vm-test-run-nix-grpc-farm
checks.x86_64-linux.farm
· build #500
· raw
1tribuchet: building on jamie2Machine state will be reset. To keep it, pass --keep-machine-state3start all VLans4(finished: start all VLans, in 0.00 seconds)5Test will time out and terminate in 900.0 seconds6run the VM test script7start all VMs8client: starting vm9lb: starting vm10worker1: starting vm11worker2: starting vm12worker2: QEMU running (pid 52)13worker2 # Disk image does not exist, creating the virtualisation disk image...14worker2 # Formatting '/build/vm-state-worker2/tmp.wNXf568WLQ', fmt=raw size=107374182415worker2 # mke2fs 1.47.4 (6-Mar-2025)16worker2 # Discarding device blocks: 0/262144 done17worker2 # Creating filesystem with 262144 4k blocks and 65536 inodes18worker2 # Filesystem UUID: 33199183-72be-4eec-a8de-c54114813c5319worker2 # Superblock backups stored on blocks:20worker2 # 32768, 98304, 163840, 22937621worker2 # 22worker2 # Allocating group tables: 0/8 done23worker2 # Writing inode tables: 0/8 done24worker2 # Creating journal (8192 blocks): done25worker2 # Writing superblocks and filesystem accounting information: 0/8 done26worker2 # 27worker2 # Virtualisation disk image created.28worker2 # Starting virtiofs daemons...29worker2 # [2026-09-22T14:50:19Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)30worker2 # [2026-09-22T14:50:19Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether31worker2 # [2026-09-22T14:50:19Z INFO virtiofsd] Waiting for vhost-user socket connection...32worker2 # [2026-09-22T14:50:19Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)33worker2 # [2026-09-22T14:50:19Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether34worker2 # [2026-09-22T14:50:19Z INFO virtiofsd] Waiting for vhost-user socket connection...35worker2 # [2026-09-22T14:50:19Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)36worker2 # [2026-09-22T14:50:19Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether37worker2 # [2026-09-22T14:50:19Z INFO virtiofsd] Waiting for vhost-user socket connection...38worker2 # [2026-09-22T14:50:19Z INFO virtiofsd] Client connected, servicing requests39worker2 # [2026-09-22T14:50:19Z INFO virtiofsd] Client connected, servicing requests40worker2 # [2026-09-22T14:50:19Z INFO virtiofsd] Client connected, servicing requests41lb # Disk image does not exist, creating the virtualisation disk image...42lb # Formatting '/build/vm-state-lb/tmp.LjjZw4AGYn', fmt=raw size=107374182443lb # mke2fs 1.47.4 (6-Mar-2025)44lb: QEMU running (pid 51)45lb # Discarding device blocks: 0/262144 done46lb # Creating filesystem with 262144 4k blocks and 65536 inodes47lb # Filesystem UUID: deb1ec60-e1b1-444c-b989-af5ef063bacb48lb # Superblock backups stored on blocks:49lb # 32768, 98304, 163840, 22937650lb # 51lb # Allocating group tables: 0/8 done52lb # Writing inode tables: 0/8 done53lb # Creating journal (8192 blocks): done54client # Disk image does not exist, creating the virtualisation disk image...55worker1: QEMU running (pid 50)56worker1 # Disk image does not exist, creating the virtualisation disk image...57client # Formatting '/build/vm-state-client/tmp.bH2lE5IMYY', fmt=raw size=107374182458client: QEMU running (pid 49)59lb # Writing superblocks and filesystem accounting information: 0/8 done60worker1 # Formatting '/build/vm-state-worker1/tmp.8KlaXJeLk7', fmt=raw size=107374182461lb # 62worker1 # mke2fs 1.47.4 (6-Mar-2025)63lb # Virtualisation disk image created.64client # mke2fs 1.47.4 (6-Mar-2025)65(finished: start all VMs, in 0.26 seconds)66worker1 # Discarding device blocks: 0/262144 done67lb # Starting virtiofs daemons...68client # Discarding device blocks: 0/262144 done69client: waiting for unit network-addresses-eth1.service70worker1 # Creating filesystem with 262144 4k blocks and 65536 inodes71lb # [2026-09-22T14:50:19Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)72client # Creating filesystem with 262144 4k blocks and 65536 inodes73client: waiting for the VM to finish booting74worker1 # Filesystem UUID: 31e3bac9-5ec2-4403-a54a-33a74a77c30975lb # [2026-09-22T14:50:19Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether76worker1 # Superblock backups stored on blocks:77lb # [2026-09-22T14:50:19Z INFO virtiofsd] Waiting for vhost-user socket connection...78worker1 # 32768, 98304, 163840, 22937679lb # [2026-09-22T14:50:19Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)80worker1 # 81lb # [2026-09-22T14:50:19Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether82worker1 # Allocating group tables: 0/8 done83lb # [2026-09-22T14:50:19Z INFO virtiofsd] Waiting for vhost-user socket connection...84worker1 # Writing inode tables: 0/8 done85lb # [2026-09-22T14:50:19Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)86worker1 # Creating journal (8192 blocks): done87lb # [2026-09-22T14:50:19Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether88worker1 # Writing superblocks and filesystem accounting information: 0/8 done89lb # [2026-09-22T14:50:19Z INFO virtiofsd] Waiting for vhost-user socket connection...90worker1 # 91lb # [2026-09-22T14:50:19Z INFO virtiofsd] Client connected, servicing requests92worker1 # Virtualisation disk image created.93lb # [2026-09-22T14:50:19Z INFO virtiofsd] Client connected, servicing requests94worker1 # Starting virtiofs daemons...95lb # [2026-09-22T14:50:19Z INFO virtiofsd] Client connected, servicing requests96worker1 # [2026-09-22T14:50:19Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)97client # Filesystem UUID: cc94140d-4804-45d0-bc2a-57c9123b713c98worker1 # [2026-09-22T14:50:19Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether99client # Superblock backups stored on blocks:100worker1 # [2026-09-22T14:50:19Z INFO virtiofsd] Waiting for vhost-user socket connection...101client # 32768, 98304, 163840, 229376102worker1 # [2026-09-22T14:50:19Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)103client # 104worker1 # [2026-09-22T14:50:19Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether105client # Allocating group tables: 0/8 done106worker1 # [2026-09-22T14:50:19Z INFO virtiofsd] Waiting for vhost-user socket connection...107client # Writing inode tables: 0/8 done108worker1 # [2026-09-22T14:50:19Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)109client # Creating journal (8192 blocks): done110worker1 # [2026-09-22T14:50:19Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether111client # Writing superblocks and filesystem accounting information: 0/8 done112worker1 # [2026-09-22T14:50:19Z INFO virtiofsd] Waiting for vhost-user socket connection...113client # 114worker1 # [2026-09-22T14:50:19Z INFO virtiofsd] Client connected, servicing requests115client # Virtualisation disk image created.116worker1 # [2026-09-22T14:50:19Z INFO virtiofsd] Client connected, servicing requests117client # Starting virtiofs daemons...118worker1 # [2026-09-22T14:50:19Z INFO virtiofsd] Client connected, servicing requests119client # [2026-09-22T14:50:19Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)120client # [2026-09-22T14:50:19Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether121client # [2026-09-22T14:50:19Z INFO virtiofsd] Waiting for vhost-user socket connection...122client # [2026-09-22T14:50:19Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)123client # [2026-09-22T14:50:19Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether124client # [2026-09-22T14:50:19Z INFO virtiofsd] Waiting for vhost-user socket connection...125client # [2026-09-22T14:50:19Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)126client # [2026-09-22T14:50:19Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether127client # [2026-09-22T14:50:19Z INFO virtiofsd] Waiting for vhost-user socket connection...128client # [2026-09-22T14:50:19Z INFO virtiofsd] Client connected, servicing requests129client # [2026-09-22T14:50:19Z INFO virtiofsd] Client connected, servicing requests130client # [2026-09-22T14:50:19Z INFO virtiofsd] Client connected, servicing requests131lb # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)132client # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)133worker2 # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)134worker1 # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)135lb # 136lb # 137client # 138client # 139lb # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA00140client # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA00141lb # Press Ctrl-B to configure iPXE (PCI 00:02.0)...142lb # 143lb # 144lb # 145lb # 146client # Press Ctrl-B to configure iPXE (PCI 00:02.0)...147client # 148client # 149client # 150client # 151lb # iPXE (http://ipxe.org) 00:05.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00152client # iPXE (http://ipxe.org) 00:05.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00153worker2 # 154worker2 # 155worker2 # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA00156lb # Press Ctrl-B to configure iPXE (PCI 00:05.0)...157lb # 158lb # 159client # Press Ctrl-B to configure iPXE (PCI 00:05.0)...160client # 161client # 162worker2 # Press Ctrl-B to configure iPXE (PCI 00:02.0)...163worker2 # 164worker2 # 165worker2 # 166worker2 # 167worker2 # iPXE (http://ipxe.org) 00:05.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00168worker2 # Press Ctrl-B to configure iPXE (PCI 00:05.0)...169worker2 # 170worker2 # 171worker1 # 172worker1 # 173worker1 # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA00174worker1 # Press Ctrl-B to configure iPXE (PCI 00:02.0)...175worker1 # 176worker1 # 177worker1 # 178worker1 # 179worker1 # iPXE (http://ipxe.org) 00:05.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00180lb # Booting from ROM...181worker1 # Press Ctrl-B to configure iPXE (PCI 00:05.0)...182worker1 # 183worker1 # 184client # Booting from ROM...185worker2 # Booting from ROM...186worker1 # Booting from ROM...187worker1 # Probing EDD (edd=off to disable)... ok188lb # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 14 11:36:19 UTC 2026189lb # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/qfr7fghmwm537hb3f4f5is3yb8phm6jx-nixos-system-lb-test/init regInfo=/nix/store/s1480xaqc00gh7hz31b4z2fwjpnxi3sa-closure-info/registration console=ttyS0,115200n8 console=tty0190lb # [ 0.000000] BIOS-provided physical RAM map:191lb # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable192lb # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved193lb # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved194lb # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable195client # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 14 11:36:19 UTC 2026196lb # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved197lb # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved198lb # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved199lb # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved200lb # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved201client # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/a8pdxq53s01ng2mkk3f5madc7gwsg7dg-nixos-system-client-test/init regInfo=/nix/store/ynmnrh676rhn2hc10z099m3ayrwjxhli-closure-info/registration console=ttyS0,115200n8 console=tty0202client # [ 0.000000] BIOS-provided physical RAM map:203lb # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved204client # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable205lb # [ 0.000000] NX (Execute Disable) protection: active206worker2 # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 14 11:36:19 UTC 2026207lb # [ 0.000000] APIC: Static calls initialized208client # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved209lb # [ 0.000000] SMBIOS 2.8 present.210client # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved211lb # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014212client # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable213lb # [ 0.000000] DMI: Memory slots populated: 1/1214lb # [ 0.000000] Hypervisor detected: KVM215client # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved216lb # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000217worker2 # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/y3l0b39dv4r3c1f6dc92g5l6qissh9w6-nixos-system-worker2-test/init regInfo=/nix/store/xmack8hdf3hqn7r1jh8r53m2jf1qpal2-closure-info/registration console=ttyS0,115200n8 console=tty0218client # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved219lb # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00220worker2 # [ 0.000000] BIOS-provided physical RAM map:221client # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved222lb # [ 0.000000] kvm-clock: using sched offset of 461751990 cycles223worker2 # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable224client # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved225worker2 # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved226lb # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns227client # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved228lb # [ 0.000005] tsc: Detected 2400.012 MHz processor229worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved230client # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved231lb # [ 0.000811] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000232worker2 # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable233client # [ 0.000000] NX (Execute Disable) protection: active234client # [ 0.000000] APIC: Static calls initialized235lb # [ 0.000838] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs236worker2 # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved237client # [ 0.000000] SMBIOS 2.8 present.238lb # [ 0.000841] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT239worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved240client # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014241lb # [ 0.002731] found SMP MP-table at [mem 0x000f5450-0x000f545f]242worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved243client # [ 0.000000] DMI: Memory slots populated: 1/1244lb # [ 0.002743] Using GB pages for direct mapping245client # [ 0.000000] Hypervisor detected: KVM246worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved247lb # [ 0.002869] RAMDISK: [mem 0x5e36e000-0x5ffcffff]248client # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000249lb # [ 0.002878] ACPI: Early table checksum verification disabled250worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved251client # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00252lb # [ 0.002880] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )253worker2 # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved254client # [ 0.000001] kvm-clock: using sched offset of 472392748 cycles255worker2 # [ 0.000000] NX (Execute Disable) protection: active256lb # [ 0.002884] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)257worker2 # [ 0.000000] APIC: Static calls initialized258client # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns259worker2 # [ 0.000000] SMBIOS 2.8 present.260lb # [ 0.002888] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)261client # [ 0.000005] tsc: Detected 2400.012 MHz processor262client # [ 0.000813] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000263worker2 # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014264lb # [ 0.002895] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)265worker2 # [ 0.000000] DMI: Memory slots populated: 1/1266lb # [ 0.002897] ACPI: FACS 0x000000005FFE0000 000040267worker2 # [ 0.000000] Hypervisor detected: KVM268client # [ 0.000840] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs269worker2 # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000270lb # [ 0.002899] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)271client # [ 0.000843] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT272worker2 # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00273client # [ 0.002723] found SMP MP-table at [mem 0x000f5450-0x000f545f]274lb # [ 0.002900] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)275worker2 # [ 0.000000] kvm-clock: using sched offset of 483593901 cycles276client # [ 0.002734] Using GB pages for direct mapping277client # [ 0.002855] RAMDISK: [mem 0x5e370000-0x5ffcffff]278lb # [ 0.002902] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)279client # [ 0.002864] ACPI: Early table checksum verification disabled280worker2 # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns281lb # [ 0.002903] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)282worker2 # [ 0.000005] tsc: Detected 2400.012 MHz processor283client # [ 0.002867] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )284worker2 # [ 0.000820] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000285lb # [ 0.002904] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]286client # [ 0.002871] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)287worker2 # [ 0.000847] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs288lb # [ 0.002905] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]289client # [ 0.002875] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)290worker2 # [ 0.000849] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT291lb # [ 0.002906] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]292client # [ 0.002882] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)293worker2 # [ 0.002743] found SMP MP-table at [mem 0x000f5450-0x000f545f]294lb # [ 0.002906] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]295client # [ 0.002884] ACPI: FACS 0x000000005FFE0000 000040296worker2 # [ 0.002754] Using GB pages for direct mapping297lb # [ 0.002907] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]298worker2 # [ 0.002899] RAMDISK: [mem 0x5e36f000-0x5ffcffff]299client # [ 0.002886] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)300worker2 # [ 0.002908] ACPI: Early table checksum verification disabled301lb # [ 0.002907] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]302worker2 # [ 0.002911] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )303client # [ 0.002888] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)304lb # [ 0.002908] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]305lb # [ 0.003133] No NUMA configuration found306worker2 # [ 0.002915] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)307client # [ 0.002889] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)308lb # [ 0.003134] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]309client # [ 0.002891] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)310worker2 # [ 0.002919] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)311lb # [ 0.003143] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]312lb # [ 0.005774] Zone ranges:313client # [ 0.002892] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]314worker2 # [ 0.002926] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)315lb # [ 0.005775] DMA [mem 0x0000000000001000-0x0000000000ffffff]316worker2 # [ 0.002928] ACPI: FACS 0x000000005FFE0000 000040317client # [ 0.002893] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]318lb # [ 0.005776] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]319lb # [ 0.005777] Normal empty320worker1 # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 14 11:36:19 UTC 2026321client # [ 0.002893] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]322lb # [ 0.005778] Device empty323worker2 # [ 0.002929] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)324lb # [ 0.005778] Movable zone start for each node325client # [ 0.002894] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]326worker2 # [ 0.002931] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)327lb # [ 0.005779] Early memory node ranges328client # [ 0.002894] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]329lb # [ 0.005779] node 0: [mem 0x0000000000001000-0x000000000009efff]330worker2 # [ 0.002933] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)331client # [ 0.002895] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]332lb # [ 0.005781] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]333worker2 # [ 0.002934] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)334client # [ 0.002895] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]335worker1 # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/jgp4wsjngw95hp4pxq5znxv19ks4ns55-nixos-system-worker1-test/init regInfo=/nix/store/9k9kms4c76xsaw26sj0n98apgzdm7i27-closure-info/registration console=ttyS0,115200n8 console=tty0336lb # [ 0.005782] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]337client # [ 0.003128] No NUMA configuration found338worker2 # [ 0.002935] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]339worker1 # [ 0.000000] BIOS-provided physical RAM map:340lb # [ 0.005801] On node 0, zone DMA: 1 pages in unavailable ranges341client # [ 0.003129] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]342worker1 # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable343lb # [ 0.006068] On node 0, zone DMA: 97 pages in unavailable ranges344worker2 # [ 0.002937] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]345client # [ 0.003131] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]346lb # [ 0.033405] On node 0, zone DMA32: 40 pages in unavailable ranges347client # [ 0.005402] Zone ranges:348worker1 # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved349worker2 # [ 0.002937] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]350lb # [ 0.033858] ACPI: PM-Timer IO Port: 0x608351client # [ 0.005403] DMA [mem 0x0000000000001000-0x0000000000ffffff]352worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved353worker2 # [ 0.002938] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]354lb # [ 0.033869] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])355client # [ 0.005405] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]356worker1 # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable357worker2 # [ 0.002938] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]358client # [ 0.005406] Normal empty359lb # [ 0.033895] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23360client # [ 0.005407] Device empty361worker1 # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved362worker2 # [ 0.002939] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]363client # [ 0.005407] Movable zone start for each node364lb # [ 0.033897] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)365client # [ 0.005408] Early memory node ranges366worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved367worker2 # [ 0.002939] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]368lb # [ 0.033899] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)369client # [ 0.005408] node 0: [mem 0x0000000000001000-0x000000000009efff]370worker2 # [ 0.003174] No NUMA configuration found371worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved372lb # [ 0.033900] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)373worker2 # [ 0.003175] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]374client # [ 0.005409] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]375worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved376lb # [ 0.033901] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)377worker2 # [ 0.003177] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]378client # [ 0.005410] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]379worker2 # [ 0.005492] Zone ranges:380worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved381lb # [ 0.033901] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)382client # [ 0.005432] On node 0, zone DMA: 1 pages in unavailable ranges383worker2 # [ 0.005493] DMA [mem 0x0000000000001000-0x0000000000ffffff]384worker1 # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved385lb # [ 0.033904] ACPI: Using ACPI (MADT) for SMP configuration information386client # [ 0.005692] On node 0, zone DMA: 97 pages in unavailable ranges387worker1 # [ 0.000000] NX (Execute Disable) protection: active388worker2 # [ 0.005495] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]389lb # [ 0.033905] ACPI: HPET id: 0x8086a201 base: 0xfed00000390worker2 # [ 0.005496] Normal empty391client # [ 0.031965] On node 0, zone DMA32: 40 pages in unavailable ranges392worker2 # [ 0.005497] Device empty393lb # [ 0.033908] TSC deadline timer available394worker1 # [ 0.000000] APIC: Static calls initialized395client # [ 0.032426] ACPI: PM-Timer IO Port: 0x608396worker1 # [ 0.000000] SMBIOS 2.8 present.397lb # [ 0.033911] CPU topo: Max. logical packages: 1398worker2 # [ 0.005498] Movable zone start for each node399worker2 # [ 0.005499] Early memory node ranges400client # [ 0.032437] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])401lb # [ 0.033912] CPU topo: Max. logical dies: 1402worker1 # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014403lb # [ 0.033912] CPU topo: Max. dies per package: 1404worker2 # [ 0.005499] node 0: [mem 0x0000000000001000-0x000000000009efff]405client # [ 0.032466] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23406worker1 # [ 0.000000] DMI: Memory slots populated: 1/1407lb # [ 0.033916] CPU topo: Max. threads per core: 1408worker1 # [ 0.000000] Hypervisor detected: KVM409worker2 # [ 0.005500] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]410client # [ 0.032469] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)411lb # [ 0.033916] CPU topo: Num. cores per package: 1412worker1 # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000413worker2 # [ 0.005501] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]414client # [ 0.032470] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)415lb # [ 0.033917] CPU topo: Num. threads per package: 1416worker1 # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00417worker2 # [ 0.005505] On node 0, zone DMA: 1 pages in unavailable ranges418client # [ 0.032472] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)419lb # [ 0.033917] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs420worker1 # [ 0.000001] kvm-clock: using sched offset of 502198679 cycles421worker2 # [ 0.005520] On node 0, zone DMA: 97 pages in unavailable ranges422client # [ 0.032472] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)423lb # [ 0.033934] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()424worker2 # [ 0.028969] On node 0, zone DMA32: 40 pages in unavailable ranges425worker1 # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns426client # [ 0.032473] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)427lb # [ 0.033963] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]428worker2 # [ 0.029428] ACPI: PM-Timer IO Port: 0x608429worker1 # [ 0.000005] tsc: Detected 2400.012 MHz processor430client # [ 0.032476] ACPI: Using ACPI (MADT) for SMP configuration information431worker2 # [ 0.029440] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])432lb # [ 0.033964] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]433worker1 # [ 0.000817] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000434client # [ 0.032476] ACPI: HPET id: 0x8086a201 base: 0xfed00000435worker2 # [ 0.029469] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23436lb # [ 0.033965] [mem 0x60000000-0xafffffff] available for PCI devices437client # [ 0.032480] TSC deadline timer available438worker1 # [ 0.000844] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs439lb # [ 0.033967] Booting paravirtualized kernel on KVM440client # [ 0.032484] CPU topo: Max. logical packages: 1441worker2 # [ 0.029472] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)442client # [ 0.032485] CPU topo: Max. logical dies: 1443worker1 # [ 0.000847] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT444worker2 # [ 0.029474] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)445client # [ 0.032486] CPU topo: Max. dies per package: 1446lb # [ 0.033969] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns447worker1 # [ 0.002758] found SMP MP-table at [mem 0x000f5450-0x000f545f]448client # [ 0.032489] CPU topo: Max. threads per core: 1449worker2 # [ 0.029475] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)450worker1 # [ 0.002769] Using GB pages for direct mapping451lb # [ 0.038441] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1452client # [ 0.032490] CPU topo: Num. cores per package: 1453worker1 # [ 0.002921] RAMDISK: [mem 0x5e370000-0x5ffcffff]454worker2 # [ 0.029476] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)455lb # [ 0.040650] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152456client # [ 0.032490] CPU topo: Num. threads per package: 1457worker1 # [ 0.002930] ACPI: Early table checksum verification disabled458worker2 # [ 0.029476] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)459client # [ 0.032490] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs460lb # [ 0.040693] kvm-guest: PV spinlocks disabled, single CPU461worker1 # [ 0.002933] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )462worker2 # [ 0.029479] ACPI: Using ACPI (MADT) for SMP configuration information463client # [ 0.032509] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()464worker2 # [ 0.029480] ACPI: HPET id: 0x8086a201 base: 0xfed00000465worker1 # [ 0.002937] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)466worker2 # [ 0.029496] TSC deadline timer available467client # [ 0.032539] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]468worker2 # [ 0.029500] CPU topo: Max. logical packages: 1469worker1 # [ 0.002941] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)470client # [ 0.032541] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]471worker2 # [ 0.029501] CPU topo: Max. logical dies: 1472worker2 # [ 0.029501] CPU topo: Max. dies per package: 1473worker1 # [ 0.002948] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)474client # [ 0.032542] [mem 0x60000000-0xafffffff] available for PCI devices475lb # [ 0.040695] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/qfr7fghmwm537hb3f4f5is3yb8phm6jx-nixos-system-lb-test/init regInfo=/nix/store/s1480xaqc00gh7hz31b4z2fwjpnxi3sa-closure-info/registration console=ttyS0,115200n8 console=tty0476worker2 # [ 0.029504] CPU topo: Max. threads per core: 1477worker1 # [ 0.002950] ACPI: FACS 0x000000005FFE0000 000040478client # [ 0.032544] Booting paravirtualized kernel on KVM479worker2 # [ 0.029505] CPU topo: Num. cores per package: 1480worker1 # [ 0.002951] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)481worker2 # [ 0.029505] CPU topo: Num. threads per package: 1482client # [ 0.032547] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns483lb # [ 0.040787] Unknown kernel command line parameters "regInfo=/nix/store/s1480xaqc00gh7hz31b4z2fwjpnxi3sa-closure-info/registration", will be passed to user space.484worker2 # [ 0.029506] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs485lb # [ 0.040800] random: crng init done486worker1 # [ 0.002953] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)487client # [ 0.037056] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1488worker2 # [ 0.029524] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()489lb # [ 0.040801] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes490worker1 # [ 0.002955] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)491client # [ 0.039731] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152492worker2 # [ 0.029554] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]493client # [ 0.039783] kvm-guest: PV spinlocks disabled, single CPU494lb # [ 0.042977] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)495worker1 # [ 0.002956] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)496worker2 # [ 0.029556] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]497lb # [ 0.044059] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)498worker1 # [ 0.002957] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]499lb # [ 0.044097] Fallback order for Node 0: 0500worker2 # [ 0.029557] [mem 0x60000000-0xafffffff] available for PCI devices501worker1 # [ 0.002958] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]502worker2 # [ 0.029559] Booting paravirtualized kernel on KVM503lb # [ 0.044100] Built 1 zonelists, mobility grouping on. Total pages: 393078504lb # [ 0.044101] Policy zone: DMA32505worker1 # [ 0.002959] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]506worker2 # [ 0.029562] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns507lb # [ 0.047310] mem auto-init: stack:all(zero), heap alloc:on, heap free:off508worker1 # [ 0.002959] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]509client # [ 0.039784] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/a8pdxq53s01ng2mkk3f5madc7gwsg7dg-nixos-system-client-test/init regInfo=/nix/store/ynmnrh676rhn2hc10z099m3ayrwjxhli-closure-info/registration console=ttyS0,115200n8 console=tty0510worker2 # [ 0.034052] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1511lb # [ 0.050451] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1512worker1 # [ 0.002960] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]513lb # [ 0.053841] allocated 3145728 bytes of page_ext514worker2 # [ 0.034168] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152515worker1 # [ 0.002960] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]516client # [ 0.039879] Unknown kernel command line parameters "regInfo=/nix/store/ynmnrh676rhn2hc10z099m3ayrwjxhli-closure-info/registration", will be passed to user space.517lb # [ 0.063576] ftrace: allocating 48787 entries in 192 pages518worker2 # [ 0.034215] kvm-guest: PV spinlocks disabled, single CPU519client # [ 0.039891] random: crng init done520lb # [ 0.063578] ftrace: allocated 192 pages with 2 groups521worker1 # [ 0.002961] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]522lb # [ 0.064398] Dynamic Preempt: lazy523client # [ 0.039892] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes524worker1 # [ 0.003195] No NUMA configuration found525lb # [ 0.064545] rcu: Preemptible hierarchical RCU implementation.526worker1 # [ 0.003196] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]527client # [ 0.042143] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)528lb # [ 0.064545] rcu: RCU event tracing is enabled.529worker1 # [ 0.003198] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]530worker1 # [ 0.005508] Zone ranges:531client # [ 0.043270] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)532lb # [ 0.064546] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.533client # [ 0.043301] Fallback order for Node 0: 0534lb # [ 0.064548] Trampoline variant of Tasks RCU enabled.535worker1 # [ 0.005509] DMA [mem 0x0000000000001000-0x0000000000ffffff]536worker2 # [ 0.034217] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/y3l0b39dv4r3c1f6dc92g5l6qissh9w6-nixos-system-worker2-test/init regInfo=/nix/store/xmack8hdf3hqn7r1jh8r53m2jf1qpal2-closure-info/registration console=ttyS0,115200n8 console=tty0537lb # [ 0.064548] Rude variant of Tasks RCU enabled.538client # [ 0.043305] Built 1 zonelists, mobility grouping on. Total pages: 393078539worker1 # [ 0.005511] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]540lb # [ 0.064548] Tracing variant of Tasks RCU enabled.541worker1 # [ 0.005512] Normal empty542client # [ 0.043306] Policy zone: DMA32543worker1 # [ 0.005513] Device empty544client # [ 0.045905] mem auto-init: stack:all(zero), heap alloc:on, heap free:off545worker2 # [ 0.034311] Unknown kernel command line parameters "regInfo=/nix/store/xmack8hdf3hqn7r1jh8r53m2jf1qpal2-closure-info/registration", will be passed to user space.546lb # [ 0.064549] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.547worker1 # [ 0.005514] Movable zone start for each node548worker1 # [ 0.005514] Early memory node ranges549worker2 # [ 0.034323] random: crng init done550client # [ 0.049041] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1551lb # [ 0.064550] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1552client # [ 0.052575] allocated 3145728 bytes of page_ext553worker1 # [ 0.005515] node 0: [mem 0x0000000000001000-0x000000000009efff]554worker2 # [ 0.034324] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes555client # [ 0.062668] ftrace: allocating 48787 entries in 192 pages556lb # [ 0.064603] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.557worker1 # [ 0.005516] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]558worker2 # [ 0.034412] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)559client # [ 0.062670] ftrace: allocated 192 pages with 2 groups560client # [ 0.063517] Dynamic Preempt: lazy561worker1 # [ 0.005517] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]562lb # [ 0.064605] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.563worker2 # [ 0.035489] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)564worker1 # [ 0.005537] On node 0, zone DMA: 1 pages in unavailable ranges565client # [ 0.063678] rcu: Preemptible hierarchical RCU implementation.566worker2 # [ 0.035519] Fallback order for Node 0: 0567lb # [ 0.064605] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.568client # [ 0.063679] rcu: RCU event tracing is enabled.569worker1 # [ 0.005798] On node 0, zone DMA: 97 pages in unavailable ranges570worker2 # [ 0.035522] Built 1 zonelists, mobility grouping on. Total pages: 393078571lb # [ 0.069007] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16572client # [ 0.063680] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.573worker2 # [ 0.035523] Policy zone: DMA32574worker1 # [ 0.032392] On node 0, zone DMA32: 40 pages in unavailable ranges575worker1 # [ 0.032855] ACPI: PM-Timer IO Port: 0x608576client # [ 0.063681] Trampoline variant of Tasks RCU enabled.577lb # [ 0.069295] rcu: srcu_init: Setting srcu_struct sizes based on contention.578worker2 # [ 0.037948] mem auto-init: stack:all(zero), heap alloc:on, heap free:off579client # [ 0.063681] Rude variant of Tasks RCU enabled.580worker1 # [ 0.032868] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])581worker2 # [ 0.041028] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1582client # [ 0.063681] Tracing variant of Tasks RCU enabled.583lb # [ 0.069302] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns584worker1 # [ 0.032897] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23585worker2 # [ 0.044344] allocated 3145728 bytes of page_ext586client # [ 0.063682] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.587worker2 # [ 0.054335] ftrace: allocating 48787 entries in 192 pages588worker1 # [ 0.032900] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)589lb # [ 0.069404] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)590worker2 # [ 0.054337] ftrace: allocated 192 pages with 2 groups591client # [ 0.063683] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1592lb # [ 0.072932] Console: colour VGA+ 80x25593worker1 # [ 0.032902] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)594worker2 # [ 0.055389] Dynamic Preempt: lazy595lb # [ 0.072935] printk: legacy console [tty0] enabled596worker1 # [ 0.032903] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)597client # [ 0.063697] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.598worker2 # [ 0.055575] rcu: Preemptible hierarchical RCU implementation.599lb # [ 0.102497] printk: legacy console [ttyS0] enabled600worker2 # [ 0.055576] rcu: RCU event tracing is enabled.601worker1 # [ 0.032904] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)602lb # [ 0.198813] ACPI: Core revision 20250807603client # [ 0.063698] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.604worker2 # [ 0.055577] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.605worker1 # [ 0.032905] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)606worker2 # [ 0.055579] Trampoline variant of Tasks RCU enabled.607client # [ 0.063699] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.608worker1 # [ 0.032907] ACPI: Using ACPI (MADT) for SMP configuration information609worker2 # [ 0.055579] Rude variant of Tasks RCU enabled.610lb # [ 0.199740] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns611client # [ 0.068081] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16612worker1 # [ 0.032908] ACPI: HPET id: 0x8086a201 base: 0xfed00000613worker2 # [ 0.055579] Tracing variant of Tasks RCU enabled.614worker1 # [ 0.032912] TSC deadline timer available615lb # [ 0.201222] APIC: Switch to symmetric I/O mode setup616client # [ 0.068357] rcu: srcu_init: Setting srcu_struct sizes based on contention.617worker2 # [ 0.055580] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.618worker1 # [ 0.032916] CPU topo: Max. logical packages: 1619lb # [ 0.202206] x2apic enabled620worker1 # [ 0.032917] CPU topo: Max. logical dies: 1621worker2 # [ 0.055581] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1622client # [ 0.068363] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns623worker1 # [ 0.032918] CPU topo: Max. dies per package: 1624lb # [ 0.202980] APIC: Switched APIC routing to: physical x2apic625worker1 # [ 0.032921] CPU topo: Max. threads per core: 1626worker2 # [ 0.055600] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.627client # [ 0.068463] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)628worker1 # [ 0.032921] CPU topo: Num. cores per package: 1629client # [ 0.072048] Console: colour VGA+ 80x25630worker1 # [ 0.032922] CPU topo: Num. threads per package: 1631worker2 # [ 0.055602] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.632client # [ 0.072050] printk: legacy console [tty0] enabled633lb # [ 0.204763] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1634worker1 # [ 0.032922] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs635client # [ 0.101943] printk: legacy console [ttyS0] enabled636worker2 # [ 0.055602] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.637worker1 # [ 0.032941] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()638client # [ 0.197638] ACPI: Core revision 20250807639worker2 # [ 0.060026] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16640lb # [ 0.205723] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns641worker1 # [ 0.032972] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]642worker2 # [ 0.060311] rcu: srcu_init: Setting srcu_struct sizes based on contention.643client # [ 0.198464] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns644worker1 # [ 0.032974] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]645lb # [ 0.207369] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)646worker2 # [ 0.060318] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns647client # [ 0.199955] APIC: Switch to symmetric I/O mode setup648worker1 # [ 0.032976] [mem 0x60000000-0xafffffff] available for PCI devices649lb # [ 0.208455] x86/cpu: User Mode Instruction Prevention (UMIP) activated650client # [ 0.200912] x2apic enabled651worker1 # [ 0.032977] Booting paravirtualized kernel on KVM652worker2 # [ 0.060425] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)653worker2 # [ 0.064206] Console: colour VGA+ 80x25654lb # [ 0.209507] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127655worker1 # [ 0.032980] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns656client # [ 0.201688] APIC: Switched APIC routing to: physical x2apic657worker2 # [ 0.064210] printk: legacy console [tty0] enabled658worker2 # [ 0.093839] printk: legacy console [ttyS0] enabled659worker1 # [ 0.037500] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1660lb # [ 0.210367] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0661worker2 # [ 0.190170] ACPI: Core revision 20250807662worker1 # [ 0.039684] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152663client # [ 0.203566] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1664worker1 # [ 0.039731] kvm-guest: PV spinlocks disabled, single CPU665lb # [ 0.211371] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto666worker2 # [ 0.191024] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns667client # [ 0.204540] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns668lb # [ 0.213367] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl669worker2 # [ 0.192566] APIC: Switch to symmetric I/O mode setup670worker2 # [ 0.193514] x2apic enabled671lb # [ 0.214367] Transient Scheduler Attacks: Vulnerable: No microcode672client # [ 0.206188] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)673worker2 # [ 0.194266] APIC: Switched APIC routing to: physical x2apic674lb # [ 0.215367] Spectre V2 : Mitigation: Enhanced / Automatic IBRS675worker1 # [ 0.039733] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/jgp4wsjngw95hp4pxq5znxv19ks4ns55-nixos-system-worker1-test/init regInfo=/nix/store/9k9kms4c76xsaw26sj0n98apgzdm7i27-closure-info/registration console=ttyS0,115200n8 console=tty0676client # [ 0.208269] x86/cpu: User Mode Instruction Prevention (UMIP) activated677lb # [ 0.216367] Speculative Return Stack Overflow: Mitigation: Safe RET678client # [ 0.209321] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127679worker1 # [ 0.039827] Unknown kernel command line parameters "regInfo=/nix/store/9k9kms4c76xsaw26sj0n98apgzdm7i27-closure-info/registration", will be passed to user space.680worker2 # [ 0.196062] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1681worker1 # [ 0.039839] random: crng init done682client # [ 0.210846] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0683worker1 # [ 0.039840] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes684worker2 # [ 0.197023] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns685client # [ 0.211183] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto686worker1 # [ 0.041949] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)687worker2 # [ 0.198705] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)688worker1 # [ 0.043204] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)689lb # [ 0.217367] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization690client # [ 0.213180] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl691worker1 # [ 0.043235] Fallback order for Node 0: 0692lb # [ 0.219374] Spectre V2 : Enabling IBPB for BPF693worker2 # [ 0.199791] x86/cpu: User Mode Instruction Prevention (UMIP) activated694worker1 # [ 0.043238] Built 1 zonelists, mobility grouping on. Total pages: 393078695client # [ 0.214180] Transient Scheduler Attacks: Vulnerable: No microcode696worker1 # [ 0.043239] Policy zone: DMA32697worker2 # [ 0.200838] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127698lb # [ 0.220367] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier699client # [ 0.215179] Spectre V2 : Mitigation: Enhanced / Automatic IBRS700worker1 # [ 0.045756] mem auto-init: stack:all(zero), heap alloc:on, heap free:off701lb # [ 0.221368] active return thunk: srso_alias_return_thunk702worker2 # [ 0.201702] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0703worker1 # [ 0.048915] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1704client # [ 0.216180] Speculative Return Stack Overflow: Mitigation: Safe RET705worker1 # [ 0.052214] allocated 3145728 bytes of page_ext706lb # [ 0.222388] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'707worker1 # [ 0.061897] ftrace: allocating 48787 entries in 192 pages708worker2 # [ 0.202706] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto709worker1 # [ 0.061899] ftrace: allocated 192 pages with 2 groups710lb # [ 0.223367] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'711worker1 # [ 0.062753] Dynamic Preempt: lazy712worker1 # [ 0.062938] rcu: Preemptible hierarchical RCU implementation.713worker2 # [ 0.203702] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl714lb # [ 0.224366] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'715worker1 # [ 0.062939] rcu: RCU event tracing is enabled.716worker2 # [ 0.204702] Transient Scheduler Attacks: Vulnerable: No microcode717lb # [ 0.225367] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'718client # [ 0.217179] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization719worker1 # [ 0.062940] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.720worker1 # [ 0.062941] Trampoline variant of Tasks RCU enabled.721client # [ 0.219187] Spectre V2 : Enabling IBPB for BPF722worker2 # [ 0.205702] Spectre V2 : Mitigation: Enhanced / Automatic IBRS723lb # [ 0.226367] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'724worker1 # [ 0.062942] Rude variant of Tasks RCU enabled.725worker2 # [ 0.206702] Speculative Return Stack Overflow: Mitigation: Safe RET726worker1 # [ 0.062942] Tracing variant of Tasks RCU enabled.727lb # [ 0.227366] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'728client # [ 0.220180] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier729worker1 # [ 0.062942] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.730client # [ 0.221180] active return thunk: srso_alias_return_thunk731lb # [ 0.228367] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'732worker1 # [ 0.062943] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1733lb # [ 0.229367] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256734client # [ 0.222201] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'735worker1 # [ 0.062964] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.736lb # [ 0.231058] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64737client # [ 0.223180] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'738worker1 # [ 0.062966] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.739worker2 # [ 0.207702] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization740lb # [ 0.231366] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512741client # [ 0.224894] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'742worker2 # [ 0.209709] Spectre V2 : Enabling IBPB for BPF743worker1 # [ 0.062967] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.744lb # [ 0.232367] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024745client # [ 0.225893] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'746worker1 # [ 0.067608] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16747worker2 # [ 0.210418] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier748lb # [ 0.233366] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8749client # [ 0.226180] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'750worker1 # [ 0.067891] rcu: srcu_init: Setting srcu_struct sizes based on contention.751worker2 # [ 0.211666] active return thunk: srso_alias_return_thunk752client # [ 0.227180] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'753worker1 # [ 0.067898] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns754lb # [ 0.234367] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.755worker2 # [ 0.212724] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'756client # [ 0.228179] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'757worker1 # [ 0.067999] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)758worker2 # [ 0.213702] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'759worker1 # [ 0.071548] Console: colour VGA+ 80x25760client # [ 0.230081] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256761worker1 # [ 0.071551] printk: legacy console [tty0] enabled762worker2 # [ 0.214702] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'763worker1 # [ 0.101084] printk: legacy console [ttyS0] enabled764client # [ 0.230896] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64765worker2 # [ 0.215702] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'766worker1 # [ 0.197177] ACPI: Core revision 20250807767client # [ 0.231898] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512768worker2 # [ 0.216701] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'769client # [ 0.232898] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024770worker1 # [ 0.198039] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns771worker2 # [ 0.217702] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'772client # [ 0.233895] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8773worker1 # [ 0.199629] APIC: Switch to symmetric I/O mode setup774worker2 # [ 0.218702] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'775worker1 # [ 0.200578] x2apic enabled776client # [ 0.234896] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.777worker2 # [ 0.219703] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256778worker1 # [ 0.201340] APIC: Switched APIC routing to: physical x2apic779worker2 # [ 0.220702] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64780worker2 # [ 0.221702] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512781worker1 # [ 0.203141] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1782worker2 # [ 0.222702] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024783worker1 # [ 0.204103] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns784worker2 # [ 0.223702] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8785worker1 # [ 0.205731] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)786worker2 # [ 0.224702] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.787worker1 # [ 0.206816] x86/cpu: User Mode Instruction Prevention (UMIP) activated788worker1 # [ 0.207916] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127789worker1 # [ 0.208728] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0790worker1 # [ 0.209732] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto791worker1 # [ 0.211728] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl792worker1 # [ 0.212728] Transient Scheduler Attacks: Vulnerable: No microcode793worker1 # [ 0.213728] Spectre V2 : Mitigation: Enhanced / Automatic IBRS794worker1 # [ 0.214728] Speculative Return Stack Overflow: Mitigation: Safe RET795worker1 # [ 0.215728] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization796worker1 # [ 0.217735] Spectre V2 : Enabling IBPB for BPF797worker1 # [ 0.218729] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier798worker1 # [ 0.219729] active return thunk: srso_alias_return_thunk799worker1 # [ 0.220749] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'800worker1 # [ 0.221728] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'801worker1 # [ 0.222728] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'802worker1 # [ 0.223728] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'803worker1 # [ 0.224728] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'804lb # [ 0.268727] Freeing SMP alternatives memory: 44K805worker1 # [ 0.225728] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'806lb # [ 0.269369] pid_max: default: 32768 minimum: 301807worker1 # [ 0.226728] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'808lb # [ 0.270380] LSM: initializing lsm=capability,landlock,yama,bpf,ima809lb # [ 0.271437] landlock: Up and running.810worker1 # [ 0.227728] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256811lb # [ 0.272059] Yama: becoming mindful.812worker1 # [ 0.228728] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64813lb # [ 0.272580] LSM support for eBPF active814worker1 # [ 0.229728] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512815lb # [ 0.273483] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)816worker1 # [ 0.230728] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024817worker1 # [ 0.231728] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8818lb # [ 0.274405] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)819client # [ 0.269469] Freeing SMP alternatives memory: 44K820client # [ 0.270183] pid_max: default: 32768 minimum: 301821worker1 # [ 0.232728] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.822client # [ 0.271009] LSM: initializing lsm=capability,landlock,yama,bpf,ima823lb # [ 0.277253] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)824client # [ 0.272276] landlock: Up and running.825client # [ 0.272912] Yama: becoming mindful.826client # [ 0.273393] LSM support for eBPF active827lb # [ 0.278297] Performance Events: Fam17h+ core perfctr, AMD PMU driver.828lb # [ 0.279205] ... version: 2829client # [ 0.274291] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)830lb # [ 0.279931] ... bit width: 48831worker2 # [ 0.259181] Freeing SMP alternatives memory: 44K832lb # [ 0.280386] ... generic counters: 6833worker2 # [ 0.259704] pid_max: default: 32768 minimum: 301834client # [ 0.275216] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)835lb # [ 0.281057] ... generic bitmap: 000000000000003f836worker2 # [ 0.260800] LSM: initializing lsm=capability,landlock,yama,bpf,ima837lb # [ 0.281369] ... fixed-purpose counters: 0838worker2 # [ 0.261808] landlock: Up and running.839lb # [ 0.282157] ... fixed-purpose bitmap: 0000000000000000840worker2 # [ 0.262414] Yama: becoming mindful.841client # [ 0.278240] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)842lb # [ 0.282369] ... value mask: 0000ffffffffffff843worker2 # [ 0.263327] LSM support for eBPF active844lb # [ 0.283291] ... max period: 00007fffffffffff845client # [ 0.279813] Performance Events: Fam17h+ core perfctr, AMD PMU driver.846worker2 # [ 0.263804] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)847lb # [ 0.284056] ... global_ctrl mask: 000000000000003f848client # [ 0.280188] ... version: 2849lb # [ 0.284489] signal: max sigframe size: 3376850client # [ 0.280903] ... bit width: 48851worker2 # [ 0.264740] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)852client # [ 0.281182] ... generic counters: 6853lb # [ 0.285243] rcu: Hierarchical SRCU implementation.854client # [ 0.281929] ... generic bitmap: 000000000000003f855lb # [ 0.285955] rcu: Max phase no-delay instances is 400.856client # [ 0.282183] ... fixed-purpose counters: 0857worker2 # [ 0.267779] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)858client # [ 0.282951] ... fixed-purpose bitmap: 0000000000000000859client # [ 0.283182] ... value mask: 0000ffffffffffff860worker2 # [ 0.269275] Performance Events: Fam17h+ core perfctr, AMD PMU driver.861worker2 # [ 0.269711] ... version: 2862client # [ 0.284113] ... max period: 00007fffffffffff863worker2 # [ 0.270416] ... bit width: 48864client # [ 0.284866] ... global_ctrl mask: 000000000000003f865worker2 # [ 0.270704] ... generic counters: 6866client # [ 0.285310] signal: max sigframe size: 3376867lb # [ 0.290988] smp: Bringing up secondary CPUs ...868worker2 # [ 0.271452] ... generic bitmap: 000000000000003f869lb # [ 0.291384] smp: Brought up 1 node, 1 CPU870client # [ 0.286069] rcu: Hierarchical SRCU implementation.871worker2 # [ 0.271705] ... fixed-purpose counters: 0872client # [ 0.286774] rcu: Max phase no-delay instances is 400.873lb # [ 0.292047] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)874worker2 # [ 0.272384] ... fixed-purpose bitmap: 0000000000000000875worker2 # [ 0.272704] ... value mask: 0000ffffffffffff876worker2 # [ 0.273613] ... max period: 00007fffffffffff877lb # [ 0.292572] Memory: 1454572K/1572312K available (17247K kernel code, 2727K rwdata, 13616K rodata, 3652K init, 2976K bss, 109320K reserved, 0K cma-reserved)878worker2 # [ 0.274399] ... global_ctrl mask: 000000000000003f879lb # [ 0.293596] devtmpfs: initialized880worker2 # [ 0.274801] signal: max sigframe size: 3376881lb # [ 0.294451] x86/mm: Memory block size: 128MB882worker2 # [ 0.275629] rcu: Hierarchical SRCU implementation.883client # [ 0.291860] smp: Bringing up secondary CPUs ...884worker2 # [ 0.276335] rcu: Max phase no-delay instances is 400.885client # [ 0.292195] smp: Brought up 1 node, 1 CPU886lb # [ 0.296180] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)887client # [ 0.292913] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)888lb # [ 0.297360] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).889lb # [ 0.298441] pinctrl core: initialized pinctrl subsystem890client # [ 0.293387] Memory: 1454596K/1572312K available (17247K kernel code, 2727K rwdata, 13616K rodata, 3652K init, 2976K bss, 109308K reserved, 0K cma-reserved)891lb # [ 0.299614] PM: RTC time: 14:50:19, date: 2026-09-22892client # [ 0.294427] devtmpfs: initialized893worker2 # [ 0.281264] smp: Bringing up secondary CPUs ...894client # [ 0.295256] x86/mm: Memory block size: 128MB895worker2 # [ 0.281719] smp: Brought up 1 node, 1 CPU896worker2 # [ 0.282413] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)897worker1 # [ 0.267783] Freeing SMP alternatives memory: 44K898client # [ 0.297054] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)899lb # [ 0.303155] NET: Registered PF_NETLINK/PF_ROUTE protocol family900worker1 # [ 0.268616] pid_max: default: 32768 minimum: 301901client # [ 0.298108] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).902worker2 # [ 0.282958] Memory: 1454588K/1572312K available (17247K kernel code, 2727K rwdata, 13616K rodata, 3652K init, 2976K bss, 109312K reserved, 0K cma-reserved)903worker1 # [ 0.269357] LSM: initializing lsm=capability,landlock,yama,bpf,ima904lb # [ 0.304683] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations905client # [ 0.299268] pinctrl core: initialized pinctrl subsystem906worker2 # [ 0.283973] devtmpfs: initialized907worker1 # [ 0.269828] landlock: Up and running.908worker1 # [ 0.270731] Yama: becoming mindful.909lb # [ 0.305390] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations910worker2 # [ 0.284774] x86/mm: Memory block size: 128MB911client # [ 0.300403] PM: RTC time: 14:50:19, date: 2026-09-22912worker1 # [ 0.271939] LSM support for eBPF active913lb # [ 0.306649] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations914worker1 # [ 0.272682] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)915worker2 # [ 0.286568] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)916lb # [ 0.307378] audit: initializing netlink subsys (disabled)917worker1 # [ 0.273775] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)918lb # [ 0.308495] thermal_sys: Registered thermal governor 'fair_share'919worker2 # [ 0.287607] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).920client # [ 0.304159] NET: Registered PF_NETLINK/PF_ROUTE protocol family921worker2 # [ 0.288785] pinctrl core: initialized pinctrl subsystem922lb # [ 0.308497] thermal_sys: Registered thermal governor 'bang_bang'923lb # [ 0.309371] thermal_sys: Registered thermal governor 'step_wise'924worker1 # [ 0.276160] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)925worker2 # [ 0.289886] PM: RTC time: 14:50:19, date: 2026-09-22926client # [ 0.305404] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations927lb # [ 0.310315] thermal_sys: Registered thermal governor 'user_space'928client # [ 0.306207] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations929worker1 # [ 0.277311] Performance Events: Fam17h+ core perfctr, AMD PMU driver.930worker1 # [ 0.277737] ... version: 2931lb # [ 0.311078] audit: type=2000 audit(1790088619.735:1): state=initialized audit_enabled=0 res=1932worker1 # [ 0.278444] ... bit width: 48933client # [ 0.307459] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations934lb # [ 0.312372] thermal_sys: Registered thermal governor 'power_allocator'935worker1 # [ 0.278731] ... generic counters: 6936worker2 # [ 0.293500] NET: Registered PF_NETLINK/PF_ROUTE protocol family937client # [ 0.308193] audit: initializing netlink subsys (disabled)938lb # [ 0.312397] cpuidle: using governor menu939worker1 # [ 0.279397] ... generic bitmap: 000000000000003f940worker1 # [ 0.279730] ... fixed-purpose counters: 0941client # [ 0.309301] thermal_sys: Registered thermal governor 'fair_share'942worker2 # [ 0.294900] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations943lb # [ 0.314565] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5944worker1 # [ 0.280413] ... fixed-purpose bitmap: 0000000000000000945client # [ 0.309303] thermal_sys: Registered thermal governor 'bang_bang'946worker2 # [ 0.295730] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations947worker1 # [ 0.280730] ... value mask: 0000ffffffffffff948client # [ 0.310183] thermal_sys: Registered thermal governor 'step_wise'949lb # [ 0.315634] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]950worker1 # [ 0.281569] ... max period: 00007fffffffffff951client # [ 0.311158] thermal_sys: Registered thermal governor 'user_space'952worker2 # [ 0.296993] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations953lb # [ 0.316372] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry954worker1 # [ 0.282360] ... global_ctrl mask: 000000000000003f955worker2 # [ 0.297715] audit: initializing netlink subsys (disabled)956lb # [ 0.317380] PCI: Using configuration type 1 for base access957client # [ 0.311922] audit: type=2000 audit(1790088619.745:1): state=initialized audit_enabled=0 res=1958worker1 # [ 0.282830] signal: max sigframe size: 3376959worker1 # [ 0.283619] rcu: Hierarchical SRCU implementation.960worker2 # [ 0.298818] thermal_sys: Registered thermal governor 'fair_share'961client # [ 0.313185] thermal_sys: Registered thermal governor 'power_allocator'962lb # [ 0.318450] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.963worker1 # [ 0.284298] rcu: Max phase no-delay instances is 400.964client # [ 0.313199] cpuidle: using governor menu965worker2 # [ 0.298821] thermal_sys: Registered thermal governor 'bang_bang'966worker2 # [ 0.299705] thermal_sys: Registered thermal governor 'step_wise'967client # [ 0.315400] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5968worker2 # [ 0.300626] thermal_sys: Registered thermal governor 'user_space'969client # [ 0.316412] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]970worker2 # [ 0.301493] audit: type=2000 audit(1790088619.747:1): state=initialized audit_enabled=0 res=1971client # [ 0.317185] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry972worker2 # [ 0.303708] thermal_sys: Registered thermal governor 'power_allocator'973worker2 # [ 0.303723] cpuidle: using governor menu974worker1 # [ 0.289299] smp: Bringing up secondary CPUs ...975client # [ 0.318194] PCI: Using configuration type 1 for base access976worker1 # [ 0.289745] smp: Brought up 1 node, 1 CPU977lb # [ 0.325382] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages978worker2 # [ 0.305908] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5979client # [ 0.319314] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.980worker1 # [ 0.290401] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)981lb # [ 0.326369] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page982worker2 # [ 0.306944] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]983worker1 # [ 0.290984] Memory: 1454572K/1572312K available (17247K kernel code, 2727K rwdata, 13616K rodata, 3652K init, 2976K bss, 109308K reserved, 0K cma-reserved)984worker2 # [ 0.307708] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry985worker1 # [ 0.291979] devtmpfs: initialized986worker2 # [ 0.308716] PCI: Using configuration type 1 for base access987worker1 # [ 0.292756] x86/mm: Memory block size: 128MB988worker2 # [ 0.309847] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.989client # [ 0.324414] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages990lb # [ 0.331370] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages991client # [ 0.325183] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page992worker1 # [ 0.294555] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)993lb # [ 0.332370] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page994worker1 # [ 0.295609] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).995worker1 # [ 0.296812] pinctrl core: initialized pinctrl subsystem996worker1 # [ 0.297876] PM: RTC time: 14:50:19, date: 2026-09-22997worker2 # [ 0.314935] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages998client # [ 0.330183] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages999worker2 # [ 0.315705] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page1000client # [ 0.331182] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page1001worker1 # [ 0.301331] NET: Registered PF_NETLINK/PF_ROUTE protocol family1002worker1 # [ 0.302225] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations1003worker1 # [ 0.302754] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations1004worker2 # [ 0.320707] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages1005worker1 # [ 0.304004] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations1006lb # [ 0.343614] ACPI: Added _OSI(Module Device)1007worker2 # [ 0.321706] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page1008worker1 # [ 0.304740] audit: initializing netlink subsys (disabled)1009lb # [ 0.344295] ACPI: Added _OSI(Processor Device)1010lb # [ 0.345216] ACPI: Added _OSI(Processor Aggregator Device)1011worker1 # [ 0.305918] thermal_sys: Registered thermal governor 'fair_share'1012worker1 # [ 0.305920] thermal_sys: Registered thermal governor 'bang_bang'1013worker1 # [ 0.306731] thermal_sys: Registered thermal governor 'step_wise'1014worker1 # [ 0.307666] thermal_sys: Registered thermal governor 'user_space'1015client # [ 0.341538] ACPI: Added _OSI(Module Device)1016client # [ 0.342184] ACPI: Added _OSI(Processor Device)1017worker1 # [ 0.308449] audit: type=2000 audit(1790088619.777:1): state=initialized audit_enabled=0 res=11018client # [ 0.342938] ACPI: Added _OSI(Processor Aggregator Device)1019worker1 # [ 0.309733] thermal_sys: Registered thermal governor 'power_allocator'1020worker1 # [ 0.309748] cpuidle: using governor menu1021worker1 # [ 0.311944] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.51022lb # [ 0.352926] ACPI: 1 ACPI AML tables successfully acquired and loaded1023worker2 # [ 0.332137] ACPI: Added _OSI(Module Device)1024worker1 # [ 0.312957] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]1025worker2 # [ 0.332707] ACPI: Added _OSI(Processor Device)1026worker1 # [ 0.313734] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry1027worker2 # [ 0.333489] ACPI: Added _OSI(Processor Aggregator Device)1028worker1 # [ 0.314741] PCI: Using configuration type 1 for base access1029lb # [ 0.356768] ACPI: Interpreter enabled1030worker1 # [ 0.315826] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.1031lb # [ 0.357385] ACPI: PM: (supports S0 S3 S4 S5)1032lb # [ 0.358100] ACPI: Using IOAPIC for interrupt routing1033client # [ 0.351727] ACPI: 1 ACPI AML tables successfully acquired and loaded1034lb # [ 0.358406] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1035lb # [ 0.359369] PCI: Using E820 reservations for host bridge windows1036worker1 # [ 0.320857] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages1037client # [ 0.355419] ACPI: Interpreter enabled1038worker1 # [ 0.321731] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page1039client # [ 0.356198] ACPI: PM: (supports S0 S3 S4 S5)1040lb # [ 0.362523] ACPI: Enabled 2 GPEs in block 00 to 3F1041client # [ 0.356903] ACPI: Using IOAPIC for interrupt routing1042worker2 # [ 0.343944] ACPI: 1 ACPI AML tables successfully acquired and loaded1043client # [ 0.357252] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1044client # [ 0.358183] PCI: Using E820 reservations for host bridge windows1045worker1 # [ 0.326732] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages1046worker1 # [ 0.327731] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page1047worker2 # [ 0.348126] ACPI: Interpreter enabled1048client # [ 0.361317] ACPI: Enabled 2 GPEs in block 00 to 3F1049worker2 # [ 0.348720] ACPI: PM: (supports S0 S3 S4 S5)1050worker2 # [ 0.349434] ACPI: Using IOAPIC for interrupt routing1051lb # [ 0.370410] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1052lb # [ 0.371377] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1053worker2 # [ 0.351769] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1054lb # [ 0.372457] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1055worker2 # [ 0.352705] PCI: Using E820 reservations for host bridge windows1056lb # [ 0.373493] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1057lb # [ 0.374720] PCI host bridge to bus 0000:001058lb # [ 0.375373] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1059worker2 # [ 0.355815] ACPI: Enabled 2 GPEs in block 00 to 3F1060client # [ 0.369962] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1061lb # [ 0.376369] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1062worker1 # [ 0.338078] ACPI: Added _OSI(Module Device)1063worker1 # [ 0.338732] ACPI: Added _OSI(Processor Device)1064client # [ 0.370935] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1065lb # [ 0.377369] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1066worker1 # [ 0.339455] ACPI: Added _OSI(Processor Aggregator Device)1067lb # [ 0.378370] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1068client # [ 0.372913] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1069lb # [ 0.379370] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1070client # [ 0.374209] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1071client # [ 0.375574] PCI host bridge to bus 0000:001072lb # [ 0.380369] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1073lb # [ 0.381370] pci_bus 0000:00: root bus resource [bus 00-ff]1074client # [ 0.376194] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1075worker2 # [ 0.363670] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1076client # [ 0.377182] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1077lb # [ 0.382397] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1078client # [ 0.378182] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1079worker2 # [ 0.364442] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1080lb # [ 0.383791] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1081client # [ 0.379182] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1082worker2 # [ 0.365781] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1083worker1 # [ 0.347766] ACPI: 1 ACPI AML tables successfully acquired and loaded1084client # [ 0.380182] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1085worker2 # [ 0.366827] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1086worker2 # [ 0.368080] PCI host bridge to bus 0000:001087client # [ 0.381182] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1088client # [ 0.382182] pci_bus 0000:00: root bus resource [bus 00-ff]1089worker2 # [ 0.368709] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1090worker2 # [ 0.369704] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1091client # [ 0.383162] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1092worker1 # [ 0.352154] ACPI: Interpreter enabled1093worker1 # [ 0.352746] ACPI: PM: (supports S0 S3 S4 S5)1094worker2 # [ 0.370709] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1095worker1 # [ 0.353451] ACPI: Using IOAPIC for interrupt routing1096client # [ 0.384626] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1097worker2 # [ 0.371733] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1098worker2 # [ 0.372704] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1099lb # [ 0.386471] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1100worker2 # [ 0.373704] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1101lb # [ 0.387505] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1102worker1 # [ 0.355785] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1103worker2 # [ 0.374705] pci_bus 0000:00: root bus resource [bus 00-ff]1104worker1 # [ 0.356731] PCI: Using E820 reservations for host bridge windows1105lb # [ 0.388430] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1106worker2 # [ 0.375707] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1107lb # [ 0.389689] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1108worker2 # [ 0.377146] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1109worker1 # [ 0.359840] ACPI: Enabled 2 GPEs in block 00 to 3F1110lb # [ 0.391113] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1111client # [ 0.387346] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1112client # [ 0.388195] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1113client # [ 0.389208] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1114lb # [ 0.394432] pci 0000:00:02.0: BAR 0 [io 0xc100-0xc11f]1115lb # [ 0.395303] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1116client # [ 0.390717] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1117lb # [ 0.396092] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1118client # [ 0.391901] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1119worker1 # [ 0.367548] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1120lb # [ 0.396376] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1121worker2 # [ 0.379785] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1122worker1 # [ 0.368484] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1123worker2 # [ 0.380718] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1124lb # [ 0.397940] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1125worker2 # [ 0.381654] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1126worker1 # [ 0.369809] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1127client # [ 0.395222] pci 0000:00:02.0: BAR 0 [io 0xc100-0xc11f]1128worker1 # [ 0.370853] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1129client # [ 0.396097] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1130worker2 # [ 0.382620] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1131worker1 # [ 0.372077] PCI host bridge to bus 0000:001132client # [ 0.396942] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1133lb # [ 0.400401] pci 0000:00:03.0: BAR 0 [io 0xc120-0xc13f]1134worker1 # [ 0.372735] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1135client # [ 0.397189] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1136lb # [ 0.401260] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1137worker2 # [ 0.384485] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1138worker1 # [ 0.373731] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1139lb # [ 0.402104] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1140worker1 # [ 0.374730] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1141client # [ 0.399152] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1142lb # [ 0.402951] pci 0000:00:04.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1143worker1 # [ 0.375730] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1144worker2 # [ 0.387724] pci 0000:00:02.0: BAR 0 [io 0xc100-0xc11f]1145worker2 # [ 0.388561] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1146worker1 # [ 0.376730] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1147client # [ 0.402223] pci 0000:00:03.0: BAR 0 [io 0xc120-0xc13f]1148worker2 # [ 0.389436] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1149worker1 # [ 0.377735] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1150client # [ 0.403111] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1151worker2 # [ 0.389710] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1152worker1 # [ 0.378731] pci_bus 0000:00: root bus resource [bus 00-ff]1153lb # [ 0.406258] pci 0000:00:04.0: BAR 0 [io 0xc000-0xc07f]1154client # [ 0.403956] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1155worker1 # [ 0.379666] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1156lb # [ 0.407005] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1157worker2 # [ 0.391231] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1158client # [ 0.405615] pci 0000:00:04.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1159lb # [ 0.407391] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1160worker1 # [ 0.381187] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1161lb # [ 0.408944] pci 0000:00:05.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1162worker2 # [ 0.394686] pci 0000:00:03.0: BAR 0 [io 0xc120-0xc13f]1163worker2 # [ 0.395372] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1164worker2 # [ 0.395727] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1165client # [ 0.409169] pci 0000:00:04.0: BAR 0 [io 0xc000-0xc07f]1166lb # [ 0.411405] pci 0000:00:05.0: BAR 0 [io 0xc140-0xc15f]1167client # [ 0.409873] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1168lb # [ 0.412272] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1169worker2 # [ 0.397251] pci 0000:00:04.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1170client # [ 0.410205] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1171lb # [ 0.413133] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1172client # [ 0.411736] pci 0000:00:05.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1173worker1 # [ 0.383799] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1174lb # [ 0.413376] pci 0000:00:05.0: ROM [mem 0xfeb80000-0xfebbffff pref]1175worker1 # [ 0.384744] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1176worker2 # [ 0.399927] pci 0000:00:04.0: BAR 0 [io 0xc000-0xc07f]1177worker1 # [ 0.385686] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1178lb # [ 0.414894] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1179client # [ 0.414225] pci 0000:00:05.0: BAR 0 [io 0xc140-0xc15f]1180worker2 # [ 0.401367] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1181worker1 # [ 0.386746] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1182worker2 # [ 0.401727] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1183client # [ 0.415132] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1184lb # [ 0.416407] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1185client # [ 0.416205] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1186lb # [ 0.417391] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1187worker1 # [ 0.388379] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1188worker2 # [ 0.403254] pci 0000:00:05.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1189client # [ 0.417188] pci 0000:00:05.0: ROM [mem 0xfeb80000-0xfebbffff pref]1190lb # [ 0.418924] pci 0000:00:07.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1191client # [ 0.418752] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1192worker2 # [ 0.405619] pci 0000:00:05.0: BAR 0 [io 0xc140-0xc15f]1193worker1 # [ 0.390750] pci 0000:00:02.0: BAR 0 [io 0xc100-0xc11f]1194worker2 # [ 0.406362] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1195worker1 # [ 0.391585] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1196worker2 # [ 0.406727] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1197worker1 # [ 0.392455] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1198client # [ 0.420235] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1199lb # [ 0.420776] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1200worker1 # [ 0.392737] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1201lb # [ 0.422108] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1202client # [ 0.421198] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1203worker2 # [ 0.407711] pci 0000:00:05.0: ROM [mem 0xfeb80000-0xfebbffff pref]1204lb # [ 0.422993] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1205client # [ 0.422854] pci 0000:00:07.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1206worker1 # [ 0.394219] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1207worker2 # [ 0.409298] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1208lb # [ 0.424382] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1209worker2 # [ 0.410730] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1210worker1 # [ 0.397649] pci 0000:00:03.0: BAR 0 [io 0xc120-0xc13f]1211client # [ 0.424884] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1212lb # [ 0.425392] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1213worker2 # [ 0.411664] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1214worker1 # [ 0.398388] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1215client # [ 0.425206] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1216worker1 # [ 0.398753] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1217lb # [ 0.426932] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1218worker2 # [ 0.413079] pci 0000:00:07.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1219client # [ 0.426764] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1220worker1 # [ 0.400304] pci 0000:00:04.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1221worker2 # [ 0.414729] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1222lb # [ 0.429370] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1223client # [ 0.428195] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1224lb # [ 0.430302] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1225client # [ 0.429188] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1226worker2 # [ 0.415715] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1227lb # [ 0.431728] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1228client # [ 0.430759] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1229worker2 # [ 0.417303] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1230worker1 # [ 0.403755] pci 0000:00:04.0: BAR 0 [io 0xc000-0xc07f]1231worker1 # [ 0.404628] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1232worker2 # [ 0.418727] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1233worker1 # [ 0.405503] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1234lb # [ 0.434235] pci 0000:00:0a.0: BAR 0 [io 0xc080-0xc0bf]1235client # [ 0.432825] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1236worker2 # [ 0.419690] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1237lb # [ 0.435020] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1238client # [ 0.433205] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1239worker1 # [ 0.407273] pci 0000:00:05.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1240lb # [ 0.435391] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1241worker2 # [ 0.421084] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1242client # [ 0.434726] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1243worker1 # [ 0.409679] pci 0000:00:05.0: BAR 0 [io 0xc140-0xc15f]1244lb # [ 0.436917] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1245worker2 # [ 0.422729] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1246worker1 # [ 0.410359] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1247client # [ 0.437234] pci 0000:00:0a.0: BAR 0 [io 0xc080-0xc0bf]1248worker1 # [ 0.410752] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1249client # [ 0.438099] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1250lb # [ 0.438377] pci 0000:00:0b.0: BAR 0 [io 0xc160-0xc17f]1251worker2 # [ 0.423831] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1252lb # [ 0.439194] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1253worker1 # [ 0.411736] pci 0000:00:05.0: ROM [mem 0xfeb80000-0xfebbffff pref]1254client # [ 0.438923] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1255lb # [ 0.440095] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1256worker2 # [ 0.425425] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1257worker1 # [ 0.413291] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1258client # [ 0.439816] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1259lb # [ 0.440955] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1260worker2 # [ 0.428442] pci 0000:00:0a.0: BAR 0 [io 0xc080-0xc0bf]1261worker1 # [ 0.414738] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1262lb # [ 0.442243] pci 0000:00:1d.0: BAR 4 [io 0xc180-0xc19f]1263worker2 # [ 0.428713] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1264worker1 # [ 0.415737] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1265client # [ 0.441988] pci 0000:00:0b.0: BAR 0 [io 0xc160-0xc17f]1266worker2 # [ 0.429649] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1267client # [ 0.442855] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1268worker1 # [ 0.417352] pci 0000:00:07.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1269lb # [ 0.443229] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1270client # [ 0.443205] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1271worker2 # [ 0.431091] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1272lb # [ 0.445176] pci 0000:00:1d.1: BAR 4 [io 0xc1a0-0xc1bf]1273worker1 # [ 0.418744] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1274client # [ 0.444747] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1275worker2 # [ 0.432721] pci 0000:00:0b.0: BAR 0 [io 0xc160-0xc17f]1276lb # [ 0.446181] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1277worker1 # [ 0.419692] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1278worker2 # [ 0.433551] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1279client # [ 0.445958] pci 0000:00:1d.0: BAR 4 [io 0xc180-0xc19f]1280lb # [ 0.448039] pci 0000:00:1d.2: BAR 4 [io 0xc1c0-0xc1df]1281worker2 # [ 0.434418] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1282worker1 # [ 0.421127] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1283client # [ 0.446396] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1284worker2 # [ 0.435301] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1285worker1 # [ 0.422611] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1286client # [ 0.447966] pci 0000:00:1d.1: BAR 4 [io 0xc1a0-0xc1bf]1287worker2 # [ 0.436485] pci 0000:00:1d.0: BAR 4 [io 0xc180-0xc19f]1288worker1 # [ 0.423573] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1289client # [ 0.449035] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1290worker2 # [ 0.436906] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1291worker1 # [ 0.425135] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1292client # [ 0.450937] pci 0000:00:1d.2: BAR 4 [io 0xc1c0-0xc1df]1293worker2 # [ 0.438722] pci 0000:00:1d.1: BAR 4 [io 0xc1a0-0xc1bf]1294worker1 # [ 0.426752] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1295worker2 # [ 0.439745] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1296worker1 # [ 0.427706] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1297worker2 # [ 0.441253] pci 0000:00:1d.2: BAR 4 [io 0xc1c0-0xc1df]1298worker1 # [ 0.429177] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1299lb # [ 0.448603] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1300worker1 # [ 0.430750] pci 0000:00:0a.0: BAR 0 [io 0xc080-0xc0bf]1301worker1 # [ 0.431660] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1302lb # [ 0.451142] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1303worker1 # [ 0.432553] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1304lb # [ 0.451655] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1305worker1 # [ 0.434178] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1306lb # [ 0.452652] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1307client # [ 0.451487] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1308lb # [ 0.453613] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1309worker1 # [ 0.436520] pci 0000:00:0b.0: BAR 0 [io 0xc160-0xc17f]1310worker1 # [ 0.436738] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1311client # [ 0.454150] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1312lb # [ 0.455421] pci 0000:00:1f.2: BAR 4 [io 0xc1e0-0xc1ff]1313worker2 # [ 0.441930] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1314worker1 # [ 0.437721] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1315lb # [ 0.456237] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1316client # [ 0.455166] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1317worker2 # [ 0.444217] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1318worker1 # [ 0.439122] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1319client # [ 0.456511] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1320lb # [ 0.457422] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1321worker2 # [ 0.445000] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1322worker1 # [ 0.440296] pci 0000:00:1d.0: BAR 4 [io 0xc180-0xc19f]1323client # [ 0.457433] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1324lb # [ 0.459084] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1325worker2 # [ 0.446001] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1326worker1 # [ 0.440935] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1327client # [ 0.459241] pci 0000:00:1f.2: BAR 4 [io 0xc1e0-0xc1ff]1328worker2 # [ 0.446962] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1329worker1 # [ 0.442331] pci 0000:00:1d.1: BAR 4 [io 0xc1a0-0xc1bf]1330client # [ 0.460137] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1331worker2 # [ 0.448745] pci 0000:00:1f.2: BAR 4 [io 0xc1e0-0xc1ff]1332worker1 # [ 0.442940] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1333lb # [ 0.462425] ACPI: PCI: Interrupt link LNKA configured for IRQ 101334worker2 # [ 0.449573] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1335client # [ 0.461271] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1336lb # [ 0.463467] ACPI: PCI: Interrupt link LNKB configured for IRQ 101337worker1 # [ 0.445246] pci 0000:00:1d.2: BAR 4 [io 0xc1c0-0xc1df]1338worker2 # [ 0.450711] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1339client # [ 0.463025] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1340lb # [ 0.464475] ACPI: PCI: Interrupt link LNKC configured for IRQ 111341lb # [ 0.465497] ACPI: PCI: Interrupt link LNKD configured for IRQ 111342worker2 # [ 0.452391] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1343lb # [ 0.466471] ACPI: PCI: Interrupt link LNKE configured for IRQ 101344lb # [ 0.467466] ACPI: PCI: Interrupt link LNKF configured for IRQ 101345client # [ 0.466941] ACPI: PCI: Interrupt link LNKA configured for IRQ 101346lb # [ 0.468469] ACPI: PCI: Interrupt link LNKG configured for IRQ 111347client # [ 0.467304] ACPI: PCI: Interrupt link LNKB configured for IRQ 101348worker2 # [ 0.456155] ACPI: PCI: Interrupt link LNKA configured for IRQ 101349lb # [ 0.469472] ACPI: PCI: Interrupt link LNKH configured for IRQ 111350client # [ 0.468281] ACPI: PCI: Interrupt link LNKC configured for IRQ 111351worker2 # [ 0.456814] ACPI: PCI: Interrupt link LNKB configured for IRQ 101352lb # [ 0.470408] ACPI: PCI: Interrupt link GSIA configured for IRQ 161353client # [ 0.469305] ACPI: PCI: Interrupt link LNKD configured for IRQ 111354worker2 # [ 0.457808] ACPI: PCI: Interrupt link LNKC configured for IRQ 111355lb # [ 0.471382] ACPI: PCI: Interrupt link GSIB configured for IRQ 171356client # [ 0.470285] ACPI: PCI: Interrupt link LNKE configured for IRQ 101357worker2 # [ 0.458801] ACPI: PCI: Interrupt link LNKD configured for IRQ 111358lb # [ 0.472382] ACPI: PCI: Interrupt link GSIC configured for IRQ 181359client # [ 0.471279] ACPI: PCI: Interrupt link LNKF configured for IRQ 101360lb # [ 0.473333] ACPI: PCI: Interrupt link GSID configured for IRQ 191361worker2 # [ 0.459804] ACPI: PCI: Interrupt link LNKE configured for IRQ 101362client # [ 0.472283] ACPI: PCI: Interrupt link LNKG configured for IRQ 111363lb # [ 0.474126] ACPI: PCI: Interrupt link GSIE configured for IRQ 201364worker2 # [ 0.460806] ACPI: PCI: Interrupt link LNKF configured for IRQ 101365lb # [ 0.475085] ACPI: PCI: Interrupt link GSIF configured for IRQ 211366client # [ 0.473279] ACPI: PCI: Interrupt link LNKH configured for IRQ 111367worker2 # [ 0.461816] ACPI: PCI: Interrupt link LNKG configured for IRQ 111368worker1 # [ 0.446016] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1369lb # [ 0.475382] ACPI: PCI: Interrupt link GSIG configured for IRQ 221370client # [ 0.474221] ACPI: PCI: Interrupt link GSIA configured for IRQ 161371worker2 # [ 0.462808] ACPI: PCI: Interrupt link LNKH configured for IRQ 111372worker1 # [ 0.448295] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1373lb # [ 0.476313] ACPI: PCI: Interrupt link GSIH configured for IRQ 231374client # [ 0.475162] ACPI: PCI: Interrupt link GSIB configured for IRQ 171375client # [ 0.475912] ACPI: PCI: Interrupt link GSIC configured for IRQ 181376worker2 # [ 0.463744] ACPI: PCI: Interrupt link GSIA configured for IRQ 161377worker1 # [ 0.449007] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1378lb # [ 0.478139] iommu: Default domain type: Translated1379client # [ 0.476194] ACPI: PCI: Interrupt link GSID configured for IRQ 191380worker2 # [ 0.464721] ACPI: PCI: Interrupt link GSIB configured for IRQ 171381lb # [ 0.478377] iommu: DMA domain TLB invalidation policy: lazy mode1382client # [ 0.477144] ACPI: PCI: Interrupt link GSIE configured for IRQ 201383worker2 # [ 0.465688] ACPI: PCI: Interrupt link GSIC configured for IRQ 181384worker1 # [ 0.450055] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1385lb # [ 0.479580] ACPI: bus type USB registered1386client # [ 0.477923] ACPI: PCI: Interrupt link GSIF configured for IRQ 211387worker2 # [ 0.466423] ACPI: PCI: Interrupt link GSID configured for IRQ 191388lb # [ 0.480387] usbcore: registered new interface driver usbfs1389worker1 # [ 0.451092] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1390worker2 # [ 0.466723] ACPI: PCI: Interrupt link GSIE configured for IRQ 201391client # [ 0.478195] ACPI: PCI: Interrupt link GSIG configured for IRQ 221392lb # [ 0.481266] usbcore: registered new interface driver hub1393worker2 # [ 0.467648] ACPI: PCI: Interrupt link GSIF configured for IRQ 211394client # [ 0.479195] ACPI: PCI: Interrupt link GSIH configured for IRQ 231395lb # [ 0.482031] usbcore: registered new device driver usb1396worker1 # [ 0.452685] pci 0000:00:1f.2: BAR 4 [io 0xc1e0-0xc1ff]1397worker2 # [ 0.468434] ACPI: PCI: Interrupt link GSIG configured for IRQ 221398worker1 # [ 0.453341] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1399worker2 # [ 0.468721] ACPI: PCI: Interrupt link GSIH configured for IRQ 231400client # [ 0.481213] iommu: Default domain type: Translated1401lb # [ 0.483283] NetLabel: Initializing1402lb # [ 0.483762] NetLabel: domain hash size = 1281403client # [ 0.482037] iommu: DMA domain TLB invalidation policy: lazy mode1404worker1 # [ 0.454084] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1405lb # [ 0.484369] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1406client # [ 0.483211] ACPI: bus type USB registered1407worker2 # [ 0.470913] iommu: Default domain type: Translated1408lb # [ 0.485410] NetLabel: unlabeled traffic allowed by default1409client # [ 0.483988] usbcore: registered new interface driver usbfs1410worker2 # [ 0.471718] iommu: DMA domain TLB invalidation policy: lazy mode1411worker1 # [ 0.455418] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1412lb # [ 0.486339] PCI: Using ACPI for IRQ routing1413client # [ 0.484877] usbcore: registered new interface driver hub1414worker2 # [ 0.472975] ACPI: bus type USB registered1415client # [ 0.485191] usbcore: registered new device driver usb1416worker2 # [ 0.473738] usbcore: registered new interface driver usbfs1417worker2 # [ 0.474653] usbcore: registered new interface driver hub1418client # [ 0.486885] NetLabel: Initializing1419worker2 # [ 0.475430] usbcore: registered new device driver usb1420client # [ 0.487188] NetLabel: domain hash size = 1281421worker1 # [ 0.459036] ACPI: PCI: Interrupt link LNKA configured for IRQ 101422client # [ 0.488105] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1423worker1 # [ 0.459837] ACPI: PCI: Interrupt link LNKB configured for IRQ 101424worker2 # [ 0.476800] NetLabel: Initializing1425client # [ 0.489038] NetLabel: unlabeled traffic allowed by default1426worker2 # [ 0.477417] NetLabel: domain hash size = 1281427worker1 # [ 0.460840] ACPI: PCI: Interrupt link LNKC configured for IRQ 111428client # [ 0.489930] PCI: Using ACPI for IRQ routing1429worker2 # [ 0.477705] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1430worker1 # [ 0.461877] ACPI: PCI: Interrupt link LNKD configured for IRQ 111431worker2 # [ 0.478666] NetLabel: unlabeled traffic allowed by default1432worker1 # [ 0.462832] ACPI: PCI: Interrupt link LNKE configured for IRQ 101433worker2 # [ 0.479454] PCI: Using ACPI for IRQ routing1434worker1 # [ 0.463833] ACPI: PCI: Interrupt link LNKF configured for IRQ 101435worker1 # [ 0.464833] ACPI: PCI: Interrupt link LNKG configured for IRQ 111436worker1 # [ 0.465828] ACPI: PCI: Interrupt link LNKH configured for IRQ 111437worker1 # [ 0.466770] ACPI: PCI: Interrupt link GSIA configured for IRQ 161438worker1 # [ 0.467747] ACPI: PCI: Interrupt link GSIB configured for IRQ 171439worker1 # [ 0.468747] ACPI: PCI: Interrupt link GSIC configured for IRQ 181440worker1 # [ 0.469747] ACPI: PCI: Interrupt link GSID configured for IRQ 191441worker1 # [ 0.470744] ACPI: PCI: Interrupt link GSIE configured for IRQ 201442worker1 # [ 0.471712] ACPI: PCI: Interrupt link GSIF configured for IRQ 211443worker1 # [ 0.472475] ACPI: PCI: Interrupt link GSIG configured for IRQ 221444worker1 # [ 0.472745] ACPI: PCI: Interrupt link GSIH configured for IRQ 231445worker1 # [ 0.475796] iommu: Default domain type: Translated1446worker1 # [ 0.476635] iommu: DMA domain TLB invalidation policy: lazy mode1447worker1 # [ 0.477799] ACPI: bus type USB registered1448worker1 # [ 0.478532] usbcore: registered new interface driver usbfs1449worker1 # [ 0.479415] usbcore: registered new interface driver hub1450worker1 # [ 0.479743] usbcore: registered new device driver usb1451worker1 # [ 0.481571] NetLabel: Initializing1452worker1 # [ 0.482255] NetLabel: domain hash size = 1281453worker1 # [ 0.482731] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1454worker1 # [ 0.483786] NetLabel: unlabeled traffic allowed by default1455worker1 # [ 0.484688] PCI: Using ACPI for IRQ routing1456lb # [ 0.530384] pci 0000:00:01.0: vgaarb: setting as boot VGA device1457lb # [ 0.531365] pci 0000:00:01.0: vgaarb: bridge control possible1458lb # [ 0.531365] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1459lb # [ 0.531373] vgaarb: loaded1460lb # [ 0.532160] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01461lb # [ 0.532371] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1462client # [ 0.534471] pci 0000:00:01.0: vgaarb: setting as boot VGA device1463worker2 # [ 0.522731] pci 0000:00:01.0: vgaarb: setting as boot VGA device1464client # [ 0.535178] pci 0000:00:01.0: vgaarb: bridge control possible1465worker2 # [ 0.523701] pci 0000:00:01.0: vgaarb: bridge control possible1466lb # [ 0.536485] clocksource: Switched to clocksource kvm-clock1467client # [ 0.535178] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1468worker2 # [ 0.523701] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1469client # [ 0.535188] vgaarb: loaded1470worker2 # [ 0.523710] vgaarb: loaded1471client # [ 0.535931] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01472worker2 # [ 0.524512] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01473lb # [ 0.539878] VFS: Disk quotas dquot_6.6.01474client # [ 0.536185] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1475worker2 # [ 0.525330] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1476lb # [ 0.540552] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1477lb # [ 0.541961] pnp: PnP ACPI init1478lb # [ 0.542785] ACPI: IRQ 4 override to edge(!), high(!)1479client # [ 0.539271] clocksource: Switched to clocksource kvm-clock1480lb # [ 0.543759] system 00:04: [mem 0xb0000000-0xbfffffff window] has been reserved1481lb # [ 0.545310] pnp: PnP ACPI: found 5 devices1482worker2 # [ 0.530813] clocksource: Switched to clocksource kvm-clock1483client # [ 0.542841] VFS: Disk quotas dquot_6.6.01484client # [ 0.543621] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1485client # [ 0.545053] pnp: PnP ACPI init1486worker2 # [ 0.534673] VFS: Disk quotas dquot_6.6.01487client # [ 0.545905] ACPI: IRQ 4 override to edge(!), high(!)1488worker2 # [ 0.535430] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1489worker2 # [ 0.536803] pnp: PnP ACPI init1490client # [ 0.546880] system 00:04: [mem 0xb0000000-0xbfffffff window] has been reserved1491client # [ 0.548447] pnp: PnP ACPI: found 5 devices1492worker2 # [ 0.537669] ACPI: IRQ 4 override to edge(!), high(!)1493lb # [ 0.552903] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1494worker2 # [ 0.538618] system 00:04: [mem 0xb0000000-0xbfffffff window] has been reserved1495lb # [ 0.554338] clocksource: Switched to clocksource acpi_pm1496lb # [ 0.555352] NET: Registered PF_INET protocol family1497worker2 # [ 0.540162] pnp: PnP ACPI: found 5 devices1498lb # [ 0.556544] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1499client # [ 0.556077] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1500client # [ 0.557592] clocksource: Switched to clocksource acpi_pm1501client # [ 0.558610] NET: Registered PF_INET protocol family1502worker2 # [ 0.547949] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1503client # [ 0.559869] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1504worker2 # [ 0.549815] clocksource: Switched to clocksource acpi_pm1505worker1 # [ 0.534911] pci 0000:00:01.0: vgaarb: setting as boot VGA device1506worker2 # [ 0.550768] NET: Registered PF_INET protocol family1507worker1 # [ 0.535727] pci 0000:00:01.0: vgaarb: bridge control possible1508worker2 # [ 0.551978] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1509worker1 # [ 0.535727] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1510worker1 # [ 0.535736] vgaarb: loaded1511worker1 # [ 0.536496] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01512worker1 # [ 0.536732] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1513lb # [ 0.573207] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1514lb # [ 0.574835] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1515lb # [ 0.576231] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1516worker1 # [ 0.542831] clocksource: Switched to clocksource kvm-clock1517lb # [ 0.577656] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1518worker1 # [ 0.544608] VFS: Disk quotas dquot_6.6.01519lb # [ 0.578934] TCP: Hash tables configured (established 16384 bind 16384)1520lb # [ 0.580089] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1521lb # [ 0.581409] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1522worker1 # [ 0.547196] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1523client # [ 0.577126] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1524lb # [ 0.582503] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1525worker1 # [ 0.548521] pnp: PnP ACPI init1526lb # [ 0.583680] NET: Registered PF_UNIX/PF_LOCAL protocol family1527client # [ 0.578669] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1528worker1 # [ 0.549332] ACPI: IRQ 4 override to edge(!), high(!)1529lb # [ 0.584597] NET: Registered PF_XDP protocol family1530worker2 # [ 0.569079] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1531client # [ 0.580018] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1532worker1 # [ 0.550251] system 00:04: [mem 0xb0000000-0xbfffffff window] has been reserved1533lb # [ 0.585411] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1534worker1 # [ 0.551739] pnp: PnP ACPI: found 5 devices1535client # [ 0.581331] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1536worker2 # [ 0.570601] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1537lb # [ 0.586497] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1538client # [ 0.582530] TCP: Hash tables configured (established 16384 bind 16384)1539worker2 # [ 0.571857] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1540lb # [ 0.587441] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1541lb # [ 0.588516] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1542client # [ 0.583659] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1543worker2 # [ 0.573156] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1544lb # [ 0.589554] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1545client # [ 0.584919] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1546worker2 # [ 0.574344] TCP: Hash tables configured (established 16384 bind 16384)1547lb # [ 0.590612] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1548client # [ 0.585991] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1549worker2 # [ 0.575460] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1550client # [ 0.587129] NET: Registered PF_UNIX/PF_LOCAL protocol family1551lb # [ 0.592307] ACPI: \_SB_.GSIA: Enabled at IRQ 161552worker2 # [ 0.576668] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1553client # [ 0.588071] NET: Registered PF_XDP protocol family1554worker2 # [ 0.577742] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1555client # [ 0.588888] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1556worker2 # [ 0.578954] NET: Registered PF_UNIX/PF_LOCAL protocol family1557worker1 # [ 0.559379] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1558lb # [ 0.594785] ACPI: \_SB_.GSIB: Enabled at IRQ 171559client # [ 0.589917] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1560worker2 # [ 0.579904] NET: Registered PF_XDP protocol family1561worker1 # [ 0.560806] clocksource: Switched to clocksource acpi_pm1562client # [ 0.590873] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1563worker2 # [ 0.580810] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1564worker1 # [ 0.561772] NET: Registered PF_INET protocol family1565lb # [ 0.596706] ACPI: \_SB_.GSIC: Enabled at IRQ 181566client # [ 0.591974] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1567worker2 # [ 0.581814] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1568client # [ 0.593018] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1569worker1 # [ 0.563023] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1570worker2 # [ 0.582842] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1571lb # [ 0.598604] ACPI: \_SB_.GSID: Enabled at IRQ 191572client # [ 0.594110] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1573worker2 # [ 0.583920] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1574lb # [ 0.600207] PCI: CLS 0 bytes, default 641575client # [ 0.595783] ACPI: \_SB_.GSIA: Enabled at IRQ 161576worker2 # [ 0.585049] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1577worker2 # [ 0.586121] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1578lb # [ 0.601217] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1579lb # [ 0.602994] Trying to unpack rootfs image as initramfs...1580client # [ 0.598580] ACPI: \_SB_.GSIB: Enabled at IRQ 171581worker2 # [ 0.587921] ACPI: \_SB_.GSIA: Enabled at IRQ 161582client # [ 0.600543] ACPI: \_SB_.GSIC: Enabled at IRQ 181583worker2 # [ 0.590376] ACPI: \_SB_.GSIB: Enabled at IRQ 171584client # [ 0.602560] ACPI: \_SB_.GSID: Enabled at IRQ 191585worker2 # [ 0.592364] ACPI: \_SB_.GSIC: Enabled at IRQ 181586client # [ 0.604371] PCI: CLS 0 bytes, default 641587worker2 # [ 0.594313] ACPI: \_SB_.GSID: Enabled at IRQ 191588client # [ 0.605273] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1589worker2 # [ 0.595998] PCI: CLS 0 bytes, default 641590client # [ 0.607063] Trying to unpack rootfs image as initramfs...1591worker2 # [ 0.596917] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1592worker2 # [ 0.598714] Trying to unpack rootfs image as initramfs...1593worker1 # [ 0.579977] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1594worker1 # [ 0.581386] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1595worker1 # [ 0.582671] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1596worker1 # [ 0.584022] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1597worker1 # [ 0.585213] TCP: Hash tables configured (established 16384 bind 16384)1598worker1 # [ 0.586353] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1599worker1 # [ 0.587657] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1600worker1 # [ 0.588743] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1601worker1 # [ 0.589964] NET: Registered PF_UNIX/PF_LOCAL protocol family1602worker1 # [ 0.590902] NET: Registered PF_XDP protocol family1603worker1 # [ 0.591732] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1604worker1 # [ 0.592726] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1605worker1 # [ 0.593721] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1606worker1 # [ 0.594807] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1607worker1 # [ 0.595907] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1608worker1 # [ 0.596995] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1609worker1 # [ 0.598684] ACPI: \_SB_.GSIA: Enabled at IRQ 161610worker1 # [ 0.601427] ACPI: \_SB_.GSIB: Enabled at IRQ 171611worker1 # [ 0.603442] ACPI: \_SB_.GSIC: Enabled at IRQ 181612worker1 # [ 0.605367] ACPI: \_SB_.GSID: Enabled at IRQ 191613worker1 # [ 0.607037] PCI: CLS 0 bytes, default 641614worker1 # [ 0.607971] Trying to unpack rootfs image as initramfs...1615lb # [ 0.647155] Initialise system trusted keyrings1616worker1 # [ 0.612024] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1617lb # [ 0.650140] workingset: timestamp_bits=40 max_order=19 bucket_order=01618client # [ 0.653165] Initialise system trusted keyrings1619worker2 # [ 0.643856] Initialise system trusted keyrings1620client # [ 0.657031] workingset: timestamp_bits=40 max_order=19 bucket_order=01621worker2 # [ 0.647469] workingset: timestamp_bits=40 max_order=19 bucket_order=01622lb # [ 0.672207] Key type asymmetric registered1623lb # [ 0.672942] Asymmetric key parser 'x509' registered1624lb # [ 0.673827] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1625lb # [ 0.679242] io scheduler mq-deadline registered1626lb # [ 0.680034] io scheduler kyber registered1627client # [ 0.676808] Key type asymmetric registered1628lb # [ 0.683635] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1629worker2 # [ 0.668836] Key type asymmetric registered1630lb # [ 0.684939] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1631client # [ 0.681851] Asymmetric key parser 'x509' registered1632worker2 # [ 0.671416] Asymmetric key parser 'x509' registered1633client # [ 0.682724] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1634worker2 # [ 0.672269] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1635worker1 # [ 0.655261] Initialise system trusted keyrings1636lb # [ 0.690376] Linux agpgart interface v0.1031637client # [ 0.685994] io scheduler mq-deadline registered1638worker2 # [ 0.675536] io scheduler mq-deadline registered1639worker1 # [ 0.656223] workingset: timestamp_bits=40 max_order=19 bucket_order=01640lb # [ 0.691170] ACPI: bus type drm_connector registered1641client # [ 0.686716] io scheduler kyber registered1642worker2 # [ 0.676283] io scheduler kyber registered1643lb # [ 0.694433] usbcore: registered new interface driver usbserial_generic1644client # [ 0.690394] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1645worker2 # [ 0.680034] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1646lb # [ 0.695542] usbserial: USB Serial support registered for generic1647client # [ 0.691670] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1648worker2 # [ 0.681328] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1649lb # [ 0.698105] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1650lb # [ 0.698105] Please enable it if your BIOS has the CPPC option.1651client # [ 0.695614] Linux agpgart interface v0.1031652worker2 # [ 0.685294] Linux agpgart interface v0.1031653lb # [ 0.700447] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1654client # [ 0.696420] ACPI: bus type drm_connector registered1655worker2 # [ 0.686052] ACPI: bus type drm_connector registered1656client # [ 0.699386] usbcore: registered new interface driver usbserial_generic1657lb # [ 0.704275] drop_monitor: Initializing network drop monitor service1658client # [ 0.700482] usbserial: USB Serial support registered for generic1659lb # [ 0.705462] NET: Registered PF_INET6 protocol family1660worker2 # [ 0.690979] usbcore: registered new interface driver usbserial_generic1661worker2 # [ 0.692102] usbserial: USB Serial support registered for generic1662lb # [ 0.708504] Segment Routing with IPv61663client # [ 0.704856] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1664client # [ 0.704856] Please enable it if your BIOS has the CPPC option.1665lb # [ 0.711118] In-situ OAM (IOAM) with IPv61666worker2 # [ 0.695419] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1667worker2 # [ 0.695419] Please enable it if your BIOS has the CPPC option.1668client # [ 0.707168] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1669lb # [ 0.713377] IPI shorthand broadcast: enabled1670worker2 # [ 0.697662] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1671worker1 # [ 0.679018] Key type asymmetric registered1672worker1 # [ 0.679721] Asymmetric key parser 'x509' registered1673client # [ 0.710048] drop_monitor: Initializing network drop monitor service1674worker1 # [ 0.680570] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1675client # [ 0.711248] NET: Registered PF_INET6 protocol family1676worker2 # [ 0.700572] drop_monitor: Initializing network drop monitor service1677worker2 # [ 0.701763] NET: Registered PF_INET6 protocol family1678client # [ 0.714294] Segment Routing with IPv61679worker2 # [ 0.704929] Segment Routing with IPv61680worker1 # [ 0.685980] io scheduler mq-deadline registered1681worker1 # [ 0.686725] io scheduler kyber registered1682client # [ 0.716863] In-situ OAM (IOAM) with IPv61683lb # [ 0.721710] sched_clock: Marking stable (587015764, 134064100)->(796640947, -75561083)1684client # [ 0.717968] IPI shorthand broadcast: enabled1685worker2 # [ 0.707441] In-situ OAM (IOAM) with IPv61686worker2 # [ 0.708506] IPI shorthand broadcast: enabled1687worker1 # [ 0.689405] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1688worker1 # [ 0.690659] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1689lb # [ 0.726290] registered taskstats version 11690lb # [ 0.727304] Loading compiled-in X.509 certificates1691worker1 # [ 0.696138] Linux agpgart interface v0.1031692client # [ 0.726261] sched_clock: Marking stable (592014505, 133815731)->(797704861, -71874625)1693worker1 # [ 0.696888] ACPI: bus type drm_connector registered1694worker2 # [ 0.718841] sched_clock: Marking stable (584014597, 134382728)->(787675231, -69277906)1695worker1 # [ 0.700183] usbcore: registered new interface driver usbserial_generic1696client # [ 0.731089] registered taskstats version 11697worker1 # [ 0.701216] usbserial: USB Serial support registered for generic1698client # [ 0.732062] Loading compiled-in X.509 certificates1699worker2 # [ 0.721501] registered taskstats version 11700worker2 # [ 0.722457] Loading compiled-in X.509 certificates1701worker1 # [ 0.704871] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1702worker1 # [ 0.704871] Please enable it if your BIOS has the CPPC option.1703worker1 # [ 0.707016] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1704worker1 # [ 0.710010] drop_monitor: Initializing network drop monitor service1705worker1 # [ 0.711162] NET: Registered PF_INET6 protocol family1706lb # [ 0.746094] Demotion targets for Node 0: null1707lb # [ 0.746999] Key type .fscrypt registered1708worker1 # [ 0.714353] Segment Routing with IPv61709lb # [ 0.749089] Key type fscrypt-provisioning registered1710lb # [ 0.750040] ima: No TPM chip found, activating TPM-bypass!1711lb # [ 0.751009] ima: Allocated hash algorithm: sha11712worker1 # [ 0.716873] In-situ OAM (IOAM) with IPv61713worker1 # [ 0.717961] IPI shorthand broadcast: enabled1714lb # [ 0.754118] ima: No architecture policies found1715client # [ 0.749847] Demotion targets for Node 0: null1716worker2 # [ 0.741412] Demotion targets for Node 0: null1717lb # [ 0.757096] PM: Magic number: 10:943:8391718client # [ 0.752877] Key type .fscrypt registered1719client # [ 0.753590] Key type fscrypt-provisioning registered1720lb # [ 0.758712] RAS: Correctable Errors collector initialized.1721client # [ 0.754598] ima: No TPM chip found, activating TPM-bypass!1722worker2 # [ 0.744457] Key type .fscrypt registered1723worker2 # [ 0.745112] Key type fscrypt-provisioning registered1724worker1 # [ 0.726194] sched_clock: Marking stable (592014560, 133829411)->(792205568, -66361597)1725client # [ 0.756849] ima: Allocated hash algorithm: sha11726worker2 # [ 0.746114] ima: No TPM chip found, activating TPM-bypass!1727client # [ 0.757650] ima: No architecture policies found1728worker2 # [ 0.748413] ima: Allocated hash algorithm: sha11729worker2 # [ 0.749175] ima: No architecture policies found1730worker1 # [ 0.730954] registered taskstats version 11731client # [ 0.761843] PM: Magic number: 10:943:8391732worker1 # [ 0.731907] Loading compiled-in X.509 certificates1733lb # [ 0.766782] clk: Disabling unused clocks1734worker2 # [ 0.751416] PM: Magic number: 10:943:8391735client # [ 0.763495] RAS: Correctable Errors collector initialized.1736worker2 # [ 0.753166] RAS: Correctable Errors collector initialized.1737lb # [ 0.771092] PM: genpd: Disabling unused power domains1738client # [ 0.772538] clk: Disabling unused clocks1739worker2 # [ 0.764189] clk: Disabling unused clocks1740client # [ 0.774854] PM: genpd: Disabling unused power domains1741worker2 # [ 0.766423] PM: genpd: Disabling unused power domains1742worker1 # [ 0.749857] Demotion targets for Node 0: null1743worker1 # [ 0.750721] Key type .fscrypt registered1744worker1 # [ 0.753849] Key type fscrypt-provisioning registered1745worker1 # [ 0.754762] ima: No TPM chip found, activating TPM-bypass!1746worker1 # [ 0.755684] ima: Allocated hash algorithm: sha11747worker1 # [ 0.757957] ima: No architecture policies found1748worker1 # [ 0.758949] PM: Magic number: 10:943:8391749worker1 # [ 0.760631] RAS: Correctable Errors collector initialized.1750worker1 # [ 0.769547] clk: Disabling unused clocks1751worker1 # [ 0.772874] PM: genpd: Disabling unused power domains1752lb # [ 0.927171] Freeing initrd memory: 29064K1753lb # [ 0.930380] Freeing unused decrypted memory: 2028K1754lb # [ 0.933096] Freeing unused kernel image (initmem) memory: 3652K1755lb # [ 0.934238] Write protecting the kernel read-only data: 32768k1756lb # [ 0.936158] Freeing unused kernel image (text/rodata gap) memory: 1184K1757lb # [ 0.937681] Freeing unused kernel image (rodata/data gap) memory: 720K1758client # [ 0.942174] Freeing initrd memory: 29056K1759client # [ 0.945404] Freeing unused decrypted memory: 2028K1760client # [ 0.948203] Freeing unused kernel image (initmem) memory: 3652K1761worker2 # [ 0.938055] Freeing initrd memory: 29060K1762client # [ 0.949352] Write protecting the kernel read-only data: 32768k1763client # [ 0.951259] Freeing unused kernel image (text/rodata gap) memory: 1184K1764worker2 # [ 0.941429] Freeing unused decrypted memory: 2028K1765client # [ 0.952797] Freeing unused kernel image (rodata/data gap) memory: 720K1766worker2 # [ 0.944338] Freeing unused kernel image (initmem) memory: 3652K1767worker2 # [ 0.945511] Write protecting the kernel read-only data: 32768k1768worker2 # [ 0.947480] Freeing unused kernel image (text/rodata gap) memory: 1184K1769worker2 # [ 0.949016] Freeing unused kernel image (rodata/data gap) memory: 720K1770worker1 # [ 0.938761] Freeing initrd memory: 29056K1771worker1 # [ 0.941998] Freeing unused decrypted memory: 2028K1772worker1 # [ 0.944591] Freeing unused kernel image (initmem) memory: 3652K1773worker1 # [ 0.945674] Write protecting the kernel read-only data: 32768k1774worker1 # [ 0.947509] Freeing unused kernel image (text/rodata gap) memory: 1184K1775worker1 # [ 0.949033] Freeing unused kernel image (rodata/data gap) memory: 720K1776lb # [ 0.988761] x86/mm: Checked W+X mappings: passed, no W+X pages found.1777lb # [ 0.989873] Run /init as init process1778lb # [ 1.000516] systemd[1]: Inserted module 'autofs4'1779client # [ 1.003905] x86/mm: Checked W+X mappings: passed, no W+X pages found.1780client # [ 1.005006] Run /init as init process1781worker2 # [ 1.000136] x86/mm: Checked W+X mappings: passed, no W+X pages found.1782worker2 # [ 1.001332] Run /init as init process1783lb # [ 1.020433] fuse: init (API version 7.45)1784client # [ 1.015880] systemd[1]: Inserted module 'autofs4'1785lb # [ 1.027996] ACPI: \_SB_.GSIG: Enabled at IRQ 221786worker2 # [ 1.012824] systemd[1]: Inserted module 'autofs4'1787lb # [ 1.030397] ACPI: \_SB_.GSIH: Enabled at IRQ 231788lb # [ 1.033499] ACPI: \_SB_.GSIE: Enabled at IRQ 201789lb # [ 1.035489] ACPI: \_SB_.GSIF: Enabled at IRQ 211790worker1 # [ 1.002114] x86/mm: Checked W+X mappings: passed, no W+X pages found.1791worker1 # [ 1.003169] Run /init as init process1792client # [ 1.035745] fuse: init (API version 7.45)1793lb # [ 1.041874] virtiofs virtio5: discovered new tag: nix-store1794lb # [ 1.043730] virtiofs virtio5: virtio_fs_setup_dax: No cache capability1795client # [ 1.043150] ACPI: \_SB_.GSIG: Enabled at IRQ 221796worker2 # [ 1.032892] fuse: init (API version 7.45)1797worker1 # [ 1.014622] systemd[1]: Inserted module 'autofs4'1798client # [ 1.045914] ACPI: \_SB_.GSIH: Enabled at IRQ 231799lb # [ 1.051245] virtiofs virtio6: discovered new tag: shared1800lb # [ 1.052794] virtiofs virtio6: virtio_fs_setup_dax: No cache capability1801client # [ 1.049516] ACPI: \_SB_.GSIE: Enabled at IRQ 201802worker2 # [ 1.040408] ACPI: \_SB_.GSIG: Enabled at IRQ 221803lb # [ 1.056270] virtiofs virtio7: discovered new tag: xchg1804client # [ 1.052082] ACPI: \_SB_.GSIF: Enabled at IRQ 211805lb # [ 1.057774] virtiofs virtio7: virtio_fs_setup_dax: No cache capability1806worker2 # [ 1.042763] ACPI: \_SB_.GSIH: Enabled at IRQ 231807worker2 # [ 1.045636] ACPI: \_SB_.GSIE: Enabled at IRQ 201808client # [ 1.058054] virtiofs virtio5: discovered new tag: nix-store1809worker2 # [ 1.047637] ACPI: \_SB_.GSIF: Enabled at IRQ 211810client # [ 1.059636] virtiofs virtio5: virtio_fs_setup_dax: No cache capability1811worker2 # [ 1.052646] virtiofs virtio5: discovered new tag: nix-store1812worker1 # [ 1.034525] fuse: init (API version 7.45)1813worker2 # [ 1.054603] virtiofs virtio5: virtio_fs_setup_dax: No cache capability1814client # [ 1.066977] virtiofs virtio6: discovered new tag: shared1815client # [ 1.068405] virtiofs virtio6: virtio_fs_setup_dax: No cache capability1816worker1 # [ 1.042005] ACPI: \_SB_.GSIG: Enabled at IRQ 221817client # [ 1.072235] virtiofs virtio7: discovered new tag: xchg1818client # [ 1.073803] virtiofs virtio7: virtio_fs_setup_dax: No cache capability1819worker2 # [ 1.063354] virtiofs virtio6: discovered new tag: shared1820lb # [ 1.079252] systemd[1]: Successfully made /usr/ read-only.1821worker1 # [ 1.044786] ACPI: \_SB_.GSIH: Enabled at IRQ 231822worker2 # [ 1.064945] virtiofs virtio6: virtio_fs_setup_dax: No cache capability1823worker1 # [ 1.048086] ACPI: \_SB_.GSIE: Enabled at IRQ 201824worker2 # [ 1.068454] virtiofs virtio7: discovered new tag: xchg1825worker1 # [ 1.050226] ACPI: \_SB_.GSIF: Enabled at IRQ 211826worker2 # [ 1.069901] virtiofs virtio7: virtio_fs_setup_dax: No cache capability1827worker1 # [ 1.055763] virtiofs virtio5: discovered new tag: nix-store1828worker1 # [ 1.057375] virtiofs virtio5: virtio_fs_setup_dax: No cache capability1829worker1 # [ 1.064952] virtiofs virtio6: discovered new tag: shared1830worker1 # [ 1.066553] virtiofs virtio6: virtio_fs_setup_dax: No cache capability1831client # [ 1.097642] systemd[1]: Successfully made /usr/ read-only.1832worker1 # [ 1.069859] virtiofs virtio7: discovered new tag: xchg1833worker1 # [ 1.071381] virtiofs virtio7: virtio_fs_setup_dax: No cache capability1834worker2 # [ 1.093703] systemd[1]: Successfully made /usr/ read-only.1835worker1 # [ 1.092647] systemd[1]: Successfully made /usr/ read-only.1836lb # [ 1.414965] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1837lb # [ 1.422274] systemd[1]: Detected virtualization kvm.1838lb # [ 1.424382] systemd[1]: Detected architecture x86-64.1839lb # [ 1.426496] systemd[1]: Running in initrd.1840lb # [ 1.428967] systemd[1]: Initializing machine ID from random generator.1841lb # [ 1.431894] systemd[1]: Hostname set to <lb>.1842client # [ 1.434202] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1843client # [ 1.446170] systemd[1]: Detected virtualization kvm.1844client # [ 1.448285] systemd[1]: Detected architecture x86-64.1845client # [ 1.450425] systemd[1]: Running in initrd.1846worker2 # [ 1.430374] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1847worker2 # [ 1.442308] systemd[1]: Detected virtualization kvm.1848client # [ 1.452953] systemd[1]: Initializing machine ID from random generator.1849worker2 # [ 1.444449] systemd[1]: Detected architecture x86-64.1850client # [ 1.455773] systemd[1]: Hostname set to <client>.1851worker2 # [ 1.446518] systemd[1]: Running in initrd.1852worker2 # [ 1.448943] systemd[1]: Initializing machine ID from random generator.1853worker1 # [ 1.428185] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1854worker1 # [ 1.433098] systemd[1]: Detected virtualization kvm.1855worker2 # [ 1.451847] systemd[1]: Hostname set to <worker2>.1856worker1 # [ 1.433925] systemd[1]: Detected architecture x86-64.1857worker1 # [ 1.434752] systemd[1]: Running in initrd.1858worker1 # [ 1.435801] systemd[1]: Initializing machine ID from random generator.1859worker1 # [ 1.437022] systemd[1]: Hostname set to <worker1>.1860lb # [ 1.660685] systemd[1]: bpf-restrict-fs: LSM BPF program attached1861client # [ 1.663943] systemd[1]: bpf-restrict-fs: LSM BPF program attached1862worker1 # [ 1.642205] systemd[1]: bpf-restrict-fs: LSM BPF program attached1863worker2 # [ 1.676115] systemd[1]: bpf-restrict-fs: LSM BPF program attached1864client # [ 1.703701] systemd[1]: Queued start job for default target Initrd Default Target.1865client # [ 1.708260] systemd[1]: Created slice Slice /system/modprobe.1866client # [ 1.709440] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1867client # [ 1.710881] systemd[1]: Expecting device /dev/disk/by-label/nixos...1868lb # [ 1.715769] systemd[1]: Queued start job for default target Initrd Default Target.1869client # [ 1.711978] systemd[1]: Reached target Path Units.1870client # [ 1.712842] systemd[1]: Reached target Slice Units.1871client # [ 1.713732] systemd[1]: Reached target Swaps.1872client # [ 1.714576] systemd[1]: Reached target Timer Units.1873client # [ 1.715562] systemd[1]: Listening on D-Bus System Message Bus Socket.1874lb # [ 1.720465] systemd[1]: Created slice Slice /system/modprobe.1875client # [ 1.716759] systemd[1]: Listening on Journal Socket (/dev/log).1876lb # [ 1.721626] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1877client # [ 1.717969] systemd[1]: Listening on Journal Sockets.1878lb # [ 1.723098] systemd[1]: Expecting device /dev/disk/by-label/nixos...1879client # [ 1.719004] systemd[1]: Listening on udev Control Socket.1880lb # [ 1.724202] systemd[1]: Reached target Path Units.1881client # [ 1.720118] systemd[1]: Listening on udev Kernel Socket.1882lb # [ 1.725139] systemd[1]: Reached target Slice Units.1883client # [ 1.721078] systemd[1]: Reached target Socket Units.1884lb # [ 1.726009] systemd[1]: Reached target Swaps.1885lb # [ 1.726855] systemd[1]: Reached target Timer Units.1886client # [ 1.722907] systemd[1]: Starting Create List of Static Device Nodes...1887lb # [ 1.727859] systemd[1]: Listening on D-Bus System Message Bus Socket.1888worker1 # [ 1.693873] systemd[1]: Queued start job for default target Initrd Default Target.1889lb # [ 1.729129] systemd[1]: Listening on Journal Socket (/dev/log).1890lb # [ 1.730279] systemd[1]: Listening on Journal Sockets.1891lb # [ 1.731307] systemd[1]: Listening on udev Control Socket.1892client # [ 1.726667] systemd[1]: Starting Load Kernel Module configfs...1893worker1 # [ 1.697141] systemd[1]: Created slice Slice /system/modprobe.1894lb # [ 1.732338] systemd[1]: Listening on udev Kernel Socket.1895lb # [ 1.733318] systemd[1]: Reached target Socket Units.1896worker1 # [ 1.698288] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1897worker1 # [ 1.699633] systemd[1]: Expecting device /dev/disk/by-label/nixos...1898worker1 # [ 1.700728] systemd[1]: Reached target Path Units.1899lb # [ 1.735213] systemd[1]: Starting Create List of Static Device Nodes...1900worker1 # [ 1.701595] systemd[1]: Reached target Slice Units.1901worker1 # [ 1.702446] systemd[1]: Reached target Swaps.1902worker1 # [ 1.703215] systemd[1]: Reached target Timer Units.1903worker1 # [ 1.704154] systemd[1]: Listening on D-Bus System Message Bus Socket.1904worker1 # [ 1.705300] systemd[1]: Listening on Journal Socket (/dev/log).1905worker1 # [ 1.706430] systemd[1]: Listening on Journal Sockets.1906lb # [ 1.741183] systemd[1]: Starting Load Kernel Module configfs...1907worker1 # [ 1.707423] systemd[1]: Listening on udev Control Socket.1908worker1 # [ 1.708414] systemd[1]: Listening on udev Kernel Socket.1909worker1 # [ 1.709387] systemd[1]: Reached target Socket Units.1910worker2 # [ 1.729255] systemd[1]: Queued start job for default target Initrd Default Target.1911worker1 # [ 1.711147] systemd[1]: Starting Create List of Static Device Nodes...1912worker2 # [ 1.733773] systemd[1]: Created slice Slice /system/modprobe.1913worker1 # [ 1.714693] systemd[1]: Starting Load Kernel Module configfs...1914worker2 # [ 1.734973] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1915worker2 # [ 1.736445] systemd[1]: Expecting device /dev/disk/by-label/nixos...1916client # [ 1.736908] systemd[1]: Starting Journal Service...1917worker2 # [ 1.737591] systemd[1]: Reached target Path Units.1918worker2 # [ 1.738594] systemd[1]: Reached target Slice Units.1919worker2 # [ 1.739531] systemd[1]: Reached target Swaps.1920worker2 # [ 1.740355] systemd[1]: Reached target Timer Units.1921worker2 # [ 1.741325] systemd[1]: Listening on D-Bus System Message Bus Socket.1922worker2 # [ 1.742590] systemd[1]: Listening on Journal Socket (/dev/log).1923worker2 # [ 1.743799] systemd[1]: Listening on Journal Sockets.1924lb # [ 1.748217] systemd[1]: Starting Journal Service...1925worker2 # [ 1.744838] systemd[1]: Listening on udev Control Socket.1926worker2 # [ 1.745901] systemd[1]: Listening on udev Kernel Socket.1927worker2 # [ 1.746917] systemd[1]: Reached target Socket Units.1928worker2 # [ 1.748783] systemd[1]: Starting Create List of Static Device Nodes...1929client # [ 1.762538] systemd[1]: Starting Load Kernel Modules...1930worker2 # [ 1.753470] systemd[1]: Starting Load Kernel Module configfs...1931worker1 # [ 1.726107] systemd[1]: Starting Journal Service...1932lb # [ 1.775262] systemd[1]: Starting Load Kernel Modules...1933client # [ 1.769965] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1934lb # [ 1.779185] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1935client # [ 1.777039] systemd[1]: Starting Coldplug All udev Devices...1936worker1 # [ 1.749065] systemd[1]: Starting Load Kernel Modules...1937worker1 # [ 1.752952] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1938lb # [ 1.788276] systemd[1]: Starting Coldplug All udev Devices...1939worker2 # [ 1.763506] systemd[1]: Starting Journal Service...1940client # [ 1.788183] systemd-journald[66]: Collecting audit messages is disabled.1941worker1 # [ 1.761029] systemd[1]: Starting Coldplug All udev Devices...1942client # [ 1.791912] systemd[1]: Finished Create List of Static Device Nodes.1943lb # [ 1.798677] systemd-journald[66]: Collecting audit messages is disabled.1944worker2 # [ 1.785639] systemd[1]: Starting Load Kernel Modules...1945lb # [ 1.801756] systemd[1]: Finished Create List of Static Device Nodes.1946client # [ 1.798028] systemd[1]: modprobe@configfs.service: Deactivated successfully.1947client # [ 1.803284] systemd[1]: Finished Load Kernel Module configfs.1948lb # [ 1.807765] systemd[1]: modprobe@configfs.service: Deactivated successfully.1949worker2 # [ 1.792523] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1950worker1 # [ 1.774951] systemd[1]: Finished Create List of Static Device Nodes.1951worker1 # [ 1.776345] systemd-journald[66]: Collecting audit messages is disabled.1952worker2 # [ 1.797577] systemd[1]: Starting Coldplug All udev Devices...1953client # [ 1.808765] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1954worker1 # [ 1.780120] systemd[1]: modprobe@configfs.service: Deactivated successfully.1955lb # [ 1.814860] systemd[1]: Finished Load Kernel Module configfs.1956lb # [ 1.819528] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1957worker1 # [ 1.786371] systemd[1]: Finished Load Kernel Module configfs.1958client # [ 1.820922] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1959worker2 # [ 1.810982] systemd-journald[66]: Collecting audit messages is disabled.1960worker1 # [ 1.792275] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1961worker2 # [ 1.812520] systemd[1]: Finished Create List of Static Device Nodes.1962worker2 # [ 1.816136] systemd[1]: modprobe@configfs.service: Deactivated successfully.1963lb # [ 1.833150] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1964client # [ 1.830984] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1965worker1 # [ 1.803914] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1966worker2 # [ 1.824804] systemd[1]: Finished Load Kernel Module configfs.1967client # [ 1.838914] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1968lb # [ 1.843202] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1969worker2 # [ 1.830850] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1970worker1 # [ 1.815022] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1971lb # [ 1.851262] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1972worker2 # [ 1.841484] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1973worker1 # [ 1.823039] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1974worker2 # [ 1.850875] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1975client # [ 1.862495] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1976lb # [ 1.874681] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1977worker2 # [ 1.861430] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1978worker1 # [ 1.844387] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1979client # [ 1.875971] systemd[1]: Starting Create Static Device Nodes in /dev...1980lb # [ 1.883291] systemd[1]: Starting Create Static Device Nodes in /dev...1981worker1 # [ 1.853053] systemd[1]: Starting Create Static Device Nodes in /dev...1982client # [ 1.883913] systemd[1]: Finished Load Kernel Modules.1983lb # [ 1.896155] systemd[1]: Finished Load Kernel Modules.1984worker1 # [ 1.864392] systemd[1]: Finished Load Kernel Modules.1985client # [ 1.894925] systemd[1]: Starting Apply Kernel Variables...1986worker2 # [ 1.883962] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1987lb # [ 1.906153] systemd[1]: Starting Apply Kernel Variables...1988worker2 # [ 1.892624] systemd[1]: Starting Create Static Device Nodes in /dev...1989worker1 # [ 1.874553] systemd[1]: Starting Apply Kernel Variables...1990client # [ 1.775436] systemd-modules-load[67]: Inserted module 'dm_mod'[ 1.910450] systemd[1]: Started Journal Service.1991client # 1992client # [ 1.780330] systemd-modules-load[67]: Inserted module 'virtio_balloon'1993client # [ 1.781389] systemd-modules-load[67]: Inserted module 'virtio_gpu'1994worker2 # [ 1.904931] systemd[1]: Finished Load Kernel Modules.1995lb # [ 1.925268] systemd[1]: Started Journal Service.1996client # [ 1.787032] systemd[1]: Finished Create Static Device Nodes in /dev.1997lb # [ 1.792227] systemd-modules-load[68]: Inserted module 'dm_mod'1998lb # [ 1.794308] systemd-modules-load[68]: Inserted module 'virtio_balloon'1999worker2 # [ 1.912900] systemd[1]: Starting Apply Kernel Variables...2000worker1 # [ 1.893914] systemd[1]: Finished Create Static Device Nodes in /dev.
loading 2,707 hidden lines…
4708worker1 # [ 38.148640] nix-daemon[869]: reaped child process 1003, status = succeeded4709lb # [ 38.189240] niks3-server[957]: 2026/09/22 14:50:57 INFO Received uploads request method=POST path=/api/pending_closures4710worker1 # [ 38.160936] nix-grpc-daemon[884]: time=2026-09-22T14:50:58.277Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4711worker1 # [ 38.162547] nix-grpc-daemon[884]: time=2026-09-22T14:50:58.279Z level=INFO msg="Uploading 5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top (232B)"4712lb # [ 38.226276] niks3-server[957]: 2026/09/22 14:50:57 INFO Registered completed upload object_key=nar/0pq8hp2qdfznpnf4pn4my9c4nd9yx6cfjn0x0bgv1rpk0536kpjy.nar.zst4713lb # [ 38.233380] niks3-server[957]: 2026/09/22 14:50:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/6/sign4714worker1 # [ 38.205530] nix-grpc-daemon[884]: time=2026-09-22T14:50:58.322Z level=INFO msg="Uploading 1 narinfos"4715lb # [ 38.240314] niks3-server[957]: 2026/09/22 14:50:57 INFO Signed narinfos id=6 count=14716lb # [ 38.243403] niks3-server[957]: 2026/09/22 14:50:57 INFO Registered completed upload object_key=log/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv4717lb # [ 38.249435] niks3-server[957]: 2026/09/22 14:50:57 INFO Registered completed upload object_key=5qgbfb7awggfq3smlyfxgz8zixnqxk2n.ls4718lb # [ 38.255412] niks3-server[957]: 2026/09/22 14:50:57 INFO Received complete upload request method=POST path=/api/pending_closures/6/complete4719lb # [ 38.262046] niks3-server[957]: 2026/09/22 14:50:57 INFO Completed upload id=64720worker1 # [ 38.228104] nix-grpc-daemon[884]: time=2026-09-22T14:50:58.345Z level=INFO msg="Upload complete. (215ms)"4721lb # [ 38.265668] niks3-server[957]: 2026/09/22 14:50:57 INFO Registered completed upload object_key=5qgbfb7awggfq3smlyfxgz8zixnqxk2n.narinfo4722worker1 # [ 38.235228] nix-grpc-daemon[787]: ts=2026-09-22T14:50:58Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43364 duration_s=1 drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv assign_id=3 outputs=14723worker1 # [ 38.248361] nix-daemon[869]: reaped child process 982, status = succeeded4724client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag t1 >&2, in 5.65 seconds)4725client: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag t1 outPath4726client: (finished: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag t1 outPath, in 0.12 seconds)4727client: must fail: test -e /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4728client: (finished: must fail: test -e /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.01 seconds)4729client: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top && grep farm-top-t1 /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4730client # copying 3 paths...4731client # copying path '/nix/store/qm96yd1cmricgf2hr21k88syh75l317v-farm-a' from 'http://lb:5751'...4732client # copying path '/nix/store/jicn175r0p4806ghnhxdvn7fsa1akkr0-farm-b' from 'http://lb:5751'...4733client # copying path '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top' from 'http://lb:5751'...4734client: (finished: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top && grep farm-top-t1 /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.17 seconds)4735(finished: subtest: DAG build is scheduled across workers and lands in the cache, in 6.00 seconds)4736subtest: repeat is answered Cached by the scheduler without building4737worker1: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4738worker1 # finding garbage collector roots...4739worker1 # deleting specified paths...4740worker1 # deleting '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top'4741worker1 # deleting unused links...4742worker1 # note: hard linking is currently saving 0.0 KiB4743worker1: (finished: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.15 seconds)4744worker2: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4745worker2 # finding garbage collector roots...4746worker2 # deleting specified paths...4747worker2 # deleting '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top'4748worker2 # deleting unused links...4749worker2 # note: hard linking is currently saving 0.0 KiB4750worker2: (finished: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.18 seconds)4751client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag t1 >&24752worker1 # [ 39.106105] nix-grpc-daemon[787]: ts=2026-09-22T14:50:59Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:433644753client # this path will be fetched (0.0 KiB download, 0.2 KiB unpacked):4754client # /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4755worker1 # [ 39.123247] nix-grpc-daemon[787]: ts=2026-09-22T14:50:59Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:433644756worker1 # [ 39.132577] nix-grpc-daemon[884]: time=2026-09-22T14:50:59.249Z level=INFO msg="All 2 paths already cached"4757worker1 # [ 39.147759] nix-grpc-daemon[787]: ts=2026-09-22T14:50:59Z level=debug event=want client=2 drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv cached=14758client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag t1 >&2, in 0.22 seconds)4759client: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag t1 outPath4760client: (finished: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag t1 outPath, in 0.10 seconds)4761lb: must succeed: curl -sf localhost:9901/clusters4762lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4763lb: must succeed: curl -sf localhost:9901/clusters4764lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)4765worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4766worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)4767(finished: subtest: repeat is answered Cached by the scheduler without building, in 0.83 seconds)4768subtest: inputs already in the cache are not sent to the scheduler4769lb: must succeed: curl -sf localhost:9901/clusters4770lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4771lb: must succeed: curl -sf localhost:9901/clusters4772lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)4773worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4774worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.04 seconds)4775client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag t1 --argstr top t1b >&24776client # don't know how to build these paths:4777client # /nix/store/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv4778worker1 # [ 39.673136] nix-grpc-daemon[787]: ts=2026-09-22T14:50:59Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:433644779worker1 # [ 39.678950] nix-grpc-daemon[787]: ts=2026-09-22T14:50:59Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:433644780worker1 # [ 39.685195] nix-grpc-daemon[884]: time=2026-09-22T14:50:59.800Z level=INFO msg="All 2 paths already cached"4781worker1 # [ 39.694138] nix-grpc-daemon[787]: ts=2026-09-22T14:50:59Z level=debug event=want client=3 drv=y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv cached=04782worker1 # [ 39.700297] nix-grpc-daemon[787]: ts=2026-09-22T14:50:59Z level=debug event=assigned drv=y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv worker=192.168.1.3:50051 clients=14783worker1 # [ 39.705389] nix-grpc-daemon[787]: ts=2026-09-22T14:50:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:433644784worker1 # [ 39.712947] nix-grpc-daemon[787]: ts=2026-09-22T14:50:59Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:433644785worker1 # [ 39.725725] nix-grpc-daemon[884]: time=2026-09-22T14:50:59.842Z level=INFO msg="All 2 paths already cached"4786client # copying 1 paths to 'lb:50051'...4787client # copying path '/nix/store/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv' to 'lb:50051'...4788worker1 # [ 39.734427] nix-grpc-daemon[787]: ts=2026-09-22T14:50:59Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:433644789worker1 # [ 39.745316] nix-daemon[869]: accepted connection from pid 787, user nix-grpc-daemon (trusted)4790worker1 # [ 39.842992] nix-daemon[869]: accepted connection from pid 1033, user nix-grpc-daemon (trusted)4791worker1 # [ 39.857406] nix-daemon[869]: reaped child process 1036, status = succeeded4792worker1 # [ 39.937930] nix-daemon[869]: accepted connection from pid 1038, user nix-grpc-daemon (trusted)4793worker1 # [ 39.951437] nix-daemon[869]: reaped child process 1041, status = succeeded4794lb # [ 39.991807] niks3-server[957]: 2026/09/22 14:50:59 INFO Received uploads request method=POST path=/api/pending_closures4795worker1 # [ 39.981969] nix-grpc-daemon[884]: time=2026-09-22T14:51:00.098Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4796worker1 # [ 39.983427] nix-grpc-daemon[884]: time=2026-09-22T14:51:00.098Z level=INFO msg="Uploading y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv (752B)"4797lb # [ 40.038637] niks3-server[957]: 2026/09/22 14:50:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/7/sign4798lb # [ 40.044232] niks3-server[957]: 2026/09/22 14:50:59 INFO Registered completed upload object_key=nar/0p14b74h5654i3y6qv7zaaib0ya421b01knaq0m00r5ypjyh2q1d.nar.zst4799lb # [ 40.057707] niks3-server[957]: 2026/09/22 14:50:59 INFO Registered completed upload object_key=y2wm8aizyyg449ia2pcscrjc2pklsrz4.ls4800lb # [ 40.068614] niks3-server[957]: 2026/09/22 14:50:59 INFO Signed narinfos id=7 count=14801worker1 # [ 40.037283] nix-grpc-daemon[884]: time=2026-09-22T14:51:00.154Z level=INFO msg="Uploading 1 narinfos"4802lb # [ 40.078578] niks3-server[957]: 2026/09/22 14:50:59 INFO Received complete upload request method=POST path=/api/pending_closures/7/complete4803lb # [ 40.090335] niks3-server[957]: 2026/09/22 14:50:59 INFO Registered completed upload object_key=y2wm8aizyyg449ia2pcscrjc2pklsrz4.narinfo4804lb # [ 40.095542] niks3-server[957]: 2026/09/22 14:50:59 INFO Completed upload id=74805worker1 # [ 40.061273] nix-grpc-daemon[884]: time=2026-09-22T14:51:00.178Z level=INFO msg="Upload complete. (305ms)"4806worker1 # [ 40.063387] nix-grpc-daemon[787]: ts=2026-09-22T14:51:00Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43364 duration_s=0 paths=1 nar_bytes_in=7524807worker1 # [ 40.066246] nix-grpc-daemon[787]: ts=2026-09-22T14:51:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:433644808worker1 # [ 40.074770] nix-daemon[869]: accepted connection from pid 787, user nix-grpc-daemon (trusted)4809worker1 # [ 40.077318] nix-daemon[869]: reaped child process 1031, status = succeeded4810client # building '/nix/store/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv' on lb:50051...4811worker1 # [ 40.081403] nix-daemon[869]: accepted connection from pid 787, user nix-grpc-daemon (trusted)4812client # farm-top> node-a: building y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv4813client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds4814client # farm-top> LOG-farm-top-t1b4815worker1 # [ 40.895767] nix-daemon[869]: reaped child process 1045, status = succeeded4816worker1 # [ 40.972715] nix-daemon[869]: accepted connection from pid 1061, user nix-grpc-daemon (trusted)4817worker1 # [ 40.986561] nix-daemon[869]: reaped child process 1064, status = succeeded4818lb # [ 41.027093] niks3-server[957]: 2026/09/22 14:51:00 INFO Received uploads request method=POST path=/api/pending_closures4819worker1 # [ 41.002374] nix-grpc-daemon[884]: time=2026-09-22T14:51:01.119Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4820worker1 # [ 41.005057] nix-grpc-daemon[884]: time=2026-09-22T14:51:01.121Z level=INFO msg="Uploading v688vnrkfcqiwaygym5ifmnr3yawh4xd-farm-top (232B)"4821lb # [ 41.068650] niks3-server[957]: 2026/09/22 14:51:00 INFO Registered completed upload object_key=nar/1j36mvvk44wddjhqvk302zl80d251m1g0d4kc4gfqy00xyklwzbf.nar.zst4822lb # [ 41.083330] niks3-server[957]: 2026/09/22 14:51:00 INFO Registered completed upload object_key=log/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv4823lb # [ 41.089861] niks3-server[957]: 2026/09/22 14:51:00 INFO Received sign narinfos request method=POST path=/api/pending_closures/8/sign4824lb # [ 41.095538] niks3-server[957]: 2026/09/22 14:51:00 INFO Signed narinfos id=8 count=14825worker1 # [ 41.061668] nix-grpc-daemon[884]: time=2026-09-22T14:51:01.178Z level=INFO msg="Uploading 1 narinfos"4826lb # [ 41.101927] niks3-server[957]: 2026/09/22 14:51:00 INFO Registered completed upload object_key=v688vnrkfcqiwaygym5ifmnr3yawh4xd.ls4827lb # [ 41.107342] niks3-server[957]: 2026/09/22 14:51:00 INFO Received complete upload request method=POST path=/api/pending_closures/8/complete4828lb # [ 41.111926] niks3-server[957]: 2026/09/22 14:51:00 INFO Completed upload id=84829worker1 # [ 41.077814] nix-grpc-daemon[884]: time=2026-09-22T14:51:01.194Z level=INFO msg="Upload complete. (190ms)"4830lb # [ 41.113680] niks3-server[957]: 2026/09/22 14:51:00 INFO Registered completed upload object_key=v688vnrkfcqiwaygym5ifmnr3yawh4xd.narinfo4831worker1 # [ 41.085198] nix-grpc-daemon[787]: ts=2026-09-22T14:51:01Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43364 duration_s=1 drv=y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv assign_id=4 outputs=14832worker1 # [ 41.098371] nix-daemon[869]: reaped child process 1043, status = succeeded4833client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag t1 --argstr top t1b >&2, in 1.60 seconds)4834client: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag t1 --argstr top t1b outPath4835client: (finished: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag t1 --argstr top t1b outPath, in 0.12 seconds)4836lb: must succeed: curl -sf localhost:9901/clusters4837lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4838lb: must succeed: curl -sf localhost:9901/clusters4839lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)4840worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4841worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.04 seconds)4842(finished: subtest: inputs already in the cache are not sent to the scheduler, in 1.95 seconds)4843subtest: outputs read back through any worker carry the cache signature4844client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag sig >&24845client # don't know how to build these paths:4846client # /nix/store/cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv4847worker2 # [ 41.564501] nix-grpc-daemon[783]: ts=2026-09-22T14:51:01Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:558364848worker2 # [ 41.570067] nix-grpc-daemon[783]: ts=2026-09-22T14:51:01Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:558364849worker1 # [ 41.572231] nix-grpc-daemon[787]: ts=2026-09-22T14:51:01Z level=debug event=want client=4 drv=lqi1rvcqjj8q9l279yzgrgaghp394zs5-farm-a.drv cached=04850worker1 # [ 41.576708] nix-grpc-daemon[787]: ts=2026-09-22T14:51:01Z level=debug event=assigned drv=lqi1rvcqjj8q9l279yzgrgaghp394zs5-farm-a.drv worker=192.168.1.4:50051 clients=14851worker2 # [ 41.597677] nix-grpc-daemon[783]: ts=2026-09-22T14:51:01Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558364852worker2 # [ 41.608761] nix-grpc-daemon[783]: ts=2026-09-22T14:51:01Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:558364853client # copying 1 paths to 'lb:50051'...4854client # copying path '/nix/store/lqi1rvcqjj8q9l279yzgrgaghp394zs5-farm-a.drv' to 'lb:50051'...4855worker2 # [ 41.619949] nix-grpc-daemon[783]: ts=2026-09-22T14:51:01Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:558364856worker2 # [ 41.631212] nix-daemon[850]: accepted connection from pid 783, user nix-grpc-daemon (trusted)4857worker2 # [ 41.653241] nix-grpc-daemon[783]: ts=2026-09-22T14:51:01Z level=info event=niks3_push_started pid=8784858worker2 # [ 41.804378] nix-grpc-daemon[878]: time=2026-09-22T14:51:01.709Z level=INFO msg="Configuring client TLS" cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker2.pem key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker2.key ca=""4859worker2 # [ 41.900135] nix-daemon[850]: accepted connection from pid 890, user nix-grpc-daemon (trusted)4860worker2 # [ 41.913073] nix-daemon[850]: reaped child process 894, status = succeeded4861worker2 # [ 41.995477] nix-daemon[850]: accepted connection from pid 896, user nix-grpc-daemon (trusted)4862worker2 # [ 42.009052] nix-daemon[850]: reaped child process 899, status = succeeded4863lb # [ 42.032165] niks3-server[957]: 2026/09/22 14:51:01 INFO Received uploads request method=POST path=/api/pending_closures4864worker2 # [ 42.025354] nix-grpc-daemon[878]: time=2026-09-22T14:51:01.931Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4865worker2 # [ 42.028620] nix-grpc-daemon[878]: time=2026-09-22T14:51:01.933Z level=INFO msg="Uploading lqi1rvcqjj8q9l279yzgrgaghp394zs5-farm-a.drv (504B)"4866lb # [ 42.066747] niks3-server[957]: 2026/09/22 14:51:01 INFO Registered completed upload object_key=nar/0bddczcnpndyzwmafx9yqfqrkrp1jg9rr5hsixz6rpy99kg84g1g.nar.zst4867lb # [ 42.074400] niks3-server[957]: 2026/09/22 14:51:01 INFO Received sign narinfos request method=POST path=/api/pending_closures/9/sign4868worker2 # [ 42.065605] nix-grpc-daemon[878]: time=2026-09-22T14:51:01.972Z level=INFO msg="Uploading 1 narinfos"4869lb # [ 42.081278] niks3-server[957]: 2026/09/22 14:51:01 INFO Signed narinfos id=9 count=14870lb # [ 42.095330] niks3-server[957]: 2026/09/22 14:51:01 INFO Registered completed upload object_key=lqi1rvcqjj8q9l279yzgrgaghp394zs5.ls4871lb # [ 42.103146] niks3-server[957]: 2026/09/22 14:51:01 INFO Received complete upload request method=POST path=/api/pending_closures/9/complete4872worker2 # [ 42.095113] nix-grpc-daemon[878]: time=2026-09-22T14:51:02.001Z level=INFO msg="Upload complete. (287ms)"4873lb # [ 42.110625] niks3-server[957]: 2026/09/22 14:51:01 INFO Completed upload id=94874worker2 # [ 42.097655] nix-grpc-daemon[783]: ts=2026-09-22T14:51:02Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55836 duration_s=0 paths=1 nar_bytes_in=5044875lb # [ 42.113198] niks3-server[957]: 2026/09/22 14:51:01 INFO Registered completed upload object_key=lqi1rvcqjj8q9l279yzgrgaghp394zs5.narinfo4876worker2 # [ 42.103442] nix-grpc-daemon[783]: ts=2026-09-22T14:51:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558364877worker2 # [ 42.112207] nix-daemon[850]: accepted connection from pid 783, user nix-grpc-daemon (trusted)4878worker2 # [ 42.114601] nix-daemon[850]: reaped child process 876, status = succeeded4879worker2 # [ 42.119785] nix-daemon[850]: accepted connection from pid 783, user nix-grpc-daemon (trusted)4880client # building '/nix/store/lqi1rvcqjj8q9l279yzgrgaghp394zs5-farm-a.drv' on lb:50051...4881client # farm-a> worker2: building lqi1rvcqjj8q9l279yzgrgaghp394zs5-farm-a.drv4882client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds4883client # farm-a> LOG-farm-a-sig4884worker2 # [ 42.963369] nix-daemon[850]: reaped child process 903, status = succeeded4885worker2 # [ 43.034441] nix-daemon[850]: accepted connection from pid 919, user nix-grpc-daemon (trusted)4886worker2 # [ 43.048468] nix-daemon[850]: reaped child process 922, status = succeeded4887lb # [ 43.069128] niks3-server[957]: 2026/09/22 14:51:02 INFO Received uploads request method=POST path=/api/pending_closures4888worker2 # [ 43.061443] nix-grpc-daemon[878]: time=2026-09-22T14:51:02.967Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4889worker2 # [ 43.063159] nix-grpc-daemon[878]: time=2026-09-22T14:51:02.969Z level=INFO msg="Uploading qi07z82gylcaha6pl0vfp8s3j6f2gl91-farm-a (128B)"4890lb # [ 43.103911] niks3-server[957]: 2026/09/22 14:51:02 INFO Registered completed upload object_key=log/lqi1rvcqjj8q9l279yzgrgaghp394zs5-farm-a.drv4891lb # [ 43.115908] niks3-server[957]: 2026/09/22 14:51:02 INFO Registered completed upload object_key=nar/05mvm4xykg9y2d7fm3g502ld2ajkd821jwng7v5g8vd73ynsbnah.nar.zst4892lb # [ 43.129063] niks3-server[957]: 2026/09/22 14:51:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/10/sign4893lb # [ 43.134732] niks3-server[957]: 2026/09/22 14:51:02 INFO Signed narinfos id=10 count=14894worker2 # [ 43.122394] nix-grpc-daemon[878]: time=2026-09-22T14:51:03.028Z level=INFO msg="Uploading 1 narinfos"4895lb # [ 43.140966] niks3-server[957]: 2026/09/22 14:51:02 INFO Registered completed upload object_key=qi07z82gylcaha6pl0vfp8s3j6f2gl91.ls4896lb # [ 43.149330] niks3-server[957]: 2026/09/22 14:51:02 INFO Received complete upload request method=POST path=/api/pending_closures/10/complete4897lb # [ 43.152364] niks3-server[957]: 2026/09/22 14:51:02 INFO Completed upload id=104898worker2 # [ 43.136925] nix-grpc-daemon[878]: time=2026-09-22T14:51:03.043Z level=INFO msg="Upload complete. (209ms)"4899lb # [ 43.155661] niks3-server[957]: 2026/09/22 14:51:02 INFO Registered completed upload object_key=qi07z82gylcaha6pl0vfp8s3j6f2gl91.narinfo4900worker2 # [ 43.143168] nix-grpc-daemon[783]: ts=2026-09-22T14:51:03Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55836 duration_s=1 drv=lqi1rvcqjj8q9l279yzgrgaghp394zs5-farm-a.drv assign_id=5 outputs=14901worker1 # [ 43.130358] nix-grpc-daemon[787]: ts=2026-09-22T14:51:03Z level=debug event=want client=4 drv=08wvq73c5i2kpl6g578glyb1rcqhv2iw-farm-b.drv cached=04902worker1 # [ 43.134676] nix-grpc-daemon[787]: ts=2026-09-22T14:51:03Z level=debug event=assigned drv=08wvq73c5i2kpl6g578glyb1rcqhv2iw-farm-b.drv worker=192.168.1.4:50051 clients=14903worker2 # [ 43.157534] nix-daemon[850]: reaped child process 901, status = succeeded4904worker2 # [ 43.191294] nix-grpc-daemon[783]: ts=2026-09-22T14:51:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558364905worker2 # [ 43.201113] nix-grpc-daemon[783]: ts=2026-09-22T14:51:03Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:558364906worker2 # [ 43.214378] nix-grpc-daemon[878]: time=2026-09-22T14:51:03.120Z level=INFO msg="All 1 paths already cached"4907client # copying 1 paths to 'lb:50051'...4908client # copying path '/nix/store/08wvq73c5i2kpl6g578glyb1rcqhv2iw-farm-b.drv' to 'lb:50051'...4909worker2 # [ 43.221785] nix-grpc-daemon[783]: ts=2026-09-22T14:51:03Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:558364910worker2 # [ 43.231160] nix-daemon[850]: accepted connection from pid 783, user nix-grpc-daemon (trusted)4911worker2 # [ 43.329487] nix-daemon[850]: accepted connection from pid 926, user nix-grpc-daemon (trusted)4912worker2 # [ 43.345242] nix-daemon[850]: reaped child process 929, status = succeeded4913worker2 # [ 43.428258] nix-daemon[850]: accepted connection from pid 931, user nix-grpc-daemon (trusted)4914worker2 # [ 43.441573] nix-daemon[850]: reaped child process 934, status = succeeded4915lb # [ 43.464398] niks3-server[957]: 2026/09/22 14:51:02 INFO Received uploads request method=POST path=/api/pending_closures4916worker2 # [ 43.457322] nix-grpc-daemon[878]: time=2026-09-22T14:51:03.363Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4917worker2 # [ 43.458796] nix-grpc-daemon[878]: time=2026-09-22T14:51:03.363Z level=INFO msg="Uploading 08wvq73c5i2kpl6g578glyb1rcqhv2iw-farm-b.drv (624B)"4918lb # [ 43.490203] niks3-server[957]: 2026/09/22 14:51:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/11/sign4919lb # [ 43.495111] niks3-server[957]: 2026/09/22 14:51:02 INFO Signed narinfos id=11 count=14920worker2 # [ 43.482664] nix-grpc-daemon[878]: time=2026-09-22T14:51:03.388Z level=INFO msg="Uploading 1 narinfos"4921lb # [ 43.504073] niks3-server[957]: 2026/09/22 14:51:02 INFO Registered completed upload object_key=nar/1m1mvw8bmb91glp26qj699lxqv98q2iz8wm4ik3miz0xj6vi3mw6.nar.zst4922lb # [ 43.517141] niks3-server[957]: 2026/09/22 14:51:02 INFO Received complete upload request method=POST path=/api/pending_closures/11/complete4923lb # [ 43.523890] niks3-server[957]: 2026/09/22 14:51:02 INFO Registered completed upload object_key=08wvq73c5i2kpl6g578glyb1rcqhv2iw.ls4924lb # [ 43.528890] niks3-server[957]: 2026/09/22 14:51:02 INFO Completed upload id=114925worker2 # [ 43.515580] nix-grpc-daemon[878]: time=2026-09-22T14:51:03.422Z level=INFO msg="Upload complete. (272ms)"4926lb # [ 43.532363] niks3-server[957]: 2026/09/22 14:51:02 INFO Registered completed upload object_key=08wvq73c5i2kpl6g578glyb1rcqhv2iw.narinfo4927worker2 # [ 43.518034] nix-grpc-daemon[783]: ts=2026-09-22T14:51:03Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55836 duration_s=0 paths=1 nar_bytes_in=6244928worker2 # [ 43.522519] nix-grpc-daemon[783]: ts=2026-09-22T14:51:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558364929worker2 # [ 43.528388] nix-daemon[850]: accepted connection from pid 783, user nix-grpc-daemon (trusted)4930worker2 # [ 43.530689] nix-daemon[850]: reaped child process 924, status = succeeded4931client # building '/nix/store/08wvq73c5i2kpl6g578glyb1rcqhv2iw-farm-b.drv' on lb:50051...4932client # farm-b> worker2: building 08wvq73c5i2kpl6g578glyb1rcqhv2iw-farm-b.drv4933worker2 # [ 43.535874] nix-daemon[850]: accepted connection from pid 783, user nix-grpc-daemon (trusted)4934client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds4935client # farm-b> LOG-farm-b-sig4936worker2 # [ 44.769078] nix-daemon[850]: reaped child process 938, status = succeeded4937worker2 # [ 44.838987] nix-daemon[850]: accepted connection from pid 954, user nix-grpc-daemon (trusted)4938worker2 # [ 44.852628] nix-daemon[850]: reaped child process 957, status = succeeded4939lb # [ 44.876382] niks3-server[957]: 2026/09/22 14:51:04 INFO Received uploads request method=POST path=/api/pending_closures4940worker2 # [ 44.868385] nix-grpc-daemon[878]: time=2026-09-22T14:51:04.774Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4941worker2 # [ 44.870292] nix-grpc-daemon[878]: time=2026-09-22T14:51:04.776Z level=INFO msg="Uploading 2f96mrg6ljwjqld1y3yaibmb07jkz3by-farm-b (176B)"4942lb # [ 44.910862] niks3-server[957]: 2026/09/22 14:51:04 INFO Registered completed upload object_key=log/08wvq73c5i2kpl6g578glyb1rcqhv2iw-farm-b.drv4943lb # [ 44.926712] niks3-server[957]: 2026/09/22 14:51:04 INFO Registered completed upload object_key=nar/0y3xlf15z3z540fgrc9sgnn2rl0aj3w6x9mjqd50hhqgmaxbiivd.nar.zst4944lb # [ 44.930559] niks3-server[957]: 2026/09/22 14:51:04 INFO Received sign narinfos request method=POST path=/api/pending_closures/12/sign4945lb # [ 44.935130] niks3-server[957]: 2026/09/22 14:51:04 INFO Signed narinfos id=12 count=14946worker2 # [ 44.923002] nix-grpc-daemon[878]: time=2026-09-22T14:51:04.829Z level=INFO msg="Uploading 1 narinfos"4947lb # [ 44.947623] niks3-server[957]: 2026/09/22 14:51:04 INFO Registered completed upload object_key=2f96mrg6ljwjqld1y3yaibmb07jkz3by.ls4948lb # [ 44.949862] niks3-server[957]: 2026/09/22 14:51:04 INFO Received complete upload request method=POST path=/api/pending_closures/12/complete4949lb # [ 44.955192] niks3-server[957]: 2026/09/22 14:51:04 INFO Completed upload id=124950lb # [ 44.957455] niks3-server[957]: 2026/09/22 14:51:04 INFO Registered completed upload object_key=2f96mrg6ljwjqld1y3yaibmb07jkz3by.narinfo4951worker2 # [ 44.941116] nix-grpc-daemon[878]: time=2026-09-22T14:51:04.847Z level=INFO msg="Upload complete. (197ms)"4952worker2 # [ 44.946645] nix-grpc-daemon[783]: ts=2026-09-22T14:51:04Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55836 duration_s=1 drv=08wvq73c5i2kpl6g578glyb1rcqhv2iw-farm-b.drv assign_id=6 outputs=14953worker1 # [ 44.934564] nix-grpc-daemon[787]: ts=2026-09-22T14:51:05Z level=debug event=want client=4 drv=cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv cached=04954worker1 # [ 44.939066] nix-grpc-daemon[787]: ts=2026-09-22T14:51:05Z level=debug event=assigned drv=cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv worker=192.168.1.4:50051 clients=14955worker2 # [ 44.961798] nix-daemon[850]: reaped child process 936, status = succeeded4956worker2 # [ 44.996243] nix-grpc-daemon[783]: ts=2026-09-22T14:51:04Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558364957worker2 # [ 45.008707] nix-grpc-daemon[783]: ts=2026-09-22T14:51:04Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:558364958worker2 # [ 45.021104] nix-grpc-daemon[878]: time=2026-09-22T14:51:04.927Z level=INFO msg="All 2 paths already cached"4959client # copying 1 paths to 'lb:50051'...4960client # copying path '/nix/store/cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv' to 'lb:50051'...4961worker2 # [ 45.031128] nix-grpc-daemon[783]: ts=2026-09-22T14:51:04Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:558364962worker2 # [ 45.046169] nix-daemon[850]: accepted connection from pid 783, user nix-grpc-daemon (trusted)4963worker2 # [ 45.145479] nix-daemon[850]: accepted connection from pid 961, user nix-grpc-daemon (trusted)4964worker2 # [ 45.159382] nix-daemon[850]: reaped child process 964, status = succeeded4965worker2 # [ 45.233929] nix-daemon[850]: accepted connection from pid 967, user nix-grpc-daemon (trusted)4966worker2 # [ 45.247508] nix-daemon[850]: reaped child process 970, status = succeeded4967lb # [ 45.267895] niks3-server[957]: 2026/09/22 14:51:04 INFO Received uploads request method=POST path=/api/pending_closures4968worker2 # [ 45.260294] nix-grpc-daemon[878]: time=2026-09-22T14:51:05.166Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4969worker2 # [ 45.261631] nix-grpc-daemon[878]: time=2026-09-22T14:51:05.166Z level=INFO msg="Uploading cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv (752B)"4970lb # [ 45.293153] niks3-server[957]: 2026/09/22 14:51:04 INFO Registered completed upload object_key=nar/0w21abkvlzg18nqkl2x4zxs7d0376kbld1n0fkr68vcpqpq1g5zg.nar.zst4971lb # [ 45.304970] niks3-server[957]: 2026/09/22 14:51:04 INFO Received sign narinfos request method=POST path=/api/pending_closures/13/sign4972worker2 # [ 45.296530] nix-grpc-daemon[878]: time=2026-09-22T14:51:05.202Z level=INFO msg="Uploading 1 narinfos"4973lb # [ 45.311818] niks3-server[957]: 2026/09/22 14:51:04 INFO Signed narinfos id=13 count=14974lb # [ 45.319290] niks3-server[957]: 2026/09/22 14:51:04 INFO Registered completed upload object_key=cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx.ls4975lb # [ 45.331223] niks3-server[957]: 2026/09/22 14:51:04 INFO Received complete upload request method=POST path=/api/pending_closures/13/complete4976worker2 # [ 45.321330] nix-grpc-daemon[878]: time=2026-09-22T14:51:05.227Z level=INFO msg="Upload complete. (262ms)"4977lb # [ 45.336629] niks3-server[957]: 2026/09/22 14:51:04 INFO Completed upload id=134978worker2 # [ 45.323478] nix-grpc-daemon[783]: ts=2026-09-22T14:51:05Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55836 duration_s=0 paths=1 nar_bytes_in=7524979lb # [ 45.343590] niks3-server[957]: 2026/09/22 14:51:04 INFO Registered completed upload object_key=cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx.narinfo4980worker2 # [ 45.327609] nix-grpc-daemon[783]: ts=2026-09-22T14:51:05Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558364981worker2 # [ 45.337467] nix-daemon[850]: accepted connection from pid 783, user nix-grpc-daemon (trusted)4982worker2 # [ 45.339765] nix-daemon[850]: reaped child process 959, status = succeeded4983client # building '/nix/store/cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv' on lb:50051...4984worker2 # [ 45.344955] nix-daemon[850]: accepted connection from pid 783, user nix-grpc-daemon (trusted)4985client # farm-top> worker2: building cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv4986lb # [ 45.797427] envoy[905]: conn peer=[2001:db8:1::1]:35922 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9507 tx=4367 ms=54994987lb # [ 45.801365] envoy[905]: conn peer=[2001:db8:1::1]:35936 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=991 tx=476 ms=554988lb # [ 45.804850] envoy[905]: conn peer=[2001:db8:1::1]:35938 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4169 tx=1918 ms=14654989client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds4990client # farm-top> LOG-farm-top-sig4991worker2 # [ 46.185753] nix-daemon[850]: reaped child process 974, status = succeeded4992worker2 # [ 46.267895] nix-daemon[850]: accepted connection from pid 990, user nix-grpc-daemon (trusted)4993worker2 # [ 46.280627] nix-daemon[850]: reaped child process 993, status = succeeded4994lb # [ 46.303388] niks3-server[957]: 2026/09/22 14:51:05 INFO Received uploads request method=POST path=/api/pending_closures4995worker2 # [ 46.295639] nix-grpc-daemon[878]: time=2026-09-22T14:51:06.201Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4996worker2 # [ 46.298061] nix-grpc-daemon[878]: time=2026-09-22T14:51:06.203Z level=INFO msg="Uploading dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top (232B)"4997lb # [ 46.341192] niks3-server[957]: 2026/09/22 14:51:05 INFO Registered completed upload object_key=log/cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv4998lb # [ 46.349115] niks3-server[957]: 2026/09/22 14:51:05 INFO Received sign narinfos request method=POST path=/api/pending_closures/14/sign4999lb # [ 46.354590] niks3-server[957]: 2026/09/22 14:51:05 INFO Signed narinfos id=14 count=15000worker2 # [ 46.342309] nix-grpc-daemon[878]: time=2026-09-22T14:51:06.248Z level=INFO msg="Uploading 1 narinfos"5001lb # [ 46.363073] niks3-server[957]: 2026/09/22 14:51:05 INFO Registered completed upload object_key=nar/0h4zvhqjxsvdqli2l6i5ajh5sh40hz57gzchdirnvbisq2aks0k8.nar.zst5002lb # [ 46.374959] niks3-server[957]: 2026/09/22 14:51:05 INFO Received complete upload request method=POST path=/api/pending_closures/14/complete5003lb # [ 46.381218] niks3-server[957]: 2026/09/22 14:51:05 INFO Registered completed upload object_key=dp6wdd8xfalk9qg1xqalcnk58hz7fnr5.ls5004lb # [ 46.469802] niks3-server[957]: 2026/09/22 14:51:05 INFO Completed upload id=145005worker2 # [ 46.457446] nix-grpc-daemon[878]: time=2026-09-22T14:51:06.363Z level=INFO msg="Upload complete. (305ms)"5006lb # [ 46.473246] niks3-server[957]: 2026/09/22 14:51:05 INFO Registered completed upload object_key=dp6wdd8xfalk9qg1xqalcnk58hz7fnr5.narinfo5007worker2 # [ 46.464511] nix-grpc-daemon[783]: ts=2026-09-22T14:51:06Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55836 duration_s=1 drv=cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv assign_id=7 outputs=15008worker2 # [ 46.479895] nix-daemon[850]: reaped child process 972, status = succeeded5009client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag sig >&2, in 5.08 seconds)5010client: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag sig outPath5011client: (finished: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag sig outPath, in 0.12 seconds)5012client: must succeed: nix path-info --sigs --store 'grpc://worker1:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&ca-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ca.pem' /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top5013worker1 # [ 46.708902] nix-grpc-daemon[787]: ts=2026-09-22T14:51:06Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:428105014client # these 3 paths will be fetched (0.0 KiB download, 0.5 KiB unpacked):5015client # /nix/store/qi07z82gylcaha6pl0vfp8s3j6f2gl91-farm-a5016client # /nix/store/2f96mrg6ljwjqld1y3yaibmb07jkz3by-farm-b5017client # /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top5018worker1 # [ 46.752753] nix-grpc-daemon[787]: ts=2026-09-22T14:51:06Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:428105019worker1 # [ 46.756904] nix-grpc-daemon[787]: copying path '/nix/store/qi07z82gylcaha6pl0vfp8s3j6f2gl91-farm-a' from 'http://lb:5751'...5020worker1 # [ 46.764276] nix-grpc-daemon[787]: copying path '/nix/store/2f96mrg6ljwjqld1y3yaibmb07jkz3by-farm-b' from 'http://lb:5751'...5021worker1 # [ 46.769430] nix-grpc-daemon[787]: copying path '/nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top' from 'http://lb:5751'...5022client: (finished: must succeed: nix path-info --sigs --store 'grpc://worker1:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&ca-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ca.pem' /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top, in 0.19 seconds)5023client: must succeed: nix path-info --sigs --store 'grpc://worker2:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&ca-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ca.pem' /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top5024worker2 # [ 46.917893] nix-grpc-daemon[783]: ts=2026-09-22T14:51:06Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:433125025worker2 # [ 46.924489] nix-grpc-daemon[783]: ts=2026-09-22T14:51:06Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:433125026client: (finished: must succeed: nix path-info --sigs --store 'grpc://worker2:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&ca-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ca.pem' /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top, in 0.16 seconds)5027client: must succeed: nix-store --delete /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top && nix copy --from 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --option trusted-public-keys 'farm-test-1:RkClDwvfixdOwourBI4UD9hudE3xfU5EBQcMFUVuRV8=' /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top5028client # finding garbage collector roots...5029client # removing stale temporary roots file "/nix/var/nix/temproots/896"5030client # removing stale temporary roots file "/nix/var/nix/temproots/851"5031client # removing stale temporary roots file "/nix/var/nix/temproots/859"5032client # removing stale temporary roots file "/nix/var/nix/temproots/873"5033client # removing stale temporary roots file "/nix/var/nix/temproots/801"5034client # removing stale temporary roots file "/nix/var/nix/temproots/818"5035client # removing stale temporary roots file "/nix/var/nix/temproots/839"5036client # removing stale temporary roots file "/nix/var/nix/temproots/881"5037client # deleting specified paths...5038client # deleting '/nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top'5039client # deleting unused links...5040client # note: hard linking is currently saving 0.0 KiB5041worker1 # [ 47.185685] nix-grpc-daemon[787]: ts=2026-09-22T14:51:07Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:433645042worker1 # [ 47.192406] nix-grpc-daemon[787]: ts=2026-09-22T14:51:07Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:433645043worker1 # [ 47.201303] nix-grpc-daemon[884]: time=2026-09-22T14:51:07.317Z level=INFO msg="All 1 paths already cached"5044worker2 # [ 47.227927] nix-grpc-daemon[783]: ts=2026-09-22T14:51:07Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:558365045worker2 # [ 47.234301] nix-grpc-daemon[783]: ts=2026-09-22T14:51:07Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:558365046client # copying 3 paths...5047worker1 # [ 47.225643] nix-grpc-daemon[787]: ts=2026-09-22T14:51:07Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:433645048client # copying path '/nix/store/qi07z82gylcaha6pl0vfp8s3j6f2gl91-farm-a' from 'grpc://lb:50051'...5049worker1 # [ 47.238337] nix-grpc-daemon[787]: ts=2026-09-22T14:51:07Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:433645050worker1 # [ 47.240784] nix-grpc-daemon[787]: ts=2026-09-22T14:51:07Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:43364 duration_s=0 paths=1 nar_bytes_out=2325051worker2 # [ 47.262764] nix-grpc-daemon[783]: ts=2026-09-22T14:51:07Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:558365052worker2 # [ 47.269400] nix-grpc-daemon[783]: ts=2026-09-22T14:51:07Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:558365053worker2 # [ 47.274859] nix-grpc-daemon[783]: ts=2026-09-22T14:51:07Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:55836 duration_s=0 paths=1 nar_bytes_out=1765054worker2 # [ 47.279377] nix-grpc-daemon[783]: ts=2026-09-22T14:51:07Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:55836 duration_s=0 paths=1 nar_bytes_out=1285055client # copying path '/nix/store/2f96mrg6ljwjqld1y3yaibmb07jkz3by-farm-b' from 'grpc://lb:50051'...5056client # copying path '/nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top' from 'grpc://lb:50051'...5057client: (finished: must succeed: nix-store --delete /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top && nix copy --from 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --option trusted-public-keys 'farm-test-1:RkClDwvfixdOwourBI4UD9hudE3xfU5EBQcMFUVuRV8=' /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top, in 0.38 seconds)5058(finished: subtest: outputs read back through any worker carry the cache signature, in 5.93 seconds)5059subtest: two clients wanting the same drv build it once5060client: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag dedup5061client # [ 47.388977] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key" --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag dedup.5062client # Running as unit: dedup1.service; invocation ID: a37de5db731b41409eb1bf81f691d4215063client: (finished: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag dedup, in 0.03 seconds)5064client: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag dedup5065client # [ 47.527413] nix[949]: don't know how to build these paths:5066client # [ 47.528301] nix[949]: /nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv5067worker1 # [ 47.495911] nix-grpc-daemon[787]: ts=2026-09-22T14:51:07Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:433645068worker1 # [ 47.506755] nix-grpc-daemon[787]: ts=2026-09-22T14:51:07Z level=debug event=want client=5 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv cached=05069worker1 # [ 47.511525] nix-grpc-daemon[787]: ts=2026-09-22T14:51:07Z level=debug event=assigned drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv worker=192.168.1.3:50051 clients=15070worker1 # [ 47.551327] nix-grpc-daemon[787]: ts=2026-09-22T14:51:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:433645071worker1 # [ 47.564436] nix-grpc-daemon[787]: ts=2026-09-22T14:51:07Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:433645072client # [ 47.605219] nix[949]: copying 1 paths to 'lb:50051'...5073worker1 # [ 47.575955] nix-grpc-daemon[787]: ts=2026-09-22T14:51:07Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:433645074client # [ 47.607103] nix[949]: copying path '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' to 'lb:50051'...5075worker1 # [ 47.588466] nix-daemon[869]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5076worker1 # [ 47.696445] nix-daemon[869]: accepted connection from pid 1085, user nix-grpc-daemon (trusted)5077worker1 # [ 47.711836] nix-daemon[869]: reaped child process 1088, status = succeeded5078worker1 # [ 47.793533] nix-daemon[869]: accepted connection from pid 1090, user nix-grpc-daemon (trusted)5079worker1 # [ 47.810373] nix-daemon[869]: reaped child process 1093, status = succeeded5080lb # [ 47.851312] niks3-server[957]: 2026/09/22 14:51:07 INFO Received uploads request method=POST path=/api/pending_closures5081worker1 # [ 47.824850] nix-grpc-daemon[884]: time=2026-09-22T14:51:07.942Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5082worker1 # [ 47.826415] nix-grpc-daemon[884]: time=2026-09-22T14:51:07.943Z level=INFO msg="Uploading k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv (432B)"5083lb # [ 47.878170] niks3-server[957]: 2026/09/22 14:51:07 INFO Registered completed upload object_key=nar/1rprin71wk1rn18ghpyxh5nm1akxfkjivwxfnlzqzmri5xdvl2l0.nar.zst5084lb # [ 47.893480] niks3-server[957]: 2026/09/22 14:51:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/15/sign5085worker1 # [ 47.865501] nix-grpc-daemon[884]: time=2026-09-22T14:51:07.982Z level=INFO msg="Uploading 1 narinfos"5086lb # [ 47.899789] niks3-server[957]: 2026/09/22 14:51:07 INFO Signed narinfos id=15 count=15087lb # [ 47.906347] niks3-server[957]: 2026/09/22 14:51:07 INFO Registered completed upload object_key=k75hvg0drf5n5cgm1w4hms82xb1l5nmz.ls5088lb # [ 47.921105] niks3-server[957]: 2026/09/22 14:51:07 INFO Received complete upload request method=POST path=/api/pending_closures/15/complete5089lb # [ 47.928304] niks3-server[957]: 2026/09/22 14:51:07 INFO Completed upload id=155090worker1 # [ 47.894223] nix-grpc-daemon[884]: time=2026-09-22T14:51:08.011Z level=INFO msg="Upload complete. (294ms)"5091lb # [ 47.930820] niks3-server[957]: 2026/09/22 14:51:07 INFO Registered completed upload object_key=k75hvg0drf5n5cgm1w4hms82xb1l5nmz.narinfo5092worker1 # [ 47.898761] nix-grpc-daemon[787]: ts=2026-09-22T14:51:08Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43364 duration_s=0 paths=1 nar_bytes_in=4325093worker1 # [ 47.905842] nix-grpc-daemon[787]: ts=2026-09-22T14:51:08Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:433645094worker1 # [ 47.915486] nix-daemon[869]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5095worker1 # [ 47.925339] nix-daemon[869]: reaped child process 1083, status = succeeded5096worker1 # [ 47.929684] nix-daemon[869]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5097client # [ 47.959591] nix[949]: building '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' on lb:50051...5098client # [ 48.444618] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key" --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag dedup.5099client # Running as unit: dedup2.service; invocation ID: 6e12fe9e4c0f4125a46e8d3e7953323d5100client: (finished: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag dedup, in 1.07 seconds)5101worker1 # [ 48.578833] nix-grpc-daemon[787]: ts=2026-09-22T14:51:08Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:433645102client # [ 48.615351] nix[965]: this derivation will be built:5103worker2 # [ 48.607227] nix-grpc-daemon[783]: ts=2026-09-22T14:51:08Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:558365104worker2 # [ 48.612913] nix-grpc-daemon[783]: copying path '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' from 'http://lb:5751'...5105client # [ 48.637602] nix[965]: /nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv5106worker1 # [ 48.611840] nix-grpc-daemon[787]: ts=2026-09-22T14:51:08Z level=debug event=want client=6 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv cached=05107worker1 # [ 48.619099] nix-grpc-daemon[787]: ts=2026-09-22T14:51:08Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:433645108client # [ 48.651766] nix[965]: building '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' on lb:50051...5109client: must succeed: sleep 25110client: (finished: must succeed: sleep 2, in 2.04 seconds)5111client: waiting for success: ! systemctl is-active dedup1 dedup25112worker1 # [ 51.748962] nix-daemon[869]: reaped child process 1097, status = succeeded5113worker1 # [ 51.819617] nix-daemon[869]: accepted connection from pid 1129, user nix-grpc-daemon (trusted)5114worker1 # [ 51.833253] nix-daemon[869]: reaped child process 1132, status = succeeded5115lb # [ 51.873390] niks3-server[957]: 2026/09/22 14:51:11 INFO Received uploads request method=POST path=/api/pending_closures5116worker1 # [ 51.848170] nix-grpc-daemon[884]: time=2026-09-22T14:51:11.965Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5117worker1 # [ 51.851058] nix-grpc-daemon[884]: time=2026-09-22T14:51:11.966Z level=INFO msg="Uploading pihyvyr6gd3qnm368ii42wvcpgsfv1di-slow-dedup (120B)"5118lb # [ 51.929837] niks3-server[957]: 2026/09/22 14:51:11 INFO Received sign narinfos request method=POST path=/api/pending_closures/16/sign5119lb # [ 51.935896] niks3-server[957]: 2026/09/22 14:51:11 INFO Signed narinfos id=16 count=15120worker1 # [ 51.905599] nix-grpc-daemon[884]: time=2026-09-22T14:51:12.022Z level=INFO msg="Uploading 1 narinfos"5121lb # [ 51.944157] niks3-server[957]: 2026/09/22 14:51:11 INFO Registered completed upload object_key=log/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv5122lb # [ 51.949590] niks3-server[957]: 2026/09/22 14:51:11 INFO Registered completed upload object_key=nar/02xhr0h0sj1ji8m0s8995xzbbb8mhbqdpdl1ggx94yqqaykihvz7.nar.zst5123lb # [ 51.958670] niks3-server[957]: 2026/09/22 14:51:11 INFO Registered completed upload object_key=pihyvyr6gd3qnm368ii42wvcpgsfv1di.ls5124lb # [ 51.966037] niks3-server[957]: 2026/09/22 14:51:11 INFO Received complete upload request method=POST path=/api/pending_closures/16/complete5125lb # [ 51.970571] niks3-server[957]: 2026/09/22 14:51:11 INFO Completed upload id=165126lb # [ 51.972437] niks3-server[957]: 2026/09/22 14:51:11 INFO Registered completed upload object_key=pihyvyr6gd3qnm368ii42wvcpgsfv1di.narinfo5127worker1 # [ 51.938137] nix-grpc-daemon[884]: time=2026-09-22T14:51:12.055Z level=INFO msg="Upload complete. (197ms)"5128worker1 # [ 51.946732] nix-grpc-daemon[787]: ts=2026-09-22T14:51:12Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43364 duration_s=4 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv assign_id=8 outputs=15129worker1 # [ 51.954291] nix-grpc-daemon[787]: ts=2026-09-22T14:51:12Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43364 duration_s=3 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv assign_id=8 outputs=15130worker1 # [ 51.966992] nix-daemon[869]: reaped child process 1095, status = succeeded5131client # [ 52.013328] systemd[1]: dedup2.service: Deactivated successfully.5132client # [ 52.017389] systemd[1]: dedup2.service: Consumed 145ms CPU time over 3.572s wall clock time, 35.4M memory peak, 6.5K incoming IP traffic, 5.9K outgoing IP traffic.5133client # [ 52.051145] systemd[1]: dedup1.service: Deactivated successfully.5134client # [ 52.052684] systemd[1]: dedup1.service: Consumed 126ms CPU time over 4.663s wall clock time, 39.5M memory peak, 7.3K incoming IP traffic, 8.5K outgoing IP traffic.5135client: (finished: waiting for success: ! systemctl is-active dedup1 dedup2, in 1.06 seconds)5136client: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }5137client: (finished: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }, in 0.04 seconds)5138worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true5139worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.05 seconds)5140worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true5141worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.05 seconds)5142worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5143worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)5144worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5145worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)5146(finished: subtest: two clients wanting the same drv build it once, in 5.69 seconds)5147subtest: --timeout and --max-silent-time reach the builder5148worker1 # [ 53.179291] nix-grpc-daemon[787]: ts=2026-09-22T14:51:13Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:433645149worker1 # [ 53.192555] nix-grpc-daemon[787]: ts=2026-09-22T14:51:13Z level=debug event=want client=7 drv=qkzy7qnn48bqc3ll9lb2m9zsrx1dr87k-slow-tmot.drv cached=05150worker1 # [ 53.197337] nix-grpc-daemon[787]: ts=2026-09-22T14:51:13Z level=debug event=assigned drv=qkzy7qnn48bqc3ll9lb2m9zsrx1dr87k-slow-tmot.drv worker=192.168.1.4:50051 clients=15151worker2 # [ 53.219067] nix-grpc-daemon[783]: ts=2026-09-22T14:51:13Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558365152worker2 # [ 53.230914] nix-grpc-daemon[783]: ts=2026-09-22T14:51:13Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:558365153worker2 # [ 53.241593] nix-grpc-daemon[783]: ts=2026-09-22T14:51:13Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:558365154worker2 # [ 53.252675] nix-daemon[850]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5155worker2 # [ 53.353773] nix-daemon[850]: accepted connection from pid 1020, user nix-grpc-daemon (trusted)5156worker2 # [ 53.367699] nix-daemon[850]: reaped child process 1023, status = succeeded5157worker2 # [ 53.450329] nix-daemon[850]: accepted connection from pid 1025, user nix-grpc-daemon (trusted)5158worker2 # [ 53.464053] nix-daemon[850]: reaped child process 1028, status = succeeded5159lb # [ 53.486328] niks3-server[957]: 2026/09/22 14:51:12 INFO Received uploads request method=POST path=/api/pending_closures5160worker2 # [ 53.478686] nix-grpc-daemon[878]: time=2026-09-22T14:51:13.384Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5161worker2 # [ 53.480582] nix-grpc-daemon[878]: time=2026-09-22T14:51:13.386Z level=INFO msg="Uploading qkzy7qnn48bqc3ll9lb2m9zsrx1dr87k-slow-tmot.drv (424B)"5162lb # [ 53.514721] niks3-server[957]: 2026/09/22 14:51:12 INFO Registered completed upload object_key=nar/1s4sciw269ksj1s7ymwz5cy4dw9n7bxv3k8wqw69h694spdh4ha2.nar.zst5163lb # [ 53.524974] niks3-server[957]: 2026/09/22 14:51:12 INFO Received sign narinfos request method=POST path=/api/pending_closures/17/sign5164lb # [ 53.532052] niks3-server[957]: 2026/09/22 14:51:12 INFO Signed narinfos id=17 count=15165worker2 # [ 53.519352] nix-grpc-daemon[878]: time=2026-09-22T14:51:13.425Z level=INFO msg="Uploading 1 narinfos"5166lb # [ 53.538483] niks3-server[957]: 2026/09/22 14:51:12 INFO Registered completed upload object_key=qkzy7qnn48bqc3ll9lb2m9zsrx1dr87k.ls5167lb # [ 53.551097] niks3-server[957]: 2026/09/22 14:51:12 INFO Received complete upload request method=POST path=/api/pending_closures/17/complete5168lb # [ 53.557054] niks3-server[957]: 2026/09/22 14:51:12 INFO Completed upload id=175169worker2 # [ 53.544265] nix-grpc-daemon[878]: time=2026-09-22T14:51:13.450Z level=INFO msg="Upload complete. (279ms)"5170lb # [ 53.560651] niks3-server[957]: 2026/09/22 14:51:12 INFO Registered completed upload object_key=qkzy7qnn48bqc3ll9lb2m9zsrx1dr87k.narinfo5171worker2 # [ 53.546775] nix-grpc-daemon[783]: ts=2026-09-22T14:51:13Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55836 duration_s=0 paths=1 nar_bytes_in=4245172worker2 # [ 53.550546] nix-grpc-daemon[783]: ts=2026-09-22T14:51:13Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558365173worker2 # [ 53.559236] nix-daemon[850]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5174worker2 # [ 53.564884] nix-daemon[850]: reaped child process 1018, status = succeeded5175worker2 # [ 53.569730] nix-daemon[850]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5176lb # [ 55.798174] envoy[905]: conn peer=[2001:db8:1::1]:35954 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9481 tx=4358 ms=49185177lb # [ 55.801896] envoy[905]: conn peer=[2001:db8:1::1]:34410 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1051 tx=2889 ms=1335178lb # [ 55.805381] envoy[905]: conn peer=[2001:db8:1::1]:34426 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=410 tx=410 ms=805179lb # [ 55.808820] envoy[905]: conn peer=[2001:db8:1::1]:34444 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=382 tx=330 ms=695180lb # [ 55.812240] envoy[905]: conn peer=[2001:db8:1::1]:34442 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=408 tx=376 ms=765181lb # [ 55.815684] envoy[905]: conn peer=[2001:db8:1::1]:34464 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1576 tx=1598 ms=33845182lb # [ 55.819772] envoy[905]: conn peer=[2001:db8:1::1]:34448 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3011 tx=1561 ms=45125183worker2 # [ 59.182767] nix-grpc-daemon[783]: ts=2026-09-22T14:51:19Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55836 duration_s=5 drv=qkzy7qnn48bqc3ll9lb2m9zsrx1dr87k-slow-tmot.drv assign_id=9 outputs=05184worker2 # [ 59.207964] nix-daemon[850]: reaped child process 1030, status = succeeded5185worker2 # [ 59.210648] nix-daemon[850]: reaped child process 1032, status = succeeded5186worker2 # [ 59.385673] nix-grpc-daemon[783]: ts=2026-09-22T14:51:19Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:558365187worker1 # [ 59.380898] nix-grpc-daemon[787]: ts=2026-09-22T14:51:19Z level=debug event=want client=8 drv=fvgcnys8vngcgklsp349d5pjxr4c66sg-slow-tmom.drv cached=05188worker1 # [ 59.386796] nix-grpc-daemon[787]: ts=2026-09-22T14:51:19Z level=debug event=assigned drv=fvgcnys8vngcgklsp349d5pjxr4c66sg-slow-tmom.drv worker=192.168.1.3:50051 clients=15189worker1 # [ 59.393402] nix-grpc-daemon[787]: ts=2026-09-22T14:51:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:433645190worker1 # [ 59.404576] nix-grpc-daemon[787]: ts=2026-09-22T14:51:19Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:433645191worker1 # [ 59.415181] nix-grpc-daemon[787]: ts=2026-09-22T14:51:19Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:433645192worker1 # [ 59.426809] nix-daemon[869]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5193worker1 # [ 59.536572] nix-daemon[869]: accepted connection from pid 1149, user nix-grpc-daemon (trusted)5194worker1 # [ 59.552861] nix-daemon[869]: reaped child process 1152, status = succeeded5195worker1 # [ 59.643615] nix-daemon[869]: accepted connection from pid 1154, user nix-grpc-daemon (trusted)5196worker1 # [ 59.656737] nix-daemon[869]: reaped child process 1157, status = succeeded5197lb # [ 59.698791] niks3-server[957]: 2026/09/22 14:51:19 INFO Received uploads request method=POST path=/api/pending_closures5198worker1 # [ 59.672615] nix-grpc-daemon[884]: time=2026-09-22T14:51:19.789Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5199worker1 # [ 59.674157] nix-grpc-daemon[884]: time=2026-09-22T14:51:19.791Z level=INFO msg="Uploading fvgcnys8vngcgklsp349d5pjxr4c66sg-slow-tmom.drv (424B)"5200lb # [ 59.736730] niks3-server[957]: 2026/09/22 14:51:19 INFO Registered completed upload object_key=nar/013y5sy2r6v0qdwv0dzpgvdfy08bds4mfh2qpvbgjazzrg2v71hp.nar.zst5201lb # [ 59.742842] niks3-server[957]: 2026/09/22 14:51:19 INFO Received sign narinfos request method=POST path=/api/pending_closures/18/sign5202lb # [ 59.747735] niks3-server[957]: 2026/09/22 14:51:19 INFO Signed narinfos id=18 count=15203worker1 # [ 59.716684] nix-grpc-daemon[884]: time=2026-09-22T14:51:19.833Z level=INFO msg="Uploading 1 narinfos"5204lb # [ 59.762491] niks3-server[957]: 2026/09/22 14:51:19 INFO Registered completed upload object_key=fvgcnys8vngcgklsp349d5pjxr4c66sg.ls5205lb # [ 59.768181] niks3-server[957]: 2026/09/22 14:51:19 INFO Received complete upload request method=POST path=/api/pending_closures/18/complete5206lb # [ 59.775994] niks3-server[957]: 2026/09/22 14:51:19 INFO Completed upload id=185207worker1 # [ 59.742096] nix-grpc-daemon[884]: time=2026-09-22T14:51:19.859Z level=INFO msg="Upload complete. (303ms)"5208lb # [ 59.780463] niks3-server[957]: 2026/09/22 14:51:19 INFO Registered completed upload object_key=fvgcnys8vngcgklsp349d5pjxr4c66sg.narinfo5209worker1 # [ 59.746642] nix-grpc-daemon[787]: ts=2026-09-22T14:51:19Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43364 duration_s=0 paths=1 nar_bytes_in=4245210worker1 # [ 59.752265] nix-grpc-daemon[787]: ts=2026-09-22T14:51:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:433645211worker1 # [ 59.762496] nix-daemon[869]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5212worker1 # [ 59.773194] nix-daemon[869]: reaped child process 1147, status = succeeded5213worker1 # [ 59.776371] nix-daemon[869]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5214lb # [ 65.799431] envoy[905]: conn peer=[2001:db8:1::1]:36238 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2983 tx=953 ms=60375215worker1 # [ 65.788973] nix-grpc-daemon[787]: ts=2026-09-22T14:51:25Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43364 duration_s=6 drv=fvgcnys8vngcgklsp349d5pjxr4c66sg-slow-tmom.drv assign_id=10 outputs=05216worker1 # [ 65.801455] nix-daemon[869]: reaped child process 1159, status = succeeded5217worker1 # [ 65.802497] nix-daemon[869]: reaped child process 1161, status = succeeded5218(finished: subtest: --timeout and --max-silent-time reach the builder, in 12.86 seconds)5219subtest: the first client leaving does not fail the second5220client: must succeed: systemd-run --unit share1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag share5221client # [ 65.938592] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key" --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag share.5222client # Running as unit: share1.service; invocation ID: 9353260fa9914afa93e48d86455308695223client: (finished: must succeed: systemd-run --unit share1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag share, in 0.03 seconds)5224client: must succeed: sleep 1; systemd-run --unit share2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag share5225worker2 # [ 66.060432] nix-grpc-daemon[783]: ts=2026-09-22T14:51:25Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:558365226client # [ 66.077646] nix[1039]: don't know how to build these paths:5227client # [ 66.078598] nix[1039]: /nix/store/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv5228worker1 # [ 66.056408] nix-grpc-daemon[787]: ts=2026-09-22T14:51:26Z level=debug event=want client=9 drv=j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv cached=05229worker2 # [ 66.076868] nix-grpc-daemon[783]: ts=2026-09-22T14:51:25Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558365230worker1 # [ 66.060774] nix-grpc-daemon[787]: ts=2026-09-22T14:51:26Z level=debug event=assigned drv=j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv worker=192.168.1.4:50051 clients=15231worker2 # [ 66.087304] nix-grpc-daemon[783]: ts=2026-09-22T14:51:25Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:558365232client # [ 66.106216] nix[1039]: copying 1 paths to 'lb:50051'...5233worker2 # [ 66.095154] nix-grpc-daemon[783]: ts=2026-09-22T14:51:26Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:558365234client # [ 66.107049] nix[1039]: copying path '/nix/store/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv' to 'lb:50051'...5235worker2 # [ 66.099970] nix-daemon[850]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5236worker2 # [ 66.184444] nix-daemon[850]: accepted connection from pid 1055, user nix-grpc-daemon (trusted)5237worker2 # [ 66.197679] nix-daemon[850]: reaped child process 1058, status = succeeded5238worker2 # [ 66.278179] nix-daemon[850]: accepted connection from pid 1060, user nix-grpc-daemon (trusted)5239worker2 # [ 66.290863] nix-daemon[850]: reaped child process 1063, status = succeeded5240lb # [ 66.315266] niks3-server[957]: 2026/09/22 14:51:25 INFO Received uploads request method=POST path=/api/pending_closures5241worker2 # [ 66.319974] nix-grpc-daemon[878]: time=2026-09-22T14:51:26.226Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5242worker2 # [ 66.321797] nix-grpc-daemon[878]: time=2026-09-22T14:51:26.228Z level=INFO msg="Uploading j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv (432B)"5243lb # [ 66.358586] niks3-server[957]: 2026/09/22 14:51:25 INFO Received sign narinfos request method=POST path=/api/pending_closures/19/sign5244worker2 # [ 66.347613] nix-grpc-daemon[878]: time=2026-09-22T14:51:26.254Z level=INFO msg="Uploading 1 narinfos"5245lb # [ 66.364238] niks3-server[957]: 2026/09/22 14:51:25 INFO Signed narinfos id=19 count=15246lb # [ 66.374887] niks3-server[957]: 2026/09/22 14:51:25 INFO Received complete upload request method=POST path=/api/pending_closures/19/complete5247lb # [ 66.381167] niks3-server[957]: 2026/09/22 14:51:25 INFO Registered completed upload object_key=nar/0s0b0gfhn36p0aylk1qa64cgw01bi7a8ca93iv4zg2q7wqcjcqhs.nar.zst5248lb # [ 66.385829] niks3-server[957]: 2026/09/22 14:51:25 INFO Registered completed upload object_key=j2xs6vfpc2dkbabf87x31z8q4j8chjfw.ls5249lb # [ 66.393990] niks3-server[957]: 2026/09/22 14:51:25 INFO Completed upload id=195250worker2 # [ 66.378530] nix-grpc-daemon[878]: time=2026-09-22T14:51:26.285Z level=INFO msg="Upload complete. (272ms)"5251worker2 # [ 66.380578] nix-grpc-daemon[783]: ts=2026-09-22T14:51:26Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55836 duration_s=0 paths=1 nar_bytes_in=4325252worker2 # [ 66.383443] nix-grpc-daemon[783]: ts=2026-09-22T14:51:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558365253worker2 # [ 66.390965] nix-daemon[850]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5254worker2 # [ 66.393275] nix-daemon[850]: reaped child process 1053, status = succeeded5255worker2 # [ 66.398622] nix-daemon[850]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5256lb # [ 66.413937] niks3-server[957]: 2026/09/22 14:51:25 INFO Registered completed upload object_key=j2xs6vfpc2dkbabf87x31z8q4j8chjfw.narinfo5257client # [ 66.409533] nix[1039]: building '/nix/store/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv' on lb:50051...5258client # [ 66.979985] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key" --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag share.5259client # Running as unit: share2.service; invocation ID: 02ba42944fbb4eecac4b61680ccdf2405260client: (finished: must succeed: sleep 1; systemd-run --unit share2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag share, in 1.04 seconds)5261client # [ 67.107702] nix[1055]: don't know how to build these paths:5262client # [ 67.108656] nix[1055]: /nix/store/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv5263worker1 # [ 67.076720] nix-grpc-daemon[787]: ts=2026-09-22T14:51:27Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:433645264worker1 # [ 67.086367] nix-grpc-daemon[787]: ts=2026-09-22T14:51:27Z level=debug event=want client=10 drv=j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv cached=05265worker2 # [ 67.107945] nix-grpc-daemon[783]: ts=2026-09-22T14:51:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558365266client # [ 67.121819] nix[1055]: building '/nix/store/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv' on lb:50051...5267client: must succeed: sleep 2; systemctl kill -s INT share15268client # [ 70.219266] systemd[1]: share1.service: Sent signal SIGINT to main process 1039 (nix) on client request.5269client # [ 70.226884] nix[1039]: shutting downj2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED)5270client # [ 70.232904] nix[1039]: error: interrupted by the user5271client: (finished: must succeed: sleep 2; systemctl kill -s INT share1, in 2.06 seconds)5272client: waiting for success: ! systemctl is-active share15273client # [ 70.275255] systemd[1]: share1.service: Main process exited, code=exited, status=1/FAILURE5274client # [ 70.276755] systemd[1]: share1.service: Failed with result 'exit-code'.5275client # [ 70.278714] systemd[1]: share1.service: Consumed 118ms CPU time over 4.339s wall clock time, 41.5M memory peak, 6.1K incoming IP traffic, 8K outgoing IP traffic.5276client: (finished: waiting for success: ! systemctl is-active share1, in 1.08 seconds)5277client: waiting for success: ! systemctl is-active share25278worker2 # [ 71.654388] nix-daemon[850]: reaped child process 1067, status = succeeded5279worker2 # [ 71.712800] nix-daemon[850]: accepted connection from pid 1101, user nix-grpc-daemon (trusted)5280worker2 # [ 71.726257] nix-daemon[850]: reaped child process 1104, status = succeeded5281lb # [ 71.749387] niks3-server[957]: 2026/09/22 14:51:31 INFO Received uploads request method=POST path=/api/pending_closures5282worker2 # [ 71.741472] nix-grpc-daemon[878]: time=2026-09-22T14:51:31.647Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5283worker2 # [ 71.743048] nix-grpc-daemon[878]: time=2026-09-22T14:51:31.649Z level=INFO msg="Uploading b9lg6znd0rinl36gy6ixq1dcj6b8bhhw-slow-share (120B)"5284lb # [ 71.786688] niks3-server[957]: 2026/09/22 14:51:31 INFO Registered completed upload object_key=nar/172qq270pqbv9ipsik19pmld0m7fj8r5nyyxq3slfx9n7zcaa9s0.nar.zst5285lb # [ 71.801144] niks3-server[957]: 2026/09/22 14:51:31 INFO Received sign narinfos request method=POST path=/api/pending_closures/20/sign5286lb # [ 71.811392] niks3-server[957]: 2026/09/22 14:51:31 INFO Signed narinfos id=20 count=15287worker2 # [ 71.795856] nix-grpc-daemon[878]: time=2026-09-22T14:51:31.702Z level=INFO msg="Uploading 1 narinfos"5288lb # [ 71.814051] niks3-server[957]: 2026/09/22 14:51:31 INFO Registered completed upload object_key=log/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv5289lb # [ 71.825118] niks3-server[957]: 2026/09/22 14:51:31 INFO Received complete upload request method=POST path=/api/pending_closures/20/complete5290lb # [ 71.829421] niks3-server[957]: 2026/09/22 14:51:31 INFO Registered completed upload object_key=b9lg6znd0rinl36gy6ixq1dcj6b8bhhw.ls5291lb # [ 71.833471] niks3-server[957]: 2026/09/22 14:51:31 INFO Completed upload id=205292worker2 # [ 71.818264] nix-grpc-daemon[878]: time=2026-09-22T14:51:31.724Z level=INFO msg="Upload complete. (208ms)"5293lb # [ 71.835769] niks3-server[957]: 2026/09/22 14:51:31 INFO Registered completed upload object_key=b9lg6znd0rinl36gy6ixq1dcj6b8bhhw.narinfo5294worker2 # [ 71.824932] nix-grpc-daemon[783]: ts=2026-09-22T14:51:31Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55836 duration_s=4 drv=j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv assign_id=11 outputs=15295worker2 # [ 71.831423] nix-grpc-daemon[783]: ts=2026-09-22T14:51:31Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55836 duration_s=5 drv=j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv assign_id=11 outputs=15296worker2 # [ 71.844309] nix-daemon[850]: reaped child process 1065, status = succeeded5297client # [ 71.907963] systemd[1]: share2.service: Deactivated successfully.5298client # [ 71.912241] systemd[1]: share2.service: Consumed 118ms CPU time over 4.930s wall clock time, 35.4M memory peak, 6K incoming IP traffic, 5.8K outgoing IP traffic.5299client: (finished: waiting for success: ! systemctl is-active share2, in 1.08 seconds)5300client: must succeed: systemctl show -p Result share2 | grep -q success || { journalctl -u share2 >&2; false; }5301client: (finished: must succeed: systemctl show -p Result share2 | grep -q success || { journalctl -u share2 >&2; false; }, in 0.03 seconds)5302(finished: subtest: the first client leaving does not fail the second, in 6.68 seconds)5303subtest: build hook: nix-daemon with builders = grpc://lb5304client: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --print-out-paths --no-link -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag hook 2>/tmp/hook.log5305client # [ 72.685842] systemd[1]: Started Nix Daemon.5306client # [ 72.772430] nix-daemon[1107]: accepted connection from pid 1104, user root (trusted)5307worker2 # [ 72.881702] nix-grpc-daemon[783]: ts=2026-09-22T14:51:32Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:558365308worker1 # [ 72.867415] nix-grpc-daemon[787]: ts=2026-09-22T14:51:32Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:433645309worker1 # [ 72.878172] nix-grpc-daemon[787]: ts=2026-09-22T14:51:32Z level=debug event=want client=11 drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv cached=05310worker1 # [ 72.883736] nix-grpc-daemon[787]: ts=2026-09-22T14:51:32Z level=debug event=assigned drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv worker=192.168.1.3:50051 clients=15311worker1 # [ 72.889286] nix-grpc-daemon[787]: ts=2026-09-22T14:51:33Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:433645312worker1 # [ 72.901182] nix-grpc-daemon[787]: ts=2026-09-22T14:51:33Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:433645313worker1 # [ 72.912231] nix-grpc-daemon[787]: ts=2026-09-22T14:51:33Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:433645314worker1 # [ 72.923571] nix-daemon[869]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5315worker1 # [ 73.021759] nix-daemon[869]: accepted connection from pid 1206, user nix-grpc-daemon (trusted)5316worker1 # [ 73.035891] nix-daemon[869]: reaped child process 1209, status = succeeded5317worker1 # [ 73.118167] nix-daemon[869]: accepted connection from pid 1211, user nix-grpc-daemon (trusted)5318worker1 # [ 73.130942] nix-daemon[869]: reaped child process 1214, status = succeeded5319lb # [ 73.171448] niks3-server[957]: 2026/09/22 14:51:32 INFO Received uploads request method=POST path=/api/pending_closures5320worker1 # [ 73.145108] nix-grpc-daemon[884]: time=2026-09-22T14:51:33.262Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5321worker1 # [ 73.146974] nix-grpc-daemon[884]: time=2026-09-22T14:51:33.263Z level=INFO msg="Uploading pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv (512B)"5322lb # [ 73.196673] niks3-server[957]: 2026/09/22 14:51:32 INFO Registered completed upload object_key=nar/08mhnmxkz6k9gck9zisdh6jb3dqb53xqy0jak6cm9lpasdvr4sk7.nar.zst5323lb # [ 73.208668] niks3-server[957]: 2026/09/22 14:51:32 INFO Received sign narinfos request method=POST path=/api/pending_closures/21/sign5324lb # [ 73.214357] niks3-server[957]: 2026/09/22 14:51:32 INFO Signed narinfos id=21 count=15325worker1 # [ 73.184313] nix-grpc-daemon[884]: time=2026-09-22T14:51:33.301Z level=INFO msg="Uploading 1 narinfos"5326lb # [ 73.225865] niks3-server[957]: 2026/09/22 14:51:32 INFO Registered completed upload object_key=pd51i21khb2xrd9paajf80pb96sw8zn0.ls5327lb # [ 73.234998] niks3-server[957]: 2026/09/22 14:51:32 INFO Received complete upload request method=POST path=/api/pending_closures/21/complete5328lb # [ 73.239935] niks3-server[957]: 2026/09/22 14:51:32 INFO Completed upload id=215329worker1 # [ 73.208272] nix-grpc-daemon[884]: time=2026-09-22T14:51:33.325Z level=INFO msg="Upload complete. (273ms)"5330worker1 # [ 73.211273] nix-grpc-daemon[787]: ts=2026-09-22T14:51:33Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43364 duration_s=0 paths=1 nar_bytes_in=5125331lb # [ 73.249699] niks3-server[957]: 2026/09/22 14:51:32 INFO Registered completed upload object_key=pd51i21khb2xrd9paajf80pb96sw8zn0.narinfo5332worker1 # [ 73.216557] nix-grpc-daemon[787]: ts=2026-09-22T14:51:33Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:433645333worker1 # [ 73.226120] nix-daemon[869]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5334worker1 # [ 73.232695] nix-daemon[869]: reaped child process 1204, status = succeeded5335worker1 # [ 73.237610] nix-daemon[869]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5336worker1 # [ 74.253884] nix-daemon[869]: reaped child process 1218, status = succeeded5337worker1 # [ 74.328622] nix-daemon[869]: accepted connection from pid 1234, user nix-grpc-daemon (trusted)5338worker1 # [ 74.342354] nix-daemon[869]: reaped child process 1237, status = succeeded5339lb # [ 74.382645] niks3-server[957]: 2026/09/22 14:51:33 INFO Received uploads request method=POST path=/api/pending_closures5340worker1 # [ 74.357090] nix-grpc-daemon[884]: time=2026-09-22T14:51:34.474Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5341worker1 # [ 74.358578] nix-grpc-daemon[884]: time=2026-09-22T14:51:34.475Z level=INFO msg="Uploading 6zgzdy628dcdg0dbc5qi7nisk08n4kp2-farm-a (128B)"5342lb # [ 74.420204] niks3-server[957]: 2026/09/22 14:51:33 INFO Registered completed upload object_key=nar/0pbbxl4kxy7zl7x4lp43ywljwn82mxzna2wgpb1bz1z765nwp9ki.nar.zst5343lb # [ 74.433294] niks3-server[957]: 2026/09/22 14:51:33 INFO Registered completed upload object_key=log/pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv5344lb # [ 74.443105] niks3-server[957]: 2026/09/22 14:51:33 INFO Received sign narinfos request method=POST path=/api/pending_closures/22/sign5345lb # [ 74.448098] niks3-server[957]: 2026/09/22 14:51:33 INFO Signed narinfos id=22 count=15346worker1 # [ 74.416559] nix-grpc-daemon[884]: time=2026-09-22T14:51:34.533Z level=INFO msg="Uploading 1 narinfos"5347lb # [ 74.453782] niks3-server[957]: 2026/09/22 14:51:33 INFO Registered completed upload object_key=6zgzdy628dcdg0dbc5qi7nisk08n4kp2.ls5348lb # [ 74.464110] niks3-server[957]: 2026/09/22 14:51:33 INFO Received complete upload request method=POST path=/api/pending_closures/22/complete5349lb # [ 74.468074] niks3-server[957]: 2026/09/22 14:51:33 INFO Completed upload id=225350worker1 # [ 74.433933] nix-grpc-daemon[884]: time=2026-09-22T14:51:34.551Z level=INFO msg="Upload complete. (189ms)"5351lb # [ 74.469750] niks3-server[957]: 2026/09/22 14:51:33 INFO Registered completed upload object_key=6zgzdy628dcdg0dbc5qi7nisk08n4kp2.narinfo5352worker1 # [ 74.436908] nix-grpc-daemon[787]: ts=2026-09-22T14:51:34Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43364 duration_s=1 drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv assign_id=12 outputs=15353worker1 # [ 74.445570] nix-daemon[869]: reaped child process 1216, status = succeeded5354worker1 # [ 74.483894] nix-grpc-daemon[787]: ts=2026-09-22T14:51:34Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:433645355worker2 # [ 74.519814] nix-grpc-daemon[783]: ts=2026-09-22T14:51:34Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:558365356worker2 # [ 74.528770] nix-grpc-daemon[783]: copying path '/nix/store/6zgzdy628dcdg0dbc5qi7nisk08n4kp2-farm-a' from 'http://lb:5751'...5357worker2 # [ 74.536657] nix-grpc-daemon[783]: ts=2026-09-22T14:51:34Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:55836 duration_s=0 paths=1 nar_bytes_out=1285358worker1 # [ 74.683864] nix-grpc-daemon[787]: ts=2026-09-22T14:51:34Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:433645359worker2 # [ 74.710444] nix-grpc-daemon[783]: ts=2026-09-22T14:51:34Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:558365360worker2 # [ 74.720472] nix-grpc-daemon[878]: time=2026-09-22T14:51:34.626Z level=INFO msg="All 1 paths already cached"5361worker1 # [ 74.715799] nix-grpc-daemon[787]: ts=2026-09-22T14:51:34Z level=debug event=want client=12 drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv cached=05362worker1 # [ 74.721223] nix-grpc-daemon[787]: ts=2026-09-22T14:51:34Z level=debug event=assigned drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv worker=192.168.1.3:50051 clients=15363worker1 # [ 74.726690] nix-grpc-daemon[787]: ts=2026-09-22T14:51:34Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:433645364worker1 # [ 74.738903] nix-grpc-daemon[787]: ts=2026-09-22T14:51:34Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:433645365worker1 # [ 74.751412] nix-grpc-daemon[884]: time=2026-09-22T14:51:34.868Z level=INFO msg="All 1 paths already cached"5366worker1 # [ 74.758686] nix-grpc-daemon[787]: ts=2026-09-22T14:51:34Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:433645367worker1 # [ 74.768175] nix-daemon[869]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5368worker1 # [ 74.864745] nix-daemon[869]: accepted connection from pid 1243, user nix-grpc-daemon (trusted)5369worker1 # [ 74.877494] nix-daemon[869]: reaped child process 1246, status = succeeded5370worker1 # [ 74.960493] nix-daemon[869]: accepted connection from pid 1248, user nix-grpc-daemon (trusted)5371worker1 # [ 74.973319] nix-daemon[869]: reaped child process 1251, status = succeeded5372lb # [ 75.013942] niks3-server[957]: 2026/09/22 14:51:34 INFO Received uploads request method=POST path=/api/pending_closures5373worker1 # [ 74.984399] nix-grpc-daemon[884]: time=2026-09-22T14:51:35.101Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5374worker1 # [ 74.985802] nix-grpc-daemon[884]: time=2026-09-22T14:51:35.101Z level=INFO msg="Uploading vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv (624B)"5375lb # [ 75.029143] niks3-server[957]: 2026/09/22 14:51:34 INFO Received sign narinfos request method=POST path=/api/pending_closures/23/sign5376lb # [ 75.031693] niks3-server[957]: 2026/09/22 14:51:34 INFO Signed narinfos id=23 count=15377worker1 # [ 74.999247] nix-grpc-daemon[884]: time=2026-09-22T14:51:35.116Z level=INFO msg="Uploading 1 narinfos"5378lb # [ 75.033409] niks3-server[957]: 2026/09/22 14:51:34 INFO Registered completed upload object_key=nar/1190412lxhybpj2b0886g97h0cpm7xx8r4virf8q4n7i91409ivg.nar.zst5379lb # [ 75.039088] niks3-server[957]: 2026/09/22 14:51:34 INFO Registered completed upload object_key=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8.ls5380lb # [ 75.044391] niks3-server[957]: 2026/09/22 14:51:34 INFO Received complete upload request method=POST path=/api/pending_closures/23/complete5381lb # [ 75.048505] niks3-server[957]: 2026/09/22 14:51:34 INFO Completed upload id=235382worker1 # [ 75.014302] nix-grpc-daemon[884]: time=2026-09-22T14:51:35.131Z level=INFO msg="Upload complete. (234ms)"5383lb # [ 75.050383] niks3-server[957]: 2026/09/22 14:51:34 INFO Registered completed upload object_key=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8.narinfo5384worker1 # [ 75.016517] nix-grpc-daemon[787]: ts=2026-09-22T14:51:35Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43364 duration_s=0 paths=1 nar_bytes_in=6245385worker1 # [ 75.020596] nix-grpc-daemon[787]: ts=2026-09-22T14:51:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:433645386worker1 # [ 75.026196] nix-daemon[869]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5387worker1 # [ 75.028423] nix-daemon[869]: reaped child process 1241, status = succeeded5388worker1 # [ 75.033529] nix-daemon[869]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5389worker1 # [ 75.679309] nix-daemon[869]: reaped child process 1255, status = succeeded5390worker1 # [ 75.759187] nix-daemon[869]: accepted connection from pid 1271, user nix-grpc-daemon (trusted)5391lb # [ 75.799959] envoy[905]: conn peer=[2001:db8:1::1]:36252 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3008 tx=951 ms=64395392lb # [ 75.804601] envoy[905]: conn peer=[2001:db8:1::1]:50988 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2965 tx=770 ms=41735393worker1 # [ 75.773067] nix-daemon[869]: reaped child process 1274, status = succeeded5394lb # [ 75.808204] envoy[905]: conn peer=[2001:db8:1::1]:50998 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1453 tx=1197 ms=47825395lb # [ 75.812179] envoy[905]: conn peer=[2001:db8:1::1]:58046 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=461 tx=334 ms=305396lb # [ 75.816433] envoy[905]: conn peer=[2001:db8:1::1]:58034 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3385 tx=1922 ms=16725397lb # [ 75.821264] niks3-server[957]: 2026/09/22 14:51:35 INFO Received uploads request method=POST path=/api/pending_closures5398worker1 # [ 75.814077] nix-grpc-daemon[884]: time=2026-09-22T14:51:35.931Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5399worker1 # [ 75.815918] nix-grpc-daemon[884]: time=2026-09-22T14:51:35.932Z level=INFO msg="Uploading mrrcl9jb4hh6rly2df2vfym5sq1235hf-farm-b (176B)"5400lb # [ 75.886653] niks3-server[957]: 2026/09/22 14:51:35 INFO Registered completed upload object_key=nar/1mgm44m4ndgi5l3yi2qv3v3sx48190m8d4ycw3r4pfzy4z1wq2s6.nar.zst5401lb # [ 75.890059] niks3-server[957]: 2026/09/22 14:51:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/24/sign5402lb # [ 75.896084] niks3-server[957]: 2026/09/22 14:51:35 INFO Signed narinfos id=24 count=15403worker1 # [ 75.864657] nix-grpc-daemon[884]: time=2026-09-22T14:51:35.981Z level=INFO msg="Uploading 1 narinfos"5404lb # [ 75.900748] niks3-server[957]: 2026/09/22 14:51:35 INFO Registered completed upload object_key=log/vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv5405lb # [ 75.905614] niks3-server[957]: 2026/09/22 14:51:35 INFO Registered completed upload object_key=mrrcl9jb4hh6rly2df2vfym5sq1235hf.ls5406lb # [ 75.914299] niks3-server[957]: 2026/09/22 14:51:35 INFO Received complete upload request method=POST path=/api/pending_closures/24/complete5407lb # [ 75.917722] niks3-server[957]: 2026/09/22 14:51:35 INFO Completed upload id=245408worker1 # [ 75.883723] nix-grpc-daemon[884]: time=2026-09-22T14:51:36.000Z level=INFO msg="Upload complete. (243ms)"5409lb # [ 75.919970] niks3-server[957]: 2026/09/22 14:51:35 INFO Registered completed upload object_key=mrrcl9jb4hh6rly2df2vfym5sq1235hf.narinfo5410worker1 # [ 75.890360] nix-grpc-daemon[787]: ts=2026-09-22T14:51:36Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43364 duration_s=0 drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv assign_id=13 outputs=15411worker2 # [ 75.918757] nix-grpc-daemon[783]: ts=2026-09-22T14:51:35Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:558365412worker1 # [ 75.906325] nix-daemon[869]: reaped child process 1253, status = succeeded5413worker2 # [ 75.927320] nix-grpc-daemon[783]: copying path '/nix/store/mrrcl9jb4hh6rly2df2vfym5sq1235hf-farm-b' from 'http://lb:5751'...5414worker2 # [ 75.947744] nix-grpc-daemon[783]: ts=2026-09-22T14:51:35Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:558365415worker2 # [ 75.952834] nix-grpc-daemon[783]: ts=2026-09-22T14:51:35Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:55836 duration_s=0 paths=1 nar_bytes_out=1765416worker2 # [ 76.162179] nix-grpc-daemon[783]: ts=2026-09-22T14:51:36Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:558365417worker2 # [ 76.168986] nix-grpc-daemon[783]: ts=2026-09-22T14:51:36Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:558365418worker2 # [ 76.174293] nix-grpc-daemon[878]: time=2026-09-22T14:51:36.079Z level=INFO msg="All 2 paths already cached"5419worker1 # [ 76.166724] nix-grpc-daemon[787]: ts=2026-09-22T14:51:36Z level=debug event=want client=13 drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv cached=05420worker1 # [ 76.169038] nix-grpc-daemon[787]: ts=2026-09-22T14:51:36Z level=debug event=assigned drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv worker=192.168.1.3:50051 clients=15421worker1 # [ 76.171720] nix-grpc-daemon[787]: ts=2026-09-22T14:51:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:433645422worker1 # [ 76.179098] nix-grpc-daemon[787]: ts=2026-09-22T14:51:36Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:433645423worker1 # [ 76.187346] nix-grpc-daemon[884]: time=2026-09-22T14:51:36.304Z level=INFO msg="All 2 paths already cached"5424worker1 # [ 76.191620] nix-grpc-daemon[787]: ts=2026-09-22T14:51:36Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:433645425worker1 # [ 76.195192] nix-daemon[869]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5426worker1 # [ 76.284138] nix-daemon[869]: accepted connection from pid 1280, user nix-grpc-daemon (trusted)5427worker1 # [ 76.299070] nix-daemon[869]: reaped child process 1283, status = succeeded5428worker1 # [ 76.380100] nix-daemon[869]: accepted connection from pid 1285, user nix-grpc-daemon (trusted)5429worker1 # [ 76.394186] nix-daemon[869]: reaped child process 1288, status = succeeded5430lb # [ 76.433084] niks3-server[957]: 2026/09/22 14:51:35 INFO Received uploads request method=POST path=/api/pending_closures5431worker1 # [ 76.406178] nix-grpc-daemon[884]: time=2026-09-22T14:51:36.523Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5432worker1 # [ 76.408187] nix-grpc-daemon[884]: time=2026-09-22T14:51:36.525Z level=INFO msg="Uploading bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv (752B)"5433lb # [ 76.464368] niks3-server[957]: 2026/09/22 14:51:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/25/sign5434worker1 # [ 76.435167] nix-grpc-daemon[884]: time=2026-09-22T14:51:36.552Z level=INFO msg="Uploading 1 narinfos"5435lb # [ 76.469225] niks3-server[957]: 2026/09/22 14:51:35 INFO Signed narinfos id=25 count=15436lb # [ 76.477471] niks3-server[957]: 2026/09/22 14:51:35 INFO Registered completed upload object_key=bshr7sr3z72cf1fa1a1b8fifr45jgpsw.ls5437lb # [ 76.481812] niks3-server[957]: 2026/09/22 14:51:35 INFO Registered completed upload object_key=nar/0d4r016fc570pq02p1j53gahxygvvyr62937d84sa58ri3rzk000.nar.zst5438lb # [ 76.493296] niks3-server[957]: 2026/09/22 14:51:35 INFO Received complete upload request method=POST path=/api/pending_closures/25/complete5439worker1 # [ 76.464517] nix-grpc-daemon[884]: time=2026-09-22T14:51:36.581Z level=INFO msg="Upload complete. (263ms)"5440lb # [ 76.498994] niks3-server[957]: 2026/09/22 14:51:35 INFO Completed upload id=255441worker1 # [ 76.466937] nix-grpc-daemon[787]: ts=2026-09-22T14:51:36Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43364 duration_s=0 paths=1 nar_bytes_in=7525442worker1 # [ 76.469725] nix-grpc-daemon[787]: ts=2026-09-22T14:51:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:433645443lb # [ 76.505474] niks3-server[957]: 2026/09/22 14:51:35 INFO Registered completed upload object_key=bshr7sr3z72cf1fa1a1b8fifr45jgpsw.narinfo5444worker1 # [ 76.474938] nix-daemon[869]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5445worker1 # [ 76.480125] nix-daemon[869]: reaped child process 1278, status = succeeded5446worker1 # [ 76.484945] nix-daemon[869]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5447worker1 # [ 77.501939] nix-daemon[869]: reaped child process 1292, status = succeeded5448worker1 # [ 77.574908] nix-daemon[869]: accepted connection from pid 1308, user nix-grpc-daemon (trusted)5449worker1 # [ 77.587789] nix-daemon[869]: reaped child process 1311, status = succeeded5450lb # [ 77.629791] niks3-server[957]: 2026/09/22 14:51:37 INFO Received uploads request method=POST path=/api/pending_closures5451worker1 # [ 77.603323] nix-grpc-daemon[884]: time=2026-09-22T14:51:37.720Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5452worker1 # [ 77.604936] nix-grpc-daemon[884]: time=2026-09-22T14:51:37.722Z level=INFO msg="Uploading aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top (232B)"5453lb # [ 77.673681] niks3-server[957]: 2026/09/22 14:51:37 INFO Registered completed upload object_key=nar/1r64sjcvz2vpjvwv3psvwf0bfifdzwiafvy1m89v5is1mm18ds87.nar.zst5454lb # [ 77.682653] niks3-server[957]: 2026/09/22 14:51:37 INFO Registered completed upload object_key=log/bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv5455lb # [ 77.690946] niks3-server[957]: 2026/09/22 14:51:37 INFO Received sign narinfos request method=POST path=/api/pending_closures/26/sign5456lb # [ 77.696992] niks3-server[957]: 2026/09/22 14:51:37 INFO Signed narinfos id=26 count=15457worker1 # [ 77.662575] nix-grpc-daemon[884]: time=2026-09-22T14:51:37.779Z level=INFO msg="Uploading 1 narinfos"5458lb # [ 77.704131] niks3-server[957]: 2026/09/22 14:51:37 INFO Registered completed upload object_key=aayfavl0hksy8qs9qc49znrwnfc5d2sr.ls5459lb # [ 77.708852] niks3-server[957]: 2026/09/22 14:51:37 INFO Received complete upload request method=POST path=/api/pending_closures/26/complete5460lb # [ 77.711822] niks3-server[957]: 2026/09/22 14:51:37 INFO Completed upload id=265461worker1 # [ 77.677752] nix-grpc-daemon[884]: time=2026-09-22T14:51:37.794Z level=INFO msg="Upload complete. (185ms)"5462lb # [ 77.714641] niks3-server[957]: 2026/09/22 14:51:37 INFO Registered completed upload object_key=aayfavl0hksy8qs9qc49znrwnfc5d2sr.narinfo5463worker2 # [ 77.707350] nix-grpc-daemon[783]: ts=2026-09-22T14:51:37Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:558365464worker1 # [ 77.685273] nix-grpc-daemon[787]: ts=2026-09-22T14:51:37Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43364 duration_s=1 drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv assign_id=14 outputs=15465worker2 # [ 77.712682] nix-grpc-daemon[783]: copying path '/nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top' from 'http://lb:5751'...5466worker1 # [ 77.698664] nix-daemon[869]: reaped child process 1290, status = succeeded5467worker1 # [ 77.711604] nix-grpc-daemon[787]: ts=2026-09-22T14:51:37Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:433645468worker1 # [ 77.716073] nix-grpc-daemon[787]: ts=2026-09-22T14:51:37Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:43364 duration_s=0 paths=1 nar_bytes_out=2325469client # [ 77.855493] nix-daemon[1107]: reaped child process 1109, status = succeeded5470client: (finished: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --print-out-paths --no-link -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag hook 2>/tmp/hook.log, in 5.27 seconds)5471client: must succeed: grep farm-top-hook /nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top5472client: (finished: must succeed: grep farm-top-hook /nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top, in 0.02 seconds)5473client: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }5474client: (finished: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }, in 0.02 seconds)5475client: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }5476client: (finished: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }, in 0.02 seconds)5477(finished: subtest: build hook: nix-daemon with builders = grpc://lb, in 5.33 seconds)5478subtest: an input only one worker has still reaches the builder5479client: must succeed: nix-build --no-out-link /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix -A input --argstr tag w1only5480client # this derivation will be built:5481client # /nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv5482client # building '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv'...5483client: (finished: must succeed: nix-build --no-out-link /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix -A input --argstr tag w1only, in 0.23 seconds)5484client: must succeed: nix-store --export /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only > /tmp/shared/inp.closure5485client: (finished: must succeed: nix-store --export /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only > /tmp/shared/inp.closure, in 0.09 seconds)5486worker1: must succeed: nix-store --import < /tmp/shared/inp.closure5487worker1: (finished: must succeed: nix-store --import < /tmp/shared/inp.closure, in 0.12 seconds)5488worker2: must fail: test -e /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only5489worker2: (finished: must fail: test -e /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only, in 0.02 seconds)5490client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix job --argstr tag w1only --argstr salt a >&25491client # [ 78.512746] nix-daemon[1107]: accepted connection from pid 1190, user root (trusted)5492client # this derivation will be built:5493client # /nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv5494worker2 # [ 78.618525] nix-grpc-daemon[783]: ts=2026-09-22T14:51:38Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:558365495client # building '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' on 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux'...5496worker1 # [ 78.602687] nix-grpc-daemon[787]: ts=2026-09-22T14:51:38Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:433645497worker1 # [ 78.709932] nix-daemon[869]: accepted connection from pid 1318, user nix-grpc-daemon (trusted)5498worker1 # [ 78.722841] nix-daemon[869]: reaped child process 1321, status = succeeded5499lb # [ 78.761987] niks3-server[957]: 2026/09/22 14:51:38 INFO Received uploads request method=POST path=/api/pending_closures5500worker1 # [ 78.736829] nix-grpc-daemon[884]: time=2026-09-22T14:51:38.853Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5501worker1 # [ 78.738675] nix-grpc-daemon[884]: time=2026-09-22T14:51:38.855Z level=INFO msg="Uploading gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only (120B)"5502lb # [ 78.791197] niks3-server[957]: 2026/09/22 14:51:38 INFO Registered completed upload object_key=nar/08ghl87f009pwqz85nbqv9hm0ln6gai6ggzilfmw3qnkaalfi6ss.nar.zst5503lb # [ 78.802730] niks3-server[957]: 2026/09/22 14:51:38 INFO Received sign narinfos request method=POST path=/api/pending_closures/27/sign5504lb # [ 78.807835] niks3-server[957]: 2026/09/22 14:51:38 INFO Signed narinfos id=27 count=15505worker1 # [ 78.778500] nix-grpc-daemon[884]: time=2026-09-22T14:51:38.895Z level=INFO msg="Uploading 1 narinfos"5506lb # [ 78.815578] niks3-server[957]: 2026/09/22 14:51:38 INFO Registered completed upload object_key=gycxa7jajm34fqsmi3xn0zjr3pvaya0x.ls5507lb # [ 78.828095] niks3-server[957]: 2026/09/22 14:51:38 INFO Received complete upload request method=POST path=/api/pending_closures/27/complete5508lb # [ 78.834174] niks3-server[957]: 2026/09/22 14:51:38 INFO Completed upload id=275509worker1 # [ 78.800078] nix-grpc-daemon[884]: time=2026-09-22T14:51:38.917Z level=INFO msg="Upload complete. (192ms)"5510lb # [ 78.839799] niks3-server[957]: 2026/09/22 14:51:38 INFO Registered completed upload object_key=gycxa7jajm34fqsmi3xn0zjr3pvaya0x.narinfo5511worker1 # [ 78.821648] nix-grpc-daemon[787]: ts=2026-09-22T14:51:38Z level=debug event=want client=14 drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv cached=05512worker1 # [ 78.825950] nix-grpc-daemon[787]: ts=2026-09-22T14:51:38Z level=debug event=assigned drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv worker=192.168.1.4:50051 clients=15513worker2 # [ 78.895532] nix-grpc-daemon[783]: ts=2026-09-22T14:51:38Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558365514worker2 # [ 78.907392] nix-grpc-daemon[783]: ts=2026-09-22T14:51:38Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:558365515client # copying 2 paths to 'lb:50051'...5516client # copying path '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv' to 'lb:50051'...5517worker2 # [ 78.922673] nix-grpc-daemon[783]: ts=2026-09-22T14:51:38Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:558365518client # copying path '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' to 'lb:50051'...5519worker2 # [ 78.931661] nix-daemon[850]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5520worker2 # [ 79.036662] nix-daemon[850]: accepted connection from pid 1115, user nix-grpc-daemon (trusted)5521worker2 # [ 79.051590] nix-daemon[850]: reaped child process 1118, status = succeeded5522worker2 # [ 79.131179] nix-daemon[850]: accepted connection from pid 1120, user nix-grpc-daemon (trusted)5523worker2 # [ 79.144297] nix-daemon[850]: reaped child process 1123, status = succeeded5524lb # [ 79.166614] niks3-server[957]: 2026/09/22 14:51:38 INFO Received uploads request method=POST path=/api/pending_closures5525lb # [ 79.177090] niks3-server[957]: 2026/09/22 14:51:38 INFO Received uploads request method=POST path=/api/pending_closures5526worker2 # [ 79.169213] nix-grpc-daemon[878]: time=2026-09-22T14:51:39.075Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"5527worker2 # [ 79.170747] nix-grpc-daemon[878]: time=2026-09-22T14:51:39.077Z level=INFO msg="Uploading s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv (416B)"5528worker2 # [ 79.173348] nix-grpc-daemon[878]: time=2026-09-22T14:51:39.079Z level=INFO msg="Uploading k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv (560B)"5529lb # [ 79.214113] niks3-server[957]: 2026/09/22 14:51:38 INFO Registered completed upload object_key=nar/0s7awvvc5phwxzwwzhl47kzq38pdgwj6jp3kwxf5w2xddnxnkl6h.nar.zst5530lb # [ 79.218642] niks3-server[957]: 2026/09/22 14:51:38 INFO Registered completed upload object_key=nar/136ms4k64sfvy4m17vvd9w9yncq4zydv4cjbhchdcg4j7ccbx3b9.nar.zst5531lb # [ 79.234994] niks3-server[957]: 2026/09/22 14:51:38 INFO Received sign narinfos request method=POST path=/api/pending_closures/29/sign5532lb # [ 79.239930] niks3-server[957]: 2026/09/22 14:51:38 INFO Signed narinfos id=29 count=25533lb # [ 79.242958] niks3-server[957]: 2026/09/22 14:51:38 INFO Received sign narinfos request method=POST path=/api/pending_closures/28/sign5534lb # [ 79.246671] niks3-server[957]: 2026/09/22 14:51:38 INFO Signed narinfos id=28 count=15535worker2 # [ 79.231588] nix-grpc-daemon[878]: time=2026-09-22T14:51:39.137Z level=INFO msg="Uploading 3 narinfos"5536lb # [ 79.251178] niks3-server[957]: 2026/09/22 14:51:38 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.ls5537lb # [ 79.253268] niks3-server[957]: 2026/09/22 14:51:38 INFO Registered completed upload object_key=k9va03aiirjmgih65p90fpa9mr396d2b.ls5538lb # [ 79.261620] niks3-server[957]: 2026/09/22 14:51:38 INFO Registered completed upload object_key=k9va03aiirjmgih65p90fpa9mr396d2b.narinfo5539lb # [ 79.268223] niks3-server[957]: 2026/09/22 14:51:38 INFO Received complete upload request method=POST path=/api/pending_closures/28/complete5540lb # [ 79.271799] niks3-server[957]: 2026/09/22 14:51:38 INFO Completed upload id=285541lb # [ 79.273730] niks3-server[957]: 2026/09/22 14:51:38 INFO Received complete upload request method=POST path=/api/pending_closures/29/complete5542lb # [ 79.276570] niks3-server[957]: 2026/09/22 14:51:38 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.narinfo5543lb # [ 79.278700] niks3-server[957]: 2026/09/22 14:51:38 INFO Completed upload id=295544worker2 # [ 79.263392] nix-grpc-daemon[878]: time=2026-09-22T14:51:39.169Z level=INFO msg="Upload complete. (316ms)"5545lb # [ 79.280686] niks3-server[957]: 2026/09/22 14:51:38 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.narinfo5546worker2 # [ 79.268712] nix-grpc-daemon[783]: ts=2026-09-22T14:51:39Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55836 duration_s=0 paths=2 nar_bytes_in=9765547worker2 # [ 79.274735] nix-grpc-daemon[783]: ts=2026-09-22T14:51:39Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558365548worker2 # [ 79.284652] nix-daemon[850]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5549worker2 # [ 79.287338] nix-daemon[850]: reaped child process 1113, status = succeeded5550worker2 # [ 79.294251] nix-grpc-daemon[783]: copying path '/nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only' from 'http://lb:5751'...5551client # building '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' on lb:50051...5552worker2 # [ 79.300954] nix-daemon[850]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5553client # uses-input-w1onlya> worker2: building k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv5554client # uses-input-w1onlya> disabling binary cache 'https://cache.nixos.org' for 60 seconds5555worker2 # [ 80.920450] nix-daemon[850]: reaped child process 1128, status = succeeded5556worker2 # [ 80.989395] nix-daemon[850]: accepted connection from pid 1144, user nix-grpc-daemon (trusted)5557worker2 # [ 81.002703] nix-daemon[850]: reaped child process 1147, status = succeeded5558lb # [ 81.023918] niks3-server[957]: 2026/09/22 14:51:40 INFO Received uploads request method=POST path=/api/pending_closures5559worker2 # [ 81.017422] nix-grpc-daemon[878]: time=2026-09-22T14:51:40.923Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5560lb # [ 81.055329] niks3-server[957]: 2026/09/22 14:51:40 INFO Registered completed upload object_key=7fihpafywah8yif4mhbmz6i2i6adiqwj.ls5561lb # [ 81.063105] niks3-server[957]: 2026/09/22 14:51:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/30/sign5562lb # [ 81.069239] niks3-server[957]: 2026/09/22 14:51:40 INFO Signed narinfos id=30 count=15563worker2 # [ 81.056506] nix-grpc-daemon[878]: time=2026-09-22T14:51:40.963Z level=INFO msg="Uploading 1 narinfos"5564lb # [ 81.075125] niks3-server[957]: 2026/09/22 14:51:40 INFO Registered completed upload object_key=log/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv5565lb # [ 81.089589] niks3-server[957]: 2026/09/22 14:51:40 INFO Received complete upload request method=POST path=/api/pending_closures/30/complete5566lb # [ 81.096123] niks3-server[957]: 2026/09/22 14:51:40 INFO Completed upload id=305567worker2 # [ 81.080642] nix-grpc-daemon[878]: time=2026-09-22T14:51:40.987Z level=INFO msg="Upload complete. (169ms)"5568lb # [ 81.098770] niks3-server[957]: 2026/09/22 14:51:40 INFO Registered completed upload object_key=7fihpafywah8yif4mhbmz6i2i6adiqwj.narinfo5569worker2 # [ 81.085429] nix-grpc-daemon[783]: ts=2026-09-22T14:51:40Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55836 duration_s=1 drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv assign_id=15 outputs=15570client # copying 1 paths...5571worker2 # [ 81.094789] nix-grpc-daemon[783]: ts=2026-09-22T14:51:41Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:558365572client # copying path '/nix/store/7fihpafywah8yif4mhbmz6i2i6adiqwj-uses-input-w1onlya' from 'grpc://lb:50051'...5573worker2 # [ 81.102106] nix-daemon[850]: reaped child process 1125, status = succeeded5574worker2 # [ 81.112475] nix-grpc-daemon[783]: ts=2026-09-22T14:51:41Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:558365575worker2 # [ 81.114611] nix-grpc-daemon[783]: ts=2026-09-22T14:51:41Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:55836 duration_s=0 paths=1 nar_bytes_out=1205576client # [ 81.235384] nix-daemon[1107]: reaped child process 1193, status = succeeded5577client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix job --argstr tag w1only --argstr salt a >&2, in 2.84 seconds)5578client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix job --argstr tag w1only --argstr salt b >&25579client # [ 81.338337] nix-daemon[1107]: accepted connection from pid 1212, user root (trusted)5580client # this derivation will be built:5581client # /nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5582client # building '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' on 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux'...5583worker1 # [ 81.422476] nix-grpc-daemon[787]: ts=2026-09-22T14:51:41Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:433645584worker1 # [ 81.428214] nix-grpc-daemon[787]: ts=2026-09-22T14:51:41Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:433645585worker1 # [ 81.433939] nix-grpc-daemon[884]: time=2026-09-22T14:51:41.549Z level=INFO msg="All 1 paths already cached"5586worker1 # [ 81.443409] nix-grpc-daemon[787]: ts=2026-09-22T14:51:41Z level=debug event=want client=15 drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv cached=05587worker1 # [ 81.448519] nix-grpc-daemon[787]: ts=2026-09-22T14:51:41Z level=debug event=assigned drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv worker=192.168.1.3:50051 clients=15588worker1 # [ 81.453411] nix-grpc-daemon[787]: ts=2026-09-22T14:51:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:433645589worker1 # [ 81.461064] nix-grpc-daemon[787]: ts=2026-09-22T14:51:41Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:433645590client # copying 1 paths to 'lb:50051'...5591client # copying path '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' to 'lb:50051'...5592worker1 # [ 81.479551] nix-grpc-daemon[787]: ts=2026-09-22T14:51:41Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:433645593worker1 # [ 81.490505] nix-daemon[869]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5594worker1 # [ 81.504842] nix-grpc-daemon[787]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 97 ms (attempt 1/5)5595worker1 # [ 81.607825] nix-grpc-daemon[787]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 127 ms (attempt 2/5)5596worker1 # [ 81.736283] nix-grpc-daemon[787]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 246 ms (attempt 3/5)5597worker1 # [ 81.983456] nix-grpc-daemon[787]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 732 ms (attempt 4/5)5598worker1 # [ 82.718788] nix-grpc-daemon[787]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5599worker1 # [ 82.722811] nix-grpc-daemon[787]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5600worker1 # [ 82.727966] nix-grpc-daemon[787]: copying path '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv' from 'http://lb:5751'...5601worker1 # [ 82.844563] nix-daemon[869]: accepted connection from pid 1334, user nix-grpc-daemon (trusted)5602worker1 # [ 82.859144] nix-daemon[869]: reaped child process 1337, status = succeeded5603worker1 # [ 82.939075] nix-daemon[869]: accepted connection from pid 1339, user nix-grpc-daemon (trusted)5604worker1 # [ 82.955151] nix-daemon[869]: reaped child process 1342, status = succeeded5605lb # [ 82.995868] niks3-server[957]: 2026/09/22 14:51:42 INFO Received uploads request method=POST path=/api/pending_closures5606worker1 # [ 82.970175] nix-grpc-daemon[884]: time=2026-09-22T14:51:43.086Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5607worker1 # [ 82.971649] nix-grpc-daemon[884]: time=2026-09-22T14:51:43.086Z level=INFO msg="Uploading 2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv (560B)"5608lb # [ 83.021978] niks3-server[957]: 2026/09/22 14:51:42 INFO Registered completed upload object_key=nar/0bp8pfsa12j91bab6f50rs7hh6sphjrig1zqlr7v80mn285qxax2.nar.zst5609lb # [ 83.032506] niks3-server[957]: 2026/09/22 14:51:42 INFO Received sign narinfos request method=POST path=/api/pending_closures/31/sign5610lb # [ 83.036933] niks3-server[957]: 2026/09/22 14:51:42 INFO Signed narinfos id=31 count=15611worker1 # [ 83.005604] nix-grpc-daemon[884]: time=2026-09-22T14:51:43.122Z level=INFO msg="Uploading 1 narinfos"5612lb # [ 83.043192] niks3-server[957]: 2026/09/22 14:51:42 INFO Registered completed upload object_key=2ign0w9nlrywl4qfkq5i314zinl1cxqs.ls5613lb # [ 83.054622] niks3-server[957]: 2026/09/22 14:51:42 INFO Received complete upload request method=POST path=/api/pending_closures/31/complete5614lb # [ 83.061379] niks3-server[957]: 2026/09/22 14:51:42 INFO Completed upload id=315615worker1 # [ 83.027340] nix-grpc-daemon[884]: time=2026-09-22T14:51:43.144Z level=INFO msg="Upload complete. (281ms)"5616worker1 # [ 83.029873] nix-grpc-daemon[787]: ts=2026-09-22T14:51:43Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43364 duration_s=1 paths=1 nar_bytes_in=5605617lb # [ 83.068686] niks3-server[957]: 2026/09/22 14:51:42 INFO Registered completed upload object_key=2ign0w9nlrywl4qfkq5i314zinl1cxqs.narinfo5618worker1 # [ 83.034319] nix-grpc-daemon[787]: ts=2026-09-22T14:51:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:433645619worker1 # [ 83.041150] nix-daemon[869]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5620worker1 # [ 83.048577] nix-daemon[869]: reaped child process 1327, status = succeeded5621worker1 # [ 83.052314] nix-daemon[869]: accepted connection from pid 787, user nix-grpc-daemon (trusted)5622client # building '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' on lb:50051...5623client # uses-input-w1onlyb> node-a: building 2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5624client # uses-input-w1onlyb> disabling binary cache 'https://cache.nixos.org' for 60 seconds5625worker1 # [ 84.092228] nix-daemon[869]: reaped child process 1346, status = succeeded5626worker1 # [ 84.173334] nix-daemon[869]: accepted connection from pid 1362, user nix-grpc-daemon (trusted)5627worker1 # [ 84.186383] nix-daemon[869]: reaped child process 1365, status = succeeded5628lb # [ 84.228962] niks3-server[957]: 2026/09/22 14:51:43 INFO Received uploads request method=POST path=/api/pending_closures5629worker1 # [ 84.202667] nix-grpc-daemon[884]: time=2026-09-22T14:51:44.319Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5630lb # [ 84.262393] niks3-server[957]: 2026/09/22 14:51:43 INFO Received sign narinfos request method=POST path=/api/pending_closures/32/sign5631worker1 # [ 84.233930] nix-grpc-daemon[884]: time=2026-09-22T14:51:44.351Z level=INFO msg="Uploading 1 narinfos"5632lb # [ 84.268117] niks3-server[957]: 2026/09/22 14:51:43 INFO Signed narinfos id=32 count=15633lb # [ 84.276117] niks3-server[957]: 2026/09/22 14:51:43 INFO Registered completed upload object_key=log/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5634lb # [ 84.286956] niks3-server[957]: 2026/09/22 14:51:43 INFO Registered completed upload object_key=rs93p5yqksm4zlblqlsfa8kxl8lkhrbg.ls5635lb # [ 84.300100] niks3-server[957]: 2026/09/22 14:51:43 INFO Received complete upload request method=POST path=/api/pending_closures/32/complete5636worker1 # [ 84.271902] nix-grpc-daemon[884]: time=2026-09-22T14:51:44.389Z level=INFO msg="Upload complete. (212ms)"5637lb # [ 84.306570] niks3-server[957]: 2026/09/22 14:51:43 INFO Completed upload id=325638lb # [ 84.309782] niks3-server[957]: 2026/09/22 14:51:43 INFO Registered completed upload object_key=rs93p5yqksm4zlblqlsfa8kxl8lkhrbg.narinfo5639worker1 # [ 84.276574] nix-grpc-daemon[787]: ts=2026-09-22T14:51:44Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43364 duration_s=1 drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv assign_id=16 outputs=15640client # copying 1 paths...5641worker1 # [ 84.287434] nix-grpc-daemon[787]: ts=2026-09-22T14:51:44Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:433645642worker1 # [ 84.294844] nix-daemon[869]: reaped child process 1344, status = succeeded5643client # copying path '/nix/store/rs93p5yqksm4zlblqlsfa8kxl8lkhrbg-uses-input-w1onlyb' from 'grpc://lb:50051'...5644worker1 # [ 84.308797] nix-grpc-daemon[787]: ts=2026-09-22T14:51:44Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:433645645worker1 # [ 84.312814] nix-grpc-daemon[787]: ts=2026-09-22T14:51:44Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:43364 duration_s=0 paths=1 nar_bytes_out=1205646client # [ 84.451656] nix-daemon[1107]: reaped child process 1215, status = succeeded5647client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix job --argstr tag w1only --argstr salt b >&2, in 3.22 seconds)5648client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix job --argstr tag w1only --argstr salt c >&25649client # [ 84.556167] nix-daemon[1107]: accepted connection from pid 1234, user root (trusted)5650client # this derivation will be built:5651client # /nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5652client # building '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' on 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux'...5653worker2 # [ 84.667403] nix-grpc-daemon[783]: ts=2026-09-22T14:51:44Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:558365654worker1 # [ 84.651233] nix-grpc-daemon[787]: ts=2026-09-22T14:51:44Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:433645655worker1 # [ 84.660738] nix-grpc-daemon[884]: time=2026-09-22T14:51:44.777Z level=INFO msg="All 1 paths already cached"5656worker1 # [ 84.675182] nix-grpc-daemon[787]: ts=2026-09-22T14:51:44Z level=debug event=want client=16 drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv cached=05657worker1 # [ 84.679441] nix-grpc-daemon[787]: ts=2026-09-22T14:51:44Z level=debug event=assigned drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv worker=192.168.1.4:50051 clients=15658worker2 # [ 84.746748] nix-grpc-daemon[783]: ts=2026-09-22T14:51:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558365659worker2 # [ 84.758990] nix-grpc-daemon[783]: ts=2026-09-22T14:51:44Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:558365660worker2 # [ 84.772675] nix-grpc-daemon[878]: time=2026-09-22T14:51:44.678Z level=INFO msg="All 1 paths already cached"5661client # copying 1 paths to 'lb:50051'...5662client # copying path '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' to 'lb:50051'...5663worker2 # [ 84.781808] nix-grpc-daemon[783]: ts=2026-09-22T14:51:44Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:558365664worker2 # [ 84.793247] nix-daemon[850]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5665worker2 # [ 84.897588] nix-daemon[850]: accepted connection from pid 1151, user nix-grpc-daemon (trusted)5666worker2 # [ 84.912054] nix-daemon[850]: reaped child process 1154, status = succeeded5667worker2 # [ 84.993174] nix-daemon[850]: accepted connection from pid 1156, user nix-grpc-daemon (trusted)5668worker2 # [ 85.006368] nix-daemon[850]: reaped child process 1159, status = succeeded5669lb # [ 85.028416] niks3-server[957]: 2026/09/22 14:51:44 INFO Received uploads request method=POST path=/api/pending_closures5670worker2 # [ 85.022277] nix-grpc-daemon[878]: time=2026-09-22T14:51:44.928Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5671worker2 # [ 85.023637] nix-grpc-daemon[878]: time=2026-09-22T14:51:44.928Z level=INFO msg="Uploading ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv (560B)"5672lb # [ 85.054837] niks3-server[957]: 2026/09/22 14:51:44 INFO Registered completed upload object_key=nar/1iq9qm6vw4b3p68mlkiw8nf0phmz21m8g79izsy14sydxmd4gmqa.nar.zst5673lb # [ 85.059771] niks3-server[957]: 2026/09/22 14:51:44 INFO Received sign narinfos request method=POST path=/api/pending_closures/33/sign5674lb # [ 85.061899] niks3-server[957]: 2026/09/22 14:51:44 INFO Signed narinfos id=33 count=15675worker2 # [ 85.046280] nix-grpc-daemon[878]: time=2026-09-22T14:51:44.952Z level=INFO msg="Uploading 1 narinfos"5676lb # [ 85.068794] niks3-server[957]: 2026/09/22 14:51:44 INFO Registered completed upload object_key=ych53lrh5qfmdxlnx672apx2v80x3aan.ls5677lb # [ 85.070329] niks3-server[957]: 2026/09/22 14:51:44 INFO Received complete upload request method=POST path=/api/pending_closures/33/complete5678lb # [ 85.073753] niks3-server[957]: 2026/09/22 14:51:44 INFO Completed upload id=335679worker2 # [ 85.059693] nix-grpc-daemon[878]: time=2026-09-22T14:51:44.966Z level=INFO msg="Upload complete. (254ms)"5680lb # [ 85.076409] niks3-server[957]: 2026/09/22 14:51:44 INFO Registered completed upload object_key=ych53lrh5qfmdxlnx672apx2v80x3aan.narinfo5681worker2 # [ 85.061995] nix-grpc-daemon[783]: ts=2026-09-22T14:51:44Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55836 duration_s=0 paths=1 nar_bytes_in=5605682worker2 # [ 85.064941] nix-grpc-daemon[783]: ts=2026-09-22T14:51:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558365683worker2 # [ 85.072107] nix-daemon[850]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5684worker2 # [ 85.074456] nix-daemon[850]: reaped child process 1149, status = succeeded5685worker2 # [ 85.079552] nix-daemon[850]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5686client # building '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' on lb:50051...5687client # uses-input-w1onlyc> worker2: building ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5688client # uses-input-w1onlyc> disabling binary cache 'https://cache.nixos.org' for 60 seconds5689lb # [ 85.800557] envoy[905]: conn peer=[2001:db8:1::1]:58066 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=435 tx=378 ms=565690lb # [ 85.804188] envoy[905]: conn peer=[2001:db8:1::1]:58052 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3876 tx=2165 ms=13045691lb # [ 85.807886] envoy[905]: conn peer=[2001:db8:1::1]:58084 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=437 tx=410 ms=555692lb # [ 85.811350] envoy[905]: conn peer=[2001:db8:1::1]:58076 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4311 tx=2368 ms=16255693lb # [ 85.814859] envoy[905]: conn peer=[2001:db8:1::1]:58098 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=421 tx=328 ms=525694lb # [ 85.818274] envoy[905]: conn peer=[2001:db8:1::1]:58092 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3881 tx=2067 ms=25485695lb # [ 85.821817] envoy[905]: conn peer=[2001:db8:1::1]:58104 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3745 tx=2124 ms=29415696lb # [ 85.825294] envoy[905]: conn peer=[2001:db8:1::1]:38610 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=447 tx=328 ms=575697worker2 # [ 85.927500] nix-daemon[850]: reaped child process 1163, status = succeeded5698worker2 # [ 86.007095] nix-daemon[850]: accepted connection from pid 1179, user nix-grpc-daemon (trusted)5699worker2 # [ 86.018569] nix-daemon[850]: reaped child process 1182, status = succeeded5700lb # [ 86.042440] niks3-server[957]: 2026/09/22 14:51:45 INFO Received uploads request method=POST path=/api/pending_closures5701worker2 # [ 86.034794] nix-grpc-daemon[878]: time=2026-09-22T14:51:45.941Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5702lb # [ 86.070713] niks3-server[957]: 2026/09/22 14:51:45 INFO Received sign narinfos request method=POST path=/api/pending_closures/34/sign5703lb # [ 86.076460] niks3-server[957]: 2026/09/22 14:51:45 INFO Signed narinfos id=34 count=15704worker2 # [ 86.064094] nix-grpc-daemon[878]: time=2026-09-22T14:51:45.970Z level=INFO msg="Uploading 1 narinfos"5705lb # [ 86.084103] niks3-server[957]: 2026/09/22 14:51:45 INFO Registered completed upload object_key=af2xkgzqdrcsfmk8dgmypxmxww9cx3rk.ls5706lb # [ 86.095577] niks3-server[957]: 2026/09/22 14:51:45 INFO Registered completed upload object_key=log/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5707lb # [ 86.103632] niks3-server[957]: 2026/09/22 14:51:45 INFO Received complete upload request method=POST path=/api/pending_closures/34/complete5708lb # [ 86.107827] niks3-server[957]: 2026/09/22 14:51:45 INFO Completed upload id=345709worker2 # [ 86.094561] nix-grpc-daemon[878]: time=2026-09-22T14:51:46.001Z level=INFO msg="Upload complete. (208ms)"5710lb # [ 86.113471] niks3-server[957]: 2026/09/22 14:51:45 INFO Registered completed upload object_key=af2xkgzqdrcsfmk8dgmypxmxww9cx3rk.narinfo5711worker2 # [ 86.099396] nix-grpc-daemon[783]: ts=2026-09-22T14:51:46Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55836 duration_s=1 drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv assign_id=17 outputs=15712client # copying 1 paths...5713worker2 # [ 86.109863] nix-grpc-daemon[783]: ts=2026-09-22T14:51:46Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:558365714client # copying path '/nix/store/af2xkgzqdrcsfmk8dgmypxmxww9cx3rk-uses-input-w1onlyc' from 'grpc://lb:50051'...5715worker2 # [ 86.116651] nix-daemon[850]: reaped child process 1161, status = succeeded5716worker2 # [ 86.127499] nix-grpc-daemon[783]: ts=2026-09-22T14:51:46Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:558365717worker2 # [ 86.131492] nix-grpc-daemon[783]: ts=2026-09-22T14:51:46Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:55836 duration_s=0 paths=1 nar_bytes_out=1205718client # [ 86.251618] nix-daemon[1107]: reaped child process 1237, status = succeeded5719client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix job --argstr tag w1only --argstr salt c >&2, in 1.80 seconds)5720(finished: subtest: an input only one worker has still reaches the builder, in 8.33 seconds)5721subtest: upload whose reference the worker lacks is completed from the cache5722client: must succeed: nix-build --no-out-link /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix -A input --argstr tag viacache5723client # this derivation will be built:5724client # /nix/store/zpv7g9hl0xcjhjz0g0hd92sv1sf580zh-local-input-viacache.drv5725client # building '/nix/store/zpv7g9hl0xcjhjz0g0hd92sv1sf580zh-local-input-viacache.drv'...5726client: (finished: must succeed: nix-build --no-out-link /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix -A input --argstr tag viacache, in 0.25 seconds)5727client: must succeed: nix-build --no-out-link /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix -A referrer --argstr tag viacache5728client # this derivation will be built:5729client # /nix/store/8higgmxp1c6ag4yj6sk9g6z66n43x70l-refers-to-input-viacache.drv5730client # building '/nix/store/8higgmxp1c6ag4yj6sk9g6z66n43x70l-refers-to-input-viacache.drv'...5731client: (finished: must succeed: nix-build --no-out-link /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix -A referrer --argstr tag viacache, in 0.25 seconds)5732client: must succeed: nix-store --export /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache > /tmp/shared/ref.closure5733client: (finished: must succeed: nix-store --export /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache > /tmp/shared/ref.closure, in 0.10 seconds)5734worker1: must succeed: nix-store --import < /tmp/shared/ref.closure5735worker1: (finished: must succeed: nix-store --import < /tmp/shared/ref.closure, in 0.13 seconds)5736worker2 # [ 87.030739] systemd[1]: Stopping Nix worker-protocol over gRPC...5737worker2 # [ 87.857966] nix-grpc-daemon[783]: ts=2026-09-22T14:51:47Z level=info event=signal_exit5738worker1 # [ 89.563879] nix-grpc-daemon[787]: ts=2026-09-22T14:51:49Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:433565739worker1 # [ 89.567690] nix-grpc-daemon[787]: ts=2026-09-22T14:51:49Z level=info event=worker_gone addr=192.168.1.4:500515740worker2 # [ 89.587184] nix-grpc-daemon[783]: ts=2026-09-22T14:51:49Z level=info event=scheduler_restarting5741worker2 # [ 89.598593] nix-grpc-daemon[783]: ts=2026-09-22T14:51:49Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5742worker2 # [ 89.616651] nix-daemon[850]: reaped child process 852, status = succeeded5743worker2 # [ 89.983143] nix-grpc-daemon[783]: E0000 00:00:1790088709.889344 828 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 18, started true, shutdown true5744worker2 # [ 89.985193] nix-grpc-daemon[783]: E0000 00:00:1790088709.889546 828 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5745worker2 # [ 91.109554] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5746worker2 # [ 91.114878] systemd[1]: Stopped Nix worker-protocol over gRPC.5747worker2 # [ 91.117629] systemd[1]: nix-grpc-daemon.service: Consumed 2.231s CPU time over 1min 14.410s wall clock time, 60.2M memory peak, 77.1K incoming IP traffic, 179.6K outgoing IP traffic.5748worker2 # [ 91.129655] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5749worker2 # [ 91.134258] systemd[1]: Closed Nix worker-protocol over gRPC.5750worker2 # [ 91.138203] systemd[1]: nix-grpc-daemon.socket: Consumed 319us CPU time over 1min 22.377s wall clock time, 256K memory peak, 86.8K incoming IP traffic, 75.7K outgoing IP traffic.5751lb: waiting until exactly ['worker2'] endpoints of x86_64-linux are unhealthy5752lb: must succeed: curl -sf localhost:9901/clusters5753lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)5754lb: (finished: waiting until exactly ['worker2'] endpoints of x86_64-linux are unhealthy, in 0.02 seconds)5755client: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache >&25756client # /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache5757worker1 # [ 91.297367] nix-grpc-daemon[787]: ts=2026-09-22T14:51:51Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:433645758worker1 # [ 91.301389] nix-grpc-daemon[787]: ts=2026-09-22T14:51:51Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:433645759client: (finished: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache >&2, in 0.16 seconds)5760worker2: must fail: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache5761worker2: (finished: must fail: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache, in 0.02 seconds)5762worker2 # [ 91.398533] systemd[1]: Listening on Nix worker-protocol over gRPC.5763worker1 # [ 91.435120] systemd[1]: Stopping Nix worker-protocol over gRPC...5764worker1 # [ 91.957764] nix-grpc-daemon[787]: ts=2026-09-22T14:51:52Z level=info event=signal_exit5765worker2 # [ 92.101845] systemd[1]: Starting scheduler-token.service...5766worker2 # [ 92.139748] systemd[1]: scheduler-token.service: Deactivated successfully.5767worker2 # [ 92.141504] systemd[1]: Finished scheduler-token.service.5768worker2 # [ 92.142557] systemd[1]: scheduler-token.service: Consumed 20ms CPU time over 39ms wall clock time, 2.3M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5769worker2 # [ 92.147419] systemd[1]: Starting Nix worker-protocol over gRPC...5770worker2 # [ 92.228159] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:52Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5771worker2 # [ 92.230154] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:52Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5772worker2 # [ 92.233713] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:52Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15773worker2 # [ 92.237828] nix-grpc-daemon[1206]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5774worker2 # [ 92.239155] nix-grpc-daemon[1206]: E0000 00:00:1790088712.144021 1206 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5775worker2 # [ 92.244371] nix-grpc-daemon[1206]: E0000 00:00:1790088712.149348 1206 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5776worker2 # [ 92.248132] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:52Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5777worker2 # [ 92.249653] systemd[1]: Started Nix worker-protocol over gRPC.5778worker2 # [ 92.256072] nix-grpc-daemon[1206]: E0000 00:00:1790088712.161882 1248 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5779worker2 # [ 92.269936] nix-grpc-daemon[1206]: E0000 00:00:1790088712.176427 1248 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5780worker2 # [ 92.275054] nix-grpc-daemon[1206]: E0000 00:00:1790088712.178713 1248 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5781worker2 # [ 92.284021] nix-grpc-daemon[1206]: E0000 00:00:1790088712.190531 1248 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5782worker2 # [ 92.293122] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:52Z level=info event=scheduler_connected addr=lb:500515783worker2 # [ 92.294576] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5784worker2 # [ 92.496210] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:52Z level=info event=scheduler_connected addr=lb:500515785worker2 # [ 92.500168] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5786lb # [ 92.598844] niks3-server[957]: 2026/09/22 14:51:51 INFO lead: released remote=192.168.1.3:487685787worker2 # [ 92.701332] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:52Z level=info event=scheduler_connected addr=lb:500515788worker2 # [ 92.705175] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5789worker2 # [ 92.906254] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:52Z level=info event=scheduler_connected addr=lb:500515790worker2 # [ 92.910234] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5791worker2 # [ 93.110725] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:53Z level=info event=scheduler_connected addr=lb:500515792worker2 # [ 93.112630] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5793worker2 # [ 93.315287] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:53Z level=info event=scheduler_connected addr=lb:500515794worker2 # [ 93.318530] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5795worker2 # [ 93.516073] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:53Z level=info event=scheduler_connected addr=lb:500515796worker2 # [ 93.520143] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5797worker1 # [ 93.566214] nix-grpc-daemon[787]: ts=2026-09-22T14:51:53Z level=info event=scheduler_restarting5798worker1 # [ 93.567790] nix-grpc-daemon[787]: ts=2026-09-22T14:51:53Z level=info event=scheduler_restarting addr=lb:500515799worker1 # [ 93.569339] nix-grpc-daemon[787]: ts=2026-09-22T14:51:53Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:433565800worker1 # [ 93.570732] nix-grpc-daemon[787]: ts=2026-09-22T14:51:53Z level=info event=worker_gone addr=192.168.1.3:500515801worker1 # [ 93.590192] nix-daemon[869]: reaped child process 871, status = succeeded5802worker2 # [ 93.721058] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:53Z level=info event=scheduler_connected addr=lb:500515803worker2 # [ 93.724910] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5804worker1 # [ 93.772876] nix-grpc-daemon[787]: ts=2026-09-22T14:51:53Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5805worker2 # [ 93.926127] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:53Z level=info event=scheduler_connected addr=lb:500515806worker2 # [ 93.930067] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5807worker1 # [ 94.098716] nix-grpc-daemon[787]: E0000 00:00:1790088714.215704 832 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 19, started true, shutdown true5808worker1 # [ 94.103603] nix-grpc-daemon[787]: E0000 00:00:1790088714.215881 832 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5809worker2 # [ 94.131264] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:54Z level=info event=scheduler_connected addr=lb:500515810worker2 # [ 94.134969] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5811worker2 # [ 94.333454] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:54Z level=info event=scheduler_connected addr=lb:500515812worker2 # [ 94.334992] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5813worker2 # [ 94.535065] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:54Z level=info event=scheduler_connected addr=lb:500515814worker2 # [ 94.538794] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5815worker2 # [ 94.740069] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:54Z level=info event=scheduler_connected addr=lb:500515816worker2 # [ 94.744207] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5817worker2 # [ 94.944745] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:54Z level=info event=scheduler_connected addr=lb:500515818worker2 # [ 94.946707] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5819worker1 # [ 95.046592] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5820worker1 # [ 95.051395] systemd[1]: Stopped Nix worker-protocol over gRPC.5821worker1 # [ 95.054938] systemd[1]: nix-grpc-daemon.service: Consumed 3.168s CPU time over 1min 19.287s wall clock time, 70.9M memory peak, 112.4K incoming IP traffic, 244.4K outgoing IP traffic.5822worker1 # [ 95.067202] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5823worker1 # [ 95.071519] systemd[1]: Closed Nix worker-protocol over gRPC.5824worker1 # [ 95.075720] systemd[1]: nix-grpc-daemon.socket: Consumed 298us CPU time over 1min 26.396s wall clock time, 256K memory peak, 122.4K incoming IP traffic, 105K outgoing IP traffic.5825lb: waiting until exactly ['worker1'] endpoints of x86_64-linux are unhealthy5826lb: must succeed: curl -sf localhost:9901/clusters5827lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)5828lb: (finished: waiting until exactly ['worker1'] endpoints of x86_64-linux are unhealthy, in 0.03 seconds)5829client: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache >&25830worker2 # [ 95.148153] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:55Z level=info event=scheduler_connected addr=lb:500515831worker2 # [ 95.152164] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5832worker2 # [ 95.271258] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:55Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:358445833worker2 # [ 95.283346] nix-daemon[850]: accepted connection from pid 1206, user nix-grpc-daemon (trusted)5834worker2 # [ 95.299535] nix-grpc-daemon[1206]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 24 ms (attempt 1/5)5835worker2 # [ 95.330907] nix-grpc-daemon[1206]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 96 ms (attempt 2/5)5836worker2 # [ 95.353674] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:55Z level=info event=scheduler_connected addr=lb:500515837worker2 # [ 95.357428] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5838worker2 # [ 95.428999] nix-grpc-daemon[1206]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 369 ms (attempt 3/5)5839worker2 # [ 95.558805] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:55Z level=info event=scheduler_connected addr=lb:500515840worker2 # [ 95.562708] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5841worker2 # [ 95.763945] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:55Z level=info event=scheduler_connected addr=lb:500515842worker2 # [ 95.767850] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5843lb # [ 95.800765] envoy[905]: conn peer=[2001:db8:1::1]:38612 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3755 tx=2126 ms=15165844lb # [ 95.804738] envoy[905]: conn peer=[2001:db8:1::1]:38628 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=447 tx=328 ms=565845lb # [ 95.808209] envoy[905]: conn peer=[2001:db8:1::4]:47642 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=51979 tx=2765 ms=727015846lb # [ 95.812074] envoy[905]: conn peer=[2001:db8:1::1]:38634 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=591 tx=522 ms=125847lb # [ 95.815471] envoy[905]: conn peer=[2001:db8:1::3]:53396 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=15189 tx=3173 ms=777865848worker2 # [ 95.799869] nix-grpc-daemon[1206]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 567 ms (attempt 4/5)5849worker2 # [ 95.968973] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:55Z level=info event=scheduler_connected addr=lb:500515850worker2 # [ 95.973155] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5851worker2 # [ 96.174316] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:56Z level=info event=scheduler_connected addr=lb:500515852worker2 # [ 96.178446] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5853worker2 # [ 96.368900] nix-grpc-daemon[1206]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5854worker2 # [ 96.370396] nix-grpc-daemon[1206]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5855worker2 # [ 96.378586] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:56Z level=info event=scheduler_connected addr=lb:500515856worker2 # [ 96.380233] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5857client # copying 2 paths...5858worker2 # [ 96.389907] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:56Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:358445859client # copying path '/nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache' to 'grpc://lb:50051'...5860client # copying path '/nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache' to 'grpc://lb:50051'...5861worker2 # [ 96.400482] nix-daemon[850]: accepted connection from pid 1206, user nix-grpc-daemon (trusted)5862worker2 # [ 96.410595] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:56Z level=info event=niks3_push_started pid=12665863worker2 # [ 96.499604] nix-grpc-daemon[1266]: time=2026-09-22T14:51:56.405Z level=INFO msg="Configuring client TLS" cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker2.pem key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker2.key ca=""5864worker2 # [ 96.581531] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:56Z level=info event=scheduler_connected addr=lb:500515865worker2 # [ 96.585216] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5866worker2 # [ 96.589850] nix-daemon[850]: accepted connection from pid 1279, user nix-grpc-daemon (trusted)5867worker2 # [ 96.604281] nix-daemon[850]: reaped child process 1282, status = succeeded5868lb # [ 96.624545] niks3-server[957]: 2026/09/22 14:51:55 INFO Received uploads request method=POST path=/api/pending_closures5869lb # [ 96.637435] niks3-server[957]: 2026/09/22 14:51:56 INFO Received uploads request method=POST path=/api/pending_closures5870worker2 # [ 96.629477] nix-grpc-daemon[1266]: time=2026-09-22T14:51:56.535Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"5871worker2 # [ 96.631099] nix-grpc-daemon[1266]: time=2026-09-22T14:51:56.537Z level=INFO msg="Uploading 6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache (184B)"5872worker2 # [ 96.633243] nix-grpc-daemon[1266]: time=2026-09-22T14:51:56.539Z level=INFO msg="Uploading cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache (128B)"5873lb # [ 96.688170] niks3-server[957]: 2026/09/22 14:51:56 INFO Registered completed upload object_key=nar/1kmnh5f2l7jjcgaq9b9988bjix6z8zjf6v350xllq5hlzgnb1f75.nar.zst5874lb # [ 96.694159] niks3-server[957]: 2026/09/22 14:51:56 INFO Registered completed upload object_key=nar/15y5akaw5jxrf5q28c25cfgfdz5zw27ihvpagi1mmvpnaw9na2a5.nar.zst5875lb # [ 96.708396] niks3-server[957]: 2026/09/22 14:51:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/35/sign5876lb # [ 96.710552] niks3-server[957]: 2026/09/22 14:51:56 INFO Signed narinfos id=35 count=15877lb # [ 96.713144] niks3-server[957]: 2026/09/22 14:51:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/36/sign5878lb # [ 96.716772] niks3-server[957]: 2026/09/22 14:51:56 INFO Signed narinfos id=36 count=25879worker2 # [ 96.702136] nix-grpc-daemon[1266]: time=2026-09-22T14:51:56.607Z level=INFO msg="Uploading 3 narinfos"5880lb # [ 96.720107] niks3-server[957]: 2026/09/22 14:51:56 INFO Registered completed upload object_key=6zp5fijxpjrd2vv7855sb0z7aag6mm8d.ls5881lb # [ 96.722839] niks3-server[957]: 2026/09/22 14:51:56 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.ls5882lb # [ 96.736947] niks3-server[957]: 2026/09/22 14:51:56 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.narinfo5883lb # [ 96.740106] niks3-server[957]: 2026/09/22 14:51:56 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.narinfo5884lb # [ 96.742360] niks3-server[957]: 2026/09/22 14:51:56 INFO Received complete upload request method=POST path=/api/pending_closures/35/complete5885lb # [ 96.746862] niks3-server[957]: 2026/09/22 14:51:56 INFO Completed upload id=355886lb # [ 96.748519] niks3-server[957]: 2026/09/22 14:51:56 INFO Received complete upload request method=POST path=/api/pending_closures/36/complete5887lb # [ 96.751198] niks3-server[957]: 2026/09/22 14:51:56 INFO Registered completed upload object_key=6zp5fijxpjrd2vv7855sb0z7aag6mm8d.narinfo5888lb # [ 96.753455] niks3-server[957]: 2026/09/22 14:51:56 INFO Completed upload id=365889worker2 # [ 96.738316] nix-grpc-daemon[1266]: time=2026-09-22T14:51:56.644Z level=INFO msg="Upload complete. (235ms)"5890worker2 # [ 96.743774] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:56Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35844 duration_s=0 paths=2 nar_bytes_in=3125891worker2 # [ 96.757946] nix-daemon[850]: reaped child process 1264, status = succeeded5892client: (finished: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache >&2, in 1.64 seconds)5893worker2: must succeed: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache && test -e /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache5894worker2 # [ 96.785283] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:56Z level=info event=scheduler_connected addr=lb:500515895worker2 # [ 96.790213] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5896worker2: (finished: must succeed: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache && test -e /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache, in 0.03 seconds)5897worker1 # [ 96.832972] systemd[1]: Listening on Nix worker-protocol over gRPC.5898worker1 # [ 96.841513] systemd[1]: Starting Nix worker-protocol over gRPC...5899worker1 # [ 96.935559] nix-grpc-daemon[1390]: ts=2026-09-22T14:51:57Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5900worker1 # [ 96.940084] nix-grpc-daemon[1390]: ts=2026-09-22T14:51:57Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5901worker1 # [ 96.943234] nix-grpc-daemon[1390]: ts=2026-09-22T14:51:57Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15902worker1 # [ 96.945208] nix-grpc-daemon[1390]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5903worker1 # [ 96.946617] nix-grpc-daemon[1390]: E0000 00:00:1790088717.059972 1390 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5904worker1 # [ 96.957276] nix-grpc-daemon[1390]: E0000 00:00:1790088717.067923 1390 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5905worker1 # [ 96.959854] nix-grpc-daemon[1390]: ts=2026-09-22T14:51:57Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5906worker1 # [ 96.961564] systemd[1]: Started Nix worker-protocol over gRPC.5907lb # [ 96.992951] niks3-server[957]: 2026/09/22 14:51:56 INFO lead: acquired remote=192.168.1.3:328545908worker1 # [ 96.966190] nix-grpc-daemon[1390]: ts=2026-09-22T14:51:57Z level=info event=scheduler_take_over5909worker1 # [ 96.968148] nix-grpc-daemon[1390]: E0000 00:00:1790088717.079849 1432 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5910worker2 # [ 96.990498] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:56Z level=info event=scheduler_connected addr=lb:500515911worker2 # [ 96.994690] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5912worker1 # [ 96.977937] nix-grpc-daemon[1390]: E0000 00:00:1790088717.095204 1432 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5913worker1 # [ 96.981152] nix-grpc-daemon[1390]: E0000 00:00:1790088717.097565 1432 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5914lb: waiting until exactly no endpoints of x86_64-linux are unhealthy5915lb: must succeed: curl -sf localhost:9901/clusters5916worker1 # [ 96.993804] nix-grpc-daemon[1390]: E0000 00:00:1790088717.111068 1432 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5917worker1 # [ 97.006477] nix-grpc-daemon[1390]: ts=2026-09-22T14:51:57Z level=info event=scheduler_connected addr=lb:500515918worker1 # [ 97.008577] nix-grpc-daemon[1390]: ts=2026-09-22T14:51:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5919lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5920worker2 # [ 97.195998] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:57Z level=info event=scheduler_connected addr=lb:500515921worker1 # [ 97.188209] nix-grpc-daemon[1390]: ts=2026-09-22T14:51:57Z level=info event=oidc_jwks provider=mock keys=15922worker1 # [ 97.191950] nix-grpc-daemon[1390]: ts=2026-09-22T14:51:57Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:413725923worker1 # [ 97.195849] nix-grpc-daemon[1390]: ts=2026-09-22T14:51:57Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05924worker1 # [ 97.209537] nix-grpc-daemon[1390]: ts=2026-09-22T14:51:57Z level=info event=scheduler_connected addr=lb:500515925worker1 # [ 97.213491] nix-grpc-daemon[1390]: ts=2026-09-22T14:51:57Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:413725926worker1 # [ 97.216990] nix-grpc-daemon[1390]: ts=2026-09-22T14:51:57Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05927lb: must succeed: curl -sf localhost:9901/clusters5928lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5929lb: (finished: waiting until exactly no endpoints of x86_64-linux are unhealthy, in 1.10 seconds)5930(finished: subtest: upload whose reference the worker lacks is completed from the cache, in 11.86 seconds)5931subtest: a niks3 restart does not cost the leader its role5932lb: must succeed: curl -sf localhost:9901/clusters5933lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)5934lb: must succeed: curl -sf localhost:9901/clusters5935lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)5936lb # [ 98.193152] systemd[1]: Stopping niks3 server...5937lb # [ 98.194438] niks3-server[957]: 2026/09/22 14:51:57 INFO Shutdown signal received, draining in-flight requests timeout=10s5938worker1 # [ 98.159955] nix-grpc-daemon[1390]: ts=2026-09-22T14:51:58Z level=info event=scheduler_lock_lost status=2005939worker2 # [ 98.179509] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:58Z level=info event=scheduler_lock_lost status=2005940lb # [ 98.201118] niks3-server[957]: 2026/09/22 14:51:57 INFO lead: released remote=192.168.1.3:328545941lb # [ 98.208358] systemd[1]: niks3.service: Deactivated successfully.5942lb # [ 98.210161] systemd[1]: Stopped niks3 server.5943lb # [ 98.210833] systemd[1]: niks3.service: Consumed 698ms CPU time over 1min 19.894s wall clock time, 28.7M memory peak, 93.8K incoming IP traffic, 135.7K outgoing IP traffic.5944lb # [ 98.217923] systemd[1]: Starting niks3 server...5945lb # [ 98.265788] niks3-server[1099]: 2026/09/22 14:51:57 goose: no migrations to run. current version: 202609200000005946lb # [ 98.271242] niks3-server[1099]: 2026/09/22 14:51:57 OK 1_commit_pending_closure.sql (3.95ms)5947lb # [ 98.274477] niks3-server[1099]: 2026/09/22 14:51:57 OK 2_object_stats_trigger.sql (2.04ms)5948lb # [ 98.275585] niks3-server[1099]: 2026/09/22 14:51:57 goose: up to current file version: 25949lb # [ 98.277815] niks3-server[1099]: 2026/09/22 14:51:57 INFO Loaded signing key name=farm-test-1 path=/nix/store/hk0nhypl37jvkn3gr36v3y7qx593p39i-key5950lb # [ 98.288703] niks3-server[1099]: 2026/09/22 14:51:57 INFO Read proxy enabled — serving cache objects from S35951lb # [ 98.291107] niks3-server[1099]: 2026/09/22 14:51:57 INFO Using socket-activated listener address=0.0.0.0:57515952lb # [ 98.293328] systemd[1]: Started niks3 server.5953lb # [ 98.294477] niks3-server[1099]: 2026/09/22 14:51:57 INFO systemd watchdog enabled interval=15s5954lb # [ 98.297122] niks3-server[1099]: 2026/09/22 14:51:57 INFO Starting HTTP server address=0.0.0.0:57515955lb: waiting for unit niks3.service5956lb: (finished: waiting for unit niks3.service, in 0.02 seconds)5957lb: must succeed: curl -sf localhost:9901/clusters5958lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)5959worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5960worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)5961lb: must succeed: curl -sf localhost:9901/clusters5962lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)5963lb: must succeed: curl -sf localhost:9901/clusters5964lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)5965worker1: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield5966worker1: (finished: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield, in 0.04 seconds)5967client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag after-niks3-restart >&25968worker2 # [ 98.565557] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:58Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:358445969client # don't know how to build these paths:5970client # /nix/store/v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv5971worker1 # [ 98.561167] nix-grpc-daemon[1390]: ts=2026-09-22T14:51:58Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:413865972worker1 # [ 98.572640] nix-daemon[869]: accepted connection from pid 1390, user nix-grpc-daemon (trusted)5973worker1 # [ 98.589356] nix-grpc-daemon[1390]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 35 ms (attempt 1/5)5974worker1 # [ 98.632529] nix-grpc-daemon[1390]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 60 ms (attempt 2/5)5975worker1 # [ 98.694565] nix-grpc-daemon[1390]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 319 ms (attempt 3/5)5976worker1 # [ 99.015481] nix-grpc-daemon[1390]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 63 ms (attempt 4/5)5977worker1 # [ 99.081142] nix-grpc-daemon[1390]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5978worker1 # [ 99.084619] nix-grpc-daemon[1390]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5979worker1 # [ 99.111829] nix-grpc-daemon[1390]: ts=2026-09-22T14:51:59Z level=debug event=want client=1 drv=6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv cached=05980worker1 # [ 99.115824] nix-grpc-daemon[1390]: ts=2026-09-22T14:51:59Z level=debug event=assigned drv=6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv worker=192.168.1.4:50051 clients=15981worker2 # [ 99.175596] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:358445982lb # [ 99.198237] niks3-server[1099]: 2026/09/22 14:51:58 INFO lead: acquired remote=192.168.1.3:328745983worker2 # [ 99.196205] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:59Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:358445984client # copying 1 paths to 'lb:50051'...5985client # copying path '/nix/store/6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv' to 'lb:50051'...5986worker2 # [ 99.207259] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:59Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:358445987worker2 # [ 99.215754] nix-daemon[850]: accepted connection from pid 1206, user nix-grpc-daemon (trusted)5988worker2 # [ 99.334487] nix-daemon[850]: accepted connection from pid 1291, user nix-grpc-daemon (trusted)5989worker2 # [ 99.348907] nix-daemon[850]: reaped child process 1294, status = succeeded5990worker2 # [ 99.425701] nix-daemon[850]: accepted connection from pid 1296, user nix-grpc-daemon (trusted)5991worker2 # [ 99.439258] nix-daemon[850]: reaped child process 1299, status = succeeded5992lb # [ 99.461578] niks3-server[1099]: 2026/09/22 14:51:58 INFO Received uploads request method=POST path=/api/pending_closures5993worker2 # [ 99.457672] nix-grpc-daemon[1266]: time=2026-09-22T14:51:59.363Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5994worker2 # [ 99.459080] nix-grpc-daemon[1266]: time=2026-09-22T14:51:59.363Z level=INFO msg="Uploading 6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv (536B)"5995lb # [ 99.498138] niks3-server[1099]: 2026/09/22 14:51:58 INFO Registered completed upload object_key=nar/0wjrljh500qj2dh74gryz4ik45bn1iv9w72490z0f2w0l2x4c87h.nar.zst5996lb # [ 99.504530] niks3-server[1099]: 2026/09/22 14:51:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/37/sign5997lb # [ 99.525067] niks3-server[1099]: 2026/09/22 14:51:58 INFO Signed narinfos id=37 count=15998worker2 # [ 99.512670] nix-grpc-daemon[1266]: time=2026-09-22T14:51:59.419Z level=INFO msg="Uploading 1 narinfos"5999lb # [ 99.541287] niks3-server[1099]: 2026/09/22 14:51:58 INFO Registered completed upload object_key=6finahhb2q4vla8rjb6gi4z4cvf71qmc.ls6000lb # [ 99.543394] niks3-server[1099]: 2026/09/22 14:51:58 INFO Received complete upload request method=POST path=/api/pending_closures/37/complete6001lb # [ 99.549250] niks3-server[1099]: 2026/09/22 14:51:58 INFO Completed upload id=376002lb # [ 99.551229] niks3-server[1099]: 2026/09/22 14:51:58 INFO Registered completed upload object_key=6finahhb2q4vla8rjb6gi4z4cvf71qmc.narinfo6003worker2 # [ 99.535180] nix-grpc-daemon[1266]: time=2026-09-22T14:51:59.441Z level=INFO msg="Upload complete. (305ms)"6004worker2 # [ 99.539579] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:59Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35844 duration_s=0 paths=1 nar_bytes_in=5366005worker2 # [ 99.545515] nix-grpc-daemon[1206]: ts=2026-09-22T14:51:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:358446006worker2 # [ 99.556342] nix-daemon[850]: accepted connection from pid 1206, user nix-grpc-daemon (trusted)6007worker2 # [ 99.566167] nix-daemon[850]: reaped child process 1289, status = succeeded6008client # building '/nix/store/6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv' on lb:50051...6009client # farm-a> worker2: building 6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv6010worker2 # [ 99.571348] nix-daemon[850]: accepted connection from pid 1206, user nix-grpc-daemon (trusted)6011client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds6012client # farm-a> LOG-farm-a-after-niks3-restart6013worker2 # [ 100.989787] nix-daemon[850]: reaped child process 1303, status = succeeded6014worker2 # [ 101.063396] nix-daemon[850]: accepted connection from pid 1319, user nix-grpc-daemon (trusted)6015worker2 # [ 101.075407] nix-daemon[850]: reaped child process 1322, status = succeeded6016lb # [ 101.097742] niks3-server[1099]: 2026/09/22 14:52:00 INFO Received uploads request method=POST path=/api/pending_closures6017worker2 # [ 101.092790] nix-grpc-daemon[1266]: time=2026-09-22T14:52:00.999Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6018worker2 # [ 101.094335] nix-grpc-daemon[1266]: time=2026-09-22T14:52:01.000Z level=INFO msg="Uploading vllq1krpcvis69h349hy98vfz55ld1wv-farm-a (144B)"6019lb # [ 101.132612] niks3-server[1099]: 2026/09/22 14:52:00 INFO Registered completed upload object_key=log/6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv6020lb # [ 101.148756] niks3-server[1099]: 2026/09/22 14:52:00 INFO Registered completed upload object_key=nar/1diw08qzlsh0zkrx4kvj1mvbc57g7r65j1f459ankpg9yc5rgcpx.nar.zst6021lb # [ 101.159248] niks3-server[1099]: 2026/09/22 14:52:00 INFO Received sign narinfos request method=POST path=/api/pending_closures/38/sign6022worker2 # [ 101.149361] nix-grpc-daemon[1266]: time=2026-09-22T14:52:01.055Z level=INFO msg="Uploading 1 narinfos"6023lb # [ 101.164491] niks3-server[1099]: 2026/09/22 14:52:00 INFO Signed narinfos id=38 count=16024lb # [ 101.170692] niks3-server[1099]: 2026/09/22 14:52:00 INFO Registered completed upload object_key=vllq1krpcvis69h349hy98vfz55ld1wv.ls6025lb # [ 101.178259] niks3-server[1099]: 2026/09/22 14:52:00 INFO Received complete upload request method=POST path=/api/pending_closures/38/complete6026lb # [ 101.181341] niks3-server[1099]: 2026/09/22 14:52:00 INFO Completed upload id=386027worker2 # [ 101.166146] nix-grpc-daemon[1266]: time=2026-09-22T14:52:01.072Z level=INFO msg="Upload complete. (186ms)"6028lb # [ 101.183897] niks3-server[1099]: 2026/09/22 14:52:00 INFO Registered completed upload object_key=vllq1krpcvis69h349hy98vfz55ld1wv.narinfo6029worker1 # [ 101.158776] nix-grpc-daemon[1390]: ts=2026-09-22T14:52:01Z level=debug event=want client=1 drv=5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv cached=06030worker2 # [ 101.172788] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:01Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35844 duration_s=1 drv=6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv assign_id=1 outputs=16031worker1 # [ 101.160641] nix-grpc-daemon[1390]: ts=2026-09-22T14:52:01Z level=debug event=assigned drv=5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv worker=192.168.1.4:50051 clients=16032worker2 # [ 101.188324] nix-daemon[850]: reaped child process 1301, status = succeeded6033worker2 # [ 101.221144] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:01Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:358446034worker2 # [ 101.233472] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:01Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:358446035worker2 # [ 101.246927] nix-grpc-daemon[1266]: time=2026-09-22T14:52:01.153Z level=INFO msg="All 1 paths already cached"6036client # copying 1 paths to 'lb:50051'...6037client # copying path '/nix/store/5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv' to 'lb:50051'...6038worker2 # [ 101.254507] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:01Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:358446039worker2 # [ 101.262671] nix-daemon[850]: accepted connection from pid 1206, user nix-grpc-daemon (trusted)6040worker2 # [ 101.358647] nix-daemon[850]: accepted connection from pid 1326, user nix-grpc-daemon (trusted)6041worker2 # [ 101.371832] nix-daemon[850]: reaped child process 1329, status = succeeded6042worker2 # [ 101.451507] nix-daemon[850]: accepted connection from pid 1331, user nix-grpc-daemon (trusted)6043worker2 # [ 101.465670] nix-daemon[850]: reaped child process 1334, status = succeeded6044lb # [ 101.486904] niks3-server[1099]: 2026/09/22 14:52:00 INFO Received uploads request method=POST path=/api/pending_closures6045worker2 # [ 101.479176] nix-grpc-daemon[1266]: time=2026-09-22T14:52:01.385Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6046worker2 # [ 101.481109] nix-grpc-daemon[1266]: time=2026-09-22T14:52:01.387Z level=INFO msg="Uploading 5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv (656B)"6047lb # [ 101.516400] niks3-server[1099]: 2026/09/22 14:52:00 INFO Received sign narinfos request method=POST path=/api/pending_closures/39/sign6048worker2 # [ 101.506491] nix-grpc-daemon[1266]: time=2026-09-22T14:52:01.412Z level=INFO msg="Uploading 1 narinfos"6049lb # [ 101.523184] niks3-server[1099]: 2026/09/22 14:52:00 INFO Signed narinfos id=39 count=16050lb # [ 101.530718] niks3-server[1099]: 2026/09/22 14:52:00 INFO Registered completed upload object_key=nar/0sjnwmbwn7i8g7g7czwn9r2pzaaa1j1yx3dfa4gw29ynhmpz9v3j.nar.zst6051lb # [ 101.542840] niks3-server[1099]: 2026/09/22 14:52:00 INFO Registered completed upload object_key=5g2krgx4inyydhy0jrhhylv4xhhlm7j7.ls6052lb # [ 101.549675] niks3-server[1099]: 2026/09/22 14:52:00 INFO Received complete upload request method=POST path=/api/pending_closures/39/complete6053lb # [ 101.554565] niks3-server[1099]: 2026/09/22 14:52:00 INFO Completed upload id=396054worker2 # [ 101.541427] nix-grpc-daemon[1266]: time=2026-09-22T14:52:01.447Z level=INFO msg="Upload complete. (266ms)"6055worker2 # [ 101.543737] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:01Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35844 duration_s=0 paths=1 nar_bytes_in=6566056lb # [ 101.561114] niks3-server[1099]: 2026/09/22 14:52:00 INFO Registered completed upload object_key=5g2krgx4inyydhy0jrhhylv4xhhlm7j7.narinfo6057worker2 # [ 101.547718] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:01Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:358446058worker2 # [ 101.554177] nix-daemon[850]: accepted connection from pid 1206, user nix-grpc-daemon (trusted)6059worker2 # [ 101.556579] nix-daemon[850]: reaped child process 1324, status = succeeded6060client # building '/nix/store/5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv' on lb:50051...6061client # farm-b> worker2: building 5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv6062worker2 # [ 101.561776] nix-daemon[850]: accepted connection from pid 1206, user nix-grpc-daemon (trusted)6063client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds6064client # farm-b> LOG-farm-b-after-niks3-restart6065worker2 # [ 102.777922] nix-daemon[850]: reaped child process 1338, status = succeeded6066worker2 # [ 102.852090] nix-daemon[850]: accepted connection from pid 1354, user nix-grpc-daemon (trusted)6067worker2 # [ 102.864089] nix-daemon[850]: reaped child process 1357, status = succeeded6068lb # [ 102.887198] niks3-server[1099]: 2026/09/22 14:52:02 INFO Received uploads request method=POST path=/api/pending_closures6069worker2 # [ 102.883230] nix-grpc-daemon[1266]: time=2026-09-22T14:52:02.789Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6070worker2 # [ 102.885573] nix-grpc-daemon[1266]: time=2026-09-22T14:52:02.791Z level=INFO msg="Uploading vkqg7lcax8h1mxs0g750ficqiii1falw-farm-b (192B)"6071lb # [ 102.927624] niks3-server[1099]: 2026/09/22 14:52:02 INFO Registered completed upload object_key=nar/12zl6l8xhwx5hdivnvf1vw18yzvz3764mhdd9n03x81phzqblxj0.nar.zst6072lb # [ 102.934165] niks3-server[1099]: 2026/09/22 14:52:02 INFO Registered completed upload object_key=log/5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv6073lb # [ 102.943205] niks3-server[1099]: 2026/09/22 14:52:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/40/sign6074lb # [ 102.947349] niks3-server[1099]: 2026/09/22 14:52:02 INFO Signed narinfos id=40 count=16075worker2 # [ 102.931952] nix-grpc-daemon[1266]: time=2026-09-22T14:52:02.838Z level=INFO msg="Uploading 1 narinfos"6076lb # [ 102.953740] niks3-server[1099]: 2026/09/22 14:52:02 INFO Registered completed upload object_key=vkqg7lcax8h1mxs0g750ficqiii1falw.ls6077lb # [ 102.957532] niks3-server[1099]: 2026/09/22 14:52:02 INFO Received complete upload request method=POST path=/api/pending_closures/40/complete6078lb # [ 102.962434] niks3-server[1099]: 2026/09/22 14:52:02 INFO Completed upload id=406079lb # [ 102.964532] niks3-server[1099]: 2026/09/22 14:52:02 INFO Registered completed upload object_key=vkqg7lcax8h1mxs0g750ficqiii1falw.narinfo6080worker2 # [ 102.946968] nix-grpc-daemon[1266]: time=2026-09-22T14:52:02.853Z level=INFO msg="Upload complete. (178ms)"6081worker2 # [ 102.953370] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:02Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35844 duration_s=1 drv=5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv assign_id=2 outputs=16082worker1 # [ 102.941877] nix-grpc-daemon[1390]: ts=2026-09-22T14:52:03Z level=debug event=want client=1 drv=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv cached=06083worker2 # [ 102.966232] nix-daemon[850]: reaped child process 1336, status = succeeded6084worker1 # [ 102.946257] nix-grpc-daemon[1390]: ts=2026-09-22T14:52:03Z level=debug event=assigned drv=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv worker=192.168.1.4:50051 clients=16085worker2 # [ 103.004255] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:358446086worker2 # [ 103.016204] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:02Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:358446087worker2 # [ 103.030939] nix-grpc-daemon[1266]: time=2026-09-22T14:52:02.937Z level=INFO msg="All 2 paths already cached"6088client # copying 1 paths to 'lb:50051'...6089client # copying path '/nix/store/v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv' to 'lb:50051'...6090worker2 # [ 103.039884] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:02Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:358446091worker2 # [ 103.048348] nix-daemon[850]: accepted connection from pid 1206, user nix-grpc-daemon (trusted)6092worker2 # [ 103.146987] nix-daemon[850]: accepted connection from pid 1361, user nix-grpc-daemon (trusted)6093worker2 # [ 103.161257] nix-daemon[850]: reaped child process 1364, status = succeeded6094worker2 # [ 103.245136] nix-daemon[850]: accepted connection from pid 1366, user nix-grpc-daemon (trusted)6095worker2 # [ 103.260592] nix-daemon[850]: reaped child process 1369, status = succeeded6096lb # [ 103.280132] niks3-server[1099]: 2026/09/22 14:52:02 INFO Received uploads request method=POST path=/api/pending_closures6097worker2 # [ 103.273917] nix-grpc-daemon[1266]: time=2026-09-22T14:52:03.180Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6098worker2 # [ 103.275398] nix-grpc-daemon[1266]: time=2026-09-22T14:52:03.181Z level=INFO msg="Uploading v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv (784B)"6099lb # [ 103.320586] niks3-server[1099]: 2026/09/22 14:52:02 INFO Registered completed upload object_key=nar/0ppnnsmr641nka691g6vrp1crps1bbn31zm7nc5za9pl2y2iwb0f.nar.zst6100lb # [ 103.333792] niks3-server[1099]: 2026/09/22 14:52:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/41/sign6101lb # [ 103.339257] niks3-server[1099]: 2026/09/22 14:52:02 INFO Signed narinfos id=41 count=16102worker2 # [ 103.326201] nix-grpc-daemon[1266]: time=2026-09-22T14:52:03.232Z level=INFO msg="Uploading 1 narinfos"6103lb # [ 103.345680] niks3-server[1099]: 2026/09/22 14:52:02 INFO Registered completed upload object_key=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c.ls6104lb # [ 103.353893] niks3-server[1099]: 2026/09/22 14:52:02 INFO Received complete upload request method=POST path=/api/pending_closures/41/complete6105lb # [ 103.359651] niks3-server[1099]: 2026/09/22 14:52:02 INFO Completed upload id=416106worker2 # [ 103.344278] nix-grpc-daemon[1266]: time=2026-09-22T14:52:03.250Z level=INFO msg="Upload complete. (282ms)"6107lb # [ 103.361250] niks3-server[1099]: 2026/09/22 14:52:02 INFO Registered completed upload object_key=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c.narinfo6108worker2 # [ 103.348289] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:03Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35844 duration_s=0 paths=1 nar_bytes_in=7846109worker2 # [ 103.353277] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:358446110worker2 # [ 103.362715] nix-daemon[850]: accepted connection from pid 1206, user nix-grpc-daemon (trusted)6111worker2 # [ 103.370441] nix-daemon[850]: reaped child process 1359, status = succeeded6112worker2 # [ 103.375598] nix-daemon[850]: accepted connection from pid 1206, user nix-grpc-daemon (trusted)6113client # building '/nix/store/v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv' on lb:50051...6114client # farm-top> worker2: building v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv6115client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds6116client # farm-top> LOG-farm-top-after-niks3-restart6117worker2 # [ 104.611655] nix-daemon[850]: reaped child process 1373, status = succeeded6118worker2 # [ 104.700764] nix-daemon[850]: accepted connection from pid 1389, user nix-grpc-daemon (trusted)6119worker2 # [ 104.714176] nix-daemon[850]: reaped child process 1392, status = succeeded6120lb # [ 104.738475] niks3-server[1099]: 2026/09/22 14:52:04 INFO Received uploads request method=POST path=/api/pending_closures6121worker2 # [ 104.733259] nix-grpc-daemon[1266]: time=2026-09-22T14:52:04.639Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6122worker2 # [ 104.736663] nix-grpc-daemon[1266]: time=2026-09-22T14:52:04.641Z level=INFO msg="Uploading lyi0sw9pvmhznkzgfl39callq9c0n87r-farm-top (248B)"6123lb # [ 104.768884] niks3-server[1099]: 2026/09/22 14:52:04 INFO Registered completed upload object_key=log/v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv6124lb # [ 104.788791] niks3-server[1099]: 2026/09/22 14:52:04 INFO Registered completed upload object_key=nar/141220x0digv3f58fbh9c79avwg0fwgsk5jyac9ymkxmr8zs2prw.nar.zst6125lb # [ 104.798181] niks3-server[1099]: 2026/09/22 14:52:04 INFO Received sign narinfos request method=POST path=/api/pending_closures/42/sign6126worker2 # [ 104.787530] nix-grpc-daemon[1266]: time=2026-09-22T14:52:04.693Z level=INFO msg="Uploading 1 narinfos"6127lb # [ 104.804180] niks3-server[1099]: 2026/09/22 14:52:04 INFO Signed narinfos id=42 count=16128lb # [ 104.809696] niks3-server[1099]: 2026/09/22 14:52:04 INFO Registered completed upload object_key=lyi0sw9pvmhznkzgfl39callq9c0n87r.ls6129lb # [ 104.817063] niks3-server[1099]: 2026/09/22 14:52:04 INFO Received complete upload request method=POST path=/api/pending_closures/42/complete6130lb # [ 104.819993] niks3-server[1099]: 2026/09/22 14:52:04 INFO Completed upload id=426131worker2 # [ 104.804662] nix-grpc-daemon[1266]: time=2026-09-22T14:52:04.711Z level=INFO msg="Upload complete. (220ms)"6132lb # [ 104.823355] niks3-server[1099]: 2026/09/22 14:52:04 INFO Registered completed upload object_key=lyi0sw9pvmhznkzgfl39callq9c0n87r.narinfo6133worker2 # [ 104.811804] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:04Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35844 duration_s=1 drv=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv assign_id=3 outputs=16134worker2 # [ 104.829169] nix-daemon[850]: reaped child process 1371, status = succeeded6135client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag after-niks3-restart >&2, in 6.40 seconds)6136client: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag after-niks3-restart outPath6137client: (finished: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag after-niks3-restart outPath, in 0.12 seconds)6138(finished: subtest: a niks3 restart does not cost the leader its role, in 6.87 seconds)6139subtest: scheduler down: the other node takes the lock and keeps it6140lb: must succeed: curl -sf localhost:9901/clusters6141lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)6142lb: must succeed: curl -sf localhost:9901/clusters6143lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)6144lb: must succeed: curl -sf localhost:9901/clusters6145lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)6146lb: must succeed: curl -sf localhost:9901/clusters6147lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)6148worker1 # [ 105.051336] systemd[1]: Stopping Nix worker-protocol over gRPC...6149lb # [ 105.800660] envoy[905]: conn peer=[2001:db8:1::1]:47956 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="TLS_error:|33554464:system_library:OPENSSL_internal:Broken_pipe:TLS_error_end" rx=1004 tx=282 ms=14836150lb # [ 105.806150] envoy[905]: conn peer=[2001:db8:1::1]:47968 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9709 tx=4427 ms=62636151worker1 # [ 105.956674] nix-grpc-daemon[1390]: ts=2026-09-22T14:52:06Z level=info event=signal_exit6152lb # [ 106.203878] niks3-server[1099]: 2026/09/22 14:52:05 INFO lead: released remote=192.168.1.3:328746153worker2 # [ 107.189976] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:07Z level=info event=scheduler_restarting addr=lb:500516154worker1 # [ 107.174914] nix-grpc-daemon[1390]: ts=2026-09-22T14:52:07Z level=info event=scheduler_restarting6155worker1 # [ 107.178609] nix-grpc-daemon[1390]: ts=2026-09-22T14:52:07Z level=info event=scheduler_restarting addr=lb:500516156worker1 # [ 107.182253] nix-grpc-daemon[1390]: ts=2026-09-22T14:52:07Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:413726157worker1 # [ 107.186264] nix-grpc-daemon[1390]: ts=2026-09-22T14:52:07Z level=info event=worker_gone addr=192.168.1.3:500516158worker1 # [ 107.189578] nix-grpc-daemon[1390]: ts=2026-09-22T14:52:07Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:413726159worker1 # [ 107.194245] nix-grpc-daemon[1390]: ts=2026-09-22T14:52:07Z level=info event=worker_gone addr=192.168.1.4:500516160worker1 # [ 107.205590] nix-daemon[869]: reaped child process 1450, status = succeeded6161worker2 # [ 107.390935] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:07Z level=info event=scheduler_connected addr=lb:500516162worker2 # [ 107.395319] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6163worker2 # [ 107.595621] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:07Z level=info event=scheduler_connected addr=lb:500516164worker2 # [ 107.597437] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6165worker2 # [ 107.798233] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:07Z level=info event=scheduler_connected addr=lb:500516166worker2 # [ 107.800065] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6167worker2 # [ 108.001398] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:07Z level=info event=scheduler_connected addr=lb:500516168worker2 # [ 108.005481] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6169worker1 # [ 108.069790] nix-grpc-daemon[1390]: E0000 00:00:1790088728.186802 1435 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 10, started true, shutdown true6170worker1 # [ 108.074929] nix-grpc-daemon[1390]: E0000 00:00:1790088728.186965 1435 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true6171worker2 # [ 108.207484] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:08Z level=info event=scheduler_connected addr=lb:500516172worker2 # [ 108.211369] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6173worker1 # [ 108.376585] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.6174worker1 # [ 108.379325] systemd[1]: Stopped Nix worker-protocol over gRPC.6175worker1 # [ 108.380449] systemd[1]: nix-grpc-daemon.service: Consumed 218ms CPU time over 11.538s wall clock time, 13.4M memory peak, 10.3K incoming IP traffic, 14.6K outgoing IP traffic.6176worker1 # [ 108.384683] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.6177worker1 # [ 108.386368] systemd[1]: Closed Nix worker-protocol over gRPC.6178worker1 # [ 108.387946] systemd[1]: nix-grpc-daemon.socket: Consumed 595us CPU time over 11.560s wall clock time, 256K memory peak, 24.3K incoming IP traffic, 18.3K outgoing IP traffic.6179worker2: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over6180worker2 # [ 108.412209] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:08Z level=info event=scheduler_connected addr=lb:500516181worker2 # [ 108.416458] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6182worker2 # [ 108.617609] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:08Z level=info event=scheduler_connected addr=lb:500516183worker2 # [ 108.621546] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6184worker2 # [ 108.822852] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:08Z level=info event=scheduler_connected addr=lb:500516185worker2 # [ 108.826784] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6186worker2 # [ 109.027938] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:08Z level=info event=scheduler_connected addr=lb:500516187worker2 # [ 109.031890] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6188lb # [ 109.208382] niks3-server[1099]: 2026/09/22 14:52:08 INFO lead: acquired remote=192.168.1.4:371186189worker2 # [ 109.192661] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:09Z level=info event=scheduler_take_over6190worker2 # [ 109.233670] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:09Z level=info event=scheduler_connected addr=lb:500516191worker2 # [ 109.237459] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6192worker2 # [ 109.438574] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:09Z level=info event=scheduler_connected addr=lb:500516193worker2 # [ 109.456234] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:09Z level=info event=oidc_jwks provider=mock keys=16194worker2 # [ 109.459865] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:09Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:558946195worker2 # [ 109.463729] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:09Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06196worker2: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over, in 1.10 seconds)6197worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6198worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.02 seconds)6199client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag on-standby >&26200worker2 # [ 109.678147] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:09Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:358446201client # don't know how to build these paths:6202client # /nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv6203worker2 # [ 109.687196] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:09Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:358446204worker2 # [ 109.708924] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:09Z level=debug event=want client=1 drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv cached=06205worker2 # [ 109.714094] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:09Z level=debug event=assigned drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv worker=192.168.1.4:50051 clients=16206worker2 # [ 109.752781] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:358446207worker2 # [ 109.764623] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:09Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:358446208client # copying 1 paths to 'lb:50051'...6209client # copying path '/nix/store/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv' to 'lb:50051'...6210worker2 # [ 109.776375] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:09Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:358446211worker2 # [ 109.787264] nix-daemon[850]: accepted connection from pid 1206, user nix-grpc-daemon (trusted)6212worker2 # [ 109.890752] nix-daemon[850]: accepted connection from pid 1413, user nix-grpc-daemon (trusted)6213worker2 # [ 109.904199] nix-daemon[850]: reaped child process 1416, status = succeeded6214worker2 # [ 109.989307] nix-daemon[850]: accepted connection from pid 1418, user nix-grpc-daemon (trusted)6215worker2 # [ 110.003642] nix-daemon[850]: reaped child process 1421, status = succeeded6216lb # [ 110.025318] niks3-server[1099]: 2026/09/22 14:52:09 INFO Received uploads request method=POST path=/api/pending_closures6217worker2 # [ 110.018521] nix-grpc-daemon[1266]: time=2026-09-22T14:52:09.924Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6218worker2 # [ 110.020061] nix-grpc-daemon[1266]: time=2026-09-22T14:52:09.924Z level=INFO msg="Uploading 0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv (520B)"6219lb # [ 110.054174] niks3-server[1099]: 2026/09/22 14:52:09 INFO Registered completed upload object_key=nar/013ynk63lqgbvdpr2qqb6pflhzfcrl005f3zkvnag3x7hqxjhibs.nar.zst6220lb # [ 110.060611] niks3-server[1099]: 2026/09/22 14:52:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/43/sign6221lb # [ 110.066778] niks3-server[1099]: 2026/09/22 14:52:09 INFO Signed narinfos id=43 count=16222worker2 # [ 110.054669] nix-grpc-daemon[1266]: time=2026-09-22T14:52:09.961Z level=INFO msg="Uploading 1 narinfos"6223lb # [ 110.087876] niks3-server[1099]: 2026/09/22 14:52:09 INFO Received complete upload request method=POST path=/api/pending_closures/43/complete6224lb # [ 110.094358] niks3-server[1099]: 2026/09/22 14:52:09 INFO Registered completed upload object_key=0i80y1x9s0d06s86lhapa650pgsxgi0g.ls6225lb # [ 110.102869] niks3-server[1099]: 2026/09/22 14:52:09 INFO Completed upload id=436226worker2 # [ 110.087183] nix-grpc-daemon[1266]: time=2026-09-22T14:52:09.993Z level=INFO msg="Upload complete. (288ms)"6227lb # [ 110.106337] niks3-server[1099]: 2026/09/22 14:52:09 INFO Registered completed upload object_key=0i80y1x9s0d06s86lhapa650pgsxgi0g.narinfo6228worker2 # [ 110.091879] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:09Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35844 duration_s=0 paths=1 nar_bytes_in=5206229worker2 # [ 110.099259] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:358446230worker2 # [ 110.108473] nix-daemon[850]: accepted connection from pid 1206, user nix-grpc-daemon (trusted)6231worker2 # [ 110.118043] nix-daemon[850]: reaped child process 1411, status = succeeded6232worker2 # [ 110.122329] nix-daemon[850]: accepted connection from pid 1206, user nix-grpc-daemon (trusted)6233client # building '/nix/store/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv' on lb:50051...6234client # farm-a> worker2: building 0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv6235client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds6236client # farm-a> LOG-farm-a-on-standby6237worker2 # [ 111.566230] nix-daemon[850]: reaped child process 1425, status = succeeded6238worker2 # [ 111.643897] nix-daemon[850]: accepted connection from pid 1441, user nix-grpc-daemon (trusted)6239worker2 # [ 111.656783] nix-daemon[850]: reaped child process 1444, status = succeeded6240lb # [ 111.678721] niks3-server[1099]: 2026/09/22 14:52:11 INFO Received uploads request method=POST path=/api/pending_closures6241worker2 # [ 111.676302] nix-grpc-daemon[1266]: time=2026-09-22T14:52:11.582Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6242worker2 # [ 111.677800] nix-grpc-daemon[1266]: time=2026-09-22T14:52:11.584Z level=INFO msg="Uploading fp54697jryq5rvfg8ml51wzkr9xjvhrg-farm-a (136B)"6243lb # [ 111.721377] niks3-server[1099]: 2026/09/22 14:52:11 INFO Registered completed upload object_key=nar/0x76wzgdd1nk8rm930z8l69v9961jpc8qhzmgganv0nhagq8lv9c.nar.zst6244lb # [ 111.739541] niks3-server[1099]: 2026/09/22 14:52:11 INFO Registered completed upload object_key=log/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv6245lb # [ 111.742587] niks3-server[1099]: 2026/09/22 14:52:11 INFO Received sign narinfos request method=POST path=/api/pending_closures/44/sign6246lb # [ 111.747098] niks3-server[1099]: 2026/09/22 14:52:11 INFO Signed narinfos id=44 count=16247worker2 # [ 111.734565] nix-grpc-daemon[1266]: time=2026-09-22T14:52:11.640Z level=INFO msg="Uploading 1 narinfos"6248lb # [ 111.758167] niks3-server[1099]: 2026/09/22 14:52:11 INFO Registered completed upload object_key=fp54697jryq5rvfg8ml51wzkr9xjvhrg.ls6249lb # [ 111.762182] niks3-server[1099]: 2026/09/22 14:52:11 INFO Received complete upload request method=POST path=/api/pending_closures/44/complete6250lb # [ 111.768038] niks3-server[1099]: 2026/09/22 14:52:11 INFO Completed upload id=446251worker2 # [ 111.752691] nix-grpc-daemon[1266]: time=2026-09-22T14:52:11.659Z level=INFO msg="Upload complete. (221ms)"6252lb # [ 111.770613] niks3-server[1099]: 2026/09/22 14:52:11 INFO Registered completed upload object_key=fp54697jryq5rvfg8ml51wzkr9xjvhrg.narinfo6253worker2 # [ 111.759700] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:11Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35844 duration_s=1 drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv assign_id=1 outputs=16254worker2 # [ 111.773679] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:11Z level=debug event=want client=1 drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv cached=06255worker2 # [ 111.778188] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:11Z level=debug event=assigned drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv worker=192.168.1.4:50051 clients=16256worker2 # [ 111.787699] nix-daemon[850]: reaped child process 1423, status = succeeded6257worker2 # [ 111.816672] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:11Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:358446258worker2 # [ 111.827537] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:11Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:358446259worker2 # [ 111.837725] nix-grpc-daemon[1266]: time=2026-09-22T14:52:11.744Z level=INFO msg="All 1 paths already cached"6260client # copying 1 paths to 'lb:50051'...6261client # copying path '/nix/store/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv' to 'lb:50051'...6262worker2 # [ 111.845569] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:11Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:358446263worker2 # [ 111.855130] nix-daemon[850]: accepted connection from pid 1206, user nix-grpc-daemon (trusted)6264worker2 # [ 111.946859] nix-daemon[850]: accepted connection from pid 1450, user nix-grpc-daemon (trusted)6265worker2 # [ 111.960722] nix-daemon[850]: reaped child process 1453, status = succeeded6266worker2 # [ 112.041847] nix-daemon[850]: accepted connection from pid 1455, user nix-grpc-daemon (trusted)6267worker2 # [ 112.055848] nix-daemon[850]: reaped child process 1458, status = succeeded6268lb # [ 112.076652] niks3-server[1099]: 2026/09/22 14:52:11 INFO Received uploads request method=POST path=/api/pending_closures6269worker2 # [ 112.066301] nix-grpc-daemon[1266]: time=2026-09-22T14:52:11.972Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6270worker2 # [ 112.068247] nix-grpc-daemon[1266]: time=2026-09-22T14:52:11.974Z level=INFO msg="Uploading 6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv (640B)"6271lb # [ 112.092246] niks3-server[1099]: 2026/09/22 14:52:11 INFO Registered completed upload object_key=nar/12kaxaz59hi3nqzgsxphwdpbajdwrfjqknsv96cnrhpibnnzjlg0.nar.zst6272lb # [ 112.095528] niks3-server[1099]: 2026/09/22 14:52:11 INFO Received sign narinfos request method=POST path=/api/pending_closures/45/sign6273lb # [ 112.097207] niks3-server[1099]: 2026/09/22 14:52:11 INFO Signed narinfos id=45 count=16274worker2 # [ 112.083046] nix-grpc-daemon[1266]: time=2026-09-22T14:52:11.989Z level=INFO msg="Uploading 1 narinfos"6275lb # [ 112.103877] niks3-server[1099]: 2026/09/22 14:52:11 INFO Registered completed upload object_key=6532q0zszdl4cnys1gb39vpngyzd7y8j.ls6276lb # [ 112.105806] niks3-server[1099]: 2026/09/22 14:52:11 INFO Received complete upload request method=POST path=/api/pending_closures/45/complete6277lb # [ 112.109523] niks3-server[1099]: 2026/09/22 14:52:11 INFO Completed upload id=456278worker2 # [ 112.093752] nix-grpc-daemon[1266]: time=2026-09-22T14:52:12.000Z level=INFO msg="Upload complete. (229ms)"6279worker2 # [ 112.095855] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:12Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35844 duration_s=0 paths=1 nar_bytes_in=6406280lb # [ 112.112151] niks3-server[1099]: 2026/09/22 14:52:11 INFO Registered completed upload object_key=6532q0zszdl4cnys1gb39vpngyzd7y8j.narinfo6281worker2 # [ 112.100458] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:12Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:358446282worker2 # [ 112.106306] nix-daemon[850]: accepted connection from pid 1206, user nix-grpc-daemon (trusted)6283worker2 # [ 112.108716] nix-daemon[850]: reaped child process 1448, status = succeeded6284worker2 # [ 112.113873] nix-daemon[850]: accepted connection from pid 1206, user nix-grpc-daemon (trusted)6285client # building '/nix/store/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv' on lb:50051...6286client # farm-b> worker2: building 6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv6287client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds6288client # farm-b> LOG-farm-b-on-standby6289worker2 # [ 112.953478] nix-daemon[850]: reaped child process 1462, status = succeeded6290worker2 # [ 113.026739] nix-daemon[850]: accepted connection from pid 1478, user nix-grpc-daemon (trusted)6291worker2 # [ 113.040285] nix-daemon[850]: reaped child process 1481, status = succeeded6292lb # [ 113.063201] niks3-server[1099]: 2026/09/22 14:52:12 INFO Received uploads request method=POST path=/api/pending_closures6293worker2 # [ 113.056290] nix-grpc-daemon[1266]: time=2026-09-22T14:52:12.962Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6294worker2 # [ 113.059056] nix-grpc-daemon[1266]: time=2026-09-22T14:52:12.964Z level=INFO msg="Uploading 593ccq0xcckfsdii9xb4x5h0bapmahkp-farm-b (184B)"6295lb # [ 113.098338] niks3-server[1099]: 2026/09/22 14:52:12 INFO Registered completed upload object_key=log/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv6296lb # [ 113.115623] niks3-server[1099]: 2026/09/22 14:52:12 INFO Received sign narinfos request method=POST path=/api/pending_closures/46/sign6297lb # [ 113.123617] niks3-server[1099]: 2026/09/22 14:52:12 INFO Signed narinfos id=46 count=16298worker2 # [ 113.111424] nix-grpc-daemon[1266]: time=2026-09-22T14:52:13.017Z level=INFO msg="Uploading 1 narinfos"6299lb # [ 113.127721] niks3-server[1099]: 2026/09/22 14:52:12 INFO Registered completed upload object_key=nar/0k21bb46wfcxlq6p91pdvlzw7kkc20p87xpxr7zq5c5zr17k6rda.nar.zst6300lb # [ 113.136343] niks3-server[1099]: 2026/09/22 14:52:12 INFO Registered completed upload object_key=593ccq0xcckfsdii9xb4x5h0bapmahkp.ls6301lb # [ 113.145350] niks3-server[1099]: 2026/09/22 14:52:12 INFO Received complete upload request method=POST path=/api/pending_closures/46/complete6302lb # [ 113.148625] niks3-server[1099]: 2026/09/22 14:52:12 INFO Completed upload id=466303worker2 # [ 113.134570] nix-grpc-daemon[1266]: time=2026-09-22T14:52:13.041Z level=INFO msg="Upload complete. (213ms)"6304lb # [ 113.152769] niks3-server[1099]: 2026/09/22 14:52:12 INFO Registered completed upload object_key=593ccq0xcckfsdii9xb4x5h0bapmahkp.narinfo6305worker2 # [ 113.141587] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:13Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35844 duration_s=1 drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv assign_id=2 outputs=16306worker2 # [ 113.153577] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:13Z level=debug event=want client=1 drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv cached=06307worker2 # [ 113.158191] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:13Z level=debug event=assigned drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv worker=192.168.1.4:50051 clients=16308worker2 # [ 113.167288] nix-daemon[850]: reaped child process 1460, status = succeeded6309worker2 # [ 113.197143] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:13Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:358446310worker2 # [ 113.209408] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:13Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:358446311worker2 # [ 113.221553] nix-grpc-daemon[1266]: time=2026-09-22T14:52:13.127Z level=INFO msg="All 2 paths already cached"6312client # copying 1 paths to 'lb:50051'...6313client # copying path '/nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv' to 'lb:50051'...6314worker2 # [ 113.230311] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:13Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:358446315worker2 # [ 113.240176] nix-daemon[850]: accepted connection from pid 1206, user nix-grpc-daemon (trusted)6316worker2 # [ 113.338904] nix-daemon[850]: accepted connection from pid 1487, user nix-grpc-daemon (trusted)6317worker2 # [ 113.353687] nix-daemon[850]: reaped child process 1490, status = succeeded6318worker2 # [ 113.435275] nix-daemon[850]: accepted connection from pid 1492, user nix-grpc-daemon (trusted)6319worker2 # [ 113.448511] nix-daemon[850]: reaped child process 1495, status = succeeded6320lb # [ 113.471377] niks3-server[1099]: 2026/09/22 14:52:12 INFO Received uploads request method=POST path=/api/pending_closures6321worker2 # [ 113.464732] nix-grpc-daemon[1266]: time=2026-09-22T14:52:13.370Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6322worker2 # [ 113.466172] nix-grpc-daemon[1266]: time=2026-09-22T14:52:13.371Z level=INFO msg="Uploading bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv (768B)"6323lb # [ 113.502218] niks3-server[1099]: 2026/09/22 14:52:12 INFO Received sign narinfos request method=POST path=/api/pending_closures/47/sign6324worker2 # [ 113.493190] nix-grpc-daemon[1266]: time=2026-09-22T14:52:13.399Z level=INFO msg="Uploading 1 narinfos"6325lb # [ 113.509129] niks3-server[1099]: 2026/09/22 14:52:12 INFO Signed narinfos id=47 count=16326lb # [ 113.512526] niks3-server[1099]: 2026/09/22 14:52:12 INFO Registered completed upload object_key=nar/0y5h6gmqp28k8bkzgrmc35pj8c8qhdi5bc0l17s14j217c1rzscm.nar.zst6327lb # [ 113.520149] niks3-server[1099]: 2026/09/22 14:52:12 INFO Registered completed upload object_key=bgavyqwkalx49wvzslr2znbhxfj7h1mb.ls6328lb # [ 113.531750] niks3-server[1099]: 2026/09/22 14:52:12 INFO Received complete upload request method=POST path=/api/pending_closures/47/complete6329lb # [ 113.535306] niks3-server[1099]: 2026/09/22 14:52:12 INFO Completed upload id=476330worker2 # [ 113.521247] nix-grpc-daemon[1266]: time=2026-09-22T14:52:13.427Z level=INFO msg="Upload complete. (270ms)"6331lb # [ 113.537710] niks3-server[1099]: 2026/09/22 14:52:12 INFO Registered completed upload object_key=bgavyqwkalx49wvzslr2znbhxfj7h1mb.narinfo6332worker2 # [ 113.524405] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:13Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35844 duration_s=0 paths=1 nar_bytes_in=7686333worker2 # [ 113.529302] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:13Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:358446334worker2 # [ 113.539581] nix-daemon[850]: accepted connection from pid 1206, user nix-grpc-daemon (trusted)6335worker2 # [ 113.545299] nix-daemon[850]: reaped child process 1485, status = succeeded6336worker2 # [ 113.548279] nix-daemon[850]: accepted connection from pid 1206, user nix-grpc-daemon (trusted)6337client # building '/nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv' on lb:50051...6338client # farm-top> worker2: building bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv6339client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds6340client # farm-top> LOG-farm-top-on-standby6341worker2 # [ 114.795725] nix-daemon[850]: reaped child process 1499, status = succeeded6342worker2 # [ 114.866523] nix-daemon[850]: accepted connection from pid 1515, user nix-grpc-daemon (trusted)6343worker2 # [ 114.881185] nix-daemon[850]: reaped child process 1518, status = succeeded6344lb # [ 114.901931] niks3-server[1099]: 2026/09/22 14:52:14 INFO Received uploads request method=POST path=/api/pending_closures6345worker2 # [ 114.895496] nix-grpc-daemon[1266]: time=2026-09-22T14:52:14.801Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6346worker2 # [ 114.896984] nix-grpc-daemon[1266]: time=2026-09-22T14:52:14.803Z level=INFO msg="Uploading bna0adh7g5jsp1sgpmaqnl7brapllfgc-farm-top (240B)"6347lb # [ 114.940958] niks3-server[1099]: 2026/09/22 14:52:14 INFO Registered completed upload object_key=nar/0bglib3l3asq436b8p5hj4yyk8c77b2qg09pcdiw1w2dfpvnd026.nar.zst6348lb # [ 114.950154] niks3-server[1099]: 2026/09/22 14:52:14 INFO Received sign narinfos request method=POST path=/api/pending_closures/48/sign6349lb # [ 114.959558] niks3-server[1099]: 2026/09/22 14:52:14 INFO Signed narinfos id=48 count=16350worker2 # [ 114.946596] nix-grpc-daemon[1266]: time=2026-09-22T14:52:14.852Z level=INFO msg="Uploading 1 narinfos"6351lb # [ 114.963613] niks3-server[1099]: 2026/09/22 14:52:14 INFO Registered completed upload object_key=log/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv6352lb # [ 114.967942] niks3-server[1099]: 2026/09/22 14:52:14 INFO Registered completed upload object_key=bna0adh7g5jsp1sgpmaqnl7brapllfgc.ls6353lb # [ 114.977806] niks3-server[1099]: 2026/09/22 14:52:14 INFO Received complete upload request method=POST path=/api/pending_closures/48/complete6354lb # [ 114.981641] niks3-server[1099]: 2026/09/22 14:52:14 INFO Completed upload id=486355lb # [ 114.983204] niks3-server[1099]: 2026/09/22 14:52:14 INFO Registered completed upload object_key=bna0adh7g5jsp1sgpmaqnl7brapllfgc.narinfo6356worker2 # [ 114.966618] nix-grpc-daemon[1266]: time=2026-09-22T14:52:14.873Z level=INFO msg="Upload complete. (210ms)"6357worker2 # [ 114.974805] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:14Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35844 duration_s=1 drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv assign_id=3 outputs=16358worker2 # [ 114.988765] nix-daemon[850]: reaped child process 1497, status = succeeded6359client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag on-standby >&2, in 5.48 seconds)6360client: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag on-standby outPath6361client: (finished: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag on-standby outPath, in 0.12 seconds)6362worker1 # [ 115.163750] systemd[1]: Listening on Nix worker-protocol over gRPC.6363worker1 # [ 115.172197] systemd[1]: Starting Nix worker-protocol over gRPC...6364worker1 # [ 115.271823] nix-grpc-daemon[1480]: ts=2026-09-22T14:52:15Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6365worker1 # [ 115.276301] nix-grpc-daemon[1480]: ts=2026-09-22T14:52:15Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6366worker1 # [ 115.281206] nix-grpc-daemon[1480]: ts=2026-09-22T14:52:15Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=16367worker1 # [ 115.286153] nix-grpc-daemon[1480]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6368worker1 # [ 115.290558] nix-grpc-daemon[1480]: E0000 00:00:1790088735.396825 1480 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6369worker1 # [ 115.293359] nix-grpc-daemon[1480]: E0000 00:00:1790088735.398206 1480 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6370worker1 # [ 115.298407] nix-grpc-daemon[1480]: ts=2026-09-22T14:52:15Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6371worker1 # [ 115.301253] systemd[1]: Started Nix worker-protocol over gRPC.6372worker1 # [ 115.306176] nix-grpc-daemon[1480]: E0000 00:00:1790088735.413824 1521 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6373worker1 # [ 115.312542] nix-grpc-daemon[1480]: E0000 00:00:1790088735.429725 1521 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6374worker1 # [ 115.316152] nix-grpc-daemon[1480]: E0000 00:00:1790088735.432063 1521 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6375lb: must succeed: curl -sf localhost:9901/clusters6376worker1 # [ 115.328065] nix-grpc-daemon[1480]: E0000 00:00:1790088735.445279 1521 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6377worker1 # [ 115.339506] nix-grpc-daemon[1480]: ts=2026-09-22T14:52:15Z level=info event=scheduler_connected addr=lb:500516378worker2 # [ 115.358747] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:15Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:558946379worker2 # [ 115.363555] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:15Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06380lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)6381worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6382worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)6383lb: must succeed: curl -sf localhost:9901/clusters6384lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)6385lb: must succeed: curl -sf localhost:9901/clusters6386lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)6387worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6388worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)6389(finished: subtest: scheduler down: the other node takes the lock and keeps it, in 10.54 seconds)6390subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise6391client: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag rs6392client # [ 115.569259] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key" --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag rs.6393client # Running as unit: rs.service; invocation ID: 5bc332ff128e40f39753583909a457aa6394client: (finished: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag rs, in 0.06 seconds)6395worker2 # [ 115.696307] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:15Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:358446396client # [ 115.714457] nix[1360]: don't know how to build these paths:6397client # [ 115.715460] nix[1360]: /nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv6398worker2 # [ 115.708996] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:15Z level=debug event=want client=2 drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv cached=06399worker2 # [ 115.712048] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:15Z level=debug event=assigned drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv worker=192.168.1.4:50051 clients=16400worker2 # [ 115.752430] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:15Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:358446401worker2 # [ 115.760937] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:15Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:358446402client # [ 115.779204] nix[1360]: copying 1 paths to 'lb:50051'...6403worker2 # [ 115.768347] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:15Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:358446404client # [ 115.780955] nix[1360]: copying path '/nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv' to 'lb:50051'...6405worker2 # [ 115.775968] nix-daemon[850]: accepted connection from pid 1206, user nix-grpc-daemon (trusted)6406lb # [ 115.800862] envoy[905]: conn peer=[2001:db8:1::3]:35628 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=751 tx=265 ms=103756407lb # [ 115.806602] envoy[905]: conn peer=[2001:db8:1::1]:45696 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9603 tx=4418 ms=53096408worker2 # [ 115.873747] nix-daemon[850]: accepted connection from pid 1531, user nix-grpc-daemon (trusted)6409worker2 # [ 115.888567] nix-daemon[850]: reaped child process 1534, status = succeeded6410worker2 # [ 115.966426] nix-daemon[850]: accepted connection from pid 1536, user nix-grpc-daemon (trusted)6411worker2 # [ 115.980587] nix-daemon[850]: reaped child process 1539, status = succeeded6412lb # [ 116.002358] niks3-server[1099]: 2026/09/22 14:52:15 INFO Received uploads request method=POST path=/api/pending_closures6413worker2 # [ 115.994605] nix-grpc-daemon[1266]: time=2026-09-22T14:52:15.900Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6414worker2 # [ 115.996489] nix-grpc-daemon[1266]: time=2026-09-22T14:52:15.902Z level=INFO msg="Uploading 2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv (416B)"6415lb # [ 116.031139] niks3-server[1099]: 2026/09/22 14:52:15 INFO Received sign narinfos request method=POST path=/api/pending_closures/49/sign6416lb # [ 116.039748] niks3-server[1099]: 2026/09/22 14:52:15 INFO Signed narinfos id=49 count=16417worker2 # [ 116.027536] nix-grpc-daemon[1266]: time=2026-09-22T14:52:15.933Z level=INFO msg="Uploading 1 narinfos"6418lb # [ 116.045706] niks3-server[1099]: 2026/09/22 14:52:15 INFO Registered completed upload object_key=nar/1p73bzh7zqz68fm48xgidr9p1hfd08x3vp8znnaaycsmagbr8hl6.nar.zst6419lb # [ 116.051630] niks3-server[1099]: 2026/09/22 14:52:15 INFO Registered completed upload object_key=2765168rrfa4dbs7pfx66j88710jx2fm.ls6420lb # [ 116.062243] niks3-server[1099]: 2026/09/22 14:52:15 INFO Received complete upload request method=POST path=/api/pending_closures/49/complete6421lb # [ 116.068319] niks3-server[1099]: 2026/09/22 14:52:15 INFO Completed upload id=496422worker2 # [ 116.052652] nix-grpc-daemon[1266]: time=2026-09-22T14:52:15.959Z level=INFO msg="Upload complete. (269ms)"6423worker2 # [ 116.056126] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:15Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35844 duration_s=0 paths=1 nar_bytes_in=4166424lb # [ 116.072360] niks3-server[1099]: 2026/09/22 14:52:15 INFO Registered completed upload object_key=2765168rrfa4dbs7pfx66j88710jx2fm.narinfo6425worker2 # [ 116.060477] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:15Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:358446426worker2 # [ 116.066071] nix-daemon[850]: accepted connection from pid 1206, user nix-grpc-daemon (trusted)6427worker2 # [ 116.068306] nix-daemon[850]: reaped child process 1529, status = succeeded6428worker2 # [ 116.073478] nix-daemon[850]: accepted connection from pid 1206, user nix-grpc-daemon (trusted)6429client # [ 116.084532] nix[1360]: building '/nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv' on lb:50051...6430lb: must succeed: curl -sf localhost:9901/clusters6431lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)6432lb: must succeed: curl -sf localhost:9901/clusters6433lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)6434lb: must succeed: curl -sf localhost:9901/clusters6435lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)6436lb: must succeed: curl -sf localhost:9901/clusters6437lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)6438worker1: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true6439worker1 # [ 118.131142] systemd-journald[283]: Received client request to rotate journal, rotating.6440worker1 # [ 118.149953] systemd-journald[283]: Vacuuming done, freed 0B of archived journals from /var/log/journal/14fe4d04cc2945fa960c7c3d6339bf16.6441worker1: (finished: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true, in 0.06 seconds)6442worker2: must succeed: systemctl restart --no-block nix-grpc-daemon.service6443worker2 # [ 118.082741] systemd[1]: Stopping Nix worker-protocol over gRPC...6444worker2: (finished: must succeed: systemctl restart --no-block nix-grpc-daemon.service, in 0.04 seconds)6445client: must succeed: sleep 5.06446worker2 # [ 118.258842] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:18Z level=info event=draining6447client: (finished: must succeed: sleep 5.0, in 5.03 seconds)6448worker1: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr='6449worker2 # [ 123.526373] nix-daemon[850]: reaped child process 1543, status = succeeded6450worker2 # [ 123.601257] nix-daemon[850]: accepted connection from pid 1584, user nix-grpc-daemon (trusted)6451worker2 # [ 123.614491] nix-daemon[850]: reaped child process 1587, status = succeeded6452lb # [ 123.635904] niks3-server[1099]: 2026/09/22 14:52:23 INFO Received uploads request method=POST path=/api/pending_closures6453worker2 # [ 123.624953] nix-grpc-daemon[1266]: time=2026-09-22T14:52:23.531Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6454worker2 # [ 123.626410] nix-grpc-daemon[1266]: time=2026-09-22T14:52:23.532Z level=INFO msg="Uploading 6qby8qgv9vwglq2yryw4zqyixrizh5d2-slow-rs (120B)"6455lb # [ 123.658972] niks3-server[1099]: 2026/09/22 14:52:23 INFO Registered completed upload object_key=nar/19z3qdigvc4gggg31dgc0wi5w85pgaq9dk7bvngh0yhckpy55jgq.nar.zst6456lb # [ 123.662449] niks3-server[1099]: 2026/09/22 14:52:23 INFO Received sign narinfos request method=POST path=/api/pending_closures/50/sign6457lb # [ 123.664323] niks3-server[1099]: 2026/09/22 14:52:23 INFO Registered completed upload object_key=log/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv6458lb # [ 123.667864] niks3-server[1099]: 2026/09/22 14:52:23 INFO Signed narinfos id=50 count=16459worker2 # [ 123.652377] nix-grpc-daemon[1266]: time=2026-09-22T14:52:23.558Z level=INFO msg="Uploading 1 narinfos"6460lb # [ 123.670650] niks3-server[1099]: 2026/09/22 14:52:23 INFO Registered completed upload object_key=6qby8qgv9vwglq2yryw4zqyixrizh5d2.ls6461lb # [ 123.677566] niks3-server[1099]: 2026/09/22 14:52:23 INFO Received complete upload request method=POST path=/api/pending_closures/50/complete6462lb # [ 123.680050] niks3-server[1099]: 2026/09/22 14:52:23 INFO Completed upload id=506463worker2 # [ 123.664470] nix-grpc-daemon[1266]: time=2026-09-22T14:52:23.570Z level=INFO msg="Upload complete. (176ms)"6464lb # [ 123.682095] niks3-server[1099]: 2026/09/22 14:52:23 INFO Registered completed upload object_key=6qby8qgv9vwglq2yryw4zqyixrizh5d2.narinfo6465worker2 # [ 123.666962] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:23Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35844 duration_s=7 drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv assign_id=4 outputs=16466worker2 # [ 123.675412] nix-daemon[850]: reaped child process 1541, status = succeeded6467client # [ 123.712333] systemd[1]: rs.service: Deactivated successfully.6468client # [ 123.716463] systemd[1]: rs.service: Consumed 137ms CPU time over 8.146s wall clock time, 41.4M memory peak, 7.1K incoming IP traffic, 8.2K outgoing IP traffic.6469worker2 # [ 124.260703] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:24Z level=info event=signal_exit6470lb # [ 125.212829] niks3-server[1099]: 2026/09/22 14:52:24 INFO lead: released remote=192.168.1.4:371186471lb # [ 125.327182] niks3-server[1099]: 2026/09/22 14:52:24 INFO lead: acquired remote=192.168.1.3:529406472worker1 # [ 125.297157] nix-grpc-daemon[1480]: ts=2026-09-22T14:52:25Z level=info event=scheduler_take_over6473lb # [ 125.801841] envoy[905]: conn peer=[2001:db8:1::1]:37268 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2943 tx=1522 ms=79886474worker1 # [ 126.177707] nix-grpc-daemon[1480]: ts=2026-09-22T14:52:26Z level=info event=scheduler_restarting addr=lb:500516475worker2 # [ 126.200380] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:26Z level=info event=scheduler_restarting6476worker2 # [ 126.203931] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:26Z level=info event=scheduler_restarting addr=lb:500516477worker2 # [ 126.207643] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:26Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:558946478worker2 # [ 126.211769] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:26Z level=info event=worker_gone addr=192.168.1.3:500516479worker2 # [ 126.214956] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:26Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:558946480worker2 # [ 126.218480] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:26Z level=info event=worker_gone addr=192.168.1.4:500516481worker2 # [ 126.238253] nix-daemon[850]: reaped child process 1258, status = succeeded6482worker2 # [ 126.321493] nix-grpc-daemon[1206]: E0000 00:00:1790088746.227848 1251 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 18, started true, shutdown true6483worker2 # [ 126.323599] nix-grpc-daemon[1206]: E0000 00:00:1790088746.227988 1251 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 9, started true, shutdown true6484worker1 # [ 126.380624] nix-grpc-daemon[1480]: ts=2026-09-22T14:52:26Z level=info event=scheduler_connected addr=lb:500516485worker1 # [ 126.387937] nix-grpc-daemon[1480]: ts=2026-09-22T14:52:26Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:422226486worker2 # [ 126.411252] nix-grpc-daemon[1206]: ts=2026-09-22T14:52:26Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"6487worker1 # [ 126.392220] nix-grpc-daemon[1480]: ts=2026-09-22T14:52:26Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06488worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr=', in 3.14 seconds)6489client: waiting for success: ! systemctl is-active rs6490client: (finished: waiting for success: ! systemctl is-active rs, in 0.04 seconds)6491client: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }6492client: (finished: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }, in 0.03 seconds)6493client: must fail: journalctl -u rs | grep -q 'reconnecting'6494client: (finished: must fail: journalctl -u rs | grep -q 'reconnecting', in 0.01 seconds)6495lb: must succeed: curl -sf localhost:9901/clusters6496lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6497worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6498worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)6499worker2 # [ 127.446245] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.6500worker2 # [ 127.448245] systemd[1]: Stopped Nix worker-protocol over gRPC.6501worker2 # [ 127.449101] systemd[1]: nix-grpc-daemon.service: Consumed 2.145s CPU time over 35.300s wall clock time, 47.1M memory peak, 83.7K incoming IP traffic, 189K outgoing IP traffic.6502worker2 # [ 127.455564] systemd[1]: Starting scheduler-token.service...6503worker2 # [ 127.492876] systemd[1]: scheduler-token.service: Deactivated successfully.6504worker2 # [ 127.494426] systemd[1]: Finished scheduler-token.service.6505worker2 # [ 127.495779] systemd[1]: scheduler-token.service: Consumed 17ms CPU time over 38ms wall clock time, 2.3M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.6506worker2 # [ 127.500167] systemd[1]: Starting Nix worker-protocol over gRPC...6507worker2 # [ 127.577853] nix-grpc-daemon[1597]: ts=2026-09-22T14:52:27Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6508worker2 # [ 127.580146] nix-grpc-daemon[1597]: ts=2026-09-22T14:52:27Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6509worker2 # [ 127.584595] nix-grpc-daemon[1597]: ts=2026-09-22T14:52:27Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=16510worker2 # [ 127.588058] nix-grpc-daemon[1597]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6511worker2 # [ 127.589366] nix-grpc-daemon[1597]: E0000 00:00:1790088747.493809 1597 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6512worker2 # [ 127.595689] nix-grpc-daemon[1597]: E0000 00:00:1790088747.498809 1597 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6513worker2 # [ 127.598594] nix-grpc-daemon[1597]: ts=2026-09-22T14:52:27Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6514worker2 # [ 127.601324] systemd[1]: Started Nix worker-protocol over gRPC.6515worker2 # [ 127.603207] nix-grpc-daemon[1597]: E0000 00:00:1790088747.508793 1637 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6516worker2 # [ 127.614331] nix-grpc-daemon[1597]: E0000 00:00:1790088747.520818 1637 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6517worker2 # [ 127.619055] nix-grpc-daemon[1597]: E0000 00:00:1790088747.523087 1637 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6518lb: must succeed: curl -sf localhost:9901/clusters6519worker2 # [ 127.628425] nix-grpc-daemon[1597]: E0000 00:00:1790088747.534925 1637 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6520worker2 # [ 127.639414] nix-grpc-daemon[1597]: ts=2026-09-22T14:52:27Z level=info event=scheduler_connected addr=lb:500516521worker1 # [ 127.630307] nix-grpc-daemon[1480]: ts=2026-09-22T14:52:27Z level=info event=oidc_jwks provider=mock keys=16522worker1 # [ 127.634992] nix-grpc-daemon[1480]: ts=2026-09-22T14:52:27Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:422226523worker1 # [ 127.638973] nix-grpc-daemon[1480]: ts=2026-09-22T14:52:27Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06524lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6525worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6526worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.03 seconds)6527(finished: subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise, in 12.19 seconds)6528subtest: scheduler restart mid-build: build finishes, no double build6529client: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag mid6530client # [ 127.761638] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key" --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag mid.6531client # Running as unit: mid.service; invocation ID: 3ab38ac012ff443887ecf4c4d91683b96532client: (finished: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag mid, in 0.06 seconds)6533worker1 # [ 127.871472] nix-grpc-daemon[1480]: ts=2026-09-22T14:52:27Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:422306534worker1 # [ 127.876536] nix-daemon[869]: accepted connection from pid 1480, user nix-grpc-daemon (trusted)6535client # [ 127.913139] nix[1396]: don't know how to build these paths:6536client # [ 127.914259] nix[1396]: /nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv6537worker1 # [ 127.887664] nix-grpc-daemon[1480]: ts=2026-09-22T14:52:28Z level=debug event=want client=1 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv cached=06538worker1 # [ 127.889515] nix-grpc-daemon[1480]: ts=2026-09-22T14:52:28Z level=debug event=assigned drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv worker=192.168.1.3:50051 clients=16539worker1 # [ 127.934795] nix-grpc-daemon[1480]: ts=2026-09-22T14:52:28Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:422306540worker1 # [ 127.939503] nix-grpc-daemon[1480]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 20 ms (attempt 1/5)6541worker1 # [ 127.962405] nix-grpc-daemon[1480]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 107 ms (attempt 2/5)6542worker1 # [ 128.070494] nix-grpc-daemon[1480]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 374 ms (attempt 3/5)6543worker1 # [ 128.447492] nix-grpc-daemon[1480]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 91 ms (attempt 4/5)6544worker1 # [ 128.540245] nix-grpc-daemon[1480]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6545worker1 # [ 128.543644] nix-grpc-daemon[1480]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6546worker1 # [ 128.557130] nix-grpc-daemon[1480]: ts=2026-09-22T14:52:28Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:422306547client # [ 128.592146] nix[1396]: copying 1 paths to 'lb:50051'...6548worker1 # [ 128.562363] nix-grpc-daemon[1480]: ts=2026-09-22T14:52:28Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:422306549client # [ 128.593236] nix[1396]: copying path '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' to 'lb:50051'...6550worker1 # [ 128.568446] nix-daemon[869]: accepted connection from pid 1480, user nix-grpc-daemon (trusted)6551worker1 # [ 128.583305] nix-grpc-daemon[1480]: ts=2026-09-22T14:52:28Z level=info event=niks3_push_started pid=16086552worker1 # [ 128.675606] nix-grpc-daemon[1608]: time=2026-09-22T14:52:28.792Z level=INFO msg="Configuring client TLS" cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker1.pem key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker1.key ca=""6553worker1 # [ 128.760240] nix-daemon[869]: accepted connection from pid 1621, user nix-grpc-daemon (trusted)6554worker1 # [ 128.772848] nix-daemon[869]: reaped child process 1624, status = succeeded6555worker1 # [ 128.865168] nix-daemon[869]: accepted connection from pid 1626, user nix-grpc-daemon (trusted)6556worker1 # [ 128.885953] nix-daemon[869]: reaped child process 1633, status = succeeded6557lb # [ 128.923387] niks3-server[1099]: 2026/09/22 14:52:28 INFO Received uploads request method=POST path=/api/pending_closures6558worker1 # [ 128.898180] nix-grpc-daemon[1608]: time=2026-09-22T14:52:29.015Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6559worker1 # [ 128.899746] nix-grpc-daemon[1608]: time=2026-09-22T14:52:29.017Z level=INFO msg="Uploading n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv (424B)"6560lb # [ 128.958956] niks3-server[1099]: 2026/09/22 14:52:28 INFO Registered completed upload object_key=nar/0pdyhjlvyi83rdx6rr15l7kj7a7hlcbqgllxm29xw3hvr3akprgq.nar.zst6561lb # [ 128.965768] niks3-server[1099]: 2026/09/22 14:52:28 INFO Received sign narinfos request method=POST path=/api/pending_closures/51/sign6562worker1 # [ 128.937563] nix-grpc-daemon[1608]: time=2026-09-22T14:52:29.054Z level=INFO msg="Uploading 1 narinfos"6563lb # [ 128.973220] niks3-server[1099]: 2026/09/22 14:52:28 INFO Signed narinfos id=51 count=16564lb # [ 128.982874] niks3-server[1099]: 2026/09/22 14:52:28 INFO Registered completed upload object_key=n23kf6h4k511p0qpzsydmqrbq2qn9g5w.ls6565lb # [ 128.989115] niks3-server[1099]: 2026/09/22 14:52:28 INFO Received complete upload request method=POST path=/api/pending_closures/51/complete6566lb # [ 128.996803] niks3-server[1099]: 2026/09/22 14:52:28 INFO Completed upload id=516567worker1 # [ 128.963483] nix-grpc-daemon[1608]: time=2026-09-22T14:52:29.079Z level=INFO msg="Upload complete. (283ms)"6568worker1 # [ 128.965607] nix-grpc-daemon[1480]: ts=2026-09-22T14:52:29Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42230 duration_s=0 paths=1 nar_bytes_in=4246569lb # [ 129.001595] niks3-server[1099]: 2026/09/22 14:52:28 INFO Registered completed upload object_key=n23kf6h4k511p0qpzsydmqrbq2qn9g5w.narinfo6570worker1 # [ 128.969189] nix-grpc-daemon[1480]: ts=2026-09-22T14:52:29Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:422306571worker1 # [ 128.976967] nix-daemon[869]: accepted connection from pid 1480, user nix-grpc-daemon (trusted)6572worker1 # [ 128.982143] nix-daemon[869]: reaped child process 1606, status = succeeded6573worker1 # [ 128.986998] nix-daemon[869]: accepted connection from pid 1480, user nix-grpc-daemon (trusted)6574client # [ 129.016407] nix[1396]: building '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' on lb:50051...6575lb: must succeed: curl -sf localhost:9901/clusters6576lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)6577lb: must succeed: curl -sf localhost:9901/clusters6578lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)6579worker1: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service6580worker1 # [ 130.160322] systemd[1]: nix-grpc-daemon.service: Sent signal SIGKILL to main process 1480 (nix-grpc-daemon) on client request.6581worker1 # [ 130.162319] systemd[1]: nix-grpc-daemon.service: Killed unit cgroup '/system.slice/nix-grpc-daemon.service' with SIGKILL on client request.6582lb # [ 130.202619] niks3-server[1099]: 2026/09/22 14:52:29 INFO lead: released remote=192.168.1.3:529406583client # [ 130.201726] nix[1396]: n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), asking the scheduler again6584client # [ 130.204136] nix[1396]: scheduler at lb:50051: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), reconnecting6585worker2 # [ 130.190891] nix-grpc-daemon[1597]: ts=2026-09-22T14:52:30Z level=info event=scheduler_disconnected addr=lb:50051 error="upstream connect error or disconnect/reset before headers. reset reason: connection termination"6586worker1 # [ 130.177176] systemd[1]: nix-grpc-daemon.service: Main process exited, code=killed, status=9/KILL6587worker1 # [ 130.178610] systemd[1]: nix-grpc-daemon.service: Failed with result 'signal'.6588worker1 # [ 130.179844] systemd[1]: nix-grpc-daemon.service: Consumed 412ms CPU time over 15.001s wall clock time, 35.6M memory peak, 14.5K incoming IP traffic, 26.1K outgoing IP traffic.6589worker1 # [ 130.194790] systemd[1]: nix-grpc-daemon.service: Scheduled restart job immediately on client request, restart counter is at 1.6590worker1 # [ 130.200964] systemd[1]: Starting Nix worker-protocol over gRPC...6591worker1 # [ 130.212453] nix-daemon[869]: reaped child process 1635, status = succeeded6592worker1 # [ 130.216161] nix-daemon[869]: reaped child process 1598, status = succeeded6593worker1 # [ 130.223494] nix-daemon[1637]: unexpected Nix daemon error: error: interrupted by the user6594worker1 # [ 130.234977] nix-daemon[869]: reaped child process 1637, status = failed with exit code 16595worker1 # [ 130.286351] nix-grpc-daemon[1670]: ts=2026-09-22T14:52:30Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6596worker1 # [ 130.288550] nix-grpc-daemon[1670]: ts=2026-09-22T14:52:30Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6597worker1 # [ 130.291789] nix-grpc-daemon[1670]: ts=2026-09-22T14:52:30Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=16598worker1 # [ 130.296057] nix-grpc-daemon[1670]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6599worker1 # [ 130.298169] nix-grpc-daemon[1670]: E0000 00:00:1790088750.413235 1670 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6600worker1 # [ 130.305198] nix-grpc-daemon[1670]: E0000 00:00:1790088750.418826 1670 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6601worker1 # [ 130.308167] nix-grpc-daemon[1670]: ts=2026-09-22T14:52:30Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6602worker1 # [ 130.309786] systemd[1]: Started Nix worker-protocol over gRPC.6603lb # [ 130.347455] niks3-server[1099]: 2026/09/22 14:52:29 INFO lead: acquired remote=192.168.1.3:514706604worker1 # [ 130.314080] nix-grpc-daemon[1670]: ts=2026-09-22T14:52:30Z level=info event=scheduler_take_over6605worker1 # [ 130.315183] nix-grpc-daemon[1670]: E0000 00:00:1790088750.430753 1711 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6606worker1: (finished: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service, in 0.19 seconds)6607client: must succeed: sleep 5.06608worker1 # [ 130.327673] nix-grpc-daemon[1670]: E0000 00:00:1790088750.444893 1711 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6609worker1 # [ 130.331136] nix-grpc-daemon[1670]: E0000 00:00:1790088750.447068 1711 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6610worker1 # [ 130.341678] nix-grpc-daemon[1670]: E0000 00:00:1790088750.458919 1711 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6611worker1 # [ 130.349039] nix-grpc-daemon[1670]: ts=2026-09-22T14:52:30Z level=info event=scheduler_connected addr=lb:500516612worker1 # [ 130.353576] nix-grpc-daemon[1670]: ts=2026-09-22T14:52:30Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:422406613worker1 # [ 130.355048] nix-grpc-daemon[1670]: ts=2026-09-22T14:52:30Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06614worker2 # [ 130.391492] nix-grpc-daemon[1597]: ts=2026-09-22T14:52:30Z level=info event=scheduler_connected addr=lb:500516615worker1 # [ 130.376062] nix-grpc-daemon[1670]: ts=2026-09-22T14:52:30Z level=info event=oidc_jwks provider=mock keys=16616worker1 # [ 130.377421] nix-grpc-daemon[1670]: ts=2026-09-22T14:52:30Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:422406617worker1 # [ 130.379035] nix-grpc-daemon[1670]: ts=2026-09-22T14:52:30Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06618worker1 # [ 130.682851] nix-grpc-daemon[1670]: ts=2026-09-22T14:52:30Z level=debug event=want client=1 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv cached=06619worker1 # [ 130.688322] nix-grpc-daemon[1670]: ts=2026-09-22T14:52:30Z level=debug event=assigned drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv worker=192.168.1.3:50051 clients=16620worker1 # [ 130.730156] nix-grpc-daemon[1670]: ts=2026-09-22T14:52:30Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:422826621worker1 # [ 130.739298] nix-daemon[869]: accepted connection from pid 1670, user nix-grpc-daemon (trusted)6622worker1 # [ 130.754841] nix-daemon[869]: accepted connection from pid 1670, user nix-grpc-daemon (trusted)6623client # [ 130.797605] nix[1396]: building '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' on lb:50051...6624worker1 # [ 130.769418] nix-daemon[869]: accepted connection from pid 1670, user nix-grpc-daemon (trusted)6625client: (finished: must succeed: sleep 5.0, in 5.02 seconds)6626client: waiting for success: ! systemctl is-active mid6627worker1 # [ 135.590823] nix-grpc-daemon[1670]: ts=2026-09-22T14:52:35Z level=info event=niks3_push_started pid=17536628worker1 # [ 135.631084] nix-daemon[869]: reaped child process 1728, status = succeeded6629worker1 # [ 135.711555] nix-grpc-daemon[1753]: time=2026-09-22T14:52:35.828Z level=INFO msg="Configuring client TLS" cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker1.pem key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker1.key ca=""6630lb # [ 135.803130] envoy[905]: conn peer=[2001:db8:1::4]:51306 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=31475 tx=2001 ms=341126631lb # [ 135.807248] envoy[905]: conn peer=[2001:db8:1::3]:33536 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=794 tx=311 ms=148386632worker1 # [ 135.796746] nix-daemon[869]: accepted connection from pid 1766, user nix-grpc-daemon (trusted)6633worker1 # [ 135.809473] nix-daemon[869]: reaped child process 1769, status = succeeded6634lb # [ 135.852405] niks3-server[1099]: 2026/09/22 14:52:35 INFO Received uploads request method=POST path=/api/pending_closures6635worker1 # [ 135.826270] nix-grpc-daemon[1753]: time=2026-09-22T14:52:35.943Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6636worker1 # [ 135.827898] nix-grpc-daemon[1753]: time=2026-09-22T14:52:35.945Z level=INFO msg="Uploading cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj-slow-mid (120B)"6637lb # [ 135.899180] niks3-server[1099]: 2026/09/22 14:52:35 INFO Registered completed upload object_key=log/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv6638lb # [ 135.919935] niks3-server[1099]: 2026/09/22 14:52:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/52/sign6639worker1 # [ 135.889310] nix-grpc-daemon[1753]: time=2026-09-22T14:52:36.006Z level=INFO msg="Uploading 1 narinfos"6640lb # [ 135.924331] niks3-server[1099]: 2026/09/22 14:52:35 INFO Signed narinfos id=52 count=16641lb # [ 135.936560] niks3-server[1099]: 2026/09/22 14:52:35 INFO Registered completed upload object_key=nar/0462jb7zwy2d73k3q6v09xnk3bx318v8ik5ckmp32w2hzvwj4k2d.nar.zst6642lb # [ 135.939629] niks3-server[1099]: 2026/09/22 14:52:35 INFO Received complete upload request method=POST path=/api/pending_closures/52/complete6643lb # [ 135.946653] niks3-server[1099]: 2026/09/22 14:52:35 INFO Registered completed upload object_key=cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj.ls6644lb # [ 135.949362] niks3-server[1099]: 2026/09/22 14:52:35 INFO Completed upload id=526645worker1 # [ 135.915316] nix-grpc-daemon[1753]: time=2026-09-22T14:52:36.032Z level=INFO msg="Upload complete. (200ms)"6646lb # [ 135.953098] niks3-server[1099]: 2026/09/22 14:52:35 INFO Registered completed upload object_key=cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj.narinfo6647worker1 # [ 135.924128] nix-grpc-daemon[1670]: ts=2026-09-22T14:52:36Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42282 duration_s=5 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv assign_id=1 outputs=16648worker1 # [ 135.936958] nix-daemon[869]: reaped child process 1726, status = succeeded6649client # [ 135.990441] systemd[1]: mid.service: Deactivated successfully.6650client # [ 135.994184] systemd[1]: mid.service: Consumed 145ms CPU time over 8.232s wall clock time, 43.4M memory peak, 8.7K incoming IP traffic, 10.4K outgoing IP traffic.6651client: (finished: waiting for success: ! systemctl is-active mid, in 1.08 seconds)6652client: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }6653client: (finished: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }, in 0.03 seconds)6654(finished: subtest: scheduler restart mid-build: build finishes, no double build, in 8.90 seconds)6655subtest: interrupting the client stops the build on the worker6656client: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag intr6657client # [ 136.635174] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key" --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag intr.6658client # Running as unit: intr.service; invocation ID: aec4b87d301d4fb9ac6554210cde21df6659client: (finished: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag intr, in 0.03 seconds)6660worker1 # [ 136.733639] nix-grpc-daemon[1670]: ts=2026-09-22T14:52:36Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:422826661client # [ 136.769988] nix[1432]: don't know how to build these paths:6662client # [ 136.770959] nix[1432]: /nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv6663worker1 # [ 136.746676] nix-grpc-daemon[1670]: ts=2026-09-22T14:52:36Z level=debug event=want client=2 drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv cached=06664worker1 # [ 136.750921] nix-grpc-daemon[1670]: ts=2026-09-22T14:52:36Z level=debug event=assigned drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv worker=192.168.1.4:50051 clients=16665worker2 # [ 136.815858] nix-grpc-daemon[1597]: ts=2026-09-22T14:52:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:367046666worker2 # [ 136.827374] nix-daemon[850]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)6667worker2 # [ 136.843622] nix-grpc-daemon[1597]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 31 ms (attempt 1/5)6668worker2 # [ 136.882134] nix-grpc-daemon[1597]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 29 ms (attempt 2/5)6669worker2 # [ 136.912912] nix-grpc-daemon[1597]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 217 ms (attempt 3/5)6670worker2 # [ 137.132254] nix-grpc-daemon[1597]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 259 ms (attempt 4/5)6671worker2 # [ 137.393587] nix-grpc-daemon[1597]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6672worker2 # [ 137.397465] nix-grpc-daemon[1597]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6673worker2 # [ 137.414126] nix-grpc-daemon[1597]: ts=2026-09-22T14:52:37Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:367046674client # [ 137.435570] nix[1432]: copying 1 paths to 'lb:50051'...6675client # [ 137.440173] nix[1432]: copying path '/nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv' to 'lb:50051'...6676worker2 # [ 137.428394] nix-grpc-daemon[1597]: ts=2026-09-22T14:52:37Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:367046677worker2 # [ 137.439097] nix-daemon[850]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)6678worker2 # [ 137.458685] nix-grpc-daemon[1597]: ts=2026-09-22T14:52:37Z level=info event=niks3_push_started pid=16856679worker2 # [ 137.572835] nix-grpc-daemon[1685]: time=2026-09-22T14:52:37.478Z level=INFO msg="Configuring client TLS" cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker2.pem key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker2.key ca=""6680worker2 # [ 137.660153] nix-daemon[850]: accepted connection from pid 1698, user nix-grpc-daemon (trusted)6681worker2 # [ 137.673511] nix-daemon[850]: reaped child process 1701, status = succeeded6682worker2 # [ 137.755988] nix-daemon[850]: accepted connection from pid 1703, user nix-grpc-daemon (trusted)6683worker2 # [ 137.769608] nix-daemon[850]: reaped child process 1706, status = succeeded6684lb # [ 137.792227] niks3-server[1099]: 2026/09/22 14:52:37 INFO Received uploads request method=POST path=/api/pending_closures6685worker2 # [ 137.787465] nix-grpc-daemon[1685]: time=2026-09-22T14:52:37.693Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6686worker2 # [ 137.790675] nix-grpc-daemon[1685]: time=2026-09-22T14:52:37.693Z level=INFO msg="Uploading x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv (424B)"6687lb # [ 137.828739] niks3-server[1099]: 2026/09/22 14:52:37 INFO Registered completed upload object_key=nar/0dw6lfdicxarqmcw4m9j94irjhgkdvc1nww51zwj454yj03v5iri.nar.zst6688lb # [ 137.838778] niks3-server[1099]: 2026/09/22 14:52:37 INFO Received sign narinfos request method=POST path=/api/pending_closures/53/sign6689lb # [ 137.846228] niks3-server[1099]: 2026/09/22 14:52:37 INFO Signed narinfos id=53 count=16690worker2 # [ 137.833734] nix-grpc-daemon[1685]: time=2026-09-22T14:52:37.739Z level=INFO msg="Uploading 1 narinfos"6691lb # [ 137.857496] niks3-server[1099]: 2026/09/22 14:52:37 INFO Registered completed upload object_key=x43jxvh618x0k4fbhc2wyxhr1lymdssr.ls6692lb # [ 137.867420] niks3-server[1099]: 2026/09/22 14:52:37 INFO Received complete upload request method=POST path=/api/pending_closures/53/complete6693lb # [ 137.874846] niks3-server[1099]: 2026/09/22 14:52:37 INFO Registered completed upload object_key=x43jxvh618x0k4fbhc2wyxhr1lymdssr.narinfo6694lb # [ 137.878227] niks3-server[1099]: 2026/09/22 14:52:37 INFO Completed upload id=536695worker2 # [ 137.862731] nix-grpc-daemon[1685]: time=2026-09-22T14:52:37.769Z level=INFO msg="Upload complete. (286ms)"6696worker2 # [ 137.865457] nix-grpc-daemon[1597]: ts=2026-09-22T14:52:37Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36704 duration_s=0 paths=1 nar_bytes_in=4246697worker2 # [ 137.868183] nix-grpc-daemon[1597]: ts=2026-09-22T14:52:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:367046698worker2 # [ 137.873950] nix-daemon[850]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)6699worker2 # [ 137.879123] nix-daemon[850]: reaped child process 1683, status = succeeded6700client # [ 137.894775] nix[1432]: building '/nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv' on lb:50051...6701worker2 # [ 137.883777] nix-daemon[850]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)6702client: must succeed: systemctl kill -s INT intr6703client # [ 138.929307] systemd[1]: intr.service: Sent signal SIGINT to main process 1432 (nix) on client request.6704client # [ 138.932534] nix[1432]: shutting downx43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED)6705worker2 # [ 138.921233] nix-grpc-daemon[1597]: ts=2026-09-22T14:52:38Z level=info event=revoked_running drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv6706client # [ 138.934806] nix[1432]: error: interrupted by the user6707client: (finished: must succeed: systemctl kill -s INT intr, in 0.03 seconds)6708client # [ 138.955166] systemd[1]: intr.service: Main process exited, code=exited, status=1/FAILURE6709client # [ 138.956463] systemd[1]: intr.service: Failed with result 'exit-code'.6710client # [ 138.958627] systemd[1]: intr.service: Consumed 116ms CPU time over 2.319s wall clock time, 41.4M memory peak, 6.5K incoming IP traffic, 8.2K outgoing IP traffic.6711worker2 # [ 139.102586] nix-daemon[850]: reaped child process 1712, status = succeeded6712worker2 # [ 139.105352] nix-daemon[1714]: unexpected Nix daemon error: error: interrupted by the user6713worker2 # [ 139.108955] nix-daemon[850]: reaped child process 1714, status = failed with exit code 16714(finished: subtest: interrupting the client stops the build on the worker, in 3.48 seconds)6715subtest: a deploy mid-build drains: the build finishes, the new generation takes over6716client: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag sw6717client # [ 140.142624] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key" --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag sw.6718client # Running as unit: sw.service; invocation ID: 3151bbaaffbf4274a1a5a8faa465acf16719client: (finished: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag sw, in 0.06 seconds)6720worker1 # [ 140.245803] nix-grpc-daemon[1670]: ts=2026-09-22T14:52:40Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:422826721client # [ 140.282063] nix[1452]: don't know how to build these paths:6722client # [ 140.282958] nix[1452]: /nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv6723worker1 # [ 140.259103] nix-grpc-daemon[1670]: ts=2026-09-22T14:52:40Z level=debug event=want client=3 drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv cached=06724worker1 # [ 140.263366] nix-grpc-daemon[1670]: ts=2026-09-22T14:52:40Z level=debug event=assigned drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv worker=192.168.1.3:50051 clients=16725worker1 # [ 140.301811] nix-grpc-daemon[1670]: ts=2026-09-22T14:52:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:422826726worker1 # [ 140.313989] nix-grpc-daemon[1670]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 43 ms (attempt 1/5)6727worker1 # [ 140.363529] nix-grpc-daemon[1670]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 187 ms (attempt 2/5)6728worker1 # [ 140.551711] nix-grpc-daemon[1670]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 371 ms (attempt 3/5)6729worker1 # [ 140.927320] nix-grpc-daemon[1670]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 779 ms (attempt 4/5)6730worker1 # [ 141.709709] nix-grpc-daemon[1670]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6731worker1 # [ 141.714251] nix-grpc-daemon[1670]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6732worker1 # [ 141.731862] nix-grpc-daemon[1670]: ts=2026-09-22T14:52:41Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:422826733client # [ 141.773772] nix[1452]: copying 1 paths to 'lb:50051'...6734client # [ 141.778248] nix[1452]: copying path '/nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv' to 'lb:50051'...6735worker1 # [ 141.748998] nix-grpc-daemon[1670]: ts=2026-09-22T14:52:41Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:422826736worker1 # [ 141.760973] nix-daemon[869]: accepted connection from pid 1670, user nix-grpc-daemon (trusted)6737worker1 # [ 141.863831] nix-daemon[869]: accepted connection from pid 1809, user nix-grpc-daemon (trusted)6738worker1 # [ 141.882604] nix-daemon[869]: reaped child process 1812, status = succeeded6739worker1 # [ 141.957322] nix-daemon[869]: accepted connection from pid 1814, user nix-grpc-daemon (trusted)6740worker1 # [ 141.970919] nix-daemon[869]: reaped child process 1817, status = succeeded6741lb # [ 142.011676] niks3-server[1099]: 2026/09/22 14:52:41 INFO Received uploads request method=POST path=/api/pending_closures6742worker1 # [ 141.984137] nix-grpc-daemon[1753]: time=2026-09-22T14:52:42.100Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6743worker1 # [ 141.985813] nix-grpc-daemon[1753]: time=2026-09-22T14:52:42.100Z level=INFO msg="Uploading q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv (416B)"6744lb # [ 142.025776] niks3-server[1099]: 2026/09/22 14:52:41 INFO Registered completed upload object_key=nar/1rby34xp94wipnqa4abnh2771jnqp34zv82lymaabrhj07vmwiix.nar.zst6745lb # [ 142.030052] niks3-server[1099]: 2026/09/22 14:52:41 INFO Received sign narinfos request method=POST path=/api/pending_closures/54/sign6746lb # [ 142.031876] niks3-server[1099]: 2026/09/22 14:52:41 INFO Signed narinfos id=54 count=16747worker1 # [ 141.999181] nix-grpc-daemon[1753]: time=2026-09-22T14:52:42.116Z level=INFO msg="Uploading 1 narinfos"6748lb # [ 142.036859] niks3-server[1099]: 2026/09/22 14:52:41 INFO Registered completed upload object_key=q9pl41yvj7mi4fqnyiarzy5py0107h7b.ls6749lb # [ 142.040627] niks3-server[1099]: 2026/09/22 14:52:41 INFO Received complete upload request method=POST path=/api/pending_closures/54/complete6750lb # [ 142.045380] niks3-server[1099]: 2026/09/22 14:52:41 INFO Completed upload id=546751worker1 # [ 142.011070] nix-grpc-daemon[1753]: time=2026-09-22T14:52:42.128Z level=INFO msg="Upload complete. (238ms)"6752lb # [ 142.047286] niks3-server[1099]: 2026/09/22 14:52:41 INFO Registered completed upload object_key=q9pl41yvj7mi4fqnyiarzy5py0107h7b.narinfo6753worker1 # [ 142.013402] nix-grpc-daemon[1670]: ts=2026-09-22T14:52:42Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42282 duration_s=0 paths=1 nar_bytes_in=4166754worker1 # [ 142.017420] nix-grpc-daemon[1670]: ts=2026-09-22T14:52:42Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:422826755worker1 # [ 142.023135] nix-daemon[869]: accepted connection from pid 1670, user nix-grpc-daemon (trusted)6756worker1 # [ 142.025458] nix-daemon[869]: reaped child process 1807, status = succeeded6757worker1 # [ 142.030505] nix-daemon[869]: accepted connection from pid 1670, user nix-grpc-daemon (trusted)6758client # [ 142.060311] nix[1452]: building '/nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv' on lb:50051...6759worker1: must succeed: systemctl reload nix-grpc-daemon.service6760worker1 # [ 143.515163] systemd[1]: Reloading Nix worker-protocol over gRPC...6761worker1 # [ 143.540172] systemd[1]: Reloaded Nix worker-protocol over gRPC.6762worker1: (finished: must succeed: systemctl reload nix-grpc-daemon.service, in 0.06 seconds)6763worker1: must succeed: pgrep -f 'read -t [9]0 x'6764worker1: (finished: must succeed: pgrep -f 'read -t [9]0 x', in 0.03 seconds)6765client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag during-drain >&26766worker1 # [ 143.691791] nix-grpc-daemon[1670]: ts=2026-09-22T14:52:43Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:422826767client # don't know how to build these paths:6768client # /nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6769worker2 # [ 143.717463] nix-grpc-daemon[1597]: ts=2026-09-22T14:52:43Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:367046770worker1 # [ 143.713699] nix-grpc-daemon[1670]: ts=2026-09-22T14:52:43Z level=debug event=want client=4 drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv cached=06771worker1 # [ 143.715641] nix-grpc-daemon[1670]: ts=2026-09-22T14:52:43Z level=debug event=assigned drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv worker=192.168.1.4:50051 clients=16772worker2 # [ 143.780805] nix-grpc-daemon[1597]: ts=2026-09-22T14:52:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:367046773worker2 # [ 143.791270] nix-grpc-daemon[1597]: ts=2026-09-22T14:52:43Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:367046774client # copying 1 paths to 'lb:50051'...6775worker2 # [ 143.802230] nix-grpc-daemon[1597]: ts=2026-09-22T14:52:43Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:367046776client # copying path '/nix/store/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv' to 'lb:50051'...6777worker2 # [ 143.824220] nix-daemon[850]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)6778worker2 # [ 143.926585] nix-daemon[850]: accepted connection from pid 1765, user nix-grpc-daemon (trusted)6779worker2 # [ 143.940536] nix-daemon[850]: reaped child process 1768, status = succeeded6780worker2 # [ 144.020882] nix-daemon[850]: accepted connection from pid 1770, user nix-grpc-daemon (trusted)6781worker2 # [ 144.034260] nix-daemon[850]: reaped child process 1773, status = succeeded6782lb # [ 144.054812] niks3-server[1099]: 2026/09/22 14:52:43 INFO Received uploads request method=POST path=/api/pending_closures6783worker2 # [ 144.044615] nix-grpc-daemon[1685]: time=2026-09-22T14:52:43.950Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6784worker2 # [ 144.046090] nix-grpc-daemon[1685]: time=2026-09-22T14:52:43.952Z level=INFO msg="Uploading s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv (528B)"6785lb # [ 144.071963] niks3-server[1099]: 2026/09/22 14:52:43 INFO Registered completed upload object_key=nar/0n1m04akn0svzkx0szkj4q7kvaw06fa9yx6k2jibkbkkv85mm55n.nar.zst6786lb # [ 144.075160] niks3-server[1099]: 2026/09/22 14:52:43 INFO Received sign narinfos request method=POST path=/api/pending_closures/55/sign6787lb # [ 144.077908] niks3-server[1099]: 2026/09/22 14:52:43 INFO Signed narinfos id=55 count=16788worker2 # [ 144.062558] nix-grpc-daemon[1685]: time=2026-09-22T14:52:43.968Z level=INFO msg="Uploading 1 narinfos"6789lb # [ 144.081647] niks3-server[1099]: 2026/09/22 14:52:43 INFO Registered completed upload object_key=s7lncxagqa332zn7mhwmk3ry9d83sbvq.ls6790lb # [ 144.086250] niks3-server[1099]: 2026/09/22 14:52:43 INFO Received complete upload request method=POST path=/api/pending_closures/55/complete6791lb # [ 144.089646] niks3-server[1099]: 2026/09/22 14:52:43 INFO Completed upload id=556792worker2 # [ 144.073867] nix-grpc-daemon[1685]: time=2026-09-22T14:52:43.980Z level=INFO msg="Upload complete. (237ms)"6793lb # [ 144.091386] niks3-server[1099]: 2026/09/22 14:52:43 INFO Registered completed upload object_key=s7lncxagqa332zn7mhwmk3ry9d83sbvq.narinfo6794worker2 # [ 144.076060] nix-grpc-daemon[1597]: ts=2026-09-22T14:52:43Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36704 duration_s=0 paths=1 nar_bytes_in=5286795worker2 # [ 144.079861] nix-grpc-daemon[1597]: ts=2026-09-22T14:52:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:367046796worker2 # [ 144.086080] nix-daemon[850]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)6797worker2 # [ 144.088377] nix-daemon[850]: reaped child process 1763, status = succeeded6798client # building '/nix/store/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv' on lb:50051...6799worker2 # [ 144.093460] nix-daemon[850]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)6800client # farm-a> worker2: building s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv6801worker1 # [ 144.309371] nix-grpc-daemon[1670]: ts=2026-09-22T14:52:44Z level=info event=draining6802client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds6803client # farm-a> LOG-farm-a-during-drain6804worker2 # [ 145.536468] nix-daemon[850]: reaped child process 1777, status = succeeded6805worker2 # [ 145.616091] nix-daemon[850]: accepted connection from pid 1793, user nix-grpc-daemon (trusted)6806worker2 # [ 145.628642] nix-daemon[850]: reaped child process 1796, status = succeeded6807lb # [ 145.651796] niks3-server[1099]: 2026/09/22 14:52:45 INFO Received uploads request method=POST path=/api/pending_closures6808worker2 # [ 145.646289] nix-grpc-daemon[1685]: time=2026-09-22T14:52:45.552Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6809worker2 # [ 145.647769] nix-grpc-daemon[1685]: time=2026-09-22T14:52:45.554Z level=INFO msg="Uploading zhl4afvdy7vs34m6ngc3z1xk04qgpcfs-farm-a (136B)"6810lb # [ 145.690731] niks3-server[1099]: 2026/09/22 14:52:45 INFO Registered completed upload object_key=log/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv6811lb # [ 145.703772] niks3-server[1099]: 2026/09/22 14:52:45 INFO Registered completed upload object_key=nar/0r0px9vw070lid3q4jhnmka11sk2y8kj26ysyvjy9dplavd7mdz4.nar.zst6812lb # [ 145.713671] niks3-server[1099]: 2026/09/22 14:52:45 INFO Received sign narinfos request method=POST path=/api/pending_closures/56/sign6813lb # [ 145.717701] niks3-server[1099]: 2026/09/22 14:52:45 INFO Signed narinfos id=56 count=16814worker2 # [ 145.705136] nix-grpc-daemon[1685]: time=2026-09-22T14:52:45.611Z level=INFO msg="Uploading 1 narinfos"6815lb # [ 145.731391] niks3-server[1099]: 2026/09/22 14:52:45 INFO Registered completed upload object_key=zhl4afvdy7vs34m6ngc3z1xk04qgpcfs.ls6816lb # [ 145.734754] niks3-server[1099]: 2026/09/22 14:52:45 INFO Received complete upload request method=POST path=/api/pending_closures/56/complete6817lb # [ 145.738628] niks3-server[1099]: 2026/09/22 14:52:45 INFO Completed upload id=566818worker2 # [ 145.723365] nix-grpc-daemon[1685]: time=2026-09-22T14:52:45.629Z level=INFO msg="Upload complete. (221ms)"6819lb # [ 145.741612] niks3-server[1099]: 2026/09/22 14:52:45 INFO Registered completed upload object_key=zhl4afvdy7vs34m6ngc3z1xk04qgpcfs.narinfo6820worker2 # [ 145.729913] nix-grpc-daemon[1597]: ts=2026-09-22T14:52:45Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36704 duration_s=1 drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv assign_id=4 outputs=16821worker1 # [ 145.716441] nix-grpc-daemon[1670]: ts=2026-09-22T14:52:45Z level=debug event=want client=4 drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv cached=06822worker1 # [ 145.718206] nix-grpc-daemon[1670]: ts=2026-09-22T14:52:45Z level=debug event=assigned drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv worker=192.168.1.4:50051 clients=16823worker2 # [ 145.746265] nix-daemon[850]: reaped child process 1775, status = succeeded6824worker2 # [ 145.779447] nix-grpc-daemon[1597]: ts=2026-09-22T14:52:45Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:367046825lb # [ 145.804523] envoy[905]: conn peer=[2001:db8:1::1]:54780 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3921 tx=1938 ms=80726826lb # [ 145.808856] envoy[905]: conn peer=[2001:db8:1::1]:42090 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2967 tx=850 ms=21856827worker2 # [ 145.798592] nix-grpc-daemon[1597]: ts=2026-09-22T14:52:45Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:367046828worker2 # [ 145.812377] nix-grpc-daemon[1685]: time=2026-09-22T14:52:45.718Z level=INFO msg="All 1 paths already cached"6829client # copying 1 paths to 'lb:50051'...6830worker2 # [ 145.821454] nix-grpc-daemon[1597]: ts=2026-09-22T14:52:45Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:367046831client # copying path '/nix/store/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv' to 'lb:50051'...6832worker2 # [ 145.842123] nix-daemon[850]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)6833worker2 # [ 145.941448] nix-daemon[850]: accepted connection from pid 1800, user nix-grpc-daemon (trusted)6834worker2 # [ 145.956195] nix-daemon[850]: reaped child process 1803, status = succeeded6835worker2 # [ 146.038470] nix-daemon[850]: accepted connection from pid 1805, user nix-grpc-daemon (trusted)6836worker2 # [ 146.051441] nix-daemon[850]: reaped child process 1808, status = succeeded6837lb # [ 146.074224] niks3-server[1099]: 2026/09/22 14:52:45 INFO Received uploads request method=POST path=/api/pending_closures6838worker2 # [ 146.066701] nix-grpc-daemon[1685]: time=2026-09-22T14:52:45.973Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6839worker2 # [ 146.068738] nix-grpc-daemon[1685]: time=2026-09-22T14:52:45.974Z level=INFO msg="Uploading pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv (640B)"6840lb # [ 146.101266] niks3-server[1099]: 2026/09/22 14:52:45 INFO Registered completed upload object_key=nar/111ql9hf3mg62sp5fwspqwm8cl88198mlcc1ij8flm7l424sz63r.nar.zst6841lb # [ 146.115115] niks3-server[1099]: 2026/09/22 14:52:45 INFO Received sign narinfos request method=POST path=/api/pending_closures/57/sign6842worker2 # [ 146.104726] nix-grpc-daemon[1685]: time=2026-09-22T14:52:46.011Z level=INFO msg="Uploading 1 narinfos"6843lb # [ 146.120057] niks3-server[1099]: 2026/09/22 14:52:45 INFO Signed narinfos id=57 count=16844lb # [ 146.125795] niks3-server[1099]: 2026/09/22 14:52:45 INFO Registered completed upload object_key=pmcfhwqsl1hiyc8gjd850vixdadvskwn.ls6845lb # [ 146.140420] niks3-server[1099]: 2026/09/22 14:52:45 INFO Received complete upload request method=POST path=/api/pending_closures/57/complete6846worker2 # [ 146.132235] nix-grpc-daemon[1685]: time=2026-09-22T14:52:46.038Z level=INFO msg="Upload complete. (278ms)"6847lb # [ 146.147678] niks3-server[1099]: 2026/09/22 14:52:45 INFO Completed upload id=576848worker2 # [ 146.134553] nix-grpc-daemon[1597]: ts=2026-09-22T14:52:46Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36704 duration_s=0 paths=1 nar_bytes_in=6406849lb # [ 146.152315] niks3-server[1099]: 2026/09/22 14:52:45 INFO Registered completed upload object_key=pmcfhwqsl1hiyc8gjd850vixdadvskwn.narinfo6850worker2 # [ 146.139689] nix-grpc-daemon[1597]: ts=2026-09-22T14:52:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:367046851worker2 # [ 146.149738] nix-daemon[850]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)6852worker2 # [ 146.154091] nix-daemon[850]: reaped child process 1798, status = succeeded6853worker2 # [ 146.157970] nix-daemon[850]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)6854client # building '/nix/store/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv' on lb:50051...6855client # farm-b> worker2: building pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv6856client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds6857client # farm-b> LOG-farm-b-during-drain6858worker2 # [ 147.188557] nix-daemon[850]: reaped child process 1812, status = succeeded6859worker2 # [ 147.264315] nix-daemon[850]: accepted connection from pid 1828, user nix-grpc-daemon (trusted)6860worker2 # [ 147.278970] nix-daemon[850]: reaped child process 1831, status = succeeded6861lb # [ 147.298641] niks3-server[1099]: 2026/09/22 14:52:46 INFO Received uploads request method=POST path=/api/pending_closures6862worker2 # [ 147.291515] nix-grpc-daemon[1685]: time=2026-09-22T14:52:47.197Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6863worker2 # [ 147.293137] nix-grpc-daemon[1685]: time=2026-09-22T14:52:47.199Z level=INFO msg="Uploading l458nfm9f1f1zyghsy09ijx02qwk7v65-farm-b (184B)"6864lb # [ 147.329357] niks3-server[1099]: 2026/09/22 14:52:46 INFO Registered completed upload object_key=nar/0m50ra4cd4i97vcbcrdsw7xn8f0b3zc5pdyr2ab8kmp3hscd8j80.nar.zst6865lb # [ 147.347141] niks3-server[1099]: 2026/09/22 14:52:46 INFO Registered completed upload object_key=log/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv6866lb # [ 147.352869] niks3-server[1099]: 2026/09/22 14:52:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/58/sign6867lb # [ 147.359197] niks3-server[1099]: 2026/09/22 14:52:46 INFO Signed narinfos id=58 count=16868worker2 # [ 147.343622] nix-grpc-daemon[1685]: time=2026-09-22T14:52:47.249Z level=INFO msg="Uploading 1 narinfos"6869lb # [ 147.368545] niks3-server[1099]: 2026/09/22 14:52:46 INFO Received complete upload request method=POST path=/api/pending_closures/58/complete6870lb # [ 147.377283] niks3-server[1099]: 2026/09/22 14:52:46 INFO Completed upload id=586871lb # [ 147.378864] niks3-server[1099]: 2026/09/22 14:52:46 INFO Registered completed upload object_key=l458nfm9f1f1zyghsy09ijx02qwk7v65.ls6872worker2 # [ 147.363210] nix-grpc-daemon[1685]: time=2026-09-22T14:52:47.269Z level=INFO msg="Upload complete. (197ms)"6873lb # [ 147.381349] niks3-server[1099]: 2026/09/22 14:52:46 INFO Registered completed upload object_key=l458nfm9f1f1zyghsy09ijx02qwk7v65.narinfo6874worker2 # [ 147.369776] nix-grpc-daemon[1597]: ts=2026-09-22T14:52:47Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36704 duration_s=1 drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv assign_id=5 outputs=16875worker1 # [ 147.355762] nix-grpc-daemon[1670]: ts=2026-09-22T14:52:47Z level=debug event=want client=4 drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv cached=06876worker1 # [ 147.358213] nix-grpc-daemon[1670]: ts=2026-09-22T14:52:47Z level=debug event=assigned drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv worker=192.168.1.4:50051 clients=16877worker2 # [ 147.384992] nix-daemon[850]: reaped child process 1810, status = succeeded6878worker2 # [ 147.417969] nix-grpc-daemon[1597]: ts=2026-09-22T14:52:47Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:367046879worker2 # [ 147.430398] nix-grpc-daemon[1597]: ts=2026-09-22T14:52:47Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:367046880worker2 # [ 147.443239] nix-grpc-daemon[1685]: time=2026-09-22T14:52:47.349Z level=INFO msg="All 2 paths already cached"6881client # copying 1 paths to 'lb:50051'...6882client # copying path '/nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv' to 'lb:50051'...6883worker2 # [ 147.452749] nix-grpc-daemon[1597]: ts=2026-09-22T14:52:47Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:367046884worker2 # [ 147.467169] nix-daemon[850]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)6885worker2 # [ 147.566474] nix-daemon[850]: accepted connection from pid 1835, user nix-grpc-daemon (trusted)6886worker2 # [ 147.580303] nix-daemon[850]: reaped child process 1838, status = succeeded6887worker2 # [ 147.662189] nix-daemon[850]: accepted connection from pid 1840, user nix-grpc-daemon (trusted)6888worker2 # [ 147.675420] nix-daemon[850]: reaped child process 1843, status = succeeded6889lb # [ 147.697666] niks3-server[1099]: 2026/09/22 14:52:47 INFO Received uploads request method=POST path=/api/pending_closures6890worker2 # [ 147.690365] nix-grpc-daemon[1685]: time=2026-09-22T14:52:47.596Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6891worker2 # [ 147.692238] nix-grpc-daemon[1685]: time=2026-09-22T14:52:47.598Z level=INFO msg="Uploading w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv (768B)"6892lb # [ 147.730183] niks3-server[1099]: 2026/09/22 14:52:47 INFO Registered completed upload object_key=nar/1crrgw2l7pdmf28ls634k8y85m0pc1v13pxxp4rd6yg5mk3kr5fs.nar.zst6893lb # [ 147.736489] niks3-server[1099]: 2026/09/22 14:52:47 INFO Received sign narinfos request method=POST path=/api/pending_closures/59/sign6894lb # [ 147.742433] niks3-server[1099]: 2026/09/22 14:52:47 INFO Signed narinfos id=59 count=16895worker2 # [ 147.730405] nix-grpc-daemon[1685]: time=2026-09-22T14:52:47.636Z level=INFO msg="Uploading 1 narinfos"6896lb # [ 147.752646] niks3-server[1099]: 2026/09/22 14:52:47 INFO Registered completed upload object_key=w4kjbckcz25f2jivly0r4lbq18qf474f.ls6897lb # [ 147.762579] niks3-server[1099]: 2026/09/22 14:52:47 INFO Received complete upload request method=POST path=/api/pending_closures/59/complete6898lb # [ 147.770108] niks3-server[1099]: 2026/09/22 14:52:47 INFO Completed upload id=596899worker2 # [ 147.754206] nix-grpc-daemon[1685]: time=2026-09-22T14:52:47.660Z level=INFO msg="Upload complete. (274ms)"6900lb # [ 147.773424] niks3-server[1099]: 2026/09/22 14:52:47 INFO Registered completed upload object_key=w4kjbckcz25f2jivly0r4lbq18qf474f.narinfo6901worker2 # [ 147.758397] nix-grpc-daemon[1597]: ts=2026-09-22T14:52:47Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36704 duration_s=0 paths=1 nar_bytes_in=7686902worker2 # [ 147.763449] nix-grpc-daemon[1597]: ts=2026-09-22T14:52:47Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:367046903worker2 # [ 147.772714] nix-daemon[850]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)6904worker2 # [ 147.780528] nix-daemon[850]: reaped child process 1833, status = succeeded6905worker2 # [ 147.785640] nix-daemon[850]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)6906client # building '/nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv' on lb:50051...6907client # farm-top> worker2: building w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6908client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds6909client # farm-top> LOG-farm-top-during-drain6910worker2 # [ 149.002334] nix-daemon[850]: reaped child process 1847, status = succeeded6911worker2 # [ 149.077978] nix-daemon[850]: accepted connection from pid 1863, user nix-grpc-daemon (trusted)6912worker2 # [ 149.090165] nix-daemon[850]: reaped child process 1866, status = succeeded6913lb # [ 149.112500] niks3-server[1099]: 2026/09/22 14:52:48 INFO Received uploads request method=POST path=/api/pending_closures6914worker2 # [ 149.101265] nix-grpc-daemon[1685]: time=2026-09-22T14:52:49.007Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6915worker2 # [ 149.102724] nix-grpc-daemon[1685]: time=2026-09-22T14:52:49.009Z level=INFO msg="Uploading 08ap9ki0d5gixmyr2sac4hj06j9jsqhg-farm-top (240B)"6916lb # [ 149.131451] niks3-server[1099]: 2026/09/22 14:52:48 INFO Registered completed upload object_key=nar/01f0abadm30zaav33qck3x7ypr9nn6rqh2vbrvih4gyb8c26x9in.nar.zst6917lb # [ 149.135564] niks3-server[1099]: 2026/09/22 14:52:48 INFO Registered completed upload object_key=log/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6918lb # [ 149.140860] niks3-server[1099]: 2026/09/22 14:52:48 INFO Received sign narinfos request method=POST path=/api/pending_closures/60/sign6919lb # [ 149.146111] niks3-server[1099]: 2026/09/22 14:52:48 INFO Signed narinfos id=60 count=16920worker2 # [ 149.132133] nix-grpc-daemon[1685]: time=2026-09-22T14:52:49.038Z level=INFO msg="Uploading 1 narinfos"6921lb # [ 149.149048] niks3-server[1099]: 2026/09/22 14:52:48 INFO Registered completed upload object_key=08ap9ki0d5gixmyr2sac4hj06j9jsqhg.ls6922lb # [ 149.153546] niks3-server[1099]: 2026/09/22 14:52:48 INFO Received complete upload request method=POST path=/api/pending_closures/60/complete6923lb # [ 149.157552] niks3-server[1099]: 2026/09/22 14:52:48 INFO Completed upload id=606924worker2 # [ 149.143171] nix-grpc-daemon[1685]: time=2026-09-22T14:52:49.049Z level=INFO msg="Upload complete. (150ms)"6925lb # [ 149.159849] niks3-server[1099]: 2026/09/22 14:52:48 INFO Registered completed upload object_key=08ap9ki0d5gixmyr2sac4hj06j9jsqhg.narinfo6926worker2 # [ 149.145938] nix-grpc-daemon[1597]: ts=2026-09-22T14:52:49Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36704 duration_s=1 drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv assign_id=6 outputs=16927worker2 # [ 149.153107] nix-daemon[850]: reaped child process 1845, status = succeeded6928client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag during-drain >&2, in 5.58 seconds)6929client: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag during-drain outPath6930client: (finished: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag during-drain outPath, in 0.11 seconds)6931worker1: must succeed: pkill -f 'read -t [9]0 x'6932worker1: (finished: must succeed: pkill -f 'read -t [9]0 x', in 0.04 seconds)6933client: waiting for success: ! systemctl is-active sw6934worker1 # [ 149.477138] nix-daemon[869]: reaped child process 1821, status = succeeded6935worker1 # [ 149.553680] nix-daemon[869]: accepted connection from pid 1869, user nix-grpc-daemon (trusted)6936worker1 # [ 149.566103] nix-daemon[869]: reaped child process 1872, status = succeeded6937lb # [ 149.606281] niks3-server[1099]: 2026/09/22 14:52:48 INFO Received uploads request method=POST path=/api/pending_closures6938worker1 # [ 149.579871] nix-grpc-daemon[1753]: time=2026-09-22T14:52:49.696Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6939worker1 # [ 149.581400] nix-grpc-daemon[1753]: time=2026-09-22T14:52:49.698Z level=INFO msg="Uploading mlk55snqrikr02pwsxjamphvl61rvl0r-slow-sw (120B)"6940lb # [ 149.646219] niks3-server[1099]: 2026/09/22 14:52:49 INFO Received sign narinfos request method=POST path=/api/pending_closures/61/sign6941worker1 # [ 149.622459] nix-grpc-daemon[1753]: time=2026-09-22T14:52:49.739Z level=INFO msg="Uploading 1 narinfos"6942lb # [ 149.656432] niks3-server[1099]: 2026/09/22 14:52:49 INFO Signed narinfos id=61 count=16943lb # [ 149.662266] niks3-server[1099]: 2026/09/22 14:52:49 INFO Registered completed upload object_key=mlk55snqrikr02pwsxjamphvl61rvl0r.ls6944lb # [ 149.666957] niks3-server[1099]: 2026/09/22 14:52:49 INFO Registered completed upload object_key=nar/0vi6cx31wc05wns04pr5vr2bak9iib1idndvy6gik6c2rz9q8c19.nar.zst6945lb # [ 149.679842] niks3-server[1099]: 2026/09/22 14:52:49 INFO Received complete upload request method=POST path=/api/pending_closures/61/complete6946lb # [ 149.688572] niks3-server[1099]: 2026/09/22 14:52:49 INFO Completed upload id=616947worker1 # [ 149.654248] nix-grpc-daemon[1753]: time=2026-09-22T14:52:49.771Z level=INFO msg="Upload complete. (209ms)"6948lb # [ 149.689934] niks3-server[1099]: 2026/09/22 14:52:49 INFO Registered completed upload object_key=log/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv6949lb # [ 149.692548] niks3-server[1099]: 2026/09/22 14:52:49 INFO Registered completed upload object_key=mlk55snqrikr02pwsxjamphvl61rvl0r.narinfo6950worker1 # [ 149.662570] nix-grpc-daemon[1670]: ts=2026-09-22T14:52:49Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42282 duration_s=7 drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv assign_id=3 outputs=16951worker1 # [ 149.675706] nix-daemon[869]: reaped child process 1819, status = succeeded6952client # [ 149.711528] systemd[1]: sw.service: Deactivated successfully.6953client # [ 149.713089] systemd[1]: sw.service: Consumed 135ms CPU time over 9.570s wall clock time, 43.4M memory peak, 7.5K incoming IP traffic, 8.5K outgoing IP traffic.6954worker1 # [ 150.311331] nix-grpc-daemon[1670]: ts=2026-09-22T14:52:50Z level=info event=signal_exit6955lb # [ 150.351826] niks3-server[1099]: 2026/09/22 14:52:49 INFO lead: released remote=192.168.1.3:514706956client: (finished: waiting for success: ! systemctl is-active sw, in 1.04 seconds)6957client: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }6958client: (finished: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }, in 0.02 seconds)6959worker1: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service6960worker2 # [ 151.338366] nix-grpc-daemon[1597]: ts=2026-09-22T14:52:51Z level=info event=scheduler_restarting addr=lb:500516961worker1 # [ 151.316819] nix-grpc-daemon[1670]: ts=2026-09-22T14:52:51Z level=info event=scheduler_restarting6962worker1 # [ 151.325561] nix-grpc-daemon[1670]: ts=2026-09-22T14:52:51Z level=info event=scheduler_restarting addr=lb:500516963worker1 # [ 151.328656] nix-grpc-daemon[1670]: ts=2026-09-22T14:52:51Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:422406964worker1 # [ 151.332240] nix-grpc-daemon[1670]: ts=2026-09-22T14:52:51Z level=info event=worker_gone addr=192.168.1.4:500516965worker1 # [ 151.334988] nix-grpc-daemon[1670]: ts=2026-09-22T14:52:51Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:422406966worker1 # [ 151.338080] nix-grpc-daemon[1670]: ts=2026-09-22T14:52:51Z level=info event=worker_gone addr=192.168.1.3:500516967worker1 # [ 151.348326] nix-daemon[869]: reaped child process 1724, status = succeeded6968worker1 # [ 151.435607] nix-grpc-daemon[1670]: E0000 00:00:1790088771.552709 1715 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 17, started true, shutdown true6969worker1 # [ 151.441102] nix-grpc-daemon[1670]: E0000 00:00:1790088771.558095 1715 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 9, started true, shutdown true6970worker2 # [ 151.538590] nix-grpc-daemon[1597]: ts=2026-09-22T14:52:51Z level=info event=scheduler_connected addr=lb:500516971worker2 # [ 151.542812] nix-grpc-daemon[1597]: ts=2026-09-22T14:52:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6972worker1 # [ 151.533489] nix-grpc-daemon[1670]: ts=2026-09-22T14:52:51Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"6973worker2 # [ 151.743904] nix-grpc-daemon[1597]: ts=2026-09-22T14:52:51Z level=info event=scheduler_connected addr=lb:500516974worker2 # [ 151.747893] nix-grpc-daemon[1597]: ts=2026-09-22T14:52:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6975worker2 # [ 151.948421] nix-grpc-daemon[1597]: ts=2026-09-22T14:52:51Z level=info event=scheduler_connected addr=lb:500516976worker2 # [ 151.950731] nix-grpc-daemon[1597]: ts=2026-09-22T14:52:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6977worker2 # [ 152.152132] nix-grpc-daemon[1597]: ts=2026-09-22T14:52:52Z level=info event=scheduler_connected addr=lb:500516978worker2 # [ 152.156192] nix-grpc-daemon[1597]: ts=2026-09-22T14:52:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6979worker1 # [ 152.331800] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.6980worker1 # [ 152.333128] systemd[1]: nix-grpc-daemon.service: Consumed 647ms CPU time over 22.158s wall clock time, 36.5M memory peak, 24.7K incoming IP traffic, 32K outgoing IP traffic.6981worker1 # [ 152.340355] systemd[1]: Starting Nix worker-protocol over gRPC...6982worker2 # [ 152.358280] nix-grpc-daemon[1597]: ts=2026-09-22T14:52:52Z level=info event=scheduler_connected addr=lb:500516983worker2 # [ 152.361506] nix-grpc-daemon[1597]: ts=2026-09-22T14:52:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6984worker1 # [ 152.416948] nix-grpc-daemon[1886]: ts=2026-09-22T14:52:52Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6985worker1 # [ 152.419133] nix-grpc-daemon[1886]: ts=2026-09-22T14:52:52Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6986worker1 # [ 152.422384] nix-grpc-daemon[1886]: ts=2026-09-22T14:52:52Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=16987worker1 # [ 152.426686] nix-grpc-daemon[1886]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6988worker1 # [ 152.429917] nix-grpc-daemon[1886]: E0000 00:00:1790088772.543826 1886 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6989worker1 # [ 152.433473] nix-grpc-daemon[1886]: E0000 00:00:1790088772.547033 1886 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6990worker1 # [ 152.439412] nix-grpc-daemon[1886]: ts=2026-09-22T14:52:52Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6991worker1 # [ 152.442508] systemd[1]: Started Nix worker-protocol over gRPC.6992lb # [ 152.476782] niks3-server[1099]: 2026/09/22 14:52:51 INFO lead: acquired remote=192.168.1.3:512126993worker1 # [ 152.445433] nix-grpc-daemon[1886]: E0000 00:00:1790088772.560050 1927 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6994worker1 # [ 152.451064] nix-grpc-daemon[1886]: ts=2026-09-22T14:52:52Z level=info event=scheduler_take_over6995worker1 # [ 152.460069] nix-grpc-daemon[1886]: E0000 00:00:1790088772.575953 1927 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6996worker1 # [ 152.462232] nix-grpc-daemon[1886]: E0000 00:00:1790088772.576074 1927 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6997worker1: (finished: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service, in 2.10 seconds)6998lb: waiting until exactly no endpoints of x86_64-linux are unhealthy6999lb: must succeed: curl -sf localhost:9901/clusters7000worker1 # [ 152.473964] nix-grpc-daemon[1886]: E0000 00:00:1790088772.591197 1927 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.7001worker1 # [ 152.485107] nix-grpc-daemon[1886]: ts=2026-09-22T14:52:52Z level=info event=scheduler_connected addr=lb:500517002worker1 # [ 152.486668] nix-grpc-daemon[1886]: ts=2026-09-22T14:52:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"7003lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)7004worker2 # [ 152.559862] nix-grpc-daemon[1597]: ts=2026-09-22T14:52:52Z level=info event=scheduler_connected addr=lb:500517005worker2 # [ 152.563856] nix-grpc-daemon[1597]: ts=2026-09-22T14:52:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"7006worker1 # [ 152.687711] nix-grpc-daemon[1886]: ts=2026-09-22T14:52:52Z level=info event=scheduler_connected addr=lb:500517007worker1 # [ 152.691778] nix-grpc-daemon[1886]: ts=2026-09-22T14:52:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"7008worker2 # [ 152.765174] nix-grpc-daemon[1597]: ts=2026-09-22T14:52:52Z level=info event=scheduler_connected addr=lb:500517009worker2 # [ 152.769090] nix-grpc-daemon[1597]: ts=2026-09-22T14:52:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"7010worker1 # [ 152.892572] nix-grpc-daemon[1886]: ts=2026-09-22T14:52:53Z level=info event=scheduler_connected addr=lb:500517011worker1 # [ 152.895067] nix-grpc-daemon[1886]: ts=2026-09-22T14:52:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"7012worker2 # [ 152.970287] nix-grpc-daemon[1597]: ts=2026-09-22T14:52:52Z level=info event=scheduler_connected addr=lb:500517013worker2 # [ 152.974312] nix-grpc-daemon[1597]: ts=2026-09-22T14:52:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"7014worker1 # [ 153.095474] nix-grpc-daemon[1886]: ts=2026-09-22T14:52:53Z level=info event=scheduler_connected addr=lb:500517015worker1 # [ 153.097366] nix-grpc-daemon[1886]: ts=2026-09-22T14:52:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"7016worker2 # [ 153.175575] nix-grpc-daemon[1597]: ts=2026-09-22T14:52:53Z level=info event=scheduler_connected addr=lb:500517017worker2 # [ 153.179568] nix-grpc-daemon[1597]: ts=2026-09-22T14:52:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"7018worker1 # [ 153.297950] nix-grpc-daemon[1886]: ts=2026-09-22T14:52:53Z level=info event=scheduler_connected addr=lb:500517019worker1 # [ 153.299977] nix-grpc-daemon[1886]: ts=2026-09-22T14:52:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"7020worker2 # [ 153.381683] nix-grpc-daemon[1597]: ts=2026-09-22T14:52:53Z level=info event=scheduler_connected addr=lb:500517021worker1 # [ 153.368466] nix-grpc-daemon[1886]: ts=2026-09-22T14:52:53Z level=info event=oidc_jwks provider=mock keys=17022worker1 # [ 153.369987] nix-grpc-daemon[1886]: ts=2026-09-22T14:52:53Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:511467023worker1 # [ 153.371659] nix-grpc-daemon[1886]: ts=2026-09-22T14:52:53Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=07024worker1 # [ 153.503318] nix-grpc-daemon[1886]: ts=2026-09-22T14:52:53Z level=info event=scheduler_connected addr=lb:500517025worker1 # [ 153.506491] nix-grpc-daemon[1886]: ts=2026-09-22T14:52:53Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:511467026worker1 # [ 153.510189] nix-grpc-daemon[1886]: ts=2026-09-22T14:52:53Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=07027lb: must succeed: curl -sf localhost:9901/clusters7028lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)7029lb: (finished: waiting until exactly no endpoints of x86_64-linux are unhealthy, in 1.10 seconds)7030lb: must succeed: curl -sf localhost:9901/clusters7031lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)7032worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics7033worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)7034(finished: subtest: a deploy mid-build drains: the build finishes, the new generation takes over, in 13.57 seconds)7035subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does7036client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&17037worker1 # [ 153.786183] nix-grpc-daemon[1886]: ts=2026-09-22T14:52:53Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:511607038worker1 # [ 153.797588] nix-daemon[869]: accepted connection from pid 1886, user nix-grpc-daemon (trusted)7039worker1 # [ 153.811257] nix-grpc-daemon[1886]: ts=2026-09-22T14:52:53Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:511607040worker1 # [ 153.821627] nix-grpc-daemon[1886]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 52 ms (attempt 1/5)7041worker1 # [ 153.880921] nix-grpc-daemon[1886]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 168 ms (attempt 2/5)7042worker1 # [ 154.050321] nix-grpc-daemon[1886]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 186 ms (attempt 3/5)7043worker1 # [ 154.237675] nix-grpc-daemon[1886]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 330 ms (attempt 4/5)7044worker1 # [ 154.586602] nix-grpc-daemon[1886]: disabling binary cache 'https://cache.nixos.org' for 60 seconds7045worker1 # [ 154.589995] nix-grpc-daemon[1886]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org7046worker1 # [ 154.637187] nix-grpc-daemon[1886]: ts=2026-09-22T14:52:54Z level=debug event=want client=1 drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv cached=07047worker1 # [ 154.642707] nix-grpc-daemon[1886]: ts=2026-09-22T14:52:54Z level=debug event=assigned drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv worker=192.168.1.3:50051 clients=17048worker1 # [ 154.647257] nix-grpc-daemon[1886]: ts=2026-09-22T14:52:54Z level=debug event=want client=1 drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv cached=07049worker1 # [ 154.651218] nix-grpc-daemon[1886]: ts=2026-09-22T14:52:54Z level=debug event=want client=1 drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv cached=07050worker1 # [ 154.682163] nix-grpc-daemon[1886]: ts=2026-09-22T14:52:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:511607051worker1 # [ 154.694902] nix-grpc-daemon[1886]: ts=2026-09-22T14:52:54Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:511607052worker1 # [ 154.706336] nix-grpc-daemon[1886]: ts=2026-09-22T14:52:54Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:511607053worker1 # [ 154.714547] nix-daemon[869]: accepted connection from pid 1886, user nix-grpc-daemon (trusted)7054worker1 # [ 154.736072] nix-grpc-daemon[1886]: ts=2026-09-22T14:52:54Z level=info event=niks3_push_started pid=19517055worker1 # [ 154.834650] nix-grpc-daemon[1951]: time=2026-09-22T14:52:54.951Z level=INFO msg="Configuring client TLS" cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker1.pem key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker1.key ca=""7056worker1 # [ 154.920822] nix-daemon[869]: accepted connection from pid 1964, user nix-grpc-daemon (trusted)7057worker1 # [ 154.933324] nix-daemon[869]: reaped child process 1967, status = succeeded7058worker1 # [ 155.013990] nix-daemon[869]: accepted connection from pid 1969, user nix-grpc-daemon (trusted)7059worker1 # [ 155.026943] nix-daemon[869]: reaped child process 1972, status = succeeded7060lb # [ 155.068625] niks3-server[1099]: 2026/09/22 14:52:54 INFO Received uploads request method=POST path=/api/pending_closures7061worker1 # [ 155.045064] nix-grpc-daemon[1951]: time=2026-09-22T14:52:55.161Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7062worker1 # [ 155.046461] nix-grpc-daemon[1951]: time=2026-09-22T14:52:55.161Z level=INFO msg="Uploading cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv (504B)"7063lb # [ 155.112851] niks3-server[1099]: 2026/09/22 14:52:54 INFO Received sign narinfos request method=POST path=/api/pending_closures/62/sign7064lb # [ 155.120495] niks3-server[1099]: 2026/09/22 14:52:54 INFO Registered completed upload object_key=nar/1ifhv0ll3mnj6a2hrp33fzwnm2yl2zjxwqfrqjriibhs5cbnyiwi.nar.zst7065worker1 # [ 155.104396] nix-grpc-daemon[1951]: time=2026-09-22T14:52:55.221Z level=INFO msg="Uploading 1 narinfos"7066lb # [ 155.138184] niks3-server[1099]: 2026/09/22 14:52:54 INFO Signed narinfos id=62 count=17067lb # [ 155.150814] niks3-server[1099]: 2026/09/22 14:52:54 INFO Registered completed upload object_key=cv151lgjr4lv54sqzb49xk9w1ak9xsm9.ls7068lb # [ 155.153989] niks3-server[1099]: 2026/09/22 14:52:54 INFO Received complete upload request method=POST path=/api/pending_closures/62/complete7069lb # [ 155.158565] niks3-server[1099]: 2026/09/22 14:52:54 INFO Completed upload id=627070worker1 # [ 155.124356] nix-grpc-daemon[1951]: time=2026-09-22T14:52:55.241Z level=INFO msg="Upload complete. (286ms)"7071lb # [ 155.161449] niks3-server[1099]: 2026/09/22 14:52:54 INFO Registered completed upload object_key=cv151lgjr4lv54sqzb49xk9w1ak9xsm9.narinfo7072worker1 # [ 155.126668] nix-grpc-daemon[1886]: ts=2026-09-22T14:52:55Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:51160 duration_s=0 paths=1 nar_bytes_in=5047073worker1 # [ 155.129834] nix-grpc-daemon[1886]: ts=2026-09-22T14:52:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:511607074worker1 # [ 155.137496] nix-daemon[869]: accepted connection from pid 1886, user nix-grpc-daemon (trusted)7075worker1 # [ 155.144202] nix-daemon[869]: reaped child process 1949, status = succeeded7076worker1 # [ 155.147126] nix-daemon[869]: accepted connection from pid 1886, user nix-grpc-daemon (trusted)7077lb # [ 155.804159] envoy[905]: conn peer=[2001:db8:1::1]:49886 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9594 tx=4412 ms=54547078lb # [ 155.808633] envoy[905]: conn peer=[2001:db8:1::1]:42092 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3005 tx=1582 ms=94267079lb # [ 155.812616] envoy[905]: conn peer=[2001:db8:1::3]:34874 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=958 tx=344 ms=211777080worker1 # [ 156.378416] nix-daemon[869]: reaped child process 1976, status = succeeded7081worker1 # [ 156.455471] nix-daemon[869]: accepted connection from pid 1992, user nix-grpc-daemon (trusted)7082worker1 # [ 156.468069] nix-daemon[869]: reaped child process 1995, status = succeeded7083lb # [ 156.508559] niks3-server[1099]: 2026/09/22 14:52:55 INFO Received uploads request method=POST path=/api/pending_closures7084worker1 # [ 156.482849] nix-grpc-daemon[1951]: time=2026-09-22T14:52:56.599Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7085worker1 # [ 156.484816] nix-grpc-daemon[1951]: time=2026-09-22T14:52:56.601Z level=INFO msg="Uploading wfziwgf0gxmjhy17lyfmn47628s166p7-farm-a (128B)"7086lb # [ 156.539184] niks3-server[1099]: 2026/09/22 14:52:55 INFO Registered completed upload object_key=nar/05i7ka1qa7vmhiys96ln3pp3nj3rfdlslnbydrxdqj40cmvmqcwf.nar.zst7087lb # [ 156.542086] niks3-server[1099]: 2026/09/22 14:52:55 INFO Received sign narinfos request method=POST path=/api/pending_closures/63/sign7088lb # [ 156.545585] niks3-server[1099]: 2026/09/22 14:52:55 INFO Signed narinfos id=63 count=17089worker1 # [ 156.512599] nix-grpc-daemon[1951]: time=2026-09-22T14:52:56.629Z level=INFO msg="Uploading 1 narinfos"7090lb # [ 156.547052] niks3-server[1099]: 2026/09/22 14:52:55 INFO Registered completed upload object_key=wfziwgf0gxmjhy17lyfmn47628s166p7.ls7091lb # [ 156.550320] niks3-server[1099]: 2026/09/22 14:52:55 INFO Registered completed upload object_key=log/cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv7092lb # [ 156.555936] niks3-server[1099]: 2026/09/22 14:52:55 INFO Received complete upload request method=POST path=/api/pending_closures/63/complete7093lb # [ 156.559241] niks3-server[1099]: 2026/09/22 14:52:55 INFO Completed upload id=637094worker1 # [ 156.526324] nix-grpc-daemon[1951]: time=2026-09-22T14:52:56.643Z level=INFO msg="Upload complete. (171ms)"7095lb # [ 156.563619] niks3-server[1099]: 2026/09/22 14:52:55 INFO Registered completed upload object_key=wfziwgf0gxmjhy17lyfmn47628s166p7.narinfo7096worker1 # [ 156.530691] nix-grpc-daemon[1886]: ts=2026-09-22T14:52:56Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:51160 duration_s=1 drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv assign_id=1 outputs=17097worker1 # [ 156.536274] nix-grpc-daemon[1886]: ts=2026-09-22T14:52:56Z level=debug event=assigned drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv worker=192.168.1.3:50051 clients=17098worker1 # [ 156.538622] nix-grpc-daemon[1886]: ts=2026-09-22T14:52:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:511607099worker1 # [ 156.540361] nix-grpc-daemon[1886]: ts=2026-09-22T14:52:56Z level=debug event=want client=1 drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv cached=07100worker1 # [ 156.547306] nix-grpc-daemon[1886]: ts=2026-09-22T14:52:56Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:511607101worker1 # [ 156.549263] nix-daemon[869]: reaped child process 1974, status = succeeded7102worker1 # [ 156.553854] nix-grpc-daemon[1886]: ts=2026-09-22T14:52:56Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:511607103worker1 # [ 156.557538] nix-daemon[869]: accepted connection from pid 1886, user nix-grpc-daemon (trusted)7104worker1 # [ 156.635640] nix-daemon[869]: accepted connection from pid 2001, user nix-grpc-daemon (trusted)7105worker1 # [ 156.649665] nix-daemon[869]: reaped child process 2004, status = succeeded7106worker1 # [ 156.732704] nix-daemon[869]: accepted connection from pid 2006, user nix-grpc-daemon (trusted)7107worker1 # [ 156.746601] nix-daemon[869]: reaped child process 2009, status = succeeded7108lb # [ 156.787865] niks3-server[1099]: 2026/09/22 14:52:56 INFO Received uploads request method=POST path=/api/pending_closures7109worker1 # [ 156.763059] nix-grpc-daemon[1951]: time=2026-09-22T14:52:56.879Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7110worker1 # [ 156.764453] nix-grpc-daemon[1951]: time=2026-09-22T14:52:56.879Z level=INFO msg="Uploading fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv (504B)"7111lb # [ 156.817612] niks3-server[1099]: 2026/09/22 14:52:56 INFO Registered completed upload object_key=nar/170l67nd7hvasq2l36jibfpivx7r9ssdy0wlzifggpplhcgf2rk0.nar.zst7112lb # [ 156.826117] niks3-server[1099]: 2026/09/22 14:52:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/64/sign7113worker1 # [ 156.797550] nix-grpc-daemon[1951]: time=2026-09-22T14:52:56.914Z level=INFO msg="Uploading 1 narinfos"7114lb # [ 156.831530] niks3-server[1099]: 2026/09/22 14:52:56 INFO Signed narinfos id=64 count=17115lb # [ 156.845161] niks3-server[1099]: 2026/09/22 14:52:56 INFO Registered completed upload object_key=fggw0ziag6jlrc830j6x1b6qw68kybhf.ls7116lb # [ 156.851359] niks3-server[1099]: 2026/09/22 14:52:56 INFO Received complete upload request method=POST path=/api/pending_closures/64/complete7117lb # [ 156.861892] niks3-server[1099]: 2026/09/22 14:52:56 INFO Completed upload id=647118worker1 # [ 156.827863] nix-grpc-daemon[1951]: time=2026-09-22T14:52:56.945Z level=INFO msg="Upload complete. (264ms)"7119lb # [ 156.863577] niks3-server[1099]: 2026/09/22 14:52:56 INFO Registered completed upload object_key=fggw0ziag6jlrc830j6x1b6qw68kybhf.narinfo7120worker1 # [ 156.831920] nix-grpc-daemon[1886]: ts=2026-09-22T14:52:56Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:51160 duration_s=0 paths=1 nar_bytes_in=5047121worker1 # [ 156.837085] nix-grpc-daemon[1886]: ts=2026-09-22T14:52:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:511607122worker1 # [ 156.846725] nix-daemon[869]: accepted connection from pid 1886, user nix-grpc-daemon (trusted)7123worker1 # [ 156.856881] nix-daemon[869]: reaped child process 1999, status = succeeded7124worker1 # [ 156.859686] nix-daemon[869]: accepted connection from pid 1886, user nix-grpc-daemon (trusted)7125worker1 # [ 158.276338] nix-daemon[869]: reaped child process 2013, status = succeeded7126worker1 # [ 158.353906] nix-daemon[869]: accepted connection from pid 2029, user nix-grpc-daemon (trusted)7127worker1 # [ 158.368408] nix-daemon[869]: reaped child process 2032, status = succeeded7128lb # [ 158.407532] niks3-server[1099]: 2026/09/22 14:52:57 INFO Received uploads request method=POST path=/api/pending_closures7129worker1 # [ 158.381838] nix-grpc-daemon[1951]: time=2026-09-22T14:52:58.498Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7130worker1 # [ 158.383298] nix-grpc-daemon[1951]: time=2026-09-22T14:52:58.500Z level=INFO msg="Uploading 63zyaryk3r1ikmmzi1w30vadf20kdw3y-farm-a (128B)"7131lb # [ 158.439804] niks3-server[1099]: 2026/09/22 14:52:57 INFO Registered completed upload object_key=log/fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv7132lb # [ 158.456824] niks3-server[1099]: 2026/09/22 14:52:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/65/sign7133lb # [ 158.462149] niks3-server[1099]: 2026/09/22 14:52:57 INFO Signed narinfos id=65 count=17134worker1 # [ 158.431518] nix-grpc-daemon[1951]: time=2026-09-22T14:52:58.548Z level=INFO msg="Uploading 1 narinfos"7135lb # [ 158.471470] niks3-server[1099]: 2026/09/22 14:52:57 INFO Registered completed upload object_key=nar/09na6djn66dpys8h08gqw39p0vw7vxqsdzbiw55rknyna8s239bm.nar.zst7136lb # [ 158.476155] niks3-server[1099]: 2026/09/22 14:52:57 INFO Registered completed upload object_key=63zyaryk3r1ikmmzi1w30vadf20kdw3y.ls7137lb # [ 158.485025] niks3-server[1099]: 2026/09/22 14:52:57 INFO Received complete upload request method=POST path=/api/pending_closures/65/complete7138lb # [ 158.488220] niks3-server[1099]: 2026/09/22 14:52:57 INFO Completed upload id=657139worker1 # [ 158.455175] nix-grpc-daemon[1951]: time=2026-09-22T14:52:58.572Z level=INFO msg="Upload complete. (187ms)"7140lb # [ 158.489994] niks3-server[1099]: 2026/09/22 14:52:57 INFO Registered completed upload object_key=63zyaryk3r1ikmmzi1w30vadf20kdw3y.narinfo7141worker1 # [ 158.459482] nix-grpc-daemon[1886]: ts=2026-09-22T14:52:58Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:51160 duration_s=1 drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv assign_id=2 outputs=17142worker1 # [ 158.465585] nix-grpc-daemon[1886]: ts=2026-09-22T14:52:58Z level=debug event=assigned drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv worker=192.168.1.3:50051 clients=17143worker1 # [ 158.467907] nix-grpc-daemon[1886]: ts=2026-09-22T14:52:58Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:511607144worker1 # [ 158.471169] nix-grpc-daemon[1886]: ts=2026-09-22T14:52:58Z level=debug event=want client=1 drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv cached=07145worker1 # [ 158.477328] nix-grpc-daemon[1886]: ts=2026-09-22T14:52:58Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:511607146worker1 # [ 158.481707] nix-daemon[869]: reaped child process 2011, status = succeeded7147worker1 # [ 158.483627] nix-grpc-daemon[1886]: ts=2026-09-22T14:52:58Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:511607148worker1 # [ 158.488069] nix-daemon[869]: accepted connection from pid 1886, user nix-grpc-daemon (trusted)7149worker1 # [ 158.575814] nix-daemon[869]: accepted connection from pid 2038, user nix-grpc-daemon (trusted)7150worker1 # [ 158.588787] nix-daemon[869]: reaped child process 2041, status = succeeded7151worker1 # [ 158.673605] nix-daemon[869]: accepted connection from pid 2043, user nix-grpc-daemon (trusted)7152worker1 # [ 158.687394] nix-daemon[869]: reaped child process 2046, status = succeeded7153lb # [ 158.727658] niks3-server[1099]: 2026/09/22 14:52:58 INFO Received uploads request method=POST path=/api/pending_closures7154worker1 # [ 158.701037] nix-grpc-daemon[1951]: time=2026-09-22T14:52:58.818Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7155worker1 # [ 158.702933] nix-grpc-daemon[1951]: time=2026-09-22T14:52:58.819Z level=INFO msg="Uploading j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv (504B)"7156lb # [ 158.751702] niks3-server[1099]: 2026/09/22 14:52:58 INFO Registered completed upload object_key=nar/0qr7hdc916yrqpvqx2bwc5l0rwy3zm6rsdlcf8rk9iid53b80xsn.nar.zst7157lb # [ 158.763843] niks3-server[1099]: 2026/09/22 14:52:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/66/sign7158lb # [ 158.768306] niks3-server[1099]: 2026/09/22 14:52:58 INFO Signed narinfos id=66 count=17159worker1 # [ 158.737188] nix-grpc-daemon[1951]: time=2026-09-22T14:52:58.854Z level=INFO msg="Uploading 1 narinfos"7160lb # [ 158.780144] niks3-server[1099]: 2026/09/22 14:52:58 INFO Registered completed upload object_key=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz.ls7161lb # [ 158.789858] niks3-server[1099]: 2026/09/22 14:52:58 INFO Received complete upload request method=POST path=/api/pending_closures/66/complete7162lb # [ 158.797966] niks3-server[1099]: 2026/09/22 14:52:58 INFO Completed upload id=667163worker1 # [ 158.765443] nix-grpc-daemon[1951]: time=2026-09-22T14:52:58.882Z level=INFO msg="Upload complete. (270ms)"7164lb # [ 158.800591] niks3-server[1099]: 2026/09/22 14:52:58 INFO Registered completed upload object_key=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz.narinfo7165worker1 # [ 158.769418] nix-grpc-daemon[1886]: ts=2026-09-22T14:52:58Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:51160 duration_s=0 paths=1 nar_bytes_in=5047166worker1 # [ 158.775153] nix-grpc-daemon[1886]: ts=2026-09-22T14:52:58Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:511607167worker1 # [ 158.783084] nix-daemon[869]: accepted connection from pid 1886, user nix-grpc-daemon (trusted)7168worker1 # [ 158.790536] nix-daemon[869]: reaped child process 2036, status = succeeded7169worker1 # [ 158.795409] nix-daemon[869]: accepted connection from pid 1886, user nix-grpc-daemon (trusted)7170worker1 # [ 160.241905] nix-daemon[869]: reaped child process 2050, status = succeeded7171worker1 # [ 160.323232] nix-daemon[869]: accepted connection from pid 2066, user nix-grpc-daemon (trusted)7172worker1 # [ 160.337237] nix-daemon[869]: reaped child process 2069, status = succeeded7173lb # [ 160.379910] niks3-server[1099]: 2026/09/22 14:52:59 INFO Received uploads request method=POST path=/api/pending_closures7174worker1 # [ 160.354826] nix-grpc-daemon[1951]: time=2026-09-22T14:53:00.471Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7175worker1 # [ 160.356452] nix-grpc-daemon[1951]: time=2026-09-22T14:53:00.473Z level=INFO msg="Uploading pnim4ls33hwi0fl1wv96w4yj4afz9w4m-farm-a (128B)"7176lb # [ 160.419885] niks3-server[1099]: 2026/09/22 14:52:59 INFO Registered completed upload object_key=nar/1j2671k9mycq8lxyfx5k6a66d5narfqw2wqwlpwxgwzi93vnvbdn.nar.zst7177lb # [ 160.428383] niks3-server[1099]: 2026/09/22 14:52:59 INFO Registered completed upload object_key=log/j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv7178lb # [ 160.438367] niks3-server[1099]: 2026/09/22 14:52:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/67/sign7179lb # [ 160.443736] niks3-server[1099]: 2026/09/22 14:52:59 INFO Signed narinfos id=67 count=17180worker1 # [ 160.412063] nix-grpc-daemon[1951]: time=2026-09-22T14:53:00.529Z level=INFO msg="Uploading 1 narinfos"7181lb # [ 160.447916] niks3-server[1099]: 2026/09/22 14:52:59 INFO Registered completed upload object_key=pnim4ls33hwi0fl1wv96w4yj4afz9w4m.ls7182lb # [ 160.459545] niks3-server[1099]: 2026/09/22 14:52:59 INFO Received complete upload request method=POST path=/api/pending_closures/67/complete7183lb # [ 160.462336] niks3-server[1099]: 2026/09/22 14:52:59 INFO Completed upload id=677184lb # [ 160.463967] niks3-server[1099]: 2026/09/22 14:52:59 INFO Registered completed upload object_key=pnim4ls33hwi0fl1wv96w4yj4afz9w4m.narinfo7185worker1 # [ 160.429568] nix-grpc-daemon[1951]: time=2026-09-22T14:53:00.546Z level=INFO msg="Upload complete. (225ms)"7186worker1 # [ 160.441077] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:00Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:51160 duration_s=1 drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv assign_id=3 outputs=17187worker1 # [ 160.449546] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:00Z level=debug event=assigned drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv worker=192.168.1.3:50051 clients=17188worker1 # [ 160.453689] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:511607189worker1 # [ 160.458257] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:00Z level=debug event=want client=1 drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv cached=07190worker1 # [ 160.468945] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:00Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:511607191worker1 # [ 160.471481] nix-daemon[869]: reaped child process 2048, status = succeeded7192worker1 # [ 160.476639] nix-grpc-daemon[1951]: time=2026-09-22T14:53:00.593Z level=INFO msg="All 1 paths already cached"7193worker1 # [ 160.481340] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:00Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:511607194worker1 # [ 160.486425] nix-daemon[869]: accepted connection from pid 1886, user nix-grpc-daemon (trusted)7195worker1 # [ 160.577542] nix-daemon[869]: accepted connection from pid 2075, user nix-grpc-daemon (trusted)7196worker1 # [ 160.590985] nix-daemon[869]: reaped child process 2078, status = succeeded7197worker1 # [ 160.681782] nix-daemon[869]: accepted connection from pid 2080, user nix-grpc-daemon (trusted)7198worker1 # [ 160.695462] nix-daemon[869]: reaped child process 2083, status = succeeded7199lb # [ 160.736921] niks3-server[1099]: 2026/09/22 14:53:00 INFO Received uploads request method=POST path=/api/pending_closures7200worker1 # [ 160.710054] nix-grpc-daemon[1951]: time=2026-09-22T14:53:00.827Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7201worker1 # [ 160.711544] nix-grpc-daemon[1951]: time=2026-09-22T14:53:00.827Z level=INFO msg="Uploading 6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv (624B)"7202lb # [ 160.765970] niks3-server[1099]: 2026/09/22 14:53:00 INFO Registered completed upload object_key=nar/06n0qn8haxwg1vfa3qwa7h14mzz5fz7nizpv8bn995f4gddci6y6.nar.zst7203lb # [ 160.774124] niks3-server[1099]: 2026/09/22 14:53:00 INFO Received sign narinfos request method=POST path=/api/pending_closures/68/sign7204lb # [ 160.779256] niks3-server[1099]: 2026/09/22 14:53:00 INFO Signed narinfos id=68 count=17205worker1 # [ 160.748081] nix-grpc-daemon[1951]: time=2026-09-22T14:53:00.865Z level=INFO msg="Uploading 1 narinfos"7206lb # [ 160.792677] niks3-server[1099]: 2026/09/22 14:53:00 INFO Registered completed upload object_key=6cjjd3vbzygfms2ccxvnmabbgq5pv310.ls7207lb # [ 160.799902] niks3-server[1099]: 2026/09/22 14:53:00 INFO Received complete upload request method=POST path=/api/pending_closures/68/complete7208worker1 # [ 160.772690] nix-grpc-daemon[1951]: time=2026-09-22T14:53:00.889Z level=INFO msg="Upload complete. (281ms)"7209lb # [ 160.807481] niks3-server[1099]: 2026/09/22 14:53:00 INFO Completed upload id=687210lb # [ 160.811120] niks3-server[1099]: 2026/09/22 14:53:00 INFO Registered completed upload object_key=6cjjd3vbzygfms2ccxvnmabbgq5pv310.narinfo7211worker1 # [ 160.774970] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:00Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:51160 duration_s=0 paths=1 nar_bytes_in=6247212worker1 # [ 160.780655] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:511607213worker1 # [ 160.790754] nix-daemon[869]: accepted connection from pid 1886, user nix-grpc-daemon (trusted)7214worker1 # [ 160.795126] nix-daemon[869]: reaped child process 2073, status = succeeded7215worker1 # [ 160.800312] nix-daemon[869]: accepted connection from pid 1886, user nix-grpc-daemon (trusted)7216worker1 # [ 161.415424] nix-daemon[869]: reaped child process 2087, status = succeeded7217worker1 # [ 161.493219] nix-daemon[869]: accepted connection from pid 2103, user nix-grpc-daemon (trusted)7218worker1 # [ 161.506985] nix-daemon[869]: reaped child process 2106, status = succeeded7219lb # [ 161.548904] niks3-server[1099]: 2026/09/22 14:53:00 INFO Received uploads request method=POST path=/api/pending_closures7220worker1 # [ 161.522733] nix-grpc-daemon[1951]: time=2026-09-22T14:53:01.639Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7221worker1 # [ 161.524219] nix-grpc-daemon[1951]: time=2026-09-22T14:53:01.641Z level=INFO msg="Uploading 7gfg490kigdqainqjk27vmks1h3wpv5q-farm-b (176B)"7222lb # [ 161.588339] niks3-server[1099]: 2026/09/22 14:53:00 INFO Registered completed upload object_key=log/6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv7223lb # [ 161.594698] niks3-server[1099]: 2026/09/22 14:53:00 INFO Registered completed upload object_key=nar/07q9c11jk9djv06znvadbhgp1j3kwqf4z0z60kwx2sjqjkpssnf0.nar.zst7224lb # [ 161.605103] niks3-server[1099]: 2026/09/22 14:53:00 INFO Received sign narinfos request method=POST path=/api/pending_closures/69/sign7225lb # [ 161.608050] niks3-server[1099]: 2026/09/22 14:53:00 INFO Signed narinfos id=69 count=17226worker1 # [ 161.574089] nix-grpc-daemon[1951]: time=2026-09-22T14:53:01.691Z level=INFO msg="Uploading 1 narinfos"7227lb # [ 161.614064] niks3-server[1099]: 2026/09/22 14:53:00 INFO Registered completed upload object_key=7gfg490kigdqainqjk27vmks1h3wpv5q.ls7228lb # [ 161.616911] niks3-server[1099]: 2026/09/22 14:53:00 INFO Received complete upload request method=POST path=/api/pending_closures/69/complete7229lb # [ 161.620930] niks3-server[1099]: 2026/09/22 14:53:00 INFO Completed upload id=697230worker1 # [ 161.586885] nix-grpc-daemon[1951]: time=2026-09-22T14:53:01.704Z level=INFO msg="Upload complete. (180ms)"7231lb # [ 161.623055] niks3-server[1099]: 2026/09/22 14:53:00 INFO Registered completed upload object_key=7gfg490kigdqainqjk27vmks1h3wpv5q.narinfo7232worker1 # [ 161.594676] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:01Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:51160 duration_s=0 drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv assign_id=4 outputs=17233worker1 # [ 161.601616] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:01Z level=debug event=assigned drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv worker=192.168.1.3:50051 clients=17234worker1 # [ 161.606263] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:01Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:511607235worker1 # [ 161.610130] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:01Z level=debug event=want client=1 drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv cached=07236worker1 # [ 161.616263] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:01Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:511607237worker1 # [ 161.618205] nix-daemon[869]: reaped child process 2085, status = succeeded7238worker1 # [ 161.623490] nix-grpc-daemon[1951]: time=2026-09-22T14:53:01.740Z level=INFO msg="All 1 paths already cached"7239worker1 # [ 161.627395] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:01Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:511607240worker1 # [ 161.631042] nix-daemon[869]: accepted connection from pid 1886, user nix-grpc-daemon (trusted)7241worker1 # [ 161.717486] nix-daemon[869]: accepted connection from pid 2112, user nix-grpc-daemon (trusted)7242worker1 # [ 161.731926] nix-daemon[869]: reaped child process 2115, status = succeeded7243worker1 # [ 161.814265] nix-daemon[869]: accepted connection from pid 2117, user nix-grpc-daemon (trusted)7244worker1 # [ 161.827848] nix-daemon[869]: reaped child process 2120, status = succeeded7245lb # [ 161.868232] niks3-server[1099]: 2026/09/22 14:53:01 INFO Received uploads request method=POST path=/api/pending_closures7246worker1 # [ 161.843703] nix-grpc-daemon[1951]: time=2026-09-22T14:53:01.960Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7247worker1 # [ 161.845370] nix-grpc-daemon[1951]: time=2026-09-22T14:53:01.962Z level=INFO msg="Uploading 6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv (624B)"7248lb # [ 161.905596] niks3-server[1099]: 2026/09/22 14:53:01 INFO Received sign narinfos request method=POST path=/api/pending_closures/70/sign7249lb # [ 161.912238] niks3-server[1099]: 2026/09/22 14:53:01 INFO Registered completed upload object_key=nar/0w4dmpczwycj1p7ngvcwdynpk5hh6r5ivyz44k3lx2gh5vpk4l0i.nar.zst7250worker1 # [ 161.881784] nix-grpc-daemon[1951]: time=2026-09-22T14:53:01.999Z level=INFO msg="Uploading 1 narinfos"7251lb # [ 161.915972] niks3-server[1099]: 2026/09/22 14:53:01 INFO Signed narinfos id=70 count=17252lb # [ 161.928173] niks3-server[1099]: 2026/09/22 14:53:01 INFO Registered completed upload object_key=6cs4w0lyx4kldp39y5k29p7c0234cffw.ls7253lb # [ 161.931271] niks3-server[1099]: 2026/09/22 14:53:01 INFO Received complete upload request method=POST path=/api/pending_closures/70/complete7254lb # [ 161.934814] niks3-server[1099]: 2026/09/22 14:53:01 INFO Completed upload id=707255worker1 # [ 161.901840] nix-grpc-daemon[1951]: time=2026-09-22T14:53:02.019Z level=INFO msg="Upload complete. (265ms)"7256lb # [ 161.936498] niks3-server[1099]: 2026/09/22 14:53:01 INFO Registered completed upload object_key=6cs4w0lyx4kldp39y5k29p7c0234cffw.narinfo7257worker1 # [ 161.904914] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:02Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:51160 duration_s=0 paths=1 nar_bytes_in=6247258worker1 # [ 161.909810] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:511607259worker1 # [ 161.920530] nix-daemon[869]: accepted connection from pid 1886, user nix-grpc-daemon (trusted)7260worker1 # [ 161.924328] nix-daemon[869]: reaped child process 2110, status = succeeded7261worker1 # [ 161.929199] nix-daemon[869]: accepted connection from pid 1886, user nix-grpc-daemon (trusted)7262worker1 # [ 163.144580] nix-daemon[869]: reaped child process 2124, status = succeeded7263worker1 # [ 163.221784] nix-daemon[869]: accepted connection from pid 2140, user nix-grpc-daemon (trusted)7264worker1 # [ 163.235180] nix-daemon[869]: reaped child process 2143, status = succeeded7265lb # [ 163.277542] niks3-server[1099]: 2026/09/22 14:53:02 INFO Received uploads request method=POST path=/api/pending_closures7266worker1 # [ 163.251476] nix-grpc-daemon[1951]: time=2026-09-22T14:53:03.368Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7267worker1 # [ 163.253387] nix-grpc-daemon[1951]: time=2026-09-22T14:53:03.370Z level=INFO msg="Uploading xxjh88rg0bhdcm1c6b12clv7l5hfbaa9-farm-b (176B)"7268lb # [ 163.316563] niks3-server[1099]: 2026/09/22 14:53:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/71/sign7269lb # [ 163.332144] niks3-server[1099]: 2026/09/22 14:53:02 INFO Signed narinfos id=71 count=17270worker1 # [ 163.300522] nix-grpc-daemon[1951]: time=2026-09-22T14:53:03.417Z level=INFO msg="Uploading 1 narinfos"7271lb # [ 163.336807] niks3-server[1099]: 2026/09/22 14:53:02 INFO Registered completed upload object_key=nar/1a8ws2jbn6nqpggi64yfgxvlwny3nffvyw6svm59is55hih5ppq4.nar.zst7272lb # [ 163.341851] niks3-server[1099]: 2026/09/22 14:53:02 INFO Registered completed upload object_key=log/6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv7273lb # [ 163.346135] niks3-server[1099]: 2026/09/22 14:53:02 INFO Registered completed upload object_key=xxjh88rg0bhdcm1c6b12clv7l5hfbaa9.ls7274lb # [ 163.355479] niks3-server[1099]: 2026/09/22 14:53:02 INFO Received complete upload request method=POST path=/api/pending_closures/71/complete7275lb # [ 163.360357] niks3-server[1099]: 2026/09/22 14:53:02 INFO Completed upload id=717276worker1 # [ 163.326175] nix-grpc-daemon[1951]: time=2026-09-22T14:53:03.443Z level=INFO msg="Upload complete. (190ms)"7277lb # [ 163.361921] niks3-server[1099]: 2026/09/22 14:53:02 INFO Registered completed upload object_key=xxjh88rg0bhdcm1c6b12clv7l5hfbaa9.narinfo7278worker1 # [ 163.328902] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:03Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:51160 duration_s=1 drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv assign_id=5 outputs=17279worker1 # [ 163.332550] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:03Z level=debug event=assigned drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv worker=192.168.1.3:50051 clients=17280worker1 # [ 163.336623] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:511607281worker1 # [ 163.339709] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:03Z level=debug event=want client=1 drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv cached=07282worker1 # [ 163.345277] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:03Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:511607283worker1 # [ 163.348257] nix-daemon[869]: reaped child process 2122, status = succeeded7284worker1 # [ 163.351712] nix-grpc-daemon[1951]: time=2026-09-22T14:53:03.469Z level=INFO msg="All 1 paths already cached"7285worker1 # [ 163.355882] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:03Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:511607286worker1 # [ 163.360129] nix-daemon[869]: accepted connection from pid 1886, user nix-grpc-daemon (trusted)7287worker1 # [ 163.446906] nix-daemon[869]: accepted connection from pid 2149, user nix-grpc-daemon (trusted)7288worker1 # [ 163.459449] nix-daemon[869]: reaped child process 2152, status = succeeded7289worker1 # [ 163.545338] nix-daemon[869]: accepted connection from pid 2154, user nix-grpc-daemon (trusted)7290worker1 # [ 163.557604] nix-daemon[869]: reaped child process 2157, status = succeeded7291lb # [ 163.600845] niks3-server[1099]: 2026/09/22 14:53:02 INFO Received uploads request method=POST path=/api/pending_closures7292worker1 # [ 163.573675] nix-grpc-daemon[1951]: time=2026-09-22T14:53:03.690Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7293worker1 # [ 163.575239] nix-grpc-daemon[1951]: time=2026-09-22T14:53:03.692Z level=INFO msg="Uploading 8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv (624B)"7294lb # [ 163.632753] niks3-server[1099]: 2026/09/22 14:53:03 INFO Received sign narinfos request method=POST path=/api/pending_closures/72/sign7295worker1 # [ 163.605360] nix-grpc-daemon[1951]: time=2026-09-22T14:53:03.722Z level=INFO msg="Uploading 1 narinfos"7296lb # [ 163.639535] niks3-server[1099]: 2026/09/22 14:53:03 INFO Signed narinfos id=72 count=17297lb # [ 163.645093] niks3-server[1099]: 2026/09/22 14:53:03 INFO Registered completed upload object_key=nar/1dh50s9lbv3zd84hw5q9axrhfi7ms38cl3gyrb9gas00gni4siz2.nar.zst7298lb # [ 163.650817] niks3-server[1099]: 2026/09/22 14:53:03 INFO Registered completed upload object_key=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz.ls7299lb # [ 163.663399] niks3-server[1099]: 2026/09/22 14:53:03 INFO Received complete upload request method=POST path=/api/pending_closures/72/complete7300worker1 # [ 163.634162] nix-grpc-daemon[1951]: time=2026-09-22T14:53:03.751Z level=INFO msg="Upload complete. (268ms)"7301lb # [ 163.668829] niks3-server[1099]: 2026/09/22 14:53:03 INFO Completed upload id=727302worker1 # [ 163.636302] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:03Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:51160 duration_s=0 paths=1 nar_bytes_in=6247303lb # [ 163.674381] niks3-server[1099]: 2026/09/22 14:53:03 INFO Registered completed upload object_key=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz.narinfo7304worker1 # [ 163.640604] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:511607305worker1 # [ 163.650727] nix-daemon[869]: accepted connection from pid 1886, user nix-grpc-daemon (trusted)7306worker1 # [ 163.653038] nix-daemon[869]: reaped child process 2147, status = succeeded7307worker1 # [ 163.658492] nix-daemon[869]: accepted connection from pid 1886, user nix-grpc-daemon (trusted)7308worker1 # [ 164.876821] nix-daemon[869]: reaped child process 2161, status = succeeded7309worker1 # [ 164.947651] nix-daemon[869]: accepted connection from pid 2178, user nix-grpc-daemon (trusted)7310worker1 # [ 164.959843] nix-daemon[869]: reaped child process 2181, status = succeeded7311lb # [ 165.001808] niks3-server[1099]: 2026/09/22 14:53:04 INFO Received uploads request method=POST path=/api/pending_closures7312worker1 # [ 164.975578] nix-grpc-daemon[1951]: time=2026-09-22T14:53:05.092Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7313worker1 # [ 164.977266] nix-grpc-daemon[1951]: time=2026-09-22T14:53:05.094Z level=INFO msg="Uploading wha0h9jdcpcy0srjm92hg7pyknwdqp6h-farm-b (176B)"7314lb # [ 165.044810] niks3-server[1099]: 2026/09/22 14:53:04 INFO Received sign narinfos request method=POST path=/api/pending_closures/73/sign7315lb # [ 165.050766] niks3-server[1099]: 2026/09/22 14:53:04 INFO Signed narinfos id=73 count=17316worker1 # [ 165.020680] nix-grpc-daemon[1951]: time=2026-09-22T14:53:05.137Z level=INFO msg="Uploading 1 narinfos"7317lb # [ 165.056062] niks3-server[1099]: 2026/09/22 14:53:04 INFO Registered completed upload object_key=log/8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv7318lb # [ 165.062258] niks3-server[1099]: 2026/09/22 14:53:04 INFO Registered completed upload object_key=nar/1xpjwixqgm25ifjqv3ldg9cbq2njxf26272sngvq3zhy6zr1mvwg.nar.zst7319lb # [ 165.071058] niks3-server[1099]: 2026/09/22 14:53:04 INFO Registered completed upload object_key=wha0h9jdcpcy0srjm92hg7pyknwdqp6h.ls7320lb # [ 165.080614] niks3-server[1099]: 2026/09/22 14:53:04 INFO Received complete upload request method=POST path=/api/pending_closures/73/complete7321lb # [ 165.087673] niks3-server[1099]: 2026/09/22 14:53:04 INFO Completed upload id=737322lb # [ 165.089731] niks3-server[1099]: 2026/09/22 14:53:04 INFO Registered completed upload object_key=wha0h9jdcpcy0srjm92hg7pyknwdqp6h.narinfo7323worker1 # [ 165.054975] nix-grpc-daemon[1951]: time=2026-09-22T14:53:05.172Z level=INFO msg="Upload complete. (189ms)"7324worker1 # [ 165.064826] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:05Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:51160 duration_s=1 drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv assign_id=6 outputs=17325worker1 # [ 165.074352] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:05Z level=debug event=assigned drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv worker=192.168.1.3:50051 clients=17326worker1 # [ 165.079199] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:05Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:511607327worker1 # [ 165.082489] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:05Z level=debug event=want client=1 drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv cached=07328worker1 # [ 165.093314] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:05Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:511607329worker1 # [ 165.096391] nix-daemon[869]: reaped child process 2159, status = succeeded7330worker1 # [ 165.100863] nix-grpc-daemon[1951]: time=2026-09-22T14:53:05.218Z level=INFO msg="All 2 paths already cached"7331worker1 # [ 165.105380] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:05Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:511607332worker1 # [ 165.110055] nix-daemon[869]: accepted connection from pid 1886, user nix-grpc-daemon (trusted)7333worker1 # [ 165.191288] nix-daemon[869]: accepted connection from pid 2187, user nix-grpc-daemon (trusted)7334worker1 # [ 165.204356] nix-daemon[869]: reaped child process 2190, status = succeeded7335worker1 # [ 165.283139] nix-daemon[869]: accepted connection from pid 2192, user nix-grpc-daemon (trusted)7336worker1 # [ 165.295464] nix-daemon[869]: reaped child process 2195, status = succeeded7337lb # [ 165.337078] niks3-server[1099]: 2026/09/22 14:53:04 INFO Received uploads request method=POST path=/api/pending_closures7338worker1 # [ 165.309842] nix-grpc-daemon[1951]: time=2026-09-22T14:53:05.427Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7339worker1 # [ 165.311380] nix-grpc-daemon[1951]: time=2026-09-22T14:53:05.428Z level=INFO msg="Uploading 2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv (752B)"7340lb # [ 165.363283] niks3-server[1099]: 2026/09/22 14:53:04 INFO Registered completed upload object_key=nar/10gk7wwjbmc835pm215qvrdyjshy6nblydlws97ya3kcny5wz30c.nar.zst7341lb # [ 165.374678] niks3-server[1099]: 2026/09/22 14:53:04 INFO Received sign narinfos request method=POST path=/api/pending_closures/74/sign7342worker1 # [ 165.347772] nix-grpc-daemon[1951]: time=2026-09-22T14:53:05.465Z level=INFO msg="Uploading 1 narinfos"7343lb # [ 165.382068] niks3-server[1099]: 2026/09/22 14:53:04 INFO Signed narinfos id=74 count=17344lb # [ 165.389170] niks3-server[1099]: 2026/09/22 14:53:04 INFO Registered completed upload object_key=2gb1j4yk6j00fr2sscj3nqixv67k9b9g.ls7345lb # [ 165.401109] niks3-server[1099]: 2026/09/22 14:53:04 INFO Received complete upload request method=POST path=/api/pending_closures/74/complete7346lb # [ 165.407167] niks3-server[1099]: 2026/09/22 14:53:04 INFO Completed upload id=747347worker1 # [ 165.372858] nix-grpc-daemon[1951]: time=2026-09-22T14:53:05.490Z level=INFO msg="Upload complete. (258ms)"7348lb # [ 165.411832] niks3-server[1099]: 2026/09/22 14:53:04 INFO Registered completed upload object_key=2gb1j4yk6j00fr2sscj3nqixv67k9b9g.narinfo7349worker1 # [ 165.376575] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:05Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:51160 duration_s=0 paths=1 nar_bytes_in=7527350worker1 # [ 165.382100] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:05Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:511607351worker1 # [ 165.391511] nix-daemon[869]: accepted connection from pid 1886, user nix-grpc-daemon (trusted)7352worker1 # [ 165.395468] nix-daemon[869]: reaped child process 2185, status = succeeded7353worker1 # [ 165.400221] nix-daemon[869]: accepted connection from pid 1886, user nix-grpc-daemon (trusted)7354worker1 # [ 166.232762] nix-daemon[869]: reaped child process 2199, status = succeeded7355worker1 # [ 166.322576] nix-daemon[869]: accepted connection from pid 2215, user nix-grpc-daemon (trusted)7356worker1 # [ 166.335406] nix-daemon[869]: reaped child process 2218, status = succeeded7357lb # [ 166.379301] niks3-server[1099]: 2026/09/22 14:53:05 INFO Received uploads request method=POST path=/api/pending_closures7358worker1 # [ 166.352845] nix-grpc-daemon[1951]: time=2026-09-22T14:53:06.469Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7359worker1 # [ 166.354315] nix-grpc-daemon[1951]: time=2026-09-22T14:53:06.471Z level=INFO msg="Uploading 78nhq5a8myvxam7sxfszi5ihz2rlq2gs-farm-top (232B)"7360lb # [ 166.420758] niks3-server[1099]: 2026/09/22 14:53:05 INFO Registered completed upload object_key=log/2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv7361lb # [ 166.431120] niks3-server[1099]: 2026/09/22 14:53:05 INFO Registered completed upload object_key=nar/1gsz81ckdr6lfkhhqmxw3bg6xqqnlz2wx6dzhpycascv9pq13fg7.nar.zst7362lb # [ 166.439454] niks3-server[1099]: 2026/09/22 14:53:05 INFO Received sign narinfos request method=POST path=/api/pending_closures/75/sign7363lb # [ 166.447454] niks3-server[1099]: 2026/09/22 14:53:05 INFO Signed narinfos id=75 count=17364worker1 # [ 166.413399] nix-grpc-daemon[1951]: time=2026-09-22T14:53:06.530Z level=INFO msg="Uploading 1 narinfos"7365lb # [ 166.452108] niks3-server[1099]: 2026/09/22 14:53:05 INFO Registered completed upload object_key=78nhq5a8myvxam7sxfszi5ihz2rlq2gs.ls7366lb # [ 166.463711] niks3-server[1099]: 2026/09/22 14:53:05 INFO Received complete upload request method=POST path=/api/pending_closures/75/complete7367lb # [ 166.467069] niks3-server[1099]: 2026/09/22 14:53:05 INFO Completed upload id=757368worker1 # [ 166.432701] nix-grpc-daemon[1951]: time=2026-09-22T14:53:06.549Z level=INFO msg="Upload complete. (224ms)"7369lb # [ 166.469715] niks3-server[1099]: 2026/09/22 14:53:05 INFO Registered completed upload object_key=78nhq5a8myvxam7sxfszi5ihz2rlq2gs.narinfo7370worker1 # [ 166.441595] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:06Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:51160 duration_s=1 drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv assign_id=7 outputs=17371worker1 # [ 166.448310] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:06Z level=debug event=assigned drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv worker=192.168.1.3:50051 clients=17372worker1 # [ 166.459409] nix-daemon[869]: reaped child process 2197, status = succeeded7373worker1 # [ 166.484746] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:511607374worker1 # [ 166.496911] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:06Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:511607375worker1 # [ 166.509341] nix-grpc-daemon[1951]: time=2026-09-22T14:53:06.626Z level=INFO msg="All 2 paths already cached"7376worker1 # [ 166.516917] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:06Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:511607377worker1 # [ 166.528139] nix-daemon[869]: accepted connection from pid 1886, user nix-grpc-daemon (trusted)7378worker1 # [ 166.621857] nix-daemon[869]: accepted connection from pid 2222, user nix-grpc-daemon (trusted)7379worker1 # [ 166.636527] nix-daemon[869]: reaped child process 2225, status = succeeded7380worker1 # [ 166.713700] nix-daemon[869]: accepted connection from pid 2227, user nix-grpc-daemon (trusted)7381worker1 # [ 166.727124] nix-daemon[869]: reaped child process 2230, status = succeeded7382lb # [ 166.767933] niks3-server[1099]: 2026/09/22 14:53:06 INFO Received uploads request method=POST path=/api/pending_closures7383worker1 # [ 166.740768] nix-grpc-daemon[1951]: time=2026-09-22T14:53:06.858Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7384worker1 # [ 166.742668] nix-grpc-daemon[1951]: time=2026-09-22T14:53:06.859Z level=INFO msg="Uploading yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv (752B)"7385lb # [ 166.792431] niks3-server[1099]: 2026/09/22 14:53:06 INFO Registered completed upload object_key=nar/1ibq19fcqq0l9xim9ly5lzg1pk7qh90qsbpya453vnh8p619zvbs.nar.zst7386lb # [ 166.804950] niks3-server[1099]: 2026/09/22 14:53:06 INFO Received sign narinfos request method=POST path=/api/pending_closures/76/sign7387lb # [ 166.809830] niks3-server[1099]: 2026/09/22 14:53:06 INFO Signed narinfos id=76 count=17388worker1 # [ 166.778388] nix-grpc-daemon[1951]: time=2026-09-22T14:53:06.895Z level=INFO msg="Uploading 1 narinfos"7389lb # [ 166.822827] niks3-server[1099]: 2026/09/22 14:53:06 INFO Registered completed upload object_key=yjsciba0gpd99p30v1win6wnqyd4v7b6.ls7390lb # [ 166.830862] niks3-server[1099]: 2026/09/22 14:53:06 INFO Received complete upload request method=POST path=/api/pending_closures/76/complete7391worker1 # [ 166.801540] nix-grpc-daemon[1951]: time=2026-09-22T14:53:06.918Z level=INFO msg="Upload complete. (262ms)"7392lb # [ 166.835620] niks3-server[1099]: 2026/09/22 14:53:06 INFO Completed upload id=767393worker1 # [ 166.803670] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:06Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:51160 duration_s=0 paths=1 nar_bytes_in=7527394lb # [ 166.841630] niks3-server[1099]: 2026/09/22 14:53:06 INFO Registered completed upload object_key=yjsciba0gpd99p30v1win6wnqyd4v7b6.narinfo7395worker1 # [ 166.807665] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:511607396worker1 # [ 166.815093] nix-daemon[869]: accepted connection from pid 1886, user nix-grpc-daemon (trusted)7397worker1 # [ 166.816360] nix-daemon[869]: reaped child process 2220, status = succeeded7398worker1 # [ 166.821673] nix-daemon[869]: accepted connection from pid 1886, user nix-grpc-daemon (trusted)7399worker1 # [ 167.841093] nix-daemon[869]: reaped child process 2234, status = succeeded7400worker1 # [ 167.913379] nix-daemon[869]: accepted connection from pid 2250, user nix-grpc-daemon (trusted)7401worker1 # [ 167.928550] nix-daemon[869]: reaped child process 2253, status = succeeded7402lb # [ 167.967637] niks3-server[1099]: 2026/09/22 14:53:07 INFO Received uploads request method=POST path=/api/pending_closures7403worker1 # [ 167.941347] nix-grpc-daemon[1951]: time=2026-09-22T14:53:08.058Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7404worker1 # [ 167.942880] nix-grpc-daemon[1951]: time=2026-09-22T14:53:08.060Z level=INFO msg="Uploading qc1853riggbxq0nbn4qjdymnhx5pncg8-farm-top (232B)"7405lb # [ 167.993661] niks3-server[1099]: 2026/09/22 14:53:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/77/sign7406lb # [ 167.995842] niks3-server[1099]: 2026/09/22 14:53:07 INFO Registered completed upload object_key=nar/1k6m6033nbrw39zmvqlrbvmg39ygyn710za2hl2fkpsxfx40hl13.nar.zst7407lb # [ 167.997985] niks3-server[1099]: 2026/09/22 14:53:07 INFO Signed narinfos id=77 count=17408worker1 # [ 167.963893] nix-grpc-daemon[1951]: time=2026-09-22T14:53:08.081Z level=INFO msg="Uploading 1 narinfos"7409lb # [ 168.000578] niks3-server[1099]: 2026/09/22 14:53:07 INFO Registered completed upload object_key=qc1853riggbxq0nbn4qjdymnhx5pncg8.ls7410lb # [ 168.003077] niks3-server[1099]: 2026/09/22 14:53:07 INFO Registered completed upload object_key=log/yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv7411lb # [ 168.008062] niks3-server[1099]: 2026/09/22 14:53:07 INFO Received complete upload request method=POST path=/api/pending_closures/77/complete7412lb # [ 168.010696] niks3-server[1099]: 2026/09/22 14:53:07 INFO Completed upload id=777413worker1 # [ 167.976420] nix-grpc-daemon[1951]: time=2026-09-22T14:53:08.093Z level=INFO msg="Upload complete. (147ms)"7414lb # [ 168.014487] niks3-server[1099]: 2026/09/22 14:53:07 INFO Registered completed upload object_key=qc1853riggbxq0nbn4qjdymnhx5pncg8.narinfo7415worker1 # [ 167.979236] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:08Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:51160 duration_s=1 drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv assign_id=8 outputs=17416worker1 # [ 167.983290] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:08Z level=debug event=assigned drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv worker=192.168.1.3:50051 clients=17417worker1 # [ 167.989254] nix-daemon[869]: reaped child process 2232, status = succeeded7418worker1 # [ 168.025155] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:08Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:511607419worker1 # [ 168.033593] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:08Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:511607420worker1 # [ 168.043840] nix-grpc-daemon[1951]: time=2026-09-22T14:53:08.161Z level=INFO msg="All 2 paths already cached"7421worker1 # [ 168.050536] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:08Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:511607422worker1 # [ 168.059078] nix-daemon[869]: accepted connection from pid 1886, user nix-grpc-daemon (trusted)7423worker1 # [ 168.144483] nix-daemon[869]: accepted connection from pid 2257, user nix-grpc-daemon (trusted)7424worker1 # [ 168.157742] nix-daemon[869]: reaped child process 2260, status = succeeded7425worker1 # [ 168.238887] nix-daemon[869]: accepted connection from pid 2262, user nix-grpc-daemon (trusted)7426worker1 # [ 168.252251] nix-daemon[869]: reaped child process 2265, status = succeeded7427lb # [ 168.292707] niks3-server[1099]: 2026/09/22 14:53:07 INFO Received uploads request method=POST path=/api/pending_closures7428worker1 # [ 168.265673] nix-grpc-daemon[1951]: time=2026-09-22T14:53:08.382Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7429worker1 # [ 168.267599] nix-grpc-daemon[1951]: time=2026-09-22T14:53:08.384Z level=INFO msg="Uploading yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv (752B)"7430lb # [ 168.316901] niks3-server[1099]: 2026/09/22 14:53:07 INFO Registered completed upload object_key=nar/1msrgzhylalg20dw693shbnzf48k8pvzp85llwgrnc970rnrliy7.nar.zst7431lb # [ 168.329471] niks3-server[1099]: 2026/09/22 14:53:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/78/sign7432lb # [ 168.334461] niks3-server[1099]: 2026/09/22 14:53:07 INFO Signed narinfos id=78 count=17433worker1 # [ 168.303027] nix-grpc-daemon[1951]: time=2026-09-22T14:53:08.420Z level=INFO msg="Uploading 1 narinfos"7434lb # [ 168.346158] niks3-server[1099]: 2026/09/22 14:53:07 INFO Registered completed upload object_key=yym31x3l626nfpm249qapx1iw7y15mqb.ls7435lb # [ 168.355770] niks3-server[1099]: 2026/09/22 14:53:07 INFO Received complete upload request method=POST path=/api/pending_closures/78/complete7436worker1 # [ 168.326667] nix-grpc-daemon[1951]: time=2026-09-22T14:53:08.443Z level=INFO msg="Upload complete. (260ms)"7437lb # [ 168.361197] niks3-server[1099]: 2026/09/22 14:53:07 INFO Completed upload id=787438worker1 # [ 168.328797] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:08Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:51160 duration_s=0 paths=1 nar_bytes_in=7527439lb # [ 168.364735] niks3-server[1099]: 2026/09/22 14:53:07 INFO Registered completed upload object_key=yym31x3l626nfpm249qapx1iw7y15mqb.narinfo7440worker1 # [ 168.334813] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:08Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:511607441worker1 # [ 168.344593] nix-daemon[869]: accepted connection from pid 1886, user nix-grpc-daemon (trusted)7442worker1 # [ 168.346260] nix-daemon[869]: reaped child process 2255, status = succeeded7443worker1 # [ 168.351381] nix-daemon[869]: accepted connection from pid 1886, user nix-grpc-daemon (trusted)7444worker1 # [ 169.188736] nix-daemon[869]: reaped child process 2269, status = succeeded7445worker1 # [ 169.272264] nix-daemon[869]: accepted connection from pid 2285, user nix-grpc-daemon (trusted)7446worker1 # [ 169.284780] nix-daemon[869]: reaped child process 2288, status = succeeded7447lb # [ 169.326623] niks3-server[1099]: 2026/09/22 14:53:08 INFO Received uploads request method=POST path=/api/pending_closures7448worker1 # [ 169.301472] nix-grpc-daemon[1951]: time=2026-09-22T14:53:09.418Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7449worker1 # [ 169.302924] nix-grpc-daemon[1951]: time=2026-09-22T14:53:09.420Z level=INFO msg="Uploading 0y5v4dgfslb2x47r2h6j0zjvpzg6frg6-farm-top (232B)"7450lb # [ 169.369705] niks3-server[1099]: 2026/09/22 14:53:08 INFO Registered completed upload object_key=log/yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv7451lb # [ 169.379259] niks3-server[1099]: 2026/09/22 14:53:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/79/sign7452worker1 # [ 169.350550] nix-grpc-daemon[1951]: time=2026-09-22T14:53:09.467Z level=INFO msg="Uploading 1 narinfos"7453lb # [ 169.385150] niks3-server[1099]: 2026/09/22 14:53:08 INFO Signed narinfos id=79 count=17454lb # [ 169.392808] niks3-server[1099]: 2026/09/22 14:53:08 INFO Registered completed upload object_key=nar/1h7kcfflb8hb0ylxshiprrirnrs580cwzkz50cz6il8jag61wy50.nar.zst7455lb # [ 169.398754] niks3-server[1099]: 2026/09/22 14:53:08 INFO Registered completed upload object_key=0y5v4dgfslb2x47r2h6j0zjvpzg6frg6.ls7456lb # [ 169.406838] niks3-server[1099]: 2026/09/22 14:53:08 INFO Received complete upload request method=POST path=/api/pending_closures/79/complete7457lb # [ 169.410851] niks3-server[1099]: 2026/09/22 14:53:08 INFO Completed upload id=797458worker1 # [ 169.376468] nix-grpc-daemon[1951]: time=2026-09-22T14:53:09.493Z level=INFO msg="Upload complete. (218ms)"7459lb # [ 169.412850] niks3-server[1099]: 2026/09/22 14:53:08 INFO Registered completed upload object_key=0y5v4dgfslb2x47r2h6j0zjvpzg6frg6.narinfo7460worker1 # [ 169.379077] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:09Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:51160 duration_s=1 drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv assign_id=9 outputs=17461worker1 # [ 169.386531] nix-daemon[869]: reaped child process 2267, status = succeeded7462client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&1, in 15.84 seconds)7463worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics7464worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.02 seconds)7465client: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag f5 --arg features '["gpu"]'7466client # [ 169.568768] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build -L --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&restart-grace=30" --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag f5 --arg features "[\"gpu\"]".7467client # Running as unit: gpu.service; invocation ID: a47ba2fca73b4bab82eb7d32303ec2947468client: (finished: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag f5 --arg features '["gpu"]', in 0.06 seconds)7469lb: must succeed: curl -sf localhost:9901/clusters7470lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)7471lb: must succeed: curl -sf localhost:9901/clusters7472lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)7473worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics7474worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.02 seconds)7475worker1 # [ 169.668657] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:09Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:511607476client # [ 169.702290] nix[1532]: don't know how to build these paths:7477client # [ 169.703158] nix[1532]: /nix/store/ynb3m2rijlmsagnc1dpj672gknpqrkcv-farm-top.drv7478worker1 # [ 169.675695] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:09Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:511607479client # [ 169.723243] nix[1532]: warning: z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv: no connected, non-draining worker for system 'x86_64-linux' offers features {gpu}, waiting up to 30s for a worker7480worker1 # [ 169.692573] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:09Z level=debug event=want client=2 drv=z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv cached=07481lb: must succeed: curl -sf localhost:9901/clusters7482lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)7483lb: must succeed: curl -sf localhost:9901/clusters7484lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)7485worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics7486worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)7487client: waiting for success: systemctl show -p Result gpu | grep -q exit-code7488lb # [ 175.805223] envoy[905]: conn peer=[2001:db8:1::1]:33436 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=27025 tx=12318 ms=156557489client # [ 199.724724] nix[1532]: z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv: no connected, non-draining worker for system 'x86_64-linux' offers features {gpu}7490client # [ 199.733175] nix[1532]: error: dependency 'z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv' failed7491client # [ 199.763691] systemd[1]: gpu.service: Main process exited, code=exited, status=1/FAILURE7492client # [ 199.767672] systemd[1]: gpu.service: Failed with result 'exit-code'.7493client # [ 199.771571] systemd[1]: gpu.service: Consumed 134ms CPU time over 30.198s wall clock time, 35.6M memory peak, 5K incoming IP traffic, 5K outgoing IP traffic.7494client: (finished: waiting for success: systemctl show -p Result gpu | grep -q exit-code, in 29.28 seconds)7495client: must succeed: journalctl -u gpu -o cat7496client: (finished: must succeed: journalctl -u gpu -o cat, in 0.02 seconds)7497lb: must succeed: curl -sf localhost:9901/clusters7498lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)7499lb: must succeed: curl -sf localhost:9901/clusters7500lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)7501worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics7502worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.02 seconds)7503(finished: subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does, in 46.54 seconds)7504subtest: low disk drains a worker and builds go to the other7505worker1: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill7506worker1: (finished: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill, in 0.92 seconds)7507worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free'7508worker1 # [ 201.452282] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:41Z level=info event=unhealthy reason=min_free7509worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free', in 1.07 seconds)7510client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag drain >&27511client # don't know how to build these paths:7512client # /nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv7513worker2 # [ 202.337437] nix-grpc-daemon[1597]: ts=2026-09-22T14:53:42Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:367047514worker1 # [ 202.323291] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:42Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:511607515worker1 # [ 202.343158] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:42Z level=debug event=want client=3 drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv cached=07516worker1 # [ 202.347399] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:42Z level=debug event=assigned drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv worker=192.168.1.4:50051 clients=17517worker2 # [ 202.406930] nix-grpc-daemon[1597]: ts=2026-09-22T14:53:42Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:367047518worker2 # [ 202.419549] nix-grpc-daemon[1597]: ts=2026-09-22T14:53:42Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:367047519client # copying 1 paths to 'lb:50051'...7520client # copying path '/nix/store/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv' to 'lb:50051'...7521worker2 # [ 202.432256] nix-grpc-daemon[1597]: ts=2026-09-22T14:53:42Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:367047522worker2 # [ 202.443217] nix-daemon[850]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)7523worker2 # [ 202.546040] nix-daemon[850]: accepted connection from pid 1870, user nix-grpc-daemon (trusted)7524worker2 # [ 202.560181] nix-daemon[850]: reaped child process 1873, status = succeeded7525worker2 # [ 202.641862] nix-daemon[850]: accepted connection from pid 1875, user nix-grpc-daemon (trusted)7526worker2 # [ 202.654479] nix-daemon[850]: reaped child process 1878, status = succeeded7527lb # [ 202.679647] niks3-server[1099]: 2026/09/22 14:53:42 INFO Received uploads request method=POST path=/api/pending_closures7528worker2 # [ 202.670693] nix-grpc-daemon[1685]: time=2026-09-22T14:53:42.576Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7529worker2 # [ 202.672707] nix-grpc-daemon[1685]: time=2026-09-22T14:53:42.578Z level=INFO msg="Uploading a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv (512B)"7530lb # [ 202.709605] niks3-server[1099]: 2026/09/22 14:53:42 INFO Registered completed upload object_key=nar/1qxv3rw2cwpq06kampqz63l69ci673kh4xz41xmflqybqycfk573.nar.zst7531lb # [ 202.729936] niks3-server[1099]: 2026/09/22 14:53:42 INFO Received sign narinfos request method=POST path=/api/pending_closures/80/sign7532worker2 # [ 202.718556] nix-grpc-daemon[1685]: time=2026-09-22T14:53:42.624Z level=INFO msg="Uploading 1 narinfos"7533lb # [ 202.734055] niks3-server[1099]: 2026/09/22 14:53:42 INFO Signed narinfos id=80 count=17534lb # [ 202.744951] niks3-server[1099]: 2026/09/22 14:53:42 INFO Received complete upload request method=POST path=/api/pending_closures/80/complete7535lb # [ 202.750603] niks3-server[1099]: 2026/09/22 14:53:42 INFO Completed upload id=807536worker2 # [ 202.735337] nix-grpc-daemon[1685]: time=2026-09-22T14:53:42.641Z level=INFO msg="Upload complete. (280ms)"7537worker2 # [ 202.739523] nix-grpc-daemon[1597]: ts=2026-09-22T14:53:42Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36704 duration_s=0 paths=1 nar_bytes_in=5127538lb # [ 202.760958] niks3-server[1099]: 2026/09/22 14:53:42 INFO Registered completed upload object_key=a33ymyck2p14d34j7jl4zc8zvry6cysy.ls7539worker2 # [ 202.744465] nix-grpc-daemon[1597]: ts=2026-09-22T14:53:42Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:367047540lb # [ 202.763863] niks3-server[1099]: 2026/09/22 14:53:42 INFO Registered completed upload object_key=a33ymyck2p14d34j7jl4zc8zvry6cysy.narinfo7541worker2 # [ 202.756140] nix-daemon[850]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)7542worker2 # [ 202.760760] nix-daemon[850]: reaped child process 1868, status = succeeded7543worker2 # [ 202.765048] nix-daemon[850]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)7544client # building '/nix/store/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv' on lb:50051...7545client # farm-a> worker2: building a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv7546client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds7547client # farm-a> LOG-farm-a-drain7548worker2 # [ 204.216465] nix-daemon[850]: reaped child process 1882, status = succeeded7549worker2 # [ 204.273085] nix-daemon[850]: accepted connection from pid 1898, user nix-grpc-daemon (trusted)7550worker2 # [ 204.285382] nix-daemon[850]: reaped child process 1901, status = succeeded7551lb # [ 204.307902] niks3-server[1099]: 2026/09/22 14:53:43 INFO Received uploads request method=POST path=/api/pending_closures7552worker2 # [ 204.300927] nix-grpc-daemon[1685]: time=2026-09-22T14:53:44.207Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7553worker2 # [ 204.304512] nix-grpc-daemon[1685]: time=2026-09-22T14:53:44.209Z level=INFO msg="Uploading rjdc0hf9mcchhia9m3zkqy1q586i3z3r-farm-a (128B)"7554lb # [ 204.343156] niks3-server[1099]: 2026/09/22 14:53:43 INFO Registered completed upload object_key=log/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv7555lb # [ 204.359772] niks3-server[1099]: 2026/09/22 14:53:43 INFO Registered completed upload object_key=nar/1bf9z7y0ikx07b4zjag42zyrv2pq1ycw4461s579kfpmni9kz4dw.nar.zst7556lb # [ 204.368241] niks3-server[1099]: 2026/09/22 14:53:43 INFO Received sign narinfos request method=POST path=/api/pending_closures/81/sign7557lb # [ 204.372777] niks3-server[1099]: 2026/09/22 14:53:43 INFO Signed narinfos id=81 count=17558worker2 # [ 204.359918] nix-grpc-daemon[1685]: time=2026-09-22T14:53:44.266Z level=INFO msg="Uploading 1 narinfos"7559lb # [ 204.381765] niks3-server[1099]: 2026/09/22 14:53:43 INFO Registered completed upload object_key=rjdc0hf9mcchhia9m3zkqy1q586i3z3r.ls7560lb # [ 204.383840] niks3-server[1099]: 2026/09/22 14:53:43 INFO Received complete upload request method=POST path=/api/pending_closures/81/complete7561lb # [ 204.387403] niks3-server[1099]: 2026/09/22 14:53:43 INFO Completed upload id=817562worker2 # [ 204.373333] nix-grpc-daemon[1685]: time=2026-09-22T14:53:44.279Z level=INFO msg="Upload complete. (200ms)"7563lb # [ 204.390811] niks3-server[1099]: 2026/09/22 14:53:43 INFO Registered completed upload object_key=rjdc0hf9mcchhia9m3zkqy1q586i3z3r.narinfo7564worker2 # [ 204.379609] nix-grpc-daemon[1597]: ts=2026-09-22T14:53:44Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36704 duration_s=1 drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv assign_id=10 outputs=17565worker1 # [ 204.367346] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:44Z level=debug event=want client=3 drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv cached=07566worker1 # [ 204.371864] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:44Z level=debug event=assigned drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv worker=192.168.1.4:50051 clients=17567worker2 # [ 204.395733] nix-daemon[850]: reaped child process 1880, status = succeeded7568worker2 # [ 204.428869] nix-grpc-daemon[1597]: ts=2026-09-22T14:53:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:367047569worker2 # [ 204.440605] nix-grpc-daemon[1597]: ts=2026-09-22T14:53:44Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:367047570worker2 # [ 204.457154] nix-grpc-daemon[1685]: time=2026-09-22T14:53:44.363Z level=INFO msg="All 1 paths already cached"7571client # copying 1 paths to 'lb:50051'...7572client # copying path '/nix/store/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv' to 'lb:50051'...7573worker2 # [ 204.465938] nix-grpc-daemon[1597]: ts=2026-09-22T14:53:44Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:367047574worker2 # [ 204.476274] nix-daemon[850]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)7575worker2 # [ 204.572354] nix-daemon[850]: accepted connection from pid 1905, user nix-grpc-daemon (trusted)7576worker2 # [ 204.585615] nix-daemon[850]: reaped child process 1908, status = succeeded7577worker2 # [ 204.669373] nix-daemon[850]: accepted connection from pid 1910, user nix-grpc-daemon (trusted)7578worker2 # [ 204.682620] nix-daemon[850]: reaped child process 1913, status = succeeded7579lb # [ 204.705190] niks3-server[1099]: 2026/09/22 14:53:44 INFO Received uploads request method=POST path=/api/pending_closures7580worker2 # [ 204.697495] nix-grpc-daemon[1685]: time=2026-09-22T14:53:44.604Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7581worker2 # [ 204.699386] nix-grpc-daemon[1685]: time=2026-09-22T14:53:44.605Z level=INFO msg="Uploading xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv (624B)"7582lb # [ 204.731583] niks3-server[1099]: 2026/09/22 14:53:44 INFO Registered completed upload object_key=nar/06skgfsii1iysx6lv8hy20iyys0q28pmz13rs51634x6bgrlvs7k.nar.zst7583lb # [ 204.747264] niks3-server[1099]: 2026/09/22 14:53:44 INFO Received sign narinfos request method=POST path=/api/pending_closures/82/sign7584lb # [ 204.752199] niks3-server[1099]: 2026/09/22 14:53:44 INFO Signed narinfos id=82 count=17585worker2 # [ 204.739639] nix-grpc-daemon[1685]: time=2026-09-22T14:53:44.645Z level=INFO msg="Uploading 1 narinfos"7586lb # [ 204.758711] niks3-server[1099]: 2026/09/22 14:53:44 INFO Registered completed upload object_key=xihm5w957gm8bh55zxlc6dnczbbl387q.ls7587lb # [ 204.769640] niks3-server[1099]: 2026/09/22 14:53:44 INFO Received complete upload request method=POST path=/api/pending_closures/82/complete7588lb # [ 204.774058] niks3-server[1099]: 2026/09/22 14:53:44 INFO Completed upload id=827589worker2 # [ 204.760952] nix-grpc-daemon[1685]: time=2026-09-22T14:53:44.667Z level=INFO msg="Upload complete. (273ms)"7590lb # [ 204.779287] niks3-server[1099]: 2026/09/22 14:53:44 INFO Registered completed upload object_key=xihm5w957gm8bh55zxlc6dnczbbl387q.narinfo7591worker2 # [ 204.763473] nix-grpc-daemon[1597]: ts=2026-09-22T14:53:44Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36704 duration_s=0 paths=1 nar_bytes_in=6247592worker2 # [ 204.766284] nix-grpc-daemon[1597]: ts=2026-09-22T14:53:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:367047593worker2 # [ 204.773463] nix-daemon[850]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)7594worker2 # [ 204.779193] nix-daemon[850]: reaped child process 1903, status = succeeded7595worker2 # [ 204.783640] nix-daemon[850]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)7596client # building '/nix/store/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv' on lb:50051...7597client # farm-b> worker2: building xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv7598client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds7599client # farm-b> LOG-farm-b-drain7600worker2 # [ 205.620471] nix-daemon[850]: reaped child process 1917, status = succeeded7601worker2 # [ 205.688089] nix-daemon[850]: accepted connection from pid 1933, user nix-grpc-daemon (trusted)7602worker2 # [ 205.699826] nix-daemon[850]: reaped child process 1936, status = succeeded7603lb # [ 205.725820] niks3-server[1099]: 2026/09/22 14:53:45 INFO Received uploads request method=POST path=/api/pending_closures7604worker2 # [ 205.716637] nix-grpc-daemon[1685]: time=2026-09-22T14:53:45.622Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7605worker2 # [ 205.719810] nix-grpc-daemon[1685]: time=2026-09-22T14:53:45.624Z level=INFO msg="Uploading pmvxn7ymjj8pdn6gb65j5kqb45n38x0p-farm-b (176B)"7606lb # [ 205.763219] niks3-server[1099]: 2026/09/22 14:53:45 INFO Registered completed upload object_key=nar/17gyl0zgiivck81rhv539x50216ksc3nsf369ismwd4j1zwsdqrb.nar.zst7607lb # [ 205.776211] niks3-server[1099]: 2026/09/22 14:53:45 INFO Registered completed upload object_key=log/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv7608lb # [ 205.784063] niks3-server[1099]: 2026/09/22 14:53:45 INFO Received sign narinfos request method=POST path=/api/pending_closures/83/sign7609lb # [ 205.789366] niks3-server[1099]: 2026/09/22 14:53:45 INFO Signed narinfos id=83 count=17610worker2 # [ 205.777179] nix-grpc-daemon[1685]: time=2026-09-22T14:53:45.683Z level=INFO msg="Uploading 1 narinfos"7611lb # [ 205.795795] niks3-server[1099]: 2026/09/22 14:53:45 INFO Registered completed upload object_key=pmvxn7ymjj8pdn6gb65j5kqb45n38x0p.ls7612lb # [ 205.804894] niks3-server[1099]: 2026/09/22 14:53:45 INFO Received complete upload request method=POST path=/api/pending_closures/83/complete7613lb # [ 205.807584] envoy[905]: conn peer=[2001:db8:1::1]:57380 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=895 tx=498 ms=300477614lb # [ 205.810368] niks3-server[1099]: 2026/09/22 14:53:45 INFO Completed upload id=837615worker2 # [ 205.795091] nix-grpc-daemon[1685]: time=2026-09-22T14:53:45.701Z level=INFO msg="Upload complete. (204ms)"7616lb # [ 205.813060] niks3-server[1099]: 2026/09/22 14:53:45 INFO Registered completed upload object_key=pmvxn7ymjj8pdn6gb65j5kqb45n38x0p.narinfo7617worker2 # [ 205.800844] nix-grpc-daemon[1597]: ts=2026-09-22T14:53:45Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36704 duration_s=1 drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv assign_id=11 outputs=17618worker2 # [ 205.815228] nix-daemon[850]: reaped child process 1915, status = succeeded7619worker1 # [ 205.830910] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:45Z level=debug event=want client=3 drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv cached=07620worker1 # [ 205.835697] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:45Z level=debug event=assigned drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv worker=192.168.1.4:50051 clients=17621worker2 # [ 205.856555] nix-grpc-daemon[1597]: ts=2026-09-22T14:53:45Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:367047622worker2 # [ 205.868412] nix-grpc-daemon[1597]: ts=2026-09-22T14:53:45Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:367047623worker2 # [ 205.881889] nix-grpc-daemon[1685]: time=2026-09-22T14:53:45.788Z level=INFO msg="All 2 paths already cached"7624client # copying 1 paths to 'lb:50051'...7625client # copying path '/nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv' to 'lb:50051'...7626worker2 # [ 205.889754] nix-grpc-daemon[1597]: ts=2026-09-22T14:53:45Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:367047627worker2 # [ 205.900127] nix-daemon[850]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)7628worker2 # [ 205.998277] nix-daemon[850]: accepted connection from pid 1940, user nix-grpc-daemon (trusted)7629worker2 # [ 206.011558] nix-daemon[850]: reaped child process 1943, status = succeeded7630worker2 # [ 206.093561] nix-daemon[850]: accepted connection from pid 1945, user nix-grpc-daemon (trusted)7631worker2 # [ 206.107213] nix-daemon[850]: reaped child process 1948, status = succeeded7632lb # [ 206.128394] niks3-server[1099]: 2026/09/22 14:53:45 INFO Received uploads request method=POST path=/api/pending_closures7633worker2 # [ 206.117186] nix-grpc-daemon[1685]: time=2026-09-22T14:53:46.023Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7634worker2 # [ 206.118781] nix-grpc-daemon[1685]: time=2026-09-22T14:53:46.025Z level=INFO msg="Uploading q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv (752B)"7635lb # [ 206.143833] niks3-server[1099]: 2026/09/22 14:53:45 INFO Registered completed upload object_key=nar/1m4vpncprrwddjpk4qry79vzd9g39s9pjs6874i2aarpm7ax0ykq.nar.zst7636lb # [ 206.147880] niks3-server[1099]: 2026/09/22 14:53:45 INFO Received sign narinfos request method=POST path=/api/pending_closures/84/sign7637lb # [ 206.150548] niks3-server[1099]: 2026/09/22 14:53:45 INFO Signed narinfos id=84 count=17638worker2 # [ 206.135301] nix-grpc-daemon[1685]: time=2026-09-22T14:53:46.041Z level=INFO msg="Uploading 1 narinfos"7639lb # [ 206.153876] niks3-server[1099]: 2026/09/22 14:53:45 INFO Registered completed upload object_key=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s.ls7640lb # [ 206.160234] niks3-server[1099]: 2026/09/22 14:53:45 INFO Received complete upload request method=POST path=/api/pending_closures/84/complete7641lb # [ 206.163582] niks3-server[1099]: 2026/09/22 14:53:45 INFO Completed upload id=847642worker2 # [ 206.147989] nix-grpc-daemon[1685]: time=2026-09-22T14:53:46.054Z level=INFO msg="Upload complete. (236ms)"7643lb # [ 206.165643] niks3-server[1099]: 2026/09/22 14:53:45 INFO Registered completed upload object_key=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s.narinfo7644worker2 # [ 206.150276] nix-grpc-daemon[1597]: ts=2026-09-22T14:53:46Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36704 duration_s=0 paths=1 nar_bytes_in=7527645worker2 # [ 206.154320] nix-grpc-daemon[1597]: ts=2026-09-22T14:53:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:367047646worker2 # [ 206.160588] nix-daemon[850]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)7647worker2 # [ 206.162907] nix-daemon[850]: reaped child process 1938, status = succeeded7648worker2 # [ 206.168165] nix-daemon[850]: accepted connection from pid 1597, user nix-grpc-daemon (trusted)7649client # building '/nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv' on lb:50051...7650client # farm-top> worker2: building q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv7651client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds7652client # farm-top> LOG-farm-top-drain7653worker2 # [ 207.207616] nix-daemon[850]: reaped child process 1952, status = succeeded7654worker2 # [ 207.293091] nix-daemon[850]: accepted connection from pid 1968, user nix-grpc-daemon (trusted)7655worker2 # [ 207.307140] nix-daemon[850]: reaped child process 1971, status = succeeded7656lb # [ 207.327703] niks3-server[1099]: 2026/09/22 14:53:46 INFO Received uploads request method=POST path=/api/pending_closures7657worker2 # [ 207.320216] nix-grpc-daemon[1685]: time=2026-09-22T14:53:47.226Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7658worker2 # [ 207.322673] nix-grpc-daemon[1685]: time=2026-09-22T14:53:47.228Z level=INFO msg="Uploading hgmfp1yw6m31jn6cpma19ikl6dy1b17r-farm-top (232B)"7659lb # [ 207.361069] niks3-server[1099]: 2026/09/22 14:53:46 INFO Registered completed upload object_key=log/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv7660lb # [ 207.374668] niks3-server[1099]: 2026/09/22 14:53:46 INFO Registered completed upload object_key=nar/1xin0j0xlljjqnd952hnxy8yw45d9j9qpj7m241vv6cckfq8kk4p.nar.zst7661lb # [ 207.381394] niks3-server[1099]: 2026/09/22 14:53:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/85/sign7662lb # [ 207.388541] niks3-server[1099]: 2026/09/22 14:53:46 INFO Signed narinfos id=85 count=17663worker2 # [ 207.375970] nix-grpc-daemon[1685]: time=2026-09-22T14:53:47.282Z level=INFO msg="Uploading 1 narinfos"7664lb # [ 207.398746] niks3-server[1099]: 2026/09/22 14:53:46 INFO Registered completed upload object_key=hgmfp1yw6m31jn6cpma19ikl6dy1b17r.ls7665lb # [ 207.405685] niks3-server[1099]: 2026/09/22 14:53:46 INFO Received complete upload request method=POST path=/api/pending_closures/85/complete7666lb # [ 207.408308] niks3-server[1099]: 2026/09/22 14:53:46 INFO Completed upload id=857667worker2 # [ 207.392826] nix-grpc-daemon[1685]: time=2026-09-22T14:53:47.299Z level=INFO msg="Upload complete. (216ms)"7668lb # [ 207.411124] niks3-server[1099]: 2026/09/22 14:53:46 INFO Registered completed upload object_key=hgmfp1yw6m31jn6cpma19ikl6dy1b17r.narinfo7669worker2 # [ 207.396158] nix-grpc-daemon[1597]: ts=2026-09-22T14:53:47Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36704 duration_s=1 drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv assign_id=12 outputs=17670worker2 # [ 207.403935] nix-daemon[850]: reaped child process 1950, status = succeeded7671client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag drain >&2, in 5.25 seconds)7672client: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag drain outPath7673client: (finished: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag drain outPath, in 0.12 seconds)7674worker1: must succeed: rm /nix/.rw-store/fill7675worker1: (finished: must succeed: rm /nix/.rw-store/fill, in 0.12 seconds)7676worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free'7677worker1 # [ 208.453749] nix-grpc-daemon[1886]: ts=2026-09-22T14:53:48Z level=info event=healthy reason=min_free7678worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free', in 1.05 seconds)7679(finished: subtest: low disk drains a worker and builds go to the other, in 8.59 seconds)7680(finished: run the VM test script, in 209.73 seconds)7681test script finished in 209.89s7682cleanup7683kill QemuMachine (pid 49)7684client # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/d64q19q1xjdwfhqx6czvrjgrhq0n3lcc-python3-3.14.7/bin/python3.14)7685client # [2026-09-22T14:53:48Z INFO virtiofsd] Client disconnected, shutting down7686client # [2026-09-22T14:53:48Z INFO virtiofsd] Client disconnected, shutting down7687client # [2026-09-22T14:53:48Z INFO virtiofsd] Client disconnected, shutting down7688kill QemuMachine (pid 51)7689lb # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/d64q19q1xjdwfhqx6czvrjgrhq0n3lcc-python3-3.14.7/bin/python3.14)7690lb # [2026-09-22T14:53:48Z INFO virtiofsd] Client disconnected, shutting down7691lb # [2026-09-22T14:53:48Z INFO virtiofsd] Client disconnected, shutting down7692lb # [2026-09-22T14:53:48Z INFO virtiofsd] Client disconnected, shutting down7693kill QemuMachine (pid 50)7694worker1 # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/d64q19q1xjdwfhqx6czvrjgrhq0n3lcc-python3-3.14.7/bin/python3.14)7695worker1 # [2026-09-22T14:53:49Z INFO virtiofsd] Client disconnected, shutting down7696worker1 # [2026-09-22T14:53:49Z INFO virtiofsd] Client disconnected, shutting down7697worker1 # [2026-09-22T14:53:49Z INFO virtiofsd] Client disconnected, shutting down7698kill QemuMachine (pid 52)7699worker2 # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/d64q19q1xjdwfhqx6czvrjgrhq0n3lcc-python3-3.14.7/bin/python3.14)7700worker2 # [2026-09-22T14:53:49Z INFO virtiofsd] Client disconnected, shutting down7701worker2 # [2026-09-22T14:53:49Z INFO virtiofsd] Client disconnected, shutting down7702worker2 # [2026-09-22T14:53:49Z INFO virtiofsd] Client disconnected, shutting down7703(finished: cleanup, in 1.12 seconds)7704additionally exposed symbols:7705 client, lb, worker1, worker2,7706 vlan1,7707 start_all, test_script, machines, machines_qemu, machines_nspawn, vlans, driver, log, os, create_machine, subtest, run_tests, join_all, retry, serial_stdout_off, serial_stdout_on, polling_condition, BaseMachine, QemuMachine, NspawnMachine, t, debug, dump_machine_ssh