vm-test-run-nix-grpc-farm
checks.x86_64-linux.farm
· build #309
· raw
1tribuchet: building on jamie2Machine state will be reset. To keep it, pass --keep-machine-state3start all VLans4(finished: start all VLans, in 0.00 seconds)5Test will time out and terminate in 900.0 seconds6run the VM test script7start all VMs8client: starting vm9lb: starting vm10worker1: starting vm11worker2: starting vm12client # Disk image does not exist, creating the virtualisation disk image...13client # Formatting '/build/vm-state-client/tmp.219q9ZEwUW', fmt=raw size=107374182414client # mke2fs 1.47.4 (6-Mar-2025)15client: QEMU running (pid 49)16client # Discarding device blocks: 0/262144 done17client # Creating filesystem with 262144 4k blocks and 65536 inodes18client # Filesystem UUID: 4d9035df-3c16-4559-b977-9cbdae4293b519client # Superblock backups stored on blocks:20client # 32768, 98304, 163840, 22937621client # 22client # Allocating group tables: 0/8 done23client # Writing inode tables: 0/8 done24client # Creating journal (8192 blocks): done25client # Writing superblocks and filesystem accounting information: 0/8 done26client # 27client # Virtualisation disk image created.28lb # Disk image does not exist, creating the virtualisation disk image...29lb: QEMU running (pid 50)30lb # Formatting '/build/vm-state-lb/tmp.VQ20jRW1uK', fmt=raw size=107374182431lb # mke2fs 1.47.4 (6-Mar-2025)32lb # Discarding device blocks: 0/262144 done33lb # Creating filesystem with 262144 4k blocks and 65536 inodes34lb # Filesystem UUID: cb2f54a7-15ba-4eb7-9a1a-970c463e3bf535lb # Superblock backups stored on blocks:36lb # 32768, 98304, 163840, 22937637lb # 38lb # Allocating group tables: 0/8 done39lb # Writing inode tables: 0/8 done40lb # Creating journal (8192 blocks): done41lb # Writing superblocks and filesystem accounting information: 0/8 done42lb # 43lb # Virtualisation disk image created.44worker1 # Disk image does not exist, creating the virtualisation disk image...45worker1 # Formatting '/build/vm-state-worker1/tmp.zLM4pQ9b9L', fmt=raw size=107374182446worker1: QEMU running (pid 51)47worker1 # mke2fs 1.47.4 (6-Mar-2025)48worker1 # Discarding device blocks: 0/262144 done49worker1 # Creating filesystem with 262144 4k blocks and 65536 inodes50worker1 # Filesystem UUID: cc21be11-2a48-4011-bbdf-d84500a7159c51worker1 # Superblock backups stored on blocks:52worker1 # 32768, 98304, 163840, 22937653worker1 # 54worker1 # Allocating group tables: 0/8 done55worker1 # Writing inode tables: 0/8 done56worker1 # Creating journal (8192 blocks): done57worker1 # Writing superblocks and filesystem accounting information: 0/8 done58worker1 # 59worker1 # Virtualisation disk image created.60worker2: QEMU running (pid 52)61worker2 # Disk image does not exist, creating the virtualisation disk image...62worker2 # Formatting '/build/vm-state-worker2/tmp.8w5GEjk2J7', fmt=raw size=107374182463worker2 # mke2fs 1.47.4 (6-Mar-2025)64worker2 # Discarding device blocks: 0/262144 done65(finished: start all VMs, in 0.33 seconds)66worker2 # Creating filesystem with 262144 4k blocks and 65536 inodes67worker2 # Filesystem UUID: b8ea6ff1-8926-43ec-9859-89f4541397f668worker2 # Superblock backups stored on blocks:69worker2 # 32768, 98304, 163840, 22937670worker2 # 71worker2 # Allocating group tables: 0/8 done72worker2 # Writing inode tables: 0/8 done73worker2 # Creating journal (8192 blocks): done74worker2 # Writing superblocks and filesystem accounting information: 0/8 done75worker2 # 76worker2 # Virtualisation disk image created.77client: waiting for unit network-addresses-eth1.service78client: waiting for the VM to finish booting79lb # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)80worker1 # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)81client # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)82worker2 # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)83lb # 84lb # 85lb # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA0086worker1 # 87worker1 # 88lb # Press Ctrl-B to configure iPXE (PCI 00:02.0)...89lb # 90lb # 91lb # 92lb # 93worker1 # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA0094lb # iPXE (http://ipxe.org) 00:08.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB0095client # 96client # 97client # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA0098worker1 # Press Ctrl-B to configure iPXE (PCI 00:02.0)...99worker1 # 100worker1 # 101worker1 # 102worker1 # 103lb # Press Ctrl-B to configure iPXE (PCI 00:08.0)...104worker1 # iPXE (http://ipxe.org) 00:08.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00105lb # 106lb # 107client # Press Ctrl-B to configure iPXE (PCI 00:02.0)...108client # 109client # 110client # 111client # 112worker2 # 113worker2 # 114client # iPXE (http://ipxe.org) 00:08.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00115worker2 # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA00116worker1 # Press Ctrl-B to configure iPXE (PCI 00:08.0)...117worker1 # 118worker1 # 119worker2 # Press Ctrl-B to configure iPXE (PCI 00:02.0)...120worker2 # 121worker2 # 122worker2 # 123worker2 # 124client # Press Ctrl-B to configure iPXE (PCI 00:08.0)...125client # 126client # 127worker2 # iPXE (http://ipxe.org) 00:08.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00128worker2 # Press Ctrl-B to configure iPXE (PCI 00:08.0)...129worker2 # 130worker2 # 131lb # Booting from ROM...132worker1 # Booting from ROM...133client # Booting from ROM...134worker2 # Booting from ROM...135lb # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 7 15:23:00 UTC 2026136lb # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/0qln5mjy767qjp5wbmxjg8jay8k05dsm-nixos-system-lb-test/init regInfo=/nix/store/nwvvzfrgncwhnws6wdjigc6jviavrdxi-closure-info/registration console=ttyS0,115200n8 console=tty0137lb # [ 0.000000] BIOS-provided physical RAM map:138lb # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable139lb # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved140worker1 # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 7 15:23:00 UTC 2026141lb # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved142lb # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable143lb # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved144lb # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved145lb # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved146worker1 # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/59sadfclj99p3nsjrb8w62ips04h7jqd-nixos-system-worker1-test/init regInfo=/nix/store/99gfyjr6j4z2yxhdjhdikn7nidv9r40d-closure-info/registration console=ttyS0,115200n8 console=tty0147worker1 # [ 0.000000] BIOS-provided physical RAM map:148lb # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved149worker1 # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable150lb # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved151worker1 # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved152lb # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved153lb # [ 0.000000] NX (Execute Disable) protection: active154worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved155lb # [ 0.000000] APIC: Static calls initialized156worker1 # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable157lb # [ 0.000000] SMBIOS 2.8 present.158worker1 # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved159lb # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014160worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved161lb # [ 0.000000] DMI: Memory slots populated: 1/1162lb # [ 0.000000] Hypervisor detected: KVM163worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved164client # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 7 15:23:00 UTC 2026165lb # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000166worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved167worker2 # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 7 15:23:00 UTC 2026168lb # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00169worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved170lb # [ 0.000001] kvm-clock: using sched offset of 466255418 cycles171worker1 # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved172lb # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns173worker1 # [ 0.000000] NX (Execute Disable) protection: active174lb # [ 0.000005] tsc: Detected 2400.012 MHz processor175worker1 # [ 0.000000] APIC: Static calls initialized176client # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/mmlfvh5rvsdyfwvf1j0yz3mcyc2x0fc8-nixos-system-client-test/init regInfo=/nix/store/b7318vdpqnkbnwygbbjcxfpnmwp5hw2v-closure-info/registration console=ttyS0,115200n8 console=tty0177worker1 # [ 0.000000] SMBIOS 2.8 present.178lb # [ 0.000806] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000179client # [ 0.000000] BIOS-provided physical RAM map:180worker2 # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/lkap5dyl76lw2f4vq53rgc5wq8hbp23z-nixos-system-worker2-test/init regInfo=/nix/store/dyzapyk53fkckjy67ggaal3lwcfb6vkk-closure-info/registration console=ttyS0,115200n8 console=tty0181worker2 # [ 0.000000] BIOS-provided physical RAM map:182client # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable183lb # [ 0.000834] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs184worker1 # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014185worker2 # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable186worker1 # [ 0.000000] DMI: Memory slots populated: 1/1187client # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved188lb # [ 0.000837] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT189worker1 # [ 0.000000] Hypervisor detected: KVM190worker2 # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved191lb # [ 0.002734] found SMP MP-table at [mem 0x000f5450-0x000f545f]192client # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved193worker1 # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000194lb # [ 0.002746] Using GB pages for direct mapping195client # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable196worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved197worker1 # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00198lb # [ 0.002817] RAMDISK: [mem 0x5e367000-0x5ffcffff]199worker1 # [ 0.000000] kvm-clock: using sched offset of 464267948 cycles200worker2 # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable201client # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved202lb # [ 0.002828] ACPI: Early table checksum verification disabled203client # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved204worker2 # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved205lb # [ 0.002831] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )206worker1 # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns207worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved208client # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved209worker1 # [ 0.000005] tsc: Detected 2400.012 MHz processor210lb # [ 0.002836] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)211worker1 # [ 0.000819] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000212client # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved213worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved214lb # [ 0.002840] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)215client # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved216worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved217worker1 # [ 0.000847] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs218lb # [ 0.002848] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)219client # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved220worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved221worker1 # [ 0.000850] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT222lb # [ 0.002850] ACPI: FACS 0x000000005FFE0000 000040223client # [ 0.000000] NX (Execute Disable) protection: active224worker1 # [ 0.002768] found SMP MP-table at [mem 0x000f5450-0x000f545f]225worker2 # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved226client # [ 0.000000] APIC: Static calls initialized227worker1 # [ 0.002782] Using GB pages for direct mapping228client # [ 0.000000] SMBIOS 2.8 present.229lb # [ 0.002852] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)230worker1 # [ 0.002871] RAMDISK: [mem 0x5e367000-0x5ffcffff]231worker2 # [ 0.000000] NX (Execute Disable) protection: active232worker2 # [ 0.000000] APIC: Static calls initialized233worker1 # [ 0.002881] ACPI: Early table checksum verification disabled234worker2 # [ 0.000000] SMBIOS 2.8 present.235client # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014236lb # [ 0.002853] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)237worker1 # [ 0.002885] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )238client # [ 0.000000] DMI: Memory slots populated: 1/1239worker2 # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014240client # [ 0.000000] Hypervisor detected: KVM241lb # [ 0.002855] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)242worker2 # [ 0.000000] DMI: Memory slots populated: 1/1243worker1 # [ 0.002889] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)244client # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000245worker2 # [ 0.000000] Hypervisor detected: KVM246lb # [ 0.002857] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)247client # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00248worker1 # [ 0.002893] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)249worker2 # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000250client # [ 0.000000] kvm-clock: using sched offset of 492515627 cycles251lb # [ 0.002858] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]252worker2 # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00253worker1 # [ 0.002901] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)254worker2 # [ 0.000000] kvm-clock: using sched offset of 465878720 cycles255lb # [ 0.002859] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]256worker1 # [ 0.002903] ACPI: FACS 0x000000005FFE0000 000040257client # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns258lb # [ 0.002860] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]259client # [ 0.000006] tsc: Detected 2400.012 MHz processor260worker1 # [ 0.002904] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)261worker2 # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns262client # [ 0.000813] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000263lb # [ 0.002860] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]264worker2 # [ 0.000006] tsc: Detected 2400.012 MHz processor265worker1 # [ 0.002906] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)266worker2 # [ 0.000812] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000267client # [ 0.000841] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs268lb # [ 0.002860] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]269worker1 # [ 0.002907] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)270client # [ 0.000843] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT271worker2 # [ 0.000840] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs272lb # [ 0.002861] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]273client # [ 0.002756] found SMP MP-table at [mem 0x000f5450-0x000f545f]274lb # [ 0.002861] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]275worker1 # [ 0.002909] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)276worker2 # [ 0.000843] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT277client # [ 0.002769] Using GB pages for direct mapping278lb # [ 0.003105] No NUMA configuration found279worker2 # [ 0.002752] found SMP MP-table at [mem 0x000f5450-0x000f545f]280client # [ 0.002911] RAMDISK: [mem 0x5e366000-0x5ffcffff]281worker1 # [ 0.002910] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]282worker2 # [ 0.002766] Using GB pages for direct mapping283lb # [ 0.003106] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]284client # [ 0.002920] ACPI: Early table checksum verification disabled285worker1 # [ 0.002911] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]286worker2 # [ 0.002862] RAMDISK: [mem 0x5e367000-0x5ffcffff]287lb # [ 0.003108] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]288client # [ 0.002924] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )289lb # [ 0.005395] Zone ranges:290worker2 # [ 0.002870] ACPI: Early table checksum verification disabled291worker1 # [ 0.002912] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]292lb # [ 0.005396] DMA [mem 0x0000000000001000-0x0000000000ffffff]293worker2 # [ 0.002874] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )294client # [ 0.002928] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)295worker1 # [ 0.002912] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]296lb # [ 0.005398] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]297lb # [ 0.005399] Normal empty298worker2 # [ 0.002879] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)299client # [ 0.002933] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)300worker1 # [ 0.002913] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]301lb # [ 0.005400] Device empty302lb # [ 0.005400] Movable zone start for each node303worker1 # [ 0.002913] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]304worker2 # [ 0.002883] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)305client # [ 0.002941] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)306lb # [ 0.005401] Early memory node ranges307client # [ 0.002943] ACPI: FACS 0x000000005FFE0000 000040308worker1 # [ 0.002914] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]309worker2 # [ 0.002891] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)310lb # [ 0.005401] node 0: [mem 0x0000000000001000-0x000000000009efff]311worker1 # [ 0.003149] No NUMA configuration found312worker2 # [ 0.002893] ACPI: FACS 0x000000005FFE0000 000040313client # [ 0.002944] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)314lb # [ 0.005403] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]315worker1 # [ 0.003150] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]316lb # [ 0.005403] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]317worker2 # [ 0.002894] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)318worker1 # [ 0.003153] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]319client # [ 0.002946] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)320worker1 # [ 0.005477] Zone ranges:321lb # [ 0.005424] On node 0, zone DMA: 1 pages in unavailable ranges322worker2 # [ 0.002896] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)323client # [ 0.002948] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)324worker1 # [ 0.005478] DMA [mem 0x0000000000001000-0x0000000000ffffff]325lb # [ 0.005685] On node 0, zone DMA: 97 pages in unavailable ranges326worker1 # [ 0.005483] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]327worker2 # [ 0.002898] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)328lb # [ 0.033557] On node 0, zone DMA32: 40 pages in unavailable ranges329client # [ 0.002949] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)330worker1 # [ 0.005484] Normal empty331worker1 # [ 0.005485] Device empty332lb # [ 0.034051] ACPI: PM-Timer IO Port: 0x608333client # [ 0.002950] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]334worker2 # [ 0.002899] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)335worker1 # [ 0.005485] Movable zone start for each node336lb # [ 0.034065] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])337worker1 # [ 0.005486] Early memory node ranges338worker2 # [ 0.002907] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]339client # [ 0.002951] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]340lb # [ 0.034096] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23341worker1 # [ 0.005486] node 0: [mem 0x0000000000001000-0x000000000009efff]342client # [ 0.002952] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]343worker2 # [ 0.002908] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]344lb # [ 0.034099] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)345worker1 # [ 0.005488] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]346client # [ 0.002953] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]347worker2 # [ 0.002908] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]348lb # [ 0.034101] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)349worker1 # [ 0.005489] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]350client # [ 0.002953] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]351worker2 # [ 0.002909] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]352lb # [ 0.034102] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)353worker1 # [ 0.005512] On node 0, zone DMA: 1 pages in unavailable ranges354worker2 # [ 0.002909] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]355worker1 # [ 0.005788] On node 0, zone DMA: 97 pages in unavailable ranges356client # [ 0.002954] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]357lb # [ 0.034103] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)358worker1 # [ 0.033418] On node 0, zone DMA32: 40 pages in unavailable ranges359client # [ 0.002954] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]360worker2 # [ 0.002910] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]361lb # [ 0.034104] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)362client # [ 0.003186] No NUMA configuration found363worker1 # [ 0.033912] ACPI: PM-Timer IO Port: 0x608364lb # [ 0.034106] ACPI: Using ACPI (MADT) for SMP configuration information365worker2 # [ 0.002910] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]366worker1 # [ 0.033926] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])367client # [ 0.003187] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]368worker2 # [ 0.003141] No NUMA configuration found369lb # [ 0.034108] ACPI: HPET id: 0x8086a201 base: 0xfed00000370lb # [ 0.034113] TSC deadline timer available371worker1 # [ 0.033957] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23372client # [ 0.003189] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]373worker2 # [ 0.003142] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]374client # [ 0.005619] Zone ranges:375lb # [ 0.034117] CPU topo: Max. logical packages: 1376worker1 # [ 0.033960] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)377worker2 # [ 0.003145] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]378lb # [ 0.034118] CPU topo: Max. logical dies: 1379client # [ 0.005621] DMA [mem 0x0000000000001000-0x0000000000ffffff]380worker2 # [ 0.005445] Zone ranges:381worker1 # [ 0.033962] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)382lb # [ 0.034119] CPU topo: Max. dies per package: 1383client # [ 0.005622] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]384worker2 # [ 0.005446] DMA [mem 0x0000000000001000-0x0000000000ffffff]385lb # [ 0.034123] CPU topo: Max. threads per core: 1386client # [ 0.005623] Normal empty387worker1 # [ 0.033963] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)388client # [ 0.005624] Device empty389worker2 # [ 0.005447] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]390lb # [ 0.034123] CPU topo: Num. cores per package: 1391client # [ 0.005625] Movable zone start for each node392worker2 # [ 0.005449] Normal empty393worker1 # [ 0.033965] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)394lb # [ 0.034124] CPU topo: Num. threads per package: 1395client # [ 0.005626] Early memory node ranges396worker2 # [ 0.005450] Device empty397worker1 # [ 0.033965] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)398worker2 # [ 0.005450] Movable zone start for each node399lb # [ 0.034124] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs400client # [ 0.005626] node 0: [mem 0x0000000000001000-0x000000000009efff]401worker2 # [ 0.005451] Early memory node ranges402worker1 # [ 0.033968] ACPI: Using ACPI (MADT) for SMP configuration information403lb # [ 0.034147] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()404client # [ 0.005627] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]405worker2 # [ 0.005451] node 0: [mem 0x0000000000001000-0x000000000009efff]406worker1 # [ 0.033969] ACPI: HPET id: 0x8086a201 base: 0xfed00000407client # [ 0.005628] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]408worker1 # [ 0.033974] TSC deadline timer available409lb # [ 0.034182] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]410worker2 # [ 0.005452] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]411worker1 # [ 0.033979] CPU topo: Max. logical packages: 1412client # [ 0.005650] On node 0, zone DMA: 1 pages in unavailable ranges413worker2 # [ 0.005454] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]414lb # [ 0.034184] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]415worker1 # [ 0.033980] CPU topo: Max. logical dies: 1416client # [ 0.005947] On node 0, zone DMA: 97 pages in unavailable ranges417worker2 # [ 0.005476] On node 0, zone DMA: 1 pages in unavailable ranges418lb # [ 0.034185] [mem 0x60000000-0xafffffff] available for PCI devices419worker1 # [ 0.033980] CPU topo: Max. dies per package: 1420client # [ 0.035423] On node 0, zone DMA32: 40 pages in unavailable ranges421lb # [ 0.034187] Booting paravirtualized kernel on KVM422worker2 # [ 0.005741] On node 0, zone DMA: 97 pages in unavailable ranges423worker1 # [ 0.033983] CPU topo: Max. threads per core: 1424client # [ 0.035920] ACPI: PM-Timer IO Port: 0x608425worker1 # [ 0.033984] CPU topo: Num. cores per package: 1426worker2 # [ 0.034112] On node 0, zone DMA32: 40 pages in unavailable ranges427client # [ 0.035936] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])428worker1 # [ 0.033985] CPU topo: Num. threads per package: 1429worker2 # [ 0.034607] ACPI: PM-Timer IO Port: 0x608430lb # [ 0.034191] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns431worker1 # [ 0.033985] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs432client # [ 0.035968] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23433worker2 # [ 0.034622] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])434lb # [ 0.038826] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1435worker1 # [ 0.034009] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()436worker2 # [ 0.034654] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23437lb # [ 0.041160] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152438client # [ 0.035970] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)439client # [ 0.035972] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)440worker2 # [ 0.034657] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)441worker1 # [ 0.034045] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]442lb # [ 0.041205] kvm-guest: PV spinlocks disabled, single CPU443client # [ 0.035974] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)444worker2 # [ 0.034659] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)445worker1 # [ 0.034047] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]446client # [ 0.035975] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)447worker2 # [ 0.034660] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)448worker1 # [ 0.034048] [mem 0x60000000-0xafffffff] available for PCI devices449worker1 # [ 0.034050] Booting paravirtualized kernel on KVM450worker2 # [ 0.034661] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)451client # [ 0.035976] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)452client # [ 0.035978] ACPI: Using ACPI (MADT) for SMP configuration information453worker2 # [ 0.034662] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)454worker1 # [ 0.034054] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns455lb # [ 0.041206] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/0qln5mjy767qjp5wbmxjg8jay8k05dsm-nixos-system-lb-test/init regInfo=/nix/store/nwvvzfrgncwhnws6wdjigc6jviavrdxi-closure-info/registration console=ttyS0,115200n8 console=tty0456client # [ 0.035979] ACPI: HPET id: 0x8086a201 base: 0xfed00000457worker2 # [ 0.034665] ACPI: Using ACPI (MADT) for SMP configuration information458client # [ 0.035985] TSC deadline timer available459worker1 # [ 0.038705] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1460worker2 # [ 0.034666] ACPI: HPET id: 0x8086a201 base: 0xfed00000461client # [ 0.035990] CPU topo: Max. logical packages: 1462worker2 # [ 0.034671] TSC deadline timer available463worker1 # [ 0.041045] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152464client # [ 0.035990] CPU topo: Max. logical dies: 1465lb # [ 0.041306] Unknown kernel command line parameters "regInfo=/nix/store/nwvvzfrgncwhnws6wdjigc6jviavrdxi-closure-info/registration", will be passed to user space.466worker2 # [ 0.034675] CPU topo: Max. logical packages: 1467lb # [ 0.041320] random: crng init done468client # [ 0.035991] CPU topo: Max. dies per package: 1469worker1 # [ 0.041089] kvm-guest: PV spinlocks disabled, single CPU470worker2 # [ 0.034676] CPU topo: Max. logical dies: 1471client # [ 0.035994] CPU topo: Max. threads per core: 1472worker2 # [ 0.034677] CPU topo: Max. dies per package: 1473lb # [ 0.041321] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes474client # [ 0.035995] CPU topo: Num. cores per package: 1475worker2 # [ 0.034680] CPU topo: Max. threads per core: 1476client # [ 0.035995] CPU topo: Num. threads per package: 1477lb # [ 0.043597] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)478worker2 # [ 0.034681] CPU topo: Num. cores per package: 1479client # [ 0.035996] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs480worker2 # [ 0.034681] CPU topo: Num. threads per package: 1481lb # [ 0.044790] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)482lb # [ 0.044831] Fallback order for Node 0: 0483worker2 # [ 0.034682] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs484client # [ 0.036024] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()485worker1 # [ 0.041091] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/59sadfclj99p3nsjrb8w62ips04h7jqd-nixos-system-worker1-test/init regInfo=/nix/store/99gfyjr6j4z2yxhdjhdikn7nidv9r40d-closure-info/registration console=ttyS0,115200n8 console=tty0486lb # [ 0.044835] Built 1 zonelists, mobility grouping on. Total pages: 393078487worker2 # [ 0.034706] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()488client # [ 0.036064] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]489lb # [ 0.044836] Policy zone: DMA32490worker2 # [ 0.034743] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]491client # [ 0.036066] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]492lb # [ 0.047602] mem auto-init: stack:all(zero), heap alloc:on, heap free:off493worker1 # [ 0.041191] Unknown kernel command line parameters "regInfo=/nix/store/99gfyjr6j4z2yxhdjhdikn7nidv9r40d-closure-info/registration", will be passed to user space.494client # [ 0.036068] [mem 0x60000000-0xafffffff] available for PCI devices495worker2 # [ 0.034745] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]496worker1 # [ 0.041204] random: crng init done497lb # [ 0.051376] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1498client # [ 0.036070] Booting paravirtualized kernel on KVM499lb # [ 0.054975] allocated 3145728 bytes of page_ext500worker2 # [ 0.034747] [mem 0x60000000-0xafffffff] available for PCI devices501worker1 # [ 0.041205] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes502worker2 # [ 0.034748] Booting paravirtualized kernel on KVM503lb # [ 0.065638] ftrace: allocating 48731 entries in 192 pages504client # [ 0.036075] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns505worker1 # [ 0.043472] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)506lb # [ 0.065641] ftrace: allocated 192 pages with 2 groups507lb # [ 0.066852] Dynamic Preempt: lazy508worker2 # [ 0.034752] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns509client # [ 0.040692] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1510worker1 # [ 0.044869] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)511lb # [ 0.067058] rcu: Preemptible hierarchical RCU implementation.512worker1 # [ 0.044910] Fallback order for Node 0: 0513client # [ 0.041795] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152514worker2 # [ 0.039407] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1515lb # [ 0.067059] rcu: RCU event tracing is enabled.516worker1 # [ 0.044913] Built 1 zonelists, mobility grouping on. Total pages: 393078517client # [ 0.041844] kvm-guest: PV spinlocks disabled, single CPU518worker2 # [ 0.042023] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152519lb # [ 0.067060] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.520worker1 # [ 0.044914] Policy zone: DMA32521worker2 # [ 0.042272] kvm-guest: PV spinlocks disabled, single CPU522lb # [ 0.067062] Trampoline variant of Tasks RCU enabled.523worker1 # [ 0.047509] mem auto-init: stack:all(zero), heap alloc:on, heap free:off524lb # [ 0.067062] Rude variant of Tasks RCU enabled.525worker1 # [ 0.050940] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1526lb # [ 0.067062] Tracing variant of Tasks RCU enabled.527worker1 # [ 0.054345] allocated 3145728 bytes of page_ext528lb # [ 0.067063] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.529worker1 # [ 0.064953] ftrace: allocating 48731 entries in 192 pages530worker1 # [ 0.064957] ftrace: allocated 192 pages with 2 groups531lb # [ 0.067064] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1532client # [ 0.041846] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/mmlfvh5rvsdyfwvf1j0yz3mcyc2x0fc8-nixos-system-client-test/init regInfo=/nix/store/b7318vdpqnkbnwygbbjcxfpnmwp5hw2v-closure-info/registration console=ttyS0,115200n8 console=tty0533worker1 # [ 0.065889] Dynamic Preempt: lazy534worker2 # [ 0.042274] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/lkap5dyl76lw2f4vq53rgc5wq8hbp23z-nixos-system-worker2-test/init regInfo=/nix/store/dyzapyk53fkckjy67ggaal3lwcfb6vkk-closure-info/registration console=ttyS0,115200n8 console=tty0535lb # [ 0.067087] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.536worker1 # [ 0.066021] rcu: Preemptible hierarchical RCU implementation.537worker1 # [ 0.066022] rcu: RCU event tracing is enabled.538client # [ 0.041972] Unknown kernel command line parameters "regInfo=/nix/store/b7318vdpqnkbnwygbbjcxfpnmwp5hw2v-closure-info/registration", will be passed to user space.539client # [ 0.041988] random: crng init done540lb # [ 0.067089] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.541worker1 # [ 0.066023] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.542worker2 # [ 0.042374] Unknown kernel command line parameters "regInfo=/nix/store/dyzapyk53fkckjy67ggaal3lwcfb6vkk-closure-info/registration", will be passed to user space.543worker1 # [ 0.066025] Trampoline variant of Tasks RCU enabled.544worker2 # [ 0.042390] random: crng init done545client # [ 0.041989] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes546lb # [ 0.067090] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.547worker1 # [ 0.066026] Rude variant of Tasks RCU enabled.548worker2 # [ 0.042391] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes549lb # [ 0.071892] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16550client # [ 0.043865] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)551worker1 # [ 0.066026] Tracing variant of Tasks RCU enabled.552lb # [ 0.072187] rcu: srcu_init: Setting srcu_struct sizes based on contention.553worker2 # [ 0.044627] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)554client # [ 0.043935] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)555worker1 # [ 0.066027] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.556client # [ 0.043970] Fallback order for Node 0: 0557worker2 # [ 0.045767] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)558worker1 # [ 0.066028] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1559lb # [ 0.072193] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns560worker2 # [ 0.045808] Fallback order for Node 0: 0561client # [ 0.043974] Built 1 zonelists, mobility grouping on. Total pages: 393078562client # [ 0.043975] Policy zone: DMA32563worker1 # [ 0.066038] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.564worker2 # [ 0.045811] Built 1 zonelists, mobility grouping on. Total pages: 393078565lb # [ 0.072309] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)566client # [ 0.044132] mem auto-init: stack:all(zero), heap alloc:on, heap free:off567worker2 # [ 0.045813] Policy zone: DMA32568lb # [ 0.075986] Console: colour VGA+ 80x25569worker1 # [ 0.066040] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.570client # [ 0.048279] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1571lb # [ 0.075989] printk: legacy console [tty0] enabled572worker2 # [ 0.048409] mem auto-init: stack:all(zero), heap alloc:on, heap free:off573client # [ 0.052524] allocated 3145728 bytes of page_ext574lb # [ 0.106239] printk: legacy console [ttyS0] enabled575worker1 # [ 0.066041] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.576worker2 # [ 0.051845] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1577client # [ 0.063923] ftrace: allocating 48731 entries in 192 pages578lb # [ 0.203676] ACPI: Core revision 20250807579worker2 # [ 0.055768] allocated 3145728 bytes of page_ext580worker1 # [ 0.070985] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16581client # [ 0.063926] ftrace: allocated 192 pages with 2 groups582worker2 # [ 0.066274] ftrace: allocating 48731 entries in 192 pages583client # [ 0.064910] Dynamic Preempt: lazy584worker1 # [ 0.071263] rcu: srcu_init: Setting srcu_struct sizes based on contention.585worker2 # [ 0.066277] ftrace: allocated 192 pages with 2 groups586client # [ 0.065078] rcu: Preemptible hierarchical RCU implementation.587worker2 # [ 0.067164] Dynamic Preempt: lazy588lb # [ 0.204729] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns589client # [ 0.065078] rcu: RCU event tracing is enabled.590worker1 # [ 0.071269] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns591worker2 # [ 0.067310] rcu: Preemptible hierarchical RCU implementation.592client # [ 0.065079] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.593lb # [ 0.206300] APIC: Switch to symmetric I/O mode setup594worker2 # [ 0.067311] rcu: RCU event tracing is enabled.595worker1 # [ 0.071385] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)596client # [ 0.065081] Trampoline variant of Tasks RCU enabled.597lb # [ 0.207244] x2apic enabled598worker2 # [ 0.067311] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.599worker1 # [ 0.075007] Console: colour VGA+ 80x25600client # [ 0.065082] Rude variant of Tasks RCU enabled.601worker2 # [ 0.067313] Trampoline variant of Tasks RCU enabled.602worker1 # [ 0.075011] printk: legacy console [tty0] enabled603client # [ 0.065082] Tracing variant of Tasks RCU enabled.604lb # [ 0.207984] APIC: Switched APIC routing to: physical x2apic605worker2 # [ 0.067314] Rude variant of Tasks RCU enabled.606worker1 # [ 0.105564] printk: legacy console [ttyS0] enabled607worker2 # [ 0.067314] Tracing variant of Tasks RCU enabled.608client # [ 0.065083] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.609worker1 # [ 0.202038] ACPI: Core revision 20250807610client # [ 0.065084] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1611worker2 # [ 0.067315] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.612lb # [ 0.209763] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1613worker2 # [ 0.067315] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1614client # [ 0.065145] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.615worker1 # [ 0.202911] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns616worker2 # [ 0.067369] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.617lb # [ 0.210725] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns618worker1 # [ 0.204436] APIC: Switch to symmetric I/O mode setup619client # [ 0.065148] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.620worker1 # [ 0.205355] x2apic enabled621worker2 # [ 0.067371] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.622client # [ 0.065149] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.623lb # [ 0.212339] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)624worker1 # [ 0.206080] APIC: Switched APIC routing to: physical x2apic625client # [ 0.069726] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16626worker2 # [ 0.067372] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.627lb # [ 0.213428] x86/cpu: User Mode Instruction Prevention (UMIP) activated628client # [ 0.070018] rcu: srcu_init: Setting srcu_struct sizes based on contention.629worker2 # [ 0.072181] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16630lb # [ 0.214457] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127631worker2 # [ 0.072487] rcu: srcu_init: Setting srcu_struct sizes based on contention.632worker1 # [ 0.207846] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1633client # [ 0.070025] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns634lb # [ 0.215336] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0635worker2 # [ 0.072494] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns636client # [ 0.070152] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)637worker1 # [ 0.208776] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns638client # [ 0.073738] Console: colour VGA+ 80x25639worker2 # [ 0.072602] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)640client # [ 0.073741] printk: legacy console [tty0] enabled641lb # [ 0.216340] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto642worker2 # [ 0.076201] Console: colour VGA+ 80x25643worker1 # [ 0.210327] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)644client # [ 0.103910] printk: legacy console [ttyS0] enabled645worker2 # [ 0.076204] printk: legacy console [tty0] enabled646client # [ 0.199765] ACPI: Core revision 20250807647worker2 # [ 0.106762] printk: legacy console [ttyS0] enabled648lb # [ 0.217336] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl649worker1 # [ 0.212411] x86/cpu: User Mode Instruction Prevention (UMIP) activated650worker2 # [ 0.202791] ACPI: Core revision 20250807651lb # [ 0.218336] Transient Scheduler Attacks: Vulnerable: No microcode652worker1 # [ 0.213463] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127653client # [ 0.200638] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns654lb # [ 0.219336] Spectre V2 : Mitigation: Enhanced / Automatic IBRS655worker1 # [ 0.214324] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0656worker2 # [ 0.203651] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns657client # [ 0.202219] APIC: Switch to symmetric I/O mode setup658lb # [ 0.220336] Speculative Return Stack Overflow: Mitigation: Safe RET659client # [ 0.203158] x2apic enabled660worker2 # [ 0.205189] APIC: Switch to symmetric I/O mode setup661worker1 # [ 0.215327] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto662worker2 # [ 0.206132] x2apic enabled663client # [ 0.203935] APIC: Switched APIC routing to: physical x2apic664worker2 # [ 0.206861] APIC: Switched APIC routing to: physical x2apic665worker1 # [ 0.216324] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl666worker1 # [ 0.218324] Transient Scheduler Attacks: Vulnerable: No microcode667client # [ 0.205732] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1668lb # [ 0.221340] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization669worker2 # [ 0.208679] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1670worker1 # [ 0.219323] Spectre V2 : Mitigation: Enhanced / Automatic IBRS671lb # [ 0.223345] Spectre V2 : Enabling IBPB for BPF672worker1 # [ 0.220203] Speculative Return Stack Overflow: Mitigation: Safe RET673client # [ 0.206707] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns674worker2 # [ 0.209648] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns675lb # [ 0.224337] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier676client # [ 0.208350] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)677lb # [ 0.225337] active return thunk: srso_alias_return_thunk678worker2 # [ 0.211289] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)679client # [ 0.209436] x86/cpu: User Mode Instruction Prevention (UMIP) activated680worker2 # [ 0.212372] x86/cpu: User Mode Instruction Prevention (UMIP) activated681lb # [ 0.226357] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'682client # [ 0.210473] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127683worker1 # [ 0.221032] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization684lb # [ 0.227336] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'685worker2 # [ 0.213460] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127686worker1 # [ 0.222331] Spectre V2 : Enabling IBPB for BPF687client # [ 0.211347] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0688lb # [ 0.228336] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'689worker2 # [ 0.214286] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0690worker1 # [ 0.223324] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier691lb # [ 0.229336] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'692client # [ 0.212350] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto693worker1 # [ 0.224324] active return thunk: srso_alias_return_thunk694worker2 # [ 0.215289] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto695lb # [ 0.230336] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'696client # [ 0.214347] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl697worker1 # [ 0.225345] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'698lb # [ 0.231340] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'699worker2 # [ 0.216287] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl700client # [ 0.215347] Transient Scheduler Attacks: Vulnerable: No microcode701worker1 # [ 0.226323] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'702worker2 # [ 0.217286] Transient Scheduler Attacks: Vulnerable: No microcode703lb # [ 0.232336] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'704client # [ 0.216347] Spectre V2 : Mitigation: Enhanced / Automatic IBRS705worker1 # [ 0.227323] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'706worker2 # [ 0.218286] Spectre V2 : Mitigation: Enhanced / Automatic IBRS707lb # [ 0.233336] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256708client # [ 0.217347] Speculative Return Stack Overflow: Mitigation: Safe RET709worker1 # [ 0.228323] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'710worker2 # [ 0.219286] Speculative Return Stack Overflow: Mitigation: Safe RET711lb # [ 0.234336] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64712worker1 # [ 0.229323] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'713lb # [ 0.235336] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512714worker1 # [ 0.230323] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'715lb # [ 0.236336] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024716worker1 # [ 0.231323] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'717lb # [ 0.237336] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8718client # [ 0.218347] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization719worker1 # [ 0.232323] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256720worker2 # [ 0.220285] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization721client # [ 0.220355] Spectre V2 : Enabling IBPB for BPF722lb # [ 0.238336] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.723worker2 # [ 0.222294] Spectre V2 : Enabling IBPB for BPF724worker1 # [ 0.234025] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64725client # [ 0.221350] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier726worker1 # [ 0.234323] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512727worker2 # [ 0.223287] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier728client # [ 0.222348] active return thunk: srso_alias_return_thunk729worker1 # [ 0.235323] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024730worker2 # [ 0.224287] active return thunk: srso_alias_return_thunk731worker1 # [ 0.236323] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8732client # [ 0.223367] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'733worker2 # [ 0.225307] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'734client # [ 0.224347] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'735worker1 # [ 0.237324] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.736worker2 # [ 0.226286] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'737client # [ 0.225347] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'738worker2 # [ 0.227286] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'739client # [ 0.226347] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'740worker2 # [ 0.228286] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'741client # [ 0.227347] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'742worker2 # [ 0.229286] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'743client # [ 0.228347] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'744worker2 # [ 0.230286] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'745client # [ 0.229347] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'746worker2 # [ 0.231286] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'747client # [ 0.230347] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256748worker2 # [ 0.232286] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256749client # [ 0.232057] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64750worker2 # [ 0.233286] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64751client # [ 0.232347] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512752worker2 # [ 0.234286] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512753client # [ 0.233347] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024754worker2 # [ 0.235286] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024755client # [ 0.234347] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8756worker2 # [ 0.236286] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8757client # [ 0.235347] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.758worker2 # [ 0.237286] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.759lb # [ 0.275117] Freeing SMP alternatives memory: 44K760lb # [ 0.275339] pid_max: default: 32768 minimum: 301761lb # [ 0.276423] LSM: initializing lsm=capability,landlock,yama,bpf,ima762lb # [ 0.277448] landlock: Up and running.763lb # [ 0.278337] Yama: becoming mindful.764lb # [ 0.279397] LSM support for eBPF active765worker1 # [ 0.274058] Freeing SMP alternatives memory: 44K766worker1 # [ 0.274327] pid_max: default: 32768 minimum: 301767lb # [ 0.280159] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)768worker1 # [ 0.275420] LSM: initializing lsm=capability,landlock,yama,bpf,ima769lb # [ 0.281381] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)770worker1 # [ 0.276428] landlock: Up and running.771worker1 # [ 0.277324] Yama: becoming mindful.772worker1 # [ 0.278122] LSM support for eBPF active773lb # [ 0.283792] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)774worker1 # [ 0.278890] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)775lb # [ 0.284932] Performance Events: Fam17h+ core perfctr, AMD PMU driver.776worker1 # [ 0.279363] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)777lb # [ 0.285346] ... version: 2778lb # [ 0.286057] ... bit width: 48779lb # [ 0.286338] ... generic counters: 6780lb # [ 0.287012] ... generic bitmap: 000000000000003f781worker1 # [ 0.282999] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)782lb # [ 0.287338] ... fixed-purpose counters: 0783lb # [ 0.288026] ... fixed-purpose bitmap: 0000000000000000784worker2 # [ 0.272726] Freeing SMP alternatives memory: 44K785client # [ 0.271604] Freeing SMP alternatives memory: 44K786lb # [ 0.288338] ... value mask: 0000ffffffffffff787worker1 # [ 0.284034] Performance Events: Fam17h+ core perfctr, AMD PMU driver.788worker2 # [ 0.273290] pid_max: default: 32768 minimum: 301789client # [ 0.272351] pid_max: default: 32768 minimum: 301790lb # [ 0.289177] ... max period: 00007fffffffffff791worker1 # [ 0.285291] ... version: 2792client # [ 0.273436] LSM: initializing lsm=capability,landlock,yama,bpf,ima793worker2 # [ 0.274394] LSM: initializing lsm=capability,landlock,yama,bpf,ima794lb # [ 0.290013] ... global_ctrl mask: 000000000000003f795worker1 # [ 0.286031] ... bit width: 48796client # [ 0.274465] landlock: Up and running.797worker2 # [ 0.275421] landlock: Up and running.798lb # [ 0.290457] signal: max sigframe size: 3376799client # [ 0.275085] Yama: becoming mindful.800worker2 # [ 0.276288] Yama: becoming mindful.801worker1 # [ 0.286396] ... generic counters: 6802lb # [ 0.291288] rcu: Hierarchical SRCU implementation.803client # [ 0.275993] LSM support for eBPF active804worker2 # [ 0.277340] LSM support for eBPF active805worker1 # [ 0.287328] ... generic bitmap: 000000000000003f806lb # [ 0.291934] rcu: Max phase no-delay instances is 400.807client # [ 0.276497] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)808worker1 # [ 0.288328] ... fixed-purpose counters: 0809worker2 # [ 0.278129] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)810worker1 # [ 0.289255] ... fixed-purpose bitmap: 0000000000000000811client # [ 0.277388] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)812worker2 # [ 0.279331] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)813worker1 # [ 0.290230] ... value mask: 0000ffffffffffff814worker1 # [ 0.291329] ... max period: 00007fffffffffff815worker2 # [ 0.281791] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)816client # [ 0.279591] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)817lb # [ 0.297158] smp: Bringing up secondary CPUs ...818worker1 # [ 0.292328] ... global_ctrl mask: 000000000000003f819worker2 # [ 0.282870] Performance Events: Fam17h+ core perfctr, AMD PMU driver.820lb # [ 0.298065] smp: Brought up 1 node, 1 CPU821client # [ 0.281006] Performance Events: Fam17h+ core perfctr, AMD PMU driver.822worker1 # [ 0.293455] signal: max sigframe size: 3376823worker2 # [ 0.283292] ... version: 2824client # [ 0.281358] ... version: 2825lb # [ 0.298340] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)826worker1 # [ 0.294269] rcu: Hierarchical SRCU implementation.827worker2 # [ 0.284013] ... bit width: 48828client # [ 0.282072] ... bit width: 48829worker1 # [ 0.295001] rcu: Max phase no-delay instances is 400.830worker2 # [ 0.284291] ... generic counters: 6831client # [ 0.282351] ... generic counters: 6832worker2 # [ 0.285014] ... generic bitmap: 000000000000003f833client # [ 0.283085] ... generic bitmap: 000000000000003f834lb # [ 0.299506] Memory: 1454564K/1572312K available (17223K kernel code, 2726K rwdata, 13584K rodata, 3644K init, 2988K bss, 109344K reserved, 0K cma-reserved)835worker2 # [ 0.285305] ... fixed-purpose counters: 0836client # [ 0.283352] ... fixed-purpose counters: 0837lb # [ 0.300605] devtmpfs: initialized838worker2 # [ 0.286004] ... fixed-purpose bitmap: 0000000000000000839client # [ 0.284081] ... fixed-purpose bitmap: 0000000000000000840lb # [ 0.301413] x86/mm: Memory block size: 128MB841worker2 # [ 0.286289] ... value mask: 0000ffffffffffff842client # [ 0.284350] ... value mask: 0000ffffffffffff843worker2 # [ 0.287217] ... max period: 00007fffffffffff844client # [ 0.285307] ... max period: 00007fffffffffff845lb # [ 0.303236] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)846worker2 # [ 0.287971] ... global_ctrl mask: 000000000000003f847client # [ 0.286079] ... global_ctrl mask: 000000000000003f848worker1 # [ 0.300215] smp: Bringing up secondary CPUs ...849worker2 # [ 0.288417] signal: max sigframe size: 3376850worker1 # [ 0.300931] smp: Brought up 1 node, 1 CPU851client # [ 0.286477] signal: max sigframe size: 3376852lb # [ 0.304281] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).853worker2 # [ 0.289228] rcu: Hierarchical SRCU implementation.854worker1 # [ 0.301328] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)855client # [ 0.287380] rcu: Hierarchical SRCU implementation.856lb # [ 0.305467] pinctrl core: initialized pinctrl subsystem857worker2 # [ 0.289872] rcu: Max phase no-delay instances is 400.858client # [ 0.288238] rcu: Max phase no-delay instances is 400.859lb # [ 0.306597] PM: RTC time: 12:20:15, date: 2026-09-20860worker1 # [ 0.302529] Memory: 1454576K/1572312K available (17223K kernel code, 2726K rwdata, 13584K rodata, 3644K init, 2988K bss, 109344K reserved, 0K cma-reserved)861worker1 # [ 0.303616] devtmpfs: initialized862worker1 # [ 0.304442] x86/mm: Memory block size: 128MB863lb # [ 0.310263] NET: Registered PF_NETLINK/PF_ROUTE protocol family864worker2 # [ 0.295095] smp: Bringing up secondary CPUs ...865worker1 # [ 0.306264] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)866client # [ 0.293943] smp: Bringing up secondary CPUs ...867lb # [ 0.311604] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations868worker2 # [ 0.295835] smp: Brought up 1 node, 1 CPU869client # [ 0.294367] smp: Brought up 1 node, 1 CPU870worker1 # [ 0.307255] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).871worker2 # [ 0.296290] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)872lb # [ 0.312362] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations873client # [ 0.295043] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)874worker1 # [ 0.308420] pinctrl core: initialized pinctrl subsystem875worker1 # [ 0.309539] PM: RTC time: 12:20:15, date: 2026-09-20876lb # [ 0.313631] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations877worker2 # [ 0.297450] Memory: 1454580K/1572312K available (17223K kernel code, 2726K rwdata, 13584K rodata, 3644K init, 2988K bss, 109344K reserved, 0K cma-reserved)878lb # [ 0.314354] audit: initializing netlink subsys (disabled)879client # [ 0.295585] Memory: 1454540K/1572312K available (17223K kernel code, 2726K rwdata, 13584K rodata, 3644K init, 2988K bss, 109348K reserved, 0K cma-reserved)880worker2 # [ 0.298572] devtmpfs: initialized881client # [ 0.296667] devtmpfs: initialized882worker2 # [ 0.299338] x86/mm: Memory block size: 128MB883lb # [ 0.315526] thermal_sys: Registered thermal governor 'fair_share'884client # [ 0.297520] x86/mm: Memory block size: 128MB885lb # [ 0.315528] thermal_sys: Registered thermal governor 'bang_bang'886worker1 # [ 0.313246] NET: Registered PF_NETLINK/PF_ROUTE protocol family887lb # [ 0.316340] thermal_sys: Registered thermal governor 'step_wise'888worker2 # [ 0.301119] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)889lb # [ 0.317277] thermal_sys: Registered thermal governor 'user_space'890client # [ 0.299379] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)891worker1 # [ 0.314541] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations892worker2 # [ 0.302152] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).893lb # [ 0.318048] audit: type=2000 audit(1789906816.181:1): state=initialized audit_enabled=0 res=1894worker2 # [ 0.303388] pinctrl core: initialized pinctrl subsystem895client # [ 0.300375] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).896worker1 # [ 0.315347] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations897lb # [ 0.319352] thermal_sys: Registered thermal governor 'power_allocator'898client # [ 0.301481] pinctrl core: initialized pinctrl subsystem899worker2 # [ 0.304558] PM: RTC time: 12:20:15, date: 2026-09-20900lb # [ 0.319380] cpuidle: using governor menu901worker1 # [ 0.316617] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations902client # [ 0.302639] PM: RTC time: 12:20:15, date: 2026-09-20903worker1 # [ 0.317339] audit: initializing netlink subsys (disabled)904lb # [ 0.321614] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5905worker1 # [ 0.318328] audit: type=2000 audit(1789906816.193:1): state=initialized audit_enabled=0 res=1906lb # [ 0.322585] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]907worker1 # [ 0.319560] thermal_sys: Registered thermal governor 'fair_share'908worker2 # [ 0.308588] NET: Registered PF_NETLINK/PF_ROUTE protocol family909lb # [ 0.323343] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry910worker1 # [ 0.319562] thermal_sys: Registered thermal governor 'bang_bang'911client # [ 0.306408] NET: Registered PF_NETLINK/PF_ROUTE protocol family912lb # [ 0.324363] PCI: Using configuration type 1 for base access913worker1 # [ 0.320327] thermal_sys: Registered thermal governor 'step_wise'914worker2 # [ 0.309813] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations915client # [ 0.307912] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations916worker1 # [ 0.321248] thermal_sys: Registered thermal governor 'user_space'917worker2 # [ 0.310316] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations918lb # [ 0.325520] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.919worker1 # [ 0.322048] thermal_sys: Registered thermal governor 'power_allocator'920client # [ 0.308377] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations921worker1 # [ 0.322344] cpuidle: using governor menu922worker2 # [ 0.311584] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations923client # [ 0.309659] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations924worker2 # [ 0.312306] audit: initializing netlink subsys (disabled)925worker1 # [ 0.324620] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5926client # [ 0.310363] audit: initializing netlink subsys (disabled)927worker2 # [ 0.313486] thermal_sys: Registered thermal governor 'fair_share'928worker1 # [ 0.325586] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]929worker2 # [ 0.313488] thermal_sys: Registered thermal governor 'bang_bang'930client # [ 0.311626] thermal_sys: Registered thermal governor 'fair_share'931lb # [ 0.330513] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages932client # [ 0.311628] thermal_sys: Registered thermal governor 'bang_bang'933worker2 # [ 0.314290] thermal_sys: Registered thermal governor 'step_wise'934worker1 # [ 0.326330] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry935lb # [ 0.331340] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page936client # [ 0.312351] thermal_sys: Registered thermal governor 'step_wise'937worker1 # [ 0.327340] PCI: Using configuration type 1 for base access938worker2 # [ 0.315213] thermal_sys: Registered thermal governor 'user_space'939client # [ 0.313280] thermal_sys: Registered thermal governor 'user_space'940worker2 # [ 0.316013] audit: type=2000 audit(1789906816.198:1): state=initialized audit_enabled=0 res=1941worker1 # [ 0.328434] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.942client # [ 0.314076] audit: type=2000 audit(1789906816.199:1): state=initialized audit_enabled=0 res=1943worker2 # [ 0.317294] thermal_sys: Registered thermal governor 'power_allocator'944worker2 # [ 0.317327] cpuidle: using governor menu945client # [ 0.315353] thermal_sys: Registered thermal governor 'power_allocator'946client # [ 0.315383] cpuidle: using governor menu947lb # [ 0.336341] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages948worker2 # [ 0.319562] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5949lb # [ 0.337340] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page950client # [ 0.317634] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5951worker2 # [ 0.320603] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]952worker1 # [ 0.333668] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages953worker2 # [ 0.321293] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry954client # [ 0.318948] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]955worker1 # [ 0.334327] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page956worker2 # [ 0.322305] PCI: Using configuration type 1 for base access957client # [ 0.319353] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry958worker1 # [ 0.335327] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages959worker2 # [ 0.323428] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.960client # [ 0.320364] PCI: Using configuration type 1 for base access961client # [ 0.321582] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.962worker2 # [ 0.328631] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages963worker1 # [ 0.342328] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page964lb # [ 0.347836] ACPI: Added _OSI(Module Device)965worker2 # [ 0.329291] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page966lb # [ 0.348344] ACPI: Added _OSI(Processor Device)967client # [ 0.326736] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages968worker2 # [ 0.330290] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages969lb # [ 0.349189] ACPI: Added _OSI(Processor Aggregator Device)970client # [ 0.327352] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page971client # [ 0.328351] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages972worker1 # [ 0.350709] ACPI: Added _OSI(Module Device)973worker1 # [ 0.351328] ACPI: Added _OSI(Processor Device)974worker1 # [ 0.352081] ACPI: Added _OSI(Processor Aggregator Device)975worker2 # [ 0.339291] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page976client # [ 0.335353] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page977lb # [ 0.360117] ACPI: 1 ACPI AML tables successfully acquired and loaded978lb # [ 0.364163] ACPI: Interpreter enabled979worker2 # [ 0.345713] ACPI: Added _OSI(Module Device)980lb # [ 0.364798] ACPI: PM: (supports S0 S3 S4 S5)981worker2 # [ 0.346291] ACPI: Added _OSI(Processor Device)982lb # [ 0.365340] ACPI: Using IOAPIC for interrupt routing983worker2 # [ 0.347018] ACPI: Added _OSI(Processor Aggregator Device)984client # [ 0.343759] ACPI: Added _OSI(Module Device)985worker1 # [ 0.361020] ACPI: 1 ACPI AML tables successfully acquired and loaded986lb # [ 0.366200] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug987client # [ 0.344352] ACPI: Added _OSI(Processor Device)988client # [ 0.345133] ACPI: Added _OSI(Processor Aggregator Device)989worker1 # [ 0.364783] ACPI: Interpreter enabled990worker1 # [ 0.365344] ACPI: PM: (supports S0 S3 S4 S5)991lb # [ 0.371341] PCI: Using E820 reservations for host bridge windows992worker1 # [ 0.366039] ACPI: Using IOAPIC for interrupt routing993lb # [ 0.372456] ACPI: Enabled 2 GPEs in block 00 to 3F994worker2 # [ 0.355805] ACPI: 1 ACPI AML tables successfully acquired and loaded995worker1 # [ 0.368401] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug996worker1 # [ 0.369327] PCI: Using E820 reservations for host bridge windows997client # [ 0.355089] ACPI: 1 ACPI AML tables successfully acquired and loaded998worker2 # [ 0.359704] ACPI: Interpreter enabled999worker1 # [ 0.372444] ACPI: Enabled 2 GPEs in block 00 to 3F1000worker2 # [ 0.360308] ACPI: PM: (supports S0 S3 S4 S5)1001client # [ 0.356331] ACPI: Interpreter enabled1002worker2 # [ 0.361008] ACPI: Using IOAPIC for interrupt routing1003client # [ 0.356863] ACPI: PM: (supports S0 S3 S4 S5)1004lb # [ 0.380865] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1005client # [ 0.357352] ACPI: Using IOAPIC for interrupt routing1006lb # [ 0.381350] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1007client # [ 0.358225] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1008lb # [ 0.382420] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1009worker2 # [ 0.363352] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1010worker2 # [ 0.364289] PCI: Using E820 reservations for host bridge windows1011lb # [ 0.383459] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1012lb # [ 0.384716] PCI host bridge to bus 0000:001013lb # [ 0.385344] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1014worker2 # [ 0.367370] ACPI: Enabled 2 GPEs in block 00 to 3F1015lb # [ 0.386341] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1016worker1 # [ 0.380819] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1017client # [ 0.363350] PCI: Using E820 reservations for host bridge windows1018client # [ 0.364468] ACPI: Enabled 2 GPEs in block 00 to 3F1019lb # [ 0.387339] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1020worker1 # [ 0.381333] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1021lb # [ 0.388339] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1022worker1 # [ 0.382408] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1023lb # [ 0.389339] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1024worker1 # [ 0.383458] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1025lb # [ 0.390339] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1026worker1 # [ 0.384716] PCI host bridge to bus 0000:001027lb # [ 0.391340] pci_bus 0000:00: root bus resource [bus 00-ff]1028worker1 # [ 0.385332] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1029worker1 # [ 0.386326] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1030lb # [ 0.392331] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1031worker1 # [ 0.387326] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1032worker2 # [ 0.376080] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1033lb # [ 0.393924] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1034worker1 # [ 0.388326] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1035worker2 # [ 0.377063] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1036client # [ 0.373216] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1037worker1 # [ 0.389327] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1038worker2 # [ 0.378380] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1039worker1 # [ 0.390334] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1040worker2 # [ 0.379415] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1041client # [ 0.375270] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1042worker1 # [ 0.391327] pci_bus 0000:00: root bus resource [bus 00-ff]1043worker2 # [ 0.380665] PCI host bridge to bus 0000:001044client # [ 0.376442] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1045worker1 # [ 0.392297] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1046worker2 # [ 0.381293] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1047client # [ 0.377475] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1048worker2 # [ 0.382289] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1049worker1 # [ 0.393812] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1050client # [ 0.378692] PCI host bridge to bus 0000:001051worker2 # [ 0.383289] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1052client # [ 0.379354] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1053worker2 # [ 0.384289] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1054client # [ 0.380350] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1055worker2 # [ 0.385289] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1056client # [ 0.381349] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1057worker2 # [ 0.386289] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1058client # [ 0.382350] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1059worker2 # [ 0.387290] pci_bus 0000:00: root bus resource [bus 00-ff]1060lb # [ 0.396445] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1061client # [ 0.383349] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1062worker2 # [ 0.388345] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1063lb # [ 0.397363] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1064client # [ 0.384349] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1065lb # [ 0.398376] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1066client # [ 0.385351] pci_bus 0000:00: root bus resource [bus 00-ff]1067worker2 # [ 0.389749] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1068worker1 # [ 0.396417] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1069client # [ 0.386387] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1070worker1 # [ 0.397339] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1071lb # [ 0.399867] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1072worker1 # [ 0.398295] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1073client # [ 0.387805] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1074worker1 # [ 0.399514] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1075lb # [ 0.401454] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1076worker1 # [ 0.401152] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1077lb # [ 0.404357] pci 0000:00:02.0: BAR 0 [io 0xc140-0xc15f]1078worker2 # [ 0.392366] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1079lb # [ 0.405353] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1080worker1 # [ 0.403363] pci 0000:00:02.0: BAR 0 [io 0xc140-0xc15f]1081worker2 # [ 0.393350] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1082worker1 # [ 0.404332] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1083lb # [ 0.406370] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1084worker2 # [ 0.394299] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1085worker1 # [ 0.405313] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1086client # [ 0.390420] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1087lb # [ 0.407354] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1088worker2 # [ 0.395630] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1089worker1 # [ 0.406278] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1090client # [ 0.391363] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1091client # [ 0.392366] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1092worker1 # [ 0.407695] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1093worker2 # [ 0.397081] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1094lb # [ 0.409286] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1095client # [ 0.393759] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1096worker1 # [ 0.409343] pci 0000:00:03.0: BAR 0 [io 0xc160-0xc17f]1097worker2 # [ 0.399305] pci 0000:00:02.0: BAR 0 [io 0xc140-0xc15f]1098client # [ 0.395129] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1099worker1 # [ 0.410189] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1100worker2 # [ 0.400156] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1101lb # [ 0.412354] pci 0000:00:03.0: BAR 0 [io 0xc160-0xc17f]1102worker1 # [ 0.411069] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1103worker2 # [ 0.401017] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1104lb # [ 0.413352] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1105worker2 # [ 0.401294] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1106worker1 # [ 0.411906] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1107client # [ 0.397357] pci 0000:00:02.0: BAR 0 [io 0xc140-0xc15f]1108lb # [ 0.414373] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1109client # [ 0.398214] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1110worker2 # [ 0.402892] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1111client # [ 0.399108] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1112lb # [ 0.416093] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1113worker1 # [ 0.414254] pci 0000:00:04.0: BAR 0 [io 0xc080-0xc0bf]1114client # [ 0.400211] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1115worker2 # [ 0.404310] pci 0000:00:03.0: BAR 0 [io 0xc160-0xc17f]1116worker1 # [ 0.414966] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1117worker2 # [ 0.405132] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1118worker1 # [ 0.415348] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1119client # [ 0.401838] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1120worker2 # [ 0.406002] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1121worker1 # [ 0.416900] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1122lb # [ 0.419355] pci 0000:00:04.0: BAR 0 [io 0xc080-0xc0bf]1123client # [ 0.403358] pci 0000:00:03.0: BAR 0 [io 0xc160-0xc17f]1124worker2 # [ 0.406857] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1125client # [ 0.404225] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1126lb # [ 0.420354] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1127worker1 # [ 0.419101] pci 0000:00:05.0: BAR 0 [io 0xc180-0xc19f]1128client # [ 0.405131] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1129lb # [ 0.421374] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1130worker1 # [ 0.419334] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1131worker2 # [ 0.409311] pci 0000:00:04.0: BAR 0 [io 0xc080-0xc0bf]1132client # [ 0.406785] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1133worker2 # [ 0.410279] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1134worker1 # [ 0.420303] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1135lb # [ 0.423099] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1136worker2 # [ 0.411069] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1137worker1 # [ 0.421953] pci 0000:00:06.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1138worker2 # [ 0.412759] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1139client # [ 0.409360] pci 0000:00:04.0: BAR 0 [io 0xc080-0xc0bf]1140lb # [ 0.426383] pci 0000:00:05.0: BAR 0 [io 0xc180-0xc19f]1141worker1 # [ 0.423358] pci 0000:00:06.0: BAR 0 [io 0xc1a0-0xc1bf]1142client # [ 0.410271] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1143worker1 # [ 0.424216] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1144lb # [ 0.427354] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1145client # [ 0.411165] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1146worker2 # [ 0.415271] pci 0000:00:05.0: BAR 0 [io 0xc180-0xc19f]1147worker1 # [ 0.425062] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1148worker2 # [ 0.415959] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1149lb # [ 0.428372] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1150client # [ 0.412823] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1151worker1 # [ 0.425888] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1152worker2 # [ 0.416312] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1153lb # [ 0.430102] pci 0000:00:06.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1154worker2 # [ 0.417896] pci 0000:00:06.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1155client # [ 0.415320] pci 0000:00:05.0: BAR 0 [io 0xc180-0xc19f]1156client # [ 0.416031] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1157worker1 # [ 0.428502] pci 0000:00:07.0: BAR 0 [io 0xc000-0xc07f]1158lb # [ 0.433375] pci 0000:00:06.0: BAR 0 [io 0xc1a0-0xc1bf]1159worker2 # [ 0.419420] pci 0000:00:06.0: BAR 0 [io 0xc1a0-0xc1bf]1160client # [ 0.416376] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1161worker1 # [ 0.429341] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1162lb # [ 0.434354] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1163worker2 # [ 0.420304] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1164worker1 # [ 0.430363] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1165client # [ 0.417944] pci 0000:00:06.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1166worker2 # [ 0.421326] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1167lb # [ 0.435374] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1168worker1 # [ 0.431974] pci 0000:00:08.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1169client # [ 0.419358] pci 0000:00:06.0: BAR 0 [io 0xc1a0-0xc1bf]1170worker2 # [ 0.423088] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1171lb # [ 0.437173] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1172client # [ 0.420231] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1173worker1 # [ 0.434368] pci 0000:00:08.0: BAR 0 [io 0xc1c0-0xc1df]1174client # [ 0.421130] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1175worker1 # [ 0.435262] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1176client # [ 0.422885] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1177worker2 # [ 0.426327] pci 0000:00:07.0: BAR 0 [io 0xc000-0xc07f]1178worker1 # [ 0.436072] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1179lb # [ 0.440354] pci 0000:00:07.0: BAR 0 [io 0xc000-0xc07f]1180worker1 # [ 0.436335] pci 0000:00:08.0: ROM [mem 0xfeb80000-0xfebbffff pref]1181worker2 # [ 0.427303] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1182lb # [ 0.441353] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1183worker2 # [ 0.428324] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1184lb # [ 0.442389] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1185client # [ 0.425361] pci 0000:00:07.0: BAR 0 [io 0xc000-0xc07f]1186worker1 # [ 0.438030] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1187client # [ 0.426243] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1188client # [ 0.427135] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1189worker2 # [ 0.430119] pci 0000:00:08.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1190worker1 # [ 0.440340] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1191lb # [ 0.444464] pci 0000:00:08.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1192client # [ 0.428862] pci 0000:00:08.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1193worker1 # [ 0.441353] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1194worker2 # [ 0.433304] pci 0000:00:08.0: BAR 0 [io 0xc1c0-0xc1df]1195lb # [ 0.447418] pci 0000:00:08.0: BAR 0 [io 0xc1c0-0xc1df]1196client # [ 0.430407] pci 0000:00:08.0: BAR 0 [io 0xc1c0-0xc1df]1197worker2 # [ 0.434302] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1198lb # [ 0.448354] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1199worker1 # [ 0.443038] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1200client # [ 0.431302] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1201worker2 # [ 0.435323] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1202lb # [ 0.449377] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1203client # [ 0.432135] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1204worker2 # [ 0.436300] pci 0000:00:08.0: ROM [mem 0xfeb80000-0xfebbffff pref]1205lb # [ 0.450354] pci 0000:00:08.0: ROM [mem 0xfeb80000-0xfebbffff pref]1206worker1 # [ 0.446191] pci 0000:00:0a.0: BAR 0 [io 0xc0c0-0xc0ff]1207client # [ 0.433194] pci 0000:00:08.0: ROM [mem 0xfeb80000-0xfebbffff pref]1208worker1 # [ 0.447085] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1209worker2 # [ 0.438066] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1210client # [ 0.434665] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1211worker1 # [ 0.448202] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1212lb # [ 0.452098] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1213client # [ 0.436186] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1214worker1 # [ 0.450019] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1215client # [ 0.437150] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1216worker2 # [ 0.440311] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1217lb # [ 0.454360] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1218worker2 # [ 0.441322] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1219client # [ 0.438701] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1220lb # [ 0.455376] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1221worker1 # [ 0.453263] pci 0000:00:0b.0: BAR 0 [io 0xc1e0-0xc1ff]1222worker2 # [ 0.443062] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1223worker1 # [ 0.454055] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1224lb # [ 0.457099] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1225client # [ 0.440510] pci 0000:00:0a.0: BAR 0 [io 0xc0c0-0xc0ff]1226worker1 # [ 0.454355] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1227client # [ 0.441356] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1228client # [ 0.442349] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1229worker2 # [ 0.446228] pci 0000:00:0a.0: BAR 0 [io 0xc0c0-0xc0ff]1230worker1 # [ 0.456004] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1231worker2 # [ 0.447102] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1232lb # [ 0.460961] pci 0000:00:0a.0: BAR 0 [io 0xc0c0-0xc0ff]1233client # [ 0.443912] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1234worker1 # [ 0.458079] pci 0000:00:1d.0: BAR 4 [io 0xc200-0xc21f]1235lb # [ 0.461353] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1236worker2 # [ 0.448277] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1237client # [ 0.445369] pci 0000:00:0b.0: BAR 0 [io 0xc1e0-0xc1ff]1238worker1 # [ 0.458591] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1239lb # [ 0.462372] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1240client # [ 0.446232] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1241client # [ 0.447078] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1242worker2 # [ 0.450091] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1243worker1 # [ 0.460041] pci 0000:00:1d.1: BAR 4 [io 0xc220-0xc23f]1244lb # [ 0.463995] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1245client # [ 0.447939] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1246lb # [ 0.465348] pci 0000:00:0b.0: BAR 0 [io 0xc1e0-0xc1ff]1247worker2 # [ 0.453201] pci 0000:00:0b.0: BAR 0 [io 0xc1e0-0xc1ff]1248client # [ 0.448918] pci 0000:00:1d.0: BAR 4 [io 0xc200-0xc21f]1249lb # [ 0.466188] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1250worker2 # [ 0.454148] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1251lb # [ 0.467043] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1252client # [ 0.449553] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1253worker2 # [ 0.455242] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1254client # [ 0.450927] pci 0000:00:1d.1: BAR 4 [io 0xc220-0xc23f]1255lb # [ 0.468167] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1256lb # [ 0.470049] pci 0000:00:1d.0: BAR 4 [io 0xc200-0xc21f]1257worker2 # [ 0.457348] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1258lb # [ 0.470552] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1259worker2 # [ 0.459122] pci 0000:00:1d.0: BAR 4 [io 0xc200-0xc21f]1260lb # [ 0.471990] pci 0000:00:1d.1: BAR 4 [io 0xc220-0xc23f]1261worker2 # [ 0.460391] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1262worker2 # [ 0.462123] pci 0000:00:1d.1: BAR 4 [io 0xc220-0xc23f]1263worker1 # [ 0.460621] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1264worker1 # [ 0.463363] pci 0000:00:1d.2: BAR 4 [io 0xc240-0xc25f]1265client # [ 0.451544] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1266worker1 # [ 0.464628] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1267client # [ 0.453882] pci 0000:00:1d.2: BAR 4 [io 0xc240-0xc25f]1268worker1 # [ 0.466336] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1269client # [ 0.454624] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1270client # [ 0.455879] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1271worker1 # [ 0.467679] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1272lb # [ 0.472552] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1273client # [ 0.456607] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1274worker1 # [ 0.468662] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1275lb # [ 0.475025] pci 0000:00:1d.2: BAR 4 [io 0xc240-0xc25f]1276client # [ 0.457636] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1277worker1 # [ 0.469663] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1278lb # [ 0.475577] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1279client # [ 0.458593] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1280lb # [ 0.476972] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1281worker1 # [ 0.471355] pci 0000:00:1f.2: BAR 4 [io 0xc260-0xc27f]1282client # [ 0.460351] pci 0000:00:1f.2: BAR 4 [io 0xc260-0xc27f]1283worker1 # [ 0.472213] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1284client # [ 0.461213] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1285lb # [ 0.477619] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1286worker2 # [ 0.463399] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1287worker1 # [ 0.473391] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1288lb # [ 0.478629] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1289client # [ 0.462356] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1290client # [ 0.463891] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1291lb # [ 0.479598] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1292worker2 # [ 0.466828] pci 0000:00:1d.2: BAR 4 [io 0xc240-0xc25f]1293worker1 # [ 0.475371] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1294lb # [ 0.481310] pci 0000:00:1f.2: BAR 4 [io 0xc260-0xc27f]1295worker2 # [ 0.467653] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1296lb # [ 0.481998] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1297worker2 # [ 0.469324] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1298client # [ 0.467279] ACPI: PCI: Interrupt link LNKA configured for IRQ 101299worker1 # [ 0.479497] ACPI: PCI: Interrupt link LNKA configured for IRQ 101300lb # [ 0.483442] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1301client # [ 0.468198] ACPI: PCI: Interrupt link LNKB configured for IRQ 101302worker2 # [ 0.470535] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1303worker1 # [ 0.480438] ACPI: PCI: Interrupt link LNKB configured for IRQ 101304client # [ 0.469156] ACPI: PCI: Interrupt link LNKC configured for IRQ 111305lb # [ 0.485039] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1306worker1 # [ 0.481424] ACPI: PCI: Interrupt link LNKC configured for IRQ 111307worker2 # [ 0.471606] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1308client # [ 0.470176] ACPI: PCI: Interrupt link LNKD configured for IRQ 111309worker1 # [ 0.482428] ACPI: PCI: Interrupt link LNKD configured for IRQ 111310client # [ 0.471164] ACPI: PCI: Interrupt link LNKE configured for IRQ 101311worker2 # [ 0.472586] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1312worker1 # [ 0.483420] ACPI: PCI: Interrupt link LNKE configured for IRQ 101313client # [ 0.472317] ACPI: PCI: Interrupt link LNKF configured for IRQ 101314worker1 # [ 0.484424] ACPI: PCI: Interrupt link LNKF configured for IRQ 101315lb # [ 0.488424] ACPI: PCI: Interrupt link LNKA configured for IRQ 101316worker1 # [ 0.485443] ACPI: PCI: Interrupt link LNKG configured for IRQ 111317worker2 # [ 0.474317] pci 0000:00:1f.2: BAR 4 [io 0xc260-0xc27f]1318lb # [ 0.489452] ACPI: PCI: Interrupt link LNKB configured for IRQ 101319worker2 # [ 0.475150] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1320worker1 # [ 0.486426] ACPI: PCI: Interrupt link LNKH configured for IRQ 111321client # [ 0.473209] ACPI: PCI: Interrupt link LNKG configured for IRQ 111322lb # [ 0.490436] ACPI: PCI: Interrupt link LNKC configured for IRQ 111323client # [ 0.474507] ACPI: PCI: Interrupt link LNKH configured for IRQ 111324worker1 # [ 0.487366] ACPI: PCI: Interrupt link GSIA configured for IRQ 161325worker2 # [ 0.476342] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1326lb # [ 0.491443] ACPI: PCI: Interrupt link LNKD configured for IRQ 111327client # [ 0.475388] ACPI: PCI: Interrupt link GSIA configured for IRQ 161328worker1 # [ 0.488342] ACPI: PCI: Interrupt link GSIB configured for IRQ 171329lb # [ 0.492438] ACPI: PCI: Interrupt link LNKE configured for IRQ 101330client # [ 0.476326] ACPI: PCI: Interrupt link GSIB configured for IRQ 171331worker1 # [ 0.489333] ACPI: PCI: Interrupt link GSIC configured for IRQ 181332worker2 # [ 0.478348] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1333lb # [ 0.493443] ACPI: PCI: Interrupt link LNKF configured for IRQ 101334client # [ 0.477074] ACPI: PCI: Interrupt link GSIC configured for IRQ 181335worker1 # [ 0.490282] ACPI: PCI: Interrupt link GSID configured for IRQ 191336lb # [ 0.494441] ACPI: PCI: Interrupt link LNKG configured for IRQ 111337client # [ 0.477362] ACPI: PCI: Interrupt link GSID configured for IRQ 191338worker1 # [ 0.491046] ACPI: PCI: Interrupt link GSIE configured for IRQ 201339lb # [ 0.495443] ACPI: PCI: Interrupt link LNKH configured for IRQ 111340client # [ 0.478294] ACPI: PCI: Interrupt link GSIE configured for IRQ 201341worker1 # [ 0.491339] ACPI: PCI: Interrupt link GSIF configured for IRQ 211342worker1 # [ 0.492269] ACPI: PCI: Interrupt link GSIG configured for IRQ 221343lb # [ 0.496440] ACPI: PCI: Interrupt link GSIA configured for IRQ 161344client # [ 0.479051] ACPI: PCI: Interrupt link GSIF configured for IRQ 211345worker1 # [ 0.493095] ACPI: PCI: Interrupt link GSIH configured for IRQ 231346lb # [ 0.497443] ACPI: PCI: Interrupt link GSIB configured for IRQ 171347worker2 # [ 0.482272] ACPI: PCI: Interrupt link LNKA configured for IRQ 101348client # [ 0.480095] ACPI: PCI: Interrupt link GSIG configured for IRQ 221349client # [ 0.480366] ACPI: PCI: Interrupt link GSIH configured for IRQ 231350worker2 # [ 0.483400] ACPI: PCI: Interrupt link LNKB configured for IRQ 101351worker1 # [ 0.495775] iommu: Default domain type: Translated1352worker2 # [ 0.484430] ACPI: PCI: Interrupt link LNKC configured for IRQ 111353client # [ 0.482525] iommu: Default domain type: Translated1354worker1 # [ 0.496338] iommu: DMA domain TLB invalidation policy: lazy mode1355worker2 # [ 0.485400] ACPI: PCI: Interrupt link LNKD configured for IRQ 111356client # [ 0.483310] iommu: DMA domain TLB invalidation policy: lazy mode1357lb # [ 0.498368] ACPI: PCI: Interrupt link GSIC configured for IRQ 181358worker1 # [ 0.497591] ACPI: bus type USB registered1359worker2 # [ 0.486395] ACPI: PCI: Interrupt link LNKE configured for IRQ 101360client # [ 0.484366] ACPI: bus type USB registered1361lb # [ 0.499371] ACPI: PCI: Interrupt link GSID configured for IRQ 191362worker1 # [ 0.498317] usbcore: registered new interface driver usbfs1363worker2 # [ 0.487403] ACPI: PCI: Interrupt link LNKF configured for IRQ 101364client # [ 0.485095] usbcore: registered new interface driver usbfs1365lb # [ 0.500355] ACPI: PCI: Interrupt link GSIE configured for IRQ 201366client # [ 0.485365] usbcore: registered new interface driver hub1367worker2 # [ 0.488394] ACPI: PCI: Interrupt link LNKG configured for IRQ 111368lb # [ 0.501304] ACPI: PCI: Interrupt link GSIF configured for IRQ 211369client # [ 0.486224] usbcore: registered new device driver usb1370worker2 # [ 0.489385] ACPI: PCI: Interrupt link LNKH configured for IRQ 111371worker1 # [ 0.499240] usbcore: registered new interface driver hub1372lb # [ 0.502108] ACPI: PCI: Interrupt link GSIG configured for IRQ 221373worker2 # [ 0.490328] ACPI: PCI: Interrupt link GSIA configured for IRQ 161374worker1 # [ 0.500349] usbcore: registered new device driver usb1375lb # [ 0.502359] ACPI: PCI: Interrupt link GSIH configured for IRQ 231376client # [ 0.487803] NetLabel: Initializing1377client # [ 0.488351] NetLabel: domain hash size = 1281378worker2 # [ 0.491289] ACPI: PCI: Interrupt link GSIB configured for IRQ 171379worker1 # [ 0.502315] NetLabel: Initializing1380client # [ 0.489053] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1381lb # [ 0.504443] iommu: Default domain type: Translated1382worker2 # [ 0.492312] ACPI: PCI: Interrupt link GSIC configured for IRQ 181383worker1 # [ 0.502999] NetLabel: domain hash size = 1281384client # [ 0.489407] NetLabel: unlabeled traffic allowed by default1385lb # [ 0.505240] iommu: DMA domain TLB invalidation policy: lazy mode1386worker2 # [ 0.493301] ACPI: PCI: Interrupt link GSID configured for IRQ 191387worker1 # [ 0.503326] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1388client # [ 0.490335] PCI: Using ACPI for IRQ routing1389lb # [ 0.506321] ACPI: bus type USB registered1390worker2 # [ 0.494290] ACPI: PCI: Interrupt link GSIE configured for IRQ 201391worker1 # [ 0.504270] NetLabel: unlabeled traffic allowed by default1392lb # [ 0.506866] usbcore: registered new interface driver usbfs1393worker2 # [ 0.495238] ACPI: PCI: Interrupt link GSIF configured for IRQ 211394worker1 # [ 0.505010] PCI: Using ACPI for IRQ routing1395worker2 # [ 0.496053] ACPI: PCI: Interrupt link GSIG configured for IRQ 221396lb # [ 0.507356] usbcore: registered new interface driver hub1397lb # [ 0.508354] usbcore: registered new device driver usb1398worker2 # [ 0.496352] ACPI: PCI: Interrupt link GSIH configured for IRQ 231399lb # [ 0.510249] NetLabel: Initializing1400worker2 # [ 0.498397] iommu: Default domain type: Translated1401lb # [ 0.510796] NetLabel: domain hash size = 1281402lb # [ 0.511339] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1403lb # [ 0.512330] NetLabel: unlabeled traffic allowed by default1404worker2 # [ 0.499232] iommu: DMA domain TLB invalidation policy: lazy mode1405lb # [ 0.513055] PCI: Using ACPI for IRQ routing1406worker2 # [ 0.500609] ACPI: bus type USB registered1407worker2 # [ 0.501349] usbcore: registered new interface driver usbfs1408worker2 # [ 0.502304] usbcore: registered new interface driver hub1409worker2 # [ 0.503251] usbcore: registered new device driver usb1410worker2 # [ 0.504936] NetLabel: Initializing1411worker2 # [ 0.505290] NetLabel: domain hash size = 1281412worker2 # [ 0.506078] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1413worker2 # [ 0.507068] NetLabel: unlabeled traffic allowed by default1414worker2 # [ 0.507301] PCI: Using ACPI for IRQ routing1415client # [ 0.533387] pci 0000:00:01.0: vgaarb: setting as boot VGA device1416client # [ 0.534346] pci 0000:00:01.0: vgaarb: bridge control possible1417client # [ 0.534346] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1418worker1 # [ 0.549008] pci 0000:00:01.0: vgaarb: setting as boot VGA device1419client # [ 0.534353] vgaarb: loaded1420worker1 # [ 0.549322] pci 0000:00:01.0: vgaarb: bridge control possible1421client # [ 0.535084] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01422client # [ 0.535352] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1423worker1 # [ 0.549322] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1424worker1 # [ 0.549332] vgaarb: loaded1425worker1 # [ 0.550096] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01426worker1 # [ 0.550326] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1427client # [ 0.538443] clocksource: Switched to clocksource kvm-clock1428lb # [ 0.556840] pci 0000:00:01.0: vgaarb: setting as boot VGA device1429client # [ 0.540169] VFS: Disk quotas dquot_6.6.01430lb # [ 0.557335] pci 0000:00:01.0: vgaarb: bridge control possible1431lb # [ 0.557335] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1432lb # [ 0.557343] vgaarb: loaded1433client # [ 0.542755] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1434lb # [ 0.558107] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01435worker1 # [ 0.556425] clocksource: Switched to clocksource kvm-clock1436client # [ 0.544088] pnp: PnP ACPI init1437lb # [ 0.558341] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1438client # [ 0.544914] ACPI: IRQ 4 override to edge(!), high(!)1439client # [ 0.545968] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved1440worker1 # [ 0.560164] VFS: Disk quotas dquot_6.6.01441worker2 # [ 0.551324] pci 0000:00:01.0: vgaarb: setting as boot VGA device1442client # [ 0.547561] pnp: PnP ACPI: found 6 devices1443worker1 # [ 0.560951] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1444worker2 # [ 0.552285] pci 0000:00:01.0: vgaarb: bridge control possible1445worker1 # [ 0.562382] pnp: PnP ACPI init1446lb # [ 0.562437] clocksource: Switched to clocksource kvm-clock1447worker2 # [ 0.552285] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1448worker1 # [ 0.563270] ACPI: IRQ 4 override to edge(!), high(!)1449worker2 # [ 0.552296] vgaarb: loaded1450lb # [ 0.564134] VFS: Disk quotas dquot_6.6.01451worker2 # [ 0.553077] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01452worker1 # [ 0.564318] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved1453worker2 # [ 0.553291] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1454worker1 # [ 0.565887] pnp: PnP ACPI: found 6 devices1455lb # [ 0.566777] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1456lb # [ 0.568225] pnp: PnP ACPI init1457client # [ 0.555193] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1458lb # [ 0.569129] ACPI: IRQ 4 override to edge(!), high(!)1459client # [ 0.556742] clocksource: Switched to clocksource acpi_pm1460lb # [ 0.570202] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved1461client # [ 0.557700] NET: Registered PF_INET protocol family1462lb # [ 0.571739] pnp: PnP ACPI: found 6 devices1463worker2 # [ 0.559388] clocksource: Switched to clocksource kvm-clock1464client # [ 0.558914] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1465worker1 # [ 0.573598] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1466worker2 # [ 0.562919] VFS: Disk quotas dquot_6.6.01467worker1 # [ 0.575098] clocksource: Switched to clocksource acpi_pm1468worker1 # [ 0.576097] NET: Registered PF_INET protocol family1469worker2 # [ 0.563641] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1470worker2 # [ 0.565080] pnp: PnP ACPI init1471worker1 # [ 0.577312] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1472worker2 # [ 0.565949] ACPI: IRQ 4 override to edge(!), high(!)1473worker2 # [ 0.566972] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved1474lb # [ 0.579507] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1475worker2 # [ 0.568546] pnp: PnP ACPI: found 6 devices1476lb # [ 0.580985] clocksource: Switched to clocksource acpi_pm1477lb # [ 0.581996] NET: Registered PF_INET protocol family1478lb # [ 0.583589] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1479worker2 # [ 0.576240] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1480client # [ 0.575873] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1481worker2 # [ 0.577755] clocksource: Switched to clocksource acpi_pm1482client # [ 0.577432] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1483worker2 # [ 0.578731] NET: Registered PF_INET protocol family1484client # [ 0.578743] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1485worker2 # [ 0.579926] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1486client # [ 0.580079] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1487client # [ 0.581401] TCP: Hash tables configured (established 16384 bind 16384)1488worker1 # [ 0.594750] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1489client # [ 0.582559] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1490client # [ 0.583815] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1491worker1 # [ 0.596540] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1492client # [ 0.584935] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1493worker1 # [ 0.598496] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1494client # [ 0.586239] NET: Registered PF_UNIX/PF_LOCAL protocol family1495client # [ 0.587244] NET: Registered PF_XDP protocol family1496worker1 # [ 0.600169] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1497client # [ 0.588094] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1498lb # [ 0.601346] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1499worker1 # [ 0.601690] TCP: Hash tables configured (established 16384 bind 16384)1500client # [ 0.589130] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1501client # [ 0.590232] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1502lb # [ 0.603271] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1503worker1 # [ 0.603174] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1504client # [ 0.591288] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1505worker1 # [ 0.604660] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1506lb # [ 0.604647] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1507client # [ 0.592428] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1508worker1 # [ 0.605785] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1509lb # [ 0.606100] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1510client # [ 0.593503] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1511worker1 # [ 0.606987] NET: Registered PF_UNIX/PF_LOCAL protocol family1512lb # [ 0.607420] TCP: Hash tables configured (established 16384 bind 16384)1513worker1 # [ 0.607973] NET: Registered PF_XDP protocol family1514client # [ 0.595194] ACPI: \_SB_.GSIA: Enabled at IRQ 161515lb # [ 0.608636] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1516worker1 # [ 0.608796] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1517worker1 # [ 0.609814] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1518lb # [ 0.609923] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1519worker2 # [ 0.597323] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1520client # [ 0.597372] ACPI: \_SB_.GSIB: Enabled at IRQ 171521worker1 # [ 0.610781] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1522lb # [ 0.611073] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1523worker2 # [ 0.598921] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1524worker1 # [ 0.611862] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1525lb # [ 0.612455] NET: Registered PF_UNIX/PF_LOCAL protocol family1526client # [ 0.599368] ACPI: \_SB_.GSIC: Enabled at IRQ 181527worker2 # [ 0.600271] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1528worker1 # [ 0.612956] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1529lb # [ 0.613523] NET: Registered PF_XDP protocol family1530worker2 # [ 0.601629] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1531worker1 # [ 0.614037] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1532client # [ 0.601516] ACPI: \_SB_.GSID: Enabled at IRQ 191533lb # [ 0.614424] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1534worker2 # [ 0.602871] TCP: Hash tables configured (established 16384 bind 16384)1535worker1 # [ 0.615721] ACPI: \_SB_.GSIA: Enabled at IRQ 161536lb # [ 0.615525] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1537client # [ 0.603213] PCI: CLS 0 bytes, default 641538worker2 # [ 0.604054] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1539lb # [ 0.616697] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1540worker2 # [ 0.605328] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1541client # [ 0.604096] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1542lb # [ 0.617987] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1543worker1 # [ 0.618436] ACPI: \_SB_.GSIB: Enabled at IRQ 171544worker2 # [ 0.606410] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1545lb # [ 0.619209] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1546client # [ 0.606035] Trying to unpack rootfs image as initramfs...1547worker2 # [ 0.607694] NET: Registered PF_UNIX/PF_LOCAL protocol family1548worker1 # [ 0.620412] ACPI: \_SB_.GSIC: Enabled at IRQ 181549lb # [ 0.620328] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1550worker2 # [ 0.608649] NET: Registered PF_XDP protocol family1551lb # [ 0.621985] ACPI: \_SB_.GSIA: Enabled at IRQ 161552worker2 # [ 0.609565] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1553worker1 # [ 0.622395] ACPI: \_SB_.GSID: Enabled at IRQ 191554worker2 # [ 0.610616] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1555worker1 # [ 0.624093] PCI: CLS 0 bytes, default 641556lb # [ 0.624336] ACPI: \_SB_.GSIB: Enabled at IRQ 171557worker2 # [ 0.611684] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1558worker2 # [ 0.612748] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1559worker1 # [ 0.624970] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1560lb # [ 0.626525] ACPI: \_SB_.GSIC: Enabled at IRQ 181561worker2 # [ 0.613888] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1562worker1 # [ 0.626696] Trying to unpack rootfs image as initramfs...1563worker2 # [ 0.615003] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1564lb # [ 0.628476] ACPI: \_SB_.GSID: Enabled at IRQ 191565worker2 # [ 0.616839] ACPI: \_SB_.GSIA: Enabled at IRQ 161566lb # [ 0.630195] PCI: CLS 0 bytes, default 641567worker2 # [ 0.619169] ACPI: \_SB_.GSIB: Enabled at IRQ 171568lb # [ 0.631059] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1569lb # [ 0.632925] Trying to unpack rootfs image as initramfs...1570worker2 # [ 0.621923] ACPI: \_SB_.GSIC: Enabled at IRQ 181571worker2 # [ 0.623942] ACPI: \_SB_.GSID: Enabled at IRQ 191572worker2 # [ 0.625665] PCI: CLS 0 bytes, default 641573worker2 # [ 0.626626] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1574worker2 # [ 0.628408] Trying to unpack rootfs image as initramfs...1575client # [ 0.655839] Initialise system trusted keyrings1576worker1 # [ 0.672547] Initialise system trusted keyrings1577client # [ 0.659399] workingset: timestamp_bits=40 max_order=19 bucket_order=01578worker1 # [ 0.676134] workingset: timestamp_bits=40 max_order=19 bucket_order=01579lb # [ 0.679514] Initialise system trusted keyrings1580lb # [ 0.683213] workingset: timestamp_bits=40 max_order=19 bucket_order=01581worker2 # [ 0.680767] Initialise system trusted keyrings1582worker2 # [ 0.682013] workingset: timestamp_bits=40 max_order=19 bucket_order=01583client # [ 0.683405] Key type asymmetric registered1584client # [ 0.684190] Asymmetric key parser 'x509' registered1585worker1 # [ 0.697707] Key type asymmetric registered1586worker1 # [ 0.698495] Asymmetric key parser 'x509' registered1587client # [ 0.685105] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1588client # [ 0.688534] io scheduler mq-deadline registered1589worker1 # [ 0.701149] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1590client # [ 0.689361] io scheduler kyber registered1591worker1 # [ 0.704187] io scheduler mq-deadline registered1592lb # [ 0.704842] Key type asymmetric registered1593worker1 # [ 0.704995] io scheduler kyber registered1594lb # [ 0.705645] Asymmetric key parser 'x509' registered1595client # [ 0.693069] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1596worker1 # [ 0.707773] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1597lb # [ 0.708244] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1598client # [ 0.696383] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1599lb # [ 0.711233] io scheduler mq-deadline registered1600lb # [ 0.712031] io scheduler kyber registered1601worker1 # [ 0.712252] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1602client # [ 0.700126] Linux agpgart interface v0.1031603client # [ 0.700955] ACPI: bus type drm_connector registered1604lb # [ 0.714864] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1605worker1 # [ 0.716286] Linux agpgart interface v0.1031606worker1 # [ 0.717099] ACPI: bus type drm_connector registered1607lb # [ 0.718192] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1608client # [ 0.705969] usbcore: registered new interface driver usbserial_generic1609client # [ 0.707124] usbserial: USB Serial support registered for generic1610worker1 # [ 0.720410] usbcore: registered new interface driver usbserial_generic1611worker1 # [ 0.721504] usbserial: USB Serial support registered for generic1612lb # [ 0.721885] Linux agpgart interface v0.1031613lb # [ 0.722787] ACPI: bus type drm_connector registered1614worker2 # [ 0.710286] Key type asymmetric registered1615client # [ 0.709362] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1616worker2 # [ 0.711161] Asymmetric key parser 'x509' registered1617client # [ 0.709362] Please enable it if your BIOS has the CPPC option.1618client # [ 0.711745] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1619lb # [ 0.725714] usbcore: registered new interface driver usbserial_generic1620worker1 # [ 0.725090] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1621worker1 # [ 0.725090] Please enable it if your BIOS has the CPPC option.1622worker2 # [ 0.713882] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1623lb # [ 0.726856] usbserial: USB Serial support registered for generic1624worker1 # [ 0.727449] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1625client # [ 0.715570] drop_monitor: Initializing network drop monitor service1626worker2 # [ 0.716868] io scheduler mq-deadline registered1627worker2 # [ 0.717668] io scheduler kyber registered1628client # [ 0.716905] NET: Registered PF_INET6 protocol family1629lb # [ 0.730178] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1630lb # [ 0.730178] Please enable it if your BIOS has the CPPC option.1631lb # [ 0.732493] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1632worker1 # [ 0.732313] drop_monitor: Initializing network drop monitor service1633worker2 # [ 0.721460] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1634worker1 # [ 0.733994] NET: Registered PF_INET6 protocol family1635client # [ 0.721929] Segment Routing with IPv61636lb # [ 0.736334] drop_monitor: Initializing network drop monitor service1637client # [ 0.724377] In-situ OAM (IOAM) with IPv61638lb # [ 0.737514] NET: Registered PF_INET6 protocol family1639worker2 # [ 0.724840] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1640worker1 # [ 0.737796] Segment Routing with IPv61641client # [ 0.725542] IPI shorthand broadcast: enabled1642worker2 # [ 0.728500] Linux agpgart interface v0.1031643worker1 # [ 0.741137] In-situ OAM (IOAM) with IPv61644worker2 # [ 0.729306] ACPI: bus type drm_connector registered1645lb # [ 0.742512] Segment Routing with IPv61646worker1 # [ 0.742620] IPI shorthand broadcast: enabled1647lb # [ 0.743211] In-situ OAM (IOAM) with IPv61648worker2 # [ 0.732374] usbcore: registered new interface driver usbserial_generic1649worker2 # [ 0.733514] usbserial: USB Serial support registered for generic1650lb # [ 0.746497] IPI shorthand broadcast: enabled1651client # [ 0.734287] sched_clock: Marking stable (599018168, 134321438)->(811202944, -77863338)1652worker2 # [ 0.735856] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1653worker2 # [ 0.735856] Please enable it if your BIOS has the CPPC option.1654worker2 # [ 0.738139] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1655client # [ 0.739751] registered taskstats version 11656lb # [ 0.752890] sched_clock: Marking stable (616014685, 136142459)->(836033000, -83875856)1657client # [ 0.740853] Loading compiled-in X.509 certificates1658worker2 # [ 0.741979] drop_monitor: Initializing network drop monitor service1659worker2 # [ 0.743163] NET: Registered PF_INET6 protocol family1660worker1 # [ 0.754615] sched_clock: Marking stable (619025252, 135053815)->(830972665, -76893598)1661lb # [ 0.758368] registered taskstats version 11662lb # [ 0.759310] Loading compiled-in X.509 certificates1663worker2 # [ 0.747393] Segment Routing with IPv61664worker1 # [ 0.760430] registered taskstats version 11665worker1 # [ 0.761757] Loading compiled-in X.509 certificates1666worker2 # [ 0.749862] In-situ OAM (IOAM) with IPv61667worker2 # [ 0.752151] IPI shorthand broadcast: enabled1668worker2 # [ 0.760579] sched_clock: Marking stable (625015127, 134791716)->(839152794, -79345951)1669client # [ 0.760372] Demotion targets for Node 0: null1670client # [ 0.761441] Key type .fscrypt registered1671client # [ 0.762170] Key type fscrypt-provisioning registered1672worker2 # [ 0.764036] registered taskstats version 11673worker2 # [ 0.765051] Loading compiled-in X.509 certificates1674lb # [ 0.778173] Demotion targets for Node 0: null1675client # [ 0.765540] ima: No TPM chip found, activating TPM-bypass!1676lb # [ 0.779115] Key type .fscrypt registered1677lb # [ 0.779807] Key type fscrypt-provisioning registered1678client # [ 0.766549] ima: Allocated hash algorithm: sha11679lb # [ 0.780899] ima: No TPM chip found, activating TPM-bypass!1680client # [ 0.769383] ima: No architecture policies found1681client # [ 0.770375] PM: Magic number: 10:66:3301682client # [ 0.771062] tty tty51: hash matches1683lb # [ 0.784183] ima: Allocated hash algorithm: sha11684lb # [ 0.784967] ima: No architecture policies found1685worker1 # [ 0.785778] Demotion targets for Node 0: null1686client # [ 0.774813] RAS: Correctable Errors collector initialized.1687lb # [ 0.789175] PM: Magic number: 10:66:3301688worker1 # [ 0.789152] Key type .fscrypt registered1689lb # [ 0.789900] tty tty51: hash matches1690worker1 # [ 0.790047] Key type fscrypt-provisioning registered1691lb # [ 0.793497] RAS: Correctable Errors collector initialized.1692worker1 # [ 0.793239] ima: No TPM chip found, activating TPM-bypass!1693worker1 # [ 0.794307] ima: Allocated hash algorithm: sha11694client # [ 0.783685] clk: Disabling unused clocks1695worker2 # [ 0.784834] Demotion targets for Node 0: null1696worker1 # [ 0.797112] ima: No architecture policies found1697worker2 # [ 0.785756] Key type .fscrypt registered1698worker1 # [ 0.798097] PM: Magic number: 10:66:3301699worker1 # [ 0.798785] tty tty51: hash matches1700client # [ 0.786350] PM: genpd: Disabling unused power domains1701worker2 # [ 0.787821] Key type fscrypt-provisioning registered1702worker2 # [ 0.788828] ima: No TPM chip found, activating TPM-bypass!1703lb # [ 0.802266] clk: Disabling unused clocks1704worker2 # [ 0.789735] ima: Allocated hash algorithm: sha11705worker1 # [ 0.802552] RAS: Correctable Errors collector initialized.1706lb # [ 0.802954] PM: genpd: Disabling unused power domains1707worker2 # [ 0.791852] ima: No architecture policies found1708worker2 # [ 0.794818] PM: Magic number: 10:66:3301709worker2 # [ 0.795580] tty tty51: hash matches1710worker1 # [ 0.811473] clk: Disabling unused clocks1711worker2 # [ 0.799215] RAS: Correctable Errors collector initialized.1712worker1 # [ 0.814082] PM: genpd: Disabling unused power domains1713worker2 # [ 0.808079] clk: Disabling unused clocks1714worker2 # [ 0.810820] PM: genpd: Disabling unused power domains1715lb # [ 0.983809] Freeing initrd memory: 29092K1716lb # [ 0.987073] Freeing unused decrypted memory: 2028K1717worker1 # [ 0.989564] Freeing initrd memory: 29092K1718lb # [ 0.989752] Freeing unused kernel image (initmem) memory: 3644K1719client # [ 0.976821] Freeing initrd memory: 29096K1720lb # [ 0.990872] Write protecting the kernel read-only data: 32768k1721worker1 # [ 0.992792] Freeing unused decrypted memory: 2028K1722lb # [ 0.992835] Freeing unused kernel image (text/rodata gap) memory: 1208K1723client # [ 0.980156] Freeing unused decrypted memory: 2028K1724lb # [ 0.994381] Freeing unused kernel image (rodata/data gap) memory: 752K1725worker1 # [ 0.995493] Freeing unused kernel image (initmem) memory: 3644K1726client # [ 0.982915] Freeing unused kernel image (initmem) memory: 3644K1727worker1 # [ 0.996558] Write protecting the kernel read-only data: 32768k1728client # [ 0.984032] Write protecting the kernel read-only data: 32768k1729worker1 # [ 0.998391] Freeing unused kernel image (text/rodata gap) memory: 1208K1730client # [ 0.985966] Freeing unused kernel image (text/rodata gap) memory: 1208K1731worker1 # [ 0.999845] Freeing unused kernel image (rodata/data gap) memory: 752K1732client # [ 0.987544] Freeing unused kernel image (rodata/data gap) memory: 752K1733worker2 # [ 0.997031] Freeing initrd memory: 29092K1734worker2 # [ 1.000350] Freeing unused decrypted memory: 2028K1735worker2 # [ 1.003130] Freeing unused kernel image (initmem) memory: 3644K1736worker2 # [ 1.004291] Write protecting the kernel read-only data: 32768k1737worker2 # [ 1.006277] Freeing unused kernel image (text/rodata gap) memory: 1208K1738worker2 # [ 1.007947] Freeing unused kernel image (rodata/data gap) memory: 752K1739lb # [ 1.045717] x86/mm: Checked W+X mappings: passed, no W+X pages found.1740lb # [ 1.046860] Run /init as init process1741worker1 # [ 1.051237] x86/mm: Checked W+X mappings: passed, no W+X pages found.1742worker1 # [ 1.052324] Run /init as init process1743client # [ 1.039097] x86/mm: Checked W+X mappings: passed, no W+X pages found.1744client # [ 1.040230] Run /init as init process1745lb # [ 1.057566] systemd[1]: Inserted module 'autofs4'1746worker1 # [ 1.062652] systemd[1]: Inserted module 'autofs4'1747client # [ 1.051260] systemd[1]: Inserted module 'autofs4'1748worker2 # [ 1.059907] x86/mm: Checked W+X mappings: passed, no W+X pages found.1749worker2 # [ 1.061118] Run /init as init process1750lb # [ 1.074566] fuse: init (API version 7.45)1751worker1 # [ 1.079552] fuse: init (API version 7.45)1752lb # [ 1.080585] ACPI: \_SB_.GSIG: Enabled at IRQ 221753client # [ 1.069911] fuse: init (API version 7.45)1754lb # [ 1.083645] ACPI: \_SB_.GSIH: Enabled at IRQ 231755worker2 # [ 1.071658] systemd[1]: Inserted module 'autofs4'1756worker1 # [ 1.085471] ACPI: \_SB_.GSIG: Enabled at IRQ 221757worker1 # [ 1.087996] ACPI: \_SB_.GSIH: Enabled at IRQ 231758lb # [ 1.087856] ACPI: \_SB_.GSIE: Enabled at IRQ 201759lb # [ 1.091399] ACPI: \_SB_.GSIF: Enabled at IRQ 211760client # [ 1.078094] ACPI: \_SB_.GSIG: Enabled at IRQ 221761worker1 # [ 1.091358] ACPI: \_SB_.GSIE: Enabled at IRQ 201762worker1 # [ 1.093763] ACPI: \_SB_.GSIF: Enabled at IRQ 211763client # [ 1.081609] ACPI: \_SB_.GSIH: Enabled at IRQ 231764client # [ 1.085658] ACPI: \_SB_.GSIE: Enabled at IRQ 201765worker2 # [ 1.088594] fuse: init (API version 7.45)1766client # [ 1.088175] ACPI: \_SB_.GSIF: Enabled at IRQ 211767worker2 # [ 1.094736] ACPI: \_SB_.GSIG: Enabled at IRQ 221768worker2 # [ 1.097216] ACPI: \_SB_.GSIH: Enabled at IRQ 231769worker2 # [ 1.100343] ACPI: \_SB_.GSIE: Enabled at IRQ 201770worker2 # [ 1.102647] ACPI: \_SB_.GSIF: Enabled at IRQ 211771worker1 # [ 1.119972] systemd[1]: Successfully made /usr/ read-only.1772lb # [ 1.120941] systemd[1]: Successfully made /usr/ read-only.1773client # [ 1.119857] systemd[1]: Successfully made /usr/ read-only.1774worker2 # [ 1.131781] systemd[1]: Successfully made /usr/ read-only.1775worker1 # [ 1.455470] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1776worker1 # [ 1.460233] systemd[1]: Detected virtualization kvm.1777lb # [ 1.456579] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1778worker1 # [ 1.461073] systemd[1]: Detected architecture x86-64.1779lb # [ 1.461399] systemd[1]: Detected virtualization kvm.1780worker1 # [ 1.461919] systemd[1]: Running in initrd.1781lb # [ 1.462248] systemd[1]: Detected architecture x86-64.1782lb # [ 1.463081] systemd[1]: Running in initrd.1783worker1 # [ 1.462989] systemd[1]: Initializing machine ID from random generator.1784worker1 # [ 1.464204] systemd[1]: Hostname set to <worker1>.1785lb # [ 1.464202] systemd[1]: Initializing machine ID from random generator.1786lb # [ 1.465368] systemd[1]: Hostname set to <lb>.1787client # [ 1.455785] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1788client # [ 1.460579] systemd[1]: Detected virtualization kvm.1789client # [ 1.461446] systemd[1]: Detected architecture x86-64.1790client # [ 1.462453] systemd[1]: Running in initrd.1791client # [ 1.463701] systemd[1]: Initializing machine ID from random generator.1792client # [ 1.464963] systemd[1]: Hostname set to <client>.1793worker2 # [ 1.467594] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1794worker2 # [ 1.472353] systemd[1]: Detected virtualization kvm.1795worker2 # [ 1.473206] systemd[1]: Detected architecture x86-64.1796worker2 # [ 1.474120] systemd[1]: Running in initrd.1797worker2 # [ 1.475311] systemd[1]: Initializing machine ID from random generator.1798worker2 # [ 1.476524] systemd[1]: Hostname set to <worker2>.1799worker1 # [ 1.671214] systemd[1]: bpf-restrict-fs: LSM BPF program attached1800lb # [ 1.673617] systemd[1]: bpf-restrict-fs: LSM BPF program attached1801client # [ 1.664523] systemd[1]: bpf-restrict-fs: LSM BPF program attached1802worker2 # [ 1.688287] systemd[1]: bpf-restrict-fs: LSM BPF program attached1803client # [ 1.706908] systemd[1]: Queued start job for default target Initrd Default Target.1804client # [ 1.711838] systemd[1]: Created slice Slice /system/modprobe.1805worker1 # [ 1.725726] systemd[1]: Queued start job for default target Initrd Default Target.1806client # [ 1.713085] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1807lb # [ 1.727108] systemd[1]: Queued start job for default target Initrd Default Target.1808client # [ 1.714503] systemd[1]: Expecting device /dev/disk/by-label/nixos...1809client # [ 1.715628] systemd[1]: Reached target Path Units.1810client # [ 1.716547] systemd[1]: Reached target Slice Units.1811client # [ 1.717485] systemd[1]: Reached target Swaps.1812worker1 # [ 1.730792] systemd[1]: Created slice Slice /system/modprobe.1813client # [ 1.718303] systemd[1]: Reached target Timer Units.1814lb # [ 1.731522] systemd[1]: Created slice Slice /system/modprobe.1815worker1 # [ 1.732139] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1816client # [ 1.719352] systemd[1]: Listening on D-Bus System Message Bus Socket.1817lb # [ 1.732710] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1818worker1 # [ 1.733516] systemd[1]: Expecting device /dev/disk/by-label/nixos...1819client # [ 1.720710] systemd[1]: Listening on Journal Socket (/dev/log).1820lb # [ 1.734225] systemd[1]: Expecting device /dev/disk/by-label/nixos...1821worker1 # [ 1.734560] systemd[1]: Reached target Path Units.1822client # [ 1.721903] systemd[1]: Listening on Journal Sockets.1823lb # [ 1.735307] systemd[1]: Reached target Path Units.1824worker1 # [ 1.735392] systemd[1]: Reached target Slice Units.1825lb # [ 1.736114] systemd[1]: Reached target Slice Units.1826client # [ 1.722918] systemd[1]: Listening on udev Control Socket.1827worker1 # [ 1.736225] systemd[1]: Reached target Swaps.1828lb # [ 1.736989] systemd[1]: Reached target Swaps.1829worker1 # [ 1.737028] systemd[1]: Reached target Timer Units.1830client # [ 1.723938] systemd[1]: Listening on udev Kernel Socket.1831lb # [ 1.737838] systemd[1]: Reached target Timer Units.1832client # [ 1.724899] systemd[1]: Reached target Socket Units.1833worker1 # [ 1.738001] systemd[1]: Listening on D-Bus System Message Bus Socket.1834lb # [ 1.738819] systemd[1]: Listening on D-Bus System Message Bus Socket.1835worker1 # [ 1.739244] systemd[1]: Listening on Journal Socket (/dev/log).1836lb # [ 1.740127] systemd[1]: Listening on Journal Socket (/dev/log).1837worker1 # [ 1.740387] systemd[1]: Listening on Journal Sockets.1838client # [ 1.727182] systemd[1]: Starting Create List of Static Device Nodes...1839lb # [ 1.741299] systemd[1]: Listening on Journal Sockets.1840worker1 # [ 1.741382] systemd[1]: Listening on udev Control Socket.1841lb # [ 1.742355] systemd[1]: Listening on udev Control Socket.1842worker1 # [ 1.742397] systemd[1]: Listening on udev Kernel Socket.1843lb # [ 1.743406] systemd[1]: Listening on udev Kernel Socket.1844worker1 # [ 1.743378] systemd[1]: Reached target Socket Units.1845lb # [ 1.744383] systemd[1]: Reached target Socket Units.1846client # [ 1.732422] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1847worker1 # [ 1.745582] systemd[1]: Starting Create List of Static Device Nodes...1848lb # [ 1.746300] systemd[1]: Starting Create List of Static Device Nodes...1849lb # [ 1.749981] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1850worker1 # [ 1.751163] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1851client # [ 1.741485] systemd[1]: Starting Load Kernel Module configfs...1852lb # [ 1.756885] systemd[1]: Starting Load Kernel Module configfs...1853worker2 # [ 1.744515] systemd[1]: Queued start job for default target Initrd Default Target.1854worker1 # [ 1.758151] systemd[1]: Starting Load Kernel Module configfs...1855worker2 # [ 1.749247] systemd[1]: Created slice Slice /system/modprobe.1856worker2 # [ 1.750473] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1857worker2 # [ 1.751991] systemd[1]: Expecting device /dev/disk/by-label/nixos...1858worker2 # [ 1.753143] systemd[1]: Reached target Path Units.1859worker2 # [ 1.754036] systemd[1]: Reached target Slice Units.1860worker2 # [ 1.754940] systemd[1]: Reached target Swaps.1861worker2 # [ 1.755760] systemd[1]: Reached target Timer Units.1862worker2 # [ 1.756754] systemd[1]: Listening on D-Bus System Message Bus Socket.1863client # [ 1.757111] systemd[1]: Starting Journal Service...1864worker2 # [ 1.758017] systemd[1]: Listening on Journal Socket (/dev/log).1865lb # [ 1.771349] systemd[1]: Starting Journal Service...1866worker2 # [ 1.759242] systemd[1]: Listening on Journal Sockets.1867worker2 # [ 1.760243] systemd[1]: Listening on udev Control Socket.1868worker2 # [ 1.761312] systemd[1]: Listening on udev Kernel Socket.1869worker1 # [ 1.774468] systemd[1]: Starting Journal Service...1870worker2 # [ 1.762303] systemd[1]: Reached target Socket Units.1871worker2 # [ 1.764186] systemd[1]: Starting Create List of Static Device Nodes...1872worker2 # [ 1.768879] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1873worker2 # [ 1.774007] systemd[1]: Starting Load Kernel Module configfs...1874client # [ 1.767894] systemd[1]: Starting Load Kernel Modules...1875lb # [ 1.785241] systemd[1]: Starting Load Kernel Modules...1876worker1 # [ 1.788725] systemd[1]: Starting Load Kernel Modules...1877worker2 # [ 1.789453] systemd[1]: Starting Journal Service...1878lb # [ 1.805048] netfs: FS-Cache loaded1879client # [ 1.793562] netfs: FS-Cache loaded1880lb # [ 1.809920] 9pnet: Installing 9P2000 support1881worker1 # [ 1.810753] netfs: FS-Cache loaded1882client # [ 1.799126] 9pnet: Installing 9P2000 support1883worker1 # [ 1.817804] 9pnet: Installing 9P2000 support1884lb # [ 1.817782] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1885worker1 # [ 1.823412] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1886worker2 # [ 1.800961] systemd[1]: Starting Load Kernel Modules...1887client # [ 1.813220] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1888lb # [ 1.832327] systemd[1]: Starting Coldplug All udev Devices...1889worker1 # [ 1.834331] systemd[1]: Starting Coldplug All udev Devices...1890worker2 # [ 1.823654] netfs: FS-Cache loaded1891client # [ 1.822592] systemd[1]: Starting Coldplug All udev Devices...1892worker2 # [ 1.830487] 9pnet: Installing 9P2000 support1893worker2 # [ 1.832756] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1894worker1 # [ 1.850156] systemd[1]: Finished Create List of Static Device Nodes.1895lb # [ 1.852393] systemd[1]: Finished Create List of Static Device Nodes.1896client # [ 1.840613] systemd[1]: Finished Create List of Static Device Nodes.1897worker1 # [ 1.854990] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1898worker2 # [ 1.843149] systemd[1]: Starting Coldplug All udev Devices...1899client # [ 1.846098] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1900worker1 # [ 1.863558] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1901lb # [ 1.863932] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1902client # [ 1.852340] systemd-journald[67]: Collecting audit messages is disabled.1903worker1 # [ 1.867547] systemd-journald[67]: Collecting audit messages is disabled.1904worker1 # [ 1.869404] systemd[1]: modprobe@configfs.service: Deactivated successfully.1905client # [ 1.857771] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1906lb # [ 1.872907] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1907client # [ 1.860862] systemd[1]: modprobe@configfs.service: Deactivated successfully.1908worker2 # [ 1.861891] systemd[1]: Finished Create List of Static Device Nodes.1909worker1 # [ 1.877504] systemd[1]: Finished Load Kernel Module configfs.1910lb # [ 1.879559] systemd-journald[67]: Collecting audit messages is disabled.1911client # [ 1.866208] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1912worker1 # [ 1.879457] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1913lb # [ 1.881619] systemd[1]: modprobe@configfs.service: Deactivated successfully.1914worker2 # [ 1.869221] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1915client # [ 1.870723] systemd[1]: Finished Load Kernel Module configfs.1916worker1 # [ 1.884867] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1917client # [ 1.874904] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1918worker2 # [ 1.877245] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1919lb # [ 1.890876] systemd[1]: Finished Load Kernel Module configfs.1920worker1 # [ 1.891113] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1921client # [ 1.879509] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1922worker2 # [ 1.882294] systemd[1]: modprobe@configfs.service: Deactivated successfully.1923lb # [ 1.893953] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1924worker1 # [ 1.896344] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1925lb # [ 1.898098] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1926worker2 # [ 1.887133] systemd-journald[67]: Collecting audit messages is disabled.1927client # [ 1.886558] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1928worker2 # [ 1.890227] systemd[1]: Finished Load Kernel Module configfs.1929worker2 # [ 1.894255] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1930lb # [ 1.908230] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1931worker2 # [ 1.898387] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1932lb # [ 1.914303] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1933worker2 # [ 1.903859] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1934worker2 # [ 1.907887] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1935client # [ 1.910978] systemd[1]: Finished Load Kernel Modules.1936worker1 # [ 1.928750] systemd[1]: Finished Load Kernel Modules.1937client # [ 1.919552] systemd[1]: Starting Apply Kernel Variables...1938worker1 # [ 1.940579] systemd[1]: Starting Apply Kernel Variables...1939client # [ 1.935469] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1940worker2 # [ 1.939413] systemd[1]: Finished Load Kernel Modules.1941lb # [ 1.956420] systemd[1]: Finished Load Kernel Modules.1942worker2 # [ 1.948036] systemd[1]: Starting Apply Kernel Variables...1943client # [ 1.947435] systemd[1]: Starting Create Static Device Nodes in /dev...1944worker1 # [ 1.966588] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1945lb # [ 1.967284] systemd[1]: Starting Apply Kernel Variables...1946worker2 # [ 1.963943] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1947worker1 # [ 1.980196] systemd[1]: Starting Create Static Device Nodes in /dev...1948client # [ 1.972491] systemd[1]: Finished Apply Kernel Variables.1949worker2 # [ 1.976207] systemd[1]: Starting Create Static Device Nodes in /dev...1950lb # [ 1.998400] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1951client # [ 1.990170] systemd[1]: Started Journal Service.1952worker2 # [ 1.992093] systemd[1]: Finished Apply Kernel Variables.1953client # [ 1.854856] systemd-modules-load[68]: Inserted module 'dm_mod'1954client # [ 1.858992] systemd-modules-load[68]: Inserted module 'virtio_balloon'1955worker1 # [ 2.006262] systemd[1]: Finished Apply Kernel Variables.1956client # [ 1.860619] systemd-modules-load[68]: Inserted module 'virtio_gpu'1957client # [ 1.863988] systemd[1]: Finished Create Static Device Nodes in /dev.1958lb # [ 2.013616] systemd[1]: Starting Create Static Device Nodes in /dev...1959worker1 # [ 2.017510] systemd[1]: Started Journal Service.1960worker1 # [ 1.879949] systemd-modules-load[68]: Inserted module 'dm_mod'1961worker1 # [ 1.885312] systemd-modules-load[68]: Inserted module 'virtio_balloon'1962client # [ 1.873170] systemd[1]: Reached target Preparation for Local File Systems.1963client # [ 1.874269] systemd[1]: Reached target Local File Systems.1964worker1 # [ 1.888440] systemd-modules-load[68]: Inserted module 'virtio_gpu'1965client # [ 1.878070] systemd[1]: Starting Create System Files and Directories...1966client # [ 1.882652] systemd[1]: Starting Rule-based Manager for Device Events and Files...1967worker2 # [ 1.882597] systemd-modules-load[68]: Inserted module 'dm_mod'[ 2.018845] systemd[1]: Started Journal Service.1968worker2 # 1969worker2 # [ 1.885383] systemd-modules-load[68]: Inserted module 'virtio_balloon'1970worker2 # [ 1.888395] systemd-modules-load[68]: Inserted module 'virtio_gpu'1971worker1 # [ 1.904194] systemd[1]: Finished Create Static Device Nodes in /dev.1972worker1 # [ 1.907075] systemd[1]: Reached target Preparation for Local File Systems.1973lb # [ 2.042323] systemd[1]: Finished Apply Kernel Variables.1974worker1 # [ 1.908283] systemd[1]: Reached target Local File Systems.1975worker2 # [ 1.900156] systemd[1]: Finished Create Static Device Nodes in /dev.1976lb # [ 1.908973] systemd-modules-load[68]: Inserted module 'dm_mod'1977worker1 # [ 1.913141] systemd[1]: Starting Create System Files and Directories...1978lb # [ 1.913155] systemd-modules-load[68]: Inserted module 'virtio_balloon'1979lb # [ 2.051975] systemd[1]: Started Journal Service.1980worker1 # [ 1.917722] systemd[1]: Starting Rule-based Manager for Device Events and Files...1981lb # [ 1.917110] systemd-modules-load[68]: Inserted module 'virtio_gpu'1982worker2 # [ 1.908181] systemd[1]: Reached target Preparation for Local File Systems.1983worker2 # [ 1.909467] systemd[1]: Reached target Local File Systems.1984worker2 # [ 1.913071] systemd[1]: Starting Create System Files and Directories...1985client # [ 1.916398] systemd[1]: Finished Create System Files and Directories.1986worker2 # [ 1.916512] systemd[1]: Starting Rule-based Manager for Device Events and Files...1987client # [ 1.936409] systemd-udevd[82]: Using default interface naming scheme 'v261'.1988lb # [ 1.950101] systemd[1]: Finished Create Static Device Nodes in /dev.1989lb # [ 1.951562] systemd[1]: Reached target Preparation for Local File Systems.1990lb # [ 1.955080] systemd[1]: Reached target Local File Systems.1991lb # [ 1.960130] systemd[1]: Starting Create System Files and Directories...1992worker1 # [ 1.963601] systemd[1]: Finished Create System Files and Directories.1993worker2 # [ 1.954190] systemd[1]: Finished Create System Files and Directories.1994lb # [ 1.966379] systemd[1]: Starting Rule-based Manager for Device Events and Files...1995client # [ 1.962336] systemd[1]: Started Rule-based Manager for Device Events and Files.1996worker1 # [ 1.986462] systemd-udevd[83]: Using default interface naming scheme 'v261'.1997worker2 # [ 1.980329] systemd-udevd[83]: Using default interface naming scheme 'v261'.1998worker1 # [ 2.012216] systemd[1]: Started Rule-based Manager for Device Events and Files.1999lb # [ 2.011458] systemd[1]: Finished Create System Files and Directories.2000client # [ 2.004494] systemd[1]: Finished Coldplug All udev Devices.
loading 1,903 hidden lines…
3904worker1: waiting for the VM to finish booting3905worker1: Guest shell says: b'Spawning backdoor root shell...\n'3906worker1: connected to guest root shell3907worker1: (connecting took 0.00 seconds)3908worker1: (finished: waiting for the VM to finish booting, in 0.00 seconds)3909lb # [ 12.590105] postgresql-pre-start[799]: selecting default "shared_buffers" ... 128MB3910worker1: (finished: waiting for unit network-addresses-eth1.service, in 0.07 seconds)3911worker1: must succeed: ip -4 addr show | grep -qF 192.168.1.3/ || { ip -4 addr >&2; false; }3912lb # [ 12.617623] dhcpcd[784]: eth0: carrier acquired3913worker1: (finished: must succeed: ip -4 addr show | grep -qF 192.168.1.3/ || { ip -4 addr >&2; false; }, in 0.03 seconds)3914worker2: waiting for unit network-addresses-eth1.service3915worker2: waiting for the VM to finish booting3916worker2: Guest shell says: b'Spawning backdoor root shell...\n'3917worker2: connected to guest root shell3918worker2: (connecting took 0.00 seconds)3919worker2: (finished: waiting for the VM to finish booting, in 0.00 seconds)3920lb # [ 12.629100] dhcpcd[784]: DUID 00:01:00:01:32:42:8c:0c:52:54:00:12:34:563921lb # [ 12.630072] dhcpcd[784]: eth0: IAID 00:12:34:563922lb # [ 12.630748] dhcpcd[784]: eth0: adding address fe80::5054:ff:fe12:34563923worker2: (finished: waiting for unit network-addresses-eth1.service, in 0.07 seconds)3924worker2: must succeed: ip -4 addr show | grep -qF 192.168.1.4/ || { ip -4 addr >&2; false; }3925worker2: (finished: must succeed: ip -4 addr show | grep -qF 192.168.1.4/ || { ip -4 addr >&2; false; }, in 0.03 seconds)3926lb: waiting for unit niks3.service3927client # [ 12.899574] dhcpcd[635]: eth0: soliciting an IPv6 router3928client # [ 12.902094] dhcpcd[635]: eth0: Router Advertisement from fe80::23929client # [ 12.904127] dhcpcd[635]: eth0: adding address fec0::5054:ff:fe12:3456/643930client # [ 12.906178] dhcpcd[635]: eth0: adding route to fec0::/643931client # [ 12.907806] dhcpcd[635]: eth0: adding default route via fe80::23932lb # [ 12.957156] rustfs-bucket-start[852]: mb s3://farm3933lb # [ 12.963126] systemd[1]: Finished rustfs-bucket.service.3934worker2 # [ 12.972447] dhcpcd[747]: eth0: soliciting an IPv6 router3935worker2 # [ 12.975444] dhcpcd[747]: eth0: Router Advertisement from fe80::23936worker2 # [ 12.977751] dhcpcd[747]: eth0: adding address fec0::5054:ff:fe12:3456/643937worker2 # [ 12.980087] dhcpcd[747]: eth0: adding route to fec0::/643938worker2 # [ 12.981955] dhcpcd[747]: eth0: adding default route via fe80::23939lb # [ 14.437227] dhcpcd[784]: eth0: soliciting a DHCP lease3940lb # [ 14.445755] postgresql-pre-start[799]: selecting default time zone ... UTC3941lb # [ 14.450296] postgresql-pre-start[799]: creating configuration files ... ok3942lb # [ 14.595389] NET: Registered PF_PACKET protocol family3943lb # [ 14.464637] dhcpcd[784]: eth0: offered 10.0.2.15 from 10.0.2.23944lb # [ 14.468285] dhcpcd[784]: eth0: probing address 10.0.2.15/243945lb # [ 14.731366] postgresql-pre-start[799]: running bootstrap script ... ok3946lb # [ 15.264306] dhcpcd[784]: eth0: soliciting an IPv6 router3947lb # [ 15.265659] dhcpcd[784]: eth0: Router Advertisement from fe80::23948lb # [ 15.266698] dhcpcd[784]: eth0: adding address fec0::5054:ff:fe12:3456/643949lb # [ 15.268134] dhcpcd[784]: eth0: adding route to fec0::/643950lb # [ 15.269392] dhcpcd[784]: eth0: adding default route via fe80::23951worker2 # [ 15.327623] dhcpcd[747]: eth0: leased 10.0.2.15 for 86400 seconds3952worker2 # [ 15.329346] dhcpcd[747]: eth0: adding route to 10.0.2.0/243953worker2 # [ 15.330707] dhcpcd[747]: eth0: adding default route via 10.0.2.23954lb # [ 15.428104] postgresql-pre-start[799]: performing post-bootstrap initialization ... ok3955worker2 # [ 15.444825] systemd[1]: Started DHCP Client.3956worker2 # [ 15.447182] systemd[1]: Reached target Multi-User System.3957worker2 # [ 15.447971] systemd[1]: Startup finished in 929ms (kernel) + 4.185s (initrd) + 10.331s (userspace) = 15.447s.3958lb # [ 15.589922] postgresql-pre-start[799]: syncing data to disk ... ok3959lb # [ 15.590874] postgresql-pre-start[799]: initdb: warning: enabling "trust" authentication for local connections3960lb # [ 15.592246] postgresql-pre-start[799]: initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.3961lb # [ 15.594392] postgresql-pre-start[799]: Success. You can now start the database server using:3962lb # [ 15.595686] postgresql-pre-start[799]: pg_ctl -D /var/lib/postgresql/18 -l logfile start3963lb # [ 15.741934] postgres[886]: [886] LOG: starting PostgreSQL 18.6 on x86_64-pc-linux-gnu, compiled by clang version 21.1.8, 64-bit3964lb # [ 15.746171] postgres[886]: [886] LOG: listening on IPv6 address "::1", port 54323965lb # [ 15.747223] postgres[886]: [886] LOG: listening on IPv4 address "127.0.0.1", port 54323966lb # [ 15.751912] postgres[886]: [886] LOG: listening on Unix socket "/run/postgresql/.s.PGSQL.5432"3967lb # [ 15.764384] postgres[895]: [895] LOG: database system was shut down at 2026-09-20 12:20:31 GMT3968lb # [ 15.769923] postgres[886]: [886] LOG: database system is ready to accept connections3969lb # [ 15.775866] systemd[1]: Started PostgreSQL Server.3970lb # [ 15.783069] systemd[1]: Starting niks3 server...3971lb # [ 15.787100] systemd[1]: Starting PostgreSQL Setup Scripts...3972client # [ 15.920892] dhcpcd[635]: eth0: leased 10.0.2.15 for 86400 seconds3973client # [ 15.921877] dhcpcd[635]: eth0: adding route to 10.0.2.0/243974client # [ 15.922645] dhcpcd[635]: eth0: adding default route via 10.0.2.23975client # [ 16.003725] systemd[1]: Started DHCP Client.3976client # [ 16.005881] systemd[1]: Reached target Multi-User System.3977client # [ 16.006909] systemd[1]: Startup finished in 909ms (kernel) + 4.081s (initrd) + 11.015s (userspace) = 16.006s.3978lb # [ 16.049416] postgresql-setup-start[909]: CREATE DATABASE3979lb # [ 16.078868] postgres[921]: [921] FATAL: role "niks3" does not exist3980lb # [ 16.088134] postgres[922]: [922] FATAL: role "niks3" does not exist3981lb # [ 16.093394] niks3-server[899]: 2026/09/20 12:20:32 ERROR Failed to run server error="failed to connect to database: failed to migrate db: failed to connect to `user=niks3 database=niks3`: /var/run/postgresql/.s.PGSQL.5432 (/var/run/postgresql): server error: FATAL: role \"niks3\" does not exist (SQLSTATE 28000); failed to connect to `user=niks3 database=niks3`: /var/run/postgresql/.s.PGSQL.5432 (/var/run/postgresql): server error: FATAL: role \"niks3\" does not exist (SQLSTATE 28000)"3982lb # [ 16.100345] systemd[1]: niks3.service: Main process exited, code=exited, status=1/FAILURE3983lb # [ 16.102119] systemd[1]: niks3.service: Failed with result 'exit-code'.3984lb # [ 16.104373] systemd[1]: Failed to start niks3 server.3985lb # [ 16.122404] postgresql-setup-start[923]: CREATE ROLE3986lb # [ 16.146162] postgresql-setup-start[928]: ALTER DATABASE3987lb # [ 16.152558] systemd[1]: Finished PostgreSQL Setup Scripts.3988lb # [ 16.154601] systemd[1]: Reached target PostgreSQL.3989worker1 # [ 17.300719] dhcpcd[694]: eth0: leased 10.0.2.15 for 86400 seconds3990worker1 # [ 17.302261] dhcpcd[694]: eth0: adding route to 10.0.2.0/243991worker1 # [ 17.303595] dhcpcd[694]: eth0: adding default route via 10.0.2.23992worker1 # [ 17.465490] systemd[1]: Started DHCP Client.3993worker1 # [ 17.468501] systemd[1]: Reached target Multi-User System.3994worker1 # [ 17.469914] systemd[1]: Startup finished in 920ms (kernel) + 4.970s (initrd) + 11.577s (userspace) = 17.468s.3995lb # [ 20.210726] dhcpcd[784]: eth0: leased 10.0.2.15 for 86400 seconds3996lb # [ 20.213712] dhcpcd[784]: eth0: adding route to 10.0.2.0/243997lb # [ 20.216378] dhcpcd[784]: eth0: adding default route via 10.0.2.23998lb # [ 20.360608] systemd[1]: Started DHCP Client.3999lb # [ 20.363411] systemd[1]: Reached target Network is Online.4000lb # [ 20.368663] systemd[1]: Started Envoy reverse proxy.4001lb # [ 20.373513] systemd[1]: Started mock-oidc.service.4002lb # [ 20.376821] systemd[1]: Reached target Multi-User System.4003lb # [ 20.378452] systemd[1]: Startup finished in 914ms (kernel) + 5.498s (initrd) + 13.965s (userspace) = 20.378s.4004lb # [ 20.482341] mock-oidc-server[1009]: Mock OIDC Server running4005lb # [ 20.483797] mock-oidc-server[1009]: OIDC Address: 192.168.1.2:80804006lb # [ 20.485132] mock-oidc-server[1009]: Issue Address: 0.0.0.0:80814007lb # [ 20.486170] mock-oidc-server[1009]: Issuer: http://192.168.1.2:8080/oidc4008lb # [ 20.487297] mock-oidc-server[1009]: JWKS: http://192.168.1.2:8080/oidc/.well-known/jwks.json4009lb # [ 20.488658] mock-oidc-server[1009]: Discovery: http://192.168.1.2:8080/oidc/.well-known/openid-configuration4010lb # [ 20.490390] mock-oidc-server[1009]: Issue tokens: http://0.0.0.0:8081/issue?sub=...4011lb # [ 20.924950] envoy[1008]: [2026-09-20 12:20:37.090][1008][info][main] [source/server/server.cc:467] initializing epoch 0 (base id=0, hot restart version=11.104)4012lb # [ 20.927129] envoy[1008]: [2026-09-20 12:20:37.090][1008][info][main] [source/server/server.cc:469] statically linked extensions:4013lb # [ 20.928963] envoy[1008]: [2026-09-20 12:20:37.090][1008][info][main] [source/server/server.cc:471] envoy.upstream_options: envoy.extensions.upstreams.http.reverse_tunnel.v3.ReverseTunnelUpstreamCodecOptions, envoy.extensions.upstreams.http.v3.HttpProtocolOptions, envoy.extensions.upstreams.tcp.v3.TcpProtocolOptions, envoy.upstreams.http.http_protocol_options, envoy.upstreams.tcp.tcp_protocol_options4014lb # [ 20.933349] envoy[1008]: [2026-09-20 12:20:37.090][1008][info][main] [source/server/server.cc:471] envoy.dubbo_proxy.filters: envoy.filters.dubbo.router4015lb # [ 20.935177] envoy[1008]: [2026-09-20 12:20:37.090][1008][info][main] [source/server/server.cc:471] envoy.listener_manager_impl: envoy.listener_manager_impl.default, envoy.listener_manager_impl.validation4016lb # [ 20.937658] envoy[1008]: [2026-09-20 12:20:37.090][1008][info][main] [source/server/server.cc:471] envoy.thrift_proxy.transports: auto, framed, header, unframed4017lb # [ 20.939773] envoy[1008]: [2026-09-20 12:20:37.090][1008][info][main] [source/server/server.cc:471] network.connection.client: default, envoy_internal4018lb # [ 20.942546] envoy[1008]: [2026-09-20 12:20:37.090][1008][info][main] [source/server/server.cc:471] envoy.regex_engines: envoy.regex_engines.google_re24019lb # [ 20.944230] envoy[1008]: [2026-09-20 12:20:37.090][1008][info][main] [source/server/server.cc:471] envoy.bootstrap: envoy.bootstrap.dynamic_modules, envoy.bootstrap.internal_listener, envoy.bootstrap.reverse_tunnel.downstream_socket_interface, envoy.bootstrap.reverse_tunnel.upstream_socket_interface, envoy.bootstrap.wasm, envoy.extensions.network.socket_interface.default_socket_interface, envoy.extensions.network.socket_interface.sockmap4020lb # [ 20.948881] envoy[1008]: [2026-09-20 12:20:37.090][1008][info][main] [source/server/server.cc:471] envoy.filters.http.upstream: envoy.buffer, envoy.ext_proc, envoy.extensions.filters.http.dynamic_modules, envoy.filters.http.admission_control, envoy.filters.http.aws_lambda, envoy.filters.http.aws_request_signing, envoy.filters.http.buffer, envoy.filters.http.composite, envoy.filters.http.credential_injector, envoy.filters.http.ext_proc, envoy.filters.http.header_mutation, envoy.filters.http.lua, envoy.filters.http.match_delegate, envoy.filters.http.upstream_codec, envoy.filters.http.upstream_rbac, envoy.filters.http.wasm4021lb # [ 20.963250] envoy[1008]: [2026-09-20 12:20:37.090][1008][info][main] [source/server/server.cc:471] envoy.rate_limit_descriptors: envoy.rate_limit_descriptors.expr4022lb # [ 20.965942] envoy[1008]: [2026-09-20 12:20:37.090][1008][info][main] [source/server/server.cc:471] envoy.request_id: envoy.request_id.uuid4023lb # [ 20.967602] envoy[1008]: [2026-09-20 12:20:37.090][1008][info][main] [source/server/server.cc:471] envoy.route.early_data_policy: envoy.route.early_data_policy.default4024lb # [ 20.969567] envoy[1008]: [2026-09-20 12:20:37.090][1008][info][main] [source/server/server.cc:471] envoy.quic.connection_id_generator: envoy.quic.connection_id_generator.quic_lb, envoy.quic.deterministic_connection_id_generator4025lb # [ 20.972193] envoy[1008]: [2026-09-20 12:20:37.090][1008][info][main] [source/server/server.cc:471] envoy.ssl.server_context_factory: envoy.ssl.server_context_factory.default4026lb # [ 20.975251] envoy[1008]: [2026-09-20 12:20:37.090][1008][info][main] [source/server/server.cc:471] filter_state.object: envoy.filters.listener.original_dst.local_ip, envoy.filters.listener.original_dst.remote_ip, envoy.hashable_string, envoy.network.application_protocols, envoy.network.ip, envoy.network.network_namespace, envoy.network.transport_socket.http_11_proxy.info, envoy.network.transport_socket.original_dst_address, envoy.network.upstream_server_name, envoy.network.upstream_subject_alt_names, envoy.ratelimit.hits_addend, envoy.router.debug_config, envoy.string, envoy.tcp_proxy.cluster, envoy.tcp_proxy.disable_tunneling, envoy.tcp_proxy.per_connection_idle_timeout_ms, envoy.upstream.dynamic_host, envoy.upstream.dynamic_port4027lb # [ 20.983405] envoy[1008]: [2026-09-20 12:20:37.090][1008][info][main] [source/server/server.cc:471] envoy.wasm.runtime: envoy.wasm.runtime.null, envoy.wasm.runtime.v84028lb # [ 20.985446] envoy[1008]: [2026-09-20 12:20:37.090][1008][info][main] [source/server/server.cc:471] envoy.matching.stat_tag_matching_data.input: stat_tag_value_input4029lb # [ 20.987480] envoy[1008]: [2026-09-20 12:20:37.090][1008][info][main] [source/server/server.cc:471] envoy.health_checkers: envoy.health_checkers.dynamic_modules, envoy.health_checkers.grpc, envoy.health_checkers.http, envoy.health_checkers.redis, envoy.health_checkers.tcp, envoy.health_checkers.thrift4030lb # [ 20.991131] envoy[1008]: [2026-09-20 12:20:37.090][1008][info][main] [source/server/server.cc:471] envoy.health_check.event_sinks: envoy.health_check.event_sink.file4031lb # [ 20.993280] envoy[1008]: [2026-09-20 12:20:37.090][1008][info][main] [source/server/server.cc:471] envoy.http.stateful_header_formatters: envoy.http.stateful_header_formatters.preserve_case, preserve_case4032lb # [ 20.995486] envoy[1008]: [2026-09-20 12:20:37.090][1008][info][main] [source/server/server.cc:471] envoy.geoip_providers: envoy.geoip_providers.maxmind4033lb # [ 20.997169] envoy[1008]: [2026-09-20 12:20:37.093][1008][info][main] [source/server/server.cc:471] envoy.dubbo_proxy.protocols: dubbo4034lb # [ 20.999209] envoy[1008]: [2026-09-20 12:20:37.093][1008][info][main] [source/server/server.cc:471] envoy.config.validators: envoy.config.validators.minimum_clusters, envoy.config.validators.minimum_clusters_validator4035lb # [ 21.001690] envoy[1008]: [2026-09-20 12:20:37.094][1008][info][main] [source/server/server.cc:471] envoy.upstream.local_address_selector: envoy.upstream.local_address_selector.default_local_address_selector, envoy.upstream.local_address_selector.filter_state_override4036lb # [ 21.004637] envoy[1008]: [2026-09-20 12:20:37.094][1008][info][main] [source/server/server.cc:471] envoy.tls.cert_validator: envoy.tls.cert_validator.default, envoy.tls.cert_validator.dynamic_modules, envoy.tls.cert_validator.spiffe4037lb # [ 21.007171] envoy[1008]: [2026-09-20 12:20:37.094][1008][info][main] [source/server/server.cc:471] envoy.built_in_formatters: envoy.built_in_formatters.cel, envoy.built_in_formatters.http.default, envoy.built_in_formatters.metadata, envoy.built_in_formatters.stream_info.default, envoy.built_in_formatters.xfcc_value4038lb # [ 21.010632] envoy[1008]: [2026-09-20 12:20:37.094][1008][info][main] [source/server/server.cc:471] envoy.matching.http.custom_matchers: envoy.matching.custom_matchers.domain_matcher, envoy.matching.custom_matchers.ip_range_matcher4039lb # [ 21.013165] envoy[1008]: [2026-09-20 12:20:37.094][1008][info][main] [source/server/server.cc:471] envoy.http.cache: envoy.extensions.http.cache.file_system_http_cache, envoy.extensions.http.cache.simple4040lb # [ 21.016197] envoy[1008]: [2026-09-20 12:20:37.094][1008][info][main] [source/server/server.cc:471] envoy.clusters: envoy.cluster.dns, envoy.cluster.eds, envoy.cluster.logical_dns, envoy.cluster.original_dst, envoy.cluster.static, envoy.cluster.strict_dns, envoy.clusters.aggregate, envoy.clusters.composite, envoy.clusters.dynamic_forward_proxy, envoy.clusters.dynamic_modules, envoy.clusters.mcp_multicluster, envoy.clusters.redis, envoy.clusters.reverse_connection4041lb # [ 21.021451] envoy[1008]: [2026-09-20 12:20:37.094][1008][info][main] [source/server/server.cc:471] envoy.http.cache_v2: envoy.extensions.http.cache_v2.file_system_http_cache, envoy.extensions.http.cache_v2.simple4042lb # [ 21.023825] envoy[1008]: [2026-09-20 12:20:37.094][1008][info][main] [source/server/server.cc:471] envoy.http.injected_credentials: envoy.http.injected_credentials.generic, envoy.http.injected_credentials.oauth24043lb # [ 21.026586] envoy[1008]: [2026-09-20 12:20:37.094][1008][info][main] [source/server/server.cc:471] envoy.http.ext_proc.response_processors: envoy.extensions.http.ext_proc.save_processing_response4044lb # [ 21.029120] envoy[1008]: [2026-09-20 12:20:37.094][1008][info][main] [source/server/server.cc:471] envoy.rbac.principals: envoy.rbac.principals.mtls_authenticated4045lb # [ 21.030970] envoy[1008]: [2026-09-20 12:20:37.095][1008][info][main] [source/server/server.cc:471] envoy.matching.input_matchers: envoy.matching.input_matchers.generic_request_matcher, envoy.matching.matchers.cel_matcher, envoy.matching.matchers.consistent_hashing, envoy.matching.matchers.dynamic_modules, envoy.matching.matchers.ip, envoy.matching.matchers.metadata_matcher, envoy.matching.matchers.runtime_fraction4046lb # [ 21.036610] envoy[1008]: [2026-09-20 12:20:37.095][1008][info][main] [source/server/server.cc:471] envoy.matching.stat_matching_data.input: stat_full_name_match_input4047lb # [ 21.038549] envoy[1008]: [2026-09-20 12:20:37.095][1008][info][main] [source/server/server.cc:471] envoy.filters.udp_listener: envoy.filters.udp.dns_filter, envoy.filters.udp_listener.dynamic_modules, envoy.filters.udp_listener.udp_proxy4048lb # [ 21.041070] envoy[1008]: [2026-09-20 12:20:37.095][1008][info][main] [source/server/server.cc:471] envoy.generic_proxy.filters: envoy.filters.generic.router4049lb # [ 21.042849] envoy[1008]: [2026-09-20 12:20:37.095][1008][info][main] [source/server/server.cc:471] envoy.compression.decompressor: envoy.compression.brotli.decompressor, envoy.compression.gzip.decompressor, envoy.compression.zstd.decompressor4050lb # [ 21.045408] envoy[1008]: [2026-09-20 12:20:37.095][1008][info][main] [source/server/server.cc:471] envoy.resource_monitors: envoy.resource_monitors.cgroup_memory, envoy.resource_monitors.cpu_utilization, envoy.resource_monitors.fixed_heap, envoy.resource_monitors.injected_resource4051lb # [ 21.048712] envoy[1008]: [2026-09-20 12:20:37.095][1008][info][main] [source/server/server.cc:471] envoy.http.original_ip_detection: envoy.http.original_ip_detection.custom_header, envoy.http.original_ip_detection.xff4052lb # [ 21.051179] envoy[1008]: [2026-09-20 12:20:37.095][1008][info][main] [source/server/server.cc:471] envoy.content_parsers: envoy.content_parsers.json4053lb # [ 21.052881] envoy[1008]: [2026-09-20 12:20:37.095][1008][info][main] [source/server/server.cc:471] envoy.router.cluster_specifier_plugin: envoy.router.cluster_specifier_plugin.lua, envoy.router.cluster_specifier_plugin.matcher4054lb # [ 21.055469] envoy[1008]: [2026-09-20 12:20:37.095][1008][info][main] [source/server/server.cc:471] envoy.matching.http.input: envoy.matching.inputs.cel_data_input, envoy.matching.inputs.destination_ip, envoy.matching.inputs.destination_port, envoy.matching.inputs.direct_source_ip, envoy.matching.inputs.dns_san, envoy.matching.inputs.dynamic_metadata, envoy.matching.inputs.dynamic_module_data_input, envoy.matching.inputs.filter_state, envoy.matching.inputs.local_reply, envoy.matching.inputs.network_namespace, envoy.matching.inputs.request_headers, envoy.matching.inputs.request_trailers, envoy.matching.inputs.response_headers, envoy.matching.inputs.response_trailers, envoy.matching.inputs.server_name, envoy.matching.inputs.source_ip, envoy.matching.inputs.source_port, envoy.matching.inputs.source_type, envoy.matching.inputs.status_code_class_input, envoy.matching.inputs.status_code_input, envoy.matching.inputs.subject, envoy.matching.inputs.uri_san, query_params4055lb # [ 21.065517] envoy[1008]: [2026-09-20 12:20:37.095][1008][info][main] [source/server/server.cc:471] envoy.transport_sockets.upstream: envoy.transport_sockets.alts, envoy.transport_sockets.dynamic_modules, envoy.transport_sockets.http_11_proxy, envoy.transport_sockets.internal_upstream, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, envoy.transport_sockets.upstream_proxy_protocol, raw_buffer, starttls, tls4056lb # [ 21.071286] envoy[1008]: [2026-09-20 12:20:37.095][1008][info][main] [source/server/server.cc:471] envoy.tls.certificate_selectors: envoy.tls.certificate_selectors.on_demand_secret4057lb # [ 21.073508] envoy[1008]: [2026-09-20 12:20:37.095][1008][info][main] [source/server/server.cc:471] envoy.filters.listener: envoy.filters.listener.dynamic_modules, envoy.filters.listener.http_inspector, envoy.filters.listener.local_ratelimit, envoy.filters.listener.original_dst, envoy.filters.listener.original_src, envoy.filters.listener.proxy_protocol, envoy.filters.listener.set_filter_state, envoy.filters.listener.tls_inspector, envoy.listener.http_inspector, envoy.listener.original_dst, envoy.listener.original_src, envoy.listener.proxy_protocol, envoy.listener.tls_inspector4058lb # [ 21.079398] envoy[1008]: [2026-09-20 12:20:37.095][1008][info][main] [source/server/server.cc:471] envoy.quic.server.crypto_stream: envoy.quic.crypto_stream.server.quiche4059lb # [ 21.081308] envoy[1008]: [2026-09-20 12:20:37.099][1008][info][main] [source/server/server.cc:471] envoy.formatter: envoy.formatter.cel, envoy.formatter.dynamic_modules, envoy.formatter.file_content, envoy.formatter.generic_secret, envoy.formatter.metadata, envoy.formatter.req_without_query4060lb # [ 21.084406] envoy[1008]: [2026-09-20 12:20:37.101][1008][info][main] [source/server/server.cc:471] envoy.quic.connection_debug_visitor: envoy.quic.connection_debug_visitor.basic, envoy.quic.connection_debug_visitor.quic_stats4061lb # [ 21.086867] envoy[1008]: [2026-09-20 12:20:37.101][1008][info][main] [source/server/server.cc:471] envoy.quic.proof_source: envoy.quic.proof_source.filter_chain4062lb # [ 21.088798] envoy[1008]: [2026-09-20 12:20:37.101][1008][info][main] [source/server/server.cc:471] envoy.rds_factory: envoy.rds_factory.default4063lb # [ 21.090530] envoy[1008]: [2026-09-20 12:20:37.101][1008][info][main] [source/server/server.cc:471] envoy.http.ext_proc.processing_request_modifiers: envoy.extensions.http.ext_proc.mapped_attribute_builder4064lb # [ 21.093220] envoy[1008]: [2026-09-20 12:20:37.101][1008][info][main] [source/server/server.cc:471] envoy.access_loggers: envoy.access_loggers.dynamic_modules, envoy.access_loggers.file, envoy.access_loggers.fluentd, envoy.access_loggers.http_grpc, envoy.access_loggers.open_telemetry, envoy.access_loggers.stats, envoy.access_loggers.stderr, envoy.access_loggers.stdout, envoy.access_loggers.tcp_grpc, envoy.access_loggers.wasm, envoy.file_access_log, envoy.fluentd_access_log, envoy.http_grpc_access_log, envoy.open_telemetry_access_log, envoy.stderr_access_log, envoy.stdout_access_log, envoy.tcp_grpc_access_log, envoy.wasm_access_log4065lb # [ 21.100345] envoy[1008]: [2026-09-20 12:20:37.103][1008][info][main] [source/server/server.cc:471] envoy.connection_handler: envoy.connection_handler.default4066lb # [ 21.102915] envoy[1008]: [2026-09-20 12:20:37.103][1008][info][main] [source/server/server.cc:471] envoy.http.forward_client_cert_details: forward_client_cert4067lb # [ 21.105231] envoy[1008]: [2026-09-20 12:20:37.103][1008][info][main] [source/server/server.cc:471] envoy.rbac.matchers: envoy.rbac.matchers.upstream_ip_port4068lb # [ 21.107551] envoy[1008]: [2026-09-20 12:20:37.103][1008][info][main] [source/server/server.cc:471] envoy.path.rewrite: envoy.path.rewrite.uri_template.uri_template_rewriter4069lb # [ 21.110141] envoy[1008]: [2026-09-20 12:20:37.103][1008][info][main] [source/server/server.cc:471] envoy.grpc_credentials: envoy.grpc_credentials.default, envoy.grpc_credentials.file_based_metadata4070lb # [ 21.112964] envoy[1008]: [2026-09-20 12:20:37.103][1008][info][main] [source/server/server.cc:471] envoy.resolvers: envoy.ip, envoy.resolvers.reverse_connection4071lb # [ 21.115273] envoy[1008]: [2026-09-20 12:20:37.103][1008][info][main] [source/server/server.cc:471] envoy.thrift_proxy.protocols: auto, binary, binary/non-strict, compact, twitter4072lb # [ 21.117733] envoy[1008]: [2026-09-20 12:20:37.105][1008][info][main] [source/server/server.cc:471] envoy.matching.network.input: envoy.matching.inputs.application_protocol, envoy.matching.inputs.destination_ip, envoy.matching.inputs.destination_port, envoy.matching.inputs.direct_source_ip, envoy.matching.inputs.dns_san, envoy.matching.inputs.filter_state, envoy.matching.inputs.network_namespace, envoy.matching.inputs.server_name, envoy.matching.inputs.source_ip, envoy.matching.inputs.source_port, envoy.matching.inputs.source_type, envoy.matching.inputs.subject, envoy.matching.inputs.transport_protocol, envoy.matching.inputs.uri_san4073lb # [ 21.125734] envoy[1008]: [2026-09-20 12:20:37.105][1008][info][main] [source/server/server.cc:471] envoy.matching.generic_proxy_request_input.input: envoy.matching.generic_proxy.input.host, envoy.matching.generic_proxy.input.method, envoy.matching.generic_proxy.input.path, envoy.matching.generic_proxy.input.property, envoy.matching.generic_proxy.input.request, envoy.matching.generic_proxy.input.service4074lb # [ 21.131158] envoy[1008]: [2026-09-20 12:20:37.105][1008][info][main] [source/server/server.cc:471] envoy.http.stateful_session: envoy.http.stateful_session.cookie, envoy.http.stateful_session.envelope, envoy.http.stateful_session.header4075lb # [ 21.134585] envoy[1008]: [2026-09-20 12:20:37.105][1008][info][main] [source/server/server.cc:471] envoy.quic.server_preferred_address: quic.server_preferred_address.datasource, quic.server_preferred_address.fixed4076lb # [ 21.137585] envoy[1008]: [2026-09-20 12:20:37.105][1008][info][main] [source/server/server.cc:471] envoy.matching.action: envoy.matching.action.transport_socket.name, envoy.matching.actions.format_string, filter-chain-name4077lb # [ 21.140847] envoy[1008]: [2026-09-20 12:20:37.105][1008][info][main] [source/server/server.cc:471] envoy.transport_sockets.downstream: envoy.transport_sockets.alts, envoy.transport_sockets.dynamic_modules, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, raw_buffer, starttls, tls4078lb # [ 21.146644] envoy[1008]: [2026-09-20 12:20:37.105][1008][info][main] [source/server/server.cc:471] envoy.generic_proxy.codecs: envoy.generic_proxy.codecs.dubbo, envoy.generic_proxy.codecs.http14079lb # [ 21.149475] envoy[1008]: [2026-09-20 12:20:37.105][1008][info][main] [source/server/server.cc:471] envoy.quic.client_packet_writer: envoy.quic.packet_writer.default4080lb # [ 21.152158] envoy[1008]: [2026-09-20 12:20:37.105][1008][info][main] [source/server/server.cc:471] envoy.udp_packet_writer: envoy.udp_packet_writer.default, envoy.udp_packet_writer.gso4081lb # [ 21.154617] envoy[1008]: [2026-09-20 12:20:37.105][1008][info][main] [source/server/server.cc:471] envoy.matching.common_inputs: envoy.matching.common_inputs.environment_variable4082lb # [ 21.157660] envoy[1008]: [2026-09-20 12:20:37.108][1008][info][main] [source/server/server.cc:471] envoy.tls.upstream_certificate_selectors: envoy.tls.certificate_selectors.on_demand_secret4083lb # [ 21.160189] envoy[1008]: [2026-09-20 12:20:37.108][1008][info][main] [source/server/server.cc:471] envoy.path.match: envoy.path.match.uri_template.uri_template_matcher4084lb # [ 21.162479] envoy[1008]: [2026-09-20 12:20:37.108][1008][info][main] [source/server/server.cc:471] envoy.filters.network: envoy.echo, envoy.ext_authz, envoy.filters.network.connection_limit, envoy.filters.network.direct_response, envoy.filters.network.dubbo_proxy, envoy.filters.network.dynamic_modules, envoy.filters.network.echo, envoy.filters.network.ext_authz, envoy.filters.network.ext_proc, envoy.filters.network.generic_proxy, envoy.filters.network.geoip, envoy.filters.network.http_connection_manager, envoy.filters.network.local_ratelimit, envoy.filters.network.mongo_proxy, envoy.filters.network.ratelimit, envoy.filters.network.rbac, envoy.filters.network.redis_proxy, envoy.filters.network.reverse_tunnel, envoy.filters.network.reverse_tunnel_drain_aware_http_connection_manager, envoy.filters.network.set_filter_state, envoy.filters.network.sni_cluster, envoy.filters.network.sni_dynamic_forward_proxy, envoy.filters.network.tcp_bandwidth_limit, envoy.filters.network.tcp_proxy, envoy.filters.network.thrift_proxy, envoy.filters.network.wasm, envoy.filters.network.zookeeper_proxy, envoy.http_connection_manager, envoy.mongo_proxy, envoy.ratelimit, envoy.redis_proxy, envoy.tcp_proxy4085lb # [ 21.177326] envoy[1008]: [2026-09-20 12:20:37.108][1008][info][main] [source/server/server.cc:471] envoy.tls.upstream_certificate_mappers: envoy.tls.certificate_mappers.static_name, envoy.tls.upstream_certificate_mappers.filter_state_override4086lb # [ 21.180722] envoy[1008]: [2026-09-20 12:20:37.108][1008][info][main] [source/server/server.cc:471] envoy.http.early_header_mutation: envoy.http.early_header_mutation.header_mutation4087lb # [ 21.183355] envoy[1008]: [2026-09-20 12:20:37.108][1008][info][main] [source/server/server.cc:471] envoy.string_matcher: envoy.string_matcher.lua4088lb # [ 21.185625] envoy[1008]: [2026-09-20 12:20:37.108][1008][info][main] [source/server/server.cc:471] envoy.http.custom_response: envoy.extensions.http.custom_response.local_response_policy, envoy.extensions.http.custom_response.redirect_policy4089lb # [ 21.189053] envoy[1008]: [2026-09-20 12:20:37.108][1008][info][main] [source/server/server.cc:471] envoy.filters.udp.session: envoy.filters.udp.session.dynamic_forward_proxy, envoy.filters.udp.session.ext_authz, envoy.filters.udp.session.http_capsule4090lb # [ 21.192422] envoy[1008]: [2026-09-20 12:20:37.108][1008][info][main] [source/server/server.cc:471] envoy.stats_sinks: envoy.dog_statsd, envoy.graphite_statsd, envoy.metrics_service, envoy.open_telemetry_stat_sink, envoy.stat_sinks.dog_statsd, envoy.stat_sinks.dynamic_modules, envoy.stat_sinks.graphite_statsd, envoy.stat_sinks.hystrix, envoy.stat_sinks.metrics_service, envoy.stat_sinks.open_telemetry, envoy.stat_sinks.statsd, envoy.stat_sinks.wasm, envoy.statsd4091lb # [ 21.198169] envoy[1008]: [2026-09-20 12:20:37.108][1008][info][main] [source/server/server.cc:471] envoy.retry_host_predicates: envoy.retry_host_predicates.omit_canary_hosts, envoy.retry_host_predicates.omit_host_metadata, envoy.retry_host_predicates.previous_hosts4092lb # [ 21.201715] envoy[1008]: [2026-09-20 12:20:37.108][1008][info][main] [source/server/server.cc:471] envoy.guarddog_actions: envoy.watchdog.abort_action, envoy.watchdog.profile_action4093lb # [ 21.204375] envoy[1008]: [2026-09-20 12:20:37.108][1008][info][main] [source/server/server.cc:471] envoy.common.key_value: envoy.key_value.file_based4094lb # [ 21.206560] envoy[1008]: [2026-09-20 12:20:37.108][1008][info][main] [source/server/server.cc:471] envoy.config_subscription: envoy.config_subscription.ads, envoy.config_subscription.ads_collection, envoy.config_subscription.aggregated_grpc_collection, envoy.config_subscription.delta_grpc, envoy.config_subscription.delta_grpc_collection, envoy.config_subscription.filesystem, envoy.config_subscription.filesystem_collection, envoy.config_subscription.grpc, envoy.config_subscription.rest4095lb # [ 21.212825] envoy[1008]: [2026-09-20 12:20:37.108][1008][info][main] [source/server/server.cc:471] envoy.filters.http: envoy.bandwidth_limit, envoy.buffer, envoy.cors, envoy.csrf, envoy.ext_authz, envoy.ext_proc, envoy.extensions.filters.http.dynamic_modules, envoy.fault, envoy.filters.http.a2a, envoy.filters.http.adaptive_concurrency, envoy.filters.http.admission_control, envoy.filters.http.ai_protocol_manager, envoy.filters.http.alternate_protocols_cache, envoy.filters.http.api_key_auth, envoy.filters.http.aws_lambda, envoy.filters.http.aws_request_signing, envoy.filters.http.bandwidth_limit, envoy.filters.http.bandwidth_share, envoy.filters.http.basic_auth, envoy.filters.http.buffer, envoy.filters.http.cache, envoy.filters.http.cache_v2, envoy.filters.http.cdn_loop, envoy.filters.http.composite, envoy.filters.http.compressor, envoy.filters.http.connect_grpc_bridge, envoy.filters.http.cors, envoy.filters.http.credential_injector, envoy.filters.http.csrf, envoy.filters.http.custom_response, envoy.filters.http.decompressor, envoy.filters.http.dynamic_forward_proxy, envoy.filters.http.ext_authz, envoy.filters.http.ext_proc, envoy.filters.http.fault, envoy.filters.http.file_server, envoy.filters.http.file_system_buffer, envoy.filters.http.filter_chain, envoy.filters.http.gcp_authn, envoy.filters.http.geoip, envoy.filters.http.grpc_field_extraction, envoy.filters.http.grpc_http1_bridge, envoy.filters.http.grpc_http1_reverse_bridge, envoy.filters.http.grpc_json_reverse_transcoder, envoy.filters.http.grpc_json_transcoder, envoy.filters.http.grpc_stats, envoy.filters.http.grpc_web, envoy.filters.http.header_mutation, envoy.filters.http.header_to_metadata, envoy.filters.http.health_check, envoy.filters.http.ip_tagging, envoy.filters.http.json_to_metadata, envoy.filters.http.jwt_authn, envoy.filters.http.local_ratelimit, envoy.filters.http.lua, envoy.filters.http.match_delegate, envoy.filters.http.mcp, envoy.filters.http.mcp_json_rest_bridge, envoy.filters.http.mcp_router, envoy.filters.http.oauth2, envoy.filters.http.on_demand, envoy.filters.http.original_src, envoy.filters.http.proto_api_scrubber, envoy.filters.http.proto_message_extraction, envoy.filters.http.rate_limit_quota, envoy.filters.http.ratelimit, envoy.filters.http.rbac, envoy.filters.http.router, envoy.filters.http.set_filter_state, envoy.filters.http.set_metadata, envoy.filters.http.sse_to_metadata, envoy.filters.http.stateful_session, envoy.filters.http.tap, envoy.filters.http.thrift_to_metadata, envoy.filters.http.transform, envoy.filters.http.wasm, envoy.geoip, envoy.grpc_http1_bridge, envoy.grpc_json_transcoder, envoy.grpc_web, envoy.health_check, envoy.ip_tagging, envoy.local_rate_limit, envoy.lua, envoy.rate_limit, envoy.router4096lb # [ 21.242658] envoy[1008]: [2026-09-20 12:20:37.108][1008][info][main] [source/server/server.cc:471] envoy.thrift_proxy.filters: envoy.filters.thrift.header_to_metadata, envoy.filters.thrift.payload_to_metadata, envoy.filters.thrift.rate_limit, envoy.filters.thrift.router4097lb # [ 21.246469] envoy[1008]: [2026-09-20 12:20:37.108][1008][info][main] [source/server/server.cc:471] envoy.tracers: envoy.tracers.datadog, envoy.tracers.dynamic_modules, envoy.tracers.fluentd, envoy.tracers.opentelemetry, envoy.tracers.skywalking, envoy.tracers.xray, envoy.tracers.zipkin, envoy.zipkin4098lb # [ 21.250579] envoy[1008]: [2026-09-20 12:20:37.108][1008][info][main] [source/server/server.cc:471] envoy.tracers.opentelemetry.resource_detectors: envoy.tracers.opentelemetry.resource_detectors.dynatrace, envoy.tracers.opentelemetry.resource_detectors.environment, envoy.tracers.opentelemetry.resource_detectors.static_config4099lb # [ 21.255067] envoy[1008]: [2026-09-20 12:20:37.108][1008][info][main] [source/server/server.cc:471] envoy.network.dns_resolver: envoy.network.dns_resolver.cares, envoy.network.dns_resolver.getaddrinfo, envoy.network.dns_resolver.hickory4100lb # [ 21.258375] envoy[1008]: [2026-09-20 12:20:37.108][1008][info][main] [source/server/server.cc:471] envoy.config_mux: envoy.config_mux.delta_grpc_mux_factory, envoy.config_mux.grpc_mux_factory, envoy.config_mux.new_grpc_mux_factory, envoy.config_mux.sotw_grpc_mux_factory4101lb # [ 21.262063] envoy[1008]: [2026-09-20 12:20:37.108][1008][info][main] [source/server/server.cc:471] envoy.tls.certificate_mappers: envoy.tls.certificate_mappers.sni, envoy.tls.certificate_mappers.static_name4102lb # [ 21.265184] envoy[1008]: [2026-09-20 12:20:37.108][1008][info][main] [source/server/server.cc:471] envoy.filters.upstream_network: envoy.filters.upstream_network.reverse_tunnel_lifecycle4103lb # [ 21.267866] envoy[1008]: [2026-09-20 12:20:37.108][1008][info][main] [source/server/server.cc:471] envoy.http.header_validators: envoy.http.header_validators.envoy_default4104lb # [ 21.270330] envoy[1008]: [2026-09-20 12:20:37.108][1008][info][main] [source/server/server.cc:471] envoy.matching.transport_socket.input: envoy.matching.inputs.endpoint_metadata, envoy.matching.inputs.locality_metadata, envoy.matching.inputs.transport_socket_filter_state4105lb # [ 21.274104] envoy[1008]: [2026-09-20 12:20:37.108][1008][info][main] [source/server/server.cc:471] envoy.load_balancing_policies: envoy.load_balancing_policies.client_side_weighted_round_robin, envoy.load_balancing_policies.cluster_provided, envoy.load_balancing_policies.dynamic_modules, envoy.load_balancing_policies.least_request, envoy.load_balancing_policies.load_aware_locality, envoy.load_balancing_policies.maglev, envoy.load_balancing_policies.override_host, envoy.load_balancing_policies.random, envoy.load_balancing_policies.ring_hash, envoy.load_balancing_policies.round_robin, envoy.load_balancing_policies.subset, envoy.load_balancing_policies.wrr_locality4106lb # [ 21.282305] envoy[1008]: [2026-09-20 12:20:37.108][1008][info][main] [source/server/server.cc:471] envoy.access_loggers.extension_filters: envoy.access_loggers.extension_filters.cel, envoy.access_loggers.extension_filters.process_ratelimit4107lb # [ 21.285690] envoy[1008]: [2026-09-20 12:20:37.108][1008][info][main] [source/server/server.cc:471] envoy.route_config_update_requester: envoy.route_config_update_requester.default4108lb # [ 21.288548] envoy[1008]: [2026-09-20 12:20:37.108][1008][info][main] [source/server/server.cc:471] envoy.compression.compressor: envoy.compression.brotli.compressor, envoy.compression.gzip.compressor, envoy.compression.zstd.compressor4109lb # [ 21.291611] envoy[1008]: [2026-09-20 12:20:37.108][1008][info][main] [source/server/server.cc:471] envoy.tracers.opentelemetry.samplers: envoy.tracers.opentelemetry.samplers.always_on, envoy.tracers.opentelemetry.samplers.cel, envoy.tracers.opentelemetry.samplers.dynatrace, envoy.tracers.opentelemetry.samplers.parent_based, envoy.tracers.opentelemetry.samplers.trace_id_ratio_based4110lb # [ 21.296973] envoy[1008]: [2026-09-20 12:20:37.108][1008][info][main] [source/server/server.cc:471] envoy.srds_factory: envoy.srds_factory.default4111lb # [ 21.298988] envoy[1008]: [2026-09-20 12:20:37.108][1008][info][main] [source/server/server.cc:471] quic.http_server_connection: quic.http_server_connection.default4112lb # [ 21.301184] envoy[1008]: [2026-09-20 12:20:37.108][1008][info][main] [source/server/server.cc:471] envoy.retry_priorities: envoy.retry_priorities.previous_priorities4113lb # [ 21.303858] envoy[1008]: [2026-09-20 12:20:37.108][1008][info][main] [source/server/server.cc:471] envoy.matching.network.custom_matchers: envoy.matching.custom_matchers.domain_matcher, envoy.matching.custom_matchers.ip_range_matcher4114lb # [ 21.307759] envoy[1008]: [2026-09-20 12:20:37.108][1008][info][main] [source/server/server.cc:471] envoy.upstreams: envoy.filters.connection_pools.tcp.generic4115lb # [ 21.310071] envoy[1008]: [2026-09-20 12:20:37.108][1008][info][main] [source/server/server.cc:471] envoy.dubbo_proxy.serializers: dubbo.hessian24116lb # [ 21.312278] envoy[1008]: [2026-09-20 12:20:37.108][1008][info][main] [source/server/server.cc:471] envoy.internal_redirect_predicates: envoy.internal_redirect_predicates.allow_listed_routes, envoy.internal_redirect_predicates.previous_routes, envoy.internal_redirect_predicates.safe_cross_scheme4117lb # [ 21.316054] envoy[1008]: [2026-09-20 12:20:37.239][1008][info][main] [source/server/server.cc:531] HTTP header map info:4118lb # [ 21.317726] envoy[1008]: [2026-09-20 12:20:37.306][1008][info][main] [source/server/server.cc:534] request header map: 664 bytes: :authority,:method,:path,:protocol,:scheme,accept,accept-encoding,access-control-request-headers,access-control-request-method,access-control-request-private-network,authentication,authorization,cache-control,cdn-loop,connection,content-encoding,content-length,content-type,expect,grpc-accept-encoding,grpc-timeout,if-match,if-modified-since,if-none-match,if-range,if-unmodified-since,keep-alive,origin,pragma,proxy-connection,proxy-status,referer,te,transfer-encoding,upgrade,user-agent,via,x-client-trace-id,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-downstream-service-cluster,x-envoy-downstream-service-node,x-envoy-expected-rq-timeout-ms,x-envoy-external-address,x-envoy-force-trace,x-envoy-hedge-on-per-try-timeout,x-envoy-internal,x-envoy-ip-tags,x-envoy-is-timeout-retry,x-envoy-max-retries,x-envoy-original-host,x-envoy-original-path,x-envoy-original-url,x-envoy-retriable-header-names,x-envoy-retriable-status-codes,x-envoy-retry-grpc-on,x-envoy-retry-on,x-envoy-upstream-alt-stat-name,x-envoy-upstream-rq-per-try-timeout-ms,x-envoy-upstream-rq-timeout-alt-response,x-envoy-upstream-rq-timeout-ms,x-envoy-upstream-stream-duration-ms,x-forwarded-client-cert,x-forwarded-for,x-forwarded-host,x-forwarded-port,x-forwarded-proto,x-request-id4119lb # [ 21.333700] envoy[1008]: [2026-09-20 12:20:37.306][1008][info][main] [source/server/server.cc:534] request trailer map: 120 bytes:4120lb # [ 21.335814] envoy[1008]: [2026-09-20 12:20:37.306][1008][info][main] [source/server/server.cc:534] response header map: 432 bytes: :status,access-control-allow-credentials,access-control-allow-headers,access-control-allow-methods,access-control-allow-origin,access-control-allow-private-network,access-control-expose-headers,access-control-max-age,age,cache-control,connection,content-encoding,content-length,content-type,date,etag,expires,grpc-message,grpc-status,keep-alive,last-modified,location,proxy-connection,proxy-status,server,transfer-encoding,upgrade,vary,via,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-degraded,x-envoy-immediate-health-check-fail,x-envoy-ratelimited,x-envoy-upstream-canary,x-envoy-upstream-healthchecked-cluster,x-envoy-upstream-service-time,x-request-id4121lb # [ 21.344182] envoy[1008]: [2026-09-20 12:20:37.306][1008][info][main] [source/server/server.cc:534] response trailer map: 144 bytes: grpc-message,grpc-status4122lb # [ 21.395529] envoy[1008]: [2026-09-20 12:20:37.561][1008][info][main] [source/server/server.cc:968] runtime: {}4123lb # [ 21.401336] envoy[1008]: [2026-09-20 12:20:37.567][1008][info][main] [source/server/server.cc:420] Raised file descriptor limits from 1024 to 524288.4124lb # [ 21.403501] envoy[1008]: [2026-09-20 12:20:37.569][1008][info][main] [source/server/server.cc:796] Starting admin HTTP server at 127.0.0.1:99014125lb # [ 21.406344] envoy[1008]: [2026-09-20 12:20:37.572][1008][info][admin] [source/server/admin/admin.cc:66] admin address: 127.0.0.1:99014126lb # [ 21.409424] envoy[1008]: [2026-09-20 12:20:37.575][1008][info][config] [source/server/configuration_impl.cc:181] loading tracing configuration4127lb # [ 21.411352] envoy[1008]: [2026-09-20 12:20:37.577][1008][info][config] [source/server/configuration_impl.cc:132] loading 0 static secret(s)4128lb # [ 21.413210] envoy[1008]: [2026-09-20 12:20:37.577][1008][info][config] [source/server/configuration_impl.cc:138] loading 2 cluster(s)4129lb # [ 21.463707] envoy[1008]: [2026-09-20 12:20:37.629][1008][info][config] [source/server/configuration_impl.cc:148] loading 1 listener(s)4130worker1 # [ 21.467829] systemd[1]: Starting Nix worker-protocol over gRPC...4131worker2 # [ 21.454728] systemd[1]: Reached target Network is Online.4132worker2 # [ 21.463479] systemd[1]: Starting scheduler-token.service...4133lb # [ 21.491066] envoy[1008]: [2026-09-20 12:20:37.657][1008][info][config] [source/server/configuration_impl.cc:164] loading stats configuration4134lb # [ 21.494276] envoy[1008]: [2026-09-20 12:20:37.660][1008][warning][main] [source/server/server.cc:1038] There is no configured limit to the number of allowed active downstream connections. Configure a limit in `envoy.resource_monitors.global_downstream_max_connections` resource monitor.4135lb # [ 21.497880] envoy[1008]: [2026-09-20 12:20:37.660][1008][info][main] [source/server/server.cc:1077] starting main dispatch loop4136worker2 # [ 21.594260] systemd[1]: scheduler-token.service: Deactivated successfully.4137worker2 # [ 21.596741] systemd[1]: Finished scheduler-token.service.4138worker2 # [ 21.598673] systemd[1]: scheduler-token.service: Consumed 41ms CPU time over 133ms wall clock time, 3.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.4139worker2 # [ 21.606875] systemd[1]: Starting Nix worker-protocol over gRPC...4140worker1 # [ 21.657981] nix-grpc-daemon[786]: ts=2026-09-20T12:20:36Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"4141worker1 # [ 21.667325] nix-grpc-daemon[786]: ts=2026-09-20T12:20:36Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc4142worker1 # [ 21.675765] nix-grpc-daemon[786]: ts=2026-09-20T12:20:36Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=14143worker1 # [ 21.711609] nix-grpc-daemon[786]: ts=2026-09-20T12:20:36Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket4144worker1 # [ 21.714912] systemd[1]: Started Nix worker-protocol over gRPC.4145worker1 # [ 21.759266] nix-grpc-daemon[786]: ts=2026-09-20T12:20:36Z level=info event=scheduler_disconnected addr=lb:50051 error="failed to connect to all addresses; last error: UNKNOWN: ipv4:192.168.1.2:50051: Failed to connect to remote host: Connection refused"4146lb # [ 21.760975] envoy[1008]: [2026-09-20 12:20:37.926][1008][info][runtime] [source/common/runtime/runtime_impl.cc:567] RTDS has finished initialization4147lb # [ 21.764058] envoy[1008]: [2026-09-20 12:20:37.927][1008][info][upstream] [source/common/upstream/cluster_manager_impl.cc:250] cm init: all clusters initialized4148lb # [ 21.766372] envoy[1008]: [2026-09-20 12:20:37.927][1008][info][main] [source/server/server.cc:1057] all clusters initialized. initializing init manager4149lb # [ 21.768341] envoy[1008]: [2026-09-20 12:20:37.927][1008][info][config] [source/common/listener_manager/listener_manager_impl.cc:1067] all dependencies initialized. starting workers4150worker2 # [ 21.794192] nix-grpc-daemon[857]: ts=2026-09-20T12:20:37Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"4151worker2 # [ 21.803281] nix-grpc-daemon[857]: ts=2026-09-20T12:20:37Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc4152worker2 # [ 21.809316] nix-grpc-daemon[857]: ts=2026-09-20T12:20:37Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=14153worker2 # [ 21.855983] nix-grpc-daemon[857]: ts=2026-09-20T12:20:37Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket4154worker2 # [ 21.859203] systemd[1]: Started Nix worker-protocol over gRPC.4155worker2 # [ 21.912207] nix-grpc-daemon[857]: ts=2026-09-20T12:20:37Z level=info event=scheduler_connected addr=lb:500514156worker2 # [ 21.923608] nix-grpc-daemon[857]: ts=2026-09-20T12:20:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4157worker1 # [ 21.960427] nix-grpc-daemon[786]: ts=2026-09-20T12:20:37Z level=info event=scheduler_disconnected addr=lb:50051 error="failed to connect to all addresses; last error: UNKNOWN: ipv4:192.168.1.2:50051: Failed to connect to remote host: Connection refused"4158worker2 # [ 22.126287] nix-grpc-daemon[857]: ts=2026-09-20T12:20:37Z level=info event=scheduler_connected addr=lb:500514159worker2 # [ 22.130170] nix-grpc-daemon[857]: ts=2026-09-20T12:20:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4160worker2 # [ 22.331203] nix-grpc-daemon[857]: ts=2026-09-20T12:20:37Z level=info event=scheduler_connected addr=lb:500514161worker2 # [ 22.336137] nix-grpc-daemon[857]: ts=2026-09-20T12:20:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4162worker1 # [ 22.361102] nix-grpc-daemon[786]: ts=2026-09-20T12:20:37Z level=info event=scheduler_disconnected addr=lb:50051 error="failed to connect to all addresses; last error: UNKNOWN: ipv4:192.168.1.2:50051: Failed to connect to remote host: Connection refused"4163worker2 # [ 22.537634] nix-grpc-daemon[857]: ts=2026-09-20T12:20:37Z level=info event=scheduler_connected addr=lb:500514164worker2 # [ 22.541581] nix-grpc-daemon[857]: ts=2026-09-20T12:20:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4165worker2 # [ 22.742269] nix-grpc-daemon[857]: ts=2026-09-20T12:20:38Z level=info event=scheduler_connected addr=lb:500514166worker2 # [ 22.746292] nix-grpc-daemon[857]: ts=2026-09-20T12:20:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4167worker2 # [ 22.947196] nix-grpc-daemon[857]: ts=2026-09-20T12:20:38Z level=info event=scheduler_connected addr=lb:500514168worker2 # [ 22.949289] nix-grpc-daemon[857]: ts=2026-09-20T12:20:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4169worker2 # [ 23.150645] nix-grpc-daemon[857]: ts=2026-09-20T12:20:38Z level=info event=scheduler_connected addr=lb:500514170worker1 # [ 23.165954] nix-grpc-daemon[786]: ts=2026-09-20T12:20:38Z level=info event=scheduler_connected addr=lb:500514171worker2 # [ 23.154827] nix-grpc-daemon[857]: ts=2026-09-20T12:20:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4172worker1 # [ 23.170711] nix-grpc-daemon[786]: ts=2026-09-20T12:20:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4173worker2 # [ 23.355958] nix-grpc-daemon[857]: ts=2026-09-20T12:20:38Z level=info event=scheduler_connected addr=lb:500514174worker2 # [ 23.358615] nix-grpc-daemon[857]: ts=2026-09-20T12:20:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4175worker1 # [ 23.371694] nix-grpc-daemon[786]: ts=2026-09-20T12:20:38Z level=info event=scheduler_connected addr=lb:500514176worker1 # [ 23.374149] nix-grpc-daemon[786]: ts=2026-09-20T12:20:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4177worker2 # [ 23.559362] nix-grpc-daemon[857]: ts=2026-09-20T12:20:38Z level=info event=scheduler_connected addr=lb:500514178worker2 # [ 23.561410] nix-grpc-daemon[857]: ts=2026-09-20T12:20:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4179worker1 # [ 23.574954] nix-grpc-daemon[786]: ts=2026-09-20T12:20:38Z level=info event=scheduler_connected addr=lb:500514180worker1 # [ 23.577649] nix-grpc-daemon[786]: ts=2026-09-20T12:20:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4181worker2 # [ 23.762386] nix-grpc-daemon[857]: ts=2026-09-20T12:20:39Z level=info event=scheduler_connected addr=lb:500514182worker2 # [ 23.764104] nix-grpc-daemon[857]: ts=2026-09-20T12:20:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4183worker1 # [ 23.777995] nix-grpc-daemon[786]: ts=2026-09-20T12:20:39Z level=info event=scheduler_connected addr=lb:500514184worker1 # [ 23.779957] nix-grpc-daemon[786]: ts=2026-09-20T12:20:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4185worker2 # [ 23.966848] nix-grpc-daemon[857]: ts=2026-09-20T12:20:39Z level=info event=scheduler_connected addr=lb:500514186worker1 # [ 23.980584] nix-grpc-daemon[786]: ts=2026-09-20T12:20:39Z level=info event=scheduler_connected addr=lb:500514187worker2 # [ 23.968997] nix-grpc-daemon[857]: ts=2026-09-20T12:20:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4188worker1 # [ 23.982578] nix-grpc-daemon[786]: ts=2026-09-20T12:20:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4189worker1 # [ 24.183511] nix-grpc-daemon[786]: ts=2026-09-20T12:20:39Z level=info event=scheduler_connected addr=lb:500514190worker2 # [ 24.170711] nix-grpc-daemon[857]: ts=2026-09-20T12:20:39Z level=info event=scheduler_connected addr=lb:500514191worker1 # [ 24.185480] nix-grpc-daemon[786]: ts=2026-09-20T12:20:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4192worker2 # [ 24.174745] nix-grpc-daemon[857]: ts=2026-09-20T12:20:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4193worker1 # [ 24.386374] nix-grpc-daemon[786]: ts=2026-09-20T12:20:39Z level=info event=scheduler_connected addr=lb:500514194worker1 # [ 24.388549] nix-grpc-daemon[786]: ts=2026-09-20T12:20:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4195worker2 # [ 24.376332] nix-grpc-daemon[857]: ts=2026-09-20T12:20:39Z level=info event=scheduler_connected addr=lb:500514196worker2 # [ 24.378957] nix-grpc-daemon[857]: ts=2026-09-20T12:20:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4197worker1 # [ 24.589211] nix-grpc-daemon[786]: ts=2026-09-20T12:20:39Z level=info event=scheduler_connected addr=lb:500514198worker1 # [ 24.591233] nix-grpc-daemon[786]: ts=2026-09-20T12:20:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4199worker2 # [ 24.579997] nix-grpc-daemon[857]: ts=2026-09-20T12:20:39Z level=info event=scheduler_connected addr=lb:500514200worker2 # [ 24.583304] nix-grpc-daemon[857]: ts=2026-09-20T12:20:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4201worker1 # [ 24.791772] nix-grpc-daemon[786]: ts=2026-09-20T12:20:40Z level=info event=scheduler_connected addr=lb:500514202worker1 # [ 24.793636] nix-grpc-daemon[786]: ts=2026-09-20T12:20:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4203worker2 # [ 24.782454] nix-grpc-daemon[857]: ts=2026-09-20T12:20:40Z level=info event=scheduler_connected addr=lb:500514204worker2 # [ 24.786334] nix-grpc-daemon[857]: ts=2026-09-20T12:20:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4205worker1 # [ 24.994331] nix-grpc-daemon[786]: ts=2026-09-20T12:20:40Z level=info event=scheduler_connected addr=lb:500514206worker1 # [ 24.996679] nix-grpc-daemon[786]: ts=2026-09-20T12:20:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4207worker2 # [ 24.985151] nix-grpc-daemon[857]: ts=2026-09-20T12:20:40Z level=info event=scheduler_connected addr=lb:500514208worker2 # [ 24.986894] nix-grpc-daemon[857]: ts=2026-09-20T12:20:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4209worker2 # [ 25.185940] nix-grpc-daemon[857]: ts=2026-09-20T12:20:40Z level=info event=scheduler_connected addr=lb:500514210worker1 # [ 25.198098] nix-grpc-daemon[786]: ts=2026-09-20T12:20:40Z level=info event=scheduler_connected addr=lb:500514211worker2 # [ 25.189598] nix-grpc-daemon[857]: ts=2026-09-20T12:20:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4212worker1 # [ 25.202613] nix-grpc-daemon[786]: ts=2026-09-20T12:20:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4213worker1 # [ 25.402533] nix-grpc-daemon[786]: ts=2026-09-20T12:20:40Z level=info event=scheduler_connected addr=lb:500514214worker2 # [ 25.391627] nix-grpc-daemon[857]: ts=2026-09-20T12:20:40Z level=info event=scheduler_connected addr=lb:500514215worker1 # [ 25.404699] nix-grpc-daemon[786]: ts=2026-09-20T12:20:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4216worker2 # [ 25.394586] nix-grpc-daemon[857]: ts=2026-09-20T12:20:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4217worker1 # [ 25.605856] nix-grpc-daemon[786]: ts=2026-09-20T12:20:40Z level=info event=scheduler_connected addr=lb:500514218worker2 # [ 25.595510] nix-grpc-daemon[857]: ts=2026-09-20T12:20:40Z level=info event=scheduler_connected addr=lb:500514219worker1 # [ 25.608207] nix-grpc-daemon[786]: ts=2026-09-20T12:20:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4220worker2 # [ 25.597668] nix-grpc-daemon[857]: ts=2026-09-20T12:20:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4221worker1 # [ 25.808821] nix-grpc-daemon[786]: ts=2026-09-20T12:20:41Z level=info event=scheduler_connected addr=lb:500514222worker2 # [ 25.798576] nix-grpc-daemon[857]: ts=2026-09-20T12:20:41Z level=info event=scheduler_connected addr=lb:500514223worker1 # [ 25.811529] nix-grpc-daemon[786]: ts=2026-09-20T12:20:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4224worker2 # [ 25.800827] nix-grpc-daemon[857]: ts=2026-09-20T12:20:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4225worker1 # [ 26.012303] nix-grpc-daemon[786]: ts=2026-09-20T12:20:41Z level=info event=scheduler_connected addr=lb:500514226worker2 # [ 26.002925] nix-grpc-daemon[857]: ts=2026-09-20T12:20:41Z level=info event=scheduler_connected addr=lb:500514227worker1 # [ 26.015120] nix-grpc-daemon[786]: ts=2026-09-20T12:20:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4228worker2 # [ 26.004453] nix-grpc-daemon[857]: ts=2026-09-20T12:20:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4229worker2 # [ 26.204455] nix-grpc-daemon[857]: ts=2026-09-20T12:20:41Z level=info event=scheduler_connected addr=lb:500514230worker2 # [ 26.206784] nix-grpc-daemon[857]: ts=2026-09-20T12:20:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4231worker1 # [ 26.217152] nix-grpc-daemon[786]: ts=2026-09-20T12:20:41Z level=info event=scheduler_connected addr=lb:500514232worker1 # [ 26.221721] nix-grpc-daemon[786]: ts=2026-09-20T12:20:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4233lb # [ 26.242449] systemd[1]: niks3.service: Scheduled restart job, restart counter is at 1.4234lb # [ 26.249846] systemd[1]: Starting niks3 server...4235lb # [ 26.352407] postgres[1051]: [1051] ERROR: relation "goose_db_version" does not exist at character 364236lb # [ 26.354394] postgres[1051]: [1051] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC4237lb # [ 26.412477] niks3-server[1046]: 2026/09/20 12:20:42 OK 20241026095416_initial_model.sql (32.53ms)4238lb # [ 26.418326] niks3-server[1046]: 2026/09/20 12:20:42 OK 20251210153512_drop_unused_gin_index.sql (3.52ms)4239worker2 # [ 26.408138] nix-grpc-daemon[857]: ts=2026-09-20T12:20:41Z level=info event=scheduler_connected addr=lb:500514240worker1 # [ 26.422615] nix-grpc-daemon[786]: ts=2026-09-20T12:20:41Z level=info event=scheduler_connected addr=lb:500514241worker2 # [ 26.411263] nix-grpc-daemon[857]: ts=2026-09-20T12:20:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4242worker1 # [ 26.425447] nix-grpc-daemon[786]: ts=2026-09-20T12:20:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4243lb # [ 26.428094] niks3-server[1046]: 2026/09/20 12:20:42 OK 20251218171726_add_pins.sql (6.75ms)4244lb # [ 26.436222] niks3-server[1046]: 2026/09/20 12:20:42 OK 20260628120000_add_object_size_and_stats.sql (8.3ms)4245lb # [ 26.442868] niks3-server[1046]: 2026/09/20 12:20:42 OK 20260905000000_add_claims.sql (4.52ms)4246lb # [ 26.449555] niks3-server[1046]: 2026/09/20 12:20:42 OK 20260920000000_drop_claims.sql (6.53ms)4247lb # [ 26.451238] niks3-server[1046]: 2026/09/20 12:20:42 goose: successfully migrated database to version: 202609200000004248lb # [ 26.458630] niks3-server[1046]: 2026/09/20 12:20:42 OK 1_commit_pending_closure.sql (9.16ms)4249lb # [ 26.462283] niks3-server[1046]: 2026/09/20 12:20:42 OK 2_object_stats_trigger.sql (1.8ms)4250lb # [ 26.464189] niks3-server[1046]: 2026/09/20 12:20:42 goose: up to current file version: 24251lb # [ 26.470853] niks3-server[1046]: 2026/09/20 12:20:42 INFO Loaded signing key name=farm-test-1 path=/nix/store/c745bsy529madd2s6hz4z5is2436ni04-key4252lb # [ 26.565444] niks3-server[1046]: 2026/09/20 12:20:42 INFO Read proxy enabled — serving cache objects from S34253lb # [ 26.567824] niks3-server[1046]: 2026/09/20 12:20:42 INFO Using socket-activated listener address=0.0.0.0:57514254lb # [ 26.577437] systemd[1]: Started niks3 server.4255lb # [ 26.578490] niks3-server[1046]: 2026/09/20 12:20:42 INFO systemd watchdog enabled interval=15s4256lb # [ 26.582239] niks3-server[1046]: 2026/09/20 12:20:42 INFO Starting HTTP server address=0.0.0.0:57514257lb # [ 26.589149] niks3-server[1046]: 2026/09/20 12:20:42 INFO lead: acquired remote=192.168.1.4:374364258worker2 # [ 26.587664] nix-grpc-daemon[857]: ts=2026-09-20T12:20:41Z level=info event=scheduler_take_over4259worker2 # [ 26.612469] nix-grpc-daemon[857]: ts=2026-09-20T12:20:41Z level=info event=scheduler_connected addr=lb:500514260worker1 # [ 26.626144] nix-grpc-daemon[786]: ts=2026-09-20T12:20:41Z level=info event=scheduler_connected addr=lb:500514261worker2 # [ 26.615275] nix-grpc-daemon[857]: ts=2026-09-20T12:20:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4262worker1 # [ 26.628834] nix-grpc-daemon[786]: ts=2026-09-20T12:20:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4263lb: (finished: waiting for unit niks3.service, in 13.96 seconds)4264lb: waiting for TCP port 5751 on localhost4265lb # Connection to localhost (127.0.0.1) 5751 port [tcp/*] succeeded!4266lb: (finished: waiting for TCP port 5751 on localhost, in 0.09 seconds)4267lb: waiting for unit envoy.service4268lb: (finished: waiting for unit envoy.service, in 0.06 seconds)4269lb: waiting for TCP port 50051 on localhost4270worker2 # [ 26.816779] nix-grpc-daemon[857]: ts=2026-09-20T12:20:42Z level=info event=scheduler_connected addr=lb:500514271worker1 # [ 26.829689] nix-grpc-daemon[786]: ts=2026-09-20T12:20:42Z level=info event=scheduler_connected addr=lb:500514272worker2 # [ 26.819614] nix-grpc-daemon[857]: ts=2026-09-20T12:20:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4273worker1 # [ 26.832416] nix-grpc-daemon[786]: ts=2026-09-20T12:20:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4274lb # Connection to localhost (::1) 50051 port [tcp/*] succeeded!4275lb # [ 26.864484] envoy[1008]: conn peer=[::1]:60452 tls=TLSv1.2 subject="-" sni=- flags=- tls_fail="-" rx=0 tx=0 ms=44276lb: (finished: waiting for TCP port 50051 on localhost, in 0.04 seconds)4277worker2 # [ 26.961358] systemd[1]: Starting scheduler-token.service...4278worker1 # [ 27.033056] nix-grpc-daemon[786]: ts=2026-09-20T12:20:42Z level=info event=scheduler_connected addr=lb:500514279worker2 # [ 27.022044] nix-grpc-daemon[857]: ts=2026-09-20T12:20:42Z level=info event=scheduler_connected addr=lb:500514280worker2 # [ 27.034705] nix-grpc-daemon[857]: ts=2026-09-20T12:20:42Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:417324281worker2 # [ 27.040728] nix-grpc-daemon[857]: ts=2026-09-20T12:20:42Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=04282worker2 # [ 27.046463] systemd[1]: scheduler-token.service: Deactivated successfully.4283worker2 # [ 27.051329] systemd[1]: Finished scheduler-token.service.4284worker2 # [ 27.053699] systemd[1]: scheduler-token.service: Consumed 29ms CPU time over 87ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.4285worker2 # [ 27.059667] nix-grpc-daemon[857]: ts=2026-09-20T12:20:42Z level=info event=oidc_jwks provider=mock keys=14286worker2 # [ 27.061228] nix-grpc-daemon[857]: ts=2026-09-20T12:20:42Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:417324287worker2 # [ 27.063226] nix-grpc-daemon[857]: ts=2026-09-20T12:20:42Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=04288lb: waiting until exactly no endpoints of x86_64-linux are unhealthy4289lb: must succeed: curl -sf localhost:9901/clusters4290lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)4291lb: (finished: waiting until exactly no endpoints of x86_64-linux are unhealthy, in 0.07 seconds)4292subtest: one node schedules and both builders hold a WorkerSession on it4293lb: must succeed: curl -sf localhost:9901/clusters4294lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4295worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4296worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)4297lb: must succeed: curl -sf localhost:9901/clusters4298lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4299lb: must succeed: curl -sf localhost:9901/clusters4300lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4301lb: must succeed: curl -sf localhost:9901/clusters4302lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4303lb: must succeed: curl -sf localhost:9901/clusters4304lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4305worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open'4306worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open', in 0.05 seconds)4307worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4308worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)4309lb: must succeed: curl -sf localhost:9901/clusters4310lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4311lb: must succeed: curl -sf localhost:9901/clusters4312lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4313worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4314worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)4315worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4316worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)4317worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4318worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)4319worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4320worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.03 seconds)4321(finished: subtest: one node schedules and both builders hold a WorkerSession on it, in 0.63 seconds)4322subtest: balancer auth: client cert, bearer token, nothing, unknown CN4323worker1 # [ 28.075223] nix-grpc-daemon[786]: ts=2026-09-20T12:20:43Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:384104324worker1 # [ 28.081471] systemd[1]: Started Nix Daemon.4325worker1 # [ 28.276312] nix-daemon[845]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4326worker1 # [ 28.315500] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 7 ms (attempt 1/5)4327worker1 # [ 28.325183] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 82 ms (attempt 2/5)4328worker1 # [ 28.410179] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 121 ms (attempt 3/5)4329worker1 # [ 28.537046] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 759 ms (attempt 4/5)4330worker1 # [ 29.304795] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4331worker1 # [ 29.336372] nix-grpc-daemon[786]: ts=2026-09-20T12:20:44Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:384104332worker1 # [ 29.353864] nix-grpc-daemon[786]: ts=2026-09-20T12:20:44Z level=debug event=path_info_miss path=mmlfvh5rvsdyfwvf1j0yz3mcyc2x0fc8-nixos-system-client-test error="error: path '/nix/store/mmlfvh5rvsdyfwvf1j0yz3mcyc2x0fc8-nixos-system-client-test' is required, but there is no substituter that can build it"4333worker1 # [ 29.365756] nix-grpc-daemon[786]: ts=2026-09-20T12:20:44Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:384104334worker1 # [ 29.375415] nix-grpc-daemon[786]: ts=2026-09-20T12:20:44Z level=debug event=path_info_miss path=mmlfvh5rvsdyfwvf1j0yz3mcyc2x0fc8-nixos-system-client-test error="error: path '/nix/store/mmlfvh5rvsdyfwvf1j0yz3mcyc2x0fc8-nixos-system-client-test' is required, but there is no substituter that can build it"4335client: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt4336client: (finished: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt, in 0.08 seconds)4337client: must succeed: install -D /nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key4338client: (finished: must succeed: install -D /nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key, in 0.04 seconds)4339worker1 # [ 29.739592] nix-grpc-daemon[786]: ts=2026-09-20T12:20:44Z level=info event=oidc_jwks provider=mock keys=14340worker1 # [ 29.742085] nix-grpc-daemon[786]: ts=2026-09-20T12:20:44Z level=debug event=rpc_start method=QueryMissing cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:384104341worker1 # [ 29.753722] nix-grpc-daemon[786]: ts=2026-09-20T12:20:44Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:384104342worker1 # [ 29.762489] nix-grpc-daemon[786]: ts=2026-09-20T12:20:45Z level=debug event=path_info_miss path=mmlfvh5rvsdyfwvf1j0yz3mcyc2x0fc8-nixos-system-client-test error="error: path '/nix/store/mmlfvh5rvsdyfwvf1j0yz3mcyc2x0fc8-nixos-system-client-test' is required, but there is no substituter that can build it"4343worker1 # [ 29.774044] nix-grpc-daemon[786]: ts=2026-09-20T12:20:45Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:384104344worker1 # [ 29.780357] nix-grpc-daemon[786]: ts=2026-09-20T12:20:45Z level=debug event=path_info_miss path=mmlfvh5rvsdyfwvf1j0yz3mcyc2x0fc8-nixos-system-client-test error="error: path '/nix/store/mmlfvh5rvsdyfwvf1j0yz3mcyc2x0fc8-nixos-system-client-test' is required, but there is no substituter that can build it"4345client: must succeed: rm -r /var/lib/nix-grpc-store4346client: (finished: must succeed: rm -r /var/lib/nix-grpc-store, in 0.03 seconds)4347worker2 # [ 30.029693] nix-grpc-daemon[857]: ts=2026-09-20T12:20:45Z level=info event=denied method=QueryMissing cn=- role=none4348worker1 # [ 30.249042] nix-grpc-daemon[786]: ts=2026-09-20T12:20:45Z level=info event=denied method=QueryMissing cn=stranger role=none4349(finished: subtest: balancer auth: client cert, bearer token, nothing, unknown CN, in 2.50 seconds)4350subtest: DAG build is scheduled across workers and lands in the cache4351client: must succeed: nix build -L --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 >&24352worker2 # [ 30.471942] nix-grpc-daemon[857]: ts=2026-09-20T12:20:45Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:417344353worker2 # [ 30.479596] systemd[1]: Started Nix Daemon.4354worker2 # [ 30.684389] nix-daemon[937]: accepted connection from pid 857, user nix-grpc-daemon (trusted)4355client # don't know how to build these paths:4356client # /nix/store/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv4357worker2 # [ 30.705487] nix-grpc-daemon[857]: ts=2026-09-20T12:20:46Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:417344358worker2 # [ 30.731066] nix-grpc-daemon[857]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 7 ms (attempt 1/5)4359worker2 # [ 30.738159] nix-grpc-daemon[857]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 125 ms (attempt 2/5)4360worker2 # [ 30.866156] nix-grpc-daemon[857]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 264 ms (attempt 3/5)4361worker2 # [ 31.133601] nix-grpc-daemon[857]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 540 ms (attempt 4/5)4362lb # [ 31.473818] envoy[1008]: conn peer=[2001:db8:1::1]:57346 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=715 tx=365 ms=13594363lb # [ 31.475765] envoy[1008]: conn peer=[2001:db8:1::1]:57362 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=2596 tx=364 ms=564364lb # [ 31.477531] envoy[1008]: conn peer=[2001:db8:1::1]:57372 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=423 tx=199 ms=214365lb # [ 31.479166] envoy[1008]: conn peer=[2001:db8:1::1]:57378 tls=TLSv1.3 subject="CN=stranger" sni=- flags=- tls_fail="-" rx=423 tx=181 ms=74366worker2 # [ 31.682296] nix-grpc-daemon[857]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4367worker2 # [ 31.713359] nix-grpc-daemon[857]: ts=2026-09-20T12:20:47Z level=debug event=want client=1 drv=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv cached=04368worker2 # [ 31.718643] nix-grpc-daemon[857]: ts=2026-09-20T12:20:47Z level=debug event=assigned drv=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv worker=192.168.1.3:50051 clients=14369worker1 # [ 31.780362] nix-grpc-daemon[786]: ts=2026-09-20T12:20:47Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:384104370worker1 # [ 31.802700] nix-grpc-daemon[786]: ts=2026-09-20T12:20:47Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:384104371client # copying 1 paths to 'lb:50051'...4372worker1 # [ 31.817888] nix-grpc-daemon[786]: ts=2026-09-20T12:20:47Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:384104373client # copying path '/nix/store/3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv' to 'lb:50051'...4374worker1 # [ 31.847653] nix-daemon[845]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4375worker1 # [ 31.873105] nix-grpc-daemon[786]: ts=2026-09-20T12:20:47Z level=info event=niks3_push_started pid=8554376worker1 # [ 32.110163] nix-grpc-daemon[855]: time=2026-09-20T12:20:47.347Z level=INFO msg="Configuring client TLS" cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/worker1.pem key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/worker1.key ca=""4377worker1 # [ 32.278409] nix-daemon[845]: accepted connection from pid 868, user nix-grpc-daemon (trusted)4378worker1 # [ 32.295400] nix-daemon[845]: reaped child process 871, status = succeeded4379worker1 # [ 32.436127] nix-daemon[845]: accepted connection from pid 873, user nix-grpc-daemon (trusted)4380worker1 # [ 32.449644] nix-daemon[845]: reaped child process 876, status = succeeded4381lb # [ 32.459562] niks3-server[1046]: 2026/09/20 12:20:48 INFO Received uploads request method=POST path=/api/pending_closures4382worker1 # [ 32.485861] nix-grpc-daemon[855]: time=2026-09-20T12:20:47.726Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4383worker1 # [ 32.487962] nix-grpc-daemon[855]: time=2026-09-20T12:20:47.728Z level=INFO msg="Uploading 3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv (504B)"4384lb # [ 32.524207] niks3-server[1046]: 2026/09/20 12:20:48 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign4385lb # [ 32.541214] niks3-server[1046]: 2026/09/20 12:20:48 INFO Registered completed upload object_key=nar/15vlbxqk65jh0v1qgc8308zz4diydc8wp113i0cc4ylv0zc5w964.nar.zst4386lb # [ 32.552064] niks3-server[1046]: 2026/09/20 12:20:48 INFO Signed narinfos id=1 count=14387worker1 # [ 32.555399] nix-grpc-daemon[855]: time=2026-09-20T12:20:47.795Z level=INFO msg="Uploading 1 narinfos"4388lb # [ 32.565969] niks3-server[1046]: 2026/09/20 12:20:48 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete4389lb # [ 32.569175] niks3-server[1046]: 2026/09/20 12:20:48 INFO Registered completed upload object_key=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1.ls4390lb # [ 32.577309] niks3-server[1046]: 2026/09/20 12:20:48 INFO Registered completed upload object_key=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1.narinfo4391lb # [ 32.583122] niks3-server[1046]: 2026/09/20 12:20:48 INFO Completed upload id=14392worker1 # [ 32.584678] nix-grpc-daemon[855]: time=2026-09-20T12:20:47.824Z level=INFO msg="Upload complete. (463ms)"4393worker1 # [ 32.590357] nix-grpc-daemon[786]: ts=2026-09-20T12:20:47Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38410 duration_s=0 paths=1 nar_bytes_in=5044394worker1 # [ 32.597575] nix-grpc-daemon[786]: ts=2026-09-20T12:20:47Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:384104395worker1 # [ 32.611128] nix-daemon[845]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4396worker1 # [ 32.618745] nix-daemon[845]: reaped child process 853, status = succeeded4397client # building '/nix/store/3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv' on lb:50051...4398client # farm-a> node-a: building 3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv4399worker1 # [ 32.629687] nix-daemon[845]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4400client # farm-a> LOG-farm-a-t14401worker1 # [ 33.854194] nix-daemon[845]: reaped child process 880, status = succeeded4402worker1 # [ 33.976581] nix-daemon[845]: accepted connection from pid 896, user nix-grpc-daemon (trusted)4403worker1 # [ 33.989984] nix-daemon[845]: reaped child process 899, status = succeeded4404lb # [ 33.997911] niks3-server[1046]: 2026/09/20 12:20:50 INFO Received uploads request method=POST path=/api/pending_closures4405worker1 # [ 34.005836] nix-grpc-daemon[855]: time=2026-09-20T12:20:49.246Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4406worker1 # [ 34.007640] nix-grpc-daemon[855]: time=2026-09-20T12:20:49.248Z level=INFO msg="Uploading qm96yd1cmricgf2hr21k88syh75l317v-farm-a (128B)"4407lb # [ 34.030358] niks3-server[1046]: 2026/09/20 12:20:50 INFO Registered completed upload object_key=log/3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv4408lb # [ 34.034877] niks3-server[1046]: 2026/09/20 12:20:50 INFO Registered completed upload object_key=nar/1rmil4ja9jzfwn6l5hwbir70b6plx99sgk0pnjjy377ngbc3vz28.nar.zst4409lb # [ 34.038255] niks3-server[1046]: 2026/09/20 12:20:50 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign4410lb # [ 34.040140] niks3-server[1046]: 2026/09/20 12:20:50 INFO Signed narinfos id=2 count=14411worker1 # [ 34.042459] nix-grpc-daemon[855]: time=2026-09-20T12:20:49.282Z level=INFO msg="Uploading 1 narinfos"4412lb # [ 34.044664] niks3-server[1046]: 2026/09/20 12:20:50 INFO Registered completed upload object_key=qm96yd1cmricgf2hr21k88syh75l317v.ls4413lb # [ 34.051554] niks3-server[1046]: 2026/09/20 12:20:50 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete4414lb # [ 34.054649] niks3-server[1046]: 2026/09/20 12:20:50 INFO Completed upload id=24415worker1 # [ 34.055721] nix-grpc-daemon[855]: time=2026-09-20T12:20:49.296Z level=INFO msg="Upload complete. (216ms)"4416lb # [ 34.056273] niks3-server[1046]: 2026/09/20 12:20:50 INFO Registered completed upload object_key=qm96yd1cmricgf2hr21k88syh75l317v.narinfo4417worker1 # [ 34.058939] nix-grpc-daemon[786]: ts=2026-09-20T12:20:49Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38410 duration_s=1 drv=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv assign_id=1 outputs=14418worker1 # [ 34.066052] nix-daemon[845]: reaped child process 878, status = succeeded4419worker2 # [ 34.058983] nix-grpc-daemon[857]: ts=2026-09-20T12:20:49Z level=debug event=want client=1 drv=zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv cached=04420worker2 # [ 34.060833] nix-grpc-daemon[857]: ts=2026-09-20T12:20:49Z level=debug event=assigned drv=zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv worker=192.168.1.3:50051 clients=14421worker1 # [ 34.116357] nix-grpc-daemon[786]: ts=2026-09-20T12:20:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:384104422worker1 # [ 34.123573] nix-grpc-daemon[786]: ts=2026-09-20T12:20:49Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:384104423worker1 # [ 34.130156] nix-grpc-daemon[855]: time=2026-09-20T12:20:49.370Z level=INFO msg="All 1 paths already cached"4424client # copying 1 paths to 'lb:50051'...4425worker1 # [ 34.134295] nix-grpc-daemon[786]: ts=2026-09-20T12:20:49Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:384104426client # copying path '/nix/store/zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv' to 'lb:50051'...4427worker1 # [ 34.144431] nix-daemon[845]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4428worker1 # [ 34.273266] nix-daemon[845]: accepted connection from pid 903, user nix-grpc-daemon (trusted)4429worker1 # [ 34.286799] nix-daemon[845]: reaped child process 906, status = succeeded4430worker1 # [ 34.416062] nix-daemon[845]: accepted connection from pid 908, user nix-grpc-daemon (trusted)4431worker1 # [ 34.428599] nix-daemon[845]: reaped child process 911, status = succeeded4432lb # [ 34.437613] niks3-server[1046]: 2026/09/20 12:20:50 INFO Received uploads request method=POST path=/api/pending_closures4433worker1 # [ 34.451670] nix-grpc-daemon[855]: time=2026-09-20T12:20:49.691Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4434worker1 # [ 34.453032] nix-grpc-daemon[855]: time=2026-09-20T12:20:49.691Z level=INFO msg="Uploading zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv (624B)"4435lb # [ 34.468643] niks3-server[1046]: 2026/09/20 12:20:50 INFO Registered completed upload object_key=nar/19sdpvl15ladfjss8gp73f6binhg0wgk0323wmn91mihvj530pxq.nar.zst4436lb # [ 34.480117] niks3-server[1046]: 2026/09/20 12:20:50 INFO Received sign narinfos request method=POST path=/api/pending_closures/3/sign4437lb # [ 34.486118] niks3-server[1046]: 2026/09/20 12:20:50 INFO Signed narinfos id=3 count=14438worker1 # [ 34.490275] nix-grpc-daemon[855]: time=2026-09-20T12:20:49.730Z level=INFO msg="Uploading 1 narinfos"4439lb # [ 34.492526] niks3-server[1046]: 2026/09/20 12:20:50 INFO Registered completed upload object_key=zbhcql1mkb05ig74hjj18iqpvqy3rfbp.ls4440lb # [ 34.502134] niks3-server[1046]: 2026/09/20 12:20:50 INFO Received complete upload request method=POST path=/api/pending_closures/3/complete4441lb # [ 34.510043] niks3-server[1046]: 2026/09/20 12:20:50 INFO Registered completed upload object_key=zbhcql1mkb05ig74hjj18iqpvqy3rfbp.narinfo4442lb # [ 34.513171] niks3-server[1046]: 2026/09/20 12:20:50 INFO Completed upload id=34443worker1 # [ 34.514225] nix-grpc-daemon[855]: time=2026-09-20T12:20:49.754Z level=INFO msg="Upload complete. (364ms)"4444worker1 # [ 34.517073] nix-grpc-daemon[786]: ts=2026-09-20T12:20:49Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38410 duration_s=0 paths=1 nar_bytes_in=6244445worker1 # [ 34.519510] nix-grpc-daemon[786]: ts=2026-09-20T12:20:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:384104446worker1 # [ 34.526302] nix-daemon[845]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4447worker1 # [ 34.530125] nix-daemon[845]: reaped child process 901, status = succeeded4448client # building '/nix/store/zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv' on lb:50051...4449worker1 # [ 34.534145] nix-daemon[845]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4450client # farm-b> node-a: building zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv4451client # farm-b> LOG-farm-b-t14452worker1 # [ 35.760681] nix-daemon[845]: reaped child process 915, status = succeeded4453worker1 # [ 35.914278] nix-daemon[845]: accepted connection from pid 930, user nix-grpc-daemon (trusted)4454worker1 # [ 35.927635] nix-daemon[845]: reaped child process 933, status = succeeded4455lb # [ 35.940652] niks3-server[1046]: 2026/09/20 12:20:52 INFO Received uploads request method=POST path=/api/pending_closures4456worker1 # [ 35.950618] nix-grpc-daemon[855]: time=2026-09-20T12:20:51.190Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4457worker1 # [ 35.952562] nix-grpc-daemon[855]: time=2026-09-20T12:20:51.192Z level=INFO msg="Uploading jicn175r0p4806ghnhxdvn7fsa1akkr0-farm-b (176B)"4458lb # [ 35.983096] niks3-server[1046]: 2026/09/20 12:20:52 INFO Received sign narinfos request method=POST path=/api/pending_closures/4/sign4459lb # [ 35.986850] niks3-server[1046]: 2026/09/20 12:20:52 INFO Signed narinfos id=4 count=14460worker1 # [ 35.991889] nix-grpc-daemon[855]: time=2026-09-20T12:20:51.232Z level=INFO msg="Uploading 1 narinfos"4461lb # [ 35.999740] niks3-server[1046]: 2026/09/20 12:20:52 INFO Registered completed upload object_key=nar/0n7l6rw4y681bymb8bwra2nh13s7ybxcm2l7y8iwy0k5mmnaznjg.nar.zst4462lb # [ 36.009149] niks3-server[1046]: 2026/09/20 12:20:52 INFO Registered completed upload object_key=log/zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv4463lb # [ 36.017347] niks3-server[1046]: 2026/09/20 12:20:52 INFO Received complete upload request method=POST path=/api/pending_closures/4/complete4464lb # [ 36.022282] niks3-server[1046]: 2026/09/20 12:20:52 INFO Registered completed upload object_key=jicn175r0p4806ghnhxdvn7fsa1akkr0.ls4465lb # [ 36.024162] niks3-server[1046]: 2026/09/20 12:20:52 INFO Registered completed upload object_key=jicn175r0p4806ghnhxdvn7fsa1akkr0.narinfo4466lb # [ 36.027702] niks3-server[1046]: 2026/09/20 12:20:52 INFO Completed upload id=44467worker1 # [ 36.029235] nix-grpc-daemon[855]: time=2026-09-20T12:20:51.269Z level=INFO msg="Upload complete. (286ms)"4468worker2 # [ 36.027952] nix-grpc-daemon[857]: ts=2026-09-20T12:20:51Z level=debug event=want client=1 drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv cached=04469worker1 # [ 36.035902] nix-grpc-daemon[786]: ts=2026-09-20T12:20:51Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38410 duration_s=1 drv=zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv assign_id=2 outputs=14470worker2 # [ 36.029753] nix-grpc-daemon[857]: ts=2026-09-20T12:20:51Z level=debug event=assigned drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv worker=192.168.1.3:50051 clients=14471worker1 # [ 36.050682] nix-daemon[845]: reaped child process 913, status = succeeded4472worker1 # [ 36.084339] nix-grpc-daemon[786]: ts=2026-09-20T12:20:51Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:384104473worker1 # [ 36.097695] nix-grpc-daemon[786]: ts=2026-09-20T12:20:51Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:384104474worker1 # [ 36.116913] nix-grpc-daemon[855]: time=2026-09-20T12:20:51.357Z level=INFO msg="All 2 paths already cached"4475client # copying 1 paths to 'lb:50051'...4476worker1 # [ 36.125599] nix-grpc-daemon[786]: ts=2026-09-20T12:20:51Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:384104477client # copying path '/nix/store/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv' to 'lb:50051'...4478worker1 # [ 36.136112] nix-daemon[845]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4479worker1 # [ 36.281359] nix-daemon[845]: accepted connection from pid 937, user nix-grpc-daemon (trusted)4480worker1 # [ 36.294808] nix-daemon[845]: reaped child process 940, status = succeeded4481worker1 # [ 36.439794] nix-daemon[845]: accepted connection from pid 942, user nix-grpc-daemon (trusted)4482worker1 # [ 36.452720] nix-daemon[845]: reaped child process 945, status = succeeded4483lb # [ 36.461384] niks3-server[1046]: 2026/09/20 12:20:52 INFO Received uploads request method=POST path=/api/pending_closures4484worker1 # [ 36.561369] nix-grpc-daemon[855]: time=2026-09-20T12:20:51.801Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4485worker1 # [ 36.563794] nix-grpc-daemon[855]: time=2026-09-20T12:20:51.803Z level=INFO msg="Uploading c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv (752B)"4486lb # [ 36.583180] niks3-server[1046]: 2026/09/20 12:20:52 INFO Registered completed upload object_key=nar/1nf01qbgzkrsqfjd6d2bx60xxkfkbrljmfrvypvjci1c69ad7knd.nar.zst4487lb # [ 36.593708] niks3-server[1046]: 2026/09/20 12:20:52 INFO Received sign narinfos request method=POST path=/api/pending_closures/5/sign4488lb # [ 36.602704] niks3-server[1046]: 2026/09/20 12:20:52 INFO Signed narinfos id=5 count=14489worker1 # [ 36.607108] nix-grpc-daemon[855]: time=2026-09-20T12:20:51.847Z level=INFO msg="Uploading 1 narinfos"4490lb # [ 36.606650] niks3-server[1046]: 2026/09/20 12:20:52 INFO Registered completed upload object_key=c7ysj984j3hnalhcvn7z0328xf5fr0zd.ls4491lb # [ 36.618567] niks3-server[1046]: 2026/09/20 12:20:52 INFO Received complete upload request method=POST path=/api/pending_closures/5/complete4492lb # [ 36.622499] niks3-server[1046]: 2026/09/20 12:20:52 INFO Completed upload id=54493worker1 # [ 36.625172] nix-grpc-daemon[855]: time=2026-09-20T12:20:51.865Z level=INFO msg="Upload complete. (482ms)"4494lb # [ 36.624296] niks3-server[1046]: 2026/09/20 12:20:52 INFO Registered completed upload object_key=c7ysj984j3hnalhcvn7z0328xf5fr0zd.narinfo4495worker1 # [ 36.627346] nix-grpc-daemon[786]: ts=2026-09-20T12:20:51Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38410 duration_s=0 paths=1 nar_bytes_in=7524496worker1 # [ 36.630689] nix-grpc-daemon[786]: ts=2026-09-20T12:20:51Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:384104497worker1 # [ 36.635583] nix-daemon[845]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4498worker1 # [ 36.642145] nix-daemon[845]: reaped child process 935, status = succeeded4499client # building '/nix/store/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv' on lb:50051...4500worker1 # [ 36.646094] nix-daemon[845]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4501client # farm-top> node-a: building c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv4502client # farm-top> LOG-farm-top-t14503worker1 # [ 37.670731] nix-daemon[845]: reaped child process 950, status = succeeded4504worker1 # [ 37.819965] nix-daemon[845]: accepted connection from pid 965, user nix-grpc-daemon (trusted)4505worker1 # [ 37.831675] nix-daemon[845]: reaped child process 968, status = succeeded4506lb # [ 37.839685] niks3-server[1046]: 2026/09/20 12:20:54 INFO Received uploads request method=POST path=/api/pending_closures4507worker1 # [ 37.849170] nix-grpc-daemon[855]: time=2026-09-20T12:20:53.089Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4508worker1 # [ 37.851053] nix-grpc-daemon[855]: time=2026-09-20T12:20:53.091Z level=INFO msg="Uploading 5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top (232B)"4509lb # [ 37.877704] niks3-server[1046]: 2026/09/20 12:20:54 INFO Registered completed upload object_key=nar/0pq8hp2qdfznpnf4pn4my9c4nd9yx6cfjn0x0bgv1rpk0536kpjy.nar.zst4510lb # [ 37.893970] niks3-server[1046]: 2026/09/20 12:20:54 INFO Received sign narinfos request method=POST path=/api/pending_closures/6/sign4511lb # [ 37.899414] niks3-server[1046]: 2026/09/20 12:20:54 INFO Signed narinfos id=6 count=14512worker1 # [ 37.903032] nix-grpc-daemon[855]: time=2026-09-20T12:20:53.143Z level=INFO msg="Uploading 1 narinfos"4513lb # [ 37.906184] niks3-server[1046]: 2026/09/20 12:20:54 INFO Registered completed upload object_key=log/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv4514lb # [ 37.909869] niks3-server[1046]: 2026/09/20 12:20:54 INFO Registered completed upload object_key=5qgbfb7awggfq3smlyfxgz8zixnqxk2n.ls4515lb # [ 37.919206] niks3-server[1046]: 2026/09/20 12:20:54 INFO Received complete upload request method=POST path=/api/pending_closures/6/complete4516lb # [ 37.922506] niks3-server[1046]: 2026/09/20 12:20:54 INFO Completed upload id=64517lb # [ 37.925101] niks3-server[1046]: 2026/09/20 12:20:54 INFO Registered completed upload object_key=5qgbfb7awggfq3smlyfxgz8zixnqxk2n.narinfo4518worker1 # [ 37.925082] nix-grpc-daemon[855]: time=2026-09-20T12:20:53.165Z level=INFO msg="Upload complete. (271ms)"4519worker1 # [ 37.931100] nix-grpc-daemon[786]: ts=2026-09-20T12:20:53Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38410 duration_s=1 drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv assign_id=3 outputs=14520worker1 # [ 37.947703] nix-daemon[845]: reaped child process 948, status = succeeded4521client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 >&2, in 7.69 seconds)4522client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 outPath4523client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 outPath, in 0.19 seconds)4524client: must fail: test -e /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4525client: (finished: must fail: test -e /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.02 seconds)4526client: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top && grep farm-top-t1 /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4527client # copying 3 paths...4528client # copying path '/nix/store/qm96yd1cmricgf2hr21k88syh75l317v-farm-a' from 'http://lb:5751'...4529client # copying path '/nix/store/jicn175r0p4806ghnhxdvn7fsa1akkr0-farm-b' from 'http://lb:5751'...4530client # copying path '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top' from 'http://lb:5751'...4531client: (finished: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top && grep farm-top-t1 /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.28 seconds)4532(finished: subtest: DAG build is scheduled across workers and lands in the cache, in 8.25 seconds)4533subtest: repeat is answered Cached by the scheduler without building4534worker1: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4535worker1 # finding garbage collector roots...4536worker1 # deleting specified paths...4537worker1 # deleting '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top'4538worker1 # deleting unused links...4539worker1 # note: hard linking is currently saving 0.0 KiB4540worker1: (finished: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.24 seconds)4541worker2: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4542worker2 # finding garbage collector roots...4543worker2 # deleting specified paths...4544worker2 # deleting '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top'4545worker2 # deleting unused links...4546worker2 # note: hard linking is currently saving 0.0 KiB4547worker2: (finished: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.27 seconds)4548client: must succeed: nix build -L --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 >&24549worker2 # [ 39.214167] nix-grpc-daemon[857]: ts=2026-09-20T12:20:54Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:417344550client # don't know how to build these paths:4551client # /nix/store/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv4552worker1 # [ 39.235205] nix-grpc-daemon[786]: ts=2026-09-20T12:20:54Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:384104553worker1 # [ 39.243309] nix-grpc-daemon[855]: time=2026-09-20T12:20:54.483Z level=INFO msg="All 2 paths already cached"4554worker2 # [ 39.242396] nix-grpc-daemon[857]: ts=2026-09-20T12:20:54Z level=debug event=want client=2 drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv cached=14555client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 >&2, in 0.26 seconds)4556client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 outPath4557client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 outPath, in 0.16 seconds)4558lb: must succeed: curl -sf localhost:9901/clusters4559lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)4560lb: must succeed: curl -sf localhost:9901/clusters4561lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4562worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4563worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)4564(finished: subtest: repeat is answered Cached by the scheduler without building, in 1.14 seconds)4565subtest: inputs already in the cache are not sent to the scheduler4566lb: must succeed: curl -sf localhost:9901/clusters4567lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4568lb: must succeed: curl -sf localhost:9901/clusters4569lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4570worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4571worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.03 seconds)4572client: must succeed: nix build -L --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 --argstr top t1b >&24573worker2 # [ 39.935423] nix-grpc-daemon[857]: ts=2026-09-20T12:20:55Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:417344574client # don't know how to build these paths:4575client # /nix/store/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv4576worker1 # [ 39.953635] nix-grpc-daemon[786]: ts=2026-09-20T12:20:55Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:384104577worker1 # [ 39.961562] nix-grpc-daemon[855]: time=2026-09-20T12:20:55.201Z level=INFO msg="All 2 paths already cached"4578worker2 # [ 39.959819] nix-grpc-daemon[857]: ts=2026-09-20T12:20:55Z level=debug event=want client=3 drv=y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv cached=04579worker2 # [ 39.961613] nix-grpc-daemon[857]: ts=2026-09-20T12:20:55Z level=debug event=assigned drv=y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv worker=192.168.1.3:50051 clients=14580worker1 # [ 40.015512] nix-grpc-daemon[786]: ts=2026-09-20T12:20:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:384104581worker1 # [ 40.027457] nix-grpc-daemon[786]: ts=2026-09-20T12:20:55Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:384104582worker1 # [ 40.040162] nix-grpc-daemon[855]: time=2026-09-20T12:20:55.280Z level=INFO msg="All 2 paths already cached"4583client # copying 1 paths to 'lb:50051'...4584client # copying path '/nix/store/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv' to 'lb:50051'...4585worker1 # [ 40.048515] nix-grpc-daemon[786]: ts=2026-09-20T12:20:55Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:384104586worker1 # [ 40.057894] nix-daemon[845]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4587worker1 # [ 40.240053] nix-daemon[845]: accepted connection from pid 986, user nix-grpc-daemon (trusted)4588worker1 # [ 40.254296] nix-daemon[845]: reaped child process 989, status = succeeded4589worker1 # [ 40.412419] nix-daemon[845]: accepted connection from pid 991, user nix-grpc-daemon (trusted)4590worker1 # [ 40.425036] nix-daemon[845]: reaped child process 994, status = succeeded4591lb # [ 40.439734] niks3-server[1046]: 2026/09/20 12:20:56 INFO Received uploads request method=POST path=/api/pending_closures4592worker1 # [ 40.446087] nix-grpc-daemon[855]: time=2026-09-20T12:20:55.686Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4593worker1 # [ 40.448178] nix-grpc-daemon[855]: time=2026-09-20T12:20:55.688Z level=INFO msg="Uploading y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv (752B)"4594worker1 # [ 40.460805] nix-grpc-daemon[855]: time=2026-09-20T12:20:55.701Z level=INFO msg="Uploading 1 narinfos"4595lb # [ 40.459623] niks3-server[1046]: 2026/09/20 12:20:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/7/sign4596lb # [ 40.461295] niks3-server[1046]: 2026/09/20 12:20:56 INFO Signed narinfos id=7 count=14597lb # [ 40.465360] niks3-server[1046]: 2026/09/20 12:20:56 INFO Registered completed upload object_key=nar/0p14b74h5654i3y6qv7zaaib0ya421b01knaq0m00r5ypjyh2q1d.nar.zst4598lb # [ 40.468348] niks3-server[1046]: 2026/09/20 12:20:56 INFO Registered completed upload object_key=y2wm8aizyyg449ia2pcscrjc2pklsrz4.ls4599lb # [ 40.472471] niks3-server[1046]: 2026/09/20 12:20:56 INFO Received complete upload request method=POST path=/api/pending_closures/7/complete4600lb # [ 40.476871] niks3-server[1046]: 2026/09/20 12:20:56 INFO Completed upload id=74601worker1 # [ 40.477945] nix-grpc-daemon[855]: time=2026-09-20T12:20:55.718Z level=INFO msg="Upload complete. (405ms)"4602lb # [ 40.478726] niks3-server[1046]: 2026/09/20 12:20:56 INFO Registered completed upload object_key=y2wm8aizyyg449ia2pcscrjc2pklsrz4.narinfo4603worker1 # [ 40.481213] nix-grpc-daemon[786]: ts=2026-09-20T12:20:55Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38410 duration_s=0 paths=1 nar_bytes_in=7524604worker1 # [ 40.484784] nix-grpc-daemon[786]: ts=2026-09-20T12:20:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:384104605worker1 # [ 40.491638] nix-daemon[845]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4606worker1 # [ 40.494279] nix-daemon[845]: reaped child process 984, status = succeeded4607client # building '/nix/store/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv' on lb:50051...4608client # farm-top> node-a: building y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv4609worker1 # [ 40.500464] nix-daemon[845]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4610lb # [ 41.474625] envoy[1008]: conn peer=[2001:db8:1::1]:57388 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9481 tx=3817 ms=74964611lb # [ 41.478794] envoy[1008]: conn peer=[2001:db8:1::1]:40922 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=991 tx=480 ms=394612client # farm-top> LOG-farm-top-t1b4613worker1 # [ 41.730707] nix-daemon[845]: reaped child process 998, status = succeeded4614worker1 # [ 41.846904] nix-daemon[845]: accepted connection from pid 1013, user nix-grpc-daemon (trusted)4615worker1 # [ 41.860688] nix-daemon[845]: reaped child process 1016, status = succeeded4616lb # [ 41.869783] niks3-server[1046]: 2026/09/20 12:20:58 INFO Received uploads request method=POST path=/api/pending_closures4617worker1 # [ 41.876650] nix-grpc-daemon[855]: time=2026-09-20T12:20:57.116Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4618worker1 # [ 41.878517] nix-grpc-daemon[855]: time=2026-09-20T12:20:57.118Z level=INFO msg="Uploading v688vnrkfcqiwaygym5ifmnr3yawh4xd-farm-top (232B)"4619lb # [ 41.895050] niks3-server[1046]: 2026/09/20 12:20:58 INFO Registered completed upload object_key=nar/1j36mvvk44wddjhqvk302zl80d251m1g0d4kc4gfqy00xyklwzbf.nar.zst4620lb # [ 41.899062] niks3-server[1046]: 2026/09/20 12:20:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/8/sign4621lb # [ 41.902048] niks3-server[1046]: 2026/09/20 12:20:58 INFO Signed narinfos id=8 count=14622worker1 # [ 41.904699] nix-grpc-daemon[855]: time=2026-09-20T12:20:57.145Z level=INFO msg="Uploading 1 narinfos"4623lb # [ 41.907091] niks3-server[1046]: 2026/09/20 12:20:58 INFO Registered completed upload object_key=log/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv4624lb # [ 41.909953] niks3-server[1046]: 2026/09/20 12:20:58 INFO Registered completed upload object_key=v688vnrkfcqiwaygym5ifmnr3yawh4xd.ls4625lb # [ 41.914196] niks3-server[1046]: 2026/09/20 12:20:58 INFO Received complete upload request method=POST path=/api/pending_closures/8/complete4626lb # [ 41.918089] niks3-server[1046]: 2026/09/20 12:20:58 INFO Completed upload id=84627worker1 # [ 41.919161] nix-grpc-daemon[855]: time=2026-09-20T12:20:57.159Z level=INFO msg="Upload complete. (209ms)"4628lb # [ 41.921262] niks3-server[1046]: 2026/09/20 12:20:58 INFO Registered completed upload object_key=v688vnrkfcqiwaygym5ifmnr3yawh4xd.narinfo4629worker1 # [ 41.921810] nix-grpc-daemon[786]: ts=2026-09-20T12:20:57Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38410 duration_s=1 drv=y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv assign_id=4 outputs=14630worker1 # [ 41.928716] nix-daemon[845]: reaped child process 996, status = succeeded4631client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 --argstr top t1b >&2, in 2.22 seconds)4632client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 --argstr top t1b outPath4633client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 --argstr top t1b outPath, in 0.17 seconds)4634lb: must succeed: curl -sf localhost:9901/clusters4635lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4636lb: must succeed: curl -sf localhost:9901/clusters4637lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4638worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4639worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.03 seconds)4640(finished: subtest: inputs already in the cache are not sent to the scheduler, in 2.62 seconds)4641subtest: two clients wanting the same drv build it once4642client: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup4643client # [ 42.313795] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup.4644client # Running as unit: dedup1.service; invocation ID: 535b14af95e442eb872ea226691930e74645client: (finished: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup, in 0.05 seconds)4646client: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup4647worker2 # [ 42.496485] nix-grpc-daemon[857]: ts=2026-09-20T12:20:57Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:417344648client # [ 42.502443] nix[960]: don't know how to build these paths:4649client # [ 42.503294] nix[960]: /nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv4650worker2 # [ 42.508039] nix-grpc-daemon[857]: ts=2026-09-20T12:20:57Z level=debug event=want client=4 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv cached=04651worker2 # [ 42.510391] nix-grpc-daemon[857]: ts=2026-09-20T12:20:57Z level=debug event=assigned drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv worker=192.168.1.4:50051 clients=14652worker2 # [ 42.512789] nix-grpc-daemon[857]: ts=2026-09-20T12:20:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:417344653worker2 # [ 42.526525] nix-grpc-daemon[857]: ts=2026-09-20T12:20:57Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:417344654client # [ 42.532107] nix[960]: copying 1 paths to 'lb:50051'...4655worker2 # [ 42.532727] nix-grpc-daemon[857]: ts=2026-09-20T12:20:57Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:417344656client # [ 42.533072] nix[960]: copying path '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' to 'lb:50051'...4657worker2 # [ 42.540694] nix-daemon[937]: accepted connection from pid 857, user nix-grpc-daemon (trusted)4658worker2 # [ 42.563065] nix-grpc-daemon[857]: ts=2026-09-20T12:20:57Z level=info event=niks3_push_started pid=9874659worker2 # [ 42.863896] nix-grpc-daemon[987]: time=2026-09-20T12:20:58.181Z level=INFO msg="Configuring client TLS" cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/worker2.pem key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/worker2.key ca=""4660worker2 # [ 43.071510] nix-daemon[937]: accepted connection from pid 1000, user nix-grpc-daemon (trusted)4661worker2 # [ 43.088396] nix-daemon[937]: reaped child process 1003, status = succeeded4662worker2 # [ 43.229266] nix-daemon[937]: accepted connection from pid 1005, user nix-grpc-daemon (trusted)4663worker2 # [ 43.242914] nix-daemon[937]: reaped child process 1008, status = succeeded4664lb # [ 43.269092] niks3-server[1046]: 2026/09/20 12:20:59 INFO Received uploads request method=POST path=/api/pending_closures4665worker2 # [ 43.265267] nix-grpc-daemon[987]: time=2026-09-20T12:20:58.586Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4666worker2 # [ 43.267492] nix-grpc-daemon[987]: time=2026-09-20T12:20:58.589Z level=INFO msg="Uploading k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv (432B)"4667lb # [ 43.314151] niks3-server[1046]: 2026/09/20 12:20:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/9/sign4668lb # [ 43.321091] niks3-server[1046]: 2026/09/20 12:20:59 INFO Registered completed upload object_key=nar/1rprin71wk1rn18ghpyxh5nm1akxfkjivwxfnlzqzmri5xdvl2l0.nar.zst4669lb # [ 43.331127] niks3-server[1046]: 2026/09/20 12:20:59 INFO Signed narinfos id=9 count=14670worker2 # [ 43.322896] nix-grpc-daemon[987]: time=2026-09-20T12:20:58.644Z level=INFO msg="Uploading 1 narinfos"4671lb # [ 43.338591] niks3-server[1046]: 2026/09/20 12:20:59 INFO Registered completed upload object_key=k75hvg0drf5n5cgm1w4hms82xb1l5nmz.ls4672lb # [ 43.347973] niks3-server[1046]: 2026/09/20 12:20:59 INFO Received complete upload request method=POST path=/api/pending_closures/9/complete4673lb # [ 43.352492] niks3-server[1046]: 2026/09/20 12:20:59 INFO Completed upload id=94674worker2 # [ 43.341681] nix-grpc-daemon[987]: time=2026-09-20T12:20:58.663Z level=INFO msg="Upload complete. (451ms)"4675lb # [ 43.354798] niks3-server[1046]: 2026/09/20 12:20:59 INFO Registered completed upload object_key=k75hvg0drf5n5cgm1w4hms82xb1l5nmz.narinfo4676worker2 # [ 43.344824] nix-grpc-daemon[857]: ts=2026-09-20T12:20:58Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41734 duration_s=0 paths=1 nar_bytes_in=4324677worker2 # [ 43.349141] nix-grpc-daemon[857]: ts=2026-09-20T12:20:58Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:417344678worker2 # [ 43.355594] nix-daemon[937]: accepted connection from pid 857, user nix-grpc-daemon (trusted)4679worker2 # [ 43.362811] nix-daemon[937]: reaped child process 985, status = succeeded4680worker2 # [ 43.368950] nix-daemon[937]: accepted connection from pid 857, user nix-grpc-daemon (trusted)4681client # [ 43.368320] nix[960]: building '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' on lb:50051...4682client # [ 43.389098] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup.4683client # Running as unit: dedup2.service; invocation ID: f0574802c6564e87bdd392959a48fd134684client: (finished: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup, in 1.09 seconds)4685worker2 # [ 43.654284] nix-grpc-daemon[857]: ts=2026-09-20T12:20:58Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:417344686client # [ 43.665911] nix[977]: this derivation will be built:4687worker1 # [ 43.679554] nix-grpc-daemon[786]: ts=2026-09-20T12:20:58Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:384104688worker1 # [ 43.686305] nix-grpc-daemon[786]: copying path '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' from 'http://lb:5751'...4689client # [ 43.686895] nix[977]: /nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv4690worker2 # [ 43.694858] nix-grpc-daemon[857]: ts=2026-09-20T12:20:59Z level=debug event=want client=5 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv cached=04691client # [ 43.701155] nix[977]: building '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' on lb:50051...4692worker2 # [ 43.701270] nix-grpc-daemon[857]: ts=2026-09-20T12:20:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:417344693client: must succeed: sleep 24694client: (finished: must succeed: sleep 2, in 2.04 seconds)4695client: waiting for success: ! systemctl is-active dedup1 dedup24696worker2 # [ 46.988415] nix-daemon[937]: reaped child process 1012, status = succeeded4697worker2 # [ 47.113968] nix-daemon[937]: accepted connection from pid 1043, user nix-grpc-daemon (trusted)4698worker2 # [ 47.127195] nix-daemon[937]: reaped child process 1046, status = succeeded4699lb # [ 47.155387] niks3-server[1046]: 2026/09/20 12:21:03 INFO Received uploads request method=POST path=/api/pending_closures4700worker2 # [ 47.152961] nix-grpc-daemon[987]: time=2026-09-20T12:21:02.474Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4701worker2 # [ 47.154676] nix-grpc-daemon[987]: time=2026-09-20T12:21:02.476Z level=INFO msg="Uploading pihyvyr6gd3qnm368ii42wvcpgsfv1di-slow-dedup (120B)"4702lb # [ 47.208829] niks3-server[1046]: 2026/09/20 12:21:03 INFO Registered completed upload object_key=nar/02xhr0h0sj1ji8m0s8995xzbbb8mhbqdpdl1ggx94yqqaykihvz7.nar.zst4703lb # [ 47.214341] niks3-server[1046]: 2026/09/20 12:21:03 INFO Registered completed upload object_key=log/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv4704lb # [ 47.221211] niks3-server[1046]: 2026/09/20 12:21:03 INFO Received sign narinfos request method=POST path=/api/pending_closures/10/sign4705lb # [ 47.226462] niks3-server[1046]: 2026/09/20 12:21:03 INFO Signed narinfos id=10 count=14706worker2 # [ 47.218585] nix-grpc-daemon[987]: time=2026-09-20T12:21:02.539Z level=INFO msg="Uploading 1 narinfos"4707lb # [ 47.229244] niks3-server[1046]: 2026/09/20 12:21:03 INFO Registered completed upload object_key=pihyvyr6gd3qnm368ii42wvcpgsfv1di.ls4708lb # [ 47.239673] niks3-server[1046]: 2026/09/20 12:21:03 INFO Received complete upload request method=POST path=/api/pending_closures/10/complete4709lb # [ 47.242512] niks3-server[1046]: 2026/09/20 12:21:03 INFO Completed upload id=104710lb # [ 47.244826] niks3-server[1046]: 2026/09/20 12:21:03 INFO Registered completed upload object_key=pihyvyr6gd3qnm368ii42wvcpgsfv1di.narinfo4711worker2 # [ 47.233006] nix-grpc-daemon[987]: time=2026-09-20T12:21:02.554Z level=INFO msg="Upload complete. (249ms)"4712worker2 # [ 47.239609] nix-grpc-daemon[857]: ts=2026-09-20T12:21:02Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41734 duration_s=3 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv assign_id=5 outputs=14713worker2 # [ 47.246154] nix-grpc-daemon[857]: ts=2026-09-20T12:21:02Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41734 duration_s=3 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv assign_id=5 outputs=14714worker2 # [ 47.256266] nix-daemon[937]: reaped child process 1010, status = succeeded4715client # [ 47.278177] systemd[1]: dedup2.service: Deactivated successfully.4716client # [ 47.281856] systemd[1]: dedup2.service: Consumed 135ms CPU time over 3.891s wall clock time, 31.2M memory peak, 8K incoming IP traffic, 6.7K outgoing IP traffic.4717client # [ 47.309040] systemd[1]: dedup1.service: Deactivated successfully.4718client # [ 47.310746] systemd[1]: dedup1.service: Consumed 124ms CPU time over 4.996s wall clock time, 37.3M memory peak, 8.6K incoming IP traffic, 9.1K outgoing IP traffic.4719client: (finished: waiting for success: ! systemctl is-active dedup1 dedup2, in 1.09 seconds)4720client: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }4721client: (finished: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }, in 0.04 seconds)4722worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true4723worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.03 seconds)4724worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true4725worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.04 seconds)4726worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4727worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)4728worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4729worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)4730(finished: subtest: two clients wanting the same drv build it once, in 5.83 seconds)4731subtest: build hook: nix-daemon with builders = grpc://lb4732client: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --print-out-paths --no-link -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag hook 2>/tmp/hook.log4733client # [ 48.288505] systemd[1]: Started Nix Daemon.4734client # [ 48.458518] nix-daemon[1018]: accepted connection from pid 1015, user root (trusted)4735worker1 # [ 48.661511] nix-grpc-daemon[786]: ts=2026-09-20T12:21:03Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:384104736worker2 # [ 48.656101] nix-grpc-daemon[857]: ts=2026-09-20T12:21:03Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:417344737worker2 # [ 48.667147] nix-grpc-daemon[857]: ts=2026-09-20T12:21:03Z level=debug event=want client=6 drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv cached=04738worker2 # [ 48.671226] nix-grpc-daemon[857]: ts=2026-09-20T12:21:03Z level=debug event=assigned drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv worker=192.168.1.3:50051 clients=14739worker1 # [ 48.722859] nix-grpc-daemon[786]: ts=2026-09-20T12:21:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:384104740worker1 # [ 48.734645] nix-grpc-daemon[786]: ts=2026-09-20T12:21:03Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:384104741worker1 # [ 48.745140] nix-grpc-daemon[786]: ts=2026-09-20T12:21:03Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:384104742worker1 # [ 48.763218] nix-daemon[845]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4743worker1 # [ 48.943122] nix-daemon[845]: accepted connection from pid 1043, user nix-grpc-daemon (trusted)4744worker1 # [ 48.958313] nix-daemon[845]: reaped child process 1046, status = succeeded4745worker1 # [ 49.105484] nix-daemon[845]: accepted connection from pid 1048, user nix-grpc-daemon (trusted)4746worker1 # [ 49.121767] nix-daemon[845]: reaped child process 1051, status = succeeded4747lb # [ 49.134086] niks3-server[1046]: 2026/09/20 12:21:05 INFO Received uploads request method=POST path=/api/pending_closures4748worker1 # [ 49.140230] nix-grpc-daemon[855]: time=2026-09-20T12:21:04.380Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4749worker1 # [ 49.141663] nix-grpc-daemon[855]: time=2026-09-20T12:21:04.380Z level=INFO msg="Uploading pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv (512B)"4750lb # [ 49.149504] niks3-server[1046]: 2026/09/20 12:21:05 INFO Registered completed upload object_key=nar/08mhnmxkz6k9gck9zisdh6jb3dqb53xqy0jak6cm9lpasdvr4sk7.nar.zst4751lb # [ 49.159129] niks3-server[1046]: 2026/09/20 12:21:05 INFO Received sign narinfos request method=POST path=/api/pending_closures/11/sign4752worker1 # [ 49.166297] nix-grpc-daemon[855]: time=2026-09-20T12:21:04.406Z level=INFO msg="Uploading 1 narinfos"4753lb # [ 49.165385] niks3-server[1046]: 2026/09/20 12:21:05 INFO Signed narinfos id=11 count=14754lb # [ 49.171533] niks3-server[1046]: 2026/09/20 12:21:05 INFO Registered completed upload object_key=pd51i21khb2xrd9paajf80pb96sw8zn0.ls4755lb # [ 49.177263] niks3-server[1046]: 2026/09/20 12:21:05 INFO Received complete upload request method=POST path=/api/pending_closures/11/complete4756lb # [ 49.182727] niks3-server[1046]: 2026/09/20 12:21:05 INFO Completed upload id=114757worker1 # [ 49.183874] nix-grpc-daemon[855]: time=2026-09-20T12:21:04.424Z level=INFO msg="Upload complete. (408ms)"4758lb # [ 49.184591] niks3-server[1046]: 2026/09/20 12:21:05 INFO Registered completed upload object_key=pd51i21khb2xrd9paajf80pb96sw8zn0.narinfo4759worker1 # [ 49.185959] nix-grpc-daemon[786]: ts=2026-09-20T12:21:04Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38410 duration_s=0 paths=1 nar_bytes_in=5124760worker1 # [ 49.190332] nix-grpc-daemon[786]: ts=2026-09-20T12:21:04Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:384104761worker1 # [ 49.196954] nix-daemon[845]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4762worker1 # [ 49.198270] nix-daemon[845]: reaped child process 1041, status = succeeded4763worker1 # [ 49.203751] nix-daemon[845]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4764worker1 # [ 50.017883] nix-daemon[845]: reaped child process 1055, status = succeeded4765worker1 # [ 50.170204] nix-daemon[845]: accepted connection from pid 1070, user nix-grpc-daemon (trusted)4766worker1 # [ 50.182550] nix-daemon[845]: reaped child process 1073, status = succeeded4767lb # [ 50.198612] niks3-server[1046]: 2026/09/20 12:21:06 INFO Received uploads request method=POST path=/api/pending_closures4768worker1 # [ 50.208520] nix-grpc-daemon[855]: time=2026-09-20T12:21:05.448Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4769worker1 # [ 50.210047] nix-grpc-daemon[855]: time=2026-09-20T12:21:05.450Z level=INFO msg="Uploading 6zgzdy628dcdg0dbc5qi7nisk08n4kp2-farm-a (128B)"4770lb # [ 50.237223] niks3-server[1046]: 2026/09/20 12:21:06 INFO Registered completed upload object_key=nar/0pbbxl4kxy7zl7x4lp43ywljwn82mxzna2wgpb1bz1z765nwp9ki.nar.zst4771lb # [ 50.252317] niks3-server[1046]: 2026/09/20 12:21:06 INFO Registered completed upload object_key=log/pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv4772lb # [ 50.260952] niks3-server[1046]: 2026/09/20 12:21:06 INFO Received sign narinfos request method=POST path=/api/pending_closures/12/sign4773worker1 # [ 50.268491] nix-grpc-daemon[855]: time=2026-09-20T12:21:05.508Z level=INFO msg="Uploading 1 narinfos"4774lb # [ 50.268109] niks3-server[1046]: 2026/09/20 12:21:06 INFO Signed narinfos id=12 count=14775lb # [ 50.274048] niks3-server[1046]: 2026/09/20 12:21:06 INFO Registered completed upload object_key=6zgzdy628dcdg0dbc5qi7nisk08n4kp2.ls4776lb # [ 50.284143] niks3-server[1046]: 2026/09/20 12:21:06 INFO Received complete upload request method=POST path=/api/pending_closures/12/complete4777lb # [ 50.287054] niks3-server[1046]: 2026/09/20 12:21:06 INFO Completed upload id=124778worker1 # [ 50.289493] nix-grpc-daemon[855]: time=2026-09-20T12:21:05.529Z level=INFO msg="Upload complete. (280ms)"4779lb # [ 50.289885] niks3-server[1046]: 2026/09/20 12:21:06 INFO Registered completed upload object_key=6zgzdy628dcdg0dbc5qi7nisk08n4kp2.narinfo4780worker1 # [ 50.293362] nix-grpc-daemon[786]: ts=2026-09-20T12:21:05Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38410 duration_s=1 drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv assign_id=6 outputs=14781worker1 # [ 50.299159] nix-daemon[845]: reaped child process 1053, status = succeeded4782worker2 # [ 50.327886] nix-grpc-daemon[857]: ts=2026-09-20T12:21:05Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:417344783worker2 # [ 50.339390] nix-grpc-daemon[857]: copying path '/nix/store/6zgzdy628dcdg0dbc5qi7nisk08n4kp2-farm-a' from 'http://lb:5751'...4784worker1 # [ 50.381329] nix-grpc-daemon[786]: ts=2026-09-20T12:21:05Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:384104785worker1 # [ 50.384064] nix-grpc-daemon[786]: ts=2026-09-20T12:21:05Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:38410 duration_s=0 paths=1 nar_bytes_out=1284786worker1 # [ 50.579834] nix-grpc-daemon[786]: ts=2026-09-20T12:21:05Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:384104787worker1 # [ 50.585758] nix-grpc-daemon[786]: ts=2026-09-20T12:21:05Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:384104788worker1 # [ 50.590808] nix-grpc-daemon[855]: time=2026-09-20T12:21:05.830Z level=INFO msg="All 1 paths already cached"4789worker2 # [ 50.590408] nix-grpc-daemon[857]: ts=2026-09-20T12:21:05Z level=debug event=want client=7 drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv cached=04790worker2 # [ 50.595594] nix-grpc-daemon[857]: ts=2026-09-20T12:21:05Z level=debug event=assigned drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv worker=192.168.1.3:50051 clients=14791worker1 # [ 50.610247] nix-grpc-daemon[786]: ts=2026-09-20T12:21:05Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:384104792worker1 # [ 50.623463] nix-grpc-daemon[786]: ts=2026-09-20T12:21:05Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:384104793worker1 # [ 50.639137] nix-grpc-daemon[855]: time=2026-09-20T12:21:05.879Z level=INFO msg="All 1 paths already cached"4794worker1 # [ 50.646218] nix-grpc-daemon[786]: ts=2026-09-20T12:21:05Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:384104795worker1 # [ 50.657251] nix-daemon[845]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4796worker1 # [ 50.842720] nix-daemon[845]: accepted connection from pid 1077, user nix-grpc-daemon (trusted)4797worker1 # [ 50.857287] nix-daemon[845]: reaped child process 1080, status = succeeded4798worker1 # [ 51.032422] nix-daemon[845]: accepted connection from pid 1082, user nix-grpc-daemon (trusted)4799worker1 # [ 51.046379] nix-daemon[845]: reaped child process 1085, status = succeeded4800lb # [ 51.053545] niks3-server[1046]: 2026/09/20 12:21:07 INFO Received uploads request method=POST path=/api/pending_closures4801worker1 # [ 51.059402] nix-grpc-daemon[855]: time=2026-09-20T12:21:06.299Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4802worker1 # [ 51.061086] nix-grpc-daemon[855]: time=2026-09-20T12:21:06.301Z level=INFO msg="Uploading vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv (624B)"4803lb # [ 51.073993] niks3-server[1046]: 2026/09/20 12:21:07 INFO Registered completed upload object_key=nar/1190412lxhybpj2b0886g97h0cpm7xx8r4virf8q4n7i91409ivg.nar.zst4804lb # [ 51.078308] niks3-server[1046]: 2026/09/20 12:21:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/13/sign4805lb # [ 51.081890] niks3-server[1046]: 2026/09/20 12:21:07 INFO Signed narinfos id=13 count=14806worker1 # [ 51.082788] nix-grpc-daemon[855]: time=2026-09-20T12:21:06.323Z level=INFO msg="Uploading 1 narinfos"4807lb # [ 51.084269] niks3-server[1046]: 2026/09/20 12:21:07 INFO Registered completed upload object_key=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8.ls4808lb # [ 51.090718] niks3-server[1046]: 2026/09/20 12:21:07 INFO Received complete upload request method=POST path=/api/pending_closures/13/complete4809lb # [ 51.093755] niks3-server[1046]: 2026/09/20 12:21:07 INFO Completed upload id=134810worker1 # [ 51.096228] nix-grpc-daemon[855]: time=2026-09-20T12:21:06.336Z level=INFO msg="Upload complete. (426ms)"4811lb # [ 51.096269] niks3-server[1046]: 2026/09/20 12:21:07 INFO Registered completed upload object_key=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8.narinfo4812worker1 # [ 51.098361] nix-grpc-daemon[786]: ts=2026-09-20T12:21:06Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38410 duration_s=0 paths=1 nar_bytes_in=6244813worker1 # [ 51.102510] nix-grpc-daemon[786]: ts=2026-09-20T12:21:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:384104814worker1 # [ 51.108260] nix-daemon[845]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4815worker1 # [ 51.112050] nix-daemon[845]: reaped child process 1075, status = succeeded4816worker1 # [ 51.117064] nix-daemon[845]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4817lb # [ 51.476199] envoy[1008]: conn peer=[2001:db8:1::1]:40938 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4139 tx=1710 ms=20254818lb # [ 51.479933] envoy[1008]: conn peer=[2001:db8:1::1]:33524 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1589 tx=1487 ms=36084819lb # [ 51.483491] envoy[1008]: conn peer=[2001:db8:1::1]:40954 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2985 tx=1372 ms=48044820lb # [ 51.487101] envoy[1008]: conn peer=[2001:db8:1::1]:33526 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3369 tx=1756 ms=17374821lb # [ 51.490702] envoy[1008]: conn peer=[2001:db8:1::1]:33532 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=409 tx=333 ms=174822worker1 # [ 51.928989] nix-daemon[845]: reaped child process 1089, status = succeeded4823worker1 # [ 52.051338] nix-daemon[845]: accepted connection from pid 1104, user nix-grpc-daemon (trusted)4824worker1 # [ 52.062094] nix-daemon[845]: reaped child process 1107, status = succeeded4825lb # [ 52.072658] niks3-server[1046]: 2026/09/20 12:21:08 INFO Received uploads request method=POST path=/api/pending_closures4826worker1 # [ 52.079474] nix-grpc-daemon[855]: time=2026-09-20T12:21:07.319Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4827worker1 # [ 52.081864] nix-grpc-daemon[855]: time=2026-09-20T12:21:07.321Z level=INFO msg="Uploading mrrcl9jb4hh6rly2df2vfym5sq1235hf-farm-b (176B)"4828lb # [ 52.100142] niks3-server[1046]: 2026/09/20 12:21:08 INFO Registered completed upload object_key=log/vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv4829lb # [ 52.108085] niks3-server[1046]: 2026/09/20 12:21:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/14/sign4830worker1 # [ 52.112870] nix-grpc-daemon[855]: time=2026-09-20T12:21:07.353Z level=INFO msg="Uploading 1 narinfos"4831lb # [ 52.112764] niks3-server[1046]: 2026/09/20 12:21:08 INFO Signed narinfos id=14 count=14832lb # [ 52.117370] niks3-server[1046]: 2026/09/20 12:21:08 INFO Registered completed upload object_key=nar/1mgm44m4ndgi5l3yi2qv3v3sx48190m8d4ycw3r4pfzy4z1wq2s6.nar.zst4833lb # [ 52.124606] niks3-server[1046]: 2026/09/20 12:21:08 INFO Received complete upload request method=POST path=/api/pending_closures/14/complete4834lb # [ 52.128044] niks3-server[1046]: 2026/09/20 12:21:08 INFO Registered completed upload object_key=mrrcl9jb4hh6rly2df2vfym5sq1235hf.ls4835lb # [ 52.141965] niks3-server[1046]: 2026/09/20 12:21:08 INFO Completed upload id=144836worker1 # [ 52.142997] nix-grpc-daemon[855]: time=2026-09-20T12:21:07.383Z level=INFO msg="Upload complete. (220ms)"4837worker1 # [ 52.145530] nix-grpc-daemon[786]: ts=2026-09-20T12:21:07Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38410 duration_s=1 drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv assign_id=7 outputs=14838worker1 # [ 52.151483] nix-grpc-daemon[786]: ts=2026-09-20T12:21:07Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:384104839lb # [ 52.152062] niks3-server[1046]: 2026/09/20 12:21:08 INFO Registered completed upload object_key=mrrcl9jb4hh6rly2df2vfym5sq1235hf.narinfo4840worker1 # [ 52.154418] nix-daemon[845]: reaped child process 1087, status = succeeded4841worker1 # [ 52.165772] nix-grpc-daemon[786]: ts=2026-09-20T12:21:07Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:384104842worker1 # [ 52.167668] nix-grpc-daemon[786]: ts=2026-09-20T12:21:07Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:38410 duration_s=0 paths=1 nar_bytes_out=1764843worker1 # [ 52.435209] nix-grpc-daemon[786]: ts=2026-09-20T12:21:07Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:384104844worker2 # [ 52.429400] nix-grpc-daemon[857]: ts=2026-09-20T12:21:07Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:417344845worker2 # [ 52.484556] nix-grpc-daemon[987]: time=2026-09-20T12:21:07.805Z level=INFO msg="All 1 paths already cached"4846worker2 # [ 52.502269] nix-grpc-daemon[857]: ts=2026-09-20T12:21:07Z level=debug event=want client=8 drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv cached=04847worker1 # [ 52.521955] nix-grpc-daemon[786]: ts=2026-09-20T12:21:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:384104848worker2 # [ 52.507330] nix-grpc-daemon[857]: ts=2026-09-20T12:21:07Z level=debug event=assigned drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv worker=192.168.1.3:50051 clients=14849worker1 # [ 52.530830] nix-grpc-daemon[786]: ts=2026-09-20T12:21:07Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:384104850worker1 # [ 52.538644] nix-grpc-daemon[855]: time=2026-09-20T12:21:07.779Z level=INFO msg="All 2 paths already cached"4851worker1 # [ 52.542984] nix-grpc-daemon[786]: ts=2026-09-20T12:21:07Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:384104852worker1 # [ 52.548630] nix-daemon[845]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4853worker1 # [ 52.710275] nix-daemon[845]: accepted connection from pid 1111, user nix-grpc-daemon (trusted)4854worker1 # [ 52.723448] nix-daemon[845]: reaped child process 1114, status = succeeded4855worker1 # [ 52.875786] nix-daemon[845]: accepted connection from pid 1116, user nix-grpc-daemon (trusted)4856worker1 # [ 52.888423] nix-daemon[845]: reaped child process 1119, status = succeeded4857lb # [ 52.896828] niks3-server[1046]: 2026/09/20 12:21:09 INFO Received uploads request method=POST path=/api/pending_closures4858worker1 # [ 52.907488] nix-grpc-daemon[855]: time=2026-09-20T12:21:08.147Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4859worker1 # [ 52.908847] nix-grpc-daemon[855]: time=2026-09-20T12:21:08.147Z level=INFO msg="Uploading bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv (752B)"4860lb # [ 52.926069] niks3-server[1046]: 2026/09/20 12:21:09 INFO Registered completed upload object_key=nar/0d4r016fc570pq02p1j53gahxygvvyr62937d84sa58ri3rzk000.nar.zst4861lb # [ 52.933934] niks3-server[1046]: 2026/09/20 12:21:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/15/sign4862lb # [ 52.938358] niks3-server[1046]: 2026/09/20 12:21:09 INFO Signed narinfos id=15 count=14863worker1 # [ 52.942632] nix-grpc-daemon[855]: time=2026-09-20T12:21:08.182Z level=INFO msg="Uploading 1 narinfos"4864lb # [ 52.949930] niks3-server[1046]: 2026/09/20 12:21:09 INFO Registered completed upload object_key=bshr7sr3z72cf1fa1a1b8fifr45jgpsw.ls4865lb # [ 52.957071] niks3-server[1046]: 2026/09/20 12:21:09 INFO Received complete upload request method=POST path=/api/pending_closures/15/complete4866lb # [ 52.962934] niks3-server[1046]: 2026/09/20 12:21:09 INFO Completed upload id=154867worker1 # [ 52.966242] nix-grpc-daemon[855]: time=2026-09-20T12:21:08.206Z level=INFO msg="Upload complete. (412ms)"4868worker1 # [ 52.968538] nix-grpc-daemon[786]: ts=2026-09-20T12:21:08Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38410 duration_s=0 paths=1 nar_bytes_in=7524869worker1 # [ 52.971505] nix-grpc-daemon[786]: ts=2026-09-20T12:21:08Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:384104870worker1 # [ 52.976594] nix-daemon[845]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4871lb # [ 52.977612] niks3-server[1046]: 2026/09/20 12:21:09 INFO Registered completed upload object_key=bshr7sr3z72cf1fa1a1b8fifr45jgpsw.narinfo4872worker1 # [ 52.981587] nix-daemon[845]: reaped child process 1109, status = succeeded4873worker1 # [ 52.986873] nix-daemon[845]: accepted connection from pid 786, user nix-grpc-daemon (trusted)4874worker1 # [ 54.214234] nix-daemon[845]: reaped child process 1123, status = succeeded4875worker1 # [ 54.353206] nix-daemon[845]: accepted connection from pid 1138, user nix-grpc-daemon (trusted)4876worker1 # [ 54.366085] nix-daemon[845]: reaped child process 1141, status = succeeded4877lb # [ 54.379529] niks3-server[1046]: 2026/09/20 12:21:10 INFO Received uploads request method=POST path=/api/pending_closures4878worker1 # [ 54.384410] nix-grpc-daemon[855]: time=2026-09-20T12:21:09.624Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4879worker1 # [ 54.386170] nix-grpc-daemon[855]: time=2026-09-20T12:21:09.626Z level=INFO msg="Uploading aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top (232B)"4880lb # [ 54.396177] niks3-server[1046]: 2026/09/20 12:21:10 INFO Registered completed upload object_key=nar/1r64sjcvz2vpjvwv3psvwf0bfifdzwiafvy1m89v5is1mm18ds87.nar.zst4881lb # [ 54.402870] niks3-server[1046]: 2026/09/20 12:21:10 INFO Received sign narinfos request method=POST path=/api/pending_closures/16/sign4882lb # [ 54.405588] niks3-server[1046]: 2026/09/20 12:21:10 INFO Signed narinfos id=16 count=14883worker1 # [ 54.406673] nix-grpc-daemon[855]: time=2026-09-20T12:21:09.647Z level=INFO msg="Uploading 1 narinfos"4884lb # [ 54.408104] niks3-server[1046]: 2026/09/20 12:21:10 INFO Registered completed upload object_key=log/bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv4885lb # [ 54.410328] niks3-server[1046]: 2026/09/20 12:21:10 INFO Registered completed upload object_key=aayfavl0hksy8qs9qc49znrwnfc5d2sr.ls4886lb # [ 54.415283] niks3-server[1046]: 2026/09/20 12:21:10 INFO Received complete upload request method=POST path=/api/pending_closures/16/complete4887lb # [ 54.418524] niks3-server[1046]: 2026/09/20 12:21:10 INFO Completed upload id=164888worker1 # [ 54.420808] nix-grpc-daemon[855]: time=2026-09-20T12:21:09.661Z level=INFO msg="Upload complete. (226ms)"4889lb # [ 54.420266] niks3-server[1046]: 2026/09/20 12:21:10 INFO Registered completed upload object_key=aayfavl0hksy8qs9qc49znrwnfc5d2sr.narinfo4890worker1 # [ 54.423686] nix-grpc-daemon[786]: ts=2026-09-20T12:21:09Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38410 duration_s=1 drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv assign_id=8 outputs=14891worker1 # [ 54.428708] nix-grpc-daemon[786]: ts=2026-09-20T12:21:09Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:384104892worker1 # [ 54.432632] nix-daemon[845]: reaped child process 1121, status = succeeded4893worker2 # [ 54.434801] nix-grpc-daemon[857]: ts=2026-09-20T12:21:09Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:417344894worker2 # [ 54.445261] nix-grpc-daemon[857]: copying path '/nix/store/mrrcl9jb4hh6rly2df2vfym5sq1235hf-farm-b' from 'http://lb:5751'...4895worker2 # [ 54.455707] nix-grpc-daemon[857]: copying path '/nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top' from 'http://lb:5751'...4896worker2 # [ 54.466571] nix-grpc-daemon[857]: ts=2026-09-20T12:21:09Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:41734 duration_s=0 paths=1 nar_bytes_out=2324897client # [ 54.583789] nix-daemon[1018]: reaped child process 1020, status = succeeded4898client: (finished: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --print-out-paths --no-link -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag hook 2>/tmp/hook.log, in 6.50 seconds)4899client: must succeed: grep farm-top-hook /nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top4900client: (finished: must succeed: grep farm-top-hook /nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top, in 0.03 seconds)4901client: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }4902client: (finished: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }, in 0.04 seconds)4903client: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }4904client: (finished: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }, in 0.03 seconds)4905(finished: subtest: build hook: nix-daemon with builders = grpc://lb, in 6.59 seconds)4906subtest: an input only one worker has still reaches the builder4907client: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A input --argstr tag w1only4908client # this derivation will be built:4909client # /nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv4910client # building '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv'...4911client: (finished: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A input --argstr tag w1only, in 0.40 seconds)4912client: must succeed: nix-store --export /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only > /tmp/shared/inp.closure4913client: (finished: must succeed: nix-store --export /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only > /tmp/shared/inp.closure, in 0.17 seconds)4914worker1: must succeed: nix-store --import < /tmp/shared/inp.closure4915worker1: (finished: must succeed: nix-store --import < /tmp/shared/inp.closure, in 0.15 seconds)4916worker2: must fail: test -e /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only4917worker2: (finished: must fail: test -e /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only, in 0.01 seconds)4918client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt a >&24919client # [ 55.557236] nix-daemon[1018]: accepted connection from pid 1101, user root (trusted)4920client # this derivation will be built:4921client # /nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv4922client # building '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' on 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key&system=x86_64-linux'...4923worker2 # [ 55.904714] nix-grpc-daemon[857]: ts=2026-09-20T12:21:11Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:417344924worker2 # [ 55.910412] nix-grpc-daemon[857]: ts=2026-09-20T12:21:11Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:417344925client # copying 1 paths...4926client # copying path '/nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only' to 'grpc://lb:50051'...4927worker2 # [ 55.921668] nix-grpc-daemon[857]: ts=2026-09-20T12:21:11Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:417344928worker2 # [ 55.932148] nix-daemon[937]: accepted connection from pid 857, user nix-grpc-daemon (trusted)4929worker2 # [ 56.089629] nix-daemon[937]: accepted connection from pid 1071, user nix-grpc-daemon (trusted)4930worker2 # [ 56.107752] nix-daemon[937]: reaped child process 1074, status = succeeded4931lb # [ 56.132129] niks3-server[1046]: 2026/09/20 12:21:12 INFO Received uploads request method=POST path=/api/pending_closures4932worker2 # [ 56.231515] nix-grpc-daemon[987]: time=2026-09-20T12:21:11.552Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4933worker2 # [ 56.237379] nix-grpc-daemon[987]: time=2026-09-20T12:21:11.556Z level=INFO msg="Uploading gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only (120B)"4934lb # [ 56.258839] niks3-server[1046]: 2026/09/20 12:21:12 INFO Received sign narinfos request method=POST path=/api/pending_closures/17/sign4935lb # [ 56.260930] niks3-server[1046]: 2026/09/20 12:21:12 INFO Registered completed upload object_key=nar/08ghl87f009pwqz85nbqv9hm0ln6gai6ggzilfmw3qnkaalfi6ss.nar.zst4936lb # [ 56.262859] niks3-server[1046]: 2026/09/20 12:21:12 INFO Signed narinfos id=17 count=14937worker2 # [ 56.253993] nix-grpc-daemon[987]: time=2026-09-20T12:21:11.574Z level=INFO msg="Uploading 1 narinfos"4938lb # [ 56.269251] niks3-server[1046]: 2026/09/20 12:21:12 INFO Received complete upload request method=POST path=/api/pending_closures/17/complete4939lb # [ 56.301667] niks3-server[1046]: 2026/09/20 12:21:12 INFO Registered completed upload object_key=gycxa7jajm34fqsmi3xn0zjr3pvaya0x.ls4940lb # [ 56.305667] niks3-server[1046]: 2026/09/20 12:21:12 INFO Completed upload id=174941lb # [ 56.308148] niks3-server[1046]: 2026/09/20 12:21:12 INFO Registered completed upload object_key=gycxa7jajm34fqsmi3xn0zjr3pvaya0x.narinfo4942worker2 # [ 56.295151] nix-grpc-daemon[987]: time=2026-09-20T12:21:11.616Z level=INFO msg="Upload complete. (351ms)"4943worker2 # [ 56.300554] nix-grpc-daemon[857]: ts=2026-09-20T12:21:11Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41734 duration_s=0 paths=1 nar_bytes_in=1204944worker2 # [ 56.314588] nix-grpc-daemon[857]: ts=2026-09-20T12:21:11Z level=debug event=want client=9 drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv cached=04945worker2 # [ 56.318886] nix-grpc-daemon[857]: ts=2026-09-20T12:21:11Z level=debug event=assigned drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv worker=192.168.1.4:50051 clients=14946worker2 # [ 56.331608] nix-daemon[937]: reaped child process 1069, status = succeeded4947worker2 # [ 56.357057] nix-grpc-daemon[857]: ts=2026-09-20T12:21:11Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:417344948worker2 # [ 56.366081] nix-grpc-daemon[857]: ts=2026-09-20T12:21:11Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:417344949client # copying 2 paths to 'lb:50051'...4950worker2 # [ 56.378348] nix-grpc-daemon[857]: ts=2026-09-20T12:21:11Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:417344951client # copying path '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv' to 'lb:50051'...4952client # copying path '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' to 'lb:50051'...4953worker2 # [ 56.388072] nix-daemon[937]: accepted connection from pid 857, user nix-grpc-daemon (trusted)4954worker2 # [ 56.502485] nix-daemon[937]: accepted connection from pid 1080, user nix-grpc-daemon (trusted)4955worker2 # [ 56.515990] nix-daemon[937]: reaped child process 1083, status = succeeded4956worker2 # [ 56.625507] nix-daemon[937]: accepted connection from pid 1086, user nix-grpc-daemon (trusted)4957worker2 # [ 56.638195] nix-daemon[937]: reaped child process 1089, status = succeeded4958lb # [ 56.656384] niks3-server[1046]: 2026/09/20 12:21:12 INFO Received uploads request method=POST path=/api/pending_closures4959lb # [ 56.660870] niks3-server[1046]: 2026/09/20 12:21:12 INFO Received uploads request method=POST path=/api/pending_closures4960worker2 # [ 56.654298] nix-grpc-daemon[987]: time=2026-09-20T12:21:11.975Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"4961worker2 # [ 56.655639] nix-grpc-daemon[987]: time=2026-09-20T12:21:11.975Z level=INFO msg="Uploading s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv (416B)"4962worker2 # [ 56.657202] nix-grpc-daemon[987]: time=2026-09-20T12:21:11.975Z level=INFO msg="Uploading k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv (560B)"4963lb # [ 56.676985] niks3-server[1046]: 2026/09/20 12:21:12 INFO Registered completed upload object_key=nar/136ms4k64sfvy4m17vvd9w9yncq4zydv4cjbhchdcg4j7ccbx3b9.nar.zst4964lb # [ 56.680161] niks3-server[1046]: 2026/09/20 12:21:12 INFO Registered completed upload object_key=nar/0s7awvvc5phwxzwwzhl47kzq38pdgwj6jp3kwxf5w2xddnxnkl6h.nar.zst4965lb # [ 56.687397] niks3-server[1046]: 2026/09/20 12:21:12 INFO Received sign narinfos request method=POST path=/api/pending_closures/18/sign4966lb # [ 56.690308] niks3-server[1046]: 2026/09/20 12:21:12 INFO Signed narinfos id=18 count=14967lb # [ 56.691493] niks3-server[1046]: 2026/09/20 12:21:12 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.ls4968lb # [ 56.693813] niks3-server[1046]: 2026/09/20 12:21:12 INFO Received sign narinfos request method=POST path=/api/pending_closures/19/sign4969lb # [ 56.696119] niks3-server[1046]: 2026/09/20 12:21:12 INFO Signed narinfos id=19 count=24970worker2 # [ 56.685705] nix-grpc-daemon[987]: time=2026-09-20T12:21:12.006Z level=INFO msg="Uploading 3 narinfos"4971lb # [ 56.697742] niks3-server[1046]: 2026/09/20 12:21:12 INFO Registered completed upload object_key=k9va03aiirjmgih65p90fpa9mr396d2b.ls4972lb # [ 56.710327] niks3-server[1046]: 2026/09/20 12:21:12 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.narinfo4973lb # [ 56.713576] niks3-server[1046]: 2026/09/20 12:21:12 INFO Received complete upload request method=POST path=/api/pending_closures/19/complete4974lb # [ 56.717394] niks3-server[1046]: 2026/09/20 12:21:12 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.narinfo4975lb # [ 56.719711] niks3-server[1046]: 2026/09/20 12:21:12 INFO Completed upload id=194976lb # [ 56.721380] niks3-server[1046]: 2026/09/20 12:21:12 INFO Received complete upload request method=POST path=/api/pending_closures/18/complete4977lb # [ 56.723389] niks3-server[1046]: 2026/09/20 12:21:12 INFO Registered completed upload object_key=k9va03aiirjmgih65p90fpa9mr396d2b.narinfo4978lb # [ 56.726069] niks3-server[1046]: 2026/09/20 12:21:12 INFO Completed upload id=184979worker2 # [ 56.715085] nix-grpc-daemon[987]: time=2026-09-20T12:21:12.036Z level=INFO msg="Upload complete. (320ms)"4980worker2 # [ 56.717120] nix-grpc-daemon[857]: ts=2026-09-20T12:21:12Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41734 duration_s=0 paths=2 nar_bytes_in=9764981worker2 # [ 56.720042] nix-grpc-daemon[857]: ts=2026-09-20T12:21:12Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:417344982worker2 # [ 56.724536] nix-daemon[937]: accepted connection from pid 857, user nix-grpc-daemon (trusted)4983worker2 # [ 56.732189] nix-daemon[937]: reaped child process 1078, status = succeeded4984worker2 # [ 56.735399] nix-daemon[937]: accepted connection from pid 857, user nix-grpc-daemon (trusted)4985client # building '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' on lb:50051...4986client # uses-input-w1onlya> worker2: building k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv4987worker2 # [ 58.150486] nix-daemon[937]: reaped child process 1093, status = succeeded4988worker2 # [ 58.289817] nix-daemon[937]: accepted connection from pid 1108, user nix-grpc-daemon (trusted)4989worker2 # [ 58.301424] nix-daemon[937]: reaped child process 1111, status = succeeded4990lb # [ 58.327612] niks3-server[1046]: 2026/09/20 12:21:14 INFO Received uploads request method=POST path=/api/pending_closures4991worker2 # [ 58.331751] nix-grpc-daemon[987]: time=2026-09-20T12:21:13.653Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"4992lb # [ 58.364190] niks3-server[1046]: 2026/09/20 12:21:14 INFO Received sign narinfos request method=POST path=/api/pending_closures/20/sign4993lb # [ 58.369242] niks3-server[1046]: 2026/09/20 12:21:14 INFO Signed narinfos id=20 count=14994worker2 # [ 58.361350] nix-grpc-daemon[987]: time=2026-09-20T12:21:13.682Z level=INFO msg="Uploading 1 narinfos"4995lb # [ 58.376759] niks3-server[1046]: 2026/09/20 12:21:14 INFO Registered completed upload object_key=log/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv4996lb # [ 58.389208] niks3-server[1046]: 2026/09/20 12:21:14 INFO Registered completed upload object_key=7fihpafywah8yif4mhbmz6i2i6adiqwj.ls4997lb # [ 58.396779] niks3-server[1046]: 2026/09/20 12:21:14 INFO Received complete upload request method=POST path=/api/pending_closures/20/complete4998worker2 # [ 58.391415] nix-grpc-daemon[987]: time=2026-09-20T12:21:13.713Z level=INFO msg="Upload complete. (248ms)"4999lb # [ 58.402339] niks3-server[1046]: 2026/09/20 12:21:14 INFO Completed upload id=205000worker2 # [ 58.394787] nix-grpc-daemon[857]: ts=2026-09-20T12:21:13Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41734 duration_s=1 drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv assign_id=9 outputs=15001lb # [ 58.406790] niks3-server[1046]: 2026/09/20 12:21:14 INFO Registered completed upload object_key=7fihpafywah8yif4mhbmz6i2i6adiqwj.narinfo5002worker2 # [ 58.401680] nix-daemon[937]: reaped child process 1091, status = succeeded5003client # copying 1 paths...5004worker2 # [ 58.438493] nix-grpc-daemon[857]: ts=2026-09-20T12:21:13Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:417345005client # copying path '/nix/store/7fihpafywah8yif4mhbmz6i2i6adiqwj-uses-input-w1onlya' from 'grpc://lb:50051'...5006worker2 # [ 58.453510] nix-grpc-daemon[857]: ts=2026-09-20T12:21:13Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:417345007worker2 # [ 58.455508] nix-grpc-daemon[857]: ts=2026-09-20T12:21:13Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:41734 duration_s=0 paths=1 nar_bytes_out=1205008client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt a >&2, in 3.14 seconds)5009client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt b >&25010client # [ 58.561295] nix-daemon[1018]: reaped child process 1104, status = succeeded5011client # [ 58.707725] nix-daemon[1018]: accepted connection from pid 1123, user root (trusted)5012client # this derivation will be built:5013client # /nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5014worker1 # [ 58.876042] nix-grpc-daemon[786]: ts=2026-09-20T12:21:14Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:384105015worker2 # [ 58.866390] nix-grpc-daemon[857]: ts=2026-09-20T12:21:14Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:417345016client # building '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' on 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key&system=x86_64-linux'...5017worker2 # [ 58.873373] nix-grpc-daemon[987]: time=2026-09-20T12:21:14.195Z level=INFO msg="All 1 paths already cached"5018worker2 # [ 58.884876] nix-grpc-daemon[857]: ts=2026-09-20T12:21:14Z level=debug event=want client=10 drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv cached=05019worker2 # [ 58.886789] nix-grpc-daemon[857]: ts=2026-09-20T12:21:14Z level=debug event=assigned drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv worker=192.168.1.3:50051 clients=15020worker1 # [ 58.903396] nix-grpc-daemon[786]: ts=2026-09-20T12:21:14Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:384105021worker1 # [ 58.914348] nix-grpc-daemon[786]: ts=2026-09-20T12:21:14Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:384105022client # copying 1 paths to 'lb:50051'...5023client # copying path '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' to 'lb:50051'...5024worker1 # [ 58.924872] nix-grpc-daemon[786]: ts=2026-09-20T12:21:14Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:384105025worker1 # [ 58.929575] nix-daemon[845]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5026worker1 # [ 58.940674] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 30 ms (attempt 1/5)5027worker1 # [ 58.973352] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 164 ms (attempt 2/5)5028worker1 # [ 59.140686] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 104 ms (attempt 3/5)5029worker1 # [ 59.251212] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 157 ms (attempt 4/5)5030worker1 # [ 59.415300] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5031worker1 # [ 59.420793] nix-grpc-daemon[786]: copying path '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv' from 'http://lb:5751'...5032worker1 # [ 59.599203] nix-daemon[845]: accepted connection from pid 1154, user nix-grpc-daemon (trusted)5033worker1 # [ 59.616845] nix-daemon[845]: reaped child process 1157, status = succeeded5034worker1 # [ 59.767694] nix-daemon[845]: accepted connection from pid 1159, user nix-grpc-daemon (trusted)5035worker1 # [ 59.784540] nix-daemon[845]: reaped child process 1162, status = succeeded5036lb # [ 59.798839] niks3-server[1046]: 2026/09/20 12:21:15 INFO Received uploads request method=POST path=/api/pending_closures5037worker1 # [ 60.026380] nix-grpc-daemon[855]: time=2026-09-20T12:21:15.265Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5038worker1 # [ 60.029904] nix-grpc-daemon[855]: time=2026-09-20T12:21:15.265Z level=INFO msg="Uploading 2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv (560B)"5039lb # [ 60.047972] niks3-server[1046]: 2026/09/20 12:21:16 INFO Received sign narinfos request method=POST path=/api/pending_closures/21/sign5040lb # [ 60.053204] niks3-server[1046]: 2026/09/20 12:21:16 INFO Signed narinfos id=21 count=15041worker1 # [ 60.057973] nix-grpc-daemon[855]: time=2026-09-20T12:21:15.298Z level=INFO msg="Uploading 1 narinfos"5042lb # [ 60.059770] niks3-server[1046]: 2026/09/20 12:21:16 INFO Registered completed upload object_key=nar/0bp8pfsa12j91bab6f50rs7hh6sphjrig1zqlr7v80mn285qxax2.nar.zst5043lb # [ 60.072776] niks3-server[1046]: 2026/09/20 12:21:16 INFO Received complete upload request method=POST path=/api/pending_closures/21/complete5044lb # [ 60.076194] niks3-server[1046]: 2026/09/20 12:21:16 INFO Registered completed upload object_key=2ign0w9nlrywl4qfkq5i314zinl1cxqs.ls5045lb # [ 60.081180] niks3-server[1046]: 2026/09/20 12:21:16 INFO Completed upload id=215046worker1 # [ 60.081951] nix-grpc-daemon[855]: time=2026-09-20T12:21:15.322Z level=INFO msg="Upload complete. (638ms)"5047lb # [ 60.084885] niks3-server[1046]: 2026/09/20 12:21:16 INFO Registered completed upload object_key=2ign0w9nlrywl4qfkq5i314zinl1cxqs.narinfo5048worker1 # [ 60.087816] nix-grpc-daemon[786]: ts=2026-09-20T12:21:15Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38410 duration_s=1 paths=1 nar_bytes_in=5605049worker1 # [ 60.095315] nix-grpc-daemon[786]: ts=2026-09-20T12:21:15Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:384105050worker1 # [ 60.105666] nix-daemon[845]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5051worker1 # [ 60.123294] nix-daemon[845]: reaped child process 1148, status = succeeded5052client # building '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' on lb:50051...5053worker1 # [ 60.129518] nix-daemon[845]: accepted connection from pid 786, user nix-grpc-daemon (trusted)5054client # uses-input-w1onlyb> node-a: building 2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5055worker1 # [ 60.947145] nix-daemon[845]: reaped child process 1166, status = succeeded5056worker1 # [ 61.085587] nix-daemon[845]: accepted connection from pid 1181, user nix-grpc-daemon (trusted)5057worker1 # [ 61.098400] nix-daemon[845]: reaped child process 1184, status = succeeded5058lb # [ 61.107832] niks3-server[1046]: 2026/09/20 12:21:17 INFO Received uploads request method=POST path=/api/pending_closures5059worker1 # [ 61.116965] nix-grpc-daemon[855]: time=2026-09-20T12:21:16.357Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5060lb # [ 61.125731] niks3-server[1046]: 2026/09/20 12:21:17 INFO Received sign narinfos request method=POST path=/api/pending_closures/22/sign5061lb # [ 61.127938] niks3-server[1046]: 2026/09/20 12:21:17 INFO Signed narinfos id=22 count=15062worker1 # [ 61.129218] nix-grpc-daemon[855]: time=2026-09-20T12:21:16.369Z level=INFO msg="Uploading 1 narinfos"5063lb # [ 61.131744] niks3-server[1046]: 2026/09/20 12:21:17 INFO Registered completed upload object_key=rs93p5yqksm4zlblqlsfa8kxl8lkhrbg.ls5064lb # [ 61.133891] niks3-server[1046]: 2026/09/20 12:21:17 INFO Registered completed upload object_key=log/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5065lb # [ 61.137982] niks3-server[1046]: 2026/09/20 12:21:17 INFO Received complete upload request method=POST path=/api/pending_closures/22/complete5066lb # [ 61.141605] niks3-server[1046]: 2026/09/20 12:21:17 INFO Completed upload id=225067worker1 # [ 61.142594] nix-grpc-daemon[855]: time=2026-09-20T12:21:16.383Z level=INFO msg="Upload complete. (202ms)"5068lb # [ 61.142883] niks3-server[1046]: 2026/09/20 12:21:17 INFO Registered completed upload object_key=rs93p5yqksm4zlblqlsfa8kxl8lkhrbg.narinfo5069worker1 # [ 61.145119] nix-grpc-daemon[786]: ts=2026-09-20T12:21:16Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38410 duration_s=1 drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv assign_id=10 outputs=15070worker1 # [ 61.151669] nix-daemon[845]: reaped child process 1164, status = succeeded5071client # copying 1 paths...5072worker2 # [ 61.180851] nix-grpc-daemon[857]: ts=2026-09-20T12:21:16Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:417345073worker2 # [ 61.194066] nix-grpc-daemon[857]: copying path '/nix/store/rs93p5yqksm4zlblqlsfa8kxl8lkhrbg-uses-input-w1onlyb' from 'http://lb:5751'...5074client # copying path '/nix/store/rs93p5yqksm4zlblqlsfa8kxl8lkhrbg-uses-input-w1onlyb' from 'grpc://lb:50051'...5075worker1 # [ 61.234076] nix-grpc-daemon[786]: ts=2026-09-20T12:21:16Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:384105076worker1 # [ 61.235917] nix-grpc-daemon[786]: ts=2026-09-20T12:21:16Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:38410 duration_s=0 paths=1 nar_bytes_out=1205077client # [ 61.277870] nix-daemon[1018]: reaped child process 1126, status = succeeded5078client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt b >&2, in 2.74 seconds)5079client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt c >&25080client # [ 61.438087] nix-daemon[1018]: accepted connection from pid 1145, user root (trusted)5081client # this derivation will be built:5082client # /nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5083lb # [ 61.475185] envoy[1008]: conn peer=[2001:db8:1::1]:33546 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3850 tx=1858 ms=16415084lb # [ 61.476750] envoy[1008]: conn peer=[2001:db8:1::1]:33554 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=435 tx=378 ms=545085lb # [ 61.478166] envoy[1008]: conn peer=[2001:db8:1::1]:33564 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4315 tx=2084 ms=20965086lb # [ 61.479716] envoy[1008]: conn peer=[2001:db8:1::1]:58638 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=437 tx=411 ms=865087lb # [ 61.481250] envoy[1008]: conn peer=[2001:db8:1::1]:58654 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4193 tx=1753 ms=26035088lb # [ 61.482703] envoy[1008]: conn peer=[2001:db8:1::1]:58658 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=447 tx=328 ms=545089lb # [ 61.484178] envoy[1008]: conn peer=[2001:db8:1::1]:58666 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3712 tx=1915 ms=23715090lb # [ 61.485636] envoy[1008]: conn peer=[2001:db8:1::1]:58668 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=421 tx=328 ms=95091worker2 # [ 61.581655] nix-grpc-daemon[857]: ts=2026-09-20T12:21:16Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:417345092worker2 # [ 61.584480] nix-grpc-daemon[857]: ts=2026-09-20T12:21:16Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:417345093client # building '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' on 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key&system=x86_64-linux'...5094worker2 # [ 61.586728] nix-grpc-daemon[987]: time=2026-09-20T12:21:16.908Z level=INFO msg="All 1 paths already cached"5095worker2 # [ 61.598633] nix-grpc-daemon[857]: ts=2026-09-20T12:21:16Z level=debug event=want client=11 drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv cached=05096worker2 # [ 61.604431] nix-grpc-daemon[857]: ts=2026-09-20T12:21:16Z level=debug event=assigned drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv worker=192.168.1.4:50051 clients=15097worker2 # [ 61.610596] nix-grpc-daemon[857]: ts=2026-09-20T12:21:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:417345098worker2 # [ 61.622390] nix-grpc-daemon[857]: ts=2026-09-20T12:21:16Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:417345099worker2 # [ 61.634449] nix-grpc-daemon[987]: time=2026-09-20T12:21:16.955Z level=INFO msg="All 1 paths already cached"5100client # copying 1 paths to 'lb:50051'...5101client # copying path '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' to 'lb:50051'...5102worker2 # [ 61.641418] nix-grpc-daemon[857]: ts=2026-09-20T12:21:16Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:417345103worker2 # [ 61.650902] nix-daemon[937]: accepted connection from pid 857, user nix-grpc-daemon (trusted)5104worker2 # [ 61.836701] nix-daemon[937]: accepted connection from pid 1120, user nix-grpc-daemon (trusted)5105worker2 # [ 61.856478] nix-daemon[937]: reaped child process 1123, status = succeeded5106worker2 # [ 62.009829] nix-daemon[937]: accepted connection from pid 1125, user nix-grpc-daemon (trusted)5107worker2 # [ 62.027451] nix-daemon[937]: reaped child process 1128, status = succeeded5108lb # [ 62.052723] niks3-server[1046]: 2026/09/20 12:21:18 INFO Received uploads request method=POST path=/api/pending_closures5109worker2 # [ 62.201266] nix-grpc-daemon[987]: time=2026-09-20T12:21:17.522Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5110worker2 # [ 62.206178] nix-grpc-daemon[987]: time=2026-09-20T12:21:17.526Z level=INFO msg="Uploading ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv (560B)"5111lb # [ 62.240839] niks3-server[1046]: 2026/09/20 12:21:18 INFO Received sign narinfos request method=POST path=/api/pending_closures/23/sign5112lb # [ 62.246453] niks3-server[1046]: 2026/09/20 12:21:18 INFO Registered completed upload object_key=nar/1iq9qm6vw4b3p68mlkiw8nf0phmz21m8g79izsy14sydxmd4gmqa.nar.zst5113lb # [ 62.252375] niks3-server[1046]: 2026/09/20 12:21:18 INFO Signed narinfos id=23 count=15114worker2 # [ 62.244686] nix-grpc-daemon[987]: time=2026-09-20T12:21:17.566Z level=INFO msg="Uploading 1 narinfos"5115lb # [ 62.268657] niks3-server[1046]: 2026/09/20 12:21:18 INFO Received complete upload request method=POST path=/api/pending_closures/23/complete5116lb # [ 62.274698] niks3-server[1046]: 2026/09/20 12:21:18 INFO Registered completed upload object_key=ych53lrh5qfmdxlnx672apx2v80x3aan.ls5117worker2 # [ 62.269820] nix-grpc-daemon[987]: time=2026-09-20T12:21:17.591Z level=INFO msg="Upload complete. (605ms)"5118lb # [ 62.280927] niks3-server[1046]: 2026/09/20 12:21:18 INFO Completed upload id=235119worker2 # [ 62.272449] nix-grpc-daemon[857]: ts=2026-09-20T12:21:17Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41734 duration_s=0 paths=1 nar_bytes_in=5605120worker2 # [ 62.275377] nix-grpc-daemon[857]: ts=2026-09-20T12:21:17Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:417345121lb # [ 62.288768] niks3-server[1046]: 2026/09/20 12:21:18 INFO Registered completed upload object_key=ych53lrh5qfmdxlnx672apx2v80x3aan.narinfo5122worker2 # [ 62.283505] nix-daemon[937]: accepted connection from pid 857, user nix-grpc-daemon (trusted)5123worker2 # [ 62.287529] nix-daemon[937]: reaped child process 1118, status = succeeded5124worker2 # [ 62.293157] nix-daemon[937]: accepted connection from pid 857, user nix-grpc-daemon (trusted)5125client # building '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' on lb:50051...5126client # uses-input-w1onlyc> worker2: building ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5127worker2 # [ 63.108290] nix-daemon[937]: reaped child process 1132, status = succeeded5128worker2 # [ 63.241907] nix-daemon[937]: accepted connection from pid 1147, user nix-grpc-daemon (trusted)5129worker2 # [ 63.253899] nix-daemon[937]: reaped child process 1150, status = succeeded5130lb # [ 63.279364] niks3-server[1046]: 2026/09/20 12:21:19 INFO Received uploads request method=POST path=/api/pending_closures5131worker2 # [ 63.272668] nix-grpc-daemon[987]: time=2026-09-20T12:21:18.594Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5132lb # [ 63.294155] niks3-server[1046]: 2026/09/20 12:21:19 INFO Received sign narinfos request method=POST path=/api/pending_closures/24/sign5133worker2 # [ 63.285930] nix-grpc-daemon[987]: time=2026-09-20T12:21:18.607Z level=INFO msg="Uploading 1 narinfos"5134lb # [ 63.297201] niks3-server[1046]: 2026/09/20 12:21:19 INFO Signed narinfos id=24 count=15135lb # [ 63.299970] niks3-server[1046]: 2026/09/20 12:21:19 INFO Registered completed upload object_key=af2xkgzqdrcsfmk8dgmypxmxww9cx3rk.ls5136lb # [ 63.302193] niks3-server[1046]: 2026/09/20 12:21:19 INFO Registered completed upload object_key=log/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5137lb # [ 63.307253] niks3-server[1046]: 2026/09/20 12:21:19 INFO Received complete upload request method=POST path=/api/pending_closures/24/complete5138lb # [ 63.310606] niks3-server[1046]: 2026/09/20 12:21:19 INFO Completed upload id=245139worker2 # [ 63.302673] nix-grpc-daemon[987]: time=2026-09-20T12:21:18.624Z level=INFO msg="Upload complete. (202ms)"5140lb # [ 63.314466] niks3-server[1046]: 2026/09/20 12:21:19 INFO Registered completed upload object_key=af2xkgzqdrcsfmk8dgmypxmxww9cx3rk.narinfo5141worker2 # [ 63.306597] nix-grpc-daemon[857]: ts=2026-09-20T12:21:18Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41734 duration_s=1 drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv assign_id=11 outputs=15142worker2 # [ 63.313092] nix-daemon[937]: reaped child process 1130, status = succeeded5143client # copying 1 paths...5144worker2 # [ 63.351262] nix-grpc-daemon[857]: ts=2026-09-20T12:21:18Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:417345145client # copying path '/nix/store/af2xkgzqdrcsfmk8dgmypxmxww9cx3rk-uses-input-w1onlyc' from 'grpc://lb:50051'...5146worker1 # [ 63.383466] nix-grpc-daemon[786]: ts=2026-09-20T12:21:18Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:384105147worker1 # [ 63.393353] nix-grpc-daemon[786]: copying path '/nix/store/af2xkgzqdrcsfmk8dgmypxmxww9cx3rk-uses-input-w1onlyc' from 'http://lb:5751'...5148worker1 # [ 63.403450] nix-grpc-daemon[786]: ts=2026-09-20T12:21:18Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:38410 duration_s=0 paths=1 nar_bytes_out=1205149client # [ 63.498090] nix-daemon[1018]: reaped child process 1148, status = succeeded5150client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt c >&2, in 2.21 seconds)5151(finished: subtest: an input only one worker has still reaches the builder, in 8.82 seconds)5152subtest: upload whose reference the worker lacks is completed from the cache5153client: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A input --argstr tag viacache5154client # this derivation will be built:5155client # /nix/store/zpv7g9hl0xcjhjz0g0hd92sv1sf580zh-local-input-viacache.drv5156client # building '/nix/store/zpv7g9hl0xcjhjz0g0hd92sv1sf580zh-local-input-viacache.drv'...5157client: (finished: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A input --argstr tag viacache, in 0.35 seconds)5158client: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A referrer --argstr tag viacache5159client # this derivation will be built:5160client # /nix/store/8higgmxp1c6ag4yj6sk9g6z66n43x70l-refers-to-input-viacache.drv5161client # building '/nix/store/8higgmxp1c6ag4yj6sk9g6z66n43x70l-refers-to-input-viacache.drv'...5162client: (finished: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A referrer --argstr tag viacache, in 0.39 seconds)5163client: must succeed: nix-store --export /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache > /tmp/shared/ref.closure5164client: (finished: must succeed: nix-store --export /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache > /tmp/shared/ref.closure, in 0.16 seconds)5165worker1: must succeed: nix-store --import < /tmp/shared/ref.closure5166worker1: (finished: must succeed: nix-store --import < /tmp/shared/ref.closure, in 0.17 seconds)5167worker2 # [ 64.626603] systemd[1]: Stopping Nix worker-protocol over gRPC...5168worker2 # [ 64.879276] nix-grpc-daemon[857]: ts=2026-09-20T12:21:20Z level=info event=signal_exit5169worker2 # [ 65.124233] nix-grpc-daemon[857]: ts=2026-09-20T12:21:20Z level=info event=scheduler_yield5170worker2 # [ 65.128750] nix-grpc-daemon[857]: ts=2026-09-20T12:21:20Z level=info event=scheduler_restarting addr=lb:500515171lb # [ 65.139793] niks3-server[1046]: 2026/09/20 12:21:21 INFO lead: released remote=192.168.1.4:374365172worker2 # [ 65.136235] nix-grpc-daemon[857]: ts=2026-09-20T12:21:20Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:417325173worker1 # [ 65.150141] nix-grpc-daemon[786]: ts=2026-09-20T12:21:20Z level=info event=scheduler_restarting addr=lb:500515174worker2 # [ 65.140715] nix-grpc-daemon[857]: ts=2026-09-20T12:21:20Z level=info event=worker_gone addr=192.168.1.4:500515175worker2 # [ 65.143809] nix-grpc-daemon[857]: ts=2026-09-20T12:21:20Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:417325176worker2 # [ 65.147172] nix-grpc-daemon[857]: ts=2026-09-20T12:21:20Z level=info event=worker_gone addr=192.168.1.3:500515177worker2 # [ 65.329829] nix-grpc-daemon[857]: ts=2026-09-20T12:21:20Z level=info event=scheduler_connected addr=lb:500515178worker2 # [ 65.333799] nix-grpc-daemon[857]: ts=2026-09-20T12:21:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5179worker1 # [ 65.349970] nix-grpc-daemon[786]: ts=2026-09-20T12:21:20Z level=info event=scheduler_connected addr=lb:500515180worker1 # [ 65.354210] nix-grpc-daemon[786]: ts=2026-09-20T12:21:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5181worker2 # [ 65.534350] nix-grpc-daemon[857]: ts=2026-09-20T12:21:20Z level=info event=scheduler_connected addr=lb:500515182worker2 # [ 65.536123] nix-grpc-daemon[857]: ts=2026-09-20T12:21:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5183worker1 # [ 65.554527] nix-grpc-daemon[786]: ts=2026-09-20T12:21:20Z level=info event=scheduler_connected addr=lb:500515184worker1 # [ 65.556379] nix-grpc-daemon[786]: ts=2026-09-20T12:21:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5185worker2 # [ 65.736741] nix-grpc-daemon[857]: ts=2026-09-20T12:21:21Z level=info event=scheduler_connected addr=lb:500515186worker2 # [ 65.738489] nix-grpc-daemon[857]: ts=2026-09-20T12:21:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5187worker1 # [ 65.756744] nix-grpc-daemon[786]: ts=2026-09-20T12:21:20Z level=info event=scheduler_connected addr=lb:500515188worker1 # [ 65.758572] nix-grpc-daemon[786]: ts=2026-09-20T12:21:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5189worker2 # [ 65.939519] nix-grpc-daemon[857]: ts=2026-09-20T12:21:21Z level=info event=scheduler_connected addr=lb:500515190worker2 # [ 65.941279] nix-grpc-daemon[857]: ts=2026-09-20T12:21:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5191worker1 # [ 65.959126] nix-grpc-daemon[786]: ts=2026-09-20T12:21:21Z level=info event=scheduler_connected addr=lb:500515192worker1 # [ 65.960738] nix-grpc-daemon[786]: ts=2026-09-20T12:21:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5193worker2 # [ 66.127782] nix-grpc-daemon[857]: ts=2026-09-20T12:21:21Z level=info event=scheduler_restarting5194worker2 # [ 66.143831] nix-daemon[937]: reaped child process 939, status = succeeded5195worker2 # [ 66.147701] nix-grpc-daemon[857]: ts=2026-09-20T12:21:21Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5196worker1 # [ 66.161684] nix-grpc-daemon[786]: ts=2026-09-20T12:21:21Z level=info event=scheduler_connected addr=lb:500515197worker1 # [ 66.163331] nix-grpc-daemon[786]: ts=2026-09-20T12:21:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5198worker1 # [ 66.363850] nix-grpc-daemon[786]: ts=2026-09-20T12:21:21Z level=info event=scheduler_connected addr=lb:500515199worker1 # [ 66.365639] nix-grpc-daemon[786]: ts=2026-09-20T12:21:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5200worker1 # [ 66.566273] nix-grpc-daemon[786]: ts=2026-09-20T12:21:21Z level=info event=scheduler_connected addr=lb:500515201worker1 # [ 66.567936] nix-grpc-daemon[786]: ts=2026-09-20T12:21:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5202lb # [ 66.589260] niks3-server[1046]: 2026/09/20 12:21:22 INFO lead: acquired remote=192.168.1.3:400745203worker1 # [ 66.593272] nix-grpc-daemon[786]: ts=2026-09-20T12:21:21Z level=info event=scheduler_take_over5204worker1 # [ 66.768365] nix-grpc-daemon[786]: ts=2026-09-20T12:21:22Z level=info event=scheduler_connected addr=lb:500515205worker1 # [ 66.770361] nix-grpc-daemon[786]: ts=2026-09-20T12:21:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5206worker1 # [ 66.971358] nix-grpc-daemon[786]: ts=2026-09-20T12:21:22Z level=info event=scheduler_connected addr=lb:500515207worker1 # [ 66.984745] nix-grpc-daemon[786]: ts=2026-09-20T12:21:22Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:477485208worker2 # [ 66.977134] nix-grpc-daemon[857]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5209worker2 # [ 66.978618] nix-grpc-daemon[857]: E0000 00:00:1789906882.297994 900 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 16, started true, shutdown true5210worker2 # [ 66.980736] nix-grpc-daemon[857]: E0000 00:00:1789906882.302324 900 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5211worker1 # [ 66.988835] nix-grpc-daemon[786]: ts=2026-09-20T12:21:22Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05212worker2 # [ 68.147773] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5213worker2 # [ 68.151467] systemd[1]: Stopped Nix worker-protocol over gRPC.5214worker2 # [ 68.155423] systemd[1]: nix-grpc-daemon.service: Consumed 1.429s CPU time over 46.540s wall clock time, 50.5M memory peak, 55.3K incoming IP traffic, 147.7K outgoing IP traffic.5215worker2 # [ 68.166447] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5216worker2 # [ 68.169891] systemd[1]: Closed Nix worker-protocol over gRPC.5217worker2 # [ 68.173179] systemd[1]: nix-grpc-daemon.socket: Consumed 492us CPU time over 58.719s wall clock time, 256K memory peak, 68.9K incoming IP traffic, 58.5K outgoing IP traffic.5218lb: waiting until exactly ['worker2'] endpoints of x86_64-linux are unhealthy5219lb: must succeed: curl -sf localhost:9901/clusters5220lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5221lb: (finished: waiting until exactly ['worker2'] endpoints of x86_64-linux are unhealthy, in 0.07 seconds)5222client: must succeed: nix path-info --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache >&25223worker1 # [ 68.452168] nix-grpc-daemon[786]: ts=2026-09-20T12:21:23Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:384105224client # /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache5225worker1 # [ 68.454856] nix-grpc-daemon[786]: ts=2026-09-20T12:21:23Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:384105226client: (finished: must succeed: nix path-info --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache >&2, in 0.21 seconds)5227worker2: must fail: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache5228worker2: (finished: must fail: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache, in 0.03 seconds)5229worker2 # [ 68.548843] systemd[1]: Listening on Nix worker-protocol over gRPC.5230worker1 # [ 68.599467] systemd[1]: Stopping Nix worker-protocol over gRPC...5231worker1 # [ 68.735266] nix-grpc-daemon[786]: ts=2026-09-20T12:21:23Z level=info event=signal_exit5232worker1 # [ 68.980591] nix-grpc-daemon[786]: ts=2026-09-20T12:21:24Z level=info event=scheduler_yield5233worker1 # [ 68.985380] nix-grpc-daemon[786]: ts=2026-09-20T12:21:24Z level=info event=scheduler_restarting addr=lb:500515234lb # [ 68.983792] niks3-server[1046]: 2026/09/20 12:21:25 INFO lead: released remote=192.168.1.3:400745235worker1 # [ 68.991161] nix-grpc-daemon[786]: ts=2026-09-20T12:21:24Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:477485236worker1 # [ 68.995553] nix-grpc-daemon[786]: ts=2026-09-20T12:21:24Z level=info event=worker_gone addr=192.168.1.3:500515237worker2 # [ 69.135811] systemd[1]: Starting scheduler-token.service...5238worker1 # [ 69.185406] nix-grpc-daemon[786]: ts=2026-09-20T12:21:24Z level=info event=scheduler_connected addr=lb:500515239worker1 # [ 69.187965] nix-grpc-daemon[786]: ts=2026-09-20T12:21:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5240worker2 # [ 69.225431] systemd[1]: scheduler-token.service: Deactivated successfully.5241worker2 # [ 69.227229] systemd[1]: Finished scheduler-token.service.5242worker2 # [ 69.228424] systemd[1]: scheduler-token.service: Consumed 36ms CPU time over 91ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5243worker2 # [ 69.234435] systemd[1]: Starting Nix worker-protocol over gRPC...5244worker1 # [ 69.388729] nix-grpc-daemon[786]: ts=2026-09-20T12:21:24Z level=info event=scheduler_connected addr=lb:500515245worker1 # [ 69.392591] nix-grpc-daemon[786]: ts=2026-09-20T12:21:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5246worker2 # [ 69.399536] nix-grpc-daemon[1174]: ts=2026-09-20T12:21:24Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5247worker2 # [ 69.402262] nix-grpc-daemon[1174]: ts=2026-09-20T12:21:24Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5248worker2 # [ 69.406583] nix-grpc-daemon[1174]: ts=2026-09-20T12:21:24Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15249worker2 # [ 69.415396] nix-grpc-daemon[1174]: ts=2026-09-20T12:21:24Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5250worker2 # [ 69.419161] systemd[1]: Started Nix worker-protocol over gRPC.5251worker2 # [ 69.428064] nix-grpc-daemon[1174]: ts=2026-09-20T12:21:24Z level=info event=scheduler_take_over5252lb # [ 69.440334] niks3-server[1046]: 2026/09/20 12:21:25 INFO lead: acquired remote=192.168.1.4:358645253worker2 # [ 69.439755] nix-grpc-daemon[1174]: ts=2026-09-20T12:21:24Z level=info event=scheduler_connected addr=lb:500515254worker2 # [ 69.441411] nix-grpc-daemon[1174]: ts=2026-09-20T12:21:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5255worker1 # [ 69.593309] nix-grpc-daemon[786]: ts=2026-09-20T12:21:24Z level=info event=scheduler_connected addr=lb:500515256worker1 # [ 69.595091] nix-grpc-daemon[786]: ts=2026-09-20T12:21:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5257worker2 # [ 69.643194] nix-grpc-daemon[1174]: ts=2026-09-20T12:21:24Z level=info event=scheduler_connected addr=lb:500515258worker2 # [ 69.647297] nix-grpc-daemon[1174]: ts=2026-09-20T12:21:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5259worker1 # [ 69.795497] nix-grpc-daemon[786]: ts=2026-09-20T12:21:25Z level=info event=scheduler_connected addr=lb:500515260worker1 # [ 69.797490] nix-grpc-daemon[786]: ts=2026-09-20T12:21:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5261worker2 # [ 69.848696] nix-grpc-daemon[1174]: ts=2026-09-20T12:21:25Z level=info event=scheduler_connected addr=lb:500515262worker2 # [ 69.852712] nix-grpc-daemon[1174]: ts=2026-09-20T12:21:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5263worker1 # [ 69.985381] nix-grpc-daemon[786]: ts=2026-09-20T12:21:25Z level=info event=scheduler_restarting5264worker1 # [ 70.012507] nix-grpc-daemon[786]: ts=2026-09-20T12:21:25Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5265worker1 # [ 70.025458] nix-daemon[845]: reaped child process 847, status = succeeded5266worker2 # [ 70.053959] nix-grpc-daemon[1174]: ts=2026-09-20T12:21:25Z level=info event=scheduler_connected addr=lb:500515267worker2 # [ 70.057954] nix-grpc-daemon[1174]: ts=2026-09-20T12:21:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5268worker1 # [ 70.119460] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5269worker1 # [ 70.122536] systemd[1]: Stopped Nix worker-protocol over gRPC.5270worker1 # [ 70.126941] systemd[1]: nix-grpc-daemon.service: Consumed 2.518s CPU time over 48.651s wall clock time, 58.8M memory peak, 79.6K incoming IP traffic, 147.1K outgoing IP traffic.5271worker1 # [ 70.139600] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5272worker1 # [ 70.143347] systemd[1]: Closed Nix worker-protocol over gRPC.5273worker1 # [ 70.146586] systemd[1]: nix-grpc-daemon.socket: Consumed 308us CPU time over 59.961s wall clock time, 256K memory peak, 74.6K incoming IP traffic, 63.3K outgoing IP traffic.5274lb: waiting until exactly ['worker1'] endpoints of x86_64-linux are unhealthy5275lb: must succeed: curl -sf localhost:9901/clusters5276lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5277lb: (finished: waiting until exactly ['worker1'] endpoints of x86_64-linux are unhealthy, in 0.06 seconds)5278client: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache >&25279worker2 # [ 70.259059] nix-grpc-daemon[1174]: ts=2026-09-20T12:21:25Z level=info event=scheduler_connected addr=lb:500515280worker2 # [ 70.262914] nix-grpc-daemon[1174]: ts=2026-09-20T12:21:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5281client # lb:50051: upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, retrying5282worker2 # [ 70.464672] nix-grpc-daemon[1174]: ts=2026-09-20T12:21:25Z level=info event=scheduler_connected addr=lb:500515283worker2 # [ 70.478137] nix-grpc-daemon[1174]: ts=2026-09-20T12:21:25Z level=info event=oidc_jwks provider=mock keys=15284worker2 # [ 70.481223] nix-grpc-daemon[1174]: ts=2026-09-20T12:21:25Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:601825285worker2 # [ 70.485534] nix-grpc-daemon[1174]: ts=2026-09-20T12:21:25Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05286worker2 # [ 70.896682] nix-grpc-daemon[1174]: ts=2026-09-20T12:21:26Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:601865287worker2 # [ 70.904108] nix-daemon[937]: accepted connection from pid 1174, user nix-grpc-daemon (trusted)5288worker2 # [ 70.914317] nix-grpc-daemon[1174]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 52 ms (attempt 1/5)5289worker2 # [ 70.969321] nix-grpc-daemon[1174]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 7 ms (attempt 2/5)5290worker2 # [ 70.979321] nix-grpc-daemon[1174]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 377 ms (attempt 3/5)5291worker2 # [ 71.359404] nix-grpc-daemon[1174]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 92 ms (attempt 4/5)5292worker2 # [ 71.451899] nix-grpc-daemon[1174]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5293client # copying 2 paths...5294client # copying path '/nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache' to 'grpc://lb:50051'...5295client # copying path '/nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache' to 'grpc://lb:50051'...5296lb # [ 71.476737] envoy[1008]: conn peer=[2001:db8:1::1]:35354 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=447 tx=329 ms=735297worker2 # [ 71.470921] nix-grpc-daemon[1174]: ts=2026-09-20T12:21:26Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:601865298lb # [ 71.481789] envoy[1008]: conn peer=[2001:db8:1::1]:58680 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3714 tx=1805 ms=18595299worker2 # [ 71.476059] nix-daemon[937]: accepted connection from pid 1174, user nix-grpc-daemon (trusted)5300lb # [ 71.485395] envoy[1008]: conn peer=[2001:db8:1::4]:51404 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=25731 tx=1461 ms=442515301lb # [ 71.489447] envoy[1008]: conn peer=[2001:db8:1::1]:35356 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=591 tx=519 ms=145302lb # [ 71.493094] envoy[1008]: conn peer=192.168.1.3:35038 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=8540 tx=1989 ms=474235303worker2 # [ 71.487302] nix-grpc-daemon[1174]: ts=2026-09-20T12:21:26Z level=info event=niks3_push_started pid=12305304worker2 # [ 71.644665] nix-grpc-daemon[1230]: time=2026-09-20T12:21:26.965Z level=INFO msg="Configuring client TLS" cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/worker2.pem key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/worker2.key ca=""5305worker2 # [ 71.806315] nix-daemon[937]: accepted connection from pid 1243, user nix-grpc-daemon (trusted)5306worker2 # [ 71.822948] nix-daemon[937]: reaped child process 1246, status = succeeded5307lb # [ 71.849978] niks3-server[1046]: 2026/09/20 12:21:28 INFO Received uploads request method=POST path=/api/pending_closures5308lb # [ 71.858625] niks3-server[1046]: 2026/09/20 12:21:28 INFO Received uploads request method=POST path=/api/pending_closures5309worker2 # [ 71.854823] nix-grpc-daemon[1230]: time=2026-09-20T12:21:27.176Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"5310worker2 # [ 71.856652] nix-grpc-daemon[1230]: time=2026-09-20T12:21:27.178Z level=INFO msg="Uploading 6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache (184B)"5311worker2 # [ 71.858637] nix-grpc-daemon[1230]: time=2026-09-20T12:21:27.180Z level=INFO msg="Uploading cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache (128B)"5312lb # [ 71.904682] niks3-server[1046]: 2026/09/20 12:21:28 INFO Registered completed upload object_key=nar/15y5akaw5jxrf5q28c25cfgfdz5zw27ihvpagi1mmvpnaw9na2a5.nar.zst5313lb # [ 71.912166] niks3-server[1046]: 2026/09/20 12:21:28 INFO Registered completed upload object_key=nar/1kmnh5f2l7jjcgaq9b9988bjix6z8zjf6v350xllq5hlzgnb1f75.nar.zst5314lb # [ 71.930913] niks3-server[1046]: 2026/09/20 12:21:28 INFO Received sign narinfos request method=POST path=/api/pending_closures/25/sign5315lb # [ 71.933339] niks3-server[1046]: 2026/09/20 12:21:28 INFO Signed narinfos id=25 count=15316lb # [ 71.936875] niks3-server[1046]: 2026/09/20 12:21:28 INFO Received sign narinfos request method=POST path=/api/pending_closures/26/sign5317lb # [ 71.939779] niks3-server[1046]: 2026/09/20 12:21:28 INFO Registered completed upload object_key=6zp5fijxpjrd2vv7855sb0z7aag6mm8d.ls5318lb # [ 71.943920] niks3-server[1046]: 2026/09/20 12:21:28 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.ls5319lb # [ 71.948784] niks3-server[1046]: 2026/09/20 12:21:28 INFO Signed narinfos id=26 count=25320worker2 # [ 71.941292] nix-grpc-daemon[1230]: time=2026-09-20T12:21:27.261Z level=INFO msg="Uploading 3 narinfos"5321lb # [ 71.966268] niks3-server[1046]: 2026/09/20 12:21:28 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.narinfo5322lb # [ 71.968580] niks3-server[1046]: 2026/09/20 12:21:28 INFO Received complete upload request method=POST path=/api/pending_closures/25/complete5323lb # [ 71.974168] niks3-server[1046]: 2026/09/20 12:21:28 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.narinfo5324lb # [ 71.976543] niks3-server[1046]: 2026/09/20 12:21:28 INFO Registered completed upload object_key=6zp5fijxpjrd2vv7855sb0z7aag6mm8d.narinfo5325lb # [ 71.984905] niks3-server[1046]: 2026/09/20 12:21:28 INFO Completed upload id=255326lb # [ 71.986143] niks3-server[1046]: 2026/09/20 12:21:28 INFO Received complete upload request method=POST path=/api/pending_closures/26/complete5327lb # [ 71.989222] niks3-server[1046]: 2026/09/20 12:21:28 INFO Completed upload id=265328worker2 # [ 71.978601] nix-grpc-daemon[1230]: time=2026-09-20T12:21:27.300Z level=INFO msg="Upload complete. (330ms)"5329worker2 # [ 71.983623] nix-grpc-daemon[1174]: ts=2026-09-20T12:21:27Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60186 duration_s=0 paths=2 nar_bytes_in=3125330worker2 # [ 71.998475] nix-daemon[937]: reaped child process 1228, status = succeeded5331client: (finished: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache >&2, in 1.81 seconds)5332worker2: must succeed: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache && test -e /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache5333worker2: (finished: must succeed: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache && test -e /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache, in 0.03 seconds)5334worker1 # [ 72.089967] systemd[1]: Listening on Nix worker-protocol over gRPC.5335worker1 # [ 72.094137] systemd[1]: Starting Nix worker-protocol over gRPC...5336worker1 # [ 72.254915] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:27Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5337worker1 # [ 72.256675] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:27Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5338worker1 # [ 72.260576] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:27Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15339worker1 # [ 72.267062] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:27Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5340worker1 # [ 72.272465] systemd[1]: Started Nix worker-protocol over gRPC.5341lb: waiting until exactly no endpoints of x86_64-linux are unhealthy5342lb: must succeed: curl -sf localhost:9901/clusters5343worker1 # [ 72.282916] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:27Z level=info event=scheduler_connected addr=lb:500515344worker2 # [ 72.273381] nix-grpc-daemon[1174]: ts=2026-09-20T12:21:27Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:601825345worker2 # [ 72.278300] nix-grpc-daemon[1174]: ts=2026-09-20T12:21:27Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05346lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5347lb: must succeed: curl -sf localhost:9901/clusters5348lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5349lb: (finished: waiting until exactly no endpoints of x86_64-linux are unhealthy, in 1.12 seconds)5350(finished: subtest: upload whose reference the worker lacks is completed from the cache, in 9.88 seconds)5351subtest: scheduler down: the other node takes the lock and keeps it5352lb: must succeed: curl -sf localhost:9901/clusters5353lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5354lb: must succeed: curl -sf localhost:9901/clusters5355lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5356lb: must succeed: curl -sf localhost:9901/clusters5357lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5358lb: must succeed: curl -sf localhost:9901/clusters5359lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5360worker2 # [ 73.599487] systemd[1]: Stopping Nix worker-protocol over gRPC...5361worker2 # [ 74.419740] nix-grpc-daemon[1174]: ts=2026-09-20T12:21:29Z level=info event=signal_exit5362worker2 # [ 74.426784] nix-grpc-daemon[1174]: ts=2026-09-20T12:21:29Z level=info event=scheduler_yield5363lb # [ 74.439764] niks3-server[1046]: 2026/09/20 12:21:30 INFO lead: released remote=192.168.1.4:358645364worker2 # [ 74.434169] nix-grpc-daemon[1174]: ts=2026-09-20T12:21:29Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:601825365worker2 # [ 74.436183] nix-grpc-daemon[1174]: ts=2026-09-20T12:21:29Z level=info event=worker_gone addr=192.168.1.3:500515366worker1 # [ 74.446541] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:29Z level=info event=scheduler_restarting addr=lb:500515367worker2 # [ 74.437961] nix-grpc-daemon[1174]: ts=2026-09-20T12:21:29Z level=info event=scheduler_restarting addr=lb:500515368worker2 # [ 74.439182] nix-grpc-daemon[1174]: ts=2026-09-20T12:21:29Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:601825369worker2 # [ 74.440485] nix-grpc-daemon[1174]: ts=2026-09-20T12:21:29Z level=info event=worker_gone addr=192.168.1.4:500515370worker1 # [ 74.646486] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:29Z level=info event=scheduler_connected addr=lb:500515371worker2 # [ 74.637331] nix-grpc-daemon[1174]: ts=2026-09-20T12:21:29Z level=info event=scheduler_connected addr=lb:500515372worker2 # [ 74.639270] nix-grpc-daemon[1174]: ts=2026-09-20T12:21:29Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5373worker1 # [ 74.650419] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:29Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5374worker1 # [ 74.850559] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:30Z level=info event=scheduler_connected addr=lb:500515375worker1 # [ 74.852439] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:30Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5376worker2 # [ 74.840743] nix-grpc-daemon[1174]: ts=2026-09-20T12:21:30Z level=info event=scheduler_connected addr=lb:500515377worker2 # [ 74.844925] nix-grpc-daemon[1174]: ts=2026-09-20T12:21:30Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5378worker1 # [ 75.053458] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:30Z level=info event=scheduler_connected addr=lb:500515379worker1 # [ 75.057548] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:30Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5380worker2 # [ 75.046397] nix-grpc-daemon[1174]: ts=2026-09-20T12:21:30Z level=info event=scheduler_connected addr=lb:500515381worker2 # [ 75.050176] nix-grpc-daemon[1174]: ts=2026-09-20T12:21:30Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5382worker1 # [ 75.258564] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:30Z level=info event=scheduler_connected addr=lb:500515383worker2 # [ 75.250911] nix-grpc-daemon[1174]: ts=2026-09-20T12:21:30Z level=info event=scheduler_connected addr=lb:500515384worker2 # [ 75.252682] nix-grpc-daemon[1174]: ts=2026-09-20T12:21:30Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5385worker1 # [ 75.262750] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:30Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5386worker2 # [ 75.430046] nix-grpc-daemon[1174]: ts=2026-09-20T12:21:30Z level=info event=scheduler_restarting5387worker2 # [ 75.441181] nix-daemon[937]: reaped child process 1223, status = succeeded5388worker2 # [ 75.447325] nix-grpc-daemon[1174]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5389worker2 # [ 75.448841] nix-grpc-daemon[1174]: E0000 00:00:1789906890.768897 1217 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 16, started true, shutdown true5390worker2 # [ 75.450684] nix-grpc-daemon[1174]: E0000 00:00:1789906890.770416 1217 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 9, started true, shutdown true5391worker1 # [ 75.464808] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:30Z level=info event=scheduler_connected addr=lb:500515392worker2 # [ 75.457945] nix-grpc-daemon[1174]: ts=2026-09-20T12:21:30Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5393worker1 # [ 75.467980] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:30Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5394worker1 # [ 75.665445] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:30Z level=info event=scheduler_connected addr=lb:500515395worker1 # [ 75.669394] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:30Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5396worker1 # [ 75.869960] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:31Z level=info event=scheduler_connected addr=lb:500515397worker1 # [ 75.873922] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:31Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5398worker1 # [ 76.074220] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:31Z level=info event=scheduler_connected addr=lb:500515399worker1 # [ 76.075976] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:31Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5400worker1 # [ 76.276370] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:31Z level=info event=scheduler_connected addr=lb:500515401worker1 # [ 76.278228] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:31Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5402worker1 # [ 76.480484] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:31Z level=info event=scheduler_connected addr=lb:500515403worker1 # [ 76.483568] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:31Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5404worker1 # [ 76.681319] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:31Z level=info event=scheduler_connected addr=lb:500515405worker1 # [ 76.685131] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:31Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5406worker1 # [ 76.885287] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:32Z level=info event=scheduler_connected addr=lb:500515407worker1 # [ 76.887147] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:32Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5408worker1 # [ 77.087539] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:32Z level=info event=scheduler_connected addr=lb:500515409worker1 # [ 77.089255] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:32Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5410worker1 # [ 77.276261] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:32Z level=info event=scheduler_take_over5411lb # [ 77.275856] niks3-server[1046]: 2026/09/20 12:21:33 INFO lead: acquired remote=192.168.1.3:530485412worker1 # [ 77.289612] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:32Z level=info event=scheduler_connected addr=lb:500515413worker1 # [ 77.291236] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:32Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5414worker2 # [ 77.429254] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5415worker2 # [ 77.431274] systemd[1]: Stopped Nix worker-protocol over gRPC.5416worker2 # [ 77.432133] systemd[1]: nix-grpc-daemon.service: Consumed 348ms CPU time over 8.194s wall clock time, 30.6M memory peak, 20.2K incoming IP traffic, 31.4K outgoing IP traffic.5417worker2 # [ 77.436584] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5418worker2 # [ 77.438308] systemd[1]: Closed Nix worker-protocol over gRPC.5419worker2 # [ 77.439487] systemd[1]: nix-grpc-daemon.socket: Consumed 381us CPU time over 8.895s wall clock time, 256K memory peak, 18.7K incoming IP traffic, 13.4K outgoing IP traffic.5420worker1: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over5421worker1 # [ 77.492756] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:32Z level=info event=scheduler_connected addr=lb:500515422worker1 # [ 77.497297] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:32Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5423worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over, in 0.07 seconds)5424worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5425worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.03 seconds)5426worker1 # [ 77.697275] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:32Z level=info event=scheduler_connected addr=lb:500515427worker1 # [ 77.701119] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:32Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5428worker1 # [ 77.901805] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:33Z level=info event=scheduler_connected addr=lb:500515429worker1 # [ 77.908353] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:33Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:550845430worker1 # [ 77.912734] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:33Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05431worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5432worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)5433client: must succeed: nix build -L --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag on-standby >&25434worker1 # [ 78.818279] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:34Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:550865435worker1 # [ 78.823612] nix-daemon[845]: accepted connection from pid 1208, user nix-grpc-daemon (trusted)5436client # don't know how to build these paths:5437client # /nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv5438worker1 # [ 78.831812] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:34Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:550865439worker1 # [ 78.837158] nix-grpc-daemon[1208]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 80 ms (attempt 1/5)5440worker1 # [ 78.919564] nix-grpc-daemon[1208]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 132 ms (attempt 2/5)5441worker1 # [ 79.054248] nix-grpc-daemon[1208]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 277 ms (attempt 3/5)5442worker1 # [ 79.334260] nix-grpc-daemon[1208]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 470 ms (attempt 4/5)5443worker1 # [ 79.809984] nix-grpc-daemon[1208]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5444worker1 # [ 79.832351] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:35Z level=debug event=want client=1 drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv cached=05445worker1 # [ 79.834468] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:35Z level=debug event=assigned drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv worker=192.168.1.3:50051 clients=15446worker1 # [ 79.875827] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:550865447worker1 # [ 79.883753] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:35Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:550865448client # copying 1 paths to 'lb:50051'...5449client # copying path '/nix/store/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv' to 'lb:50051'...5450worker1 # [ 79.890830] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:35Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:550865451worker1 # [ 79.898185] nix-daemon[845]: accepted connection from pid 1208, user nix-grpc-daemon (trusted)5452worker1 # [ 79.915625] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:35Z level=info event=niks3_push_started pid=12785453worker1 # [ 80.081333] nix-grpc-daemon[1278]: time=2026-09-20T12:21:35.320Z level=INFO msg="Configuring client TLS" cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/worker1.pem key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/worker1.key ca=""5454worker1 # [ 80.223770] nix-daemon[845]: accepted connection from pid 1291, user nix-grpc-daemon (trusted)5455worker1 # [ 80.234647] nix-daemon[845]: reaped child process 1294, status = succeeded5456worker1 # [ 80.389558] nix-daemon[845]: accepted connection from pid 1296, user nix-grpc-daemon (trusted)5457worker1 # [ 80.401378] nix-daemon[845]: reaped child process 1299, status = succeeded5458lb # [ 80.414256] niks3-server[1046]: 2026/09/20 12:21:36 INFO Received uploads request method=POST path=/api/pending_closures5459worker1 # [ 80.420699] nix-grpc-daemon[1278]: time=2026-09-20T12:21:35.661Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5460worker1 # [ 80.422685] nix-grpc-daemon[1278]: time=2026-09-20T12:21:35.662Z level=INFO msg="Uploading 0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv (520B)"5461lb # [ 80.435665] niks3-server[1046]: 2026/09/20 12:21:36 INFO Registered completed upload object_key=nar/013ynk63lqgbvdpr2qqb6pflhzfcrl005f3zkvnag3x7hqxjhibs.nar.zst5462lb # [ 80.442209] niks3-server[1046]: 2026/09/20 12:21:36 INFO Received sign narinfos request method=POST path=/api/pending_closures/27/sign5463worker1 # [ 80.448747] nix-grpc-daemon[1278]: time=2026-09-20T12:21:35.689Z level=INFO msg="Uploading 1 narinfos"5464lb # [ 80.447531] niks3-server[1046]: 2026/09/20 12:21:36 INFO Signed narinfos id=27 count=15465lb # [ 80.457378] niks3-server[1046]: 2026/09/20 12:21:36 INFO Received complete upload request method=POST path=/api/pending_closures/27/complete5466lb # [ 80.459592] niks3-server[1046]: 2026/09/20 12:21:36 INFO Registered completed upload object_key=0i80y1x9s0d06s86lhapa650pgsxgi0g.ls5467lb # [ 80.464887] niks3-server[1046]: 2026/09/20 12:21:36 INFO Completed upload id=275468worker1 # [ 80.467269] nix-grpc-daemon[1278]: time=2026-09-20T12:21:35.707Z level=INFO msg="Upload complete. (386ms)"5469lb # [ 80.466709] niks3-server[1046]: 2026/09/20 12:21:36 INFO Registered completed upload object_key=0i80y1x9s0d06s86lhapa650pgsxgi0g.narinfo5470worker1 # [ 80.469551] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:35Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55086 duration_s=0 paths=1 nar_bytes_in=5205471worker1 # [ 80.472482] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:550865472worker1 # [ 80.478792] nix-daemon[845]: accepted connection from pid 1208, user nix-grpc-daemon (trusted)5473worker1 # [ 80.482389] nix-daemon[845]: reaped child process 1276, status = succeeded5474client # building '/nix/store/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv' on lb:50051...5475client # farm-a> node-a: building 0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv5476worker1 # [ 80.487041] nix-daemon[845]: accepted connection from pid 1208, user nix-grpc-daemon (trusted)5477lb # [ 81.476943] envoy[1008]: conn peer=[2001:db8:1::1]:35368 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="TLS_error:|33554464:system_library:OPENSSL_internal:Broken_pipe:TLS_error_end" rx=1164 tx=403 ms=16065478lb # [ 81.482683] envoy[1008]: conn peer=[2001:db8:1::4]:34042 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=8603 tx=535 ms=60225479client # farm-a> LOG-farm-a-on-standby5480worker1 # [ 81.715966] nix-daemon[845]: reaped child process 1303, status = succeeded5481worker1 # [ 81.851717] nix-daemon[845]: accepted connection from pid 1318, user nix-grpc-daemon (trusted)5482worker1 # [ 81.864486] nix-daemon[845]: reaped child process 1321, status = succeeded5483lb # [ 81.877497] niks3-server[1046]: 2026/09/20 12:21:38 INFO Received uploads request method=POST path=/api/pending_closures5484worker1 # [ 81.887107] nix-grpc-daemon[1278]: time=2026-09-20T12:21:37.127Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5485worker1 # [ 81.888898] nix-grpc-daemon[1278]: time=2026-09-20T12:21:37.129Z level=INFO msg="Uploading fp54697jryq5rvfg8ml51wzkr9xjvhrg-farm-a (136B)"5486lb # [ 81.911643] niks3-server[1046]: 2026/09/20 12:21:38 INFO Registered completed upload object_key=log/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv5487lb # [ 81.927933] niks3-server[1046]: 2026/09/20 12:21:38 INFO Registered completed upload object_key=nar/0x76wzgdd1nk8rm930z8l69v9961jpc8qhzmgganv0nhagq8lv9c.nar.zst5488lb # [ 81.940096] niks3-server[1046]: 2026/09/20 12:21:38 INFO Received sign narinfos request method=POST path=/api/pending_closures/28/sign5489lb # [ 81.946555] niks3-server[1046]: 2026/09/20 12:21:38 INFO Signed narinfos id=28 count=15490worker1 # [ 81.949983] nix-grpc-daemon[1278]: time=2026-09-20T12:21:37.190Z level=INFO msg="Uploading 1 narinfos"5491lb # [ 81.949573] niks3-server[1046]: 2026/09/20 12:21:38 INFO Registered completed upload object_key=fp54697jryq5rvfg8ml51wzkr9xjvhrg.ls5492lb # [ 81.960108] niks3-server[1046]: 2026/09/20 12:21:38 INFO Received complete upload request method=POST path=/api/pending_closures/28/complete5493lb # [ 81.964161] niks3-server[1046]: 2026/09/20 12:21:38 INFO Completed upload id=285494worker1 # [ 81.965270] nix-grpc-daemon[1278]: time=2026-09-20T12:21:37.205Z level=INFO msg="Upload complete. (270ms)"5495lb # [ 81.965697] niks3-server[1046]: 2026/09/20 12:21:38 INFO Registered completed upload object_key=fp54697jryq5rvfg8ml51wzkr9xjvhrg.narinfo5496worker1 # [ 81.968341] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:37Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55086 duration_s=1 drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv assign_id=1 outputs=15497worker1 # [ 81.978587] nix-daemon[845]: reaped child process 1301, status = succeeded5498worker1 # [ 81.983290] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:37Z level=debug event=want client=1 drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv cached=05499worker1 # [ 81.985397] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:37Z level=debug event=assigned drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv worker=192.168.1.3:50051 clients=15500worker1 # [ 82.028433] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:550865501worker1 # [ 82.038320] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:37Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:550865502worker1 # [ 82.048585] nix-grpc-daemon[1278]: time=2026-09-20T12:21:37.289Z level=INFO msg="All 1 paths already cached"5503client # copying 1 paths to 'lb:50051'...5504client # copying path '/nix/store/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv' to 'lb:50051'...5505worker1 # [ 82.055321] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:37Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:550865506worker1 # [ 82.061089] nix-daemon[845]: accepted connection from pid 1208, user nix-grpc-daemon (trusted)5507worker1 # [ 82.223422] nix-daemon[845]: accepted connection from pid 1327, user nix-grpc-daemon (trusted)5508worker1 # [ 82.236609] nix-daemon[845]: reaped child process 1330, status = succeeded5509worker1 # [ 82.388505] nix-daemon[845]: accepted connection from pid 1332, user nix-grpc-daemon (trusted)5510worker1 # [ 82.400744] nix-daemon[845]: reaped child process 1335, status = succeeded5511lb # [ 82.413683] niks3-server[1046]: 2026/09/20 12:21:38 INFO Received uploads request method=POST path=/api/pending_closures5512worker1 # [ 82.425037] nix-grpc-daemon[1278]: time=2026-09-20T12:21:37.665Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5513worker1 # [ 82.426383] nix-grpc-daemon[1278]: time=2026-09-20T12:21:37.665Z level=INFO msg="Uploading 6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv (640B)"5514lb # [ 82.445257] niks3-server[1046]: 2026/09/20 12:21:38 INFO Registered completed upload object_key=nar/12kaxaz59hi3nqzgsxphwdpbajdwrfjqknsv96cnrhpibnnzjlg0.nar.zst5515lb # [ 82.456097] niks3-server[1046]: 2026/09/20 12:21:38 INFO Received sign narinfos request method=POST path=/api/pending_closures/29/sign5516worker1 # [ 82.461536] nix-grpc-daemon[1278]: time=2026-09-20T12:21:37.701Z level=INFO msg="Uploading 1 narinfos"5517lb # [ 82.460516] niks3-server[1046]: 2026/09/20 12:21:38 INFO Signed narinfos id=29 count=15518lb # [ 82.468207] niks3-server[1046]: 2026/09/20 12:21:38 INFO Registered completed upload object_key=6532q0zszdl4cnys1gb39vpngyzd7y8j.ls5519lb # [ 82.479355] niks3-server[1046]: 2026/09/20 12:21:38 INFO Received complete upload request method=POST path=/api/pending_closures/29/complete5520lb # [ 82.486373] niks3-server[1046]: 2026/09/20 12:21:38 INFO Completed upload id=295521worker1 # [ 82.487637] nix-grpc-daemon[1278]: time=2026-09-20T12:21:37.727Z level=INFO msg="Upload complete. (421ms)"5522lb # [ 82.489052] niks3-server[1046]: 2026/09/20 12:21:38 INFO Registered completed upload object_key=6532q0zszdl4cnys1gb39vpngyzd7y8j.narinfo5523worker1 # [ 82.491467] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:37Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55086 duration_s=0 paths=1 nar_bytes_in=6405524worker1 # [ 82.497276] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:550865525worker1 # [ 82.505532] nix-daemon[845]: accepted connection from pid 1208, user nix-grpc-daemon (trusted)5526worker1 # [ 82.513537] nix-daemon[845]: reaped child process 1325, status = succeeded5527client # building '/nix/store/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv' on lb:50051...5528client # farm-b> node-a: building 6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv5529worker1 # [ 82.519750] nix-daemon[845]: accepted connection from pid 1208, user nix-grpc-daemon (trusted)5530client # farm-b> LOG-farm-b-on-standby5531worker1 # [ 83.747327] nix-daemon[845]: reaped child process 1339, status = succeeded5532worker1 # [ 83.895827] nix-daemon[845]: accepted connection from pid 1354, user nix-grpc-daemon (trusted)5533worker1 # [ 83.908350] nix-daemon[845]: reaped child process 1357, status = succeeded5534lb # [ 83.922764] niks3-server[1046]: 2026/09/20 12:21:40 INFO Received uploads request method=POST path=/api/pending_closures5535worker1 # [ 83.933078] nix-grpc-daemon[1278]: time=2026-09-20T12:21:39.173Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5536worker1 # [ 83.935341] nix-grpc-daemon[1278]: time=2026-09-20T12:21:39.175Z level=INFO msg="Uploading 593ccq0xcckfsdii9xb4x5h0bapmahkp-farm-b (184B)"5537lb # [ 83.967136] niks3-server[1046]: 2026/09/20 12:21:40 INFO Registered completed upload object_key=nar/0k21bb46wfcxlq6p91pdvlzw7kkc20p87xpxr7zq5c5zr17k6rda.nar.zst5538lb # [ 83.970887] niks3-server[1046]: 2026/09/20 12:21:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/30/sign5539worker1 # [ 83.979613] nix-grpc-daemon[1278]: time=2026-09-20T12:21:39.219Z level=INFO msg="Uploading 1 narinfos"5540lb # [ 83.979771] niks3-server[1046]: 2026/09/20 12:21:40 INFO Signed narinfos id=30 count=15541lb # [ 83.984635] niks3-server[1046]: 2026/09/20 12:21:40 INFO Registered completed upload object_key=593ccq0xcckfsdii9xb4x5h0bapmahkp.ls5542lb # [ 83.989526] niks3-server[1046]: 2026/09/20 12:21:40 INFO Registered completed upload object_key=log/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv5543lb # [ 83.999532] niks3-server[1046]: 2026/09/20 12:21:40 INFO Received complete upload request method=POST path=/api/pending_closures/30/complete5544lb # [ 84.003300] niks3-server[1046]: 2026/09/20 12:21:40 INFO Completed upload id=305545worker1 # [ 84.004418] nix-grpc-daemon[1278]: time=2026-09-20T12:21:39.244Z level=INFO msg="Upload complete. (276ms)"5546lb # [ 84.004722] niks3-server[1046]: 2026/09/20 12:21:40 INFO Registered completed upload object_key=593ccq0xcckfsdii9xb4x5h0bapmahkp.narinfo5547worker1 # [ 84.007426] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:39Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55086 duration_s=1 drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv assign_id=2 outputs=15548worker1 # [ 84.017722] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:39Z level=debug event=want client=1 drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv cached=05549worker1 # [ 84.019624] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:39Z level=debug event=assigned drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv worker=192.168.1.3:50051 clients=15550worker1 # [ 84.024395] nix-daemon[845]: reaped child process 1337, status = succeeded5551worker1 # [ 84.060992] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:39Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:550865552worker1 # [ 84.067433] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:39Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:550865553worker1 # [ 84.073395] nix-grpc-daemon[1278]: time=2026-09-20T12:21:39.313Z level=INFO msg="All 2 paths already cached"5554client # copying 1 paths to 'lb:50051'...5555client # copying path '/nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv' to 'lb:50051'...5556worker1 # [ 84.077142] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:39Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:550865557worker1 # [ 84.081164] nix-daemon[845]: accepted connection from pid 1208, user nix-grpc-daemon (trusted)5558worker1 # [ 84.203443] nix-daemon[845]: accepted connection from pid 1363, user nix-grpc-daemon (trusted)5559worker1 # [ 84.215854] nix-daemon[845]: reaped child process 1366, status = succeeded5560worker1 # [ 84.366739] nix-daemon[845]: accepted connection from pid 1368, user nix-grpc-daemon (trusted)5561worker1 # [ 84.379571] nix-daemon[845]: reaped child process 1371, status = succeeded5562lb # [ 84.392779] niks3-server[1046]: 2026/09/20 12:21:40 INFO Received uploads request method=POST path=/api/pending_closures5563worker1 # [ 84.397973] nix-grpc-daemon[1278]: time=2026-09-20T12:21:39.638Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5564worker1 # [ 84.399635] nix-grpc-daemon[1278]: time=2026-09-20T12:21:39.640Z level=INFO msg="Uploading bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv (768B)"5565lb # [ 84.417717] niks3-server[1046]: 2026/09/20 12:21:40 INFO Registered completed upload object_key=nar/0y5h6gmqp28k8bkzgrmc35pj8c8qhdi5bc0l17s14j217c1rzscm.nar.zst5566lb # [ 84.432059] niks3-server[1046]: 2026/09/20 12:21:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/31/sign5567lb # [ 84.436578] niks3-server[1046]: 2026/09/20 12:21:40 INFO Signed narinfos id=31 count=15568worker1 # [ 84.442588] nix-grpc-daemon[1278]: time=2026-09-20T12:21:39.682Z level=INFO msg="Uploading 1 narinfos"5569lb # [ 84.446253] niks3-server[1046]: 2026/09/20 12:21:40 INFO Registered completed upload object_key=bgavyqwkalx49wvzslr2znbhxfj7h1mb.ls5570lb # [ 84.459407] niks3-server[1046]: 2026/09/20 12:21:40 INFO Received complete upload request method=POST path=/api/pending_closures/31/complete5571lb # [ 84.464493] niks3-server[1046]: 2026/09/20 12:21:40 INFO Completed upload id=315572worker1 # [ 84.468247] nix-grpc-daemon[1278]: time=2026-09-20T12:21:39.708Z level=INFO msg="Upload complete. (380ms)"5573lb # [ 84.467973] niks3-server[1046]: 2026/09/20 12:21:40 INFO Registered completed upload object_key=bgavyqwkalx49wvzslr2znbhxfj7h1mb.narinfo5574worker1 # [ 84.470515] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:39Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55086 duration_s=0 paths=1 nar_bytes_in=7685575worker1 # [ 84.473959] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:39Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:550865576worker1 # [ 84.479145] nix-daemon[845]: accepted connection from pid 1208, user nix-grpc-daemon (trusted)5577worker1 # [ 84.484058] nix-daemon[845]: reaped child process 1361, status = succeeded5578worker1 # [ 84.489159] nix-daemon[845]: accepted connection from pid 1208, user nix-grpc-daemon (trusted)5579client # building '/nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv' on lb:50051...5580client # farm-top> node-a: building bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv5581client # farm-top> LOG-farm-top-on-standby5582worker1 # [ 85.515675] nix-daemon[845]: reaped child process 1375, status = succeeded5583worker1 # [ 85.675312] nix-daemon[845]: accepted connection from pid 1390, user nix-grpc-daemon (trusted)5584worker1 # [ 85.688765] nix-daemon[845]: reaped child process 1393, status = succeeded5585lb # [ 85.700575] niks3-server[1046]: 2026/09/20 12:21:41 INFO Received uploads request method=POST path=/api/pending_closures5586worker1 # [ 85.712101] nix-grpc-daemon[1278]: time=2026-09-20T12:21:40.952Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5587worker1 # [ 85.713586] nix-grpc-daemon[1278]: time=2026-09-20T12:21:40.954Z level=INFO msg="Uploading bna0adh7g5jsp1sgpmaqnl7brapllfgc-farm-top (240B)"5588lb # [ 85.746506] niks3-server[1046]: 2026/09/20 12:21:41 INFO Registered completed upload object_key=log/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv5589lb # [ 85.750837] niks3-server[1046]: 2026/09/20 12:21:41 INFO Registered completed upload object_key=nar/0bglib3l3asq436b8p5hj4yyk8c77b2qg09pcdiw1w2dfpvnd026.nar.zst5590lb # [ 85.762863] niks3-server[1046]: 2026/09/20 12:21:41 INFO Received sign narinfos request method=POST path=/api/pending_closures/32/sign5591lb # [ 85.767776] niks3-server[1046]: 2026/09/20 12:21:41 INFO Signed narinfos id=32 count=15592worker1 # [ 85.771436] nix-grpc-daemon[1278]: time=2026-09-20T12:21:41.011Z level=INFO msg="Uploading 1 narinfos"5593lb # [ 85.773911] niks3-server[1046]: 2026/09/20 12:21:41 INFO Registered completed upload object_key=bna0adh7g5jsp1sgpmaqnl7brapllfgc.ls5594lb # [ 85.782150] niks3-server[1046]: 2026/09/20 12:21:41 INFO Received complete upload request method=POST path=/api/pending_closures/32/complete5595lb # [ 85.786451] niks3-server[1046]: 2026/09/20 12:21:41 INFO Completed upload id=325596worker1 # [ 85.787363] nix-grpc-daemon[1278]: time=2026-09-20T12:21:41.027Z level=INFO msg="Upload complete. (288ms)"5597lb # [ 85.789051] niks3-server[1046]: 2026/09/20 12:21:41 INFO Registered completed upload object_key=bna0adh7g5jsp1sgpmaqnl7brapllfgc.narinfo5598worker1 # [ 85.790602] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:41Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55086 duration_s=1 drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv assign_id=3 outputs=15599worker1 # [ 85.797592] nix-daemon[845]: reaped child process 1373, status = succeeded5600client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag on-standby >&2, in 7.21 seconds)5601client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag on-standby outPath5602client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag on-standby outPath, in 0.16 seconds)5603worker2 # [ 86.010618] systemd[1]: Listening on Nix worker-protocol over gRPC.5604worker2 # [ 86.012350] systemd[1]: Starting scheduler-token.service...5605worker2 # [ 86.061409] systemd[1]: scheduler-token.service: Deactivated successfully.5606worker2 # [ 86.062983] systemd[1]: Finished scheduler-token.service.5607worker2 # [ 86.064370] systemd[1]: scheduler-token.service: Consumed 17ms CPU time over 50ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5608worker2 # [ 86.069558] systemd[1]: Starting Nix worker-protocol over gRPC...5609worker2 # [ 86.225451] nix-grpc-daemon[1270]: ts=2026-09-20T12:21:41Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5610worker2 # [ 86.227279] nix-grpc-daemon[1270]: ts=2026-09-20T12:21:41Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5611worker2 # [ 86.231426] nix-grpc-daemon[1270]: ts=2026-09-20T12:21:41Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15612worker2 # [ 86.238533] nix-grpc-daemon[1270]: ts=2026-09-20T12:21:41Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5613worker2 # [ 86.240195] systemd[1]: Started Nix worker-protocol over gRPC.5614lb: must succeed: curl -sf localhost:9901/clusters5615worker2 # [ 86.254328] nix-grpc-daemon[1270]: ts=2026-09-20T12:21:41Z level=info event=scheduler_connected addr=lb:500515616worker1 # [ 86.275138] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:41Z level=info event=oidc_jwks provider=mock keys=15617worker1 # [ 86.279276] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:41Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:550845618worker1 # [ 86.283449] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:41Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05619lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5620worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5621worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)5622lb: must succeed: curl -sf localhost:9901/clusters5623lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5624lb: must succeed: curl -sf localhost:9901/clusters5625lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5626worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5627worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)5628(finished: subtest: scheduler down: the other node takes the lock and keeps it, in 13.13 seconds)5629subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise5630client: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag rs5631client # [ 86.562764] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag rs.5632client # Running as unit: rs.service; invocation ID: b89ad3e943e04658815e878602a2370b5633client: (finished: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag rs, in 0.07 seconds)5634worker2 # [ 86.783790] nix-grpc-daemon[1270]: ts=2026-09-20T12:21:42Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:388685635worker2 # [ 86.796483] nix-daemon[937]: accepted connection from pid 1270, user nix-grpc-daemon (trusted)5636client # [ 86.810302] nix[1249]: don't know how to build these paths:5637client # [ 86.811350] nix[1249]: /nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv5638worker1 # [ 86.831097] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:42Z level=debug event=want client=2 drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv cached=05639worker1 # [ 86.832998] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:42Z level=debug event=assigned drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv worker=192.168.1.3:50051 clients=15640worker1 # [ 86.874618] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:42Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:550865641worker1 # [ 86.882449] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:42Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:550865642client # [ 86.875335] nix[1249]: copying 1 paths to 'lb:50051'...5643worker1 # [ 86.890880] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:42Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:550865644client # [ 86.880179] nix[1249]: copying path '/nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv' to 'lb:50051'...5645worker1 # [ 86.900466] nix-daemon[845]: accepted connection from pid 1208, user nix-grpc-daemon (trusted)5646worker1 # [ 87.029502] nix-daemon[845]: accepted connection from pid 1406, user nix-grpc-daemon (trusted)5647worker1 # [ 87.043323] nix-daemon[845]: reaped child process 1409, status = succeeded5648worker1 # [ 87.178210] nix-daemon[845]: accepted connection from pid 1411, user nix-grpc-daemon (trusted)5649worker1 # [ 87.191870] nix-daemon[845]: reaped child process 1414, status = succeeded5650lb # [ 87.199934] niks3-server[1046]: 2026/09/20 12:21:43 INFO Received uploads request method=POST path=/api/pending_closures5651worker1 # [ 87.206471] nix-grpc-daemon[1278]: time=2026-09-20T12:21:42.446Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5652worker1 # [ 87.207869] nix-grpc-daemon[1278]: time=2026-09-20T12:21:42.446Z level=INFO msg="Uploading 2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv (416B)"5653lb # [ 87.225059] niks3-server[1046]: 2026/09/20 12:21:43 INFO Registered completed upload object_key=nar/1p73bzh7zqz68fm48xgidr9p1hfd08x3vp8znnaaycsmagbr8hl6.nar.zst5654lb # [ 87.238325] niks3-server[1046]: 2026/09/20 12:21:43 INFO Received sign narinfos request method=POST path=/api/pending_closures/33/sign5655worker1 # [ 87.246089] nix-grpc-daemon[1278]: time=2026-09-20T12:21:42.486Z level=INFO msg="Uploading 1 narinfos"5656lb # [ 87.245080] niks3-server[1046]: 2026/09/20 12:21:43 INFO Signed narinfos id=33 count=15657lb # [ 87.251280] niks3-server[1046]: 2026/09/20 12:21:43 INFO Registered completed upload object_key=2765168rrfa4dbs7pfx66j88710jx2fm.ls5658lb # [ 87.262711] niks3-server[1046]: 2026/09/20 12:21:43 INFO Received complete upload request method=POST path=/api/pending_closures/33/complete5659lb # [ 87.269333] niks3-server[1046]: 2026/09/20 12:21:43 INFO Completed upload id=335660worker1 # [ 87.270408] nix-grpc-daemon[1278]: time=2026-09-20T12:21:42.510Z level=INFO msg="Upload complete. (365ms)"5661lb # [ 87.271973] niks3-server[1046]: 2026/09/20 12:21:43 INFO Registered completed upload object_key=2765168rrfa4dbs7pfx66j88710jx2fm.narinfo5662worker1 # [ 87.273383] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:42Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55086 duration_s=0 paths=1 nar_bytes_in=4165663worker1 # [ 87.277746] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:42Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:550865664worker1 # [ 87.283464] nix-daemon[845]: accepted connection from pid 1208, user nix-grpc-daemon (trusted)5665worker1 # [ 87.285908] nix-daemon[845]: reaped child process 1404, status = succeeded5666worker1 # [ 87.291217] nix-daemon[845]: accepted connection from pid 1208, user nix-grpc-daemon (trusted)5667client # [ 87.278957] nix[1249]: building '/nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv' on lb:50051...5668lb: must succeed: curl -sf localhost:9901/clusters5669lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5670lb: must succeed: curl -sf localhost:9901/clusters5671lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5672lb: must succeed: curl -sf localhost:9901/clusters5673lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5674lb: must succeed: curl -sf localhost:9901/clusters5675lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5676worker2: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true5677worker2 # [ 89.393880] systemd-journald[286]: Received client request to rotate journal, rotating.5678worker2 # [ 89.416128] systemd-journald[286]: Vacuuming done, freed 0B of archived journals from /var/log/journal/410dbe809b954a878dde8674d91a8ad6.5679worker2: (finished: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true, in 0.07 seconds)5680worker1: must succeed: systemctl restart --no-block nix-grpc-daemon.service5681worker1 # [ 89.344657] systemd[1]: Stopping Nix worker-protocol over gRPC...5682worker1: (finished: must succeed: systemctl restart --no-block nix-grpc-daemon.service, in 0.04 seconds)5683client: must succeed: sleep 5.05684worker1 # [ 90.273902] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:45Z level=info event=draining5685lb # [ 91.477419] envoy[1008]: conn peer=[2001:db8:1::1]:33110 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9620 tx=3854 ms=69875686client: (finished: must succeed: sleep 5.0, in 5.03 seconds)5687worker2: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr='5688worker1 # [ 94.728337] nix-daemon[845]: reaped child process 1418, status = succeeded5689worker1 # [ 94.893689] nix-daemon[845]: accepted connection from pid 1458, user nix-grpc-daemon (trusted)5690worker1 # [ 94.906909] nix-daemon[845]: reaped child process 1461, status = succeeded5691lb # [ 94.919297] niks3-server[1046]: 2026/09/20 12:21:51 INFO Received uploads request method=POST path=/api/pending_closures5692worker1 # [ 94.928058] nix-grpc-daemon[1278]: time=2026-09-20T12:21:50.168Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5693worker1 # [ 94.929763] nix-grpc-daemon[1278]: time=2026-09-20T12:21:50.170Z level=INFO msg="Uploading 6qby8qgv9vwglq2yryw4zqyixrizh5d2-slow-rs (120B)"5694lb # [ 94.963173] niks3-server[1046]: 2026/09/20 12:21:51 INFO Received sign narinfos request method=POST path=/api/pending_closures/34/sign5695worker1 # [ 94.969327] nix-grpc-daemon[1278]: time=2026-09-20T12:21:50.209Z level=INFO msg="Uploading 1 narinfos"5696lb # [ 94.969074] niks3-server[1046]: 2026/09/20 12:21:51 INFO Signed narinfos id=34 count=15697lb # [ 94.976045] niks3-server[1046]: 2026/09/20 12:21:51 INFO Registered completed upload object_key=nar/19z3qdigvc4gggg31dgc0wi5w85pgaq9dk7bvngh0yhckpy55jgq.nar.zst5698lb # [ 94.987239] niks3-server[1046]: 2026/09/20 12:21:51 INFO Received complete upload request method=POST path=/api/pending_closures/34/complete5699lb # [ 94.990936] niks3-server[1046]: 2026/09/20 12:21:51 INFO Completed upload id=345700worker1 # [ 94.991996] nix-grpc-daemon[1278]: time=2026-09-20T12:21:50.232Z level=INFO msg="Upload complete. (286ms)"5701lb # [ 94.994245] niks3-server[1046]: 2026/09/20 12:21:51 INFO Registered completed upload object_key=6qby8qgv9vwglq2yryw4zqyixrizh5d2.ls5702worker1 # [ 94.996336] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:50Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55086 duration_s=7 drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv assign_id=4 outputs=15703lb # [ 94.997486] niks3-server[1046]: 2026/09/20 12:21:51 INFO Registered completed upload object_key=6qby8qgv9vwglq2yryw4zqyixrizh5d2.narinfo5704worker1 # [ 95.002489] nix-daemon[845]: reaped child process 1416, status = succeeded5705lb # [ 95.002284] niks3-server[1046]: 2026/09/20 12:21:51 INFO Registered completed upload object_key=log/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv5706client # [ 95.026090] systemd[1]: rs.service: Deactivated successfully.5707client # [ 95.030877] systemd[1]: rs.service: Consumed 138ms CPU time over 8.466s wall clock time, 39.1M memory peak, 8.5K incoming IP traffic, 9.1K outgoing IP traffic.5708worker1 # [ 95.276458] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:50Z level=info event=signal_exit5709worker1 # [ 95.497172] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:50Z level=info event=scheduler_yield5710lb # [ 95.497855] niks3-server[1046]: 2026/09/20 12:21:51 INFO lead: released remote=192.168.1.3:530485711worker1 # [ 95.503888] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:50Z level=info event=scheduler_restarting addr=lb:500515712worker2 # [ 95.493639] nix-grpc-daemon[1270]: ts=2026-09-20T12:21:50Z level=info event=scheduler_restarting addr=lb:500515713worker1 # [ 95.508388] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:50Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:550845714worker1 # [ 95.511737] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:50Z level=info event=worker_gone addr=192.168.1.3:500515715worker1 # [ 95.514718] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:50Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:550845716worker1 # [ 95.517969] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:50Z level=info event=worker_gone addr=192.168.1.4:500515717worker1 # [ 95.705322] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:50Z level=info event=scheduler_connected addr=lb:500515718worker2 # [ 95.694161] nix-grpc-daemon[1270]: ts=2026-09-20T12:21:51Z level=info event=scheduler_connected addr=lb:500515719worker1 # [ 95.711095] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:50Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"5720worker2 # [ 95.700138] nix-grpc-daemon[1270]: ts=2026-09-20T12:21:51Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"5721worker2: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr=', in 1.10 seconds)5722client: waiting for success: ! systemctl is-active rs5723client: (finished: waiting for success: ! systemctl is-active rs, in 0.06 seconds)5724client: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }5725client: (finished: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }, in 0.03 seconds)5726client: must fail: journalctl -u rs | grep -q 'reconnecting'5727client: (finished: must fail: journalctl -u rs | grep -q 'reconnecting', in 0.03 seconds)5728lb: must succeed: curl -sf localhost:9901/clusters5729worker1 # [ 95.911529] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:51Z level=info event=scheduler_connected addr=lb:500515730worker1 # [ 95.913372] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5731worker2 # [ 95.901472] nix-grpc-daemon[1270]: ts=2026-09-20T12:21:51Z level=info event=scheduler_connected addr=lb:500515732worker2 # [ 95.905314] nix-grpc-daemon[1270]: ts=2026-09-20T12:21:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5733lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5734worker1 # [ 96.114430] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:51Z level=info event=scheduler_connected addr=lb:500515735worker1 # [ 96.116638] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5736worker2 # [ 96.106346] nix-grpc-daemon[1270]: ts=2026-09-20T12:21:51Z level=info event=scheduler_connected addr=lb:500515737worker2 # [ 96.110344] nix-grpc-daemon[1270]: ts=2026-09-20T12:21:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5738lb # [ 96.260713] niks3-server[1046]: 2026/09/20 12:21:52 INFO lead: acquired remote=192.168.1.4:424285739worker2 # [ 96.252879] nix-grpc-daemon[1270]: ts=2026-09-20T12:21:51Z level=info event=scheduler_take_over5740worker1 # [ 96.317797] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:51Z level=info event=scheduler_connected addr=lb:500515741worker1 # [ 96.321744] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5742worker2 # [ 96.311146] nix-grpc-daemon[1270]: ts=2026-09-20T12:21:51Z level=info event=scheduler_connected addr=lb:500515743worker2 # [ 96.314890] nix-grpc-daemon[1270]: ts=2026-09-20T12:21:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5744worker1 # [ 96.501150] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:51Z level=info event=scheduler_restarting5745worker2 # [ 96.517935] nix-grpc-daemon[1270]: ts=2026-09-20T12:21:51Z level=info event=scheduler_connected addr=lb:500515746worker1 # [ 96.535255] nix-daemon[845]: reaped child process 1269, status = succeeded5747worker2 # [ 96.522224] nix-grpc-daemon[1270]: ts=2026-09-20T12:21:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5748worker1 # [ 96.537599] nix-grpc-daemon[1208]: ts=2026-09-20T12:21:51Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5749worker2 # [ 96.718473] nix-grpc-daemon[1270]: ts=2026-09-20T12:21:52Z level=info event=scheduler_connected addr=lb:500515750worker2 # [ 96.722421] nix-grpc-daemon[1270]: ts=2026-09-20T12:21:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5751worker1 # [ 96.884459] nix-grpc-daemon[1208]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5752worker1 # [ 96.886196] nix-grpc-daemon[1208]: E0000 00:00:1789906912.124381 1251 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 16, started true, shutdown true5753worker1 # [ 96.888510] nix-grpc-daemon[1208]: E0000 00:00:1789906912.124909 1251 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5754lb: must succeed: curl -sf localhost:9901/clusters5755worker2 # [ 96.923700] nix-grpc-daemon[1270]: ts=2026-09-20T12:21:52Z level=info event=scheduler_connected addr=lb:500515756worker2 # [ 96.927641] nix-grpc-daemon[1270]: ts=2026-09-20T12:21:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5757lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)5758worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5759worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)5760worker2 # [ 97.128551] nix-grpc-daemon[1270]: ts=2026-09-20T12:21:52Z level=info event=scheduler_connected addr=lb:500515761worker2 # [ 97.130673] nix-grpc-daemon[1270]: ts=2026-09-20T12:21:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5762worker2 # [ 97.331568] nix-grpc-daemon[1270]: ts=2026-09-20T12:21:52Z level=info event=scheduler_connected addr=lb:500515763worker2 # [ 97.333308] nix-grpc-daemon[1270]: ts=2026-09-20T12:21:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5764worker2 # [ 97.535087] nix-grpc-daemon[1270]: ts=2026-09-20T12:21:52Z level=info event=scheduler_connected addr=lb:500515765worker2 # [ 97.546149] nix-grpc-daemon[1270]: ts=2026-09-20T12:21:52Z level=info event=oidc_jwks provider=mock keys=15766worker2 # [ 97.550398] nix-grpc-daemon[1270]: ts=2026-09-20T12:21:52Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:333605767worker2 # [ 97.554306] nix-grpc-daemon[1270]: ts=2026-09-20T12:21:52Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05768lb: must succeed: curl -sf localhost:9901/clusters5769lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)5770worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5771worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)5772worker1 # [ 98.413803] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5773worker1 # [ 98.418244] systemd[1]: Stopped Nix worker-protocol over gRPC.5774worker1 # [ 98.420615] systemd[1]: nix-grpc-daemon.service: Consumed 1.294s CPU time over 26.319s wall clock time, 33.5M memory peak, 49.4K incoming IP traffic, 97K outgoing IP traffic.5775worker1 # [ 98.439535] systemd[1]: Starting Nix worker-protocol over gRPC...5776worker1 # [ 98.588454] nix-grpc-daemon[1468]: ts=2026-09-20T12:21:53Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5777worker1 # [ 98.590216] nix-grpc-daemon[1468]: ts=2026-09-20T12:21:53Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5778worker1 # [ 98.593911] nix-grpc-daemon[1468]: ts=2026-09-20T12:21:53Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15779worker1 # [ 98.600704] nix-grpc-daemon[1468]: ts=2026-09-20T12:21:53Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5780worker1 # [ 98.602299] systemd[1]: Started Nix worker-protocol over gRPC.5781worker1 # [ 98.614796] nix-grpc-daemon[1468]: ts=2026-09-20T12:21:53Z level=info event=scheduler_connected addr=lb:500515782worker2 # [ 98.605281] nix-grpc-daemon[1270]: ts=2026-09-20T12:21:53Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:333605783worker2 # [ 98.607255] nix-grpc-daemon[1270]: ts=2026-09-20T12:21:53Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05784lb: must succeed: curl -sf localhost:9901/clusters5785lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5786worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5787worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.03 seconds)5788(finished: subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise, in 12.70 seconds)5789subtest: scheduler restart mid-build: build finishes, no double build5790client: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag mid5791client # [ 99.240660] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag mid.5792client # Running as unit: mid.service; invocation ID: adffc7c6a2cd4ab0ad070710e245113b5793client: (finished: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag mid, in 0.04 seconds)5794worker2 # [ 99.438400] nix-grpc-daemon[1270]: ts=2026-09-20T12:21:54Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:388685795client # [ 99.444839] nix[1285]: don't know how to build these paths:5796client # [ 99.446204] nix[1285]: /nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv5797worker2 # [ 99.454160] nix-grpc-daemon[1270]: ts=2026-09-20T12:21:54Z level=debug event=want client=1 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv cached=05798worker2 # [ 99.458730] nix-grpc-daemon[1270]: ts=2026-09-20T12:21:54Z level=debug event=assigned drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv worker=192.168.1.4:50051 clients=15799worker2 # [ 99.503563] nix-grpc-daemon[1270]: ts=2026-09-20T12:21:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:388685800worker2 # [ 99.517082] nix-grpc-daemon[1270]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 43 ms (attempt 1/5)5801worker2 # [ 99.565729] nix-grpc-daemon[1270]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 43 ms (attempt 2/5)5802worker2 # [ 99.612229] nix-grpc-daemon[1270]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 374 ms (attempt 3/5)5803worker2 # [ 99.992054] nix-grpc-daemon[1270]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 727 ms (attempt 4/5)5804worker2 # [ 100.726114] nix-grpc-daemon[1270]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5805worker2 # [ 100.740542] nix-grpc-daemon[1270]: ts=2026-09-20T12:21:56Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:388685806client # [ 100.748918] nix[1285]: copying 1 paths to 'lb:50051'...5807worker2 # [ 100.753107] nix-grpc-daemon[1270]: ts=2026-09-20T12:21:56Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:388685808client # [ 100.752982] nix[1285]: copying path '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' to 'lb:50051'...5809worker2 # [ 100.761786] nix-daemon[937]: accepted connection from pid 1270, user nix-grpc-daemon (trusted)5810worker2 # [ 100.772296] nix-grpc-daemon[1270]: ts=2026-09-20T12:21:56Z level=info event=niks3_push_started pid=13895811worker2 # [ 100.962892] nix-grpc-daemon[1389]: time=2026-09-20T12:21:56.284Z level=INFO msg="Configuring client TLS" cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/worker2.pem key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/worker2.key ca=""5812worker2 # [ 101.119477] nix-daemon[937]: accepted connection from pid 1402, user nix-grpc-daemon (trusted)5813worker2 # [ 101.130719] nix-daemon[937]: reaped child process 1405, status = succeeded5814worker2 # [ 101.303176] nix-daemon[937]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)5815worker2 # [ 101.315158] nix-daemon[937]: reaped child process 1410, status = succeeded5816lb # [ 101.343614] niks3-server[1046]: 2026/09/20 12:21:57 INFO Received uploads request method=POST path=/api/pending_closures5817worker2 # [ 101.338917] nix-grpc-daemon[1389]: time=2026-09-20T12:21:56.660Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5818worker2 # [ 101.341329] nix-grpc-daemon[1389]: time=2026-09-20T12:21:56.662Z level=INFO msg="Uploading n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv (424B)"5819lb # [ 101.367775] niks3-server[1046]: 2026/09/20 12:21:57 INFO Registered completed upload object_key=nar/0pdyhjlvyi83rdx6rr15l7kj7a7hlcbqgllxm29xw3hvr3akprgq.nar.zst5820lb # [ 101.372978] niks3-server[1046]: 2026/09/20 12:21:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/35/sign5821lb # [ 101.375961] niks3-server[1046]: 2026/09/20 12:21:57 INFO Signed narinfos id=35 count=15822worker2 # [ 101.367137] nix-grpc-daemon[1389]: time=2026-09-20T12:21:56.688Z level=INFO msg="Uploading 1 narinfos"5823lb # [ 101.381334] niks3-server[1046]: 2026/09/20 12:21:57 INFO Registered completed upload object_key=n23kf6h4k511p0qpzsydmqrbq2qn9g5w.ls5824lb # [ 101.387063] niks3-server[1046]: 2026/09/20 12:21:57 INFO Received complete upload request method=POST path=/api/pending_closures/35/complete5825worker2 # [ 101.382886] nix-grpc-daemon[1389]: time=2026-09-20T12:21:56.704Z level=INFO msg="Upload complete. (415ms)"5826lb # [ 101.394308] niks3-server[1046]: 2026/09/20 12:21:57 INFO Completed upload id=355827worker2 # [ 101.386070] nix-grpc-daemon[1270]: ts=2026-09-20T12:21:56Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38868 duration_s=0 paths=1 nar_bytes_in=4245828lb # [ 101.397983] niks3-server[1046]: 2026/09/20 12:21:57 INFO Registered completed upload object_key=n23kf6h4k511p0qpzsydmqrbq2qn9g5w.narinfo5829worker2 # [ 101.389742] nix-grpc-daemon[1270]: ts=2026-09-20T12:21:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:388685830worker2 # [ 101.395133] nix-daemon[937]: accepted connection from pid 1270, user nix-grpc-daemon (trusted)5831worker2 # [ 101.397446] nix-daemon[937]: reaped child process 1387, status = succeeded5832worker2 # [ 101.402942] nix-daemon[937]: accepted connection from pid 1270, user nix-grpc-daemon (trusted)5833client # [ 101.402282] nix[1285]: building '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' on lb:50051...5834lb # [ 101.478421] envoy[1008]: conn peer=[2001:db8:1::1]:48188 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2947 tx=1348 ms=82265835lb # [ 101.481667] envoy[1008]: conn peer=[2001:db8:1::3]:35360 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=5568 tx=1321 ms=242475836lb: must succeed: curl -sf localhost:9901/clusters5837lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5838lb: must succeed: curl -sf localhost:9901/clusters5839lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5840worker2: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service5841worker2 # [ 102.927272] systemd[1]: nix-grpc-daemon.service: Sent signal SIGKILL to main process 1270 (nix-grpc-daemon) on client request.5842worker2 # [ 102.930362] systemd[1]: nix-grpc-daemon.service: Killed unit cgroup '/system.slice/nix-grpc-daemon.service' with SIGKILL on client request.5843worker1 # [ 102.947330] nix-grpc-daemon[1468]: ts=2026-09-20T12:21:58Z level=info event=scheduler_disconnected addr=lb:50051 error="upstream connect error or disconnect/reset before headers. reset reason: connection termination"5844lb # [ 102.948527] niks3-server[1046]: 2026/09/20 12:21:59 INFO lead: released remote=192.168.1.4:424285845client # [ 102.936884] nix[1285]: n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), asking the scheduler again5846client # [ 102.941724] nix[1285]: scheduler at lb:50051: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), reconnecting5847worker2 # [ 102.949712] systemd[1]: Starting scheduler-token.service...5848worker2 # [ 102.961294] systemd[1]: nix-grpc-daemon.service: Main process exited, code=killed, status=9/KILL5849worker2 # [ 102.963940] systemd[1]: nix-grpc-daemon.service: Failed with result 'signal'.5850worker2 # [ 102.965454] systemd[1]: nix-grpc-daemon.service: Consumed 440ms CPU time over 16.891s wall clock time, 31.3M memory peak, 17.2K incoming IP traffic, 39.1K outgoing IP traffic.5851worker2 # [ 102.974946] nix-daemon[937]: reaped child process 1326, status = succeeded5852worker2 # [ 102.976436] nix-daemon[937]: reaped child process 1412, status = succeeded5853worker2 # [ 102.986242] nix-daemon[1414]: unexpected Nix daemon error: error: interrupted by the user5854worker2 # [ 102.990486] nix-daemon[937]: reaped child process 1414, status = failed with exit code 15855worker2 # [ 103.027551] systemd[1]: scheduler-token.service: Deactivated successfully.5856worker2 # [ 103.030273] systemd[1]: Finished scheduler-token.service.5857worker2 # [ 103.032260] systemd[1]: scheduler-token.service: Consumed 17ms CPU time over 80ms wall clock time, 2.2M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5858worker2 # [ 103.034424] systemd[1]: nix-grpc-daemon.service: Scheduled restart job immediately on client request, restart counter is at 1.5859worker2 # [ 103.038851] systemd[1]: Starting scheduler-token.service...5860worker2 # [ 103.097460] systemd[1]: scheduler-token.service: Deactivated successfully.5861worker2 # [ 103.099151] systemd[1]: Finished scheduler-token.service.5862worker2 # [ 103.100373] systemd[1]: scheduler-token.service: Consumed 17ms CPU time over 60ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5863worker2 # [ 103.105713] systemd[1]: Starting Nix worker-protocol over gRPC...5864worker1 # [ 103.148071] nix-grpc-daemon[1468]: ts=2026-09-20T12:21:58Z level=info event=scheduler_connected addr=lb:500515865worker2 # [ 103.268989] nix-grpc-daemon[1456]: ts=2026-09-20T12:21:58Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5866worker2 # [ 103.271131] nix-grpc-daemon[1456]: ts=2026-09-20T12:21:58Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5867worker2 # [ 103.274388] nix-grpc-daemon[1456]: ts=2026-09-20T12:21:58Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15868worker2 # [ 103.280734] nix-grpc-daemon[1456]: ts=2026-09-20T12:21:58Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5869worker2 # [ 103.282324] systemd[1]: Started Nix worker-protocol over gRPC.5870worker2: (finished: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service, in 0.40 seconds)5871client: must succeed: sleep 5.05872worker2 # [ 103.292522] nix-grpc-daemon[1456]: ts=2026-09-20T12:21:58Z level=info event=scheduler_take_over5873lb # [ 103.303642] niks3-server[1046]: 2026/09/20 12:21:59 INFO lead: acquired remote=192.168.1.4:594805874worker2 # [ 103.299555] nix-grpc-daemon[1456]: ts=2026-09-20T12:21:58Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:406765875worker2 # [ 103.301894] nix-grpc-daemon[1456]: ts=2026-09-20T12:21:58Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05876worker2 # [ 103.305054] nix-grpc-daemon[1456]: ts=2026-09-20T12:21:58Z level=info event=scheduler_connected addr=lb:500515877worker2 # [ 103.310061] nix-grpc-daemon[1456]: ts=2026-09-20T12:21:58Z level=info event=oidc_jwks provider=mock keys=15878worker2 # [ 103.311395] nix-grpc-daemon[1456]: ts=2026-09-20T12:21:58Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:406765879worker2 # [ 103.312987] nix-grpc-daemon[1456]: ts=2026-09-20T12:21:58Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05880worker2 # [ 103.451690] nix-grpc-daemon[1456]: ts=2026-09-20T12:21:58Z level=debug event=want client=1 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv cached=05881worker2 # [ 103.455994] nix-grpc-daemon[1456]: ts=2026-09-20T12:21:58Z level=debug event=assigned drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv worker=192.168.1.3:50051 clients=15882worker1 # [ 103.515962] nix-grpc-daemon[1468]: ts=2026-09-20T12:21:58Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:409525883worker1 # [ 103.527137] nix-daemon[845]: accepted connection from pid 1468, user nix-grpc-daemon (trusted)5884worker1 # [ 103.545601] nix-grpc-daemon[1468]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 32 ms (attempt 1/5)5885worker1 # [ 103.583421] nix-grpc-daemon[1468]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 16 ms (attempt 2/5)5886worker1 # [ 103.604997] nix-grpc-daemon[1468]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 7 ms (attempt 3/5)5887worker1 # [ 103.614415] nix-grpc-daemon[1468]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 153 ms (attempt 4/5)5888worker1 # [ 103.770175] nix-grpc-daemon[1468]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5889worker1 # [ 103.788624] nix-grpc-daemon[1468]: copying path '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' from 'http://lb:5751'...5890worker1 # [ 103.808386] nix-daemon[845]: accepted connection from pid 1468, user nix-grpc-daemon (trusted)5891client # [ 103.810728] nix[1285]: building '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' on lb:50051...5892worker1 # [ 103.824546] nix-daemon[845]: accepted connection from pid 1468, user nix-grpc-daemon (trusted)5893client: (finished: must succeed: sleep 5.0, in 5.04 seconds)5894client: waiting for success: ! systemctl is-active mid5895worker1 # [ 108.647491] nix-grpc-daemon[1468]: ts=2026-09-20T12:22:03Z level=info event=niks3_push_started pid=15695896worker1 # [ 108.679365] nix-daemon[845]: reaped child process 1545, status = succeeded5897worker1 # [ 108.843415] nix-grpc-daemon[1569]: time=2026-09-20T12:22:04.083Z level=INFO msg="Configuring client TLS" cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/worker1.pem key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/worker1.key ca=""5898worker1 # [ 108.980326] nix-daemon[845]: accepted connection from pid 1582, user nix-grpc-daemon (trusted)5899worker1 # [ 108.991053] nix-daemon[845]: reaped child process 1585, status = succeeded5900lb # [ 109.001792] niks3-server[1046]: 2026/09/20 12:22:05 INFO Received uploads request method=POST path=/api/pending_closures5901worker1 # [ 109.011958] nix-grpc-daemon[1569]: time=2026-09-20T12:22:04.252Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5902worker1 # [ 109.013434] nix-grpc-daemon[1569]: time=2026-09-20T12:22:04.253Z level=INFO msg="Uploading cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj-slow-mid (120B)"5903lb # [ 109.049227] niks3-server[1046]: 2026/09/20 12:22:05 INFO Registered completed upload object_key=nar/0462jb7zwy2d73k3q6v09xnk3bx318v8ik5ckmp32w2hzvwj4k2d.nar.zst5904lb # [ 109.059305] niks3-server[1046]: 2026/09/20 12:22:05 INFO Received sign narinfos request method=POST path=/api/pending_closures/36/sign5905lb # [ 109.070066] niks3-server[1046]: 2026/09/20 12:22:05 INFO Registered completed upload object_key=log/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv5906lb # [ 109.082871] niks3-server[1046]: 2026/09/20 12:22:05 INFO Signed narinfos id=36 count=15907worker1 # [ 109.084092] nix-grpc-daemon[1569]: time=2026-09-20T12:22:04.324Z level=INFO msg="Uploading 1 narinfos"5908lb # [ 109.093585] niks3-server[1046]: 2026/09/20 12:22:05 INFO Registered completed upload object_key=cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj.ls5909lb # [ 109.096847] niks3-server[1046]: 2026/09/20 12:22:05 INFO Received complete upload request method=POST path=/api/pending_closures/36/complete5910lb # [ 109.102906] niks3-server[1046]: 2026/09/20 12:22:05 INFO Completed upload id=365911worker1 # [ 109.104388] nix-grpc-daemon[1569]: time=2026-09-20T12:22:04.344Z level=INFO msg="Upload complete. (257ms)"5912lb # [ 109.108303] niks3-server[1046]: 2026/09/20 12:22:05 INFO Registered completed upload object_key=cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj.narinfo5913worker1 # [ 109.110177] nix-grpc-daemon[1468]: ts=2026-09-20T12:22:04Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40952 duration_s=5 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv assign_id=1 outputs=15914worker1 # [ 109.123052] nix-daemon[845]: reaped child process 1543, status = succeeded5915client # [ 109.126096] systemd[1]: mid.service: Deactivated successfully.5916client # [ 109.128073] systemd[1]: mid.service: Consumed 132ms CPU time over 9.887s wall clock time, 37.3M memory peak, 9.6K incoming IP traffic, 10.6K outgoing IP traffic.5917client: (finished: waiting for success: ! systemctl is-active mid, in 1.11 seconds)5918client: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }5919client: (finished: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }, in 0.04 seconds)5920(finished: subtest: scheduler restart mid-build: build finishes, no double build, in 10.50 seconds)5921subtest: interrupting the client stops the build on the worker5922client: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag intr5923client # [ 109.744764] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag intr.5924client # Running as unit: intr.service; invocation ID: 30d86a0581a3483d9f575f0ea40ab19b5925client: (finished: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag intr, in 0.04 seconds)5926worker2 # [ 109.960741] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:05Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:407045927worker2 # [ 109.972322] nix-daemon[937]: accepted connection from pid 1456, user nix-grpc-daemon (trusted)5928client # [ 109.984461] nix[1321]: don't know how to build these paths:5929client # [ 109.985393] nix[1321]: /nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv5930worker2 # [ 109.993324] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:05Z level=debug event=want client=2 drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv cached=05931worker2 # [ 109.995180] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:05Z level=debug event=assigned drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv worker=192.168.1.4:50051 clients=15932worker2 # [ 110.042578] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:05Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:407045933worker2 # [ 110.054389] nix-grpc-daemon[1456]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 85 ms (attempt 1/5)5934worker2 # [ 110.145243] nix-grpc-daemon[1456]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 71 ms (attempt 2/5)5935worker2 # [ 110.216876] nix-grpc-daemon[1456]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 143 ms (attempt 3/5)5936worker2 # [ 110.365853] nix-grpc-daemon[1456]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 605 ms (attempt 4/5)5937worker2 # [ 110.979536] nix-grpc-daemon[1456]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5938worker2 # [ 110.991651] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:06Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:407045939client # [ 110.998111] nix[1321]: copying 1 paths to 'lb:50051'...5940client # [ 110.999856] nix[1321]: copying path '/nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv' to 'lb:50051'...5941worker2 # [ 111.000852] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:06Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:407045942worker2 # [ 111.006353] nix-daemon[937]: accepted connection from pid 1456, user nix-grpc-daemon (trusted)5943worker2 # [ 111.016064] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:06Z level=info event=niks3_push_started pid=15325944worker2 # [ 111.192446] nix-grpc-daemon[1532]: time=2026-09-20T12:22:06.513Z level=INFO msg="Configuring client TLS" cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/worker2.pem key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/worker2.key ca=""5945worker2 # [ 111.353786] nix-daemon[937]: accepted connection from pid 1545, user nix-grpc-daemon (trusted)5946worker2 # [ 111.364988] nix-daemon[937]: reaped child process 1548, status = succeeded5947lb # [ 111.479760] envoy[1008]: conn peer=[2001:db8:1::4]:37554 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=9471 tx=674 ms=166835948lb # [ 111.483767] envoy[1008]: conn peer=[2001:db8:1::1]:38484 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3835 tx=1659 ms=96785949worker2 # [ 111.538238] nix-daemon[937]: accepted connection from pid 1550, user nix-grpc-daemon (trusted)5950worker2 # [ 111.550546] nix-daemon[937]: reaped child process 1553, status = succeeded5951lb # [ 111.577839] niks3-server[1046]: 2026/09/20 12:22:07 INFO Received uploads request method=POST path=/api/pending_closures5952worker2 # [ 111.602374] nix-grpc-daemon[1532]: time=2026-09-20T12:22:06.924Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5953worker2 # [ 111.604286] nix-grpc-daemon[1532]: time=2026-09-20T12:22:06.925Z level=INFO msg="Uploading x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv (424B)"5954lb # [ 111.644619] niks3-server[1046]: 2026/09/20 12:22:07 INFO Registered completed upload object_key=nar/0dw6lfdicxarqmcw4m9j94irjhgkdvc1nww51zwj454yj03v5iri.nar.zst5955lb # [ 111.650666] niks3-server[1046]: 2026/09/20 12:22:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/37/sign5956lb # [ 111.655704] niks3-server[1046]: 2026/09/20 12:22:07 INFO Signed narinfos id=37 count=15957worker2 # [ 111.648423] nix-grpc-daemon[1532]: time=2026-09-20T12:22:06.969Z level=INFO msg="Uploading 1 narinfos"5958lb # [ 111.666633] niks3-server[1046]: 2026/09/20 12:22:07 INFO Registered completed upload object_key=x43jxvh618x0k4fbhc2wyxhr1lymdssr.ls5959lb # [ 111.675443] niks3-server[1046]: 2026/09/20 12:22:07 INFO Received complete upload request method=POST path=/api/pending_closures/37/complete5960lb # [ 111.686066] niks3-server[1046]: 2026/09/20 12:22:07 INFO Completed upload id=375961lb # [ 111.687558] niks3-server[1046]: 2026/09/20 12:22:07 INFO Registered completed upload object_key=x43jxvh618x0k4fbhc2wyxhr1lymdssr.narinfo5962worker2 # [ 111.675375] nix-grpc-daemon[1532]: time=2026-09-20T12:22:06.996Z level=INFO msg="Upload complete. (478ms)"5963worker2 # [ 111.680064] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:07Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40704 duration_s=0 paths=1 nar_bytes_in=4245964worker2 # [ 111.686652] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:407045965worker2 # [ 111.698109] nix-daemon[937]: accepted connection from pid 1456, user nix-grpc-daemon (trusted)5966worker2 # [ 111.707670] nix-daemon[937]: reaped child process 1530, status = succeeded5967client # [ 111.712905] nix[1321]: building '/nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv' on lb:50051...5968worker2 # [ 111.714613] nix-daemon[937]: accepted connection from pid 1456, user nix-grpc-daemon (trusted)5969client: must succeed: systemctl kill -s INT intr5970client # [ 113.259728] systemd[1]: intr.service: Sent signal SIGINT to main process 1321 (nix) on client request.5971client # [ 113.262924] nix[1321]: shutting downx43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED)5972client: (finished: must succeed: systemctl kill -s INT intr, in 0.04 seconds)5973worker2 # [ 113.264474] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:08Z level=info event=revoked_running drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv5974client # [ 113.266642] nix[1321]: error: interrupted by the user5975client # [ 113.287995] systemd[1]: intr.service: Main process exited, code=exited, status=1/FAILURE5976client # [ 113.289432] systemd[1]: intr.service: Failed with result 'exit-code'.5977client # [ 113.291860] systemd[1]: intr.service: Consumed 113ms CPU time over 3.546s wall clock time, 39.2M memory peak, 7.7K incoming IP traffic, 8.7K outgoing IP traffic.5978worker2 # [ 113.337183] nix-daemon[1557]: unexpected Nix daemon error: error: interrupted by the user5979worker2 # [ 113.345438] nix-daemon[937]: reaped child process 1555, status = succeeded5980worker2 # [ 113.346579] nix-daemon[937]: reaped child process 1557, status = failed with exit code 15981(finished: subtest: interrupting the client stops the build on the worker, in 3.66 seconds)5982subtest: a deploy mid-build drains: the build finishes, the new generation takes over5983client: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag sw5984client # [ 113.405876] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag sw.5985client # Running as unit: sw.service; invocation ID: 3716f8b431c64ce285070ca189c919445986client: (finished: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag sw, in 0.04 seconds)5987client # [ 113.628343] nix[1341]: don't know how to build these paths:5988client # [ 113.629879] nix[1341]: /nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv5989worker2 # [ 113.627352] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:08Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:407045990worker2 # [ 113.639138] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:08Z level=debug event=want client=3 drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv cached=05991worker2 # [ 113.643585] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:08Z level=debug event=assigned drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv worker=192.168.1.3:50051 clients=15992worker1 # [ 113.696673] nix-grpc-daemon[1468]: ts=2026-09-20T12:22:08Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:409525993worker1 # [ 113.709285] nix-grpc-daemon[1468]: ts=2026-09-20T12:22:08Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:409525994client # [ 113.704167] nix[1341]: copying 1 paths to 'lb:50051'...5995worker1 # [ 113.717121] nix-grpc-daemon[1468]: ts=2026-09-20T12:22:08Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:409525996client # [ 113.705247] nix[1341]: copying path '/nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv' to 'lb:50051'...5997worker1 # [ 113.723639] nix-daemon[845]: accepted connection from pid 1468, user nix-grpc-daemon (trusted)5998worker1 # [ 113.887458] nix-daemon[845]: accepted connection from pid 1613, user nix-grpc-daemon (trusted)5999worker1 # [ 113.900142] nix-daemon[845]: reaped child process 1616, status = succeeded6000worker1 # [ 114.052145] nix-daemon[845]: accepted connection from pid 1618, user nix-grpc-daemon (trusted)6001worker1 # [ 114.063844] nix-daemon[845]: reaped child process 1621, status = succeeded6002lb # [ 114.079431] niks3-server[1046]: 2026/09/20 12:22:10 INFO Received uploads request method=POST path=/api/pending_closures6003worker1 # [ 114.172917] nix-grpc-daemon[1569]: time=2026-09-20T12:22:09.412Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6004worker1 # [ 114.178671] nix-grpc-daemon[1569]: time=2026-09-20T12:22:09.416Z level=INFO msg="Uploading q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv (416B)"6005lb # [ 114.197809] niks3-server[1046]: 2026/09/20 12:22:10 INFO Registered completed upload object_key=nar/1rby34xp94wipnqa4abnh2771jnqp34zv82lymaabrhj07vmwiix.nar.zst6006lb # [ 114.206987] niks3-server[1046]: 2026/09/20 12:22:10 INFO Received sign narinfos request method=POST path=/api/pending_closures/38/sign6007lb # [ 114.213179] niks3-server[1046]: 2026/09/20 12:22:10 INFO Signed narinfos id=38 count=16008worker1 # [ 114.214265] nix-grpc-daemon[1569]: time=2026-09-20T12:22:09.454Z level=INFO msg="Uploading 1 narinfos"6009lb # [ 114.220943] niks3-server[1046]: 2026/09/20 12:22:10 INFO Registered completed upload object_key=q9pl41yvj7mi4fqnyiarzy5py0107h7b.ls6010lb # [ 114.233227] niks3-server[1046]: 2026/09/20 12:22:10 INFO Received complete upload request method=POST path=/api/pending_closures/38/complete6011lb # [ 114.240770] niks3-server[1046]: 2026/09/20 12:22:10 INFO Completed upload id=386012worker1 # [ 114.242070] nix-grpc-daemon[1569]: time=2026-09-20T12:22:09.482Z level=INFO msg="Upload complete. (511ms)"6013lb # [ 114.244510] niks3-server[1046]: 2026/09/20 12:22:10 INFO Registered completed upload object_key=q9pl41yvj7mi4fqnyiarzy5py0107h7b.narinfo6014worker1 # [ 114.247160] nix-grpc-daemon[1468]: ts=2026-09-20T12:22:09Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40952 duration_s=0 paths=1 nar_bytes_in=4166015worker1 # [ 114.252601] nix-grpc-daemon[1468]: ts=2026-09-20T12:22:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:409526016worker1 # [ 114.265131] nix-daemon[845]: accepted connection from pid 1468, user nix-grpc-daemon (trusted)6017worker1 # [ 114.273473] nix-daemon[845]: reaped child process 1611, status = succeeded6018client # [ 114.268642] nix[1341]: building '/nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv' on lb:50051...6019worker1 # [ 114.282747] nix-daemon[845]: accepted connection from pid 1468, user nix-grpc-daemon (trusted)6020worker1: must succeed: systemctl reload nix-grpc-daemon.service6021worker1 # [ 115.948521] systemd[1]: Reloading Nix worker-protocol over gRPC...6022worker1 # [ 115.989408] systemd[1]: Reloaded Nix worker-protocol over gRPC.6023worker1: (finished: must succeed: systemctl reload nix-grpc-daemon.service, in 0.08 seconds)6024worker1: must succeed: pgrep -f 'read -t [9]0 x'6025worker1: (finished: must succeed: pgrep -f 'read -t [9]0 x', in 0.04 seconds)6026client: must succeed: nix build -L --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag during-drain >&26027client # don't know how to build these paths:6028client # /nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6029worker2 # [ 116.238988] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:11Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:407046030worker1 # [ 116.256434] nix-grpc-daemon[1468]: ts=2026-09-20T12:22:11Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:409526031worker2 # [ 116.264728] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:11Z level=debug event=want client=4 drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv cached=06032worker2 # [ 116.269653] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:11Z level=debug event=assigned drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv worker=192.168.1.4:50051 clients=16033worker2 # [ 116.308901] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:11Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:407046034worker2 # [ 116.322390] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:11Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:407046035client # copying 1 paths to 'lb:50051'...6036worker2 # [ 116.334218] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:11Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:407046037client # copying path '/nix/store/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv' to 'lb:50051'...6038worker2 # [ 116.346442] nix-daemon[937]: accepted connection from pid 1456, user nix-grpc-daemon (trusted)6039worker2 # [ 116.528377] nix-daemon[937]: accepted connection from pid 1607, user nix-grpc-daemon (trusted)6040worker2 # [ 116.540435] nix-daemon[937]: reaped child process 1610, status = succeeded6041worker1 # [ 116.606692] nix-grpc-daemon[1468]: ts=2026-09-20T12:22:11Z level=info event=draining6042worker2 # [ 116.697806] nix-daemon[937]: accepted connection from pid 1612, user nix-grpc-daemon (trusted)6043worker2 # [ 116.709721] nix-daemon[937]: reaped child process 1615, status = succeeded6044lb # [ 116.738608] niks3-server[1046]: 2026/09/20 12:22:12 INFO Received uploads request method=POST path=/api/pending_closures6045worker2 # [ 116.735186] nix-grpc-daemon[1532]: time=2026-09-20T12:22:12.056Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6046worker2 # [ 116.737083] nix-grpc-daemon[1532]: time=2026-09-20T12:22:12.058Z level=INFO msg="Uploading s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv (528B)"6047lb # [ 116.773628] niks3-server[1046]: 2026/09/20 12:22:12 INFO Received sign narinfos request method=POST path=/api/pending_closures/39/sign6048lb # [ 116.779142] niks3-server[1046]: 2026/09/20 12:22:12 INFO Registered completed upload object_key=nar/0n1m04akn0svzkx0szkj4q7kvaw06fa9yx6k2jibkbkkv85mm55n.nar.zst6049lb # [ 116.787654] niks3-server[1046]: 2026/09/20 12:22:12 INFO Signed narinfos id=39 count=16050worker2 # [ 116.780605] nix-grpc-daemon[1532]: time=2026-09-20T12:22:12.102Z level=INFO msg="Uploading 1 narinfos"6051lb # [ 116.797223] niks3-server[1046]: 2026/09/20 12:22:12 INFO Registered completed upload object_key=s7lncxagqa332zn7mhwmk3ry9d83sbvq.ls6052lb # [ 116.808902] niks3-server[1046]: 2026/09/20 12:22:12 INFO Received complete upload request method=POST path=/api/pending_closures/39/complete6053lb # [ 116.819058] niks3-server[1046]: 2026/09/20 12:22:12 INFO Completed upload id=396054worker2 # [ 116.808402] nix-grpc-daemon[1532]: time=2026-09-20T12:22:12.129Z level=INFO msg="Upload complete. (450ms)"6055lb # [ 116.823374] niks3-server[1046]: 2026/09/20 12:22:12 INFO Registered completed upload object_key=s7lncxagqa332zn7mhwmk3ry9d83sbvq.narinfo6056worker2 # [ 116.811974] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:12Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40704 duration_s=0 paths=1 nar_bytes_in=5286057worker2 # [ 116.819365] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:12Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:407046058worker2 # [ 116.828470] nix-daemon[937]: accepted connection from pid 1456, user nix-grpc-daemon (trusted)6059worker2 # [ 116.836861] nix-daemon[937]: reaped child process 1605, status = succeeded6060worker2 # [ 116.842447] nix-daemon[937]: accepted connection from pid 1456, user nix-grpc-daemon (trusted)6061client # building '/nix/store/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv' on lb:50051...6062client # farm-a> worker2: building s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv6063client # farm-a> LOG-farm-a-during-drain6064worker2 # [ 118.464304] nix-daemon[937]: reaped child process 1619, status = succeeded6065worker2 # [ 118.598117] nix-daemon[937]: accepted connection from pid 1634, user nix-grpc-daemon (trusted)6066worker2 # [ 118.609755] nix-daemon[937]: reaped child process 1637, status = succeeded6067lb # [ 118.634089] niks3-server[1046]: 2026/09/20 12:22:14 INFO Received uploads request method=POST path=/api/pending_closures6068worker2 # [ 118.627993] nix-grpc-daemon[1532]: time=2026-09-20T12:22:13.949Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6069worker2 # [ 118.631540] nix-grpc-daemon[1532]: time=2026-09-20T12:22:13.951Z level=INFO msg="Uploading zhl4afvdy7vs34m6ngc3z1xk04qgpcfs-farm-a (136B)"6070lb # [ 118.653132] niks3-server[1046]: 2026/09/20 12:22:14 INFO Registered completed upload object_key=log/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv6071lb # [ 118.660351] niks3-server[1046]: 2026/09/20 12:22:14 INFO Registered completed upload object_key=nar/0r0px9vw070lid3q4jhnmka11sk2y8kj26ysyvjy9dplavd7mdz4.nar.zst6072lb # [ 118.662718] niks3-server[1046]: 2026/09/20 12:22:14 INFO Received sign narinfos request method=POST path=/api/pending_closures/40/sign6073lb # [ 118.664785] niks3-server[1046]: 2026/09/20 12:22:14 INFO Signed narinfos id=40 count=16074worker2 # [ 118.654233] nix-grpc-daemon[1532]: time=2026-09-20T12:22:13.975Z level=INFO msg="Uploading 1 narinfos"6075lb # [ 118.667865] niks3-server[1046]: 2026/09/20 12:22:14 INFO Registered completed upload object_key=zhl4afvdy7vs34m6ngc3z1xk04qgpcfs.ls6076lb # [ 118.673518] niks3-server[1046]: 2026/09/20 12:22:14 INFO Received complete upload request method=POST path=/api/pending_closures/40/complete6077lb # [ 118.676917] niks3-server[1046]: 2026/09/20 12:22:14 INFO Completed upload id=406078worker2 # [ 118.665796] nix-grpc-daemon[1532]: time=2026-09-20T12:22:13.987Z level=INFO msg="Upload complete. (215ms)"6079lb # [ 118.678403] niks3-server[1046]: 2026/09/20 12:22:14 INFO Registered completed upload object_key=zhl4afvdy7vs34m6ngc3z1xk04qgpcfs.narinfo6080worker2 # [ 118.668233] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:13Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40704 duration_s=1 drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv assign_id=4 outputs=16081worker2 # [ 118.675132] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:13Z level=debug event=want client=4 drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv cached=06082worker2 # [ 118.677185] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:13Z level=debug event=assigned drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv worker=192.168.1.4:50051 clients=16083worker2 # [ 118.681342] nix-daemon[937]: reaped child process 1617, status = succeeded6084worker2 # [ 118.718506] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:14Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:407046085worker2 # [ 118.725335] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:14Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:407046086worker2 # [ 118.733787] nix-grpc-daemon[1532]: time=2026-09-20T12:22:14.055Z level=INFO msg="All 1 paths already cached"6087client # copying 1 paths to 'lb:50051'...6088worker2 # [ 118.740556] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:14Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:407046089client # copying path '/nix/store/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv' to 'lb:50051'...6090worker2 # [ 118.757336] nix-daemon[937]: accepted connection from pid 1456, user nix-grpc-daemon (trusted)6091worker2 # [ 118.922180] nix-daemon[937]: accepted connection from pid 1643, user nix-grpc-daemon (trusted)6092worker2 # [ 118.935035] nix-daemon[937]: reaped child process 1646, status = succeeded6093worker2 # [ 119.086224] nix-daemon[937]: accepted connection from pid 1648, user nix-grpc-daemon (trusted)6094worker2 # [ 119.098194] nix-daemon[937]: reaped child process 1651, status = succeeded6095lb # [ 119.124757] niks3-server[1046]: 2026/09/20 12:22:15 INFO Received uploads request method=POST path=/api/pending_closures6096worker2 # [ 119.123434] nix-grpc-daemon[1532]: time=2026-09-20T12:22:14.445Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6097worker2 # [ 119.124911] nix-grpc-daemon[1532]: time=2026-09-20T12:22:14.446Z level=INFO msg="Uploading pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv (640B)"6098lb # [ 119.157719] niks3-server[1046]: 2026/09/20 12:22:15 INFO Registered completed upload object_key=nar/111ql9hf3mg62sp5fwspqwm8cl88198mlcc1ij8flm7l424sz63r.nar.zst6099lb # [ 119.166319] niks3-server[1046]: 2026/09/20 12:22:15 INFO Received sign narinfos request method=POST path=/api/pending_closures/41/sign6100worker2 # [ 119.161967] nix-grpc-daemon[1532]: time=2026-09-20T12:22:14.483Z level=INFO msg="Uploading 1 narinfos"6101lb # [ 119.173090] niks3-server[1046]: 2026/09/20 12:22:15 INFO Signed narinfos id=41 count=16102lb # [ 119.179910] niks3-server[1046]: 2026/09/20 12:22:15 INFO Registered completed upload object_key=pmcfhwqsl1hiyc8gjd850vixdadvskwn.ls6103lb # [ 119.189102] niks3-server[1046]: 2026/09/20 12:22:15 INFO Received complete upload request method=POST path=/api/pending_closures/41/complete6104lb # [ 119.193855] niks3-server[1046]: 2026/09/20 12:22:15 INFO Completed upload id=416105worker2 # [ 119.185593] nix-grpc-daemon[1532]: time=2026-09-20T12:22:14.507Z level=INFO msg="Upload complete. (417ms)"6106lb # [ 119.198506] niks3-server[1046]: 2026/09/20 12:22:15 INFO Registered completed upload object_key=pmcfhwqsl1hiyc8gjd850vixdadvskwn.narinfo6107worker2 # [ 119.189197] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:14Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40704 duration_s=0 paths=1 nar_bytes_in=6406108worker2 # [ 119.194189] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:14Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:407046109worker2 # [ 119.204106] nix-daemon[937]: accepted connection from pid 1456, user nix-grpc-daemon (trusted)6110worker2 # [ 119.208561] nix-daemon[937]: reaped child process 1641, status = succeeded6111worker2 # [ 119.213422] nix-daemon[937]: accepted connection from pid 1456, user nix-grpc-daemon (trusted)6112client # building '/nix/store/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv' on lb:50051...6113client # farm-b> worker2: building pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv6114client # farm-b> LOG-farm-b-during-drain6115worker2 # [ 120.439818] nix-daemon[937]: reaped child process 1655, status = succeeded6116worker2 # [ 120.595592] nix-daemon[937]: accepted connection from pid 1670, user nix-grpc-daemon (trusted)6117worker2 # [ 120.607776] nix-daemon[937]: reaped child process 1673, status = succeeded6118lb # [ 120.635092] niks3-server[1046]: 2026/09/20 12:22:16 INFO Received uploads request method=POST path=/api/pending_closures6119worker2 # [ 120.631258] nix-grpc-daemon[1532]: time=2026-09-20T12:22:15.952Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6120worker2 # [ 120.632625] nix-grpc-daemon[1532]: time=2026-09-20T12:22:15.952Z level=INFO msg="Uploading l458nfm9f1f1zyghsy09ijx02qwk7v65-farm-b (184B)"6121lb # [ 120.669780] niks3-server[1046]: 2026/09/20 12:22:16 INFO Registered completed upload object_key=nar/0m50ra4cd4i97vcbcrdsw7xn8f0b3zc5pdyr2ab8kmp3hscd8j80.nar.zst6122lb # [ 120.677397] niks3-server[1046]: 2026/09/20 12:22:16 INFO Registered completed upload object_key=log/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv6123lb # [ 120.688711] niks3-server[1046]: 2026/09/20 12:22:16 INFO Received sign narinfos request method=POST path=/api/pending_closures/42/sign6124lb # [ 120.693978] niks3-server[1046]: 2026/09/20 12:22:16 INFO Signed narinfos id=42 count=16125worker2 # [ 120.683072] nix-grpc-daemon[1532]: time=2026-09-20T12:22:16.004Z level=INFO msg="Uploading 1 narinfos"6126lb # [ 120.699224] niks3-server[1046]: 2026/09/20 12:22:16 INFO Registered completed upload object_key=l458nfm9f1f1zyghsy09ijx02qwk7v65.ls6127lb # [ 120.707572] niks3-server[1046]: 2026/09/20 12:22:16 INFO Received complete upload request method=POST path=/api/pending_closures/42/complete6128lb # [ 120.711497] niks3-server[1046]: 2026/09/20 12:22:16 INFO Completed upload id=426129worker2 # [ 120.700677] nix-grpc-daemon[1532]: time=2026-09-20T12:22:16.022Z level=INFO msg="Upload complete. (275ms)"6130lb # [ 120.713280] niks3-server[1046]: 2026/09/20 12:22:16 INFO Registered completed upload object_key=l458nfm9f1f1zyghsy09ijx02qwk7v65.narinfo6131worker2 # [ 120.707652] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:16Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40704 duration_s=1 drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv assign_id=5 outputs=16132worker2 # [ 120.719485] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:16Z level=debug event=want client=4 drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv cached=06133worker2 # [ 120.722711] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:16Z level=debug event=assigned drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv worker=192.168.1.4:50051 clients=16134worker2 # [ 120.731206] nix-daemon[937]: reaped child process 1653, status = succeeded6135worker2 # [ 120.763658] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:407046136worker2 # [ 120.777063] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:16Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:407046137worker2 # [ 120.790074] nix-grpc-daemon[1532]: time=2026-09-20T12:22:16.111Z level=INFO msg="All 2 paths already cached"6138client # copying 1 paths to 'lb:50051'...6139worker2 # [ 120.798706] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:16Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:407046140client # copying path '/nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv' to 'lb:50051'...6141worker2 # [ 120.814168] nix-daemon[937]: accepted connection from pid 1456, user nix-grpc-daemon (trusted)6142worker2 # [ 120.977695] nix-daemon[937]: accepted connection from pid 1679, user nix-grpc-daemon (trusted)6143worker2 # [ 120.990723] nix-daemon[937]: reaped child process 1682, status = succeeded6144worker2 # [ 121.167160] nix-daemon[937]: accepted connection from pid 1684, user nix-grpc-daemon (trusted)6145worker2 # [ 121.179235] nix-daemon[937]: reaped child process 1687, status = succeeded6146lb # [ 121.203819] niks3-server[1046]: 2026/09/20 12:22:17 INFO Received uploads request method=POST path=/api/pending_closures6147worker2 # [ 121.205637] nix-grpc-daemon[1532]: time=2026-09-20T12:22:16.527Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6148worker2 # [ 121.207101] nix-grpc-daemon[1532]: time=2026-09-20T12:22:16.527Z level=INFO msg="Uploading w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv (768B)"6149lb # [ 121.236616] niks3-server[1046]: 2026/09/20 12:22:17 INFO Registered completed upload object_key=nar/1crrgw2l7pdmf28ls634k8y85m0pc1v13pxxp4rd6yg5mk3kr5fs.nar.zst6150lb # [ 121.250087] niks3-server[1046]: 2026/09/20 12:22:17 INFO Received sign narinfos request method=POST path=/api/pending_closures/43/sign6151lb # [ 121.257111] niks3-server[1046]: 2026/09/20 12:22:17 INFO Signed narinfos id=43 count=16152worker2 # [ 121.249713] nix-grpc-daemon[1532]: time=2026-09-20T12:22:16.571Z level=INFO msg="Uploading 1 narinfos"6153lb # [ 121.261107] niks3-server[1046]: 2026/09/20 12:22:17 INFO Registered completed upload object_key=w4kjbckcz25f2jivly0r4lbq18qf474f.ls6154lb # [ 121.274337] niks3-server[1046]: 2026/09/20 12:22:17 INFO Received complete upload request method=POST path=/api/pending_closures/43/complete6155lb # [ 121.279550] niks3-server[1046]: 2026/09/20 12:22:17 INFO Completed upload id=436156worker2 # [ 121.270414] nix-grpc-daemon[1532]: time=2026-09-20T12:22:16.591Z level=INFO msg="Upload complete. (443ms)"6157lb # [ 121.282708] niks3-server[1046]: 2026/09/20 12:22:17 INFO Registered completed upload object_key=w4kjbckcz25f2jivly0r4lbq18qf474f.narinfo6158worker2 # [ 121.275130] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:16Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40704 duration_s=0 paths=1 nar_bytes_in=7686159worker2 # [ 121.280826] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:407046160worker2 # [ 121.291814] nix-daemon[937]: accepted connection from pid 1456, user nix-grpc-daemon (trusted)6161worker2 # [ 121.297218] nix-daemon[937]: reaped child process 1677, status = succeeded6162worker2 # [ 121.303189] nix-daemon[937]: accepted connection from pid 1456, user nix-grpc-daemon (trusted)6163client # building '/nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv' on lb:50051...6164client # farm-top> worker2: building w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6165lb # [ 121.480242] envoy[1008]: conn peer=[2001:db8:1::1]:36546 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2937 tx=757 ms=33246166client # farm-top> LOG-farm-top-during-drain6167worker2 # [ 122.330881] nix-daemon[937]: reaped child process 1691, status = succeeded6168worker2 # [ 122.515089] nix-daemon[937]: accepted connection from pid 1706, user nix-grpc-daemon (trusted)6169worker2 # [ 122.525927] nix-daemon[937]: reaped child process 1709, status = succeeded6170lb # [ 122.555158] niks3-server[1046]: 2026/09/20 12:22:18 INFO Received uploads request method=POST path=/api/pending_closures6171worker2 # [ 122.549097] nix-grpc-daemon[1532]: time=2026-09-20T12:22:17.870Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6172worker2 # [ 122.551242] nix-grpc-daemon[1532]: time=2026-09-20T12:22:17.872Z level=INFO msg="Uploading 08ap9ki0d5gixmyr2sac4hj06j9jsqhg-farm-top (240B)"6173lb # [ 122.573460] niks3-server[1046]: 2026/09/20 12:22:18 INFO Registered completed upload object_key=nar/01f0abadm30zaav33qck3x7ypr9nn6rqh2vbrvih4gyb8c26x9in.nar.zst6174lb # [ 122.580226] niks3-server[1046]: 2026/09/20 12:22:18 INFO Received sign narinfos request method=POST path=/api/pending_closures/44/sign6175lb # [ 122.583712] niks3-server[1046]: 2026/09/20 12:22:18 INFO Signed narinfos id=44 count=16176worker2 # [ 122.575093] nix-grpc-daemon[1532]: time=2026-09-20T12:22:17.896Z level=INFO msg="Uploading 1 narinfos"6177lb # [ 122.586885] niks3-server[1046]: 2026/09/20 12:22:18 INFO Registered completed upload object_key=08ap9ki0d5gixmyr2sac4hj06j9jsqhg.ls6178lb # [ 122.589680] niks3-server[1046]: 2026/09/20 12:22:18 INFO Registered completed upload object_key=log/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6179lb # [ 122.595393] niks3-server[1046]: 2026/09/20 12:22:18 INFO Received complete upload request method=POST path=/api/pending_closures/44/complete6180lb # [ 122.600211] niks3-server[1046]: 2026/09/20 12:22:18 INFO Completed upload id=446181worker2 # [ 122.589223] nix-grpc-daemon[1532]: time=2026-09-20T12:22:17.910Z level=INFO msg="Upload complete. (276ms)"6182lb # [ 122.601850] niks3-server[1046]: 2026/09/20 12:22:18 INFO Registered completed upload object_key=08ap9ki0d5gixmyr2sac4hj06j9jsqhg.narinfo6183worker2 # [ 122.591861] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:17Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40704 duration_s=1 drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv assign_id=6 outputs=16184worker2 # [ 122.599160] nix-daemon[937]: reaped child process 1689, status = succeeded6185client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag during-drain >&2, in 6.61 seconds)6186client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag during-drain outPath6187client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag during-drain outPath, in 0.22 seconds)6188worker1: must succeed: pkill -f 'read -t [9]0 x'6189worker1: (finished: must succeed: pkill -f 'read -t [9]0 x', in 0.05 seconds)6190client: waiting for success: ! systemctl is-active sw6191worker1 # [ 123.119656] nix-daemon[845]: reaped child process 1625, status = succeeded6192worker1 # [ 123.277087] nix-daemon[845]: accepted connection from pid 1666, user nix-grpc-daemon (trusted)6193worker1 # [ 123.288384] nix-daemon[845]: reaped child process 1669, status = succeeded6194lb # [ 123.301418] niks3-server[1046]: 2026/09/20 12:22:19 INFO Received uploads request method=POST path=/api/pending_closures6195worker1 # [ 123.322633] nix-grpc-daemon[1569]: time=2026-09-20T12:22:18.562Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6196worker1 # [ 123.324158] nix-grpc-daemon[1569]: time=2026-09-20T12:22:18.562Z level=INFO msg="Uploading mlk55snqrikr02pwsxjamphvl61rvl0r-slow-sw (120B)"6197lb # [ 123.346722] niks3-server[1046]: 2026/09/20 12:22:19 INFO Registered completed upload object_key=nar/0vi6cx31wc05wns04pr5vr2bak9iib1idndvy6gik6c2rz9q8c19.nar.zst6198lb # [ 123.357217] niks3-server[1046]: 2026/09/20 12:22:19 INFO Received sign narinfos request method=POST path=/api/pending_closures/45/sign6199lb # [ 123.362130] niks3-server[1046]: 2026/09/20 12:22:19 INFO Signed narinfos id=45 count=16200worker1 # [ 123.363998] nix-grpc-daemon[1569]: time=2026-09-20T12:22:18.603Z level=INFO msg="Uploading 1 narinfos"6201lb # [ 123.371309] niks3-server[1046]: 2026/09/20 12:22:19 INFO Received complete upload request method=POST path=/api/pending_closures/45/complete6202lb # [ 123.396418] niks3-server[1046]: 2026/09/20 12:22:19 INFO Registered completed upload object_key=log/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv6203lb # [ 123.406364] niks3-server[1046]: 2026/09/20 12:22:19 INFO Registered completed upload object_key=mlk55snqrikr02pwsxjamphvl61rvl0r.ls6204lb # [ 123.418946] niks3-server[1046]: 2026/09/20 12:22:19 INFO Completed upload id=456205worker1 # [ 123.419870] nix-grpc-daemon[1569]: time=2026-09-20T12:22:18.660Z level=INFO msg="Upload complete. (318ms)"6206lb # [ 123.426502] niks3-server[1046]: 2026/09/20 12:22:19 INFO Registered completed upload object_key=mlk55snqrikr02pwsxjamphvl61rvl0r.narinfo6207worker1 # [ 123.426602] nix-grpc-daemon[1468]: ts=2026-09-20T12:22:18Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40952 duration_s=9 drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv assign_id=3 outputs=16208worker1 # [ 123.440536] nix-daemon[845]: reaped child process 1623, status = succeeded6209client # [ 123.458455] systemd[1]: sw.service: Deactivated successfully.6210client # [ 123.462943] systemd[1]: sw.service: Consumed 137ms CPU time over 10.055s wall clock time, 39.1M memory peak, 8.6K incoming IP traffic, 9K outgoing IP traffic.6211worker1 # [ 123.608762] nix-grpc-daemon[1468]: ts=2026-09-20T12:22:18Z level=info event=signal_exit6212client: (finished: waiting for success: ! systemctl is-active sw, in 1.07 seconds)6213client: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }6214client: (finished: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }, in 0.03 seconds)6215worker1: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service6216worker2 # [ 124.606817] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:19Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:406766217worker2 # [ 124.610494] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:19Z level=info event=worker_gone addr=192.168.1.3:500516218worker1 # [ 124.620505] nix-grpc-daemon[1468]: ts=2026-09-20T12:22:19Z level=info event=scheduler_restarting6219worker1 # [ 124.630356] nix-grpc-daemon[1468]: ts=2026-09-20T12:22:19Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"6220worker1 # [ 124.646993] nix-daemon[845]: reaped child process 1537, status = succeeded6221worker1 # [ 125.453412] nix-grpc-daemon[1468]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6222worker1 # [ 125.456963] nix-grpc-daemon[1468]: E0000 00:00:1789906940.693608 1511 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 15, started true, shutdown true6223worker1 # [ 125.461839] nix-grpc-daemon[1468]: E0000 00:00:1789906940.693719 1511 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true6224worker1 # [ 126.642389] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.6225worker1 # [ 126.647194] systemd[1]: nix-grpc-daemon.service: Consumed 619ms CPU time over 28.205s wall clock time, 29.9M memory peak, 21.1K incoming IP traffic, 27.6K outgoing IP traffic.6226worker1 # [ 126.663233] systemd[1]: Starting Nix worker-protocol over gRPC...6227worker1 # [ 126.811302] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:22Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6228worker1 # [ 126.813244] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:22Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6229worker1 # [ 126.816600] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:22Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=16230worker1 # [ 126.823525] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:22Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6231worker1 # [ 126.827924] systemd[1]: Started Nix worker-protocol over gRPC.6232worker1: (finished: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service, in 2.82 seconds)6233lb: waiting until exactly no endpoints of x86_64-linux are unhealthy6234lb: must succeed: curl -sf localhost:9901/clusters6235worker1 # [ 126.839582] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:22Z level=info event=scheduler_connected addr=lb:500516236worker2 # [ 126.830566] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:22Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:406766237worker2 # [ 126.834752] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:22Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06238lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)6239lb: must succeed: curl -sf localhost:9901/clusters6240lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)6241lb: (finished: waiting until exactly no endpoints of x86_64-linux are unhealthy, in 1.12 seconds)6242lb: must succeed: curl -sf localhost:9901/clusters6243lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)6244worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6245worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6246(finished: subtest: a deploy mid-build drains: the build finishes, the new generation takes over, in 14.68 seconds)6247subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does6248client: must succeed: nix build -L --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&16249worker2 # [ 128.265733] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:23Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:407046250worker1 # [ 128.289678] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:23Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:466406251worker1 # [ 128.301106] nix-daemon[845]: accepted connection from pid 1683, user nix-grpc-daemon (trusted)6252worker1 # [ 128.316933] nix-grpc-daemon[1683]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 98 ms (attempt 1/5)6253worker1 # [ 128.420853] nix-grpc-daemon[1683]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 37 ms (attempt 2/5)6254worker1 # [ 128.463679] nix-grpc-daemon[1683]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 141 ms (attempt 3/5)6255worker1 # [ 128.610698] nix-grpc-daemon[1683]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 518 ms (attempt 4/5)6256worker1 # [ 129.135604] nix-grpc-daemon[1683]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6257worker2 # [ 129.169898] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:24Z level=debug event=want client=5 drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv cached=06258worker2 # [ 129.174314] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:24Z level=debug event=assigned drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv worker=192.168.1.3:50051 clients=16259worker2 # [ 129.178662] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:24Z level=debug event=want client=5 drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv cached=06260worker2 # [ 129.182554] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:24Z level=debug event=want client=5 drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv cached=06261worker1 # [ 129.227153] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:466406262worker1 # [ 129.240675] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:24Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:466406263worker1 # [ 129.252452] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:24Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:466406264worker1 # [ 129.260965] nix-daemon[845]: accepted connection from pid 1683, user nix-grpc-daemon (trusted)6265worker1 # [ 129.282942] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:24Z level=info event=niks3_push_started pid=17396266worker1 # [ 129.459762] nix-grpc-daemon[1739]: time=2026-09-20T12:22:24.699Z level=INFO msg="Configuring client TLS" cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/worker1.pem key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/worker1.key ca=""6267worker1 # [ 129.603090] nix-daemon[845]: accepted connection from pid 1752, user nix-grpc-daemon (trusted)6268worker1 # [ 129.613713] nix-daemon[845]: reaped child process 1755, status = succeeded6269worker1 # [ 129.765531] nix-daemon[845]: accepted connection from pid 1757, user nix-grpc-daemon (trusted)6270worker1 # [ 129.777612] nix-daemon[845]: reaped child process 1760, status = succeeded6271lb # [ 129.791475] niks3-server[1046]: 2026/09/20 12:22:25 INFO Received uploads request method=POST path=/api/pending_closures6272worker1 # [ 129.801105] nix-grpc-daemon[1739]: time=2026-09-20T12:22:25.041Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6273worker1 # [ 129.802660] nix-grpc-daemon[1739]: time=2026-09-20T12:22:25.043Z level=INFO msg="Uploading cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv (504B)"6274lb # [ 129.827409] niks3-server[1046]: 2026/09/20 12:22:25 INFO Registered completed upload object_key=nar/1ifhv0ll3mnj6a2hrp33fzwnm2yl2zjxwqfrqjriibhs5cbnyiwi.nar.zst6275lb # [ 129.834461] niks3-server[1046]: 2026/09/20 12:22:26 INFO Received sign narinfos request method=POST path=/api/pending_closures/46/sign6276lb # [ 129.839636] niks3-server[1046]: 2026/09/20 12:22:26 INFO Signed narinfos id=46 count=16277worker1 # [ 129.843596] nix-grpc-daemon[1739]: time=2026-09-20T12:22:25.084Z level=INFO msg="Uploading 1 narinfos"6278lb # [ 129.849996] niks3-server[1046]: 2026/09/20 12:22:26 INFO Registered completed upload object_key=cv151lgjr4lv54sqzb49xk9w1ak9xsm9.ls6279lb # [ 129.861057] niks3-server[1046]: 2026/09/20 12:22:26 INFO Received complete upload request method=POST path=/api/pending_closures/46/complete6280lb # [ 129.866085] niks3-server[1046]: 2026/09/20 12:22:26 INFO Completed upload id=466281worker1 # [ 129.869442] nix-grpc-daemon[1739]: time=2026-09-20T12:22:25.109Z level=INFO msg="Upload complete. (405ms)"6282lb # [ 129.869635] niks3-server[1046]: 2026/09/20 12:22:26 INFO Registered completed upload object_key=cv151lgjr4lv54sqzb49xk9w1ak9xsm9.narinfo6283worker1 # [ 129.871282] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:25Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46640 duration_s=0 paths=1 nar_bytes_in=5046284worker1 # [ 129.880101] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:25Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:466406285worker1 # [ 129.888534] nix-daemon[845]: accepted connection from pid 1683, user nix-grpc-daemon (trusted)6286worker1 # [ 129.890931] nix-daemon[845]: reaped child process 1737, status = succeeded6287worker1 # [ 129.896204] nix-daemon[845]: accepted connection from pid 1683, user nix-grpc-daemon (trusted)6288worker1 # [ 131.123964] nix-daemon[845]: reaped child process 1764, status = succeeded6289worker1 # [ 131.267914] nix-daemon[845]: accepted connection from pid 1779, user nix-grpc-daemon (trusted)6290worker1 # [ 131.280201] nix-daemon[845]: reaped child process 1782, status = succeeded6291lb # [ 131.291717] niks3-server[1046]: 2026/09/20 12:22:27 INFO Received uploads request method=POST path=/api/pending_closures6292worker1 # [ 131.298204] nix-grpc-daemon[1739]: time=2026-09-20T12:22:26.538Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6293worker1 # [ 131.299706] nix-grpc-daemon[1739]: time=2026-09-20T12:22:26.540Z level=INFO msg="Uploading wfziwgf0gxmjhy17lyfmn47628s166p7-farm-a (128B)"6294lb # [ 131.312565] niks3-server[1046]: 2026/09/20 12:22:27 INFO Registered completed upload object_key=log/cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv6295lb # [ 131.317379] niks3-server[1046]: 2026/09/20 12:22:27 INFO Registered completed upload object_key=nar/05i7ka1qa7vmhiys96ln3pp3nj3rfdlslnbydrxdqj40cmvmqcwf.nar.zst6296lb # [ 131.321763] niks3-server[1046]: 2026/09/20 12:22:27 INFO Received sign narinfos request method=POST path=/api/pending_closures/47/sign6297lb # [ 131.324266] niks3-server[1046]: 2026/09/20 12:22:27 INFO Signed narinfos id=47 count=16298worker1 # [ 131.325616] nix-grpc-daemon[1739]: time=2026-09-20T12:22:26.565Z level=INFO msg="Uploading 1 narinfos"6299lb # [ 131.327115] niks3-server[1046]: 2026/09/20 12:22:27 INFO Registered completed upload object_key=wfziwgf0gxmjhy17lyfmn47628s166p7.ls6300lb # [ 131.332130] niks3-server[1046]: 2026/09/20 12:22:27 INFO Received complete upload request method=POST path=/api/pending_closures/47/complete6301lb # [ 131.335692] niks3-server[1046]: 2026/09/20 12:22:27 INFO Completed upload id=476302worker1 # [ 131.336799] nix-grpc-daemon[1739]: time=2026-09-20T12:22:26.577Z level=INFO msg="Upload complete. (233ms)"6303lb # [ 131.338083] niks3-server[1046]: 2026/09/20 12:22:27 INFO Registered completed upload object_key=wfziwgf0gxmjhy17lyfmn47628s166p7.narinfo6304worker1 # [ 131.339351] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:26Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46640 duration_s=1 drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv assign_id=7 outputs=16305worker2 # [ 131.330249] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:26Z level=debug event=assigned drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv worker=192.168.1.3:50051 clients=16306worker1 # [ 131.346324] nix-daemon[845]: reaped child process 1762, status = succeeded6307worker2 # [ 131.335476] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:26Z level=debug event=want client=5 drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv cached=06308worker1 # [ 131.347853] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:466406309worker1 # [ 131.354349] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:26Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:466406310worker1 # [ 131.360206] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:26Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:466406311worker1 # [ 131.364257] nix-daemon[845]: accepted connection from pid 1683, user nix-grpc-daemon (trusted)6312lb # [ 131.480114] envoy[1008]: conn peer=[2001:db8:1::1]:44192 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9641 tx=3870 ms=63786313lb # [ 131.481815] envoy[1008]: conn peer=[2001:db8:1::1]:44178 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2949 tx=1349 ms=98036314lb # [ 131.483300] envoy[1008]: conn peer=[2001:db8:1::3]:33534 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=1114 tx=448 ms=260146315worker1 # [ 131.509989] nix-daemon[845]: accepted connection from pid 1786, user nix-grpc-daemon (trusted)6316worker1 # [ 131.521991] nix-daemon[845]: reaped child process 1789, status = succeeded6317worker1 # [ 131.675872] nix-daemon[845]: accepted connection from pid 1791, user nix-grpc-daemon (trusted)6318worker1 # [ 131.688080] nix-daemon[845]: reaped child process 1794, status = succeeded6319lb # [ 131.700055] niks3-server[1046]: 2026/09/20 12:22:27 INFO Received uploads request method=POST path=/api/pending_closures6320worker1 # [ 131.705701] nix-grpc-daemon[1739]: time=2026-09-20T12:22:26.946Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6321worker1 # [ 131.707390] nix-grpc-daemon[1739]: time=2026-09-20T12:22:26.947Z level=INFO msg="Uploading fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv (504B)"6322lb # [ 131.716476] niks3-server[1046]: 2026/09/20 12:22:27 INFO Registered completed upload object_key=nar/170l67nd7hvasq2l36jibfpivx7r9ssdy0wlzifggpplhcgf2rk0.nar.zst6323lb # [ 131.720621] niks3-server[1046]: 2026/09/20 12:22:27 INFO Received sign narinfos request method=POST path=/api/pending_closures/48/sign6324lb # [ 131.722381] niks3-server[1046]: 2026/09/20 12:22:27 INFO Signed narinfos id=48 count=16325worker1 # [ 131.724563] nix-grpc-daemon[1739]: time=2026-09-20T12:22:26.965Z level=INFO msg="Uploading 1 narinfos"6326lb # [ 131.726287] niks3-server[1046]: 2026/09/20 12:22:27 INFO Registered completed upload object_key=fggw0ziag6jlrc830j6x1b6qw68kybhf.ls6327lb # [ 131.730750] niks3-server[1046]: 2026/09/20 12:22:27 INFO Received complete upload request method=POST path=/api/pending_closures/48/complete6328lb # [ 131.734339] niks3-server[1046]: 2026/09/20 12:22:27 INFO Completed upload id=486329worker1 # [ 131.735226] nix-grpc-daemon[1739]: time=2026-09-20T12:22:26.975Z level=INFO msg="Upload complete. (365ms)"6330lb # [ 131.736212] niks3-server[1046]: 2026/09/20 12:22:27 INFO Registered completed upload object_key=fggw0ziag6jlrc830j6x1b6qw68kybhf.narinfo6331worker1 # [ 131.738071] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:26Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46640 duration_s=0 paths=1 nar_bytes_in=5046332worker1 # [ 131.741352] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:466406333worker1 # [ 131.746913] nix-daemon[845]: accepted connection from pid 1683, user nix-grpc-daemon (trusted)6334worker1 # [ 131.749224] nix-daemon[845]: reaped child process 1784, status = succeeded6335worker1 # [ 131.754403] nix-daemon[845]: accepted connection from pid 1683, user nix-grpc-daemon (trusted)6336worker1 # [ 132.783246] nix-daemon[845]: reaped child process 1798, status = succeeded6337worker1 # [ 132.932193] nix-daemon[845]: accepted connection from pid 1813, user nix-grpc-daemon (trusted)6338worker1 # [ 132.943853] nix-daemon[845]: reaped child process 1816, status = succeeded6339lb # [ 132.956959] niks3-server[1046]: 2026/09/20 12:22:29 INFO Received uploads request method=POST path=/api/pending_closures6340worker1 # [ 132.965958] nix-grpc-daemon[1739]: time=2026-09-20T12:22:28.206Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6341worker1 # [ 132.967679] nix-grpc-daemon[1739]: time=2026-09-20T12:22:28.207Z level=INFO msg="Uploading 63zyaryk3r1ikmmzi1w30vadf20kdw3y-farm-a (128B)"6342lb # [ 132.994157] niks3-server[1046]: 2026/09/20 12:22:29 INFO Registered completed upload object_key=log/fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv6343lb # [ 133.004208] niks3-server[1046]: 2026/09/20 12:22:29 INFO Registered completed upload object_key=nar/09na6djn66dpys8h08gqw39p0vw7vxqsdzbiw55rknyna8s239bm.nar.zst6344lb # [ 133.013810] niks3-server[1046]: 2026/09/20 12:22:29 INFO Received sign narinfos request method=POST path=/api/pending_closures/49/sign6345worker1 # [ 133.020061] nix-grpc-daemon[1739]: time=2026-09-20T12:22:28.260Z level=INFO msg="Uploading 1 narinfos"6346lb # [ 133.019126] niks3-server[1046]: 2026/09/20 12:22:29 INFO Signed narinfos id=49 count=16347lb # [ 133.023829] niks3-server[1046]: 2026/09/20 12:22:29 INFO Registered completed upload object_key=63zyaryk3r1ikmmzi1w30vadf20kdw3y.ls6348lb # [ 133.031985] niks3-server[1046]: 2026/09/20 12:22:29 INFO Received complete upload request method=POST path=/api/pending_closures/49/complete6349lb # [ 133.037813] niks3-server[1046]: 2026/09/20 12:22:29 INFO Completed upload id=496350worker1 # [ 133.039134] nix-grpc-daemon[1739]: time=2026-09-20T12:22:28.279Z level=INFO msg="Upload complete. (277ms)"6351lb # [ 133.040574] niks3-server[1046]: 2026/09/20 12:22:29 INFO Registered completed upload object_key=63zyaryk3r1ikmmzi1w30vadf20kdw3y.narinfo6352worker1 # [ 133.047568] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:28Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46640 duration_s=1 drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv assign_id=8 outputs=16353worker2 # [ 133.037170] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:28Z level=debug event=assigned drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv worker=192.168.1.3:50051 clients=16354worker1 # [ 133.053504] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:28Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:466406355worker2 # [ 133.042871] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:28Z level=debug event=want client=5 drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv cached=06356worker1 # [ 133.063369] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:28Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:466406357worker1 # [ 133.071515] nix-daemon[845]: reaped child process 1796, status = succeeded6358worker1 # [ 133.074515] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:28Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:466406359worker1 # [ 133.080060] nix-daemon[845]: accepted connection from pid 1683, user nix-grpc-daemon (trusted)6360worker1 # [ 133.225690] nix-daemon[845]: accepted connection from pid 1820, user nix-grpc-daemon (trusted)6361worker1 # [ 133.237992] nix-daemon[845]: reaped child process 1823, status = succeeded6362worker1 # [ 133.391465] nix-daemon[845]: accepted connection from pid 1825, user nix-grpc-daemon (trusted)6363worker1 # [ 133.402653] nix-daemon[845]: reaped child process 1828, status = succeeded6364lb # [ 133.415200] niks3-server[1046]: 2026/09/20 12:22:29 INFO Received uploads request method=POST path=/api/pending_closures6365worker1 # [ 133.423807] nix-grpc-daemon[1739]: time=2026-09-20T12:22:28.664Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6366worker1 # [ 133.425453] nix-grpc-daemon[1739]: time=2026-09-20T12:22:28.665Z level=INFO msg="Uploading j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv (504B)"6367lb # [ 133.444324] niks3-server[1046]: 2026/09/20 12:22:29 INFO Registered completed upload object_key=nar/0qr7hdc916yrqpvqx2bwc5l0rwy3zm6rsdlcf8rk9iid53b80xsn.nar.zst6368lb # [ 133.453159] niks3-server[1046]: 2026/09/20 12:22:29 INFO Received sign narinfos request method=POST path=/api/pending_closures/50/sign6369lb # [ 133.459669] niks3-server[1046]: 2026/09/20 12:22:29 INFO Signed narinfos id=50 count=16370worker1 # [ 133.463948] nix-grpc-daemon[1739]: time=2026-09-20T12:22:28.704Z level=INFO msg="Uploading 1 narinfos"6371lb # [ 133.463359] niks3-server[1046]: 2026/09/20 12:22:29 INFO Registered completed upload object_key=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz.ls6372lb # [ 133.474489] niks3-server[1046]: 2026/09/20 12:22:29 INFO Received complete upload request method=POST path=/api/pending_closures/50/complete6373lb # [ 133.480579] niks3-server[1046]: 2026/09/20 12:22:29 INFO Completed upload id=506374worker1 # [ 133.481682] nix-grpc-daemon[1739]: time=2026-09-20T12:22:28.722Z level=INFO msg="Upload complete. (396ms)"6375lb # [ 133.484939] niks3-server[1046]: 2026/09/20 12:22:29 INFO Registered completed upload object_key=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz.narinfo6376worker1 # [ 133.485154] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:28Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46640 duration_s=0 paths=1 nar_bytes_in=5046377worker1 # [ 133.490304] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:28Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:466406378worker1 # [ 133.498411] nix-daemon[845]: accepted connection from pid 1683, user nix-grpc-daemon (trusted)6379worker1 # [ 133.505489] nix-daemon[845]: reaped child process 1818, status = succeeded6380worker1 # [ 133.509073] nix-daemon[845]: accepted connection from pid 1683, user nix-grpc-daemon (trusted)6381worker1 # [ 134.334849] nix-daemon[845]: reaped child process 1832, status = succeeded6382worker1 # [ 134.485599] nix-daemon[845]: accepted connection from pid 1847, user nix-grpc-daemon (trusted)6383worker1 # [ 134.497807] nix-daemon[845]: reaped child process 1850, status = succeeded6384lb # [ 134.512452] niks3-server[1046]: 2026/09/20 12:22:30 INFO Received uploads request method=POST path=/api/pending_closures6385worker1 # [ 134.523185] nix-grpc-daemon[1739]: time=2026-09-20T12:22:29.763Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6386worker1 # [ 134.524866] nix-grpc-daemon[1739]: time=2026-09-20T12:22:29.765Z level=INFO msg="Uploading pnim4ls33hwi0fl1wv96w4yj4afz9w4m-farm-a (128B)"6387lb # [ 134.545811] niks3-server[1046]: 2026/09/20 12:22:30 INFO Registered completed upload object_key=nar/1j2671k9mycq8lxyfx5k6a66d5narfqw2wqwlpwxgwzi93vnvbdn.nar.zst6388lb # [ 134.559203] niks3-server[1046]: 2026/09/20 12:22:30 INFO Received sign narinfos request method=POST path=/api/pending_closures/51/sign6389worker1 # [ 134.569086] nix-grpc-daemon[1739]: time=2026-09-20T12:22:29.809Z level=INFO msg="Uploading 1 narinfos"6390lb # [ 134.568093] niks3-server[1046]: 2026/09/20 12:22:30 INFO Signed narinfos id=51 count=16391lb # [ 134.571484] niks3-server[1046]: 2026/09/20 12:22:30 INFO Registered completed upload object_key=pnim4ls33hwi0fl1wv96w4yj4afz9w4m.ls6392lb # [ 134.575702] niks3-server[1046]: 2026/09/20 12:22:30 INFO Registered completed upload object_key=log/j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv6393lb # [ 134.583214] niks3-server[1046]: 2026/09/20 12:22:30 INFO Received complete upload request method=POST path=/api/pending_closures/51/complete6394lb # [ 134.586664] niks3-server[1046]: 2026/09/20 12:22:30 INFO Completed upload id=516395worker1 # [ 134.587727] nix-grpc-daemon[1739]: time=2026-09-20T12:22:29.828Z level=INFO msg="Upload complete. (272ms)"6396lb # [ 134.588534] niks3-server[1046]: 2026/09/20 12:22:30 INFO Registered completed upload object_key=pnim4ls33hwi0fl1wv96w4yj4afz9w4m.narinfo6397worker2 # [ 134.580405] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:29Z level=debug event=assigned drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv worker=192.168.1.3:50051 clients=16398worker1 # [ 134.590872] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:29Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46640 duration_s=1 drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv assign_id=9 outputs=16399worker2 # [ 134.587824] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:29Z level=debug event=want client=5 drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv cached=06400worker1 # [ 134.598935] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:29Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:466406401worker1 # [ 134.608240] nix-daemon[845]: reaped child process 1830, status = succeeded6402worker1 # [ 134.610347] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:29Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:466406403worker1 # [ 134.616077] nix-grpc-daemon[1739]: time=2026-09-20T12:22:29.856Z level=INFO msg="All 1 paths already cached"6404worker1 # [ 134.619752] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:29Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:466406405worker1 # [ 134.623329] nix-daemon[845]: accepted connection from pid 1683, user nix-grpc-daemon (trusted)6406worker1 # [ 134.767067] nix-daemon[845]: accepted connection from pid 1854, user nix-grpc-daemon (trusted)6407worker1 # [ 134.778813] nix-daemon[845]: reaped child process 1857, status = succeeded6408worker1 # [ 134.930562] nix-daemon[845]: accepted connection from pid 1859, user nix-grpc-daemon (trusted)6409worker1 # [ 134.942602] nix-daemon[845]: reaped child process 1862, status = succeeded6410lb # [ 134.957786] niks3-server[1046]: 2026/09/20 12:22:31 INFO Received uploads request method=POST path=/api/pending_closures6411worker1 # [ 134.976623] nix-grpc-daemon[1739]: time=2026-09-20T12:22:30.216Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6412worker1 # [ 134.978183] nix-grpc-daemon[1739]: time=2026-09-20T12:22:30.216Z level=INFO msg="Uploading 6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv (624B)"6413lb # [ 135.000271] niks3-server[1046]: 2026/09/20 12:22:31 INFO Received sign narinfos request method=POST path=/api/pending_closures/52/sign6414lb # [ 135.005290] niks3-server[1046]: 2026/09/20 12:22:31 INFO Registered completed upload object_key=nar/06n0qn8haxwg1vfa3qwa7h14mzz5fz7nizpv8bn995f4gddci6y6.nar.zst6415worker1 # [ 135.012519] nix-grpc-daemon[1739]: time=2026-09-20T12:22:30.252Z level=INFO msg="Uploading 1 narinfos"6416lb # [ 135.011719] niks3-server[1046]: 2026/09/20 12:22:31 INFO Signed narinfos id=52 count=16417lb # [ 135.015723] niks3-server[1046]: 2026/09/20 12:22:31 INFO Registered completed upload object_key=6cjjd3vbzygfms2ccxvnmabbgq5pv310.ls6418lb # [ 135.031595] niks3-server[1046]: 2026/09/20 12:22:31 INFO Received complete upload request method=POST path=/api/pending_closures/52/complete6419lb # [ 135.037908] niks3-server[1046]: 2026/09/20 12:22:31 INFO Completed upload id=526420worker1 # [ 135.038959] nix-grpc-daemon[1739]: time=2026-09-20T12:22:30.279Z level=INFO msg="Upload complete. (409ms)"6421lb # [ 135.040239] niks3-server[1046]: 2026/09/20 12:22:31 INFO Registered completed upload object_key=6cjjd3vbzygfms2ccxvnmabbgq5pv310.narinfo6422worker1 # [ 135.043286] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:30Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46640 duration_s=0 paths=1 nar_bytes_in=6246423worker1 # [ 135.050516] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:30Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:466406424worker1 # [ 135.060270] nix-daemon[845]: accepted connection from pid 1683, user nix-grpc-daemon (trusted)6425worker1 # [ 135.068335] nix-daemon[845]: reaped child process 1852, status = succeeded6426worker1 # [ 135.075217] nix-daemon[845]: accepted connection from pid 1683, user nix-grpc-daemon (trusted)6427worker1 # [ 136.099946] nix-daemon[845]: reaped child process 1866, status = succeeded6428worker1 # [ 136.255750] nix-daemon[845]: accepted connection from pid 1881, user nix-grpc-daemon (trusted)6429worker1 # [ 136.267512] nix-daemon[845]: reaped child process 1884, status = succeeded6430lb # [ 136.280263] niks3-server[1046]: 2026/09/20 12:22:32 INFO Received uploads request method=POST path=/api/pending_closures6431worker1 # [ 136.287892] nix-grpc-daemon[1739]: time=2026-09-20T12:22:31.528Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6432worker1 # [ 136.289577] nix-grpc-daemon[1739]: time=2026-09-20T12:22:31.529Z level=INFO msg="Uploading 7gfg490kigdqainqjk27vmks1h3wpv5q-farm-b (176B)"6433lb # [ 136.304632] niks3-server[1046]: 2026/09/20 12:22:32 INFO Registered completed upload object_key=log/6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv6434lb # [ 136.308532] niks3-server[1046]: 2026/09/20 12:22:32 INFO Registered completed upload object_key=nar/07q9c11jk9djv06znvadbhgp1j3kwqf4z0z60kwx2sjqjkpssnf0.nar.zst6435lb # [ 136.310956] niks3-server[1046]: 2026/09/20 12:22:32 INFO Received sign narinfos request method=POST path=/api/pending_closures/53/sign6436lb # [ 136.313500] niks3-server[1046]: 2026/09/20 12:22:32 INFO Signed narinfos id=53 count=16437worker1 # [ 136.314741] nix-grpc-daemon[1739]: time=2026-09-20T12:22:31.555Z level=INFO msg="Uploading 1 narinfos"6438lb # [ 136.319295] niks3-server[1046]: 2026/09/20 12:22:32 INFO Received complete upload request method=POST path=/api/pending_closures/53/complete6439lb # [ 136.515090] niks3-server[1046]: 2026/09/20 12:22:32 INFO Registered completed upload object_key=7gfg490kigdqainqjk27vmks1h3wpv5q.ls6440lb # [ 136.527268] niks3-server[1046]: 2026/09/20 12:22:32 INFO Completed upload id=536441worker1 # [ 136.528114] nix-grpc-daemon[1739]: time=2026-09-20T12:22:31.768Z level=INFO msg="Upload complete. (445ms)"6442worker2 # [ 136.521785] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:31Z level=debug event=assigned drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv worker=192.168.1.3:50051 clients=16443lb # [ 136.534240] niks3-server[1046]: 2026/09/20 12:22:32 INFO Registered completed upload object_key=7gfg490kigdqainqjk27vmks1h3wpv5q.narinfo6444worker1 # [ 136.534450] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:31Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46640 duration_s=1 drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv assign_id=10 outputs=16445worker2 # [ 136.530321] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:31Z level=debug event=want client=5 drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv cached=06446worker1 # [ 136.542927] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:31Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:466406447worker1 # [ 136.556622] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:31Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:466406448worker1 # [ 136.563167] nix-daemon[845]: reaped child process 1864, status = succeeded6449worker1 # [ 136.576451] nix-grpc-daemon[1739]: time=2026-09-20T12:22:31.816Z level=INFO msg="All 1 paths already cached"6450worker1 # [ 136.583563] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:31Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:466406451worker1 # [ 136.593155] nix-daemon[845]: accepted connection from pid 1683, user nix-grpc-daemon (trusted)6452worker1 # [ 136.749653] nix-daemon[845]: accepted connection from pid 1888, user nix-grpc-daemon (trusted)6453worker1 # [ 136.761729] nix-daemon[845]: reaped child process 1891, status = succeeded6454worker1 # [ 136.913617] nix-daemon[845]: accepted connection from pid 1893, user nix-grpc-daemon (trusted)6455worker1 # [ 136.925535] nix-daemon[845]: reaped child process 1896, status = succeeded6456lb # [ 136.939885] niks3-server[1046]: 2026/09/20 12:22:33 INFO Received uploads request method=POST path=/api/pending_closures6457worker1 # [ 136.948528] nix-grpc-daemon[1739]: time=2026-09-20T12:22:32.189Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6458worker1 # [ 136.950414] nix-grpc-daemon[1739]: time=2026-09-20T12:22:32.190Z level=INFO msg="Uploading 6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv (624B)"6459lb # [ 136.968223] niks3-server[1046]: 2026/09/20 12:22:33 INFO Registered completed upload object_key=nar/0w4dmpczwycj1p7ngvcwdynpk5hh6r5ivyz44k3lx2gh5vpk4l0i.nar.zst6460lb # [ 136.980805] niks3-server[1046]: 2026/09/20 12:22:33 INFO Received sign narinfos request method=POST path=/api/pending_closures/54/sign6461lb # [ 136.989791] niks3-server[1046]: 2026/09/20 12:22:33 INFO Registered completed upload object_key=6cs4w0lyx4kldp39y5k29p7c0234cffw.ls6462lb # [ 136.994484] niks3-server[1046]: 2026/09/20 12:22:33 INFO Signed narinfos id=54 count=16463worker1 # [ 136.998511] nix-grpc-daemon[1739]: time=2026-09-20T12:22:32.238Z level=INFO msg="Uploading 1 narinfos"6464lb # [ 137.006255] niks3-server[1046]: 2026/09/20 12:22:33 INFO Received complete upload request method=POST path=/api/pending_closures/54/complete6465lb # [ 137.011390] niks3-server[1046]: 2026/09/20 12:22:33 INFO Completed upload id=546466worker1 # [ 137.012519] nix-grpc-daemon[1739]: time=2026-09-20T12:22:32.252Z level=INFO msg="Upload complete. (408ms)"6467lb # [ 137.015885] niks3-server[1046]: 2026/09/20 12:22:33 INFO Registered completed upload object_key=6cs4w0lyx4kldp39y5k29p7c0234cffw.narinfo6468worker1 # [ 137.016096] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:32Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46640 duration_s=0 paths=1 nar_bytes_in=6246469worker1 # [ 137.022084] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:466406470worker1 # [ 137.030069] nix-daemon[845]: accepted connection from pid 1683, user nix-grpc-daemon (trusted)6471worker1 # [ 137.037561] nix-daemon[845]: reaped child process 1886, status = succeeded6472worker1 # [ 137.043607] nix-daemon[845]: accepted connection from pid 1683, user nix-grpc-daemon (trusted)6473worker1 # [ 138.672557] nix-daemon[845]: reaped child process 1900, status = succeeded6474worker1 # [ 138.813346] nix-daemon[845]: accepted connection from pid 1915, user nix-grpc-daemon (trusted)6475worker1 # [ 138.825922] nix-daemon[845]: reaped child process 1918, status = succeeded6476lb # [ 138.841883] niks3-server[1046]: 2026/09/20 12:22:35 INFO Received uploads request method=POST path=/api/pending_closures6477worker1 # [ 139.097664] nix-grpc-daemon[1739]: time=2026-09-20T12:22:34.337Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6478worker1 # [ 139.106688] nix-grpc-daemon[1739]: time=2026-09-20T12:22:34.337Z level=INFO msg="Uploading xxjh88rg0bhdcm1c6b12clv7l5hfbaa9-farm-b (176B)"6479lb # [ 139.129621] niks3-server[1046]: 2026/09/20 12:22:35 INFO Registered completed upload object_key=log/6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv6480lb # [ 139.142735] niks3-server[1046]: 2026/09/20 12:22:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/55/sign6481lb # [ 139.149720] niks3-server[1046]: 2026/09/20 12:22:35 INFO Signed narinfos id=55 count=16482worker1 # [ 139.150356] nix-grpc-daemon[1739]: time=2026-09-20T12:22:34.390Z level=INFO msg="Uploading 1 narinfos"6483lb # [ 139.152286] niks3-server[1046]: 2026/09/20 12:22:35 INFO Registered completed upload object_key=nar/1a8ws2jbn6nqpggi64yfgxvlwny3nffvyw6svm59is55hih5ppq4.nar.zst6484lb # [ 139.166127] niks3-server[1046]: 2026/09/20 12:22:35 INFO Registered completed upload object_key=xxjh88rg0bhdcm1c6b12clv7l5hfbaa9.ls6485lb # [ 139.169069] niks3-server[1046]: 2026/09/20 12:22:35 INFO Received complete upload request method=POST path=/api/pending_closures/55/complete6486lb # [ 139.181260] niks3-server[1046]: 2026/09/20 12:22:35 INFO Completed upload id=556487worker1 # [ 139.182393] nix-grpc-daemon[1739]: time=2026-09-20T12:22:34.422Z level=INFO msg="Upload complete. (529ms)"6488lb # [ 139.186925] niks3-server[1046]: 2026/09/20 12:22:35 INFO Registered completed upload object_key=xxjh88rg0bhdcm1c6b12clv7l5hfbaa9.narinfo6489worker1 # [ 139.189340] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:34Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46640 duration_s=2 drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv assign_id=11 outputs=16490worker2 # [ 139.177916] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:34Z level=debug event=assigned drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv worker=192.168.1.3:50051 clients=16491worker1 # [ 139.198163] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:34Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:466406492worker2 # [ 139.189497] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:34Z level=debug event=want client=5 drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv cached=06493worker1 # [ 139.212317] nix-daemon[845]: reaped child process 1898, status = succeeded6494worker1 # [ 139.215730] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:34Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:466406495worker1 # [ 139.228733] nix-grpc-daemon[1739]: time=2026-09-20T12:22:34.469Z level=INFO msg="All 1 paths already cached"6496worker1 # [ 139.236103] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:34Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:466406497worker1 # [ 139.246169] nix-daemon[845]: accepted connection from pid 1683, user nix-grpc-daemon (trusted)6498worker1 # [ 139.394931] nix-daemon[845]: accepted connection from pid 1922, user nix-grpc-daemon (trusted)6499worker1 # [ 139.408051] nix-daemon[845]: reaped child process 1925, status = succeeded6500worker1 # [ 139.550230] nix-daemon[845]: accepted connection from pid 1927, user nix-grpc-daemon (trusted)6501worker1 # [ 139.562232] nix-daemon[845]: reaped child process 1930, status = succeeded6502lb # [ 139.574886] niks3-server[1046]: 2026/09/20 12:22:35 INFO Received uploads request method=POST path=/api/pending_closures6503worker1 # [ 139.583928] nix-grpc-daemon[1739]: time=2026-09-20T12:22:34.824Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6504worker1 # [ 139.585859] nix-grpc-daemon[1739]: time=2026-09-20T12:22:34.826Z level=INFO msg="Uploading 8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv (624B)"6505lb # [ 139.603382] niks3-server[1046]: 2026/09/20 12:22:35 INFO Registered completed upload object_key=nar/1dh50s9lbv3zd84hw5q9axrhfi7ms38cl3gyrb9gas00gni4siz2.nar.zst6506lb # [ 139.613398] niks3-server[1046]: 2026/09/20 12:22:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/56/sign6507worker1 # [ 139.619333] nix-grpc-daemon[1739]: time=2026-09-20T12:22:34.859Z level=INFO msg="Uploading 1 narinfos"6508lb # [ 139.619276] niks3-server[1046]: 2026/09/20 12:22:35 INFO Signed narinfos id=56 count=16509lb # [ 139.625087] niks3-server[1046]: 2026/09/20 12:22:35 INFO Registered completed upload object_key=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz.ls6510lb # [ 139.636744] niks3-server[1046]: 2026/09/20 12:22:35 INFO Received complete upload request method=POST path=/api/pending_closures/56/complete6511worker1 # [ 139.643116] nix-grpc-daemon[1739]: time=2026-09-20T12:22:34.883Z level=INFO msg="Upload complete. (386ms)"6512lb # [ 139.642304] niks3-server[1046]: 2026/09/20 12:22:35 INFO Completed upload id=566513worker1 # [ 139.645185] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:34Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46640 duration_s=0 paths=1 nar_bytes_in=6246514lb # [ 139.645570] niks3-server[1046]: 2026/09/20 12:22:35 INFO Registered completed upload object_key=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz.narinfo6515worker1 # [ 139.651435] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:34Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:466406516worker1 # [ 139.656379] nix-daemon[845]: accepted connection from pid 1683, user nix-grpc-daemon (trusted)6517worker1 # [ 139.658633] nix-daemon[845]: reaped child process 1920, status = succeeded6518worker1 # [ 139.663867] nix-daemon[845]: accepted connection from pid 1683, user nix-grpc-daemon (trusted)6519worker1 # [ 140.677215] nix-daemon[845]: reaped child process 1934, status = succeeded6520worker1 # [ 140.816489] nix-daemon[845]: accepted connection from pid 1949, user nix-grpc-daemon (trusted)6521worker1 # [ 140.827271] nix-daemon[845]: reaped child process 1952, status = succeeded6522lb # [ 140.843153] niks3-server[1046]: 2026/09/20 12:22:37 INFO Received uploads request method=POST path=/api/pending_closures6523worker1 # [ 140.848471] nix-grpc-daemon[1739]: time=2026-09-20T12:22:36.088Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6524worker1 # [ 140.850259] nix-grpc-daemon[1739]: time=2026-09-20T12:22:36.090Z level=INFO msg="Uploading wha0h9jdcpcy0srjm92hg7pyknwdqp6h-farm-b (176B)"6525lb # [ 140.860778] niks3-server[1046]: 2026/09/20 12:22:37 INFO Registered completed upload object_key=log/8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv6526lb # [ 140.864972] niks3-server[1046]: 2026/09/20 12:22:37 INFO Registered completed upload object_key=nar/1xpjwixqgm25ifjqv3ldg9cbq2njxf26272sngvq3zhy6zr1mvwg.nar.zst6527lb # [ 140.869340] niks3-server[1046]: 2026/09/20 12:22:37 INFO Received sign narinfos request method=POST path=/api/pending_closures/57/sign6528lb # [ 140.871602] niks3-server[1046]: 2026/09/20 12:22:37 INFO Signed narinfos id=57 count=16529worker1 # [ 140.872604] nix-grpc-daemon[1739]: time=2026-09-20T12:22:36.112Z level=INFO msg="Uploading 1 narinfos"6530lb # [ 140.873800] niks3-server[1046]: 2026/09/20 12:22:37 INFO Registered completed upload object_key=wha0h9jdcpcy0srjm92hg7pyknwdqp6h.ls6531lb # [ 140.878637] niks3-server[1046]: 2026/09/20 12:22:37 INFO Received complete upload request method=POST path=/api/pending_closures/57/complete6532lb # [ 140.882249] niks3-server[1046]: 2026/09/20 12:22:37 INFO Completed upload id=576533worker1 # [ 140.883342] nix-grpc-daemon[1739]: time=2026-09-20T12:22:36.123Z level=INFO msg="Upload complete. (213ms)"6534lb # [ 140.884412] niks3-server[1046]: 2026/09/20 12:22:37 INFO Registered completed upload object_key=wha0h9jdcpcy0srjm92hg7pyknwdqp6h.narinfo6535worker1 # [ 140.885831] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:36Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46640 duration_s=1 drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv assign_id=12 outputs=16536worker2 # [ 140.876353] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:36Z level=debug event=assigned drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv worker=192.168.1.3:50051 clients=16537worker1 # [ 140.890588] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:466406538worker2 # [ 140.881539] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:36Z level=debug event=want client=5 drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv cached=06539worker1 # [ 140.896077] nix-daemon[845]: reaped child process 1932, status = succeeded6540worker1 # [ 140.897510] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:36Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:466406541worker1 # [ 140.902937] nix-grpc-daemon[1739]: time=2026-09-20T12:22:36.143Z level=INFO msg="All 2 paths already cached"6542worker1 # [ 140.906696] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:36Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:466406543worker1 # [ 140.911545] nix-daemon[845]: accepted connection from pid 1683, user nix-grpc-daemon (trusted)6544worker1 # [ 141.064083] nix-daemon[845]: accepted connection from pid 1956, user nix-grpc-daemon (trusted)6545worker1 # [ 141.074852] nix-daemon[845]: reaped child process 1959, status = succeeded6546worker1 # [ 141.230845] nix-daemon[845]: accepted connection from pid 1961, user nix-grpc-daemon (trusted)6547worker1 # [ 141.242937] nix-daemon[845]: reaped child process 1964, status = succeeded6548lb # [ 141.256817] niks3-server[1046]: 2026/09/20 12:22:37 INFO Received uploads request method=POST path=/api/pending_closures6549worker1 # [ 141.265340] nix-grpc-daemon[1739]: time=2026-09-20T12:22:36.505Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6550worker1 # [ 141.267258] nix-grpc-daemon[1739]: time=2026-09-20T12:22:36.507Z level=INFO msg="Uploading 2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv (752B)"6551lb # [ 141.281831] niks3-server[1046]: 2026/09/20 12:22:37 INFO Registered completed upload object_key=nar/10gk7wwjbmc835pm215qvrdyjshy6nblydlws97ya3kcny5wz30c.nar.zst6552lb # [ 141.294978] niks3-server[1046]: 2026/09/20 12:22:37 INFO Received sign narinfos request method=POST path=/api/pending_closures/58/sign6553worker1 # [ 141.301672] nix-grpc-daemon[1739]: time=2026-09-20T12:22:36.542Z level=INFO msg="Uploading 1 narinfos"6554lb # [ 141.300734] niks3-server[1046]: 2026/09/20 12:22:37 INFO Signed narinfos id=58 count=16555lb # [ 141.306085] niks3-server[1046]: 2026/09/20 12:22:37 INFO Registered completed upload object_key=2gb1j4yk6j00fr2sscj3nqixv67k9b9g.ls6556lb # [ 141.316606] niks3-server[1046]: 2026/09/20 12:22:37 INFO Received complete upload request method=POST path=/api/pending_closures/58/complete6557worker1 # [ 141.323726] nix-grpc-daemon[1739]: time=2026-09-20T12:22:36.564Z level=INFO msg="Upload complete. (407ms)"6558lb # [ 141.322670] niks3-server[1046]: 2026/09/20 12:22:37 INFO Completed upload id=586559worker1 # [ 141.325984] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:36Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46640 duration_s=0 paths=1 nar_bytes_in=7526560lb # [ 141.328119] niks3-server[1046]: 2026/09/20 12:22:37 INFO Registered completed upload object_key=2gb1j4yk6j00fr2sscj3nqixv67k9b9g.narinfo6561worker1 # [ 141.331267] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:466406562worker1 # [ 141.340692] nix-daemon[845]: accepted connection from pid 1683, user nix-grpc-daemon (trusted)6563worker1 # [ 141.344537] nix-daemon[845]: reaped child process 1954, status = succeeded6564worker1 # [ 141.349488] nix-daemon[845]: accepted connection from pid 1683, user nix-grpc-daemon (trusted)6565worker1 # [ 141.962458] nix-daemon[845]: reaped child process 1968, status = succeeded6566worker1 # [ 142.103768] nix-daemon[845]: accepted connection from pid 1983, user nix-grpc-daemon (trusted)6567worker1 # [ 142.114537] nix-daemon[845]: reaped child process 1986, status = succeeded6568lb # [ 142.127952] niks3-server[1046]: 2026/09/20 12:22:38 INFO Received uploads request method=POST path=/api/pending_closures6569worker1 # [ 142.137264] nix-grpc-daemon[1739]: time=2026-09-20T12:22:37.377Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6570worker1 # [ 142.138846] nix-grpc-daemon[1739]: time=2026-09-20T12:22:37.379Z level=INFO msg="Uploading 78nhq5a8myvxam7sxfszi5ihz2rlq2gs-farm-top (232B)"6571lb # [ 142.164576] niks3-server[1046]: 2026/09/20 12:22:38 INFO Registered completed upload object_key=log/2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv6572lb # [ 142.182667] niks3-server[1046]: 2026/09/20 12:22:38 INFO Received sign narinfos request method=POST path=/api/pending_closures/59/sign6573worker1 # [ 142.190272] nix-grpc-daemon[1739]: time=2026-09-20T12:22:37.430Z level=INFO msg="Uploading 1 narinfos"6574lb # [ 142.189044] niks3-server[1046]: 2026/09/20 12:22:38 INFO Signed narinfos id=59 count=16575lb # [ 142.192819] niks3-server[1046]: 2026/09/20 12:22:38 INFO Registered completed upload object_key=nar/1gsz81ckdr6lfkhhqmxw3bg6xqqnlz2wx6dzhpycascv9pq13fg7.nar.zst6576lb # [ 142.196834] niks3-server[1046]: 2026/09/20 12:22:38 INFO Registered completed upload object_key=78nhq5a8myvxam7sxfszi5ihz2rlq2gs.ls6577lb # [ 142.206550] niks3-server[1046]: 2026/09/20 12:22:38 INFO Received complete upload request method=POST path=/api/pending_closures/59/complete6578lb # [ 142.208957] niks3-server[1046]: 2026/09/20 12:22:38 INFO Completed upload id=596579worker1 # [ 142.210445] nix-grpc-daemon[1739]: time=2026-09-20T12:22:37.450Z level=INFO msg="Upload complete. (255ms)"6580lb # [ 142.211505] niks3-server[1046]: 2026/09/20 12:22:38 INFO Registered completed upload object_key=78nhq5a8myvxam7sxfszi5ihz2rlq2gs.narinfo6581worker2 # [ 142.204277] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:37Z level=debug event=assigned drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv worker=192.168.1.3:50051 clients=16582worker1 # [ 142.215989] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:37Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46640 duration_s=0 drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv assign_id=13 outputs=16583worker1 # [ 142.227161] nix-daemon[845]: reaped child process 1966, status = succeeded6584worker1 # [ 142.260592] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:466406585worker1 # [ 142.273464] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:37Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:466406586worker1 # [ 142.285512] nix-grpc-daemon[1739]: time=2026-09-20T12:22:37.525Z level=INFO msg="All 2 paths already cached"6587worker1 # [ 142.292870] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:37Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:466406588worker1 # [ 142.300795] nix-daemon[845]: accepted connection from pid 1683, user nix-grpc-daemon (trusted)6589worker1 # [ 142.455024] nix-daemon[845]: accepted connection from pid 1990, user nix-grpc-daemon (trusted)6590worker1 # [ 142.467128] nix-daemon[845]: reaped child process 1993, status = succeeded6591worker1 # [ 142.612713] nix-daemon[845]: accepted connection from pid 1995, user nix-grpc-daemon (trusted)6592worker1 # [ 142.624159] nix-daemon[845]: reaped child process 1998, status = succeeded6593lb # [ 142.636978] niks3-server[1046]: 2026/09/20 12:22:38 INFO Received uploads request method=POST path=/api/pending_closures6594worker1 # [ 142.645927] nix-grpc-daemon[1739]: time=2026-09-20T12:22:37.886Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6595worker1 # [ 142.647698] nix-grpc-daemon[1739]: time=2026-09-20T12:22:37.887Z level=INFO msg="Uploading yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv (752B)"6596lb # [ 142.666679] niks3-server[1046]: 2026/09/20 12:22:38 INFO Registered completed upload object_key=nar/1ibq19fcqq0l9xim9ly5lzg1pk7qh90qsbpya453vnh8p619zvbs.nar.zst6597lb # [ 142.675571] niks3-server[1046]: 2026/09/20 12:22:38 INFO Received sign narinfos request method=POST path=/api/pending_closures/60/sign6598lb # [ 142.681305] niks3-server[1046]: 2026/09/20 12:22:38 INFO Signed narinfos id=60 count=16599worker1 # [ 142.687211] nix-grpc-daemon[1739]: time=2026-09-20T12:22:37.927Z level=INFO msg="Uploading 1 narinfos"6600lb # [ 142.689359] niks3-server[1046]: 2026/09/20 12:22:38 INFO Registered completed upload object_key=yjsciba0gpd99p30v1win6wnqyd4v7b6.ls6601lb # [ 142.699098] niks3-server[1046]: 2026/09/20 12:22:38 INFO Received complete upload request method=POST path=/api/pending_closures/60/complete6602lb # [ 142.704683] niks3-server[1046]: 2026/09/20 12:22:38 INFO Completed upload id=606603worker1 # [ 142.705821] nix-grpc-daemon[1739]: time=2026-09-20T12:22:37.946Z level=INFO msg="Upload complete. (393ms)"6604lb # [ 142.709427] niks3-server[1046]: 2026/09/20 12:22:38 INFO Registered completed upload object_key=yjsciba0gpd99p30v1win6wnqyd4v7b6.narinfo6605worker1 # [ 142.709252] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:37Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46640 duration_s=0 paths=1 nar_bytes_in=7526606worker1 # [ 142.714281] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:466406607worker1 # [ 142.723910] nix-daemon[845]: accepted connection from pid 1683, user nix-grpc-daemon (trusted)6608worker1 # [ 142.727475] nix-daemon[845]: reaped child process 1988, status = succeeded6609worker1 # [ 142.732347] nix-daemon[845]: accepted connection from pid 1683, user nix-grpc-daemon (trusted)6610worker1 # [ 143.344696] nix-daemon[845]: reaped child process 2002, status = succeeded6611worker1 # [ 143.486522] nix-daemon[845]: accepted connection from pid 2017, user nix-grpc-daemon (trusted)6612worker1 # [ 143.498811] nix-daemon[845]: reaped child process 2020, status = succeeded6613lb # [ 143.512498] niks3-server[1046]: 2026/09/20 12:22:39 INFO Received uploads request method=POST path=/api/pending_closures6614worker1 # [ 143.546583] nix-grpc-daemon[1739]: time=2026-09-20T12:22:38.786Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6615worker1 # [ 143.548368] nix-grpc-daemon[1739]: time=2026-09-20T12:22:38.788Z level=INFO msg="Uploading qc1853riggbxq0nbn4qjdymnhx5pncg8-farm-top (232B)"6616lb # [ 143.573941] niks3-server[1046]: 2026/09/20 12:22:39 INFO Registered completed upload object_key=nar/1k6m6033nbrw39zmvqlrbvmg39ygyn710za2hl2fkpsxfx40hl13.nar.zst6617lb # [ 143.585635] niks3-server[1046]: 2026/09/20 12:22:39 INFO Received sign narinfos request method=POST path=/api/pending_closures/61/sign6618lb # [ 143.590248] niks3-server[1046]: 2026/09/20 12:22:39 INFO Signed narinfos id=61 count=16619worker1 # [ 143.594662] nix-grpc-daemon[1739]: time=2026-09-20T12:22:38.834Z level=INFO msg="Uploading 1 narinfos"6620lb # [ 143.596621] niks3-server[1046]: 2026/09/20 12:22:39 INFO Registered completed upload object_key=log/yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv6621lb # [ 143.606373] niks3-server[1046]: 2026/09/20 12:22:39 INFO Received complete upload request method=POST path=/api/pending_closures/61/complete6622lb # [ 143.612057] niks3-server[1046]: 2026/09/20 12:22:39 INFO Completed upload id=616623worker1 # [ 143.613293] nix-grpc-daemon[1739]: time=2026-09-20T12:22:38.853Z level=INFO msg="Upload complete. (273ms)"6624lb # [ 143.616845] niks3-server[1046]: 2026/09/20 12:22:39 INFO Registered completed upload object_key=qc1853riggbxq0nbn4qjdymnhx5pncg8.ls6625lb # [ 143.619313] niks3-server[1046]: 2026/09/20 12:22:39 INFO Registered completed upload object_key=qc1853riggbxq0nbn4qjdymnhx5pncg8.narinfo6626worker2 # [ 143.607382] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:38Z level=debug event=assigned drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv worker=192.168.1.3:50051 clients=16627worker1 # [ 143.618902] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:38Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46640 duration_s=0 drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv assign_id=14 outputs=16628worker1 # [ 143.633741] nix-daemon[845]: reaped child process 2000, status = succeeded6629worker1 # [ 143.663280] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:38Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:466406630worker1 # [ 143.676270] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:38Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:466406631worker1 # [ 143.688571] nix-grpc-daemon[1739]: time=2026-09-20T12:22:38.929Z level=INFO msg="All 2 paths already cached"6632worker1 # [ 143.696173] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:38Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:466406633worker1 # [ 143.706154] nix-daemon[845]: accepted connection from pid 1683, user nix-grpc-daemon (trusted)6634worker1 # [ 143.864903] nix-daemon[845]: accepted connection from pid 2024, user nix-grpc-daemon (trusted)6635worker1 # [ 143.876768] nix-daemon[845]: reaped child process 2027, status = succeeded6636worker1 # [ 144.024792] nix-daemon[845]: accepted connection from pid 2029, user nix-grpc-daemon (trusted)6637worker1 # [ 144.036724] nix-daemon[845]: reaped child process 2032, status = succeeded6638lb # [ 144.047778] niks3-server[1046]: 2026/09/20 12:22:40 INFO Received uploads request method=POST path=/api/pending_closures6639worker1 # [ 144.056217] nix-grpc-daemon[1739]: time=2026-09-20T12:22:39.296Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6640worker1 # [ 144.057923] nix-grpc-daemon[1739]: time=2026-09-20T12:22:39.298Z level=INFO msg="Uploading yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv (752B)"6641lb # [ 144.077471] niks3-server[1046]: 2026/09/20 12:22:40 INFO Registered completed upload object_key=nar/1msrgzhylalg20dw693shbnzf48k8pvzp85llwgrnc970rnrliy7.nar.zst6642lb # [ 144.087842] niks3-server[1046]: 2026/09/20 12:22:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/62/sign6643worker1 # [ 144.096661] nix-grpc-daemon[1739]: time=2026-09-20T12:22:39.337Z level=INFO msg="Uploading 1 narinfos"6644lb # [ 144.092916] niks3-server[1046]: 2026/09/20 12:22:40 INFO Signed narinfos id=62 count=16645lb # [ 144.098876] niks3-server[1046]: 2026/09/20 12:22:40 INFO Registered completed upload object_key=yym31x3l626nfpm249qapx1iw7y15mqb.ls6646lb # [ 144.111812] niks3-server[1046]: 2026/09/20 12:22:40 INFO Received complete upload request method=POST path=/api/pending_closures/62/complete6647lb # [ 144.116353] niks3-server[1046]: 2026/09/20 12:22:40 INFO Completed upload id=626648worker1 # [ 144.117316] nix-grpc-daemon[1739]: time=2026-09-20T12:22:39.357Z level=INFO msg="Upload complete. (399ms)"6649lb # [ 144.119040] niks3-server[1046]: 2026/09/20 12:22:40 INFO Registered completed upload object_key=yym31x3l626nfpm249qapx1iw7y15mqb.narinfo6650worker1 # [ 144.121107] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:39Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46640 duration_s=0 paths=1 nar_bytes_in=7526651worker1 # [ 144.126574] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:39Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:466406652worker1 # [ 144.134883] nix-daemon[845]: accepted connection from pid 1683, user nix-grpc-daemon (trusted)6653worker1 # [ 144.139164] nix-daemon[845]: reaped child process 2022, status = succeeded6654worker1 # [ 144.144961] nix-daemon[845]: accepted connection from pid 1683, user nix-grpc-daemon (trusted)6655worker1 # [ 145.572859] nix-daemon[845]: reaped child process 2036, status = succeeded6656worker1 # [ 145.700089] nix-daemon[845]: accepted connection from pid 2051, user nix-grpc-daemon (trusted)6657worker1 # [ 145.711905] nix-daemon[845]: reaped child process 2054, status = succeeded6658lb # [ 145.719751] niks3-server[1046]: 2026/09/20 12:22:41 INFO Received uploads request method=POST path=/api/pending_closures6659worker1 # [ 145.732093] nix-grpc-daemon[1739]: time=2026-09-20T12:22:40.972Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6660worker1 # [ 145.735060] nix-grpc-daemon[1739]: time=2026-09-20T12:22:40.974Z level=INFO msg="Uploading 0y5v4dgfslb2x47r2h6j0zjvpzg6frg6-farm-top (232B)"6661lb # [ 145.760674] niks3-server[1046]: 2026/09/20 12:22:41 INFO Registered completed upload object_key=log/yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv6662lb # [ 145.771732] niks3-server[1046]: 2026/09/20 12:22:41 INFO Registered completed upload object_key=nar/1h7kcfflb8hb0ylxshiprrirnrs580cwzkz50cz6il8jag61wy50.nar.zst6663lb # [ 145.779774] niks3-server[1046]: 2026/09/20 12:22:41 INFO Received sign narinfos request method=POST path=/api/pending_closures/63/sign6664worker1 # [ 145.786873] nix-grpc-daemon[1739]: time=2026-09-20T12:22:41.027Z level=INFO msg="Uploading 1 narinfos"6665lb # [ 145.786326] niks3-server[1046]: 2026/09/20 12:22:41 INFO Signed narinfos id=63 count=16666lb # [ 145.790438] niks3-server[1046]: 2026/09/20 12:22:41 INFO Registered completed upload object_key=0y5v4dgfslb2x47r2h6j0zjvpzg6frg6.ls6667lb # [ 145.800070] niks3-server[1046]: 2026/09/20 12:22:41 INFO Received complete upload request method=POST path=/api/pending_closures/63/complete6668lb # [ 145.805112] niks3-server[1046]: 2026/09/20 12:22:41 INFO Completed upload id=636669worker1 # [ 145.806289] nix-grpc-daemon[1739]: time=2026-09-20T12:22:41.046Z level=INFO msg="Upload complete. (254ms)"6670lb # [ 145.807759] niks3-server[1046]: 2026/09/20 12:22:41 INFO Registered completed upload object_key=0y5v4dgfslb2x47r2h6j0zjvpzg6frg6.narinfo6671worker1 # [ 145.813316] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:41Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46640 duration_s=1 drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv assign_id=15 outputs=16672worker1 # [ 145.825650] nix-daemon[845]: reaped child process 2034, status = succeeded6673client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&1, in 17.80 seconds)6674worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6675worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)6676client: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag f5 --arg features '["gpu"]'6677client # [ 145.970867] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build -L --store "grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key&restart-grace=30" --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag f5 --arg features "[\"gpu\"]".6678client # Running as unit: gpu.service; invocation ID: eaa533a25424443da69fff71087373eb6679client: (finished: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag f5 --arg features '["gpu"]', in 0.07 seconds)6680lb: must succeed: curl -sf localhost:9901/clusters6681lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)6682lb: must succeed: curl -sf localhost:9901/clusters6683lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6684worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6685worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6686worker1 # [ 146.192950] nix-grpc-daemon[1683]: ts=2026-09-20T12:22:41Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:466406687client # [ 146.184627] nix[1421]: don't know how to build these paths:6688client # [ 146.185756] nix[1421]: /nix/store/ynb3m2rijlmsagnc1dpj672gknpqrkcv-farm-top.drv6689worker2 # [ 146.187611] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:41Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:407046690client # [ 146.201674] nix[1421]: warning: z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv: no connected, non-draining worker for system 'x86_64-linux' offers features {gpu}, waiting up to 30s for a worker6691worker2 # [ 146.201532] nix-grpc-daemon[1456]: ts=2026-09-20T12:22:41Z level=debug event=want client=6 drv=z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv cached=06692lb: must succeed: curl -sf localhost:9901/clusters6693lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)6694lb: must succeed: curl -sf localhost:9901/clusters6695lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)6696worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6697worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)6698client: waiting for success: systemctl show -p Result gpu | grep -q exit-code6699lb # [ 151.482493] envoy[1008]: conn peer=[2001:db8:1::1]:48378 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=27085 tx=10648 ms=175536700client # [ 176.204403] nix[1421]: z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv: no connected, non-draining worker for system 'x86_64-linux' offers features {gpu}6701client # [ 176.211186] nix[1421]: error: dependency 'z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv' failed6702client # [ 176.254112] systemd[1]: gpu.service: Main process exited, code=exited, status=1/FAILURE6703client # [ 176.256544] systemd[1]: gpu.service: Failed with result 'exit-code'.6704client # [ 176.260319] systemd[1]: gpu.service: Consumed 133ms CPU time over 30.283s wall clock time, 35.4M memory peak, 6.7K incoming IP traffic, 5.8K outgoing IP traffic.6705client: (finished: waiting for success: systemctl show -p Result gpu | grep -q exit-code, in 29.60 seconds)6706client: must succeed: journalctl -u gpu -o cat6707client: (finished: must succeed: journalctl -u gpu -o cat, in 0.02 seconds)6708lb: must succeed: curl -sf localhost:9901/clusters6709lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)6710lb: must succeed: curl -sf localhost:9901/clusters6711lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)6712worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6713worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)6714(finished: subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does, in 49.05 seconds)6715subtest: low disk drains a worker and builds go to the other6716worker1: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill6717worker1: (finished: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill, in 0.94 seconds)6718worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free'6719worker1 # [ 178.843944] nix-grpc-daemon[1683]: ts=2026-09-20T12:23:14Z level=info event=unhealthy reason=min_free6720worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free', in 1.11 seconds)6721client: must succeed: nix build -L --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag drain >&26722client # don't know how to build these paths:6723client # /nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv6724worker1 # [ 179.380673] nix-grpc-daemon[1683]: ts=2026-09-20T12:23:14Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:466406725worker2 # [ 179.373521] nix-grpc-daemon[1456]: ts=2026-09-20T12:23:14Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:407046726worker2 # [ 179.395149] nix-grpc-daemon[1456]: ts=2026-09-20T12:23:14Z level=debug event=want client=7 drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv cached=06727worker2 # [ 179.400464] nix-grpc-daemon[1456]: ts=2026-09-20T12:23:14Z level=debug event=assigned drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv worker=192.168.1.4:50051 clients=16728worker2 # [ 179.439711] nix-grpc-daemon[1456]: ts=2026-09-20T12:23:14Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:407046729worker2 # [ 179.451536] nix-grpc-daemon[1456]: ts=2026-09-20T12:23:14Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:407046730client # copying 1 paths to 'lb:50051'...6731client # copying path '/nix/store/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv' to 'lb:50051'...6732worker2 # [ 179.463709] nix-grpc-daemon[1456]: ts=2026-09-20T12:23:14Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:407046733worker2 # [ 179.474374] nix-daemon[937]: accepted connection from pid 1456, user nix-grpc-daemon (trusted)6734worker2 # [ 179.634472] nix-daemon[937]: accepted connection from pid 1743, user nix-grpc-daemon (trusted)6735worker2 # [ 179.647281] nix-daemon[937]: reaped child process 1746, status = succeeded6736worker2 # [ 179.779829] nix-daemon[937]: accepted connection from pid 1748, user nix-grpc-daemon (trusted)6737worker2 # [ 179.792605] nix-daemon[937]: reaped child process 1751, status = succeeded6738lb # [ 179.812793] niks3-server[1046]: 2026/09/20 12:23:15 INFO Received uploads request method=POST path=/api/pending_closures6739worker2 # [ 179.811467] nix-grpc-daemon[1532]: time=2026-09-20T12:23:15.132Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6740worker2 # [ 179.813264] nix-grpc-daemon[1532]: time=2026-09-20T12:23:15.134Z level=INFO msg="Uploading a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv (512B)"6741lb # [ 179.844317] niks3-server[1046]: 2026/09/20 12:23:16 INFO Registered completed upload object_key=nar/1qxv3rw2cwpq06kampqz63l69ci673kh4xz41xmflqybqycfk573.nar.zst6742lb # [ 179.858354] niks3-server[1046]: 2026/09/20 12:23:16 INFO Received sign narinfos request method=POST path=/api/pending_closures/64/sign6743worker2 # [ 179.852088] nix-grpc-daemon[1532]: time=2026-09-20T12:23:15.173Z level=INFO msg="Uploading 1 narinfos"6744lb # [ 179.863478] niks3-server[1046]: 2026/09/20 12:23:16 INFO Signed narinfos id=64 count=16745lb # [ 179.871387] niks3-server[1046]: 2026/09/20 12:23:16 INFO Registered completed upload object_key=a33ymyck2p14d34j7jl4zc8zvry6cysy.ls6746lb # [ 179.879162] niks3-server[1046]: 2026/09/20 12:23:16 INFO Received complete upload request method=POST path=/api/pending_closures/64/complete6747lb # [ 179.887339] niks3-server[1046]: 2026/09/20 12:23:16 INFO Completed upload id=646748worker2 # [ 179.876243] nix-grpc-daemon[1532]: time=2026-09-20T12:23:15.197Z level=INFO msg="Upload complete. (390ms)"6749lb # [ 179.889591] niks3-server[1046]: 2026/09/20 12:23:16 INFO Registered completed upload object_key=a33ymyck2p14d34j7jl4zc8zvry6cysy.narinfo6750worker2 # [ 179.880114] nix-grpc-daemon[1456]: ts=2026-09-20T12:23:15Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40704 duration_s=0 paths=1 nar_bytes_in=5126751worker2 # [ 179.884715] nix-grpc-daemon[1456]: ts=2026-09-20T12:23:15Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:407046752worker2 # [ 179.894204] nix-daemon[937]: accepted connection from pid 1456, user nix-grpc-daemon (trusted)6753worker2 # [ 179.900573] nix-daemon[937]: reaped child process 1741, status = succeeded6754client # building '/nix/store/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv' on lb:50051...6755client # farm-a> worker2: building a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv6756worker2 # [ 179.905399] nix-daemon[937]: accepted connection from pid 1456, user nix-grpc-daemon (trusted)6757worker2 # [ 180.917855] nix-daemon[937]: reaped child process 1755, status = succeeded6758client # farm-a> LOG-farm-a-drain6759worker2 # [ 181.043267] nix-daemon[937]: accepted connection from pid 1770, user nix-grpc-daemon (trusted)6760worker2 # [ 181.055327] nix-daemon[937]: reaped child process 1773, status = succeeded6761lb # [ 181.081123] niks3-server[1046]: 2026/09/20 12:23:17 INFO Received uploads request method=POST path=/api/pending_closures6762worker2 # [ 181.081293] nix-grpc-daemon[1532]: time=2026-09-20T12:23:16.402Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6763worker2 # [ 181.082774] nix-grpc-daemon[1532]: time=2026-09-20T12:23:16.404Z level=INFO msg="Uploading rjdc0hf9mcchhia9m3zkqy1q586i3z3r-farm-a (128B)"6764lb # [ 181.117596] niks3-server[1046]: 2026/09/20 12:23:17 INFO Registered completed upload object_key=log/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv6765lb # [ 181.133127] niks3-server[1046]: 2026/09/20 12:23:17 INFO Registered completed upload object_key=nar/1bf9z7y0ikx07b4zjag42zyrv2pq1ycw4461s579kfpmni9kz4dw.nar.zst6766lb # [ 181.139785] niks3-server[1046]: 2026/09/20 12:23:17 INFO Received sign narinfos request method=POST path=/api/pending_closures/65/sign6767worker2 # [ 181.132891] nix-grpc-daemon[1532]: time=2026-09-20T12:23:16.454Z level=INFO msg="Uploading 1 narinfos"6768lb # [ 181.143683] niks3-server[1046]: 2026/09/20 12:23:17 INFO Signed narinfos id=65 count=16769lb # [ 181.148495] niks3-server[1046]: 2026/09/20 12:23:17 INFO Registered completed upload object_key=rjdc0hf9mcchhia9m3zkqy1q586i3z3r.ls6770lb # [ 181.156317] niks3-server[1046]: 2026/09/20 12:23:17 INFO Received complete upload request method=POST path=/api/pending_closures/65/complete6771lb # [ 181.159969] niks3-server[1046]: 2026/09/20 12:23:17 INFO Completed upload id=656772worker2 # [ 181.148988] nix-grpc-daemon[1532]: time=2026-09-20T12:23:16.470Z level=INFO msg="Upload complete. (237ms)"6773lb # [ 181.162507] niks3-server[1046]: 2026/09/20 12:23:17 INFO Registered completed upload object_key=rjdc0hf9mcchhia9m3zkqy1q586i3z3r.narinfo6774worker2 # [ 181.152665] nix-grpc-daemon[1456]: ts=2026-09-20T12:23:16Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40704 duration_s=1 drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv assign_id=16 outputs=16775worker2 # [ 181.163593] nix-grpc-daemon[1456]: ts=2026-09-20T12:23:16Z level=debug event=want client=7 drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv cached=06776worker2 # [ 181.166575] nix-grpc-daemon[1456]: ts=2026-09-20T12:23:16Z level=debug event=assigned drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv worker=192.168.1.4:50051 clients=16777worker2 # [ 181.171125] nix-daemon[937]: reaped child process 1753, status = succeeded6778worker2 # [ 181.210906] nix-grpc-daemon[1456]: ts=2026-09-20T12:23:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:407046779worker2 # [ 181.224152] nix-grpc-daemon[1456]: ts=2026-09-20T12:23:16Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:407046780worker2 # [ 181.237924] nix-grpc-daemon[1532]: time=2026-09-20T12:23:16.559Z level=INFO msg="All 1 paths already cached"6781client # copying 1 paths to 'lb:50051'...6782client # copying path '/nix/store/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv' to 'lb:50051'...6783worker2 # [ 181.246731] nix-grpc-daemon[1456]: ts=2026-09-20T12:23:16Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:407046784worker2 # [ 181.255266] nix-daemon[937]: accepted connection from pid 1456, user nix-grpc-daemon (trusted)6785worker2 # [ 181.400457] nix-daemon[937]: accepted connection from pid 1779, user nix-grpc-daemon (trusted)6786worker2 # [ 181.413593] nix-daemon[937]: reaped child process 1782, status = succeeded6787lb # [ 181.484299] envoy[1008]: conn peer=[2001:db8:1::1]:59572 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=895 tx=498 ms=300416788worker2 # [ 181.564315] nix-daemon[937]: accepted connection from pid 1784, user nix-grpc-daemon (trusted)6789worker2 # [ 181.576486] nix-daemon[937]: reaped child process 1787, status = succeeded6790lb # [ 181.596742] niks3-server[1046]: 2026/09/20 12:23:17 INFO Received uploads request method=POST path=/api/pending_closures6791worker2 # [ 181.594777] nix-grpc-daemon[1532]: time=2026-09-20T12:23:16.916Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6792worker2 # [ 181.596305] nix-grpc-daemon[1532]: time=2026-09-20T12:23:16.917Z level=INFO msg="Uploading xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv (624B)"6793lb # [ 181.628539] niks3-server[1046]: 2026/09/20 12:23:17 INFO Registered completed upload object_key=nar/06skgfsii1iysx6lv8hy20iyys0q28pmz13rs51634x6bgrlvs7k.nar.zst6794lb # [ 181.637928] niks3-server[1046]: 2026/09/20 12:23:17 INFO Received sign narinfos request method=POST path=/api/pending_closures/66/sign6795worker2 # [ 181.633225] nix-grpc-daemon[1532]: time=2026-09-20T12:23:16.954Z level=INFO msg="Uploading 1 narinfos"6796lb # [ 181.644641] niks3-server[1046]: 2026/09/20 12:23:17 INFO Signed narinfos id=66 count=16797lb # [ 181.649100] niks3-server[1046]: 2026/09/20 12:23:17 INFO Registered completed upload object_key=xihm5w957gm8bh55zxlc6dnczbbl387q.ls6798lb # [ 181.661527] niks3-server[1046]: 2026/09/20 12:23:17 INFO Received complete upload request method=POST path=/api/pending_closures/66/complete6799lb # [ 181.665872] niks3-server[1046]: 2026/09/20 12:23:17 INFO Completed upload id=666800worker2 # [ 181.657316] nix-grpc-daemon[1532]: time=2026-09-20T12:23:16.978Z level=INFO msg="Upload complete. (389ms)"6801lb # [ 181.668785] niks3-server[1046]: 2026/09/20 12:23:17 INFO Registered completed upload object_key=xihm5w957gm8bh55zxlc6dnczbbl387q.narinfo6802worker2 # [ 181.660726] nix-grpc-daemon[1456]: ts=2026-09-20T12:23:16Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40704 duration_s=0 paths=1 nar_bytes_in=6246803worker2 # [ 181.665533] nix-grpc-daemon[1456]: ts=2026-09-20T12:23:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:407046804worker2 # [ 181.675135] nix-daemon[937]: accepted connection from pid 1456, user nix-grpc-daemon (trusted)6805worker2 # [ 181.681733] nix-daemon[937]: reaped child process 1777, status = succeeded6806client # building '/nix/store/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv' on lb:50051...6807worker2 # [ 181.685975] nix-daemon[937]: accepted connection from pid 1456, user nix-grpc-daemon (trusted)6808client # farm-b> worker2: building xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv6809client # farm-b> LOG-farm-b-drain6810worker2 # [ 182.709667] nix-daemon[937]: reaped child process 1791, status = succeeded6811worker2 # [ 182.867833] nix-daemon[937]: accepted connection from pid 1806, user nix-grpc-daemon (trusted)6812worker2 # [ 182.878926] nix-daemon[937]: reaped child process 1809, status = succeeded6813lb # [ 182.905484] niks3-server[1046]: 2026/09/20 12:23:19 INFO Received uploads request method=POST path=/api/pending_closures6814worker2 # [ 182.902695] nix-grpc-daemon[1532]: time=2026-09-20T12:23:18.224Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6815worker2 # [ 182.905059] nix-grpc-daemon[1532]: time=2026-09-20T12:23:18.225Z level=INFO msg="Uploading pmvxn7ymjj8pdn6gb65j5kqb45n38x0p-farm-b (176B)"6816lb # [ 182.941959] niks3-server[1046]: 2026/09/20 12:23:19 INFO Registered completed upload object_key=log/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv6817lb # [ 182.950930] niks3-server[1046]: 2026/09/20 12:23:19 INFO Registered completed upload object_key=nar/17gyl0zgiivck81rhv539x50216ksc3nsf369ismwd4j1zwsdqrb.nar.zst6818lb # [ 182.961639] niks3-server[1046]: 2026/09/20 12:23:19 INFO Received sign narinfos request method=POST path=/api/pending_closures/67/sign6819lb # [ 182.968051] niks3-server[1046]: 2026/09/20 12:23:19 INFO Signed narinfos id=67 count=16820worker2 # [ 182.959865] nix-grpc-daemon[1532]: time=2026-09-20T12:23:18.281Z level=INFO msg="Uploading 1 narinfos"6821lb # [ 182.971104] niks3-server[1046]: 2026/09/20 12:23:19 INFO Registered completed upload object_key=pmvxn7ymjj8pdn6gb65j5kqb45n38x0p.ls6822lb # [ 182.981634] niks3-server[1046]: 2026/09/20 12:23:19 INFO Received complete upload request method=POST path=/api/pending_closures/67/complete6823lb # [ 182.986486] niks3-server[1046]: 2026/09/20 12:23:19 INFO Completed upload id=676824worker2 # [ 182.975690] nix-grpc-daemon[1532]: time=2026-09-20T12:23:18.297Z level=INFO msg="Upload complete. (283ms)"6825lb # [ 182.989540] niks3-server[1046]: 2026/09/20 12:23:19 INFO Registered completed upload object_key=pmvxn7ymjj8pdn6gb65j5kqb45n38x0p.narinfo6826worker2 # [ 182.983948] nix-grpc-daemon[1456]: ts=2026-09-20T12:23:18Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40704 duration_s=1 drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv assign_id=17 outputs=16827worker2 # [ 182.993885] nix-grpc-daemon[1456]: ts=2026-09-20T12:23:18Z level=debug event=want client=7 drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv cached=06828worker2 # [ 182.998393] nix-grpc-daemon[1456]: ts=2026-09-20T12:23:18Z level=debug event=assigned drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv worker=192.168.1.4:50051 clients=16829worker2 # [ 183.002219] nix-grpc-daemon[1456]: ts=2026-09-20T12:23:18Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:407046830worker2 # [ 183.011694] nix-daemon[937]: reaped child process 1789, status = succeeded6831worker2 # [ 183.013508] nix-grpc-daemon[1456]: ts=2026-09-20T12:23:18Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:407046832worker2 # [ 183.020805] nix-grpc-daemon[1532]: time=2026-09-20T12:23:18.342Z level=INFO msg="All 2 paths already cached"6833client # copying 1 paths to 'lb:50051'...6834worker2 # [ 183.026084] nix-grpc-daemon[1456]: ts=2026-09-20T12:23:18Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:407046835client # copying path '/nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv' to 'lb:50051'...6836worker2 # [ 183.032362] nix-daemon[937]: accepted connection from pid 1456, user nix-grpc-daemon (trusted)6837worker2 # [ 183.186410] nix-daemon[937]: accepted connection from pid 1815, user nix-grpc-daemon (trusted)6838worker2 # [ 183.200155] nix-daemon[937]: reaped child process 1818, status = succeeded6839worker2 # [ 183.359519] nix-daemon[937]: accepted connection from pid 1820, user nix-grpc-daemon (trusted)6840worker2 # [ 183.372529] nix-daemon[937]: reaped child process 1823, status = succeeded6841lb # [ 183.399542] niks3-server[1046]: 2026/09/20 12:23:19 INFO Received uploads request method=POST path=/api/pending_closures6842worker2 # [ 183.397131] nix-grpc-daemon[1532]: time=2026-09-20T12:23:18.718Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6843worker2 # [ 183.398613] nix-grpc-daemon[1532]: time=2026-09-20T12:23:18.720Z level=INFO msg="Uploading q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv (752B)"6844lb # [ 183.432162] niks3-server[1046]: 2026/09/20 12:23:19 INFO Registered completed upload object_key=nar/1m4vpncprrwddjpk4qry79vzd9g39s9pjs6874i2aarpm7ax0ykq.nar.zst6845lb # [ 183.439890] niks3-server[1046]: 2026/09/20 12:23:19 INFO Received sign narinfos request method=POST path=/api/pending_closures/68/sign6846worker2 # [ 183.433525] nix-grpc-daemon[1532]: time=2026-09-20T12:23:18.755Z level=INFO msg="Uploading 1 narinfos"6847lb # [ 183.444744] niks3-server[1046]: 2026/09/20 12:23:19 INFO Signed narinfos id=68 count=16848lb # [ 183.449765] niks3-server[1046]: 2026/09/20 12:23:19 INFO Registered completed upload object_key=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s.ls6849lb # [ 183.460713] niks3-server[1046]: 2026/09/20 12:23:19 INFO Received complete upload request method=POST path=/api/pending_closures/68/complete6850lb # [ 183.466512] niks3-server[1046]: 2026/09/20 12:23:19 INFO Completed upload id=686851worker2 # [ 183.455646] nix-grpc-daemon[1532]: time=2026-09-20T12:23:18.777Z level=INFO msg="Upload complete. (418ms)"6852lb # [ 183.469131] niks3-server[1046]: 2026/09/20 12:23:19 INFO Registered completed upload object_key=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s.narinfo6853worker2 # [ 183.459420] nix-grpc-daemon[1456]: ts=2026-09-20T12:23:18Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40704 duration_s=0 paths=1 nar_bytes_in=7526854worker2 # [ 183.464810] nix-grpc-daemon[1456]: ts=2026-09-20T12:23:18Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:407046855worker2 # [ 183.473968] nix-daemon[937]: accepted connection from pid 1456, user nix-grpc-daemon (trusted)6856worker2 # [ 183.478690] nix-daemon[937]: reaped child process 1813, status = succeeded6857worker2 # [ 183.483913] nix-daemon[937]: accepted connection from pid 1456, user nix-grpc-daemon (trusted)6858client # building '/nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv' on lb:50051...6859client # farm-top> worker2: building q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv6860client # farm-top> LOG-farm-top-drain6861worker2 # [ 184.497541] nix-daemon[937]: reaped child process 1827, status = succeeded6862worker2 # [ 184.614994] nix-daemon[937]: accepted connection from pid 1842, user nix-grpc-daemon (trusted)6863worker2 # [ 184.627876] nix-daemon[937]: reaped child process 1845, status = succeeded6864lb # [ 184.650695] niks3-server[1046]: 2026/09/20 12:23:20 INFO Received uploads request method=POST path=/api/pending_closures6865worker2 # [ 184.647574] nix-grpc-daemon[1532]: time=2026-09-20T12:23:19.969Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6866worker2 # [ 184.649256] nix-grpc-daemon[1532]: time=2026-09-20T12:23:19.970Z level=INFO msg="Uploading hgmfp1yw6m31jn6cpma19ikl6dy1b17r-farm-top (232B)"6867lb # [ 184.679454] niks3-server[1046]: 2026/09/20 12:23:20 INFO Registered completed upload object_key=log/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv6868lb # [ 184.696547] niks3-server[1046]: 2026/09/20 12:23:20 INFO Registered completed upload object_key=nar/1xin0j0xlljjqnd952hnxy8yw45d9j9qpj7m241vv6cckfq8kk4p.nar.zst6869lb # [ 184.703523] niks3-server[1046]: 2026/09/20 12:23:20 INFO Received sign narinfos request method=POST path=/api/pending_closures/69/sign6870lb # [ 184.708359] niks3-server[1046]: 2026/09/20 12:23:20 INFO Signed narinfos id=69 count=16871worker2 # [ 184.700417] nix-grpc-daemon[1532]: time=2026-09-20T12:23:20.021Z level=INFO msg="Uploading 1 narinfos"6872lb # [ 184.716360] niks3-server[1046]: 2026/09/20 12:23:20 INFO Registered completed upload object_key=hgmfp1yw6m31jn6cpma19ikl6dy1b17r.ls6873lb # [ 184.724256] niks3-server[1046]: 2026/09/20 12:23:20 INFO Received complete upload request method=POST path=/api/pending_closures/69/complete6874lb # [ 184.730103] niks3-server[1046]: 2026/09/20 12:23:20 INFO Completed upload id=696875worker2 # [ 184.719386] nix-grpc-daemon[1532]: time=2026-09-20T12:23:20.040Z level=INFO msg="Upload complete. (228ms)"6876lb # [ 184.733314] niks3-server[1046]: 2026/09/20 12:23:20 INFO Registered completed upload object_key=hgmfp1yw6m31jn6cpma19ikl6dy1b17r.narinfo6877worker2 # [ 184.725606] nix-grpc-daemon[1456]: ts=2026-09-20T12:23:20Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40704 duration_s=1 drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv assign_id=18 outputs=16878worker2 # [ 184.738880] nix-daemon[937]: reaped child process 1825, status = succeeded6879client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?ca-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ca.pem&client-cert=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.pem&client-key=/nix/store/8vm13krkgkcd458yp3j43c8n27rmhbpz-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag drain >&2, in 5.63 seconds)6880client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag drain outPath6881client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag drain outPath, in 0.19 seconds)6882worker1: must succeed: rm /nix/.rw-store/fill6883worker1: (finished: must succeed: rm /nix/.rw-store/fill, in 0.13 seconds)6884worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free'6885worker1 # [ 185.847660] nix-grpc-daemon[1683]: ts=2026-09-20T12:23:21Z level=info event=healthy reason=min_free6886worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free', in 1.11 seconds)6887(finished: subtest: low disk drains a worker and builds go to the other, in 9.18 seconds)6888(finished: run the VM test script, in 187.29 seconds)6889test script finished in 187.49s6890cleanup6891kill QemuMachine (pid 49)6892client # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/b5bpi6zfajzzrwwpgba2q6li3nnya4bs-python3-3.14.7/bin/python3.14)6893kill QemuMachine (pid 50)6894lb # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/b5bpi6zfajzzrwwpgba2q6li3nnya4bs-python3-3.14.7/bin/python3.14)6895kill QemuMachine (pid 51)6896worker1 # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/b5bpi6zfajzzrwwpgba2q6li3nnya4bs-python3-3.14.7/bin/python3.14)6897kill QemuMachine (pid 52)6898worker2 # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/b5bpi6zfajzzrwwpgba2q6li3nnya4bs-python3-3.14.7/bin/python3.14)6899(finished: cleanup, in 1.08 seconds)6900additionally exposed symbols:6901 client, lb, worker1, worker2,6902 vlan1,6903 start_all, test_script, machines, machines_qemu, machines_nspawn, vlans, driver, log, os, create_machine, subtest, run_tests, join_all, retry, serial_stdout_off, serial_stdout_on, polling_condition, BaseMachine, QemuMachine, NspawnMachine, t, debug, dump_machine_ssh