vm-test-run-nix-grpc-farm
checks.aarch64-linux.farm
· build #480
· raw
1tribuchet: building on eliza2Machine state will be reset. To keep it, pass --keep-machine-state3start all VLans4(finished: start all VLans, in 0.00 seconds)5Test will time out and terminate in 900.0 seconds6run the VM test script7start all VMs8client: starting vm9lb: starting vm10worker1: starting vm11worker2: starting vm12worker1 # Disk image does not exist, creating the virtualisation disk image...13lb # Disk image does not exist, creating the virtualisation disk image...14worker1 # Formatting '/build/vm-state-worker1/tmp.UvaNkIoGPg', fmt=raw size=107374182415worker1: QEMU running (pid 51)16lb # Formatting '/build/vm-state-lb/tmp.0UL2hI1yNH', fmt=raw size=107374182417lb: QEMU running (pid 49)18worker2 # Disk image does not exist, creating the virtualisation disk image...19worker2: QEMU running (pid 52)20lb # mke2fs 1.47.4 (6-Mar-2025)21client # Disk image does not exist, creating the virtualisation disk image...22client: QEMU running (pid 50)23lb # Discarding device blocks: 0/262144 done24client # Formatting '/build/vm-state-client/tmp.fadhmClmP3', fmt=raw size=107374182425worker2 # Formatting '/build/vm-state-worker2/tmp.hhniQSML51', fmt=raw size=107374182426worker1 # mke2fs 1.47.4 (6-Mar-2025)27lb # Creating filesystem with 262144 4k blocks and 65536 inodes28client # mke2fs 1.47.4 (6-Mar-2025)29(finished: start all VMs, in 0.46 seconds)30lb # Filesystem UUID: 32ce671e-284c-4bdb-b1b3-4bb2ba4325d831client # Discarding device blocks: 0/262144 done32client: waiting for unit network-addresses-eth1.service33lb # Superblock backups stored on blocks:34worker2 # mke2fs 1.47.4 (6-Mar-2025)35worker1 # Discarding device blocks: 0/262144 done36lb # 32768, 98304, 163840, 22937637client: waiting for the VM to finish booting38client # Creating filesystem with 262144 4k blocks and 65536 inodes39lb # 40worker2 # Discarding device blocks: 0/262144 done41worker1 # Creating filesystem with 262144 4k blocks and 65536 inodes42worker2 # Creating filesystem with 262144 4k blocks and 65536 inodes43worker1 # Filesystem UUID: 1ca2ebb5-06c7-441d-ade7-bf2e4a4cf31444lb # Allocating group tables: 0/8 done45client # Filesystem UUID: 1254f221-7786-43be-840c-ce3d9642ab1046worker2 # Filesystem UUID: e3dfeb75-f1f6-4868-8710-4a19bb11e72047worker1 # Superblock backups stored on blocks:48lb # Writing inode tables: 0/8 done49client # Superblock backups stored on blocks:50worker2 # Superblock backups stored on blocks:51worker1 # 32768, 98304, 163840, 22937652lb # Creating journal (8192 blocks): done53client # 32768, 98304, 163840, 22937654worker2 # 32768, 98304, 163840, 22937655worker1 # 56lb # Writing superblocks and filesystem accounting information: 0/8 done57client # 58worker2 # 59worker1 # Allocating group tables: 0/8 done60lb # 61client # Allocating group tables: 0/8 done62worker2 # Allocating group tables: 0/8 done63worker1 # Writing inode tables: 0/8 done64lb # Virtualisation disk image created.65client # Writing inode tables: 0/8 done66worker2 # Writing inode tables: 0/8 done67worker1 # Creating journal (8192 blocks): done68lb # Starting virtiofs daemons...69client # Creating journal (8192 blocks): done70worker2 # Creating journal (8192 blocks): done71worker1 # Writing superblocks and filesystem accounting information: 0/8 done72lb # [2026-09-21T14:43:34Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)73client # Writing superblocks and filesystem accounting information: 0/8 done74worker2 # Writing superblocks and filesystem accounting information: 0/8 done75worker1 # 76lb # [2026-09-21T14:43:34Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether77client # 78worker2 # 79worker1 # Virtualisation disk image created.80lb # [2026-09-21T14:43:34Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)81client # Virtualisation disk image created.82worker2 # Virtualisation disk image created.83worker1 # Starting virtiofs daemons...84lb # [2026-09-21T14:43:34Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether85client # Starting virtiofs daemons...86worker2 # Starting virtiofs daemons...87worker1 # [2026-09-21T14:43:34Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)88lb # [2026-09-21T14:43:34Z INFO virtiofsd] Waiting for vhost-user socket connection...89client # [2026-09-21T14:43:34Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)90worker2 # [2026-09-21T14:43:34Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)91worker1 # [2026-09-21T14:43:34Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether92lb # [2026-09-21T14:43:34Z INFO virtiofsd] Waiting for vhost-user socket connection...93client # [2026-09-21T14:43:34Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether94worker2 # [2026-09-21T14:43:34Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether95worker1 # [2026-09-21T14:43:34Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)96lb # [2026-09-21T14:43:34Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)97client # [2026-09-21T14:43:34Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)98worker2 # [2026-09-21T14:43:34Z INFO virtiofsd] Waiting for vhost-user socket connection...99worker1 # [2026-09-21T14:43:34Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether100lb # [2026-09-21T14:43:34Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether101client # [2026-09-21T14:43:34Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether102worker2 # [2026-09-21T14:43:34Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)103worker1 # [2026-09-21T14:43:34Z INFO virtiofsd] Waiting for vhost-user socket connection...104lb # [2026-09-21T14:43:34Z INFO virtiofsd] Waiting for vhost-user socket connection...105client # [2026-09-21T14:43:34Z INFO virtiofsd] Waiting for vhost-user socket connection...106worker2 # [2026-09-21T14:43:34Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether107worker1 # [2026-09-21T14:43:34Z INFO virtiofsd] Waiting for vhost-user socket connection...108lb # [2026-09-21T14:43:34Z INFO virtiofsd] Client connected, servicing requests109client # [2026-09-21T14:43:34Z INFO virtiofsd] Waiting for vhost-user socket connection...110worker2 # [2026-09-21T14:43:34Z INFO virtiofsd] Waiting for vhost-user socket connection...111worker1 # [2026-09-21T14:43:34Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)112lb # [2026-09-21T14:43:34Z INFO virtiofsd] Client connected, servicing requests113client # [2026-09-21T14:43:34Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)114worker2 # [2026-09-21T14:43:34Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)115worker1 # [2026-09-21T14:43:34Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether116lb # [2026-09-21T14:43:34Z INFO virtiofsd] Client connected, servicing requests117client # [2026-09-21T14:43:34Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether118worker2 # [2026-09-21T14:43:34Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether119client # [2026-09-21T14:43:34Z INFO virtiofsd] Waiting for vhost-user socket connection...120worker2 # [2026-09-21T14:43:34Z INFO virtiofsd] Waiting for vhost-user socket connection...121client # [2026-09-21T14:43:34Z INFO virtiofsd] Client connected, servicing requests122worker2 # [2026-09-21T14:43:34Z INFO virtiofsd] Client connected, servicing requests123client # [2026-09-21T14:43:34Z INFO virtiofsd] Client connected, servicing requests124worker2 # [2026-09-21T14:43:34Z INFO virtiofsd] Client connected, servicing requests125client # [2026-09-21T14:43:34Z INFO virtiofsd] Client connected, servicing requests126worker2 # [2026-09-21T14:43:34Z INFO virtiofsd] Client connected, servicing requests127worker1 # [2026-09-21T14:43:34Z INFO virtiofsd] Waiting for vhost-user socket connection...128worker1 # [2026-09-21T14:43:34Z INFO virtiofsd] Client connected, servicing requests129worker1 # [2026-09-21T14:43:34Z INFO virtiofsd] Client connected, servicing requests130worker1 # [2026-09-21T14:43:34Z INFO virtiofsd] Client connected, servicing requests131lb # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40]132worker1 # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40]133lb # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 14 11:36:19 UTC 2026134lb # [ 0.000000] KASLR enabled135worker1 # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 14 11:36:19 UTC 2026136lb # [ 0.000000] random: crng init done137worker1 # [ 0.000000] KASLR enabled138lb # [ 0.000000] Machine model: linux,dummy-virt139worker1 # [ 0.000000] random: crng init done140lb # [ 0.000000] efi: UEFI not found.141worker1 # [ 0.000000] Machine model: linux,dummy-virt142worker1 # [ 0.000000] efi: UEFI not found.143lb # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT144client # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40]145worker1 # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT146lb # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff]147worker1 # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff]148lb # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f]149lb # [ 0.000000] Zone ranges:150client # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 14 11:36:19 UTC 2026151worker1 # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f]152client # [ 0.000000] KASLR enabled153worker1 # [ 0.000000] Zone ranges:154lb # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff]155client # [ 0.000000] random: crng init done156lb # [ 0.000000] DMA32 empty157client # [ 0.000000] Machine model: linux,dummy-virt158worker1 # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff]159lb # [ 0.000000] Normal empty160client # [ 0.000000] efi: UEFI not found.161worker1 # [ 0.000000] DMA32 empty162lb # [ 0.000000] Device empty163worker1 # [ 0.000000] Normal empty164lb # [ 0.000000] Movable zone start for each node165worker1 # [ 0.000000] Device empty166client # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT167lb # [ 0.000000] Early memory node ranges168worker1 # [ 0.000000] Movable zone start for each node169client # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff]170worker1 # [ 0.000000] Early memory node ranges171lb # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff]172client # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f]173worker1 # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff]174client # [ 0.000000] Zone ranges:175lb # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff]176lb # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000177worker1 # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff]178client # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff]179client # [ 0.000000] DMA32 empty180lb # [ 0.000000] psci: probing for conduit method from DT.181worker1 # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000182client # [ 0.000000] Normal empty183client # [ 0.000000] Device empty184lb # [ 0.000000] psci: PSCIv1.3 detected in firmware.185worker1 # [ 0.000000] psci: probing for conduit method from DT.186client # [ 0.000000] Movable zone start for each node187lb # [ 0.000000] psci: Using standard PSCI v0.2 function IDs188worker1 # [ 0.000000] psci: PSCIv1.3 detected in firmware.189client # [ 0.000000] Early memory node ranges190lb # [ 0.000000] psci: Trusted OS migration not required191worker1 # [ 0.000000] psci: Using standard PSCI v0.2 function IDs192client # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff]193lb # [ 0.000000] psci: SMC Calling Convention v1.1194worker1 # [ 0.000000] psci: Trusted OS migration not required195worker1 # [ 0.000000] psci: SMC Calling Convention v1.1196client # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff]197lb # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003)198client # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000199worker1 # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003)200lb # [ 0.000000] percpu: Embedded 76 pages/cpu s186712 r8192 d116392 u311296201client # [ 0.000000] psci: probing for conduit method from DT.202lb # [ 0.000000] Detected PIPT I-cache on CPU0203client # [ 0.000000] psci: PSCIv1.3 detected in firmware.204worker1 # [ 0.000000] percpu: Embedded 76 pages/cpu s186712 r8192 d116392 u311296205worker1 # [ 0.000000] Detected PIPT I-cache on CPU0206client # [ 0.000000] psci: Using standard PSCI v0.2 function IDs207lb # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm)208client # [ 0.000000] psci: Trusted OS migration not required209lb # [ 0.000000] CPU features: detected: GICv3 CPU interface210worker1 # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm)211client # [ 0.000000] psci: SMC Calling Convention v1.1212lb # [ 0.000000] CPU features: detected: Spectre-v4213worker1 # [ 0.000000] CPU features: detected: GICv3 CPU interface214lb # [ 0.000000] CPU features: detected: Spectre-BHB215client # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003)216worker1 # [ 0.000000] CPU features: detected: Spectre-v4217worker1 # [ 0.000000] CPU features: detected: Spectre-BHB218lb # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38219client # [ 0.000000] percpu: Embedded 76 pages/cpu s186712 r8192 d116392 u311296220worker1 # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38221lb # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23222client # [ 0.000000] Detected PIPT I-cache on CPU0223lb # [ 0.000000] alternatives: applying boot alternatives224worker1 # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23225client # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm)226worker1 # [ 0.000000] alternatives: applying boot alternatives227client # [ 0.000000] CPU features: detected: GICv3 CPU interface228worker2 # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40]229client # [ 0.000000] CPU features: detected: Spectre-v4230client # [ 0.000000] CPU features: detected: Spectre-BHB231client # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38232worker2 # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 14 11:36:19 UTC 2026233worker2 # [ 0.000000] KASLR enabled234client # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23235lb # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/gcbvv54lc1mkgavjm2x1hn2y3wpbcps9-nixos-system-lb-test/init regInfo=/nix/store/zdpr9k12xnmsfn00vay1jzchyfdyhwwc-closure-info/registration console=ttyAMA0,115200n8 console=tty0236worker2 # [ 0.000000] random: crng init done237client # [ 0.000000] alternatives: applying boot alternatives238worker2 # [ 0.000000] Machine model: linux,dummy-virt239worker1 # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/6xkk768q3zcqgvlrnm0xb1lyjz91891g-nixos-system-worker1-test/init regInfo=/nix/store/3946mr0lkcyn2ygj7zagrllcql5yvyk5-closure-info/registration console=ttyAMA0,115200n8 console=tty0240worker2 # [ 0.000000] efi: UEFI not found.241lb # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/zdpr9k12xnmsfn00vay1jzchyfdyhwwc-closure-info/registration", will be passed to user space.242worker2 # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT243worker1 # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/3946mr0lkcyn2ygj7zagrllcql5yvyk5-closure-info/registration", will be passed to user space.244lb # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes245worker2 # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff]246worker1 # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes247lb # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)248worker2 # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f]249worker2 # [ 0.000000] Zone ranges:250client # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/6v5lkzqzv8y8aql79zri93a767rl7bwj-nixos-system-client-test/init regInfo=/nix/store/d6pwbdkbnvnr66k6s6bqylk0yc9mwdmr-closure-info/registration console=ttyAMA0,115200n8 console=tty0251worker1 # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)252lb # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)253worker2 # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff]254worker2 # [ 0.000000] DMA32 empty255lb # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB256worker1 # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)257worker2 # [ 0.000000] Normal empty258lb # [ 0.000000] software IO TLB: area num 1.259worker2 # [ 0.000000] Device empty260worker1 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB261worker2 # [ 0.000000] Movable zone start for each node262lb # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB263worker1 # [ 0.000000] software IO TLB: area num 1.264client # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/d6pwbdkbnvnr66k6s6bqylk0yc9mwdmr-closure-info/registration", will be passed to user space.265worker2 # [ 0.000000] Early memory node ranges266worker1 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB267lb # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB)268client # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes269worker2 # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff]270lb # [ 0.000000] Fallback order for Node 0: 0271worker1 # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB)272worker2 # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff]273client # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)274worker1 # [ 0.000000] Fallback order for Node 0: 0275lb # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216276worker2 # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000277lb # [ 0.000000] Policy zone: DMA278client # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)279worker1 # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216280worker2 # [ 0.000000] psci: probing for conduit method from DT.281lb # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off282worker2 # [ 0.000000] psci: PSCIv1.3 detected in firmware.283client # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB284worker1 # [ 0.000000] Policy zone: DMA285client # [ 0.000000] software IO TLB: area num 1.286lb # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1287worker2 # [ 0.000000] psci: Using standard PSCI v0.2 function IDs288lb # [ 0.000000] allocated 3145728 bytes of page_ext289worker2 # [ 0.000000] psci: Trusted OS migration not required290worker1 # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off291client # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB292lb # [ 0.000000] ftrace: allocating 74950 entries in 294 pages293worker2 # [ 0.000000] psci: SMC Calling Convention v1.1294worker1 # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1295client # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB)296lb # [ 0.000000] ftrace: allocated 294 pages with 4 groups297worker1 # [ 0.000000] allocated 3145728 bytes of page_ext298worker2 # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003)299client # [ 0.000000] Fallback order for Node 0: 0300lb # [ 0.000000] rcu: Hierarchical RCU implementation.301worker1 # [ 0.000000] ftrace: allocating 74950 entries in 294 pages302lb # [ 0.000000] rcu: RCU event tracing is enabled.303client # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216304worker2 # [ 0.000000] percpu: Embedded 76 pages/cpu s186712 r8192 d116392 u311296305worker1 # [ 0.000000] ftrace: allocated 294 pages with 4 groups306client # [ 0.000000] Policy zone: DMA307worker2 # [ 0.000000] Detected PIPT I-cache on CPU0308lb # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.309worker1 # [ 0.000000] rcu: Hierarchical RCU implementation.310client # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off311lb # [ 0.000000] Trampoline variant of Tasks RCU enabled.312worker1 # [ 0.000000] rcu: RCU event tracing is enabled.313worker2 # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm)314lb # [ 0.000000] Rude variant of Tasks RCU enabled.315client # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1316worker2 # [ 0.000000] CPU features: detected: GICv3 CPU interface317worker1 # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.318lb # [ 0.000000] Tracing variant of Tasks RCU enabled.319client # [ 0.000000] allocated 3145728 bytes of page_ext320worker2 # [ 0.000000] CPU features: detected: Spectre-v4321worker1 # [ 0.000000] Trampoline variant of Tasks RCU enabled.322client # [ 0.000000] ftrace: allocating 74950 entries in 294 pages323lb # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.324worker1 # [ 0.000000] Rude variant of Tasks RCU enabled.325worker2 # [ 0.000000] CPU features: detected: Spectre-BHB326client # [ 0.000000] ftrace: allocated 294 pages with 4 groups327worker1 # [ 0.000000] Tracing variant of Tasks RCU enabled.328worker2 # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38329lb # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1330client # [ 0.000000] rcu: Hierarchical RCU implementation.331worker2 # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23332worker1 # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.333client # [ 0.000000] rcu: RCU event tracing is enabled.334lb # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.335worker2 # [ 0.000000] alternatives: applying boot alternatives336client # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.337worker1 # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1338client # [ 0.000000] Trampoline variant of Tasks RCU enabled.339lb # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.340worker1 # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.341client # [ 0.000000] Rude variant of Tasks RCU enabled.342lb # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.343client # [ 0.000000] Tracing variant of Tasks RCU enabled.344worker1 # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.345lb # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0346client # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.347lb # [ 0.000000] GICv3: 256 SPIs implemented348worker1 # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.349lb # [ 0.000000] GICv3: 0 Extended SPIs implemented350client # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1351worker2 # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/1z69h5i65q7pa63098rza1vngylyipj5-nixos-system-worker2-test/init regInfo=/nix/store/a7nf1gfcihx1v6ly6fqcjk8c321ypyrd-closure-info/registration console=ttyAMA0,115200n8 console=tty0352worker1 # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0353lb # [ 0.000000] Root IRQ handler: gic_handle_irq354worker1 # [ 0.000000] GICv3: 256 SPIs implemented355lb # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI356client # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.357worker1 # [ 0.000000] GICv3: 0 Extended SPIs implemented358lb # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0359worker2 # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/a7nf1gfcihx1v6ly6fqcjk8c321ypyrd-closure-info/registration", will be passed to user space.360worker1 # [ 0.000000] Root IRQ handler: gic_handle_irq361client # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.362lb # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000363worker1 # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI364worker2 # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes365lb # [ 0.000000] ITS [mem 0x08080000-0x0809ffff]366worker1 # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0367client # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.368worker2 # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)369client # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0370worker1 # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000371lb # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1)372client # [ 0.000000] GICv3: 256 SPIs implemented373worker1 # [ 0.000000] ITS [mem 0x08080000-0x0809ffff]374worker2 # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)375client # [ 0.000000] GICv3: 0 Extended SPIs implemented376lb # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1)377worker2 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB378client # [ 0.000000] Root IRQ handler: gic_handle_irq379worker1 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1)380worker2 # [ 0.000000] software IO TLB: area num 1.381client # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI382lb # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000383client # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0384worker2 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB385worker1 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1)386lb # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000387client # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000388worker1 # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000389worker2 # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB)390lb # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.391client # [ 0.000000] ITS [mem 0x08080000-0x0809ffff]392worker2 # [ 0.000000] Fallback order for Node 0: 0393worker1 # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000394lb # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns395worker2 # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216396client # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1)397worker2 # [ 0.000000] Policy zone: DMA398worker1 # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.399lb # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt).400worker2 # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off401client # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1)402worker1 # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns403worker2 # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1404lb # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns405client # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000406worker1 # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt).407worker2 # [ 0.000000] allocated 3145728 bytes of page_ext408client # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000409lb # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns410worker2 # [ 0.000000] ftrace: allocating 74950 entries in 294 pages411lb # [ 0.000031] arm-pv: using stolen time PV412worker1 # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns413worker2 # [ 0.000000] ftrace: allocated 294 pages with 4 groups414client # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.415worker2 # [ 0.000000] rcu: Hierarchical RCU implementation.416worker1 # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns417lb # [ 0.000476] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)418client # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns419worker1 # [ 0.000031] arm-pv: using stolen time PV420worker2 # [ 0.000000] rcu: RCU event tracing is enabled.421lb # [ 0.000670] Console: colour dummy device 80x25422worker2 # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.423client # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt).424lb # [ 0.000678] printk: legacy console [tty0] enabled425worker1 # [ 0.000544] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)426worker2 # [ 0.000000] Trampoline variant of Tasks RCU enabled.427worker1 # [ 0.000726] Console: colour dummy device 80x25428worker2 # [ 0.000000] Rude variant of Tasks RCU enabled.429client # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns430lb # [ 0.000871] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000)431worker1 # [ 0.000734] printk: legacy console [tty0] enabled432worker2 # [ 0.000000] Tracing variant of Tasks RCU enabled.433lb # [ 0.000878] pid_max: default: 32768 minimum: 301434client # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns435lb # [ 0.000977] LSM: initializing lsm=capability,landlock,yama,bpf,ima436worker2 # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.437client # [ 0.000032] arm-pv: using stolen time PV438worker1 # [ 0.000925] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000)439lb # [ 0.001120] landlock: Up and running.440lb # [ 0.001123] Yama: becoming mindful.441worker1 # [ 0.000932] pid_max: default: 32768 minimum: 301442worker2 # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1443client # [ 0.000502] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)444lb # [ 0.001656] LSM support for eBPF active445worker1 # [ 0.001006] LSM: initializing lsm=capability,landlock,yama,bpf,ima446client # [ 0.000680] Console: colour dummy device 80x25447worker2 # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.448worker1 # [ 0.001192] landlock: Up and running.449lb # [ 0.001829] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)450worker1 # [ 0.001195] Yama: becoming mindful.451client # [ 0.000688] printk: legacy console [tty0] enabled452worker1 # [ 0.001678] LSM support for eBPF active453worker2 # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.454lb # [ 0.001862] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)455client # [ 0.000899] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000)456lb # [ 0.003776] rcu: Hierarchical SRCU implementation.457worker1 # [ 0.001809] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)458worker2 # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.459client # [ 0.000906] pid_max: default: 32768 minimum: 301460lb # [ 0.003782] rcu: Max phase no-delay instances is 1000.461worker1 # [ 0.001841] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)462worker2 # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0463client # [ 0.000980] LSM: initializing lsm=capability,landlock,yama,bpf,ima464lb # [ 0.005115] fsl-mc MSI: its@8080000 domain created465worker2 # [ 0.000000] GICv3: 256 SPIs implemented466worker1 # [ 0.003724] rcu: Hierarchical SRCU implementation.467client # [ 0.001137] landlock: Up and running.468lb # [ 0.005214] EFI services will not be available.469worker2 # [ 0.000000] GICv3: 0 Extended SPIs implemented470client # [ 0.001140] Yama: becoming mindful.471worker1 # [ 0.003729] rcu: Max phase no-delay instances is 1000.472lb # [ 0.005311] smp: Bringing up secondary CPUs ...473client # [ 0.001633] LSM support for eBPF active474worker2 # [ 0.000000] Root IRQ handler: gic_handle_irq475lb # [ 0.005319] smp: Brought up 1 node, 1 CPU476worker1 # [ 0.005056] fsl-mc MSI: its@8080000 domain created477worker2 # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI478worker1 # [ 0.005151] EFI services will not be available.479lb # [ 0.005323] SMP: Total of 1 processors activated.480client # [ 0.001752] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)481lb # [ 0.005328] CPU: All CPU(s) started at EL1482worker2 # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0483worker1 # [ 0.005256] smp: Bringing up secondary CPUs ...484worker1 # [ 0.005265] smp: Brought up 1 node, 1 CPU485client # [ 0.001785] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)486lb # [ 0.005338] CPU features: detected: Branch Target Identification487worker2 # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000488worker1 # [ 0.005268] SMP: Total of 1 processors activated.489client # [ 0.003576] rcu: Hierarchical SRCU implementation.490worker2 # [ 0.000000] ITS [mem 0x08080000-0x0809ffff]491lb # [ 0.005343] CPU features: detected: ARMv8.4 Translation Table Level492worker1 # [ 0.005274] CPU: All CPU(s) started at EL1493client # [ 0.003581] rcu: Max phase no-delay instances is 1000.494client # [ 0.004843] fsl-mc MSI: its@8080000 domain created495worker1 # [ 0.005287] CPU features: detected: Branch Target Identification496worker2 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449d0000 (indirect, esz 8, psz 64K, shr 1)497lb # [ 0.005346] CPU features: detected: Instruction cache invalidation not required for I/D coherence498client # [ 0.004935] EFI services will not be available.499worker1 # [ 0.005292] CPU features: detected: ARMv8.4 Translation Table Level500client # [ 0.005002] smp: Bringing up secondary CPUs ...501lb # [ 0.005350] CPU features: detected: Data cache clean to the PoU not required for I/D coherence502worker2 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @449e0000 (flat, esz 8, psz 64K, shr 1)503client # [ 0.005010] smp: Brought up 1 node, 1 CPU504worker1 # [ 0.005295] CPU features: detected: Instruction cache invalidation not required for I/D coherence505lb # [ 0.005354] CPU features: detected: Common not Private translations506worker2 # [ 0.000000] GICv3: using LPI property table @0x00000000449f0000507client # [ 0.005013] SMP: Total of 1 processors activated.508lb # [ 0.005357] CPU features: detected: CRC32 instructions509client # [ 0.005018] CPU: All CPU(s) started at EL1510worker1 # [ 0.005299] CPU features: detected: Data cache clean to the PoU not required for I/D coherence511worker2 # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a10000512client # [ 0.005028] CPU features: detected: Branch Target Identification513lb # [ 0.005359] CPU features: detected: Data cache clean to Point of Deep Persistence514worker1 # [ 0.005303] CPU features: detected: Common not Private translations515worker2 # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.516client # [ 0.005032] CPU features: detected: ARMv8.4 Translation Table Level517worker1 # [ 0.005306] CPU features: detected: CRC32 instructions518lb # [ 0.005363] CPU features: detected: Data cache clean to Point of Persistence519worker2 # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns520worker1 # [ 0.005309] CPU features: detected: Data cache clean to Point of Deep Persistence521client # [ 0.005035] CPU features: detected: Instruction cache invalidation not required for I/D coherence522lb # [ 0.005366] CPU features: detected: Data independent timing control (DIT)523worker2 # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt).524lb # [ 0.005369] CPU features: detected: E0PD525worker1 # [ 0.005313] CPU features: detected: Data cache clean to Point of Persistence526client # [ 0.005039] CPU features: detected: Data cache clean to the PoU not required for I/D coherence527lb # [ 0.005371] CPU features: detected: Enhanced Counter Virtualization528worker1 # [ 0.005316] CPU features: detected: Data independent timing control (DIT)529client # [ 0.005042] CPU features: detected: Common not Private translations530worker2 # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns531lb # [ 0.005374] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF)532worker1 # [ 0.005319] CPU features: detected: E0PD533client # [ 0.005046] CPU features: detected: CRC32 instructions534worker2 # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns535lb # [ 0.005378] CPU features: detected: Enhanced Virtualization Traps536worker1 # [ 0.005322] CPU features: detected: Enhanced Counter Virtualization537worker2 # [ 0.000029] arm-pv: using stolen time PV538client # [ 0.005048] CPU features: detected: Data cache clean to Point of Deep Persistence539lb # [ 0.005381] CPU features: detected: Fine Grained Traps540worker1 # [ 0.005325] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF)541client # [ 0.005069] CPU features: detected: Data cache clean to Point of Persistence542lb # [ 0.005384] CPU features: detected: Generic authentication (architected QARMA5 algorithm)543worker2 # [ 0.000419] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)544worker1 # [ 0.005328] CPU features: detected: Enhanced Virtualization Traps545client # [ 0.005072] CPU features: detected: Data independent timing control (DIT)546worker2 # [ 0.000580] Console: colour dummy device 80x25547worker1 # [ 0.005331] CPU features: detected: Fine Grained Traps548lb # [ 0.005389] CPU features: detected: RCpc load-acquire (LDAPR)549worker2 # [ 0.000588] printk: legacy console [tty0] enabled550client # [ 0.005076] CPU features: detected: E0PD551lb # [ 0.005392] CPU features: detected: LSE atomic instructions552worker1 # [ 0.005335] CPU features: detected: Generic authentication (architected QARMA5 algorithm)553client # [ 0.005078] CPU features: detected: Enhanced Counter Virtualization554lb # [ 0.005394] CPU features: detected: Privileged Access Never555worker1 # [ 0.005339] CPU features: detected: RCpc load-acquire (LDAPR)556worker2 # [ 0.000785] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000)557lb # [ 0.005397] CPU features: detected: PMUv3558client # [ 0.005081] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF)559worker1 # [ 0.005342] CPU features: detected: LSE atomic instructions560worker2 # [ 0.000792] pid_max: default: 32768 minimum: 301561lb # [ 0.005400] CPU features: detected: RAS Extension Support562client # [ 0.005085] CPU features: detected: Enhanced Virtualization Traps563worker1 # [ 0.005345] CPU features: detected: Privileged Access Never564worker2 # [ 0.000864] LSM: initializing lsm=capability,landlock,yama,bpf,ima565lb # [ 0.005403] CPU features: detected: RASv1p1 Extension Support566client # [ 0.005088] CPU features: detected: Fine Grained Traps567worker1 # [ 0.005348] CPU features: detected: PMUv3568worker2 # [ 0.001008] landlock: Up and running.569lb # [ 0.005405] CPU features: detected: Random Number Generator570worker2 # [ 0.001011] Yama: becoming mindful.571worker1 # [ 0.005350] CPU features: detected: RAS Extension Support572client # [ 0.005091] CPU features: detected: Generic authentication (architected QARMA5 algorithm)573worker2 # [ 0.001466] LSM support for eBPF active574worker1 # [ 0.005353] CPU features: detected: RASv1p1 Extension Support575client # [ 0.005095] CPU features: detected: RCpc load-acquire (LDAPR)576lb # [ 0.005408] CPU features: detected: Speculation barrier (SB)577worker2 # [ 0.001620] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)578worker1 # [ 0.005356] CPU features: detected: Random Number Generator579client # [ 0.005098] CPU features: detected: LSE atomic instructions580lb # [ 0.005411] CPU features: detected: Stage-2 Force Write-Back581worker1 # [ 0.005358] CPU features: detected: Speculation barrier (SB)582worker2 # [ 0.001653] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)583client # [ 0.005101] CPU features: detected: Privileged Access Never584lb # [ 0.005414] CPU features: detected: TLB range maintenance instructions585worker2 # [ 0.003597] rcu: Hierarchical SRCU implementation.586client # [ 0.005104] CPU features: detected: PMUv3587worker1 # [ 0.005361] CPU features: detected: Stage-2 Force Write-Back588worker2 # [ 0.003602] rcu: Max phase no-delay instances is 1000.589lb # [ 0.005419] CPU features: detected: Speculative Store Bypassing Safe (SSBS)590client # [ 0.005106] CPU features: detected: RAS Extension Support591worker1 # [ 0.005364] CPU features: detected: TLB range maintenance instructions592worker2 # [ 0.004840] fsl-mc MSI: its@8080000 domain created593lb # [ 0.005459] alternatives: applying system-wide alternatives594client # [ 0.005109] CPU features: detected: RASv1p1 Extension Support595worker1 # [ 0.005369] CPU features: detected: Speculative Store Bypassing Safe (SSBS)596worker2 # [ 0.004929] EFI services will not be available.597client # [ 0.005112] CPU features: detected: Random Number Generator598lb # [ 0.008461] CPU features: detected: BBM Level 2 without TLB conflict abort599worker2 # [ 0.005008] smp: Bringing up secondary CPUs ...600worker1 # [ 0.005408] alternatives: applying system-wide alternatives601worker2 # [ 0.005016] smp: Brought up 1 node, 1 CPU602client # [ 0.005114] CPU features: detected: Speculation barrier (SB)603worker1 # [ 0.008387] CPU features: detected: BBM Level 2 without TLB conflict abort604worker2 # [ 0.005019] SMP: Total of 1 processors activated.605client # [ 0.005117] CPU features: detected: Stage-2 Force Write-Back606lb # [ 0.008613] Memory: 1405680K/1572864K available (24448K kernel code, 7094K rwdata, 26372K rodata, 4736K init, 1107K bss, 124636K reserved, 32768K cma-reserved)607worker2 # [ 0.005024] CPU: All CPU(s) started at EL1608lb # [ 0.009349] devtmpfs: initialized609client # [ 0.005120] CPU features: detected: TLB range maintenance instructions610worker2 # [ 0.005034] CPU features: detected: Branch Target Identification611worker1 # [ 0.008590] Memory: 1405600K/1572864K available (24448K kernel code, 7094K rwdata, 26372K rodata, 4736K init, 1107K bss, 124664K reserved, 32768K cma-reserved)612lb # [ 0.011231] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)613client # [ 0.005124] CPU features: detected: Speculative Store Bypassing Safe (SSBS)614worker1 # [ 0.009318] devtmpfs: initialized615worker2 # [ 0.005038] CPU features: detected: ARMv8.4 Translation Table Level616client # [ 0.005163] alternatives: applying system-wide alternatives617lb # [ 0.011254] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).618worker1 # [ 0.011109] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)619worker2 # [ 0.005042] CPU features: detected: Instruction cache invalidation not required for I/D coherence620client # [ 0.008109] CPU features: detected: BBM Level 2 without TLB conflict abort621lb # [ 0.011501] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL622worker1 # [ 0.011131] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).623lb # [ 0.011506] 0 pages in range for non-PLT usage624worker2 # [ 0.005046] CPU features: detected: Data cache clean to the PoU not required for I/D coherence625lb # [ 0.011507] 508272 pages in range for PLT usage626worker1 # [ 0.011344] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL627worker2 # [ 0.005049] CPU features: detected: Common not Private translations628client # [ 0.008322] Memory: 1405612K/1572864K available (24448K kernel code, 7094K rwdata, 26372K rodata, 4736K init, 1107K bss, 124660K reserved, 32768K cma-reserved)629lb # [ 0.011617] pinctrl core: initialized pinctrl subsystem630worker1 # [ 0.011348] 0 pages in range for non-PLT usage631client # [ 0.009132] devtmpfs: initialized632worker2 # [ 0.005053] CPU features: detected: CRC32 instructions633lb # [ 0.012405] DMI not present or invalid.634worker1 # [ 0.011349] 508272 pages in range for PLT usage635client # [ 0.010872] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)636lb # [ 0.015620] NET: Registered PF_NETLINK/PF_ROUTE protocol family637worker2 # [ 0.005056] CPU features: detected: Data cache clean to Point of Deep Persistence638worker1 # [ 0.011453] pinctrl core: initialized pinctrl subsystem639worker1 # [ 0.012215] DMI not present or invalid.640worker2 # [ 0.005059] CPU features: detected: Data cache clean to Point of Persistence641lb # [ 0.017813] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations642client # [ 0.010899] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).643worker1 # [ 0.015279] NET: Registered PF_NETLINK/PF_ROUTE protocol family644client # [ 0.011107] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL645worker2 # [ 0.005062] CPU features: detected: Data independent timing control (DIT)646lb # [ 0.018007] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations647worker1 # [ 0.017539] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations648lb # [ 0.018215] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations649worker2 # [ 0.005065] CPU features: detected: E0PD650worker1 # [ 0.017733] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations651lb # [ 0.018236] audit: initializing netlink subsys (disabled)652worker2 # [ 0.005068] CPU features: detected: Enhanced Counter Virtualization653worker1 # [ 0.017931] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations654lb # [ 0.018820] thermal_sys: Registered thermal governor 'fair_share'655worker2 # [ 0.005071] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF)656worker1 # [ 0.017951] audit: initializing netlink subsys (disabled)657lb # [ 0.018821] thermal_sys: Registered thermal governor 'bang_bang'658worker2 # [ 0.005074] CPU features: detected: Enhanced Virtualization Traps659worker1 # [ 0.018507] thermal_sys: Registered thermal governor 'fair_share'660lb # [ 0.018825] thermal_sys: Registered thermal governor 'step_wise'661worker2 # [ 0.005077] CPU features: detected: Fine Grained Traps662worker1 # [ 0.018509] thermal_sys: Registered thermal governor 'bang_bang'663lb # [ 0.018828] thermal_sys: Registered thermal governor 'user_space'664worker1 # [ 0.018512] thermal_sys: Registered thermal governor 'step_wise'665worker2 # [ 0.005081] CPU features: detected: Generic authentication (architected QARMA5 algorithm)666lb # [ 0.018833] thermal_sys: Registered thermal governor 'power_allocator'667worker1 # [ 0.018515] thermal_sys: Registered thermal governor 'user_space'668worker2 # [ 0.005085] CPU features: detected: RCpc load-acquire (LDAPR)669lb # [ 0.018857] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1670worker2 # [ 0.005088] CPU features: detected: LSE atomic instructions671lb # [ 0.018866] cpuidle: using governor ladder672worker2 # [ 0.005091] CPU features: detected: Privileged Access Never673lb # [ 0.018871] cpuidle: using governor menu674worker2 # [ 0.005094] CPU features: detected: PMUv3675lb # [ 0.019059] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers.676worker2 # [ 0.005097] CPU features: detected: RAS Extension Support677lb # [ 0.019075] ASID allocator initialised with 65536 entries678client # [ 0.011111] 0 pages in range for non-PLT usage679worker2 # [ 0.005099] CPU features: detected: RASv1p1 Extension Support680lb # [ 0.020310] Serial: AMBA PL011 UART driver681client # [ 0.011112] 508272 pages in range for PLT usage682worker2 # [ 0.005102] CPU features: detected: Random Number Generator683client # [ 0.011221] pinctrl core: initialized pinctrl subsystem684lb # [ 0.025589] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1685client # [ 0.012021] DMI not present or invalid.686worker2 # [ 0.005105] CPU features: detected: Speculation barrier (SB)687lb # [ 0.025742] printk: console [ttyAMA0] enabled688client # [ 0.015108] NET: Registered PF_NETLINK/PF_ROUTE protocol family689worker2 # [ 0.005107] CPU features: detected: Stage-2 Force Write-Back690client # [ 0.017324] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations691worker2 # [ 0.005110] CPU features: detected: TLB range maintenance instructions692worker2 # [ 0.005115] CPU features: detected: Speculative Store Bypassing Safe (SSBS)693client # [ 0.017509] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations694worker2 # [ 0.005151] alternatives: applying system-wide alternatives695client # [ 0.017710] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations696worker2 # [ 0.008232] CPU features: detected: BBM Level 2 without TLB conflict abort697client # [ 0.017731] audit: initializing netlink subsys (disabled)698worker1 # [ 0.018520] thermal_sys: Registered thermal governor 'power_allocator'699client # [ 0.018258] thermal_sys: Registered thermal governor 'fair_share'700worker1 # [ 0.018544] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1701client # [ 0.018260] thermal_sys: Registered thermal governor 'bang_bang'702worker2 # [ 0.008450] Memory: 1405764K/1572864K available (24448K kernel code, 7094K rwdata, 26372K rodata, 4736K init, 1107K bss, 124632K reserved, 32768K cma-reserved)703worker1 # [ 0.018553] cpuidle: using governor ladder704worker2 # [ 0.009221] devtmpfs: initialized705client # [ 0.018264] thermal_sys: Registered thermal governor 'step_wise'706worker1 # [ 0.018558] cpuidle: using governor menu707worker2 # [ 0.011021] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)708client # [ 0.018267] thermal_sys: Registered thermal governor 'user_space'709worker1 # [ 0.018746] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers.710lb # [ 0.141760] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages711worker1 # [ 0.018764] ASID allocator initialised with 65536 entries712client # [ 0.018273] thermal_sys: Registered thermal governor 'power_allocator'713worker2 # [ 0.011043] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).714lb # [ 0.141780] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page715worker1 # [ 0.019959] Serial: AMBA PL011 UART driver716client # [ 0.018308] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1717lb # [ 0.141785] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages718worker2 # [ 0.011235] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL719client # [ 0.018316] cpuidle: using governor ladder720worker2 # [ 0.011239] 0 pages in range for non-PLT usage721worker1 # [ 0.025199] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1722lb # [ 0.141789] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page723client # [ 0.018322] cpuidle: using governor menu724worker2 # [ 0.011241] 508272 pages in range for PLT usage725worker1 # [ 0.025362] printk: console [ttyAMA0] enabled726lb # [ 0.141794] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages727worker2 # [ 0.011335] pinctrl core: initialized pinctrl subsystem728client # [ 0.018503] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers.729worker2 # [ 0.012120] DMI not present or invalid.730lb # [ 0.141798] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page731client # [ 0.018517] ASID allocator initialised with 65536 entries732worker1 # [ 0.144865] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages733worker2 # [ 0.015274] NET: Registered PF_NETLINK/PF_ROUTE protocol family734client # [ 0.019748] Serial: AMBA PL011 UART driver735lb # [ 0.141803] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages736worker1 # [ 0.144886] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page737worker2 # [ 0.017506] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations738lb # [ 0.141807] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page739client # [ 0.024976] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1740worker1 # [ 0.144892] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages741client # [ 0.025110] printk: console [ttyAMA0] enabled742worker2 # [ 0.017717] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations743worker1 # [ 0.144896] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page744worker2 # [ 0.017919] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations745worker1 # [ 0.144900] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages746client # [ 0.143765] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages747lb # [ 0.149008] fbcon: Taking over console748worker2 # [ 0.017945] audit: initializing netlink subsys (disabled)749worker1 # [ 0.144904] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page750client # [ 0.143785] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page751lb # [ 0.149023] ACPI: Interpreter disabled.752worker2 # [ 0.018528] thermal_sys: Registered thermal governor 'fair_share'753worker1 # [ 0.144909] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages754client # [ 0.143790] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages755worker2 # [ 0.018530] thermal_sys: Registered thermal governor 'bang_bang'756worker1 # [ 0.144913] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page757client # [ 0.143794] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page758worker2 # [ 0.018533] thermal_sys: Registered thermal governor 'step_wise'759client # [ 0.143798] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages760worker2 # [ 0.018536] thermal_sys: Registered thermal governor 'user_space'761client # [ 0.143803] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page762worker2 # [ 0.018541] thermal_sys: Registered thermal governor 'power_allocator'763client # [ 0.143807] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages764worker1 # [ 0.152198] fbcon: Taking over console765worker2 # [ 0.018566] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1766worker1 # [ 0.152211] ACPI: Interpreter disabled.767client # [ 0.143811] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page768worker2 # [ 0.018574] cpuidle: using governor ladder769worker1 # [ 0.154071] iommu: Default domain type: Translated770worker2 # [ 0.018580] cpuidle: using governor menu771worker1 # [ 0.154080] iommu: DMA domain TLB invalidation policy: strict mode772worker2 # [ 0.018769] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers.773worker1 # [ 0.155839] SCSI subsystem initialized774worker2 # [ 0.018783] ASID allocator initialised with 65536 entries775client # [ 0.151194] fbcon: Taking over console776worker2 # [ 0.019982] Serial: AMBA PL011 UART driver777client # [ 0.151209] ACPI: Interpreter disabled.778client # [ 0.153091] iommu: Default domain type: Translated779worker2 # [ 0.025247] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1780lb # [ 0.158048] iommu: Default domain type: Translated781worker2 # [ 0.025370] printk: console [ttyAMA0] enabled782client # [ 0.153101] iommu: DMA domain TLB invalidation policy: strict mode783lb # [ 0.158059] iommu: DMA domain TLB invalidation policy: strict mode784lb # [ 0.158430] SCSI subsystem initialized785client # [ 0.154794] SCSI subsystem initialized786lb # [ 0.160382] usbcore: registered new interface driver usbfs787lb # [ 0.160414] usbcore: registered new interface driver hub788lb # [ 0.160433] usbcore: registered new device driver usb789lb # [ 0.160727] pps_core: LinuxPPS API ver. 1 registered790lb # [ 0.160733] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>791lb # [ 0.160743] PTP clock support registered792lb # [ 0.160793] EDAC MC: Ver: 3.0.0793worker1 # [ 0.163530] usbcore: registered new interface driver usbfs794lb # [ 0.165369] scmi_core: SCMI protocol bus registered795worker1 # [ 0.163564] usbcore: registered new interface driver hub796worker2 # [ 0.143083] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages797worker1 # [ 0.163596] usbcore: registered new device driver usb798lb # [ 0.166317] FPGA manager framework799worker1 # [ 0.163873] pps_core: LinuxPPS API ver. 1 registered800worker2 # [ 0.143101] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page801lb # [ 0.167247] vgaarb: loaded802worker2 # [ 0.143106] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages803worker1 # [ 0.163879] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>804lb # [ 0.167851] clocksource: Switched to clocksource arch_sys_counter805client # [ 0.162464] usbcore: registered new interface driver usbfs806worker1 # [ 0.163888] PTP clock support registered807worker2 # [ 0.143111] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page808client # [ 0.162497] usbcore: registered new interface driver hub809worker1 # [ 0.163938] EDAC MC: Ver: 3.0.0810worker2 # [ 0.143115] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages811client # [ 0.162523] usbcore: registered new device driver usb812worker1 # [ 0.168513] scmi_core: SCMI protocol bus registered813client # [ 0.162806] pps_core: LinuxPPS API ver. 1 registered814worker2 # [ 0.143120] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page815worker1 # [ 0.169504] FPGA manager framework816worker2 # [ 0.143124] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages817client # [ 0.162812] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>818worker1 # [ 0.170475] vgaarb: loaded819client # [ 0.162822] PTP clock support registered820worker2 # [ 0.143128] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page821client # [ 0.162868] EDAC MC: Ver: 3.0.0822worker1 # [ 0.171111] clocksource: Switched to clocksource arch_sys_counter823client # [ 0.167444] scmi_core: SCMI protocol bus registered824worker1 # [ 0.172485] VFS: Disk quotas dquot_6.6.0825worker2 # [ 0.150907] fbcon: Taking over console826client # [ 0.168387] FPGA manager framework827worker1 # [ 0.172522] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)828worker2 # [ 0.150924] ACPI: Interpreter disabled.829client # [ 0.169360] vgaarb: loaded830lb # [ 0.174193] VFS: Disk quotas dquot_6.6.0831worker1 # [ 0.174081] netfs: FS-Cache loaded832lb # [ 0.174242] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)833worker1 # [ 0.174194] pnp: PnP ACPI: disabled834client # [ 0.169975] clocksource: Switched to clocksource arch_sys_counter835lb # [ 0.175864] netfs: FS-Cache loaded836lb # [ 0.175981] pnp: PnP ACPI: disabled837client # [ 0.171339] VFS: Disk quotas dquot_6.6.0838client # [ 0.171370] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)839client # [ 0.172941] netfs: FS-Cache loaded840client # [ 0.173077] pnp: PnP ACPI: disabled841lb # [ 0.179825] NET: Registered PF_INET protocol family842worker2 # [ 0.159962] iommu: Default domain type: Translated843worker2 # [ 0.159973] iommu: DMA domain TLB invalidation policy: strict mode844worker1 # [ 0.180122] NET: Registered PF_INET protocol family845worker2 # [ 0.160291] SCSI subsystem initialized846lb # [ 0.180711] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)847worker1 # [ 0.180395] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)848worker2 # [ 0.162283] usbcore: registered new interface driver usbfs849worker2 # [ 0.162315] usbcore: registered new interface driver hub850worker2 # [ 0.162333] usbcore: registered new device driver usb851client # [ 0.178969] NET: Registered PF_INET protocol family852worker2 # [ 0.162606] pps_core: LinuxPPS API ver. 1 registered853client # [ 0.179232] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)854worker2 # [ 0.162612] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>855worker2 # [ 0.162634] PTP clock support registered856worker2 # [ 0.162682] EDAC MC: Ver: 3.0.0857worker2 # [ 0.167335] scmi_core: SCMI protocol bus registered858worker2 # [ 0.168324] FPGA manager framework859worker2 # [ 0.169314] vgaarb: loaded860worker2 # [ 0.169930] clocksource: Switched to clocksource arch_sys_counter861worker2 # [ 0.173619] VFS: Disk quotas dquot_6.6.0862worker2 # [ 0.173657] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)863worker2 # [ 0.177459] netfs: FS-Cache loaded864worker2 # [ 0.177573] pnp: PnP ACPI: disabled865worker2 # [ 0.181521] NET: Registered PF_INET protocol family866worker2 # [ 0.181776] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)867lb # [ 0.208748] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)868lb # [ 0.208790] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)869lb # [ 0.208817] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)870lb # [ 0.208891] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)871client # [ 0.208478] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)872lb # [ 0.209036] TCP: Hash tables configured (established 16384 bind 16384)873client # [ 0.208520] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)874worker1 # [ 0.210317] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)875lb # [ 0.209117] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)876client # [ 0.208545] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)877worker1 # [ 0.210360] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)878lb # [ 0.209153] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)879client # [ 0.208614] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)880lb # [ 0.209190] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)881worker1 # [ 0.210384] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)882lb # [ 0.209293] NET: Registered PF_UNIX/PF_LOCAL protocol family883client # [ 0.208756] TCP: Hash tables configured (established 16384 bind 16384)884worker1 # [ 0.210453] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)885lb # [ 0.209314] NET: Registered PF_XDP protocol family886client # [ 0.208839] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)887lb # [ 0.209332] PCI: CLS 0 bytes, default 64888worker1 # [ 0.210597] TCP: Hash tables configured (established 16384 bind 16384)889lb # [ 0.209579] Trying to unpack rootfs image as initramfs...890client # [ 0.208875] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)891worker1 # [ 0.210674] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)892client # [ 0.208911] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)893worker1 # [ 0.210708] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)894client # [ 0.209025] NET: Registered PF_UNIX/PF_LOCAL protocol family895worker1 # [ 0.210745] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)896client # [ 0.209087] NET: Registered PF_XDP protocol family897client # [ 0.209105] PCI: CLS 0 bytes, default 64898worker1 # [ 0.210834] NET: Registered PF_UNIX/PF_LOCAL protocol family899client # [ 0.209359] Trying to unpack rootfs image as initramfs...900worker1 # [ 0.210859] NET: Registered PF_XDP protocol family901worker1 # [ 0.210876] PCI: CLS 0 bytes, default 64902worker1 # [ 0.211118] Trying to unpack rootfs image as initramfs...903lb # [ 0.225343] kvm [1]: HYP mode not available904client # [ 0.223520] kvm [1]: HYP mode not available905worker1 # [ 0.228843] kvm [1]: HYP mode not available906worker2 # [ 0.211045] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)907worker2 # [ 0.211090] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)908worker2 # [ 0.211119] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)909worker2 # [ 0.211191] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)910worker2 # [ 0.211334] TCP: Hash tables configured (established 16384 bind 16384)911worker2 # [ 0.211420] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)912worker2 # [ 0.211455] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)913worker2 # [ 0.211490] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)914worker2 # [ 0.211605] NET: Registered PF_UNIX/PF_LOCAL protocol family915worker2 # [ 0.211665] NET: Registered PF_XDP protocol family916worker2 # [ 0.211682] PCI: CLS 0 bytes, default 64917worker2 # [ 0.211951] Trying to unpack rootfs image as initramfs...918worker2 # [ 0.227798] kvm [1]: HYP mode not available919lb # [ 0.316460] Initialise system trusted keyrings920lb # [ 0.317200] workingset: timestamp_bits=42 max_order=19 bucket_order=0921lb # [ 0.318498] squashfs: version 4.0 (2009/01/31) Phillip Lougher922client # [ 0.314515] Initialise system trusted keyrings923lb # [ 0.319280] 9p: Installing v9fs 9p2000 file system support924worker1 # [ 0.323649] Initialise system trusted keyrings925worker1 # [ 0.324420] workingset: timestamp_bits=42 max_order=19 bucket_order=0926worker1 # [ 0.325730] squashfs: version 4.0 (2009/01/31) Phillip Lougher927client # [ 0.322032] workingset: timestamp_bits=42 max_order=19 bucket_order=0928worker1 # [ 0.326511] 9p: Installing v9fs 9p2000 file system support929client # [ 0.323422] squashfs: version 4.0 (2009/01/31) Phillip Lougher930client # [ 0.324195] 9p: Installing v9fs 9p2000 file system support931worker2 # [ 0.322471] Initialise system trusted keyrings932worker2 # [ 0.323199] workingset: timestamp_bits=42 max_order=19 bucket_order=0933worker2 # [ 0.324513] squashfs: version 4.0 (2009/01/31) Phillip Lougher934lb # [ 0.348278] Key type asymmetric registered935worker2 # [ 0.325302] 9p: Installing v9fs 9p2000 file system support936lb # [ 0.348307] Asymmetric key parser 'x509' registered937lb # [ 0.348379] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242)938client # [ 0.344872] Key type asymmetric registered939lb # [ 0.350486] io scheduler mq-deadline registered940client # [ 0.344890] Asymmetric key parser 'x509' registered941lb # [ 0.350497] io scheduler kyber registered942client # [ 0.344976] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242)943client # [ 0.347134] io scheduler mq-deadline registered944client # [ 0.347144] io scheduler kyber registered945worker1 # [ 0.355258] Key type asymmetric registered946worker1 # [ 0.355282] Asymmetric key parser 'x509' registered947worker1 # [ 0.355364] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242)948worker1 # [ 0.357514] io scheduler mq-deadline registered949worker1 # [ 0.357526] io scheduler kyber registered950lb # [ 0.359984] pl061_gpio 9030000.pl061: PL061 GPIO chip registered951client # [ 0.358115] pl061_gpio 9030000.pl061: PL061 GPIO chip registered952lb # [ 0.361377] ledtrig-cpu: registered to indicate activity on CPUs953lb # [ 0.361779] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:954lb # [ 0.361796] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000955lb # [ 0.361808] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000956client # [ 0.359501] ledtrig-cpu: registered to indicate activity on CPUs957lb # [ 0.361817] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000958client # [ 0.359923] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:959worker1 # [ 0.367256] pl061_gpio 9030000.pl061: PL061 GPIO chip registered960lb # [ 0.361838] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits961client # [ 0.359940] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000962lb # [ 0.361864] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]963client # [ 0.359952] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000964lb # [ 0.361938] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00965client # [ 0.359961] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000966lb # [ 0.361948] pci_bus 0000:00: root bus resource [bus 00-ff]967worker1 # [ 0.368636] ledtrig-cpu: registered to indicate activity on CPUs968client # [ 0.359993] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits969lb # [ 0.361955] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]970worker1 # [ 0.369084] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:971lb # [ 0.361960] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]972client # [ 0.360018] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]973lb # [ 0.361966] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]974worker1 # [ 0.369102] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000975client # [ 0.360094] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00976client # [ 0.360104] pci_bus 0000:00: root bus resource [bus 00-ff]977lb # [ 0.362019] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint978worker1 # [ 0.369115] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000979client # [ 0.360110] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]980lb # [ 0.362470] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint981worker1 # [ 0.369124] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000982client # [ 0.360115] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]983lb # [ 0.362654] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f]984worker2 # [ 0.354032] Key type asymmetric registered985lb # [ 0.362671] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff]986worker1 # [ 0.369147] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits987worker2 # [ 0.354060] Asymmetric key parser 'x509' registered988client # [ 0.360120] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]989lb # [ 0.362701] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]990worker1 # [ 0.369173] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]991worker2 # [ 0.354139] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242)992client # [ 0.360182] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint993lb # [ 0.362717] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref]994worker2 # [ 0.356221] io scheduler mq-deadline registered995worker1 # [ 0.369274] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00996client # [ 0.360616] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint997worker2 # [ 0.356232] io scheduler kyber registered998lb # [ 0.363170] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint999worker1 # [ 0.369284] pci_bus 0000:00: root bus resource [bus 00-ff]1000client # [ 0.360802] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f]1001lb # [ 0.363352] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f]1002worker1 # [ 0.369290] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]1003client # [ 0.360818] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff]1004lb # [ 0.363368] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff]1005worker1 # [ 0.369296] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]1006client # [ 0.360848] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1007lb # [ 0.363397] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1008client # [ 0.360864] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref]1009worker1 # [ 0.369301] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]1010client # [ 0.361332] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1011worker1 # [ 0.369356] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint1012client # [ 0.361514] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f]1013worker1 # [ 0.369798] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1014client # [ 0.361529] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff]1015worker1 # [ 0.369983] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f]1016client # [ 0.361559] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1017worker1 # [ 0.369999] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff]1018worker1 # [ 0.370029] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1019worker1 # [ 0.370044] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref]1020worker1 # [ 0.370503] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1021worker1 # [ 0.370685] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f]1022worker2 # [ 0.366073] pl061_gpio 9030000.pl061: PL061 GPIO chip registered1023worker1 # [ 0.370700] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff]1024worker1 # [ 0.370729] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1025worker1 # [ 0.390154] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1026worker2 # [ 0.367494] ledtrig-cpu: registered to indicate activity on CPUs1027worker1 # [ 0.390340] pci 0000:00:03.0: BAR 0 [io 0x0000-0x007f]1028worker2 # [ 0.367900] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:1029worker1 # [ 0.390356] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff]1030worker1 # [ 0.390386] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1031worker2 # [ 0.367917] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x00000000001032worker1 # [ 0.390847] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1033worker2 # [ 0.367930] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x00100000001034lb # [ 0.387915] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1035worker1 # [ 0.391030] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f]1036lb # [ 0.388122] pci 0000:00:03.0: BAR 0 [io 0x0000-0x007f]1037worker2 # [ 0.367938] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x80000000001038worker1 # [ 0.391045] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff]1039lb # [ 0.388138] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff]1040worker1 # [ 0.391075] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1041worker2 # [ 0.367973] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits1042lb # [ 0.388169] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1043worker1 # [ 0.391093] pci 0000:00:04.0: ROM [mem 0x00000000-0x0003ffff pref]1044worker2 # [ 0.367998] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]1045lb # [ 0.388675] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1046client # [ 0.386061] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1047worker2 # [ 0.368090] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:001048lb # [ 0.388859] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f]1049client # [ 0.386262] pci 0000:00:03.0: BAR 0 [io 0x0000-0x007f]1050worker1 # [ 0.398597] pci 0000:00:05.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint1051worker2 # [ 0.368100] pci_bus 0000:00: root bus resource [bus 00-ff]1052lb # [ 0.388875] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff]1053client # [ 0.386280] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff]1054worker1 # [ 0.398786] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff]1055worker2 # [ 0.368106] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]1056lb # [ 0.388904] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1057client # [ 0.386311] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1058worker1 # [ 0.398815] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1059lb # [ 0.388923] pci 0000:00:04.0: ROM [mem 0x00000000-0x0003ffff pref]1060worker2 # [ 0.368111] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]1061client # [ 0.386807] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1062worker2 # [ 0.368116] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]1063lb # [ 0.389374] pci 0000:00:05.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint1064client # [ 0.386991] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f]1065lb # [ 0.389559] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff]1066client # [ 0.387007] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff]1067worker2 # [ 0.368174] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint1068lb # [ 0.389589] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1069client # [ 0.387036] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1070worker2 # [ 0.368623] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1071client # [ 0.387054] pci 0000:00:04.0: ROM [mem 0x00000000-0x0003ffff pref]1072lb # [ 0.390068] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1073worker2 # [ 0.368814] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f]1074lb # [ 0.390276] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff]1075worker2 # [ 0.368830] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff]1076client # [ 0.387505] pci 0000:00:05.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint1077worker2 # [ 0.368861] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1078client # [ 0.387692] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff]1079lb # [ 0.390310] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1080worker2 # [ 0.368878] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref]1081client # [ 0.387721] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1082lb # [ 0.390740] pci 0000:00:07.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint1083lb # [ 0.390922] pci 0000:00:07.0: BAR 0 [mem 0x00000000-0x00000fff]1084worker2 # [ 0.369341] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1085client # [ 0.388171] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1086worker2 # [ 0.369527] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f]1087client # [ 0.388357] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff]1088lb # [ 0.391168] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1089worker2 # [ 0.369543] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff]1090client # [ 0.388386] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1091lb # [ 0.391354] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff]1092worker2 # [ 0.369573] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1093lb # [ 0.391383] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1094client # [ 0.388770] pci 0000:00:07.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint1095client # [ 0.388950] pci 0000:00:07.0: BAR 0 [mem 0x00000000-0x00000fff]1096lb # [ 0.391834] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1097worker2 # [ 0.389053] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1098lb # [ 0.392034] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff]1099client # [ 0.389222] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1100worker2 # [ 0.389247] pci 0000:00:03.0: BAR 0 [io 0x0000-0x007f]1101lb # [ 0.392064] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1102client # [ 0.389412] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff]1103worker2 # [ 0.389264] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff]1104client # [ 0.389442] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1105lb # [ 0.392518] pci 0000:00:0a.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1106worker2 # [ 0.389294] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1107lb # [ 0.392703] pci 0000:00:0a.0: BAR 1 [mem 0x00000000-0x00000fff]1108client # [ 0.389898] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1109worker2 # [ 0.389763] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1110lb # [ 0.392733] pci 0000:00:0a.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1111client # [ 0.390099] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff]1112worker1 # [ 0.407382] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1113client # [ 0.390129] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1114worker1 # [ 0.407591] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff]1115lb # [ 0.393194] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1116lb # [ 0.393469] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f]1117worker1 # [ 0.407621] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1118client # [ 0.390593] pci 0000:00:0a.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1119lb # [ 0.393486] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff]1120client # [ 0.390788] pci 0000:00:0a.0: BAR 1 [mem 0x00000000-0x00000fff]1121worker1 # [ 0.408076] pci 0000:00:07.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint1122lb # [ 0.393515] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1123client # [ 0.390817] pci 0000:00:0a.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1124worker1 # [ 0.408262] pci 0000:00:07.0: BAR 0 [mem 0x00000000-0x00000fff]1125lb # [ 0.393967] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1126client # [ 0.391292] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1127worker1 # [ 0.408512] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1128lb # [ 0.394156] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f]1129client # [ 0.391567] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f]1130worker1 # [ 0.408698] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff]1131lb # [ 0.394173] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff]1132client # [ 0.391585] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff]1133worker1 # [ 0.408728] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1134lb # [ 0.394202] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1135client # [ 0.391614] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1136worker1 # [ 0.409179] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1137lb # [ 0.394779] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned1138client # [ 0.392084] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1139worker1 # [ 0.409376] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff]1140lb # [ 0.394790] pci 0000:00:04.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned1141client # [ 0.392266] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f]1142worker1 # [ 0.409406] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1143client # [ 0.392283] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff]1144lb # [ 0.394796] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned1145worker1 # [ 0.409863] pci 0000:00:0a.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1146client # [ 0.392313] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1147lb # [ 0.394842] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned1148worker1 # [ 0.410052] pci 0000:00:0a.0: BAR 1 [mem 0x00000000-0x00000fff]1149client # [ 0.392885] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned1150lb # [ 0.394889] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned1151worker1 # [ 0.410081] pci 0000:00:0a.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1152client # [ 0.392896] pci 0000:00:04.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned1153lb # [ 0.394934] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned1154worker1 # [ 0.410542] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1155client # [ 0.392902] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned1156worker1 # [ 0.410833] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f]1157lb # [ 0.394980] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned1158client # [ 0.392947] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned1159worker1 # [ 0.410851] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff]1160lb # [ 0.395026] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned1161worker1 # [ 0.410881] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1162client # [ 0.392995] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned1163lb # [ 0.395071] pci 0000:00:08.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned1164worker1 # [ 0.411357] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1165client # [ 0.393040] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned1166lb # [ 0.395116] pci 0000:00:09.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned1167worker1 # [ 0.411542] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f]1168client # [ 0.393096] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned1169worker1 # [ 0.411557] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff]1170worker2 # [ 0.402029] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f]1171lb # [ 0.395162] pci 0000:00:0a.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned1172client # [ 0.393142] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned1173worker2 # [ 0.402062] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff]1174worker1 # [ 0.411588] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1175lb # [ 0.395208] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned1176client # [ 0.393189] pci 0000:00:08.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned1177worker2 # [ 0.402094] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1178worker1 # [ 0.412186] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned1179lb # [ 0.395277] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned1180worker2 # [ 0.402114] pci 0000:00:04.0: ROM [mem 0x00000000-0x0003ffff pref]1181worker1 # [ 0.412195] pci 0000:00:04.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned1182client # [ 0.393235] pci 0000:00:09.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned1183lb # [ 0.395322] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned1184worker2 # [ 0.402623] pci 0000:00:05.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint1185client # [ 0.393280] pci 0000:00:0a.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned1186worker1 # [ 0.412200] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned1187lb # [ 0.395344] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned1188worker2 # [ 0.402820] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff]1189lb # [ 0.395365] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned1190client # [ 0.393327] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned1191worker1 # [ 0.412247] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned1192worker2 # [ 0.402850] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1193lb # [ 0.395387] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned1194client # [ 0.393412] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned1195worker1 # [ 0.412293] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned1196worker2 # [ 0.403318] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1197lb # [ 0.395415] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned1198client # [ 0.393473] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned1199worker1 # [ 0.412341] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned1200worker2 # [ 0.403511] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff]1201lb # [ 0.395437] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned1202client # [ 0.393495] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned1203worker1 # [ 0.412393] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned1204worker2 # [ 0.403554] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1205lb # [ 0.395461] pci 0000:00:07.0: BAR 0 [mem 0x10086000-0x10086fff]: assigned1206client # [ 0.393517] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned1207worker1 # [ 0.412440] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned1208worker2 # [ 0.404013] pci 0000:00:07.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint1209lb # [ 0.395487] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned1210client # [ 0.393538] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned1211worker2 # [ 0.404199] pci 0000:00:07.0: BAR 0 [mem 0x00000000-0x00000fff]1212lb # [ 0.395511] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned1213worker1 # [ 0.412486] pci 0000:00:08.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned1214client # [ 0.393559] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned1215lb # [ 0.395534] pci 0000:00:0a.0: BAR 1 [mem 0x10089000-0x10089fff]: assigned1216worker2 # [ 0.404459] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1217client # [ 0.393581] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned1218worker1 # [ 0.412532] pci 0000:00:09.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned1219worker2 # [ 0.404655] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff]1220lb # [ 0.395558] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned1221client # [ 0.393602] pci 0000:00:07.0: BAR 0 [mem 0x10086000-0x10086fff]: assigned1222worker1 # [ 0.412578] pci 0000:00:0a.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned1223worker2 # [ 0.404685] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1224lb # [ 0.395581] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned1225client # [ 0.393627] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned1226worker1 # [ 0.412626] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned1227lb # [ 0.395602] pci 0000:00:03.0: BAR 0 [io 0x1000-0x107f]: assigned1228worker2 # [ 0.405147] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1229client # [ 0.393649] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned1230worker1 # [ 0.412701] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned1231lb # [ 0.395625] pci 0000:00:0b.0: BAR 0 [io 0x1080-0x10bf]: assigned1232worker2 # [ 0.405341] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff]1233client # [ 0.393671] pci 0000:00:0a.0: BAR 1 [mem 0x10089000-0x10089fff]: assigned1234lb # [ 0.395647] pci 0000:00:01.0: BAR 0 [io 0x10c0-0x10df]: assigned1235worker1 # [ 0.412749] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned1236worker2 # [ 0.405372] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1237client # [ 0.393693] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned1238lb # [ 0.395670] pci 0000:00:02.0: BAR 0 [io 0x10e0-0x10ff]: assigned1239worker1 # [ 0.412770] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned1240client # [ 0.393716] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned1241worker2 # [ 0.405832] pci 0000:00:0a.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1242lb # [ 0.395692] pci 0000:00:04.0: BAR 0 [io 0x1100-0x111f]: assigned1243worker1 # [ 0.412792] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned1244client # [ 0.393737] pci 0000:00:03.0: BAR 0 [io 0x1000-0x107f]: assigned1245worker2 # [ 0.406040] pci 0000:00:0a.0: BAR 1 [mem 0x00000000-0x00000fff]1246lb # [ 0.395716] pci 0000:00:0c.0: BAR 0 [io 0x1120-0x113f]: assigned1247worker1 # [ 0.412813] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned1248client # [ 0.393758] pci 0000:00:0b.0: BAR 0 [io 0x1080-0x10bf]: assigned1249worker2 # [ 0.406072] pci 0000:00:0a.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1250lb # [ 0.395743] pci_bus 0000:00: resource 4 [io 0x0000-0xffff]1251worker1 # [ 0.412835] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned1252client # [ 0.393779] pci 0000:00:01.0: BAR 0 [io 0x10c0-0x10df]: assigned1253lb # [ 0.395752] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff]1254worker2 # [ 0.406540] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1255worker1 # [ 0.412856] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned1256client # [ 0.393800] pci 0000:00:02.0: BAR 0 [io 0x10e0-0x10ff]: assigned1257worker2 # [ 0.406830] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f]1258lb # [ 0.395757] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff]1259client # [ 0.393821] pci 0000:00:04.0: BAR 0 [io 0x1100-0x111f]: assigned1260worker1 # [ 0.412877] pci 0000:00:07.0: BAR 0 [mem 0x10086000-0x10086fff]: assigned1261worker2 # [ 0.406849] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff]1262client # [ 0.393843] pci 0000:00:0c.0: BAR 0 [io 0x1120-0x113f]: assigned1263worker1 # [ 0.412902] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned1264worker2 # [ 0.406879] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1265client # [ 0.393869] pci_bus 0000:00: resource 4 [io 0x0000-0xffff]1266worker1 # [ 0.412924] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned1267client # [ 0.393878] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff]1268worker2 # [ 0.407361] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1269worker1 # [ 0.412946] pci 0000:00:0a.0: BAR 1 [mem 0x10089000-0x10089fff]: assigned1270worker2 # [ 0.407562] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f]1271client # [ 0.393883] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff]1272worker1 # [ 0.412968] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned1273worker2 # [ 0.407578] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff]1274worker1 # [ 0.412990] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned1275worker2 # [ 0.407609] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1276worker1 # [ 0.413011] pci 0000:00:03.0: BAR 0 [io 0x1000-0x107f]: assigned1277worker2 # [ 0.408185] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned1278worker1 # [ 0.413032] pci 0000:00:0b.0: BAR 0 [io 0x1080-0x10bf]: assigned1279worker2 # [ 0.408196] pci 0000:00:04.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned1280worker1 # [ 0.413052] pci 0000:00:01.0: BAR 0 [io 0x10c0-0x10df]: assigned1281worker1 # [ 0.413073] pci 0000:00:02.0: BAR 0 [io 0x10e0-0x10ff]: assigned1282worker2 # [ 0.408201] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned1283worker1 # [ 0.413094] pci 0000:00:04.0: BAR 0 [io 0x1100-0x111f]: assigned1284lb # [ 0.456652] pci 0000:00:07.0: enabling device (0000 -> 0002)1285worker2 # [ 0.408247] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned1286worker1 # [ 0.413115] pci 0000:00:0c.0: BAR 0 [io 0x1120-0x113f]: assigned1287worker2 # [ 0.408295] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned1288worker1 # [ 0.413141] pci_bus 0000:00: resource 4 [io 0x0000-0xffff]1289worker1 # [ 0.413150] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff]1290worker2 # [ 0.408344] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned1291worker1 # [ 0.413155] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff]1292worker2 # [ 0.408393] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned1293client # [ 0.454776] pci 0000:00:07.0: enabling device (0000 -> 0002)1294worker1 # [ 0.413977] pci 0000:00:07.0: enabling device (0000 -> 0002)1295worker2 # [ 0.408440] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned1296worker2 # [ 0.408487] pci 0000:00:08.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned1297worker2 # [ 0.408535] pci 0000:00:09.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned1298worker2 # [ 0.408582] pci 0000:00:0a.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned1299worker2 # [ 0.408630] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned1300worker2 # [ 0.408726] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned1301worker2 # [ 0.408774] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned1302worker2 # [ 0.408796] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned1303worker2 # [ 0.408818] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned1304worker2 # [ 0.408840] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned1305worker2 # [ 0.408862] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned1306worker2 # [ 0.408883] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned1307worker2 # [ 0.408905] pci 0000:00:07.0: BAR 0 [mem 0x10086000-0x10086fff]: assigned1308worker2 # [ 0.408930] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned1309worker2 # [ 0.408953] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned1310worker2 # [ 0.408975] pci 0000:00:0a.0: BAR 1 [mem 0x10089000-0x10089fff]: assigned1311worker2 # [ 0.408998] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned1312worker2 # [ 0.409020] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned1313worker2 # [ 0.409042] pci 0000:00:03.0: BAR 0 [io 0x1000-0x107f]: assigned1314worker2 # [ 0.409064] pci 0000:00:0b.0: BAR 0 [io 0x1080-0x10bf]: assigned1315worker2 # [ 0.409085] pci 0000:00:01.0: BAR 0 [io 0x10c0-0x10df]: assigned1316worker2 # [ 0.409107] pci 0000:00:02.0: BAR 0 [io 0x10e0-0x10ff]: assigned1317worker2 # [ 0.409128] pci 0000:00:04.0: BAR 0 [io 0x1100-0x111f]: assigned1318worker2 # [ 0.409150] pci 0000:00:0c.0: BAR 0 [io 0x1120-0x113f]: assigned1319worker2 # [ 0.409177] pci_bus 0000:00: resource 4 [io 0x0000-0xffff]1320worker2 # [ 0.409186] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff]1321lb # [ 0.482635] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)1322worker2 # [ 0.409191] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff]1323client # [ 0.480360] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)1324worker1 # [ 0.486269] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)1325worker2 # [ 0.466080] pci 0000:00:07.0: enabling device (0000 -> 0002)1326lb # [ 0.489284] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003)1327lb # [ 0.491396] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003)1328client # [ 0.487146] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003)1329worker1 # [ 0.492543] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003)1330client # [ 0.489174] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003)1331worker1 # [ 0.494873] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003)1332lb # [ 0.501458] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003)1333lb # [ 0.503518] virtio-pci 0000:00:05.0: enabling device (0000 -> 0002)1334client # [ 0.499329] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003)1335client # [ 0.501470] virtio-pci 0000:00:05.0: enabling device (0000 -> 0002)1336worker1 # [ 0.504567] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003)1337worker1 # [ 0.506952] virtio-pci 0000:00:05.0: enabling device (0000 -> 0002)1338worker2 # [ 0.487365] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)1339worker2 # [ 0.489576] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003)1340client # [ 0.512874] virtio-pci 0000:00:06.0: enabling device (0000 -> 0002)1341lb # [ 0.515275] virtio-pci 0000:00:06.0: enabling device (0000 -> 0002)1342worker1 # [ 0.518237] virtio-pci 0000:00:06.0: enabling device (0000 -> 0002)1343lb # [ 0.517108] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002)1344lb # [ 0.518900] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002)1345worker2 # [ 0.500585] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003)1346client # [ 0.515505] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002)1347worker1 # [ 0.524196] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002)1348worker1 # [ 0.526017] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002)1349client # [ 0.517338] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002)1350client # [ 0.519242] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0002)1351client # [ 0.521224] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003)1352lb # [ 0.529073] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0002)1353lb # [ 0.531130] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003)1354worker1 # [ 0.536236] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0002)1355worker2 # [ 0.510496] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003)1356worker1 # [ 0.538608] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003)1357worker2 # [ 0.514507] virtio-pci 0000:00:05.0: enabling device (0000 -> 0002)1358worker2 # [ 0.516578] virtio-pci 0000:00:06.0: enabling device (0000 -> 0002)1359client # [ 0.535529] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003)1360lb # [ 0.542889] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003)1361worker1 # [ 0.550109] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003)1362worker2 # [ 0.526115] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002)1363worker2 # [ 0.528676] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002)1364client # [ 0.551238] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1365lb # [ 0.557115] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1366client # [ 0.553875] msm_serial: driver initialized1367lb # [ 0.559807] msm_serial: driver initialized1368worker1 # [ 0.560361] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1369client # [ 0.554593] SuperH (H)SCI(F) driver initialized1370client # [ 0.554651] STM32 USART driver initialized1371lb # [ 0.560511] SuperH (H)SCI(F) driver initialized1372worker1 # [ 0.563104] msm_serial: driver initialized1373lb # [ 0.560572] STM32 USART driver initialized1374worker2 # [ 0.538484] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0002)1375worker2 # [ 0.541401] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003)1376worker1 # [ 0.563793] SuperH (H)SCI(F) driver initialized1377worker1 # [ 0.563850] STM32 USART driver initialized1378worker2 # [ 0.553186] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003)1379worker2 # [ 0.563188] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1380lb # [ 0.590717] loop: module loaded1381worker2 # [ 0.566245] msm_serial: driver initialized1382worker2 # [ 0.566395] SuperH (H)SCI(F) driver initialized1383lb # [ 0.590914] virtio_blk virtio2: 1/0/0 default/read/poll queues1384client # [ 0.586582] loop: module loaded1385worker2 # [ 0.566448] STM32 USART driver initialized1386client # [ 0.586771] virtio_blk virtio2: 1/0/0 default/read/poll queues1387lb # [ 0.591771] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB)1388client # [ 0.587537] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB)1389worker1 # [ 0.597714] loop: module loaded1390worker1 # [ 0.597920] virtio_blk virtio2: 1/0/0 default/read/poll queues1391worker1 # [ 0.598652] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB)1392lb # [ 0.596464] megasas: 07.734.00.00-rc11393lb # [ 0.597140] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff]1394client # [ 0.594634] megasas: 07.734.00.00-rc11395worker1 # [ 0.603744] megasas: 07.734.00.00-rc11396client # [ 0.595342] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff]1397worker1 # [ 0.604460] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff]1398client # [ 0.598567] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001399client # [ 0.598591] Intel/Sharp Extended Query Table at 0x00311400worker1 # [ 0.607113] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001401client # [ 0.600230] Using buffer write method1402client # [ 0.600288] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff]1403lb # [ 0.617110] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001404worker1 # [ 0.615175] Intel/Sharp Extended Query Table at 0x00311405worker1 # [ 0.616850] Using buffer write method1406lb # [ 0.617145] Intel/Sharp Extended Query Table at 0x00311407lb # [ 0.618813] Using buffer write method1408worker1 # [ 0.616934] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff]1409lb # [ 0.618894] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff]1410client # [ 0.614016] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001411worker1 # [ 0.618772] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001412client # [ 0.614037] Intel/Sharp Extended Query Table at 0x00311413worker1 # [ 0.618795] Intel/Sharp Extended Query Table at 0x00311414client # [ 0.615775] Using buffer write method1415client # [ 0.615803] Concatenating MTD devices:1416client # [ 0.615807] (0): "0.flash"1417worker2 # [ 0.600803] loop: module loaded1418client # [ 0.615811] (1): "0.flash"1419client # [ 0.615821] into device "0.flash"1420worker2 # [ 0.600983] virtio_blk virtio2: 1/0/0 default/read/poll queues1421worker2 # [ 0.601784] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB)1422lb # [ 0.623899] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001423lb # [ 0.623919] Intel/Sharp Extended Query Table at 0x00311424lb # [ 0.625865] Using buffer write method1425lb # [ 0.625892] Concatenating MTD devices:1426lb # [ 0.625896] (0): "0.flash"1427lb # [ 0.625899] (1): "0.flash"1428lb # [ 0.625911] into device "0.flash"1429worker1 # [ 0.632209] Using buffer write method1430worker1 # [ 0.632243] Concatenating MTD devices:1431worker1 # [ 0.632247] (0): "0.flash"1432worker1 # [ 0.632251] (1): "0.flash"1433worker1 # [ 0.632261] into device "0.flash"1434worker2 # [ 0.614572] megasas: 07.734.00.00-rc11435worker2 # [ 0.615278] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff]1436worker2 # [ 0.617833] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001437worker2 # [ 0.617861] Intel/Sharp Extended Query Table at 0x00311438worker2 # [ 0.619500] Using buffer write method1439worker2 # [ 0.619593] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff]1440worker2 # [ 0.621602] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001441worker2 # [ 0.621627] Intel/Sharp Extended Query Table at 0x00311442worker2 # [ 0.639944] Using buffer write method1443worker2 # [ 0.639980] Concatenating MTD devices:1444worker2 # [ 0.639984] (0): "0.flash"1445worker2 # [ 0.639988] (1): "0.flash"1446worker2 # [ 0.639997] into device "0.flash"1447worker1 # [ 0.877131] Freeing initrd memory: 26976K1448client # [ 0.874250] Freeing initrd memory: 26972K1449lb # [ 0.879691] Freeing initrd memory: 26948K1450worker1 # [ 0.883269] tun: Universal TUN/TAP device driver, 1.61451client # [ 0.880253] tun: Universal TUN/TAP device driver, 1.61452lb # [ 0.885802] tun: Universal TUN/TAP device driver, 1.61453worker1 # [ 0.886849] thunder_xcv, ver 1.01454worker1 # [ 0.886890] thunder_bgx, ver 1.01455worker1 # [ 0.886912] nicpf, ver 1.01456client # [ 0.884055] thunder_xcv, ver 1.01457client # [ 0.884099] thunder_bgx, ver 1.01458client # [ 0.884121] nicpf, ver 1.01459client # [ 0.884683] e1000: Intel(R) PRO/1000 Network Driver1460lb # [ 0.889395] thunder_xcv, ver 1.01461client # [ 0.884690] e1000: Copyright (c) 1999-2006 Intel Corporation.1462lb # [ 0.889439] thunder_bgx, ver 1.01463worker1 # [ 0.888667] e1000: Intel(R) PRO/1000 Network Driver1464lb # [ 0.889463] nicpf, ver 1.01465client # [ 0.884714] e1000e: Intel(R) PRO/1000 Network Driver1466worker1 # [ 0.888678] e1000: Copyright (c) 1999-2006 Intel Corporation.1467lb # [ 0.890041] e1000: Intel(R) PRO/1000 Network Driver1468client # [ 0.884721] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.1469worker1 # [ 0.888702] e1000e: Intel(R) PRO/1000 Network Driver1470lb # [ 0.890048] e1000: Copyright (c) 1999-2006 Intel Corporation.1471client # [ 0.884751] igb: Intel(R) Gigabit Ethernet Network Driver1472worker1 # [ 0.888710] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.1473lb # [ 0.890073] e1000e: Intel(R) PRO/1000 Network Driver1474client # [ 0.884756] igb: Copyright (c) 2007-2014 Intel Corporation.1475worker1 # [ 0.888739] igb: Intel(R) Gigabit Ethernet Network Driver1476lb # [ 0.890080] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.1477client # [ 0.884779] igbvf: Intel(R) Gigabit Virtual Function Network Driver1478worker1 # [ 0.888745] igb: Copyright (c) 2007-2014 Intel Corporation.1479lb # [ 0.890110] igb: Intel(R) Gigabit Ethernet Network Driver1480client # [ 0.884785] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.1481worker1 # [ 0.888783] igbvf: Intel(R) Gigabit Virtual Function Network Driver1482lb # [ 0.890116] igb: Copyright (c) 2007-2014 Intel Corporation.1483client # [ 0.884920] sky2: driver version 1.301484worker1 # [ 0.888789] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.1485lb # [ 0.890140] igbvf: Intel(R) Gigabit Virtual Function Network Driver1486worker1 # [ 0.888941] sky2: driver version 1.301487lb # [ 0.890158] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.1488worker1 # [ 0.890497] usbcore: registered new interface driver usb-storage1489client # [ 0.893435] usbcore: registered new interface driver usb-storage1490lb # [ 0.890304] sky2: driver version 1.301491worker1 # [ 0.890608] usbcore: registered new interface driver usbserial_generic1492lb # [ 0.892534] ehci-pci 0000:00:07.0: EHCI Host Controller1493client # [ 0.893486] usbcore: registered new interface driver usbserial_generic1494worker1 # [ 0.890622] usbserial: USB Serial support registered for generic1495client # [ 0.893500] usbserial: USB Serial support registered for generic1496lb # [ 0.892559] ehci-pci 0000:00:07.0: new USB bus registered, assigned bus number 11497worker1 # [ 0.891844] ehci-pci 0000:00:07.0: EHCI Host Controller1498lb # [ 0.892827] ehci-pci 0000:00:07.0: irq 16, io mem 0x100860001499worker1 # [ 0.891869] ehci-pci 0000:00:07.0: new USB bus registered, assigned bus number 11500client # [ 0.896855] ehci-pci 0000:00:07.0: EHCI Host Controller1501worker1 # [ 0.892132] ehci-pci 0000:00:07.0: irq 16, io mem 0x100860001502client # [ 0.896885] ehci-pci 0000:00:07.0: new USB bus registered, assigned bus number 11503lb # [ 0.902037] usbcore: registered new interface driver usb-storage1504client # [ 0.897231] ehci-pci 0000:00:07.0: irq 16, io mem 0x100860001505lb # [ 0.902128] usbcore: registered new interface driver usbserial_generic1506worker1 # [ 0.902559] hv_vmbus: registering driver hyperv_keyboard1507lb # [ 0.902142] usbserial: USB Serial support registered for generic1508worker1 # [ 0.903175] ehci-pci 0000:00:07.0: USB 2.0 started, EHCI 1.001509client # [ 0.899795] hv_vmbus: registering driver hyperv_keyboard1510worker2 # [ 0.881461] Freeing initrd memory: 26944K1511lb # [ 0.902797] hv_vmbus: registering driver hyperv_keyboard1512worker1 # [ 0.903462] hub 1-0:1.0: USB hub found1513worker1 # [ 0.903481] hub 1-0:1.0: 6 ports detected1514client # [ 0.901325] rtc-pl031 9010000.pl031: registered as rtc01515lb # [ 0.906376] ehci-pci 0000:00:07.0: USB 2.0 started, EHCI 1.001516lb # [ 0.906677] hub 1-0:1.0: USB hub found1517worker1 # [ 0.906429] rtc-pl031 9010000.pl031: registered as rtc01518client # [ 0.901354] rtc-pl031 9010000.pl031: setting system clock to 2026-09-21T14:43:36 UTC (1790001816)1519lb # [ 0.906698] hub 1-0:1.0: 6 ports detected1520client # [ 0.901653] i2c_dev: i2c /dev entries driver1521worker1 # [ 0.906455] rtc-pl031 9010000.pl031: setting system clock to 2026-09-21T14:43:36 UTC (1790001816)1522worker1 # [ 0.906765] i2c_dev: i2c /dev entries driver1523lb # [ 0.909058] rtc-pl031 9010000.pl031: registered as rtc01524worker2 # [ 0.887519] tun: Universal TUN/TAP device driver, 1.61525lb # [ 0.909086] rtc-pl031 9010000.pl031: setting system clock to 2026-09-21T14:43:36 UTC (1790001816)1526lb # [ 0.909407] i2c_dev: i2c /dev entries driver1527client # [ 0.906966] sdhci: Secure Digital Host Controller Interface driver1528client # [ 0.906977] sdhci: Copyright(c) Pierre Ossman1529lb # [ 0.914763] sdhci: Secure Digital Host Controller Interface driver1530worker1 # [ 0.911995] sdhci: Secure Digital Host Controller Interface driver1531client # [ 0.907239] Synopsys Designware Multimedia Card Interface Driver1532lb # [ 0.914774] sdhci: Copyright(c) Pierre Ossman1533worker1 # [ 0.912005] sdhci: Copyright(c) Pierre Ossman1534client # [ 0.907618] sdhci-pltfm: SDHCI platform and OF driver helper1535lb # [ 0.915038] Synopsys Designware Multimedia Card Interface Driver1536worker1 # [ 0.912259] Synopsys Designware Multimedia Card Interface Driver1537worker2 # [ 0.891234] thunder_xcv, ver 1.01538client # [ 0.909116] hid: raw HID events driver (C) Jiri Kosina1539worker2 # [ 0.891278] thunder_bgx, ver 1.01540lb # [ 0.915410] sdhci-pltfm: SDHCI platform and OF driver helper1541worker1 # [ 0.912622] sdhci-pltfm: SDHCI platform and OF driver helper1542worker2 # [ 0.891299] nicpf, ver 1.01543client # [ 0.909352] usbcore: registered new interface driver usbhid1544worker1 # [ 0.914129] hid: raw HID events driver (C) Jiri Kosina1545worker2 # [ 0.891885] e1000: Intel(R) PRO/1000 Network Driver1546client # [ 0.909359] usbhid: USB HID core driver1547worker1 # [ 0.914362] usbcore: registered new interface driver usbhid1548worker2 # [ 0.891893] e1000: Copyright (c) 1999-2006 Intel Corporation.1549worker1 # [ 0.914369] usbhid: USB HID core driver1550lb # [ 0.919624] hid: raw HID events driver (C) Jiri Kosina1551worker2 # [ 0.891917] e1000e: Intel(R) PRO/1000 Network Driver1552client # [ 0.914381] ehci-pci 0000:00:07.0: USB 2.0 started, EHCI 1.001553client # [ 0.914694] hub 1-0:1.0: USB hub found1554worker2 # [ 0.891925] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.1555client # [ 0.914714] hub 1-0:1.0: 6 ports detected1556worker2 # [ 0.891956] igb: Intel(R) Gigabit Ethernet Network Driver1557lb # [ 0.920507] usbcore: registered new interface driver usbhid1558lb # [ 0.920517] usbhid: USB HID core driver1559worker2 # [ 0.891962] igb: Copyright (c) 2007-2014 Intel Corporation.1560worker2 # [ 0.891985] igbvf: Intel(R) Gigabit Virtual Function Network Driver1561worker1 # [ 0.919853] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available1562client # [ 0.917139] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available1563worker2 # [ 0.891991] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.1564worker1 # [ 0.921378] drop_monitor: Initializing network drop monitor service1565worker2 # [ 0.892127] sky2: driver version 1.301566lb # [ 0.922775] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available1567worker1 # [ 0.921514] NET: Registered PF_INET6 protocol family1568worker2 # [ 0.893675] usbcore: registered new interface driver usb-storage1569client # [ 0.919714] drop_monitor: Initializing network drop monitor service1570worker1 # [ 0.924450] Segment Routing with IPv61571worker2 # [ 0.893724] usbcore: registered new interface driver usbserial_generic1572worker1 # [ 0.924472] In-situ OAM (IOAM) with IPv61573client # [ 0.919892] NET: Registered PF_INET6 protocol family1574lb # [ 0.925394] drop_monitor: Initializing network drop monitor service1575worker2 # [ 0.893738] usbserial: USB Serial support registered for generic1576worker1 # [ 0.924500] NET: Registered PF_PACKET protocol family1577lb # [ 0.925555] NET: Registered PF_INET6 protocol family1578worker2 # [ 0.894712] ehci-pci 0000:00:07.0: EHCI Host Controller1579client # [ 0.921784] Segment Routing with IPv61580client # [ 0.921802] In-situ OAM (IOAM) with IPv61581lb # [ 0.927482] Segment Routing with IPv61582worker2 # [ 0.894738] ehci-pci 0000:00:07.0: new USB bus registered, assigned bus number 11583lb # [ 0.927501] In-situ OAM (IOAM) with IPv61584client # [ 0.921830] NET: Registered PF_PACKET protocol family1585worker1 # [ 0.926086] 9pnet: Installing 9P2000 support1586worker2 # [ 0.894982] ehci-pci 0000:00:07.0: irq 16, io mem 0x100860001587worker1 # [ 0.926131] Key type dns_resolver registered1588lb # [ 0.927529] NET: Registered PF_PACKET protocol family1589worker2 # [ 0.906403] ehci-pci 0000:00:07.0: USB 2.0 started, EHCI 1.001590worker2 # [ 0.906697] hub 1-0:1.0: USB hub found1591worker2 # [ 0.906712] hub 1-0:1.0: 6 ports detected1592client # [ 0.923466] 9pnet: Installing 9P2000 support1593client # [ 0.923514] Key type dns_resolver registered1594lb # [ 0.929295] 9pnet: Installing 9P2000 support1595lb # [ 0.929342] Key type dns_resolver registered1596worker2 # [ 0.909196] hv_vmbus: registering driver hyperv_keyboard1597worker2 # [ 0.910854] rtc-pl031 9010000.pl031: registered as rtc01598worker2 # [ 0.910878] rtc-pl031 9010000.pl031: setting system clock to 2026-09-21T14:43:36 UTC (1790001816)1599worker2 # [ 0.911177] i2c_dev: i2c /dev entries driver1600worker1 # [ 0.932864] registered taskstats version 11601worker1 # [ 0.933013] Loading compiled-in X.509 certificates1602client # [ 0.930096] registered taskstats version 11603client # [ 0.930246] Loading compiled-in X.509 certificates1604lb # [ 0.935968] registered taskstats version 11605lb # [ 0.936117] Loading compiled-in X.509 certificates1606worker2 # [ 0.916502] sdhci: Secure Digital Host Controller Interface driver1607worker2 # [ 0.916512] sdhci: Copyright(c) Pierre Ossman1608worker2 # [ 0.916778] Synopsys Designware Multimedia Card Interface Driver1609worker2 # [ 0.917145] sdhci-pltfm: SDHCI platform and OF driver helper1610worker1 # [ 0.941457] Demotion targets for Node 0: null1611worker1 # [ 0.941556] Key type .fscrypt registered1612worker1 # [ 0.941562] Key type fscrypt-provisioning registered1613client # [ 0.938931] Demotion targets for Node 0: null1614worker1 # [ 0.941654] ima: No TPM chip found, activating TPM-bypass!1615worker2 # [ 0.921415] hid: raw HID events driver (C) Jiri Kosina1616client # [ 0.939036] Key type .fscrypt registered1617worker1 # [ 0.941673] ima: Allocated hash algorithm: sha11618lb # [ 0.944698] Demotion targets for Node 0: null1619worker2 # [ 0.921681] usbcore: registered new interface driver usbhid1620client # [ 0.939042] Key type fscrypt-provisioning registered1621lb # [ 0.944797] Key type .fscrypt registered1622worker1 # [ 0.941694] ima: No architecture policies found1623worker2 # [ 0.921687] usbhid: USB HID core driver1624client # [ 0.939139] ima: No TPM chip found, activating TPM-bypass!1625lb # [ 0.944804] Key type fscrypt-provisioning registered1626client # [ 0.939158] ima: Allocated hash algorithm: sha11627lb # [ 0.944894] ima: No TPM chip found, activating TPM-bypass!1628worker1 # [ 0.945589] input: gpio-keys as /devices/platform/gpio-keys/input/input01629client # [ 0.939180] ima: No architecture policies found1630lb # [ 0.944913] ima: Allocated hash algorithm: sha11631lb # [ 0.944935] ima: No architecture policies found1632client # [ 0.942997] input: gpio-keys as /devices/platform/gpio-keys/input/input01633worker2 # [ 0.924538] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available1634lb # [ 0.948750] input: gpio-keys as /devices/platform/gpio-keys/input/input01635worker2 # [ 0.927153] drop_monitor: Initializing network drop monitor service1636worker2 # [ 0.927311] NET: Registered PF_INET6 protocol family1637worker2 # [ 0.929232] Segment Routing with IPv61638worker2 # [ 0.929251] In-situ OAM (IOAM) with IPv61639worker2 # [ 0.929280] NET: Registered PF_PACKET protocol family1640worker2 # [ 0.931124] 9pnet: Installing 9P2000 support1641worker2 # [ 0.931169] Key type dns_resolver registered1642worker2 # [ 0.937992] registered taskstats version 11643worker2 # [ 0.938150] Loading compiled-in X.509 certificates1644worker1 # [ 0.963811] clk: Disabling unused clocks1645client # [ 0.960834] clk: Disabling unused clocks1646worker1 # [ 0.963834] PM: genpd: Disabling unused power domains1647client # [ 0.960861] PM: genpd: Disabling unused power domains1648lb # [ 0.966954] clk: Disabling unused clocks1649lb # [ 0.966979] PM: genpd: Disabling unused power domains1650worker1 # [ 0.968021] Freeing unused kernel memory: 4736K1651worker1 # [ 0.968224] Run /init as init process1652worker2 # [ 0.946790] Demotion targets for Node 0: null1653worker2 # [ 0.946897] Key type .fscrypt registered1654client # [ 0.965076] Freeing unused kernel memory: 4736K1655worker2 # [ 0.946903] Key type fscrypt-provisioning registered1656client # [ 0.965276] Run /init as init process1657worker2 # [ 0.947002] ima: No TPM chip found, activating TPM-bypass!1658lb # [ 0.971131] Freeing unused kernel memory: 4736K1659lb # [ 0.971343] Run /init as init process1660worker2 # [ 0.947021] ima: Allocated hash algorithm: sha11661worker2 # [ 0.947044] ima: No architecture policies found1662worker2 # [ 0.951188] input: gpio-keys as /devices/platform/gpio-keys/input/input01663client # [ 0.980174] systemd[1]: Successfully made /usr/ read-only.1664lb # [ 0.986077] systemd[1]: Successfully made /usr/ read-only.1665worker1 # [ 0.985386] systemd[1]: Successfully made /usr/ read-only.1666worker2 # [ 0.969298] clk: Disabling unused clocks1667worker2 # [ 0.969321] PM: genpd: Disabling unused power domains1668worker2 # [ 0.973441] Freeing unused kernel memory: 4736K1669worker2 # [ 0.973639] Run /init as init process1670worker2 # [ 0.988164] systemd[1]: Successfully made /usr/ read-only.1671lb # [ 1.151918] usb 1-1: new high-speed USB device number 2 using ehci-pci1672worker1 # [ 1.151205] usb 1-1: new high-speed USB device number 2 using ehci-pci1673client # [ 1.162044] usb 1-1: new high-speed USB device number 2 using ehci-pci1674worker2 # [ 1.154012] usb 1-1: new high-speed USB device number 2 using ehci-pci1675worker1 # [ 1.303664] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input11676lb # [ 1.304530] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input11677client # [ 1.314233] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input11678lb # [ 1.320889] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1679worker1 # [ 1.320258] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1680lb # [ 1.333102] systemd[1]: Detected virtualization qemu.1681worker1 # [ 1.332389] systemd[1]: Detected virtualization qemu.1682worker2 # [ 1.308260] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input11683lb # [ 1.335145] systemd[1]: Detected architecture arm64.1684worker1 # [ 1.334466] systemd[1]: Detected architecture arm64.1685client # [ 1.320348] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1686lb # [ 1.337131] systemd[1]: Running in initrd.1687worker1 # [ 1.336453] systemd[1]: Running in initrd.1688client # [ 1.332805] systemd[1]: Detected virtualization qemu.1689client # [ 1.334939] systemd[1]: Detected architecture arm64.1690lb # [ 1.339761] systemd[1]: Initializing machine ID from random generator.1691worker1 # [ 1.339080] systemd[1]: Initializing machine ID from random generator.1692client # [ 1.336854] systemd[1]: Running in initrd.1693lb # [ 1.342912] systemd[1]: Hostname set to <lb>.1694worker1 # [ 1.341945] systemd[1]: Hostname set to <worker1>.1695client # [ 1.339521] systemd[1]: Initializing machine ID from random generator.1696client # [ 1.342487] systemd[1]: Hostname set to <client>.1697worker2 # [ 1.323001] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1698worker2 # [ 1.335174] systemd[1]: Detected virtualization qemu.1699worker2 # [ 1.337214] systemd[1]: Detected architecture arm64.1700worker2 # [ 1.339256] systemd[1]: Running in initrd.1701worker2 # [ 1.341805] systemd[1]: Initializing machine ID from random generator.1702worker2 # [ 1.344788] systemd[1]: Hostname set to <worker2>.1703lb # [ 1.388160] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:07.0-1/input01704worker1 # [ 1.391290] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:07.0-1/input01705client # [ 1.398295] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:07.0-1/input01706worker2 # [ 1.394212] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:07.0-1/input01707lb # [ 1.511924] usb 1-2: new high-speed USB device number 3 using ehci-pci1708worker1 # [ 1.511200] usb 1-2: new high-speed USB device number 3 using ehci-pci1709client # [ 1.518067] usb 1-2: new high-speed USB device number 3 using ehci-pci1710worker2 # [ 1.518022] usb 1-2: new high-speed USB device number 3 using ehci-pci1711worker1 # [ 1.590148] systemd[1]: bpf-restrict-fs: LSM BPF program attached1712worker1 # [ 1.647191] systemd[1]: Queued start job for default target Initrd Default Target.1713lb # [ 1.655147] systemd[1]: bpf-restrict-fs: LSM BPF program attached1714worker1 # [ 1.657443] systemd[1]: Created slice Slice /system/modprobe.1715worker1 # [ 1.658527] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1716client # [ 1.653678] systemd[1]: bpf-restrict-fs: LSM BPF program attached1717worker1 # [ 1.659858] systemd[1]: Expecting device /dev/disk/by-label/nixos...1718worker1 # [ 1.661164] systemd[1]: Reached target Path Units.1719worker1 # [ 1.661901] systemd[1]: Reached target Slice Units.1720worker1 # [ 1.662826] systemd[1]: Reached target Swaps.1721worker1 # [ 1.663886] systemd[1]: Reached target Timer Units.1722worker1 # [ 1.664065] systemd[1]: Listening on D-Bus System Message Bus Socket.1723worker1 # [ 1.664233] systemd[1]: Listening on Journal Socket (/dev/log).1724worker1 # [ 1.664366] systemd[1]: Listening on Journal Sockets.1725worker1 # [ 1.664493] systemd[1]: Listening on udev Control Socket.1726worker1 # [ 1.664595] systemd[1]: Listening on udev Kernel Socket.1727worker1 # [ 1.664614] systemd[1]: Reached target Socket Units.1728worker2 # [ 1.648567] systemd[1]: bpf-restrict-fs: LSM BPF program attached1729lb # [ 1.668329] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input21730worker1 # [ 1.672586] systemd[1]: Starting Create List of Static Device Nodes...1731worker1 # [ 1.674007] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs1732worker1 # [ 1.676407] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input21733lb # [ 1.676131] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:07.0-2/input01734worker1 # [ 1.676559] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:07.0-2/input01735worker1 # [ 1.684839] systemd[1]: Mounting Kernel Configuration File System...1736client # [ 1.678717] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input21737client # [ 1.684476] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:07.0-2/input01738worker1 # [ 1.696580] systemd[1]: Starting Journal Service...1739worker2 # [ 1.678776] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input21740worker2 # [ 1.683828] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:07.0-2/input01741worker1 # [ 1.718432] systemd[1]: Starting Load Kernel Modules...1742worker1 # [ 1.718540] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1743worker1 # [ 1.738616] systemd-journald[72]: Collecting audit messages is disabled.1744worker1 # [ 1.747309] systemd[1]: Starting Coldplug All udev Devices...1745worker1 # [ 1.759374] systemd[1]: Finished Create List of Static Device Nodes.1746lb # [ 1.765151] systemd[1]: Queued start job for default target Initrd Default Target.1747lb # [ 1.774224] systemd[1]: Created slice Slice /system/modprobe.1748client # [ 1.769853] systemd[1]: Queued start job for default target Initrd Default Target.1749lb # [ 1.775466] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1750worker1 # [ 1.771534] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1751lb # [ 1.776830] systemd[1]: Expecting device /dev/disk/by-label/nixos...1752worker1 # [ 1.771812] systemd[1]: Mounted Kernel Configuration File System.1753lb # [ 1.777877] systemd[1]: Reached target Path Units.1754lb # [ 1.778696] systemd[1]: Reached target Slice Units.1755lb # [ 1.779493] systemd[1]: Reached target Swaps.1756lb # [ 1.780295] systemd[1]: Reached target Timer Units.1757lb # [ 1.781274] systemd[1]: Listening on D-Bus System Message Bus Socket.1758lb # [ 1.782473] systemd[1]: Listening on Journal Socket (/dev/log).1759lb # [ 1.783544] systemd[1]: Listening on Journal Sockets.1760worker2 # [ 1.762810] systemd[1]: Queued start job for default target Initrd Default Target.1761lb # [ 1.784535] systemd[1]: Listening on udev Control Socket.1762lb # [ 1.784659] systemd[1]: Listening on udev Kernel Socket.1763client # [ 1.781700] systemd[1]: Created slice Slice /system/modprobe.1764lb # [ 1.784682] systemd[1]: Reached target Socket Units.1765client # [ 1.783004] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1766lb # [ 1.788908] systemd[1]: Starting Create List of Static Device Nodes...1767client # [ 1.784370] systemd[1]: Expecting device /dev/disk/by-label/nixos...1768client # [ 1.785585] systemd[1]: Reached target Path Units.1769lb # [ 1.790017] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs1770client # [ 1.786487] systemd[1]: Reached target Slice Units.1771client # [ 1.787340] systemd[1]: Reached target Swaps.1772client # [ 1.788108] systemd[1]: Reached target Timer Units.1773client # [ 1.789168] systemd[1]: Listening on D-Bus System Message Bus Socket.1774worker2 # [ 1.772399] systemd[1]: Created slice Slice /system/modprobe.1775worker2 # [ 1.773592] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1776client # [ 1.790494] systemd[1]: Listening on Journal Socket (/dev/log).1777lb # [ 1.797435] systemd[1]: Mounting Kernel Configuration File System...1778worker2 # [ 1.775009] systemd[1]: Expecting device /dev/disk/by-label/nixos...1779client # [ 1.790639] systemd[1]: Listening on Journal Sockets.1780worker2 # [ 1.776081] systemd[1]: Reached target Path Units.1781client # [ 1.790776] systemd[1]: Listening on udev Control Socket.1782worker2 # [ 1.776904] systemd[1]: Reached target Slice Units.1783client # [ 1.790903] systemd[1]: Listening on udev Kernel Socket.1784worker2 # [ 1.777738] systemd[1]: Reached target Swaps.1785client # [ 1.790928] systemd[1]: Reached target Socket Units.1786worker2 # [ 1.778564] systemd[1]: Reached target Timer Units.1787client # [ 1.796960] systemd[1]: Starting Create List of Static Device Nodes...1788worker2 # [ 1.779613] systemd[1]: Listening on D-Bus System Message Bus Socket.1789worker2 # [ 1.780839] systemd[1]: Listening on Journal Socket (/dev/log).1790client # [ 1.798190] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs1791worker1 # [ 1.802408] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1792worker2 # [ 1.782028] systemd[1]: Listening on Journal Sockets.1793worker2 # [ 1.783008] systemd[1]: Listening on udev Control Socket.1794worker2 # [ 1.784090] systemd[1]: Listening on udev Kernel Socket.1795worker2 # [ 1.784970] systemd[1]: Reached target Socket Units.1796lb # [ 1.808131] systemd[1]: Starting Journal Service...1797worker2 # [ 1.787466] systemd[1]: Starting Create List of Static Device Nodes...1798worker2 # [ 1.788666] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs1799client # [ 1.809481] systemd[1]: Mounting Kernel Configuration File System...1800worker1 # [ 1.811169] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1801worker1 # [ 1.817187] [drm] pci: virtio-gpu-pci detected at 0000:00:05.01802worker1 # [ 1.817417] [drm] features: -virgl +edid -resource_blob -host_visible1803worker1 # [ 1.817427] [drm] features: -context_init1804worker1 # [ 1.818096] [drm] number of scanouts: 11805worker2 # [ 1.798094] systemd[1]: Mounting Kernel Configuration File System...1806worker1 # [ 1.818114] [drm] number of cap sets: 01807worker2 # [ 1.806260] systemd[1]: Starting Journal Service...1808lb # [ 1.836073] systemd[1]: Starting Load Kernel Modules...1809worker1 # [ 1.835792] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1810lb # [ 1.836198] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1811worker1 # [ 1.838423] systemd[1]: Starting Create Static Device Nodes in /dev...1812client # [ 1.838763] systemd[1]: Starting Journal Service...1813client # [ 1.841450] systemd[1]: Starting Load Kernel Modules...1814lb # [ 1.848059] systemd[1]: Starting Coldplug All udev Devices...1815client # [ 1.842573] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1816worker2 # [ 1.830918] systemd[1]: Starting Load Kernel Modules...1817worker2 # [ 1.831919] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1818client # [ 1.852217] systemd[1]: Starting Coldplug All udev Devices...1819worker1 # [ 1.851660] virtio-pci 0000:00:05.0: [drm] Registered 1 planes with drm panic1820worker1 # [ 1.851677] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:05.0 on minor 01821lb # [ 1.855909] systemd-journald[72]: Collecting audit messages is disabled.1822lb # [ 1.872008] systemd[1]: Finished Create List of Static Device Nodes.1823lb # [ 1.872705] systemd[1]: Mounted Kernel Configuration File System.1824client # [ 1.867641] systemd[1]: Finished Create List of Static Device Nodes.1825client # [ 1.868367] systemd[1]: Mounted Kernel Configuration File System.1826worker2 # [ 1.856922] systemd-journald[72]: Collecting audit messages is disabled.1827worker2 # [ 1.862132] systemd[1]: Starting Coldplug All udev Devices...1828worker2 # [ 1.864567] systemd[1]: Finished Create List of Static Device Nodes.1829lb # [ 1.888096] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1830worker1 # [ 1.891783] systemd[1]: Finished Create Static Device Nodes in /dev.1831worker1 # [ 1.892002] systemd[1]: Reached target Preparation for Local File Systems.1832worker1 # [ 1.892026] systemd[1]: Reached target Local File Systems.1833worker1 # [ 1.895902] systemd[1]: Starting Rule-based Manager for Device Events and Files...1834worker2 # [ 1.878154] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1835worker2 # [ 1.879640] systemd[1]: Mounted Kernel Configuration File System.1836worker1 # [ 1.903332] Console: switching to colour frame buffer device 160x501837worker1 # [ 1.904010] virtio-pci 0000:00:05.0: [drm] fb0: virtio_gpudrmfb frame buffer device1838client # [ 1.894142] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1839client # [ 1.898024] systemd-journald[72]: Collecting audit messages is disabled.1840lb # [ 1.924514] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1841lb # [ 1.927022] systemd[1]: Starting Create Static Device Nodes in /dev...1842worker1 # [ 1.927385] systemd[1]: Finished Load Kernel Modules.1843worker1 # [ 1.930617] systemd[1]: Starting Apply Kernel Variables...1844client # [ 1.925222] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1845client # [ 1.926121] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1846client # [ 1.930047] systemd[1]: Starting Create Static Device Nodes in /dev...1847worker1 # [ 1.920476] systemd-modules-load[74]: Inserted module 'dm_mod'1848worker1 # [ 1.921732] systemd-modules-load[74]: Module 'virtio_balloon' is built in1849worker1 # [ 1.922815] systemd-modules-load[74]: Module 'virtio_console' is built in1850worker1 # [ 1.923837] systemd-modules-load[74]: Inserted module 'virtio_gpu'1851client # [ 1.938079] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1852worker1 # [ 1.944079] systemd[1]: Started Journal Service.1853worker2 # [ 1.918650] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1854worker2 # [ 1.927306] systemd[1]: Starting Create Static Device Nodes in /dev...1855worker1 # [ 1.937644] systemd-modules-load[74]: Module 'virtio_rng' is built in1856client # [ 1.947724] [drm] pci: virtio-gpu-pci detected at 0000:00:05.01857client # [ 1.947958] [drm] features: -virgl +edid -resource_blob -host_visible1858client # [ 1.947969] [drm] features: -context_init1859client # [ 1.948694] [drm] number of scanouts: 11860client # [ 1.948711] [drm] number of cap sets: 01861lb # [ 1.966641] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1862worker1 # [ 1.955237] systemd[1]: Starting Create System Files and Directories...1863worker2 # [ 1.948640] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1864lb # [ 1.975899] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1865client # [ 1.970626] virtio-pci 0000:00:05.0: [drm] Registered 1 planes with drm panic1866client # [ 1.970645] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:05.0 on minor 01867lb # [ 1.983684] [drm] pci: virtio-gpu-pci detected at 0000:00:05.01868lb # [ 1.985301] systemd[1]: Finished Create Static Device Nodes in /dev.1869lb # [ 1.985670] systemd[1]: Reached target Preparation for Local File Systems.1870worker2 # [ 1.958000] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1871lb # [ 1.985701] systemd[1]: Reached target Local File Systems.1872worker2 # [ 1.964169] [drm] pci: virtio-gpu-pci detected at 0000:00:05.01873worker2 # [ 1.964454] [drm] features: -virgl +edid -resource_blob -host_visible1874worker2 # [ 1.964465] [drm] features: -context_init1875worker2 # [ 1.965182] [drm] number of scanouts: 11876worker2 # [ 1.965201] [drm] number of cap sets: 01877worker1 # [ 1.977109] systemd[1]: Finished Apply Kernel Variables.1878lb # [ 1.996367] systemd[1]: Starting Rule-based Manager for Device Events and Files...1879lb # [ 2.004056] [drm] features: -virgl +edid -resource_blob -host_visible1880client # [ 1.993354] Console: switching to colour frame buffer device 160x501881lb # [ 2.004074] [drm] features: -context_init1882lb # [ 2.004831] [drm] number of scanouts: 11883lb # [ 2.004849] [drm] number of cap sets: 01884worker2 # [ 1.982637] systemd[1]: Finished Create Static Device Nodes in /dev.1885worker2 # [ 1.982859] systemd[1]: Reached target Preparation for Local File Systems.1886worker2 # [ 1.982886] systemd[1]: Reached target Local File Systems.1887client # [ 2.001719] systemd[1]: Finished Create Static Device Nodes in /dev.1888worker1 # [ 1.992380] systemd-udevd[79]: Using default interface naming scheme 'v261'.1889client # [ 2.002091] systemd[1]: Reached target Preparation for Local File Systems.1890client # [ 2.002114] systemd[1]: Reached target Local File Systems.1891worker1 # [ 1.996079] systemd[1]: Finished Create System Files and Directories.1892worker2 # [ 1.990313] systemd[1]: Starting Rule-based Manager for Device Events and Files...1893lb # [ 2.012206] virtio-pci 0000:00:05.0: [drm] Registered 1 planes with drm panic1894lb # [ 2.012222] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:05.0 on minor 01895client # [ 2.014454] systemd[1]: Starting Rule-based Manager for Device Events and Files...1896worker2 # [ 1.998270] virtio-pci 0000:00:05.0: [drm] Registered 1 planes with drm panic1897worker2 # [ 1.998285] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:05.0 on minor 01898client # [ 2.027905] virtio-pci 0000:00:05.0: [drm] fb0: virtio_gpudrmfb frame buffer device1899worker1 # [ 2.021053] systemd[1]: Started Rule-based Manager for Device Events and Files.1900lb # [ 2.036199] Console: switching to colour frame buffer device 160x501901lb # [ 2.032918] systemd-modules-load[74]: Inserted module 'dm_mod'1902lb # [ 2.034938] systemd-modules-load[74]: Module 'virtio_balloon' is built in1903worker2 # [ 2.021462] Console: switching to colour frame buffer device 160x501904client # [ 2.042283] systemd[1]: Finished Load Kernel Modules.1905client # [ 2.045556] systemd[1]: Starting Apply Kernel Variables...1906worker2 # [ 2.019611] systemd-modules-load[74]: Inserted module 'dm_mod'1907worker2 # [ 2.034793] systemd[1]: Started Journal Service.1908lb # [ 2.045499] systemd-modules-load[74]: Module 'virtio_console' is built in1909lb # [ 2.056027] systemd[1]: Started Journal Service.1910worker2 # [ 2.023000] systemd-modules-load[74]: Module 'virtio_balloon' is built in1911lb # [ 2.049070] systemd[1]: Starting Create System Files and Directories...1912worker2 # [ 2.046111] virtio-pci 0000:00:05.0: [drm] fb0: virtio_gpudrmfb frame buffer device1913worker2 # [ 2.034449] systemd-modules-load[74]: Module 'virtio_console' is built in1914worker2 # [ 2.037756] systemd[1]: Starting Create System Files and Directories...1915lb # [ 2.072452] virtio-pci 0000:00:05.0: [drm] fb0: virtio_gpudrmfb frame buffer device1916worker2 # [ 2.040840] systemd-modules-load[74]: Inserted module 'virtio_gpu'1917lb # [ 2.066607] systemd-modules-load[74]: Inserted module 'virtio_gpu'1918lb # [ 2.072237] systemd-modules-load[74]: Module 'virtio_rng' is built in1919worker2 # [ 2.052565] systemd-modules-load[74]: Module 'virtio_rng' is built in1920worker2 # [ 2.056413] systemd[1]: Finished Load Kernel Modules.1921client # [ 2.088319] systemd[1]: Started Journal Service.1922lb # [ 2.090351] systemd[1]: Finished Load Kernel Modules.1923client # [ 2.084323] systemd-modules-load[73]: Inserted module 'dm_mod'1924client # [ 2.085565] systemd-modules-load[73]: Module 'virtio_balloon' is built in1925worker1 # [ 2.092107] systemd[1]: Starting Virtual Console Setup...1926lb # [ 2.097281] systemd-udevd[78]: Using default interface naming scheme 'v261'.1927client # [ 2.092435] systemd-modules-load[73]: Module 'virtio_console' is built in1928worker2 # [ 2.079458] systemd-udevd[78]: Using default interface naming scheme 'v261'.1929client # [ 2.100412] systemd-modules-load[73]: Inserted module 'virtio_gpu'1930lb # [ 2.108940] systemd[1]: Starting Apply Kernel Variables...1931client # [ 2.102799] systemd-modules-load[73]: Module 'virtio_rng' is built in1932lb # [ 2.112820] systemd[1]: Finished Create System Files and Directories.1933worker2 # [ 2.092881] systemd[1]: Starting Apply Kernel Variables...1934client # [ 2.112595] systemd[1]: Starting Create System Files and Directories...1935client # [ 2.116979] systemd[1]: Finished Apply Kernel Variables.1936client # [ 2.118211] systemd-udevd[78]: Using default interface naming scheme 'v261'.1937worker2 # [ 2.110751] systemd[1]: Finished Apply Kernel Variables.1938client # [ 2.131373] systemd[1]: Finished Create System Files and Directories.1939lb # [ 2.141350] systemd[1]: Finished Apply Kernel Variables.1940lb # [ 2.145374] systemd[1]: Started Rule-based Manager for Device Events and Files.1941worker2 # [ 2.128866] systemd[1]: Finished Create System Files and Directories.1942client # [ 2.150420] systemd[1]: Started Rule-based Manager for Device Events and Files.1943worker2 # [ 2.137322] systemd[1]: Started Rule-based Manager for Device Events and Files.1944worker1 # [ 2.169263] systemd-vconsole-setup[102]: Configuration of first virtual console was skipped, ignoring remaining ones.1945worker1 # [ 2.172074] systemd[1]: Finished Virtual Console Setup.1946lb # [ 2.204089] systemd[1]: Starting Virtual Console Setup...1947client # [ 2.207790] systemd[1]: Starting Virtual Console Setup...1948worker2 # [ 2.192567] systemd[1]: Starting Virtual Console Setup...1949lb # [ 2.256488] systemd-vconsole-setup[101]: Configuration of first virtual console was skipped, ignoring remaining ones.1950lb # [ 2.259756] systemd[1]: Finished Virtual Console Setup.1951client # [ 2.256430] systemd-vconsole-setup[101]: Configuration of first virtual console was skipped, ignoring remaining ones.1952client # [ 2.259789] systemd[1]: Finished Virtual Console Setup.1953worker2 # [ 2.244667] systemd-vconsole-setup[100]: Configuration of first virtual console was skipped, ignoring remaining ones.1954worker2 # [ 2.247918] systemd[1]: Finished Virtual Console Setup.1955worker1 # [ 2.770647] systemd[1]: Finished Coldplug All udev Devices.1956worker1 # [ 2.771605] systemd[1]: Reached target System Initialization.1957worker1 # [ 2.772552] systemd[1]: Reached target Basic System.1958client # [ 2.860207] systemd[1]: Finished Coldplug All udev Devices.1959client # [ 2.862383] systemd[1]: Reached target System Initialization.1960client # [ 2.864099] systemd[1]: Reached target Basic System.1961worker2 # [ 2.860217] systemd[1]: Finished Coldplug All udev Devices.1962worker2 # [ 2.861140] systemd[1]: Reached target System Initialization.1963worker2 # [ 2.861954] systemd[1]: Reached target Basic System.1964lb # [ 2.891945] systemd[1]: Finished Coldplug All udev Devices.1965lb # [ 2.892921] systemd[1]: Reached target System Initialization.1966lb # [ 2.893738] systemd[1]: Reached target Basic System.1967worker1 # [ 2.930247] (udev-worker)[95]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name.1968worker1 # [ 2.941609] (udev-worker)[95]: Network interface NamePolicy= disabled on kernel command line.1969worker1 # [ 2.958676] (udev-worker)[93]: Network interface NamePolicy= disabled on kernel command line.1970worker2 # [ 3.007513] (udev-worker)[94]: Network interface NamePolicy= disabled on kernel command line.1971worker1 # [ 3.032970] systemd[1]: Found device /dev/disk/by-label/nixos.1972worker1 # [ 3.037240] systemd[1]: Reached target Initrd Root Device.1973client # [ 3.033954] (udev-worker)[90]: Network interface NamePolicy= disabled on kernel command line.1974worker1 # [ 3.041504] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos...1975lb # [ 3.057850] (udev-worker)[94]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name.1976client # [ 3.059716] (udev-worker)[100]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name.1977client # [ 3.064267] (udev-worker)[100]: Network interface NamePolicy= disabled on kernel command line.1978lb # [ 3.073874] (udev-worker)[96]: Network interface NamePolicy= disabled on kernel command line.1979worker2 # [ 3.054510] (udev-worker)[95]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name.1980lb # [ 3.080805] (udev-worker)[94]: Network interface NamePolicy= disabled on kernel command line.1981worker2 # [ 3.058445] (udev-worker)[95]: Network interface NamePolicy= disabled on kernel command line.1982worker1 # [ 3.087830] systemd-fsck[111]: nixos: clean, 12/65536 files, 13019/262144 blocks1983worker1 # [ 3.096151] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos.1984worker1 # [ 3.098188] systemd[1]: Mounting /sysroot...1985client # [ 3.120950] systemd[1]: Found device /dev/disk/by-label/nixos.1986client # [ 3.127596] systemd[1]: Reached target Initrd Root Device.1987lb # [ 3.138473] systemd[1]: Found device /dev/disk/by-label/nixos.1988client # [ 3.131862] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos...1989lb # [ 3.140821] systemd[1]: Reached target Initrd Root Device.1990worker1 # [ 3.149727] EXT4-fs (vda): mounted filesystem 1ca2ebb5-06c7-441d-ade7-bf2e4a4cf314 r/w with ordered data mode. Quota mode: none.1991lb # [ 3.144478] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos...1992worker1 # [ 3.141646] systemd[1]: Mounted /sysroot.1993worker1 # [ 3.143297] systemd[1]: Reached target Initrd Root File System.1994worker1 # [ 3.147137] systemd[1]: Starting Mountpoints Configured in the Real Root...1995worker2 # [ 3.132826] systemd[1]: Found device /dev/disk/by-label/nixos.1996worker2 # [ 3.134498] systemd[1]: Reached target Initrd Root Device.1997worker2 # [ 3.138763] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos...1998worker1 # [ 3.171073] systemd-sysroot-fstab-check[119]: /sysroot should be mounted in the initrd, will request daemon-reload.1999worker1 # [ 3.176113] systemd[1]: Reload requested from client PID 119 ('systemd-sysroot') (unit initrd-parse-etc.service)...2000worker1 # [ 3.181618] systemd[1]: Reloading...
loading 1,829 hidden lines…
3830worker2 # [ 26.026246] nix-grpc-daemon[770]: ts=2026-09-21T14:44:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3831worker1 # [ 26.219906] nix-grpc-daemon[713]: ts=2026-09-21T14:44:01Z level=info event=scheduler_connected addr=lb:500513832worker1 # [ 26.225041] nix-grpc-daemon[713]: ts=2026-09-21T14:44:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3833worker2 # [ 26.229124] nix-grpc-daemon[770]: ts=2026-09-21T14:44:01Z level=info event=scheduler_connected addr=lb:500513834worker2 # [ 26.233206] nix-grpc-daemon[770]: ts=2026-09-21T14:44:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3835worker1 # [ 26.427619] nix-grpc-daemon[713]: ts=2026-09-21T14:44:02Z level=info event=scheduler_connected addr=lb:500513836worker1 # [ 26.431795] nix-grpc-daemon[713]: ts=2026-09-21T14:44:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3837worker2 # [ 26.429894] nix-grpc-daemon[770]: ts=2026-09-21T14:44:02Z level=info event=scheduler_connected addr=lb:500513838worker2 # [ 26.434934] nix-grpc-daemon[770]: ts=2026-09-21T14:44:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3839lb: must succeed: curl -sf localhost:9901/clusters3840lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)3841worker1 # [ 26.628813] nix-grpc-daemon[713]: ts=2026-09-21T14:44:02Z level=info event=scheduler_connected addr=lb:500513842worker1 # [ 26.634235] nix-grpc-daemon[713]: ts=2026-09-21T14:44:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3843worker2 # [ 26.636056] nix-grpc-daemon[770]: ts=2026-09-21T14:44:02Z level=info event=scheduler_connected addr=lb:500513844worker2 # [ 26.641065] nix-grpc-daemon[770]: ts=2026-09-21T14:44:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3845worker1 # [ 26.834930] nix-grpc-daemon[713]: ts=2026-09-21T14:44:02Z level=info event=scheduler_connected addr=lb:500513846worker1 # [ 26.840100] nix-grpc-daemon[713]: ts=2026-09-21T14:44:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3847worker2 # [ 26.842032] nix-grpc-daemon[770]: ts=2026-09-21T14:44:02Z level=info event=scheduler_connected addr=lb:500513848worker2 # [ 26.847183] nix-grpc-daemon[770]: ts=2026-09-21T14:44:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3849worker1 # [ 27.041148] nix-grpc-daemon[713]: ts=2026-09-21T14:44:02Z level=info event=scheduler_connected addr=lb:500513850worker1 # [ 27.046177] nix-grpc-daemon[713]: ts=2026-09-21T14:44:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3851worker2 # [ 27.047902] nix-grpc-daemon[770]: ts=2026-09-21T14:44:02Z level=info event=scheduler_connected addr=lb:500513852worker2 # [ 27.052912] nix-grpc-daemon[770]: ts=2026-09-21T14:44:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3853worker1 # [ 27.246871] nix-grpc-daemon[713]: ts=2026-09-21T14:44:02Z level=info event=scheduler_connected addr=lb:500513854worker1 # [ 27.251980] nix-grpc-daemon[713]: ts=2026-09-21T14:44:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3855worker2 # [ 27.255598] nix-grpc-daemon[770]: ts=2026-09-21T14:44:02Z level=info event=scheduler_connected addr=lb:500513856worker2 # [ 27.259745] nix-grpc-daemon[770]: ts=2026-09-21T14:44:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3857worker1 # [ 27.454749] nix-grpc-daemon[713]: ts=2026-09-21T14:44:03Z level=info event=scheduler_connected addr=lb:500513858worker1 # [ 27.458911] nix-grpc-daemon[713]: ts=2026-09-21T14:44:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3859worker2 # [ 27.456157] nix-grpc-daemon[770]: ts=2026-09-21T14:44:03Z level=info event=scheduler_connected addr=lb:500513860worker2 # [ 27.461169] nix-grpc-daemon[770]: ts=2026-09-21T14:44:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3861lb: must succeed: curl -sf localhost:9901/clusters3862lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)3863worker1 # [ 27.655272] nix-grpc-daemon[713]: ts=2026-09-21T14:44:03Z level=info event=scheduler_connected addr=lb:500513864worker1 # [ 27.666731] nix-grpc-daemon[713]: ts=2026-09-21T14:44:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3865worker2 # [ 27.662255] nix-grpc-daemon[770]: ts=2026-09-21T14:44:03Z level=info event=scheduler_connected addr=lb:500513866worker2 # [ 27.671502] nix-grpc-daemon[770]: ts=2026-09-21T14:44:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3867worker1 # [ 27.864563] nix-grpc-daemon[713]: ts=2026-09-21T14:44:03Z level=info event=scheduler_connected addr=lb:500513868worker1 # [ 27.869668] nix-grpc-daemon[713]: ts=2026-09-21T14:44:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3869worker2 # [ 27.869327] nix-grpc-daemon[770]: ts=2026-09-21T14:44:03Z level=info event=scheduler_connected addr=lb:500513870worker2 # [ 27.874613] nix-grpc-daemon[770]: ts=2026-09-21T14:44:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3871worker1 # [ 28.070479] nix-grpc-daemon[713]: ts=2026-09-21T14:44:03Z level=info event=scheduler_connected addr=lb:500513872worker1 # [ 28.075750] nix-grpc-daemon[713]: ts=2026-09-21T14:44:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3873worker2 # [ 28.075701] nix-grpc-daemon[770]: ts=2026-09-21T14:44:03Z level=info event=scheduler_connected addr=lb:500513874worker2 # [ 28.080944] nix-grpc-daemon[770]: ts=2026-09-21T14:44:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3875worker1 # [ 28.276664] nix-grpc-daemon[713]: ts=2026-09-21T14:44:03Z level=info event=scheduler_connected addr=lb:500513876worker1 # [ 28.281939] nix-grpc-daemon[713]: ts=2026-09-21T14:44:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3877worker2 # [ 28.283663] nix-grpc-daemon[770]: ts=2026-09-21T14:44:03Z level=info event=scheduler_connected addr=lb:500513878worker2 # [ 28.287861] nix-grpc-daemon[770]: ts=2026-09-21T14:44:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3879worker1 # [ 28.485006] nix-grpc-daemon[713]: ts=2026-09-21T14:44:04Z level=info event=scheduler_connected addr=lb:500513880worker1 # [ 28.489218] nix-grpc-daemon[713]: ts=2026-09-21T14:44:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3881worker2 # [ 28.484557] nix-grpc-daemon[770]: ts=2026-09-21T14:44:04Z level=info event=scheduler_connected addr=lb:500513882worker2 # [ 28.489542] nix-grpc-daemon[770]: ts=2026-09-21T14:44:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3883lb: must succeed: curl -sf localhost:9901/clusters3884worker1 # [ 28.685810] nix-grpc-daemon[713]: ts=2026-09-21T14:44:04Z level=info event=scheduler_connected addr=lb:500513885worker1 # [ 28.691232] nix-grpc-daemon[713]: ts=2026-09-21T14:44:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3886lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)3887worker2 # [ 28.690692] nix-grpc-daemon[770]: ts=2026-09-21T14:44:04Z level=info event=scheduler_connected addr=lb:500513888worker2 # [ 28.695748] nix-grpc-daemon[770]: ts=2026-09-21T14:44:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3889worker1 # [ 28.892130] nix-grpc-daemon[713]: ts=2026-09-21T14:44:04Z level=info event=scheduler_connected addr=lb:500513890worker1 # [ 28.897528] nix-grpc-daemon[713]: ts=2026-09-21T14:44:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3891worker2 # [ 28.896986] nix-grpc-daemon[770]: ts=2026-09-21T14:44:04Z level=info event=scheduler_connected addr=lb:500513892worker2 # [ 28.902049] nix-grpc-daemon[770]: ts=2026-09-21T14:44:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3893worker1 # [ 29.098554] nix-grpc-daemon[713]: ts=2026-09-21T14:44:04Z level=info event=scheduler_connected addr=lb:500513894worker1 # [ 29.104152] nix-grpc-daemon[713]: ts=2026-09-21T14:44:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3895worker2 # [ 29.102913] nix-grpc-daemon[770]: ts=2026-09-21T14:44:04Z level=info event=scheduler_connected addr=lb:500513896worker2 # [ 29.108079] nix-grpc-daemon[770]: ts=2026-09-21T14:44:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3897worker1 # [ 29.304714] nix-grpc-daemon[713]: ts=2026-09-21T14:44:04Z level=info event=scheduler_connected addr=lb:500513898worker1 # [ 29.309943] nix-grpc-daemon[713]: ts=2026-09-21T14:44:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3899worker2 # [ 29.310904] nix-grpc-daemon[770]: ts=2026-09-21T14:44:04Z level=info event=scheduler_connected addr=lb:500513900worker2 # [ 29.315015] nix-grpc-daemon[770]: ts=2026-09-21T14:44:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3901worker1 # [ 29.513063] nix-grpc-daemon[713]: ts=2026-09-21T14:44:05Z level=info event=scheduler_connected addr=lb:500513902worker1 # [ 29.517215] nix-grpc-daemon[713]: ts=2026-09-21T14:44:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3903worker2 # [ 29.511465] nix-grpc-daemon[770]: ts=2026-09-21T14:44:05Z level=info event=scheduler_connected addr=lb:500513904worker2 # [ 29.516645] nix-grpc-daemon[770]: ts=2026-09-21T14:44:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3905lb: must succeed: curl -sf localhost:9901/clusters3906worker1 # [ 29.713773] nix-grpc-daemon[713]: ts=2026-09-21T14:44:05Z level=info event=scheduler_connected addr=lb:500513907worker1 # [ 29.719194] nix-grpc-daemon[713]: ts=2026-09-21T14:44:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3908worker2 # [ 29.717874] nix-grpc-daemon[770]: ts=2026-09-21T14:44:05Z level=info event=scheduler_connected addr=lb:500513909worker2 # [ 29.722979] nix-grpc-daemon[770]: ts=2026-09-21T14:44:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3910lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)3911worker1 # [ 29.920248] nix-grpc-daemon[713]: ts=2026-09-21T14:44:05Z level=info event=scheduler_connected addr=lb:500513912worker1 # [ 29.925363] nix-grpc-daemon[713]: ts=2026-09-21T14:44:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3913worker2 # [ 29.924119] nix-grpc-daemon[770]: ts=2026-09-21T14:44:05Z level=info event=scheduler_connected addr=lb:500513914worker2 # [ 29.929157] nix-grpc-daemon[770]: ts=2026-09-21T14:44:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3915worker1 # [ 30.126275] nix-grpc-daemon[713]: ts=2026-09-21T14:44:05Z level=info event=scheduler_connected addr=lb:500513916worker1 # [ 30.131341] nix-grpc-daemon[713]: ts=2026-09-21T14:44:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3917worker2 # [ 30.130202] nix-grpc-daemon[770]: ts=2026-09-21T14:44:05Z level=info event=scheduler_connected addr=lb:500513918worker2 # [ 30.135243] nix-grpc-daemon[770]: ts=2026-09-21T14:44:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3919worker1 # [ 30.332288] nix-grpc-daemon[713]: ts=2026-09-21T14:44:05Z level=info event=scheduler_connected addr=lb:500513920worker1 # [ 30.337442] nix-grpc-daemon[713]: ts=2026-09-21T14:44:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3921worker2 # [ 30.338158] nix-grpc-daemon[770]: ts=2026-09-21T14:44:05Z level=info event=scheduler_connected addr=lb:500513922worker2 # [ 30.342272] nix-grpc-daemon[770]: ts=2026-09-21T14:44:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3923worker1 # [ 30.540392] nix-grpc-daemon[713]: ts=2026-09-21T14:44:06Z level=info event=scheduler_connected addr=lb:500513924worker1 # [ 30.544467] nix-grpc-daemon[713]: ts=2026-09-21T14:44:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3925worker2 # [ 30.538933] nix-grpc-daemon[770]: ts=2026-09-21T14:44:06Z level=info event=scheduler_connected addr=lb:500513926worker2 # [ 30.544079] nix-grpc-daemon[770]: ts=2026-09-21T14:44:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3927worker1 # [ 30.741125] nix-grpc-daemon[713]: ts=2026-09-21T14:44:06Z level=info event=scheduler_connected addr=lb:500513928worker1 # [ 30.746391] nix-grpc-daemon[713]: ts=2026-09-21T14:44:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3929lb: must succeed: curl -sf localhost:9901/clusters3930worker2 # [ 30.745075] nix-grpc-daemon[770]: ts=2026-09-21T14:44:06Z level=info event=scheduler_connected addr=lb:500513931worker2 # [ 30.750057] nix-grpc-daemon[770]: ts=2026-09-21T14:44:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3932lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)3933worker1 # [ 30.948399] nix-grpc-daemon[713]: ts=2026-09-21T14:44:06Z level=info event=scheduler_connected addr=lb:500513934worker1 # [ 30.955701] nix-grpc-daemon[713]: ts=2026-09-21T14:44:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3935worker2 # [ 30.951416] nix-grpc-daemon[770]: ts=2026-09-21T14:44:06Z level=info event=scheduler_connected addr=lb:500513936worker2 # [ 30.963253] nix-grpc-daemon[770]: ts=2026-09-21T14:44:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3937worker1 # [ 31.153902] nix-grpc-daemon[713]: ts=2026-09-21T14:44:06Z level=info event=scheduler_connected addr=lb:500513938worker1 # [ 31.159193] nix-grpc-daemon[713]: ts=2026-09-21T14:44:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3939worker2 # [ 31.159524] nix-grpc-daemon[770]: ts=2026-09-21T14:44:06Z level=info event=scheduler_connected addr=lb:500513940worker2 # [ 31.164719] nix-grpc-daemon[770]: ts=2026-09-21T14:44:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3941worker1 # [ 31.359909] nix-grpc-daemon[713]: ts=2026-09-21T14:44:06Z level=info event=scheduler_connected addr=lb:500513942worker1 # [ 31.365051] nix-grpc-daemon[713]: ts=2026-09-21T14:44:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3943worker2 # [ 31.367964] nix-grpc-daemon[770]: ts=2026-09-21T14:44:06Z level=info event=scheduler_connected addr=lb:500513944worker2 # [ 31.371968] nix-grpc-daemon[770]: ts=2026-09-21T14:44:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3945worker1 # [ 31.567814] nix-grpc-daemon[713]: ts=2026-09-21T14:44:07Z level=info event=scheduler_connected addr=lb:500513946worker1 # [ 31.572119] nix-grpc-daemon[713]: ts=2026-09-21T14:44:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3947worker2 # [ 31.568264] nix-grpc-daemon[770]: ts=2026-09-21T14:44:07Z level=info event=scheduler_connected addr=lb:500513948worker2 # [ 31.573185] nix-grpc-daemon[770]: ts=2026-09-21T14:44:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3949worker1 # [ 31.768477] nix-grpc-daemon[713]: ts=2026-09-21T14:44:07Z level=info event=scheduler_connected addr=lb:500513950worker1 # [ 31.773502] nix-grpc-daemon[713]: ts=2026-09-21T14:44:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3951worker2 # [ 31.774382] nix-grpc-daemon[770]: ts=2026-09-21T14:44:07Z level=info event=scheduler_connected addr=lb:500513952worker2 # [ 31.779405] nix-grpc-daemon[770]: ts=2026-09-21T14:44:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3953lb: must succeed: curl -sf localhost:9901/clusters3954lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)3955worker1 # [ 31.974425] nix-grpc-daemon[713]: ts=2026-09-21T14:44:07Z level=info event=scheduler_connected addr=lb:500513956worker1 # [ 31.979633] nix-grpc-daemon[713]: ts=2026-09-21T14:44:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3957worker2 # [ 31.980291] nix-grpc-daemon[770]: ts=2026-09-21T14:44:07Z level=info event=scheduler_connected addr=lb:500513958worker2 # [ 31.985289] nix-grpc-daemon[770]: ts=2026-09-21T14:44:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3959worker1 # [ 32.180438] nix-grpc-daemon[713]: ts=2026-09-21T14:44:07Z level=info event=scheduler_connected addr=lb:500513960worker1 # [ 32.185604] nix-grpc-daemon[713]: ts=2026-09-21T14:44:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3961worker2 # [ 32.186339] nix-grpc-daemon[770]: ts=2026-09-21T14:44:07Z level=info event=scheduler_connected addr=lb:500513962worker2 # [ 32.191210] nix-grpc-daemon[770]: ts=2026-09-21T14:44:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3963worker1 # [ 32.387348] nix-grpc-daemon[713]: ts=2026-09-21T14:44:07Z level=info event=scheduler_connected addr=lb:500513964worker1 # [ 32.393274] nix-grpc-daemon[713]: ts=2026-09-21T14:44:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3965worker2 # [ 32.395506] nix-grpc-daemon[770]: ts=2026-09-21T14:44:07Z level=info event=scheduler_connected addr=lb:500513966worker2 # [ 32.399923] nix-grpc-daemon[770]: ts=2026-09-21T14:44:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3967worker1 # [ 32.593077] nix-grpc-daemon[713]: ts=2026-09-21T14:44:08Z level=info event=scheduler_connected addr=lb:500513968worker1 # [ 32.597350] nix-grpc-daemon[713]: ts=2026-09-21T14:44:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3969worker2 # [ 32.597111] nix-grpc-daemon[770]: ts=2026-09-21T14:44:08Z level=info event=scheduler_connected addr=lb:500513970worker2 # [ 32.602780] nix-grpc-daemon[770]: ts=2026-09-21T14:44:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3971worker1 # [ 32.794158] nix-grpc-daemon[713]: ts=2026-09-21T14:44:08Z level=info event=scheduler_connected addr=lb:500513972worker1 # [ 32.799888] nix-grpc-daemon[713]: ts=2026-09-21T14:44:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3973worker2 # [ 32.804428] nix-grpc-daemon[770]: ts=2026-09-21T14:44:08Z level=info event=scheduler_connected addr=lb:500513974worker2 # [ 32.809999] nix-grpc-daemon[770]: ts=2026-09-21T14:44:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3975lb: must succeed: curl -sf localhost:9901/clusters3976worker1 # [ 33.001301] nix-grpc-daemon[713]: ts=2026-09-21T14:44:08Z level=info event=scheduler_connected addr=lb:500513977worker1 # [ 33.007104] nix-grpc-daemon[713]: ts=2026-09-21T14:44:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3978lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.12 seconds)3979worker2 # [ 33.011660] nix-grpc-daemon[770]: ts=2026-09-21T14:44:08Z level=info event=scheduler_connected addr=lb:500513980worker2 # [ 33.017985] nix-grpc-daemon[770]: ts=2026-09-21T14:44:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3981worker1 # [ 33.208130] nix-grpc-daemon[713]: ts=2026-09-21T14:44:08Z level=info event=scheduler_connected addr=lb:500513982worker1 # [ 33.213618] nix-grpc-daemon[713]: ts=2026-09-21T14:44:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3983worker2 # [ 33.219524] nix-grpc-daemon[770]: ts=2026-09-21T14:44:08Z level=info event=scheduler_connected addr=lb:500513984worker2 # [ 33.226013] nix-grpc-daemon[770]: ts=2026-09-21T14:44:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3985worker1 # [ 33.414705] nix-grpc-daemon[713]: ts=2026-09-21T14:44:09Z level=info event=scheduler_connected addr=lb:500513986worker1 # [ 33.420937] nix-grpc-daemon[713]: ts=2026-09-21T14:44:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3987worker2 # [ 33.430809] nix-grpc-daemon[770]: ts=2026-09-21T14:44:09Z level=info event=scheduler_connected addr=lb:500513988worker2 # [ 33.435199] nix-grpc-daemon[770]: ts=2026-09-21T14:44:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3989worker1 # [ 33.624119] nix-grpc-daemon[713]: ts=2026-09-21T14:44:09Z level=info event=scheduler_connected addr=lb:500513990worker1 # [ 33.630485] nix-grpc-daemon[713]: ts=2026-09-21T14:44:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3991worker2 # [ 33.631635] nix-grpc-daemon[770]: ts=2026-09-21T14:44:09Z level=info event=scheduler_connected addr=lb:500513992worker2 # [ 33.637949] nix-grpc-daemon[770]: ts=2026-09-21T14:44:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3993worker1 # [ 33.831450] nix-grpc-daemon[713]: ts=2026-09-21T14:44:09Z level=info event=scheduler_connected addr=lb:500513994worker1 # [ 33.837569] nix-grpc-daemon[713]: ts=2026-09-21T14:44:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3995worker2 # [ 33.839175] nix-grpc-daemon[770]: ts=2026-09-21T14:44:09Z level=info event=scheduler_connected addr=lb:500513996worker2 # [ 33.845482] nix-grpc-daemon[770]: ts=2026-09-21T14:44:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3997lb: must succeed: curl -sf localhost:9901/clusters3998worker1 # [ 34.038614] nix-grpc-daemon[713]: ts=2026-09-21T14:44:09Z level=info event=scheduler_connected addr=lb:500513999worker1 # [ 34.044392] nix-grpc-daemon[713]: ts=2026-09-21T14:44:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4000worker2 # [ 34.047067] nix-grpc-daemon[770]: ts=2026-09-21T14:44:09Z level=info event=scheduler_connected addr=lb:500514001worker2 # [ 34.052907] nix-grpc-daemon[770]: ts=2026-09-21T14:44:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4002worker1 # [ 34.123223] nix-grpc-daemon[713]: ts=2026-09-21T14:44:09Z level=info event=scheduler_take_over4003lb # [ 34.129281] niks3-server[979]: 2026/09/21 14:44:09 INFO lead: acquired remote=192.168.1.3:499344004lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.12 seconds)4005worker1 # [ 34.245736] nix-grpc-daemon[713]: ts=2026-09-21T14:44:09Z level=info event=scheduler_connected addr=lb:500514006worker1 # [ 34.272785] nix-grpc-daemon[713]: ts=2026-09-21T14:44:09Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:529184007worker2 # [ 34.254688] nix-grpc-daemon[770]: ts=2026-09-21T14:44:09Z level=info event=scheduler_connected addr=lb:500514008worker1 # [ 34.290473] nix-grpc-daemon[713]: ts=2026-09-21T14:44:09Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=04009worker1 # [ 34.311211] nix-grpc-daemon[713]: ts=2026-09-21T14:44:09Z level=info event=oidc_jwks provider=mock keys=14010worker1 # [ 34.319751] nix-grpc-daemon[713]: ts=2026-09-21T14:44:09Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:529184011worker1 # [ 34.332277] nix-grpc-daemon[713]: ts=2026-09-21T14:44:09Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=04012lb: must succeed: curl -sf localhost:9901/clusters4013lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.10 seconds)4014worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4015worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.10 seconds)4016lb: must succeed: curl -sf localhost:9901/clusters4017lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.10 seconds)4018lb: must succeed: curl -sf localhost:9901/clusters4019lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)4020lb: must succeed: curl -sf localhost:9901/clusters4021lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4022lb: must succeed: curl -sf localhost:9901/clusters4023lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4024worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open'4025worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open', in 0.07 seconds)4026worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4027worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)4028lb: must succeed: curl -sf localhost:9901/clusters4029lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.10 seconds)4030lb: must succeed: curl -sf localhost:9901/clusters4031lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4032worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4033worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.10 seconds)4034worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4035worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.09 seconds)4036worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4037worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)4038worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4039worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)4040(finished: subtest: one node schedules and both builders hold a WorkerSession on it, in 10.72 seconds)4041subtest: balancer auth: client cert, bearer token, nothing, unknown CN4042worker2 # [ 36.494316] nix-grpc-daemon[770]: ts=2026-09-21T14:44:12Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:453964043worker2 # [ 36.523595] systemd[1]: Started Nix Daemon.4044worker2 # [ 36.731572] nix-daemon[838]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4045worker2 # [ 36.756175] nix-grpc-daemon[770]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 56 ms (attempt 1/5)4046worker2 # [ 36.816207] nix-grpc-daemon[770]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 176 ms (attempt 2/5)4047worker2 # [ 36.995929] nix-grpc-daemon[770]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 31 ms (attempt 3/5)4048worker2 # [ 37.029876] nix-grpc-daemon[770]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 653 ms (attempt 4/5)4049worker2 # [ 37.698577] nix-grpc-daemon[770]: disabling binary cache 'https://cache.nixos.org' for 60 seconds4050worker2 # [ 37.706731] nix-grpc-daemon[770]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4051worker1 # [ 37.777300] nix-grpc-daemon[713]: ts=2026-09-21T14:44:13Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:529344052worker1 # [ 37.791245] systemd[1]: Started Nix Daemon.4053worker1 # [ 37.978456] nix-daemon[852]: accepted connection from pid 713, user nix-grpc-daemon (trusted)4054worker1 # [ 38.001207] nix-grpc-daemon[713]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 35 ms (attempt 1/5)4055worker1 # [ 38.039808] nix-grpc-daemon[713]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 129 ms (attempt 2/5)4056worker1 # [ 38.172115] nix-grpc-daemon[713]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 251 ms (attempt 3/5)4057worker1 # [ 38.433337] nix-grpc-daemon[713]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 738 ms (attempt 4/5)4058worker1 # [ 39.175285] nix-grpc-daemon[713]: disabling binary cache 'https://cache.nixos.org' for 60 seconds4059worker1 # [ 39.180589] nix-grpc-daemon[713]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4060worker1 # [ 39.195817] nix-grpc-daemon[713]: ts=2026-09-21T14:44:14Z level=debug event=path_info_miss path=6v5lkzqzv8y8aql79zri93a767rl7bwj-nixos-system-client-test error="error: path '/nix/store/6v5lkzqzv8y8aql79zri93a767rl7bwj-nixos-system-client-test' is required, but there is no substituter that can build it"4061worker2 # [ 39.190115] nix-grpc-daemon[770]: ts=2026-09-21T14:44:14Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:453964062worker2 # [ 39.203045] nix-grpc-daemon[770]: ts=2026-09-21T14:44:14Z level=debug event=path_info_miss path=6v5lkzqzv8y8aql79zri93a767rl7bwj-nixos-system-client-test error="error: path '/nix/store/6v5lkzqzv8y8aql79zri93a767rl7bwj-nixos-system-client-test' is required, but there is no substituter that can build it"4063client: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt4064client: (finished: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt, in 0.08 seconds)4065client: must succeed: install -D /nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key4066client: (finished: must succeed: install -D /nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key, in 0.04 seconds)4067worker1 # [ 39.551857] nix-grpc-daemon[713]: ts=2026-09-21T14:44:15Z level=debug event=rpc_start method=QueryMissing cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:529344068worker1 # [ 39.568956] nix-grpc-daemon[713]: ts=2026-09-21T14:44:15Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:529344069worker1 # [ 39.580965] nix-grpc-daemon[713]: ts=2026-09-21T14:44:15Z level=debug event=path_info_miss path=6v5lkzqzv8y8aql79zri93a767rl7bwj-nixos-system-client-test error="error: path '/nix/store/6v5lkzqzv8y8aql79zri93a767rl7bwj-nixos-system-client-test' is required, but there is no substituter that can build it"4070worker2 # [ 39.576292] nix-grpc-daemon[770]: ts=2026-09-21T14:44:15Z level=info event=oidc_jwks provider=mock keys=14071worker2 # [ 39.584159] nix-grpc-daemon[770]: ts=2026-09-21T14:44:15Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:453964072worker2 # [ 39.592183] nix-grpc-daemon[770]: ts=2026-09-21T14:44:15Z level=debug event=path_info_miss path=6v5lkzqzv8y8aql79zri93a767rl7bwj-nixos-system-client-test error="error: path '/nix/store/6v5lkzqzv8y8aql79zri93a767rl7bwj-nixos-system-client-test' is required, but there is no substituter that can build it"4073client: must succeed: rm -r /var/lib/nix-grpc-store4074client: (finished: must succeed: rm -r /var/lib/nix-grpc-store, in 0.03 seconds)4075worker1 # [ 39.836237] nix-grpc-daemon[713]: ts=2026-09-21T14:44:15Z level=info event=denied method=QueryMissing cn=- role=none4076worker2 # [ 39.990298] nix-grpc-daemon[770]: ts=2026-09-21T14:44:15Z level=info event=denied method=QueryMissing cn=stranger role=none4077(finished: subtest: balancer auth: client cert, bearer token, nothing, unknown CN, in 3.86 seconds)4078subtest: DAG build is scheduled across workers and lands in the cache4079client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 >&24080client # don't know how to build these paths:4081client # /nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv4082worker2 # [ 40.161113] nix-grpc-daemon[770]: ts=2026-09-21T14:44:15Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:453964083worker2 # [ 40.170186] nix-grpc-daemon[770]: ts=2026-09-21T14:44:15Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:453964084worker1 # [ 40.227878] nix-grpc-daemon[713]: ts=2026-09-21T14:44:15Z level=debug event=want client=1 drv=619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv cached=04085worker1 # [ 40.233203] nix-grpc-daemon[713]: ts=2026-09-21T14:44:15Z level=debug event=assigned drv=619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv worker=192.168.1.3:50051 clients=14086worker1 # [ 40.277667] nix-grpc-daemon[713]: ts=2026-09-21T14:44:15Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:529344087worker1 # [ 40.295301] nix-grpc-daemon[713]: ts=2026-09-21T14:44:15Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:529344088client # copying 1 paths to 'lb:50051'...4089worker1 # [ 40.311627] nix-grpc-daemon[713]: ts=2026-09-21T14:44:15Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:529344090client # copying path '/nix/store/619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv' to 'lb:50051'...4091worker1 # [ 40.333658] nix-daemon[852]: accepted connection from pid 713, user nix-grpc-daemon (trusted)4092worker1 # [ 40.376810] nix-grpc-daemon[713]: ts=2026-09-21T14:44:15Z level=info event=niks3_push_started pid=8664093worker1 # [ 40.557630] nix-grpc-daemon[866]: time=2026-09-21T14:44:16.165Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.key ca=""4094worker1 # [ 40.669138] nix-daemon[852]: accepted connection from pid 878, user nix-grpc-daemon (trusted)4095worker1 # [ 40.691651] nix-daemon[852]: reaped child process 881, status = succeeded4096worker1 # [ 40.786912] nix-daemon[852]: accepted connection from pid 883, user nix-grpc-daemon (trusted)4097worker1 # [ 40.805453] nix-daemon[852]: reaped child process 886, status = succeeded4098lb # [ 40.809425] niks3-server[979]: 2026/09/21 14:44:16 INFO Received uploads request method=POST path=/api/pending_closures4099worker1 # [ 40.829683] nix-grpc-daemon[866]: time=2026-09-21T14:44:16.437Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4100worker1 # [ 40.831154] nix-grpc-daemon[866]: time=2026-09-21T14:44:16.437Z level=INFO msg="Uploading 619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv (504B)"4101lb # [ 40.889158] niks3-server[979]: 2026/09/21 14:44:16 INFO Registered completed upload object_key=nar/1il50j4vra11xfnqgjhjgrjn1753frlhxnss6mh1dmp5ww3is34w.nar.zst4102lb # [ 40.903342] niks3-server[979]: 2026/09/21 14:44:16 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign4103worker1 # [ 40.906859] nix-grpc-daemon[866]: time=2026-09-21T14:44:16.514Z level=INFO msg="Uploading 1 narinfos"4104lb # [ 40.916644] niks3-server[979]: 2026/09/21 14:44:16 INFO Signed narinfos id=1 count=14105lb # [ 40.931562] niks3-server[979]: 2026/09/21 14:44:16 INFO Registered completed upload object_key=619ccx60pfv52kv4hhgqpj1ivdmfcr5r.ls4106lb # [ 40.940152] niks3-server[979]: 2026/09/21 14:44:16 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete4107lb # [ 40.948406] niks3-server[979]: 2026/09/21 14:44:16 INFO Completed upload id=14108worker1 # [ 40.945938] nix-grpc-daemon[866]: time=2026-09-21T14:44:16.553Z level=INFO msg="Upload complete. (383ms)"4109lb # [ 40.956390] niks3-server[979]: 2026/09/21 14:44:16 INFO Registered completed upload object_key=619ccx60pfv52kv4hhgqpj1ivdmfcr5r.narinfo4110worker1 # [ 40.957865] nix-grpc-daemon[713]: ts=2026-09-21T14:44:16Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52934 duration_s=0 paths=1 nar_bytes_in=5044111worker1 # [ 40.983140] nix-grpc-daemon[713]: ts=2026-09-21T14:44:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:529344112worker1 # [ 40.991133] nix-daemon[852]: accepted connection from pid 713, user nix-grpc-daemon (trusted)4113worker1 # [ 41.001076] nix-daemon[852]: reaped child process 864, status = succeeded4114client # building '/nix/store/619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv' on lb:50051...4115client # farm-a> node-a: building 619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv4116worker1 # [ 41.020326] nix-daemon[852]: accepted connection from pid 713, user nix-grpc-daemon (trusted)4117client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds4118client # farm-a> LOG-farm-a-t14119worker1 # [ 42.065798] nix-daemon[852]: reaped child process 890, status = succeeded4120worker1 # [ 42.168960] nix-daemon[852]: accepted connection from pid 907, user nix-grpc-daemon (trusted)4121worker1 # [ 42.189175] nix-daemon[852]: reaped child process 910, status = succeeded4122lb # [ 42.198373] niks3-server[979]: 2026/09/21 14:44:17 INFO Received uploads request method=POST path=/api/pending_closures4123worker1 # [ 42.207313] nix-grpc-daemon[866]: time=2026-09-21T14:44:17.813Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4124worker1 # [ 42.210098] nix-grpc-daemon[866]: time=2026-09-21T14:44:17.813Z level=INFO msg="Uploading dg404zlvxfrspnxl6cc1qkc6i83m5a3a-farm-a (128B)"4125lb # [ 42.227449] niks3-server[979]: 2026/09/21 14:44:17 INFO Registered completed upload object_key=log/619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv4126lb # [ 42.251857] niks3-server[979]: 2026/09/21 14:44:17 INFO Registered completed upload object_key=nar/1rmil4ja9jzfwn6l5hwbir70b6plx99sgk0pnjjy377ngbc3vz28.nar.zst4127worker1 # [ 42.276112] nix-grpc-daemon[866]: time=2026-09-21T14:44:17.883Z level=INFO msg="Uploading 1 narinfos"4128lb # [ 42.273358] niks3-server[979]: 2026/09/21 14:44:17 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign4129lb # [ 42.284317] niks3-server[979]: 2026/09/21 14:44:17 INFO Signed narinfos id=2 count=14130lb # [ 42.287362] niks3-server[979]: 2026/09/21 14:44:17 INFO Registered completed upload object_key=dg404zlvxfrspnxl6cc1qkc6i83m5a3a.ls4131lb # [ 42.304144] niks3-server[979]: 2026/09/21 14:44:17 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete4132lb # [ 42.315349] niks3-server[979]: 2026/09/21 14:44:17 INFO Completed upload id=24133worker1 # [ 42.311545] nix-grpc-daemon[866]: time=2026-09-21T14:44:17.920Z level=INFO msg="Upload complete. (267ms)"4134lb # [ 42.319803] niks3-server[979]: 2026/09/21 14:44:17 INFO Registered completed upload object_key=dg404zlvxfrspnxl6cc1qkc6i83m5a3a.narinfo4135worker1 # [ 42.337089] nix-grpc-daemon[713]: ts=2026-09-21T14:44:17Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52934 duration_s=1 drv=619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv assign_id=1 outputs=14136worker1 # [ 42.348338] nix-grpc-daemon[713]: ts=2026-09-21T14:44:17Z level=debug event=want client=1 drv=79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv cached=04137worker1 # [ 42.354688] nix-grpc-daemon[713]: ts=2026-09-21T14:44:17Z level=debug event=assigned drv=79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv worker=192.168.1.3:50051 clients=14138worker1 # [ 42.358868] nix-daemon[852]: reaped child process 888, status = succeeded4139worker1 # [ 42.376597] nix-grpc-daemon[713]: ts=2026-09-21T14:44:17Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:529344140worker1 # [ 42.392413] nix-grpc-daemon[713]: ts=2026-09-21T14:44:18Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:529344141worker1 # [ 42.406515] nix-grpc-daemon[866]: time=2026-09-21T14:44:18.015Z level=INFO msg="All 1 paths already cached"4142client # copying 1 paths to 'lb:50051'...4143worker1 # [ 42.416118] nix-grpc-daemon[713]: ts=2026-09-21T14:44:18Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:529344144client # copying path '/nix/store/79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv' to 'lb:50051'...4145worker1 # [ 42.438337] nix-daemon[852]: accepted connection from pid 713, user nix-grpc-daemon (trusted)4146worker1 # [ 42.599702] nix-daemon[852]: accepted connection from pid 916, user nix-grpc-daemon (trusted)4147worker1 # [ 42.619574] nix-daemon[852]: reaped child process 919, status = succeeded4148worker1 # [ 42.724669] nix-daemon[852]: accepted connection from pid 921, user nix-grpc-daemon (trusted)4149worker1 # [ 42.741131] nix-daemon[852]: reaped child process 924, status = succeeded4150lb # [ 42.750460] niks3-server[979]: 2026/09/21 14:44:18 INFO Received uploads request method=POST path=/api/pending_closures4151worker1 # [ 42.755691] nix-grpc-daemon[866]: time=2026-09-21T14:44:18.363Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4152worker1 # [ 42.757397] nix-grpc-daemon[866]: time=2026-09-21T14:44:18.363Z level=INFO msg="Uploading 79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv (624B)"4153lb # [ 42.783143] niks3-server[979]: 2026/09/21 14:44:18 INFO Registered completed upload object_key=nar/1xk4kbz8l387hfrqq2xi6as79has336h62f2ra95472yqjclksjg.nar.zst4154worker1 # [ 42.789320] nix-grpc-daemon[866]: time=2026-09-21T14:44:18.397Z level=INFO msg="Uploading 1 narinfos"4155lb # [ 42.798642] niks3-server[979]: 2026/09/21 14:44:18 INFO Received sign narinfos request method=POST path=/api/pending_closures/3/sign4156lb # [ 42.809281] niks3-server[979]: 2026/09/21 14:44:18 INFO Signed narinfos id=3 count=14157lb # [ 42.811683] niks3-server[979]: 2026/09/21 14:44:18 INFO Registered completed upload object_key=79z38hfi82zcmb9yq1gcmfviimgf6z2l.ls4158lb # [ 42.823704] niks3-server[979]: 2026/09/21 14:44:18 INFO Received complete upload request method=POST path=/api/pending_closures/3/complete4159lb # [ 42.830452] niks3-server[979]: 2026/09/21 14:44:18 INFO Completed upload id=34160worker1 # [ 42.826491] nix-grpc-daemon[866]: time=2026-09-21T14:44:18.435Z level=INFO msg="Upload complete. (371ms)"4161lb # [ 42.832630] niks3-server[979]: 2026/09/21 14:44:18 INFO Registered completed upload object_key=79z38hfi82zcmb9yq1gcmfviimgf6z2l.narinfo4162worker1 # [ 42.836282] nix-grpc-daemon[713]: ts=2026-09-21T14:44:18Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52934 duration_s=0 paths=1 nar_bytes_in=6244163worker1 # [ 42.851248] nix-grpc-daemon[713]: ts=2026-09-21T14:44:18Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:529344164worker1 # [ 42.860563] nix-daemon[852]: accepted connection from pid 713, user nix-grpc-daemon (trusted)4165worker1 # [ 42.863701] nix-daemon[852]: reaped child process 914, status = succeeded4166client # building '/nix/store/79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv' on lb:50051...4167worker1 # [ 42.874780] nix-daemon[852]: accepted connection from pid 713, user nix-grpc-daemon (trusted)4168client # farm-b> node-a: building 79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv4169client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds4170client # farm-b> LOG-farm-b-t14171worker1 # [ 44.390837] nix-daemon[852]: reaped child process 928, status = succeeded4172worker1 # [ 44.455034] nix-daemon[852]: accepted connection from pid 944, user nix-grpc-daemon (trusted)4173worker1 # [ 44.480002] nix-daemon[852]: reaped child process 947, status = succeeded4174lb # [ 44.494574] niks3-server[979]: 2026/09/21 14:44:20 INFO Received uploads request method=POST path=/api/pending_closures4175worker1 # [ 44.501408] nix-grpc-daemon[866]: time=2026-09-21T14:44:20.108Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4176worker1 # [ 44.503219] nix-grpc-daemon[866]: time=2026-09-21T14:44:20.109Z level=INFO msg="Uploading 8n9g36vw9957r150a7wg1k6yj9fbqm8r-farm-b (176B)"4177lb # [ 44.529932] niks3-server[979]: 2026/09/21 14:44:20 INFO Registered completed upload object_key=nar/1mx8h9irf9ykfzxs6q3lwxbzxwv5d2s9mh2x0sky7448mlc1qxrx.nar.zst4178lb # [ 44.546006] niks3-server[979]: 2026/09/21 14:44:20 INFO Registered completed upload object_key=log/79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv4179lb # [ 44.553987] niks3-server[979]: 2026/09/21 14:44:20 INFO Received sign narinfos request method=POST path=/api/pending_closures/4/sign4180worker1 # [ 44.550710] nix-grpc-daemon[866]: time=2026-09-21T14:44:20.158Z level=INFO msg="Uploading 1 narinfos"4181lb # [ 44.563364] niks3-server[979]: 2026/09/21 14:44:20 INFO Signed narinfos id=4 count=14182worker1 # [ 44.576323] nix-grpc-daemon[866]: time=2026-09-21T14:44:20.184Z level=INFO msg="Upload complete. (288ms)"4183lb # [ 44.573355] niks3-server[979]: 2026/09/21 14:44:20 INFO Received complete upload request method=POST path=/api/pending_closures/4/complete4184lb # [ 44.588865] niks3-server[979]: 2026/09/21 14:44:20 INFO Completed upload id=44185lb # [ 44.590998] niks3-server[979]: 2026/09/21 14:44:20 INFO Registered completed upload object_key=8n9g36vw9957r150a7wg1k6yj9fbqm8r.ls4186lb # [ 44.596775] niks3-server[979]: 2026/09/21 14:44:20 INFO Registered completed upload object_key=8n9g36vw9957r150a7wg1k6yj9fbqm8r.narinfo4187worker1 # [ 44.594251] nix-grpc-daemon[713]: ts=2026-09-21T14:44:20Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52934 duration_s=1 drv=79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv assign_id=2 outputs=14188worker1 # [ 44.615036] nix-grpc-daemon[713]: ts=2026-09-21T14:44:20Z level=debug event=want client=1 drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv cached=04189worker1 # [ 44.620658] nix-grpc-daemon[713]: ts=2026-09-21T14:44:20Z level=debug event=assigned drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv worker=192.168.1.3:50051 clients=14190worker1 # [ 44.626177] nix-daemon[852]: reaped child process 926, status = succeeded4191worker1 # [ 44.646103] nix-grpc-daemon[713]: ts=2026-09-21T14:44:20Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:529344192worker1 # [ 44.660971] nix-grpc-daemon[713]: ts=2026-09-21T14:44:20Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:529344193worker1 # [ 44.675194] nix-grpc-daemon[866]: time=2026-09-21T14:44:20.283Z level=INFO msg="All 2 paths already cached"4194client # copying 1 paths to 'lb:50051'...4195worker1 # [ 44.685302] nix-grpc-daemon[713]: ts=2026-09-21T14:44:20Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:529344196client # copying path '/nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv' to 'lb:50051'...4197worker1 # [ 44.702748] nix-daemon[852]: accepted connection from pid 713, user nix-grpc-daemon (trusted)4198worker1 # [ 44.856739] nix-daemon[852]: accepted connection from pid 954, user nix-grpc-daemon (trusted)4199worker1 # [ 44.875479] nix-daemon[852]: reaped child process 957, status = succeeded4200worker1 # [ 44.978449] nix-daemon[852]: accepted connection from pid 959, user nix-grpc-daemon (trusted)4201worker1 # [ 44.994764] nix-daemon[852]: reaped child process 962, status = succeeded4202lb # [ 45.005102] niks3-server[979]: 2026/09/21 14:44:20 INFO Received uploads request method=POST path=/api/pending_closures4203worker1 # [ 45.011137] nix-grpc-daemon[866]: time=2026-09-21T14:44:20.619Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4204worker1 # [ 45.012676] nix-grpc-daemon[866]: time=2026-09-21T14:44:20.619Z level=INFO msg="Uploading czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv (752B)"4205lb # [ 45.039295] niks3-server[979]: 2026/09/21 14:44:20 INFO Registered completed upload object_key=nar/02rh0zwkcxgd7rc74r5h676rr0m8dbayk3glb776b2mdlirxh9lv.nar.zst4206worker1 # [ 45.047967] nix-grpc-daemon[866]: time=2026-09-21T14:44:20.656Z level=INFO msg="Uploading 1 narinfos"4207lb # [ 45.054586] niks3-server[979]: 2026/09/21 14:44:20 INFO Received sign narinfos request method=POST path=/api/pending_closures/5/sign4208lb # [ 45.064711] niks3-server[979]: 2026/09/21 14:44:20 INFO Signed narinfos id=5 count=14209lb # [ 45.069998] niks3-server[979]: 2026/09/21 14:44:20 INFO Registered completed upload object_key=czd6ljynjaw3bl3w8iz1nx68whkdis8f.ls4210worker1 # [ 45.077203] nix-grpc-daemon[866]: time=2026-09-21T14:44:20.685Z level=INFO msg="Upload complete. (356ms)"4211lb # [ 45.081386] niks3-server[979]: 2026/09/21 14:44:20 INFO Received complete upload request method=POST path=/api/pending_closures/5/complete4212lb # [ 45.086415] niks3-server[979]: 2026/09/21 14:44:20 INFO Completed upload id=54213worker1 # [ 45.084221] nix-grpc-daemon[713]: ts=2026-09-21T14:44:20Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52934 duration_s=0 paths=1 nar_bytes_in=7524214lb # [ 45.090404] niks3-server[979]: 2026/09/21 14:44:20 INFO Registered completed upload object_key=czd6ljynjaw3bl3w8iz1nx68whkdis8f.narinfo4215worker1 # [ 45.097486] nix-grpc-daemon[713]: ts=2026-09-21T14:44:20Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:529344216worker1 # [ 45.103540] nix-daemon[852]: accepted connection from pid 713, user nix-grpc-daemon (trusted)4217worker1 # [ 45.108832] nix-daemon[852]: reaped child process 952, status = succeeded4218client # building '/nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv' on lb:50051...4219client # farm-top> node-a: building czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv4220worker1 # [ 45.123538] nix-daemon[852]: accepted connection from pid 713, user nix-grpc-daemon (trusted)4221client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds4222client # farm-top> LOG-farm-top-t14223worker1 # [ 46.008956] nix-daemon[852]: reaped child process 966, status = succeeded4224worker1 # [ 46.110598] nix-daemon[852]: accepted connection from pid 982, user nix-grpc-daemon (trusted)4225worker1 # [ 46.131298] nix-daemon[852]: reaped child process 985, status = succeeded4226lb # [ 46.139890] niks3-server[979]: 2026/09/21 14:44:21 INFO Received uploads request method=POST path=/api/pending_closures4227worker1 # [ 46.149936] nix-grpc-daemon[866]: time=2026-09-21T14:44:21.756Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4228worker1 # [ 46.151846] nix-grpc-daemon[866]: time=2026-09-21T14:44:21.756Z level=INFO msg="Uploading nblg066ygaz2avzlmix3rnngr62hld8f-farm-top (232B)"4229lb # [ 46.182314] niks3-server[979]: 2026/09/21 14:44:21 INFO Registered completed upload object_key=log/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv4230worker1 # [ 46.195779] nix-grpc-daemon[866]: time=2026-09-21T14:44:21.803Z level=INFO msg="Uploading 1 narinfos"4231lb # [ 46.200096] niks3-server[979]: 2026/09/21 14:44:21 INFO Received sign narinfos request method=POST path=/api/pending_closures/6/sign4232lb # [ 46.203086] niks3-server[979]: 2026/09/21 14:44:21 INFO Signed narinfos id=6 count=14233lb # [ 46.211961] niks3-server[979]: 2026/09/21 14:44:21 INFO Registered completed upload object_key=nar/1wplyx8g7zja504cjvqmxis1q6laml9bidgiq4ya0i9p2pvqb9qp.nar.zst4234lb # [ 46.218688] niks3-server[979]: 2026/09/21 14:44:21 INFO Registered completed upload object_key=nblg066ygaz2avzlmix3rnngr62hld8f.ls4235lb # [ 46.225352] niks3-server[979]: 2026/09/21 14:44:21 INFO Received complete upload request method=POST path=/api/pending_closures/6/complete4236lb # [ 46.229013] niks3-server[979]: 2026/09/21 14:44:21 INFO Completed upload id=64237worker1 # [ 46.224897] nix-grpc-daemon[866]: time=2026-09-21T14:44:21.833Z level=INFO msg="Upload complete. (290ms)"4238lb # [ 46.231800] niks3-server[979]: 2026/09/21 14:44:21 INFO Registered completed upload object_key=nblg066ygaz2avzlmix3rnngr62hld8f.narinfo4239worker1 # [ 46.237241] nix-grpc-daemon[713]: ts=2026-09-21T14:44:21Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52934 duration_s=1 drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv assign_id=3 outputs=14240worker1 # [ 46.256240] nix-daemon[852]: reaped child process 964, status = succeeded4241client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 >&2, in 6.25 seconds)4242client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 outPath4243client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 outPath, in 0.18 seconds)4244client: must fail: test -e /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top4245client: (finished: must fail: test -e /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.02 seconds)4246client: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top && grep farm-top-t1 /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top4247client # copying 3 paths...4248client # copying path '/nix/store/dg404zlvxfrspnxl6cc1qkc6i83m5a3a-farm-a' from 'http://lb:5751'...4249client # copying path '/nix/store/8n9g36vw9957r150a7wg1k6yj9fbqm8r-farm-b' from 'http://lb:5751'...4250client # copying path '/nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top' from 'http://lb:5751'...4251client: (finished: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top && grep farm-top-t1 /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.22 seconds)4252(finished: subtest: DAG build is scheduled across workers and lands in the cache, in 6.77 seconds)4253subtest: repeat is answered Cached by the scheduler without building4254worker1: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top4255lb # [ 46.884880] envoy[888]: conn peer=[2001:db8:1::1]:50630 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=732 tx=407 ms=27514256lb # [ 46.889833] envoy[888]: conn peer=[2001:db8:1::1]:50634 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=2596 tx=364 ms=744257lb # [ 46.894372] envoy[888]: conn peer=[2001:db8:1::1]:50646 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=423 tx=198 ms=154258lb # [ 46.899239] envoy[888]: conn peer=[2001:db8:1::1]:50656 tls=TLSv1.3 subject="CN=stranger" sni=- flags=- tls_fail="-" rx=423 tx=181 ms=134259lb # [ 46.903926] envoy[888]: conn peer=[2001:db8:1::1]:50672 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9562 tx=4074 ms=60714260worker1 # finding garbage collector roots...4261worker1 # deleting specified paths...4262worker1 # deleting '/nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top'4263worker1 # deleting unused links...4264worker1 # note: hard linking is currently saving 0.0 KiB4265worker1: (finished: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.22 seconds)4266worker2: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top4267worker2 # finding garbage collector roots...4268worker2 # deleting specified paths...4269worker2 # deleting '/nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top'4270worker2 # deleting unused links...4271worker2 # note: hard linking is currently saving 0.0 KiB4272worker2: (finished: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.24 seconds)4273client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 >&24274client # don't know how to build these paths:4275client # /nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv4276worker2 # [ 47.468627] nix-grpc-daemon[770]: ts=2026-09-21T14:44:23Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:453964277worker1 # [ 47.493341] nix-grpc-daemon[713]: ts=2026-09-21T14:44:23Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:529344278worker1 # [ 47.506771] nix-grpc-daemon[866]: time=2026-09-21T14:44:23.114Z level=INFO msg="All 2 paths already cached"4279worker1 # [ 47.523526] nix-grpc-daemon[713]: ts=2026-09-21T14:44:23Z level=debug event=want client=2 drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv cached=14280client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 >&2, in 0.34 seconds)4281client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 outPath4282client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 outPath, in 0.18 seconds)4283lb: must succeed: curl -sf localhost:9901/clusters4284lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)4285lb: must succeed: curl -sf localhost:9901/clusters4286lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4287worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4288worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)4289(finished: subtest: repeat is answered Cached by the scheduler without building, in 1.25 seconds)4290subtest: inputs already in the cache are not sent to the scheduler4291lb: must succeed: curl -sf localhost:9901/clusters4292lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)4293lb: must succeed: curl -sf localhost:9901/clusters4294lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4295worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4296worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.06 seconds)4297client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 --argstr top t1b >&24298client # don't know how to build these paths:4299client # /nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv4300worker2 # [ 48.413614] nix-grpc-daemon[770]: ts=2026-09-21T14:44:24Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:453964301worker1 # [ 48.463317] nix-grpc-daemon[713]: ts=2026-09-21T14:44:24Z level=debug event=want client=3 drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv cached=04302worker1 # [ 48.470433] nix-grpc-daemon[713]: ts=2026-09-21T14:44:24Z level=debug event=assigned drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv worker=192.168.1.3:50051 clients=14303worker1 # [ 48.510074] nix-grpc-daemon[713]: ts=2026-09-21T14:44:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:529344304worker1 # [ 48.525311] nix-grpc-daemon[713]: ts=2026-09-21T14:44:24Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:529344305client # copying 1 paths to 'lb:50051'...4306worker1 # [ 48.541485] nix-grpc-daemon[866]: time=2026-09-21T14:44:24.148Z level=INFO msg="All 2 paths already cached"4307client # copying path '/nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv' to 'lb:50051'...4308worker1 # [ 48.546761] nix-grpc-daemon[713]: ts=2026-09-21T14:44:24Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:529344309worker1 # [ 48.561833] nix-daemon[852]: accepted connection from pid 713, user nix-grpc-daemon (trusted)4310worker1 # [ 48.722817] nix-daemon[852]: accepted connection from pid 1014, user nix-grpc-daemon (trusted)4311worker1 # [ 48.744441] nix-daemon[852]: reaped child process 1017, status = succeeded4312worker1 # [ 48.840951] nix-daemon[852]: accepted connection from pid 1019, user nix-grpc-daemon (trusted)4313worker1 # [ 48.858164] nix-daemon[852]: reaped child process 1022, status = succeeded4314lb # [ 48.870624] niks3-server[979]: 2026/09/21 14:44:24 INFO Received uploads request method=POST path=/api/pending_closures4315worker1 # [ 48.876361] nix-grpc-daemon[866]: time=2026-09-21T14:44:24.484Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4316worker1 # [ 48.877860] nix-grpc-daemon[866]: time=2026-09-21T14:44:24.484Z level=INFO msg="Uploading blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv (752B)"4317lb # [ 48.903240] niks3-server[979]: 2026/09/21 14:44:24 INFO Registered completed upload object_key=nar/1vy3ds565pni9mnkldcs6alimlb6bg39cimm60s4swkk5yjhwlqf.nar.zst4318worker1 # [ 48.910041] nix-grpc-daemon[866]: time=2026-09-21T14:44:24.518Z level=INFO msg="Uploading 1 narinfos"4319lb # [ 48.919005] niks3-server[979]: 2026/09/21 14:44:24 INFO Received sign narinfos request method=POST path=/api/pending_closures/7/sign4320lb # [ 48.928372] niks3-server[979]: 2026/09/21 14:44:24 INFO Signed narinfos id=7 count=14321lb # [ 48.937058] niks3-server[979]: 2026/09/21 14:44:24 INFO Registered completed upload object_key=blgxcsidls7na1m5da3m0bja1prb87vl.ls4322worker1 # [ 48.937075] nix-grpc-daemon[866]: time=2026-09-21T14:44:24.545Z level=INFO msg="Upload complete. (358ms)"4323lb # [ 48.946010] niks3-server[979]: 2026/09/21 14:44:24 INFO Received complete upload request method=POST path=/api/pending_closures/7/complete4324worker1 # [ 48.944132] nix-grpc-daemon[713]: ts=2026-09-21T14:44:24Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52934 duration_s=0 paths=1 nar_bytes_in=7524325lb # [ 48.953421] niks3-server[979]: 2026/09/21 14:44:24 INFO Completed upload id=74326lb # [ 48.955947] niks3-server[979]: 2026/09/21 14:44:24 INFO Registered completed upload object_key=blgxcsidls7na1m5da3m0bja1prb87vl.narinfo4327worker1 # [ 48.955943] nix-grpc-daemon[713]: ts=2026-09-21T14:44:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:529344328worker1 # [ 48.961934] nix-daemon[852]: accepted connection from pid 713, user nix-grpc-daemon (trusted)4329worker1 # [ 48.966462] nix-daemon[852]: reaped child process 1012, status = succeeded4330client # building '/nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv' on lb:50051...4331client # farm-top> node-a: building blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv4332worker1 # [ 48.979781] nix-daemon[852]: accepted connection from pid 713, user nix-grpc-daemon (trusted)4333worker2 # [ 48.421549] nix-grpc-daemon[770]: ts=2026-09-21T14:44:24Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:453964334client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds4335client # farm-top> LOG-farm-top-t1b4336worker1 # [ 50.280528] nix-daemon[852]: reaped child process 1026, status = succeeded4337worker1 # [ 50.356391] nix-daemon[852]: accepted connection from pid 1043, user nix-grpc-daemon (trusted)4338worker1 # [ 50.375765] nix-daemon[852]: reaped child process 1046, status = succeeded4339lb # [ 50.387387] niks3-server[979]: 2026/09/21 14:44:25 INFO Received uploads request method=POST path=/api/pending_closures4340worker1 # [ 50.395416] nix-grpc-daemon[866]: time=2026-09-21T14:44:26.001Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4341worker1 # [ 50.397239] nix-grpc-daemon[866]: time=2026-09-21T14:44:26.001Z level=INFO msg="Uploading 0xnysbbdbmapgf23wb2dbsfhgh18yxvp-farm-top (232B)"4342lb # [ 50.419506] niks3-server[979]: 2026/09/21 14:44:26 INFO Registered completed upload object_key=log/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv4343worker1 # [ 50.434955] nix-grpc-daemon[866]: time=2026-09-21T14:44:26.043Z level=INFO msg="Uploading 1 narinfos"4344lb # [ 50.439537] niks3-server[979]: 2026/09/21 14:44:26 INFO Received sign narinfos request method=POST path=/api/pending_closures/8/sign4345lb # [ 50.450690] niks3-server[979]: 2026/09/21 14:44:26 INFO Signed narinfos id=8 count=14346lb # [ 50.455533] niks3-server[979]: 2026/09/21 14:44:26 INFO Registered completed upload object_key=nar/1f6pjd04v1xmbxqqjhczkkfk4ns9qg6ssnspdalp41q9hkxf9yfr.nar.zst4347lb # [ 50.461303] niks3-server[979]: 2026/09/21 14:44:26 INFO Registered completed upload object_key=0xnysbbdbmapgf23wb2dbsfhgh18yxvp.ls4348lb # [ 50.471733] niks3-server[979]: 2026/09/21 14:44:26 INFO Received complete upload request method=POST path=/api/pending_closures/8/complete4349lb # [ 50.478769] niks3-server[979]: 2026/09/21 14:44:26 INFO Completed upload id=84350worker1 # [ 50.473733] nix-grpc-daemon[866]: time=2026-09-21T14:44:26.082Z level=INFO msg="Upload complete. (281ms)"4351lb # [ 50.481945] niks3-server[979]: 2026/09/21 14:44:26 INFO Registered completed upload object_key=0xnysbbdbmapgf23wb2dbsfhgh18yxvp.narinfo4352worker1 # [ 50.489158] nix-grpc-daemon[713]: ts=2026-09-21T14:44:26Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52934 duration_s=1 drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv assign_id=4 outputs=14353worker1 # [ 50.504873] nix-daemon[852]: reaped child process 1024, status = succeeded4354client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 --argstr top t1b >&2, in 2.30 seconds)4355client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 --argstr top t1b outPath4356client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 --argstr top t1b outPath, in 0.18 seconds)4357lb: must succeed: curl -sf localhost:9901/clusters4358lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)4359lb: must succeed: curl -sf localhost:9901/clusters4360lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4361worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4362worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.06 seconds)4363(finished: subtest: inputs already in the cache are not sent to the scheduler, in 2.83 seconds)4364subtest: two clients wanting the same drv build it once4365client: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup4366client # [ 50.945983] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup.4367client # Running as unit: dedup1.service; invocation ID: c7f676b3c6224775932eacc5964babd44368client: (finished: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup, in 0.09 seconds)4369client: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup4370client # [ 51.133646] nix[928]: don't know how to build these paths:4371client # [ 51.134851] nix[928]: /nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv4372worker1 # [ 51.135323] nix-grpc-daemon[713]: ts=2026-09-21T14:44:26Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:529344373worker1 # [ 51.148250] nix-grpc-daemon[713]: ts=2026-09-21T14:44:26Z level=debug event=want client=4 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv cached=04374worker1 # [ 51.154111] nix-grpc-daemon[713]: ts=2026-09-21T14:44:26Z level=debug event=assigned drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv worker=192.168.1.4:50051 clients=14375worker2 # [ 51.138545] nix-grpc-daemon[770]: ts=2026-09-21T14:44:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:453964376worker2 # [ 51.155746] nix-grpc-daemon[770]: ts=2026-09-21T14:44:26Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:453964377client # [ 51.187259] nix[928]: copying 1 paths to 'lb:50051'...4378worker2 # [ 51.170882] nix-grpc-daemon[770]: ts=2026-09-21T14:44:26Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:453964379client # [ 51.189886] nix[928]: copying path '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' to 'lb:50051'...4380worker2 # [ 51.185286] nix-daemon[838]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4381worker2 # [ 51.212124] nix-grpc-daemon[770]: ts=2026-09-21T14:44:26Z level=info event=niks3_push_started pid=8664382worker2 # [ 51.379362] nix-grpc-daemon[866]: time=2026-09-21T14:44:26.981Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.key ca=""4383worker2 # [ 51.488478] nix-daemon[838]: accepted connection from pid 879, user nix-grpc-daemon (trusted)4384worker2 # [ 51.509257] nix-daemon[838]: reaped child process 882, status = succeeded4385worker2 # [ 51.615281] nix-daemon[838]: accepted connection from pid 884, user nix-grpc-daemon (trusted)4386worker2 # [ 51.631413] nix-daemon[838]: reaped child process 887, status = succeeded4387lb # [ 51.665966] niks3-server[979]: 2026/09/21 14:44:27 INFO Received uploads request method=POST path=/api/pending_closures4388worker2 # [ 51.650028] nix-grpc-daemon[866]: time=2026-09-21T14:44:27.251Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4389worker2 # [ 51.651510] nix-grpc-daemon[866]: time=2026-09-21T14:44:27.251Z level=INFO msg="Uploading cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv (432B)"4390lb # [ 51.705242] niks3-server[979]: 2026/09/21 14:44:27 INFO Registered completed upload object_key=nar/0l7vpaziqxpw0srrsgimqvaa0dyxa9glicn3ilkwk06l64hn837w.nar.zst4391worker2 # [ 51.692070] nix-grpc-daemon[866]: time=2026-09-21T14:44:27.294Z level=INFO msg="Uploading 1 narinfos"4392lb # [ 51.719873] niks3-server[979]: 2026/09/21 14:44:27 INFO Received sign narinfos request method=POST path=/api/pending_closures/9/sign4393lb # [ 51.729511] niks3-server[979]: 2026/09/21 14:44:27 INFO Signed narinfos id=9 count=14394lb # [ 51.738408] niks3-server[979]: 2026/09/21 14:44:27 INFO Registered completed upload object_key=cp23830c44gs2dw4xy80s5wqyjm7vw84.ls4395worker2 # [ 51.719817] nix-grpc-daemon[866]: time=2026-09-21T14:44:27.322Z level=INFO msg="Upload complete. (336ms)"4396lb # [ 51.744617] niks3-server[979]: 2026/09/21 14:44:27 INFO Received complete upload request method=POST path=/api/pending_closures/9/complete4397lb # [ 51.750325] niks3-server[979]: 2026/09/21 14:44:27 INFO Completed upload id=94398worker2 # [ 51.728257] nix-grpc-daemon[770]: ts=2026-09-21T14:44:27Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:45396 duration_s=0 paths=1 nar_bytes_in=4324399lb # [ 51.756195] niks3-server[979]: 2026/09/21 14:44:27 INFO Registered completed upload object_key=cp23830c44gs2dw4xy80s5wqyjm7vw84.narinfo4400worker2 # [ 51.739371] nix-grpc-daemon[770]: ts=2026-09-21T14:44:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:453964401worker2 # [ 51.746056] nix-daemon[838]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4402worker2 # [ 51.750691] nix-daemon[838]: reaped child process 864, status = succeeded4403client # [ 51.778352] nix[928]: building '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' on lb:50051...4404worker2 # [ 51.763894] nix-daemon[838]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4405client # [ 52.042858] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup.4406client # Running as unit: dedup2.service; invocation ID: eabf0abde3d9437884e2d14c419fb5c14407client: (finished: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup, in 1.11 seconds)4408client # [ 52.290282] nix[944]: don't know how to build these paths:4409worker1 # [ 52.289298] nix-grpc-daemon[713]: ts=2026-09-21T14:44:27Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:529344410client # [ 52.291439] nix[944]: /nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv4411worker1 # [ 52.302625] nix-grpc-daemon[713]: ts=2026-09-21T14:44:27Z level=debug event=want client=5 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv cached=04412worker2 # [ 52.288436] nix-grpc-daemon[770]: ts=2026-09-21T14:44:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:453964413client # [ 52.309641] nix[944]: building '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' on lb:50051...4414client: must succeed: sleep 24415client: (finished: must succeed: sleep 2, in 2.05 seconds)4416client: waiting for success: ! systemctl is-active dedup1 dedup24417worker2 # [ 55.595822] nix-daemon[838]: reaped child process 891, status = succeeded4418worker2 # [ 55.687582] nix-daemon[838]: accepted connection from pid 921, user nix-grpc-daemon (trusted)4419worker2 # [ 55.708871] nix-daemon[838]: reaped child process 924, status = succeeded4420lb # [ 55.737275] niks3-server[979]: 2026/09/21 14:44:31 INFO Received uploads request method=POST path=/api/pending_closures4421worker2 # [ 55.724415] nix-grpc-daemon[866]: time=2026-09-21T14:44:31.325Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4422worker2 # [ 55.725952] nix-grpc-daemon[866]: time=2026-09-21T14:44:31.325Z level=INFO msg="Uploading vamz940my4dma9a2daklr9pw9x9pp9hp-slow-dedup (120B)"4423lb # [ 55.782998] niks3-server[979]: 2026/09/21 14:44:31 INFO Registered completed upload object_key=log/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv4424lb # [ 55.794817] niks3-server[979]: 2026/09/21 14:44:31 INFO Registered completed upload object_key=nar/02xhr0h0sj1ji8m0s8995xzbbb8mhbqdpdl1ggx94yqqaykihvz7.nar.zst4425worker2 # [ 55.780723] nix-grpc-daemon[866]: time=2026-09-21T14:44:31.382Z level=INFO msg="Uploading 1 narinfos"4426lb # [ 55.809510] niks3-server[979]: 2026/09/21 14:44:31 INFO Received sign narinfos request method=POST path=/api/pending_closures/10/sign4427lb # [ 55.815992] niks3-server[979]: 2026/09/21 14:44:31 INFO Signed narinfos id=10 count=14428lb # [ 55.820868] niks3-server[979]: 2026/09/21 14:44:31 INFO Registered completed upload object_key=vamz940my4dma9a2daklr9pw9x9pp9hp.ls4429worker2 # [ 55.807265] nix-grpc-daemon[866]: time=2026-09-21T14:44:31.410Z level=INFO msg="Upload complete. (231ms)"4430lb # [ 55.832973] niks3-server[979]: 2026/09/21 14:44:31 INFO Received complete upload request method=POST path=/api/pending_closures/10/complete4431lb # [ 55.842222] niks3-server[979]: 2026/09/21 14:44:31 INFO Completed upload id=104432worker2 # [ 55.816907] nix-grpc-daemon[770]: ts=2026-09-21T14:44:31Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:45396 duration_s=3 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv assign_id=5 outputs=14433lb # [ 55.848961] niks3-server[979]: 2026/09/21 14:44:31 INFO Registered completed upload object_key=vamz940my4dma9a2daklr9pw9x9pp9hp.narinfo4434worker2 # [ 55.833010] nix-grpc-daemon[770]: ts=2026-09-21T14:44:31Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:45396 duration_s=4 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv assign_id=5 outputs=14435worker2 # [ 55.838984] nix-daemon[838]: reaped child process 889, status = succeeded4436client # [ 55.870514] systemd[1]: dedup2.service: Deactivated successfully.4437client # [ 55.875316] systemd[1]: dedup2.service: Consumed 219ms CPU time over 3.829s wall clock time, 33.3M memory peak, 5.8K incoming IP traffic, 5.7K outgoing IP traffic.4438client # [ 55.923531] systemd[1]: dedup1.service: Deactivated successfully.4439client # [ 55.926870] systemd[1]: dedup1.service: Consumed 169ms CPU time over 4.982s wall clock time, 43.5M memory peak, 7.2K incoming IP traffic, 8.4K outgoing IP traffic.4440client: (finished: waiting for success: ! systemctl is-active dedup1 dedup2, in 1.12 seconds)4441client: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }4442client: (finished: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }, in 0.06 seconds)4443worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true4444worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.07 seconds)4445worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true4446lb # [ 56.885494] envoy[888]: conn peer=[2001:db8:1::1]:37458 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=993 tx=480 ms=564447lb # [ 56.892421] envoy[888]: conn peer=[2001:db8:1::1]:37462 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4198 tx=1783 ms=20644448lb # [ 56.897296] envoy[888]: conn peer=[2001:db8:1::1]:54232 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1430 tx=1085 ms=35694449lb # [ 56.902984] envoy[888]: conn peer=[2001:db8:1::1]:54218 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3005 tx=1453 ms=47644450worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.07 seconds)4451worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4452worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)4453worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4454worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)4455(finished: subtest: two clients wanting the same drv build it once, in 6.15 seconds)4456subtest: build hook: nix-daemon with builders = grpc://lb4457client: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --print-out-paths --no-link -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag hook 2>/tmp/hook.log4458client # [ 57.204482] systemd[1]: Started Nix Daemon.4459client # [ 57.320544] nix-daemon[986]: accepted connection from pid 983, user root (trusted)4460worker2 # [ 57.453182] nix-grpc-daemon[770]: ts=2026-09-21T14:44:33Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:453964461worker2 # [ 57.459444] nix-grpc-daemon[770]: ts=2026-09-21T14:44:33Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:453964462worker1 # [ 57.501263] nix-grpc-daemon[713]: ts=2026-09-21T14:44:33Z level=debug event=want client=6 drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv cached=04463worker1 # [ 57.501575] nix-grpc-daemon[713]: ts=2026-09-21T14:44:33Z level=debug event=assigned drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv worker=192.168.1.3:50051 clients=14464worker1 # [ 57.544216] nix-grpc-daemon[713]: ts=2026-09-21T14:44:33Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:529344465worker1 # [ 57.560430] nix-grpc-daemon[713]: ts=2026-09-21T14:44:33Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:529344466worker1 # [ 57.575495] nix-grpc-daemon[713]: ts=2026-09-21T14:44:33Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:529344467worker1 # [ 57.590444] nix-daemon[852]: accepted connection from pid 713, user nix-grpc-daemon (trusted)4468worker1 # [ 57.758956] nix-daemon[852]: accepted connection from pid 1082, user nix-grpc-daemon (trusted)4469worker1 # [ 57.782349] nix-daemon[852]: reaped child process 1085, status = succeeded4470worker1 # [ 57.877527] nix-daemon[852]: accepted connection from pid 1087, user nix-grpc-daemon (trusted)4471worker1 # [ 57.894160] nix-daemon[852]: reaped child process 1090, status = succeeded4472lb # [ 57.905712] niks3-server[979]: 2026/09/21 14:44:33 INFO Received uploads request method=POST path=/api/pending_closures4473worker1 # [ 57.911784] nix-grpc-daemon[866]: time=2026-09-21T14:44:33.519Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4474worker1 # [ 57.913410] nix-grpc-daemon[866]: time=2026-09-21T14:44:33.519Z level=INFO msg="Uploading xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv (512B)"4475lb # [ 57.945427] niks3-server[979]: 2026/09/21 14:44:33 INFO Registered completed upload object_key=nar/1f1i5kab3qhx7pg1k0ma5r1m7ipfazkfkfb944dm7qcjr466pwij.nar.zst4476worker1 # [ 57.953863] nix-grpc-daemon[866]: time=2026-09-21T14:44:33.562Z level=INFO msg="Uploading 1 narinfos"4477lb # [ 57.959286] niks3-server[979]: 2026/09/21 14:44:33 INFO Received sign narinfos request method=POST path=/api/pending_closures/11/sign4478lb # [ 57.969276] niks3-server[979]: 2026/09/21 14:44:33 INFO Signed narinfos id=11 count=14479lb # [ 57.974505] niks3-server[979]: 2026/09/21 14:44:33 INFO Registered completed upload object_key=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv.ls4480lb # [ 57.980559] niks3-server[979]: 2026/09/21 14:44:33 INFO Received complete upload request method=POST path=/api/pending_closures/11/complete4481lb # [ 57.989389] niks3-server[979]: 2026/09/21 14:44:33 INFO Completed upload id=114482worker1 # [ 57.984654] nix-grpc-daemon[866]: time=2026-09-21T14:44:33.593Z level=INFO msg="Upload complete. (377ms)"4483lb # [ 57.991860] niks3-server[979]: 2026/09/21 14:44:33 INFO Registered completed upload object_key=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv.narinfo4484worker1 # [ 57.994167] nix-grpc-daemon[713]: ts=2026-09-21T14:44:33Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52934 duration_s=0 paths=1 nar_bytes_in=5124485worker1 # [ 58.012450] nix-grpc-daemon[713]: ts=2026-09-21T14:44:33Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:529344486worker1 # [ 58.022521] nix-daemon[852]: accepted connection from pid 713, user nix-grpc-daemon (trusted)4487worker1 # [ 58.028859] nix-daemon[852]: reaped child process 1080, status = succeeded4488worker1 # [ 58.042298] nix-daemon[852]: accepted connection from pid 713, user nix-grpc-daemon (trusted)4489worker1 # [ 58.940386] nix-daemon[852]: reaped child process 1094, status = succeeded4490worker1 # [ 59.007241] nix-daemon[852]: accepted connection from pid 1110, user nix-grpc-daemon (trusted)4491worker1 # [ 59.025047] nix-daemon[852]: reaped child process 1113, status = succeeded4492lb # [ 59.038398] niks3-server[979]: 2026/09/21 14:44:34 INFO Received uploads request method=POST path=/api/pending_closures4493worker1 # [ 59.044438] nix-grpc-daemon[866]: time=2026-09-21T14:44:34.649Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4494worker1 # [ 59.046063] nix-grpc-daemon[866]: time=2026-09-21T14:44:34.649Z level=INFO msg="Uploading yfs64sxmsg77fnzczdvc3g3yfdwh9fks-farm-a (128B)"4495lb # [ 59.072151] niks3-server[979]: 2026/09/21 14:44:34 INFO Registered completed upload object_key=log/xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv4496lb # [ 59.086861] niks3-server[979]: 2026/09/21 14:44:34 INFO Registered completed upload object_key=nar/0pbbxl4kxy7zl7x4lp43ywljwn82mxzna2wgpb1bz1z765nwp9ki.nar.zst4497worker1 # [ 59.097899] nix-grpc-daemon[866]: time=2026-09-21T14:44:34.706Z level=INFO msg="Uploading 1 narinfos"4498lb # [ 59.101698] niks3-server[979]: 2026/09/21 14:44:34 INFO Received sign narinfos request method=POST path=/api/pending_closures/12/sign4499lb # [ 59.114058] niks3-server[979]: 2026/09/21 14:44:34 INFO Signed narinfos id=12 count=14500lb # [ 59.121530] niks3-server[979]: 2026/09/21 14:44:34 INFO Registered completed upload object_key=yfs64sxmsg77fnzczdvc3g3yfdwh9fks.ls4501lb # [ 59.133545] niks3-server[979]: 2026/09/21 14:44:34 INFO Received complete upload request method=POST path=/api/pending_closures/12/complete4502lb # [ 59.140722] niks3-server[979]: 2026/09/21 14:44:34 INFO Completed upload id=124503worker1 # [ 59.135775] nix-grpc-daemon[866]: time=2026-09-21T14:44:34.744Z level=INFO msg="Upload complete. (279ms)"4504lb # [ 59.144065] niks3-server[979]: 2026/09/21 14:44:34 INFO Registered completed upload object_key=yfs64sxmsg77fnzczdvc3g3yfdwh9fks.narinfo4505worker1 # [ 59.147656] nix-grpc-daemon[713]: ts=2026-09-21T14:44:34Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52934 duration_s=1 drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv assign_id=6 outputs=14506worker1 # [ 59.166729] nix-daemon[852]: reaped child process 1092, status = succeeded4507worker1 # [ 59.192820] nix-grpc-daemon[713]: ts=2026-09-21T14:44:34Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:529344508worker2 # [ 59.195947] nix-grpc-daemon[770]: ts=2026-09-21T14:44:34Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:453964509worker2 # [ 59.206113] nix-grpc-daemon[770]: copying path '/nix/store/yfs64sxmsg77fnzczdvc3g3yfdwh9fks-farm-a' from 'http://lb:5751'...4510worker2 # [ 59.224307] nix-grpc-daemon[770]: ts=2026-09-21T14:44:34Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:45396 duration_s=0 paths=1 nar_bytes_out=1284511worker1 # [ 59.533771] nix-grpc-daemon[713]: ts=2026-09-21T14:44:35Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:529344512worker2 # [ 59.518046] nix-grpc-daemon[770]: ts=2026-09-21T14:44:35Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:453964513worker2 # [ 59.528745] nix-grpc-daemon[866]: time=2026-09-21T14:44:35.130Z level=INFO msg="All 1 paths already cached"4514worker1 # [ 59.567175] nix-grpc-daemon[713]: ts=2026-09-21T14:44:35Z level=debug event=want client=7 drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv cached=04515worker1 # [ 59.574135] nix-grpc-daemon[713]: ts=2026-09-21T14:44:35Z level=debug event=assigned drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv worker=192.168.1.3:50051 clients=14516worker1 # [ 59.582922] nix-grpc-daemon[713]: ts=2026-09-21T14:44:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:529344517worker1 # [ 59.594307] nix-grpc-daemon[713]: ts=2026-09-21T14:44:35Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:529344518worker1 # [ 59.608769] nix-grpc-daemon[866]: time=2026-09-21T14:44:35.217Z level=INFO msg="All 1 paths already cached"4519worker1 # [ 59.617880] nix-grpc-daemon[713]: ts=2026-09-21T14:44:35Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:529344520worker1 # [ 59.630778] nix-daemon[852]: accepted connection from pid 713, user nix-grpc-daemon (trusted)4521worker1 # [ 59.786344] nix-daemon[852]: accepted connection from pid 1119, user nix-grpc-daemon (trusted)4522worker1 # [ 59.809365] nix-daemon[852]: reaped child process 1122, status = succeeded4523worker1 # [ 59.911720] nix-daemon[852]: accepted connection from pid 1124, user nix-grpc-daemon (trusted)4524worker1 # [ 59.931315] nix-daemon[852]: reaped child process 1127, status = succeeded4525lb # [ 59.939250] niks3-server[979]: 2026/09/21 14:44:35 INFO Received uploads request method=POST path=/api/pending_closures4526worker1 # [ 59.944898] nix-grpc-daemon[866]: time=2026-09-21T14:44:35.553Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4527worker1 # [ 59.946400] nix-grpc-daemon[866]: time=2026-09-21T14:44:35.553Z level=INFO msg="Uploading r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv (624B)"4528lb # [ 59.967591] niks3-server[979]: 2026/09/21 14:44:35 INFO Registered completed upload object_key=nar/15k66v3mh3v6p5vymv4pv3v3sfl56i2b1ia2rf093c37w6bhc04m.nar.zst4529lb # [ 59.982697] niks3-server[979]: 2026/09/21 14:44:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/13/sign4530worker1 # [ 59.986099] nix-grpc-daemon[866]: time=2026-09-21T14:44:35.594Z level=INFO msg="Uploading 1 narinfos"4531lb # [ 59.994694] niks3-server[979]: 2026/09/21 14:44:35 INFO Signed narinfos id=13 count=14532lb # [ 60.006306] niks3-server[979]: 2026/09/21 14:44:35 INFO Received complete upload request method=POST path=/api/pending_closures/13/complete4533lb # [ 60.022163] niks3-server[979]: 2026/09/21 14:44:35 INFO Registered completed upload object_key=r1jsr4b2g1qwly395z6idx3z2362ih8i.ls4534lb # [ 60.032350] niks3-server[979]: 2026/09/21 14:44:35 INFO Completed upload id=134535worker1 # [ 60.023773] nix-grpc-daemon[866]: time=2026-09-21T14:44:35.632Z level=INFO msg="Upload complete. (377ms)"4536worker1 # [ 60.032501] nix-grpc-daemon[713]: ts=2026-09-21T14:44:35Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52934 duration_s=0 paths=1 nar_bytes_in=6244537worker1 # [ 60.051863] nix-grpc-daemon[713]: ts=2026-09-21T14:44:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:529344538lb # [ 60.057227] niks3-server[979]: 2026/09/21 14:44:35 INFO Registered completed upload object_key=r1jsr4b2g1qwly395z6idx3z2362ih8i.narinfo4539worker1 # [ 60.060444] nix-daemon[852]: accepted connection from pid 713, user nix-grpc-daemon (trusted)4540worker1 # [ 60.068793] nix-daemon[852]: reaped child process 1117, status = succeeded4541worker1 # [ 60.082614] nix-daemon[852]: accepted connection from pid 713, user nix-grpc-daemon (trusted)4542worker1 # [ 60.907422] nix-daemon[852]: reaped child process 1131, status = succeeded4543worker1 # [ 61.016497] nix-daemon[852]: accepted connection from pid 1147, user nix-grpc-daemon (trusted)4544worker1 # [ 61.038261] nix-daemon[852]: reaped child process 1150, status = succeeded4545lb # [ 61.040191] niks3-server[979]: 2026/09/21 14:44:36 INFO Received uploads request method=POST path=/api/pending_closures4546worker1 # [ 61.051524] nix-grpc-daemon[866]: time=2026-09-21T14:44:36.658Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4547worker1 # [ 61.053092] nix-grpc-daemon[866]: time=2026-09-21T14:44:36.658Z level=INFO msg="Uploading la6ryzsihag4zpxgxls54fsg34dvlalk-farm-b (176B)"4548lb # [ 61.094179] niks3-server[979]: 2026/09/21 14:44:36 INFO Registered completed upload object_key=log/r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv4549worker1 # [ 61.100920] nix-grpc-daemon[866]: time=2026-09-21T14:44:36.709Z level=INFO msg="Uploading 1 narinfos"4550lb # [ 61.106287] niks3-server[979]: 2026/09/21 14:44:36 INFO Registered completed upload object_key=nar/069d0xnyw2q345iqzx6agxc5fxifc6q36bkkklw2aa4qwanxdd1g.nar.zst4551lb # [ 61.116550] niks3-server[979]: 2026/09/21 14:44:36 INFO Received sign narinfos request method=POST path=/api/pending_closures/14/sign4552lb # [ 61.119870] niks3-server[979]: 2026/09/21 14:44:36 INFO Signed narinfos id=14 count=14553lb # [ 61.134322] niks3-server[979]: 2026/09/21 14:44:36 INFO Registered completed upload object_key=la6ryzsihag4zpxgxls54fsg34dvlalk.ls4554lb # [ 61.139181] niks3-server[979]: 2026/09/21 14:44:36 INFO Received complete upload request method=POST path=/api/pending_closures/14/complete4555lb # [ 61.143812] niks3-server[979]: 2026/09/21 14:44:36 INFO Completed upload id=144556worker1 # [ 61.139845] nix-grpc-daemon[866]: time=2026-09-21T14:44:36.748Z level=INFO msg="Upload complete. (247ms)"4557lb # [ 61.146803] niks3-server[979]: 2026/09/21 14:44:36 INFO Registered completed upload object_key=la6ryzsihag4zpxgxls54fsg34dvlalk.narinfo4558worker1 # [ 61.153169] nix-grpc-daemon[713]: ts=2026-09-21T14:44:36Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52934 duration_s=1 drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv assign_id=7 outputs=14559worker1 # [ 61.172460] nix-grpc-daemon[713]: ts=2026-09-21T14:44:36Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:529344560worker1 # [ 61.179871] nix-daemon[852]: reaped child process 1129, status = succeeded4561worker2 # [ 61.169944] nix-grpc-daemon[770]: ts=2026-09-21T14:44:36Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:453964562worker2 # [ 61.182958] nix-grpc-daemon[770]: copying path '/nix/store/la6ryzsihag4zpxgxls54fsg34dvlalk-farm-b' from 'http://lb:5751'...4563worker2 # [ 61.202461] nix-grpc-daemon[770]: ts=2026-09-21T14:44:36Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:45396 duration_s=0 paths=1 nar_bytes_out=1764564worker1 # [ 61.506687] nix-grpc-daemon[713]: ts=2026-09-21T14:44:37Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:529344565worker2 # [ 61.489981] nix-grpc-daemon[770]: ts=2026-09-21T14:44:37Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:453964566worker2 # [ 61.500540] nix-grpc-daemon[866]: time=2026-09-21T14:44:37.102Z level=INFO msg="All 2 paths already cached"4567worker1 # [ 61.538513] nix-grpc-daemon[713]: ts=2026-09-21T14:44:37Z level=debug event=want client=8 drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv cached=04568worker1 # [ 61.545834] nix-grpc-daemon[713]: ts=2026-09-21T14:44:37Z level=debug event=assigned drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv worker=192.168.1.3:50051 clients=14569worker1 # [ 61.554670] nix-grpc-daemon[713]: ts=2026-09-21T14:44:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:529344570worker1 # [ 61.565961] nix-grpc-daemon[713]: ts=2026-09-21T14:44:37Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:529344571worker1 # [ 61.580260] nix-grpc-daemon[866]: time=2026-09-21T14:44:37.188Z level=INFO msg="All 2 paths already cached"4572worker1 # [ 61.589794] nix-grpc-daemon[713]: ts=2026-09-21T14:44:37Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:529344573worker1 # [ 61.604632] nix-daemon[852]: accepted connection from pid 713, user nix-grpc-daemon (trusted)4574worker1 # [ 61.754330] nix-daemon[852]: accepted connection from pid 1156, user nix-grpc-daemon (trusted)4575worker1 # [ 61.776734] nix-daemon[852]: reaped child process 1159, status = succeeded4576worker1 # [ 61.878113] nix-daemon[852]: accepted connection from pid 1161, user nix-grpc-daemon (trusted)4577worker1 # [ 61.898141] nix-daemon[852]: reaped child process 1164, status = succeeded4578lb # [ 61.905058] niks3-server[979]: 2026/09/21 14:44:37 INFO Received uploads request method=POST path=/api/pending_closures4579worker1 # [ 61.910100] nix-grpc-daemon[866]: time=2026-09-21T14:44:37.518Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4580worker1 # [ 61.911554] nix-grpc-daemon[866]: time=2026-09-21T14:44:37.518Z level=INFO msg="Uploading aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv (752B)"4581lb # [ 61.932503] niks3-server[979]: 2026/09/21 14:44:37 INFO Registered completed upload object_key=nar/1ahq7sp1sl2scxqzfjd8gqimpqfkjbyi9ivxdhdwx0jzh2gbv74b.nar.zst4582worker1 # [ 61.949956] nix-grpc-daemon[866]: time=2026-09-21T14:44:37.558Z level=INFO msg="Uploading 1 narinfos"4583lb # [ 61.946816] niks3-server[979]: 2026/09/21 14:44:37 INFO Received sign narinfos request method=POST path=/api/pending_closures/15/sign4584lb # [ 61.961088] niks3-server[979]: 2026/09/21 14:44:37 INFO Signed narinfos id=15 count=14585lb # [ 61.963784] niks3-server[979]: 2026/09/21 14:44:37 INFO Registered completed upload object_key=aa4abikjj4w30bfx1p8hxmq0916qbz1c.ls4586lb # [ 61.977581] niks3-server[979]: 2026/09/21 14:44:37 INFO Received complete upload request method=POST path=/api/pending_closures/15/complete4587worker1 # [ 61.981341] nix-grpc-daemon[866]: time=2026-09-21T14:44:37.589Z level=INFO msg="Upload complete. (360ms)"4588lb # [ 61.987241] niks3-server[979]: 2026/09/21 14:44:37 INFO Completed upload id=154589lb # [ 61.990758] niks3-server[979]: 2026/09/21 14:44:37 INFO Registered completed upload object_key=aa4abikjj4w30bfx1p8hxmq0916qbz1c.narinfo4590worker1 # [ 61.988265] nix-grpc-daemon[713]: ts=2026-09-21T14:44:37Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52934 duration_s=0 paths=1 nar_bytes_in=7524591worker1 # [ 62.002940] nix-grpc-daemon[713]: ts=2026-09-21T14:44:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:529344592worker1 # [ 62.009390] nix-daemon[852]: accepted connection from pid 713, user nix-grpc-daemon (trusted)4593worker1 # [ 62.014629] nix-daemon[852]: reaped child process 1154, status = succeeded4594worker1 # [ 62.025271] nix-daemon[852]: accepted connection from pid 713, user nix-grpc-daemon (trusted)4595worker1 # [ 63.262839] nix-daemon[852]: reaped child process 1168, status = succeeded4596worker1 # [ 63.357512] nix-daemon[852]: accepted connection from pid 1184, user nix-grpc-daemon (trusted)4597worker1 # [ 63.374422] nix-daemon[852]: reaped child process 1187, status = succeeded4598lb # [ 63.386756] niks3-server[979]: 2026/09/21 14:44:38 INFO Received uploads request method=POST path=/api/pending_closures4599worker1 # [ 63.392403] nix-grpc-daemon[866]: time=2026-09-21T14:44:39.000Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4600worker1 # [ 63.393883] nix-grpc-daemon[866]: time=2026-09-21T14:44:39.000Z level=INFO msg="Uploading 39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top (232B)"4601lb # [ 63.435944] niks3-server[979]: 2026/09/21 14:44:39 INFO Received sign narinfos request method=POST path=/api/pending_closures/16/sign4602lb # [ 63.445432] niks3-server[979]: 2026/09/21 14:44:39 INFO Signed narinfos id=16 count=14603worker1 # [ 63.441866] nix-grpc-daemon[866]: time=2026-09-21T14:44:39.050Z level=INFO msg="Uploading 1 narinfos"4604lb # [ 63.455779] niks3-server[979]: 2026/09/21 14:44:39 INFO Received complete upload request method=POST path=/api/pending_closures/16/complete4605lb # [ 63.533952] niks3-server[979]: 2026/09/21 14:44:39 INFO Registered completed upload object_key=nar/0vq3rhfhhbqyx5pbvaa9ddj9wbw9arph334y3bp3idfrikwnjc92.nar.zst4606lb # [ 63.548555] niks3-server[979]: 2026/09/21 14:44:39 INFO Registered completed upload object_key=log/aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv4607lb # [ 63.558060] niks3-server[979]: 2026/09/21 14:44:39 INFO Registered completed upload object_key=39qxwzy2zkij70y9zvbr59pb0df38w5h.ls4608worker1 # [ 63.560981] nix-grpc-daemon[866]: time=2026-09-21T14:44:39.169Z level=INFO msg="Upload complete. (324ms)"4609lb # [ 63.570227] niks3-server[979]: 2026/09/21 14:44:39 INFO Completed upload id=164610lb # [ 63.575949] niks3-server[979]: 2026/09/21 14:44:39 INFO Registered completed upload object_key=39qxwzy2zkij70y9zvbr59pb0df38w5h.narinfo4611worker1 # [ 63.577268] nix-grpc-daemon[713]: ts=2026-09-21T14:44:39Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52934 duration_s=1 drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv assign_id=8 outputs=14612worker1 # [ 63.595137] nix-grpc-daemon[713]: ts=2026-09-21T14:44:39Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:529344613worker1 # [ 63.600903] nix-daemon[852]: reaped child process 1166, status = succeeded4614worker1 # [ 63.614924] nix-grpc-daemon[713]: ts=2026-09-21T14:44:39Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:529344615worker1 # [ 63.620261] nix-grpc-daemon[713]: ts=2026-09-21T14:44:39Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:52934 duration_s=0 paths=1 nar_bytes_out=2324616client # [ 63.738428] nix-daemon[986]: reaped child process 988, status = succeeded4617client: (finished: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --print-out-paths --no-link -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag hook 2>/tmp/hook.log, in 6.73 seconds)4618client: must succeed: grep farm-top-hook /nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top4619client: (finished: must succeed: grep farm-top-hook /nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top, in 0.04 seconds)4620client: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }4621client: (finished: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }, in 0.05 seconds)4622client: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }4623client: (finished: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }, in 0.03 seconds)4624(finished: subtest: build hook: nix-daemon with builders = grpc://lb, in 6.85 seconds)4625subtest: an input only one worker has still reaches the builder4626client: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A input --argstr tag w1only4627client # this derivation will be built:4628client # /nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv4629client # building '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv'...4630client: (finished: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A input --argstr tag w1only, in 0.29 seconds)4631client: must succeed: nix-store --export /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only > /tmp/shared/inp.closure4632client: (finished: must succeed: nix-store --export /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only > /tmp/shared/inp.closure, in 0.12 seconds)4633worker1: must succeed: nix-store --import < /tmp/shared/inp.closure4634worker1: (finished: must succeed: nix-store --import < /tmp/shared/inp.closure, in 0.19 seconds)4635worker2: must fail: test -e /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only4636worker2: (finished: must fail: test -e /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only, in 0.03 seconds)4637client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt a >&24638client # [ 64.695621] nix-daemon[986]: accepted connection from pid 1069, user root (trusted)4639client # this derivation will be built:4640client # /nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv4641client # building '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' on 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux'...4642worker1 # [ 64.854286] nix-grpc-daemon[713]: ts=2026-09-21T14:44:40Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:529344643worker2 # [ 64.840365] nix-grpc-daemon[770]: ts=2026-09-21T14:44:40Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:453964644client # copying 1 paths...4645client # copying path '/nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only' to 'grpc://lb:50051'...4646worker1 # [ 64.871924] nix-grpc-daemon[713]: ts=2026-09-21T14:44:40Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:529344647worker1 # [ 64.886924] nix-daemon[852]: accepted connection from pid 713, user nix-grpc-daemon (trusted)4648worker1 # [ 65.050872] nix-daemon[852]: accepted connection from pid 1196, user nix-grpc-daemon (trusted)4649worker1 # [ 65.073614] nix-daemon[852]: reaped child process 1199, status = succeeded4650lb # [ 65.081249] niks3-server[979]: 2026/09/21 14:44:40 INFO Received uploads request method=POST path=/api/pending_closures4651worker1 # [ 65.086716] nix-grpc-daemon[866]: time=2026-09-21T14:44:40.694Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4652worker1 # [ 65.088469] nix-grpc-daemon[866]: time=2026-09-21T14:44:40.694Z level=INFO msg="Uploading dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only (120B)"4653lb # [ 65.108828] niks3-server[979]: 2026/09/21 14:44:40 INFO Registered completed upload object_key=nar/08ghl87f009pwqz85nbqv9hm0ln6gai6ggzilfmw3qnkaalfi6ss.nar.zst4654lb # [ 65.124683] niks3-server[979]: 2026/09/21 14:44:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/17/sign4655worker1 # [ 65.128942] nix-grpc-daemon[866]: time=2026-09-21T14:44:40.737Z level=INFO msg="Uploading 1 narinfos"4656lb # [ 65.137441] niks3-server[979]: 2026/09/21 14:44:40 INFO Signed narinfos id=17 count=14657lb # [ 65.145995] niks3-server[979]: 2026/09/21 14:44:40 INFO Received complete upload request method=POST path=/api/pending_closures/17/complete4658lb # [ 65.263977] niks3-server[979]: 2026/09/21 14:44:40 INFO Registered completed upload object_key=dymj0d1vj8q0ns6p096qk7fii766dap6.ls4659lb # [ 65.275490] niks3-server[979]: 2026/09/21 14:44:40 INFO Completed upload id=174660worker1 # [ 65.271705] nix-grpc-daemon[866]: time=2026-09-21T14:44:40.880Z level=INFO msg="Upload complete. (369ms)"4661lb # [ 65.284172] niks3-server[979]: 2026/09/21 14:44:40 INFO Registered completed upload object_key=dymj0d1vj8q0ns6p096qk7fii766dap6.narinfo4662worker1 # [ 65.284426] nix-grpc-daemon[713]: ts=2026-09-21T14:44:40Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52934 duration_s=0 paths=1 nar_bytes_in=1204663worker1 # [ 65.304337] nix-grpc-daemon[713]: ts=2026-09-21T14:44:40Z level=debug event=want client=9 drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv cached=04664worker1 # [ 65.311065] nix-grpc-daemon[713]: ts=2026-09-21T14:44:40Z level=debug event=assigned drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv worker=192.168.1.4:50051 clients=14665worker1 # [ 65.318123] nix-daemon[852]: reaped child process 1194, status = succeeded4666worker2 # [ 65.323846] nix-grpc-daemon[770]: ts=2026-09-21T14:44:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:453964667worker2 # [ 65.339552] nix-grpc-daemon[770]: ts=2026-09-21T14:44:40Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:453964668client # copying 2 paths to 'lb:50051'...4669worker2 # [ 65.358200] nix-grpc-daemon[770]: ts=2026-09-21T14:44:40Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:453964670client # copying path '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv' to 'lb:50051'...4671client # copying path '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' to 'lb:50051'...4672worker2 # [ 65.374512] nix-daemon[838]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4673worker2 # [ 65.549194] nix-daemon[838]: accepted connection from pid 942, user nix-grpc-daemon (trusted)4674worker2 # [ 65.568997] nix-daemon[838]: reaped child process 945, status = succeeded4675worker2 # [ 65.673306] nix-daemon[838]: accepted connection from pid 947, user nix-grpc-daemon (trusted)4676worker2 # [ 65.691050] nix-daemon[838]: reaped child process 950, status = succeeded4677lb # [ 65.721001] niks3-server[979]: 2026/09/21 14:44:41 INFO Received uploads request method=POST path=/api/pending_closures4678lb # [ 65.730827] niks3-server[979]: 2026/09/21 14:44:41 INFO Received uploads request method=POST path=/api/pending_closures4679worker2 # [ 65.716117] nix-grpc-daemon[866]: time=2026-09-21T14:44:41.318Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"4680worker2 # [ 65.717589] nix-grpc-daemon[866]: time=2026-09-21T14:44:41.318Z level=INFO msg="Uploading 4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv (416B)"4681worker2 # [ 65.719248] nix-grpc-daemon[866]: time=2026-09-21T14:44:41.318Z level=INFO msg="Uploading 71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv (568B)"4682lb # [ 65.763555] niks3-server[979]: 2026/09/21 14:44:41 INFO Registered completed upload object_key=nar/0i2vbprwp0wxa61nkss61gzivqmmjh6klkgv0rlfnxk46rcaba1v.nar.zst4683lb # [ 65.785036] niks3-server[979]: 2026/09/21 14:44:41 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.ls4684lb # [ 65.793668] niks3-server[979]: 2026/09/21 14:44:41 INFO Registered completed upload object_key=nar/0p6zgq1gdngshl62q0hv3lslqwv90s7fmflwc7wgnycf4ka6ik69.nar.zst4685lb # [ 65.802361] niks3-server[979]: 2026/09/21 14:44:41 INFO Received sign narinfos request method=POST path=/api/pending_closures/18/sign4686worker2 # [ 65.785897] nix-grpc-daemon[866]: time=2026-09-21T14:44:41.387Z level=INFO msg="Uploading 3 narinfos"4687lb # [ 65.813635] niks3-server[979]: 2026/09/21 14:44:41 INFO Signed narinfos id=18 count=14688lb # [ 65.819211] niks3-server[979]: 2026/09/21 14:44:41 INFO Received sign narinfos request method=POST path=/api/pending_closures/19/sign4689lb # [ 65.827378] niks3-server[979]: 2026/09/21 14:44:41 INFO Signed narinfos id=19 count=24690lb # [ 65.832730] niks3-server[979]: 2026/09/21 14:44:41 INFO Registered completed upload object_key=71l2x9mybp0i90r6pdnglwjnayj4xpl8.ls4691lb # [ 65.852524] niks3-server[979]: 2026/09/21 14:44:41 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.narinfo4692lb # [ 65.855214] niks3-server[979]: 2026/09/21 14:44:41 INFO Received complete upload request method=POST path=/api/pending_closures/18/complete4693lb # [ 65.861950] niks3-server[979]: 2026/09/21 14:44:41 INFO Completed upload id=184694lb # [ 65.863568] niks3-server[979]: 2026/09/21 14:44:41 INFO Received complete upload request method=POST path=/api/pending_closures/19/complete4695lb # [ 65.867534] niks3-server[979]: 2026/09/21 14:44:41 INFO Registered completed upload object_key=71l2x9mybp0i90r6pdnglwjnayj4xpl8.narinfo4696lb # [ 65.871272] niks3-server[979]: 2026/09/21 14:44:41 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.narinfo4697lb # [ 65.874448] niks3-server[979]: 2026/09/21 14:44:41 INFO Completed upload id=194698worker2 # [ 65.850264] nix-grpc-daemon[866]: time=2026-09-21T14:44:41.453Z level=INFO msg="Upload complete. (455ms)"4699worker2 # [ 65.862273] nix-grpc-daemon[770]: ts=2026-09-21T14:44:41Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:45396 duration_s=0 paths=2 nar_bytes_in=9844700worker2 # [ 65.881963] nix-grpc-daemon[770]: ts=2026-09-21T14:44:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:453964701worker2 # [ 65.893208] nix-daemon[838]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4702worker2 # [ 65.904364] nix-daemon[838]: reaped child process 940, status = succeeded4703worker2 # [ 65.917140] nix-grpc-daemon[770]: copying path '/nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only' from 'http://lb:5751'...4704client # building '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' on lb:50051...4705worker2 # [ 65.933877] nix-daemon[838]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4706client # uses-input-w1onlya> worker2: building 71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv4707client # uses-input-w1onlya> disabling binary cache 'https://cache.nixos.org' for 60 seconds4708worker2 # [ 66.769417] nix-daemon[838]: reaped child process 955, status = succeeded4709lb # [ 66.886591] envoy[888]: conn peer=[2001:db8:1::1]:54244 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3403 tx=1717 ms=18324710lb # [ 66.891534] envoy[888]: conn peer=[2001:db8:1::1]:54246 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=436 tx=334 ms=944711worker2 # [ 66.872594] nix-daemon[838]: accepted connection from pid 971, user nix-grpc-daemon (trusted)4712lb # [ 66.896276] envoy[888]: conn peer=[2001:db8:1::1]:54250 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3888 tx=1932 ms=17474713lb # [ 66.901290] envoy[888]: conn peer=[2001:db8:1::1]:51782 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=436 tx=376 ms=974714lb # [ 66.907709] envoy[888]: conn peer=[2001:db8:1::1]:51784 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4339 tx=2165 ms=21664715worker2 # [ 66.889427] nix-daemon[838]: reaped child process 974, status = succeeded4716lb # [ 66.912556] envoy[888]: conn peer=[2001:db8:1::1]:51796 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=438 tx=406 ms=644717lb # [ 66.922083] niks3-server[979]: 2026/09/21 14:44:42 INFO Received uploads request method=POST path=/api/pending_closures4718worker2 # [ 66.905094] nix-grpc-daemon[866]: time=2026-09-21T14:44:42.507Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"4719lb # [ 66.948585] niks3-server[979]: 2026/09/21 14:44:42 INFO Registered completed upload object_key=2i7aa0cdpp60vpdm180r56gqfh25m7hc.ls4720lb # [ 66.959872] niks3-server[979]: 2026/09/21 14:44:42 INFO Received sign narinfos request method=POST path=/api/pending_closures/20/sign4721worker2 # [ 66.941209] nix-grpc-daemon[866]: time=2026-09-21T14:44:42.543Z level=INFO msg="Uploading 1 narinfos"4722lb # [ 66.970277] niks3-server[979]: 2026/09/21 14:44:42 INFO Signed narinfos id=20 count=14723lb # [ 66.975393] niks3-server[979]: 2026/09/21 14:44:42 INFO Registered completed upload object_key=log/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv4724lb # [ 66.984470] niks3-server[979]: 2026/09/21 14:44:42 INFO Received complete upload request method=POST path=/api/pending_closures/20/complete4725lb # [ 66.991774] niks3-server[979]: 2026/09/21 14:44:42 INFO Registered completed upload object_key=2i7aa0cdpp60vpdm180r56gqfh25m7hc.narinfo4726lb # [ 66.996612] niks3-server[979]: 2026/09/21 14:44:42 INFO Completed upload id=204727worker2 # [ 66.972302] nix-grpc-daemon[866]: time=2026-09-21T14:44:42.575Z level=INFO msg="Upload complete. (222ms)"4728client # copying 1 paths...4729worker2 # [ 66.984322] nix-grpc-daemon[770]: ts=2026-09-21T14:44:42Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:45396 duration_s=1 drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv assign_id=9 outputs=14730client # copying path '/nix/store/2i7aa0cdpp60vpdm180r56gqfh25m7hc-uses-input-w1onlya' from 'grpc://lb:50051'...4731worker2 # [ 66.999157] nix-grpc-daemon[770]: ts=2026-09-21T14:44:42Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:453964732worker2 # [ 67.003316] nix-daemon[838]: reaped child process 952, status = succeeded4733worker2 # [ 67.020229] nix-grpc-daemon[770]: ts=2026-09-21T14:44:42Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:453964734worker2 # [ 67.025040] nix-grpc-daemon[770]: ts=2026-09-21T14:44:42Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:45396 duration_s=0 paths=1 nar_bytes_out=1204735client # [ 67.163573] nix-daemon[986]: reaped child process 1072, status = succeeded4736client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt a >&2, in 2.67 seconds)4737client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt b >&24738client # [ 67.343182] nix-daemon[986]: accepted connection from pid 1091, user root (trusted)4739client # this derivation will be built:4740client # /nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv4741worker2 # [ 67.474226] nix-grpc-daemon[770]: ts=2026-09-21T14:44:43Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:453964742client # building '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' on 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux'...4743worker2 # [ 67.481754] nix-grpc-daemon[770]: ts=2026-09-21T14:44:43Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:453964744worker2 # [ 67.488387] nix-grpc-daemon[866]: time=2026-09-21T14:44:43.090Z level=INFO msg="All 1 paths already cached"4745worker1 # [ 67.521338] nix-grpc-daemon[713]: ts=2026-09-21T14:44:43Z level=debug event=want client=10 drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv cached=04746worker1 # [ 67.528579] nix-grpc-daemon[713]: ts=2026-09-21T14:44:43Z level=debug event=assigned drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv worker=192.168.1.3:50051 clients=14747worker1 # [ 67.538280] nix-grpc-daemon[713]: ts=2026-09-21T14:44:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:529344748worker1 # [ 67.549296] nix-grpc-daemon[713]: ts=2026-09-21T14:44:43Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:529344749client # copying 1 paths to 'lb:50051'...4750client # copying path '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' to 'lb:50051'...4751worker1 # [ 67.571090] nix-grpc-daemon[713]: ts=2026-09-21T14:44:43Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:529344752worker1 # [ 67.585899] nix-daemon[852]: accepted connection from pid 713, user nix-grpc-daemon (trusted)4753worker1 # [ 67.606411] nix-grpc-daemon[713]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 2 ms (attempt 1/5)4754worker1 # [ 67.616113] nix-grpc-daemon[713]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 161 ms (attempt 2/5)4755worker1 # [ 67.786152] nix-grpc-daemon[713]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 243 ms (attempt 3/5)4756worker1 # [ 68.031241] nix-grpc-daemon[713]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 169 ms (attempt 4/5)4757worker1 # [ 68.202928] nix-grpc-daemon[713]: disabling binary cache 'https://cache.nixos.org' for 60 seconds4758worker1 # [ 68.211899] nix-grpc-daemon[713]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4759worker1 # [ 68.220563] nix-grpc-daemon[713]: copying path '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv' from 'http://lb:5751'...4760worker1 # [ 68.395891] nix-daemon[852]: accepted connection from pid 1212, user nix-grpc-daemon (trusted)4761worker1 # [ 68.414689] nix-daemon[852]: reaped child process 1215, status = succeeded4762worker1 # [ 68.522431] nix-daemon[852]: accepted connection from pid 1217, user nix-grpc-daemon (trusted)4763worker1 # [ 68.541204] nix-daemon[852]: reaped child process 1220, status = succeeded4764lb # [ 68.550738] niks3-server[979]: 2026/09/21 14:44:44 INFO Received uploads request method=POST path=/api/pending_closures4765worker1 # [ 68.558120] nix-grpc-daemon[866]: time=2026-09-21T14:44:44.166Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4766worker1 # [ 68.559645] nix-grpc-daemon[866]: time=2026-09-21T14:44:44.166Z level=INFO msg="Uploading 12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv (568B)"4767lb # [ 68.580616] niks3-server[979]: 2026/09/21 14:44:44 INFO Registered completed upload object_key=nar/14jznd749vfl63s1gv7i29r5z6jcai2byp2nz83dr5gib6z2cyaz.nar.zst4768worker1 # [ 68.588605] nix-grpc-daemon[866]: time=2026-09-21T14:44:44.197Z level=INFO msg="Uploading 1 narinfos"4769lb # [ 68.594214] niks3-server[979]: 2026/09/21 14:44:44 INFO Received sign narinfos request method=POST path=/api/pending_closures/21/sign4770lb # [ 68.605309] niks3-server[979]: 2026/09/21 14:44:44 INFO Signed narinfos id=21 count=14771lb # [ 68.613570] niks3-server[979]: 2026/09/21 14:44:44 INFO Registered completed upload object_key=12ialcyr3l4m98rzfgk6apzimy55z3j8.ls4772worker1 # [ 68.621657] nix-grpc-daemon[866]: time=2026-09-21T14:44:44.230Z level=INFO msg="Upload complete. (388ms)"4773lb # [ 68.626443] niks3-server[979]: 2026/09/21 14:44:44 INFO Received complete upload request method=POST path=/api/pending_closures/21/complete4774lb # [ 68.636881] niks3-server[979]: 2026/09/21 14:44:44 INFO Completed upload id=214775worker1 # [ 68.629058] nix-grpc-daemon[713]: ts=2026-09-21T14:44:44Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52934 duration_s=1 paths=1 nar_bytes_in=5684776lb # [ 68.640506] niks3-server[979]: 2026/09/21 14:44:44 INFO Registered completed upload object_key=12ialcyr3l4m98rzfgk6apzimy55z3j8.narinfo4777worker1 # [ 68.639614] nix-grpc-daemon[713]: ts=2026-09-21T14:44:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:529344778worker1 # [ 68.654505] nix-daemon[852]: accepted connection from pid 713, user nix-grpc-daemon (trusted)4779worker1 # [ 68.657299] nix-daemon[852]: reaped child process 1205, status = succeeded4780client # building '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' on lb:50051...4781worker1 # [ 68.669760] nix-daemon[852]: accepted connection from pid 713, user nix-grpc-daemon (trusted)4782client # uses-input-w1onlyb> node-a: building 12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv4783client # uses-input-w1onlyb> disabling binary cache 'https://cache.nixos.org' for 60 seconds4784worker1 # [ 70.340108] nix-daemon[852]: reaped child process 1224, status = succeeded4785worker1 # [ 70.440821] nix-daemon[852]: accepted connection from pid 1240, user nix-grpc-daemon (trusted)4786worker1 # [ 70.459698] nix-daemon[852]: reaped child process 1243, status = succeeded4787lb # [ 70.469567] niks3-server[979]: 2026/09/21 14:44:46 INFO Received uploads request method=POST path=/api/pending_closures4788worker1 # [ 70.475773] nix-grpc-daemon[866]: time=2026-09-21T14:44:46.082Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"4789lb # [ 70.500794] niks3-server[979]: 2026/09/21 14:44:46 INFO Registered completed upload object_key=jfy62nvxcd0b9rkhly152q21ddbrcn28.ls4790worker1 # [ 70.508070] nix-grpc-daemon[866]: time=2026-09-21T14:44:46.116Z level=INFO msg="Uploading 1 narinfos"4791lb # [ 70.515113] niks3-server[979]: 2026/09/21 14:44:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/22/sign4792lb # [ 70.525726] niks3-server[979]: 2026/09/21 14:44:46 INFO Signed narinfos id=22 count=14793lb # [ 70.532692] niks3-server[979]: 2026/09/21 14:44:46 INFO Registered completed upload object_key=log/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv4794worker1 # [ 70.540277] nix-grpc-daemon[866]: time=2026-09-21T14:44:46.148Z level=INFO msg="Upload complete. (259ms)"4795lb # [ 70.544903] niks3-server[979]: 2026/09/21 14:44:46 INFO Received complete upload request method=POST path=/api/pending_closures/22/complete4796lb # [ 70.551635] niks3-server[979]: 2026/09/21 14:44:46 INFO Completed upload id=224797client # copying 1 paths...4798lb # [ 70.555763] niks3-server[979]: 2026/09/21 14:44:46 INFO Registered completed upload object_key=jfy62nvxcd0b9rkhly152q21ddbrcn28.narinfo4799worker1 # [ 70.553069] nix-grpc-daemon[713]: ts=2026-09-21T14:44:46Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52934 duration_s=1 drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv assign_id=10 outputs=14800client # copying path '/nix/store/jfy62nvxcd0b9rkhly152q21ddbrcn28-uses-input-w1onlyb' from 'grpc://lb:50051'...4801worker1 # [ 70.571787] nix-grpc-daemon[713]: ts=2026-09-21T14:44:46Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:529344802worker1 # [ 70.576920] nix-daemon[852]: reaped child process 1222, status = succeeded4803worker2 # [ 70.570948] nix-grpc-daemon[770]: ts=2026-09-21T14:44:46Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:453964804worker2 # [ 70.584367] nix-grpc-daemon[770]: copying path '/nix/store/jfy62nvxcd0b9rkhly152q21ddbrcn28-uses-input-w1onlyb' from 'http://lb:5751'...4805worker2 # [ 70.600931] nix-grpc-daemon[770]: ts=2026-09-21T14:44:46Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:45396 duration_s=0 paths=1 nar_bytes_out=1204806client # [ 70.747640] nix-daemon[986]: reaped child process 1094, status = succeeded4807client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt b >&2, in 3.58 seconds)4808client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt c >&24809client # [ 70.922718] nix-daemon[986]: accepted connection from pid 1113, user root (trusted)4810client # this derivation will be built:4811client # /nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv4812worker1 # [ 71.075029] nix-grpc-daemon[713]: ts=2026-09-21T14:44:46Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:529344813client # building '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' on 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux'...4814worker2 # [ 71.058192] nix-grpc-daemon[770]: ts=2026-09-21T14:44:46Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:453964815worker2 # [ 71.069105] nix-grpc-daemon[866]: time=2026-09-21T14:44:46.671Z level=INFO msg="All 1 paths already cached"4816worker1 # [ 71.106315] nix-grpc-daemon[713]: ts=2026-09-21T14:44:46Z level=debug event=want client=11 drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv cached=04817worker2 # [ 71.088743] nix-grpc-daemon[770]: ts=2026-09-21T14:44:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:453964818worker1 # [ 71.112754] nix-grpc-daemon[713]: ts=2026-09-21T14:44:46Z level=debug event=assigned drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv worker=192.168.1.4:50051 clients=14819worker2 # [ 71.103262] nix-grpc-daemon[770]: ts=2026-09-21T14:44:46Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:453964820worker2 # [ 71.117684] nix-grpc-daemon[866]: time=2026-09-21T14:44:46.720Z level=INFO msg="All 1 paths already cached"4821client # copying 1 paths to 'lb:50051'...4822client # copying path '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' to 'lb:50051'...4823worker2 # [ 71.128538] nix-grpc-daemon[770]: ts=2026-09-21T14:44:46Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:453964824worker2 # [ 71.142821] nix-daemon[838]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4825worker2 # [ 71.307181] nix-daemon[838]: accepted connection from pid 979, user nix-grpc-daemon (trusted)4826worker2 # [ 71.338718] nix-daemon[838]: reaped child process 982, status = succeeded4827worker2 # [ 71.436539] nix-daemon[838]: accepted connection from pid 984, user nix-grpc-daemon (trusted)4828worker2 # [ 71.457299] nix-daemon[838]: reaped child process 987, status = succeeded4829lb # [ 71.487583] niks3-server[979]: 2026/09/21 14:44:47 INFO Received uploads request method=POST path=/api/pending_closures4830worker2 # [ 71.473135] nix-grpc-daemon[866]: time=2026-09-21T14:44:47.075Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4831worker2 # [ 71.474598] nix-grpc-daemon[866]: time=2026-09-21T14:44:47.075Z level=INFO msg="Uploading n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv (568B)"4832lb # [ 71.520808] niks3-server[979]: 2026/09/21 14:44:47 INFO Registered completed upload object_key=nar/1qd3ps964395j2cc3rlqd58m21s7f6na9s0k2z5kclx0c3l4sl40.nar.zst4833lb # [ 71.537954] niks3-server[979]: 2026/09/21 14:44:47 INFO Received sign narinfos request method=POST path=/api/pending_closures/23/sign4834worker2 # [ 71.519243] nix-grpc-daemon[866]: time=2026-09-21T14:44:47.121Z level=INFO msg="Uploading 1 narinfos"4835lb # [ 71.546453] niks3-server[979]: 2026/09/21 14:44:47 INFO Signed narinfos id=23 count=14836lb # [ 71.559149] niks3-server[979]: 2026/09/21 14:44:47 INFO Registered completed upload object_key=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr.ls4837lb # [ 71.570484] niks3-server[979]: 2026/09/21 14:44:47 INFO Received complete upload request method=POST path=/api/pending_closures/23/complete4838worker2 # [ 71.550585] nix-grpc-daemon[866]: time=2026-09-21T14:44:47.153Z level=INFO msg="Upload complete. (390ms)"4839lb # [ 71.581756] niks3-server[979]: 2026/09/21 14:44:47 INFO Completed upload id=234840lb # [ 71.584941] niks3-server[979]: 2026/09/21 14:44:47 INFO Registered completed upload object_key=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr.narinfo4841worker2 # [ 71.560642] nix-grpc-daemon[770]: ts=2026-09-21T14:44:47Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:45396 duration_s=0 paths=1 nar_bytes_in=5684842worker2 # [ 71.574073] nix-grpc-daemon[770]: ts=2026-09-21T14:44:47Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:453964843worker2 # [ 71.590233] nix-daemon[838]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4844worker2 # [ 71.594131] nix-daemon[838]: reaped child process 977, status = succeeded4845client # building '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' on lb:50051...4846worker2 # [ 71.608948] nix-daemon[838]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4847client # uses-input-w1onlyc> worker2: building n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv4848client # uses-input-w1onlyc> disabling binary cache 'https://cache.nixos.org' for 60 seconds4849worker2 # [ 72.319044] nix-daemon[838]: reaped child process 991, status = succeeded4850worker2 # [ 72.406424] nix-daemon[838]: accepted connection from pid 1007, user nix-grpc-daemon (trusted)4851worker2 # [ 72.428253] nix-daemon[838]: reaped child process 1010, status = succeeded4852lb # [ 72.450092] niks3-server[979]: 2026/09/21 14:44:48 INFO Received uploads request method=POST path=/api/pending_closures4853worker2 # [ 72.440458] nix-grpc-daemon[866]: time=2026-09-21T14:44:48.040Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"4854lb # [ 72.485469] niks3-server[979]: 2026/09/21 14:44:48 INFO Received sign narinfos request method=POST path=/api/pending_closures/24/sign4855worker2 # [ 72.467267] nix-grpc-daemon[866]: time=2026-09-21T14:44:48.069Z level=INFO msg="Uploading 1 narinfos"4856lb # [ 72.501695] niks3-server[979]: 2026/09/21 14:44:48 INFO Signed narinfos id=24 count=14857lb # [ 72.509535] niks3-server[979]: 2026/09/21 14:44:48 INFO Registered completed upload object_key=13r5ipj38sk1irdi724wm1mbl9lw720b.ls4858lb # [ 72.525119] niks3-server[979]: 2026/09/21 14:44:48 INFO Received complete upload request method=POST path=/api/pending_closures/24/complete4859lb # [ 72.531266] niks3-server[979]: 2026/09/21 14:44:48 INFO Registered completed upload object_key=log/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv4860lb # [ 72.537176] niks3-server[979]: 2026/09/21 14:44:48 INFO Completed upload id=244861worker2 # [ 72.513405] nix-grpc-daemon[866]: time=2026-09-21T14:44:48.116Z level=INFO msg="Upload complete. (292ms)"4862lb # [ 72.544277] niks3-server[979]: 2026/09/21 14:44:48 INFO Registered completed upload object_key=13r5ipj38sk1irdi724wm1mbl9lw720b.narinfo4863worker2 # [ 72.524317] nix-grpc-daemon[770]: ts=2026-09-21T14:44:48Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:45396 duration_s=0 drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv assign_id=11 outputs=14864worker2 # [ 72.539632] nix-daemon[838]: reaped child process 989, status = succeeded4865client # copying 1 paths...4866worker2 # [ 72.578235] nix-grpc-daemon[770]: ts=2026-09-21T14:44:48Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:453964867client # copying path '/nix/store/13r5ipj38sk1irdi724wm1mbl9lw720b-uses-input-w1onlyc' from 'grpc://lb:50051'...4868worker2 # [ 72.628278] nix-grpc-daemon[770]: ts=2026-09-21T14:44:48Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:453964869worker2 # [ 72.633843] nix-grpc-daemon[770]: ts=2026-09-21T14:44:48Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:45396 duration_s=0 paths=1 nar_bytes_out=1204870client # [ 72.848763] nix-daemon[986]: reaped child process 1116, status = succeeded4871client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt c >&2, in 2.11 seconds)4872(finished: subtest: an input only one worker has still reaches the builder, in 9.00 seconds)4873subtest: upload whose reference the worker lacks is completed from the cache4874client: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A input --argstr tag viacache4875client # this derivation will be built:4876client # /nix/store/x1rrnjihpa1xvpyxxxdrjzp3c8z8nayq-local-input-viacache.drv4877client # building '/nix/store/x1rrnjihpa1xvpyxxxdrjzp3c8z8nayq-local-input-viacache.drv'...4878client: (finished: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A input --argstr tag viacache, in 0.57 seconds)4879client: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A referrer --argstr tag viacache4880client # this derivation will be built:4881client # /nix/store/1m587dk4visgyryf6n0mgp2rlp7pbjd1-refers-to-input-viacache.drv4882client # building '/nix/store/1m587dk4visgyryf6n0mgp2rlp7pbjd1-refers-to-input-viacache.drv'...4883client: (finished: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A referrer --argstr tag viacache, in 0.28 seconds)4884client: must succeed: nix-store --export /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache > /tmp/shared/ref.closure4885client: (finished: must succeed: nix-store --export /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache > /tmp/shared/ref.closure, in 0.11 seconds)4886worker1: must succeed: nix-store --import < /tmp/shared/ref.closure4887worker1: (finished: must succeed: nix-store --import < /tmp/shared/ref.closure, in 0.20 seconds)4888worker2 # [ 74.103353] systemd[1]: Stopping Nix worker-protocol over gRPC...4889worker2 # [ 74.958159] nix-grpc-daemon[770]: ts=2026-09-21T14:44:50Z level=info event=signal_exit4890worker1 # [ 76.127462] nix-grpc-daemon[713]: ts=2026-09-21T14:44:51Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:529184891worker1 # [ 76.127952] nix-grpc-daemon[713]: ts=2026-09-21T14:44:51Z level=info event=worker_gone addr=192.168.1.4:500514892worker2 # [ 76.107912] nix-grpc-daemon[770]: ts=2026-09-21T14:44:51Z level=info event=scheduler_restarting4893worker2 # [ 76.136496] nix-grpc-daemon[770]: ts=2026-09-21T14:44:51Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"4894worker2 # [ 76.167074] nix-daemon[838]: reaped child process 840, status = succeeded4895lb # [ 76.886431] envoy[888]: conn peer=[2001:db8:1::1]:51808 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4234 tx=1852 ms=22474896lb # [ 76.891803] envoy[888]: conn peer=[2001:db8:1::1]:51818 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=422 tx=328 ms=644897lb # [ 76.897481] envoy[888]: conn peer=[2001:db8:1::1]:51826 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3741 tx=1900 ms=31884898lb # [ 76.902952] envoy[888]: conn peer=[2001:db8:1::1]:51828 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=448 tx=329 ms=934899lb # [ 76.907389] envoy[888]: conn peer=[2001:db8:1::1]:38668 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=422 tx=328 ms=774900lb # [ 76.912663] envoy[888]: conn peer=[2001:db8:1::1]:38662 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3725 tx=1884 ms=16364901lb # [ 76.917205] envoy[888]: conn peer=[2001:db8:1::4]:60712 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=61818 tx=3056 ms=581334902worker2 # [ 77.057012] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.4903worker2 # [ 77.060842] systemd[1]: Stopped Nix worker-protocol over gRPC.4904worker2 # [ 77.069517] systemd[1]: nix-grpc-daemon.service: Consumed 1.819s CPU time over 59.271s wall clock time, 53.9M memory peak, 51.2K incoming IP traffic, 146.9K outgoing IP traffic.4905worker2 # [ 77.084573] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.4906worker2 # [ 77.091918] systemd[1]: Closed Nix worker-protocol over gRPC.4907worker2 # [ 77.093924] systemd[1]: nix-grpc-daemon.socket: Consumed 520us CPU time over 1min 7.725s wall clock time, 256K memory peak, 58.2K incoming IP traffic, 51.6K outgoing IP traffic.4908lb: waiting until exactly ['worker2'] endpoints of aarch64-linux are unhealthy4909lb: must succeed: curl -sf localhost:9901/clusters4910lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)4911lb: (finished: waiting until exactly ['worker2'] endpoints of aarch64-linux are unhealthy, in 0.07 seconds)4912client: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache >&24913worker1 # [ 77.418922] nix-grpc-daemon[713]: ts=2026-09-21T14:44:53Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:529344914worker1 # [ 77.427668] nix-grpc-daemon[713]: ts=2026-09-21T14:44:53Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:529344915client # /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache4916client: (finished: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache >&2, in 0.26 seconds)4917worker2: must fail: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache4918worker2: (finished: must fail: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache, in 0.03 seconds)4919worker2 # [ 77.538426] systemd[1]: Listening on Nix worker-protocol over gRPC.4920worker1 # [ 77.631237] systemd[1]: Stopping Nix worker-protocol over gRPC...4921worker2 # [ 78.098821] systemd[1]: Starting scheduler-token.service...4922worker1 # [ 78.139268] nix-grpc-daemon[713]: ts=2026-09-21T14:44:53Z level=info event=signal_exit4923worker2 # [ 78.194464] systemd[1]: scheduler-token.service: Deactivated successfully.4924worker2 # [ 78.204984] systemd[1]: Finished scheduler-token.service.4925worker2 # [ 78.206958] systemd[1]: scheduler-token.service: Consumed 62ms CPU time over 100ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.4926worker2 # [ 78.214941] systemd[1]: Starting Nix worker-protocol over gRPC...4927worker2 # [ 78.349223] nix-grpc-daemon[1034]: ts=2026-09-21T14:44:53Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"4928worker2 # [ 78.351420] nix-grpc-daemon[1034]: ts=2026-09-21T14:44:53Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc4929worker2 # [ 78.355399] nix-grpc-daemon[1034]: ts=2026-09-21T14:44:53Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=14930worker2 # [ 78.361053] nix-grpc-daemon[1034]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR4931worker2 # [ 78.362469] nix-grpc-daemon[1034]: E0000 00:00:1790001893.963478 1034 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4932worker2 # [ 78.404137] nix-grpc-daemon[1034]: E0000 00:00:1790001893.969710 1034 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4933worker2 # [ 78.406406] nix-grpc-daemon[1034]: ts=2026-09-21T14:44:53Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket4934worker2 # [ 78.407980] nix-grpc-daemon[1034]: E0000 00:00:1790001893.981178 1075 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4935worker2 # [ 78.419070] nix-grpc-daemon[1034]: E0000 00:00:1790001893.999885 1075 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4936worker2 # [ 78.421893] nix-grpc-daemon[1034]: E0000 00:00:1790001894.000038 1075 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4937worker2 # [ 78.424433] systemd[1]: Started Nix worker-protocol over gRPC.4938worker2 # [ 78.425260] nix-grpc-daemon[1034]: E0000 00:00:1790001894.021233 1075 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4939worker2 # [ 78.432715] nix-grpc-daemon[1034]: ts=2026-09-21T14:44:54Z level=info event=scheduler_connected addr=lb:500514940worker2 # [ 78.434289] nix-grpc-daemon[1034]: ts=2026-09-21T14:44:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4941worker2 # [ 78.634527] nix-grpc-daemon[1034]: ts=2026-09-21T14:44:54Z level=info event=scheduler_connected addr=lb:500514942worker2 # [ 78.639567] nix-grpc-daemon[1034]: ts=2026-09-21T14:44:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4943worker2 # [ 78.840685] nix-grpc-daemon[1034]: ts=2026-09-21T14:44:54Z level=info event=scheduler_connected addr=lb:500514944worker2 # [ 78.845781] nix-grpc-daemon[1034]: ts=2026-09-21T14:44:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4945worker2 # [ 79.046791] nix-grpc-daemon[1034]: ts=2026-09-21T14:44:54Z level=info event=scheduler_connected addr=lb:500514946worker2 # [ 79.051904] nix-grpc-daemon[1034]: ts=2026-09-21T14:44:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4947lb # [ 79.128450] niks3-server[979]: 2026/09/21 14:44:54 INFO lead: released remote=192.168.1.3:499344948worker2 # [ 79.253031] nix-grpc-daemon[1034]: ts=2026-09-21T14:44:54Z level=info event=scheduler_connected addr=lb:500514949worker2 # [ 79.258023] nix-grpc-daemon[1034]: ts=2026-09-21T14:44:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4950worker2 # [ 79.460306] nix-grpc-daemon[1034]: ts=2026-09-21T14:44:55Z level=info event=scheduler_connected addr=lb:500514951worker2 # [ 79.465310] nix-grpc-daemon[1034]: ts=2026-09-21T14:44:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4952worker2 # [ 79.666289] nix-grpc-daemon[1034]: ts=2026-09-21T14:44:55Z level=info event=scheduler_connected addr=lb:500514953worker2 # [ 79.671418] nix-grpc-daemon[1034]: ts=2026-09-21T14:44:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4954worker2 # [ 79.872661] nix-grpc-daemon[1034]: ts=2026-09-21T14:44:55Z level=info event=scheduler_connected addr=lb:500514955worker2 # [ 79.877725] nix-grpc-daemon[1034]: ts=2026-09-21T14:44:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4956worker2 # [ 80.078833] nix-grpc-daemon[1034]: ts=2026-09-21T14:44:55Z level=info event=scheduler_connected addr=lb:500514957worker2 # [ 80.083952] nix-grpc-daemon[1034]: ts=2026-09-21T14:44:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4958worker1 # [ 80.125255] nix-grpc-daemon[713]: ts=2026-09-21T14:44:55Z level=info event=scheduler_restarting4959worker1 # [ 80.130121] nix-grpc-daemon[713]: ts=2026-09-21T14:44:55Z level=info event=scheduler_restarting addr=lb:500514960worker1 # [ 80.135669] nix-grpc-daemon[713]: ts=2026-09-21T14:44:55Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:529184961worker1 # [ 80.140884] nix-grpc-daemon[713]: ts=2026-09-21T14:44:55Z level=info event=worker_gone addr=192.168.1.3:500514962worker1 # [ 80.174036] nix-daemon[852]: reaped child process 854, status = succeeded4963worker1 # [ 80.236504] nix-grpc-daemon[713]: E0000 00:00:1790001895.844796 758 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 17, started true, shutdown true4964worker1 # [ 80.257455] nix-grpc-daemon[713]: E0000 00:00:1790001895.865880 758 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 9, started true, shutdown true4965worker2 # [ 80.285134] nix-grpc-daemon[1034]: ts=2026-09-21T14:44:55Z level=info event=scheduler_connected addr=lb:500514966worker2 # [ 80.290183] nix-grpc-daemon[1034]: ts=2026-09-21T14:44:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4967worker1 # [ 80.349069] nix-grpc-daemon[713]: ts=2026-09-21T14:44:55Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"4968worker2 # [ 80.492857] nix-grpc-daemon[1034]: ts=2026-09-21T14:44:56Z level=info event=scheduler_connected addr=lb:500514969worker2 # [ 80.496961] nix-grpc-daemon[1034]: ts=2026-09-21T14:44:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4970worker2 # [ 80.693590] nix-grpc-daemon[1034]: ts=2026-09-21T14:44:56Z level=info event=scheduler_connected addr=lb:500514971worker2 # [ 80.698432] nix-grpc-daemon[1034]: ts=2026-09-21T14:44:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4972worker2 # [ 80.899396] nix-grpc-daemon[1034]: ts=2026-09-21T14:44:56Z level=info event=scheduler_connected addr=lb:500514973worker2 # [ 80.904597] nix-grpc-daemon[1034]: ts=2026-09-21T14:44:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4974worker1 # [ 81.024315] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.4975worker1 # [ 81.029114] systemd[1]: Stopped Nix worker-protocol over gRPC.4976worker1 # [ 81.035011] systemd[1]: nix-grpc-daemon.service: Consumed 3.946s CPU time over 1min 4.138s wall clock time, 68.2M memory peak, 102.5K incoming IP traffic, 211.7K outgoing IP traffic.4977worker1 # [ 81.053550] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.4978worker1 # [ 81.066123] systemd[1]: Closed Nix worker-protocol over gRPC.4979lb: waiting until exactly ['worker1'] endpoints of aarch64-linux are unhealthy4980lb: must succeed: curl -sf localhost:9901/clusters4981worker1 # [ 81.069484] systemd[1]: nix-grpc-daemon.socket: Consumed 544us CPU time over 1min 11.082s wall clock time, 256K memory peak, 95.8K incoming IP traffic, 78.5K outgoing IP traffic.4982worker2 # [ 81.105867] nix-grpc-daemon[1034]: ts=2026-09-21T14:44:56Z level=info event=scheduler_connected addr=lb:500514983worker2 # [ 81.111009] nix-grpc-daemon[1034]: ts=2026-09-21T14:44:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4984lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)4985lb: (finished: waiting until exactly ['worker1'] endpoints of aarch64-linux are unhealthy, in 0.07 seconds)4986client: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache >&24987worker2 # [ 81.312069] nix-grpc-daemon[1034]: ts=2026-09-21T14:44:56Z level=info event=scheduler_connected addr=lb:500514988worker2 # [ 81.317120] nix-grpc-daemon[1034]: ts=2026-09-21T14:44:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4989worker2 # [ 81.340152] nix-grpc-daemon[1034]: ts=2026-09-21T14:44:56Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:390224990worker2 # [ 81.355787] nix-daemon[838]: accepted connection from pid 1034, user nix-grpc-daemon (trusted)4991worker2 # [ 81.384354] nix-grpc-daemon[1034]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 69 ms (attempt 1/5)4992worker2 # [ 81.455057] nix-grpc-daemon[1034]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 147 ms (attempt 2/5)4993worker2 # [ 81.519931] nix-grpc-daemon[1034]: ts=2026-09-21T14:44:57Z level=info event=scheduler_connected addr=lb:500514994worker2 # [ 81.524155] nix-grpc-daemon[1034]: ts=2026-09-21T14:44:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4995worker2 # [ 81.604072] nix-grpc-daemon[1034]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 123 ms (attempt 3/5)4996worker2 # [ 81.720867] nix-grpc-daemon[1034]: ts=2026-09-21T14:44:57Z level=info event=scheduler_connected addr=lb:500514997worker2 # [ 81.725862] nix-grpc-daemon[1034]: ts=2026-09-21T14:44:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4998worker2 # [ 81.733113] nix-grpc-daemon[1034]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 500 ms (attempt 4/5)4999worker2 # [ 81.926969] nix-grpc-daemon[1034]: ts=2026-09-21T14:44:57Z level=info event=scheduler_connected addr=lb:500515000worker2 # [ 81.932194] nix-grpc-daemon[1034]: ts=2026-09-21T14:44:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5001worker2 # [ 82.133234] nix-grpc-daemon[1034]: ts=2026-09-21T14:44:57Z level=info event=scheduler_connected addr=lb:500515002worker2 # [ 82.138278] nix-grpc-daemon[1034]: ts=2026-09-21T14:44:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5003worker2 # [ 82.243517] nix-grpc-daemon[1034]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5004worker2 # [ 82.249603] nix-grpc-daemon[1034]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5005client # copying 2 paths...5006client # copying path '/nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache' to 'grpc://lb:50051'...5007worker2 # [ 82.273074] nix-grpc-daemon[1034]: ts=2026-09-21T14:44:57Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:390225008client # copying path '/nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache' to 'grpc://lb:50051'...5009worker2 # [ 82.290097] nix-daemon[838]: accepted connection from pid 1034, user nix-grpc-daemon (trusted)5010worker2 # [ 82.324714] nix-grpc-daemon[1034]: ts=2026-09-21T14:44:57Z level=info event=niks3_push_started pid=10945011worker2 # [ 82.348449] nix-grpc-daemon[1034]: ts=2026-09-21T14:44:57Z level=info event=scheduler_connected addr=lb:500515012worker2 # [ 82.351131] nix-grpc-daemon[1034]: ts=2026-09-21T14:44:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5013worker2 # [ 82.489196] nix-grpc-daemon[1094]: time=2026-09-21T14:44:58.091Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.key ca=""5014worker2 # [ 82.544774] nix-grpc-daemon[1034]: ts=2026-09-21T14:44:58Z level=info event=scheduler_connected addr=lb:500515015worker2 # [ 82.547681] nix-grpc-daemon[1034]: ts=2026-09-21T14:44:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5016worker2 # [ 82.600452] nix-daemon[838]: accepted connection from pid 1106, user nix-grpc-daemon (trusted)5017worker2 # [ 82.618472] nix-daemon[838]: reaped child process 1109, status = succeeded5018lb # [ 82.649367] niks3-server[979]: 2026/09/21 14:44:58 INFO Received uploads request method=POST path=/api/pending_closures5019lb # [ 82.660624] niks3-server[979]: 2026/09/21 14:44:58 INFO Received uploads request method=POST path=/api/pending_closures5020worker2 # [ 82.646228] nix-grpc-daemon[1094]: time=2026-09-21T14:44:58.247Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"5021worker2 # [ 82.647682] nix-grpc-daemon[1094]: time=2026-09-21T14:44:58.247Z level=INFO msg="Uploading syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache (128B)"5022worker2 # [ 82.650555] nix-grpc-daemon[1094]: time=2026-09-21T14:44:58.247Z level=INFO msg="Uploading 4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache (184B)"5023lb # [ 82.714016] niks3-server[979]: 2026/09/21 14:44:58 INFO Registered completed upload object_key=nar/1kmnh5f2l7jjcgaq9b9988bjix6z8zjf6v350xllq5hlzgnb1f75.nar.zst5024lb # [ 82.731024] niks3-server[979]: 2026/09/21 14:44:58 INFO Registered completed upload object_key=nar/1rf1rg73pvnaa9kk6sy9r1ydj24b4g0kb0wnx0z5hxl1nvkmlavq.nar.zst5025lb # [ 82.741151] niks3-server[979]: 2026/09/21 14:44:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/25/sign5026worker2 # [ 82.724155] nix-grpc-daemon[1094]: time=2026-09-21T14:44:58.326Z level=INFO msg="Uploading 3 narinfos"5027lb # [ 82.750239] niks3-server[979]: 2026/09/21 14:44:58 INFO Signed narinfos id=25 count=15028lb # [ 82.756096] niks3-server[979]: 2026/09/21 14:44:58 INFO Registered completed upload object_key=syvivri0wisb7fd33rxcg9kz63kc2bv7.ls5029lb # [ 82.759439] niks3-server[979]: 2026/09/21 14:44:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/26/sign5030lb # [ 82.769286] niks3-server[979]: 2026/09/21 14:44:58 INFO Signed narinfos id=26 count=25031lb # [ 82.771281] niks3-server[979]: 2026/09/21 14:44:58 INFO Registered completed upload object_key=4rig83zwv4xj1zajs08ima13w0yh676v.ls5032worker2 # [ 82.748107] nix-grpc-daemon[1034]: ts=2026-09-21T14:44:58Z level=info event=scheduler_connected addr=lb:500515033worker2 # [ 82.753132] nix-grpc-daemon[1034]: ts=2026-09-21T14:44:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5034lb # [ 82.790236] niks3-server[979]: 2026/09/21 14:44:58 INFO Registered completed upload object_key=4rig83zwv4xj1zajs08ima13w0yh676v.narinfo5035lb # [ 82.798444] niks3-server[979]: 2026/09/21 14:44:58 INFO Received complete upload request method=POST path=/api/pending_closures/25/complete5036lb # [ 82.806078] niks3-server[979]: 2026/09/21 14:44:58 INFO Completed upload id=255037lb # [ 82.808570] niks3-server[979]: 2026/09/21 14:44:58 INFO Received complete upload request method=POST path=/api/pending_closures/26/complete5038worker2 # [ 82.783554] nix-grpc-daemon[1094]: time=2026-09-21T14:44:58.386Z level=INFO msg="Upload complete. (290ms)"5039lb # [ 82.810964] niks3-server[979]: 2026/09/21 14:44:58 INFO Registered completed upload object_key=syvivri0wisb7fd33rxcg9kz63kc2bv7.narinfo5040lb # [ 82.815817] niks3-server[979]: 2026/09/21 14:44:58 INFO Registered completed upload object_key=syvivri0wisb7fd33rxcg9kz63kc2bv7.narinfo5041lb # [ 82.819650] niks3-server[979]: 2026/09/21 14:44:58 INFO Completed upload id=265042worker2 # [ 82.796117] nix-grpc-daemon[1034]: ts=2026-09-21T14:44:58Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39022 duration_s=0 paths=2 nar_bytes_in=3125043worker2 # [ 82.810643] nix-daemon[838]: reaped child process 1092, status = succeeded5044client: (finished: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache >&2, in 1.71 seconds)5045worker2: must succeed: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache && test -e /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache5046worker2: (finished: must succeed: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache && test -e /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache, in 0.03 seconds)5047worker1 # [ 82.951103] systemd[1]: Listening on Nix worker-protocol over gRPC.5048worker1 # [ 82.960972] systemd[1]: Starting Nix worker-protocol over gRPC...5049worker2 # [ 82.954210] nix-grpc-daemon[1034]: ts=2026-09-21T14:44:58Z level=info event=scheduler_connected addr=lb:500515050worker2 # [ 82.959411] nix-grpc-daemon[1034]: ts=2026-09-21T14:44:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5051worker1 # [ 83.107173] nix-grpc-daemon[1268]: ts=2026-09-21T14:44:58Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5052worker1 # [ 83.112731] nix-grpc-daemon[1268]: ts=2026-09-21T14:44:58Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5053worker1 # [ 83.114617] nix-grpc-daemon[1268]: ts=2026-09-21T14:44:58Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15054worker1 # [ 83.135509] nix-grpc-daemon[1268]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5055lb # [ 83.142893] niks3-server[979]: 2026/09/21 14:44:58 INFO lead: acquired remote=192.168.1.3:436545056lb: waiting until exactly no endpoints of aarch64-linux are unhealthy5057lb: must succeed: curl -sf localhost:9901/clusters5058worker1 # [ 83.156126] nix-grpc-daemon[1268]: E0000 00:00:1790001898.727148 1268 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5059worker1 # [ 83.175005] nix-grpc-daemon[1268]: E0000 00:00:1790001898.729737 1268 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5060worker1 # [ 83.177839] nix-grpc-daemon[1268]: ts=2026-09-21T14:44:58Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5061worker1 # [ 83.179647] systemd[1]: Started Nix worker-protocol over gRPC.5062worker1 # [ 83.180551] nix-grpc-daemon[1268]: ts=2026-09-21T14:44:58Z level=info event=scheduler_take_over5063worker1 # [ 83.181739] nix-grpc-daemon[1268]: E0000 00:00:1790001898.747163 1307 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5064worker2 # [ 83.160509] nix-grpc-daemon[1034]: ts=2026-09-21T14:44:58Z level=info event=scheduler_connected addr=lb:500515065worker1 # [ 83.184221] nix-grpc-daemon[1268]: E0000 00:00:1790001898.768272 1307 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5066worker1 # [ 83.190824] nix-grpc-daemon[1268]: E0000 00:00:1790001898.768435 1307 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5067worker1 # [ 83.197523] nix-grpc-daemon[1268]: E0000 00:00:1790001898.782794 1307 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5068worker1 # [ 83.199792] nix-grpc-daemon[1268]: ts=2026-09-21T14:44:58Z level=info event=scheduler_connected addr=lb:500515069worker1 # [ 83.201159] nix-grpc-daemon[1268]: ts=2026-09-21T14:44:58Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:429125070worker1 # [ 83.202660] nix-grpc-daemon[1268]: ts=2026-09-21T14:44:58Z level=info event=oidc_jwks provider=mock keys=15071worker1 # [ 83.203859] nix-grpc-daemon[1268]: ts=2026-09-21T14:44:58Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:429125072worker1 # [ 83.205623] nix-grpc-daemon[1268]: ts=2026-09-21T14:44:58Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=05073worker1 # [ 83.207316] nix-grpc-daemon[1268]: ts=2026-09-21T14:44:58Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=05074lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5075lb: (finished: waiting until exactly no endpoints of aarch64-linux are unhealthy, in 0.07 seconds)5076(finished: subtest: upload whose reference the worker lacks is completed from the cache, in 10.34 seconds)5077subtest: a niks3 restart does not cost the leader its role5078lb: must succeed: curl -sf localhost:9901/clusters5079lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5080lb: must succeed: curl -sf localhost:9901/clusters5081lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)5082lb # [ 83.340766] niks3-server[979]: 2026/09/21 14:44:58 INFO Shutdown signal received, draining in-flight requests timeout=10s5083worker1 # [ 83.340603] nix-grpc-daemon[1268]: ts=2026-09-21T14:44:58Z level=info event=scheduler_lock_lost status=2005084worker2 # [ 83.320189] nix-grpc-daemon[1034]: ts=2026-09-21T14:44:58Z level=info event=scheduler_lock_lost status=2005085lb # [ 83.350076] niks3-server[979]: 2026/09/21 14:44:58 INFO lead: released remote=192.168.1.3:436545086lb # [ 83.351617] systemd[1]: Stopping niks3 server...5087lb # [ 86.884489] envoy[888]: conn peer=[2001:db8:1::1]:38678 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=591 tx=522 ms=315088lb # [ 86.884799] envoy[888]: conn peer=[2001:db8:1::3]:33518 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=13257 tx=3626 ms=631495089lb # [ 86.884940] envoy[888]: conn peer=[2001:db8:1::1]:50472 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1008 tx=305 ms=14695090lb # [ 88.076466] systemd[1]: niks3.service: Deactivated successfully.5091lb # [ 88.081817] systemd[1]: Stopped niks3 server.5092lb # [ 88.086893] systemd[1]: niks3.service: Consumed 493ms CPU time over 1min 4.174s wall clock time, 25.5M memory peak, 68.2K incoming IP traffic, 98.9K outgoing IP traffic.5093lb # [ 88.110756] systemd[1]: Starting niks3 server...5094lb # [ 88.220503] niks3-server[1112]: 2026/09/21 14:45:03 goose: no migrations to run. current version: 202609200000005095lb # [ 88.227788] niks3-server[1112]: 2026/09/21 14:45:03 OK 1_commit_pending_closure.sql (7.75ms)5096lb # [ 88.232144] niks3-server[1112]: 2026/09/21 14:45:03 OK 2_object_stats_trigger.sql (3.94ms)5097lb # [ 88.233466] niks3-server[1112]: 2026/09/21 14:45:03 goose: up to current file version: 25098lb # [ 88.235294] niks3-server[1112]: 2026/09/21 14:45:03 INFO Loaded signing key name=farm-test-1 path=/nix/store/3qiyvklx8qgcqqsb1swilx6drs1cdj8m-key5099lb # [ 88.247445] niks3-server[1112]: 2026/09/21 14:45:03 INFO Read proxy enabled — serving cache objects from S35100lb # [ 88.254556] niks3-server[1112]: 2026/09/21 14:45:03 INFO Using socket-activated listener address=0.0.0.0:57515101lb # [ 88.258234] niks3-server[1112]: 2026/09/21 14:45:03 INFO systemd watchdog enabled interval=15s5102lb: waiting for unit niks3.service5103lb # [ 88.259793] niks3-server[1112]: 2026/09/21 14:45:03 INFO Starting HTTP server address=0.0.0.0:57515104lb # [ 88.262329] niks3-server[1112]: 2026/09/21 14:45:03 INFO lead: acquired remote=192.168.1.3:436665105lb # [ 88.263547] systemd[1]: Started niks3 server.5106lb: (finished: waiting for unit niks3.service, in 0.03 seconds)5107lb: must succeed: curl -sf localhost:9901/clusters5108lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)5109worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5110worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5111lb: must succeed: curl -sf localhost:9901/clusters5112lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5113lb: must succeed: curl -sf localhost:9901/clusters5114lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5115worker1: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield5116worker1: (finished: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield, in 0.07 seconds)5117client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag after-niks3-restart >&25118worker1 # [ 88.749036] nix-grpc-daemon[1268]: ts=2026-09-21T14:45:04Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:429285119worker1 # [ 88.764271] nix-daemon[852]: accepted connection from pid 1268, user nix-grpc-daemon (trusted)5120client # don't know how to build these paths:5121client # /nix/store/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv5122worker1 # [ 88.788731] nix-grpc-daemon[1268]: ts=2026-09-21T14:45:04Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:429285123worker1 # [ 88.805462] nix-grpc-daemon[1268]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 22 ms (attempt 1/5)5124worker1 # [ 88.836503] nix-grpc-daemon[1268]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 48 ms (attempt 2/5)5125worker1 # [ 88.893680] nix-grpc-daemon[1268]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 167 ms (attempt 3/5)5126worker1 # [ 89.062757] nix-grpc-daemon[1268]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 219 ms (attempt 4/5)5127worker1 # [ 89.286617] nix-grpc-daemon[1268]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5128worker1 # [ 89.292346] nix-grpc-daemon[1268]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5129worker1 # [ 89.337464] nix-grpc-daemon[1268]: ts=2026-09-21T14:45:04Z level=debug event=want client=1 drv=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv cached=05130worker1 # [ 89.344113] nix-grpc-daemon[1268]: ts=2026-09-21T14:45:04Z level=debug event=assigned drv=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv worker=192.168.1.3:50051 clients=15131worker1 # [ 89.384206] nix-grpc-daemon[1268]: ts=2026-09-21T14:45:04Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:429285132worker1 # [ 89.401951] nix-grpc-daemon[1268]: ts=2026-09-21T14:45:05Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:429285133client # copying 1 paths to 'lb:50051'...5134client # copying path '/nix/store/mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv' to 'lb:50051'...5135worker1 # [ 89.416164] nix-grpc-daemon[1268]: ts=2026-09-21T14:45:05Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:429285136worker1 # [ 89.432383] nix-daemon[852]: accepted connection from pid 1268, user nix-grpc-daemon (trusted)5137worker1 # [ 89.460631] nix-grpc-daemon[1268]: ts=2026-09-21T14:45:05Z level=info event=niks3_push_started pid=13405138worker1 # [ 89.612940] nix-grpc-daemon[1340]: time=2026-09-21T14:45:05.220Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.key ca=""5139worker1 # [ 89.718480] nix-daemon[852]: accepted connection from pid 1353, user nix-grpc-daemon (trusted)5140worker1 # [ 89.736670] nix-daemon[852]: reaped child process 1356, status = succeeded5141worker1 # [ 89.836776] nix-daemon[852]: accepted connection from pid 1358, user nix-grpc-daemon (trusted)5142worker1 # [ 89.853097] nix-daemon[852]: reaped child process 1361, status = succeeded5143lb # [ 89.863537] niks3-server[1112]: 2026/09/21 14:45:05 INFO Received uploads request method=POST path=/api/pending_closures5144worker1 # [ 89.874938] nix-grpc-daemon[1340]: time=2026-09-21T14:45:05.482Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5145worker1 # [ 89.876730] nix-grpc-daemon[1340]: time=2026-09-21T14:45:05.482Z level=INFO msg="Uploading mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv (544B)"5146lb # [ 89.923243] niks3-server[1112]: 2026/09/21 14:45:05 INFO Registered completed upload object_key=nar/0hn539w6achi7azzyhpihsi61g85x9f535dxv438yq5y50ksrp1d.nar.zst5147lb # [ 89.937978] niks3-server[1112]: 2026/09/21 14:45:05 INFO Received sign narinfos request method=POST path=/api/pending_closures/27/sign5148worker1 # [ 89.941234] nix-grpc-daemon[1340]: time=2026-09-21T14:45:05.549Z level=INFO msg="Uploading 1 narinfos"5149lb # [ 89.949653] niks3-server[1112]: 2026/09/21 14:45:05 INFO Signed narinfos id=27 count=15150lb # [ 89.955672] niks3-server[1112]: 2026/09/21 14:45:05 INFO Registered completed upload object_key=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z.ls5151lb # [ 89.969255] niks3-server[1112]: 2026/09/21 14:45:05 INFO Received complete upload request method=POST path=/api/pending_closures/27/complete5152lb # [ 89.982548] niks3-server[1112]: 2026/09/21 14:45:05 INFO Completed upload id=275153worker1 # [ 89.978925] nix-grpc-daemon[1340]: time=2026-09-21T14:45:05.587Z level=INFO msg="Upload complete. (366ms)"5154worker1 # [ 89.996258] nix-grpc-daemon[1268]: ts=2026-09-21T14:45:05Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42928 duration_s=0 paths=1 nar_bytes_in=5445155worker1 # [ 90.010427] nix-grpc-daemon[1268]: ts=2026-09-21T14:45:05Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:429285156worker1 # [ 90.020643] nix-grpc-daemon[1340]: time=2026-09-21T14:45:05.599Z level=WARN msg="Failed to register uploaded object" key=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z.narinfo error="server returned 404: object is not pending upload\n"5157worker1 # [ 90.032329] nix-daemon[852]: accepted connection from pid 1268, user nix-grpc-daemon (trusted)5158worker1 # [ 90.038605] nix-daemon[852]: reaped child process 1338, status = succeeded5159client # building '/nix/store/mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv' on lb:50051...5160client # farm-a> node-a: building mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv5161worker1 # [ 90.045869] nix-daemon[852]: accepted connection from pid 1268, user nix-grpc-daemon (trusted)5162client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds5163client # farm-a> LOG-farm-a-after-niks3-restart5164worker1 # [ 90.685514] nix-daemon[852]: reaped child process 1365, status = succeeded5165worker1 # [ 90.832831] nix-daemon[852]: accepted connection from pid 1381, user nix-grpc-daemon (trusted)5166worker1 # [ 90.851724] nix-daemon[852]: reaped child process 1384, status = succeeded5167lb # [ 90.861406] niks3-server[1112]: 2026/09/21 14:45:06 INFO Received uploads request method=POST path=/api/pending_closures5168worker1 # [ 90.872654] nix-grpc-daemon[1340]: time=2026-09-21T14:45:06.479Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5169worker1 # [ 90.878661] nix-grpc-daemon[1340]: time=2026-09-21T14:45:06.479Z level=INFO msg="Uploading j3iswx3ynilvcycpyw0z667966c9kpzx-farm-a (144B)"5170lb # [ 90.924686] niks3-server[1112]: 2026/09/21 14:45:06 INFO Registered completed upload object_key=log/mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv5171worker1 # [ 90.931828] nix-grpc-daemon[1340]: time=2026-09-21T14:45:06.538Z level=INFO msg="Uploading 1 narinfos"5172lb # [ 90.937146] niks3-server[1112]: 2026/09/21 14:45:06 INFO Registered completed upload object_key=nar/1diw08qzlsh0zkrx4kvj1mvbc57g7r65j1f459ankpg9yc5rgcpx.nar.zst5173lb # [ 90.945801] niks3-server[1112]: 2026/09/21 14:45:06 INFO Received sign narinfos request method=POST path=/api/pending_closures/28/sign5174lb # [ 90.955228] niks3-server[1112]: 2026/09/21 14:45:06 INFO Signed narinfos id=28 count=15175lb # [ 90.960544] niks3-server[1112]: 2026/09/21 14:45:06 INFO Registered completed upload object_key=j3iswx3ynilvcycpyw0z667966c9kpzx.ls5176worker1 # [ 90.955877] nix-grpc-daemon[1340]: time=2026-09-21T14:45:06.564Z level=INFO msg="Upload complete. (298ms)"5177lb # [ 90.963788] niks3-server[1112]: 2026/09/21 14:45:06 INFO Received complete upload request method=POST path=/api/pending_closures/28/complete5178lb # [ 90.969261] niks3-server[1112]: 2026/09/21 14:45:06 INFO Completed upload id=285179lb # [ 90.972191] niks3-server[1112]: 2026/09/21 14:45:06 INFO Registered completed upload object_key=j3iswx3ynilvcycpyw0z667966c9kpzx.narinfo5180worker1 # [ 90.976663] nix-grpc-daemon[1268]: ts=2026-09-21T14:45:06Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42928 duration_s=0 drv=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv assign_id=1 outputs=15181worker1 # [ 90.995275] nix-grpc-daemon[1268]: ts=2026-09-21T14:45:06Z level=debug event=want client=1 drv=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv cached=05182worker1 # [ 90.999990] nix-grpc-daemon[1268]: ts=2026-09-21T14:45:06Z level=debug event=assigned drv=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv worker=192.168.1.3:50051 clients=15183worker1 # [ 91.004879] nix-daemon[852]: reaped child process 1363, status = succeeded5184worker1 # [ 91.020515] nix-grpc-daemon[1268]: ts=2026-09-21T14:45:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:429285185worker1 # [ 91.035196] nix-grpc-daemon[1268]: ts=2026-09-21T14:45:06Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:429285186worker1 # [ 91.050404] nix-grpc-daemon[1340]: time=2026-09-21T14:45:06.658Z level=INFO msg="All 1 paths already cached"5187client # copying 1 paths to 'lb:50051'...5188client # copying path '/nix/store/5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv' to 'lb:50051'...5189worker1 # [ 91.060597] nix-grpc-daemon[1268]: ts=2026-09-21T14:45:06Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:429285190worker1 # [ 91.074480] nix-daemon[852]: accepted connection from pid 1268, user nix-grpc-daemon (trusted)5191worker1 # [ 91.225905] nix-daemon[852]: accepted connection from pid 1390, user nix-grpc-daemon (trusted)5192worker1 # [ 91.245918] nix-daemon[852]: reaped child process 1393, status = succeeded5193worker1 # [ 91.346228] nix-daemon[852]: accepted connection from pid 1395, user nix-grpc-daemon (trusted)5194worker1 # [ 91.362960] nix-daemon[852]: reaped child process 1398, status = succeeded5195lb # [ 91.373973] niks3-server[1112]: 2026/09/21 14:45:06 INFO Received uploads request method=POST path=/api/pending_closures5196worker1 # [ 91.380094] nix-grpc-daemon[1340]: time=2026-09-21T14:45:06.988Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5197worker1 # [ 91.381545] nix-grpc-daemon[1340]: time=2026-09-21T14:45:06.988Z level=INFO msg="Uploading 5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv (656B)"5198lb # [ 91.406012] niks3-server[1112]: 2026/09/21 14:45:07 INFO Registered completed upload object_key=nar/1xpcisa10g5by3yknnrs1xwhxhf4s243h7gwzjii0799pwbhfjfa.nar.zst5199worker1 # [ 91.415349] nix-grpc-daemon[1340]: time=2026-09-21T14:45:07.023Z level=INFO msg="Uploading 1 narinfos"5200lb # [ 91.420616] niks3-server[1112]: 2026/09/21 14:45:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/29/sign5201lb # [ 91.434195] niks3-server[1112]: 2026/09/21 14:45:07 INFO Signed narinfos id=29 count=15202lb # [ 91.439593] niks3-server[1112]: 2026/09/21 14:45:07 INFO Registered completed upload object_key=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch.ls5203lb # [ 91.451252] niks3-server[1112]: 2026/09/21 14:45:07 INFO Received complete upload request method=POST path=/api/pending_closures/29/complete5204lb # [ 91.457184] niks3-server[1112]: 2026/09/21 14:45:07 INFO Completed upload id=295205worker1 # [ 91.453360] nix-grpc-daemon[1340]: time=2026-09-21T14:45:07.061Z level=INFO msg="Upload complete. (363ms)"5206lb # [ 91.462699] niks3-server[1112]: 2026/09/21 14:45:07 INFO Registered completed upload object_key=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch.narinfo5207worker1 # [ 91.460157] nix-grpc-daemon[1268]: ts=2026-09-21T14:45:07Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42928 duration_s=0 paths=1 nar_bytes_in=6565208worker1 # [ 91.474147] nix-grpc-daemon[1268]: ts=2026-09-21T14:45:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:429285209worker1 # [ 91.480425] nix-daemon[852]: accepted connection from pid 1268, user nix-grpc-daemon (trusted)5210worker1 # [ 91.485259] nix-daemon[852]: reaped child process 1388, status = succeeded5211client # building '/nix/store/5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv' on lb:50051...5212client # farm-b> node-a: building 5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv5213worker1 # [ 91.498679] nix-daemon[852]: accepted connection from pid 1268, user nix-grpc-daemon (trusted)5214client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds5215client # farm-b> LOG-farm-b-after-niks3-restart5216worker1 # [ 92.549175] nix-daemon[852]: reaped child process 1402, status = succeeded5217worker1 # [ 92.647370] nix-daemon[852]: accepted connection from pid 1418, user nix-grpc-daemon (trusted)5218worker1 # [ 92.663558] nix-daemon[852]: reaped child process 1421, status = succeeded5219lb # [ 92.675448] niks3-server[1112]: 2026/09/21 14:45:08 INFO Received uploads request method=POST path=/api/pending_closures5220worker1 # [ 92.681774] nix-grpc-daemon[1340]: time=2026-09-21T14:45:08.289Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5221worker1 # [ 92.683248] nix-grpc-daemon[1340]: time=2026-09-21T14:45:08.289Z level=INFO msg="Uploading myar0ljvsqm9h7fxyh6qrqkg0175r3xn-farm-b (192B)"5222lb # [ 92.718760] niks3-server[1112]: 2026/09/21 14:45:08 INFO Registered completed upload object_key=nar/0zvv5rxmyk9afay83j52m7ipyhrqivhg6h6a0ksjd658pxzwi78w.nar.zst5223worker1 # [ 92.729465] nix-grpc-daemon[1340]: time=2026-09-21T14:45:08.337Z level=INFO msg="Uploading 1 narinfos"5224lb # [ 92.735350] niks3-server[1112]: 2026/09/21 14:45:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/30/sign5225lb # [ 92.748574] niks3-server[1112]: 2026/09/21 14:45:08 INFO Registered completed upload object_key=log/5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv5226lb # [ 92.751595] niks3-server[1112]: 2026/09/21 14:45:08 INFO Signed narinfos id=30 count=15227worker1 # [ 92.760116] nix-grpc-daemon[1340]: time=2026-09-21T14:45:08.368Z level=INFO msg="Upload complete. (251ms)"5228lb # [ 92.766622] niks3-server[1112]: 2026/09/21 14:45:08 INFO Received complete upload request method=POST path=/api/pending_closures/30/complete5229lb # [ 92.775965] niks3-server[1112]: 2026/09/21 14:45:08 INFO Completed upload id=305230lb # [ 92.777720] niks3-server[1112]: 2026/09/21 14:45:08 INFO Registered completed upload object_key=myar0ljvsqm9h7fxyh6qrqkg0175r3xn.narinfo5231worker1 # [ 92.776936] nix-grpc-daemon[1268]: ts=2026-09-21T14:45:08Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42928 duration_s=1 drv=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv assign_id=2 outputs=15232lb # [ 92.784773] niks3-server[1112]: 2026/09/21 14:45:08 INFO Registered completed upload object_key=myar0ljvsqm9h7fxyh6qrqkg0175r3xn.ls5233worker1 # [ 92.793443] nix-grpc-daemon[1268]: ts=2026-09-21T14:45:08Z level=debug event=want client=1 drv=6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv cached=05234worker1 # [ 92.798086] nix-grpc-daemon[1268]: ts=2026-09-21T14:45:08Z level=debug event=assigned drv=6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv worker=192.168.1.3:50051 clients=15235worker1 # [ 92.802853] nix-daemon[852]: reaped child process 1400, status = succeeded5236worker1 # [ 92.820233] nix-grpc-daemon[1268]: ts=2026-09-21T14:45:08Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:429285237worker1 # [ 92.833893] nix-grpc-daemon[1268]: ts=2026-09-21T14:45:08Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:429285238worker1 # [ 92.848833] nix-grpc-daemon[1340]: time=2026-09-21T14:45:08.457Z level=INFO msg="All 2 paths already cached"5239client # copying 1 paths to 'lb:50051'...5240client # copying path '/nix/store/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv' to 'lb:50051'...5241worker1 # [ 92.858538] nix-grpc-daemon[1268]: ts=2026-09-21T14:45:08Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:429285242worker1 # [ 92.872418] nix-daemon[852]: accepted connection from pid 1268, user nix-grpc-daemon (trusted)5243worker1 # [ 93.017381] nix-daemon[852]: accepted connection from pid 1427, user nix-grpc-daemon (trusted)5244worker1 # [ 93.034450] nix-daemon[852]: reaped child process 1430, status = succeeded5245worker1 # [ 93.138100] nix-daemon[852]: accepted connection from pid 1432, user nix-grpc-daemon (trusted)5246worker1 # [ 93.154096] nix-daemon[852]: reaped child process 1435, status = succeeded5247lb # [ 93.163812] niks3-server[1112]: 2026/09/21 14:45:08 INFO Received uploads request method=POST path=/api/pending_closures5248worker1 # [ 93.173335] nix-grpc-daemon[1340]: time=2026-09-21T14:45:08.781Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5249worker1 # [ 93.174785] nix-grpc-daemon[1340]: time=2026-09-21T14:45:08.781Z level=INFO msg="Uploading 6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv (784B)"5250lb # [ 93.199243] niks3-server[1112]: 2026/09/21 14:45:08 INFO Registered completed upload object_key=nar/1fnvw3c0v6n1j60nd5iagl3bggzcvixsw103w98mwhnpmnmgb248.nar.zst5251worker1 # [ 93.208805] nix-grpc-daemon[1340]: time=2026-09-21T14:45:08.817Z level=INFO msg="Uploading 1 narinfos"5252lb # [ 93.216570] niks3-server[1112]: 2026/09/21 14:45:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/31/sign5253lb # [ 93.227333] niks3-server[1112]: 2026/09/21 14:45:08 INFO Signed narinfos id=31 count=15254lb # [ 93.234265] niks3-server[1112]: 2026/09/21 14:45:08 INFO Registered completed upload object_key=6j7gr603xfslr5lxcbh72430f5q1r4b0.ls5255worker1 # [ 93.242008] nix-grpc-daemon[1340]: time=2026-09-21T14:45:08.850Z level=INFO msg="Upload complete. (355ms)"5256lb # [ 93.246798] niks3-server[1112]: 2026/09/21 14:45:08 INFO Received complete upload request method=POST path=/api/pending_closures/31/complete5257worker1 # [ 93.248242] nix-grpc-daemon[1268]: ts=2026-09-21T14:45:08Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42928 duration_s=0 paths=1 nar_bytes_in=7845258lb # [ 93.256317] niks3-server[1112]: 2026/09/21 14:45:08 INFO Completed upload id=315259lb # [ 93.259004] niks3-server[1112]: 2026/09/21 14:45:08 INFO Registered completed upload object_key=6j7gr603xfslr5lxcbh72430f5q1r4b0.narinfo5260worker1 # [ 93.262524] nix-grpc-daemon[1268]: ts=2026-09-21T14:45:08Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:429285261worker1 # [ 93.269652] nix-daemon[852]: accepted connection from pid 1268, user nix-grpc-daemon (trusted)5262worker1 # [ 93.274943] nix-daemon[852]: reaped child process 1425, status = succeeded5263client # building '/nix/store/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv' on lb:50051...5264worker1 # [ 93.286661] nix-daemon[852]: accepted connection from pid 1268, user nix-grpc-daemon (trusted)5265client # farm-top> node-a: building 6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv5266client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds5267client # farm-top> LOG-farm-top-after-niks3-restart5268worker1 # [ 94.368123] nix-daemon[852]: reaped child process 1439, status = succeeded5269worker1 # [ 94.469240] nix-daemon[852]: accepted connection from pid 1455, user nix-grpc-daemon (trusted)5270worker1 # [ 94.489294] nix-daemon[852]: reaped child process 1458, status = succeeded5271lb # [ 94.499853] niks3-server[1112]: 2026/09/21 14:45:10 INFO Received uploads request method=POST path=/api/pending_closures5272worker1 # [ 94.507424] nix-grpc-daemon[1340]: time=2026-09-21T14:45:10.114Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5273worker1 # [ 94.509380] nix-grpc-daemon[1340]: time=2026-09-21T14:45:10.114Z level=INFO msg="Uploading z8mx9hni3m7cymmz0ddfhg3jhqvi9mlc-farm-top (248B)"5274lb # [ 94.537525] niks3-server[1112]: 2026/09/21 14:45:10 INFO Registered completed upload object_key=log/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv5275lb # [ 94.549065] niks3-server[1112]: 2026/09/21 14:45:10 INFO Registered completed upload object_key=nar/0cypkbh15calmx90jgh2waxpq661jalf4c99dgb7j606z61xh0ld.nar.zst5276worker1 # [ 94.564984] nix-grpc-daemon[1340]: time=2026-09-21T14:45:10.173Z level=INFO msg="Uploading 1 narinfos"5277lb # [ 94.562430] niks3-server[1112]: 2026/09/21 14:45:10 INFO Received sign narinfos request method=POST path=/api/pending_closures/32/sign5278lb # [ 94.573290] niks3-server[1112]: 2026/09/21 14:45:10 INFO Signed narinfos id=32 count=15279lb # [ 94.579707] niks3-server[1112]: 2026/09/21 14:45:10 INFO Registered completed upload object_key=z8mx9hni3m7cymmz0ddfhg3jhqvi9mlc.ls5280lb # [ 94.593406] niks3-server[1112]: 2026/09/21 14:45:10 INFO Received complete upload request method=POST path=/api/pending_closures/32/complete5281lb # [ 94.600487] niks3-server[1112]: 2026/09/21 14:45:10 INFO Completed upload id=325282worker1 # [ 94.596056] nix-grpc-daemon[1340]: time=2026-09-21T14:45:10.204Z level=INFO msg="Upload complete. (297ms)"5283lb # [ 94.603526] niks3-server[1112]: 2026/09/21 14:45:10 INFO Registered completed upload object_key=z8mx9hni3m7cymmz0ddfhg3jhqvi9mlc.narinfo5284worker1 # [ 94.609346] nix-grpc-daemon[1268]: ts=2026-09-21T14:45:10Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42928 duration_s=1 drv=6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv assign_id=3 outputs=15285worker1 # [ 94.629507] nix-daemon[852]: reaped child process 1437, status = succeeded5286client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag after-niks3-restart >&2, in 6.13 seconds)5287client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag after-niks3-restart outPath5288client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag after-niks3-restart outPath, in 0.18 seconds)5289(finished: subtest: a niks3 restart does not cost the leader its role, in 11.62 seconds)5290subtest: scheduler down: the other node takes the lock and keeps it5291lb: must succeed: curl -sf localhost:9901/clusters5292lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)5293lb: must succeed: curl -sf localhost:9901/clusters5294lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5295lb: must succeed: curl -sf localhost:9901/clusters5296lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)5297lb: must succeed: curl -sf localhost:9901/clusters5298lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)5299worker1 # [ 95.074234] systemd[1]: Stopping Nix worker-protocol over gRPC...5300worker1 # [ 95.133324] nix-grpc-daemon[1268]: ts=2026-09-21T14:45:10Z level=info event=signal_exit5301lb # [ 95.264714] niks3-server[1112]: 2026/09/21 14:45:10 INFO lead: released remote=192.168.1.3:436665302worker2 # [ 96.242969] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:11Z level=info event=scheduler_restarting addr=lb:500515303worker1 # [ 96.267000] nix-grpc-daemon[1268]: ts=2026-09-21T14:45:11Z level=info event=scheduler_restarting5304worker1 # [ 96.271876] nix-grpc-daemon[1268]: ts=2026-09-21T14:45:11Z level=info event=scheduler_restarting addr=lb:500515305worker1 # [ 96.276672] nix-grpc-daemon[1268]: ts=2026-09-21T14:45:11Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:429125306worker1 # [ 96.281955] nix-grpc-daemon[1268]: ts=2026-09-21T14:45:11Z level=info event=worker_gone addr=192.168.1.4:500515307worker1 # [ 96.287233] nix-grpc-daemon[1268]: ts=2026-09-21T14:45:11Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:429125308worker1 # [ 96.296507] nix-grpc-daemon[1268]: ts=2026-09-21T14:45:11Z level=info event=worker_gone addr=192.168.1.3:500515309worker1 # [ 96.313766] nix-daemon[852]: reaped child process 1330, status = succeeded5310worker2 # [ 96.442535] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:12Z level=info event=scheduler_connected addr=lb:500515311worker2 # [ 96.447961] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5312worker1 # [ 96.482950] nix-grpc-daemon[1268]: ts=2026-09-21T14:45:12Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5313worker2 # [ 96.649188] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:12Z level=info event=scheduler_connected addr=lb:500515314worker2 # [ 96.654315] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5315worker2 # [ 96.855456] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:12Z level=info event=scheduler_connected addr=lb:500515316worker2 # [ 96.861062] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5317lb # [ 96.888576] envoy[888]: conn peer=[2001:db8:1::1]:50488 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9710 tx=4086 ms=58805318lb # [ 96.888819] envoy[888]: conn peer=[2001:db8:1::3]:36964 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=1065 tx=425 ms=132865319worker2 # [ 97.062102] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:12Z level=info event=scheduler_connected addr=lb:500515320worker2 # [ 97.067178] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5321worker1 # [ 97.178549] nix-grpc-daemon[1268]: E0000 00:00:1790001912.786818 1313 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 17, started true, shutdown true5322worker1 # [ 97.178955] nix-grpc-daemon[1268]: E0000 00:00:1790001912.787041 1313 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5323worker2 # [ 97.270004] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:12Z level=info event=scheduler_connected addr=lb:500515324worker2 # [ 97.274177] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5325worker2 # [ 97.470780] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:13Z level=info event=scheduler_connected addr=lb:500515326worker2 # [ 97.475977] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5327worker2 # [ 97.676971] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:13Z level=info event=scheduler_connected addr=lb:500515328worker2 # [ 97.682193] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5329worker2 # [ 97.883104] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:13Z level=info event=scheduler_connected addr=lb:500515330worker2 # [ 97.888455] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5331worker2 # [ 98.089599] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:13Z level=info event=scheduler_connected addr=lb:500515332worker2 # [ 98.094659] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5333worker2 # [ 98.227951] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:13Z level=info event=scheduler_take_over5334lb # [ 98.253166] niks3-server[1112]: 2026/09/21 14:45:13 INFO lead: acquired remote=192.168.1.4:594225335worker2 # [ 98.297496] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:13Z level=info event=scheduler_connected addr=lb:500515336worker2 # [ 98.301581] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5337worker1 # [ 98.416696] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5338worker1 # [ 98.421359] systemd[1]: Stopped Nix worker-protocol over gRPC.5339worker1 # [ 98.428399] systemd[1]: nix-grpc-daemon.service: Consumed 1.484s CPU time over 15.459s wall clock time, 38M memory peak, 36.1K incoming IP traffic, 56.4K outgoing IP traffic.5340worker1 # [ 98.442606] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5341worker1 # [ 98.454956] systemd[1]: Closed Nix worker-protocol over gRPC.5342worker2: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over5343worker1 # [ 98.458970] systemd[1]: nix-grpc-daemon.socket: Consumed 1ms CPU time over 15.508s wall clock time, 256K memory peak, 36.2K incoming IP traffic, 26.5K outgoing IP traffic.5344worker2 # [ 98.498124] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:14Z level=info event=scheduler_connected addr=lb:500515345worker2 # [ 98.520319] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:14Z level=info event=oidc_jwks provider=mock keys=15346worker2 # [ 98.523119] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:14Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:509525347worker2 # [ 98.533194] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:14Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=05348worker2: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over, in 0.10 seconds)5349worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5350worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)5351client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag on-standby >&25352worker2 # [ 98.785730] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:14Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:390225353client # don't know how to build these paths:5354client # /nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv5355worker2 # [ 98.795736] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:14Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:390225356worker2 # [ 98.823533] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:14Z level=debug event=want client=1 drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv cached=05357worker2 # [ 98.830165] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:14Z level=debug event=assigned drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv worker=192.168.1.4:50051 clients=15358worker2 # [ 98.874198] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:14Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:390225359worker2 # [ 98.889462] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:14Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:390225360client # copying 1 paths to 'lb:50051'...5361client # copying path '/nix/store/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv' to 'lb:50051'...5362worker2 # [ 98.905224] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:14Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:390225363worker2 # [ 98.920001] nix-daemon[838]: accepted connection from pid 1034, user nix-grpc-daemon (trusted)5364worker2 # [ 99.078651] nix-daemon[838]: accepted connection from pid 1129, user nix-grpc-daemon (trusted)5365worker2 # [ 99.097273] nix-daemon[838]: reaped child process 1132, status = succeeded5366worker2 # [ 99.204156] nix-daemon[838]: accepted connection from pid 1134, user nix-grpc-daemon (trusted)5367worker2 # [ 99.219665] nix-daemon[838]: reaped child process 1137, status = succeeded5368lb # [ 99.250321] niks3-server[1112]: 2026/09/21 14:45:14 INFO Received uploads request method=POST path=/api/pending_closures5369worker2 # [ 99.238639] nix-grpc-daemon[1094]: time=2026-09-21T14:45:14.841Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5370worker2 # [ 99.240162] nix-grpc-daemon[1094]: time=2026-09-21T14:45:14.841Z level=INFO msg="Uploading adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv (520B)"5371lb # [ 99.288773] niks3-server[1112]: 2026/09/21 14:45:14 INFO Registered completed upload object_key=nar/0scd0jwin78ibwq92wkajw2zj7adk8bih7yhx8bxz1706hh53507.nar.zst5372worker2 # [ 99.275445] nix-grpc-daemon[1094]: time=2026-09-21T14:45:14.878Z level=INFO msg="Uploading 1 narinfos"5373lb # [ 99.302748] niks3-server[1112]: 2026/09/21 14:45:14 INFO Received sign narinfos request method=POST path=/api/pending_closures/33/sign5374lb # [ 99.314940] niks3-server[1112]: 2026/09/21 14:45:14 INFO Signed narinfos id=33 count=15375lb # [ 99.321368] niks3-server[1112]: 2026/09/21 14:45:14 INFO Registered completed upload object_key=adj3g5w8jh83wl959p7daq6igl0gaay3.ls5376worker2 # [ 99.309895] nix-grpc-daemon[1094]: time=2026-09-21T14:45:14.912Z level=INFO msg="Upload complete. (372ms)"5377lb # [ 99.334502] niks3-server[1112]: 2026/09/21 14:45:14 INFO Received complete upload request method=POST path=/api/pending_closures/33/complete5378lb # [ 99.343800] niks3-server[1112]: 2026/09/21 14:45:14 INFO Completed upload id=335379worker2 # [ 99.319765] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:14Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39022 duration_s=0 paths=1 nar_bytes_in=5205380lb # [ 99.350174] niks3-server[1112]: 2026/09/21 14:45:14 INFO Registered completed upload object_key=adj3g5w8jh83wl959p7daq6igl0gaay3.narinfo5381worker2 # [ 99.333910] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:14Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:390225382worker2 # [ 99.342475] nix-daemon[838]: accepted connection from pid 1034, user nix-grpc-daemon (trusted)5383worker2 # [ 99.350215] nix-daemon[838]: reaped child process 1127, status = succeeded5384client # building '/nix/store/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv' on lb:50051...5385worker2 # [ 99.357758] nix-daemon[838]: accepted connection from pid 1034, user nix-grpc-daemon (trusted)5386client # farm-a> worker2: building adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv5387client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds5388client # farm-a> LOG-farm-a-on-standby5389worker2 # [ 100.632900] nix-daemon[838]: reaped child process 1141, status = succeeded5390worker2 # [ 100.738911] nix-daemon[838]: accepted connection from pid 1157, user nix-grpc-daemon (trusted)5391worker2 # [ 100.755897] nix-daemon[838]: reaped child process 1160, status = succeeded5392lb # [ 100.789573] niks3-server[1112]: 2026/09/21 14:45:16 INFO Received uploads request method=POST path=/api/pending_closures5393worker2 # [ 100.776427] nix-grpc-daemon[1094]: time=2026-09-21T14:45:16.377Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5394worker2 # [ 100.778269] nix-grpc-daemon[1094]: time=2026-09-21T14:45:16.377Z level=INFO msg="Uploading 7qyaw17jjsjxkkgb1kfm3wxvczk56aan-farm-a (136B)"5395lb # [ 100.826792] niks3-server[1112]: 2026/09/21 14:45:16 INFO Registered completed upload object_key=log/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv5396lb # [ 100.840575] niks3-server[1112]: 2026/09/21 14:45:16 INFO Registered completed upload object_key=nar/0x76wzgdd1nk8rm930z8l69v9961jpc8qhzmgganv0nhagq8lv9c.nar.zst5397worker2 # [ 100.830120] nix-grpc-daemon[1094]: time=2026-09-21T14:45:16.432Z level=INFO msg="Uploading 1 narinfos"5398lb # [ 100.857046] niks3-server[1112]: 2026/09/21 14:45:16 INFO Received sign narinfos request method=POST path=/api/pending_closures/34/sign5399lb # [ 100.866804] niks3-server[1112]: 2026/09/21 14:45:16 INFO Signed narinfos id=34 count=15400lb # [ 100.872227] niks3-server[1112]: 2026/09/21 14:45:16 INFO Registered completed upload object_key=7qyaw17jjsjxkkgb1kfm3wxvczk56aan.ls5401lb # [ 100.877955] niks3-server[1112]: 2026/09/21 14:45:16 INFO Received complete upload request method=POST path=/api/pending_closures/34/complete5402lb # [ 100.888057] niks3-server[1112]: 2026/09/21 14:45:16 INFO Completed upload id=345403worker2 # [ 100.862747] nix-grpc-daemon[1094]: time=2026-09-21T14:45:16.465Z level=INFO msg="Upload complete. (291ms)"5404lb # [ 100.891421] niks3-server[1112]: 2026/09/21 14:45:16 INFO Registered completed upload object_key=7qyaw17jjsjxkkgb1kfm3wxvczk56aan.narinfo5405worker2 # [ 100.884850] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:16Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39022 duration_s=1 drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv assign_id=1 outputs=15406worker2 # [ 100.900332] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:16Z level=debug event=want client=1 drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv cached=05407worker2 # [ 100.905531] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:16Z level=debug event=assigned drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv worker=192.168.1.4:50051 clients=15408worker2 # [ 100.909673] nix-daemon[838]: reaped child process 1139, status = succeeded5409worker2 # [ 100.927702] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:390225410worker2 # [ 100.941779] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:16Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:390225411worker2 # [ 100.956893] nix-grpc-daemon[1094]: time=2026-09-21T14:45:16.559Z level=INFO msg="All 1 paths already cached"5412client # copying 1 paths to 'lb:50051'...5413client # copying path '/nix/store/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv' to 'lb:50051'...5414worker2 # [ 100.966654] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:16Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:390225415worker2 # [ 100.981151] nix-daemon[838]: accepted connection from pid 1034, user nix-grpc-daemon (trusted)5416worker2 # [ 101.133127] nix-daemon[838]: accepted connection from pid 1166, user nix-grpc-daemon (trusted)5417worker2 # [ 101.152320] nix-daemon[838]: reaped child process 1169, status = succeeded5418worker2 # [ 101.252560] nix-daemon[838]: accepted connection from pid 1171, user nix-grpc-daemon (trusted)5419worker2 # [ 101.271675] nix-daemon[838]: reaped child process 1174, status = succeeded5420lb # [ 101.295174] niks3-server[1112]: 2026/09/21 14:45:16 INFO Received uploads request method=POST path=/api/pending_closures5421worker2 # [ 101.281025] nix-grpc-daemon[1094]: time=2026-09-21T14:45:16.883Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5422worker2 # [ 101.282485] nix-grpc-daemon[1094]: time=2026-09-21T14:45:16.883Z level=INFO msg="Uploading s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv (640B)"5423lb # [ 101.328031] niks3-server[1112]: 2026/09/21 14:45:16 INFO Registered completed upload object_key=nar/0g5yl4v9pymang94vyp5ib5wi5jwf0bpp184y8qjavkpag12wmlv.nar.zst5424worker2 # [ 101.316923] nix-grpc-daemon[1094]: time=2026-09-21T14:45:16.919Z level=INFO msg="Uploading 1 narinfos"5425lb # [ 101.345081] niks3-server[1112]: 2026/09/21 14:45:16 INFO Received sign narinfos request method=POST path=/api/pending_closures/35/sign5426lb # [ 101.355365] niks3-server[1112]: 2026/09/21 14:45:16 INFO Signed narinfos id=35 count=15427lb # [ 101.361062] niks3-server[1112]: 2026/09/21 14:45:16 INFO Registered completed upload object_key=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g.ls5428lb # [ 101.370488] niks3-server[1112]: 2026/09/21 14:45:16 INFO Received complete upload request method=POST path=/api/pending_closures/35/complete5429lb # [ 101.376357] niks3-server[1112]: 2026/09/21 14:45:16 INFO Completed upload id=355430worker2 # [ 101.351613] nix-grpc-daemon[1094]: time=2026-09-21T14:45:16.954Z level=INFO msg="Upload complete. (355ms)"5431lb # [ 101.379487] niks3-server[1112]: 2026/09/21 14:45:16 INFO Registered completed upload object_key=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g.narinfo5432worker2 # [ 101.360273] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:16Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39022 duration_s=0 paths=1 nar_bytes_in=6405433worker2 # [ 101.373108] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:390225434worker2 # [ 101.382020] nix-daemon[838]: accepted connection from pid 1034, user nix-grpc-daemon (trusted)5435worker2 # [ 101.391672] nix-daemon[838]: reaped child process 1164, status = succeeded5436client # building '/nix/store/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv' on lb:50051...5437client # farm-b> worker2: building s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv5438worker2 # [ 101.399946] nix-daemon[838]: accepted connection from pid 1034, user nix-grpc-daemon (trusted)5439client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds5440client # farm-b> LOG-farm-b-on-standby5441worker2 # [ 102.859204] nix-daemon[838]: reaped child process 1178, status = succeeded5442worker2 # [ 102.964052] nix-daemon[838]: accepted connection from pid 1194, user nix-grpc-daemon (trusted)5443worker2 # [ 102.981800] nix-daemon[838]: reaped child process 1197, status = succeeded5444lb # [ 103.014035] niks3-server[1112]: 2026/09/21 14:45:18 INFO Received uploads request method=POST path=/api/pending_closures5445worker2 # [ 103.003805] nix-grpc-daemon[1094]: time=2026-09-21T14:45:18.605Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5446worker2 # [ 103.005774] nix-grpc-daemon[1094]: time=2026-09-21T14:45:18.605Z level=INFO msg="Uploading 6hsxxv65wr2jz8cgv2v66iq16niq1gsv-farm-b (184B)"5447lb # [ 103.051845] niks3-server[1112]: 2026/09/21 14:45:18 INFO Registered completed upload object_key=nar/1grn64wcl2mc78k1jvrz5mcrk1ld70lhf8b9jpiqy8mvs6gfzq0v.nar.zst5448lb # [ 103.070279] niks3-server[1112]: 2026/09/21 14:45:18 INFO Registered completed upload object_key=log/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv5449worker2 # [ 103.055142] nix-grpc-daemon[1094]: time=2026-09-21T14:45:18.657Z level=INFO msg="Uploading 1 narinfos"5450lb # [ 103.080776] niks3-server[1112]: 2026/09/21 14:45:18 INFO Received sign narinfos request method=POST path=/api/pending_closures/36/sign5451lb # [ 103.089770] niks3-server[1112]: 2026/09/21 14:45:18 INFO Signed narinfos id=36 count=15452lb # [ 103.091807] niks3-server[1112]: 2026/09/21 14:45:18 INFO Registered completed upload object_key=6hsxxv65wr2jz8cgv2v66iq16niq1gsv.ls5453lb # [ 103.102750] niks3-server[1112]: 2026/09/21 14:45:18 INFO Received complete upload request method=POST path=/api/pending_closures/36/complete5454lb # [ 103.108959] niks3-server[1112]: 2026/09/21 14:45:18 INFO Completed upload id=365455worker2 # [ 103.084046] nix-grpc-daemon[1094]: time=2026-09-21T14:45:18.686Z level=INFO msg="Upload complete. (273ms)"5456lb # [ 103.111031] niks3-server[1112]: 2026/09/21 14:45:18 INFO Registered completed upload object_key=6hsxxv65wr2jz8cgv2v66iq16niq1gsv.narinfo5457worker2 # [ 103.101197] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:18Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39022 duration_s=1 drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv assign_id=2 outputs=15458worker2 # [ 103.116377] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:18Z level=debug event=want client=1 drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv cached=05459worker2 # [ 103.122674] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:18Z level=debug event=assigned drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv worker=192.168.1.4:50051 clients=15460worker2 # [ 103.127156] nix-daemon[838]: reaped child process 1176, status = succeeded5461worker2 # [ 103.144121] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:18Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:390225462worker2 # [ 103.159094] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:18Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:390225463worker2 # [ 103.173320] nix-grpc-daemon[1094]: time=2026-09-21T14:45:18.776Z level=INFO msg="All 2 paths already cached"5464client # copying 1 paths to 'lb:50051'...5465client # copying path '/nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv' to 'lb:50051'...5466worker2 # [ 103.182968] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:18Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:390225467worker2 # [ 103.196114] nix-daemon[838]: accepted connection from pid 1034, user nix-grpc-daemon (trusted)5468worker2 # [ 103.348854] nix-daemon[838]: accepted connection from pid 1203, user nix-grpc-daemon (trusted)5469worker2 # [ 103.367913] nix-daemon[838]: reaped child process 1206, status = succeeded5470worker2 # [ 103.469329] nix-daemon[838]: accepted connection from pid 1208, user nix-grpc-daemon (trusted)5471worker2 # [ 103.488117] nix-daemon[838]: reaped child process 1211, status = succeeded5472lb # [ 103.511477] niks3-server[1112]: 2026/09/21 14:45:19 INFO Received uploads request method=POST path=/api/pending_closures5473worker2 # [ 103.497079] nix-grpc-daemon[1094]: time=2026-09-21T14:45:19.099Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5474worker2 # [ 103.498527] nix-grpc-daemon[1094]: time=2026-09-21T14:45:19.099Z level=INFO msg="Uploading 7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv (768B)"5475lb # [ 103.539218] niks3-server[1112]: 2026/09/21 14:45:19 INFO Registered completed upload object_key=nar/1zyx46mv29cfygdhc5ai40qr4vl88lkr1lilm0h72j23w6wakgmc.nar.zst5476lb # [ 103.554277] niks3-server[1112]: 2026/09/21 14:45:19 INFO Received sign narinfos request method=POST path=/api/pending_closures/37/sign5477worker2 # [ 103.536222] nix-grpc-daemon[1094]: time=2026-09-21T14:45:19.138Z level=INFO msg="Uploading 1 narinfos"5478lb # [ 103.566366] niks3-server[1112]: 2026/09/21 14:45:19 INFO Signed narinfos id=37 count=15479lb # [ 103.572501] niks3-server[1112]: 2026/09/21 14:45:19 INFO Registered completed upload object_key=7s5szig4ffga53df6rls98r9lhxsrsc5.ls5480lb # [ 103.581671] niks3-server[1112]: 2026/09/21 14:45:19 INFO Received complete upload request method=POST path=/api/pending_closures/37/complete5481worker2 # [ 103.562390] nix-grpc-daemon[1094]: time=2026-09-21T14:45:19.165Z level=INFO msg="Upload complete. (351ms)"5482lb # [ 103.590234] niks3-server[1112]: 2026/09/21 14:45:19 INFO Completed upload id=375483lb # [ 103.593924] niks3-server[1112]: 2026/09/21 14:45:19 INFO Registered completed upload object_key=7s5szig4ffga53df6rls98r9lhxsrsc5.narinfo5484worker2 # [ 103.572231] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:19Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39022 duration_s=0 paths=1 nar_bytes_in=7685485worker2 # [ 103.583025] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:390225486worker2 # [ 103.590405] nix-daemon[838]: accepted connection from pid 1034, user nix-grpc-daemon (trusted)5487worker2 # [ 103.595357] nix-daemon[838]: reaped child process 1201, status = succeeded5488client # building '/nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv' on lb:50051...5489worker2 # [ 103.609635] nix-daemon[838]: accepted connection from pid 1034, user nix-grpc-daemon (trusted)5490client # farm-top> worker2: building 7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv5491client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds5492client # farm-top> LOG-farm-top-on-standby5493worker2 # [ 104.916904] nix-daemon[838]: reaped child process 1215, status = succeeded5494worker2 # [ 104.986638] nix-daemon[838]: accepted connection from pid 1231, user nix-grpc-daemon (trusted)5495worker2 # [ 105.006726] nix-daemon[838]: reaped child process 1234, status = succeeded5496lb # [ 105.037842] niks3-server[1112]: 2026/09/21 14:45:20 INFO Received uploads request method=POST path=/api/pending_closures5497worker2 # [ 105.022736] nix-grpc-daemon[1094]: time=2026-09-21T14:45:20.623Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5498worker2 # [ 105.024599] nix-grpc-daemon[1094]: time=2026-09-21T14:45:20.624Z level=INFO msg="Uploading 1ini8mfd1rzqyp8jgqihp4k889588mpx-farm-top (240B)"5499lb # [ 105.075001] niks3-server[1112]: 2026/09/21 14:45:20 INFO Registered completed upload object_key=log/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv5500worker2 # [ 105.065043] nix-grpc-daemon[1094]: time=2026-09-21T14:45:20.667Z level=INFO msg="Uploading 1 narinfos"5501lb # [ 105.089154] niks3-server[1112]: 2026/09/21 14:45:20 INFO Received sign narinfos request method=POST path=/api/pending_closures/38/sign5502lb # [ 105.100358] niks3-server[1112]: 2026/09/21 14:45:20 INFO Signed narinfos id=38 count=15503lb # [ 105.107693] niks3-server[1112]: 2026/09/21 14:45:20 INFO Registered completed upload object_key=nar/1mncazpqn39nifc2fwbx80la5i264r8xc1lj8fnlbx006znc1925.nar.zst5504lb # [ 105.120903] niks3-server[1112]: 2026/09/21 14:45:20 INFO Registered completed upload object_key=1ini8mfd1rzqyp8jgqihp4k889588mpx.ls5505lb # [ 105.132939] niks3-server[1112]: 2026/09/21 14:45:20 INFO Received complete upload request method=POST path=/api/pending_closures/38/complete5506worker2 # [ 105.114917] nix-grpc-daemon[1094]: time=2026-09-21T14:45:20.717Z level=INFO msg="Upload complete. (295ms)"5507lb # [ 105.141185] niks3-server[1112]: 2026/09/21 14:45:20 INFO Completed upload id=385508lb # [ 105.148266] niks3-server[1112]: 2026/09/21 14:45:20 INFO Registered completed upload object_key=1ini8mfd1rzqyp8jgqihp4k889588mpx.narinfo5509worker2 # [ 105.129129] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:20Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39022 duration_s=1 drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv assign_id=3 outputs=15510worker2 # [ 105.145890] nix-daemon[838]: reaped child process 1213, status = succeeded5511client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag on-standby >&2, in 6.59 seconds)5512client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag on-standby outPath5513client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag on-standby outPath, in 0.17 seconds)5514worker1 # [ 105.431050] systemd[1]: Listening on Nix worker-protocol over gRPC.5515worker1 # [ 105.443146] systemd[1]: Starting Nix worker-protocol over gRPC...5516worker1 # [ 105.579760] nix-grpc-daemon[1476]: ts=2026-09-21T14:45:21Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5517worker1 # [ 105.582231] nix-grpc-daemon[1476]: ts=2026-09-21T14:45:21Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5518worker1 # [ 105.586002] nix-grpc-daemon[1476]: ts=2026-09-21T14:45:21Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15519worker1 # [ 105.591266] nix-grpc-daemon[1476]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5520worker1 # [ 105.604071] nix-grpc-daemon[1476]: E0000 00:00:1790001921.199047 1476 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5521lb: must succeed: curl -sf localhost:9901/clusters5522worker1 # [ 105.632160] nix-grpc-daemon[1476]: E0000 00:00:1790001921.202578 1476 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5523worker1 # [ 105.634437] nix-grpc-daemon[1476]: ts=2026-09-21T14:45:21Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5524worker1 # [ 105.641680] nix-grpc-daemon[1476]: E0000 00:00:1790001921.211242 1518 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5525worker1 # [ 105.644576] nix-grpc-daemon[1476]: E0000 00:00:1790001921.229620 1518 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5526worker1 # [ 105.647006] nix-grpc-daemon[1476]: E0000 00:00:1790001921.229835 1518 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5527worker1 # [ 105.649462] systemd[1]: Started Nix worker-protocol over gRPC.5528worker1 # [ 105.650369] nix-grpc-daemon[1476]: E0000 00:00:1790001921.249413 1518 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5529worker1 # [ 105.659535] nix-grpc-daemon[1476]: ts=2026-09-21T14:45:21Z level=info event=scheduler_connected addr=lb:500515530worker2 # [ 105.642940] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:21Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:509525531worker2 # [ 105.648549] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:21Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=05532lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5533worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5534worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5535lb: must succeed: curl -sf localhost:9901/clusters5536lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)5537lb: must succeed: curl -sf localhost:9901/clusters5538lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5539worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5540worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5541(finished: subtest: scheduler down: the other node takes the lock and keeps it, in 11.11 seconds)5542subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise5543client: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag rs5544client # [ 106.007883] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag rs.5545client # Running as unit: rs.service; invocation ID: d76edc67bf7e450abc6182afa90620735546client: (finished: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag rs, in 0.09 seconds)5547client # [ 106.222686] nix[1239]: don't know how to build these paths:5548client # [ 106.223721] nix[1239]: /nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv5549worker2 # [ 106.204947] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:21Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:390225550worker2 # [ 106.218028] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:21Z level=debug event=want client=2 drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv cached=05551worker2 # [ 106.225205] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:21Z level=debug event=assigned drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv worker=192.168.1.4:50051 clients=15552worker2 # [ 106.264860] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:21Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:390225553worker2 # [ 106.282535] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:21Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:390225554client # [ 106.315241] nix[1239]: copying 1 paths to 'lb:50051'...5555worker2 # [ 106.298798] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:21Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:390225556client # [ 106.317917] nix[1239]: copying path '/nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv' to 'lb:50051'...5557worker2 # [ 106.313555] nix-daemon[838]: accepted connection from pid 1034, user nix-grpc-daemon (trusted)5558worker2 # [ 106.475295] nix-daemon[838]: accepted connection from pid 1248, user nix-grpc-daemon (trusted)5559worker2 # [ 106.496264] nix-daemon[838]: reaped child process 1251, status = succeeded5560worker2 # [ 106.604615] nix-daemon[838]: accepted connection from pid 1253, user nix-grpc-daemon (trusted)5561worker2 # [ 106.625659] nix-daemon[838]: reaped child process 1256, status = succeeded5562lb # [ 106.649923] niks3-server[1112]: 2026/09/21 14:45:22 INFO Received uploads request method=POST path=/api/pending_closures5563worker2 # [ 106.635045] nix-grpc-daemon[1094]: time=2026-09-21T14:45:22.237Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5564worker2 # [ 106.636638] nix-grpc-daemon[1094]: time=2026-09-21T14:45:22.237Z level=INFO msg="Uploading ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv (416B)"5565lb # [ 106.678325] niks3-server[1112]: 2026/09/21 14:45:22 INFO Registered completed upload object_key=nar/0lynldv3n40qnhi1516wgiry3i3pja3sanwyyd8q4nslr915d7r1.nar.zst5566lb # [ 106.692471] niks3-server[1112]: 2026/09/21 14:45:22 INFO Received sign narinfos request method=POST path=/api/pending_closures/39/sign5567worker2 # [ 106.673662] nix-grpc-daemon[1094]: time=2026-09-21T14:45:22.276Z level=INFO msg="Uploading 1 narinfos"5568lb # [ 106.701803] niks3-server[1112]: 2026/09/21 14:45:22 INFO Signed narinfos id=39 count=15569lb # [ 106.711491] niks3-server[1112]: 2026/09/21 14:45:22 INFO Registered completed upload object_key=ajji8m1zd5q5wi64gp7vclb5dd8hr19m.ls5570worker2 # [ 106.699587] nix-grpc-daemon[1094]: time=2026-09-21T14:45:22.302Z level=INFO msg="Upload complete. (369ms)"5571lb # [ 106.724262] niks3-server[1112]: 2026/09/21 14:45:22 INFO Received complete upload request method=POST path=/api/pending_closures/39/complete5572lb # [ 106.733330] niks3-server[1112]: 2026/09/21 14:45:22 INFO Completed upload id=395573worker2 # [ 106.708354] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:22Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39022 duration_s=0 paths=1 nar_bytes_in=4165574lb # [ 106.738692] niks3-server[1112]: 2026/09/21 14:45:22 INFO Registered completed upload object_key=ajji8m1zd5q5wi64gp7vclb5dd8hr19m.narinfo5575worker2 # [ 106.716571] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:22Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:390225576worker2 # [ 106.725792] nix-daemon[838]: accepted connection from pid 1034, user nix-grpc-daemon (trusted)5577worker2 # [ 106.730260] nix-daemon[838]: reaped child process 1246, status = succeeded5578client # [ 106.756578] nix[1239]: building '/nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv' on lb:50051...5579worker2 # [ 106.741723] nix-daemon[838]: accepted connection from pid 1034, user nix-grpc-daemon (trusted)5580lb # [ 106.887901] envoy[888]: conn peer=[2001:db8:1::1]:35426 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9705 tx=4096 ms=63545581lb: must succeed: curl -sf localhost:9901/clusters5582lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5583lb: must succeed: curl -sf localhost:9901/clusters5584lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5585lb: must succeed: curl -sf localhost:9901/clusters5586lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5587lb: must succeed: curl -sf localhost:9901/clusters5588lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)5589worker1: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true5590worker1 # [ 108.845778] systemd-journald[273]: Received client request to rotate journal, rotating.5591worker1 # [ 108.866073] systemd-journald[273]: Vacuuming done, freed 0B of archived journals from /var/log/journal/5cb0941880ae47a1834ad4fcf5bbc469.5592worker1: (finished: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true, in 0.09 seconds)5593worker2: must succeed: systemctl restart --no-block nix-grpc-daemon.service5594worker2 # [ 108.899640] systemd[1]: Stopping Nix worker-protocol over gRPC...5595worker2: (finished: must succeed: systemctl restart --no-block nix-grpc-daemon.service, in 0.05 seconds)5596client: must succeed: sleep 5.05597worker2 # [ 109.381996] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:24Z level=info event=draining5598client: (finished: must succeed: sleep 5.0, in 5.05 seconds)5599worker1: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr='5600worker2 # [ 114.450913] nix-daemon[838]: reaped child process 1260, status = succeeded5601worker2 # [ 114.523788] nix-daemon[838]: accepted connection from pid 1301, user nix-grpc-daemon (trusted)5602worker2 # [ 114.542267] nix-daemon[838]: reaped child process 1304, status = succeeded5603lb # [ 114.572795] niks3-server[1112]: 2026/09/21 14:45:30 INFO Received uploads request method=POST path=/api/pending_closures5604worker2 # [ 114.560059] nix-grpc-daemon[1094]: time=2026-09-21T14:45:30.160Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5605worker2 # [ 114.561760] nix-grpc-daemon[1094]: time=2026-09-21T14:45:30.160Z level=INFO msg="Uploading k48d52qj0czpjmzj15yq3fd14zfw5iq6-slow-rs (120B)"5606lb # [ 114.612701] niks3-server[1112]: 2026/09/21 14:45:30 INFO Registered completed upload object_key=nar/19z3qdigvc4gggg31dgc0wi5w85pgaq9dk7bvngh0yhckpy55jgq.nar.zst5607lb # [ 114.625142] niks3-server[1112]: 2026/09/21 14:45:30 INFO Registered completed upload object_key=log/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv5608worker2 # [ 114.608145] nix-grpc-daemon[1094]: time=2026-09-21T14:45:30.210Z level=INFO msg="Uploading 1 narinfos"5609lb # [ 114.638297] niks3-server[1112]: 2026/09/21 14:45:30 INFO Received sign narinfos request method=POST path=/api/pending_closures/40/sign5610lb # [ 114.646324] niks3-server[1112]: 2026/09/21 14:45:30 INFO Signed narinfos id=40 count=15611lb # [ 114.654901] niks3-server[1112]: 2026/09/21 14:45:30 INFO Registered completed upload object_key=k48d52qj0czpjmzj15yq3fd14zfw5iq6.ls5612lb # [ 114.659483] niks3-server[1112]: 2026/09/21 14:45:30 INFO Received complete upload request method=POST path=/api/pending_closures/40/complete5613lb # [ 114.666286] niks3-server[1112]: 2026/09/21 14:45:30 INFO Completed upload id=405614lb # [ 114.670795] niks3-server[1112]: 2026/09/21 14:45:30 INFO Registered completed upload object_key=k48d52qj0czpjmzj15yq3fd14zfw5iq6.narinfo5615worker2 # [ 114.642029] nix-grpc-daemon[1094]: time=2026-09-21T14:45:30.244Z level=INFO msg="Upload complete. (286ms)"5616worker2 # [ 114.657467] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:30Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39022 duration_s=7 drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv assign_id=4 outputs=15617worker2 # [ 114.674865] nix-daemon[838]: reaped child process 1258, status = succeeded5618client # [ 114.705896] systemd[1]: rs.service: Deactivated successfully.5619client # [ 114.711039] systemd[1]: rs.service: Consumed 202ms CPU time over 8.699s wall clock time, 41.2M memory peak, 7.2K incoming IP traffic, 8.4K outgoing IP traffic.5620worker2 # [ 115.385110] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:30Z level=info event=signal_exit5621lb # [ 116.258034] niks3-server[1112]: 2026/09/21 14:45:31 INFO lead: released remote=192.168.1.4:594225622lb # [ 116.890321] envoy[888]: conn peer=[2001:db8:1::1]:56808 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2979 tx=1430 ms=84645623worker1 # [ 117.255750] nix-grpc-daemon[1476]: ts=2026-09-21T14:45:32Z level=info event=scheduler_restarting addr=lb:500515624worker2 # [ 117.238986] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:32Z level=info event=scheduler_restarting5625worker2 # [ 117.239438] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:32Z level=info event=scheduler_restarting addr=lb:500515626worker2 # [ 117.239566] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:32Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:509525627worker2 # [ 117.239695] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:32Z level=info event=worker_gone addr=192.168.1.4:500515628worker2 # [ 117.239807] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:32Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:509525629worker2 # [ 117.239913] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:32Z level=info event=worker_gone addr=192.168.1.3:500515630worker2 # [ 117.287263] nix-daemon[838]: reaped child process 1086, status = succeeded5631worker1 # [ 117.455835] nix-grpc-daemon[1476]: ts=2026-09-21T14:45:33Z level=info event=scheduler_connected addr=lb:500515632worker1 # [ 117.466286] nix-grpc-daemon[1476]: ts=2026-09-21T14:45:33Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5633worker2 # [ 117.455754] nix-grpc-daemon[1034]: ts=2026-09-21T14:45:33Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5634worker2 # [ 117.464418] nix-grpc-daemon[1034]: E0000 00:00:1790001933.066902 1079 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 18, started true, shutdown true5635worker2 # [ 117.470936] nix-grpc-daemon[1034]: E0000 00:00:1790001933.067351 1079 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5636worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr=', in 3.27 seconds)5637client: waiting for success: ! systemctl is-active rs5638client: (finished: waiting for success: ! systemctl is-active rs, in 0.06 seconds)5639client: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }5640client: (finished: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }, in 0.05 seconds)5641client: must fail: journalctl -u rs | grep -q 'reconnecting'5642client: (finished: must fail: journalctl -u rs | grep -q 'reconnecting', in 0.03 seconds)5643lb: must succeed: curl -sf localhost:9901/clusters5644worker1 # [ 117.663601] nix-grpc-daemon[1476]: ts=2026-09-21T14:45:33Z level=info event=scheduler_connected addr=lb:500515645worker1 # [ 117.669126] nix-grpc-daemon[1476]: ts=2026-09-21T14:45:33Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5646lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5647worker2 # [ 117.782294] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5648worker2 # [ 117.791157] systemd[1]: Stopped Nix worker-protocol over gRPC.5649worker2 # [ 117.796842] systemd[1]: nix-grpc-daemon.service: Consumed 2.157s CPU time over 39.571s wall clock time, 39.1M memory peak, 61.3K incoming IP traffic, 136.2K outgoing IP traffic.5650worker2 # [ 117.811169] systemd[1]: Starting scheduler-token.service...5651worker1 # [ 117.869909] nix-grpc-daemon[1476]: ts=2026-09-21T14:45:33Z level=info event=scheduler_connected addr=lb:500515652worker1 # [ 117.875171] nix-grpc-daemon[1476]: ts=2026-09-21T14:45:33Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5653worker2 # [ 117.888488] systemd[1]: scheduler-token.service: Deactivated successfully.5654worker2 # [ 117.893197] systemd[1]: Finished scheduler-token.service.5655worker2 # [ 117.895007] systemd[1]: scheduler-token.service: Consumed 47ms CPU time over 81ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5656worker2 # [ 117.916382] systemd[1]: Starting Nix worker-protocol over gRPC...5657worker2 # [ 118.028433] nix-grpc-daemon[1314]: ts=2026-09-21T14:45:33Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5658worker2 # [ 118.030671] nix-grpc-daemon[1314]: ts=2026-09-21T14:45:33Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5659worker2 # [ 118.034414] nix-grpc-daemon[1314]: ts=2026-09-21T14:45:33Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15660worker2 # [ 118.040076] nix-grpc-daemon[1314]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5661worker2 # [ 118.041519] nix-grpc-daemon[1314]: E0000 00:00:1790001933.641775 1314 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5662lb # [ 118.077112] niks3-server[1112]: 2026/09/21 14:45:33 INFO lead: acquired remote=192.168.1.4:599585663worker1 # [ 118.076257] nix-grpc-daemon[1476]: ts=2026-09-21T14:45:33Z level=info event=scheduler_connected addr=lb:500515664worker1 # [ 118.081745] nix-grpc-daemon[1476]: ts=2026-09-21T14:45:33Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5665worker2 # [ 118.081497] nix-grpc-daemon[1314]: E0000 00:00:1790001933.647936 1314 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5666worker2 # [ 118.084233] nix-grpc-daemon[1314]: ts=2026-09-21T14:45:33Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5667worker2 # [ 118.086001] nix-grpc-daemon[1314]: E0000 00:00:1790001933.653853 1354 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5668worker2 # [ 118.088288] nix-grpc-daemon[1314]: ts=2026-09-21T14:45:33Z level=info event=scheduler_take_over5669worker2 # [ 118.092135] nix-grpc-daemon[1314]: E0000 00:00:1790001933.669153 1354 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5670worker2 # [ 118.094518] nix-grpc-daemon[1314]: E0000 00:00:1790001933.669292 1354 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5671worker2 # [ 118.097200] nix-grpc-daemon[1314]: E0000 00:00:1790001933.683539 1354 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5672worker2 # [ 118.101716] systemd[1]: Started Nix worker-protocol over gRPC.5673worker2 # [ 118.102605] nix-grpc-daemon[1314]: ts=2026-09-21T14:45:33Z level=info event=scheduler_connected addr=lb:500515674worker2 # [ 118.103862] nix-grpc-daemon[1314]: ts=2026-09-21T14:45:33Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5675worker1 # [ 118.284471] nix-grpc-daemon[1476]: ts=2026-09-21T14:45:33Z level=info event=scheduler_connected addr=lb:500515676worker1 # [ 118.288621] nix-grpc-daemon[1476]: ts=2026-09-21T14:45:33Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5677worker2 # [ 118.302999] nix-grpc-daemon[1314]: ts=2026-09-21T14:45:33Z level=info event=scheduler_connected addr=lb:500515678worker2 # [ 118.307962] nix-grpc-daemon[1314]: ts=2026-09-21T14:45:33Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5679worker1 # [ 118.484830] nix-grpc-daemon[1476]: ts=2026-09-21T14:45:34Z level=info event=scheduler_connected addr=lb:500515680worker1 # [ 118.490615] nix-grpc-daemon[1476]: ts=2026-09-21T14:45:34Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5681worker2 # [ 118.509057] nix-grpc-daemon[1314]: ts=2026-09-21T14:45:34Z level=info event=scheduler_connected addr=lb:500515682worker2 # [ 118.514229] nix-grpc-daemon[1314]: ts=2026-09-21T14:45:34Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5683worker1 # [ 118.690608] nix-grpc-daemon[1476]: ts=2026-09-21T14:45:34Z level=info event=scheduler_connected addr=lb:500515684worker1 # [ 118.695880] nix-grpc-daemon[1476]: ts=2026-09-21T14:45:34Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5685lb: must succeed: curl -sf localhost:9901/clusters5686worker2 # [ 118.715264] nix-grpc-daemon[1314]: ts=2026-09-21T14:45:34Z level=info event=scheduler_connected addr=lb:500515687worker2 # [ 118.720739] nix-grpc-daemon[1314]: ts=2026-09-21T14:45:34Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5688lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5689worker1 # [ 118.896886] nix-grpc-daemon[1476]: ts=2026-09-21T14:45:34Z level=info event=scheduler_connected addr=lb:500515690worker2 # [ 118.882149] nix-grpc-daemon[1314]: ts=2026-09-21T14:45:34Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:349345691worker2 # [ 118.887395] nix-grpc-daemon[1314]: ts=2026-09-21T14:45:34Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=05692worker2 # [ 118.921716] nix-grpc-daemon[1314]: ts=2026-09-21T14:45:34Z level=info event=scheduler_connected addr=lb:500515693worker2 # [ 118.935132] nix-grpc-daemon[1314]: ts=2026-09-21T14:45:34Z level=info event=oidc_jwks provider=mock keys=15694worker2 # [ 118.939655] nix-grpc-daemon[1314]: ts=2026-09-21T14:45:34Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:349345695worker2 # [ 118.945098] nix-grpc-daemon[1314]: ts=2026-09-21T14:45:34Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=05696lb: must succeed: curl -sf localhost:9901/clusters5697lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5698worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5699worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)5700(finished: subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise, in 13.97 seconds)5701subtest: scheduler restart mid-build: build finishes, no double build5702client: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag mid5703client # [ 119.980979] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag mid.5704client # Running as unit: mid.service; invocation ID: c50a24620b6f47c29c720228249ab9b65705client: (finished: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag mid, in 0.09 seconds)5706worker2 # [ 120.173742] nix-grpc-daemon[1314]: ts=2026-09-21T14:45:35Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:349465707worker2 # [ 120.186791] nix-daemon[838]: accepted connection from pid 1314, user nix-grpc-daemon (trusted)5708client # [ 120.221429] nix[1275]: don't know how to build these paths:5709client # [ 120.222935] nix[1275]: /nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv5710worker2 # [ 120.216707] nix-grpc-daemon[1314]: ts=2026-09-21T14:45:35Z level=debug event=want client=1 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv cached=05711worker2 # [ 120.223458] nix-grpc-daemon[1314]: ts=2026-09-21T14:45:35Z level=debug event=assigned drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv worker=192.168.1.3:50051 clients=15712worker1 # [ 120.303920] nix-grpc-daemon[1476]: ts=2026-09-21T14:45:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:385825713worker1 # [ 120.317873] nix-daemon[852]: accepted connection from pid 1476, user nix-grpc-daemon (trusted)5714worker1 # [ 120.341427] nix-grpc-daemon[1476]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 8 ms (attempt 1/5)5715worker1 # [ 120.357841] nix-grpc-daemon[1476]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 197 ms (attempt 2/5)5716worker1 # [ 120.556948] nix-grpc-daemon[1476]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 335 ms (attempt 3/5)5717worker1 # [ 120.893913] nix-grpc-daemon[1476]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 277 ms (attempt 4/5)5718worker1 # [ 121.175600] nix-grpc-daemon[1476]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5719worker1 # [ 121.184466] nix-grpc-daemon[1476]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5720worker1 # [ 121.202711] nix-grpc-daemon[1476]: ts=2026-09-21T14:45:36Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:385825721client # [ 121.211308] nix[1275]: copying 1 paths to 'lb:50051'...5722worker1 # [ 121.214318] nix-grpc-daemon[1476]: ts=2026-09-21T14:45:36Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:385825723client # [ 121.211702] nix[1275]: copying path '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' to 'lb:50051'...5724worker1 # [ 121.232102] nix-daemon[852]: accepted connection from pid 1476, user nix-grpc-daemon (trusted)5725worker1 # [ 121.274088] nix-grpc-daemon[1476]: ts=2026-09-21T14:45:36Z level=info event=niks3_push_started pid=16005726worker1 # [ 121.400715] nix-grpc-daemon[1600]: time=2026-09-21T14:45:37.008Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.key ca=""5727worker1 # [ 121.509412] nix-daemon[852]: accepted connection from pid 1613, user nix-grpc-daemon (trusted)5728worker1 # [ 121.525725] nix-daemon[852]: reaped child process 1616, status = succeeded5729worker1 # [ 121.631263] nix-daemon[852]: accepted connection from pid 1618, user nix-grpc-daemon (trusted)5730worker1 # [ 121.647485] nix-daemon[852]: reaped child process 1621, status = succeeded5731lb # [ 121.658437] niks3-server[1112]: 2026/09/21 14:45:37 INFO Received uploads request method=POST path=/api/pending_closures5732worker1 # [ 121.664689] nix-grpc-daemon[1600]: time=2026-09-21T14:45:37.272Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5733worker1 # [ 121.666156] nix-grpc-daemon[1600]: time=2026-09-21T14:45:37.272Z level=INFO msg="Uploading chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv (424B)"5734lb # [ 121.694053] niks3-server[1112]: 2026/09/21 14:45:37 INFO Registered completed upload object_key=nar/14lxjrdcxwbhc1kgg39y29gnbx5cz80j4852d1i4w3rvq7g4b04g.nar.zst5735lb # [ 121.718450] niks3-server[1112]: 2026/09/21 14:45:37 INFO Received sign narinfos request method=POST path=/api/pending_closures/41/sign5736worker1 # [ 121.720046] nix-grpc-daemon[1600]: time=2026-09-21T14:45:37.328Z level=INFO msg="Uploading 1 narinfos"5737lb # [ 121.727390] niks3-server[1112]: 2026/09/21 14:45:37 INFO Signed narinfos id=41 count=15738lb # [ 121.734895] niks3-server[1112]: 2026/09/21 14:45:37 INFO Registered completed upload object_key=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc.ls5739lb # [ 121.747123] niks3-server[1112]: 2026/09/21 14:45:37 INFO Received complete upload request method=POST path=/api/pending_closures/41/complete5740worker1 # [ 121.750607] nix-grpc-daemon[1600]: time=2026-09-21T14:45:37.359Z level=INFO msg="Upload complete. (346ms)"5741lb # [ 121.758167] niks3-server[1112]: 2026/09/21 14:45:37 INFO Completed upload id=415742worker1 # [ 121.761380] nix-grpc-daemon[1476]: ts=2026-09-21T14:45:37Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38582 duration_s=0 paths=1 nar_bytes_in=4245743worker1 # [ 121.778624] nix-grpc-daemon[1476]: ts=2026-09-21T14:45:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:385825744worker1 # [ 121.788441] nix-grpc-daemon[1600]: time=2026-09-21T14:45:37.381Z level=WARN msg="Failed to register uploaded object" key=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc.narinfo error="server returned 404: object is not pending upload\n"5745worker1 # [ 121.800296] nix-daemon[852]: accepted connection from pid 1476, user nix-grpc-daemon (trusted)5746worker1 # [ 121.802842] nix-daemon[852]: reaped child process 1598, status = succeeded5747client # [ 121.806651] nix[1275]: building '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' on lb:50051...5748worker1 # [ 121.812402] nix-daemon[852]: accepted connection from pid 1476, user nix-grpc-daemon (trusted)5749lb: must succeed: curl -sf localhost:9901/clusters5750lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5751lb: must succeed: curl -sf localhost:9901/clusters5752lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5753worker2: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service5754worker2 # [ 123.920994] systemd[1]: nix-grpc-daemon.service: Sent signal SIGKILL to main process 1314 (nix-grpc-daemon) on client request.5755worker2 # [ 123.927522] systemd[1]: nix-grpc-daemon.service: Killed unit cgroup '/system.slice/nix-grpc-daemon.service' with SIGKILL on client request.5756lb # [ 123.962597] niks3-server[1112]: 2026/09/21 14:45:39 INFO lead: released remote=192.168.1.4:599585757worker1 # [ 123.964472] nix-grpc-daemon[1476]: ts=2026-09-21T14:45:39Z level=info event=scheduler_disconnected addr=lb:50051 error="Stream removed (RST_STREAM (Received RST_STREAM with error code 2))"5758client # [ 123.962678] nix[1275]: scheduler at lb:50051: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), reconnecting5759worker2 # [ 123.955019] systemd[1]: nix-grpc-daemon.service: Main process exited, code=killed, status=9/KILL5760worker2 # [ 123.975517] systemd[1]: nix-grpc-daemon.service: Failed with result 'signal'.5761worker2 # [ 123.984237] systemd[1]: nix-grpc-daemon.service: Consumed 223ms CPU time over 6.044s wall clock time, 14.4M memory peak, 10K incoming IP traffic, 14.8K outgoing IP traffic.5762worker2 # [ 123.986559] systemd[1]: Starting scheduler-token.service...5763worker2 # [ 123.991201] nix-daemon[838]: reaped child process 1373, status = succeeded5764worker2 # [ 124.033684] systemd[1]: scheduler-token.service: Deactivated successfully.5765worker2 # [ 124.037928] systemd[1]: Finished scheduler-token.service.5766worker2 # [ 124.039169] systemd[1]: scheduler-token.service: Consumed 25ms CPU time over 63ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5767worker2 # [ 124.047046] systemd[1]: nix-grpc-daemon.service: Scheduled restart job immediately on client request, restart counter is at 1.5768worker2 # [ 124.052236] systemd[1]: Starting scheduler-token.service...5769worker2 # [ 124.098924] systemd[1]: scheduler-token.service: Deactivated successfully.5770worker2 # [ 124.102043] systemd[1]: Finished scheduler-token.service.5771worker2 # [ 124.103034] systemd[1]: scheduler-token.service: Consumed 27ms CPU time over 54ms wall clock time, 2.3M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5772worker2 # [ 124.109498] systemd[1]: Starting Nix worker-protocol over gRPC...5773worker1 # [ 124.167413] nix-grpc-daemon[1476]: ts=2026-09-21T14:45:39Z level=info event=scheduler_connected addr=lb:500515774worker2 # [ 124.216749] nix-grpc-daemon[1407]: ts=2026-09-21T14:45:39Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5775worker2 # [ 124.218590] nix-grpc-daemon[1407]: ts=2026-09-21T14:45:39Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5776worker2 # [ 124.222249] nix-grpc-daemon[1407]: ts=2026-09-21T14:45:39Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15777worker2 # [ 124.228195] nix-grpc-daemon[1407]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5778lb # [ 124.266367] niks3-server[1112]: 2026/09/21 14:45:39 INFO lead: acquired remote=192.168.1.4:423525779worker2 # [ 124.260244] nix-grpc-daemon[1407]: E0000 00:00:1790001939.829575 1407 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5780worker2 # [ 124.262613] nix-grpc-daemon[1407]: E0000 00:00:1790001939.834524 1407 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5781worker2: (finished: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service, in 0.41 seconds)5782client: must succeed: sleep 5.05783worker2 # [ 124.280441] nix-grpc-daemon[1407]: ts=2026-09-21T14:45:39Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5784worker2 # [ 124.282676] nix-grpc-daemon[1407]: E0000 00:00:1790001939.842961 1447 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5785worker2 # [ 124.285319] nix-grpc-daemon[1407]: ts=2026-09-21T14:45:39Z level=info event=scheduler_take_over5786worker2 # [ 124.286423] nix-grpc-daemon[1407]: ts=2026-09-21T14:45:39Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:454105787worker2 # [ 124.290611] nix-grpc-daemon[1407]: ts=2026-09-21T14:45:39Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=15788worker2 # [ 124.294904] nix-grpc-daemon[1407]: E0000 00:00:1790001939.862840 1447 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5789worker2 # [ 124.297601] systemd[1]: Started Nix worker-protocol over gRPC.5790worker2 # [ 124.298870] nix-grpc-daemon[1407]: E0000 00:00:1790001939.868031 1447 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5791worker2 # [ 124.301644] nix-grpc-daemon[1407]: E0000 00:00:1790001939.882697 1447 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5792worker2 # [ 124.305685] nix-grpc-daemon[1407]: ts=2026-09-21T14:45:39Z level=info event=scheduler_connected addr=lb:500515793worker2 # [ 124.306950] nix-grpc-daemon[1407]: ts=2026-09-21T14:45:39Z level=info event=oidc_jwks provider=mock keys=15794worker2 # [ 124.308230] nix-grpc-daemon[1407]: ts=2026-09-21T14:45:39Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:454105795worker2 # [ 124.309887] nix-grpc-daemon[1407]: ts=2026-09-21T14:45:39Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=05796lb # [ 126.889794] envoy[888]: conn peer=[2001:db8:1::4]:59488 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=27833 tx=1725 ms=390165797lb # [ 126.895066] envoy[888]: conn peer=[2001:db8:1::4]:44096 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=4102 tx=385 ms=58585798client: (finished: must succeed: sleep 5.0, in 5.05 seconds)5799client: waiting for success: ! systemctl is-active mid5800worker1 # [ 129.675218] nix-daemon[852]: reaped child process 1625, status = succeeded5801worker1 # [ 129.789654] nix-daemon[852]: accepted connection from pid 1662, user nix-grpc-daemon (trusted)5802worker1 # [ 129.808670] nix-daemon[852]: reaped child process 1665, status = succeeded5803lb # [ 129.819455] niks3-server[1112]: 2026/09/21 14:45:45 INFO Received uploads request method=POST path=/api/pending_closures5804worker1 # [ 129.827736] nix-grpc-daemon[1600]: time=2026-09-21T14:45:45.433Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5805worker1 # [ 129.830617] nix-grpc-daemon[1600]: time=2026-09-21T14:45:45.433Z level=INFO msg="Uploading fk2vdwmn2msb3ja19s2jwak35ignz889-slow-mid (120B)"5806lb # [ 129.850327] niks3-server[1112]: 2026/09/21 14:45:45 INFO Registered completed upload object_key=log/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv5807lb # [ 129.885294] niks3-server[1112]: 2026/09/21 14:45:45 INFO Registered completed upload object_key=nar/0462jb7zwy2d73k3q6v09xnk3bx318v8ik5ckmp32w2hzvwj4k2d.nar.zst5808lb # [ 129.893638] niks3-server[1112]: 2026/09/21 14:45:45 INFO Received sign narinfos request method=POST path=/api/pending_closures/42/sign5809worker1 # [ 129.894774] nix-grpc-daemon[1600]: time=2026-09-21T14:45:45.501Z level=INFO msg="Uploading 1 narinfos"5810lb # [ 129.901257] niks3-server[1112]: 2026/09/21 14:45:45 INFO Signed narinfos id=42 count=15811lb # [ 129.903294] niks3-server[1112]: 2026/09/21 14:45:45 INFO Registered completed upload object_key=fk2vdwmn2msb3ja19s2jwak35ignz889.ls5812lb # [ 129.914574] niks3-server[1112]: 2026/09/21 14:45:45 INFO Received complete upload request method=POST path=/api/pending_closures/42/complete5813lb # [ 129.921394] niks3-server[1112]: 2026/09/21 14:45:45 INFO Completed upload id=425814worker1 # [ 129.916743] nix-grpc-daemon[1600]: time=2026-09-21T14:45:45.525Z level=INFO msg="Upload complete. (288ms)"5815worker1 # [ 129.929484] nix-grpc-daemon[1476]: ts=2026-09-21T14:45:45Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38582 duration_s=8 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv assign_id=1 outputs=15816worker1 # [ 129.947520] nix-grpc-daemon[1600]: time=2026-09-21T14:45:45.549Z level=WARN msg="Failed to register uploaded object" key=fk2vdwmn2msb3ja19s2jwak35ignz889.narinfo error="server returned 404: object is not pending upload\n"5817worker1 # [ 129.954997] nix-daemon[852]: reaped child process 1623, status = succeeded5818client # [ 129.957765] systemd[1]: mid.service: Deactivated successfully.5819client # [ 129.962411] systemd[1]: mid.service: Consumed 197ms CPU time over 9.978s wall clock time, 41.4M memory peak, 7.9K incoming IP traffic, 8.9K outgoing IP traffic.5820client: (finished: waiting for success: ! systemctl is-active mid, in 1.12 seconds)5821client: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }5822client: (finished: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }, in 0.05 seconds)5823(finished: subtest: scheduler restart mid-build: build finishes, no double build, in 10.84 seconds)5824subtest: interrupting the client stops the build on the worker5825client: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag intr5826client # [ 130.795526] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag intr.5827client # Running as unit: intr.service; invocation ID: 7f818f0441d94b4382af7024342248125828client: (finished: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag intr, in 0.05 seconds)5829worker2 # [ 130.954433] nix-grpc-daemon[1407]: ts=2026-09-21T14:45:46Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:334045830worker2 # [ 130.969688] nix-daemon[838]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)5831client # [ 131.003028] nix[1311]: don't know how to build these paths:5832client # [ 131.004543] nix[1311]: /nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv5833worker2 # [ 130.995081] nix-grpc-daemon[1407]: ts=2026-09-21T14:45:46Z level=debug event=want client=2 drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv cached=05834worker2 # [ 131.001966] nix-grpc-daemon[1407]: ts=2026-09-21T14:45:46Z level=debug event=assigned drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv worker=192.168.1.4:50051 clients=15835worker2 # [ 131.047668] nix-grpc-daemon[1407]: ts=2026-09-21T14:45:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:334045836worker2 # [ 131.062994] nix-grpc-daemon[1407]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 31 ms (attempt 1/5)5837worker2 # [ 131.102817] nix-grpc-daemon[1407]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 61 ms (attempt 2/5)5838worker2 # [ 131.166225] nix-grpc-daemon[1407]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 365 ms (attempt 3/5)5839worker2 # [ 131.540886] nix-grpc-daemon[1407]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 614 ms (attempt 4/5)5840worker2 # [ 132.158379] nix-grpc-daemon[1407]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5841worker2 # [ 132.165567] nix-grpc-daemon[1407]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5842worker2 # [ 132.182567] nix-grpc-daemon[1407]: ts=2026-09-21T14:45:47Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:334045843client # [ 132.214143] nix[1311]: copying 1 paths to 'lb:50051'...5844worker2 # [ 132.200096] nix-grpc-daemon[1407]: ts=2026-09-21T14:45:47Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:334045845client # [ 132.220182] nix[1311]: copying path '/nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv' to 'lb:50051'...5846worker2 # [ 132.217367] nix-daemon[838]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)5847worker2 # [ 132.252096] nix-grpc-daemon[1407]: ts=2026-09-21T14:45:47Z level=info event=niks3_push_started pid=14855848worker2 # [ 132.397160] nix-grpc-daemon[1485]: time=2026-09-21T14:45:47.999Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.key ca=""5849worker2 # [ 132.501654] nix-daemon[838]: accepted connection from pid 1497, user nix-grpc-daemon (trusted)5850worker2 # [ 132.517940] nix-daemon[838]: reaped child process 1500, status = succeeded5851worker2 # [ 132.619876] nix-daemon[838]: accepted connection from pid 1502, user nix-grpc-daemon (trusted)5852worker2 # [ 132.635959] nix-daemon[838]: reaped child process 1505, status = succeeded5853lb # [ 132.669566] niks3-server[1112]: 2026/09/21 14:45:48 INFO Received uploads request method=POST path=/api/pending_closures5854worker2 # [ 132.657639] nix-grpc-daemon[1485]: time=2026-09-21T14:45:48.259Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5855worker2 # [ 132.659294] nix-grpc-daemon[1485]: time=2026-09-21T14:45:48.259Z level=INFO msg="Uploading 85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv (424B)"5856lb # [ 132.717717] niks3-server[1112]: 2026/09/21 14:45:48 INFO Registered completed upload object_key=nar/0h460h76prjyvhqhyxlhz1xrmh3s520jbd19gaagy4hp4vc92r1p.nar.zst5857worker2 # [ 132.703482] nix-grpc-daemon[1485]: time=2026-09-21T14:45:48.306Z level=INFO msg="Uploading 1 narinfos"5858lb # [ 132.731891] niks3-server[1112]: 2026/09/21 14:45:48 INFO Received sign narinfos request method=POST path=/api/pending_closures/43/sign5859lb # [ 132.740576] niks3-server[1112]: 2026/09/21 14:45:48 INFO Signed narinfos id=43 count=15860lb # [ 132.748276] niks3-server[1112]: 2026/09/21 14:45:48 INFO Registered completed upload object_key=85wa5czf6m5ng93xyq44r1affb74csh8.ls5861lb # [ 132.751159] niks3-server[1112]: 2026/09/21 14:45:48 INFO Received complete upload request method=POST path=/api/pending_closures/43/complete5862lb # [ 132.757359] niks3-server[1112]: 2026/09/21 14:45:48 INFO Completed upload id=435863worker2 # [ 132.731766] nix-grpc-daemon[1485]: time=2026-09-21T14:45:48.334Z level=INFO msg="Upload complete. (331ms)"5864lb # [ 132.759603] niks3-server[1112]: 2026/09/21 14:45:48 INFO Registered completed upload object_key=85wa5czf6m5ng93xyq44r1affb74csh8.narinfo5865worker2 # [ 132.740324] nix-grpc-daemon[1407]: ts=2026-09-21T14:45:48Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33404 duration_s=0 paths=1 nar_bytes_in=4245866worker2 # [ 132.755609] nix-grpc-daemon[1407]: ts=2026-09-21T14:45:48Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:334045867worker2 # [ 132.764231] nix-daemon[838]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)5868worker2 # [ 132.768904] nix-daemon[838]: reaped child process 1483, status = succeeded5869client # [ 132.796811] nix[1311]: building '/nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv' on lb:50051...5870worker2 # [ 132.782765] nix-daemon[838]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)5871client: must succeed: systemctl kill -s INT intr5872worker2 # [ 134.407268] nix-grpc-daemon[1407]: ts=2026-09-21T14:45:50Z level=info event=revoked_running drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv5873client # [ 134.421304] systemd[1]: intr.service: Sent signal SIGINT to main process 1311 (nix) on client request.5874client # [ 134.430381] nix[1311]: shutting down85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED)5875client # [ 134.444417] nix[1311]: error: interrupted by the user5876client: (finished: must succeed: systemctl kill -s INT intr, in 0.07 seconds)5877client # [ 134.474981] systemd[1]: intr.service: Main process exited, code=exited, status=1/FAILURE5878client # [ 134.480795] systemd[1]: intr.service: Failed with result 'exit-code'.5879client # [ 134.483334] systemd[1]: intr.service: Consumed 168ms CPU time over 3.683s wall clock time, 41.4M memory peak, 6.5K incoming IP traffic, 8.3K outgoing IP traffic.5880worker2 # [ 134.607607] nix-daemon[838]: reaped child process 1507, status = succeeded5881worker2 # [ 134.611535] nix-daemon[1509]: unexpected Nix daemon error: error: interrupted by the user5882worker2 # [ 134.617858] nix-daemon[838]: reaped child process 1509, status = failed with exit code 15883(finished: subtest: interrupting the client stops the build on the worker, in 4.97 seconds)5884subtest: a deploy mid-build drains: the build finishes, the new generation takes over5885client: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag sw5886client # [ 135.792740] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag sw.5887client # Running as unit: sw.service; invocation ID: e4f2857809b04789b1751e6a7f9766ce5888client: (finished: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag sw, in 0.08 seconds)5889client # [ 135.993914] nix[1332]: don't know how to build these paths:5890client # [ 135.994867] nix[1332]: /nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv5891worker1 # [ 135.996463] nix-grpc-daemon[1476]: ts=2026-09-21T14:45:51Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:385825892worker2 # [ 135.989378] nix-grpc-daemon[1407]: ts=2026-09-21T14:45:51Z level=debug event=want client=3 drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv cached=05893worker2 # [ 135.994996] nix-grpc-daemon[1407]: ts=2026-09-21T14:45:51Z level=debug event=assigned drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv worker=192.168.1.3:50051 clients=15894worker1 # [ 136.056122] nix-grpc-daemon[1476]: ts=2026-09-21T14:45:51Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:385825895worker1 # [ 136.072442] nix-grpc-daemon[1476]: ts=2026-09-21T14:45:51Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:385825896client # [ 136.081608] nix[1332]: copying 1 paths to 'lb:50051'...5897client # [ 136.087251] nix[1332]: copying path '/nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv' to 'lb:50051'...5898worker1 # [ 136.089975] nix-grpc-daemon[1476]: ts=2026-09-21T14:45:51Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:385825899worker1 # [ 136.105069] nix-daemon[852]: accepted connection from pid 1476, user nix-grpc-daemon (trusted)5900worker1 # [ 136.264632] nix-daemon[852]: accepted connection from pid 1697, user nix-grpc-daemon (trusted)5901worker1 # [ 136.287800] nix-daemon[852]: reaped child process 1700, status = succeeded5902worker1 # [ 136.383926] nix-daemon[852]: accepted connection from pid 1702, user nix-grpc-daemon (trusted)5903worker1 # [ 136.400372] nix-daemon[852]: reaped child process 1705, status = succeeded5904lb # [ 136.411944] niks3-server[1112]: 2026/09/21 14:45:52 INFO Received uploads request method=POST path=/api/pending_closures5905worker1 # [ 136.421170] nix-grpc-daemon[1600]: time=2026-09-21T14:45:52.029Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5906worker1 # [ 136.422594] nix-grpc-daemon[1600]: time=2026-09-21T14:45:52.029Z level=INFO msg="Uploading x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv (416B)"5907lb # [ 136.443663] niks3-server[1112]: 2026/09/21 14:45:52 INFO Registered completed upload object_key=nar/0jczp45bkr8vp1ymznxf6da1iv8dzh42pnl36wiprbzwhpi2ddia.nar.zst5908worker1 # [ 136.454038] nix-grpc-daemon[1600]: time=2026-09-21T14:45:52.062Z level=INFO msg="Uploading 1 narinfos"5909lb # [ 136.457688] niks3-server[1112]: 2026/09/21 14:45:52 INFO Received sign narinfos request method=POST path=/api/pending_closures/44/sign5910lb # [ 136.470631] niks3-server[1112]: 2026/09/21 14:45:52 INFO Signed narinfos id=44 count=15911lb # [ 136.477822] niks3-server[1112]: 2026/09/21 14:45:52 INFO Registered completed upload object_key=x0m4asg5xx7vgj4dip2crwn6zhxqngvs.ls5912lb # [ 136.490127] niks3-server[1112]: 2026/09/21 14:45:52 INFO Received complete upload request method=POST path=/api/pending_closures/44/complete5913worker1 # [ 136.492798] nix-grpc-daemon[1600]: time=2026-09-21T14:45:52.101Z level=INFO msg="Upload complete. (370ms)"5914lb # [ 136.500079] niks3-server[1112]: 2026/09/21 14:45:52 INFO Completed upload id=445915lb # [ 136.503807] niks3-server[1112]: 2026/09/21 14:45:52 INFO Registered completed upload object_key=x0m4asg5xx7vgj4dip2crwn6zhxqngvs.narinfo5916worker1 # [ 136.500272] nix-grpc-daemon[1476]: ts=2026-09-21T14:45:52Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38582 duration_s=0 paths=1 nar_bytes_in=4165917worker1 # [ 136.514771] nix-grpc-daemon[1476]: ts=2026-09-21T14:45:52Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:385825918worker1 # [ 136.523925] nix-daemon[852]: accepted connection from pid 1476, user nix-grpc-daemon (trusted)5919worker1 # [ 136.532323] nix-daemon[852]: reaped child process 1695, status = succeeded5920client # [ 136.534317] nix[1332]: building '/nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv' on lb:50051...5921worker1 # [ 136.540188] nix-daemon[852]: accepted connection from pid 1476, user nix-grpc-daemon (trusted)5922lb # [ 136.889725] envoy[888]: conn peer=[2001:db8:1::1]:32912 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3078 tx=1518 ms=97535923lb # [ 136.894601] envoy[888]: conn peer=[2001:db8:1::1]:52740 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2987 tx=853 ms=34855924worker1: must succeed: systemctl reload nix-grpc-daemon.service5925worker1 # [ 138.388333] systemd[1]: Reloading Nix worker-protocol over gRPC...5926worker1 # [ 138.420499] systemd[1]: Reloaded Nix worker-protocol over gRPC.5927worker1: (finished: must succeed: systemctl reload nix-grpc-daemon.service, in 0.08 seconds)5928worker1: must succeed: pgrep -f 'read -t [9]0 x'5929worker1: (finished: must succeed: pgrep -f 'read -t [9]0 x', in 0.05 seconds)5930client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag during-drain >&25931worker1 # [ 138.607944] nix-grpc-daemon[1476]: ts=2026-09-21T14:45:54Z level=info event=draining5932client # don't know how to build these paths:5933client # /nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv5934worker2 # [ 138.677008] nix-grpc-daemon[1407]: ts=2026-09-21T14:45:54Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:334045935worker2 # [ 138.684818] nix-grpc-daemon[1407]: ts=2026-09-21T14:45:54Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:334045936worker2 # [ 138.709313] nix-grpc-daemon[1407]: ts=2026-09-21T14:45:54Z level=debug event=want client=4 drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv cached=05937worker2 # [ 138.716085] nix-grpc-daemon[1407]: ts=2026-09-21T14:45:54Z level=debug event=assigned drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv worker=192.168.1.4:50051 clients=15938worker2 # [ 138.761118] nix-grpc-daemon[1407]: ts=2026-09-21T14:45:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:334045939worker2 # [ 138.776453] nix-grpc-daemon[1407]: ts=2026-09-21T14:45:54Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:334045940client # copying 1 paths to 'lb:50051'...5941worker2 # [ 138.792421] nix-grpc-daemon[1407]: ts=2026-09-21T14:45:54Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:334045942client # copying path '/nix/store/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv' to 'lb:50051'...5943worker2 # [ 138.822058] nix-daemon[838]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)5944worker2 # [ 138.990277] nix-daemon[838]: accepted connection from pid 1564, user nix-grpc-daemon (trusted)5945worker2 # [ 139.009890] nix-daemon[838]: reaped child process 1567, status = succeeded5946worker2 # [ 139.107583] nix-daemon[838]: accepted connection from pid 1569, user nix-grpc-daemon (trusted)5947worker2 # [ 139.122961] nix-daemon[838]: reaped child process 1572, status = succeeded5948lb # [ 139.157806] niks3-server[1112]: 2026/09/21 14:45:54 INFO Received uploads request method=POST path=/api/pending_closures5949worker2 # [ 139.142482] nix-grpc-daemon[1485]: time=2026-09-21T14:45:54.744Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5950worker2 # [ 139.143997] nix-grpc-daemon[1485]: time=2026-09-21T14:45:54.744Z level=INFO msg="Uploading k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv (528B)"5951lb # [ 139.185547] niks3-server[1112]: 2026/09/21 14:45:54 INFO Registered completed upload object_key=nar/1h3fzlmfdwdiqr732wwrq7jnpw7n6p8f9qx1089c6g6x2i4csv75.nar.zst5952lb # [ 139.198377] niks3-server[1112]: 2026/09/21 14:45:54 INFO Received sign narinfos request method=POST path=/api/pending_closures/45/sign5953worker2 # [ 139.181373] nix-grpc-daemon[1485]: time=2026-09-21T14:45:54.784Z level=INFO msg="Uploading 1 narinfos"5954lb # [ 139.212550] niks3-server[1112]: 2026/09/21 14:45:54 INFO Signed narinfos id=45 count=15955lb # [ 139.215299] niks3-server[1112]: 2026/09/21 14:45:54 INFO Registered completed upload object_key=k3l35rf796jrnjns08398p1l8fcx178a.ls5956lb # [ 139.228676] niks3-server[1112]: 2026/09/21 14:45:54 INFO Received complete upload request method=POST path=/api/pending_closures/45/complete5957lb # [ 139.236256] niks3-server[1112]: 2026/09/21 14:45:54 INFO Completed upload id=455958worker2 # [ 139.211176] nix-grpc-daemon[1485]: time=2026-09-21T14:45:54.814Z level=INFO msg="Upload complete. (370ms)"5959lb # [ 139.238867] niks3-server[1112]: 2026/09/21 14:45:54 INFO Registered completed upload object_key=k3l35rf796jrnjns08398p1l8fcx178a.narinfo5960worker2 # [ 139.220276] nix-grpc-daemon[1407]: ts=2026-09-21T14:45:54Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33404 duration_s=0 paths=1 nar_bytes_in=5285961worker2 # [ 139.234946] nix-grpc-daemon[1407]: ts=2026-09-21T14:45:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:334045962worker2 # [ 139.244634] nix-daemon[838]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)5963worker2 # [ 139.247510] nix-daemon[838]: reaped child process 1562, status = succeeded5964client # building '/nix/store/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv' on lb:50051...5965worker2 # [ 139.258737] nix-daemon[838]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)5966client # farm-a> worker2: building k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv5967client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds5968client # farm-a> LOG-farm-a-during-drain5969worker2 # [ 140.160950] nix-daemon[838]: reaped child process 1576, status = succeeded5970worker2 # [ 140.230001] nix-daemon[838]: accepted connection from pid 1592, user nix-grpc-daemon (trusted)5971worker2 # [ 140.250504] nix-daemon[838]: reaped child process 1595, status = succeeded5972lb # [ 140.274604] niks3-server[1112]: 2026/09/21 14:45:55 INFO Received uploads request method=POST path=/api/pending_closures5973worker2 # [ 140.260494] nix-grpc-daemon[1485]: time=2026-09-21T14:45:55.862Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5974worker2 # [ 140.262042] nix-grpc-daemon[1485]: time=2026-09-21T14:45:55.862Z level=INFO msg="Uploading 42c24pndjip9nvdj8j25mkv0650griqf-farm-a (136B)"5975lb # [ 140.319753] niks3-server[1112]: 2026/09/21 14:45:55 INFO Registered completed upload object_key=nar/0r0px9vw070lid3q4jhnmka11sk2y8kj26ysyvjy9dplavd7mdz4.nar.zst5976lb # [ 140.332859] niks3-server[1112]: 2026/09/21 14:45:55 INFO Registered completed upload object_key=log/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv5977worker2 # [ 140.319463] nix-grpc-daemon[1485]: time=2026-09-21T14:45:55.921Z level=INFO msg="Uploading 1 narinfos"5978lb # [ 140.343974] niks3-server[1112]: 2026/09/21 14:45:55 INFO Received sign narinfos request method=POST path=/api/pending_closures/46/sign5979lb # [ 140.351254] niks3-server[1112]: 2026/09/21 14:45:55 INFO Signed narinfos id=46 count=15980lb # [ 140.356381] niks3-server[1112]: 2026/09/21 14:45:55 INFO Registered completed upload object_key=42c24pndjip9nvdj8j25mkv0650griqf.ls5981lb # [ 140.364373] niks3-server[1112]: 2026/09/21 14:45:55 INFO Received complete upload request method=POST path=/api/pending_closures/46/complete5982worker2 # [ 140.343621] nix-grpc-daemon[1485]: time=2026-09-21T14:45:55.946Z level=INFO msg="Upload complete. (272ms)"5983lb # [ 140.370633] niks3-server[1112]: 2026/09/21 14:45:55 INFO Completed upload id=465984lb # [ 140.373165] niks3-server[1112]: 2026/09/21 14:45:55 INFO Registered completed upload object_key=42c24pndjip9nvdj8j25mkv0650griqf.narinfo5985worker2 # [ 140.361542] nix-grpc-daemon[1407]: ts=2026-09-21T14:45:55Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33404 duration_s=1 drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv assign_id=3 outputs=15986worker2 # [ 140.375852] nix-grpc-daemon[1407]: ts=2026-09-21T14:45:55Z level=debug event=want client=4 drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv cached=05987worker2 # [ 140.379989] nix-grpc-daemon[1407]: ts=2026-09-21T14:45:55Z level=debug event=assigned drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv worker=192.168.1.4:50051 clients=15988worker2 # [ 140.383675] nix-daemon[838]: reaped child process 1574, status = succeeded5989worker2 # [ 140.413424] nix-grpc-daemon[1407]: ts=2026-09-21T14:45:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:334045990worker2 # [ 140.428489] nix-grpc-daemon[1407]: ts=2026-09-21T14:45:56Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:334045991client # copying 1 paths to 'lb:50051'...5992worker2 # [ 140.444759] nix-grpc-daemon[1485]: time=2026-09-21T14:45:56.046Z level=INFO msg="All 1 paths already cached"5993worker2 # [ 140.450397] nix-grpc-daemon[1407]: ts=2026-09-21T14:45:56Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:334045994client # copying path '/nix/store/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv' to 'lb:50051'...5995worker2 # [ 140.472978] nix-daemon[838]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)5996worker2 # [ 140.625833] nix-daemon[838]: accepted connection from pid 1601, user nix-grpc-daemon (trusted)5997worker2 # [ 140.645274] nix-daemon[838]: reaped child process 1604, status = succeeded5998worker2 # [ 140.748118] nix-daemon[838]: accepted connection from pid 1606, user nix-grpc-daemon (trusted)5999worker2 # [ 140.765231] nix-daemon[838]: reaped child process 1609, status = succeeded6000lb # [ 140.795240] niks3-server[1112]: 2026/09/21 14:45:56 INFO Received uploads request method=POST path=/api/pending_closures6001worker2 # [ 140.779904] nix-grpc-daemon[1485]: time=2026-09-21T14:45:56.382Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6002worker2 # [ 140.781614] nix-grpc-daemon[1485]: time=2026-09-21T14:45:56.382Z level=INFO msg="Uploading im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv (640B)"6003lb # [ 140.821878] niks3-server[1112]: 2026/09/21 14:45:56 INFO Registered completed upload object_key=nar/0rxmdnq61ka29n0b2as2hdbz91wi2ap17db890js8r5plhpg436g.nar.zst6004worker2 # [ 140.810520] nix-grpc-daemon[1485]: time=2026-09-21T14:45:56.413Z level=INFO msg="Uploading 1 narinfos"6005lb # [ 140.836570] niks3-server[1112]: 2026/09/21 14:45:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/47/sign6006lb # [ 140.849365] niks3-server[1112]: 2026/09/21 14:45:56 INFO Signed narinfos id=47 count=16007lb # [ 140.854604] niks3-server[1112]: 2026/09/21 14:45:56 INFO Registered completed upload object_key=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l.ls6008worker2 # [ 140.840906] nix-grpc-daemon[1485]: time=2026-09-21T14:45:56.443Z level=INFO msg="Upload complete. (351ms)"6009lb # [ 140.866722] niks3-server[1112]: 2026/09/21 14:45:56 INFO Received complete upload request method=POST path=/api/pending_closures/47/complete6010lb # [ 140.871947] niks3-server[1112]: 2026/09/21 14:45:56 INFO Completed upload id=476011worker2 # [ 140.848139] nix-grpc-daemon[1407]: ts=2026-09-21T14:45:56Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33404 duration_s=0 paths=1 nar_bytes_in=6406012lb # [ 140.874931] niks3-server[1112]: 2026/09/21 14:45:56 INFO Registered completed upload object_key=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l.narinfo6013worker2 # [ 140.860312] nix-grpc-daemon[1407]: ts=2026-09-21T14:45:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:334046014worker2 # [ 140.869503] nix-daemon[838]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6015worker2 # [ 140.874149] nix-daemon[838]: reaped child process 1599, status = succeeded6016client # building '/nix/store/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv' on lb:50051...6017client # farm-b> worker2: building im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv6018worker2 # [ 140.887392] nix-daemon[838]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6019client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds6020client # farm-b> LOG-farm-b-during-drain6021worker2 # [ 142.151984] nix-daemon[838]: reaped child process 1613, status = succeeded6022worker2 # [ 142.267361] nix-daemon[838]: accepted connection from pid 1629, user nix-grpc-daemon (trusted)6023worker2 # [ 142.285618] nix-daemon[838]: reaped child process 1632, status = succeeded6024lb # [ 142.318818] niks3-server[1112]: 2026/09/21 14:45:57 INFO Received uploads request method=POST path=/api/pending_closures6025worker2 # [ 142.302912] nix-grpc-daemon[1485]: time=2026-09-21T14:45:57.904Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6026worker2 # [ 142.304778] nix-grpc-daemon[1485]: time=2026-09-21T14:45:57.904Z level=INFO msg="Uploading daks7y2c761g04qpzsjx25629wigr8js-farm-b (184B)"6027lb # [ 142.352518] niks3-server[1112]: 2026/09/21 14:45:57 INFO Registered completed upload object_key=log/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv6028lb # [ 142.374946] niks3-server[1112]: 2026/09/21 14:45:57 INFO Registered completed upload object_key=nar/0lnpxl34nrzp147jy5bafczkvxcl00hcf69qjhsljmv2ip9jisfh.nar.zst6029worker2 # [ 142.360198] nix-grpc-daemon[1485]: time=2026-09-21T14:45:57.962Z level=INFO msg="Uploading 1 narinfos"6030lb # [ 142.385752] niks3-server[1112]: 2026/09/21 14:45:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/48/sign6031lb # [ 142.392534] niks3-server[1112]: 2026/09/21 14:45:57 INFO Signed narinfos id=48 count=16032lb # [ 142.398565] niks3-server[1112]: 2026/09/21 14:45:57 INFO Received complete upload request method=POST path=/api/pending_closures/48/complete6033lb # [ 142.409657] niks3-server[1112]: 2026/09/21 14:45:58 INFO Registered completed upload object_key=daks7y2c761g04qpzsjx25629wigr8js.ls6034lb # [ 142.418760] niks3-server[1112]: 2026/09/21 14:45:58 INFO Completed upload id=486035worker2 # [ 142.384396] nix-grpc-daemon[1485]: time=2026-09-21T14:45:57.987Z level=INFO msg="Upload complete. (286ms)"6036worker2 # [ 142.405386] nix-grpc-daemon[1407]: ts=2026-09-21T14:45:57Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33404 duration_s=1 drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv assign_id=4 outputs=16037worker2 # [ 142.420311] nix-grpc-daemon[1407]: ts=2026-09-21T14:45:58Z level=debug event=want client=4 drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv cached=06038worker2 # [ 142.426867] nix-grpc-daemon[1407]: ts=2026-09-21T14:45:58Z level=debug event=assigned drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv worker=192.168.1.4:50051 clients=16039worker2 # [ 142.432193] nix-grpc-daemon[1485]: time=2026-09-21T14:45:58.019Z level=WARN msg="Failed to register uploaded object" key=daks7y2c761g04qpzsjx25629wigr8js.narinfo error="server returned 404: object is not pending upload\n"6040worker2 # [ 142.438826] nix-daemon[838]: reaped child process 1611, status = succeeded6041worker2 # [ 142.448072] nix-grpc-daemon[1407]: ts=2026-09-21T14:45:58Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:334046042worker2 # [ 142.462452] nix-grpc-daemon[1407]: ts=2026-09-21T14:45:58Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:334046043worker2 # [ 142.481328] nix-grpc-daemon[1485]: time=2026-09-21T14:45:58.084Z level=INFO msg="All 2 paths already cached"6044client # copying 1 paths to 'lb:50051'...6045worker2 # [ 142.491895] nix-grpc-daemon[1407]: ts=2026-09-21T14:45:58Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:334046046client # copying path '/nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv' to 'lb:50051'...6047worker2 # [ 142.514903] nix-daemon[838]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6048worker2 # [ 142.667379] nix-daemon[838]: accepted connection from pid 1638, user nix-grpc-daemon (trusted)6049worker2 # [ 142.689270] nix-daemon[838]: reaped child process 1641, status = succeeded6050worker2 # [ 142.790029] nix-daemon[838]: accepted connection from pid 1643, user nix-grpc-daemon (trusted)6051worker2 # [ 142.810260] nix-daemon[838]: reaped child process 1646, status = succeeded6052lb # [ 142.835933] niks3-server[1112]: 2026/09/21 14:45:58 INFO Received uploads request method=POST path=/api/pending_closures6053worker2 # [ 142.823870] nix-grpc-daemon[1485]: time=2026-09-21T14:45:58.426Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6054worker2 # [ 142.825589] nix-grpc-daemon[1485]: time=2026-09-21T14:45:58.426Z level=INFO msg="Uploading lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv (768B)"6055worker2 # [ 142.859734] nix-grpc-daemon[1485]: time=2026-09-21T14:45:58.462Z level=INFO msg="Uploading 1 narinfos"6056lb # [ 142.884081] niks3-server[1112]: 2026/09/21 14:45:58 INFO Registered completed upload object_key=nar/066jhl91shf8ar6h3cqqb855j5cswl7170wrg5lzf3fhpka0xywn.nar.zst6057lb # [ 142.899189] niks3-server[1112]: 2026/09/21 14:45:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/49/sign6058lb # [ 142.911394] niks3-server[1112]: 2026/09/21 14:45:58 INFO Signed narinfos id=49 count=16059worker2 # [ 142.891920] nix-grpc-daemon[1485]: time=2026-09-21T14:45:58.494Z level=INFO msg="Upload complete. (360ms)"6060lb # [ 142.918305] niks3-server[1112]: 2026/09/21 14:45:58 INFO Registered completed upload object_key=lbk8wy49c881163x07gb8z669j2fg1ff.ls6061lb # [ 142.923866] niks3-server[1112]: 2026/09/21 14:45:58 INFO Received complete upload request method=POST path=/api/pending_closures/49/complete6062lb # [ 142.928872] niks3-server[1112]: 2026/09/21 14:45:58 INFO Completed upload id=496063lb # [ 142.931763] niks3-server[1112]: 2026/09/21 14:45:58 INFO Registered completed upload object_key=lbk8wy49c881163x07gb8z669j2fg1ff.narinfo6064worker2 # [ 142.899330] nix-grpc-daemon[1407]: ts=2026-09-21T14:45:58Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33404 duration_s=0 paths=1 nar_bytes_in=7686065worker2 # [ 142.913270] nix-grpc-daemon[1407]: ts=2026-09-21T14:45:58Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:334046066worker2 # [ 142.923333] nix-daemon[838]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6067worker2 # [ 142.926331] nix-daemon[838]: reaped child process 1636, status = succeeded6068client # building '/nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv' on lb:50051...6069worker2 # [ 142.939919] nix-daemon[838]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6070client # farm-top> worker2: building lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv6071client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds6072client # farm-top> LOG-farm-top-during-drain6073worker2 # [ 144.017945] nix-daemon[838]: reaped child process 1650, status = succeeded6074worker2 # [ 144.120767] nix-daemon[838]: accepted connection from pid 1666, user nix-grpc-daemon (trusted)6075worker2 # [ 144.139818] nix-daemon[838]: reaped child process 1669, status = succeeded6076lb # [ 144.171069] niks3-server[1112]: 2026/09/21 14:45:59 INFO Received uploads request method=POST path=/api/pending_closures6077worker2 # [ 144.158320] nix-grpc-daemon[1485]: time=2026-09-21T14:45:59.759Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6078worker2 # [ 144.159999] nix-grpc-daemon[1485]: time=2026-09-21T14:45:59.759Z level=INFO msg="Uploading xdwmx2f4n5n8bggglniz2n35any0xz4w-farm-top (240B)"6079lb # [ 144.210469] niks3-server[1112]: 2026/09/21 14:45:59 INFO Registered completed upload object_key=nar/1bpy9ga4n2446pary6msidbs8mnil1a34jn8p8yl2scvdg691mz2.nar.zst6080worker2 # [ 144.201755] nix-grpc-daemon[1485]: time=2026-09-21T14:45:59.804Z level=INFO msg="Uploading 1 narinfos"6081lb # [ 144.226012] niks3-server[1112]: 2026/09/21 14:45:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/50/sign6082lb # [ 144.237389] niks3-server[1112]: 2026/09/21 14:45:59 INFO Signed narinfos id=50 count=16083lb # [ 144.243362] niks3-server[1112]: 2026/09/21 14:45:59 INFO Registered completed upload object_key=log/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv6084worker2 # [ 144.229096] nix-grpc-daemon[1485]: time=2026-09-21T14:45:59.832Z level=INFO msg="Upload complete. (278ms)"6085lb # [ 144.254679] niks3-server[1112]: 2026/09/21 14:45:59 INFO Registered completed upload object_key=xdwmx2f4n5n8bggglniz2n35any0xz4w.ls6086lb # [ 144.262441] niks3-server[1112]: 2026/09/21 14:45:59 INFO Received complete upload request method=POST path=/api/pending_closures/50/complete6087lb # [ 144.268527] niks3-server[1112]: 2026/09/21 14:45:59 INFO Completed upload id=506088lb # [ 144.272204] niks3-server[1112]: 2026/09/21 14:45:59 INFO Registered completed upload object_key=xdwmx2f4n5n8bggglniz2n35any0xz4w.narinfo6089worker2 # [ 144.245168] nix-grpc-daemon[1407]: ts=2026-09-21T14:45:59Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33404 duration_s=1 drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv assign_id=5 outputs=16090worker2 # [ 144.261632] nix-daemon[838]: reaped child process 1648, status = succeeded6091client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag during-drain >&2, in 5.83 seconds)6092client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag during-drain outPath6093client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag during-drain outPath, in 0.18 seconds)6094worker1: must succeed: pkill -f 'read -t [9]0 x'6095worker1: (finished: must succeed: pkill -f 'read -t [9]0 x', in 0.05 seconds)6096client: waiting for success: ! systemctl is-active sw6097worker1 # [ 144.577458] nix-daemon[852]: reaped child process 1709, status = succeeded6098worker1 # [ 144.668899] nix-daemon[852]: accepted connection from pid 1750, user nix-grpc-daemon (trusted)6099worker1 # [ 144.689295] nix-daemon[852]: reaped child process 1753, status = succeeded6100lb # [ 144.694433] niks3-server[1112]: 2026/09/21 14:46:00 INFO Received uploads request method=POST path=/api/pending_closures6101worker1 # [ 144.699877] nix-grpc-daemon[1600]: time=2026-09-21T14:46:00.307Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6102worker1 # [ 144.701562] nix-grpc-daemon[1600]: time=2026-09-21T14:46:00.307Z level=INFO msg="Uploading qr8vyl2zl3nxnin3vw41c6z35vj7087f-slow-sw (120B)"6103lb # [ 144.728115] niks3-server[1112]: 2026/09/21 14:46:00 INFO Registered completed upload object_key=nar/0vi6cx31wc05wns04pr5vr2bak9iib1idndvy6gik6c2rz9q8c19.nar.zst6104lb # [ 144.746606] niks3-server[1112]: 2026/09/21 14:46:00 INFO Registered completed upload object_key=log/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv6105worker1 # [ 144.751873] nix-grpc-daemon[1600]: time=2026-09-21T14:46:00.360Z level=INFO msg="Uploading 1 narinfos"6106lb # [ 144.756909] niks3-server[1112]: 2026/09/21 14:46:00 INFO Received sign narinfos request method=POST path=/api/pending_closures/51/sign6107lb # [ 144.765728] niks3-server[1112]: 2026/09/21 14:46:00 INFO Signed narinfos id=51 count=16108lb # [ 144.767746] niks3-server[1112]: 2026/09/21 14:46:00 INFO Registered completed upload object_key=qr8vyl2zl3nxnin3vw41c6z35vj7087f.ls6109lb # [ 144.774986] niks3-server[1112]: 2026/09/21 14:46:00 INFO Received complete upload request method=POST path=/api/pending_closures/51/complete6110worker1 # [ 144.776550] nix-grpc-daemon[1600]: time=2026-09-21T14:46:00.385Z level=INFO msg="Upload complete. (220ms)"6111lb # [ 144.782488] niks3-server[1112]: 2026/09/21 14:46:00 INFO Completed upload id=516112lb # [ 144.786229] niks3-server[1112]: 2026/09/21 14:46:00 INFO Registered completed upload object_key=qr8vyl2zl3nxnin3vw41c6z35vj7087f.narinfo6113worker1 # [ 144.789006] nix-grpc-daemon[1476]: ts=2026-09-21T14:46:00Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38582 duration_s=8 drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv assign_id=2 outputs=16114worker1 # [ 144.802020] nix-daemon[852]: reaped child process 1707, status = succeeded6115client # [ 144.818383] systemd[1]: sw.service: Deactivated successfully.6116client # [ 144.823485] systemd[1]: sw.service: Consumed 193ms CPU time over 9.026s wall clock time, 41.4M memory peak, 7.1K incoming IP traffic, 8.4K outgoing IP traffic.6117worker1 # [ 145.610744] nix-grpc-daemon[1476]: ts=2026-09-21T14:46:01Z level=info event=signal_exit6118client: (finished: waiting for success: ! systemctl is-active sw, in 1.09 seconds)6119client: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }6120client: (finished: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }, in 0.05 seconds)6121worker1: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service6122lb # [ 146.890531] envoy[888]: conn peer=[2001:db8:1::1]:52756 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9621 tx=4090 ms=55856123lb # [ 146.895528] envoy[888]: conn peer=[2001:db8:1::1]:52750 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2978 tx=1405 ms=88086124worker1 # [ 147.611470] nix-grpc-daemon[1476]: ts=2026-09-21T14:46:03Z level=info event=scheduler_restarting6125worker2 # [ 147.590246] nix-grpc-daemon[1407]: ts=2026-09-21T14:46:03Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:454106126worker2 # [ 147.595070] nix-grpc-daemon[1407]: ts=2026-09-21T14:46:03Z level=info event=worker_gone addr=192.168.1.3:500516127worker1 # [ 147.636981] nix-grpc-daemon[1476]: ts=2026-09-21T14:46:03Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"6128worker1 # [ 147.662337] nix-daemon[852]: reaped child process 1588, status = succeeded6129worker1 # [ 148.015026] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.6130worker1 # [ 148.023208] systemd[1]: nix-grpc-daemon.service: Consumed 1.147s CPU time over 42.577s wall clock time, 36.1M memory peak, 26.5K incoming IP traffic, 37.8K outgoing IP traffic.6131worker1 # [ 148.043265] systemd[1]: Starting Nix worker-protocol over gRPC...6132worker1 # [ 148.191016] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:03Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6133worker1 # [ 148.193355] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:03Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6134worker1 # [ 148.197116] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:03Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=16135worker1 # [ 148.203625] nix-grpc-daemon[1767]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6136worker1 # [ 148.220216] nix-grpc-daemon[1767]: E0000 00:00:1790001963.810066 1767 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6137worker1: (finished: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service, in 2.56 seconds)6138lb: waiting until exactly no endpoints of aarch64-linux are unhealthy6139lb: must succeed: curl -sf localhost:9901/clusters6140worker1 # [ 148.249572] nix-grpc-daemon[1767]: E0000 00:00:1790001963.812067 1767 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6141worker1 # [ 148.251839] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:03Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6142worker1 # [ 148.254602] nix-grpc-daemon[1767]: E0000 00:00:1790001963.821492 1807 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6143worker1 # [ 148.257679] nix-grpc-daemon[1767]: E0000 00:00:1790001963.840491 1807 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6144worker1 # [ 148.260154] nix-grpc-daemon[1767]: E0000 00:00:1790001963.840654 1807 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6145worker1 # [ 148.262451] nix-grpc-daemon[1767]: E0000 00:00:1790001963.857988 1807 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6146worker1 # [ 148.264808] systemd[1]: Started Nix worker-protocol over gRPC.6147worker1 # [ 148.271392] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:03Z level=info event=scheduler_connected addr=lb:500516148worker2 # [ 148.254038] nix-grpc-daemon[1407]: ts=2026-09-21T14:46:03Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:454106149worker2 # [ 148.259837] nix-grpc-daemon[1407]: ts=2026-09-21T14:46:03Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=06150lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)6151lb: must succeed: curl -sf localhost:9901/clusters6152lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)6153lb: (finished: waiting until exactly no endpoints of aarch64-linux are unhealthy, in 1.14 seconds)6154lb: must succeed: curl -sf localhost:9901/clusters6155lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6156worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6157worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6158(finished: subtest: a deploy mid-build drains: the build finishes, the new generation takes over, in 13.77 seconds)6159subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does6160client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&16161worker1 # [ 149.720778] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:05Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:346566162worker1 # [ 149.735694] nix-daemon[852]: accepted connection from pid 1767, user nix-grpc-daemon (trusted)6163worker1 # [ 149.755450] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:05Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:346566164worker1 # [ 149.763376] nix-grpc-daemon[1767]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 33 ms (attempt 1/5)6165worker1 # [ 149.801993] nix-grpc-daemon[1767]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 18 ms (attempt 2/5)6166worker1 # [ 149.821685] nix-grpc-daemon[1767]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 304 ms (attempt 3/5)6167worker1 # [ 150.128156] nix-grpc-daemon[1767]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 410 ms (attempt 4/5)6168worker1 # [ 150.540643] nix-grpc-daemon[1767]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6169worker1 # [ 150.549329] nix-grpc-daemon[1767]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6170worker2 # [ 150.571804] nix-grpc-daemon[1407]: ts=2026-09-21T14:46:06Z level=debug event=want client=5 drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv cached=06171worker2 # [ 150.577876] nix-grpc-daemon[1407]: ts=2026-09-21T14:46:06Z level=debug event=assigned drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv worker=192.168.1.3:50051 clients=16172worker2 # [ 150.583811] nix-grpc-daemon[1407]: ts=2026-09-21T14:46:06Z level=debug event=want client=5 drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv cached=06173worker2 # [ 150.589141] nix-grpc-daemon[1407]: ts=2026-09-21T14:46:06Z level=debug event=want client=5 drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv cached=06174worker1 # [ 150.637768] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:346566175worker1 # [ 150.652803] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:06Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:346566176worker1 # [ 150.667302] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:06Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:346566177worker1 # [ 150.681622] nix-daemon[852]: accepted connection from pid 1767, user nix-grpc-daemon (trusted)6178worker1 # [ 150.711218] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:06Z level=info event=niks3_push_started pid=18276179worker1 # [ 150.858280] nix-grpc-daemon[1827]: time=2026-09-21T14:46:06.466Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.key ca=""6180worker1 # [ 150.963237] nix-daemon[852]: accepted connection from pid 1839, user nix-grpc-daemon (trusted)6181worker1 # [ 150.978889] nix-daemon[852]: reaped child process 1842, status = succeeded6182worker1 # [ 151.083215] nix-daemon[852]: accepted connection from pid 1844, user nix-grpc-daemon (trusted)6183worker1 # [ 151.098345] nix-daemon[852]: reaped child process 1847, status = succeeded6184lb # [ 151.110231] niks3-server[1112]: 2026/09/21 14:46:06 INFO Received uploads request method=POST path=/api/pending_closures6185worker1 # [ 151.118104] nix-grpc-daemon[1827]: time=2026-09-21T14:46:06.725Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6186worker1 # [ 151.119569] nix-grpc-daemon[1827]: time=2026-09-21T14:46:06.725Z level=INFO msg="Uploading 3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv (512B)"6187lb # [ 151.149011] niks3-server[1112]: 2026/09/21 14:46:06 INFO Registered completed upload object_key=nar/0zxfsmqijfc2dkglls3yp1x3w0w1na7vx39fxc124zddjzpaygd8.nar.zst6188worker1 # [ 151.164134] nix-grpc-daemon[1827]: time=2026-09-21T14:46:06.772Z level=INFO msg="Uploading 1 narinfos"6189lb # [ 151.167863] niks3-server[1112]: 2026/09/21 14:46:06 INFO Received sign narinfos request method=POST path=/api/pending_closures/52/sign6190lb # [ 151.179478] niks3-server[1112]: 2026/09/21 14:46:06 INFO Signed narinfos id=52 count=16191lb # [ 151.196200] niks3-server[1112]: 2026/09/21 14:46:06 INFO Registered completed upload object_key=3223c4sl4ncpldbvw558vqwddijwzfpd.ls6192lb # [ 151.206221] niks3-server[1112]: 2026/09/21 14:46:06 INFO Received complete upload request method=POST path=/api/pending_closures/52/complete6193lb # [ 151.212264] niks3-server[1112]: 2026/09/21 14:46:06 INFO Completed upload id=526194worker1 # [ 151.207735] nix-grpc-daemon[1827]: time=2026-09-21T14:46:06.816Z level=INFO msg="Upload complete. (345ms)"6195lb # [ 151.215466] niks3-server[1112]: 2026/09/21 14:46:06 INFO Registered completed upload object_key=3223c4sl4ncpldbvw558vqwddijwzfpd.narinfo6196worker1 # [ 151.219838] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:06Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34656 duration_s=0 paths=1 nar_bytes_in=5126197worker1 # [ 151.236612] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:346566198worker1 # [ 151.246112] nix-daemon[852]: accepted connection from pid 1767, user nix-grpc-daemon (trusted)6199worker1 # [ 151.250181] nix-daemon[852]: reaped child process 1825, status = succeeded6200worker1 # [ 151.262016] nix-daemon[852]: accepted connection from pid 1767, user nix-grpc-daemon (trusted)6201worker1 # [ 152.155733] nix-daemon[852]: reaped child process 1851, status = succeeded6202worker1 # [ 152.240775] nix-daemon[852]: accepted connection from pid 1868, user nix-grpc-daemon (trusted)6203worker1 # [ 152.258781] nix-daemon[852]: reaped child process 1871, status = succeeded6204lb # [ 152.271365] niks3-server[1112]: 2026/09/21 14:46:07 INFO Received uploads request method=POST path=/api/pending_closures6205worker1 # [ 152.280083] nix-grpc-daemon[1827]: time=2026-09-21T14:46:07.885Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6206worker1 # [ 152.281858] nix-grpc-daemon[1827]: time=2026-09-21T14:46:07.885Z level=INFO msg="Uploading dla3w0zf02z5jhccwa27i85bbgzrrsvb-farm-a (128B)"6207lb # [ 152.304089] niks3-server[1112]: 2026/09/21 14:46:07 INFO Registered completed upload object_key=log/3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv6208worker1 # [ 152.329323] nix-grpc-daemon[1827]: time=2026-09-21T14:46:07.937Z level=INFO msg="Uploading 1 narinfos"6209lb # [ 152.332894] niks3-server[1112]: 2026/09/21 14:46:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/53/sign6210lb # [ 152.343744] niks3-server[1112]: 2026/09/21 14:46:07 INFO Signed narinfos id=53 count=16211lb # [ 152.349366] niks3-server[1112]: 2026/09/21 14:46:07 INFO Registered completed upload object_key=nar/05i7ka1qa7vmhiys96ln3pp3nj3rfdlslnbydrxdqj40cmvmqcwf.nar.zst6212lb # [ 152.362049] niks3-server[1112]: 2026/09/21 14:46:07 INFO Received complete upload request method=POST path=/api/pending_closures/53/complete6213worker1 # [ 152.357760] nix-grpc-daemon[1827]: time=2026-09-21T14:46:07.966Z level=INFO msg="Upload complete. (282ms)"6214lb # [ 152.372158] niks3-server[1112]: 2026/09/21 14:46:07 INFO Completed upload id=536215worker2 # [ 152.348447] nix-grpc-daemon[1407]: ts=2026-09-21T14:46:07Z level=debug event=assigned drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv worker=192.168.1.3:50051 clients=16216worker1 # [ 152.370383] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:07Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34656 duration_s=1 drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv assign_id=6 outputs=16217lb # [ 152.378390] niks3-server[1112]: 2026/09/21 14:46:07 INFO Registered completed upload object_key=dla3w0zf02z5jhccwa27i85bbgzrrsvb.narinfo6218worker2 # [ 152.359771] nix-grpc-daemon[1407]: ts=2026-09-21T14:46:07Z level=debug event=want client=5 drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv cached=06219lb # [ 152.386563] niks3-server[1112]: 2026/09/21 14:46:07 INFO Registered completed upload object_key=dla3w0zf02z5jhccwa27i85bbgzrrsvb.ls6220worker1 # [ 152.396241] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:346566221worker1 # [ 152.400516] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:07Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:346566222worker1 # [ 152.406624] nix-daemon[852]: reaped child process 1849, status = succeeded6223worker1 # [ 152.418217] nix-daemon[852]: accepted connection from pid 1767, user nix-grpc-daemon (trusted)6224worker1 # [ 152.424352] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:08Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:346566225worker1 # [ 152.554352] nix-daemon[852]: accepted connection from pid 1875, user nix-grpc-daemon (trusted)6226worker1 # [ 152.573798] nix-daemon[852]: reaped child process 1878, status = succeeded6227worker1 # [ 152.672690] nix-daemon[852]: accepted connection from pid 1880, user nix-grpc-daemon (trusted)6228worker1 # [ 152.688928] nix-daemon[852]: reaped child process 1883, status = succeeded6229lb # [ 152.698447] niks3-server[1112]: 2026/09/21 14:46:08 INFO Received uploads request method=POST path=/api/pending_closures6230worker1 # [ 152.703365] nix-grpc-daemon[1827]: time=2026-09-21T14:46:08.311Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6231worker1 # [ 152.704991] nix-grpc-daemon[1827]: time=2026-09-21T14:46:08.311Z level=INFO msg="Uploading pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv (512B)"6232lb # [ 152.725873] niks3-server[1112]: 2026/09/21 14:46:08 INFO Registered completed upload object_key=nar/0g5bxr683biqh358mkd9xrclwllan4cgnv4pzijas586xj68wwx0.nar.zst6233worker1 # [ 152.735616] nix-grpc-daemon[1827]: time=2026-09-21T14:46:08.344Z level=INFO msg="Uploading 1 narinfos"6234lb # [ 152.740863] niks3-server[1112]: 2026/09/21 14:46:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/54/sign6235lb # [ 152.752517] niks3-server[1112]: 2026/09/21 14:46:08 INFO Signed narinfos id=54 count=16236lb # [ 152.761216] niks3-server[1112]: 2026/09/21 14:46:08 INFO Registered completed upload object_key=pplpzcnjfn295nm16cg0iihpfc9iffry.ls6237worker1 # [ 152.768354] nix-grpc-daemon[1827]: time=2026-09-21T14:46:08.376Z level=INFO msg="Upload complete. (337ms)"6238lb # [ 152.773545] niks3-server[1112]: 2026/09/21 14:46:08 INFO Received complete upload request method=POST path=/api/pending_closures/54/complete6239lb # [ 152.782565] niks3-server[1112]: 2026/09/21 14:46:08 INFO Completed upload id=546240lb # [ 152.785381] niks3-server[1112]: 2026/09/21 14:46:08 INFO Registered completed upload object_key=pplpzcnjfn295nm16cg0iihpfc9iffry.narinfo6241worker1 # [ 152.776125] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:08Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34656 duration_s=0 paths=1 nar_bytes_in=5126242worker1 # [ 152.789325] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:08Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:346566243worker1 # [ 152.795249] nix-daemon[852]: accepted connection from pid 1767, user nix-grpc-daemon (trusted)6244worker1 # [ 152.799963] nix-daemon[852]: reaped child process 1873, status = succeeded6245worker1 # [ 152.810202] nix-daemon[852]: accepted connection from pid 1767, user nix-grpc-daemon (trusted)6246worker1 # [ 153.708636] nix-daemon[852]: reaped child process 1887, status = succeeded6247worker1 # [ 153.774271] nix-daemon[852]: accepted connection from pid 1903, user nix-grpc-daemon (trusted)6248worker1 # [ 153.792817] nix-daemon[852]: reaped child process 1906, status = succeeded6249lb # [ 153.803461] niks3-server[1112]: 2026/09/21 14:46:09 INFO Received uploads request method=POST path=/api/pending_closures6250worker1 # [ 153.808407] nix-grpc-daemon[1827]: time=2026-09-21T14:46:09.416Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6251worker1 # [ 153.810765] nix-grpc-daemon[1827]: time=2026-09-21T14:46:09.418Z level=INFO msg="Uploading fhc1zkcygpzppf0bj5fr2k85lf7p87v9-farm-a (128B)"6252lb # [ 153.840109] niks3-server[1112]: 2026/09/21 14:46:09 INFO Registered completed upload object_key=nar/09na6djn66dpys8h08gqw39p0vw7vxqsdzbiw55rknyna8s239bm.nar.zst6253worker1 # [ 153.853431] nix-grpc-daemon[1827]: time=2026-09-21T14:46:09.461Z level=INFO msg="Uploading 1 narinfos"6254lb # [ 153.857493] niks3-server[1112]: 2026/09/21 14:46:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/55/sign6255lb # [ 153.868212] niks3-server[1112]: 2026/09/21 14:46:09 INFO Signed narinfos id=55 count=16256lb # [ 153.874852] niks3-server[1112]: 2026/09/21 14:46:09 INFO Registered completed upload object_key=log/pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv6257lb # [ 153.886988] niks3-server[1112]: 2026/09/21 14:46:09 INFO Registered completed upload object_key=fhc1zkcygpzppf0bj5fr2k85lf7p87v9.ls6258worker1 # [ 153.888914] nix-grpc-daemon[1827]: time=2026-09-21T14:46:09.497Z level=INFO msg="Upload complete. (267ms)"6259lb # [ 153.895770] niks3-server[1112]: 2026/09/21 14:46:09 INFO Received complete upload request method=POST path=/api/pending_closures/55/complete6260lb # [ 153.900837] niks3-server[1112]: 2026/09/21 14:46:09 INFO Completed upload id=556261lb # [ 153.904422] niks3-server[1112]: 2026/09/21 14:46:09 INFO Registered completed upload object_key=fhc1zkcygpzppf0bj5fr2k85lf7p87v9.narinfo6262worker2 # [ 153.878437] nix-grpc-daemon[1407]: ts=2026-09-21T14:46:09Z level=debug event=assigned drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv worker=192.168.1.3:50051 clients=16263worker1 # [ 153.901135] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:09Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34656 duration_s=1 drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv assign_id=7 outputs=16264worker2 # [ 153.890921] nix-grpc-daemon[1407]: ts=2026-09-21T14:46:09Z level=debug event=want client=5 drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv cached=06265worker1 # [ 153.920426] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:346566266worker1 # [ 153.931520] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:09Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:346566267worker1 # [ 153.937916] nix-daemon[852]: reaped child process 1885, status = succeeded6268worker1 # [ 153.945486] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:09Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:346566269worker1 # [ 153.959964] nix-daemon[852]: accepted connection from pid 1767, user nix-grpc-daemon (trusted)6270worker1 # [ 154.099431] nix-daemon[852]: accepted connection from pid 1910, user nix-grpc-daemon (trusted)6271worker1 # [ 154.116811] nix-daemon[852]: reaped child process 1913, status = succeeded6272worker1 # [ 154.219249] nix-daemon[852]: accepted connection from pid 1915, user nix-grpc-daemon (trusted)6273worker1 # [ 154.234026] nix-daemon[852]: reaped child process 1918, status = succeeded6274lb # [ 154.245165] niks3-server[1112]: 2026/09/21 14:46:09 INFO Received uploads request method=POST path=/api/pending_closures6275worker1 # [ 154.250729] nix-grpc-daemon[1827]: time=2026-09-21T14:46:09.859Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6276worker1 # [ 154.252285] nix-grpc-daemon[1827]: time=2026-09-21T14:46:09.859Z level=INFO msg="Uploading sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv (512B)"6277lb # [ 154.271540] niks3-server[1112]: 2026/09/21 14:46:09 INFO Registered completed upload object_key=nar/1fxz200fmpbckafrn03b2snx9knqpbwf8rwqlbvg68356qj7aj5w.nar.zst6278worker1 # [ 154.288087] nix-grpc-daemon[1827]: time=2026-09-21T14:46:09.896Z level=INFO msg="Uploading 1 narinfos"6279lb # [ 154.285303] niks3-server[1112]: 2026/09/21 14:46:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/56/sign6280lb # [ 154.297466] niks3-server[1112]: 2026/09/21 14:46:09 INFO Signed narinfos id=56 count=16281lb # [ 154.304260] niks3-server[1112]: 2026/09/21 14:46:09 INFO Registered completed upload object_key=sy43wp9v52pni1wms6q0p3qybpc2pwad.ls6282lb # [ 154.315701] niks3-server[1112]: 2026/09/21 14:46:09 INFO Received complete upload request method=POST path=/api/pending_closures/56/complete6283lb # [ 154.329737] niks3-server[1112]: 2026/09/21 14:46:09 INFO Completed upload id=566284worker1 # [ 154.325665] nix-grpc-daemon[1827]: time=2026-09-21T14:46:09.934Z level=INFO msg="Upload complete. (356ms)"6285lb # [ 154.331825] niks3-server[1112]: 2026/09/21 14:46:09 INFO Registered completed upload object_key=sy43wp9v52pni1wms6q0p3qybpc2pwad.narinfo6286worker1 # [ 154.332268] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:09Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34656 duration_s=0 paths=1 nar_bytes_in=5126287worker1 # [ 154.345824] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:346566288worker1 # [ 154.352357] nix-daemon[852]: accepted connection from pid 1767, user nix-grpc-daemon (trusted)6289worker1 # [ 154.357586] nix-daemon[852]: reaped child process 1908, status = succeeded6290worker1 # [ 154.371393] nix-daemon[852]: accepted connection from pid 1767, user nix-grpc-daemon (trusted)6291worker1 # [ 155.608001] nix-daemon[852]: reaped child process 1922, status = succeeded6292worker1 # [ 155.726023] nix-daemon[852]: accepted connection from pid 1938, user nix-grpc-daemon (trusted)6293worker1 # [ 155.743901] nix-daemon[852]: reaped child process 1941, status = succeeded6294lb # [ 155.755548] niks3-server[1112]: 2026/09/21 14:46:11 INFO Received uploads request method=POST path=/api/pending_closures6295worker1 # [ 155.764052] nix-grpc-daemon[1827]: time=2026-09-21T14:46:11.370Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6296worker1 # [ 155.765687] nix-grpc-daemon[1827]: time=2026-09-21T14:46:11.370Z level=INFO msg="Uploading g5laypvfk2469p8binw3mksk2hv7zhnh-farm-a (128B)"6297lb # [ 155.789816] niks3-server[1112]: 2026/09/21 14:46:11 INFO Registered completed upload object_key=log/sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv6298lb # [ 155.802036] niks3-server[1112]: 2026/09/21 14:46:11 INFO Registered completed upload object_key=nar/1j2671k9mycq8lxyfx5k6a66d5narfqw2wqwlpwxgwzi93vnvbdn.nar.zst6299lb # [ 155.816120] niks3-server[1112]: 2026/09/21 14:46:11 INFO Received sign narinfos request method=POST path=/api/pending_closures/57/sign6300worker1 # [ 155.818067] nix-grpc-daemon[1827]: time=2026-09-21T14:46:11.426Z level=INFO msg="Uploading 1 narinfos"6301lb # [ 155.824289] niks3-server[1112]: 2026/09/21 14:46:11 INFO Signed narinfos id=57 count=16302lb # [ 155.832416] niks3-server[1112]: 2026/09/21 14:46:11 INFO Registered completed upload object_key=g5laypvfk2469p8binw3mksk2hv7zhnh.ls6303lb # [ 155.846125] niks3-server[1112]: 2026/09/21 14:46:11 INFO Received complete upload request method=POST path=/api/pending_closures/57/complete6304lb # [ 155.853491] niks3-server[1112]: 2026/09/21 14:46:11 INFO Completed upload id=576305worker1 # [ 155.847946] nix-grpc-daemon[1827]: time=2026-09-21T14:46:11.456Z level=INFO msg="Upload complete. (266ms)"6306lb # [ 155.855794] niks3-server[1112]: 2026/09/21 14:46:11 INFO Registered completed upload object_key=g5laypvfk2469p8binw3mksk2hv7zhnh.narinfo6307worker2 # [ 155.838391] nix-grpc-daemon[1407]: ts=2026-09-21T14:46:11Z level=debug event=assigned drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv worker=192.168.1.3:50051 clients=16308worker1 # [ 155.859891] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:11Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34656 duration_s=1 drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv assign_id=8 outputs=16309worker2 # [ 155.850738] nix-grpc-daemon[1407]: ts=2026-09-21T14:46:11Z level=debug event=want client=5 drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv cached=06310worker1 # [ 155.880324] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:11Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:346566311worker1 # [ 155.889860] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:11Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:346566312worker1 # [ 155.895275] nix-daemon[852]: reaped child process 1920, status = succeeded6313worker1 # [ 155.901668] nix-grpc-daemon[1827]: time=2026-09-21T14:46:11.510Z level=INFO msg="All 1 paths already cached"6314worker1 # [ 155.910151] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:11Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:346566315worker1 # [ 155.923454] nix-daemon[852]: accepted connection from pid 1767, user nix-grpc-daemon (trusted)6316worker1 # [ 156.065732] nix-daemon[852]: accepted connection from pid 1945, user nix-grpc-daemon (trusted)6317worker1 # [ 156.082274] nix-daemon[852]: reaped child process 1948, status = succeeded6318worker1 # [ 156.183585] nix-daemon[852]: accepted connection from pid 1950, user nix-grpc-daemon (trusted)6319worker1 # [ 156.197958] nix-daemon[852]: reaped child process 1953, status = succeeded6320lb # [ 156.209793] niks3-server[1112]: 2026/09/21 14:46:11 INFO Received uploads request method=POST path=/api/pending_closures6321worker1 # [ 156.214706] nix-grpc-daemon[1827]: time=2026-09-21T14:46:11.823Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6322worker1 # [ 156.216576] nix-grpc-daemon[1827]: time=2026-09-21T14:46:11.823Z level=INFO msg="Uploading cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv (624B)"6323lb # [ 156.235995] niks3-server[1112]: 2026/09/21 14:46:11 INFO Registered completed upload object_key=nar/1w4sykcjji9zlw0a5xibxycy6x3jyb4drd29x2k9g7x5whfwwpsq.nar.zst6324worker1 # [ 156.244223] nix-grpc-daemon[1827]: time=2026-09-21T14:46:11.852Z level=INFO msg="Uploading 1 narinfos"6325lb # [ 156.249875] niks3-server[1112]: 2026/09/21 14:46:11 INFO Received sign narinfos request method=POST path=/api/pending_closures/58/sign6326lb # [ 156.261504] niks3-server[1112]: 2026/09/21 14:46:11 INFO Signed narinfos id=58 count=16327lb # [ 156.267679] niks3-server[1112]: 2026/09/21 14:46:11 INFO Registered completed upload object_key=cnf7j5vjqyj799isbg26wgjnrlk208gi.ls6328lb # [ 156.276777] niks3-server[1112]: 2026/09/21 14:46:11 INFO Received complete upload request method=POST path=/api/pending_closures/58/complete6329worker1 # [ 156.278657] nix-grpc-daemon[1827]: time=2026-09-21T14:46:11.887Z level=INFO msg="Upload complete. (342ms)"6330lb # [ 156.283835] niks3-server[1112]: 2026/09/21 14:46:11 INFO Completed upload id=586331lb # [ 156.290203] niks3-server[1112]: 2026/09/21 14:46:11 INFO Registered completed upload object_key=cnf7j5vjqyj799isbg26wgjnrlk208gi.narinfo6332worker1 # [ 156.286828] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:11Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34656 duration_s=0 paths=1 nar_bytes_in=6246333worker1 # [ 156.300403] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:11Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:346566334worker1 # [ 156.306264] nix-daemon[852]: accepted connection from pid 1767, user nix-grpc-daemon (trusted)6335worker1 # [ 156.310732] nix-daemon[852]: reaped child process 1943, status = succeeded6336worker1 # [ 156.321876] nix-daemon[852]: accepted connection from pid 1767, user nix-grpc-daemon (trusted)6337lb # [ 156.892519] envoy[888]: conn peer=[2001:db8:1::3]:50074 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=2175 tx=710 ms=419656338worker1 # [ 157.559574] nix-daemon[852]: reaped child process 1957, status = succeeded6339worker1 # [ 157.655382] nix-daemon[852]: accepted connection from pid 1973, user nix-grpc-daemon (trusted)6340worker1 # [ 157.674156] nix-daemon[852]: reaped child process 1976, status = succeeded6341lb # [ 157.683709] niks3-server[1112]: 2026/09/21 14:46:13 INFO Received uploads request method=POST path=/api/pending_closures6342worker1 # [ 157.689498] nix-grpc-daemon[1827]: time=2026-09-21T14:46:13.297Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6343worker1 # [ 157.693742] nix-grpc-daemon[1827]: time=2026-09-21T14:46:13.299Z level=INFO msg="Uploading p1334cl91hzsgv8bmiqzkwvhbwq24vn1-farm-b (176B)"6344lb # [ 157.726541] niks3-server[1112]: 2026/09/21 14:46:13 INFO Registered completed upload object_key=nar/0pa7j07n0n5p9wcfihhx3357xrxs4h8s57kncplymh51lvzvz2kh.nar.zst6345worker1 # [ 157.734906] nix-grpc-daemon[1827]: time=2026-09-21T14:46:13.343Z level=INFO msg="Uploading 1 narinfos"6346lb # [ 157.738775] niks3-server[1112]: 2026/09/21 14:46:13 INFO Received sign narinfos request method=POST path=/api/pending_closures/59/sign6347lb # [ 157.750460] niks3-server[1112]: 2026/09/21 14:46:13 INFO Signed narinfos id=59 count=16348lb # [ 157.756361] niks3-server[1112]: 2026/09/21 14:46:13 INFO Registered completed upload object_key=log/cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv6349lb # [ 157.764267] niks3-server[1112]: 2026/09/21 14:46:13 INFO Registered completed upload object_key=p1334cl91hzsgv8bmiqzkwvhbwq24vn1.ls6350worker1 # [ 157.770993] nix-grpc-daemon[1827]: time=2026-09-21T14:46:13.379Z level=INFO msg="Upload complete. (237ms)"6351lb # [ 157.776636] niks3-server[1112]: 2026/09/21 14:46:13 INFO Received complete upload request method=POST path=/api/pending_closures/59/complete6352lb # [ 157.783931] niks3-server[1112]: 2026/09/21 14:46:13 INFO Completed upload id=596353worker2 # [ 157.759333] nix-grpc-daemon[1407]: ts=2026-09-21T14:46:13Z level=debug event=assigned drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv worker=192.168.1.3:50051 clients=16354lb # [ 157.789786] niks3-server[1112]: 2026/09/21 14:46:13 INFO Registered completed upload object_key=p1334cl91hzsgv8bmiqzkwvhbwq24vn1.narinfo6355worker2 # [ 157.772130] nix-grpc-daemon[1407]: ts=2026-09-21T14:46:13Z level=debug event=want client=5 drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv cached=06356worker1 # [ 157.785341] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:13Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34656 duration_s=1 drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv assign_id=9 outputs=16357worker1 # [ 157.800544] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:13Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:346566358worker1 # [ 157.806448] nix-daemon[852]: reaped child process 1955, status = succeeded6359worker1 # [ 157.814400] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:13Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:346566360worker1 # [ 157.826739] nix-grpc-daemon[1827]: time=2026-09-21T14:46:13.435Z level=INFO msg="All 1 paths already cached"6361worker1 # [ 157.835353] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:13Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:346566362worker1 # [ 157.848862] nix-daemon[852]: accepted connection from pid 1767, user nix-grpc-daemon (trusted)6363worker1 # [ 157.991569] nix-daemon[852]: accepted connection from pid 1980, user nix-grpc-daemon (trusted)6364worker1 # [ 158.008891] nix-daemon[852]: reaped child process 1983, status = succeeded6365worker1 # [ 158.112491] nix-daemon[852]: accepted connection from pid 1985, user nix-grpc-daemon (trusted)6366worker1 # [ 158.127555] nix-daemon[852]: reaped child process 1988, status = succeeded6367lb # [ 158.138008] niks3-server[1112]: 2026/09/21 14:46:13 INFO Received uploads request method=POST path=/api/pending_closures6368worker1 # [ 158.144772] nix-grpc-daemon[1827]: time=2026-09-21T14:46:13.753Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6369worker1 # [ 158.146254] nix-grpc-daemon[1827]: time=2026-09-21T14:46:13.753Z level=INFO msg="Uploading 7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv (624B)"6370lb # [ 158.170519] niks3-server[1112]: 2026/09/21 14:46:13 INFO Registered completed upload object_key=nar/0l0zss9ls4bhz18slrrmz7l1j93xz7424q7314p64rpgrpqf6b7k.nar.zst6371lb # [ 158.184885] niks3-server[1112]: 2026/09/21 14:46:13 INFO Received sign narinfos request method=POST path=/api/pending_closures/60/sign6372worker1 # [ 158.185687] nix-grpc-daemon[1827]: time=2026-09-21T14:46:13.794Z level=INFO msg="Uploading 1 narinfos"6373lb # [ 158.195319] niks3-server[1112]: 2026/09/21 14:46:13 INFO Signed narinfos id=60 count=16374lb # [ 158.200649] niks3-server[1112]: 2026/09/21 14:46:13 INFO Registered completed upload object_key=7vjzkx8444zm2h7882vsj6bmj73zqjq3.ls6375lb # [ 158.209557] niks3-server[1112]: 2026/09/21 14:46:13 INFO Received complete upload request method=POST path=/api/pending_closures/60/complete6376worker1 # [ 158.210764] nix-grpc-daemon[1827]: time=2026-09-21T14:46:13.819Z level=INFO msg="Upload complete. (348ms)"6377lb # [ 158.216575] niks3-server[1112]: 2026/09/21 14:46:13 INFO Completed upload id=606378lb # [ 158.220542] niks3-server[1112]: 2026/09/21 14:46:13 INFO Registered completed upload object_key=7vjzkx8444zm2h7882vsj6bmj73zqjq3.narinfo6379worker1 # [ 158.220284] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:13Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34656 duration_s=0 paths=1 nar_bytes_in=6246380worker1 # [ 158.234335] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:13Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:346566381worker1 # [ 158.243194] nix-daemon[852]: accepted connection from pid 1767, user nix-grpc-daemon (trusted)6382worker1 # [ 158.246075] nix-daemon[852]: reaped child process 1978, status = succeeded6383worker1 # [ 158.258930] nix-daemon[852]: accepted connection from pid 1767, user nix-grpc-daemon (trusted)6384worker1 # [ 159.761692] nix-daemon[852]: reaped child process 1992, status = succeeded6385worker1 # [ 159.833731] nix-daemon[852]: accepted connection from pid 2008, user nix-grpc-daemon (trusted)6386worker1 # [ 159.853765] nix-daemon[852]: reaped child process 2011, status = succeeded6387lb # [ 159.856906] niks3-server[1112]: 2026/09/21 14:46:15 INFO Received uploads request method=POST path=/api/pending_closures6388worker1 # [ 159.863195] nix-grpc-daemon[1827]: time=2026-09-21T14:46:15.470Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6389worker1 # [ 159.864952] nix-grpc-daemon[1827]: time=2026-09-21T14:46:15.470Z level=INFO msg="Uploading nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8-farm-b (176B)"6390lb # [ 159.888793] niks3-server[1112]: 2026/09/21 14:46:15 INFO Registered completed upload object_key=log/7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv6391lb # [ 159.904129] niks3-server[1112]: 2026/09/21 14:46:15 INFO Registered completed upload object_key=nar/0kqa5m59sp6aj1ixqj6q54wwqxkcn98fw1rc9hfkcdk460wjdl0l.nar.zst6392worker1 # [ 159.914096] nix-grpc-daemon[1827]: time=2026-09-21T14:46:15.522Z level=INFO msg="Uploading 1 narinfos"6393lb # [ 159.917622] niks3-server[1112]: 2026/09/21 14:46:15 INFO Received sign narinfos request method=POST path=/api/pending_closures/61/sign6394lb # [ 159.928669] niks3-server[1112]: 2026/09/21 14:46:15 INFO Signed narinfos id=61 count=16395lb # [ 159.936128] niks3-server[1112]: 2026/09/21 14:46:15 INFO Registered completed upload object_key=nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8.ls6396lb # [ 159.950515] niks3-server[1112]: 2026/09/21 14:46:15 INFO Received complete upload request method=POST path=/api/pending_closures/61/complete6397lb # [ 159.958586] niks3-server[1112]: 2026/09/21 14:46:15 INFO Completed upload id=616398worker1 # [ 159.954011] nix-grpc-daemon[1827]: time=2026-09-21T14:46:15.562Z level=INFO msg="Upload complete. (284ms)"6399lb # [ 159.961738] niks3-server[1112]: 2026/09/21 14:46:15 INFO Registered completed upload object_key=nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8.narinfo6400worker2 # [ 159.944700] nix-grpc-daemon[1407]: ts=2026-09-21T14:46:15Z level=debug event=assigned drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv worker=192.168.1.3:50051 clients=16401worker1 # [ 159.969439] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:15Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34656 duration_s=1 drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv assign_id=10 outputs=16402worker2 # [ 159.956570] nix-grpc-daemon[1407]: ts=2026-09-21T14:46:15Z level=debug event=want client=5 drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv cached=06403worker1 # [ 159.985447] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:15Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:346566404worker1 # [ 159.996946] nix-daemon[852]: reaped child process 1990, status = succeeded6405worker1 # [ 159.999225] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:15Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:346566406worker1 # [ 160.010941] nix-grpc-daemon[1827]: time=2026-09-21T14:46:15.619Z level=INFO msg="All 1 paths already cached"6407worker1 # [ 160.019494] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:15Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:346566408worker1 # [ 160.032273] nix-daemon[852]: accepted connection from pid 1767, user nix-grpc-daemon (trusted)6409worker1 # [ 160.180085] nix-daemon[852]: accepted connection from pid 2015, user nix-grpc-daemon (trusted)6410worker1 # [ 160.208231] nix-daemon[852]: reaped child process 2018, status = succeeded6411worker1 # [ 160.300261] nix-daemon[852]: accepted connection from pid 2020, user nix-grpc-daemon (trusted)6412worker1 # [ 160.315663] nix-daemon[852]: reaped child process 2023, status = succeeded6413lb # [ 160.326243] niks3-server[1112]: 2026/09/21 14:46:15 INFO Received uploads request method=POST path=/api/pending_closures6414worker1 # [ 160.330668] nix-grpc-daemon[1827]: time=2026-09-21T14:46:15.939Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6415worker1 # [ 160.332202] nix-grpc-daemon[1827]: time=2026-09-21T14:46:15.939Z level=INFO msg="Uploading cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv (624B)"6416lb # [ 160.352260] niks3-server[1112]: 2026/09/21 14:46:15 INFO Registered completed upload object_key=nar/0f6g0vggy0qbqsj64l1nivyqdh6q12kh8620ysnf210ani0727vf.nar.zst6417worker1 # [ 160.370873] nix-grpc-daemon[1827]: time=2026-09-21T14:46:15.979Z level=INFO msg="Uploading 1 narinfos"6418lb # [ 160.368238] niks3-server[1112]: 2026/09/21 14:46:15 INFO Received sign narinfos request method=POST path=/api/pending_closures/62/sign6419lb # [ 160.381407] niks3-server[1112]: 2026/09/21 14:46:15 INFO Signed narinfos id=62 count=16420lb # [ 160.388254] niks3-server[1112]: 2026/09/21 14:46:15 INFO Registered completed upload object_key=cgwyij19i3fl7sqlyz54vzfr4zajg077.ls6421worker1 # [ 160.396890] nix-grpc-daemon[1827]: time=2026-09-21T14:46:16.005Z level=INFO msg="Upload complete. (351ms)"6422lb # [ 160.402070] niks3-server[1112]: 2026/09/21 14:46:15 INFO Received complete upload request method=POST path=/api/pending_closures/62/complete6423worker1 # [ 160.404126] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:16Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34656 duration_s=0 paths=1 nar_bytes_in=6246424lb # [ 160.412120] niks3-server[1112]: 2026/09/21 14:46:16 INFO Completed upload id=626425lb # [ 160.415249] niks3-server[1112]: 2026/09/21 14:46:16 INFO Registered completed upload object_key=cgwyij19i3fl7sqlyz54vzfr4zajg077.narinfo6426worker1 # [ 160.419256] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:346566427worker1 # [ 160.426276] nix-daemon[852]: accepted connection from pid 1767, user nix-grpc-daemon (trusted)6428worker1 # [ 160.431209] nix-daemon[852]: reaped child process 2013, status = succeeded6429worker1 # [ 160.442777] nix-daemon[852]: accepted connection from pid 1767, user nix-grpc-daemon (trusted)6430worker1 # [ 162.084757] nix-daemon[852]: reaped child process 2027, status = succeeded6431worker1 # [ 162.193426] nix-daemon[852]: accepted connection from pid 2043, user nix-grpc-daemon (trusted)6432worker1 # [ 162.212355] nix-daemon[852]: reaped child process 2046, status = succeeded6433lb # [ 162.221880] niks3-server[1112]: 2026/09/21 14:46:17 INFO Received uploads request method=POST path=/api/pending_closures6434worker1 # [ 162.228391] nix-grpc-daemon[1827]: time=2026-09-21T14:46:17.835Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6435worker1 # [ 162.230789] nix-grpc-daemon[1827]: time=2026-09-21T14:46:17.835Z level=INFO msg="Uploading hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5-farm-b (176B)"6436lb # [ 162.258604] niks3-server[1112]: 2026/09/21 14:46:17 INFO Registered completed upload object_key=nar/0x964sxcxsdh3nfjarc1ib52kr50kl8zxl7h34jq5ydx763drras.nar.zst6437worker1 # [ 162.267764] nix-grpc-daemon[1827]: time=2026-09-21T14:46:17.876Z level=INFO msg="Uploading 1 narinfos"6438lb # [ 162.272133] niks3-server[1112]: 2026/09/21 14:46:17 INFO Received sign narinfos request method=POST path=/api/pending_closures/63/sign6439lb # [ 162.283009] niks3-server[1112]: 2026/09/21 14:46:17 INFO Signed narinfos id=63 count=16440lb # [ 162.288965] niks3-server[1112]: 2026/09/21 14:46:17 INFO Registered completed upload object_key=log/cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv6441lb # [ 162.296830] niks3-server[1112]: 2026/09/21 14:46:17 INFO Registered completed upload object_key=hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5.ls6442lb # [ 162.302901] niks3-server[1112]: 2026/09/21 14:46:17 INFO Received complete upload request method=POST path=/api/pending_closures/63/complete6443lb # [ 162.311129] niks3-server[1112]: 2026/09/21 14:46:17 INFO Completed upload id=636444worker1 # [ 162.306420] nix-grpc-daemon[1827]: time=2026-09-21T14:46:17.914Z level=INFO msg="Upload complete. (250ms)"6445lb # [ 162.314245] niks3-server[1112]: 2026/09/21 14:46:17 INFO Registered completed upload object_key=hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5.narinfo6446worker2 # [ 162.297337] nix-grpc-daemon[1407]: ts=2026-09-21T14:46:17Z level=debug event=assigned drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv worker=192.168.1.3:50051 clients=16447worker1 # [ 162.317059] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:17Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34656 duration_s=1 drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv assign_id=11 outputs=16448worker2 # [ 162.310170] nix-grpc-daemon[1407]: ts=2026-09-21T14:46:17Z level=debug event=want client=5 drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv cached=06449worker1 # [ 162.335518] nix-daemon[852]: reaped child process 2025, status = succeeded6450worker1 # [ 162.372293] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:17Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:346566451worker1 # [ 162.386611] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:17Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:346566452worker1 # [ 162.400630] nix-grpc-daemon[1827]: time=2026-09-21T14:46:18.009Z level=INFO msg="All 2 paths already cached"6453worker1 # [ 162.409648] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:18Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:346566454worker1 # [ 162.423380] nix-daemon[852]: accepted connection from pid 1767, user nix-grpc-daemon (trusted)6455worker1 # [ 162.578764] nix-daemon[852]: accepted connection from pid 2050, user nix-grpc-daemon (trusted)6456worker1 # [ 162.598842] nix-daemon[852]: reaped child process 2053, status = succeeded6457worker1 # [ 162.700855] nix-daemon[852]: accepted connection from pid 2055, user nix-grpc-daemon (trusted)6458worker1 # [ 162.718274] nix-daemon[852]: reaped child process 2058, status = succeeded6459lb # [ 162.726653] niks3-server[1112]: 2026/09/21 14:46:18 INFO Received uploads request method=POST path=/api/pending_closures6460worker1 # [ 162.731417] nix-grpc-daemon[1827]: time=2026-09-21T14:46:18.339Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6461worker1 # [ 162.733019] nix-grpc-daemon[1827]: time=2026-09-21T14:46:18.339Z level=INFO msg="Uploading dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv (752B)"6462lb # [ 162.753291] niks3-server[1112]: 2026/09/21 14:46:18 INFO Registered completed upload object_key=nar/0s7ff7rwqx5p9mdzddqgzd9bbd5j8njfc62k07mgnpy19810bs86.nar.zst6463worker1 # [ 162.763888] nix-grpc-daemon[1827]: time=2026-09-21T14:46:18.372Z level=INFO msg="Uploading 1 narinfos"6464lb # [ 162.767805] niks3-server[1112]: 2026/09/21 14:46:18 INFO Received sign narinfos request method=POST path=/api/pending_closures/64/sign6465lb # [ 162.780085] niks3-server[1112]: 2026/09/21 14:46:18 INFO Signed narinfos id=64 count=16466lb # [ 162.787550] niks3-server[1112]: 2026/09/21 14:46:18 INFO Registered completed upload object_key=dca2k4dvnk52m7gijqfq8q99ih3ni8ny.ls6467lb # [ 162.800562] niks3-server[1112]: 2026/09/21 14:46:18 INFO Received complete upload request method=POST path=/api/pending_closures/64/complete6468worker1 # [ 162.803186] nix-grpc-daemon[1827]: time=2026-09-21T14:46:18.411Z level=INFO msg="Upload complete. (363ms)"6469lb # [ 162.810193] niks3-server[1112]: 2026/09/21 14:46:18 INFO Completed upload id=646470lb # [ 162.814845] niks3-server[1112]: 2026/09/21 14:46:18 INFO Registered completed upload object_key=dca2k4dvnk52m7gijqfq8q99ih3ni8ny.narinfo6471worker1 # [ 162.812256] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:18Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34656 duration_s=0 paths=1 nar_bytes_in=7526472worker1 # [ 162.824452] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:18Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:346566473worker1 # [ 162.831865] nix-daemon[852]: accepted connection from pid 1767, user nix-grpc-daemon (trusted)6474worker1 # [ 162.837842] nix-daemon[852]: reaped child process 2048, status = succeeded6475worker1 # [ 162.852368] nix-daemon[852]: accepted connection from pid 1767, user nix-grpc-daemon (trusted)6476worker1 # [ 163.942975] nix-daemon[852]: reaped child process 2062, status = succeeded6477worker1 # [ 164.011460] nix-daemon[852]: accepted connection from pid 2078, user nix-grpc-daemon (trusted)6478worker1 # [ 164.028987] nix-daemon[852]: reaped child process 2081, status = succeeded6479lb # [ 164.039894] niks3-server[1112]: 2026/09/21 14:46:19 INFO Received uploads request method=POST path=/api/pending_closures6480worker1 # [ 164.048424] nix-grpc-daemon[1827]: time=2026-09-21T14:46:19.654Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6481worker1 # [ 164.050021] nix-grpc-daemon[1827]: time=2026-09-21T14:46:19.654Z level=INFO msg="Uploading d88xgckxx0s0n5ly2d0jw1qy1641z8pq-farm-top (232B)"6482lb # [ 164.084708] niks3-server[1112]: 2026/09/21 14:46:19 INFO Registered completed upload object_key=nar/0knxhkhz3x04vk8as5wv5ar2d0h02prv7q5f42qm1xjgr70ancd0.nar.zst6483worker1 # [ 164.091127] nix-grpc-daemon[1827]: time=2026-09-21T14:46:19.699Z level=INFO msg="Uploading 1 narinfos"6484lb # [ 164.097679] niks3-server[1112]: 2026/09/21 14:46:19 INFO Received sign narinfos request method=POST path=/api/pending_closures/65/sign6485lb # [ 164.108975] niks3-server[1112]: 2026/09/21 14:46:19 INFO Signed narinfos id=65 count=16486lb # [ 164.111308] niks3-server[1112]: 2026/09/21 14:46:19 INFO Registered completed upload object_key=log/dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv6487lb # [ 164.119916] niks3-server[1112]: 2026/09/21 14:46:19 INFO Registered completed upload object_key=d88xgckxx0s0n5ly2d0jw1qy1641z8pq.ls6488worker1 # [ 164.124191] nix-grpc-daemon[1827]: time=2026-09-21T14:46:19.732Z level=INFO msg="Upload complete. (262ms)"6489lb # [ 164.128625] niks3-server[1112]: 2026/09/21 14:46:19 INFO Received complete upload request method=POST path=/api/pending_closures/65/complete6490lb # [ 164.133920] niks3-server[1112]: 2026/09/21 14:46:19 INFO Completed upload id=656491lb # [ 164.138532] niks3-server[1112]: 2026/09/21 14:46:19 INFO Registered completed upload object_key=d88xgckxx0s0n5ly2d0jw1qy1641z8pq.narinfo6492worker2 # [ 164.111690] nix-grpc-daemon[1407]: ts=2026-09-21T14:46:19Z level=debug event=assigned drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv worker=192.168.1.3:50051 clients=16493worker1 # [ 164.137077] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:19Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34656 duration_s=1 drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv assign_id=12 outputs=16494worker1 # [ 164.156350] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:346566495worker1 # [ 164.163423] nix-daemon[852]: reaped child process 2060, status = succeeded6496worker1 # [ 164.167882] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:19Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:346566497worker1 # [ 164.176586] nix-grpc-daemon[1827]: time=2026-09-21T14:46:19.785Z level=INFO msg="All 2 paths already cached"6498worker1 # [ 164.184487] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:19Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:346566499worker1 # [ 164.196318] nix-daemon[852]: accepted connection from pid 1767, user nix-grpc-daemon (trusted)6500worker1 # [ 164.341177] nix-daemon[852]: accepted connection from pid 2085, user nix-grpc-daemon (trusted)6501worker1 # [ 164.356998] nix-daemon[852]: reaped child process 2088, status = succeeded6502worker1 # [ 164.459256] nix-daemon[852]: accepted connection from pid 2090, user nix-grpc-daemon (trusted)6503worker1 # [ 164.475045] nix-daemon[852]: reaped child process 2093, status = succeeded6504lb # [ 164.485613] niks3-server[1112]: 2026/09/21 14:46:20 INFO Received uploads request method=POST path=/api/pending_closures6505worker1 # [ 164.490063] nix-grpc-daemon[1827]: time=2026-09-21T14:46:20.098Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6506worker1 # [ 164.491504] nix-grpc-daemon[1827]: time=2026-09-21T14:46:20.098Z level=INFO msg="Uploading 2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv (752B)"6507lb # [ 164.515180] niks3-server[1112]: 2026/09/21 14:46:20 INFO Registered completed upload object_key=nar/0cpiyiv7frlhngqhhc703bglz5q2cpg4n8sgds4sfg7nvw0fcdn5.nar.zst6508worker1 # [ 164.522483] nix-grpc-daemon[1827]: time=2026-09-21T14:46:20.130Z level=INFO msg="Uploading 1 narinfos"6509lb # [ 164.531098] niks3-server[1112]: 2026/09/21 14:46:20 INFO Received sign narinfos request method=POST path=/api/pending_closures/66/sign6510lb # [ 164.539365] niks3-server[1112]: 2026/09/21 14:46:20 INFO Signed narinfos id=66 count=16511lb # [ 164.547426] niks3-server[1112]: 2026/09/21 14:46:20 INFO Received complete upload request method=POST path=/api/pending_closures/66/complete6512worker1 # [ 164.549165] nix-grpc-daemon[1827]: time=2026-09-21T14:46:20.157Z level=INFO msg="Upload complete. (340ms)"6513lb # [ 164.555545] niks3-server[1112]: 2026/09/21 14:46:20 INFO Completed upload id=666514lb # [ 164.558522] niks3-server[1112]: 2026/09/21 14:46:20 INFO Registered completed upload object_key=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9.narinfo6515lb # [ 164.563565] niks3-server[1112]: 2026/09/21 14:46:20 INFO Registered completed upload object_key=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9.ls6516worker1 # [ 164.556108] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:20Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34656 duration_s=0 paths=1 nar_bytes_in=7526517worker1 # [ 164.569419] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:20Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:346566518worker1 # [ 164.572933] nix-daemon[852]: reaped child process 2083, status = succeeded6519worker1 # [ 164.579581] nix-daemon[852]: accepted connection from pid 1767, user nix-grpc-daemon (trusted)6520worker1 # [ 164.592236] nix-daemon[852]: accepted connection from pid 1767, user nix-grpc-daemon (trusted)6521worker1 # [ 165.664287] nix-daemon[852]: reaped child process 2097, status = succeeded6522worker1 # [ 165.767923] nix-daemon[852]: accepted connection from pid 2113, user nix-grpc-daemon (trusted)6523worker1 # [ 165.788696] nix-daemon[852]: reaped child process 2116, status = succeeded6524lb # [ 165.794926] niks3-server[1112]: 2026/09/21 14:46:21 INFO Received uploads request method=POST path=/api/pending_closures6525worker1 # [ 165.801687] nix-grpc-daemon[1827]: time=2026-09-21T14:46:21.408Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6526worker1 # [ 165.803272] nix-grpc-daemon[1827]: time=2026-09-21T14:46:21.409Z level=INFO msg="Uploading vz84kyfcw804d0iyx7nj61j47ms9zxmn-farm-top (232B)"6527lb # [ 165.831928] niks3-server[1112]: 2026/09/21 14:46:21 INFO Registered completed upload object_key=log/2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv6528lb # [ 165.843358] niks3-server[1112]: 2026/09/21 14:46:21 INFO Registered completed upload object_key=nar/0aaz4iim7fa45zws98gbhp73l23kbm0fvp4anssnxa96v666ljhs.nar.zst6529worker1 # [ 165.853406] nix-grpc-daemon[1827]: time=2026-09-21T14:46:21.461Z level=INFO msg="Uploading 1 narinfos"6530lb # [ 165.857439] niks3-server[1112]: 2026/09/21 14:46:21 INFO Received sign narinfos request method=POST path=/api/pending_closures/67/sign6531lb # [ 165.864667] niks3-server[1112]: 2026/09/21 14:46:21 INFO Signed narinfos id=67 count=16532lb # [ 165.866990] niks3-server[1112]: 2026/09/21 14:46:21 INFO Registered completed upload object_key=vz84kyfcw804d0iyx7nj61j47ms9zxmn.ls6533lb # [ 165.879182] niks3-server[1112]: 2026/09/21 14:46:21 INFO Received complete upload request method=POST path=/api/pending_closures/67/complete6534lb # [ 165.887991] niks3-server[1112]: 2026/09/21 14:46:21 INFO Completed upload id=676535worker1 # [ 165.883197] nix-grpc-daemon[1827]: time=2026-09-21T14:46:21.491Z level=INFO msg="Upload complete. (281ms)"6536lb # [ 165.891836] niks3-server[1112]: 2026/09/21 14:46:21 INFO Registered completed upload object_key=vz84kyfcw804d0iyx7nj61j47ms9zxmn.narinfo6537worker2 # [ 165.874664] nix-grpc-daemon[1407]: ts=2026-09-21T14:46:21Z level=debug event=assigned drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv worker=192.168.1.3:50051 clients=16538worker1 # [ 165.896626] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:21Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34656 duration_s=1 drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv assign_id=13 outputs=16539worker1 # [ 165.914119] nix-daemon[852]: reaped child process 2095, status = succeeded6540worker1 # [ 165.939955] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:21Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:346566541worker1 # [ 165.955265] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:21Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:346566542worker1 # [ 165.969491] nix-grpc-daemon[1827]: time=2026-09-21T14:46:21.577Z level=INFO msg="All 2 paths already cached"6543worker1 # [ 165.978325] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:21Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:346566544worker1 # [ 165.991952] nix-daemon[852]: accepted connection from pid 1767, user nix-grpc-daemon (trusted)6545worker1 # [ 166.144120] nix-daemon[852]: accepted connection from pid 2120, user nix-grpc-daemon (trusted)6546worker1 # [ 166.161967] nix-daemon[852]: reaped child process 2123, status = succeeded6547worker1 # [ 166.262694] nix-daemon[852]: accepted connection from pid 2125, user nix-grpc-daemon (trusted)6548worker1 # [ 166.278193] nix-daemon[852]: reaped child process 2128, status = succeeded6549lb # [ 166.289749] niks3-server[1112]: 2026/09/21 14:46:21 INFO Received uploads request method=POST path=/api/pending_closures6550worker1 # [ 166.294143] nix-grpc-daemon[1827]: time=2026-09-21T14:46:21.902Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6551worker1 # [ 166.295592] nix-grpc-daemon[1827]: time=2026-09-21T14:46:21.902Z level=INFO msg="Uploading qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv (752B)"6552lb # [ 166.315725] niks3-server[1112]: 2026/09/21 14:46:21 INFO Registered completed upload object_key=nar/05zmsnqx4aj55yncx917mz6d3hzrvz2rx2x75313vvy1kq1zv9fv.nar.zst6553worker1 # [ 166.325789] nix-grpc-daemon[1827]: time=2026-09-21T14:46:21.934Z level=INFO msg="Uploading 1 narinfos"6554lb # [ 166.330866] niks3-server[1112]: 2026/09/21 14:46:21 INFO Received sign narinfos request method=POST path=/api/pending_closures/68/sign6555lb # [ 166.342781] niks3-server[1112]: 2026/09/21 14:46:21 INFO Signed narinfos id=68 count=16556lb # [ 166.348646] niks3-server[1112]: 2026/09/21 14:46:21 INFO Registered completed upload object_key=qx10pa00pilyp5xdzd48l88d5jz143mr.ls6557lb # [ 166.356207] niks3-server[1112]: 2026/09/21 14:46:21 INFO Received complete upload request method=POST path=/api/pending_closures/68/complete6558worker1 # [ 166.356767] nix-grpc-daemon[1827]: time=2026-09-21T14:46:21.965Z level=INFO msg="Upload complete. (348ms)"6559lb # [ 166.364114] niks3-server[1112]: 2026/09/21 14:46:21 INFO Completed upload id=686560worker1 # [ 166.364147] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:21Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34656 duration_s=0 paths=1 nar_bytes_in=7526561lb # [ 166.367950] niks3-server[1112]: 2026/09/21 14:46:21 INFO Registered completed upload object_key=qx10pa00pilyp5xdzd48l88d5jz143mr.narinfo6562worker1 # [ 166.375979] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:21Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:346566563worker1 # [ 166.379462] nix-daemon[852]: reaped child process 2118, status = succeeded6564worker1 # [ 166.386328] nix-daemon[852]: accepted connection from pid 1767, user nix-grpc-daemon (trusted)6565worker1 # [ 166.398631] nix-daemon[852]: accepted connection from pid 1767, user nix-grpc-daemon (trusted)6566worker1 # [ 167.628201] nix-daemon[852]: reaped child process 2132, status = succeeded6567worker1 # [ 167.728384] nix-daemon[852]: accepted connection from pid 2148, user nix-grpc-daemon (trusted)6568worker1 # [ 167.746048] nix-daemon[852]: reaped child process 2151, status = succeeded6569lb # [ 167.757882] niks3-server[1112]: 2026/09/21 14:46:23 INFO Received uploads request method=POST path=/api/pending_closures6570worker1 # [ 167.767160] nix-grpc-daemon[1827]: time=2026-09-21T14:46:23.372Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6571worker1 # [ 167.768769] nix-grpc-daemon[1827]: time=2026-09-21T14:46:23.372Z level=INFO msg="Uploading iagwbb072hnm79qb9wsrjp5907zc7aa2-farm-top (232B)"6572lb # [ 167.797141] niks3-server[1112]: 2026/09/21 14:46:23 INFO Registered completed upload object_key=log/qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv6573lb # [ 167.810308] niks3-server[1112]: 2026/09/21 14:46:23 INFO Registered completed upload object_key=nar/053jd22j3kv7v6j0q7cspdpc0h1c1xmywmajl6sd5hzwg50z2yj1.nar.zst6574worker1 # [ 167.817388] nix-grpc-daemon[1827]: time=2026-09-21T14:46:23.425Z level=INFO msg="Uploading 1 narinfos"6575lb # [ 167.824440] niks3-server[1112]: 2026/09/21 14:46:23 INFO Received sign narinfos request method=POST path=/api/pending_closures/69/sign6576lb # [ 167.832858] niks3-server[1112]: 2026/09/21 14:46:23 INFO Signed narinfos id=69 count=16577lb # [ 167.835329] niks3-server[1112]: 2026/09/21 14:46:23 INFO Registered completed upload object_key=iagwbb072hnm79qb9wsrjp5907zc7aa2.ls6578lb # [ 167.846359] niks3-server[1112]: 2026/09/21 14:46:23 INFO Received complete upload request method=POST path=/api/pending_closures/69/complete6579lb # [ 167.851716] niks3-server[1112]: 2026/09/21 14:46:23 INFO Completed upload id=696580worker1 # [ 167.847862] nix-grpc-daemon[1827]: time=2026-09-21T14:46:23.456Z level=INFO msg="Upload complete. (240ms)"6581lb # [ 167.855326] niks3-server[1112]: 2026/09/21 14:46:23 INFO Registered completed upload object_key=iagwbb072hnm79qb9wsrjp5907zc7aa2.narinfo6582worker1 # [ 167.861143] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:23Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34656 duration_s=1 drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv assign_id=14 outputs=16583worker1 # [ 167.877979] nix-daemon[852]: reaped child process 2130, status = succeeded6584client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&1, in 18.43 seconds)6585worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6586worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)6587client: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag f5 --arg features '["gpu"]'6588client # [ 168.052829] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build -L --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&restart-grace=30" --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag f5 --arg features "[\"gpu\"]".6589client # Running as unit: gpu.service; invocation ID: 865a0a652f284df391777fa094053b016590client: (finished: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag f5 --arg features '["gpu"]', in 0.09 seconds)6591lb: must succeed: curl -sf localhost:9901/clusters6592lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)6593lb: must succeed: curl -sf localhost:9901/clusters6594lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6595worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6596worker2 # [ 168.227412] nix-grpc-daemon[1407]: ts=2026-09-21T14:46:23Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:334046597client # [ 168.256153] nix[1411]: don't know how to build these paths:6598client # [ 168.257254] nix[1411]: /nix/store/3i1nziwpdqq19lzpqy4rywzxnm8dlps6-farm-top.drv6599worker2 # [ 168.245397] nix-grpc-daemon[1407]: ts=2026-09-21T14:46:23Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:334046600worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.09 seconds)6601worker2 # [ 168.266362] nix-grpc-daemon[1407]: ts=2026-09-21T14:46:23Z level=debug event=want client=6 drv=1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv cached=06602client # [ 168.284734] nix[1411]: warning: 1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv: no connected, non-draining worker for system 'aarch64-linux' offers features {gpu}, waiting up to 30s for a worker6603lb: must succeed: curl -sf localhost:9901/clusters6604lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)6605lb: must succeed: curl -sf localhost:9901/clusters6606lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6607worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6608worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6609client: waiting for success: systemctl show -p Result gpu | grep -q exit-code6610lb # [ 176.893472] envoy[888]: conn peer=[2001:db8:1::1]:55428 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=27131 tx=11378 ms=181626611client # [ 198.289739] nix[1411]: 1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv: no connected, non-draining worker for system 'aarch64-linux' offers features {gpu}6612client # [ 198.302185] nix[1411]: error: dependency '1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv' failed6613client # [ 198.330757] systemd[1]: gpu.service: Main process exited, code=exited, status=1/FAILURE6614client # [ 198.341621] systemd[1]: gpu.service: Failed with result 'exit-code'.6615client # [ 198.345451] systemd[1]: gpu.service: Consumed 185ms CPU time over 30.287s wall clock time, 35.4M memory peak, 5.1K incoming IP traffic, 5K outgoing IP traffic.6616client: (finished: waiting for success: systemctl show -p Result gpu | grep -q exit-code, in 29.03 seconds)6617client: must succeed: journalctl -u gpu -o cat6618client: (finished: must succeed: journalctl -u gpu -o cat, in 0.03 seconds)6619lb: must succeed: curl -sf localhost:9901/clusters6620lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)6621lb: must succeed: curl -sf localhost:9901/clusters6622lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6623worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6624worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6625(finished: subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does, in 49.21 seconds)6626subtest: low disk drains a worker and builds go to the other6627worker1: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill6628worker1: (finished: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill, in 1.21 seconds)6629worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free'6630worker1 # [ 200.242789] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:55Z level=info event=unhealthy reason=min_free6631worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free', in 1.10 seconds)6632client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag drain >&26633client # don't know how to build these paths:6634client # /nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv6635worker1 # [ 201.225143] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:56Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:346566636worker1 # [ 201.233310] nix-grpc-daemon[1767]: ts=2026-09-21T14:46:56Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:346566637worker2 # [ 201.238936] nix-grpc-daemon[1407]: ts=2026-09-21T14:46:56Z level=debug event=want client=7 drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv cached=06638worker2 # [ 201.244585] nix-grpc-daemon[1407]: ts=2026-09-21T14:46:56Z level=debug event=assigned drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv worker=192.168.1.4:50051 clients=16639worker2 # [ 201.284608] nix-grpc-daemon[1407]: ts=2026-09-21T14:46:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:334046640worker2 # [ 201.299316] nix-grpc-daemon[1407]: ts=2026-09-21T14:46:56Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:334046641client # copying 1 paths to 'lb:50051'...6642client # copying path '/nix/store/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv' to 'lb:50051'...6643worker2 # [ 201.315583] nix-grpc-daemon[1407]: ts=2026-09-21T14:46:56Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:334046644worker2 # [ 201.331004] nix-daemon[838]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6645worker2 # [ 201.490620] nix-daemon[838]: accepted connection from pid 1703, user nix-grpc-daemon (trusted)6646worker2 # [ 201.508872] nix-daemon[838]: reaped child process 1706, status = succeeded6647worker2 # [ 201.616510] nix-daemon[838]: accepted connection from pid 1708, user nix-grpc-daemon (trusted)6648worker2 # [ 201.634059] nix-daemon[838]: reaped child process 1711, status = succeeded6649lb # [ 201.664163] niks3-server[1112]: 2026/09/21 14:46:57 INFO Received uploads request method=POST path=/api/pending_closures6650worker2 # [ 201.649810] nix-grpc-daemon[1485]: time=2026-09-21T14:46:57.252Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6651worker2 # [ 201.651281] nix-grpc-daemon[1485]: time=2026-09-21T14:46:57.252Z level=INFO msg="Uploading kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv (512B)"6652lb # [ 201.704456] niks3-server[1112]: 2026/09/21 14:46:57 INFO Registered completed upload object_key=nar/1sfy0gcq4znfcj1f68c749r5x2jwmwsfbfp6xdflz5xkndjwjjm3.nar.zst6653lb # [ 201.717057] niks3-server[1112]: 2026/09/21 14:46:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/70/sign6654worker2 # [ 201.697404] nix-grpc-daemon[1485]: time=2026-09-21T14:46:57.300Z level=INFO msg="Uploading 1 narinfos"6655lb # [ 201.723847] niks3-server[1112]: 2026/09/21 14:46:57 INFO Signed narinfos id=70 count=16656lb # [ 201.738700] niks3-server[1112]: 2026/09/21 14:46:57 INFO Registered completed upload object_key=kksl1jqfg56vabgqvrkz5vki6nq7r59r.ls6657lb # [ 201.750773] niks3-server[1112]: 2026/09/21 14:46:57 INFO Received complete upload request method=POST path=/api/pending_closures/70/complete6658lb # [ 201.758135] niks3-server[1112]: 2026/09/21 14:46:57 INFO Completed upload id=706659worker2 # [ 201.732032] nix-grpc-daemon[1485]: time=2026-09-21T14:46:57.334Z level=INFO msg="Upload complete. (384ms)"6660lb # [ 201.759997] niks3-server[1112]: 2026/09/21 14:46:57 INFO Registered completed upload object_key=kksl1jqfg56vabgqvrkz5vki6nq7r59r.narinfo6661worker2 # [ 201.743856] nix-grpc-daemon[1407]: ts=2026-09-21T14:46:57Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33404 duration_s=0 paths=1 nar_bytes_in=5126662worker2 # [ 201.758552] nix-grpc-daemon[1407]: ts=2026-09-21T14:46:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:334046663worker2 # [ 201.765138] nix-daemon[838]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6664worker2 # [ 201.770313] nix-daemon[838]: reaped child process 1701, status = succeeded6665client # building '/nix/store/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv' on lb:50051...6666client # farm-a> worker2: building kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv6667worker2 # [ 201.783254] nix-daemon[838]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6668client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds6669client # farm-a> LOG-farm-a-drain6670worker2 # [ 203.414760] nix-daemon[838]: reaped child process 1715, status = succeeded6671worker2 # [ 203.520088] nix-daemon[838]: accepted connection from pid 1731, user nix-grpc-daemon (trusted)6672worker2 # [ 203.536474] nix-daemon[838]: reaped child process 1734, status = succeeded6673lb # [ 203.569274] niks3-server[1112]: 2026/09/21 14:46:59 INFO Received uploads request method=POST path=/api/pending_closures6674worker2 # [ 203.555111] nix-grpc-daemon[1485]: time=2026-09-21T14:46:59.155Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6675worker2 # [ 203.556720] nix-grpc-daemon[1485]: time=2026-09-21T14:46:59.155Z level=INFO msg="Uploading 272b39wn7cbnhkcc2rrz2f6kmhhmcwnr-farm-a (128B)"6676lb # [ 203.605788] niks3-server[1112]: 2026/09/21 14:46:59 INFO Registered completed upload object_key=nar/1bf9z7y0ikx07b4zjag42zyrv2pq1ycw4461s579kfpmni9kz4dw.nar.zst6677lb # [ 203.619639] niks3-server[1112]: 2026/09/21 14:46:59 INFO Registered completed upload object_key=log/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv6678worker2 # [ 203.602678] nix-grpc-daemon[1485]: time=2026-09-21T14:46:59.205Z level=INFO msg="Uploading 1 narinfos"6679lb # [ 203.630939] niks3-server[1112]: 2026/09/21 14:46:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/71/sign6680lb # [ 203.638791] niks3-server[1112]: 2026/09/21 14:46:59 INFO Signed narinfos id=71 count=16681lb # [ 203.645472] niks3-server[1112]: 2026/09/21 14:46:59 INFO Received complete upload request method=POST path=/api/pending_closures/71/complete6682worker2 # [ 203.626728] nix-grpc-daemon[1485]: time=2026-09-21T14:46:59.229Z level=INFO msg="Upload complete. (233ms)"6683lb # [ 203.653328] niks3-server[1112]: 2026/09/21 14:46:59 INFO Completed upload id=716684lb # [ 203.657287] niks3-server[1112]: 2026/09/21 14:46:59 INFO Registered completed upload object_key=272b39wn7cbnhkcc2rrz2f6kmhhmcwnr.narinfo6685lb # [ 203.661609] niks3-server[1112]: 2026/09/21 14:46:59 INFO Registered completed upload object_key=272b39wn7cbnhkcc2rrz2f6kmhhmcwnr.ls6686worker2 # [ 203.641241] nix-grpc-daemon[1407]: ts=2026-09-21T14:46:59Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33404 duration_s=1 drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv assign_id=15 outputs=16687worker2 # [ 203.657740] nix-grpc-daemon[1407]: ts=2026-09-21T14:46:59Z level=debug event=want client=7 drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv cached=06688worker2 # [ 203.662768] nix-grpc-daemon[1407]: ts=2026-09-21T14:46:59Z level=debug event=assigned drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv worker=192.168.1.4:50051 clients=16689worker2 # [ 203.667541] nix-daemon[838]: reaped child process 1713, status = succeeded6690worker2 # [ 203.703673] nix-grpc-daemon[1407]: ts=2026-09-21T14:46:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:334046691worker2 # [ 203.718553] nix-grpc-daemon[1407]: ts=2026-09-21T14:46:59Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:334046692worker2 # [ 203.732404] nix-grpc-daemon[1485]: time=2026-09-21T14:46:59.335Z level=INFO msg="All 1 paths already cached"6693client # copying 1 paths to 'lb:50051'...6694client # copying path '/nix/store/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv' to 'lb:50051'...6695worker2 # [ 203.743214] nix-grpc-daemon[1407]: ts=2026-09-21T14:46:59Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:334046696worker2 # [ 203.756733] nix-daemon[838]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6697worker2 # [ 203.911064] nix-daemon[838]: accepted connection from pid 1740, user nix-grpc-daemon (trusted)6698worker2 # [ 203.928676] nix-daemon[838]: reaped child process 1743, status = succeeded6699worker2 # [ 204.033879] nix-daemon[838]: accepted connection from pid 1745, user nix-grpc-daemon (trusted)6700worker2 # [ 204.049586] nix-daemon[838]: reaped child process 1748, status = succeeded6701lb # [ 204.081179] niks3-server[1112]: 2026/09/21 14:46:59 INFO Received uploads request method=POST path=/api/pending_closures6702worker2 # [ 204.065377] nix-grpc-daemon[1485]: time=2026-09-21T14:46:59.668Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6703worker2 # [ 204.066892] nix-grpc-daemon[1485]: time=2026-09-21T14:46:59.668Z level=INFO msg="Uploading fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv (632B)"6704lb # [ 204.107692] niks3-server[1112]: 2026/09/21 14:46:59 INFO Registered completed upload object_key=nar/1ax4z88sb8d24gsvydxvgdy2lijkmhnycxhr5war1j3sq97cyzqf.nar.zst6705worker2 # [ 204.101807] nix-grpc-daemon[1485]: time=2026-09-21T14:46:59.704Z level=INFO msg="Uploading 1 narinfos"6706lb # [ 204.126090] niks3-server[1112]: 2026/09/21 14:46:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/72/sign6707lb # [ 204.136906] niks3-server[1112]: 2026/09/21 14:46:59 INFO Signed narinfos id=72 count=16708lb # [ 204.144538] niks3-server[1112]: 2026/09/21 14:46:59 INFO Registered completed upload object_key=fa7adcm0x8sbgmrsfx912fp8xakk99g6.ls6709lb # [ 204.155418] niks3-server[1112]: 2026/09/21 14:46:59 INFO Received complete upload request method=POST path=/api/pending_closures/72/complete6710worker2 # [ 204.135980] nix-grpc-daemon[1485]: time=2026-09-21T14:46:59.738Z level=INFO msg="Upload complete. (362ms)"6711lb # [ 204.162113] niks3-server[1112]: 2026/09/21 14:46:59 INFO Completed upload id=726712lb # [ 204.166285] niks3-server[1112]: 2026/09/21 14:46:59 INFO Registered completed upload object_key=fa7adcm0x8sbgmrsfx912fp8xakk99g6.narinfo6713worker2 # [ 204.144977] nix-grpc-daemon[1407]: ts=2026-09-21T14:46:59Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33404 duration_s=0 paths=1 nar_bytes_in=6326714worker2 # [ 204.158237] nix-grpc-daemon[1407]: ts=2026-09-21T14:46:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:334046715worker2 # [ 204.167008] nix-daemon[838]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6716worker2 # [ 204.170449] nix-daemon[838]: reaped child process 1738, status = succeeded6717client # building '/nix/store/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv' on lb:50051...6718worker2 # [ 204.182339] nix-daemon[838]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6719client # farm-b> worker2: building fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv6720client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds6721client # farm-b> LOG-farm-b-drain6722worker2 # [ 205.656658] nix-daemon[838]: reaped child process 1752, status = succeeded6723worker2 # [ 205.762530] nix-daemon[838]: accepted connection from pid 1768, user nix-grpc-daemon (trusted)6724worker2 # [ 205.782642] nix-daemon[838]: reaped child process 1771, status = succeeded6725lb # [ 205.813130] niks3-server[1112]: 2026/09/21 14:47:01 INFO Received uploads request method=POST path=/api/pending_closures6726worker2 # [ 205.800420] nix-grpc-daemon[1485]: time=2026-09-21T14:47:01.400Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6727worker2 # [ 205.802062] nix-grpc-daemon[1485]: time=2026-09-21T14:47:01.400Z level=INFO msg="Uploading g774lks418zc45bd9hwhinm61pwcwnr9-farm-b (176B)"6728lb # [ 205.849709] niks3-server[1112]: 2026/09/21 14:47:01 INFO Registered completed upload object_key=log/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv6729lb # [ 205.866580] niks3-server[1112]: 2026/09/21 14:47:01 INFO Registered completed upload object_key=nar/1ammzz4zx9sggy8v73zxk3j08vbqxgpw14n8lx2fh57bx24yzvnj.nar.zst6730lb # [ 205.879345] niks3-server[1112]: 2026/09/21 14:47:01 INFO Received sign narinfos request method=POST path=/api/pending_closures/73/sign6731lb # [ 205.884269] niks3-server[1112]: 2026/09/21 14:47:01 INFO Signed narinfos id=73 count=16732worker2 # [ 205.859653] nix-grpc-daemon[1485]: time=2026-09-21T14:47:01.462Z level=INFO msg="Uploading 1 narinfos"6733lb # [ 205.891979] niks3-server[1112]: 2026/09/21 14:47:01 INFO Registered completed upload object_key=g774lks418zc45bd9hwhinm61pwcwnr9.ls6734lb # [ 205.900687] niks3-server[1112]: 2026/09/21 14:47:01 INFO Received complete upload request method=POST path=/api/pending_closures/73/complete6735lb # [ 205.908307] niks3-server[1112]: 2026/09/21 14:47:01 INFO Completed upload id=736736worker2 # [ 205.882971] nix-grpc-daemon[1485]: time=2026-09-21T14:47:01.485Z level=INFO msg="Upload complete. (287ms)"6737lb # [ 205.910787] niks3-server[1112]: 2026/09/21 14:47:01 INFO Registered completed upload object_key=g774lks418zc45bd9hwhinm61pwcwnr9.narinfo6738worker2 # [ 205.901015] nix-grpc-daemon[1407]: ts=2026-09-21T14:47:01Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33404 duration_s=1 drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv assign_id=16 outputs=16739worker2 # [ 205.916377] nix-grpc-daemon[1407]: ts=2026-09-21T14:47:01Z level=debug event=want client=7 drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv cached=06740worker2 # [ 205.922830] nix-grpc-daemon[1407]: ts=2026-09-21T14:47:01Z level=debug event=assigned drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv worker=192.168.1.4:50051 clients=16741worker2 # [ 205.927277] nix-daemon[838]: reaped child process 1750, status = succeeded6742worker2 # [ 205.943693] nix-grpc-daemon[1407]: ts=2026-09-21T14:47:01Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:334046743worker2 # [ 205.958936] nix-grpc-daemon[1407]: ts=2026-09-21T14:47:01Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:334046744worker2 # [ 205.973316] nix-grpc-daemon[1485]: time=2026-09-21T14:47:01.576Z level=INFO msg="All 2 paths already cached"6745client # copying 1 paths to 'lb:50051'...6746client # copying path '/nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv' to 'lb:50051'...6747worker2 # [ 205.983651] nix-grpc-daemon[1407]: ts=2026-09-21T14:47:01Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:334046748worker2 # [ 205.997541] nix-daemon[838]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6749worker2 # [ 206.157427] nix-daemon[838]: accepted connection from pid 1777, user nix-grpc-daemon (trusted)6750worker2 # [ 206.183205] nix-daemon[838]: reaped child process 1780, status = succeeded6751worker2 # [ 206.278623] nix-daemon[838]: accepted connection from pid 1782, user nix-grpc-daemon (trusted)6752worker2 # [ 206.293553] nix-daemon[838]: reaped child process 1785, status = succeeded6753lb # [ 206.325384] niks3-server[1112]: 2026/09/21 14:47:01 INFO Received uploads request method=POST path=/api/pending_closures6754worker2 # [ 206.309504] nix-grpc-daemon[1485]: time=2026-09-21T14:47:01.912Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6755worker2 # [ 206.311014] nix-grpc-daemon[1485]: time=2026-09-21T14:47:01.912Z level=INFO msg="Uploading 77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv (760B)"6756lb # [ 206.351590] niks3-server[1112]: 2026/09/21 14:47:01 INFO Registered completed upload object_key=nar/1hrr94bvb37dwmf1giypzl9qnff72xm540ak7q49j34fyzia9ngb.nar.zst6757worker2 # [ 206.341106] nix-grpc-daemon[1485]: time=2026-09-21T14:47:01.943Z level=INFO msg="Uploading 1 narinfos"6758lb # [ 206.365452] niks3-server[1112]: 2026/09/21 14:47:01 INFO Received sign narinfos request method=POST path=/api/pending_closures/74/sign6759lb # [ 206.375607] niks3-server[1112]: 2026/09/21 14:47:01 INFO Signed narinfos id=74 count=16760lb # [ 206.383679] niks3-server[1112]: 2026/09/21 14:47:01 INFO Registered completed upload object_key=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l.ls6761lb # [ 206.396211] niks3-server[1112]: 2026/09/21 14:47:01 INFO Received complete upload request method=POST path=/api/pending_closures/74/complete6762worker2 # [ 206.376129] nix-grpc-daemon[1485]: time=2026-09-21T14:47:01.979Z level=INFO msg="Upload complete. (362ms)"6763lb # [ 206.405659] niks3-server[1112]: 2026/09/21 14:47:02 INFO Completed upload id=746764lb # [ 206.408753] niks3-server[1112]: 2026/09/21 14:47:02 INFO Registered completed upload object_key=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l.narinfo6765worker2 # [ 206.384746] nix-grpc-daemon[1407]: ts=2026-09-21T14:47:01Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33404 duration_s=0 paths=1 nar_bytes_in=7606766worker2 # [ 206.392593] nix-grpc-daemon[1407]: ts=2026-09-21T14:47:01Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:334046767worker2 # [ 206.404410] nix-daemon[838]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6768worker2 # [ 206.406966] nix-daemon[838]: reaped child process 1775, status = succeeded6769worker2 # [ 206.417217] nix-daemon[838]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6770client # building '/nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv' on lb:50051...6771client # farm-top> worker2: building 77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv6772lb # [ 206.896668] envoy[888]: conn peer=[2001:db8:1::1]:57888 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=897 tx=500 ms=300676773client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds6774client # farm-top> LOG-farm-top-drain6775worker2 # [ 208.326502] nix-daemon[838]: reaped child process 1789, status = succeeded6776worker2 # [ 208.398904] nix-daemon[838]: accepted connection from pid 1810, user nix-grpc-daemon (trusted)6777worker2 # [ 208.417468] nix-daemon[838]: reaped child process 1813, status = succeeded6778lb # [ 208.446871] niks3-server[1112]: 2026/09/21 14:47:04 INFO Received uploads request method=POST path=/api/pending_closures6779worker2 # [ 208.435761] nix-grpc-daemon[1485]: time=2026-09-21T14:47:04.037Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6780worker2 # [ 208.437651] nix-grpc-daemon[1485]: time=2026-09-21T14:47:04.037Z level=INFO msg="Uploading bzkv277k4v9r9h4aq5h13ddcavmp36fg-farm-top (232B)"6781lb # [ 208.488085] niks3-server[1112]: 2026/09/21 14:47:04 INFO Registered completed upload object_key=log/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv6782lb # [ 208.498434] niks3-server[1112]: 2026/09/21 14:47:04 INFO Registered completed upload object_key=nar/0l8sy8grn9pmq2n58ag0hx791zdfyxwk9319ynjqmn7yy7wl64s7.nar.zst6783worker2 # [ 208.485608] nix-grpc-daemon[1485]: time=2026-09-21T14:47:04.088Z level=INFO msg="Uploading 1 narinfos"6784lb # [ 208.513394] niks3-server[1112]: 2026/09/21 14:47:04 INFO Received sign narinfos request method=POST path=/api/pending_closures/75/sign6785lb # [ 208.522327] niks3-server[1112]: 2026/09/21 14:47:04 INFO Signed narinfos id=75 count=16786lb # [ 208.529718] niks3-server[1112]: 2026/09/21 14:47:04 INFO Registered completed upload object_key=bzkv277k4v9r9h4aq5h13ddcavmp36fg.ls6787worker2 # [ 208.511546] nix-grpc-daemon[1485]: time=2026-09-21T14:47:04.114Z level=INFO msg="Upload complete. (281ms)"6788lb # [ 208.538239] niks3-server[1112]: 2026/09/21 14:47:04 INFO Received complete upload request method=POST path=/api/pending_closures/75/complete6789lb # [ 208.543900] niks3-server[1112]: 2026/09/21 14:47:04 INFO Completed upload id=756790lb # [ 208.547967] niks3-server[1112]: 2026/09/21 14:47:04 INFO Registered completed upload object_key=bzkv277k4v9r9h4aq5h13ddcavmp36fg.narinfo6791worker2 # [ 208.524783] nix-grpc-daemon[1407]: ts=2026-09-21T14:47:04Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33404 duration_s=2 drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv assign_id=17 outputs=16792worker2 # [ 208.544257] nix-daemon[838]: reaped child process 1787, status = succeeded6793client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag drain >&2, in 7.58 seconds)6794client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag drain outPath6795client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag drain outPath, in 0.18 seconds)6796worker1: must succeed: rm /nix/.rw-store/fill6797worker1: (finished: must succeed: rm /nix/.rw-store/fill, in 0.15 seconds)6798worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free'6799worker1 # [ 209.253254] nix-grpc-daemon[1767]: ts=2026-09-21T14:47:04Z level=info event=healthy reason=min_free6800worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free', in 1.09 seconds)6801(finished: subtest: low disk drains a worker and builds go to the other, in 11.39 seconds)6802(finished: run the VM test script, in 211.01 seconds)6803test script finished in 211.26s6804cleanup6805kill QemuMachine (pid 50)6806client # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/3fl7bdkdxk2k4nssy1d1161isbzn6bsr-python3-3.14.7/bin/python3.14)6807client # [2026-09-21T14:47:05Z INFO virtiofsd] Client disconnected, shutting down6808client # [2026-09-21T14:47:05Z INFO virtiofsd] Client disconnected, shutting down6809client # [2026-09-21T14:47:05Z INFO virtiofsd] Client disconnected, shutting down6810kill QemuMachine (pid 49)6811lb # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/3fl7bdkdxk2k4nssy1d1161isbzn6bsr-python3-3.14.7/bin/python3.14)6812lb # [2026-09-21T14:47:05Z INFO virtiofsd] Client disconnected, shutting down6813lb # [2026-09-21T14:47:05Z INFO virtiofsd] Client disconnected, shutting down6814lb # [2026-09-21T14:47:05Z INFO virtiofsd] Client disconnected, shutting down6815kill QemuMachine (pid 51)6816worker1 # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/3fl7bdkdxk2k4nssy1d1161isbzn6bsr-python3-3.14.7/bin/python3.14)6817worker1 # [2026-09-21T14:47:06Z INFO virtiofsd] Client disconnected, shutting down6818worker1 # [2026-09-21T14:47:06Z INFO virtiofsd] Client disconnected, shutting down6819worker1 # [2026-09-21T14:47:06Z INFO virtiofsd] Client disconnected, shutting down6820kill QemuMachine (pid 52)6821worker2 # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/3fl7bdkdxk2k4nssy1d1161isbzn6bsr-python3-3.14.7/bin/python3.14)6822worker2 # [2026-09-21T14:47:06Z INFO virtiofsd] Client disconnected, shutting down6823worker2 # [2026-09-21T14:47:06Z INFO virtiofsd] Client disconnected, shutting down6824worker2 # [2026-09-21T14:47:06Z INFO virtiofsd] Client disconnected, shutting down6825(finished: cleanup, in 1.19 seconds)6826additionally exposed symbols:6827 client, lb, worker1, worker2,6828 vlan1,6829 start_all, test_script, machines, machines_qemu, machines_nspawn, vlans, driver, log, os, create_machine, subtest, run_tests, join_all, retry, serial_stdout_off, serial_stdout_on, polling_condition, BaseMachine, QemuMachine, NspawnMachine, t, debug, dump_machine_ssh