vm-test-run-nix-grpc-farm
checks.aarch64-linux.farm
· build #479
· raw
1tribuchet: building on eliza2Machine state will be reset. To keep it, pass --keep-machine-state3start all VLans4(finished: start all VLans, in 0.00 seconds)5Test will time out and terminate in 900.0 seconds6run the VM test script7start all VMs8client: starting vm9lb: starting vm10worker1: starting vm11worker2: starting vm12worker1 # Disk image does not exist, creating the virtualisation disk image...13client # Disk image does not exist, creating the virtualisation disk image...14worker1 # Formatting '/build/vm-state-worker1/tmp.KyWLMXtAth', fmt=raw size=107374182415client: QEMU running (pid 49)16lb # Disk image does not exist, creating the virtualisation disk image...17worker1 # mke2fs 1.47.4 (6-Mar-2025)18client # Formatting '/build/vm-state-client/tmp.CRRsMTuYGn', fmt=raw size=107374182419lb # Formatting '/build/vm-state-lb/tmp.acQ1AAQA2J', fmt=raw size=107374182420worker2: QEMU running (pid 51)21worker2 # Disk image does not exist, creating the virtualisation disk image...22client # mke2fs 1.47.4 (6-Mar-2025)23lb # mke2fs 1.47.4 (6-Mar-2025)24worker1: QEMU running (pid 52)25lb: QEMU running (pid 50)26worker1 # Discarding device blocks: 0/262144 done27worker2 # Formatting '/build/vm-state-worker2/tmp.J6K5MgQ6Oy', fmt=raw size=107374182428lb # Discarding device blocks: 0/262144 done29client # Discarding device blocks: 0/262144 done30worker1 # Creating filesystem with 262144 4k blocks and 65536 inodes31lb # Creating filesystem with 262144 4k blocks and 65536 inodes32client # Creating filesystem with 262144 4k blocks and 65536 inodes33worker1 # Filesystem UUID: 58c98773-28c8-46e8-8fa1-b320d4a88aa034lb # Filesystem UUID: 110a27e5-9947-4241-82fa-e97c2e01489235client # Filesystem UUID: 820818e7-b13f-4385-ad4b-fb2af3f1dbfb36worker1 # Superblock backups stored on blocks:37lb # Superblock backups stored on blocks:38client # Superblock backups stored on blocks:39worker1 # 32768, 98304, 163840, 22937640lb # 32768, 98304, 163840, 22937641client # 32768, 98304, 163840, 22937642worker1 # 43lb # 44client # 45worker1 # Allocating group tables: 0/8 done46lb # Allocating group tables: 0/8 done47client # Allocating group tables: 0/8 done48worker1 # Writing inode tables: 0/8 done49lb # Writing inode tables: 0/8 done50client # Writing inode tables: 0/8 done51worker1 # Creating journal (8192 blocks): done52lb # Creating journal (8192 blocks): done53client # Creating journal (8192 blocks): done54worker1 # Writing superblocks and filesystem accounting information: 0/8 done55lb # Writing superblocks and filesystem accounting information: 0/8 done56client # Writing superblocks and filesystem accounting information: 0/8 done57worker1 # 58lb # 59client # 60worker1 # Virtualisation disk image created.61lb # Virtualisation disk image created.62client # Virtualisation disk image created.63worker1 # Starting virtiofs daemons...64lb # Starting virtiofs daemons...65client # Starting virtiofs daemons...66worker1 # [2026-09-21T14:29:22Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)67lb # [2026-09-21T14:29:22Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)68client # [2026-09-21T14:29:22Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)69worker1 # [2026-09-21T14:29:22Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether70lb # [2026-09-21T14:29:22Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether71client # [2026-09-21T14:29:22Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether72worker1 # [2026-09-21T14:29:22Z INFO virtiofsd] Waiting for vhost-user socket connection...73lb # [2026-09-21T14:29:22Z INFO virtiofsd] Waiting for vhost-user socket connection...74client # [2026-09-21T14:29:22Z INFO virtiofsd] Waiting for vhost-user socket connection...75worker1 # [2026-09-21T14:29:22Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)76lb # [2026-09-21T14:29:22Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)77client # [2026-09-21T14:29:22Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)78worker1 # [2026-09-21T14:29:22Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether79lb # [2026-09-21T14:29:22Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether80client # [2026-09-21T14:29:22Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether81worker1 # [2026-09-21T14:29:22Z INFO virtiofsd] Waiting for vhost-user socket connection...82lb # [2026-09-21T14:29:22Z INFO virtiofsd] Waiting for vhost-user socket connection...83client # [2026-09-21T14:29:22Z INFO virtiofsd] Waiting for vhost-user socket connection...84worker1 # [2026-09-21T14:29:22Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)85lb # [2026-09-21T14:29:22Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)86client # [2026-09-21T14:29:22Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)87worker1 # [2026-09-21T14:29:22Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether88lb # [2026-09-21T14:29:22Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether89client # [2026-09-21T14:29:22Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether90worker1 # [2026-09-21T14:29:22Z INFO virtiofsd] Waiting for vhost-user socket connection...91lb # [2026-09-21T14:29:22Z INFO virtiofsd] Waiting for vhost-user socket connection...92client # [2026-09-21T14:29:22Z INFO virtiofsd] Waiting for vhost-user socket connection...93worker1 # [2026-09-21T14:29:22Z INFO virtiofsd] Client connected, servicing requests94lb # [2026-09-21T14:29:22Z INFO virtiofsd] Client connected, servicing requests95client # [2026-09-21T14:29:22Z INFO virtiofsd] Client connected, servicing requests96worker1 # [2026-09-21T14:29:22Z INFO virtiofsd] Client connected, servicing requests97lb # [2026-09-21T14:29:22Z INFO virtiofsd] Client connected, servicing requests98client # [2026-09-21T14:29:22Z INFO virtiofsd] Client connected, servicing requests99worker1 # [2026-09-21T14:29:22Z INFO virtiofsd] Client connected, servicing requests100lb # [2026-09-21T14:29:22Z INFO virtiofsd] Client connected, servicing requests101client # [2026-09-21T14:29:22Z INFO virtiofsd] Client connected, servicing requests102(finished: start all VMs, in 0.45 seconds)103worker2 # mke2fs 1.47.4 (6-Mar-2025)104client: waiting for unit network-addresses-eth1.service105worker2 # Discarding device blocks: 0/262144 done106client: waiting for the VM to finish booting107worker2 # Creating filesystem with 262144 4k blocks and 65536 inodes108worker2 # Filesystem UUID: 956997b5-d2f1-4d3f-950a-1ef97894be22109worker2 # Superblock backups stored on blocks:110worker2 # 32768, 98304, 163840, 229376111worker2 # 112worker2 # Allocating group tables: 0/8 done113worker2 # Writing inode tables: 0/8 done114worker2 # Creating journal (8192 blocks): done115worker2 # Writing superblocks and filesystem accounting information: 0/8 done116worker2 # 117worker2 # Virtualisation disk image created.118worker2 # Starting virtiofs daemons...119worker2 # [2026-09-21T14:29:22Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)120worker2 # [2026-09-21T14:29:22Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether121worker2 # [2026-09-21T14:29:22Z INFO virtiofsd] Waiting for vhost-user socket connection...122worker2 # [2026-09-21T14:29:22Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)123worker2 # [2026-09-21T14:29:22Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether124worker2 # [2026-09-21T14:29:22Z INFO virtiofsd] Waiting for vhost-user socket connection...125worker2 # [2026-09-21T14:29:22Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)126worker2 # [2026-09-21T14:29:22Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether127worker2 # [2026-09-21T14:29:22Z INFO virtiofsd] Waiting for vhost-user socket connection...128worker2 # [2026-09-21T14:29:22Z INFO virtiofsd] Client connected, servicing requests129worker2 # [2026-09-21T14:29:22Z INFO virtiofsd] Client connected, servicing requests130worker2 # [2026-09-21T14:29:22Z INFO virtiofsd] Client connected, servicing requests131client # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40]132client # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 14 11:36:19 UTC 2026133client # [ 0.000000] KASLR enabled134worker1 # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40]135client # [ 0.000000] random: crng init done136client # [ 0.000000] Machine model: linux,dummy-virt137client # [ 0.000000] efi: UEFI not found.138lb # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40]139worker1 # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 14 11:36:19 UTC 2026140worker1 # [ 0.000000] KASLR enabled141client # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT142worker1 # [ 0.000000] random: crng init done143worker1 # [ 0.000000] Machine model: linux,dummy-virt144lb # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 14 11:36:19 UTC 2026145client # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff]146lb # [ 0.000000] KASLR enabled147worker1 # [ 0.000000] efi: UEFI not found.148lb # [ 0.000000] random: crng init done149client # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f]150client # [ 0.000000] Zone ranges:151worker1 # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT152lb # [ 0.000000] Machine model: linux,dummy-virt153lb # [ 0.000000] efi: UEFI not found.154client # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff]155worker1 # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff]156client # [ 0.000000] DMA32 empty157lb # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT158client # [ 0.000000] Normal empty159worker1 # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f]160client # [ 0.000000] Device empty161worker1 # [ 0.000000] Zone ranges:162lb # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff]163client # [ 0.000000] Movable zone start for each node164worker1 # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff]165worker2 # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40]166client # [ 0.000000] Early memory node ranges167lb # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f]168worker1 # [ 0.000000] DMA32 empty169lb # [ 0.000000] Zone ranges:170worker1 # [ 0.000000] Normal empty171client # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff]172worker1 # [ 0.000000] Device empty173lb # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff]174worker2 # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 14 11:36:19 UTC 2026175lb # [ 0.000000] DMA32 empty176client # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff]177worker1 # [ 0.000000] Movable zone start for each node178lb # [ 0.000000] Normal empty179worker1 # [ 0.000000] Early memory node ranges180worker2 # [ 0.000000] KASLR enabled181lb # [ 0.000000] Device empty182worker2 # [ 0.000000] random: crng init done183client # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000184lb # [ 0.000000] Movable zone start for each node185worker2 # [ 0.000000] Machine model: linux,dummy-virt186worker1 # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff]187client # [ 0.000000] psci: probing for conduit method from DT.188worker2 # [ 0.000000] efi: UEFI not found.189lb # [ 0.000000] Early memory node ranges190client # [ 0.000000] psci: PSCIv1.3 detected in firmware.191worker1 # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff]192lb # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff]193worker2 # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT194worker1 # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000195client # [ 0.000000] psci: Using standard PSCI v0.2 function IDs196client # [ 0.000000] psci: Trusted OS migration not required197worker1 # [ 0.000000] psci: probing for conduit method from DT.198worker2 # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff]199lb # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff]200client # [ 0.000000] psci: SMC Calling Convention v1.1201worker1 # [ 0.000000] psci: PSCIv1.3 detected in firmware.202lb # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000203worker2 # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f]204worker2 # [ 0.000000] Zone ranges:205lb # [ 0.000000] psci: probing for conduit method from DT.206worker1 # [ 0.000000] psci: Using standard PSCI v0.2 function IDs207client # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003)208worker2 # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff]209lb # [ 0.000000] psci: PSCIv1.3 detected in firmware.210worker1 # [ 0.000000] psci: Trusted OS migration not required211worker2 # [ 0.000000] DMA32 empty212client # [ 0.000000] percpu: Embedded 76 pages/cpu s186712 r8192 d116392 u311296213lb # [ 0.000000] psci: Using standard PSCI v0.2 function IDs214worker1 # [ 0.000000] psci: SMC Calling Convention v1.1215worker2 # [ 0.000000] Normal empty216client # [ 0.000000] Detected PIPT I-cache on CPU0217worker2 # [ 0.000000] Device empty218lb # [ 0.000000] psci: Trusted OS migration not required219worker1 # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003)220lb # [ 0.000000] psci: SMC Calling Convention v1.1221worker2 # [ 0.000000] Movable zone start for each node222client # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm)223worker2 # [ 0.000000] Early memory node ranges224worker1 # [ 0.000000] percpu: Embedded 76 pages/cpu s186712 r8192 d116392 u311296225client # [ 0.000000] CPU features: detected: GICv3 CPU interface226lb # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003)227worker2 # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff]228worker1 # [ 0.000000] Detected PIPT I-cache on CPU0229client # [ 0.000000] CPU features: detected: Spectre-v4230lb # [ 0.000000] percpu: Embedded 76 pages/cpu s186712 r8192 d116392 u311296231worker2 # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff]232client # [ 0.000000] CPU features: detected: Spectre-BHB233worker1 # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm)234lb # [ 0.000000] Detected PIPT I-cache on CPU0235worker2 # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000236client # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38237worker1 # [ 0.000000] CPU features: detected: GICv3 CPU interface238worker2 # [ 0.000000] psci: probing for conduit method from DT.239worker1 # [ 0.000000] CPU features: detected: Spectre-v4240lb # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm)241client # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23242worker2 # [ 0.000000] psci: PSCIv1.3 detected in firmware.243worker1 # [ 0.000000] CPU features: detected: Spectre-BHB244lb # [ 0.000000] CPU features: detected: GICv3 CPU interface245client # [ 0.000000] alternatives: applying boot alternatives246worker2 # [ 0.000000] psci: Using standard PSCI v0.2 function IDs247lb # [ 0.000000] CPU features: detected: Spectre-v4248worker1 # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38249worker2 # [ 0.000000] psci: Trusted OS migration not required250lb # [ 0.000000] CPU features: detected: Spectre-BHB251worker2 # [ 0.000000] psci: SMC Calling Convention v1.1252worker1 # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23253lb # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38254worker1 # [ 0.000000] alternatives: applying boot alternatives255worker2 # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003)256lb # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23257lb # [ 0.000000] alternatives: applying boot alternatives258worker2 # [ 0.000000] percpu: Embedded 76 pages/cpu s186712 r8192 d116392 u311296259worker2 # [ 0.000000] Detected PIPT I-cache on CPU0260client # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/h4vx3h280swjvvjn29az46l9rzjm19b0-nixos-system-client-test/init regInfo=/nix/store/6ig250j7wmxv991qlm8mbgf1azqxg23b-closure-info/registration console=ttyAMA0,115200n8 console=tty0261worker2 # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm)262worker2 # [ 0.000000] CPU features: detected: GICv3 CPU interface263worker1 # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/7p3yw1z65xci37mflkzx15bhrw3vfh31-nixos-system-worker1-test/init regInfo=/nix/store/jg2d42h74ikrc0ciimlw3jcfbq7lz74b-closure-info/registration console=ttyAMA0,115200n8 console=tty0264worker2 # [ 0.000000] CPU features: detected: Spectre-v4265client # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/6ig250j7wmxv991qlm8mbgf1azqxg23b-closure-info/registration", will be passed to user space.266worker2 # [ 0.000000] CPU features: detected: Spectre-BHB267lb # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/gcbvv54lc1mkgavjm2x1hn2y3wpbcps9-nixos-system-lb-test/init regInfo=/nix/store/zdpr9k12xnmsfn00vay1jzchyfdyhwwc-closure-info/registration console=ttyAMA0,115200n8 console=tty0268client # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes269worker2 # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38270worker2 # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23271client # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)272worker1 # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/jg2d42h74ikrc0ciimlw3jcfbq7lz74b-closure-info/registration", will be passed to user space.273lb # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/zdpr9k12xnmsfn00vay1jzchyfdyhwwc-closure-info/registration", will be passed to user space.274worker2 # [ 0.000000] alternatives: applying boot alternatives275worker1 # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes276client # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)277lb # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes278client # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB279worker1 # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)280client # [ 0.000000] software IO TLB: area num 1.281lb # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)282worker1 # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)283client # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB284lb # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)285worker1 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB286client # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB)287lb # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB288worker1 # [ 0.000000] software IO TLB: area num 1.289worker2 # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/w9nmysfyz54wqzsa6x95wb1nn52nrgzb-nixos-system-worker2-test/init regInfo=/nix/store/6gr9sfvasldfgapaymqd64qfipiw4h77-closure-info/registration console=ttyAMA0,115200n8 console=tty0290lb # [ 0.000000] software IO TLB: area num 1.291client # [ 0.000000] Fallback order for Node 0: 0292worker1 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB293client # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216294lb # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB295client # [ 0.000000] Policy zone: DMA296worker1 # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB)297worker2 # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/6gr9sfvasldfgapaymqd64qfipiw4h77-closure-info/registration", will be passed to user space.298lb # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB)299worker1 # [ 0.000000] Fallback order for Node 0: 0300client # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off301lb # [ 0.000000] Fallback order for Node 0: 0302worker2 # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes303worker1 # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216304client # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1305lb # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216306worker1 # [ 0.000000] Policy zone: DMA307client # [ 0.000000] allocated 3145728 bytes of page_ext308worker2 # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)309worker1 # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off310client # [ 0.000000] ftrace: allocating 74950 entries in 294 pages311lb # [ 0.000000] Policy zone: DMA312worker2 # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)313client # [ 0.000000] ftrace: allocated 294 pages with 4 groups314worker1 # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1315lb # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off316client # [ 0.000000] rcu: Hierarchical RCU implementation.317worker1 # [ 0.000000] allocated 3145728 bytes of page_ext318lb # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1319client # [ 0.000000] rcu: RCU event tracing is enabled.320worker1 # [ 0.000000] ftrace: allocating 74950 entries in 294 pages321worker2 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB322lb # [ 0.000000] allocated 3145728 bytes of page_ext323worker2 # [ 0.000000] software IO TLB: area num 1.324worker1 # [ 0.000000] ftrace: allocated 294 pages with 4 groups325client # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.326lb # [ 0.000000] ftrace: allocating 74950 entries in 294 pages327worker1 # [ 0.000000] rcu: Hierarchical RCU implementation.328worker2 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB329client # [ 0.000000] Trampoline variant of Tasks RCU enabled.330lb # [ 0.000000] ftrace: allocated 294 pages with 4 groups331worker1 # [ 0.000000] rcu: RCU event tracing is enabled.332client # [ 0.000000] Rude variant of Tasks RCU enabled.333worker2 # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB)334lb # [ 0.000000] rcu: Hierarchical RCU implementation.335client # [ 0.000000] Tracing variant of Tasks RCU enabled.336worker1 # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.337worker2 # [ 0.000000] Fallback order for Node 0: 0338lb # [ 0.000000] rcu: RCU event tracing is enabled.339worker1 # [ 0.000000] Trampoline variant of Tasks RCU enabled.340client # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.341worker2 # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216342lb # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.343worker1 # [ 0.000000] Rude variant of Tasks RCU enabled.344worker2 # [ 0.000000] Policy zone: DMA345lb # [ 0.000000] Trampoline variant of Tasks RCU enabled.346client # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1347worker1 # [ 0.000000] Tracing variant of Tasks RCU enabled.348worker2 # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off349lb # [ 0.000000] Rude variant of Tasks RCU enabled.350worker1 # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.351client # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.352lb # [ 0.000000] Tracing variant of Tasks RCU enabled.353worker2 # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1354worker2 # [ 0.000000] allocated 3145728 bytes of page_ext355worker1 # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1356lb # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.357client # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.358worker2 # [ 0.000000] ftrace: allocating 74950 entries in 294 pages359lb # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1360worker1 # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.361worker2 # [ 0.000000] ftrace: allocated 294 pages with 4 groups362client # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.363worker2 # [ 0.000000] rcu: Hierarchical RCU implementation.364lb # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.365client # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0366worker1 # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.367worker2 # [ 0.000000] rcu: RCU event tracing is enabled.368client # [ 0.000000] GICv3: 256 SPIs implemented369lb # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.370client # [ 0.000000] GICv3: 0 Extended SPIs implemented371worker2 # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.372worker1 # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.373client # [ 0.000000] Root IRQ handler: gic_handle_irq374worker2 # [ 0.000000] Trampoline variant of Tasks RCU enabled.375worker1 # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0376lb # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.377worker2 # [ 0.000000] Rude variant of Tasks RCU enabled.378client # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI379worker1 # [ 0.000000] GICv3: 256 SPIs implemented380client # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0381worker2 # [ 0.000000] Tracing variant of Tasks RCU enabled.382worker1 # [ 0.000000] GICv3: 0 Extended SPIs implemented383lb # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0384lb # [ 0.000000] GICv3: 256 SPIs implemented385worker1 # [ 0.000000] Root IRQ handler: gic_handle_irq386client # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000387worker2 # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.388lb # [ 0.000000] GICv3: 0 Extended SPIs implemented389worker1 # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI390client # [ 0.000000] ITS [mem 0x08080000-0x0809ffff]391lb # [ 0.000000] Root IRQ handler: gic_handle_irq392worker1 # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0393worker2 # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1394lb # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI395client # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1)396worker1 # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000397lb # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0398worker2 # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.399worker1 # [ 0.000000] ITS [mem 0x08080000-0x0809ffff]400lb # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000401client # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1)402worker2 # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.403lb # [ 0.000000] ITS [mem 0x08080000-0x0809ffff]404worker1 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1)405client # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000406worker2 # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.407lb # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1)408client # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000409worker1 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1)410worker2 # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0411worker2 # [ 0.000000] GICv3: 256 SPIs implemented412client # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.413worker1 # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000414lb # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1)415worker2 # [ 0.000000] GICv3: 0 Extended SPIs implemented416worker2 # [ 0.000000] Root IRQ handler: gic_handle_irq417worker1 # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000418lb # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000419client # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns420worker2 # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI421worker1 # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.422lb # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000423client # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt).424worker2 # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0425lb # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.426worker2 # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000427worker1 # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns428client # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns429worker2 # [ 0.000000] ITS [mem 0x08080000-0x0809ffff]430worker1 # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt).431lb # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns432client # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns433lb # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt).434worker2 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1)435client # [ 0.000030] arm-pv: using stolen time PV436worker1 # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns437worker2 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1)438lb # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns439client # [ 0.000402] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)440worker1 # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns441client # [ 0.000578] Console: colour dummy device 80x25442worker1 # [ 0.000031] arm-pv: using stolen time PV443worker2 # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000444lb # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns445client # [ 0.000585] printk: legacy console [tty0] enabled446worker2 # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000447lb # [ 0.000042] arm-pv: using stolen time PV448worker1 # [ 0.000406] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)449worker2 # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.450worker1 # [ 0.000574] Console: colour dummy device 80x25451client # [ 0.000773] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000)452lb # [ 0.000394] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)453worker1 # [ 0.000581] printk: legacy console [tty0] enabled454client # [ 0.000779] pid_max: default: 32768 minimum: 301455lb # [ 0.000580] Console: colour dummy device 80x25456worker2 # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns457client # [ 0.000852] LSM: initializing lsm=capability,landlock,yama,bpf,ima458lb # [ 0.000587] printk: legacy console [tty0] enabled459client # [ 0.000991] landlock: Up and running.460worker1 # [ 0.000766] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000)461worker2 # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt).462client # [ 0.000994] Yama: becoming mindful.463worker1 # [ 0.000773] pid_max: default: 32768 minimum: 301464lb # [ 0.000770] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000)465client # [ 0.001450] LSM support for eBPF active466worker1 # [ 0.000848] LSM: initializing lsm=capability,landlock,yama,bpf,ima467worker2 # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns468lb # [ 0.000777] pid_max: default: 32768 minimum: 301469worker1 # [ 0.000985] landlock: Up and running.470client # [ 0.001598] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)471worker1 # [ 0.000988] Yama: becoming mindful.472lb # [ 0.000855] LSM: initializing lsm=capability,landlock,yama,bpf,ima473worker2 # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns474lb # [ 0.000969] landlock: Up and running.475worker1 # [ 0.001444] LSM support for eBPF active476client # [ 0.001631] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)477worker2 # [ 0.000031] arm-pv: using stolen time PV478lb # [ 0.000972] Yama: becoming mindful.479client # [ 0.003488] rcu: Hierarchical SRCU implementation.480lb # [ 0.001389] LSM support for eBPF active481worker1 # [ 0.001613] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)482worker2 # [ 0.000388] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)483client # [ 0.003493] rcu: Max phase no-delay instances is 1000.484lb # [ 0.001541] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)485worker1 # [ 0.001647] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)486worker2 # [ 0.000542] Console: colour dummy device 80x25487client # [ 0.004748] fsl-mc MSI: its@8080000 domain created488worker2 # [ 0.000549] printk: legacy console [tty0] enabled489worker1 # [ 0.003478] rcu: Hierarchical SRCU implementation.490client # [ 0.004834] EFI services will not be available.491lb # [ 0.001574] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)492worker1 # [ 0.003483] rcu: Max phase no-delay instances is 1000.493client # [ 0.004900] smp: Bringing up secondary CPUs ...494lb # [ 0.003378] rcu: Hierarchical SRCU implementation.495client # [ 0.004909] smp: Brought up 1 node, 1 CPU496worker2 # [ 0.000748] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000)497worker1 # [ 0.004750] fsl-mc MSI: its@8080000 domain created498lb # [ 0.003383] rcu: Max phase no-delay instances is 1000.499client # [ 0.004912] SMP: Total of 1 processors activated.500worker1 # [ 0.004837] EFI services will not be available.501worker2 # [ 0.000754] pid_max: default: 32768 minimum: 301502lb # [ 0.004667] fsl-mc MSI: its@8080000 domain created503client # [ 0.004917] CPU: All CPU(s) started at EL1504worker1 # [ 0.004949] smp: Bringing up secondary CPUs ...505worker2 # [ 0.000852] LSM: initializing lsm=capability,landlock,yama,bpf,ima506lb # [ 0.004766] EFI services will not be available.507worker1 # [ 0.004958] smp: Brought up 1 node, 1 CPU508client # [ 0.004927] CPU features: detected: Branch Target Identification509worker2 # [ 0.001019] landlock: Up and running.510lb # [ 0.004845] smp: Bringing up secondary CPUs ...511worker1 # [ 0.004961] SMP: Total of 1 processors activated.512worker2 # [ 0.001022] Yama: becoming mindful.513client # [ 0.004931] CPU features: detected: ARMv8.4 Translation Table Level514lb # [ 0.004853] smp: Brought up 1 node, 1 CPU515worker2 # [ 0.001457] LSM support for eBPF active516worker1 # [ 0.004966] CPU: All CPU(s) started at EL1517lb # [ 0.004856] SMP: Total of 1 processors activated.518worker1 # [ 0.004976] CPU features: detected: Branch Target Identification519lb # [ 0.004861] CPU: All CPU(s) started at EL1520client # [ 0.004934] CPU features: detected: Instruction cache invalidation not required for I/D coherence521worker2 # [ 0.001594] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)522lb # [ 0.004871] CPU features: detected: Branch Target Identification523worker1 # [ 0.004980] CPU features: detected: ARMv8.4 Translation Table Level524worker2 # [ 0.001633] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)525client # [ 0.004938] CPU features: detected: Data cache clean to the PoU not required for I/D coherence526lb # [ 0.004876] CPU features: detected: ARMv8.4 Translation Table Level527worker2 # [ 0.003486] rcu: Hierarchical SRCU implementation.528client # [ 0.004941] CPU features: detected: Common not Private translations529worker1 # [ 0.004984] CPU features: detected: Instruction cache invalidation not required for I/D coherence530worker2 # [ 0.003491] rcu: Max phase no-delay instances is 1000.531client # [ 0.004945] CPU features: detected: CRC32 instructions532worker1 # [ 0.004987] CPU features: detected: Data cache clean to the PoU not required for I/D coherence533lb # [ 0.004879] CPU features: detected: Instruction cache invalidation not required for I/D coherence534worker2 # [ 0.004763] fsl-mc MSI: its@8080000 domain created535worker2 # [ 0.004855] EFI services will not be available.536client # [ 0.004947] CPU features: detected: Data cache clean to Point of Deep Persistence537worker1 # [ 0.004991] CPU features: detected: Common not Private translations538lb # [ 0.004883] CPU features: detected: Data cache clean to the PoU not required for I/D coherence539worker2 # [ 0.004938] smp: Bringing up secondary CPUs ...540worker1 # [ 0.004994] CPU features: detected: CRC32 instructions541client # [ 0.004951] CPU features: detected: Data cache clean to Point of Persistence542worker2 # [ 0.004946] smp: Brought up 1 node, 1 CPU543lb # [ 0.004887] CPU features: detected: Common not Private translations544lb # [ 0.004890] CPU features: detected: CRC32 instructions545client # [ 0.004954] CPU features: detected: Data independent timing control (DIT)546worker2 # [ 0.004949] SMP: Total of 1 processors activated.547worker1 # [ 0.004997] CPU features: detected: Data cache clean to Point of Deep Persistence548worker2 # [ 0.004955] CPU: All CPU(s) started at EL1549lb # [ 0.004893] CPU features: detected: Data cache clean to Point of Deep Persistence550worker1 # [ 0.005000] CPU features: detected: Data cache clean to Point of Persistence551worker2 # [ 0.004965] CPU features: detected: Branch Target Identification552lb # [ 0.004897] CPU features: detected: Data cache clean to Point of Persistence553worker1 # [ 0.005004] CPU features: detected: Data independent timing control (DIT)554worker2 # [ 0.004969] CPU features: detected: ARMv8.4 Translation Table Level555worker1 # [ 0.005007] CPU features: detected: E0PD556lb # [ 0.004900] CPU features: detected: Data independent timing control (DIT)557lb # [ 0.004903] CPU features: detected: E0PD558worker2 # [ 0.004972] CPU features: detected: Instruction cache invalidation not required for I/D coherence559worker1 # [ 0.005009] CPU features: detected: Enhanced Counter Virtualization560lb # [ 0.004906] CPU features: detected: Enhanced Counter Virtualization561worker1 # [ 0.005012] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF)562worker2 # [ 0.004976] CPU features: detected: Data cache clean to the PoU not required for I/D coherence563lb # [ 0.004909] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF)564worker1 # [ 0.005016] CPU features: detected: Enhanced Virtualization Traps565worker2 # [ 0.004980] CPU features: detected: Common not Private translations566lb # [ 0.004912] CPU features: detected: Enhanced Virtualization Traps567worker1 # [ 0.005019] CPU features: detected: Fine Grained Traps568worker2 # [ 0.004983] CPU features: detected: CRC32 instructions569lb # [ 0.004915] CPU features: detected: Fine Grained Traps570worker1 # [ 0.005022] CPU features: detected: Generic authentication (architected QARMA5 algorithm)571worker2 # [ 0.004986] CPU features: detected: Data cache clean to Point of Deep Persistence572lb # [ 0.004919] CPU features: detected: Generic authentication (architected QARMA5 algorithm)573worker1 # [ 0.005027] CPU features: detected: RCpc load-acquire (LDAPR)574worker2 # [ 0.004990] CPU features: detected: Data cache clean to Point of Persistence575lb # [ 0.004924] CPU features: detected: RCpc load-acquire (LDAPR)576worker1 # [ 0.005029] CPU features: detected: LSE atomic instructions577worker2 # [ 0.004992] CPU features: detected: Data independent timing control (DIT)578lb # [ 0.004927] CPU features: detected: LSE atomic instructions579worker1 # [ 0.005032] CPU features: detected: Privileged Access Never580worker2 # [ 0.004996] CPU features: detected: E0PD581lb # [ 0.004929] CPU features: detected: Privileged Access Never582worker1 # [ 0.005035] CPU features: detected: PMUv3583client # [ 0.004957] CPU features: detected: E0PD584worker2 # [ 0.004998] CPU features: detected: Enhanced Counter Virtualization585worker1 # [ 0.005038] CPU features: detected: RAS Extension Support586lb # [ 0.004932] CPU features: detected: PMUv3587client # [ 0.004960] CPU features: detected: Enhanced Counter Virtualization588lb # [ 0.004935] CPU features: detected: RAS Extension Support589worker2 # [ 0.005001] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF)590worker1 # [ 0.005040] CPU features: detected: RASv1p1 Extension Support591client # [ 0.004963] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF)592lb # [ 0.004938] CPU features: detected: RASv1p1 Extension Support593worker2 # [ 0.005005] CPU features: detected: Enhanced Virtualization Traps594worker1 # [ 0.005043] CPU features: detected: Random Number Generator595client # [ 0.004967] CPU features: detected: Enhanced Virtualization Traps596lb # [ 0.004940] CPU features: detected: Random Number Generator597worker2 # [ 0.005008] CPU features: detected: Fine Grained Traps598worker1 # [ 0.005046] CPU features: detected: Speculation barrier (SB)599client # [ 0.004969] CPU features: detected: Fine Grained Traps600lb # [ 0.004943] CPU features: detected: Speculation barrier (SB)601worker1 # [ 0.005048] CPU features: detected: Stage-2 Force Write-Back602worker2 # [ 0.005012] CPU features: detected: Generic authentication (architected QARMA5 algorithm)603client # [ 0.004972] CPU features: detected: Generic authentication (architected QARMA5 algorithm)604lb # [ 0.004946] CPU features: detected: Stage-2 Force Write-Back605worker1 # [ 0.005051] CPU features: detected: TLB range maintenance instructions606worker2 # [ 0.005017] CPU features: detected: RCpc load-acquire (LDAPR)607client # [ 0.004977] CPU features: detected: RCpc load-acquire (LDAPR)608lb # [ 0.004949] CPU features: detected: TLB range maintenance instructions609worker2 # [ 0.005020] CPU features: detected: LSE atomic instructions610worker1 # [ 0.005056] CPU features: detected: Speculative Store Bypassing Safe (SSBS)611client # [ 0.004980] CPU features: detected: LSE atomic instructions612lb # [ 0.004954] CPU features: detected: Speculative Store Bypassing Safe (SSBS)613worker2 # [ 0.005023] CPU features: detected: Privileged Access Never614worker1 # [ 0.005093] alternatives: applying system-wide alternatives615client # [ 0.004982] CPU features: detected: Privileged Access Never616worker2 # [ 0.005026] CPU features: detected: PMUv3617lb # [ 0.004991] alternatives: applying system-wide alternatives618client # [ 0.004985] CPU features: detected: PMUv3619worker1 # [ 0.008132] CPU features: detected: BBM Level 2 without TLB conflict abort620worker2 # [ 0.005029] CPU features: detected: RAS Extension Support621lb # [ 0.007922] CPU features: detected: BBM Level 2 without TLB conflict abort622client # [ 0.004988] CPU features: detected: RAS Extension Support623worker2 # [ 0.005032] CPU features: detected: RASv1p1 Extension Support624client # [ 0.004991] CPU features: detected: RASv1p1 Extension Support625worker2 # [ 0.005034] CPU features: detected: Random Number Generator626worker1 # [ 0.008308] Memory: 1405652K/1572864K available (24448K kernel code, 7094K rwdata, 26372K rodata, 4736K init, 1107K bss, 124664K reserved, 32768K cma-reserved)627client # [ 0.004993] CPU features: detected: Random Number Generator628lb # [ 0.008161] Memory: 1405660K/1572864K available (24448K kernel code, 7094K rwdata, 26372K rodata, 4736K init, 1107K bss, 124636K reserved, 32768K cma-reserved)629worker1 # [ 0.008719] devtmpfs: initialized630worker2 # [ 0.005037] CPU features: detected: Speculation barrier (SB)631lb # [ 0.008531] devtmpfs: initialized632client # [ 0.004996] CPU features: detected: Speculation barrier (SB)633worker2 # [ 0.005040] CPU features: detected: Stage-2 Force Write-Back634worker1 # [ 0.010538] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)635client # [ 0.004998] CPU features: detected: Stage-2 Force Write-Back636lb # [ 0.010375] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)637worker2 # [ 0.005042] CPU features: detected: TLB range maintenance instructions638worker1 # [ 0.010562] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).639client # [ 0.005001] CPU features: detected: TLB range maintenance instructions640lb # [ 0.010397] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).641worker2 # [ 0.005047] CPU features: detected: Speculative Store Bypassing Safe (SSBS)642worker1 # [ 0.010751] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL643client # [ 0.005005] CPU features: detected: Speculative Store Bypassing Safe (SSBS)644lb # [ 0.010603] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL645client # [ 0.005041] alternatives: applying system-wide alternatives646lb # [ 0.010608] 0 pages in range for non-PLT usage647client # [ 0.008008] CPU features: detected: BBM Level 2 without TLB conflict abort648lb # [ 0.010609] 508272 pages in range for PLT usage649lb # [ 0.010719] pinctrl core: initialized pinctrl subsystem650lb # [ 0.011521] DMI not present or invalid.651client # [ 0.008258] Memory: 1405628K/1572864K available (24448K kernel code, 7094K rwdata, 26372K rodata, 4736K init, 1107K bss, 124660K reserved, 32768K cma-reserved)652lb # [ 0.014804] NET: Registered PF_NETLINK/PF_ROUTE protocol family653client # [ 0.008642] devtmpfs: initialized654lb # [ 0.016990] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations655client # [ 0.010442] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)656lb # [ 0.017179] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations657client # [ 0.010464] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).658lb # [ 0.017377] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations659client # [ 0.010645] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL660lb # [ 0.017401] audit: initializing netlink subsys (disabled)661client # [ 0.010649] 0 pages in range for non-PLT usage662client # [ 0.010650] 508272 pages in range for PLT usage663lb # [ 0.017948] thermal_sys: Registered thermal governor 'fair_share'664client # [ 0.010759] pinctrl core: initialized pinctrl subsystem665lb # [ 0.017950] thermal_sys: Registered thermal governor 'bang_bang'666worker2 # [ 0.005084] alternatives: applying system-wide alternatives667worker1 # [ 0.010756] 0 pages in range for non-PLT usage668client # [ 0.011564] DMI not present or invalid.669lb # [ 0.017954] thermal_sys: Registered thermal governor 'step_wise'670worker1 # [ 0.010757] 508272 pages in range for PLT usage671worker2 # [ 0.008107] CPU features: detected: BBM Level 2 without TLB conflict abort672client # [ 0.014768] NET: Registered PF_NETLINK/PF_ROUTE protocol family673worker1 # [ 0.010853] pinctrl core: initialized pinctrl subsystem674client # [ 0.017055] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations675worker1 # [ 0.011619] DMI not present or invalid.676worker2 # [ 0.008262] Memory: 1405716K/1572864K available (24448K kernel code, 7094K rwdata, 26372K rodata, 4736K init, 1107K bss, 124632K reserved, 32768K cma-reserved)677worker1 # [ 0.014931] NET: Registered PF_NETLINK/PF_ROUTE protocol family678client # [ 0.017233] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations679worker2 # [ 0.008620] devtmpfs: initialized680worker1 # [ 0.017151] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations681client # [ 0.017427] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations682worker2 # [ 0.010421] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)683client # [ 0.017448] audit: initializing netlink subsys (disabled)684worker1 # [ 0.017337] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations685worker2 # [ 0.010444] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).686client # [ 0.018083] thermal_sys: Registered thermal governor 'fair_share'687worker1 # [ 0.017546] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations688worker2 # [ 0.010624] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL689client # [ 0.018085] thermal_sys: Registered thermal governor 'bang_bang'690worker1 # [ 0.017567] audit: initializing netlink subsys (disabled)691worker2 # [ 0.010628] 0 pages in range for non-PLT usage692client # [ 0.018089] thermal_sys: Registered thermal governor 'step_wise'693worker2 # [ 0.010629] 508272 pages in range for PLT usage694worker1 # [ 0.018123] thermal_sys: Registered thermal governor 'fair_share'695worker2 # [ 0.010741] pinctrl core: initialized pinctrl subsystem696client # [ 0.018092] thermal_sys: Registered thermal governor 'user_space'697worker1 # [ 0.018125] thermal_sys: Registered thermal governor 'bang_bang'698worker2 # [ 0.011479] DMI not present or invalid.699client # [ 0.018097] thermal_sys: Registered thermal governor 'power_allocator'700worker1 # [ 0.018129] thermal_sys: Registered thermal governor 'step_wise'701worker2 # [ 0.014466] NET: Registered PF_NETLINK/PF_ROUTE protocol family702worker1 # [ 0.018131] thermal_sys: Registered thermal governor 'user_space'703client # [ 0.018122] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1704worker2 # [ 0.016690] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations705client # [ 0.018131] cpuidle: using governor ladder706worker1 # [ 0.018137] thermal_sys: Registered thermal governor 'power_allocator'707lb # [ 0.017957] thermal_sys: Registered thermal governor 'user_space'708client # [ 0.018137] cpuidle: using governor menu709worker2 # [ 0.016856] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations710lb # [ 0.017962] thermal_sys: Registered thermal governor 'power_allocator'711worker1 # [ 0.018161] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1712client # [ 0.018323] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers.713worker2 # [ 0.017052] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations714worker1 # [ 0.018169] cpuidle: using governor ladder715lb # [ 0.017996] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1716client # [ 0.018339] ASID allocator initialised with 65536 entries717worker1 # [ 0.018175] cpuidle: using governor menu718worker2 # [ 0.017072] audit: initializing netlink subsys (disabled)719lb # [ 0.018006] cpuidle: using governor ladder720client # [ 0.019577] Serial: AMBA PL011 UART driver721worker1 # [ 0.018353] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers.722worker2 # [ 0.017646] thermal_sys: Registered thermal governor 'fair_share'723lb # [ 0.018012] cpuidle: using governor menu724client # [ 0.024820] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1725worker2 # [ 0.017648] thermal_sys: Registered thermal governor 'bang_bang'726worker1 # [ 0.018368] ASID allocator initialised with 65536 entries727lb # [ 0.018195] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers.728client # [ 0.024964] printk: console [ttyAMA0] enabled729worker1 # [ 0.019558] Serial: AMBA PL011 UART driver730lb # [ 0.018210] ASID allocator initialised with 65536 entries731worker2 # [ 0.017652] thermal_sys: Registered thermal governor 'step_wise'732lb # [ 0.019377] Serial: AMBA PL011 UART driver733worker2 # [ 0.017654] thermal_sys: Registered thermal governor 'user_space'734worker1 # [ 0.024707] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1735client # [ 0.144509] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages736worker1 # [ 0.024823] printk: console [ttyAMA0] enabled737lb # [ 0.024646] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1738worker2 # [ 0.017660] thermal_sys: Registered thermal governor 'power_allocator'739client # [ 0.144530] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page740lb # [ 0.024792] printk: console [ttyAMA0] enabled741worker2 # [ 0.017684] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1742client # [ 0.144536] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages743worker1 # [ 0.144039] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages744worker2 # [ 0.017693] cpuidle: using governor ladder745lb # [ 0.143116] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages746worker2 # [ 0.017699] cpuidle: using governor menu747client # [ 0.144540] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page748worker1 # [ 0.144059] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page749lb # [ 0.143135] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page750worker2 # [ 0.017894] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers.751client # [ 0.144544] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages752worker1 # [ 0.144065] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages753lb # [ 0.143140] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages754worker2 # [ 0.017909] ASID allocator initialised with 65536 entries755client # [ 0.144549] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page756worker1 # [ 0.144069] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page757worker2 # [ 0.019165] Serial: AMBA PL011 UART driver758lb # [ 0.143145] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page759worker1 # [ 0.144073] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages760client # [ 0.144553] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages761lb # [ 0.143149] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages762worker2 # [ 0.024272] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1763client # [ 0.144557] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page764worker1 # [ 0.144078] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page765worker2 # [ 0.024422] printk: console [ttyAMA0] enabled766lb # [ 0.143153] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page767worker1 # [ 0.144082] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages768lb # [ 0.143157] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages769worker1 # [ 0.144086] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page770worker2 # [ 0.142503] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages771lb # [ 0.143161] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page772client # [ 0.151833] fbcon: Taking over console773worker2 # [ 0.142522] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page774client # [ 0.151848] ACPI: Interpreter disabled.775worker2 # [ 0.142527] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages776worker1 # [ 0.151363] fbcon: Taking over console777worker2 # [ 0.142531] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page778worker1 # [ 0.151377] ACPI: Interpreter disabled.779lb # [ 0.150460] fbcon: Taking over console780worker2 # [ 0.142535] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages781lb # [ 0.150474] ACPI: Interpreter disabled.782worker1 # [ 0.153206] iommu: Default domain type: Translated783lb # [ 0.152346] iommu: Default domain type: Translated784worker2 # [ 0.142539] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page785worker1 # [ 0.153215] iommu: DMA domain TLB invalidation policy: strict mode786lb # [ 0.152356] iommu: DMA domain TLB invalidation policy: strict mode787worker2 # [ 0.142543] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages788worker1 # [ 0.154918] SCSI subsystem initialized789worker2 # [ 0.142548] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page790client # [ 0.158118] iommu: Default domain type: Translated791client # [ 0.158128] iommu: DMA domain TLB invalidation policy: strict mode792client # [ 0.159787] SCSI subsystem initialized793worker2 # [ 0.150079] fbcon: Taking over console794worker2 # [ 0.150094] ACPI: Interpreter disabled.795client # [ 0.160458] usbcore: registered new interface driver usbfs796client # [ 0.160490] usbcore: registered new interface driver hub797client # [ 0.160507] usbcore: registered new device driver usb798client # [ 0.160777] pps_core: LinuxPPS API ver. 1 registered799client # [ 0.160784] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>800lb # [ 0.160918] SCSI subsystem initialized801client # [ 0.160794] PTP clock support registered802client # [ 0.160851] EDAC MC: Ver: 3.0.0803worker1 # [ 0.162610] usbcore: registered new interface driver usbfs804worker2 # [ 0.156332] iommu: Default domain type: Translated805lb # [ 0.161610] usbcore: registered new interface driver usbfs806client # [ 0.165447] scmi_core: SCMI protocol bus registered807worker1 # [ 0.162636] usbcore: registered new interface driver hub808worker2 # [ 0.156341] iommu: DMA domain TLB invalidation policy: strict mode809lb # [ 0.161650] usbcore: registered new interface driver hub810worker1 # [ 0.162660] usbcore: registered new device driver usb811client # [ 0.166396] FPGA manager framework812lb # [ 0.161666] usbcore: registered new device driver usb813worker2 # [ 0.157959] SCSI subsystem initialized814worker1 # [ 0.162926] pps_core: LinuxPPS API ver. 1 registered815client # [ 0.167346] vgaarb: loaded816lb # [ 0.161971] pps_core: LinuxPPS API ver. 1 registered817worker1 # [ 0.162932] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>818client # [ 0.167984] clocksource: Switched to clocksource arch_sys_counter819worker2 # [ 0.158650] usbcore: registered new interface driver usbfs820lb # [ 0.161977] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>821worker1 # [ 0.162942] PTP clock support registered822worker2 # [ 0.158682] usbcore: registered new interface driver hub823lb # [ 0.161987] PTP clock support registered824worker1 # [ 0.162981] EDAC MC: Ver: 3.0.0825lb # [ 0.162028] EDAC MC: Ver: 3.0.0826worker2 # [ 0.158705] usbcore: registered new device driver usb827worker1 # [ 0.167505] scmi_core: SCMI protocol bus registered828worker2 # [ 0.159009] pps_core: LinuxPPS API ver. 1 registered829lb # [ 0.166509] scmi_core: SCMI protocol bus registered830worker1 # [ 0.168475] FPGA manager framework831worker2 # [ 0.159015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>832lb # [ 0.167443] FPGA manager framework833worker1 # [ 0.169420] vgaarb: loaded834worker2 # [ 0.159026] PTP clock support registered835lb # [ 0.168420] vgaarb: loaded836worker2 # [ 0.159075] EDAC MC: Ver: 3.0.0837worker2 # [ 0.163550] scmi_core: SCMI protocol bus registered838worker1 # [ 0.170029] clocksource: Switched to clocksource arch_sys_counter839lb # [ 0.169067] clocksource: Switched to clocksource arch_sys_counter840worker1 # [ 0.170632] VFS: Disk quotas dquot_6.6.0841worker2 # [ 0.164521] FPGA manager framework842lb # [ 0.169666] VFS: Disk quotas dquot_6.6.0843worker1 # [ 0.170663] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)844client # [ 0.174197] VFS: Disk quotas dquot_6.6.0845worker2 # [ 0.165461] vgaarb: loaded846lb # [ 0.169692] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)847worker1 # [ 0.172988] netfs: FS-Cache loaded848client # [ 0.174251] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)849worker2 # [ 0.166070] clocksource: Switched to clocksource arch_sys_counter850lb # [ 0.171960] netfs: FS-Cache loaded851worker1 # [ 0.173083] pnp: PnP ACPI: disabled852client # [ 0.175857] netfs: FS-Cache loaded853lb # [ 0.172090] pnp: PnP ACPI: disabled854client # [ 0.175974] pnp: PnP ACPI: disabled855client # [ 0.179817] NET: Registered PF_INET protocol family856worker2 # [ 0.170019] VFS: Disk quotas dquot_6.6.0857worker2 # [ 0.170053] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)858worker2 # [ 0.173659] netfs: FS-Cache loaded859worker2 # [ 0.173786] pnp: PnP ACPI: disabled860worker1 # [ 0.178976] NET: Registered PF_INET protocol family861client # [ 0.180689] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)862lb # [ 0.178216] NET: Registered PF_INET protocol family863worker1 # [ 0.179243] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)864lb # [ 0.178476] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)865worker2 # [ 0.177555] NET: Registered PF_INET protocol family866worker2 # [ 0.177810] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)867client # [ 0.209663] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)868lb # [ 0.207562] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)869client # [ 0.209710] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)870lb # [ 0.207614] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)871client # [ 0.209740] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)872lb # [ 0.207642] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)873client # [ 0.209814] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)874lb # [ 0.207719] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)875client # [ 0.209958] TCP: Hash tables configured (established 16384 bind 16384)876lb # [ 0.207863] TCP: Hash tables configured (established 16384 bind 16384)877worker1 # [ 0.208650] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)878client # [ 0.210044] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)879lb # [ 0.207951] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)880worker1 # [ 0.208701] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)881client # [ 0.210081] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)882lb # [ 0.207987] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)883worker1 # [ 0.208732] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)884client # [ 0.210116] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)885lb # [ 0.208034] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)886client # [ 0.210214] NET: Registered PF_UNIX/PF_LOCAL protocol family887worker1 # [ 0.208810] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)888worker2 # [ 0.208044] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)889lb # [ 0.208153] NET: Registered PF_UNIX/PF_LOCAL protocol family890client # [ 0.210235] NET: Registered PF_XDP protocol family891worker1 # [ 0.208954] TCP: Hash tables configured (established 16384 bind 16384)892lb # [ 0.208172] NET: Registered PF_XDP protocol family893client # [ 0.210253] PCI: CLS 0 bytes, default 64894worker2 # [ 0.208098] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)895lb # [ 0.208190] PCI: CLS 0 bytes, default 64896client # [ 0.210490] Trying to unpack rootfs image as initramfs...897worker1 # [ 0.209038] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)898lb # [ 0.208438] Trying to unpack rootfs image as initramfs...899worker2 # [ 0.208132] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)900worker1 # [ 0.209074] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)901worker2 # [ 0.208219] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)902worker1 # [ 0.209109] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)903worker2 # [ 0.208362] TCP: Hash tables configured (established 16384 bind 16384)904worker1 # [ 0.209201] NET: Registered PF_UNIX/PF_LOCAL protocol family905worker1 # [ 0.209221] NET: Registered PF_XDP protocol family906worker2 # [ 0.208449] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)907worker1 # [ 0.209240] PCI: CLS 0 bytes, default 64908worker2 # [ 0.208487] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)909worker1 # [ 0.209501] Trying to unpack rootfs image as initramfs...910worker2 # [ 0.208523] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)911worker2 # [ 0.208627] NET: Registered PF_UNIX/PF_LOCAL protocol family912worker2 # [ 0.208666] NET: Registered PF_XDP protocol family913worker2 # [ 0.208685] PCI: CLS 0 bytes, default 64914client # [ 0.225503] kvm [1]: HYP mode not available915worker2 # [ 0.208916] Trying to unpack rootfs image as initramfs...916lb # [ 0.222566] kvm [1]: HYP mode not available917worker1 # [ 0.227702] kvm [1]: HYP mode not available918worker2 # [ 0.223622] kvm [1]: HYP mode not available919client # [ 0.320582] Initialise system trusted keyrings920lb # [ 0.317754] Initialise system trusted keyrings921client # [ 0.321320] workingset: timestamp_bits=42 max_order=19 bucket_order=0922lb # [ 0.318497] workingset: timestamp_bits=42 max_order=19 bucket_order=0923client # [ 0.322615] squashfs: version 4.0 (2009/01/31) Phillip Lougher924lb # [ 0.319803] squashfs: version 4.0 (2009/01/31) Phillip Lougher925worker2 # [ 0.314741] Initialise system trusted keyrings926client # [ 0.323407] 9p: Installing v9fs 9p2000 file system support927lb # [ 0.320611] 9p: Installing v9fs 9p2000 file system support928worker1 # [ 0.322612] Initialise system trusted keyrings929worker2 # [ 0.315479] workingset: timestamp_bits=42 max_order=19 bucket_order=0930worker1 # [ 0.323341] workingset: timestamp_bits=42 max_order=19 bucket_order=0931worker2 # [ 0.316704] squashfs: version 4.0 (2009/01/31) Phillip Lougher932worker1 # [ 0.324629] squashfs: version 4.0 (2009/01/31) Phillip Lougher933worker1 # [ 0.325411] 9p: Installing v9fs 9p2000 file system support934worker2 # [ 0.322103] 9p: Installing v9fs 9p2000 file system support935worker2 # [ 0.342834] Key type asymmetric registered936client # [ 0.352123] Key type asymmetric registered937worker2 # [ 0.342855] Asymmetric key parser 'x509' registered938client # [ 0.352152] Asymmetric key parser 'x509' registered939lb # [ 0.349306] Key type asymmetric registered940worker2 # [ 0.342913] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242)941lb # [ 0.349333] Asymmetric key parser 'x509' registered942client # [ 0.352234] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242)943worker2 # [ 0.345006] io scheduler mq-deadline registered944lb # [ 0.349400] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242)945client # [ 0.354339] io scheduler mq-deadline registered946worker2 # [ 0.345017] io scheduler kyber registered947client # [ 0.354350] io scheduler kyber registered948lb # [ 0.351461] io scheduler mq-deadline registered949lb # [ 0.351472] io scheduler kyber registered950worker1 # [ 0.354159] Key type asymmetric registered951worker1 # [ 0.354188] Asymmetric key parser 'x509' registered952worker1 # [ 0.354256] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242)953worker1 # [ 0.356395] io scheduler mq-deadline registered954worker1 # [ 0.356406] io scheduler kyber registered955worker2 # [ 0.354227] pl061_gpio 9030000.pl061: PL061 GPIO chip registered956client # [ 0.364129] pl061_gpio 9030000.pl061: PL061 GPIO chip registered957lb # [ 0.361201] pl061_gpio 9030000.pl061: PL061 GPIO chip registered958client # [ 0.365550] ledtrig-cpu: registered to indicate activity on CPUs959lb # [ 0.362621] ledtrig-cpu: registered to indicate activity on CPUs960client # [ 0.365958] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:961worker1 # [ 0.366169] pl061_gpio 9030000.pl061: PL061 GPIO chip registered962lb # [ 0.363017] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:963client # [ 0.365976] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000964lb # [ 0.363034] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000965client # [ 0.365996] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000966lb # [ 0.363046] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000967worker1 # [ 0.367540] ledtrig-cpu: registered to indicate activity on CPUs968client # [ 0.366005] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000969lb # [ 0.363055] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000970worker1 # [ 0.367957] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:971client # [ 0.366026] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits972lb # [ 0.363083] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits973worker1 # [ 0.367974] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000974client # [ 0.366052] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]975lb # [ 0.363113] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]976client # [ 0.366131] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00977worker1 # [ 0.367998] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000978lb # [ 0.363190] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00979client # [ 0.366140] pci_bus 0000:00: root bus resource [bus 00-ff]980worker1 # [ 0.368007] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000981lb # [ 0.363200] pci_bus 0000:00: root bus resource [bus 00-ff]982client # [ 0.366147] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]983lb # [ 0.363206] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]984worker1 # [ 0.368034] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits985client # [ 0.366153] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]986lb # [ 0.363211] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]987client # [ 0.366158] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]988worker1 # [ 0.368058] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]989lb # [ 0.363216] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]990client # [ 0.366212] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint991worker2 # [ 0.362108] ledtrig-cpu: registered to indicate activity on CPUs992worker1 # [ 0.368140] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00993lb # [ 0.363278] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint994worker1 # [ 0.368150] pci_bus 0000:00: root bus resource [bus 00-ff]995worker2 # [ 0.362627] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:996client # [ 0.366645] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint997worker1 # [ 0.368155] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]998lb # [ 0.363713] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint999client # [ 0.366833] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f]1000worker2 # [ 0.362647] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x00000000001001lb # [ 0.363897] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f]1002worker1 # [ 0.368161] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]1003client # [ 0.366849] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff]1004lb # [ 0.363913] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff]1005worker2 # [ 0.362660] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x00100000001006client # [ 0.366879] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1007worker1 # [ 0.368166] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]1008lb # [ 0.363943] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1009client # [ 0.366895] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref]1010worker2 # [ 0.362668] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x80000000001011worker1 # [ 0.368221] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint1012lb # [ 0.363959] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref]1013client # [ 0.367357] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1014worker2 # [ 0.362689] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits1015worker1 # [ 0.368662] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1016lb # [ 0.364430] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1017client # [ 0.367557] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f]1018worker1 # [ 0.368848] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f]1019lb # [ 0.364612] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f]1020client # [ 0.367573] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff]1021worker2 # [ 0.362715] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]1022worker1 # [ 0.368864] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff]1023lb # [ 0.364627] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff]1024worker2 # [ 0.362795] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:001025client # [ 0.367603] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1026worker1 # [ 0.368893] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1027lb # [ 0.364656] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1028worker2 # [ 0.362804] pci_bus 0000:00: root bus resource [bus 00-ff]1029worker1 # [ 0.368909] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref]1030worker2 # [ 0.362810] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]1031worker1 # [ 0.369365] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1032worker2 # [ 0.362816] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]1033client # [ 0.386623] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1034worker1 # [ 0.369565] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f]1035lb # [ 0.383733] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1036worker2 # [ 0.362821] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]1037client # [ 0.386810] pci 0000:00:03.0: BAR 0 [io 0x0000-0x007f]1038lb # [ 0.383919] pci 0000:00:03.0: BAR 0 [io 0x0000-0x007f]1039worker1 # [ 0.369581] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff]1040client # [ 0.386826] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff]1041worker2 # [ 0.362927] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint1042lb # [ 0.383935] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff]1043worker1 # [ 0.369610] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1044client # [ 0.386856] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1045worker2 # [ 0.363367] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1046lb # [ 0.383964] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1047client # [ 0.387320] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1048worker2 # [ 0.363552] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f]1049lb # [ 0.384435] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1050client # [ 0.387511] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f]1051worker2 # [ 0.363568] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff]1052worker1 # [ 0.388791] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1053lb # [ 0.384622] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f]1054client # [ 0.387526] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff]1055worker2 # [ 0.363598] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1056worker1 # [ 0.388977] pci 0000:00:03.0: BAR 0 [io 0x0000-0x007f]1057lb # [ 0.384637] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff]1058client # [ 0.387556] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1059worker2 # [ 0.363613] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref]1060worker1 # [ 0.388993] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff]1061lb # [ 0.384666] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1062client # [ 0.387574] pci 0000:00:04.0: ROM [mem 0x00000000-0x0003ffff pref]1063worker1 # [ 0.389022] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1064worker2 # [ 0.364069] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1065lb # [ 0.384685] pci 0000:00:04.0: ROM [mem 0x00000000-0x0003ffff pref]1066worker2 # [ 0.364252] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f]1067worker1 # [ 0.389493] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1068worker2 # [ 0.364267] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff]1069worker1 # [ 0.389678] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f]1070worker2 # [ 0.364297] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1071worker1 # [ 0.389693] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff]1072worker1 # [ 0.389723] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1073worker2 # [ 0.364741] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1074worker2 # [ 0.364922] pci 0000:00:03.0: BAR 0 [io 0x0000-0x007f]1075worker1 # [ 0.389742] pci 0000:00:04.0: ROM [mem 0x00000000-0x0003ffff pref]1076worker2 # [ 0.364938] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff]1077worker2 # [ 0.364967] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1078worker2 # [ 0.365420] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1079worker2 # [ 0.365604] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f]1080worker2 # [ 0.365619] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff]1081worker2 # [ 0.365649] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1082worker2 # [ 0.365667] pci 0000:00:04.0: ROM [mem 0x00000000-0x0003ffff pref]1083worker2 # [ 0.366137] pci 0000:00:05.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint1084worker2 # [ 0.366324] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff]1085worker2 # [ 0.366354] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1086worker2 # [ 0.366820] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1087worker2 # [ 0.367007] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff]1088client # [ 0.400108] pci 0000:00:05.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint1089lb # [ 0.397189] pci 0000:00:05.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint1090worker2 # [ 0.367036] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1091client # [ 0.400320] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff]1092lb # [ 0.397391] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff]1093client # [ 0.400352] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1094worker2 # [ 0.367433] pci 0000:00:07.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint1095lb # [ 0.397422] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1096worker2 # [ 0.367615] pci 0000:00:07.0: BAR 0 [mem 0x00000000-0x00000fff]1097client # [ 0.400844] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1098lb # [ 0.397908] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1099client # [ 0.401033] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff]1100worker2 # [ 0.367899] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1101lb # [ 0.398095] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff]1102worker2 # [ 0.368087] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff]1103client # [ 0.401062] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1104worker1 # [ 0.402277] pci 0000:00:05.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint1105lb # [ 0.398125] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1106worker2 # [ 0.368116] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1107worker1 # [ 0.402480] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff]1108client # [ 0.401456] pci 0000:00:07.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint1109lb # [ 0.398512] pci 0000:00:07.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint1110worker2 # [ 0.368562] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1111worker1 # [ 0.402511] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1112client # [ 0.401638] pci 0000:00:07.0: BAR 0 [mem 0x00000000-0x00000fff]1113lb # [ 0.398694] pci 0000:00:07.0: BAR 0 [mem 0x00000000-0x00000fff]1114worker2 # [ 0.368745] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff]1115worker1 # [ 0.402976] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1116client # [ 0.401917] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1117lb # [ 0.398951] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1118worker2 # [ 0.368775] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1119worker1 # [ 0.403161] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff]1120client # [ 0.402104] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff]1121lb # [ 0.399138] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff]1122worker2 # [ 0.369242] pci 0000:00:0a.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1123worker1 # [ 0.403191] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1124client # [ 0.402134] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1125lb # [ 0.399168] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1126worker2 # [ 0.369426] pci 0000:00:0a.0: BAR 1 [mem 0x00000000-0x00000fff]1127worker1 # [ 0.403612] pci 0000:00:07.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint1128client # [ 0.402590] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1129lb # [ 0.399614] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1130worker2 # [ 0.369456] pci 0000:00:0a.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1131worker1 # [ 0.403797] pci 0000:00:07.0: BAR 0 [mem 0x00000000-0x00000fff]1132client # [ 0.402776] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff]1133lb # [ 0.399800] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff]1134worker2 # [ 0.369908] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1135client # [ 0.402806] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1136worker1 # [ 0.404046] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1137lb # [ 0.399829] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1138worker1 # [ 0.404231] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff]1139client # [ 0.403282] pci 0000:00:0a.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1140lb # [ 0.400292] pci 0000:00:0a.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1141worker1 # [ 0.404261] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1142client # [ 0.403476] pci 0000:00:0a.0: BAR 1 [mem 0x00000000-0x00000fff]1143lb # [ 0.400477] pci 0000:00:0a.0: BAR 1 [mem 0x00000000-0x00000fff]1144client # [ 0.403506] pci 0000:00:0a.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1145worker1 # [ 0.404714] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1146lb # [ 0.400507] pci 0000:00:0a.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1147worker1 # [ 0.404899] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff]1148client # [ 0.403970] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1149lb # [ 0.400966] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1150worker1 # [ 0.404929] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1151client # [ 0.404257] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f]1152lb # [ 0.401234] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f]1153client # [ 0.404274] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff]1154worker1 # [ 0.405380] pci 0000:00:0a.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1155lb # [ 0.401250] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff]1156client # [ 0.404303] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1157worker1 # [ 0.405580] pci 0000:00:0a.0: BAR 1 [mem 0x00000000-0x00000fff]1158lb # [ 0.401279] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1159worker1 # [ 0.405610] pci 0000:00:0a.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1160client # [ 0.404765] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1161lb # [ 0.401735] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1162client # [ 0.404947] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f]1163lb # [ 0.401919] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f]1164worker1 # [ 0.406096] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1165client # [ 0.404963] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff]1166lb # [ 0.401935] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff]1167worker1 # [ 0.406378] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f]1168client # [ 0.404993] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1169worker1 # [ 0.406397] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff]1170lb # [ 0.401964] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1171client # [ 0.405567] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned1172worker1 # [ 0.406428] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1173lb # [ 0.402536] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned1174client # [ 0.405578] pci 0000:00:04.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned1175worker1 # [ 0.406895] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1176lb # [ 0.402548] pci 0000:00:04.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned1177worker1 # [ 0.407087] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f]1178client # [ 0.405583] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned1179lb # [ 0.402553] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned1180worker1 # [ 0.407103] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff]1181client # [ 0.405628] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned1182lb # [ 0.402598] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned1183worker1 # [ 0.407132] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1184client # [ 0.405675] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned1185lb # [ 0.402645] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned1186worker2 # [ 0.418269] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f]1187worker1 # [ 0.407700] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned1188client # [ 0.405726] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned1189worker2 # [ 0.418297] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff]1190lb # [ 0.402691] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned1191worker1 # [ 0.407711] pci 0000:00:04.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned1192worker2 # [ 0.418327] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1193client # [ 0.405773] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned1194lb # [ 0.402737] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned1195worker1 # [ 0.407716] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned1196worker2 # [ 0.418873] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1197client # [ 0.405820] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned1198lb # [ 0.402784] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned1199worker1 # [ 0.407761] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned1200worker2 # [ 0.419066] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f]1201client # [ 0.405866] pci 0000:00:08.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned1202lb # [ 0.402829] pci 0000:00:08.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned1203worker2 # [ 0.419082] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff]1204worker1 # [ 0.407809] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned1205client # [ 0.405912] pci 0000:00:09.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned1206lb # [ 0.402875] pci 0000:00:09.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned1207worker2 # [ 0.419112] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1208worker1 # [ 0.407856] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned1209worker2 # [ 0.419676] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned1210client # [ 0.405958] pci 0000:00:0a.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned1211lb # [ 0.402920] pci 0000:00:0a.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned1212worker1 # [ 0.407903] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned1213worker2 # [ 0.419687] pci 0000:00:04.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned1214client # [ 0.406005] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned1215lb # [ 0.402966] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned1216worker1 # [ 0.407949] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned1217worker2 # [ 0.419693] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned1218client # [ 0.406077] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned1219lb # [ 0.403036] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned1220worker1 # [ 0.407995] pci 0000:00:08.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned1221worker2 # [ 0.419739] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned1222lb # [ 0.403081] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned1223client # [ 0.406123] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned1224worker1 # [ 0.408043] pci 0000:00:09.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned1225lb # [ 0.403103] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned1226worker2 # [ 0.419785] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned1227client # [ 0.406144] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned1228worker1 # [ 0.408089] pci 0000:00:0a.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned1229client # [ 0.406166] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned1230lb # [ 0.403124] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned1231worker2 # [ 0.419833] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned1232lb # [ 0.403145] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned1233client # [ 0.406187] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned1234worker1 # [ 0.408135] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned1235worker2 # [ 0.419880] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned1236lb # [ 0.403166] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned1237client # [ 0.406209] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned1238worker1 # [ 0.408207] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned1239worker2 # [ 0.419928] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned1240client # [ 0.406230] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned1241lb # [ 0.403187] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned1242worker1 # [ 0.408253] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned1243worker2 # [ 0.419976] pci 0000:00:08.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned1244lb # [ 0.403208] pci 0000:00:07.0: BAR 0 [mem 0x10086000-0x10086fff]: assigned1245client # [ 0.406252] pci 0000:00:07.0: BAR 0 [mem 0x10086000-0x10086fff]: assigned1246worker1 # [ 0.408274] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned1247worker2 # [ 0.420022] pci 0000:00:09.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned1248lb # [ 0.403233] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned1249client # [ 0.406277] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned1250worker1 # [ 0.408295] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned1251lb # [ 0.403255] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned1252client # [ 0.406299] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned1253worker1 # [ 0.408316] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned1254worker2 # [ 0.420069] pci 0000:00:0a.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned1255lb # [ 0.403277] pci 0000:00:0a.0: BAR 1 [mem 0x10089000-0x10089fff]: assigned1256client # [ 0.406321] pci 0000:00:0a.0: BAR 1 [mem 0x10089000-0x10089fff]: assigned1257worker1 # [ 0.408337] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned1258worker2 # [ 0.420116] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned1259lb # [ 0.403299] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned1260worker1 # [ 0.408358] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned1261client # [ 0.406344] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned1262worker2 # [ 0.420236] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned1263lb # [ 0.403321] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned1264client # [ 0.406366] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned1265worker1 # [ 0.408379] pci 0000:00:07.0: BAR 0 [mem 0x10086000-0x10086fff]: assigned1266worker2 # [ 0.420282] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned1267lb # [ 0.403342] pci 0000:00:03.0: BAR 0 [io 0x1000-0x107f]: assigned1268client # [ 0.406387] pci 0000:00:03.0: BAR 0 [io 0x1000-0x107f]: assigned1269worker1 # [ 0.408405] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned1270worker2 # [ 0.420304] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned1271lb # [ 0.403363] pci 0000:00:0b.0: BAR 0 [io 0x1080-0x10bf]: assigned1272client # [ 0.406408] pci 0000:00:0b.0: BAR 0 [io 0x1080-0x10bf]: assigned1273worker1 # [ 0.408427] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned1274worker2 # [ 0.420325] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned1275lb # [ 0.403383] pci 0000:00:01.0: BAR 0 [io 0x10c0-0x10df]: assigned1276client # [ 0.406429] pci 0000:00:01.0: BAR 0 [io 0x10c0-0x10df]: assigned1277worker1 # [ 0.408449] pci 0000:00:0a.0: BAR 1 [mem 0x10089000-0x10089fff]: assigned1278lb # [ 0.403404] pci 0000:00:02.0: BAR 0 [io 0x10e0-0x10ff]: assigned1279worker2 # [ 0.420347] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned1280client # [ 0.406451] pci 0000:00:02.0: BAR 0 [io 0x10e0-0x10ff]: assigned1281worker1 # [ 0.408471] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned1282lb # [ 0.403425] pci 0000:00:04.0: BAR 0 [io 0x1100-0x111f]: assigned1283worker2 # [ 0.420367] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned1284client # [ 0.406472] pci 0000:00:04.0: BAR 0 [io 0x1100-0x111f]: assigned1285worker1 # [ 0.408493] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned1286lb # [ 0.403446] pci 0000:00:0c.0: BAR 0 [io 0x1120-0x113f]: assigned1287worker2 # [ 0.420389] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned1288client # [ 0.406493] pci 0000:00:0c.0: BAR 0 [io 0x1120-0x113f]: assigned1289worker1 # [ 0.408514] pci 0000:00:03.0: BAR 0 [io 0x1000-0x107f]: assigned1290lb # [ 0.403472] pci_bus 0000:00: resource 4 [io 0x0000-0xffff]1291client # [ 0.406519] pci_bus 0000:00: resource 4 [io 0x0000-0xffff]1292worker2 # [ 0.420409] pci 0000:00:07.0: BAR 0 [mem 0x10086000-0x10086fff]: assigned1293worker1 # [ 0.408535] pci 0000:00:0b.0: BAR 0 [io 0x1080-0x10bf]: assigned1294lb # [ 0.403481] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff]1295client # [ 0.406528] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff]1296worker2 # [ 0.420435] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned1297worker1 # [ 0.408556] pci 0000:00:01.0: BAR 0 [io 0x10c0-0x10df]: assigned1298lb # [ 0.403485] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff]1299client # [ 0.406532] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff]1300worker2 # [ 0.420457] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned1301worker1 # [ 0.408577] pci 0000:00:02.0: BAR 0 [io 0x10e0-0x10ff]: assigned1302lb # [ 0.404301] pci 0000:00:07.0: enabling device (0000 -> 0002)1303client # [ 0.407346] pci 0000:00:07.0: enabling device (0000 -> 0002)1304worker2 # [ 0.420478] pci 0000:00:0a.0: BAR 1 [mem 0x10089000-0x10089fff]: assigned1305worker1 # [ 0.408598] pci 0000:00:04.0: BAR 0 [io 0x1100-0x111f]: assigned1306worker2 # [ 0.420500] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned1307worker1 # [ 0.408619] pci 0000:00:0c.0: BAR 0 [io 0x1120-0x113f]: assigned1308worker1 # [ 0.408645] pci_bus 0000:00: resource 4 [io 0x0000-0xffff]1309worker2 # [ 0.420522] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned1310worker1 # [ 0.408654] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff]1311worker2 # [ 0.420543] pci 0000:00:03.0: BAR 0 [io 0x1000-0x107f]: assigned1312worker1 # [ 0.408659] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff]1313worker2 # [ 0.420564] pci 0000:00:0b.0: BAR 0 [io 0x1080-0x10bf]: assigned1314worker1 # [ 0.409475] pci 0000:00:07.0: enabling device (0000 -> 0002)1315worker2 # [ 0.420584] pci 0000:00:01.0: BAR 0 [io 0x10c0-0x10df]: assigned1316worker2 # [ 0.420605] pci 0000:00:02.0: BAR 0 [io 0x10e0-0x10ff]: assigned1317worker2 # [ 0.420625] pci 0000:00:04.0: BAR 0 [io 0x1100-0x111f]: assigned1318worker2 # [ 0.420646] pci 0000:00:0c.0: BAR 0 [io 0x1120-0x113f]: assigned1319worker2 # [ 0.420672] pci_bus 0000:00: resource 4 [io 0x0000-0xffff]1320worker2 # [ 0.420681] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff]1321worker2 # [ 0.420686] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff]1322worker2 # [ 0.421493] pci 0000:00:07.0: enabling device (0000 -> 0002)1323client # [ 0.483930] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)1324lb # [ 0.480731] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)1325worker2 # [ 0.475725] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)1326worker2 # [ 0.477797] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003)1327worker1 # [ 0.485503] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)1328client # [ 0.493383] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003)1329lb # [ 0.490444] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003)1330worker2 # [ 0.488720] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003)1331client # [ 0.497574] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003)1332worker1 # [ 0.495386] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003)1333client # [ 0.499469] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003)1334lb # [ 0.494948] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003)1335lb # [ 0.496898] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003)1336lb # [ 0.498846] virtio-pci 0000:00:05.0: enabling device (0000 -> 0002)1337lb # [ 0.500912] virtio-pci 0000:00:06.0: enabling device (0000 -> 0002)1338worker2 # [ 0.494624] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003)1339worker1 # [ 0.499955] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003)1340worker2 # [ 0.496631] virtio-pci 0000:00:05.0: enabling device (0000 -> 0002)1341worker1 # [ 0.501867] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003)1342worker1 # [ 0.503856] virtio-pci 0000:00:05.0: enabling device (0000 -> 0002)1343worker1 # [ 0.505901] virtio-pci 0000:00:06.0: enabling device (0000 -> 0002)1344client # [ 0.513486] virtio-pci 0000:00:05.0: enabling device (0000 -> 0002)1345client # [ 0.515849] virtio-pci 0000:00:06.0: enabling device (0000 -> 0002)1346worker2 # [ 0.507111] virtio-pci 0000:00:06.0: enabling device (0000 -> 0002)1347worker2 # [ 0.508852] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002)1348client # [ 0.519169] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002)1349lb # [ 0.518527] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002)1350client # [ 0.522049] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002)1351lb # [ 0.520577] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002)1352worker1 # [ 0.523612] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002)1353lb # [ 0.524015] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0002)1354worker1 # [ 0.525531] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002)1355worker2 # [ 0.518835] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002)1356worker2 # [ 0.521395] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0002)1357worker1 # [ 0.529070] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0002)1358worker2 # [ 0.524810] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003)1359lb # [ 0.532245] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003)1360client # [ 0.532014] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0002)1361client # [ 0.533912] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003)1362worker1 # [ 0.537991] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003)1363client # [ 0.536230] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003)1364worker2 # [ 0.536157] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003)1365lb # [ 0.542687] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003)1366client # [ 0.549485] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1367worker1 # [ 0.547469] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003)1368client # [ 0.557183] msm_serial: driver initialized1369client # [ 0.557327] SuperH (H)SCI(F) driver initialized1370client # [ 0.557383] STM32 USART driver initialized1371lb # [ 0.558364] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1372worker1 # [ 0.561327] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1373lb # [ 0.560938] msm_serial: driver initialized1374worker2 # [ 0.557885] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1375worker1 # [ 0.564087] msm_serial: driver initialized1376lb # [ 0.561643] SuperH (H)SCI(F) driver initialized1377worker1 # [ 0.564232] SuperH (H)SCI(F) driver initialized1378lb # [ 0.561705] STM32 USART driver initialized1379worker1 # [ 0.564287] STM32 USART driver initialized1380worker2 # [ 0.560476] msm_serial: driver initialized1381worker2 # [ 0.560627] SuperH (H)SCI(F) driver initialized1382worker2 # [ 0.560681] STM32 USART driver initialized1383client # [ 0.586939] loop: module loaded1384client # [ 0.587115] virtio_blk virtio2: 1/0/0 default/read/poll queues1385client # [ 0.587899] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB)1386lb # [ 0.594000] loop: module loaded1387worker2 # [ 0.589658] loop: module loaded1388lb # [ 0.594182] virtio_blk virtio2: 1/0/0 default/read/poll queues1389worker2 # [ 0.589855] virtio_blk virtio2: 1/0/0 default/read/poll queues1390lb # [ 0.594976] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB)1391worker2 # [ 0.591854] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB)1392worker1 # [ 0.598754] loop: module loaded1393worker1 # [ 0.598932] virtio_blk virtio2: 1/0/0 default/read/poll queues1394worker1 # [ 0.599711] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB)1395client # [ 0.600589] megasas: 07.734.00.00-rc11396client # [ 0.601287] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff]1397client # [ 0.603486] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001398client # [ 0.603510] Intel/Sharp Extended Query Table at 0x00311399client # [ 0.605082] Using buffer write method1400client # [ 0.605157] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff]1401client # [ 0.606810] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001402client # [ 0.606831] Intel/Sharp Extended Query Table at 0x00311403lb # [ 0.605731] megasas: 07.734.00.00-rc11404lb # [ 0.606410] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff]1405lb # [ 0.608687] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001406worker2 # [ 0.602693] megasas: 07.734.00.00-rc11407lb # [ 0.608711] Intel/Sharp Extended Query Table at 0x00311408lb # [ 0.610305] Using buffer write method1409worker2 # [ 0.603407] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff]1410worker1 # [ 0.610670] megasas: 07.734.00.00-rc11411client # [ 0.620516] Using buffer write method1412lb # [ 0.610368] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff]1413client # [ 0.620541] Concatenating MTD devices:1414worker1 # [ 0.611424] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff]1415worker2 # [ 0.605579] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001416client # [ 0.620545] (0): "0.flash"1417client # [ 0.620549] (1): "0.flash"1418worker2 # [ 0.605603] Intel/Sharp Extended Query Table at 0x00311419client # [ 0.620557] into device "0.flash"1420worker2 # [ 0.607159] Using buffer write method1421worker1 # [ 0.613346] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001422worker1 # [ 0.613372] Intel/Sharp Extended Query Table at 0x00311423lb # [ 0.612132] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001424worker2 # [ 0.607241] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff]1425worker1 # [ 0.615069] Using buffer write method1426lb # [ 0.612155] Intel/Sharp Extended Query Table at 0x00311427worker2 # [ 0.608911] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001428worker1 # [ 0.615128] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff]1429worker2 # [ 0.608932] Intel/Sharp Extended Query Table at 0x00311430worker1 # [ 0.616852] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001431worker1 # [ 0.616875] Intel/Sharp Extended Query Table at 0x00311432lb # [ 0.630614] Using buffer write method1433lb # [ 0.630647] Concatenating MTD devices:1434lb # [ 0.630651] (0): "0.flash"1435lb # [ 0.630655] (1): "0.flash"1436lb # [ 0.630662] into device "0.flash"1437worker2 # [ 0.627686] Using buffer write method1438worker1 # [ 0.634904] Using buffer write method1439worker2 # [ 0.627718] Concatenating MTD devices:1440worker1 # [ 0.634938] Concatenating MTD devices:1441worker2 # [ 0.627722] (0): "0.flash"1442worker1 # [ 0.634942] (0): "0.flash"1443worker2 # [ 0.627726] (1): "0.flash"1444worker1 # [ 0.634946] (1): "0.flash"1445worker2 # [ 0.627736] into device "0.flash"1446worker1 # [ 0.634955] into device "0.flash"1447client # [ 0.860378] Freeing initrd memory: 26972K1448client # [ 0.866437] tun: Universal TUN/TAP device driver, 1.61449lb # [ 0.864745] Freeing initrd memory: 26948K1450client # [ 0.870246] thunder_xcv, ver 1.01451client # [ 0.870291] thunder_bgx, ver 1.01452client # [ 0.870314] nicpf, ver 1.01453client # [ 0.870873] e1000: Intel(R) PRO/1000 Network Driver1454client # [ 0.870880] e1000: Copyright (c) 1999-2006 Intel Corporation.1455lb # [ 0.870718] tun: Universal TUN/TAP device driver, 1.61456client # [ 0.870904] e1000e: Intel(R) PRO/1000 Network Driver1457client # [ 0.870913] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.1458client # [ 0.870941] igb: Intel(R) Gigabit Ethernet Network Driver1459client # [ 0.870948] igb: Copyright (c) 2007-2014 Intel Corporation.1460client # [ 0.870970] igbvf: Intel(R) Gigabit Virtual Function Network Driver1461client # [ 0.870976] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.1462worker2 # [ 0.869181] Freeing initrd memory: 26944K1463client # [ 0.871114] sky2: driver version 1.301464client # [ 0.880705] ehci-pci 0000:00:07.0: EHCI Host Controller1465lb # [ 0.874600] thunder_xcv, ver 1.01466lb # [ 0.874646] thunder_bgx, ver 1.01467worker1 # [ 0.879890] Freeing initrd memory: 26976K1468client # [ 0.880737] ehci-pci 0000:00:07.0: new USB bus registered, assigned bus number 11469lb # [ 0.874669] nicpf, ver 1.01470client # [ 0.880985] ehci-pci 0000:00:07.0: irq 16, io mem 0x100860001471lb # [ 0.875235] e1000: Intel(R) PRO/1000 Network Driver1472lb # [ 0.875243] e1000: Copyright (c) 1999-2006 Intel Corporation.1473client # [ 0.883422] usbcore: registered new interface driver usb-storage1474lb # [ 0.875266] e1000e: Intel(R) PRO/1000 Network Driver1475worker2 # [ 0.875180] tun: Universal TUN/TAP device driver, 1.61476client # [ 0.883501] usbcore: registered new interface driver usbserial_generic1477lb # [ 0.875274] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.1478client # [ 0.883516] usbserial: USB Serial support registered for generic1479lb # [ 0.875303] igb: Intel(R) Gigabit Ethernet Network Driver1480lb # [ 0.875309] igb: Copyright (c) 2007-2014 Intel Corporation.1481lb # [ 0.875332] igbvf: Intel(R) Gigabit Virtual Function Network Driver1482client # [ 0.886390] hv_vmbus: registering driver hyperv_keyboard1483lb # [ 0.875338] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.1484worker1 # [ 0.885925] tun: Universal TUN/TAP device driver, 1.61485lb # [ 0.875475] sky2: driver version 1.301486lb # [ 0.877061] usbcore: registered new interface driver usb-storage1487client # [ 0.887983] rtc-pl031 9010000.pl031: registered as rtc01488lb # [ 0.877823] ehci-pci 0000:00:07.0: EHCI Host Controller1489client # [ 0.888018] rtc-pl031 9010000.pl031: setting system clock to 2026-09-21T14:29:24 UTC (1790000964)1490lb # [ 0.877849] ehci-pci 0000:00:07.0: new USB bus registered, assigned bus number 11491client # [ 0.888318] i2c_dev: i2c /dev entries driver1492lb # [ 0.878153] ehci-pci 0000:00:07.0: irq 16, io mem 0x100860001493worker2 # [ 0.879010] thunder_xcv, ver 1.01494worker1 # [ 0.889667] thunder_xcv, ver 1.01495worker2 # [ 0.879055] thunder_bgx, ver 1.01496worker1 # [ 0.889710] thunder_bgx, ver 1.01497worker2 # [ 0.879077] nicpf, ver 1.01498worker1 # [ 0.889733] nicpf, ver 1.01499worker2 # [ 0.879655] e1000: Intel(R) PRO/1000 Network Driver1500lb # [ 0.888394] usbcore: registered new interface driver usbserial_generic1501worker2 # [ 0.879662] e1000: Copyright (c) 1999-2006 Intel Corporation.1502lb # [ 0.888414] usbserial: USB Serial support registered for generic1503worker2 # [ 0.879686] e1000e: Intel(R) PRO/1000 Network Driver1504lb # [ 0.889061] hv_vmbus: registering driver hyperv_keyboard1505worker2 # [ 0.879693] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.1506lb # [ 0.889113] ehci-pci 0000:00:07.0: USB 2.0 started, EHCI 1.001507worker2 # [ 0.879722] igb: Intel(R) Gigabit Ethernet Network Driver1508lb # [ 0.889406] hub 1-0:1.0: USB hub found1509client # [ 0.893699] sdhci: Secure Digital Host Controller Interface driver1510lb # [ 0.889427] hub 1-0:1.0: 6 ports detected1511worker2 # [ 0.879727] igb: Copyright (c) 2007-2014 Intel Corporation.1512client # [ 0.893710] sdhci: Copyright(c) Pierre Ossman1513worker1 # [ 0.891495] e1000: Intel(R) PRO/1000 Network Driver1514worker2 # [ 0.879750] igbvf: Intel(R) Gigabit Virtual Function Network Driver1515client # [ 0.893981] Synopsys Designware Multimedia Card Interface Driver1516worker1 # [ 0.891506] e1000: Copyright (c) 1999-2006 Intel Corporation.1517worker2 # [ 0.879756] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.1518client # [ 0.894354] sdhci-pltfm: SDHCI platform and OF driver helper1519worker2 # [ 0.879893] sky2: driver version 1.301520worker1 # [ 0.891531] e1000e: Intel(R) PRO/1000 Network Driver1521client # [ 0.895895] hid: raw HID events driver (C) Jiri Kosina1522lb # [ 0.894404] rtc-pl031 9010000.pl031: registered as rtc01523worker1 # [ 0.891539] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.1524worker2 # [ 0.881471] usbcore: registered new interface driver usb-storage1525client # [ 0.896019] ehci-pci 0000:00:07.0: USB 2.0 started, EHCI 1.001526worker1 # [ 0.891574] igb: Intel(R) Gigabit Ethernet Network Driver1527lb # [ 0.894434] rtc-pl031 9010000.pl031: setting system clock to 2026-09-21T14:29:24 UTC (1790000964)1528worker2 # [ 0.881558] usbcore: registered new interface driver usbserial_generic1529worker1 # [ 0.891580] igb: Copyright (c) 2007-2014 Intel Corporation.1530lb # [ 0.894727] i2c_dev: i2c /dev entries driver1531client # [ 0.900311] hub 1-0:1.0: USB hub found1532worker2 # [ 0.881573] usbserial: USB Serial support registered for generic1533client # [ 0.900330] hub 1-0:1.0: 6 ports detected1534worker1 # [ 0.891604] igbvf: Intel(R) Gigabit Virtual Function Network Driver1535worker2 # [ 0.882803] ehci-pci 0000:00:07.0: EHCI Host Controller1536worker1 # [ 0.891610] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.1537worker2 # [ 0.882829] ehci-pci 0000:00:07.0: new USB bus registered, assigned bus number 11538worker1 # [ 0.891743] sky2: driver version 1.301539worker2 # [ 0.882985] ehci-pci 0000:00:07.0: irq 16, io mem 0x100860001540client # [ 0.901838] usbcore: registered new interface driver usbhid1541worker1 # [ 0.893251] usbcore: registered new interface driver usb-storage1542client # [ 0.901846] usbhid: USB HID core driver1543worker1 # [ 0.893445] usbcore: registered new interface driver usbserial_generic1544worker2 # [ 0.894447] ehci-pci 0000:00:07.0: USB 2.0 started, EHCI 1.001545worker1 # [ 0.893467] usbserial: USB Serial support registered for generic1546lb # [ 0.899957] sdhci: Secure Digital Host Controller Interface driver1547worker2 # [ 0.894752] hub 1-0:1.0: USB hub found1548worker2 # [ 0.894773] hub 1-0:1.0: 6 ports detected1549lb # [ 0.899971] sdhci: Copyright(c) Pierre Ossman1550worker1 # [ 0.894741] ehci-pci 0000:00:07.0: EHCI Host Controller1551lb # [ 0.900247] Synopsys Designware Multimedia Card Interface Driver1552worker1 # [ 0.894767] ehci-pci 0000:00:07.0: new USB bus registered, assigned bus number 11553client # [ 0.904165] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available1554lb # [ 0.900614] sdhci-pltfm: SDHCI platform and OF driver helper1555worker2 # [ 0.896874] hv_vmbus: registering driver hyperv_keyboard1556worker1 # [ 0.894995] ehci-pci 0000:00:07.0: irq 16, io mem 0x100860001557client # [ 0.905697] drop_monitor: Initializing network drop monitor service1558client # [ 0.905827] NET: Registered PF_INET6 protocol family1559worker1 # [ 0.905241] hv_vmbus: registering driver hyperv_keyboard1560worker2 # [ 0.898367] rtc-pl031 9010000.pl031: registered as rtc01561lb # [ 0.904748] hid: raw HID events driver (C) Jiri Kosina1562client # [ 0.908801] Segment Routing with IPv61563lb # [ 0.905007] usbcore: registered new interface driver usbhid1564client # [ 0.908818] In-situ OAM (IOAM) with IPv61565worker1 # [ 0.906664] ehci-pci 0000:00:07.0: USB 2.0 started, EHCI 1.001566worker2 # [ 0.898392] rtc-pl031 9010000.pl031: setting system clock to 2026-09-21T14:29:24 UTC (1790000964)1567lb # [ 0.905014] usbhid: USB HID core driver1568client # [ 0.908847] NET: Registered PF_PACKET protocol family1569worker1 # [ 0.907007] hub 1-0:1.0: USB hub found1570worker2 # [ 0.898711] i2c_dev: i2c /dev entries driver1571worker1 # [ 0.907040] hub 1-0:1.0: 6 ports detected1572client # [ 0.910440] 9pnet: Installing 9P2000 support1573worker1 # [ 0.909198] rtc-pl031 9010000.pl031: registered as rtc01574client # [ 0.910529] Key type dns_resolver registered1575worker1 # [ 0.909225] rtc-pl031 9010000.pl031: setting system clock to 2026-09-21T14:29:24 UTC (1790000964)1576worker1 # [ 0.909544] i2c_dev: i2c /dev entries driver1577lb # [ 0.907871] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available1578lb # [ 0.909486] drop_monitor: Initializing network drop monitor service1579lb # [ 0.909620] NET: Registered PF_INET6 protocol family1580worker2 # [ 0.903933] sdhci: Secure Digital Host Controller Interface driver1581worker2 # [ 0.903944] sdhci: Copyright(c) Pierre Ossman1582lb # [ 0.912640] Segment Routing with IPv61583worker2 # [ 0.904212] Synopsys Designware Multimedia Card Interface Driver1584lb # [ 0.912659] In-situ OAM (IOAM) with IPv61585lb # [ 0.912686] NET: Registered PF_PACKET protocol family1586worker2 # [ 0.904592] sdhci-pltfm: SDHCI platform and OF driver helper1587worker2 # [ 0.908771] hid: raw HID events driver (C) Jiri Kosina1588worker2 # [ 0.909035] usbcore: registered new interface driver usbhid1589client # [ 0.917505] registered taskstats version 11590worker1 # [ 0.914776] sdhci: Secure Digital Host Controller Interface driver1591worker2 # [ 0.909042] usbhid: USB HID core driver1592lb # [ 0.914504] 9pnet: Installing 9P2000 support1593client # [ 0.917647] Loading compiled-in X.509 certificates1594worker1 # [ 0.914786] sdhci: Copyright(c) Pierre Ossman1595lb # [ 0.914547] Key type dns_resolver registered1596worker1 # [ 0.915045] Synopsys Designware Multimedia Card Interface Driver1597worker1 # [ 0.915407] sdhci-pltfm: SDHCI platform and OF driver helper1598worker1 # [ 0.916898] hid: raw HID events driver (C) Jiri Kosina1599worker2 # [ 0.911969] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available1600worker1 # [ 0.917146] usbcore: registered new interface driver usbhid1601worker1 # [ 0.917152] usbhid: USB HID core driver1602worker2 # [ 0.913500] drop_monitor: Initializing network drop monitor service1603worker2 # [ 0.913665] NET: Registered PF_INET6 protocol family1604worker2 # [ 0.916686] Segment Routing with IPv61605worker2 # [ 0.916720] In-situ OAM (IOAM) with IPv61606worker2 # [ 0.916746] NET: Registered PF_PACKET protocol family1607client # [ 0.926552] Demotion targets for Node 0: null1608worker1 # [ 0.922716] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available1609client # [ 0.926656] Key type .fscrypt registered1610worker1 # [ 0.924227] drop_monitor: Initializing network drop monitor service1611client # [ 0.926663] Key type fscrypt-provisioning registered1612worker1 # [ 0.924406] NET: Registered PF_INET6 protocol family1613client # [ 0.926755] ima: No TPM chip found, activating TPM-bypass!1614lb # [ 0.921244] registered taskstats version 11615client # [ 0.926775] ima: Allocated hash algorithm: sha11616lb # [ 0.921398] Loading compiled-in X.509 certificates1617worker1 # [ 0.927407] Segment Routing with IPv61618client # [ 0.926795] ima: No architecture policies found1619worker1 # [ 0.927429] In-situ OAM (IOAM) with IPv61620worker1 # [ 0.927460] NET: Registered PF_PACKET protocol family1621client # [ 0.930736] input: gpio-keys as /devices/platform/gpio-keys/input/input01622worker2 # [ 0.918547] 9pnet: Installing 9P2000 support1623worker2 # [ 0.918591] Key type dns_resolver registered1624worker1 # [ 0.929067] 9pnet: Installing 9P2000 support1625worker1 # [ 0.929110] Key type dns_resolver registered1626lb # [ 0.929818] Demotion targets for Node 0: null1627lb # [ 0.929919] Key type .fscrypt registered1628lb # [ 0.929925] Key type fscrypt-provisioning registered1629worker2 # [ 0.925330] registered taskstats version 11630lb # [ 0.930022] ima: No TPM chip found, activating TPM-bypass!1631worker2 # [ 0.925469] Loading compiled-in X.509 certificates1632lb # [ 0.930042] ima: Allocated hash algorithm: sha11633lb # [ 0.930067] ima: No architecture policies found1634lb # [ 0.934023] input: gpio-keys as /devices/platform/gpio-keys/input/input01635worker1 # [ 0.935863] registered taskstats version 11636worker1 # [ 0.936039] Loading compiled-in X.509 certificates1637worker2 # [ 0.933933] Demotion targets for Node 0: null1638worker2 # [ 0.934039] Key type .fscrypt registered1639worker2 # [ 0.934046] Key type fscrypt-provisioning registered1640worker2 # [ 0.935887] ima: No TPM chip found, activating TPM-bypass!1641worker2 # [ 0.935910] ima: Allocated hash algorithm: sha11642worker2 # [ 0.935933] ima: No architecture policies found1643worker1 # [ 0.944515] Demotion targets for Node 0: null1644worker1 # [ 0.944618] Key type .fscrypt registered1645worker2 # [ 0.938250] input: gpio-keys as /devices/platform/gpio-keys/input/input01646worker1 # [ 0.944625] Key type fscrypt-provisioning registered1647worker1 # [ 0.944715] ima: No TPM chip found, activating TPM-bypass!1648worker1 # [ 0.944734] ima: Allocated hash algorithm: sha11649worker1 # [ 0.944754] ima: No architecture policies found1650client # [ 0.948227] clk: Disabling unused clocks1651client # [ 0.948247] PM: genpd: Disabling unused power domains1652worker1 # [ 0.948795] input: gpio-keys as /devices/platform/gpio-keys/input/input01653client # [ 0.952385] Freeing unused kernel memory: 4736K1654client # [ 0.952568] Run /init as init process1655lb # [ 0.951689] clk: Disabling unused clocks1656lb # [ 0.951716] PM: genpd: Disabling unused power domains1657lb # [ 0.955874] Freeing unused kernel memory: 4736K1658lb # [ 0.956136] Run /init as init process1659worker2 # [ 0.955893] clk: Disabling unused clocks1660worker2 # [ 0.955922] PM: genpd: Disabling unused power domains1661worker2 # [ 0.960085] Freeing unused kernel memory: 4736K1662worker2 # [ 0.960308] Run /init as init process1663client # [ 0.970287] systemd[1]: Successfully made /usr/ read-only.1664worker1 # [ 0.967145] clk: Disabling unused clocks1665worker1 # [ 0.967174] PM: genpd: Disabling unused power domains1666worker1 # [ 0.971416] Freeing unused kernel memory: 4736K1667worker1 # [ 0.971663] Run /init as init process1668lb # [ 0.971299] systemd[1]: Successfully made /usr/ read-only.1669worker2 # [ 0.976238] systemd[1]: Successfully made /usr/ read-only.1670worker1 # [ 0.988203] systemd[1]: Successfully made /usr/ read-only.1671lb # [ 1.137153] usb 1-1: new high-speed USB device number 2 using ehci-pci1672client # [ 1.148050] usb 1-1: new high-speed USB device number 2 using ehci-pci1673worker2 # [ 1.142154] usb 1-1: new high-speed USB device number 2 using ehci-pci1674worker1 # [ 1.154111] usb 1-1: new high-speed USB device number 2 using ehci-pci1675lb # [ 1.289663] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input11676client # [ 1.300716] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input11677worker2 # [ 1.294714] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input11678worker1 # [ 1.306783] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input11679lb # [ 1.305980] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1680lb # [ 1.312515] systemd[1]: Detected virtualization qemu.1681lb # [ 1.313445] systemd[1]: Detected architecture arm64.1682lb # [ 1.314308] systemd[1]: Running in initrd.1683client # [ 1.307058] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1684lb # [ 1.315519] systemd[1]: Initializing machine ID from random generator.1685lb # [ 1.316874] systemd[1]: Hostname set to <lb>.1686client # [ 1.307122] systemd[1]: Detected virtualization qemu.1687client # [ 1.307204] systemd[1]: Detected architecture arm64.1688client # [ 1.307230] systemd[1]: Running in initrd.1689client # [ 1.326088] systemd[1]: Initializing machine ID from random generator.1690client # [ 1.328945] systemd[1]: Hostname set to <client>.1691worker2 # [ 1.311034] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1692worker2 # [ 1.323149] systemd[1]: Detected virtualization qemu.1693worker2 # [ 1.325128] systemd[1]: Detected architecture arm64.1694worker1 # [ 1.323073] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1695worker2 # [ 1.327100] systemd[1]: Running in initrd.1696worker1 # [ 1.335186] systemd[1]: Detected virtualization qemu.1697worker1 # [ 1.337182] systemd[1]: Detected architecture arm64.1698worker2 # [ 1.329687] systemd[1]: Initializing machine ID from random generator.1699worker1 # [ 1.339217] systemd[1]: Running in initrd.1700worker2 # [ 1.332743] systemd[1]: Hostname set to <worker2>.1701worker1 # [ 1.341888] systemd[1]: Initializing machine ID from random generator.1702worker1 # [ 1.344903] systemd[1]: Hostname set to <worker1>.1703lb # [ 1.369308] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:07.0-1/input01704client # [ 1.384257] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:07.0-1/input01705worker1 # [ 1.390230] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:07.0-1/input01706worker2 # [ 1.382379] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:07.0-1/input01707lb # [ 1.489154] usb 1-2: new high-speed USB device number 3 using ehci-pci1708client # [ 1.504048] usb 1-2: new high-speed USB device number 3 using ehci-pci1709worker2 # [ 1.502164] usb 1-2: new high-speed USB device number 3 using ehci-pci1710worker1 # [ 1.510099] usb 1-2: new high-speed USB device number 3 using ehci-pci1711lb # [ 1.571576] systemd[1]: bpf-restrict-fs: LSM BPF program attached1712worker1 # [ 1.616358] systemd[1]: bpf-restrict-fs: LSM BPF program attached1713client # [ 1.631089] systemd[1]: bpf-restrict-fs: LSM BPF program attached1714lb # [ 1.632263] systemd[1]: Queued start job for default target Initrd Default Target.1715worker2 # [ 1.633564] systemd[1]: bpf-restrict-fs: LSM BPF program attached1716lb # [ 1.640949] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input21717lb # [ 1.644601] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:07.0-2/input01718lb # [ 1.648100] systemd[1]: Created slice Slice /system/modprobe.1719lb # [ 1.649313] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1720lb # [ 1.649354] systemd[1]: Expecting device /dev/disk/by-label/nixos...1721lb # [ 1.649387] systemd[1]: Reached target Path Units.1722lb # [ 1.649405] systemd[1]: Reached target Slice Units.1723lb # [ 1.649424] systemd[1]: Reached target Swaps.1724lb # [ 1.649443] systemd[1]: Reached target Timer Units.1725lb # [ 1.649621] systemd[1]: Listening on D-Bus System Message Bus Socket.1726lb # [ 1.649767] systemd[1]: Listening on Journal Socket (/dev/log).1727lb # [ 1.649908] systemd[1]: Listening on Journal Sockets.1728lb # [ 1.650038] systemd[1]: Listening on udev Control Socket.1729lb # [ 1.650145] systemd[1]: Listening on udev Kernel Socket.1730lb # [ 1.650167] systemd[1]: Reached target Socket Units.1731client # [ 1.660707] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input21732lb # [ 1.662160] systemd[1]: Starting Create List of Static Device Nodes...1733lb # [ 1.663192] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs1734client # [ 1.666648] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:07.0-2/input01735worker1 # [ 1.666487] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input21736worker2 # [ 1.660419] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input21737lb # [ 1.670312] systemd[1]: Mounting Kernel Configuration File System...1738worker1 # [ 1.670188] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:07.0-2/input01739worker2 # [ 1.665803] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:07.0-2/input01740lb # [ 1.685471] systemd[1]: Starting Journal Service...1741lb # [ 1.693278] systemd[1]: Starting Load Kernel Modules...1742lb # [ 1.694120] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1743lb # [ 1.709473] systemd[1]: Starting Coldplug All udev Devices...1744worker1 # [ 1.713487] systemd[1]: Queued start job for default target Initrd Default Target.1745worker1 # [ 1.725143] systemd[1]: Created slice Slice /system/modprobe.1746worker1 # [ 1.726515] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1747worker1 # [ 1.727833] systemd[1]: Expecting device /dev/disk/by-label/nixos...1748worker1 # [ 1.728889] systemd[1]: Reached target Path Units.1749worker1 # [ 1.729707] systemd[1]: Reached target Slice Units.1750worker1 # [ 1.730616] systemd[1]: Reached target Swaps.1751worker1 # [ 1.731356] systemd[1]: Reached target Timer Units.1752worker1 # [ 1.732456] systemd[1]: Listening on D-Bus System Message Bus Socket.1753worker1 # [ 1.733697] systemd[1]: Listening on Journal Socket (/dev/log).1754lb # [ 1.733224] systemd[1]: Finished Create List of Static Device Nodes.1755worker1 # [ 1.734893] systemd[1]: Listening on Journal Sockets.1756worker1 # [ 1.735036] systemd[1]: Listening on udev Control Socket.1757worker1 # [ 1.735159] systemd[1]: Listening on udev Kernel Socket.1758worker1 # [ 1.735184] systemd[1]: Reached target Socket Units.1759lb # [ 1.737410] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1760client # [ 1.741808] systemd[1]: Queued start job for default target Initrd Default Target.1761worker1 # [ 1.740066] systemd[1]: Starting Create List of Static Device Nodes...1762worker1 # [ 1.741281] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs1763client # [ 1.750068] systemd[1]: Created slice Slice /system/modprobe.1764client # [ 1.751147] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1765worker1 # [ 1.750192] systemd[1]: Mounting Kernel Configuration File System...1766client # [ 1.752578] systemd[1]: Expecting device /dev/disk/by-label/nixos...1767client # [ 1.753741] systemd[1]: Reached target Path Units.1768client # [ 1.754716] systemd[1]: Reached target Slice Units.1769client # [ 1.755620] systemd[1]: Reached target Swaps.1770lb # [ 1.752159] systemd-journald[72]: Collecting audit messages is disabled.1771client # [ 1.756516] systemd[1]: Reached target Timer Units.1772client # [ 1.757604] systemd[1]: Listening on D-Bus System Message Bus Socket.1773worker2 # [ 1.748502] systemd[1]: Queued start job for default target Initrd Default Target.1774client # [ 1.758956] systemd[1]: Listening on Journal Socket (/dev/log).1775worker1 # [ 1.758310] systemd[1]: Starting Journal Service...1776client # [ 1.760257] systemd[1]: Listening on Journal Sockets.1777client # [ 1.760414] systemd[1]: Listening on udev Control Socket.1778client # [ 1.760538] systemd[1]: Listening on udev Kernel Socket.1779client # [ 1.760563] systemd[1]: Reached target Socket Units.1780worker2 # [ 1.756298] systemd[1]: Created slice Slice /system/modprobe.1781client # [ 1.765987] systemd[1]: Starting Create List of Static Device Nodes...1782worker2 # [ 1.757392] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1783client # [ 1.767237] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs1784worker2 # [ 1.758681] systemd[1]: Expecting device /dev/disk/by-label/nixos...1785worker2 # [ 1.758713] systemd[1]: Reached target Path Units.1786worker2 # [ 1.758732] systemd[1]: Reached target Slice Units.1787worker2 # [ 1.758752] systemd[1]: Reached target Swaps.1788worker2 # [ 1.758771] systemd[1]: Reached target Timer Units.1789worker2 # [ 1.758963] systemd[1]: Listening on D-Bus System Message Bus Socket.1790worker2 # [ 1.759106] systemd[1]: Listening on Journal Socket (/dev/log).1791worker2 # [ 1.759240] systemd[1]: Listening on Journal Sockets.1792worker2 # [ 1.759364] systemd[1]: Listening on udev Control Socket.1793client # [ 1.776170] systemd[1]: Mounting Kernel Configuration File System...1794worker2 # [ 1.759470] systemd[1]: Listening on udev Kernel Socket.1795worker2 # [ 1.759490] systemd[1]: Reached target Socket Units.1796worker2 # [ 1.769477] systemd[1]: Starting Create List of Static Device Nodes...1797worker2 # [ 1.769562] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs1798lb # [ 1.777416] systemd[1]: Mounted Kernel Configuration File System.1799client # [ 1.784294] systemd[1]: Starting Journal Service...1800worker2 # [ 1.775227] systemd[1]: Mounting Kernel Configuration File System...1801lb # [ 1.785345] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1802lb # [ 1.789505] systemd[1]: Starting Create Static Device Nodes in /dev...1803worker2 # [ 1.785267] systemd[1]: Starting Journal Service...1804worker1 # [ 1.794226] systemd[1]: Starting Load Kernel Modules...1805worker1 # [ 1.794360] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1806worker1 # [ 1.797594] systemd[1]: Starting Coldplug All udev Devices...1807lb # [ 1.802767] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1808client # [ 1.807946] systemd[1]: Starting Load Kernel Modules...1809client # [ 1.809065] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1810worker2 # [ 1.801947] systemd[1]: Starting Load Kernel Modules...1811worker2 # [ 1.803115] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1812worker2 # [ 1.809766] systemd[1]: Starting Coldplug All udev Devices...1813lb # [ 1.817164] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1814worker1 # [ 1.819864] systemd-journald[72]: Collecting audit messages is disabled.1815lb # [ 1.830336] [drm] pci: virtio-gpu-pci detected at 0000:00:05.01816lb # [ 1.830573] [drm] features: -virgl +edid -resource_blob -host_visible1817lb # [ 1.830583] [drm] features: -context_init1818lb # [ 1.831306] [drm] number of scanouts: 11819lb # [ 1.831324] [drm] number of cap sets: 01820worker1 # [ 1.830889] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1821worker1 # [ 1.834287] systemd[1]: Finished Create List of Static Device Nodes.1822worker1 # [ 1.834968] systemd[1]: Mounted Kernel Configuration File System.1823worker2 # [ 1.830294] systemd[1]: Finished Create List of Static Device Nodes.1824client # [ 1.840565] systemd[1]: Starting Coldplug All udev Devices...1825worker2 # [ 1.831197] systemd[1]: Mounted Kernel Configuration File System.1826client # [ 1.842924] systemd-journald[72]: Collecting audit messages is disabled.1827client # [ 1.844964] systemd[1]: Finished Create List of Static Device Nodes.1828client # [ 1.845644] systemd[1]: Mounted Kernel Configuration File System.1829worker1 # [ 1.842388] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1830lb # [ 1.849704] systemd[1]: Finished Create Static Device Nodes in /dev.1831worker2 # [ 1.842515] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1832lb # [ 1.849883] systemd[1]: Reached target Preparation for Local File Systems.1833lb # [ 1.849911] systemd[1]: Reached target Local File Systems.1834lb # [ 1.853805] systemd[1]: Starting Rule-based Manager for Device Events and Files...1835client # [ 1.856357] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1836lb # [ 1.861446] virtio-pci 0000:00:05.0: [drm] Registered 1 planes with drm panic1837lb # [ 1.861463] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:05.0 on minor 01838worker1 # [ 1.862436] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1839worker1 # [ 1.868696] [drm] pci: virtio-gpu-pci detected at 0000:00:05.01840worker1 # [ 1.868922] [drm] features: -virgl +edid -resource_blob -host_visible1841worker1 # [ 1.868932] [drm] features: -context_init1842worker1 # [ 1.869674] [drm] number of scanouts: 11843worker1 # [ 1.869692] [drm] number of cap sets: 01844worker2 # [ 1.869862] systemd-journald[72]: Collecting audit messages is disabled.1845worker1 # [ 1.886746] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1846worker1 # [ 1.889205] systemd[1]: Starting Create Static Device Nodes in /dev...1847worker2 # [ 1.882780] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1848worker2 # [ 1.884342] systemd[1]: Starting Create Static Device Nodes in /dev...1849lb # [ 1.885380] Console: switching to colour frame buffer device 160x501850lb # [ 1.891870] virtio-pci 0000:00:05.0: [drm] fb0: virtio_gpudrmfb frame buffer device1851client # [ 1.898970] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1852client # [ 1.899290] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1853client # [ 1.903343] systemd[1]: Starting Create Static Device Nodes in /dev...1854worker1 # [ 1.898468] virtio-pci 0000:00:05.0: [drm] Registered 1 planes with drm panic1855worker1 # [ 1.898487] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:05.0 on minor 01856worker2 # [ 1.900322] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1857client # [ 1.920070] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1858lb # [ 1.917701] systemd[1]: Finished Load Kernel Modules.1859lb # [ 1.925483] systemd[1]: Starting Apply Kernel Variables...1860worker2 # [ 1.914146] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1861worker2 # [ 1.920245] [drm] pci: virtio-gpu-pci detected at 0000:00:05.01862client # [ 1.927518] [drm] pci: virtio-gpu-pci detected at 0000:00:05.01863worker2 # [ 1.920514] [drm] features: -virgl +edid -resource_blob -host_visible1864client # [ 1.927752] [drm] features: -virgl +edid -resource_blob -host_visible1865worker2 # [ 1.920524] [drm] features: -context_init1866client # [ 1.927762] [drm] features: -context_init1867worker2 # [ 1.921200] [drm] number of scanouts: 11868worker2 # [ 1.921218] [drm] number of cap sets: 01869worker1 # [ 1.928925] Console: switching to colour frame buffer device 160x501870worker1 # [ 1.935387] virtio-pci 0000:00:05.0: [drm] fb0: virtio_gpudrmfb frame buffer device1871worker1 # [ 1.942733] systemd[1]: Finished Create Static Device Nodes in /dev.1872worker1 # [ 1.943097] systemd[1]: Reached target Preparation for Local File Systems.1873worker1 # [ 1.943134] systemd[1]: Reached target Local File Systems.1874client # [ 1.948554] systemd[1]: Finished Create Static Device Nodes in /dev.1875client # [ 1.948880] systemd[1]: Reached target Preparation for Local File Systems.1876worker2 # [ 1.938718] systemd[1]: Finished Create Static Device Nodes in /dev.1877client # [ 1.948908] systemd[1]: Reached target Local File Systems.1878worker2 # [ 1.938916] systemd[1]: Reached target Preparation for Local File Systems.1879worker2 # [ 1.938943] systemd[1]: Reached target Local File Systems.1880client # [ 1.952766] systemd[1]: Starting Rule-based Manager for Device Events and Files...1881worker2 # [ 1.940439] systemd[1]: Starting Rule-based Manager for Device Events and Files...1882lb # [ 1.940313] systemd-modules-load[73]: Inserted module 'dm_mod'1883worker1 # [ 1.950400] systemd[1]: Starting Rule-based Manager for Device Events and Files...1884lb # [ 1.941411] systemd-modules-load[73]: Module 'virtio_balloon' is built in1885lb # [ 1.942505] systemd-modules-load[73]: Module 'virtio_console' is built in1886lb # [ 1.943587] systemd-modules-load[73]: Inserted module 'virtio_gpu'1887client # [ 1.960713] [drm] number of scanouts: 11888client # [ 1.960743] [drm] number of cap sets: 01889worker2 # [ 1.950569] virtio-pci 0000:00:05.0: [drm] Registered 1 planes with drm panic1890worker2 # [ 1.950589] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:05.0 on minor 01891lb # [ 1.961319] systemd[1]: Started Journal Service.1892worker1 # [ 1.966629] systemd[1]: Finished Load Kernel Modules.1893client # [ 1.964388] virtio-pci 0000:00:05.0: [drm] Registered 1 planes with drm panic1894client # [ 1.964402] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:05.0 on minor 01895lb # [ 1.964415] systemd-modules-load[73]: Module 'virtio_rng' is built in1896lb # [ 1.965743] systemd-udevd[79]: Using default interface naming scheme 'v261'.1897worker1 # [ 1.978538] systemd[1]: Starting Apply Kernel Variables...1898lb # [ 1.972894] systemd[1]: Starting Create System Files and Directories...1899client # [ 1.984453] Console: switching to colour frame buffer device 160x501900worker2 # [ 1.978386] Console: switching to colour frame buffer device 160x501901worker2 # [ 1.982588] virtio-pci 0000:00:05.0: [drm] fb0: virtio_gpudrmfb frame buffer device1902lb # [ 1.984697] systemd[1]: Finished Apply Kernel Variables.1903worker1 # [ 1.996429] systemd-modules-load[73]: Inserted module 'dm_mod'1904lb # [ 1.997755] systemd[1]: Finished Create System Files and Directories.1905worker1 # [ 2.011565] systemd[1]: Started Journal Service.1906worker2 # [ 2.006640] systemd[1]: Finished Load Kernel Modules.1907worker1 # [ 2.000729] systemd-modules-load[73]: Module 'virtio_balloon' is built in1908worker1 # [ 2.001850] systemd-modules-load[73]: Module 'virtio_console' is built in1909worker1 # [ 2.002924] systemd-modules-load[73]: Inserted module 'virtio_gpu'1910client # [ 2.019670] virtio-pci 0000:00:05.0: [drm] fb0: virtio_gpudrmfb frame buffer device1911worker1 # [ 2.003947] systemd-modules-load[73]: Module 'virtio_rng' is built in1912worker2 # [ 2.010747] systemd[1]: Starting Apply Kernel Variables...1913client # [ 2.025303] systemd[1]: Finished Load Kernel Modules.1914lb # [ 2.016952] systemd[1]: Started Rule-based Manager for Device Events and Files.1915client # [ 2.036699] systemd[1]: Starting Apply Kernel Variables...1916client # [ 2.036879] systemd[1]: Started Journal Service.1917worker1 # [ 2.024454] systemd[1]: Starting Create System Files and Directories...1918client # [ 2.023940] systemd-modules-load[74]: Inserted module 'dm_mod'1919client # [ 2.041007] systemd-modules-load[74]: Module 'virtio_balloon' is built in1920client # [ 2.042143] systemd-modules-load[74]: Module 'virtio_console' is built in1921client # [ 2.043226] systemd-modules-load[74]: Inserted module 'virtio_gpu'1922worker1 # [ 2.040452] systemd-udevd[79]: Using default interface naming scheme 'v261'.1923worker1 # [ 2.042935] systemd[1]: Finished Apply Kernel Variables.1924client # [ 2.056183] systemd-modules-load[74]: Module 'virtio_rng' is built in1925worker1 # [ 2.053467] systemd[1]: Finished Create System Files and Directories.1926client # [ 2.060570] systemd-udevd[79]: Using default interface naming scheme 'v261'.1927worker2 # [ 2.062073] systemd[1]: Started Journal Service.1928worker2 # [ 2.052422] systemd-modules-load[73]: Inserted module 'dm_mod'1929worker2 # [ 2.056413] systemd-modules-load[73]: Module 'virtio_balloon' is built in1930client # [ 2.069250] systemd[1]: Starting Create System Files and Directories...1931worker2 # [ 2.060413] systemd-modules-load[73]: Module 'virtio_console' is built in1932worker2 # [ 2.064268] systemd-modules-load[73]: Inserted module 'virtio_gpu'1933worker2 # [ 2.067457] systemd-modules-load[73]: Module 'virtio_rng' is built in1934client # [ 2.079869] systemd[1]: Finished Apply Kernel Variables.1935lb # [ 2.076509] systemd[1]: Starting Virtual Console Setup...1936worker1 # [ 2.078270] systemd[1]: Started Rule-based Manager for Device Events and Files.1937worker2 # [ 2.072613] systemd-udevd[78]: Using default interface naming scheme 'v261'.1938worker2 # [ 2.073715] systemd[1]: Starting Create System Files and Directories...1939worker2 # [ 2.074787] systemd[1]: Finished Apply Kernel Variables.1940client # [ 2.109077] systemd[1]: Finished Create System Files and Directories.1941client # [ 2.116732] systemd[1]: Started Rule-based Manager for Device Events and Files.1942worker2 # [ 2.105975] systemd[1]: Started Rule-based Manager for Device Events and Files.1943worker2 # [ 2.107957] systemd[1]: Finished Create System Files and Directories.1944lb # [ 2.124517] systemd-vconsole-setup[101]: Configuration of first virtual console was skipped, ignoring remaining ones.1945lb # [ 2.127934] systemd[1]: Finished Virtual Console Setup.1946worker1 # [ 2.129029] systemd[1]: Starting Virtual Console Setup...1947client # [ 2.176257] systemd[1]: Starting Virtual Console Setup...1948worker2 # [ 2.163835] systemd[1]: Starting Virtual Console Setup...1949worker1 # [ 2.184516] systemd-vconsole-setup[101]: Configuration of first virtual console was skipped, ignoring remaining ones.1950worker1 # [ 2.187988] systemd[1]: Finished Virtual Console Setup.1951worker2 # [ 2.212472] systemd-vconsole-setup[102]: Configuration of first virtual console was skipped, ignoring remaining ones.1952client # [ 2.224478] systemd-vconsole-setup[102]: Configuration of first virtual console was skipped, ignoring remaining ones.1953worker2 # [ 2.215930] systemd[1]: Finished Virtual Console Setup.1954client # [ 2.227951] systemd[1]: Finished Virtual Console Setup.1955lb # [ 2.748942] systemd[1]: Finished Coldplug All udev Devices.1956lb # [ 2.749914] systemd[1]: Reached target System Initialization.1957lb # [ 2.752096] systemd[1]: Reached target Basic System.1958worker1 # [ 2.798654] systemd[1]: Finished Coldplug All udev Devices.1959worker1 # [ 2.799823] systemd[1]: Reached target System Initialization.1960worker1 # [ 2.804174] systemd[1]: Reached target Basic System.1961worker2 # [ 2.803919] systemd[1]: Finished Coldplug All udev Devices.1962worker2 # [ 2.804997] systemd[1]: Reached target System Initialization.1963worker2 # [ 2.808097] systemd[1]: Reached target Basic System.1964client # [ 2.864941] systemd[1]: Finished Coldplug All udev Devices.1965client # [ 2.865867] systemd[1]: Reached target System Initialization.1966client # [ 2.866674] systemd[1]: Reached target Basic System.1967lb # [ 2.883978] (udev-worker)[93]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name.1968lb # [ 2.926282] (udev-worker)[93]: Network interface NamePolicy= disabled on kernel command line.1969lb # [ 2.929874] (udev-worker)[91]: Network interface NamePolicy= disabled on kernel command line.1970worker1 # [ 2.931642] (udev-worker)[96]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name.1971worker2 # [ 2.959701] (udev-worker)[100]: Network interface NamePolicy= disabled on kernel command line.1972worker1 # [ 2.975435] (udev-worker)[100]: Network interface NamePolicy= disabled on kernel command line.1973worker1 # [ 2.978758] (udev-worker)[96]: Network interface NamePolicy= disabled on kernel command line.1974lb # [ 2.988571] systemd[1]: Found device /dev/disk/by-label/nixos.1975lb # [ 2.991962] systemd[1]: Reached target Initrd Root Device.1976worker2 # [ 2.988565] (udev-worker)[93]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name.1977lb # [ 2.996814] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos...1978client # [ 3.006537] (udev-worker)[98]: Network interface NamePolicy= disabled on kernel command line.1979worker2 # [ 2.996200] (udev-worker)[93]: Network interface NamePolicy= disabled on kernel command line.1980client # [ 3.041934] (udev-worker)[93]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name.1981client # [ 3.048742] (udev-worker)[93]: Network interface NamePolicy= disabled on kernel command line.1982lb # [ 3.045317] systemd-fsck[110]: nixos: clean, 12/65536 files, 13019/262144 blocks1983lb # [ 3.052112] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos.1984lb # [ 3.056772] systemd[1]: Mounting /sysroot...1985worker2 # [ 3.052473] systemd[1]: Found device /dev/disk/by-label/nixos.1986worker2 # [ 3.054580] systemd[1]: Reached target Initrd Root Device.1987worker2 # [ 3.058728] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos...1988worker1 # [ 3.067582] systemd[1]: Found device /dev/disk/by-label/nixos.1989worker1 # [ 3.070698] systemd[1]: Reached target Initrd Root Device.1990worker1 # [ 3.074665] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos...1991lb # [ 3.105696] EXT4-fs (vda): mounted filesystem 110a27e5-9947-4241-82fa-e97c2e014892 r/w with ordered data mode. Quota mode: none.1992lb # [ 3.098632] systemd[1]: Mounted /sysroot.1993lb # [ 3.100400] systemd[1]: Reached target Initrd Root File System.1994client # [ 3.110034] systemd[1]: Found device /dev/disk/by-label/nixos.1995lb # [ 3.108116] systemd[1]: Starting Mountpoints Configured in the Real Root...1996client # [ 3.114212] systemd[1]: Reached target Initrd Root Device.1997worker2 # [ 3.105627] systemd-fsck[111]: nixos: clean, 12/65536 files, 13019/262144 blocks1998client # [ 3.119097] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos...1999worker2 # [ 3.111348] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos.2000worker1 # [ 3.122517] systemd-fsck[110]: nixos: clean, 12/65536 files, 13019/262144 blocks
loading 1,779 hidden lines…
3780worker1 # [ 25.687604] nix-grpc-daemon[768]: ts=2026-09-21T14:29:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3781worker2 # [ 25.754485] nix-grpc-daemon[770]: ts=2026-09-21T14:29:49Z level=info event=scheduler_connected addr=lb:500513782worker2 # [ 25.759637] nix-grpc-daemon[770]: ts=2026-09-21T14:29:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3783worker1 # [ 25.888990] nix-grpc-daemon[768]: ts=2026-09-21T14:29:49Z level=info event=scheduler_connected addr=lb:500513784worker1 # [ 25.894435] nix-grpc-daemon[768]: ts=2026-09-21T14:29:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3785worker2 # [ 25.961619] nix-grpc-daemon[770]: ts=2026-09-21T14:29:49Z level=info event=scheduler_connected addr=lb:500513786worker2 # [ 25.967298] nix-grpc-daemon[770]: ts=2026-09-21T14:29:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3787worker1 # [ 26.096723] nix-grpc-daemon[768]: ts=2026-09-21T14:29:49Z level=info event=scheduler_connected addr=lb:500513788worker1 # [ 26.102263] nix-grpc-daemon[768]: ts=2026-09-21T14:29:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3789worker2 # [ 26.168915] nix-grpc-daemon[770]: ts=2026-09-21T14:29:49Z level=info event=scheduler_connected addr=lb:500513790worker2 # [ 26.174838] nix-grpc-daemon[770]: ts=2026-09-21T14:29:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3791worker1 # [ 26.303735] nix-grpc-daemon[768]: ts=2026-09-21T14:29:49Z level=info event=scheduler_connected addr=lb:500513792worker1 # [ 26.309995] nix-grpc-daemon[768]: ts=2026-09-21T14:29:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3793lb: must succeed: curl -sf localhost:9901/clusters3794worker2 # [ 26.376738] nix-grpc-daemon[770]: ts=2026-09-21T14:29:49Z level=info event=scheduler_connected addr=lb:500513795worker2 # [ 26.382137] nix-grpc-daemon[770]: ts=2026-09-21T14:29:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3796lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.10 seconds)3797worker1 # [ 26.513718] nix-grpc-daemon[768]: ts=2026-09-21T14:29:50Z level=info event=scheduler_connected addr=lb:500513798worker1 # [ 26.518379] nix-grpc-daemon[768]: ts=2026-09-21T14:29:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3799worker2 # [ 26.586487] nix-grpc-daemon[770]: ts=2026-09-21T14:29:50Z level=info event=scheduler_connected addr=lb:500513800worker2 # [ 26.590953] nix-grpc-daemon[770]: ts=2026-09-21T14:29:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3801worker1 # [ 26.719838] nix-grpc-daemon[768]: ts=2026-09-21T14:29:50Z level=info event=scheduler_connected addr=lb:500513802worker1 # [ 26.726489] nix-grpc-daemon[768]: ts=2026-09-21T14:29:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3803worker2 # [ 26.787565] nix-grpc-daemon[770]: ts=2026-09-21T14:29:50Z level=info event=scheduler_connected addr=lb:500513804worker2 # [ 26.793968] nix-grpc-daemon[770]: ts=2026-09-21T14:29:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3805worker1 # [ 26.927565] nix-grpc-daemon[768]: ts=2026-09-21T14:29:50Z level=info event=scheduler_connected addr=lb:500513806worker1 # [ 26.933492] nix-grpc-daemon[768]: ts=2026-09-21T14:29:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3807worker2 # [ 26.995734] nix-grpc-daemon[770]: ts=2026-09-21T14:29:50Z level=info event=scheduler_connected addr=lb:500513808worker2 # [ 27.001273] nix-grpc-daemon[770]: ts=2026-09-21T14:29:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3809worker1 # [ 27.135644] nix-grpc-daemon[768]: ts=2026-09-21T14:29:50Z level=info event=scheduler_connected addr=lb:500513810worker1 # [ 27.141512] nix-grpc-daemon[768]: ts=2026-09-21T14:29:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3811worker2 # [ 27.203325] nix-grpc-daemon[770]: ts=2026-09-21T14:29:50Z level=info event=scheduler_connected addr=lb:500513812worker2 # [ 27.208883] nix-grpc-daemon[770]: ts=2026-09-21T14:29:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3813worker1 # [ 27.338978] nix-grpc-daemon[768]: ts=2026-09-21T14:29:50Z level=info event=scheduler_connected addr=lb:500513814worker1 # [ 27.344693] nix-grpc-daemon[768]: ts=2026-09-21T14:29:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3815worker2 # [ 27.407808] nix-grpc-daemon[770]: ts=2026-09-21T14:29:51Z level=info event=scheduler_connected addr=lb:500513816worker2 # [ 27.412219] nix-grpc-daemon[770]: ts=2026-09-21T14:29:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3817lb: must succeed: curl -sf localhost:9901/clusters3818lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)3819worker1 # [ 27.549020] nix-grpc-daemon[768]: ts=2026-09-21T14:29:51Z level=info event=scheduler_connected addr=lb:500513820worker1 # [ 27.553457] nix-grpc-daemon[768]: ts=2026-09-21T14:29:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3821worker2 # [ 27.613634] nix-grpc-daemon[770]: ts=2026-09-21T14:29:51Z level=info event=scheduler_connected addr=lb:500513822worker2 # [ 27.618519] nix-grpc-daemon[770]: ts=2026-09-21T14:29:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3823worker1 # [ 27.750116] nix-grpc-daemon[768]: ts=2026-09-21T14:29:51Z level=info event=scheduler_connected addr=lb:500513824worker1 # [ 27.756206] nix-grpc-daemon[768]: ts=2026-09-21T14:29:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3825worker2 # [ 27.814029] nix-grpc-daemon[770]: ts=2026-09-21T14:29:51Z level=info event=scheduler_connected addr=lb:500513826worker2 # [ 27.819924] nix-grpc-daemon[770]: ts=2026-09-21T14:29:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3827worker1 # [ 27.957769] nix-grpc-daemon[768]: ts=2026-09-21T14:29:51Z level=info event=scheduler_connected addr=lb:500513828worker1 # [ 27.964191] nix-grpc-daemon[768]: ts=2026-09-21T14:29:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3829worker2 # [ 28.021771] nix-grpc-daemon[770]: ts=2026-09-21T14:29:51Z level=info event=scheduler_connected addr=lb:500513830worker2 # [ 28.027220] nix-grpc-daemon[770]: ts=2026-09-21T14:29:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3831worker1 # [ 28.165787] nix-grpc-daemon[768]: ts=2026-09-21T14:29:51Z level=info event=scheduler_connected addr=lb:500513832worker1 # [ 28.171633] nix-grpc-daemon[768]: ts=2026-09-21T14:29:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3833worker2 # [ 28.229394] nix-grpc-daemon[770]: ts=2026-09-21T14:29:51Z level=info event=scheduler_connected addr=lb:500513834worker2 # [ 28.235152] nix-grpc-daemon[770]: ts=2026-09-21T14:29:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3835worker1 # [ 28.373040] nix-grpc-daemon[768]: ts=2026-09-21T14:29:51Z level=info event=scheduler_connected addr=lb:500513836worker1 # [ 28.378782] nix-grpc-daemon[768]: ts=2026-09-21T14:29:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3837worker2 # [ 28.436768] nix-grpc-daemon[770]: ts=2026-09-21T14:29:52Z level=info event=scheduler_connected addr=lb:500513838worker2 # [ 28.442472] nix-grpc-daemon[770]: ts=2026-09-21T14:29:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3839lb: must succeed: curl -sf localhost:9901/clusters3840worker1 # [ 28.583231] nix-grpc-daemon[768]: ts=2026-09-21T14:29:52Z level=info event=scheduler_connected addr=lb:500513841worker1 # [ 28.587560] nix-grpc-daemon[768]: ts=2026-09-21T14:29:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3842lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds)3843worker2 # [ 28.643525] nix-grpc-daemon[770]: ts=2026-09-21T14:29:52Z level=info event=scheduler_connected addr=lb:500513844worker2 # [ 28.647880] nix-grpc-daemon[770]: ts=2026-09-21T14:29:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3845worker1 # [ 28.784357] nix-grpc-daemon[768]: ts=2026-09-21T14:29:52Z level=info event=scheduler_connected addr=lb:500513846worker1 # [ 28.789882] nix-grpc-daemon[768]: ts=2026-09-21T14:29:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3847worker2 # [ 28.844215] nix-grpc-daemon[770]: ts=2026-09-21T14:29:52Z level=info event=scheduler_connected addr=lb:500513848worker2 # [ 28.849773] nix-grpc-daemon[770]: ts=2026-09-21T14:29:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3849worker1 # [ 28.990883] nix-grpc-daemon[768]: ts=2026-09-21T14:29:52Z level=info event=scheduler_connected addr=lb:500513850worker1 # [ 28.996483] nix-grpc-daemon[768]: ts=2026-09-21T14:29:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3851worker2 # [ 29.051713] nix-grpc-daemon[770]: ts=2026-09-21T14:29:52Z level=info event=scheduler_connected addr=lb:500513852worker2 # [ 29.058187] nix-grpc-daemon[770]: ts=2026-09-21T14:29:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3853worker1 # [ 29.197952] nix-grpc-daemon[768]: ts=2026-09-21T14:29:52Z level=info event=scheduler_connected addr=lb:500513854worker1 # [ 29.204433] nix-grpc-daemon[768]: ts=2026-09-21T14:29:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3855worker2 # [ 29.260549] nix-grpc-daemon[770]: ts=2026-09-21T14:29:52Z level=info event=scheduler_connected addr=lb:500513856worker2 # [ 29.266157] nix-grpc-daemon[770]: ts=2026-09-21T14:29:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3857worker1 # [ 29.405431] nix-grpc-daemon[768]: ts=2026-09-21T14:29:53Z level=info event=scheduler_connected addr=lb:500513858worker1 # [ 29.411280] nix-grpc-daemon[768]: ts=2026-09-21T14:29:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3859worker2 # [ 29.467429] nix-grpc-daemon[770]: ts=2026-09-21T14:29:53Z level=info event=scheduler_connected addr=lb:500513860worker2 # [ 29.473445] nix-grpc-daemon[770]: ts=2026-09-21T14:29:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3861lb: must succeed: curl -sf localhost:9901/clusters3862worker1 # [ 29.615849] nix-grpc-daemon[768]: ts=2026-09-21T14:29:53Z level=info event=scheduler_connected addr=lb:500513863worker1 # [ 29.620285] nix-grpc-daemon[768]: ts=2026-09-21T14:29:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3864worker2 # [ 29.679240] nix-grpc-daemon[770]: ts=2026-09-21T14:29:53Z level=info event=scheduler_connected addr=lb:500513865worker2 # [ 29.683833] nix-grpc-daemon[770]: ts=2026-09-21T14:29:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3866lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.10 seconds)3867worker1 # [ 29.817476] nix-grpc-daemon[768]: ts=2026-09-21T14:29:53Z level=info event=scheduler_connected addr=lb:500513868worker1 # [ 29.823706] nix-grpc-daemon[768]: ts=2026-09-21T14:29:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3869worker2 # [ 29.879830] nix-grpc-daemon[770]: ts=2026-09-21T14:29:53Z level=info event=scheduler_connected addr=lb:500513870worker2 # [ 29.885607] nix-grpc-daemon[770]: ts=2026-09-21T14:29:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3871worker1 # [ 30.025142] nix-grpc-daemon[768]: ts=2026-09-21T14:29:53Z level=info event=scheduler_connected addr=lb:500513872worker1 # [ 30.031369] nix-grpc-daemon[768]: ts=2026-09-21T14:29:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3873worker2 # [ 30.087280] nix-grpc-daemon[770]: ts=2026-09-21T14:29:53Z level=info event=scheduler_connected addr=lb:500513874worker2 # [ 30.092753] nix-grpc-daemon[770]: ts=2026-09-21T14:29:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3875worker1 # [ 30.233087] nix-grpc-daemon[768]: ts=2026-09-21T14:29:53Z level=info event=scheduler_connected addr=lb:500513876worker1 # [ 30.239910] nix-grpc-daemon[768]: ts=2026-09-21T14:29:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3877worker2 # [ 30.294677] nix-grpc-daemon[770]: ts=2026-09-21T14:29:53Z level=info event=scheduler_connected addr=lb:500513878worker2 # [ 30.300610] nix-grpc-daemon[770]: ts=2026-09-21T14:29:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3879worker1 # [ 30.439944] nix-grpc-daemon[768]: ts=2026-09-21T14:29:54Z level=info event=scheduler_connected addr=lb:500513880worker1 # [ 30.445697] nix-grpc-daemon[768]: ts=2026-09-21T14:29:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3881worker2 # [ 30.502380] nix-grpc-daemon[770]: ts=2026-09-21T14:29:54Z level=info event=scheduler_connected addr=lb:500513882worker2 # [ 30.508233] nix-grpc-daemon[770]: ts=2026-09-21T14:29:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3883worker1 # [ 30.650069] nix-grpc-daemon[768]: ts=2026-09-21T14:29:54Z level=info event=scheduler_connected addr=lb:500513884worker1 # [ 30.654380] nix-grpc-daemon[768]: ts=2026-09-21T14:29:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3885lb: must succeed: curl -sf localhost:9901/clusters3886worker2 # [ 30.712521] nix-grpc-daemon[770]: ts=2026-09-21T14:29:54Z level=info event=scheduler_connected addr=lb:500513887worker2 # [ 30.716758] nix-grpc-daemon[770]: ts=2026-09-21T14:29:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3888lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.10 seconds)3889worker1 # [ 30.850630] nix-grpc-daemon[768]: ts=2026-09-21T14:29:54Z level=info event=scheduler_connected addr=lb:500513890worker1 # [ 30.854984] nix-grpc-daemon[768]: ts=2026-09-21T14:29:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3891worker2 # [ 30.914313] nix-grpc-daemon[770]: ts=2026-09-21T14:29:54Z level=info event=scheduler_connected addr=lb:500513892worker2 # [ 30.920158] nix-grpc-daemon[770]: ts=2026-09-21T14:29:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3893worker1 # [ 31.056131] nix-grpc-daemon[768]: ts=2026-09-21T14:29:54Z level=info event=scheduler_connected addr=lb:500513894worker1 # [ 31.061546] nix-grpc-daemon[768]: ts=2026-09-21T14:29:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3895worker2 # [ 31.122086] nix-grpc-daemon[770]: ts=2026-09-21T14:29:54Z level=info event=scheduler_connected addr=lb:500513896worker2 # [ 31.126842] nix-grpc-daemon[770]: ts=2026-09-21T14:29:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3897worker1 # [ 31.262712] nix-grpc-daemon[768]: ts=2026-09-21T14:29:54Z level=info event=scheduler_connected addr=lb:500513898worker1 # [ 31.270229] nix-grpc-daemon[768]: ts=2026-09-21T14:29:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3899worker2 # [ 31.330339] nix-grpc-daemon[770]: ts=2026-09-21T14:29:54Z level=info event=scheduler_connected addr=lb:500513900worker2 # [ 31.334714] nix-grpc-daemon[770]: ts=2026-09-21T14:29:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3901worker1 # [ 31.471837] nix-grpc-daemon[768]: ts=2026-09-21T14:29:55Z level=info event=scheduler_connected addr=lb:500513902worker1 # [ 31.477589] nix-grpc-daemon[768]: ts=2026-09-21T14:29:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3903worker2 # [ 31.532129] nix-grpc-daemon[770]: ts=2026-09-21T14:29:55Z level=info event=scheduler_connected addr=lb:500513904worker2 # [ 31.537665] nix-grpc-daemon[770]: ts=2026-09-21T14:29:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3905worker1 # [ 31.681321] nix-grpc-daemon[768]: ts=2026-09-21T14:29:55Z level=info event=scheduler_connected addr=lb:500513906worker1 # [ 31.685615] nix-grpc-daemon[768]: ts=2026-09-21T14:29:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3907worker2 # [ 31.743016] nix-grpc-daemon[770]: ts=2026-09-21T14:29:55Z level=info event=scheduler_connected addr=lb:500513908worker2 # [ 31.747380] nix-grpc-daemon[770]: ts=2026-09-21T14:29:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3909lb: must succeed: curl -sf localhost:9901/clusters3910worker1 # [ 31.882250] nix-grpc-daemon[768]: ts=2026-09-21T14:29:55Z level=info event=scheduler_connected addr=lb:500513911worker1 # [ 31.888574] nix-grpc-daemon[768]: ts=2026-09-21T14:29:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3912lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.10 seconds)3913worker2 # [ 31.943522] nix-grpc-daemon[770]: ts=2026-09-21T14:29:55Z level=info event=scheduler_connected addr=lb:500513914worker2 # [ 31.949388] nix-grpc-daemon[770]: ts=2026-09-21T14:29:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3915worker1 # [ 32.090350] nix-grpc-daemon[768]: ts=2026-09-21T14:29:55Z level=info event=scheduler_connected addr=lb:500513916worker1 # [ 32.096000] nix-grpc-daemon[768]: ts=2026-09-21T14:29:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3917worker2 # [ 32.151211] nix-grpc-daemon[770]: ts=2026-09-21T14:29:55Z level=info event=scheduler_connected addr=lb:500513918worker2 # [ 32.157087] nix-grpc-daemon[770]: ts=2026-09-21T14:29:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3919worker1 # [ 32.296714] nix-grpc-daemon[768]: ts=2026-09-21T14:29:55Z level=info event=scheduler_connected addr=lb:500513920worker1 # [ 32.301810] nix-grpc-daemon[768]: ts=2026-09-21T14:29:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3921worker2 # [ 32.358055] nix-grpc-daemon[770]: ts=2026-09-21T14:29:55Z level=info event=scheduler_connected addr=lb:500513922worker2 # [ 32.363271] nix-grpc-daemon[770]: ts=2026-09-21T14:29:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3923worker1 # [ 32.502531] nix-grpc-daemon[768]: ts=2026-09-21T14:29:56Z level=info event=scheduler_connected addr=lb:500513924worker1 # [ 32.507800] nix-grpc-daemon[768]: ts=2026-09-21T14:29:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3925worker2 # [ 32.565341] nix-grpc-daemon[770]: ts=2026-09-21T14:29:56Z level=info event=scheduler_connected addr=lb:500513926worker2 # [ 32.570630] nix-grpc-daemon[770]: ts=2026-09-21T14:29:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3927worker1 # [ 32.710719] nix-grpc-daemon[768]: ts=2026-09-21T14:29:56Z level=info event=scheduler_connected addr=lb:500513928worker1 # [ 32.714875] nix-grpc-daemon[768]: ts=2026-09-21T14:29:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3929worker2 # [ 32.773427] nix-grpc-daemon[770]: ts=2026-09-21T14:29:56Z level=info event=scheduler_connected addr=lb:500513930worker2 # [ 32.777755] nix-grpc-daemon[770]: ts=2026-09-21T14:29:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3931lb: must succeed: curl -sf localhost:9901/clusters3932worker1 # [ 32.911172] nix-grpc-daemon[768]: ts=2026-09-21T14:29:56Z level=info event=scheduler_connected addr=lb:500513933worker1 # [ 32.916556] nix-grpc-daemon[768]: ts=2026-09-21T14:29:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3934lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)3935worker2 # [ 32.974228] nix-grpc-daemon[770]: ts=2026-09-21T14:29:56Z level=info event=scheduler_connected addr=lb:500513936worker2 # [ 32.979497] nix-grpc-daemon[770]: ts=2026-09-21T14:29:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3937worker1 # [ 33.117194] nix-grpc-daemon[768]: ts=2026-09-21T14:29:56Z level=info event=scheduler_connected addr=lb:500513938worker1 # [ 33.122232] nix-grpc-daemon[768]: ts=2026-09-21T14:29:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3939worker2 # [ 33.180701] nix-grpc-daemon[770]: ts=2026-09-21T14:29:56Z level=info event=scheduler_connected addr=lb:500513940worker2 # [ 33.185828] nix-grpc-daemon[770]: ts=2026-09-21T14:29:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3941worker1 # [ 33.323341] nix-grpc-daemon[768]: ts=2026-09-21T14:29:56Z level=info event=scheduler_connected addr=lb:500513942worker1 # [ 33.328775] nix-grpc-daemon[768]: ts=2026-09-21T14:29:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3943worker2 # [ 33.387059] nix-grpc-daemon[770]: ts=2026-09-21T14:29:57Z level=info event=scheduler_connected addr=lb:500513944worker2 # [ 33.392634] nix-grpc-daemon[770]: ts=2026-09-21T14:29:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3945worker1 # [ 33.529524] nix-grpc-daemon[768]: ts=2026-09-21T14:29:57Z level=info event=scheduler_connected addr=lb:500513946worker1 # [ 33.534893] nix-grpc-daemon[768]: ts=2026-09-21T14:29:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3947worker2 # [ 33.593638] nix-grpc-daemon[770]: ts=2026-09-21T14:29:57Z level=info event=scheduler_connected addr=lb:500513948worker2 # [ 33.599069] nix-grpc-daemon[770]: ts=2026-09-21T14:29:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3949worker1 # [ 33.737207] nix-grpc-daemon[768]: ts=2026-09-21T14:29:57Z level=info event=scheduler_connected addr=lb:500513950worker1 # [ 33.742060] nix-grpc-daemon[768]: ts=2026-09-21T14:29:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3951worker2 # [ 33.802678] nix-grpc-daemon[770]: ts=2026-09-21T14:29:57Z level=info event=scheduler_connected addr=lb:500513952worker2 # [ 33.806833] nix-grpc-daemon[770]: ts=2026-09-21T14:29:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3953worker1 # [ 33.919063] nix-grpc-daemon[768]: ts=2026-09-21T14:29:57Z level=info event=scheduler_take_over3954lb # [ 33.919951] niks3-server[975]: 2026/09/21 14:29:57 INFO lead: acquired remote=192.168.1.3:502323955worker1 # [ 33.942872] nix-grpc-daemon[768]: ts=2026-09-21T14:29:57Z level=info event=scheduler_connected addr=lb:500513956worker1 # [ 33.947959] nix-grpc-daemon[768]: ts=2026-09-21T14:29:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3957lb: must succeed: curl -sf localhost:9901/clusters3958worker2 # [ 34.003575] nix-grpc-daemon[770]: ts=2026-09-21T14:29:57Z level=info event=scheduler_connected addr=lb:500513959worker2 # [ 34.009026] nix-grpc-daemon[770]: ts=2026-09-21T14:29:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3960lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)3961worker1 # [ 34.148995] nix-grpc-daemon[768]: ts=2026-09-21T14:29:57Z level=info event=scheduler_connected addr=lb:500513962worker1 # [ 34.153943] nix-grpc-daemon[768]: ts=2026-09-21T14:29:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3963worker2 # [ 34.210108] nix-grpc-daemon[770]: ts=2026-09-21T14:29:57Z level=info event=scheduler_connected addr=lb:500513964worker2 # [ 34.215229] nix-grpc-daemon[770]: ts=2026-09-21T14:29:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3965worker1 # [ 34.354441] nix-grpc-daemon[768]: ts=2026-09-21T14:29:57Z level=info event=scheduler_connected addr=lb:500513966worker1 # [ 34.377282] nix-grpc-daemon[768]: ts=2026-09-21T14:29:57Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:389403967worker1 # [ 34.383463] nix-grpc-daemon[768]: ts=2026-09-21T14:29:57Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=03968worker2 # [ 34.416228] nix-grpc-daemon[770]: ts=2026-09-21T14:29:58Z level=info event=scheduler_connected addr=lb:500513969worker1 # [ 34.437784] nix-grpc-daemon[768]: ts=2026-09-21T14:29:58Z level=info event=oidc_jwks provider=mock keys=13970worker1 # [ 34.442141] nix-grpc-daemon[768]: ts=2026-09-21T14:29:58Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:389403971worker1 # [ 34.447504] nix-grpc-daemon[768]: ts=2026-09-21T14:29:58Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=03972lb: must succeed: curl -sf localhost:9901/clusters3973lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)3974worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics3975worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)3976lb: must succeed: curl -sf localhost:9901/clusters3977lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)3978lb: must succeed: curl -sf localhost:9901/clusters3979lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)3980lb: must succeed: curl -sf localhost:9901/clusters3981lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)3982lb: must succeed: curl -sf localhost:9901/clusters3983lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)3984worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open'3985worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open', in 0.06 seconds)3986worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics3987worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)3988lb: must succeed: curl -sf localhost:9901/clusters3989lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)3990lb: must succeed: curl -sf localhost:9901/clusters3991lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)3992worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics3993worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)3994worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics3995worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)3996worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics3997worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)3998worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics3999worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)4000(finished: subtest: one node schedules and both builders hold a WorkerSession on it, in 10.55 seconds)4001subtest: balancer auth: client cert, bearer token, nothing, unknown CN4002worker2 # [ 36.119624] nix-grpc-daemon[770]: ts=2026-09-21T14:29:59Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:490144003worker2 # [ 36.137604] systemd[1]: Started Nix Daemon.4004worker2 # [ 36.333910] nix-daemon[838]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4005worker2 # [ 36.355072] nix-grpc-daemon[770]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 56 ms (attempt 1/5)4006worker2 # [ 36.414835] nix-grpc-daemon[770]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 6 ms (attempt 2/5)4007worker2 # [ 36.421662] nix-grpc-daemon[770]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 264 ms (attempt 3/5)4008worker2 # [ 36.691618] nix-grpc-daemon[770]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 658 ms (attempt 4/5)4009worker2 # [ 37.353324] nix-grpc-daemon[770]: disabling binary cache 'https://cache.nixos.org' for 60 seconds4010worker2 # [ 37.358512] nix-grpc-daemon[770]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4011worker2 # [ 37.378373] nix-grpc-daemon[770]: ts=2026-09-21T14:30:00Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:490144012worker2 # [ 37.394258] nix-grpc-daemon[770]: ts=2026-09-21T14:30:01Z level=debug event=path_info_miss path=h4vx3h280swjvvjn29az46l9rzjm19b0-nixos-system-client-test error="error: path '/nix/store/h4vx3h280swjvvjn29az46l9rzjm19b0-nixos-system-client-test' is required, but there is no substituter that can build it"4013worker2 # [ 37.407781] nix-grpc-daemon[770]: ts=2026-09-21T14:30:01Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:490144014worker2 # [ 37.415280] nix-grpc-daemon[770]: ts=2026-09-21T14:30:01Z level=debug event=path_info_miss path=h4vx3h280swjvvjn29az46l9rzjm19b0-nixos-system-client-test error="error: path '/nix/store/h4vx3h280swjvvjn29az46l9rzjm19b0-nixos-system-client-test' is required, but there is no substituter that can build it"4015client: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt4016client: (finished: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt, in 0.08 seconds)4017client: must succeed: install -D /nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key4018client: (finished: must succeed: install -D /nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key, in 0.04 seconds)4019worker1 # [ 37.765507] nix-grpc-daemon[768]: ts=2026-09-21T14:30:01Z level=debug event=rpc_start method=QueryMissing cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:389564020worker1 # [ 37.783237] systemd[1]: Started Nix Daemon.4021worker1 # [ 37.995096] nix-daemon[851]: accepted connection from pid 768, user nix-grpc-daemon (trusted)4022worker1 # [ 38.013295] nix-grpc-daemon[768]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 57 ms (attempt 1/5)4023worker1 # [ 38.074236] nix-grpc-daemon[768]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 86 ms (attempt 2/5)4024worker1 # [ 38.161957] nix-grpc-daemon[768]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 388 ms (attempt 3/5)4025worker1 # [ 38.553185] nix-grpc-daemon[768]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 281 ms (attempt 4/5)4026worker1 # [ 38.845382] nix-grpc-daemon[768]: disabling binary cache 'https://cache.nixos.org' for 60 seconds4027worker1 # [ 38.851225] nix-grpc-daemon[768]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4028worker1 # [ 38.868404] nix-grpc-daemon[768]: ts=2026-09-21T14:30:02Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:389564029worker1 # [ 38.883332] nix-grpc-daemon[768]: ts=2026-09-21T14:30:02Z level=debug event=path_info_miss path=h4vx3h280swjvvjn29az46l9rzjm19b0-nixos-system-client-test error="error: path '/nix/store/h4vx3h280swjvvjn29az46l9rzjm19b0-nixos-system-client-test' is required, but there is no substituter that can build it"4030worker1 # [ 38.896356] nix-grpc-daemon[768]: ts=2026-09-21T14:30:02Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:389564031worker1 # [ 38.907604] nix-grpc-daemon[768]: ts=2026-09-21T14:30:02Z level=debug event=path_info_miss path=h4vx3h280swjvvjn29az46l9rzjm19b0-nixos-system-client-test error="error: path '/nix/store/h4vx3h280swjvvjn29az46l9rzjm19b0-nixos-system-client-test' is required, but there is no substituter that can build it"4032client: must succeed: rm -r /var/lib/nix-grpc-store4033client: (finished: must succeed: rm -r /var/lib/nix-grpc-store, in 0.04 seconds)4034lb # [ 39.006731] envoy[909]: conn peer=[2001:db8:1::1]:34896 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=715 tx=365 ms=13274035lb # [ 39.012062] envoy[909]: conn peer=[2001:db8:1::1]:34902 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=2596 tx=365 ms=11744036worker2 # [ 39.161389] nix-grpc-daemon[770]: ts=2026-09-21T14:30:02Z level=info event=denied method=QueryMissing cn=- role=none4037worker2 # [ 39.335228] nix-grpc-daemon[770]: ts=2026-09-21T14:30:02Z level=info event=denied method=QueryMissing cn=stranger role=none4038(finished: subtest: balancer auth: client cert, bearer token, nothing, unknown CN, in 3.52 seconds)4039subtest: DAG build is scheduled across workers and lands in the cache4040client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 >&24041worker1 # [ 39.511134] nix-grpc-daemon[768]: ts=2026-09-21T14:30:03Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:389564042client # don't know how to build these paths:4043client # /nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv4044worker1 # [ 39.522169] nix-grpc-daemon[768]: ts=2026-09-21T14:30:03Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:389564045worker1 # [ 39.563560] nix-grpc-daemon[768]: ts=2026-09-21T14:30:03Z level=debug event=want client=1 drv=619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv cached=04046worker1 # [ 39.569696] nix-grpc-daemon[768]: ts=2026-09-21T14:30:03Z level=debug event=assigned drv=619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv worker=192.168.1.3:50051 clients=14047worker1 # [ 39.624199] nix-grpc-daemon[768]: ts=2026-09-21T14:30:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:389564048worker1 # [ 39.642542] nix-grpc-daemon[768]: ts=2026-09-21T14:30:03Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:389564049client # copying 1 paths to 'lb:50051'...4050worker1 # [ 39.659937] nix-grpc-daemon[768]: ts=2026-09-21T14:30:03Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:389564051client # copying path '/nix/store/619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv' to 'lb:50051'...4052worker1 # [ 39.688667] nix-daemon[851]: accepted connection from pid 768, user nix-grpc-daemon (trusted)4053worker1 # [ 39.730432] nix-grpc-daemon[768]: ts=2026-09-21T14:30:03Z level=info event=niks3_push_started pid=8654054worker1 # [ 39.925419] nix-grpc-daemon[865]: time=2026-09-21T14:30:03.528Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.key ca=""4055worker1 # [ 40.039996] nix-daemon[851]: accepted connection from pid 878, user nix-grpc-daemon (trusted)4056worker1 # [ 40.059978] nix-daemon[851]: reaped child process 881, status = succeeded4057worker1 # [ 40.160296] nix-daemon[851]: accepted connection from pid 883, user nix-grpc-daemon (trusted)4058worker1 # [ 40.176996] nix-daemon[851]: reaped child process 886, status = succeeded4059lb # [ 40.182711] niks3-server[975]: 2026/09/21 14:30:03 INFO Received uploads request method=POST path=/api/pending_closures4060worker1 # [ 40.210226] nix-grpc-daemon[865]: time=2026-09-21T14:30:03.813Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4061worker1 # [ 40.211858] nix-grpc-daemon[865]: time=2026-09-21T14:30:03.813Z level=INFO msg="Uploading 619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv (504B)"4062lb # [ 40.267615] niks3-server[975]: 2026/09/21 14:30:03 INFO Registered completed upload object_key=nar/1il50j4vra11xfnqgjhjgrjn1753frlhxnss6mh1dmp5ww3is34w.nar.zst4063lb # [ 40.281256] niks3-server[975]: 2026/09/21 14:30:03 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign4064worker1 # [ 40.281510] nix-grpc-daemon[865]: time=2026-09-21T14:30:03.885Z level=INFO msg="Uploading 1 narinfos"4065lb # [ 40.291929] niks3-server[975]: 2026/09/21 14:30:03 INFO Signed narinfos id=1 count=14066lb # [ 40.306463] niks3-server[975]: 2026/09/21 14:30:03 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete4067lb # [ 40.327081] niks3-server[975]: 2026/09/21 14:30:03 INFO Registered completed upload object_key=619ccx60pfv52kv4hhgqpj1ivdmfcr5r.ls4068lb # [ 40.340088] niks3-server[975]: 2026/09/21 14:30:03 INFO Completed upload id=14069worker1 # [ 40.341031] nix-grpc-daemon[865]: time=2026-09-21T14:30:03.945Z level=INFO msg="Upload complete. (412ms)"4070worker1 # [ 40.351603] nix-grpc-daemon[865]: time=2026-09-21T14:30:03.950Z level=WARN msg="Failed to register uploaded object" key=619ccx60pfv52kv4hhgqpj1ivdmfcr5r.narinfo error="server returned 404: object is not pending upload\n"4071worker1 # [ 40.372285] nix-grpc-daemon[768]: ts=2026-09-21T14:30:03Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38956 duration_s=0 paths=1 nar_bytes_in=5044072worker1 # [ 40.383000] nix-grpc-daemon[768]: ts=2026-09-21T14:30:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:389564073worker1 # [ 40.396322] nix-daemon[851]: accepted connection from pid 768, user nix-grpc-daemon (trusted)4074worker1 # [ 40.403095] nix-daemon[851]: reaped child process 863, status = succeeded4075client # building '/nix/store/619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv' on lb:50051...4076client # farm-a> node-a: building 619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv4077worker1 # [ 40.411856] nix-daemon[851]: accepted connection from pid 768, user nix-grpc-daemon (trusted)4078client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds4079client # farm-a> LOG-farm-a-t14080worker1 # [ 41.471828] nix-daemon[851]: reaped child process 890, status = succeeded4081worker1 # [ 41.601457] nix-daemon[851]: accepted connection from pid 907, user nix-grpc-daemon (trusted)4082worker1 # [ 41.624562] nix-daemon[851]: reaped child process 910, status = succeeded4083lb # [ 41.623879] niks3-server[975]: 2026/09/21 14:30:05 INFO Received uploads request method=POST path=/api/pending_closures4084worker1 # [ 41.640434] nix-grpc-daemon[865]: time=2026-09-21T14:30:05.244Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4085worker1 # [ 41.642178] nix-grpc-daemon[865]: time=2026-09-21T14:30:05.244Z level=INFO msg="Uploading dg404zlvxfrspnxl6cc1qkc6i83m5a3a-farm-a (128B)"4086lb # [ 41.687157] niks3-server[975]: 2026/09/21 14:30:05 INFO Registered completed upload object_key=nar/1rmil4ja9jzfwn6l5hwbir70b6plx99sgk0pnjjy377ngbc3vz28.nar.zst4087worker1 # [ 41.697731] nix-grpc-daemon[865]: time=2026-09-21T14:30:05.300Z level=INFO msg="Uploading 1 narinfos"4088lb # [ 41.700896] niks3-server[975]: 2026/09/21 14:30:05 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign4089lb # [ 41.703806] niks3-server[975]: 2026/09/21 14:30:05 INFO Signed narinfos id=2 count=14090lb # [ 41.711508] niks3-server[975]: 2026/09/21 14:30:05 INFO Registered completed upload object_key=log/619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv4091lb # [ 41.718745] niks3-server[975]: 2026/09/21 14:30:05 INFO Registered completed upload object_key=dg404zlvxfrspnxl6cc1qkc6i83m5a3a.ls4092worker1 # [ 41.723335] nix-grpc-daemon[865]: time=2026-09-21T14:30:05.327Z level=INFO msg="Upload complete. (298ms)"4093lb # [ 41.723787] niks3-server[975]: 2026/09/21 14:30:05 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete4094lb # [ 41.727159] niks3-server[975]: 2026/09/21 14:30:05 INFO Completed upload id=24095lb # [ 41.730725] niks3-server[975]: 2026/09/21 14:30:05 INFO Registered completed upload object_key=dg404zlvxfrspnxl6cc1qkc6i83m5a3a.narinfo4096worker1 # [ 41.748917] nix-grpc-daemon[768]: ts=2026-09-21T14:30:05Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38956 duration_s=1 drv=619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv assign_id=1 outputs=14097worker1 # [ 41.760404] nix-grpc-daemon[768]: ts=2026-09-21T14:30:05Z level=debug event=want client=1 drv=79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv cached=04098worker1 # [ 41.765609] nix-grpc-daemon[768]: ts=2026-09-21T14:30:05Z level=debug event=assigned drv=79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv worker=192.168.1.3:50051 clients=14099worker1 # [ 41.769588] nix-daemon[851]: reaped child process 888, status = succeeded4100worker1 # [ 41.788960] nix-grpc-daemon[768]: ts=2026-09-21T14:30:05Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:389564101worker1 # [ 41.810019] nix-grpc-daemon[768]: ts=2026-09-21T14:30:05Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:389564102worker1 # [ 41.824679] nix-grpc-daemon[865]: time=2026-09-21T14:30:05.429Z level=INFO msg="All 1 paths already cached"4103client # copying 1 paths to 'lb:50051'...4104worker1 # [ 41.833935] nix-grpc-daemon[768]: ts=2026-09-21T14:30:05Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:389564105client # copying path '/nix/store/79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv' to 'lb:50051'...4106worker1 # [ 41.855672] nix-daemon[851]: accepted connection from pid 768, user nix-grpc-daemon (trusted)4107worker1 # [ 42.013124] nix-daemon[851]: accepted connection from pid 916, user nix-grpc-daemon (trusted)4108worker1 # [ 42.033597] nix-daemon[851]: reaped child process 919, status = succeeded4109worker1 # [ 42.132399] nix-daemon[851]: accepted connection from pid 921, user nix-grpc-daemon (trusted)4110worker1 # [ 42.147812] nix-daemon[851]: reaped child process 924, status = succeeded4111lb # [ 42.153486] niks3-server[975]: 2026/09/21 14:30:05 INFO Received uploads request method=POST path=/api/pending_closures4112worker1 # [ 42.163922] nix-grpc-daemon[865]: time=2026-09-21T14:30:05.768Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4113worker1 # [ 42.165607] nix-grpc-daemon[865]: time=2026-09-21T14:30:05.768Z level=INFO msg="Uploading 79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv (624B)"4114lb # [ 42.182853] niks3-server[975]: 2026/09/21 14:30:05 INFO Registered completed upload object_key=nar/1xk4kbz8l387hfrqq2xi6as79has336h62f2ra95472yqjclksjg.nar.zst4115worker1 # [ 42.196559] nix-grpc-daemon[865]: time=2026-09-21T14:30:05.801Z level=INFO msg="Uploading 1 narinfos"4116lb # [ 42.198186] niks3-server[975]: 2026/09/21 14:30:05 INFO Received sign narinfos request method=POST path=/api/pending_closures/3/sign4117lb # [ 42.210873] niks3-server[975]: 2026/09/21 14:30:05 INFO Signed narinfos id=3 count=14118lb # [ 42.216292] niks3-server[975]: 2026/09/21 14:30:05 INFO Registered completed upload object_key=79z38hfi82zcmb9yq1gcmfviimgf6z2l.ls4119worker1 # [ 42.224384] nix-grpc-daemon[865]: time=2026-09-21T14:30:05.829Z level=INFO msg="Upload complete. (351ms)"4120lb # [ 42.224118] niks3-server[975]: 2026/09/21 14:30:05 INFO Received complete upload request method=POST path=/api/pending_closures/3/complete4121worker1 # [ 42.231662] nix-grpc-daemon[768]: ts=2026-09-21T14:30:05Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38956 duration_s=0 paths=1 nar_bytes_in=6244122lb # [ 42.234248] niks3-server[975]: 2026/09/21 14:30:05 INFO Completed upload id=34123lb # [ 42.236973] niks3-server[975]: 2026/09/21 14:30:05 INFO Registered completed upload object_key=79z38hfi82zcmb9yq1gcmfviimgf6z2l.narinfo4124worker1 # [ 42.244606] nix-grpc-daemon[768]: ts=2026-09-21T14:30:05Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:389564125worker1 # [ 42.252623] nix-daemon[851]: accepted connection from pid 768, user nix-grpc-daemon (trusted)4126worker1 # [ 42.255368] nix-daemon[851]: reaped child process 914, status = succeeded4127client # building '/nix/store/79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv' on lb:50051...4128worker1 # [ 42.265988] nix-daemon[851]: accepted connection from pid 768, user nix-grpc-daemon (trusted)4129client # farm-b> node-a: building 79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv4130client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds4131client # farm-b> LOG-farm-b-t14132worker1 # [ 43.312836] nix-daemon[851]: reaped child process 928, status = succeeded4133worker1 # [ 43.407870] nix-daemon[851]: accepted connection from pid 944, user nix-grpc-daemon (trusted)4134worker1 # [ 43.425684] nix-daemon[851]: reaped child process 947, status = succeeded4135lb # [ 43.432810] niks3-server[975]: 2026/09/21 14:30:07 INFO Received uploads request method=POST path=/api/pending_closures4136worker1 # [ 43.443847] nix-grpc-daemon[865]: time=2026-09-21T14:30:07.047Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4137worker1 # [ 43.445420] nix-grpc-daemon[865]: time=2026-09-21T14:30:07.047Z level=INFO msg="Uploading 8n9g36vw9957r150a7wg1k6yj9fbqm8r-farm-b (176B)"4138lb # [ 43.464812] niks3-server[975]: 2026/09/21 14:30:07 INFO Registered completed upload object_key=nar/1mx8h9irf9ykfzxs6q3lwxbzxwv5d2s9mh2x0sky7448mlc1qxrx.nar.zst4139lb # [ 43.486956] niks3-server[975]: 2026/09/21 14:30:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/4/sign4140worker1 # [ 43.492515] nix-grpc-daemon[865]: time=2026-09-21T14:30:07.096Z level=INFO msg="Uploading 1 narinfos"4141lb # [ 43.496325] niks3-server[975]: 2026/09/21 14:30:07 INFO Signed narinfos id=4 count=14142lb # [ 43.498612] niks3-server[975]: 2026/09/21 14:30:07 INFO Registered completed upload object_key=8n9g36vw9957r150a7wg1k6yj9fbqm8r.ls4143lb # [ 43.511969] niks3-server[975]: 2026/09/21 14:30:07 INFO Received complete upload request method=POST path=/api/pending_closures/4/complete4144lb # [ 43.520973] niks3-server[975]: 2026/09/21 14:30:07 INFO Registered completed upload object_key=log/79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv4145lb # [ 43.524406] niks3-server[975]: 2026/09/21 14:30:07 INFO Registered completed upload object_key=8n9g36vw9957r150a7wg1k6yj9fbqm8r.narinfo4146lb # [ 43.528882] niks3-server[975]: 2026/09/21 14:30:07 INFO Completed upload id=44147worker1 # [ 43.528703] nix-grpc-daemon[865]: time=2026-09-21T14:30:07.133Z level=INFO msg="Upload complete. (251ms)"4148worker1 # [ 43.549082] nix-grpc-daemon[768]: ts=2026-09-21T14:30:07Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38956 duration_s=1 drv=79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv assign_id=2 outputs=14149worker1 # [ 43.560399] nix-grpc-daemon[768]: ts=2026-09-21T14:30:07Z level=debug event=want client=1 drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv cached=04150worker1 # [ 43.568321] nix-grpc-daemon[768]: ts=2026-09-21T14:30:07Z level=debug event=assigned drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv worker=192.168.1.3:50051 clients=14151worker1 # [ 43.573396] nix-daemon[851]: reaped child process 926, status = succeeded4152worker1 # [ 43.588983] nix-grpc-daemon[768]: ts=2026-09-21T14:30:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:389564153worker1 # [ 43.604330] nix-grpc-daemon[768]: ts=2026-09-21T14:30:07Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:389564154worker1 # [ 43.619163] nix-grpc-daemon[865]: time=2026-09-21T14:30:07.223Z level=INFO msg="All 2 paths already cached"4155client # copying 1 paths to 'lb:50051'...4156client # copying path '/nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv' to 'lb:50051'...4157worker1 # [ 43.629389] nix-grpc-daemon[768]: ts=2026-09-21T14:30:07Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:389564158worker1 # [ 43.642958] nix-daemon[851]: accepted connection from pid 768, user nix-grpc-daemon (trusted)4159worker1 # [ 43.792138] nix-daemon[851]: accepted connection from pid 953, user nix-grpc-daemon (trusted)4160worker1 # [ 43.820867] nix-daemon[851]: reaped child process 956, status = succeeded4161worker1 # [ 43.910414] nix-daemon[851]: accepted connection from pid 958, user nix-grpc-daemon (trusted)4162worker1 # [ 43.927671] nix-daemon[851]: reaped child process 961, status = succeeded4163lb # [ 43.933627] niks3-server[975]: 2026/09/21 14:30:07 INFO Received uploads request method=POST path=/api/pending_closures4164worker1 # [ 43.946158] nix-grpc-daemon[865]: time=2026-09-21T14:30:07.550Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4165worker1 # [ 43.947636] nix-grpc-daemon[865]: time=2026-09-21T14:30:07.550Z level=INFO msg="Uploading czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv (752B)"4166lb # [ 43.968797] niks3-server[975]: 2026/09/21 14:30:07 INFO Registered completed upload object_key=nar/02rh0zwkcxgd7rc74r5h676rr0m8dbayk3glb776b2mdlirxh9lv.nar.zst4167worker1 # [ 43.980570] nix-grpc-daemon[865]: time=2026-09-21T14:30:07.585Z level=INFO msg="Uploading 1 narinfos"4168lb # [ 43.984350] niks3-server[975]: 2026/09/21 14:30:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/5/sign4169lb # [ 43.992874] niks3-server[975]: 2026/09/21 14:30:07 INFO Signed narinfos id=5 count=14170lb # [ 44.000457] niks3-server[975]: 2026/09/21 14:30:07 INFO Registered completed upload object_key=czd6ljynjaw3bl3w8iz1nx68whkdis8f.ls4171lb # [ 44.012189] niks3-server[975]: 2026/09/21 14:30:07 INFO Received complete upload request method=POST path=/api/pending_closures/5/complete4172worker1 # [ 44.016118] nix-grpc-daemon[865]: time=2026-09-21T14:30:07.616Z level=INFO msg="Upload complete. (353ms)"4173lb # [ 44.017547] niks3-server[975]: 2026/09/21 14:30:07 INFO Completed upload id=54174worker1 # [ 44.019054] nix-grpc-daemon[768]: ts=2026-09-21T14:30:07Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38956 duration_s=0 paths=1 nar_bytes_in=7524175lb # [ 44.022948] niks3-server[975]: 2026/09/21 14:30:07 INFO Registered completed upload object_key=czd6ljynjaw3bl3w8iz1nx68whkdis8f.narinfo4176worker1 # [ 44.036414] nix-grpc-daemon[768]: ts=2026-09-21T14:30:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:389564177worker1 # [ 44.045288] nix-daemon[851]: accepted connection from pid 768, user nix-grpc-daemon (trusted)4178worker1 # [ 44.051862] nix-daemon[851]: reaped child process 951, status = succeeded4179client # building '/nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv' on lb:50051...4180worker1 # [ 44.070038] nix-daemon[851]: accepted connection from pid 768, user nix-grpc-daemon (trusted)4181client # farm-top> node-a: building czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv4182client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds4183client # farm-top> LOG-farm-top-t14184worker1 # [ 45.137612] nix-daemon[851]: reaped child process 965, status = succeeded4185worker1 # [ 45.262640] nix-daemon[851]: accepted connection from pid 982, user nix-grpc-daemon (trusted)4186worker1 # [ 45.279807] nix-daemon[851]: reaped child process 985, status = succeeded4187lb # [ 45.289176] niks3-server[975]: 2026/09/21 14:30:08 INFO Received uploads request method=POST path=/api/pending_closures4188worker1 # [ 45.306404] nix-grpc-daemon[865]: time=2026-09-21T14:30:08.909Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4189worker1 # [ 45.308505] nix-grpc-daemon[865]: time=2026-09-21T14:30:08.909Z level=INFO msg="Uploading nblg066ygaz2avzlmix3rnngr62hld8f-farm-top (232B)"4190lb # [ 45.333816] niks3-server[975]: 2026/09/21 14:30:08 INFO Registered completed upload object_key=nar/1wplyx8g7zja504cjvqmxis1q6laml9bidgiq4ya0i9p2pvqb9qp.nar.zst4191lb # [ 45.341025] niks3-server[975]: 2026/09/21 14:30:08 INFO Registered completed upload object_key=log/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv4192lb # [ 45.348066] niks3-server[975]: 2026/09/21 14:30:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/6/sign4193lb # [ 45.354291] niks3-server[975]: 2026/09/21 14:30:08 INFO Signed narinfos id=6 count=14194worker1 # [ 45.353264] nix-grpc-daemon[865]: time=2026-09-21T14:30:08.957Z level=INFO msg="Uploading 1 narinfos"4195lb # [ 45.358442] niks3-server[975]: 2026/09/21 14:30:08 INFO Registered completed upload object_key=nblg066ygaz2avzlmix3rnngr62hld8f.ls4196lb # [ 45.370017] niks3-server[975]: 2026/09/21 14:30:08 INFO Received complete upload request method=POST path=/api/pending_closures/6/complete4197lb # [ 45.377208] niks3-server[975]: 2026/09/21 14:30:08 INFO Completed upload id=64198worker1 # [ 45.376552] nix-grpc-daemon[865]: time=2026-09-21T14:30:08.981Z level=INFO msg="Upload complete. (291ms)"4199lb # [ 45.378739] niks3-server[975]: 2026/09/21 14:30:08 INFO Registered completed upload object_key=nblg066ygaz2avzlmix3rnngr62hld8f.narinfo4200worker1 # [ 45.389152] nix-grpc-daemon[768]: ts=2026-09-21T14:30:08Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38956 duration_s=1 drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv assign_id=3 outputs=14201worker1 # [ 45.408574] nix-daemon[851]: reaped child process 963, status = succeeded4202client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 >&2, in 6.08 seconds)4203client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 outPath4204client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 outPath, in 0.18 seconds)4205client: must fail: test -e /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top4206client: (finished: must fail: test -e /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.02 seconds)4207client: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top && grep farm-top-t1 /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top4208client # copying 3 paths...4209client # copying path '/nix/store/dg404zlvxfrspnxl6cc1qkc6i83m5a3a-farm-a' from 'http://lb:5751'...4210client # copying path '/nix/store/8n9g36vw9957r150a7wg1k6yj9fbqm8r-farm-b' from 'http://lb:5751'...4211client # copying path '/nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top' from 'http://lb:5751'...4212client: (finished: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top && grep farm-top-t1 /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.23 seconds)4213(finished: subtest: DAG build is scheduled across workers and lands in the cache, in 6.60 seconds)4214subtest: repeat is answered Cached by the scheduler without building4215worker1: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top4216worker1 # finding garbage collector roots...4217worker1 # deleting specified paths...4218worker1 # deleting '/nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top'4219worker1 # deleting unused links...4220worker1 # note: hard linking is currently saving 0.0 KiB4221worker1: (finished: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.22 seconds)4222worker2: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top4223worker2 # finding garbage collector roots...4224worker2 # deleting specified paths...4225worker2 # deleting '/nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top'4226worker2 # deleting unused links...4227worker2 # note: hard linking is currently saving 0.0 KiB4228worker2: (finished: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.23 seconds)4229client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 >&24230worker1 # [ 46.632655] nix-grpc-daemon[768]: ts=2026-09-21T14:30:10Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:389564231client # this path will be fetched (0.0 KiB download, 0.2 KiB unpacked):4232client # /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top4233worker1 # [ 46.649618] nix-grpc-daemon[768]: ts=2026-09-21T14:30:10Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:389564234worker1 # [ 46.660682] nix-grpc-daemon[865]: time=2026-09-21T14:30:10.264Z level=INFO msg="All 2 paths already cached"4235worker1 # [ 46.677076] nix-grpc-daemon[768]: ts=2026-09-21T14:30:10Z level=debug event=want client=2 drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv cached=14236client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 >&2, in 0.30 seconds)4237client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 outPath4238client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 outPath, in 0.15 seconds)4239lb: must succeed: curl -sf localhost:9901/clusters4240lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)4241lb: must succeed: curl -sf localhost:9901/clusters4242lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4243worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4244worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)4245(finished: subtest: repeat is answered Cached by the scheduler without building, in 1.18 seconds)4246subtest: inputs already in the cache are not sent to the scheduler4247lb: must succeed: curl -sf localhost:9901/clusters4248lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)4249lb: must succeed: curl -sf localhost:9901/clusters4250lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4251worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4252worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.06 seconds)4253client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 --argstr top t1b >&24254client # don't know how to build these paths:4255client # /nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv4256worker1 # [ 47.521763] nix-grpc-daemon[768]: ts=2026-09-21T14:30:11Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:389564257worker1 # [ 47.529188] nix-grpc-daemon[768]: ts=2026-09-21T14:30:11Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:389564258worker1 # [ 47.535918] nix-grpc-daemon[865]: time=2026-09-21T14:30:11.139Z level=INFO msg="All 2 paths already cached"4259worker1 # [ 47.547705] nix-grpc-daemon[768]: ts=2026-09-21T14:30:11Z level=debug event=want client=3 drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv cached=04260worker1 # [ 47.554684] nix-grpc-daemon[768]: ts=2026-09-21T14:30:11Z level=debug event=assigned drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv worker=192.168.1.3:50051 clients=14261worker1 # [ 47.594124] nix-grpc-daemon[768]: ts=2026-09-21T14:30:11Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:389564262worker1 # [ 47.609550] nix-grpc-daemon[768]: ts=2026-09-21T14:30:11Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:389564263worker1 # [ 47.624601] nix-grpc-daemon[865]: time=2026-09-21T14:30:11.229Z level=INFO msg="All 2 paths already cached"4264client # copying 1 paths to 'lb:50051'...4265client # copying path '/nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv' to 'lb:50051'...4266worker1 # [ 47.635477] nix-grpc-daemon[768]: ts=2026-09-21T14:30:11Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:389564267worker1 # [ 47.650182] nix-daemon[851]: accepted connection from pid 768, user nix-grpc-daemon (trusted)4268worker1 # [ 47.805384] nix-daemon[851]: accepted connection from pid 1014, user nix-grpc-daemon (trusted)4269worker1 # [ 47.828571] nix-daemon[851]: reaped child process 1017, status = succeeded4270worker1 # [ 47.925848] nix-daemon[851]: accepted connection from pid 1019, user nix-grpc-daemon (trusted)4271worker1 # [ 47.946744] nix-daemon[851]: reaped child process 1022, status = succeeded4272lb # [ 47.945860] niks3-server[975]: 2026/09/21 14:30:11 INFO Received uploads request method=POST path=/api/pending_closures4273worker1 # [ 47.954338] nix-grpc-daemon[865]: time=2026-09-21T14:30:11.558Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4274worker1 # [ 47.955790] nix-grpc-daemon[865]: time=2026-09-21T14:30:11.558Z level=INFO msg="Uploading blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv (752B)"4275lb # [ 47.971218] niks3-server[975]: 2026/09/21 14:30:11 INFO Registered completed upload object_key=nar/1vy3ds565pni9mnkldcs6alimlb6bg39cimm60s4swkk5yjhwlqf.nar.zst4276lb # [ 47.987136] niks3-server[975]: 2026/09/21 14:30:11 INFO Received sign narinfos request method=POST path=/api/pending_closures/7/sign4277worker1 # [ 47.994154] nix-grpc-daemon[865]: time=2026-09-21T14:30:11.598Z level=INFO msg="Uploading 1 narinfos"4278lb # [ 47.999634] niks3-server[975]: 2026/09/21 14:30:11 INFO Signed narinfos id=7 count=14279lb # [ 48.005027] niks3-server[975]: 2026/09/21 14:30:11 INFO Registered completed upload object_key=blgxcsidls7na1m5da3m0bja1prb87vl.ls4280lb # [ 48.014804] niks3-server[975]: 2026/09/21 14:30:11 INFO Received complete upload request method=POST path=/api/pending_closures/7/complete4281lb # [ 48.021532] niks3-server[975]: 2026/09/21 14:30:11 INFO Completed upload id=74282worker1 # [ 48.020526] nix-grpc-daemon[865]: time=2026-09-21T14:30:11.625Z level=INFO msg="Upload complete. (353ms)"4283lb # [ 48.023903] niks3-server[975]: 2026/09/21 14:30:11 INFO Registered completed upload object_key=blgxcsidls7na1m5da3m0bja1prb87vl.narinfo4284worker1 # [ 48.030295] nix-grpc-daemon[768]: ts=2026-09-21T14:30:11Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38956 duration_s=0 paths=1 nar_bytes_in=7524285worker1 # [ 48.041873] nix-grpc-daemon[768]: ts=2026-09-21T14:30:11Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:389564286worker1 # [ 48.048981] nix-daemon[851]: accepted connection from pid 768, user nix-grpc-daemon (trusted)4287worker1 # [ 48.056156] nix-daemon[851]: reaped child process 1012, status = succeeded4288worker1 # [ 48.061055] nix-daemon[851]: accepted connection from pid 768, user nix-grpc-daemon (trusted)4289client # building '/nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv' on lb:50051...4290client # farm-top> node-a: building blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv4291client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds4292lb # [ 49.004769] envoy[909]: conn peer=[2001:db8:1::1]:34916 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=423 tx=198 ms=174293client # farm-top> LOG-farm-top-t1b4294worker1 # [ 49.097358] nix-daemon[851]: reaped child process 1026, status = succeeded4295worker1 # [ 49.188750] nix-daemon[851]: accepted connection from pid 1042, user nix-grpc-daemon (trusted)4296worker1 # [ 49.211008] nix-daemon[851]: reaped child process 1045, status = succeeded4297lb # [ 49.009410] envoy[909]: conn peer=[2001:db8:1::1]:34918 tls=TLSv1.3 subject="CN=stranger" sni=- flags=- tls_fail="-" rx=423 tx=181 ms=124298lb # [ 49.009536] envoy[909]: conn peer=[2001:db8:1::1]:34926 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9562 tx=4049 ms=58974299lb # [ 49.009701] envoy[909]: conn peer=[2001:db8:1::1]:57370 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=993 tx=480 ms=674300lb # [ 49.212288] niks3-server[975]: 2026/09/21 14:30:12 INFO Received uploads request method=POST path=/api/pending_closures4301worker1 # [ 49.240435] nix-grpc-daemon[865]: time=2026-09-21T14:30:12.843Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4302worker1 # [ 49.242018] nix-grpc-daemon[865]: time=2026-09-21T14:30:12.843Z level=INFO msg="Uploading 0xnysbbdbmapgf23wb2dbsfhgh18yxvp-farm-top (232B)"4303lb # [ 49.268363] niks3-server[975]: 2026/09/21 14:30:12 INFO Registered completed upload object_key=log/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv4304lb # [ 49.283069] niks3-server[975]: 2026/09/21 14:30:12 INFO Registered completed upload object_key=nar/1f6pjd04v1xmbxqqjhczkkfk4ns9qg6ssnspdalp41q9hkxf9yfr.nar.zst4305lb # [ 49.295705] niks3-server[975]: 2026/09/21 14:30:12 INFO Received sign narinfos request method=POST path=/api/pending_closures/8/sign4306worker1 # [ 49.303509] nix-grpc-daemon[865]: time=2026-09-21T14:30:12.907Z level=INFO msg="Uploading 1 narinfos"4307lb # [ 49.306058] niks3-server[975]: 2026/09/21 14:30:12 INFO Signed narinfos id=8 count=14308lb # [ 49.311285] niks3-server[975]: 2026/09/21 14:30:12 INFO Registered completed upload object_key=0xnysbbdbmapgf23wb2dbsfhgh18yxvp.ls4309lb # [ 49.322879] niks3-server[975]: 2026/09/21 14:30:12 INFO Received complete upload request method=POST path=/api/pending_closures/8/complete4310lb # [ 49.329340] niks3-server[975]: 2026/09/21 14:30:12 INFO Completed upload id=84311worker1 # [ 49.329395] nix-grpc-daemon[865]: time=2026-09-21T14:30:12.934Z level=INFO msg="Upload complete. (258ms)"4312lb # [ 49.331825] niks3-server[975]: 2026/09/21 14:30:12 INFO Registered completed upload object_key=0xnysbbdbmapgf23wb2dbsfhgh18yxvp.narinfo4313worker1 # [ 49.345125] nix-grpc-daemon[768]: ts=2026-09-21T14:30:12Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38956 duration_s=1 drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv assign_id=4 outputs=14314worker1 # [ 49.360683] nix-daemon[851]: reaped child process 1024, status = succeeded4315client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 --argstr top t1b >&2, in 2.08 seconds)4316client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 --argstr top t1b outPath4317client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 --argstr top t1b outPath, in 0.18 seconds)4318lb: must succeed: curl -sf localhost:9901/clusters4319lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)4320lb: must succeed: curl -sf localhost:9901/clusters4321lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4322worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4323worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.06 seconds)4324(finished: subtest: inputs already in the cache are not sent to the scheduler, in 2.60 seconds)4325subtest: two clients wanting the same drv build it once4326client: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup4327client # [ 49.811351] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup.4328client # Running as unit: dedup1.service; invocation ID: d7d08954594c4132b1783d66970024ea4329client: (finished: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup, in 0.10 seconds)4330client: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup4331client # [ 50.007267] nix[867]: don't know how to build these paths:4332client # [ 50.008780] nix[867]: /nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv4333worker1 # [ 50.001451] nix-grpc-daemon[768]: ts=2026-09-21T14:30:13Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:389564334worker1 # [ 50.016126] nix-grpc-daemon[768]: ts=2026-09-21T14:30:13Z level=debug event=want client=4 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv cached=04335worker1 # [ 50.022488] nix-grpc-daemon[768]: ts=2026-09-21T14:30:13Z level=debug event=assigned drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv worker=192.168.1.4:50051 clients=14336worker2 # [ 50.022353] nix-grpc-daemon[770]: ts=2026-09-21T14:30:13Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:490144337worker2 # [ 50.038209] nix-grpc-daemon[770]: ts=2026-09-21T14:30:13Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:490144338client # [ 50.061553] nix[867]: copying 1 paths to 'lb:50051'...4339client # [ 50.067157] nix[867]: copying path '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' to 'lb:50051'...4340worker2 # [ 50.055846] nix-grpc-daemon[770]: ts=2026-09-21T14:30:13Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:490144341worker2 # [ 50.071590] nix-daemon[838]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4342worker2 # [ 50.101215] nix-grpc-daemon[770]: ts=2026-09-21T14:30:13Z level=info event=niks3_push_started pid=8664343worker2 # [ 50.273162] nix-grpc-daemon[866]: time=2026-09-21T14:30:13.888Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.key ca=""4344worker2 # [ 50.383941] nix-daemon[838]: accepted connection from pid 878, user nix-grpc-daemon (trusted)4345worker2 # [ 50.407142] nix-daemon[838]: reaped child process 881, status = succeeded4346worker2 # [ 50.511298] nix-daemon[838]: accepted connection from pid 883, user nix-grpc-daemon (trusted)4347worker2 # [ 50.528332] nix-daemon[838]: reaped child process 886, status = succeeded4348lb # [ 50.543065] niks3-server[975]: 2026/09/21 14:30:14 INFO Received uploads request method=POST path=/api/pending_closures4349worker2 # [ 50.548066] nix-grpc-daemon[866]: time=2026-09-21T14:30:14.162Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4350worker2 # [ 50.549557] nix-grpc-daemon[866]: time=2026-09-21T14:30:14.162Z level=INFO msg="Uploading cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv (432B)"4351lb # [ 50.591368] niks3-server[975]: 2026/09/21 14:30:14 INFO Registered completed upload object_key=nar/0l7vpaziqxpw0srrsgimqvaa0dyxa9glicn3ilkwk06l64hn837w.nar.zst4352worker2 # [ 50.594841] nix-grpc-daemon[866]: time=2026-09-21T14:30:14.209Z level=INFO msg="Uploading 1 narinfos"4353lb # [ 50.604807] niks3-server[975]: 2026/09/21 14:30:14 INFO Received sign narinfos request method=POST path=/api/pending_closures/9/sign4354lb # [ 50.614208] niks3-server[975]: 2026/09/21 14:30:14 INFO Signed narinfos id=9 count=14355lb # [ 50.627074] niks3-server[975]: 2026/09/21 14:30:14 INFO Received complete upload request method=POST path=/api/pending_closures/9/complete4356worker2 # [ 50.626596] nix-grpc-daemon[866]: time=2026-09-21T14:30:14.241Z level=INFO msg="Upload complete. (349ms)"4357lb # [ 50.634422] niks3-server[975]: 2026/09/21 14:30:14 INFO Completed upload id=94358lb # [ 50.641366] niks3-server[975]: 2026/09/21 14:30:14 INFO Registered completed upload object_key=cp23830c44gs2dw4xy80s5wqyjm7vw84.ls4359worker2 # [ 50.636787] nix-grpc-daemon[866]: time=2026-09-21T14:30:14.242Z level=WARN msg="Failed to register uploaded object" key=cp23830c44gs2dw4xy80s5wqyjm7vw84.narinfo error="server returned 404: object is not pending upload\n"4360worker2 # [ 50.651370] nix-grpc-daemon[770]: ts=2026-09-21T14:30:14Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49014 duration_s=0 paths=1 nar_bytes_in=4324361worker2 # [ 50.659704] nix-grpc-daemon[770]: ts=2026-09-21T14:30:14Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:490144362worker2 # [ 50.667841] nix-daemon[838]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4363worker2 # [ 50.672834] nix-daemon[838]: reaped child process 864, status = succeeded4364client # [ 50.683672] nix[867]: building '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' on lb:50051...4365worker2 # [ 50.676774] nix-daemon[838]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4366client # [ 50.913850] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup.4367client # Running as unit: dedup2.service; invocation ID: 935fc72e86184ef7829682b5cd46a7534368client: (finished: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup, in 1.11 seconds)4369worker1 # [ 51.133229] nix-grpc-daemon[768]: ts=2026-09-21T14:30:14Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:389564370client # [ 51.145390] nix[883]: don't know how to build these paths:4371client # [ 51.147724] nix[883]: /nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv4372worker1 # [ 51.151257] nix-grpc-daemon[768]: ts=2026-09-21T14:30:14Z level=debug event=want client=5 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv cached=04373worker2 # [ 51.151062] nix-grpc-daemon[770]: ts=2026-09-21T14:30:14Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:490144374client # [ 51.168253] nix[883]: building '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' on lb:50051...4375client: must succeed: sleep 24376client: (finished: must succeed: sleep 2, in 2.06 seconds)4377client: waiting for success: ! systemctl is-active dedup1 dedup24378worker2 # [ 54.554658] nix-daemon[838]: reaped child process 890, status = succeeded4379worker2 # [ 54.622630] nix-daemon[838]: accepted connection from pid 920, user nix-grpc-daemon (trusted)4380worker2 # [ 54.639870] nix-daemon[838]: reaped child process 923, status = succeeded4381lb # [ 54.653010] niks3-server[975]: 2026/09/21 14:30:18 INFO Received uploads request method=POST path=/api/pending_closures4382worker2 # [ 54.656382] nix-grpc-daemon[866]: time=2026-09-21T14:30:18.271Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4383worker2 # [ 54.657866] nix-grpc-daemon[866]: time=2026-09-21T14:30:18.271Z level=INFO msg="Uploading vamz940my4dma9a2daklr9pw9x9pp9hp-slow-dedup (120B)"4384lb # [ 54.698378] niks3-server[975]: 2026/09/21 14:30:18 INFO Received sign narinfos request method=POST path=/api/pending_closures/10/sign4385worker2 # [ 54.702401] nix-grpc-daemon[866]: time=2026-09-21T14:30:18.316Z level=INFO msg="Uploading 1 narinfos"4386lb # [ 54.714486] niks3-server[975]: 2026/09/21 14:30:18 INFO Signed narinfos id=10 count=14387lb # [ 54.721948] niks3-server[975]: 2026/09/21 14:30:18 INFO Registered completed upload object_key=nar/02xhr0h0sj1ji8m0s8995xzbbb8mhbqdpdl1ggx94yqqaykihvz7.nar.zst4388lb # [ 54.733166] niks3-server[975]: 2026/09/21 14:30:18 INFO Registered completed upload object_key=log/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv4389worker2 # [ 54.739038] nix-grpc-daemon[866]: time=2026-09-21T14:30:18.354Z level=INFO msg="Upload complete. (249ms)"4390lb # [ 54.746158] niks3-server[975]: 2026/09/21 14:30:18 INFO Registered completed upload object_key=vamz940my4dma9a2daklr9pw9x9pp9hp.ls4391lb # [ 54.753436] niks3-server[975]: 2026/09/21 14:30:18 INFO Received complete upload request method=POST path=/api/pending_closures/10/complete4392worker2 # [ 54.749006] nix-grpc-daemon[770]: ts=2026-09-21T14:30:18Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49014 duration_s=3 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv assign_id=5 outputs=14393lb # [ 54.759765] niks3-server[975]: 2026/09/21 14:30:18 INFO Completed upload id=104394lb # [ 54.763285] niks3-server[975]: 2026/09/21 14:30:18 INFO Registered completed upload object_key=vamz940my4dma9a2daklr9pw9x9pp9hp.narinfo4395worker2 # [ 54.764317] nix-grpc-daemon[770]: ts=2026-09-21T14:30:18Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49014 duration_s=4 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv assign_id=5 outputs=14396worker2 # [ 54.770889] nix-daemon[838]: reaped child process 888, status = succeeded4397client # [ 54.799199] systemd[1]: dedup2.service: Deactivated successfully.4398client # [ 54.804547] systemd[1]: dedup2.service: Consumed 203ms CPU time over 3.889s wall clock time, 33.2M memory peak, 5.8K incoming IP traffic, 5.6K outgoing IP traffic.4399client # [ 54.844809] systemd[1]: dedup1.service: Deactivated successfully.4400client # [ 54.848356] systemd[1]: dedup1.service: Consumed 172ms CPU time over 5.036s wall clock time, 39.4M memory peak, 7.2K incoming IP traffic, 8.4K outgoing IP traffic.4401client: (finished: waiting for success: ! systemctl is-active dedup1 dedup2, in 1.13 seconds)4402client: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }4403client: (finished: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }, in 0.06 seconds)4404worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true4405worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.07 seconds)4406worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true4407worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.07 seconds)4408worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4409worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)4410worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4411worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)4412(finished: subtest: two clients wanting the same drv build it once, in 6.15 seconds)4413subtest: build hook: nix-daemon with builders = grpc://lb4414client: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --print-out-paths --no-link -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag hook 2>/tmp/hook.log4415client # [ 56.076576] systemd[1]: Started Nix Daemon.4416client # [ 56.195454] nix-daemon[925]: accepted connection from pid 922, user root (trusted)4417worker2 # [ 56.333924] nix-grpc-daemon[770]: ts=2026-09-21T14:30:19Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:490144418worker2 # [ 56.343168] nix-grpc-daemon[770]: ts=2026-09-21T14:30:19Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:490144419worker1 # [ 56.371381] nix-grpc-daemon[768]: ts=2026-09-21T14:30:19Z level=debug event=want client=6 drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv cached=04420worker1 # [ 56.371768] nix-grpc-daemon[768]: ts=2026-09-21T14:30:19Z level=debug event=assigned drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv worker=192.168.1.3:50051 clients=14421worker1 # [ 56.417070] nix-grpc-daemon[768]: ts=2026-09-21T14:30:20Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:389564422worker1 # [ 56.432619] nix-grpc-daemon[768]: ts=2026-09-21T14:30:20Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:389564423worker1 # [ 56.447601] nix-grpc-daemon[768]: ts=2026-09-21T14:30:20Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:389564424worker1 # [ 56.465436] nix-daemon[851]: accepted connection from pid 768, user nix-grpc-daemon (trusted)4425worker1 # [ 56.623354] nix-daemon[851]: accepted connection from pid 1081, user nix-grpc-daemon (trusted)4426worker1 # [ 56.647360] nix-daemon[851]: reaped child process 1084, status = succeeded4427worker1 # [ 56.747160] nix-daemon[851]: accepted connection from pid 1086, user nix-grpc-daemon (trusted)4428worker1 # [ 56.766058] nix-daemon[851]: reaped child process 1089, status = succeeded4429lb # [ 56.769825] niks3-server[975]: 2026/09/21 14:30:20 INFO Received uploads request method=POST path=/api/pending_closures4430worker1 # [ 56.779791] nix-grpc-daemon[865]: time=2026-09-21T14:30:20.383Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4431worker1 # [ 56.781431] nix-grpc-daemon[865]: time=2026-09-21T14:30:20.383Z level=INFO msg="Uploading xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv (512B)"4432lb # [ 56.800761] niks3-server[975]: 2026/09/21 14:30:20 INFO Registered completed upload object_key=nar/1f1i5kab3qhx7pg1k0ma5r1m7ipfazkfkfb944dm7qcjr466pwij.nar.zst4433worker1 # [ 56.825522] nix-grpc-daemon[865]: time=2026-09-21T14:30:20.430Z level=INFO msg="Uploading 1 narinfos"4434lb # [ 56.819014] niks3-server[975]: 2026/09/21 14:30:20 INFO Received sign narinfos request method=POST path=/api/pending_closures/11/sign4435lb # [ 56.829358] niks3-server[975]: 2026/09/21 14:30:20 INFO Signed narinfos id=11 count=14436lb # [ 56.835138] niks3-server[975]: 2026/09/21 14:30:20 INFO Registered completed upload object_key=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv.ls4437lb # [ 56.845511] niks3-server[975]: 2026/09/21 14:30:20 INFO Received complete upload request method=POST path=/api/pending_closures/11/complete4438worker1 # [ 56.846137] nix-grpc-daemon[865]: time=2026-09-21T14:30:20.450Z level=INFO msg="Upload complete. (363ms)"4439lb # [ 56.851241] niks3-server[975]: 2026/09/21 14:30:20 INFO Completed upload id=114440lb # [ 56.854291] niks3-server[975]: 2026/09/21 14:30:20 INFO Registered completed upload object_key=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv.narinfo4441worker1 # [ 56.855534] nix-grpc-daemon[768]: ts=2026-09-21T14:30:20Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38956 duration_s=0 paths=1 nar_bytes_in=5124442worker1 # [ 56.867724] nix-grpc-daemon[768]: ts=2026-09-21T14:30:20Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:389564443worker1 # [ 56.876502] nix-daemon[851]: accepted connection from pid 768, user nix-grpc-daemon (trusted)4444worker1 # [ 56.879198] nix-daemon[851]: reaped child process 1079, status = succeeded4445worker1 # [ 56.891970] nix-daemon[851]: accepted connection from pid 768, user nix-grpc-daemon (trusted)4446worker1 # [ 58.372816] nix-daemon[851]: reaped child process 1093, status = succeeded4447worker1 # [ 58.479749] nix-daemon[851]: accepted connection from pid 1109, user nix-grpc-daemon (trusted)4448worker1 # [ 58.498077] nix-daemon[851]: reaped child process 1112, status = succeeded4449lb # [ 58.503358] niks3-server[975]: 2026/09/21 14:30:22 INFO Received uploads request method=POST path=/api/pending_closures4450worker1 # [ 58.516348] nix-grpc-daemon[865]: time=2026-09-21T14:30:22.117Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4451worker1 # [ 58.518040] nix-grpc-daemon[865]: time=2026-09-21T14:30:22.117Z level=INFO msg="Uploading yfs64sxmsg77fnzczdvc3g3yfdwh9fks-farm-a (128B)"4452lb # [ 58.536421] niks3-server[975]: 2026/09/21 14:30:22 INFO Registered completed upload object_key=log/xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv4453lb # [ 58.555511] niks3-server[975]: 2026/09/21 14:30:22 INFO Registered completed upload object_key=nar/0pbbxl4kxy7zl7x4lp43ywljwn82mxzna2wgpb1bz1z765nwp9ki.nar.zst4454worker1 # [ 58.566111] nix-grpc-daemon[865]: time=2026-09-21T14:30:22.170Z level=INFO msg="Uploading 1 narinfos"4455lb # [ 58.568262] niks3-server[975]: 2026/09/21 14:30:22 INFO Received sign narinfos request method=POST path=/api/pending_closures/12/sign4456lb # [ 58.571580] niks3-server[975]: 2026/09/21 14:30:22 INFO Signed narinfos id=12 count=14457lb # [ 58.580355] niks3-server[975]: 2026/09/21 14:30:22 INFO Registered completed upload object_key=yfs64sxmsg77fnzczdvc3g3yfdwh9fks.ls4458lb # [ 58.588196] niks3-server[975]: 2026/09/21 14:30:22 INFO Received complete upload request method=POST path=/api/pending_closures/12/complete4459lb # [ 58.592291] niks3-server[975]: 2026/09/21 14:30:22 INFO Completed upload id=124460lb # [ 58.596429] niks3-server[975]: 2026/09/21 14:30:22 INFO Registered completed upload object_key=yfs64sxmsg77fnzczdvc3g3yfdwh9fks.narinfo4461worker1 # [ 58.592429] nix-grpc-daemon[865]: time=2026-09-21T14:30:22.197Z level=INFO msg="Upload complete. (289ms)"4462worker1 # [ 58.605292] nix-grpc-daemon[768]: ts=2026-09-21T14:30:22Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38956 duration_s=1 drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv assign_id=6 outputs=14463worker2 # [ 58.602272] nix-grpc-daemon[770]: ts=2026-09-21T14:30:22Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:490144464worker1 # [ 58.624665] nix-daemon[851]: reaped child process 1091, status = succeeded4465worker2 # [ 58.616743] nix-grpc-daemon[770]: copying path '/nix/store/yfs64sxmsg77fnzczdvc3g3yfdwh9fks-farm-a' from 'http://lb:5751'...4466worker2 # [ 58.658890] nix-grpc-daemon[770]: ts=2026-09-21T14:30:22Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:490144467worker2 # [ 58.664386] nix-grpc-daemon[770]: ts=2026-09-21T14:30:22Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:49014 duration_s=0 paths=1 nar_bytes_out=1284468worker1 # [ 58.939253] nix-grpc-daemon[768]: ts=2026-09-21T14:30:22Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:389564469worker2 # [ 58.936972] nix-grpc-daemon[770]: ts=2026-09-21T14:30:22Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:490144470worker2 # [ 58.949843] nix-grpc-daemon[866]: time=2026-09-21T14:30:22.564Z level=INFO msg="All 1 paths already cached"4471worker1 # [ 58.975967] nix-grpc-daemon[768]: ts=2026-09-21T14:30:22Z level=debug event=want client=7 drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv cached=04472worker1 # [ 58.985286] nix-grpc-daemon[768]: ts=2026-09-21T14:30:22Z level=debug event=assigned drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv worker=192.168.1.3:50051 clients=14473worker1 # [ 58.993896] nix-grpc-daemon[768]: ts=2026-09-21T14:30:22Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:389564474worker1 # [ 59.001480] nix-grpc-daemon[768]: ts=2026-09-21T14:30:22Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:389564475lb # [ 59.008569] envoy[909]: conn peer=[2001:db8:1::1]:57372 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4198 tx=1806 ms=18304476worker1 # [ 59.017297] nix-grpc-daemon[865]: time=2026-09-21T14:30:22.621Z level=INFO msg="All 1 paths already cached"4477lb # [ 59.015764] envoy[909]: conn peer=[2001:db8:1::1]:45608 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1443 tx=1085 ms=36454478lb # [ 59.022307] envoy[909]: conn peer=[2001:db8:1::1]:57382 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3018 tx=1453 ms=48224479lb # [ 59.028284] envoy[909]: conn peer=[2001:db8:1::1]:45616 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3401 tx=1854 ms=23844480lb # [ 59.035446] envoy[909]: conn peer=[2001:db8:1::1]:45630 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=410 tx=333 ms=654481worker1 # [ 59.036186] nix-grpc-daemon[768]: ts=2026-09-21T14:30:22Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:389564482worker1 # [ 59.049294] nix-daemon[851]: accepted connection from pid 768, user nix-grpc-daemon (trusted)4483worker1 # [ 59.212732] nix-daemon[851]: accepted connection from pid 1118, user nix-grpc-daemon (trusted)4484worker1 # [ 59.236180] nix-daemon[851]: reaped child process 1121, status = succeeded4485worker1 # [ 59.335556] nix-daemon[851]: accepted connection from pid 1123, user nix-grpc-daemon (trusted)4486worker1 # [ 59.353732] nix-daemon[851]: reaped child process 1126, status = succeeded4487lb # [ 59.359956] niks3-server[975]: 2026/09/21 14:30:22 INFO Received uploads request method=POST path=/api/pending_closures4488worker1 # [ 59.370044] nix-grpc-daemon[865]: time=2026-09-21T14:30:22.974Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4489worker1 # [ 59.371578] nix-grpc-daemon[865]: time=2026-09-21T14:30:22.974Z level=INFO msg="Uploading r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv (624B)"4490lb # [ 59.387252] niks3-server[975]: 2026/09/21 14:30:23 INFO Registered completed upload object_key=nar/15k66v3mh3v6p5vymv4pv3v3sfl56i2b1ia2rf093c37w6bhc04m.nar.zst4491worker1 # [ 59.402711] nix-grpc-daemon[865]: time=2026-09-21T14:30:23.007Z level=INFO msg="Uploading 1 narinfos"4492lb # [ 59.402326] niks3-server[975]: 2026/09/21 14:30:23 INFO Received sign narinfos request method=POST path=/api/pending_closures/13/sign4493lb # [ 59.413715] niks3-server[975]: 2026/09/21 14:30:23 INFO Signed narinfos id=13 count=14494lb # [ 59.422796] niks3-server[975]: 2026/09/21 14:30:23 INFO Registered completed upload object_key=r1jsr4b2g1qwly395z6idx3z2362ih8i.ls4495worker1 # [ 59.429458] nix-grpc-daemon[865]: time=2026-09-21T14:30:23.034Z level=INFO msg="Upload complete. (363ms)"4496lb # [ 59.429245] niks3-server[975]: 2026/09/21 14:30:23 INFO Received complete upload request method=POST path=/api/pending_closures/13/complete4497lb # [ 59.437498] niks3-server[975]: 2026/09/21 14:30:23 INFO Completed upload id=134498worker1 # [ 59.437677] nix-grpc-daemon[768]: ts=2026-09-21T14:30:23Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38956 duration_s=0 paths=1 nar_bytes_in=6244499lb # [ 59.439481] niks3-server[975]: 2026/09/21 14:30:23 INFO Registered completed upload object_key=r1jsr4b2g1qwly395z6idx3z2362ih8i.narinfo4500worker1 # [ 59.444442] nix-grpc-daemon[768]: ts=2026-09-21T14:30:23Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:389564501worker1 # [ 59.459274] nix-daemon[851]: accepted connection from pid 768, user nix-grpc-daemon (trusted)4502worker1 # [ 59.462610] nix-daemon[851]: reaped child process 1116, status = succeeded4503worker1 # [ 59.475385] nix-daemon[851]: accepted connection from pid 768, user nix-grpc-daemon (trusted)4504worker1 # [ 60.512723] nix-daemon[851]: reaped child process 1130, status = succeeded4505worker1 # [ 60.598202] nix-daemon[851]: accepted connection from pid 1146, user nix-grpc-daemon (trusted)4506worker1 # [ 60.618027] nix-daemon[851]: reaped child process 1149, status = succeeded4507lb # [ 60.623148] niks3-server[975]: 2026/09/21 14:30:24 INFO Received uploads request method=POST path=/api/pending_closures4508worker1 # [ 60.634806] nix-grpc-daemon[865]: time=2026-09-21T14:30:24.238Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4509worker1 # [ 60.636354] nix-grpc-daemon[865]: time=2026-09-21T14:30:24.238Z level=INFO msg="Uploading la6ryzsihag4zpxgxls54fsg34dvlalk-farm-b (176B)"4510lb # [ 60.660801] niks3-server[975]: 2026/09/21 14:30:24 INFO Registered completed upload object_key=nar/069d0xnyw2q345iqzx6agxc5fxifc6q36bkkklw2aa4qwanxdd1g.nar.zst4511lb # [ 60.672340] niks3-server[975]: 2026/09/21 14:30:24 INFO Registered completed upload object_key=log/r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv4512worker1 # [ 60.682527] nix-grpc-daemon[865]: time=2026-09-21T14:30:24.287Z level=INFO msg="Uploading 1 narinfos"4513lb # [ 60.683939] niks3-server[975]: 2026/09/21 14:30:24 INFO Received sign narinfos request method=POST path=/api/pending_closures/14/sign4514lb # [ 60.693029] niks3-server[975]: 2026/09/21 14:30:24 INFO Signed narinfos id=14 count=14515lb # [ 60.695160] niks3-server[975]: 2026/09/21 14:30:24 INFO Registered completed upload object_key=la6ryzsihag4zpxgxls54fsg34dvlalk.ls4516lb # [ 60.704469] niks3-server[975]: 2026/09/21 14:30:24 INFO Received complete upload request method=POST path=/api/pending_closures/14/complete4517lb # [ 60.709850] niks3-server[975]: 2026/09/21 14:30:24 INFO Completed upload id=144518lb # [ 60.712458] niks3-server[975]: 2026/09/21 14:30:24 INFO Registered completed upload object_key=la6ryzsihag4zpxgxls54fsg34dvlalk.narinfo4519worker1 # [ 60.709977] nix-grpc-daemon[865]: time=2026-09-21T14:30:24.314Z level=INFO msg="Upload complete. (223ms)"4520worker1 # [ 60.721033] nix-grpc-daemon[768]: ts=2026-09-21T14:30:24Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38956 duration_s=1 drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv assign_id=7 outputs=14521worker1 # [ 60.733695] nix-daemon[851]: reaped child process 1128, status = succeeded4522worker1 # [ 60.764460] nix-grpc-daemon[768]: ts=2026-09-21T14:30:24Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:389564523worker1 # [ 60.794454] nix-grpc-daemon[768]: ts=2026-09-21T14:30:24Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:389564524worker1 # [ 60.799372] nix-grpc-daemon[768]: ts=2026-09-21T14:30:24Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:38956 duration_s=0 paths=1 nar_bytes_out=1764525worker2 # [ 61.064338] nix-grpc-daemon[770]: ts=2026-09-21T14:30:24Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:490144526worker1 # [ 61.073916] nix-grpc-daemon[768]: ts=2026-09-21T14:30:24Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:389564527worker1 # [ 61.084374] nix-grpc-daemon[865]: time=2026-09-21T14:30:24.689Z level=INFO msg="All 2 paths already cached"4528worker1 # [ 61.102210] nix-grpc-daemon[768]: ts=2026-09-21T14:30:24Z level=debug event=want client=8 drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv cached=04529worker1 # [ 61.109365] nix-grpc-daemon[768]: ts=2026-09-21T14:30:24Z level=debug event=assigned drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv worker=192.168.1.3:50051 clients=14530worker1 # [ 61.118021] nix-grpc-daemon[768]: ts=2026-09-21T14:30:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:389564531worker1 # [ 61.129100] nix-grpc-daemon[768]: ts=2026-09-21T14:30:24Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:389564532worker1 # [ 61.142874] nix-grpc-daemon[865]: time=2026-09-21T14:30:24.747Z level=INFO msg="All 2 paths already cached"4533worker1 # [ 61.151910] nix-grpc-daemon[768]: ts=2026-09-21T14:30:24Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:389564534worker1 # [ 61.165939] nix-daemon[851]: accepted connection from pid 768, user nix-grpc-daemon (trusted)4535worker1 # [ 61.322335] nix-daemon[851]: accepted connection from pid 1155, user nix-grpc-daemon (trusted)4536worker1 # [ 61.341720] nix-daemon[851]: reaped child process 1158, status = succeeded4537worker1 # [ 61.444268] nix-daemon[851]: accepted connection from pid 1160, user nix-grpc-daemon (trusted)4538worker1 # [ 61.461680] nix-daemon[851]: reaped child process 1163, status = succeeded4539lb # [ 61.466512] niks3-server[975]: 2026/09/21 14:30:25 INFO Received uploads request method=POST path=/api/pending_closures4540worker1 # [ 61.478442] nix-grpc-daemon[865]: time=2026-09-21T14:30:25.082Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4541worker1 # [ 61.480310] nix-grpc-daemon[865]: time=2026-09-21T14:30:25.082Z level=INFO msg="Uploading aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv (752B)"4542lb # [ 61.495045] niks3-server[975]: 2026/09/21 14:30:25 INFO Registered completed upload object_key=nar/1ahq7sp1sl2scxqzfjd8gqimpqfkjbyi9ivxdhdwx0jzh2gbv74b.nar.zst4543lb # [ 61.510746] niks3-server[975]: 2026/09/21 14:30:25 INFO Received sign narinfos request method=POST path=/api/pending_closures/15/sign4544worker1 # [ 61.517590] nix-grpc-daemon[865]: time=2026-09-21T14:30:25.122Z level=INFO msg="Uploading 1 narinfos"4545lb # [ 61.523521] niks3-server[975]: 2026/09/21 14:30:25 INFO Signed narinfos id=15 count=14546lb # [ 61.529130] niks3-server[975]: 2026/09/21 14:30:25 INFO Registered completed upload object_key=aa4abikjj4w30bfx1p8hxmq0916qbz1c.ls4547lb # [ 61.539597] niks3-server[975]: 2026/09/21 14:30:25 INFO Received complete upload request method=POST path=/api/pending_closures/15/complete4548worker1 # [ 61.545421] nix-grpc-daemon[865]: time=2026-09-21T14:30:25.150Z level=INFO msg="Upload complete. (363ms)"4549lb # [ 61.547291] niks3-server[975]: 2026/09/21 14:30:25 INFO Completed upload id=154550lb # [ 61.552549] niks3-server[975]: 2026/09/21 14:30:25 INFO Registered completed upload object_key=aa4abikjj4w30bfx1p8hxmq0916qbz1c.narinfo4551worker1 # [ 61.556122] nix-grpc-daemon[768]: ts=2026-09-21T14:30:25Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38956 duration_s=0 paths=1 nar_bytes_in=7524552worker1 # [ 61.561231] nix-grpc-daemon[768]: ts=2026-09-21T14:30:25Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:389564553worker1 # [ 61.573398] nix-daemon[851]: accepted connection from pid 768, user nix-grpc-daemon (trusted)4554worker1 # [ 61.578894] nix-daemon[851]: reaped child process 1153, status = succeeded4555worker1 # [ 61.592638] nix-daemon[851]: accepted connection from pid 768, user nix-grpc-daemon (trusted)4556worker1 # [ 62.646383] nix-daemon[851]: reaped child process 1167, status = succeeded4557worker1 # [ 62.764123] nix-daemon[851]: accepted connection from pid 1183, user nix-grpc-daemon (trusted)4558worker1 # [ 62.782110] nix-daemon[851]: reaped child process 1186, status = succeeded4559lb # [ 62.788559] niks3-server[975]: 2026/09/21 14:30:26 INFO Received uploads request method=POST path=/api/pending_closures4560worker1 # [ 62.800424] nix-grpc-daemon[865]: time=2026-09-21T14:30:26.402Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4561worker1 # [ 62.802034] nix-grpc-daemon[865]: time=2026-09-21T14:30:26.402Z level=INFO msg="Uploading 39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top (232B)"4562lb # [ 62.837376] niks3-server[975]: 2026/09/21 14:30:26 INFO Registered completed upload object_key=log/aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv4563lb # [ 62.848141] niks3-server[975]: 2026/09/21 14:30:26 INFO Registered completed upload object_key=nar/0vq3rhfhhbqyx5pbvaa9ddj9wbw9arph334y3bp3idfrikwnjc92.nar.zst4564worker1 # [ 62.860170] nix-grpc-daemon[865]: time=2026-09-21T14:30:26.464Z level=INFO msg="Uploading 1 narinfos"4565lb # [ 62.862201] niks3-server[975]: 2026/09/21 14:30:26 INFO Received sign narinfos request method=POST path=/api/pending_closures/16/sign4566lb # [ 62.869873] niks3-server[975]: 2026/09/21 14:30:26 INFO Signed narinfos id=16 count=14567lb # [ 62.875460] niks3-server[975]: 2026/09/21 14:30:26 INFO Registered completed upload object_key=39qxwzy2zkij70y9zvbr59pb0df38w5h.ls4568lb # [ 62.887718] niks3-server[975]: 2026/09/21 14:30:26 INFO Received complete upload request method=POST path=/api/pending_closures/16/complete4569lb # [ 62.893109] niks3-server[975]: 2026/09/21 14:30:26 INFO Completed upload id=164570worker1 # [ 62.892885] nix-grpc-daemon[865]: time=2026-09-21T14:30:26.497Z level=INFO msg="Upload complete. (290ms)"4571lb # [ 62.896373] niks3-server[975]: 2026/09/21 14:30:26 INFO Registered completed upload object_key=39qxwzy2zkij70y9zvbr59pb0df38w5h.narinfo4572worker1 # [ 62.905470] nix-grpc-daemon[768]: ts=2026-09-21T14:30:26Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38956 duration_s=1 drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv assign_id=8 outputs=14573worker2 # [ 62.905971] nix-grpc-daemon[770]: ts=2026-09-21T14:30:26Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:490144574worker1 # [ 62.925380] nix-daemon[851]: reaped child process 1165, status = succeeded4575worker2 # [ 62.919993] nix-grpc-daemon[770]: copying path '/nix/store/la6ryzsihag4zpxgxls54fsg34dvlalk-farm-b' from 'http://lb:5751'...4576worker2 # [ 62.935409] nix-grpc-daemon[770]: copying path '/nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top' from 'http://lb:5751'...4577worker2 # [ 62.974253] nix-grpc-daemon[770]: ts=2026-09-21T14:30:26Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:490144578worker2 # [ 62.980304] nix-grpc-daemon[770]: ts=2026-09-21T14:30:26Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:49014 duration_s=0 paths=1 nar_bytes_out=2324579client # [ 63.119828] nix-daemon[925]: reaped child process 927, status = succeeded4580client: (finished: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --print-out-paths --no-link -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag hook 2>/tmp/hook.log, in 7.25 seconds)4581client: must succeed: grep farm-top-hook /nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top4582client: (finished: must succeed: grep farm-top-hook /nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top, in 0.04 seconds)4583client: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }4584client: (finished: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }, in 0.05 seconds)4585client: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }4586client: (finished: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }, in 0.03 seconds)4587(finished: subtest: build hook: nix-daemon with builders = grpc://lb, in 7.38 seconds)4588subtest: an input only one worker has still reaches the builder4589client: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A input --argstr tag w1only4590client # this derivation will be built:4591client # /nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv4592client # building '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv'...4593client: (finished: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A input --argstr tag w1only, in 0.30 seconds)4594client: must succeed: nix-store --export /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only > /tmp/shared/inp.closure4595client: (finished: must succeed: nix-store --export /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only > /tmp/shared/inp.closure, in 0.12 seconds)4596worker1: must succeed: nix-store --import < /tmp/shared/inp.closure4597worker1: (finished: must succeed: nix-store --import < /tmp/shared/inp.closure, in 0.19 seconds)4598worker2: must fail: test -e /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only4599worker2: (finished: must fail: test -e /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only, in 0.03 seconds)4600client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt a >&24601client # [ 64.088043] nix-daemon[925]: accepted connection from pid 1008, user root (trusted)4602client # this derivation will be built:4603client # /nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv4604client # building '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' on 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux'...4605worker1 # [ 64.238738] nix-grpc-daemon[768]: ts=2026-09-21T14:30:27Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:389564606worker1 # [ 64.246421] nix-grpc-daemon[768]: ts=2026-09-21T14:30:27Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:389564607worker1 # [ 64.415895] nix-daemon[851]: accepted connection from pid 1193, user nix-grpc-daemon (trusted)4608worker1 # [ 64.435105] nix-daemon[851]: reaped child process 1196, status = succeeded4609lb # [ 64.442075] niks3-server[975]: 2026/09/21 14:30:28 INFO Received uploads request method=POST path=/api/pending_closures4610worker1 # [ 64.469687] nix-grpc-daemon[865]: time=2026-09-21T14:30:28.073Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4611worker1 # [ 64.471761] nix-grpc-daemon[865]: time=2026-09-21T14:30:28.073Z level=INFO msg="Uploading dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only (120B)"4612lb # [ 64.490304] niks3-server[975]: 2026/09/21 14:30:28 INFO Registered completed upload object_key=nar/08ghl87f009pwqz85nbqv9hm0ln6gai6ggzilfmw3qnkaalfi6ss.nar.zst4613worker1 # [ 64.504576] nix-grpc-daemon[865]: time=2026-09-21T14:30:28.108Z level=INFO msg="Uploading 1 narinfos"4614lb # [ 64.505898] niks3-server[975]: 2026/09/21 14:30:28 INFO Received sign narinfos request method=POST path=/api/pending_closures/17/sign4615lb # [ 64.516347] niks3-server[975]: 2026/09/21 14:30:28 INFO Signed narinfos id=17 count=14616lb # [ 64.519478] niks3-server[975]: 2026/09/21 14:30:28 INFO Registered completed upload object_key=dymj0d1vj8q0ns6p096qk7fii766dap6.ls4617lb # [ 64.531877] niks3-server[975]: 2026/09/21 14:30:28 INFO Received complete upload request method=POST path=/api/pending_closures/17/complete4618lb # [ 64.541316] niks3-server[975]: 2026/09/21 14:30:28 INFO Completed upload id=174619worker1 # [ 64.542130] nix-grpc-daemon[865]: time=2026-09-21T14:30:28.146Z level=INFO msg="Upload complete. (296ms)"4620lb # [ 64.545318] niks3-server[975]: 2026/09/21 14:30:28 INFO Registered completed upload object_key=dymj0d1vj8q0ns6p096qk7fii766dap6.narinfo4621worker1 # [ 64.567161] nix-grpc-daemon[768]: ts=2026-09-21T14:30:28Z level=debug event=want client=9 drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv cached=04622worker1 # [ 64.573470] nix-grpc-daemon[768]: ts=2026-09-21T14:30:28Z level=debug event=assigned drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv worker=192.168.1.4:50051 clients=14623worker2 # [ 64.609834] nix-grpc-daemon[770]: ts=2026-09-21T14:30:28Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:490144624worker2 # [ 64.624355] nix-grpc-daemon[770]: ts=2026-09-21T14:30:28Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:490144625client # copying 2 paths to 'lb:50051'...4626client # copying path '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv' to 'lb:50051'...4627worker2 # [ 64.644436] nix-grpc-daemon[770]: ts=2026-09-21T14:30:28Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:490144628client # copying path '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' to 'lb:50051'...4629worker2 # [ 64.661154] nix-daemon[838]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4630worker2 # [ 64.829388] nix-daemon[838]: accepted connection from pid 942, user nix-grpc-daemon (trusted)4631worker2 # [ 64.857976] nix-daemon[838]: reaped child process 945, status = succeeded4632worker2 # [ 64.959331] nix-daemon[838]: accepted connection from pid 947, user nix-grpc-daemon (trusted)4633worker2 # [ 64.978935] nix-daemon[838]: reaped child process 950, status = succeeded4634lb # [ 64.989741] niks3-server[975]: 2026/09/21 14:30:28 INFO Received uploads request method=POST path=/api/pending_closures4635lb # [ 65.000456] niks3-server[975]: 2026/09/21 14:30:28 INFO Received uploads request method=POST path=/api/pending_closures4636worker2 # [ 65.004335] nix-grpc-daemon[866]: time=2026-09-21T14:30:28.618Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"4637worker2 # [ 65.005817] nix-grpc-daemon[866]: time=2026-09-21T14:30:28.618Z level=INFO msg="Uploading 71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv (568B)"4638worker2 # [ 65.007581] nix-grpc-daemon[866]: time=2026-09-21T14:30:28.618Z level=INFO msg="Uploading 4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv (416B)"4639lb # [ 65.044054] niks3-server[975]: 2026/09/21 14:30:28 INFO Registered completed upload object_key=nar/0i2vbprwp0wxa61nkss61gzivqmmjh6klkgv0rlfnxk46rcaba1v.nar.zst4640lb # [ 65.056123] niks3-server[975]: 2026/09/21 14:30:28 INFO Registered completed upload object_key=nar/0p6zgq1gdngshl62q0hv3lslqwv90s7fmflwc7wgnycf4ka6ik69.nar.zst4641lb # [ 65.068796] niks3-server[975]: 2026/09/21 14:30:28 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.ls4642lb # [ 65.075520] niks3-server[975]: 2026/09/21 14:30:28 INFO Received sign narinfos request method=POST path=/api/pending_closures/18/sign4643lb # [ 65.082113] niks3-server[975]: 2026/09/21 14:30:28 INFO Signed narinfos id=18 count=14644worker2 # [ 65.076124] nix-grpc-daemon[866]: time=2026-09-21T14:30:28.690Z level=INFO msg="Uploading 3 narinfos"4645lb # [ 65.086665] niks3-server[975]: 2026/09/21 14:30:28 INFO Received sign narinfos request method=POST path=/api/pending_closures/19/sign4646lb # [ 65.092601] niks3-server[975]: 2026/09/21 14:30:28 INFO Signed narinfos id=19 count=24647lb # [ 65.094320] niks3-server[975]: 2026/09/21 14:30:28 INFO Registered completed upload object_key=71l2x9mybp0i90r6pdnglwjnayj4xpl8.ls4648lb # [ 65.107722] niks3-server[975]: 2026/09/21 14:30:28 INFO Registered completed upload object_key=71l2x9mybp0i90r6pdnglwjnayj4xpl8.narinfo4649lb # [ 65.123046] niks3-server[975]: 2026/09/21 14:30:28 INFO Received complete upload request method=POST path=/api/pending_closures/18/complete4650lb # [ 65.127529] niks3-server[975]: 2026/09/21 14:30:28 INFO Completed upload id=184651lb # [ 65.133025] niks3-server[975]: 2026/09/21 14:30:28 INFO Received complete upload request method=POST path=/api/pending_closures/19/complete4652lb # [ 65.135299] niks3-server[975]: 2026/09/21 14:30:28 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.narinfo4653lb # [ 65.139243] niks3-server[975]: 2026/09/21 14:30:28 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.narinfo4654lb # [ 65.141693] niks3-server[975]: 2026/09/21 14:30:28 INFO Completed upload id=194655worker2 # [ 65.133100] nix-grpc-daemon[866]: time=2026-09-21T14:30:28.748Z level=INFO msg="Upload complete. (452ms)"4656worker2 # [ 65.144462] nix-grpc-daemon[770]: ts=2026-09-21T14:30:28Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49014 duration_s=0 paths=2 nar_bytes_in=9844657worker2 # [ 65.159477] nix-grpc-daemon[770]: ts=2026-09-21T14:30:28Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:490144658worker2 # [ 65.175678] nix-daemon[838]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4659worker2 # [ 65.182864] nix-daemon[838]: reaped child process 940, status = succeeded4660worker2 # [ 65.203129] nix-grpc-daemon[770]: copying path '/nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only' from 'http://lb:5751'...4661client # building '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' on lb:50051...4662worker2 # [ 65.222658] nix-daemon[838]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4663client # uses-input-w1onlya> worker2: building 71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv4664client # uses-input-w1onlya> disabling binary cache 'https://cache.nixos.org' for 60 seconds4665worker2 # [ 66.659374] nix-daemon[838]: reaped child process 955, status = succeeded4666worker2 # [ 66.765682] nix-daemon[838]: accepted connection from pid 971, user nix-grpc-daemon (trusted)4667worker2 # [ 66.786225] nix-daemon[838]: reaped child process 974, status = succeeded4668lb # [ 66.797472] niks3-server[975]: 2026/09/21 14:30:30 INFO Received uploads request method=POST path=/api/pending_closures4669worker2 # [ 66.802525] nix-grpc-daemon[866]: time=2026-09-21T14:30:30.417Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"4670worker2 # [ 66.838667] nix-grpc-daemon[866]: time=2026-09-21T14:30:30.453Z level=INFO msg="Uploading 1 narinfos"4671lb # [ 66.833868] niks3-server[975]: 2026/09/21 14:30:30 INFO Received sign narinfos request method=POST path=/api/pending_closures/20/sign4672lb # [ 66.851130] niks3-server[975]: 2026/09/21 14:30:30 INFO Signed narinfos id=20 count=14673lb # [ 66.859514] niks3-server[975]: 2026/09/21 14:30:30 INFO Registered completed upload object_key=log/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv4674lb # [ 66.873696] niks3-server[975]: 2026/09/21 14:30:30 INFO Registered completed upload object_key=2i7aa0cdpp60vpdm180r56gqfh25m7hc.ls4675worker2 # [ 66.873465] nix-grpc-daemon[866]: time=2026-09-21T14:30:30.489Z level=INFO msg="Upload complete. (232ms)"4676lb # [ 66.881332] niks3-server[975]: 2026/09/21 14:30:30 INFO Received complete upload request method=POST path=/api/pending_closures/20/complete4677lb # [ 66.887311] niks3-server[975]: 2026/09/21 14:30:30 INFO Completed upload id=204678lb # [ 66.890448] niks3-server[975]: 2026/09/21 14:30:30 INFO Registered completed upload object_key=2i7aa0cdpp60vpdm180r56gqfh25m7hc.narinfo4679worker2 # [ 66.885077] nix-grpc-daemon[770]: ts=2026-09-21T14:30:30Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49014 duration_s=1 drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv assign_id=9 outputs=14680client # copying 1 paths...4681worker1 # [ 66.903743] nix-grpc-daemon[768]: ts=2026-09-21T14:30:30Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:389564682worker2 # [ 66.900436] nix-daemon[838]: reaped child process 952, status = succeeded4683worker1 # [ 66.922047] nix-grpc-daemon[768]: copying path '/nix/store/2i7aa0cdpp60vpdm180r56gqfh25m7hc-uses-input-w1onlya' from 'http://lb:5751'...4684client # copying path '/nix/store/2i7aa0cdpp60vpdm180r56gqfh25m7hc-uses-input-w1onlya' from 'grpc://lb:50051'...4685worker2 # [ 66.974188] nix-grpc-daemon[770]: ts=2026-09-21T14:30:30Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:490144686worker2 # [ 66.981913] nix-grpc-daemon[770]: ts=2026-09-21T14:30:30Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:49014 duration_s=0 paths=1 nar_bytes_out=1204687client # [ 67.135759] nix-daemon[925]: reaped child process 1011, status = succeeded4688client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt a >&2, in 3.24 seconds)4689client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt b >&24690client # [ 67.328248] nix-daemon[925]: accepted connection from pid 1029, user root (trusted)4691client # this derivation will be built:4692client # /nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv4693worker1 # [ 67.492072] nix-grpc-daemon[768]: ts=2026-09-21T14:30:31Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:389564694client # building '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' on 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux'...4695worker1 # [ 67.502039] nix-grpc-daemon[768]: ts=2026-09-21T14:30:31Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:389564696worker1 # [ 67.510418] nix-grpc-daemon[865]: time=2026-09-21T14:30:31.112Z level=INFO msg="All 1 paths already cached"4697worker1 # [ 67.524219] nix-grpc-daemon[768]: ts=2026-09-21T14:30:31Z level=debug event=want client=10 drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv cached=04698worker1 # [ 67.532672] nix-grpc-daemon[768]: ts=2026-09-21T14:30:31Z level=debug event=assigned drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv worker=192.168.1.3:50051 clients=14699worker1 # [ 67.543061] nix-grpc-daemon[768]: ts=2026-09-21T14:30:31Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:389564700worker1 # [ 67.555715] nix-grpc-daemon[768]: ts=2026-09-21T14:30:31Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:389564701client # copying 1 paths to 'lb:50051'...4702client # copying path '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' to 'lb:50051'...4703worker1 # [ 67.580118] nix-grpc-daemon[768]: ts=2026-09-21T14:30:31Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:389564704worker1 # [ 67.597291] nix-daemon[851]: accepted connection from pid 768, user nix-grpc-daemon (trusted)4705worker1 # [ 67.626039] nix-grpc-daemon[768]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 18 ms (attempt 1/5)4706worker1 # [ 67.652509] nix-grpc-daemon[768]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 113 ms (attempt 2/5)4707worker1 # [ 67.774735] nix-grpc-daemon[768]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 300 ms (attempt 3/5)4708worker1 # [ 68.078075] nix-grpc-daemon[768]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 717 ms (attempt 4/5)4709worker1 # [ 68.798365] nix-grpc-daemon[768]: disabling binary cache 'https://cache.nixos.org' for 60 seconds4710worker1 # [ 68.798712] nix-grpc-daemon[768]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4711worker1 # [ 68.810088] nix-grpc-daemon[768]: copying path '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv' from 'http://lb:5751'...4712worker1 # [ 69.004851] nix-daemon[851]: accepted connection from pid 1210, user nix-grpc-daemon (trusted)4713lb # [ 69.005734] envoy[909]: conn peer=[2001:db8:1::1]:45638 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3900 tx=1931 ms=19134714lb # [ 69.010673] envoy[909]: conn peer=[2001:db8:1::1]:45642 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=436 tx=375 ms=644715lb # [ 69.015838] envoy[909]: conn peer=[2001:db8:1::1]:56434 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4351 tx=2278 ms=19754716lb # [ 69.020772] envoy[909]: conn peer=[2001:db8:1::1]:56448 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=438 tx=406 ms=704717worker1 # [ 69.030510] nix-daemon[851]: reaped child process 1213, status = succeeded4718lb # [ 69.025609] envoy[909]: conn peer=[2001:db8:1::1]:56458 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3916 tx=1972 ms=28104719lb # [ 69.031948] envoy[909]: conn peer=[2001:db8:1::1]:56468 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=448 tx=328 ms=744720worker1 # [ 69.135606] nix-daemon[851]: accepted connection from pid 1215, user nix-grpc-daemon (trusted)4721worker1 # [ 69.156674] nix-daemon[851]: reaped child process 1218, status = succeeded4722lb # [ 69.161894] niks3-server[975]: 2026/09/21 14:30:32 INFO Received uploads request method=POST path=/api/pending_closures4723worker1 # [ 69.263001] nix-grpc-daemon[865]: time=2026-09-21T14:30:32.865Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4724worker1 # [ 69.268361] nix-grpc-daemon[865]: time=2026-09-21T14:30:32.865Z level=INFO msg="Uploading 12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv (568B)"4725lb # [ 69.290029] niks3-server[975]: 2026/09/21 14:30:32 INFO Registered completed upload object_key=nar/14jznd749vfl63s1gv7i29r5z6jcai2byp2nz83dr5gib6z2cyaz.nar.zst4726worker1 # [ 69.305772] nix-grpc-daemon[865]: time=2026-09-21T14:30:32.909Z level=INFO msg="Uploading 1 narinfos"4727lb # [ 69.305202] niks3-server[975]: 2026/09/21 14:30:32 INFO Received sign narinfos request method=POST path=/api/pending_closures/21/sign4728lb # [ 69.313285] niks3-server[975]: 2026/09/21 14:30:32 INFO Signed narinfos id=21 count=14729lb # [ 69.322269] niks3-server[975]: 2026/09/21 14:30:32 INFO Registered completed upload object_key=12ialcyr3l4m98rzfgk6apzimy55z3j8.ls4730lb # [ 69.334229] niks3-server[975]: 2026/09/21 14:30:32 INFO Received complete upload request method=POST path=/api/pending_closures/21/complete4731worker1 # [ 69.342622] nix-grpc-daemon[865]: time=2026-09-21T14:30:32.947Z level=INFO msg="Upload complete. (503ms)"4732lb # [ 69.348752] niks3-server[975]: 2026/09/21 14:30:32 INFO Completed upload id=214733lb # [ 69.357266] niks3-server[975]: 2026/09/21 14:30:32 INFO Registered completed upload object_key=12ialcyr3l4m98rzfgk6apzimy55z3j8.narinfo4734worker1 # [ 69.353152] nix-grpc-daemon[768]: ts=2026-09-21T14:30:32Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38956 duration_s=1 paths=1 nar_bytes_in=5684735worker1 # [ 69.373225] nix-grpc-daemon[768]: ts=2026-09-21T14:30:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:389564736worker1 # [ 69.383169] nix-daemon[851]: accepted connection from pid 768, user nix-grpc-daemon (trusted)4737worker1 # [ 69.391825] nix-daemon[851]: reaped child process 1203, status = succeeded4738client # building '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' on lb:50051...4739client # uses-input-w1onlyb> node-a: building 12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv4740worker1 # [ 69.410551] nix-daemon[851]: accepted connection from pid 768, user nix-grpc-daemon (trusted)4741client # uses-input-w1onlyb> disabling binary cache 'https://cache.nixos.org' for 60 seconds4742worker1 # [ 70.480953] nix-daemon[851]: reaped child process 1222, status = succeeded4743worker1 # [ 70.596624] nix-daemon[851]: accepted connection from pid 1238, user nix-grpc-daemon (trusted)4744worker1 # [ 70.618723] nix-daemon[851]: reaped child process 1241, status = succeeded4745lb # [ 70.616699] niks3-server[975]: 2026/09/21 14:30:34 INFO Received uploads request method=POST path=/api/pending_closures4746worker1 # [ 70.629685] nix-grpc-daemon[865]: time=2026-09-21T14:30:34.232Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"4747lb # [ 70.648076] niks3-server[975]: 2026/09/21 14:30:34 INFO Received sign narinfos request method=POST path=/api/pending_closures/22/sign4748worker1 # [ 70.660189] nix-grpc-daemon[865]: time=2026-09-21T14:30:34.264Z level=INFO msg="Uploading 1 narinfos"4749lb # [ 70.666502] niks3-server[975]: 2026/09/21 14:30:34 INFO Signed narinfos id=22 count=14750lb # [ 70.673925] niks3-server[975]: 2026/09/21 14:30:34 INFO Registered completed upload object_key=jfy62nvxcd0b9rkhly152q21ddbrcn28.ls4751worker1 # [ 70.685989] nix-grpc-daemon[865]: time=2026-09-21T14:30:34.290Z level=INFO msg="Upload complete. (259ms)"4752lb # [ 70.681681] niks3-server[975]: 2026/09/21 14:30:34 INFO Registered completed upload object_key=log/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv4753lb # [ 70.694450] niks3-server[975]: 2026/09/21 14:30:34 INFO Received complete upload request method=POST path=/api/pending_closures/22/complete4754lb # [ 70.697603] niks3-server[975]: 2026/09/21 14:30:34 INFO Completed upload id=224755worker1 # [ 70.697047] nix-grpc-daemon[768]: ts=2026-09-21T14:30:34Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38956 duration_s=1 drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv assign_id=10 outputs=14756lb # [ 70.699603] niks3-server[975]: 2026/09/21 14:30:34 INFO Registered completed upload object_key=jfy62nvxcd0b9rkhly152q21ddbrcn28.narinfo4757worker1 # [ 70.713101] nix-daemon[851]: reaped child process 1220, status = succeeded4758client # copying 1 paths...4759worker2 # [ 70.737621] nix-grpc-daemon[770]: ts=2026-09-21T14:30:34Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:490144760worker2 # [ 70.752461] nix-grpc-daemon[770]: copying path '/nix/store/jfy62nvxcd0b9rkhly152q21ddbrcn28-uses-input-w1onlyb' from 'http://lb:5751'...4761client # copying path '/nix/store/jfy62nvxcd0b9rkhly152q21ddbrcn28-uses-input-w1onlyb' from 'grpc://lb:50051'...4762worker1 # [ 70.804332] nix-grpc-daemon[768]: ts=2026-09-21T14:30:34Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:389564763worker1 # [ 70.810477] nix-grpc-daemon[768]: ts=2026-09-21T14:30:34Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:38956 duration_s=0 paths=1 nar_bytes_out=1204764client # [ 70.895905] nix-daemon[925]: reaped child process 1032, status = succeeded4765client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt b >&2, in 3.76 seconds)4766client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt c >&24767client # [ 71.068616] nix-daemon[925]: accepted connection from pid 1051, user root (trusted)4768client # this derivation will be built:4769client # /nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv4770worker2 # [ 71.207360] nix-grpc-daemon[770]: ts=2026-09-21T14:30:34Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:490144771client # building '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' on 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux'...4772worker1 # [ 71.218081] nix-grpc-daemon[768]: ts=2026-09-21T14:30:34Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:389564773worker1 # [ 71.229678] nix-grpc-daemon[865]: time=2026-09-21T14:30:34.834Z level=INFO msg="All 1 paths already cached"4774worker1 # [ 71.246344] nix-grpc-daemon[768]: ts=2026-09-21T14:30:34Z level=debug event=want client=11 drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv cached=04775worker2 # [ 71.243307] nix-grpc-daemon[770]: ts=2026-09-21T14:30:34Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:490144776worker1 # [ 71.253055] nix-grpc-daemon[768]: ts=2026-09-21T14:30:34Z level=debug event=assigned drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv worker=192.168.1.4:50051 clients=14777worker2 # [ 71.258881] nix-grpc-daemon[770]: ts=2026-09-21T14:30:34Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:490144778worker2 # [ 71.275845] nix-grpc-daemon[866]: time=2026-09-21T14:30:34.890Z level=INFO msg="All 1 paths already cached"4779client # copying 1 paths to 'lb:50051'...4780client # copying path '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' to 'lb:50051'...4781worker2 # [ 71.285880] nix-grpc-daemon[770]: ts=2026-09-21T14:30:34Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:490144782worker2 # [ 71.301273] nix-daemon[838]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4783worker2 # [ 71.461379] nix-daemon[838]: accepted connection from pid 979, user nix-grpc-daemon (trusted)4784worker2 # [ 71.482961] nix-daemon[838]: reaped child process 982, status = succeeded4785worker2 # [ 71.584340] nix-daemon[838]: accepted connection from pid 984, user nix-grpc-daemon (trusted)4786worker2 # [ 71.601744] nix-daemon[838]: reaped child process 987, status = succeeded4787lb # [ 71.613124] niks3-server[975]: 2026/09/21 14:30:35 INFO Received uploads request method=POST path=/api/pending_closures4788worker2 # [ 71.615664] nix-grpc-daemon[866]: time=2026-09-21T14:30:35.230Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4789worker2 # [ 71.617357] nix-grpc-daemon[866]: time=2026-09-21T14:30:35.230Z level=INFO msg="Uploading n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv (568B)"4790lb # [ 71.642123] niks3-server[975]: 2026/09/21 14:30:35 INFO Registered completed upload object_key=nar/1qd3ps964395j2cc3rlqd58m21s7f6na9s0k2z5kclx0c3l4sl40.nar.zst4791lb # [ 71.657952] niks3-server[975]: 2026/09/21 14:30:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/23/sign4792worker2 # [ 71.657047] nix-grpc-daemon[866]: time=2026-09-21T14:30:35.272Z level=INFO msg="Uploading 1 narinfos"4793lb # [ 71.670102] niks3-server[975]: 2026/09/21 14:30:35 INFO Signed narinfos id=23 count=14794lb # [ 71.676213] niks3-server[975]: 2026/09/21 14:30:35 INFO Registered completed upload object_key=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr.ls4795lb # [ 71.686403] niks3-server[975]: 2026/09/21 14:30:35 INFO Received complete upload request method=POST path=/api/pending_closures/23/complete4796worker2 # [ 71.684336] nix-grpc-daemon[866]: time=2026-09-21T14:30:35.299Z level=INFO msg="Upload complete. (363ms)"4797lb # [ 71.694377] niks3-server[975]: 2026/09/21 14:30:35 INFO Completed upload id=234798lb # [ 71.697714] niks3-server[975]: 2026/09/21 14:30:35 INFO Registered completed upload object_key=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr.narinfo4799worker2 # [ 71.696749] nix-grpc-daemon[770]: ts=2026-09-21T14:30:35Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49014 duration_s=0 paths=1 nar_bytes_in=5684800worker2 # [ 71.706976] nix-grpc-daemon[770]: ts=2026-09-21T14:30:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:490144801worker2 # [ 71.714149] nix-daemon[838]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4802worker2 # [ 71.718984] nix-daemon[838]: reaped child process 977, status = succeeded4803client # building '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' on lb:50051...4804worker2 # [ 71.732187] nix-daemon[838]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4805client # uses-input-w1onlyc> worker2: building n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv4806client # uses-input-w1onlyc> disabling binary cache 'https://cache.nixos.org' for 60 seconds4807worker2 # [ 72.372529] nix-daemon[838]: reaped child process 991, status = succeeded4808worker2 # [ 72.474450] nix-daemon[838]: accepted connection from pid 1007, user nix-grpc-daemon (trusted)4809worker2 # [ 72.493381] nix-daemon[838]: reaped child process 1010, status = succeeded4810lb # [ 72.498630] niks3-server[975]: 2026/09/21 14:30:36 INFO Received uploads request method=POST path=/api/pending_closures4811worker2 # [ 72.503487] nix-grpc-daemon[866]: time=2026-09-21T14:30:36.117Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"4812lb # [ 72.530373] niks3-server[975]: 2026/09/21 14:30:36 INFO Registered completed upload object_key=13r5ipj38sk1irdi724wm1mbl9lw720b.ls4813worker2 # [ 72.535496] nix-grpc-daemon[866]: time=2026-09-21T14:30:36.150Z level=INFO msg="Uploading 1 narinfos"4814lb # [ 72.541770] niks3-server[975]: 2026/09/21 14:30:36 INFO Received sign narinfos request method=POST path=/api/pending_closures/24/sign4815lb # [ 72.554461] niks3-server[975]: 2026/09/21 14:30:36 INFO Signed narinfos id=24 count=14816lb # [ 72.561671] niks3-server[975]: 2026/09/21 14:30:36 INFO Registered completed upload object_key=log/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv4817worker2 # [ 72.570226] nix-grpc-daemon[866]: time=2026-09-21T14:30:36.185Z level=INFO msg="Upload complete. (228ms)"4818lb # [ 72.576576] niks3-server[975]: 2026/09/21 14:30:36 INFO Received complete upload request method=POST path=/api/pending_closures/24/complete4819lb # [ 72.583496] niks3-server[975]: 2026/09/21 14:30:36 INFO Completed upload id=244820lb # [ 72.586668] niks3-server[975]: 2026/09/21 14:30:36 INFO Registered completed upload object_key=13r5ipj38sk1irdi724wm1mbl9lw720b.narinfo4821worker2 # [ 72.581217] nix-grpc-daemon[770]: ts=2026-09-21T14:30:36Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49014 duration_s=0 drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv assign_id=11 outputs=14822worker2 # [ 72.594724] nix-daemon[838]: reaped child process 989, status = succeeded4823client # copying 1 paths...4824worker2 # [ 72.627898] nix-grpc-daemon[770]: ts=2026-09-21T14:30:36Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:490144825client # copying path '/nix/store/13r5ipj38sk1irdi724wm1mbl9lw720b-uses-input-w1onlyc' from 'grpc://lb:50051'...4826worker1 # [ 72.661080] nix-grpc-daemon[768]: ts=2026-09-21T14:30:36Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:389564827worker1 # [ 72.673944] nix-grpc-daemon[768]: copying path '/nix/store/13r5ipj38sk1irdi724wm1mbl9lw720b-uses-input-w1onlyc' from 'http://lb:5751'...4828worker1 # [ 72.690209] nix-grpc-daemon[768]: ts=2026-09-21T14:30:36Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:38956 duration_s=0 paths=1 nar_bytes_out=1204829client # [ 72.818725] nix-daemon[925]: reaped child process 1054, status = succeeded4830client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt c >&2, in 1.92 seconds)4831(finished: subtest: an input only one worker has still reaches the builder, in 9.57 seconds)4832subtest: upload whose reference the worker lacks is completed from the cache4833client: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A input --argstr tag viacache4834client # this derivation will be built:4835client # /nix/store/x1rrnjihpa1xvpyxxxdrjzp3c8z8nayq-local-input-viacache.drv4836client # building '/nix/store/x1rrnjihpa1xvpyxxxdrjzp3c8z8nayq-local-input-viacache.drv'...4837client: (finished: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A input --argstr tag viacache, in 0.34 seconds)4838client: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A referrer --argstr tag viacache4839client # this derivation will be built:4840client # /nix/store/1m587dk4visgyryf6n0mgp2rlp7pbjd1-refers-to-input-viacache.drv4841client # building '/nix/store/1m587dk4visgyryf6n0mgp2rlp7pbjd1-refers-to-input-viacache.drv'...4842client: (finished: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A referrer --argstr tag viacache, in 0.29 seconds)4843client: must succeed: nix-store --export /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache > /tmp/shared/ref.closure4844client: (finished: must succeed: nix-store --export /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache > /tmp/shared/ref.closure, in 0.12 seconds)4845worker1: must succeed: nix-store --import < /tmp/shared/ref.closure4846worker1: (finished: must succeed: nix-store --import < /tmp/shared/ref.closure, in 0.19 seconds)4847worker2 # [ 73.823643] systemd[1]: Stopping Nix worker-protocol over gRPC...4848worker2 # [ 74.338322] nix-grpc-daemon[770]: ts=2026-09-21T14:30:37Z level=info event=signal_exit4849worker2 # [ 75.915716] nix-grpc-daemon[770]: ts=2026-09-21T14:30:39Z level=info event=scheduler_restarting4850worker1 # [ 75.922310] nix-grpc-daemon[768]: ts=2026-09-21T14:30:39Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:389404851worker1 # [ 75.927108] nix-grpc-daemon[768]: ts=2026-09-21T14:30:39Z level=info event=worker_gone addr=192.168.1.4:500514852worker2 # [ 75.945164] nix-grpc-daemon[770]: ts=2026-09-21T14:30:39Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"4853worker2 # [ 75.970126] nix-daemon[838]: reaped child process 840, status = succeeded4854worker2 # [ 76.467916] nix-grpc-daemon[770]: E0000 00:00:1790001040.083135 815 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 16, started true, shutdown true4855worker2 # [ 76.474696] nix-grpc-daemon[770]: E0000 00:00:1790001040.083439 815 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true4856worker2 # [ 77.922129] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.4857worker2 # [ 77.928563] systemd[1]: Stopped Nix worker-protocol over gRPC.4858worker2 # [ 77.931125] systemd[1]: nix-grpc-daemon.service: Consumed 1.808s CPU time over 58.797s wall clock time, 53.6M memory peak, 49K incoming IP traffic, 139.2K outgoing IP traffic.4859worker2 # [ 77.949020] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.4860worker2 # [ 77.954501] systemd[1]: Closed Nix worker-protocol over gRPC.4861worker2 # [ 77.956661] systemd[1]: nix-grpc-daemon.socket: Consumed 640us CPU time over 1min 8.032s wall clock time, 256K memory peak, 56.4K incoming IP traffic, 49.7K outgoing IP traffic.4862lb: waiting until exactly ['worker2'] endpoints of aarch64-linux are unhealthy4863lb: must succeed: curl -sf localhost:9901/clusters4864lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)4865lb: (finished: waiting until exactly ['worker2'] endpoints of aarch64-linux are unhealthy, in 0.07 seconds)4866client: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache >&24867worker1 # [ 78.254549] nix-grpc-daemon[768]: ts=2026-09-21T14:30:41Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:389564868worker1 # [ 78.263453] nix-grpc-daemon[768]: ts=2026-09-21T14:30:41Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:389564869client # /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache4870client: (finished: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache >&2, in 0.25 seconds)4871worker2: must fail: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache4872worker2: (finished: must fail: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache, in 0.03 seconds)4873worker2 # [ 78.377896] systemd[1]: Listening on Nix worker-protocol over gRPC.4874worker1 # [ 78.454832] systemd[1]: Stopping Nix worker-protocol over gRPC...4875worker2 # [ 78.919860] systemd[1]: Starting scheduler-token.service...4876lb # [ 79.006782] envoy[909]: conn peer=[2001:db8:1::1]:56478 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3753 tx=2013 ms=33354877worker2 # [ 79.004095] systemd[1]: scheduler-token.service: Deactivated successfully.4878worker2 # [ 79.012189] systemd[1]: Finished scheduler-token.service.4879worker2 # [ 79.014348] systemd[1]: scheduler-token.service: Consumed 56ms CPU time over 87ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.4880worker2 # [ 79.036689] systemd[1]: Starting Nix worker-protocol over gRPC...4881worker2 # [ 79.162280] nix-grpc-daemon[1034]: ts=2026-09-21T14:30:42Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"4882worker2 # [ 79.164571] nix-grpc-daemon[1034]: ts=2026-09-21T14:30:42Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc4883worker2 # [ 79.168434] nix-grpc-daemon[1034]: ts=2026-09-21T14:30:42Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=14884worker2 # [ 79.173828] nix-grpc-daemon[1034]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR4885worker2 # [ 79.175264] nix-grpc-daemon[1034]: E0000 00:00:1790001042.788766 1034 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4886worker2 # [ 79.220086] nix-grpc-daemon[1034]: E0000 00:00:1790001042.794859 1034 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4887worker2 # [ 79.222334] nix-grpc-daemon[1034]: ts=2026-09-21T14:30:42Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket4888worker2 # [ 79.223972] nix-grpc-daemon[1034]: E0000 00:00:1790001042.806164 1075 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4889worker2 # [ 79.230519] nix-grpc-daemon[1034]: E0000 00:00:1790001042.824003 1075 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4890worker2 # [ 79.233351] nix-grpc-daemon[1034]: E0000 00:00:1790001042.824136 1075 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4891worker2 # [ 79.235886] systemd[1]: Started Nix worker-protocol over gRPC.4892worker2 # [ 79.236771] nix-grpc-daemon[1034]: E0000 00:00:1790001042.845284 1075 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4893worker2 # [ 79.244211] nix-grpc-daemon[1034]: ts=2026-09-21T14:30:42Z level=info event=scheduler_connected addr=lb:500514894worker1 # [ 79.252996] nix-grpc-daemon[768]: ts=2026-09-21T14:30:42Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:389404895worker1 # [ 79.260836] nix-grpc-daemon[768]: ts=2026-09-21T14:30:42Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=04896worker1 # [ 79.272732] nix-grpc-daemon[768]: ts=2026-09-21T14:30:42Z level=info event=signal_exit4897lb # [ 79.011912] envoy[909]: conn peer=[2001:db8:1::1]:56494 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=448 tx=328 ms=264898lb # [ 79.012085] envoy[909]: conn peer=[2001:db8:1::1]:43084 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3724 tx=1881 ms=15354899lb # [ 79.012204] envoy[909]: conn peer=[2001:db8:1::1]:43086 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=448 tx=329 ms=924900lb # [ 79.012318] envoy[909]: conn peer=[2001:db8:1::4]:56572 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=57252 tx=2865 ms=565424901lb # [ 79.012433] envoy[909]: conn peer=[2001:db8:1::1]:43100 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=591 tx=522 ms=304902lb # [ 79.913777] niks3-server[975]: 2026/09/21 14:30:43 INFO lead: released remote=192.168.1.3:502324903worker1 # [ 80.915182] nix-grpc-daemon[768]: ts=2026-09-21T14:30:44Z level=info event=scheduler_restarting4904worker2 # [ 80.910680] nix-grpc-daemon[1034]: ts=2026-09-21T14:30:44Z level=info event=scheduler_restarting addr=lb:500514905worker1 # [ 80.927878] nix-grpc-daemon[768]: ts=2026-09-21T14:30:44Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:389404906worker1 # [ 80.932965] nix-grpc-daemon[768]: ts=2026-09-21T14:30:44Z level=info event=worker_gone addr=192.168.1.4:500514907worker1 # [ 80.937592] nix-grpc-daemon[768]: ts=2026-09-21T14:30:44Z level=info event=scheduler_restarting addr=lb:500514908worker1 # [ 80.942096] nix-grpc-daemon[768]: ts=2026-09-21T14:30:44Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:389404909worker1 # [ 80.947356] nix-grpc-daemon[768]: ts=2026-09-21T14:30:44Z level=info event=worker_gone addr=192.168.1.3:500514910worker1 # [ 80.973512] nix-daemon[851]: reaped child process 853, status = succeeded4911worker2 # [ 81.110844] nix-grpc-daemon[1034]: ts=2026-09-21T14:30:44Z level=info event=scheduler_connected addr=lb:500514912worker2 # [ 81.116177] nix-grpc-daemon[1034]: ts=2026-09-21T14:30:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4913worker1 # [ 81.149205] nix-grpc-daemon[768]: ts=2026-09-21T14:30:44Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"4914worker2 # [ 81.317587] nix-grpc-daemon[1034]: ts=2026-09-21T14:30:44Z level=info event=scheduler_connected addr=lb:500514915worker2 # [ 81.322918] nix-grpc-daemon[1034]: ts=2026-09-21T14:30:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4916worker1 # [ 81.363569] nix-grpc-daemon[768]: E0000 00:00:1790001044.967950 813 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 17, started true, shutdown true4917worker1 # [ 81.404478] nix-grpc-daemon[768]: E0000 00:00:1790001045.009059 813 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true4918worker2 # [ 81.523803] nix-grpc-daemon[1034]: ts=2026-09-21T14:30:45Z level=info event=scheduler_connected addr=lb:500514919worker2 # [ 81.529069] nix-grpc-daemon[1034]: ts=2026-09-21T14:30:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4920worker2 # [ 81.730136] nix-grpc-daemon[1034]: ts=2026-09-21T14:30:45Z level=info event=scheduler_connected addr=lb:500514921worker2 # [ 81.735382] nix-grpc-daemon[1034]: ts=2026-09-21T14:30:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4922worker1 # [ 81.878928] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.4923worker1 # [ 81.886832] systemd[1]: Stopped Nix worker-protocol over gRPC.4924worker1 # [ 81.893657] systemd[1]: nix-grpc-daemon.service: Consumed 4.032s CPU time over 1min 2.868s wall clock time, 68.6M memory peak, 100.5K incoming IP traffic, 208.6K outgoing IP traffic.4925worker1 # [ 81.913062] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.4926worker1 # [ 81.915166] systemd[1]: Closed Nix worker-protocol over gRPC.4927lb: waiting until exactly ['worker1'] endpoints of aarch64-linux are unhealthy4928lb: must succeed: curl -sf localhost:9901/clusters4929worker1 # [ 81.920619] systemd[1]: nix-grpc-daemon.socket: Consumed 561us CPU time over 1min 12.468s wall clock time, 256K memory peak, 98.2K incoming IP traffic, 80.2K outgoing IP traffic.4930worker2 # [ 81.938187] nix-grpc-daemon[1034]: ts=2026-09-21T14:30:45Z level=info event=scheduler_connected addr=lb:500514931worker2 # [ 81.942305] nix-grpc-daemon[1034]: ts=2026-09-21T14:30:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4932lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)4933lb: (finished: waiting until exactly ['worker1'] endpoints of aarch64-linux are unhealthy, in 0.08 seconds)4934client: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache >&24935worker2 # [ 82.139089] nix-grpc-daemon[1034]: ts=2026-09-21T14:30:45Z level=info event=scheduler_connected addr=lb:500514936worker2 # [ 82.144425] nix-grpc-daemon[1034]: ts=2026-09-21T14:30:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4937worker2 # [ 82.211031] nix-grpc-daemon[1034]: ts=2026-09-21T14:30:45Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:420564938worker2 # [ 82.224820] nix-daemon[838]: accepted connection from pid 1034, user nix-grpc-daemon (trusted)4939worker2 # [ 82.252827] nix-grpc-daemon[1034]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 45 ms (attempt 1/5)4940worker2 # [ 82.305517] nix-grpc-daemon[1034]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 71 ms (attempt 2/5)4941worker2 # [ 82.345471] nix-grpc-daemon[1034]: ts=2026-09-21T14:30:45Z level=info event=scheduler_connected addr=lb:500514942worker2 # [ 82.350550] nix-grpc-daemon[1034]: ts=2026-09-21T14:30:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4943worker2 # [ 82.385654] nix-grpc-daemon[1034]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 249 ms (attempt 3/5)4944worker2 # [ 82.551487] nix-grpc-daemon[1034]: ts=2026-09-21T14:30:46Z level=info event=scheduler_connected addr=lb:500514945worker2 # [ 82.557275] nix-grpc-daemon[1034]: ts=2026-09-21T14:30:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4946worker2 # [ 82.637045] nix-grpc-daemon[1034]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 479 ms (attempt 4/5)4947worker2 # [ 82.758453] nix-grpc-daemon[1034]: ts=2026-09-21T14:30:46Z level=info event=scheduler_connected addr=lb:500514948worker2 # [ 82.764197] nix-grpc-daemon[1034]: ts=2026-09-21T14:30:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4949worker2 # [ 82.967557] nix-grpc-daemon[1034]: ts=2026-09-21T14:30:46Z level=info event=scheduler_connected addr=lb:500514950worker2 # [ 82.971858] nix-grpc-daemon[1034]: ts=2026-09-21T14:30:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4951worker2 # [ 83.119303] nix-grpc-daemon[1034]: disabling binary cache 'https://cache.nixos.org' for 60 seconds4952worker2 # [ 83.125741] nix-grpc-daemon[1034]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4953client # copying 2 paths...4954client # copying path '/nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache' to 'grpc://lb:50051'...4955worker2 # [ 83.148084] nix-grpc-daemon[1034]: ts=2026-09-21T14:30:46Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:420564956client # copying path '/nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache' to 'grpc://lb:50051'...4957worker2 # [ 83.165483] nix-daemon[838]: accepted connection from pid 1034, user nix-grpc-daemon (trusted)4958worker2 # [ 83.171235] nix-grpc-daemon[1034]: ts=2026-09-21T14:30:46Z level=info event=scheduler_connected addr=lb:500514959worker2 # [ 83.178379] nix-grpc-daemon[1034]: ts=2026-09-21T14:30:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4960worker2 # [ 83.205280] nix-grpc-daemon[1034]: ts=2026-09-21T14:30:46Z level=info event=niks3_push_started pid=10944961worker2 # [ 83.363501] nix-grpc-daemon[1094]: time=2026-09-21T14:30:46.978Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.key ca=""4962worker2 # [ 83.378774] nix-grpc-daemon[1034]: ts=2026-09-21T14:30:46Z level=info event=scheduler_connected addr=lb:500514963worker2 # [ 83.381488] nix-grpc-daemon[1034]: ts=2026-09-21T14:30:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4964worker2 # [ 83.475325] nix-daemon[838]: accepted connection from pid 1107, user nix-grpc-daemon (trusted)4965worker2 # [ 83.491397] nix-daemon[838]: reaped child process 1110, status = succeeded4966lb # [ 83.506819] niks3-server[975]: 2026/09/21 14:30:47 INFO Received uploads request method=POST path=/api/pending_closures4967lb # [ 83.517516] niks3-server[975]: 2026/09/21 14:30:47 INFO Received uploads request method=POST path=/api/pending_closures4968worker2 # [ 83.521997] nix-grpc-daemon[1094]: time=2026-09-21T14:30:47.135Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"4969worker2 # [ 83.523475] nix-grpc-daemon[1094]: time=2026-09-21T14:30:47.135Z level=INFO msg="Uploading syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache (128B)"4970worker2 # [ 83.526898] nix-grpc-daemon[1094]: time=2026-09-21T14:30:47.136Z level=INFO msg="Uploading 4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache (184B)"4971lb # [ 83.573620] niks3-server[975]: 2026/09/21 14:30:47 INFO Registered completed upload object_key=nar/1kmnh5f2l7jjcgaq9b9988bjix6z8zjf6v350xllq5hlzgnb1f75.nar.zst4972worker2 # [ 83.582055] nix-grpc-daemon[1034]: ts=2026-09-21T14:30:47Z level=info event=scheduler_connected addr=lb:500514973lb # [ 83.593112] niks3-server[975]: 2026/09/21 14:30:47 INFO Registered completed upload object_key=nar/1rf1rg73pvnaa9kk6sy9r1ydj24b4g0kb0wnx0z5hxl1nvkmlavq.nar.zst4974worker2 # [ 83.586113] nix-grpc-daemon[1034]: ts=2026-09-21T14:30:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4975worker2 # [ 83.596204] nix-grpc-daemon[1094]: time=2026-09-21T14:30:47.210Z level=INFO msg="Uploading 3 narinfos"4976lb # [ 83.602843] niks3-server[975]: 2026/09/21 14:30:47 INFO Received sign narinfos request method=POST path=/api/pending_closures/25/sign4977lb # [ 83.610887] niks3-server[975]: 2026/09/21 14:30:47 INFO Signed narinfos id=25 count=14978lb # [ 83.614408] niks3-server[975]: 2026/09/21 14:30:47 INFO Registered completed upload object_key=syvivri0wisb7fd33rxcg9kz63kc2bv7.ls4979lb # [ 83.619810] niks3-server[975]: 2026/09/21 14:30:47 INFO Received sign narinfos request method=POST path=/api/pending_closures/26/sign4980lb # [ 83.626786] niks3-server[975]: 2026/09/21 14:30:47 INFO Signed narinfos id=26 count=24981lb # [ 83.632428] niks3-server[975]: 2026/09/21 14:30:47 INFO Registered completed upload object_key=4rig83zwv4xj1zajs08ima13w0yh676v.ls4982lb # [ 83.638412] niks3-server[975]: 2026/09/21 14:30:47 INFO Registered completed upload object_key=4rig83zwv4xj1zajs08ima13w0yh676v.narinfo4983lb # [ 83.644259] niks3-server[975]: 2026/09/21 14:30:47 INFO Registered completed upload object_key=syvivri0wisb7fd33rxcg9kz63kc2bv7.narinfo4984lb # [ 83.651809] niks3-server[975]: 2026/09/21 14:30:47 INFO Received complete upload request method=POST path=/api/pending_closures/25/complete4985lb # [ 83.658997] niks3-server[975]: 2026/09/21 14:30:47 INFO Completed upload id=254986worker2 # [ 83.652193] nix-grpc-daemon[1094]: time=2026-09-21T14:30:47.267Z level=INFO msg="Upload complete. (284ms)"4987lb # [ 83.661068] niks3-server[975]: 2026/09/21 14:30:47 INFO Received complete upload request method=POST path=/api/pending_closures/26/complete4988lb # [ 83.663533] niks3-server[975]: 2026/09/21 14:30:47 INFO Completed upload id=264989lb # [ 83.668374] niks3-server[975]: 2026/09/21 14:30:47 INFO Registered completed upload object_key=syvivri0wisb7fd33rxcg9kz63kc2bv7.narinfo4990worker2 # [ 83.662931] nix-grpc-daemon[1034]: ts=2026-09-21T14:30:47Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42056 duration_s=0 paths=2 nar_bytes_in=3124991worker2 # [ 83.680147] nix-daemon[838]: reaped child process 1092, status = succeeded4992client: (finished: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache >&2, in 1.71 seconds)4993worker2: must succeed: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache && test -e /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache4994worker2: (finished: must succeed: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache && test -e /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache, in 0.03 seconds)4995worker2 # [ 83.787178] nix-grpc-daemon[1034]: ts=2026-09-21T14:30:47Z level=info event=scheduler_connected addr=lb:500514996worker2 # [ 83.792512] nix-grpc-daemon[1034]: ts=2026-09-21T14:30:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4997worker1 # [ 83.812965] systemd[1]: Listening on Nix worker-protocol over gRPC.4998worker1 # [ 83.820414] systemd[1]: Starting Nix worker-protocol over gRPC...4999worker1 # [ 83.974838] nix-grpc-daemon[1267]: ts=2026-09-21T14:30:47Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5000worker1 # [ 83.977281] nix-grpc-daemon[1267]: ts=2026-09-21T14:30:47Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5001worker1 # [ 83.981310] nix-grpc-daemon[1267]: ts=2026-09-21T14:30:47Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15002worker1 # [ 83.986958] nix-grpc-daemon[1267]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5003worker1 # [ 84.004127] nix-grpc-daemon[1267]: E0000 00:00:1790001047.591057 1267 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5004worker2 # [ 83.995445] nix-grpc-daemon[1034]: ts=2026-09-21T14:30:47Z level=info event=scheduler_connected addr=lb:500515005worker1 # [ 84.006506] nix-grpc-daemon[1267]: E0000 00:00:1790001047.594695 1267 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5006lb # [ 84.006677] niks3-server[975]: 2026/09/21 14:30:47 INFO lead: acquired remote=192.168.1.3:534285007worker2 # [ 83.999768] nix-grpc-daemon[1034]: ts=2026-09-21T14:30:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5008lb: waiting until exactly no endpoints of aarch64-linux are unhealthy5009lb: must succeed: curl -sf localhost:9901/clusters5010worker1 # [ 84.042647] nix-grpc-daemon[1267]: ts=2026-09-21T14:30:47Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5011worker1 # [ 84.044859] nix-grpc-daemon[1267]: E0000 00:00:1790001047.608153 1308 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5012worker1 # [ 84.047592] systemd[1]: Started Nix worker-protocol over gRPC.5013worker1 # [ 84.048472] nix-grpc-daemon[1267]: ts=2026-09-21T14:30:47Z level=info event=scheduler_take_over5014worker1 # [ 84.049573] nix-grpc-daemon[1267]: E0000 00:00:1790001047.630918 1308 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5015worker1 # [ 84.051780] nix-grpc-daemon[1267]: E0000 00:00:1790001047.631133 1308 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5016worker1 # [ 84.054262] nix-grpc-daemon[1267]: E0000 00:00:1790001047.646587 1308 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5017worker1 # [ 84.062030] nix-grpc-daemon[1267]: ts=2026-09-21T14:30:47Z level=info event=scheduler_connected addr=lb:500515018worker1 # [ 84.068136] nix-grpc-daemon[1267]: ts=2026-09-21T14:30:47Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:354185019worker1 # [ 84.069845] nix-grpc-daemon[1267]: ts=2026-09-21T14:30:47Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=05020lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5021lb: (finished: waiting until exactly no endpoints of aarch64-linux are unhealthy, in 0.08 seconds)5022(finished: subtest: upload whose reference the worker lacks is completed from the cache, in 11.28 seconds)5023subtest: a niks3 restart does not cost the leader its role5024lb: must succeed: curl -sf localhost:9901/clusters5025lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5026lb: must succeed: curl -sf localhost:9901/clusters5027lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)5028worker2 # [ 84.196369] nix-grpc-daemon[1034]: ts=2026-09-21T14:30:47Z level=info event=scheduler_connected addr=lb:500515029worker1 # [ 84.212135] nix-grpc-daemon[1267]: ts=2026-09-21T14:30:47Z level=info event=oidc_jwks provider=mock keys=15030worker1 # [ 84.217445] nix-grpc-daemon[1267]: ts=2026-09-21T14:30:47Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:354185031worker1 # [ 84.223184] nix-grpc-daemon[1267]: ts=2026-09-21T14:30:47Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=05032worker1 # [ 84.239624] nix-grpc-daemon[1267]: ts=2026-09-21T14:30:47Z level=info event=scheduler_lock_lost status=2005033worker2 # [ 84.232803] nix-grpc-daemon[1034]: ts=2026-09-21T14:30:47Z level=info event=scheduler_lock_lost status=2005034lb # [ 84.238644] niks3-server[975]: 2026/09/21 14:30:47 INFO Shutdown signal received, draining in-flight requests timeout=10s5035lb # [ 84.245498] niks3-server[975]: 2026/09/21 14:30:47 INFO lead: released remote=192.168.1.3:534285036lb # [ 84.247412] systemd[1]: Stopping niks3 server...5037lb # [ 89.017318] envoy[909]: conn peer=[2001:db8:1::3]:51318 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=11907 tx=3300 ms=618405038lb # [ 89.023547] envoy[909]: conn peer=[2001:db8:1::1]:37512 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1007 tx=304 ms=14705039lb # [ 89.042256] systemd[1]: niks3.service: Deactivated successfully.5040lb # [ 89.047379] systemd[1]: Stopped niks3 server.5041lb # [ 89.052486] systemd[1]: niks3.service: Consumed 503ms CPU time over 1min 5.415s wall clock time, 25.6M memory peak, 72.1K incoming IP traffic, 102K outgoing IP traffic.5042lb # [ 89.077954] systemd[1]: Starting niks3 server...5043lb # [ 89.179275] niks3-server[1111]: 2026/09/21 14:30:52 goose: no migrations to run. current version: 202609200000005044lb # [ 89.185383] niks3-server[1111]: 2026/09/21 14:30:52 OK 1_commit_pending_closure.sql (8.32ms)5045lb # [ 89.190262] niks3-server[1111]: 2026/09/21 14:30:52 OK 2_object_stats_trigger.sql (2.67ms)5046lb # [ 89.191808] niks3-server[1111]: 2026/09/21 14:30:52 goose: up to current file version: 25047lb # [ 89.194868] niks3-server[1111]: 2026/09/21 14:30:52 INFO Loaded signing key name=farm-test-1 path=/nix/store/3qiyvklx8qgcqqsb1swilx6drs1cdj8m-key5048lb # [ 89.207577] niks3-server[1111]: 2026/09/21 14:30:52 INFO Read proxy enabled — serving cache objects from S35049lb # [ 89.214688] niks3-server[1111]: 2026/09/21 14:30:52 INFO Using socket-activated listener address=0.0.0.0:57515050lb # [ 89.217123] niks3-server[1111]: 2026/09/21 14:30:52 INFO systemd watchdog enabled interval=15s5051lb # [ 89.219630] niks3-server[1111]: 2026/09/21 14:30:52 INFO Starting HTTP server address=0.0.0.0:57515052lb # [ 89.221886] niks3-server[1111]: 2026/09/21 14:30:52 INFO lead: acquired remote=192.168.1.3:534385053lb # [ 89.223258] systemd[1]: Started niks3 server.5054lb: waiting for unit niks3.service5055lb: (finished: waiting for unit niks3.service, in 0.03 seconds)5056lb: must succeed: curl -sf localhost:9901/clusters5057lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)5058worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5059worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5060lb: must succeed: curl -sf localhost:9901/clusters5061lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5062lb: must succeed: curl -sf localhost:9901/clusters5063lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5064worker1: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield5065worker1: (finished: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield, in 0.07 seconds)5066client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag after-niks3-restart >&25067worker2 # [ 89.728884] nix-grpc-daemon[1034]: ts=2026-09-21T14:30:53Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:420565068client # don't know how to build these paths:5069client # /nix/store/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv5070worker1 # [ 89.753975] nix-grpc-daemon[1267]: ts=2026-09-21T14:30:53Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:354285071worker1 # [ 89.767060] nix-daemon[851]: accepted connection from pid 1267, user nix-grpc-daemon (trusted)5072worker1 # [ 89.791465] nix-grpc-daemon[1267]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 77 ms (attempt 1/5)5073worker1 # [ 89.877315] nix-grpc-daemon[1267]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 138 ms (attempt 2/5)5074worker1 # [ 90.017397] nix-grpc-daemon[1267]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 16 ms (attempt 3/5)5075worker1 # [ 90.035082] nix-grpc-daemon[1267]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 663 ms (attempt 4/5)5076worker1 # [ 90.708000] nix-grpc-daemon[1267]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5077worker1 # [ 90.709108] nix-grpc-daemon[1267]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5078worker1 # [ 90.759501] nix-grpc-daemon[1267]: ts=2026-09-21T14:30:54Z level=debug event=want client=1 drv=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv cached=05079worker1 # [ 90.766088] nix-grpc-daemon[1267]: ts=2026-09-21T14:30:54Z level=debug event=assigned drv=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv worker=192.168.1.3:50051 clients=15080worker1 # [ 90.813435] nix-grpc-daemon[1267]: ts=2026-09-21T14:30:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:354285081worker1 # [ 90.829174] nix-grpc-daemon[1267]: ts=2026-09-21T14:30:54Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:354285082client # copying 1 paths to 'lb:50051'...5083client # copying path '/nix/store/mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv' to 'lb:50051'...5084worker1 # [ 90.843330] nix-grpc-daemon[1267]: ts=2026-09-21T14:30:54Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:354285085worker1 # [ 90.858903] nix-daemon[851]: accepted connection from pid 1267, user nix-grpc-daemon (trusted)5086worker1 # [ 90.887201] nix-grpc-daemon[1267]: ts=2026-09-21T14:30:54Z level=info event=niks3_push_started pid=13395087worker1 # [ 91.034711] nix-grpc-daemon[1339]: time=2026-09-21T14:30:54.638Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.key ca=""5088worker1 # [ 91.145059] nix-daemon[851]: accepted connection from pid 1351, user nix-grpc-daemon (trusted)5089worker1 # [ 91.164260] nix-daemon[851]: reaped child process 1354, status = succeeded5090worker1 # [ 91.262095] nix-daemon[851]: accepted connection from pid 1356, user nix-grpc-daemon (trusted)5091worker1 # [ 91.280432] nix-daemon[851]: reaped child process 1359, status = succeeded5092lb # [ 91.278713] niks3-server[1111]: 2026/09/21 14:30:54 INFO Received uploads request method=POST path=/api/pending_closures5093worker1 # [ 91.296123] nix-grpc-daemon[1339]: time=2026-09-21T14:30:54.900Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5094worker1 # [ 91.297554] nix-grpc-daemon[1339]: time=2026-09-21T14:30:54.900Z level=INFO msg="Uploading mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv (544B)"5095lb # [ 91.339810] niks3-server[1111]: 2026/09/21 14:30:54 INFO Received sign narinfos request method=POST path=/api/pending_closures/27/sign5096lb # [ 91.359232] niks3-server[1111]: 2026/09/21 14:30:54 INFO Registered completed upload object_key=nar/0hn539w6achi7azzyhpihsi61g85x9f535dxv438yq5y50ksrp1d.nar.zst5097lb # [ 91.379782] niks3-server[1111]: 2026/09/21 14:30:54 INFO Signed narinfos id=27 count=15098worker1 # [ 91.383363] nix-grpc-daemon[1339]: time=2026-09-21T14:30:54.987Z level=INFO msg="Uploading 1 narinfos"5099lb # [ 91.392077] niks3-server[1111]: 2026/09/21 14:30:55 INFO Registered completed upload object_key=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z.ls5100lb # [ 91.400966] niks3-server[1111]: 2026/09/21 14:30:55 INFO Received complete upload request method=POST path=/api/pending_closures/27/complete5101lb # [ 91.407554] niks3-server[1111]: 2026/09/21 14:30:55 INFO Completed upload id=275102worker1 # [ 91.406780] nix-grpc-daemon[1339]: time=2026-09-21T14:30:55.011Z level=INFO msg="Upload complete. (368ms)"5103lb # [ 91.414765] niks3-server[1111]: 2026/09/21 14:30:55 INFO Registered completed upload object_key=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z.narinfo5104worker1 # [ 91.418898] nix-grpc-daemon[1267]: ts=2026-09-21T14:30:55Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35428 duration_s=0 paths=1 nar_bytes_in=5445105worker1 # [ 91.438848] nix-grpc-daemon[1267]: ts=2026-09-21T14:30:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:354285106worker1 # [ 91.443079] nix-daemon[851]: accepted connection from pid 1267, user nix-grpc-daemon (trusted)5107worker1 # [ 91.452241] nix-daemon[851]: reaped child process 1337, status = succeeded5108client # building '/nix/store/mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv' on lb:50051...5109worker1 # [ 91.462021] nix-daemon[851]: accepted connection from pid 1267, user nix-grpc-daemon (trusted)5110client # farm-a> node-a: building mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv5111client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds5112client # farm-a> LOG-farm-a-after-niks3-restart5113worker1 # [ 93.128566] nix-daemon[851]: reaped child process 1363, status = succeeded5114worker1 # [ 93.258773] nix-daemon[851]: accepted connection from pid 1379, user nix-grpc-daemon (trusted)5115worker1 # [ 93.275682] nix-daemon[851]: reaped child process 1382, status = succeeded5116lb # [ 93.281714] niks3-server[1111]: 2026/09/21 14:30:56 INFO Received uploads request method=POST path=/api/pending_closures5117worker1 # [ 93.300336] nix-grpc-daemon[1339]: time=2026-09-21T14:30:56.900Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5118worker1 # [ 93.302309] nix-grpc-daemon[1339]: time=2026-09-21T14:30:56.900Z level=INFO msg="Uploading j3iswx3ynilvcycpyw0z667966c9kpzx-farm-a (144B)"5119lb # [ 93.320593] niks3-server[1111]: 2026/09/21 14:30:56 INFO Registered completed upload object_key=log/mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv5120lb # [ 93.339080] niks3-server[1111]: 2026/09/21 14:30:56 INFO Registered completed upload object_key=nar/1diw08qzlsh0zkrx4kvj1mvbc57g7r65j1f459ankpg9yc5rgcpx.nar.zst5121lb # [ 93.352577] niks3-server[1111]: 2026/09/21 14:30:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/28/sign5122lb # [ 93.360500] niks3-server[1111]: 2026/09/21 14:30:56 INFO Signed narinfos id=28 count=15123worker1 # [ 93.359982] nix-grpc-daemon[1339]: time=2026-09-21T14:30:56.963Z level=INFO msg="Uploading 1 narinfos"5124lb # [ 93.365572] niks3-server[1111]: 2026/09/21 14:30:56 INFO Registered completed upload object_key=j3iswx3ynilvcycpyw0z667966c9kpzx.ls5125lb # [ 93.376213] niks3-server[1111]: 2026/09/21 14:30:56 INFO Received complete upload request method=POST path=/api/pending_closures/28/complete5126lb # [ 93.382144] niks3-server[1111]: 2026/09/21 14:30:57 INFO Completed upload id=285127worker1 # [ 93.382528] nix-grpc-daemon[1339]: time=2026-09-21T14:30:56.987Z level=INFO msg="Upload complete. (309ms)"5128lb # [ 93.384650] niks3-server[1111]: 2026/09/21 14:30:57 INFO Registered completed upload object_key=j3iswx3ynilvcycpyw0z667966c9kpzx.narinfo5129worker1 # [ 93.404710] nix-grpc-daemon[1267]: ts=2026-09-21T14:30:56Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35428 duration_s=1 drv=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv assign_id=1 outputs=15130worker1 # [ 93.420309] nix-grpc-daemon[1267]: ts=2026-09-21T14:30:57Z level=debug event=want client=1 drv=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv cached=05131worker1 # [ 93.423975] nix-grpc-daemon[1267]: ts=2026-09-21T14:30:57Z level=debug event=assigned drv=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv worker=192.168.1.3:50051 clients=15132worker1 # [ 93.428508] nix-daemon[851]: reaped child process 1361, status = succeeded5133worker1 # [ 93.448760] nix-grpc-daemon[1267]: ts=2026-09-21T14:30:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:354285134worker1 # [ 93.463843] nix-grpc-daemon[1267]: ts=2026-09-21T14:30:57Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:354285135worker1 # [ 93.478786] nix-grpc-daemon[1339]: time=2026-09-21T14:30:57.083Z level=INFO msg="All 1 paths already cached"5136client # copying 1 paths to 'lb:50051'...5137client # copying path '/nix/store/5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv' to 'lb:50051'...5138worker1 # [ 93.488670] nix-grpc-daemon[1267]: ts=2026-09-21T14:30:57Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:354285139worker1 # [ 93.502611] nix-daemon[851]: accepted connection from pid 1267, user nix-grpc-daemon (trusted)5140worker1 # [ 93.654773] nix-daemon[851]: accepted connection from pid 1388, user nix-grpc-daemon (trusted)5141worker1 # [ 93.673285] nix-daemon[851]: reaped child process 1391, status = succeeded5142worker1 # [ 93.774391] nix-daemon[851]: accepted connection from pid 1393, user nix-grpc-daemon (trusted)5143worker1 # [ 93.791862] nix-daemon[851]: reaped child process 1396, status = succeeded5144lb # [ 93.795205] niks3-server[1111]: 2026/09/21 14:30:57 INFO Received uploads request method=POST path=/api/pending_closures5145worker1 # [ 93.805171] nix-grpc-daemon[1339]: time=2026-09-21T14:30:57.409Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5146worker1 # [ 93.806638] nix-grpc-daemon[1339]: time=2026-09-21T14:30:57.409Z level=INFO msg="Uploading 5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv (656B)"5147lb # [ 93.822299] niks3-server[1111]: 2026/09/21 14:30:57 INFO Registered completed upload object_key=nar/1xpcisa10g5by3yknnrs1xwhxhf4s243h7gwzjii0799pwbhfjfa.nar.zst5148worker1 # [ 93.837448] nix-grpc-daemon[1339]: time=2026-09-21T14:30:57.441Z level=INFO msg="Uploading 1 narinfos"5149lb # [ 93.836953] niks3-server[1111]: 2026/09/21 14:30:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/29/sign5150lb # [ 93.849695] niks3-server[1111]: 2026/09/21 14:30:57 INFO Signed narinfos id=29 count=15151lb # [ 93.858073] niks3-server[1111]: 2026/09/21 14:30:57 INFO Registered completed upload object_key=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch.ls5152lb # [ 93.874718] niks3-server[1111]: 2026/09/21 14:30:57 INFO Received complete upload request method=POST path=/api/pending_closures/29/complete5153lb # [ 93.883721] niks3-server[1111]: 2026/09/21 14:30:57 INFO Completed upload id=295154worker1 # [ 93.884502] nix-grpc-daemon[1339]: time=2026-09-21T14:30:57.489Z level=INFO msg="Upload complete. (366ms)"5155lb # [ 93.887256] niks3-server[1111]: 2026/09/21 14:30:57 INFO Registered completed upload object_key=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch.narinfo5156worker1 # [ 93.895563] nix-grpc-daemon[1267]: ts=2026-09-21T14:30:57Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35428 duration_s=0 paths=1 nar_bytes_in=6565157worker1 # [ 93.908599] nix-grpc-daemon[1267]: ts=2026-09-21T14:30:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:354285158worker1 # [ 93.917089] nix-daemon[851]: accepted connection from pid 1267, user nix-grpc-daemon (trusted)5159worker1 # [ 93.922956] nix-daemon[851]: reaped child process 1386, status = succeeded5160client # building '/nix/store/5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv' on lb:50051...5161client # farm-b> node-a: building 5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv5162worker1 # [ 93.936161] nix-daemon[851]: accepted connection from pid 1267, user nix-grpc-daemon (trusted)5163client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds5164client # farm-b> LOG-farm-b-after-niks3-restart5165worker1 # [ 94.966615] nix-daemon[851]: reaped child process 1400, status = succeeded5166worker1 # [ 95.082285] nix-daemon[851]: accepted connection from pid 1416, user nix-grpc-daemon (trusted)5167worker1 # [ 95.103090] nix-daemon[851]: reaped child process 1419, status = succeeded5168lb # [ 95.106869] niks3-server[1111]: 2026/09/21 14:30:58 INFO Received uploads request method=POST path=/api/pending_closures5169worker1 # [ 95.121150] nix-grpc-daemon[1339]: time=2026-09-21T14:30:58.724Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5170worker1 # [ 95.122710] nix-grpc-daemon[1339]: time=2026-09-21T14:30:58.725Z level=INFO msg="Uploading myar0ljvsqm9h7fxyh6qrqkg0175r3xn-farm-b (192B)"5171lb # [ 95.152266] niks3-server[1111]: 2026/09/21 14:30:58 INFO Registered completed upload object_key=log/5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv5172lb # [ 95.161431] niks3-server[1111]: 2026/09/21 14:30:58 INFO Registered completed upload object_key=nar/0zvv5rxmyk9afay83j52m7ipyhrqivhg6h6a0ksjd658pxzwi78w.nar.zst5173worker1 # [ 95.172795] nix-grpc-daemon[1339]: time=2026-09-21T14:30:58.777Z level=INFO msg="Uploading 1 narinfos"5174lb # [ 95.173030] niks3-server[1111]: 2026/09/21 14:30:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/30/sign5175lb # [ 95.179944] niks3-server[1111]: 2026/09/21 14:30:58 INFO Signed narinfos id=30 count=15176lb # [ 95.186922] niks3-server[1111]: 2026/09/21 14:30:58 INFO Registered completed upload object_key=myar0ljvsqm9h7fxyh6qrqkg0175r3xn.ls5177worker1 # [ 95.191938] nix-grpc-daemon[1339]: time=2026-09-21T14:30:58.796Z level=INFO msg="Upload complete. (245ms)"5178lb # [ 95.194001] niks3-server[1111]: 2026/09/21 14:30:58 INFO Received complete upload request method=POST path=/api/pending_closures/30/complete5179lb # [ 95.198494] niks3-server[1111]: 2026/09/21 14:30:58 INFO Completed upload id=305180lb # [ 95.200966] niks3-server[1111]: 2026/09/21 14:30:58 INFO Registered completed upload object_key=myar0ljvsqm9h7fxyh6qrqkg0175r3xn.narinfo5181worker1 # [ 95.208765] nix-grpc-daemon[1267]: ts=2026-09-21T14:30:58Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35428 duration_s=1 drv=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv assign_id=2 outputs=15182worker1 # [ 95.224659] nix-grpc-daemon[1267]: ts=2026-09-21T14:30:58Z level=debug event=want client=1 drv=6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv cached=05183worker1 # [ 95.229298] nix-grpc-daemon[1267]: ts=2026-09-21T14:30:58Z level=debug event=assigned drv=6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv worker=192.168.1.3:50051 clients=15184worker1 # [ 95.234148] nix-daemon[851]: reaped child process 1398, status = succeeded5185worker1 # [ 95.252506] nix-grpc-daemon[1267]: ts=2026-09-21T14:30:58Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:354285186worker1 # [ 95.266563] nix-grpc-daemon[1267]: ts=2026-09-21T14:30:58Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:354285187worker1 # [ 95.279881] nix-grpc-daemon[1339]: time=2026-09-21T14:30:58.884Z level=INFO msg="All 2 paths already cached"5188client # copying 1 paths to 'lb:50051'...5189client # copying path '/nix/store/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv' to 'lb:50051'...5190worker1 # [ 95.290360] nix-grpc-daemon[1267]: ts=2026-09-21T14:30:58Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:354285191worker1 # [ 95.303984] nix-daemon[851]: accepted connection from pid 1267, user nix-grpc-daemon (trusted)5192worker1 # [ 95.451529] nix-daemon[851]: accepted connection from pid 1425, user nix-grpc-daemon (trusted)5193worker1 # [ 95.468711] nix-daemon[851]: reaped child process 1428, status = succeeded5194worker1 # [ 95.572159] nix-daemon[851]: accepted connection from pid 1430, user nix-grpc-daemon (trusted)5195worker1 # [ 95.588899] nix-daemon[851]: reaped child process 1433, status = succeeded5196lb # [ 95.593921] niks3-server[1111]: 2026/09/21 14:30:59 INFO Received uploads request method=POST path=/api/pending_closures5197worker1 # [ 95.603784] nix-grpc-daemon[1339]: time=2026-09-21T14:30:59.208Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5198worker1 # [ 95.605435] nix-grpc-daemon[1339]: time=2026-09-21T14:30:59.208Z level=INFO msg="Uploading 6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv (784B)"5199lb # [ 95.622957] niks3-server[1111]: 2026/09/21 14:30:59 INFO Registered completed upload object_key=nar/1fnvw3c0v6n1j60nd5iagl3bggzcvixsw103w98mwhnpmnmgb248.nar.zst5200worker1 # [ 95.638015] nix-grpc-daemon[1339]: time=2026-09-21T14:30:59.242Z level=INFO msg="Uploading 1 narinfos"5201lb # [ 95.637650] niks3-server[1111]: 2026/09/21 14:30:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/31/sign5202lb # [ 95.649155] niks3-server[1111]: 2026/09/21 14:30:59 INFO Signed narinfos id=31 count=15203lb # [ 95.657943] niks3-server[1111]: 2026/09/21 14:30:59 INFO Registered completed upload object_key=6j7gr603xfslr5lxcbh72430f5q1r4b0.ls5204worker1 # [ 95.668489] nix-grpc-daemon[1339]: time=2026-09-21T14:30:59.273Z level=INFO msg="Upload complete. (349ms)"5205lb # [ 95.669432] niks3-server[1111]: 2026/09/21 14:30:59 INFO Received complete upload request method=POST path=/api/pending_closures/31/complete5206lb # [ 95.675494] niks3-server[1111]: 2026/09/21 14:30:59 INFO Completed upload id=315207worker1 # [ 95.675768] nix-grpc-daemon[1267]: ts=2026-09-21T14:30:59Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35428 duration_s=0 paths=1 nar_bytes_in=7845208lb # [ 95.679462] niks3-server[1111]: 2026/09/21 14:30:59 INFO Registered completed upload object_key=6j7gr603xfslr5lxcbh72430f5q1r4b0.narinfo5209worker1 # [ 95.687570] nix-grpc-daemon[1267]: ts=2026-09-21T14:30:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:354285210worker1 # [ 95.695834] nix-daemon[851]: accepted connection from pid 1267, user nix-grpc-daemon (trusted)5211worker1 # [ 95.699064] nix-daemon[851]: reaped child process 1423, status = succeeded5212client # building '/nix/store/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv' on lb:50051...5213worker1 # [ 95.709885] nix-daemon[851]: accepted connection from pid 1267, user nix-grpc-daemon (trusted)5214client # farm-top> node-a: building 6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv5215client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds5216client # farm-top> LOG-farm-top-after-niks3-restart5217worker1 # [ 97.010147] nix-daemon[851]: reaped child process 1437, status = succeeded5218worker1 # [ 97.077497] nix-daemon[851]: accepted connection from pid 1453, user nix-grpc-daemon (trusted)5219worker1 # [ 97.098785] nix-daemon[851]: reaped child process 1456, status = succeeded5220lb # [ 97.104739] niks3-server[1111]: 2026/09/21 14:31:00 INFO Received uploads request method=POST path=/api/pending_closures5221worker1 # [ 97.118717] nix-grpc-daemon[1339]: time=2026-09-21T14:31:00.721Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5222worker1 # [ 97.120390] nix-grpc-daemon[1339]: time=2026-09-21T14:31:00.721Z level=INFO msg="Uploading z8mx9hni3m7cymmz0ddfhg3jhqvi9mlc-farm-top (248B)"5223lb # [ 97.147298] niks3-server[1111]: 2026/09/21 14:31:00 INFO Registered completed upload object_key=log/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv5224lb # [ 97.158027] niks3-server[1111]: 2026/09/21 14:31:00 INFO Registered completed upload object_key=nar/0cypkbh15calmx90jgh2waxpq661jalf4c99dgb7j606z61xh0ld.nar.zst5225lb # [ 97.168454] niks3-server[1111]: 2026/09/21 14:31:00 INFO Received sign narinfos request method=POST path=/api/pending_closures/32/sign5226worker1 # [ 97.176425] nix-grpc-daemon[1339]: time=2026-09-21T14:31:00.780Z level=INFO msg="Uploading 1 narinfos"5227lb # [ 97.177699] niks3-server[1111]: 2026/09/21 14:31:00 INFO Signed narinfos id=32 count=15228lb # [ 97.183512] niks3-server[1111]: 2026/09/21 14:31:00 INFO Registered completed upload object_key=z8mx9hni3m7cymmz0ddfhg3jhqvi9mlc.ls5229lb # [ 97.190921] niks3-server[1111]: 2026/09/21 14:31:00 INFO Received complete upload request method=POST path=/api/pending_closures/32/complete5230lb # [ 97.199900] niks3-server[1111]: 2026/09/21 14:31:00 INFO Completed upload id=325231worker1 # [ 97.200128] nix-grpc-daemon[1339]: time=2026-09-21T14:31:00.804Z level=INFO msg="Upload complete. (277ms)"5232lb # [ 97.202893] niks3-server[1111]: 2026/09/21 14:31:00 INFO Registered completed upload object_key=z8mx9hni3m7cymmz0ddfhg3jhqvi9mlc.narinfo5233worker1 # [ 97.212685] nix-grpc-daemon[1267]: ts=2026-09-21T14:31:00Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35428 duration_s=1 drv=6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv assign_id=3 outputs=15234worker1 # [ 97.232403] nix-daemon[851]: reaped child process 1435, status = succeeded5235client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag after-niks3-restart >&2, in 7.75 seconds)5236client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag after-niks3-restart outPath5237client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag after-niks3-restart outPath, in 0.18 seconds)5238(finished: subtest: a niks3 restart does not cost the leader its role, in 13.33 seconds)5239subtest: scheduler down: the other node takes the lock and keeps it5240lb: must succeed: curl -sf localhost:9901/clusters5241lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5242lb: must succeed: curl -sf localhost:9901/clusters5243lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5244lb: must succeed: curl -sf localhost:9901/clusters5245lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5246lb: must succeed: curl -sf localhost:9901/clusters5247lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)5248worker1 # [ 97.683704] systemd[1]: Stopping Nix worker-protocol over gRPC...5249worker1 # [ 98.000667] nix-grpc-daemon[1267]: ts=2026-09-21T14:31:01Z level=info event=signal_exit5250lb # [ 98.236643] niks3-server[1111]: 2026/09/21 14:31:01 INFO lead: released remote=192.168.1.3:534385251lb # [ 99.016787] envoy[909]: conn peer=[2001:db8:1::1]:37520 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9740 tx=4153 ms=74915252lb # [ 99.212811] niks3-server[1111]: 2026/09/21 14:31:02 INFO lead: acquired remote=192.168.1.4:399825253worker2 # [ 99.211567] nix-grpc-daemon[1034]: ts=2026-09-21T14:31:02Z level=info event=scheduler_take_over5254worker2 # [ 99.224319] nix-grpc-daemon[1034]: ts=2026-09-21T14:31:02Z level=info event=scheduler_restarting addr=lb:500515255worker1 # [ 99.237664] nix-grpc-daemon[1267]: ts=2026-09-21T14:31:02Z level=info event=scheduler_restarting5256worker1 # [ 99.241454] nix-grpc-daemon[1267]: ts=2026-09-21T14:31:02Z level=info event=scheduler_restarting addr=lb:500515257worker1 # [ 99.246007] nix-grpc-daemon[1267]: ts=2026-09-21T14:31:02Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:354185258worker1 # [ 99.251123] nix-grpc-daemon[1267]: ts=2026-09-21T14:31:02Z level=info event=worker_gone addr=192.168.1.3:500515259worker1 # [ 99.255623] nix-grpc-daemon[1267]: ts=2026-09-21T14:31:02Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:354185260worker1 # [ 99.260862] nix-grpc-daemon[1267]: ts=2026-09-21T14:31:02Z level=info event=worker_gone addr=192.168.1.4:500515261worker1 # [ 99.284906] nix-daemon[851]: reaped child process 1329, status = succeeded5262worker2 # [ 99.425564] nix-grpc-daemon[1034]: ts=2026-09-21T14:31:03Z level=info event=scheduler_connected addr=lb:500515263worker1 # [ 99.451403] nix-grpc-daemon[1267]: ts=2026-09-21T14:31:03Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5264worker2 # [ 99.451754] nix-grpc-daemon[1034]: ts=2026-09-21T14:31:03Z level=info event=oidc_jwks provider=mock keys=15265worker2 # [ 99.456411] nix-grpc-daemon[1034]: ts=2026-09-21T14:31:03Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:549685266worker2 # [ 99.461835] nix-grpc-daemon[1034]: ts=2026-09-21T14:31:03Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=05267worker1 # [ 100.070869] nix-grpc-daemon[1267]: E0000 00:00:1790001063.675295 1312 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 17, started true, shutdown true5268worker1 # [ 100.077294] nix-grpc-daemon[1267]: E0000 00:00:1790001063.675520 1312 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5269worker1 # [ 101.391342] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5270worker1 # [ 101.396113] systemd[1]: Stopped Nix worker-protocol over gRPC.5271worker1 # [ 101.403666] systemd[1]: nix-grpc-daemon.service: Consumed 1.488s CPU time over 17.574s wall clock time, 38.3M memory peak, 36K incoming IP traffic, 56.5K outgoing IP traffic.5272worker1 # [ 101.415969] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5273worker1 # [ 101.425077] systemd[1]: Closed Nix worker-protocol over gRPC.5274worker1 # [ 101.429143] systemd[1]: nix-grpc-daemon.socket: Consumed 1ms CPU time over 17.618s wall clock time, 256K memory peak, 37K incoming IP traffic, 27.5K outgoing IP traffic.5275worker2: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over5276worker2: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over, in 0.08 seconds)5277worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5278worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)5279client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag on-standby >&25280client # don't know how to build these paths:5281client # /nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv5282worker2 # [ 101.767057] nix-grpc-daemon[1034]: ts=2026-09-21T14:31:05Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:420565283worker2 # [ 101.774643] nix-grpc-daemon[1034]: ts=2026-09-21T14:31:05Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:420565284worker2 # [ 101.803135] nix-grpc-daemon[1034]: ts=2026-09-21T14:31:05Z level=debug event=want client=1 drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv cached=05285worker2 # [ 101.810510] nix-grpc-daemon[1034]: ts=2026-09-21T14:31:05Z level=debug event=assigned drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv worker=192.168.1.4:50051 clients=15286worker2 # [ 101.850096] nix-grpc-daemon[1034]: ts=2026-09-21T14:31:05Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:420565287worker2 # [ 101.866227] nix-grpc-daemon[1034]: ts=2026-09-21T14:31:05Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:420565288client # copying 1 paths to 'lb:50051'...5289client # copying path '/nix/store/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv' to 'lb:50051'...5290worker2 # [ 101.882625] nix-grpc-daemon[1034]: ts=2026-09-21T14:31:05Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:420565291worker2 # [ 101.897822] nix-daemon[838]: accepted connection from pid 1034, user nix-grpc-daemon (trusted)5292worker2 # [ 102.065037] nix-daemon[838]: accepted connection from pid 1130, user nix-grpc-daemon (trusted)5293worker2 # [ 102.085734] nix-daemon[838]: reaped child process 1133, status = succeeded5294worker2 # [ 102.202294] nix-daemon[838]: accepted connection from pid 1135, user nix-grpc-daemon (trusted)5295worker2 # [ 102.222274] nix-daemon[838]: reaped child process 1138, status = succeeded5296lb # [ 102.233127] niks3-server[1111]: 2026/09/21 14:31:05 INFO Received uploads request method=POST path=/api/pending_closures5297worker2 # [ 102.237297] nix-grpc-daemon[1094]: time=2026-09-21T14:31:05.852Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5298worker2 # [ 102.238806] nix-grpc-daemon[1094]: time=2026-09-21T14:31:05.852Z level=INFO msg="Uploading adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv (520B)"5299lb # [ 102.269778] niks3-server[1111]: 2026/09/21 14:31:05 INFO Registered completed upload object_key=nar/0scd0jwin78ibwq92wkajw2zj7adk8bih7yhx8bxz1706hh53507.nar.zst5300worker2 # [ 102.274363] nix-grpc-daemon[1094]: time=2026-09-21T14:31:05.889Z level=INFO msg="Uploading 1 narinfos"5301lb # [ 102.281898] niks3-server[1111]: 2026/09/21 14:31:05 INFO Received sign narinfos request method=POST path=/api/pending_closures/33/sign5302lb # [ 102.293109] niks3-server[1111]: 2026/09/21 14:31:05 INFO Signed narinfos id=33 count=15303lb # [ 102.301849] niks3-server[1111]: 2026/09/21 14:31:05 INFO Registered completed upload object_key=adj3g5w8jh83wl959p7daq6igl0gaay3.ls5304lb # [ 102.312311] niks3-server[1111]: 2026/09/21 14:31:05 INFO Received complete upload request method=POST path=/api/pending_closures/33/complete5305lb # [ 102.322315] niks3-server[1111]: 2026/09/21 14:31:05 INFO Completed upload id=335306worker2 # [ 102.316316] nix-grpc-daemon[1094]: time=2026-09-21T14:31:05.931Z level=INFO msg="Upload complete. (401ms)"5307lb # [ 102.327115] niks3-server[1111]: 2026/09/21 14:31:05 INFO Registered completed upload object_key=adj3g5w8jh83wl959p7daq6igl0gaay3.narinfo5308worker2 # [ 102.327513] nix-grpc-daemon[1034]: ts=2026-09-21T14:31:05Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42056 duration_s=0 paths=1 nar_bytes_in=5205309worker2 # [ 102.339812] nix-grpc-daemon[1034]: ts=2026-09-21T14:31:05Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:420565310worker2 # [ 102.349677] nix-daemon[838]: accepted connection from pid 1034, user nix-grpc-daemon (trusted)5311worker2 # [ 102.352361] nix-daemon[838]: reaped child process 1128, status = succeeded5312client # building '/nix/store/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv' on lb:50051...5313worker2 # [ 102.364347] nix-daemon[838]: accepted connection from pid 1034, user nix-grpc-daemon (trusted)5314client # farm-a> worker2: building adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv5315client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds5316client # farm-a> LOG-farm-a-on-standby5317worker2 # [ 103.407628] nix-daemon[838]: reaped child process 1142, status = succeeded5318worker2 # [ 103.551879] nix-daemon[838]: accepted connection from pid 1158, user nix-grpc-daemon (trusted)5319worker2 # [ 103.579766] nix-daemon[838]: reaped child process 1161, status = succeeded5320lb # [ 103.586532] niks3-server[1111]: 2026/09/21 14:31:07 INFO Received uploads request method=POST path=/api/pending_closures5321worker2 # [ 103.592427] nix-grpc-daemon[1094]: time=2026-09-21T14:31:07.205Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5322worker2 # [ 103.593881] nix-grpc-daemon[1094]: time=2026-09-21T14:31:07.205Z level=INFO msg="Uploading 7qyaw17jjsjxkkgb1kfm3wxvczk56aan-farm-a (136B)"5323lb # [ 103.623710] niks3-server[1111]: 2026/09/21 14:31:07 INFO Registered completed upload object_key=log/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv5324lb # [ 103.639645] niks3-server[1111]: 2026/09/21 14:31:07 INFO Registered completed upload object_key=nar/0x76wzgdd1nk8rm930z8l69v9961jpc8qhzmgganv0nhagq8lv9c.nar.zst5325worker2 # [ 103.646313] nix-grpc-daemon[1094]: time=2026-09-21T14:31:07.260Z level=INFO msg="Uploading 1 narinfos"5326lb # [ 103.652266] niks3-server[1111]: 2026/09/21 14:31:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/34/sign5327lb # [ 103.655694] niks3-server[1111]: 2026/09/21 14:31:07 INFO Signed narinfos id=34 count=15328lb # [ 103.667034] niks3-server[1111]: 2026/09/21 14:31:07 INFO Registered completed upload object_key=7qyaw17jjsjxkkgb1kfm3wxvczk56aan.ls5329lb # [ 103.680646] niks3-server[1111]: 2026/09/21 14:31:07 INFO Received complete upload request method=POST path=/api/pending_closures/34/complete5330lb # [ 103.685931] niks3-server[1111]: 2026/09/21 14:31:07 INFO Completed upload id=345331worker2 # [ 103.679789] nix-grpc-daemon[1094]: time=2026-09-21T14:31:07.295Z level=INFO msg="Upload complete. (297ms)"5332lb # [ 103.688664] niks3-server[1111]: 2026/09/21 14:31:07 INFO Registered completed upload object_key=7qyaw17jjsjxkkgb1kfm3wxvczk56aan.narinfo5333worker2 # [ 103.702359] nix-grpc-daemon[1034]: ts=2026-09-21T14:31:07Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42056 duration_s=1 drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv assign_id=1 outputs=15334worker2 # [ 103.716397] nix-grpc-daemon[1034]: ts=2026-09-21T14:31:07Z level=debug event=want client=1 drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv cached=05335worker2 # [ 103.720463] nix-grpc-daemon[1034]: ts=2026-09-21T14:31:07Z level=debug event=assigned drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv worker=192.168.1.4:50051 clients=15336worker2 # [ 103.723975] nix-daemon[838]: reaped child process 1140, status = succeeded5337worker2 # [ 103.747213] nix-grpc-daemon[1034]: ts=2026-09-21T14:31:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:420565338worker2 # [ 103.764144] nix-grpc-daemon[1034]: ts=2026-09-21T14:31:07Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:420565339worker2 # [ 103.778388] nix-grpc-daemon[1094]: time=2026-09-21T14:31:07.393Z level=INFO msg="All 1 paths already cached"5340client # copying 1 paths to 'lb:50051'...5341client # copying path '/nix/store/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv' to 'lb:50051'...5342worker2 # [ 103.788885] nix-grpc-daemon[1034]: ts=2026-09-21T14:31:07Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:420565343worker2 # [ 103.806064] nix-daemon[838]: accepted connection from pid 1034, user nix-grpc-daemon (trusted)5344worker2 # [ 103.961373] nix-daemon[838]: accepted connection from pid 1167, user nix-grpc-daemon (trusted)5345worker2 # [ 103.980214] nix-daemon[838]: reaped child process 1170, status = succeeded5346worker2 # [ 104.092697] nix-daemon[838]: accepted connection from pid 1172, user nix-grpc-daemon (trusted)5347worker2 # [ 104.108298] nix-daemon[838]: reaped child process 1175, status = succeeded5348lb # [ 104.120642] niks3-server[1111]: 2026/09/21 14:31:07 INFO Received uploads request method=POST path=/api/pending_closures5349worker2 # [ 104.123379] nix-grpc-daemon[1094]: time=2026-09-21T14:31:07.738Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5350worker2 # [ 104.125030] nix-grpc-daemon[1094]: time=2026-09-21T14:31:07.738Z level=INFO msg="Uploading s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv (640B)"5351lb # [ 104.150457] niks3-server[1111]: 2026/09/21 14:31:07 INFO Registered completed upload object_key=nar/0g5yl4v9pymang94vyp5ib5wi5jwf0bpp184y8qjavkpag12wmlv.nar.zst5352lb # [ 104.167583] niks3-server[1111]: 2026/09/21 14:31:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/35/sign5353worker2 # [ 104.166750] nix-grpc-daemon[1094]: time=2026-09-21T14:31:07.782Z level=INFO msg="Uploading 1 narinfos"5354lb # [ 104.179526] niks3-server[1111]: 2026/09/21 14:31:07 INFO Signed narinfos id=35 count=15355lb # [ 104.187214] niks3-server[1111]: 2026/09/21 14:31:07 INFO Registered completed upload object_key=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g.ls5356lb # [ 104.198916] niks3-server[1111]: 2026/09/21 14:31:07 INFO Received complete upload request method=POST path=/api/pending_closures/35/complete5357lb # [ 104.204871] niks3-server[1111]: 2026/09/21 14:31:07 INFO Completed upload id=355358worker2 # [ 104.198077] nix-grpc-daemon[1094]: time=2026-09-21T14:31:07.813Z level=INFO msg="Upload complete. (375ms)"5359lb # [ 104.208608] niks3-server[1111]: 2026/09/21 14:31:07 INFO Registered completed upload object_key=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g.narinfo5360worker2 # [ 104.208647] nix-grpc-daemon[1034]: ts=2026-09-21T14:31:07Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42056 duration_s=0 paths=1 nar_bytes_in=6405361worker2 # [ 104.223889] nix-grpc-daemon[1034]: ts=2026-09-21T14:31:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:420565362worker2 # [ 104.232722] nix-daemon[838]: accepted connection from pid 1034, user nix-grpc-daemon (trusted)5363worker2 # [ 104.235590] nix-daemon[838]: reaped child process 1165, status = succeeded5364client # building '/nix/store/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv' on lb:50051...5365client # farm-b> worker2: building s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv5366worker2 # [ 104.250361] nix-daemon[838]: accepted connection from pid 1034, user nix-grpc-daemon (trusted)5367client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds5368client # farm-b> LOG-farm-b-on-standby5369worker2 # [ 105.296577] nix-daemon[838]: reaped child process 1179, status = succeeded5370worker2 # [ 105.387989] nix-daemon[838]: accepted connection from pid 1195, user nix-grpc-daemon (trusted)5371worker2 # [ 105.404848] nix-daemon[838]: reaped child process 1198, status = succeeded5372lb # [ 105.418549] niks3-server[1111]: 2026/09/21 14:31:09 INFO Received uploads request method=POST path=/api/pending_closures5373worker2 # [ 105.421550] nix-grpc-daemon[1094]: time=2026-09-21T14:31:09.036Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5374worker2 # [ 105.422999] nix-grpc-daemon[1094]: time=2026-09-21T14:31:09.036Z level=INFO msg="Uploading 6hsxxv65wr2jz8cgv2v66iq16niq1gsv-farm-b (184B)"5375lb # [ 105.456582] niks3-server[1111]: 2026/09/21 14:31:09 INFO Registered completed upload object_key=nar/1grn64wcl2mc78k1jvrz5mcrk1ld70lhf8b9jpiqy8mvs6gfzq0v.nar.zst5376lb # [ 105.469960] niks3-server[1111]: 2026/09/21 14:31:09 INFO Registered completed upload object_key=log/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv5377worker2 # [ 105.473558] nix-grpc-daemon[1094]: time=2026-09-21T14:31:09.088Z level=INFO msg="Uploading 1 narinfos"5378lb # [ 105.480128] niks3-server[1111]: 2026/09/21 14:31:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/36/sign5379lb # [ 105.483820] niks3-server[1111]: 2026/09/21 14:31:09 INFO Signed narinfos id=36 count=15380lb # [ 105.494450] niks3-server[1111]: 2026/09/21 14:31:09 INFO Registered completed upload object_key=6hsxxv65wr2jz8cgv2v66iq16niq1gsv.ls5381lb # [ 105.505626] niks3-server[1111]: 2026/09/21 14:31:09 INFO Received complete upload request method=POST path=/api/pending_closures/36/complete5382lb # [ 105.510011] niks3-server[1111]: 2026/09/21 14:31:09 INFO Completed upload id=365383lb # [ 105.512850] niks3-server[1111]: 2026/09/21 14:31:09 INFO Registered completed upload object_key=6hsxxv65wr2jz8cgv2v66iq16niq1gsv.narinfo5384worker2 # [ 105.497475] nix-grpc-daemon[1094]: time=2026-09-21T14:31:09.113Z level=INFO msg="Upload complete. (238ms)"5385worker2 # [ 105.511361] nix-grpc-daemon[1034]: ts=2026-09-21T14:31:09Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42056 duration_s=1 drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv assign_id=2 outputs=15386worker2 # [ 105.530022] nix-grpc-daemon[1034]: ts=2026-09-21T14:31:09Z level=debug event=want client=1 drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv cached=05387worker2 # [ 105.534491] nix-grpc-daemon[1034]: ts=2026-09-21T14:31:09Z level=debug event=assigned drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv worker=192.168.1.4:50051 clients=15388worker2 # [ 105.539095] nix-daemon[838]: reaped child process 1177, status = succeeded5389worker2 # [ 105.565672] nix-grpc-daemon[1034]: ts=2026-09-21T14:31:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:420565390worker2 # [ 105.581818] nix-grpc-daemon[1034]: ts=2026-09-21T14:31:09Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:420565391worker2 # [ 105.596722] nix-grpc-daemon[1094]: time=2026-09-21T14:31:09.212Z level=INFO msg="All 2 paths already cached"5392client # copying 1 paths to 'lb:50051'...5393client # copying path '/nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv' to 'lb:50051'...5394worker2 # [ 105.606998] nix-grpc-daemon[1034]: ts=2026-09-21T14:31:09Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:420565395worker2 # [ 105.621907] nix-daemon[838]: accepted connection from pid 1034, user nix-grpc-daemon (trusted)5396worker2 # [ 105.771797] nix-daemon[838]: accepted connection from pid 1204, user nix-grpc-daemon (trusted)5397worker2 # [ 105.792401] nix-daemon[838]: reaped child process 1207, status = succeeded5398worker2 # [ 105.890905] nix-daemon[838]: accepted connection from pid 1209, user nix-grpc-daemon (trusted)5399worker2 # [ 105.906262] nix-daemon[838]: reaped child process 1212, status = succeeded5400lb # [ 105.920246] niks3-server[1111]: 2026/09/21 14:31:09 INFO Received uploads request method=POST path=/api/pending_closures5401worker2 # [ 105.923714] nix-grpc-daemon[1094]: time=2026-09-21T14:31:09.539Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5402worker2 # [ 105.925392] nix-grpc-daemon[1094]: time=2026-09-21T14:31:09.539Z level=INFO msg="Uploading 7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv (768B)"5403lb # [ 105.952133] niks3-server[1111]: 2026/09/21 14:31:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/37/sign5404worker2 # [ 105.954827] nix-grpc-daemon[1094]: time=2026-09-21T14:31:09.570Z level=INFO msg="Uploading 1 narinfos"5405lb # [ 105.970548] niks3-server[1111]: 2026/09/21 14:31:09 INFO Signed narinfos id=37 count=15406lb # [ 105.981499] niks3-server[1111]: 2026/09/21 14:31:09 INFO Registered completed upload object_key=nar/1zyx46mv29cfygdhc5ai40qr4vl88lkr1lilm0h72j23w6wakgmc.nar.zst5407worker2 # [ 105.992098] nix-grpc-daemon[1094]: time=2026-09-21T14:31:09.607Z level=INFO msg="Upload complete. (354ms)"5408lb # [ 106.002090] niks3-server[1111]: 2026/09/21 14:31:09 INFO Received complete upload request method=POST path=/api/pending_closures/37/complete5409lb # [ 106.011259] niks3-server[1111]: 2026/09/21 14:31:09 INFO Completed upload id=375410lb # [ 106.013550] niks3-server[1111]: 2026/09/21 14:31:09 INFO Registered completed upload object_key=7s5szig4ffga53df6rls98r9lhxsrsc5.narinfo5411worker2 # [ 106.000142] nix-grpc-daemon[1034]: ts=2026-09-21T14:31:09Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42056 duration_s=0 paths=1 nar_bytes_in=7685412worker2 # [ 106.014150] nix-grpc-daemon[1094]: time=2026-09-21T14:31:09.620Z level=WARN msg="Failed to register uploaded object" key=7s5szig4ffga53df6rls98r9lhxsrsc5.ls error="server returned 404: object is not pending upload\n"5413worker2 # [ 106.024399] nix-grpc-daemon[1034]: ts=2026-09-21T14:31:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:420565414worker2 # [ 106.027437] nix-daemon[838]: accepted connection from pid 1034, user nix-grpc-daemon (trusted)5415worker2 # [ 106.034749] nix-daemon[838]: reaped child process 1202, status = succeeded5416client # building '/nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv' on lb:50051...5417worker2 # [ 106.041683] nix-daemon[838]: accepted connection from pid 1034, user nix-grpc-daemon (trusted)5418client # farm-top> worker2: building 7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv5419client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds5420client # farm-top> LOG-farm-top-on-standby5421worker2 # [ 107.503819] nix-daemon[838]: reaped child process 1216, status = succeeded5422worker2 # [ 107.645115] nix-daemon[838]: accepted connection from pid 1232, user nix-grpc-daemon (trusted)5423worker2 # [ 107.662011] nix-daemon[838]: reaped child process 1235, status = succeeded5424lb # [ 107.677647] niks3-server[1111]: 2026/09/21 14:31:11 INFO Received uploads request method=POST path=/api/pending_closures5425worker2 # [ 107.688382] nix-grpc-daemon[1094]: time=2026-09-21T14:31:11.301Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5426worker2 # [ 107.690156] nix-grpc-daemon[1094]: time=2026-09-21T14:31:11.302Z level=INFO msg="Uploading 1ini8mfd1rzqyp8jgqihp4k889588mpx-farm-top (240B)"5427lb # [ 107.733490] niks3-server[1111]: 2026/09/21 14:31:11 INFO Registered completed upload object_key=log/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv5428lb # [ 107.741198] niks3-server[1111]: 2026/09/21 14:31:11 INFO Registered completed upload object_key=nar/1mncazpqn39nifc2fwbx80la5i264r8xc1lj8fnlbx006znc1925.nar.zst5429worker2 # [ 107.743519] nix-grpc-daemon[1094]: time=2026-09-21T14:31:11.358Z level=INFO msg="Uploading 1 narinfos"5430lb # [ 107.751220] niks3-server[1111]: 2026/09/21 14:31:11 INFO Received sign narinfos request method=POST path=/api/pending_closures/38/sign5431lb # [ 107.758842] niks3-server[1111]: 2026/09/21 14:31:11 INFO Signed narinfos id=38 count=15432lb # [ 107.762738] niks3-server[1111]: 2026/09/21 14:31:11 INFO Registered completed upload object_key=1ini8mfd1rzqyp8jgqihp4k889588mpx.ls5433lb # [ 107.770386] niks3-server[1111]: 2026/09/21 14:31:11 INFO Received complete upload request method=POST path=/api/pending_closures/38/complete5434lb # [ 107.777586] niks3-server[1111]: 2026/09/21 14:31:11 INFO Completed upload id=385435worker2 # [ 107.770295] nix-grpc-daemon[1094]: time=2026-09-21T14:31:11.385Z level=INFO msg="Upload complete. (317ms)"5436lb # [ 107.780285] niks3-server[1111]: 2026/09/21 14:31:11 INFO Registered completed upload object_key=1ini8mfd1rzqyp8jgqihp4k889588mpx.narinfo5437worker2 # [ 107.785161] nix-grpc-daemon[1034]: ts=2026-09-21T14:31:11Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42056 duration_s=1 drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv assign_id=3 outputs=15438worker2 # [ 107.801331] nix-daemon[838]: reaped child process 1214, status = succeeded5439client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag on-standby >&2, in 6.29 seconds)5440client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag on-standby outPath5441client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag on-standby outPath, in 0.18 seconds)5442worker1 # [ 108.102860] systemd[1]: Listening on Nix worker-protocol over gRPC.5443worker1 # [ 108.116548] systemd[1]: Starting Nix worker-protocol over gRPC...5444worker1 # [ 108.261721] nix-grpc-daemon[1474]: ts=2026-09-21T14:31:11Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5445worker1 # [ 108.264063] nix-grpc-daemon[1474]: ts=2026-09-21T14:31:11Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5446worker1 # [ 108.268375] nix-grpc-daemon[1474]: ts=2026-09-21T14:31:11Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15447worker1 # [ 108.275313] nix-grpc-daemon[1474]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5448worker1 # [ 108.296185] nix-grpc-daemon[1474]: E0000 00:00:1790001071.877485 1474 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5449worker1 # [ 108.298504] nix-grpc-daemon[1474]: E0000 00:00:1790001071.879900 1474 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5450lb: must succeed: curl -sf localhost:9901/clusters5451worker1 # [ 108.323799] nix-grpc-daemon[1474]: ts=2026-09-21T14:31:11Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5452worker1 # [ 108.326086] nix-grpc-daemon[1474]: E0000 00:00:1790001071.889837 1514 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5453worker1 # [ 108.328655] nix-grpc-daemon[1474]: E0000 00:00:1790001071.910798 1514 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5454worker1 # [ 108.330869] nix-grpc-daemon[1474]: E0000 00:00:1790001071.911018 1514 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5455worker1 # [ 108.333305] nix-grpc-daemon[1474]: E0000 00:00:1790001071.927497 1514 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5456worker1 # [ 108.338566] systemd[1]: Started Nix worker-protocol over gRPC.5457worker1 # [ 108.342058] nix-grpc-daemon[1474]: ts=2026-09-21T14:31:11Z level=info event=scheduler_connected addr=lb:500515458worker2 # [ 108.336198] nix-grpc-daemon[1034]: ts=2026-09-21T14:31:11Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:549685459worker2 # [ 108.341705] nix-grpc-daemon[1034]: ts=2026-09-21T14:31:11Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=05460lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds)5461worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5462worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)5463lb: must succeed: curl -sf localhost:9901/clusters5464lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)5465lb: must succeed: curl -sf localhost:9901/clusters5466lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5467worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5468worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5469(finished: subtest: scheduler down: the other node takes the lock and keeps it, in 11.20 seconds)5470subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise5471client: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag rs5472client # [ 108.710238] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag rs.5473client # Running as unit: rs.service; invocation ID: 0070e96b7a8744749de15b830967075f5474client: (finished: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag rs, in 0.09 seconds)5475worker1 # [ 108.915018] nix-grpc-daemon[1474]: ts=2026-09-21T14:31:12Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:398005476worker1 # [ 108.929935] nix-daemon[851]: accepted connection from pid 1474, user nix-grpc-daemon (trusted)5477client # [ 108.952056] nix[1178]: don't know how to build these paths:5478client # [ 108.953605] nix[1178]: /nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv5479worker2 # [ 108.958086] nix-grpc-daemon[1034]: ts=2026-09-21T14:31:12Z level=debug event=want client=2 drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv cached=05480worker2 # [ 108.958446] nix-grpc-daemon[1034]: ts=2026-09-21T14:31:12Z level=debug event=assigned drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv worker=192.168.1.4:50051 clients=15481worker2 # [ 109.002053] nix-grpc-daemon[1034]: ts=2026-09-21T14:31:12Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:420565482lb # [ 109.018482] envoy[909]: conn peer=[2001:db8:1::3]:53184 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=1039 tx=425 ms=153885483worker2 # [ 109.024699] nix-grpc-daemon[1034]: ts=2026-09-21T14:31:12Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:420565484client # [ 109.050571] nix[1178]: copying 1 paths to 'lb:50051'...5485worker2 # [ 109.038558] nix-grpc-daemon[1034]: ts=2026-09-21T14:31:12Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:420565486client # [ 109.053498] nix[1178]: copying path '/nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv' to 'lb:50051'...5487worker2 # [ 109.060877] nix-daemon[838]: accepted connection from pid 1034, user nix-grpc-daemon (trusted)5488worker2 # [ 109.225485] nix-daemon[838]: accepted connection from pid 1248, user nix-grpc-daemon (trusted)5489worker2 # [ 109.246628] nix-daemon[838]: reaped child process 1251, status = succeeded5490worker2 # [ 109.349424] nix-daemon[838]: accepted connection from pid 1253, user nix-grpc-daemon (trusted)5491worker2 # [ 109.368608] nix-daemon[838]: reaped child process 1256, status = succeeded5492lb # [ 109.025396] envoy[909]: conn peer=[2001:db8:1::1]:44970 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9662 tx=4076 ms=60325493lb # [ 109.374186] niks3-server[1111]: 2026/09/21 14:31:12 INFO Received uploads request method=POST path=/api/pending_closures5494worker2 # [ 109.382790] nix-grpc-daemon[1094]: time=2026-09-21T14:31:12.997Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5495worker2 # [ 109.384340] nix-grpc-daemon[1094]: time=2026-09-21T14:31:12.997Z level=INFO msg="Uploading ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv (416B)"5496lb # [ 109.408192] niks3-server[1111]: 2026/09/21 14:31:13 INFO Registered completed upload object_key=nar/0lynldv3n40qnhi1516wgiry3i3pja3sanwyyd8q4nslr915d7r1.nar.zst5497lb # [ 109.425260] niks3-server[1111]: 2026/09/21 14:31:13 INFO Received sign narinfos request method=POST path=/api/pending_closures/39/sign5498worker2 # [ 109.426206] nix-grpc-daemon[1094]: time=2026-09-21T14:31:13.041Z level=INFO msg="Uploading 1 narinfos"5499lb # [ 109.438397] niks3-server[1111]: 2026/09/21 14:31:13 INFO Signed narinfos id=39 count=15500lb # [ 109.445706] niks3-server[1111]: 2026/09/21 14:31:13 INFO Registered completed upload object_key=ajji8m1zd5q5wi64gp7vclb5dd8hr19m.ls5501lb # [ 109.456682] niks3-server[1111]: 2026/09/21 14:31:13 INFO Received complete upload request method=POST path=/api/pending_closures/39/complete5502lb # [ 109.469245] niks3-server[1111]: 2026/09/21 14:31:13 INFO Completed upload id=395503worker2 # [ 109.457484] nix-grpc-daemon[1094]: time=2026-09-21T14:31:13.073Z level=INFO msg="Upload complete. (379ms)"5504lb # [ 109.475206] niks3-server[1111]: 2026/09/21 14:31:13 INFO Registered completed upload object_key=ajji8m1zd5q5wi64gp7vclb5dd8hr19m.narinfo5505worker2 # [ 109.468388] nix-grpc-daemon[1034]: ts=2026-09-21T14:31:13Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42056 duration_s=0 paths=1 nar_bytes_in=4165506worker2 # [ 109.478789] nix-grpc-daemon[1034]: ts=2026-09-21T14:31:13Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:420565507worker2 # [ 109.485852] nix-daemon[838]: accepted connection from pid 1034, user nix-grpc-daemon (trusted)5508worker2 # [ 109.491159] nix-daemon[838]: reaped child process 1246, status = succeeded5509client # [ 109.512857] nix[1178]: building '/nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv' on lb:50051...5510worker2 # [ 109.505108] nix-daemon[838]: accepted connection from pid 1034, user nix-grpc-daemon (trusted)5511lb: must succeed: curl -sf localhost:9901/clusters5512lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5513lb: must succeed: curl -sf localhost:9901/clusters5514lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5515lb: must succeed: curl -sf localhost:9901/clusters5516lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5517lb: must succeed: curl -sf localhost:9901/clusters5518lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)5519worker1: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true5520worker1 # [ 111.512736] systemd-journald[261]: Received client request to rotate journal, rotating.5521worker1 # [ 111.546444] systemd-journald[261]: Vacuuming done, freed 0B of archived journals from /var/log/journal/ccaedcf676434df5b264ee453d1ee23c.5522worker1: (finished: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true, in 0.10 seconds)5523worker2: must succeed: systemctl restart --no-block nix-grpc-daemon.service5524worker2 # [ 111.587006] systemd[1]: Stopping Nix worker-protocol over gRPC...5525worker2: (finished: must succeed: systemctl restart --no-block nix-grpc-daemon.service, in 0.05 seconds)5526client: must succeed: sleep 5.05527worker2 # [ 112.206202] nix-grpc-daemon[1034]: ts=2026-09-21T14:31:15Z level=info event=draining5528client: (finished: must succeed: sleep 5.0, in 5.05 seconds)5529worker1: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr='5530worker2 # [ 116.936598] nix-daemon[838]: reaped child process 1260, status = succeeded5531worker2 # [ 117.028592] nix-daemon[838]: accepted connection from pid 1301, user nix-grpc-daemon (trusted)5532worker2 # [ 117.049946] nix-daemon[838]: reaped child process 1304, status = succeeded5533lb # [ 117.055365] niks3-server[1111]: 2026/09/21 14:31:20 INFO Received uploads request method=POST path=/api/pending_closures5534worker2 # [ 117.060381] nix-grpc-daemon[1094]: time=2026-09-21T14:31:20.674Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5535worker2 # [ 117.061840] nix-grpc-daemon[1094]: time=2026-09-21T14:31:20.674Z level=INFO msg="Uploading k48d52qj0czpjmzj15yq3fd14zfw5iq6-slow-rs (120B)"5536lb # [ 117.091608] niks3-server[1111]: 2026/09/21 14:31:20 INFO Registered completed upload object_key=log/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv5537lb # [ 117.106736] niks3-server[1111]: 2026/09/21 14:31:20 INFO Registered completed upload object_key=nar/19z3qdigvc4gggg31dgc0wi5w85pgaq9dk7bvngh0yhckpy55jgq.nar.zst5538lb # [ 117.119875] niks3-server[1111]: 2026/09/21 14:31:20 INFO Received sign narinfos request method=POST path=/api/pending_closures/40/sign5539worker2 # [ 117.118431] nix-grpc-daemon[1094]: time=2026-09-21T14:31:20.733Z level=INFO msg="Uploading 1 narinfos"5540lb # [ 117.129049] niks3-server[1111]: 2026/09/21 14:31:20 INFO Signed narinfos id=40 count=15541lb # [ 117.133398] niks3-server[1111]: 2026/09/21 14:31:20 INFO Registered completed upload object_key=k48d52qj0czpjmzj15yq3fd14zfw5iq6.ls5542lb # [ 117.142220] niks3-server[1111]: 2026/09/21 14:31:20 INFO Received complete upload request method=POST path=/api/pending_closures/40/complete5543lb # [ 117.148647] niks3-server[1111]: 2026/09/21 14:31:20 INFO Completed upload id=405544worker2 # [ 117.142411] nix-grpc-daemon[1094]: time=2026-09-21T14:31:20.757Z level=INFO msg="Upload complete. (222ms)"5545lb # [ 117.152071] niks3-server[1111]: 2026/09/21 14:31:20 INFO Registered completed upload object_key=k48d52qj0czpjmzj15yq3fd14zfw5iq6.narinfo5546worker2 # [ 117.157080] nix-grpc-daemon[1034]: ts=2026-09-21T14:31:20Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42056 duration_s=7 drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv assign_id=4 outputs=15547worker2 # [ 117.171269] nix-daemon[838]: reaped child process 1258, status = succeeded5548client # [ 117.199655] systemd[1]: rs.service: Deactivated successfully.5549client # [ 117.203853] systemd[1]: rs.service: Consumed 207ms CPU time over 8.493s wall clock time, 41.2M memory peak, 7.1K incoming IP traffic, 8.2K outgoing IP traffic.5550worker2 # [ 117.208206] nix-grpc-daemon[1034]: ts=2026-09-21T14:31:20Z level=info event=signal_exit5551lb # [ 117.222143] niks3-server[1111]: 2026/09/21 14:31:20 INFO lead: released remote=192.168.1.4:399825552worker1 # [ 118.223968] nix-grpc-daemon[1474]: ts=2026-09-21T14:31:21Z level=info event=scheduler_restarting addr=lb:500515553worker2 # [ 118.220882] nix-grpc-daemon[1034]: ts=2026-09-21T14:31:21Z level=info event=scheduler_restarting5554worker2 # [ 118.225778] nix-grpc-daemon[1034]: ts=2026-09-21T14:31:21Z level=info event=scheduler_restarting addr=lb:500515555worker2 # [ 118.231139] nix-grpc-daemon[1034]: ts=2026-09-21T14:31:21Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:549685556worker2 # [ 118.236333] nix-grpc-daemon[1034]: ts=2026-09-21T14:31:21Z level=info event=worker_gone addr=192.168.1.4:500515557worker2 # [ 118.240905] nix-grpc-daemon[1034]: ts=2026-09-21T14:31:21Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:549685558worker2 # [ 118.245752] nix-grpc-daemon[1034]: ts=2026-09-21T14:31:21Z level=info event=worker_gone addr=192.168.1.3:500515559worker2 # [ 118.271534] nix-daemon[838]: reaped child process 1086, status = succeeded5560worker1 # [ 118.285249] nix-grpc-daemon[1474]: ts=2026-09-21T14:31:21Z level=info event=scheduler_take_over5561lb # [ 118.285533] niks3-server[1111]: 2026/09/21 14:31:21 INFO lead: acquired remote=192.168.1.3:357825562worker2 # [ 118.284629] nix-grpc-daemon[1034]: E0000 00:00:1790001081.899968 1079 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 18, started true, shutdown true5563worker2 # [ 118.291183] nix-grpc-daemon[1034]: E0000 00:00:1790001081.900176 1079 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 9, started true, shutdown true5564worker1 # [ 118.423521] nix-grpc-daemon[1474]: ts=2026-09-21T14:31:22Z level=info event=scheduler_connected addr=lb:500515565worker1 # [ 118.432562] nix-grpc-daemon[1474]: ts=2026-09-21T14:31:22Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:408505566worker1 # [ 118.438043] nix-grpc-daemon[1474]: ts=2026-09-21T14:31:22Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=05567worker2 # [ 118.438071] nix-grpc-daemon[1034]: ts=2026-09-21T14:31:22Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5568worker2 # [ 118.506197] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5569worker2 # [ 118.512442] systemd[1]: Stopped Nix worker-protocol over gRPC.5570worker2 # [ 118.514958] systemd[1]: nix-grpc-daemon.service: Consumed 2.126s CPU time over 39.481s wall clock time, 38.9M memory peak, 58.7K incoming IP traffic, 119.8K outgoing IP traffic.5571worker2 # [ 118.533870] systemd[1]: Starting scheduler-token.service...5572worker2 # [ 118.605160] systemd[1]: scheduler-token.service: Deactivated successfully.5573worker2 # [ 118.612441] systemd[1]: Finished scheduler-token.service.5574worker2 # [ 118.614082] systemd[1]: scheduler-token.service: Consumed 44ms CPU time over 75ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5575worker2 # [ 118.620482] systemd[1]: Starting Nix worker-protocol over gRPC...5576worker2 # [ 118.738578] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:22Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5577worker2 # [ 118.740888] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:22Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5578worker2 # [ 118.744784] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:22Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15579worker2 # [ 118.751211] nix-grpc-daemon[1314]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5580worker2 # [ 118.774743] nix-grpc-daemon[1314]: E0000 00:00:1790001082.364659 1314 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5581worker2 # [ 118.792667] nix-grpc-daemon[1314]: E0000 00:00:1790001082.366680 1314 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5582worker2 # [ 118.795504] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:22Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5583worker2 # [ 118.797390] nix-grpc-daemon[1314]: E0000 00:00:1790001082.374794 1353 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5584worker2 # [ 118.799650] nix-grpc-daemon[1314]: E0000 00:00:1790001082.390184 1353 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5585worker2 # [ 118.805029] nix-grpc-daemon[1314]: E0000 00:00:1790001082.392861 1353 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5586worker2 # [ 118.809351] nix-grpc-daemon[1314]: E0000 00:00:1790001082.407347 1353 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5587worker2 # [ 118.811987] systemd[1]: Started Nix worker-protocol over gRPC.5588worker2 # [ 118.812887] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:22Z level=info event=scheduler_connected addr=lb:500515589worker1 # [ 118.826028] nix-grpc-daemon[1474]: ts=2026-09-21T14:31:22Z level=info event=oidc_jwks provider=mock keys=15590worker1 # [ 118.830477] nix-grpc-daemon[1474]: ts=2026-09-21T14:31:22Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:408505591worker1 # [ 118.835863] nix-grpc-daemon[1474]: ts=2026-09-21T14:31:22Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=05592lb # [ 119.016123] envoy[909]: conn peer=[2001:db8:1::1]:44978 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2966 tx=1406 ms=82635593lb # [ 119.016426] envoy[909]: conn peer=[2001:db8:1::4]:36510 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=14939 tx=1178 ms=391845594worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr=', in 2.20 seconds)5595client: waiting for success: ! systemctl is-active rs5596client: (finished: waiting for success: ! systemctl is-active rs, in 0.06 seconds)5597client: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }5598client: (finished: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }, in 0.05 seconds)5599client: must fail: journalctl -u rs | grep -q 'reconnecting'5600client: (finished: must fail: journalctl -u rs | grep -q 'reconnecting', in 0.03 seconds)5601lb: must succeed: curl -sf localhost:9901/clusters5602lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5603worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5604worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5605(finished: subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise, in 10.74 seconds)5606subtest: scheduler restart mid-build: build finishes, no double build5607client: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag mid5608client # [ 119.448803] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag mid.5609client # Running as unit: mid.service; invocation ID: 8d6eb52572ba42f3aa6c7794ace636cd5610client: (finished: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag mid, in 0.09 seconds)5611worker2 # [ 119.649536] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:23Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:578025612worker2 # [ 119.663983] nix-daemon[838]: accepted connection from pid 1314, user nix-grpc-daemon (trusted)5613client # [ 119.693972] nix[1214]: don't know how to build these paths:5614client # [ 119.695378] nix[1214]: /nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv5615worker1 # [ 119.706243] nix-grpc-daemon[1474]: ts=2026-09-21T14:31:23Z level=debug event=want client=1 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv cached=05616worker1 # [ 119.706577] nix-grpc-daemon[1474]: ts=2026-09-21T14:31:23Z level=debug event=assigned drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv worker=192.168.1.3:50051 clients=15617worker1 # [ 119.748895] nix-grpc-daemon[1474]: ts=2026-09-21T14:31:23Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:398005618worker1 # [ 119.766736] nix-grpc-daemon[1474]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 12 ms (attempt 1/5)5619worker1 # [ 119.787814] nix-grpc-daemon[1474]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 23 ms (attempt 2/5)5620worker1 # [ 119.820400] nix-grpc-daemon[1474]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 167 ms (attempt 3/5)5621worker1 # [ 119.996916] nix-grpc-daemon[1474]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 798 ms (attempt 4/5)5622worker1 # [ 120.798457] nix-grpc-daemon[1474]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5623worker1 # [ 120.803816] nix-grpc-daemon[1474]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5624worker1 # [ 120.823672] nix-grpc-daemon[1474]: ts=2026-09-21T14:31:24Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:398005625client # [ 120.841277] nix[1214]: copying 1 paths to 'lb:50051'...5626worker1 # [ 120.840518] nix-grpc-daemon[1474]: ts=2026-09-21T14:31:24Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:398005627client # [ 120.846522] nix[1214]: copying path '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' to 'lb:50051'...5628worker1 # [ 120.854229] nix-daemon[851]: accepted connection from pid 1474, user nix-grpc-daemon (trusted)5629worker1 # [ 120.891088] nix-grpc-daemon[1474]: ts=2026-09-21T14:31:24Z level=info event=niks3_push_started pid=15975630worker1 # [ 121.040494] nix-grpc-daemon[1597]: time=2026-09-21T14:31:24.644Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.key ca=""5631worker1 # [ 121.149319] nix-daemon[851]: accepted connection from pid 1609, user nix-grpc-daemon (trusted)5632worker1 # [ 121.166222] nix-daemon[851]: reaped child process 1612, status = succeeded5633worker1 # [ 121.268481] nix-daemon[851]: accepted connection from pid 1614, user nix-grpc-daemon (trusted)5634worker1 # [ 121.284526] nix-daemon[851]: reaped child process 1617, status = succeeded5635lb # [ 121.291101] niks3-server[1111]: 2026/09/21 14:31:24 INFO Received uploads request method=POST path=/api/pending_closures5636worker1 # [ 121.302805] nix-grpc-daemon[1597]: time=2026-09-21T14:31:24.906Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5637worker1 # [ 121.304634] nix-grpc-daemon[1597]: time=2026-09-21T14:31:24.906Z level=INFO msg="Uploading chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv (424B)"5638lb # [ 121.326114] niks3-server[1111]: 2026/09/21 14:31:24 INFO Registered completed upload object_key=nar/14lxjrdcxwbhc1kgg39y29gnbx5cz80j4852d1i4w3rvq7g4b04g.nar.zst5639worker1 # [ 121.339687] nix-grpc-daemon[1597]: time=2026-09-21T14:31:24.943Z level=INFO msg="Uploading 1 narinfos"5640lb # [ 121.338671] niks3-server[1111]: 2026/09/21 14:31:24 INFO Received sign narinfos request method=POST path=/api/pending_closures/41/sign5641lb # [ 121.354203] niks3-server[1111]: 2026/09/21 14:31:24 INFO Signed narinfos id=41 count=15642lb # [ 121.360347] niks3-server[1111]: 2026/09/21 14:31:24 INFO Registered completed upload object_key=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc.ls5643lb # [ 121.372382] niks3-server[1111]: 2026/09/21 14:31:24 INFO Received complete upload request method=POST path=/api/pending_closures/41/complete5644lb # [ 121.379784] niks3-server[1111]: 2026/09/21 14:31:24 INFO Completed upload id=415645worker1 # [ 121.380571] nix-grpc-daemon[1597]: time=2026-09-21T14:31:24.985Z level=INFO msg="Upload complete. (336ms)"5646lb # [ 121.383488] niks3-server[1111]: 2026/09/21 14:31:25 INFO Registered completed upload object_key=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc.narinfo5647worker1 # [ 121.388257] nix-grpc-daemon[1474]: ts=2026-09-21T14:31:24Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39800 duration_s=0 paths=1 nar_bytes_in=4245648worker1 # [ 121.402974] nix-grpc-daemon[1474]: ts=2026-09-21T14:31:25Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:398005649worker1 # [ 121.410360] nix-daemon[851]: accepted connection from pid 1474, user nix-grpc-daemon (trusted)5650worker1 # [ 121.415565] nix-daemon[851]: reaped child process 1595, status = succeeded5651worker1 # [ 121.427631] nix-daemon[851]: accepted connection from pid 1474, user nix-grpc-daemon (trusted)5652client # [ 121.430770] nix[1214]: building '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' on lb:50051...5653lb: must succeed: curl -sf localhost:9901/clusters5654lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5655lb: must succeed: curl -sf localhost:9901/clusters5656lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5657worker1: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service5658worker1 # [ 123.311059] systemd[1]: nix-grpc-daemon.service: Sent signal SIGKILL to main process 1474 (nix-grpc-daemon) on client request.5659worker1 # [ 123.313234] systemd[1]: nix-grpc-daemon.service: Killed unit cgroup '/system.slice/nix-grpc-daemon.service' with SIGKILL on client request.5660lb # [ 123.314055] niks3-server[1111]: 2026/09/21 14:31:26 INFO lead: released remote=192.168.1.3:357825661client # [ 123.325507] nix[1214]: chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), asking the scheduler again5662client # [ 123.328637] nix[1214]: scheduler at lb:50051: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), reconnecting5663worker1: (finished: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service, in 0.06 seconds)5664client: must succeed: sleep 5.05665worker2 # [ 123.315213] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:26Z level=info event=scheduler_disconnected addr=lb:50051 error="upstream connect error or disconnect/reset before headers. reset reason: connection termination"5666worker1 # [ 123.337450] systemd[1]: nix-grpc-daemon.service: Main process exited, code=killed, status=9/KILL5667worker1 # [ 123.338988] systemd[1]: nix-grpc-daemon.service: Failed with result 'signal'.5668worker1 # [ 123.343678] systemd[1]: nix-grpc-daemon.service: Consumed 591ms CPU time over 15.220s wall clock time, 34.5M memory peak, 14.7K incoming IP traffic, 19.9K outgoing IP traffic.5669worker1 # [ 123.362209] nix-daemon[851]: reaped child process 1533, status = succeeded5670worker1 # [ 123.363661] nix-daemon[851]: reaped child process 1619, status = succeeded5671worker1 # [ 123.372140] nix-daemon[1621]: unexpected Nix daemon error: error: interrupted by the user5672worker1 # [ 123.375917] nix-daemon[851]: reaped child process 1621, status = failed with exit code 15673worker2 # [ 123.515814] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:27Z level=info event=scheduler_connected addr=lb:500515674worker1 # [ 123.656541] systemd[1]: nix-grpc-daemon.service: Scheduled restart job, restart counter is at 1.5675worker1 # [ 123.669167] systemd[1]: Starting Nix worker-protocol over gRPC...5676lb # [ 123.767468] niks3-server[1111]: 2026/09/21 14:31:27 INFO lead: acquired remote=192.168.1.4:413245677worker2 # [ 123.761447] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:27Z level=info event=scheduler_take_over5678worker1 # [ 123.836505] nix-grpc-daemon[1658]: ts=2026-09-21T14:31:27Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5679worker1 # [ 123.838359] nix-grpc-daemon[1658]: ts=2026-09-21T14:31:27Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5680worker1 # [ 123.842047] nix-grpc-daemon[1658]: ts=2026-09-21T14:31:27Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15681worker1 # [ 123.847446] nix-grpc-daemon[1658]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5682worker1 # [ 123.862607] nix-grpc-daemon[1658]: E0000 00:00:1790001087.451289 1658 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5683worker1 # [ 123.867370] nix-grpc-daemon[1658]: E0000 00:00:1790001087.454856 1658 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5684client # [ 123.882984] nix[1214]: scheduler at lb:50051: scheduler passive, another node serves, reconnecting5685worker2 # [ 123.871905] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:27Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"5686worker1 # [ 123.896106] nix-grpc-daemon[1658]: ts=2026-09-21T14:31:27Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5687worker1 # [ 123.897758] nix-grpc-daemon[1658]: E0000 00:00:1790001087.470548 1700 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5688worker1 # [ 123.905918] systemd[1]: Started Nix worker-protocol over gRPC.5689worker1 # [ 123.907273] nix-grpc-daemon[1658]: E0000 00:00:1790001087.491320 1700 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5690worker1 # [ 123.909866] nix-grpc-daemon[1658]: E0000 00:00:1790001087.491440 1700 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5691worker1 # [ 123.915085] nix-grpc-daemon[1658]: E0000 00:00:1790001087.509978 1700 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5692worker1 # [ 123.919638] nix-grpc-daemon[1658]: ts=2026-09-21T14:31:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5693worker2 # [ 124.073552] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:27Z level=info event=scheduler_connected addr=lb:500515694worker2 # [ 124.078707] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5695worker1 # [ 124.120649] nix-grpc-daemon[1658]: ts=2026-09-21T14:31:27Z level=info event=scheduler_connected addr=lb:500515696worker1 # [ 124.125670] nix-grpc-daemon[1658]: ts=2026-09-21T14:31:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5697worker2 # [ 124.279726] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:27Z level=info event=scheduler_connected addr=lb:500515698worker2 # [ 124.284936] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5699worker1 # [ 124.326547] nix-grpc-daemon[1658]: ts=2026-09-21T14:31:27Z level=info event=scheduler_connected addr=lb:500515700worker1 # [ 124.331631] nix-grpc-daemon[1658]: ts=2026-09-21T14:31:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5701worker2 # [ 124.487719] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:28Z level=info event=scheduler_connected addr=lb:500515702worker2 # [ 124.491958] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5703worker1 # [ 124.532443] nix-grpc-daemon[1658]: ts=2026-09-21T14:31:28Z level=info event=scheduler_connected addr=lb:500515704worker1 # [ 124.537411] nix-grpc-daemon[1658]: ts=2026-09-21T14:31:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5705worker2 # [ 124.688279] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:28Z level=info event=scheduler_connected addr=lb:500515706worker2 # [ 124.693225] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5707worker1 # [ 124.738100] nix-grpc-daemon[1658]: ts=2026-09-21T14:31:28Z level=info event=scheduler_connected addr=lb:500515708worker1 # [ 124.743046] nix-grpc-daemon[1658]: ts=2026-09-21T14:31:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5709client # [ 124.886398] nix[1214]: scheduler at lb:50051: no healthy upstream, reconnecting5710worker2 # [ 124.894178] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:28Z level=info event=scheduler_connected addr=lb:500515711worker2 # [ 124.899390] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5712worker1 # [ 124.945535] nix-grpc-daemon[1658]: ts=2026-09-21T14:31:28Z level=info event=scheduler_connected addr=lb:500515713worker1 # [ 124.949664] nix-grpc-daemon[1658]: ts=2026-09-21T14:31:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5714worker2 # [ 125.100370] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:28Z level=info event=scheduler_connected addr=lb:500515715worker2 # [ 125.105354] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5716worker1 # [ 125.146100] nix-grpc-daemon[1658]: ts=2026-09-21T14:31:28Z level=info event=scheduler_connected addr=lb:500515717worker1 # [ 125.151252] nix-grpc-daemon[1658]: ts=2026-09-21T14:31:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5718worker2 # [ 125.306492] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:28Z level=info event=scheduler_connected addr=lb:500515719worker2 # [ 125.311591] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5720worker1 # [ 125.352269] nix-grpc-daemon[1658]: ts=2026-09-21T14:31:28Z level=info event=scheduler_connected addr=lb:500515721worker1 # [ 125.357276] nix-grpc-daemon[1658]: ts=2026-09-21T14:31:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5722worker2 # [ 125.514379] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:29Z level=info event=scheduler_connected addr=lb:500515723worker2 # [ 125.518471] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:29Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5724worker1 # [ 125.558026] nix-grpc-daemon[1658]: ts=2026-09-21T14:31:29Z level=info event=scheduler_connected addr=lb:500515725worker2 # [ 125.565703] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:29Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:527545726worker2 # [ 125.571051] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:29Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=05727worker2 # [ 125.715080] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:29Z level=info event=scheduler_connected addr=lb:500515728worker2 # [ 125.728134] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:29Z level=info event=oidc_jwks provider=mock keys=15729worker2 # [ 125.733102] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:29Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:527545730worker2 # [ 125.738429] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:29Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=05731worker2 # [ 126.885997] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:30Z level=debug event=want client=1 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv cached=05732worker2 # [ 126.891906] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:30Z level=debug event=assigned drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv worker=192.168.1.3:50051 clients=15733worker1 # [ 126.943216] nix-grpc-daemon[1658]: ts=2026-09-21T14:31:30Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:433325734worker1 # [ 126.957353] nix-daemon[851]: accepted connection from pid 1658, user nix-grpc-daemon (trusted)5735worker1 # [ 126.975502] nix-daemon[851]: accepted connection from pid 1658, user nix-grpc-daemon (trusted)5736client # [ 126.999405] nix[1214]: building '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' on lb:50051...5737worker1 # [ 126.996776] nix-daemon[851]: accepted connection from pid 1658, user nix-grpc-daemon (trusted)5738client: (finished: must succeed: sleep 5.0, in 5.03 seconds)5739client: waiting for success: ! systemctl is-active mid5740worker1 # [ 128.619549] nix-grpc-daemon[1658]: ts=2026-09-21T14:31:32Z level=info event=niks3_push_started pid=17395741worker1 # [ 128.656649] nix-daemon[851]: reaped child process 1714, status = succeeded5742worker1 # [ 128.773904] nix-grpc-daemon[1739]: time=2026-09-21T14:31:32.378Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.key ca=""5743worker1 # [ 128.880143] nix-daemon[851]: accepted connection from pid 1752, user nix-grpc-daemon (trusted)5744worker1 # [ 128.900839] nix-daemon[851]: reaped child process 1755, status = succeeded5745lb # [ 128.907027] niks3-server[1111]: 2026/09/21 14:31:32 INFO Received uploads request method=POST path=/api/pending_closures5746worker1 # [ 128.919002] nix-grpc-daemon[1739]: time=2026-09-21T14:31:32.521Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5747worker1 # [ 128.922244] nix-grpc-daemon[1739]: time=2026-09-21T14:31:32.521Z level=INFO msg="Uploading fk2vdwmn2msb3ja19s2jwak35ignz889-slow-mid (120B)"5748lb # [ 128.962724] niks3-server[1111]: 2026/09/21 14:31:32 INFO Registered completed upload object_key=log/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv5749lb # [ 128.972197] niks3-server[1111]: 2026/09/21 14:31:32 INFO Registered completed upload object_key=nar/0462jb7zwy2d73k3q6v09xnk3bx318v8ik5ckmp32w2hzvwj4k2d.nar.zst5750lb # [ 128.985822] niks3-server[1111]: 2026/09/21 14:31:32 INFO Received sign narinfos request method=POST path=/api/pending_closures/42/sign5751lb # [ 129.000314] niks3-server[1111]: 2026/09/21 14:31:32 INFO Registered completed upload object_key=fk2vdwmn2msb3ja19s2jwak35ignz889.ls5752lb # [ 129.013504] niks3-server[1111]: 2026/09/21 14:31:32 INFO Signed narinfos id=42 count=15753worker1 # [ 129.015873] nix-grpc-daemon[1739]: time=2026-09-21T14:31:32.618Z level=INFO msg="Uploading 1 narinfos"5754lb # [ 129.020757] envoy[909]: conn peer=[2001:db8:1::3]:35758 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=666 tx=311 ms=149935755lb # [ 129.030068] niks3-server[1111]: 2026/09/21 14:31:32 INFO Received complete upload request method=POST path=/api/pending_closures/42/complete5756lb # [ 129.041239] niks3-server[1111]: 2026/09/21 14:31:32 INFO Registered completed upload object_key=fk2vdwmn2msb3ja19s2jwak35ignz889.narinfo5757lb # [ 129.051254] niks3-server[1111]: 2026/09/21 14:31:32 INFO Completed upload id=425758worker1 # [ 129.052222] nix-grpc-daemon[1739]: time=2026-09-21T14:31:32.656Z level=INFO msg="Upload complete. (274ms)"5759worker1 # [ 129.064176] nix-grpc-daemon[1658]: ts=2026-09-21T14:31:32Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43332 duration_s=2 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv assign_id=1 outputs=15760worker1 # [ 129.082293] nix-daemon[851]: reaped child process 1712, status = succeeded5761client # [ 129.102613] systemd[1]: mid.service: Deactivated successfully.5762client # [ 129.107556] systemd[1]: mid.service: Consumed 209ms CPU time over 9.655s wall clock time, 41.3M memory peak, 9.1K incoming IP traffic, 11.4K outgoing IP traffic.5763client: (finished: waiting for success: ! systemctl is-active mid, in 1.12 seconds)5764client: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }5765client: (finished: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }, in 0.05 seconds)5766(finished: subtest: scheduler restart mid-build: build finishes, no double build, in 10.42 seconds)5767subtest: interrupting the client stops the build on the worker5768client: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag intr5769client # [ 129.836898] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag intr.5770client # Running as unit: intr.service; invocation ID: 83e3423efdd3470c9e86c7aaaf5962905771client: (finished: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag intr, in 0.05 seconds)5772worker1 # [ 130.004717] nix-grpc-daemon[1658]: ts=2026-09-21T14:31:33Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:433325773client # [ 130.016555] nix[1250]: don't know how to build these paths:5774client # [ 130.017497] nix[1250]: /nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv5775worker2 # [ 130.021453] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:33Z level=debug event=want client=2 drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv cached=05776worker2 # [ 130.026915] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:33Z level=debug event=assigned drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv worker=192.168.1.4:50051 clients=15777worker2 # [ 130.066608] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:33Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:578025778worker2 # [ 130.084202] nix-grpc-daemon[1314]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 42 ms (attempt 1/5)5779worker2 # [ 130.135306] nix-grpc-daemon[1314]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 64 ms (attempt 2/5)5780worker2 # [ 130.201206] nix-grpc-daemon[1314]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 379 ms (attempt 3/5)5781worker2 # [ 130.582237] nix-grpc-daemon[1314]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 298 ms (attempt 4/5)5782worker2 # [ 130.885200] nix-grpc-daemon[1314]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5783worker2 # [ 130.890507] nix-grpc-daemon[1314]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5784worker2 # [ 130.910595] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:34Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:578025785client # [ 130.934155] nix[1250]: copying 1 paths to 'lb:50051'...5786client # [ 130.938649] nix[1250]: copying path '/nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv' to 'lb:50051'...5787worker2 # [ 130.928640] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:34Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:578025788worker2 # [ 130.943631] nix-daemon[838]: accepted connection from pid 1314, user nix-grpc-daemon (trusted)5789worker2 # [ 130.976595] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:34Z level=info event=niks3_push_started pid=14105790worker2 # [ 131.134866] nix-grpc-daemon[1410]: time=2026-09-21T14:31:34.749Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.key ca=""5791worker2 # [ 131.256822] nix-daemon[838]: accepted connection from pid 1426, user nix-grpc-daemon (trusted)5792worker2 # [ 131.275429] nix-daemon[838]: reaped child process 1429, status = succeeded5793worker2 # [ 131.374716] nix-daemon[838]: accepted connection from pid 1431, user nix-grpc-daemon (trusted)5794worker2 # [ 131.393696] nix-daemon[838]: reaped child process 1434, status = succeeded5795lb # [ 131.399842] niks3-server[1111]: 2026/09/21 14:31:35 INFO Received uploads request method=POST path=/api/pending_closures5796worker2 # [ 131.406449] nix-grpc-daemon[1410]: time=2026-09-21T14:31:35.020Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5797worker2 # [ 131.407917] nix-grpc-daemon[1410]: time=2026-09-21T14:31:35.021Z level=INFO msg="Uploading 85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv (424B)"5798lb # [ 131.448109] niks3-server[1111]: 2026/09/21 14:31:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/43/sign5799worker2 # [ 131.450129] nix-grpc-daemon[1410]: time=2026-09-21T14:31:35.065Z level=INFO msg="Uploading 1 narinfos"5800lb # [ 131.461369] niks3-server[1111]: 2026/09/21 14:31:35 INFO Registered completed upload object_key=nar/0h460h76prjyvhqhyxlhz1xrmh3s520jbd19gaagy4hp4vc92r1p.nar.zst5801lb # [ 131.475693] niks3-server[1111]: 2026/09/21 14:31:35 INFO Signed narinfos id=43 count=15802worker2 # [ 131.477316] nix-grpc-daemon[1410]: time=2026-09-21T14:31:35.092Z level=INFO msg="Upload complete. (338ms)"5803lb # [ 131.484545] niks3-server[1111]: 2026/09/21 14:31:35 INFO Received complete upload request method=POST path=/api/pending_closures/43/complete5804lb # [ 131.491739] niks3-server[1111]: 2026/09/21 14:31:35 INFO Completed upload id=435805worker2 # [ 131.483941] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:35Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:57802 duration_s=0 paths=1 nar_bytes_in=4245806lb # [ 131.495568] niks3-server[1111]: 2026/09/21 14:31:35 INFO Registered completed upload object_key=85wa5czf6m5ng93xyq44r1affb74csh8.narinfo5807lb # [ 131.502653] niks3-server[1111]: 2026/09/21 14:31:35 INFO Registered completed upload object_key=85wa5czf6m5ng93xyq44r1affb74csh8.ls5808worker2 # [ 131.497788] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:578025809worker2 # [ 131.503012] nix-daemon[838]: accepted connection from pid 1314, user nix-grpc-daemon (trusted)5810worker2 # [ 131.507508] nix-daemon[838]: reaped child process 1408, status = succeeded5811worker2 # [ 131.517156] nix-daemon[838]: accepted connection from pid 1314, user nix-grpc-daemon (trusted)5812client # [ 131.528639] nix[1250]: building '/nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv' on lb:50051...5813client: must succeed: systemctl kill -s INT intr5814worker2 # [ 133.515829] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:37Z level=info event=revoked_running drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv5815client # [ 133.524876] systemd[1]: intr.service: Sent signal SIGINT to main process 1250 (nix) on client request.5816client # [ 133.534222] nix[1250]: shutting down85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED)5817client # [ 133.548387] nix[1250]: error: interrupted by the user5818client: (finished: must succeed: systemctl kill -s INT intr, in 0.08 seconds)5819worker2 # [ 133.547571] nix-daemon[838]: reaped child process 1436, status = succeeded5820worker2 # [ 133.549651] nix-daemon[1438]: unexpected Nix daemon error: error: interrupted by the user5821worker2 # [ 133.555827] nix-daemon[838]: reaped child process 1438, status = failed with exit code 15822client # [ 133.581323] systemd[1]: intr.service: Main process exited, code=exited, status=1/FAILURE5823client # [ 133.586708] systemd[1]: intr.service: Failed with result 'exit-code'.5824client # [ 133.591363] systemd[1]: intr.service: Consumed 169ms CPU time over 3.748s wall clock time, 41.4M memory peak, 6.4K incoming IP traffic, 8.3K outgoing IP traffic.5825(finished: subtest: interrupting the client stops the build on the worker, in 3.86 seconds)5826subtest: a deploy mid-build drains: the build finishes, the new generation takes over5827client: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag sw5828client # [ 133.727396] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag sw.5829client # Running as unit: sw.service; invocation ID: fa4ba4ebfb0b407b9070a90747b929a65830client: (finished: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag sw, in 0.08 seconds)5831client # [ 133.920219] nix[1271]: don't know how to build these paths:5832client # [ 133.921353] nix[1271]: /nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv5833worker1 # [ 133.914831] nix-grpc-daemon[1658]: ts=2026-09-21T14:31:37Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:433325834worker2 # [ 133.920054] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:37Z level=debug event=want client=3 drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv cached=05835worker2 # [ 133.926091] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:37Z level=debug event=assigned drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv worker=192.168.1.3:50051 clients=15836worker1 # [ 133.972768] nix-grpc-daemon[1658]: ts=2026-09-21T14:31:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:433325837worker1 # [ 133.987556] nix-grpc-daemon[1658]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 59 ms (attempt 1/5)5838worker1 # [ 134.055518] nix-grpc-daemon[1658]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 138 ms (attempt 2/5)5839worker1 # [ 134.195328] nix-grpc-daemon[1658]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 377 ms (attempt 3/5)5840worker1 # [ 134.574342] nix-grpc-daemon[1658]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 196 ms (attempt 4/5)5841worker1 # [ 134.772572] nix-grpc-daemon[1658]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5842worker1 # [ 134.778837] nix-grpc-daemon[1658]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5843worker1 # [ 134.796826] nix-grpc-daemon[1658]: ts=2026-09-21T14:31:38Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:433325844client # [ 134.816160] nix[1271]: copying 1 paths to 'lb:50051'...5845worker1 # [ 134.812634] nix-grpc-daemon[1658]: ts=2026-09-21T14:31:38Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:433325846client # [ 134.816459] nix[1271]: copying path '/nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv' to 'lb:50051'...5847worker1 # [ 134.829740] nix-daemon[851]: accepted connection from pid 1658, user nix-grpc-daemon (trusted)5848worker1 # [ 134.999561] nix-daemon[851]: accepted connection from pid 1787, user nix-grpc-daemon (trusted)5849worker1 # [ 135.048736] nix-daemon[851]: reaped child process 1794, status = succeeded5850worker1 # [ 135.135006] nix-daemon[851]: accepted connection from pid 1796, user nix-grpc-daemon (trusted)5851worker1 # [ 135.154295] nix-daemon[851]: reaped child process 1799, status = succeeded5852lb # [ 135.154057] niks3-server[1111]: 2026/09/21 14:31:38 INFO Received uploads request method=POST path=/api/pending_closures5853worker1 # [ 135.173385] nix-grpc-daemon[1739]: time=2026-09-21T14:31:38.777Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5854worker1 # [ 135.174817] nix-grpc-daemon[1739]: time=2026-09-21T14:31:38.777Z level=INFO msg="Uploading x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv (416B)"5855lb # [ 135.196535] niks3-server[1111]: 2026/09/21 14:31:38 INFO Registered completed upload object_key=nar/0jczp45bkr8vp1ymznxf6da1iv8dzh42pnl36wiprbzwhpi2ddia.nar.zst5856worker1 # [ 135.209466] nix-grpc-daemon[1739]: time=2026-09-21T14:31:38.813Z level=INFO msg="Uploading 1 narinfos"5857lb # [ 135.209712] niks3-server[1111]: 2026/09/21 14:31:38 INFO Received sign narinfos request method=POST path=/api/pending_closures/44/sign5858lb # [ 135.220286] niks3-server[1111]: 2026/09/21 14:31:38 INFO Signed narinfos id=44 count=15859lb # [ 135.225585] niks3-server[1111]: 2026/09/21 14:31:38 INFO Registered completed upload object_key=x0m4asg5xx7vgj4dip2crwn6zhxqngvs.ls5860lb # [ 135.240593] niks3-server[1111]: 2026/09/21 14:31:38 INFO Received complete upload request method=POST path=/api/pending_closures/44/complete5861lb # [ 135.246673] niks3-server[1111]: 2026/09/21 14:31:38 INFO Completed upload id=445862worker1 # [ 135.246986] nix-grpc-daemon[1739]: time=2026-09-21T14:31:38.851Z level=INFO msg="Upload complete. (400ms)"5863lb # [ 135.249437] niks3-server[1111]: 2026/09/21 14:31:38 INFO Registered completed upload object_key=x0m4asg5xx7vgj4dip2crwn6zhxqngvs.narinfo5864worker1 # [ 135.259099] nix-grpc-daemon[1658]: ts=2026-09-21T14:31:38Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43332 duration_s=0 paths=1 nar_bytes_in=4165865worker1 # [ 135.275045] nix-grpc-daemon[1658]: ts=2026-09-21T14:31:38Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:433325866worker1 # [ 135.284328] nix-daemon[851]: accepted connection from pid 1658, user nix-grpc-daemon (trusted)5867worker1 # [ 135.286686] nix-daemon[851]: reaped child process 1785, status = succeeded5868worker1 # [ 135.299281] nix-daemon[851]: accepted connection from pid 1658, user nix-grpc-daemon (trusted)5869client # [ 135.303032] nix[1271]: building '/nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv' on lb:50051...5870worker1: must succeed: systemctl reload nix-grpc-daemon.service5871worker1 # [ 136.437931] systemd[1]: Reloading Nix worker-protocol over gRPC...5872worker1 # [ 136.486245] systemd[1]: Reloaded Nix worker-protocol over gRPC.5873worker1: (finished: must succeed: systemctl reload nix-grpc-daemon.service, in 0.08 seconds)5874worker1: must succeed: pgrep -f 'read -t [9]0 x'5875worker1: (finished: must succeed: pgrep -f 'read -t [9]0 x', in 0.05 seconds)5876client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag during-drain >&25877worker2 # [ 136.766344] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:40Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:578025878client # don't know how to build these paths:5879client # /nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv5880worker2 # [ 136.776295] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:40Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:578025881worker2 # [ 136.802415] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:40Z level=debug event=want client=4 drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv cached=05882worker2 # [ 136.809546] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:40Z level=debug event=assigned drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv worker=192.168.1.4:50051 clients=15883worker1 # [ 136.865431] nix-grpc-daemon[1658]: ts=2026-09-21T14:31:40Z level=info event=draining5884worker2 # [ 136.855832] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:578025885worker2 # [ 136.871872] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:40Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:578025886client # copying 1 paths to 'lb:50051'...5887worker2 # [ 136.887938] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:40Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:578025888client # copying path '/nix/store/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv' to 'lb:50051'...5889worker2 # [ 136.911329] nix-daemon[838]: accepted connection from pid 1314, user nix-grpc-daemon (trusted)5890worker2 # [ 137.075439] nix-daemon[838]: accepted connection from pid 1489, user nix-grpc-daemon (trusted)5891worker2 # [ 137.103739] nix-daemon[838]: reaped child process 1492, status = succeeded5892worker2 # [ 137.199499] nix-daemon[838]: accepted connection from pid 1494, user nix-grpc-daemon (trusted)5893worker2 # [ 137.219136] nix-daemon[838]: reaped child process 1497, status = succeeded5894lb # [ 137.230876] niks3-server[1111]: 2026/09/21 14:31:40 INFO Received uploads request method=POST path=/api/pending_closures5895worker2 # [ 137.235629] nix-grpc-daemon[1410]: time=2026-09-21T14:31:40.850Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5896worker2 # [ 137.237280] nix-grpc-daemon[1410]: time=2026-09-21T14:31:40.850Z level=INFO msg="Uploading k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv (528B)"5897lb # [ 137.262619] niks3-server[1111]: 2026/09/21 14:31:40 INFO Registered completed upload object_key=nar/1h3fzlmfdwdiqr732wwrq7jnpw7n6p8f9qx1089c6g6x2i4csv75.nar.zst5898lb # [ 137.282328] niks3-server[1111]: 2026/09/21 14:31:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/45/sign5899worker2 # [ 137.283100] nix-grpc-daemon[1410]: time=2026-09-21T14:31:40.898Z level=INFO msg="Uploading 1 narinfos"5900lb # [ 137.294608] niks3-server[1111]: 2026/09/21 14:31:40 INFO Signed narinfos id=45 count=15901lb # [ 137.300932] niks3-server[1111]: 2026/09/21 14:31:40 INFO Registered completed upload object_key=k3l35rf796jrnjns08398p1l8fcx178a.ls5902lb # [ 137.310196] niks3-server[1111]: 2026/09/21 14:31:40 INFO Received complete upload request method=POST path=/api/pending_closures/45/complete5903lb # [ 137.320120] niks3-server[1111]: 2026/09/21 14:31:40 INFO Completed upload id=455904worker2 # [ 137.313814] nix-grpc-daemon[1410]: time=2026-09-21T14:31:40.929Z level=INFO msg="Upload complete. (384ms)"5905lb # [ 137.322617] niks3-server[1111]: 2026/09/21 14:31:40 INFO Registered completed upload object_key=k3l35rf796jrnjns08398p1l8fcx178a.narinfo5906worker2 # [ 137.323854] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:40Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:57802 duration_s=0 paths=1 nar_bytes_in=5285907worker2 # [ 137.340368] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:578025908worker2 # [ 137.350110] nix-daemon[838]: accepted connection from pid 1314, user nix-grpc-daemon (trusted)5909worker2 # [ 137.353001] nix-daemon[838]: reaped child process 1487, status = succeeded5910client # building '/nix/store/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv' on lb:50051...5911worker2 # [ 137.365660] nix-daemon[838]: accepted connection from pid 1314, user nix-grpc-daemon (trusted)5912client # farm-a> worker2: building k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv5913client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds5914client # farm-a> LOG-farm-a-during-drain5915worker2 # [ 138.860406] nix-daemon[838]: reaped child process 1501, status = succeeded5916worker2 # [ 138.924912] nix-daemon[838]: accepted connection from pid 1517, user nix-grpc-daemon (trusted)5917worker2 # [ 138.944719] nix-daemon[838]: reaped child process 1520, status = succeeded5918lb # [ 138.958029] niks3-server[1111]: 2026/09/21 14:31:42 INFO Received uploads request method=POST path=/api/pending_closures5919worker2 # [ 138.960606] nix-grpc-daemon[1410]: time=2026-09-21T14:31:42.575Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5920worker2 # [ 138.965780] nix-grpc-daemon[1410]: time=2026-09-21T14:31:42.575Z level=INFO msg="Uploading 42c24pndjip9nvdj8j25mkv0650griqf-farm-a (136B)"5921lb # [ 138.992324] niks3-server[1111]: 2026/09/21 14:31:42 INFO Registered completed upload object_key=log/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv5922lb # [ 139.010010] niks3-server[1111]: 2026/09/21 14:31:42 INFO Registered completed upload object_key=nar/0r0px9vw070lid3q4jhnmka11sk2y8kj26ysyvjy9dplavd7mdz4.nar.zst5923worker2 # [ 139.018671] nix-grpc-daemon[1410]: time=2026-09-21T14:31:42.633Z level=INFO msg="Uploading 1 narinfos"5924lb # [ 139.026637] envoy[909]: conn peer=[2001:db8:1::1]:53716 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4301 tx=2002 ms=94235925lb # [ 139.035744] envoy[909]: conn peer=[2001:db8:1::1]:58242 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2987 tx=828 ms=35485926lb # [ 139.045334] niks3-server[1111]: 2026/09/21 14:31:42 INFO Received sign narinfos request method=POST path=/api/pending_closures/46/sign5927lb # [ 139.051967] niks3-server[1111]: 2026/09/21 14:31:42 INFO Signed narinfos id=46 count=15928lb # [ 139.055665] niks3-server[1111]: 2026/09/21 14:31:42 INFO Registered completed upload object_key=42c24pndjip9nvdj8j25mkv0650griqf.ls5929worker2 # [ 139.045463] nix-grpc-daemon[1410]: time=2026-09-21T14:31:42.661Z level=INFO msg="Upload complete. (269ms)"5930lb # [ 139.060688] niks3-server[1111]: 2026/09/21 14:31:42 INFO Received complete upload request method=POST path=/api/pending_closures/46/complete5931lb # [ 139.064891] niks3-server[1111]: 2026/09/21 14:31:42 INFO Completed upload id=465932lb # [ 139.066894] niks3-server[1111]: 2026/09/21 14:31:42 INFO Registered completed upload object_key=42c24pndjip9nvdj8j25mkv0650griqf.narinfo5933worker2 # [ 139.057342] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:42Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:57802 duration_s=1 drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv assign_id=4 outputs=15934worker2 # [ 139.080505] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:42Z level=debug event=want client=4 drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv cached=05935worker2 # [ 139.085275] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:42Z level=debug event=assigned drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv worker=192.168.1.4:50051 clients=15936worker2 # [ 139.089612] nix-daemon[838]: reaped child process 1499, status = succeeded5937worker2 # [ 139.122182] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:42Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:578025938worker2 # [ 139.143088] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:42Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:578025939worker2 # [ 139.157457] nix-grpc-daemon[1410]: time=2026-09-21T14:31:42.772Z level=INFO msg="All 1 paths already cached"5940client # copying 1 paths to 'lb:50051'...5941worker2 # [ 139.167856] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:42Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:578025942client # copying path '/nix/store/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv' to 'lb:50051'...5943worker2 # [ 139.190687] nix-daemon[838]: accepted connection from pid 1314, user nix-grpc-daemon (trusted)5944worker2 # [ 139.350798] nix-daemon[838]: accepted connection from pid 1526, user nix-grpc-daemon (trusted)5945worker2 # [ 139.371659] nix-daemon[838]: reaped child process 1529, status = succeeded5946worker2 # [ 139.473133] nix-daemon[838]: accepted connection from pid 1531, user nix-grpc-daemon (trusted)5947worker2 # [ 139.492546] nix-daemon[838]: reaped child process 1534, status = succeeded5948lb # [ 139.501308] niks3-server[1111]: 2026/09/21 14:31:43 INFO Received uploads request method=POST path=/api/pending_closures5949worker2 # [ 139.504833] nix-grpc-daemon[1410]: time=2026-09-21T14:31:43.120Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5950worker2 # [ 139.506295] nix-grpc-daemon[1410]: time=2026-09-21T14:31:43.120Z level=INFO msg="Uploading im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv (640B)"5951lb # [ 139.538975] niks3-server[1111]: 2026/09/21 14:31:43 INFO Registered completed upload object_key=nar/0rxmdnq61ka29n0b2as2hdbz91wi2ap17db890js8r5plhpg436g.nar.zst5952lb # [ 139.552900] niks3-server[1111]: 2026/09/21 14:31:43 INFO Received sign narinfos request method=POST path=/api/pending_closures/47/sign5953worker2 # [ 139.555342] nix-grpc-daemon[1410]: time=2026-09-21T14:31:43.170Z level=INFO msg="Uploading 1 narinfos"5954lb # [ 139.566298] niks3-server[1111]: 2026/09/21 14:31:43 INFO Signed narinfos id=47 count=15955lb # [ 139.571673] niks3-server[1111]: 2026/09/21 14:31:43 INFO Registered completed upload object_key=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l.ls5956lb # [ 139.585011] niks3-server[1111]: 2026/09/21 14:31:43 INFO Received complete upload request method=POST path=/api/pending_closures/47/complete5957worker2 # [ 139.584720] nix-grpc-daemon[1410]: time=2026-09-21T14:31:43.200Z level=INFO msg="Upload complete. (376ms)"5958lb # [ 139.592818] niks3-server[1111]: 2026/09/21 14:31:43 INFO Completed upload id=475959lb # [ 139.594990] niks3-server[1111]: 2026/09/21 14:31:43 INFO Registered completed upload object_key=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l.narinfo5960worker2 # [ 139.596166] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:43Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:57802 duration_s=0 paths=1 nar_bytes_in=6405961worker2 # [ 139.605072] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:578025962worker2 # [ 139.616628] nix-daemon[838]: accepted connection from pid 1314, user nix-grpc-daemon (trusted)5963worker2 # [ 139.621612] nix-daemon[838]: reaped child process 1524, status = succeeded5964client # building '/nix/store/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv' on lb:50051...5965client # farm-b> worker2: building im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv5966worker2 # [ 139.636163] nix-daemon[838]: accepted connection from pid 1314, user nix-grpc-daemon (trusted)5967client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds5968client # farm-b> LOG-farm-b-during-drain5969worker2 # [ 140.462794] nix-daemon[838]: reaped child process 1538, status = succeeded5970worker2 # [ 140.558906] nix-daemon[838]: accepted connection from pid 1554, user nix-grpc-daemon (trusted)5971worker2 # [ 140.579471] nix-daemon[838]: reaped child process 1557, status = succeeded5972lb # [ 140.591185] niks3-server[1111]: 2026/09/21 14:31:44 INFO Received uploads request method=POST path=/api/pending_closures5973worker2 # [ 140.596610] nix-grpc-daemon[1410]: time=2026-09-21T14:31:44.211Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5974worker2 # [ 140.598055] nix-grpc-daemon[1410]: time=2026-09-21T14:31:44.211Z level=INFO msg="Uploading daks7y2c761g04qpzsjx25629wigr8js-farm-b (184B)"5975lb # [ 140.631257] niks3-server[1111]: 2026/09/21 14:31:44 INFO Registered completed upload object_key=nar/0lnpxl34nrzp147jy5bafczkvxcl00hcf69qjhsljmv2ip9jisfh.nar.zst5976lb # [ 140.644133] niks3-server[1111]: 2026/09/21 14:31:44 INFO Registered completed upload object_key=log/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv5977worker2 # [ 140.648696] nix-grpc-daemon[1410]: time=2026-09-21T14:31:44.263Z level=INFO msg="Uploading 1 narinfos"5978lb # [ 140.658112] niks3-server[1111]: 2026/09/21 14:31:44 INFO Received sign narinfos request method=POST path=/api/pending_closures/48/sign5979lb # [ 140.666193] niks3-server[1111]: 2026/09/21 14:31:44 INFO Signed narinfos id=48 count=15980lb # [ 140.672280] niks3-server[1111]: 2026/09/21 14:31:44 INFO Registered completed upload object_key=daks7y2c761g04qpzsjx25629wigr8js.ls5981lb # [ 140.682615] niks3-server[1111]: 2026/09/21 14:31:44 INFO Received complete upload request method=POST path=/api/pending_closures/48/complete5982lb # [ 140.688950] niks3-server[1111]: 2026/09/21 14:31:44 INFO Completed upload id=485983worker2 # [ 140.681641] nix-grpc-daemon[1410]: time=2026-09-21T14:31:44.297Z level=INFO msg="Upload complete. (236ms)"5984lb # [ 140.691416] niks3-server[1111]: 2026/09/21 14:31:44 INFO Registered completed upload object_key=daks7y2c761g04qpzsjx25629wigr8js.narinfo5985worker2 # [ 140.701035] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:44Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:57802 duration_s=1 drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv assign_id=5 outputs=15986worker2 # [ 140.716507] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:44Z level=debug event=want client=4 drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv cached=05987worker2 # [ 140.723045] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:44Z level=debug event=assigned drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv worker=192.168.1.4:50051 clients=15988worker2 # [ 140.727750] nix-daemon[838]: reaped child process 1536, status = succeeded5989worker2 # [ 140.745962] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:578025990worker2 # [ 140.761185] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:44Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:578025991worker2 # [ 140.776503] nix-grpc-daemon[1410]: time=2026-09-21T14:31:44.392Z level=INFO msg="All 2 paths already cached"5992client # copying 1 paths to 'lb:50051'...5993client # copying path '/nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv' to 'lb:50051'...5994worker2 # [ 140.786521] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:44Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:578025995worker2 # [ 140.800301] nix-daemon[838]: accepted connection from pid 1314, user nix-grpc-daemon (trusted)5996worker2 # [ 140.959432] nix-daemon[838]: accepted connection from pid 1563, user nix-grpc-daemon (trusted)5997worker2 # [ 140.980975] nix-daemon[838]: reaped child process 1566, status = succeeded5998worker2 # [ 141.077420] nix-daemon[838]: accepted connection from pid 1568, user nix-grpc-daemon (trusted)5999worker2 # [ 141.093790] nix-daemon[838]: reaped child process 1571, status = succeeded6000lb # [ 141.106795] niks3-server[1111]: 2026/09/21 14:31:44 INFO Received uploads request method=POST path=/api/pending_closures6001worker2 # [ 141.109837] nix-grpc-daemon[1410]: time=2026-09-21T14:31:44.725Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6002worker2 # [ 141.111333] nix-grpc-daemon[1410]: time=2026-09-21T14:31:44.725Z level=INFO msg="Uploading lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv (768B)"6003lb # [ 141.133546] niks3-server[1111]: 2026/09/21 14:31:44 INFO Registered completed upload object_key=nar/066jhl91shf8ar6h3cqqb855j5cswl7170wrg5lzf3fhpka0xywn.nar.zst6004lb # [ 141.148684] niks3-server[1111]: 2026/09/21 14:31:44 INFO Received sign narinfos request method=POST path=/api/pending_closures/49/sign6005worker2 # [ 141.149342] nix-grpc-daemon[1410]: time=2026-09-21T14:31:44.764Z level=INFO msg="Uploading 1 narinfos"6006lb # [ 141.161727] niks3-server[1111]: 2026/09/21 14:31:44 INFO Signed narinfos id=49 count=16007lb # [ 141.167214] niks3-server[1111]: 2026/09/21 14:31:44 INFO Registered completed upload object_key=lbk8wy49c881163x07gb8z669j2fg1ff.ls6008lb # [ 141.176684] niks3-server[1111]: 2026/09/21 14:31:44 INFO Received complete upload request method=POST path=/api/pending_closures/49/complete6009worker2 # [ 141.175345] nix-grpc-daemon[1410]: time=2026-09-21T14:31:44.790Z level=INFO msg="Upload complete. (359ms)"6010lb # [ 141.183960] niks3-server[1111]: 2026/09/21 14:31:44 INFO Completed upload id=496011lb # [ 141.187662] niks3-server[1111]: 2026/09/21 14:31:44 INFO Registered completed upload object_key=lbk8wy49c881163x07gb8z669j2fg1ff.narinfo6012worker2 # [ 141.184271] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:44Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:57802 duration_s=0 paths=1 nar_bytes_in=7686013worker2 # [ 141.195182] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:578026014worker2 # [ 141.202200] nix-daemon[838]: accepted connection from pid 1314, user nix-grpc-daemon (trusted)6015worker2 # [ 141.207032] nix-daemon[838]: reaped child process 1561, status = succeeded6016client # building '/nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv' on lb:50051...6017worker2 # [ 141.218846] nix-daemon[838]: accepted connection from pid 1314, user nix-grpc-daemon (trusted)6018client # farm-top> worker2: building lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv6019client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds6020client # farm-top> LOG-farm-top-during-drain6021worker2 # [ 142.260804] nix-daemon[838]: reaped child process 1575, status = succeeded6022worker2 # [ 142.365168] nix-daemon[838]: accepted connection from pid 1591, user nix-grpc-daemon (trusted)6023worker2 # [ 142.387817] nix-daemon[838]: reaped child process 1594, status = succeeded6024lb # [ 142.393482] niks3-server[1111]: 2026/09/21 14:31:46 INFO Received uploads request method=POST path=/api/pending_closures6025worker2 # [ 142.398564] nix-grpc-daemon[1410]: time=2026-09-21T14:31:46.012Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6026worker2 # [ 142.400260] nix-grpc-daemon[1410]: time=2026-09-21T14:31:46.012Z level=INFO msg="Uploading xdwmx2f4n5n8bggglniz2n35any0xz4w-farm-top (240B)"6027lb # [ 142.426549] niks3-server[1111]: 2026/09/21 14:31:46 INFO Registered completed upload object_key=log/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv6028lb # [ 142.444102] niks3-server[1111]: 2026/09/21 14:31:46 INFO Registered completed upload object_key=nar/1bpy9ga4n2446pary6msidbs8mnil1a34jn8p8yl2scvdg691mz2.nar.zst6029lb # [ 142.455700] niks3-server[1111]: 2026/09/21 14:31:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/50/sign6030worker2 # [ 142.455175] nix-grpc-daemon[1410]: time=2026-09-21T14:31:46.070Z level=INFO msg="Uploading 1 narinfos"6031lb # [ 142.466395] niks3-server[1111]: 2026/09/21 14:31:46 INFO Signed narinfos id=50 count=16032lb # [ 142.472311] niks3-server[1111]: 2026/09/21 14:31:46 INFO Registered completed upload object_key=xdwmx2f4n5n8bggglniz2n35any0xz4w.ls6033lb # [ 142.480632] niks3-server[1111]: 2026/09/21 14:31:46 INFO Received complete upload request method=POST path=/api/pending_closures/50/complete6034lb # [ 142.490661] niks3-server[1111]: 2026/09/21 14:31:46 INFO Completed upload id=506035worker2 # [ 142.482906] nix-grpc-daemon[1410]: time=2026-09-21T14:31:46.098Z level=INFO msg="Upload complete. (251ms)"6036lb # [ 142.495417] niks3-server[1111]: 2026/09/21 14:31:46 INFO Registered completed upload object_key=xdwmx2f4n5n8bggglniz2n35any0xz4w.narinfo6037worker2 # [ 142.495122] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:46Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:57802 duration_s=1 drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv assign_id=6 outputs=16038worker2 # [ 142.515223] nix-daemon[838]: reaped child process 1573, status = succeeded6039client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag during-drain >&2, in 6.01 seconds)6040client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag during-drain outPath6041client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag during-drain outPath, in 0.18 seconds)6042worker1: must succeed: pkill -f 'read -t [9]0 x'6043worker1: (finished: must succeed: pkill -f 'read -t [9]0 x', in 0.05 seconds)6044client: waiting for success: ! systemctl is-active sw6045worker1 # [ 142.972828] nix-daemon[851]: reaped child process 1803, status = succeeded6046worker1 # [ 143.082760] nix-daemon[851]: accepted connection from pid 1840, user nix-grpc-daemon (trusted)6047worker1 # [ 143.104457] nix-daemon[851]: reaped child process 1843, status = succeeded6048lb # [ 143.106683] niks3-server[1111]: 2026/09/21 14:31:46 INFO Received uploads request method=POST path=/api/pending_closures6049worker1 # [ 143.117269] nix-grpc-daemon[1739]: time=2026-09-21T14:31:46.721Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6050worker1 # [ 143.118837] nix-grpc-daemon[1739]: time=2026-09-21T14:31:46.721Z level=INFO msg="Uploading qr8vyl2zl3nxnin3vw41c6z35vj7087f-slow-sw (120B)"6051lb # [ 143.138422] niks3-server[1111]: 2026/09/21 14:31:46 INFO Registered completed upload object_key=log/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv6052lb # [ 143.154392] niks3-server[1111]: 2026/09/21 14:31:46 INFO Registered completed upload object_key=nar/0vi6cx31wc05wns04pr5vr2bak9iib1idndvy6gik6c2rz9q8c19.nar.zst6053lb # [ 143.166344] niks3-server[1111]: 2026/09/21 14:31:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/51/sign6054worker1 # [ 143.172123] nix-grpc-daemon[1739]: time=2026-09-21T14:31:46.775Z level=INFO msg="Uploading 1 narinfos"6055lb # [ 143.176542] niks3-server[1111]: 2026/09/21 14:31:46 INFO Signed narinfos id=51 count=16056lb # [ 143.181824] niks3-server[1111]: 2026/09/21 14:31:46 INFO Registered completed upload object_key=qr8vyl2zl3nxnin3vw41c6z35vj7087f.ls6057lb # [ 143.190736] niks3-server[1111]: 2026/09/21 14:31:46 INFO Received complete upload request method=POST path=/api/pending_closures/51/complete6058lb # [ 143.196886] niks3-server[1111]: 2026/09/21 14:31:46 INFO Completed upload id=516059worker1 # [ 143.197116] nix-grpc-daemon[1739]: time=2026-09-21T14:31:46.801Z level=INFO msg="Upload complete. (278ms)"6060lb # [ 143.200471] niks3-server[1111]: 2026/09/21 14:31:46 INFO Registered completed upload object_key=qr8vyl2zl3nxnin3vw41c6z35vj7087f.narinfo6061worker1 # [ 143.209081] nix-grpc-daemon[1658]: ts=2026-09-21T14:31:46Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43332 duration_s=7 drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv assign_id=3 outputs=16062worker1 # [ 143.228749] nix-daemon[851]: reaped child process 1801, status = succeeded6063client # [ 143.248532] systemd[1]: sw.service: Deactivated successfully.6064client # [ 143.253677] systemd[1]: sw.service: Consumed 186ms CPU time over 9.524s wall clock time, 41.4M memory peak, 7.5K incoming IP traffic, 8.5K outgoing IP traffic.6065worker1 # [ 143.867280] nix-grpc-daemon[1658]: ts=2026-09-21T14:31:47Z level=info event=signal_exit6066client: (finished: waiting for success: ! systemctl is-active sw, in 1.10 seconds)6067client: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }6068client: (finished: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }, in 0.06 seconds)6069worker1: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service6070worker2 # [ 145.867006] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:49Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:527546071worker2 # [ 145.871963] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:49Z level=info event=worker_gone addr=192.168.1.3:500516072worker1 # [ 145.879697] nix-grpc-daemon[1658]: ts=2026-09-21T14:31:49Z level=info event=scheduler_restarting6073worker1 # [ 145.897013] nix-grpc-daemon[1658]: ts=2026-09-21T14:31:49Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"6074worker1 # [ 145.922855] nix-daemon[851]: reaped child process 1710, status = succeeded6075worker1 # [ 145.926835] nix-grpc-daemon[1658]: E0000 00:00:1790001109.531174 1703 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 16, started true, shutdown true6076worker1 # [ 145.933606] nix-grpc-daemon[1658]: E0000 00:00:1790001109.531463 1703 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true6077worker1 # [ 147.897782] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.6078worker1 # [ 147.903987] systemd[1]: nix-grpc-daemon.service: Consumed 929ms CPU time over 24.568s wall clock time, 35.5M memory peak, 21.5K incoming IP traffic, 29.3K outgoing IP traffic.6079worker1 # [ 147.924619] systemd[1]: Starting Nix worker-protocol over gRPC...6080worker1 # [ 148.072299] nix-grpc-daemon[1857]: ts=2026-09-21T14:31:51Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6081worker1 # [ 148.074671] nix-grpc-daemon[1857]: ts=2026-09-21T14:31:51Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6082worker1 # [ 148.078615] nix-grpc-daemon[1857]: ts=2026-09-21T14:31:51Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=16083worker1 # [ 148.084060] nix-grpc-daemon[1857]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6084worker1 # [ 148.098587] nix-grpc-daemon[1857]: E0000 00:00:1790001111.688353 1857 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6085worker1: (finished: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service, in 4.19 seconds)6086lb: waiting until exactly no endpoints of aarch64-linux are unhealthy6087lb: must succeed: curl -sf localhost:9901/clusters6088worker1 # [ 148.128158] nix-grpc-daemon[1857]: E0000 00:00:1790001111.692128 1857 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6089worker1 # [ 148.130422] nix-grpc-daemon[1857]: ts=2026-09-21T14:31:51Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6090worker1 # [ 148.136500] nix-grpc-daemon[1857]: E0000 00:00:1790001111.701151 1896 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6091worker1 # [ 148.139342] nix-grpc-daemon[1857]: E0000 00:00:1790001111.720453 1896 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6092worker1 # [ 148.141917] nix-grpc-daemon[1857]: E0000 00:00:1790001111.720609 1896 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6093worker1 # [ 148.144256] systemd[1]: Started Nix worker-protocol over gRPC.6094worker1 # [ 148.145157] nix-grpc-daemon[1857]: E0000 00:00:1790001111.740304 1896 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6095worker1 # [ 148.154773] nix-grpc-daemon[1857]: ts=2026-09-21T14:31:51Z level=info event=scheduler_connected addr=lb:500516096worker2 # [ 148.149856] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:51Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:527546097worker2 # [ 148.156196] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:51Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=06098lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6099lb # [ 149.020368] envoy[909]: conn peer=[2001:db8:1::1]:42164 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9621 tx=4089 ms=57536100lb # [ 149.020614] envoy[909]: conn peer=[2001:db8:1::1]:42148 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2982 tx=1446 ms=93086101lb # [ 149.020711] envoy[909]: conn peer=[2001:db8:1::3]:59774 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=1813 tx=616 ms=219646102lb: must succeed: curl -sf localhost:9901/clusters6103lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)6104lb: (finished: waiting until exactly no endpoints of aarch64-linux are unhealthy, in 1.15 seconds)6105lb: must succeed: curl -sf localhost:9901/clusters6106lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)6107worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6108worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6109(finished: subtest: a deploy mid-build drains: the build finishes, the new generation takes over, in 15.72 seconds)6110subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does6111client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&16112worker1 # [ 149.613897] nix-grpc-daemon[1857]: ts=2026-09-21T14:31:53Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:392366113worker1 # [ 149.628850] nix-daemon[851]: accepted connection from pid 1857, user nix-grpc-daemon (trusted)6114worker2 # [ 149.646128] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:53Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:578026115worker2 # [ 149.690149] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:53Z level=debug event=want client=5 drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv cached=06116worker2 # [ 149.696400] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:53Z level=debug event=assigned drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv worker=192.168.1.3:50051 clients=16117worker2 # [ 149.702321] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:53Z level=debug event=want client=5 drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv cached=06118worker2 # [ 149.707531] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:53Z level=debug event=want client=5 drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv cached=06119worker1 # [ 149.744527] nix-grpc-daemon[1857]: ts=2026-09-21T14:31:53Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:392366120worker1 # [ 149.763571] nix-grpc-daemon[1857]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 73 ms (attempt 1/5)6121worker1 # [ 149.845926] nix-grpc-daemon[1857]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 17 ms (attempt 2/5)6122worker1 # [ 149.864850] nix-grpc-daemon[1857]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 53 ms (attempt 3/5)6123worker1 # [ 149.920584] nix-grpc-daemon[1857]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 500 ms (attempt 4/5)6124worker1 # [ 150.422536] nix-grpc-daemon[1857]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6125worker1 # [ 150.428345] nix-grpc-daemon[1857]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6126worker1 # [ 150.446266] nix-grpc-daemon[1857]: ts=2026-09-21T14:31:54Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:392366127worker1 # [ 150.461820] nix-grpc-daemon[1857]: ts=2026-09-21T14:31:54Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:392366128worker1 # [ 150.475346] nix-daemon[851]: accepted connection from pid 1857, user nix-grpc-daemon (trusted)6129worker1 # [ 150.502617] nix-grpc-daemon[1857]: ts=2026-09-21T14:31:54Z level=info event=niks3_push_started pid=19176130worker1 # [ 150.649285] nix-grpc-daemon[1917]: time=2026-09-21T14:31:54.253Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.key ca=""6131worker1 # [ 150.758380] nix-daemon[851]: accepted connection from pid 1930, user nix-grpc-daemon (trusted)6132worker1 # [ 150.782426] nix-daemon[851]: reaped child process 1933, status = succeeded6133worker1 # [ 150.875868] nix-daemon[851]: accepted connection from pid 1935, user nix-grpc-daemon (trusted)6134worker1 # [ 150.890274] nix-daemon[851]: reaped child process 1938, status = succeeded6135lb # [ 150.898501] niks3-server[1111]: 2026/09/21 14:31:54 INFO Received uploads request method=POST path=/api/pending_closures6136worker1 # [ 150.909333] nix-grpc-daemon[1917]: time=2026-09-21T14:31:54.512Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6137worker1 # [ 150.910784] nix-grpc-daemon[1917]: time=2026-09-21T14:31:54.512Z level=INFO msg="Uploading 3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv (512B)"6138lb # [ 150.942185] niks3-server[1111]: 2026/09/21 14:31:54 INFO Registered completed upload object_key=nar/0zxfsmqijfc2dkglls3yp1x3w0w1na7vx39fxc124zddjzpaygd8.nar.zst6139lb # [ 150.961356] niks3-server[1111]: 2026/09/21 14:31:54 INFO Received sign narinfos request method=POST path=/api/pending_closures/52/sign6140worker1 # [ 150.967607] nix-grpc-daemon[1917]: time=2026-09-21T14:31:54.571Z level=INFO msg="Uploading 1 narinfos"6141lb # [ 150.970134] niks3-server[1111]: 2026/09/21 14:31:54 INFO Signed narinfos id=52 count=16142lb # [ 150.977025] niks3-server[1111]: 2026/09/21 14:31:54 INFO Registered completed upload object_key=3223c4sl4ncpldbvw558vqwddijwzfpd.ls6143lb # [ 150.987974] niks3-server[1111]: 2026/09/21 14:31:54 INFO Received complete upload request method=POST path=/api/pending_closures/52/complete6144lb # [ 150.996729] niks3-server[1111]: 2026/09/21 14:31:54 INFO Completed upload id=526145worker1 # [ 150.999983] nix-grpc-daemon[1917]: time=2026-09-21T14:31:54.604Z level=INFO msg="Upload complete. (347ms)"6146lb # [ 151.000568] niks3-server[1111]: 2026/09/21 14:31:54 INFO Registered completed upload object_key=3223c4sl4ncpldbvw558vqwddijwzfpd.narinfo6147worker1 # [ 151.009231] nix-grpc-daemon[1857]: ts=2026-09-21T14:31:54Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39236 duration_s=0 paths=1 nar_bytes_in=5126148worker1 # [ 151.023717] nix-grpc-daemon[1857]: ts=2026-09-21T14:31:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:392366149worker1 # [ 151.033792] nix-daemon[851]: accepted connection from pid 1857, user nix-grpc-daemon (trusted)6150worker1 # [ 151.036855] nix-daemon[851]: reaped child process 1915, status = succeeded6151worker1 # [ 151.051030] nix-daemon[851]: accepted connection from pid 1857, user nix-grpc-daemon (trusted)6152worker1 # [ 151.947439] nix-daemon[851]: reaped child process 1942, status = succeeded6153worker1 # [ 152.013359] nix-daemon[851]: accepted connection from pid 1958, user nix-grpc-daemon (trusted)6154worker1 # [ 152.029963] nix-daemon[851]: reaped child process 1961, status = succeeded6155lb # [ 152.038516] niks3-server[1111]: 2026/09/21 14:31:55 INFO Received uploads request method=POST path=/api/pending_closures6156worker1 # [ 152.049211] nix-grpc-daemon[1917]: time=2026-09-21T14:31:55.651Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6157worker1 # [ 152.051144] nix-grpc-daemon[1917]: time=2026-09-21T14:31:55.651Z level=INFO msg="Uploading dla3w0zf02z5jhccwa27i85bbgzrrsvb-farm-a (128B)"6158lb # [ 152.097891] niks3-server[1111]: 2026/09/21 14:31:55 INFO Registered completed upload object_key=log/3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv6159lb # [ 152.106909] niks3-server[1111]: 2026/09/21 14:31:55 INFO Registered completed upload object_key=nar/05i7ka1qa7vmhiys96ln3pp3nj3rfdlslnbydrxdqj40cmvmqcwf.nar.zst6160worker1 # [ 152.116128] nix-grpc-daemon[1917]: time=2026-09-21T14:31:55.720Z level=INFO msg="Uploading 1 narinfos"6161lb # [ 152.116599] niks3-server[1111]: 2026/09/21 14:31:55 INFO Received sign narinfos request method=POST path=/api/pending_closures/53/sign6162lb # [ 152.124227] niks3-server[1111]: 2026/09/21 14:31:55 INFO Signed narinfos id=53 count=16163lb # [ 152.128624] niks3-server[1111]: 2026/09/21 14:31:55 INFO Registered completed upload object_key=dla3w0zf02z5jhccwa27i85bbgzrrsvb.ls6164lb # [ 152.142072] niks3-server[1111]: 2026/09/21 14:31:55 INFO Received complete upload request method=POST path=/api/pending_closures/53/complete6165lb # [ 152.148519] niks3-server[1111]: 2026/09/21 14:31:55 INFO Completed upload id=536166worker1 # [ 152.146867] nix-grpc-daemon[1917]: time=2026-09-21T14:31:55.751Z level=INFO msg="Upload complete. (285ms)"6167lb # [ 152.150463] niks3-server[1111]: 2026/09/21 14:31:55 INFO Registered completed upload object_key=dla3w0zf02z5jhccwa27i85bbgzrrsvb.narinfo6168worker2 # [ 152.150224] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:55Z level=debug event=assigned drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv worker=192.168.1.3:50051 clients=16169worker1 # [ 152.161356] nix-grpc-daemon[1857]: ts=2026-09-21T14:31:55Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39236 duration_s=1 drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv assign_id=7 outputs=16170worker2 # [ 152.163710] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:55Z level=debug event=want client=5 drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv cached=06171worker1 # [ 152.180428] nix-grpc-daemon[1857]: ts=2026-09-21T14:31:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:392366172worker1 # [ 152.189179] nix-daemon[851]: reaped child process 1940, status = succeeded6173worker1 # [ 152.193467] nix-grpc-daemon[1857]: ts=2026-09-21T14:31:55Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:392366174worker1 # [ 152.201930] nix-grpc-daemon[1857]: ts=2026-09-21T14:31:55Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:392366175worker1 # [ 152.212548] nix-daemon[851]: accepted connection from pid 1857, user nix-grpc-daemon (trusted)6176worker1 # [ 152.352116] nix-daemon[851]: accepted connection from pid 1965, user nix-grpc-daemon (trusted)6177worker1 # [ 152.374988] nix-daemon[851]: reaped child process 1968, status = succeeded6178worker1 # [ 152.471328] nix-daemon[851]: accepted connection from pid 1970, user nix-grpc-daemon (trusted)6179worker1 # [ 152.486624] nix-daemon[851]: reaped child process 1973, status = succeeded6180lb # [ 152.493317] niks3-server[1111]: 2026/09/21 14:31:56 INFO Received uploads request method=POST path=/api/pending_closures6181worker1 # [ 152.502485] nix-grpc-daemon[1917]: time=2026-09-21T14:31:56.106Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6182worker1 # [ 152.503948] nix-grpc-daemon[1917]: time=2026-09-21T14:31:56.106Z level=INFO msg="Uploading pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv (512B)"6183lb # [ 152.520517] niks3-server[1111]: 2026/09/21 14:31:56 INFO Registered completed upload object_key=nar/0g5bxr683biqh358mkd9xrclwllan4cgnv4pzijas586xj68wwx0.nar.zst6184lb # [ 152.535754] niks3-server[1111]: 2026/09/21 14:31:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/54/sign6185worker1 # [ 152.542976] nix-grpc-daemon[1917]: time=2026-09-21T14:31:56.147Z level=INFO msg="Uploading 1 narinfos"6186lb # [ 152.546986] niks3-server[1111]: 2026/09/21 14:31:56 INFO Signed narinfos id=54 count=16187lb # [ 152.552731] niks3-server[1111]: 2026/09/21 14:31:56 INFO Registered completed upload object_key=pplpzcnjfn295nm16cg0iihpfc9iffry.ls6188worker1 # [ 152.564351] nix-grpc-daemon[1917]: time=2026-09-21T14:31:56.168Z level=INFO msg="Upload complete. (338ms)"6189lb # [ 152.563709] niks3-server[1111]: 2026/09/21 14:31:56 INFO Received complete upload request method=POST path=/api/pending_closures/54/complete6190lb # [ 152.568790] niks3-server[1111]: 2026/09/21 14:31:56 INFO Completed upload id=546191lb # [ 152.571619] niks3-server[1111]: 2026/09/21 14:31:56 INFO Registered completed upload object_key=pplpzcnjfn295nm16cg0iihpfc9iffry.narinfo6192worker1 # [ 152.572794] nix-grpc-daemon[1857]: ts=2026-09-21T14:31:56Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39236 duration_s=0 paths=1 nar_bytes_in=5126193worker1 # [ 152.582483] nix-grpc-daemon[1857]: ts=2026-09-21T14:31:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:392366194worker1 # [ 152.587633] nix-daemon[851]: accepted connection from pid 1857, user nix-grpc-daemon (trusted)6195worker1 # [ 152.591677] nix-daemon[851]: reaped child process 1963, status = succeeded6196worker1 # [ 152.601342] nix-daemon[851]: accepted connection from pid 1857, user nix-grpc-daemon (trusted)6197worker1 # [ 153.644442] nix-daemon[851]: reaped child process 1977, status = succeeded6198worker1 # [ 153.750676] nix-daemon[851]: accepted connection from pid 1993, user nix-grpc-daemon (trusted)6199worker1 # [ 153.768291] nix-daemon[851]: reaped child process 1996, status = succeeded6200lb # [ 153.775709] niks3-server[1111]: 2026/09/21 14:31:57 INFO Received uploads request method=POST path=/api/pending_closures6201worker1 # [ 153.786262] nix-grpc-daemon[1917]: time=2026-09-21T14:31:57.390Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6202worker1 # [ 153.787752] nix-grpc-daemon[1917]: time=2026-09-21T14:31:57.390Z level=INFO msg="Uploading fhc1zkcygpzppf0bj5fr2k85lf7p87v9-farm-a (128B)"6203lb # [ 153.814773] niks3-server[1111]: 2026/09/21 14:31:57 INFO Registered completed upload object_key=nar/09na6djn66dpys8h08gqw39p0vw7vxqsdzbiw55rknyna8s239bm.nar.zst6204lb # [ 153.826982] niks3-server[1111]: 2026/09/21 14:31:57 INFO Registered completed upload object_key=log/pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv6205worker1 # [ 153.835832] nix-grpc-daemon[1917]: time=2026-09-21T14:31:57.440Z level=INFO msg="Uploading 1 narinfos"6206lb # [ 153.837256] niks3-server[1111]: 2026/09/21 14:31:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/55/sign6207lb # [ 153.844565] niks3-server[1111]: 2026/09/21 14:31:57 INFO Signed narinfos id=55 count=16208lb # [ 153.846713] niks3-server[1111]: 2026/09/21 14:31:57 INFO Registered completed upload object_key=fhc1zkcygpzppf0bj5fr2k85lf7p87v9.ls6209lb # [ 153.854678] niks3-server[1111]: 2026/09/21 14:31:57 INFO Received complete upload request method=POST path=/api/pending_closures/55/complete6210lb # [ 153.862156] niks3-server[1111]: 2026/09/21 14:31:57 INFO Completed upload id=556211worker1 # [ 153.862960] nix-grpc-daemon[1917]: time=2026-09-21T14:31:57.467Z level=INFO msg="Upload complete. (251ms)"6212lb # [ 153.864663] niks3-server[1111]: 2026/09/21 14:31:57 INFO Registered completed upload object_key=fhc1zkcygpzppf0bj5fr2k85lf7p87v9.narinfo6213worker2 # [ 153.867933] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:57Z level=debug event=assigned drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv worker=192.168.1.3:50051 clients=16214worker1 # [ 153.875565] nix-grpc-daemon[1857]: ts=2026-09-21T14:31:57Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39236 duration_s=1 drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv assign_id=8 outputs=16215worker2 # [ 153.875546] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:57Z level=debug event=want client=5 drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv cached=06216worker1 # [ 153.894348] nix-grpc-daemon[1857]: ts=2026-09-21T14:31:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:392366217worker1 # [ 153.905747] nix-grpc-daemon[1857]: ts=2026-09-21T14:31:57Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:392366218worker1 # [ 153.912532] nix-grpc-daemon[1857]: ts=2026-09-21T14:31:57Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:392366219worker1 # [ 153.920226] nix-daemon[851]: accepted connection from pid 1857, user nix-grpc-daemon (trusted)6220worker1 # [ 153.928265] nix-daemon[851]: reaped child process 1975, status = succeeded6221worker1 # [ 154.068560] nix-daemon[851]: accepted connection from pid 2000, user nix-grpc-daemon (trusted)6222worker1 # [ 154.086592] nix-daemon[851]: reaped child process 2003, status = succeeded6223worker1 # [ 154.201562] nix-daemon[851]: accepted connection from pid 2005, user nix-grpc-daemon (trusted)6224worker1 # [ 154.220707] nix-daemon[851]: reaped child process 2008, status = succeeded6225lb # [ 154.218447] niks3-server[1111]: 2026/09/21 14:31:57 INFO Received uploads request method=POST path=/api/pending_closures6226worker1 # [ 154.227914] nix-grpc-daemon[1917]: time=2026-09-21T14:31:57.832Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6227worker1 # [ 154.229633] nix-grpc-daemon[1917]: time=2026-09-21T14:31:57.832Z level=INFO msg="Uploading sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv (512B)"6228lb # [ 154.245727] niks3-server[1111]: 2026/09/21 14:31:57 INFO Registered completed upload object_key=nar/1fxz200fmpbckafrn03b2snx9knqpbwf8rwqlbvg68356qj7aj5w.nar.zst6229worker1 # [ 154.260058] nix-grpc-daemon[1917]: time=2026-09-21T14:31:57.864Z level=INFO msg="Uploading 1 narinfos"6230lb # [ 154.262184] niks3-server[1111]: 2026/09/21 14:31:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/56/sign6231lb # [ 154.272442] niks3-server[1111]: 2026/09/21 14:31:57 INFO Signed narinfos id=56 count=16232lb # [ 154.275414] niks3-server[1111]: 2026/09/21 14:31:57 INFO Registered completed upload object_key=sy43wp9v52pni1wms6q0p3qybpc2pwad.ls6233worker1 # [ 154.293397] nix-grpc-daemon[1917]: time=2026-09-21T14:31:57.898Z level=INFO msg="Upload complete. (357ms)"6234lb # [ 154.283991] niks3-server[1111]: 2026/09/21 14:31:57 INFO Received complete upload request method=POST path=/api/pending_closures/56/complete6235lb # [ 154.297215] niks3-server[1111]: 2026/09/21 14:31:57 INFO Completed upload id=566236worker1 # [ 154.300237] nix-grpc-daemon[1857]: ts=2026-09-21T14:31:57Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39236 duration_s=0 paths=1 nar_bytes_in=5126237lb # [ 154.303066] niks3-server[1111]: 2026/09/21 14:31:57 INFO Registered completed upload object_key=sy43wp9v52pni1wms6q0p3qybpc2pwad.narinfo6238worker1 # [ 154.314495] nix-grpc-daemon[1857]: ts=2026-09-21T14:31:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:392366239worker1 # [ 154.320553] nix-daemon[851]: accepted connection from pid 1857, user nix-grpc-daemon (trusted)6240worker1 # [ 154.325924] nix-daemon[851]: reaped child process 1998, status = succeeded6241worker1 # [ 154.335830] nix-daemon[851]: accepted connection from pid 1857, user nix-grpc-daemon (trusted)6242worker1 # [ 155.645427] nix-daemon[851]: reaped child process 2012, status = succeeded6243worker1 # [ 155.716048] nix-daemon[851]: accepted connection from pid 2028, user nix-grpc-daemon (trusted)6244worker1 # [ 155.734744] nix-daemon[851]: reaped child process 2031, status = succeeded6245lb # [ 155.742984] niks3-server[1111]: 2026/09/21 14:31:59 INFO Received uploads request method=POST path=/api/pending_closures6246worker1 # [ 155.754835] nix-grpc-daemon[1917]: time=2026-09-21T14:31:59.359Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6247worker1 # [ 155.758997] nix-grpc-daemon[1917]: time=2026-09-21T14:31:59.361Z level=INFO msg="Uploading g5laypvfk2469p8binw3mksk2hv7zhnh-farm-a (128B)"6248lb # [ 155.782529] niks3-server[1111]: 2026/09/21 14:31:59 INFO Registered completed upload object_key=log/sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv6249lb # [ 155.800250] niks3-server[1111]: 2026/09/21 14:31:59 INFO Registered completed upload object_key=nar/1j2671k9mycq8lxyfx5k6a66d5narfqw2wqwlpwxgwzi93vnvbdn.nar.zst6250worker1 # [ 155.812048] nix-grpc-daemon[1917]: time=2026-09-21T14:31:59.416Z level=INFO msg="Uploading 1 narinfos"6251lb # [ 155.814311] niks3-server[1111]: 2026/09/21 14:31:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/57/sign6252lb # [ 155.823194] niks3-server[1111]: 2026/09/21 14:31:59 INFO Signed narinfos id=57 count=16253lb # [ 155.827554] niks3-server[1111]: 2026/09/21 14:31:59 INFO Registered completed upload object_key=g5laypvfk2469p8binw3mksk2hv7zhnh.ls6254worker1 # [ 155.836449] nix-grpc-daemon[1917]: time=2026-09-21T14:31:59.441Z level=INFO msg="Upload complete. (288ms)"6255lb # [ 155.836875] niks3-server[1111]: 2026/09/21 14:31:59 INFO Received complete upload request method=POST path=/api/pending_closures/57/complete6256lb # [ 155.839850] niks3-server[1111]: 2026/09/21 14:31:59 INFO Completed upload id=576257lb # [ 155.845704] niks3-server[1111]: 2026/09/21 14:31:59 INFO Registered completed upload object_key=g5laypvfk2469p8binw3mksk2hv7zhnh.narinfo6258worker2 # [ 155.839784] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:59Z level=debug event=assigned drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv worker=192.168.1.3:50051 clients=16259worker1 # [ 155.849042] nix-grpc-daemon[1857]: ts=2026-09-21T14:31:59Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39236 duration_s=1 drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv assign_id=9 outputs=16260worker2 # [ 155.852622] nix-grpc-daemon[1314]: ts=2026-09-21T14:31:59Z level=debug event=want client=5 drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv cached=06261worker1 # [ 155.867898] nix-daemon[851]: reaped child process 2010, status = succeeded6262worker1 # [ 155.901132] nix-grpc-daemon[1857]: ts=2026-09-21T14:31:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:392366263worker1 # [ 155.915878] nix-grpc-daemon[1857]: ts=2026-09-21T14:31:59Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:392366264worker1 # [ 155.930843] nix-grpc-daemon[1917]: time=2026-09-21T14:31:59.535Z level=INFO msg="All 1 paths already cached"6265worker1 # [ 155.939741] nix-grpc-daemon[1857]: ts=2026-09-21T14:31:59Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:392366266worker1 # [ 155.953687] nix-daemon[851]: accepted connection from pid 1857, user nix-grpc-daemon (trusted)6267worker1 # [ 156.113604] nix-daemon[851]: accepted connection from pid 2035, user nix-grpc-daemon (trusted)6268worker1 # [ 156.140449] nix-daemon[851]: reaped child process 2038, status = succeeded6269worker1 # [ 156.230955] nix-daemon[851]: accepted connection from pid 2040, user nix-grpc-daemon (trusted)6270worker1 # [ 156.246452] nix-daemon[851]: reaped child process 2043, status = succeeded6271lb # [ 156.251933] niks3-server[1111]: 2026/09/21 14:31:59 INFO Received uploads request method=POST path=/api/pending_closures6272worker1 # [ 156.262342] nix-grpc-daemon[1917]: time=2026-09-21T14:31:59.866Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6273worker1 # [ 156.263811] nix-grpc-daemon[1917]: time=2026-09-21T14:31:59.866Z level=INFO msg="Uploading cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv (624B)"6274lb # [ 156.278831] niks3-server[1111]: 2026/09/21 14:31:59 INFO Registered completed upload object_key=nar/1w4sykcjji9zlw0a5xibxycy6x3jyb4drd29x2k9g7x5whfwwpsq.nar.zst6275lb # [ 156.292394] niks3-server[1111]: 2026/09/21 14:31:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/58/sign6276worker1 # [ 156.298758] nix-grpc-daemon[1917]: time=2026-09-21T14:31:59.903Z level=INFO msg="Uploading 1 narinfos"6277lb # [ 156.303246] niks3-server[1111]: 2026/09/21 14:31:59 INFO Signed narinfos id=58 count=16278lb # [ 156.310132] niks3-server[1111]: 2026/09/21 14:31:59 INFO Registered completed upload object_key=cnf7j5vjqyj799isbg26wgjnrlk208gi.ls6279lb # [ 156.322093] niks3-server[1111]: 2026/09/21 14:31:59 INFO Received complete upload request method=POST path=/api/pending_closures/58/complete6280worker1 # [ 156.327407] nix-grpc-daemon[1917]: time=2026-09-21T14:31:59.932Z level=INFO msg="Upload complete. (358ms)"6281lb # [ 156.331225] niks3-server[1111]: 2026/09/21 14:31:59 INFO Completed upload id=586282worker1 # [ 156.336258] nix-grpc-daemon[1857]: ts=2026-09-21T14:31:59Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39236 duration_s=0 paths=1 nar_bytes_in=6246283lb # [ 156.335219] niks3-server[1111]: 2026/09/21 14:31:59 INFO Registered completed upload object_key=cnf7j5vjqyj799isbg26wgjnrlk208gi.narinfo6284worker1 # [ 156.349301] nix-grpc-daemon[1857]: ts=2026-09-21T14:31:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:392366285worker1 # [ 156.359323] nix-daemon[851]: accepted connection from pid 1857, user nix-grpc-daemon (trusted)6286worker1 # [ 156.361860] nix-daemon[851]: reaped child process 2033, status = succeeded6287worker1 # [ 156.375811] nix-daemon[851]: accepted connection from pid 1857, user nix-grpc-daemon (trusted)6288worker1 # [ 157.848373] nix-daemon[851]: reaped child process 2047, status = succeeded6289worker1 # [ 157.944299] nix-daemon[851]: accepted connection from pid 2063, user nix-grpc-daemon (trusted)6290worker1 # [ 157.961812] nix-daemon[851]: reaped child process 2066, status = succeeded6291lb # [ 157.969154] niks3-server[1111]: 2026/09/21 14:32:01 INFO Received uploads request method=POST path=/api/pending_closures6292worker1 # [ 157.983191] nix-grpc-daemon[1917]: time=2026-09-21T14:32:01.585Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6293worker1 # [ 157.985156] nix-grpc-daemon[1917]: time=2026-09-21T14:32:01.585Z level=INFO msg="Uploading p1334cl91hzsgv8bmiqzkwvhbwq24vn1-farm-b (176B)"6294lb # [ 158.011268] niks3-server[1111]: 2026/09/21 14:32:01 INFO Registered completed upload object_key=nar/0pa7j07n0n5p9wcfihhx3357xrxs4h8s57kncplymh51lvzvz2kh.nar.zst6295worker1 # [ 158.022434] nix-grpc-daemon[1917]: time=2026-09-21T14:32:01.626Z level=INFO msg="Uploading 1 narinfos"6296lb # [ 158.023392] niks3-server[1111]: 2026/09/21 14:32:01 INFO Received sign narinfos request method=POST path=/api/pending_closures/59/sign6297lb # [ 158.034263] niks3-server[1111]: 2026/09/21 14:32:01 INFO Signed narinfos id=59 count=16298lb # [ 158.040407] niks3-server[1111]: 2026/09/21 14:32:01 INFO Registered completed upload object_key=log/cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv6299lb # [ 158.050363] niks3-server[1111]: 2026/09/21 14:32:01 INFO Registered completed upload object_key=p1334cl91hzsgv8bmiqzkwvhbwq24vn1.ls6300worker1 # [ 158.058239] nix-grpc-daemon[1917]: time=2026-09-21T14:32:01.662Z level=INFO msg="Upload complete. (271ms)"6301lb # [ 158.058836] niks3-server[1111]: 2026/09/21 14:32:01 INFO Received complete upload request method=POST path=/api/pending_closures/59/complete6302lb # [ 158.064389] niks3-server[1111]: 2026/09/21 14:32:01 INFO Completed upload id=596303lb # [ 158.067961] niks3-server[1111]: 2026/09/21 14:32:01 INFO Registered completed upload object_key=p1334cl91hzsgv8bmiqzkwvhbwq24vn1.narinfo6304worker2 # [ 158.059428] nix-grpc-daemon[1314]: ts=2026-09-21T14:32:01Z level=debug event=assigned drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv worker=192.168.1.3:50051 clients=16305worker1 # [ 158.069653] nix-grpc-daemon[1857]: ts=2026-09-21T14:32:01Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39236 duration_s=1 drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv assign_id=10 outputs=16306worker2 # [ 158.071723] nix-grpc-daemon[1314]: ts=2026-09-21T14:32:01Z level=debug event=want client=5 drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv cached=06307worker1 # [ 158.092950] nix-grpc-daemon[1857]: ts=2026-09-21T14:32:01Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:392366308worker1 # [ 158.103825] nix-grpc-daemon[1857]: ts=2026-09-21T14:32:01Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:392366309worker1 # [ 158.109931] nix-daemon[851]: reaped child process 2045, status = succeeded6310worker1 # [ 158.113366] nix-grpc-daemon[1917]: time=2026-09-21T14:32:01.717Z level=INFO msg="All 1 paths already cached"6311worker1 # [ 158.117544] nix-grpc-daemon[1857]: ts=2026-09-21T14:32:01Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:392366312worker1 # [ 158.131442] nix-daemon[851]: accepted connection from pid 1857, user nix-grpc-daemon (trusted)6313worker1 # [ 158.264476] nix-daemon[851]: accepted connection from pid 2070, user nix-grpc-daemon (trusted)6314worker1 # [ 158.287460] nix-daemon[851]: reaped child process 2073, status = succeeded6315worker1 # [ 158.384247] nix-daemon[851]: accepted connection from pid 2075, user nix-grpc-daemon (trusted)6316worker1 # [ 158.402885] nix-daemon[851]: reaped child process 2078, status = succeeded6317lb # [ 158.403457] niks3-server[1111]: 2026/09/21 14:32:02 INFO Received uploads request method=POST path=/api/pending_closures6318worker1 # [ 158.413719] nix-grpc-daemon[1917]: time=2026-09-21T14:32:02.018Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6319worker1 # [ 158.415156] nix-grpc-daemon[1917]: time=2026-09-21T14:32:02.018Z level=INFO msg="Uploading 7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv (624B)"6320lb # [ 158.433978] niks3-server[1111]: 2026/09/21 14:32:02 INFO Registered completed upload object_key=nar/0l0zss9ls4bhz18slrrmz7l1j93xz7424q7314p64rpgrpqf6b7k.nar.zst6321worker1 # [ 158.447786] nix-grpc-daemon[1917]: time=2026-09-21T14:32:02.052Z level=INFO msg="Uploading 1 narinfos"6322lb # [ 158.450790] niks3-server[1111]: 2026/09/21 14:32:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/60/sign6323lb # [ 158.459075] niks3-server[1111]: 2026/09/21 14:32:02 INFO Signed narinfos id=60 count=16324worker1 # [ 158.467642] nix-grpc-daemon[1917]: time=2026-09-21T14:32:02.072Z level=INFO msg="Upload complete. (327ms)"6325lb # [ 158.467290] niks3-server[1111]: 2026/09/21 14:32:02 INFO Registered completed upload object_key=7vjzkx8444zm2h7882vsj6bmj73zqjq3.ls6326worker1 # [ 158.475692] nix-grpc-daemon[1857]: ts=2026-09-21T14:32:02Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39236 duration_s=0 paths=1 nar_bytes_in=6246327lb # [ 158.476111] niks3-server[1111]: 2026/09/21 14:32:02 INFO Received complete upload request method=POST path=/api/pending_closures/60/complete6328lb # [ 158.479334] niks3-server[1111]: 2026/09/21 14:32:02 INFO Completed upload id=606329lb # [ 158.484238] niks3-server[1111]: 2026/09/21 14:32:02 INFO Registered completed upload object_key=7vjzkx8444zm2h7882vsj6bmj73zqjq3.narinfo6330worker1 # [ 158.486159] nix-grpc-daemon[1857]: ts=2026-09-21T14:32:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:392366331worker1 # [ 158.491773] nix-daemon[851]: accepted connection from pid 1857, user nix-grpc-daemon (trusted)6332worker1 # [ 158.497746] nix-daemon[851]: reaped child process 2068, status = succeeded6333worker1 # [ 158.508709] nix-daemon[851]: accepted connection from pid 1857, user nix-grpc-daemon (trusted)6334worker1 # [ 159.196630] nix-daemon[851]: reaped child process 2082, status = succeeded6335worker1 # [ 159.264230] nix-daemon[851]: accepted connection from pid 2098, user nix-grpc-daemon (trusted)6336worker1 # [ 159.283005] nix-daemon[851]: reaped child process 2101, status = succeeded6337lb # [ 159.287437] niks3-server[1111]: 2026/09/21 14:32:02 INFO Received uploads request method=POST path=/api/pending_closures6338worker1 # [ 159.300405] nix-grpc-daemon[1917]: time=2026-09-21T14:32:02.902Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6339worker1 # [ 159.301903] nix-grpc-daemon[1917]: time=2026-09-21T14:32:02.902Z level=INFO msg="Uploading nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8-farm-b (176B)"6340lb # [ 159.327367] niks3-server[1111]: 2026/09/21 14:32:02 INFO Registered completed upload object_key=nar/0kqa5m59sp6aj1ixqj6q54wwqxkcn98fw1rc9hfkcdk460wjdl0l.nar.zst6341lb # [ 159.340519] niks3-server[1111]: 2026/09/21 14:32:02 INFO Registered completed upload object_key=log/7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv6342worker1 # [ 159.350363] nix-grpc-daemon[1917]: time=2026-09-21T14:32:02.954Z level=INFO msg="Uploading 1 narinfos"6343lb # [ 159.352461] niks3-server[1111]: 2026/09/21 14:32:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/61/sign6344lb # [ 159.362910] niks3-server[1111]: 2026/09/21 14:32:02 INFO Signed narinfos id=61 count=16345lb # [ 159.369152] niks3-server[1111]: 2026/09/21 14:32:02 INFO Registered completed upload object_key=nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8.ls6346worker1 # [ 159.382222] nix-grpc-daemon[1917]: time=2026-09-21T14:32:02.986Z level=INFO msg="Upload complete. (264ms)"6347lb # [ 159.382317] niks3-server[1111]: 2026/09/21 14:32:02 INFO Received complete upload request method=POST path=/api/pending_closures/61/complete6348lb # [ 159.388252] niks3-server[1111]: 2026/09/21 14:32:02 INFO Completed upload id=616349worker2 # [ 159.383393] nix-grpc-daemon[1314]: ts=2026-09-21T14:32:02Z level=debug event=assigned drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv worker=192.168.1.3:50051 clients=16350lb # [ 159.394833] niks3-server[1111]: 2026/09/21 14:32:03 INFO Registered completed upload object_key=nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8.narinfo6351worker1 # [ 159.393683] nix-grpc-daemon[1857]: ts=2026-09-21T14:32:02Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39236 duration_s=0 drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv assign_id=11 outputs=16352worker2 # [ 159.404237] nix-grpc-daemon[1314]: ts=2026-09-21T14:32:03Z level=debug event=want client=5 drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv cached=06353worker1 # [ 159.416561] nix-grpc-daemon[1857]: ts=2026-09-21T14:32:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:392366354worker1 # [ 159.424672] nix-daemon[851]: reaped child process 2080, status = succeeded6355worker1 # [ 159.450315] nix-grpc-daemon[1857]: ts=2026-09-21T14:32:03Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:392366356worker1 # [ 159.463969] nix-grpc-daemon[1917]: time=2026-09-21T14:32:03.068Z level=INFO msg="All 1 paths already cached"6357worker1 # [ 159.470103] nix-grpc-daemon[1857]: ts=2026-09-21T14:32:03Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:392366358worker1 # [ 159.483948] nix-daemon[851]: accepted connection from pid 1857, user nix-grpc-daemon (trusted)6359worker1 # [ 159.639974] nix-daemon[851]: accepted connection from pid 2105, user nix-grpc-daemon (trusted)6360worker1 # [ 159.659511] nix-daemon[851]: reaped child process 2108, status = succeeded6361worker1 # [ 159.760973] nix-daemon[851]: accepted connection from pid 2110, user nix-grpc-daemon (trusted)6362worker1 # [ 159.780914] nix-daemon[851]: reaped child process 2113, status = succeeded6363lb # [ 159.778354] niks3-server[1111]: 2026/09/21 14:32:03 INFO Received uploads request method=POST path=/api/pending_closures6364worker1 # [ 159.787836] nix-grpc-daemon[1917]: time=2026-09-21T14:32:03.392Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6365worker1 # [ 159.789492] nix-grpc-daemon[1917]: time=2026-09-21T14:32:03.392Z level=INFO msg="Uploading cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv (624B)"6366lb # [ 159.807841] niks3-server[1111]: 2026/09/21 14:32:03 INFO Registered completed upload object_key=nar/0f6g0vggy0qbqsj64l1nivyqdh6q12kh8620ysnf210ani0727vf.nar.zst6367worker1 # [ 159.820982] nix-grpc-daemon[1917]: time=2026-09-21T14:32:03.425Z level=INFO msg="Uploading 1 narinfos"6368lb # [ 159.821214] niks3-server[1111]: 2026/09/21 14:32:03 INFO Received sign narinfos request method=POST path=/api/pending_closures/62/sign6369lb # [ 159.832914] niks3-server[1111]: 2026/09/21 14:32:03 INFO Signed narinfos id=62 count=16370lb # [ 159.838840] niks3-server[1111]: 2026/09/21 14:32:03 INFO Registered completed upload object_key=cgwyij19i3fl7sqlyz54vzfr4zajg077.ls6371lb # [ 159.848823] niks3-server[1111]: 2026/09/21 14:32:03 INFO Received complete upload request method=POST path=/api/pending_closures/62/complete6372lb # [ 159.855245] niks3-server[1111]: 2026/09/21 14:32:03 INFO Completed upload id=626373worker1 # [ 159.855269] nix-grpc-daemon[1917]: time=2026-09-21T14:32:03.459Z level=INFO msg="Upload complete. (355ms)"6374lb # [ 159.857691] niks3-server[1111]: 2026/09/21 14:32:03 INFO Registered completed upload object_key=cgwyij19i3fl7sqlyz54vzfr4zajg077.narinfo6375worker1 # [ 159.864707] nix-grpc-daemon[1857]: ts=2026-09-21T14:32:03Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39236 duration_s=0 paths=1 nar_bytes_in=6246376worker1 # [ 159.876148] nix-grpc-daemon[1857]: ts=2026-09-21T14:32:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:392366377worker1 # [ 159.882486] nix-daemon[851]: accepted connection from pid 1857, user nix-grpc-daemon (trusted)6378worker1 # [ 159.887031] nix-daemon[851]: reaped child process 2103, status = succeeded6379worker1 # [ 159.899678] nix-daemon[851]: accepted connection from pid 1857, user nix-grpc-daemon (trusted)6380worker1 # [ 161.204672] nix-daemon[851]: reaped child process 2117, status = succeeded6381worker1 # [ 161.273742] nix-daemon[851]: accepted connection from pid 2133, user nix-grpc-daemon (trusted)6382worker1 # [ 161.296616] nix-daemon[851]: reaped child process 2136, status = succeeded6383lb # [ 161.298184] niks3-server[1111]: 2026/09/21 14:32:04 INFO Received uploads request method=POST path=/api/pending_closures6384worker1 # [ 161.311632] nix-grpc-daemon[1917]: time=2026-09-21T14:32:04.913Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6385worker1 # [ 161.313377] nix-grpc-daemon[1917]: time=2026-09-21T14:32:04.913Z level=INFO msg="Uploading hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5-farm-b (176B)"6386lb # [ 161.338420] niks3-server[1111]: 2026/09/21 14:32:04 INFO Registered completed upload object_key=nar/0x964sxcxsdh3nfjarc1ib52kr50kl8zxl7h34jq5ydx763drras.nar.zst6387lb # [ 161.350963] niks3-server[1111]: 2026/09/21 14:32:04 INFO Registered completed upload object_key=log/cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv6388worker1 # [ 161.361644] nix-grpc-daemon[1917]: time=2026-09-21T14:32:04.966Z level=INFO msg="Uploading 1 narinfos"6389lb # [ 161.360850] niks3-server[1111]: 2026/09/21 14:32:04 INFO Received sign narinfos request method=POST path=/api/pending_closures/63/sign6390lb # [ 161.370546] niks3-server[1111]: 2026/09/21 14:32:04 INFO Signed narinfos id=63 count=16391lb # [ 161.375373] niks3-server[1111]: 2026/09/21 14:32:04 INFO Registered completed upload object_key=hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5.ls6392lb # [ 161.384286] niks3-server[1111]: 2026/09/21 14:32:05 INFO Received complete upload request method=POST path=/api/pending_closures/63/complete6393worker1 # [ 161.393287] nix-grpc-daemon[1917]: time=2026-09-21T14:32:04.997Z level=INFO msg="Upload complete. (281ms)"6394lb # [ 161.395303] niks3-server[1111]: 2026/09/21 14:32:05 INFO Completed upload id=636395lb # [ 161.399029] niks3-server[1111]: 2026/09/21 14:32:05 INFO Registered completed upload object_key=hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5.narinfo6396worker2 # [ 161.394715] nix-grpc-daemon[1314]: ts=2026-09-21T14:32:05Z level=debug event=assigned drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv worker=192.168.1.3:50051 clients=16397worker2 # [ 161.407491] nix-grpc-daemon[1314]: ts=2026-09-21T14:32:05Z level=debug event=want client=5 drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv cached=06398worker1 # [ 161.405202] nix-grpc-daemon[1857]: ts=2026-09-21T14:32:05Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39236 duration_s=1 drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv assign_id=12 outputs=16399worker1 # [ 161.422638] nix-grpc-daemon[1857]: ts=2026-09-21T14:32:05Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:392366400worker1 # [ 161.430162] nix-daemon[851]: reaped child process 2115, status = succeeded6401worker1 # [ 161.438047] nix-grpc-daemon[1857]: ts=2026-09-21T14:32:05Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:392366402worker1 # [ 161.450705] nix-grpc-daemon[1917]: time=2026-09-21T14:32:05.055Z level=INFO msg="All 2 paths already cached"6403worker1 # [ 161.458653] nix-grpc-daemon[1857]: ts=2026-09-21T14:32:05Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:392366404worker1 # [ 161.470576] nix-daemon[851]: accepted connection from pid 1857, user nix-grpc-daemon (trusted)6405worker1 # [ 161.619361] nix-daemon[851]: accepted connection from pid 2140, user nix-grpc-daemon (trusted)6406worker1 # [ 161.638449] nix-daemon[851]: reaped child process 2143, status = succeeded6407worker1 # [ 161.737208] nix-daemon[851]: accepted connection from pid 2145, user nix-grpc-daemon (trusted)6408worker1 # [ 161.755733] nix-daemon[851]: reaped child process 2148, status = succeeded6409lb # [ 161.753752] niks3-server[1111]: 2026/09/21 14:32:05 INFO Received uploads request method=POST path=/api/pending_closures6410worker1 # [ 161.763019] nix-grpc-daemon[1917]: time=2026-09-21T14:32:05.367Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6411worker1 # [ 161.764670] nix-grpc-daemon[1917]: time=2026-09-21T14:32:05.367Z level=INFO msg="Uploading dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv (752B)"6412lb # [ 161.780199] niks3-server[1111]: 2026/09/21 14:32:05 INFO Registered completed upload object_key=nar/0s7ff7rwqx5p9mdzddqgzd9bbd5j8njfc62k07mgnpy19810bs86.nar.zst6413worker1 # [ 161.794330] nix-grpc-daemon[1917]: time=2026-09-21T14:32:05.398Z level=INFO msg="Uploading 1 narinfos"6414lb # [ 161.796139] niks3-server[1111]: 2026/09/21 14:32:05 INFO Received sign narinfos request method=POST path=/api/pending_closures/64/sign6415lb # [ 161.807643] niks3-server[1111]: 2026/09/21 14:32:05 INFO Signed narinfos id=64 count=16416lb # [ 161.815094] niks3-server[1111]: 2026/09/21 14:32:05 INFO Registered completed upload object_key=dca2k4dvnk52m7gijqfq8q99ih3ni8ny.ls6417worker1 # [ 161.826877] nix-grpc-daemon[1917]: time=2026-09-21T14:32:05.431Z level=INFO msg="Upload complete. (343ms)"6418lb # [ 161.826828] niks3-server[1111]: 2026/09/21 14:32:05 INFO Received complete upload request method=POST path=/api/pending_closures/64/complete6419lb # [ 161.832374] niks3-server[1111]: 2026/09/21 14:32:05 INFO Completed upload id=646420worker1 # [ 161.833188] nix-grpc-daemon[1857]: ts=2026-09-21T14:32:05Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39236 duration_s=0 paths=1 nar_bytes_in=7526421lb # [ 161.835601] niks3-server[1111]: 2026/09/21 14:32:05 INFO Registered completed upload object_key=dca2k4dvnk52m7gijqfq8q99ih3ni8ny.narinfo6422worker1 # [ 161.845184] nix-grpc-daemon[1857]: ts=2026-09-21T14:32:05Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:392366423worker1 # [ 161.853586] nix-daemon[851]: accepted connection from pid 1857, user nix-grpc-daemon (trusted)6424worker1 # [ 161.856429] nix-daemon[851]: reaped child process 2138, status = succeeded6425worker1 # [ 161.866959] nix-daemon[851]: accepted connection from pid 1857, user nix-grpc-daemon (trusted)6426worker1 # [ 163.373655] nix-daemon[851]: reaped child process 2152, status = succeeded6427worker1 # [ 163.441687] nix-daemon[851]: accepted connection from pid 2168, user nix-grpc-daemon (trusted)6428worker1 # [ 163.458285] nix-daemon[851]: reaped child process 2171, status = succeeded6429lb # [ 163.465997] niks3-server[1111]: 2026/09/21 14:32:07 INFO Received uploads request method=POST path=/api/pending_closures6430worker1 # [ 163.476483] nix-grpc-daemon[1917]: time=2026-09-21T14:32:07.080Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6431worker1 # [ 163.478429] nix-grpc-daemon[1917]: time=2026-09-21T14:32:07.080Z level=INFO msg="Uploading d88xgckxx0s0n5ly2d0jw1qy1641z8pq-farm-top (232B)"6432lb # [ 163.508802] niks3-server[1111]: 2026/09/21 14:32:07 INFO Registered completed upload object_key=nar/0knxhkhz3x04vk8as5wv5ar2d0h02prv7q5f42qm1xjgr70ancd0.nar.zst6433worker1 # [ 163.520714] nix-grpc-daemon[1917]: time=2026-09-21T14:32:07.125Z level=INFO msg="Uploading 1 narinfos"6434lb # [ 163.521784] niks3-server[1111]: 2026/09/21 14:32:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/65/sign6435lb # [ 163.533378] niks3-server[1111]: 2026/09/21 14:32:07 INFO Signed narinfos id=65 count=16436lb # [ 163.540138] niks3-server[1111]: 2026/09/21 14:32:07 INFO Registered completed upload object_key=log/dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv6437lb # [ 163.552528] niks3-server[1111]: 2026/09/21 14:32:07 INFO Registered completed upload object_key=d88xgckxx0s0n5ly2d0jw1qy1641z8pq.ls6438worker1 # [ 163.557676] nix-grpc-daemon[1917]: time=2026-09-21T14:32:07.162Z level=INFO msg="Upload complete. (278ms)"6439lb # [ 163.559714] niks3-server[1111]: 2026/09/21 14:32:07 INFO Received complete upload request method=POST path=/api/pending_closures/65/complete6440lb # [ 163.565384] niks3-server[1111]: 2026/09/21 14:32:07 INFO Completed upload id=656441lb # [ 163.567845] niks3-server[1111]: 2026/09/21 14:32:07 INFO Registered completed upload object_key=d88xgckxx0s0n5ly2d0jw1qy1641z8pq.narinfo6442worker2 # [ 163.560739] nix-grpc-daemon[1314]: ts=2026-09-21T14:32:07Z level=debug event=assigned drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv worker=192.168.1.3:50051 clients=16443worker1 # [ 163.569107] nix-grpc-daemon[1857]: ts=2026-09-21T14:32:07Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39236 duration_s=1 drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv assign_id=13 outputs=16444worker1 # [ 163.589078] nix-grpc-daemon[1857]: ts=2026-09-21T14:32:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:392366445worker1 # [ 163.599929] nix-grpc-daemon[1857]: ts=2026-09-21T14:32:07Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:392366446worker1 # [ 163.605009] nix-daemon[851]: reaped child process 2150, status = succeeded6447worker1 # [ 163.611700] nix-grpc-daemon[1917]: time=2026-09-21T14:32:07.216Z level=INFO msg="All 2 paths already cached"6448worker1 # [ 163.620186] nix-grpc-daemon[1857]: ts=2026-09-21T14:32:07Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:392366449worker1 # [ 163.633470] nix-daemon[851]: accepted connection from pid 1857, user nix-grpc-daemon (trusted)6450worker1 # [ 163.774651] nix-daemon[851]: accepted connection from pid 2175, user nix-grpc-daemon (trusted)6451worker1 # [ 163.792765] nix-daemon[851]: reaped child process 2178, status = succeeded6452worker1 # [ 163.893375] nix-daemon[851]: accepted connection from pid 2180, user nix-grpc-daemon (trusted)6453worker1 # [ 163.909571] nix-daemon[851]: reaped child process 2183, status = succeeded6454lb # [ 163.915362] niks3-server[1111]: 2026/09/21 14:32:07 INFO Received uploads request method=POST path=/api/pending_closures6455worker1 # [ 163.924561] nix-grpc-daemon[1917]: time=2026-09-21T14:32:07.529Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6456worker1 # [ 163.926052] nix-grpc-daemon[1917]: time=2026-09-21T14:32:07.529Z level=INFO msg="Uploading 2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv (752B)"6457lb # [ 163.941607] niks3-server[1111]: 2026/09/21 14:32:07 INFO Registered completed upload object_key=nar/0cpiyiv7frlhngqhhc703bglz5q2cpg4n8sgds4sfg7nvw0fcdn5.nar.zst6458worker1 # [ 163.958558] nix-grpc-daemon[1917]: time=2026-09-21T14:32:07.563Z level=INFO msg="Uploading 1 narinfos"6459lb # [ 163.958427] niks3-server[1111]: 2026/09/21 14:32:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/66/sign6460lb # [ 163.968637] niks3-server[1111]: 2026/09/21 14:32:07 INFO Signed narinfos id=66 count=16461lb # [ 163.971374] niks3-server[1111]: 2026/09/21 14:32:07 INFO Registered completed upload object_key=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9.ls6462worker1 # [ 163.987495] nix-grpc-daemon[1917]: time=2026-09-21T14:32:07.592Z level=INFO msg="Upload complete. (342ms)"6463lb # [ 163.987686] niks3-server[1111]: 2026/09/21 14:32:07 INFO Received complete upload request method=POST path=/api/pending_closures/66/complete6464lb # [ 163.992254] niks3-server[1111]: 2026/09/21 14:32:07 INFO Completed upload id=666465lb # [ 163.995120] niks3-server[1111]: 2026/09/21 14:32:07 INFO Registered completed upload object_key=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9.narinfo6466worker1 # [ 163.994484] nix-grpc-daemon[1857]: ts=2026-09-21T14:32:07Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39236 duration_s=0 paths=1 nar_bytes_in=7526467worker1 # [ 164.004732] nix-grpc-daemon[1857]: ts=2026-09-21T14:32:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:392366468worker1 # [ 164.015925] nix-daemon[851]: accepted connection from pid 1857, user nix-grpc-daemon (trusted)6469worker1 # [ 164.023189] nix-daemon[851]: reaped child process 2173, status = succeeded6470worker1 # [ 164.030119] nix-daemon[851]: accepted connection from pid 1857, user nix-grpc-daemon (trusted)6471worker1 # [ 165.512417] nix-daemon[851]: reaped child process 2187, status = succeeded6472worker1 # [ 165.610372] nix-daemon[851]: accepted connection from pid 2203, user nix-grpc-daemon (trusted)6473worker1 # [ 165.631585] nix-daemon[851]: reaped child process 2206, status = succeeded6474lb # [ 165.635662] niks3-server[1111]: 2026/09/21 14:32:09 INFO Received uploads request method=POST path=/api/pending_closures6475worker1 # [ 165.647898] nix-grpc-daemon[1917]: time=2026-09-21T14:32:09.250Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6476worker1 # [ 165.649847] nix-grpc-daemon[1917]: time=2026-09-21T14:32:09.250Z level=INFO msg="Uploading vz84kyfcw804d0iyx7nj61j47ms9zxmn-farm-top (232B)"6477lb # [ 165.670358] niks3-server[1111]: 2026/09/21 14:32:09 INFO Registered completed upload object_key=log/2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv6478lb # [ 165.684673] niks3-server[1111]: 2026/09/21 14:32:09 INFO Registered completed upload object_key=nar/0aaz4iim7fa45zws98gbhp73l23kbm0fvp4anssnxa96v666ljhs.nar.zst6479worker1 # [ 165.700357] nix-grpc-daemon[1917]: time=2026-09-21T14:32:09.304Z level=INFO msg="Uploading 1 narinfos"6480lb # [ 165.699348] niks3-server[1111]: 2026/09/21 14:32:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/67/sign6481lb # [ 165.707657] niks3-server[1111]: 2026/09/21 14:32:09 INFO Signed narinfos id=67 count=16482lb # [ 165.715911] niks3-server[1111]: 2026/09/21 14:32:09 INFO Registered completed upload object_key=vz84kyfcw804d0iyx7nj61j47ms9zxmn.ls6483lb # [ 165.726909] niks3-server[1111]: 2026/09/21 14:32:09 INFO Received complete upload request method=POST path=/api/pending_closures/67/complete6484lb # [ 165.734081] niks3-server[1111]: 2026/09/21 14:32:09 INFO Completed upload id=676485worker1 # [ 165.732766] nix-grpc-daemon[1917]: time=2026-09-21T14:32:09.337Z level=INFO msg="Upload complete. (293ms)"6486lb # [ 165.736451] niks3-server[1111]: 2026/09/21 14:32:09 INFO Registered completed upload object_key=vz84kyfcw804d0iyx7nj61j47ms9zxmn.narinfo6487worker2 # [ 165.738126] nix-grpc-daemon[1314]: ts=2026-09-21T14:32:09Z level=debug event=assigned drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv worker=192.168.1.3:50051 clients=16488worker1 # [ 165.746213] nix-grpc-daemon[1857]: ts=2026-09-21T14:32:09Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39236 duration_s=1 drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv assign_id=14 outputs=16489worker1 # [ 165.766797] nix-grpc-daemon[1857]: ts=2026-09-21T14:32:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:392366490worker1 # [ 165.773785] nix-daemon[851]: reaped child process 2185, status = succeeded6491worker1 # [ 165.781794] nix-grpc-daemon[1857]: ts=2026-09-21T14:32:09Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:392366492worker1 # [ 165.794677] nix-grpc-daemon[1917]: time=2026-09-21T14:32:09.399Z level=INFO msg="All 2 paths already cached"6493worker1 # [ 165.802585] nix-grpc-daemon[1857]: ts=2026-09-21T14:32:09Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:392366494worker1 # [ 165.814949] nix-daemon[851]: accepted connection from pid 1857, user nix-grpc-daemon (trusted)6495worker1 # [ 165.963961] nix-daemon[851]: accepted connection from pid 2210, user nix-grpc-daemon (trusted)6496worker1 # [ 165.981451] nix-daemon[851]: reaped child process 2213, status = succeeded6497worker1 # [ 166.081795] nix-daemon[851]: accepted connection from pid 2215, user nix-grpc-daemon (trusted)6498worker1 # [ 166.100313] nix-daemon[851]: reaped child process 2218, status = succeeded6499lb # [ 166.098435] niks3-server[1111]: 2026/09/21 14:32:09 INFO Received uploads request method=POST path=/api/pending_closures6500worker1 # [ 166.109956] nix-grpc-daemon[1917]: time=2026-09-21T14:32:09.714Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6501worker1 # [ 166.111418] nix-grpc-daemon[1917]: time=2026-09-21T14:32:09.714Z level=INFO msg="Uploading qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv (752B)"6502lb # [ 166.126811] niks3-server[1111]: 2026/09/21 14:32:09 INFO Registered completed upload object_key=nar/05zmsnqx4aj55yncx917mz6d3hzrvz2rx2x75313vvy1kq1zv9fv.nar.zst6503worker1 # [ 166.141215] nix-grpc-daemon[1917]: time=2026-09-21T14:32:09.745Z level=INFO msg="Uploading 1 narinfos"6504lb # [ 166.143209] niks3-server[1111]: 2026/09/21 14:32:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/68/sign6505lb # [ 166.154227] niks3-server[1111]: 2026/09/21 14:32:09 INFO Signed narinfos id=68 count=16506lb # [ 166.160296] niks3-server[1111]: 2026/09/21 14:32:09 INFO Registered completed upload object_key=qx10pa00pilyp5xdzd48l88d5jz143mr.ls6507lb # [ 166.169310] niks3-server[1111]: 2026/09/21 14:32:09 INFO Received complete upload request method=POST path=/api/pending_closures/68/complete6508worker1 # [ 166.175832] nix-grpc-daemon[1917]: time=2026-09-21T14:32:09.780Z level=INFO msg="Upload complete. (347ms)"6509lb # [ 166.176179] niks3-server[1111]: 2026/09/21 14:32:09 INFO Completed upload id=686510lb # [ 166.180811] niks3-server[1111]: 2026/09/21 14:32:09 INFO Registered completed upload object_key=qx10pa00pilyp5xdzd48l88d5jz143mr.narinfo6511worker1 # [ 166.184228] nix-grpc-daemon[1857]: ts=2026-09-21T14:32:09Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39236 duration_s=0 paths=1 nar_bytes_in=7526512worker1 # [ 166.192371] nix-grpc-daemon[1857]: ts=2026-09-21T14:32:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:392366513worker1 # [ 166.205392] nix-daemon[851]: accepted connection from pid 1857, user nix-grpc-daemon (trusted)6514worker1 # [ 166.208509] nix-daemon[851]: reaped child process 2208, status = succeeded6515worker1 # [ 166.221501] nix-daemon[851]: accepted connection from pid 1857, user nix-grpc-daemon (trusted)6516worker1 # [ 167.264911] nix-daemon[851]: reaped child process 2222, status = succeeded6517worker1 # [ 167.372832] nix-daemon[851]: accepted connection from pid 2238, user nix-grpc-daemon (trusted)6518worker1 # [ 167.393726] nix-daemon[851]: reaped child process 2241, status = succeeded6519lb # [ 167.394092] niks3-server[1111]: 2026/09/21 14:32:11 INFO Received uploads request method=POST path=/api/pending_closures6520worker1 # [ 167.404501] nix-grpc-daemon[1917]: time=2026-09-21T14:32:11.007Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6521worker1 # [ 167.406620] nix-grpc-daemon[1917]: time=2026-09-21T14:32:11.007Z level=INFO msg="Uploading iagwbb072hnm79qb9wsrjp5907zc7aa2-farm-top (232B)"6522lb # [ 167.429125] niks3-server[1111]: 2026/09/21 14:32:11 INFO Registered completed upload object_key=nar/053jd22j3kv7v6j0q7cspdpc0h1c1xmywmajl6sd5hzwg50z2yj1.nar.zst6523worker1 # [ 167.447789] nix-grpc-daemon[1917]: time=2026-09-21T14:32:11.052Z level=INFO msg="Uploading 1 narinfos"6524lb # [ 167.447510] niks3-server[1111]: 2026/09/21 14:32:11 INFO Received sign narinfos request method=POST path=/api/pending_closures/69/sign6525lb # [ 167.457726] niks3-server[1111]: 2026/09/21 14:32:11 INFO Signed narinfos id=69 count=16526lb # [ 167.463627] niks3-server[1111]: 2026/09/21 14:32:11 INFO Registered completed upload object_key=log/qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv6527lb # [ 167.474182] niks3-server[1111]: 2026/09/21 14:32:11 INFO Registered completed upload object_key=iagwbb072hnm79qb9wsrjp5907zc7aa2.ls6528worker1 # [ 167.480702] nix-grpc-daemon[1917]: time=2026-09-21T14:32:11.085Z level=INFO msg="Upload complete. (250ms)"6529lb # [ 167.481218] niks3-server[1111]: 2026/09/21 14:32:11 INFO Received complete upload request method=POST path=/api/pending_closures/69/complete6530lb # [ 167.486225] niks3-server[1111]: 2026/09/21 14:32:11 INFO Completed upload id=696531lb # [ 167.488961] niks3-server[1111]: 2026/09/21 14:32:11 INFO Registered completed upload object_key=iagwbb072hnm79qb9wsrjp5907zc7aa2.narinfo6532worker1 # [ 167.492930] nix-grpc-daemon[1857]: ts=2026-09-21T14:32:11Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39236 duration_s=1 drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv assign_id=15 outputs=16533worker1 # [ 167.507432] nix-daemon[851]: reaped child process 2220, status = succeeded6534client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&1, in 18.18 seconds)6535worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6536worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)6537client: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag f5 --arg features '["gpu"]'6538client # [ 167.693315] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build -L --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&restart-grace=30" --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag f5 --arg features "[\"gpu\"]".6539client # Running as unit: gpu.service; invocation ID: 225be635f1bb46ab8675130593aebe876540client: (finished: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag f5 --arg features '["gpu"]', in 0.09 seconds)6541lb: must succeed: curl -sf localhost:9901/clusters6542lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)6543lb: must succeed: curl -sf localhost:9901/clusters6544lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)6545worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6546client # [ 167.903644] nix[1350]: don't know how to build these paths:6547client # [ 167.905687] nix[1350]: /nix/store/3i1nziwpdqq19lzpqy4rywzxnm8dlps6-farm-top.drv6548worker2 # [ 167.892910] nix-grpc-daemon[1314]: ts=2026-09-21T14:32:11Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:578026549worker2 # [ 167.898340] nix-grpc-daemon[1314]: ts=2026-09-21T14:32:11Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:578026550worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)6551client # [ 167.933694] nix[1350]: warning: 1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv: no connected, non-draining worker for system 'aarch64-linux' offers features {gpu}, waiting up to 30s for a worker6552worker2 # [ 167.921643] nix-grpc-daemon[1314]: ts=2026-09-21T14:32:11Z level=debug event=want client=6 drv=1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv cached=06553lb: must succeed: curl -sf localhost:9901/clusters6554lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)6555lb: must succeed: curl -sf localhost:9901/clusters6556lb # [ 169.024863] envoy[909]: conn peer=[2001:db8:1::1]:45880 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=27075 tx=11467 ms=179046557lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6558worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6559worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)6560client: waiting for success: systemctl show -p Result gpu | grep -q exit-code6561client # [ 197.935346] nix[1350]: 1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv: no connected, non-draining worker for system 'aarch64-linux' offers features {gpu}6562client # [ 197.945633] nix[1350]: error: dependency '1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv' failed6563client # [ 197.986118] systemd[1]: gpu.service: Main process exited, code=exited, status=1/FAILURE6564client # [ 197.993870] systemd[1]: gpu.service: Failed with result 'exit-code'.6565client # [ 197.997437] systemd[1]: gpu.service: Consumed 197ms CPU time over 30.299s wall clock time, 35.4M memory peak, 5.1K incoming IP traffic, 5K outgoing IP traffic.6566client: (finished: waiting for success: systemctl show -p Result gpu | grep -q exit-code, in 29.08 seconds)6567client: must succeed: journalctl -u gpu -o cat6568client: (finished: must succeed: journalctl -u gpu -o cat, in 0.03 seconds)6569lb: must succeed: curl -sf localhost:9901/clusters6570lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)6571lb: must succeed: curl -sf localhost:9901/clusters6572lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6573worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6574worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6575(finished: subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does, in 49.02 seconds)6576subtest: low disk drains a worker and builds go to the other6577worker1: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill6578lb # [ 199.025929] envoy[909]: conn peer=[2001:db8:1::1]:33390 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=897 tx=499 ms=300616579worker1: (finished: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill, in 1.21 seconds)6580worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free'6581worker1 # [ 200.114022] nix-grpc-daemon[1857]: ts=2026-09-21T14:32:43Z level=info event=unhealthy reason=min_free6582worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free', in 1.10 seconds)6583client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag drain >&26584client # don't know how to build these paths:6585client # /nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv6586worker1 # [ 200.926198] nix-grpc-daemon[1857]: ts=2026-09-21T14:32:44Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:392366587worker2 # [ 200.927859] nix-grpc-daemon[1314]: ts=2026-09-21T14:32:44Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:578026588worker2 # [ 200.952828] nix-grpc-daemon[1314]: ts=2026-09-21T14:32:44Z level=debug event=want client=7 drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv cached=06589worker2 # [ 200.959399] nix-grpc-daemon[1314]: ts=2026-09-21T14:32:44Z level=debug event=assigned drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv worker=192.168.1.4:50051 clients=16590worker2 # [ 200.998788] nix-grpc-daemon[1314]: ts=2026-09-21T14:32:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:578026591worker2 # [ 201.013766] nix-grpc-daemon[1314]: ts=2026-09-21T14:32:44Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:578026592client # copying 1 paths to 'lb:50051'...6593client # copying path '/nix/store/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv' to 'lb:50051'...6594worker2 # [ 201.029261] nix-grpc-daemon[1314]: ts=2026-09-21T14:32:44Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:578026595worker2 # [ 201.044167] nix-daemon[838]: accepted connection from pid 1314, user nix-grpc-daemon (trusted)6596worker2 # [ 201.203311] nix-daemon[838]: accepted connection from pid 1628, user nix-grpc-daemon (trusted)6597worker2 # [ 201.223188] nix-daemon[838]: reaped child process 1631, status = succeeded6598worker2 # [ 201.328307] nix-daemon[838]: accepted connection from pid 1633, user nix-grpc-daemon (trusted)6599worker2 # [ 201.346187] nix-daemon[838]: reaped child process 1636, status = succeeded6600lb # [ 201.358092] niks3-server[1111]: 2026/09/21 14:32:44 INFO Received uploads request method=POST path=/api/pending_closures6601worker2 # [ 201.361368] nix-grpc-daemon[1410]: time=2026-09-21T14:32:44.976Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6602worker2 # [ 201.362856] nix-grpc-daemon[1410]: time=2026-09-21T14:32:44.976Z level=INFO msg="Uploading kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv (512B)"6603lb # [ 201.389631] niks3-server[1111]: 2026/09/21 14:32:45 INFO Registered completed upload object_key=nar/1sfy0gcq4znfcj1f68c749r5x2jwmwsfbfp6xdflz5xkndjwjjm3.nar.zst6604lb # [ 201.402235] niks3-server[1111]: 2026/09/21 14:32:45 INFO Received sign narinfos request method=POST path=/api/pending_closures/70/sign6605worker2 # [ 201.402385] nix-grpc-daemon[1410]: time=2026-09-21T14:32:45.017Z level=INFO msg="Uploading 1 narinfos"6606lb # [ 201.414065] niks3-server[1111]: 2026/09/21 14:32:45 INFO Signed narinfos id=70 count=16607lb # [ 201.424412] niks3-server[1111]: 2026/09/21 14:32:45 INFO Registered completed upload object_key=kksl1jqfg56vabgqvrkz5vki6nq7r59r.ls6608lb # [ 201.441350] niks3-server[1111]: 2026/09/21 14:32:45 INFO Received complete upload request method=POST path=/api/pending_closures/70/complete6609worker2 # [ 201.439370] nix-grpc-daemon[1410]: time=2026-09-21T14:32:45.054Z level=INFO msg="Upload complete. (378ms)"6610lb # [ 201.448264] niks3-server[1111]: 2026/09/21 14:32:45 INFO Completed upload id=706611worker2 # [ 201.448277] nix-grpc-daemon[1314]: ts=2026-09-21T14:32:45Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:57802 duration_s=0 paths=1 nar_bytes_in=5126612lb # [ 201.462279] niks3-server[1111]: 2026/09/21 14:32:45 INFO Registered completed upload object_key=kksl1jqfg56vabgqvrkz5vki6nq7r59r.narinfo6613worker2 # [ 201.461599] nix-grpc-daemon[1314]: ts=2026-09-21T14:32:45Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:578026614worker2 # [ 201.468202] nix-daemon[838]: accepted connection from pid 1314, user nix-grpc-daemon (trusted)6615worker2 # [ 201.473036] nix-daemon[838]: reaped child process 1626, status = succeeded6616client # building '/nix/store/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv' on lb:50051...6617worker2 # [ 201.485843] nix-daemon[838]: accepted connection from pid 1314, user nix-grpc-daemon (trusted)6618client # farm-a> worker2: building kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv6619client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds6620client # farm-a> LOG-farm-a-drain6621worker2 # [ 202.569532] nix-daemon[838]: reaped child process 1640, status = succeeded6622worker2 # [ 202.644495] nix-daemon[838]: accepted connection from pid 1656, user nix-grpc-daemon (trusted)6623worker2 # [ 202.667297] nix-daemon[838]: reaped child process 1659, status = succeeded6624lb # [ 202.678319] niks3-server[1111]: 2026/09/21 14:32:46 INFO Received uploads request method=POST path=/api/pending_closures6625worker2 # [ 202.684433] nix-grpc-daemon[1410]: time=2026-09-21T14:32:46.297Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6626worker2 # [ 202.685952] nix-grpc-daemon[1410]: time=2026-09-21T14:32:46.297Z level=INFO msg="Uploading 272b39wn7cbnhkcc2rrz2f6kmhhmcwnr-farm-a (128B)"6627lb # [ 202.722137] niks3-server[1111]: 2026/09/21 14:32:46 INFO Registered completed upload object_key=log/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv6628lb # [ 202.732706] niks3-server[1111]: 2026/09/21 14:32:46 INFO Registered completed upload object_key=nar/1bf9z7y0ikx07b4zjag42zyrv2pq1ycw4461s579kfpmni9kz4dw.nar.zst6629lb # [ 202.745053] niks3-server[1111]: 2026/09/21 14:32:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/71/sign6630worker2 # [ 202.743354] nix-grpc-daemon[1410]: time=2026-09-21T14:32:46.358Z level=INFO msg="Uploading 1 narinfos"6631lb # [ 202.752601] niks3-server[1111]: 2026/09/21 14:32:46 INFO Signed narinfos id=71 count=16632lb # [ 202.754585] niks3-server[1111]: 2026/09/21 14:32:46 INFO Registered completed upload object_key=272b39wn7cbnhkcc2rrz2f6kmhhmcwnr.ls6633lb # [ 202.764391] niks3-server[1111]: 2026/09/21 14:32:46 INFO Received complete upload request method=POST path=/api/pending_closures/71/complete6634lb # [ 202.770754] niks3-server[1111]: 2026/09/21 14:32:46 INFO Completed upload id=716635worker2 # [ 202.763292] nix-grpc-daemon[1410]: time=2026-09-21T14:32:46.378Z level=INFO msg="Upload complete. (268ms)"6636lb # [ 202.773982] niks3-server[1111]: 2026/09/21 14:32:46 INFO Registered completed upload object_key=272b39wn7cbnhkcc2rrz2f6kmhhmcwnr.narinfo6637worker2 # [ 202.784218] nix-grpc-daemon[1314]: ts=2026-09-21T14:32:46Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:57802 duration_s=1 drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv assign_id=16 outputs=16638worker2 # [ 202.798384] nix-grpc-daemon[1314]: ts=2026-09-21T14:32:46Z level=debug event=want client=7 drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv cached=06639worker2 # [ 202.802225] nix-grpc-daemon[1314]: ts=2026-09-21T14:32:46Z level=debug event=assigned drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv worker=192.168.1.4:50051 clients=16640worker2 # [ 202.806312] nix-daemon[838]: reaped child process 1638, status = succeeded6641worker2 # [ 202.826370] nix-grpc-daemon[1314]: ts=2026-09-21T14:32:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:578026642worker2 # [ 202.841177] nix-grpc-daemon[1314]: ts=2026-09-21T14:32:46Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:578026643worker2 # [ 202.855520] nix-grpc-daemon[1410]: time=2026-09-21T14:32:46.471Z level=INFO msg="All 1 paths already cached"6644client # copying 1 paths to 'lb:50051'...6645client # copying path '/nix/store/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv' to 'lb:50051'...6646worker2 # [ 202.865584] nix-grpc-daemon[1314]: ts=2026-09-21T14:32:46Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:578026647worker2 # [ 202.878903] nix-daemon[838]: accepted connection from pid 1314, user nix-grpc-daemon (trusted)6648worker2 # [ 203.036771] nix-daemon[838]: accepted connection from pid 1665, user nix-grpc-daemon (trusted)6649worker2 # [ 203.056469] nix-daemon[838]: reaped child process 1668, status = succeeded6650worker2 # [ 203.158208] nix-daemon[838]: accepted connection from pid 1670, user nix-grpc-daemon (trusted)6651worker2 # [ 203.174224] nix-daemon[838]: reaped child process 1673, status = succeeded6652lb # [ 203.187128] niks3-server[1111]: 2026/09/21 14:32:46 INFO Received uploads request method=POST path=/api/pending_closures6653worker2 # [ 203.189244] nix-grpc-daemon[1410]: time=2026-09-21T14:32:46.804Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6654worker2 # [ 203.190718] nix-grpc-daemon[1410]: time=2026-09-21T14:32:46.804Z level=INFO msg="Uploading fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv (632B)"6655lb # [ 203.216214] niks3-server[1111]: 2026/09/21 14:32:46 INFO Registered completed upload object_key=nar/1ax4z88sb8d24gsvydxvgdy2lijkmhnycxhr5war1j3sq97cyzqf.nar.zst6656worker2 # [ 203.222029] nix-grpc-daemon[1410]: time=2026-09-21T14:32:46.837Z level=INFO msg="Uploading 1 narinfos"6657lb # [ 203.230328] niks3-server[1111]: 2026/09/21 14:32:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/72/sign6658lb # [ 203.243295] niks3-server[1111]: 2026/09/21 14:32:46 INFO Signed narinfos id=72 count=16659lb # [ 203.250780] niks3-server[1111]: 2026/09/21 14:32:46 INFO Registered completed upload object_key=fa7adcm0x8sbgmrsfx912fp8xakk99g6.ls6660worker2 # [ 203.255595] nix-grpc-daemon[1410]: time=2026-09-21T14:32:46.871Z level=INFO msg="Upload complete. (360ms)"6661lb # [ 203.263930] niks3-server[1111]: 2026/09/21 14:32:46 INFO Received complete upload request method=POST path=/api/pending_closures/72/complete6662lb # [ 203.269380] niks3-server[1111]: 2026/09/21 14:32:46 INFO Completed upload id=726663worker2 # [ 203.264280] nix-grpc-daemon[1314]: ts=2026-09-21T14:32:46Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:57802 duration_s=0 paths=1 nar_bytes_in=6326664lb # [ 203.273266] niks3-server[1111]: 2026/09/21 14:32:46 INFO Registered completed upload object_key=fa7adcm0x8sbgmrsfx912fp8xakk99g6.narinfo6665worker2 # [ 203.275675] nix-grpc-daemon[1314]: ts=2026-09-21T14:32:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:578026666worker2 # [ 203.282020] nix-daemon[838]: accepted connection from pid 1314, user nix-grpc-daemon (trusted)6667worker2 # [ 203.287210] nix-daemon[838]: reaped child process 1663, status = succeeded6668client # building '/nix/store/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv' on lb:50051...6669worker2 # [ 203.299198] nix-daemon[838]: accepted connection from pid 1314, user nix-grpc-daemon (trusted)6670client # farm-b> worker2: building fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv6671client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds6672client # farm-b> LOG-farm-b-drain6673worker2 # [ 204.163873] nix-daemon[838]: reaped child process 1677, status = succeeded6674worker2 # [ 204.295406] nix-daemon[838]: accepted connection from pid 1693, user nix-grpc-daemon (trusted)6675worker2 # [ 204.312177] nix-daemon[838]: reaped child process 1696, status = succeeded6676lb # [ 204.327983] niks3-server[1111]: 2026/09/21 14:32:47 INFO Received uploads request method=POST path=/api/pending_closures6677worker2 # [ 204.332431] nix-grpc-daemon[1410]: time=2026-09-21T14:32:47.946Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6678worker2 # [ 204.334111] nix-grpc-daemon[1410]: time=2026-09-21T14:32:47.946Z level=INFO msg="Uploading g774lks418zc45bd9hwhinm61pwcwnr9-farm-b (176B)"6679lb # [ 204.370270] niks3-server[1111]: 2026/09/21 14:32:47 INFO Registered completed upload object_key=nar/1ammzz4zx9sggy8v73zxk3j08vbqxgpw14n8lx2fh57bx24yzvnj.nar.zst6680lb # [ 204.384405] niks3-server[1111]: 2026/09/21 14:32:48 INFO Registered completed upload object_key=log/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv6681worker2 # [ 204.386340] nix-grpc-daemon[1410]: time=2026-09-21T14:32:48.001Z level=INFO msg="Uploading 1 narinfos"6682lb # [ 204.395973] niks3-server[1111]: 2026/09/21 14:32:48 INFO Received sign narinfos request method=POST path=/api/pending_closures/73/sign6683lb # [ 204.404157] niks3-server[1111]: 2026/09/21 14:32:48 INFO Signed narinfos id=73 count=16684lb # [ 204.409417] niks3-server[1111]: 2026/09/21 14:32:48 INFO Registered completed upload object_key=g774lks418zc45bd9hwhinm61pwcwnr9.ls6685lb # [ 204.415490] niks3-server[1111]: 2026/09/21 14:32:48 INFO Received complete upload request method=POST path=/api/pending_closures/73/complete6686lb # [ 204.421263] niks3-server[1111]: 2026/09/21 14:32:48 INFO Completed upload id=736687worker2 # [ 204.414499] nix-grpc-daemon[1410]: time=2026-09-21T14:32:48.029Z level=INFO msg="Upload complete. (303ms)"6688lb # [ 204.423813] niks3-server[1111]: 2026/09/21 14:32:48 INFO Registered completed upload object_key=g774lks418zc45bd9hwhinm61pwcwnr9.narinfo6689worker2 # [ 204.436570] nix-grpc-daemon[1314]: ts=2026-09-21T14:32:48Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:57802 duration_s=1 drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv assign_id=17 outputs=16690worker2 # [ 204.448413] nix-grpc-daemon[1314]: ts=2026-09-21T14:32:48Z level=debug event=want client=7 drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv cached=06691worker2 # [ 204.454068] nix-grpc-daemon[1314]: ts=2026-09-21T14:32:48Z level=debug event=assigned drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv worker=192.168.1.4:50051 clients=16692worker2 # [ 204.458627] nix-daemon[838]: reaped child process 1675, status = succeeded6693worker2 # [ 204.477713] nix-grpc-daemon[1314]: ts=2026-09-21T14:32:48Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:578026694worker2 # [ 204.492945] nix-grpc-daemon[1314]: ts=2026-09-21T14:32:48Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:578026695worker2 # [ 204.506943] nix-grpc-daemon[1410]: time=2026-09-21T14:32:48.122Z level=INFO msg="All 2 paths already cached"6696client # copying 1 paths to 'lb:50051'...6697client # copying path '/nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv' to 'lb:50051'...6698worker2 # [ 204.517478] nix-grpc-daemon[1314]: ts=2026-09-21T14:32:48Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:578026699worker2 # [ 204.531038] nix-daemon[838]: accepted connection from pid 1314, user nix-grpc-daemon (trusted)6700worker2 # [ 204.695014] nix-daemon[838]: accepted connection from pid 1702, user nix-grpc-daemon (trusted)6701worker2 # [ 204.713946] nix-daemon[838]: reaped child process 1705, status = succeeded6702worker2 # [ 204.821987] nix-daemon[838]: accepted connection from pid 1707, user nix-grpc-daemon (trusted)6703worker2 # [ 204.837907] nix-daemon[838]: reaped child process 1710, status = succeeded6704lb # [ 204.851671] niks3-server[1111]: 2026/09/21 14:32:48 INFO Received uploads request method=POST path=/api/pending_closures6705worker2 # [ 204.854108] nix-grpc-daemon[1410]: time=2026-09-21T14:32:48.469Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6706worker2 # [ 204.855605] nix-grpc-daemon[1410]: time=2026-09-21T14:32:48.469Z level=INFO msg="Uploading 77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv (760B)"6707lb # [ 204.878045] niks3-server[1111]: 2026/09/21 14:32:48 INFO Registered completed upload object_key=nar/1hrr94bvb37dwmf1giypzl9qnff72xm540ak7q49j34fyzia9ngb.nar.zst6708lb # [ 204.892294] niks3-server[1111]: 2026/09/21 14:32:48 INFO Received sign narinfos request method=POST path=/api/pending_closures/74/sign6709worker2 # [ 204.892840] nix-grpc-daemon[1410]: time=2026-09-21T14:32:48.508Z level=INFO msg="Uploading 1 narinfos"6710lb # [ 204.904908] niks3-server[1111]: 2026/09/21 14:32:48 INFO Signed narinfos id=74 count=16711lb # [ 204.911525] niks3-server[1111]: 2026/09/21 14:32:48 INFO Registered completed upload object_key=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l.ls6712worker2 # [ 204.917685] nix-grpc-daemon[1410]: time=2026-09-21T14:32:48.533Z level=INFO msg="Upload complete. (370ms)"6713lb # [ 204.924485] niks3-server[1111]: 2026/09/21 14:32:48 INFO Received complete upload request method=POST path=/api/pending_closures/74/complete6714lb # [ 204.931921] niks3-server[1111]: 2026/09/21 14:32:48 INFO Completed upload id=746715worker2 # [ 204.925546] nix-grpc-daemon[1314]: ts=2026-09-21T14:32:48Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:57802 duration_s=0 paths=1 nar_bytes_in=7606716lb # [ 204.935867] niks3-server[1111]: 2026/09/21 14:32:48 INFO Registered completed upload object_key=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l.narinfo6717worker2 # [ 204.938811] nix-grpc-daemon[1314]: ts=2026-09-21T14:32:48Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:578026718worker2 # [ 204.948769] nix-daemon[838]: accepted connection from pid 1314, user nix-grpc-daemon (trusted)6719worker2 # [ 204.951641] nix-daemon[838]: reaped child process 1700, status = succeeded6720client # building '/nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv' on lb:50051...6721worker2 # [ 204.963418] nix-daemon[838]: accepted connection from pid 1314, user nix-grpc-daemon (trusted)6722client # farm-top> worker2: building 77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv6723client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds6724client # farm-top> LOG-farm-top-drain6725worker2 # [ 206.478156] nix-daemon[838]: reaped child process 1714, status = succeeded6726worker2 # [ 206.552084] nix-daemon[838]: accepted connection from pid 1730, user nix-grpc-daemon (trusted)6727worker2 # [ 206.573984] nix-daemon[838]: reaped child process 1733, status = succeeded6728lb # [ 206.581894] niks3-server[1111]: 2026/09/21 14:32:50 INFO Received uploads request method=POST path=/api/pending_closures6729worker2 # [ 206.584452] nix-grpc-daemon[1410]: time=2026-09-21T14:32:50.199Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6730worker2 # [ 206.586142] nix-grpc-daemon[1410]: time=2026-09-21T14:32:50.199Z level=INFO msg="Uploading bzkv277k4v9r9h4aq5h13ddcavmp36fg-farm-top (232B)"6731lb # [ 206.628215] niks3-server[1111]: 2026/09/21 14:32:50 INFO Registered completed upload object_key=nar/0l8sy8grn9pmq2n58ag0hx791zdfyxwk9319ynjqmn7yy7wl64s7.nar.zst6732worker2 # [ 206.635753] nix-grpc-daemon[1410]: time=2026-09-21T14:32:50.250Z level=INFO msg="Uploading 1 narinfos"6733lb # [ 206.648324] niks3-server[1111]: 2026/09/21 14:32:50 INFO Received sign narinfos request method=POST path=/api/pending_closures/75/sign6734lb # [ 206.659784] niks3-server[1111]: 2026/09/21 14:32:50 INFO Signed narinfos id=75 count=16735lb # [ 206.665843] niks3-server[1111]: 2026/09/21 14:32:50 INFO Registered completed upload object_key=bzkv277k4v9r9h4aq5h13ddcavmp36fg.ls6736lb # [ 206.677395] niks3-server[1111]: 2026/09/21 14:32:50 INFO Registered completed upload object_key=log/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv6737lb # [ 206.681281] niks3-server[1111]: 2026/09/21 14:32:50 INFO Received complete upload request method=POST path=/api/pending_closures/75/complete6738lb # [ 206.686472] niks3-server[1111]: 2026/09/21 14:32:50 INFO Completed upload id=756739worker2 # [ 206.679468] nix-grpc-daemon[1410]: time=2026-09-21T14:32:50.294Z level=INFO msg="Upload complete. (302ms)"6740lb # [ 206.689580] niks3-server[1111]: 2026/09/21 14:32:50 INFO Registered completed upload object_key=bzkv277k4v9r9h4aq5h13ddcavmp36fg.narinfo6741worker2 # [ 206.693371] nix-grpc-daemon[1314]: ts=2026-09-21T14:32:50Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:57802 duration_s=1 drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv assign_id=18 outputs=16742worker2 # [ 206.711944] nix-daemon[838]: reaped child process 1712, status = succeeded6743client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag drain >&2, in 6.04 seconds)6744client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag drain outPath6745client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag drain outPath, in 0.17 seconds)6746worker1: must succeed: rm /nix/.rw-store/fill6747worker1: (finished: must succeed: rm /nix/.rw-store/fill, in 0.20 seconds)6748worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free'6749worker1 # [ 208.126395] nix-grpc-daemon[1857]: ts=2026-09-21T14:32:51Z level=info event=healthy reason=min_free6750worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free', in 1.11 seconds)6751(finished: subtest: low disk drains a worker and builds go to the other, in 9.96 seconds)6752(finished: run the VM test script, in 209.24 seconds)6753test script finished in 209.49s6754cleanup6755kill QemuMachine (pid 49)6756client # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/3fl7bdkdxk2k4nssy1d1161isbzn6bsr-python3-3.14.7/bin/python3.14)6757client # [2026-09-21T14:32:52Z INFO virtiofsd] Client disconnected, shutting down6758client # [2026-09-21T14:32:52Z INFO virtiofsd] Client disconnected, shutting down6759client # [2026-09-21T14:32:52Z INFO virtiofsd] Client disconnected, shutting down6760kill QemuMachine (pid 50)6761lb # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/3fl7bdkdxk2k4nssy1d1161isbzn6bsr-python3-3.14.7/bin/python3.14)6762lb # [2026-09-21T14:32:52Z INFO virtiofsd] Client disconnected, shutting down6763lb # [2026-09-21T14:32:52Z INFO virtiofsd] Client disconnected, shutting down6764lb # [2026-09-21T14:32:52Z INFO virtiofsd] Client disconnected, shutting down6765kill QemuMachine (pid 52)6766worker1 # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/3fl7bdkdxk2k4nssy1d1161isbzn6bsr-python3-3.14.7/bin/python3.14)6767worker1 # [2026-09-21T14:32:52Z INFO virtiofsd] Client disconnected, shutting down6768worker1 # [2026-09-21T14:32:52Z INFO virtiofsd] Client disconnected, shutting down6769worker1 # [2026-09-21T14:32:52Z INFO virtiofsd] Client disconnected, shutting down6770kill QemuMachine (pid 51)6771worker2 # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/3fl7bdkdxk2k4nssy1d1161isbzn6bsr-python3-3.14.7/bin/python3.14)6772worker2 # [2026-09-21T14:32:53Z INFO virtiofsd] Client disconnected, shutting down6773worker2 # [2026-09-21T14:32:53Z INFO virtiofsd] Client disconnected, shutting down6774worker2 # [2026-09-21T14:32:53Z INFO virtiofsd] Client disconnected, shutting down6775(finished: cleanup, in 1.04 seconds)6776additionally exposed symbols:6777 client, lb, worker1, worker2,6778 vlan1,6779 start_all, test_script, machines, machines_qemu, machines_nspawn, vlans, driver, log, os, create_machine, subtest, run_tests, join_all, retry, serial_stdout_off, serial_stdout_on, polling_condition, BaseMachine, QemuMachine, NspawnMachine, t, debug, dump_machine_ssh