vm-test-run-nix-grpc-farm
checks.aarch64-linux.farm
· build #464
· raw
1tribuchet: building on eliza2Machine state will be reset. To keep it, pass --keep-machine-state3start all VLans4(finished: start all VLans, in 0.00 seconds)5Test will time out and terminate in 900.0 seconds6run the VM test script7start all VMs8client: starting vm9lb: starting vm10worker1: starting vm11worker2: starting vm12lb # Disk image does not exist, creating the virtualisation disk image...13client # Disk image does not exist, creating the virtualisation disk image...14lb # Formatting '/build/vm-state-lb/tmp.xVeMw2ahYp', fmt=raw size=107374182415worker2: QEMU running (pid 52)16worker2 # Disk image does not exist, creating the virtualisation disk image...17lb: QEMU running (pid 50)18lb # mke2fs 1.47.4 (6-Mar-2025)19worker1: QEMU running (pid 51)20worker2 # Formatting '/build/vm-state-worker2/tmp.XyX5gpLYng', fmt=raw size=107374182421worker1 # Disk image does not exist, creating the virtualisation disk image...22client # Formatting '/build/vm-state-client/tmp.1motHJCeyK', fmt=raw size=107374182423worker2 # mke2fs 1.47.4 (6-Mar-2025)24worker1 # Formatting '/build/vm-state-worker1/tmp.j9mpZC42KH', fmt=raw size=107374182425client # mke2fs 1.47.4 (6-Mar-2025)26worker2 # Discarding device blocks: 0/262144 done27worker1 # mke2fs 1.47.4 (6-Mar-2025)28client # Discarding device blocks: 0/262144 done29worker2 # Creating filesystem with 262144 4k blocks and 65536 inodes30worker1 # Discarding device blocks: 0/262144 done31client # Creating filesystem with 262144 4k blocks and 65536 inodes32worker2 # Filesystem UUID: 97bd670e-dfd6-452b-b95d-333e56646b5433worker1 # Creating filesystem with 262144 4k blocks and 65536 inodes34client # Filesystem UUID: d8120e89-9010-4664-aa39-806ccbc94f1135client # Superblock backups stored on blocks:36worker2 # Superblock backups stored on blocks:37lb # Discarding device blocks: 0/262144 done38client # 32768, 98304, 163840, 22937639worker2 # 32768, 98304, 163840, 22937640lb # Creating filesystem with 262144 4k blocks and 65536 inodes41client # 42worker2 # 43lb # Filesystem UUID: 354440ba-1f96-4418-b8f3-d4670856c55844client # Allocating group tables: 0/8 done45worker2 # Allocating group tables: 0/8 done46lb # Superblock backups stored on blocks:47client # Writing inode tables: 0/8 done48worker2 # Writing inode tables: 0/8 done49lb # 32768, 98304, 163840, 22937650client # Creating journal (8192 blocks): done51worker2 # Creating journal (8192 blocks): done52lb # 53client # Writing superblocks and filesystem accounting information: 0/8 done54worker2 # Writing superblocks and filesystem accounting information: 0/8 done55lb # Allocating group tables: 0/8 done56client # 57worker2 # 58lb # Writing inode tables: 0/8 done59client # Virtualisation disk image created.60worker2 # Virtualisation disk image created.61lb # Creating journal (8192 blocks): done62client: QEMU running (pid 49)63lb # Writing superblocks and filesystem accounting information: 0/8 done64worker1 # Filesystem UUID: 081bd967-e28b-42b1-aa39-a7d26f23863b65lb # 66worker1 # Superblock backups stored on blocks:67(finished: start all VMs, in 0.52 seconds)68worker1 # 32768, 98304, 163840, 22937669client: waiting for unit network-addresses-eth1.service70worker1 # 71client: waiting for the VM to finish booting72worker1 # Allocating group tables: 0/8 done73lb # Virtualisation disk image created.74worker1 # Writing inode tables: 0/8 done75worker1 # Creating journal (8192 blocks): done76worker1 # Writing superblocks and filesystem accounting information: 0/8 done77worker1 # 78worker1 # Virtualisation disk image created.79lb # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40]80lb # [ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 7 15:23:00 UTC 202681lb # [ 0.000000] KASLR enabled82lb # [ 0.000000] random: crng init done83lb # [ 0.000000] Machine model: linux,dummy-virt84lb # [ 0.000000] efi: UEFI not found.85lb # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT86lb # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff]87lb # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f]88lb # [ 0.000000] Zone ranges:89lb # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff]90lb # [ 0.000000] DMA32 empty91lb # [ 0.000000] Normal empty92lb # [ 0.000000] Device empty93lb # [ 0.000000] Movable zone start for each node94lb # [ 0.000000] Early memory node ranges95lb # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff]96lb # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff]97lb # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c40000098lb # [ 0.000000] psci: probing for conduit method from DT.99lb # [ 0.000000] psci: PSCIv1.3 detected in firmware.100lb # [ 0.000000] psci: Using standard PSCI v0.2 function IDs101lb # [ 0.000000] psci: Trusted OS migration not required102lb # [ 0.000000] psci: SMC Calling Convention v1.1103lb # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003)104lb # [ 0.000000] percpu: Embedded 76 pages/cpu s186648 r8192 d116456 u311296105lb # [ 0.000000] Detected PIPT I-cache on CPU0106lb # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm)107lb # [ 0.000000] CPU features: detected: GICv3 CPU interface108lb # [ 0.000000] CPU features: detected: Spectre-v4109lb # [ 0.000000] CPU features: detected: Spectre-BHB110lb # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38111worker2 # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40]112lb # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23113lb # [ 0.000000] alternatives: applying boot alternatives114worker2 # [ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 7 15:23:00 UTC 2026115worker2 # [ 0.000000] KASLR enabled116worker2 # [ 0.000000] random: crng init done117worker2 # [ 0.000000] Machine model: linux,dummy-virt118worker2 # [ 0.000000] efi: UEFI not found.119worker2 # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT120client # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40]121lb # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/168cy51b13gm4gbaddwdhrwvcvzdxbrq-nixos-system-lb-test/init regInfo=/nix/store/mnw4siin8g9mlnyg6j8vpnpyazygjysc-closure-info/registration console=ttyAMA0,115200n8 console=tty0122worker2 # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff]123worker2 # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f]124client # [ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 7 15:23:00 UTC 2026125worker2 # [ 0.000000] Zone ranges:126client # [ 0.000000] KASLR enabled127worker1 # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40]128client # [ 0.000000] random: crng init done129worker2 # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff]130client # [ 0.000000] Machine model: linux,dummy-virt131worker2 # [ 0.000000] DMA32 empty132lb # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/mnw4siin8g9mlnyg6j8vpnpyazygjysc-closure-info/registration", will be passed to user space.133worker2 # [ 0.000000] Normal empty134client # [ 0.000000] efi: UEFI not found.135worker2 # [ 0.000000] Device empty136worker1 # [ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 7 15:23:00 UTC 2026137worker1 # [ 0.000000] KASLR enabled138lb # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes139client # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT140worker2 # [ 0.000000] Movable zone start for each node141worker1 # [ 0.000000] random: crng init done142worker2 # [ 0.000000] Early memory node ranges143lb # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)144client # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff]145worker1 # [ 0.000000] Machine model: linux,dummy-virt146worker2 # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff]147worker1 # [ 0.000000] efi: UEFI not found.148client # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f]149lb # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)150client # [ 0.000000] Zone ranges:151worker2 # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff]152worker1 # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT153client # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff]154worker2 # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000155lb # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB156client # [ 0.000000] DMA32 empty157worker1 # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff]158lb # [ 0.000000] software IO TLB: area num 1.159worker2 # [ 0.000000] psci: probing for conduit method from DT.160client # [ 0.000000] Normal empty161worker1 # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f]162client # [ 0.000000] Device empty163lb # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB164worker2 # [ 0.000000] psci: PSCIv1.3 detected in firmware.165worker1 # [ 0.000000] Zone ranges:166client # [ 0.000000] Movable zone start for each node167worker2 # [ 0.000000] psci: Using standard PSCI v0.2 function IDs168client # [ 0.000000] Early memory node ranges169worker1 # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff]170lb # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB)171worker2 # [ 0.000000] psci: Trusted OS migration not required172worker1 # [ 0.000000] DMA32 empty173client # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff]174lb # [ 0.000000] Fallback order for Node 0: 0175worker1 # [ 0.000000] Normal empty176worker2 # [ 0.000000] psci: SMC Calling Convention v1.1177worker1 # [ 0.000000] Device empty178lb # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216179client # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff]180worker1 # [ 0.000000] Movable zone start for each node181lb # [ 0.000000] Policy zone: DMA182worker2 # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003)183worker1 # [ 0.000000] Early memory node ranges184client # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000185worker2 # [ 0.000000] percpu: Embedded 76 pages/cpu s186648 r8192 d116456 u311296186lb # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off187client # [ 0.000000] psci: probing for conduit method from DT.188worker1 # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff]189worker2 # [ 0.000000] Detected PIPT I-cache on CPU0190client # [ 0.000000] psci: PSCIv1.3 detected in firmware.191lb # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1192worker1 # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff]193client # [ 0.000000] psci: Using standard PSCI v0.2 function IDs194lb # [ 0.000000] allocated 3145728 bytes of page_ext195worker2 # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm)196worker1 # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000197client # [ 0.000000] psci: Trusted OS migration not required198lb # [ 0.000000] ftrace: allocating 74885 entries in 294 pages199worker1 # [ 0.000000] psci: probing for conduit method from DT.200worker2 # [ 0.000000] CPU features: detected: GICv3 CPU interface201client # [ 0.000000] psci: SMC Calling Convention v1.1202lb # [ 0.000000] ftrace: allocated 294 pages with 4 groups203worker2 # [ 0.000000] CPU features: detected: Spectre-v4204worker1 # [ 0.000000] psci: PSCIv1.3 detected in firmware.205lb # [ 0.000000] rcu: Hierarchical RCU implementation.206worker2 # [ 0.000000] CPU features: detected: Spectre-BHB207worker1 # [ 0.000000] psci: Using standard PSCI v0.2 function IDs208client # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003)209lb # [ 0.000000] rcu: RCU event tracing is enabled.210worker1 # [ 0.000000] psci: Trusted OS migration not required211worker2 # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38212client # [ 0.000000] percpu: Embedded 76 pages/cpu s186648 r8192 d116456 u311296213worker1 # [ 0.000000] psci: SMC Calling Convention v1.1214lb # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.215worker2 # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23216client # [ 0.000000] Detected PIPT I-cache on CPU0217lb # [ 0.000000] Trampoline variant of Tasks RCU enabled.218worker2 # [ 0.000000] alternatives: applying boot alternatives219worker1 # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003)220lb # [ 0.000000] Rude variant of Tasks RCU enabled.221client # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm)222lb # [ 0.000000] Tracing variant of Tasks RCU enabled.223worker1 # [ 0.000000] percpu: Embedded 76 pages/cpu s186648 r8192 d116456 u311296224client # [ 0.000000] CPU features: detected: GICv3 CPU interface225worker1 # [ 0.000000] Detected PIPT I-cache on CPU0226client # [ 0.000000] CPU features: detected: Spectre-v4227lb # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.228client # [ 0.000000] CPU features: detected: Spectre-BHB229worker1 # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm)230lb # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1231client # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38232worker1 # [ 0.000000] CPU features: detected: GICv3 CPU interface233worker1 # [ 0.000000] CPU features: detected: Spectre-v4234client # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23235worker2 # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/jzf4xkwa8xmlj8vk00ys4gir3nz7fy8j-nixos-system-worker2-test/init regInfo=/nix/store/8vck4wm0zxi7gd9ibf4y3jhycpzbwnl6-closure-info/registration console=ttyAMA0,115200n8 console=tty0236lb # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.237worker1 # [ 0.000000] CPU features: detected: Spectre-BHB238client # [ 0.000000] alternatives: applying boot alternatives239lb # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.240worker1 # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38241worker1 # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23242lb # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.243worker2 # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/8vck4wm0zxi7gd9ibf4y3jhycpzbwnl6-closure-info/registration", will be passed to user space.244worker1 # [ 0.000000] alternatives: applying boot alternatives245lb # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0246worker2 # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes247lb # [ 0.000000] GICv3: 256 SPIs implemented248lb # [ 0.000000] GICv3: 0 Extended SPIs implemented249worker2 # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)250lb # [ 0.000000] Root IRQ handler: gic_handle_irq251client # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/p6p9342ifnppad60sm4s8zcrc0dk790i-nixos-system-client-test/init regInfo=/nix/store/mfi7s969scingkrqknkvk3lldv44xdhn-closure-info/registration console=ttyAMA0,115200n8 console=tty0252lb # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI253worker2 # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)254lb # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0255worker2 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB256client # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/mfi7s969scingkrqknkvk3lldv44xdhn-closure-info/registration", will be passed to user space.257worker1 # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/143xd0r0k7bpy3hb59k6jhcv7f9kn0pn-nixos-system-worker1-test/init regInfo=/nix/store/kyyw7i260gz287j3y494hm7f7ygzcgmp-closure-info/registration console=ttyAMA0,115200n8 console=tty0258worker2 # [ 0.000000] software IO TLB: area num 1.259lb # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000260lb # [ 0.000000] ITS [mem 0x08080000-0x0809ffff]261client # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes262worker2 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB263lb # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1)264client # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)265worker1 # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/kyyw7i260gz287j3y494hm7f7ygzcgmp-closure-info/registration", will be passed to user space.266worker2 # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB)267worker2 # [ 0.000000] Fallback order for Node 0: 0268client # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)269lb # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1)270worker2 # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216271worker1 # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes272worker2 # [ 0.000000] Policy zone: DMA273lb # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000274client # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB275worker1 # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)276worker2 # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off277lb # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000278client # [ 0.000000] software IO TLB: area num 1.279worker2 # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1280worker1 # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)281lb # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.282client # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB283worker2 # [ 0.000000] allocated 3145728 bytes of page_ext284worker1 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB285worker2 # [ 0.000000] ftrace: allocating 74885 entries in 294 pages286client # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB)287lb # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns288worker1 # [ 0.000000] software IO TLB: area num 1.289worker2 # [ 0.000000] ftrace: allocated 294 pages with 4 groups290client # [ 0.000000] Fallback order for Node 0: 0291lb # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt).292worker1 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB293worker2 # [ 0.000000] rcu: Hierarchical RCU implementation.294client # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216295worker2 # [ 0.000000] rcu: RCU event tracing is enabled.296client # [ 0.000000] Policy zone: DMA297worker1 # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB)298lb # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns299worker1 # [ 0.000000] Fallback order for Node 0: 0300worker2 # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.301client # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off302worker2 # [ 0.000000] Trampoline variant of Tasks RCU enabled.303lb # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns304worker1 # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216305client # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1306worker2 # [ 0.000000] Rude variant of Tasks RCU enabled.307worker1 # [ 0.000000] Policy zone: DMA308lb # [ 0.000035] arm-pv: using stolen time PV309client # [ 0.000000] allocated 3145728 bytes of page_ext310worker2 # [ 0.000000] Tracing variant of Tasks RCU enabled.311worker1 # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off312client # [ 0.000000] ftrace: allocating 74885 entries in 294 pages313lb # [ 0.000651] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)314worker2 # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.315worker1 # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1316client # [ 0.000000] ftrace: allocated 294 pages with 4 groups317lb # [ 0.000816] Console: colour dummy device 80x25318worker1 # [ 0.000000] allocated 3145728 bytes of page_ext319worker2 # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1320client # [ 0.000000] rcu: Hierarchical RCU implementation.321lb # [ 0.000824] printk: legacy console [tty0] enabled322client # [ 0.000000] rcu: RCU event tracing is enabled.323worker1 # [ 0.000000] ftrace: allocating 74885 entries in 294 pages324worker2 # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.325worker1 # [ 0.000000] ftrace: allocated 294 pages with 4 groups326client # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.327lb # [ 0.001016] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000)328worker1 # [ 0.000000] rcu: Hierarchical RCU implementation.329lb # [ 0.001023] pid_max: default: 32768 minimum: 301330client # [ 0.000000] Trampoline variant of Tasks RCU enabled.331worker2 # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.332worker1 # [ 0.000000] rcu: RCU event tracing is enabled.333client # [ 0.000000] Rude variant of Tasks RCU enabled.334lb # [ 0.001092] LSM: initializing lsm=capability,landlock,yama,bpf,ima335worker1 # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.336lb # [ 0.001258] landlock: Up and running.337client # [ 0.000000] Tracing variant of Tasks RCU enabled.338worker2 # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.339lb # [ 0.001261] Yama: becoming mindful.340worker1 # [ 0.000000] Trampoline variant of Tasks RCU enabled.341worker2 # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0342lb # [ 0.001893] LSM support for eBPF active343client # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.344worker1 # [ 0.000000] Rude variant of Tasks RCU enabled.345worker2 # [ 0.000000] GICv3: 256 SPIs implemented346client # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1347worker1 # [ 0.000000] Tracing variant of Tasks RCU enabled.348worker2 # [ 0.000000] GICv3: 0 Extended SPIs implemented349lb # [ 0.002044] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)350worker2 # [ 0.000000] Root IRQ handler: gic_handle_irq351worker1 # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.352lb # [ 0.002077] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)353client # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.354worker2 # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI355worker1 # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1356lb # [ 0.003244] cacheinfo: Unable to detect cache hierarchy for CPU 0357worker2 # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0358client # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.359lb # [ 0.004023] rcu: Hierarchical SRCU implementation.360worker2 # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000361worker1 # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.362lb # [ 0.004027] rcu: Max phase no-delay instances is 1000.363worker2 # [ 0.000000] ITS [mem 0x08080000-0x0809ffff]364client # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.365lb # [ 0.005247] fsl-mc MSI: its@8080000 domain created366worker1 # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.367client # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0368lb # [ 0.005347] EFI services will not be available.369worker2 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449e0000 (indirect, esz 8, psz 64K, shr 1)370client # [ 0.000000] GICv3: 256 SPIs implemented371lb # [ 0.005415] smp: Bringing up secondary CPUs ...372worker1 # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.373client # [ 0.000000] GICv3: 0 Extended SPIs implemented374lb # [ 0.005423] smp: Brought up 1 node, 1 CPU375worker2 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @449f0000 (flat, esz 8, psz 64K, shr 1)376client # [ 0.000000] Root IRQ handler: gic_handle_irq377worker1 # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0378lb # [ 0.005426] SMP: Total of 1 processors activated.379client # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI380worker1 # [ 0.000000] GICv3: 256 SPIs implemented381lb # [ 0.005431] CPU: All CPU(s) started at EL1382worker2 # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000383worker1 # [ 0.000000] GICv3: 0 Extended SPIs implemented384client # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0385lb # [ 0.005442] CPU features: detected: Branch Target Identification386worker2 # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000387worker1 # [ 0.000000] Root IRQ handler: gic_handle_irq388client # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000389lb # [ 0.005447] CPU features: detected: ARMv8.4 Translation Table Level390worker1 # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI391worker2 # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.392client # [ 0.000000] ITS [mem 0x08080000-0x0809ffff]393worker1 # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0394lb # [ 0.005450] CPU features: detected: Instruction cache invalidation not required for I/D coherence395worker2 # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns396client # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1)397worker1 # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000398worker1 # [ 0.000000] ITS [mem 0x08080000-0x0809ffff]399worker2 # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt).400lb # [ 0.005454] CPU features: detected: Data cache clean to the PoU not required for I/D coherence401client # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1)402lb # [ 0.005458] CPU features: detected: Common not Private translations403worker1 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1)404client # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000405worker2 # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns406lb # [ 0.005461] CPU features: detected: CRC32 instructions407client # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000408worker2 # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns409worker1 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1)410lb # [ 0.005464] CPU features: detected: Data cache clean to Point of Deep Persistence411worker2 # [ 0.000030] arm-pv: using stolen time PV412client # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.413worker1 # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000414lb # [ 0.005467] CPU features: detected: Data cache clean to Point of Persistence415worker1 # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000416worker2 # [ 0.000395] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)417lb # [ 0.005470] CPU features: detected: Data independent timing control (DIT)418client # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns419worker2 # [ 0.000560] Console: colour dummy device 80x25420lb # [ 0.005473] CPU features: detected: E0PD421worker1 # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.422client # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt).423worker2 # [ 0.000569] printk: legacy console [tty0] enabled424lb # [ 0.005476] CPU features: detected: Enhanced Counter Virtualization425worker1 # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns426lb # [ 0.005479] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF)427client # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns428worker2 # [ 0.000768] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000)429worker1 # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt).430lb # [ 0.005482] CPU features: detected: Enhanced Virtualization Traps431worker2 # [ 0.000774] pid_max: default: 32768 minimum: 301432client # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns433lb # [ 0.005485] CPU features: detected: Fine Grained Traps434client # [ 0.000029] arm-pv: using stolen time PV435worker2 # [ 0.000842] LSM: initializing lsm=capability,landlock,yama,bpf,ima436worker1 # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns437worker2 # [ 0.000986] landlock: Up and running.438lb # [ 0.005489] CPU features: detected: Generic authentication (architected QARMA5 algorithm)439worker2 # [ 0.000988] Yama: becoming mindful.440worker1 # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns441client # [ 0.000399] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)442worker2 # [ 0.001435] LSM support for eBPF active443lb # [ 0.005493] CPU features: detected: RCpc load-acquire (LDAPR)444worker1 # [ 0.000045] arm-pv: using stolen time PV445client # [ 0.000554] Console: colour dummy device 80x25446lb # [ 0.005496] CPU features: detected: LSE atomic instructions447worker2 # [ 0.001579] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)448client # [ 0.000561] printk: legacy console [tty0] enabled449lb # [ 0.005499] CPU features: detected: Privileged Access Never450worker1 # [ 0.000421] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)451worker2 # [ 0.001612] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)452lb # [ 0.005502] CPU features: detected: PMUv3453worker1 # [ 0.000604] Console: colour dummy device 80x25454client # [ 0.000750] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000)455worker2 # [ 0.002785] cacheinfo: Unable to detect cache hierarchy for CPU 0456lb # [ 0.005505] CPU features: detected: RAS Extension Support457worker1 # [ 0.000612] printk: legacy console [tty0] enabled458client # [ 0.000756] pid_max: default: 32768 minimum: 301459worker2 # [ 0.003654] rcu: Hierarchical SRCU implementation.460lb # [ 0.005507] CPU features: detected: RASv1p1 Extension Support461client # [ 0.000826] LSM: initializing lsm=capability,landlock,yama,bpf,ima462worker2 # [ 0.003658] rcu: Max phase no-delay instances is 1000.463lb # [ 0.005510] CPU features: detected: Random Number Generator464client # [ 0.000977] landlock: Up and running.465worker1 # [ 0.000800] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000)466worker2 # [ 0.004857] fsl-mc MSI: its@8080000 domain created467client # [ 0.000979] Yama: becoming mindful.468worker1 # [ 0.000807] pid_max: default: 32768 minimum: 301469lb # [ 0.005513] CPU features: detected: Speculation barrier (SB)470client # [ 0.001466] LSM support for eBPF active471worker2 # [ 0.004947] EFI services will not be available.472worker1 # [ 0.000885] LSM: initializing lsm=capability,landlock,yama,bpf,ima473worker2 # [ 0.005012] smp: Bringing up secondary CPUs ...474lb # [ 0.005515] CPU features: detected: Stage-2 Force Write-Back475worker1 # [ 0.001008] landlock: Up and running.476client # [ 0.001638] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)477worker2 # [ 0.005020] smp: Brought up 1 node, 1 CPU478worker1 # [ 0.001010] Yama: becoming mindful.479lb # [ 0.005518] CPU features: detected: TLB range maintenance instructions480worker2 # [ 0.005023] SMP: Total of 1 processors activated.481worker1 # [ 0.001447] LSM support for eBPF active482client # [ 0.001672] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)483worker2 # [ 0.005028] CPU: All CPU(s) started at EL1484lb # [ 0.005522] CPU features: detected: Speculative Store Bypassing Safe (SSBS)485client # [ 0.002793] cacheinfo: Unable to detect cache hierarchy for CPU 0486worker1 # [ 0.001578] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)487worker2 # [ 0.005040] CPU features: detected: Branch Target Identification488lb # [ 0.005563] alternatives: applying system-wide alternatives489client # [ 0.003478] rcu: Hierarchical SRCU implementation.490worker1 # [ 0.001610] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)491worker2 # [ 0.005045] CPU features: detected: ARMv8.4 Translation Table Level492client # [ 0.003482] rcu: Max phase no-delay instances is 1000.493lb # [ 0.008596] CPU features: detected: BBM Level 2 without TLB conflict abort494worker1 # [ 0.002694] cacheinfo: Unable to detect cache hierarchy for CPU 0495client # [ 0.004643] fsl-mc MSI: its@8080000 domain created496worker2 # [ 0.005048] CPU features: detected: Instruction cache invalidation not required for I/D coherence497worker1 # [ 0.003447] rcu: Hierarchical SRCU implementation.498client # [ 0.004730] EFI services will not be available.499worker1 # [ 0.003451] rcu: Max phase no-delay instances is 1000.500lb # [ 0.008779] Memory: 1406520K/1572864K available (24448K kernel code, 7090K rwdata, 26348K rodata, 4736K init, 1107K bss, 123764K reserved, 32768K cma-reserved)501client # [ 0.004793] smp: Bringing up secondary CPUs ...502worker2 # [ 0.005051] CPU features: detected: Data cache clean to the PoU not required for I/D coherence503lb # [ 0.009191] devtmpfs: initialized504client # [ 0.004801] smp: Brought up 1 node, 1 CPU505worker1 # [ 0.004682] fsl-mc MSI: its@8080000 domain created506worker2 # [ 0.005055] CPU features: detected: Common not Private translations507worker1 # [ 0.004773] EFI services will not be available.508client # [ 0.004804] SMP: Total of 1 processors activated.509lb # [ 0.011069] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)510client # [ 0.004809] CPU: All CPU(s) started at EL1511worker2 # [ 0.005058] CPU features: detected: CRC32 instructions512worker1 # [ 0.004854] smp: Bringing up secondary CPUs ...513worker1 # [ 0.004862] smp: Brought up 1 node, 1 CPU514client # [ 0.004819] CPU features: detected: Branch Target Identification515lb # [ 0.011099] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).516worker2 # [ 0.005061] CPU features: detected: Data cache clean to Point of Deep Persistence517worker1 # [ 0.004865] SMP: Total of 1 processors activated.518client # [ 0.004824] CPU features: detected: ARMv8.4 Translation Table Level519lb # [ 0.011302] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL520worker1 # [ 0.004871] CPU: All CPU(s) started at EL1521worker2 # [ 0.005065] CPU features: detected: Data cache clean to Point of Persistence522lb # [ 0.011306] 0 pages in range for non-PLT usage523worker1 # [ 0.004882] CPU features: detected: Branch Target Identification524client # [ 0.004827] CPU features: detected: Instruction cache invalidation not required for I/D coherence525worker2 # [ 0.005068] CPU features: detected: Data independent timing control (DIT)526lb # [ 0.011307] 508288 pages in range for PLT usage527worker2 # [ 0.005071] CPU features: detected: E0PD528worker1 # [ 0.004887] CPU features: detected: ARMv8.4 Translation Table Level529lb # [ 0.011419] pinctrl core: initialized pinctrl subsystem530client # [ 0.004830] CPU features: detected: Data cache clean to the PoU not required for I/D coherence531worker2 # [ 0.005073] CPU features: detected: Enhanced Counter Virtualization532lb # [ 0.012198] DMI not present or invalid.533worker1 # [ 0.004890] CPU features: detected: Instruction cache invalidation not required for I/D coherence534client # [ 0.004834] CPU features: detected: Common not Private translations535lb # [ 0.015822] NET: Registered PF_NETLINK/PF_ROUTE protocol family536worker2 # [ 0.005076] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF)537client # [ 0.004837] CPU features: detected: CRC32 instructions538worker1 # [ 0.004894] CPU features: detected: Data cache clean to the PoU not required for I/D coherence539worker2 # [ 0.005080] CPU features: detected: Enhanced Virtualization Traps540lb # [ 0.018037] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations541client # [ 0.004840] CPU features: detected: Data cache clean to Point of Deep Persistence542worker2 # [ 0.005083] CPU features: detected: Fine Grained Traps543worker1 # [ 0.004897] CPU features: detected: Common not Private translations544lb # [ 0.018226] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations545worker1 # [ 0.004901] CPU features: detected: CRC32 instructions546client # [ 0.004843] CPU features: detected: Data cache clean to Point of Persistence547worker2 # [ 0.005087] CPU features: detected: Generic authentication (architected QARMA5 algorithm)548lb # [ 0.018421] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations549worker1 # [ 0.004904] CPU features: detected: Data cache clean to Point of Deep Persistence550client # [ 0.004847] CPU features: detected: Data independent timing control (DIT)551worker2 # [ 0.005092] CPU features: detected: RCpc load-acquire (LDAPR)552client # [ 0.004850] CPU features: detected: E0PD553lb # [ 0.018442] audit: initializing netlink subsys (disabled)554worker1 # [ 0.004907] CPU features: detected: Data cache clean to Point of Persistence555worker2 # [ 0.005095] CPU features: detected: LSE atomic instructions556lb # [ 0.019059] thermal_sys: Registered thermal governor 'fair_share'557client # [ 0.004852] CPU features: detected: Enhanced Counter Virtualization558worker2 # [ 0.005098] CPU features: detected: Privileged Access Never559worker1 # [ 0.004910] CPU features: detected: Data independent timing control (DIT)560lb # [ 0.019061] thermal_sys: Registered thermal governor 'bang_bang'561worker2 # [ 0.005101] CPU features: detected: PMUv3562worker1 # [ 0.004914] CPU features: detected: E0PD563client # [ 0.004855] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF)564worker2 # [ 0.005103] CPU features: detected: RAS Extension Support565lb # [ 0.019065] thermal_sys: Registered thermal governor 'step_wise'566client # [ 0.004858] CPU features: detected: Enhanced Virtualization Traps567worker1 # [ 0.004916] CPU features: detected: Enhanced Counter Virtualization568worker2 # [ 0.005106] CPU features: detected: RASv1p1 Extension Support569lb # [ 0.019070] thermal_sys: Registered thermal governor 'user_space'570client # [ 0.004862] CPU features: detected: Fine Grained Traps571worker1 # [ 0.004919] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF)572worker2 # [ 0.005109] CPU features: detected: Random Number Generator573client # [ 0.004865] CPU features: detected: Generic authentication (architected QARMA5 algorithm)574worker1 # [ 0.004922] CPU features: detected: Enhanced Virtualization Traps575worker2 # [ 0.005111] CPU features: detected: Speculation barrier (SB)576worker1 # [ 0.004925] CPU features: detected: Fine Grained Traps577client # [ 0.004869] CPU features: detected: RCpc load-acquire (LDAPR)578worker2 # [ 0.005114] CPU features: detected: Stage-2 Force Write-Back579client # [ 0.004872] CPU features: detected: LSE atomic instructions580worker1 # [ 0.004929] CPU features: detected: Generic authentication (architected QARMA5 algorithm)581worker2 # [ 0.005117] CPU features: detected: TLB range maintenance instructions582client # [ 0.004876] CPU features: detected: Privileged Access Never583worker1 # [ 0.004933] CPU features: detected: RCpc load-acquire (LDAPR)584client # [ 0.004878] CPU features: detected: PMUv3585worker2 # [ 0.005122] CPU features: detected: Speculative Store Bypassing Safe (SSBS)586worker1 # [ 0.004936] CPU features: detected: LSE atomic instructions587client # [ 0.004881] CPU features: detected: RAS Extension Support588worker2 # [ 0.005160] alternatives: applying system-wide alternatives589worker1 # [ 0.004939] CPU features: detected: Privileged Access Never590client # [ 0.004884] CPU features: detected: RASv1p1 Extension Support591worker2 # [ 0.008199] CPU features: detected: BBM Level 2 without TLB conflict abort592worker1 # [ 0.004942] CPU features: detected: PMUv3593client # [ 0.004886] CPU features: detected: Random Number Generator594worker1 # [ 0.004945] CPU features: detected: RAS Extension Support595client # [ 0.004889] CPU features: detected: Speculation barrier (SB)596worker1 # [ 0.004947] CPU features: detected: RASv1p1 Extension Support597worker2 # [ 0.008345] Memory: 1406536K/1572864K available (24448K kernel code, 7090K rwdata, 26348K rodata, 4736K init, 1107K bss, 123772K reserved, 32768K cma-reserved)598client # [ 0.004891] CPU features: detected: Stage-2 Force Write-Back599worker1 # [ 0.004950] CPU features: detected: Random Number Generator600worker2 # [ 0.008722] devtmpfs: initialized601client # [ 0.004894] CPU features: detected: TLB range maintenance instructions602worker1 # [ 0.004952] CPU features: detected: Speculation barrier (SB)603worker2 # [ 0.010809] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)604worker1 # [ 0.004955] CPU features: detected: Stage-2 Force Write-Back605client # [ 0.004899] CPU features: detected: Speculative Store Bypassing Safe (SSBS)606lb # [ 0.019073] thermal_sys: Registered thermal governor 'power_allocator'607worker2 # [ 0.010831] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).608client # [ 0.004934] alternatives: applying system-wide alternatives609worker1 # [ 0.004958] CPU features: detected: TLB range maintenance instructions610worker2 # [ 0.011031] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL611lb # [ 0.019097] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1612client # [ 0.007857] CPU features: detected: BBM Level 2 without TLB conflict abort613worker1 # [ 0.004962] CPU features: detected: Speculative Store Bypassing Safe (SSBS)614lb # [ 0.019105] cpuidle: using governor ladder615lb # [ 0.019111] cpuidle: using governor menu616worker1 # [ 0.004999] alternatives: applying system-wide alternatives617lb # [ 0.019291] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers.618worker1 # [ 0.008083] CPU features: detected: BBM Level 2 without TLB conflict abort619lb # [ 0.019306] ASID allocator initialised with 65536 entries620lb # [ 0.020524] Serial: AMBA PL011 UART driver621worker1 # [ 0.008293] Memory: 1406508K/1572864K available (24448K kernel code, 7090K rwdata, 26348K rodata, 4736K init, 1107K bss, 123772K reserved, 32768K cma-reserved)622lb # [ 0.025690] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1623worker1 # [ 0.008672] devtmpfs: initialized624lb # [ 0.025839] printk: console [ttyAMA0] enabled625worker1 # [ 0.010519] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)626lb # [ 0.145901] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages627worker1 # [ 0.010541] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).628lb # [ 0.145924] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page629worker1 # [ 0.010735] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL630worker1 # [ 0.010740] 0 pages in range for non-PLT usage631lb # [ 0.145930] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages632worker1 # [ 0.010741] 508288 pages in range for PLT usage633lb # [ 0.145935] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page634worker2 # [ 0.011036] 0 pages in range for non-PLT usage635lb # [ 0.145939] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages636worker2 # [ 0.011037] 508288 pages in range for PLT usage637worker2 # [ 0.011158] pinctrl core: initialized pinctrl subsystem638lb # [ 0.145943] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page639client # [ 0.008075] Memory: 1406508K/1572864K available (24448K kernel code, 7090K rwdata, 26348K rodata, 4736K init, 1107K bss, 123784K reserved, 32768K cma-reserved)640worker2 # [ 0.011998] DMI not present or invalid.641client # [ 0.008484] devtmpfs: initialized642lb # [ 0.145948] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages643worker2 # [ 0.015222] NET: Registered PF_NETLINK/PF_ROUTE protocol family644client # [ 0.010264] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)645lb # [ 0.145952] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page646worker2 # [ 0.017444] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations647client # [ 0.010290] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).648worker2 # [ 0.017636] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations649client # [ 0.010509] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL650client # [ 0.010513] 0 pages in range for non-PLT usage651worker2 # [ 0.017831] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations652lb # [ 0.153375] fbcon: Taking over console653client # [ 0.010514] 508288 pages in range for PLT usage654lb # [ 0.153391] ACPI: Interpreter disabled.655worker2 # [ 0.017852] audit: initializing netlink subsys (disabled)656client # [ 0.010624] pinctrl core: initialized pinctrl subsystem657lb # [ 0.155197] iommu: Default domain type: Translated658worker2 # [ 0.018405] thermal_sys: Registered thermal governor 'fair_share'659client # [ 0.011376] DMI not present or invalid.660lb # [ 0.155206] iommu: DMA domain TLB invalidation policy: strict mode661worker2 # [ 0.018407] thermal_sys: Registered thermal governor 'bang_bang'662client # [ 0.014408] NET: Registered PF_NETLINK/PF_ROUTE protocol family663worker2 # [ 0.018411] thermal_sys: Registered thermal governor 'step_wise'664lb # [ 0.156859] SCSI subsystem initialized665client # [ 0.016716] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations666worker1 # [ 0.010852] pinctrl core: initialized pinctrl subsystem667worker1 # [ 0.011574] DMI not present or invalid.668worker2 # [ 0.018416] thermal_sys: Registered thermal governor 'user_space'669client # [ 0.016884] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations670worker1 # [ 0.014797] NET: Registered PF_NETLINK/PF_ROUTE protocol family671worker2 # [ 0.018419] thermal_sys: Registered thermal governor 'power_allocator'672client # [ 0.017077] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations673worker1 # [ 0.017041] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations674worker2 # [ 0.018449] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1675client # [ 0.017098] audit: initializing netlink subsys (disabled)676worker2 # [ 0.018457] cpuidle: using governor ladder677worker1 # [ 0.017225] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations678worker2 # [ 0.018463] cpuidle: using governor menu679client # [ 0.017641] thermal_sys: Registered thermal governor 'fair_share'680worker1 # [ 0.017424] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations681client # [ 0.017643] thermal_sys: Registered thermal governor 'bang_bang'682worker2 # [ 0.018660] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers.683worker2 # [ 0.018674] ASID allocator initialised with 65536 entries684worker1 # [ 0.017446] audit: initializing netlink subsys (disabled)685client # [ 0.017647] thermal_sys: Registered thermal governor 'step_wise'686worker2 # [ 0.019860] Serial: AMBA PL011 UART driver687worker1 # [ 0.018024] thermal_sys: Registered thermal governor 'fair_share'688client # [ 0.017652] thermal_sys: Registered thermal governor 'user_space'689worker1 # [ 0.018026] thermal_sys: Registered thermal governor 'bang_bang'690worker2 # [ 0.024973] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1691client # [ 0.017655] thermal_sys: Registered thermal governor 'power_allocator'692worker2 # [ 0.025088] printk: console [ttyAMA0] enabled693worker1 # [ 0.018030] thermal_sys: Registered thermal governor 'step_wise'694client # [ 0.017677] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1695worker1 # [ 0.018036] thermal_sys: Registered thermal governor 'user_space'696client # [ 0.017686] cpuidle: using governor ladder697client # [ 0.017692] cpuidle: using governor menu698worker2 # [ 0.144403] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages699lb # [ 0.164335] usbcore: registered new interface driver usbfs700worker1 # [ 0.018039] thermal_sys: Registered thermal governor 'power_allocator'701lb # [ 0.164367] usbcore: registered new interface driver hub702worker2 # [ 0.144424] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page703client # [ 0.017879] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers.704worker1 # [ 0.018064] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1705lb # [ 0.164385] usbcore: registered new device driver usb706client # [ 0.017894] ASID allocator initialised with 65536 entries707worker1 # [ 0.018072] cpuidle: using governor ladder708worker2 # [ 0.144430] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages709lb # [ 0.164702] pps_core: LinuxPPS API ver. 1 registered710client # [ 0.019041] Serial: AMBA PL011 UART driver711worker1 # [ 0.018078] cpuidle: using governor menu712worker2 # [ 0.144434] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page713worker1 # [ 0.018276] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers.714lb # [ 0.164708] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>715client # [ 0.024121] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1716worker2 # [ 0.144438] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages717worker1 # [ 0.018292] ASID allocator initialised with 65536 entries718lb # [ 0.164718] PTP clock support registered719client # [ 0.024239] printk: console [ttyAMA0] enabled720worker2 # [ 0.144443] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page721lb # [ 0.164759] EDAC MC: Ver: 3.0.0722worker1 # [ 0.019432] Serial: AMBA PL011 UART driver723worker2 # [ 0.144447] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages724lb # [ 0.169222] scmi_core: SCMI protocol bus registered725worker1 # [ 0.024665] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1726client # [ 0.144192] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages727worker2 # [ 0.144451] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page728worker1 # [ 0.024782] printk: console [ttyAMA0] enabled729client # [ 0.144211] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page730lb # [ 0.170154] FPGA manager framework731lb # [ 0.171108] vgaarb: loaded732client # [ 0.144216] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages733worker1 # [ 0.144459] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages734client # [ 0.144220] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page735worker1 # [ 0.144481] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page736worker2 # [ 0.151649] fbcon: Taking over console737lb # [ 0.171756] clocksource: Switched to clocksource arch_sys_counter738client # [ 0.144225] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages739worker2 # [ 0.151665] ACPI: Interpreter disabled.740worker1 # [ 0.144487] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages741lb # [ 0.172361] VFS: Disk quotas dquot_6.6.0742client # [ 0.144229] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page743worker2 # [ 0.153495] iommu: Default domain type: Translated744lb # [ 0.172395] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)745worker1 # [ 0.144491] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page746client # [ 0.144233] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages747worker2 # [ 0.153505] iommu: DMA domain TLB invalidation policy: strict mode748worker1 # [ 0.144495] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages749lb # [ 0.174658] netfs: FS-Cache loaded750client # [ 0.144237] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page751lb # [ 0.174787] pnp: PnP ACPI: disabled752worker2 # [ 0.155209] SCSI subsystem initialized753worker1 # [ 0.144499] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page754worker1 # [ 0.144504] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages755worker1 # [ 0.144508] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page756client # [ 0.151501] fbcon: Taking over console757client # [ 0.151515] ACPI: Interpreter disabled.758client # [ 0.153311] iommu: Default domain type: Translated759client # [ 0.153320] iommu: DMA domain TLB invalidation policy: strict mode760worker1 # [ 0.151886] fbcon: Taking over console761worker1 # [ 0.151901] ACPI: Interpreter disabled.762client # [ 0.155033] SCSI subsystem initialized763worker1 # [ 0.153750] iommu: Default domain type: Translated764worker1 # [ 0.153760] iommu: DMA domain TLB invalidation policy: strict mode765client # [ 0.155714] usbcore: registered new interface driver usbfs766client # [ 0.155749] usbcore: registered new interface driver hub767worker1 # [ 0.155484] SCSI subsystem initialized768client # [ 0.155768] usbcore: registered new device driver usb769client # [ 0.156029] pps_core: LinuxPPS API ver. 1 registered770worker2 # [ 0.162695] usbcore: registered new interface driver usbfs771lb # [ 0.180580] NET: Registered PF_INET protocol family772worker2 # [ 0.162745] usbcore: registered new interface driver hub773client # [ 0.156035] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>774worker2 # [ 0.162763] usbcore: registered new device driver usb775lb # [ 0.180869] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)776client # [ 0.156045] PTP clock support registered777worker2 # [ 0.163033] pps_core: LinuxPPS API ver. 1 registered778client # [ 0.156105] EDAC MC: Ver: 3.0.0779worker2 # [ 0.163038] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>780worker2 # [ 0.163048] PTP clock support registered781worker2 # [ 0.163097] EDAC MC: Ver: 3.0.0782worker2 # [ 0.167762] scmi_core: SCMI protocol bus registered783worker2 # [ 0.168760] FPGA manager framework784worker2 # [ 0.169728] vgaarb: loaded785worker1 # [ 0.163094] usbcore: registered new interface driver usbfs786worker1 # [ 0.163129] usbcore: registered new interface driver hub787worker2 # [ 0.170402] clocksource: Switched to clocksource arch_sys_counter788worker1 # [ 0.163148] usbcore: registered new device driver usb789worker2 # [ 0.171036] VFS: Disk quotas dquot_6.6.0790worker1 # [ 0.163423] pps_core: LinuxPPS API ver. 1 registered791worker2 # [ 0.171068] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)792client # [ 0.167540] scmi_core: SCMI protocol bus registered793worker1 # [ 0.163430] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>794worker2 # [ 0.173499] netfs: FS-Cache loaded795worker1 # [ 0.163440] PTP clock support registered796worker2 # [ 0.173598] pnp: PnP ACPI: disabled797client # [ 0.168603] FPGA manager framework798worker1 # [ 0.163488] EDAC MC: Ver: 3.0.0799client # [ 0.169555] vgaarb: loaded800worker1 # [ 0.168119] scmi_core: SCMI protocol bus registered801worker1 # [ 0.169081] FPGA manager framework802client # [ 0.170208] clocksource: Switched to clocksource arch_sys_counter803worker1 # [ 0.170018] vgaarb: loaded804client # [ 0.170817] VFS: Disk quotas dquot_6.6.0805client # [ 0.170849] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)806worker1 # [ 0.170661] clocksource: Switched to clocksource arch_sys_counter807client # [ 0.173284] netfs: FS-Cache loaded808worker1 # [ 0.171253] VFS: Disk quotas dquot_6.6.0809client # [ 0.173610] pnp: PnP ACPI: disabled810worker1 # [ 0.171278] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)811worker1 # [ 0.173613] netfs: FS-Cache loaded812worker1 # [ 0.173728] pnp: PnP ACPI: disabled813worker2 # [ 0.179704] NET: Registered PF_INET protocol family814worker2 # [ 0.179964] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)815client # [ 0.179466] NET: Registered PF_INET protocol family816client # [ 0.179738] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)817worker1 # [ 0.179635] NET: Registered PF_INET protocol family818worker1 # [ 0.179900] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)819lb # [ 0.210371] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)820lb # [ 0.210423] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)821lb # [ 0.210455] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)822lb # [ 0.210528] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)823lb # [ 0.210671] TCP: Hash tables configured (established 16384 bind 16384)824lb # [ 0.210775] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)825lb # [ 0.210812] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)826lb # [ 0.210848] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)827lb # [ 0.210953] NET: Registered PF_UNIX/PF_LOCAL protocol family828lb # [ 0.210976] NET: Registered PF_XDP protocol family829lb # [ 0.210994] PCI: CLS 0 bytes, default 64830lb # [ 0.211252] Trying to unpack rootfs image as initramfs...831lb # [ 0.229440] kvm [1]: HYP mode not available832worker2 # [ 0.209520] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)833worker2 # [ 0.209575] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)834worker2 # [ 0.209610] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)835worker2 # [ 0.209692] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)836worker2 # [ 0.209834] TCP: Hash tables configured (established 16384 bind 16384)837worker2 # [ 0.209923] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)838client # [ 0.209387] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)839worker2 # [ 0.209959] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)840client # [ 0.209450] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)841worker2 # [ 0.209997] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)842client # [ 0.209485] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)843worker2 # [ 0.210088] NET: Registered PF_UNIX/PF_LOCAL protocol family844worker1 # [ 0.209735] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)845worker2 # [ 0.210133] NET: Registered PF_XDP protocol family846client # [ 0.209577] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)847worker1 # [ 0.209790] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)848worker2 # [ 0.210151] PCI: CLS 0 bytes, default 64849client # [ 0.209743] TCP: Hash tables configured (established 16384 bind 16384)850worker2 # [ 0.210429] Trying to unpack rootfs image as initramfs...851worker1 # [ 0.209820] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)852client # [ 0.209833] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)853worker1 # [ 0.209910] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)854client # [ 0.209868] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)855worker1 # [ 0.210054] TCP: Hash tables configured (established 16384 bind 16384)856client # [ 0.209903] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)857worker1 # [ 0.210147] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)858client # [ 0.210013] NET: Registered PF_UNIX/PF_LOCAL protocol family859worker1 # [ 0.210185] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)860client # [ 0.210053] NET: Registered PF_XDP protocol family861client # [ 0.210071] PCI: CLS 0 bytes, default 64862worker1 # [ 0.210222] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)863client # [ 0.210305] Trying to unpack rootfs image as initramfs...864worker1 # [ 0.210321] NET: Registered PF_UNIX/PF_LOCAL protocol family865worker1 # [ 0.210351] NET: Registered PF_XDP protocol family866worker1 # [ 0.210369] PCI: CLS 0 bytes, default 64867worker2 # [ 0.228094] kvm [1]: HYP mode not available868worker1 # [ 0.210639] Trying to unpack rootfs image as initramfs...869client # [ 0.228084] kvm [1]: HYP mode not available870worker1 # [ 0.228643] kvm [1]: HYP mode not available871lb # [ 0.324381] Initialise system trusted keyrings872lb # [ 0.325131] workingset: timestamp_bits=42 max_order=19 bucket_order=0873lb # [ 0.326332] squashfs: version 4.0 (2009/01/31) Phillip Lougher874lb # [ 0.327099] 9p: Installing v9fs 9p2000 file system support875worker2 # [ 0.326829] Initialise system trusted keyrings876worker2 # [ 0.327600] workingset: timestamp_bits=42 max_order=19 bucket_order=0877worker2 # [ 0.328941] squashfs: version 4.0 (2009/01/31) Phillip Lougher878worker2 # [ 0.329706] 9p: Installing v9fs 9p2000 file system support879client # [ 0.325932] Initialise system trusted keyrings880lb # [ 0.351017] Key type asymmetric registered881lb # [ 0.351048] Asymmetric key parser 'x509' registered882client # [ 0.326748] workingset: timestamp_bits=42 max_order=19 bucket_order=0883lb # [ 0.351126] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242)884client # [ 0.328019] squashfs: version 4.0 (2009/01/31) Phillip Lougher885client # [ 0.328770] 9p: Installing v9fs 9p2000 file system support886worker1 # [ 0.328966] Initialise system trusted keyrings887worker1 # [ 0.329738] workingset: timestamp_bits=42 max_order=19 bucket_order=0888worker1 # [ 0.331045] squashfs: version 4.0 (2009/01/31) Phillip Lougher889worker1 # [ 0.331813] 9p: Installing v9fs 9p2000 file system support890lb # [ 0.359818] io scheduler mq-deadline registered891lb # [ 0.359840] io scheduler kyber registered892lb # [ 0.365000] pl061_gpio 9030000.pl061: PL061 GPIO chip registered893worker2 # [ 0.350387] Key type asymmetric registered894lb # [ 0.366278] ledtrig-cpu: registered to indicate activity on CPUs895lb # [ 0.366639] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:896lb # [ 0.366655] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000897lb # [ 0.366676] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000898lb # [ 0.366685] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000899lb # [ 0.366736] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits900client # [ 0.352744] Key type asymmetric registered901lb # [ 0.366758] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]902client # [ 0.352776] Asymmetric key parser 'x509' registered903worker2 # [ 0.358442] Asymmetric key parser 'x509' registered904lb # [ 0.366836] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00905worker2 # [ 0.358529] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242)906client # [ 0.352857] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242)907lb # [ 0.366846] pci_bus 0000:00: root bus resource [bus 00-ff]908worker2 # [ 0.360129] io scheduler mq-deadline registered909lb # [ 0.366852] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]910worker2 # [ 0.360141] io scheduler kyber registered911lb # [ 0.366857] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]912lb # [ 0.366862] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]913lb # [ 0.366919] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint914lb # [ 0.367365] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint915lb # [ 0.367548] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f]916lb # [ 0.367564] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff]917lb # [ 0.367593] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]918worker1 # [ 0.359763] Key type asymmetric registered919lb # [ 0.367610] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref]920client # [ 0.358258] io scheduler mq-deadline registered921worker1 # [ 0.359791] Asymmetric key parser 'x509' registered922client # [ 0.358275] io scheduler kyber registered923lb # [ 0.368070] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint924worker1 # [ 0.359866] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242)925lb # [ 0.368251] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f]926worker1 # [ 0.361968] io scheduler mq-deadline registered927lb # [ 0.368267] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff]928worker1 # [ 0.361980] io scheduler kyber registered929lb # [ 0.368296] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]930lb # [ 0.368742] pci 0000:00:03.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint931worker2 # [ 0.370558] pl061_gpio 9030000.pl061: PL061 GPIO chip registered932lb # [ 0.368921] pci 0000:00:03.0: BAR 0 [io 0x0000-0x003f]933client # [ 0.366350] pl061_gpio 9030000.pl061: PL061 GPIO chip registered934lb # [ 0.368937] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff]935lb # [ 0.368965] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]936lb # [ 0.369407] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint937worker2 # [ 0.372004] ledtrig-cpu: registered to indicate activity on CPUs938lb # [ 0.369586] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f]939worker2 # [ 0.372403] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:940lb # [ 0.369602] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff]941lb # [ 0.369634] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]942worker2 # [ 0.372420] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000943lb # [ 0.370089] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint944worker2 # [ 0.372439] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000945lb # [ 0.370268] pci 0000:00:05.0: BAR 0 [io 0x0000-0x001f]946lb # [ 0.370283] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff]947worker2 # [ 0.372448] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000948client # [ 0.367708] ledtrig-cpu: registered to indicate activity on CPUs949lb # [ 0.370312] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]950worker2 # [ 0.372469] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits951client # [ 0.368076] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:952lb # [ 0.370791] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint953worker2 # [ 0.372494] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]954lb # [ 0.370974] pci 0000:00:06.0: BAR 0 [io 0x0000-0x007f]955client # [ 0.368092] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000956worker2 # [ 0.372574] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00957lb # [ 0.370990] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff]958client # [ 0.368104] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000959worker2 # [ 0.372584] pci_bus 0000:00: root bus resource [bus 00-ff]960worker1 # [ 0.374943] pl061_gpio 9030000.pl061: PL061 GPIO chip registered961lb # [ 0.371019] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]962worker2 # [ 0.372590] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]963client # [ 0.368112] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000964lb # [ 0.371458] pci 0000:00:07.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint965worker2 # [ 0.372595] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]966lb # [ 0.371641] pci 0000:00:07.0: BAR 0 [io 0x0000-0x001f]967client # [ 0.368139] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits968worker2 # [ 0.372600] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]969lb # [ 0.371656] pci 0000:00:07.0: BAR 1 [mem 0x00000000-0x00000fff]970client # [ 0.368161] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]971worker1 # [ 0.376343] ledtrig-cpu: registered to indicate activity on CPUs972worker2 # [ 0.372693] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint973lb # [ 0.371686] pci 0000:00:07.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]974client # [ 0.368258] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00975lb # [ 0.371702] pci 0000:00:07.0: ROM [mem 0x00000000-0x0003ffff pref]976worker1 # [ 0.376725] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:977worker2 # [ 0.373136] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint978client # [ 0.368267] pci_bus 0000:00: root bus resource [bus 00-ff]979worker2 # [ 0.373324] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f]980worker1 # [ 0.376742] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000981client # [ 0.368273] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]982worker2 # [ 0.373341] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff]983worker1 # [ 0.376754] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000984client # [ 0.368278] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]985worker2 # [ 0.373370] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]986client # [ 0.368283] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]987worker1 # [ 0.376763] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000988worker2 # [ 0.373387] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref]989client # [ 0.368384] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint990worker2 # [ 0.373844] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint991worker1 # [ 0.376784] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits992worker2 # [ 0.374029] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f]993client # [ 0.368820] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint994worker1 # [ 0.376806] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]995worker2 # [ 0.374045] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff]996client # [ 0.369006] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f]997worker1 # [ 0.376897] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00998worker2 # [ 0.374075] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]999client # [ 0.369022] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff]1000worker1 # [ 0.376907] pci_bus 0000:00: root bus resource [bus 00-ff]1001client # [ 0.369051] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1002worker1 # [ 0.376912] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]1003client # [ 0.369067] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref]1004worker1 # [ 0.376917] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]1005worker2 # [ 0.393177] pci 0000:00:03.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1006client # [ 0.369520] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1007worker1 # [ 0.376923] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]1008worker2 # [ 0.393365] pci 0000:00:03.0: BAR 0 [io 0x0000-0x003f]1009client # [ 0.369733] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f]1010worker2 # [ 0.393381] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff]1011lb # [ 0.412289] pci 0000:00:08.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint1012worker1 # [ 0.377017] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint1013client # [ 0.369750] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff]1014worker2 # [ 0.393410] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1015lb # [ 0.412500] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff]1016worker1 # [ 0.377450] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1017client # [ 0.369779] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1018lb # [ 0.412532] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1019worker1 # [ 0.377635] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f]1020worker2 # [ 0.393871] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1021client # [ 0.370262] pci 0000:00:03.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1022worker2 # [ 0.394053] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f]1023worker1 # [ 0.377651] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff]1024lb # [ 0.412993] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1025client # [ 0.370447] pci 0000:00:03.0: BAR 0 [io 0x0000-0x003f]1026worker2 # [ 0.394069] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff]1027worker1 # [ 0.377681] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1028lb # [ 0.413180] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff]1029client # [ 0.370462] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff]1030worker2 # [ 0.394100] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1031worker1 # [ 0.377697] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref]1032lb # [ 0.413210] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1033client # [ 0.370492] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1034worker1 # [ 0.378163] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1035lb # [ 0.413604] pci 0000:00:0a.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint1036client # [ 0.370948] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1037worker1 # [ 0.378348] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f]1038lb # [ 0.413786] pci 0000:00:0a.0: BAR 0 [mem 0x00000000-0x00000fff]1039client # [ 0.371134] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f]1040worker1 # [ 0.378363] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff]1041client # [ 0.371150] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff]1042lb # [ 0.414029] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1043worker1 # [ 0.378393] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1044lb # [ 0.414330] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f]1045client # [ 0.371182] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1046lb # [ 0.414347] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff]1047client # [ 0.371667] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1048lb # [ 0.414378] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1049client # [ 0.371854] pci 0000:00:05.0: BAR 0 [io 0x0000-0x001f]1050client # [ 0.371870] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff]1051worker1 # [ 0.397045] pci 0000:00:03.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1052lb # [ 0.414858] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1053lb # [ 0.415043] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f]1054worker1 # [ 0.397236] pci 0000:00:03.0: BAR 0 [io 0x0000-0x003f]1055client # [ 0.371899] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1056worker1 # [ 0.397252] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff]1057lb # [ 0.415058] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff]1058client # [ 0.372350] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1059worker1 # [ 0.397281] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1060lb # [ 0.415088] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1061client # [ 0.372533] pci 0000:00:06.0: BAR 0 [io 0x0000-0x007f]1062client # [ 0.372549] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff]1063lb # [ 0.415672] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned1064worker1 # [ 0.397733] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1065worker1 # [ 0.397916] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f]1066client # [ 0.372578] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1067lb # [ 0.415683] pci 0000:00:07.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned1068worker1 # [ 0.397931] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff]1069lb # [ 0.415689] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned1070client # [ 0.373027] pci 0000:00:07.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1071worker1 # [ 0.397963] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1072client # [ 0.373211] pci 0000:00:07.0: BAR 0 [io 0x0000-0x001f]1073lb # [ 0.415735] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned1074client # [ 0.373227] pci 0000:00:07.0: BAR 1 [mem 0x00000000-0x00000fff]1075worker1 # [ 0.398420] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1076worker1 # [ 0.398604] pci 0000:00:05.0: BAR 0 [io 0x0000-0x001f]1077client # [ 0.373257] pci 0000:00:07.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1078worker1 # [ 0.398620] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff]1079client # [ 0.373273] pci 0000:00:07.0: ROM [mem 0x00000000-0x0003ffff pref]1080worker1 # [ 0.398649] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1081client # [ 0.373790] pci 0000:00:08.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint1082worker2 # [ 0.406637] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1083client # [ 0.373979] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff]1084worker2 # [ 0.406849] pci 0000:00:05.0: BAR 0 [io 0x0000-0x001f]1085client # [ 0.374008] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1086worker2 # [ 0.406867] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff]1087worker2 # [ 0.406897] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1088worker2 # [ 0.407355] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1089worker2 # [ 0.407541] pci 0000:00:06.0: BAR 0 [io 0x0000-0x007f]1090worker2 # [ 0.407556] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff]1091lb # [ 0.435831] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned1092worker2 # [ 0.407586] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1093lb # [ 0.435887] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned1094worker2 # [ 0.408051] pci 0000:00:07.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1095worker2 # [ 0.408237] pci 0000:00:07.0: BAR 0 [io 0x0000-0x001f]1096lb # [ 0.435934] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned1097worker2 # [ 0.408253] pci 0000:00:07.0: BAR 1 [mem 0x00000000-0x00000fff]1098lb # [ 0.435982] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned1099worker2 # [ 0.408282] pci 0000:00:07.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1100lb # [ 0.436031] pci 0000:00:07.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned1101worker2 # [ 0.408299] pci 0000:00:07.0: ROM [mem 0x00000000-0x0003ffff pref]1102lb # [ 0.436083] pci 0000:00:08.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned1103worker2 # [ 0.408791] pci 0000:00:08.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint1104worker2 # [ 0.408985] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff]1105lb # [ 0.436130] pci 0000:00:09.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned1106worker2 # [ 0.409015] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1107lb # [ 0.436178] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned1108client # [ 0.418586] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1109worker2 # [ 0.409476] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1110lb # [ 0.436277] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned1111worker2 # [ 0.409664] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff]1112client # [ 0.418796] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff]1113lb # [ 0.436324] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned1114worker2 # [ 0.409694] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1115client # [ 0.418826] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1116lb # [ 0.436347] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned1117worker1 # [ 0.415166] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1118worker2 # [ 0.410083] pci 0000:00:0a.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint1119client # [ 0.419227] pci 0000:00:0a.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint1120worker1 # [ 0.415366] pci 0000:00:06.0: BAR 0 [io 0x0000-0x007f]1121lb # [ 0.436372] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned1122client # [ 0.419409] pci 0000:00:0a.0: BAR 0 [mem 0x00000000-0x00000fff]1123worker2 # [ 0.410269] pci 0000:00:0a.0: BAR 0 [mem 0x00000000-0x00000fff]1124worker1 # [ 0.415382] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff]1125lb # [ 0.436395] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned1126worker2 # [ 0.410534] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1127client # [ 0.419657] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1128worker1 # [ 0.415411] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1129lb # [ 0.436421] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned1130worker2 # [ 0.410828] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f]1131client # [ 0.419962] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f]1132worker1 # [ 0.415867] pci 0000:00:07.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1133lb # [ 0.436443] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned1134worker2 # [ 0.410846] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff]1135client # [ 0.419979] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff]1136worker1 # [ 0.416057] pci 0000:00:07.0: BAR 0 [io 0x0000-0x001f]1137lb # [ 0.436465] pci 0000:00:07.0: BAR 1 [mem 0x10086000-0x10086fff]: assigned1138worker2 # [ 0.410875] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1139client # [ 0.420008] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1140worker1 # [ 0.416073] pci 0000:00:07.0: BAR 1 [mem 0x00000000-0x00000fff]1141lb # [ 0.436487] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned1142worker2 # [ 0.411333] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1143client # [ 0.420468] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1144worker1 # [ 0.416102] pci 0000:00:07.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1145lb # [ 0.436509] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned1146worker2 # [ 0.411526] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f]1147client # [ 0.420652] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f]1148worker1 # [ 0.416118] pci 0000:00:07.0: ROM [mem 0x00000000-0x0003ffff pref]1149lb # [ 0.436532] pci 0000:00:0a.0: BAR 0 [mem 0x10089000-0x10089fff]: assigned1150worker2 # [ 0.411542] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff]1151client # [ 0.420668] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff]1152worker1 # [ 0.416605] pci 0000:00:08.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint1153lb # [ 0.436555] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned1154worker2 # [ 0.411572] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1155client # [ 0.420698] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1156worker1 # [ 0.416792] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff]1157lb # [ 0.436577] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned1158worker2 # [ 0.412177] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned1159client # [ 0.421286] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned1160worker1 # [ 0.416822] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1161lb # [ 0.436599] pci 0000:00:06.0: BAR 0 [io 0x1000-0x107f]: assigned1162client # [ 0.421297] pci 0000:00:07.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned1163worker2 # [ 0.412189] pci 0000:00:07.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned1164lb # [ 0.436620] pci 0000:00:03.0: BAR 0 [io 0x1080-0x10bf]: assigned1165worker1 # [ 0.417269] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1166worker2 # [ 0.412195] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned1167client # [ 0.421303] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned1168lb # [ 0.436642] pci 0000:00:0b.0: BAR 0 [io 0x10c0-0x10ff]: assigned1169worker1 # [ 0.417455] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff]1170lb # [ 0.436664] pci 0000:00:01.0: BAR 0 [io 0x1100-0x111f]: assigned1171worker2 # [ 0.412240] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned1172worker1 # [ 0.417484] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1173client # [ 0.421349] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned1174lb # [ 0.436685] pci 0000:00:02.0: BAR 0 [io 0x1120-0x113f]: assigned1175worker2 # [ 0.412289] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned1176client # [ 0.421396] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned1177worker1 # [ 0.417868] pci 0000:00:0a.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint1178lb # [ 0.436707] pci 0000:00:04.0: BAR 0 [io 0x1140-0x115f]: assigned1179worker1 # [ 0.418052] pci 0000:00:0a.0: BAR 0 [mem 0x00000000-0x00000fff]1180worker2 # [ 0.412337] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned1181client # [ 0.421443] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned1182lb # [ 0.436728] pci 0000:00:05.0: BAR 0 [io 0x1160-0x117f]: assigned1183worker1 # [ 0.418299] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1184worker2 # [ 0.412386] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned1185lb # [ 0.436750] pci 0000:00:07.0: BAR 0 [io 0x1180-0x119f]: assigned1186client # [ 0.421490] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned1187worker1 # [ 0.418598] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f]1188lb # [ 0.436772] pci 0000:00:0c.0: BAR 0 [io 0x11a0-0x11bf]: assigned1189worker2 # [ 0.412433] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned1190client # [ 0.421537] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned1191worker1 # [ 0.418615] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff]1192lb # [ 0.436800] pci_bus 0000:00: resource 4 [io 0x0000-0xffff]1193worker2 # [ 0.412479] pci 0000:00:07.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned1194worker1 # [ 0.418645] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1195client # [ 0.421585] pci 0000:00:07.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned1196lb # [ 0.436810] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff]1197worker2 # [ 0.412527] pci 0000:00:08.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned1198lb # [ 0.436815] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff]1199client # [ 0.421651] pci 0000:00:08.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned1200worker1 # [ 0.419115] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1201lb # [ 0.437973] pci 0000:00:0a.0: enabling device (0000 -> 0002)1202worker1 # [ 0.419297] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f]1203worker2 # [ 0.412574] pci 0000:00:09.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned1204client # [ 0.421699] pci 0000:00:09.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned1205worker1 # [ 0.419312] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff]1206worker2 # [ 0.412621] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned1207client # [ 0.421745] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned1208worker1 # [ 0.419342] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1209worker2 # [ 0.412695] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned1210client # [ 0.421821] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned1211worker1 # [ 0.419942] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned1212worker2 # [ 0.412741] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned1213client # [ 0.421866] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned1214worker1 # [ 0.419953] pci 0000:00:07.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned1215worker2 # [ 0.412763] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned1216client # [ 0.421888] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned1217worker1 # [ 0.419959] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned1218worker2 # [ 0.412785] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned1219client # [ 0.421909] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned1220worker1 # [ 0.420003] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned1221worker2 # [ 0.412807] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned1222client # [ 0.421931] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned1223worker1 # [ 0.420061] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned1224worker2 # [ 0.412833] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned1225client # [ 0.421955] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned1226worker2 # [ 0.412855] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned1227worker1 # [ 0.420108] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned1228client # [ 0.421977] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned1229worker2 # [ 0.412877] pci 0000:00:07.0: BAR 1 [mem 0x10086000-0x10086fff]: assigned1230client # [ 0.421998] pci 0000:00:07.0: BAR 1 [mem 0x10086000-0x10086fff]: assigned1231worker1 # [ 0.420154] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned1232worker2 # [ 0.412903] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned1233client # [ 0.422020] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned1234worker1 # [ 0.420200] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned1235worker2 # [ 0.412925] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned1236client # [ 0.422042] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned1237worker1 # [ 0.420246] pci 0000:00:07.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned1238worker2 # [ 0.412949] pci 0000:00:0a.0: BAR 0 [mem 0x10089000-0x10089fff]: assigned1239client # [ 0.422063] pci 0000:00:0a.0: BAR 0 [mem 0x10089000-0x10089fff]: assigned1240worker2 # [ 0.412972] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned1241worker1 # [ 0.420292] pci 0000:00:08.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned1242client # [ 0.422086] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned1243worker2 # [ 0.412995] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned1244worker1 # [ 0.420338] pci 0000:00:09.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned1245client # [ 0.422108] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned1246worker2 # [ 0.413017] pci 0000:00:06.0: BAR 0 [io 0x1000-0x107f]: assigned1247client # [ 0.422129] pci 0000:00:06.0: BAR 0 [io 0x1000-0x107f]: assigned1248worker1 # [ 0.420384] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned1249worker2 # [ 0.413039] pci 0000:00:03.0: BAR 0 [io 0x1080-0x10bf]: assigned1250client # [ 0.422151] pci 0000:00:03.0: BAR 0 [io 0x1080-0x10bf]: assigned1251worker1 # [ 0.420446] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned1252worker2 # [ 0.413061] pci 0000:00:0b.0: BAR 0 [io 0x10c0-0x10ff]: assigned1253client # [ 0.422171] pci 0000:00:0b.0: BAR 0 [io 0x10c0-0x10ff]: assigned1254worker2 # [ 0.413084] pci 0000:00:01.0: BAR 0 [io 0x1100-0x111f]: assigned1255worker1 # [ 0.420491] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned1256client # [ 0.422193] pci 0000:00:01.0: BAR 0 [io 0x1100-0x111f]: assigned1257worker2 # [ 0.413106] pci 0000:00:02.0: BAR 0 [io 0x1120-0x113f]: assigned1258worker1 # [ 0.420513] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned1259worker2 # [ 0.413128] pci 0000:00:04.0: BAR 0 [io 0x1140-0x115f]: assigned1260worker1 # [ 0.420533] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned1261worker2 # [ 0.413149] pci 0000:00:05.0: BAR 0 [io 0x1160-0x117f]: assigned1262worker1 # [ 0.420555] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned1263worker2 # [ 0.413171] pci 0000:00:07.0: BAR 0 [io 0x1180-0x119f]: assigned1264worker1 # [ 0.420579] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned1265worker2 # [ 0.413193] pci 0000:00:0c.0: BAR 0 [io 0x11a0-0x11bf]: assigned1266worker1 # [ 0.420600] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned1267worker2 # [ 0.413220] pci_bus 0000:00: resource 4 [io 0x0000-0xffff]1268worker1 # [ 0.420621] pci 0000:00:07.0: BAR 1 [mem 0x10086000-0x10086fff]: assigned1269worker2 # [ 0.413230] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff]1270worker1 # [ 0.420643] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned1271worker2 # [ 0.413235] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff]1272worker2 # [ 0.414398] pci 0000:00:0a.0: enabling device (0000 -> 0002)1273worker1 # [ 0.420664] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned1274worker1 # [ 0.420686] pci 0000:00:0a.0: BAR 0 [mem 0x10089000-0x10089fff]: assigned1275worker1 # [ 0.420708] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned1276worker1 # [ 0.420730] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned1277client # [ 0.462255] pci 0000:00:02.0: BAR 0 [io 0x1120-0x113f]: assigned1278worker1 # [ 0.420751] pci 0000:00:06.0: BAR 0 [io 0x1000-0x107f]: assigned1279client # [ 0.462289] pci 0000:00:04.0: BAR 0 [io 0x1140-0x115f]: assigned1280worker1 # [ 0.420773] pci 0000:00:03.0: BAR 0 [io 0x1080-0x10bf]: assigned1281lb # [ 0.490180] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)1282client # [ 0.462312] pci 0000:00:05.0: BAR 0 [io 0x1160-0x117f]: assigned1283worker1 # [ 0.420794] pci 0000:00:0b.0: BAR 0 [io 0x10c0-0x10ff]: assigned1284client # [ 0.462334] pci 0000:00:07.0: BAR 0 [io 0x1180-0x119f]: assigned1285worker1 # [ 0.420815] pci 0000:00:01.0: BAR 0 [io 0x1100-0x111f]: assigned1286client # [ 0.462357] pci 0000:00:0c.0: BAR 0 [io 0x11a0-0x11bf]: assigned1287worker1 # [ 0.420836] pci 0000:00:02.0: BAR 0 [io 0x1120-0x113f]: assigned1288client # [ 0.462387] pci_bus 0000:00: resource 4 [io 0x0000-0xffff]1289worker1 # [ 0.420857] pci 0000:00:04.0: BAR 0 [io 0x1140-0x115f]: assigned1290client # [ 0.462396] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff]1291worker1 # [ 0.420879] pci 0000:00:05.0: BAR 0 [io 0x1160-0x117f]: assigned1292client # [ 0.462401] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff]1293worker1 # [ 0.420900] pci 0000:00:07.0: BAR 0 [io 0x1180-0x119f]: assigned1294worker2 # [ 0.475000] pci 0000:00:0a.0: quirk_usb_early_handoff+0x0/0xa60 took 59186 usecs1295client # [ 0.463593] pci 0000:00:0a.0: enabling device (0000 -> 0002)1296worker1 # [ 0.420921] pci 0000:00:0c.0: BAR 0 [io 0x11a0-0x11bf]: assigned1297worker1 # [ 0.420948] pci_bus 0000:00: resource 4 [io 0x0000-0xffff]1298worker1 # [ 0.420957] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff]1299worker1 # [ 0.420962] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff]1300worker1 # [ 0.422143] pci 0000:00:0a.0: enabling device (0000 -> 0002)1301lb # [ 0.500446] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003)1302lb # [ 0.503045] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003)1303worker1 # [ 0.478738] pci 0000:00:0a.0: quirk_usb_early_handoff+0x0/0xa60 took 55272 usecs1304lb # [ 0.506909] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003)1305lb # [ 0.513079] virtio-pci 0000:00:05.0: enabling device (0000 -> 0003)1306worker2 # [ 0.495971] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)1307client # [ 0.490272] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)1308worker2 # [ 0.498348] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003)1309client # [ 0.492494] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003)1310lb # [ 0.524353] virtio-pci 0000:00:06.0: enabling device (0000 -> 0003)1311lb # [ 0.526679] virtio-pci 0000:00:07.0: enabling device (0000 -> 0003)1312worker1 # [ 0.498947] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)1313worker1 # [ 0.501228] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003)1314worker2 # [ 0.509306] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003)1315client # [ 0.502454] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003)1316client # [ 0.505052] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003)1317worker2 # [ 0.515461] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003)1318worker2 # [ 0.517541] virtio-pci 0000:00:05.0: enabling device (0000 -> 0003)1319lb # [ 0.538018] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002)1320worker1 # [ 0.511752] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003)1321worker1 # [ 0.513831] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003)1322client # [ 0.515155] virtio-pci 0000:00:05.0: enabling device (0000 -> 0003)1323client # [ 0.517221] virtio-pci 0000:00:06.0: enabling device (0000 -> 0003)1324worker2 # [ 0.527880] virtio-pci 0000:00:06.0: enabling device (0000 -> 0003)1325worker2 # [ 0.530048] virtio-pci 0000:00:07.0: enabling device (0000 -> 0003)1326worker1 # [ 0.524207] virtio-pci 0000:00:05.0: enabling device (0000 -> 0003)1327worker1 # [ 0.526324] virtio-pci 0000:00:06.0: enabling device (0000 -> 0003)1328client # [ 0.528787] virtio-pci 0000:00:07.0: enabling device (0000 -> 0003)1329lb # [ 0.548027] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002)1330lb # [ 0.551359] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003)1331lb # [ 0.554194] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003)1332client # [ 0.531844] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002)1333client # [ 0.534011] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002)1334worker2 # [ 0.541640] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002)1335worker1 # [ 0.537806] virtio-pci 0000:00:07.0: enabling device (0000 -> 0003)1336worker1 # [ 0.541012] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002)1337worker2 # [ 0.548210] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002)1338client # [ 0.543676] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003)1339client # [ 0.546093] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003)1340worker2 # [ 0.550256] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003)1341lb # [ 0.572569] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1342lb # [ 0.575332] msm_serial: driver initialized1343lb # [ 0.575489] SuperH (H)SCI(F) driver initialized1344lb # [ 0.575546] STM32 USART driver initialized1345worker1 # [ 0.546848] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002)1346worker1 # [ 0.549996] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003)1347worker2 # [ 0.559944] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003)1348client # [ 0.563429] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1349worker2 # [ 0.573535] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1350worker1 # [ 0.568320] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003)1351worker2 # [ 0.576504] msm_serial: driver initialized1352worker2 # [ 0.576666] SuperH (H)SCI(F) driver initialized1353client # [ 0.571169] msm_serial: driver initialized1354worker2 # [ 0.576724] STM32 USART driver initialized1355client # [ 0.571319] SuperH (H)SCI(F) driver initialized1356client # [ 0.571371] STM32 USART driver initialized1357worker1 # [ 0.574217] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1358lb # [ 0.606490] loop: module loaded1359lb # [ 0.606721] virtio_blk virtio5: 1/0/0 default/read/poll queues1360worker1 # [ 0.580037] msm_serial: driver initialized1361lb # [ 0.607489] virtio_blk virtio5: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB)1362worker1 # [ 0.580207] SuperH (H)SCI(F) driver initialized1363worker1 # [ 0.580260] STM32 USART driver initialized1364lb # [ 0.615904] megasas: 07.734.00.00-rc11365lb # [ 0.616738] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff]1366lb # [ 0.619082] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001367lb # [ 0.619108] Intel/Sharp Extended Query Table at 0x00311368lb # [ 0.620738] Using buffer write method1369client # [ 0.600959] loop: module loaded1370client # [ 0.601151] virtio_blk virtio5: 1/0/0 default/read/poll queues1371lb # [ 0.620804] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff]1372client # [ 0.601902] virtio_blk virtio5: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB)1373lb # [ 0.622640] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001374lb # [ 0.622672] Intel/Sharp Extended Query Table at 0x00311375client # [ 0.606864] megasas: 07.734.00.00-rc11376client # [ 0.607551] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff]1377worker2 # [ 0.616224] loop: module loaded1378worker2 # [ 0.616431] virtio_blk virtio5: 1/0/0 default/read/poll queues1379worker2 # [ 0.617279] virtio_blk virtio5: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB)1380lb # [ 0.641254] Using buffer write method1381lb # [ 0.641296] Concatenating MTD devices:1382lb # [ 0.641300] (0): "0.flash"1383lb # [ 0.641305] (1): "0.flash"1384lb # [ 0.641308] into device "0.flash"1385worker1 # [ 0.614953] loop: module loaded1386worker1 # [ 0.615150] virtio_blk virtio5: 1/0/0 default/read/poll queues1387worker1 # [ 0.615962] virtio_blk virtio5: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB)1388client # [ 0.618927] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001389client # [ 0.618961] Intel/Sharp Extended Query Table at 0x00311390client # [ 0.620528] Using buffer write method1391client # [ 0.620604] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff]1392worker2 # [ 0.627240] megasas: 07.734.00.00-rc11393worker2 # [ 0.628121] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff]1394worker2 # [ 0.630455] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001395worker2 # [ 0.630480] Intel/Sharp Extended Query Table at 0x00311396worker1 # [ 0.623331] megasas: 07.734.00.00-rc11397worker2 # [ 0.632063] Using buffer write method1398worker1 # [ 0.624090] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff]1399worker2 # [ 0.632133] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff]1400worker1 # [ 0.625963] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001401worker1 # [ 0.625990] Intel/Sharp Extended Query Table at 0x00311402worker2 # [ 0.633808] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001403worker2 # [ 0.633833] Intel/Sharp Extended Query Table at 0x00311404worker1 # [ 0.627678] Using buffer write method1405worker1 # [ 0.627750] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff]1406client # [ 0.634958] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001407client # [ 0.635000] Intel/Sharp Extended Query Table at 0x00311408worker1 # [ 0.629651] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001409client # [ 0.636780] Using buffer write method1410client # [ 0.636810] Concatenating MTD devices:1411worker1 # [ 0.629675] Intel/Sharp Extended Query Table at 0x00311412client # [ 0.636814] (0): "0.flash"1413client # [ 0.636818] (1): "0.flash"1414client # [ 0.636821] into device "0.flash"1415worker2 # [ 0.647438] Using buffer write method1416worker2 # [ 0.647479] Concatenating MTD devices:1417worker2 # [ 0.647484] (0): "0.flash"1418worker2 # [ 0.647488] (1): "0.flash"1419worker2 # [ 0.647491] into device "0.flash"1420worker1 # [ 0.649338] Using buffer write method1421worker1 # [ 0.649381] Concatenating MTD devices:1422worker1 # [ 0.649386] (0): "0.flash"1423worker1 # [ 0.649391] (1): "0.flash"1424worker1 # [ 0.649394] into device "0.flash"1425lb # [ 0.884932] Freeing initrd memory: 26140K1426lb # [ 0.890959] tun: Universal TUN/TAP device driver, 1.61427lb # [ 0.894657] thunder_xcv, ver 1.01428lb # [ 0.894717] thunder_bgx, ver 1.01429lb # [ 0.894740] nicpf, ver 1.01430lb # [ 0.895293] e1000: Intel(R) PRO/1000 Network Driver1431lb # [ 0.895300] e1000: Copyright (c) 1999-2006 Intel Corporation.1432lb # [ 0.895323] e1000e: Intel(R) PRO/1000 Network Driver1433lb # [ 0.895332] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.1434lb # [ 0.895360] igb: Intel(R) Gigabit Ethernet Network Driver1435lb # [ 0.895366] igb: Copyright (c) 2007-2014 Intel Corporation.1436lb # [ 0.895389] igbvf: Intel(R) Gigabit Virtual Function Network Driver1437lb # [ 0.895395] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.1438lb # [ 0.895529] sky2: driver version 1.301439lb # [ 0.904284] usbcore: registered new interface driver usb-storage1440client # [ 0.879134] Freeing initrd memory: 26160K1441lb # [ 0.904380] usbcore: registered new interface driver usbserial_generic1442lb # [ 0.904394] usbserial: USB Serial support registered for generic1443lb # [ 0.904994] hv_vmbus: registering driver hyperv_keyboard1444lb # [ 0.906227] ehci-pci 0000:00:0a.0: EHCI Host Controller1445client # [ 0.885087] tun: Universal TUN/TAP device driver, 1.61446lb # [ 0.906252] ehci-pci 0000:00:0a.0: new USB bus registered, assigned bus number 11447lb # [ 0.906451] ehci-pci 0000:00:0a.0: irq 15, io mem 0x100890001448lb # [ 0.911938] rtc-pl031 9010000.pl031: registered as rtc01449lb # [ 0.911969] rtc-pl031 9010000.pl031: setting system clock to 2026-09-21T09:13:10 UTC (1789981990)1450lb # [ 0.912316] i2c_dev: i2c /dev entries driver1451client # [ 0.888711] thunder_xcv, ver 1.01452client # [ 0.888757] thunder_bgx, ver 1.01453client # [ 0.888780] nicpf, ver 1.01454client # [ 0.889317] e1000: Intel(R) PRO/1000 Network Driver1455client # [ 0.889325] e1000: Copyright (c) 1999-2006 Intel Corporation.1456client # [ 0.889349] e1000e: Intel(R) PRO/1000 Network Driver1457client # [ 0.889358] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.1458client # [ 0.889387] igb: Intel(R) Gigabit Ethernet Network Driver1459client # [ 0.889393] igb: Copyright (c) 2007-2014 Intel Corporation.1460lb # [ 0.917294] sdhci: Secure Digital Host Controller Interface driver1461worker1 # [ 0.891925] Freeing initrd memory: 26148K1462lb # [ 0.917306] sdhci: Copyright(c) Pierre Ossman1463client # [ 0.889416] igbvf: Intel(R) Gigabit Virtual Function Network Driver1464lb # [ 0.917579] Synopsys Designware Multimedia Card Interface Driver1465client # [ 0.889422] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.1466client # [ 0.889553] sky2: driver version 1.301467lb # [ 0.917960] sdhci-pltfm: SDHCI platform and OF driver helper1468worker2 # [ 0.902385] Freeing initrd memory: 26148K1469lb # [ 0.919691] hid: raw HID events driver (C) Jiri Kosina1470lb # [ 0.923035] ehci-pci 0000:00:0a.0: USB 2.0 started, EHCI 1.001471client # [ 0.898848] ehci-pci 0000:00:0a.0: EHCI Host Controller1472lb # [ 0.923427] hub 1-0:1.0: USB hub found1473worker1 # [ 0.898005] tun: Universal TUN/TAP device driver, 1.61474lb # [ 0.923446] hub 1-0:1.0: 6 ports detected1475client # [ 0.898895] ehci-pci 0000:00:0a.0: new USB bus registered, assigned bus number 11476client # [ 0.899087] ehci-pci 0000:00:0a.0: irq 15, io mem 0x100890001477lb # [ 0.925607] usbcore: registered new interface driver usbhid1478lb # [ 0.925619] usbhid: USB HID core driver1479client # [ 0.901540] usbcore: registered new interface driver usb-storage1480client # [ 0.901757] usbcore: registered new interface driver usbserial_generic1481client # [ 0.901776] usbserial: USB Serial support registered for generic1482worker2 # [ 0.908991] tun: Universal TUN/TAP device driver, 1.61483worker1 # [ 0.901788] thunder_xcv, ver 1.01484worker1 # [ 0.901833] thunder_bgx, ver 1.01485worker1 # [ 0.901855] nicpf, ver 1.01486client # [ 0.904608] hv_vmbus: registering driver hyperv_keyboard1487lb # [ 0.927962] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available1488worker1 # [ 0.902383] e1000: Intel(R) PRO/1000 Network Driver1489lb # [ 0.929516] drop_monitor: Initializing network drop monitor service1490worker1 # [ 0.902391] e1000: Copyright (c) 1999-2006 Intel Corporation.1491client # [ 0.906074] rtc-pl031 9010000.pl031: registered as rtc01492lb # [ 0.929705] NET: Registered PF_INET6 protocol family1493worker1 # [ 0.902415] e1000e: Intel(R) PRO/1000 Network Driver1494client # [ 0.906101] rtc-pl031 9010000.pl031: setting system clock to 2026-09-21T09:13:10 UTC (1789981990)1495worker1 # [ 0.902424] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.1496lb # [ 0.932707] Segment Routing with IPv61497worker1 # [ 0.902452] igb: Intel(R) Gigabit Ethernet Network Driver1498lb # [ 0.932726] In-situ OAM (IOAM) with IPv61499lb # [ 0.932769] NET: Registered PF_PACKET protocol family1500worker1 # [ 0.902458] igb: Copyright (c) 2007-2014 Intel Corporation.1501client # [ 0.908110] i2c_dev: i2c /dev entries driver1502worker2 # [ 0.912916] thunder_xcv, ver 1.01503worker1 # [ 0.902481] igbvf: Intel(R) Gigabit Virtual Function Network Driver1504worker2 # [ 0.912972] thunder_bgx, ver 1.01505worker2 # [ 0.913000] nicpf, ver 1.01506worker1 # [ 0.902487] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.1507worker2 # [ 0.913663] e1000: Intel(R) PRO/1000 Network Driver1508lb # [ 0.934396] 9pnet: Installing 9P2000 support1509worker1 # [ 0.902614] sky2: driver version 1.301510worker2 # [ 0.913670] e1000: Copyright (c) 1999-2006 Intel Corporation.1511worker2 # [ 0.913700] e1000e: Intel(R) PRO/1000 Network Driver1512worker2 # [ 0.913709] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.1513worker2 # [ 0.913744] igb: Intel(R) Gigabit Ethernet Network Driver1514worker2 # [ 0.913750] igb: Copyright (c) 2007-2014 Intel Corporation.1515client # [ 0.911351] sdhci: Secure Digital Host Controller Interface driver1516lb # [ 0.937097] Key type dns_resolver registered1517worker2 # [ 0.913778] igbvf: Intel(R) Gigabit Virtual Function Network Driver1518client # [ 0.911360] sdhci: Copyright(c) Pierre Ossman1519worker1 # [ 0.911217] usbcore: registered new interface driver usb-storage1520worker2 # [ 0.913784] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.1521client # [ 0.911619] Synopsys Designware Multimedia Card Interface Driver1522worker2 # [ 0.913943] sky2: driver version 1.301523worker1 # [ 0.911365] usbcore: registered new interface driver usbserial_generic1524client # [ 0.911986] sdhci-pltfm: SDHCI platform and OF driver helper1525worker1 # [ 0.911380] usbserial: USB Serial support registered for generic1526client # [ 0.913669] hid: raw HID events driver (C) Jiri Kosina1527worker1 # [ 0.911955] hv_vmbus: registering driver hyperv_keyboard1528client # [ 0.913932] usbcore: registered new interface driver usbhid1529worker1 # [ 0.913180] ehci-pci 0000:00:0a.0: EHCI Host Controller1530client # [ 0.913938] usbhid: USB HID core driver1531lb # [ 0.943502] registered taskstats version 11532worker1 # [ 0.913205] ehci-pci 0000:00:0a.0: new USB bus registered, assigned bus number 11533client # [ 0.918733] ehci-pci 0000:00:0a.0: USB 2.0 started, EHCI 1.001534lb # [ 0.943662] Loading compiled-in X.509 certificates1535worker1 # [ 0.913411] ehci-pci 0000:00:0a.0: irq 15, io mem 0x100890001536worker2 # [ 0.923107] usbcore: registered new interface driver usb-storage1537client # [ 0.919927] hub 1-0:1.0: USB hub found1538client # [ 0.920402] hub 1-0:1.0: 6 ports detected1539worker2 # [ 0.923233] usbcore: registered new interface driver usbserial_generic1540worker2 # [ 0.923251] usbserial: USB Serial support registered for generic1541worker1 # [ 0.918824] rtc-pl031 9010000.pl031: registered as rtc01542worker2 # [ 0.923961] hv_vmbus: registering driver hyperv_keyboard1543worker1 # [ 0.918856] rtc-pl031 9010000.pl031: setting system clock to 2026-09-21T09:13:10 UTC (1789981990)1544client # [ 0.922062] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available1545worker1 # [ 0.919197] i2c_dev: i2c /dev entries driver1546worker2 # [ 0.928414] ehci-pci 0000:00:0a.0: EHCI Host Controller1547worker2 # [ 0.928447] ehci-pci 0000:00:0a.0: new USB bus registered, assigned bus number 11548client # [ 0.924614] drop_monitor: Initializing network drop monitor service1549worker2 # [ 0.928767] ehci-pci 0000:00:0a.0: irq 15, io mem 0x100890001550client # [ 0.924794] NET: Registered PF_INET6 protocol family1551client # [ 0.926792] Segment Routing with IPv61552client # [ 0.926810] In-situ OAM (IOAM) with IPv61553worker2 # [ 0.931474] rtc-pl031 9010000.pl031: registered as rtc01554lb # [ 0.952288] Demotion targets for Node 0: null1555client # [ 0.926840] NET: Registered PF_PACKET protocol family1556lb # [ 0.952395] Key type .fscrypt registered1557worker1 # [ 0.923985] sdhci: Secure Digital Host Controller Interface driver1558worker2 # [ 0.931506] rtc-pl031 9010000.pl031: setting system clock to 2026-09-21T09:13:10 UTC (1789981990)1559lb # [ 0.952402] Key type fscrypt-provisioning registered1560worker1 # [ 0.923995] sdhci: Copyright(c) Pierre Ossman1561worker2 # [ 0.931877] i2c_dev: i2c /dev entries driver1562lb # [ 0.952499] ima: No TPM chip found, activating TPM-bypass!1563worker1 # [ 0.924283] Synopsys Designware Multimedia Card Interface Driver1564lb # [ 0.952519] ima: Allocated hash algorithm: sha11565client # [ 0.928479] 9pnet: Installing 9P2000 support1566worker1 # [ 0.924638] sdhci-pltfm: SDHCI platform and OF driver helper1567lb # [ 0.952544] ima: No architecture policies found1568worker1 # [ 0.926294] hid: raw HID events driver (C) Jiri Kosina1569worker1 # [ 0.926548] usbcore: registered new interface driver usbhid1570worker1 # [ 0.926555] usbhid: USB HID core driver1571lb # [ 0.956879] input: gpio-keys as /devices/platform/gpio-keys/input/input01572worker1 # [ 0.931096] ehci-pci 0000:00:0a.0: USB 2.0 started, EHCI 1.001573worker1 # [ 0.932144] hub 1-0:1.0: USB hub found1574worker1 # [ 0.932626] hub 1-0:1.0: 6 ports detected1575client # [ 0.931191] Key type dns_resolver registered1576worker2 # [ 0.936812] sdhci: Secure Digital Host Controller Interface driver1577worker2 # [ 0.936823] sdhci: Copyright(c) Pierre Ossman1578worker2 # [ 0.937098] Synopsys Designware Multimedia Card Interface Driver1579worker2 # [ 0.937479] sdhci-pltfm: SDHCI platform and OF driver helper1580worker1 # [ 0.934515] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available1581worker2 # [ 0.938434] ehci-pci 0000:00:0a.0: USB 2.0 started, EHCI 1.001582worker2 # [ 0.938812] hub 1-0:1.0: USB hub found1583worker2 # [ 0.938835] hub 1-0:1.0: 6 ports detected1584client # [ 0.937334] registered taskstats version 11585client # [ 0.937474] Loading compiled-in X.509 certificates1586worker1 # [ 0.937121] drop_monitor: Initializing network drop monitor service1587worker1 # [ 0.937272] NET: Registered PF_INET6 protocol family1588worker1 # [ 0.939217] Segment Routing with IPv61589worker1 # [ 0.939237] In-situ OAM (IOAM) with IPv61590worker2 # [ 0.944759] hid: raw HID events driver (C) Jiri Kosina1591worker1 # [ 0.939265] NET: Registered PF_PACKET protocol family1592worker2 # [ 0.945094] usbcore: registered new interface driver usbhid1593worker2 # [ 0.945101] usbhid: USB HID core driver1594worker1 # [ 0.940921] 9pnet: Installing 9P2000 support1595worker2 # [ 0.948263] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available1596worker2 # [ 0.950060] drop_monitor: Initializing network drop monitor service1597client # [ 0.946206] Demotion targets for Node 0: null1598worker2 # [ 0.950243] NET: Registered PF_INET6 protocol family1599client # [ 0.946376] Key type .fscrypt registered1600client # [ 0.946406] Key type fscrypt-provisioning registered1601worker1 # [ 0.943662] Key type dns_resolver registered1602worker2 # [ 0.953304] Segment Routing with IPv61603client # [ 0.946521] ima: No TPM chip found, activating TPM-bypass!1604worker2 # [ 0.953326] In-situ OAM (IOAM) with IPv61605client # [ 0.946565] ima: Allocated hash algorithm: sha11606worker2 # [ 0.953353] NET: Registered PF_PACKET protocol family1607client # [ 0.946630] ima: No architecture policies found1608lb # [ 0.974568] clk: Disabling unused clocks1609client # [ 0.950652] input: gpio-keys as /devices/platform/gpio-keys/input/input01610lb # [ 0.974601] PM: genpd: Disabling unused power domains1611worker1 # [ 0.949735] registered taskstats version 11612worker1 # [ 0.949897] Loading compiled-in X.509 certificates1613worker2 # [ 0.955057] 9pnet: Installing 9P2000 support1614worker2 # [ 0.957227] Key type dns_resolver registered1615lb # [ 0.978938] Freeing unused kernel memory: 4736K1616lb # [ 0.979130] Run /init as init process1617worker1 # [ 0.958897] Demotion targets for Node 0: null1618worker2 # [ 0.965437] registered taskstats version 11619worker1 # [ 0.959015] Key type .fscrypt registered1620worker1 # [ 0.959021] Key type fscrypt-provisioning registered1621worker2 # [ 0.965643] Loading compiled-in X.509 certificates1622worker1 # [ 0.959118] ima: No TPM chip found, activating TPM-bypass!1623worker1 # [ 0.959137] ima: Allocated hash algorithm: sha11624worker1 # [ 0.959159] ima: No architecture policies found1625worker1 # [ 0.963173] input: gpio-keys as /devices/platform/gpio-keys/input/input01626lb # [ 0.994021] systemd[1]: Successfully made /usr/ read-only.1627worker2 # [ 0.974556] Demotion targets for Node 0: null1628client # [ 0.968243] clk: Disabling unused clocks1629worker2 # [ 0.974667] Key type .fscrypt registered1630client # [ 0.968275] PM: genpd: Disabling unused power domains1631worker2 # [ 0.974674] Key type fscrypt-provisioning registered1632worker2 # [ 0.974794] ima: No TPM chip found, activating TPM-bypass!1633worker2 # [ 0.974814] ima: Allocated hash algorithm: sha11634worker2 # [ 0.974838] ima: No architecture policies found1635client # [ 0.972504] Freeing unused kernel memory: 4736K1636client # [ 0.972724] Run /init as init process1637worker2 # [ 0.979076] input: gpio-keys as /devices/platform/gpio-keys/input/input01638worker1 # [ 0.982154] clk: Disabling unused clocks1639worker1 # [ 0.982184] PM: genpd: Disabling unused power domains1640worker1 # [ 0.986448] Freeing unused kernel memory: 4736K1641client # [ 0.988475] systemd[1]: Successfully made /usr/ read-only.1642worker1 # [ 0.987244] Run /init as init process1643worker2 # [ 0.997512] clk: Disabling unused clocks1644worker2 # [ 0.997544] PM: genpd: Disabling unused power domains1645worker2 # [ 1.001917] Freeing unused kernel memory: 4736K1646worker2 # [ 1.002107] Run /init as init process1647worker1 # [ 1.000998] systemd[1]: Successfully made /usr/ read-only.1648worker2 # [ 1.016799] systemd[1]: Successfully made /usr/ read-only.1649lb # [ 1.167808] usb 1-1: new high-speed USB device number 2 using ehci-pci1650client # [ 1.166251] usb 1-1: new high-speed USB device number 2 using ehci-pci1651worker2 # [ 1.186482] usb 1-1: new high-speed USB device number 2 using ehci-pci1652worker1 # [ 1.182705] usb 1-1: new high-speed USB device number 2 using ehci-pci1653lb # [ 1.319758] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:0a.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input11654lb # [ 1.328743] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1655lb # [ 1.336310] systemd[1]: Detected virtualization qemu.1656lb # [ 1.336377] systemd[1]: Detected architecture arm64.1657lb # [ 1.336391] systemd[1]: Running in initrd.1658lb # [ 1.337006] systemd[1]: Initializing machine ID from random generator.1659lb # [ 1.337174] systemd[1]: Hostname set to <lb>.1660client # [ 1.318487] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:0a.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input11661worker2 # [ 1.336031] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:0a.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input11662client # [ 1.323886] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1663worker1 # [ 1.332586] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:0a.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input11664client # [ 1.323939] systemd[1]: Detected virtualization qemu.1665client # [ 1.324006] systemd[1]: Detected architecture arm64.1666client # [ 1.324027] systemd[1]: Running in initrd.1667client # [ 1.324893] systemd[1]: Initializing machine ID from random generator.1668worker1 # [ 1.335435] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1669client # [ 1.325237] systemd[1]: Hostname set to <client>.1670worker1 # [ 1.335464] systemd[1]: Detected virtualization qemu.1671worker1 # [ 1.335507] systemd[1]: Detected architecture arm64.1672worker1 # [ 1.335518] systemd[1]: Running in initrd.1673worker1 # [ 1.336034] systemd[1]: Initializing machine ID from random generator.1674worker1 # [ 1.336171] systemd[1]: Hostname set to <worker1>.1675worker2 # [ 1.351533] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1676worker2 # [ 1.360302] systemd[1]: Detected virtualization qemu.1677worker2 # [ 1.362538] systemd[1]: Detected architecture arm64.1678worker2 # [ 1.364499] systemd[1]: Running in initrd.1679worker2 # [ 1.367170] systemd[1]: Initializing machine ID from random generator.1680worker2 # [ 1.369978] systemd[1]: Hostname set to <worker2>.1681lb # [ 1.399952] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:0a.0-1/input01682client # [ 1.406450] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:0a.0-1/input01683worker1 # [ 1.414851] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:0a.0-1/input01684worker2 # [ 1.422610] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:0a.0-1/input01685lb # [ 1.519809] usb 1-2: new high-speed USB device number 3 using ehci-pci1686client # [ 1.526299] usb 1-2: new high-speed USB device number 3 using ehci-pci1687worker1 # [ 1.534786] usb 1-2: new high-speed USB device number 3 using ehci-pci1688worker2 # [ 1.542446] usb 1-2: new high-speed USB device number 3 using ehci-pci1689lb # [ 1.623448] systemd[1]: bpf-restrict-fs: LSM BPF program attached1690worker2 # [ 1.612563] systemd[1]: bpf-restrict-fs: LSM BPF program attached1691worker1 # [ 1.618220] systemd[1]: bpf-restrict-fs: LSM BPF program attached1692client # [ 1.638383] systemd[1]: bpf-restrict-fs: LSM BPF program attached1693lb # [ 1.675936] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:0a.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input21694lb # [ 1.680039] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:0a.0-2/input01695worker2 # [ 1.669012] systemd[1]: Queued start job for default target Initrd Default Target.1696worker2 # [ 1.676660] systemd[1]: Created slice Slice /system/modprobe.1697worker2 # [ 1.677697] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1698worker2 # [ 1.678960] systemd[1]: Expecting device /dev/disk/by-label/nixos...1699worker2 # [ 1.679864] systemd[1]: Reached target Path Units.1700worker2 # [ 1.680604] systemd[1]: Reached target Slice Units.1701worker2 # [ 1.681310] systemd[1]: Reached target Swaps.1702worker2 # [ 1.681944] systemd[1]: Reached target Timer Units.1703worker2 # [ 1.682909] systemd[1]: Listening on D-Bus System Message Bus Socket.1704worker2 # [ 1.683996] systemd[1]: Listening on Journal Socket (/dev/log).1705worker2 # [ 1.684966] systemd[1]: Listening on Journal Sockets.1706worker2 # [ 1.685788] systemd[1]: Listening on udev Control Socket.1707worker2 # [ 1.686851] systemd[1]: Listening on udev Kernel Socket.1708worker2 # [ 1.687788] systemd[1]: Reached target Socket Units.1709client # [ 1.680254] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:0a.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input21710worker2 # [ 1.690735] systemd[1]: Starting Create List of Static Device Nodes...1711client # [ 1.686443] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:0a.0-2/input01712worker2 # [ 1.694214] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:0a.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input21713worker2 # [ 1.698652] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:0a.0-2/input01714worker1 # [ 1.693439] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:0a.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input21715lb # [ 1.722163] systemd[1]: Queued start job for default target Initrd Default Target.1716worker2 # [ 1.702611] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1717worker1 # [ 1.693614] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:0a.0-2/input01718worker2 # [ 1.703598] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs1719worker2 # [ 1.710034] systemd[1]: Mounting Kernel Configuration File System...1720worker1 # [ 1.704666] systemd[1]: Queued start job for default target Initrd Default Target.1721lb # [ 1.734977] systemd[1]: Created slice Slice /system/modprobe.1722lb # [ 1.736203] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1723lb # [ 1.737504] systemd[1]: Expecting device /dev/disk/by-label/nixos...1724lb # [ 1.738554] systemd[1]: Reached target Path Units.1725lb # [ 1.739382] systemd[1]: Reached target Slice Units.1726lb # [ 1.740264] systemd[1]: Reached target Swaps.1727worker1 # [ 1.713314] systemd[1]: Created slice Slice /system/modprobe.1728lb # [ 1.741022] systemd[1]: Reached target Timer Units.1729lb # [ 1.742114] systemd[1]: Listening on D-Bus System Message Bus Socket.1730lb # [ 1.743383] systemd[1]: Listening on Journal Socket (/dev/log).1731worker1 # [ 1.714722] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1732lb # [ 1.744590] systemd[1]: Listening on Journal Sockets.1733worker1 # [ 1.714769] systemd[1]: Expecting device /dev/disk/by-label/nixos...1734lb # [ 1.745594] systemd[1]: Listening on udev Control Socket.1735worker1 # [ 1.714804] systemd[1]: Reached target Path Units.1736lb # [ 1.746621] systemd[1]: Listening on udev Kernel Socket.1737worker1 # [ 1.714828] systemd[1]: Reached target Slice Units.1738lb # [ 1.747562] systemd[1]: Reached target Socket Units.1739worker1 # [ 1.714851] systemd[1]: Reached target Swaps.1740worker1 # [ 1.714876] systemd[1]: Reached target Timer Units.1741worker1 # [ 1.715084] systemd[1]: Listening on D-Bus System Message Bus Socket.1742worker1 # [ 1.715252] systemd[1]: Listening on Journal Socket (/dev/log).1743worker1 # [ 1.715421] systemd[1]: Listening on Journal Sockets.1744lb # [ 1.750221] systemd[1]: Starting Create List of Static Device Nodes...1745worker1 # [ 1.715569] systemd[1]: Listening on udev Control Socket.1746worker1 # [ 1.715689] systemd[1]: Listening on udev Kernel Socket.1747worker1 # [ 1.715714] systemd[1]: Reached target Socket Units.1748worker2 # [ 1.734630] systemd[1]: Starting Journal Service...1749worker1 # [ 1.728915] systemd[1]: Starting Create List of Static Device Nodes...1750lb # [ 1.759960] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1751worker2 # [ 1.742681] systemd[1]: Starting Load Kernel Modules...1752worker1 # [ 1.735968] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1753lb # [ 1.760055] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs1754worker2 # [ 1.743526] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1755worker1 # [ 1.737105] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs1756lb # [ 1.768054] systemd[1]: Mounting Kernel Configuration File System...1757client # [ 1.744034] systemd[1]: Queued start job for default target Initrd Default Target.1758worker1 # [ 1.746859] systemd[1]: Mounting Kernel Configuration File System...1759worker2 # [ 1.756146] systemd[1]: Starting Coldplug All udev Devices...1760client # [ 1.752481] systemd[1]: Created slice Slice /system/modprobe.1761client # [ 1.753674] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1762client # [ 1.755125] systemd[1]: Expecting device /dev/disk/by-label/nixos...1763client # [ 1.756229] systemd[1]: Reached target Path Units.1764client # [ 1.757072] systemd[1]: Reached target Slice Units.1765worker1 # [ 1.756249] systemd[1]: Starting Journal Service...1766client # [ 1.757950] systemd[1]: Reached target Swaps.1767client # [ 1.758791] systemd[1]: Reached target Timer Units.1768client # [ 1.759826] systemd[1]: Listening on D-Bus System Message Bus Socket.1769client # [ 1.761186] systemd[1]: Listening on Journal Socket (/dev/log).1770client # [ 1.762432] systemd[1]: Listening on Journal Sockets.1771client # [ 1.763464] systemd[1]: Listening on udev Control Socket.1772client # [ 1.764515] systemd[1]: Listening on udev Kernel Socket.1773client # [ 1.765441] systemd[1]: Reached target Socket Units.1774client # [ 1.768081] systemd[1]: Starting Create List of Static Device Nodes...1775worker2 # [ 1.774562] systemd[1]: Finished Create List of Static Device Nodes.1776worker2 # [ 1.775701] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1777lb # [ 1.800040] systemd[1]: Starting Journal Service...1778client # [ 1.775079] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1779client # [ 1.776278] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs1780worker2 # [ 1.783192] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1781worker2 # [ 1.783594] systemd[1]: Mounted Kernel Configuration File System.1782worker1 # [ 1.779461] systemd[1]: Starting Load Kernel Modules...1783lb # [ 1.807998] systemd[1]: Starting Load Kernel Modules...1784worker1 # [ 1.781041] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1785lb # [ 1.808890] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1786client # [ 1.786425] systemd[1]: Mounting Kernel Configuration File System...1787worker2 # [ 1.792052] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1788client # [ 1.795695] systemd[1]: Starting Journal Service...1789worker1 # [ 1.794879] systemd[1]: Starting Coldplug All udev Devices...1790worker2 # [ 1.798909] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1791lb # [ 1.824172] systemd[1]: Starting Coldplug All udev Devices...1792worker2 # [ 1.811927] systemd-journald[73]: Collecting audit messages is disabled.1793worker2 # [ 1.812823] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1794lb # [ 1.838657] systemd[1]: Finished Create List of Static Device Nodes.1795worker1 # [ 1.811067] systemd[1]: Finished Create List of Static Device Nodes.1796lb # [ 1.840701] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1797worker1 # [ 1.811929] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1798worker1 # [ 1.812251] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1799worker1 # [ 1.812489] systemd[1]: Mounted Kernel Configuration File System.1800worker2 # [ 1.826776] [drm] pci: virtio-gpu-pci detected at 0000:00:08.01801worker2 # [ 1.827012] [drm] features: -virgl +edid -resource_blob -host_visible1802worker2 # [ 1.827022] [drm] features: -context_init1803worker2 # [ 1.827723] [drm] number of scanouts: 11804worker2 # [ 1.827740] [drm] number of cap sets: 01805lb # [ 1.852356] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1806lb # [ 1.852748] systemd[1]: Mounted Kernel Configuration File System.1807client # [ 1.830478] systemd[1]: Starting Load Kernel Modules...1808client # [ 1.830609] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1809worker1 # [ 1.827165] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1810worker1 # [ 1.831379] systemd-journald[73]: Collecting audit messages is disabled.1811lb # [ 1.860167] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1812client # [ 1.838409] systemd[1]: Starting Coldplug All udev Devices...1813worker2 # [ 1.843635] virtio-pci 0000:00:08.0: [drm] Registered 1 planes with drm panic1814worker2 # [ 1.843664] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:08.0 on minor 01815lb # [ 1.869871] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1816client # [ 1.847168] systemd-journald[73]: Collecting audit messages is disabled.1817worker2 # [ 1.851113] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1818client # [ 1.848128] systemd[1]: Finished Create List of Static Device Nodes.1819worker2 # [ 1.853837] systemd[1]: Starting Create Static Device Nodes in /dev...1820lb # [ 1.874133] systemd-journald[73]: Collecting audit messages is disabled.1821client # [ 1.849804] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1822worker1 # [ 1.849699] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1823client # [ 1.861407] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1824client # [ 1.861794] systemd[1]: Mounted Kernel Configuration File System.1825lb # [ 1.887926] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1826worker1 # [ 1.862713] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1827worker1 # [ 1.867403] [drm] pci: virtio-gpu-pci detected at 0000:00:08.01828worker1 # [ 1.867639] [drm] features: -virgl +edid -resource_blob -host_visible1829worker1 # [ 1.867649] [drm] features: -context_init1830worker1 # [ 1.868365] [drm] number of scanouts: 11831worker1 # [ 1.868383] [drm] number of cap sets: 01832lb # [ 1.899643] [drm] pci: virtio-gpu-pci detected at 0000:00:08.01833client # [ 1.875282] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1834lb # [ 1.905452] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1835worker2 # [ 1.878721] Console: switching to colour frame buffer device 160x501836worker2 # [ 1.885475] virtio-pci 0000:00:08.0: [drm] fb0: virtio_gpudrmfb frame buffer device1837lb # [ 1.911987] [drm] features: -virgl +edid -resource_blob -host_visible1838lb # [ 1.912006] [drm] features: -context_init1839lb # [ 1.912761] [drm] number of scanouts: 11840lb # [ 1.912779] [drm] number of cap sets: 01841worker1 # [ 1.887058] virtio-pci 0000:00:08.0: [drm] Registered 1 planes with drm panic1842worker1 # [ 1.887083] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:08.0 on minor 01843worker1 # [ 1.895372] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1844lb # [ 1.920123] systemd[1]: Starting Create Static Device Nodes in /dev...1845worker1 # [ 1.898167] systemd[1]: Starting Create Static Device Nodes in /dev...1846client # [ 1.901271] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1847lb # [ 1.924984] virtio-pci 0000:00:08.0: [drm] Registered 1 planes with drm panic1848worker2 # [ 1.911461] systemd[1]: Finished Load Kernel Modules.1849lb # [ 1.925005] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:08.0 on minor 01850client # [ 1.910264] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1851worker2 # [ 1.918955] systemd[1]: Starting Apply Kernel Variables...1852client # [ 1.919544] [drm] pci: virtio-gpu-pci detected at 0000:00:08.01853client # [ 1.919779] [drm] features: -virgl +edid -resource_blob -host_visible1854client # [ 1.919789] [drm] features: -context_init1855client # [ 1.920503] [drm] number of scanouts: 11856client # [ 1.920520] [drm] number of cap sets: 01857worker2 # [ 1.934546] systemd[1]: Finished Create Static Device Nodes in /dev.1858worker1 # [ 1.923068] Console: switching to colour frame buffer device 160x501859worker2 # [ 1.936886] systemd[1]: Reached target Preparation for Local File Systems.1860lb # [ 1.952687] Console: switching to colour frame buffer device 160x501861lb # [ 1.959280] virtio-pci 0000:00:08.0: [drm] fb0: virtio_gpudrmfb frame buffer device1862worker2 # [ 1.942559] systemd[1]: Reached target Local File Systems.1863client # [ 1.938974] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1864client # [ 1.941507] systemd[1]: Starting Create Static Device Nodes in /dev...1865worker2 # [ 1.950851] systemd[1]: Starting Rule-based Manager for Device Events and Files...1866worker1 # [ 1.947505] virtio-pci 0000:00:08.0: [drm] fb0: virtio_gpudrmfb frame buffer device1867client # [ 1.946581] virtio-pci 0000:00:08.0: [drm] Registered 1 planes with drm panic1868client # [ 1.946602] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:08.0 on minor 01869lb # [ 1.984448] systemd[1]: Finished Load Kernel Modules.1870lb # [ 1.988233] systemd[1]: Starting Apply Kernel Variables...1871worker1 # [ 1.961474] systemd[1]: Finished Load Kernel Modules.1872worker1 # [ 1.964725] systemd[1]: Starting Apply Kernel Variables...1873lb # [ 2.004024] systemd[1]: Finished Create Static Device Nodes in /dev.1874lb # [ 2.004463] systemd[1]: Reached target Preparation for Local File Systems.1875lb # [ 2.004503] systemd[1]: Reached target Local File Systems.1876worker2 # [ 1.986564] systemd[1]: Finished Apply Kernel Variables.1877worker1 # [ 1.983627] systemd[1]: Finished Create Static Device Nodes in /dev.1878worker1 # [ 1.985971] systemd[1]: Reached target Preparation for Local File Systems.1879worker1 # [ 1.986016] systemd[1]: Reached target Local File Systems.1880client # [ 1.982541] Console: switching to colour frame buffer device 160x501881client # [ 1.989225] virtio-pci 0000:00:08.0: [drm] fb0: virtio_gpudrmfb frame buffer device1882worker2 # [ 1.994240] systemd[1]: Started Journal Service.1883lb # [ 2.020137] systemd[1]: Starting Rule-based Manager for Device Events and Files...1884client # [ 2.002906] systemd[1]: Finished Create Static Device Nodes in /dev.1885worker2 # [ 1.996327] systemd-modules-load[74]: Inserted module 'dm_mod'1886client # [ 2.003307] systemd[1]: Reached target Preparation for Local File Systems.1887worker2 # [ 1.997476] systemd-modules-load[74]: Module 'virtio_balloon' is built in1888client # [ 2.003344] systemd[1]: Reached target Local File Systems.1889worker2 # [ 1.998609] systemd-modules-load[74]: Module 'virtio_console' is built in1890worker2 # [ 1.999667] systemd-modules-load[74]: Inserted module 'virtio_gpu'1891worker1 # [ 2.003043] systemd[1]: Starting Rule-based Manager for Device Events and Files...1892client # [ 2.008565] systemd[1]: Starting Rule-based Manager for Device Events and Files...1893worker2 # [ 2.009309] systemd-modules-load[74]: Module 'virtio_rng' is built in1894worker2 # [ 2.010356] systemd[1]: Starting Create System Files and Directories...1895worker1 # [ 2.004417] systemd-modules-load[74]: Inserted module 'dm_mod'1896worker1 # [ 2.005547] systemd-modules-load[74]: Module 'virtio_balloon' is built in1897worker1 # [ 2.021391] systemd[1]: Started Journal Service.1898lb # [ 2.052143] systemd[1]: Finished Apply Kernel Variables.1899client # [ 2.030409] systemd[1]: Finished Load Kernel Modules.1900lb # [ 2.055188] systemd[1]: Started Journal Service.1901worker1 # [ 2.016450] systemd-modules-load[74]: Module 'virtio_console' is built in1902lb # [ 2.044420] systemd-modules-load[74]: Inserted module 'dm_mod'1903lb # [ 2.045680] systemd-modules-load[74]: Module 'virtio_balloon' is built in1904worker1 # [ 2.020545] systemd-modules-load[74]: Inserted module 'virtio_gpu'1905worker1 # [ 2.024283] systemd-modules-load[74]: Module 'virtio_rng' is built in1906worker1 # [ 2.025321] systemd[1]: Finished Apply Kernel Variables.1907lb # [ 2.052305] systemd-modules-load[74]: Module 'virtio_console' is built in1908client # [ 2.039559] systemd[1]: Starting Apply Kernel Variables...1909worker1 # [ 2.028386] systemd[1]: Starting Create System Files and Directories...1910client # [ 2.026383] systemd-modules-load[75]: Inserted module 'dm_mod'1911client # [ 2.027655] systemd-modules-load[75]: Module 'virtio_balloon' is built in1912lb # [ 2.056334] systemd-modules-load[74]: Inserted module 'virtio_gpu'1913client # [ 2.028858] systemd-modules-load[75]: Module 'virtio_console' is built in1914client # [ 2.029923] systemd-modules-load[75]: Inserted module 'virtio_gpu'1915client # [ 2.030875] systemd-modules-load[75]: Module 'virtio_rng' is built in1916worker2 # [ 2.041444] systemd-udevd[81]: Using default interface naming scheme 'v261'.1917client # [ 2.050431] systemd[1]: Started Journal Service.1918worker2 # [ 2.044730] systemd[1]: Finished Create System Files and Directories.1919lb # [ 2.064310] systemd-modules-load[74]: Module 'virtio_rng' is built in1920lb # [ 2.069671] systemd[1]: Starting Create System Files and Directories...1921client # [ 2.062605] systemd[1]: Starting Create System Files and Directories...1922worker1 # [ 2.065620] systemd[1]: Finished Create System Files and Directories.1923worker2 # [ 2.073529] systemd[1]: Started Rule-based Manager for Device Events and Files.1924worker1 # [ 2.077788] systemd-udevd[81]: Using default interface naming scheme 'v261'.1925lb # [ 2.103544] systemd[1]: Finished Create System Files and Directories.1926lb # [ 2.106232] systemd-udevd[81]: Using default interface naming scheme 'v261'.1927client # [ 2.086806] systemd-udevd[79]: Using default interface naming scheme 'v261'.1928client # [ 2.093429] systemd[1]: Finished Apply Kernel Variables.1929worker1 # [ 2.109377] systemd[1]: Started Rule-based Manager for Device Events and Files.1930client # [ 2.108765] systemd[1]: Finished Create System Files and Directories.1931lb # [ 2.138202] systemd[1]: Started Rule-based Manager for Device Events and Files.1932client # [ 2.129280] systemd[1]: Started Rule-based Manager for Device Events and Files.1933worker2 # [ 2.140251] systemd[1]: Starting Virtual Console Setup...1934worker1 # [ 2.168813] systemd[1]: Starting Virtual Console Setup...1935lb # [ 2.198462] systemd[1]: Starting Virtual Console Setup...1936client # [ 2.192106] systemd[1]: Starting Virtual Console Setup...1937worker2 # [ 2.212815] systemd-vconsole-setup[105]: Configuration of first virtual console was skipped, ignoring remaining ones.1938worker2 # [ 2.216069] systemd[1]: Finished Virtual Console Setup.1939lb # [ 2.244539] systemd-vconsole-setup[103]: Configuration of first virtual console was skipped, ignoring remaining ones.1940worker1 # [ 2.220556] systemd-vconsole-setup[103]: Configuration of first virtual console was skipped, ignoring remaining ones.1941lb # [ 2.248215] systemd[1]: Finished Virtual Console Setup.1942worker1 # [ 2.224163] systemd[1]: Finished Virtual Console Setup.1943client # [ 2.240513] systemd-vconsole-setup[103]: Configuration of first virtual console was skipped, ignoring remaining ones.1944client # [ 2.244043] systemd[1]: Finished Virtual Console Setup.1945worker1 # [ 2.832374] systemd[1]: Finished Coldplug All udev Devices.1946worker1 # [ 2.833595] systemd[1]: Reached target System Initialization.1947worker1 # [ 2.834450] systemd[1]: Reached target Basic System.1948worker2 # [ 2.845204] systemd[1]: Finished Coldplug All udev Devices.1949worker2 # [ 2.846217] systemd[1]: Reached target System Initialization.1950worker2 # [ 2.847080] systemd[1]: Reached target Basic System.1951lb # [ 2.869086] systemd[1]: Finished Coldplug All udev Devices.1952lb # [ 2.870244] systemd[1]: Reached target System Initialization.1953lb # [ 2.871094] systemd[1]: Reached target Basic System.1954client # [ 2.861472] systemd[1]: Finished Coldplug All udev Devices.1955client # [ 2.864133] systemd[1]: Reached target System Initialization.1956client # [ 2.864989] systemd[1]: Reached target Basic System.1957worker1 # [ 2.959897] (udev-worker)[95]: Network interface NamePolicy= disabled on kernel command line.1958worker2 # [ 2.983901] (udev-worker)[104]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name.1959lb # [ 3.032276] (udev-worker)[106]: Network interface NamePolicy= disabled on kernel command line.1960lb # [ 3.036566] (udev-worker)[97]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name.1961lb # [ 3.039155] (udev-worker)[97]: Network interface NamePolicy= disabled on kernel command line.1962worker2 # [ 3.025935] (udev-worker)[104]: Network interface NamePolicy= disabled on kernel command line.1963worker1 # [ 3.022654] (udev-worker)[99]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name.1964worker2 # [ 3.033554] (udev-worker)[100]: Network interface NamePolicy= disabled on kernel command line.1965worker1 # [ 3.028309] (udev-worker)[99]: Network interface NamePolicy= disabled on kernel command line.1966client # [ 3.031446] (udev-worker)[92]: Network interface NamePolicy= disabled on kernel command line.1967client # [ 3.049443] (udev-worker)[96]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name.1968client # [ 3.054316] (udev-worker)[96]: Network interface NamePolicy= disabled on kernel command line.1969worker2 # [ 3.087866] systemd[1]: Found device /dev/disk/by-label/nixos.1970worker2 # [ 3.092228] systemd[1]: Reached target Initrd Root Device.1971worker1 # [ 3.090413] systemd[1]: Found device /dev/disk/by-label/nixos.1972worker2 # [ 3.097294] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos...1973worker1 # [ 3.092254] systemd[1]: Reached target Initrd Root Device.1974worker1 # [ 3.095696] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos...1975lb # [ 3.127846] systemd[1]: Found device /dev/disk/by-label/nixos.1976lb # [ 3.137413] systemd[1]: Reached target Initrd Root Device.1977lb # [ 3.142258] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos...1978client # [ 3.116125] systemd[1]: Found device /dev/disk/by-label/nixos.1979client # [ 3.117584] systemd[1]: Reached target Initrd Root Device.1980client # [ 3.120686] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos...1981worker2 # [ 3.153729] systemd-fsck[114]: nixos: clean, 12/65536 files, 13019/262144 blocks1982worker2 # [ 3.160244] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos.1983worker1 # [ 3.155410] systemd-fsck[112]: nixos: clean, 12/65536 files, 13019/262144 blocks1984worker2 # [ 3.168369] systemd[1]: Mounting /sysroot...1985worker1 # [ 3.161823] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos.1986lb # [ 3.196514] systemd-fsck[112]: nixos: clean, 12/65536 files, 13019/262144 blocks1987worker1 # [ 3.176205] systemd[1]: Mounting /sysroot...1988client # [ 3.175558] systemd-fsck[112]: nixos: clean, 12/65536 files, 13019/262144 blocks1989lb # [ 3.204106] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos.1990client # [ 3.181621] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos.1991lb # [ 3.211337] systemd[1]: Mounting /sysroot...1992client # [ 3.188551] systemd[1]: Mounting /sysroot...1993worker2 # [ 3.218344] EXT4-fs (vda): mounted filesystem 97bd670e-dfd6-452b-b95d-333e56646b54 r/w with ordered data mode. Quota mode: none.1994worker2 # [ 3.211120] systemd[1]: Mounted /sysroot.1995worker2 # [ 3.213512] systemd[1]: Reached target Initrd Root File System.1996worker2 # [ 3.219160] systemd[1]: Starting Mountpoints Configured in the Real Root...1997worker1 # [ 3.228719] EXT4-fs (vda): mounted filesystem 081bd967-e28b-42b1-aa39-a7d26f23863b r/w with ordered data mode. Quota mode: none.1998lb # [ 3.258920] EXT4-fs (vda): mounted filesystem 354440ba-1f96-4418-b8f3-d4670856c558 r/w with ordered data mode. Quota mode: none.1999worker1 # [ 3.221094] systemd[1]: Mounted /sysroot.2000worker1 # [ 3.222215] systemd[1]: Reached target Initrd Root File System.
loading 1,715 hidden lines…
3716worker2 # [ 26.726594] nix-grpc-daemon[781]: ts=2026-09-21T09:13:36Z level=info event=scheduler_connected addr=lb:500513717worker2 # [ 26.732440] nix-grpc-daemon[781]: ts=2026-09-21T09:13:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3718worker1 # [ 26.773401] nix-grpc-daemon[774]: ts=2026-09-21T09:13:36Z level=info event=scheduler_connected addr=lb:500513719worker1 # [ 26.779572] nix-grpc-daemon[774]: ts=2026-09-21T09:13:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3720worker2 # [ 26.933952] nix-grpc-daemon[781]: ts=2026-09-21T09:13:36Z level=info event=scheduler_connected addr=lb:500513721worker2 # [ 26.939424] nix-grpc-daemon[781]: ts=2026-09-21T09:13:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3722worker1 # [ 26.980565] nix-grpc-daemon[774]: ts=2026-09-21T09:13:36Z level=info event=scheduler_connected addr=lb:500513723worker1 # [ 26.985900] nix-grpc-daemon[774]: ts=2026-09-21T09:13:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3724worker2 # [ 27.140632] nix-grpc-daemon[781]: ts=2026-09-21T09:13:36Z level=info event=scheduler_connected addr=lb:500513725worker2 # [ 27.145631] nix-grpc-daemon[781]: ts=2026-09-21T09:13:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3726worker1 # [ 27.186786] nix-grpc-daemon[774]: ts=2026-09-21T09:13:36Z level=info event=scheduler_connected addr=lb:500513727worker1 # [ 27.192164] nix-grpc-daemon[774]: ts=2026-09-21T09:13:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3728worker2 # [ 27.346917] nix-grpc-daemon[781]: ts=2026-09-21T09:13:36Z level=info event=scheduler_connected addr=lb:500513729worker2 # [ 27.352244] nix-grpc-daemon[781]: ts=2026-09-21T09:13:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3730worker1 # [ 27.394782] nix-grpc-daemon[774]: ts=2026-09-21T09:13:36Z level=info event=scheduler_connected addr=lb:500513731worker1 # [ 27.399028] nix-grpc-daemon[774]: ts=2026-09-21T09:13:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3732worker2 # [ 27.555851] nix-grpc-daemon[781]: ts=2026-09-21T09:13:37Z level=info event=scheduler_connected addr=lb:500513733worker2 # [ 27.560226] nix-grpc-daemon[781]: ts=2026-09-21T09:13:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3734worker1 # [ 27.595341] nix-grpc-daemon[774]: ts=2026-09-21T09:13:37Z level=info event=scheduler_connected addr=lb:500513735worker1 # [ 27.600866] nix-grpc-daemon[774]: ts=2026-09-21T09:13:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3736worker2 # [ 27.756527] nix-grpc-daemon[781]: ts=2026-09-21T09:13:37Z level=info event=scheduler_connected addr=lb:500513737worker2 # [ 27.761873] nix-grpc-daemon[781]: ts=2026-09-21T09:13:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3738worker1 # [ 27.801926] nix-grpc-daemon[774]: ts=2026-09-21T09:13:37Z level=info event=scheduler_connected addr=lb:500513739worker1 # [ 27.807362] nix-grpc-daemon[774]: ts=2026-09-21T09:13:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3740worker2 # [ 27.963210] nix-grpc-daemon[781]: ts=2026-09-21T09:13:37Z level=info event=scheduler_connected addr=lb:500513741worker2 # [ 27.968569] nix-grpc-daemon[781]: ts=2026-09-21T09:13:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3742worker1 # [ 28.008277] nix-grpc-daemon[774]: ts=2026-09-21T09:13:37Z level=info event=scheduler_connected addr=lb:500513743worker1 # [ 28.013416] nix-grpc-daemon[774]: ts=2026-09-21T09:13:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3744worker2 # [ 28.169704] nix-grpc-daemon[781]: ts=2026-09-21T09:13:37Z level=info event=scheduler_connected addr=lb:500513745worker2 # [ 28.174780] nix-grpc-daemon[781]: ts=2026-09-21T09:13:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3746worker1 # [ 28.214351] nix-grpc-daemon[774]: ts=2026-09-21T09:13:37Z level=info event=scheduler_connected addr=lb:500513747worker1 # [ 28.219608] nix-grpc-daemon[774]: ts=2026-09-21T09:13:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3748worker2 # [ 28.376046] nix-grpc-daemon[781]: ts=2026-09-21T09:13:37Z level=info event=scheduler_connected addr=lb:500513749worker2 # [ 28.381112] nix-grpc-daemon[781]: ts=2026-09-21T09:13:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3750worker1 # [ 28.422226] nix-grpc-daemon[774]: ts=2026-09-21T09:13:38Z level=info event=scheduler_connected addr=lb:500513751worker1 # [ 28.426441] nix-grpc-daemon[774]: ts=2026-09-21T09:13:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3752worker2 # [ 28.583866] nix-grpc-daemon[781]: ts=2026-09-21T09:13:38Z level=info event=scheduler_connected addr=lb:500513753worker2 # [ 28.588171] nix-grpc-daemon[781]: ts=2026-09-21T09:13:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3754worker1 # [ 28.622754] nix-grpc-daemon[774]: ts=2026-09-21T09:13:38Z level=info event=scheduler_connected addr=lb:500513755worker1 # [ 28.628117] nix-grpc-daemon[774]: ts=2026-09-21T09:13:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3756worker2 # [ 28.784991] nix-grpc-daemon[781]: ts=2026-09-21T09:13:38Z level=info event=scheduler_connected addr=lb:500513757worker2 # [ 28.790353] nix-grpc-daemon[781]: ts=2026-09-21T09:13:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3758worker1 # [ 28.829209] nix-grpc-daemon[774]: ts=2026-09-21T09:13:38Z level=info event=scheduler_connected addr=lb:500513759worker1 # [ 28.834748] nix-grpc-daemon[774]: ts=2026-09-21T09:13:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3760worker2 # [ 28.991391] nix-grpc-daemon[781]: ts=2026-09-21T09:13:38Z level=info event=scheduler_connected addr=lb:500513761worker2 # [ 28.995260] nix-grpc-daemon[781]: ts=2026-09-21T09:13:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3762worker1 # [ 29.035467] nix-grpc-daemon[774]: ts=2026-09-21T09:13:38Z level=info event=scheduler_connected addr=lb:500513763worker1 # [ 29.038618] nix-grpc-daemon[774]: ts=2026-09-21T09:13:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3764worker2 # [ 29.196766] nix-grpc-daemon[781]: ts=2026-09-21T09:13:38Z level=info event=scheduler_connected addr=lb:500513765worker2 # [ 29.201946] nix-grpc-daemon[781]: ts=2026-09-21T09:13:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3766worker1 # [ 29.239510] nix-grpc-daemon[774]: ts=2026-09-21T09:13:38Z level=info event=scheduler_connected addr=lb:500513767worker1 # [ 29.243329] nix-grpc-daemon[774]: ts=2026-09-21T09:13:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3768worker2 # [ 29.403089] nix-grpc-daemon[781]: ts=2026-09-21T09:13:38Z level=info event=scheduler_connected addr=lb:500513769worker2 # [ 29.408697] nix-grpc-daemon[781]: ts=2026-09-21T09:13:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3770worker1 # [ 29.446040] nix-grpc-daemon[774]: ts=2026-09-21T09:13:39Z level=info event=scheduler_connected addr=lb:500513771worker1 # [ 29.450519] nix-grpc-daemon[774]: ts=2026-09-21T09:13:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3772worker2 # [ 29.611586] nix-grpc-daemon[781]: ts=2026-09-21T09:13:39Z level=info event=scheduler_connected addr=lb:500513773worker2 # [ 29.615804] nix-grpc-daemon[781]: ts=2026-09-21T09:13:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3774worker1 # [ 29.646812] nix-grpc-daemon[774]: ts=2026-09-21T09:13:39Z level=info event=scheduler_connected addr=lb:500513775worker1 # [ 29.652348] nix-grpc-daemon[774]: ts=2026-09-21T09:13:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3776worker2 # [ 29.812440] nix-grpc-daemon[781]: ts=2026-09-21T09:13:39Z level=info event=scheduler_connected addr=lb:500513777worker2 # [ 29.817535] nix-grpc-daemon[781]: ts=2026-09-21T09:13:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3778worker1 # [ 29.853431] nix-grpc-daemon[774]: ts=2026-09-21T09:13:39Z level=info event=scheduler_connected addr=lb:500513779worker1 # [ 29.858607] nix-grpc-daemon[774]: ts=2026-09-21T09:13:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3780worker2 # [ 30.018886] nix-grpc-daemon[781]: ts=2026-09-21T09:13:39Z level=info event=scheduler_connected addr=lb:500513781worker2 # [ 30.024335] nix-grpc-daemon[781]: ts=2026-09-21T09:13:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3782worker1 # [ 30.059478] nix-grpc-daemon[774]: ts=2026-09-21T09:13:39Z level=info event=scheduler_connected addr=lb:500513783worker1 # [ 30.065217] nix-grpc-daemon[774]: ts=2026-09-21T09:13:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3784worker2 # [ 30.225312] nix-grpc-daemon[781]: ts=2026-09-21T09:13:39Z level=info event=scheduler_connected addr=lb:500513785worker2 # [ 30.230390] nix-grpc-daemon[781]: ts=2026-09-21T09:13:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3786worker1 # [ 30.266085] nix-grpc-daemon[774]: ts=2026-09-21T09:13:39Z level=info event=scheduler_connected addr=lb:500513787worker1 # [ 30.271536] nix-grpc-daemon[774]: ts=2026-09-21T09:13:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3788worker2 # [ 30.431720] nix-grpc-daemon[781]: ts=2026-09-21T09:13:40Z level=info event=scheduler_connected addr=lb:500513789worker2 # [ 30.437814] nix-grpc-daemon[781]: ts=2026-09-21T09:13:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3790worker1 # [ 30.474621] nix-grpc-daemon[774]: ts=2026-09-21T09:13:40Z level=info event=scheduler_connected addr=lb:500513791worker1 # [ 30.478834] nix-grpc-daemon[774]: ts=2026-09-21T09:13:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3792worker2 # [ 30.640370] nix-grpc-daemon[781]: ts=2026-09-21T09:13:40Z level=info event=scheduler_connected addr=lb:500513793worker2 # [ 30.644588] nix-grpc-daemon[781]: ts=2026-09-21T09:13:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3794worker1 # [ 30.675460] nix-grpc-daemon[774]: ts=2026-09-21T09:13:40Z level=info event=scheduler_connected addr=lb:500513795worker1 # [ 30.681032] nix-grpc-daemon[774]: ts=2026-09-21T09:13:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3796worker2 # [ 30.841091] nix-grpc-daemon[781]: ts=2026-09-21T09:13:40Z level=info event=scheduler_connected addr=lb:500513797worker2 # [ 30.846420] nix-grpc-daemon[781]: ts=2026-09-21T09:13:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3798worker1 # [ 30.881937] nix-grpc-daemon[774]: ts=2026-09-21T09:13:40Z level=info event=scheduler_connected addr=lb:500513799worker1 # [ 30.887289] nix-grpc-daemon[774]: ts=2026-09-21T09:13:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3800worker2 # [ 31.047587] nix-grpc-daemon[781]: ts=2026-09-21T09:13:40Z level=info event=scheduler_connected addr=lb:500513801worker2 # [ 31.053088] nix-grpc-daemon[781]: ts=2026-09-21T09:13:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3802worker1 # [ 31.088542] nix-grpc-daemon[774]: ts=2026-09-21T09:13:40Z level=info event=scheduler_connected addr=lb:500513803worker1 # [ 31.093772] nix-grpc-daemon[774]: ts=2026-09-21T09:13:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3804worker2 # [ 31.254328] nix-grpc-daemon[781]: ts=2026-09-21T09:13:40Z level=info event=scheduler_connected addr=lb:500513805worker2 # [ 31.259754] nix-grpc-daemon[781]: ts=2026-09-21T09:13:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3806worker1 # [ 31.294558] nix-grpc-daemon[774]: ts=2026-09-21T09:13:40Z level=info event=scheduler_connected addr=lb:500513807worker1 # [ 31.299748] nix-grpc-daemon[774]: ts=2026-09-21T09:13:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3808worker2 # [ 31.461356] nix-grpc-daemon[781]: ts=2026-09-21T09:13:41Z level=info event=scheduler_connected addr=lb:500513809worker2 # [ 31.466799] nix-grpc-daemon[781]: ts=2026-09-21T09:13:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3810worker1 # [ 31.502400] nix-grpc-daemon[774]: ts=2026-09-21T09:13:41Z level=info event=scheduler_connected addr=lb:500513811worker1 # [ 31.506717] nix-grpc-daemon[774]: ts=2026-09-21T09:13:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3812worker2 # [ 31.670010] nix-grpc-daemon[781]: ts=2026-09-21T09:13:41Z level=info event=scheduler_connected addr=lb:500513813worker2 # [ 31.674148] nix-grpc-daemon[781]: ts=2026-09-21T09:13:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3814worker1 # [ 31.702861] nix-grpc-daemon[774]: ts=2026-09-21T09:13:41Z level=info event=scheduler_connected addr=lb:500513815worker1 # [ 31.708212] nix-grpc-daemon[774]: ts=2026-09-21T09:13:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3816worker2 # [ 31.870694] nix-grpc-daemon[781]: ts=2026-09-21T09:13:41Z level=info event=scheduler_connected addr=lb:500513817worker2 # [ 31.875880] nix-grpc-daemon[781]: ts=2026-09-21T09:13:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3818worker1 # [ 31.908935] nix-grpc-daemon[774]: ts=2026-09-21T09:13:41Z level=info event=scheduler_connected addr=lb:500513819worker1 # [ 31.914028] nix-grpc-daemon[774]: ts=2026-09-21T09:13:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3820worker2 # [ 32.077129] nix-grpc-daemon[781]: ts=2026-09-21T09:13:41Z level=info event=scheduler_connected addr=lb:500513821worker2 # [ 32.082209] nix-grpc-daemon[781]: ts=2026-09-21T09:13:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3822worker1 # [ 32.114819] nix-grpc-daemon[774]: ts=2026-09-21T09:13:41Z level=info event=scheduler_connected addr=lb:500513823worker1 # [ 32.120119] nix-grpc-daemon[774]: ts=2026-09-21T09:13:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3824worker2 # [ 32.284378] nix-grpc-daemon[781]: ts=2026-09-21T09:13:41Z level=info event=scheduler_connected addr=lb:500513825worker2 # [ 32.295511] nix-grpc-daemon[781]: ts=2026-09-21T09:13:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3826worker1 # [ 32.321056] nix-grpc-daemon[774]: ts=2026-09-21T09:13:41Z level=info event=scheduler_connected addr=lb:500513827worker1 # [ 32.326380] nix-grpc-daemon[774]: ts=2026-09-21T09:13:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3828worker2 # [ 32.492713] nix-grpc-daemon[781]: ts=2026-09-21T09:13:42Z level=info event=scheduler_connected addr=lb:500513829worker2 # [ 32.498030] nix-grpc-daemon[781]: ts=2026-09-21T09:13:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3830worker1 # [ 32.529686] nix-grpc-daemon[774]: ts=2026-09-21T09:13:42Z level=info event=scheduler_connected addr=lb:500513831worker1 # [ 32.537063] nix-grpc-daemon[774]: ts=2026-09-21T09:13:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3832worker2 # [ 32.700885] nix-grpc-daemon[781]: ts=2026-09-21T09:13:42Z level=info event=scheduler_connected addr=lb:500513833worker2 # [ 32.705343] nix-grpc-daemon[781]: ts=2026-09-21T09:13:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3834worker1 # [ 32.729648] nix-grpc-daemon[774]: ts=2026-09-21T09:13:42Z level=info event=scheduler_connected addr=lb:500513835worker1 # [ 32.734913] nix-grpc-daemon[774]: ts=2026-09-21T09:13:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3836worker2 # [ 32.902469] nix-grpc-daemon[781]: ts=2026-09-21T09:13:42Z level=info event=scheduler_connected addr=lb:500513837worker2 # [ 32.907983] nix-grpc-daemon[781]: ts=2026-09-21T09:13:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3838worker1 # [ 32.935716] nix-grpc-daemon[774]: ts=2026-09-21T09:13:42Z level=info event=scheduler_connected addr=lb:500513839worker1 # [ 32.941149] nix-grpc-daemon[774]: ts=2026-09-21T09:13:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3840worker2 # [ 33.109191] nix-grpc-daemon[781]: ts=2026-09-21T09:13:42Z level=info event=scheduler_connected addr=lb:500513841worker2 # [ 33.114480] nix-grpc-daemon[781]: ts=2026-09-21T09:13:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3842worker1 # [ 33.142017] nix-grpc-daemon[774]: ts=2026-09-21T09:13:42Z level=info event=scheduler_connected addr=lb:500513843worker1 # [ 33.147469] nix-grpc-daemon[774]: ts=2026-09-21T09:13:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3844worker2 # [ 33.315874] nix-grpc-daemon[781]: ts=2026-09-21T09:13:42Z level=info event=scheduler_connected addr=lb:500513845worker2 # [ 33.321132] nix-grpc-daemon[781]: ts=2026-09-21T09:13:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3846worker1 # [ 33.348270] nix-grpc-daemon[774]: ts=2026-09-21T09:13:42Z level=info event=scheduler_connected addr=lb:500513847worker1 # [ 33.353413] nix-grpc-daemon[774]: ts=2026-09-21T09:13:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3848worker2 # [ 33.522210] nix-grpc-daemon[781]: ts=2026-09-21T09:13:43Z level=info event=scheduler_connected addr=lb:500513849worker2 # [ 33.527492] nix-grpc-daemon[781]: ts=2026-09-21T09:13:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3850worker1 # [ 33.556194] nix-grpc-daemon[774]: ts=2026-09-21T09:13:43Z level=info event=scheduler_connected addr=lb:500513851worker1 # [ 33.560299] nix-grpc-daemon[774]: ts=2026-09-21T09:13:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3852worker2 # [ 33.730654] nix-grpc-daemon[781]: ts=2026-09-21T09:13:43Z level=info event=scheduler_connected addr=lb:500513853worker2 # [ 33.734863] nix-grpc-daemon[781]: ts=2026-09-21T09:13:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3854worker1 # [ 33.756548] nix-grpc-daemon[774]: ts=2026-09-21T09:13:43Z level=info event=scheduler_connected addr=lb:500513855worker1 # [ 33.761658] nix-grpc-daemon[774]: ts=2026-09-21T09:13:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3856worker2 # [ 33.931668] nix-grpc-daemon[781]: ts=2026-09-21T09:13:43Z level=info event=scheduler_connected addr=lb:500513857worker2 # [ 33.937356] nix-grpc-daemon[781]: ts=2026-09-21T09:13:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3858worker1 # [ 33.962498] nix-grpc-daemon[774]: ts=2026-09-21T09:13:43Z level=info event=scheduler_connected addr=lb:500513859worker1 # [ 33.967731] nix-grpc-daemon[774]: ts=2026-09-21T09:13:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3860worker2 # [ 34.138631] nix-grpc-daemon[781]: ts=2026-09-21T09:13:43Z level=info event=scheduler_connected addr=lb:500513861worker2 # [ 34.144506] nix-grpc-daemon[781]: ts=2026-09-21T09:13:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3862worker1 # [ 34.168553] nix-grpc-daemon[774]: ts=2026-09-21T09:13:43Z level=info event=scheduler_connected addr=lb:500513863worker1 # [ 34.173707] nix-grpc-daemon[774]: ts=2026-09-21T09:13:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3864worker2 # [ 34.345195] nix-grpc-daemon[781]: ts=2026-09-21T09:13:43Z level=info event=scheduler_connected addr=lb:500513865worker2 # [ 34.350327] nix-grpc-daemon[781]: ts=2026-09-21T09:13:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3866worker1 # [ 34.374855] nix-grpc-daemon[774]: ts=2026-09-21T09:13:43Z level=info event=scheduler_connected addr=lb:500513867worker1 # [ 34.380254] nix-grpc-daemon[774]: ts=2026-09-21T09:13:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3868worker2 # [ 34.551528] nix-grpc-daemon[781]: ts=2026-09-21T09:13:44Z level=info event=scheduler_connected addr=lb:500513869worker2 # [ 34.556879] nix-grpc-daemon[781]: ts=2026-09-21T09:13:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3870worker1 # [ 34.582713] nix-grpc-daemon[774]: ts=2026-09-21T09:13:44Z level=info event=scheduler_connected addr=lb:500513871worker1 # [ 34.586891] nix-grpc-daemon[774]: ts=2026-09-21T09:13:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3872worker2 # [ 34.760221] nix-grpc-daemon[781]: ts=2026-09-21T09:13:44Z level=info event=scheduler_connected addr=lb:500513873worker2 # [ 34.764325] nix-grpc-daemon[781]: ts=2026-09-21T09:13:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3874worker1 # [ 34.783568] nix-grpc-daemon[774]: ts=2026-09-21T09:13:44Z level=info event=scheduler_connected addr=lb:500513875worker1 # [ 34.788830] nix-grpc-daemon[774]: ts=2026-09-21T09:13:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3876worker2 # [ 34.961249] nix-grpc-daemon[781]: ts=2026-09-21T09:13:44Z level=info event=scheduler_connected addr=lb:500513877worker2 # [ 34.966506] nix-grpc-daemon[781]: ts=2026-09-21T09:13:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3878worker1 # [ 34.989675] nix-grpc-daemon[774]: ts=2026-09-21T09:13:44Z level=info event=scheduler_connected addr=lb:500513879worker1 # [ 34.994886] nix-grpc-daemon[774]: ts=2026-09-21T09:13:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3880worker2 # [ 35.167691] nix-grpc-daemon[781]: ts=2026-09-21T09:13:44Z level=info event=scheduler_connected addr=lb:500513881worker2 # [ 35.173119] nix-grpc-daemon[781]: ts=2026-09-21T09:13:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3882worker1 # [ 35.195694] nix-grpc-daemon[774]: ts=2026-09-21T09:13:44Z level=info event=scheduler_connected addr=lb:500513883worker1 # [ 35.200953] nix-grpc-daemon[774]: ts=2026-09-21T09:13:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3884worker2 # [ 35.374322] nix-grpc-daemon[781]: ts=2026-09-21T09:13:44Z level=info event=scheduler_connected addr=lb:500513885worker2 # [ 35.379683] nix-grpc-daemon[781]: ts=2026-09-21T09:13:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3886worker1 # [ 35.401591] nix-grpc-daemon[774]: ts=2026-09-21T09:13:44Z level=info event=scheduler_connected addr=lb:500513887worker1 # [ 35.406700] nix-grpc-daemon[774]: ts=2026-09-21T09:13:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3888worker2 # [ 35.582359] nix-grpc-daemon[781]: ts=2026-09-21T09:13:45Z level=info event=scheduler_connected addr=lb:500513889worker2 # [ 35.593003] nix-grpc-daemon[781]: ts=2026-09-21T09:13:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3890worker1 # [ 35.609708] nix-grpc-daemon[774]: ts=2026-09-21T09:13:45Z level=info event=scheduler_connected addr=lb:500513891worker1 # [ 35.614058] nix-grpc-daemon[774]: ts=2026-09-21T09:13:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3892worker2 # [ 35.791037] nix-grpc-daemon[781]: ts=2026-09-21T09:13:45Z level=info event=scheduler_connected addr=lb:500513893worker2 # [ 35.795242] nix-grpc-daemon[781]: ts=2026-09-21T09:13:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3894worker1 # [ 35.810481] nix-grpc-daemon[774]: ts=2026-09-21T09:13:45Z level=info event=scheduler_connected addr=lb:500513895worker1 # [ 35.819142] nix-grpc-daemon[774]: ts=2026-09-21T09:13:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3896worker2 # [ 35.992069] nix-grpc-daemon[781]: ts=2026-09-21T09:13:45Z level=info event=scheduler_connected addr=lb:500513897worker2 # [ 35.997114] nix-grpc-daemon[781]: ts=2026-09-21T09:13:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3898worker1 # [ 36.019561] nix-grpc-daemon[774]: ts=2026-09-21T09:13:45Z level=info event=scheduler_connected addr=lb:500513899worker1 # [ 36.025134] nix-grpc-daemon[774]: ts=2026-09-21T09:13:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3900worker2 # [ 36.198183] nix-grpc-daemon[781]: ts=2026-09-21T09:13:45Z level=info event=scheduler_connected addr=lb:500513901worker2 # [ 36.203401] nix-grpc-daemon[781]: ts=2026-09-21T09:13:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3902worker1 # [ 36.225982] nix-grpc-daemon[774]: ts=2026-09-21T09:13:45Z level=info event=scheduler_connected addr=lb:500513903worker1 # [ 36.231441] nix-grpc-daemon[774]: ts=2026-09-21T09:13:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3904worker2 # [ 36.404600] nix-grpc-daemon[781]: ts=2026-09-21T09:13:45Z level=info event=scheduler_connected addr=lb:500513905worker2 # [ 36.409690] nix-grpc-daemon[781]: ts=2026-09-21T09:13:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3906worker1 # [ 36.432225] nix-grpc-daemon[774]: ts=2026-09-21T09:13:46Z level=info event=scheduler_connected addr=lb:500513907worker1 # [ 36.437853] nix-grpc-daemon[774]: ts=2026-09-21T09:13:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3908worker2 # [ 36.610878] nix-grpc-daemon[781]: ts=2026-09-21T09:13:46Z level=info event=scheduler_connected addr=lb:500513909worker2 # [ 36.616375] nix-grpc-daemon[781]: ts=2026-09-21T09:13:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3910worker1 # [ 36.640369] nix-grpc-daemon[774]: ts=2026-09-21T09:13:46Z level=info event=scheduler_connected addr=lb:500513911worker1 # [ 36.644515] nix-grpc-daemon[774]: ts=2026-09-21T09:13:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3912worker2 # [ 36.819366] nix-grpc-daemon[781]: ts=2026-09-21T09:13:46Z level=info event=scheduler_connected addr=lb:500513913worker2 # [ 36.823653] nix-grpc-daemon[781]: ts=2026-09-21T09:13:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3914worker1 # [ 36.840910] nix-grpc-daemon[774]: ts=2026-09-21T09:13:46Z level=info event=scheduler_connected addr=lb:500513915worker1 # [ 36.846291] nix-grpc-daemon[774]: ts=2026-09-21T09:13:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3916worker2 # [ 37.020411] nix-grpc-daemon[781]: ts=2026-09-21T09:13:46Z level=info event=scheduler_connected addr=lb:500513917worker2 # [ 37.025726] nix-grpc-daemon[781]: ts=2026-09-21T09:13:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3918worker1 # [ 37.047081] nix-grpc-daemon[774]: ts=2026-09-21T09:13:46Z level=info event=scheduler_connected addr=lb:500513919worker1 # [ 37.052606] nix-grpc-daemon[774]: ts=2026-09-21T09:13:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3920worker2 # [ 37.226795] nix-grpc-daemon[781]: ts=2026-09-21T09:13:46Z level=info event=scheduler_connected addr=lb:500513921worker2 # [ 37.232190] nix-grpc-daemon[781]: ts=2026-09-21T09:13:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3922worker1 # [ 37.253490] nix-grpc-daemon[774]: ts=2026-09-21T09:13:46Z level=info event=scheduler_connected addr=lb:500513923worker1 # [ 37.258842] nix-grpc-daemon[774]: ts=2026-09-21T09:13:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3924worker2 # [ 37.433350] nix-grpc-daemon[781]: ts=2026-09-21T09:13:47Z level=info event=scheduler_connected addr=lb:500513925worker2 # [ 37.438710] nix-grpc-daemon[781]: ts=2026-09-21T09:13:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3926worker1 # [ 37.459595] nix-grpc-daemon[774]: ts=2026-09-21T09:13:47Z level=info event=scheduler_connected addr=lb:500513927worker1 # [ 37.465159] nix-grpc-daemon[774]: ts=2026-09-21T09:13:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3928worker2 # [ 37.639607] nix-grpc-daemon[781]: ts=2026-09-21T09:13:47Z level=info event=scheduler_connected addr=lb:500513929worker2 # [ 37.645119] nix-grpc-daemon[781]: ts=2026-09-21T09:13:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3930worker1 # [ 37.667738] nix-grpc-daemon[774]: ts=2026-09-21T09:13:47Z level=info event=scheduler_connected addr=lb:500513931worker1 # [ 37.672150] nix-grpc-daemon[774]: ts=2026-09-21T09:13:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3932worker2 # [ 37.848387] nix-grpc-daemon[781]: ts=2026-09-21T09:13:47Z level=info event=scheduler_connected addr=lb:500513933worker2 # [ 37.852544] nix-grpc-daemon[781]: ts=2026-09-21T09:13:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3934worker1 # [ 37.868107] nix-grpc-daemon[774]: ts=2026-09-21T09:13:47Z level=info event=scheduler_connected addr=lb:500513935worker1 # [ 37.873374] nix-grpc-daemon[774]: ts=2026-09-21T09:13:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3936worker2 # [ 38.049390] nix-grpc-daemon[781]: ts=2026-09-21T09:13:47Z level=info event=scheduler_connected addr=lb:500513937worker2 # [ 38.054669] nix-grpc-daemon[781]: ts=2026-09-21T09:13:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3938worker1 # [ 38.074323] nix-grpc-daemon[774]: ts=2026-09-21T09:13:47Z level=info event=scheduler_connected addr=lb:500513939worker1 # [ 38.079669] nix-grpc-daemon[774]: ts=2026-09-21T09:13:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3940worker2 # [ 38.255846] nix-grpc-daemon[781]: ts=2026-09-21T09:13:47Z level=info event=scheduler_connected addr=lb:500513941worker2 # [ 38.261548] nix-grpc-daemon[781]: ts=2026-09-21T09:13:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3942worker1 # [ 38.280294] nix-grpc-daemon[774]: ts=2026-09-21T09:13:47Z level=info event=scheduler_connected addr=lb:500513943worker1 # [ 38.285605] nix-grpc-daemon[774]: ts=2026-09-21T09:13:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3944worker2 # [ 38.462192] nix-grpc-daemon[781]: ts=2026-09-21T09:13:48Z level=info event=scheduler_connected addr=lb:500513945worker2 # [ 38.467790] nix-grpc-daemon[781]: ts=2026-09-21T09:13:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3946worker1 # [ 38.486208] nix-grpc-daemon[774]: ts=2026-09-21T09:13:48Z level=info event=scheduler_connected addr=lb:500513947worker1 # [ 38.491642] nix-grpc-daemon[774]: ts=2026-09-21T09:13:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3948worker1 # [ 38.552135] nix-grpc-daemon[774]: ts=2026-09-21T09:13:48Z level=info event=scheduler_lock_lost error="error: http://lb:5751/api/farm/lead: Timeout was reached"3949worker2 # [ 38.669064] nix-grpc-daemon[781]: ts=2026-09-21T09:13:48Z level=info event=scheduler_connected addr=lb:500513950worker2 # [ 38.674264] nix-grpc-daemon[781]: ts=2026-09-21T09:13:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3951worker1 # [ 38.694293] nix-grpc-daemon[774]: ts=2026-09-21T09:13:48Z level=info event=scheduler_connected addr=lb:500513952worker1 # [ 38.698605] nix-grpc-daemon[774]: ts=2026-09-21T09:13:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3953worker2 # [ 38.706541] nix-grpc-daemon[781]: ts=2026-09-21T09:13:48Z level=info event=scheduler_lock_lost error="error: http://lb:5751/api/farm/lead: Timeout was reached"3954worker2 # [ 38.877016] nix-grpc-daemon[781]: ts=2026-09-21T09:13:48Z level=info event=scheduler_connected addr=lb:500513955worker2 # [ 38.885830] nix-grpc-daemon[781]: ts=2026-09-21T09:13:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3956worker1 # [ 38.894939] nix-grpc-daemon[774]: ts=2026-09-21T09:13:48Z level=info event=scheduler_connected addr=lb:500513957worker1 # [ 38.900589] nix-grpc-daemon[774]: ts=2026-09-21T09:13:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3958worker2 # [ 39.083090] nix-grpc-daemon[781]: ts=2026-09-21T09:13:48Z level=info event=scheduler_connected addr=lb:500513959worker2 # [ 39.088646] nix-grpc-daemon[781]: ts=2026-09-21T09:13:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3960worker1 # [ 39.101408] nix-grpc-daemon[774]: ts=2026-09-21T09:13:48Z level=info event=scheduler_connected addr=lb:500513961worker1 # [ 39.106735] nix-grpc-daemon[774]: ts=2026-09-21T09:13:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3962worker2 # [ 39.289732] nix-grpc-daemon[781]: ts=2026-09-21T09:13:48Z level=info event=scheduler_connected addr=lb:500513963worker2 # [ 39.295105] nix-grpc-daemon[781]: ts=2026-09-21T09:13:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3964worker1 # [ 39.307938] nix-grpc-daemon[774]: ts=2026-09-21T09:13:48Z level=info event=scheduler_connected addr=lb:500513965worker1 # [ 39.313453] nix-grpc-daemon[774]: ts=2026-09-21T09:13:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3966worker2 # [ 39.496426] nix-grpc-daemon[781]: ts=2026-09-21T09:13:49Z level=info event=scheduler_connected addr=lb:500513967worker2 # [ 39.502092] nix-grpc-daemon[781]: ts=2026-09-21T09:13:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3968worker1 # [ 39.513990] nix-grpc-daemon[774]: ts=2026-09-21T09:13:49Z level=info event=scheduler_connected addr=lb:500513969worker1 # [ 39.519666] nix-grpc-daemon[774]: ts=2026-09-21T09:13:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3970worker2 # [ 39.702962] nix-grpc-daemon[781]: ts=2026-09-21T09:13:49Z level=info event=scheduler_connected addr=lb:500513971worker2 # [ 39.710572] nix-grpc-daemon[781]: ts=2026-09-21T09:13:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3972worker1 # [ 39.722591] nix-grpc-daemon[774]: ts=2026-09-21T09:13:49Z level=info event=scheduler_connected addr=lb:500513973worker1 # [ 39.726841] nix-grpc-daemon[774]: ts=2026-09-21T09:13:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3974worker2 # [ 39.912149] nix-grpc-daemon[781]: ts=2026-09-21T09:13:49Z level=info event=scheduler_connected addr=lb:500513975worker2 # [ 39.916294] nix-grpc-daemon[781]: ts=2026-09-21T09:13:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3976worker1 # [ 39.922745] nix-grpc-daemon[774]: ts=2026-09-21T09:13:49Z level=info event=scheduler_connected addr=lb:500513977worker1 # [ 39.928082] nix-grpc-daemon[774]: ts=2026-09-21T09:13:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3978worker2 # [ 40.112843] nix-grpc-daemon[781]: ts=2026-09-21T09:13:49Z level=info event=scheduler_connected addr=lb:500513979worker2 # [ 40.118052] nix-grpc-daemon[781]: ts=2026-09-21T09:13:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3980worker1 # [ 40.128902] nix-grpc-daemon[774]: ts=2026-09-21T09:13:49Z level=info event=scheduler_connected addr=lb:500513981worker1 # [ 40.134067] nix-grpc-daemon[774]: ts=2026-09-21T09:13:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3982worker2 # [ 40.319352] nix-grpc-daemon[781]: ts=2026-09-21T09:13:49Z level=info event=scheduler_connected addr=lb:500513983worker2 # [ 40.324832] nix-grpc-daemon[781]: ts=2026-09-21T09:13:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3984worker1 # [ 40.334888] nix-grpc-daemon[774]: ts=2026-09-21T09:13:49Z level=info event=scheduler_connected addr=lb:500513985worker1 # [ 40.340086] nix-grpc-daemon[774]: ts=2026-09-21T09:13:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3986worker2 # [ 40.525981] nix-grpc-daemon[781]: ts=2026-09-21T09:13:50Z level=info event=scheduler_connected addr=lb:500513987worker2 # [ 40.531313] nix-grpc-daemon[781]: ts=2026-09-21T09:13:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3988worker1 # [ 40.541044] nix-grpc-daemon[774]: ts=2026-09-21T09:13:50Z level=info event=scheduler_connected addr=lb:500513989worker1 # [ 40.546619] nix-grpc-daemon[774]: ts=2026-09-21T09:13:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3990worker2 # [ 40.732723] nix-grpc-daemon[781]: ts=2026-09-21T09:13:50Z level=info event=scheduler_connected addr=lb:500513991worker2 # [ 40.738063] nix-grpc-daemon[781]: ts=2026-09-21T09:13:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3992worker1 # [ 40.749090] nix-grpc-daemon[774]: ts=2026-09-21T09:13:50Z level=info event=scheduler_connected addr=lb:500513993worker1 # [ 40.753230] nix-grpc-daemon[774]: ts=2026-09-21T09:13:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3994worker2 # [ 40.940861] nix-grpc-daemon[781]: ts=2026-09-21T09:13:50Z level=info event=scheduler_connected addr=lb:500513995worker2 # [ 40.945652] nix-grpc-daemon[781]: ts=2026-09-21T09:13:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3996worker1 # [ 40.949502] nix-grpc-daemon[774]: ts=2026-09-21T09:13:50Z level=info event=scheduler_connected addr=lb:500513997worker1 # [ 40.954623] nix-grpc-daemon[774]: ts=2026-09-21T09:13:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3998worker2 # [ 41.146888] nix-grpc-daemon[781]: ts=2026-09-21T09:13:50Z level=info event=scheduler_connected addr=lb:500513999worker2 # [ 41.152344] nix-grpc-daemon[781]: ts=2026-09-21T09:13:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4000worker1 # [ 41.155404] nix-grpc-daemon[774]: ts=2026-09-21T09:13:50Z level=info event=scheduler_connected addr=lb:500514001worker1 # [ 41.160691] nix-grpc-daemon[774]: ts=2026-09-21T09:13:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4002worker2 # [ 41.353544] nix-grpc-daemon[781]: ts=2026-09-21T09:13:50Z level=info event=scheduler_connected addr=lb:500514003worker2 # [ 41.358862] nix-grpc-daemon[781]: ts=2026-09-21T09:13:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4004worker1 # [ 41.361663] nix-grpc-daemon[774]: ts=2026-09-21T09:13:50Z level=info event=scheduler_connected addr=lb:500514005worker1 # [ 41.366978] nix-grpc-daemon[774]: ts=2026-09-21T09:13:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4006worker2 # [ 41.560743] nix-grpc-daemon[781]: ts=2026-09-21T09:13:51Z level=info event=scheduler_connected addr=lb:500514007worker2 # [ 41.566213] nix-grpc-daemon[781]: ts=2026-09-21T09:13:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4008worker1 # [ 41.567997] nix-grpc-daemon[774]: ts=2026-09-21T09:13:51Z level=info event=scheduler_connected addr=lb:500514009worker1 # [ 41.573439] nix-grpc-daemon[774]: ts=2026-09-21T09:13:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4010worker2 # [ 41.767341] nix-grpc-daemon[781]: ts=2026-09-21T09:13:51Z level=info event=scheduler_connected addr=lb:500514011worker2 # [ 41.772913] nix-grpc-daemon[781]: ts=2026-09-21T09:13:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4012worker1 # [ 41.775800] nix-grpc-daemon[774]: ts=2026-09-21T09:13:51Z level=info event=scheduler_connected addr=lb:500514013worker1 # [ 41.780127] nix-grpc-daemon[774]: ts=2026-09-21T09:13:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4014worker2 # [ 41.975115] nix-grpc-daemon[781]: ts=2026-09-21T09:13:51Z level=info event=scheduler_connected addr=lb:500514015worker2 # [ 41.980540] nix-grpc-daemon[781]: ts=2026-09-21T09:13:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4016worker1 # [ 41.976251] nix-grpc-daemon[774]: ts=2026-09-21T09:13:51Z level=info event=scheduler_connected addr=lb:500514017worker1 # [ 41.981338] nix-grpc-daemon[774]: ts=2026-09-21T09:13:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4018lb # [ 42.086647] rustfs-bucket-start[1117]: mb s3://farm4019lb # [ 42.101390] systemd[1]: Finished rustfs-bucket.service.4020lb # [ 42.110250] systemd[1]: Starting niks3 server...4021worker2 # [ 42.181700] nix-grpc-daemon[781]: ts=2026-09-21T09:13:51Z level=info event=scheduler_connected addr=lb:500514022worker2 # [ 42.187051] nix-grpc-daemon[781]: ts=2026-09-21T09:13:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4023worker1 # [ 42.182188] nix-grpc-daemon[774]: ts=2026-09-21T09:13:51Z level=info event=scheduler_connected addr=lb:500514024worker1 # [ 42.187307] nix-grpc-daemon[774]: ts=2026-09-21T09:13:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4025lb # [ 42.332160] postgres[1134]: [1134] ERROR: relation "goose_db_version" does not exist at character 364026lb # [ 42.333524] postgres[1134]: [1134] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC4027lb # [ 42.362369] niks3-server[1127]: 2026/09/21 09:13:51 OK 20241026095416_initial_model.sql (19.09ms)4028lb # [ 42.366140] niks3-server[1127]: 2026/09/21 09:13:51 OK 20251210153512_drop_unused_gin_index.sql (4.15ms)4029lb # [ 42.371500] niks3-server[1127]: 2026/09/21 09:13:51 OK 20251218171726_add_pins.sql (5.31ms)4030lb # [ 42.376304] niks3-server[1127]: 2026/09/21 09:13:51 OK 20260628120000_add_object_size_and_stats.sql (4.77ms)4031lb # [ 42.381613] niks3-server[1127]: 2026/09/21 09:13:51 OK 20260905000000_add_claims.sql (5.28ms)4032lb # [ 42.386087] niks3-server[1127]: 2026/09/21 09:13:51 OK 20260920000000_drop_claims.sql (4.35ms)4033lb # [ 42.388150] niks3-server[1127]: 2026/09/21 09:13:51 goose: successfully migrated database to version: 202609200000004034lb # [ 42.392733] niks3-server[1127]: 2026/09/21 09:13:51 OK 1_commit_pending_closure.sql (6.63ms)4035lb # [ 42.395262] niks3-server[1127]: 2026/09/21 09:13:51 OK 2_object_stats_trigger.sql (2.4ms)4036lb # [ 42.396791] niks3-server[1127]: 2026/09/21 09:13:51 goose: up to current file version: 24037lb # [ 42.400343] niks3-server[1127]: 2026/09/21 09:13:52 INFO Loaded signing key name=farm-test-1 path=/nix/store/8lzz3wcf17v05vfdp1zcfaa9hipw3f70-key4038worker2 # [ 42.388494] nix-grpc-daemon[781]: ts=2026-09-21T09:13:51Z level=info event=scheduler_connected addr=lb:500514039worker2 # [ 42.393671] nix-grpc-daemon[781]: ts=2026-09-21T09:13:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4040worker1 # [ 42.389475] nix-grpc-daemon[774]: ts=2026-09-21T09:13:51Z level=info event=scheduler_connected addr=lb:500514041worker1 # [ 42.394209] nix-grpc-daemon[774]: ts=2026-09-21T09:13:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4042lb # [ 42.443837] niks3-server[1127]: 2026/09/21 09:13:52 INFO Read proxy enabled — serving cache objects from S34043lb # [ 42.446681] niks3-server[1127]: 2026/09/21 09:13:52 INFO Using socket-activated listener address=0.0.0.0:57514044lb # [ 42.449687] niks3-server[1127]: 2026/09/21 09:13:52 INFO systemd watchdog enabled interval=15s4045lb # [ 42.454276] systemd[1]: Started niks3 server.4046lb # [ 42.455454] niks3-server[1127]: 2026/09/21 09:13:52 INFO Starting HTTP server address=0.0.0.0:57514047lb # [ 42.461379] systemd[1]: Reached target Multi-User System.4048lb # [ 42.462169] systemd[1]: Startup finished in 967ms (kernel) + 5.427s (initrd) + 36.057s (userspace) = 42.452s.4049worker2 # [ 42.454691] nix-grpc-daemon[781]: ts=2026-09-21T09:13:52Z level=info event=scheduler_take_over4050lb # [ 42.475050] niks3-server[1127]: 2026/09/21 09:13:52 INFO lead: acquired remote=192.168.1.4:485764051worker2 # [ 42.594629] nix-grpc-daemon[781]: ts=2026-09-21T09:13:52Z level=info event=scheduler_connected addr=lb:500514052worker1 # [ 42.595564] nix-grpc-daemon[774]: ts=2026-09-21T09:13:52Z level=info event=scheduler_connected addr=lb:500514053worker2 # [ 42.606452] nix-grpc-daemon[781]: ts=2026-09-21T09:13:52Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:344744054worker2 # [ 42.612582] nix-grpc-daemon[781]: ts=2026-09-21T09:13:52Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=04055worker2 # [ 42.625992] nix-grpc-daemon[781]: ts=2026-09-21T09:13:52Z level=info event=oidc_jwks provider=mock keys=14056worker2 # [ 42.630465] nix-grpc-daemon[781]: ts=2026-09-21T09:13:52Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:344744057worker2 # [ 42.635861] nix-grpc-daemon[781]: ts=2026-09-21T09:13:52Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=04058lb: (finished: waiting for unit niks3.service, in 28.26 seconds)4059lb: waiting for TCP port 5751 on localhost4060lb # Connection to localhost (127.0.0.1) 5751 port [tcp/*] succeeded!4061lb: (finished: waiting for TCP port 5751 on localhost, in 0.09 seconds)4062lb: waiting for unit envoy.service4063lb: (finished: waiting for unit envoy.service, in 0.07 seconds)4064lb: waiting for TCP port 50051 on localhost4065lb # Connection to localhost (::1) 50051 port [tcp/*] succeeded!4066lb # [ 43.064775] envoy[913]: conn peer=[::1]:57190 tls=TLSv1.2 subject="-" sni=- flags=- tls_fail="-" rx=0 tx=0 ms=34067lb: (finished: waiting for TCP port 50051 on localhost, in 0.04 seconds)4068worker2 # [ 43.235768] systemd[1]: Starting scheduler-token.service...4069worker2 # [ 43.333702] systemd[1]: scheduler-token.service: Deactivated successfully.4070worker2 # [ 43.337721] systemd[1]: Finished scheduler-token.service.4071worker2 # [ 43.341885] systemd[1]: scheduler-token.service: Consumed 41ms CPU time over 101ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.4072lb: waiting until exactly no endpoints of aarch64-linux are unhealthy4073lb: must succeed: curl -sf localhost:9901/clusters4074lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.10 seconds)4075lb: (finished: waiting until exactly no endpoints of aarch64-linux are unhealthy, in 0.10 seconds)4076subtest: one node schedules and both builders hold a WorkerSession on it4077lb: must succeed: curl -sf localhost:9901/clusters4078lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4079worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4080worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)4081lb: must succeed: curl -sf localhost:9901/clusters4082lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)4083lb: must succeed: curl -sf localhost:9901/clusters4084lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4085lb: must succeed: curl -sf localhost:9901/clusters4086lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4087lb: must succeed: curl -sf localhost:9901/clusters4088lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4089worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open'4090worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open', in 0.07 seconds)4091worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4092worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)4093lb: must succeed: curl -sf localhost:9901/clusters4094lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)4095lb: must succeed: curl -sf localhost:9901/clusters4096lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)4097worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4098worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.09 seconds)4099worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4100worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)4101worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4102worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)4103worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4104worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)4105(finished: subtest: one node schedules and both builders hold a WorkerSession on it, in 0.88 seconds)4106subtest: balancer auth: client cert, bearer token, nothing, unknown CN4107worker1 # [ 44.713327] nix-grpc-daemon[774]: ts=2026-09-21T09:13:54Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:353504108worker1 # [ 44.733041] systemd[1]: Started Nix Daemon.4109worker1 # [ 45.032635] nix-daemon[843]: accepted connection from pid 774, user nix-grpc-daemon (trusted)4110worker1 # [ 45.068630] nix-grpc-daemon[774]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 25 ms (attempt 1/5)4111worker1 # [ 45.096577] nix-grpc-daemon[774]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 109 ms (attempt 2/5)4112worker1 # [ 45.208858] nix-grpc-daemon[774]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 90 ms (attempt 3/5)4113worker1 # [ 45.306622] nix-grpc-daemon[774]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 21 ms (attempt 4/5)4114worker1 # [ 45.337874] nix-grpc-daemon[774]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4115worker2 # [ 45.385255] nix-grpc-daemon[781]: ts=2026-09-21T09:13:54Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:344904116worker2 # [ 45.402321] systemd[1]: Started Nix Daemon.4117worker2 # [ 45.770406] nix-daemon[870]: accepted connection from pid 781, user nix-grpc-daemon (trusted)4118worker2 # [ 45.802862] nix-grpc-daemon[781]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 45 ms (attempt 1/5)4119worker2 # [ 45.852178] nix-grpc-daemon[781]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 173 ms (attempt 2/5)4120worker2 # [ 46.025696] nix-grpc-daemon[781]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 204 ms (attempt 3/5)4121worker2 # [ 46.237445] nix-grpc-daemon[781]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 217 ms (attempt 4/5)4122worker2 # [ 46.465816] nix-grpc-daemon[781]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4123worker2 # [ 46.481658] nix-grpc-daemon[781]: ts=2026-09-21T09:13:56Z level=debug event=path_info_miss path=p6p9342ifnppad60sm4s8zcrc0dk790i-nixos-system-client-test error="error: path '/nix/store/p6p9342ifnppad60sm4s8zcrc0dk790i-nixos-system-client-test' is required, but there is no substituter that can build it"4124worker2 # [ 46.495351] nix-grpc-daemon[781]: ts=2026-09-21T09:13:56Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:344904125worker2 # [ 46.507839] nix-grpc-daemon[781]: ts=2026-09-21T09:13:56Z level=debug event=path_info_miss path=p6p9342ifnppad60sm4s8zcrc0dk790i-nixos-system-client-test error="error: path '/nix/store/p6p9342ifnppad60sm4s8zcrc0dk790i-nixos-system-client-test' is required, but there is no substituter that can build it"4126client: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt4127client: (finished: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt, in 0.10 seconds)4128client: must succeed: install -D /nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key4129client: (finished: must succeed: install -D /nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key, in 0.05 seconds)4130worker1 # [ 46.957897] nix-grpc-daemon[774]: ts=2026-09-21T09:13:56Z level=info event=oidc_jwks provider=mock keys=14131worker1 # [ 46.958258] nix-grpc-daemon[774]: ts=2026-09-21T09:13:56Z level=debug event=rpc_start method=QueryMissing cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:353504132worker1 # [ 46.972547] nix-grpc-daemon[774]: ts=2026-09-21T09:13:56Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:353504133worker1 # [ 46.987030] nix-grpc-daemon[774]: ts=2026-09-21T09:13:56Z level=debug event=path_info_miss path=p6p9342ifnppad60sm4s8zcrc0dk790i-nixos-system-client-test error="error: path '/nix/store/p6p9342ifnppad60sm4s8zcrc0dk790i-nixos-system-client-test' is required, but there is no substituter that can build it"4134worker1 # [ 47.002098] nix-grpc-daemon[774]: ts=2026-09-21T09:13:56Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:353504135worker1 # [ 47.009708] nix-grpc-daemon[774]: ts=2026-09-21T09:13:56Z level=debug event=path_info_miss path=p6p9342ifnppad60sm4s8zcrc0dk790i-nixos-system-client-test error="error: path '/nix/store/p6p9342ifnppad60sm4s8zcrc0dk790i-nixos-system-client-test' is required, but there is no substituter that can build it"4136client: must succeed: rm -r /var/lib/nix-grpc-store4137client: (finished: must succeed: rm -r /var/lib/nix-grpc-store, in 0.04 seconds)4138worker2 # [ 47.324128] nix-grpc-daemon[781]: ts=2026-09-21T09:13:56Z level=info event=denied method=QueryMissing cn=- role=none4139worker1 # [ 47.581583] nix-grpc-daemon[774]: ts=2026-09-21T09:13:57Z level=info event=denied method=QueryMissing cn=stranger role=none4140(finished: subtest: balancer auth: client cert, bearer token, nothing, unknown CN, in 3.29 seconds)4141subtest: DAG build is scheduled across workers and lands in the cache4142client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 >&24143worker1 # [ 47.850927] nix-grpc-daemon[774]: ts=2026-09-21T09:13:57Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:353504144client # don't know how to build these paths:4145client # /nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv4146worker2 # [ 47.873455] nix-grpc-daemon[781]: ts=2026-09-21T09:13:57Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:344904147worker2 # [ 47.913344] nix-grpc-daemon[781]: ts=2026-09-21T09:13:57Z level=debug event=want client=1 drv=619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv cached=04148worker2 # [ 47.919402] nix-grpc-daemon[781]: ts=2026-09-21T09:13:57Z level=debug event=assigned drv=619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv worker=192.168.1.3:50051 clients=14149worker1 # [ 47.964553] nix-grpc-daemon[774]: ts=2026-09-21T09:13:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:353504150worker1 # [ 47.981416] nix-grpc-daemon[774]: ts=2026-09-21T09:13:57Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:353504151client # copying 1 paths to 'lb:50051'...4152worker1 # [ 47.998484] nix-grpc-daemon[774]: ts=2026-09-21T09:13:57Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:353504153client # copying path '/nix/store/619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv' to 'lb:50051'...4154worker1 # [ 48.029730] nix-daemon[843]: accepted connection from pid 774, user nix-grpc-daemon (trusted)4155worker1 # [ 48.063978] nix-grpc-daemon[774]: ts=2026-09-21T09:13:57Z level=info event=niks3_push_started pid=8534156worker1 # [ 48.345675] nix-grpc-daemon[853]: time=2026-09-21T09:13:57.939Z level=INFO msg="Configuring client TLS" cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker1.pem key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker1.key ca=""4157worker1 # [ 48.542506] nix-daemon[843]: accepted connection from pid 866, user nix-grpc-daemon (trusted)4158worker1 # [ 48.561238] nix-daemon[843]: reaped child process 869, status = succeeded4159worker1 # [ 48.734692] nix-daemon[843]: accepted connection from pid 871, user nix-grpc-daemon (trusted)4160worker1 # [ 48.755022] nix-daemon[843]: reaped child process 874, status = succeeded4161lb # [ 48.793792] niks3-server[1127]: 2026/09/21 09:13:58 INFO Received uploads request method=POST path=/api/pending_closures4162worker1 # [ 48.794194] nix-grpc-daemon[853]: time=2026-09-21T09:13:58.389Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4163worker1 # [ 48.797872] nix-grpc-daemon[853]: time=2026-09-21T09:13:58.391Z level=INFO msg="Uploading 619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv (504B)"4164lb # [ 48.873656] niks3-server[1127]: 2026/09/21 09:13:58 INFO Registered completed upload object_key=nar/1il50j4vra11xfnqgjhjgrjn1753frlhxnss6mh1dmp5ww3is34w.nar.zst4165lb # [ 48.884314] niks3-server[1127]: 2026/09/21 09:13:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign4166lb # [ 48.891762] niks3-server[1127]: 2026/09/21 09:13:58 INFO Registered completed upload object_key=619ccx60pfv52kv4hhgqpj1ivdmfcr5r.ls4167worker1 # [ 48.866657] nix-grpc-daemon[853]: time=2026-09-21T09:13:58.461Z level=INFO msg="Uploading 1 narinfos"4168lb # [ 48.897891] niks3-server[1127]: 2026/09/21 09:13:58 INFO Signed narinfos id=1 count=14169lb # [ 48.906914] niks3-server[1127]: 2026/09/21 09:13:58 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete4170lb # [ 48.915071] niks3-server[1127]: 2026/09/21 09:13:58 INFO Registered completed upload object_key=619ccx60pfv52kv4hhgqpj1ivdmfcr5r.narinfo4171lb # [ 48.927971] niks3-server[1127]: 2026/09/21 09:13:58 INFO Completed upload id=14172worker1 # [ 48.902909] nix-grpc-daemon[853]: time=2026-09-21T09:13:58.498Z level=INFO msg="Upload complete. (549ms)"4173worker1 # [ 48.916667] nix-grpc-daemon[774]: ts=2026-09-21T09:13:58Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35350 duration_s=0 paths=1 nar_bytes_in=5044174worker1 # [ 48.933122] nix-grpc-daemon[774]: ts=2026-09-21T09:13:58Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:353504175worker1 # [ 48.943706] nix-daemon[843]: accepted connection from pid 774, user nix-grpc-daemon (trusted)4176worker1 # [ 48.955903] nix-daemon[843]: reaped child process 851, status = succeeded4177client # building '/nix/store/619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv' on lb:50051...4178client # farm-a> node-a: building 619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv4179worker1 # [ 48.966091] nix-daemon[843]: accepted connection from pid 774, user nix-grpc-daemon (trusted)4180client # farm-a> LOG-farm-a-t14181worker1 # [ 49.988231] nix-daemon[843]: reaped child process 878, status = succeeded4182worker1 # [ 50.152961] nix-daemon[843]: accepted connection from pid 893, user nix-grpc-daemon (trusted)4183worker1 # [ 50.171025] nix-daemon[843]: reaped child process 896, status = succeeded4184lb # [ 50.208694] niks3-server[1127]: 2026/09/21 09:13:59 INFO Received uploads request method=POST path=/api/pending_closures4185worker1 # [ 50.196728] nix-grpc-daemon[853]: time=2026-09-21T09:13:59.791Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4186worker1 # [ 50.198617] nix-grpc-daemon[853]: time=2026-09-21T09:13:59.791Z level=INFO msg="Uploading dg404zlvxfrspnxl6cc1qkc6i83m5a3a-farm-a (128B)"4187lb # [ 50.257403] niks3-server[1127]: 2026/09/21 09:13:59 INFO Registered completed upload object_key=log/619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv4188lb # [ 50.269246] niks3-server[1127]: 2026/09/21 09:13:59 INFO Registered completed upload object_key=nar/1rmil4ja9jzfwn6l5hwbir70b6plx99sgk0pnjjy377ngbc3vz28.nar.zst4189lb # [ 50.278906] niks3-server[1127]: 2026/09/21 09:13:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign4190lb # [ 50.285152] niks3-server[1127]: 2026/09/21 09:13:59 INFO Signed narinfos id=2 count=14191worker1 # [ 50.260995] nix-grpc-daemon[853]: time=2026-09-21T09:13:59.855Z level=INFO msg="Uploading 1 narinfos"4192lb # [ 50.288292] niks3-server[1127]: 2026/09/21 09:13:59 INFO Registered completed upload object_key=dg404zlvxfrspnxl6cc1qkc6i83m5a3a.ls4193lb # [ 50.299620] niks3-server[1127]: 2026/09/21 09:13:59 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete4194lb # [ 50.312250] niks3-server[1127]: 2026/09/21 09:13:59 INFO Completed upload id=24195worker1 # [ 50.287037] nix-grpc-daemon[853]: time=2026-09-21T09:13:59.882Z level=INFO msg="Upload complete. (309ms)"4196lb # [ 50.315824] niks3-server[1127]: 2026/09/21 09:13:59 INFO Registered completed upload object_key=dg404zlvxfrspnxl6cc1qkc6i83m5a3a.narinfo4197worker1 # [ 50.300124] nix-grpc-daemon[774]: ts=2026-09-21T09:13:59Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35350 duration_s=1 drv=619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv assign_id=1 outputs=14198worker2 # [ 50.317470] nix-grpc-daemon[781]: ts=2026-09-21T09:13:59Z level=debug event=want client=1 drv=79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv cached=04199worker1 # [ 50.315219] nix-daemon[843]: reaped child process 876, status = succeeded4200worker2 # [ 50.317856] nix-grpc-daemon[781]: ts=2026-09-21T09:13:59Z level=debug event=assigned drv=79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv worker=192.168.1.3:50051 clients=14201worker1 # [ 50.355218] nix-grpc-daemon[774]: ts=2026-09-21T09:13:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:353504202worker1 # [ 50.370928] nix-grpc-daemon[774]: ts=2026-09-21T09:13:59Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:353504203worker1 # [ 50.386267] nix-grpc-daemon[853]: time=2026-09-21T09:13:59.981Z level=INFO msg="All 1 paths already cached"4204client # copying 1 paths to 'lb:50051'...4205worker1 # [ 50.396421] nix-grpc-daemon[774]: ts=2026-09-21T09:13:59Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:353504206client # copying path '/nix/store/79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv' to 'lb:50051'...4207worker1 # [ 50.418706] nix-daemon[843]: accepted connection from pid 774, user nix-grpc-daemon (trusted)4208worker1 # [ 50.644248] nix-daemon[843]: accepted connection from pid 900, user nix-grpc-daemon (trusted)4209worker1 # [ 50.663881] nix-daemon[843]: reaped child process 903, status = succeeded4210worker1 # [ 50.845556] nix-daemon[843]: accepted connection from pid 905, user nix-grpc-daemon (trusted)4211worker1 # [ 50.864418] nix-daemon[843]: reaped child process 908, status = succeeded4212lb # [ 50.906366] niks3-server[1127]: 2026/09/21 09:14:00 INFO Received uploads request method=POST path=/api/pending_closures4213worker1 # [ 50.891416] nix-grpc-daemon[853]: time=2026-09-21T09:14:00.486Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4214worker1 # [ 50.893223] nix-grpc-daemon[853]: time=2026-09-21T09:14:00.486Z level=INFO msg="Uploading 79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv (624B)"4215lb # [ 50.936503] niks3-server[1127]: 2026/09/21 09:14:00 INFO Registered completed upload object_key=nar/1xk4kbz8l387hfrqq2xi6as79has336h62f2ra95472yqjclksjg.nar.zst4216lb # [ 50.948864] niks3-server[1127]: 2026/09/21 09:14:00 INFO Received sign narinfos request method=POST path=/api/pending_closures/3/sign4217worker1 # [ 50.930589] nix-grpc-daemon[853]: time=2026-09-21T09:14:00.526Z level=INFO msg="Uploading 1 narinfos"4218lb # [ 50.961240] niks3-server[1127]: 2026/09/21 09:14:00 INFO Signed narinfos id=3 count=14219lb # [ 50.967475] niks3-server[1127]: 2026/09/21 09:14:00 INFO Registered completed upload object_key=79z38hfi82zcmb9yq1gcmfviimgf6z2l.ls4220lb # [ 50.979141] envoy[913]: conn peer=[2001:db8:1::1]:46030 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=732 tx=430 ms=18424221lb # [ 50.983845] envoy[913]: conn peer=[2001:db8:1::1]:46036 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=2596 tx=364 ms=734222worker1 # [ 50.957887] nix-grpc-daemon[853]: time=2026-09-21T09:14:00.553Z level=INFO msg="Upload complete. (520ms)"4223lb # [ 50.987216] envoy[913]: conn peer=[2001:db8:1::1]:46048 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=423 tx=198 ms=124224lb # [ 50.990729] envoy[913]: conn peer=[2001:db8:1::1]:46054 tls=TLSv1.3 subject="CN=stranger" sni=- flags=- tls_fail="-" rx=423 tx=181 ms=134225lb # [ 50.994564] niks3-server[1127]: 2026/09/21 09:14:00 INFO Received complete upload request method=POST path=/api/pending_closures/3/complete4226worker1 # [ 50.968250] nix-grpc-daemon[774]: ts=2026-09-21T09:14:00Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35350 duration_s=0 paths=1 nar_bytes_in=6244227lb # [ 50.997661] niks3-server[1127]: 2026/09/21 09:14:00 INFO Completed upload id=34228lb # [ 50.999655] niks3-server[1127]: 2026/09/21 09:14:00 INFO Registered completed upload object_key=79z38hfi82zcmb9yq1gcmfviimgf6z2l.narinfo4229worker1 # [ 50.979610] nix-grpc-daemon[774]: ts=2026-09-21T09:14:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:353504230worker1 # [ 50.990146] nix-daemon[843]: accepted connection from pid 774, user nix-grpc-daemon (trusted)4231worker1 # [ 50.994128] nix-daemon[843]: reaped child process 898, status = succeeded4232client # building '/nix/store/79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv' on lb:50051...4233client # farm-b> node-a: building 79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv4234worker1 # [ 51.007917] nix-daemon[843]: accepted connection from pid 774, user nix-grpc-daemon (trusted)4235client # farm-b> LOG-farm-b-t14236worker1 # [ 52.455688] nix-daemon[843]: reaped child process 912, status = succeeded4237worker1 # [ 52.638496] nix-daemon[843]: accepted connection from pid 927, user nix-grpc-daemon (trusted)4238worker1 # [ 52.656233] nix-daemon[843]: reaped child process 930, status = succeeded4239lb # [ 52.694400] niks3-server[1127]: 2026/09/21 09:14:02 INFO Received uploads request method=POST path=/api/pending_closures4240worker1 # [ 52.679972] nix-grpc-daemon[853]: time=2026-09-21T09:14:02.274Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4241worker1 # [ 52.681555] nix-grpc-daemon[853]: time=2026-09-21T09:14:02.274Z level=INFO msg="Uploading 8n9g36vw9957r150a7wg1k6yj9fbqm8r-farm-b (176B)"4242lb # [ 52.727097] niks3-server[1127]: 2026/09/21 09:14:02 INFO Registered completed upload object_key=log/79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv4243lb # [ 52.741430] niks3-server[1127]: 2026/09/21 09:14:02 INFO Registered completed upload object_key=nar/1mx8h9irf9ykfzxs6q3lwxbzxwv5d2s9mh2x0sky7448mlc1qxrx.nar.zst4244worker1 # [ 52.736164] nix-grpc-daemon[853]: time=2026-09-21T09:14:02.331Z level=INFO msg="Uploading 1 narinfos"4245lb # [ 52.755966] niks3-server[1127]: 2026/09/21 09:14:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/4/sign4246lb # [ 52.766913] niks3-server[1127]: 2026/09/21 09:14:02 INFO Signed narinfos id=4 count=14247lb # [ 52.772892] niks3-server[1127]: 2026/09/21 09:14:02 INFO Registered completed upload object_key=8n9g36vw9957r150a7wg1k6yj9fbqm8r.ls4248lb # [ 52.779685] niks3-server[1127]: 2026/09/21 09:14:02 INFO Received complete upload request method=POST path=/api/pending_closures/4/complete4249lb # [ 52.787634] niks3-server[1127]: 2026/09/21 09:14:02 INFO Completed upload id=44250worker1 # [ 52.762462] nix-grpc-daemon[853]: time=2026-09-21T09:14:02.358Z level=INFO msg="Upload complete. (343ms)"4251lb # [ 52.790550] niks3-server[1127]: 2026/09/21 09:14:02 INFO Registered completed upload object_key=8n9g36vw9957r150a7wg1k6yj9fbqm8r.narinfo4252worker1 # [ 52.777260] nix-grpc-daemon[774]: ts=2026-09-21T09:14:02Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35350 duration_s=1 drv=79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv assign_id=2 outputs=14253worker2 # [ 52.786733] nix-grpc-daemon[781]: ts=2026-09-21T09:14:02Z level=debug event=want client=1 drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv cached=04254worker1 # [ 52.794049] nix-daemon[843]: reaped child process 910, status = succeeded4255worker1 # [ 52.827150] nix-grpc-daemon[774]: ts=2026-09-21T09:14:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:353504256worker1 # [ 52.842964] nix-grpc-daemon[774]: ts=2026-09-21T09:14:02Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:353504257worker1 # [ 52.858469] nix-grpc-daemon[853]: time=2026-09-21T09:14:02.454Z level=INFO msg="All 2 paths already cached"4258client # copying 1 paths to 'lb:50051'...4259worker1 # [ 52.869027] nix-grpc-daemon[774]: ts=2026-09-21T09:14:02Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:353504260client # copying path '/nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv' to 'lb:50051'...4261worker1 # [ 52.893064] nix-daemon[843]: accepted connection from pid 774, user nix-grpc-daemon (trusted)4262worker1 # [ 53.099590] nix-daemon[843]: accepted connection from pid 934, user nix-grpc-daemon (trusted)4263worker1 # [ 53.120609] nix-daemon[843]: reaped child process 937, status = succeeded4264worker1 # [ 53.297804] nix-daemon[843]: accepted connection from pid 939, user nix-grpc-daemon (trusted)4265worker1 # [ 53.316395] nix-daemon[843]: reaped child process 942, status = succeeded4266lb # [ 53.354438] niks3-server[1127]: 2026/09/21 09:14:02 INFO Received uploads request method=POST path=/api/pending_closures4267worker1 # [ 53.339592] nix-grpc-daemon[853]: time=2026-09-21T09:14:02.934Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4268worker1 # [ 53.341265] nix-grpc-daemon[853]: time=2026-09-21T09:14:02.935Z level=INFO msg="Uploading czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv (752B)"4269lb # [ 53.382318] niks3-server[1127]: 2026/09/21 09:14:02 INFO Registered completed upload object_key=nar/02rh0zwkcxgd7rc74r5h676rr0m8dbayk3glb776b2mdlirxh9lv.nar.zst4270worker1 # [ 53.371312] nix-grpc-daemon[853]: time=2026-09-21T09:14:02.966Z level=INFO msg="Uploading 1 narinfos"4271lb # [ 53.397941] niks3-server[1127]: 2026/09/21 09:14:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/5/sign4272lb # [ 53.409282] niks3-server[1127]: 2026/09/21 09:14:02 INFO Signed narinfos id=5 count=14273lb # [ 53.415198] niks3-server[1127]: 2026/09/21 09:14:03 INFO Registered completed upload object_key=czd6ljynjaw3bl3w8iz1nx68whkdis8f.ls4274lb # [ 53.422672] niks3-server[1127]: 2026/09/21 09:14:03 INFO Received complete upload request method=POST path=/api/pending_closures/5/complete4275lb # [ 53.429264] niks3-server[1127]: 2026/09/21 09:14:03 INFO Completed upload id=54276worker1 # [ 53.403920] nix-grpc-daemon[853]: time=2026-09-21T09:14:02.999Z level=INFO msg="Upload complete. (492ms)"4277worker2 # [ 52.792809] nix-grpc-daemon[781]: ts=2026-09-21T09:14:02Z level=debug event=assigned drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv worker=192.168.1.3:50051 clients=14278lb # [ 53.433797] niks3-server[1127]: 2026/09/21 09:14:03 INFO Registered completed upload object_key=czd6ljynjaw3bl3w8iz1nx68whkdis8f.narinfo4279worker1 # [ 53.413846] nix-grpc-daemon[774]: ts=2026-09-21T09:14:03Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35350 duration_s=0 paths=1 nar_bytes_in=7524280worker1 # [ 53.420761] nix-grpc-daemon[774]: ts=2026-09-21T09:14:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:353504281worker1 # [ 53.431509] nix-daemon[843]: accepted connection from pid 774, user nix-grpc-daemon (trusted)4282worker1 # [ 53.436329] nix-daemon[843]: reaped child process 932, status = succeeded4283client # building '/nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv' on lb:50051...4284client # farm-top> node-a: building czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv4285worker1 # [ 53.449950] nix-daemon[843]: accepted connection from pid 774, user nix-grpc-daemon (trusted)4286client # farm-top> LOG-farm-top-t14287worker1 # [ 54.295516] nix-daemon[843]: reaped child process 946, status = succeeded4288worker1 # [ 54.475722] nix-daemon[843]: accepted connection from pid 961, user nix-grpc-daemon (trusted)4289worker1 # [ 54.494360] nix-daemon[843]: reaped child process 964, status = succeeded4290lb # [ 54.534085] niks3-server[1127]: 2026/09/21 09:14:04 INFO Received uploads request method=POST path=/api/pending_closures4291worker1 # [ 54.518928] nix-grpc-daemon[853]: time=2026-09-21T09:14:04.112Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4292worker1 # [ 54.521076] nix-grpc-daemon[853]: time=2026-09-21T09:14:04.112Z level=INFO msg="Uploading nblg066ygaz2avzlmix3rnngr62hld8f-farm-top (232B)"4293lb # [ 54.572563] niks3-server[1127]: 2026/09/21 09:14:04 INFO Registered completed upload object_key=log/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv4294lb # [ 54.584986] niks3-server[1127]: 2026/09/21 09:14:04 INFO Registered completed upload object_key=nar/1wplyx8g7zja504cjvqmxis1q6laml9bidgiq4ya0i9p2pvqb9qp.nar.zst4295worker1 # [ 54.573171] nix-grpc-daemon[853]: time=2026-09-21T09:14:04.168Z level=INFO msg="Uploading 1 narinfos"4296lb # [ 54.592961] niks3-server[1127]: 2026/09/21 09:14:04 INFO Received sign narinfos request method=POST path=/api/pending_closures/6/sign4297lb # [ 54.605072] niks3-server[1127]: 2026/09/21 09:14:04 INFO Signed narinfos id=6 count=14298lb # [ 54.607500] niks3-server[1127]: 2026/09/21 09:14:04 INFO Registered completed upload object_key=nblg066ygaz2avzlmix3rnngr62hld8f.ls4299lb # [ 54.616856] niks3-server[1127]: 2026/09/21 09:14:04 INFO Received complete upload request method=POST path=/api/pending_closures/6/complete4300lb # [ 54.628320] niks3-server[1127]: 2026/09/21 09:14:04 INFO Completed upload id=64301worker1 # [ 54.603097] nix-grpc-daemon[853]: time=2026-09-21T09:14:04.198Z level=INFO msg="Upload complete. (343ms)"4302lb # [ 54.631564] niks3-server[1127]: 2026/09/21 09:14:04 INFO Registered completed upload object_key=nblg066ygaz2avzlmix3rnngr62hld8f.narinfo4303worker1 # [ 54.617258] nix-grpc-daemon[774]: ts=2026-09-21T09:14:04Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35350 duration_s=1 drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv assign_id=3 outputs=14304worker1 # [ 54.634813] nix-daemon[843]: reaped child process 944, status = succeeded4305client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 >&2, in 7.06 seconds)4306client: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 outPath4307client: (finished: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 outPath, in 0.24 seconds)4308client: must fail: test -e /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top4309client: (finished: must fail: test -e /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.04 seconds)4310client: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top && grep farm-top-t1 /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top4311client # copying 3 paths...4312client # copying path '/nix/store/dg404zlvxfrspnxl6cc1qkc6i83m5a3a-farm-a' from 'http://lb:5751'...4313client # copying path '/nix/store/8n9g36vw9957r150a7wg1k6yj9fbqm8r-farm-b' from 'http://lb:5751'...4314client # copying path '/nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top' from 'http://lb:5751'...4315client: (finished: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top && grep farm-top-t1 /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.30 seconds)4316(finished: subtest: DAG build is scheduled across workers and lands in the cache, in 7.74 seconds)4317subtest: repeat is answered Cached by the scheduler without building4318worker1: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top4319worker1 # finding garbage collector roots...4320worker1 # deleting specified paths...4321worker1 # deleting '/nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top'4322worker1 # deleting unused links...4323worker1 # note: hard linking is currently saving 0.0 KiB4324worker1: (finished: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.28 seconds)4325worker2: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top4326worker2 # finding garbage collector roots...4327worker2 # deleting specified paths...4328worker2 # deleting '/nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top'4329worker2 # deleting unused links...4330worker2 # note: hard linking is currently saving 0.0 KiB4331worker2: (finished: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.29 seconds)4332client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 >&24333worker1 # [ 56.183083] nix-grpc-daemon[774]: ts=2026-09-21T09:14:05Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:353504334client # this path will be fetched (0.0 KiB download, 0.2 KiB unpacked):4335client # /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top4336worker1 # [ 56.202989] nix-grpc-daemon[774]: ts=2026-09-21T09:14:05Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:353504337worker1 # [ 56.214359] nix-grpc-daemon[853]: time=2026-09-21T09:14:05.809Z level=INFO msg="All 2 paths already cached"4338worker2 # [ 56.240748] nix-grpc-daemon[781]: ts=2026-09-21T09:14:05Z level=debug event=want client=2 drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv cached=14339client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 >&2, in 0.35 seconds)4340client: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 outPath4341client: (finished: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 outPath, in 0.21 seconds)4342lb: must succeed: curl -sf localhost:9901/clusters4343lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds)4344lb: must succeed: curl -sf localhost:9901/clusters4345lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)4346worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4347worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)4348(finished: subtest: repeat is answered Cached by the scheduler without building, in 1.46 seconds)4349subtest: inputs already in the cache are not sent to the scheduler4350lb: must succeed: curl -sf localhost:9901/clusters4351lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)4352lb: must succeed: curl -sf localhost:9901/clusters4353lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4354worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4355worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.07 seconds)4356client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 --argstr top t1b >&24357worker1 # [ 57.273154] nix-grpc-daemon[774]: ts=2026-09-21T09:14:06Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:353504358client # don't know how to build these paths:4359client # /nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv4360worker1 # [ 57.283937] nix-grpc-daemon[774]: ts=2026-09-21T09:14:06Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:353504361worker1 # [ 57.295135] nix-grpc-daemon[853]: time=2026-09-21T09:14:06.889Z level=INFO msg="All 2 paths already cached"4362worker2 # [ 57.320713] nix-grpc-daemon[781]: ts=2026-09-21T09:14:06Z level=debug event=want client=3 drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv cached=04363worker2 # [ 57.321092] nix-grpc-daemon[781]: ts=2026-09-21T09:14:06Z level=debug event=assigned drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv worker=192.168.1.3:50051 clients=14364worker1 # [ 57.358997] nix-grpc-daemon[774]: ts=2026-09-21T09:14:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:353504365worker1 # [ 57.374597] nix-grpc-daemon[774]: ts=2026-09-21T09:14:06Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:353504366worker1 # [ 57.389973] nix-grpc-daemon[853]: time=2026-09-21T09:14:06.985Z level=INFO msg="All 2 paths already cached"4367client # copying 1 paths to 'lb:50051'...4368client # copying path '/nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv' to 'lb:50051'...4369worker1 # [ 57.400656] nix-grpc-daemon[774]: ts=2026-09-21T09:14:06Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:353504370worker1 # [ 57.420955] nix-daemon[843]: accepted connection from pid 774, user nix-grpc-daemon (trusted)4371worker1 # [ 57.626435] nix-daemon[843]: accepted connection from pid 981, user nix-grpc-daemon (trusted)4372worker1 # [ 57.647604] nix-daemon[843]: reaped child process 984, status = succeeded4373worker1 # [ 57.822239] nix-daemon[843]: accepted connection from pid 986, user nix-grpc-daemon (trusted)4374worker1 # [ 57.842167] nix-daemon[843]: reaped child process 989, status = succeeded4375lb # [ 57.881855] niks3-server[1127]: 2026/09/21 09:14:07 INFO Received uploads request method=POST path=/api/pending_closures4376worker1 # [ 57.866111] nix-grpc-daemon[853]: time=2026-09-21T09:14:07.461Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4377worker1 # [ 57.867693] nix-grpc-daemon[853]: time=2026-09-21T09:14:07.461Z level=INFO msg="Uploading blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv (752B)"4378lb # [ 57.908420] niks3-server[1127]: 2026/09/21 09:14:07 INFO Registered completed upload object_key=nar/1vy3ds565pni9mnkldcs6alimlb6bg39cimm60s4swkk5yjhwlqf.nar.zst4379lb # [ 57.924253] niks3-server[1127]: 2026/09/21 09:14:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/7/sign4380worker1 # [ 57.904518] nix-grpc-daemon[853]: time=2026-09-21T09:14:07.499Z level=INFO msg="Uploading 1 narinfos"4381lb # [ 57.936322] niks3-server[1127]: 2026/09/21 09:14:07 INFO Signed narinfos id=7 count=14382lb # [ 57.943653] niks3-server[1127]: 2026/09/21 09:14:07 INFO Registered completed upload object_key=blgxcsidls7na1m5da3m0bja1prb87vl.ls4383lb # [ 57.953881] niks3-server[1127]: 2026/09/21 09:14:07 INFO Received complete upload request method=POST path=/api/pending_closures/7/complete4384lb # [ 57.959073] niks3-server[1127]: 2026/09/21 09:14:07 INFO Completed upload id=74385worker1 # [ 57.933706] nix-grpc-daemon[853]: time=2026-09-21T09:14:07.529Z level=INFO msg="Upload complete. (494ms)"4386lb # [ 57.961891] niks3-server[1127]: 2026/09/21 09:14:07 INFO Registered completed upload object_key=blgxcsidls7na1m5da3m0bja1prb87vl.narinfo4387worker1 # [ 57.940331] nix-grpc-daemon[774]: ts=2026-09-21T09:14:07Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35350 duration_s=0 paths=1 nar_bytes_in=7524388worker1 # [ 57.954807] nix-grpc-daemon[774]: ts=2026-09-21T09:14:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:353504389worker1 # [ 57.962422] nix-daemon[843]: accepted connection from pid 774, user nix-grpc-daemon (trusted)4390worker1 # [ 57.967638] nix-daemon[843]: reaped child process 979, status = succeeded4391client # building '/nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv' on lb:50051...4392worker1 # [ 57.980134] nix-daemon[843]: accepted connection from pid 774, user nix-grpc-daemon (trusted)4393client # farm-top> node-a: building blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv4394client # farm-top> LOG-farm-top-t1b4395worker1 # [ 59.631073] nix-daemon[843]: reaped child process 993, status = succeeded4396worker1 # [ 59.819105] nix-daemon[843]: accepted connection from pid 1008, user nix-grpc-daemon (trusted)4397worker1 # [ 59.838069] nix-daemon[843]: reaped child process 1011, status = succeeded4398lb # [ 59.878155] niks3-server[1127]: 2026/09/21 09:14:09 INFO Received uploads request method=POST path=/api/pending_closures4399worker1 # [ 59.864579] nix-grpc-daemon[853]: time=2026-09-21T09:14:09.458Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4400worker1 # [ 59.866646] nix-grpc-daemon[853]: time=2026-09-21T09:14:09.458Z level=INFO msg="Uploading 0xnysbbdbmapgf23wb2dbsfhgh18yxvp-farm-top (232B)"4401lb # [ 59.922257] niks3-server[1127]: 2026/09/21 09:14:09 INFO Registered completed upload object_key=log/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv4402lb # [ 59.934021] niks3-server[1127]: 2026/09/21 09:14:09 INFO Registered completed upload object_key=nar/1f6pjd04v1xmbxqqjhczkkfk4ns9qg6ssnspdalp41q9hkxf9yfr.nar.zst4403worker1 # [ 59.925204] nix-grpc-daemon[853]: time=2026-09-21T09:14:09.520Z level=INFO msg="Uploading 1 narinfos"4404lb # [ 59.949951] niks3-server[1127]: 2026/09/21 09:14:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/8/sign4405lb # [ 59.958928] niks3-server[1127]: 2026/09/21 09:14:09 INFO Signed narinfos id=8 count=14406lb # [ 59.964561] niks3-server[1127]: 2026/09/21 09:14:09 INFO Registered completed upload object_key=0xnysbbdbmapgf23wb2dbsfhgh18yxvp.ls4407lb # [ 59.975550] niks3-server[1127]: 2026/09/21 09:14:09 INFO Received complete upload request method=POST path=/api/pending_closures/8/complete4408lb # [ 59.979614] niks3-server[1127]: 2026/09/21 09:14:09 INFO Registered completed upload object_key=0xnysbbdbmapgf23wb2dbsfhgh18yxvp.narinfo4409lb # [ 59.984394] niks3-server[1127]: 2026/09/21 09:14:09 INFO Completed upload id=84410worker1 # [ 59.958834] nix-grpc-daemon[853]: time=2026-09-21T09:14:09.554Z level=INFO msg="Upload complete. (364ms)"4411worker1 # [ 59.973347] nix-grpc-daemon[774]: ts=2026-09-21T09:14:09Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35350 duration_s=2 drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv assign_id=4 outputs=14412worker1 # [ 59.989507] nix-daemon[843]: reaped child process 991, status = succeeded4413client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 --argstr top t1b >&2, in 3.01 seconds)4414client: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 --argstr top t1b outPath4415client: (finished: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 --argstr top t1b outPath, in 0.25 seconds)4416lb: must succeed: curl -sf localhost:9901/clusters4417lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds)4418lb: must succeed: curl -sf localhost:9901/clusters4419lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)4420worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4421worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.07 seconds)4422(finished: subtest: inputs already in the cache are not sent to the scheduler, in 3.67 seconds)4423subtest: two clients wanting the same drv build it once4424client: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag dedup4425client # [ 60.565449] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key" --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag dedup.4426client # Running as unit: dedup1.service; invocation ID: f0210f4699484edb8affbe14f1ed545a4427client: (finished: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag dedup, in 0.10 seconds)4428client: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag dedup4429client # [ 60.849720] nix[879]: don't know how to build these paths:4430client # [ 60.852083] nix[879]: /nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv4431worker1 # [ 60.849764] nix-grpc-daemon[774]: ts=2026-09-21T09:14:10Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:353504432worker2 # [ 60.874750] nix-grpc-daemon[781]: ts=2026-09-21T09:14:10Z level=debug event=want client=4 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv cached=04433worker2 # [ 60.884978] nix-grpc-daemon[781]: ts=2026-09-21T09:14:10Z level=debug event=assigned drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv worker=192.168.1.4:50051 clients=14434worker2 # [ 60.893972] nix-grpc-daemon[781]: ts=2026-09-21T09:14:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:344904435worker2 # [ 60.917022] nix-grpc-daemon[781]: ts=2026-09-21T09:14:10Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:344904436client # [ 60.924160] nix[879]: copying 1 paths to 'lb:50051'...4437client # [ 60.928325] nix[879]: copying path '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' to 'lb:50051'...4438worker2 # [ 60.935667] nix-grpc-daemon[781]: ts=2026-09-21T09:14:10Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:344904439lb # [ 60.966007] envoy[913]: conn peer=[2001:db8:1::1]:46068 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9523 tx=3839 ms=67904440lb # [ 60.971234] envoy[913]: conn peer=[2001:db8:1::1]:60170 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=993 tx=476 ms=744441worker2 # [ 60.955593] nix-daemon[870]: accepted connection from pid 781, user nix-grpc-daemon (trusted)4442lb # [ 60.978275] envoy[913]: conn peer=[2001:db8:1::1]:60184 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4215 tx=1745 ms=27094443worker2 # [ 60.985148] nix-grpc-daemon[781]: ts=2026-09-21T09:14:10Z level=info event=niks3_push_started pid=9184444worker2 # [ 61.301123] nix-grpc-daemon[918]: time=2026-09-21T09:14:10.881Z level=INFO msg="Configuring client TLS" cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker2.pem key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker2.key ca=""4445worker2 # [ 61.508821] nix-daemon[870]: accepted connection from pid 931, user nix-grpc-daemon (trusted)4446worker2 # [ 61.529196] nix-daemon[870]: reaped child process 934, status = succeeded4447worker2 # [ 61.711073] nix-daemon[870]: accepted connection from pid 936, user nix-grpc-daemon (trusted)4448client # [ 61.693915] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key" --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag dedup.4449worker2 # [ 61.729459] nix-daemon[870]: reaped child process 939, status = succeeded4450client: (finished: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag dedup, in 1.14 seconds)4451lb # [ 61.762758] niks3-server[1127]: 2026/09/21 09:14:11 INFO Received uploads request method=POST path=/api/pending_closures4452worker2 # [ 61.756218] nix-grpc-daemon[918]: time=2026-09-21T09:14:11.338Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4453worker2 # [ 61.759886] nix-grpc-daemon[918]: time=2026-09-21T09:14:11.340Z level=INFO msg="Uploading cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv (432B)"4454lb # [ 61.806203] niks3-server[1127]: 2026/09/21 09:14:11 INFO Registered completed upload object_key=nar/0l7vpaziqxpw0srrsgimqvaa0dyxa9glicn3ilkwk06l64hn837w.nar.zst4455lb # [ 61.818771] niks3-server[1127]: 2026/09/21 09:14:11 INFO Received sign narinfos request method=POST path=/api/pending_closures/9/sign4456worker2 # [ 61.807355] nix-grpc-daemon[918]: time=2026-09-21T09:14:11.389Z level=INFO msg="Uploading 1 narinfos"4457lb # [ 61.830368] niks3-server[1127]: 2026/09/21 09:14:11 INFO Signed narinfos id=9 count=14458lb # [ 61.838793] niks3-server[1127]: 2026/09/21 09:14:11 INFO Registered completed upload object_key=cp23830c44gs2dw4xy80s5wqyjm7vw84.ls4459lb # [ 61.853786] niks3-server[1127]: 2026/09/21 09:14:11 INFO Received complete upload request method=POST path=/api/pending_closures/9/complete4460lb # [ 61.860979] niks3-server[1127]: 2026/09/21 09:14:11 INFO Completed upload id=94461lb # [ 61.864449] niks3-server[1127]: 2026/09/21 09:14:11 INFO Registered completed upload object_key=cp23830c44gs2dw4xy80s5wqyjm7vw84.narinfo4462worker2 # [ 61.841727] nix-grpc-daemon[918]: time=2026-09-21T09:14:11.424Z level=INFO msg="Upload complete. (530ms)"4463worker2 # [ 61.852786] nix-grpc-daemon[781]: ts=2026-09-21T09:14:11Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34490 duration_s=0 paths=1 nar_bytes_in=4324464worker2 # [ 61.874256] nix-grpc-daemon[781]: ts=2026-09-21T09:14:11Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:344904465worker2 # [ 61.887118] nix-daemon[870]: accepted connection from pid 781, user nix-grpc-daemon (trusted)4466worker2 # [ 61.895497] nix-daemon[870]: reaped child process 916, status = succeeded4467client # Running as unit: dedup2.service; invocation ID: e6e3990c82424225ad1e3dc253765c7f4468worker2 # [ 61.907813] nix-daemon[870]: accepted connection from pid 781, user nix-grpc-daemon (trusted)4469client # [ 61.898015] nix[879]: building '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' on lb:50051...4470client # [ 62.088064] nix[895]: don't know how to build these paths:4471client # [ 62.089008] nix[895]: /nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv4472worker1 # [ 62.089598] nix-grpc-daemon[774]: ts=2026-09-21T09:14:11Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:353504473worker2 # [ 62.110485] nix-grpc-daemon[781]: ts=2026-09-21T09:14:11Z level=debug event=want client=5 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv cached=04474worker2 # [ 62.123091] nix-grpc-daemon[781]: ts=2026-09-21T09:14:11Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:344904475client # [ 62.121729] nix[895]: building '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' on lb:50051...4476client: must succeed: sleep 24477client: (finished: must succeed: sleep 2, in 2.06 seconds)4478client: waiting for success: ! systemctl is-active dedup1 dedup24479worker2 # [ 65.559124] nix-daemon[870]: reaped child process 947, status = succeeded4480worker2 # [ 65.744596] nix-daemon[870]: accepted connection from pid 974, user nix-grpc-daemon (trusted)4481worker2 # [ 65.762575] nix-daemon[870]: reaped child process 977, status = succeeded4482lb # [ 65.793700] niks3-server[1127]: 2026/09/21 09:14:15 INFO Received uploads request method=POST path=/api/pending_closures4483worker2 # [ 65.787649] nix-grpc-daemon[918]: time=2026-09-21T09:14:15.369Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4484worker2 # [ 65.789712] nix-grpc-daemon[918]: time=2026-09-21T09:14:15.369Z level=INFO msg="Uploading vamz940my4dma9a2daklr9pw9x9pp9hp-slow-dedup (120B)"4485lb # [ 65.842328] niks3-server[1127]: 2026/09/21 09:14:15 INFO Registered completed upload object_key=log/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv4486lb # [ 65.853555] niks3-server[1127]: 2026/09/21 09:14:15 INFO Registered completed upload object_key=nar/02xhr0h0sj1ji8m0s8995xzbbb8mhbqdpdl1ggx94yqqaykihvz7.nar.zst4487worker2 # [ 65.849445] nix-grpc-daemon[918]: time=2026-09-21T09:14:15.430Z level=INFO msg="Uploading 1 narinfos"4488lb # [ 65.867974] niks3-server[1127]: 2026/09/21 09:14:15 INFO Received sign narinfos request method=POST path=/api/pending_closures/10/sign4489lb # [ 65.872793] niks3-server[1127]: 2026/09/21 09:14:15 INFO Signed narinfos id=10 count=14490lb # [ 65.875160] niks3-server[1127]: 2026/09/21 09:14:15 INFO Registered completed upload object_key=vamz940my4dma9a2daklr9pw9x9pp9hp.ls4491lb # [ 65.886760] niks3-server[1127]: 2026/09/21 09:14:15 INFO Received complete upload request method=POST path=/api/pending_closures/10/complete4492lb # [ 65.893934] niks3-server[1127]: 2026/09/21 09:14:15 INFO Completed upload id=104493worker2 # [ 65.875692] nix-grpc-daemon[918]: time=2026-09-21T09:14:15.458Z level=INFO msg="Upload complete. (349ms)"4494lb # [ 65.897331] niks3-server[1127]: 2026/09/21 09:14:15 INFO Registered completed upload object_key=vamz940my4dma9a2daklr9pw9x9pp9hp.narinfo4495worker2 # [ 65.893178] nix-grpc-daemon[781]: ts=2026-09-21T09:14:15Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34490 duration_s=3 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv assign_id=5 outputs=14496worker2 # [ 65.907217] nix-grpc-daemon[781]: ts=2026-09-21T09:14:15Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34490 duration_s=4 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv assign_id=5 outputs=14497worker2 # [ 65.913063] nix-daemon[870]: reaped child process 944, status = succeeded4498client # [ 65.926175] systemd[1]: dedup2.service: Deactivated successfully.4499client # [ 65.934777] systemd[1]: dedup2.service: Consumed 230ms CPU time over 4.241s wall clock time, 33.2M memory peak, 5.9K incoming IP traffic, 5.6K outgoing IP traffic.4500client # [ 65.978232] systemd[1]: dedup1.service: Deactivated successfully.4501client # [ 65.982413] systemd[1]: dedup1.service: Consumed 184ms CPU time over 5.416s wall clock time, 41.3M memory peak, 7.2K incoming IP traffic, 8.4K outgoing IP traffic.4502client: (finished: waiting for success: ! systemctl is-active dedup1 dedup2, in 1.14 seconds)4503client: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }4504client: (finished: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }, in 0.07 seconds)4505worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true4506worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.08 seconds)4507worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true4508worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.07 seconds)4509worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4510worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)4511worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4512worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)4513(finished: subtest: two clients wanting the same drv build it once, in 6.35 seconds)4514subtest: build hook: nix-daemon with builders = grpc://lb4515client: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --print-out-paths --no-link -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag hook 2>/tmp/hook.log4516client # [ 67.062332] systemd[1]: Started Nix Daemon.4517client # [ 67.248424] nix-daemon[937]: accepted connection from pid 934, user root (trusted)4518worker1 # [ 67.470767] nix-grpc-daemon[774]: ts=2026-09-21T09:14:17Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:353504519worker2 # [ 67.482175] nix-grpc-daemon[781]: ts=2026-09-21T09:14:17Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:344904520worker2 # [ 67.501548] nix-grpc-daemon[781]: ts=2026-09-21T09:14:17Z level=debug event=want client=6 drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv cached=04521worker2 # [ 67.507576] nix-grpc-daemon[781]: ts=2026-09-21T09:14:17Z level=debug event=assigned drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv worker=192.168.1.3:50051 clients=14522worker1 # [ 67.542986] nix-grpc-daemon[774]: ts=2026-09-21T09:14:17Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:353504523worker1 # [ 67.559610] nix-grpc-daemon[774]: ts=2026-09-21T09:14:17Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:353504524worker1 # [ 67.576123] nix-grpc-daemon[774]: ts=2026-09-21T09:14:17Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:353504525worker1 # [ 67.591687] nix-daemon[843]: accepted connection from pid 774, user nix-grpc-daemon (trusted)4526worker1 # [ 67.807500] nix-daemon[843]: accepted connection from pid 1036, user nix-grpc-daemon (trusted)4527worker1 # [ 67.829947] nix-daemon[843]: reaped child process 1039, status = succeeded4528worker1 # [ 68.006556] nix-daemon[843]: accepted connection from pid 1041, user nix-grpc-daemon (trusted)4529worker1 # [ 68.024126] nix-daemon[843]: reaped child process 1044, status = succeeded4530lb # [ 68.062748] niks3-server[1127]: 2026/09/21 09:14:17 INFO Received uploads request method=POST path=/api/pending_closures4531worker1 # [ 68.047193] nix-grpc-daemon[853]: time=2026-09-21T09:14:17.642Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4532worker1 # [ 68.048812] nix-grpc-daemon[853]: time=2026-09-21T09:14:17.642Z level=INFO msg="Uploading xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv (512B)"4533lb # [ 68.089465] niks3-server[1127]: 2026/09/21 09:14:17 INFO Registered completed upload object_key=nar/1f1i5kab3qhx7pg1k0ma5r1m7ipfazkfkfb944dm7qcjr466pwij.nar.zst4534lb # [ 68.104453] niks3-server[1127]: 2026/09/21 09:14:17 INFO Received sign narinfos request method=POST path=/api/pending_closures/11/sign4535worker1 # [ 68.084536] nix-grpc-daemon[853]: time=2026-09-21T09:14:17.680Z level=INFO msg="Uploading 1 narinfos"4536lb # [ 68.116680] niks3-server[1127]: 2026/09/21 09:14:17 INFO Signed narinfos id=11 count=14537lb # [ 68.123746] niks3-server[1127]: 2026/09/21 09:14:17 INFO Registered completed upload object_key=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv.ls4538lb # [ 68.131809] niks3-server[1127]: 2026/09/21 09:14:17 INFO Received complete upload request method=POST path=/api/pending_closures/11/complete4539worker1 # [ 68.111649] nix-grpc-daemon[853]: time=2026-09-21T09:14:17.707Z level=INFO msg="Upload complete. (501ms)"4540lb # [ 68.139049] niks3-server[1127]: 2026/09/21 09:14:17 INFO Completed upload id=114541lb # [ 68.142465] niks3-server[1127]: 2026/09/21 09:14:17 INFO Registered completed upload object_key=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv.narinfo4542worker1 # [ 68.120235] nix-grpc-daemon[774]: ts=2026-09-21T09:14:17Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35350 duration_s=0 paths=1 nar_bytes_in=5124543worker1 # [ 68.130897] nix-grpc-daemon[774]: ts=2026-09-21T09:14:17Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:353504544worker1 # [ 68.137624] nix-daemon[843]: accepted connection from pid 774, user nix-grpc-daemon (trusted)4545worker1 # [ 68.142751] nix-daemon[843]: reaped child process 1034, status = succeeded4546worker1 # [ 68.155964] nix-daemon[843]: accepted connection from pid 774, user nix-grpc-daemon (trusted)4547worker1 # [ 69.399111] nix-daemon[843]: reaped child process 1048, status = succeeded4548worker1 # [ 69.591690] nix-daemon[843]: accepted connection from pid 1063, user nix-grpc-daemon (trusted)4549worker1 # [ 69.609649] nix-daemon[843]: reaped child process 1066, status = succeeded4550lb # [ 69.648956] niks3-server[1127]: 2026/09/21 09:14:19 INFO Received uploads request method=POST path=/api/pending_closures4551worker1 # [ 69.636460] nix-grpc-daemon[853]: time=2026-09-21T09:14:19.229Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4552worker1 # [ 69.638107] nix-grpc-daemon[853]: time=2026-09-21T09:14:19.229Z level=INFO msg="Uploading yfs64sxmsg77fnzczdvc3g3yfdwh9fks-farm-a (128B)"4553lb # [ 69.693624] niks3-server[1127]: 2026/09/21 09:14:19 INFO Registered completed upload object_key=log/xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv4554lb # [ 69.709584] niks3-server[1127]: 2026/09/21 09:14:19 INFO Received sign narinfos request method=POST path=/api/pending_closures/12/sign4555worker1 # [ 69.690478] nix-grpc-daemon[853]: time=2026-09-21T09:14:19.285Z level=INFO msg="Uploading 1 narinfos"4556lb # [ 69.725840] niks3-server[1127]: 2026/09/21 09:14:19 INFO Signed narinfos id=12 count=14557lb # [ 69.732597] niks3-server[1127]: 2026/09/21 09:14:19 INFO Registered completed upload object_key=nar/0pbbxl4kxy7zl7x4lp43ywljwn82mxzna2wgpb1bz1z765nwp9ki.nar.zst4558lb # [ 69.744249] niks3-server[1127]: 2026/09/21 09:14:19 INFO Received complete upload request method=POST path=/api/pending_closures/12/complete4559lb # [ 69.753205] niks3-server[1127]: 2026/09/21 09:14:19 INFO Registered completed upload object_key=yfs64sxmsg77fnzczdvc3g3yfdwh9fks.ls4560lb # [ 69.757761] niks3-server[1127]: 2026/09/21 09:14:19 INFO Completed upload id=124561lb # [ 69.760540] niks3-server[1127]: 2026/09/21 09:14:19 INFO Registered completed upload object_key=yfs64sxmsg77fnzczdvc3g3yfdwh9fks.narinfo4562worker1 # [ 69.731769] nix-grpc-daemon[853]: time=2026-09-21T09:14:19.327Z level=INFO msg="Upload complete. (365ms)"4563worker1 # [ 69.745399] nix-grpc-daemon[774]: ts=2026-09-21T09:14:19Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35350 duration_s=1 drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv assign_id=6 outputs=14564worker2 # [ 69.754741] nix-grpc-daemon[781]: ts=2026-09-21T09:14:19Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:344904565worker1 # [ 69.762544] nix-daemon[843]: reaped child process 1046, status = succeeded4566worker2 # [ 69.770631] nix-grpc-daemon[781]: copying path '/nix/store/yfs64sxmsg77fnzczdvc3g3yfdwh9fks-farm-a' from 'http://lb:5751'...4567worker1 # [ 69.809804] nix-grpc-daemon[774]: ts=2026-09-21T09:14:19Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:353504568worker1 # [ 69.816792] nix-grpc-daemon[774]: ts=2026-09-21T09:14:19Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:35350 duration_s=0 paths=1 nar_bytes_out=1284569worker1 # [ 70.089486] nix-grpc-daemon[774]: ts=2026-09-21T09:14:19Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:353504570worker1 # [ 70.098340] nix-grpc-daemon[774]: ts=2026-09-21T09:14:19Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:353504571worker1 # [ 70.107948] nix-grpc-daemon[853]: time=2026-09-21T09:14:19.703Z level=INFO msg="All 1 paths already cached"4572worker2 # [ 70.133407] nix-grpc-daemon[781]: ts=2026-09-21T09:14:19Z level=debug event=want client=7 drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv cached=04573worker2 # [ 70.139895] nix-grpc-daemon[781]: ts=2026-09-21T09:14:19Z level=debug event=assigned drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv worker=192.168.1.3:50051 clients=14574worker1 # [ 70.135880] nix-grpc-daemon[774]: ts=2026-09-21T09:14:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:353504575worker1 # [ 70.151340] nix-grpc-daemon[774]: ts=2026-09-21T09:14:19Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:353504576worker1 # [ 70.165844] nix-grpc-daemon[853]: time=2026-09-21T09:14:19.761Z level=INFO msg="All 1 paths already cached"4577worker1 # [ 70.175305] nix-grpc-daemon[774]: ts=2026-09-21T09:14:19Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:353504578worker1 # [ 70.189867] nix-daemon[843]: accepted connection from pid 774, user nix-grpc-daemon (trusted)4579worker1 # [ 70.410289] nix-daemon[843]: accepted connection from pid 1070, user nix-grpc-daemon (trusted)4580worker1 # [ 70.429758] nix-daemon[843]: reaped child process 1073, status = succeeded4581worker1 # [ 70.604849] nix-daemon[843]: accepted connection from pid 1075, user nix-grpc-daemon (trusted)4582worker1 # [ 70.622484] nix-daemon[843]: reaped child process 1078, status = succeeded4583lb # [ 70.659868] niks3-server[1127]: 2026/09/21 09:14:20 INFO Received uploads request method=POST path=/api/pending_closures4584worker1 # [ 70.644478] nix-grpc-daemon[853]: time=2026-09-21T09:14:20.239Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4585worker1 # [ 70.645968] nix-grpc-daemon[853]: time=2026-09-21T09:14:20.239Z level=INFO msg="Uploading r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv (624B)"4586lb # [ 70.687068] niks3-server[1127]: 2026/09/21 09:14:20 INFO Registered completed upload object_key=nar/15k66v3mh3v6p5vymv4pv3v3sfl56i2b1ia2rf093c37w6bhc04m.nar.zst4587lb # [ 70.702076] niks3-server[1127]: 2026/09/21 09:14:20 INFO Received sign narinfos request method=POST path=/api/pending_closures/13/sign4588worker1 # [ 70.683182] nix-grpc-daemon[853]: time=2026-09-21T09:14:20.278Z level=INFO msg="Uploading 1 narinfos"4589lb # [ 70.714409] niks3-server[1127]: 2026/09/21 09:14:20 INFO Signed narinfos id=13 count=14590lb # [ 70.720653] niks3-server[1127]: 2026/09/21 09:14:20 INFO Registered completed upload object_key=r1jsr4b2g1qwly395z6idx3z2362ih8i.ls4591lb # [ 70.729627] niks3-server[1127]: 2026/09/21 09:14:20 INFO Received complete upload request method=POST path=/api/pending_closures/13/complete4592lb # [ 70.734610] niks3-server[1127]: 2026/09/21 09:14:20 INFO Registered completed upload object_key=r1jsr4b2g1qwly395z6idx3z2362ih8i.narinfo4593lb # [ 70.738638] niks3-server[1127]: 2026/09/21 09:14:20 INFO Completed upload id=134594worker1 # [ 70.713461] nix-grpc-daemon[853]: time=2026-09-21T09:14:20.309Z level=INFO msg="Upload complete. (505ms)"4595worker1 # [ 70.723080] nix-grpc-daemon[774]: ts=2026-09-21T09:14:20Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35350 duration_s=0 paths=1 nar_bytes_in=6244596worker1 # [ 70.736246] nix-grpc-daemon[774]: ts=2026-09-21T09:14:20Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:353504597worker1 # [ 70.742062] nix-daemon[843]: accepted connection from pid 774, user nix-grpc-daemon (trusted)4598worker1 # [ 70.746917] nix-daemon[843]: reaped child process 1068, status = succeeded4599worker1 # [ 70.760103] nix-daemon[843]: accepted connection from pid 774, user nix-grpc-daemon (trusted)4600lb # [ 70.968477] envoy[913]: conn peer=[2001:db8:1::1]:48960 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1443 tx=1085 ms=38114601lb # [ 70.974371] envoy[913]: conn peer=[2001:db8:1::1]:48948 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2979 tx=1374 ms=50934602lb # [ 70.979207] envoy[913]: conn peer=[2001:db8:1::1]:48988 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=410 tx=333 ms=304603lb # [ 70.983884] envoy[913]: conn peer=[2001:db8:1::1]:48974 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3407 tx=1774 ms=23654604worker1 # [ 71.580128] nix-daemon[843]: reaped child process 1082, status = succeeded4605worker1 # [ 71.733776] nix-daemon[843]: accepted connection from pid 1097, user nix-grpc-daemon (trusted)4606worker1 # [ 71.752337] nix-daemon[843]: reaped child process 1100, status = succeeded4607lb # [ 71.791543] niks3-server[1127]: 2026/09/21 09:14:21 INFO Received uploads request method=POST path=/api/pending_closures4608worker1 # [ 71.775534] nix-grpc-daemon[853]: time=2026-09-21T09:14:21.370Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4609worker1 # [ 71.778731] nix-grpc-daemon[853]: time=2026-09-21T09:14:21.372Z level=INFO msg="Uploading la6ryzsihag4zpxgxls54fsg34dvlalk-farm-b (176B)"4610lb # [ 71.835432] niks3-server[1127]: 2026/09/21 09:14:21 INFO Received sign narinfos request method=POST path=/api/pending_closures/14/sign4611worker1 # [ 71.820991] nix-grpc-daemon[853]: time=2026-09-21T09:14:21.416Z level=INFO msg="Uploading 1 narinfos"4612lb # [ 71.850523] niks3-server[1127]: 2026/09/21 09:14:21 INFO Signed narinfos id=14 count=14613lb # [ 71.860443] niks3-server[1127]: 2026/09/21 09:14:21 INFO Registered completed upload object_key=log/r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv4614lb # [ 71.872254] niks3-server[1127]: 2026/09/21 09:14:21 INFO Registered completed upload object_key=la6ryzsihag4zpxgxls54fsg34dvlalk.ls4615lb # [ 71.875788] niks3-server[1127]: 2026/09/21 09:14:21 INFO Registered completed upload object_key=nar/069d0xnyw2q345iqzx6agxc5fxifc6q36bkkklw2aa4qwanxdd1g.nar.zst4616worker1 # [ 71.854964] nix-grpc-daemon[853]: time=2026-09-21T09:14:21.450Z level=INFO msg="Upload complete. (285ms)"4617lb # [ 71.880769] niks3-server[1127]: 2026/09/21 09:14:21 INFO Received complete upload request method=POST path=/api/pending_closures/14/complete4618lb # [ 71.885445] niks3-server[1127]: 2026/09/21 09:14:21 INFO Completed upload id=144619worker1 # [ 71.863832] nix-grpc-daemon[774]: ts=2026-09-21T09:14:21Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35350 duration_s=1 drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv assign_id=7 outputs=14620worker2 # [ 71.875481] nix-grpc-daemon[781]: ts=2026-09-21T09:14:21Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:344904621worker1 # [ 71.878350] nix-grpc-daemon[853]: time=2026-09-21T09:14:21.454Z level=WARN msg="Failed to register uploaded object" key=la6ryzsihag4zpxgxls54fsg34dvlalk.narinfo error="server returned 404: object is not pending upload\n"4622worker1 # [ 71.884092] nix-daemon[843]: reaped child process 1080, status = succeeded4623worker2 # [ 71.890292] nix-grpc-daemon[781]: copying path '/nix/store/la6ryzsihag4zpxgxls54fsg34dvlalk-farm-b' from 'http://lb:5751'...4624worker2 # [ 71.933155] nix-grpc-daemon[781]: ts=2026-09-21T09:14:21Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:344904625worker2 # [ 71.939643] nix-grpc-daemon[781]: ts=2026-09-21T09:14:21Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:34490 duration_s=0 paths=1 nar_bytes_out=1764626worker1 # [ 72.245853] nix-grpc-daemon[774]: ts=2026-09-21T09:14:21Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:353504627worker1 # [ 72.253526] nix-grpc-daemon[774]: ts=2026-09-21T09:14:21Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:353504628worker1 # [ 72.260549] nix-grpc-daemon[853]: time=2026-09-21T09:14:21.854Z level=INFO msg="All 2 paths already cached"4629worker2 # [ 72.279434] nix-grpc-daemon[781]: ts=2026-09-21T09:14:21Z level=debug event=want client=8 drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv cached=04630worker2 # [ 72.285567] nix-grpc-daemon[781]: ts=2026-09-21T09:14:21Z level=debug event=assigned drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv worker=192.168.1.3:50051 clients=14631worker1 # [ 72.318605] nix-grpc-daemon[774]: ts=2026-09-21T09:14:21Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:353504632worker1 # [ 72.333961] nix-grpc-daemon[774]: ts=2026-09-21T09:14:21Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:353504633worker1 # [ 72.348660] nix-grpc-daemon[853]: time=2026-09-21T09:14:21.944Z level=INFO msg="All 2 paths already cached"4634worker1 # [ 72.354798] nix-grpc-daemon[774]: ts=2026-09-21T09:14:21Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:353504635worker1 # [ 72.369018] nix-daemon[843]: accepted connection from pid 774, user nix-grpc-daemon (trusted)4636worker1 # [ 72.574469] nix-daemon[843]: accepted connection from pid 1104, user nix-grpc-daemon (trusted)4637worker1 # [ 72.592581] nix-daemon[843]: reaped child process 1107, status = succeeded4638worker1 # [ 72.766327] nix-daemon[843]: accepted connection from pid 1109, user nix-grpc-daemon (trusted)4639worker1 # [ 72.783773] nix-daemon[843]: reaped child process 1112, status = succeeded4640lb # [ 72.821368] niks3-server[1127]: 2026/09/21 09:14:22 INFO Received uploads request method=POST path=/api/pending_closures4641worker1 # [ 72.806113] nix-grpc-daemon[853]: time=2026-09-21T09:14:22.401Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4642worker1 # [ 72.807619] nix-grpc-daemon[853]: time=2026-09-21T09:14:22.401Z level=INFO msg="Uploading aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv (752B)"4643lb # [ 72.850129] niks3-server[1127]: 2026/09/21 09:14:22 INFO Registered completed upload object_key=nar/1ahq7sp1sl2scxqzfjd8gqimpqfkjbyi9ivxdhdwx0jzh2gbv74b.nar.zst4644worker1 # [ 72.837319] nix-grpc-daemon[853]: time=2026-09-21T09:14:22.432Z level=INFO msg="Uploading 1 narinfos"4645lb # [ 72.866974] niks3-server[1127]: 2026/09/21 09:14:22 INFO Received sign narinfos request method=POST path=/api/pending_closures/15/sign4646lb # [ 72.877104] niks3-server[1127]: 2026/09/21 09:14:22 INFO Signed narinfos id=15 count=14647worker1 # [ 72.862260] nix-grpc-daemon[853]: time=2026-09-21T09:14:22.457Z level=INFO msg="Upload complete. (475ms)"4648lb # [ 72.887724] niks3-server[1127]: 2026/09/21 09:14:22 INFO Registered completed upload object_key=aa4abikjj4w30bfx1p8hxmq0916qbz1c.ls4649lb # [ 72.892395] niks3-server[1127]: 2026/09/21 09:14:22 INFO Received complete upload request method=POST path=/api/pending_closures/15/complete4650lb # [ 72.896617] niks3-server[1127]: 2026/09/21 09:14:22 INFO Completed upload id=154651worker1 # [ 72.871963] nix-grpc-daemon[774]: ts=2026-09-21T09:14:22Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35350 duration_s=0 paths=1 nar_bytes_in=7524652lb # [ 72.899014] niks3-server[1127]: 2026/09/21 09:14:22 INFO Registered completed upload object_key=aa4abikjj4w30bfx1p8hxmq0916qbz1c.narinfo4653worker1 # [ 72.882836] nix-grpc-daemon[774]: ts=2026-09-21T09:14:22Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:353504654worker1 # [ 72.888830] nix-daemon[843]: accepted connection from pid 774, user nix-grpc-daemon (trusted)4655worker1 # [ 72.893396] nix-daemon[843]: reaped child process 1102, status = succeeded4656worker1 # [ 72.906158] nix-daemon[843]: accepted connection from pid 774, user nix-grpc-daemon (trusted)4657worker1 # [ 74.133261] nix-daemon[843]: reaped child process 1116, status = succeeded4658worker1 # [ 74.304772] nix-daemon[843]: accepted connection from pid 1131, user nix-grpc-daemon (trusted)4659worker1 # [ 74.324868] nix-daemon[843]: reaped child process 1134, status = succeeded4660lb # [ 74.362544] niks3-server[1127]: 2026/09/21 09:14:23 INFO Received uploads request method=POST path=/api/pending_closures4661worker1 # [ 74.348398] nix-grpc-daemon[853]: time=2026-09-21T09:14:23.942Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4662worker1 # [ 74.349945] nix-grpc-daemon[853]: time=2026-09-21T09:14:23.942Z level=INFO msg="Uploading 39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top (232B)"4663lb # [ 74.410975] niks3-server[1127]: 2026/09/21 09:14:24 INFO Received sign narinfos request method=POST path=/api/pending_closures/16/sign4664worker1 # [ 74.394431] nix-grpc-daemon[853]: time=2026-09-21T09:14:23.989Z level=INFO msg="Uploading 1 narinfos"4665lb # [ 74.429312] niks3-server[1127]: 2026/09/21 09:14:24 INFO Signed narinfos id=16 count=14666lb # [ 74.431715] niks3-server[1127]: 2026/09/21 09:14:24 INFO Registered completed upload object_key=log/aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv4667lb # [ 74.440939] niks3-server[1127]: 2026/09/21 09:14:24 INFO Registered completed upload object_key=nar/0vq3rhfhhbqyx5pbvaa9ddj9wbw9arph334y3bp3idfrikwnjc92.nar.zst4668worker1 # [ 74.424446] nix-grpc-daemon[853]: time=2026-09-21T09:14:24.020Z level=INFO msg="Upload complete. (308ms)"4669lb # [ 74.450497] niks3-server[1127]: 2026/09/21 09:14:24 INFO Registered completed upload object_key=39qxwzy2zkij70y9zvbr59pb0df38w5h.ls4670lb # [ 74.454172] niks3-server[1127]: 2026/09/21 09:14:24 INFO Received complete upload request method=POST path=/api/pending_closures/16/complete4671lb # [ 74.457660] niks3-server[1127]: 2026/09/21 09:14:24 INFO Completed upload id=164672lb # [ 74.460495] niks3-server[1127]: 2026/09/21 09:14:24 INFO Registered completed upload object_key=39qxwzy2zkij70y9zvbr59pb0df38w5h.narinfo4673worker1 # [ 74.436944] nix-grpc-daemon[774]: ts=2026-09-21T09:14:24Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35350 duration_s=1 drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv assign_id=8 outputs=14674worker1 # [ 74.451227] nix-grpc-daemon[774]: ts=2026-09-21T09:14:24Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:353504675worker1 # [ 74.455639] nix-daemon[843]: reaped child process 1114, status = succeeded4676worker2 # [ 74.476717] nix-grpc-daemon[781]: ts=2026-09-21T09:14:24Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:344904677worker2 # [ 74.490826] nix-grpc-daemon[781]: copying path '/nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top' from 'http://lb:5751'...4678worker2 # [ 74.507056] nix-grpc-daemon[781]: ts=2026-09-21T09:14:24Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:34490 duration_s=0 paths=1 nar_bytes_out=2324679client # [ 74.640511] nix-daemon[937]: reaped child process 939, status = succeeded4680client: (finished: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --print-out-paths --no-link -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag hook 2>/tmp/hook.log, in 7.82 seconds)4681client: must succeed: grep farm-top-hook /nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top4682client: (finished: must succeed: grep farm-top-hook /nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top, in 0.04 seconds)4683client: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }4684client: (finished: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }, in 0.05 seconds)4685client: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }4686client: (finished: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }, in 0.03 seconds)4687(finished: subtest: build hook: nix-daemon with builders = grpc://lb, in 7.95 seconds)4688subtest: an input only one worker has still reaches the builder4689client: must succeed: nix-build --no-out-link /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix -A input --argstr tag w1only4690client # this derivation will be built:4691client # /nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv4692client # building '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv'...4693client: (finished: must succeed: nix-build --no-out-link /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix -A input --argstr tag w1only, in 0.44 seconds)4694client: must succeed: nix-store --export /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only > /tmp/shared/inp.closure4695client: (finished: must succeed: nix-store --export /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only > /tmp/shared/inp.closure, in 0.20 seconds)4696worker1: must succeed: nix-store --import < /tmp/shared/inp.closure4697worker1: (finished: must succeed: nix-store --import < /tmp/shared/inp.closure, in 0.25 seconds)4698worker2: must fail: test -e /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only4699worker2: (finished: must fail: test -e /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only, in 0.04 seconds)4700client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix job --argstr tag w1only --argstr salt a >&24701client # [ 75.934015] nix-daemon[937]: accepted connection from pid 1018, user root (trusted)4702client # this derivation will be built:4703client # /nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv4704client # building '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' on 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux'...4705worker2 # [ 76.162339] nix-grpc-daemon[781]: ts=2026-09-21T09:14:25Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:344904706worker1 # [ 76.161382] nix-grpc-daemon[774]: ts=2026-09-21T09:14:25Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:353504707worker1 # [ 76.376465] nix-daemon[843]: accepted connection from pid 1141, user nix-grpc-daemon (trusted)4708worker1 # [ 76.394576] nix-daemon[843]: reaped child process 1144, status = succeeded4709lb # [ 76.433753] niks3-server[1127]: 2026/09/21 09:14:26 INFO Received uploads request method=POST path=/api/pending_closures4710worker1 # [ 76.420498] nix-grpc-daemon[853]: time=2026-09-21T09:14:26.015Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4711worker1 # [ 76.422124] nix-grpc-daemon[853]: time=2026-09-21T09:14:26.015Z level=INFO msg="Uploading dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only (120B)"4712lb # [ 76.466802] niks3-server[1127]: 2026/09/21 09:14:26 INFO Registered completed upload object_key=nar/08ghl87f009pwqz85nbqv9hm0ln6gai6ggzilfmw3qnkaalfi6ss.nar.zst4713worker1 # [ 76.456239] nix-grpc-daemon[853]: time=2026-09-21T09:14:26.051Z level=INFO msg="Uploading 1 narinfos"4714lb # [ 76.484298] niks3-server[1127]: 2026/09/21 09:14:26 INFO Received sign narinfos request method=POST path=/api/pending_closures/17/sign4715lb # [ 76.494102] niks3-server[1127]: 2026/09/21 09:14:26 INFO Signed narinfos id=17 count=14716lb # [ 76.498995] niks3-server[1127]: 2026/09/21 09:14:26 INFO Registered completed upload object_key=dymj0d1vj8q0ns6p096qk7fii766dap6.ls4717lb # [ 76.508412] niks3-server[1127]: 2026/09/21 09:14:26 INFO Received complete upload request method=POST path=/api/pending_closures/17/complete4718lb # [ 76.513896] niks3-server[1127]: 2026/09/21 09:14:26 INFO Registered completed upload object_key=dymj0d1vj8q0ns6p096qk7fii766dap6.narinfo4719lb # [ 76.517882] niks3-server[1127]: 2026/09/21 09:14:26 INFO Completed upload id=174720worker1 # [ 76.492706] nix-grpc-daemon[853]: time=2026-09-21T09:14:26.088Z level=INFO msg="Upload complete. (332ms)"4721worker2 # [ 76.524282] nix-grpc-daemon[781]: ts=2026-09-21T09:14:26Z level=debug event=want client=9 drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv cached=04722worker2 # [ 76.531976] nix-grpc-daemon[781]: ts=2026-09-21T09:14:26Z level=debug event=assigned drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv worker=192.168.1.4:50051 clients=14723worker2 # [ 76.579238] nix-grpc-daemon[781]: ts=2026-09-21T09:14:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:344904724worker2 # [ 76.595516] nix-grpc-daemon[781]: ts=2026-09-21T09:14:26Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:344904725client # copying 2 paths to 'lb:50051'...4726client # copying path '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv' to 'lb:50051'...4727worker2 # [ 76.615799] nix-grpc-daemon[781]: ts=2026-09-21T09:14:26Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:344904728client # copying path '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' to 'lb:50051'...4729worker2 # [ 76.632118] nix-daemon[870]: accepted connection from pid 781, user nix-grpc-daemon (trusted)4730worker2 # [ 76.856302] nix-daemon[870]: accepted connection from pid 1004, user nix-grpc-daemon (trusted)4731worker2 # [ 76.875862] nix-daemon[870]: reaped child process 1007, status = succeeded4732worker2 # [ 77.054669] nix-daemon[870]: accepted connection from pid 1009, user nix-grpc-daemon (trusted)4733worker2 # [ 77.071142] nix-daemon[870]: reaped child process 1012, status = succeeded4734lb # [ 77.104707] niks3-server[1127]: 2026/09/21 09:14:26 INFO Received uploads request method=POST path=/api/pending_closures4735lb # [ 77.118959] niks3-server[1127]: 2026/09/21 09:14:26 INFO Received uploads request method=POST path=/api/pending_closures4736worker2 # [ 77.113484] nix-grpc-daemon[918]: time=2026-09-21T09:14:26.695Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"4737worker2 # [ 77.114973] nix-grpc-daemon[918]: time=2026-09-21T09:14:26.695Z level=INFO msg="Uploading 71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv (568B)"4738worker2 # [ 77.116812] nix-grpc-daemon[918]: time=2026-09-21T09:14:26.695Z level=INFO msg="Uploading 4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv (416B)"4739lb # [ 77.160047] niks3-server[1127]: 2026/09/21 09:14:26 INFO Registered completed upload object_key=nar/0p6zgq1gdngshl62q0hv3lslqwv90s7fmflwc7wgnycf4ka6ik69.nar.zst4740lb # [ 77.172201] niks3-server[1127]: 2026/09/21 09:14:26 INFO Registered completed upload object_key=nar/0i2vbprwp0wxa61nkss61gzivqmmjh6klkgv0rlfnxk46rcaba1v.nar.zst4741lb # [ 77.186499] niks3-server[1127]: 2026/09/21 09:14:26 INFO Received sign narinfos request method=POST path=/api/pending_closures/18/sign4742worker2 # [ 77.176127] nix-grpc-daemon[918]: time=2026-09-21T09:14:26.757Z level=INFO msg="Uploading 3 narinfos"4743lb # [ 77.196368] niks3-server[1127]: 2026/09/21 09:14:26 INFO Signed narinfos id=18 count=14744lb # [ 77.198329] niks3-server[1127]: 2026/09/21 09:14:26 INFO Received sign narinfos request method=POST path=/api/pending_closures/19/sign4745lb # [ 77.207053] niks3-server[1127]: 2026/09/21 09:14:26 INFO Signed narinfos id=19 count=24746lb # [ 77.211045] niks3-server[1127]: 2026/09/21 09:14:26 INFO Registered completed upload object_key=71l2x9mybp0i90r6pdnglwjnayj4xpl8.ls4747lb # [ 77.214945] niks3-server[1127]: 2026/09/21 09:14:26 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.ls4748lb # [ 77.226182] niks3-server[1127]: 2026/09/21 09:14:26 INFO Registered completed upload object_key=71l2x9mybp0i90r6pdnglwjnayj4xpl8.narinfo4749lb # [ 77.234221] niks3-server[1127]: 2026/09/21 09:14:26 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.narinfo4750lb # [ 77.240514] niks3-server[1127]: 2026/09/21 09:14:26 INFO Received complete upload request method=POST path=/api/pending_closures/18/complete4751lb # [ 77.245018] niks3-server[1127]: 2026/09/21 09:14:26 INFO Completed upload id=184752lb # [ 77.247048] niks3-server[1127]: 2026/09/21 09:14:26 INFO Received complete upload request method=POST path=/api/pending_closures/19/complete4753lb # [ 77.250775] niks3-server[1127]: 2026/09/21 09:14:26 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.narinfo4754lb # [ 77.253772] niks3-server[1127]: 2026/09/21 09:14:26 INFO Completed upload id=194755worker2 # [ 77.235484] nix-grpc-daemon[918]: time=2026-09-21T09:14:26.818Z level=INFO msg="Upload complete. (580ms)"4756worker2 # [ 77.247208] nix-grpc-daemon[781]: ts=2026-09-21T09:14:26Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34490 duration_s=0 paths=2 nar_bytes_in=9844757worker2 # [ 77.267477] nix-grpc-daemon[781]: ts=2026-09-21T09:14:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:344904758worker2 # [ 77.277613] nix-daemon[870]: accepted connection from pid 781, user nix-grpc-daemon (trusted)4759worker2 # [ 77.287142] nix-daemon[870]: reaped child process 1002, status = succeeded4760worker2 # [ 77.298872] nix-grpc-daemon[781]: copying path '/nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only' from 'http://lb:5751'...4761client # building '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' on lb:50051...4762worker2 # [ 77.319015] nix-daemon[870]: accepted connection from pid 781, user nix-grpc-daemon (trusted)4763client # uses-input-w1onlya> worker2: building 71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv4764worker2 # [ 78.369276] nix-daemon[870]: reaped child process 1017, status = succeeded4765worker2 # [ 78.540971] nix-daemon[870]: accepted connection from pid 1032, user nix-grpc-daemon (trusted)4766worker2 # [ 78.560737] nix-daemon[870]: reaped child process 1035, status = succeeded4767lb # [ 78.591734] niks3-server[1127]: 2026/09/21 09:14:28 INFO Received uploads request method=POST path=/api/pending_closures4768worker2 # [ 78.583759] nix-grpc-daemon[918]: time=2026-09-21T09:14:28.165Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"4769lb # [ 78.621220] niks3-server[1127]: 2026/09/21 09:14:28 INFO Received sign narinfos request method=POST path=/api/pending_closures/20/sign4770worker2 # [ 78.614759] nix-grpc-daemon[918]: time=2026-09-21T09:14:28.197Z level=INFO msg="Uploading 1 narinfos"4771lb # [ 78.636446] niks3-server[1127]: 2026/09/21 09:14:28 INFO Signed narinfos id=20 count=14772lb # [ 78.648644] niks3-server[1127]: 2026/09/21 09:14:28 INFO Received complete upload request method=POST path=/api/pending_closures/20/complete4773lb # [ 78.755326] niks3-server[1127]: 2026/09/21 09:14:28 INFO Registered completed upload object_key=2i7aa0cdpp60vpdm180r56gqfh25m7hc.ls4774lb # [ 78.762567] niks3-server[1127]: 2026/09/21 09:14:28 INFO Registered completed upload object_key=log/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv4775lb # [ 78.769746] niks3-server[1127]: 2026/09/21 09:14:28 INFO Completed upload id=204776worker2 # [ 78.750962] nix-grpc-daemon[918]: time=2026-09-21T09:14:28.333Z level=INFO msg="Upload complete. (413ms)"4777lb # [ 78.773708] niks3-server[1127]: 2026/09/21 09:14:28 INFO Registered completed upload object_key=2i7aa0cdpp60vpdm180r56gqfh25m7hc.narinfo4778worker2 # [ 78.763129] nix-grpc-daemon[781]: ts=2026-09-21T09:14:28Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34490 duration_s=1 drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv assign_id=9 outputs=14779worker2 # [ 78.783314] nix-daemon[870]: reaped child process 1014, status = succeeded4780client # copying 1 paths...4781worker1 # [ 78.805228] nix-grpc-daemon[774]: ts=2026-09-21T09:14:28Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:353504782worker1 # [ 78.823947] nix-grpc-daemon[774]: copying path '/nix/store/2i7aa0cdpp60vpdm180r56gqfh25m7hc-uses-input-w1onlya' from 'http://lb:5751'...4783client # copying path '/nix/store/2i7aa0cdpp60vpdm180r56gqfh25m7hc-uses-input-w1onlya' from 'grpc://lb:50051'...4784worker1 # [ 78.876126] nix-grpc-daemon[774]: ts=2026-09-21T09:14:28Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:353504785worker1 # [ 78.881591] nix-grpc-daemon[774]: ts=2026-09-21T09:14:28Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:35350 duration_s=0 paths=1 nar_bytes_out=1204786client # [ 79.007051] nix-daemon[937]: reaped child process 1021, status = succeeded4787client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix job --argstr tag w1only --argstr salt a >&2, in 3.31 seconds)4788client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix job --argstr tag w1only --argstr salt b >&24789client # [ 79.228913] nix-daemon[937]: accepted connection from pid 1040, user root (trusted)4790client # this derivation will be built:4791client # /nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv4792worker1 # [ 79.445118] nix-grpc-daemon[774]: ts=2026-09-21T09:14:29Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:353504793client # building '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' on 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux'...4794worker2 # [ 79.457450] nix-grpc-daemon[781]: ts=2026-09-21T09:14:29Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:344904795worker2 # [ 79.469680] nix-grpc-daemon[918]: time=2026-09-21T09:14:29.052Z level=INFO msg="All 1 paths already cached"4796worker2 # [ 79.487011] nix-grpc-daemon[781]: ts=2026-09-21T09:14:29Z level=debug event=want client=10 drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv cached=04797worker1 # [ 79.482753] nix-grpc-daemon[774]: ts=2026-09-21T09:14:29Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:353504798worker2 # [ 79.493635] nix-grpc-daemon[781]: ts=2026-09-21T09:14:29Z level=debug event=assigned drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv worker=192.168.1.3:50051 clients=14799worker1 # [ 79.498533] nix-grpc-daemon[774]: ts=2026-09-21T09:14:29Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:353504800client # copying 1 paths to 'lb:50051'...4801client # copying path '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' to 'lb:50051'...4802worker1 # [ 79.519296] nix-grpc-daemon[774]: ts=2026-09-21T09:14:29Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:353504803worker1 # [ 79.534390] nix-daemon[843]: accepted connection from pid 774, user nix-grpc-daemon (trusted)4804worker1 # [ 79.556731] nix-grpc-daemon[774]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 83 ms (attempt 1/5)4805worker1 # [ 79.646057] nix-grpc-daemon[774]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 161 ms (attempt 2/5)4806worker1 # [ 79.815774] nix-grpc-daemon[774]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 197 ms (attempt 3/5)4807worker1 # [ 80.020575] nix-grpc-daemon[774]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 689 ms (attempt 4/5)4808worker1 # [ 80.717829] nix-grpc-daemon[774]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4809worker1 # [ 80.725805] nix-grpc-daemon[774]: copying path '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv' from 'http://lb:5751'...4810lb # [ 80.965401] envoy[913]: conn peer=[2001:db8:1::1]:47356 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=436 tx=375 ms=694811lb # [ 80.970351] envoy[913]: conn peer=[2001:db8:1::1]:48994 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3860 tx=1989 ms=19004812lb # [ 80.975397] envoy[913]: conn peer=[2001:db8:1::1]:47368 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4338 tx=2109 ms=23154813lb # [ 80.981337] envoy[913]: conn peer=[2001:db8:1::1]:47384 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=438 tx=407 ms=954814lb # [ 80.986015] envoy[913]: conn peer=[2001:db8:1::1]:47390 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3861 tx=1877 ms=27814815worker1 # [ 80.965691] nix-daemon[843]: accepted connection from pid 1153, user nix-grpc-daemon (trusted)4816lb # [ 80.990883] envoy[913]: conn peer=[2001:db8:1::1]:47404 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=448 tx=328 ms=644817worker1 # [ 80.985155] nix-daemon[843]: reaped child process 1156, status = succeeded4818worker1 # [ 81.158782] nix-daemon[843]: accepted connection from pid 1158, user nix-grpc-daemon (trusted)4819worker1 # [ 81.179305] nix-daemon[843]: reaped child process 1161, status = succeeded4820lb # [ 81.218087] niks3-server[1127]: 2026/09/21 09:14:30 INFO Received uploads request method=POST path=/api/pending_closures4821worker1 # [ 81.241196] nix-grpc-daemon[853]: time=2026-09-21T09:14:30.836Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4822worker1 # [ 81.243481] nix-grpc-daemon[853]: time=2026-09-21T09:14:30.838Z level=INFO msg="Uploading 12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv (568B)"4823lb # [ 81.285892] niks3-server[1127]: 2026/09/21 09:14:30 INFO Registered completed upload object_key=nar/14jznd749vfl63s1gv7i29r5z6jcai2byp2nz83dr5gib6z2cyaz.nar.zst4824lb # [ 81.299969] niks3-server[1127]: 2026/09/21 09:14:30 INFO Received sign narinfos request method=POST path=/api/pending_closures/21/sign4825worker1 # [ 81.283725] nix-grpc-daemon[853]: time=2026-09-21T09:14:30.879Z level=INFO msg="Uploading 1 narinfos"4826lb # [ 81.309053] niks3-server[1127]: 2026/09/21 09:14:30 INFO Signed narinfos id=21 count=14827lb # [ 81.315871] niks3-server[1127]: 2026/09/21 09:14:30 INFO Registered completed upload object_key=12ialcyr3l4m98rzfgk6apzimy55z3j8.ls4828lb # [ 81.330000] niks3-server[1127]: 2026/09/21 09:14:30 INFO Received complete upload request method=POST path=/api/pending_closures/21/complete4829lb # [ 81.338080] niks3-server[1127]: 2026/09/21 09:14:30 INFO Completed upload id=214830worker1 # [ 81.312084] nix-grpc-daemon[853]: time=2026-09-21T09:14:30.907Z level=INFO msg="Upload complete. (556ms)"4831lb # [ 81.341711] niks3-server[1127]: 2026/09/21 09:14:30 INFO Registered completed upload object_key=12ialcyr3l4m98rzfgk6apzimy55z3j8.narinfo4832worker1 # [ 81.324178] nix-grpc-daemon[774]: ts=2026-09-21T09:14:30Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35350 duration_s=1 paths=1 nar_bytes_in=5684833worker1 # [ 81.343529] nix-grpc-daemon[774]: ts=2026-09-21T09:14:30Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:353504834worker1 # [ 81.353092] nix-daemon[843]: accepted connection from pid 774, user nix-grpc-daemon (trusted)4835worker1 # [ 81.359178] nix-daemon[843]: reaped child process 1147, status = succeeded4836client # building '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' on lb:50051...4837worker1 # [ 81.373846] nix-daemon[843]: accepted connection from pid 774, user nix-grpc-daemon (trusted)4838client # uses-input-w1onlyb> node-a: building 12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv4839worker1 # [ 82.404178] nix-daemon[843]: reaped child process 1165, status = succeeded4840worker1 # [ 82.573509] nix-daemon[843]: accepted connection from pid 1180, user nix-grpc-daemon (trusted)4841worker1 # [ 82.590730] nix-daemon[843]: reaped child process 1183, status = succeeded4842lb # [ 82.630670] niks3-server[1127]: 2026/09/21 09:14:32 INFO Received uploads request method=POST path=/api/pending_closures4843worker1 # [ 82.612384] nix-grpc-daemon[853]: time=2026-09-21T09:14:32.206Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"4844lb # [ 82.658276] niks3-server[1127]: 2026/09/21 09:14:32 INFO Registered completed upload object_key=log/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv4845worker1 # [ 82.644914] nix-grpc-daemon[853]: time=2026-09-21T09:14:32.240Z level=INFO msg="Uploading 1 narinfos"4846lb # [ 82.674097] niks3-server[1127]: 2026/09/21 09:14:32 INFO Received sign narinfos request method=POST path=/api/pending_closures/22/sign4847lb # [ 82.683693] niks3-server[1127]: 2026/09/21 09:14:32 INFO Signed narinfos id=22 count=14848lb # [ 82.691574] niks3-server[1127]: 2026/09/21 09:14:32 INFO Registered completed upload object_key=jfy62nvxcd0b9rkhly152q21ddbrcn28.ls4849worker1 # [ 82.670513] nix-grpc-daemon[853]: time=2026-09-21T09:14:32.266Z level=INFO msg="Upload complete. (282ms)"4850lb # [ 82.696485] niks3-server[1127]: 2026/09/21 09:14:32 INFO Received complete upload request method=POST path=/api/pending_closures/22/complete4851lb # [ 82.702303] niks3-server[1127]: 2026/09/21 09:14:32 INFO Completed upload id=224852lb # [ 82.704746] niks3-server[1127]: 2026/09/21 09:14:32 INFO Registered completed upload object_key=jfy62nvxcd0b9rkhly152q21ddbrcn28.narinfo4853worker1 # [ 82.679863] nix-grpc-daemon[774]: ts=2026-09-21T09:14:32Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35350 duration_s=1 drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv assign_id=10 outputs=14854worker1 # [ 82.690866] nix-daemon[843]: reaped child process 1163, status = succeeded4855client # copying 1 paths...4856worker1 # [ 82.727054] nix-grpc-daemon[774]: ts=2026-09-21T09:14:32Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:353504857client # copying path '/nix/store/jfy62nvxcd0b9rkhly152q21ddbrcn28-uses-input-w1onlyb' from 'grpc://lb:50051'...4858worker2 # [ 82.770549] nix-grpc-daemon[781]: ts=2026-09-21T09:14:32Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:344904859worker2 # [ 82.785246] nix-grpc-daemon[781]: copying path '/nix/store/jfy62nvxcd0b9rkhly152q21ddbrcn28-uses-input-w1onlyb' from 'http://lb:5751'...4860worker2 # [ 82.802320] nix-grpc-daemon[781]: ts=2026-09-21T09:14:32Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:34490 duration_s=0 paths=1 nar_bytes_out=1204861client # [ 82.918566] nix-daemon[937]: reaped child process 1043, status = succeeded4862client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix job --argstr tag w1only --argstr salt b >&2, in 3.92 seconds)4863client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix job --argstr tag w1only --argstr salt c >&24864client # [ 83.145299] nix-daemon[937]: accepted connection from pid 1062, user root (trusted)4865client # this derivation will be built:4866client # /nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv4867worker2 # [ 83.369944] nix-grpc-daemon[781]: ts=2026-09-21T09:14:32Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:344904868client # building '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' on 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux'...4869worker1 # [ 83.367155] nix-grpc-daemon[774]: ts=2026-09-21T09:14:32Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:353504870worker1 # [ 83.378513] nix-grpc-daemon[853]: time=2026-09-21T09:14:32.974Z level=INFO msg="All 1 paths already cached"4871worker2 # [ 83.403545] nix-grpc-daemon[781]: ts=2026-09-21T09:14:32Z level=debug event=want client=11 drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv cached=04872worker2 # [ 83.411651] nix-grpc-daemon[781]: ts=2026-09-21T09:14:32Z level=debug event=assigned drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv worker=192.168.1.4:50051 clients=14873worker2 # [ 83.421179] nix-grpc-daemon[781]: ts=2026-09-21T09:14:33Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:344904874worker2 # [ 83.433041] nix-grpc-daemon[781]: ts=2026-09-21T09:14:33Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:344904875worker2 # [ 83.444441] nix-grpc-daemon[918]: time=2026-09-21T09:14:33.026Z level=INFO msg="All 1 paths already cached"4876client # copying 1 paths to 'lb:50051'...4877client # copying path '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' to 'lb:50051'...4878worker2 # [ 83.454347] nix-grpc-daemon[781]: ts=2026-09-21T09:14:33Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:344904879worker2 # [ 83.469355] nix-daemon[870]: accepted connection from pid 781, user nix-grpc-daemon (trusted)4880worker2 # [ 83.669778] nix-daemon[870]: accepted connection from pid 1044, user nix-grpc-daemon (trusted)4881worker2 # [ 83.688861] nix-daemon[870]: reaped child process 1047, status = succeeded4882worker2 # [ 83.862651] nix-daemon[870]: accepted connection from pid 1049, user nix-grpc-daemon (trusted)4883worker2 # [ 83.879642] nix-daemon[870]: reaped child process 1052, status = succeeded4884lb # [ 83.911420] niks3-server[1127]: 2026/09/21 09:14:33 INFO Received uploads request method=POST path=/api/pending_closures4885worker2 # [ 83.902225] nix-grpc-daemon[918]: time=2026-09-21T09:14:33.484Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4886worker2 # [ 83.903758] nix-grpc-daemon[918]: time=2026-09-21T09:14:33.484Z level=INFO msg="Uploading n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv (568B)"4887lb # [ 83.938247] niks3-server[1127]: 2026/09/21 09:14:33 INFO Registered completed upload object_key=nar/1qd3ps964395j2cc3rlqd58m21s7f6na9s0k2z5kclx0c3l4sl40.nar.zst4888lb # [ 83.953911] niks3-server[1127]: 2026/09/21 09:14:33 INFO Received sign narinfos request method=POST path=/api/pending_closures/23/sign4889worker2 # [ 83.941259] nix-grpc-daemon[918]: time=2026-09-21T09:14:33.523Z level=INFO msg="Uploading 1 narinfos"4890lb # [ 83.966306] niks3-server[1127]: 2026/09/21 09:14:33 INFO Signed narinfos id=23 count=14891lb # [ 83.972259] niks3-server[1127]: 2026/09/21 09:14:33 INFO Registered completed upload object_key=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr.ls4892lb # [ 83.979064] niks3-server[1127]: 2026/09/21 09:14:33 INFO Received complete upload request method=POST path=/api/pending_closures/23/complete4893lb # [ 83.985439] niks3-server[1127]: 2026/09/21 09:14:33 INFO Completed upload id=234894worker2 # [ 83.967065] nix-grpc-daemon[918]: time=2026-09-21T09:14:33.549Z level=INFO msg="Upload complete. (480ms)"4895lb # [ 83.987893] niks3-server[1127]: 2026/09/21 09:14:33 INFO Registered completed upload object_key=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr.narinfo4896worker2 # [ 83.977697] nix-grpc-daemon[781]: ts=2026-09-21T09:14:33Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34490 duration_s=0 paths=1 nar_bytes_in=5684897worker2 # [ 83.984994] nix-grpc-daemon[781]: ts=2026-09-21T09:14:33Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:344904898worker2 # [ 83.995157] nix-daemon[870]: accepted connection from pid 781, user nix-grpc-daemon (trusted)4899worker2 # [ 83.999685] nix-daemon[870]: reaped child process 1042, status = succeeded4900client # building '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' on lb:50051...4901worker2 # [ 84.011507] nix-daemon[870]: accepted connection from pid 781, user nix-grpc-daemon (trusted)4902client # uses-input-w1onlyc> worker2: building n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv4903worker2 # [ 85.048543] nix-daemon[870]: reaped child process 1056, status = succeeded4904worker2 # [ 85.219239] nix-daemon[870]: accepted connection from pid 1071, user nix-grpc-daemon (trusted)4905worker2 # [ 85.236984] nix-daemon[870]: reaped child process 1074, status = succeeded4906lb # [ 85.269410] niks3-server[1127]: 2026/09/21 09:14:34 INFO Received uploads request method=POST path=/api/pending_closures4907worker2 # [ 85.261284] nix-grpc-daemon[918]: time=2026-09-21T09:14:34.842Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"4908lb # [ 85.297537] niks3-server[1127]: 2026/09/21 09:14:34 INFO Received sign narinfos request method=POST path=/api/pending_closures/24/sign4909worker2 # [ 85.287811] nix-grpc-daemon[918]: time=2026-09-21T09:14:34.870Z level=INFO msg="Uploading 1 narinfos"4910lb # [ 85.311447] niks3-server[1127]: 2026/09/21 09:14:34 INFO Signed narinfos id=24 count=14911lb # [ 85.319318] niks3-server[1127]: 2026/09/21 09:14:34 INFO Registered completed upload object_key=13r5ipj38sk1irdi724wm1mbl9lw720b.ls4912lb # [ 85.327135] niks3-server[1127]: 2026/09/21 09:14:34 INFO Registered completed upload object_key=log/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv4913lb # [ 85.337815] niks3-server[1127]: 2026/09/21 09:14:34 INFO Received complete upload request method=POST path=/api/pending_closures/24/complete4914lb # [ 85.344312] niks3-server[1127]: 2026/09/21 09:14:34 INFO Completed upload id=244915worker2 # [ 85.329358] nix-grpc-daemon[918]: time=2026-09-21T09:14:34.911Z level=INFO msg="Upload complete. (305ms)"4916lb # [ 85.347971] niks3-server[1127]: 2026/09/21 09:14:34 INFO Registered completed upload object_key=13r5ipj38sk1irdi724wm1mbl9lw720b.narinfo4917worker2 # [ 85.341024] nix-grpc-daemon[781]: ts=2026-09-21T09:14:34Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34490 duration_s=1 drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv assign_id=11 outputs=14918worker2 # [ 85.354998] nix-daemon[870]: reaped child process 1054, status = succeeded4919client # copying 1 paths...4920worker2 # [ 85.385474] nix-grpc-daemon[781]: ts=2026-09-21T09:14:34Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:344904921client # copying path '/nix/store/13r5ipj38sk1irdi724wm1mbl9lw720b-uses-input-w1onlyc' from 'grpc://lb:50051'...4922worker1 # [ 85.409099] nix-grpc-daemon[774]: ts=2026-09-21T09:14:35Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:353504923worker1 # [ 85.422245] nix-grpc-daemon[774]: copying path '/nix/store/13r5ipj38sk1irdi724wm1mbl9lw720b-uses-input-w1onlyc' from 'http://lb:5751'...4924worker1 # [ 85.440685] nix-grpc-daemon[774]: ts=2026-09-21T09:14:35Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:35350 duration_s=0 paths=1 nar_bytes_out=1204925client # [ 85.571911] nix-daemon[937]: reaped child process 1065, status = succeeded4926client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix job --argstr tag w1only --argstr salt c >&2, in 2.65 seconds)4927(finished: subtest: an input only one worker has still reaches the builder, in 10.80 seconds)4928subtest: upload whose reference the worker lacks is completed from the cache4929client: must succeed: nix-build --no-out-link /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix -A input --argstr tag viacache4930client # this derivation will be built:4931client # /nix/store/x1rrnjihpa1xvpyxxxdrjzp3c8z8nayq-local-input-viacache.drv4932client # building '/nix/store/x1rrnjihpa1xvpyxxxdrjzp3c8z8nayq-local-input-viacache.drv'...4933client: (finished: must succeed: nix-build --no-out-link /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix -A input --argstr tag viacache, in 0.45 seconds)4934client: must succeed: nix-build --no-out-link /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix -A referrer --argstr tag viacache4935client # this derivation will be built:4936client # /nix/store/1m587dk4visgyryf6n0mgp2rlp7pbjd1-refers-to-input-viacache.drv4937client # building '/nix/store/1m587dk4visgyryf6n0mgp2rlp7pbjd1-refers-to-input-viacache.drv'...4938client: (finished: must succeed: nix-build --no-out-link /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix -A referrer --argstr tag viacache, in 0.43 seconds)4939client: must succeed: nix-store --export /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache > /tmp/shared/ref.closure4940client: (finished: must succeed: nix-store --export /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache > /tmp/shared/ref.closure, in 0.20 seconds)4941worker1: must succeed: nix-store --import < /tmp/shared/ref.closure4942worker1: (finished: must succeed: nix-store --import < /tmp/shared/ref.closure, in 0.24 seconds)4943worker2 # [ 86.990901] systemd[1]: Stopping Nix worker-protocol over gRPC...4944worker2 # [ 87.524980] nix-grpc-daemon[781]: ts=2026-09-21T09:14:37Z level=info event=signal_exit4945worker2 # [ 87.975396] nix-grpc-daemon[781]: ts=2026-09-21T09:14:37Z level=info event=scheduler_yield4946lb # [ 87.998301] niks3-server[1127]: 2026/09/21 09:14:37 INFO lead: released remote=192.168.1.4:485764947worker2 # [ 87.981037] nix-grpc-daemon[781]: ts=2026-09-21T09:14:37Z level=info event=scheduler_restarting addr=lb:500514948worker2 # [ 87.986639] nix-grpc-daemon[781]: ts=2026-09-21T09:14:37Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:344744949worker1 # [ 87.982620] nix-grpc-daemon[774]: ts=2026-09-21T09:14:37Z level=info event=scheduler_restarting addr=lb:500514950worker2 # [ 87.991356] nix-grpc-daemon[781]: ts=2026-09-21T09:14:37Z level=info event=worker_gone addr=192.168.1.4:500514951worker2 # [ 87.997015] nix-grpc-daemon[781]: ts=2026-09-21T09:14:37Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:344744952worker2 # [ 88.001480] nix-grpc-daemon[781]: ts=2026-09-21T09:14:37Z level=info event=worker_gone addr=192.168.1.3:500514953worker2 # [ 88.182051] nix-grpc-daemon[781]: ts=2026-09-21T09:14:37Z level=info event=scheduler_connected addr=lb:500514954worker2 # [ 88.187110] nix-grpc-daemon[781]: ts=2026-09-21T09:14:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4955worker1 # [ 88.182851] nix-grpc-daemon[774]: ts=2026-09-21T09:14:37Z level=info event=scheduler_connected addr=lb:500514956worker1 # [ 88.188086] nix-grpc-daemon[774]: ts=2026-09-21T09:14:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4957worker2 # [ 88.388412] nix-grpc-daemon[781]: ts=2026-09-21T09:14:37Z level=info event=scheduler_connected addr=lb:500514958worker2 # [ 88.393366] nix-grpc-daemon[781]: ts=2026-09-21T09:14:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4959worker1 # [ 88.388930] nix-grpc-daemon[774]: ts=2026-09-21T09:14:37Z level=info event=scheduler_connected addr=lb:500514960worker1 # [ 88.393900] nix-grpc-daemon[774]: ts=2026-09-21T09:14:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4961worker2 # [ 88.596188] nix-grpc-daemon[781]: ts=2026-09-21T09:14:38Z level=info event=scheduler_connected addr=lb:500514962worker2 # [ 88.600231] nix-grpc-daemon[781]: ts=2026-09-21T09:14:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4963worker1 # [ 88.594610] nix-grpc-daemon[774]: ts=2026-09-21T09:14:38Z level=info event=scheduler_connected addr=lb:500514964worker1 # [ 88.599906] nix-grpc-daemon[774]: ts=2026-09-21T09:14:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4965worker2 # [ 88.796953] nix-grpc-daemon[781]: ts=2026-09-21T09:14:38Z level=info event=scheduler_connected addr=lb:500514966worker2 # [ 88.802287] nix-grpc-daemon[781]: ts=2026-09-21T09:14:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4967worker1 # [ 88.800966] nix-grpc-daemon[774]: ts=2026-09-21T09:14:38Z level=info event=scheduler_connected addr=lb:500514968worker1 # [ 88.805971] nix-grpc-daemon[774]: ts=2026-09-21T09:14:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4969worker2 # [ 88.981290] nix-grpc-daemon[781]: ts=2026-09-21T09:14:38Z level=info event=scheduler_restarting4970worker1 # [ 89.008508] nix-grpc-daemon[774]: ts=2026-09-21T09:14:38Z level=info event=scheduler_connected addr=lb:500514971worker1 # [ 89.012540] nix-grpc-daemon[774]: ts=2026-09-21T09:14:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4972worker2 # [ 89.025839] nix-grpc-daemon[781]: ts=2026-09-21T09:14:38Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"4973worker2 # [ 89.042014] nix-daemon[870]: reaped child process 873, status = succeeded4974worker1 # [ 89.208989] nix-grpc-daemon[774]: ts=2026-09-21T09:14:38Z level=info event=scheduler_connected addr=lb:500514975worker1 # [ 89.214156] nix-grpc-daemon[774]: ts=2026-09-21T09:14:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4976worker1 # [ 89.414879] nix-grpc-daemon[774]: ts=2026-09-21T09:14:39Z level=info event=scheduler_connected addr=lb:500514977worker1 # [ 89.420067] nix-grpc-daemon[774]: ts=2026-09-21T09:14:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4978worker1 # [ 89.620950] nix-grpc-daemon[774]: ts=2026-09-21T09:14:39Z level=info event=scheduler_connected addr=lb:500514979worker1 # [ 89.625949] nix-grpc-daemon[774]: ts=2026-09-21T09:14:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4980worker2 # [ 89.657805] nix-grpc-daemon[781]: E0000 00:00:1789982079.239181 826 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 16, started true, shutdown true4981worker2 # [ 89.669759] nix-grpc-daemon[781]: E0000 00:00:1789982079.252348 826 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true4982worker1 # [ 89.826734] nix-grpc-daemon[774]: ts=2026-09-21T09:14:39Z level=info event=scheduler_connected addr=lb:500514983worker1 # [ 89.831661] nix-grpc-daemon[774]: ts=2026-09-21T09:14:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4984worker1 # [ 90.034305] nix-grpc-daemon[774]: ts=2026-09-21T09:14:39Z level=info event=scheduler_connected addr=lb:500514985worker1 # [ 90.038490] nix-grpc-daemon[774]: ts=2026-09-21T09:14:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4986worker1 # [ 90.234872] nix-grpc-daemon[774]: ts=2026-09-21T09:14:39Z level=info event=scheduler_connected addr=lb:500514987worker1 # [ 90.239971] nix-grpc-daemon[774]: ts=2026-09-21T09:14:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4988worker1 # [ 90.440845] nix-grpc-daemon[774]: ts=2026-09-21T09:14:40Z level=info event=scheduler_connected addr=lb:500514989worker1 # [ 90.445895] nix-grpc-daemon[774]: ts=2026-09-21T09:14:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4990worker1 # [ 90.646550] nix-grpc-daemon[774]: ts=2026-09-21T09:14:40Z level=info event=scheduler_connected addr=lb:500514991worker1 # [ 90.651579] nix-grpc-daemon[774]: ts=2026-09-21T09:14:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4992worker1 # [ 90.852532] nix-grpc-daemon[774]: ts=2026-09-21T09:14:40Z level=info event=scheduler_connected addr=lb:500514993worker1 # [ 90.857630] nix-grpc-daemon[774]: ts=2026-09-21T09:14:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4994worker2 # [ 90.915018] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.4995worker2 # [ 90.918218] systemd[1]: Stopped Nix worker-protocol over gRPC.4996worker2 # [ 90.922745] systemd[1]: nix-grpc-daemon.service: Consumed 2.111s CPU time over 1min 9.765s wall clock time, 55.2M memory peak, 65.5K incoming IP traffic, 205.7K outgoing IP traffic.4997worker2 # [ 90.933780] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.4998worker2 # [ 90.935851] systemd[1]: Closed Nix worker-protocol over gRPC.4999worker2 # [ 90.940612] systemd[1]: nix-grpc-daemon.socket: Consumed 544us CPU time over 1min 20.154s wall clock time, 256K memory peak, 78.4K incoming IP traffic, 67K outgoing IP traffic.5000lb: waiting until exactly ['worker2'] endpoints of aarch64-linux are unhealthy5001lb: must succeed: curl -sf localhost:9901/clusters5002lb # [ 90.963872] envoy[913]: conn peer=[2001:db8:1::1]:40524 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=448 tx=329 ms=965003lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5004lb: (finished: waiting until exactly ['worker2'] endpoints of aarch64-linux are unhealthy, in 0.09 seconds)5005client: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache >&25006worker1 # [ 91.060179] nix-grpc-daemon[774]: ts=2026-09-21T09:14:40Z level=info event=scheduler_connected addr=lb:500515007worker1 # [ 91.064106] nix-grpc-daemon[774]: ts=2026-09-21T09:14:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5008worker1 # [ 91.260749] nix-grpc-daemon[774]: ts=2026-09-21T09:14:40Z level=info event=scheduler_connected addr=lb:500515009worker1 # [ 91.265760] nix-grpc-daemon[774]: ts=2026-09-21T09:14:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5010worker1 # [ 91.285247] nix-grpc-daemon[774]: ts=2026-09-21T09:14:40Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:353505011worker1 # [ 91.293472] nix-grpc-daemon[774]: ts=2026-09-21T09:14:40Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:353505012client # /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache5013client: (finished: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache >&2, in 0.30 seconds)5014worker2: must fail: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache5015worker2: (finished: must fail: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache, in 0.03 seconds)5016worker2 # [ 91.438149] systemd[1]: Listening on Nix worker-protocol over gRPC.5017worker1 # [ 91.468681] nix-grpc-daemon[774]: ts=2026-09-21T09:14:41Z level=info event=scheduler_connected addr=lb:500515018worker1 # [ 91.473866] nix-grpc-daemon[774]: ts=2026-09-21T09:14:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5019worker1 # [ 91.520749] systemd[1]: Stopping Nix worker-protocol over gRPC...5020lb # [ 90.968861] envoy[913]: conn peer=[2001:db8:1::1]:47408 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3713 tx=1804 ms=34095021lb # [ 90.969019] envoy[913]: conn peer=[2001:db8:1::1]:40534 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3713 tx=1805 ms=21375022lb # [ 90.969125] envoy[913]: conn peer=[2001:db8:1::1]:40542 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=448 tx=329 ms=975023lb # [ 90.969225] envoy[913]: conn peer=[2001:db8:1::4]:37328 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=82395 tx=3985 ms=674465024worker1 # [ 91.669609] nix-grpc-daemon[774]: ts=2026-09-21T09:14:41Z level=info event=scheduler_connected addr=lb:500515025worker1 # [ 91.674872] nix-grpc-daemon[774]: ts=2026-09-21T09:14:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5026worker1 # [ 91.875630] nix-grpc-daemon[774]: ts=2026-09-21T09:14:41Z level=info event=scheduler_connected addr=lb:500515027worker1 # [ 91.881028] nix-grpc-daemon[774]: ts=2026-09-21T09:14:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5028worker2 # [ 91.904163] systemd[1]: Starting scheduler-token.service...5029worker2 # [ 92.008810] systemd[1]: scheduler-token.service: Deactivated successfully.5030worker2 # [ 92.015058] systemd[1]: Finished scheduler-token.service.5031worker2 # [ 92.019362] systemd[1]: scheduler-token.service: Consumed 49ms CPU time over 107ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5032worker2 # [ 92.024169] systemd[1]: Starting Nix worker-protocol over gRPC...5033worker1 # [ 92.083470] nix-grpc-daemon[774]: ts=2026-09-21T09:14:41Z level=info event=scheduler_connected addr=lb:500515034worker1 # [ 92.087723] nix-grpc-daemon[774]: ts=2026-09-21T09:14:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5035worker2 # [ 92.210839] nix-grpc-daemon[1098]: ts=2026-09-21T09:14:41Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5036worker2 # [ 92.213357] nix-grpc-daemon[1098]: ts=2026-09-21T09:14:41Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5037worker2 # [ 92.217388] nix-grpc-daemon[1098]: ts=2026-09-21T09:14:41Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15038worker2 # [ 92.224055] nix-grpc-daemon[1098]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5039worker2 # [ 92.225494] nix-grpc-daemon[1098]: E0000 00:00:1789982081.806624 1098 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5040worker2 # [ 92.239791] nix-grpc-daemon[1098]: E0000 00:00:1789982081.812862 1098 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5041worker2 # [ 92.243989] nix-grpc-daemon[1098]: ts=2026-09-21T09:14:41Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5042worker2 # [ 92.246504] systemd[1]: Started Nix worker-protocol over gRPC.5043lb # [ 92.262388] niks3-server[1127]: 2026/09/21 09:14:41 INFO lead: acquired remote=192.168.1.4:581265044worker2 # [ 92.247773] nix-grpc-daemon[1098]: ts=2026-09-21T09:14:41Z level=info event=scheduler_take_over5045worker2 # [ 92.250901] nix-grpc-daemon[1098]: E0000 00:00:1789982081.833082 1140 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5046worker2 # [ 92.280139] nix-grpc-daemon[1098]: E0000 00:00:1789982081.852715 1140 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5047worker2 # [ 92.282445] nix-grpc-daemon[1098]: E0000 00:00:1789982081.852862 1140 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5048worker2 # [ 92.290022] nix-grpc-daemon[1098]: E0000 00:00:1789982081.872035 1140 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5049worker1 # [ 92.284541] nix-grpc-daemon[774]: ts=2026-09-21T09:14:41Z level=info event=scheduler_connected addr=lb:500515050worker2 # [ 92.300672] nix-grpc-daemon[1098]: ts=2026-09-21T09:14:41Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:398805051worker2 # [ 92.304926] nix-grpc-daemon[1098]: ts=2026-09-21T09:14:41Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=05052worker2 # [ 92.307002] nix-grpc-daemon[1098]: ts=2026-09-21T09:14:41Z level=info event=scheduler_connected addr=lb:500515053worker2 # [ 92.311440] nix-grpc-daemon[1098]: ts=2026-09-21T09:14:41Z level=info event=oidc_jwks provider=mock keys=15054worker2 # [ 92.312993] nix-grpc-daemon[1098]: ts=2026-09-21T09:14:41Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:398805055worker2 # [ 92.314631] nix-grpc-daemon[1098]: ts=2026-09-21T09:14:41Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=05056worker1 # [ 92.353606] nix-grpc-daemon[774]: ts=2026-09-21T09:14:41Z level=info event=signal_exit5057worker2 # [ 93.455101] nix-grpc-daemon[1098]: ts=2026-09-21T09:14:43Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:398805058worker2 # [ 93.459776] nix-grpc-daemon[1098]: ts=2026-09-21T09:14:43Z level=info event=worker_gone addr=192.168.1.3:500515059worker1 # [ 93.469261] nix-grpc-daemon[774]: ts=2026-09-21T09:14:43Z level=info event=scheduler_restarting5060worker1 # [ 93.484649] nix-grpc-daemon[774]: ts=2026-09-21T09:14:43Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5061worker1 # [ 93.501429] nix-daemon[843]: reaped child process 845, status = succeeded5062worker1 # [ 93.504296] nix-grpc-daemon[774]: E0000 00:00:1789982083.099546 819 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 15, started true, shutdown true5063worker1 # [ 93.515344] nix-grpc-daemon[774]: E0000 00:00:1789982083.110883 819 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5064worker1 # [ 94.827194] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5065worker1 # [ 94.830481] systemd[1]: Stopped Nix worker-protocol over gRPC.5066worker1 # [ 94.836518] systemd[1]: nix-grpc-daemon.service: Consumed 3.873s CPU time over 1min 13.873s wall clock time, 63M memory peak, 103.5K incoming IP traffic, 190K outgoing IP traffic.5067worker1 # [ 94.851617] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5068worker1 # [ 94.857630] systemd[1]: Closed Nix worker-protocol over gRPC.5069worker1 # [ 94.860315] systemd[1]: nix-grpc-daemon.socket: Consumed 696us CPU time over 1min 23.347s wall clock time, 256K memory peak, 86.3K incoming IP traffic, 69.7K outgoing IP traffic.5070lb: waiting until exactly ['worker1'] endpoints of aarch64-linux are unhealthy5071lb: must succeed: curl -sf localhost:9901/clusters5072lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5073lb: (finished: waiting until exactly ['worker1'] endpoints of aarch64-linux are unhealthy, in 0.08 seconds)5074client: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache >&25075worker2 # [ 95.241936] nix-grpc-daemon[1098]: ts=2026-09-21T09:14:44Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:398905076worker2 # [ 95.265843] nix-daemon[870]: accepted connection from pid 1098, user nix-grpc-daemon (trusted)5077worker2 # [ 95.296234] nix-grpc-daemon[1098]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 50 ms (attempt 1/5)5078worker2 # [ 95.352382] nix-grpc-daemon[1098]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 126 ms (attempt 2/5)5079worker2 # [ 95.486193] nix-grpc-daemon[1098]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 261 ms (attempt 3/5)5080worker2 # [ 95.747692] nix-grpc-daemon[1098]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 635 ms (attempt 4/5)5081worker2 # [ 96.392909] nix-grpc-daemon[1098]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5082client # copying 2 paths...5083client # copying path '/nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache' to 'grpc://lb:50051'...5084worker2 # [ 96.431846] nix-grpc-daemon[1098]: ts=2026-09-21T09:14:46Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:398905085client # copying path '/nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache' to 'grpc://lb:50051'...5086worker2 # [ 96.461066] nix-daemon[870]: accepted connection from pid 1098, user nix-grpc-daemon (trusted)5087worker2 # [ 96.498991] nix-grpc-daemon[1098]: ts=2026-09-21T09:14:46Z level=info event=niks3_push_started pid=11575088worker2 # [ 96.785169] nix-grpc-daemon[1157]: time=2026-09-21T09:14:46.367Z level=INFO msg="Configuring client TLS" cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker2.pem key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker2.key ca=""5089worker2 # [ 97.000190] nix-daemon[870]: accepted connection from pid 1170, user nix-grpc-daemon (trusted)5090worker2 # [ 97.021884] nix-daemon[870]: reaped child process 1173, status = succeeded5091lb # [ 97.055829] niks3-server[1127]: 2026/09/21 09:14:46 INFO Received uploads request method=POST path=/api/pending_closures5092lb # [ 97.069867] niks3-server[1127]: 2026/09/21 09:14:46 INFO Received uploads request method=POST path=/api/pending_closures5093worker2 # [ 97.068166] nix-grpc-daemon[1157]: time=2026-09-21T09:14:46.647Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"5094worker2 # [ 97.070099] nix-grpc-daemon[1157]: time=2026-09-21T09:14:46.647Z level=INFO msg="Uploading 4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache (184B)"5095worker2 # [ 97.073646] nix-grpc-daemon[1157]: time=2026-09-21T09:14:46.648Z level=INFO msg="Uploading syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache (128B)"5096lb # [ 97.144155] niks3-server[1127]: 2026/09/21 09:14:46 INFO Registered completed upload object_key=nar/1kmnh5f2l7jjcgaq9b9988bjix6z8zjf6v350xllq5hlzgnb1f75.nar.zst5097lb # [ 97.168818] niks3-server[1127]: 2026/09/21 09:14:46 INFO Registered completed upload object_key=nar/1rf1rg73pvnaa9kk6sy9r1ydj24b4g0kb0wnx0z5hxl1nvkmlavq.nar.zst5098lb # [ 97.184482] niks3-server[1127]: 2026/09/21 09:14:46 INFO Registered completed upload object_key=syvivri0wisb7fd33rxcg9kz63kc2bv7.ls5099lb # [ 97.192237] niks3-server[1127]: 2026/09/21 09:14:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/26/sign5100worker2 # [ 97.174775] nix-grpc-daemon[1157]: time=2026-09-21T09:14:46.755Z level=INFO msg="Uploading 3 narinfos"5101lb # [ 97.200660] niks3-server[1127]: 2026/09/21 09:14:46 INFO Signed narinfos id=26 count=25102lb # [ 97.203001] niks3-server[1127]: 2026/09/21 09:14:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/25/sign5103lb # [ 97.213008] niks3-server[1127]: 2026/09/21 09:14:46 INFO Signed narinfos id=25 count=15104lb # [ 97.215056] niks3-server[1127]: 2026/09/21 09:14:46 INFO Registered completed upload object_key=4rig83zwv4xj1zajs08ima13w0yh676v.ls5105lb # [ 97.225225] niks3-server[1127]: 2026/09/21 09:14:46 INFO Registered completed upload object_key=4rig83zwv4xj1zajs08ima13w0yh676v.narinfo5106lb # [ 97.233293] niks3-server[1127]: 2026/09/21 09:14:46 INFO Received complete upload request method=POST path=/api/pending_closures/25/complete5107lb # [ 97.242105] niks3-server[1127]: 2026/09/21 09:14:46 INFO Registered completed upload object_key=syvivri0wisb7fd33rxcg9kz63kc2bv7.narinfo5108lb # [ 97.247161] niks3-server[1127]: 2026/09/21 09:14:46 INFO Registered completed upload object_key=syvivri0wisb7fd33rxcg9kz63kc2bv7.narinfo5109lb # [ 97.262163] niks3-server[1127]: 2026/09/21 09:14:46 INFO Completed upload id=255110lb # [ 97.265107] niks3-server[1127]: 2026/09/21 09:14:46 INFO Received complete upload request method=POST path=/api/pending_closures/26/complete5111lb # [ 97.268001] niks3-server[1127]: 2026/09/21 09:14:46 INFO Completed upload id=265112worker2 # [ 97.249934] nix-grpc-daemon[1157]: time=2026-09-21T09:14:46.832Z level=INFO msg="Upload complete. (458ms)"5113worker2 # [ 97.260417] nix-grpc-daemon[1098]: ts=2026-09-21T09:14:46Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39890 duration_s=0 paths=2 nar_bytes_in=3125114worker2 # [ 97.281356] nix-daemon[870]: reaped child process 1155, status = succeeded5115client: (finished: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache >&2, in 2.35 seconds)5116worker2: must succeed: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache && test -e /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache5117worker2: (finished: must succeed: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache && test -e /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache, in 0.04 seconds)5118worker1 # [ 97.433276] systemd[1]: Listening on Nix worker-protocol over gRPC.5119worker1 # [ 97.442252] systemd[1]: Starting Nix worker-protocol over gRPC...5120worker1 # [ 97.641453] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:47Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5121worker1 # [ 97.643703] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:47Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5122worker1 # [ 97.648093] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:47Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15123worker1 # [ 97.656073] nix-grpc-daemon[1207]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5124worker1 # [ 97.657579] nix-grpc-daemon[1207]: E0000 00:00:1789982087.250347 1207 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5125worker1 # [ 97.668679] nix-grpc-daemon[1207]: E0000 00:00:1789982087.256954 1207 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5126worker1 # [ 97.673780] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:47Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5127worker1 # [ 97.675998] systemd[1]: Started Nix worker-protocol over gRPC.5128worker1 # [ 97.678493] nix-grpc-daemon[1207]: E0000 00:00:1789982087.273422 1246 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5129lb: waiting until exactly no endpoints of aarch64-linux are unhealthy5130lb: must succeed: curl -sf localhost:9901/clusters5131worker1 # [ 97.708221] nix-grpc-daemon[1207]: E0000 00:00:1789982087.291781 1246 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5132worker1 # [ 97.710508] nix-grpc-daemon[1207]: E0000 00:00:1789982087.291976 1246 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5133worker1 # [ 97.716755] nix-grpc-daemon[1207]: E0000 00:00:1789982087.311438 1246 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5134worker1 # [ 97.740288] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:47Z level=info event=scheduler_connected addr=lb:500515135worker2 # [ 97.750493] nix-grpc-daemon[1098]: ts=2026-09-21T09:14:47Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:398805136worker2 # [ 97.755533] nix-grpc-daemon[1098]: ts=2026-09-21T09:14:47Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=05137lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.11 seconds)5138lb: must succeed: curl -sf localhost:9901/clusters5139lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5140lb: (finished: waiting until exactly no endpoints of aarch64-linux are unhealthy, in 1.19 seconds)5141(finished: subtest: upload whose reference the worker lacks is completed from the cache, in 13.29 seconds)5142subtest: scheduler down: the other node takes the lock and keeps it5143lb: must succeed: curl -sf localhost:9901/clusters5144lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)5145lb: must succeed: curl -sf localhost:9901/clusters5146lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5147lb: must succeed: curl -sf localhost:9901/clusters5148lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5149lb: must succeed: curl -sf localhost:9901/clusters5150lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5151worker2 # [ 99.148813] systemd[1]: Stopping Nix worker-protocol over gRPC...5152worker2 # [ 99.236686] nix-grpc-daemon[1098]: ts=2026-09-21T09:14:48Z level=info event=signal_exit5153worker2 # [ 99.446733] nix-grpc-daemon[1098]: ts=2026-09-21T09:14:49Z level=info event=scheduler_yield5154lb # [ 99.471142] niks3-server[1127]: 2026/09/21 09:14:49 INFO lead: released remote=192.168.1.4:581265155worker1 # [ 99.447310] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:49Z level=info event=scheduler_restarting addr=lb:500515156worker2 # [ 99.454098] nix-grpc-daemon[1098]: ts=2026-09-21T09:14:49Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:398805157worker2 # [ 99.460693] nix-grpc-daemon[1098]: ts=2026-09-21T09:14:49Z level=info event=worker_gone addr=192.168.1.3:500515158worker2 # [ 99.466697] nix-grpc-daemon[1098]: ts=2026-09-21T09:14:49Z level=info event=scheduler_restarting addr=lb:500515159worker2 # [ 99.470896] nix-grpc-daemon[1098]: ts=2026-09-21T09:14:49Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:398805160worker2 # [ 99.475476] nix-grpc-daemon[1098]: ts=2026-09-21T09:14:49Z level=info event=worker_gone addr=192.168.1.4:500515161worker1 # [ 99.646551] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:49Z level=info event=scheduler_connected addr=lb:500515162worker1 # [ 99.653547] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5163worker2 # [ 99.662022] nix-grpc-daemon[1098]: ts=2026-09-21T09:14:49Z level=info event=scheduler_connected addr=lb:500515164worker2 # [ 99.667178] nix-grpc-daemon[1098]: ts=2026-09-21T09:14:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5165worker1 # [ 99.854364] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:49Z level=info event=scheduler_connected addr=lb:500515166worker1 # [ 99.859818] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5167worker2 # [ 99.868286] nix-grpc-daemon[1098]: ts=2026-09-21T09:14:49Z level=info event=scheduler_connected addr=lb:500515168worker2 # [ 99.873542] nix-grpc-daemon[1098]: ts=2026-09-21T09:14:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5169worker1 # [ 100.060714] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:49Z level=info event=scheduler_connected addr=lb:500515170worker1 # [ 100.066077] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5171worker2 # [ 100.074498] nix-grpc-daemon[1098]: ts=2026-09-21T09:14:49Z level=info event=scheduler_connected addr=lb:500515172worker2 # [ 100.079576] nix-grpc-daemon[1098]: ts=2026-09-21T09:14:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5173worker1 # [ 100.266952] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:49Z level=info event=scheduler_connected addr=lb:500515174worker1 # [ 100.272413] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5175worker2 # [ 100.282127] nix-grpc-daemon[1098]: ts=2026-09-21T09:14:49Z level=info event=scheduler_connected addr=lb:500515176worker2 # [ 100.286299] nix-grpc-daemon[1098]: ts=2026-09-21T09:14:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5177worker2 # [ 100.451965] nix-grpc-daemon[1098]: ts=2026-09-21T09:14:50Z level=info event=scheduler_restarting5178worker1 # [ 100.475184] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:50Z level=info event=scheduler_connected addr=lb:500515179worker1 # [ 100.479666] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5180worker2 # [ 100.502699] nix-daemon[870]: reaped child process 1150, status = succeeded5181worker2 # [ 100.505734] nix-grpc-daemon[1098]: ts=2026-09-21T09:14:50Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5182worker1 # [ 100.675862] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:50Z level=info event=scheduler_connected addr=lb:500515183worker1 # [ 100.681384] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5184worker1 # [ 100.881974] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:50Z level=info event=scheduler_connected addr=lb:500515185worker1 # [ 100.887196] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5186lb # [ 100.969385] envoy[913]: conn peer=[2001:db8:1::1]:55286 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=591 tx=519 ms=285187lb # [ 100.974280] envoy[913]: conn peer=[2001:db8:1::3]:58994 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=18805 tx=4953 ms=720485188lb # [ 100.979418] envoy[913]: conn peer=[2001:db8:1::1]:55302 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="TLS_error:|33554464:system_library:OPENSSL_internal:Broken_pipe:TLS_error_end" rx=1011 tx=304 ms=20415189lb # [ 100.987737] envoy[913]: conn peer=[2001:db8:1::4]:35538 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=4158 tx=349 ms=81905190worker1 # [ 101.088147] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:50Z level=info event=scheduler_connected addr=lb:500515191worker1 # [ 101.093344] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5192worker2 # [ 101.267956] nix-grpc-daemon[1098]: E0000 00:00:1789982090.850401 1143 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 16, started true, shutdown true5193worker2 # [ 101.268601] nix-grpc-daemon[1098]: E0000 00:00:1789982090.850674 1143 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5194worker1 # [ 101.294058] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:50Z level=info event=scheduler_connected addr=lb:500515195worker1 # [ 101.298858] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5196worker1 # [ 101.501593] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:51Z level=info event=scheduler_connected addr=lb:500515197worker1 # [ 101.505668] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5198worker1 # [ 101.702115] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:51Z level=info event=scheduler_connected addr=lb:500515199worker1 # [ 101.707396] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5200worker1 # [ 101.908126] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:51Z level=info event=scheduler_connected addr=lb:500515201worker1 # [ 101.913312] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5202worker1 # [ 102.115797] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:51Z level=info event=scheduler_connected addr=lb:500515203worker1 # [ 102.121346] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5204worker2 # [ 102.271305] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5205worker2 # [ 102.277726] systemd[1]: Stopped Nix worker-protocol over gRPC.5206worker2 # [ 102.283383] systemd[1]: nix-grpc-daemon.service: Consumed 582ms CPU time over 10.250s wall clock time, 34.2M memory peak, 19.2K incoming IP traffic, 25.4K outgoing IP traffic.5207worker2 # [ 102.299165] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5208worker2 # [ 102.301429] systemd[1]: Closed Nix worker-protocol over gRPC.5209worker2 # [ 102.304472] systemd[1]: nix-grpc-daemon.socket: Consumed 841us CPU time over 10.867s wall clock time, 256K memory peak, 21.4K incoming IP traffic, 16K outgoing IP traffic.5210worker1: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over5211worker1 # [ 102.322110] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:51Z level=info event=scheduler_connected addr=lb:500515212worker1 # [ 102.330108] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5213worker1 # [ 102.531746] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:52Z level=info event=scheduler_connected addr=lb:500515214worker1 # [ 102.535937] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5215worker1 # [ 102.675098] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:52Z level=info event=scheduler_take_over5216lb # [ 102.701308] niks3-server[1127]: 2026/09/21 09:14:52 INFO lead: acquired remote=192.168.1.3:591425217worker1 # [ 102.732628] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:52Z level=info event=scheduler_connected addr=lb:500515218worker1 # [ 102.737874] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5219worker1 # [ 102.938739] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:52Z level=info event=scheduler_connected addr=lb:500515220worker1 # [ 102.948194] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:52Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:603885221worker1 # [ 102.954286] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:52Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=05222worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over, in 1.19 seconds)5223worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5224worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)5225client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag on-standby >&25226worker1 # [ 103.818840] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:53Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:603965227worker1 # [ 103.833124] nix-daemon[843]: accepted connection from pid 1207, user nix-grpc-daemon (trusted)5228client # don't know how to build these paths:5229client # /nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv5230worker1 # [ 103.857096] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:53Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:603965231worker1 # [ 103.872756] nix-grpc-daemon[1207]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 65 ms (attempt 1/5)5232worker1 # [ 103.944994] nix-grpc-daemon[1207]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 137 ms (attempt 2/5)5233worker1 # [ 104.089634] nix-grpc-daemon[1207]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 52 ms (attempt 3/5)5234worker1 # [ 104.149143] nix-grpc-daemon[1207]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 773 ms (attempt 4/5)5235worker1 # [ 104.931566] nix-grpc-daemon[1207]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5236worker1 # [ 104.965185] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:54Z level=debug event=want client=1 drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv cached=05237worker1 # [ 104.971722] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:54Z level=debug event=assigned drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv worker=192.168.1.3:50051 clients=15238worker1 # [ 105.013214] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:603965239worker1 # [ 105.028358] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:54Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:603965240client # copying 1 paths to 'lb:50051'...5241client # copying path '/nix/store/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv' to 'lb:50051'...5242worker1 # [ 105.042387] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:54Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:603965243worker1 # [ 105.060210] nix-daemon[843]: accepted connection from pid 1207, user nix-grpc-daemon (trusted)5244worker1 # [ 105.088209] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:54Z level=info event=niks3_push_started pid=12835245worker1 # [ 105.310864] nix-grpc-daemon[1283]: time=2026-09-21T09:14:54.905Z level=INFO msg="Configuring client TLS" cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker1.pem key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker1.key ca=""5246worker1 # [ 105.486980] nix-daemon[843]: accepted connection from pid 1296, user nix-grpc-daemon (trusted)5247worker1 # [ 105.505006] nix-daemon[843]: reaped child process 1299, status = succeeded5248worker1 # [ 105.684105] nix-daemon[843]: accepted connection from pid 1301, user nix-grpc-daemon (trusted)5249worker1 # [ 105.699910] nix-daemon[843]: reaped child process 1304, status = succeeded5250lb # [ 105.740643] niks3-server[1127]: 2026/09/21 09:14:55 INFO Received uploads request method=POST path=/api/pending_closures5251worker1 # [ 105.728068] nix-grpc-daemon[1283]: time=2026-09-21T09:14:55.323Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5252worker1 # [ 105.729582] nix-grpc-daemon[1283]: time=2026-09-21T09:14:55.323Z level=INFO msg="Uploading adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv (520B)"5253lb # [ 105.787182] niks3-server[1127]: 2026/09/21 09:14:55 INFO Registered completed upload object_key=nar/0scd0jwin78ibwq92wkajw2zj7adk8bih7yhx8bxz1706hh53507.nar.zst5254worker1 # [ 105.772622] nix-grpc-daemon[1283]: time=2026-09-21T09:14:55.367Z level=INFO msg="Uploading 1 narinfos"5255lb # [ 105.797498] niks3-server[1127]: 2026/09/21 09:14:55 INFO Received sign narinfos request method=POST path=/api/pending_closures/27/sign5256lb # [ 105.810898] niks3-server[1127]: 2026/09/21 09:14:55 INFO Signed narinfos id=27 count=15257lb # [ 105.817886] niks3-server[1127]: 2026/09/21 09:14:55 INFO Registered completed upload object_key=adj3g5w8jh83wl959p7daq6igl0gaay3.ls5258lb # [ 105.833117] niks3-server[1127]: 2026/09/21 09:14:55 INFO Received complete upload request method=POST path=/api/pending_closures/27/complete5259lb # [ 105.838036] niks3-server[1127]: 2026/09/21 09:14:55 INFO Completed upload id=275260worker1 # [ 105.812512] nix-grpc-daemon[1283]: time=2026-09-21T09:14:55.408Z level=INFO msg="Upload complete. (497ms)"5261lb # [ 105.841404] niks3-server[1127]: 2026/09/21 09:14:55 INFO Registered completed upload object_key=adj3g5w8jh83wl959p7daq6igl0gaay3.narinfo5262worker1 # [ 105.823942] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:55Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60396 duration_s=0 paths=1 nar_bytes_in=5205263worker1 # [ 105.838166] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:603965264worker1 # [ 105.844638] nix-daemon[843]: accepted connection from pid 1207, user nix-grpc-daemon (trusted)5265worker1 # [ 105.851193] nix-daemon[843]: reaped child process 1281, status = succeeded5266client # building '/nix/store/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv' on lb:50051...5267client # farm-a> node-a: building adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv5268worker1 # [ 105.864212] nix-daemon[843]: accepted connection from pid 1207, user nix-grpc-daemon (trusted)5269client # farm-a> LOG-farm-a-on-standby5270worker1 # [ 107.509022] nix-daemon[843]: reaped child process 1308, status = succeeded5271worker1 # [ 107.701759] nix-daemon[843]: accepted connection from pid 1323, user nix-grpc-daemon (trusted)5272worker1 # [ 107.719400] nix-daemon[843]: reaped child process 1326, status = succeeded5273lb # [ 107.757331] niks3-server[1127]: 2026/09/21 09:14:57 INFO Received uploads request method=POST path=/api/pending_closures5274worker1 # [ 107.744347] nix-grpc-daemon[1283]: time=2026-09-21T09:14:57.336Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5275worker1 # [ 107.746076] nix-grpc-daemon[1283]: time=2026-09-21T09:14:57.336Z level=INFO msg="Uploading 7qyaw17jjsjxkkgb1kfm3wxvczk56aan-farm-a (136B)"5276lb # [ 107.790579] niks3-server[1127]: 2026/09/21 09:14:57 INFO Registered completed upload object_key=log/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv5277lb # [ 107.802592] niks3-server[1127]: 2026/09/21 09:14:57 INFO Registered completed upload object_key=nar/0x76wzgdd1nk8rm930z8l69v9961jpc8qhzmgganv0nhagq8lv9c.nar.zst5278lb # [ 107.820167] niks3-server[1127]: 2026/09/21 09:14:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/28/sign5279worker1 # [ 107.802997] nix-grpc-daemon[1283]: time=2026-09-21T09:14:57.397Z level=INFO msg="Uploading 1 narinfos"5280lb # [ 107.831831] niks3-server[1127]: 2026/09/21 09:14:57 INFO Signed narinfos id=28 count=15281lb # [ 107.839033] niks3-server[1127]: 2026/09/21 09:14:57 INFO Registered completed upload object_key=7qyaw17jjsjxkkgb1kfm3wxvczk56aan.ls5282lb # [ 107.846930] niks3-server[1127]: 2026/09/21 09:14:57 INFO Received complete upload request method=POST path=/api/pending_closures/28/complete5283lb # [ 107.855440] niks3-server[1127]: 2026/09/21 09:14:57 INFO Completed upload id=285284worker1 # [ 107.829558] nix-grpc-daemon[1283]: time=2026-09-21T09:14:57.425Z level=INFO msg="Upload complete. (351ms)"5285lb # [ 107.858735] niks3-server[1127]: 2026/09/21 09:14:57 INFO Registered completed upload object_key=7qyaw17jjsjxkkgb1kfm3wxvczk56aan.narinfo5286worker1 # [ 107.854030] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:57Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60396 duration_s=2 drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv assign_id=1 outputs=15287worker1 # [ 107.873589] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:57Z level=debug event=want client=1 drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv cached=05288worker1 # [ 107.878331] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:57Z level=debug event=assigned drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv worker=192.168.1.3:50051 clients=15289worker1 # [ 107.883217] nix-daemon[843]: reaped child process 1306, status = succeeded5290worker1 # [ 107.898983] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:603965291worker1 # [ 107.913714] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:57Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:603965292worker1 # [ 107.928770] nix-grpc-daemon[1283]: time=2026-09-21T09:14:57.524Z level=INFO msg="All 1 paths already cached"5293client # copying 1 paths to 'lb:50051'...5294worker1 # [ 107.937085] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:57Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:603965295client # copying path '/nix/store/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv' to 'lb:50051'...5296worker1 # [ 107.952143] nix-daemon[843]: accepted connection from pid 1207, user nix-grpc-daemon (trusted)5297worker1 # [ 108.157596] nix-daemon[843]: accepted connection from pid 1332, user nix-grpc-daemon (trusted)5298worker1 # [ 108.177268] nix-daemon[843]: reaped child process 1335, status = succeeded5299worker1 # [ 108.351525] nix-daemon[843]: accepted connection from pid 1337, user nix-grpc-daemon (trusted)5300worker1 # [ 108.371294] nix-daemon[843]: reaped child process 1340, status = succeeded5301lb # [ 108.407841] niks3-server[1127]: 2026/09/21 09:14:58 INFO Received uploads request method=POST path=/api/pending_closures5302worker1 # [ 108.395979] nix-grpc-daemon[1283]: time=2026-09-21T09:14:57.991Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5303worker1 # [ 108.397531] nix-grpc-daemon[1283]: time=2026-09-21T09:14:57.991Z level=INFO msg="Uploading s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv (640B)"5304lb # [ 108.441619] niks3-server[1127]: 2026/09/21 09:14:58 INFO Registered completed upload object_key=nar/0g5yl4v9pymang94vyp5ib5wi5jwf0bpp184y8qjavkpag12wmlv.nar.zst5305lb # [ 108.455663] niks3-server[1127]: 2026/09/21 09:14:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/29/sign5306worker1 # [ 108.438125] nix-grpc-daemon[1283]: time=2026-09-21T09:14:58.033Z level=INFO msg="Uploading 1 narinfos"5307lb # [ 108.469320] niks3-server[1127]: 2026/09/21 09:14:58 INFO Signed narinfos id=29 count=15308lb # [ 108.475777] niks3-server[1127]: 2026/09/21 09:14:58 INFO Registered completed upload object_key=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g.ls5309lb # [ 108.486159] niks3-server[1127]: 2026/09/21 09:14:58 INFO Received complete upload request method=POST path=/api/pending_closures/29/complete5310lb # [ 108.491405] niks3-server[1127]: 2026/09/21 09:14:58 INFO Completed upload id=295311worker1 # [ 108.465667] nix-grpc-daemon[1283]: time=2026-09-21T09:14:58.061Z level=INFO msg="Upload complete. (498ms)"5312lb # [ 108.494633] niks3-server[1127]: 2026/09/21 09:14:58 INFO Registered completed upload object_key=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g.narinfo5313worker1 # [ 108.472470] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:58Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60396 duration_s=0 paths=1 nar_bytes_in=6405314worker1 # [ 108.488288] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:58Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:603965315worker1 # [ 108.494387] nix-daemon[843]: accepted connection from pid 1207, user nix-grpc-daemon (trusted)5316worker1 # [ 108.499406] nix-daemon[843]: reaped child process 1330, status = succeeded5317client # building '/nix/store/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv' on lb:50051...5318client # farm-b> node-a: building s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv5319worker1 # [ 108.515481] nix-daemon[843]: accepted connection from pid 1207, user nix-grpc-daemon (trusted)5320client # farm-b> LOG-farm-b-on-standby5321worker1 # [ 109.351016] nix-daemon[843]: reaped child process 1344, status = succeeded5322worker1 # [ 109.524295] nix-daemon[843]: accepted connection from pid 1359, user nix-grpc-daemon (trusted)5323worker1 # [ 109.544562] nix-daemon[843]: reaped child process 1362, status = succeeded5324lb # [ 109.581605] niks3-server[1127]: 2026/09/21 09:14:59 INFO Received uploads request method=POST path=/api/pending_closures5325worker1 # [ 109.566712] nix-grpc-daemon[1283]: time=2026-09-21T09:14:59.162Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5326worker1 # [ 109.569847] nix-grpc-daemon[1283]: time=2026-09-21T09:14:59.164Z level=INFO msg="Uploading 6hsxxv65wr2jz8cgv2v66iq16niq1gsv-farm-b (184B)"5327lb # [ 109.621671] niks3-server[1127]: 2026/09/21 09:14:59 INFO Registered completed upload object_key=nar/1grn64wcl2mc78k1jvrz5mcrk1ld70lhf8b9jpiqy8mvs6gfzq0v.nar.zst5328lb # [ 109.632930] niks3-server[1127]: 2026/09/21 09:14:59 INFO Registered completed upload object_key=log/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv5329worker1 # [ 109.619521] nix-grpc-daemon[1283]: time=2026-09-21T09:14:59.214Z level=INFO msg="Uploading 1 narinfos"5330lb # [ 109.644570] niks3-server[1127]: 2026/09/21 09:14:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/30/sign5331lb # [ 109.650482] niks3-server[1127]: 2026/09/21 09:14:59 INFO Signed narinfos id=30 count=15332lb # [ 109.655758] niks3-server[1127]: 2026/09/21 09:14:59 INFO Registered completed upload object_key=6hsxxv65wr2jz8cgv2v66iq16niq1gsv.ls5333lb # [ 109.668185] niks3-server[1127]: 2026/09/21 09:14:59 INFO Received complete upload request method=POST path=/api/pending_closures/30/complete5334lb # [ 109.673854] niks3-server[1127]: 2026/09/21 09:14:59 INFO Completed upload id=305335worker1 # [ 109.648773] nix-grpc-daemon[1283]: time=2026-09-21T09:14:59.244Z level=INFO msg="Upload complete. (323ms)"5336lb # [ 109.677604] niks3-server[1127]: 2026/09/21 09:14:59 INFO Registered completed upload object_key=6hsxxv65wr2jz8cgv2v66iq16niq1gsv.narinfo5337worker1 # [ 109.668966] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:59Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60396 duration_s=1 drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv assign_id=2 outputs=15338worker1 # [ 109.684650] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:59Z level=debug event=want client=1 drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv cached=05339worker1 # [ 109.689745] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:59Z level=debug event=assigned drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv worker=192.168.1.3:50051 clients=15340worker1 # [ 109.693700] nix-daemon[843]: reaped child process 1342, status = succeeded5341worker1 # [ 109.710954] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:603965342worker1 # [ 109.727152] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:59Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:603965343client # copying 1 paths to 'lb:50051'...5344worker1 # [ 109.741854] nix-grpc-daemon[1283]: time=2026-09-21T09:14:59.337Z level=INFO msg="All 2 paths already cached"5345client # copying path '/nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv' to 'lb:50051'...5346worker1 # [ 109.748359] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:59Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:603965347worker1 # [ 109.761778] nix-daemon[843]: accepted connection from pid 1207, user nix-grpc-daemon (trusted)5348worker1 # [ 109.959604] nix-daemon[843]: accepted connection from pid 1368, user nix-grpc-daemon (trusted)5349worker1 # [ 109.977145] nix-daemon[843]: reaped child process 1371, status = succeeded5350worker1 # [ 110.154514] nix-daemon[843]: accepted connection from pid 1373, user nix-grpc-daemon (trusted)5351worker1 # [ 110.171359] nix-daemon[843]: reaped child process 1376, status = succeeded5352lb # [ 110.212100] niks3-server[1127]: 2026/09/21 09:14:59 INFO Received uploads request method=POST path=/api/pending_closures5353worker1 # [ 110.196097] nix-grpc-daemon[1283]: time=2026-09-21T09:14:59.791Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5354worker1 # [ 110.197582] nix-grpc-daemon[1283]: time=2026-09-21T09:14:59.791Z level=INFO msg="Uploading 7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv (768B)"5355lb # [ 110.243393] niks3-server[1127]: 2026/09/21 09:14:59 INFO Registered completed upload object_key=nar/1zyx46mv29cfygdhc5ai40qr4vl88lkr1lilm0h72j23w6wakgmc.nar.zst5356worker1 # [ 110.231296] nix-grpc-daemon[1283]: time=2026-09-21T09:14:59.826Z level=INFO msg="Uploading 1 narinfos"5357lb # [ 110.260273] niks3-server[1127]: 2026/09/21 09:14:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/31/sign5358lb # [ 110.271127] niks3-server[1127]: 2026/09/21 09:14:59 INFO Signed narinfos id=31 count=15359lb # [ 110.278633] niks3-server[1127]: 2026/09/21 09:14:59 INFO Registered completed upload object_key=7s5szig4ffga53df6rls98r9lhxsrsc5.ls5360lb # [ 110.289167] niks3-server[1127]: 2026/09/21 09:14:59 INFO Received complete upload request method=POST path=/api/pending_closures/31/complete5361worker1 # [ 110.269310] nix-grpc-daemon[1283]: time=2026-09-21T09:14:59.865Z level=INFO msg="Upload complete. (492ms)"5362lb # [ 110.294826] niks3-server[1127]: 2026/09/21 09:14:59 INFO Completed upload id=315363lb # [ 110.298070] niks3-server[1127]: 2026/09/21 09:14:59 INFO Registered completed upload object_key=7s5szig4ffga53df6rls98r9lhxsrsc5.narinfo5364worker1 # [ 110.276273] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:59Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60396 duration_s=0 paths=1 nar_bytes_in=7685365worker1 # [ 110.290272] nix-grpc-daemon[1207]: ts=2026-09-21T09:14:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:603965366worker1 # [ 110.300177] nix-daemon[843]: accepted connection from pid 1207, user nix-grpc-daemon (trusted)5367worker1 # [ 110.303125] nix-daemon[843]: reaped child process 1366, status = succeeded5368client # building '/nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv' on lb:50051...5369worker1 # [ 110.316643] nix-daemon[843]: accepted connection from pid 1207, user nix-grpc-daemon (trusted)5370client # farm-top> node-a: building 7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv5371client # farm-top> LOG-farm-top-on-standby5372worker1 # [ 111.542506] nix-daemon[843]: reaped child process 1380, status = succeeded5373worker1 # [ 111.709107] nix-daemon[843]: accepted connection from pid 1395, user nix-grpc-daemon (trusted)5374worker1 # [ 111.729257] nix-daemon[843]: reaped child process 1398, status = succeeded5375lb # [ 111.769717] niks3-server[1127]: 2026/09/21 09:15:01 INFO Received uploads request method=POST path=/api/pending_closures5376worker1 # [ 111.756435] nix-grpc-daemon[1283]: time=2026-09-21T09:15:01.351Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5377worker1 # [ 111.757978] nix-grpc-daemon[1283]: time=2026-09-21T09:15:01.351Z level=INFO msg="Uploading 1ini8mfd1rzqyp8jgqihp4k889588mpx-farm-top (240B)"5378lb # [ 111.816173] niks3-server[1127]: 2026/09/21 09:15:01 INFO Registered completed upload object_key=nar/1mncazpqn39nifc2fwbx80la5i264r8xc1lj8fnlbx006znc1925.nar.zst5379lb # [ 111.825227] niks3-server[1127]: 2026/09/21 09:15:01 INFO Registered completed upload object_key=log/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv5380worker1 # [ 111.809175] nix-grpc-daemon[1283]: time=2026-09-21T09:15:01.404Z level=INFO msg="Uploading 1 narinfos"5381lb # [ 111.837829] niks3-server[1127]: 2026/09/21 09:15:01 INFO Received sign narinfos request method=POST path=/api/pending_closures/32/sign5382lb # [ 111.846058] niks3-server[1127]: 2026/09/21 09:15:01 INFO Signed narinfos id=32 count=15383lb # [ 111.847966] niks3-server[1127]: 2026/09/21 09:15:01 INFO Registered completed upload object_key=1ini8mfd1rzqyp8jgqihp4k889588mpx.ls5384worker1 # [ 111.829947] nix-grpc-daemon[1283]: time=2026-09-21T09:15:01.425Z level=INFO msg="Upload complete. (300ms)"5385lb # [ 111.855949] niks3-server[1127]: 2026/09/21 09:15:01 INFO Received complete upload request method=POST path=/api/pending_closures/32/complete5386lb # [ 111.860161] niks3-server[1127]: 2026/09/21 09:15:01 INFO Completed upload id=325387lb # [ 111.862254] niks3-server[1127]: 2026/09/21 09:15:01 INFO Registered completed upload object_key=1ini8mfd1rzqyp8jgqihp4k889588mpx.narinfo5388worker1 # [ 111.841206] nix-grpc-daemon[1207]: ts=2026-09-21T09:15:01Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60396 duration_s=1 drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv assign_id=3 outputs=15389worker1 # [ 111.854618] nix-daemon[843]: reaped child process 1378, status = succeeded5390client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag on-standby >&2, in 8.35 seconds)5391client: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag on-standby outPath5392client: (finished: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag on-standby outPath, in 0.23 seconds)5393worker2 # [ 112.220931] systemd[1]: Listening on Nix worker-protocol over gRPC.5394worker2 # [ 112.226929] systemd[1]: Starting scheduler-token.service...5395worker2 # [ 112.318113] systemd[1]: scheduler-token.service: Deactivated successfully.5396worker2 # [ 112.324112] systemd[1]: Finished scheduler-token.service.5397worker2 # [ 112.327137] systemd[1]: scheduler-token.service: Consumed 37ms CPU time over 94ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5398worker2 # [ 112.330888] systemd[1]: Starting Nix worker-protocol over gRPC...5399worker2 # [ 112.505898] nix-grpc-daemon[1197]: ts=2026-09-21T09:15:02Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5400worker2 # [ 112.508490] nix-grpc-daemon[1197]: ts=2026-09-21T09:15:02Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5401worker2 # [ 112.512591] nix-grpc-daemon[1197]: ts=2026-09-21T09:15:02Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15402worker2 # [ 112.520065] nix-grpc-daemon[1197]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5403worker2 # [ 112.521535] nix-grpc-daemon[1197]: E0000 00:00:1789982102.101626 1197 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5404worker2 # [ 112.537056] nix-grpc-daemon[1197]: E0000 00:00:1789982102.108048 1197 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5405lb: must succeed: curl -sf localhost:9901/clusters5406worker2 # [ 112.542675] nix-grpc-daemon[1197]: ts=2026-09-21T09:15:02Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5407worker2 # [ 112.568282] systemd[1]: Started Nix worker-protocol over gRPC.5408worker2 # [ 112.569117] nix-grpc-daemon[1197]: E0000 00:00:1789982102.129173 1239 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5409worker2 # [ 112.571407] nix-grpc-daemon[1197]: E0000 00:00:1789982102.145808 1239 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5410worker2 # [ 112.584267] nix-grpc-daemon[1197]: E0000 00:00:1789982102.145953 1239 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5411worker2 # [ 112.587051] nix-grpc-daemon[1197]: E0000 00:00:1789982102.166258 1239 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5412worker2 # [ 112.600183] nix-grpc-daemon[1197]: ts=2026-09-21T09:15:02Z level=info event=scheduler_connected addr=lb:500515413worker1 # [ 112.603807] nix-grpc-daemon[1207]: ts=2026-09-21T09:15:02Z level=info event=oidc_jwks provider=mock keys=15414worker1 # [ 112.608402] nix-grpc-daemon[1207]: ts=2026-09-21T09:15:02Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:603885415worker1 # [ 112.613727] nix-grpc-daemon[1207]: ts=2026-09-21T09:15:02Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=05416lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5417worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5418worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)5419lb: must succeed: curl -sf localhost:9901/clusters5420lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5421lb: must succeed: curl -sf localhost:9901/clusters5422lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5423worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5424worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)5425(finished: subtest: scheduler down: the other node takes the lock and keeps it, in 14.04 seconds)5426subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise5427client: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag rs5428client # [ 112.983487] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key" --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag rs.5429client: (finished: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag rs, in 0.10 seconds)5430client # Running as unit: rs.service; invocation ID: d11348792e2342d0a3821875fa9107e45431client # [ 113.252456] nix[1166]: don't know how to build these paths:5432client # [ 113.254331] nix[1166]: /nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv5433worker1 # [ 113.254051] nix-grpc-daemon[1207]: ts=2026-09-21T09:15:02Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:603965434worker1 # [ 113.267187] nix-grpc-daemon[1207]: ts=2026-09-21T09:15:02Z level=debug event=want client=2 drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv cached=05435worker1 # [ 113.274534] nix-grpc-daemon[1207]: ts=2026-09-21T09:15:02Z level=debug event=assigned drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv worker=192.168.1.3:50051 clients=15436worker1 # [ 113.314700] nix-grpc-daemon[1207]: ts=2026-09-21T09:15:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:603965437worker1 # [ 113.330569] nix-grpc-daemon[1207]: ts=2026-09-21T09:15:02Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:603965438client # [ 113.340949] nix[1166]: copying 1 paths to 'lb:50051'...5439client # [ 113.345080] nix[1166]: copying path '/nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv' to 'lb:50051'...5440worker1 # [ 113.348114] nix-grpc-daemon[1207]: ts=2026-09-21T09:15:02Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:603965441worker1 # [ 113.363140] nix-daemon[843]: accepted connection from pid 1207, user nix-grpc-daemon (trusted)5442worker1 # [ 113.574614] nix-daemon[843]: accepted connection from pid 1411, user nix-grpc-daemon (trusted)5443worker1 # [ 113.594094] nix-daemon[843]: reaped child process 1414, status = succeeded5444worker1 # [ 113.769388] nix-daemon[843]: accepted connection from pid 1416, user nix-grpc-daemon (trusted)5445worker1 # [ 113.786552] nix-daemon[843]: reaped child process 1419, status = succeeded5446lb # [ 113.827145] niks3-server[1127]: 2026/09/21 09:15:03 INFO Received uploads request method=POST path=/api/pending_closures5447worker1 # [ 113.811689] nix-grpc-daemon[1283]: time=2026-09-21T09:15:03.406Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5448worker1 # [ 113.813424] nix-grpc-daemon[1283]: time=2026-09-21T09:15:03.406Z level=INFO msg="Uploading ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv (416B)"5449lb # [ 113.853129] niks3-server[1127]: 2026/09/21 09:15:03 INFO Registered completed upload object_key=nar/0lynldv3n40qnhi1516wgiry3i3pja3sanwyyd8q4nslr915d7r1.nar.zst5450lb # [ 113.868764] niks3-server[1127]: 2026/09/21 09:15:03 INFO Received sign narinfos request method=POST path=/api/pending_closures/33/sign5451worker1 # [ 113.848886] nix-grpc-daemon[1283]: time=2026-09-21T09:15:03.444Z level=INFO msg="Uploading 1 narinfos"5452lb # [ 113.880851] niks3-server[1127]: 2026/09/21 09:15:03 INFO Signed narinfos id=33 count=15453lb # [ 113.888163] niks3-server[1127]: 2026/09/21 09:15:03 INFO Registered completed upload object_key=ajji8m1zd5q5wi64gp7vclb5dd8hr19m.ls5454lb # [ 113.898059] niks3-server[1127]: 2026/09/21 09:15:03 INFO Received complete upload request method=POST path=/api/pending_closures/33/complete5455lb # [ 113.904341] niks3-server[1127]: 2026/09/21 09:15:03 INFO Completed upload id=335456worker1 # [ 113.877815] nix-grpc-daemon[1283]: time=2026-09-21T09:15:03.473Z level=INFO msg="Upload complete. (497ms)"5457lb # [ 113.908517] niks3-server[1127]: 2026/09/21 09:15:03 INFO Registered completed upload object_key=ajji8m1zd5q5wi64gp7vclb5dd8hr19m.narinfo5458worker1 # [ 113.888510] nix-grpc-daemon[1207]: ts=2026-09-21T09:15:03Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60396 duration_s=0 paths=1 nar_bytes_in=4165459worker1 # [ 113.902291] nix-grpc-daemon[1207]: ts=2026-09-21T09:15:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:603965460worker1 # [ 113.908932] nix-daemon[843]: accepted connection from pid 1207, user nix-grpc-daemon (trusted)5461worker1 # [ 113.914145] nix-daemon[843]: reaped child process 1409, status = succeeded5462client # [ 113.922306] nix[1166]: building '/nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv' on lb:50051...5463worker1 # [ 113.928205] nix-daemon[843]: accepted connection from pid 1207, user nix-grpc-daemon (trusted)5464lb: must succeed: curl -sf localhost:9901/clusters5465lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5466lb: must succeed: curl -sf localhost:9901/clusters5467lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)5468lb: must succeed: curl -sf localhost:9901/clusters5469lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5470lb: must succeed: curl -sf localhost:9901/clusters5471lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5472worker2: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true5473worker2 # [ 115.934239] systemd-journald[269]: Received client request to rotate journal, rotating.5474worker2 # [ 115.958218] systemd-journald[269]: Vacuuming done, freed 0B of archived journals from /var/log/journal/ad6ef90e50024f498dcd3264791bf1a9.5475worker2: (finished: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true, in 0.09 seconds)5476worker1: must succeed: systemctl restart --no-block nix-grpc-daemon.service5477worker1 # [ 116.002703] systemd[1]: Stopping Nix worker-protocol over gRPC...5478worker1: (finished: must succeed: systemctl restart --no-block nix-grpc-daemon.service, in 0.06 seconds)5479client: must succeed: sleep 5.05480worker1 # [ 116.685219] nix-grpc-daemon[1207]: ts=2026-09-21T09:15:06Z level=info event=draining5481lb # [ 120.970791] envoy[913]: conn peer=[2001:db8:1::1]:48698 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9623 tx=3861 ms=80445482client: (finished: must succeed: sleep 5.0, in 5.06 seconds)5483worker1 # [ 121.355999] nix-daemon[843]: reaped child process 1423, status = succeeded5484worker2: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr='5485worker1 # [ 121.515158] nix-daemon[843]: accepted connection from pid 1463, user nix-grpc-daemon (trusted)5486worker1 # [ 121.533593] nix-daemon[843]: reaped child process 1466, status = succeeded5487lb # [ 121.572812] niks3-server[1127]: 2026/09/21 09:15:11 INFO Received uploads request method=POST path=/api/pending_closures5488worker1 # [ 121.559229] nix-grpc-daemon[1283]: time=2026-09-21T09:15:11.152Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5489worker1 # [ 121.560841] nix-grpc-daemon[1283]: time=2026-09-21T09:15:11.152Z level=INFO msg="Uploading k48d52qj0czpjmzj15yq3fd14zfw5iq6-slow-rs (120B)"5490lb # [ 121.617383] niks3-server[1127]: 2026/09/21 09:15:11 INFO Registered completed upload object_key=log/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv5491worker1 # [ 121.601218] nix-grpc-daemon[1283]: time=2026-09-21T09:15:11.196Z level=INFO msg="Uploading 1 narinfos"5492lb # [ 121.625752] niks3-server[1127]: 2026/09/21 09:15:11 INFO Received sign narinfos request method=POST path=/api/pending_closures/34/sign5493lb # [ 121.638648] niks3-server[1127]: 2026/09/21 09:15:11 INFO Signed narinfos id=34 count=15494lb # [ 121.645062] niks3-server[1127]: 2026/09/21 09:15:11 INFO Registered completed upload object_key=nar/19z3qdigvc4gggg31dgc0wi5w85pgaq9dk7bvngh0yhckpy55jgq.nar.zst5495lb # [ 121.654682] niks3-server[1127]: 2026/09/21 09:15:11 INFO Registered completed upload object_key=k48d52qj0czpjmzj15yq3fd14zfw5iq6.ls5496worker1 # [ 121.633787] nix-grpc-daemon[1283]: time=2026-09-21T09:15:11.229Z level=INFO msg="Upload complete. (291ms)"5497lb # [ 121.659971] niks3-server[1127]: 2026/09/21 09:15:11 INFO Received complete upload request method=POST path=/api/pending_closures/34/complete5498lb # [ 121.666515] niks3-server[1127]: 2026/09/21 09:15:11 INFO Completed upload id=345499lb # [ 121.669428] niks3-server[1127]: 2026/09/21 09:15:11 INFO Registered completed upload object_key=k48d52qj0czpjmzj15yq3fd14zfw5iq6.narinfo5500worker1 # [ 121.645266] nix-grpc-daemon[1207]: ts=2026-09-21T09:15:11Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60396 duration_s=7 drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv assign_id=4 outputs=15501worker1 # [ 121.658817] nix-daemon[843]: reaped child process 1421, status = succeeded5502worker1 # [ 121.685375] nix-grpc-daemon[1207]: ts=2026-09-21T09:15:11Z level=info event=signal_exit5503client # [ 121.687219] systemd[1]: rs.service: Deactivated successfully.5504client # [ 121.691577] systemd[1]: rs.service: Consumed 178ms CPU time over 8.707s wall clock time, 41.2M memory peak, 6.8K incoming IP traffic, 8.1K outgoing IP traffic.5505worker1 # [ 121.896362] nix-grpc-daemon[1207]: ts=2026-09-21T09:15:11Z level=info event=scheduler_yield5506worker1 # [ 121.902122] nix-grpc-daemon[1207]: ts=2026-09-21T09:15:11Z level=info event=scheduler_restarting addr=lb:500515507lb # [ 121.921605] niks3-server[1127]: 2026/09/21 09:15:11 INFO lead: released remote=192.168.1.3:591425508worker2 # [ 121.910764] nix-grpc-daemon[1197]: ts=2026-09-21T09:15:11Z level=info event=scheduler_restarting addr=lb:500515509worker1 # [ 121.908566] nix-grpc-daemon[1207]: ts=2026-09-21T09:15:11Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:603885510worker1 # [ 121.913092] nix-grpc-daemon[1207]: ts=2026-09-21T09:15:11Z level=info event=worker_gone addr=192.168.1.3:500515511worker1 # [ 121.917125] nix-grpc-daemon[1207]: ts=2026-09-21T09:15:11Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:603885512worker1 # [ 121.921535] nix-grpc-daemon[1207]: ts=2026-09-21T09:15:11Z level=info event=worker_gone addr=192.168.1.4:500515513worker1 # [ 122.103135] nix-grpc-daemon[1207]: ts=2026-09-21T09:15:11Z level=info event=scheduler_connected addr=lb:500515514worker2 # [ 122.110745] nix-grpc-daemon[1197]: ts=2026-09-21T09:15:11Z level=info event=scheduler_connected addr=lb:500515515worker1 # [ 122.108593] nix-grpc-daemon[1207]: ts=2026-09-21T09:15:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5516worker2 # [ 122.116222] nix-grpc-daemon[1197]: ts=2026-09-21T09:15:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5517worker1 # [ 122.309393] nix-grpc-daemon[1207]: ts=2026-09-21T09:15:11Z level=info event=scheduler_connected addr=lb:500515518worker2 # [ 122.317209] nix-grpc-daemon[1197]: ts=2026-09-21T09:15:11Z level=info event=scheduler_connected addr=lb:500515519worker1 # [ 122.314627] nix-grpc-daemon[1207]: ts=2026-09-21T09:15:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5520worker2 # [ 122.322438] nix-grpc-daemon[1197]: ts=2026-09-21T09:15:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5521worker2: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr=', in 1.12 seconds)5522client: waiting for success: ! systemctl is-active rs5523worker1 # [ 122.515475] nix-grpc-daemon[1207]: ts=2026-09-21T09:15:12Z level=info event=scheduler_connected addr=lb:500515524worker2 # [ 122.523493] nix-grpc-daemon[1197]: ts=2026-09-21T09:15:12Z level=info event=scheduler_connected addr=lb:500515525worker1 # [ 122.520964] nix-grpc-daemon[1207]: ts=2026-09-21T09:15:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5526worker2 # [ 122.528552] nix-grpc-daemon[1197]: ts=2026-09-21T09:15:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5527lb # [ 122.557176] niks3-server[1127]: 2026/09/21 09:15:12 INFO lead: acquired remote=192.168.1.4:425085528worker2 # [ 122.539774] nix-grpc-daemon[1197]: ts=2026-09-21T09:15:12Z level=info event=scheduler_take_over5529client: (finished: waiting for success: ! systemctl is-active rs, in 0.07 seconds)5530client: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }5531client: (finished: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }, in 0.05 seconds)5532client: must fail: journalctl -u rs | grep -q 'reconnecting'5533client: (finished: must fail: journalctl -u rs | grep -q 'reconnecting', in 0.03 seconds)5534lb: must succeed: curl -sf localhost:9901/clusters5535worker1 # [ 122.723512] nix-grpc-daemon[1207]: ts=2026-09-21T09:15:12Z level=info event=scheduler_connected addr=lb:500515536worker2 # [ 122.730542] nix-grpc-daemon[1197]: ts=2026-09-21T09:15:12Z level=info event=scheduler_connected addr=lb:500515537worker1 # [ 122.727790] nix-grpc-daemon[1207]: ts=2026-09-21T09:15:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5538worker2 # [ 122.734796] nix-grpc-daemon[1197]: ts=2026-09-21T09:15:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5539lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5540worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5541worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5542worker1 # [ 122.902084] nix-grpc-daemon[1207]: ts=2026-09-21T09:15:12Z level=info event=scheduler_restarting5543worker2 # [ 122.933901] nix-grpc-daemon[1197]: ts=2026-09-21T09:15:12Z level=info event=scheduler_connected addr=lb:500515544worker2 # [ 122.938129] nix-grpc-daemon[1197]: ts=2026-09-21T09:15:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5545worker1 # [ 122.946012] nix-grpc-daemon[1207]: ts=2026-09-21T09:15:12Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5546worker1 # [ 122.962458] nix-daemon[843]: reaped child process 1274, status = succeeded5547worker2 # [ 123.135152] nix-grpc-daemon[1197]: ts=2026-09-21T09:15:12Z level=info event=scheduler_connected addr=lb:500515548worker2 # [ 123.140438] nix-grpc-daemon[1197]: ts=2026-09-21T09:15:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5549worker2 # [ 123.341671] nix-grpc-daemon[1197]: ts=2026-09-21T09:15:12Z level=info event=scheduler_connected addr=lb:500515550worker2 # [ 123.346713] nix-grpc-daemon[1197]: ts=2026-09-21T09:15:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5551worker2 # [ 123.547904] nix-grpc-daemon[1197]: ts=2026-09-21T09:15:13Z level=info event=scheduler_connected addr=lb:500515552worker2 # [ 123.553015] nix-grpc-daemon[1197]: ts=2026-09-21T09:15:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5553worker2 # [ 123.753977] nix-grpc-daemon[1197]: ts=2026-09-21T09:15:13Z level=info event=scheduler_connected addr=lb:500515554worker2 # [ 123.769997] nix-grpc-daemon[1197]: ts=2026-09-21T09:15:13Z level=info event=oidc_jwks provider=mock keys=15555worker2 # [ 123.774404] nix-grpc-daemon[1197]: ts=2026-09-21T09:15:13Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:571585556worker2 # [ 123.779604] nix-grpc-daemon[1197]: ts=2026-09-21T09:15:13Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=05557lb: must succeed: curl -sf localhost:9901/clusters5558lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5559worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5560worker1 # [ 123.894452] nix-grpc-daemon[1207]: E0000 00:00:1789982113.488746 1252 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 16, started true, shutdown true5561worker1 # [ 123.900960] nix-grpc-daemon[1207]: E0000 00:00:1789982113.488977 1252 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5562worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5563worker1 # [ 124.762402] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5564worker1 # [ 124.765617] systemd[1]: Stopped Nix worker-protocol over gRPC.5565worker1 # [ 124.771670] systemd[1]: nix-grpc-daemon.service: Consumed 1.827s CPU time over 27.318s wall clock time, 36.8M memory peak, 48.4K incoming IP traffic, 89.8K outgoing IP traffic.5566worker1 # [ 124.789484] systemd[1]: Starting Nix worker-protocol over gRPC...5567lb: must succeed: curl -sf localhost:9901/clusters5568worker1 # [ 124.984544] nix-grpc-daemon[1473]: ts=2026-09-21T09:15:14Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5569worker1 # [ 124.987358] nix-grpc-daemon[1473]: ts=2026-09-21T09:15:14Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5570worker1 # [ 124.991561] nix-grpc-daemon[1473]: ts=2026-09-21T09:15:14Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15571worker1 # [ 125.000087] nix-grpc-daemon[1473]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5572worker1 # [ 125.001571] nix-grpc-daemon[1473]: E0000 00:00:1789982114.593754 1473 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5573worker1 # [ 125.007819] nix-grpc-daemon[1473]: E0000 00:00:1789982114.595811 1473 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5574worker1 # [ 125.012286] nix-grpc-daemon[1473]: ts=2026-09-21T09:15:14Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5575worker1 # [ 125.014384] systemd[1]: Started Nix worker-protocol over gRPC.5576worker1 # [ 125.019368] nix-grpc-daemon[1473]: E0000 00:00:1789982114.613697 1512 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5577lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5578worker1 # [ 125.048164] nix-grpc-daemon[1473]: E0000 00:00:1789982114.631174 1512 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5579worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5580worker1 # [ 125.050431] nix-grpc-daemon[1473]: E0000 00:00:1789982114.631351 1512 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5581worker1 # [ 125.056999] nix-grpc-daemon[1473]: E0000 00:00:1789982114.651089 1512 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5582worker1 # [ 125.064891] nix-grpc-daemon[1473]: ts=2026-09-21T09:15:14Z level=info event=scheduler_connected addr=lb:500515583worker2 # [ 125.075471] nix-grpc-daemon[1197]: ts=2026-09-21T09:15:14Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:571585584worker2 # [ 125.075753] nix-grpc-daemon[1197]: ts=2026-09-21T09:15:14Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=05585worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.09 seconds)5586(finished: subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise, in 12.22 seconds)5587subtest: scheduler restart mid-build: build finishes, no double build5588client: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag mid5589client # [ 125.207759] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key" --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag mid.5590client # Running as unit: mid.service; invocation ID: 5a854f21515d4d17bfe706eb67e333a15591client: (finished: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag mid, in 0.10 seconds)5592worker2 # [ 125.479995] nix-grpc-daemon[1197]: ts=2026-09-21T09:15:15Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:571665593worker2 # [ 125.494776] nix-daemon[870]: accepted connection from pid 1197, user nix-grpc-daemon (trusted)5594client # [ 125.503303] nix[1202]: don't know how to build these paths:5595client # [ 125.504611] nix[1202]: /nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv5596worker2 # [ 125.524368] nix-grpc-daemon[1197]: ts=2026-09-21T09:15:15Z level=debug event=want client=1 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv cached=05597worker2 # [ 125.531068] nix-grpc-daemon[1197]: ts=2026-09-21T09:15:15Z level=debug event=assigned drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv worker=192.168.1.4:50051 clients=15598worker2 # [ 125.578288] nix-grpc-daemon[1197]: ts=2026-09-21T09:15:15Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:571665599worker2 # [ 125.599088] nix-grpc-daemon[1197]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 82 ms (attempt 1/5)5600worker2 # [ 125.688767] nix-grpc-daemon[1197]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 151 ms (attempt 2/5)5601worker2 # [ 125.847479] nix-grpc-daemon[1197]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 188 ms (attempt 3/5)5602worker2 # [ 126.043175] nix-grpc-daemon[1197]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 329 ms (attempt 4/5)5603worker2 # [ 126.374192] nix-grpc-daemon[1197]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5604worker2 # [ 126.395260] nix-grpc-daemon[1197]: ts=2026-09-21T09:15:15Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:571665605client # [ 126.400235] nix[1202]: copying 1 paths to 'lb:50051'...5606worker2 # [ 126.410753] nix-grpc-daemon[1197]: ts=2026-09-21T09:15:15Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:571665607client # [ 126.404158] nix[1202]: copying path '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' to 'lb:50051'...5608worker2 # [ 126.427627] nix-daemon[870]: accepted connection from pid 1197, user nix-grpc-daemon (trusted)5609worker2 # [ 126.456970] nix-grpc-daemon[1197]: ts=2026-09-21T09:15:16Z level=info event=niks3_push_started pid=13155610worker2 # [ 126.691404] nix-grpc-daemon[1315]: time=2026-09-21T09:15:16.273Z level=INFO msg="Configuring client TLS" cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker2.pem key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker2.key ca=""5611worker2 # [ 126.866012] nix-daemon[870]: accepted connection from pid 1332, user nix-grpc-daemon (trusted)5612worker2 # [ 126.883136] nix-daemon[870]: reaped child process 1335, status = succeeded5613worker2 # [ 127.058853] nix-daemon[870]: accepted connection from pid 1337, user nix-grpc-daemon (trusted)5614worker2 # [ 127.074841] nix-daemon[870]: reaped child process 1340, status = succeeded5615lb # [ 127.105981] niks3-server[1127]: 2026/09/21 09:15:16 INFO Received uploads request method=POST path=/api/pending_closures5616worker2 # [ 127.097607] nix-grpc-daemon[1315]: time=2026-09-21T09:15:16.679Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5617worker2 # [ 127.099053] nix-grpc-daemon[1315]: time=2026-09-21T09:15:16.679Z level=INFO msg="Uploading chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv (424B)"5618lb # [ 127.140311] niks3-server[1127]: 2026/09/21 09:15:16 INFO Registered completed upload object_key=nar/14lxjrdcxwbhc1kgg39y29gnbx5cz80j4852d1i4w3rvq7g4b04g.nar.zst5619lb # [ 127.156957] niks3-server[1127]: 2026/09/21 09:15:16 INFO Received sign narinfos request method=POST path=/api/pending_closures/35/sign5620worker2 # [ 127.145992] nix-grpc-daemon[1315]: time=2026-09-21T09:15:16.728Z level=INFO msg="Uploading 1 narinfos"5621lb # [ 127.172340] niks3-server[1127]: 2026/09/21 09:15:16 INFO Signed narinfos id=35 count=15622lb # [ 127.175592] niks3-server[1127]: 2026/09/21 09:15:16 INFO Registered completed upload object_key=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc.ls5623lb # [ 127.187331] niks3-server[1127]: 2026/09/21 09:15:16 INFO Received complete upload request method=POST path=/api/pending_closures/35/complete5624lb # [ 127.199696] niks3-server[1127]: 2026/09/21 09:15:16 INFO Registered completed upload object_key=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc.narinfo5625lb # [ 127.223584] niks3-server[1127]: 2026/09/21 09:15:16 INFO Completed upload id=355626worker2 # [ 127.205462] nix-grpc-daemon[1315]: time=2026-09-21T09:15:16.788Z level=INFO msg="Upload complete. (510ms)"5627worker2 # [ 127.220294] nix-grpc-daemon[1197]: ts=2026-09-21T09:15:16Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:57166 duration_s=0 paths=1 nar_bytes_in=4245628worker2 # [ 127.232580] nix-grpc-daemon[1197]: ts=2026-09-21T09:15:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:571665629worker2 # [ 127.243309] nix-daemon[870]: accepted connection from pid 1197, user nix-grpc-daemon (trusted)5630worker2 # [ 127.251037] nix-daemon[870]: reaped child process 1313, status = succeeded5631client # [ 127.254152] nix[1202]: building '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' on lb:50051...5632worker2 # [ 127.264411] nix-daemon[870]: accepted connection from pid 1197, user nix-grpc-daemon (trusted)5633lb: must succeed: curl -sf localhost:9901/clusters5634lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5635lb: must succeed: curl -sf localhost:9901/clusters5636lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)5637worker2: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service5638worker2 # [ 129.244453] systemd[1]: nix-grpc-daemon.service: Sent signal SIGKILL to main process 1197 (nix-grpc-daemon) on client request.5639worker2 # [ 129.246023] systemd[1]: nix-grpc-daemon.service: Killed unit cgroup '/system.slice/nix-grpc-daemon.service' with SIGKILL on client request.5640lb # [ 129.281898] niks3-server[1127]: 2026/09/21 09:15:18 INFO lead: released remote=192.168.1.4:425085641worker2 # [ 129.274098] systemd[1]: Starting scheduler-token.service...5642client # [ 129.262946] nix[1202]: chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), asking the scheduler again5643worker1 # [ 129.265264] nix-grpc-daemon[1473]: ts=2026-09-21T09:15:18Z level=info event=scheduler_disconnected addr=lb:50051 error="upstream connect error or disconnect/reset before headers. reset reason: connection termination"5644client # [ 129.269330] nix[1202]: scheduler at lb:50051: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), reconnecting5645worker2 # [ 129.287494] systemd[1]: nix-grpc-daemon.service: Main process exited, code=killed, status=9/KILL5646worker2 # [ 129.294046] systemd[1]: nix-grpc-daemon.service: Failed with result 'signal'.5647worker2 # [ 129.295357] systemd[1]: nix-grpc-daemon.service: Consumed 580ms CPU time over 16.960s wall clock time, 34.5M memory peak, 17.2K incoming IP traffic, 33.6K outgoing IP traffic.5648worker2 # [ 129.313880] nix-daemon[870]: reaped child process 1306, status = succeeded5649worker2 # [ 129.315527] nix-daemon[870]: reaped child process 1342, status = succeeded5650worker2 # [ 129.324397] nix-daemon[1344]: unexpected Nix daemon error: error: interrupted by the user5651worker2 # [ 129.332764] nix-daemon[870]: reaped child process 1344, status = failed with exit code 15652worker2 # [ 129.376391] systemd[1]: scheduler-token.service: Deactivated successfully.5653worker2 # [ 129.377339] systemd[1]: Finished scheduler-token.service.5654worker2 # [ 129.378067] systemd[1]: scheduler-token.service: Consumed 23ms CPU time over 99ms wall clock time, 2.2M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5655worker2 # [ 129.379918] systemd[1]: nix-grpc-daemon.service: Scheduled restart job immediately on client request, restart counter is at 1.5656worker2 # [ 129.385360] systemd[1]: Starting scheduler-token.service...5657worker2 # [ 129.456001] systemd[1]: scheduler-token.service: Deactivated successfully.5658worker2 # [ 129.459819] systemd[1]: Finished scheduler-token.service.5659worker2 # [ 129.461013] systemd[1]: scheduler-token.service: Consumed 22ms CPU time over 70ms wall clock time, 2.2M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5660worker2 # [ 129.464311] systemd[1]: Starting Nix worker-protocol over gRPC...5661worker1 # [ 129.465342] nix-grpc-daemon[1473]: ts=2026-09-21T09:15:19Z level=info event=scheduler_connected addr=lb:500515662worker2 # [ 129.631114] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:19Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5663worker2 # [ 129.633628] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:19Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5664worker2 # [ 129.637649] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:19Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15665worker2 # [ 129.644067] nix-grpc-daemon[1386]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5666worker2 # [ 129.645505] nix-grpc-daemon[1386]: E0000 00:00:1789982119.226606 1386 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5667worker2 # [ 129.662303] nix-grpc-daemon[1386]: E0000 00:00:1789982119.232804 1386 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5668worker2 # [ 129.665696] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:19Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5669lb # [ 129.684732] niks3-server[1127]: 2026/09/21 09:15:19 INFO lead: acquired remote=192.168.1.4:447645670worker2 # [ 129.668915] systemd[1]: Started Nix worker-protocol over gRPC.5671worker2 # [ 129.670055] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:19Z level=info event=scheduler_take_over5672worker2: (finished: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service, in 0.49 seconds)5673client: must succeed: sleep 5.05674worker2 # [ 129.700179] nix-grpc-daemon[1386]: E0000 00:00:1789982119.254338 1428 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5675worker2 # [ 129.702520] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:19Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:571705676worker2 # [ 129.709945] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:19Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=05677worker2 # [ 129.712236] nix-grpc-daemon[1386]: E0000 00:00:1789982119.273143 1428 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5678worker2 # [ 129.714623] nix-grpc-daemon[1386]: E0000 00:00:1789982119.273280 1428 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5679worker2 # [ 129.719767] nix-grpc-daemon[1386]: E0000 00:00:1789982119.292067 1428 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5680worker2 # [ 129.725113] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:19Z level=info event=scheduler_connected addr=lb:500515681worker2 # [ 129.730053] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:19Z level=info event=oidc_jwks provider=mock keys=15682worker2 # [ 129.731367] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:19Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:571705683worker2 # [ 129.733312] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:19Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=05684worker2 # [ 129.786047] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:19Z level=debug event=want client=1 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv cached=05685worker2 # [ 129.789495] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:19Z level=debug event=assigned drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv worker=192.168.1.3:50051 clients=15686worker1 # [ 129.842565] nix-grpc-daemon[1473]: ts=2026-09-21T09:15:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:452225687worker1 # [ 129.858378] nix-daemon[843]: accepted connection from pid 1473, user nix-grpc-daemon (trusted)5688worker1 # [ 129.884494] nix-grpc-daemon[1473]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 75 ms (attempt 1/5)5689worker1 # [ 129.967129] nix-grpc-daemon[1473]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 36 ms (attempt 2/5)5690worker1 # [ 130.011191] nix-grpc-daemon[1473]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 257 ms (attempt 3/5)5691worker1 # [ 130.277715] nix-grpc-daemon[1473]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 367 ms (attempt 4/5)5692worker1 # [ 130.645708] nix-grpc-daemon[1473]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5693worker1 # [ 130.670383] nix-grpc-daemon[1473]: copying path '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' from 'http://lb:5751'...5694worker1 # [ 130.699366] nix-daemon[843]: accepted connection from pid 1473, user nix-grpc-daemon (trusted)5695client # [ 130.715631] nix[1202]: building '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' on lb:50051...5696worker1 # [ 130.722315] nix-daemon[843]: accepted connection from pid 1473, user nix-grpc-daemon (trusted)5697lb # [ 130.975135] envoy[913]: conn peer=[2001:db8:1::1]:35730 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2936 tx=1332 ms=84095698lb # [ 130.980143] envoy[913]: conn peer=[2001:db8:1::3]:53266 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=4160 tx=1315 ms=252055699lb # [ 130.985258] envoy[913]: conn peer=[2001:db8:1::4]:58204 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=8006 tx=595 ms=166825700client: (finished: must succeed: sleep 5.0, in 5.06 seconds)5701client: waiting for success: ! systemctl is-active mid5702worker1 # [ 135.149111] nix-grpc-daemon[1473]: ts=2026-09-21T09:15:24Z level=info event=niks3_push_started pid=15775703worker1 # [ 135.193460] nix-daemon[843]: reaped child process 1553, status = succeeded5704worker1 # [ 135.385466] nix-grpc-daemon[1577]: time=2026-09-21T09:15:24.980Z level=INFO msg="Configuring client TLS" cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker1.pem key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker1.key ca=""5705worker1 # [ 135.560198] nix-daemon[843]: accepted connection from pid 1590, user nix-grpc-daemon (trusted)5706worker1 # [ 135.578313] nix-daemon[843]: reaped child process 1593, status = succeeded5707lb # [ 135.617224] niks3-server[1127]: 2026/09/21 09:15:25 INFO Received uploads request method=POST path=/api/pending_closures5708worker1 # [ 135.602911] nix-grpc-daemon[1577]: time=2026-09-21T09:15:25.196Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5709worker1 # [ 135.605730] nix-grpc-daemon[1577]: time=2026-09-21T09:15:25.196Z level=INFO msg="Uploading fk2vdwmn2msb3ja19s2jwak35ignz889-slow-mid (120B)"5710lb # [ 135.665437] niks3-server[1127]: 2026/09/21 09:15:25 INFO Registered completed upload object_key=nar/0462jb7zwy2d73k3q6v09xnk3bx318v8ik5ckmp32w2hzvwj4k2d.nar.zst5711lb # [ 135.678256] niks3-server[1127]: 2026/09/21 09:15:25 INFO Received sign narinfos request method=POST path=/api/pending_closures/36/sign5712worker1 # [ 135.654456] nix-grpc-daemon[1577]: time=2026-09-21T09:15:25.248Z level=INFO msg="Uploading 1 narinfos"5713lb # [ 135.688225] niks3-server[1127]: 2026/09/21 09:15:25 INFO Signed narinfos id=36 count=15714lb # [ 135.690791] niks3-server[1127]: 2026/09/21 09:15:25 INFO Registered completed upload object_key=log/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv5715worker1 # [ 135.680246] nix-grpc-daemon[1577]: time=2026-09-21T09:15:25.275Z level=INFO msg="Upload complete. (291ms)"5716lb # [ 135.706527] niks3-server[1127]: 2026/09/21 09:15:25 INFO Received complete upload request method=POST path=/api/pending_closures/36/complete5717lb # [ 135.720361] niks3-server[1127]: 2026/09/21 09:15:25 INFO Completed upload id=365718worker1 # [ 135.688145] nix-grpc-daemon[1473]: ts=2026-09-21T09:15:25Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:45222 duration_s=5 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv assign_id=1 outputs=15719worker1 # [ 135.703547] nix-daemon[843]: reaped child process 1551, status = succeeded5720worker1 # [ 135.712100] nix-grpc-daemon[1577]: time=2026-09-21T09:15:25.307Z level=WARN msg="Failed to register uploaded object" key=fk2vdwmn2msb3ja19s2jwak35ignz889.narinfo error="server returned 404: object is not pending upload\n"5721lb # [ 135.744438] niks3-server[1127]: 2026/09/21 09:15:25 INFO Registered completed upload object_key=fk2vdwmn2msb3ja19s2jwak35ignz889.ls5722client # [ 135.736740] systemd[1]: mid.service: Deactivated successfully.5723client # [ 135.740876] systemd[1]: mid.service: Consumed 184ms CPU time over 10.532s wall clock time, 39.2M memory peak, 8.1K incoming IP traffic, 9.7K outgoing IP traffic.5724client: (finished: waiting for success: ! systemctl is-active mid, in 1.14 seconds)5725client: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }5726client: (finished: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }, in 0.06 seconds)5727(finished: subtest: scheduler restart mid-build: build finishes, no double build, in 11.11 seconds)5728subtest: interrupting the client stops the build on the worker5729client: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag intr5730client # [ 136.283172] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key" --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag intr.5731client # Running as unit: intr.service; invocation ID: d64b9d980a774276a4efc99b9d9f71105732client: (finished: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag intr, in 0.05 seconds)5733worker2 # [ 136.542702] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:26Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:466165734worker2 # [ 136.557213] nix-daemon[870]: accepted connection from pid 1386, user nix-grpc-daemon (trusted)5735client # [ 136.565954] nix[1238]: don't know how to build these paths:5736client # [ 136.568413] nix[1238]: /nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv5737worker2 # [ 136.588831] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:26Z level=debug event=want client=2 drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv cached=05738worker2 # [ 136.595487] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:26Z level=debug event=assigned drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv worker=192.168.1.4:50051 clients=15739worker2 # [ 136.642015] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:466165740worker2 # [ 136.659871] nix-grpc-daemon[1386]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 80 ms (attempt 1/5)5741worker2 # [ 136.747605] nix-grpc-daemon[1386]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 95 ms (attempt 2/5)5742worker2 # [ 136.850315] nix-grpc-daemon[1386]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 188 ms (attempt 3/5)5743worker2 # [ 137.046053] nix-grpc-daemon[1386]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 119 ms (attempt 4/5)5744worker2 # [ 137.173537] nix-grpc-daemon[1386]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5745worker2 # [ 137.194566] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:26Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:466165746client # [ 137.200139] nix[1238]: copying 1 paths to 'lb:50051'...5747client # [ 137.204057] nix[1238]: copying path '/nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv' to 'lb:50051'...5748worker2 # [ 137.214205] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:26Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:466165749worker2 # [ 137.229221] nix-daemon[870]: accepted connection from pid 1386, user nix-grpc-daemon (trusted)5750worker2 # [ 137.258099] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:26Z level=info event=niks3_push_started pid=14615751worker2 # [ 137.469485] nix-grpc-daemon[1461]: time=2026-09-21T09:15:27.051Z level=INFO msg="Configuring client TLS" cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker2.pem key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker2.key ca=""5752worker2 # [ 137.654272] nix-daemon[870]: accepted connection from pid 1474, user nix-grpc-daemon (trusted)5753worker2 # [ 137.671782] nix-daemon[870]: reaped child process 1481, status = succeeded5754worker2 # [ 137.846776] nix-daemon[870]: accepted connection from pid 1483, user nix-grpc-daemon (trusted)5755worker2 # [ 137.865381] nix-daemon[870]: reaped child process 1486, status = succeeded5756lb # [ 137.898077] niks3-server[1127]: 2026/09/21 09:15:27 INFO Received uploads request method=POST path=/api/pending_closures5757worker2 # [ 137.894592] nix-grpc-daemon[1461]: time=2026-09-21T09:15:27.476Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5758worker2 # [ 137.896346] nix-grpc-daemon[1461]: time=2026-09-21T09:15:27.476Z level=INFO msg="Uploading 85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv (424B)"5759lb # [ 137.937883] niks3-server[1127]: 2026/09/21 09:15:27 INFO Registered completed upload object_key=nar/0h460h76prjyvhqhyxlhz1xrmh3s520jbd19gaagy4hp4vc92r1p.nar.zst5760lb # [ 137.949309] niks3-server[1127]: 2026/09/21 09:15:27 INFO Received sign narinfos request method=POST path=/api/pending_closures/37/sign5761worker2 # [ 137.938771] nix-grpc-daemon[1461]: time=2026-09-21T09:15:27.521Z level=INFO msg="Uploading 1 narinfos"5762lb # [ 137.962886] niks3-server[1127]: 2026/09/21 09:15:27 INFO Signed narinfos id=37 count=15763lb # [ 137.984084] niks3-server[1127]: 2026/09/21 09:15:27 INFO Received complete upload request method=POST path=/api/pending_closures/37/complete5764worker2 # [ 137.977380] nix-grpc-daemon[1461]: time=2026-09-21T09:15:27.560Z level=INFO msg="Upload complete. (504ms)"5765lb # [ 137.995083] niks3-server[1127]: 2026/09/21 09:15:27 INFO Registered completed upload object_key=85wa5czf6m5ng93xyq44r1affb74csh8.ls5766lb # [ 138.001793] niks3-server[1127]: 2026/09/21 09:15:27 INFO Completed upload id=375767lb # [ 138.005146] niks3-server[1127]: 2026/09/21 09:15:27 INFO Registered completed upload object_key=85wa5czf6m5ng93xyq44r1affb74csh8.narinfo5768worker2 # [ 137.988614] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:27Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46616 duration_s=0 paths=1 nar_bytes_in=4245769worker2 # [ 138.000711] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:466165770worker2 # [ 138.012174] nix-daemon[870]: accepted connection from pid 1386, user nix-grpc-daemon (trusted)5771worker2 # [ 138.016914] nix-daemon[870]: reaped child process 1459, status = succeeded5772client # [ 138.019385] nix[1238]: building '/nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv' on lb:50051...5773worker2 # [ 138.031531] nix-daemon[870]: accepted connection from pid 1386, user nix-grpc-daemon (trusted)5774client: must succeed: systemctl kill -s INT intr5775worker2 # [ 139.989596] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:29Z level=info event=revoked_running drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv5776client # [ 139.977227] systemd[1]: intr.service: Sent signal SIGINT to main process 1238 (nix) on client request.5777client # [ 139.986476] nix[1238]: shutting down85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED)5778client: (finished: must succeed: systemctl kill -s INT intr, in 0.07 seconds)5779client # [ 139.996530] nix[1238]: error: interrupted by the user5780client # [ 140.037273] systemd[1]: intr.service: Main process exited, code=exited, status=1/FAILURE5781client # [ 140.042092] systemd[1]: intr.service: Failed with result 'exit-code'.5782client # [ 140.044684] systemd[1]: intr.service: Consumed 158ms CPU time over 3.757s wall clock time, 41.2M memory peak, 6.1K incoming IP traffic, 8K outgoing IP traffic.5783worker2 # [ 140.057045] nix-daemon[870]: reaped child process 1488, status = succeeded5784worker2 # [ 140.059348] nix-daemon[1490]: unexpected Nix daemon error: error: interrupted by the user5785worker2 # [ 140.067813] nix-daemon[870]: reaped child process 1490, status = failed with exit code 15786(finished: subtest: interrupting the client stops the build on the worker, in 3.88 seconds)5787subtest: a deploy mid-build drains: the build finishes, the new generation takes over5788client: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag sw5789client # [ 140.191554] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key" --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag sw.5790client # Running as unit: sw.service; invocation ID: 7f9064b5e9924b88b5ac6865a541a2475791client: (finished: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag sw, in 0.08 seconds)5792worker1 # [ 140.440630] nix-grpc-daemon[1473]: ts=2026-09-21T09:15:30Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:452225793client # [ 140.447515] nix[1258]: don't know how to build these paths:5794client # [ 140.448669] nix[1258]: /nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv5795worker2 # [ 140.467361] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:30Z level=debug event=want client=3 drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv cached=05796worker2 # [ 140.473512] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:30Z level=debug event=assigned drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv worker=192.168.1.3:50051 clients=15797worker1 # [ 140.507004] nix-grpc-daemon[1473]: ts=2026-09-21T09:15:30Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:452225798worker1 # [ 140.523549] nix-grpc-daemon[1473]: ts=2026-09-21T09:15:30Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:452225799client # [ 140.536264] nix[1258]: copying 1 paths to 'lb:50051'...5800worker1 # [ 140.538964] nix-grpc-daemon[1473]: ts=2026-09-21T09:15:30Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:452225801client # [ 140.538571] nix[1258]: copying path '/nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv' to 'lb:50051'...5802worker1 # [ 140.555695] nix-daemon[843]: accepted connection from pid 1473, user nix-grpc-daemon (trusted)5803worker1 # [ 140.768337] nix-daemon[843]: accepted connection from pid 1621, user nix-grpc-daemon (trusted)5804worker1 # [ 140.789236] nix-daemon[843]: reaped child process 1624, status = succeeded5805lb # [ 140.972003] envoy[913]: conn peer=[2001:db8:1::1]:38380 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3828 tx=1685 ms=102415806lb # [ 140.972448] envoy[913]: conn peer=[2001:db8:1::1]:49892 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2944 tx=757 ms=34935807worker1 # [ 140.972672] nix-daemon[843]: accepted connection from pid 1626, user nix-grpc-daemon (trusted)5808worker1 # [ 140.989854] nix-daemon[843]: reaped child process 1629, status = succeeded5809lb # [ 141.028905] niks3-server[1127]: 2026/09/21 09:15:30 INFO Received uploads request method=POST path=/api/pending_closures5810worker1 # [ 141.017122] nix-grpc-daemon[1577]: time=2026-09-21T09:15:30.612Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5811worker1 # [ 141.018646] nix-grpc-daemon[1577]: time=2026-09-21T09:15:30.612Z level=INFO msg="Uploading x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv (416B)"5812lb # [ 141.062319] niks3-server[1127]: 2026/09/21 09:15:30 INFO Registered completed upload object_key=nar/0jczp45bkr8vp1ymznxf6da1iv8dzh42pnl36wiprbzwhpi2ddia.nar.zst5813worker1 # [ 141.051765] nix-grpc-daemon[1577]: time=2026-09-21T09:15:30.647Z level=INFO msg="Uploading 1 narinfos"5814lb # [ 141.078795] niks3-server[1127]: 2026/09/21 09:15:30 INFO Received sign narinfos request method=POST path=/api/pending_closures/38/sign5815lb # [ 141.090073] niks3-server[1127]: 2026/09/21 09:15:30 INFO Signed narinfos id=38 count=15816lb # [ 141.102998] niks3-server[1127]: 2026/09/21 09:15:30 INFO Registered completed upload object_key=x0m4asg5xx7vgj4dip2crwn6zhxqngvs.ls5817lb # [ 141.106924] niks3-server[1127]: 2026/09/21 09:15:30 INFO Received complete upload request method=POST path=/api/pending_closures/38/complete5818lb # [ 141.111773] niks3-server[1127]: 2026/09/21 09:15:30 INFO Completed upload id=385819worker1 # [ 141.085758] nix-grpc-daemon[1577]: time=2026-09-21T09:15:30.681Z level=INFO msg="Upload complete. (512ms)"5820lb # [ 141.114784] niks3-server[1127]: 2026/09/21 09:15:30 INFO Registered completed upload object_key=x0m4asg5xx7vgj4dip2crwn6zhxqngvs.narinfo5821worker1 # [ 141.096366] nix-grpc-daemon[1473]: ts=2026-09-21T09:15:30Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:45222 duration_s=0 paths=1 nar_bytes_in=4165822worker1 # [ 141.112349] nix-grpc-daemon[1473]: ts=2026-09-21T09:15:30Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:452225823worker1 # [ 141.118599] nix-daemon[843]: accepted connection from pid 1473, user nix-grpc-daemon (trusted)5824worker1 # [ 141.124507] nix-daemon[843]: reaped child process 1619, status = succeeded5825client # [ 141.133203] nix[1258]: building '/nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv' on lb:50051...5826worker1 # [ 141.139610] nix-daemon[843]: accepted connection from pid 1473, user nix-grpc-daemon (trusted)5827worker1: must succeed: systemctl reload nix-grpc-daemon.service5828worker1 # [ 142.889362] systemd[1]: Reloading Nix worker-protocol over gRPC...5829worker1 # [ 142.935092] systemd[1]: Reloaded Nix worker-protocol over gRPC.5830worker1: (finished: must succeed: systemctl reload nix-grpc-daemon.service, in 0.10 seconds)5831worker1: must succeed: pgrep -f 'read -t [9]0 x'5832worker1: (finished: must succeed: pgrep -f 'read -t [9]0 x', in 0.06 seconds)5833client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag during-drain >&25834worker1 # [ 143.015016] nix-grpc-daemon[1473]: ts=2026-09-21T09:15:32Z level=info event=draining5835client # don't know how to build these paths:5836client # /nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv5837worker2 # [ 143.278076] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:32Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:466165838worker2 # [ 143.285736] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:32Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:466165839worker2 # [ 143.310760] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:32Z level=debug event=want client=4 drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv cached=05840worker2 # [ 143.318047] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:32Z level=debug event=assigned drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv worker=192.168.1.4:50051 clients=15841worker2 # [ 143.361833] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:466165842worker2 # [ 143.377198] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:32Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:466165843client # copying 1 paths to 'lb:50051'...5844worker2 # [ 143.393443] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:32Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:466165845client # copying path '/nix/store/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv' to 'lb:50051'...5846worker2 # [ 143.417150] nix-daemon[870]: accepted connection from pid 1386, user nix-grpc-daemon (trusted)5847worker2 # [ 143.628821] nix-daemon[870]: accepted connection from pid 1540, user nix-grpc-daemon (trusted)5848worker2 # [ 143.648555] nix-daemon[870]: reaped child process 1543, status = succeeded5849worker2 # [ 143.822523] nix-daemon[870]: accepted connection from pid 1545, user nix-grpc-daemon (trusted)5850worker2 # [ 143.839226] nix-daemon[870]: reaped child process 1548, status = succeeded5851lb # [ 143.870305] niks3-server[1127]: 2026/09/21 09:15:33 INFO Received uploads request method=POST path=/api/pending_closures5852worker2 # [ 143.864336] nix-grpc-daemon[1461]: time=2026-09-21T09:15:33.446Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5853worker2 # [ 143.865794] nix-grpc-daemon[1461]: time=2026-09-21T09:15:33.446Z level=INFO msg="Uploading k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv (528B)"5854lb # [ 143.907835] niks3-server[1127]: 2026/09/21 09:15:33 INFO Registered completed upload object_key=nar/1h3fzlmfdwdiqr732wwrq7jnpw7n6p8f9qx1089c6g6x2i4csv75.nar.zst5855lb # [ 143.925621] niks3-server[1127]: 2026/09/21 09:15:33 INFO Received sign narinfos request method=POST path=/api/pending_closures/39/sign5856worker2 # [ 143.913105] nix-grpc-daemon[1461]: time=2026-09-21T09:15:33.495Z level=INFO msg="Uploading 1 narinfos"5857lb # [ 143.936518] niks3-server[1127]: 2026/09/21 09:15:33 INFO Signed narinfos id=39 count=15858lb # [ 143.938732] niks3-server[1127]: 2026/09/21 09:15:33 INFO Registered completed upload object_key=k3l35rf796jrnjns08398p1l8fcx178a.ls5859lb # [ 143.948796] niks3-server[1127]: 2026/09/21 09:15:33 INFO Received complete upload request method=POST path=/api/pending_closures/39/complete5860lb # [ 143.955755] niks3-server[1127]: 2026/09/21 09:15:33 INFO Completed upload id=395861worker2 # [ 143.937422] nix-grpc-daemon[1461]: time=2026-09-21T09:15:33.520Z level=INFO msg="Upload complete. (503ms)"5862lb # [ 143.958446] niks3-server[1127]: 2026/09/21 09:15:33 INFO Registered completed upload object_key=k3l35rf796jrnjns08398p1l8fcx178a.narinfo5863worker2 # [ 143.947540] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:33Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46616 duration_s=0 paths=1 nar_bytes_in=5285864worker2 # [ 143.960252] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:33Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:466165865worker2 # [ 143.967278] nix-daemon[870]: accepted connection from pid 1386, user nix-grpc-daemon (trusted)5866worker2 # [ 143.972623] nix-daemon[870]: reaped child process 1538, status = succeeded5867client # building '/nix/store/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv' on lb:50051...5868client # farm-a> worker2: building k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv5869worker2 # [ 143.986137] nix-daemon[870]: accepted connection from pid 1386, user nix-grpc-daemon (trusted)5870client # farm-a> LOG-farm-a-during-drain5871worker2 # [ 145.228801] nix-daemon[870]: reaped child process 1552, status = succeeded5872worker2 # [ 145.406221] nix-daemon[870]: accepted connection from pid 1567, user nix-grpc-daemon (trusted)5873worker2 # [ 145.423249] nix-daemon[870]: reaped child process 1570, status = succeeded5874lb # [ 145.454152] niks3-server[1127]: 2026/09/21 09:15:35 INFO Received uploads request method=POST path=/api/pending_closures5875worker2 # [ 145.448496] nix-grpc-daemon[1461]: time=2026-09-21T09:15:35.030Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5876worker2 # [ 145.450456] nix-grpc-daemon[1461]: time=2026-09-21T09:15:35.030Z level=INFO msg="Uploading 42c24pndjip9nvdj8j25mkv0650griqf-farm-a (136B)"5877lb # [ 145.501196] niks3-server[1127]: 2026/09/21 09:15:35 INFO Registered completed upload object_key=nar/0r0px9vw070lid3q4jhnmka11sk2y8kj26ysyvjy9dplavd7mdz4.nar.zst5878lb # [ 145.514681] niks3-server[1127]: 2026/09/21 09:15:35 INFO Registered completed upload object_key=log/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv5879lb # [ 145.522825] niks3-server[1127]: 2026/09/21 09:15:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/40/sign5880lb # [ 145.528821] niks3-server[1127]: 2026/09/21 09:15:35 INFO Signed narinfos id=40 count=15881worker2 # [ 145.510665] nix-grpc-daemon[1461]: time=2026-09-21T09:15:35.092Z level=INFO msg="Uploading 1 narinfos"5882lb # [ 145.534240] niks3-server[1127]: 2026/09/21 09:15:35 INFO Registered completed upload object_key=42c24pndjip9nvdj8j25mkv0650griqf.ls5883lb # [ 145.543560] niks3-server[1127]: 2026/09/21 09:15:35 INFO Received complete upload request method=POST path=/api/pending_closures/40/complete5884worker2 # [ 145.530540] nix-grpc-daemon[1461]: time=2026-09-21T09:15:35.113Z level=INFO msg="Upload complete. (333ms)"5885lb # [ 145.550745] niks3-server[1127]: 2026/09/21 09:15:35 INFO Completed upload id=405886lb # [ 145.553590] niks3-server[1127]: 2026/09/21 09:15:35 INFO Registered completed upload object_key=42c24pndjip9nvdj8j25mkv0650griqf.narinfo5887worker2 # [ 145.545595] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:35Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46616 duration_s=1 drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv assign_id=4 outputs=15888worker2 # [ 145.566060] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:35Z level=debug event=want client=4 drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv cached=05889worker2 # [ 145.570551] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:35Z level=debug event=assigned drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv worker=192.168.1.4:50051 clients=15890worker2 # [ 145.575166] nix-daemon[870]: reaped child process 1550, status = succeeded5891worker2 # [ 145.602155] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:466165892worker2 # [ 145.617636] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:35Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:466165893worker2 # [ 145.632744] nix-grpc-daemon[1461]: time=2026-09-21T09:15:35.215Z level=INFO msg="All 1 paths already cached"5894client # copying 1 paths to 'lb:50051'...5895worker2 # [ 145.642534] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:35Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:466165896client # copying path '/nix/store/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv' to 'lb:50051'...5897worker2 # [ 145.667063] nix-daemon[870]: accepted connection from pid 1386, user nix-grpc-daemon (trusted)5898worker2 # [ 145.876901] nix-daemon[870]: accepted connection from pid 1576, user nix-grpc-daemon (trusted)5899worker2 # [ 145.896535] nix-daemon[870]: reaped child process 1579, status = succeeded5900worker2 # [ 146.067940] nix-daemon[870]: accepted connection from pid 1581, user nix-grpc-daemon (trusted)5901worker2 # [ 146.086181] nix-daemon[870]: reaped child process 1584, status = succeeded5902lb # [ 146.115445] niks3-server[1127]: 2026/09/21 09:15:35 INFO Received uploads request method=POST path=/api/pending_closures5903worker2 # [ 146.107009] nix-grpc-daemon[1461]: time=2026-09-21T09:15:35.689Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5904worker2 # [ 146.108572] nix-grpc-daemon[1461]: time=2026-09-21T09:15:35.689Z level=INFO msg="Uploading im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv (640B)"5905lb # [ 146.144632] niks3-server[1127]: 2026/09/21 09:15:35 INFO Registered completed upload object_key=nar/0rxmdnq61ka29n0b2as2hdbz91wi2ap17db890js8r5plhpg436g.nar.zst5906worker2 # [ 146.139573] nix-grpc-daemon[1461]: time=2026-09-21T09:15:35.722Z level=INFO msg="Uploading 1 narinfos"5907lb # [ 146.160253] niks3-server[1127]: 2026/09/21 09:15:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/41/sign5908lb # [ 146.171329] niks3-server[1127]: 2026/09/21 09:15:35 INFO Signed narinfos id=41 count=15909lb # [ 146.177337] niks3-server[1127]: 2026/09/21 09:15:35 INFO Registered completed upload object_key=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l.ls5910worker2 # [ 146.167589] nix-grpc-daemon[1461]: time=2026-09-21T09:15:35.750Z level=INFO msg="Upload complete. (483ms)"5911lb # [ 146.186559] niks3-server[1127]: 2026/09/21 09:15:35 INFO Received complete upload request method=POST path=/api/pending_closures/41/complete5912lb # [ 146.191450] niks3-server[1127]: 2026/09/21 09:15:35 INFO Completed upload id=415913worker2 # [ 146.176209] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:35Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46616 duration_s=0 paths=1 nar_bytes_in=6405914lb # [ 146.195467] niks3-server[1127]: 2026/09/21 09:15:35 INFO Registered completed upload object_key=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l.narinfo5915worker2 # [ 146.185623] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:466165916worker2 # [ 146.192443] nix-daemon[870]: accepted connection from pid 1386, user nix-grpc-daemon (trusted)5917worker2 # [ 146.196938] nix-daemon[870]: reaped child process 1574, status = succeeded5918client # building '/nix/store/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv' on lb:50051...5919client # farm-b> worker2: building im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv5920worker2 # [ 146.209727] nix-daemon[870]: accepted connection from pid 1386, user nix-grpc-daemon (trusted)5921client # farm-b> LOG-farm-b-during-drain5922worker2 # [ 147.431838] nix-daemon[870]: reaped child process 1588, status = succeeded5923worker2 # [ 147.595870] nix-daemon[870]: accepted connection from pid 1603, user nix-grpc-daemon (trusted)5924worker2 # [ 147.615461] nix-daemon[870]: reaped child process 1606, status = succeeded5925lb # [ 147.646457] niks3-server[1127]: 2026/09/21 09:15:37 INFO Received uploads request method=POST path=/api/pending_closures5926worker2 # [ 147.638838] nix-grpc-daemon[1461]: time=2026-09-21T09:15:37.220Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5927worker2 # [ 147.640467] nix-grpc-daemon[1461]: time=2026-09-21T09:15:37.220Z level=INFO msg="Uploading daks7y2c761g04qpzsjx25629wigr8js-farm-b (184B)"5928lb # [ 147.679983] niks3-server[1127]: 2026/09/21 09:15:37 INFO Registered completed upload object_key=log/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv5929lb # [ 147.695856] niks3-server[1127]: 2026/09/21 09:15:37 INFO Registered completed upload object_key=nar/0lnpxl34nrzp147jy5bafczkvxcl00hcf69qjhsljmv2ip9jisfh.nar.zst5930lb # [ 147.712616] niks3-server[1127]: 2026/09/21 09:15:37 INFO Received sign narinfos request method=POST path=/api/pending_closures/42/sign5931worker2 # [ 147.701492] nix-grpc-daemon[1461]: time=2026-09-21T09:15:37.283Z level=INFO msg="Uploading 1 narinfos"5932lb # [ 147.725901] niks3-server[1127]: 2026/09/21 09:15:37 INFO Signed narinfos id=42 count=15933lb # [ 147.731821] niks3-server[1127]: 2026/09/21 09:15:37 INFO Registered completed upload object_key=daks7y2c761g04qpzsjx25629wigr8js.ls5934lb # [ 147.739572] niks3-server[1127]: 2026/09/21 09:15:37 INFO Received complete upload request method=POST path=/api/pending_closures/42/complete5935worker2 # [ 147.727381] nix-grpc-daemon[1461]: time=2026-09-21T09:15:37.310Z level=INFO msg="Upload complete. (308ms)"5936lb # [ 147.747312] niks3-server[1127]: 2026/09/21 09:15:37 INFO Completed upload id=425937lb # [ 147.750249] niks3-server[1127]: 2026/09/21 09:15:37 INFO Registered completed upload object_key=daks7y2c761g04qpzsjx25629wigr8js.narinfo5938worker2 # [ 147.745669] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:37Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46616 duration_s=1 drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv assign_id=5 outputs=15939worker2 # [ 147.762857] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:37Z level=debug event=want client=4 drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv cached=05940worker2 # [ 147.767411] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:37Z level=debug event=assigned drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv worker=192.168.1.4:50051 clients=15941worker2 # [ 147.772276] nix-daemon[870]: reaped child process 1586, status = succeeded5942worker2 # [ 147.789805] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:466165943worker2 # [ 147.804541] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:37Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:466165944worker2 # [ 147.820572] nix-grpc-daemon[1461]: time=2026-09-21T09:15:37.403Z level=INFO msg="All 2 paths already cached"5945client # copying 1 paths to 'lb:50051'...5946client # copying path '/nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv' to 'lb:50051'...5947worker2 # [ 147.830978] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:37Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:466165948worker2 # [ 147.849094] nix-daemon[870]: accepted connection from pid 1386, user nix-grpc-daemon (trusted)5949worker2 # [ 148.053373] nix-daemon[870]: accepted connection from pid 1612, user nix-grpc-daemon (trusted)5950worker2 # [ 148.071438] nix-daemon[870]: reaped child process 1615, status = succeeded5951worker2 # [ 148.248675] nix-daemon[870]: accepted connection from pid 1617, user nix-grpc-daemon (trusted)5952worker2 # [ 148.264913] nix-daemon[870]: reaped child process 1620, status = succeeded5953lb # [ 148.296796] niks3-server[1127]: 2026/09/21 09:15:37 INFO Received uploads request method=POST path=/api/pending_closures5954worker2 # [ 148.288094] nix-grpc-daemon[1461]: time=2026-09-21T09:15:37.870Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5955worker2 # [ 148.289978] nix-grpc-daemon[1461]: time=2026-09-21T09:15:37.870Z level=INFO msg="Uploading lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv (768B)"5956lb # [ 148.326310] niks3-server[1127]: 2026/09/21 09:15:37 INFO Registered completed upload object_key=nar/066jhl91shf8ar6h3cqqb855j5cswl7170wrg5lzf3fhpka0xywn.nar.zst5957worker2 # [ 148.320968] nix-grpc-daemon[1461]: time=2026-09-21T09:15:37.903Z level=INFO msg="Uploading 1 narinfos"5958lb # [ 148.343434] niks3-server[1127]: 2026/09/21 09:15:37 INFO Received sign narinfos request method=POST path=/api/pending_closures/43/sign5959lb # [ 148.354221] niks3-server[1127]: 2026/09/21 09:15:37 INFO Signed narinfos id=43 count=15960worker2 # [ 148.345917] nix-grpc-daemon[1461]: time=2026-09-21T09:15:37.928Z level=INFO msg="Upload complete. (480ms)"5961lb # [ 148.364623] niks3-server[1127]: 2026/09/21 09:15:37 INFO Registered completed upload object_key=lbk8wy49c881163x07gb8z669j2fg1ff.ls5962lb # [ 148.369715] niks3-server[1127]: 2026/09/21 09:15:37 INFO Received complete upload request method=POST path=/api/pending_closures/43/complete5963worker2 # [ 148.352240] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:37Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46616 duration_s=0 paths=1 nar_bytes_in=7685964lb # [ 148.374317] niks3-server[1127]: 2026/09/21 09:15:37 INFO Completed upload id=435965lb # [ 148.376491] niks3-server[1127]: 2026/09/21 09:15:37 INFO Registered completed upload object_key=lbk8wy49c881163x07gb8z669j2fg1ff.narinfo5966worker2 # [ 148.364752] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:466165967worker2 # [ 148.373552] nix-daemon[870]: accepted connection from pid 1386, user nix-grpc-daemon (trusted)5968worker2 # [ 148.376857] nix-daemon[870]: reaped child process 1610, status = succeeded5969client # building '/nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv' on lb:50051...5970worker2 # [ 148.388805] nix-daemon[870]: accepted connection from pid 1386, user nix-grpc-daemon (trusted)5971client # farm-top> worker2: building lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv5972client # farm-top> LOG-farm-top-during-drain5973worker2 # [ 150.228622] nix-daemon[870]: reaped child process 1624, status = succeeded5974worker2 # [ 150.412746] nix-daemon[870]: accepted connection from pid 1639, user nix-grpc-daemon (trusted)5975worker2 # [ 150.431448] nix-daemon[870]: reaped child process 1642, status = succeeded5976lb # [ 150.462507] niks3-server[1127]: 2026/09/21 09:15:40 INFO Received uploads request method=POST path=/api/pending_closures5977worker2 # [ 150.455427] nix-grpc-daemon[1461]: time=2026-09-21T09:15:40.036Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5978worker2 # [ 150.457065] nix-grpc-daemon[1461]: time=2026-09-21T09:15:40.036Z level=INFO msg="Uploading xdwmx2f4n5n8bggglniz2n35any0xz4w-farm-top (240B)"5979lb # [ 150.499466] niks3-server[1127]: 2026/09/21 09:15:40 INFO Registered completed upload object_key=log/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv5980worker2 # [ 150.498180] nix-grpc-daemon[1461]: time=2026-09-21T09:15:40.080Z level=INFO msg="Uploading 1 narinfos"5981lb # [ 150.516411] niks3-server[1127]: 2026/09/21 09:15:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/44/sign5982lb # [ 150.523969] niks3-server[1127]: 2026/09/21 09:15:40 INFO Signed narinfos id=44 count=15983lb # [ 150.530554] niks3-server[1127]: 2026/09/21 09:15:40 INFO Registered completed upload object_key=nar/1bpy9ga4n2446pary6msidbs8mnil1a34jn8p8yl2scvdg691mz2.nar.zst5984lb # [ 150.539528] niks3-server[1127]: 2026/09/21 09:15:40 INFO Registered completed upload object_key=xdwmx2f4n5n8bggglniz2n35any0xz4w.ls5985lb # [ 150.548254] niks3-server[1127]: 2026/09/21 09:15:40 INFO Received complete upload request method=POST path=/api/pending_closures/44/complete5986lb # [ 150.554257] niks3-server[1127]: 2026/09/21 09:15:40 INFO Registered completed upload object_key=xdwmx2f4n5n8bggglniz2n35any0xz4w.narinfo5987worker2 # [ 150.535839] nix-grpc-daemon[1461]: time=2026-09-21T09:15:40.118Z level=INFO msg="Upload complete. (336ms)"5988lb # [ 150.558447] niks3-server[1127]: 2026/09/21 09:15:40 INFO Completed upload id=445989worker2 # [ 150.549129] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:40Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46616 duration_s=2 drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv assign_id=6 outputs=15990worker2 # [ 150.566001] nix-daemon[870]: reaped child process 1622, status = succeeded5991client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag during-drain >&2, in 7.59 seconds)5992client: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag during-drain outPath5993client: (finished: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag during-drain outPath, in 0.23 seconds)5994worker1: must succeed: pkill -f 'read -t [9]0 x'5995worker1: (finished: must succeed: pkill -f 'read -t [9]0 x', in 0.06 seconds)5996client: waiting for success: ! systemctl is-active sw5997lb # [ 150.973741] envoy[913]: conn peer=[2001:db8:1::1]:43120 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9668 tx=3887 ms=72875998worker1 # [ 150.980909] nix-daemon[843]: reaped child process 1633, status = succeeded5999worker1 # [ 151.142974] nix-daemon[843]: accepted connection from pid 1673, user nix-grpc-daemon (trusted)6000worker1 # [ 151.160332] nix-daemon[843]: reaped child process 1676, status = succeeded6001lb # [ 151.197293] niks3-server[1127]: 2026/09/21 09:15:40 INFO Received uploads request method=POST path=/api/pending_closures6002worker1 # [ 151.182285] nix-grpc-daemon[1577]: time=2026-09-21T09:15:40.777Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6003worker1 # [ 151.183818] nix-grpc-daemon[1577]: time=2026-09-21T09:15:40.777Z level=INFO msg="Uploading qr8vyl2zl3nxnin3vw41c6z35vj7087f-slow-sw (120B)"6004lb # [ 151.228396] niks3-server[1127]: 2026/09/21 09:15:40 INFO Registered completed upload object_key=log/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv6005lb # [ 151.242578] niks3-server[1127]: 2026/09/21 09:15:40 INFO Registered completed upload object_key=nar/0vi6cx31wc05wns04pr5vr2bak9iib1idndvy6gik6c2rz9q8c19.nar.zst6006lb # [ 151.253496] niks3-server[1127]: 2026/09/21 09:15:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/45/sign6007worker1 # [ 151.232819] nix-grpc-daemon[1577]: time=2026-09-21T09:15:40.827Z level=INFO msg="Uploading 1 narinfos"6008lb # [ 151.262546] niks3-server[1127]: 2026/09/21 09:15:40 INFO Signed narinfos id=45 count=16009lb # [ 151.267760] niks3-server[1127]: 2026/09/21 09:15:40 INFO Registered completed upload object_key=qr8vyl2zl3nxnin3vw41c6z35vj7087f.ls6010lb # [ 151.274921] niks3-server[1127]: 2026/09/21 09:15:40 INFO Received complete upload request method=POST path=/api/pending_closures/45/complete6011lb # [ 151.281668] niks3-server[1127]: 2026/09/21 09:15:40 INFO Registered completed upload object_key=qr8vyl2zl3nxnin3vw41c6z35vj7087f.narinfo6012lb # [ 151.286824] niks3-server[1127]: 2026/09/21 09:15:40 INFO Completed upload id=456013worker1 # [ 151.261479] nix-grpc-daemon[1577]: time=2026-09-21T09:15:40.857Z level=INFO msg="Upload complete. (303ms)"6014worker1 # [ 151.272986] nix-grpc-daemon[1473]: ts=2026-09-21T09:15:40Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:45222 duration_s=10 drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv assign_id=3 outputs=16015worker1 # [ 151.285200] nix-daemon[843]: reaped child process 1631, status = succeeded6016client # [ 151.312255] systemd[1]: sw.service: Deactivated successfully.6017client # [ 151.317387] systemd[1]: sw.service: Consumed 170ms CPU time over 11.124s wall clock time, 41.2M memory peak, 6.7K incoming IP traffic, 8.1K outgoing IP traffic.6018client: (finished: waiting for success: ! systemctl is-active sw, in 1.11 seconds)6019client: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }6020worker1 # [ 152.017388] nix-grpc-daemon[1473]: ts=2026-09-21T09:15:41Z level=info event=signal_exit6021client: (finished: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }, in 0.04 seconds)6022worker1: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service6023worker2 # [ 153.253052] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:42Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:571706024worker2 # [ 153.257966] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:42Z level=info event=worker_gone addr=192.168.1.3:500516025worker1 # [ 153.252816] nix-grpc-daemon[1473]: ts=2026-09-21T09:15:42Z level=info event=scheduler_restarting6026worker1 # [ 153.271241] nix-grpc-daemon[1473]: ts=2026-09-21T09:15:42Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"6027worker1 # [ 153.296180] nix-daemon[843]: reaped child process 1545, status = succeeded6028worker1 # [ 153.828279] nix-grpc-daemon[1473]: E0000 00:00:1789982143.423609 1518 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 15, started true, shutdown true6029worker1 # [ 153.834764] nix-grpc-daemon[1473]: E0000 00:00:1789982143.423893 1518 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true6030worker1 # [ 155.065311] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.6031worker1 # [ 155.071296] systemd[1]: nix-grpc-daemon.service: Consumed 892ms CPU time over 30.274s wall clock time, 33.5M memory peak, 21K incoming IP traffic, 27.1K outgoing IP traffic.6032worker1 # [ 155.092493] systemd[1]: Starting Nix worker-protocol over gRPC...6033worker1 # [ 155.286711] nix-grpc-daemon[1690]: ts=2026-09-21T09:15:44Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6034worker1 # [ 155.289367] nix-grpc-daemon[1690]: ts=2026-09-21T09:15:44Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6035worker1 # [ 155.293688] nix-grpc-daemon[1690]: ts=2026-09-21T09:15:44Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=16036worker1 # [ 155.301492] nix-grpc-daemon[1690]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6037worker1 # [ 155.303002] nix-grpc-daemon[1690]: E0000 00:00:1789982144.896437 1690 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6038worker1 # [ 155.314120] nix-grpc-daemon[1690]: E0000 00:00:1789982144.902598 1690 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6039worker1 # [ 155.318616] nix-grpc-daemon[1690]: ts=2026-09-21T09:15:44Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6040worker1: (finished: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service, in 3.29 seconds)6041lb: waiting until exactly no endpoints of aarch64-linux are unhealthy6042lb: must succeed: curl -sf localhost:9901/clusters6043worker1 # [ 155.352038] systemd[1]: Started Nix worker-protocol over gRPC.6044worker1 # [ 155.353007] nix-grpc-daemon[1690]: E0000 00:00:1789982144.918896 1730 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6045worker1 # [ 155.358703] nix-grpc-daemon[1690]: E0000 00:00:1789982144.934202 1730 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6046worker1 # [ 155.361703] nix-grpc-daemon[1690]: E0000 00:00:1789982144.934377 1730 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6047worker1 # [ 155.364203] nix-grpc-daemon[1690]: E0000 00:00:1789982144.953567 1730 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6048worker1 # [ 155.376866] nix-grpc-daemon[1690]: ts=2026-09-21T09:15:44Z level=info event=scheduler_connected addr=lb:500516049worker2 # [ 155.386732] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:44Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:571706050worker2 # [ 155.392516] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:44Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=06051lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds)6052lb: must succeed: curl -sf localhost:9901/clusters6053lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds)6054lb: (finished: waiting until exactly no endpoints of aarch64-linux are unhealthy, in 1.18 seconds)6055lb: must succeed: curl -sf localhost:9901/clusters6056lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)6057worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6058worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)6059(finished: subtest: a deploy mid-build drains: the build finishes, the new generation takes over, in 16.53 seconds)6060subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does6061client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&16062worker2 # [ 156.929354] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:46Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:466166063worker1 # [ 156.933019] nix-grpc-daemon[1690]: ts=2026-09-21T09:15:46Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:366206064worker1 # [ 156.949076] nix-daemon[843]: accepted connection from pid 1690, user nix-grpc-daemon (trusted)6065worker1 # [ 156.974617] nix-grpc-daemon[1690]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 14 ms (attempt 1/5)6066worker1 # [ 156.996820] nix-grpc-daemon[1690]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 167 ms (attempt 2/5)6067worker1 # [ 157.171488] nix-grpc-daemon[1690]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 184 ms (attempt 3/5)6068worker1 # [ 157.363417] nix-grpc-daemon[1690]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 233 ms (attempt 4/5)6069worker1 # [ 157.604832] nix-grpc-daemon[1690]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6070worker2 # [ 157.663281] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:47Z level=debug event=want client=5 drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv cached=06071worker2 # [ 157.663630] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:47Z level=debug event=assigned drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv worker=192.168.1.3:50051 clients=16072worker2 # [ 157.663810] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:47Z level=debug event=want client=5 drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv cached=06073worker2 # [ 157.663942] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:47Z level=debug event=want client=5 drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv cached=06074worker1 # [ 157.702815] nix-grpc-daemon[1690]: ts=2026-09-21T09:15:47Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:366206075worker1 # [ 157.720437] nix-grpc-daemon[1690]: ts=2026-09-21T09:15:47Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:366206076worker1 # [ 157.735171] nix-grpc-daemon[1690]: ts=2026-09-21T09:15:47Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:366206077worker1 # [ 157.750980] nix-daemon[843]: accepted connection from pid 1690, user nix-grpc-daemon (trusted)6078worker1 # [ 157.779050] nix-grpc-daemon[1690]: ts=2026-09-21T09:15:47Z level=info event=niks3_push_started pid=17496079worker1 # [ 158.004307] nix-grpc-daemon[1749]: time=2026-09-21T09:15:47.599Z level=INFO msg="Configuring client TLS" cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker1.pem key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker1.key ca=""6080worker1 # [ 158.183372] nix-daemon[843]: accepted connection from pid 1762, user nix-grpc-daemon (trusted)6081worker1 # [ 158.200697] nix-daemon[843]: reaped child process 1765, status = succeeded6082worker1 # [ 158.381284] nix-daemon[843]: accepted connection from pid 1767, user nix-grpc-daemon (trusted)6083worker1 # [ 158.398699] nix-daemon[843]: reaped child process 1770, status = succeeded6084lb # [ 158.437723] niks3-server[1127]: 2026/09/21 09:15:48 INFO Received uploads request method=POST path=/api/pending_closures6085worker1 # [ 158.424987] nix-grpc-daemon[1749]: time=2026-09-21T09:15:48.019Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6086worker1 # [ 158.426501] nix-grpc-daemon[1749]: time=2026-09-21T09:15:48.019Z level=INFO msg="Uploading 3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv (512B)"6087lb # [ 158.479015] niks3-server[1127]: 2026/09/21 09:15:48 INFO Registered completed upload object_key=nar/0zxfsmqijfc2dkglls3yp1x3w0w1na7vx39fxc124zddjzpaygd8.nar.zst6088worker1 # [ 158.473839] nix-grpc-daemon[1749]: time=2026-09-21T09:15:48.069Z level=INFO msg="Uploading 1 narinfos"6089lb # [ 158.492307] niks3-server[1127]: 2026/09/21 09:15:48 INFO Received sign narinfos request method=POST path=/api/pending_closures/46/sign6090lb # [ 158.505501] niks3-server[1127]: 2026/09/21 09:15:48 INFO Signed narinfos id=46 count=16091lb # [ 158.512226] niks3-server[1127]: 2026/09/21 09:15:48 INFO Registered completed upload object_key=3223c4sl4ncpldbvw558vqwddijwzfpd.ls6092lb # [ 158.524138] niks3-server[1127]: 2026/09/21 09:15:48 INFO Received complete upload request method=POST path=/api/pending_closures/46/complete6093worker1 # [ 158.505746] nix-grpc-daemon[1749]: time=2026-09-21T09:15:48.101Z level=INFO msg="Upload complete. (497ms)"6094lb # [ 158.531792] niks3-server[1127]: 2026/09/21 09:15:48 INFO Completed upload id=466095lb # [ 158.534406] niks3-server[1127]: 2026/09/21 09:15:48 INFO Registered completed upload object_key=3223c4sl4ncpldbvw558vqwddijwzfpd.narinfo6096worker1 # [ 158.515067] nix-grpc-daemon[1690]: ts=2026-09-21T09:15:48Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36620 duration_s=0 paths=1 nar_bytes_in=5126097worker1 # [ 158.529968] nix-grpc-daemon[1690]: ts=2026-09-21T09:15:48Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:366206098worker1 # [ 158.539730] nix-daemon[843]: accepted connection from pid 1690, user nix-grpc-daemon (trusted)6099worker1 # [ 158.542466] nix-daemon[843]: reaped child process 1747, status = succeeded6100worker1 # [ 158.556190] nix-daemon[843]: accepted connection from pid 1690, user nix-grpc-daemon (trusted)6101worker1 # [ 159.595745] nix-daemon[843]: reaped child process 1774, status = succeeded6102worker1 # [ 159.771721] nix-daemon[843]: accepted connection from pid 1789, user nix-grpc-daemon (trusted)6103worker1 # [ 159.791351] nix-daemon[843]: reaped child process 1792, status = succeeded6104lb # [ 159.829728] niks3-server[1127]: 2026/09/21 09:15:49 INFO Received uploads request method=POST path=/api/pending_closures6105worker1 # [ 159.816418] nix-grpc-daemon[1749]: time=2026-09-21T09:15:49.409Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6106worker1 # [ 159.817942] nix-grpc-daemon[1749]: time=2026-09-21T09:15:49.409Z level=INFO msg="Uploading dla3w0zf02z5jhccwa27i85bbgzrrsvb-farm-a (128B)"6107lb # [ 159.859724] niks3-server[1127]: 2026/09/21 09:15:49 INFO Registered completed upload object_key=log/3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv6108lb # [ 159.878591] niks3-server[1127]: 2026/09/21 09:15:49 INFO Registered completed upload object_key=nar/05i7ka1qa7vmhiys96ln3pp3nj3rfdlslnbydrxdqj40cmvmqcwf.nar.zst6109worker1 # [ 159.867495] nix-grpc-daemon[1749]: time=2026-09-21T09:15:49.462Z level=INFO msg="Uploading 1 narinfos"6110lb # [ 159.895915] niks3-server[1127]: 2026/09/21 09:15:49 INFO Received sign narinfos request method=POST path=/api/pending_closures/47/sign6111lb # [ 159.905410] niks3-server[1127]: 2026/09/21 09:15:49 INFO Signed narinfos id=47 count=16112lb # [ 159.909186] niks3-server[1127]: 2026/09/21 09:15:49 INFO Registered completed upload object_key=dla3w0zf02z5jhccwa27i85bbgzrrsvb.ls6113worker1 # [ 159.892182] nix-grpc-daemon[1749]: time=2026-09-21T09:15:49.487Z level=INFO msg="Upload complete. (325ms)"6114lb # [ 159.917845] niks3-server[1127]: 2026/09/21 09:15:49 INFO Received complete upload request method=POST path=/api/pending_closures/47/complete6115lb # [ 159.921562] niks3-server[1127]: 2026/09/21 09:15:49 INFO Completed upload id=476116lb # [ 159.924802] niks3-server[1127]: 2026/09/21 09:15:49 INFO Registered completed upload object_key=dla3w0zf02z5jhccwa27i85bbgzrrsvb.narinfo6117worker2 # [ 159.908580] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:49Z level=debug event=assigned drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv worker=192.168.1.3:50051 clients=16118worker1 # [ 159.902807] nix-grpc-daemon[1690]: ts=2026-09-21T09:15:49Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36620 duration_s=1 drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv assign_id=7 outputs=16119worker2 # [ 159.921327] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:49Z level=debug event=want client=5 drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv cached=06120worker1 # [ 159.920337] nix-grpc-daemon[1690]: ts=2026-09-21T09:15:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:366206121worker1 # [ 159.928171] nix-daemon[843]: reaped child process 1772, status = succeeded6122worker1 # [ 159.931184] nix-grpc-daemon[1690]: ts=2026-09-21T09:15:49Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:366206123worker1 # [ 159.942933] nix-grpc-daemon[1690]: ts=2026-09-21T09:15:49Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:366206124worker1 # [ 159.953372] nix-daemon[843]: accepted connection from pid 1690, user nix-grpc-daemon (trusted)6125worker1 # [ 160.144440] nix-daemon[843]: accepted connection from pid 1796, user nix-grpc-daemon (trusted)6126worker1 # [ 160.164671] nix-daemon[843]: reaped child process 1799, status = succeeded6127worker1 # [ 160.340246] nix-daemon[843]: accepted connection from pid 1801, user nix-grpc-daemon (trusted)6128worker1 # [ 160.359411] nix-daemon[843]: reaped child process 1804, status = succeeded6129lb # [ 160.398582] niks3-server[1127]: 2026/09/21 09:15:50 INFO Received uploads request method=POST path=/api/pending_closures6130worker1 # [ 160.381260] nix-grpc-daemon[1749]: time=2026-09-21T09:15:49.977Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6131worker1 # [ 160.383482] nix-grpc-daemon[1749]: time=2026-09-21T09:15:49.978Z level=INFO msg="Uploading pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv (512B)"6132lb # [ 160.426417] niks3-server[1127]: 2026/09/21 09:15:50 INFO Registered completed upload object_key=nar/0g5bxr683biqh358mkd9xrclwllan4cgnv4pzijas586xj68wwx0.nar.zst6133worker1 # [ 160.413456] nix-grpc-daemon[1749]: time=2026-09-21T09:15:50.008Z level=INFO msg="Uploading 1 narinfos"6134lb # [ 160.442931] niks3-server[1127]: 2026/09/21 09:15:50 INFO Received sign narinfos request method=POST path=/api/pending_closures/48/sign6135lb # [ 160.453675] niks3-server[1127]: 2026/09/21 09:15:50 INFO Signed narinfos id=48 count=16136lb # [ 160.463111] niks3-server[1127]: 2026/09/21 09:15:50 INFO Registered completed upload object_key=pplpzcnjfn295nm16cg0iihpfc9iffry.ls6137lb # [ 160.467072] niks3-server[1127]: 2026/09/21 09:15:50 INFO Received complete upload request method=POST path=/api/pending_closures/48/complete6138lb # [ 160.473000] niks3-server[1127]: 2026/09/21 09:15:50 INFO Completed upload id=486139worker1 # [ 160.445625] nix-grpc-daemon[1749]: time=2026-09-21T09:15:50.041Z level=INFO msg="Upload complete. (480ms)"6140lb # [ 160.476449] niks3-server[1127]: 2026/09/21 09:15:50 INFO Registered completed upload object_key=pplpzcnjfn295nm16cg0iihpfc9iffry.narinfo6141worker1 # [ 160.452400] nix-grpc-daemon[1690]: ts=2026-09-21T09:15:50Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36620 duration_s=0 paths=1 nar_bytes_in=5126142worker1 # [ 160.466500] nix-grpc-daemon[1690]: ts=2026-09-21T09:15:50Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:366206143worker1 # [ 160.472834] nix-daemon[843]: accepted connection from pid 1690, user nix-grpc-daemon (trusted)6144worker1 # [ 160.477761] nix-daemon[843]: reaped child process 1794, status = succeeded6145worker1 # [ 160.490612] nix-daemon[843]: accepted connection from pid 1690, user nix-grpc-daemon (trusted)6146lb # [ 160.973645] envoy[913]: conn peer=[2001:db8:1::1]:49908 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2935 tx=1333 ms=108436147lb # [ 160.978727] envoy[913]: conn peer=[2001:db8:1::3]:43914 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=986 tx=448 ms=281946148worker1 # [ 162.122407] nix-daemon[843]: reaped child process 1808, status = succeeded6149worker1 # [ 162.288900] nix-daemon[843]: accepted connection from pid 1823, user nix-grpc-daemon (trusted)6150worker1 # [ 162.306994] nix-daemon[843]: reaped child process 1826, status = succeeded6151lb # [ 162.344939] niks3-server[1127]: 2026/09/21 09:15:51 INFO Received uploads request method=POST path=/api/pending_closures6152worker1 # [ 162.332077] nix-grpc-daemon[1749]: time=2026-09-21T09:15:51.926Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6153worker1 # [ 162.333638] nix-grpc-daemon[1749]: time=2026-09-21T09:15:51.926Z level=INFO msg="Uploading fhc1zkcygpzppf0bj5fr2k85lf7p87v9-farm-a (128B)"6154lb # [ 162.377992] niks3-server[1127]: 2026/09/21 09:15:51 INFO Registered completed upload object_key=log/pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv6155lb # [ 162.391111] niks3-server[1127]: 2026/09/21 09:15:51 INFO Registered completed upload object_key=nar/09na6djn66dpys8h08gqw39p0vw7vxqsdzbiw55rknyna8s239bm.nar.zst6156lb # [ 162.407933] niks3-server[1127]: 2026/09/21 09:15:52 INFO Received sign narinfos request method=POST path=/api/pending_closures/49/sign6157worker1 # [ 162.389215] nix-grpc-daemon[1749]: time=2026-09-21T09:15:51.984Z level=INFO msg="Uploading 1 narinfos"6158lb # [ 162.419749] niks3-server[1127]: 2026/09/21 09:15:52 INFO Signed narinfos id=49 count=16159lb # [ 162.425996] niks3-server[1127]: 2026/09/21 09:15:52 INFO Registered completed upload object_key=fhc1zkcygpzppf0bj5fr2k85lf7p87v9.ls6160lb # [ 162.432153] niks3-server[1127]: 2026/09/21 09:15:52 INFO Received complete upload request method=POST path=/api/pending_closures/49/complete6161lb # [ 162.441422] niks3-server[1127]: 2026/09/21 09:15:52 INFO Completed upload id=496162worker1 # [ 162.415797] nix-grpc-daemon[1749]: time=2026-09-21T09:15:52.011Z level=INFO msg="Upload complete. (314ms)"6163lb # [ 162.444139] niks3-server[1127]: 2026/09/21 09:15:52 INFO Registered completed upload object_key=fhc1zkcygpzppf0bj5fr2k85lf7p87v9.narinfo6164worker2 # [ 162.433055] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:52Z level=debug event=assigned drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv worker=192.168.1.3:50051 clients=16165worker1 # [ 162.428467] nix-grpc-daemon[1690]: ts=2026-09-21T09:15:52Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36620 duration_s=1 drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv assign_id=8 outputs=16166worker2 # [ 162.445777] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:52Z level=debug event=want client=5 drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv cached=06167worker1 # [ 162.447667] nix-grpc-daemon[1690]: ts=2026-09-21T09:15:52Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:366206168worker1 # [ 162.454110] nix-daemon[843]: reaped child process 1806, status = succeeded6169worker1 # [ 162.462484] nix-grpc-daemon[1690]: ts=2026-09-21T09:15:52Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:366206170worker1 # [ 162.476134] nix-grpc-daemon[1690]: ts=2026-09-21T09:15:52Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:366206171worker1 # [ 162.488708] nix-daemon[843]: accepted connection from pid 1690, user nix-grpc-daemon (trusted)6172worker1 # [ 162.686871] nix-daemon[843]: accepted connection from pid 1830, user nix-grpc-daemon (trusted)6173worker1 # [ 162.705906] nix-daemon[843]: reaped child process 1833, status = succeeded6174worker1 # [ 162.881837] nix-daemon[843]: accepted connection from pid 1835, user nix-grpc-daemon (trusted)6175worker1 # [ 162.899672] nix-daemon[843]: reaped child process 1838, status = succeeded6176lb # [ 162.940908] niks3-server[1127]: 2026/09/21 09:15:52 INFO Received uploads request method=POST path=/api/pending_closures6177worker1 # [ 162.924141] nix-grpc-daemon[1749]: time=2026-09-21T09:15:52.519Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6178worker1 # [ 162.925717] nix-grpc-daemon[1749]: time=2026-09-21T09:15:52.519Z level=INFO msg="Uploading sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv (512B)"6179lb # [ 162.973012] niks3-server[1127]: 2026/09/21 09:15:52 INFO Registered completed upload object_key=nar/1fxz200fmpbckafrn03b2snx9knqpbwf8rwqlbvg68356qj7aj5w.nar.zst6180lb # [ 162.984107] niks3-server[1127]: 2026/09/21 09:15:52 INFO Received sign narinfos request method=POST path=/api/pending_closures/50/sign6181worker1 # [ 162.964792] nix-grpc-daemon[1749]: time=2026-09-21T09:15:52.560Z level=INFO msg="Uploading 1 narinfos"6182lb # [ 162.994168] niks3-server[1127]: 2026/09/21 09:15:52 INFO Signed narinfos id=50 count=16183lb # [ 163.000915] niks3-server[1127]: 2026/09/21 09:15:52 INFO Registered completed upload object_key=sy43wp9v52pni1wms6q0p3qybpc2pwad.ls6184lb # [ 163.009302] niks3-server[1127]: 2026/09/21 09:15:52 INFO Received complete upload request method=POST path=/api/pending_closures/50/complete6185lb # [ 163.014292] niks3-server[1127]: 2026/09/21 09:15:52 INFO Registered completed upload object_key=sy43wp9v52pni1wms6q0p3qybpc2pwad.narinfo6186lb # [ 163.018371] niks3-server[1127]: 2026/09/21 09:15:52 INFO Completed upload id=506187worker1 # [ 162.993056] nix-grpc-daemon[1749]: time=2026-09-21T09:15:52.588Z level=INFO msg="Upload complete. (490ms)"6188worker1 # [ 163.000916] nix-grpc-daemon[1690]: ts=2026-09-21T09:15:52Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36620 duration_s=0 paths=1 nar_bytes_in=5126189worker1 # [ 163.014324] nix-grpc-daemon[1690]: ts=2026-09-21T09:15:52Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:366206190worker1 # [ 163.020731] nix-daemon[843]: accepted connection from pid 1690, user nix-grpc-daemon (trusted)6191worker1 # [ 163.025561] nix-daemon[843]: reaped child process 1828, status = succeeded6192worker1 # [ 163.038692] nix-daemon[843]: accepted connection from pid 1690, user nix-grpc-daemon (trusted)6193worker1 # [ 164.087854] nix-daemon[843]: reaped child process 1842, status = succeeded6194worker1 # [ 164.278377] nix-daemon[843]: accepted connection from pid 1857, user nix-grpc-daemon (trusted)6195worker1 # [ 164.296662] nix-daemon[843]: reaped child process 1860, status = succeeded6196lb # [ 164.334173] niks3-server[1127]: 2026/09/21 09:15:53 INFO Received uploads request method=POST path=/api/pending_closures6197worker1 # [ 164.320390] nix-grpc-daemon[1749]: time=2026-09-21T09:15:53.915Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6198worker1 # [ 164.321903] nix-grpc-daemon[1749]: time=2026-09-21T09:15:53.915Z level=INFO msg="Uploading g5laypvfk2469p8binw3mksk2hv7zhnh-farm-a (128B)"6199lb # [ 164.373292] niks3-server[1127]: 2026/09/21 09:15:53 INFO Registered completed upload object_key=log/sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv6200lb # [ 164.384315] niks3-server[1127]: 2026/09/21 09:15:53 INFO Registered completed upload object_key=nar/1j2671k9mycq8lxyfx5k6a66d5narfqw2wqwlpwxgwzi93vnvbdn.nar.zst6201worker1 # [ 164.372428] nix-grpc-daemon[1749]: time=2026-09-21T09:15:53.967Z level=INFO msg="Uploading 1 narinfos"6202lb # [ 164.393549] niks3-server[1127]: 2026/09/21 09:15:53 INFO Received sign narinfos request method=POST path=/api/pending_closures/51/sign6203lb # [ 164.401996] niks3-server[1127]: 2026/09/21 09:15:54 INFO Signed narinfos id=51 count=16204lb # [ 164.406386] niks3-server[1127]: 2026/09/21 09:15:54 INFO Registered completed upload object_key=g5laypvfk2469p8binw3mksk2hv7zhnh.ls6205lb # [ 164.412249] niks3-server[1127]: 2026/09/21 09:15:54 INFO Received complete upload request method=POST path=/api/pending_closures/51/complete6206lb # [ 164.418607] niks3-server[1127]: 2026/09/21 09:15:54 INFO Completed upload id=516207worker1 # [ 164.393142] nix-grpc-daemon[1749]: time=2026-09-21T09:15:53.988Z level=INFO msg="Upload complete. (344ms)"6208lb # [ 164.420475] niks3-server[1127]: 2026/09/21 09:15:54 INFO Registered completed upload object_key=g5laypvfk2469p8binw3mksk2hv7zhnh.narinfo6209worker2 # [ 164.412177] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:53Z level=debug event=assigned drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv worker=192.168.1.3:50051 clients=16210worker1 # [ 164.405178] nix-grpc-daemon[1690]: ts=2026-09-21T09:15:53Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36620 duration_s=1 drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv assign_id=9 outputs=16211worker2 # [ 164.419988] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:54Z level=debug event=want client=5 drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv cached=06212worker1 # [ 164.423399] nix-daemon[843]: reaped child process 1840, status = succeeded6213worker1 # [ 164.451906] nix-grpc-daemon[1690]: ts=2026-09-21T09:15:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:366206214worker1 # [ 164.467255] nix-grpc-daemon[1690]: ts=2026-09-21T09:15:54Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:366206215worker1 # [ 164.482746] nix-grpc-daemon[1749]: time=2026-09-21T09:15:54.077Z level=INFO msg="All 1 paths already cached"6216worker1 # [ 164.488274] nix-grpc-daemon[1690]: ts=2026-09-21T09:15:54Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:366206217worker1 # [ 164.501944] nix-daemon[843]: accepted connection from pid 1690, user nix-grpc-daemon (trusted)6218worker1 # [ 164.708971] nix-daemon[843]: accepted connection from pid 1864, user nix-grpc-daemon (trusted)6219worker1 # [ 164.728666] nix-daemon[843]: reaped child process 1867, status = succeeded6220worker1 # [ 164.907011] nix-daemon[843]: accepted connection from pid 1869, user nix-grpc-daemon (trusted)6221worker1 # [ 164.925651] nix-daemon[843]: reaped child process 1872, status = succeeded6222lb # [ 164.964736] niks3-server[1127]: 2026/09/21 09:15:54 INFO Received uploads request method=POST path=/api/pending_closures6223worker1 # [ 164.949347] nix-grpc-daemon[1749]: time=2026-09-21T09:15:54.544Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6224worker1 # [ 164.950894] nix-grpc-daemon[1749]: time=2026-09-21T09:15:54.544Z level=INFO msg="Uploading cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv (624B)"6225lb # [ 164.996893] niks3-server[1127]: 2026/09/21 09:15:54 INFO Registered completed upload object_key=nar/1w4sykcjji9zlw0a5xibxycy6x3jyb4drd29x2k9g7x5whfwwpsq.nar.zst6226worker1 # [ 164.986176] nix-grpc-daemon[1749]: time=2026-09-21T09:15:54.581Z level=INFO msg="Uploading 1 narinfos"6227lb # [ 165.012542] niks3-server[1127]: 2026/09/21 09:15:54 INFO Received sign narinfos request method=POST path=/api/pending_closures/52/sign6228lb # [ 165.022781] niks3-server[1127]: 2026/09/21 09:15:54 INFO Signed narinfos id=52 count=16229lb # [ 165.028442] niks3-server[1127]: 2026/09/21 09:15:54 INFO Registered completed upload object_key=cnf7j5vjqyj799isbg26wgjnrlk208gi.ls6230lb # [ 165.036631] niks3-server[1127]: 2026/09/21 09:15:54 INFO Received complete upload request method=POST path=/api/pending_closures/52/complete6231lb # [ 165.043034] niks3-server[1127]: 2026/09/21 09:15:54 INFO Completed upload id=526232worker1 # [ 165.017740] nix-grpc-daemon[1749]: time=2026-09-21T09:15:54.613Z level=INFO msg="Upload complete. (498ms)"6233lb # [ 165.045829] niks3-server[1127]: 2026/09/21 09:15:54 INFO Registered completed upload object_key=cnf7j5vjqyj799isbg26wgjnrlk208gi.narinfo6234worker1 # [ 165.028385] nix-grpc-daemon[1690]: ts=2026-09-21T09:15:54Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36620 duration_s=0 paths=1 nar_bytes_in=6246235worker1 # [ 165.039259] nix-grpc-daemon[1690]: ts=2026-09-21T09:15:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:366206236worker1 # [ 165.046255] nix-daemon[843]: accepted connection from pid 1690, user nix-grpc-daemon (trusted)6237worker1 # [ 165.051155] nix-daemon[843]: reaped child process 1862, status = succeeded6238worker1 # [ 165.063071] nix-daemon[843]: accepted connection from pid 1690, user nix-grpc-daemon (trusted)6239worker1 # [ 166.713687] nix-daemon[843]: reaped child process 1876, status = succeeded6240worker1 # [ 166.891892] nix-daemon[843]: accepted connection from pid 1891, user nix-grpc-daemon (trusted)6241worker1 # [ 166.909461] nix-daemon[843]: reaped child process 1894, status = succeeded6242lb # [ 166.949199] niks3-server[1127]: 2026/09/21 09:15:56 INFO Received uploads request method=POST path=/api/pending_closures6243worker1 # [ 166.936225] nix-grpc-daemon[1749]: time=2026-09-21T09:15:56.529Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6244worker1 # [ 166.937784] nix-grpc-daemon[1749]: time=2026-09-21T09:15:56.529Z level=INFO msg="Uploading p1334cl91hzsgv8bmiqzkwvhbwq24vn1-farm-b (176B)"6245lb # [ 166.984340] niks3-server[1127]: 2026/09/21 09:15:56 INFO Registered completed upload object_key=nar/0pa7j07n0n5p9wcfihhx3357xrxs4h8s57kncplymh51lvzvz2kh.nar.zst6246worker1 # [ 166.975622] nix-grpc-daemon[1749]: time=2026-09-21T09:15:56.570Z level=INFO msg="Uploading 1 narinfos"6247lb # [ 167.000472] niks3-server[1127]: 2026/09/21 09:15:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/53/sign6248lb # [ 167.013592] niks3-server[1127]: 2026/09/21 09:15:56 INFO Signed narinfos id=53 count=16249lb # [ 167.022051] niks3-server[1127]: 2026/09/21 09:15:56 INFO Registered completed upload object_key=p1334cl91hzsgv8bmiqzkwvhbwq24vn1.ls6250lb # [ 167.028808] niks3-server[1127]: 2026/09/21 09:15:56 INFO Registered completed upload object_key=log/cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv6251lb # [ 167.037476] niks3-server[1127]: 2026/09/21 09:15:56 INFO Received complete upload request method=POST path=/api/pending_closures/53/complete6252lb # [ 167.042658] niks3-server[1127]: 2026/09/21 09:15:56 INFO Completed upload id=536253worker1 # [ 167.017182] nix-grpc-daemon[1749]: time=2026-09-21T09:15:56.612Z level=INFO msg="Upload complete. (340ms)"6254lb # [ 167.045509] niks3-server[1127]: 2026/09/21 09:15:56 INFO Registered completed upload object_key=p1334cl91hzsgv8bmiqzkwvhbwq24vn1.narinfo6255worker2 # [ 167.033866] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:56Z level=debug event=assigned drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv worker=192.168.1.3:50051 clients=16256worker1 # [ 167.029112] nix-grpc-daemon[1690]: ts=2026-09-21T09:15:56Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36620 duration_s=1 drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv assign_id=10 outputs=16257worker2 # [ 167.046987] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:56Z level=debug event=want client=5 drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv cached=06258worker1 # [ 167.046801] nix-daemon[843]: reaped child process 1874, status = succeeded6259worker1 # [ 167.086062] nix-grpc-daemon[1690]: ts=2026-09-21T09:15:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:366206260worker1 # [ 167.107196] nix-grpc-daemon[1690]: ts=2026-09-21T09:15:56Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:366206261worker1 # [ 167.122608] nix-grpc-daemon[1749]: time=2026-09-21T09:15:56.718Z level=INFO msg="All 1 paths already cached"6262worker1 # [ 167.131460] nix-grpc-daemon[1690]: ts=2026-09-21T09:15:56Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:366206263worker1 # [ 167.148121] nix-daemon[843]: accepted connection from pid 1690, user nix-grpc-daemon (trusted)6264worker1 # [ 167.355423] nix-daemon[843]: accepted connection from pid 1898, user nix-grpc-daemon (trusted)6265worker1 # [ 167.373896] nix-daemon[843]: reaped child process 1901, status = succeeded6266worker1 # [ 167.551516] nix-daemon[843]: accepted connection from pid 1903, user nix-grpc-daemon (trusted)6267worker1 # [ 167.567737] nix-daemon[843]: reaped child process 1906, status = succeeded6268lb # [ 167.607646] niks3-server[1127]: 2026/09/21 09:15:57 INFO Received uploads request method=POST path=/api/pending_closures6269worker1 # [ 167.592172] nix-grpc-daemon[1749]: time=2026-09-21T09:15:57.187Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6270worker1 # [ 167.595026] nix-grpc-daemon[1749]: time=2026-09-21T09:15:57.187Z level=INFO msg="Uploading 7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv (624B)"6271lb # [ 167.639032] niks3-server[1127]: 2026/09/21 09:15:57 INFO Registered completed upload object_key=nar/0l0zss9ls4bhz18slrrmz7l1j93xz7424q7314p64rpgrpqf6b7k.nar.zst6272lb # [ 167.652426] niks3-server[1127]: 2026/09/21 09:15:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/54/sign6273worker1 # [ 167.633384] nix-grpc-daemon[1749]: time=2026-09-21T09:15:57.228Z level=INFO msg="Uploading 1 narinfos"6274lb # [ 167.663034] niks3-server[1127]: 2026/09/21 09:15:57 INFO Signed narinfos id=54 count=16275lb # [ 167.669307] niks3-server[1127]: 2026/09/21 09:15:57 INFO Registered completed upload object_key=7vjzkx8444zm2h7882vsj6bmj73zqjq3.ls6276lb # [ 167.675958] niks3-server[1127]: 2026/09/21 09:15:57 INFO Received complete upload request method=POST path=/api/pending_closures/54/complete6277worker1 # [ 167.656868] nix-grpc-daemon[1749]: time=2026-09-21T09:15:57.252Z level=INFO msg="Upload complete. (492ms)"6278lb # [ 167.682942] niks3-server[1127]: 2026/09/21 09:15:57 INFO Completed upload id=546279lb # [ 167.685569] niks3-server[1127]: 2026/09/21 09:15:57 INFO Registered completed upload object_key=7vjzkx8444zm2h7882vsj6bmj73zqjq3.narinfo6280worker1 # [ 167.664136] nix-grpc-daemon[1690]: ts=2026-09-21T09:15:57Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36620 duration_s=0 paths=1 nar_bytes_in=6246281worker1 # [ 167.677063] nix-grpc-daemon[1690]: ts=2026-09-21T09:15:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:366206282worker1 # [ 167.684641] nix-daemon[843]: accepted connection from pid 1690, user nix-grpc-daemon (trusted)6283worker1 # [ 167.689150] nix-daemon[843]: reaped child process 1896, status = succeeded6284worker1 # [ 167.702109] nix-daemon[843]: accepted connection from pid 1690, user nix-grpc-daemon (trusted)6285worker1 # [ 168.326263] nix-daemon[843]: reaped child process 1910, status = succeeded6286worker1 # [ 168.484059] nix-daemon[843]: accepted connection from pid 1925, user nix-grpc-daemon (trusted)6287worker1 # [ 168.499782] nix-daemon[843]: reaped child process 1928, status = succeeded6288lb # [ 168.540879] niks3-server[1127]: 2026/09/21 09:15:58 INFO Received uploads request method=POST path=/api/pending_closures6289worker1 # [ 168.527839] nix-grpc-daemon[1749]: time=2026-09-21T09:15:58.120Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6290worker1 # [ 168.529540] nix-grpc-daemon[1749]: time=2026-09-21T09:15:58.120Z level=INFO msg="Uploading nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8-farm-b (176B)"6291lb # [ 168.576945] niks3-server[1127]: 2026/09/21 09:15:58 INFO Registered completed upload object_key=nar/0kqa5m59sp6aj1ixqj6q54wwqxkcn98fw1rc9hfkcdk460wjdl0l.nar.zst6292worker1 # [ 168.565137] nix-grpc-daemon[1749]: time=2026-09-21T09:15:58.160Z level=INFO msg="Uploading 1 narinfos"6293lb # [ 168.590372] niks3-server[1127]: 2026/09/21 09:15:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/55/sign6294lb # [ 168.599145] niks3-server[1127]: 2026/09/21 09:15:58 INFO Signed narinfos id=55 count=16295lb # [ 168.606535] niks3-server[1127]: 2026/09/21 09:15:58 INFO Registered completed upload object_key=log/7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv6296lb # [ 168.616205] niks3-server[1127]: 2026/09/21 09:15:58 INFO Registered completed upload object_key=nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8.ls6297worker1 # [ 168.598294] nix-grpc-daemon[1749]: time=2026-09-21T09:15:58.193Z level=INFO msg="Upload complete. (287ms)"6298lb # [ 168.625212] niks3-server[1127]: 2026/09/21 09:15:58 INFO Received complete upload request method=POST path=/api/pending_closures/55/complete6299lb # [ 168.631371] niks3-server[1127]: 2026/09/21 09:15:58 INFO Completed upload id=556300worker2 # [ 168.614327] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:58Z level=debug event=assigned drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv worker=192.168.1.3:50051 clients=16301lb # [ 168.634882] niks3-server[1127]: 2026/09/21 09:15:58 INFO Registered completed upload object_key=nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8.narinfo6302worker1 # [ 168.608974] nix-grpc-daemon[1690]: ts=2026-09-21T09:15:58Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36620 duration_s=0 drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv assign_id=11 outputs=16303worker2 # [ 168.627038] nix-grpc-daemon[1386]: ts=2026-09-21T09:15:58Z level=debug event=want client=5 drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv cached=06304worker1 # [ 168.623652] nix-grpc-daemon[1690]: ts=2026-09-21T09:15:58Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:366206305worker1 # [ 168.629332] nix-daemon[843]: reaped child process 1908, status = succeeded6306worker1 # [ 168.636232] nix-grpc-daemon[1690]: ts=2026-09-21T09:15:58Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:366206307worker1 # [ 168.648972] nix-grpc-daemon[1749]: time=2026-09-21T09:15:58.244Z level=INFO msg="All 1 paths already cached"6308worker1 # [ 168.656399] nix-grpc-daemon[1690]: ts=2026-09-21T09:15:58Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:366206309worker1 # [ 168.666274] nix-daemon[843]: accepted connection from pid 1690, user nix-grpc-daemon (trusted)6310worker1 # [ 168.850225] nix-daemon[843]: accepted connection from pid 1932, user nix-grpc-daemon (trusted)6311worker1 # [ 168.869570] nix-daemon[843]: reaped child process 1935, status = succeeded6312worker1 # [ 169.043188] nix-daemon[843]: accepted connection from pid 1937, user nix-grpc-daemon (trusted)6313worker1 # [ 169.059861] nix-daemon[843]: reaped child process 1940, status = succeeded6314lb # [ 169.096484] niks3-server[1127]: 2026/09/21 09:15:58 INFO Received uploads request method=POST path=/api/pending_closures6315worker1 # [ 169.081933] nix-grpc-daemon[1749]: time=2026-09-21T09:15:58.677Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6316worker1 # [ 169.083401] nix-grpc-daemon[1749]: time=2026-09-21T09:15:58.677Z level=INFO msg="Uploading cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv (624B)"6317lb # [ 169.130071] niks3-server[1127]: 2026/09/21 09:15:58 INFO Registered completed upload object_key=nar/0f6g0vggy0qbqsj64l1nivyqdh6q12kh8620ysnf210ani0727vf.nar.zst6318lb # [ 169.139852] niks3-server[1127]: 2026/09/21 09:15:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/56/sign6319worker1 # [ 169.120925] nix-grpc-daemon[1749]: time=2026-09-21T09:15:58.716Z level=INFO msg="Uploading 1 narinfos"6320lb # [ 169.151765] niks3-server[1127]: 2026/09/21 09:15:58 INFO Signed narinfos id=56 count=16321lb # [ 169.157600] niks3-server[1127]: 2026/09/21 09:15:58 INFO Registered completed upload object_key=cgwyij19i3fl7sqlyz54vzfr4zajg077.ls6322worker1 # [ 169.140951] nix-grpc-daemon[1749]: time=2026-09-21T09:15:58.736Z level=INFO msg="Upload complete. (462ms)"6323lb # [ 169.166716] niks3-server[1127]: 2026/09/21 09:15:58 INFO Received complete upload request method=POST path=/api/pending_closures/56/complete6324lb # [ 169.171740] niks3-server[1127]: 2026/09/21 09:15:58 INFO Completed upload id=566325worker1 # [ 169.148252] nix-grpc-daemon[1690]: ts=2026-09-21T09:15:58Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36620 duration_s=0 paths=1 nar_bytes_in=6246326lb # [ 169.174669] niks3-server[1127]: 2026/09/21 09:15:58 INFO Registered completed upload object_key=cgwyij19i3fl7sqlyz54vzfr4zajg077.narinfo6327worker1 # [ 169.159616] nix-grpc-daemon[1690]: ts=2026-09-21T09:15:58Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:366206328worker1 # [ 169.168296] nix-daemon[843]: accepted connection from pid 1690, user nix-grpc-daemon (trusted)6329worker1 # [ 169.171125] nix-daemon[843]: reaped child process 1930, status = succeeded6330worker1 # [ 169.181221] nix-daemon[843]: accepted connection from pid 1690, user nix-grpc-daemon (trusted)6331worker1 # [ 170.202465] nix-daemon[843]: reaped child process 1944, status = succeeded6332worker1 # [ 170.359092] nix-daemon[843]: accepted connection from pid 1959, user nix-grpc-daemon (trusted)6333worker1 # [ 170.377232] nix-daemon[843]: reaped child process 1962, status = succeeded6334lb # [ 170.417876] niks3-server[1127]: 2026/09/21 09:16:00 INFO Received uploads request method=POST path=/api/pending_closures6335worker1 # [ 170.401488] nix-grpc-daemon[1749]: time=2026-09-21T09:15:59.996Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6336worker1 # [ 170.402989] nix-grpc-daemon[1749]: time=2026-09-21T09:15:59.996Z level=INFO msg="Uploading hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5-farm-b (176B)"6337lb # [ 170.450724] niks3-server[1127]: 2026/09/21 09:16:00 INFO Registered completed upload object_key=log/cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv6338lb # [ 170.461961] niks3-server[1127]: 2026/09/21 09:16:00 INFO Registered completed upload object_key=nar/0x964sxcxsdh3nfjarc1ib52kr50kl8zxl7h34jq5ydx763drras.nar.zst6339lb # [ 170.477238] niks3-server[1127]: 2026/09/21 09:16:00 INFO Received sign narinfos request method=POST path=/api/pending_closures/57/sign6340lb # [ 170.482951] niks3-server[1127]: 2026/09/21 09:16:00 INFO Signed narinfos id=57 count=16341worker1 # [ 170.457806] nix-grpc-daemon[1749]: time=2026-09-21T09:16:00.053Z level=INFO msg="Uploading 1 narinfos"6342lb # [ 170.487869] niks3-server[1127]: 2026/09/21 09:16:00 INFO Registered completed upload object_key=hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5.ls6343lb # [ 170.499449] niks3-server[1127]: 2026/09/21 09:16:00 INFO Received complete upload request method=POST path=/api/pending_closures/57/complete6344lb # [ 170.508115] niks3-server[1127]: 2026/09/21 09:16:00 INFO Completed upload id=576345worker1 # [ 170.482750] nix-grpc-daemon[1749]: time=2026-09-21T09:16:00.078Z level=INFO msg="Upload complete. (290ms)"6346lb # [ 170.511141] niks3-server[1127]: 2026/09/21 09:16:00 INFO Registered completed upload object_key=hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5.narinfo6347worker2 # [ 170.499691] nix-grpc-daemon[1386]: ts=2026-09-21T09:16:00Z level=debug event=assigned drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv worker=192.168.1.3:50051 clients=16348worker1 # [ 170.493075] nix-grpc-daemon[1690]: ts=2026-09-21T09:16:00Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36620 duration_s=1 drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv assign_id=12 outputs=16349worker2 # [ 170.512334] nix-grpc-daemon[1386]: ts=2026-09-21T09:16:00Z level=debug event=want client=5 drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv cached=06350worker1 # [ 170.510748] nix-daemon[843]: reaped child process 1942, status = succeeded6351worker1 # [ 170.546980] nix-grpc-daemon[1690]: ts=2026-09-21T09:16:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:366206352worker1 # [ 170.559195] nix-grpc-daemon[1690]: ts=2026-09-21T09:16:00Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:366206353worker1 # [ 170.573407] nix-grpc-daemon[1749]: time=2026-09-21T09:16:00.169Z level=INFO msg="All 2 paths already cached"6354worker1 # [ 170.582922] nix-grpc-daemon[1690]: ts=2026-09-21T09:16:00Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:366206355worker1 # [ 170.597472] nix-daemon[843]: accepted connection from pid 1690, user nix-grpc-daemon (trusted)6356worker1 # [ 170.801016] nix-daemon[843]: accepted connection from pid 1966, user nix-grpc-daemon (trusted)6357worker1 # [ 170.818377] nix-daemon[843]: reaped child process 1969, status = succeeded6358worker1 # [ 170.996845] nix-daemon[843]: accepted connection from pid 1971, user nix-grpc-daemon (trusted)6359worker1 # [ 171.014385] nix-daemon[843]: reaped child process 1974, status = succeeded6360lb # [ 171.051080] niks3-server[1127]: 2026/09/21 09:16:00 INFO Received uploads request method=POST path=/api/pending_closures6361worker1 # [ 171.034602] nix-grpc-daemon[1749]: time=2026-09-21T09:16:00.630Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6362worker1 # [ 171.036254] nix-grpc-daemon[1749]: time=2026-09-21T09:16:00.630Z level=INFO msg="Uploading dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv (752B)"6363lb # [ 171.076610] niks3-server[1127]: 2026/09/21 09:16:00 INFO Registered completed upload object_key=nar/0s7ff7rwqx5p9mdzddqgzd9bbd5j8njfc62k07mgnpy19810bs86.nar.zst6364lb # [ 171.091160] niks3-server[1127]: 2026/09/21 09:16:00 INFO Received sign narinfos request method=POST path=/api/pending_closures/58/sign6365worker1 # [ 171.072399] nix-grpc-daemon[1749]: time=2026-09-21T09:16:00.667Z level=INFO msg="Uploading 1 narinfos"6366lb # [ 171.103770] niks3-server[1127]: 2026/09/21 09:16:00 INFO Signed narinfos id=58 count=16367lb # [ 171.109254] niks3-server[1127]: 2026/09/21 09:16:00 INFO Registered completed upload object_key=dca2k4dvnk52m7gijqfq8q99ih3ni8ny.ls6368lb # [ 171.117056] niks3-server[1127]: 2026/09/21 09:16:00 INFO Received complete upload request method=POST path=/api/pending_closures/58/complete6369worker1 # [ 171.097092] nix-grpc-daemon[1749]: time=2026-09-21T09:16:00.692Z level=INFO msg="Upload complete. (483ms)"6370lb # [ 171.122738] niks3-server[1127]: 2026/09/21 09:16:00 INFO Completed upload id=586371lb # [ 171.126776] niks3-server[1127]: 2026/09/21 09:16:00 INFO Registered completed upload object_key=dca2k4dvnk52m7gijqfq8q99ih3ni8ny.narinfo6372worker1 # [ 171.104107] nix-grpc-daemon[1690]: ts=2026-09-21T09:16:00Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36620 duration_s=0 paths=1 nar_bytes_in=7526373worker1 # [ 171.115438] nix-grpc-daemon[1690]: ts=2026-09-21T09:16:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:366206374worker1 # [ 171.122217] nix-daemon[843]: accepted connection from pid 1690, user nix-grpc-daemon (trusted)6375worker1 # [ 171.126984] nix-daemon[843]: reaped child process 1964, status = succeeded6376worker1 # [ 171.140156] nix-daemon[843]: accepted connection from pid 1690, user nix-grpc-daemon (trusted)6377worker1 # [ 172.186255] nix-daemon[843]: reaped child process 1978, status = succeeded6378worker1 # [ 172.368785] nix-daemon[843]: accepted connection from pid 1993, user nix-grpc-daemon (trusted)6379worker1 # [ 172.387817] nix-daemon[843]: reaped child process 1996, status = succeeded6380lb # [ 172.426030] niks3-server[1127]: 2026/09/21 09:16:02 INFO Received uploads request method=POST path=/api/pending_closures6381worker1 # [ 172.412398] nix-grpc-daemon[1749]: time=2026-09-21T09:16:02.005Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6382worker1 # [ 172.415258] nix-grpc-daemon[1749]: time=2026-09-21T09:16:02.005Z level=INFO msg="Uploading d88xgckxx0s0n5ly2d0jw1qy1641z8pq-farm-top (232B)"6383lb # [ 172.466810] niks3-server[1127]: 2026/09/21 09:16:02 INFO Registered completed upload object_key=nar/0knxhkhz3x04vk8as5wv5ar2d0h02prv7q5f42qm1xjgr70ancd0.nar.zst6384lb # [ 172.479372] niks3-server[1127]: 2026/09/21 09:16:02 INFO Registered completed upload object_key=log/dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv6385lb # [ 172.487424] niks3-server[1127]: 2026/09/21 09:16:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/59/sign6386worker1 # [ 172.468229] nix-grpc-daemon[1749]: time=2026-09-21T09:16:02.063Z level=INFO msg="Uploading 1 narinfos"6387lb # [ 172.496251] niks3-server[1127]: 2026/09/21 09:16:02 INFO Signed narinfos id=59 count=16388lb # [ 172.498997] niks3-server[1127]: 2026/09/21 09:16:02 INFO Registered completed upload object_key=d88xgckxx0s0n5ly2d0jw1qy1641z8pq.ls6389lb # [ 172.510172] niks3-server[1127]: 2026/09/21 09:16:02 INFO Received complete upload request method=POST path=/api/pending_closures/59/complete6390lb # [ 172.517615] niks3-server[1127]: 2026/09/21 09:16:02 INFO Completed upload id=596391worker1 # [ 172.492399] nix-grpc-daemon[1749]: time=2026-09-21T09:16:02.088Z level=INFO msg="Upload complete. (340ms)"6392lb # [ 172.520409] niks3-server[1127]: 2026/09/21 09:16:02 INFO Registered completed upload object_key=d88xgckxx0s0n5ly2d0jw1qy1641z8pq.narinfo6393worker2 # [ 172.508711] nix-grpc-daemon[1386]: ts=2026-09-21T09:16:02Z level=debug event=assigned drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv worker=192.168.1.3:50051 clients=16394worker1 # [ 172.505088] nix-grpc-daemon[1690]: ts=2026-09-21T09:16:02Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36620 duration_s=1 drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv assign_id=13 outputs=16395worker1 # [ 172.520653] nix-daemon[843]: reaped child process 1976, status = succeeded6396worker1 # [ 172.546998] nix-grpc-daemon[1690]: ts=2026-09-21T09:16:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:366206397worker1 # [ 172.561690] nix-grpc-daemon[1690]: ts=2026-09-21T09:16:02Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:366206398worker1 # [ 172.575288] nix-grpc-daemon[1749]: time=2026-09-21T09:16:02.170Z level=INFO msg="All 2 paths already cached"6399worker1 # [ 172.584301] nix-grpc-daemon[1690]: ts=2026-09-21T09:16:02Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:366206400worker1 # [ 172.598044] nix-daemon[843]: accepted connection from pid 1690, user nix-grpc-daemon (trusted)6401worker1 # [ 172.796533] nix-daemon[843]: accepted connection from pid 2000, user nix-grpc-daemon (trusted)6402worker1 # [ 172.816884] nix-daemon[843]: reaped child process 2003, status = succeeded6403worker1 # [ 172.996458] nix-daemon[843]: accepted connection from pid 2005, user nix-grpc-daemon (trusted)6404worker1 # [ 173.013035] nix-daemon[843]: reaped child process 2008, status = succeeded6405lb # [ 173.052153] niks3-server[1127]: 2026/09/21 09:16:02 INFO Received uploads request method=POST path=/api/pending_closures6406worker1 # [ 173.035465] nix-grpc-daemon[1749]: time=2026-09-21T09:16:02.630Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6407worker1 # [ 173.037303] nix-grpc-daemon[1749]: time=2026-09-21T09:16:02.630Z level=INFO msg="Uploading 2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv (752B)"6408lb # [ 173.080748] niks3-server[1127]: 2026/09/21 09:16:02 INFO Registered completed upload object_key=nar/0cpiyiv7frlhngqhhc703bglz5q2cpg4n8sgds4sfg7nvw0fcdn5.nar.zst6409lb # [ 173.092255] niks3-server[1127]: 2026/09/21 09:16:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/60/sign6410worker1 # [ 173.072947] nix-grpc-daemon[1749]: time=2026-09-21T09:16:02.668Z level=INFO msg="Uploading 1 narinfos"6411lb # [ 173.102078] niks3-server[1127]: 2026/09/21 09:16:02 INFO Signed narinfos id=60 count=16412lb # [ 173.108736] niks3-server[1127]: 2026/09/21 09:16:02 INFO Registered completed upload object_key=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9.ls6413lb # [ 173.116633] niks3-server[1127]: 2026/09/21 09:16:02 INFO Received complete upload request method=POST path=/api/pending_closures/60/complete6414lb # [ 173.122958] niks3-server[1127]: 2026/09/21 09:16:02 INFO Registered completed upload object_key=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9.narinfo6415lb # [ 173.128287] niks3-server[1127]: 2026/09/21 09:16:02 INFO Completed upload id=606416worker1 # [ 173.102583] nix-grpc-daemon[1749]: time=2026-09-21T09:16:02.698Z level=INFO msg="Upload complete. (489ms)"6417worker1 # [ 173.111319] nix-grpc-daemon[1690]: ts=2026-09-21T09:16:02Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36620 duration_s=0 paths=1 nar_bytes_in=7526418worker1 # [ 173.120945] nix-grpc-daemon[1690]: ts=2026-09-21T09:16:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:366206419worker1 # [ 173.131340] nix-daemon[843]: accepted connection from pid 1690, user nix-grpc-daemon (trusted)6420worker1 # [ 173.138120] nix-daemon[843]: reaped child process 1998, status = succeeded6421worker1 # [ 173.148186] nix-daemon[843]: accepted connection from pid 1690, user nix-grpc-daemon (trusted)6422worker1 # [ 173.977235] nix-daemon[843]: reaped child process 2012, status = succeeded6423worker1 # [ 174.144147] nix-daemon[843]: accepted connection from pid 2027, user nix-grpc-daemon (trusted)6424worker1 # [ 174.163877] nix-daemon[843]: reaped child process 2030, status = succeeded6425lb # [ 174.202422] niks3-server[1127]: 2026/09/21 09:16:03 INFO Received uploads request method=POST path=/api/pending_closures6426worker1 # [ 174.187607] nix-grpc-daemon[1749]: time=2026-09-21T09:16:03.782Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6427worker1 # [ 174.189245] nix-grpc-daemon[1749]: time=2026-09-21T09:16:03.782Z level=INFO msg="Uploading vz84kyfcw804d0iyx7nj61j47ms9zxmn-farm-top (232B)"6428lb # [ 174.240691] niks3-server[1127]: 2026/09/21 09:16:03 INFO Registered completed upload object_key=log/2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv6429lb # [ 174.251826] niks3-server[1127]: 2026/09/21 09:16:03 INFO Registered completed upload object_key=nar/0aaz4iim7fa45zws98gbhp73l23kbm0fvp4anssnxa96v666ljhs.nar.zst6430worker1 # [ 174.237220] nix-grpc-daemon[1749]: time=2026-09-21T09:16:03.832Z level=INFO msg="Uploading 1 narinfos"6431lb # [ 174.264171] niks3-server[1127]: 2026/09/21 09:16:03 INFO Received sign narinfos request method=POST path=/api/pending_closures/61/sign6432lb # [ 174.267474] niks3-server[1127]: 2026/09/21 09:16:03 INFO Signed narinfos id=61 count=16433lb # [ 174.274212] niks3-server[1127]: 2026/09/21 09:16:03 INFO Registered completed upload object_key=vz84kyfcw804d0iyx7nj61j47ms9zxmn.ls6434lb # [ 174.280393] niks3-server[1127]: 2026/09/21 09:16:03 INFO Received complete upload request method=POST path=/api/pending_closures/61/complete6435lb # [ 174.284337] niks3-server[1127]: 2026/09/21 09:16:03 INFO Registered completed upload object_key=vz84kyfcw804d0iyx7nj61j47ms9zxmn.narinfo6436lb # [ 174.288061] niks3-server[1127]: 2026/09/21 09:16:03 INFO Completed upload id=616437worker1 # [ 174.262780] nix-grpc-daemon[1749]: time=2026-09-21T09:16:03.858Z level=INFO msg="Upload complete. (305ms)"6438worker2 # [ 174.277104] nix-grpc-daemon[1386]: ts=2026-09-21T09:16:03Z level=debug event=assigned drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv worker=192.168.1.3:50051 clients=16439worker1 # [ 174.273098] nix-grpc-daemon[1690]: ts=2026-09-21T09:16:03Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36620 duration_s=1 drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv assign_id=14 outputs=16440worker1 # [ 174.287832] nix-grpc-daemon[1690]: ts=2026-09-21T09:16:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:366206441worker1 # [ 174.292519] nix-daemon[843]: reaped child process 2010, status = succeeded6442worker1 # [ 174.302253] nix-grpc-daemon[1690]: ts=2026-09-21T09:16:03Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:366206443worker1 # [ 174.313884] nix-grpc-daemon[1749]: time=2026-09-21T09:16:03.909Z level=INFO msg="All 2 paths already cached"6444worker1 # [ 174.321806] nix-grpc-daemon[1690]: ts=2026-09-21T09:16:03Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:366206445worker1 # [ 174.333862] nix-daemon[843]: accepted connection from pid 1690, user nix-grpc-daemon (trusted)6446worker1 # [ 174.524103] nix-daemon[843]: accepted connection from pid 2034, user nix-grpc-daemon (trusted)6447worker1 # [ 174.542304] nix-daemon[843]: reaped child process 2037, status = succeeded6448worker1 # [ 174.718868] nix-daemon[843]: accepted connection from pid 2039, user nix-grpc-daemon (trusted)6449worker1 # [ 174.735050] nix-daemon[843]: reaped child process 2042, status = succeeded6450lb # [ 174.772899] niks3-server[1127]: 2026/09/21 09:16:04 INFO Received uploads request method=POST path=/api/pending_closures6451worker1 # [ 174.759074] nix-grpc-daemon[1749]: time=2026-09-21T09:16:04.354Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6452worker1 # [ 174.760643] nix-grpc-daemon[1749]: time=2026-09-21T09:16:04.354Z level=INFO msg="Uploading qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv (752B)"6453lb # [ 174.804445] niks3-server[1127]: 2026/09/21 09:16:04 INFO Registered completed upload object_key=nar/05zmsnqx4aj55yncx917mz6d3hzrvz2rx2x75313vvy1kq1zv9fv.nar.zst6454worker1 # [ 174.789510] nix-grpc-daemon[1749]: time=2026-09-21T09:16:04.385Z level=INFO msg="Uploading 1 narinfos"6455lb # [ 174.820418] niks3-server[1127]: 2026/09/21 09:16:04 INFO Received sign narinfos request method=POST path=/api/pending_closures/62/sign6456lb # [ 174.830881] niks3-server[1127]: 2026/09/21 09:16:04 INFO Signed narinfos id=62 count=16457lb # [ 174.834887] niks3-server[1127]: 2026/09/21 09:16:04 INFO Registered completed upload object_key=qx10pa00pilyp5xdzd48l88d5jz143mr.ls6458lb # [ 174.843295] niks3-server[1127]: 2026/09/21 09:16:04 INFO Received complete upload request method=POST path=/api/pending_closures/62/complete6459lb # [ 174.848257] niks3-server[1127]: 2026/09/21 09:16:04 INFO Completed upload id=626460worker1 # [ 174.822694] nix-grpc-daemon[1749]: time=2026-09-21T09:16:04.418Z level=INFO msg="Upload complete. (476ms)"6461lb # [ 174.851027] niks3-server[1127]: 2026/09/21 09:16:04 INFO Registered completed upload object_key=qx10pa00pilyp5xdzd48l88d5jz143mr.narinfo6462worker1 # [ 174.832240] nix-grpc-daemon[1690]: ts=2026-09-21T09:16:04Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36620 duration_s=0 paths=1 nar_bytes_in=7526463worker1 # [ 174.843590] nix-grpc-daemon[1690]: ts=2026-09-21T09:16:04Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:366206464worker1 # [ 174.854047] nix-daemon[843]: accepted connection from pid 1690, user nix-grpc-daemon (trusted)6465worker1 # [ 174.857152] nix-daemon[843]: reaped child process 2032, status = succeeded6466worker1 # [ 174.871289] nix-daemon[843]: accepted connection from pid 1690, user nix-grpc-daemon (trusted)6467worker1 # [ 175.491632] nix-daemon[843]: reaped child process 2046, status = succeeded6468worker1 # [ 175.649668] nix-daemon[843]: accepted connection from pid 2061, user nix-grpc-daemon (trusted)6469worker1 # [ 175.667792] nix-daemon[843]: reaped child process 2064, status = succeeded6470lb # [ 175.702371] niks3-server[1127]: 2026/09/21 09:16:05 INFO Received uploads request method=POST path=/api/pending_closures6471worker1 # [ 175.684443] nix-grpc-daemon[1749]: time=2026-09-21T09:16:05.278Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6472worker1 # [ 175.685922] nix-grpc-daemon[1749]: time=2026-09-21T09:16:05.278Z level=INFO msg="Uploading iagwbb072hnm79qb9wsrjp5907zc7aa2-farm-top (232B)"6473lb # [ 175.739154] niks3-server[1127]: 2026/09/21 09:16:05 INFO Registered completed upload object_key=nar/053jd22j3kv7v6j0q7cspdpc0h1c1xmywmajl6sd5hzwg50z2yj1.nar.zst6474worker1 # [ 175.728819] nix-grpc-daemon[1749]: time=2026-09-21T09:16:05.324Z level=INFO msg="Uploading 1 narinfos"6475lb # [ 175.754525] niks3-server[1127]: 2026/09/21 09:16:05 INFO Received sign narinfos request method=POST path=/api/pending_closures/63/sign6476lb # [ 175.762512] niks3-server[1127]: 2026/09/21 09:16:05 INFO Signed narinfos id=63 count=16477lb # [ 175.767900] niks3-server[1127]: 2026/09/21 09:16:05 INFO Registered completed upload object_key=log/qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv6478lb # [ 175.773898] niks3-server[1127]: 2026/09/21 09:16:05 INFO Registered completed upload object_key=iagwbb072hnm79qb9wsrjp5907zc7aa2.ls6479worker1 # [ 175.756298] nix-grpc-daemon[1749]: time=2026-09-21T09:16:05.351Z level=INFO msg="Upload complete. (276ms)"6480lb # [ 175.782562] niks3-server[1127]: 2026/09/21 09:16:05 INFO Received complete upload request method=POST path=/api/pending_closures/63/complete6481lb # [ 175.785288] niks3-server[1127]: 2026/09/21 09:16:05 INFO Completed upload id=636482lb # [ 175.788045] niks3-server[1127]: 2026/09/21 09:16:05 INFO Registered completed upload object_key=iagwbb072hnm79qb9wsrjp5907zc7aa2.narinfo6483worker1 # [ 175.764871] nix-grpc-daemon[1690]: ts=2026-09-21T09:16:05Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36620 duration_s=0 drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv assign_id=15 outputs=16484worker1 # [ 175.779167] nix-daemon[843]: reaped child process 2044, status = succeeded6485client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&1, in 19.19 seconds)6486worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6487worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6488client: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag f5 --arg features '["gpu"]'6489client # [ 175.984635] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build -L --store "grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&restart-grace=30" --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag f5 --arg features "[\"gpu\"]".6490client # Running as unit: gpu.service; invocation ID: 6b56d758890d468884c8d1908abfc5996491client: (finished: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag f5 --arg features '["gpu"]', in 0.09 seconds)6492lb: must succeed: curl -sf localhost:9901/clusters6493lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds)6494lb: must succeed: curl -sf localhost:9901/clusters6495lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)6496worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6497worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)6498client # [ 176.243583] nix[1338]: don't know how to build these paths:6499client # [ 176.245127] nix[1338]: /nix/store/3i1nziwpdqq19lzpqy4rywzxnm8dlps6-farm-top.drv6500worker1 # [ 176.244070] nix-grpc-daemon[1690]: ts=2026-09-21T09:16:05Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:366206501worker2 # [ 176.257757] nix-grpc-daemon[1386]: ts=2026-09-21T09:16:05Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:466166502client # [ 176.275655] nix[1338]: warning: 1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv: no connected, non-draining worker for system 'aarch64-linux' offers features {gpu}, waiting up to 30s for a worker6503worker2 # [ 176.283808] nix-grpc-daemon[1386]: ts=2026-09-21T09:16:05Z level=debug event=want client=6 drv=1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv cached=06504lb: must succeed: curl -sf localhost:9901/clusters6505lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6506lb: must succeed: curl -sf localhost:9901/clusters6507lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)6508worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6509worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6510client: waiting for success: systemctl show -p Result gpu | grep -q exit-code6511lb # [ 180.978789] envoy[913]: conn peer=[2001:db8:1::1]:35510 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=27161 tx=10757 ms=188646512client # [ 206.277255] nix[1338]: 1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv: no connected, non-draining worker for system 'aarch64-linux' offers features {gpu}6513client # [ 206.287100] nix[1338]: error: dependency '1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv' failed6514client # [ 206.326931] systemd[1]: gpu.service: Main process exited, code=exited, status=1/FAILURE6515client # [ 206.337312] systemd[1]: gpu.service: Failed with result 'exit-code'.6516client # [ 206.340763] systemd[1]: gpu.service: Consumed 175ms CPU time over 30.351s wall clock time, 35.4M memory peak, 5.1K incoming IP traffic, 5K outgoing IP traffic.6517client: (finished: waiting for success: systemctl show -p Result gpu | grep -q exit-code, in 29.27 seconds)6518client: must succeed: journalctl -u gpu -o cat6519client: (finished: must succeed: journalctl -u gpu -o cat, in 0.03 seconds)6520lb: must succeed: curl -sf localhost:9901/clusters6521lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6522lb: must succeed: curl -sf localhost:9901/clusters6523lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6524worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6525worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6526(finished: subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does, in 50.32 seconds)6527subtest: low disk drains a worker and builds go to the other6528worker1: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill6529worker1: (finished: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill, in 1.09 seconds)6530worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free'6531worker1 # [ 208.334665] nix-grpc-daemon[1690]: ts=2026-09-21T09:16:37Z level=info event=unhealthy reason=min_free6532worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free', in 1.13 seconds)6533client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag drain >&26534client # don't know how to build these paths:6535client # /nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv6536worker2 # [ 209.473141] nix-grpc-daemon[1386]: ts=2026-09-21T09:16:39Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:466166537worker2 # [ 209.480794] nix-grpc-daemon[1386]: ts=2026-09-21T09:16:39Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:466166538worker2 # [ 209.504148] nix-grpc-daemon[1386]: ts=2026-09-21T09:16:39Z level=debug event=want client=7 drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv cached=06539worker2 # [ 209.511056] nix-grpc-daemon[1386]: ts=2026-09-21T09:16:39Z level=debug event=assigned drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv worker=192.168.1.4:50051 clients=16540worker2 # [ 209.549543] nix-grpc-daemon[1386]: ts=2026-09-21T09:16:39Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:466166541worker2 # [ 209.565644] nix-grpc-daemon[1386]: ts=2026-09-21T09:16:39Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:466166542client # copying 1 paths to 'lb:50051'...6543client # copying path '/nix/store/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv' to 'lb:50051'...6544worker2 # [ 209.581522] nix-grpc-daemon[1386]: ts=2026-09-21T09:16:39Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:466166545worker2 # [ 209.599413] nix-daemon[870]: accepted connection from pid 1386, user nix-grpc-daemon (trusted)6546worker2 # [ 209.808155] nix-daemon[870]: accepted connection from pid 1676, user nix-grpc-daemon (trusted)6547worker2 # [ 209.825732] nix-daemon[870]: reaped child process 1679, status = succeeded6548worker2 # [ 210.002328] nix-daemon[870]: accepted connection from pid 1681, user nix-grpc-daemon (trusted)6549worker2 # [ 210.018470] nix-daemon[870]: reaped child process 1684, status = succeeded6550lb # [ 210.050129] niks3-server[1127]: 2026/09/21 09:16:39 INFO Received uploads request method=POST path=/api/pending_closures6551worker2 # [ 210.044182] nix-grpc-daemon[1461]: time=2026-09-21T09:16:39.626Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6552worker2 # [ 210.045652] nix-grpc-daemon[1461]: time=2026-09-21T09:16:39.626Z level=INFO msg="Uploading kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv (512B)"6553lb # [ 210.082427] niks3-server[1127]: 2026/09/21 09:16:39 INFO Registered completed upload object_key=nar/1sfy0gcq4znfcj1f68c749r5x2jwmwsfbfp6xdflz5xkndjwjjm3.nar.zst6554worker2 # [ 210.078511] nix-grpc-daemon[1461]: time=2026-09-21T09:16:39.661Z level=INFO msg="Uploading 1 narinfos"6555lb # [ 210.097062] niks3-server[1127]: 2026/09/21 09:16:39 INFO Received sign narinfos request method=POST path=/api/pending_closures/64/sign6556lb # [ 210.106764] niks3-server[1127]: 2026/09/21 09:16:39 INFO Signed narinfos id=64 count=16557lb # [ 210.114675] niks3-server[1127]: 2026/09/21 09:16:39 INFO Registered completed upload object_key=kksl1jqfg56vabgqvrkz5vki6nq7r59r.ls6558lb # [ 210.124104] niks3-server[1127]: 2026/09/21 09:16:39 INFO Received complete upload request method=POST path=/api/pending_closures/64/complete6559lb # [ 210.130742] niks3-server[1127]: 2026/09/21 09:16:39 INFO Completed upload id=646560worker2 # [ 210.112323] nix-grpc-daemon[1461]: time=2026-09-21T09:16:39.695Z level=INFO msg="Upload complete. (495ms)"6561lb # [ 210.133792] niks3-server[1127]: 2026/09/21 09:16:39 INFO Registered completed upload object_key=kksl1jqfg56vabgqvrkz5vki6nq7r59r.narinfo6562worker2 # [ 210.120137] nix-grpc-daemon[1386]: ts=2026-09-21T09:16:39Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46616 duration_s=0 paths=1 nar_bytes_in=5126563worker2 # [ 210.134426] nix-grpc-daemon[1386]: ts=2026-09-21T09:16:39Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:466166564worker2 # [ 210.141105] nix-daemon[870]: accepted connection from pid 1386, user nix-grpc-daemon (trusted)6565worker2 # [ 210.146055] nix-daemon[870]: reaped child process 1674, status = succeeded6566client # building '/nix/store/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv' on lb:50051...6567client # farm-a> worker2: building kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv6568worker2 # [ 210.159444] nix-daemon[870]: accepted connection from pid 1386, user nix-grpc-daemon (trusted)6569lb # [ 210.977837] envoy[913]: conn peer=[2001:db8:1::1]:52566 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=897 tx=499 ms=300616570client # farm-a> LOG-farm-a-drain6571worker2 # [ 211.403152] nix-daemon[870]: reaped child process 1688, status = succeeded6572worker2 # [ 211.587084] nix-daemon[870]: accepted connection from pid 1703, user nix-grpc-daemon (trusted)6573worker2 # [ 211.604104] nix-daemon[870]: reaped child process 1706, status = succeeded6574lb # [ 211.637496] niks3-server[1127]: 2026/09/21 09:16:41 INFO Received uploads request method=POST path=/api/pending_closures6575worker2 # [ 211.628390] nix-grpc-daemon[1461]: time=2026-09-21T09:16:41.209Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6576worker2 # [ 211.629845] nix-grpc-daemon[1461]: time=2026-09-21T09:16:41.209Z level=INFO msg="Uploading 272b39wn7cbnhkcc2rrz2f6kmhhmcwnr-farm-a (128B)"6577lb # [ 211.671590] niks3-server[1127]: 2026/09/21 09:16:41 INFO Registered completed upload object_key=nar/1bf9z7y0ikx07b4zjag42zyrv2pq1ycw4461s579kfpmni9kz4dw.nar.zst6578lb # [ 211.689469] niks3-server[1127]: 2026/09/21 09:16:41 INFO Registered completed upload object_key=log/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv6579lb # [ 211.695108] niks3-server[1127]: 2026/09/21 09:16:41 INFO Received sign narinfos request method=POST path=/api/pending_closures/65/sign6580worker2 # [ 211.681526] nix-grpc-daemon[1461]: time=2026-09-21T09:16:41.263Z level=INFO msg="Uploading 1 narinfos"6581lb # [ 211.705465] niks3-server[1127]: 2026/09/21 09:16:41 INFO Signed narinfos id=65 count=16582lb # [ 211.707932] niks3-server[1127]: 2026/09/21 09:16:41 INFO Registered completed upload object_key=272b39wn7cbnhkcc2rrz2f6kmhhmcwnr.ls6583lb # [ 211.716838] niks3-server[1127]: 2026/09/21 09:16:41 INFO Received complete upload request method=POST path=/api/pending_closures/65/complete6584lb # [ 211.722382] niks3-server[1127]: 2026/09/21 09:16:41 INFO Completed upload id=656585worker2 # [ 211.703532] nix-grpc-daemon[1461]: time=2026-09-21T09:16:41.286Z level=INFO msg="Upload complete. (333ms)"6586worker2 # [ 211.719103] nix-grpc-daemon[1461]: time=2026-09-21T09:16:41.286Z level=WARN msg="Failed to register uploaded object" key=272b39wn7cbnhkcc2rrz2f6kmhhmcwnr.narinfo error="server returned 404: object is not pending upload\n"6587worker2 # [ 211.741110] nix-grpc-daemon[1386]: ts=2026-09-21T09:16:41Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46616 duration_s=1 drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv assign_id=16 outputs=16588worker2 # [ 211.748563] nix-grpc-daemon[1386]: ts=2026-09-21T09:16:41Z level=debug event=want client=7 drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv cached=06589worker2 # [ 211.752540] nix-grpc-daemon[1386]: ts=2026-09-21T09:16:41Z level=debug event=assigned drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv worker=192.168.1.4:50051 clients=16590worker2 # [ 211.757141] nix-daemon[870]: reaped child process 1686, status = succeeded6591worker2 # [ 211.778015] nix-grpc-daemon[1386]: ts=2026-09-21T09:16:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:466166592worker2 # [ 211.792887] nix-grpc-daemon[1386]: ts=2026-09-21T09:16:41Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:466166593worker2 # [ 211.806978] nix-grpc-daemon[1461]: time=2026-09-21T09:16:41.389Z level=INFO msg="All 1 paths already cached"6594client # copying 1 paths to 'lb:50051'...6595client # copying path '/nix/store/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv' to 'lb:50051'...6596worker2 # [ 211.816987] nix-grpc-daemon[1386]: ts=2026-09-21T09:16:41Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:466166597worker2 # [ 211.830237] nix-daemon[870]: accepted connection from pid 1386, user nix-grpc-daemon (trusted)6598worker2 # [ 212.037210] nix-daemon[870]: accepted connection from pid 1712, user nix-grpc-daemon (trusted)6599worker2 # [ 212.054974] nix-daemon[870]: reaped child process 1715, status = succeeded6600worker2 # [ 212.226495] nix-daemon[870]: accepted connection from pid 1717, user nix-grpc-daemon (trusted)6601worker2 # [ 212.242811] nix-daemon[870]: reaped child process 1720, status = succeeded6602lb # [ 212.272683] niks3-server[1127]: 2026/09/21 09:16:41 INFO Received uploads request method=POST path=/api/pending_closures6603worker2 # [ 212.262760] nix-grpc-daemon[1461]: time=2026-09-21T09:16:41.845Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6604worker2 # [ 212.264386] nix-grpc-daemon[1461]: time=2026-09-21T09:16:41.845Z level=INFO msg="Uploading fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv (632B)"6605lb # [ 212.299317] niks3-server[1127]: 2026/09/21 09:16:41 INFO Registered completed upload object_key=nar/1ax4z88sb8d24gsvydxvgdy2lijkmhnycxhr5war1j3sq97cyzqf.nar.zst6606lb # [ 212.313897] niks3-server[1127]: 2026/09/21 09:16:41 INFO Received sign narinfos request method=POST path=/api/pending_closures/66/sign6607worker2 # [ 212.300982] nix-grpc-daemon[1461]: time=2026-09-21T09:16:41.883Z level=INFO msg="Uploading 1 narinfos"6608lb # [ 212.324658] niks3-server[1127]: 2026/09/21 09:16:41 INFO Signed narinfos id=66 count=16609lb # [ 212.327622] niks3-server[1127]: 2026/09/21 09:16:41 INFO Registered completed upload object_key=fa7adcm0x8sbgmrsfx912fp8xakk99g6.ls6610lb # [ 212.337897] niks3-server[1127]: 2026/09/21 09:16:41 INFO Received complete upload request method=POST path=/api/pending_closures/66/complete6611lb # [ 212.344147] niks3-server[1127]: 2026/09/21 09:16:41 INFO Registered completed upload object_key=fa7adcm0x8sbgmrsfx912fp8xakk99g6.narinfo6612lb # [ 212.348218] niks3-server[1127]: 2026/09/21 09:16:41 INFO Completed upload id=666613worker2 # [ 212.329693] nix-grpc-daemon[1461]: time=2026-09-21T09:16:41.912Z level=INFO msg="Upload complete. (480ms)"6614worker2 # [ 212.340431] nix-grpc-daemon[1386]: ts=2026-09-21T09:16:41Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46616 duration_s=0 paths=1 nar_bytes_in=6326615worker2 # [ 212.351077] nix-grpc-daemon[1386]: ts=2026-09-21T09:16:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:466166616worker2 # [ 212.356960] nix-daemon[870]: accepted connection from pid 1386, user nix-grpc-daemon (trusted)6617worker2 # [ 212.361693] nix-daemon[870]: reaped child process 1710, status = succeeded6618client # building '/nix/store/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv' on lb:50051...6619client # farm-b> worker2: building fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv6620worker2 # [ 212.374012] nix-daemon[870]: accepted connection from pid 1386, user nix-grpc-daemon (trusted)6621client # farm-b> LOG-farm-b-drain6622worker2 # [ 213.617733] nix-daemon[870]: reaped child process 1724, status = succeeded6623worker2 # [ 213.801236] nix-daemon[870]: accepted connection from pid 1739, user nix-grpc-daemon (trusted)6624worker2 # [ 213.819620] nix-daemon[870]: reaped child process 1742, status = succeeded6625lb # [ 213.851711] niks3-server[1127]: 2026/09/21 09:16:43 INFO Received uploads request method=POST path=/api/pending_closures6626worker2 # [ 213.848457] nix-grpc-daemon[1461]: time=2026-09-21T09:16:43.427Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6627worker2 # [ 213.850169] nix-grpc-daemon[1461]: time=2026-09-21T09:16:43.427Z level=INFO msg="Uploading g774lks418zc45bd9hwhinm61pwcwnr9-farm-b (176B)"6628worker2 # [ 213.890803] nix-grpc-daemon[1461]: time=2026-09-21T09:16:43.473Z level=INFO msg="Uploading 1 narinfos"6629lb # [ 213.900747] niks3-server[1127]: 2026/09/21 09:16:43 INFO Registered completed upload object_key=nar/1ammzz4zx9sggy8v73zxk3j08vbqxgpw14n8lx2fh57bx24yzvnj.nar.zst6630lb # [ 213.912777] niks3-server[1127]: 2026/09/21 09:16:43 INFO Registered completed upload object_key=log/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv6631lb # [ 213.920495] niks3-server[1127]: 2026/09/21 09:16:43 INFO Received sign narinfos request method=POST path=/api/pending_closures/67/sign6632lb # [ 213.923462] niks3-server[1127]: 2026/09/21 09:16:43 INFO Signed narinfos id=67 count=16633lb # [ 213.926994] niks3-server[1127]: 2026/09/21 09:16:43 INFO Registered completed upload object_key=g774lks418zc45bd9hwhinm61pwcwnr9.ls6634worker2 # [ 213.915771] nix-grpc-daemon[1461]: time=2026-09-21T09:16:43.498Z level=INFO msg="Upload complete. (331ms)"6635lb # [ 213.934709] niks3-server[1127]: 2026/09/21 09:16:43 INFO Received complete upload request method=POST path=/api/pending_closures/67/complete6636lb # [ 213.938925] niks3-server[1127]: 2026/09/21 09:16:43 INFO Completed upload id=676637lb # [ 213.942861] niks3-server[1127]: 2026/09/21 09:16:43 INFO Registered completed upload object_key=g774lks418zc45bd9hwhinm61pwcwnr9.narinfo6638worker2 # [ 213.931025] nix-grpc-daemon[1386]: ts=2026-09-21T09:16:43Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46616 duration_s=1 drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv assign_id=17 outputs=16639worker2 # [ 213.948126] nix-grpc-daemon[1386]: ts=2026-09-21T09:16:43Z level=debug event=want client=7 drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv cached=06640worker2 # [ 213.951994] nix-grpc-daemon[1386]: ts=2026-09-21T09:16:43Z level=debug event=assigned drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv worker=192.168.1.4:50051 clients=16641worker2 # [ 213.956721] nix-daemon[870]: reaped child process 1722, status = succeeded6642worker2 # [ 213.989881] nix-grpc-daemon[1386]: ts=2026-09-21T09:16:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:466166643worker2 # [ 214.004757] nix-grpc-daemon[1386]: ts=2026-09-21T09:16:43Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:466166644worker2 # [ 214.018806] nix-grpc-daemon[1461]: time=2026-09-21T09:16:43.601Z level=INFO msg="All 2 paths already cached"6645client # copying 1 paths to 'lb:50051'...6646client # copying path '/nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv' to 'lb:50051'...6647worker2 # [ 214.028948] nix-grpc-daemon[1386]: ts=2026-09-21T09:16:43Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:466166648worker2 # [ 214.044457] nix-daemon[870]: accepted connection from pid 1386, user nix-grpc-daemon (trusted)6649worker2 # [ 214.244803] nix-daemon[870]: accepted connection from pid 1748, user nix-grpc-daemon (trusted)6650worker2 # [ 214.265223] nix-daemon[870]: reaped child process 1751, status = succeeded6651worker2 # [ 214.436379] nix-daemon[870]: accepted connection from pid 1753, user nix-grpc-daemon (trusted)6652worker2 # [ 214.454712] nix-daemon[870]: reaped child process 1756, status = succeeded6653lb # [ 214.484815] niks3-server[1127]: 2026/09/21 09:16:44 INFO Received uploads request method=POST path=/api/pending_closures6654worker2 # [ 214.475827] nix-grpc-daemon[1461]: time=2026-09-21T09:16:44.058Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6655worker2 # [ 214.477485] nix-grpc-daemon[1461]: time=2026-09-21T09:16:44.058Z level=INFO msg="Uploading 77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv (760B)"6656lb # [ 214.515224] niks3-server[1127]: 2026/09/21 09:16:44 INFO Registered completed upload object_key=nar/1hrr94bvb37dwmf1giypzl9qnff72xm540ak7q49j34fyzia9ngb.nar.zst6657worker2 # [ 214.507997] nix-grpc-daemon[1461]: time=2026-09-21T09:16:44.090Z level=INFO msg="Uploading 1 narinfos"6658lb # [ 214.531459] niks3-server[1127]: 2026/09/21 09:16:44 INFO Received sign narinfos request method=POST path=/api/pending_closures/68/sign6659lb # [ 214.541759] niks3-server[1127]: 2026/09/21 09:16:44 INFO Signed narinfos id=68 count=16660lb # [ 214.546255] niks3-server[1127]: 2026/09/21 09:16:44 INFO Registered completed upload object_key=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l.ls6661worker2 # [ 214.536329] nix-grpc-daemon[1461]: time=2026-09-21T09:16:44.119Z level=INFO msg="Upload complete. (473ms)"6662lb # [ 214.554977] niks3-server[1127]: 2026/09/21 09:16:44 INFO Received complete upload request method=POST path=/api/pending_closures/68/complete6663lb # [ 214.560324] niks3-server[1127]: 2026/09/21 09:16:44 INFO Completed upload id=686664worker2 # [ 214.544130] nix-grpc-daemon[1386]: ts=2026-09-21T09:16:44Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46616 duration_s=0 paths=1 nar_bytes_in=7606665lb # [ 214.563570] niks3-server[1127]: 2026/09/21 09:16:44 INFO Registered completed upload object_key=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l.narinfo6666worker2 # [ 214.554302] nix-grpc-daemon[1386]: ts=2026-09-21T09:16:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:466166667worker2 # [ 214.560886] nix-daemon[870]: accepted connection from pid 1386, user nix-grpc-daemon (trusted)6668worker2 # [ 214.565620] nix-daemon[870]: reaped child process 1746, status = succeeded6669worker2 # [ 214.576452] nix-daemon[870]: accepted connection from pid 1386, user nix-grpc-daemon (trusted)6670client # building '/nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv' on lb:50051...6671client # farm-top> worker2: building 77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv6672client # farm-top> LOG-farm-top-drain6673worker2 # [ 216.212570] nix-daemon[870]: reaped child process 1760, status = succeeded6674worker2 # [ 216.381754] nix-daemon[870]: accepted connection from pid 1775, user nix-grpc-daemon (trusted)6675worker2 # [ 216.399981] nix-daemon[870]: reaped child process 1778, status = succeeded6676lb # [ 216.436573] niks3-server[1127]: 2026/09/21 09:16:46 INFO Received uploads request method=POST path=/api/pending_closures6677worker2 # [ 216.427612] nix-grpc-daemon[1461]: time=2026-09-21T09:16:46.009Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6678worker2 # [ 216.429232] nix-grpc-daemon[1461]: time=2026-09-21T09:16:46.009Z level=INFO msg="Uploading bzkv277k4v9r9h4aq5h13ddcavmp36fg-farm-top (232B)"6679lb # [ 216.472237] niks3-server[1127]: 2026/09/21 09:16:46 INFO Registered completed upload object_key=log/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv6680lb # [ 216.484413] niks3-server[1127]: 2026/09/21 09:16:46 INFO Registered completed upload object_key=nar/0l8sy8grn9pmq2n58ag0hx791zdfyxwk9319ynjqmn7yy7wl64s7.nar.zst6681worker2 # [ 216.474630] nix-grpc-daemon[1461]: time=2026-09-21T09:16:46.057Z level=INFO msg="Uploading 1 narinfos"6682lb # [ 216.497108] niks3-server[1127]: 2026/09/21 09:16:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/69/sign6683lb # [ 216.504836] niks3-server[1127]: 2026/09/21 09:16:46 INFO Signed narinfos id=69 count=16684lb # [ 216.507316] niks3-server[1127]: 2026/09/21 09:16:46 INFO Registered completed upload object_key=bzkv277k4v9r9h4aq5h13ddcavmp36fg.ls6685lb # [ 216.515136] niks3-server[1127]: 2026/09/21 09:16:46 INFO Received complete upload request method=POST path=/api/pending_closures/69/complete6686lb # [ 216.521332] niks3-server[1127]: 2026/09/21 09:16:46 INFO Completed upload id=696687worker2 # [ 216.502411] nix-grpc-daemon[1461]: time=2026-09-21T09:16:46.085Z level=INFO msg="Upload complete. (313ms)"6688lb # [ 216.524458] niks3-server[1127]: 2026/09/21 09:16:46 INFO Registered completed upload object_key=bzkv277k4v9r9h4aq5h13ddcavmp36fg.narinfo6689worker2 # [ 216.517111] nix-grpc-daemon[1386]: ts=2026-09-21T09:16:46Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46616 duration_s=1 drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv assign_id=18 outputs=16690worker2 # [ 216.529899] nix-daemon[870]: reaped child process 1758, status = succeeded6691client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag drain >&2, in 7.37 seconds)6692client: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag drain outPath6693client: (finished: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag drain outPath, in 0.23 seconds)6694worker1: must succeed: rm /nix/.rw-store/fill6695worker1: (finished: must succeed: rm /nix/.rw-store/fill, in 0.15 seconds)6696worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free'6697worker1 # [ 217.343205] nix-grpc-daemon[1690]: ts=2026-09-21T09:16:46Z level=info event=healthy reason=min_free6698worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free', in 1.12 seconds)6699(finished: subtest: low disk drains a worker and builds go to the other, in 11.20 seconds)6700(finished: run the VM test script, in 219.13 seconds)6701test script finished in 219.41s6702cleanup6703kill QemuMachine (pid 49)6704client # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/lfh101cfjj8p8npcari9hk0lbqi33yvh-python3-3.14.7/bin/python3.14)6705kill QemuMachine (pid 50)6706lb # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/lfh101cfjj8p8npcari9hk0lbqi33yvh-python3-3.14.7/bin/python3.14)6707kill QemuMachine (pid 51)6708worker1 # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/lfh101cfjj8p8npcari9hk0lbqi33yvh-python3-3.14.7/bin/python3.14)6709kill QemuMachine (pid 52)6710worker2 # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/lfh101cfjj8p8npcari9hk0lbqi33yvh-python3-3.14.7/bin/python3.14)6711(finished: cleanup, in 1.03 seconds)6712additionally exposed symbols:6713 client, lb, worker1, worker2,6714 vlan1,6715 start_all, test_script, machines, machines_qemu, machines_nspawn, vlans, driver, log, os, create_machine, subtest, run_tests, join_all, retry, serial_stdout_off, serial_stdout_on, polling_condition, BaseMachine, QemuMachine, NspawnMachine, t, debug, dump_machine_ssh