vm-test-run-nix-grpc-farm
checks.aarch64-linux.farm
· build #470
· raw
1tribuchet: building on eliza2Machine state will be reset. To keep it, pass --keep-machine-state3start all VLans4(finished: start all VLans, in 0.00 seconds)5Test will time out and terminate in 900.0 seconds6run the VM test script7start all VMs8client: starting vm9lb: starting vm10worker1: starting vm11worker2: starting vm12client # Disk image does not exist, creating the virtualisation disk image...13client: QEMU running (pid 49)14client # Formatting '/build/vm-state-client/tmp.spmA6NA12Q', fmt=raw size=107374182415lb # Disk image does not exist, creating the virtualisation disk image...16lb: QEMU running (pid 50)17worker1 # Disk image does not exist, creating the virtualisation disk image...18worker1: QEMU running (pid 52)19lb # Formatting '/build/vm-state-lb/tmp.w0t0aFY7Eg', fmt=raw size=107374182420worker1 # Formatting '/build/vm-state-worker1/tmp.HrQQqzgI3S', fmt=raw size=107374182421client # mke2fs 1.47.4 (6-Mar-2025)22lb # mke2fs 1.47.4 (6-Mar-2025)23worker1 # mke2fs 1.47.4 (6-Mar-2025)24client # Discarding device blocks: 0/262144 done25lb # Discarding device blocks: 0/262144 done26worker1 # Discarding device blocks: 0/262144 done27client # Creating filesystem with 262144 4k blocks and 65536 inodes28lb # Creating filesystem with 262144 4k blocks and 65536 inodes29worker1 # Creating filesystem with 262144 4k blocks and 65536 inodes30client # Filesystem UUID: e1c1dd6d-d0d5-48ab-918d-58d37418a61631lb # Filesystem UUID: d5d81774-ba7c-4af2-ab79-e383fb9590b832worker1 # Filesystem UUID: 051900b9-7d73-4b0d-9cf4-daa02c92093533client # Superblock backups stored on blocks:34lb # Superblock backups stored on blocks:35worker1 # Superblock backups stored on blocks:36client # 32768, 98304, 163840, 22937637lb # 32768, 98304, 163840, 22937638worker1 # 32768, 98304, 163840, 22937639client # 40lb # 41worker1 # 42client # Allocating group tables: 0/8 done43lb # Allocating group tables: 0/8 done44worker1 # Allocating group tables: 0/8 done45client # Writing inode tables: 0/8 done46lb # Writing inode tables: 0/8 done47worker1 # Writing inode tables: 0/8 done48client # Creating journal (8192 blocks): done49lb # Creating journal (8192 blocks): done50worker1 # Creating journal (8192 blocks): done51client # Writing superblocks and filesystem accounting information: 0/8 done52lb # Writing superblocks and filesystem accounting information: 0/8 done53worker1 # Writing superblocks and filesystem accounting information: 0/8 done54client # 55lb # 56worker1 # 57client # Virtualisation disk image created.58lb # Virtualisation disk image created.59worker1 # Virtualisation disk image created.60worker2: QEMU running (pid 51)61worker2 # Disk image does not exist, creating the virtualisation disk image...62worker2 # Formatting '/build/vm-state-worker2/tmp.6qg6ZBFLhQ', fmt=raw size=107374182463worker2 # mke2fs 1.47.4 (6-Mar-2025)64worker2 # Discarding device blocks: 0/262144 done65worker2 # Creating filesystem with 262144 4k blocks and 65536 inodes66(finished: start all VMs, in 0.48 seconds)67worker2 # Filesystem UUID: 9841e95f-8727-4999-9445-4307181025de68client: waiting for unit network-addresses-eth1.service69worker2 # Superblock backups stored on blocks:70client: waiting for the VM to finish booting71worker2 # 32768, 98304, 163840, 22937672worker2 # 73worker2 # Allocating group tables: 0/8 done74worker2 # Writing inode tables: 0/8 done75worker2 # Creating journal (8192 blocks): done76worker2 # Writing superblocks and filesystem accounting information: 0/8 done77worker2 # 78worker2 # Virtualisation disk image created.79client # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40]80client # [ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 7 15:23:00 UTC 202681client # [ 0.000000] KASLR enabled82client # [ 0.000000] random: crng init done83client # [ 0.000000] Machine model: linux,dummy-virt84client # [ 0.000000] efi: UEFI not found.85client # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT86lb # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40]87client # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff]88client # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f]89client # [ 0.000000] Zone ranges:90lb # [ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 7 15:23:00 UTC 202691client # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff]92lb # [ 0.000000] KASLR enabled93client # [ 0.000000] DMA32 empty94lb # [ 0.000000] random: crng init done95client # [ 0.000000] Normal empty96lb # [ 0.000000] Machine model: linux,dummy-virt97client # [ 0.000000] Device empty98lb # [ 0.000000] efi: UEFI not found.99client # [ 0.000000] Movable zone start for each node100client # [ 0.000000] Early memory node ranges101lb # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT102client # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff]103lb # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff]104client # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff]105lb # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f]106lb # [ 0.000000] Zone ranges:107client # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000108client # [ 0.000000] psci: probing for conduit method from DT.109lb # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff]110lb # [ 0.000000] DMA32 empty111client # [ 0.000000] psci: PSCIv1.3 detected in firmware.112lb # [ 0.000000] Normal empty113lb # [ 0.000000] Device empty114client # [ 0.000000] psci: Using standard PSCI v0.2 function IDs115lb # [ 0.000000] Movable zone start for each node116client # [ 0.000000] psci: Trusted OS migration not required117lb # [ 0.000000] Early memory node ranges118client # [ 0.000000] psci: SMC Calling Convention v1.1119lb # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff]120client # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003)121lb # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff]122client # [ 0.000000] percpu: Embedded 76 pages/cpu s186648 r8192 d116456 u311296123lb # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000124worker2 # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40]125client # [ 0.000000] Detected PIPT I-cache on CPU0126lb # [ 0.000000] psci: probing for conduit method from DT.127lb # [ 0.000000] psci: PSCIv1.3 detected in firmware.128client # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm)129lb # [ 0.000000] psci: Using standard PSCI v0.2 function IDs130worker2 # [ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 7 15:23:00 UTC 2026131client # [ 0.000000] CPU features: detected: GICv3 CPU interface132worker2 # [ 0.000000] KASLR enabled133lb # [ 0.000000] psci: Trusted OS migration not required134client # [ 0.000000] CPU features: detected: Spectre-v4135worker2 # [ 0.000000] random: crng init done136lb # [ 0.000000] psci: SMC Calling Convention v1.1137client # [ 0.000000] CPU features: detected: Spectre-BHB138worker2 # [ 0.000000] Machine model: linux,dummy-virt139worker2 # [ 0.000000] efi: UEFI not found.140client # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38141lb # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003)142worker2 # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT143client # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23144lb # [ 0.000000] percpu: Embedded 76 pages/cpu s186648 r8192 d116456 u311296145client # [ 0.000000] alternatives: applying boot alternatives146worker2 # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff]147lb # [ 0.000000] Detected PIPT I-cache on CPU0148worker2 # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f]149lb # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm)150worker2 # [ 0.000000] Zone ranges:151lb # [ 0.000000] CPU features: detected: GICv3 CPU interface152worker2 # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff]153lb # [ 0.000000] CPU features: detected: Spectre-v4154worker2 # [ 0.000000] DMA32 empty155worker2 # [ 0.000000] Normal empty156lb # [ 0.000000] CPU features: detected: Spectre-BHB157worker2 # [ 0.000000] Device empty158worker2 # [ 0.000000] Movable zone start for each node159lb # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38160client # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/wzyai2lcdcb8sgsxr1wpx21k8i95k7mn-nixos-system-client-test/init regInfo=/nix/store/pdj8wc854xn099x1vdf9pc3a2nsrcvf7-closure-info/registration console=ttyAMA0,115200n8 console=tty0161worker2 # [ 0.000000] Early memory node ranges162lb # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23163worker2 # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff]164lb # [ 0.000000] alternatives: applying boot alternatives165client # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/pdj8wc854xn099x1vdf9pc3a2nsrcvf7-closure-info/registration", will be passed to user space.166worker2 # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff]167worker2 # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000168client # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes169worker2 # [ 0.000000] psci: probing for conduit method from DT.170client # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)171worker2 # [ 0.000000] psci: PSCIv1.3 detected in firmware.172worker1 # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40]173worker2 # [ 0.000000] psci: Using standard PSCI v0.2 function IDs174client # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)175worker2 # [ 0.000000] psci: Trusted OS migration not required176lb # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/sy3pchz5vg22ylc3whvkp4s8qngi8wmd-nixos-system-lb-test/init regInfo=/nix/store/4q693lxjjy3z9gghvdrv4s7qcr63j8a1-closure-info/registration console=ttyAMA0,115200n8 console=tty0177worker1 # [ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 7 15:23:00 UTC 2026178client # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB179worker2 # [ 0.000000] psci: SMC Calling Convention v1.1180worker1 # [ 0.000000] KASLR enabled181client # [ 0.000000] software IO TLB: area num 1.182worker1 # [ 0.000000] random: crng init done183worker1 # [ 0.000000] Machine model: linux,dummy-virt184worker2 # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003)185client # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB186lb # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/4q693lxjjy3z9gghvdrv4s7qcr63j8a1-closure-info/registration", will be passed to user space.187worker1 # [ 0.000000] efi: UEFI not found.188worker2 # [ 0.000000] percpu: Embedded 76 pages/cpu s186648 r8192 d116456 u311296189client # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB)190lb # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes191worker2 # [ 0.000000] Detected PIPT I-cache on CPU0192worker1 # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT193client # [ 0.000000] Fallback order for Node 0: 0194lb # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)195worker1 # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff]196client # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216197worker2 # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm)198client # [ 0.000000] Policy zone: DMA199worker1 # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f]200worker2 # [ 0.000000] CPU features: detected: GICv3 CPU interface201lb # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)202worker1 # [ 0.000000] Zone ranges:203worker2 # [ 0.000000] CPU features: detected: Spectre-v4204client # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off205lb # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB206worker1 # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff]207worker2 # [ 0.000000] CPU features: detected: Spectre-BHB208lb # [ 0.000000] software IO TLB: area num 1.209client # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1210worker1 # [ 0.000000] DMA32 empty211worker1 # [ 0.000000] Normal empty212worker2 # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38213client # [ 0.000000] allocated 3145728 bytes of page_ext214lb # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB215worker1 # [ 0.000000] Device empty216client # [ 0.000000] ftrace: allocating 74885 entries in 294 pages217worker2 # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23218worker1 # [ 0.000000] Movable zone start for each node219lb # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB)220client # [ 0.000000] ftrace: allocated 294 pages with 4 groups221worker2 # [ 0.000000] alternatives: applying boot alternatives222worker1 # [ 0.000000] Early memory node ranges223lb # [ 0.000000] Fallback order for Node 0: 0224client # [ 0.000000] rcu: Hierarchical RCU implementation.225worker1 # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff]226client # [ 0.000000] rcu: RCU event tracing is enabled.227lb # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216228worker1 # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff]229lb # [ 0.000000] Policy zone: DMA230client # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.231worker1 # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000232lb # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off233client # [ 0.000000] Trampoline variant of Tasks RCU enabled.234worker1 # [ 0.000000] psci: probing for conduit method from DT.235client # [ 0.000000] Rude variant of Tasks RCU enabled.236lb # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1237worker1 # [ 0.000000] psci: PSCIv1.3 detected in firmware.238client # [ 0.000000] Tracing variant of Tasks RCU enabled.239worker2 # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/5cljzl6wrgl74g4i9mg12h52ihgkgzia-nixos-system-worker2-test/init regInfo=/nix/store/6hjywiy50qhqnjghshq6dwj0rsaxxvm3-closure-info/registration console=ttyAMA0,115200n8 console=tty0240lb # [ 0.000000] allocated 3145728 bytes of page_ext241worker1 # [ 0.000000] psci: Using standard PSCI v0.2 function IDs242lb # [ 0.000000] ftrace: allocating 74885 entries in 294 pages243client # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.244worker1 # [ 0.000000] psci: Trusted OS migration not required245lb # [ 0.000000] ftrace: allocated 294 pages with 4 groups246worker1 # [ 0.000000] psci: SMC Calling Convention v1.1247client # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1248lb # [ 0.000000] rcu: Hierarchical RCU implementation.249worker2 # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/6hjywiy50qhqnjghshq6dwj0rsaxxvm3-closure-info/registration", will be passed to user space.250lb # [ 0.000000] rcu: RCU event tracing is enabled.251worker1 # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003)252worker2 # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes253client # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.254lb # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.255worker1 # [ 0.000000] percpu: Embedded 76 pages/cpu s186648 r8192 d116456 u311296256worker2 # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)257client # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.258lb # [ 0.000000] Trampoline variant of Tasks RCU enabled.259worker1 # [ 0.000000] Detected PIPT I-cache on CPU0260lb # [ 0.000000] Rude variant of Tasks RCU enabled.261worker2 # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)262client # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.263worker1 # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm)264lb # [ 0.000000] Tracing variant of Tasks RCU enabled.265worker2 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB266worker1 # [ 0.000000] CPU features: detected: GICv3 CPU interface267client # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0268worker2 # [ 0.000000] software IO TLB: area num 1.269lb # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.270client # [ 0.000000] GICv3: 256 SPIs implemented271worker1 # [ 0.000000] CPU features: detected: Spectre-v4272client # [ 0.000000] GICv3: 0 Extended SPIs implemented273worker2 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB274worker1 # [ 0.000000] CPU features: detected: Spectre-BHB275lb # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1276client # [ 0.000000] Root IRQ handler: gic_handle_irq277worker1 # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38278worker2 # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB)279client # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI280lb # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.281worker2 # [ 0.000000] Fallback order for Node 0: 0282worker1 # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23283client # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0284worker2 # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216285lb # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.286worker1 # [ 0.000000] alternatives: applying boot alternatives287client # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000288worker2 # [ 0.000000] Policy zone: DMA289client # [ 0.000000] ITS [mem 0x08080000-0x0809ffff]290worker2 # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off291lb # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.292lb # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0293client # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1)294worker2 # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1295lb # [ 0.000000] GICv3: 256 SPIs implemented296worker2 # [ 0.000000] allocated 3145728 bytes of page_ext297lb # [ 0.000000] GICv3: 0 Extended SPIs implemented298worker2 # [ 0.000000] ftrace: allocating 74885 entries in 294 pages299client # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1)300lb # [ 0.000000] Root IRQ handler: gic_handle_irq301client # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000302worker1 # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/9d8fygwq37kz8yh1yl6h683p1r23agf6-nixos-system-worker1-test/init regInfo=/nix/store/bdhfwc4gyq5x9cvh3llg8w2cn0zr2la0-closure-info/registration console=ttyAMA0,115200n8 console=tty0303worker2 # [ 0.000000] ftrace: allocated 294 pages with 4 groups304lb # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI305worker2 # [ 0.000000] rcu: Hierarchical RCU implementation.306lb # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0307client # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000308worker2 # [ 0.000000] rcu: RCU event tracing is enabled.309lb # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000310client # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.311worker1 # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/bdhfwc4gyq5x9cvh3llg8w2cn0zr2la0-closure-info/registration", will be passed to user space.312worker2 # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.313lb # [ 0.000000] ITS [mem 0x08080000-0x0809ffff]314worker2 # [ 0.000000] Trampoline variant of Tasks RCU enabled.315worker1 # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes316client # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns317worker2 # [ 0.000000] Rude variant of Tasks RCU enabled.318lb # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1)319client # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt).320worker1 # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)321worker2 # [ 0.000000] Tracing variant of Tasks RCU enabled.322lb # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1)323worker2 # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.324worker1 # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)325client # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns326worker1 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB327lb # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000328worker2 # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1329worker1 # [ 0.000000] software IO TLB: area num 1.330client # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns331lb # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000332client # [ 0.000033] arm-pv: using stolen time PV333worker2 # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.334worker1 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB335lb # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.336client # [ 0.000591] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)337worker1 # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB)338worker2 # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.339client # [ 0.000757] Console: colour dummy device 80x25340worker1 # [ 0.000000] Fallback order for Node 0: 0341lb # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns342client # [ 0.000765] printk: legacy console [tty0] enabled343worker2 # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.344lb # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt).345worker1 # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216346worker1 # [ 0.000000] Policy zone: DMA347worker2 # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0348worker2 # [ 0.000000] GICv3: 256 SPIs implemented349client # [ 0.000960] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000)350worker1 # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off351lb # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns352worker2 # [ 0.000000] GICv3: 0 Extended SPIs implemented353client # [ 0.000967] pid_max: default: 32768 minimum: 301354worker1 # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1355worker2 # [ 0.000000] Root IRQ handler: gic_handle_irq356lb # [ 0.000001] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns357client # [ 0.001055] LSM: initializing lsm=capability,landlock,yama,bpf,ima358worker1 # [ 0.000000] allocated 3145728 bytes of page_ext359worker2 # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI360lb # [ 0.000039] arm-pv: using stolen time PV361client # [ 0.001222] landlock: Up and running.362worker1 # [ 0.000000] ftrace: allocating 74885 entries in 294 pages363worker2 # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0364client # [ 0.001225] Yama: becoming mindful.365client # [ 0.001803] LSM support for eBPF active366worker1 # [ 0.000000] ftrace: allocated 294 pages with 4 groups367lb # [ 0.000573] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)368worker2 # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000369lb # [ 0.000728] Console: colour dummy device 80x25370worker1 # [ 0.000000] rcu: Hierarchical RCU implementation.371worker2 # [ 0.000000] ITS [mem 0x08080000-0x0809ffff]372client # [ 0.001963] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)373worker1 # [ 0.000000] rcu: RCU event tracing is enabled.374lb # [ 0.000736] printk: legacy console [tty0] enabled375client # [ 0.001997] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)376worker2 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1)377worker1 # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.378lb # [ 0.000925] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000)379client # [ 0.003203] cacheinfo: Unable to detect cache hierarchy for CPU 0380worker1 # [ 0.000000] Trampoline variant of Tasks RCU enabled.381lb # [ 0.000931] pid_max: default: 32768 minimum: 301382client # [ 0.004033] rcu: Hierarchical SRCU implementation.383worker2 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1)384worker1 # [ 0.000000] Rude variant of Tasks RCU enabled.385client # [ 0.004039] rcu: Max phase no-delay instances is 1000.386lb # [ 0.001008] LSM: initializing lsm=capability,landlock,yama,bpf,ima387worker1 # [ 0.000000] Tracing variant of Tasks RCU enabled.388worker2 # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000389client # [ 0.005241] fsl-mc MSI: its@8080000 domain created390lb # [ 0.001167] landlock: Up and running.391worker1 # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.392worker2 # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000393lb # [ 0.001170] Yama: becoming mindful.394client # [ 0.005338] EFI services will not be available.395lb # [ 0.001765] LSM support for eBPF active396client # [ 0.005448] smp: Bringing up secondary CPUs ...397worker2 # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.398worker1 # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1399client # [ 0.005458] smp: Brought up 1 node, 1 CPU400lb # [ 0.001955] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)401client # [ 0.005461] SMP: Total of 1 processors activated.402worker1 # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.403worker2 # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns404client # [ 0.005467] CPU: All CPU(s) started at EL1405lb # [ 0.001991] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)406worker2 # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt).407client # [ 0.005478] CPU features: detected: Branch Target Identification408lb # [ 0.003186] cacheinfo: Unable to detect cache hierarchy for CPU 0409worker1 # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.410lb # [ 0.003978] rcu: Hierarchical SRCU implementation.411client # [ 0.005483] CPU features: detected: ARMv8.4 Translation Table Level412worker2 # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns413lb # [ 0.003984] rcu: Max phase no-delay instances is 1000.414worker1 # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.415lb # [ 0.005274] fsl-mc MSI: its@8080000 domain created416client # [ 0.005486] CPU features: detected: Instruction cache invalidation not required for I/D coherence417worker1 # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0418worker2 # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns419worker1 # [ 0.000000] GICv3: 256 SPIs implemented420worker2 # [ 0.000032] arm-pv: using stolen time PV421client # [ 0.005490] CPU features: detected: Data cache clean to the PoU not required for I/D coherence422worker1 # [ 0.000000] GICv3: 0 Extended SPIs implemented423lb # [ 0.005375] EFI services will not be available.424worker1 # [ 0.000000] Root IRQ handler: gic_handle_irq425client # [ 0.005493] CPU features: detected: Common not Private translations426lb # [ 0.005455] smp: Bringing up secondary CPUs ...427worker2 # [ 0.000497] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)428worker1 # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI429lb # [ 0.005464] smp: Brought up 1 node, 1 CPU430client # [ 0.005497] CPU features: detected: CRC32 instructions431worker2 # [ 0.000696] Console: colour dummy device 80x25432worker1 # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0433lb # [ 0.005467] SMP: Total of 1 processors activated.434worker2 # [ 0.000704] printk: legacy console [tty0] enabled435client # [ 0.005499] CPU features: detected: Data cache clean to Point of Deep Persistence436lb # [ 0.005472] CPU: All CPU(s) started at EL1437worker1 # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000438lb # [ 0.005484] CPU features: detected: Branch Target Identification439client # [ 0.005503] CPU features: detected: Data cache clean to Point of Persistence440worker1 # [ 0.000000] ITS [mem 0x08080000-0x0809ffff]441worker2 # [ 0.000896] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000)442lb # [ 0.005489] CPU features: detected: ARMv8.4 Translation Table Level443worker2 # [ 0.000903] pid_max: default: 32768 minimum: 301444client # [ 0.005506] CPU features: detected: Data independent timing control (DIT)445worker1 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1)446client # [ 0.005509] CPU features: detected: E0PD447worker2 # [ 0.000983] LSM: initializing lsm=capability,landlock,yama,bpf,ima448lb # [ 0.005492] CPU features: detected: Instruction cache invalidation not required for I/D coherence449worker2 # [ 0.001155] landlock: Up and running.450client # [ 0.005512] CPU features: detected: Enhanced Counter Virtualization451worker2 # [ 0.001157] Yama: becoming mindful.452worker1 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1)453lb # [ 0.005496] CPU features: detected: Data cache clean to the PoU not required for I/D coherence454client # [ 0.005515] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF)455worker2 # [ 0.001671] LSM support for eBPF active456worker1 # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000457client # [ 0.005518] CPU features: detected: Enhanced Virtualization Traps458lb # [ 0.005500] CPU features: detected: Common not Private translations459worker2 # [ 0.001802] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)460worker1 # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000461lb # [ 0.005503] CPU features: detected: CRC32 instructions462client # [ 0.005521] CPU features: detected: Fine Grained Traps463worker2 # [ 0.001840] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)464worker1 # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.465lb # [ 0.005506] CPU features: detected: Data cache clean to Point of Deep Persistence466client # [ 0.005525] CPU features: detected: Generic authentication (architected QARMA5 algorithm)467worker2 # [ 0.002958] cacheinfo: Unable to detect cache hierarchy for CPU 0468client # [ 0.005529] CPU features: detected: RCpc load-acquire (LDAPR)469lb # [ 0.005509] CPU features: detected: Data cache clean to Point of Persistence470worker2 # [ 0.003693] rcu: Hierarchical SRCU implementation.471worker1 # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns472client # [ 0.005532] CPU features: detected: LSE atomic instructions473lb # [ 0.005512] CPU features: detected: Data independent timing control (DIT)474worker2 # [ 0.003697] rcu: Max phase no-delay instances is 1000.475worker1 # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt).476client # [ 0.005535] CPU features: detected: Privileged Access Never477lb # [ 0.005515] CPU features: detected: E0PD478worker2 # [ 0.004903] fsl-mc MSI: its@8080000 domain created479client # [ 0.005538] CPU features: detected: PMUv3480worker2 # [ 0.004992] EFI services will not be available.481lb # [ 0.005518] CPU features: detected: Enhanced Counter Virtualization482worker1 # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns483worker2 # [ 0.005062] smp: Bringing up secondary CPUs ...484client # [ 0.005540] CPU features: detected: RAS Extension Support485lb # [ 0.005521] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF)486worker2 # [ 0.005070] smp: Brought up 1 node, 1 CPU487worker1 # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns488worker2 # [ 0.005073] SMP: Total of 1 processors activated.489lb # [ 0.005524] CPU features: detected: Enhanced Virtualization Traps490worker1 # [ 0.000031] arm-pv: using stolen time PV491client # [ 0.005543] CPU features: detected: RASv1p1 Extension Support492worker2 # [ 0.005079] CPU: All CPU(s) started at EL1493lb # [ 0.005527] CPU features: detected: Fine Grained Traps494client # [ 0.005546] CPU features: detected: Random Number Generator495worker2 # [ 0.005089] CPU features: detected: Branch Target Identification496worker1 # [ 0.000404] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)497lb # [ 0.005530] CPU features: detected: Generic authentication (architected QARMA5 algorithm)498client # [ 0.005548] CPU features: detected: Speculation barrier (SB)499worker1 # [ 0.000579] Console: colour dummy device 80x25500worker2 # [ 0.005094] CPU features: detected: ARMv8.4 Translation Table Level501client # [ 0.005551] CPU features: detected: Stage-2 Force Write-Back502lb # [ 0.005535] CPU features: detected: RCpc load-acquire (LDAPR)503worker1 # [ 0.000586] printk: legacy console [tty0] enabled504worker2 # [ 0.005098] CPU features: detected: Instruction cache invalidation not required for I/D coherence505lb # [ 0.005538] CPU features: detected: LSE atomic instructions506client # [ 0.005554] CPU features: detected: TLB range maintenance instructions507lb # [ 0.005541] CPU features: detected: Privileged Access Never508worker1 # [ 0.000784] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000)509worker2 # [ 0.005101] CPU features: detected: Data cache clean to the PoU not required for I/D coherence510client # [ 0.005559] CPU features: detected: Speculative Store Bypassing Safe (SSBS)511lb # [ 0.005544] CPU features: detected: PMUv3512worker1 # [ 0.000791] pid_max: default: 32768 minimum: 301513client # [ 0.005600] alternatives: applying system-wide alternatives514worker2 # [ 0.005105] CPU features: detected: Common not Private translations515lb # [ 0.005546] CPU features: detected: RAS Extension Support516worker1 # [ 0.000872] LSM: initializing lsm=capability,landlock,yama,bpf,ima517worker2 # [ 0.005108] CPU features: detected: CRC32 instructions518client # [ 0.008615] CPU features: detected: BBM Level 2 without TLB conflict abort519worker1 # [ 0.001006] landlock: Up and running.520lb # [ 0.005549] CPU features: detected: RASv1p1 Extension Support521worker1 # [ 0.001008] Yama: becoming mindful.522lb # [ 0.005552] CPU features: detected: Random Number Generator523worker2 # [ 0.005110] CPU features: detected: Data cache clean to Point of Deep Persistence524worker1 # [ 0.001455] LSM support for eBPF active525lb # [ 0.005554] CPU features: detected: Speculation barrier (SB)526client # [ 0.008796] Memory: 1406508K/1572864K available (24448K kernel code, 7090K rwdata, 26348K rodata, 4736K init, 1107K bss, 123784K reserved, 32768K cma-reserved)527worker1 # [ 0.001600] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)528client # [ 0.009263] devtmpfs: initialized529lb # [ 0.005557] CPU features: detected: Stage-2 Force Write-Back530worker2 # [ 0.005114] CPU features: detected: Data cache clean to Point of Persistence531worker1 # [ 0.001638] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)532lb # [ 0.005560] CPU features: detected: TLB range maintenance instructions533client # [ 0.011109] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)534worker2 # [ 0.005117] CPU features: detected: Data independent timing control (DIT)535worker1 # [ 0.002799] cacheinfo: Unable to detect cache hierarchy for CPU 0536lb # [ 0.005566] CPU features: detected: Speculative Store Bypassing Safe (SSBS)537worker2 # [ 0.005120] CPU features: detected: E0PD538client # [ 0.011132] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).539worker1 # [ 0.003599] rcu: Hierarchical SRCU implementation.540lb # [ 0.005602] alternatives: applying system-wide alternatives541worker2 # [ 0.005123] CPU features: detected: Enhanced Counter Virtualization542client # [ 0.011315] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL543worker1 # [ 0.003604] rcu: Max phase no-delay instances is 1000.544lb # [ 0.008638] CPU features: detected: BBM Level 2 without TLB conflict abort545client # [ 0.011319] 0 pages in range for non-PLT usage546worker2 # [ 0.005126] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF)547worker1 # [ 0.004832] fsl-mc MSI: its@8080000 domain created548client # [ 0.011320] 508288 pages in range for PLT usage549worker1 # [ 0.004923] EFI services will not be available.550worker2 # [ 0.005129] CPU features: detected: Enhanced Virtualization Traps551client # [ 0.011432] pinctrl core: initialized pinctrl subsystem552worker1 # [ 0.004991] smp: Bringing up secondary CPUs ...553worker2 # [ 0.005132] CPU features: detected: Fine Grained Traps554client # [ 0.012289] DMI not present or invalid.555lb # [ 0.008814] Memory: 1406532K/1572864K available (24448K kernel code, 7090K rwdata, 26348K rodata, 4736K init, 1107K bss, 123764K reserved, 32768K cma-reserved)556worker1 # [ 0.005000] smp: Brought up 1 node, 1 CPU557lb # [ 0.009199] devtmpfs: initialized558worker1 # [ 0.005004] SMP: Total of 1 processors activated.559client # [ 0.015542] NET: Registered PF_NETLINK/PF_ROUTE protocol family560worker2 # [ 0.005136] CPU features: detected: Generic authentication (architected QARMA5 algorithm)561worker1 # [ 0.005009] CPU: All CPU(s) started at EL1562lb # [ 0.011077] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)563worker2 # [ 0.005141] CPU features: detected: RCpc load-acquire (LDAPR)564client # [ 0.018237] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations565worker1 # [ 0.005021] CPU features: detected: Branch Target Identification566worker2 # [ 0.005144] CPU features: detected: LSE atomic instructions567lb # [ 0.011102] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).568client # [ 0.018434] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations569worker1 # [ 0.005027] CPU features: detected: ARMv8.4 Translation Table Level570worker2 # [ 0.005147] CPU features: detected: Privileged Access Never571lb # [ 0.011300] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL572worker2 # [ 0.005150] CPU features: detected: PMUv3573client # [ 0.018642] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations574lb # [ 0.011305] 0 pages in range for non-PLT usage575worker1 # [ 0.005030] CPU features: detected: Instruction cache invalidation not required for I/D coherence576worker2 # [ 0.005152] CPU features: detected: RAS Extension Support577client # [ 0.018668] audit: initializing netlink subsys (disabled)578lb # [ 0.011306] 508288 pages in range for PLT usage579worker2 # [ 0.005155] CPU features: detected: RASv1p1 Extension Support580worker1 # [ 0.005034] CPU features: detected: Data cache clean to the PoU not required for I/D coherence581lb # [ 0.011428] pinctrl core: initialized pinctrl subsystem582client # [ 0.019281] thermal_sys: Registered thermal governor 'fair_share'583lb # [ 0.012164] DMI not present or invalid.584worker2 # [ 0.005158] CPU features: detected: Random Number Generator585client # [ 0.019284] thermal_sys: Registered thermal governor 'bang_bang'586worker1 # [ 0.005037] CPU features: detected: Common not Private translations587lb # [ 0.015320] NET: Registered PF_NETLINK/PF_ROUTE protocol family588worker2 # [ 0.005160] CPU features: detected: Speculation barrier (SB)589worker1 # [ 0.005040] CPU features: detected: CRC32 instructions590client # [ 0.019288] thermal_sys: Registered thermal governor 'step_wise'591worker2 # [ 0.005163] CPU features: detected: Stage-2 Force Write-Back592lb # [ 0.017629] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations593client # [ 0.019293] thermal_sys: Registered thermal governor 'user_space'594worker1 # [ 0.005043] CPU features: detected: Data cache clean to Point of Deep Persistence595worker2 # [ 0.005166] CPU features: detected: TLB range maintenance instructions596client # [ 0.019296] thermal_sys: Registered thermal governor 'power_allocator'597lb # [ 0.017813] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations598worker2 # [ 0.005171] CPU features: detected: Speculative Store Bypassing Safe (SSBS)599worker1 # [ 0.005047] CPU features: detected: Data cache clean to Point of Persistence600lb # [ 0.018013] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations601client # [ 0.019322] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1602worker2 # [ 0.005208] alternatives: applying system-wide alternatives603client # [ 0.019330] cpuidle: using governor ladder604lb # [ 0.018035] audit: initializing netlink subsys (disabled)605worker1 # [ 0.005050] CPU features: detected: Data independent timing control (DIT)606client # [ 0.019336] cpuidle: using governor menu607worker1 # [ 0.005053] CPU features: detected: E0PD608worker2 # [ 0.008208] CPU features: detected: BBM Level 2 without TLB conflict abort609lb # [ 0.018970] thermal_sys: Registered thermal governor 'fair_share'610client # [ 0.019552] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers.611worker1 # [ 0.005056] CPU features: detected: Enhanced Counter Virtualization612lb # [ 0.018975] thermal_sys: Registered thermal governor 'bang_bang'613client # [ 0.019576] ASID allocator initialised with 65536 entries614worker2 # [ 0.008348] Memory: 1406496K/1572864K available (24448K kernel code, 7090K rwdata, 26348K rodata, 4736K init, 1107K bss, 123772K reserved, 32768K cma-reserved)615client # [ 0.020819] Serial: AMBA PL011 UART driver616worker1 # [ 0.005059] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF)617lb # [ 0.018979] thermal_sys: Registered thermal governor 'step_wise'618worker2 # [ 0.008713] devtmpfs: initialized619worker1 # [ 0.005062] CPU features: detected: Enhanced Virtualization Traps620lb # [ 0.018983] thermal_sys: Registered thermal governor 'user_space'621client # [ 0.026013] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1622worker2 # [ 0.010561] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)623worker1 # [ 0.005065] CPU features: detected: Fine Grained Traps624client # [ 0.026171] printk: console [ttyAMA0] enabled625lb # [ 0.018986] thermal_sys: Registered thermal governor 'power_allocator'626worker2 # [ 0.010584] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).627worker1 # [ 0.005069] CPU features: detected: Generic authentication (architected QARMA5 algorithm)628lb # [ 0.019016] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1629worker2 # [ 0.010756] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL630worker1 # [ 0.005074] CPU features: detected: RCpc load-acquire (LDAPR)631lb # [ 0.019024] cpuidle: using governor ladder632worker2 # [ 0.010761] 0 pages in range for non-PLT usage633lb # [ 0.019030] cpuidle: using governor menu634worker1 # [ 0.005077] CPU features: detected: LSE atomic instructions635worker2 # [ 0.010762] 508288 pages in range for PLT usage636worker1 # [ 0.005081] CPU features: detected: Privileged Access Never637lb # [ 0.019225] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers.638worker2 # [ 0.010870] pinctrl core: initialized pinctrl subsystem639worker1 # [ 0.005084] CPU features: detected: PMUv3640lb # [ 0.019242] ASID allocator initialised with 65536 entries641worker2 # [ 0.011702] DMI not present or invalid.642lb # [ 0.020454] Serial: AMBA PL011 UART driver643worker1 # [ 0.005086] CPU features: detected: RAS Extension Support644worker2 # [ 0.014861] NET: Registered PF_NETLINK/PF_ROUTE protocol family645worker1 # [ 0.005089] CPU features: detected: RASv1p1 Extension Support646lb # [ 0.025887] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1647worker2 # [ 0.017554] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations648worker1 # [ 0.005091] CPU features: detected: Random Number Generator649lb # [ 0.026053] printk: console [ttyAMA0] enabled650worker1 # [ 0.005094] CPU features: detected: Speculation barrier (SB)651worker2 # [ 0.017734] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations652worker1 # [ 0.005097] CPU features: detected: Stage-2 Force Write-Back653worker2 # [ 0.017937] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations654worker1 # [ 0.005100] CPU features: detected: TLB range maintenance instructions655worker2 # [ 0.017960] audit: initializing netlink subsys (disabled)656worker1 # [ 0.005105] CPU features: detected: Speculative Store Bypassing Safe (SSBS)657worker2 # [ 0.018562] thermal_sys: Registered thermal governor 'fair_share'658worker1 # [ 0.005144] alternatives: applying system-wide alternatives659worker2 # [ 0.018564] thermal_sys: Registered thermal governor 'bang_bang'660client # [ 0.145998] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages661worker1 # [ 0.008143] CPU features: detected: BBM Level 2 without TLB conflict abort662worker2 # [ 0.018567] thermal_sys: Registered thermal governor 'step_wise'663client # [ 0.146019] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page664worker2 # [ 0.018572] thermal_sys: Registered thermal governor 'user_space'665client # [ 0.146024] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages666worker2 # [ 0.018575] thermal_sys: Registered thermal governor 'power_allocator'667worker1 # [ 0.008346] Memory: 1406512K/1572864K available (24448K kernel code, 7090K rwdata, 26348K rodata, 4736K init, 1107K bss, 123772K reserved, 32768K cma-reserved)668client # [ 0.146029] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page669worker1 # [ 0.008748] devtmpfs: initialized670worker2 # [ 0.018611] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1671client # [ 0.146033] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages672worker1 # [ 0.010612] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)673worker2 # [ 0.018620] cpuidle: using governor ladder674worker2 # [ 0.018626] cpuidle: using governor menu675client # [ 0.146037] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page676worker1 # [ 0.010635] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).677worker2 # [ 0.018812] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers.678client # [ 0.146041] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages679worker1 # [ 0.010836] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL680worker2 # [ 0.018829] ASID allocator initialised with 65536 entries681client # [ 0.146046] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page682worker1 # [ 0.010842] 0 pages in range for non-PLT usage683worker2 # [ 0.020067] Serial: AMBA PL011 UART driver684worker1 # [ 0.010843] 508288 pages in range for PLT usage685lb # [ 0.147085] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages686worker2 # [ 0.025342] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1687worker1 # [ 0.010957] pinctrl core: initialized pinctrl subsystem688lb # [ 0.147108] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page689worker1 # [ 0.011704] DMI not present or invalid.690worker2 # [ 0.025508] printk: console [ttyAMA0] enabled691client # [ 0.153399] fbcon: Taking over console692lb # [ 0.147114] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages693worker1 # [ 0.014786] NET: Registered PF_NETLINK/PF_ROUTE protocol family694client # [ 0.153417] ACPI: Interpreter disabled.695lb # [ 0.147118] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page696worker1 # [ 0.017434] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations697client # [ 0.155285] iommu: Default domain type: Translated698lb # [ 0.147123] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages699client # [ 0.155296] iommu: DMA domain TLB invalidation policy: strict mode700worker1 # [ 0.017618] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations701lb # [ 0.147127] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page702worker1 # [ 0.017815] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations703lb # [ 0.147131] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages704worker1 # [ 0.017838] audit: initializing netlink subsys (disabled)705lb # [ 0.147135] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page706worker1 # [ 0.018447] thermal_sys: Registered thermal governor 'fair_share'707worker1 # [ 0.018449] thermal_sys: Registered thermal governor 'bang_bang'708worker1 # [ 0.018453] thermal_sys: Registered thermal governor 'step_wise'709lb # [ 0.154746] fbcon: Taking over console710lb # [ 0.154771] ACPI: Interpreter disabled.711worker1 # [ 0.018458] thermal_sys: Registered thermal governor 'user_space'712worker1 # [ 0.018461] thermal_sys: Registered thermal governor 'power_allocator'713lb # [ 0.156709] iommu: Default domain type: Translated714worker1 # [ 0.018498] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1715client # [ 0.162477] SCSI subsystem initialized716lb # [ 0.156719] iommu: DMA domain TLB invalidation policy: strict mode717worker1 # [ 0.018507] cpuidle: using governor ladder718worker2 # [ 0.145664] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages719worker1 # [ 0.018513] cpuidle: using governor menu720lb # [ 0.158478] SCSI subsystem initialized721client # [ 0.163154] usbcore: registered new interface driver usbfs722worker2 # [ 0.145687] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page723worker1 # [ 0.018724] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers.724client # [ 0.163187] usbcore: registered new interface driver hub725worker2 # [ 0.145693] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages726worker1 # [ 0.018740] ASID allocator initialised with 65536 entries727client # [ 0.163204] usbcore: registered new device driver usb728worker1 # [ 0.019907] Serial: AMBA PL011 UART driver729worker2 # [ 0.145697] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page730client # [ 0.163480] pps_core: LinuxPPS API ver. 1 registered731worker2 # [ 0.145702] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages732worker1 # [ 0.025294] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1733client # [ 0.163487] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>734worker1 # [ 0.025419] printk: console [ttyAMA0] enabled735worker2 # [ 0.145706] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page736client # [ 0.163497] PTP clock support registered737client # [ 0.163556] EDAC MC: Ver: 3.0.0738worker2 # [ 0.145710] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages739client # [ 0.168065] scmi_core: SCMI protocol bus registered740worker2 # [ 0.145715] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page741lb # [ 0.163540] usbcore: registered new interface driver usbfs742client # [ 0.169037] FPGA manager framework743lb # [ 0.163574] usbcore: registered new interface driver hub744client # [ 0.170030] vgaarb: loaded745lb # [ 0.163594] usbcore: registered new device driver usb746worker2 # [ 0.153107] fbcon: Taking over console747client # [ 0.170666] clocksource: Switched to clocksource arch_sys_counter748lb # [ 0.163907] pps_core: LinuxPPS API ver. 1 registered749worker2 # [ 0.153123] ACPI: Interpreter disabled.750lb # [ 0.163918] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>751worker2 # [ 0.155043] iommu: Default domain type: Translated752lb # [ 0.163929] PTP clock support registered753worker2 # [ 0.155054] iommu: DMA domain TLB invalidation policy: strict mode754lb # [ 0.163979] EDAC MC: Ver: 3.0.0755worker2 # [ 0.156764] SCSI subsystem initialized756lb # [ 0.168666] scmi_core: SCMI protocol bus registered757client # [ 0.173813] VFS: Disk quotas dquot_6.6.0758client # [ 0.173851] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)759lb # [ 0.169663] FPGA manager framework760lb # [ 0.170673] vgaarb: loaded761worker1 # [ 0.146483] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages762lb # [ 0.171328] clocksource: Switched to clocksource arch_sys_counter763worker1 # [ 0.146508] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page764worker1 # [ 0.146514] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages765client # [ 0.177628] netfs: FS-Cache loaded766client # [ 0.177773] pnp: PnP ACPI: disabled767worker1 # [ 0.146518] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page768worker1 # [ 0.146523] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages769worker2 # [ 0.162001] usbcore: registered new interface driver usbfs770worker1 # [ 0.146527] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page771lb # [ 0.175263] VFS: Disk quotas dquot_6.6.0772worker2 # [ 0.162034] usbcore: registered new interface driver hub773worker1 # [ 0.146531] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages774worker2 # [ 0.162052] usbcore: registered new device driver usb775lb # [ 0.175311] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)776worker1 # [ 0.146536] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page777worker2 # [ 0.162323] pps_core: LinuxPPS API ver. 1 registered778worker2 # [ 0.162329] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>779client # [ 0.181828] NET: Registered PF_INET protocol family780worker2 # [ 0.162338] PTP clock support registered781worker2 # [ 0.162394] EDAC MC: Ver: 3.0.0782worker1 # [ 0.154124] fbcon: Taking over console783client # [ 0.182121] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)784worker1 # [ 0.154139] ACPI: Interpreter disabled.785lb # [ 0.179118] netfs: FS-Cache loaded786worker2 # [ 0.167018] scmi_core: SCMI protocol bus registered787lb # [ 0.179264] pnp: PnP ACPI: disabled788worker2 # [ 0.168007] FPGA manager framework789worker2 # [ 0.168995] vgaarb: loaded790worker2 # [ 0.169652] clocksource: Switched to clocksource arch_sys_counter791worker1 # [ 0.162844] iommu: Default domain type: Translated792lb # [ 0.183575] NET: Registered PF_INET protocol family793worker1 # [ 0.162855] iommu: DMA domain TLB invalidation policy: strict mode794worker2 # [ 0.176036] VFS: Disk quotas dquot_6.6.0795lb # [ 0.183849] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)796worker1 # [ 0.163234] SCSI subsystem initialized797worker2 # [ 0.176089] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)798worker1 # [ 0.165259] usbcore: registered new interface driver usbfs799worker2 # [ 0.177724] netfs: FS-Cache loaded800worker2 # [ 0.177849] pnp: PnP ACPI: disabled801worker1 # [ 0.165296] usbcore: registered new interface driver hub802worker1 # [ 0.165318] usbcore: registered new device driver usb803worker1 # [ 0.165594] pps_core: LinuxPPS API ver. 1 registered804worker1 # [ 0.165601] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>805worker1 # [ 0.165611] PTP clock support registered806worker1 # [ 0.165660] EDAC MC: Ver: 3.0.0807worker1 # [ 0.170378] scmi_core: SCMI protocol bus registered808worker1 # [ 0.171418] FPGA manager framework809worker1 # [ 0.172427] vgaarb: loaded810worker1 # [ 0.173068] clocksource: Switched to clocksource arch_sys_counter811worker2 # [ 0.181737] NET: Registered PF_INET protocol family812worker2 # [ 0.182014] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)813worker1 # [ 0.177674] VFS: Disk quotas dquot_6.6.0814worker1 # [ 0.177721] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)815worker1 # [ 0.179424] netfs: FS-Cache loaded816worker1 # [ 0.179562] pnp: PnP ACPI: disabled817worker1 # [ 0.183596] NET: Registered PF_INET protocol family818worker1 # [ 0.183870] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)819client # [ 0.212768] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)820client # [ 0.212826] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)821client # [ 0.212870] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)822client # [ 0.212948] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)823client # [ 0.213093] TCP: Hash tables configured (established 16384 bind 16384)824client # [ 0.213177] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)825client # [ 0.213218] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)826client # [ 0.213255] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)827client # [ 0.213359] NET: Registered PF_UNIX/PF_LOCAL protocol family828client # [ 0.213388] NET: Registered PF_XDP protocol family829client # [ 0.213408] PCI: CLS 0 bytes, default 64830client # [ 0.213676] Trying to unpack rootfs image as initramfs...831lb # [ 0.214398] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)832lb # [ 0.214459] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)833lb # [ 0.214493] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)834client # [ 0.228168] kvm [1]: HYP mode not available835lb # [ 0.214580] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)836lb # [ 0.214723] TCP: Hash tables configured (established 16384 bind 16384)837lb # [ 0.214818] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)838lb # [ 0.214858] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)839lb # [ 0.214897] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)840worker2 # [ 0.211379] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)841lb # [ 0.214989] NET: Registered PF_UNIX/PF_LOCAL protocol family842worker2 # [ 0.211436] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)843lb # [ 0.215042] NET: Registered PF_XDP protocol family844lb # [ 0.215075] PCI: CLS 0 bytes, default 64845worker2 # [ 0.211465] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)846lb # [ 0.215391] Trying to unpack rootfs image as initramfs...847worker2 # [ 0.211536] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)848worker2 # [ 0.211679] TCP: Hash tables configured (established 16384 bind 16384)849worker2 # [ 0.211773] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)850worker2 # [ 0.211811] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)851worker2 # [ 0.211847] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)852worker2 # [ 0.211943] NET: Registered PF_UNIX/PF_LOCAL protocol family853worker2 # [ 0.211966] NET: Registered PF_XDP protocol family854worker2 # [ 0.211985] PCI: CLS 0 bytes, default 64855worker2 # [ 0.212257] Trying to unpack rootfs image as initramfs...856lb # [ 0.233069] kvm [1]: HYP mode not available857worker1 # [ 0.213802] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)858worker1 # [ 0.213860] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)859worker1 # [ 0.213893] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)860worker1 # [ 0.213975] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)861worker1 # [ 0.214118] TCP: Hash tables configured (established 16384 bind 16384)862worker1 # [ 0.214206] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)863worker1 # [ 0.214242] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)864worker1 # [ 0.214279] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)865worker1 # [ 0.214390] NET: Registered PF_UNIX/PF_LOCAL protocol family866worker1 # [ 0.214414] NET: Registered PF_XDP protocol family867worker1 # [ 0.214432] PCI: CLS 0 bytes, default 64868worker2 # [ 0.233712] kvm [1]: HYP mode not available869worker1 # [ 0.214726] Trying to unpack rootfs image as initramfs...870worker1 # [ 0.230960] kvm [1]: HYP mode not available871client # [ 0.337485] Initialise system trusted keyrings872lb # [ 0.333213] Initialise system trusted keyrings873client # [ 0.338254] workingset: timestamp_bits=42 max_order=19 bucket_order=0874lb # [ 0.333992] workingset: timestamp_bits=42 max_order=19 bucket_order=0875client # [ 0.339613] squashfs: version 4.0 (2009/01/31) Phillip Lougher876lb # [ 0.335302] squashfs: version 4.0 (2009/01/31) Phillip Lougher877lb # [ 0.336119] 9p: Installing v9fs 9p2000 file system support878client # [ 0.340404] 9p: Installing v9fs 9p2000 file system support879worker2 # [ 0.332064] Initialise system trusted keyrings880worker2 # [ 0.333173] workingset: timestamp_bits=42 max_order=19 bucket_order=0881worker2 # [ 0.334592] squashfs: version 4.0 (2009/01/31) Phillip Lougher882worker1 # [ 0.328819] Initialise system trusted keyrings883worker2 # [ 0.335361] 9p: Installing v9fs 9p2000 file system support884client # [ 0.360399] Key type asymmetric registered885client # [ 0.360429] Asymmetric key parser 'x509' registered886client # [ 0.360511] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242)887worker1 # [ 0.337138] workingset: timestamp_bits=42 max_order=19 bucket_order=0888client # [ 0.366726] io scheduler mq-deadline registered889client # [ 0.366747] io scheduler kyber registered890worker1 # [ 0.338587] squashfs: version 4.0 (2009/01/31) Phillip Lougher891lb # [ 0.364194] Key type asymmetric registered892lb # [ 0.364229] Asymmetric key parser 'x509' registered893lb # [ 0.364316] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242)894lb # [ 0.366418] io scheduler mq-deadline registered895lb # [ 0.366429] io scheduler kyber registered896worker2 # [ 0.355435] Key type asymmetric registered897worker2 # [ 0.355465] Asymmetric key parser 'x509' registered898worker1 # [ 0.339377] 9p: Installing v9fs 9p2000 file system support899worker2 # [ 0.355547] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242)900worker2 # [ 0.361736] io scheduler mq-deadline registered901worker2 # [ 0.361757] io scheduler kyber registered902client # [ 0.378845] pl061_gpio 9030000.pl061: PL061 GPIO chip registered903worker1 # [ 0.352045] Key type asymmetric registered904worker1 # [ 0.352059] Asymmetric key parser 'x509' registered905worker1 # [ 0.352105] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242)906client # [ 0.380298] ledtrig-cpu: registered to indicate activity on CPUs907client # [ 0.380735] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:908client # [ 0.380756] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000909lb # [ 0.379976] pl061_gpio 9030000.pl061: PL061 GPIO chip registered910client # [ 0.380767] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000911client # [ 0.380776] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000912client # [ 0.380814] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits913client # [ 0.380846] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]914client # [ 0.380922] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00915worker1 # [ 0.361157] io scheduler mq-deadline registered916client # [ 0.380931] pci_bus 0000:00: root bus resource [bus 00-ff]917worker1 # [ 0.361179] io scheduler kyber registered918client # [ 0.380938] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]919worker2 # [ 0.373813] pl061_gpio 9030000.pl061: PL061 GPIO chip registered920client # [ 0.380943] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]921lb # [ 0.381364] ledtrig-cpu: registered to indicate activity on CPUs922client # [ 0.380948] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]923lb # [ 0.381779] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:924client # [ 0.381039] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint925lb # [ 0.381799] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000926worker2 # [ 0.375278] ledtrig-cpu: registered to indicate activity on CPUs927client # [ 0.381480] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint928lb # [ 0.381812] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000929client # [ 0.381668] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f]930worker2 # [ 0.375666] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:931lb # [ 0.381820] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000932client # [ 0.381684] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff]933worker2 # [ 0.375683] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000934client # [ 0.381714] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]935lb # [ 0.381843] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits936worker2 # [ 0.375696] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000937client # [ 0.381730] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref]938lb # [ 0.381865] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]939client # [ 0.382183] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint940worker2 # [ 0.375704] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000941lb # [ 0.381953] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00942client # [ 0.382367] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f]943lb # [ 0.381962] pci_bus 0000:00: root bus resource [bus 00-ff]944worker2 # [ 0.375726] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits945client # [ 0.382383] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff]946lb # [ 0.381968] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]947client # [ 0.382413] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]948worker2 # [ 0.375751] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]949lb # [ 0.381973] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]950worker2 # [ 0.375842] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00951lb # [ 0.381978] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]952worker2 # [ 0.375856] pci_bus 0000:00: root bus resource [bus 00-ff]953worker1 # [ 0.373228] pl061_gpio 9030000.pl061: PL061 GPIO chip registered954client # [ 0.401787] pci 0000:00:03.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint955lb # [ 0.382067] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint956worker2 # [ 0.375862] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]957client # [ 0.401972] pci 0000:00:03.0: BAR 0 [io 0x0000-0x003f]958worker2 # [ 0.375867] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]959lb # [ 0.382500] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint960client # [ 0.401987] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff]961lb # [ 0.382705] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f]962worker2 # [ 0.375873] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]963client # [ 0.402017] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]964lb # [ 0.382723] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff]965worker2 # [ 0.375929] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint966client # [ 0.402483] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint967worker1 # [ 0.374608] ledtrig-cpu: registered to indicate activity on CPUs968lb # [ 0.382753] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]969client # [ 0.402662] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f]970worker2 # [ 0.376369] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint971lb # [ 0.382770] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref]972worker1 # [ 0.375045] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:973worker2 # [ 0.376556] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f]974lb # [ 0.383238] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint975worker1 # [ 0.375063] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000976worker2 # [ 0.376572] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff]977lb # [ 0.383452] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f]978worker2 # [ 0.376602] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]979worker1 # [ 0.375075] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000980lb # [ 0.383469] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff]981worker2 # [ 0.376618] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref]982lb # [ 0.383500] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]983worker1 # [ 0.375083] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000984worker2 # [ 0.377084] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint985lb # [ 0.383952] pci 0000:00:03.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint986worker1 # [ 0.375106] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits987worker2 # [ 0.377269] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f]988lb # [ 0.384135] pci 0000:00:03.0: BAR 0 [io 0x0000-0x003f]989worker2 # [ 0.377285] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff]990worker1 # [ 0.375130] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]991lb # [ 0.384152] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff]992worker2 # [ 0.377315] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]993worker1 # [ 0.375218] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00994lb # [ 0.384182] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]995worker1 # [ 0.375231] pci_bus 0000:00: root bus resource [bus 00-ff]996lb # [ 0.384643] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint997worker1 # [ 0.375242] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]998lb # [ 0.384826] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f]999worker2 # [ 0.396354] pci 0000:00:03.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1000worker1 # [ 0.375247] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]1001lb # [ 0.384843] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff]1002worker2 # [ 0.396542] pci 0000:00:03.0: BAR 0 [io 0x0000-0x003f]1003worker1 # [ 0.375252] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]1004lb # [ 0.384877] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1005worker2 # [ 0.396558] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff]1006worker1 # [ 0.375311] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint1007worker2 # [ 0.396587] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1008lb # [ 0.385339] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1009lb # [ 0.385521] pci 0000:00:05.0: BAR 0 [io 0x0000-0x001f]1010worker1 # [ 0.375763] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1011worker2 # [ 0.397055] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1012lb # [ 0.385538] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff]1013worker1 # [ 0.375950] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f]1014worker2 # [ 0.397239] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f]1015lb # [ 0.385568] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1016worker1 # [ 0.375966] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff]1017worker2 # [ 0.397255] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff]1018worker1 # [ 0.375996] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1019lb # [ 0.386023] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1020worker2 # [ 0.397287] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1021lb # [ 0.386207] pci 0000:00:06.0: BAR 0 [io 0x0000-0x007f]1022worker1 # [ 0.376012] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref]1023lb # [ 0.386228] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff]1024worker1 # [ 0.376470] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1025lb # [ 0.386259] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1026worker1 # [ 0.376656] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f]1027client # [ 0.414771] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff]1028worker1 # [ 0.376671] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff]1029lb # [ 0.386719] pci 0000:00:07.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1030client # [ 0.414816] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1031lb # [ 0.386900] pci 0000:00:07.0: BAR 0 [io 0x0000-0x001f]1032worker1 # [ 0.376701] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1033client # [ 0.415345] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1034lb # [ 0.386916] pci 0000:00:07.0: BAR 1 [mem 0x00000000-0x00000fff]1035client # [ 0.415554] pci 0000:00:05.0: BAR 0 [io 0x0000-0x001f]1036lb # [ 0.386946] pci 0000:00:07.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1037client # [ 0.415571] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff]1038lb # [ 0.386963] pci 0000:00:07.0: ROM [mem 0x00000000-0x0003ffff pref]1039worker1 # [ 0.395932] pci 0000:00:03.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1040client # [ 0.415600] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1041worker1 # [ 0.396122] pci 0000:00:03.0: BAR 0 [io 0x0000-0x003f]1042worker1 # [ 0.396138] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff]1043client # [ 0.416061] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1044client # [ 0.416244] pci 0000:00:06.0: BAR 0 [io 0x0000-0x007f]1045worker1 # [ 0.396168] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1046client # [ 0.416260] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff]1047worker1 # [ 0.396635] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1048client # [ 0.416290] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1049worker1 # [ 0.396816] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f]1050worker1 # [ 0.396831] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff]1051client # [ 0.416746] pci 0000:00:07.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1052client # [ 0.416930] pci 0000:00:07.0: BAR 0 [io 0x0000-0x001f]1053worker1 # [ 0.396863] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1054client # [ 0.416945] pci 0000:00:07.0: BAR 1 [mem 0x00000000-0x00000fff]1055client # [ 0.416975] pci 0000:00:07.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1056client # [ 0.416991] pci 0000:00:07.0: ROM [mem 0x00000000-0x0003ffff pref]1057client # [ 0.417480] pci 0000:00:08.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint1058worker2 # [ 0.409845] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1059client # [ 0.417666] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff]1060worker2 # [ 0.410061] pci 0000:00:05.0: BAR 0 [io 0x0000-0x001f]1061worker2 # [ 0.410078] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff]1062client # [ 0.417696] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1063lb # [ 0.427550] pci 0000:00:08.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint1064worker2 # [ 0.410107] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1065lb # [ 0.427777] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff]1066client # [ 0.418149] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1067client # [ 0.418337] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff]1068lb # [ 0.427811] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1069worker2 # [ 0.410649] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1070worker2 # [ 0.410834] pci 0000:00:06.0: BAR 0 [io 0x0000-0x007f]1071client # [ 0.418366] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1072lb # [ 0.428358] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1073worker2 # [ 0.410850] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff]1074lb # [ 0.428547] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff]1075client # [ 0.418775] pci 0000:00:0a.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint1076worker2 # [ 0.410880] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1077client # [ 0.418958] pci 0000:00:0a.0: BAR 0 [mem 0x00000000-0x00000fff]1078lb # [ 0.428578] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1079worker2 # [ 0.411346] pci 0000:00:07.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1080client # [ 0.419205] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1081lb # [ 0.428979] pci 0000:00:0a.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint1082worker2 # [ 0.411535] pci 0000:00:07.0: BAR 0 [io 0x0000-0x001f]1083client # [ 0.419493] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f]1084lb # [ 0.429161] pci 0000:00:0a.0: BAR 0 [mem 0x00000000-0x00000fff]1085worker2 # [ 0.411551] pci 0000:00:07.0: BAR 1 [mem 0x00000000-0x00000fff]1086client # [ 0.419510] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff]1087lb # [ 0.429406] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1088worker2 # [ 0.411580] pci 0000:00:07.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1089client # [ 0.419558] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1090lb # [ 0.429667] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f]1091worker2 # [ 0.411596] pci 0000:00:07.0: ROM [mem 0x00000000-0x0003ffff pref]1092lb # [ 0.429686] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff]1093client # [ 0.420033] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1094worker2 # [ 0.412095] pci 0000:00:08.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint1095client # [ 0.420215] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f]1096lb # [ 0.429717] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1097worker2 # [ 0.412284] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff]1098client # [ 0.420231] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff]1099lb # [ 0.430177] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1100worker1 # [ 0.409434] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1101worker2 # [ 0.412314] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1102client # [ 0.420267] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1103lb # [ 0.430360] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f]1104worker1 # [ 0.409641] pci 0000:00:05.0: BAR 0 [io 0x0000-0x001f]1105client # [ 0.420861] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned1106worker2 # [ 0.412790] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1107lb # [ 0.430376] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff]1108worker1 # [ 0.409658] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff]1109worker2 # [ 0.412978] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff]1110client # [ 0.420872] pci 0000:00:07.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned1111worker1 # [ 0.409687] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1112lb # [ 0.430406] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1113worker2 # [ 0.413008] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1114lb # [ 0.431023] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned1115client # [ 0.420878] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned1116worker1 # [ 0.410148] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1117worker2 # [ 0.413396] pci 0000:00:0a.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint1118lb # [ 0.431035] pci 0000:00:07.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned1119worker1 # [ 0.410334] pci 0000:00:06.0: BAR 0 [io 0x0000-0x007f]1120client # [ 0.420922] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned1121worker1 # [ 0.410349] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff]1122worker2 # [ 0.413579] pci 0000:00:0a.0: BAR 0 [mem 0x00000000-0x00000fff]1123lb # [ 0.431041] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned1124client # [ 0.420969] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned1125worker1 # [ 0.410379] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1126worker2 # [ 0.413843] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1127lb # [ 0.431088] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned1128client # [ 0.421016] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned1129worker2 # [ 0.414116] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f]1130worker1 # [ 0.410895] pci 0000:00:07.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1131lb # [ 0.431137] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned1132client # [ 0.421062] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned1133worker2 # [ 0.414133] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff]1134worker1 # [ 0.411080] pci 0000:00:07.0: BAR 0 [io 0x0000-0x001f]1135lb # [ 0.431185] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned1136worker2 # [ 0.414163] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1137client # [ 0.421107] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned1138worker1 # [ 0.411095] pci 0000:00:07.0: BAR 1 [mem 0x00000000-0x00000fff]1139worker1 # [ 0.411124] pci 0000:00:07.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1140lb # [ 0.431234] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned1141client # [ 0.421153] pci 0000:00:07.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned1142worker2 # [ 0.414637] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1143worker1 # [ 0.411140] pci 0000:00:07.0: ROM [mem 0x00000000-0x0003ffff pref]1144worker2 # [ 0.414831] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f]1145lb # [ 0.431283] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned1146client # [ 0.421199] pci 0000:00:08.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned1147worker2 # [ 0.414847] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff]1148worker1 # [ 0.411610] pci 0000:00:08.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint1149client # [ 0.421245] pci 0000:00:09.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned1150worker2 # [ 0.414876] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1151worker1 # [ 0.411801] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff]1152client # [ 0.421291] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned1153worker2 # [ 0.415572] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned1154worker1 # [ 0.411831] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1155client # [ 0.421352] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned1156worker2 # [ 0.415587] pci 0000:00:07.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned1157worker1 # [ 0.412289] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1158client # [ 0.421398] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned1159worker1 # [ 0.412477] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff]1160worker2 # [ 0.415593] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned1161client # [ 0.421420] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned1162worker1 # [ 0.412506] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1163worker2 # [ 0.415641] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned1164client # [ 0.421441] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned1165worker1 # [ 0.412896] pci 0000:00:0a.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint1166worker2 # [ 0.415690] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned1167client # [ 0.421462] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned1168worker1 # [ 0.413100] pci 0000:00:0a.0: BAR 0 [mem 0x00000000-0x00000fff]1169worker2 # [ 0.415741] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned1170client # [ 0.421486] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned1171worker1 # [ 0.413355] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1172client # [ 0.421507] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned1173worker2 # [ 0.415792] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned1174worker1 # [ 0.413630] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f]1175client # [ 0.421529] pci 0000:00:07.0: BAR 1 [mem 0x10086000-0x10086fff]: assigned1176worker2 # [ 0.415839] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned1177worker1 # [ 0.413647] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff]1178lb # [ 0.455354] pci 0000:00:07.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned1179client # [ 0.421551] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned1180worker1 # [ 0.413676] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1181worker2 # [ 0.415886] pci 0000:00:07.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned1182lb # [ 0.455429] pci 0000:00:08.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned1183client # [ 0.421573] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned1184worker1 # [ 0.414149] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1185worker2 # [ 0.415933] pci 0000:00:08.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned1186client # [ 0.421595] pci 0000:00:0a.0: BAR 0 [mem 0x10089000-0x10089fff]: assigned1187lb # [ 0.455478] pci 0000:00:09.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned1188worker1 # [ 0.414336] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f]1189worker2 # [ 0.415983] pci 0000:00:09.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned1190client # [ 0.421617] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned1191lb # [ 0.455527] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned1192worker1 # [ 0.414352] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff]1193client # [ 0.421640] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned1194worker2 # [ 0.416032] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned1195lb # [ 0.455627] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned1196worker1 # [ 0.414384] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1197client # [ 0.421662] pci 0000:00:06.0: BAR 0 [io 0x1000-0x107f]: assigned1198worker2 # [ 0.416108] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned1199lb # [ 0.455677] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned1200worker1 # [ 0.414990] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned1201client # [ 0.421684] pci 0000:00:03.0: BAR 0 [io 0x1080-0x10bf]: assigned1202worker2 # [ 0.416156] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned1203lb # [ 0.455701] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned1204worker1 # [ 0.415001] pci 0000:00:07.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned1205client # [ 0.421706] pci 0000:00:0b.0: BAR 0 [io 0x10c0-0x10ff]: assigned1206worker2 # [ 0.416180] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned1207lb # [ 0.455724] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned1208client # [ 0.421727] pci 0000:00:01.0: BAR 0 [io 0x1100-0x111f]: assigned1209worker1 # [ 0.415007] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned1210worker2 # [ 0.416202] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned1211lb # [ 0.455747] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned1212client # [ 0.421749] pci 0000:00:02.0: BAR 0 [io 0x1120-0x113f]: assigned1213worker1 # [ 0.415052] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned1214client # [ 0.421771] pci 0000:00:04.0: BAR 0 [io 0x1140-0x115f]: assigned1215worker2 # [ 0.416225] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned1216lb # [ 0.455774] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned1217worker1 # [ 0.415100] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned1218client # [ 0.421792] pci 0000:00:05.0: BAR 0 [io 0x1160-0x117f]: assigned1219worker2 # [ 0.416250] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned1220lb # [ 0.455797] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned1221client # [ 0.421814] pci 0000:00:07.0: BAR 0 [io 0x1180-0x119f]: assigned1222worker2 # [ 0.416272] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned1223worker1 # [ 0.415148] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned1224lb # [ 0.455820] pci 0000:00:07.0: BAR 1 [mem 0x10086000-0x10086fff]: assigned1225client # [ 0.421836] pci 0000:00:0c.0: BAR 0 [io 0x11a0-0x11bf]: assigned1226worker2 # [ 0.416294] pci 0000:00:07.0: BAR 1 [mem 0x10086000-0x10086fff]: assigned1227lb # [ 0.455842] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned1228worker1 # [ 0.415195] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned1229client # [ 0.421866] pci_bus 0000:00: resource 4 [io 0x0000-0xffff]1230lb # [ 0.455865] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned1231worker2 # [ 0.416316] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned1232client # [ 0.421876] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff]1233worker1 # [ 0.415243] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned1234lb # [ 0.455887] pci 0000:00:0a.0: BAR 0 [mem 0x10089000-0x10089fff]: assigned1235worker2 # [ 0.416342] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned1236client # [ 0.421881] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff]1237worker1 # [ 0.415294] pci 0000:00:07.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned1238lb # [ 0.455911] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned1239worker2 # [ 0.416365] pci 0000:00:0a.0: BAR 0 [mem 0x10089000-0x10089fff]: assigned1240worker1 # [ 0.415340] pci 0000:00:08.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned1241lb # [ 0.455935] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned1242worker2 # [ 0.416388] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned1243lb # [ 0.455957] pci 0000:00:06.0: BAR 0 [io 0x1000-0x107f]: assigned1244worker1 # [ 0.415386] pci 0000:00:09.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned1245worker2 # [ 0.416411] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned1246lb # [ 0.455979] pci 0000:00:03.0: BAR 0 [io 0x1080-0x10bf]: assigned1247worker2 # [ 0.416432] pci 0000:00:06.0: BAR 0 [io 0x1000-0x107f]: assigned1248worker1 # [ 0.415432] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned1249lb # [ 0.456001] pci 0000:00:0b.0: BAR 0 [io 0x10c0-0x10ff]: assigned1250worker2 # [ 0.416454] pci 0000:00:03.0: BAR 0 [io 0x1080-0x10bf]: assigned1251worker1 # [ 0.415503] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned1252lb # [ 0.456023] pci 0000:00:01.0: BAR 0 [io 0x1100-0x111f]: assigned1253worker2 # [ 0.416475] pci 0000:00:0b.0: BAR 0 [io 0x10c0-0x10ff]: assigned1254worker1 # [ 0.415549] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned1255lb # [ 0.456045] pci 0000:00:02.0: BAR 0 [io 0x1120-0x113f]: assigned1256worker2 # [ 0.416496] pci 0000:00:01.0: BAR 0 [io 0x1100-0x111f]: assigned1257lb # [ 0.456068] pci 0000:00:04.0: BAR 0 [io 0x1140-0x115f]: assigned1258worker1 # [ 0.415571] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned1259worker2 # [ 0.416518] pci 0000:00:02.0: BAR 0 [io 0x1120-0x113f]: assigned1260lb # [ 0.456090] pci 0000:00:05.0: BAR 0 [io 0x1160-0x117f]: assigned1261worker1 # [ 0.415593] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned1262worker2 # [ 0.416540] pci 0000:00:04.0: BAR 0 [io 0x1140-0x115f]: assigned1263lb # [ 0.456113] pci 0000:00:07.0: BAR 0 [io 0x1180-0x119f]: assigned1264worker1 # [ 0.415616] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned1265worker2 # [ 0.416562] pci 0000:00:05.0: BAR 0 [io 0x1160-0x117f]: assigned1266lb # [ 0.456139] pci 0000:00:0c.0: BAR 0 [io 0x11a0-0x11bf]: assigned1267worker2 # [ 0.416584] pci 0000:00:07.0: BAR 0 [io 0x1180-0x119f]: assigned1268worker1 # [ 0.415641] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned1269lb # [ 0.456169] pci_bus 0000:00: resource 4 [io 0x0000-0xffff]1270client # [ 0.483112] pci 0000:00:0a.0: enabling device (0000 -> 0002)1271worker2 # [ 0.416606] pci 0000:00:0c.0: BAR 0 [io 0x11a0-0x11bf]: assigned1272worker1 # [ 0.415663] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned1273lb # [ 0.456180] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff]1274worker2 # [ 0.416633] pci_bus 0000:00: resource 4 [io 0x0000-0xffff]1275worker1 # [ 0.415689] pci 0000:00:07.0: BAR 1 [mem 0x10086000-0x10086fff]: assigned1276lb # [ 0.456185] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff]1277worker2 # [ 0.416642] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff]1278worker1 # [ 0.415711] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned1279lb # [ 0.457362] pci 0000:00:0a.0: enabling device (0000 -> 0002)1280worker2 # [ 0.416647] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff]1281worker1 # [ 0.415733] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned1282worker1 # [ 0.415755] pci 0000:00:0a.0: BAR 0 [mem 0x10089000-0x10089fff]: assigned1283worker1 # [ 0.415778] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned1284worker1 # [ 0.415801] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned1285worker1 # [ 0.415822] pci 0000:00:06.0: BAR 0 [io 0x1000-0x107f]: assigned1286worker1 # [ 0.415844] pci 0000:00:03.0: BAR 0 [io 0x1080-0x10bf]: assigned1287worker1 # [ 0.415865] pci 0000:00:0b.0: BAR 0 [io 0x10c0-0x10ff]: assigned1288worker1 # [ 0.415887] pci 0000:00:01.0: BAR 0 [io 0x1100-0x111f]: assigned1289worker1 # [ 0.415908] pci 0000:00:02.0: BAR 0 [io 0x1120-0x113f]: assigned1290worker1 # [ 0.415930] pci 0000:00:04.0: BAR 0 [io 0x1140-0x115f]: assigned1291worker1 # [ 0.415952] pci 0000:00:05.0: BAR 0 [io 0x1160-0x117f]: assigned1292worker1 # [ 0.415974] pci 0000:00:07.0: BAR 0 [io 0x1180-0x119f]: assigned1293worker1 # [ 0.415997] pci 0000:00:0c.0: BAR 0 [io 0x11a0-0x11bf]: assigned1294worker2 # [ 0.477904] pci 0000:00:0a.0: enabling device (0000 -> 0002)1295worker1 # [ 0.416023] pci_bus 0000:00: resource 4 [io 0x0000-0xffff]1296worker1 # [ 0.416033] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff]1297worker1 # [ 0.416037] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff]1298client # [ 0.504024] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)1299worker1 # [ 0.477281] pci 0000:00:0a.0: enabling device (0000 -> 0002)1300client # [ 0.506242] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003)1301lb # [ 0.507054] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)1302worker2 # [ 0.499552] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)1303client # [ 0.517400] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003)1304lb # [ 0.516843] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003)1305lb # [ 0.521382] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003)1306worker1 # [ 0.498408] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)1307worker2 # [ 0.509970] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003)1308worker1 # [ 0.500599] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003)1309client # [ 0.527338] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003)1310client # [ 0.531622] virtio-pci 0000:00:05.0: enabling device (0000 -> 0003)1311client # [ 0.533709] virtio-pci 0000:00:06.0: enabling device (0000 -> 0003)1312lb # [ 0.527530] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003)1313lb # [ 0.529658] virtio-pci 0000:00:05.0: enabling device (0000 -> 0003)1314worker2 # [ 0.517714] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003)1315worker2 # [ 0.520331] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003)1316worker1 # [ 0.511818] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003)1317client # [ 0.545086] virtio-pci 0000:00:07.0: enabling device (0000 -> 0003)1318lb # [ 0.540459] virtio-pci 0000:00:06.0: enabling device (0000 -> 0003)1319lb # [ 0.542573] virtio-pci 0000:00:07.0: enabling device (0000 -> 0003)1320worker1 # [ 0.518031] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003)1321worker2 # [ 0.530281] virtio-pci 0000:00:05.0: enabling device (0000 -> 0003)1322worker2 # [ 0.533143] virtio-pci 0000:00:06.0: enabling device (0000 -> 0003)1323worker1 # [ 0.520124] virtio-pci 0000:00:05.0: enabling device (0000 -> 0003)1324worker2 # [ 0.536804] virtio-pci 0000:00:07.0: enabling device (0000 -> 0003)1325lb # [ 0.554244] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002)1326worker1 # [ 0.531016] virtio-pci 0000:00:06.0: enabling device (0000 -> 0003)1327client # [ 0.555246] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002)1328worker2 # [ 0.545098] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002)1329client # [ 0.559608] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002)1330client # [ 0.561487] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003)1331lb # [ 0.557500] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002)1332worker1 # [ 0.537271] virtio-pci 0000:00:07.0: enabling device (0000 -> 0003)1333worker1 # [ 0.539468] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002)1334client # [ 0.573334] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003)1335worker2 # [ 0.555398] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002)1336lb # [ 0.567901] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003)1337lb # [ 0.570344] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003)1338worker1 # [ 0.549263] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002)1339worker1 # [ 0.551299] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003)1340lb # [ 0.580633] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1341client # [ 0.587945] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1342worker2 # [ 0.566192] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003)1343client # [ 0.590567] msm_serial: driver initialized1344worker2 # [ 0.568623] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003)1345lb # [ 0.583357] msm_serial: driver initialized1346lb # [ 0.583495] SuperH (H)SCI(F) driver initialized1347lb # [ 0.583551] STM32 USART driver initialized1348worker1 # [ 0.561310] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003)1349client # [ 0.591243] SuperH (H)SCI(F) driver initialized1350client # [ 0.591302] STM32 USART driver initialized1351worker2 # [ 0.578964] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1352worker1 # [ 0.570998] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1353worker2 # [ 0.587033] msm_serial: driver initialized1354worker2 # [ 0.587235] SuperH (H)SCI(F) driver initialized1355worker2 # [ 0.587288] STM32 USART driver initialized1356worker1 # [ 0.578034] msm_serial: driver initialized1357worker1 # [ 0.578181] SuperH (H)SCI(F) driver initialized1358worker1 # [ 0.578233] STM32 USART driver initialized1359client # [ 0.623672] loop: module loaded1360client # [ 0.623865] virtio_blk virtio5: 1/0/0 default/read/poll queues1361client # [ 0.624597] virtio_blk virtio5: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB)1362lb # [ 0.620567] loop: module loaded1363lb # [ 0.620759] virtio_blk virtio5: 1/0/0 default/read/poll queues1364lb # [ 0.621491] virtio_blk virtio5: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB)1365worker2 # [ 0.616934] loop: module loaded1366worker2 # [ 0.617144] virtio_blk virtio5: 1/0/0 default/read/poll queues1367client # [ 0.631345] megasas: 07.734.00.00-rc11368client # [ 0.632263] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff]1369client # [ 0.634570] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001370worker2 # [ 0.619689] virtio_blk virtio5: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB)1371client # [ 0.634602] Intel/Sharp Extended Query Table at 0x00311372worker1 # [ 0.610034] loop: module loaded1373worker1 # [ 0.610233] virtio_blk virtio5: 1/0/0 default/read/poll queues1374worker1 # [ 0.611018] virtio_blk virtio5: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB)1375lb # [ 0.632187] megasas: 07.734.00.00-rc11376lb # [ 0.633057] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff]1377lb # [ 0.635921] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001378client # [ 0.643896] Using buffer write method1379lb # [ 0.635946] Intel/Sharp Extended Query Table at 0x00311380lb # [ 0.637708] Using buffer write method1381client # [ 0.643978] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff]1382lb # [ 0.637783] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff]1383worker2 # [ 0.630381] megasas: 07.734.00.00-rc11384worker2 # [ 0.631158] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff]1385worker2 # [ 0.633125] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001386client # [ 0.655203] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001387worker2 # [ 0.633160] Intel/Sharp Extended Query Table at 0x00311388worker1 # [ 0.621761] megasas: 07.734.00.00-rc11389client # [ 0.655237] Intel/Sharp Extended Query Table at 0x00311390client # [ 0.656613] Using buffer write method1391worker1 # [ 0.622525] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff]1392client # [ 0.656642] Concatenating MTD devices:1393worker2 # [ 0.634941] Using buffer write method1394client # [ 0.656646] (0): "0.flash"1395client # [ 0.656651] (1): "0.flash"1396client # [ 0.656654] into device "0.flash"1397worker2 # [ 0.635022] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff]1398worker1 # [ 0.624625] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001399lb # [ 0.651909] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001400worker1 # [ 0.624653] Intel/Sharp Extended Query Table at 0x00311401lb # [ 0.651948] Intel/Sharp Extended Query Table at 0x00311402worker1 # [ 0.626420] Using buffer write method1403worker2 # [ 0.637074] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001404lb # [ 0.653430] Using buffer write method1405lb # [ 0.653460] Concatenating MTD devices:1406worker2 # [ 0.637125] Intel/Sharp Extended Query Table at 0x00311407lb # [ 0.653464] (0): "0.flash"1408lb # [ 0.653469] (1): "0.flash"1409lb # [ 0.653472] into device "0.flash"1410worker1 # [ 0.626490] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff]1411worker1 # [ 0.628432] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001412worker1 # [ 0.628462] Intel/Sharp Extended Query Table at 0x00311413worker2 # [ 0.657896] Using buffer write method1414worker2 # [ 0.657934] Concatenating MTD devices:1415worker2 # [ 0.657944] (0): "0.flash"1416worker2 # [ 0.657948] (1): "0.flash"1417worker2 # [ 0.657951] into device "0.flash"1418worker1 # [ 0.648999] Using buffer write method1419worker1 # [ 0.649039] Concatenating MTD devices:1420worker1 # [ 0.649043] (0): "0.flash"1421worker1 # [ 0.649047] (1): "0.flash"1422worker1 # [ 0.649050] into device "0.flash"1423client # [ 0.936084] Freeing initrd memory: 26160K1424client # [ 0.942419] tun: Universal TUN/TAP device driver, 1.61425client # [ 0.946241] thunder_xcv, ver 1.01426client # [ 0.946299] thunder_bgx, ver 1.01427client # [ 0.946329] nicpf, ver 1.01428lb # [ 0.942394] Freeing initrd memory: 26140K1429client # [ 0.948126] e1000: Intel(R) PRO/1000 Network Driver1430client # [ 0.948138] e1000: Copyright (c) 1999-2006 Intel Corporation.1431client # [ 0.948162] e1000e: Intel(R) PRO/1000 Network Driver1432lb # [ 0.948965] tun: Universal TUN/TAP device driver, 1.61433client # [ 0.948170] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.1434client # [ 0.948198] igb: Intel(R) Gigabit Ethernet Network Driver1435client # [ 0.948204] igb: Copyright (c) 2007-2014 Intel Corporation.1436client # [ 0.948227] igbvf: Intel(R) Gigabit Virtual Function Network Driver1437client # [ 0.948233] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.1438client # [ 0.948370] sky2: driver version 1.301439client # [ 0.950069] usbcore: registered new interface driver usb-storage1440client # [ 0.950159] usbcore: registered new interface driver usbserial_generic1441worker2 # [ 0.942243] Freeing initrd memory: 26148K1442client # [ 0.950173] usbserial: USB Serial support registered for generic1443client # [ 0.951586] ehci-pci 0000:00:0a.0: EHCI Host Controller1444lb # [ 0.952920] thunder_xcv, ver 1.01445client # [ 0.951620] ehci-pci 0000:00:0a.0: new USB bus registered, assigned bus number 11446lb # [ 0.952969] thunder_bgx, ver 1.01447lb # [ 0.952992] nicpf, ver 1.01448client # [ 0.952338] ehci-pci 0000:00:0a.0: irq 15, io mem 0x100890001449lb # [ 0.953586] e1000: Intel(R) PRO/1000 Network Driver1450lb # [ 0.953597] e1000: Copyright (c) 1999-2006 Intel Corporation.1451client # [ 0.962909] hv_vmbus: registering driver hyperv_keyboard1452lb # [ 0.953625] e1000e: Intel(R) PRO/1000 Network Driver1453client # [ 0.964301] ehci-pci 0000:00:0a.0: USB 2.0 started, EHCI 1.001454lb # [ 0.953633] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.1455lb # [ 0.953663] igb: Intel(R) Gigabit Ethernet Network Driver1456client # [ 0.965386] hub 1-0:1.0: USB hub found1457worker1 # [ 0.934978] Freeing initrd memory: 26148K1458client # [ 0.965868] hub 1-0:1.0: 6 ports detected1459lb # [ 0.953668] igb: Copyright (c) 2007-2014 Intel Corporation.1460worker2 # [ 0.948781] tun: Universal TUN/TAP device driver, 1.61461lb # [ 0.953696] igbvf: Intel(R) Gigabit Virtual Function Network Driver1462lb # [ 0.953702] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.1463lb # [ 0.953844] sky2: driver version 1.301464client # [ 0.967016] rtc-pl031 9010000.pl031: registered as rtc01465lb # [ 0.956142] ehci-pci 0000:00:0a.0: EHCI Host Controller1466client # [ 0.967052] rtc-pl031 9010000.pl031: setting system clock to 2026-09-21T12:08:31 UTC (1789992511)1467lb # [ 0.956170] ehci-pci 0000:00:0a.0: new USB bus registered, assigned bus number 11468client # [ 0.967365] i2c_dev: i2c /dev entries driver1469lb # [ 0.956359] ehci-pci 0000:00:0a.0: irq 15, io mem 0x100890001470worker1 # [ 0.941294] tun: Universal TUN/TAP device driver, 1.61471worker2 # [ 0.952554] thunder_xcv, ver 1.01472worker2 # [ 0.952607] thunder_bgx, ver 1.01473worker2 # [ 0.952635] nicpf, ver 1.01474worker2 # [ 0.953182] e1000: Intel(R) PRO/1000 Network Driver1475worker2 # [ 0.953188] e1000: Copyright (c) 1999-2006 Intel Corporation.1476worker2 # [ 0.953213] e1000e: Intel(R) PRO/1000 Network Driver1477lb # [ 0.966087] usbcore: registered new interface driver usb-storage1478worker1 # [ 0.945007] thunder_xcv, ver 1.01479worker2 # [ 0.953220] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.1480lb # [ 0.966221] usbcore: registered new interface driver usbserial_generic1481worker1 # [ 0.945060] thunder_bgx, ver 1.01482worker2 # [ 0.953249] igb: Intel(R) Gigabit Ethernet Network Driver1483lb # [ 0.966237] usbserial: USB Serial support registered for generic1484client # [ 0.972473] sdhci: Secure Digital Host Controller Interface driver1485worker2 # [ 0.953255] igb: Copyright (c) 2007-2014 Intel Corporation.1486client # [ 0.972488] sdhci: Copyright(c) Pierre Ossman1487lb # [ 0.966891] hv_vmbus: registering driver hyperv_keyboard1488worker2 # [ 0.953278] igbvf: Intel(R) Gigabit Virtual Function Network Driver1489client # [ 0.972751] Synopsys Designware Multimedia Card Interface Driver1490lb # [ 0.967383] ehci-pci 0000:00:0a.0: USB 2.0 started, EHCI 1.001491client # [ 0.973136] sdhci-pltfm: SDHCI platform and OF driver helper1492worker2 # [ 0.953284] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.1493lb # [ 0.967734] hub 1-0:1.0: USB hub found1494worker2 # [ 0.953431] sky2: driver version 1.301495lb # [ 0.967754] hub 1-0:1.0: 6 ports detected1496client # [ 0.977510] hid: raw HID events driver (C) Jiri Kosina1497worker1 # [ 0.945946] nicpf, ver 1.01498client # [ 0.977809] usbcore: registered new interface driver usbhid1499worker1 # [ 0.946508] e1000: Intel(R) PRO/1000 Network Driver1500client # [ 0.977820] usbhid: USB HID core driver1501lb # [ 0.973416] rtc-pl031 9010000.pl031: registered as rtc01502worker1 # [ 0.946515] e1000: Copyright (c) 1999-2006 Intel Corporation.1503worker1 # [ 0.946546] e1000e: Intel(R) PRO/1000 Network Driver1504lb # [ 0.973444] rtc-pl031 9010000.pl031: setting system clock to 2026-09-21T12:08:31 UTC (1789992511)1505worker1 # [ 0.946554] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.1506lb # [ 0.973814] i2c_dev: i2c /dev entries driver1507worker2 # [ 0.962243] usbcore: registered new interface driver usb-storage1508worker1 # [ 0.946583] igb: Intel(R) Gigabit Ethernet Network Driver1509worker2 # [ 0.962336] usbcore: registered new interface driver usbserial_generic1510worker1 # [ 0.946589] igb: Copyright (c) 2007-2014 Intel Corporation.1511client # [ 0.980770] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available1512worker2 # [ 0.962355] usbserial: USB Serial support registered for generic1513worker1 # [ 0.946612] igbvf: Intel(R) Gigabit Virtual Function Network Driver1514client # [ 0.982371] drop_monitor: Initializing network drop monitor service1515worker2 # [ 0.962974] hv_vmbus: registering driver hyperv_keyboard1516worker1 # [ 0.946621] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.1517client # [ 0.982568] NET: Registered PF_INET6 protocol family1518lb # [ 0.979120] sdhci: Secure Digital Host Controller Interface driver1519worker2 # [ 0.964286] ehci-pci 0000:00:0a.0: EHCI Host Controller1520worker1 # [ 0.946799] sky2: driver version 1.301521lb # [ 0.979135] sdhci: Copyright(c) Pierre Ossman1522worker1 # [ 0.948473] usbcore: registered new interface driver usb-storage1523worker2 # [ 0.964322] ehci-pci 0000:00:0a.0: new USB bus registered, assigned bus number 11524client # [ 0.985542] Segment Routing with IPv61525client # [ 0.985567] In-situ OAM (IOAM) with IPv61526worker2 # [ 0.964555] ehci-pci 0000:00:0a.0: irq 15, io mem 0x100890001527worker1 # [ 0.948568] usbcore: registered new interface driver usbserial_generic1528client # [ 0.985596] NET: Registered PF_PACKET protocol family1529worker1 # [ 0.948584] usbserial: USB Serial support registered for generic1530worker1 # [ 0.949845] ehci-pci 0000:00:0a.0: EHCI Host Controller1531worker2 # [ 0.969889] rtc-pl031 9010000.pl031: registered as rtc01532worker1 # [ 0.949872] ehci-pci 0000:00:0a.0: new USB bus registered, assigned bus number 11533lb # [ 0.980675] Synopsys Designware Multimedia Card Interface Driver1534worker1 # [ 0.950141] ehci-pci 0000:00:0a.0: irq 15, io mem 0x100890001535lb # [ 0.981084] sdhci-pltfm: SDHCI platform and OF driver helper1536worker2 # [ 0.969920] rtc-pl031 9010000.pl031: setting system clock to 2026-09-21T12:08:31 UTC (1789992511)1537worker2 # [ 0.970252] i2c_dev: i2c /dev entries driver1538lb # [ 0.983082] hid: raw HID events driver (C) Jiri Kosina1539worker1 # [ 0.960996] hv_vmbus: registering driver hyperv_keyboard1540client # [ 0.987435] 9pnet: Installing 9P2000 support1541worker1 # [ 0.961764] ehci-pci 0000:00:0a.0: USB 2.0 started, EHCI 1.001542client # [ 0.990089] Key type dns_resolver registered1543worker1 # [ 0.962880] hub 1-0:1.0: USB hub found1544lb # [ 0.985485] usbcore: registered new interface driver usbhid1545worker1 # [ 0.963377] hub 1-0:1.0: 6 ports detected1546lb # [ 0.985495] usbhid: USB HID core driver1547worker2 # [ 0.975225] sdhci: Secure Digital Host Controller Interface driver1548worker1 # [ 0.965129] rtc-pl031 9010000.pl031: registered as rtc01549worker2 # [ 0.975240] sdhci: Copyright(c) Pierre Ossman1550lb # [ 0.988123] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available1551worker2 # [ 0.975510] Synopsys Designware Multimedia Card Interface Driver1552worker1 # [ 0.965157] rtc-pl031 9010000.pl031: setting system clock to 2026-09-21T12:08:31 UTC (1789992511)1553worker1 # [ 0.965455] i2c_dev: i2c /dev entries driver1554worker2 # [ 0.975901] sdhci-pltfm: SDHCI platform and OF driver helper1555lb # [ 0.990112] drop_monitor: Initializing network drop monitor service1556lb # [ 0.990335] NET: Registered PF_INET6 protocol family1557worker2 # [ 0.980337] ehci-pci 0000:00:0a.0: USB 2.0 started, EHCI 1.001558lb # [ 0.993612] Segment Routing with IPv61559client # [ 0.997214] registered taskstats version 11560worker2 # [ 0.981379] hub 1-0:1.0: USB hub found1561lb # [ 0.993645] In-situ OAM (IOAM) with IPv61562worker2 # [ 0.981400] hub 1-0:1.0: 6 ports detected1563client # [ 0.997376] Loading compiled-in X.509 certificates1564lb # [ 0.993675] NET: Registered PF_PACKET protocol family1565lb # [ 0.995323] 9pnet: Installing 9P2000 support1566worker2 # [ 0.982735] hid: raw HID events driver (C) Jiri Kosina1567worker2 # [ 0.982979] usbcore: registered new interface driver usbhid1568worker1 # [ 0.970469] sdhci: Secure Digital Host Controller Interface driver1569worker2 # [ 0.982986] usbhid: USB HID core driver1570worker1 # [ 0.970489] sdhci: Copyright(c) Pierre Ossman1571worker1 # [ 0.970808] Synopsys Designware Multimedia Card Interface Driver1572worker1 # [ 0.971235] sdhci-pltfm: SDHCI platform and OF driver helper1573lb # [ 0.998334] Key type dns_resolver registered1574worker1 # [ 0.972983] hid: raw HID events driver (C) Jiri Kosina1575worker2 # [ 0.985934] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available1576worker1 # [ 0.976612] usbcore: registered new interface driver usbhid1577worker2 # [ 0.987481] drop_monitor: Initializing network drop monitor service1578worker1 # [ 0.976625] usbhid: USB HID core driver1579client # [ 1.006262] Demotion targets for Node 0: null1580worker2 # [ 0.987650] NET: Registered PF_INET6 protocol family1581client # [ 1.006412] Key type .fscrypt registered1582client # [ 1.006423] Key type fscrypt-provisioning registered1583worker2 # [ 0.990718] Segment Routing with IPv61584worker2 # [ 0.990741] In-situ OAM (IOAM) with IPv61585client # [ 1.006537] ima: No TPM chip found, activating TPM-bypass!1586worker2 # [ 0.990771] NET: Registered PF_PACKET protocol family1587client # [ 1.006562] ima: Allocated hash algorithm: sha11588client # [ 1.006586] ima: No architecture policies found1589worker1 # [ 0.978975] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available1590worker2 # [ 0.992449] 9pnet: Installing 9P2000 support1591client # [ 1.010741] input: gpio-keys as /devices/platform/gpio-keys/input/input01592worker1 # [ 0.980554] drop_monitor: Initializing network drop monitor service1593worker1 # [ 0.980708] NET: Registered PF_INET6 protocol family1594lb # [ 1.004946] registered taskstats version 11595worker1 # [ 0.983933] Segment Routing with IPv61596lb # [ 1.005124] Loading compiled-in X.509 certificates1597worker1 # [ 0.983951] In-situ OAM (IOAM) with IPv61598worker1 # [ 0.984003] NET: Registered PF_PACKET protocol family1599worker2 # [ 0.995539] Key type dns_resolver registered1600worker1 # [ 0.985880] 9pnet: Installing 9P2000 support1601worker1 # [ 0.988045] Key type dns_resolver registered1602lb # [ 1.014350] Demotion targets for Node 0: null1603lb # [ 1.014476] Key type .fscrypt registered1604lb # [ 1.014482] Key type fscrypt-provisioning registered1605lb # [ 1.014602] ima: No TPM chip found, activating TPM-bypass!1606lb # [ 1.014625] ima: Allocated hash algorithm: sha11607lb # [ 1.014648] ima: No architecture policies found1608worker2 # [ 1.002116] registered taskstats version 11609worker2 # [ 1.002278] Loading compiled-in X.509 certificates1610lb # [ 1.018650] input: gpio-keys as /devices/platform/gpio-keys/input/input01611worker1 # [ 0.995265] registered taskstats version 11612worker1 # [ 0.995456] Loading compiled-in X.509 certificates1613worker2 # [ 1.011424] Demotion targets for Node 0: null1614worker2 # [ 1.011557] Key type .fscrypt registered1615worker2 # [ 1.011567] Key type fscrypt-provisioning registered1616client # [ 1.029485] clk: Disabling unused clocks1617worker2 # [ 1.011665] ima: No TPM chip found, activating TPM-bypass!1618client # [ 1.029513] PM: genpd: Disabling unused power domains1619worker2 # [ 1.011690] ima: Allocated hash algorithm: sha11620worker2 # [ 1.011714] ima: No architecture policies found1621worker2 # [ 1.015742] input: gpio-keys as /devices/platform/gpio-keys/input/input01622worker1 # [ 1.004341] Demotion targets for Node 0: null1623worker1 # [ 1.004448] Key type .fscrypt registered1624client # [ 1.033878] Freeing unused kernel memory: 4736K1625client # [ 1.034063] Run /init as init process1626worker1 # [ 1.004454] Key type fscrypt-provisioning registered1627worker1 # [ 1.004553] ima: No TPM chip found, activating TPM-bypass!1628worker1 # [ 1.004581] ima: Allocated hash algorithm: sha11629worker1 # [ 1.004612] ima: No architecture policies found1630worker1 # [ 1.008815] input: gpio-keys as /devices/platform/gpio-keys/input/input01631lb # [ 1.036615] clk: Disabling unused clocks1632lb # [ 1.036642] PM: genpd: Disabling unused power domains1633lb # [ 1.041331] Freeing unused kernel memory: 4736K1634lb # [ 1.041537] Run /init as init process1635client # [ 1.049137] systemd[1]: Successfully made /usr/ read-only.1636worker2 # [ 1.034334] clk: Disabling unused clocks1637worker2 # [ 1.034362] PM: genpd: Disabling unused power domains1638worker2 # [ 1.038920] Freeing unused kernel memory: 4736K1639worker2 # [ 1.039130] Run /init as init process1640worker1 # [ 1.026736] clk: Disabling unused clocks1641worker1 # [ 1.026763] PM: genpd: Disabling unused power domains1642worker1 # [ 1.031243] Freeing unused kernel memory: 4736K1643worker1 # [ 1.031442] Run /init as init process1644lb # [ 1.057730] systemd[1]: Successfully made /usr/ read-only.1645worker2 # [ 1.056207] systemd[1]: Successfully made /usr/ read-only.1646worker1 # [ 1.047409] systemd[1]: Successfully made /usr/ read-only.1647client # [ 1.210790] usb 1-1: new high-speed USB device number 2 using ehci-pci1648lb # [ 1.215403] usb 1-1: new high-speed USB device number 2 using ehci-pci1649worker1 # [ 1.209161] usb 1-1: new high-speed USB device number 2 using ehci-pci1650worker2 # [ 1.225779] usb 1-1: new high-speed USB device number 2 using ehci-pci1651client # [ 1.363127] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:0a.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input11652lb # [ 1.371947] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:0a.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input11653client # [ 1.383857] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1654client # [ 1.389620] systemd[1]: Detected virtualization qemu.1655client # [ 1.390697] systemd[1]: Detected architecture arm64.1656client # [ 1.391680] systemd[1]: Running in initrd.1657client # [ 1.393107] systemd[1]: Initializing machine ID from random generator.1658client # [ 1.394538] systemd[1]: Hostname set to <client>.1659worker1 # [ 1.361428] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:0a.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input11660worker2 # [ 1.377577] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:0a.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input11661lb # [ 1.392869] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1662lb # [ 1.404934] systemd[1]: Detected virtualization qemu.1663lb # [ 1.407013] systemd[1]: Detected architecture arm64.1664lb # [ 1.409035] systemd[1]: Running in initrd.1665worker2 # [ 1.390945] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1666worker2 # [ 1.399365] systemd[1]: Detected virtualization qemu.1667worker2 # [ 1.400750] systemd[1]: Detected architecture arm64.1668lb # [ 1.411974] systemd[1]: Initializing machine ID from random generator.1669worker2 # [ 1.402088] systemd[1]: Running in initrd.1670lb # [ 1.414822] systemd[1]: Hostname set to <lb>.1671worker2 # [ 1.403967] systemd[1]: Initializing machine ID from random generator.1672worker1 # [ 1.382418] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1673worker2 # [ 1.405867] systemd[1]: Hostname set to <worker2>.1674worker1 # [ 1.394828] systemd[1]: Detected virtualization qemu.1675worker1 # [ 1.396909] systemd[1]: Detected architecture arm64.1676worker1 # [ 1.398955] systemd[1]: Running in initrd.1677worker1 # [ 1.401755] systemd[1]: Initializing machine ID from random generator.1678worker1 # [ 1.404714] systemd[1]: Hostname set to <worker1>.1679client # [ 1.451578] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:0a.0-1/input01680lb # [ 1.459613] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:0a.0-1/input01681worker1 # [ 1.449911] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:0a.0-1/input01682worker2 # [ 1.469967] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:0a.0-1/input01683client # [ 1.570756] usb 1-2: new high-speed USB device number 3 using ehci-pci1684lb # [ 1.579407] usb 1-2: new high-speed USB device number 3 using ehci-pci1685worker1 # [ 1.573130] usb 1-2: new high-speed USB device number 3 using ehci-pci1686worker2 # [ 1.589800] usb 1-2: new high-speed USB device number 3 using ehci-pci1687client # [ 1.672073] systemd[1]: bpf-restrict-fs: LSM BPF program attached1688worker2 # [ 1.656238] systemd[1]: bpf-restrict-fs: LSM BPF program attached1689lb # [ 1.686999] systemd[1]: bpf-restrict-fs: LSM BPF program attached1690client # [ 1.726757] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:0a.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input21691client # [ 1.730411] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:0a.0-2/input01692lb # [ 1.740170] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:0a.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input21693worker2 # [ 1.730857] systemd[1]: Queued start job for default target Initrd Default Target.1694worker1 # [ 1.718558] systemd[1]: bpf-restrict-fs: LSM BPF program attached1695lb # [ 1.743624] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:0a.0-2/input01696worker2 # [ 1.740077] systemd[1]: Created slice Slice /system/modprobe.1697worker2 # [ 1.741990] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1698worker2 # [ 1.743238] systemd[1]: Expecting device /dev/disk/by-label/nixos...1699worker2 # [ 1.744074] systemd[1]: Reached target Path Units.1700worker2 # [ 1.744712] systemd[1]: Reached target Slice Units.1701worker2 # [ 1.745912] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:0a.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input21702worker2 # [ 1.746084] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:0a.0-2/input01703worker2 # [ 1.749532] systemd[1]: Reached target Swaps.1704worker1 # [ 1.733203] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:0a.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input21705worker2 # [ 1.750189] systemd[1]: Reached target Timer Units.1706worker2 # [ 1.751128] systemd[1]: Listening on D-Bus System Message Bus Socket.1707worker2 # [ 1.752146] systemd[1]: Listening on Journal Socket (/dev/log).1708worker2 # [ 1.753049] systemd[1]: Listening on Journal Sockets.1709worker1 # [ 1.738836] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:0a.0-2/input01710worker2 # [ 1.753852] systemd[1]: Listening on udev Control Socket.1711worker2 # [ 1.753979] systemd[1]: Listening on udev Kernel Socket.1712worker2 # [ 1.754021] systemd[1]: Reached target Socket Units.1713worker2 # [ 1.757742] systemd[1]: Starting Create List of Static Device Nodes...1714worker2 # [ 1.765555] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1715worker2 # [ 1.766719] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs1716client # [ 1.795202] systemd[1]: Queued start job for default target Initrd Default Target.1717worker2 # [ 1.778988] systemd[1]: Mounting Kernel Configuration File System...1718client # [ 1.804869] systemd[1]: Created slice Slice /system/modprobe.1719client # [ 1.805821] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1720client # [ 1.806846] systemd[1]: Expecting device /dev/disk/by-label/nixos...1721client # [ 1.807668] systemd[1]: Reached target Path Units.1722client # [ 1.808285] systemd[1]: Reached target Slice Units.1723client # [ 1.808910] systemd[1]: Reached target Swaps.1724client # [ 1.809597] systemd[1]: Reached target Timer Units.1725client # [ 1.810400] systemd[1]: Listening on D-Bus System Message Bus Socket.1726lb # [ 1.805942] systemd[1]: Queued start job for default target Initrd Default Target.1727client # [ 1.811435] systemd[1]: Listening on Journal Socket (/dev/log).1728client # [ 1.812337] systemd[1]: Listening on Journal Sockets.1729client # [ 1.813095] systemd[1]: Listening on udev Control Socket.1730client # [ 1.813895] systemd[1]: Listening on udev Kernel Socket.1731client # [ 1.814599] systemd[1]: Reached target Socket Units.1732worker2 # [ 1.798308] systemd[1]: Starting Journal Service...1733client # [ 1.816774] systemd[1]: Starting Create List of Static Device Nodes...1734worker2 # [ 1.800543] systemd[1]: Starting Load Kernel Modules...1735worker2 # [ 1.801429] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1736lb # [ 1.818892] systemd[1]: Created slice Slice /system/modprobe.1737lb # [ 1.820192] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1738client # [ 1.826562] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1739lb # [ 1.821496] systemd[1]: Expecting device /dev/disk/by-label/nixos...1740lb # [ 1.822625] systemd[1]: Reached target Path Units.1741client # [ 1.827633] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs1742lb # [ 1.823471] systemd[1]: Reached target Slice Units.1743lb # [ 1.824298] systemd[1]: Reached target Swaps.1744lb # [ 1.825040] systemd[1]: Reached target Timer Units.1745lb # [ 1.826071] systemd[1]: Listening on D-Bus System Message Bus Socket.1746worker2 # [ 1.814903] systemd[1]: Starting Coldplug All udev Devices...1747lb # [ 1.827301] systemd[1]: Listening on Journal Socket (/dev/log).1748lb # [ 1.828568] systemd[1]: Listening on Journal Sockets.1749lb # [ 1.828780] systemd[1]: Listening on udev Control Socket.1750lb # [ 1.828910] systemd[1]: Listening on udev Kernel Socket.1751lb # [ 1.828934] systemd[1]: Reached target Socket Units.1752lb # [ 1.834114] systemd[1]: Starting Create List of Static Device Nodes...1753worker2 # [ 1.824033] systemd[1]: Finished Create List of Static Device Nodes.1754client # [ 1.840089] systemd[1]: Mounting Kernel Configuration File System...1755lb # [ 1.843555] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1756lb # [ 1.844714] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs1757worker2 # [ 1.835590] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1758client # [ 1.856184] systemd[1]: Starting Journal Service...1759lb # [ 1.851035] systemd[1]: Mounting Kernel Configuration File System...1760worker2 # [ 1.839223] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1761worker2 # [ 1.839567] systemd[1]: Mounted Kernel Configuration File System.1762worker2 # [ 1.846139] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1763client # [ 1.866871] systemd[1]: Starting Load Kernel Modules...1764client # [ 1.867734] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1765worker1 # [ 1.842262] systemd[1]: Queued start job for default target Initrd Default Target.1766client # [ 1.879095] systemd[1]: Starting Coldplug All udev Devices...1767worker1 # [ 1.852044] systemd[1]: Created slice Slice /system/modprobe.1768worker1 # [ 1.853387] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1769worker1 # [ 1.854813] systemd[1]: Expecting device /dev/disk/by-label/nixos...1770lb # [ 1.879269] systemd[1]: Starting Journal Service...1771worker1 # [ 1.855942] systemd[1]: Reached target Path Units.1772worker1 # [ 1.856830] systemd[1]: Reached target Slice Units.1773worker1 # [ 1.857789] systemd[1]: Reached target Swaps.1774worker1 # [ 1.858597] systemd[1]: Reached target Timer Units.1775worker1 # [ 1.859683] systemd[1]: Listening on D-Bus System Message Bus Socket.1776worker1 # [ 1.861009] systemd[1]: Listening on Journal Socket (/dev/log).1777lb # [ 1.884935] systemd[1]: Starting Load Kernel Modules...1778lb # [ 1.885081] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1779worker1 # [ 1.862321] systemd[1]: Listening on Journal Sockets.1780worker1 # [ 1.862482] systemd[1]: Listening on udev Control Socket.1781worker1 # [ 1.862610] systemd[1]: Listening on udev Kernel Socket.1782worker1 # [ 1.862636] systemd[1]: Reached target Socket Units.1783worker1 # [ 1.868278] systemd[1]: Starting Create List of Static Device Nodes...1784lb # [ 1.899973] systemd[1]: Starting Coldplug All udev Devices...1785worker2 # [ 1.888661] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1786worker1 # [ 1.877299] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1787worker2 # [ 1.891911] systemd-journald[73]: Collecting audit messages is disabled.1788worker1 # [ 1.877423] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs1789client # [ 1.910858] systemd[1]: Finished Create List of Static Device Nodes.1790client # [ 1.912001] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1791client # [ 1.912370] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1792worker1 # [ 1.889272] systemd[1]: Mounting Kernel Configuration File System...1793client # [ 1.923996] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1794worker2 # [ 1.901782] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1795worker2 # [ 1.906739] [drm] pci: virtio-gpu-pci detected at 0000:00:08.01796worker2 # [ 1.906994] [drm] features: -virgl +edid -resource_blob -host_visible1797worker2 # [ 1.907004] [drm] features: -context_init1798lb # [ 1.919490] systemd[1]: Finished Create List of Static Device Nodes.1799worker2 # [ 1.907819] [drm] number of scanouts: 11800worker2 # [ 1.907838] [drm] number of cap sets: 01801client # [ 1.928399] systemd-journald[73]: Collecting audit messages is disabled.1802lb # [ 1.920737] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1803lb # [ 1.921104] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1804lb # [ 1.921370] systemd[1]: Mounted Kernel Configuration File System.1805worker2 # [ 1.914344] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1806worker2 # [ 1.917075] systemd[1]: Starting Create Static Device Nodes in /dev...1807client # [ 1.939695] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1808client # [ 1.942838] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1809lb # [ 1.939787] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1810worker1 # [ 1.921953] systemd[1]: Starting Journal Service...1811worker1 # [ 1.924765] systemd[1]: Starting Load Kernel Modules...1812client # [ 1.954772] [drm] pci: virtio-gpu-pci detected at 0000:00:08.01813worker1 # [ 1.925820] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1814client # [ 1.955018] [drm] features: -virgl +edid -resource_blob -host_visible1815client # [ 1.955028] [drm] features: -context_init1816client # [ 1.955788] [drm] number of scanouts: 11817client # [ 1.955806] [drm] number of cap sets: 01818client # [ 1.959374] systemd[1]: Mounted Kernel Configuration File System.1819worker1 # [ 1.932798] systemd[1]: Starting Coldplug All udev Devices...1820worker2 # [ 1.938640] virtio-pci 0000:00:08.0: [drm] Registered 1 planes with drm panic1821worker2 # [ 1.938671] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:08.0 on minor 01822lb # [ 1.961724] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1823client # [ 1.975131] virtio-pci 0000:00:08.0: [drm] Registered 1 planes with drm panic1824client # [ 1.975157] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:08.0 on minor 01825lb # [ 1.975848] systemd-journald[73]: Collecting audit messages is disabled.1826worker2 # [ 1.966783] Console: switching to colour frame buffer device 160x501827worker1 # [ 1.961256] systemd[1]: Finished Create List of Static Device Nodes.1828worker1 # [ 1.962411] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1829worker1 # [ 1.962765] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1830worker1 # [ 1.963009] systemd[1]: Mounted Kernel Configuration File System.1831lb # [ 1.987227] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1832lb # [ 1.992202] [drm] pci: virtio-gpu-pci detected at 0000:00:08.01833lb # [ 1.992542] [drm] features: -virgl +edid -resource_blob -host_visible1834lb # [ 1.992552] [drm] features: -context_init1835lb # [ 1.993310] [drm] number of scanouts: 11836lb # [ 1.993329] [drm] number of cap sets: 01837client # [ 1.999799] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1838worker1 # [ 1.973630] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1839worker2 # [ 1.986412] systemd[1]: Finished Create Static Device Nodes in /dev.1840worker2 # [ 1.987504] systemd[1]: Reached target Preparation for Local File Systems.1841worker2 # [ 1.988376] systemd[1]: Reached target Local File Systems.1842lb # [ 2.000203] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1843client # [ 2.011273] systemd[1]: Starting Create Static Device Nodes in /dev...1844lb # [ 2.007736] systemd[1]: Starting Create Static Device Nodes in /dev...1845worker2 # [ 1.998228] systemd[1]: Starting Rule-based Manager for Device Events and Files...1846worker2 # [ 2.003368] virtio-pci 0000:00:08.0: [drm] fb0: virtio_gpudrmfb frame buffer device1847client # [ 2.015889] Console: switching to colour frame buffer device 160x501848lb # [ 2.019805] virtio-pci 0000:00:08.0: [drm] Registered 1 planes with drm panic1849lb # [ 2.019833] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:08.0 on minor 01850worker1 # [ 2.006377] systemd-journald[73]: Collecting audit messages is disabled.1851worker1 # [ 2.015099] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1852client # [ 2.043340] virtio-pci 0000:00:08.0: [drm] fb0: virtio_gpudrmfb frame buffer device1853worker2 # [ 2.033980] systemd[1]: Finished Load Kernel Modules.1854worker2 # [ 2.037600] systemd[1]: Starting Apply Kernel Variables...1855client # [ 2.055365] systemd[1]: Finished Load Kernel Modules.1856worker1 # [ 2.029051] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1857client # [ 2.058893] systemd[1]: Starting Apply Kernel Variables...1858lb # [ 2.051792] Console: switching to colour frame buffer device 160x501859worker1 # [ 2.037968] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1860worker1 # [ 2.044452] [drm] pci: virtio-gpu-pci detected at 0000:00:08.01861worker1 # [ 2.044698] [drm] features: -virgl +edid -resource_blob -host_visible1862worker1 # [ 2.044709] [drm] features: -context_init1863lb # [ 2.072046] virtio-pci 0000:00:08.0: [drm] fb0: virtio_gpudrmfb frame buffer device1864lb # [ 2.076142] systemd[1]: Finished Create Static Device Nodes in /dev.1865worker1 # [ 2.049501] systemd[1]: Starting Create Static Device Nodes in /dev...1866lb # [ 2.077333] systemd[1]: Reached target Preparation for Local File Systems.1867lb # [ 2.078789] systemd[1]: Reached target Local File Systems.1868worker1 # [ 2.061603] [drm] number of scanouts: 11869worker1 # [ 2.061633] [drm] number of cap sets: 01870worker1 # [ 2.063281] virtio-pci 0000:00:08.0: [drm] Registered 1 planes with drm panic1871client # [ 2.091164] systemd[1]: Finished Create Static Device Nodes in /dev.1872worker1 # [ 2.063299] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:08.0 on minor 01873client # [ 2.091436] systemd[1]: Reached target Preparation for Local File Systems.1874client # [ 2.091475] systemd[1]: Reached target Local File Systems.1875lb # [ 2.092051] systemd[1]: Starting Rule-based Manager for Device Events and Files...1876lb # [ 2.095604] systemd[1]: Finished Load Kernel Modules.1877client # [ 2.107100] systemd[1]: Starting Rule-based Manager for Device Events and Files...1878worker2 # [ 2.094524] systemd[1]: Finished Apply Kernel Variables.1879worker2 # [ 2.102036] systemd[1]: Started Journal Service.1880worker2 # [ 2.087958] systemd-modules-load[74]: Inserted module 'dm_mod'1881client # [ 2.125394] systemd[1]: Finished Apply Kernel Variables.1882worker2 # [ 2.096485] systemd-modules-load[74]: Module 'virtio_balloon' is built in1883worker2 # [ 2.100793] systemd-modules-load[74]: Module 'virtio_console' is built in1884worker1 # [ 2.097502] Console: switching to colour frame buffer device 160x501885lb # [ 2.127526] systemd[1]: Starting Apply Kernel Variables...1886worker2 # [ 2.104269] systemd-modules-load[74]: Inserted module 'virtio_gpu'1887worker2 # [ 2.109445] systemd-modules-load[74]: Module 'virtio_rng' is built in1888client # [ 2.128306] systemd-modules-load[74]: Inserted module 'dm_mod'1889client # [ 2.129491] systemd-modules-load[74]: Module 'virtio_balloon' is built in1890worker2 # [ 2.113443] systemd[1]: Starting Create System Files and Directories...1891client # [ 2.130604] systemd-modules-load[74]: Module 'virtio_console' is built in1892client # [ 2.131665] systemd-modules-load[74]: Inserted module 'virtio_gpu'1893worker1 # [ 2.117850] systemd[1]: Finished Create Static Device Nodes in /dev.1894client # [ 2.147411] systemd[1]: Started Journal Service.1895worker1 # [ 2.118068] systemd[1]: Reached target Preparation for Local File Systems.1896worker1 # [ 2.118093] systemd[1]: Reached target Local File Systems.1897worker1 # [ 2.122178] systemd[1]: Starting Rule-based Manager for Device Events and Files...1898worker2 # [ 2.121688] systemd-udevd[79]: Using default interface naming scheme 'v261'.1899client # [ 2.144495] systemd-modules-load[74]: Module 'virtio_rng' is built in1900worker1 # [ 2.133794] virtio-pci 0000:00:08.0: [drm] fb0: virtio_gpudrmfb frame buffer device1901client # [ 2.156137] systemd[1]: Starting Create System Files and Directories...1902lb # [ 2.170621] systemd[1]: Started Journal Service.1903lb # [ 2.169145] systemd-modules-load[74]: Inserted module 'dm_mod'1904worker1 # [ 2.161328] systemd[1]: Finished Load Kernel Modules.1905lb # [ 2.171250] systemd-modules-load[74]: Module 'virtio_balloon' is built in1906worker2 # [ 2.161212] systemd[1]: Started Rule-based Manager for Device Events and Files.1907worker1 # [ 2.163757] systemd[1]: Starting Apply Kernel Variables...1908lb # [ 2.175284] systemd-modules-load[74]: Module 'virtio_console' is built in1909client # [ 2.185460] systemd-udevd[81]: Using default interface naming scheme 'v261'.1910lb # [ 2.180493] systemd-modules-load[74]: Inserted module 'virtio_gpu'1911lb # [ 2.181498] systemd-modules-load[74]: Module 'virtio_rng' is built in1912worker2 # [ 2.167745] systemd[1]: Finished Create System Files and Directories.1913lb # [ 2.182489] systemd[1]: Starting Create System Files and Directories...1914lb # [ 2.196103] systemd[1]: Finished Apply Kernel Variables.1915client # [ 2.206431] systemd[1]: Finished Create System Files and Directories.1916lb # [ 2.205183] systemd-udevd[79]: Using default interface naming scheme 'v261'.1917worker1 # [ 2.188371] systemd-modules-load[74]: Inserted module 'dm_mod'1918worker1 # [ 2.202653] systemd[1]: Started Journal Service.1919worker1 # [ 2.192218] systemd-modules-load[74]: Module 'virtio_balloon' is built in1920worker1 # [ 2.193340] systemd-modules-load[74]: Module 'virtio_console' is built in1921worker1 # [ 2.194434] systemd-modules-load[74]: Inserted module 'virtio_gpu'1922worker1 # [ 2.195450] systemd-modules-load[74]: Module 'virtio_rng' is built in1923client # [ 2.231058] systemd[1]: Started Rule-based Manager for Device Events and Files.1924lb # [ 2.234702] systemd[1]: Finished Create System Files and Directories.1925worker1 # [ 2.216520] systemd[1]: Starting Create System Files and Directories...1926worker2 # [ 2.228607] systemd[1]: Starting Virtual Console Setup...1927worker1 # [ 2.228150] systemd[1]: Finished Apply Kernel Variables.1928lb # [ 2.250488] systemd[1]: Started Rule-based Manager for Device Events and Files.1929worker1 # [ 2.232575] systemd-udevd[79]: Using default interface naming scheme 'v261'.1930worker1 # [ 2.265548] systemd[1]: Finished Create System Files and Directories.1931client # [ 2.300378] systemd[1]: Starting Virtual Console Setup...1932worker1 # [ 2.277220] systemd[1]: Started Rule-based Manager for Device Events and Files.1933worker2 # [ 2.294254] systemd-vconsole-setup[102]: Configuration of first virtual console was skipped, ignoring remaining ones.1934worker2 # [ 2.295926] systemd[1]: Finished Virtual Console Setup.1935lb # [ 2.320275] systemd[1]: Starting Virtual Console Setup...1936client # [ 2.348524] systemd-vconsole-setup[103]: Configuration of first virtual console was skipped, ignoring remaining ones.1937client # [ 2.351933] systemd[1]: Finished Virtual Console Setup.1938worker1 # [ 2.340116] systemd[1]: Starting Virtual Console Setup...1939lb # [ 2.364480] systemd-vconsole-setup[103]: Configuration of first virtual console was skipped, ignoring remaining ones.1940lb # [ 2.368001] systemd[1]: Finished Virtual Console Setup.1941worker1 # [ 2.384452] systemd-vconsole-setup[103]: Configuration of first virtual console was skipped, ignoring remaining ones.1942worker1 # [ 2.387843] systemd[1]: Finished Virtual Console Setup.1943worker2 # [ 2.877438] systemd[1]: Finished Coldplug All udev Devices.1944worker2 # [ 2.880136] systemd[1]: Reached target System Initialization.1945worker2 # [ 2.881019] systemd[1]: Reached target Basic System.1946client # [ 2.944316] systemd[1]: Finished Coldplug All udev Devices.1947client # [ 2.948799] systemd[1]: Reached target System Initialization.1948client # [ 2.949682] systemd[1]: Reached target Basic System.1949lb # [ 2.972996] systemd[1]: Finished Coldplug All udev Devices.1950lb # [ 2.973922] systemd[1]: Reached target System Initialization.1951lb # [ 2.976154] systemd[1]: Reached target Basic System.1952worker1 # [ 2.995248] systemd[1]: Finished Coldplug All udev Devices.1953worker1 # [ 2.996500] systemd[1]: Reached target System Initialization.1954worker1 # [ 3.000110] systemd[1]: Reached target Basic System.1955worker2 # [ 3.035902] (udev-worker)[97]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name.1956worker2 # [ 3.040643] (udev-worker)[97]: Network interface NamePolicy= disabled on kernel command line.1957worker2 # [ 3.053410] (udev-worker)[94]: Network interface NamePolicy= disabled on kernel command line.1958client # [ 3.106004] (udev-worker)[99]: Network interface NamePolicy= disabled on kernel command line.1959lb # [ 3.107853] (udev-worker)[102]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name.1960client # [ 3.119024] (udev-worker)[97]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name.1961client # [ 3.126501] (udev-worker)[97]: Network interface NamePolicy= disabled on kernel command line.1962worker2 # [ 3.117790] systemd[1]: Found device /dev/disk/by-label/nixos.1963worker2 # [ 3.121552] systemd[1]: Reached target Initrd Root Device.1964worker2 # [ 3.124180] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos...1965lb # [ 3.146780] (udev-worker)[100]: Network interface NamePolicy= disabled on kernel command line.1966lb # [ 3.153934] (udev-worker)[102]: Network interface NamePolicy= disabled on kernel command line.1967client # [ 3.184561] systemd[1]: Found device /dev/disk/by-label/nixos.1968worker1 # [ 3.157652] (udev-worker)[96]: Network interface NamePolicy= disabled on kernel command line.1969client # [ 3.187049] systemd[1]: Reached target Initrd Root Device.1970client # [ 3.192197] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos...1971worker1 # [ 3.163833] (udev-worker)[97]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name.1972worker1 # [ 3.168505] (udev-worker)[97]: Network interface NamePolicy= disabled on kernel command line.1973worker2 # [ 3.182620] systemd-fsck[111]: nixos: clean, 12/65536 files, 13019/262144 blocks1974worker2 # [ 3.189865] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos.1975worker2 # [ 3.196254] systemd[1]: Mounting /sysroot...1976client # [ 3.243575] systemd-fsck[112]: nixos: clean, 12/65536 files, 13019/262144 blocks1977lb # [ 3.242333] systemd[1]: Found device /dev/disk/by-label/nixos.1978lb # [ 3.246710] systemd[1]: Reached target Initrd Root Device.1979client # [ 3.252136] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos.1980worker2 # [ 3.251997] EXT4-fs (vda): mounted filesystem 9841e95f-8727-4999-9445-4307181025de r/w with ordered data mode. Quota mode: none.1981client # [ 3.257470] systemd[1]: Mounting /sysroot...1982lb # [ 3.252112] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos...1983worker2 # [ 3.245042] systemd[1]: Mounted /sysroot.1984worker2 # [ 3.246065] systemd[1]: Reached target Initrd Root File System.1985worker2 # [ 3.249473] systemd[1]: Starting Mountpoints Configured in the Real Root...1986worker1 # [ 3.243974] systemd[1]: Found device /dev/disk/by-label/nixos.1987worker1 # [ 3.251637] systemd[1]: Reached target Initrd Root Device.1988worker1 # [ 3.257485] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos...1989client # [ 3.308216] EXT4-fs (vda): mounted filesystem e1c1dd6d-d0d5-48ab-918d-58d37418a616 r/w with ordered data mode. Quota mode: none.1990worker2 # [ 3.279514] systemd-sysroot-fstab-check[119]: /sysroot should be mounted in the initrd, will request daemon-reload.1991client # [ 3.300764] systemd[1]: Mounted /sysroot.1992worker2 # [ 3.284129] systemd[1]: Reload requested from client PID 119 ('systemd-sysroot') (unit initrd-parse-etc.service)...1993lb # [ 3.295068] systemd-fsck[112]: nixos: clean, 12/65536 files, 13019/262144 blocks1994worker2 # [ 3.289008] systemd[1]: Reloading...1995client # [ 3.306186] systemd[1]: Reached target Initrd Root File System.1996lb # [ 3.303229] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos.1997client # [ 3.312116] systemd[1]: Starting Mountpoints Configured in the Real Root...1998lb # [ 3.312307] systemd[1]: Mounting /sysroot...1999worker1 # [ 3.307608] systemd-fsck[112]: nixos: clean, 12/65536 files, 13019/262144 blocks2000client # [ 3.336144] systemd-sysroot-fstab-check[120]: /sysroot should be mounted in the initrd, will request daemon-reload.
loading 1,744 hidden lines…
3745worker1 # [ 27.638913] nix-grpc-daemon[777]: ts=2026-09-21T12:08:58Z level=info event=scheduler_connected addr=lb:500513746worker1 # [ 27.643981] nix-grpc-daemon[777]: ts=2026-09-21T12:08:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3747worker2 # [ 27.777753] nix-grpc-daemon[778]: ts=2026-09-21T12:08:58Z level=info event=scheduler_connected addr=lb:500513748worker2 # [ 27.782764] nix-grpc-daemon[778]: ts=2026-09-21T12:08:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3749worker1 # [ 27.844872] nix-grpc-daemon[777]: ts=2026-09-21T12:08:58Z level=info event=scheduler_connected addr=lb:500513750worker1 # [ 27.849901] nix-grpc-daemon[777]: ts=2026-09-21T12:08:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3751worker2 # [ 27.983832] nix-grpc-daemon[778]: ts=2026-09-21T12:08:58Z level=info event=scheduler_connected addr=lb:500513752worker2 # [ 27.989047] nix-grpc-daemon[778]: ts=2026-09-21T12:08:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3753worker1 # [ 28.050613] nix-grpc-daemon[777]: ts=2026-09-21T12:08:58Z level=info event=scheduler_connected addr=lb:500513754worker1 # [ 28.055669] nix-grpc-daemon[777]: ts=2026-09-21T12:08:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3755worker2 # [ 28.190194] nix-grpc-daemon[778]: ts=2026-09-21T12:08:58Z level=info event=scheduler_connected addr=lb:500513756worker2 # [ 28.195180] nix-grpc-daemon[778]: ts=2026-09-21T12:08:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3757worker1 # [ 28.258287] nix-grpc-daemon[777]: ts=2026-09-21T12:08:58Z level=info event=scheduler_connected addr=lb:500513758worker1 # [ 28.262404] nix-grpc-daemon[777]: ts=2026-09-21T12:08:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3759worker2 # [ 28.398043] nix-grpc-daemon[778]: ts=2026-09-21T12:08:58Z level=info event=scheduler_connected addr=lb:500513760worker2 # [ 28.402113] nix-grpc-daemon[778]: ts=2026-09-21T12:08:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3761worker1 # [ 28.458814] nix-grpc-daemon[777]: ts=2026-09-21T12:08:59Z level=info event=scheduler_connected addr=lb:500513762worker1 # [ 28.463957] nix-grpc-daemon[777]: ts=2026-09-21T12:08:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3763worker2 # [ 28.598815] nix-grpc-daemon[778]: ts=2026-09-21T12:08:59Z level=info event=scheduler_connected addr=lb:500513764worker2 # [ 28.603875] nix-grpc-daemon[778]: ts=2026-09-21T12:08:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3765worker1 # [ 28.666579] nix-grpc-daemon[777]: ts=2026-09-21T12:08:59Z level=info event=scheduler_connected addr=lb:500513766worker1 # [ 28.676225] nix-grpc-daemon[777]: ts=2026-09-21T12:08:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3767worker2 # [ 28.805023] nix-grpc-daemon[778]: ts=2026-09-21T12:08:59Z level=info event=scheduler_connected addr=lb:500513768worker2 # [ 28.810136] nix-grpc-daemon[778]: ts=2026-09-21T12:08:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3769worker1 # [ 28.867595] nix-grpc-daemon[777]: ts=2026-09-21T12:08:59Z level=info event=scheduler_connected addr=lb:500513770worker1 # [ 28.872968] nix-grpc-daemon[777]: ts=2026-09-21T12:08:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3771worker2 # [ 29.011325] nix-grpc-daemon[778]: ts=2026-09-21T12:08:59Z level=info event=scheduler_connected addr=lb:500513772worker2 # [ 29.016668] nix-grpc-daemon[778]: ts=2026-09-21T12:08:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3773worker1 # [ 29.073648] nix-grpc-daemon[777]: ts=2026-09-21T12:08:59Z level=info event=scheduler_connected addr=lb:500513774worker1 # [ 29.078912] nix-grpc-daemon[777]: ts=2026-09-21T12:08:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3775worker2 # [ 29.217760] nix-grpc-daemon[778]: ts=2026-09-21T12:08:59Z level=info event=scheduler_connected addr=lb:500513776worker2 # [ 29.222967] nix-grpc-daemon[778]: ts=2026-09-21T12:08:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3777worker1 # [ 29.281539] nix-grpc-daemon[777]: ts=2026-09-21T12:08:59Z level=info event=scheduler_connected addr=lb:500513778worker1 # [ 29.285621] nix-grpc-daemon[777]: ts=2026-09-21T12:08:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3779worker2 # [ 29.427488] nix-grpc-daemon[778]: ts=2026-09-21T12:08:59Z level=info event=scheduler_connected addr=lb:500513780worker2 # [ 29.432266] nix-grpc-daemon[778]: ts=2026-09-21T12:08:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3781worker1 # [ 29.482142] nix-grpc-daemon[777]: ts=2026-09-21T12:09:00Z level=info event=scheduler_connected addr=lb:500513782worker1 # [ 29.487268] nix-grpc-daemon[777]: ts=2026-09-21T12:09:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3783worker2 # [ 29.627847] nix-grpc-daemon[778]: ts=2026-09-21T12:09:00Z level=info event=scheduler_connected addr=lb:500513784worker2 # [ 29.633149] nix-grpc-daemon[778]: ts=2026-09-21T12:09:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3785worker1 # [ 29.688054] nix-grpc-daemon[777]: ts=2026-09-21T12:09:00Z level=info event=scheduler_connected addr=lb:500513786worker1 # [ 29.693192] nix-grpc-daemon[777]: ts=2026-09-21T12:09:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3787worker2 # [ 29.834000] nix-grpc-daemon[778]: ts=2026-09-21T12:09:00Z level=info event=scheduler_connected addr=lb:500513788worker2 # [ 29.839025] nix-grpc-daemon[778]: ts=2026-09-21T12:09:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3789worker1 # [ 29.894182] nix-grpc-daemon[777]: ts=2026-09-21T12:09:00Z level=info event=scheduler_connected addr=lb:500513790worker1 # [ 29.899251] nix-grpc-daemon[777]: ts=2026-09-21T12:09:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3791worker2 # [ 30.040171] nix-grpc-daemon[778]: ts=2026-09-21T12:09:00Z level=info event=scheduler_connected addr=lb:500513792worker2 # [ 30.045135] nix-grpc-daemon[778]: ts=2026-09-21T12:09:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3793worker1 # [ 30.099962] nix-grpc-daemon[777]: ts=2026-09-21T12:09:00Z level=info event=scheduler_connected addr=lb:500513794worker1 # [ 30.105092] nix-grpc-daemon[777]: ts=2026-09-21T12:09:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3795worker2 # [ 30.246130] nix-grpc-daemon[778]: ts=2026-09-21T12:09:00Z level=info event=scheduler_connected addr=lb:500513796worker2 # [ 30.251238] nix-grpc-daemon[778]: ts=2026-09-21T12:09:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3797worker1 # [ 30.307429] nix-grpc-daemon[777]: ts=2026-09-21T12:09:00Z level=info event=scheduler_connected addr=lb:500513798worker1 # [ 30.311610] nix-grpc-daemon[777]: ts=2026-09-21T12:09:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3799worker2 # [ 30.454152] nix-grpc-daemon[778]: ts=2026-09-21T12:09:00Z level=info event=scheduler_connected addr=lb:500513800worker2 # [ 30.458223] nix-grpc-daemon[778]: ts=2026-09-21T12:09:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3801worker1 # [ 30.507822] nix-grpc-daemon[777]: ts=2026-09-21T12:09:01Z level=info event=scheduler_connected addr=lb:500513802worker1 # [ 30.512992] nix-grpc-daemon[777]: ts=2026-09-21T12:09:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3803worker2 # [ 30.655015] nix-grpc-daemon[778]: ts=2026-09-21T12:09:01Z level=info event=scheduler_connected addr=lb:500513804worker2 # [ 30.660111] nix-grpc-daemon[778]: ts=2026-09-21T12:09:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3805worker1 # [ 30.713743] nix-grpc-daemon[777]: ts=2026-09-21T12:09:01Z level=info event=scheduler_connected addr=lb:500513806worker1 # [ 30.718870] nix-grpc-daemon[777]: ts=2026-09-21T12:09:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3807worker2 # [ 30.860922] nix-grpc-daemon[778]: ts=2026-09-21T12:09:01Z level=info event=scheduler_connected addr=lb:500513808worker2 # [ 30.864141] nix-grpc-daemon[778]: ts=2026-09-21T12:09:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3809worker1 # [ 30.919549] nix-grpc-daemon[777]: ts=2026-09-21T12:09:01Z level=info event=scheduler_connected addr=lb:500513810worker1 # [ 30.925439] nix-grpc-daemon[777]: ts=2026-09-21T12:09:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3811worker2 # [ 31.065182] nix-grpc-daemon[778]: ts=2026-09-21T12:09:01Z level=info event=scheduler_connected addr=lb:500513812worker2 # [ 31.070200] nix-grpc-daemon[778]: ts=2026-09-21T12:09:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3813worker1 # [ 31.126088] nix-grpc-daemon[777]: ts=2026-09-21T12:09:01Z level=info event=scheduler_connected addr=lb:500513814worker1 # [ 31.131113] nix-grpc-daemon[777]: ts=2026-09-21T12:09:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3815worker2 # [ 31.271225] nix-grpc-daemon[778]: ts=2026-09-21T12:09:01Z level=info event=scheduler_connected addr=lb:500513816worker2 # [ 31.276407] nix-grpc-daemon[778]: ts=2026-09-21T12:09:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3817worker1 # [ 31.333744] nix-grpc-daemon[777]: ts=2026-09-21T12:09:01Z level=info event=scheduler_connected addr=lb:500513818worker1 # [ 31.337752] nix-grpc-daemon[777]: ts=2026-09-21T12:09:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3819worker2 # [ 31.478857] nix-grpc-daemon[778]: ts=2026-09-21T12:09:02Z level=info event=scheduler_connected addr=lb:500513820worker2 # [ 31.482715] nix-grpc-daemon[778]: ts=2026-09-21T12:09:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3821worker1 # [ 31.534141] nix-grpc-daemon[777]: ts=2026-09-21T12:09:02Z level=info event=scheduler_connected addr=lb:500513822worker1 # [ 31.539170] nix-grpc-daemon[777]: ts=2026-09-21T12:09:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3823worker2 # [ 31.679840] nix-grpc-daemon[778]: ts=2026-09-21T12:09:02Z level=info event=scheduler_connected addr=lb:500513824worker2 # [ 31.684935] nix-grpc-daemon[778]: ts=2026-09-21T12:09:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3825worker1 # [ 31.739899] nix-grpc-daemon[777]: ts=2026-09-21T12:09:02Z level=info event=scheduler_connected addr=lb:500513826worker1 # [ 31.745035] nix-grpc-daemon[777]: ts=2026-09-21T12:09:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3827worker2 # [ 31.886043] nix-grpc-daemon[778]: ts=2026-09-21T12:09:02Z level=info event=scheduler_connected addr=lb:500513828worker2 # [ 31.891121] nix-grpc-daemon[778]: ts=2026-09-21T12:09:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3829worker1 # [ 31.945802] nix-grpc-daemon[777]: ts=2026-09-21T12:09:02Z level=info event=scheduler_connected addr=lb:500513830worker1 # [ 31.950891] nix-grpc-daemon[777]: ts=2026-09-21T12:09:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3831worker2 # [ 32.092161] nix-grpc-daemon[778]: ts=2026-09-21T12:09:02Z level=info event=scheduler_connected addr=lb:500513832worker2 # [ 32.097266] nix-grpc-daemon[778]: ts=2026-09-21T12:09:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3833worker1 # [ 32.151567] nix-grpc-daemon[777]: ts=2026-09-21T12:09:02Z level=info event=scheduler_connected addr=lb:500513834worker1 # [ 32.156358] nix-grpc-daemon[777]: ts=2026-09-21T12:09:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3835worker2 # [ 32.298080] nix-grpc-daemon[778]: ts=2026-09-21T12:09:02Z level=info event=scheduler_connected addr=lb:500513836worker2 # [ 32.303038] nix-grpc-daemon[778]: ts=2026-09-21T12:09:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3837worker1 # [ 32.358937] nix-grpc-daemon[777]: ts=2026-09-21T12:09:02Z level=info event=scheduler_connected addr=lb:500513838worker1 # [ 32.363087] nix-grpc-daemon[777]: ts=2026-09-21T12:09:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3839worker2 # [ 32.506052] nix-grpc-daemon[778]: ts=2026-09-21T12:09:03Z level=info event=scheduler_connected addr=lb:500513840worker2 # [ 32.510160] nix-grpc-daemon[778]: ts=2026-09-21T12:09:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3841worker1 # [ 32.559325] nix-grpc-daemon[777]: ts=2026-09-21T12:09:03Z level=info event=scheduler_connected addr=lb:500513842worker1 # [ 32.564546] nix-grpc-daemon[777]: ts=2026-09-21T12:09:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3843worker2 # [ 32.706972] nix-grpc-daemon[778]: ts=2026-09-21T12:09:03Z level=info event=scheduler_connected addr=lb:500513844worker2 # [ 32.712495] nix-grpc-daemon[778]: ts=2026-09-21T12:09:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3845worker1 # [ 32.765379] nix-grpc-daemon[777]: ts=2026-09-21T12:09:03Z level=info event=scheduler_connected addr=lb:500513846worker1 # [ 32.770469] nix-grpc-daemon[777]: ts=2026-09-21T12:09:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3847worker2 # [ 32.913847] nix-grpc-daemon[778]: ts=2026-09-21T12:09:03Z level=info event=scheduler_connected addr=lb:500513848worker2 # [ 32.918929] nix-grpc-daemon[778]: ts=2026-09-21T12:09:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3849worker1 # [ 32.971212] nix-grpc-daemon[777]: ts=2026-09-21T12:09:03Z level=info event=scheduler_connected addr=lb:500513850worker1 # [ 32.976366] nix-grpc-daemon[777]: ts=2026-09-21T12:09:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3851worker2 # [ 33.120093] nix-grpc-daemon[778]: ts=2026-09-21T12:09:03Z level=info event=scheduler_connected addr=lb:500513852worker2 # [ 33.125146] nix-grpc-daemon[778]: ts=2026-09-21T12:09:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3853worker1 # [ 33.177172] nix-grpc-daemon[777]: ts=2026-09-21T12:09:03Z level=info event=scheduler_connected addr=lb:500513854worker1 # [ 33.182163] nix-grpc-daemon[777]: ts=2026-09-21T12:09:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3855worker2 # [ 33.326101] nix-grpc-daemon[778]: ts=2026-09-21T12:09:03Z level=info event=scheduler_connected addr=lb:500513856worker2 # [ 33.331080] nix-grpc-daemon[778]: ts=2026-09-21T12:09:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3857worker1 # [ 33.384661] nix-grpc-daemon[777]: ts=2026-09-21T12:09:03Z level=info event=scheduler_connected addr=lb:500513858worker1 # [ 33.388404] nix-grpc-daemon[777]: ts=2026-09-21T12:09:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3859worker2 # [ 33.534033] nix-grpc-daemon[778]: ts=2026-09-21T12:09:04Z level=info event=scheduler_connected addr=lb:500513860worker2 # [ 33.538119] nix-grpc-daemon[778]: ts=2026-09-21T12:09:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3861worker1 # [ 33.585409] nix-grpc-daemon[777]: ts=2026-09-21T12:09:04Z level=info event=scheduler_connected addr=lb:500513862worker1 # [ 33.590063] nix-grpc-daemon[777]: ts=2026-09-21T12:09:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3863worker2 # [ 33.734952] nix-grpc-daemon[778]: ts=2026-09-21T12:09:04Z level=info event=scheduler_connected addr=lb:500513864worker2 # [ 33.740203] nix-grpc-daemon[778]: ts=2026-09-21T12:09:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3865lb # [ 33.766867] rustfs-bucket-start[1049]: mb s3://farm3866lb # [ 33.776807] systemd[1]: Finished rustfs-bucket.service.3867lb # [ 33.790302] systemd[1]: Starting niks3 server...3868worker1 # [ 33.790814] nix-grpc-daemon[777]: ts=2026-09-21T12:09:04Z level=info event=scheduler_connected addr=lb:500513869worker1 # [ 33.795911] nix-grpc-daemon[777]: ts=2026-09-21T12:09:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3870worker2 # [ 33.941157] nix-grpc-daemon[778]: ts=2026-09-21T12:09:04Z level=info event=scheduler_connected addr=lb:500513871worker2 # [ 33.946155] nix-grpc-daemon[778]: ts=2026-09-21T12:09:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3872worker1 # [ 33.996709] nix-grpc-daemon[777]: ts=2026-09-21T12:09:04Z level=info event=scheduler_connected addr=lb:500513873worker1 # [ 34.001720] nix-grpc-daemon[777]: ts=2026-09-21T12:09:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3874lb # [ 34.038686] postgres[1065]: [1065] ERROR: relation "goose_db_version" does not exist at character 363875lb # [ 34.039990] postgres[1065]: [1065] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC3876lb # [ 34.071569] niks3-server[1060]: 2026/09/21 12:09:04 OK 20241026095416_initial_model.sql (18.6ms)3877lb # [ 34.075837] niks3-server[1060]: 2026/09/21 12:09:04 OK 20251210153512_drop_unused_gin_index.sql (4.56ms)3878lb # [ 34.081541] niks3-server[1060]: 2026/09/21 12:09:04 OK 20251218171726_add_pins.sql (5.65ms)3879lb # [ 34.086199] niks3-server[1060]: 2026/09/21 12:09:04 OK 20260628120000_add_object_size_and_stats.sql (4.62ms)3880lb # [ 34.091043] niks3-server[1060]: 2026/09/21 12:09:04 OK 20260905000000_add_claims.sql (4.81ms)3881lb # [ 34.095629] niks3-server[1060]: 2026/09/21 12:09:04 OK 20260920000000_drop_claims.sql (4.48ms)3882lb # [ 34.097578] niks3-server[1060]: 2026/09/21 12:09:04 goose: successfully migrated database to version: 202609200000003883lb # [ 34.101838] niks3-server[1060]: 2026/09/21 12:09:04 OK 1_commit_pending_closure.sql (6.18ms)3884lb # [ 34.104411] niks3-server[1060]: 2026/09/21 12:09:04 OK 2_object_stats_trigger.sql (2.43ms)3885lb # [ 34.105775] niks3-server[1060]: 2026/09/21 12:09:04 goose: up to current file version: 23886lb # [ 34.109797] niks3-server[1060]: 2026/09/21 12:09:04 INFO Loaded signing key name=farm-test-1 path=/nix/store/8lzz3wcf17v05vfdp1zcfaa9hipw3f70-key3887lb # [ 34.157893] niks3-server[1060]: 2026/09/21 12:09:04 INFO Read proxy enabled — serving cache objects from S33888lb # [ 34.159721] niks3-server[1060]: 2026/09/21 12:09:04 INFO Using socket-activated listener address=0.0.0.0:57513889worker2 # [ 34.147261] nix-grpc-daemon[778]: ts=2026-09-21T12:09:04Z level=info event=scheduler_connected addr=lb:500513890lb # [ 34.163270] niks3-server[1060]: 2026/09/21 12:09:04 INFO systemd watchdog enabled interval=15s3891worker2 # [ 34.152362] nix-grpc-daemon[778]: ts=2026-09-21T12:09:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3892lb # [ 34.168368] niks3-server[1060]: 2026/09/21 12:09:04 INFO Starting HTTP server address=0.0.0.0:57513893lb # [ 34.172878] systemd[1]: Started niks3 server.3894lb # [ 34.173559] systemd[1]: Reached target Multi-User System.3895lb # [ 34.174318] systemd[1]: Startup finished in 1.031s (kernel) + 5.727s (initrd) + 27.408s (userspace) = 34.167s.3896lb # [ 34.180082] niks3-server[1060]: 2026/09/21 12:09:04 INFO lead: acquired remote=192.168.1.4:522203897worker2 # [ 34.172141] nix-grpc-daemon[778]: ts=2026-09-21T12:09:04Z level=info event=scheduler_take_over3898worker1 # [ 34.202440] nix-grpc-daemon[777]: ts=2026-09-21T12:09:04Z level=info event=scheduler_connected addr=lb:500513899worker1 # [ 34.207527] nix-grpc-daemon[777]: ts=2026-09-21T12:09:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3900worker2 # [ 34.353476] nix-grpc-daemon[778]: ts=2026-09-21T12:09:04Z level=info event=scheduler_connected addr=lb:500513901worker2 # [ 34.358454] nix-grpc-daemon[778]: ts=2026-09-21T12:09:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3902worker1 # [ 34.410121] nix-grpc-daemon[777]: ts=2026-09-21T12:09:04Z level=info event=scheduler_connected addr=lb:500513903worker1 # [ 34.414183] nix-grpc-daemon[777]: ts=2026-09-21T12:09:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3904worker2 # [ 34.562059] nix-grpc-daemon[778]: ts=2026-09-21T12:09:05Z level=info event=scheduler_connected addr=lb:500513905worker2 # [ 34.567002] nix-grpc-daemon[778]: ts=2026-09-21T12:09:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3906worker1 # [ 34.610935] nix-grpc-daemon[777]: ts=2026-09-21T12:09:05Z level=info event=scheduler_connected addr=lb:500513907worker1 # [ 34.616344] nix-grpc-daemon[777]: ts=2026-09-21T12:09:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3908lb: (finished: waiting for unit niks3.service, in 20.46 seconds)3909lb: waiting for TCP port 5751 on localhost3910lb # Connection to localhost (127.0.0.1) 5751 port [tcp/*] succeeded!3911lb: (finished: waiting for TCP port 5751 on localhost, in 0.11 seconds)3912lb: waiting for unit envoy.service3913worker2 # [ 34.768111] nix-grpc-daemon[778]: ts=2026-09-21T12:09:05Z level=info event=scheduler_connected addr=lb:500513914worker2 # [ 34.773244] nix-grpc-daemon[778]: ts=2026-09-21T12:09:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3915lb: (finished: waiting for unit envoy.service, in 0.08 seconds)3916lb: waiting for TCP port 50051 on localhost3917worker1 # [ 34.817367] nix-grpc-daemon[777]: ts=2026-09-21T12:09:05Z level=info event=scheduler_connected addr=lb:500513918worker1 # [ 34.823138] nix-grpc-daemon[777]: ts=2026-09-21T12:09:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3919lb # Connection to localhost (::1) 50051 port [tcp/*] succeeded!3920lb # [ 34.885823] envoy[910]: conn peer=[::1]:55734 tls=TLSv1.2 subject="-" sni=- flags=- tls_fail="-" rx=0 tx=0 ms=43921lb: (finished: waiting for TCP port 50051 on localhost, in 0.05 seconds)3922worker2 # [ 34.974577] nix-grpc-daemon[778]: ts=2026-09-21T12:09:05Z level=info event=scheduler_connected addr=lb:500513923worker2 # [ 34.979962] nix-grpc-daemon[778]: ts=2026-09-21T12:09:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3924worker1 # [ 35.024149] nix-grpc-daemon[777]: ts=2026-09-21T12:09:05Z level=info event=scheduler_connected addr=lb:500513925worker1 # [ 35.031107] nix-grpc-daemon[777]: ts=2026-09-21T12:09:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3926worker2 # [ 35.075331] systemd[1]: Starting scheduler-token.service...3927worker2 # [ 35.182773] nix-grpc-daemon[778]: ts=2026-09-21T12:09:05Z level=info event=scheduler_connected addr=lb:500513928worker2 # [ 35.187682] systemd[1]: scheduler-token.service: Deactivated successfully.3929worker2 # [ 35.198009] systemd[1]: Finished scheduler-token.service.3930worker2 # [ 35.200435] systemd[1]: scheduler-token.service: Consumed 39ms CPU time over 115ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.3931lb: waiting until exactly no endpoints of aarch64-linux are unhealthy3932lb: must succeed: curl -sf localhost:9901/clusters3933worker2 # [ 35.214173] nix-grpc-daemon[778]: ts=2026-09-21T12:09:05Z level=info event=oidc_jwks provider=mock keys=13934worker2 # [ 35.217673] nix-grpc-daemon[778]: ts=2026-09-21T12:09:05Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:490523935worker2 # [ 35.220555] nix-grpc-daemon[778]: ts=2026-09-21T12:09:05Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=03936worker1 # [ 35.232604] nix-grpc-daemon[777]: ts=2026-09-21T12:09:05Z level=info event=scheduler_connected addr=lb:500513937worker2 # [ 35.246030] nix-grpc-daemon[778]: ts=2026-09-21T12:09:05Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:490523938worker2 # [ 35.249164] nix-grpc-daemon[778]: ts=2026-09-21T12:09:05Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=03939lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.13 seconds)3940lb: (finished: waiting until exactly no endpoints of aarch64-linux are unhealthy, in 0.13 seconds)3941subtest: one node schedules and both builders hold a WorkerSession on it3942lb: must succeed: curl -sf localhost:9901/clusters3943lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)3944worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics3945worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)3946lb: must succeed: curl -sf localhost:9901/clusters3947lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)3948lb: must succeed: curl -sf localhost:9901/clusters3949lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)3950lb: must succeed: curl -sf localhost:9901/clusters3951lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)3952lb: must succeed: curl -sf localhost:9901/clusters3953lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)3954worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open'3955worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open', in 0.07 seconds)3956worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics3957worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)3958lb: must succeed: curl -sf localhost:9901/clusters3959lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.12 seconds)3960lb: must succeed: curl -sf localhost:9901/clusters3961lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)3962worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics3963worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.10 seconds)3964worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics3965worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)3966worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics3967worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)3968worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics3969worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)3970(finished: subtest: one node schedules and both builders hold a WorkerSession on it, in 0.99 seconds)3971subtest: balancer auth: client cert, bearer token, nothing, unknown CN3972worker1 # [ 36.736992] nix-grpc-daemon[777]: ts=2026-09-21T12:09:07Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:401103973worker1 # [ 36.754686] systemd[1]: Started Nix Daemon.3974worker1 # [ 37.057874] nix-daemon[842]: accepted connection from pid 777, user nix-grpc-daemon (trusted)3975worker1 # [ 37.081673] nix-grpc-daemon[777]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 38 ms (attempt 1/5)3976worker1 # [ 37.119800] nix-grpc-daemon[777]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 26 ms (attempt 2/5)3977worker1 # [ 37.146017] nix-grpc-daemon[777]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 255 ms (attempt 3/5)3978worker1 # [ 37.407067] nix-grpc-daemon[777]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 461 ms (attempt 4/5)3979worker1 # [ 37.878891] nix-grpc-daemon[777]: disabling binary cache 'https://cache.nixos.org' for 60 seconds3980worker1 # [ 37.885067] nix-grpc-daemon[777]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org3981worker2 # [ 37.930883] nix-grpc-daemon[778]: ts=2026-09-21T12:09:08Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:490543982worker2 # [ 37.949198] systemd[1]: Started Nix Daemon.3983worker2 # [ 38.214034] nix-daemon[863]: accepted connection from pid 778, user nix-grpc-daemon (trusted)3984worker2 # [ 38.237069] nix-grpc-daemon[778]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 81 ms (attempt 1/5)3985worker2 # [ 38.320853] nix-grpc-daemon[778]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 29 ms (attempt 2/5)3986worker2 # [ 38.350078] nix-grpc-daemon[778]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 290 ms (attempt 3/5)3987worker2 # [ 38.647114] nix-grpc-daemon[778]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 116 ms (attempt 4/5)3988worker2 # [ 38.765251] nix-grpc-daemon[778]: disabling binary cache 'https://cache.nixos.org' for 60 seconds3989worker2 # [ 38.770713] nix-grpc-daemon[778]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org3990worker2 # [ 38.783532] nix-grpc-daemon[778]: ts=2026-09-21T12:09:09Z level=debug event=path_info_miss path=wzyai2lcdcb8sgsxr1wpx21k8i95k7mn-nixos-system-client-test error="error: path '/nix/store/wzyai2lcdcb8sgsxr1wpx21k8i95k7mn-nixos-system-client-test' is required, but there is no substituter that can build it"3991worker1 # [ 38.784109] nix-grpc-daemon[777]: ts=2026-09-21T12:09:09Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:401103992worker1 # [ 38.796257] nix-grpc-daemon[777]: ts=2026-09-21T12:09:09Z level=debug event=path_info_miss path=wzyai2lcdcb8sgsxr1wpx21k8i95k7mn-nixos-system-client-test error="error: path '/nix/store/wzyai2lcdcb8sgsxr1wpx21k8i95k7mn-nixos-system-client-test' is required, but there is no substituter that can build it"3993client: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt3994client: (finished: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt, in 0.10 seconds)3995client: must succeed: install -D /nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key3996client: (finished: must succeed: install -D /nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key, in 0.05 seconds)3997worker1 # [ 39.270993] nix-grpc-daemon[777]: ts=2026-09-21T12:09:09Z level=info event=oidc_jwks provider=mock keys=13998worker1 # [ 39.276680] nix-grpc-daemon[777]: ts=2026-09-21T12:09:09Z level=debug event=rpc_start method=QueryMissing cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:401103999worker2 # [ 39.300927] nix-grpc-daemon[778]: ts=2026-09-21T12:09:09Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:490544000worker2 # [ 39.312707] nix-grpc-daemon[778]: ts=2026-09-21T12:09:09Z level=debug event=path_info_miss path=wzyai2lcdcb8sgsxr1wpx21k8i95k7mn-nixos-system-client-test error="error: path '/nix/store/wzyai2lcdcb8sgsxr1wpx21k8i95k7mn-nixos-system-client-test' is required, but there is no substituter that can build it"4001worker2 # [ 39.326156] nix-grpc-daemon[778]: ts=2026-09-21T12:09:09Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:490544002worker2 # [ 39.333916] nix-grpc-daemon[778]: ts=2026-09-21T12:09:09Z level=debug event=path_info_miss path=wzyai2lcdcb8sgsxr1wpx21k8i95k7mn-nixos-system-client-test error="error: path '/nix/store/wzyai2lcdcb8sgsxr1wpx21k8i95k7mn-nixos-system-client-test' is required, but there is no substituter that can build it"4003client: must succeed: rm -r /var/lib/nix-grpc-store4004client: (finished: must succeed: rm -r /var/lib/nix-grpc-store, in 0.04 seconds)4005worker1 # [ 39.628964] nix-grpc-daemon[777]: ts=2026-09-21T12:09:10Z level=info event=denied method=QueryMissing cn=- role=none4006worker2 # [ 39.903614] nix-grpc-daemon[778]: ts=2026-09-21T12:09:10Z level=info event=denied method=QueryMissing cn=stranger role=none4007(finished: subtest: balancer auth: client cert, bearer token, nothing, unknown CN, in 3.61 seconds)4008subtest: DAG build is scheduled across workers and lands in the cache4009client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 >&24010worker2 # [ 40.171179] nix-grpc-daemon[778]: ts=2026-09-21T12:09:10Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:490544011client # don't know how to build these paths:4012client # /nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv4013worker1 # [ 40.172937] nix-grpc-daemon[777]: ts=2026-09-21T12:09:10Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:401104014worker2 # [ 40.223182] nix-grpc-daemon[778]: ts=2026-09-21T12:09:10Z level=debug event=want client=1 drv=619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv cached=04015worker2 # [ 40.230369] nix-grpc-daemon[778]: ts=2026-09-21T12:09:10Z level=debug event=assigned drv=619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv worker=192.168.1.4:50051 clients=14016worker2 # [ 40.270143] nix-grpc-daemon[778]: ts=2026-09-21T12:09:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:490544017worker2 # [ 40.296572] nix-grpc-daemon[778]: ts=2026-09-21T12:09:10Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:490544018client # copying 1 paths to 'lb:50051'...4019worker2 # [ 40.313234] nix-grpc-daemon[778]: ts=2026-09-21T12:09:10Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:490544020client # copying path '/nix/store/619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv' to 'lb:50051'...4021worker2 # [ 40.342278] nix-daemon[863]: accepted connection from pid 778, user nix-grpc-daemon (trusted)4022worker2 # [ 40.373797] nix-grpc-daemon[778]: ts=2026-09-21T12:09:10Z level=info event=niks3_push_started pid=8784023lb # [ 40.658068] envoy[910]: conn peer=[2001:db8:1::1]:53754 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=732 tx=407 ms=21134024lb # [ 40.663317] envoy[910]: conn peer=[2001:db8:1::1]:53762 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=2596 tx=364 ms=764025lb # [ 40.668637] envoy[910]: conn peer=[2001:db8:1::1]:53774 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=423 tx=198 ms=134026lb # [ 40.674464] envoy[910]: conn peer=[2001:db8:1::1]:53782 tls=TLSv1.3 subject="CN=stranger" sni=- flags=- tls_fail="-" rx=423 tx=181 ms=184027worker2 # [ 40.670692] nix-grpc-daemon[878]: time=2026-09-21T12:09:11.213Z level=INFO msg="Configuring client TLS" cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker2.pem key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker2.key ca=""4028worker2 # [ 40.868749] nix-daemon[863]: accepted connection from pid 891, user nix-grpc-daemon (trusted)4029worker2 # [ 40.886596] nix-daemon[863]: reaped child process 894, status = succeeded4030worker2 # [ 41.067883] nix-daemon[863]: accepted connection from pid 896, user nix-grpc-daemon (trusted)4031worker2 # [ 41.084662] nix-daemon[863]: reaped child process 899, status = succeeded4032lb # [ 41.107921] niks3-server[1060]: 2026/09/21 12:09:11 INFO Received uploads request method=POST path=/api/pending_closures4033worker2 # [ 41.125375] nix-grpc-daemon[878]: time=2026-09-21T12:09:11.669Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4034worker2 # [ 41.128983] nix-grpc-daemon[878]: time=2026-09-21T12:09:11.670Z level=INFO msg="Uploading 619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv (504B)"4035lb # [ 41.187141] niks3-server[1060]: 2026/09/21 12:09:11 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign4036lb # [ 41.196149] niks3-server[1060]: 2026/09/21 12:09:11 INFO Registered completed upload object_key=nar/1il50j4vra11xfnqgjhjgrjn1753frlhxnss6mh1dmp5ww3is34w.nar.zst4037lb # [ 41.226309] niks3-server[1060]: 2026/09/21 12:09:11 INFO Signed narinfos id=1 count=14038worker2 # [ 41.219048] nix-grpc-daemon[878]: time=2026-09-21T12:09:11.761Z level=INFO msg="Uploading 1 narinfos"4039lb # [ 41.237008] niks3-server[1060]: 2026/09/21 12:09:11 INFO Registered completed upload object_key=619ccx60pfv52kv4hhgqpj1ivdmfcr5r.ls4040lb # [ 41.247977] niks3-server[1060]: 2026/09/21 12:09:11 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete4041lb # [ 41.254311] niks3-server[1060]: 2026/09/21 12:09:11 INFO Completed upload id=14042worker2 # [ 41.244409] nix-grpc-daemon[878]: time=2026-09-21T12:09:11.787Z level=INFO msg="Upload complete. (562ms)"4043lb # [ 41.261512] niks3-server[1060]: 2026/09/21 12:09:11 INFO Registered completed upload object_key=619ccx60pfv52kv4hhgqpj1ivdmfcr5r.narinfo4044worker2 # [ 41.254241] nix-grpc-daemon[778]: ts=2026-09-21T12:09:11Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49054 duration_s=0 paths=1 nar_bytes_in=5044045worker2 # [ 41.272402] nix-grpc-daemon[778]: ts=2026-09-21T12:09:11Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:490544046worker2 # [ 41.283195] nix-daemon[863]: accepted connection from pid 778, user nix-grpc-daemon (trusted)4047worker2 # [ 41.290621] nix-daemon[863]: reaped child process 876, status = succeeded4048client # building '/nix/store/619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv' on lb:50051...4049client # farm-a> worker2: building 619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv4050worker2 # [ 41.305492] nix-daemon[863]: accepted connection from pid 778, user nix-grpc-daemon (trusted)4051client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds4052client # farm-a> LOG-farm-a-t14053worker2 # [ 42.777802] nix-daemon[863]: reaped child process 903, status = succeeded4054worker2 # [ 42.965543] nix-daemon[863]: accepted connection from pid 919, user nix-grpc-daemon (trusted)4055worker2 # [ 42.982365] nix-daemon[863]: reaped child process 923, status = succeeded4056lb # [ 43.011347] niks3-server[1060]: 2026/09/21 12:09:13 INFO Received uploads request method=POST path=/api/pending_closures4057worker2 # [ 43.021961] nix-grpc-daemon[878]: time=2026-09-21T12:09:13.563Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4058worker2 # [ 43.024778] nix-grpc-daemon[878]: time=2026-09-21T12:09:13.563Z level=INFO msg="Uploading dg404zlvxfrspnxl6cc1qkc6i83m5a3a-farm-a (128B)"4059lb # [ 43.081804] niks3-server[1060]: 2026/09/21 12:09:13 INFO Registered completed upload object_key=log/619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv4060lb # [ 43.097764] niks3-server[1060]: 2026/09/21 12:09:13 INFO Registered completed upload object_key=nar/1rmil4ja9jzfwn6l5hwbir70b6plx99sgk0pnjjy377ngbc3vz28.nar.zst4061lb # [ 43.105835] niks3-server[1060]: 2026/09/21 12:09:13 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign4062lb # [ 43.111235] niks3-server[1060]: 2026/09/21 12:09:13 INFO Signed narinfos id=2 count=14063worker2 # [ 43.100202] nix-grpc-daemon[878]: time=2026-09-21T12:09:13.642Z level=INFO msg="Uploading 1 narinfos"4064lb # [ 43.113872] niks3-server[1060]: 2026/09/21 12:09:13 INFO Registered completed upload object_key=dg404zlvxfrspnxl6cc1qkc6i83m5a3a.ls4065lb # [ 43.128480] niks3-server[1060]: 2026/09/21 12:09:13 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete4066lb # [ 43.132625] niks3-server[1060]: 2026/09/21 12:09:13 INFO Completed upload id=24067worker2 # [ 43.122402] nix-grpc-daemon[878]: time=2026-09-21T12:09:13.666Z level=INFO msg="Upload complete. (399ms)"4068lb # [ 43.134544] niks3-server[1060]: 2026/09/21 12:09:13 INFO Registered completed upload object_key=dg404zlvxfrspnxl6cc1qkc6i83m5a3a.narinfo4069worker2 # [ 43.149103] nix-grpc-daemon[778]: ts=2026-09-21T12:09:13Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49054 duration_s=1 drv=619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv assign_id=1 outputs=14070worker2 # [ 43.163746] nix-grpc-daemon[778]: ts=2026-09-21T12:09:13Z level=debug event=want client=1 drv=79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv cached=04071worker2 # [ 43.170003] nix-grpc-daemon[778]: ts=2026-09-21T12:09:13Z level=debug event=assigned drv=79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv worker=192.168.1.4:50051 clients=14072worker2 # [ 43.174372] nix-daemon[863]: reaped child process 901, status = succeeded4073worker2 # [ 43.189068] nix-grpc-daemon[778]: ts=2026-09-21T12:09:13Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:490544074worker2 # [ 43.202506] nix-grpc-daemon[778]: ts=2026-09-21T12:09:13Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:490544075worker2 # [ 43.216419] nix-grpc-daemon[878]: time=2026-09-21T12:09:13.760Z level=INFO msg="All 1 paths already cached"4076client # copying 1 paths to 'lb:50051'...4077worker2 # [ 43.224624] nix-grpc-daemon[778]: ts=2026-09-21T12:09:13Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:490544078client # copying path '/nix/store/79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv' to 'lb:50051'...4079worker2 # [ 43.248981] nix-daemon[863]: accepted connection from pid 778, user nix-grpc-daemon (trusted)4080worker2 # [ 43.462921] nix-daemon[863]: accepted connection from pid 929, user nix-grpc-daemon (trusted)4081worker2 # [ 43.479206] nix-daemon[863]: reaped child process 932, status = succeeded4082worker2 # [ 43.654988] nix-daemon[863]: accepted connection from pid 934, user nix-grpc-daemon (trusted)4083worker2 # [ 43.671110] nix-daemon[863]: reaped child process 937, status = succeeded4084lb # [ 43.693298] niks3-server[1060]: 2026/09/21 12:09:14 INFO Received uploads request method=POST path=/api/pending_closures4085worker2 # [ 43.693450] nix-grpc-daemon[878]: time=2026-09-21T12:09:14.236Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4086worker2 # [ 43.694888] nix-grpc-daemon[878]: time=2026-09-21T12:09:14.236Z level=INFO msg="Uploading 79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv (624B)"4087lb # [ 43.728391] niks3-server[1060]: 2026/09/21 12:09:14 INFO Received sign narinfos request method=POST path=/api/pending_closures/3/sign4088worker2 # [ 43.725865] nix-grpc-daemon[878]: time=2026-09-21T12:09:14.269Z level=INFO msg="Uploading 1 narinfos"4089lb # [ 43.741382] niks3-server[1060]: 2026/09/21 12:09:14 INFO Registered completed upload object_key=nar/1xk4kbz8l387hfrqq2xi6as79has336h62f2ra95472yqjclksjg.nar.zst4090lb # [ 43.751611] niks3-server[1060]: 2026/09/21 12:09:14 INFO Signed narinfos id=3 count=14091worker2 # [ 43.750224] nix-grpc-daemon[878]: time=2026-09-21T12:09:14.293Z level=INFO msg="Upload complete. (484ms)"4092lb # [ 43.761090] niks3-server[1060]: 2026/09/21 12:09:14 INFO Registered completed upload object_key=79z38hfi82zcmb9yq1gcmfviimgf6z2l.ls4093lb # [ 43.766367] niks3-server[1060]: 2026/09/21 12:09:14 INFO Received complete upload request method=POST path=/api/pending_closures/3/complete4094worker2 # [ 43.759417] nix-grpc-daemon[778]: ts=2026-09-21T12:09:14Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49054 duration_s=0 paths=1 nar_bytes_in=6244095lb # [ 43.771214] niks3-server[1060]: 2026/09/21 12:09:14 INFO Completed upload id=34096lb # [ 43.773765] niks3-server[1060]: 2026/09/21 12:09:14 INFO Registered completed upload object_key=79z38hfi82zcmb9yq1gcmfviimgf6z2l.narinfo4097worker2 # [ 43.767436] nix-grpc-daemon[778]: ts=2026-09-21T12:09:14Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:490544098worker2 # [ 43.776909] nix-daemon[863]: accepted connection from pid 778, user nix-grpc-daemon (trusted)4099worker2 # [ 43.779439] nix-daemon[863]: reaped child process 927, status = succeeded4100client # building '/nix/store/79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv' on lb:50051...4101client # farm-b> worker2: building 79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv4102worker2 # [ 43.791473] nix-daemon[863]: accepted connection from pid 778, user nix-grpc-daemon (trusted)4103client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds4104client # farm-b> LOG-farm-b-t14105worker2 # [ 44.841201] nix-daemon[863]: reaped child process 941, status = succeeded4106worker2 # [ 45.014605] nix-daemon[863]: accepted connection from pid 957, user nix-grpc-daemon (trusted)4107worker2 # [ 45.030790] nix-daemon[863]: reaped child process 960, status = succeeded4108lb # [ 45.055935] niks3-server[1060]: 2026/09/21 12:09:15 INFO Received uploads request method=POST path=/api/pending_closures4109worker2 # [ 45.054321] nix-grpc-daemon[878]: time=2026-09-21T12:09:15.597Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4110worker2 # [ 45.056364] nix-grpc-daemon[878]: time=2026-09-21T12:09:15.597Z level=INFO msg="Uploading 8n9g36vw9957r150a7wg1k6yj9fbqm8r-farm-b (176B)"4111lb # [ 45.092055] niks3-server[1060]: 2026/09/21 12:09:15 INFO Registered completed upload object_key=nar/1mx8h9irf9ykfzxs6q3lwxbzxwv5d2s9mh2x0sky7448mlc1qxrx.nar.zst4112lb # [ 45.105315] niks3-server[1060]: 2026/09/21 12:09:15 INFO Registered completed upload object_key=log/79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv4113lb # [ 45.111846] niks3-server[1060]: 2026/09/21 12:09:15 INFO Received sign narinfos request method=POST path=/api/pending_closures/4/sign4114worker2 # [ 45.106819] nix-grpc-daemon[878]: time=2026-09-21T12:09:15.649Z level=INFO msg="Uploading 1 narinfos"4115lb # [ 45.122624] niks3-server[1060]: 2026/09/21 12:09:15 INFO Signed narinfos id=4 count=14116lb # [ 45.128184] niks3-server[1060]: 2026/09/21 12:09:15 INFO Registered completed upload object_key=8n9g36vw9957r150a7wg1k6yj9fbqm8r.ls4117lb # [ 45.137391] niks3-server[1060]: 2026/09/21 12:09:15 INFO Received complete upload request method=POST path=/api/pending_closures/4/complete4118lb # [ 45.142298] niks3-server[1060]: 2026/09/21 12:09:15 INFO Completed upload id=44119worker2 # [ 45.131578] nix-grpc-daemon[878]: time=2026-09-21T12:09:15.675Z level=INFO msg="Upload complete. (328ms)"4120lb # [ 45.145338] niks3-server[1060]: 2026/09/21 12:09:15 INFO Registered completed upload object_key=8n9g36vw9957r150a7wg1k6yj9fbqm8r.narinfo4121worker2 # [ 45.149047] nix-grpc-daemon[778]: ts=2026-09-21T12:09:15Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49054 duration_s=1 drv=79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv assign_id=2 outputs=14122worker2 # [ 45.164420] nix-grpc-daemon[778]: ts=2026-09-21T12:09:15Z level=debug event=want client=1 drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv cached=04123worker2 # [ 45.170021] nix-grpc-daemon[778]: ts=2026-09-21T12:09:15Z level=debug event=assigned drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv worker=192.168.1.4:50051 clients=14124worker2 # [ 45.175219] nix-daemon[863]: reaped child process 939, status = succeeded4125worker2 # [ 45.193758] nix-grpc-daemon[778]: ts=2026-09-21T12:09:15Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:490544126worker2 # [ 45.209585] nix-grpc-daemon[778]: ts=2026-09-21T12:09:15Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:490544127worker2 # [ 45.223338] nix-grpc-daemon[878]: time=2026-09-21T12:09:15.766Z level=INFO msg="All 2 paths already cached"4128client # copying 1 paths to 'lb:50051'...4129worker2 # [ 45.233508] nix-grpc-daemon[778]: ts=2026-09-21T12:09:15Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:490544130client # copying path '/nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv' to 'lb:50051'...4131worker2 # [ 45.258148] nix-daemon[863]: accepted connection from pid 778, user nix-grpc-daemon (trusted)4132worker2 # [ 45.469316] nix-daemon[863]: accepted connection from pid 966, user nix-grpc-daemon (trusted)4133worker2 # [ 45.487383] nix-daemon[863]: reaped child process 969, status = succeeded4134worker2 # [ 45.657578] nix-daemon[863]: accepted connection from pid 971, user nix-grpc-daemon (trusted)4135worker2 # [ 45.674621] nix-daemon[863]: reaped child process 974, status = succeeded4136lb # [ 45.697107] niks3-server[1060]: 2026/09/21 12:09:16 INFO Received uploads request method=POST path=/api/pending_closures4137worker2 # [ 45.698016] nix-grpc-daemon[878]: time=2026-09-21T12:09:16.241Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4138worker2 # [ 45.699486] nix-grpc-daemon[878]: time=2026-09-21T12:09:16.241Z level=INFO msg="Uploading czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv (752B)"4139lb # [ 45.725429] niks3-server[1060]: 2026/09/21 12:09:16 INFO Registered completed upload object_key=nar/02rh0zwkcxgd7rc74r5h676rr0m8dbayk3glb776b2mdlirxh9lv.nar.zst4140worker2 # [ 45.728509] nix-grpc-daemon[878]: time=2026-09-21T12:09:16.271Z level=INFO msg="Uploading 1 narinfos"4141lb # [ 45.742437] niks3-server[1060]: 2026/09/21 12:09:16 INFO Received sign narinfos request method=POST path=/api/pending_closures/5/sign4142lb # [ 45.753087] niks3-server[1060]: 2026/09/21 12:09:16 INFO Signed narinfos id=5 count=14143lb # [ 45.757925] niks3-server[1060]: 2026/09/21 12:09:16 INFO Registered completed upload object_key=czd6ljynjaw3bl3w8iz1nx68whkdis8f.ls4144worker2 # [ 45.757254] nix-grpc-daemon[878]: time=2026-09-21T12:09:16.300Z level=INFO msg="Upload complete. (481ms)"4145lb # [ 45.767728] niks3-server[1060]: 2026/09/21 12:09:16 INFO Received complete upload request method=POST path=/api/pending_closures/5/complete4146lb # [ 45.773016] niks3-server[1060]: 2026/09/21 12:09:16 INFO Completed upload id=54147worker2 # [ 45.764094] nix-grpc-daemon[778]: ts=2026-09-21T12:09:16Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49054 duration_s=0 paths=1 nar_bytes_in=7524148lb # [ 45.776297] niks3-server[1060]: 2026/09/21 12:09:16 INFO Registered completed upload object_key=czd6ljynjaw3bl3w8iz1nx68whkdis8f.narinfo4149worker2 # [ 45.774228] nix-grpc-daemon[778]: ts=2026-09-21T12:09:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:490544150worker2 # [ 45.780140] nix-daemon[863]: accepted connection from pid 778, user nix-grpc-daemon (trusted)4151worker2 # [ 45.784401] nix-daemon[863]: reaped child process 964, status = succeeded4152client # building '/nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv' on lb:50051...4153worker2 # [ 45.794230] nix-daemon[863]: accepted connection from pid 778, user nix-grpc-daemon (trusted)4154client # farm-top> worker2: building czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv4155client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds4156client # farm-top> LOG-farm-top-t14157worker2 # [ 46.819514] nix-daemon[863]: reaped child process 978, status = succeeded4158worker2 # [ 46.987402] nix-daemon[863]: accepted connection from pid 995, user nix-grpc-daemon (trusted)4159worker2 # [ 47.006041] nix-daemon[863]: reaped child process 998, status = succeeded4160lb # [ 47.031763] niks3-server[1060]: 2026/09/21 12:09:17 INFO Received uploads request method=POST path=/api/pending_closures4161worker2 # [ 47.030907] nix-grpc-daemon[878]: time=2026-09-21T12:09:17.572Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4162worker2 # [ 47.032488] nix-grpc-daemon[878]: time=2026-09-21T12:09:17.572Z level=INFO msg="Uploading nblg066ygaz2avzlmix3rnngr62hld8f-farm-top (232B)"4163lb # [ 47.068101] niks3-server[1060]: 2026/09/21 12:09:17 INFO Registered completed upload object_key=nar/1wplyx8g7zja504cjvqmxis1q6laml9bidgiq4ya0i9p2pvqb9qp.nar.zst4164lb # [ 47.082119] niks3-server[1060]: 2026/09/21 12:09:17 INFO Registered completed upload object_key=log/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv4165worker2 # [ 47.081320] nix-grpc-daemon[878]: time=2026-09-21T12:09:17.624Z level=INFO msg="Uploading 1 narinfos"4166lb # [ 47.094434] niks3-server[1060]: 2026/09/21 12:09:17 INFO Received sign narinfos request method=POST path=/api/pending_closures/6/sign4167lb # [ 47.101182] niks3-server[1060]: 2026/09/21 12:09:17 INFO Signed narinfos id=6 count=14168lb # [ 47.103296] niks3-server[1060]: 2026/09/21 12:09:17 INFO Registered completed upload object_key=nblg066ygaz2avzlmix3rnngr62hld8f.ls4169lb # [ 47.112211] niks3-server[1060]: 2026/09/21 12:09:17 INFO Received complete upload request method=POST path=/api/pending_closures/6/complete4170lb # [ 47.119843] niks3-server[1060]: 2026/09/21 12:09:17 INFO Completed upload id=64171lb # [ 47.122706] niks3-server[1060]: 2026/09/21 12:09:17 INFO Registered completed upload object_key=nblg066ygaz2avzlmix3rnngr62hld8f.narinfo4172worker2 # [ 47.109820] nix-grpc-daemon[878]: time=2026-09-21T12:09:17.653Z level=INFO msg="Upload complete. (305ms)"4173worker2 # [ 47.121188] nix-grpc-daemon[778]: ts=2026-09-21T12:09:17Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49054 duration_s=1 drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv assign_id=3 outputs=14174worker2 # [ 47.137516] nix-daemon[863]: reaped child process 976, status = succeeded4175client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 >&2, in 7.24 seconds)4176client: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 outPath4177client: (finished: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 outPath, in 0.24 seconds)4178client: must fail: test -e /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top4179client: (finished: must fail: test -e /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.03 seconds)4180client: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top && grep farm-top-t1 /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top4181client # copying 3 paths...4182client # copying path '/nix/store/dg404zlvxfrspnxl6cc1qkc6i83m5a3a-farm-a' from 'http://lb:5751'...4183client # copying path '/nix/store/8n9g36vw9957r150a7wg1k6yj9fbqm8r-farm-b' from 'http://lb:5751'...4184client # copying path '/nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top' from 'http://lb:5751'...4185client: (finished: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top && grep farm-top-t1 /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.30 seconds)4186(finished: subtest: DAG build is scheduled across workers and lands in the cache, in 7.91 seconds)4187subtest: repeat is answered Cached by the scheduler without building4188worker1: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top4189worker1 # finding garbage collector roots...4190worker1 # deleting specified paths...4191worker1 # deleting '/nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top'4192worker1 # deleting unused links...4193worker1 # note: hard linking is currently saving 0.0 KiB4194worker1: (finished: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.29 seconds)4195worker2: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top4196worker2 # finding garbage collector roots...4197worker2 # deleting specified paths...4198worker2 # deleting '/nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top'4199worker2 # deleting unused links...4200worker2 # note: hard linking is currently saving 0.0 KiB4201worker2: (finished: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.27 seconds)4202client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 >&24203worker2 # [ 48.674994] nix-grpc-daemon[778]: ts=2026-09-21T12:09:19Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:490544204client # this path will be fetched (0.0 KiB download, 0.2 KiB unpacked):4205client # /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top4206worker2 # [ 48.692619] nix-grpc-daemon[778]: ts=2026-09-21T12:09:19Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:490544207worker2 # [ 48.706533] nix-grpc-daemon[878]: time=2026-09-21T12:09:19.249Z level=INFO msg="All 2 paths already cached"4208worker2 # [ 48.723328] nix-grpc-daemon[778]: ts=2026-09-21T12:09:19Z level=debug event=want client=2 drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv cached=14209client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 >&2, in 0.36 seconds)4210client: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 outPath4211client: (finished: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 outPath, in 0.22 seconds)4212lb: must succeed: curl -sf localhost:9901/clusters4213lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds)4214lb: must succeed: curl -sf localhost:9901/clusters4215lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)4216worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4217worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)4218(finished: subtest: repeat is answered Cached by the scheduler without building, in 1.46 seconds)4219subtest: inputs already in the cache are not sent to the scheduler4220lb: must succeed: curl -sf localhost:9901/clusters4221lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)4222lb: must succeed: curl -sf localhost:9901/clusters4223lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)4224worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4225worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.06 seconds)4226client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 --argstr top t1b >&24227worker1 # [ 49.764941] nix-grpc-daemon[777]: ts=2026-09-21T12:09:20Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:401104228client # don't know how to build these paths:4229client # /nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv4230worker2 # [ 49.788662] nix-grpc-daemon[778]: ts=2026-09-21T12:09:20Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:490544231worker2 # [ 49.799434] nix-grpc-daemon[878]: time=2026-09-21T12:09:20.342Z level=INFO msg="All 2 paths already cached"4232worker2 # [ 49.814860] nix-grpc-daemon[778]: ts=2026-09-21T12:09:20Z level=debug event=want client=3 drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv cached=04233worker2 # [ 49.821741] nix-grpc-daemon[778]: ts=2026-09-21T12:09:20Z level=debug event=assigned drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv worker=192.168.1.4:50051 clients=14234worker2 # [ 49.861691] nix-grpc-daemon[778]: ts=2026-09-21T12:09:20Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:490544235worker2 # [ 49.877118] nix-grpc-daemon[778]: ts=2026-09-21T12:09:20Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:490544236worker2 # [ 49.891401] nix-grpc-daemon[878]: time=2026-09-21T12:09:20.434Z level=INFO msg="All 2 paths already cached"4237client # copying 1 paths to 'lb:50051'...4238client # copying path '/nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv' to 'lb:50051'...4239worker2 # [ 49.902231] nix-grpc-daemon[778]: ts=2026-09-21T12:09:20Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:490544240worker2 # [ 49.917256] nix-daemon[863]: accepted connection from pid 778, user nix-grpc-daemon (trusted)4241worker2 # [ 50.122764] nix-daemon[863]: accepted connection from pid 1027, user nix-grpc-daemon (trusted)4242worker2 # [ 50.141057] nix-daemon[863]: reaped child process 1030, status = succeeded4243worker2 # [ 50.314226] nix-daemon[863]: accepted connection from pid 1032, user nix-grpc-daemon (trusted)4244worker2 # [ 50.331454] nix-daemon[863]: reaped child process 1035, status = succeeded4245lb # [ 50.355486] niks3-server[1060]: 2026/09/21 12:09:20 INFO Received uploads request method=POST path=/api/pending_closures4246worker2 # [ 50.356662] nix-grpc-daemon[878]: time=2026-09-21T12:09:20.900Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4247worker2 # [ 50.358108] nix-grpc-daemon[878]: time=2026-09-21T12:09:20.900Z level=INFO msg="Uploading blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv (752B)"4248lb # [ 50.384572] niks3-server[1060]: 2026/09/21 12:09:20 INFO Registered completed upload object_key=nar/1vy3ds565pni9mnkldcs6alimlb6bg39cimm60s4swkk5yjhwlqf.nar.zst4249worker2 # [ 50.389865] nix-grpc-daemon[878]: time=2026-09-21T12:09:20.933Z level=INFO msg="Uploading 1 narinfos"4250lb # [ 50.402665] niks3-server[1060]: 2026/09/21 12:09:20 INFO Received sign narinfos request method=POST path=/api/pending_closures/7/sign4251lb # [ 50.412297] niks3-server[1060]: 2026/09/21 12:09:20 INFO Signed narinfos id=7 count=14252lb # [ 50.416476] niks3-server[1060]: 2026/09/21 12:09:20 INFO Registered completed upload object_key=blgxcsidls7na1m5da3m0bja1prb87vl.ls4253worker2 # [ 50.413674] nix-grpc-daemon[878]: time=2026-09-21T12:09:20.957Z level=INFO msg="Upload complete. (479ms)"4254lb # [ 50.424398] niks3-server[1060]: 2026/09/21 12:09:20 INFO Received complete upload request method=POST path=/api/pending_closures/7/complete4255lb # [ 50.430401] niks3-server[1060]: 2026/09/21 12:09:20 INFO Completed upload id=74256lb # [ 50.434224] niks3-server[1060]: 2026/09/21 12:09:20 INFO Registered completed upload object_key=blgxcsidls7na1m5da3m0bja1prb87vl.narinfo4257worker2 # [ 50.420223] nix-grpc-daemon[778]: ts=2026-09-21T12:09:20Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49054 duration_s=0 paths=1 nar_bytes_in=7524258worker2 # [ 50.429004] nix-grpc-daemon[778]: ts=2026-09-21T12:09:20Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:490544259worker2 # [ 50.439582] nix-daemon[863]: accepted connection from pid 778, user nix-grpc-daemon (trusted)4260worker2 # [ 50.443832] nix-daemon[863]: reaped child process 1025, status = succeeded4261client # building '/nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv' on lb:50051...4262client # farm-top> worker2: building blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv4263worker2 # [ 50.454563] nix-daemon[863]: accepted connection from pid 778, user nix-grpc-daemon (trusted)4264lb # [ 50.660253] envoy[910]: conn peer=[2001:db8:1::1]:53792 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9575 tx=4096 ms=69794265lb # [ 50.665176] envoy[910]: conn peer=[2001:db8:1::1]:57238 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=993 tx=503 ms=704266client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds4267client # farm-top> LOG-farm-top-t1b4268worker2 # [ 51.481155] nix-daemon[863]: reaped child process 1039, status = succeeded4269worker2 # [ 51.637795] nix-daemon[863]: accepted connection from pid 1055, user nix-grpc-daemon (trusted)4270worker2 # [ 51.654398] nix-daemon[863]: reaped child process 1058, status = succeeded4271lb # [ 51.681533] niks3-server[1060]: 2026/09/21 12:09:22 INFO Received uploads request method=POST path=/api/pending_closures4272worker2 # [ 51.683875] nix-grpc-daemon[878]: time=2026-09-21T12:09:22.224Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4273worker2 # [ 51.685537] nix-grpc-daemon[878]: time=2026-09-21T12:09:22.224Z level=INFO msg="Uploading 0xnysbbdbmapgf23wb2dbsfhgh18yxvp-farm-top (232B)"4274lb # [ 51.714245] niks3-server[1060]: 2026/09/21 12:09:22 INFO Registered completed upload object_key=nar/1f6pjd04v1xmbxqqjhczkkfk4ns9qg6ssnspdalp41q9hkxf9yfr.nar.zst4275lb # [ 51.729141] niks3-server[1060]: 2026/09/21 12:09:22 INFO Registered completed upload object_key=log/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv4276lb # [ 51.740120] niks3-server[1060]: 2026/09/21 12:09:22 INFO Received sign narinfos request method=POST path=/api/pending_closures/8/sign4277worker2 # [ 51.736179] nix-grpc-daemon[878]: time=2026-09-21T12:09:22.279Z level=INFO msg="Uploading 1 narinfos"4278lb # [ 51.746053] niks3-server[1060]: 2026/09/21 12:09:22 INFO Signed narinfos id=8 count=14279lb # [ 51.749904] niks3-server[1060]: 2026/09/21 12:09:22 INFO Registered completed upload object_key=0xnysbbdbmapgf23wb2dbsfhgh18yxvp.ls4280lb # [ 51.761548] niks3-server[1060]: 2026/09/21 12:09:22 INFO Received complete upload request method=POST path=/api/pending_closures/8/complete4281lb # [ 51.769385] niks3-server[1060]: 2026/09/21 12:09:22 INFO Completed upload id=84282lb # [ 51.773524] niks3-server[1060]: 2026/09/21 12:09:22 INFO Registered completed upload object_key=0xnysbbdbmapgf23wb2dbsfhgh18yxvp.narinfo4283worker2 # [ 51.757496] nix-grpc-daemon[878]: time=2026-09-21T12:09:22.301Z level=INFO msg="Upload complete. (288ms)"4284worker2 # [ 51.769071] nix-grpc-daemon[778]: ts=2026-09-21T12:09:22Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49054 duration_s=1 drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv assign_id=4 outputs=14285worker2 # [ 51.782723] nix-daemon[863]: reaped child process 1037, status = succeeded4286client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 --argstr top t1b >&2, in 2.30 seconds)4287client: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 --argstr top t1b outPath4288client: (finished: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 --argstr top t1b outPath, in 0.23 seconds)4289lb: must succeed: curl -sf localhost:9901/clusters4290lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)4291lb: must succeed: curl -sf localhost:9901/clusters4292lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)4293worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4294worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.06 seconds)4295(finished: subtest: inputs already in the cache are not sent to the scheduler, in 2.93 seconds)4296subtest: two clients wanting the same drv build it once4297client: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag dedup4298client # [ 52.334854] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key" --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag dedup.4299client # Running as unit: dedup1.service; invocation ID: ef942b66b93e4bf5bd7c67f27dca52f14300client: (finished: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag dedup, in 0.09 seconds)4301client: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag dedup4302client # [ 52.596724] nix[886]: don't know how to build these paths:4303client # [ 52.598377] nix[886]: /nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv4304worker2 # [ 52.579149] nix-grpc-daemon[778]: ts=2026-09-21T12:09:23Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:490544305worker2 # [ 52.592122] nix-grpc-daemon[778]: ts=2026-09-21T12:09:23Z level=debug event=want client=4 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv cached=04306worker2 # [ 52.598523] nix-grpc-daemon[778]: ts=2026-09-21T12:09:23Z level=debug event=assigned drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv worker=192.168.1.3:50051 clients=14307worker1 # [ 52.593735] nix-grpc-daemon[777]: ts=2026-09-21T12:09:23Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:401104308worker1 # [ 52.611238] nix-grpc-daemon[777]: ts=2026-09-21T12:09:23Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:401104309client # [ 52.650287] nix[886]: copying 1 paths to 'lb:50051'...4310client # [ 52.653024] nix[886]: copying path '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' to 'lb:50051'...4311worker1 # [ 52.626988] nix-grpc-daemon[777]: ts=2026-09-21T12:09:23Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:401104312worker1 # [ 52.642101] nix-daemon[842]: accepted connection from pid 777, user nix-grpc-daemon (trusted)4313worker1 # [ 52.670356] nix-grpc-daemon[777]: ts=2026-09-21T12:09:23Z level=info event=niks3_push_started pid=8704314worker1 # [ 52.940183] nix-grpc-daemon[870]: time=2026-09-21T12:09:23.486Z level=INFO msg="Configuring client TLS" cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker1.pem key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker1.key ca=""4315worker1 # [ 53.133563] nix-daemon[842]: accepted connection from pid 883, user nix-grpc-daemon (trusted)4316worker1 # [ 53.151327] nix-daemon[842]: reaped child process 886, status = succeeded4317worker1 # [ 53.322490] nix-daemon[842]: accepted connection from pid 888, user nix-grpc-daemon (trusted)4318worker1 # [ 53.338733] nix-daemon[842]: reaped child process 891, status = succeeded4319lb # [ 53.373332] niks3-server[1060]: 2026/09/21 12:09:23 INFO Received uploads request method=POST path=/api/pending_closures4320worker1 # [ 53.362883] nix-grpc-daemon[870]: time=2026-09-21T12:09:23.910Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4321worker1 # [ 53.366314] nix-grpc-daemon[870]: time=2026-09-21T12:09:23.912Z level=INFO msg="Uploading cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv (432B)"4322lb # [ 53.411757] niks3-server[1060]: 2026/09/21 12:09:23 INFO Registered completed upload object_key=nar/0l7vpaziqxpw0srrsgimqvaa0dyxa9glicn3ilkwk06l64hn837w.nar.zst4323lb # [ 53.426978] niks3-server[1060]: 2026/09/21 12:09:23 INFO Received sign narinfos request method=POST path=/api/pending_closures/9/sign4324worker1 # [ 53.412075] nix-grpc-daemon[870]: time=2026-09-21T12:09:23.959Z level=INFO msg="Uploading 1 narinfos"4325lb # [ 53.440151] niks3-server[1060]: 2026/09/21 12:09:23 INFO Signed narinfos id=9 count=14326lb # [ 53.446629] niks3-server[1060]: 2026/09/21 12:09:23 INFO Registered completed upload object_key=cp23830c44gs2dw4xy80s5wqyjm7vw84.ls4327client # [ 53.450841] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key" --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag dedup.4328worker1 # [ 53.438552] nix-grpc-daemon[870]: time=2026-09-21T12:09:23.986Z level=INFO msg="Upload complete. (489ms)"4329lb # [ 53.461469] niks3-server[1060]: 2026/09/21 12:09:23 INFO Received complete upload request method=POST path=/api/pending_closures/9/complete4330lb # [ 53.467242] niks3-server[1060]: 2026/09/21 12:09:24 INFO Completed upload id=94331worker1 # [ 53.448292] nix-grpc-daemon[777]: ts=2026-09-21T12:09:23Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40110 duration_s=0 paths=1 nar_bytes_in=4324332lb # [ 53.469649] niks3-server[1060]: 2026/09/21 12:09:24 INFO Registered completed upload object_key=cp23830c44gs2dw4xy80s5wqyjm7vw84.narinfo4333client: (finished: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag dedup, in 1.13 seconds)4334worker1 # [ 53.459811] nix-grpc-daemon[777]: ts=2026-09-21T12:09:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:401104335worker1 # [ 53.471940] nix-daemon[842]: accepted connection from pid 777, user nix-grpc-daemon (trusted)4336worker1 # [ 53.474349] nix-daemon[842]: reaped child process 868, status = succeeded4337client # Running as unit: dedup2.service; invocation ID: c25892e48699400b8e6ac1766af0dfa74338client # [ 53.520587] nix[886]: building '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' on lb:50051...4339worker1 # [ 53.499028] nix-daemon[842]: accepted connection from pid 777, user nix-grpc-daemon (trusted)4340client # [ 53.837706] nix[902]: don't know how to build these paths:4341client # [ 53.839492] nix[902]: /nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv4342worker2 # [ 53.821287] nix-grpc-daemon[778]: ts=2026-09-21T12:09:24Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:490544343worker2 # [ 53.834364] nix-grpc-daemon[778]: ts=2026-09-21T12:09:24Z level=debug event=want client=5 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv cached=04344worker1 # [ 53.829804] nix-grpc-daemon[777]: ts=2026-09-21T12:09:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:401104345client # [ 53.863963] nix[902]: building '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' on lb:50051...4346client: must succeed: sleep 24347client: (finished: must succeed: sleep 2, in 2.06 seconds)4348client: waiting for success: ! systemctl is-active dedup1 dedup24349worker1 # [ 57.153303] nix-daemon[842]: reaped child process 899, status = succeeded4350worker1 # [ 57.333132] nix-daemon[842]: accepted connection from pid 925, user nix-grpc-daemon (trusted)4351worker1 # [ 57.349587] nix-daemon[842]: reaped child process 928, status = succeeded4352lb # [ 57.383718] niks3-server[1060]: 2026/09/21 12:09:27 INFO Received uploads request method=POST path=/api/pending_closures4353worker1 # [ 57.375066] nix-grpc-daemon[870]: time=2026-09-21T12:09:27.921Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4354worker1 # [ 57.377380] nix-grpc-daemon[870]: time=2026-09-21T12:09:27.921Z level=INFO msg="Uploading vamz940my4dma9a2daklr9pw9x9pp9hp-slow-dedup (120B)"4355lb # [ 57.423376] niks3-server[1060]: 2026/09/21 12:09:27 INFO Registered completed upload object_key=log/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv4356lb # [ 57.437362] niks3-server[1060]: 2026/09/21 12:09:27 INFO Registered completed upload object_key=nar/02xhr0h0sj1ji8m0s8995xzbbb8mhbqdpdl1ggx94yqqaykihvz7.nar.zst4357lb # [ 57.453431] niks3-server[1060]: 2026/09/21 12:09:27 INFO Received sign narinfos request method=POST path=/api/pending_closures/10/sign4358worker1 # [ 57.440152] nix-grpc-daemon[870]: time=2026-09-21T12:09:27.986Z level=INFO msg="Uploading 1 narinfos"4359lb # [ 57.461774] niks3-server[1060]: 2026/09/21 12:09:28 INFO Signed narinfos id=10 count=14360lb # [ 57.465623] niks3-server[1060]: 2026/09/21 12:09:28 INFO Registered completed upload object_key=vamz940my4dma9a2daklr9pw9x9pp9hp.ls4361lb # [ 57.482277] niks3-server[1060]: 2026/09/21 12:09:28 INFO Received complete upload request method=POST path=/api/pending_closures/10/complete4362lb # [ 57.489165] niks3-server[1060]: 2026/09/21 12:09:28 INFO Completed upload id=104363worker1 # [ 57.467885] nix-grpc-daemon[870]: time=2026-09-21T12:09:28.015Z level=INFO msg="Upload complete. (348ms)"4364lb # [ 57.492240] niks3-server[1060]: 2026/09/21 12:09:28 INFO Registered completed upload object_key=vamz940my4dma9a2daklr9pw9x9pp9hp.narinfo4365worker1 # [ 57.481242] nix-grpc-daemon[777]: ts=2026-09-21T12:09:28Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40110 duration_s=3 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv assign_id=5 outputs=14366worker1 # [ 57.498532] nix-grpc-daemon[777]: ts=2026-09-21T12:09:28Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40110 duration_s=4 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv assign_id=5 outputs=14367worker1 # [ 57.505712] nix-daemon[842]: reaped child process 895, status = succeeded4368client # [ 57.557871] systemd[1]: dedup2.service: Deactivated successfully.4369client # [ 57.563185] systemd[1]: dedup2.service: Consumed 215ms CPU time over 4.106s wall clock time, 33.2M memory peak, 5.8K incoming IP traffic, 5.6K outgoing IP traffic.4370client # [ 57.601777] systemd[1]: dedup1.service: Deactivated successfully.4371client # [ 57.604150] systemd[1]: dedup1.service: Consumed 169ms CPU time over 5.269s wall clock time, 39.3M memory peak, 7.3K incoming IP traffic, 8.7K outgoing IP traffic.4372client: (finished: waiting for success: ! systemctl is-active dedup1 dedup2, in 1.14 seconds)4373client: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }4374client: (finished: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }, in 0.06 seconds)4375worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true4376worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.07 seconds)4377worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true4378worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.07 seconds)4379worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4380worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)4381worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4382worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)4383(finished: subtest: two clients wanting the same drv build it once, in 6.29 seconds)4384subtest: build hook: nix-daemon with builders = grpc://lb4385client: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --print-out-paths --no-link -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag hook 2>/tmp/hook.log4386client # [ 58.772594] systemd[1]: Started Nix Daemon.4387client # [ 58.957063] nix-daemon[944]: accepted connection from pid 941, user root (trusted)4388worker2 # [ 59.163585] nix-grpc-daemon[778]: ts=2026-09-21T12:09:29Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:490544389worker2 # [ 59.170311] nix-grpc-daemon[778]: ts=2026-09-21T12:09:29Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:490544390worker2 # [ 59.183878] nix-grpc-daemon[778]: ts=2026-09-21T12:09:29Z level=debug event=want client=6 drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv cached=04391worker2 # [ 59.191324] nix-grpc-daemon[778]: ts=2026-09-21T12:09:29Z level=debug event=assigned drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv worker=192.168.1.4:50051 clients=14392worker2 # [ 59.233899] nix-grpc-daemon[778]: ts=2026-09-21T12:09:29Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:490544393worker2 # [ 59.249048] nix-grpc-daemon[778]: ts=2026-09-21T12:09:29Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:490544394worker2 # [ 59.263808] nix-grpc-daemon[778]: ts=2026-09-21T12:09:29Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:490544395worker2 # [ 59.289578] nix-daemon[863]: accepted connection from pid 778, user nix-grpc-daemon (trusted)4396worker2 # [ 59.494655] nix-daemon[863]: accepted connection from pid 1094, user nix-grpc-daemon (trusted)4397worker2 # [ 59.514473] nix-daemon[863]: reaped child process 1097, status = succeeded4398worker2 # [ 59.689122] nix-daemon[863]: accepted connection from pid 1099, user nix-grpc-daemon (trusted)4399worker2 # [ 59.705388] nix-daemon[863]: reaped child process 1102, status = succeeded4400lb # [ 59.727897] niks3-server[1060]: 2026/09/21 12:09:30 INFO Received uploads request method=POST path=/api/pending_closures4401worker2 # [ 59.727345] nix-grpc-daemon[878]: time=2026-09-21T12:09:30.270Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4402worker2 # [ 59.728919] nix-grpc-daemon[878]: time=2026-09-21T12:09:30.270Z level=INFO msg="Uploading xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv (512B)"4403lb # [ 59.757312] niks3-server[1060]: 2026/09/21 12:09:30 INFO Registered completed upload object_key=nar/1f1i5kab3qhx7pg1k0ma5r1m7ipfazkfkfb944dm7qcjr466pwij.nar.zst4404worker2 # [ 59.761123] nix-grpc-daemon[878]: time=2026-09-21T12:09:30.304Z level=INFO msg="Uploading 1 narinfos"4405lb # [ 59.772219] niks3-server[1060]: 2026/09/21 12:09:30 INFO Received sign narinfos request method=POST path=/api/pending_closures/11/sign4406lb # [ 59.781492] niks3-server[1060]: 2026/09/21 12:09:30 INFO Signed narinfos id=11 count=14407lb # [ 59.788139] niks3-server[1060]: 2026/09/21 12:09:30 INFO Registered completed upload object_key=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv.ls4408lb # [ 59.797862] niks3-server[1060]: 2026/09/21 12:09:30 INFO Received complete upload request method=POST path=/api/pending_closures/11/complete4409worker2 # [ 59.792496] nix-grpc-daemon[878]: time=2026-09-21T12:09:30.336Z level=INFO msg="Upload complete. (486ms)"4410lb # [ 59.803860] niks3-server[1060]: 2026/09/21 12:09:30 INFO Completed upload id=114411lb # [ 59.807796] niks3-server[1060]: 2026/09/21 12:09:30 INFO Registered completed upload object_key=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv.narinfo4412worker2 # [ 59.800338] nix-grpc-daemon[778]: ts=2026-09-21T12:09:30Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49054 duration_s=0 paths=1 nar_bytes_in=5124413worker2 # [ 59.808846] nix-grpc-daemon[778]: ts=2026-09-21T12:09:30Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:490544414worker2 # [ 59.819591] nix-daemon[863]: accepted connection from pid 778, user nix-grpc-daemon (trusted)4415worker2 # [ 59.824493] nix-daemon[863]: reaped child process 1092, status = succeeded4416worker2 # [ 59.837524] nix-daemon[863]: accepted connection from pid 778, user nix-grpc-daemon (trusted)4417lb # [ 60.661269] envoy[910]: conn peer=[2001:db8:1::1]:57240 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4215 tx=1801 ms=20084418lb # [ 60.666332] envoy[910]: conn peer=[2001:db8:1::1]:60994 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1430 tx=1085 ms=36924419lb # [ 60.671225] envoy[910]: conn peer=[2001:db8:1::1]:60978 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3035 tx=1468 ms=49714420worker2 # [ 61.086215] nix-daemon[863]: reaped child process 1106, status = succeeded4421worker2 # [ 61.268126] nix-daemon[863]: accepted connection from pid 1122, user nix-grpc-daemon (trusted)4422worker2 # [ 61.285827] nix-daemon[863]: reaped child process 1125, status = succeeded4423lb # [ 61.309276] niks3-server[1060]: 2026/09/21 12:09:31 INFO Received uploads request method=POST path=/api/pending_closures4424worker2 # [ 61.308512] nix-grpc-daemon[878]: time=2026-09-21T12:09:31.851Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4425worker2 # [ 61.309927] nix-grpc-daemon[878]: time=2026-09-21T12:09:31.851Z level=INFO msg="Uploading yfs64sxmsg77fnzczdvc3g3yfdwh9fks-farm-a (128B)"4426lb # [ 61.351814] niks3-server[1060]: 2026/09/21 12:09:31 INFO Registered completed upload object_key=nar/0pbbxl4kxy7zl7x4lp43ywljwn82mxzna2wgpb1bz1z765nwp9ki.nar.zst4427lb # [ 61.363616] niks3-server[1060]: 2026/09/21 12:09:31 INFO Registered completed upload object_key=log/xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv4428lb # [ 61.371759] niks3-server[1060]: 2026/09/21 12:09:31 INFO Received sign narinfos request method=POST path=/api/pending_closures/12/sign4429worker2 # [ 61.366710] nix-grpc-daemon[878]: time=2026-09-21T12:09:31.910Z level=INFO msg="Uploading 1 narinfos"4430lb # [ 61.379781] niks3-server[1060]: 2026/09/21 12:09:31 INFO Signed narinfos id=12 count=14431lb # [ 61.384113] niks3-server[1060]: 2026/09/21 12:09:31 INFO Registered completed upload object_key=yfs64sxmsg77fnzczdvc3g3yfdwh9fks.ls4432lb # [ 61.390982] niks3-server[1060]: 2026/09/21 12:09:31 INFO Received complete upload request method=POST path=/api/pending_closures/12/complete4433lb # [ 61.396643] niks3-server[1060]: 2026/09/21 12:09:31 INFO Completed upload id=124434worker2 # [ 61.386447] nix-grpc-daemon[878]: time=2026-09-21T12:09:31.930Z level=INFO msg="Upload complete. (338ms)"4435lb # [ 61.398903] niks3-server[1060]: 2026/09/21 12:09:31 INFO Registered completed upload object_key=yfs64sxmsg77fnzczdvc3g3yfdwh9fks.narinfo4436worker2 # [ 61.396990] nix-grpc-daemon[778]: ts=2026-09-21T12:09:31Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49054 duration_s=1 drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv assign_id=6 outputs=14437worker1 # [ 61.390604] nix-grpc-daemon[777]: ts=2026-09-21T12:09:31Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:401104438worker2 # [ 61.415529] nix-daemon[863]: reaped child process 1104, status = succeeded4439worker1 # [ 61.411209] nix-grpc-daemon[777]: copying path '/nix/store/yfs64sxmsg77fnzczdvc3g3yfdwh9fks-farm-a' from 'http://lb:5751'...4440worker2 # [ 61.469700] nix-grpc-daemon[778]: ts=2026-09-21T12:09:32Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:490544441worker2 # [ 61.477776] nix-grpc-daemon[778]: ts=2026-09-21T12:09:32Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:49054 duration_s=0 paths=1 nar_bytes_out=1284442worker1 # [ 61.748040] nix-grpc-daemon[777]: ts=2026-09-21T12:09:32Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:401104443worker2 # [ 61.763515] nix-grpc-daemon[778]: ts=2026-09-21T12:09:32Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:490544444worker2 # [ 61.773141] nix-grpc-daemon[878]: time=2026-09-21T12:09:32.316Z level=INFO msg="All 1 paths already cached"4445worker2 # [ 61.790068] nix-grpc-daemon[778]: ts=2026-09-21T12:09:32Z level=debug event=want client=7 drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv cached=04446worker2 # [ 61.799249] nix-grpc-daemon[778]: ts=2026-09-21T12:09:32Z level=debug event=assigned drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv worker=192.168.1.4:50051 clients=14447worker2 # [ 61.807701] nix-grpc-daemon[778]: ts=2026-09-21T12:09:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:490544448worker2 # [ 61.815191] nix-grpc-daemon[778]: ts=2026-09-21T12:09:32Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:490544449worker2 # [ 61.825351] nix-grpc-daemon[878]: time=2026-09-21T12:09:32.368Z level=INFO msg="All 1 paths already cached"4450worker2 # [ 61.833918] nix-grpc-daemon[778]: ts=2026-09-21T12:09:32Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:490544451worker2 # [ 61.848113] nix-daemon[863]: accepted connection from pid 778, user nix-grpc-daemon (trusted)4452worker2 # [ 62.051586] nix-daemon[863]: accepted connection from pid 1131, user nix-grpc-daemon (trusted)4453worker2 # [ 62.070728] nix-daemon[863]: reaped child process 1134, status = succeeded4454worker2 # [ 62.241221] nix-daemon[863]: accepted connection from pid 1136, user nix-grpc-daemon (trusted)4455worker2 # [ 62.257111] nix-daemon[863]: reaped child process 1139, status = succeeded4456lb # [ 62.278510] niks3-server[1060]: 2026/09/21 12:09:32 INFO Received uploads request method=POST path=/api/pending_closures4457worker2 # [ 62.277322] nix-grpc-daemon[878]: time=2026-09-21T12:09:32.820Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4458worker2 # [ 62.278773] nix-grpc-daemon[878]: time=2026-09-21T12:09:32.820Z level=INFO msg="Uploading r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv (624B)"4459lb # [ 62.304480] niks3-server[1060]: 2026/09/21 12:09:32 INFO Registered completed upload object_key=nar/15k66v3mh3v6p5vymv4pv3v3sfl56i2b1ia2rf093c37w6bhc04m.nar.zst4460worker2 # [ 62.307947] nix-grpc-daemon[878]: time=2026-09-21T12:09:32.851Z level=INFO msg="Uploading 1 narinfos"4461lb # [ 62.321416] niks3-server[1060]: 2026/09/21 12:09:32 INFO Received sign narinfos request method=POST path=/api/pending_closures/13/sign4462lb # [ 62.333543] niks3-server[1060]: 2026/09/21 12:09:32 INFO Signed narinfos id=13 count=14463lb # [ 62.341937] niks3-server[1060]: 2026/09/21 12:09:32 INFO Registered completed upload object_key=r1jsr4b2g1qwly395z6idx3z2362ih8i.ls4464lb # [ 62.346920] niks3-server[1060]: 2026/09/21 12:09:32 INFO Received complete upload request method=POST path=/api/pending_closures/13/complete4465lb # [ 62.350697] niks3-server[1060]: 2026/09/21 12:09:32 INFO Registered completed upload object_key=r1jsr4b2g1qwly395z6idx3z2362ih8i.narinfo4466lb # [ 62.354946] niks3-server[1060]: 2026/09/21 12:09:32 INFO Completed upload id=134467worker2 # [ 62.348138] nix-grpc-daemon[878]: time=2026-09-21T12:09:32.888Z level=INFO msg="Upload complete. (481ms)"4468worker2 # [ 62.356638] nix-grpc-daemon[778]: ts=2026-09-21T12:09:32Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49054 duration_s=0 paths=1 nar_bytes_in=6244469worker2 # [ 62.367348] nix-grpc-daemon[778]: ts=2026-09-21T12:09:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:490544470worker2 # [ 62.373442] nix-daemon[863]: accepted connection from pid 778, user nix-grpc-daemon (trusted)4471worker2 # [ 62.378583] nix-daemon[863]: reaped child process 1129, status = succeeded4472worker2 # [ 62.389734] nix-daemon[863]: accepted connection from pid 778, user nix-grpc-daemon (trusted)4473worker2 # [ 63.617557] nix-daemon[863]: reaped child process 1143, status = succeeded4474worker2 # [ 63.777277] nix-daemon[863]: accepted connection from pid 1159, user nix-grpc-daemon (trusted)4475worker2 # [ 63.795956] nix-daemon[863]: reaped child process 1162, status = succeeded4476lb # [ 63.818807] niks3-server[1060]: 2026/09/21 12:09:34 INFO Received uploads request method=POST path=/api/pending_closures4477worker2 # [ 63.818815] nix-grpc-daemon[878]: time=2026-09-21T12:09:34.361Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4478worker2 # [ 63.820390] nix-grpc-daemon[878]: time=2026-09-21T12:09:34.361Z level=INFO msg="Uploading la6ryzsihag4zpxgxls54fsg34dvlalk-farm-b (176B)"4479lb # [ 63.849921] niks3-server[1060]: 2026/09/21 12:09:34 INFO Registered completed upload object_key=nar/069d0xnyw2q345iqzx6agxc5fxifc6q36bkkklw2aa4qwanxdd1g.nar.zst4480lb # [ 63.865680] niks3-server[1060]: 2026/09/21 12:09:34 INFO Registered completed upload object_key=la6ryzsihag4zpxgxls54fsg34dvlalk.ls4481worker2 # [ 63.864571] nix-grpc-daemon[878]: time=2026-09-21T12:09:34.408Z level=INFO msg="Uploading 1 narinfos"4482lb # [ 63.874990] niks3-server[1060]: 2026/09/21 12:09:34 INFO Received sign narinfos request method=POST path=/api/pending_closures/14/sign4483lb # [ 63.885652] niks3-server[1060]: 2026/09/21 12:09:34 INFO Signed narinfos id=14 count=14484lb # [ 63.895893] niks3-server[1060]: 2026/09/21 12:09:34 INFO Registered completed upload object_key=log/r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv4485worker2 # [ 63.892516] nix-grpc-daemon[878]: time=2026-09-21T12:09:34.436Z level=INFO msg="Upload complete. (291ms)"4486lb # [ 63.903265] niks3-server[1060]: 2026/09/21 12:09:34 INFO Received complete upload request method=POST path=/api/pending_closures/14/complete4487lb # [ 63.909243] niks3-server[1060]: 2026/09/21 12:09:34 INFO Completed upload id=144488lb # [ 63.912086] niks3-server[1060]: 2026/09/21 12:09:34 INFO Registered completed upload object_key=la6ryzsihag4zpxgxls54fsg34dvlalk.narinfo4489worker2 # [ 63.902991] nix-grpc-daemon[778]: ts=2026-09-21T12:09:34Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49054 duration_s=1 drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv assign_id=7 outputs=14490worker1 # [ 63.903292] nix-grpc-daemon[777]: ts=2026-09-21T12:09:34Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:401104491worker2 # [ 63.918213] nix-daemon[863]: reaped child process 1141, status = succeeded4492worker1 # [ 63.915019] nix-grpc-daemon[777]: copying path '/nix/store/la6ryzsihag4zpxgxls54fsg34dvlalk-farm-b' from 'http://lb:5751'...4493worker2 # [ 63.965796] nix-grpc-daemon[778]: ts=2026-09-21T12:09:34Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:490544494worker2 # [ 63.973994] nix-grpc-daemon[778]: ts=2026-09-21T12:09:34Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:49054 duration_s=0 paths=1 nar_bytes_out=1764495worker2 # [ 64.241869] nix-grpc-daemon[778]: ts=2026-09-21T12:09:34Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:490544496worker1 # [ 64.234374] nix-grpc-daemon[777]: ts=2026-09-21T12:09:34Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:401104497worker1 # [ 64.246560] nix-grpc-daemon[870]: time=2026-09-21T12:09:34.793Z level=INFO msg="All 2 paths already cached"4498worker2 # [ 64.276184] nix-grpc-daemon[778]: ts=2026-09-21T12:09:34Z level=debug event=want client=8 drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv cached=04499worker2 # [ 64.283051] nix-grpc-daemon[778]: ts=2026-09-21T12:09:34Z level=debug event=assigned drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv worker=192.168.1.4:50051 clients=14500worker2 # [ 64.291790] nix-grpc-daemon[778]: ts=2026-09-21T12:09:34Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:490544501worker2 # [ 64.303071] nix-grpc-daemon[778]: ts=2026-09-21T12:09:34Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:490544502worker2 # [ 64.317502] nix-grpc-daemon[878]: time=2026-09-21T12:09:34.861Z level=INFO msg="All 2 paths already cached"4503worker2 # [ 64.326491] nix-grpc-daemon[778]: ts=2026-09-21T12:09:34Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:490544504worker2 # [ 64.340350] nix-daemon[863]: accepted connection from pid 778, user nix-grpc-daemon (trusted)4505worker2 # [ 64.548853] nix-daemon[863]: accepted connection from pid 1168, user nix-grpc-daemon (trusted)4506worker2 # [ 64.565926] nix-daemon[863]: reaped child process 1171, status = succeeded4507worker2 # [ 64.741760] nix-daemon[863]: accepted connection from pid 1173, user nix-grpc-daemon (trusted)4508worker2 # [ 64.758627] nix-daemon[863]: reaped child process 1176, status = succeeded4509lb # [ 64.780383] niks3-server[1060]: 2026/09/21 12:09:35 INFO Received uploads request method=POST path=/api/pending_closures4510worker2 # [ 64.779666] nix-grpc-daemon[878]: time=2026-09-21T12:09:35.323Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4511worker2 # [ 64.781359] nix-grpc-daemon[878]: time=2026-09-21T12:09:35.323Z level=INFO msg="Uploading aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv (752B)"4512lb # [ 64.808113] niks3-server[1060]: 2026/09/21 12:09:35 INFO Registered completed upload object_key=nar/1ahq7sp1sl2scxqzfjd8gqimpqfkjbyi9ivxdhdwx0jzh2gbv74b.nar.zst4513lb # [ 64.823809] niks3-server[1060]: 2026/09/21 12:09:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/15/sign4514worker2 # [ 64.819910] nix-grpc-daemon[878]: time=2026-09-21T12:09:35.362Z level=INFO msg="Uploading 1 narinfos"4515lb # [ 64.837048] niks3-server[1060]: 2026/09/21 12:09:35 INFO Signed narinfos id=15 count=14516lb # [ 64.848633] niks3-server[1060]: 2026/09/21 12:09:35 INFO Registered completed upload object_key=aa4abikjj4w30bfx1p8hxmq0916qbz1c.ls4517lb # [ 64.852699] niks3-server[1060]: 2026/09/21 12:09:35 INFO Received complete upload request method=POST path=/api/pending_closures/15/complete4518worker2 # [ 64.846812] nix-grpc-daemon[878]: time=2026-09-21T12:09:35.390Z level=INFO msg="Upload complete. (489ms)"4519lb # [ 64.857536] niks3-server[1060]: 2026/09/21 12:09:35 INFO Completed upload id=154520lb # [ 64.860523] niks3-server[1060]: 2026/09/21 12:09:35 INFO Registered completed upload object_key=aa4abikjj4w30bfx1p8hxmq0916qbz1c.narinfo4521worker2 # [ 64.856298] nix-grpc-daemon[778]: ts=2026-09-21T12:09:35Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49054 duration_s=0 paths=1 nar_bytes_in=7524522worker2 # [ 64.867287] nix-grpc-daemon[778]: ts=2026-09-21T12:09:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:490544523worker2 # [ 64.874824] nix-daemon[863]: accepted connection from pid 778, user nix-grpc-daemon (trusted)4524worker2 # [ 64.879831] nix-daemon[863]: reaped child process 1166, status = succeeded4525worker2 # [ 64.891325] nix-daemon[863]: accepted connection from pid 778, user nix-grpc-daemon (trusted)4526worker2 # [ 66.126311] nix-daemon[863]: reaped child process 1180, status = succeeded4527worker2 # [ 66.298975] nix-daemon[863]: accepted connection from pid 1196, user nix-grpc-daemon (trusted)4528worker2 # [ 66.315990] nix-daemon[863]: reaped child process 1199, status = succeeded4529lb # [ 66.339122] niks3-server[1060]: 2026/09/21 12:09:36 INFO Received uploads request method=POST path=/api/pending_closures4530worker2 # [ 66.340389] nix-grpc-daemon[878]: time=2026-09-21T12:09:36.882Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4531worker2 # [ 66.341828] nix-grpc-daemon[878]: time=2026-09-21T12:09:36.882Z level=INFO msg="Uploading 39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top (232B)"4532lb # [ 66.382382] niks3-server[1060]: 2026/09/21 12:09:36 INFO Registered completed upload object_key=log/aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv4533lb # [ 66.393789] niks3-server[1060]: 2026/09/21 12:09:36 INFO Registered completed upload object_key=nar/0vq3rhfhhbqyx5pbvaa9ddj9wbw9arph334y3bp3idfrikwnjc92.nar.zst4534worker2 # [ 66.393788] nix-grpc-daemon[878]: time=2026-09-21T12:09:36.937Z level=INFO msg="Uploading 1 narinfos"4535lb # [ 66.405701] niks3-server[1060]: 2026/09/21 12:09:36 INFO Received sign narinfos request method=POST path=/api/pending_closures/16/sign4536lb # [ 66.414469] niks3-server[1060]: 2026/09/21 12:09:36 INFO Signed narinfos id=16 count=14537lb # [ 66.419047] niks3-server[1060]: 2026/09/21 12:09:36 INFO Registered completed upload object_key=39qxwzy2zkij70y9zvbr59pb0df38w5h.ls4538lb # [ 66.425309] niks3-server[1060]: 2026/09/21 12:09:36 INFO Received complete upload request method=POST path=/api/pending_closures/16/complete4539worker2 # [ 66.418664] nix-grpc-daemon[878]: time=2026-09-21T12:09:36.962Z level=INFO msg="Upload complete. (315ms)"4540lb # [ 66.430463] niks3-server[1060]: 2026/09/21 12:09:36 INFO Completed upload id=164541lb # [ 66.432868] niks3-server[1060]: 2026/09/21 12:09:36 INFO Registered completed upload object_key=39qxwzy2zkij70y9zvbr59pb0df38w5h.narinfo4542worker2 # [ 66.429051] nix-grpc-daemon[778]: ts=2026-09-21T12:09:36Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49054 duration_s=1 drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv assign_id=8 outputs=14543worker2 # [ 66.444877] nix-daemon[863]: reaped child process 1178, status = succeeded4544worker2 # [ 66.474341] nix-grpc-daemon[778]: ts=2026-09-21T12:09:37Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:490544545worker2 # [ 66.499801] nix-grpc-daemon[778]: ts=2026-09-21T12:09:37Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:490544546worker2 # [ 66.505078] nix-grpc-daemon[778]: ts=2026-09-21T12:09:37Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:49054 duration_s=0 paths=1 nar_bytes_out=2324547client # [ 66.647302] nix-daemon[944]: reaped child process 946, status = succeeded4548client: (finished: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --print-out-paths --no-link -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag hook 2>/tmp/hook.log, in 8.12 seconds)4549client: must succeed: grep farm-top-hook /nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top4550client: (finished: must succeed: grep farm-top-hook /nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top, in 0.05 seconds)4551client: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }4552client: (finished: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }, in 0.05 seconds)4553client: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }4554client: (finished: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }, in 0.03 seconds)4555(finished: subtest: build hook: nix-daemon with builders = grpc://lb, in 8.25 seconds)4556subtest: an input only one worker has still reaches the builder4557client: must succeed: nix-build --no-out-link /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix -A input --argstr tag w1only4558client # this derivation will be built:4559client # /nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv4560client # building '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv'...4561client: (finished: must succeed: nix-build --no-out-link /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix -A input --argstr tag w1only, in 0.43 seconds)4562client: must succeed: nix-store --export /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only > /tmp/shared/inp.closure4563client: (finished: must succeed: nix-store --export /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only > /tmp/shared/inp.closure, in 0.20 seconds)4564worker1: must succeed: nix-store --import < /tmp/shared/inp.closure4565worker1: (finished: must succeed: nix-store --import < /tmp/shared/inp.closure, in 0.24 seconds)4566worker2: must fail: test -e /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only4567worker2: (finished: must fail: test -e /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only, in 0.04 seconds)4568client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix job --argstr tag w1only --argstr salt a >&24569client # [ 67.928835] nix-daemon[944]: accepted connection from pid 1026, user root (trusted)4570client # this derivation will be built:4571client # /nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv4572client # building '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' on 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux'...4573worker1 # [ 68.117852] nix-grpc-daemon[777]: ts=2026-09-21T12:09:38Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:401104574worker1 # [ 68.124305] nix-grpc-daemon[777]: ts=2026-09-21T12:09:38Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:401104575worker1 # [ 68.337087] nix-daemon[842]: accepted connection from pid 946, user nix-grpc-daemon (trusted)4576worker1 # [ 68.352911] nix-daemon[842]: reaped child process 949, status = succeeded4577lb # [ 68.389622] niks3-server[1060]: 2026/09/21 12:09:38 INFO Received uploads request method=POST path=/api/pending_closures4578worker1 # [ 68.377099] nix-grpc-daemon[870]: time=2026-09-21T12:09:38.924Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4579worker1 # [ 68.379150] nix-grpc-daemon[870]: time=2026-09-21T12:09:38.926Z level=INFO msg="Uploading dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only (120B)"4580lb # [ 68.419795] niks3-server[1060]: 2026/09/21 12:09:38 INFO Registered completed upload object_key=nar/08ghl87f009pwqz85nbqv9hm0ln6gai6ggzilfmw3qnkaalfi6ss.nar.zst4581worker1 # [ 68.410342] nix-grpc-daemon[870]: time=2026-09-21T12:09:38.957Z level=INFO msg="Uploading 1 narinfos"4582lb # [ 68.433567] niks3-server[1060]: 2026/09/21 12:09:38 INFO Received sign narinfos request method=POST path=/api/pending_closures/17/sign4583lb # [ 68.444790] niks3-server[1060]: 2026/09/21 12:09:38 INFO Signed narinfos id=17 count=14584lb # [ 68.449807] niks3-server[1060]: 2026/09/21 12:09:38 INFO Registered completed upload object_key=dymj0d1vj8q0ns6p096qk7fii766dap6.ls4585worker1 # [ 68.435958] nix-grpc-daemon[870]: time=2026-09-21T12:09:38.983Z level=INFO msg="Upload complete. (306ms)"4586lb # [ 68.458406] niks3-server[1060]: 2026/09/21 12:09:38 INFO Received complete upload request method=POST path=/api/pending_closures/17/complete4587lb # [ 68.463100] niks3-server[1060]: 2026/09/21 12:09:38 INFO Completed upload id=174588lb # [ 68.466403] niks3-server[1060]: 2026/09/21 12:09:39 INFO Registered completed upload object_key=dymj0d1vj8q0ns6p096qk7fii766dap6.narinfo4589worker2 # [ 68.471315] nix-grpc-daemon[778]: ts=2026-09-21T12:09:39Z level=debug event=want client=9 drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv cached=04590worker2 # [ 68.477711] nix-grpc-daemon[778]: ts=2026-09-21T12:09:39Z level=debug event=assigned drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv worker=192.168.1.3:50051 clients=14591worker1 # [ 68.507029] nix-grpc-daemon[777]: ts=2026-09-21T12:09:39Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:401104592worker1 # [ 68.522599] nix-grpc-daemon[777]: ts=2026-09-21T12:09:39Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:401104593client # copying 2 paths to 'lb:50051'...4594client # copying path '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv' to 'lb:50051'...4595worker1 # [ 68.541108] nix-grpc-daemon[777]: ts=2026-09-21T12:09:39Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:401104596client # copying path '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' to 'lb:50051'...4597worker1 # [ 68.555935] nix-daemon[842]: accepted connection from pid 777, user nix-grpc-daemon (trusted)4598worker1 # [ 68.761530] nix-daemon[842]: accepted connection from pid 953, user nix-grpc-daemon (trusted)4599worker1 # [ 68.778265] nix-daemon[842]: reaped child process 956, status = succeeded4600worker1 # [ 68.949554] nix-daemon[842]: accepted connection from pid 958, user nix-grpc-daemon (trusted)4601worker1 # [ 68.965899] nix-daemon[842]: reaped child process 961, status = succeeded4602lb # [ 68.999893] niks3-server[1060]: 2026/09/21 12:09:39 INFO Received uploads request method=POST path=/api/pending_closures4603lb # [ 69.009462] niks3-server[1060]: 2026/09/21 12:09:39 INFO Received uploads request method=POST path=/api/pending_closures4604worker1 # [ 68.997017] nix-grpc-daemon[870]: time=2026-09-21T12:09:39.544Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"4605worker1 # [ 68.998503] nix-grpc-daemon[870]: time=2026-09-21T12:09:39.544Z level=INFO msg="Uploading 4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv (416B)"4606worker1 # [ 69.001532] nix-grpc-daemon[870]: time=2026-09-21T12:09:39.544Z level=INFO msg="Uploading 71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv (568B)"4607lb # [ 69.044374] niks3-server[1060]: 2026/09/21 12:09:39 INFO Registered completed upload object_key=nar/0i2vbprwp0wxa61nkss61gzivqmmjh6klkgv0rlfnxk46rcaba1v.nar.zst4608lb # [ 69.057539] niks3-server[1060]: 2026/09/21 12:09:39 INFO Registered completed upload object_key=nar/0p6zgq1gdngshl62q0hv3lslqwv90s7fmflwc7wgnycf4ka6ik69.nar.zst4609lb # [ 69.068993] niks3-server[1060]: 2026/09/21 12:09:39 INFO Registered completed upload object_key=71l2x9mybp0i90r6pdnglwjnayj4xpl8.ls4610lb # [ 69.076080] niks3-server[1060]: 2026/09/21 12:09:39 INFO Received sign narinfos request method=POST path=/api/pending_closures/19/sign4611lb # [ 69.080734] niks3-server[1060]: 2026/09/21 12:09:39 INFO Signed narinfos id=19 count=24612lb # [ 69.084174] niks3-server[1060]: 2026/09/21 12:09:39 INFO Received sign narinfos request method=POST path=/api/pending_closures/18/sign4613worker1 # [ 69.063452] nix-grpc-daemon[870]: time=2026-09-21T12:09:39.610Z level=INFO msg="Uploading 3 narinfos"4614lb # [ 69.093199] niks3-server[1060]: 2026/09/21 12:09:39 INFO Signed narinfos id=18 count=14615lb # [ 69.095265] niks3-server[1060]: 2026/09/21 12:09:39 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.ls4616lb # [ 69.114969] niks3-server[1060]: 2026/09/21 12:09:39 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.narinfo4617lb # [ 69.120829] niks3-server[1060]: 2026/09/21 12:09:39 INFO Received complete upload request method=POST path=/api/pending_closures/19/complete4618lb # [ 69.125651] niks3-server[1060]: 2026/09/21 12:09:39 INFO Completed upload id=194619lb # [ 69.128925] niks3-server[1060]: 2026/09/21 12:09:39 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.narinfo4620lb # [ 69.132537] niks3-server[1060]: 2026/09/21 12:09:39 INFO Received complete upload request method=POST path=/api/pending_closures/18/complete4621worker1 # [ 69.107728] nix-grpc-daemon[870]: time=2026-09-21T12:09:39.655Z level=WARN msg="Failed to register uploaded object" key=71l2x9mybp0i90r6pdnglwjnayj4xpl8.narinfo error="server returned 404: object is not pending upload\n"4622lb # [ 69.135675] niks3-server[1060]: 2026/09/21 12:09:39 INFO Completed upload id=184623worker1 # [ 69.128641] nix-grpc-daemon[870]: time=2026-09-21T12:09:39.663Z level=INFO msg="Upload complete. (541ms)"4624worker1 # [ 69.136902] nix-grpc-daemon[777]: ts=2026-09-21T12:09:39Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40110 duration_s=0 paths=2 nar_bytes_in=9844625worker1 # [ 69.147122] nix-grpc-daemon[777]: ts=2026-09-21T12:09:39Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:401104626worker1 # [ 69.156162] nix-daemon[842]: accepted connection from pid 777, user nix-grpc-daemon (trusted)4627worker1 # [ 69.158460] nix-daemon[842]: reaped child process 951, status = succeeded4628client # building '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' on lb:50051...4629worker1 # [ 69.168138] nix-daemon[842]: accepted connection from pid 777, user nix-grpc-daemon (trusted)4630client # uses-input-w1onlya> node-a: building 71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv4631client # uses-input-w1onlya> disabling binary cache 'https://cache.nixos.org' for 60 seconds4632worker1 # [ 70.025717] nix-daemon[842]: reaped child process 965, status = succeeded4633worker1 # [ 70.205648] nix-daemon[842]: accepted connection from pid 981, user nix-grpc-daemon (trusted)4634worker1 # [ 70.226300] nix-daemon[842]: reaped child process 984, status = succeeded4635lb # [ 70.263101] niks3-server[1060]: 2026/09/21 12:09:40 INFO Received uploads request method=POST path=/api/pending_closures4636worker1 # [ 70.253340] nix-grpc-daemon[870]: time=2026-09-21T12:09:40.800Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"4637lb # [ 70.303709] niks3-server[1060]: 2026/09/21 12:09:40 INFO Registered completed upload object_key=2i7aa0cdpp60vpdm180r56gqfh25m7hc.ls4638worker1 # [ 70.294443] nix-grpc-daemon[870]: time=2026-09-21T12:09:40.841Z level=INFO msg="Uploading 1 narinfos"4639lb # [ 70.317944] niks3-server[1060]: 2026/09/21 12:09:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/20/sign4640lb # [ 70.327135] niks3-server[1060]: 2026/09/21 12:09:40 INFO Signed narinfos id=20 count=14641lb # [ 70.332788] niks3-server[1060]: 2026/09/21 12:09:40 INFO Registered completed upload object_key=log/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv4642lb # [ 70.341007] niks3-server[1060]: 2026/09/21 12:09:40 INFO Received complete upload request method=POST path=/api/pending_closures/20/complete4643lb # [ 70.345854] niks3-server[1060]: 2026/09/21 12:09:40 INFO Completed upload id=204644worker1 # [ 70.323754] nix-grpc-daemon[870]: time=2026-09-21T12:09:40.871Z level=INFO msg="Upload complete. (344ms)"4645lb # [ 70.348417] niks3-server[1060]: 2026/09/21 12:09:40 INFO Registered completed upload object_key=2i7aa0cdpp60vpdm180r56gqfh25m7hc.narinfo4646client # copying 1 paths...4647worker1 # [ 70.337299] nix-grpc-daemon[777]: ts=2026-09-21T12:09:40Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40110 duration_s=1 drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv assign_id=9 outputs=14648worker2 # [ 70.358030] nix-grpc-daemon[778]: ts=2026-09-21T12:09:40Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:490544649worker1 # [ 70.355163] nix-daemon[842]: reaped child process 963, status = succeeded4650worker2 # [ 70.374965] nix-grpc-daemon[778]: copying path '/nix/store/2i7aa0cdpp60vpdm180r56gqfh25m7hc-uses-input-w1onlya' from 'http://lb:5751'...4651client # copying path '/nix/store/2i7aa0cdpp60vpdm180r56gqfh25m7hc-uses-input-w1onlya' from 'grpc://lb:50051'...4652worker2 # [ 70.439341] nix-grpc-daemon[778]: ts=2026-09-21T12:09:40Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:490544653worker2 # [ 70.444746] nix-grpc-daemon[778]: ts=2026-09-21T12:09:40Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:49054 duration_s=0 paths=1 nar_bytes_out=1204654client # [ 70.605442] nix-daemon[944]: reaped child process 1029, status = succeeded4655client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix job --argstr tag w1only --argstr salt a >&2, in 2.90 seconds)4656client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix job --argstr tag w1only --argstr salt b >&24657lb # [ 70.661745] envoy[910]: conn peer=[2001:db8:1::1]:51486 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=410 tx=333 ms=444658lb # [ 70.666819] envoy[910]: conn peer=[2001:db8:1::1]:32772 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3401 tx=1856 ms=23454659lb # [ 70.671830] envoy[910]: conn peer=[2001:db8:1::1]:51494 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3887 tx=2042 ms=22304660lb # [ 70.677590] envoy[910]: conn peer=[2001:db8:1::1]:51498 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=436 tx=375 ms=284661lb # [ 70.682290] envoy[910]: conn peer=[2001:db8:1::1]:51506 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4339 tx=2189 ms=23164662lb # [ 70.687112] envoy[910]: conn peer=[2001:db8:1::1]:51514 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=438 tx=406 ms=644663lb # [ 70.691944] envoy[910]: conn peer=[2001:db8:1::1]:51518 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3875 tx=1953 ms=23714664lb # [ 70.696731] envoy[910]: conn peer=[2001:db8:1::1]:51522 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=422 tx=328 ms=704665client # [ 70.854858] nix-daemon[944]: accepted connection from pid 1048, user root (trusted)4666client # this derivation will be built:4667client # /nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv4668worker1 # [ 71.087043] nix-grpc-daemon[777]: ts=2026-09-21T12:09:41Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:401104669client # building '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' on 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux'...4670worker1 # [ 71.093957] nix-grpc-daemon[777]: ts=2026-09-21T12:09:41Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:401104671worker1 # [ 71.105953] nix-grpc-daemon[870]: time=2026-09-21T12:09:41.653Z level=INFO msg="All 1 paths already cached"4672worker2 # [ 71.136233] nix-grpc-daemon[778]: ts=2026-09-21T12:09:41Z level=debug event=want client=10 drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv cached=04673worker2 # [ 71.136603] nix-grpc-daemon[778]: ts=2026-09-21T12:09:41Z level=debug event=assigned drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv worker=192.168.1.4:50051 clients=14674worker2 # [ 71.153487] nix-grpc-daemon[778]: ts=2026-09-21T12:09:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:490544675worker2 # [ 71.170501] nix-grpc-daemon[778]: ts=2026-09-21T12:09:41Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:490544676client # copying 1 paths to 'lb:50051'...4677worker2 # [ 71.193031] nix-grpc-daemon[778]: ts=2026-09-21T12:09:41Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:490544678client # copying path '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' to 'lb:50051'...4679worker2 # [ 71.211662] nix-daemon[863]: accepted connection from pid 778, user nix-grpc-daemon (trusted)4680worker2 # [ 71.242881] nix-grpc-daemon[778]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 60 ms (attempt 1/5)4681worker2 # [ 71.309100] nix-grpc-daemon[778]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 59 ms (attempt 2/5)4682worker2 # [ 71.376748] nix-grpc-daemon[778]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 84 ms (attempt 3/5)4683worker2 # [ 71.468156] nix-grpc-daemon[778]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 75 ms (attempt 4/5)4684worker2 # [ 71.543716] nix-grpc-daemon[778]: disabling binary cache 'https://cache.nixos.org' for 60 seconds4685worker2 # [ 71.550463] nix-grpc-daemon[778]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4686worker2 # [ 71.557480] nix-grpc-daemon[778]: copying path '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv' from 'http://lb:5751'...4687worker2 # [ 71.795367] nix-daemon[863]: accepted connection from pid 1216, user nix-grpc-daemon (trusted)4688worker2 # [ 71.813570] nix-daemon[863]: reaped child process 1219, status = succeeded4689worker2 # [ 71.988905] nix-daemon[863]: accepted connection from pid 1221, user nix-grpc-daemon (trusted)4690worker2 # [ 72.004608] nix-daemon[863]: reaped child process 1224, status = succeeded4691lb # [ 72.030028] niks3-server[1060]: 2026/09/21 12:09:42 INFO Received uploads request method=POST path=/api/pending_closures4692worker2 # [ 72.032035] nix-grpc-daemon[878]: time=2026-09-21T12:09:42.574Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4693worker2 # [ 72.033540] nix-grpc-daemon[878]: time=2026-09-21T12:09:42.574Z level=INFO msg="Uploading 12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv (568B)"4694lb # [ 72.062779] niks3-server[1060]: 2026/09/21 12:09:42 INFO Registered completed upload object_key=nar/14jznd749vfl63s1gv7i29r5z6jcai2byp2nz83dr5gib6z2cyaz.nar.zst4695lb # [ 72.078245] niks3-server[1060]: 2026/09/21 12:09:42 INFO Received sign narinfos request method=POST path=/api/pending_closures/21/sign4696worker2 # [ 72.074860] nix-grpc-daemon[878]: time=2026-09-21T12:09:42.618Z level=INFO msg="Uploading 1 narinfos"4697lb # [ 72.090536] niks3-server[1060]: 2026/09/21 12:09:42 INFO Signed narinfos id=21 count=14698lb # [ 72.098121] niks3-server[1060]: 2026/09/21 12:09:42 INFO Registered completed upload object_key=12ialcyr3l4m98rzfgk6apzimy55z3j8.ls4699worker2 # [ 72.100512] nix-grpc-daemon[878]: time=2026-09-21T12:09:42.644Z level=INFO msg="Upload complete. (510ms)"4700lb # [ 72.112152] niks3-server[1060]: 2026/09/21 12:09:42 INFO Received complete upload request method=POST path=/api/pending_closures/21/complete4701lb # [ 72.118526] niks3-server[1060]: 2026/09/21 12:09:42 INFO Completed upload id=214702worker2 # [ 72.108110] nix-grpc-daemon[778]: ts=2026-09-21T12:09:42Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49054 duration_s=0 paths=1 nar_bytes_in=5684703lb # [ 72.123408] niks3-server[1060]: 2026/09/21 12:09:42 INFO Registered completed upload object_key=12ialcyr3l4m98rzfgk6apzimy55z3j8.narinfo4704worker2 # [ 72.119069] nix-grpc-daemon[778]: ts=2026-09-21T12:09:42Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:490544705worker2 # [ 72.130868] nix-daemon[863]: accepted connection from pid 778, user nix-grpc-daemon (trusted)4706worker2 # [ 72.136306] nix-daemon[863]: reaped child process 1209, status = succeeded4707worker2 # [ 72.155760] nix-grpc-daemon[778]: copying path '/nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only' from 'http://lb:5751'...4708client # building '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' on lb:50051...4709worker2 # [ 72.172242] nix-daemon[863]: accepted connection from pid 778, user nix-grpc-daemon (trusted)4710client # uses-input-w1onlyb> worker2: building 12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv4711client # uses-input-w1onlyb> disabling binary cache 'https://cache.nixos.org' for 60 seconds4712worker2 # [ 73.604345] nix-daemon[863]: reaped child process 1229, status = succeeded4713worker2 # [ 73.770769] nix-daemon[863]: accepted connection from pid 1245, user nix-grpc-daemon (trusted)4714worker2 # [ 73.788939] nix-daemon[863]: reaped child process 1248, status = succeeded4715lb # [ 73.813843] niks3-server[1060]: 2026/09/21 12:09:44 INFO Received uploads request method=POST path=/api/pending_closures4716worker2 # [ 73.816056] nix-grpc-daemon[878]: time=2026-09-21T12:09:44.357Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"4717lb # [ 73.841768] niks3-server[1060]: 2026/09/21 12:09:44 INFO Registered completed upload object_key=jfy62nvxcd0b9rkhly152q21ddbrcn28.ls4718worker2 # [ 73.843391] nix-grpc-daemon[878]: time=2026-09-21T12:09:44.386Z level=INFO msg="Uploading 1 narinfos"4719lb # [ 73.854515] niks3-server[1060]: 2026/09/21 12:09:44 INFO Received sign narinfos request method=POST path=/api/pending_closures/22/sign4720lb # [ 73.865968] niks3-server[1060]: 2026/09/21 12:09:44 INFO Signed narinfos id=22 count=14721worker2 # [ 73.865081] nix-grpc-daemon[878]: time=2026-09-21T12:09:44.408Z level=INFO msg="Upload complete. (279ms)"4722lb # [ 73.874819] niks3-server[1060]: 2026/09/21 12:09:44 INFO Registered completed upload object_key=log/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv4723worker2 # [ 73.871823] nix-grpc-daemon[778]: ts=2026-09-21T12:09:44Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49054 duration_s=1 drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv assign_id=10 outputs=14724lb # [ 73.883756] niks3-server[1060]: 2026/09/21 12:09:44 INFO Received complete upload request method=POST path=/api/pending_closures/22/complete4725lb # [ 73.888267] niks3-server[1060]: 2026/09/21 12:09:44 INFO Completed upload id=224726client # copying 1 paths...4727worker2 # [ 73.881345] nix-daemon[863]: reaped child process 1226, status = succeeded4728lb # [ 73.890403] niks3-server[1060]: 2026/09/21 12:09:44 INFO Registered completed upload object_key=jfy62nvxcd0b9rkhly152q21ddbrcn28.narinfo4729worker1 # [ 73.875798] nix-grpc-daemon[777]: ts=2026-09-21T12:09:44Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:401104730worker1 # [ 73.889770] nix-grpc-daemon[777]: copying path '/nix/store/jfy62nvxcd0b9rkhly152q21ddbrcn28-uses-input-w1onlyb' from 'http://lb:5751'...4731client # copying path '/nix/store/jfy62nvxcd0b9rkhly152q21ddbrcn28-uses-input-w1onlyb' from 'grpc://lb:50051'...4732worker1 # [ 73.936275] nix-grpc-daemon[777]: ts=2026-09-21T12:09:44Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:401104733worker1 # [ 73.943042] nix-grpc-daemon[777]: ts=2026-09-21T12:09:44Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:40110 duration_s=0 paths=1 nar_bytes_out=1204734client # [ 74.098902] nix-daemon[944]: reaped child process 1051, status = succeeded4735client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix job --argstr tag w1only --argstr salt b >&2, in 3.50 seconds)4736client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix job --argstr tag w1only --argstr salt c >&24737client # [ 74.321382] nix-daemon[944]: accepted connection from pid 1070, user root (trusted)4738client # this derivation will be built:4739client # /nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv4740worker2 # [ 74.521734] nix-grpc-daemon[778]: ts=2026-09-21T12:09:45Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:490544741client # building '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' on 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux'...4742worker1 # [ 74.515011] nix-grpc-daemon[777]: ts=2026-09-21T12:09:45Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:401104743worker1 # [ 74.526697] nix-grpc-daemon[870]: time=2026-09-21T12:09:45.073Z level=INFO msg="All 1 paths already cached"4744worker2 # [ 74.555451] nix-grpc-daemon[778]: ts=2026-09-21T12:09:45Z level=debug event=want client=11 drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv cached=04745worker2 # [ 74.562107] nix-grpc-daemon[778]: ts=2026-09-21T12:09:45Z level=debug event=assigned drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv worker=192.168.1.3:50051 clients=14746worker1 # [ 74.590606] nix-grpc-daemon[777]: ts=2026-09-21T12:09:45Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:401104747worker1 # [ 74.606980] nix-grpc-daemon[777]: ts=2026-09-21T12:09:45Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:401104748worker1 # [ 74.621483] nix-grpc-daemon[870]: time=2026-09-21T12:09:45.169Z level=INFO msg="All 1 paths already cached"4749client # copying 1 paths to 'lb:50051'...4750client # copying path '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' to 'lb:50051'...4751worker1 # [ 74.632137] nix-grpc-daemon[777]: ts=2026-09-21T12:09:45Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:401104752worker1 # [ 74.647703] nix-daemon[842]: accepted connection from pid 777, user nix-grpc-daemon (trusted)4753worker1 # [ 74.862421] nix-daemon[842]: accepted connection from pid 989, user nix-grpc-daemon (trusted)4754worker1 # [ 74.882184] nix-daemon[842]: reaped child process 992, status = succeeded4755worker1 # [ 75.059189] nix-daemon[842]: accepted connection from pid 994, user nix-grpc-daemon (trusted)4756worker1 # [ 75.076336] nix-daemon[842]: reaped child process 997, status = succeeded4757lb # [ 75.114225] niks3-server[1060]: 2026/09/21 12:09:45 INFO Received uploads request method=POST path=/api/pending_closures4758worker1 # [ 75.103445] nix-grpc-daemon[870]: time=2026-09-21T12:09:45.650Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4759worker1 # [ 75.105138] nix-grpc-daemon[870]: time=2026-09-21T12:09:45.650Z level=INFO msg="Uploading n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv (568B)"4760lb # [ 75.147612] niks3-server[1060]: 2026/09/21 12:09:45 INFO Registered completed upload object_key=nar/1qd3ps964395j2cc3rlqd58m21s7f6na9s0k2z5kclx0c3l4sl40.nar.zst4761worker1 # [ 75.138513] nix-grpc-daemon[870]: time=2026-09-21T12:09:45.685Z level=INFO msg="Uploading 1 narinfos"4762lb # [ 75.159826] niks3-server[1060]: 2026/09/21 12:09:45 INFO Received sign narinfos request method=POST path=/api/pending_closures/23/sign4763lb # [ 75.166785] niks3-server[1060]: 2026/09/21 12:09:45 INFO Signed narinfos id=23 count=14764lb # [ 75.171919] niks3-server[1060]: 2026/09/21 12:09:45 INFO Registered completed upload object_key=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr.ls4765lb # [ 75.181517] niks3-server[1060]: 2026/09/21 12:09:45 INFO Received complete upload request method=POST path=/api/pending_closures/23/complete4766lb # [ 75.186076] niks3-server[1060]: 2026/09/21 12:09:45 INFO Completed upload id=234767worker1 # [ 75.159763] nix-grpc-daemon[870]: time=2026-09-21T12:09:45.707Z level=INFO msg="Upload complete. (494ms)"4768lb # [ 75.188785] niks3-server[1060]: 2026/09/21 12:09:45 INFO Registered completed upload object_key=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr.narinfo4769worker1 # [ 75.168631] nix-grpc-daemon[777]: ts=2026-09-21T12:09:45Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40110 duration_s=0 paths=1 nar_bytes_in=5684770worker1 # [ 75.178820] nix-grpc-daemon[777]: ts=2026-09-21T12:09:45Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:401104771worker1 # [ 75.184435] nix-daemon[842]: accepted connection from pid 777, user nix-grpc-daemon (trusted)4772worker1 # [ 75.188773] nix-daemon[842]: reaped child process 987, status = succeeded4773client # building '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' on lb:50051...4774worker1 # [ 75.200423] nix-daemon[842]: accepted connection from pid 777, user nix-grpc-daemon (trusted)4775client # uses-input-w1onlyc> node-a: building n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv4776client # uses-input-w1onlyc> disabling binary cache 'https://cache.nixos.org' for 60 seconds4777worker1 # [ 76.630708] nix-daemon[842]: reaped child process 1001, status = succeeded4778worker1 # [ 76.803750] nix-daemon[842]: accepted connection from pid 1017, user nix-grpc-daemon (trusted)4779worker1 # [ 76.823222] nix-daemon[842]: reaped child process 1020, status = succeeded4780lb # [ 76.858432] niks3-server[1060]: 2026/09/21 12:09:47 INFO Received uploads request method=POST path=/api/pending_closures4781worker1 # [ 76.848149] nix-grpc-daemon[870]: time=2026-09-21T12:09:47.394Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"4782lb # [ 76.887792] niks3-server[1060]: 2026/09/21 12:09:47 INFO Received sign narinfos request method=POST path=/api/pending_closures/24/sign4783worker1 # [ 76.876854] nix-grpc-daemon[870]: time=2026-09-21T12:09:47.424Z level=INFO msg="Uploading 1 narinfos"4784lb # [ 76.903284] niks3-server[1060]: 2026/09/21 12:09:47 INFO Registered completed upload object_key=13r5ipj38sk1irdi724wm1mbl9lw720b.ls4785lb # [ 76.913854] niks3-server[1060]: 2026/09/21 12:09:47 INFO Signed narinfos id=24 count=14786worker1 # [ 76.902155] nix-grpc-daemon[870]: time=2026-09-21T12:09:47.449Z level=INFO msg="Upload complete. (292ms)"4787lb # [ 76.923529] niks3-server[1060]: 2026/09/21 12:09:47 INFO Registered completed upload object_key=log/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv4788lb # [ 76.929901] niks3-server[1060]: 2026/09/21 12:09:47 INFO Received complete upload request method=POST path=/api/pending_closures/24/complete4789lb # [ 76.934366] niks3-server[1060]: 2026/09/21 12:09:47 INFO Completed upload id=244790worker1 # [ 76.912849] nix-grpc-daemon[777]: ts=2026-09-21T12:09:47Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40110 duration_s=1 drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv assign_id=11 outputs=14791client # copying 1 paths...4792lb # [ 76.937530] niks3-server[1060]: 2026/09/21 12:09:47 INFO Registered completed upload object_key=13r5ipj38sk1irdi724wm1mbl9lw720b.narinfo4793worker1 # [ 76.924449] nix-daemon[842]: reaped child process 999, status = succeeded4794worker2 # [ 76.934228] nix-grpc-daemon[778]: ts=2026-09-21T12:09:47Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:490544795worker2 # [ 76.948303] nix-grpc-daemon[778]: copying path '/nix/store/13r5ipj38sk1irdi724wm1mbl9lw720b-uses-input-w1onlyc' from 'http://lb:5751'...4796client # copying path '/nix/store/13r5ipj38sk1irdi724wm1mbl9lw720b-uses-input-w1onlyc' from 'grpc://lb:50051'...4797worker2 # [ 76.991418] nix-grpc-daemon[778]: ts=2026-09-21T12:09:47Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:490544798worker2 # [ 76.998147] nix-grpc-daemon[778]: ts=2026-09-21T12:09:47Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:49054 duration_s=0 paths=1 nar_bytes_out=1204799client # [ 77.144798] nix-daemon[944]: reaped child process 1073, status = succeeded4800client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix job --argstr tag w1only --argstr salt c >&2, in 3.04 seconds)4801(finished: subtest: an input only one worker has still reaches the builder, in 10.36 seconds)4802subtest: upload whose reference the worker lacks is completed from the cache4803client: must succeed: nix-build --no-out-link /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix -A input --argstr tag viacache4804client # this derivation will be built:4805client # /nix/store/x1rrnjihpa1xvpyxxxdrjzp3c8z8nayq-local-input-viacache.drv4806client # building '/nix/store/x1rrnjihpa1xvpyxxxdrjzp3c8z8nayq-local-input-viacache.drv'...4807client: (finished: must succeed: nix-build --no-out-link /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix -A input --argstr tag viacache, in 0.45 seconds)4808client: must succeed: nix-build --no-out-link /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix -A referrer --argstr tag viacache4809client # this derivation will be built:4810client # /nix/store/1m587dk4visgyryf6n0mgp2rlp7pbjd1-refers-to-input-viacache.drv4811client # building '/nix/store/1m587dk4visgyryf6n0mgp2rlp7pbjd1-refers-to-input-viacache.drv'...4812client: (finished: must succeed: nix-build --no-out-link /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix -A referrer --argstr tag viacache, in 0.43 seconds)4813client: must succeed: nix-store --export /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache > /tmp/shared/ref.closure4814client: (finished: must succeed: nix-store --export /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache > /tmp/shared/ref.closure, in 0.20 seconds)4815worker1: must succeed: nix-store --import < /tmp/shared/ref.closure4816worker1: (finished: must succeed: nix-store --import < /tmp/shared/ref.closure, in 0.24 seconds)4817worker2 # [ 78.537489] systemd[1]: Stopping Nix worker-protocol over gRPC...4818worker2 # [ 79.133330] nix-grpc-daemon[778]: ts=2026-09-21T12:09:49Z level=info event=signal_exit4819worker2 # [ 79.159576] nix-grpc-daemon[778]: ts=2026-09-21T12:09:49Z level=info event=scheduler_yield4820worker1 # [ 79.153884] nix-grpc-daemon[777]: ts=2026-09-21T12:09:49Z level=info event=scheduler_restarting addr=lb:500514821lb # [ 79.175816] niks3-server[1060]: 2026/09/21 12:09:49 INFO lead: released remote=192.168.1.4:522204822worker2 # [ 79.165205] nix-grpc-daemon[778]: ts=2026-09-21T12:09:49Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:490524823worker1 # [ 79.158743] nix-grpc-daemon[777]: ts=2026-09-21T12:09:49Z level=info event=scheduler_take_over4824worker2 # [ 79.171373] nix-grpc-daemon[778]: ts=2026-09-21T12:09:49Z level=info event=worker_gone addr=192.168.1.3:500514825worker2 # [ 79.175597] nix-grpc-daemon[778]: ts=2026-09-21T12:09:49Z level=info event=scheduler_restarting addr=lb:500514826worker2 # [ 79.180888] nix-grpc-daemon[778]: ts=2026-09-21T12:09:49Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:490524827worker2 # [ 79.185396] nix-grpc-daemon[778]: ts=2026-09-21T12:09:49Z level=info event=worker_gone addr=192.168.1.4:500514828worker1 # [ 79.353483] nix-grpc-daemon[777]: ts=2026-09-21T12:09:49Z level=info event=scheduler_connected addr=lb:500514829worker1 # [ 79.358731] nix-grpc-daemon[777]: ts=2026-09-21T12:09:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4830worker2 # [ 79.372655] nix-grpc-daemon[778]: ts=2026-09-21T12:09:49Z level=info event=scheduler_connected addr=lb:500514831worker2 # [ 79.377739] nix-grpc-daemon[778]: ts=2026-09-21T12:09:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4832worker1 # [ 79.559482] nix-grpc-daemon[777]: ts=2026-09-21T12:09:50Z level=info event=scheduler_connected addr=lb:500514833worker1 # [ 79.564809] nix-grpc-daemon[777]: ts=2026-09-21T12:09:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4834worker2 # [ 79.578794] nix-grpc-daemon[778]: ts=2026-09-21T12:09:50Z level=info event=scheduler_connected addr=lb:500514835worker2 # [ 79.583873] nix-grpc-daemon[778]: ts=2026-09-21T12:09:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4836lb # [ 79.186595] niks3-server[1060]: 2026/09/21 12:09:49 INFO lead: acquired remote=192.168.1.3:506784837worker1 # [ 79.765581] nix-grpc-daemon[777]: ts=2026-09-21T12:09:50Z level=info event=scheduler_connected addr=lb:500514838worker1 # [ 79.770728] nix-grpc-daemon[777]: ts=2026-09-21T12:09:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4839worker2 # [ 79.784983] nix-grpc-daemon[778]: ts=2026-09-21T12:09:50Z level=info event=scheduler_connected addr=lb:500514840worker2 # [ 79.790065] nix-grpc-daemon[778]: ts=2026-09-21T12:09:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4841worker1 # [ 79.971524] nix-grpc-daemon[777]: ts=2026-09-21T12:09:50Z level=info event=scheduler_connected addr=lb:500514842worker1 # [ 79.976795] nix-grpc-daemon[777]: ts=2026-09-21T12:09:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4843worker2 # [ 79.991142] nix-grpc-daemon[778]: ts=2026-09-21T12:09:50Z level=info event=scheduler_connected addr=lb:500514844worker2 # [ 79.996373] nix-grpc-daemon[778]: ts=2026-09-21T12:09:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4845worker2 # [ 80.169279] nix-grpc-daemon[778]: ts=2026-09-21T12:09:50Z level=info event=scheduler_restarting4846worker1 # [ 80.179466] nix-grpc-daemon[777]: ts=2026-09-21T12:09:50Z level=info event=scheduler_connected addr=lb:500514847worker1 # [ 80.183672] nix-grpc-daemon[777]: ts=2026-09-21T12:09:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4848worker2 # [ 80.224220] nix-grpc-daemon[778]: E0000 00:00:1789992590.756038 823 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 16, started true, shutdown true4849worker2 # [ 80.232356] nix-grpc-daemon[778]: ts=2026-09-21T12:09:50Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"4850worker2 # [ 80.236787] nix-daemon[863]: reaped child process 865, status = succeeded4851worker2 # [ 80.249163] nix-grpc-daemon[778]: E0000 00:00:1789992590.792571 823 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 9, started true, shutdown true4852worker1 # [ 80.380080] nix-grpc-daemon[777]: ts=2026-09-21T12:09:50Z level=info event=scheduler_connected addr=lb:500514853worker1 # [ 80.384786] nix-grpc-daemon[777]: ts=2026-09-21T12:09:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4854worker2 # [ 80.596911] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.4855worker1 # [ 80.585546] nix-grpc-daemon[777]: ts=2026-09-21T12:09:51Z level=info event=scheduler_connected addr=lb:500514856worker2 # [ 80.599885] systemd[1]: Stopped Nix worker-protocol over gRPC.4857worker1 # [ 80.590753] nix-grpc-daemon[777]: ts=2026-09-21T12:09:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4858worker2 # [ 80.612561] systemd[1]: nix-grpc-daemon.service: Consumed 3.563s CPU time over 59.793s wall clock time, 66M memory peak, 95.8K incoming IP traffic, 246.6K outgoing IP traffic.4859worker2 # [ 80.630272] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.4860worker2 # [ 80.635198] systemd[1]: Closed Nix worker-protocol over gRPC.4861lb: waiting until exactly ['worker2'] endpoints of aarch64-linux are unhealthy4862lb: must succeed: curl -sf localhost:9901/clusters4863worker2 # [ 80.639257] systemd[1]: nix-grpc-daemon.socket: Consumed 583us CPU time over 1min 8.939s wall clock time, 256K memory peak, 94.5K incoming IP traffic, 76.3K outgoing IP traffic.4864lb # [ 80.665011] envoy[910]: conn peer=[2001:db8:1::1]:59706 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3740 tx=2015 ms=29194865lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds)4866lb: (finished: waiting until exactly ['worker2'] endpoints of aarch64-linux are unhealthy, in 0.09 seconds)4867client: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache >&24868worker1 # [ 80.791734] nix-grpc-daemon[777]: ts=2026-09-21T12:09:51Z level=info event=scheduler_connected addr=lb:500514869worker1 # [ 80.797206] nix-grpc-daemon[777]: ts=2026-09-21T12:09:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4870worker1 # [ 80.976388] nix-grpc-daemon[777]: ts=2026-09-21T12:09:51Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:401104871client # /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache4872worker1 # [ 80.983840] nix-grpc-daemon[777]: ts=2026-09-21T12:09:51Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:401104873worker1 # [ 80.997921] nix-grpc-daemon[777]: ts=2026-09-21T12:09:51Z level=info event=scheduler_connected addr=lb:500514874worker1 # [ 81.003683] nix-grpc-daemon[777]: ts=2026-09-21T12:09:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4875client: (finished: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache >&2, in 0.29 seconds)4876worker2: must fail: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache4877worker2: (finished: must fail: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache, in 0.03 seconds)4878worker2 # [ 81.125702] systemd[1]: Listening on Nix worker-protocol over gRPC.4879worker1 # [ 81.193541] systemd[1]: Stopping Nix worker-protocol over gRPC...4880worker1 # [ 81.205110] nix-grpc-daemon[777]: ts=2026-09-21T12:09:51Z level=info event=scheduler_connected addr=lb:500514881worker1 # [ 81.220187] nix-grpc-daemon[777]: ts=2026-09-21T12:09:51Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:585384882worker1 # [ 81.224867] nix-grpc-daemon[777]: ts=2026-09-21T12:09:51Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=04883lb # [ 80.670565] envoy[910]: conn peer=[2001:db8:1::1]:59722 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=448 tx=328 ms=734884lb # [ 80.670715] envoy[910]: conn peer=[2001:db8:1::1]:59726 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3758 tx=2029 ms=25364885lb # [ 80.670817] envoy[910]: conn peer=[2001:db8:1::1]:59728 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=448 tx=328 ms=694886lb # [ 80.670930] envoy[910]: conn peer=[2001:db8:1::4]:55698 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=57470 tx=3295 ms=590314887worker2 # [ 81.585147] systemd[1]: Starting scheduler-token.service...4888worker2 # [ 81.682958] systemd[1]: scheduler-token.service: Deactivated successfully.4889worker2 # [ 81.688758] systemd[1]: Finished scheduler-token.service.4890worker2 # [ 81.689971] systemd[1]: scheduler-token.service: Consumed 50ms CPU time over 100ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.4891worker2 # [ 81.695643] systemd[1]: Starting Nix worker-protocol over gRPC...4892worker2 # [ 81.880579] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:52Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"4893worker2 # [ 81.883350] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:52Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc4894worker2 # [ 81.887475] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:52Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=14895worker2 # [ 81.896073] nix-grpc-daemon[1275]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR4896worker2 # [ 81.897544] nix-grpc-daemon[1275]: E0000 00:00:1789992592.437731 1275 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4897worker2 # [ 81.909501] nix-grpc-daemon[1275]: E0000 00:00:1789992592.444169 1275 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4898worker2 # [ 81.916616] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:52Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket4899worker2 # [ 81.919524] systemd[1]: Started Nix worker-protocol over gRPC.4900worker2 # [ 81.923834] nix-grpc-daemon[1275]: E0000 00:00:1789992592.465360 1314 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4901worker2 # [ 81.952228] nix-grpc-daemon[1275]: E0000 00:00:1789992592.483454 1314 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4902worker2 # [ 81.954513] nix-grpc-daemon[1275]: E0000 00:00:1789992592.483684 1314 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4903worker2 # [ 81.961117] nix-grpc-daemon[1275]: E0000 00:00:1789992592.503725 1314 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4904worker2 # [ 81.973612] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:52Z level=info event=scheduler_connected addr=lb:500514905worker1 # [ 81.964576] nix-grpc-daemon[777]: ts=2026-09-21T12:09:52Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:585384906worker1 # [ 81.970397] nix-grpc-daemon[777]: ts=2026-09-21T12:09:52Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=04907worker1 # [ 82.009606] nix-grpc-daemon[777]: ts=2026-09-21T12:09:52Z level=info event=signal_exit4908worker1 # [ 82.247494] nix-grpc-daemon[777]: ts=2026-09-21T12:09:52Z level=info event=scheduler_yield4909worker2 # [ 82.267729] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:52Z level=info event=scheduler_restarting addr=lb:500514910lb # [ 82.278183] niks3-server[1060]: 2026/09/21 12:09:52 INFO lead: released remote=192.168.1.3:506784911worker1 # [ 82.257366] nix-grpc-daemon[777]: ts=2026-09-21T12:09:52Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:585384912worker1 # [ 82.264993] nix-grpc-daemon[777]: ts=2026-09-21T12:09:52Z level=info event=worker_gone addr=192.168.1.4:500514913worker1 # [ 82.269088] nix-grpc-daemon[777]: ts=2026-09-21T12:09:52Z level=info event=scheduler_restarting addr=lb:500514914worker1 # [ 82.273047] nix-grpc-daemon[777]: ts=2026-09-21T12:09:52Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:585384915worker1 # [ 82.277387] nix-grpc-daemon[777]: ts=2026-09-21T12:09:52Z level=info event=worker_gone addr=192.168.1.3:500514916worker2 # [ 82.469013] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:53Z level=info event=scheduler_connected addr=lb:500514917worker1 # [ 82.459476] nix-grpc-daemon[777]: ts=2026-09-21T12:09:53Z level=info event=scheduler_connected addr=lb:500514918worker2 # [ 82.474333] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4919worker1 # [ 82.465149] nix-grpc-daemon[777]: ts=2026-09-21T12:09:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4920worker2 # [ 82.675604] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:53Z level=info event=scheduler_connected addr=lb:500514921worker1 # [ 82.666139] nix-grpc-daemon[777]: ts=2026-09-21T12:09:53Z level=info event=scheduler_connected addr=lb:500514922worker2 # [ 82.681012] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4923worker1 # [ 82.671487] nix-grpc-daemon[777]: ts=2026-09-21T12:09:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4924worker2 # [ 82.883656] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:53Z level=info event=scheduler_connected addr=lb:500514925worker1 # [ 82.872537] nix-grpc-daemon[777]: ts=2026-09-21T12:09:53Z level=info event=scheduler_connected addr=lb:500514926worker2 # [ 82.887928] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4927worker1 # [ 82.878458] nix-grpc-daemon[777]: ts=2026-09-21T12:09:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4928worker2 # [ 83.084690] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:53Z level=info event=scheduler_connected addr=lb:500514929worker1 # [ 83.079096] nix-grpc-daemon[777]: ts=2026-09-21T12:09:53Z level=info event=scheduler_connected addr=lb:500514930worker2 # [ 83.089715] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4931worker1 # [ 83.084333] nix-grpc-daemon[777]: ts=2026-09-21T12:09:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4932worker1 # [ 83.258637] nix-grpc-daemon[777]: ts=2026-09-21T12:09:53Z level=info event=scheduler_restarting4933worker2 # [ 83.290746] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:53Z level=info event=scheduler_connected addr=lb:500514934worker2 # [ 83.295852] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4935worker1 # [ 83.295709] nix-daemon[842]: reaped child process 844, status = succeeded4936worker1 # [ 83.310045] nix-grpc-daemon[777]: ts=2026-09-21T12:09:53Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"4937worker2 # [ 83.496952] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:54Z level=info event=scheduler_connected addr=lb:500514938worker2 # [ 83.501979] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4939worker2 # [ 83.703048] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:54Z level=info event=scheduler_connected addr=lb:500514940worker2 # [ 83.708258] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4941worker2 # [ 83.911053] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:54Z level=info event=scheduler_connected addr=lb:500514942worker2 # [ 83.916607] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4943worker2 # [ 84.112284] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:54Z level=info event=scheduler_connected addr=lb:500514944worker2 # [ 84.117516] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4945worker1 # [ 84.130118] nix-grpc-daemon[777]: E0000 00:00:1789992594.677645 822 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 17, started true, shutdown true4946worker1 # [ 84.130503] nix-grpc-daemon[777]: E0000 00:00:1789992594.677881 822 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true4947worker2 # [ 84.318808] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:54Z level=info event=scheduler_connected addr=lb:500514948worker2 # [ 84.324252] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4949worker1 # [ 84.514645] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.4950worker2 # [ 84.525364] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:55Z level=info event=scheduler_connected addr=lb:500514951worker2 # [ 84.530664] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4952worker1 # [ 84.524559] systemd[1]: Stopped Nix worker-protocol over gRPC.4953worker1 # [ 84.527091] systemd[1]: nix-grpc-daemon.service: Consumed 1.965s CPU time over 1min 3.876s wall clock time, 51.9M memory peak, 59.1K incoming IP traffic, 105.2K outgoing IP traffic.4954worker1 # [ 84.546815] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.4955worker1 # [ 84.549773] systemd[1]: Closed Nix worker-protocol over gRPC.4956worker1 # [ 84.552733] systemd[1]: nix-grpc-daemon.socket: Consumed 593us CPU time over 1min 13.963s wall clock time, 256K memory peak, 66.7K incoming IP traffic, 59.8K outgoing IP traffic.4957lb: waiting until exactly ['worker1'] endpoints of aarch64-linux are unhealthy4958lb: must succeed: curl -sf localhost:9901/clusters4959lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)4960lb: (finished: waiting until exactly ['worker1'] endpoints of aarch64-linux are unhealthy, in 0.09 seconds)4961client: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache >&24962worker2 # [ 84.731725] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:55Z level=info event=scheduler_connected addr=lb:500514963worker2 # [ 84.737214] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4964worker2 # [ 84.921290] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:55Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:485764965worker2 # [ 84.939850] nix-daemon[863]: accepted connection from pid 1275, user nix-grpc-daemon (trusted)4966worker2 # [ 84.952309] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:55Z level=info event=scheduler_connected addr=lb:500514967worker2 # [ 84.957652] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4968worker2 # [ 84.976521] nix-grpc-daemon[1275]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 46 ms (attempt 1/5)4969worker2 # [ 85.022982] nix-grpc-daemon[1275]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 14 ms (attempt 2/5)4970worker2 # [ 85.037158] nix-grpc-daemon[1275]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 85 ms (attempt 3/5)4971worker2 # [ 85.129551] nix-grpc-daemon[1275]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 754 ms (attempt 4/5)4972worker2 # [ 85.140826] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:55Z level=info event=scheduler_connected addr=lb:500514973worker2 # [ 85.145848] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4974worker2 # [ 85.346929] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:55Z level=info event=scheduler_connected addr=lb:500514975worker2 # [ 85.352834] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4976worker2 # [ 85.553936] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:56Z level=info event=scheduler_connected addr=lb:500514977worker2 # [ 85.559108] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4978worker2 # [ 85.760309] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:56Z level=info event=scheduler_connected addr=lb:500514979worker2 # [ 85.765372] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4980worker2 # [ 85.891812] nix-grpc-daemon[1275]: disabling binary cache 'https://cache.nixos.org' for 60 seconds4981worker2 # [ 85.898124] nix-grpc-daemon[1275]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4982client # copying 2 paths...4983client # copying path '/nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache' to 'grpc://lb:50051'...4984worker2 # [ 85.923442] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:56Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:485764985client # copying path '/nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache' to 'grpc://lb:50051'...4986worker2 # [ 85.942160] nix-daemon[863]: accepted connection from pid 1275, user nix-grpc-daemon (trusted)4987worker2 # [ 85.976209] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:56Z level=info event=niks3_push_started pid=13354988worker2 # [ 85.985666] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:56Z level=info event=scheduler_connected addr=lb:500514989worker2 # [ 85.990173] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4990worker2 # [ 86.187924] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:56Z level=info event=scheduler_connected addr=lb:500514991worker2 # [ 86.190221] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4992worker2 # [ 86.206034] nix-grpc-daemon[1335]: time=2026-09-21T12:09:56.749Z level=INFO msg="Configuring client TLS" cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker2.pem key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker2.key ca=""4993worker2 # [ 86.382957] nix-daemon[863]: accepted connection from pid 1348, user nix-grpc-daemon (trusted)4994worker2 # [ 86.390789] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:56Z level=info event=scheduler_connected addr=lb:500514995worker2 # [ 86.400342] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4996worker2 # [ 86.407635] nix-daemon[863]: reaped child process 1351, status = succeeded4997lb # [ 86.434044] niks3-server[1060]: 2026/09/21 12:09:56 INFO Received uploads request method=POST path=/api/pending_closures4998lb # [ 86.444329] niks3-server[1060]: 2026/09/21 12:09:56 INFO Received uploads request method=POST path=/api/pending_closures4999worker2 # [ 86.444147] nix-grpc-daemon[1335]: time=2026-09-21T12:09:56.987Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"5000worker2 # [ 86.445619] nix-grpc-daemon[1335]: time=2026-09-21T12:09:56.987Z level=INFO msg="Uploading 4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache (184B)"5001worker2 # [ 86.447333] nix-grpc-daemon[1335]: time=2026-09-21T12:09:56.987Z level=INFO msg="Uploading syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache (128B)"5002lb # [ 86.498788] niks3-server[1060]: 2026/09/21 12:09:57 INFO Registered completed upload object_key=nar/1rf1rg73pvnaa9kk6sy9r1ydj24b4g0kb0wnx0z5hxl1nvkmlavq.nar.zst5003lb # [ 86.512166] niks3-server[1060]: 2026/09/21 12:09:57 INFO Registered completed upload object_key=nar/1kmnh5f2l7jjcgaq9b9988bjix6z8zjf6v350xllq5hlzgnb1f75.nar.zst5004lb # [ 86.522203] niks3-server[1060]: 2026/09/21 12:09:57 INFO Registered completed upload object_key=syvivri0wisb7fd33rxcg9kz63kc2bv7.ls5005lb # [ 86.530316] niks3-server[1060]: 2026/09/21 12:09:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/25/sign5006lb # [ 86.537021] niks3-server[1060]: 2026/09/21 12:09:57 INFO Signed narinfos id=25 count=15007worker2 # [ 86.528171] nix-grpc-daemon[1335]: time=2026-09-21T12:09:57.070Z level=INFO msg="Uploading 3 narinfos"5008lb # [ 86.540913] niks3-server[1060]: 2026/09/21 12:09:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/26/sign5009lb # [ 86.548231] niks3-server[1060]: 2026/09/21 12:09:57 INFO Signed narinfos id=26 count=25010lb # [ 86.550454] niks3-server[1060]: 2026/09/21 12:09:57 INFO Registered completed upload object_key=4rig83zwv4xj1zajs08ima13w0yh676v.ls5011lb # [ 86.567917] niks3-server[1060]: 2026/09/21 12:09:57 INFO Registered completed upload object_key=4rig83zwv4xj1zajs08ima13w0yh676v.narinfo5012lb # [ 86.575324] niks3-server[1060]: 2026/09/21 12:09:57 INFO Received complete upload request method=POST path=/api/pending_closures/25/complete5013lb # [ 86.584241] niks3-server[1060]: 2026/09/21 12:09:57 INFO Completed upload id=255014lb # [ 86.589913] niks3-server[1060]: 2026/09/21 12:09:57 INFO Received complete upload request method=POST path=/api/pending_closures/26/complete5015worker2 # [ 86.579860] nix-grpc-daemon[1335]: time=2026-09-21T12:09:57.123Z level=INFO msg="Upload complete. (368ms)"5016lb # [ 86.593533] niks3-server[1060]: 2026/09/21 12:09:57 INFO Registered completed upload object_key=syvivri0wisb7fd33rxcg9kz63kc2bv7.narinfo5017lb # [ 86.603742] niks3-server[1060]: 2026/09/21 12:09:57 INFO Completed upload id=265018worker2 # [ 86.592395] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:57Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48576 duration_s=0 paths=2 nar_bytes_in=3125019worker2 # [ 86.611185] nix-grpc-daemon[1335]: time=2026-09-21T12:09:57.143Z level=WARN msg="Failed to register uploaded object" key=syvivri0wisb7fd33rxcg9kz63kc2bv7.narinfo error="server returned 404: object is not pending upload\n"5020worker2 # [ 86.618770] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:57Z level=info event=scheduler_connected addr=lb:500515021worker2 # [ 86.622897] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5022worker2 # [ 86.627032] nix-daemon[863]: reaped child process 1333, status = succeeded5023client: (finished: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache >&2, in 1.98 seconds)5024worker2: must succeed: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache && test -e /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache5025worker2: (finished: must succeed: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache && test -e /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache, in 0.03 seconds)5026worker1 # [ 86.745144] systemd[1]: Listening on Nix worker-protocol over gRPC.5027worker1 # [ 86.753562] systemd[1]: Starting Nix worker-protocol over gRPC...5028worker2 # [ 86.803419] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:57Z level=info event=scheduler_connected addr=lb:500515029worker2 # [ 86.808938] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5030lb # [ 86.928454] niks3-server[1060]: 2026/09/21 12:09:57 INFO lead: acquired remote=192.168.1.4:409905031worker2 # [ 86.923728] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:57Z level=info event=scheduler_take_over5032worker1 # [ 86.953608] nix-grpc-daemon[1043]: ts=2026-09-21T12:09:57Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5033worker1 # [ 86.955373] nix-grpc-daemon[1043]: ts=2026-09-21T12:09:57Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5034worker1 # [ 86.960779] nix-grpc-daemon[1043]: ts=2026-09-21T12:09:57Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15035worker1 # [ 86.968063] nix-grpc-daemon[1043]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5036worker1 # [ 86.969549] nix-grpc-daemon[1043]: E0000 00:00:1789992597.514766 1043 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5037worker1 # [ 86.980698] nix-grpc-daemon[1043]: E0000 00:00:1789992597.521305 1043 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5038worker1 # [ 86.984529] nix-grpc-daemon[1043]: ts=2026-09-21T12:09:57Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5039worker1 # [ 86.987226] systemd[1]: Started Nix worker-protocol over gRPC.5040lb: waiting until exactly no endpoints of aarch64-linux are unhealthy5041lb: must succeed: curl -sf localhost:9901/clusters5042worker1 # [ 86.992107] nix-grpc-daemon[1043]: E0000 00:00:1789992597.538162 1082 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5043worker2 # [ 87.009971] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:57Z level=info event=scheduler_connected addr=lb:500515044worker1 # [ 87.020134] nix-grpc-daemon[1043]: E0000 00:00:1789992597.555978 1082 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5045worker2 # [ 87.031605] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:57Z level=info event=oidc_jwks provider=mock keys=15046worker1 # [ 87.022539] nix-grpc-daemon[1043]: E0000 00:00:1789992597.556176 1082 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5047worker2 # [ 87.036426] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:57Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:485845048worker1 # [ 87.029346] nix-grpc-daemon[1043]: E0000 00:00:1789992597.575668 1082 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5049worker2 # [ 87.041879] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:57Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=05050worker1 # [ 87.046058] nix-grpc-daemon[1043]: ts=2026-09-21T12:09:57Z level=info event=scheduler_connected addr=lb:500515051worker2 # [ 87.058454] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:57Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:485845052worker2 # [ 87.064186] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:57Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=05053lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.10 seconds)5054lb: must succeed: curl -sf localhost:9901/clusters5055lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5056lb: (finished: waiting until exactly no endpoints of aarch64-linux are unhealthy, in 1.18 seconds)5057(finished: subtest: upload whose reference the worker lacks is completed from the cache, in 11.04 seconds)5058subtest: scheduler down: the other node takes the lock and keeps it5059lb: must succeed: curl -sf localhost:9901/clusters5060lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)5061lb: must succeed: curl -sf localhost:9901/clusters5062lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5063lb: must succeed: curl -sf localhost:9901/clusters5064lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5065lb: must succeed: curl -sf localhost:9901/clusters5066lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5067worker2 # [ 88.458072] systemd[1]: Stopping Nix worker-protocol over gRPC...5068worker2 # [ 88.909524] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:59Z level=info event=signal_exit5069worker2 # [ 89.129538] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:59Z level=info event=scheduler_yield5070lb # [ 89.147727] niks3-server[1060]: 2026/09/21 12:09:59 INFO lead: released remote=192.168.1.4:409905071worker1 # [ 89.129689] nix-grpc-daemon[1043]: ts=2026-09-21T12:09:59Z level=info event=scheduler_restarting addr=lb:500515072worker2 # [ 89.143879] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:59Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:485845073worker2 # [ 89.151103] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:59Z level=info event=worker_gone addr=192.168.1.3:500515074worker2 # [ 89.155643] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:59Z level=info event=scheduler_restarting addr=lb:500515075worker2 # [ 89.159761] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:59Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:485845076worker2 # [ 89.164538] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:59Z level=info event=worker_gone addr=192.168.1.4:500515077worker1 # [ 89.329751] nix-grpc-daemon[1043]: ts=2026-09-21T12:09:59Z level=info event=scheduler_connected addr=lb:500515078worker1 # [ 89.335106] nix-grpc-daemon[1043]: ts=2026-09-21T12:09:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5079worker2 # [ 89.350812] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:59Z level=info event=scheduler_connected addr=lb:500515080worker2 # [ 89.355991] nix-grpc-daemon[1275]: ts=2026-09-21T12:09:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5081worker1 # [ 89.535749] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:00Z level=info event=scheduler_connected addr=lb:500515082worker1 # [ 89.541094] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5083worker2 # [ 89.557221] nix-grpc-daemon[1275]: ts=2026-09-21T12:10:00Z level=info event=scheduler_connected addr=lb:500515084worker2 # [ 89.562271] nix-grpc-daemon[1275]: ts=2026-09-21T12:10:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5085worker1 # [ 89.741787] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:00Z level=info event=scheduler_connected addr=lb:500515086worker1 # [ 89.746897] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5087worker2 # [ 89.763386] nix-grpc-daemon[1275]: ts=2026-09-21T12:10:00Z level=info event=scheduler_connected addr=lb:500515088worker2 # [ 89.768691] nix-grpc-daemon[1275]: ts=2026-09-21T12:10:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5089worker1 # [ 89.947710] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:00Z level=info event=scheduler_connected addr=lb:500515090worker1 # [ 89.953056] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5091worker2 # [ 89.971281] nix-grpc-daemon[1275]: ts=2026-09-21T12:10:00Z level=info event=scheduler_connected addr=lb:500515092worker2 # [ 89.975587] nix-grpc-daemon[1275]: ts=2026-09-21T12:10:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5093worker2 # [ 90.134702] nix-grpc-daemon[1275]: ts=2026-09-21T12:10:00Z level=info event=scheduler_restarting5094worker2 # [ 90.165139] nix-daemon[863]: reaped child process 1327, status = succeeded5095worker1 # [ 90.155566] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:00Z level=info event=scheduler_connected addr=lb:500515096worker1 # [ 90.159872] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5097worker2 # [ 90.190188] nix-grpc-daemon[1275]: ts=2026-09-21T12:10:00Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5098worker1 # [ 90.356245] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:00Z level=info event=scheduler_connected addr=lb:500515099worker1 # [ 90.361329] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5100worker1 # [ 90.562189] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:01Z level=info event=scheduler_connected addr=lb:500515101worker1 # [ 90.567623] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5102lb # [ 90.663963] envoy[910]: conn peer=[2001:db8:1::1]:33196 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=591 tx=522 ms=225103lb # [ 90.664278] envoy[910]: conn peer=[2001:db8:1::3]:49188 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=12377 tx=3293 ms=622385104lb # [ 90.664438] envoy[910]: conn peer=[2001:db8:1::1]:33206 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1006 tx=282 ms=16865105lb # [ 90.664541] envoy[910]: conn peer=[2001:db8:1::4]:42108 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=21717 tx=1152 ms=82085106worker1 # [ 90.768376] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:01Z level=info event=scheduler_connected addr=lb:500515107worker1 # [ 90.773500] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5108worker2 # [ 90.940409] nix-grpc-daemon[1275]: E0000 00:00:1789992601.483718 1320 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 17, started true, shutdown true5109worker2 # [ 90.940816] nix-grpc-daemon[1275]: E0000 00:00:1789992601.483989 1320 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5110worker1 # [ 90.974373] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:01Z level=info event=scheduler_connected addr=lb:500515111worker1 # [ 90.980241] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5112worker1 # [ 91.182879] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:01Z level=info event=scheduler_connected addr=lb:500515113worker1 # [ 91.187250] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5114worker1 # [ 91.383455] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:01Z level=info event=scheduler_connected addr=lb:500515115worker1 # [ 91.388704] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5116worker1 # [ 91.589372] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:02Z level=info event=scheduler_connected addr=lb:500515117worker1 # [ 91.594398] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5118worker1 # [ 91.795196] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:02Z level=info event=scheduler_connected addr=lb:500515119worker1 # [ 91.800473] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5120worker2 # [ 91.935299] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5121worker2 # [ 91.940184] systemd[1]: Stopped Nix worker-protocol over gRPC.5122worker2 # [ 91.948213] systemd[1]: nix-grpc-daemon.service: Consumed 529ms CPU time over 10.243s wall clock time, 33.4M memory peak, 23.8K incoming IP traffic, 48.7K outgoing IP traffic.5123worker2 # [ 91.958920] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5124worker2 # [ 91.965377] systemd[1]: Closed Nix worker-protocol over gRPC.5125worker2 # [ 91.967115] systemd[1]: nix-grpc-daemon.socket: Consumed 732us CPU time over 10.850s wall clock time, 256K memory peak, 20.6K incoming IP traffic, 15.4K outgoing IP traffic.5126worker1: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over5127lb # [ 92.005305] niks3-server[1060]: 2026/09/21 12:10:02 INFO lead: acquired remote=192.168.1.3:528025128worker1 # [ 91.983485] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:02Z level=info event=scheduler_take_over5129worker1 # [ 92.001228] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:02Z level=info event=scheduler_connected addr=lb:500515130worker1 # [ 92.008146] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5131worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over, in 0.09 seconds)5132worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5133worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)5134worker1 # [ 92.210192] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:02Z level=info event=scheduler_connected addr=lb:500515135worker1 # [ 92.214370] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5136worker1 # [ 92.411087] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:02Z level=info event=scheduler_connected addr=lb:500515137worker1 # [ 92.416511] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5138worker1 # [ 92.617124] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:03Z level=info event=scheduler_connected addr=lb:500515139worker1 # [ 92.625948] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:03Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:434425140worker1 # [ 92.631306] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:03Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=05141worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5142worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)5143client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag on-standby >&25144worker1 # [ 93.453237] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:03Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:434485145worker1 # [ 93.467808] nix-daemon[842]: accepted connection from pid 1043, user nix-grpc-daemon (trusted)5146client # don't know how to build these paths:5147client # /nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv5148worker1 # [ 93.488393] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:04Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:434485149worker1 # [ 93.507501] nix-grpc-daemon[1043]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 99 ms (attempt 1/5)5150worker1 # [ 93.614181] nix-grpc-daemon[1043]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 137 ms (attempt 2/5)5151worker1 # [ 93.758849] nix-grpc-daemon[1043]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 241 ms (attempt 3/5)5152worker1 # [ 94.007474] nix-grpc-daemon[1043]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 385 ms (attempt 4/5)5153worker1 # [ 94.393214] nix-grpc-daemon[1043]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5154worker1 # [ 94.398967] nix-grpc-daemon[1043]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5155worker1 # [ 94.433795] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:04Z level=debug event=want client=1 drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv cached=05156worker1 # [ 94.440891] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:04Z level=debug event=assigned drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv worker=192.168.1.3:50051 clients=15157worker1 # [ 94.482139] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:05Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:434485158worker1 # [ 94.496253] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:05Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:434485159client # copying 1 paths to 'lb:50051'...5160client # copying path '/nix/store/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv' to 'lb:50051'...5161worker1 # [ 94.510558] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:05Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:434485162worker1 # [ 94.526031] nix-daemon[842]: accepted connection from pid 1043, user nix-grpc-daemon (trusted)5163worker1 # [ 94.555334] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:05Z level=info event=niks3_push_started pid=11175164worker1 # [ 94.769504] nix-grpc-daemon[1117]: time=2026-09-21T12:10:05.316Z level=INFO msg="Configuring client TLS" cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker1.pem key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker1.key ca=""5165worker1 # [ 94.943947] nix-daemon[842]: accepted connection from pid 1130, user nix-grpc-daemon (trusted)5166worker1 # [ 94.959704] nix-daemon[842]: reaped child process 1133, status = succeeded5167worker1 # [ 95.137464] nix-daemon[842]: accepted connection from pid 1135, user nix-grpc-daemon (trusted)5168worker1 # [ 95.153500] nix-daemon[842]: reaped child process 1138, status = succeeded5169lb # [ 95.189562] niks3-server[1060]: 2026/09/21 12:10:05 INFO Received uploads request method=POST path=/api/pending_closures5170worker1 # [ 95.182691] nix-grpc-daemon[1117]: time=2026-09-21T12:10:05.729Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5171worker1 # [ 95.184577] nix-grpc-daemon[1117]: time=2026-09-21T12:10:05.729Z level=INFO msg="Uploading adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv (520B)"5172lb # [ 95.243002] niks3-server[1060]: 2026/09/21 12:10:05 INFO Registered completed upload object_key=nar/0scd0jwin78ibwq92wkajw2zj7adk8bih7yhx8bxz1706hh53507.nar.zst5173worker1 # [ 95.231034] nix-grpc-daemon[1117]: time=2026-09-21T12:10:05.778Z level=INFO msg="Uploading 1 narinfos"5174lb # [ 95.258443] niks3-server[1060]: 2026/09/21 12:10:05 INFO Received sign narinfos request method=POST path=/api/pending_closures/27/sign5175lb # [ 95.268636] niks3-server[1060]: 2026/09/21 12:10:05 INFO Signed narinfos id=27 count=15176lb # [ 95.273279] niks3-server[1060]: 2026/09/21 12:10:05 INFO Registered completed upload object_key=adj3g5w8jh83wl959p7daq6igl0gaay3.ls5177worker1 # [ 95.259800] nix-grpc-daemon[1117]: time=2026-09-21T12:10:05.807Z level=INFO msg="Upload complete. (486ms)"5178lb # [ 95.282537] niks3-server[1060]: 2026/09/21 12:10:05 INFO Received complete upload request method=POST path=/api/pending_closures/27/complete5179lb # [ 95.288868] niks3-server[1060]: 2026/09/21 12:10:05 INFO Completed upload id=275180worker1 # [ 95.269376] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:05Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43448 duration_s=0 paths=1 nar_bytes_in=5205181lb # [ 95.292733] niks3-server[1060]: 2026/09/21 12:10:05 INFO Registered completed upload object_key=adj3g5w8jh83wl959p7daq6igl0gaay3.narinfo5182worker1 # [ 95.284262] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:05Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:434485183worker1 # [ 95.291199] nix-daemon[842]: accepted connection from pid 1043, user nix-grpc-daemon (trusted)5184worker1 # [ 95.296945] nix-daemon[842]: reaped child process 1115, status = succeeded5185client # building '/nix/store/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv' on lb:50051...5186client # farm-a> node-a: building adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv5187worker1 # [ 95.311195] nix-daemon[842]: accepted connection from pid 1043, user nix-grpc-daemon (trusted)5188client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds5189client # farm-a> LOG-farm-a-on-standby5190worker1 # [ 96.353816] nix-daemon[842]: reaped child process 1142, status = succeeded5191worker1 # [ 96.530785] nix-daemon[842]: accepted connection from pid 1158, user nix-grpc-daemon (trusted)5192worker1 # [ 96.547464] nix-daemon[842]: reaped child process 1161, status = succeeded5193lb # [ 96.586425] niks3-server[1060]: 2026/09/21 12:10:07 INFO Received uploads request method=POST path=/api/pending_closures5194worker1 # [ 96.576387] nix-grpc-daemon[1117]: time=2026-09-21T12:10:07.121Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5195worker1 # [ 96.577858] nix-grpc-daemon[1117]: time=2026-09-21T12:10:07.121Z level=INFO msg="Uploading 7qyaw17jjsjxkkgb1kfm3wxvczk56aan-farm-a (136B)"5196lb # [ 96.620113] niks3-server[1060]: 2026/09/21 12:10:07 INFO Registered completed upload object_key=log/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv5197lb # [ 96.633576] niks3-server[1060]: 2026/09/21 12:10:07 INFO Registered completed upload object_key=nar/0x76wzgdd1nk8rm930z8l69v9961jpc8qhzmgganv0nhagq8lv9c.nar.zst5198lb # [ 96.648302] niks3-server[1060]: 2026/09/21 12:10:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/28/sign5199worker1 # [ 96.635371] nix-grpc-daemon[1117]: time=2026-09-21T12:10:07.181Z level=INFO msg="Uploading 1 narinfos"5200lb # [ 96.660574] niks3-server[1060]: 2026/09/21 12:10:07 INFO Signed narinfos id=28 count=15201lb # [ 96.663706] niks3-server[1060]: 2026/09/21 12:10:07 INFO Registered completed upload object_key=7qyaw17jjsjxkkgb1kfm3wxvczk56aan.ls5202lb # [ 96.676193] niks3-server[1060]: 2026/09/21 12:10:07 INFO Received complete upload request method=POST path=/api/pending_closures/28/complete5203lb # [ 96.686626] niks3-server[1060]: 2026/09/21 12:10:07 INFO Completed upload id=285204worker1 # [ 96.665394] nix-grpc-daemon[1117]: time=2026-09-21T12:10:07.213Z level=INFO msg="Upload complete. (343ms)"5205lb # [ 96.689707] niks3-server[1060]: 2026/09/21 12:10:07 INFO Registered completed upload object_key=7qyaw17jjsjxkkgb1kfm3wxvczk56aan.narinfo5206worker1 # [ 96.692267] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:07Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43448 duration_s=1 drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv assign_id=1 outputs=15207worker1 # [ 96.703312] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:07Z level=debug event=want client=1 drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv cached=05208worker1 # [ 96.707211] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:07Z level=debug event=assigned drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv worker=192.168.1.3:50051 clients=15209worker1 # [ 96.711161] nix-daemon[842]: reaped child process 1140, status = succeeded5210worker1 # [ 96.730975] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:434485211worker1 # [ 96.745939] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:07Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:434485212worker1 # [ 96.760059] nix-grpc-daemon[1117]: time=2026-09-21T12:10:07.307Z level=INFO msg="All 1 paths already cached"5213client # copying 1 paths to 'lb:50051'...5214client # copying path '/nix/store/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv' to 'lb:50051'...5215worker1 # [ 96.770003] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:07Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:434485216worker1 # [ 96.783833] nix-daemon[842]: accepted connection from pid 1043, user nix-grpc-daemon (trusted)5217worker1 # [ 96.988957] nix-daemon[842]: accepted connection from pid 1167, user nix-grpc-daemon (trusted)5218worker1 # [ 97.005445] nix-daemon[842]: reaped child process 1170, status = succeeded5219worker1 # [ 97.180604] nix-daemon[842]: accepted connection from pid 1172, user nix-grpc-daemon (trusted)5220worker1 # [ 97.196497] nix-daemon[842]: reaped child process 1175, status = succeeded5221lb # [ 97.232397] niks3-server[1060]: 2026/09/21 12:10:07 INFO Received uploads request method=POST path=/api/pending_closures5222worker1 # [ 97.220060] nix-grpc-daemon[1117]: time=2026-09-21T12:10:07.767Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5223worker1 # [ 97.221527] nix-grpc-daemon[1117]: time=2026-09-21T12:10:07.767Z level=INFO msg="Uploading s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv (640B)"5224lb # [ 97.263572] niks3-server[1060]: 2026/09/21 12:10:07 INFO Registered completed upload object_key=nar/0g5yl4v9pymang94vyp5ib5wi5jwf0bpp184y8qjavkpag12wmlv.nar.zst5225lb # [ 97.277332] niks3-server[1060]: 2026/09/21 12:10:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/29/sign5226worker1 # [ 97.262789] nix-grpc-daemon[1117]: time=2026-09-21T12:10:07.810Z level=INFO msg="Uploading 1 narinfos"5227lb # [ 97.290954] niks3-server[1060]: 2026/09/21 12:10:07 INFO Signed narinfos id=29 count=15228lb # [ 97.297978] niks3-server[1060]: 2026/09/21 12:10:07 INFO Registered completed upload object_key=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g.ls5229lb # [ 97.307534] niks3-server[1060]: 2026/09/21 12:10:07 INFO Received complete upload request method=POST path=/api/pending_closures/29/complete5230worker1 # [ 97.291265] nix-grpc-daemon[1117]: time=2026-09-21T12:10:07.839Z level=INFO msg="Upload complete. (491ms)"5231lb # [ 97.313532] niks3-server[1060]: 2026/09/21 12:10:07 INFO Completed upload id=295232lb # [ 97.318673] niks3-server[1060]: 2026/09/21 12:10:07 INFO Registered completed upload object_key=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g.narinfo5233worker1 # [ 97.300300] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:07Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43448 duration_s=0 paths=1 nar_bytes_in=6405234worker1 # [ 97.313303] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:434485235worker1 # [ 97.320271] nix-daemon[842]: accepted connection from pid 1043, user nix-grpc-daemon (trusted)5236worker1 # [ 97.326038] nix-daemon[842]: reaped child process 1165, status = succeeded5237client # building '/nix/store/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv' on lb:50051...5238client # farm-b> node-a: building s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv5239worker1 # [ 97.340326] nix-daemon[842]: accepted connection from pid 1043, user nix-grpc-daemon (trusted)5240client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds5241client # farm-b> LOG-farm-b-on-standby5242worker1 # [ 98.387203] nix-daemon[842]: reaped child process 1179, status = succeeded5243worker1 # [ 98.571180] nix-daemon[842]: accepted connection from pid 1195, user nix-grpc-daemon (trusted)5244worker1 # [ 98.587862] nix-daemon[842]: reaped child process 1198, status = succeeded5245lb # [ 98.623908] niks3-server[1060]: 2026/09/21 12:10:09 INFO Received uploads request method=POST path=/api/pending_closures5246worker1 # [ 98.615124] nix-grpc-daemon[1117]: time=2026-09-21T12:10:09.162Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5247worker1 # [ 98.616828] nix-grpc-daemon[1117]: time=2026-09-21T12:10:09.162Z level=INFO msg="Uploading 6hsxxv65wr2jz8cgv2v66iq16niq1gsv-farm-b (184B)"5248lb # [ 98.667877] niks3-server[1060]: 2026/09/21 12:10:09 INFO Registered completed upload object_key=nar/1grn64wcl2mc78k1jvrz5mcrk1ld70lhf8b9jpiqy8mvs6gfzq0v.nar.zst5249lb # [ 98.678696] niks3-server[1060]: 2026/09/21 12:10:09 INFO Registered completed upload object_key=log/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv5250lb # [ 98.684444] niks3-server[1060]: 2026/09/21 12:10:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/30/sign5251worker1 # [ 98.667490] nix-grpc-daemon[1117]: time=2026-09-21T12:10:09.214Z level=INFO msg="Uploading 1 narinfos"5252lb # [ 98.692644] niks3-server[1060]: 2026/09/21 12:10:09 INFO Signed narinfos id=30 count=15253lb # [ 98.697595] niks3-server[1060]: 2026/09/21 12:10:09 INFO Registered completed upload object_key=6hsxxv65wr2jz8cgv2v66iq16niq1gsv.ls5254lb # [ 98.706342] niks3-server[1060]: 2026/09/21 12:10:09 INFO Received complete upload request method=POST path=/api/pending_closures/30/complete5255lb # [ 98.711489] niks3-server[1060]: 2026/09/21 12:10:09 INFO Completed upload id=305256worker1 # [ 98.690212] nix-grpc-daemon[1117]: time=2026-09-21T12:10:09.237Z level=INFO msg="Upload complete. (339ms)"5257lb # [ 98.714222] niks3-server[1060]: 2026/09/21 12:10:09 INFO Registered completed upload object_key=6hsxxv65wr2jz8cgv2v66iq16niq1gsv.narinfo5258worker1 # [ 98.708978] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:09Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43448 duration_s=1 drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv assign_id=2 outputs=15259worker1 # [ 98.723731] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:09Z level=debug event=want client=1 drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv cached=05260worker1 # [ 98.727654] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:09Z level=debug event=assigned drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv worker=192.168.1.3:50051 clients=15261worker1 # [ 98.731648] nix-daemon[842]: reaped child process 1177, status = succeeded5262worker1 # [ 98.750762] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:434485263worker1 # [ 98.766947] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:09Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:434485264worker1 # [ 98.783024] nix-grpc-daemon[1117]: time=2026-09-21T12:10:09.330Z level=INFO msg="All 2 paths already cached"5265client # copying 1 paths to 'lb:50051'...5266client # copying path '/nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv' to 'lb:50051'...5267worker1 # [ 98.794253] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:09Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:434485268worker1 # [ 98.808150] nix-daemon[842]: accepted connection from pid 1043, user nix-grpc-daemon (trusted)5269worker1 # [ 99.010725] nix-daemon[842]: accepted connection from pid 1204, user nix-grpc-daemon (trusted)5270worker1 # [ 99.027192] nix-daemon[842]: reaped child process 1207, status = succeeded5271worker1 # [ 99.204676] nix-daemon[842]: accepted connection from pid 1209, user nix-grpc-daemon (trusted)5272worker1 # [ 99.220394] nix-daemon[842]: reaped child process 1212, status = succeeded5273lb # [ 99.256557] niks3-server[1060]: 2026/09/21 12:10:09 INFO Received uploads request method=POST path=/api/pending_closures5274worker1 # [ 99.244779] nix-grpc-daemon[1117]: time=2026-09-21T12:10:09.792Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5275worker1 # [ 99.246259] nix-grpc-daemon[1117]: time=2026-09-21T12:10:09.792Z level=INFO msg="Uploading 7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv (768B)"5276lb # [ 99.283281] niks3-server[1060]: 2026/09/21 12:10:09 INFO Registered completed upload object_key=nar/1zyx46mv29cfygdhc5ai40qr4vl88lkr1lilm0h72j23w6wakgmc.nar.zst5277lb # [ 99.296786] niks3-server[1060]: 2026/09/21 12:10:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/31/sign5278worker1 # [ 99.281892] nix-grpc-daemon[1117]: time=2026-09-21T12:10:09.829Z level=INFO msg="Uploading 1 narinfos"5279lb # [ 99.307232] niks3-server[1060]: 2026/09/21 12:10:09 INFO Signed narinfos id=31 count=15280lb # [ 99.313678] niks3-server[1060]: 2026/09/21 12:10:09 INFO Registered completed upload object_key=7s5szig4ffga53df6rls98r9lhxsrsc5.ls5281lb # [ 99.320098] niks3-server[1060]: 2026/09/21 12:10:09 INFO Received complete upload request method=POST path=/api/pending_closures/31/complete5282lb # [ 99.328366] niks3-server[1060]: 2026/09/21 12:10:09 INFO Completed upload id=315283worker1 # [ 99.306943] nix-grpc-daemon[1117]: time=2026-09-21T12:10:09.854Z level=INFO msg="Upload complete. (482ms)"5284lb # [ 99.330923] niks3-server[1060]: 2026/09/21 12:10:09 INFO Registered completed upload object_key=7s5szig4ffga53df6rls98r9lhxsrsc5.narinfo5285worker1 # [ 99.316230] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:09Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43448 duration_s=0 paths=1 nar_bytes_in=7685286worker1 # [ 99.325610] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:434485287worker1 # [ 99.332003] nix-daemon[842]: accepted connection from pid 1043, user nix-grpc-daemon (trusted)5288worker1 # [ 99.336539] nix-daemon[842]: reaped child process 1202, status = succeeded5289client # building '/nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv' on lb:50051...5290worker1 # [ 99.348969] nix-daemon[842]: accepted connection from pid 1043, user nix-grpc-daemon (trusted)5291client # farm-top> node-a: building 7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv5292client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds5293client # farm-top> LOG-farm-top-on-standby5294worker1 # [ 100.581949] nix-daemon[842]: reaped child process 1216, status = succeeded5295worker1 # [ 100.758945] nix-daemon[842]: accepted connection from pid 1232, user nix-grpc-daemon (trusted)5296worker1 # [ 100.777764] nix-daemon[842]: reaped child process 1235, status = succeeded5297lb # [ 100.814257] niks3-server[1060]: 2026/09/21 12:10:11 INFO Received uploads request method=POST path=/api/pending_closures5298worker1 # [ 100.804385] nix-grpc-daemon[1117]: time=2026-09-21T12:10:11.351Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5299worker1 # [ 100.807055] nix-grpc-daemon[1117]: time=2026-09-21T12:10:11.351Z level=INFO msg="Uploading 1ini8mfd1rzqyp8jgqihp4k889588mpx-farm-top (240B)"5300lb # [ 100.850079] niks3-server[1060]: 2026/09/21 12:10:11 INFO Registered completed upload object_key=log/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv5301lb # [ 100.863285] niks3-server[1060]: 2026/09/21 12:10:11 INFO Registered completed upload object_key=nar/1mncazpqn39nifc2fwbx80la5i264r8xc1lj8fnlbx006znc1925.nar.zst5302lb # [ 100.876865] niks3-server[1060]: 2026/09/21 12:10:11 INFO Received sign narinfos request method=POST path=/api/pending_closures/32/sign5303worker1 # [ 100.860391] nix-grpc-daemon[1117]: time=2026-09-21T12:10:11.407Z level=INFO msg="Uploading 1 narinfos"5304lb # [ 100.887135] niks3-server[1060]: 2026/09/21 12:10:11 INFO Signed narinfos id=32 count=15305lb # [ 100.893288] niks3-server[1060]: 2026/09/21 12:10:11 INFO Registered completed upload object_key=1ini8mfd1rzqyp8jgqihp4k889588mpx.ls5306lb # [ 100.900807] niks3-server[1060]: 2026/09/21 12:10:11 INFO Received complete upload request method=POST path=/api/pending_closures/32/complete5307lb # [ 100.907572] niks3-server[1060]: 2026/09/21 12:10:11 INFO Registered completed upload object_key=1ini8mfd1rzqyp8jgqihp4k889588mpx.narinfo5308lb # [ 100.912363] niks3-server[1060]: 2026/09/21 12:10:11 INFO Completed upload id=325309worker1 # [ 100.890727] nix-grpc-daemon[1117]: time=2026-09-21T12:10:11.438Z level=INFO msg="Upload complete. (332ms)"5310worker1 # [ 100.905171] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:11Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43448 duration_s=1 drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv assign_id=3 outputs=15311worker1 # [ 100.921055] nix-daemon[842]: reaped child process 1214, status = succeeded5312client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag on-standby >&2, in 7.77 seconds)5313client: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag on-standby outPath5314client: (finished: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag on-standby outPath, in 0.23 seconds)5315worker2 # [ 101.287828] systemd[1]: Listening on Nix worker-protocol over gRPC.5316worker2 # [ 101.297401] systemd[1]: Starting scheduler-token.service...5317worker2 # [ 101.396158] systemd[1]: scheduler-token.service: Deactivated successfully.5318worker2 # [ 101.400631] systemd[1]: Finished scheduler-token.service.5319worker2 # [ 101.401731] systemd[1]: scheduler-token.service: Consumed 39ms CPU time over 100ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5320worker2 # [ 101.407559] systemd[1]: Starting Nix worker-protocol over gRPC...5321worker2 # [ 101.583616] nix-grpc-daemon[1375]: ts=2026-09-21T12:10:12Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5322worker2 # [ 101.586163] nix-grpc-daemon[1375]: ts=2026-09-21T12:10:12Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5323worker2 # [ 101.590217] nix-grpc-daemon[1375]: ts=2026-09-21T12:10:12Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15324worker2 # [ 101.597565] nix-grpc-daemon[1375]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5325worker2 # [ 101.598999] nix-grpc-daemon[1375]: E0000 00:00:1789992612.140365 1375 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5326worker2 # [ 101.609691] nix-grpc-daemon[1375]: E0000 00:00:1789992612.146437 1375 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5327worker2 # [ 101.614267] nix-grpc-daemon[1375]: ts=2026-09-21T12:10:12Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5328worker2 # [ 101.616422] systemd[1]: Started Nix worker-protocol over gRPC.5329worker2 # [ 101.628113] nix-grpc-daemon[1375]: E0000 00:00:1789992612.162304 1415 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5330lb: must succeed: curl -sf localhost:9901/clusters5331worker2 # [ 101.648069] nix-grpc-daemon[1375]: E0000 00:00:1789992612.180320 1415 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5332worker2 # [ 101.650328] nix-grpc-daemon[1375]: E0000 00:00:1789992612.180471 1415 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5333worker2 # [ 101.656959] nix-grpc-daemon[1375]: E0000 00:00:1789992612.199660 1415 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5334worker2 # [ 101.675436] nix-grpc-daemon[1375]: ts=2026-09-21T12:10:12Z level=info event=scheduler_connected addr=lb:500515335worker1 # [ 101.674639] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:12Z level=info event=oidc_jwks provider=mock keys=15336worker1 # [ 101.675318] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:12Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:434425337worker1 # [ 101.684661] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:12Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=05338lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds)5339worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5340worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5341lb: must succeed: curl -sf localhost:9901/clusters5342lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5343lb: must succeed: curl -sf localhost:9901/clusters5344lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)5345worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5346worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.09 seconds)5347(finished: subtest: scheduler down: the other node takes the lock and keeps it, in 13.84 seconds)5348subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise5349client: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag rs5350client # [ 102.104490] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key" --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag rs.5351client # Running as unit: rs.service; invocation ID: 9366774bcea947b3a1329ea4631573655352client: (finished: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag rs, in 0.09 seconds)5353client # [ 102.363818] nix[1174]: don't know how to build these paths:5354client # [ 102.364931] nix[1174]: /nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv5355worker1 # [ 102.336385] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:12Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:434485356worker1 # [ 102.350463] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:12Z level=debug event=want client=2 drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv cached=05357worker1 # [ 102.357621] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:12Z level=debug event=assigned drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv worker=192.168.1.3:50051 clients=15358worker1 # [ 102.394499] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:12Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:434485359worker1 # [ 102.409843] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:12Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:434485360client # [ 102.449127] nix[1174]: copying 1 paths to 'lb:50051'...5361client # [ 102.454891] nix[1174]: copying path '/nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv' to 'lb:50051'...5362worker1 # [ 102.427023] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:12Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:434485363worker1 # [ 102.442908] nix-daemon[842]: accepted connection from pid 1043, user nix-grpc-daemon (trusted)5364worker1 # [ 102.644503] nix-daemon[842]: accepted connection from pid 1248, user nix-grpc-daemon (trusted)5365worker1 # [ 102.663762] nix-daemon[842]: reaped child process 1251, status = succeeded5366worker1 # [ 102.837381] nix-daemon[842]: accepted connection from pid 1253, user nix-grpc-daemon (trusted)5367worker1 # [ 102.855659] nix-daemon[842]: reaped child process 1256, status = succeeded5368lb # [ 102.889673] niks3-server[1060]: 2026/09/21 12:10:13 INFO Received uploads request method=POST path=/api/pending_closures5369worker1 # [ 102.880004] nix-grpc-daemon[1117]: time=2026-09-21T12:10:13.427Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5370worker1 # [ 102.881683] nix-grpc-daemon[1117]: time=2026-09-21T12:10:13.427Z level=INFO msg="Uploading ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv (416B)"5371lb # [ 102.917050] niks3-server[1060]: 2026/09/21 12:10:13 INFO Registered completed upload object_key=nar/0lynldv3n40qnhi1516wgiry3i3pja3sanwyyd8q4nslr915d7r1.nar.zst5372lb # [ 102.932208] niks3-server[1060]: 2026/09/21 12:10:13 INFO Received sign narinfos request method=POST path=/api/pending_closures/33/sign5373worker1 # [ 102.916985] nix-grpc-daemon[1117]: time=2026-09-21T12:10:13.464Z level=INFO msg="Uploading 1 narinfos"5374lb # [ 102.944284] niks3-server[1060]: 2026/09/21 12:10:13 INFO Signed narinfos id=33 count=15375lb # [ 102.947394] niks3-server[1060]: 2026/09/21 12:10:13 INFO Registered completed upload object_key=ajji8m1zd5q5wi64gp7vclb5dd8hr19m.ls5376lb # [ 102.956193] niks3-server[1060]: 2026/09/21 12:10:13 INFO Received complete upload request method=POST path=/api/pending_closures/33/complete5377lb # [ 102.966667] niks3-server[1060]: 2026/09/21 12:10:13 INFO Completed upload id=335378worker1 # [ 102.945111] nix-grpc-daemon[1117]: time=2026-09-21T12:10:13.492Z level=INFO msg="Upload complete. (484ms)"5379lb # [ 102.969223] niks3-server[1060]: 2026/09/21 12:10:13 INFO Registered completed upload object_key=ajji8m1zd5q5wi64gp7vclb5dd8hr19m.narinfo5380worker1 # [ 102.952324] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:13Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43448 duration_s=0 paths=1 nar_bytes_in=4165381worker1 # [ 102.966898] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:13Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:434485382worker1 # [ 102.974450] nix-daemon[842]: accepted connection from pid 1043, user nix-grpc-daemon (trusted)5383worker1 # [ 102.979785] nix-daemon[842]: reaped child process 1246, status = succeeded5384client # [ 103.017007] nix[1174]: building '/nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv' on lb:50051...5385worker1 # [ 102.990939] nix-daemon[842]: accepted connection from pid 1043, user nix-grpc-daemon (trusted)5386lb: must succeed: curl -sf localhost:9901/clusters5387lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds)5388lb: must succeed: curl -sf localhost:9901/clusters5389lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)5390lb: must succeed: curl -sf localhost:9901/clusters5391lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5392lb: must succeed: curl -sf localhost:9901/clusters5393lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5394worker2: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true5395worker2 # [ 105.072681] systemd-journald[274]: Received client request to rotate journal, rotating.5396worker2 # [ 105.102590] systemd-journald[274]: Vacuuming done, freed 0B of archived journals from /var/log/journal/fe7923f42f4b45a39a2a9d2600ab6e51.5397worker2: (finished: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true, in 0.11 seconds)5398worker1: must succeed: systemctl restart --no-block nix-grpc-daemon.service5399worker1 # [ 105.154687] systemd[1]: Stopping Nix worker-protocol over gRPC...5400worker1: (finished: must succeed: systemctl restart --no-block nix-grpc-daemon.service, in 0.07 seconds)5401client: must succeed: sleep 5.05402worker1 # [ 105.987675] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:16Z level=info event=draining5403client: (finished: must succeed: sleep 5.0, in 5.08 seconds)5404worker2: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr='5405worker1 # [ 110.647115] nix-daemon[842]: reaped child process 1260, status = succeeded5406lb # [ 110.669007] envoy[910]: conn peer=[2001:db8:1::1]:47850 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9597 tx=4149 ms=74655407worker1 # [ 110.829148] nix-daemon[842]: accepted connection from pid 1301, user nix-grpc-daemon (trusted)5408worker1 # [ 110.846293] nix-daemon[842]: reaped child process 1304, status = succeeded5409lb # [ 110.880811] niks3-server[1060]: 2026/09/21 12:10:21 INFO Received uploads request method=POST path=/api/pending_closures5410worker1 # [ 110.872397] nix-grpc-daemon[1117]: time=2026-09-21T12:10:21.417Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5411worker1 # [ 110.874245] nix-grpc-daemon[1117]: time=2026-09-21T12:10:21.417Z level=INFO msg="Uploading k48d52qj0czpjmzj15yq3fd14zfw5iq6-slow-rs (120B)"5412lb # [ 110.915629] niks3-server[1060]: 2026/09/21 12:10:21 INFO Registered completed upload object_key=nar/19z3qdigvc4gggg31dgc0wi5w85pgaq9dk7bvngh0yhckpy55jgq.nar.zst5413worker1 # [ 110.912230] nix-grpc-daemon[1117]: time=2026-09-21T12:10:21.459Z level=INFO msg="Uploading 1 narinfos"5414lb # [ 110.932659] niks3-server[1060]: 2026/09/21 12:10:21 INFO Received sign narinfos request method=POST path=/api/pending_closures/34/sign5415lb # [ 110.946140] niks3-server[1060]: 2026/09/21 12:10:21 INFO Signed narinfos id=34 count=15416lb # [ 110.953002] niks3-server[1060]: 2026/09/21 12:10:21 INFO Registered completed upload object_key=log/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv5417lb # [ 110.960347] niks3-server[1060]: 2026/09/21 12:10:21 INFO Registered completed upload object_key=k48d52qj0czpjmzj15yq3fd14zfw5iq6.ls5418lb # [ 110.968961] niks3-server[1060]: 2026/09/21 12:10:21 INFO Received complete upload request method=POST path=/api/pending_closures/34/complete5419lb # [ 110.973340] niks3-server[1060]: 2026/09/21 12:10:21 INFO Completed upload id=345420worker1 # [ 110.951490] nix-grpc-daemon[1117]: time=2026-09-21T12:10:21.499Z level=INFO msg="Upload complete. (342ms)"5421lb # [ 110.975937] niks3-server[1060]: 2026/09/21 12:10:21 INFO Registered completed upload object_key=k48d52qj0czpjmzj15yq3fd14zfw5iq6.narinfo5422worker1 # [ 110.964939] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:21Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43448 duration_s=7 drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv assign_id=4 outputs=15423worker1 # [ 110.978951] nix-daemon[842]: reaped child process 1258, status = succeeded5424worker1 # [ 110.989404] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:21Z level=info event=signal_exit5425client # [ 111.039228] systemd[1]: rs.service: Deactivated successfully.5426client # [ 111.044127] systemd[1]: rs.service: Consumed 184ms CPU time over 8.938s wall clock time, 39.2M memory peak, 7.1K incoming IP traffic, 8.4K outgoing IP traffic.5427worker1 # [ 111.213705] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:21Z level=info event=scheduler_yield5428lb # [ 111.237335] niks3-server[1060]: 2026/09/21 12:10:21 INFO lead: released remote=192.168.1.3:528025429worker2 # [ 111.229542] nix-grpc-daemon[1375]: ts=2026-09-21T12:10:21Z level=info event=scheduler_restarting addr=lb:500515430worker1 # [ 111.228118] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:21Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:434425431worker1 # [ 111.233544] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:21Z level=info event=worker_gone addr=192.168.1.4:500515432worker1 # [ 111.239282] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:21Z level=info event=scheduler_restarting addr=lb:500515433worker1 # [ 111.243387] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:21Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:434425434worker1 # [ 111.247810] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:21Z level=info event=worker_gone addr=192.168.1.3:500515435worker2 # [ 111.430065] nix-grpc-daemon[1375]: ts=2026-09-21T12:10:21Z level=info event=scheduler_connected addr=lb:500515436worker2 # [ 111.435369] nix-grpc-daemon[1375]: ts=2026-09-21T12:10:21Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"5437worker1 # [ 111.434517] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:21Z level=info event=scheduler_connected addr=lb:500515438worker1 # [ 111.441240] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:21Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"5439lb # [ 111.623583] niks3-server[1060]: 2026/09/21 12:10:22 INFO lead: acquired remote=192.168.1.4:386485440worker2 # [ 111.614842] nix-grpc-daemon[1375]: ts=2026-09-21T12:10:22Z level=info event=scheduler_take_over5441worker2 # [ 111.637263] nix-grpc-daemon[1375]: ts=2026-09-21T12:10:22Z level=info event=scheduler_connected addr=lb:500515442worker2 # [ 111.647581] nix-grpc-daemon[1375]: ts=2026-09-21T12:10:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5443worker1 # [ 111.642223] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:22Z level=info event=scheduler_connected addr=lb:500515444worker1 # [ 111.647332] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5445worker2: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr=', in 1.13 seconds)5446client: waiting for success: ! systemctl is-active rs5447client: (finished: waiting for success: ! systemctl is-active rs, in 0.07 seconds)5448client: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }5449client: (finished: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }, in 0.05 seconds)5450client: must fail: journalctl -u rs | grep -q 'reconnecting'5451worker2 # [ 111.846090] nix-grpc-daemon[1375]: ts=2026-09-21T12:10:22Z level=info event=scheduler_connected addr=lb:500515452worker2 # [ 111.851456] nix-grpc-daemon[1375]: ts=2026-09-21T12:10:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5453client: (finished: must fail: journalctl -u rs | grep -q 'reconnecting', in 0.03 seconds)5454lb: must succeed: curl -sf localhost:9901/clusters5455worker1 # [ 111.847999] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:22Z level=info event=scheduler_connected addr=lb:500515456worker1 # [ 111.853547] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5457lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5458worker2 # [ 112.052376] nix-grpc-daemon[1375]: ts=2026-09-21T12:10:22Z level=info event=scheduler_connected addr=lb:500515459worker2 # [ 112.057409] nix-grpc-daemon[1375]: ts=2026-09-21T12:10:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5460worker1 # [ 112.055657] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:22Z level=info event=scheduler_connected addr=lb:500515461worker1 # [ 112.059762] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5462worker1 # [ 112.219427] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:22Z level=info event=scheduler_restarting5463worker2 # [ 112.260301] nix-grpc-daemon[1375]: ts=2026-09-21T12:10:22Z level=info event=scheduler_connected addr=lb:500515464worker1 # [ 112.250496] nix-daemon[842]: reaped child process 1107, status = succeeded5465worker2 # [ 112.264414] nix-grpc-daemon[1375]: ts=2026-09-21T12:10:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5466worker1 # [ 112.278496] nix-grpc-daemon[1043]: ts=2026-09-21T12:10:22Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5467worker2 # [ 112.461163] nix-grpc-daemon[1375]: ts=2026-09-21T12:10:23Z level=info event=scheduler_connected addr=lb:500515468worker2 # [ 112.466170] nix-grpc-daemon[1375]: ts=2026-09-21T12:10:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5469worker1 # [ 112.576421] nix-grpc-daemon[1043]: E0000 00:00:1789992623.123862 1088 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 16, started true, shutdown true5470worker1 # [ 112.583111] nix-grpc-daemon[1043]: E0000 00:00:1789992623.124139 1088 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5471worker2 # [ 112.667171] nix-grpc-daemon[1375]: ts=2026-09-21T12:10:23Z level=info event=scheduler_connected addr=lb:500515472worker2 # [ 112.672281] nix-grpc-daemon[1375]: ts=2026-09-21T12:10:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5473worker2 # [ 112.873322] nix-grpc-daemon[1375]: ts=2026-09-21T12:10:23Z level=info event=scheduler_connected addr=lb:500515474worker2 # [ 112.878373] nix-grpc-daemon[1375]: ts=2026-09-21T12:10:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5475lb: must succeed: curl -sf localhost:9901/clusters5476lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5477worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5478worker2 # [ 113.079531] nix-grpc-daemon[1375]: ts=2026-09-21T12:10:23Z level=info event=scheduler_connected addr=lb:500515479worker2 # [ 113.094133] nix-grpc-daemon[1375]: ts=2026-09-21T12:10:23Z level=info event=oidc_jwks provider=mock keys=15480worker2 # [ 113.103675] nix-grpc-daemon[1375]: ts=2026-09-21T12:10:23Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:348425481worker2 # [ 113.107983] nix-grpc-daemon[1375]: ts=2026-09-21T12:10:23Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=05482worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.10 seconds)5483worker1 # [ 113.837055] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5484worker1 # [ 113.840436] systemd[1]: Stopped Nix worker-protocol over gRPC.5485worker1 # [ 113.842967] systemd[1]: nix-grpc-daemon.service: Consumed 1.846s CPU time over 27.082s wall clock time, 36.7M memory peak, 49.4K incoming IP traffic, 91.9K outgoing IP traffic.5486worker1 # [ 113.862925] systemd[1]: Starting Nix worker-protocol over gRPC...5487worker1 # [ 114.059237] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:24Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5488worker1 # [ 114.061828] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:24Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5489worker1 # [ 114.065988] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:24Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15490worker1 # [ 114.073387] nix-grpc-daemon[1311]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5491worker1 # [ 114.074855] nix-grpc-daemon[1311]: E0000 00:00:1789992624.620450 1311 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5492worker1 # [ 114.082149] nix-grpc-daemon[1311]: E0000 00:00:1789992624.626594 1311 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5493worker1 # [ 114.086048] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:24Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5494worker1 # [ 114.088414] systemd[1]: Started Nix worker-protocol over gRPC.5495worker1 # [ 114.095886] nix-grpc-daemon[1311]: E0000 00:00:1789992624.641396 1352 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5496lb: must succeed: curl -sf localhost:9901/clusters5497worker1 # [ 114.124147] nix-grpc-daemon[1311]: E0000 00:00:1789992624.659879 1352 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5498worker1 # [ 114.126444] nix-grpc-daemon[1311]: E0000 00:00:1789992624.660046 1352 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5499worker1 # [ 114.132528] nix-grpc-daemon[1311]: E0000 00:00:1789992624.679422 1352 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5500worker1 # [ 114.149113] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:24Z level=info event=scheduler_connected addr=lb:500515501worker2 # [ 114.162391] nix-grpc-daemon[1375]: ts=2026-09-21T12:10:24Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:348425502lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds)5503worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5504worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)5505(finished: subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise, in 12.27 seconds)5506subtest: scheduler restart mid-build: build finishes, no double build5507client: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag mid5508client # [ 114.369782] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key" --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag mid.5509client # Running as unit: mid.service; invocation ID: 76dff21d56ce4959bd854b59b068acea5510client: (finished: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag mid, in 0.09 seconds)5511worker2 # [ 114.168560] nix-grpc-daemon[1375]: ts=2026-09-21T12:10:24Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=05512worker2 # [ 114.613571] nix-grpc-daemon[1375]: ts=2026-09-21T12:10:25Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:348545513worker2 # [ 114.641055] nix-daemon[863]: accepted connection from pid 1375, user nix-grpc-daemon (trusted)5514client # [ 114.667628] nix[1210]: don't know how to build these paths:5515client # [ 114.668964] nix[1210]: /nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv5516worker2 # [ 114.665168] nix-grpc-daemon[1375]: ts=2026-09-21T12:10:25Z level=debug event=want client=1 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv cached=05517worker2 # [ 114.672239] nix-grpc-daemon[1375]: ts=2026-09-21T12:10:25Z level=debug event=assigned drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv worker=192.168.1.4:50051 clients=15518worker2 # [ 114.709546] nix-grpc-daemon[1375]: ts=2026-09-21T12:10:25Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:348545519worker2 # [ 114.730929] nix-grpc-daemon[1375]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 24 ms (attempt 1/5)5520worker2 # [ 114.762410] nix-grpc-daemon[1375]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 70 ms (attempt 2/5)5521worker2 # [ 114.840147] nix-grpc-daemon[1375]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 127 ms (attempt 3/5)5522worker2 # [ 114.967512] nix-grpc-daemon[1375]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 775 ms (attempt 4/5)5523worker2 # [ 115.745123] nix-grpc-daemon[1375]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5524worker2 # [ 115.745477] nix-grpc-daemon[1375]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5525worker2 # [ 115.771598] nix-grpc-daemon[1375]: ts=2026-09-21T12:10:26Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:348545526client # [ 115.801163] nix[1210]: copying 1 paths to 'lb:50051'...5527worker2 # [ 115.787858] nix-grpc-daemon[1375]: ts=2026-09-21T12:10:26Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:348545528client # [ 115.806878] nix[1210]: copying path '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' to 'lb:50051'...5529worker2 # [ 115.803982] nix-daemon[863]: accepted connection from pid 1375, user nix-grpc-daemon (trusted)5530worker2 # [ 115.836153] nix-grpc-daemon[1375]: ts=2026-09-21T12:10:26Z level=info event=niks3_push_started pid=14955531worker2 # [ 116.055792] nix-grpc-daemon[1495]: time=2026-09-21T12:10:26.598Z level=INFO msg="Configuring client TLS" cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker2.pem key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker2.key ca=""5532worker2 # [ 116.231190] nix-daemon[863]: accepted connection from pid 1508, user nix-grpc-daemon (trusted)5533worker2 # [ 116.248434] nix-daemon[863]: reaped child process 1511, status = succeeded5534worker2 # [ 116.420398] nix-daemon[863]: accepted connection from pid 1513, user nix-grpc-daemon (trusted)5535worker2 # [ 116.437628] nix-daemon[863]: reaped child process 1516, status = succeeded5536lb # [ 116.461788] niks3-server[1060]: 2026/09/21 12:10:27 INFO Received uploads request method=POST path=/api/pending_closures5537worker2 # [ 116.460068] nix-grpc-daemon[1495]: time=2026-09-21T12:10:27.003Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5538worker2 # [ 116.461508] nix-grpc-daemon[1495]: time=2026-09-21T12:10:27.003Z level=INFO msg="Uploading chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv (424B)"5539lb # [ 116.496658] niks3-server[1060]: 2026/09/21 12:10:27 INFO Registered completed upload object_key=nar/14lxjrdcxwbhc1kgg39y29gnbx5cz80j4852d1i4w3rvq7g4b04g.nar.zst5540worker2 # [ 116.500741] nix-grpc-daemon[1495]: time=2026-09-21T12:10:27.043Z level=INFO msg="Uploading 1 narinfos"5541lb # [ 116.510256] niks3-server[1060]: 2026/09/21 12:10:27 INFO Received sign narinfos request method=POST path=/api/pending_closures/35/sign5542lb # [ 116.520362] niks3-server[1060]: 2026/09/21 12:10:27 INFO Signed narinfos id=35 count=15543lb # [ 116.530100] niks3-server[1060]: 2026/09/21 12:10:27 INFO Registered completed upload object_key=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc.ls5544lb # [ 116.539494] niks3-server[1060]: 2026/09/21 12:10:27 INFO Received complete upload request method=POST path=/api/pending_closures/35/complete5545worker2 # [ 116.536204] nix-grpc-daemon[1495]: time=2026-09-21T12:10:27.079Z level=INFO msg="Upload complete. (476ms)"5546lb # [ 116.550785] niks3-server[1060]: 2026/09/21 12:10:27 INFO Completed upload id=355547worker2 # [ 116.544112] nix-grpc-daemon[1375]: ts=2026-09-21T12:10:27Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34854 duration_s=0 paths=1 nar_bytes_in=4245548worker2 # [ 116.556696] nix-grpc-daemon[1375]: ts=2026-09-21T12:10:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:348545549worker2 # [ 116.566084] nix-grpc-daemon[1495]: time=2026-09-21T12:10:27.099Z level=WARN msg="Failed to register uploaded object" key=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc.narinfo error="server returned 404: object is not pending upload\n"5550worker2 # [ 116.576409] nix-daemon[863]: accepted connection from pid 1375, user nix-grpc-daemon (trusted)5551worker2 # [ 116.580108] nix-daemon[863]: reaped child process 1493, status = succeeded5552client # [ 116.600267] nix[1210]: building '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' on lb:50051...5553worker2 # [ 116.587508] nix-daemon[863]: accepted connection from pid 1375, user nix-grpc-daemon (trusted)5554lb: must succeed: curl -sf localhost:9901/clusters5555lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5556lb: must succeed: curl -sf localhost:9901/clusters5557lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5558worker2: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service5559worker2 # [ 118.352511] systemd[1]: nix-grpc-daemon.service: Sent signal SIGKILL to main process 1375 (nix-grpc-daemon) on client request.5560worker2 # [ 118.354075] systemd[1]: nix-grpc-daemon.service: Killed unit cgroup '/system.slice/nix-grpc-daemon.service' with SIGKILL on client request.5561lb # [ 118.373641] niks3-server[1060]: 2026/09/21 12:10:28 INFO lead: released remote=192.168.1.4:386485562worker2 # [ 118.381370] systemd[1]: Starting scheduler-token.service...5563client # [ 118.392507] nix[1210]: chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), asking the scheduler again5564worker1 # [ 118.364981] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:28Z level=info event=scheduler_disconnected addr=lb:50051 error="upstream connect error or disconnect/reset before headers. reset reason: connection termination"5565client # [ 118.392898] nix[1210]: scheduler at lb:50051: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), reconnecting5566worker2 # [ 118.401455] systemd[1]: nix-grpc-daemon.service: Main process exited, code=killed, status=9/KILL5567worker2 # [ 118.402955] systemd[1]: nix-grpc-daemon.service: Failed with result 'signal'.5568worker2 # [ 118.407727] systemd[1]: nix-grpc-daemon.service: Consumed 587ms CPU time over 16.993s wall clock time, 34.2M memory peak, 16.7K incoming IP traffic, 31.7K outgoing IP traffic.5569worker2 # [ 118.432948] nix-daemon[863]: reaped child process 1518, status = succeeded5570worker2 # [ 118.434094] nix-daemon[863]: reaped child process 1481, status = succeeded5571worker2 # [ 118.440597] nix-daemon[1520]: unexpected Nix daemon error: error: interrupted by the user5572worker2 # [ 118.449638] nix-daemon[863]: reaped child process 1520, status = failed with exit code 15573worker2 # [ 118.492117] systemd[1]: scheduler-token.service: Deactivated successfully.5574worker2 # [ 118.493112] systemd[1]: Finished scheduler-token.service.5575worker2 # [ 118.493839] systemd[1]: scheduler-token.service: Consumed 26ms CPU time over 109ms wall clock time, 2.2M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5576worker2 # [ 118.495727] systemd[1]: nix-grpc-daemon.service: Scheduled restart job immediately on client request, restart counter is at 1.5577worker2 # [ 118.501093] systemd[1]: Starting scheduler-token.service...5578worker2 # [ 118.571735] systemd[1]: scheduler-token.service: Deactivated successfully.5579worker2 # [ 118.575358] systemd[1]: Finished scheduler-token.service.5580worker2 # [ 118.578283] systemd[1]: scheduler-token.service: Consumed 23ms CPU time over 70ms wall clock time, 2.2M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5581worker1 # [ 118.565489] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:29Z level=info event=scheduler_connected addr=lb:500515582worker2 # [ 118.580924] systemd[1]: Starting Nix worker-protocol over gRPC...5583worker2 # [ 118.750971] nix-grpc-daemon[1563]: ts=2026-09-21T12:10:29Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5584worker2 # [ 118.753816] nix-grpc-daemon[1563]: ts=2026-09-21T12:10:29Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5585worker2 # [ 118.757851] nix-grpc-daemon[1563]: ts=2026-09-21T12:10:29Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15586worker2 # [ 118.765263] nix-grpc-daemon[1563]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5587worker2 # [ 118.766704] nix-grpc-daemon[1563]: E0000 00:00:1789992629.308044 1563 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5588worker2: (finished: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service, in 0.49 seconds)5589worker2 # [ 118.778224] nix-grpc-daemon[1563]: E0000 00:00:1789992629.314007 1563 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5590client: must succeed: sleep 5.05591worker2 # [ 118.786480] nix-grpc-daemon[1563]: ts=2026-09-21T12:10:29Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5592worker2 # [ 118.788673] systemd[1]: Started Nix worker-protocol over gRPC.5593lb # [ 118.798374] niks3-server[1060]: 2026/09/21 12:10:29 INFO lead: acquired remote=192.168.1.4:440365594worker2 # [ 118.792072] nix-grpc-daemon[1563]: E0000 00:00:1789992629.335509 1603 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5595worker1 # [ 118.782763] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:29Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"5596worker2 # [ 118.824128] nix-grpc-daemon[1563]: ts=2026-09-21T12:10:29Z level=info event=scheduler_take_over5597worker2 # [ 118.827046] nix-grpc-daemon[1563]: E0000 00:00:1789992629.353877 1603 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5598worker2 # [ 118.829814] nix-grpc-daemon[1563]: E0000 00:00:1789992629.354018 1603 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5599worker2 # [ 118.832328] nix-grpc-daemon[1563]: E0000 00:00:1789992629.370011 1603 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5600worker2 # [ 118.840455] nix-grpc-daemon[1563]: ts=2026-09-21T12:10:29Z level=info event=scheduler_connected addr=lb:500515601worker2 # [ 118.845751] nix-grpc-daemon[1563]: ts=2026-09-21T12:10:29Z level=info event=oidc_jwks provider=mock keys=15602worker2 # [ 118.847022] nix-grpc-daemon[1563]: ts=2026-09-21T12:10:29Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:348665603worker2 # [ 118.848948] nix-grpc-daemon[1563]: ts=2026-09-21T12:10:29Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=05604worker2 # [ 118.884023] nix-grpc-daemon[1563]: ts=2026-09-21T12:10:29Z level=debug event=want client=1 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv cached=05605worker2 # [ 118.886562] nix-grpc-daemon[1563]: ts=2026-09-21T12:10:29Z level=debug event=assigned drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv worker=192.168.1.4:50051 clients=15606worker2 # [ 118.941213] nix-grpc-daemon[1563]: ts=2026-09-21T12:10:29Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:348765607worker2 # [ 118.952516] nix-daemon[863]: accepted connection from pid 1563, user nix-grpc-daemon (trusted)5608worker2 # [ 118.968929] nix-daemon[863]: accepted connection from pid 1563, user nix-grpc-daemon (trusted)5609worker2 # [ 118.980344] nix-daemon[863]: accepted connection from pid 1563, user nix-grpc-daemon (trusted)5610client # [ 118.994937] nix[1210]: building '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' on lb:50051...5611worker2 # [ 118.996192] nix-grpc-daemon[1563]: ts=2026-09-21T12:10:29Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:348665612worker1 # [ 118.983915] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:29Z level=info event=scheduler_connected addr=lb:500515613worker2 # [ 118.998620] nix-grpc-daemon[1563]: ts=2026-09-21T12:10:29Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=05614lb # [ 120.663896] envoy[910]: conn peer=[2001:db8:1::1]:35206 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2979 tx=1428 ms=86415615lb # [ 120.668868] envoy[910]: conn peer=[2001:db8:1::3]:44454 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=4147 tx=1321 ms=252375616lb # [ 120.669014] envoy[910]: conn peer=[2001:db8:1::4]:39250 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=8006 tx=646 ms=167115617client: (finished: must succeed: sleep 5.0, in 5.06 seconds)5618client: waiting for success: ! systemctl is-active mid5619worker2 # [ 124.203926] nix-grpc-daemon[1563]: ts=2026-09-21T12:10:34Z level=info event=niks3_push_started pid=16465620worker2 # [ 124.221551] nix-daemon[863]: reaped child process 1621, status = succeeded5621worker2 # [ 124.401585] nix-grpc-daemon[1646]: time=2026-09-21T12:10:34.944Z level=INFO msg="Configuring client TLS" cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker2.pem key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker2.key ca=""5622worker2 # [ 124.576264] nix-daemon[863]: accepted connection from pid 1659, user nix-grpc-daemon (trusted)5623worker2 # [ 124.594337] nix-daemon[863]: reaped child process 1662, status = succeeded5624lb # [ 124.617893] niks3-server[1060]: 2026/09/21 12:10:35 INFO Received uploads request method=POST path=/api/pending_closures5625worker2 # [ 124.616728] nix-grpc-daemon[1646]: time=2026-09-21T12:10:35.160Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5626worker2 # [ 124.621498] nix-grpc-daemon[1646]: time=2026-09-21T12:10:35.162Z level=INFO msg="Uploading fk2vdwmn2msb3ja19s2jwak35ignz889-slow-mid (120B)"5627lb # [ 124.664854] niks3-server[1060]: 2026/09/21 12:10:35 INFO Registered completed upload object_key=log/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv5628worker2 # [ 124.678539] nix-grpc-daemon[1646]: time=2026-09-21T12:10:35.220Z level=INFO msg="Uploading 1 narinfos"5629lb # [ 124.687490] niks3-server[1060]: 2026/09/21 12:10:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/36/sign5630lb # [ 124.701460] niks3-server[1060]: 2026/09/21 12:10:35 INFO Signed narinfos id=36 count=15631lb # [ 124.703722] niks3-server[1060]: 2026/09/21 12:10:35 INFO Registered completed upload object_key=fk2vdwmn2msb3ja19s2jwak35ignz889.ls5632lb # [ 124.724586] niks3-server[1060]: 2026/09/21 12:10:35 INFO Received complete upload request method=POST path=/api/pending_closures/36/complete5633worker2 # [ 124.714491] nix-grpc-daemon[1646]: time=2026-09-21T12:10:35.258Z level=INFO msg="Upload complete. (308ms)"5634lb # [ 124.727615] niks3-server[1060]: 2026/09/21 12:10:35 INFO Completed upload id=365635worker2 # [ 124.728296] nix-grpc-daemon[1563]: ts=2026-09-21T12:10:35Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34876 duration_s=5 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv assign_id=1 outputs=15636lb # [ 124.742830] niks3-server[1060]: 2026/09/21 12:10:35 INFO Registered completed upload object_key=nar/0462jb7zwy2d73k3q6v09xnk3bx318v8ik5ckmp32w2hzvwj4k2d.nar.zst5637worker2 # [ 124.744383] nix-grpc-daemon[1646]: time=2026-09-21T12:10:35.280Z level=WARN msg="Failed to register uploaded object" key=fk2vdwmn2msb3ja19s2jwak35ignz889.narinfo error="server returned 404: object is not pending upload\n"5638worker2 # [ 124.750745] nix-daemon[863]: reaped child process 1619, status = succeeded5639client # [ 124.784707] systemd[1]: mid.service: Deactivated successfully.5640client # [ 124.789722] systemd[1]: mid.service: Consumed 196ms CPU time over 10.418s wall clock time, 41.2M memory peak, 8.5K incoming IP traffic, 10.4K outgoing IP traffic.5641client: (finished: waiting for success: ! systemctl is-active mid, in 1.14 seconds)5642client: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }5643client: (finished: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }, in 0.05 seconds)5644(finished: subtest: scheduler restart mid-build: build finishes, no double build, in 11.06 seconds)5645subtest: interrupting the client stops the build on the worker5646client: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag intr5647client # [ 125.403082] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key" --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag intr.5648client # Running as unit: intr.service; invocation ID: 379f5541fd034bc7b94ebeb8b4ac8f045649client: (finished: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag intr, in 0.05 seconds)5650worker2 # [ 125.628408] nix-grpc-daemon[1563]: ts=2026-09-21T12:10:36Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:348765651client # [ 125.652362] nix[1246]: don't know how to build these paths:5652client # [ 125.653263] nix[1246]: /nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv5653worker2 # [ 125.646245] nix-grpc-daemon[1563]: ts=2026-09-21T12:10:36Z level=debug event=want client=2 drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv cached=05654worker2 # [ 125.652309] nix-grpc-daemon[1563]: ts=2026-09-21T12:10:36Z level=debug event=assigned drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv worker=192.168.1.3:50051 clients=15655worker1 # [ 125.698782] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:462565656worker1 # [ 125.713168] nix-daemon[842]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)5657worker1 # [ 125.742189] nix-grpc-daemon[1311]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 22 ms (attempt 1/5)5658worker1 # [ 125.771737] nix-grpc-daemon[1311]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 110 ms (attempt 2/5)5659worker1 # [ 125.889397] nix-grpc-daemon[1311]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 8 ms (attempt 3/5)5660worker1 # [ 125.898102] nix-grpc-daemon[1311]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 134 ms (attempt 4/5)5661worker1 # [ 126.040329] nix-grpc-daemon[1311]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5662worker1 # [ 126.043934] nix-grpc-daemon[1311]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5663worker1 # [ 126.064431] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:36Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:462565664client # [ 126.107225] nix[1246]: copying 1 paths to 'lb:50051'...5665worker1 # [ 126.079644] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:36Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:462565666client # [ 126.109851] nix[1246]: copying path '/nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv' to 'lb:50051'...5667worker1 # [ 126.094831] nix-daemon[842]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)5668worker1 # [ 126.122578] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:36Z level=info event=niks3_push_started pid=14035669worker1 # [ 126.343979] nix-grpc-daemon[1403]: time=2026-09-21T12:10:36.890Z level=INFO msg="Configuring client TLS" cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker1.pem key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker1.key ca=""5670worker1 # [ 126.515458] nix-daemon[842]: accepted connection from pid 1416, user nix-grpc-daemon (trusted)5671worker1 # [ 126.531153] nix-daemon[842]: reaped child process 1419, status = succeeded5672worker1 # [ 126.719690] nix-daemon[842]: accepted connection from pid 1421, user nix-grpc-daemon (trusted)5673worker1 # [ 126.734228] nix-daemon[842]: reaped child process 1428, status = succeeded5674lb # [ 126.774433] niks3-server[1060]: 2026/09/21 12:10:37 INFO Received uploads request method=POST path=/api/pending_closures5675worker1 # [ 126.765753] nix-grpc-daemon[1403]: time=2026-09-21T12:10:37.313Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5676worker1 # [ 126.767381] nix-grpc-daemon[1403]: time=2026-09-21T12:10:37.315Z level=INFO msg="Uploading 85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv (424B)"5677lb # [ 126.814048] niks3-server[1060]: 2026/09/21 12:10:37 INFO Registered completed upload object_key=nar/0h460h76prjyvhqhyxlhz1xrmh3s520jbd19gaagy4hp4vc92r1p.nar.zst5678lb # [ 126.825712] niks3-server[1060]: 2026/09/21 12:10:37 INFO Received sign narinfos request method=POST path=/api/pending_closures/37/sign5679worker1 # [ 126.811913] nix-grpc-daemon[1403]: time=2026-09-21T12:10:37.359Z level=INFO msg="Uploading 1 narinfos"5680lb # [ 126.838239] niks3-server[1060]: 2026/09/21 12:10:37 INFO Signed narinfos id=37 count=15681lb # [ 126.860254] niks3-server[1060]: 2026/09/21 12:10:37 INFO Received complete upload request method=POST path=/api/pending_closures/37/complete5682worker1 # [ 126.850450] nix-grpc-daemon[1403]: time=2026-09-21T12:10:37.398Z level=INFO msg="Upload complete. (502ms)"5683lb # [ 126.872902] niks3-server[1060]: 2026/09/21 12:10:37 INFO Registered completed upload object_key=85wa5czf6m5ng93xyq44r1affb74csh8.ls5684lb # [ 126.878235] niks3-server[1060]: 2026/09/21 12:10:37 INFO Completed upload id=375685lb # [ 126.881690] niks3-server[1060]: 2026/09/21 12:10:37 INFO Registered completed upload object_key=85wa5czf6m5ng93xyq44r1affb74csh8.narinfo5686worker1 # [ 126.861760] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:37Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46256 duration_s=0 paths=1 nar_bytes_in=4245687worker1 # [ 126.875504] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:462565688worker1 # [ 126.885260] nix-daemon[842]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)5689worker1 # [ 126.888336] nix-daemon[842]: reaped child process 1401, status = succeeded5690client # [ 126.925996] nix[1246]: building '/nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv' on lb:50051...5691worker1 # [ 126.900974] nix-daemon[842]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)5692client: must succeed: systemctl kill -s INT intr5693worker1 # [ 129.092940] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:39Z level=info event=revoked_running drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv5694client # [ 129.115049] systemd[1]: intr.service: Sent signal SIGINT to main process 1246 (nix) on client request.5695client # [ 129.124821] nix[1246]: shutting down85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED)5696client: (finished: must succeed: systemctl kill -s INT intr, in 0.07 seconds)5697client # [ 129.135853] nix[1246]: error: interrupted by the user5698worker1 # [ 129.134406] nix-daemon[842]: reaped child process 1430, status = succeeded5699worker1 # [ 129.137697] nix-daemon[1432]: unexpected Nix daemon error: error: interrupted by the user5700worker1 # [ 129.148290] nix-daemon[842]: reaped child process 1432, status = failed with exit code 15701client # [ 129.178072] systemd[1]: intr.service: Main process exited, code=exited, status=1/FAILURE5702client # [ 129.183769] systemd[1]: intr.service: Failed with result 'exit-code'.5703client # [ 129.186588] systemd[1]: intr.service: Consumed 163ms CPU time over 3.778s wall clock time, 41.4M memory peak, 6.5K incoming IP traffic, 8.3K outgoing IP traffic.5704(finished: subtest: interrupting the client stops the build on the worker, in 3.89 seconds)5705subtest: a deploy mid-build drains: the build finishes, the new generation takes over5706client: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag sw5707client # [ 129.322098] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key" --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag sw.5708client # Running as unit: sw.service; invocation ID: 29e3f7e925c64174a821185bee0bd5595709client: (finished: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag sw, in 0.09 seconds)5710client # [ 129.581218] nix[1266]: don't know how to build these paths:5711client # [ 129.582166] nix[1266]: /nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv5712worker1 # [ 129.550393] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:40Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:462565713worker2 # [ 129.578141] nix-grpc-daemon[1563]: ts=2026-09-21T12:10:40Z level=debug event=want client=3 drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv cached=05714worker2 # [ 129.583529] nix-grpc-daemon[1563]: ts=2026-09-21T12:10:40Z level=debug event=assigned drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv worker=192.168.1.4:50051 clients=15715worker2 # [ 129.621908] nix-grpc-daemon[1563]: ts=2026-09-21T12:10:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:348765716worker2 # [ 129.638081] nix-grpc-daemon[1563]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 82 ms (attempt 1/5)5717worker2 # [ 129.727819] nix-grpc-daemon[1563]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 47 ms (attempt 2/5)5718worker2 # [ 129.775096] nix-grpc-daemon[1563]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 26 ms (attempt 3/5)5719worker2 # [ 129.809161] nix-grpc-daemon[1563]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 22 ms (attempt 4/5)5720worker2 # [ 129.838777] nix-grpc-daemon[1563]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5721worker2 # [ 129.844716] nix-grpc-daemon[1563]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5722worker2 # [ 129.863147] nix-grpc-daemon[1563]: ts=2026-09-21T12:10:40Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:348765723client # [ 129.894983] nix[1266]: copying 1 paths to 'lb:50051'...5724worker2 # [ 129.878629] nix-grpc-daemon[1563]: ts=2026-09-21T12:10:40Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:348765725client # [ 129.897981] nix[1266]: copying path '/nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv' to 'lb:50051'...5726worker2 # [ 129.893452] nix-daemon[863]: accepted connection from pid 1563, user nix-grpc-daemon (trusted)5727worker2 # [ 130.097584] nix-daemon[863]: accepted connection from pid 1698, user nix-grpc-daemon (trusted)5728worker2 # [ 130.116861] nix-daemon[863]: reaped child process 1701, status = succeeded5729worker2 # [ 130.288834] nix-daemon[863]: accepted connection from pid 1703, user nix-grpc-daemon (trusted)5730worker2 # [ 130.306618] nix-daemon[863]: reaped child process 1706, status = succeeded5731lb # [ 130.329734] niks3-server[1060]: 2026/09/21 12:10:40 INFO Received uploads request method=POST path=/api/pending_closures5732worker2 # [ 130.335112] nix-grpc-daemon[1646]: time=2026-09-21T12:10:40.878Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5733worker2 # [ 130.336745] nix-grpc-daemon[1646]: time=2026-09-21T12:10:40.878Z level=INFO msg="Uploading x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv (416B)"5734lb # [ 130.364322] niks3-server[1060]: 2026/09/21 12:10:40 INFO Registered completed upload object_key=nar/0jczp45bkr8vp1ymznxf6da1iv8dzh42pnl36wiprbzwhpi2ddia.nar.zst5735lb # [ 130.377783] niks3-server[1060]: 2026/09/21 12:10:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/38/sign5736worker2 # [ 130.374485] nix-grpc-daemon[1646]: time=2026-09-21T12:10:40.917Z level=INFO msg="Uploading 1 narinfos"5737lb # [ 130.389682] niks3-server[1060]: 2026/09/21 12:10:40 INFO Signed narinfos id=38 count=15738lb # [ 130.395576] niks3-server[1060]: 2026/09/21 12:10:40 INFO Registered completed upload object_key=x0m4asg5xx7vgj4dip2crwn6zhxqngvs.ls5739lb # [ 130.406290] niks3-server[1060]: 2026/09/21 12:10:40 INFO Received complete upload request method=POST path=/api/pending_closures/38/complete5740lb # [ 130.411374] niks3-server[1060]: 2026/09/21 12:10:40 INFO Completed upload id=385741worker2 # [ 130.401211] nix-grpc-daemon[1646]: time=2026-09-21T12:10:40.944Z level=INFO msg="Upload complete. (490ms)"5742lb # [ 130.414323] niks3-server[1060]: 2026/09/21 12:10:40 INFO Registered completed upload object_key=x0m4asg5xx7vgj4dip2crwn6zhxqngvs.narinfo5743worker2 # [ 130.408137] nix-grpc-daemon[1563]: ts=2026-09-21T12:10:40Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34876 duration_s=0 paths=1 nar_bytes_in=4165744worker2 # [ 130.421761] nix-grpc-daemon[1563]: ts=2026-09-21T12:10:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:348765745worker2 # [ 130.428675] nix-daemon[863]: accepted connection from pid 1563, user nix-grpc-daemon (trusted)5746worker2 # [ 130.433874] nix-daemon[863]: reaped child process 1696, status = succeeded5747client # [ 130.459380] nix[1266]: building '/nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv' on lb:50051...5748worker2 # [ 130.447473] nix-daemon[863]: accepted connection from pid 1563, user nix-grpc-daemon (trusted)5749lb # [ 130.665938] envoy[910]: conn peer=[2001:db8:1::1]:38738 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3940 tx=1854 ms=101345750lb # [ 130.671710] envoy[910]: conn peer=[2001:db8:1::1]:54084 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2987 tx=850 ms=35075751worker2: must succeed: timeout 20 /run/current-system/specialisation/next/bin/switch-to-configuration test >&25752worker2 # Checking switch inhibitors... done5753worker2 # [ 132.783858] nixos[1742]: switching to system configuration /nix/store/n5ldl2n7ssxrifcgvi8l27ajdvbwnl5r-nixos-system-worker2-test5754worker2 # [ 132.787067] systemd[1]: Stopped target Local File Systems.5755worker2 # [ 132.788882] systemd[1]: Stopped target Remote File Systems.5756worker2 # activating the configuration...5757worker2 # setting up /etc...5758worker2 # [ 133.569993] systemd[1]: Reload requested from client PID 1742 ('.switch-to-conf') (unit backdoor.service)...5759worker2 # [ 133.576206] systemd[1]: Reloading...5760worker2 # [ 134.474775] systemd[1]: Reloading finished in 901 ms.5761worker2 # restarting sysinit-reactivation.target5762worker2 # [ 134.562100] systemd[1]: Starting Re-setup tmpfiles on a system that is already running....5763worker2 # [ 134.719683] systemd[1]: Finished Re-setup tmpfiles on a system that is already running..5764worker2 # [ 134.724243] systemd[1]: Reached target Reactivate sysinit units.5765worker2 # reloading the following units: dbus-broker.service, nix-grpc-daemon.service5766worker2 # [ 134.728587] systemd[1]: Reloading D-Bus System Message Bus...5767worker2 # [ 134.729749] dbus-broker-launch[502]: Caught SIGHUP, trigger reload.5768worker2 # [ 134.745003] systemd[1]: Reloading Nix worker-protocol over gRPC...5769worker2 # [ 134.761478] dbus-broker-launch[502]: Looking up NSS user entry for 'systemd-timesync'...5770worker2 # [ 134.764514] dbus-broker-launch[502]: NSS returned no entry for 'systemd-timesync'5771worker2 # [ 134.768385] dbus-broker-launch[502]: Invalid user-name in /nix/store/602n2r0g7ngz2pj9sxi9qamwpihwymx6-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync"5772worker2 # [ 134.801373] systemd[1]: Reloaded Nix worker-protocol over gRPC.5773worker2 # [ 134.825842] systemd[1]: Reloaded D-Bus System Message Bus.5774worker2 # [ 134.826704] systemd[1]: Reached target Local File Systems.5775worker2 # [ 134.827494] systemd[1]: Load Kernel Module efi_pstore skipped, unmet condition check ConditionKernelModuleLoaded=!efi_pstore5776worker2 # [ 134.834597] systemd[1]: Starting Create SUID/SGID Wrappers...5777worker2 # [ 134.844316] systemd[1]: Update Boot Loader Random Seed skipped, no trigger condition checks were met.5778worker2 # [ 134.845565] systemd[1]: Clear Stale Hibernate Storage Info skipped, unmet condition check ConditionPathExists=/sys/firmware/efi/efivars/HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc675779worker2 # [ 134.847597] systemd[1]: Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore5780worker2 # [ 134.856913] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os5781worker2 # [ 134.861560] systemd[1]: TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os5782worker2 # [ 134.862755] systemd[1]: TPM PCR Measurements skipped, unmet condition check ConditionSecurity=measured-os5783worker2 # [ 134.863979] systemd[1]: Make TPM PCR Policy skipped, unmet condition check ConditionSecurity=measured-uki5784worker2 # [ 134.869952] systemd[1]: Reached target Remote File Systems.5785worker2 # [ 135.360177] systemd[1]: suid-sgid-wrappers.service: Deactivated successfully.5786worker2 # [ 135.364280] systemd[1]: Finished Create SUID/SGID Wrappers.5787worker2 # [ 135.368541] systemd[1]: Starting linger-users.service...5788worker2 # [ 135.420552] systemd[1]: linger-users.service: Deactivated successfully.5789worker2 # [ 135.424073] systemd[1]: Finished linger-users.service.5790worker2 # the following new units were started: sysinit-reactivation.target, systemd-tmpfiles-resetup.service5791worker2 # [ 135.679331] nixos[1742]: finished switching to system configuration /nix/store/n5ldl2n7ssxrifcgvi8l27ajdvbwnl5r-nixos-system-worker2-test5792worker2: (finished: must succeed: timeout 20 /run/current-system/specialisation/next/bin/switch-to-configuration test >&2, in 3.73 seconds)5793worker2: must succeed: pgrep -f 'read -t [9]0 x'5794worker2: (finished: must succeed: pgrep -f 'read -t [9]0 x', in 0.06 seconds)5795client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag during-drain >&25796worker2 # [ 135.788382] nix-grpc-daemon[1563]: ts=2026-09-21T12:10:46Z level=info event=draining5797worker1 # [ 136.021135] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:46Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:462565798client # don't know how to build these paths:5799client # /nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv5800worker1 # [ 136.034204] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:46Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:462565801worker2 # [ 136.069090] nix-grpc-daemon[1563]: ts=2026-09-21T12:10:46Z level=debug event=want client=4 drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv cached=05802worker2 # [ 136.071179] nix-grpc-daemon[1563]: ts=2026-09-21T12:10:46Z level=debug event=assigned drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv worker=192.168.1.3:50051 clients=15803worker1 # [ 136.108634] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:462565804worker1 # [ 136.123980] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:46Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:462565805client # copying 1 paths to 'lb:50051'...5806worker1 # [ 136.139811] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:46Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:462565807client # copying path '/nix/store/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv' to 'lb:50051'...5808worker1 # [ 136.164156] nix-daemon[842]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)5809worker1 # [ 136.379210] nix-daemon[842]: accepted connection from pid 1479, user nix-grpc-daemon (trusted)5810worker1 # [ 136.395409] nix-daemon[842]: reaped child process 1482, status = succeeded5811worker1 # [ 136.571892] nix-daemon[842]: accepted connection from pid 1484, user nix-grpc-daemon (trusted)5812worker1 # [ 136.587143] nix-daemon[842]: reaped child process 1487, status = succeeded5813lb # [ 136.625250] niks3-server[1060]: 2026/09/21 12:10:47 INFO Received uploads request method=POST path=/api/pending_closures5814worker1 # [ 136.616368] nix-grpc-daemon[1403]: time=2026-09-21T12:10:47.163Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5815worker1 # [ 136.617860] nix-grpc-daemon[1403]: time=2026-09-21T12:10:47.163Z level=INFO msg="Uploading k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv (528B)"5816lb # [ 136.659781] niks3-server[1060]: 2026/09/21 12:10:47 INFO Registered completed upload object_key=nar/1h3fzlmfdwdiqr732wwrq7jnpw7n6p8f9qx1089c6g6x2i4csv75.nar.zst5817worker1 # [ 136.655284] nix-grpc-daemon[1403]: time=2026-09-21T12:10:47.202Z level=INFO msg="Uploading 1 narinfos"5818lb # [ 136.677386] niks3-server[1060]: 2026/09/21 12:10:47 INFO Received sign narinfos request method=POST path=/api/pending_closures/39/sign5819lb # [ 136.686930] niks3-server[1060]: 2026/09/21 12:10:47 INFO Signed narinfos id=39 count=15820lb # [ 136.692861] niks3-server[1060]: 2026/09/21 12:10:47 INFO Registered completed upload object_key=k3l35rf796jrnjns08398p1l8fcx178a.ls5821lb # [ 136.701499] niks3-server[1060]: 2026/09/21 12:10:47 INFO Received complete upload request method=POST path=/api/pending_closures/39/complete5822lb # [ 136.708975] niks3-server[1060]: 2026/09/21 12:10:47 INFO Completed upload id=395823worker1 # [ 136.687123] nix-grpc-daemon[1403]: time=2026-09-21T12:10:47.234Z level=INFO msg="Upload complete. (505ms)"5824lb # [ 136.711393] niks3-server[1060]: 2026/09/21 12:10:47 INFO Registered completed upload object_key=k3l35rf796jrnjns08398p1l8fcx178a.narinfo5825worker1 # [ 136.696300] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:47Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46256 duration_s=0 paths=1 nar_bytes_in=5285826worker1 # [ 136.708417] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:47Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:462565827worker1 # [ 136.718054] nix-daemon[842]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)5828worker1 # [ 136.721077] nix-daemon[842]: reaped child process 1477, status = succeeded5829client # building '/nix/store/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv' on lb:50051...5830client # farm-a> node-a: building k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv5831worker1 # [ 136.734787] nix-daemon[842]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)5832client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds5833client # farm-a> LOG-farm-a-during-drain5834worker1 # [ 138.171730] nix-daemon[842]: reaped child process 1491, status = succeeded5835worker1 # [ 138.344132] nix-daemon[842]: accepted connection from pid 1507, user nix-grpc-daemon (trusted)5836worker1 # [ 138.363335] nix-daemon[842]: reaped child process 1510, status = succeeded5837lb # [ 138.398467] niks3-server[1060]: 2026/09/21 12:10:48 INFO Received uploads request method=POST path=/api/pending_closures5838worker1 # [ 138.388437] nix-grpc-daemon[1403]: time=2026-09-21T12:10:48.934Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5839worker1 # [ 138.389969] nix-grpc-daemon[1403]: time=2026-09-21T12:10:48.934Z level=INFO msg="Uploading 42c24pndjip9nvdj8j25mkv0650griqf-farm-a (136B)"5840lb # [ 138.436642] niks3-server[1060]: 2026/09/21 12:10:48 INFO Registered completed upload object_key=log/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv5841lb # [ 138.451978] niks3-server[1060]: 2026/09/21 12:10:48 INFO Registered completed upload object_key=nar/0r0px9vw070lid3q4jhnmka11sk2y8kj26ysyvjy9dplavd7mdz4.nar.zst5842worker1 # [ 138.440629] nix-grpc-daemon[1403]: time=2026-09-21T12:10:48.987Z level=INFO msg="Uploading 1 narinfos"5843lb # [ 138.463925] niks3-server[1060]: 2026/09/21 12:10:49 INFO Received sign narinfos request method=POST path=/api/pending_closures/40/sign5844lb # [ 138.472671] niks3-server[1060]: 2026/09/21 12:10:49 INFO Signed narinfos id=40 count=15845lb # [ 138.476406] niks3-server[1060]: 2026/09/21 12:10:49 INFO Registered completed upload object_key=42c24pndjip9nvdj8j25mkv0650griqf.ls5846lb # [ 138.483595] niks3-server[1060]: 2026/09/21 12:10:49 INFO Received complete upload request method=POST path=/api/pending_closures/40/complete5847lb # [ 138.487844] niks3-server[1060]: 2026/09/21 12:10:49 INFO Completed upload id=405848worker1 # [ 138.466541] nix-grpc-daemon[1403]: time=2026-09-21T12:10:49.014Z level=INFO msg="Upload complete. (321ms)"5849lb # [ 138.490973] niks3-server[1060]: 2026/09/21 12:10:49 INFO Registered completed upload object_key=42c24pndjip9nvdj8j25mkv0650griqf.narinfo5850worker2 # [ 138.492488] nix-grpc-daemon[1563]: ts=2026-09-21T12:10:49Z level=debug event=want client=4 drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv cached=05851worker1 # [ 138.477034] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:49Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46256 duration_s=1 drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv assign_id=4 outputs=15852worker2 # [ 138.494397] nix-grpc-daemon[1563]: ts=2026-09-21T12:10:49Z level=debug event=assigned drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv worker=192.168.1.3:50051 clients=15853worker1 # [ 138.492995] nix-daemon[842]: reaped child process 1489, status = succeeded5854worker1 # [ 138.530922] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:462565855worker1 # [ 138.548241] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:49Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:462565856worker1 # [ 138.562706] nix-grpc-daemon[1403]: time=2026-09-21T12:10:49.110Z level=INFO msg="All 1 paths already cached"5857client # copying 1 paths to 'lb:50051'...5858worker1 # [ 138.573045] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:49Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:462565859client # copying path '/nix/store/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv' to 'lb:50051'...5860worker1 # [ 138.596271] nix-daemon[842]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)5861worker1 # [ 138.804117] nix-daemon[842]: accepted connection from pid 1514, user nix-grpc-daemon (trusted)5862worker1 # [ 138.821830] nix-daemon[842]: reaped child process 1517, status = succeeded5863worker1 # [ 138.998534] nix-daemon[842]: accepted connection from pid 1519, user nix-grpc-daemon (trusted)5864worker1 # [ 139.016624] nix-daemon[842]: reaped child process 1522, status = succeeded5865lb # [ 139.049444] niks3-server[1060]: 2026/09/21 12:10:49 INFO Received uploads request method=POST path=/api/pending_closures5866worker1 # [ 139.037914] nix-grpc-daemon[1403]: time=2026-09-21T12:10:49.585Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5867worker1 # [ 139.039440] nix-grpc-daemon[1403]: time=2026-09-21T12:10:49.585Z level=INFO msg="Uploading im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv (640B)"5868lb # [ 139.099725] niks3-server[1060]: 2026/09/21 12:10:49 INFO Received sign narinfos request method=POST path=/api/pending_closures/41/sign5869worker1 # [ 139.086058] nix-grpc-daemon[1403]: time=2026-09-21T12:10:49.633Z level=INFO msg="Uploading 1 narinfos"5870lb # [ 139.111536] niks3-server[1060]: 2026/09/21 12:10:49 INFO Registered completed upload object_key=nar/0rxmdnq61ka29n0b2as2hdbz91wi2ap17db890js8r5plhpg436g.nar.zst5871lb # [ 139.121382] niks3-server[1060]: 2026/09/21 12:10:49 INFO Signed narinfos id=41 count=15872worker1 # [ 139.106780] nix-grpc-daemon[1403]: time=2026-09-21T12:10:49.654Z level=INFO msg="Upload complete. (492ms)"5873lb # [ 139.128792] niks3-server[1060]: 2026/09/21 12:10:49 INFO Registered completed upload object_key=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l.ls5874lb # [ 139.133152] niks3-server[1060]: 2026/09/21 12:10:49 INFO Received complete upload request method=POST path=/api/pending_closures/41/complete5875lb # [ 139.137388] niks3-server[1060]: 2026/09/21 12:10:49 INFO Completed upload id=415876lb # [ 139.139340] niks3-server[1060]: 2026/09/21 12:10:49 INFO Registered completed upload object_key=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l.narinfo5877worker1 # [ 139.116504] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:49Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46256 duration_s=0 paths=1 nar_bytes_in=6405878worker1 # [ 139.125000] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:462565879worker1 # [ 139.135122] nix-daemon[842]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)5880worker1 # [ 139.139447] nix-daemon[842]: reaped child process 1512, status = succeeded5881client # building '/nix/store/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv' on lb:50051...5882worker1 # [ 139.152231] nix-daemon[842]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)5883client # farm-b> node-a: building im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv5884client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds5885client # farm-b> LOG-farm-b-during-drain5886worker1 # [ 140.199484] nix-daemon[842]: reaped child process 1526, status = succeeded5887worker1 # [ 140.380632] nix-daemon[842]: accepted connection from pid 1542, user nix-grpc-daemon (trusted)5888worker1 # [ 140.396605] nix-daemon[842]: reaped child process 1545, status = succeeded5889lb # [ 140.434260] niks3-server[1060]: 2026/09/21 12:10:50 INFO Received uploads request method=POST path=/api/pending_closures5890worker1 # [ 140.423226] nix-grpc-daemon[1403]: time=2026-09-21T12:10:50.970Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5891worker1 # [ 140.425020] nix-grpc-daemon[1403]: time=2026-09-21T12:10:50.970Z level=INFO msg="Uploading daks7y2c761g04qpzsjx25629wigr8js-farm-b (184B)"5892lb # [ 140.464403] niks3-server[1060]: 2026/09/21 12:10:51 INFO Registered completed upload object_key=log/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv5893lb # [ 140.477582] niks3-server[1060]: 2026/09/21 12:10:51 INFO Registered completed upload object_key=nar/0lnpxl34nrzp147jy5bafczkvxcl00hcf69qjhsljmv2ip9jisfh.nar.zst5894worker1 # [ 140.469049] nix-grpc-daemon[1403]: time=2026-09-21T12:10:51.016Z level=INFO msg="Uploading 1 narinfos"5895lb # [ 140.491294] niks3-server[1060]: 2026/09/21 12:10:51 INFO Received sign narinfos request method=POST path=/api/pending_closures/42/sign5896lb # [ 140.500220] niks3-server[1060]: 2026/09/21 12:10:51 INFO Signed narinfos id=42 count=15897lb # [ 140.502416] niks3-server[1060]: 2026/09/21 12:10:51 INFO Registered completed upload object_key=daks7y2c761g04qpzsjx25629wigr8js.ls5898lb # [ 140.508272] niks3-server[1060]: 2026/09/21 12:10:51 INFO Received complete upload request method=POST path=/api/pending_closures/42/complete5899lb # [ 140.516131] niks3-server[1060]: 2026/09/21 12:10:51 INFO Completed upload id=425900worker1 # [ 140.494856] nix-grpc-daemon[1403]: time=2026-09-21T12:10:51.042Z level=INFO msg="Upload complete. (332ms)"5901lb # [ 140.518726] niks3-server[1060]: 2026/09/21 12:10:51 INFO Registered completed upload object_key=daks7y2c761g04qpzsjx25629wigr8js.narinfo5902worker2 # [ 140.518094] nix-grpc-daemon[1563]: ts=2026-09-21T12:10:51Z level=debug event=want client=4 drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv cached=05903worker2 # [ 140.520117] nix-grpc-daemon[1563]: ts=2026-09-21T12:10:51Z level=debug event=assigned drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv worker=192.168.1.3:50051 clients=15904worker1 # [ 140.505131] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:51Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46256 duration_s=1 drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv assign_id=5 outputs=15905worker1 # [ 140.524435] nix-daemon[842]: reaped child process 1524, status = succeeded5906worker1 # [ 140.559316] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:51Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:462565907worker1 # [ 140.574442] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:51Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:462565908worker1 # [ 140.589854] nix-grpc-daemon[1403]: time=2026-09-21T12:10:51.136Z level=INFO msg="All 2 paths already cached"5909client # copying 1 paths to 'lb:50051'...5910worker1 # [ 140.602153] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:51Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:462565911client # copying path '/nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv' to 'lb:50051'...5912worker1 # [ 140.625644] nix-daemon[842]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)5913worker1 # [ 140.828152] nix-daemon[842]: accepted connection from pid 1549, user nix-grpc-daemon (trusted)5914worker1 # [ 140.847373] nix-daemon[842]: reaped child process 1552, status = succeeded5915worker1 # [ 141.025884] nix-daemon[842]: accepted connection from pid 1554, user nix-grpc-daemon (trusted)5916worker1 # [ 141.042050] nix-daemon[842]: reaped child process 1557, status = succeeded5917lb # [ 141.075506] niks3-server[1060]: 2026/09/21 12:10:51 INFO Received uploads request method=POST path=/api/pending_closures5918worker1 # [ 141.063645] nix-grpc-daemon[1403]: time=2026-09-21T12:10:51.611Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5919worker1 # [ 141.065360] nix-grpc-daemon[1403]: time=2026-09-21T12:10:51.611Z level=INFO msg="Uploading lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv (768B)"5920lb # [ 141.104350] niks3-server[1060]: 2026/09/21 12:10:51 INFO Registered completed upload object_key=nar/066jhl91shf8ar6h3cqqb855j5cswl7170wrg5lzf3fhpka0xywn.nar.zst5921lb # [ 141.113714] niks3-server[1060]: 2026/09/21 12:10:51 INFO Received sign narinfos request method=POST path=/api/pending_closures/43/sign5922worker1 # [ 141.098192] nix-grpc-daemon[1403]: time=2026-09-21T12:10:51.645Z level=INFO msg="Uploading 1 narinfos"5923lb # [ 141.125621] niks3-server[1060]: 2026/09/21 12:10:51 INFO Signed narinfos id=43 count=15924lb # [ 141.132311] niks3-server[1060]: 2026/09/21 12:10:51 INFO Registered completed upload object_key=lbk8wy49c881163x07gb8z669j2fg1ff.ls5925lb # [ 141.142585] niks3-server[1060]: 2026/09/21 12:10:51 INFO Received complete upload request method=POST path=/api/pending_closures/43/complete5926lb # [ 141.147353] niks3-server[1060]: 2026/09/21 12:10:51 INFO Completed upload id=435927worker1 # [ 141.126022] nix-grpc-daemon[1403]: time=2026-09-21T12:10:51.673Z level=INFO msg="Upload complete. (482ms)"5928lb # [ 141.150027] niks3-server[1060]: 2026/09/21 12:10:51 INFO Registered completed upload object_key=lbk8wy49c881163x07gb8z669j2fg1ff.narinfo5929worker1 # [ 141.132312] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:51Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46256 duration_s=0 paths=1 nar_bytes_in=7685930worker1 # [ 141.141200] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:51Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:462565931worker1 # [ 141.151402] nix-daemon[842]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)5932worker1 # [ 141.155733] nix-daemon[842]: reaped child process 1547, status = succeeded5933client # building '/nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv' on lb:50051...5934worker1 # [ 141.166601] nix-daemon[842]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)5935client # farm-top> node-a: building lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv5936client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds5937client # farm-top> LOG-farm-top-during-drain5938worker1 # [ 142.810357] nix-daemon[842]: reaped child process 1561, status = succeeded5939worker1 # [ 143.005110] nix-daemon[842]: accepted connection from pid 1577, user nix-grpc-daemon (trusted)5940worker1 # [ 143.023922] nix-daemon[842]: reaped child process 1580, status = succeeded5941lb # [ 143.058921] niks3-server[1060]: 2026/09/21 12:10:53 INFO Received uploads request method=POST path=/api/pending_closures5942worker1 # [ 143.049829] nix-grpc-daemon[1403]: time=2026-09-21T12:10:53.595Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5943worker1 # [ 143.051482] nix-grpc-daemon[1403]: time=2026-09-21T12:10:53.595Z level=INFO msg="Uploading xdwmx2f4n5n8bggglniz2n35any0xz4w-farm-top (240B)"5944lb # [ 143.100136] niks3-server[1060]: 2026/09/21 12:10:53 INFO Registered completed upload object_key=log/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv5945worker1 # [ 143.090574] nix-grpc-daemon[1403]: time=2026-09-21T12:10:53.638Z level=INFO msg="Uploading 1 narinfos"5946lb # [ 143.111767] niks3-server[1060]: 2026/09/21 12:10:53 INFO Registered completed upload object_key=nar/1bpy9ga4n2446pary6msidbs8mnil1a34jn8p8yl2scvdg691mz2.nar.zst5947lb # [ 143.123374] niks3-server[1060]: 2026/09/21 12:10:53 INFO Received sign narinfos request method=POST path=/api/pending_closures/44/sign5948lb # [ 143.129641] niks3-server[1060]: 2026/09/21 12:10:53 INFO Signed narinfos id=44 count=15949lb # [ 143.136662] niks3-server[1060]: 2026/09/21 12:10:53 INFO Registered completed upload object_key=xdwmx2f4n5n8bggglniz2n35any0xz4w.ls5950worker1 # [ 143.119885] nix-grpc-daemon[1403]: time=2026-09-21T12:10:53.667Z level=INFO msg="Upload complete. (340ms)"5951lb # [ 143.142518] niks3-server[1060]: 2026/09/21 12:10:53 INFO Received complete upload request method=POST path=/api/pending_closures/44/complete5952lb # [ 143.149031] niks3-server[1060]: 2026/09/21 12:10:53 INFO Completed upload id=445953lb # [ 143.152675] niks3-server[1060]: 2026/09/21 12:10:53 INFO Registered completed upload object_key=xdwmx2f4n5n8bggglniz2n35any0xz4w.narinfo5954worker1 # [ 143.133054] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:53Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46256 duration_s=1 drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv assign_id=6 outputs=15955worker1 # [ 143.146561] nix-daemon[842]: reaped child process 1559, status = succeeded5956client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag during-drain >&2, in 7.44 seconds)5957client: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag during-drain outPath5958client: (finished: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag during-drain outPath, in 0.23 seconds)5959worker2: must succeed: pkill -f 'read -t [9]0 x'5960worker2: (finished: must succeed: pkill -f 'read -t [9]0 x', in 0.06 seconds)5961client: waiting for success: ! systemctl is-active sw5962worker2 # [ 143.709585] nix-daemon[863]: reaped child process 1710, status = succeeded5963worker2 # [ 143.902103] nix-daemon[863]: accepted connection from pid 1888, user nix-grpc-daemon (trusted)5964worker2 # [ 143.919498] nix-daemon[863]: reaped child process 1891, status = succeeded5965lb # [ 143.941372] niks3-server[1060]: 2026/09/21 12:10:54 INFO Received uploads request method=POST path=/api/pending_closures5966worker2 # [ 143.941642] nix-grpc-daemon[1646]: time=2026-09-21T12:10:54.484Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5967worker2 # [ 143.943531] nix-grpc-daemon[1646]: time=2026-09-21T12:10:54.484Z level=INFO msg="Uploading qr8vyl2zl3nxnin3vw41c6z35vj7087f-slow-sw (120B)"5968lb # [ 143.971502] niks3-server[1060]: 2026/09/21 12:10:54 INFO Registered completed upload object_key=log/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv5969lb # [ 143.982994] niks3-server[1060]: 2026/09/21 12:10:54 INFO Registered completed upload object_key=nar/0vi6cx31wc05wns04pr5vr2bak9iib1idndvy6gik6c2rz9q8c19.nar.zst5970worker2 # [ 143.995704] nix-grpc-daemon[1646]: time=2026-09-21T12:10:54.538Z level=INFO msg="Uploading 1 narinfos"5971lb # [ 143.998700] niks3-server[1060]: 2026/09/21 12:10:54 INFO Received sign narinfos request method=POST path=/api/pending_closures/45/sign5972lb # [ 144.011164] niks3-server[1060]: 2026/09/21 12:10:54 INFO Signed narinfos id=45 count=15973lb # [ 144.017397] niks3-server[1060]: 2026/09/21 12:10:54 INFO Registered completed upload object_key=qr8vyl2zl3nxnin3vw41c6z35vj7087f.ls5974lb # [ 144.024136] niks3-server[1060]: 2026/09/21 12:10:54 INFO Received complete upload request method=POST path=/api/pending_closures/45/complete5975lb # [ 144.032524] niks3-server[1060]: 2026/09/21 12:10:54 INFO Completed upload id=455976worker2 # [ 144.022464] nix-grpc-daemon[1646]: time=2026-09-21T12:10:54.566Z level=INFO msg="Upload complete. (349ms)"5977lb # [ 144.035209] niks3-server[1060]: 2026/09/21 12:10:54 INFO Registered completed upload object_key=qr8vyl2zl3nxnin3vw41c6z35vj7087f.narinfo5978worker2 # [ 144.037199] nix-grpc-daemon[1563]: ts=2026-09-21T12:10:54Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34876 duration_s=13 drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv assign_id=3 outputs=15979worker2 # [ 144.054452] nix-daemon[863]: reaped child process 1708, status = succeeded5980client # [ 144.092965] systemd[1]: sw.service: Deactivated successfully.5981client # [ 144.097367] systemd[1]: sw.service: Consumed 179ms CPU time over 14.774s wall clock time, 41.2M memory peak, 7.4K incoming IP traffic, 8.8K outgoing IP traffic.5982client: (finished: waiting for success: ! systemctl is-active sw, in 1.11 seconds)5983client: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }5984client: (finished: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }, in 0.05 seconds)5985worker2: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service5986worker2: (finished: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service, in 0.07 seconds)5987lb: waiting until exactly no endpoints of aarch64-linux are unhealthy5988lb: must succeed: curl -sf localhost:9901/clusters5989worker2 # [ 144.791812] nix-grpc-daemon[1563]: ts=2026-09-21T12:10:55Z level=info event=signal_exit5990lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5991worker2 # [ 145.012168] nix-grpc-daemon[1563]: ts=2026-09-21T12:10:55Z level=info event=scheduler_yield5992lb # [ 145.025349] niks3-server[1060]: 2026/09/21 12:10:55 INFO lead: released remote=192.168.1.4:440365993worker1 # [ 145.006581] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:55Z level=info event=scheduler_restarting addr=lb:500515994worker2 # [ 145.025908] nix-grpc-daemon[1563]: ts=2026-09-21T12:10:55Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:348665995worker2 # [ 145.034100] nix-grpc-daemon[1563]: ts=2026-09-21T12:10:55Z level=info event=worker_gone addr=192.168.1.3:500515996worker2 # [ 145.039500] nix-grpc-daemon[1563]: ts=2026-09-21T12:10:55Z level=info event=scheduler_restarting addr=lb:500515997worker2 # [ 145.043634] nix-grpc-daemon[1563]: ts=2026-09-21T12:10:55Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:348665998worker2 # [ 145.048191] nix-grpc-daemon[1563]: ts=2026-09-21T12:10:55Z level=info event=worker_gone addr=192.168.1.4:500515999worker1 # [ 145.206576] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:55Z level=info event=scheduler_connected addr=lb:500516000worker1 # [ 145.211932] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6001worker2 # [ 145.233093] nix-grpc-daemon[1563]: ts=2026-09-21T12:10:55Z level=info event=scheduler_connected addr=lb:500516002worker2 # [ 145.238250] nix-grpc-daemon[1563]: ts=2026-09-21T12:10:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6003worker1 # [ 145.412792] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:55Z level=info event=scheduler_connected addr=lb:500516004worker1 # [ 145.417800] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6005worker2 # [ 145.439240] nix-grpc-daemon[1563]: ts=2026-09-21T12:10:55Z level=info event=scheduler_connected addr=lb:500516006worker2 # [ 145.444741] nix-grpc-daemon[1563]: ts=2026-09-21T12:10:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6007worker1 # [ 145.618574] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:56Z level=info event=scheduler_connected addr=lb:500516008worker1 # [ 145.622680] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6009worker2 # [ 145.646560] nix-grpc-daemon[1563]: ts=2026-09-21T12:10:56Z level=info event=scheduler_connected addr=lb:500516010worker2 # [ 145.652263] nix-grpc-daemon[1563]: ts=2026-09-21T12:10:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6011lb: must succeed: curl -sf localhost:9901/clusters6012worker1 # [ 145.823582] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:56Z level=info event=scheduler_connected addr=lb:500516013worker1 # [ 145.829050] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6014worker2 # [ 145.848978] nix-grpc-daemon[1563]: ts=2026-09-21T12:10:56Z level=info event=scheduler_connected addr=lb:500516015worker2 # [ 145.854358] nix-grpc-daemon[1563]: ts=2026-09-21T12:10:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6016lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds)6017worker2 # [ 146.017347] nix-grpc-daemon[1563]: ts=2026-09-21T12:10:56Z level=info event=scheduler_restarting6018worker1 # [ 146.031486] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:56Z level=info event=scheduler_connected addr=lb:500516019worker1 # [ 146.035427] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6020worker2 # [ 146.035976] nix-grpc-daemon[1563]: E0000 00:00:1789992656.579311 1608 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 19, started true, shutdown true6021worker2 # [ 146.053468] nix-grpc-daemon[1563]: E0000 00:00:1789992656.579537 1608 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 9, started true, shutdown true6022worker2 # [ 146.080842] nix-daemon[863]: reaped child process 1617, status = succeeded6023worker2 # [ 146.082908] nix-grpc-daemon[1563]: ts=2026-09-21T12:10:56Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"6024worker1 # [ 146.232096] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:56Z level=info event=scheduler_connected addr=lb:500516025worker1 # [ 146.237217] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6026worker1 # [ 146.437854] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:56Z level=info event=scheduler_connected addr=lb:500516027worker1 # [ 146.442553] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6028worker1 # [ 146.643482] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:57Z level=info event=scheduler_connected addr=lb:500516029worker1 # [ 146.648649] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6030worker2 # [ 146.831283] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.6031worker2 # [ 146.831703] systemd[1]: nix-grpc-daemon.service: Consumed 878ms CPU time over 28.427s wall clock time, 34.6M memory peak, 25.9K incoming IP traffic, 36K outgoing IP traffic.6032worker1 # [ 146.849460] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:57Z level=info event=scheduler_connected addr=lb:500516033worker1 # [ 146.854324] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6034lb: must succeed: curl -sf localhost:9901/clusters6035lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6036worker1 # [ 147.056808] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:57Z level=info event=scheduler_connected addr=lb:500516037worker1 # [ 147.060892] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6038worker1 # [ 147.257672] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:57Z level=info event=scheduler_connected addr=lb:500516039worker1 # [ 147.262845] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6040worker1 # [ 147.463693] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:58Z level=info event=scheduler_connected addr=lb:500516041worker1 # [ 147.469007] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6042worker1 # [ 147.669747] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:58Z level=info event=scheduler_connected addr=lb:500516043worker1 # [ 147.674897] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6044worker2 # [ 147.817391] systemd[1]: Starting scheduler-token.service...6045worker1 # [ 147.875842] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:58Z level=info event=scheduler_connected addr=lb:500516046worker1 # [ 147.881062] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6047worker2 # [ 147.923212] systemd[1]: scheduler-token.service: Deactivated successfully.6048worker2 # [ 147.928736] systemd[1]: Finished scheduler-token.service.6049worker2 # [ 147.930110] systemd[1]: scheduler-token.service: Consumed 51ms CPU time over 108ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.6050worker2 # [ 147.937800] systemd[1]: Starting Nix worker-protocol over gRPC...6051lb: must succeed: curl -sf localhost:9901/clusters6052lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6053worker1 # [ 148.083381] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:58Z level=info event=scheduler_connected addr=lb:500516054worker1 # [ 148.087586] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6055worker2 # [ 148.121218] nix-grpc-daemon[1906]: ts=2026-09-21T12:10:58Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6056worker2 # [ 148.123760] nix-grpc-daemon[1906]: ts=2026-09-21T12:10:58Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6057worker2 # [ 148.127929] nix-grpc-daemon[1906]: ts=2026-09-21T12:10:58Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=16058worker2 # [ 148.136071] nix-grpc-daemon[1906]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6059worker2 # [ 148.137496] nix-grpc-daemon[1906]: E0000 00:00:1789992658.678106 1906 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6060worker2 # [ 148.148744] nix-grpc-daemon[1906]: E0000 00:00:1789992658.684300 1906 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6061worker2 # [ 148.153808] nix-grpc-daemon[1906]: ts=2026-09-21T12:10:58Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6062worker2 # [ 148.155871] systemd[1]: Started Nix worker-protocol over gRPC.6063worker2 # [ 148.159161] nix-grpc-daemon[1906]: ts=2026-09-21T12:10:58Z level=info event=scheduler_take_over6064lb # [ 148.169343] niks3-server[1060]: 2026/09/21 12:10:58 INFO lead: acquired remote=192.168.1.4:520046065worker2 # [ 148.164074] nix-grpc-daemon[1906]: E0000 00:00:1789992658.706258 1946 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6066worker2 # [ 148.195757] nix-grpc-daemon[1906]: E0000 00:00:1789992658.724309 1946 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6067worker2 # [ 148.198916] nix-grpc-daemon[1906]: E0000 00:00:1789992658.724455 1946 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6068worker2 # [ 148.201794] nix-grpc-daemon[1906]: E0000 00:00:1789992658.739243 1946 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6069worker2 # [ 148.211368] nix-grpc-daemon[1906]: ts=2026-09-21T12:10:58Z level=info event=scheduler_connected addr=lb:500516070worker2 # [ 148.218628] nix-grpc-daemon[1906]: ts=2026-09-21T12:10:58Z level=info event=oidc_jwks provider=mock keys=16071worker2 # [ 148.219958] nix-grpc-daemon[1906]: ts=2026-09-21T12:10:58Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:408346072worker2 # [ 148.221925] nix-grpc-daemon[1906]: ts=2026-09-21T12:10:58Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=06073worker1 # [ 148.283956] nix-grpc-daemon[1311]: ts=2026-09-21T12:10:58Z level=info event=scheduler_connected addr=lb:500516074worker2 # [ 148.296804] nix-grpc-daemon[1906]: ts=2026-09-21T12:10:58Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:408346075worker2 # [ 148.300968] nix-grpc-daemon[1906]: ts=2026-09-21T12:10:58Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=06076lb: must succeed: curl -sf localhost:9901/clusters6077lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6078lb: (finished: waiting until exactly no endpoints of aarch64-linux are unhealthy, in 4.42 seconds)6079lb: must succeed: curl -sf localhost:9901/clusters6080lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)6081worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6082worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)6083(finished: subtest: a deploy mid-build drains: the build finishes, the new generation takes over, in 20.05 seconds)6084subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does6085client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&16086worker2 # [ 149.548648] nix-grpc-daemon[1906]: ts=2026-09-21T12:11:00Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:408506087worker2 # [ 149.563217] nix-daemon[863]: accepted connection from pid 1906, user nix-grpc-daemon (trusted)6088worker1 # [ 149.574045] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:00Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:462566089worker2 # [ 149.633954] nix-grpc-daemon[1906]: ts=2026-09-21T12:11:00Z level=debug event=want client=1 drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv cached=06090worker2 # [ 149.640006] nix-grpc-daemon[1906]: ts=2026-09-21T12:11:00Z level=debug event=assigned drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv worker=192.168.1.3:50051 clients=16091worker2 # [ 149.646060] nix-grpc-daemon[1906]: ts=2026-09-21T12:11:00Z level=debug event=want client=1 drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv cached=06092worker2 # [ 149.651382] nix-grpc-daemon[1906]: ts=2026-09-21T12:11:00Z level=debug event=want client=1 drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv cached=06093worker1 # [ 149.671163] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:462566094worker1 # [ 149.687357] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:00Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:462566095worker1 # [ 149.702200] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:00Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:462566096worker1 # [ 149.717400] nix-daemon[842]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6097worker1 # [ 149.933593] nix-daemon[842]: accepted connection from pid 1584, user nix-grpc-daemon (trusted)6098worker1 # [ 149.952040] nix-daemon[842]: reaped child process 1587, status = succeeded6099worker1 # [ 150.129569] nix-daemon[842]: accepted connection from pid 1589, user nix-grpc-daemon (trusted)6100worker1 # [ 150.146425] nix-daemon[842]: reaped child process 1592, status = succeeded6101lb # [ 150.181748] niks3-server[1060]: 2026/09/21 12:11:00 INFO Received uploads request method=POST path=/api/pending_closures6102worker1 # [ 150.169455] nix-grpc-daemon[1403]: time=2026-09-21T12:11:00.716Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6103worker1 # [ 150.170960] nix-grpc-daemon[1403]: time=2026-09-21T12:11:00.716Z level=INFO msg="Uploading 3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv (512B)"6104lb # [ 150.213194] niks3-server[1060]: 2026/09/21 12:11:00 INFO Received sign narinfos request method=POST path=/api/pending_closures/46/sign6105worker1 # [ 150.199429] nix-grpc-daemon[1403]: time=2026-09-21T12:11:00.747Z level=INFO msg="Uploading 1 narinfos"6106lb # [ 150.225777] niks3-server[1060]: 2026/09/21 12:11:00 INFO Signed narinfos id=46 count=16107lb # [ 150.234446] niks3-server[1060]: 2026/09/21 12:11:00 INFO Registered completed upload object_key=3223c4sl4ncpldbvw558vqwddijwzfpd.ls6108lb # [ 150.245138] niks3-server[1060]: 2026/09/21 12:11:00 INFO Registered completed upload object_key=nar/0zxfsmqijfc2dkglls3yp1x3w0w1na7vx39fxc124zddjzpaygd8.nar.zst6109lb # [ 150.250973] niks3-server[1060]: 2026/09/21 12:11:00 INFO Received complete upload request method=POST path=/api/pending_closures/46/complete6110lb # [ 150.254725] niks3-server[1060]: 2026/09/21 12:11:00 INFO Registered completed upload object_key=3223c4sl4ncpldbvw558vqwddijwzfpd.narinfo6111lb # [ 150.259039] niks3-server[1060]: 2026/09/21 12:11:00 INFO Completed upload id=466112worker1 # [ 150.237442] nix-grpc-daemon[1403]: time=2026-09-21T12:11:00.785Z level=INFO msg="Upload complete. (502ms)"6113worker1 # [ 150.246120] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:00Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46256 duration_s=0 paths=1 nar_bytes_in=5126114worker1 # [ 150.259979] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:462566115worker1 # [ 150.266779] nix-daemon[842]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6116worker1 # [ 150.271628] nix-daemon[842]: reaped child process 1582, status = succeeded6117worker1 # [ 150.283378] nix-daemon[842]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6118lb # [ 150.670281] envoy[910]: conn peer=[2001:db8:1::1]:48924 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9668 tx=4122 ms=71276119lb # [ 150.675537] envoy[910]: conn peer=[2001:db8:1::1]:54100 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3025 tx=1466 ms=144846120lb # [ 150.680485] envoy[910]: conn peer=[2001:db8:1::4]:50256 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=4556 tx=539 ms=272336121worker1 # [ 151.731744] nix-daemon[842]: reaped child process 1596, status = succeeded6122worker1 # [ 151.918589] nix-daemon[842]: accepted connection from pid 1612, user nix-grpc-daemon (trusted)6123worker1 # [ 151.937493] nix-daemon[842]: reaped child process 1615, status = succeeded6124lb # [ 151.972653] niks3-server[1060]: 2026/09/21 12:11:02 INFO Received uploads request method=POST path=/api/pending_closures6125worker1 # [ 151.960507] nix-grpc-daemon[1403]: time=2026-09-21T12:11:02.508Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6126worker1 # [ 151.964868] nix-grpc-daemon[1403]: time=2026-09-21T12:11:02.509Z level=INFO msg="Uploading dla3w0zf02z5jhccwa27i85bbgzrrsvb-farm-a (128B)"6127lb # [ 152.005714] niks3-server[1060]: 2026/09/21 12:11:02 INFO Registered completed upload object_key=log/3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv6128lb # [ 152.019042] niks3-server[1060]: 2026/09/21 12:11:02 INFO Registered completed upload object_key=nar/05i7ka1qa7vmhiys96ln3pp3nj3rfdlslnbydrxdqj40cmvmqcwf.nar.zst6129lb # [ 152.033325] niks3-server[1060]: 2026/09/21 12:11:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/47/sign6130worker1 # [ 152.016746] nix-grpc-daemon[1403]: time=2026-09-21T12:11:02.564Z level=INFO msg="Uploading 1 narinfos"6131lb # [ 152.043242] niks3-server[1060]: 2026/09/21 12:11:02 INFO Signed narinfos id=47 count=16132lb # [ 152.049305] niks3-server[1060]: 2026/09/21 12:11:02 INFO Registered completed upload object_key=dla3w0zf02z5jhccwa27i85bbgzrrsvb.ls6133lb # [ 152.056715] niks3-server[1060]: 2026/09/21 12:11:02 INFO Received complete upload request method=POST path=/api/pending_closures/47/complete6134lb # [ 152.064912] niks3-server[1060]: 2026/09/21 12:11:02 INFO Completed upload id=476135worker1 # [ 152.043605] nix-grpc-daemon[1403]: time=2026-09-21T12:11:02.591Z level=INFO msg="Upload complete. (345ms)"6136lb # [ 152.068233] niks3-server[1060]: 2026/09/21 12:11:02 INFO Registered completed upload object_key=dla3w0zf02z5jhccwa27i85bbgzrrsvb.narinfo6137worker2 # [ 152.069205] nix-grpc-daemon[1906]: ts=2026-09-21T12:11:02Z level=debug event=assigned drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv worker=192.168.1.3:50051 clients=16138worker1 # [ 152.057166] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:02Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46256 duration_s=1 drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv assign_id=1 outputs=16139worker2 # [ 152.080363] nix-grpc-daemon[1906]: ts=2026-09-21T12:11:02Z level=debug event=want client=1 drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv cached=06140worker1 # [ 152.072955] nix-daemon[842]: reaped child process 1594, status = succeeded6141worker1 # [ 152.111740] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:462566142worker1 # [ 152.126711] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:02Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:462566143worker1 # [ 152.141451] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:02Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:462566144worker1 # [ 152.155067] nix-daemon[842]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6145worker1 # [ 152.354928] nix-daemon[842]: accepted connection from pid 1619, user nix-grpc-daemon (trusted)6146worker1 # [ 152.372412] nix-daemon[842]: reaped child process 1622, status = succeeded6147worker1 # [ 152.546370] nix-daemon[842]: accepted connection from pid 1624, user nix-grpc-daemon (trusted)6148worker1 # [ 152.562269] nix-daemon[842]: reaped child process 1627, status = succeeded6149lb # [ 152.595261] niks3-server[1060]: 2026/09/21 12:11:03 INFO Received uploads request method=POST path=/api/pending_closures6150worker1 # [ 152.582390] nix-grpc-daemon[1403]: time=2026-09-21T12:11:03.129Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6151worker1 # [ 152.583855] nix-grpc-daemon[1403]: time=2026-09-21T12:11:03.129Z level=INFO msg="Uploading pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv (512B)"6152lb # [ 152.624204] niks3-server[1060]: 2026/09/21 12:11:03 INFO Registered completed upload object_key=nar/0g5bxr683biqh358mkd9xrclwllan4cgnv4pzijas586xj68wwx0.nar.zst6153lb # [ 152.635753] niks3-server[1060]: 2026/09/21 12:11:03 INFO Received sign narinfos request method=POST path=/api/pending_closures/48/sign6154worker1 # [ 152.620586] nix-grpc-daemon[1403]: time=2026-09-21T12:11:03.168Z level=INFO msg="Uploading 1 narinfos"6155lb # [ 152.647100] niks3-server[1060]: 2026/09/21 12:11:03 INFO Signed narinfos id=48 count=16156lb # [ 152.653310] niks3-server[1060]: 2026/09/21 12:11:03 INFO Registered completed upload object_key=pplpzcnjfn295nm16cg0iihpfc9iffry.ls6157worker1 # [ 152.641064] nix-grpc-daemon[1403]: time=2026-09-21T12:11:03.188Z level=INFO msg="Upload complete. (470ms)"6158lb # [ 152.662923] niks3-server[1060]: 2026/09/21 12:11:03 INFO Received complete upload request method=POST path=/api/pending_closures/48/complete6159lb # [ 152.667722] niks3-server[1060]: 2026/09/21 12:11:03 INFO Completed upload id=486160lb # [ 152.670703] niks3-server[1060]: 2026/09/21 12:11:03 INFO Registered completed upload object_key=pplpzcnjfn295nm16cg0iihpfc9iffry.narinfo6161worker1 # [ 152.648277] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:03Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46256 duration_s=0 paths=1 nar_bytes_in=5126162worker1 # [ 152.659733] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:462566163worker1 # [ 152.664272] nix-daemon[842]: reaped child process 1617, status = succeeded6164worker1 # [ 152.669766] nix-daemon[842]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6165worker1 # [ 152.683055] nix-daemon[842]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6166worker1 # [ 153.922794] nix-daemon[842]: reaped child process 1631, status = succeeded6167worker1 # [ 154.105027] nix-daemon[842]: accepted connection from pid 1647, user nix-grpc-daemon (trusted)6168worker1 # [ 154.120215] nix-daemon[842]: reaped child process 1650, status = succeeded6169lb # [ 154.158007] niks3-server[1060]: 2026/09/21 12:11:04 INFO Received uploads request method=POST path=/api/pending_closures6170worker1 # [ 154.151135] nix-grpc-daemon[1403]: time=2026-09-21T12:11:04.695Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6171worker1 # [ 154.152755] nix-grpc-daemon[1403]: time=2026-09-21T12:11:04.695Z level=INFO msg="Uploading fhc1zkcygpzppf0bj5fr2k85lf7p87v9-farm-a (128B)"6172lb # [ 154.214091] niks3-server[1060]: 2026/09/21 12:11:04 INFO Registered completed upload object_key=nar/09na6djn66dpys8h08gqw39p0vw7vxqsdzbiw55rknyna8s239bm.nar.zst6173worker1 # [ 154.208533] nix-grpc-daemon[1403]: time=2026-09-21T12:11:04.756Z level=INFO msg="Uploading 1 narinfos"6174lb # [ 154.225861] niks3-server[1060]: 2026/09/21 12:11:04 INFO Registered completed upload object_key=log/pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv6175lb # [ 154.234080] niks3-server[1060]: 2026/09/21 12:11:04 INFO Received sign narinfos request method=POST path=/api/pending_closures/49/sign6176lb # [ 154.240008] niks3-server[1060]: 2026/09/21 12:11:04 INFO Signed narinfos id=49 count=16177lb # [ 154.246331] niks3-server[1060]: 2026/09/21 12:11:04 INFO Registered completed upload object_key=fhc1zkcygpzppf0bj5fr2k85lf7p87v9.ls6178worker1 # [ 154.224366] nix-grpc-daemon[1403]: time=2026-09-21T12:11:04.772Z level=INFO msg="Upload complete. (330ms)"6179lb # [ 154.248942] niks3-server[1060]: 2026/09/21 12:11:04 INFO Received complete upload request method=POST path=/api/pending_closures/49/complete6180lb # [ 154.252656] niks3-server[1060]: 2026/09/21 12:11:04 INFO Completed upload id=496181lb # [ 154.254289] niks3-server[1060]: 2026/09/21 12:11:04 INFO Registered completed upload object_key=fhc1zkcygpzppf0bj5fr2k85lf7p87v9.narinfo6182worker1 # [ 154.236896] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:04Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46256 duration_s=1 drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv assign_id=2 outputs=16183worker2 # [ 154.249155] nix-grpc-daemon[1906]: ts=2026-09-21T12:11:04Z level=debug event=assigned drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv worker=192.168.1.3:50051 clients=16184worker1 # [ 154.252650] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:04Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:462566185worker2 # [ 154.262696] nix-grpc-daemon[1906]: ts=2026-09-21T12:11:04Z level=debug event=want client=1 drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv cached=06186worker1 # [ 154.258421] nix-daemon[842]: reaped child process 1629, status = succeeded6187worker1 # [ 154.265513] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:04Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:462566188worker1 # [ 154.277537] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:04Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:462566189worker1 # [ 154.289722] nix-daemon[842]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6190worker1 # [ 154.477780] nix-daemon[842]: accepted connection from pid 1654, user nix-grpc-daemon (trusted)6191worker1 # [ 154.494568] nix-daemon[842]: reaped child process 1657, status = succeeded6192worker1 # [ 154.664207] nix-daemon[842]: accepted connection from pid 1659, user nix-grpc-daemon (trusted)6193worker1 # [ 154.678993] nix-daemon[842]: reaped child process 1662, status = succeeded6194lb # [ 154.716570] niks3-server[1060]: 2026/09/21 12:11:05 INFO Received uploads request method=POST path=/api/pending_closures6195worker1 # [ 154.706525] nix-grpc-daemon[1403]: time=2026-09-21T12:11:05.254Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6196worker1 # [ 154.707974] nix-grpc-daemon[1403]: time=2026-09-21T12:11:05.254Z level=INFO msg="Uploading sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv (512B)"6197lb # [ 154.750089] niks3-server[1060]: 2026/09/21 12:11:05 INFO Registered completed upload object_key=nar/1fxz200fmpbckafrn03b2snx9knqpbwf8rwqlbvg68356qj7aj5w.nar.zst6198lb # [ 154.759827] niks3-server[1060]: 2026/09/21 12:11:05 INFO Received sign narinfos request method=POST path=/api/pending_closures/50/sign6199worker1 # [ 154.744974] nix-grpc-daemon[1403]: time=2026-09-21T12:11:05.292Z level=INFO msg="Uploading 1 narinfos"6200lb # [ 154.771412] niks3-server[1060]: 2026/09/21 12:11:05 INFO Signed narinfos id=50 count=16201lb # [ 154.777393] niks3-server[1060]: 2026/09/21 12:11:05 INFO Registered completed upload object_key=sy43wp9v52pni1wms6q0p3qybpc2pwad.ls6202lb # [ 154.787811] niks3-server[1060]: 2026/09/21 12:11:05 INFO Received complete upload request method=POST path=/api/pending_closures/50/complete6203lb # [ 154.792721] niks3-server[1060]: 2026/09/21 12:11:05 INFO Completed upload id=506204worker1 # [ 154.771234] nix-grpc-daemon[1403]: time=2026-09-21T12:11:05.319Z level=INFO msg="Upload complete. (469ms)"6205lb # [ 154.795531] niks3-server[1060]: 2026/09/21 12:11:05 INFO Registered completed upload object_key=sy43wp9v52pni1wms6q0p3qybpc2pwad.narinfo6206worker1 # [ 154.776934] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:05Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46256 duration_s=0 paths=1 nar_bytes_in=5126207worker1 # [ 154.787792] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:05Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:462566208worker1 # [ 154.791794] nix-daemon[842]: reaped child process 1652, status = succeeded6209worker1 # [ 154.798064] nix-daemon[842]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6210worker1 # [ 154.808720] nix-daemon[842]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6211worker1 # [ 155.637567] nix-daemon[842]: reaped child process 1666, status = succeeded6212worker1 # [ 155.804322] nix-daemon[842]: accepted connection from pid 1682, user nix-grpc-daemon (trusted)6213worker1 # [ 155.819176] nix-daemon[842]: reaped child process 1685, status = succeeded6214lb # [ 155.856719] niks3-server[1060]: 2026/09/21 12:11:06 INFO Received uploads request method=POST path=/api/pending_closures6215worker1 # [ 155.844512] nix-grpc-daemon[1403]: time=2026-09-21T12:11:06.392Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6216worker1 # [ 155.847047] nix-grpc-daemon[1403]: time=2026-09-21T12:11:06.393Z level=INFO msg="Uploading g5laypvfk2469p8binw3mksk2hv7zhnh-farm-a (128B)"6217lb # [ 155.892222] niks3-server[1060]: 2026/09/21 12:11:06 INFO Registered completed upload object_key=nar/1j2671k9mycq8lxyfx5k6a66d5narfqw2wqwlpwxgwzi93vnvbdn.nar.zst6218lb # [ 155.907388] niks3-server[1060]: 2026/09/21 12:11:06 INFO Registered completed upload object_key=log/sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv6219worker1 # [ 155.896706] nix-grpc-daemon[1403]: time=2026-09-21T12:11:06.444Z level=INFO msg="Uploading 1 narinfos"6220lb # [ 155.919776] niks3-server[1060]: 2026/09/21 12:11:06 INFO Received sign narinfos request method=POST path=/api/pending_closures/51/sign6221lb # [ 155.932961] niks3-server[1060]: 2026/09/21 12:11:06 INFO Signed narinfos id=51 count=16222lb # [ 155.939180] niks3-server[1060]: 2026/09/21 12:11:06 INFO Registered completed upload object_key=g5laypvfk2469p8binw3mksk2hv7zhnh.ls6223lb # [ 155.947342] niks3-server[1060]: 2026/09/21 12:11:06 INFO Received complete upload request method=POST path=/api/pending_closures/51/complete6224lb # [ 155.955007] niks3-server[1060]: 2026/09/21 12:11:06 INFO Completed upload id=516225lb # [ 155.958743] niks3-server[1060]: 2026/09/21 12:11:06 INFO Registered completed upload object_key=g5laypvfk2469p8binw3mksk2hv7zhnh.narinfo6226worker2 # [ 155.946936] nix-grpc-daemon[1906]: ts=2026-09-21T12:11:06Z level=debug event=assigned drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv worker=192.168.1.3:50051 clients=16227worker1 # [ 155.930946] nix-grpc-daemon[1403]: time=2026-09-21T12:11:06.478Z level=INFO msg="Upload complete. (311ms)"6228worker1 # [ 155.945012] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:06Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46256 duration_s=1 drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv assign_id=3 outputs=16229worker2 # [ 155.972591] nix-grpc-daemon[1906]: ts=2026-09-21T12:11:06Z level=debug event=want client=1 drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv cached=06230worker1 # [ 155.962923] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:462566231worker1 # [ 155.968442] nix-daemon[842]: reaped child process 1664, status = succeeded6232worker1 # [ 155.998596] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:06Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:462566233worker1 # [ 156.012175] nix-grpc-daemon[1403]: time=2026-09-21T12:11:06.559Z level=INFO msg="All 1 paths already cached"6234worker1 # [ 156.020779] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:06Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:462566235worker1 # [ 156.034164] nix-daemon[842]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6236worker1 # [ 156.231872] nix-daemon[842]: accepted connection from pid 1689, user nix-grpc-daemon (trusted)6237worker1 # [ 156.247245] nix-daemon[842]: reaped child process 1692, status = succeeded6238worker1 # [ 156.419861] nix-daemon[842]: accepted connection from pid 1694, user nix-grpc-daemon (trusted)6239worker1 # [ 156.433842] nix-daemon[842]: reaped child process 1697, status = succeeded6240lb # [ 156.470941] niks3-server[1060]: 2026/09/21 12:11:07 INFO Received uploads request method=POST path=/api/pending_closures6241worker1 # [ 156.458170] nix-grpc-daemon[1403]: time=2026-09-21T12:11:07.005Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6242worker1 # [ 156.459678] nix-grpc-daemon[1403]: time=2026-09-21T12:11:07.005Z level=INFO msg="Uploading cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv (624B)"6243lb # [ 156.497694] niks3-server[1060]: 2026/09/21 12:11:07 INFO Registered completed upload object_key=nar/1w4sykcjji9zlw0a5xibxycy6x3jyb4drd29x2k9g7x5whfwwpsq.nar.zst6244worker1 # [ 156.489441] nix-grpc-daemon[1403]: time=2026-09-21T12:11:07.037Z level=INFO msg="Uploading 1 narinfos"6245lb # [ 156.513594] niks3-server[1060]: 2026/09/21 12:11:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/52/sign6246lb # [ 156.524185] niks3-server[1060]: 2026/09/21 12:11:07 INFO Signed narinfos id=52 count=16247lb # [ 156.529624] niks3-server[1060]: 2026/09/21 12:11:07 INFO Registered completed upload object_key=cnf7j5vjqyj799isbg26wgjnrlk208gi.ls6248worker1 # [ 156.514562] nix-grpc-daemon[1403]: time=2026-09-21T12:11:07.062Z level=INFO msg="Upload complete. (464ms)"6249lb # [ 156.538246] niks3-server[1060]: 2026/09/21 12:11:07 INFO Received complete upload request method=POST path=/api/pending_closures/52/complete6250worker1 # [ 156.520260] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:07Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46256 duration_s=0 paths=1 nar_bytes_in=6246251lb # [ 156.543592] niks3-server[1060]: 2026/09/21 12:11:07 INFO Completed upload id=526252lb # [ 156.546989] niks3-server[1060]: 2026/09/21 12:11:07 INFO Registered completed upload object_key=cnf7j5vjqyj799isbg26wgjnrlk208gi.narinfo6253worker1 # [ 156.532611] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:462566254worker1 # [ 156.537931] nix-daemon[842]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6255worker1 # [ 156.542414] nix-daemon[842]: reaped child process 1687, status = succeeded6256worker1 # [ 156.554737] nix-daemon[842]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6257worker1 # [ 157.777087] nix-daemon[842]: reaped child process 1701, status = succeeded6258worker1 # [ 157.941206] nix-daemon[842]: accepted connection from pid 1717, user nix-grpc-daemon (trusted)6259worker1 # [ 157.959873] nix-daemon[842]: reaped child process 1720, status = succeeded6260lb # [ 157.994620] niks3-server[1060]: 2026/09/21 12:11:08 INFO Received uploads request method=POST path=/api/pending_closures6261worker1 # [ 157.984390] nix-grpc-daemon[1403]: time=2026-09-21T12:11:08.529Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6262worker1 # [ 157.985848] nix-grpc-daemon[1403]: time=2026-09-21T12:11:08.529Z level=INFO msg="Uploading p1334cl91hzsgv8bmiqzkwvhbwq24vn1-farm-b (176B)"6263lb # [ 158.031992] niks3-server[1060]: 2026/09/21 12:11:08 INFO Registered completed upload object_key=log/cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv6264lb # [ 158.042265] niks3-server[1060]: 2026/09/21 12:11:08 INFO Registered completed upload object_key=nar/0pa7j07n0n5p9wcfihhx3357xrxs4h8s57kncplymh51lvzvz2kh.nar.zst6265lb # [ 158.055825] niks3-server[1060]: 2026/09/21 12:11:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/53/sign6266worker1 # [ 158.040467] nix-grpc-daemon[1403]: time=2026-09-21T12:11:08.587Z level=INFO msg="Uploading 1 narinfos"6267lb # [ 158.061580] niks3-server[1060]: 2026/09/21 12:11:08 INFO Signed narinfos id=53 count=16268lb # [ 158.065246] niks3-server[1060]: 2026/09/21 12:11:08 INFO Registered completed upload object_key=p1334cl91hzsgv8bmiqzkwvhbwq24vn1.ls6269lb # [ 158.078410] niks3-server[1060]: 2026/09/21 12:11:08 INFO Received complete upload request method=POST path=/api/pending_closures/53/complete6270worker1 # [ 158.061681] nix-grpc-daemon[1403]: time=2026-09-21T12:11:08.609Z level=INFO msg="Upload complete. (298ms)"6271lb # [ 158.089523] niks3-server[1060]: 2026/09/21 12:11:08 INFO Completed upload id=536272worker2 # [ 158.080826] nix-grpc-daemon[1906]: ts=2026-09-21T12:11:08Z level=debug event=assigned drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv worker=192.168.1.3:50051 clients=16273lb # [ 158.096233] niks3-server[1060]: 2026/09/21 12:11:08 INFO Registered completed upload object_key=p1334cl91hzsgv8bmiqzkwvhbwq24vn1.narinfo6274worker1 # [ 158.073048] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:08Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46256 duration_s=1 drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv assign_id=4 outputs=16275worker2 # [ 158.096147] nix-grpc-daemon[1906]: ts=2026-09-21T12:11:08Z level=debug event=want client=1 drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv cached=06276worker1 # [ 158.088707] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:08Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:462566277worker1 # [ 158.094231] nix-daemon[842]: reaped child process 1699, status = succeeded6278worker1 # [ 158.101875] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:08Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:462566279worker1 # [ 158.119495] nix-grpc-daemon[1403]: time=2026-09-21T12:11:08.667Z level=INFO msg="All 1 paths already cached"6280worker1 # [ 158.126864] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:08Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:462566281worker1 # [ 158.138389] nix-daemon[842]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6282worker1 # [ 158.331059] nix-daemon[842]: accepted connection from pid 1724, user nix-grpc-daemon (trusted)6283worker1 # [ 158.349899] nix-daemon[842]: reaped child process 1727, status = succeeded6284worker1 # [ 158.520117] nix-daemon[842]: accepted connection from pid 1729, user nix-grpc-daemon (trusted)6285worker1 # [ 158.535520] nix-daemon[842]: reaped child process 1732, status = succeeded6286lb # [ 158.569109] niks3-server[1060]: 2026/09/21 12:11:09 INFO Received uploads request method=POST path=/api/pending_closures6287worker1 # [ 158.558760] nix-grpc-daemon[1403]: time=2026-09-21T12:11:09.106Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6288worker1 # [ 158.560302] nix-grpc-daemon[1403]: time=2026-09-21T12:11:09.106Z level=INFO msg="Uploading 7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv (624B)"6289lb # [ 158.598482] niks3-server[1060]: 2026/09/21 12:11:09 INFO Registered completed upload object_key=nar/0l0zss9ls4bhz18slrrmz7l1j93xz7424q7314p64rpgrpqf6b7k.nar.zst6290lb # [ 158.612736] niks3-server[1060]: 2026/09/21 12:11:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/54/sign6291worker1 # [ 158.597620] nix-grpc-daemon[1403]: time=2026-09-21T12:11:09.145Z level=INFO msg="Uploading 1 narinfos"6292lb # [ 158.625142] niks3-server[1060]: 2026/09/21 12:11:09 INFO Signed narinfos id=54 count=16293lb # [ 158.630211] niks3-server[1060]: 2026/09/21 12:11:09 INFO Registered completed upload object_key=7vjzkx8444zm2h7882vsj6bmj73zqjq3.ls6294lb # [ 158.639566] niks3-server[1060]: 2026/09/21 12:11:09 INFO Received complete upload request method=POST path=/api/pending_closures/54/complete6295lb # [ 158.644457] niks3-server[1060]: 2026/09/21 12:11:09 INFO Registered completed upload object_key=7vjzkx8444zm2h7882vsj6bmj73zqjq3.narinfo6296lb # [ 158.648437] niks3-server[1060]: 2026/09/21 12:11:09 INFO Completed upload id=546297worker1 # [ 158.627111] nix-grpc-daemon[1403]: time=2026-09-21T12:11:09.174Z level=INFO msg="Upload complete. (475ms)"6298worker1 # [ 158.634435] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:09Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46256 duration_s=0 paths=1 nar_bytes_in=6246299worker1 # [ 158.647600] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:462566300worker1 # [ 158.656906] nix-daemon[842]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6301worker1 # [ 158.659680] nix-daemon[842]: reaped child process 1722, status = succeeded6302worker1 # [ 158.670752] nix-daemon[842]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6303worker1 # [ 159.693318] nix-daemon[842]: reaped child process 1736, status = succeeded6304worker1 # [ 159.847088] nix-daemon[842]: accepted connection from pid 1752, user nix-grpc-daemon (trusted)6305worker1 # [ 159.864334] nix-daemon[842]: reaped child process 1755, status = succeeded6306lb # [ 159.901694] niks3-server[1060]: 2026/09/21 12:11:10 INFO Received uploads request method=POST path=/api/pending_closures6307worker1 # [ 159.892388] nix-grpc-daemon[1403]: time=2026-09-21T12:11:10.439Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6308worker1 # [ 159.893842] nix-grpc-daemon[1403]: time=2026-09-21T12:11:10.439Z level=INFO msg="Uploading nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8-farm-b (176B)"6309lb # [ 159.942949] niks3-server[1060]: 2026/09/21 12:11:10 INFO Registered completed upload object_key=nar/0kqa5m59sp6aj1ixqj6q54wwqxkcn98fw1rc9hfkcdk460wjdl0l.nar.zst6310lb # [ 159.956326] niks3-server[1060]: 2026/09/21 12:11:10 INFO Registered completed upload object_key=log/7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv6311lb # [ 159.965303] niks3-server[1060]: 2026/09/21 12:11:10 INFO Received sign narinfos request method=POST path=/api/pending_closures/55/sign6312worker1 # [ 159.949127] nix-grpc-daemon[1403]: time=2026-09-21T12:11:10.496Z level=INFO msg="Uploading 1 narinfos"6313lb # [ 159.974290] niks3-server[1060]: 2026/09/21 12:11:10 INFO Signed narinfos id=55 count=16314lb # [ 159.980059] niks3-server[1060]: 2026/09/21 12:11:10 INFO Registered completed upload object_key=nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8.ls6315lb # [ 159.990431] niks3-server[1060]: 2026/09/21 12:11:10 INFO Received complete upload request method=POST path=/api/pending_closures/55/complete6316lb # [ 159.995707] niks3-server[1060]: 2026/09/21 12:11:10 INFO Completed upload id=556317worker1 # [ 159.974374] nix-grpc-daemon[1403]: time=2026-09-21T12:11:10.522Z level=INFO msg="Upload complete. (292ms)"6318lb # [ 159.998927] niks3-server[1060]: 2026/09/21 12:11:10 INFO Registered completed upload object_key=nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8.narinfo6319worker2 # [ 159.995486] nix-grpc-daemon[1906]: ts=2026-09-21T12:11:10Z level=debug event=assigned drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv worker=192.168.1.3:50051 clients=16320worker2 # [ 160.008354] nix-grpc-daemon[1906]: ts=2026-09-21T12:11:10Z level=debug event=want client=1 drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv cached=06321worker1 # [ 159.985145] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:10Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46256 duration_s=1 drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv assign_id=5 outputs=16322worker1 # [ 160.003689] nix-daemon[842]: reaped child process 1734, status = succeeded6323worker1 # [ 160.038799] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:462566324worker1 # [ 160.053628] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:10Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:462566325worker1 # [ 160.068319] nix-grpc-daemon[1403]: time=2026-09-21T12:11:10.616Z level=INFO msg="All 1 paths already cached"6326worker1 # [ 160.075066] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:10Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:462566327worker1 # [ 160.087677] nix-daemon[842]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6328worker1 # [ 160.291889] nix-daemon[842]: accepted connection from pid 1759, user nix-grpc-daemon (trusted)6329worker1 # [ 160.308520] nix-daemon[842]: reaped child process 1762, status = succeeded6330worker1 # [ 160.482912] nix-daemon[842]: accepted connection from pid 1764, user nix-grpc-daemon (trusted)6331worker1 # [ 160.498821] nix-daemon[842]: reaped child process 1767, status = succeeded6332lb # [ 160.531280] niks3-server[1060]: 2026/09/21 12:11:11 INFO Received uploads request method=POST path=/api/pending_closures6333worker1 # [ 160.519948] nix-grpc-daemon[1403]: time=2026-09-21T12:11:11.067Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6334worker1 # [ 160.521621] nix-grpc-daemon[1403]: time=2026-09-21T12:11:11.067Z level=INFO msg="Uploading cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv (624B)"6335lb # [ 160.561882] niks3-server[1060]: 2026/09/21 12:11:11 INFO Registered completed upload object_key=nar/0f6g0vggy0qbqsj64l1nivyqdh6q12kh8620ysnf210ani0727vf.nar.zst6336lb # [ 160.571843] niks3-server[1060]: 2026/09/21 12:11:11 INFO Received sign narinfos request method=POST path=/api/pending_closures/56/sign6337worker1 # [ 160.557837] nix-grpc-daemon[1403]: time=2026-09-21T12:11:11.105Z level=INFO msg="Uploading 1 narinfos"6338lb # [ 160.582316] niks3-server[1060]: 2026/09/21 12:11:11 INFO Signed narinfos id=56 count=16339lb # [ 160.587833] niks3-server[1060]: 2026/09/21 12:11:11 INFO Registered completed upload object_key=cgwyij19i3fl7sqlyz54vzfr4zajg077.ls6340lb # [ 160.596696] niks3-server[1060]: 2026/09/21 12:11:11 INFO Received complete upload request method=POST path=/api/pending_closures/56/complete6341lb # [ 160.602575] niks3-server[1060]: 2026/09/21 12:11:11 INFO Registered completed upload object_key=cgwyij19i3fl7sqlyz54vzfr4zajg077.narinfo6342lb # [ 160.607965] niks3-server[1060]: 2026/09/21 12:11:11 INFO Completed upload id=566343worker1 # [ 160.586582] nix-grpc-daemon[1403]: time=2026-09-21T12:11:11.134Z level=INFO msg="Upload complete. (483ms)"6344worker1 # [ 160.593899] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:11Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46256 duration_s=0 paths=1 nar_bytes_in=6246345worker1 # [ 160.606294] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:11Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:462566346worker1 # [ 160.615397] nix-daemon[842]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6347worker1 # [ 160.618249] nix-daemon[842]: reaped child process 1757, status = succeeded6348worker1 # [ 160.632285] nix-daemon[842]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6349worker1 # [ 162.079576] nix-daemon[842]: reaped child process 1771, status = succeeded6350worker1 # [ 162.251886] nix-daemon[842]: accepted connection from pid 1787, user nix-grpc-daemon (trusted)6351worker1 # [ 162.269765] nix-daemon[842]: reaped child process 1790, status = succeeded6352lb # [ 162.303547] niks3-server[1060]: 2026/09/21 12:11:12 INFO Received uploads request method=POST path=/api/pending_closures6353worker1 # [ 162.292429] nix-grpc-daemon[1403]: time=2026-09-21T12:11:12.839Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6354worker1 # [ 162.294109] nix-grpc-daemon[1403]: time=2026-09-21T12:11:12.839Z level=INFO msg="Uploading hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5-farm-b (176B)"6355lb # [ 162.338234] niks3-server[1060]: 2026/09/21 12:11:12 INFO Registered completed upload object_key=log/cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv6356lb # [ 162.354314] niks3-server[1060]: 2026/09/21 12:11:12 INFO Registered completed upload object_key=nar/0x964sxcxsdh3nfjarc1ib52kr50kl8zxl7h34jq5ydx763drras.nar.zst6357lb # [ 162.364319] niks3-server[1060]: 2026/09/21 12:11:12 INFO Received sign narinfos request method=POST path=/api/pending_closures/57/sign6358worker1 # [ 162.348532] nix-grpc-daemon[1403]: time=2026-09-21T12:11:12.895Z level=INFO msg="Uploading 1 narinfos"6359lb # [ 162.374267] niks3-server[1060]: 2026/09/21 12:11:12 INFO Signed narinfos id=57 count=16360lb # [ 162.379793] niks3-server[1060]: 2026/09/21 12:11:12 INFO Registered completed upload object_key=hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5.ls6361lb # [ 162.388696] niks3-server[1060]: 2026/09/21 12:11:12 INFO Received complete upload request method=POST path=/api/pending_closures/57/complete6362lb # [ 162.395303] niks3-server[1060]: 2026/09/21 12:11:12 INFO Registered completed upload object_key=hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5.narinfo6363lb # [ 162.401530] niks3-server[1060]: 2026/09/21 12:11:12 INFO Completed upload id=576364worker1 # [ 162.380272] nix-grpc-daemon[1403]: time=2026-09-21T12:11:12.928Z level=INFO msg="Upload complete. (335ms)"6365worker2 # [ 162.399749] nix-grpc-daemon[1906]: ts=2026-09-21T12:11:12Z level=debug event=assigned drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv worker=192.168.1.3:50051 clients=16366worker1 # [ 162.392979] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:12Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46256 duration_s=1 drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv assign_id=6 outputs=16367worker1 # [ 162.410300] nix-daemon[842]: reaped child process 1769, status = succeeded6368worker1 # [ 162.433118] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:12Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:462566369worker2 # [ 162.446956] nix-grpc-daemon[1906]: ts=2026-09-21T12:11:12Z level=debug event=want client=1 drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv cached=06370worker1 # [ 162.448136] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:12Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:462566371worker1 # [ 162.461618] nix-grpc-daemon[1403]: time=2026-09-21T12:11:13.009Z level=INFO msg="All 2 paths already cached"6372worker1 # [ 162.470226] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:13Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:462566373worker1 # [ 162.485544] nix-daemon[842]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6374worker1 # [ 162.682774] nix-daemon[842]: accepted connection from pid 1794, user nix-grpc-daemon (trusted)6375worker1 # [ 162.698084] nix-daemon[842]: reaped child process 1797, status = succeeded6376worker1 # [ 162.875579] nix-daemon[842]: accepted connection from pid 1799, user nix-grpc-daemon (trusted)6377worker1 # [ 162.891109] nix-daemon[842]: reaped child process 1802, status = succeeded6378lb # [ 162.927018] niks3-server[1060]: 2026/09/21 12:11:13 INFO Received uploads request method=POST path=/api/pending_closures6379worker1 # [ 162.915354] nix-grpc-daemon[1403]: time=2026-09-21T12:11:13.462Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6380worker1 # [ 162.916948] nix-grpc-daemon[1403]: time=2026-09-21T12:11:13.462Z level=INFO msg="Uploading dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv (752B)"6381lb # [ 162.956427] niks3-server[1060]: 2026/09/21 12:11:13 INFO Registered completed upload object_key=nar/0s7ff7rwqx5p9mdzddqgzd9bbd5j8njfc62k07mgnpy19810bs86.nar.zst6382lb # [ 162.966372] niks3-server[1060]: 2026/09/21 12:11:13 INFO Received sign narinfos request method=POST path=/api/pending_closures/58/sign6383worker1 # [ 162.952395] nix-grpc-daemon[1403]: time=2026-09-21T12:11:13.500Z level=INFO msg="Uploading 1 narinfos"6384lb # [ 162.973956] niks3-server[1060]: 2026/09/21 12:11:13 INFO Signed narinfos id=58 count=16385lb # [ 162.979275] niks3-server[1060]: 2026/09/21 12:11:13 INFO Registered completed upload object_key=dca2k4dvnk52m7gijqfq8q99ih3ni8ny.ls6386lb # [ 162.991067] niks3-server[1060]: 2026/09/21 12:11:13 INFO Received complete upload request method=POST path=/api/pending_closures/58/complete6387worker1 # [ 162.976259] nix-grpc-daemon[1403]: time=2026-09-21T12:11:13.524Z level=INFO msg="Upload complete. (473ms)"6388lb # [ 162.998400] niks3-server[1060]: 2026/09/21 12:11:13 INFO Completed upload id=586389lb # [ 163.002012] niks3-server[1060]: 2026/09/21 12:11:13 INFO Registered completed upload object_key=dca2k4dvnk52m7gijqfq8q99ih3ni8ny.narinfo6390worker1 # [ 162.984305] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:13Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46256 duration_s=0 paths=1 nar_bytes_in=7526391worker1 # [ 162.995292] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:13Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:462566392worker1 # [ 163.000965] nix-daemon[842]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6393worker1 # [ 163.005325] nix-daemon[842]: reaped child process 1792, status = succeeded6394worker1 # [ 163.018000] nix-daemon[842]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6395worker1 # [ 164.063464] nix-daemon[842]: reaped child process 1806, status = succeeded6396worker1 # [ 164.236083] nix-daemon[842]: accepted connection from pid 1822, user nix-grpc-daemon (trusted)6397worker1 # [ 164.254765] nix-daemon[842]: reaped child process 1825, status = succeeded6398lb # [ 164.289608] niks3-server[1060]: 2026/09/21 12:11:14 INFO Received uploads request method=POST path=/api/pending_closures6399worker1 # [ 164.278694] nix-grpc-daemon[1403]: time=2026-09-21T12:11:14.825Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6400worker1 # [ 164.280234] nix-grpc-daemon[1403]: time=2026-09-21T12:11:14.825Z level=INFO msg="Uploading d88xgckxx0s0n5ly2d0jw1qy1641z8pq-farm-top (232B)"6401lb # [ 164.331596] niks3-server[1060]: 2026/09/21 12:11:14 INFO Registered completed upload object_key=nar/0knxhkhz3x04vk8as5wv5ar2d0h02prv7q5f42qm1xjgr70ancd0.nar.zst6402worker1 # [ 164.321211] nix-grpc-daemon[1403]: time=2026-09-21T12:11:14.868Z level=INFO msg="Uploading 1 narinfos"6403lb # [ 164.340728] niks3-server[1060]: 2026/09/21 12:11:14 INFO Registered completed upload object_key=log/dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv6404lb # [ 164.355267] niks3-server[1060]: 2026/09/21 12:11:14 INFO Received sign narinfos request method=POST path=/api/pending_closures/59/sign6405lb # [ 164.360221] niks3-server[1060]: 2026/09/21 12:11:14 INFO Signed narinfos id=59 count=16406lb # [ 164.366261] niks3-server[1060]: 2026/09/21 12:11:14 INFO Registered completed upload object_key=d88xgckxx0s0n5ly2d0jw1qy1641z8pq.ls6407worker1 # [ 164.348222] nix-grpc-daemon[1403]: time=2026-09-21T12:11:14.895Z level=INFO msg="Upload complete. (320ms)"6408lb # [ 164.370013] niks3-server[1060]: 2026/09/21 12:11:14 INFO Received complete upload request method=POST path=/api/pending_closures/59/complete6409lb # [ 164.373967] niks3-server[1060]: 2026/09/21 12:11:14 INFO Completed upload id=596410lb # [ 164.376918] niks3-server[1060]: 2026/09/21 12:11:14 INFO Registered completed upload object_key=d88xgckxx0s0n5ly2d0jw1qy1641z8pq.narinfo6411worker2 # [ 164.368645] nix-grpc-daemon[1906]: ts=2026-09-21T12:11:14Z level=debug event=assigned drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv worker=192.168.1.3:50051 clients=16412worker1 # [ 164.360895] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:14Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46256 duration_s=1 drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv assign_id=7 outputs=16413worker1 # [ 164.375588] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:14Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:462566414worker1 # [ 164.381493] nix-daemon[842]: reaped child process 1804, status = succeeded6415worker1 # [ 164.388438] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:14Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:462566416worker1 # [ 164.400289] nix-grpc-daemon[1403]: time=2026-09-21T12:11:14.948Z level=INFO msg="All 2 paths already cached"6417worker1 # [ 164.407899] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:14Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:462566418worker1 # [ 164.419457] nix-daemon[842]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6419worker1 # [ 164.606172] nix-daemon[842]: accepted connection from pid 1829, user nix-grpc-daemon (trusted)6420worker1 # [ 164.623069] nix-daemon[842]: reaped child process 1832, status = succeeded6421worker1 # [ 164.797916] nix-daemon[842]: accepted connection from pid 1834, user nix-grpc-daemon (trusted)6422worker1 # [ 164.813557] nix-daemon[842]: reaped child process 1837, status = succeeded6423lb # [ 164.846226] niks3-server[1060]: 2026/09/21 12:11:15 INFO Received uploads request method=POST path=/api/pending_closures6424worker1 # [ 164.833423] nix-grpc-daemon[1403]: time=2026-09-21T12:11:15.381Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6425worker1 # [ 164.834874] nix-grpc-daemon[1403]: time=2026-09-21T12:11:15.381Z level=INFO msg="Uploading 2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv (752B)"6426lb # [ 164.871326] niks3-server[1060]: 2026/09/21 12:11:15 INFO Registered completed upload object_key=nar/0cpiyiv7frlhngqhhc703bglz5q2cpg4n8sgds4sfg7nvw0fcdn5.nar.zst6427lb # [ 164.884756] niks3-server[1060]: 2026/09/21 12:11:15 INFO Received sign narinfos request method=POST path=/api/pending_closures/60/sign6428worker1 # [ 164.871353] nix-grpc-daemon[1403]: time=2026-09-21T12:11:15.418Z level=INFO msg="Uploading 1 narinfos"6429lb # [ 164.892900] niks3-server[1060]: 2026/09/21 12:11:15 INFO Signed narinfos id=60 count=16430lb # [ 164.901842] niks3-server[1060]: 2026/09/21 12:11:15 INFO Registered completed upload object_key=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9.ls6431lb # [ 164.909916] niks3-server[1060]: 2026/09/21 12:11:15 INFO Received complete upload request method=POST path=/api/pending_closures/60/complete6432lb # [ 164.916894] niks3-server[1060]: 2026/09/21 12:11:15 INFO Completed upload id=606433worker1 # [ 164.894929] nix-grpc-daemon[1403]: time=2026-09-21T12:11:15.442Z level=INFO msg="Upload complete. (462ms)"6434lb # [ 164.919468] niks3-server[1060]: 2026/09/21 12:11:15 INFO Registered completed upload object_key=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9.narinfo6435worker1 # [ 164.904782] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:15Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46256 duration_s=0 paths=1 nar_bytes_in=7526436worker1 # [ 164.914085] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:15Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:462566437worker1 # [ 164.921369] nix-daemon[842]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6438worker1 # [ 164.925925] nix-daemon[842]: reaped child process 1827, status = succeeded6439worker1 # [ 164.938261] nix-daemon[842]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6440worker1 # [ 165.781398] nix-daemon[842]: reaped child process 1841, status = succeeded6441worker1 # [ 165.965320] nix-daemon[842]: accepted connection from pid 1857, user nix-grpc-daemon (trusted)6442worker1 # [ 165.982214] nix-daemon[842]: reaped child process 1860, status = succeeded6443lb # [ 166.017884] niks3-server[1060]: 2026/09/21 12:11:16 INFO Received uploads request method=POST path=/api/pending_closures6444worker1 # [ 166.004568] nix-grpc-daemon[1403]: time=2026-09-21T12:11:16.552Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6445worker1 # [ 166.008208] nix-grpc-daemon[1403]: time=2026-09-21T12:11:16.553Z level=INFO msg="Uploading vz84kyfcw804d0iyx7nj61j47ms9zxmn-farm-top (232B)"6446lb # [ 166.055930] niks3-server[1060]: 2026/09/21 12:11:16 INFO Registered completed upload object_key=log/2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv6447worker1 # [ 166.044394] nix-grpc-daemon[1403]: time=2026-09-21T12:11:16.591Z level=INFO msg="Uploading 1 narinfos"6448lb # [ 166.068453] niks3-server[1060]: 2026/09/21 12:11:16 INFO Received sign narinfos request method=POST path=/api/pending_closures/61/sign6449lb # [ 166.079540] niks3-server[1060]: 2026/09/21 12:11:16 INFO Signed narinfos id=61 count=16450lb # [ 166.084562] niks3-server[1060]: 2026/09/21 12:11:16 INFO Registered completed upload object_key=vz84kyfcw804d0iyx7nj61j47ms9zxmn.ls6451worker1 # [ 166.070612] nix-grpc-daemon[1403]: time=2026-09-21T12:11:16.618Z level=INFO msg="Upload complete. (323ms)"6452lb # [ 166.092485] niks3-server[1060]: 2026/09/21 12:11:16 INFO Registered completed upload object_key=nar/0aaz4iim7fa45zws98gbhp73l23kbm0fvp4anssnxa96v666ljhs.nar.zst6453lb # [ 166.097690] niks3-server[1060]: 2026/09/21 12:11:16 INFO Received complete upload request method=POST path=/api/pending_closures/61/complete6454lb # [ 166.101591] niks3-server[1060]: 2026/09/21 12:11:16 INFO Completed upload id=616455worker2 # [ 166.089160] nix-grpc-daemon[1906]: ts=2026-09-21T12:11:16Z level=debug event=assigned drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv worker=192.168.1.3:50051 clients=16456lb # [ 166.103619] niks3-server[1060]: 2026/09/21 12:11:16 INFO Registered completed upload object_key=vz84kyfcw804d0iyx7nj61j47ms9zxmn.narinfo6457worker1 # [ 166.081016] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:16Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46256 duration_s=1 drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv assign_id=8 outputs=16458worker1 # [ 166.095511] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:462566459worker1 # [ 166.100949] nix-daemon[842]: reaped child process 1839, status = succeeded6460worker1 # [ 166.107472] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:16Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:462566461worker1 # [ 166.118023] nix-grpc-daemon[1403]: time=2026-09-21T12:11:16.665Z level=INFO msg="All 2 paths already cached"6462worker1 # [ 166.124617] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:16Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:462566463worker1 # [ 166.133597] nix-daemon[842]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6464worker1 # [ 166.318625] nix-daemon[842]: accepted connection from pid 1864, user nix-grpc-daemon (trusted)6465worker1 # [ 166.335916] nix-daemon[842]: reaped child process 1867, status = succeeded6466worker1 # [ 166.510778] nix-daemon[842]: accepted connection from pid 1869, user nix-grpc-daemon (trusted)6467worker1 # [ 166.528581] nix-daemon[842]: reaped child process 1872, status = succeeded6468lb # [ 166.562235] niks3-server[1060]: 2026/09/21 12:11:17 INFO Received uploads request method=POST path=/api/pending_closures6469worker1 # [ 166.549480] nix-grpc-daemon[1403]: time=2026-09-21T12:11:17.097Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6470worker1 # [ 166.550979] nix-grpc-daemon[1403]: time=2026-09-21T12:11:17.097Z level=INFO msg="Uploading qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv (752B)"6471lb # [ 166.590370] niks3-server[1060]: 2026/09/21 12:11:17 INFO Registered completed upload object_key=nar/05zmsnqx4aj55yncx917mz6d3hzrvz2rx2x75313vvy1kq1zv9fv.nar.zst6472worker1 # [ 166.581149] nix-grpc-daemon[1403]: time=2026-09-21T12:11:17.128Z level=INFO msg="Uploading 1 narinfos"6473lb # [ 166.604512] niks3-server[1060]: 2026/09/21 12:11:17 INFO Received sign narinfos request method=POST path=/api/pending_closures/62/sign6474lb # [ 166.615088] niks3-server[1060]: 2026/09/21 12:11:17 INFO Signed narinfos id=62 count=16475lb # [ 166.619457] niks3-server[1060]: 2026/09/21 12:11:17 INFO Registered completed upload object_key=qx10pa00pilyp5xdzd48l88d5jz143mr.ls6476lb # [ 166.628312] niks3-server[1060]: 2026/09/21 12:11:17 INFO Received complete upload request method=POST path=/api/pending_closures/62/complete6477worker1 # [ 166.611995] nix-grpc-daemon[1403]: time=2026-09-21T12:11:17.159Z level=INFO msg="Upload complete. (467ms)"6478lb # [ 166.634394] niks3-server[1060]: 2026/09/21 12:11:17 INFO Completed upload id=626479lb # [ 166.637721] niks3-server[1060]: 2026/09/21 12:11:17 INFO Registered completed upload object_key=qx10pa00pilyp5xdzd48l88d5jz143mr.narinfo6480worker1 # [ 166.620155] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:17Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46256 duration_s=0 paths=1 nar_bytes_in=7526481worker1 # [ 166.631132] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:17Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:462566482worker1 # [ 166.640372] nix-daemon[842]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6483worker1 # [ 166.643891] nix-daemon[842]: reaped child process 1862, status = succeeded6484worker1 # [ 166.656652] nix-daemon[842]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)6485worker1 # [ 167.480775] nix-daemon[842]: reaped child process 1876, status = succeeded6486worker1 # [ 167.643164] nix-daemon[842]: accepted connection from pid 1892, user nix-grpc-daemon (trusted)6487worker1 # [ 167.659857] nix-daemon[842]: reaped child process 1895, status = succeeded6488lb # [ 167.697267] niks3-server[1060]: 2026/09/21 12:11:18 INFO Received uploads request method=POST path=/api/pending_closures6489worker1 # [ 167.689870] nix-grpc-daemon[1403]: time=2026-09-21T12:11:18.236Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6490worker1 # [ 167.691649] nix-grpc-daemon[1403]: time=2026-09-21T12:11:18.236Z level=INFO msg="Uploading iagwbb072hnm79qb9wsrjp5907zc7aa2-farm-top (232B)"6491lb # [ 167.730918] niks3-server[1060]: 2026/09/21 12:11:18 INFO Registered completed upload object_key=log/qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv6492lb # [ 167.750146] niks3-server[1060]: 2026/09/21 12:11:18 INFO Registered completed upload object_key=nar/053jd22j3kv7v6j0q7cspdpc0h1c1xmywmajl6sd5hzwg50z2yj1.nar.zst6493worker1 # [ 167.734632] nix-grpc-daemon[1403]: time=2026-09-21T12:11:18.282Z level=INFO msg="Uploading 1 narinfos"6494lb # [ 167.760225] niks3-server[1060]: 2026/09/21 12:11:18 INFO Received sign narinfos request method=POST path=/api/pending_closures/63/sign6495lb # [ 167.763775] niks3-server[1060]: 2026/09/21 12:11:18 INFO Signed narinfos id=63 count=16496lb # [ 167.770675] niks3-server[1060]: 2026/09/21 12:11:18 INFO Registered completed upload object_key=iagwbb072hnm79qb9wsrjp5907zc7aa2.ls6497lb # [ 167.777276] niks3-server[1060]: 2026/09/21 12:11:18 INFO Received complete upload request method=POST path=/api/pending_closures/63/complete6498lb # [ 167.781521] niks3-server[1060]: 2026/09/21 12:11:18 INFO Completed upload id=636499worker1 # [ 167.760303] nix-grpc-daemon[1403]: time=2026-09-21T12:11:18.308Z level=INFO msg="Upload complete. (295ms)"6500lb # [ 167.783959] niks3-server[1060]: 2026/09/21 12:11:18 INFO Registered completed upload object_key=iagwbb072hnm79qb9wsrjp5907zc7aa2.narinfo6501worker1 # [ 167.773077] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:18Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46256 duration_s=1 drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv assign_id=9 outputs=16502worker1 # [ 167.786738] nix-daemon[842]: reaped child process 1874, status = succeeded6503client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&1, in 18.58 seconds)6504worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6505worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6506client: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag f5 --arg features '["gpu"]'6507client # [ 168.017703] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build -L --store "grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&restart-grace=30" --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag f5 --arg features "[\"gpu\"]".6508client # Running as unit: gpu.service; invocation ID: 897359321d5b44aabba68c8170b332fb6509client: (finished: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag f5 --arg features '["gpu"]', in 0.09 seconds)6510lb: must succeed: curl -sf localhost:9901/clusters6511lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6512lb: must succeed: curl -sf localhost:9901/clusters6513lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)6514worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6515worker2 # [ 168.257107] nix-grpc-daemon[1906]: ts=2026-09-21T12:11:18Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:408506516worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.09 seconds)6517client # [ 168.279477] nix[1346]: don't know how to build these paths:6518client # [ 168.280763] nix[1346]: /nix/store/3i1nziwpdqq19lzpqy4rywzxnm8dlps6-farm-top.drv6519worker2 # [ 168.264662] nix-grpc-daemon[1906]: ts=2026-09-21T12:11:18Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:408506520worker2 # [ 168.275258] nix-grpc-daemon[1906]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 98 ms (attempt 1/5)6521worker2 # [ 168.378637] nix-grpc-daemon[1906]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 95 ms (attempt 2/5)6522worker2 # [ 168.481173] nix-grpc-daemon[1906]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 244 ms (attempt 3/5)6523worker2 # [ 168.732786] nix-grpc-daemon[1906]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 181 ms (attempt 4/5)6524worker2 # [ 168.921648] nix-grpc-daemon[1906]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6525worker2 # [ 168.928172] nix-grpc-daemon[1906]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6526worker2 # [ 168.958282] nix-grpc-daemon[1906]: ts=2026-09-21T12:11:19Z level=debug event=want client=2 drv=1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv cached=06527client # [ 168.975465] nix[1346]: warning: 1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv: no connected, non-draining worker for system 'aarch64-linux' offers features {gpu}, waiting up to 30s for a worker6528lb: must succeed: curl -sf localhost:9901/clusters6529lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6530lb: must succeed: curl -sf localhost:9901/clusters6531lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)6532worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6533worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)6534client: waiting for success: systemctl show -p Result gpu | grep -q exit-code6535lb # [ 170.675110] envoy[910]: conn peer=[2001:db8:1::1]:35248 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=27114 tx=11334 ms=182566536client # [ 198.978531] nix[1346]: 1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv: no connected, non-draining worker for system 'aarch64-linux' offers features {gpu}6537client # [ 198.991025] nix[1346]: error: dependency '1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv' failed6538client # [ 199.033022] systemd[1]: gpu.service: Main process exited, code=exited, status=1/FAILURE6539client # [ 199.038121] systemd[1]: gpu.service: Failed with result 'exit-code'.6540client # [ 199.043619] systemd[1]: gpu.service: Consumed 179ms CPU time over 31.020s wall clock time, 37.4M memory peak, 4K written to disk, 5.4K incoming IP traffic, 5.1K outgoing IP traffic.6541client: (finished: waiting for success: systemctl show -p Result gpu | grep -q exit-code, in 30.32 seconds)6542client: must succeed: journalctl -u gpu -o cat6543client: (finished: must succeed: journalctl -u gpu -o cat, in 0.03 seconds)6544lb: must succeed: curl -sf localhost:9901/clusters6545lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6546lb: must succeed: curl -sf localhost:9901/clusters6547lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)6548worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6549worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)6550(finished: subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does, in 50.77 seconds)6551subtest: low disk drains a worker and builds go to the other6552worker1: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill6553lb # [ 200.673626] envoy[910]: conn peer=[2001:db8:1::1]:52664 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=914 tx=540 ms=307276554worker1 # [ 201.125262] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:51Z level=info event=unhealthy reason=min_free6555worker1: (finished: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill, in 1.13 seconds)6556worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free'6557worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free', in 0.05 seconds)6558client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag drain >&26559client # don't know how to build these paths:6560client # /nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv6561worker2 # [ 201.504597] nix-grpc-daemon[1906]: ts=2026-09-21T12:11:52Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:408506562worker2 # [ 201.511432] nix-grpc-daemon[1906]: ts=2026-09-21T12:11:52Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:408506563worker2 # [ 201.537557] nix-grpc-daemon[1906]: ts=2026-09-21T12:11:52Z level=debug event=want client=3 drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv cached=06564worker2 # [ 201.544807] nix-grpc-daemon[1906]: ts=2026-09-21T12:11:52Z level=debug event=assigned drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv worker=192.168.1.4:50051 clients=16565worker2 # [ 201.582314] nix-grpc-daemon[1906]: ts=2026-09-21T12:11:52Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:408506566worker2 # [ 201.597975] nix-grpc-daemon[1906]: ts=2026-09-21T12:11:52Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:408506567client # copying 1 paths to 'lb:50051'...6568client # copying path '/nix/store/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv' to 'lb:50051'...6569worker2 # [ 201.613583] nix-grpc-daemon[1906]: ts=2026-09-21T12:11:52Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:408506570worker2 # [ 201.629488] nix-daemon[863]: accepted connection from pid 1906, user nix-grpc-daemon (trusted)6571worker2 # [ 201.658212] nix-grpc-daemon[1906]: ts=2026-09-21T12:11:52Z level=info event=niks3_push_started pid=19966572worker2 # [ 201.882037] nix-grpc-daemon[1996]: time=2026-09-21T12:11:52.424Z level=INFO msg="Configuring client TLS" cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker2.pem key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker2.key ca=""6573worker2 # [ 202.061225] nix-daemon[863]: accepted connection from pid 2009, user nix-grpc-daemon (trusted)6574worker2 # [ 202.078361] nix-daemon[863]: reaped child process 2012, status = succeeded6575worker2 # [ 202.251572] nix-daemon[863]: accepted connection from pid 2014, user nix-grpc-daemon (trusted)6576worker2 # [ 202.270222] nix-daemon[863]: reaped child process 2017, status = succeeded6577lb # [ 202.292597] niks3-server[1060]: 2026/09/21 12:11:52 INFO Received uploads request method=POST path=/api/pending_closures6578worker2 # [ 202.292064] nix-grpc-daemon[1996]: time=2026-09-21T12:11:52.835Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6579worker2 # [ 202.293544] nix-grpc-daemon[1996]: time=2026-09-21T12:11:52.835Z level=INFO msg="Uploading kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv (512B)"6580lb # [ 202.328423] niks3-server[1060]: 2026/09/21 12:11:52 INFO Registered completed upload object_key=nar/1sfy0gcq4znfcj1f68c749r5x2jwmwsfbfp6xdflz5xkndjwjjm3.nar.zst6581lb # [ 202.342114] niks3-server[1060]: 2026/09/21 12:11:52 INFO Received sign narinfos request method=POST path=/api/pending_closures/64/sign6582worker2 # [ 202.338659] nix-grpc-daemon[1996]: time=2026-09-21T12:11:52.881Z level=INFO msg="Uploading 1 narinfos"6583lb # [ 202.354802] niks3-server[1060]: 2026/09/21 12:11:52 INFO Signed narinfos id=64 count=16584lb # [ 202.359245] niks3-server[1060]: 2026/09/21 12:11:52 INFO Registered completed upload object_key=kksl1jqfg56vabgqvrkz5vki6nq7r59r.ls6585lb # [ 202.373993] niks3-server[1060]: 2026/09/21 12:11:52 INFO Received complete upload request method=POST path=/api/pending_closures/64/complete6586worker2 # [ 202.370265] nix-grpc-daemon[1996]: time=2026-09-21T12:11:52.913Z level=INFO msg="Upload complete. (484ms)"6587lb # [ 202.381059] niks3-server[1060]: 2026/09/21 12:11:52 INFO Completed upload id=646588lb # [ 202.385290] niks3-server[1060]: 2026/09/21 12:11:52 INFO Registered completed upload object_key=kksl1jqfg56vabgqvrkz5vki6nq7r59r.narinfo6589worker2 # [ 202.380546] nix-grpc-daemon[1906]: ts=2026-09-21T12:11:52Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40850 duration_s=0 paths=1 nar_bytes_in=5126590worker2 # [ 202.391272] nix-grpc-daemon[1906]: ts=2026-09-21T12:11:52Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:408506591worker2 # [ 202.397479] nix-daemon[863]: accepted connection from pid 1906, user nix-grpc-daemon (trusted)6592worker2 # [ 202.402275] nix-daemon[863]: reaped child process 1994, status = succeeded6593client # building '/nix/store/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv' on lb:50051...6594client # farm-a> worker2-next: building kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv6595worker2 # [ 202.415330] nix-daemon[863]: accepted connection from pid 1906, user nix-grpc-daemon (trusted)6596client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds6597client # farm-a> LOG-farm-a-drain6598worker2 # [ 203.464235] nix-daemon[863]: reaped child process 2021, status = succeeded6599worker2 # [ 203.643313] nix-daemon[863]: accepted connection from pid 2037, user nix-grpc-daemon (trusted)6600worker2 # [ 203.658966] nix-daemon[863]: reaped child process 2040, status = succeeded6601lb # [ 203.685245] niks3-server[1060]: 2026/09/21 12:11:54 INFO Received uploads request method=POST path=/api/pending_closures6602worker2 # [ 203.692267] nix-grpc-daemon[1996]: time=2026-09-21T12:11:54.232Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6603worker2 # [ 203.693905] nix-grpc-daemon[1996]: time=2026-09-21T12:11:54.232Z level=INFO msg="Uploading 272b39wn7cbnhkcc2rrz2f6kmhhmcwnr-farm-a (128B)"6604lb # [ 203.728680] niks3-server[1060]: 2026/09/21 12:11:54 INFO Registered completed upload object_key=log/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv6605lb # [ 203.740070] niks3-server[1060]: 2026/09/21 12:11:54 INFO Registered completed upload object_key=nar/1bf9z7y0ikx07b4zjag42zyrv2pq1ycw4461s579kfpmni9kz4dw.nar.zst6606lb # [ 203.751244] niks3-server[1060]: 2026/09/21 12:11:54 INFO Received sign narinfos request method=POST path=/api/pending_closures/65/sign6607worker2 # [ 203.742282] nix-grpc-daemon[1996]: time=2026-09-21T12:11:54.284Z level=INFO msg="Uploading 1 narinfos"6608lb # [ 203.759304] niks3-server[1060]: 2026/09/21 12:11:54 INFO Signed narinfos id=65 count=16609lb # [ 203.765269] niks3-server[1060]: 2026/09/21 12:11:54 INFO Registered completed upload object_key=272b39wn7cbnhkcc2rrz2f6kmhhmcwnr.ls6610lb # [ 203.774627] niks3-server[1060]: 2026/09/21 12:11:54 INFO Received complete upload request method=POST path=/api/pending_closures/65/complete6611lb # [ 203.781704] niks3-server[1060]: 2026/09/21 12:11:54 INFO Completed upload id=656612worker2 # [ 203.770027] nix-grpc-daemon[1996]: time=2026-09-21T12:11:54.313Z level=INFO msg="Upload complete. (344ms)"6613lb # [ 203.786648] niks3-server[1060]: 2026/09/21 12:11:54 INFO Registered completed upload object_key=272b39wn7cbnhkcc2rrz2f6kmhhmcwnr.narinfo6614worker2 # [ 203.788714] nix-grpc-daemon[1906]: ts=2026-09-21T12:11:54Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40850 duration_s=1 drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv assign_id=10 outputs=16615worker2 # [ 203.804236] nix-grpc-daemon[1906]: ts=2026-09-21T12:11:54Z level=debug event=want client=3 drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv cached=06616worker2 # [ 203.807633] nix-grpc-daemon[1906]: ts=2026-09-21T12:11:54Z level=debug event=assigned drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv worker=192.168.1.4:50051 clients=16617worker2 # [ 203.811800] nix-daemon[863]: reaped child process 2019, status = succeeded6618worker2 # [ 203.833901] nix-grpc-daemon[1906]: ts=2026-09-21T12:11:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:408506619worker2 # [ 203.847875] nix-grpc-daemon[1906]: ts=2026-09-21T12:11:54Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:408506620worker2 # [ 203.861019] nix-grpc-daemon[1996]: time=2026-09-21T12:11:54.404Z level=INFO msg="All 1 paths already cached"6621client # copying 1 paths to 'lb:50051'...6622client # copying path '/nix/store/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv' to 'lb:50051'...6623worker2 # [ 203.871689] nix-grpc-daemon[1906]: ts=2026-09-21T12:11:54Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:408506624worker2 # [ 203.885436] nix-daemon[863]: accepted connection from pid 1906, user nix-grpc-daemon (trusted)6625worker2 # [ 204.091254] nix-daemon[863]: accepted connection from pid 2046, user nix-grpc-daemon (trusted)6626worker2 # [ 204.108336] nix-daemon[863]: reaped child process 2049, status = succeeded6627worker2 # [ 204.285118] nix-daemon[863]: accepted connection from pid 2051, user nix-grpc-daemon (trusted)6628worker2 # [ 204.301890] nix-daemon[863]: reaped child process 2054, status = succeeded6629lb # [ 204.323227] niks3-server[1060]: 2026/09/21 12:11:54 INFO Received uploads request method=POST path=/api/pending_closures6630worker2 # [ 204.322388] nix-grpc-daemon[1996]: time=2026-09-21T12:11:54.865Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6631worker2 # [ 204.323877] nix-grpc-daemon[1996]: time=2026-09-21T12:11:54.865Z level=INFO msg="Uploading fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv (632B)"6632lb # [ 204.353076] niks3-server[1060]: 2026/09/21 12:11:54 INFO Received sign narinfos request method=POST path=/api/pending_closures/66/sign6633worker2 # [ 204.349323] nix-grpc-daemon[1996]: time=2026-09-21T12:11:54.892Z level=INFO msg="Uploading 1 narinfos"6634lb # [ 204.366630] niks3-server[1060]: 2026/09/21 12:11:54 INFO Signed narinfos id=66 count=16635lb # [ 204.374774] niks3-server[1060]: 2026/09/21 12:11:54 INFO Registered completed upload object_key=nar/1ax4z88sb8d24gsvydxvgdy2lijkmhnycxhr5war1j3sq97cyzqf.nar.zst6636lb # [ 204.387592] niks3-server[1060]: 2026/09/21 12:11:54 INFO Registered completed upload object_key=fa7adcm0x8sbgmrsfx912fp8xakk99g6.ls6637lb # [ 204.391740] niks3-server[1060]: 2026/09/21 12:11:54 INFO Received complete upload request method=POST path=/api/pending_closures/66/complete6638worker2 # [ 204.386314] nix-grpc-daemon[1996]: time=2026-09-21T12:11:54.929Z level=INFO msg="Upload complete. (485ms)"6639lb # [ 204.396653] niks3-server[1060]: 2026/09/21 12:11:54 INFO Completed upload id=666640lb # [ 204.399598] niks3-server[1060]: 2026/09/21 12:11:54 INFO Registered completed upload object_key=fa7adcm0x8sbgmrsfx912fp8xakk99g6.narinfo6641worker2 # [ 204.392234] nix-grpc-daemon[1906]: ts=2026-09-21T12:11:54Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40850 duration_s=0 paths=1 nar_bytes_in=6326642worker2 # [ 204.403972] nix-grpc-daemon[1906]: ts=2026-09-21T12:11:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:408506643worker2 # [ 204.407969] nix-daemon[863]: reaped child process 2044, status = succeeded6644worker2 # [ 204.414214] nix-daemon[863]: accepted connection from pid 1906, user nix-grpc-daemon (trusted)6645client # building '/nix/store/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv' on lb:50051...6646client # farm-b> worker2-next: building fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv6647worker2 # [ 204.427405] nix-daemon[863]: accepted connection from pid 1906, user nix-grpc-daemon (trusted)6648client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds6649client # farm-b> LOG-farm-b-drain6650worker2 # [ 205.275552] nix-daemon[863]: reaped child process 2058, status = succeeded6651worker2 # [ 205.452700] nix-daemon[863]: accepted connection from pid 2074, user nix-grpc-daemon (trusted)6652worker2 # [ 205.468714] nix-daemon[863]: reaped child process 2077, status = succeeded6653lb # [ 205.494205] niks3-server[1060]: 2026/09/21 12:11:56 INFO Received uploads request method=POST path=/api/pending_closures6654worker2 # [ 205.500486] nix-grpc-daemon[1996]: time=2026-09-21T12:11:56.043Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6655worker2 # [ 205.502067] nix-grpc-daemon[1996]: time=2026-09-21T12:11:56.043Z level=INFO msg="Uploading g774lks418zc45bd9hwhinm61pwcwnr9-farm-b (176B)"6656lb # [ 205.534497] niks3-server[1060]: 2026/09/21 12:11:56 INFO Registered completed upload object_key=log/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv6657lb # [ 205.546133] niks3-server[1060]: 2026/09/21 12:11:56 INFO Registered completed upload object_key=nar/1ammzz4zx9sggy8v73zxk3j08vbqxgpw14n8lx2fh57bx24yzvnj.nar.zst6658lb # [ 205.554645] niks3-server[1060]: 2026/09/21 12:11:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/67/sign6659worker2 # [ 205.549250] nix-grpc-daemon[1996]: time=2026-09-21T12:11:56.092Z level=INFO msg="Uploading 1 narinfos"6660lb # [ 205.563520] niks3-server[1060]: 2026/09/21 12:11:56 INFO Signed narinfos id=67 count=16661lb # [ 205.568668] niks3-server[1060]: 2026/09/21 12:11:56 INFO Registered completed upload object_key=g774lks418zc45bd9hwhinm61pwcwnr9.ls6662lb # [ 205.575606] niks3-server[1060]: 2026/09/21 12:11:56 INFO Received complete upload request method=POST path=/api/pending_closures/67/complete6663lb # [ 205.579990] niks3-server[1060]: 2026/09/21 12:11:56 INFO Completed upload id=676664worker2 # [ 205.569804] nix-grpc-daemon[1996]: time=2026-09-21T12:11:56.113Z level=INFO msg="Upload complete. (333ms)"6665lb # [ 205.582111] niks3-server[1060]: 2026/09/21 12:11:56 INFO Registered completed upload object_key=g774lks418zc45bd9hwhinm61pwcwnr9.narinfo6666worker2 # [ 205.589005] nix-grpc-daemon[1906]: ts=2026-09-21T12:11:56Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40850 duration_s=1 drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv assign_id=11 outputs=16667worker2 # [ 205.602787] nix-grpc-daemon[1906]: ts=2026-09-21T12:11:56Z level=debug event=want client=3 drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv cached=06668worker2 # [ 205.606731] nix-grpc-daemon[1906]: ts=2026-09-21T12:11:56Z level=debug event=assigned drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv worker=192.168.1.4:50051 clients=16669worker2 # [ 205.610704] nix-daemon[863]: reaped child process 2056, status = succeeded6670worker2 # [ 205.630241] nix-grpc-daemon[1906]: ts=2026-09-21T12:11:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:408506671worker2 # [ 205.645998] nix-grpc-daemon[1906]: ts=2026-09-21T12:11:56Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:408506672worker2 # [ 205.660060] nix-grpc-daemon[1996]: time=2026-09-21T12:11:56.203Z level=INFO msg="All 2 paths already cached"6673client # copying 1 paths to 'lb:50051'...6674client # copying path '/nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv' to 'lb:50051'...6675worker2 # [ 205.670093] nix-grpc-daemon[1906]: ts=2026-09-21T12:11:56Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:408506676worker2 # [ 205.683755] nix-daemon[863]: accepted connection from pid 1906, user nix-grpc-daemon (trusted)6677worker2 # [ 205.884152] nix-daemon[863]: accepted connection from pid 2083, user nix-grpc-daemon (trusted)6678worker2 # [ 205.900949] nix-daemon[863]: reaped child process 2086, status = succeeded6679worker2 # [ 206.075923] nix-daemon[863]: accepted connection from pid 2088, user nix-grpc-daemon (trusted)6680worker2 # [ 206.090918] nix-daemon[863]: reaped child process 2091, status = succeeded6681lb # [ 206.116968] niks3-server[1060]: 2026/09/21 12:11:56 INFO Received uploads request method=POST path=/api/pending_closures6682worker2 # [ 206.115543] nix-grpc-daemon[1996]: time=2026-09-21T12:11:56.658Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6683worker2 # [ 206.117172] nix-grpc-daemon[1996]: time=2026-09-21T12:11:56.658Z level=INFO msg="Uploading 77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv (760B)"6684lb # [ 206.144607] niks3-server[1060]: 2026/09/21 12:11:56 INFO Registered completed upload object_key=nar/1hrr94bvb37dwmf1giypzl9qnff72xm540ak7q49j34fyzia9ngb.nar.zst6685lb # [ 206.155953] niks3-server[1060]: 2026/09/21 12:11:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/68/sign6686worker2 # [ 206.151957] nix-grpc-daemon[1996]: time=2026-09-21T12:11:56.695Z level=INFO msg="Uploading 1 narinfos"6687lb # [ 206.167690] niks3-server[1060]: 2026/09/21 12:11:56 INFO Signed narinfos id=68 count=16688lb # [ 206.174310] niks3-server[1060]: 2026/09/21 12:11:56 INFO Registered completed upload object_key=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l.ls6689worker2 # [ 206.173540] nix-grpc-daemon[1996]: time=2026-09-21T12:11:56.717Z level=INFO msg="Upload complete. (474ms)"6690lb # [ 206.183806] niks3-server[1060]: 2026/09/21 12:11:56 INFO Received complete upload request method=POST path=/api/pending_closures/68/complete6691lb # [ 206.189643] niks3-server[1060]: 2026/09/21 12:11:56 INFO Completed upload id=686692worker2 # [ 206.183325] nix-grpc-daemon[1906]: ts=2026-09-21T12:11:56Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40850 duration_s=0 paths=1 nar_bytes_in=7606693lb # [ 206.193215] niks3-server[1060]: 2026/09/21 12:11:56 INFO Registered completed upload object_key=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l.narinfo6694worker2 # [ 206.188712] nix-grpc-daemon[1906]: ts=2026-09-21T12:11:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:408506695worker2 # [ 206.198069] nix-daemon[863]: accepted connection from pid 1906, user nix-grpc-daemon (trusted)6696worker2 # [ 206.202365] nix-daemon[863]: reaped child process 2081, status = succeeded6697client # building '/nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv' on lb:50051...6698worker2 # [ 206.214874] nix-daemon[863]: accepted connection from pid 1906, user nix-grpc-daemon (trusted)6699client # farm-top> worker2-next: building 77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv6700client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds6701client # farm-top> LOG-farm-top-drain6702worker2 # [ 207.259904] nix-daemon[863]: reaped child process 2095, status = succeeded6703worker2 # [ 207.446059] nix-daemon[863]: accepted connection from pid 2111, user nix-grpc-daemon (trusted)6704worker2 # [ 207.462550] nix-daemon[863]: reaped child process 2114, status = succeeded6705lb # [ 207.486044] niks3-server[1060]: 2026/09/21 12:11:58 INFO Received uploads request method=POST path=/api/pending_closures6706worker2 # [ 207.487565] nix-grpc-daemon[1996]: time=2026-09-21T12:11:58.029Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6707worker2 # [ 207.489190] nix-grpc-daemon[1996]: time=2026-09-21T12:11:58.029Z level=INFO msg="Uploading bzkv277k4v9r9h4aq5h13ddcavmp36fg-farm-top (232B)"6708lb # [ 207.525768] niks3-server[1060]: 2026/09/21 12:11:58 INFO Registered completed upload object_key=nar/0l8sy8grn9pmq2n58ag0hx791zdfyxwk9319ynjqmn7yy7wl64s7.nar.zst6709worker2 # [ 207.528173] nix-grpc-daemon[1996]: time=2026-09-21T12:11:58.071Z level=INFO msg="Uploading 1 narinfos"6710lb # [ 207.541122] niks3-server[1060]: 2026/09/21 12:11:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/69/sign6711lb # [ 207.550539] niks3-server[1060]: 2026/09/21 12:11:58 INFO Signed narinfos id=69 count=16712lb # [ 207.556308] niks3-server[1060]: 2026/09/21 12:11:58 INFO Registered completed upload object_key=bzkv277k4v9r9h4aq5h13ddcavmp36fg.ls6713lb # [ 207.559859] niks3-server[1060]: 2026/09/21 12:11:58 INFO Registered completed upload object_key=log/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv6714lb # [ 207.568163] niks3-server[1060]: 2026/09/21 12:11:58 INFO Received complete upload request method=POST path=/api/pending_closures/69/complete6715lb # [ 207.575969] niks3-server[1060]: 2026/09/21 12:11:58 INFO Completed upload id=696716worker2 # [ 207.565856] nix-grpc-daemon[1996]: time=2026-09-21T12:11:58.109Z level=INFO msg="Upload complete. (341ms)"6717lb # [ 207.579004] niks3-server[1060]: 2026/09/21 12:11:58 INFO Registered completed upload object_key=bzkv277k4v9r9h4aq5h13ddcavmp36fg.narinfo6718worker2 # [ 207.577058] nix-grpc-daemon[1906]: ts=2026-09-21T12:11:58Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40850 duration_s=1 drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv assign_id=12 outputs=16719worker2 # [ 207.595501] nix-daemon[863]: reaped child process 2093, status = succeeded6720client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag drain >&2, in 6.40 seconds)6721client: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag drain outPath6722client: (finished: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag drain outPath, in 0.23 seconds)6723worker1: must succeed: rm /nix/.rw-store/fill6724worker1: (finished: must succeed: rm /nix/.rw-store/fill, in 0.15 seconds)6725worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free'6726worker1 # [ 208.127386] nix-grpc-daemon[1311]: ts=2026-09-21T12:11:58Z level=info event=healthy reason=min_free6727worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free', in 0.04 seconds)6728(finished: subtest: low disk drains a worker and builds go to the other, in 8.10 seconds)6729(finished: run the VM test script, in 209.10 seconds)6730test script finished in 209.38s6731cleanup6732kill QemuMachine (pid 49)6733client # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/lfh101cfjj8p8npcari9hk0lbqi33yvh-python3-3.14.7/bin/python3.14)6734kill QemuMachine (pid 50)6735lb # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/lfh101cfjj8p8npcari9hk0lbqi33yvh-python3-3.14.7/bin/python3.14)6736kill QemuMachine (pid 52)6737worker1 # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/lfh101cfjj8p8npcari9hk0lbqi33yvh-python3-3.14.7/bin/python3.14)6738kill QemuMachine (pid 51)6739worker2 # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/lfh101cfjj8p8npcari9hk0lbqi33yvh-python3-3.14.7/bin/python3.14)6740(finished: cleanup, in 1.23 seconds)6741additionally exposed symbols:6742 client, lb, worker1, worker2,6743 vlan1,6744 start_all, test_script, machines, machines_qemu, machines_nspawn, vlans, driver, log, os, create_machine, subtest, run_tests, join_all, retry, serial_stdout_off, serial_stdout_on, polling_condition, BaseMachine, QemuMachine, NspawnMachine, t, debug, dump_machine_ssh