container-test-run-flakelet-relay
checks.x86_64-linux.nixos-test
· build #21
· raw
1tribuchet: building on jamie2Machine state will be reset. To keep it, pass --keep-machine-state3start all VLans4(finished: start all VLans, in 0.00 seconds)5Initializing machine ID from random generator.6Test will time out and terminate in 3600.0 seconds7run the VM test script8start all VMs9agent: systemd-nspawn running (pid 52)10client: systemd-nspawn running (pid 53)11agent: Waiting for journal at /build/vm-state-agent/var/log/journal...12relay: systemd-nspawn running (pid 56)13client: Waiting for journal at /build/vm-state-client/var/log/journal...14relay: Waiting for journal at /build/vm-state-relay/var/log/journal...15(finished: start all VMs, in 0.00 seconds)16relay: waiting for unit flakelet-relay.service17nixos-nspawn(client): TAP vde-tap1 not found; container will be isolated from VDE18nixos-nspawn(client): A common reason for this is that /dev/net is not available in the Nix sandbox. Try adding /dev/net to extra-sandbox-paths.19nixos-nspawn(agent): TAP vde-tap1 not found; container will be isolated from VDE20nixos-nspawn(agent): A common reason for this is that /dev/net is not available in the Nix sandbox. Try adding /dev/net to extra-sandbox-paths.21nixos-nspawn(relay): TAP vde-tap1 not found; container will be isolated from VDE22nixos-nspawn(relay): A common reason for this is that /dev/net is not available in the Nix sandbox. Try adding /dev/net to extra-sandbox-paths.23Note: in a future version of systemd-nspawn the default set of permitted socket address families will be restricted to AF_INET, AF_INET6 and AF_UNIX. Use --restrict-address-families= to configure the set of permitted socket address families, or set RestrictAddressFamilies= in a .nspawn file.24Note: in a future version of systemd-nspawn the default set of permitted socket address families will be restricted to AF_INET, AF_INET6 and AF_UNIX. Use --restrict-address-families= to configure the set of permitted socket address families, or set RestrictAddressFamilies= in a .nspawn file.25░ Spawning container client on /build/vm-state-client.26░ Spawning container relay on /build/vm-state-relay.27Note: in a future version of systemd-nspawn the default set of permitted socket address families will be restricted to AF_INET, AF_INET6 and AF_UNIX. Use --restrict-address-families= to configure the set of permitted socket address families, or set RestrictAddressFamilies= in a .nspawn file.28░ Spawning container agent on /build/vm-state-agent.29agent # [ 4473.385397] agent systemd-journald[57]: Journal started30agent # [ 4473.385439] agent systemd-journald[57]: Runtime Journal (/run/log/journal/bbdc25a12fed41c4858accd83e668c4b) is 8M, max 4G, 3.9G free.31agent # [ 4473.395608] agent systemd[1]: Finished Apply Kernel Variables.32agent # [ 4473.407513] agent systemd[1]: Finished Create Static Device Nodes in /dev gracefully.33agent # [ 4473.443807] agent systemd[1]: Starting Flush Journal to Persistent Storage...34agent # [ 4473.444581] agent systemd[1]: Starting Create Static Device Nodes in /dev...35relay # [ 4473.383230] relay systemd-journald[57]: Journal started36relay # [ 4473.383272] relay systemd-journald[57]: Runtime Journal (/run/log/journal/ce1cc1ef91d649a5b86026ce32603edc) is 8M, max 4G, 3.9G free.37relay # [ 4473.395094] relay systemd[1]: Finished Apply Kernel Variables.38relay # [ 4473.407184] relay systemd[1]: Finished Create Static Device Nodes in /dev gracefully.39relay # [ 4473.443778] relay systemd[1]: Starting Flush Journal to Persistent Storage...40relay # [ 4473.444682] relay systemd[1]: Starting Create Static Device Nodes in /dev...41relay # [ 4473.452578] relay systemd-journald[57]: Time spent on flushing to /var/log/journal/ce1cc1ef91d649a5b86026ce32603edc is 1.394ms for 6 entries.42client # [ 4473.377224] client systemd-journald[57]: Journal started43client # [ 4473.377264] client systemd-journald[57]: Runtime Journal (/run/log/journal/083857f1ee3347c0b2c47f98b0ab6489) is 8M, max 4G, 3.9G free.44client # [ 4473.387857] client systemd[1]: Finished Apply Kernel Variables.45client # [ 4473.399110] client systemd[1]: Finished Create Static Device Nodes in /dev gracefully.46client # [ 4473.412742] client systemd[1]: Starting Flush Journal to Persistent Storage...47client # [ 4473.413462] client systemd[1]: Starting Create Static Device Nodes in /dev...48client # [ 4473.449074] client systemd-journald[57]: Time spent on flushing to /var/log/journal/083857f1ee3347c0b2c47f98b0ab6489 is 1.337ms for 6 entries.49client # [ 4473.449074] client systemd-journald[57]: System Journal (/var/log/journal/083857f1ee3347c0b2c47f98b0ab6489) is 512B, max 4G, 3.9G free.50client # [ 4473.456613] client systemd[1]: Finished Create Static Device Nodes in /dev.51client # [ 4473.457313] client systemd[1]: Reached target Preparation for Local File Systems.52client # [ 4473.457418] client systemd[1]: Rule-based Manager for Device Events and Files skipped, unmet condition check ConditionPathIsReadWrite=/sys53client # [ 4473.470506] client systemd[1]: Finished Flush Journal to Persistent Storage.54client # [ 4473.526140] client systemd[1]: Finished Firewall.55agent # [ 4473.452590] agent systemd-journald[57]: Time spent on flushing to /var/log/journal/bbdc25a12fed41c4858accd83e668c4b is 1.397ms for 6 entries.56agent # [ 4473.452590] agent systemd-journald[57]: System Journal (/var/log/journal/bbdc25a12fed41c4858accd83e668c4b) is 512B, max 4G, 3.9G free.57agent # [ 4473.461846] agent systemd[1]: Finished Create Static Device Nodes in /dev.58agent # [ 4473.462541] agent systemd[1]: Reached target Preparation for Local File Systems.59agent # [ 4473.462651] agent systemd[1]: Rule-based Manager for Device Events and Files skipped, unmet condition check ConditionPathIsReadWrite=/sys60agent # [ 4473.477174] agent systemd[1]: Finished Flush Journal to Persistent Storage.61agent # [ 4473.532058] agent systemd[1]: Finished Firewall.62relay # [ 4473.452578] relay systemd-journald[57]: System Journal (/var/log/journal/ce1cc1ef91d649a5b86026ce32603edc) is 512B, max 4G, 3.9G free.63relay # [ 4473.463009] relay systemd[1]: Finished Create Static Device Nodes in /dev.64relay # [ 4473.463748] relay systemd[1]: Reached target Preparation for Local File Systems.65relay # [ 4473.463870] relay systemd[1]: Rule-based Manager for Device Events and Files skipped, unmet condition check ConditionPathIsReadWrite=/sys66relay # [ 4473.474303] relay systemd[1]: Finished Flush Journal to Persistent Storage.67relay # [ 4473.555989] relay systemd[1]: Finished Firewall.68agent # [ 4474.385174] agent systemd[1]: Mounting /run/wrappers...69agent # [ 4474.432861] agent systemd[1]: Mounted /run/wrappers.70agent # [ 4474.433591] agent systemd[1]: Reached target Local File Systems.71agent # [ 4474.434389] agent systemd[1]: Listening on Boot Loader Control Service Socket.72agent # [ 4474.435571] agent systemd[1]: Starting Create SUID/SGID Wrappers...73agent # [ 4474.435600] agent systemd[1]: Update Boot Loader Random Seed skipped, unmet condition check ConditionVirtualization=!container74agent # [ 4474.436509] agent systemd[1]: Starting Save Transient machine-id to Disk...75agent # [ 4474.437318] agent systemd[1]: Starting Create System Files and Directories...76agent # [ 4474.454242] agent systemd-tmpfiles[161]: Cannot set file attributes for '/var/empty', value=0x00000010, mask=0x00000010, ignoring: Operation not permitted77agent # [ 4474.454649] agent systemd-tmpfiles[161]: fchmod() of /var/log/journal failed: Operation not permitted78agent # [ 4474.454944] agent systemd-tmpfiles[161]: fchmod() of /var/log/journal/bbdc25a12fed41c4858accd83e668c4b failed: Operation not permitted79agent # [ 4474.455287] agent systemd-tmpfiles[161]: fchmod() of /run/log/journal failed: Operation not permitted80client # [ 4474.389038] client systemd[1]: Mounting /run/wrappers...81client # [ 4474.442834] client systemd[1]: Mounted /run/wrappers.82client # [ 4474.443859] client systemd[1]: Reached target Local File Systems.83client # [ 4474.445207] client systemd[1]: Listening on Boot Loader Control Service Socket.84client # [ 4474.446593] client systemd[1]: Starting Create SUID/SGID Wrappers...85client # [ 4474.446643] client systemd[1]: Update Boot Loader Random Seed skipped, unmet condition check ConditionVirtualization=!container86client # [ 4474.447701] client systemd[1]: Starting Save Transient machine-id to Disk...87client # [ 4474.448825] client systemd[1]: Starting Create System Files and Directories...88client # [ 4474.466629] client systemd-tmpfiles[161]: Cannot set file attributes for '/var/empty', value=0x00000010, mask=0x00000010, ignoring: Operation not permitted89client # [ 4474.467047] client systemd-tmpfiles[161]: fchmod() of /var/log/journal failed: Operation not permitted90client # [ 4474.467360] client systemd-tmpfiles[161]: fchmod() of /var/log/journal/083857f1ee3347c0b2c47f98b0ab6489 failed: Operation not permitted91client # [ 4474.467704] client systemd-tmpfiles[161]: fchmod() of /run/log/journal failed: Operation not permitted92relay # [ 4474.384614] relay systemd[1]: Mounting /run/wrappers...93relay # [ 4474.447558] relay systemd[1]: Mounted /run/wrappers.94relay # [ 4474.448768] relay systemd[1]: Reached target Local File Systems.95relay # [ 4474.450176] relay systemd[1]: Listening on Boot Loader Control Service Socket.96relay # [ 4474.451373] relay systemd[1]: Starting Create SUID/SGID Wrappers...97relay # [ 4474.451422] relay systemd[1]: Update Boot Loader Random Seed skipped, unmet condition check ConditionVirtualization=!container98relay # [ 4474.452420] relay systemd[1]: Starting Save Transient machine-id to Disk...99relay # [ 4474.453444] relay systemd[1]: Starting Create System Files and Directories...100relay # [ 4474.471254] relay systemd-tmpfiles[169]: Cannot set file attributes for '/var/empty', value=0x00000010, mask=0x00000010, ignoring: Operation not permitted101relay # [ 4474.471652] relay systemd-tmpfiles[169]: fchmod() of /var/log/journal failed: Operation not permitted102relay # [ 4474.471949] relay systemd-tmpfiles[169]: fchmod() of /var/log/journal/ce1cc1ef91d649a5b86026ce32603edc failed: Operation not permitted103relay # [ 4474.472303] relay systemd-tmpfiles[169]: fchmod() of /run/log/journal failed: Operation not permitted104relay # [ 4474.472475] relay suid-sgid-wrappers-start[176]: chmod: changing permissions of '/run/wrappers/wrappers.rPcJe1jFKm/chsh': Operation not permitted105relay # [ 4474.479457] relay systemd[1]: suid-sgid-wrappers.service: Main process exited, code=exited, status=1/FAILURE106relay # [ 4474.479497] relay systemd[1]: suid-sgid-wrappers.service: Failed with result 'exit-code'.107relay # [ 4474.479548] relay systemd[1]: Failed to start Create SUID/SGID Wrappers.108relay # [ 4474.484966] relay systemd[1]: Finished Create System Files and Directories.109relay # [ 4474.562988] relay systemd[1]: etc-machine\x2did.mount: Deactivated successfully.110relay # [ 4474.582615] relay systemd[1]: Finished Save Transient machine-id to Disk.111relay # [ 4474.586511] relay systemd[1]: Starting Rebuild Journal Catalog...112relay # [ 4474.587341] relay systemd[1]: Starting Record System Boot/Shutdown in UTMP...113relay # [ 4474.634942] relay systemd[1]: Finished Record System Boot/Shutdown in UTMP.114agent # [ 4474.457532] agent systemd[1]: Finished Create System Files and Directories.115agent # [ 4474.460687] agent suid-sgid-wrappers-start[169]: chmod: changing permissions of '/run/wrappers/wrappers.kUkFjpisAc/chsh': Operation not permitted116agent # [ 4474.470315] agent systemd[1]: suid-sgid-wrappers.service: Main process exited, code=exited, status=1/FAILURE117agent # [ 4474.470531] agent systemd[1]: suid-sgid-wrappers.service: Failed with result 'exit-code'.118agent # [ 4474.470687] agent systemd[1]: Failed to start Create SUID/SGID Wrappers.119agent # [ 4474.472339] agent systemd[1]: Starting Rebuild Journal Catalog...120agent # [ 4474.473029] agent systemd[1]: Starting Record System Boot/Shutdown in UTMP...121agent # [ 4474.485213] agent systemd[1]: Finished Record System Boot/Shutdown in UTMP.122agent # [ 4474.489579] agent systemd[1]: Finished Rebuild Journal Catalog.123agent # [ 4474.490795] agent systemd[1]: Starting Update is Completed...124agent # [ 4474.499126] agent systemd[1]: Finished Update is Completed.125agent # [ 4474.499239] agent systemd[1]: Reached target System Initialization.126agent # [ 4474.499316] agent systemd[1]: Discard unused filesystem blocks once a week skipped, unmet condition check ConditionVirtualization=!container127agent # [ 4474.499341] agent systemd[1]: Started Daily Cleanup of Temporary Directories.128agent # [ 4474.499355] agent systemd[1]: Reached target Timer Units.129agent # [ 4474.499448] agent systemd[1]: Listening on D-Bus System Message Bus Socket.130agent # [ 4474.499638] agent systemd[1]: Listening on Nix Daemon Socket.131agent # [ 4474.499730] agent systemd[1]: Listening on Virtual Machine and Container Registration Service Socket.132agent # [ 4474.499744] agent systemd[1]: Reached target Socket Units.133agent # [ 4474.499771] agent systemd[1]: Reached target Basic System.134agent # [ 4474.500867] agent systemd[1]: Starting Re-link flakelet services at boot...135agent # [ 4474.501573] agent systemd[1]: Starting Import lastlog data into lastlog2 database...136agent # [ 4474.502477] agent systemd[1]: Starting Name Service Cache Daemon (nsncd)...137agent # [ 4474.503494] agent systemd[1]: Starting resolvconf update...138agent # [ 4474.511901] agent systemd[1]: Finished Re-link flakelet services at boot.139agent # [ 4474.521747] agent systemd[1]: Starting Reconcile flakelet services with the host configuration...140agent # [ 4474.565768] agent systemd[1]: etc-machine\x2did.mount: Deactivated successfully.141agent # [ 4474.566734] agent systemd[1]: Finished Save Transient machine-id to Disk.142agent # [ 4474.566925] agent systemd[1]: Finished Import lastlog data into lastlog2 database.143agent # [ 4474.567495] agent systemd[1]: Finished Reconcile flakelet services with the host configuration.144agent # [ 4474.592046] agent systemd[1]: nscd.service: Deactivated successfully.145agent # [ 4474.625430] agent systemd[1]: Stopped Name Service Cache Daemon (nsncd).146agent # [ 4474.627143] agent systemd[1]: Finished resolvconf update.147agent # [ 4474.627998] agent systemd[1]: Reached target Preparation for Network.148agent # [ 4474.629070] agent systemd[1]: Starting Address configuration of eth1...149agent # [ 4474.630259] agent systemd[1]: Starting Extra networking commands....150agent # [ 4474.631357] agent systemd[1]: Starting Name Service Cache Daemon (nsncd)...151client # [ 4474.469948] client suid-sgid-wrappers-start[168]: chmod: changing permissions of '/run/wrappers/wrappers.yjweE5IaPf/chsh': Operation not permitted152client # [ 4474.474606] client systemd[1]: suid-sgid-wrappers.service: Main process exited, code=exited, status=1/FAILURE153client # [ 4474.474647] client systemd[1]: suid-sgid-wrappers.service: Failed with result 'exit-code'.154client # [ 4474.474695] client systemd[1]: Failed to start Create SUID/SGID Wrappers.155client # [ 4474.509707] client systemd[1]: Finished Create System Files and Directories.156client # [ 4474.572217] client systemd[1]: etc-machine\x2did.mount: Deactivated successfully.157client # [ 4474.625581] client systemd[1]: Finished Save Transient machine-id to Disk.158client # [ 4474.629639] client systemd[1]: Starting Rebuild Journal Catalog...159client # [ 4474.630266] client systemd[1]: Starting Record System Boot/Shutdown in UTMP...160client # [ 4474.642783] client systemd[1]: Finished Record System Boot/Shutdown in UTMP.161client # [ 4474.650515] client systemd[1]: Finished Rebuild Journal Catalog.162client # [ 4474.652259] client systemd[1]: Starting Update is Completed...163client # [ 4474.662702] client systemd[1]: Finished Update is Completed.164client # [ 4474.662793] client systemd[1]: Reached target System Initialization.165client # [ 4474.662856] client systemd[1]: Discard unused filesystem blocks once a week skipped, unmet condition check ConditionVirtualization=!container166client # [ 4474.662880] client systemd[1]: Started Daily Cleanup of Temporary Directories.167client # [ 4474.662893] client systemd[1]: Reached target Timer Units.168client # [ 4474.662990] client systemd[1]: Listening on D-Bus System Message Bus Socket.169client # [ 4474.663184] client systemd[1]: Listening on Nix Daemon Socket.170client # [ 4474.663284] client systemd[1]: Listening on Virtual Machine and Container Registration Service Socket.171client # [ 4474.663297] client systemd[1]: Reached target Socket Units.172client # [ 4474.663325] client systemd[1]: Reached target Basic System.173client # [ 4474.664270] client systemd[1]: Starting Import lastlog data into lastlog2 database...174client # [ 4474.665069] client systemd[1]: Starting Name Service Cache Daemon (nsncd)...175client # [ 4474.665624] client systemd[1]: Starting resolvconf update...176client # [ 4474.679849] client systemd[1]: Finished Import lastlog data into lastlog2 database.177client # [ 4474.752144] client systemd[1]: nscd.service: Deactivated successfully.178client # [ 4474.752259] client systemd[1]: Stopped Name Service Cache Daemon (nsncd).179client # [ 4474.754847] client systemd[1]: Starting Name Service Cache Daemon (nsncd)...180client # [ 4474.757869] client systemd[1]: Finished resolvconf update.181client # [ 4474.757964] client systemd[1]: Reached target Preparation for Network.182client # [ 4474.758849] client systemd[1]: Starting Address configuration of eth1...183client # [ 4474.759742] client systemd[1]: Starting Extra networking commands....184client # [ 4474.779388] client network-addresses-eth1-start[237]: adding address 192.168.1.2/24... done185client # [ 4474.783234] client network-addresses-eth1-start[237]: adding address 2001:db8:1::2/64... done186client # [ 4474.788064] client systemd[1]: Finished Address configuration of eth1.187client # [ 4474.829835] client systemd[1]: Finished Extra networking commands..188client # [ 4474.829971] client systemd[1]: Reached target Network.189client # [ 4474.877149] client systemd[1]: nscd.service: Deactivated successfully.190client # [ 4474.877334] client systemd[1]: Stopped Name Service Cache Daemon (nsncd).191client # [ 4474.880220] client systemd[1]: Starting Name Service Cache Daemon (nsncd)...192agent # [ 4474.649769] agent network-addresses-eth1-start[240]: adding address 192.168.1.1/24... done193agent # [ 4474.653592] agent network-addresses-eth1-start[240]: adding address 2001:db8:1::1/64... done194agent # [ 4474.657547] agent systemd[1]: Finished Address configuration of eth1.195agent # [ 4474.707436] agent systemd[1]: Finished Extra networking commands..196agent # [ 4474.708111] agent systemd[1]: Reached target Network.197agent # [ 4474.709270] agent systemd[1]: Starting flakelet-agent...198agent # [ 4474.709977] agent systemd[1]: Starting Update flakelet service app...199agent # [ 4474.710639] agent systemd[1]: Starting Update flakelet service other...200agent # [ 4474.751768] agent systemd[1]: nscd.service: Deactivated successfully.201agent # [ 4474.751905] agent systemd[1]: Stopped Name Service Cache Daemon (nsncd).202agent # [ 4474.754589] agent systemd[1]: Starting Name Service Cache Daemon (nsncd)...203agent # [ 4474.759545] agent flakelet[305]: app: using prebuilt artifact /nix/store/hysydfr96ds875vr0jgw5rjxi4p39h2j-flakelet-app-v1204agent # [ 4474.759815] agent flakelet[306]: other: using prebuilt artifact /nix/store/hmr990n0b4v6rpzrcpyyw738r9xi7z95-flakelet-other-v1205agent # [ 4474.760121] agent flakelet[305]: app: activating generation 1206agent # [ 4474.760279] agent flakelet[306]: other: activating generation 1207agent # [ 4474.764987] agent systemd[1]: Reload requested from client PID 327 ('systemctl') (unit flakelet-other.service)...208agent # [ 4474.765028] agent systemd[1]: Reloading...209agent # [ 4474.793281] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:12.754929Z","level":"WARN","message":"SRV _flakelet-relay._tcp.test: lookup failed","target":"flakelet_agent"}210relay # [ 4474.642462] relay systemd[1]: Finished Rebuild Journal Catalog.211relay # [ 4474.643842] relay systemd[1]: Starting Update is Completed...212relay # [ 4474.656866] relay systemd[1]: Finished Update is Completed.213relay # [ 4474.656962] relay systemd[1]: Reached target System Initialization.214relay # [ 4474.657050] relay systemd[1]: Discard unused filesystem blocks once a week skipped, unmet condition check ConditionVirtualization=!container215relay # [ 4474.657070] relay systemd[1]: Started Daily Cleanup of Temporary Directories.216relay # [ 4474.657082] relay systemd[1]: Reached target Timer Units.217relay # [ 4474.657179] relay systemd[1]: Listening on D-Bus System Message Bus Socket.218relay # [ 4474.657347] relay systemd[1]: Listening on Nix Daemon Socket.219relay # [ 4474.657444] relay systemd[1]: Listening on Virtual Machine and Container Registration Service Socket.220relay # [ 4474.657458] relay systemd[1]: Reached target Socket Units.221relay # [ 4474.657487] relay systemd[1]: Reached target Basic System.222relay # [ 4474.658847] relay systemd[1]: Started device-mock.service.223relay # [ 4474.659631] relay systemd[1]: Starting Re-link flakelet services at boot...224relay # [ 4474.660406] relay systemd[1]: Starting Import lastlog data into lastlog2 database...225relay # [ 4474.661468] relay systemd[1]: Starting Name Service Cache Daemon (nsncd)...226relay # [ 4474.662191] relay systemd[1]: Starting resolvconf update...227relay # [ 4474.671103] relay systemd[1]: Finished Re-link flakelet services at boot.228relay # [ 4474.672105] relay systemd[1]: Starting Reconcile flakelet services with the host configuration...229relay # [ 4474.678095] relay systemd[1]: Finished Import lastlog data into lastlog2 database.230relay # [ 4474.683631] relay systemd[1]: Finished Reconcile flakelet services with the host configuration.231relay # [ 4474.761319] relay systemd[1]: nscd.service: Deactivated successfully.232relay # [ 4474.761513] relay systemd[1]: Stopped Name Service Cache Daemon (nsncd).233relay # [ 4474.763134] relay systemd[1]: Finished resolvconf update.234relay # [ 4474.763885] relay systemd[1]: Reached target Preparation for Network.235relay # [ 4474.764980] relay systemd[1]: Starting Address configuration of eth1...236relay # [ 4474.765641] relay systemd[1]: Starting Extra networking commands....237relay # [ 4474.766501] relay systemd[1]: Starting Name Service Cache Daemon (nsncd)...238relay # [ 4474.784154] relay network-addresses-eth1-start[249]: adding address 192.168.1.3/24... done239relay # [ 4474.787751] relay network-addresses-eth1-start[249]: adding address 2001:db8:1::3/64... done240relay # [ 4474.792169] relay systemd[1]: Finished Address configuration of eth1.241relay # [ 4474.831169] relay systemd[1]: Finished Extra networking commands..242relay # [ 4474.831260] relay systemd[1]: Reached target Network.243relay # [ 4474.832600] relay systemd[1]: Starting Dnsmasq Daemon...244relay # [ 4474.833523] relay systemd[1]: Starting Nginx Web Server...245relay # [ 4474.834357] relay systemd[1]: Starting Update flakelet service flakelet-relay...246relay # [ 4474.845099] relay flakelet[311]: flakelet-relay: using prebuilt artifact /nix/store/fg23fm1bd55b29023k2xz2vnvx84smyx-flakelet-flakelet-relay247relay # [ 4474.845734] relay flakelet[311]: flakelet-relay: activating generation 1248relay # [ 4474.849202] relay systemd[1]: Reload requested from client PID 314 ('systemctl') (unit flakelet-flakelet-relay.service)...249relay # [ 4474.849235] relay systemd[1]: Reloading...250relay # [ 4475.159457] relay nsncd[251]: Sep 06 08:19:13.121 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket"251relay # [ 4475.262189] relay systemd[1]: Reloading finished in 412 ms.252relay # [ 4475.404686] relay systemd[1]: Started Name Service Cache Daemon (nsncd).253relay # [ 4475.405797] relay systemd[1]: Reached target Host and Network Name Lookups.254relay # [ 4475.405904] relay systemd[1]: Reached target User and Group Name Lookups.255relay # [ 4475.407117] relay systemd[1]: Starting User Login Management...256relay # [ 4475.407997] relay systemd[1]: Starting Permit User Sessions...257relay # [ 4475.421349] relay systemd[1]: Reload requested from client PID 357 ('systemctl') (unit flakelet-flakelet-relay.service)...258relay # [ 4475.421386] relay systemd[1]: Reloading...259client # [ 4475.154606] client nsncd[303]: Sep 06 08:19:13.116 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket"260client # [ 4475.154626] client systemd[1]: Started Name Service Cache Daemon (nsncd).261client # [ 4475.154726] client systemd[1]: Reached target Host and Network Name Lookups.262client # [ 4475.154836] client systemd[1]: Reached target User and Group Name Lookups.263client # [ 4475.157402] client systemd[1]: Starting User Login Management...264client # [ 4475.158839] client systemd[1]: Starting Permit User Sessions...265client # [ 4475.194309] client systemd[1]: Finished Permit User Sessions.266client # [ 4475.196424] client systemd[1]: Started Console Getty.267client # [ 4475.196480] client systemd[1]: Getty on tty1 skipped, unmet condition check ConditionPathExists=/dev/tty0268client # [ 4475.196511] client systemd[1]: Reached target Login Prompts.269agent # [ 4475.194844] agent nsncd[311]: Sep 06 08:19:13.156 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket"270agent # [ 4475.275319] agent systemd[1]: Reloading finished in 509 ms.271agent # [ 4475.407202] agent systemd[1]: Started flakelet-agent.272agent # [ 4475.407383] agent systemd[1]: Started Name Service Cache Daemon (nsncd).273agent # [ 4475.407882] agent systemd[1]: Reload requested from client PID 325 ('systemctl') (unit flakelet-app.service)...274agent # [ 4475.407923] agent systemd[1]: Reloading...275relay # [ 4475.456229] relay dnsmasq-pre-start[372]: dnsmasq: syntax check OK.276relay # [ 4475.784769] relay systemd[1]: Reloading finished in 363 ms.277agent # [ 4475.777483] agent systemd[1]: Reloading finished in 369 ms.278agent # [ 4475.885503] agent systemd[1]: Reached target Host and Network Name Lookups.279agent # [ 4475.885624] agent systemd[1]: Reached target User and Group Name Lookups.280agent # [ 4475.887135] agent systemd[1]: Starting User Login Management...281agent # [ 4475.888022] agent systemd[1]: Starting Permit User Sessions...282agent # [ 4475.895522] agent systemd[1]: Reload requested from client PID 769 ('systemctl') (unit flakelet-other.service)...283agent # [ 4475.895553] agent systemd[1]: Reloading...284relay # [ 4475.886921] relay systemd[1]: Finished Permit User Sessions.285relay # [ 4475.921560] relay systemd[1]: Started Console Getty.286relay # [ 4475.921636] relay systemd[1]: Getty on tty1 skipped, unmet condition check ConditionPathExists=/dev/tty0287relay # [ 4475.921702] relay systemd[1]: Reached target Login Prompts.288relay # [ 4475.923462] relay systemd[1]: Starting flakelet-relay...289agent # [ 4476.280582] agent systemd[1]: Reloading finished in 384 ms.290agent # [ 4476.350377] agent systemd[1]: Finished Permit User Sessions.291agent # [ 4476.354566] agent systemd[1]: Started Console Getty.292agent # [ 4476.354628] agent systemd[1]: Getty on tty1 skipped, unmet condition check ConditionPathExists=/dev/tty0293agent # [ 4476.354695] agent systemd[1]: Reached target Login Prompts.294agent # [ 4476.358875] agent systemd[1]: Reload requested from client PID 805 ('systemctl') (unit flakelet-app.service)...295agent # [ 4476.358904] agent systemd[1]: Reloading...296client # [ 4476.487803] client systemd-logind[315]: New seat seat0.297client # [ 4476.488358] client systemd[1]: Started User Login Management.298client # [ 4476.490558] client systemd[1]: Starting D-Bus System Message Bus...299client # [ 4476.491858] client systemd[1]: Starting linger-users.service...300client # [ 4476.528081] client systemd[1]: linger-users.service: Deactivated successfully.301client # [ 4476.528451] client systemd[1]: Finished linger-users.service.302client # [ 4476.528792] client systemd[1]: Reached target Multi-User System.303relay # [ 4476.494466] relay nginx-pre-start[402]: nginx: the configuration file /nix/store/01avjj26wcihp926xgasszcj5s0k7cbs-nginx.conf syntax is ok304relay # [ 4476.495188] relay nginx-pre-start[402]: nginx: configuration file /nix/store/01avjj26wcihp926xgasszcj5s0k7cbs-nginx.conf test is successful305relay # [ 4476.518781] relay systemd[1]: Started Nginx Web Server.306relay # [ 4476.590709] relay systemd[1]: Starting D-Bus System Message Bus...307agent # [ 4476.709391] agent systemd[1]: Reloading finished in 350 ms.308agent # [ 4476.780368] agent systemd[1]: Starting other v1...309agent # [ 4476.805989] agent systemd[1]: Starting app v1...310agent # [ 4476.809253] agent systemd[1]: Started other v1.311agent # [ 4476.812877] agent systemd[1]: Started app v1.312agent # [ 4476.824995] agent flakelet[306]: other: updated to generation 1313agent # [ 4476.825615] agent systemd[1]: Finished Update flakelet service other.314agent # [ 4476.828602] agent flakelet[305]: app: updated to generation 1315agent # [ 4476.830093] agent systemd[1]: Finished Update flakelet service app.316agent # [ 4476.830203] agent systemd[1]: Reached target flakelet managed services.317relay # [ 4476.867498] relay systemd-logind[354]: New seat seat0.318relay # [ 4476.867815] relay systemd[1]: Started User Login Management.319relay # [ 4476.869899] relay systemd[1]: Starting linger-users.service...320relay # [ 4476.901778] relay systemd[1]: linger-users.service: Deactivated successfully.321relay # [ 4476.902073] relay systemd[1]: Finished linger-users.service.322relay # [ 4477.100132] relay dbus-broker-launch[408]: Looking up NSS user entry for 'systemd-timesync'...323relay # [ 4477.100932] relay dbus-broker-launch[408]: NSS returned no entry for 'systemd-timesync'324relay # [ 4477.100932] relay dbus-broker-launch[408]: Invalid user-name in /nix/store/27h4g2jdw18vrabfh6p1l0n8ss5dhbp5-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync"325relay # [ 4477.101558] relay systemd[1]: Started D-Bus System Message Bus.326relay # [ 4477.106993] relay dbus-broker-launch[408]: Ready327relay # [ 4477.110193] relay systemd[1]: Started Dnsmasq Daemon.328relay # [ 4477.112793] relay dnsmasq[394]: started, version 2.93 cachesize 150329relay # [ 4477.112811] relay dnsmasq[394]: compile time options: IPv6 GNU-getopt DBus no-UBus no-i18n IDN2 DHCP DHCPv6 no-Lua TFTP conntrack ipset nftset auth DNSSEC loop-detect inotify dumpfile330relay # [ 4477.112826] relay dnsmasq[394]: DBus support enabled: connected to system bus331relay # [ 4477.112840] relay dnsmasq[394]: no upstream servers configured - please set them from DBus332relay # [ 4477.112984] relay dnsmasq[394]: read /etc/hosts - 9 names333client # [ 4477.035700] client dbus-broker-launch[325]: Looking up NSS user entry for 'systemd-timesync'...334client # [ 4477.037065] client dbus-broker-launch[325]: NSS returned no entry for 'systemd-timesync'335client # [ 4477.037065] client dbus-broker-launch[325]: Invalid user-name in /nix/store/pgfa8wj70mb4y0q358afylir6nilgvmb-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync"336client # [ 4477.037880] client systemd[1]: Started D-Bus System Message Bus.337client # [ 4477.038367] client systemd[1]: Startup finished in 4.092s.338client # [ 4477.043839] client dbus-broker-launch[325]: Ready339agent # [ 4477.358599] agent systemd-logind[767]: New seat seat0.340agent # [ 4477.360143] agent systemd[1]: Starting D-Bus System Message Bus...341agent # [ 4477.360205] agent systemd[1]: Started User Login Management.342agent # [ 4477.361242] agent systemd[1]: Starting linger-users.service...343agent # [ 4477.398542] agent systemd[1]: linger-users.service: Deactivated successfully.344agent # [ 4477.398744] agent systemd[1]: Finished linger-users.service.345agent # [ 4477.398930] agent systemd[1]: Reached target Multi-User System.346agent # [ 4477.777347] agent dbus-broker-launch[845]: Looking up NSS user entry for 'systemd-timesync'...347agent # [ 4477.778527] agent dbus-broker-launch[845]: NSS returned no entry for 'systemd-timesync'348agent # [ 4477.778527] agent dbus-broker-launch[845]: Invalid user-name in /nix/store/q3c2gxfm178wqgzv8m87na4aa72wxc04-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync"349agent # [ 4477.779647] agent systemd[1]: Started D-Bus System Message Bus.350agent # [ 4477.780189] agent systemd[1]: Startup finished in 4.831s.351agent # [ 4477.786446] agent dbus-broker-launch[845]: Ready352relay # [ 4477.892555] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:15.854170Z","level":"INFO","message":"listening on 127.0.0.1:7400","target":"flakelet_relay::relay::server"}353relay # [ 4477.893457] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:15.854789Z","level":"INFO","message":"listening on [::]:7443","target":"flakelet_relay::relay::server"}354relay # [ 4477.893068] relay systemd[1]: Started flakelet-relay.355relay # [ 4477.900228] relay systemd[1]: Starting health for flakelet-relay...356relay: (finished: waiting for unit flakelet-relay.service, in 6.16 seconds)357relay: waiting for unit nginx.service358relay # [ 4478.149772] relay systemd[1]: flakelet-relay-health.service: Deactivated successfully.359relay # [ 4478.150123] relay systemd[1]: Finished health for flakelet-relay.360relay # [ 4478.192913] relay flakelet[311]: flakelet-relay: updated to generation 1361relay # [ 4478.195750] relay systemd[1]: Finished Update flakelet service flakelet-relay.362relay # [ 4478.196297] relay systemd[1]: Reached target flakelet managed services.363relay # [ 4478.196471] relay systemd[1]: Reached target Multi-User System.364relay # [ 4478.196726] relay systemd[1]: Startup finished in 5.250s.365relay: (finished: waiting for unit nginx.service, in 0.02 seconds)366agent: waiting for unit flakelet.target367agent: (finished: waiting for unit flakelet.target, in 0.02 seconds)368agent: waiting for unit flakelet-agent.service369agent: (finished: waiting for unit flakelet-agent.service, in 0.02 seconds)370client: waiting for unit multi-user.target371client: (finished: waiting for unit multi-user.target, in 0.02 seconds)372relay: waiting for success: curl -sf http://127.0.0.1:7400/metrics | grep -q 'agent_up{host="agent"} 1'373agent # [ 4479.814456] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:17.776218Z","level":"INFO","message":"connected","relay":"https://relay:7443","relay_name":"relay","relay_version":"0.1.0","host":"agent","target":"flakelet_relay::agent::conn"}374relay # [ 4479.814850] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:17.776577Z","level":"INFO","message":"agent connected","host":"agent","version":"0.1.0","flakelets":"[Named { name: \"app\", generation: Some(1), revision: Some(\"prebuilt:app\") }, Named { name: \"other\", generation: Some(1), revision: Some(\"prebuilt:other\") }]","target":"flakelet_relay::relay::agent_conn"}375relay: (finished: waiting for success: curl -sf http://127.0.0.1:7400/metrics | grep -q 'agent_up{host="agent"} 1', in 2.05 seconds)376agent: must succeed: systemctl is-active app.service other.service377agent: (finished: must succeed: systemctl is-active app.service other.service, in 0.01 seconds)378subtest: deploy with client certificate, relay found via SRV379agent: must succeed: jq '.services.app.prebuilt="/nix/store/6ili1vxjlag9d7dy0bxrbxk6bpi5qh9p-flakelet-app-v2"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json380agent: (finished: must succeed: jq '.services.app.prebuilt="/nix/store/6ili1vxjlag9d7dy0bxrbxk6bpi5qh9p-flakelet-app-v2"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json, in 0.01 seconds)381client: must succeed: push --relay-srv test --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy agent/app 2>&1382agent # [ 4480.792257] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:18.754018Z","level":"INFO","message":"start","id":"8ea3a03b211f684880093e2c3bcf81df","flakelet":"app","rule":"admins","relay":"relay","caller":"Some(\"x509:email:admin@example.org\")","target":"flakelet_relay::agent::conn"}383agent # [ 4480.793282] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:18.755085Z","level":"INFO","message":"update starting","flakelet":"app","ids":"[\"8ea3a03b211f684880093e2c3bcf81df\"]","target":"flakelet_relay::agent::jobs"}384agent # [ 4480.855227] agent systemd[1]: Starting flakelet update app (via relay)...385agent # [ 4480.861818] agent systemd[1]: Started flakelet update app (via relay).386agent # [ 4480.865820] agent flakelet[882]: app: using prebuilt artifact /nix/store/6ili1vxjlag9d7dy0bxrbxk6bpi5qh9p-flakelet-app-v2387agent # [ 4480.866468] agent flakelet[882]: app: activating generation 2388agent # [ 4480.879311] agent systemd[1]: Stopping app v1...389agent # [ 4480.879992] agent systemd[1]: app.service: Deactivated successfully.390agent # [ 4480.880115] agent systemd[1]: Stopped app v1.391agent # [ 4480.890791] agent systemd[1]: Reload requested from client PID 891 ('systemctl') (unit flakelet-relay-job-app.service)...392agent # [ 4480.890825] agent systemd[1]: Reloading...393relay # [ 4480.791718] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:18.753463Z","level":"INFO","message":"deploy accepted","job":"d30c8ccd9850f8cfcbe259628ad443a4","caller":"x509:email:admin@example.org","targets":"[\"agent/app\"]","target":"flakelet_relay::relay::api"}394agent # [ 4481.219637] agent systemd[1]: Reloading finished in 328 ms.395agent # [ 4481.352761] agent systemd[1]: Reload requested from client PID 921 ('systemctl') (unit flakelet-relay-job-app.service)...396agent # [ 4481.352798] agent systemd[1]: Reloading...397agent # [ 4481.679851] agent systemd[1]: Reloading finished in 326 ms.398agent # [ 4481.756985] agent systemd[1]: Reload requested from client PID 950 ('systemctl') (unit flakelet-relay-job-app.service)...399agent # [ 4481.757031] agent systemd[1]: Reloading...400agent # [ 4482.082633] agent systemd[1]: Reloading finished in 325 ms.401agent # [ 4482.168462] agent systemd[1]: Reload requested from client PID 981 ('systemctl') (unit flakelet-relay-job-app.service)...402agent # [ 4482.168496] agent systemd[1]: Reloading...403agent # [ 4482.493385] agent systemd[1]: Reloading finished in 324 ms.404agent # [ 4482.575716] agent systemd[1]: Starting app v2...405agent # [ 4482.604903] agent systemd[1]: Started app v2.406agent # [ 4482.619010] agent flakelet[882]: app: updated to generation 2407agent # [ 4482.620891] agent systemd[1]: flakelet-relay-job-app.service: Deactivated successfully.408client: (finished: must succeed: push --relay-srv test --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy agent/app 2>&1, in 2.95 seconds)409agent: must succeed: systemctl show app.service -p Description | grep -q 'app v2'410agent: (finished: must succeed: systemctl show app.service -p Description | grep -q 'app v2', in 0.02 seconds)411agent: must succeed: journalctl -u flakelet-relay-job-app.service | grep -q 'generation 2'412agent: (finished: must succeed: journalctl -u flakelet-relay-job-app.service | grep -q 'generation 2', in 0.01 seconds)413(finished: subtest: deploy with client certificate, relay found via SRV, in 3.00 seconds)414subtest: deploy with OIDC token, sub principal415agent: must succeed: jq '.services.app.prebuilt="/nix/store/hysydfr96ds875vr0jgw5rjxi4p39h2j-flakelet-app-v1"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json416agent: (finished: must succeed: jq '.services.app.prebuilt="/nix/store/hysydfr96ds875vr0jgw5rjxi4p39h2j-flakelet-app-v1"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json, in 0.01 seconds)417client: must succeed: echo '{"groups": []}' | step crypto jwt sign --key /nix/store/r7sclamg09v434w8zjlfnrcrn1w79wh6-flakelet-relay-test-issuer/jwk.json --iss https://relay:8443 --aud flakelet-relay --sub 'repo:github:example/app:ref:refs/heads/main' --exp $(( $(date +%s) + 300 )) --jti $(head -c8 /dev/urandom | base32)418client: (finished: must succeed: echo '{"groups": []}' | step crypto jwt sign --key /nix/store/r7sclamg09v434w8zjlfnrcrn1w79wh6-flakelet-relay-test-issuer/jwk.json --iss https://relay:8443 --aud flakelet-relay --sub 'repo:github:example/app:ref:refs/heads/main' --exp $(( $(date +%s) + 300 )) --jti $(head -c8 /dev/urandom | base32), in 0.05 seconds)419client: must succeed: FLAKELET_RELAY_TOKEN_COMMAND='echo eyJhbGciOiJFZERTQSIsImtpZCI6ImsxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJmbGFrZWxldC1yZWxheSIsImV4cCI6MTc4ODY4MzA2MSwiZ3JvdXBzIjpbXSwiaWF0IjoxNzg4NjgyNzYxLCJpc3MiOiJodHRwczovL3JlbGF5Ojg0NDMiLCJqdGkiOiI2M0g1SFIyUDNZMlZDPT09IiwibmJmIjoxNzg4NjgyNzYxLCJzdWIiOiJyZXBvOmdpdGh1YjpleGFtcGxlL2FwcDpyZWY6cmVmcy9oZWFkcy9tYWluIn0.2MyrB94AoGvWlHKiSoKZdNE_C_aooQVRkLvsjrTxFIrIMxgwf0JrFjYOi2BwISPgllRD3HBK9nBFaU-iotvTAQ' push --relay https://relay:7443 deploy agent/app 2>&1420agent # [ 4483.716687] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:21.678433Z","level":"INFO","message":"update finished","flakelet":"app","status":"Updated","generation":"Some(2)","target":"flakelet_relay::agent::jobs"}421agent # [ 4483.841090] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:21.802874Z","level":"INFO","message":"start","id":"90a362b4e152f6b56ec01f29e11d0b79","flakelet":"app","rule":"ci","relay":"relay","caller":"Some(\"oidc:mock:repo:github:example/app:ref:refs/heads/main\")","target":"flakelet_relay::agent::conn"}422agent # [ 4483.841430] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:21.803254Z","level":"INFO","message":"update starting","flakelet":"app","ids":"[\"90a362b4e152f6b56ec01f29e11d0b79\"]","target":"flakelet_relay::agent::jobs"}423agent # [ 4483.882237] agent systemd[1]: Starting flakelet update app (via relay)...424relay # [ 4483.717805] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:21.679553Z","level":"INFO","message":"deploy finished","job":"d30c8ccd9850f8cfcbe259628ad443a4","ok":true,"target":"flakelet_relay::relay::api"}425relay # [ 4483.840821] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:21.802625Z","level":"INFO","message":"deploy accepted","job":"5d5e8cac9596c0da9bc30c6abf2d63ec","caller":"oidc:mock:repo:github:example/app:ref:refs/heads/main","targets":"[\"agent/app\"]","target":"flakelet_relay::relay::api"}426agent # [ 4484.075405] agent systemd[1]: Started flakelet update app (via relay).427agent # [ 4484.080646] agent flakelet[1059]: app: using prebuilt artifact /nix/store/hysydfr96ds875vr0jgw5rjxi4p39h2j-flakelet-app-v1428agent # [ 4484.081654] agent flakelet[1059]: app: activating generation 3429agent # [ 4484.091714] agent systemd[1]: Stopping app v2...430agent # [ 4484.092410] agent systemd[1]: app.service: Deactivated successfully.431agent # [ 4484.092539] agent systemd[1]: Stopped app v2.432agent # [ 4484.104145] agent systemd[1]: Reload requested from client PID 1066 ('systemctl') (unit flakelet-relay-job-app.service)...433agent # [ 4484.104177] agent systemd[1]: Reloading...434agent # [ 4484.431744] agent systemd[1]: Reloading finished in 327 ms.435agent # [ 4484.495070] agent systemd[1]: Reload requested from client PID 1096 ('systemctl') (unit flakelet-relay-job-app.service)...436agent # [ 4484.495102] agent systemd[1]: Reloading...437agent # [ 4484.819728] agent systemd[1]: Reloading finished in 324 ms.438agent # [ 4484.877978] agent systemd[1]: Reload requested from client PID 1128 ('systemctl') (unit flakelet-relay-job-app.service)...439agent # [ 4484.878015] agent systemd[1]: Reloading...440agent # [ 4485.205872] agent systemd[1]: Reloading finished in 327 ms.441agent # [ 4485.283027] agent systemd[1]: Reload requested from client PID 1160 ('systemctl') (unit flakelet-relay-job-app.service)...442agent # [ 4485.283061] agent systemd[1]: Reloading...443agent # [ 4485.605594] agent systemd[1]: Reloading finished in 322 ms.444agent # [ 4485.675660] agent systemd[1]: Starting app v1...445agent # [ 4485.707103] agent systemd[1]: Started app v1.446agent # [ 4485.719454] agent flakelet[1059]: app: updated to generation 3447agent # [ 4485.721121] agent systemd[1]: flakelet-relay-job-app.service: Deactivated successfully.448client: (finished: must succeed: FLAKELET_RELAY_TOKEN_COMMAND='echo eyJhbGciOiJFZERTQSIsImtpZCI6ImsxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJmbGFrZWxldC1yZWxheSIsImV4cCI6MTc4ODY4MzA2MSwiZ3JvdXBzIjpbXSwiaWF0IjoxNzg4NjgyNzYxLCJpc3MiOiJodHRwczovL3JlbGF5Ojg0NDMiLCJqdGkiOiI2M0g1SFIyUDNZMlZDPT09IiwibmJmIjoxNzg4NjgyNzYxLCJzdWIiOiJyZXBvOmdpdGh1YjpleGFtcGxlL2FwcDpyZWY6cmVmcy9oZWFkcy9tYWluIn0.2MyrB94AoGvWlHKiSoKZdNE_C_aooQVRkLvsjrTxFIrIMxgwf0JrFjYOi2BwISPgllRD3HBK9nBFaU-iotvTAQ' push --relay https://relay:7443 deploy agent/app 2>&1, in 3.02 seconds)449(finished: subtest: deploy with OIDC token, sub principal, in 3.08 seconds)450subtest: policy denies uncovered target451client: must fail: FLAKELET_RELAY_TOKEN_COMMAND='echo eyJhbGciOiJFZERTQSIsImtpZCI6ImsxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJmbGFrZWxldC1yZWxheSIsImV4cCI6MTc4ODY4MzA2MSwiZ3JvdXBzIjpbXSwiaWF0IjoxNzg4NjgyNzYxLCJpc3MiOiJodHRwczovL3JlbGF5Ojg0NDMiLCJqdGkiOiI2M0g1SFIyUDNZMlZDPT09IiwibmJmIjoxNzg4NjgyNzYxLCJzdWIiOiJyZXBvOmdpdGh1YjpleGFtcGxlL2FwcDpyZWY6cmVmcy9oZWFkcy9tYWluIn0.2MyrB94AoGvWlHKiSoKZdNE_C_aooQVRkLvsjrTxFIrIMxgwf0JrFjYOi2BwISPgllRD3HBK9nBFaU-iotvTAQ' push --relay https://relay:7443 deploy agent/other 2>&1452client: (finished: must fail: FLAKELET_RELAY_TOKEN_COMMAND='echo eyJhbGciOiJFZERTQSIsImtpZCI6ImsxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJmbGFrZWxldC1yZWxheSIsImV4cCI6MTc4ODY4MzA2MSwiZ3JvdXBzIjpbXSwiaWF0IjoxNzg4NjgyNzYxLCJpc3MiOiJodHRwczovL3JlbGF5Ojg0NDMiLCJqdGkiOiI2M0g1SFIyUDNZMlZDPT09IiwibmJmIjoxNzg4NjgyNzYxLCJzdWIiOiJyZXBvOmdpdGh1YjpleGFtcGxlL2FwcDpyZWY6cmVmcy9oZWFkcy9tYWluIn0.2MyrB94AoGvWlHKiSoKZdNE_C_aooQVRkLvsjrTxFIrIMxgwf0JrFjYOi2BwISPgllRD3HBK9nBFaU-iotvTAQ' push --relay https://relay:7443 deploy agent/other 2>&1, in 0.02 seconds)453(finished: subtest: policy denies uncovered target, in 0.02 seconds)454subtest: claim principal from groups, unchanged counts as success455client: must succeed: echo '{"groups": ["wheel"]}' | step crypto jwt sign --key /nix/store/r7sclamg09v434w8zjlfnrcrn1w79wh6-flakelet-relay-test-issuer/jwk.json --iss https://relay:8443 --aud flakelet-relay --sub 'someone' --exp $(( $(date +%s) + 300 )) --jti $(head -c8 /dev/urandom | base32)456client: (finished: must succeed: echo '{"groups": ["wheel"]}' | step crypto jwt sign --key /nix/store/r7sclamg09v434w8zjlfnrcrn1w79wh6-flakelet-relay-test-issuer/jwk.json --iss https://relay:8443 --aud flakelet-relay --sub 'someone' --exp $(( $(date +%s) + 300 )) --jti $(head -c8 /dev/urandom | base32), in 0.05 seconds)457client: must succeed: FLAKELET_RELAY_TOKEN_COMMAND='echo eyJhbGciOiJFZERTQSIsImtpZCI6ImsxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJmbGFrZWxldC1yZWxheSIsImV4cCI6MTc4ODY4MzA2NCwiZ3JvdXBzIjpbIndoZWVsIl0sImlhdCI6MTc4ODY4Mjc2NCwiaXNzIjoiaHR0cHM6Ly9yZWxheTo4NDQzIiwianRpIjoiVEZURVQ2VjM0RVJCVT09PSIsIm5iZiI6MTc4ODY4Mjc2NCwic3ViIjoic29tZW9uZSJ9.E0pUXNcHewmb9QSQMPCQDlRdohKU5ezHOcSAqapusIqYOc5xPkAfdtFVeju0f3k1y51AwQdEF5Y2hLYPfGiVBw' push --relay https://relay:7443 deploy agent/other 2>&1458agent # [ 4486.825920] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:24.787696Z","level":"INFO","message":"update finished","flakelet":"app","status":"Updated","generation":"Some(3)","target":"flakelet_relay::agent::jobs"}459agent # [ 4486.919726] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:24.881482Z","level":"INFO","message":"start","id":"2fa777ab29b029159d6c3c0e355da55b","flakelet":"other","rule":"admins","relay":"relay","caller":"Some(\"oidc:mock:groups:wheel\\noidc:mock:someone\")","target":"flakelet_relay::agent::conn"}460agent # [ 4486.920110] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:24.881933Z","level":"INFO","message":"update starting","flakelet":"other","ids":"[\"2fa777ab29b029159d6c3c0e355da55b\"]","target":"flakelet_relay::agent::jobs"}461agent # [ 4486.960197] agent systemd[1]: Starting flakelet update other (via relay)...462agent # [ 4486.991020] agent systemd[1]: Started flakelet update other (via relay).463agent # [ 4486.995404] agent flakelet[1235]: other: using prebuilt artifact /nix/store/hmr990n0b4v6rpzrcpyyw738r9xi7z95-flakelet-other-v1464agent # [ 4486.995900] agent flakelet[1235]: other: up to date465agent # [ 4486.996428] agent systemd[1]: flakelet-relay-job-other.service: Deactivated successfully.466relay # [ 4486.827284] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:24.789032Z","level":"INFO","message":"deploy finished","job":"5d5e8cac9596c0da9bc30c6abf2d63ec","ok":true,"target":"flakelet_relay::relay::api"}467relay # [ 4486.919473] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:24.881197Z","level":"INFO","message":"deploy accepted","job":"9a2b7dd3805b54e0b139af73f1e9759b","caller":"oidc:mock:groups:wheel","targets":"[\"agent/other\"]","target":"flakelet_relay::relay::api"}468client: (finished: must succeed: FLAKELET_RELAY_TOKEN_COMMAND='echo eyJhbGciOiJFZERTQSIsImtpZCI6ImsxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJmbGFrZWxldC1yZWxheSIsImV4cCI6MTc4ODY4MzA2NCwiZ3JvdXBzIjpbIndoZWVsIl0sImlhdCI6MTc4ODY4Mjc2NCwiaXNzIjoiaHR0cHM6Ly9yZWxheTo4NDQzIiwianRpIjoiVEZURVQ2VjM0RVJCVT09PSIsIm5iZiI6MTc4ODY4Mjc2NCwic3ViIjoic29tZW9uZSJ9.E0pUXNcHewmb9QSQMPCQDlRdohKU5ezHOcSAqapusIqYOc5xPkAfdtFVeju0f3k1y51AwQdEF5Y2hLYPfGiVBw' push --relay https://relay:7443 deploy agent/other 2>&1, in 1.65 seconds)469(finished: subtest: claim principal from groups, unchanged counts as success, in 1.71 seconds)470subtest: push login via device flow, cached token used afterwards471client: must succeed: push login --issuer https://relay:8443 --client-id flakelet-relay 2>&1472agent # [ 4488.552643] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:26.514380Z","level":"INFO","message":"update finished","flakelet":"other","status":"Unchanged","generation":"Some(1)","target":"flakelet_relay::agent::jobs"}473relay # [ 4488.554139] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:26.515771Z","level":"INFO","message":"deploy finished","job":"9a2b7dd3805b54e0b139af73f1e9759b","ok":true,"target":"flakelet_relay::relay::api"}474relay # [ 4488.577730] relay 93sn3sv24fj26ksccpppzwg6kbf10s4w-device-mock[186]: 127.0.0.1 - - [06/Sep/2026 08:19:26] "POST /device HTTP/1.1" 200 -475relay # [ 4489.583825] relay 93sn3sv24fj26ksccpppzwg6kbf10s4w-device-mock[186]: 127.0.0.1 - - [06/Sep/2026 08:19:27] "POST /token HTTP/1.1" 400 -476client: (finished: must succeed: push login --issuer https://relay:8443 --client-id flakelet-relay 2>&1, in 2.08 seconds)477client: must succeed: test "$(stat -c %a /root/.local/state/flakelet-push/token.json)" = 600478client: (finished: must succeed: test "$(stat -c %a /root/.local/state/flakelet-push/token.json)" = 600, in 0.01 seconds)479client: must succeed: push --relay https://relay:7443 deploy agent/other 2>&1480agent # [ 4490.670283] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:28.632024Z","level":"INFO","message":"start","id":"b57846a3f70181fa8a4da9f8bf64f448","flakelet":"other","rule":"admins","relay":"relay","caller":"Some(\"oidc:mock:groups:wheel\\noidc:mock:someone\")","target":"flakelet_relay::agent::conn"}481agent # [ 4490.671233] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:28.632652Z","level":"INFO","message":"update starting","flakelet":"other","ids":"[\"b57846a3f70181fa8a4da9f8bf64f448\"]","target":"flakelet_relay::agent::jobs"}482agent # [ 4490.724992] agent systemd[1]: Starting flakelet update other (via relay)...483agent # [ 4490.756952] agent systemd[1]: Started flakelet update other (via relay).484agent # [ 4490.761693] agent flakelet[1274]: other: using prebuilt artifact /nix/store/hmr990n0b4v6rpzrcpyyw738r9xi7z95-flakelet-other-v1485agent # [ 4490.762173] agent flakelet[1274]: other: up to date486agent # [ 4490.764825] agent systemd[1]: flakelet-relay-job-other.service: Deactivated successfully.487relay # [ 4490.637189] relay 93sn3sv24fj26ksccpppzwg6kbf10s4w-device-mock[186]: 127.0.0.1 - - [06/Sep/2026 08:19:28] "POST /token HTTP/1.1" 200 -488relay # [ 4490.669912] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:28.631717Z","level":"INFO","message":"deploy accepted","job":"df334c4fa6725c4bc85aec269c56ca12","caller":"oidc:mock:groups:wheel","targets":"[\"agent/other\"]","target":"flakelet_relay::relay::api"}489client: (finished: must succeed: push --relay https://relay:7443 deploy agent/other 2>&1, in 0.66 seconds)490client: must succeed: push --relay https://relay:7443 deploy agent/other 2>&1491agent # [ 4491.308568] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:29.270333Z","level":"INFO","message":"update finished","flakelet":"other","status":"Unchanged","generation":"Some(1)","target":"flakelet_relay::agent::jobs"}492agent # [ 4491.329373] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:29.291179Z","level":"INFO","message":"start","id":"a5922f33efb0ab418d9afa0456e5b93e","flakelet":"other","rule":"admins","relay":"relay","caller":"Some(\"oidc:mock:groups:wheel\\noidc:mock:someone\")","target":"flakelet_relay::agent::conn"}493agent # [ 4491.329754] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:29.291579Z","level":"INFO","message":"update starting","flakelet":"other","ids":"[\"a5922f33efb0ab418d9afa0456e5b93e\"]","target":"flakelet_relay::agent::jobs"}494agent # [ 4491.366555] agent systemd[1]: Starting flakelet update other (via relay)...495agent # [ 4491.414122] agent systemd[1]: Started flakelet update other (via relay).496agent # [ 4491.418724] agent flakelet[1312]: other: using prebuilt artifact /nix/store/hmr990n0b4v6rpzrcpyyw738r9xi7z95-flakelet-other-v1497agent # [ 4491.419229] agent flakelet[1312]: other: up to date498agent # [ 4491.421026] agent systemd[1]: flakelet-relay-job-other.service: Deactivated successfully.499relay # [ 4491.310119] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:29.271834Z","level":"INFO","message":"deploy finished","job":"df334c4fa6725c4bc85aec269c56ca12","ok":true,"target":"flakelet_relay::relay::api"}500relay # [ 4491.328988] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:29.290803Z","level":"INFO","message":"deploy accepted","job":"c7858ad5d61ad5e6dd238530f01d5664","caller":"oidc:mock:groups:wheel","targets":"[\"agent/other\"]","target":"flakelet_relay::relay::api"}501client: (finished: must succeed: push --relay https://relay:7443 deploy agent/other 2>&1, in 0.65 seconds)502(finished: subtest: push login via device flow, cached token used afterwards, in 3.41 seconds)503subtest: waves: failing first wave skips the second504agent: must succeed: jq '.services.app.prebuilt="/nix/store/8282rnbvyk847q4l4zby3zwcq3jpzrvw-flakelet-app-broken"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json505agent: (finished: must succeed: jq '.services.app.prebuilt="/nix/store/8282rnbvyk847q4l4zby3zwcq3jpzrvw-flakelet-app-broken"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json, in 0.02 seconds)506client: must fail: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy agent/app --wave agent/other 2>&1507agent # [ 4491.961407] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:29.923179Z","level":"INFO","message":"update finished","flakelet":"other","status":"Unchanged","generation":"Some(1)","target":"flakelet_relay::agent::jobs"}508agent # [ 4491.997083] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:29.958884Z","level":"INFO","message":"start","id":"39f02f08c2eada80dc0e2d7433d8d7f5","flakelet":"app","rule":"admins","relay":"relay","caller":"Some(\"x509:email:admin@example.org\")","target":"flakelet_relay::agent::conn"}509agent # [ 4491.997465] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:29.959288Z","level":"INFO","message":"update starting","flakelet":"app","ids":"[\"39f02f08c2eada80dc0e2d7433d8d7f5\"]","target":"flakelet_relay::agent::jobs"}510agent # [ 4492.035789] agent systemd[1]: Starting flakelet update app (via relay)...511agent # [ 4492.085227] agent systemd[1]: Started flakelet update app (via relay).512agent # [ 4492.090041] agent flakelet[1343]: app: using prebuilt artifact /nix/store/8282rnbvyk847q4l4zby3zwcq3jpzrvw-flakelet-app-broken513agent # [ 4492.090693] agent flakelet[1343]: app: activating generation 4514agent # [ 4492.100381] agent systemd[1]: Stopping app v1...515agent # [ 4492.101093] agent systemd[1]: app.service: Deactivated successfully.516agent # [ 4492.101215] agent systemd[1]: Stopped app v1.517agent # [ 4492.110174] agent systemd[1]: Reload requested from client PID 1350 ('systemctl') (unit flakelet-relay-job-app.service)...518agent # [ 4492.110207] agent systemd[1]: Reloading...519relay # [ 4491.963075] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:29.924800Z","level":"INFO","message":"deploy finished","job":"c7858ad5d61ad5e6dd238530f01d5664","ok":true,"target":"flakelet_relay::relay::api"}520relay # [ 4491.996819] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:29.958623Z","level":"INFO","message":"deploy accepted","job":"75ef5f5be4a5fe966263e5925f6745cc","caller":"x509:email:admin@example.org","targets":"[\"agent/app\", \"agent/other\"]","target":"flakelet_relay::relay::api"}521agent # [ 4492.437420] agent systemd[1]: Reloading finished in 326 ms.522agent # [ 4492.505049] agent systemd[1]: Reload requested from client PID 1380 ('systemctl') (unit flakelet-relay-job-app.service)...523agent # [ 4492.505080] agent systemd[1]: Reloading...524agent # [ 4492.826174] agent systemd[1]: Reloading finished in 320 ms.525agent # [ 4492.894205] agent systemd[1]: Reload requested from client PID 1411 ('systemctl') (unit flakelet-relay-job-app.service)...526agent # [ 4492.894239] agent systemd[1]: Reloading...527agent # [ 4493.217369] agent systemd[1]: Reloading finished in 322 ms.528agent # [ 4493.296993] agent systemd[1]: Reload requested from client PID 1444 ('systemctl') (unit flakelet-relay-job-app.service)...529agent # [ 4493.297037] agent systemd[1]: Reloading...530agent # [ 4493.622935] agent systemd[1]: Reloading finished in 325 ms.531agent # [ 4493.691119] agent systemd[1]: Starting app broken...532agent # [ 4493.721473] agent (nonexistent)[1476]: app.service: Unable to locate executable '/nonexistent': No such file or directory533agent # [ 4493.721496] agent (nonexistent)[1476]: app.service: Failed at step EXEC spawning /nonexistent: No such file or directory534agent # [ 4493.723222] agent systemd[1]: app.service: Main process exited, code=exited, status=203/EXEC535agent # [ 4493.723315] agent systemd[1]: app.service: Failed with result 'exit-code'.536agent # [ 4493.723479] agent systemd[1]: Failed to start app broken.537agent # [ 4493.753688] agent systemd[1]: Reload requested from client PID 1487 ('systemctl') (unit flakelet-relay-job-app.service)...538agent # [ 4493.753723] agent systemd[1]: Reloading...539agent # [ 4494.080822] agent systemd[1]: Reloading finished in 326 ms.540agent # [ 4494.162429] agent systemd[1]: Reload requested from client PID 1517 ('systemctl') (unit flakelet-relay-job-app.service)...541agent # [ 4494.162465] agent systemd[1]: Reloading...542agent # [ 4494.487824] agent systemd[1]: Reloading finished in 324 ms.543agent # [ 4494.573830] agent systemd[1]: Reload requested from client PID 1547 ('systemctl') (unit flakelet-relay-job-app.service)...544agent # [ 4494.573866] agent systemd[1]: Reloading...545agent # [ 4494.896667] agent systemd[1]: Reloading finished in 322 ms.546agent # [ 4494.994058] agent systemd[1]: Reload requested from client PID 1580 ('systemctl') (unit flakelet-relay-job-app.service)...547agent # [ 4494.994096] agent systemd[1]: Reloading...548agent # [ 4495.318888] agent systemd[1]: Reloading finished in 324 ms.549agent # [ 4495.388693] agent systemd[1]: Starting app v1...550agent # [ 4495.432762] agent systemd[1]: Started app v1.551agent # [ 4495.434043] agent flakelet[1343]: app: deploy failed, rolled back: systemctl start app.service failed:552agent # [ 4495.434043] agent flakelet[1343]: failed units: app.service. See journalctl -u app.service553agent # [ 4495.435905] agent systemd[1]: flakelet-relay-job-app.service: Main process exited, code=exited, status=1/FAILURE554agent # [ 4495.435998] agent systemd[1]: flakelet-relay-job-app.service: Failed with result 'exit-code'.555client: (finished: must fail: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy agent/app --wave agent/other 2>&1, in 4.15 seconds)556agent: must succeed: systemctl show app.service -p Description | grep -q 'app v1'557agent: (finished: must succeed: systemctl show app.service -p Description | grep -q 'app v1', in 0.02 seconds)558(finished: subtest: waves: failing first wave skips the second, in 4.19 seconds)559subtest: host patterns expand on the relay560agent: must succeed: jq '.services.app.prebuilt="/nix/store/hysydfr96ds875vr0jgw5rjxi4p39h2j-flakelet-app-v1"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json561agent: (finished: must succeed: jq '.services.app.prebuilt="/nix/store/hysydfr96ds875vr0jgw5rjxi4p39h2j-flakelet-app-v1"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json, in 0.01 seconds)562client: must succeed: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy '@all/app' 2>&1563agent # [ 4496.127326] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:34.089093Z","level":"INFO","message":"update finished","flakelet":"app","status":"RolledBack","generation":"Some(3)","target":"flakelet_relay::agent::jobs"}564agent # [ 4496.184042] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:34.145810Z","level":"INFO","message":"start","id":"fbc2538011f662cb0a5de236a2ca6864","flakelet":"app","rule":"admins","relay":"relay","caller":"Some(\"x509:email:admin@example.org\")","target":"flakelet_relay::agent::conn"}565agent # [ 4496.184308] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:34.146134Z","level":"INFO","message":"update starting","flakelet":"app","ids":"[\"fbc2538011f662cb0a5de236a2ca6864\"]","target":"flakelet_relay::agent::jobs"}566agent # [ 4496.233470] agent systemd[1]: Starting flakelet update app (via relay)...567agent # [ 4496.262981] agent systemd[1]: Started flakelet update app (via relay).568agent # [ 4496.267505] agent flakelet[1651]: app: using prebuilt artifact /nix/store/hysydfr96ds875vr0jgw5rjxi4p39h2j-flakelet-app-v1569agent # [ 4496.268023] agent flakelet[1651]: app: up to date570agent # [ 4496.268614] agent systemd[1]: flakelet-relay-job-app.service: Deactivated successfully.571relay # [ 4496.128363] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:34.090085Z","level":"INFO","message":"deploy finished","job":"75ef5f5be4a5fe966263e5925f6745cc","ok":false,"target":"flakelet_relay::relay::api"}572relay # [ 4496.183842] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:34.145594Z","level":"INFO","message":"deploy accepted","job":"fbebd5b39d6f5f7be34a3217fa63fe1e","caller":"x509:email:admin@example.org","targets":"[\"agent/app\"]","target":"flakelet_relay::relay::api"}573client: (finished: must succeed: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy '@all/app' 2>&1, in 0.67 seconds)574client: must fail: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy '*/app' 2>&1575agent # [ 4496.832255] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:34.794049Z","level":"INFO","message":"update finished","flakelet":"app","status":"Unchanged","generation":"Some(3)","target":"flakelet_relay::agent::jobs"}576agent # [ 4496.855469] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:34.817279Z","level":"INFO","message":"start","id":"197bf481eef456c5177b789dd1e86911","flakelet":"app","rule":"admins","relay":"relay","caller":"Some(\"x509:email:admin@example.org\")","target":"flakelet_relay::agent::conn"}577agent # [ 4496.855716] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:34.817547Z","level":"INFO","message":"update starting","flakelet":"app","ids":"[\"197bf481eef456c5177b789dd1e86911\"]","target":"flakelet_relay::agent::jobs"}578agent # [ 4496.900379] agent systemd[1]: Starting flakelet update app (via relay)...579agent # [ 4496.924915] agent systemd[1]: Started flakelet update app (via relay).580agent # [ 4496.928407] agent flakelet[1689]: app: using prebuilt artifact /nix/store/hysydfr96ds875vr0jgw5rjxi4p39h2j-flakelet-app-v1581agent # [ 4496.928927] agent flakelet[1689]: app: up to date582agent # [ 4496.929862] agent systemd[1]: flakelet-relay-job-app.service: Deactivated successfully.583relay # [ 4496.833309] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:34.795044Z","level":"INFO","message":"deploy finished","job":"fbebd5b39d6f5f7be34a3217fa63fe1e","ok":true,"target":"flakelet_relay::relay::api"}584relay # [ 4496.855101] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:34.816902Z","level":"INFO","message":"deploy accepted","job":"9a4e7c1343036c053d36fb63e74435b2","caller":"x509:email:admin@example.org","targets":"[\"agent/app\", \"ghost/app\"]","target":"flakelet_relay::relay::api"}585client: (finished: must fail: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy '*/app' 2>&1, in 0.65 seconds)586client: must fail: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy agent/app --wave '*/app' 2>&1587agent # [ 4497.480056] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:35.441806Z","level":"INFO","message":"update finished","flakelet":"app","status":"Unchanged","generation":"Some(3)","target":"flakelet_relay::agent::jobs"}588agent # [ 4497.502783] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:35.464561Z","level":"INFO","message":"start","id":"515286f668c3a60c82318dafa8807ff1","flakelet":"app","rule":"admins","relay":"relay","caller":"Some(\"x509:email:admin@example.org\")","target":"flakelet_relay::agent::conn"}589agent # [ 4497.502974] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:35.464801Z","level":"INFO","message":"update starting","flakelet":"app","ids":"[\"515286f668c3a60c82318dafa8807ff1\"]","target":"flakelet_relay::agent::jobs"}590agent # [ 4497.560831] agent systemd[1]: Starting flakelet update app (via relay)...591agent # [ 4497.588995] agent systemd[1]: Started flakelet update app (via relay).592agent # [ 4497.593263] agent flakelet[1717]: app: using prebuilt artifact /nix/store/hysydfr96ds875vr0jgw5rjxi4p39h2j-flakelet-app-v1593agent # [ 4497.593735] agent flakelet[1717]: app: up to date594agent # [ 4497.595943] agent systemd[1]: flakelet-relay-job-app.service: Deactivated successfully.595relay # [ 4497.481260] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:35.443005Z","level":"INFO","message":"deploy finished","job":"9a4e7c1343036c053d36fb63e74435b2","ok":false,"target":"flakelet_relay::relay::api"}596relay # [ 4497.502610] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:35.464411Z","level":"INFO","message":"deploy accepted","job":"3ed0982d29273bb0fda943bbfccf3663","caller":"x509:email:admin@example.org","targets":"[\"agent/app\", \"ghost/app\"]","target":"flakelet_relay::relay::api"}597client: (finished: must fail: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy agent/app --wave '*/app' 2>&1, in 0.67 seconds)598client: must fail: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy '@all/nope' 2>&1599client: (finished: must fail: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy '@all/nope' 2>&1, in 0.02 seconds)600(finished: subtest: host patterns expand on the relay, in 2.02 seconds)601subtest: detach returns once the job is accepted602agent: must succeed: jq '.services.app.prebuilt="/nix/store/ln47rsd5km8a04hd5zlkz4q1v352zqh3-flakelet-app-slow2"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json603agent: (finished: must succeed: jq '.services.app.prebuilt="/nix/store/ln47rsd5km8a04hd5zlkz4q1v352zqh3-flakelet-app-slow2"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json, in 0.01 seconds)604client: must succeed: timeout 20 push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy --detach agent/app 2>&1605client: (finished: must succeed: timeout 20 push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy --detach agent/app 2>&1, in 0.02 seconds)606agent: waiting for success: systemctl is-active flakelet-relay-job-app.service607agent # [ 4498.147781] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:36.109531Z","level":"INFO","message":"update finished","flakelet":"app","status":"Unchanged","generation":"Some(3)","target":"flakelet_relay::agent::jobs"}608agent # [ 4498.201653] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:36.163418Z","level":"INFO","message":"start","id":"12bf059d3f3942e70f9508a8e804a7c0","flakelet":"app","rule":"admins","relay":"relay","caller":"Some(\"x509:email:admin@example.org\")","target":"flakelet_relay::agent::conn"}609agent # [ 4498.201878] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:36.163705Z","level":"INFO","message":"update starting","flakelet":"app","ids":"[\"12bf059d3f3942e70f9508a8e804a7c0\"]","target":"flakelet_relay::agent::jobs"}610agent # [ 4498.254887] agent systemd[1]: Starting flakelet update app (via relay)...611agent # [ 4498.284042] agent systemd[1]: Started flakelet update app (via relay).612agent # [ 4498.288448] agent flakelet[1749]: app: using prebuilt artifact /nix/store/ln47rsd5km8a04hd5zlkz4q1v352zqh3-flakelet-app-slow2613agent # [ 4498.289085] agent flakelet[1749]: app: activating generation 4614agent # [ 4498.301401] agent systemd[1]: Stopping app v1...615agent # [ 4498.301840] agent systemd[1]: app.service: Deactivated successfully.616agent # [ 4498.302116] agent systemd[1]: Stopped app v1.617agent # [ 4498.318233] agent systemd[1]: Reload requested from client PID 1756 ('systemctl') (unit flakelet-relay-job-app.service)...618agent # [ 4498.318297] agent systemd[1]: Reloading...619relay # [ 4498.148905] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:36.110655Z","level":"INFO","message":"deploy finished","job":"3ed0982d29273bb0fda943bbfccf3663","ok":false,"target":"flakelet_relay::relay::api"}620relay # [ 4498.201442] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:36.163201Z","level":"INFO","message":"deploy accepted","job":"6800e917a4af293f2eb524113736d111","caller":"x509:email:admin@example.org","targets":"[\"agent/app\"]","target":"flakelet_relay::relay::api"}621agent # [ 4498.655514] agent systemd[1]: Reloading finished in 336 ms.622agent # [ 4498.729471] agent systemd[1]: Reload requested from client PID 1786 ('systemctl') (unit flakelet-relay-job-app.service)...623agent # [ 4498.729508] agent systemd[1]: Reloading...624agent # [ 4499.056286] agent systemd[1]: Reloading finished in 326 ms.625agent # [ 4499.165043] agent systemd[1]: Reload requested from client PID 1817 ('systemctl') (unit flakelet-relay-job-app.service)...626agent # [ 4499.165079] agent systemd[1]: Reloading...627agent: (finished: waiting for success: systemctl is-active flakelet-relay-job-app.service, in 1.35 seconds)628??? Warning (UserWarning): wait_until_fails(): passing a bare int/float as a duration is deprecated. Use datetime.timedelta instead.629 File "/nix/store/fv4yr9c0f9kyrfc00100i4rlshiiidbd-nixos-test-driver-1.1/lib/python3.14/site-packages/test_driver/duration.py", line 39630agent: waiting for failure: systemctl is-active flakelet-relay-job-app.service631??? Warning (UserWarning): execute(): passing a bare int/float as a duration is deprecated. Use datetime.timedelta instead.632 File "/nix/store/fv4yr9c0f9kyrfc00100i4rlshiiidbd-nixos-test-driver-1.1/lib/python3.14/site-packages/test_driver/duration.py", line 39633agent # [ 4499.489476] agent systemd[1]: Reloading finished in 324 ms.634agent # [ 4499.562081] agent systemd[1]: Reload requested from client PID 1852 ('systemctl') (unit flakelet-relay-job-app.service)...635agent # [ 4499.562114] agent systemd[1]: Reloading...636agent # [ 4499.888826] agent systemd[1]: Reloading finished in 326 ms.637agent # [ 4499.965774] agent systemd[1]: Starting app slow2...638agent # [ 4506.044129] agent systemd[1]: Started app slow2.639agent # [ 4506.060217] agent flakelet[1749]: app: updated to generation 4640agent # [ 4506.062018] agent systemd[1]: flakelet-relay-job-app.service: Deactivated successfully.641agent: (finished: waiting for failure: systemctl is-active flakelet-relay-job-app.service, in 7.50 seconds)642(finished: subtest: detach returns once the job is accepted, in 8.89 seconds)643subtest: unknown host is final, missing agent is retried, agents listing644client: must fail: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy nope/app 2>&1645client: (finished: must fail: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy nope/app 2>&1, in 0.02 seconds)646agent: must succeed: systemctl stop flakelet-agent647agent: (finished: must succeed: systemctl stop flakelet-agent, in 0.04 seconds)648relay: waiting for success: curl -sf http://127.0.0.1:7400/metrics | grep -q 'agent_up{host="agent"} 0'649relay: (finished: waiting for success: curl -sf http://127.0.0.1:7400/metrics | grep -q 'agent_up{host="agent"} 0', in 0.02 seconds)650client: must succeed: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy agent/other > /tmp/retry.out 2>&1 &651client: (finished: must succeed: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy agent/other > /tmp/retry.out 2>&1 &, in 0.01 seconds)652client: waiting for success: grep -q 'agent_unavailable.*retrying' /tmp/retry.out653agent # [ 4507.096134] agent systemd[1]: Stopping flakelet-agent...654agent # [ 4507.124532] agent systemd[1]: flakelet-agent.service: Deactivated successfully.655agent # [ 4507.124922] agent systemd[1]: Stopped flakelet-agent.656agent # [ 4507.125366] agent systemd[1]: flakelet-agent.service: Consumed 2.988s CPU time over 32.415s wall clock time.657relay # [ 4507.096193] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:45.057934Z","level":"INFO","message":"agent disconnected: websocket closed","host":"agent","target":"flakelet_relay::relay::agent_conn"}658client: (finished: waiting for success: grep -q 'agent_unavailable.*retrying' /tmp/retry.out, in 1.02 seconds)659agent: must succeed: systemctl start flakelet-agent660agent: (finished: must succeed: systemctl start flakelet-agent, in 0.16 seconds)661??? Warning (UserWarning): wait_until_succeeds(): passing a bare int/float as a duration is deprecated. Use datetime.timedelta instead.662 File "/nix/store/fv4yr9c0f9kyrfc00100i4rlshiiidbd-nixos-test-driver-1.1/lib/python3.14/site-packages/test_driver/duration.py", line 39663client: waiting for success: grep -q 'agent/other: unchanged' /tmp/retry.out664agent # [ 4508.183264] agent systemd[1]: Starting flakelet-agent...665agent # [ 4508.238059] agent flakelet-agent[1952]: {"timestamp":"2026-09-06T08:19:46.199840Z","level":"INFO","message":"reattaching to running unit","flakelet":"app","ids":"[\"12bf059d3f3942e70f9508a8e804a7c0\"]","target":"flakelet_relay::agent::jobs"}666agent # [ 4508.324988] agent systemd[1]: Started flakelet-agent.667agent # [ 4508.335618] agent flakelet-agent[1952]: {"timestamp":"2026-09-06T08:19:46.297412Z","level":"INFO","message":"connected","relay":"https://relay:7443","relay_name":"relay","relay_version":"0.1.0","host":"agent","target":"flakelet_relay::agent::conn"}668relay # [ 4508.335686] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:46.297423Z","level":"INFO","message":"agent connected","host":"agent","version":"0.1.0","flakelets":"[Named { name: \"app\", generation: Some(4), revision: Some(\"prebuilt:app\") }, Named { name: \"other\", generation: Some(1), revision: Some(\"prebuilt:other\") }]","target":"flakelet_relay::relay::agent_conn"}669agent # [ 4508.795884] agent flakelet-agent[1952]: {"timestamp":"2026-09-06T08:19:46.757600Z","level":"INFO","message":"update finished","flakelet":"app","status":"Updated","generation":"Some(4)","target":"flakelet_relay::agent::jobs"}670agent # [ 4510.169735] agent flakelet-agent[1952]: {"timestamp":"2026-09-06T08:19:48.131441Z","level":"INFO","message":"start","id":"0e451e175fc58384e099ff87f303965c","flakelet":"other","rule":"admins","relay":"relay","caller":"Some(\"x509:email:admin@example.org\")","target":"flakelet_relay::agent::conn"}671agent # [ 4510.170636] agent flakelet-agent[1952]: {"timestamp":"2026-09-06T08:19:48.132280Z","level":"INFO","message":"update starting","flakelet":"other","ids":"[\"0e451e175fc58384e099ff87f303965c\"]","target":"flakelet_relay::agent::jobs"}672agent # [ 4510.224171] agent systemd[1]: Starting flakelet update other (via relay)...673agent # [ 4510.250946] agent systemd[1]: Started flakelet update other (via relay).674agent # [ 4510.254481] agent flakelet[2385]: other: using prebuilt artifact /nix/store/hmr990n0b4v6rpzrcpyyw738r9xi7z95-flakelet-other-v1675agent # [ 4510.254960] agent flakelet[2385]: other: up to date676agent # [ 4510.257875] agent systemd[1]: flakelet-relay-job-other.service: Deactivated successfully.677relay # [ 4510.169348] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:48.131079Z","level":"INFO","message":"deploy accepted","job":"9254837a90831217a89cc61211f155f5","caller":"x509:email:admin@example.org","targets":"[\"agent/other\"]","target":"flakelet_relay::relay::api"}678agent # [ 4510.809494] agent flakelet-agent[1952]: {"timestamp":"2026-09-06T08:19:48.771230Z","level":"INFO","message":"update finished","flakelet":"other","status":"Unchanged","generation":"Some(1)","target":"flakelet_relay::agent::jobs"}679relay # [ 4510.811191] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:48.772916Z","level":"INFO","message":"deploy finished","job":"9254837a90831217a89cc61211f155f5","ok":true,"target":"flakelet_relay::relay::api"}680client: (finished: waiting for success: grep -q 'agent/other: unchanged' /tmp/retry.out, in 3.04 seconds)681client: must succeed: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem agents682client: (finished: must succeed: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem agents, in 0.02 seconds)683client: must succeed: echo '{"groups": []}' | step crypto jwt sign --key /nix/store/r7sclamg09v434w8zjlfnrcrn1w79wh6-flakelet-relay-test-issuer/jwk.json --iss https://relay:8443 --aud flakelet-relay --sub 'repo:github:example/app:ref:refs/heads/main' --exp $(( $(date +%s) + 300 )) --jti $(head -c8 /dev/urandom | base32)684client: (finished: must succeed: echo '{"groups": []}' | step crypto jwt sign --key /nix/store/r7sclamg09v434w8zjlfnrcrn1w79wh6-flakelet-relay-test-issuer/jwk.json --iss https://relay:8443 --aud flakelet-relay --sub 'repo:github:example/app:ref:refs/heads/main' --exp $(( $(date +%s) + 300 )) --jti $(head -c8 /dev/urandom | base32), in 0.05 seconds)685client: must succeed: FLAKELET_RELAY_TOKEN_COMMAND='echo eyJhbGciOiJFZERTQSIsImtpZCI6ImsxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJmbGFrZWxldC1yZWxheSIsImV4cCI6MTc4ODY4MzA4OSwiZ3JvdXBzIjpbXSwiaWF0IjoxNzg4NjgyNzg5LCJpc3MiOiJodHRwczovL3JlbGF5Ojg0NDMiLCJqdGkiOiJNMzdZTDVGUkRISDU2PT09IiwibmJmIjoxNzg4NjgyNzg5LCJzdWIiOiJyZXBvOmdpdGh1YjpleGFtcGxlL2FwcDpyZWY6cmVmcy9oZWFkcy9tYWluIn0.JdQ3HIIPRLegYWQp8C-urvqDBmNPEwRtRXaXrBqlWbKRw4ZdWeTdYaZFqG7eX_umvn0nukiGYyXc-zQ-WiP1BA' push --relay https://relay:7443 agents686client: (finished: must succeed: FLAKELET_RELAY_TOKEN_COMMAND='echo eyJhbGciOiJFZERTQSIsImtpZCI6ImsxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJmbGFrZWxldC1yZWxheSIsImV4cCI6MTc4ODY4MzA4OSwiZ3JvdXBzIjpbXSwiaWF0IjoxNzg4NjgyNzg5LCJpc3MiOiJodHRwczovL3JlbGF5Ojg0NDMiLCJqdGkiOiJNMzdZTDVGUkRISDU2PT09IiwibmJmIjoxNzg4NjgyNzg5LCJzdWIiOiJyZXBvOmdpdGh1YjpleGFtcGxlL2FwcDpyZWY6cmVmcy9oZWFkcy9tYWluIn0.JdQ3HIIPRLegYWQp8C-urvqDBmNPEwRtRXaXrBqlWbKRw4ZdWeTdYaZFqG7eX_umvn0nukiGYyXc-zQ-WiP1BA' push --relay https://relay:7443 agents, in 0.02 seconds)687(finished: subtest: unknown host is final, missing agent is retried, agents listing, in 4.40 seconds)688subtest: out-of-band flakelet update reaches the relay without reconnect689agent: must succeed: flakelet status --json app | jq '.[0].generation'690agent: (finished: must succeed: flakelet status --json app | jq '.[0].generation', in 0.03 seconds)691agent: must succeed: journalctl -u flakelet-agent -o cat | grep -c '"connected"' || true692agent: (finished: must succeed: journalctl -u flakelet-agent -o cat | grep -c '"connected"' || true, in 0.01 seconds)693agent: must succeed: jq '.services.app.prebuilt="/nix/store/6ili1vxjlag9d7dy0bxrbxk6bpi5qh9p-flakelet-app-v2"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json694agent: (finished: must succeed: jq '.services.app.prebuilt="/nix/store/6ili1vxjlag9d7dy0bxrbxk6bpi5qh9p-flakelet-app-v2"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json, in 0.01 seconds)695agent: must succeed: flakelet update app696app: using prebuilt artifact /nix/store/6ili1vxjlag9d7dy0bxrbxk6bpi5qh9p-flakelet-app-v2697app: activating generation 5698agent # [ 4511.540657] agent systemd[1]: Stopping app slow2...699agent # [ 4511.541392] agent systemd[1]: app.service: Deactivated successfully.700agent # [ 4511.541529] agent systemd[1]: Stopped app slow2.701agent # [ 4511.553287] agent systemd[1]: Reload requested from client PID 2433 ('systemctl')...702agent # [ 4511.553324] agent systemd[1]: Reloading...703agent # [ 4511.882921] agent systemd[1]: Reloading finished in 329 ms.704agent # [ 4511.975443] agent systemd[1]: Reload requested from client PID 2462 ('systemctl')...705agent # [ 4511.975479] agent systemd[1]: Reloading...706agent # [ 4512.298636] agent systemd[1]: Reloading finished in 322 ms.707agent # [ 4512.384036] agent systemd[1]: Reload requested from client PID 2492 ('systemctl')...708agent # [ 4512.384079] agent systemd[1]: Reloading...709agent # [ 4512.708559] agent systemd[1]: Reloading finished in 324 ms.710agent # [ 4512.794051] agent systemd[1]: Reload requested from client PID 2524 ('systemctl')...711agent # [ 4512.794089] agent systemd[1]: Reloading...712agent: (finished: must succeed: flakelet update app, in 1.72 seconds)713client: waiting for success: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem agents | grep -q 'app@5'714agent # [ 4513.118825] agent systemd[1]: Reloading finished in 324 ms.715agent # [ 4513.187571] agent systemd[1]: Starting app v2...716agent # [ 4513.224077] agent systemd[1]: Started app v2.717client: (finished: waiting for success: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem agents | grep -q 'app@5', in 2.07 seconds)718agent: must succeed: journalctl -u flakelet-agent -o cat | grep -c '"connected"' || true719agent: (finished: must succeed: journalctl -u flakelet-agent -o cat | grep -c '"connected"' || true, in 0.01 seconds)720agent: must succeed: jq '.services.app.prebuilt="/nix/store/hysydfr96ds875vr0jgw5rjxi4p39h2j-flakelet-app-v1"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json721agent: (finished: must succeed: jq '.services.app.prebuilt="/nix/store/hysydfr96ds875vr0jgw5rjxi4p39h2j-flakelet-app-v1"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json, in 0.01 seconds)722agent: must succeed: flakelet update app723app: using prebuilt artifact /nix/store/hysydfr96ds875vr0jgw5rjxi4p39h2j-flakelet-app-v1724app: activating generation 6725agent # [ 4515.348610] agent systemd[1]: Stopping app v2...726agent # [ 4515.349348] agent systemd[1]: app.service: Deactivated successfully.727agent # [ 4515.349478] agent systemd[1]: Stopped app v2.728agent # [ 4515.359426] agent systemd[1]: Reload requested from client PID 2582 ('systemctl')...729agent # [ 4515.359463] agent systemd[1]: Reloading...730agent # [ 4515.686216] agent systemd[1]: Reloading finished in 326 ms.731agent # [ 4515.753803] agent systemd[1]: Reload requested from client PID 2611 ('systemctl')...732agent # [ 4515.753840] agent systemd[1]: Reloading...733agent # [ 4516.077352] agent systemd[1]: Reloading finished in 322 ms.734agent # [ 4516.142613] agent systemd[1]: Reload requested from client PID 2639 ('systemctl')...735agent # [ 4516.142649] agent systemd[1]: Reloading...736agent # [ 4516.466688] agent systemd[1]: Reloading finished in 323 ms.737agent # [ 4516.539211] agent systemd[1]: Reload requested from client PID 2671 ('systemctl')...738agent # [ 4516.539241] agent systemd[1]: Reloading...739agent: (finished: must succeed: flakelet update app, in 1.64 seconds)740client: waiting for success: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem agents | grep -q 'app@6'741agent # [ 4516.861402] agent systemd[1]: Reloading finished in 321 ms.742agent # [ 4516.932333] agent systemd[1]: Starting app v1...743agent # [ 4516.960005] agent systemd[1]: Started app v1.744client: (finished: waiting for success: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem agents | grep -q 'app@6', in 2.06 seconds)745(finished: subtest: out-of-band flakelet update reaches the relay without reconnect, in 7.57 seconds)746subtest: certificate not listed under agents cannot connect as agent747client: must succeed: curl -s -o /dev/null -w '%{http_code}' --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/mallory/cert.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/mallory/key.pem -H 'Upgrade: websocket' -H 'Connection: Upgrade' -H 'Sec-WebSocket-Version: 13' -H 'Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==' https://relay:7443/v1/agent748client: (finished: must succeed: curl -s -o /dev/null -w '%{http_code}' --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/mallory/cert.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/mallory/key.pem -H 'Upgrade: websocket' -H 'Connection: Upgrade' -H 'Sec-WebSocket-Version: 13' -H 'Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==' https://relay:7443/v1/agent, in 0.02 seconds)749(finished: subtest: certificate not listed under agents cannot connect as agent, in 0.03 seconds)750subtest: metrics751relay: must succeed: curl -sf http://127.0.0.1:7400/metrics752relay: (finished: must succeed: curl -sf http://127.0.0.1:7400/metrics, in 0.01 seconds)753(finished: subtest: metrics, in 0.01 seconds)754subtest: agent link drop mid-run replays missed lines once755agent: must succeed: jq '.services.app.prebuilt="/nix/store/c67jj8kx3m50nyj874aqmdqf21w5iznd-flakelet-app-slow"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json756agent: (finished: must succeed: jq '.services.app.prebuilt="/nix/store/c67jj8kx3m50nyj874aqmdqf21w5iznd-flakelet-app-slow"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json, in 0.01 seconds)757agent: must succeed: date +%s758agent: (finished: must succeed: date +%s, in 0.01 seconds)759client: must succeed: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy agent/app > /tmp/slow.out 2>&1 &760client: (finished: must succeed: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy agent/app > /tmp/slow.out 2>&1 &, in 0.01 seconds)761agent: waiting for success: journalctl -u flakelet-relay-job-app.service --since=@1788682797 | grep -q 'app-slow'762relay # [ 4519.059679] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:57.021407Z","level":"INFO","message":"agent rejected: no agents entry matches","principals":"[\"x509:dns:mallory\"]","target":"flakelet_relay::relay::agent_conn"}763relay # [ 4519.116493] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:57.078247Z","level":"INFO","message":"deploy accepted","job":"fc5d49a798d1b431af8fd0099b0d7754","caller":"x509:email:admin@example.org","targets":"[\"agent/app\"]","target":"flakelet_relay::relay::api"}764agent # [ 4519.116956] agent flakelet-agent[1952]: {"timestamp":"2026-09-06T08:19:57.078702Z","level":"INFO","message":"start","id":"a0aa3eb4699c49a364fd4d93a1df7cdf","flakelet":"app","rule":"admins","relay":"relay","caller":"Some(\"x509:email:admin@example.org\")","target":"flakelet_relay::agent::conn"}765agent # [ 4519.117800] agent flakelet-agent[1952]: {"timestamp":"2026-09-06T08:19:57.079403Z","level":"INFO","message":"update starting","flakelet":"app","ids":"[\"a0aa3eb4699c49a364fd4d93a1df7cdf\"]","target":"flakelet_relay::agent::jobs"}766agent # [ 4519.169893] agent systemd[1]: Starting flakelet update app (via relay)...767agent # [ 4519.203887] agent systemd[1]: Started flakelet update app (via relay).768agent # [ 4519.207991] agent flakelet[2735]: app: using prebuilt artifact /nix/store/c67jj8kx3m50nyj874aqmdqf21w5iznd-flakelet-app-slow769agent # [ 4519.208591] agent flakelet[2735]: app: activating generation 7770agent # [ 4519.221565] agent systemd[1]: Stopping app v1...771agent # [ 4519.222063] agent systemd[1]: app.service: Deactivated successfully.772agent # [ 4519.222324] agent systemd[1]: Stopped app v1.773agent # [ 4519.239562] agent systemd[1]: Reload requested from client PID 2742 ('systemctl') (unit flakelet-relay-job-app.service)...774agent # [ 4519.239628] agent systemd[1]: Reloading...775agent # [ 4519.577957] agent systemd[1]: Reloading finished in 337 ms.776agent # [ 4519.651551] agent systemd[1]: Reload requested from client PID 2772 ('systemctl') (unit flakelet-relay-job-app.service)...777agent # [ 4519.651589] agent systemd[1]: Reloading...778agent: (finished: waiting for success: journalctl -u flakelet-relay-job-app.service --since=@1788682797 | grep -q 'app-slow', in 1.02 seconds)779agent: must succeed: for a in $(getent ahosts relay | cut -d' ' -f1 | sort -u); do ip route add unreachable $a; done780agent: (finished: must succeed: for a in $(getent ahosts relay | cut -d' ' -f1 | sort -u); do ip route add unreachable $a; done, in 0.02 seconds)781relay: must succeed: for a in $(getent ahosts agent | cut -d' ' -f1 | sort -u); do ss -K dst "[$a]"; done782relay: (finished: must succeed: for a in $(getent ahosts agent | cut -d' ' -f1 | sort -u); do ss -K dst "[$a]"; done, in 0.06 seconds)783agent: must succeed: ss -K dport = 7443784agent # [ 4519.974709] agent systemd[1]: Reloading finished in 322 ms.785agent # [ 4520.053118] agent systemd[1]: Reload requested from client PID 2801 ('systemctl') (unit flakelet-relay-job-app.service)...786agent # [ 4520.053157] agent systemd[1]: Reloading...787agent: (finished: must succeed: ss -K dport = 7443, in 0.03 seconds)788agent: waiting for success: journalctl -u flakelet-relay-job-app.service --since=@1788682797 | grep -q 'updated to generation'789agent # [ 4520.231670] agent flakelet-agent[1952]: {"timestamp":"2026-09-06T08:19:58.193287Z","level":"INFO","message":"disconnected: Software caused connection abort (os error 103)","relay":"https://relay:7443","target":"flakelet_relay::agent::conn"}790agent # [ 4520.376518] agent systemd[1]: Reloading finished in 322 ms.791agent # [ 4520.453972] agent systemd[1]: Reload requested from client PID 2849 ('systemctl') (unit flakelet-relay-job-app.service)...792agent # [ 4520.454015] agent systemd[1]: Reloading...793relay # [ 4520.201286] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:58.163025Z","level":"INFO","message":"agent disconnected: Software caused connection abort (os error 103)","host":"agent","target":"flakelet_relay::relay::agent_conn"}794agent # [ 4520.779383] agent systemd[1]: Reloading finished in 325 ms.795agent # [ 4520.849444] agent systemd[1]: Starting app slow...796agent # [ 4521.234592] agent flakelet-agent[1952]: {"timestamp":"2026-09-06T08:19:59.196320Z","level":"WARN","message":"connect failed: connect relay:7443: No route to host (os error 113)","relay":"https://relay:7443","target":"flakelet_relay::agent::conn"}797agent # [ 4523.237832] agent flakelet-agent[1952]: {"timestamp":"2026-09-06T08:20:01.199546Z","level":"WARN","message":"connect failed: connect relay:7443: No route to host (os error 113)","relay":"https://relay:7443","target":"flakelet_relay::agent::conn"}798agent # [ 4526.911028] agent systemd[1]: Started app slow.799agent # [ 4526.928747] agent flakelet[2735]: app: updated to generation 7800agent # [ 4526.931230] agent systemd[1]: flakelet-relay-job-app.service: Deactivated successfully.801agent: (finished: waiting for success: journalctl -u flakelet-relay-job-app.service --since=@1788682797 | grep -q 'updated to generation', in 7.11 seconds)802agent: must succeed: for a in $(getent ahosts relay | cut -d' ' -f1 | sort -u); do ip route del unreachable $a; done803agent: (finished: must succeed: for a in $(getent ahosts relay | cut -d' ' -f1 | sort -u); do ip route del unreachable $a; done, in 0.01 seconds)804client: waiting for success: grep -q 'agent/app: updated' /tmp/slow.out805agent # [ 4527.240505] agent flakelet-agent[1952]: {"timestamp":"2026-09-06T08:20:05.202111Z","level":"WARN","message":"connect failed: connect relay:7443: No route to host (os error 113)","relay":"https://relay:7443","target":"flakelet_relay::agent::conn"}806agent # [ 4528.037225] agent flakelet-agent[1952]: {"timestamp":"2026-09-06T08:20:05.998965Z","level":"INFO","message":"update finished","flakelet":"app","status":"Updated","generation":"Some(7)","target":"flakelet_relay::agent::jobs"}807client: (finished: waiting for success: grep -q 'agent/app: updated' /tmp/slow.out, in 8.08 seconds)808client: must succeed: cat /tmp/slow.out809client: (finished: must succeed: cat /tmp/slow.out, in 0.01 seconds)810(finished: subtest: agent link drop mid-run replays missed lines once, in 16.37 seconds)811subtest: agent restart mid-run reattaches to the unit812agent: must succeed: jq '.services.app.prebuilt="/nix/store/ln47rsd5km8a04hd5zlkz4q1v352zqh3-flakelet-app-slow2"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json813agent: (finished: must succeed: jq '.services.app.prebuilt="/nix/store/ln47rsd5km8a04hd5zlkz4q1v352zqh3-flakelet-app-slow2"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json, in 0.01 seconds)814client: must succeed: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy agent/app > /tmp/restart.out 2>&1 &815client: (finished: must succeed: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy agent/app > /tmp/restart.out 2>&1 &, in 0.00 seconds)816agent: waiting for success: journalctl -u flakelet-relay-job-app.service --since -20s | grep -q 'app-slow2'817agent # [ 4535.244493] agent flakelet-agent[1952]: {"timestamp":"2026-09-06T08:20:13.206229Z","level":"INFO","message":"connected","relay":"https://relay:7443","relay_name":"relay","relay_version":"0.1.0","host":"agent","target":"flakelet_relay::agent::conn"}818agent # [ 4535.245524] agent flakelet-agent[1952]: {"timestamp":"2026-09-06T08:20:13.206413Z","level":"INFO","message":"start","id":"a0aa3eb4699c49a364fd4d93a1df7cdf","flakelet":"app","rule":"admins","relay":"relay","caller":"Some(\"x509:email:admin@example.org\")","target":"flakelet_relay::agent::conn"}819agent # [ 4535.480858] agent flakelet-agent[1952]: {"timestamp":"2026-09-06T08:20:13.442607Z","level":"INFO","message":"start","id":"abe982fc6094ddbd2ae474b1598d1f5c","flakelet":"app","rule":"admins","relay":"relay","caller":"Some(\"x509:email:admin@example.org\")","target":"flakelet_relay::agent::conn"}820agent # [ 4535.481239] agent flakelet-agent[1952]: {"timestamp":"2026-09-06T08:20:13.443066Z","level":"INFO","message":"update starting","flakelet":"app","ids":"[\"abe982fc6094ddbd2ae474b1598d1f5c\"]","target":"flakelet_relay::agent::jobs"}821relay # [ 4535.244633] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:20:13.206315Z","level":"INFO","message":"agent connected","host":"agent","version":"0.1.0","flakelets":"[Named { name: \"app\", generation: Some(7), revision: Some(\"prebuilt:app\") }, Named { name: \"other\", generation: Some(1), revision: Some(\"prebuilt:other\") }]","target":"flakelet_relay::relay::agent_conn"}822relay # [ 4535.245570] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:20:13.206669Z","level":"INFO","message":"deploy finished","job":"fc5d49a798d1b431af8fd0099b0d7754","ok":true,"target":"flakelet_relay::relay::api"}823relay # [ 4535.480626] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:20:13.442329Z","level":"INFO","message":"deploy accepted","job":"fcec88019960b634f620a5fb8ef1b38a","caller":"x509:email:admin@example.org","targets":"[\"agent/app\"]","target":"flakelet_relay::relay::api"}824agent # [ 4535.532139] agent systemd[1]: Starting flakelet update app (via relay)...825agent # [ 4535.562963] agent systemd[1]: Started flakelet update app (via relay).826agent # [ 4535.568135] agent flakelet[3028]: app: using prebuilt artifact /nix/store/ln47rsd5km8a04hd5zlkz4q1v352zqh3-flakelet-app-slow2827agent # [ 4535.568760] agent flakelet[3028]: app: activating generation 8828agent # [ 4535.582592] agent systemd[1]: Stopping app slow...829agent # [ 4535.583063] agent systemd[1]: app.service: Deactivated successfully.830agent # [ 4535.583334] agent systemd[1]: Stopped app slow.831agent # [ 4535.600043] agent systemd[1]: Reload requested from client PID 3035 ('systemctl') (unit flakelet-relay-job-app.service)...832agent # [ 4535.600109] agent systemd[1]: Reloading...833agent # [ 4535.937722] agent systemd[1]: Reloading finished in 336 ms.834agent # [ 4536.018375] agent systemd[1]: Reload requested from client PID 3065 ('systemctl') (unit flakelet-relay-job-app.service)...835agent # [ 4536.018413] agent systemd[1]: Reloading...836agent # [ 4536.342245] agent systemd[1]: Reloading finished in 323 ms.837agent: (finished: waiting for success: journalctl -u flakelet-relay-job-app.service --since -20s | grep -q 'app-slow2', in 1.02 seconds)838agent: must succeed: systemctl restart flakelet-agent839agent # [ 4536.420649] agent systemd[1]: Reload requested from client PID 3096 ('systemctl') (unit flakelet-relay-job-app.service)...840agent # [ 4536.420685] agent systemd[1]: Reloading...841agent # [ 4536.742702] agent systemd[1]: Reloading finished in 321 ms.842agent # [ 4536.809896] agent systemd[1]: Stopping flakelet-agent...843agent # [ 4536.815529] agent systemd[1]: Reload requested from client PID 3133 ('systemctl') (unit flakelet-relay-job-app.service)...844agent # [ 4536.815570] agent systemd[1]: Reloading...845relay # [ 4536.810352] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:20:14.772094Z","level":"INFO","message":"agent disconnected: websocket closed","host":"agent","target":"flakelet_relay::relay::agent_conn"}846agent: (finished: must succeed: systemctl restart flakelet-agent, in 0.88 seconds)847agent: must succeed: test -n "$(ls /var/lib/flakelet-agent/jobs)"848agent: (finished: must succeed: test -n "$(ls /var/lib/flakelet-agent/jobs)", in 0.01 seconds)849client: waiting for success: grep -q 'agent/app: updated' /tmp/restart.out850agent # [ 4537.156765] agent systemd[1]: Reloading finished in 340 ms.851agent # [ 4537.235727] agent systemd[1]: flakelet-agent.service: Deactivated successfully.852agent # [ 4537.235906] agent systemd[1]: Stopped flakelet-agent.853agent # [ 4537.235921] agent systemd[1]: flakelet-agent.service: Consumed 2.108s CPU time over 29.052s wall clock time.854agent # [ 4537.239635] agent systemd[1]: Starting flakelet-agent...855agent # [ 4537.242727] agent systemd[1]: Starting app slow2...856agent # [ 4537.291388] agent flakelet-agent[3164]: {"timestamp":"2026-09-06T08:20:15.253199Z","level":"INFO","message":"reattaching to running unit","flakelet":"app","ids":"[\"abe982fc6094ddbd2ae474b1598d1f5c\"]","target":"flakelet_relay::agent::jobs"}857agent # [ 4537.369509] agent systemd[1]: Started flakelet-agent.858agent # [ 4537.383877] agent flakelet-agent[3164]: {"timestamp":"2026-09-06T08:20:15.345627Z","level":"INFO","message":"connected","relay":"https://relay:7443","relay_name":"relay","relay_version":"0.1.0","host":"agent","target":"flakelet_relay::agent::conn"}859agent # [ 4537.384128] agent flakelet-agent[3164]: {"timestamp":"2026-09-06T08:20:15.345801Z","level":"INFO","message":"start","id":"abe982fc6094ddbd2ae474b1598d1f5c","flakelet":"app","rule":"admins","relay":"relay","caller":"Some(\"x509:email:admin@example.org\")","target":"flakelet_relay::agent::conn"}860relay # [ 4537.383946] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:20:15.345694Z","level":"INFO","message":"agent connected","host":"agent","version":"0.1.0","flakelets":"[Named { name: \"app\", generation: Some(7), revision: Some(\"prebuilt:app\") }, Named { name: \"other\", generation: Some(1), revision: Some(\"prebuilt:other\") }]","target":"flakelet_relay::relay::agent_conn"}861agent # [ 4543.315647] agent systemd[1]: Started app slow2.862agent # [ 4543.332803] agent flakelet[3028]: app: updated to generation 8863agent # [ 4543.335592] agent systemd[1]: flakelet-relay-job-app.service: Deactivated successfully.864agent # [ 4543.876891] agent flakelet-agent[3164]: {"timestamp":"2026-09-06T08:20:21.838634Z","level":"INFO","message":"update finished","flakelet":"app","status":"Updated","generation":"Some(8)","target":"flakelet_relay::agent::jobs"}865relay # [ 4543.878436] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:20:21.840177Z","level":"INFO","message":"deploy finished","job":"fcec88019960b634f620a5fb8ef1b38a","ok":true,"target":"flakelet_relay::relay::api"}866client: (finished: waiting for success: grep -q 'agent/app: updated' /tmp/restart.out, in 7.08 seconds)867client: must succeed: cat /tmp/restart.out868client: (finished: must succeed: cat /tmp/restart.out, in 0.01 seconds)869agent: must succeed: systemctl is-active app.service870agent: (finished: must succeed: systemctl is-active app.service, in 0.01 seconds)871(finished: subtest: agent restart mid-run reattaches to the unit, in 9.04 seconds)872subtest: relay restart mid-run, push resumes via /v1/jobs873agent: must succeed: jq '.services.app.prebuilt="/nix/store/c67jj8kx3m50nyj874aqmdqf21w5iznd-flakelet-app-slow"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json874agent: (finished: must succeed: jq '.services.app.prebuilt="/nix/store/c67jj8kx3m50nyj874aqmdqf21w5iznd-flakelet-app-slow"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json, in 0.01 seconds)875agent: must succeed: date +%s876agent: (finished: must succeed: date +%s, in 0.01 seconds)877client: must succeed: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy agent/app > /tmp/resume.out 2>&1 &878client: (finished: must succeed: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy agent/app > /tmp/resume.out 2>&1 &, in 0.01 seconds)879agent: waiting for success: journalctl -u flakelet-relay-job-app.service --since=@1788682822 | grep -q 'app-slow'880agent # [ 4544.530497] agent flakelet-agent[3164]: {"timestamp":"2026-09-06T08:20:22.492248Z","level":"INFO","message":"start","id":"4a0077e6bf403a3622aaee6ccf6cc304","flakelet":"app","rule":"admins","relay":"relay","caller":"Some(\"x509:email:admin@example.org\")","target":"flakelet_relay::agent::conn"}881agent # [ 4544.531113] agent flakelet-agent[3164]: {"timestamp":"2026-09-06T08:20:22.492934Z","level":"INFO","message":"update starting","flakelet":"app","ids":"[\"4a0077e6bf403a3622aaee6ccf6cc304\"]","target":"flakelet_relay::agent::jobs"}882agent # [ 4544.582552] agent systemd[1]: Starting flakelet update app (via relay)...883agent # [ 4544.611044] agent systemd[1]: Started flakelet update app (via relay).884agent # [ 4544.614671] agent flakelet[3649]: app: using prebuilt artifact /nix/store/c67jj8kx3m50nyj874aqmdqf21w5iznd-flakelet-app-slow885agent # [ 4544.615262] agent flakelet[3649]: app: activating generation 9886agent # [ 4544.629113] agent systemd[1]: Stopping app slow2...887agent # [ 4544.630752] agent systemd[1]: app.service: Deactivated successfully.888agent # [ 4544.631090] agent systemd[1]: Stopped app slow2.889agent # [ 4544.646447] agent systemd[1]: Reload requested from client PID 3656 ('systemctl') (unit flakelet-relay-job-app.service)...890agent # [ 4544.646513] agent systemd[1]: Reloading...891relay # [ 4544.530061] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:20:22.491766Z","level":"INFO","message":"deploy accepted","job":"486e6ce5b02ce6df82a2d8eb7c4c4ecd","caller":"x509:email:admin@example.org","targets":"[\"agent/app\"]","target":"flakelet_relay::relay::api"}892agent # [ 4544.984364] agent systemd[1]: Reloading finished in 337 ms.893agent # [ 4545.075825] agent systemd[1]: Reload requested from client PID 3686 ('systemctl') (unit flakelet-relay-job-app.service)...894agent # [ 4545.075861] agent systemd[1]: Reloading...895agent: (finished: waiting for success: journalctl -u flakelet-relay-job-app.service --since=@1788682822 | grep -q 'app-slow', in 1.02 seconds)896relay: must succeed: systemctl restart flakelet-relay897agent # [ 4545.400218] agent systemd[1]: Reloading finished in 323 ms.898agent # [ 4545.468544] agent systemd[1]: Reload requested from client PID 3717 ('systemctl') (unit flakelet-relay-job-app.service)...899agent # [ 4545.468579] agent systemd[1]: Reloading...900agent # [ 4545.557873] agent flakelet-agent[3164]: {"timestamp":"2026-09-06T08:20:23.519557Z","level":"INFO","message":"disconnected: websocket closed","relay":"https://relay:7443","target":"flakelet_relay::agent::conn"}901relay # [ 4545.557338] relay systemd[1]: Stopping flakelet-relay...902relay # [ 4545.588599] relay systemd[1]: flakelet-relay.service: Deactivated successfully.903relay # [ 4545.588993] relay systemd[1]: Stopped flakelet-relay.904relay # [ 4545.589054] relay systemd[1]: flakelet-relay.service: Consumed 2.721s CPU time over 1min 9.665s wall clock time.905relay # [ 4545.625075] relay systemd[1]: Starting flakelet-relay...906agent # [ 4545.798579] agent systemd[1]: Reloading finished in 329 ms.907agent # [ 4545.890603] agent systemd[1]: Reload requested from client PID 3753 ('systemctl') (unit flakelet-relay-job-app.service)...908agent # [ 4545.890639] agent systemd[1]: Reloading...909agent # [ 4546.220379] agent systemd[1]: Reloading finished in 329 ms.910agent # [ 4546.304459] agent systemd[1]: Starting app slow...911agent # [ 4546.560167] agent flakelet-agent[3164]: {"timestamp":"2026-09-06T08:20:24.521776Z","level":"WARN","message":"connect failed: connect relay:7443: Connection refused (os error 111)","relay":"https://relay:7443","target":"flakelet_relay::agent::conn"}912relay: (finished: must succeed: systemctl restart flakelet-relay, in 1.48 seconds)913client: waiting for success: grep -q '» ok' /tmp/resume.out914relay # [ 4547.025283] relay flakelet-relay[934]: {"timestamp":"2026-09-06T08:20:24.987035Z","level":"INFO","message":"listening on 127.0.0.1:7400","target":"flakelet_relay::relay::server"}915relay # [ 4547.025785] relay flakelet-relay[934]: {"timestamp":"2026-09-06T08:20:24.987486Z","level":"INFO","message":"listening on [::]:7443","target":"flakelet_relay::relay::server"}916relay # [ 4547.025835] relay systemd[1]: Started flakelet-relay.917agent # [ 4548.574909] agent flakelet-agent[3164]: {"timestamp":"2026-09-06T08:20:26.536680Z","level":"INFO","message":"connected","relay":"https://relay:7443","relay_name":"relay","relay_version":"0.1.0","host":"agent","target":"flakelet_relay::agent::conn"}918relay # [ 4548.575049] relay flakelet-relay[934]: {"timestamp":"2026-09-06T08:20:26.536813Z","level":"INFO","message":"agent connected","host":"agent","version":"0.1.0","flakelets":"[Named { name: \"app\", generation: Some(8), revision: Some(\"prebuilt:app\") }, Named { name: \"other\", generation: Some(1), revision: Some(\"prebuilt:other\") }]","target":"flakelet_relay::relay::agent_conn"}919relay # [ 4548.577328] relay flakelet-relay[934]: {"timestamp":"2026-09-06T08:20:26.539092Z","level":"INFO","message":"job reattached","job":"486e6ce5b02ce6df82a2d8eb7c4c4ecd","targets":"[\"agent/app\"]","target":"flakelet_relay::relay::api"}920agent # [ 4552.378018] agent systemd[1]: Started app slow.921agent # [ 4552.397405] agent flakelet[3649]: app: updated to generation 9922agent # [ 4552.400011] agent systemd[1]: flakelet-relay-job-app.service: Deactivated successfully.923agent # [ 4553.479656] agent flakelet-agent[3164]: {"timestamp":"2026-09-06T08:20:31.441445Z","level":"INFO","message":"update finished","flakelet":"app","status":"Updated","generation":"Some(9)","target":"flakelet_relay::agent::jobs"}924client: (finished: waiting for success: grep -q '» ok' /tmp/resume.out, in 7.08 seconds)925client: must succeed: cat /tmp/resume.out926client: (finished: must succeed: cat /tmp/resume.out, in 0.01 seconds)927(finished: subtest: relay restart mid-run, push resumes via /v1/jobs, in 9.63 seconds)928subtest: job history comes from the agents and survives a relay restart929client: must succeed: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy --id hist1 agent/other 2>&1930agent # [ 4554.142705] agent flakelet-agent[3164]: {"timestamp":"2026-09-06T08:20:32.104457Z","level":"INFO","message":"start","id":"5be20c693e5c0b60e3f85779794dfb86","flakelet":"other","rule":"admins","relay":"relay","caller":"Some(\"x509:email:admin@example.org\")","target":"flakelet_relay::agent::conn"}931agent # [ 4554.143301] agent flakelet-agent[3164]: {"timestamp":"2026-09-06T08:20:32.105123Z","level":"INFO","message":"update starting","flakelet":"other","ids":"[\"5be20c693e5c0b60e3f85779794dfb86\"]","target":"flakelet_relay::agent::jobs"}932agent # [ 4554.193961] agent systemd[1]: Starting flakelet update other (via relay)...933agent # [ 4554.228977] agent systemd[1]: Started flakelet update other (via relay).934agent # [ 4554.233726] agent flakelet[3866]: other: using prebuilt artifact /nix/store/hmr990n0b4v6rpzrcpyyw738r9xi7z95-flakelet-other-v1935agent # [ 4554.234245] agent flakelet[3866]: other: up to date936agent # [ 4554.235625] agent systemd[1]: flakelet-relay-job-other.service: Deactivated successfully.937relay # [ 4554.142287] relay flakelet-relay[934]: {"timestamp":"2026-09-06T08:20:32.104038Z","level":"INFO","message":"deploy accepted","job":"188b06c4636b65849b3ba8daac22dc7c","caller":"x509:email:admin@example.org","targets":"[\"agent/other\"]","target":"flakelet_relay::relay::api"}938client: (finished: must succeed: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy --id hist1 agent/other 2>&1, in 0.67 seconds)939relay: must succeed: systemctl restart flakelet-relay940agent # [ 4554.783137] agent flakelet-agent[3164]: {"timestamp":"2026-09-06T08:20:32.744843Z","level":"INFO","message":"update finished","flakelet":"other","status":"Unchanged","generation":"Some(1)","target":"flakelet_relay::agent::jobs"}941agent # [ 4554.799272] agent flakelet-agent[3164]: {"timestamp":"2026-09-06T08:20:32.761082Z","level":"INFO","message":"disconnected: websocket closed","relay":"https://relay:7443","target":"flakelet_relay::agent::conn"}942relay # [ 4554.784295] relay flakelet-relay[934]: {"timestamp":"2026-09-06T08:20:32.746059Z","level":"INFO","message":"deploy finished","job":"188b06c4636b65849b3ba8daac22dc7c","ok":true,"target":"flakelet_relay::relay::api"}943relay # [ 4554.799009] relay systemd[1]: Stopping flakelet-relay...944relay # [ 4554.848547] relay systemd[1]: flakelet-relay.service: Deactivated successfully.945relay # [ 4554.848940] relay systemd[1]: Stopped flakelet-relay.946relay # [ 4554.848977] relay systemd[1]: flakelet-relay.service: Consumed 2.363s CPU time over 9.223s wall clock time.947relay # [ 4554.885233] relay systemd[1]: Starting flakelet-relay...948agent # [ 4555.801870] agent flakelet-agent[3164]: {"timestamp":"2026-09-06T08:20:33.763620Z","level":"WARN","message":"connect failed: connect relay:7443: Connection refused (os error 111)","relay":"https://relay:7443","target":"flakelet_relay::agent::conn"}949relay: (finished: must succeed: systemctl restart flakelet-relay, in 1.50 seconds)950relay: waiting for success: curl -sf http://127.0.0.1:7400/metrics | grep -q 'agent_up{host="agent"} 1'951relay # [ 4556.286999] relay flakelet-relay[1324]: {"timestamp":"2026-09-06T08:20:34.248753Z","level":"INFO","message":"listening on 127.0.0.1:7400","target":"flakelet_relay::relay::server"}952relay # [ 4556.287847] relay flakelet-relay[1324]: {"timestamp":"2026-09-06T08:20:34.249262Z","level":"INFO","message":"listening on [::]:7443","target":"flakelet_relay::relay::server"}953relay # [ 4556.287639] relay systemd[1]: Started flakelet-relay.954agent # [ 4557.812226] agent flakelet-agent[3164]: {"timestamp":"2026-09-06T08:20:35.773979Z","level":"INFO","message":"connected","relay":"https://relay:7443","relay_name":"relay","relay_version":"0.1.0","host":"agent","target":"flakelet_relay::agent::conn"}955relay # [ 4557.812363] relay flakelet-relay[1324]: {"timestamp":"2026-09-06T08:20:35.774100Z","level":"INFO","message":"agent connected","host":"agent","version":"0.1.0","flakelets":"[Named { name: \"app\", generation: Some(9), revision: Some(\"prebuilt:app\") }, Named { name: \"other\", generation: Some(1), revision: Some(\"prebuilt:other\") }]","target":"flakelet_relay::relay::agent_conn"}956relay: (finished: waiting for success: curl -sf http://127.0.0.1:7400/metrics | grep -q 'agent_up{host="agent"} 1', in 2.05 seconds)957client: must succeed: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem jobs958client: (finished: must succeed: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem jobs, in 0.02 seconds)959client: must succeed: echo '{"groups": []}' | step crypto jwt sign --key /nix/store/r7sclamg09v434w8zjlfnrcrn1w79wh6-flakelet-relay-test-issuer/jwk.json --iss https://relay:8443 --aud flakelet-relay --sub 'repo:github:example/app:ref:refs/heads/main' --exp $(( $(date +%s) + 300 )) --jti $(head -c8 /dev/urandom | base32)960client: (finished: must succeed: echo '{"groups": []}' | step crypto jwt sign --key /nix/store/r7sclamg09v434w8zjlfnrcrn1w79wh6-flakelet-relay-test-issuer/jwk.json --iss https://relay:8443 --aud flakelet-relay --sub 'repo:github:example/app:ref:refs/heads/main' --exp $(( $(date +%s) + 300 )) --jti $(head -c8 /dev/urandom | base32), in 0.05 seconds)961client: must succeed: FLAKELET_RELAY_TOKEN_COMMAND='echo eyJhbGciOiJFZERTQSIsImtpZCI6ImsxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJmbGFrZWxldC1yZWxheSIsImV4cCI6MTc4ODY4MzEzNiwiZ3JvdXBzIjpbXSwiaWF0IjoxNzg4NjgyODM2LCJpc3MiOiJodHRwczovL3JlbGF5Ojg0NDMiLCJqdGkiOiJHUVlXTEpTWDZMQ0RNPT09IiwibmJmIjoxNzg4NjgyODM2LCJzdWIiOiJyZXBvOmdpdGh1YjpleGFtcGxlL2FwcDpyZWY6cmVmcy9oZWFkcy9tYWluIn0.y6G6hWOfR5NacFu8RqsqKdsr7VlpZHUOzd8Tv3ZuBskcPfge7OCLZXeU9MokOr6XlNRFID-z6xfNUpLJ_GeCDw' push --relay https://relay:7443 jobs962client: (finished: must succeed: FLAKELET_RELAY_TOKEN_COMMAND='echo eyJhbGciOiJFZERTQSIsImtpZCI6ImsxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJmbGFrZWxldC1yZWxheSIsImV4cCI6MTc4ODY4MzEzNiwiZ3JvdXBzIjpbXSwiaWF0IjoxNzg4NjgyODM2LCJpc3MiOiJodHRwczovL3JlbGF5Ojg0NDMiLCJqdGkiOiJHUVlXTEpTWDZMQ0RNPT09IiwibmJmIjoxNzg4NjgyODM2LCJzdWIiOiJyZXBvOmdpdGh1YjpleGFtcGxlL2FwcDpyZWY6cmVmcy9oZWFkcy9tYWluIn0.y6G6hWOfR5NacFu8RqsqKdsr7VlpZHUOzd8Tv3ZuBskcPfge7OCLZXeU9MokOr6XlNRFID-z6xfNUpLJ_GeCDw' push --relay https://relay:7443 jobs, in 0.02 seconds)963(finished: subtest: job history comes from the agents and survives a relay restart, in 4.31 seconds)964subtest: dashboard login via authorization code flow965client: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -o /dev/null -w '%{http_code} %{redirect_url}' https://relay:7443/ui/ | grep -q '303 https://relay:7443/ui/login'966client: (finished: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -o /dev/null -w '%{http_code} %{redirect_url}' https://relay:7443/ui/ | grep -q '303 https://relay:7443/ui/login', in 0.02 seconds)967client: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -L https://relay:7443/ui/login968client: (finished: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -L https://relay:7443/ui/login, in 0.08 seconds)969client: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar https://relay:7443/ui/jobs970client: (finished: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar https://relay:7443/ui/jobs, in 0.02 seconds)971client: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -f https://relay:7443/v1/jobs | grep -q hist1972client: (finished: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -f https://relay:7443/v1/jobs | grep -q hist1, in 0.02 seconds)973client: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -N --max-time 10 https://relay:7443/ui/jobs/hist1/events || true974client: (finished: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -N --max-time 10 https://relay:7443/ui/jobs/hist1/events || true, in 0.02 seconds)975client: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -o /dev/null -w '%{http_code}' -X POST 'https://relay:7443/ui/deploy?arg=other' | grep -q 403976client: (finished: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -o /dev/null -w '%{http_code}' -X POST 'https://relay:7443/ui/deploy?arg=other' | grep -q 403, in 0.02 seconds)977client: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -D - -o /dev/null -H 'HX-Request: true' -X POST 'https://relay:7443/ui/deploy?arg=other'978client: (finished: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -D - -o /dev/null -H 'HX-Request: true' -X POST 'https://relay:7443/ui/deploy?arg=other', in 0.02 seconds)979client: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -N --max-time 30 https://relay:7443/ui/jobs/d8ae15318f69e01f3f3f90b63ec13bad/events || true980relay # [ 4558.477044] relay 93sn3sv24fj26ksccpppzwg6kbf10s4w-device-mock[186]: 127.0.0.1 - - [06/Sep/2026 08:20:36] "GET /authorize?response_type=code&client_id=dashboard&redirect_uri=https%3A%2F%2Frelay%3A7443%2Fui%2Fcallback&scope=openid%20profile%20email%20groups&state=uVbdmHWfX2wb9BNRiArnya6u9aSOaTQvGeORzZ0U5gM&code_challenge=CXctGMkvkoGS0KpRTAP18SpOWr8xoT57DLeNlxBfVVw&code_challenge_method=S256 HTTP/1.1" 302 -981relay # [ 4558.528979] relay 93sn3sv24fj26ksccpppzwg6kbf10s4w-device-mock[186]: 127.0.0.1 - - [06/Sep/2026 08:20:36] "POST /token HTTP/1.1" 200 -982relay # [ 4558.529785] relay flakelet-relay[1324]: {"timestamp":"2026-09-06T08:20:36.491589Z","level":"INFO","message":"login","name":"someone","principals":"[\"oidc:mock:someone\", \"oidc:mock:groups:wheel\"]","target":"flakelet_relay::relay::login"}983relay # [ 4558.586046] relay flakelet-relay[1324]: {"timestamp":"2026-09-06T08:20:36.547810Z","level":"INFO","message":"job reattached","job":"188b06c4636b65849b3ba8daac22dc7c","targets":"[\"agent/other\"]","target":"flakelet_relay::relay::api"}984relay # [ 4558.629499] relay flakelet-relay[1324]: {"timestamp":"2026-09-06T08:20:36.591257Z","level":"INFO","message":"deploy accepted","job":"b41bdb4d57a8cae9358eb92ac75f95c3","caller":"oidc:mock:someone","targets":"[\"agent/other\"]","target":"flakelet_relay::relay::api"}985relay # [ 4558.650839] relay flakelet-relay[1324]: {"timestamp":"2026-09-06T08:20:36.612640Z","level":"INFO","message":"job reattached","job":"b41bdb4d57a8cae9358eb92ac75f95c3","targets":"[\"agent/other\"]","target":"flakelet_relay::relay::api"}986agent # [ 4558.629806] agent flakelet-agent[3164]: {"timestamp":"2026-09-06T08:20:36.591577Z","level":"INFO","message":"start","id":"c2047ca37729d2845c4cc26d801ee7ce","flakelet":"other","rule":"admins","relay":"relay","caller":"Some(\"oidc:mock:someone\\noidc:mock:groups:wheel\")","target":"flakelet_relay::agent::conn"}987agent # [ 4558.630175] agent flakelet-agent[3164]: {"timestamp":"2026-09-06T08:20:36.592008Z","level":"INFO","message":"update starting","flakelet":"other","ids":"[\"c2047ca37729d2845c4cc26d801ee7ce\"]","target":"flakelet_relay::agent::jobs"}988agent # [ 4558.673534] agent systemd[1]: Starting flakelet update other (via relay)...989agent # [ 4558.704457] agent systemd[1]: Started flakelet update other (via relay).990agent # [ 4558.708802] agent flakelet[3914]: other: using prebuilt artifact /nix/store/hmr990n0b4v6rpzrcpyyw738r9xi7z95-flakelet-other-v1991agent # [ 4558.709327] agent flakelet[3914]: other: up to date992agent # [ 4558.710022] agent systemd[1]: flakelet-relay-job-other.service: Deactivated successfully.993client: (finished: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -N --max-time 30 https://relay:7443/ui/jobs/d8ae15318f69e01f3f3f90b63ec13bad/events || true, in 0.63 seconds)994client: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar https://relay:7443/ui/jobs/d8ae15318f69e01f3f3f90b63ec13bad995client: (finished: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar https://relay:7443/ui/jobs/d8ae15318f69e01f3f3f90b63ec13bad, in 0.02 seconds)996client: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -sf -o /dev/null https://relay:7443/ui/static/htmx.min.js997client: (finished: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -sf -o /dev/null https://relay:7443/ui/static/htmx.min.js, in 0.02 seconds)998client: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar 'https://relay:7443/ui/?q=status%3Ahealthy+oth'999client: (finished: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar 'https://relay:7443/ui/?q=status%3Ahealthy+oth', in 0.02 seconds)1000client: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar https://relay:7443/ui/flakelets/other1001client: (finished: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar https://relay:7443/ui/flakelets/other, in 0.02 seconds)1002client: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar 'https://relay:7443/ui/hosts?q=status%3Adisconnected'1003client: (finished: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar 'https://relay:7443/ui/hosts?q=status%3Adisconnected', in 0.02 seconds)1004client: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -o /dev/null -w '%{http_code}' -H 'HX-Request: true' -X POST 'https://relay:7443/ui/retry?arg=d8ae15318f69e01f3f3f90b63ec13bad' | grep -q 4001005client: (finished: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -o /dev/null -w '%{http_code}' -H 'HX-Request: true' -X POST 'https://relay:7443/ui/retry?arg=d8ae15318f69e01f3f3f90b63ec13bad' | grep -q 400, in 0.02 seconds)1006client: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -N --max-time 15 'https://relay:7443/ui/events?path=jobs' > /tmp/ev.out 2>&1 &1007client: (finished: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -N --max-time 15 'https://relay:7443/ui/events?path=jobs' > /tmp/ev.out 2>&1 &, in 0.01 seconds)1008client: must succeed: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy --id live1 agent/other1009» job 38796a7ba508eadfe82a1cd5def000bc via relay 0.1.0 (agents: agent 0.1.0)1010» wave 01011agent/other | Starting flakelet update other (via relay)...1012agent/other | Started flakelet update other (via relay).1013agent/other | other: using prebuilt artifact /nix/store/hmr990n0b4v6rpzrcpyyw738r9xi7z95-flakelet-other-v11014agent/other | other: up to date1015agent # [ 4559.258432] agent flakelet-agent[3164]: {"timestamp":"2026-09-06T08:20:37.220192Z","level":"INFO","message":"update finished","flakelet":"other","status":"Unchanged","generation":"Some(1)","target":"flakelet_relay::agent::jobs"}1016agent # [ 4559.400107] agent flakelet-agent[3164]: {"timestamp":"2026-09-06T08:20:37.361899Z","level":"INFO","message":"start","id":"5e317fc57863c2a7c03ebe1bdacd1356","flakelet":"other","rule":"admins","relay":"relay","caller":"Some(\"x509:email:admin@example.org\")","target":"flakelet_relay::agent::conn"}1017agent/otheragent # [ 4559.400460] agent flakelet-agent[3164]: {"timestamp":"2026-09-06T08:20:37.362292Z","level":"INFO","message":"update starting","flakelet":"other","ids":"[\"5e317fc57863c2a7c03ebe1bdacd1356\"]","target":"flakelet_relay::agent::jobs"}1018 | flakelet-relay-job-other.service: Deactivated successfully.agent # [ 4559.456745] agent systemd[1]: Starting flakelet update other (via relay)...10191020agent # [ 4559.488998] agent systemd[1]: Started flakelet update other (via relay).1021agent # [ 4559.492937] agent flakelet[3952]: other: using prebuilt artifact /nix/store/hmr990n0b4v6rpzrcpyyw738r9xi7z95-flakelet-other-v11022agent # [ 4559.493395] agent flakelet[3952]: other: up to date1023agent # [ 4559.494816] agent systemd[1]: flakelet-relay-job-other.service: Deactivated successfully.1024relay # [ 4559.399836] relay flakelet-relay[1324]: {"timestamp":"2026-09-06T08:20:37.361584Z","level":"INFO","message":"deploy accepted","job":"38796a7ba508eadfe82a1cd5def000bc","caller":"x509:email:admin@example.org","targets":"[\"agent/other\"]","target":"flakelet_relay::relay::api"}1025» agent/other: unchanged (generation 1)1026» ok1027client: (finished: must succeed: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy --id live1 agent/other, in 0.66 seconds)1028client: waiting for success: grep -q live1 /tmp/ev.out1029client: (finished: waiting for success: grep -q live1 /tmp/ev.out, in 0.01 seconds)1030client: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -X POST -o /dev/null https://relay:7443/ui/logout1031client: (finished: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -X POST -o /dev/null https://relay:7443/ui/logout, in 0.02 seconds)1032client: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -o /dev/null -w '%{http_code}' https://relay:7443/ui/ | grep -q 3031033client: (finished: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -o /dev/null -w '%{http_code}' https://relay:7443/ui/ | grep -q 303, in 0.02 seconds)1034(finished: subtest: dashboard login via authorization code flow, in 1.67 seconds)1035(finished: run the VM test script, in 87.65 seconds)1036test script finished in 87.79s1037cleanup1038kill NspawnMachine (pid 52)1039kill NspawnMachine (pid 53)1040Container agent terminated by signal KILL.1041relay # [ 4560.045157] relay flakelet-relay[1324]: {"timestamp":"2026-09-06T08:20:38.006832Z","level":"INFO","message":"deploy finished","job":"38796a7ba508eadfe82a1cd5def000bc","ok":true,"target":"flakelet_relay::relay::api"}1042kill NspawnMachine (pid 56)1043Container client terminated by signal KILL.1044(finished: cleanup, in 0.39 seconds)1045additionally exposed symbols:1046 agent, client, relay,1047 vlan1,1048 start_all, test_script, machines, machines_qemu, machines_nspawn, vlans, driver, log, os, create_machine, subtest, run_tests, join_all, retry, serial_stdout_off, serial_stdout_on, polling_condition, BaseMachine, QemuMachine, NspawnMachine, t, debug, dump_machine_ssh1049» job d30c8ccd9850f8cfcbe259628ad443a4 via relay 0.1.0 (agents: agent 0.1.0)1050» wave 01051agent/app | Starting flakelet update app (via relay)...1052agent/app | Started flakelet update app (via relay).1053agent/app | app: using prebuilt artifact /nix/store/6ili1vxjlag9d7dy0bxrbxk6bpi5qh9p-flakelet-app-v21054agent/app | app: activating generation 21055agent/app | app: updated to generation 21056agent/app | flakelet-relay-job-app.service: Deactivated successfully.1057» agent/app: updated (generation 2)1058» ok10591060» job 75ef5f5be4a5fe966263e5925f6745cc via relay 0.1.0 (agents: agent 0.1.0)1061» wave 01062agent/app | Starting flakelet update app (via relay)...1063agent/app | Started flakelet update app (via relay).1064agent/app | app: using prebuilt artifact /nix/store/8282rnbvyk847q4l4zby3zwcq3jpzrvw-flakelet-app-broken1065agent/app | app: activating generation 41066agent/app | app: deploy failed, rolled back: systemctl start app.service failed:1067agent/app | failed units: app.service. See journalctl -u app.service1068agent/app | flakelet-relay-job-app.service: Main process exited, code=exited, status=1/FAILURE1069agent/app | flakelet-relay-job-app.service: Failed with result 'exit-code'.1070» agent/app: rolled-back (generation 3)1071agent/app │ Starting app v1...1072agent/app │ Started app v1.1073agent/app │ Stopping app v1...1074agent/app │ app.service: Deactivated successfully.1075agent/app │ Stopped app v1.1076agent/app │ Starting app v2...1077agent/app │ Started app v2.1078agent/app │ Stopping app v2...1079agent/app │ app.service: Deactivated successfully.1080agent/app │ Stopped app v2.1081agent/app │ Starting app v1...1082agent/app │ Started app v1.1083agent/app │ Stopping app v1...1084agent/app │ app.service: Deactivated successfully.1085agent/app │ Stopped app v1.1086agent/app │ Starting app broken...1087agent/app │ app.service: Unable to locate executable '/nonexistent': No such file or directory1088agent/app │ app.service: Failed at step EXEC spawning /nonexistent: No such file or directory1089agent/app │ app.service: Main process exited, code=exited, status=203/EXEC1090agent/app │ app.service: Failed with result 'exit-code'.1091agent/app │ Failed to start app broken.1092agent/app │ Starting app v1...1093agent/app │ Started app v1.1094» skipped: agent/other1095» failed10961097» job fc5d49a798d1b431af8fd0099b0d7754 via relay 0.1.0 (agents: agent 0.1.0)1098» wave 01099agent/app | Starting flakelet update app (via relay)...1100agent/app | Started flakelet update app (via relay).1101agent/app | app: using prebuilt artifact /nix/store/c67jj8kx3m50nyj874aqmdqf21w5iznd-flakelet-app-slow1102agent/app | app: activating generation 71103agent/app | app: updated to generation 71104agent/app | flakelet-relay-job-app.service: Deactivated successfully.1105» agent/app: updated (generation 7)1106» ok11071108» job fcec88019960b634f620a5fb8ef1b38a via relay 0.1.0 (agents: agent 0.1.0)1109» wave 01110agent/app | Starting flakelet update app (via relay)...1111agent/app | Started flakelet update app (via relay).1112agent/app | app: using prebuilt artifact /nix/store/ln47rsd5km8a04hd5zlkz4q1v352zqh3-flakelet-app-slow21113agent/app | app: activating generation 81114agent/app | app: updated to generation 81115agent/app | flakelet-relay-job-app.service: Deactivated successfully.1116» agent/app: updated (generation 8)1117» ok11181119» job 486e6ce5b02ce6df82a2d8eb7c4c4ecd via relay 0.1.0 (agents: agent 0.1.0)1120» wave 01121agent/app | Starting flakelet update app (via relay)...1122agent/app | Started flakelet update app (via relay).1123agent/app | app: using prebuilt artifact /nix/store/c67jj8kx3m50nyj874aqmdqf21w5iznd-flakelet-app-slow1124agent/app | app: activating generation 91125flakelet-push: stream: error reading a body from connection, resuming job a70a44a7a5707cece21054c1e6c803191126flakelet-push: https://relay:7443: connect relay:7443: Connection refused (os error 111), retrying in 1s1127flakelet-push: https://relay:7443: connect relay:7443: Connection refused (os error 111), retrying in 2s1128» job 486e6ce5b02ce6df82a2d8eb7c4c4ecd via relay 0.1.0 (agents: agent 0.1.0)1129» wave 01130agent/app | app: updated to generation 91131agent/app | flakelet-relay-job-app.service: Deactivated successfully.1132» agent/app: updated (generation 9)1133» ok113411351788682832 hist1 x509:email:admin@example.org agent/other:unchanged11361788682822 a70a44a7a5707cece21054c1e6c80319 x509:email:admin@example.org agent/app:updated11371788682813 a7f19d977a022febd46d4cec67d6fcdb x509:email:admin@example.org agent/app:updated11381788682797 810d8512d83958f19d115bd4660409a3 x509:email:admin@example.org agent/app:updated11391788682788 95c539f7cf0849fb0391973a03262a4d x509:email:admin@example.org agent/other:unchanged11401788682776 d88d64c0291e3d21bee391609fc3c7ac x509:email:admin@example.org agent/app:updated11411788682775 23529942aa3b7352cdd813ac71102efd x509:email:admin@example.org agent/app:unchanged11421788682774 2050bc877a2aeaf686b3f39e89a211e1 x509:email:admin@example.org agent/app:unchanged11431788682774 aaad37000fcfecda421a60238df9df73 x509:email:admin@example.org agent/app:unchanged11441788682769 cf02666942f1874edb95b0fc688392c2 x509:email:admin@example.org agent/app:rolled-back11451788682769 ff51713f438e89d848c3cd5329261ac5 oidc:mock:groups:wheel agent/other:unchanged11461788682768 e4b7ff0ca374863ec2d2462155978928 oidc:mock:groups:wheel agent/other:unchanged11471788682764 863b961dd4e442f8b1e19b73c6592987 oidc:mock:groups:wheel agent/other:unchanged11481788682761 3dbbed54b7d9639a680752b0826f2660 oidc:mock:repo:github:example/app:ref:refs/heads/main agent/app:updated11491788682758 12d47d955f1de71c5aac940e24f6b47b x509:email:admin@example.org agent/app:updated11501151Container relay terminated by signal KILL.