tribuchet: building on jamie Machine state will be reset. To keep it, pass --keep-machine-state start all VLans (finished: start all VLans, in 0.00 seconds) Initializing machine ID from random generator. Test will time out and terminate in 3600.0 seconds run the VM test script start all VMs agent: systemd-nspawn running (pid 52) client: systemd-nspawn running (pid 54) relay: systemd-nspawn running (pid 53) agent: Waiting for journal at /build/vm-state-agent/var/log/journal... client: Waiting for journal at /build/vm-state-client/var/log/journal... relay: Waiting for journal at /build/vm-state-relay/var/log/journal... (finished: start all VMs, in 0.01 seconds) relay: waiting for unit flakelet-relay.service nixos-nspawn(agent): TAP vde-tap1 not found; container will be isolated from VDE nixos-nspawn(agent): A common reason for this is that /dev/net is not available in the Nix sandbox. Try adding /dev/net to extra-sandbox-paths. nixos-nspawn(relay): TAP vde-tap1 not found; container will be isolated from VDE nixos-nspawn(relay): A common reason for this is that /dev/net is not available in the Nix sandbox. Try adding /dev/net to extra-sandbox-paths. nixos-nspawn(client): TAP vde-tap1 not found; container will be isolated from VDE nixos-nspawn(client): A common reason for this is that /dev/net is not available in the Nix sandbox. Try adding /dev/net to extra-sandbox-paths. Note: in a future version of systemd-nspawn the default set of permitted socket address families will be restricted to AF_INET, AF_INET6 and AF_UNIX. Use --restrict-address-families= to configure the set of permitted socket address families, or set RestrictAddressFamilies= in a .nspawn file. Note: in a future version of systemd-nspawn the default set of permitted socket address families will be restricted to AF_INET, AF_INET6 and AF_UNIX. Use --restrict-address-families= to configure the set of permitted socket address families, or set RestrictAddressFamilies= in a .nspawn file. ░ Spawning container relay on /build/vm-state-relay. ░ Spawning container client on /build/vm-state-client. Note: in a future version of systemd-nspawn the default set of permitted socket address families will be restricted to AF_INET, AF_INET6 and AF_UNIX. Use --restrict-address-families= to configure the set of permitted socket address families, or set RestrictAddressFamilies= in a .nspawn file. ░ Spawning container agent on /build/vm-state-agent. agent # No journal files were found. agent # No journal boot entry found for the specified boot (+0). relay # No journal files were found. relay # No journal boot entry found for the specified boot (+0). client # No journal files were found. client # No journal boot entry found for the specified boot (+0). agent # [514002.090760] agent systemd-journald[57]: Journal started agent # [514002.090805] agent systemd-journald[57]: Runtime Journal (/run/log/journal/12177be2e33c48acbe7100da0c1d8dd9) is 8M, max 4G, 3.9G free. agent # [514002.099785] agent systemd[1]: Finished Apply Kernel Variables. agent # [514002.112176] agent systemd[1]: Finished Create Static Device Nodes in /dev gracefully. agent # [514002.146709] agent systemd[1]: Starting Flush Journal to Persistent Storage... agent # [514002.147417] agent systemd[1]: Starting Create Static Device Nodes in /dev... agent # [514002.157263] agent systemd-journald[57]: Time spent on flushing to /var/log/journal/12177be2e33c48acbe7100da0c1d8dd9 is 1.539ms for 6 entries. agent # [514002.157263] agent systemd-journald[57]: System Journal (/var/log/journal/12177be2e33c48acbe7100da0c1d8dd9) is 512B, max 4G, 3.9G free. agent # [514002.167540] agent systemd[1]: Finished Create Static Device Nodes in /dev. agent # [514002.168344] agent systemd[1]: Reached target Preparation for Local File Systems. agent # [514002.168487] agent systemd[1]: Rule-based Manager for Device Events and Files skipped, unmet condition check ConditionPathIsReadWrite=/sys agent # [514002.180823] agent systemd[1]: Finished Flush Journal to Persistent Storage. agent # [514002.273449] agent systemd[1]: Finished Firewall. agent # [514003.090368] agent systemd[1]: Mounting /run/wrappers... agent # [514003.150509] agent systemd[1]: Mounted /run/wrappers. agent # [514003.151615] agent systemd[1]: Reached target Local File Systems. agent # [514003.152994] agent systemd[1]: Listening on Boot Loader Control Service Socket. relay # [514002.087358] relay systemd-journald[57]: Journal started relay # [514002.087398] relay systemd-journald[57]: Runtime Journal (/run/log/journal/6df87aee40ef47a38d17cddc1370f193) is 8M, max 4G, 3.9G free. relay # [514002.091962] relay systemd[1]: Finished Apply Kernel Variables. relay # [514002.103739] relay systemd[1]: Finished Create Static Device Nodes in /dev gracefully. relay # [514002.117519] relay systemd[1]: Starting Flush Journal to Persistent Storage... relay # [514002.118169] relay systemd[1]: Starting Create Static Device Nodes in /dev... relay # [514002.152143] relay systemd-journald[57]: Time spent on flushing to /var/log/journal/6df87aee40ef47a38d17cddc1370f193 is 1.487ms for 6 entries. relay # [514002.152143] relay systemd-journald[57]: System Journal (/var/log/journal/6df87aee40ef47a38d17cddc1370f193) is 512B, max 4G, 3.9G free. relay # [514002.166423] relay systemd[1]: Finished Create Static Device Nodes in /dev. relay # [514002.166600] relay systemd[1]: Reached target Preparation for Local File Systems. relay # [514002.166682] relay systemd[1]: Rule-based Manager for Device Events and Files skipped, unmet condition check ConditionPathIsReadWrite=/sys relay # [514002.174489] relay systemd[1]: Finished Flush Journal to Persistent Storage. relay # [514002.273178] relay systemd[1]: Finished Firewall. relay # [514003.093369] relay systemd[1]: Mounting /run/wrappers... relay # [514003.150136] relay systemd[1]: Mounted /run/wrappers. relay # [514003.151373] relay systemd[1]: Reached target Local File Systems. relay # [514003.152647] relay systemd[1]: Listening on Boot Loader Control Service Socket. relay # [514003.154071] relay systemd[1]: Starting Create SUID/SGID Wrappers... relay # [514003.154115] relay systemd[1]: Update Boot Loader Random Seed skipped, unmet condition check ConditionVirtualization=!container relay # [514003.155725] relay systemd[1]: Starting Save Transient machine-id to Disk... client # [514002.083905] client systemd-journald[57]: Journal started client # [514002.083948] client systemd-journald[57]: Runtime Journal (/run/log/journal/d13e066fb68a46248f8b4ffa3c857cb4) is 8M, max 4G, 3.9G free. client # [514002.092194] client systemd[1]: Finished Apply Kernel Variables. client # [514002.104293] client systemd[1]: Finished Create Static Device Nodes in /dev gracefully. client # [514002.119180] client systemd[1]: Starting Flush Journal to Persistent Storage... client # [514002.119878] client systemd[1]: Starting Create Static Device Nodes in /dev... client # [514002.152793] client systemd-journald[57]: Time spent on flushing to /var/log/journal/d13e066fb68a46248f8b4ffa3c857cb4 is 1.538ms for 6 entries. client # [514002.152793] client systemd-journald[57]: System Journal (/var/log/journal/d13e066fb68a46248f8b4ffa3c857cb4) is 512B, max 4G, 3.9G free. client # [514002.160925] client systemd[1]: Finished Create Static Device Nodes in /dev. client # [514002.161649] client systemd[1]: Reached target Preparation for Local File Systems. client # [514002.161760] client systemd[1]: Rule-based Manager for Device Events and Files skipped, unmet condition check ConditionPathIsReadWrite=/sys client # [514002.178054] client systemd[1]: Finished Flush Journal to Persistent Storage. client # [514002.246042] client systemd[1]: Finished Firewall. client # [514003.092132] client systemd[1]: Mounting /run/wrappers... client # [514003.150255] client systemd[1]: Mounted /run/wrappers. client # [514003.151299] client systemd[1]: Reached target Local File Systems. client # [514003.152507] client systemd[1]: Listening on Boot Loader Control Service Socket. client # [514003.154079] client systemd[1]: Starting Create SUID/SGID Wrappers... client # [514003.154128] client systemd[1]: Update Boot Loader Random Seed skipped, unmet condition check ConditionVirtualization=!container client # [514003.155736] client systemd[1]: Starting Save Transient machine-id to Disk... client # [514003.157215] client systemd[1]: Starting Create System Files and Directories... relay # [514003.156980] relay systemd[1]: Starting Create System Files and Directories... relay # [514003.175398] relay systemd-tmpfiles[169]: Cannot set file attributes for '/var/empty', value=0x00000010, mask=0x00000010, ignoring: Operation not permitted relay # [514003.175848] relay systemd-tmpfiles[169]: fchmod() of /var/log/journal failed: Operation not permitted client # [514003.175788] client systemd-tmpfiles[161]: Cannot set file attributes for '/var/empty', value=0x00000010, mask=0x00000010, ignoring: Operation not permitted client # [514003.176245] client systemd-tmpfiles[161]: fchmod() of /var/log/journal failed: Operation not permitted relay # [514003.176179] relay systemd-tmpfiles[169]: fchmod() of /var/log/journal/6df87aee40ef47a38d17cddc1370f193 failed: Operation not permitted relay # [514003.176528] relay systemd-tmpfiles[169]: fchmod() of /run/log/journal failed: Operation not permitted agent # [514003.154480] agent systemd[1]: Starting Create SUID/SGID Wrappers... agent # [514003.154532] agent systemd[1]: Update Boot Loader Random Seed skipped, unmet condition check ConditionVirtualization=!container agent # [514003.155622] agent systemd[1]: Starting Save Transient machine-id to Disk... client # [514003.176604] client suid-sgid-wrappers-start[168]: chmod: changing permissions of '/run/wrappers/wrappers.RNZg5ujPFV/chsh': Operation not permitted agent # [514003.157073] agent systemd[1]: Starting Create System Files and Directories... client # [514003.176559] client systemd-tmpfiles[161]: fchmod() of /var/log/journal/d13e066fb68a46248f8b4ffa3c857cb4 failed: Operation not permitted agent # [514003.176609] agent systemd-tmpfiles[161]: Cannot set file attributes for '/var/empty', value=0x00000010, mask=0x00000010, ignoring: Operation not permitted client # [514003.176902] client systemd-tmpfiles[161]: fchmod() of /run/log/journal failed: Operation not permitted agent # [514003.177035] agent systemd-tmpfiles[161]: fchmod() of /var/log/journal failed: Operation not permitted agent # [514003.177329] agent systemd-tmpfiles[161]: fchmod() of /var/log/journal/12177be2e33c48acbe7100da0c1d8dd9 failed: Operation not permitted agent # [514003.177646] agent systemd-tmpfiles[161]: fchmod() of /run/log/journal failed: Operation not permitted agent # [514003.180419] agent suid-sgid-wrappers-start[168]: chmod: changing permissions of '/run/wrappers/wrappers.fFbBZovtjN/chsh': Operation not permitted agent # [514003.183700] agent systemd[1]: suid-sgid-wrappers.service: Main process exited, code=exited, status=1/FAILURE agent # [514003.183740] agent systemd[1]: suid-sgid-wrappers.service: Failed with result 'exit-code'. agent # [514003.183795] agent systemd[1]: Failed to start Create SUID/SGID Wrappers. agent # [514003.196116] agent systemd[1]: Finished Create System Files and Directories. agent # [514003.247848] agent systemd[1]: etc-machine\x2did.mount: Deactivated successfully. agent # [514003.260914] agent systemd[1]: Finished Save Transient machine-id to Disk. agent # [514003.264475] agent systemd[1]: Starting Rebuild Journal Catalog... agent # [514003.265267] agent systemd[1]: Starting Record System Boot/Shutdown in UTMP... agent # [514003.304200] agent systemd[1]: Finished Record System Boot/Shutdown in UTMP. agent # [514003.311875] agent systemd[1]: Finished Rebuild Journal Catalog. agent # [514003.312965] agent systemd[1]: Starting Update is Completed... agent # [514003.324210] agent systemd[1]: Finished Update is Completed. agent # [514003.324465] agent systemd[1]: Reached target System Initialization. agent # [514003.324569] agent systemd[1]: Discard unused filesystem blocks once a week skipped, unmet condition check ConditionVirtualization=!container agent # [514003.324610] agent systemd[1]: Started Daily Cleanup of Temporary Directories. agent # [514003.324632] agent systemd[1]: Reached target Timer Units. agent # [514003.324924] agent systemd[1]: Listening on D-Bus System Message Bus Socket. agent # [514003.325454] agent systemd[1]: Listening on Nix Daemon Socket. agent # [514003.325637] agent systemd[1]: Listening on Virtual Machine and Container Registration Service Socket. agent # [514003.325662] agent systemd[1]: Reached target Socket Units. agent # [514003.325707] agent systemd[1]: Reached target Basic System. agent # [514003.327132] agent systemd[1]: Starting Re-link flakelet services at boot... agent # [514003.328047] agent systemd[1]: Starting Import lastlog data into lastlog2 database... agent # [514003.329567] agent systemd[1]: Starting Name Service Cache Daemon (nsncd)... agent # [514003.330764] agent systemd[1]: Starting resolvconf update... agent # [514003.338855] agent systemd[1]: Finished Re-link flakelet services at boot. agent # [514003.340175] agent systemd[1]: Starting Reconcile flakelet services with the host configuration... relay # [514003.179812] relay suid-sgid-wrappers-start[176]: chmod: changing permissions of '/run/wrappers/wrappers.eoM4ge5gXw/chsh': Operation not permitted relay # [514003.183060] relay systemd[1]: suid-sgid-wrappers.service: Main process exited, code=exited, status=1/FAILURE relay # [514003.183124] relay systemd[1]: suid-sgid-wrappers.service: Failed with result 'exit-code'. relay # [514003.183164] relay systemd[1]: Failed to start Create SUID/SGID Wrappers. relay # [514003.183684] relay systemd[1]: Finished Create System Files and Directories. relay # [514003.247136] relay systemd[1]: etc-machine\x2did.mount: Deactivated successfully. relay # [514003.260037] relay systemd[1]: Finished Save Transient machine-id to Disk. relay # [514003.263479] relay systemd[1]: Starting Rebuild Journal Catalog... relay # [514003.264389] relay systemd[1]: Starting Record System Boot/Shutdown in UTMP... relay # [514003.302517] relay systemd[1]: Finished Record System Boot/Shutdown in UTMP. relay # [514003.311572] relay systemd[1]: Finished Rebuild Journal Catalog. relay # [514003.312596] relay systemd[1]: Starting Update is Completed... relay # [514003.322036] relay systemd[1]: Finished Update is Completed. relay # [514003.322860] relay systemd[1]: Reached target System Initialization. relay # [514003.323039] relay systemd[1]: Discard unused filesystem blocks once a week skipped, unmet condition check ConditionVirtualization=!container relay # [514003.323091] relay systemd[1]: Started Daily Cleanup of Temporary Directories. relay # [514003.323115] relay systemd[1]: Reached target Timer Units. relay # [514003.323282] relay systemd[1]: Listening on D-Bus System Message Bus Socket. relay # [514003.323540] relay systemd[1]: Listening on Nix Daemon Socket. relay # [514003.323818] relay systemd[1]: Listening on Virtual Machine and Container Registration Service Socket. relay # [514003.323845] relay systemd[1]: Reached target Socket Units. relay # [514003.323891] relay systemd[1]: Reached target Basic System. relay # [514003.325660] relay systemd[1]: Started device-mock.service. relay # [514003.326824] relay systemd[1]: Starting Re-link flakelet services at boot... relay # [514003.327863] relay systemd[1]: Starting Import lastlog data into lastlog2 database... relay # [514003.329109] relay systemd[1]: Starting Name Service Cache Daemon (nsncd)... relay # [514003.330103] relay systemd[1]: Starting resolvconf update... relay # [514003.338172] relay systemd[1]: Finished Re-link flakelet services at boot. relay # [514003.339115] relay systemd[1]: Starting Reconcile flakelet services with the host configuration... client # [514003.182871] client systemd[1]: suid-sgid-wrappers.service: Main process exited, code=exited, status=1/FAILURE client # [514003.182964] client systemd[1]: suid-sgid-wrappers.service: Failed with result 'exit-code'. client # [514003.183114] client systemd[1]: Failed to start Create SUID/SGID Wrappers. client # [514003.183613] client systemd[1]: Finished Create System Files and Directories. client # [514003.247163] client systemd[1]: etc-machine\x2did.mount: Deactivated successfully. client # [514003.259943] client systemd[1]: Finished Save Transient machine-id to Disk. client # [514003.262794] client systemd[1]: Starting Rebuild Journal Catalog... client # [514003.263781] client systemd[1]: Starting Record System Boot/Shutdown in UTMP... client # [514003.304054] client systemd[1]: Finished Record System Boot/Shutdown in UTMP. client # [514003.310452] client systemd[1]: Finished Rebuild Journal Catalog. client # [514003.311444] client systemd[1]: Starting Update is Completed... client # [514003.320454] client systemd[1]: Finished Update is Completed. client # [514003.320677] client systemd[1]: Reached target System Initialization. client # [514003.320780] client systemd[1]: Discard unused filesystem blocks once a week skipped, unmet condition check ConditionVirtualization=!container client # [514003.320816] client systemd[1]: Started Daily Cleanup of Temporary Directories. client # [514003.320840] client systemd[1]: Reached target Timer Units. client # [514003.320996] client systemd[1]: Listening on D-Bus System Message Bus Socket. client # [514003.321250] client systemd[1]: Listening on Nix Daemon Socket. client # [514003.321419] client systemd[1]: Listening on Virtual Machine and Container Registration Service Socket. client # [514003.321442] client systemd[1]: Reached target Socket Units. client # [514003.321491] client systemd[1]: Reached target Basic System. client # [514003.323242] client systemd[1]: Starting Import lastlog data into lastlog2 database... client # [514003.324644] client systemd[1]: Starting Name Service Cache Daemon (nsncd)... client # [514003.326077] client systemd[1]: Starting resolvconf update... client # [514003.338882] client systemd[1]: Finished Import lastlog data into lastlog2 database. relay # [514003.350127] relay systemd[1]: Finished Reconcile flakelet services with the host configuration. relay # [514003.352010] relay systemd[1]: Finished Import lastlog data into lastlog2 database. relay # [514003.398860] relay systemd[1]: Finished resolvconf update. relay # [514003.399423] relay systemd[1]: Reached target Preparation for Network. relay # [514003.406482] relay systemd[1]: Starting Address configuration of eth1... relay # [514003.412720] relay systemd[1]: Starting Extra networking commands.... relay # [514003.419323] relay systemd[1]: nscd.service: Deactivated successfully. relay # [514003.445331] relay systemd[1]: Stopped Name Service Cache Daemon (nsncd). relay # [514003.448612] relay systemd[1]: Starting Name Service Cache Daemon (nsncd)... relay # [514003.463400] relay network-addresses-eth1-start[246]: adding address 192.168.1.3/24... done relay # [514003.466615] relay network-addresses-eth1-start[246]: adding address 2001:db8:1::3/64... done relay # [514003.471656] relay systemd[1]: Finished Address configuration of eth1. relay # [514003.505515] relay systemd[1]: Finished Extra networking commands.. relay # [514003.505687] relay systemd[1]: Reached target Network. relay # [514003.506997] relay systemd[1]: Starting Dnsmasq Daemon... relay # [514003.507819] relay systemd[1]: Starting Nginx Web Server... relay # [514003.508588] relay systemd[1]: Starting Update flakelet service flakelet-relay... relay # [514003.555382] relay flakelet[311]: flakelet-relay: using prebuilt artifact /nix/store/55zvswj0bvimz7faqq19rhsjlvqasa00-flakelet-flakelet-relay relay # [514003.555986] relay flakelet[311]: flakelet-relay: activating generation 1 relay # [514003.559605] relay systemd[1]: Reload requested from client PID 316 ('systemctl') (unit flakelet-flakelet-relay.service)... relay # [514003.559640] relay systemd[1]: Reloading... client # [514003.406312] client systemd[1]: Finished resolvconf update. client # [514003.406883] client systemd[1]: Reached target Preparation for Network. client # [514003.416288] client systemd[1]: Starting Address configuration of eth1... client # [514003.425730] client systemd[1]: Starting Extra networking commands.... client # [514003.445314] client systemd[1]: nscd.service: Deactivated successfully. client # [514003.445381] client systemd[1]: Stopped Name Service Cache Daemon (nsncd). client # [514003.449077] client systemd[1]: Starting Name Service Cache Daemon (nsncd)... client # [514003.463869] client network-addresses-eth1-start[233]: adding address 192.168.1.2/24... done client # [514003.467308] client network-addresses-eth1-start[233]: adding address 2001:db8:1::2/64... done client # [514003.471987] client systemd[1]: Finished Address configuration of eth1. client # [514003.547381] client systemd[1]: Finished Extra networking commands.. client # [514003.547692] client systemd[1]: nscd.service: Deactivated successfully. client # [514003.547892] client systemd[1]: Stopped Name Service Cache Daemon (nsncd). client # [514003.550312] client systemd[1]: Reached target Network. client # [514003.551625] client systemd[1]: Starting Name Service Cache Daemon (nsncd)... agent # [514003.346444] agent systemd[1]: Finished Import lastlog data into lastlog2 database. agent # [514003.352873] agent systemd[1]: Finished Reconcile flakelet services with the host configuration. agent # [514003.445365] agent systemd[1]: Finished resolvconf update. agent # [514003.445565] agent systemd[1]: nscd.service: Deactivated successfully. agent # [514003.445615] agent systemd[1]: Stopped Name Service Cache Daemon (nsncd). agent # [514003.447924] agent systemd[1]: Reached target Preparation for Network. agent # [514003.449188] agent systemd[1]: Starting Address configuration of eth1... agent # [514003.449919] agent systemd[1]: Starting Extra networking commands.... agent # [514003.450735] agent systemd[1]: Starting Name Service Cache Daemon (nsncd)... agent # [514003.471731] agent network-addresses-eth1-start[240]: adding address 192.168.1.1/24... done agent # [514003.474967] agent network-addresses-eth1-start[240]: adding address 2001:db8:1::1/64... done agent # [514003.480062] agent systemd[1]: Finished Address configuration of eth1. agent # [514003.547369] agent systemd[1]: Finished Extra networking commands.. agent # [514003.547574] agent systemd[1]: Reached target Network. agent # [514003.548848] agent systemd[1]: Starting Update flakelet service app... agent # [514003.550054] agent systemd[1]: Starting Update flakelet service flakelet-agent... agent # [514003.550722] agent systemd[1]: Starting Update flakelet service other... agent # [514003.555651] agent systemd[1]: nscd.service: Deactivated successfully. agent # [514003.555839] agent systemd[1]: Stopped Name Service Cache Daemon (nsncd). agent # [514003.558409] agent systemd[1]: Starting Name Service Cache Daemon (nsncd)... agent # [514003.561410] agent flakelet[304]: app: using prebuilt artifact /nix/store/cqj0khcw75fy7djy7y554bcgxi50911r-flakelet-app-v1 agent # [514003.561980] agent flakelet[304]: app: activating generation 1 agent # [514003.562258] agent flakelet[305]: flakelet-agent: using prebuilt artifact /nix/store/flgya4ggb72hyvnnjcsjw6zbknb4w1s4-flakelet-flakelet-agent agent # [514003.562822] agent flakelet[305]: flakelet-agent: activating generation 1 agent # [514003.562944] agent flakelet[306]: other: using prebuilt artifact /nix/store/h4k7d4dqcfa4h9d48c0rm4x08qziqfaf-flakelet-other-v1 agent # [514003.563489] agent flakelet[306]: other: activating generation 1 agent # [514003.566198] agent systemd[1]: Reload requested from client PID 311 ('systemctl') (unit flakelet-app.service)... agent # [514003.566233] agent systemd[1]: Reloading... relay # [514003.906499] relay nsncd[251]: Sep 17 11:58:32.193 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" relay # [514003.982291] relay systemd[1]: Reloading finished in 422 ms. relay # [514004.124183] relay systemd[1]: Started Name Service Cache Daemon (nsncd). relay # [514004.125049] relay systemd[1]: Reached target Host and Network Name Lookups. relay # [514004.125192] relay systemd[1]: Reached target User and Group Name Lookups. relay # [514004.126830] relay systemd[1]: Starting User Login Management... relay # [514004.127938] relay systemd[1]: Starting Permit User Sessions... relay # [514004.144363] relay systemd[1]: Reload requested from client PID 357 ('systemctl') (unit flakelet-flakelet-relay.service)... relay # [514004.144412] relay systemd[1]: Reloading... client # [514003.899480] client nsncd[303]: Sep 17 11:58:32.186 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" client # [514003.899523] client systemd[1]: Started Name Service Cache Daemon (nsncd). client # [514003.899618] client systemd[1]: Reached target Host and Network Name Lookups. client # [514003.899734] client systemd[1]: Reached target User and Group Name Lookups. client # [514003.902516] client systemd[1]: Starting User Login Management... client # [514003.904213] client systemd[1]: Starting Permit User Sessions... client # [514003.936450] client systemd[1]: Finished Permit User Sessions. client # [514003.937736] client systemd[1]: Started Console Getty. client # [514003.937764] client systemd[1]: Getty on tty1 skipped, unmet condition check ConditionPathExists=/dev/tty0 client # [514003.937776] client systemd[1]: Reached target Login Prompts. agent # [514003.878996] agent nsncd[310]: Sep 17 11:58:32.166 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" agent # [514003.993798] agent systemd[1]: Reloading finished in 427 ms. agent # [514004.119226] agent systemd[1]: Started Name Service Cache Daemon (nsncd). agent # [514004.122344] agent systemd[1]: Reload requested from client PID 314 ('systemctl') (unit flakelet-other.service)... agent # [514004.122390] agent systemd[1]: Reloading... relay # [514004.156975] relay dnsmasq-pre-start[360]: dnsmasq: syntax check OK. relay # [514004.499568] relay systemd[1]: Reloading finished in 354 ms. agent # [514004.506585] agent systemd[1]: Reloading finished in 383 ms. agent # [514004.639993] agent systemd[1]: Reload requested from client PID 313 ('systemctl') (unit flakelet-flakelet-agent.service)... agent # [514004.640059] agent systemd[1]: Reloading... relay # [514004.642040] relay systemd[1]: Finished Permit User Sessions. relay # [514004.670168] relay systemd[1]: Started Console Getty. relay # [514004.670340] relay systemd[1]: Getty on tty1 skipped, unmet condition check ConditionPathExists=/dev/tty0 relay # [514004.670557] relay systemd[1]: Reached target Login Prompts. relay # [514004.675321] relay systemd[1]: Starting flakelet-relay... relay # [514005.006943] relay nginx-pre-start[402]: nginx: the configuration file /nix/store/a6vli3znypjpjrh5rzjh88sj1pb5al8x-nginx.conf syntax is ok relay # [514005.007755] relay nginx-pre-start[402]: nginx: configuration file /nix/store/a6vli3znypjpjrh5rzjh88sj1pb5al8x-nginx.conf test is successful relay # [514005.023630] relay systemd[1]: Started Nginx Web Server. agent # [514005.037416] agent systemd[1]: Reloading finished in 396 ms. agent # [514005.126733] agent systemd[1]: Reached target Host and Network Name Lookups. agent # [514005.126889] agent systemd[1]: Reached target User and Group Name Lookups. agent # [514005.128244] agent systemd[1]: Starting User Login Management... agent # [514005.129443] agent systemd[1]: Starting Permit User Sessions... agent # [514005.136526] agent systemd[1]: Reload requested from client PID 408 ('systemctl') (unit flakelet-other.service)... agent # [514005.136555] agent systemd[1]: Reloading... client # [514005.128992] client systemd-logind[315]: New seat seat0. client # [514005.129370] client systemd[1]: Started User Login Management. client # [514005.131641] client systemd[1]: Starting D-Bus System Message Bus... client # [514005.159621] client systemd[1]: Starting linger-users.service... client # [514005.184970] client systemd[1]: linger-users.service: Deactivated successfully. client # [514005.185323] client systemd[1]: Finished linger-users.service. client # [514005.185676] client systemd[1]: Reached target Multi-User System. relay # [514005.291970] relay systemd[1]: Starting D-Bus System Message Bus... relay # [514005.497041] relay systemd-logind[354]: New seat seat0. relay # [514005.497424] relay systemd[1]: Started User Login Management. relay # [514005.500074] relay systemd[1]: Starting linger-users.service... relay # [514005.539128] relay systemd[1]: linger-users.service: Deactivated successfully. relay # [514005.539508] relay systemd[1]: Finished linger-users.service. agent # [514005.519646] agent systemd[1]: Reloading finished in 382 ms. agent # [514005.598747] agent systemd[1]: Finished Permit User Sessions. agent # [514005.604647] agent systemd[1]: Reload requested from client PID 409 ('systemctl') (unit flakelet-app.service)... agent # [514005.604693] agent systemd[1]: Reloading... client # [514005.736095] client dbus-broker-launch[325]: Looking up NSS user entry for 'systemd-timesync'... client # [514005.737742] client dbus-broker-launch[325]: NSS returned no entry for 'systemd-timesync' client # [514005.737742] client dbus-broker-launch[325]: Invalid user-name in /nix/store/bavypp7cmwvqb019gw72zkdw9ha0biq8-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync" client # [514005.738750] client systemd[1]: Started D-Bus System Message Bus. client # [514005.739275] client systemd[1]: Startup finished in 4.079s. client # [514005.743762] client dbus-broker-launch[325]: Ready relay # [514005.820323] relay dbus-broker-launch[408]: Looking up NSS user entry for 'systemd-timesync'... relay # [514005.821603] relay dbus-broker-launch[408]: NSS returned no entry for 'systemd-timesync' relay # [514005.821603] relay dbus-broker-launch[408]: Invalid user-name in /nix/store/43w9dr1mn08yxp891i9sgc66qamk73pz-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync" relay # [514005.822727] relay systemd[1]: Started D-Bus System Message Bus. relay # [514005.828258] relay dbus-broker-launch[408]: Ready relay # [514005.831597] relay systemd[1]: Started Dnsmasq Daemon. relay # [514005.833786] relay dnsmasq[394]: started, version 2.93 cachesize 150 relay # [514005.833796] relay dnsmasq[394]: compile time options: IPv6 GNU-getopt DBus no-UBus no-i18n IDN2 DHCP DHCPv6 no-Lua TFTP conntrack ipset nftset auth DNSSEC loop-detect inotify dumpfile relay # [514005.833800] relay dnsmasq[394]: DBus support enabled: connected to system bus relay # [514005.833808] relay dnsmasq[394]: no upstream servers configured - please set them from DBus relay # [514005.833911] relay dnsmasq[394]: read /etc/hosts - 9 names agent # [514005.955736] agent systemd[1]: Reloading finished in 350 ms. agent # [514006.026180] agent systemd[1]: Started Console Getty. agent # [514006.026276] agent systemd[1]: Getty on tty1 skipped, unmet condition check ConditionPathExists=/dev/tty0 agent # [514006.026395] agent systemd[1]: Reached target Login Prompts. agent # [514006.026995] agent systemd[1]: Reload requested from client PID 410 ('systemctl') (unit flakelet-flakelet-agent.service)... agent # [514006.027034] agent systemd[1]: Reloading... agent # [514006.365215] agent systemd[1]: Reloading finished in 337 ms. agent # [514006.440674] agent systemd[1]: Starting other v1... agent # [514006.443195] agent systemd[1]: Starting app v1... agent # [514006.469016] agent systemd[1]: Starting flakelet-agent... agent # [514006.472526] agent systemd[1]: Started other v1. agent # [514006.473201] agent systemd[1]: Started app v1. agent # [514006.476383] agent systemd-logind[405]: New seat seat0. agent # [514006.488336] agent systemd[1]: Started User Login Management. agent # [514006.491784] agent systemd[1]: Starting D-Bus System Message Bus... agent # [514006.491836] agent flakelet[306]: other: updated to generation 1 agent # [514006.493235] agent systemd[1]: Starting linger-users.service... agent # [514006.493788] agent systemd[1]: Finished Update flakelet service other. agent # [514006.495453] agent flakelet[304]: app: updated to generation 1 agent # [514006.497587] agent systemd[1]: Finished Update flakelet service app. agent # [514006.504912] agent systemd[1]: linger-users.service: Deactivated successfully. agent # [514006.505072] agent systemd[1]: Finished linger-users.service. agent # [514006.518293] agent flakelet-agent[504]: {"timestamp":"2026-09-17T11:58:34.805766Z","level":"INFO","message":"local change","flakelet":"app","generation":1,"by":"host activation","target":"flakelet_relay::agent::jobs"} agent # [514006.518630] agent flakelet-agent[504]: {"timestamp":"2026-09-17T11:58:34.805935Z","level":"INFO","message":"local change","flakelet":"other","generation":1,"by":"host activation","target":"flakelet_relay::agent::jobs"} agent # [514006.518608] agent systemd[1]: Started flakelet-agent. agent # [514006.520949] agent flakelet-agent[504]: {"timestamp":"2026-09-17T11:58:34.808421Z","level":"WARN","message":"connect failed: connect relay:7443: Connection refused (os error 111)","relay":"https://relay:7443","target":"flakelet_relay::agent::conn"} agent # [514006.524830] agent flakelet[305]: flakelet-agent: updated to generation 1 agent # [514006.526792] agent systemd[1]: Finished Update flakelet service flakelet-agent. agent # [514006.527992] agent systemd[1]: Reached target flakelet managed services. agent # [514006.528087] agent systemd[1]: Reached target Multi-User System. agent # [514006.536951] agent flakelet-agent[504]: {"timestamp":"2026-09-17T11:58:34.824445Z","level":"INFO","message":"local change","flakelet":"flakelet-agent","generation":1,"by":"host activation","target":"flakelet_relay::agent::jobs"} relay # [514006.679359] relay flakelet-relay[400]: {"timestamp":"2026-09-17T11:58:34.966777Z","level":"INFO","message":"listening on 127.0.0.1:7400","target":"flakelet_relay::relay::server"} relay # [514006.680102] relay flakelet-relay[400]: {"timestamp":"2026-09-17T11:58:34.967166Z","level":"INFO","message":"listening on [::]:7443","target":"flakelet_relay::relay::server"} relay # [514006.679928] relay systemd[1]: Started flakelet-relay. relay # [514006.691229] relay systemd[1]: Starting health for flakelet-relay... relay: (finished: waiting for unit flakelet-relay.service, in 6.17 seconds) relay: waiting for unit nginx.service agent # [514006.910204] agent dbus-broker-launch[706]: Looking up NSS user entry for 'systemd-timesync'... agent # [514006.911720] agent dbus-broker-launch[706]: NSS returned no entry for 'systemd-timesync' agent # [514006.911720] agent dbus-broker-launch[706]: Invalid user-name in /nix/store/flwdz8g9rqzw07wvh1g1j68syb1p53xd-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync" agent # [514006.912234] agent systemd[1]: Started D-Bus System Message Bus. agent # [514006.912559] agent systemd[1]: Startup finished in 5.240s. agent # [514006.918289] agent dbus-broker-launch[706]: Ready relay: (finished: waiting for unit nginx.service, in 0.02 seconds) agent: waiting for unit flakelet.target relay # [514006.958063] relay systemd[1]: flakelet-relay-health.service: Deactivated successfully. relay # [514006.958427] relay systemd[1]: Finished health for flakelet-relay. relay # [514006.965971] relay flakelet[311]: flakelet-relay: updated to generation 1 relay # [514006.968600] relay systemd[1]: Finished Update flakelet service flakelet-relay. relay # [514006.969171] relay systemd[1]: Reached target flakelet managed services. relay # [514006.969343] relay systemd[1]: Reached target Multi-User System. relay # [514006.969595] relay systemd[1]: Startup finished in 5.324s. agent: (finished: waiting for unit flakelet.target, in 0.02 seconds) agent: waiting for unit flakelet-agent.service agent: (finished: waiting for unit flakelet-agent.service, in 0.02 seconds) client: waiting for unit multi-user.target client: (finished: waiting for unit multi-user.target, in 0.02 seconds) relay: waiting for success: curl -sf http://127.0.0.1:7400/metrics | grep -q 'agent_up{host="agent"} 1' agent # [514008.534127] agent flakelet-agent[504]: {"timestamp":"2026-09-17T11:58:36.821507Z","level":"INFO","message":"connected","relay":"https://relay:7443","relay_name":"relay","relay_version":"0.1.0","host":"agent","target":"flakelet_relay::agent::conn"} relay # [514008.534205] relay flakelet-relay[400]: {"timestamp":"2026-09-17T11:58:36.821597Z","level":"INFO","message":"agent connected","host":"agent","version":"0.1.0","flakelets":"[Named { name: \"app\", generation: Some(1), revision: Some(\"prebuilt:app\") }, Named { name: \"flakelet-agent\", generation: Some(1), revision: Some(\"prebuilt:flakelet-agent\") }, Named { name: \"other\", generation: Some(1), revision: Some(\"prebuilt:other\") }]","target":"flakelet_relay::relay::agent_conn"} relay: (finished: waiting for success: curl -sf http://127.0.0.1:7400/metrics | grep -q 'agent_up{host="agent"} 1', in 2.05 seconds) agent: must succeed: systemctl is-active app.service other.service agent: (finished: must succeed: systemctl is-active app.service other.service, in 0.01 seconds) subtest: deploy with client certificate, relay found via SRV agent: must succeed: jq '.services.app.prebuilt="/nix/store/1hhsfvlan5s6l308sr1yx2i20rkz1mdc-flakelet-app-v2"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json agent: (finished: must succeed: jq '.services.app.prebuilt="/nix/store/1hhsfvlan5s6l308sr1yx2i20rkz1mdc-flakelet-app-v2"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json, in 0.01 seconds) client: must succeed: push --relay-srv test --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem deploy agent/app 2>&1 agent # [514009.494430] agent flakelet-agent[504]: {"timestamp":"2026-09-17T11:58:37.781862Z","level":"INFO","message":"start","id":"d661ec204f7cb640c930552b1e6ce9e0","flakelet":"app","rule":"admins","relay":"relay","caller_name":"\"admin@example.org\"","target":"flakelet_relay::agent::conn"} agent # [514009.494945] agent flakelet-agent[504]: {"timestamp":"2026-09-17T11:58:37.782442Z","level":"INFO","message":"update starting","flakelet":"app","ids":"[\"d661ec204f7cb640c930552b1e6ce9e0\"]","target":"flakelet_relay::agent::jobs"} agent # [514009.530708] agent systemd[1]: Starting flakelet update app (via relay)... agent # [514009.560158] agent systemd[1]: Started flakelet update app (via relay). agent # [514009.564984] agent flakelet[926]: app: using prebuilt artifact /nix/store/1hhsfvlan5s6l308sr1yx2i20rkz1mdc-flakelet-app-v2 agent # [514009.565703] agent flakelet[926]: app: activating generation 2 agent # [514009.582154] agent systemd[1]: Stopping app v1... agent # [514009.584034] agent systemd[1]: app.service: Deactivated successfully. agent # [514009.584342] agent systemd[1]: Stopped app v1. agent # [514009.601195] agent systemd[1]: Reload requested from client PID 933 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [514009.601263] agent systemd[1]: Reloading... relay # [514009.494035] relay flakelet-relay[400]: {"timestamp":"2026-09-17T11:58:37.781467Z","level":"INFO","message":"deploy accepted","job":"718651f3253803c42a213aff77f3b07c","caller":"admin@example.org","targets":"[\"agent/app\"]","target":"flakelet_relay::relay::api"} agent # [514009.941312] agent systemd[1]: Reloading finished in 339 ms. agent # [514010.023302] agent systemd[1]: Reload requested from client PID 963 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [514010.023340] agent systemd[1]: Reloading... agent # [514010.347547] agent systemd[1]: Reloading finished in 323 ms. agent # [514010.411880] agent systemd[1]: Reload requested from client PID 992 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [514010.411920] agent systemd[1]: Reloading... agent # [514010.736723] agent systemd[1]: Reloading finished in 324 ms. agent # [514010.823073] agent systemd[1]: Reload requested from client PID 1026 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [514010.823108] agent systemd[1]: Reloading... agent # [514011.146401] agent systemd[1]: Reloading finished in 322 ms. agent # [514011.214388] agent systemd[1]: Starting app v2... agent # [514011.245208] agent systemd[1]: Started app v2. agent # [514011.251268] agent flakelet[926]: app: updated to generation 2 agent # [514011.254376] agent systemd[1]: flakelet-relay-job-app.service: Deactivated successfully. client: (finished: must succeed: push --relay-srv test --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem deploy agent/app 2>&1, in 2.89 seconds) agent: must succeed: systemctl show app.service -p Description | grep -q 'app v2' agent: (finished: must succeed: systemctl show app.service -p Description | grep -q 'app v2', in 0.02 seconds) agent: must succeed: journalctl -u flakelet-relay-job-app.service | grep -q 'generation 2' agent: (finished: must succeed: journalctl -u flakelet-relay-job-app.service | grep -q 'generation 2', in 0.01 seconds) agent: must succeed: flakelet status --json app | jq -c '.[0].changed.by' agent: (finished: must succeed: flakelet status --json app | jq -c '.[0].changed.by', in 0.03 seconds) (finished: subtest: deploy with client certificate, relay found via SRV, in 2.96 seconds) subtest: deploy with OIDC token, sub principal agent: must succeed: jq '.services.app.prebuilt="/nix/store/cqj0khcw75fy7djy7y554bcgxi50911r-flakelet-app-v1"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json agent: (finished: must succeed: jq '.services.app.prebuilt="/nix/store/cqj0khcw75fy7djy7y554bcgxi50911r-flakelet-app-v1"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json, in 0.01 seconds) client: must succeed: echo '{"groups": []}' | step crypto jwt sign --key /nix/store/4y8k1nvlafnzaadg5frncjjarv7aw3ry-flakelet-relay-test-issuer/jwk.json --iss https://relay:8443 --aud flakelet-relay --sub 'repo:github:example/app:ref:refs/heads/main' --exp $(( $(date +%s) + 300 )) --jti $(head -c8 /dev/urandom | base32) client: (finished: must succeed: echo '{"groups": []}' | step crypto jwt sign --key /nix/store/4y8k1nvlafnzaadg5frncjjarv7aw3ry-flakelet-relay-test-issuer/jwk.json --iss https://relay:8443 --aud flakelet-relay --sub 'repo:github:example/app:ref:refs/heads/main' --exp $(( $(date +%s) + 300 )) --jti $(head -c8 /dev/urandom | base32), in 0.05 seconds) client: must succeed: FLAKELET_RELAY_TOKEN_COMMAND='echo eyJhbGciOiJFZERTQSIsImtpZCI6ImsxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJmbGFrZWxldC1yZWxheSIsImV4cCI6MTc4OTY0NjYyMCwiZ3JvdXBzIjpbXSwiaWF0IjoxNzg5NjQ2MzIwLCJpc3MiOiJodHRwczovL3JlbGF5Ojg0NDMiLCJqdGkiOiJNV1lZN1pIR1lUSDdTPT09IiwibmJmIjoxNzg5NjQ2MzIwLCJzdWIiOiJyZXBvOmdpdGh1YjpleGFtcGxlL2FwcDpyZWY6cmVmcy9oZWFkcy9tYWluIn0.zeqrUWz4NdebMLWSxuEMIHFGwXq4s58OvldJGTuMCMUdDjDohAAL1gG-WhC9cqyJnG1cEyETvWgrDmcciey_AQ' push --relay https://relay:7443 deploy agent/app 2>&1 agent # [514012.353311] agent flakelet-agent[504]: {"timestamp":"2026-09-17T11:58:40.640724Z","level":"INFO","message":"update finished","flakelet":"app","status":"Updated","generation":"Some(2)","target":"flakelet_relay::agent::jobs"} agent # [514012.521255] agent flakelet-agent[504]: {"timestamp":"2026-09-17T11:58:40.808687Z","level":"INFO","message":"start","id":"3a624a3a8626e2d95bba62f883df4e95","flakelet":"app","rule":"ci","relay":"relay","caller_name":"\"repo:github:example/app:ref:refs/heads/main\"","target":"flakelet_relay::agent::conn"} agent # [514012.521557] agent flakelet-agent[504]: {"timestamp":"2026-09-17T11:58:40.809056Z","level":"INFO","message":"update starting","flakelet":"app","ids":"[\"3a624a3a8626e2d95bba62f883df4e95\"]","target":"flakelet_relay::agent::jobs"} agent # [514012.562176] agent systemd[1]: Starting flakelet update app (via relay)... agent # [514012.592024] agent systemd[1]: Started flakelet update app (via relay). agent # [514012.595767] agent flakelet[1096]: app: using prebuilt artifact /nix/store/cqj0khcw75fy7djy7y554bcgxi50911r-flakelet-app-v1 agent # [514012.596392] agent flakelet[1096]: app: activating generation 3 agent # [514012.609028] agent systemd[1]: Stopping app v2... relay # [514012.355045] relay flakelet-relay[400]: {"timestamp":"2026-09-17T11:58:40.642450Z","level":"INFO","message":"deploy finished","job":"718651f3253803c42a213aff77f3b07c","ok":true,"target":"flakelet_relay::relay::api"} relay # [514012.521067] relay flakelet-relay[400]: {"timestamp":"2026-09-17T11:58:40.808432Z","level":"INFO","message":"deploy accepted","job":"4667d4aaa93dd7b48cacd5a1edd7028e","caller":"repo:github:example/app:ref:refs/heads/main","targets":"[\"agent/app\"]","target":"flakelet_relay::relay::api"} agent # [514012.609486] agent systemd[1]: app.service: Deactivated successfully. agent # [514012.609740] agent systemd[1]: Stopped app v2. agent # [514012.626497] agent systemd[1]: Reload requested from client PID 1103 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [514012.626566] agent systemd[1]: Reloading... agent # [514012.967556] agent systemd[1]: Reloading finished in 340 ms. agent # [514013.048146] agent systemd[1]: Reload requested from client PID 1133 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [514013.048183] agent systemd[1]: Reloading... agent # [514013.371386] agent systemd[1]: Reloading finished in 322 ms. agent # [514013.457179] agent systemd[1]: Reload requested from client PID 1162 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [514013.457217] agent systemd[1]: Reloading... agent # [514013.785732] agent systemd[1]: Reloading finished in 328 ms. agent # [514013.873568] agent systemd[1]: Reload requested from client PID 1193 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [514013.873604] agent systemd[1]: Reloading... agent # [514014.194942] agent systemd[1]: Reloading finished in 320 ms. agent # [514014.264148] agent systemd[1]: Starting app v1... agent # [514014.295115] agent systemd[1]: Started app v1. agent # [514014.299639] agent flakelet[1096]: app: updated to generation 3 agent # [514014.302119] agent systemd[1]: flakelet-relay-job-app.service: Deactivated successfully. client: (finished: must succeed: FLAKELET_RELAY_TOKEN_COMMAND='echo eyJhbGciOiJFZERTQSIsImtpZCI6ImsxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJmbGFrZWxldC1yZWxheSIsImV4cCI6MTc4OTY0NjYyMCwiZ3JvdXBzIjpbXSwiaWF0IjoxNzg5NjQ2MzIwLCJpc3MiOiJodHRwczovL3JlbGF5Ojg0NDMiLCJqdGkiOiJNV1lZN1pIR1lUSDdTPT09IiwibmJmIjoxNzg5NjQ2MzIwLCJzdWIiOiJyZXBvOmdpdGh1YjpleGFtcGxlL2FwcDpyZWY6cmVmcy9oZWFkcy9tYWluIn0.zeqrUWz4NdebMLWSxuEMIHFGwXq4s58OvldJGTuMCMUdDjDohAAL1gG-WhC9cqyJnG1cEyETvWgrDmcciey_AQ' push --relay https://relay:7443 deploy agent/app 2>&1, in 2.93 seconds) (finished: subtest: deploy with OIDC token, sub principal, in 3.00 seconds) subtest: policy denies uncovered target client: must fail: FLAKELET_RELAY_TOKEN_COMMAND='echo eyJhbGciOiJFZERTQSIsImtpZCI6ImsxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJmbGFrZWxldC1yZWxheSIsImV4cCI6MTc4OTY0NjYyMCwiZ3JvdXBzIjpbXSwiaWF0IjoxNzg5NjQ2MzIwLCJpc3MiOiJodHRwczovL3JlbGF5Ojg0NDMiLCJqdGkiOiJNV1lZN1pIR1lUSDdTPT09IiwibmJmIjoxNzg5NjQ2MzIwLCJzdWIiOiJyZXBvOmdpdGh1YjpleGFtcGxlL2FwcDpyZWY6cmVmcy9oZWFkcy9tYWluIn0.zeqrUWz4NdebMLWSxuEMIHFGwXq4s58OvldJGTuMCMUdDjDohAAL1gG-WhC9cqyJnG1cEyETvWgrDmcciey_AQ' push --relay https://relay:7443 deploy agent/other 2>&1 client: (finished: must fail: FLAKELET_RELAY_TOKEN_COMMAND='echo eyJhbGciOiJFZERTQSIsImtpZCI6ImsxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJmbGFrZWxldC1yZWxheSIsImV4cCI6MTc4OTY0NjYyMCwiZ3JvdXBzIjpbXSwiaWF0IjoxNzg5NjQ2MzIwLCJpc3MiOiJodHRwczovL3JlbGF5Ojg0NDMiLCJqdGkiOiJNV1lZN1pIR1lUSDdTPT09IiwibmJmIjoxNzg5NjQ2MzIwLCJzdWIiOiJyZXBvOmdpdGh1YjpleGFtcGxlL2FwcDpyZWY6cmVmcy9oZWFkcy9tYWluIn0.zeqrUWz4NdebMLWSxuEMIHFGwXq4s58OvldJGTuMCMUdDjDohAAL1gG-WhC9cqyJnG1cEyETvWgrDmcciey_AQ' push --relay https://relay:7443 deploy agent/other 2>&1, in 0.02 seconds) (finished: subtest: policy denies uncovered target, in 0.03 seconds) subtest: claim principal from groups, unchanged counts as success client: must succeed: echo '{"groups": ["wheel"]}' | step crypto jwt sign --key /nix/store/4y8k1nvlafnzaadg5frncjjarv7aw3ry-flakelet-relay-test-issuer/jwk.json --iss https://relay:8443 --aud flakelet-relay --sub 'someone' --exp $(( $(date +%s) + 300 )) --jti $(head -c8 /dev/urandom | base32) client: (finished: must succeed: echo '{"groups": ["wheel"]}' | step crypto jwt sign --key /nix/store/4y8k1nvlafnzaadg5frncjjarv7aw3ry-flakelet-relay-test-issuer/jwk.json --iss https://relay:8443 --aud flakelet-relay --sub 'someone' --exp $(( $(date +%s) + 300 )) --jti $(head -c8 /dev/urandom | base32), in 0.05 seconds) client: must succeed: FLAKELET_RELAY_TOKEN_COMMAND='echo eyJhbGciOiJFZERTQSIsImtpZCI6ImsxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJmbGFrZWxldC1yZWxheSIsImV4cCI6MTc4OTY0NjYyMywiZ3JvdXBzIjpbIndoZWVsIl0sImlhdCI6MTc4OTY0NjMyMywiaXNzIjoiaHR0cHM6Ly9yZWxheTo4NDQzIiwianRpIjoiN1hTTUpURjRKTUkzTz09PSIsIm5iZiI6MTc4OTY0NjMyMywic3ViIjoic29tZW9uZSJ9.ePtT2y3uzP-gRtyjS_xKnkS9qVwQeV7IhQjVyYUVunHzntoKO7MhWVEKHpbIGJ0Qlp528x_LU11nSia6XCmHBw' push --relay https://relay:7443 deploy agent/other 2>&1 agent # [514015.409994] agent flakelet-agent[504]: {"timestamp":"2026-09-17T11:58:43.697395Z","level":"INFO","message":"update finished","flakelet":"app","status":"Updated","generation":"Some(3)","target":"flakelet_relay::agent::jobs"} agent # [514015.518238] agent flakelet-agent[504]: {"timestamp":"2026-09-17T11:58:43.805682Z","level":"INFO","message":"start","id":"6298494196acada2565ea344fb54b692","flakelet":"other","rule":"admins","relay":"relay","caller_name":"\"someone\"","target":"flakelet_relay::agent::conn"} agent # [514015.518773] agent flakelet-agent[504]: {"timestamp":"2026-09-17T11:58:43.806263Z","level":"INFO","message":"update starting","flakelet":"other","ids":"[\"6298494196acada2565ea344fb54b692\"]","target":"flakelet_relay::agent::jobs"} agent # [514015.556267] agent systemd[1]: Starting flakelet update other (via relay)... agent # [514015.585245] agent systemd[1]: Started flakelet update other (via relay). agent # [514015.591011] agent flakelet[1247]: other: using prebuilt artifact /nix/store/h4k7d4dqcfa4h9d48c0rm4x08qziqfaf-flakelet-other-v1 agent # [514015.591971] agent flakelet[1247]: other: up to date agent # [514015.592627] agent systemd[1]: flakelet-relay-job-other.service: Deactivated successfully. relay # [514015.411543] relay flakelet-relay[400]: {"timestamp":"2026-09-17T11:58:43.698910Z","level":"INFO","message":"deploy finished","job":"4667d4aaa93dd7b48cacd5a1edd7028e","ok":true,"target":"flakelet_relay::relay::api"} relay # [514015.517868] relay flakelet-relay[400]: {"timestamp":"2026-09-17T11:58:43.805287Z","level":"INFO","message":"deploy accepted","job":"908b46607c76674a74ffd9fc06af729c","caller":"someone","targets":"[\"agent/other\"]","target":"flakelet_relay::relay::api"} client: (finished: must succeed: FLAKELET_RELAY_TOKEN_COMMAND='echo eyJhbGciOiJFZERTQSIsImtpZCI6ImsxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJmbGFrZWxldC1yZWxheSIsImV4cCI6MTc4OTY0NjYyMywiZ3JvdXBzIjpbIndoZWVsIl0sImlhdCI6MTc4OTY0NjMyMywiaXNzIjoiaHR0cHM6Ly9yZWxheTo4NDQzIiwianRpIjoiN1hTTUpURjRKTUkzTz09PSIsIm5iZiI6MTc4OTY0NjMyMywic3ViIjoic29tZW9uZSJ9.ePtT2y3uzP-gRtyjS_xKnkS9qVwQeV7IhQjVyYUVunHzntoKO7MhWVEKHpbIGJ0Qlp528x_LU11nSia6XCmHBw' push --relay https://relay:7443 deploy agent/other 2>&1, in 1.64 seconds) (finished: subtest: claim principal from groups, unchanged counts as success, in 1.70 seconds) subtest: push login via device flow, cached token used afterwards client: must succeed: push login --issuer https://relay:8443 --client-id flakelet-relay 2>&1 agent # [514017.132233] agent flakelet-agent[504]: {"timestamp":"2026-09-17T11:58:45.419643Z","level":"INFO","message":"update finished","flakelet":"other","status":"Unchanged","generation":"Some(1)","target":"flakelet_relay::agent::jobs"} relay # [514017.134201] relay flakelet-relay[400]: {"timestamp":"2026-09-17T11:58:45.421639Z","level":"INFO","message":"deploy finished","job":"908b46607c76674a74ffd9fc06af729c","ok":true,"target":"flakelet_relay::relay::api"} relay # [514017.156745] relay 7gsx6vaqmkb2vf8rr0w4y1g018hi9d43-device-mock[186]: 127.0.0.1 - - [17/Sep/2026 11:58:45] "POST /device HTTP/1.1" 200 - relay # [514018.161442] relay 7gsx6vaqmkb2vf8rr0w4y1g018hi9d43-device-mock[186]: 127.0.0.1 - - [17/Sep/2026 11:58:46] "POST /token HTTP/1.1" 400 - client: (finished: must succeed: push login --issuer https://relay:8443 --client-id flakelet-relay 2>&1, in 2.08 seconds) client: must succeed: test "$(stat -c %a /root/.local/state/flakelet-push/token.json)" = 600 client: (finished: must succeed: test "$(stat -c %a /root/.local/state/flakelet-push/token.json)" = 600, in 0.01 seconds) client: must succeed: push --relay https://relay:7443 deploy agent/other 2>&1 agent # [514019.249403] agent flakelet-agent[504]: {"timestamp":"2026-09-17T11:58:47.536841Z","level":"INFO","message":"start","id":"fbe45a86fef058984ac3ba016f9b6bdb","flakelet":"other","rule":"admins","relay":"relay","caller_name":"\"dev@example.org\"","target":"flakelet_relay::agent::conn"} agent # [514019.250200] agent flakelet-agent[504]: {"timestamp":"2026-09-17T11:58:47.537689Z","level":"INFO","message":"update starting","flakelet":"other","ids":"[\"fbe45a86fef058984ac3ba016f9b6bdb\"]","target":"flakelet_relay::agent::jobs"} agent # [514019.290171] agent systemd[1]: Starting flakelet update other (via relay)... agent # [514019.318144] agent systemd[1]: Started flakelet update other (via relay). agent # [514019.322970] agent flakelet[1276]: other: using prebuilt artifact /nix/store/h4k7d4dqcfa4h9d48c0rm4x08qziqfaf-flakelet-other-v1 agent # [514019.323532] agent flakelet[1276]: other: up to date agent # [514019.326793] agent systemd[1]: flakelet-relay-job-other.service: Deactivated successfully. relay # [514019.214219] relay 7gsx6vaqmkb2vf8rr0w4y1g018hi9d43-device-mock[186]: 127.0.0.1 - - [17/Sep/2026 11:58:47] "POST /token HTTP/1.1" 200 - relay # [514019.248992] relay flakelet-relay[400]: {"timestamp":"2026-09-17T11:58:47.536460Z","level":"INFO","message":"deploy accepted","job":"8e74b0f318b0c4ae5a54615fca63e91d","caller":"dev@example.org","targets":"[\"agent/other\"]","target":"flakelet_relay::relay::api"} client: (finished: must succeed: push --relay https://relay:7443 deploy agent/other 2>&1, in 0.64 seconds) client: must succeed: push --relay https://relay:7443 deploy agent/other 2>&1 agent # [514019.863758] agent flakelet-agent[504]: {"timestamp":"2026-09-17T11:58:48.151176Z","level":"INFO","message":"update finished","flakelet":"other","status":"Unchanged","generation":"Some(1)","target":"flakelet_relay::agent::jobs"} agent # [514019.885788] agent flakelet-agent[504]: {"timestamp":"2026-09-17T11:58:48.173263Z","level":"INFO","message":"start","id":"f2af0682c4ebe9a8f92f86b612d57360","flakelet":"other","rule":"admins","relay":"relay","caller_name":"\"dev@example.org\"","target":"flakelet_relay::agent::conn"} agent # [514019.886176] agent flakelet-agent[504]: {"timestamp":"2026-09-17T11:58:48.173666Z","level":"INFO","message":"update starting","flakelet":"other","ids":"[\"f2af0682c4ebe9a8f92f86b612d57360\"]","target":"flakelet_relay::agent::jobs"} agent # [514019.923819] agent systemd[1]: Starting flakelet update other (via relay)... agent # [514019.958263] agent systemd[1]: Started flakelet update other (via relay). agent # [514019.962744] agent flakelet[1296]: other: using prebuilt artifact /nix/store/h4k7d4dqcfa4h9d48c0rm4x08qziqfaf-flakelet-other-v1 agent # [514019.963294] agent flakelet[1296]: other: up to date agent # [514019.966682] agent systemd[1]: flakelet-relay-job-other.service: Deactivated successfully. relay # [514019.865347] relay flakelet-relay[400]: {"timestamp":"2026-09-17T11:58:48.152778Z","level":"INFO","message":"deploy finished","job":"8e74b0f318b0c4ae5a54615fca63e91d","ok":true,"target":"flakelet_relay::relay::api"} relay # [514019.885606] relay flakelet-relay[400]: {"timestamp":"2026-09-17T11:58:48.173087Z","level":"INFO","message":"deploy accepted","job":"565a665a03908801241a5e0ba964e8f6","caller":"dev@example.org","targets":"[\"agent/other\"]","target":"flakelet_relay::relay::api"} client: (finished: must succeed: push --relay https://relay:7443 deploy agent/other 2>&1, in 0.64 seconds) (finished: subtest: push login via device flow, cached token used afterwards, in 3.37 seconds) subtest: waves: failing first wave skips the second agent: must succeed: jq '.services.app.prebuilt="/nix/store/qy8h9kkvfrxq0bc47vwjy6lfnk9i95ga-flakelet-app-broken"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json agent: (finished: must succeed: jq '.services.app.prebuilt="/nix/store/qy8h9kkvfrxq0bc47vwjy6lfnk9i95ga-flakelet-app-broken"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json, in 0.01 seconds) client: must fail: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem deploy agent/app --wave agent/other 2>&1 agent # [514020.504258] agent flakelet-agent[504]: {"timestamp":"2026-09-17T11:58:48.791670Z","level":"INFO","message":"update finished","flakelet":"other","status":"Unchanged","generation":"Some(1)","target":"flakelet_relay::agent::jobs"} agent # [514020.542125] agent flakelet-agent[504]: {"timestamp":"2026-09-17T11:58:48.829569Z","level":"INFO","message":"start","id":"789c9c77efd2ad47fab19564ea997f7d","flakelet":"app","rule":"admins","relay":"relay","caller_name":"\"admin@example.org\"","target":"flakelet_relay::agent::conn"} agent # [514020.542465] agent flakelet-agent[504]: {"timestamp":"2026-09-17T11:58:48.829957Z","level":"INFO","message":"update starting","flakelet":"app","ids":"[\"789c9c77efd2ad47fab19564ea997f7d\"]","target":"flakelet_relay::agent::jobs"} agent # [514020.575343] agent systemd[1]: Starting flakelet update app (via relay)... agent # [514020.607221] agent systemd[1]: Started flakelet update app (via relay). agent # [514020.611635] agent flakelet[1323]: app: using prebuilt artifact /nix/store/qy8h9kkvfrxq0bc47vwjy6lfnk9i95ga-flakelet-app-broken agent # [514020.612294] agent flakelet[1323]: app: activating generation 4 agent # [514020.629107] agent systemd[1]: Stopping app v1... agent # [514020.630769] agent systemd[1]: app.service: Deactivated successfully. agent # [514020.631044] agent systemd[1]: Stopped app v1. agent # [514020.646410] agent systemd[1]: Reload requested from client PID 1330 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [514020.646478] agent systemd[1]: Reloading... relay # [514020.505524] relay flakelet-relay[400]: {"timestamp":"2026-09-17T11:58:48.792947Z","level":"INFO","message":"deploy finished","job":"565a665a03908801241a5e0ba964e8f6","ok":true,"target":"flakelet_relay::relay::api"} relay # [514020.541853] relay flakelet-relay[400]: {"timestamp":"2026-09-17T11:58:48.829328Z","level":"INFO","message":"deploy accepted","job":"7785a7578bc59044d2221ef3e1e57ec4","caller":"admin@example.org","targets":"[\"agent/app\", \"agent/other\"]","target":"flakelet_relay::relay::api"} agent # [514020.981630] agent systemd[1]: Reloading finished in 334 ms. agent # [514021.054206] agent systemd[1]: Reload requested from client PID 1360 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [514021.054243] agent systemd[1]: Reloading... agent # [514021.376585] agent systemd[1]: Reloading finished in 321 ms. agent # [514021.452236] agent systemd[1]: Reload requested from client PID 1389 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [514021.452273] agent systemd[1]: Reloading... agent # [514021.772873] agent systemd[1]: Reloading finished in 320 ms. agent # [514021.848678] agent systemd[1]: Reload requested from client PID 1420 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [514021.848712] agent systemd[1]: Reloading... agent # [514022.169655] agent systemd[1]: Reloading finished in 320 ms. agent # [514022.242780] agent systemd[1]: Starting app broken... agent # [514022.273478] agent (nonexistent)[1450]: app.service: Unable to locate executable '/nonexistent': No such file or directory agent # [514022.273500] agent (nonexistent)[1450]: app.service: Failed at step EXEC spawning /nonexistent: No such file or directory agent # [514022.275099] agent systemd[1]: app.service: Main process exited, code=exited, status=203/EXEC agent # [514022.275208] agent systemd[1]: app.service: Failed with result 'exit-code'. agent # [514022.275572] agent systemd[1]: Failed to start app broken. agent # [514022.298672] agent systemd[1]: Reload requested from client PID 1456 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [514022.298710] agent systemd[1]: Reloading... agent # [514022.623426] agent systemd[1]: Reloading finished in 324 ms. agent # [514022.701185] agent systemd[1]: Reload requested from client PID 1489 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [514022.701220] agent systemd[1]: Reloading... agent # [514023.022443] agent systemd[1]: Reloading finished in 320 ms. agent # [514023.096970] agent systemd[1]: Reload requested from client PID 1520 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [514023.097015] agent systemd[1]: Reloading... agent # [514023.425038] agent systemd[1]: Reloading finished in 327 ms. agent # [514023.483996] agent systemd[1]: Reload requested from client PID 1550 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [514023.484041] agent systemd[1]: Reloading... agent # [514023.805170] agent systemd[1]: Reloading finished in 320 ms. agent # [514023.884435] agent systemd[1]: Starting app v1... agent # [514023.913062] agent systemd[1]: Started app v1. agent # [514023.914285] agent flakelet[1323]: app: deploy failed, rolled back: systemctl start app.service failed: agent # [514023.914285] agent flakelet[1323]: failed units: app.service. See journalctl -u app.service agent # [514023.916970] agent systemd[1]: flakelet-relay-job-app.service: Main process exited, code=exited, status=1/FAILURE agent # [514023.917076] agent systemd[1]: flakelet-relay-job-app.service: Failed with result 'exit-code'. client: (finished: must fail: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem deploy agent/app --wave agent/other 2>&1, in 4.12 seconds) agent: must succeed: systemctl show app.service -p Description | grep -q 'app v1' agent: (finished: must succeed: systemctl show app.service -p Description | grep -q 'app v1', in 0.01 seconds) (finished: subtest: waves: failing first wave skips the second, in 4.15 seconds) subtest: host patterns expand on the relay agent: must succeed: jq '.services.app.prebuilt="/nix/store/cqj0khcw75fy7djy7y554bcgxi50911r-flakelet-app-v1"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json agent: (finished: must succeed: jq '.services.app.prebuilt="/nix/store/cqj0khcw75fy7djy7y554bcgxi50911r-flakelet-app-v1"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json, in 0.01 seconds) client: must succeed: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem deploy '@all/app' 2>&1 agent # [514024.637554] agent flakelet-agent[504]: {"timestamp":"2026-09-17T11:58:52.924968Z","level":"INFO","message":"update finished","flakelet":"app","status":"RolledBack","generation":"Some(3)","target":"flakelet_relay::agent::jobs"} agent # [514024.689381] agent flakelet-agent[504]: {"timestamp":"2026-09-17T11:58:52.976822Z","level":"INFO","message":"start","id":"1c2a7a87f197df64f61cb529fd35ceb5","flakelet":"app","rule":"admins","relay":"relay","caller_name":"\"admin@example.org\"","target":"flakelet_relay::agent::conn"} agent # [514024.689791] agent flakelet-agent[504]: {"timestamp":"2026-09-17T11:58:52.977281Z","level":"INFO","message":"update starting","flakelet":"app","ids":"[\"1c2a7a87f197df64f61cb529fd35ceb5\"]","target":"flakelet_relay::agent::jobs"} agent # [514024.715803] agent systemd[1]: Starting flakelet update app (via relay)... agent # [514024.746147] agent systemd[1]: Started flakelet update app (via relay). agent # [514024.750363] agent flakelet[1610]: app: using prebuilt artifact /nix/store/cqj0khcw75fy7djy7y554bcgxi50911r-flakelet-app-v1 agent # [514024.750949] agent flakelet[1610]: app: up to date agent # [514024.753561] agent systemd[1]: flakelet-relay-job-app.service: Deactivated successfully. relay # [514024.638824] relay flakelet-relay[400]: {"timestamp":"2026-09-17T11:58:52.926259Z","level":"INFO","message":"deploy finished","job":"7785a7578bc59044d2221ef3e1e57ec4","ok":false,"target":"flakelet_relay::relay::api"} relay # [514024.689151] relay flakelet-relay[400]: {"timestamp":"2026-09-17T11:58:52.976591Z","level":"INFO","message":"deploy accepted","job":"5a94dbba39c0e0eaa5af7dc2234183bc","caller":"admin@example.org","targets":"[\"agent/app\"]","target":"flakelet_relay::relay::api"} client: (finished: must succeed: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem deploy '@all/app' 2>&1, in 0.62 seconds) client: must fail: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem deploy '*/app' 2>&1 agent # [514025.287100] agent flakelet-agent[504]: {"timestamp":"2026-09-17T11:58:53.574523Z","level":"INFO","message":"update finished","flakelet":"app","status":"Unchanged","generation":"Some(3)","target":"flakelet_relay::agent::jobs"} agent # [514025.310262] agent flakelet-agent[504]: {"timestamp":"2026-09-17T11:58:53.597750Z","level":"INFO","message":"start","id":"89d5195b8ebbb6bf2ba9c0ad8d490edc","flakelet":"app","rule":"admins","relay":"relay","caller_name":"\"admin@example.org\"","target":"flakelet_relay::agent::conn"} agent # [514025.310476] agent flakelet-agent[504]: {"timestamp":"2026-09-17T11:58:53.597970Z","level":"INFO","message":"update starting","flakelet":"app","ids":"[\"89d5195b8ebbb6bf2ba9c0ad8d490edc\"]","target":"flakelet_relay::agent::jobs"} agent # [514025.335153] agent systemd[1]: Starting flakelet update app (via relay)... agent # [514025.366069] agent systemd[1]: Started flakelet update app (via relay). agent # [514025.370376] agent flakelet[1630]: app: using prebuilt artifact /nix/store/cqj0khcw75fy7djy7y554bcgxi50911r-flakelet-app-v1 agent # [514025.370929] agent flakelet[1630]: app: up to date agent # [514025.373456] agent systemd[1]: flakelet-relay-job-app.service: Deactivated successfully. relay # [514025.288245] relay flakelet-relay[400]: {"timestamp":"2026-09-17T11:58:53.575684Z","level":"INFO","message":"deploy finished","job":"5a94dbba39c0e0eaa5af7dc2234183bc","ok":true,"target":"flakelet_relay::relay::api"} relay # [514025.309998] relay flakelet-relay[400]: {"timestamp":"2026-09-17T11:58:53.597482Z","level":"INFO","message":"deploy accepted","job":"b3eafc5ba98d10a36d91f72a3e01b3c0","caller":"admin@example.org","targets":"[\"agent/app\", \"ghost/app\"]","target":"flakelet_relay::relay::api"} client: (finished: must fail: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem deploy '*/app' 2>&1, in 0.62 seconds) client: must fail: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem deploy agent/app --wave '*/app' 2>&1 agent # [514025.910261] agent flakelet-agent[504]: {"timestamp":"2026-09-17T11:58:54.197672Z","level":"INFO","message":"update finished","flakelet":"app","status":"Unchanged","generation":"Some(3)","target":"flakelet_relay::agent::jobs"} agent # [514025.937341] agent flakelet-agent[504]: {"timestamp":"2026-09-17T11:58:54.224791Z","level":"INFO","message":"start","id":"2f8ace70734c0db79a276a41bc965567","flakelet":"app","rule":"admins","relay":"relay","caller_name":"\"admin@example.org\"","target":"flakelet_relay::agent::conn"} agent # [514025.937686] agent flakelet-agent[504]: {"timestamp":"2026-09-17T11:58:54.225181Z","level":"INFO","message":"update starting","flakelet":"app","ids":"[\"2f8ace70734c0db79a276a41bc965567\"]","target":"flakelet_relay::agent::jobs"} agent # [514025.972417] agent systemd[1]: Starting flakelet update app (via relay)... agent # [514026.005401] agent systemd[1]: Started flakelet update app (via relay). agent # [514026.010122] agent flakelet[1650]: app: using prebuilt artifact /nix/store/cqj0khcw75fy7djy7y554bcgxi50911r-flakelet-app-v1 agent # [514026.010712] agent flakelet[1650]: app: up to date agent # [514026.014553] agent systemd[1]: flakelet-relay-job-app.service: Deactivated successfully. relay # [514025.911793] relay flakelet-relay[400]: {"timestamp":"2026-09-17T11:58:54.199252Z","level":"INFO","message":"deploy finished","job":"b3eafc5ba98d10a36d91f72a3e01b3c0","ok":false,"target":"flakelet_relay::relay::api"} relay # [514025.937123] relay flakelet-relay[400]: {"timestamp":"2026-09-17T11:58:54.224611Z","level":"INFO","message":"deploy accepted","job":"3dc5b822ea258ccba30d46a074b6f09f","caller":"admin@example.org","targets":"[\"agent/app\", \"ghost/app\"]","target":"flakelet_relay::relay::api"} client: (finished: must fail: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem deploy agent/app --wave '*/app' 2>&1, in 0.64 seconds) client: must fail: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem deploy '@all/nope' 2>&1 client: (finished: must fail: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem deploy '@all/nope' 2>&1, in 0.02 seconds) (finished: subtest: host patterns expand on the relay, in 1.92 seconds) subtest: detach returns once the job is accepted agent: must succeed: jq '.services.app.prebuilt="/nix/store/yqdz0mcr1b2ljpw9vli2zvbmm43j4gr2-flakelet-app-slow2"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json agent: (finished: must succeed: jq '.services.app.prebuilt="/nix/store/yqdz0mcr1b2ljpw9vli2zvbmm43j4gr2-flakelet-app-slow2"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json, in 0.01 seconds) client: must succeed: timeout 20 push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem deploy --detach agent/app 2>&1 client: (finished: must succeed: timeout 20 push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem deploy --detach agent/app 2>&1, in 0.02 seconds) agent: waiting for success: systemctl is-active flakelet-relay-job-app.service agent # [514026.546525] agent flakelet-agent[504]: {"timestamp":"2026-09-17T11:58:54.833978Z","level":"INFO","message":"update finished","flakelet":"app","status":"Unchanged","generation":"Some(3)","target":"flakelet_relay::agent::jobs"} agent # [514026.611211] agent flakelet-agent[504]: {"timestamp":"2026-09-17T11:58:54.898655Z","level":"INFO","message":"start","id":"3863177a439ba5d75c0460e8e749614c","flakelet":"app","rule":"admins","relay":"relay","caller_name":"\"admin@example.org\"","target":"flakelet_relay::agent::conn"} agent # [514026.611763] agent flakelet-agent[504]: {"timestamp":"2026-09-17T11:58:54.899250Z","level":"INFO","message":"update starting","flakelet":"app","ids":"[\"3863177a439ba5d75c0460e8e749614c\"]","target":"flakelet_relay::agent::jobs"} agent # [514026.639558] agent systemd[1]: Starting flakelet update app (via relay)... agent # [514026.678131] agent systemd[1]: Started flakelet update app (via relay). agent # [514026.682183] agent flakelet[1678]: app: using prebuilt artifact /nix/store/yqdz0mcr1b2ljpw9vli2zvbmm43j4gr2-flakelet-app-slow2 agent # [514026.682835] agent flakelet[1678]: app: activating generation 4 agent # [514026.692336] agent systemd[1]: Stopping app v1... agent # [514026.693045] agent systemd[1]: app.service: Deactivated successfully. agent # [514026.693169] agent systemd[1]: Stopped app v1. agent # [514026.702513] agent systemd[1]: Reload requested from client PID 1685 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [514026.702549] agent systemd[1]: Reloading... relay # [514026.549668] relay flakelet-relay[400]: {"timestamp":"2026-09-17T11:58:54.837117Z","level":"INFO","message":"deploy finished","job":"3dc5b822ea258ccba30d46a074b6f09f","ok":false,"target":"flakelet_relay::relay::api"} relay # [514026.610912] relay flakelet-relay[400]: {"timestamp":"2026-09-17T11:58:54.898350Z","level":"INFO","message":"deploy accepted","job":"cf24ff62fb693ff11642f0795bb5208d","caller":"admin@example.org","targets":"[\"agent/app\"]","target":"flakelet_relay::relay::api"} agent # [514027.024799] agent systemd[1]: Reloading finished in 321 ms. agent # [514027.116483] agent systemd[1]: Reload requested from client PID 1715 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [514027.116522] agent systemd[1]: Reloading... agent # [514027.438144] agent systemd[1]: Reloading finished in 321 ms. agent # [514027.507989] agent systemd[1]: Reload requested from client PID 1744 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [514027.508031] agent systemd[1]: Reloading... agent: (finished: waiting for success: systemctl is-active flakelet-relay-job-app.service, in 1.30 seconds) ??? Warning (UserWarning): wait_until_fails(): passing a bare int/float as a duration is deprecated. Use datetime.timedelta instead. File "/nix/store/463h52kqv09byfnk9gw61bsa1h1qb8bh-nixos-test-driver-1.1/lib/python3.14/site-packages/test_driver/duration.py", line 39 agent: waiting for failure: systemctl is-active flakelet-relay-job-app.service ??? Warning (UserWarning): execute(): passing a bare int/float as a duration is deprecated. Use datetime.timedelta instead. File "/nix/store/463h52kqv09byfnk9gw61bsa1h1qb8bh-nixos-test-driver-1.1/lib/python3.14/site-packages/test_driver/duration.py", line 39 agent # [514027.829134] agent systemd[1]: Reloading finished in 320 ms. agent # [514027.924577] agent systemd[1]: Reload requested from client PID 1777 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [514027.924612] agent systemd[1]: Reloading... agent # [514028.245648] agent systemd[1]: Reloading finished in 320 ms. agent # [514028.310184] agent systemd[1]: Starting app slow2... agent: (finished: waiting for failure: systemctl is-active flakelet-relay-job-app.service, in 6.47 seconds) (finished: subtest: detach returns once the job is accepted, in 7.81 seconds) subtest: unknown host is final, missing agent is retried, agents listing client: must fail: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem deploy nope/app 2>&1 client: (finished: must fail: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem deploy nope/app 2>&1, in 0.02 seconds) agent: must succeed: systemctl stop flakelet-agent agent: (finished: must succeed: systemctl stop flakelet-agent, in 0.07 seconds) relay: waiting for success: curl -sf http://127.0.0.1:7400/metrics | grep -q 'agent_up{host="agent"} 0' relay: (finished: waiting for success: curl -sf http://127.0.0.1:7400/metrics | grep -q 'agent_up{host="agent"} 0', in 0.02 seconds) client: must succeed: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem deploy agent/other > /tmp/retry.out 2>&1 & client: (finished: must succeed: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem deploy agent/other > /tmp/retry.out 2>&1 &, in 0.01 seconds) client: waiting for success: grep -q 'agent_unavailable.*retrying' /tmp/retry.out agent # [514034.376097] agent systemd[1]: Started app slow2. agent # [514034.381243] agent flakelet[1678]: app: updated to generation 4 agent # [514034.381911] agent systemd[1]: flakelet-relay-job-app.service: Deactivated successfully. agent # [514034.418999] agent systemd[1]: Stopping flakelet-agent... agent # [514034.449200] agent systemd[1]: flakelet-agent.service: Deactivated successfully. agent # [514034.449662] agent systemd[1]: Stopped flakelet-agent. agent # [514034.450115] agent systemd[1]: flakelet-agent.service: Consumed 1.788s CPU time over 27.980s wall clock time. relay # [514034.418858] relay flakelet-relay[400]: {"timestamp":"2026-09-17T11:59:02.706296Z","level":"INFO","message":"agent disconnected: IO error: peer closed connection without sending TLS close_notify: https://docs.rs/rustls/latest/rustls/manual/_03_howto/index.html#unexpected-eof","host":"agent","target":"flakelet_relay::relay::agent_conn"} client: (finished: waiting for success: grep -q 'agent_unavailable.*retrying' /tmp/retry.out, in 1.02 seconds) agent: must succeed: systemctl start flakelet-agent agent: (finished: must succeed: systemctl start flakelet-agent, in 0.10 seconds) ??? Warning (UserWarning): wait_until_succeeds(): passing a bare int/float as a duration is deprecated. Use datetime.timedelta instead. File "/nix/store/463h52kqv09byfnk9gw61bsa1h1qb8bh-nixos-test-driver-1.1/lib/python3.14/site-packages/test_driver/duration.py", line 39 client: waiting for success: grep -q 'agent/other: unchanged' /tmp/retry.out agent # [514035.539135] agent systemd[1]: Starting flakelet-agent... agent # [514035.600714] agent flakelet-agent[1839]: {"timestamp":"2026-09-17T11:59:03.888144Z","level":"INFO","message":"reattaching to running unit","flakelet":"app","ids":"[\"3863177a439ba5d75c0460e8e749614c\"]","target":"flakelet_relay::agent::jobs"} agent # [514035.624187] agent systemd[1]: Started flakelet-agent. agent # [514035.633092] agent flakelet-agent[1839]: {"timestamp":"2026-09-17T11:59:03.920498Z","level":"INFO","message":"connected","relay":"https://relay:7443","relay_name":"relay","relay_version":"0.1.0","host":"agent","target":"flakelet_relay::agent::conn"} relay # [514035.633149] relay flakelet-relay[400]: {"timestamp":"2026-09-17T11:59:03.920593Z","level":"INFO","message":"agent connected","host":"agent","version":"0.1.0","flakelets":"[Named { name: \"app\", generation: Some(4), revision: Some(\"prebuilt:app\") }, Named { name: \"flakelet-agent\", generation: Some(1), revision: Some(\"prebuilt:flakelet-agent\") }, Named { name: \"other\", generation: Some(1), revision: Some(\"prebuilt:other\") }]","target":"flakelet_relay::relay::agent_conn"} agent # [514036.147828] agent flakelet-agent[1839]: {"timestamp":"2026-09-17T11:59:04.435276Z","level":"INFO","message":"update finished","flakelet":"app","status":"Updated","generation":"Some(4)","target":"flakelet_relay::agent::jobs"} agent # [514037.525741] agent flakelet-agent[1839]: {"timestamp":"2026-09-17T11:59:05.813167Z","level":"INFO","message":"start","id":"ec0c2636d746f01b8066afa28132c48f","flakelet":"other","rule":"admins","relay":"relay","caller_name":"\"admin@example.org\"","target":"flakelet_relay::agent::conn"} agent # [514037.526522] agent flakelet-agent[1839]: {"timestamp":"2026-09-17T11:59:05.813755Z","level":"INFO","message":"update starting","flakelet":"other","ids":"[\"ec0c2636d746f01b8066afa28132c48f\"]","target":"flakelet_relay::agent::jobs"} agent # [514037.559211] agent systemd[1]: Starting flakelet update other (via relay)... agent # [514037.585010] agent systemd[1]: Started flakelet update other (via relay). agent # [514037.589404] agent flakelet[2252]: other: using prebuilt artifact /nix/store/h4k7d4dqcfa4h9d48c0rm4x08qziqfaf-flakelet-other-v1 agent # [514037.589966] agent flakelet[2252]: other: up to date agent # [514037.591232] agent systemd[1]: flakelet-relay-job-other.service: Deactivated successfully. relay # [514037.525380] relay flakelet-relay[400]: {"timestamp":"2026-09-17T11:59:05.812826Z","level":"INFO","message":"deploy accepted","job":"1a4c54b09e47d54f346043187ba99a1d","caller":"admin@example.org","targets":"[\"agent/other\"]","target":"flakelet_relay::relay::api"} agent # [514038.118476] agent flakelet-agent[1839]: {"timestamp":"2026-09-17T11:59:06.405944Z","level":"INFO","message":"update finished","flakelet":"other","status":"Unchanged","generation":"Some(1)","target":"flakelet_relay::agent::jobs"} relay # [514038.119791] relay flakelet-relay[400]: {"timestamp":"2026-09-17T11:59:06.407228Z","level":"INFO","message":"deploy finished","job":"1a4c54b09e47d54f346043187ba99a1d","ok":true,"target":"flakelet_relay::relay::api"} client: (finished: waiting for success: grep -q 'agent/other: unchanged' /tmp/retry.out, in 3.04 seconds) client: must succeed: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem agents client: (finished: must succeed: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem agents, in 0.02 seconds) client: must succeed: echo '{"groups": []}' | step crypto jwt sign --key /nix/store/4y8k1nvlafnzaadg5frncjjarv7aw3ry-flakelet-relay-test-issuer/jwk.json --iss https://relay:8443 --aud flakelet-relay --sub 'repo:github:example/app:ref:refs/heads/main' --exp $(( $(date +%s) + 300 )) --jti $(head -c8 /dev/urandom | base32) client: (finished: must succeed: echo '{"groups": []}' | step crypto jwt sign --key /nix/store/4y8k1nvlafnzaadg5frncjjarv7aw3ry-flakelet-relay-test-issuer/jwk.json --iss https://relay:8443 --aud flakelet-relay --sub 'repo:github:example/app:ref:refs/heads/main' --exp $(( $(date +%s) + 300 )) --jti $(head -c8 /dev/urandom | base32), in 0.06 seconds) client: must succeed: FLAKELET_RELAY_TOKEN_COMMAND='echo eyJhbGciOiJFZERTQSIsImtpZCI6ImsxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJmbGFrZWxldC1yZWxheSIsImV4cCI6MTc4OTY0NjY0NiwiZ3JvdXBzIjpbXSwiaWF0IjoxNzg5NjQ2MzQ3LCJpc3MiOiJodHRwczovL3JlbGF5Ojg0NDMiLCJqdGkiOiJJVFZYVExXUUFMMlJZPT09IiwibmJmIjoxNzg5NjQ2MzQ3LCJzdWIiOiJyZXBvOmdpdGh1YjpleGFtcGxlL2FwcDpyZWY6cmVmcy9oZWFkcy9tYWluIn0.pwsQZtewO_cYtryFW9Lig3FSMKgFcr844HDRhCgrJOPjxo2cO752nxKxg6F_Vg2pgmY-Y64wZWrhWqFRboLPDQ' push --relay https://relay:7443 agents client: (finished: must succeed: FLAKELET_RELAY_TOKEN_COMMAND='echo eyJhbGciOiJFZERTQSIsImtpZCI6ImsxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJmbGFrZWxldC1yZWxheSIsImV4cCI6MTc4OTY0NjY0NiwiZ3JvdXBzIjpbXSwiaWF0IjoxNzg5NjQ2MzQ3LCJpc3MiOiJodHRwczovL3JlbGF5Ojg0NDMiLCJqdGkiOiJJVFZYVExXUUFMMlJZPT09IiwibmJmIjoxNzg5NjQ2MzQ3LCJzdWIiOiJyZXBvOmdpdGh1YjpleGFtcGxlL2FwcDpyZWY6cmVmcy9oZWFkcy9tYWluIn0.pwsQZtewO_cYtryFW9Lig3FSMKgFcr844HDRhCgrJOPjxo2cO752nxKxg6F_Vg2pgmY-Y64wZWrhWqFRboLPDQ' push --relay https://relay:7443 agents, in 0.02 seconds) (finished: subtest: unknown host is final, missing agent is retried, agents listing, in 4.39 seconds) subtest: out-of-band flakelet update reaches the relay without reconnect agent: must succeed: flakelet status --json app | jq '.[0].generation' agent: (finished: must succeed: flakelet status --json app | jq '.[0].generation', in 0.03 seconds) agent: must succeed: journalctl -u flakelet-agent -o cat | grep -c '"connected"' || true agent: (finished: must succeed: journalctl -u flakelet-agent -o cat | grep -c '"connected"' || true, in 0.01 seconds) agent: must succeed: jq '.services.app.prebuilt="/nix/store/1hhsfvlan5s6l308sr1yx2i20rkz1mdc-flakelet-app-v2"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json agent: (finished: must succeed: jq '.services.app.prebuilt="/nix/store/1hhsfvlan5s6l308sr1yx2i20rkz1mdc-flakelet-app-v2"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json, in 0.01 seconds) agent: must succeed: flakelet update app app: using prebuilt artifact /nix/store/1hhsfvlan5s6l308sr1yx2i20rkz1mdc-flakelet-app-v2 app: activating generation 5 agent # [514038.857141] agent systemd[1]: Stopping app slow2... agent # [514038.857606] agent systemd[1]: app.service: Deactivated successfully. agent # [514038.857872] agent systemd[1]: Stopped app slow2. agent # [514038.877334] agent systemd[1]: Reload requested from client PID 2286 ('systemctl')... agent # [514038.877399] agent systemd[1]: Reloading... agent # [514039.222236] agent systemd[1]: Reloading finished in 344 ms. agent # [514039.282881] agent systemd[1]: Reload requested from client PID 2315 ('systemctl')... agent # [514039.282919] agent systemd[1]: Reloading... agent # [514039.609047] agent systemd[1]: Reloading finished in 325 ms. agent # [514039.697865] agent systemd[1]: Reload requested from client PID 2345 ('systemctl')... agent # [514039.697906] agent systemd[1]: Reloading... agent # [514040.025119] agent systemd[1]: Reloading finished in 326 ms. agent # [514040.088973] agent systemd[1]: Reload requested from client PID 2376 ('systemctl')... agent # [514040.089015] agent systemd[1]: Reloading... agent: (finished: must succeed: flakelet update app, in 1.71 seconds) client: waiting for success: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem agents | grep -q 'app@5' agent # [514040.417375] agent systemd[1]: Reloading finished in 328 ms. agent # [514040.498116] agent systemd[1]: Starting app v2... agent # [514040.530047] agent systemd[1]: Started app v2. agent # [514041.645974] agent flakelet-agent[1839]: {"timestamp":"2026-09-17T11:59:09.933444Z","level":"INFO","message":"local change","flakelet":"app","generation":5,"by":"root (shell)","target":"flakelet_relay::agent::jobs"} client: (finished: waiting for success: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem agents | grep -q 'app@5', in 2.07 seconds) agent: must succeed: journalctl -u flakelet-agent -o cat | grep -c '"connected"' || true agent: (finished: must succeed: journalctl -u flakelet-agent -o cat | grep -c '"connected"' || true, in 0.01 seconds) client: waiting for success: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem jobs | grep local-app-5- client: (finished: waiting for success: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem jobs | grep local-app-5-, in 0.02 seconds) client: must succeed: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem jobs | grep local-app- client: (finished: must succeed: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem jobs | grep local-app-, in 0.02 seconds) agent: must succeed: jq '.services.app.prebuilt="/nix/store/cqj0khcw75fy7djy7y554bcgxi50911r-flakelet-app-v1"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json agent: (finished: must succeed: jq '.services.app.prebuilt="/nix/store/cqj0khcw75fy7djy7y554bcgxi50911r-flakelet-app-v1"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json, in 0.01 seconds) agent: must succeed: flakelet update app app: using prebuilt artifact /nix/store/cqj0khcw75fy7djy7y554bcgxi50911r-flakelet-app-v1 app: activating generation 6 agent # [514042.709318] agent systemd[1]: Stopping app v2... agent # [514042.709732] agent systemd[1]: app.service: Deactivated successfully. agent # [514042.709992] agent systemd[1]: Stopped app v2. agent # [514042.727053] agent systemd[1]: Reload requested from client PID 2422 ('systemctl')... agent # [514042.727119] agent systemd[1]: Reloading... agent # [514043.066217] agent systemd[1]: Reloading finished in 338 ms. agent # [514043.163410] agent systemd[1]: Reload requested from client PID 2451 ('systemctl')... agent # [514043.163448] agent systemd[1]: Reloading... agent # [514043.486083] agent systemd[1]: Reloading finished in 322 ms. agent # [514043.535738] agent systemd[1]: Reload requested from client PID 2479 ('systemctl')... agent # [514043.535774] agent systemd[1]: Reloading... agent # [514043.857376] agent systemd[1]: Reloading finished in 321 ms. agent # [514043.949339] agent systemd[1]: Reload requested from client PID 2511 ('systemctl')... agent # [514043.949374] agent systemd[1]: Reloading... agent: (finished: must succeed: flakelet update app, in 1.71 seconds) client: waiting for success: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem agents | grep -q 'app@6' agent # [514044.269616] agent systemd[1]: Reloading finished in 319 ms. agent # [514044.351432] agent systemd[1]: Starting app v1... agent # [514044.383037] agent systemd[1]: Started app v1. agent # [514045.649359] agent flakelet-agent[1839]: {"timestamp":"2026-09-17T11:59:13.936829Z","level":"INFO","message":"local change","flakelet":"app","generation":6,"by":"root (shell)","target":"flakelet_relay::agent::jobs"} client: (finished: waiting for success: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem agents | grep -q 'app@6', in 2.07 seconds) (finished: subtest: out-of-band flakelet update reaches the relay without reconnect, in 7.68 seconds) subtest: certificate not listed under agents cannot connect as agent client: must succeed: curl -s -o /dev/null -w '%{http_code}' --cacert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/ca.pem --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/mallory/cert.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/mallory/key.pem -H 'Upgrade: websocket' -H 'Connection: Upgrade' -H 'Sec-WebSocket-Version: 13' -H 'Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==' https://relay:7443/v1/agent client: (finished: must succeed: curl -s -o /dev/null -w '%{http_code}' --cacert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/ca.pem --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/mallory/cert.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/mallory/key.pem -H 'Upgrade: websocket' -H 'Connection: Upgrade' -H 'Sec-WebSocket-Version: 13' -H 'Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==' https://relay:7443/v1/agent, in 0.03 seconds) (finished: subtest: certificate not listed under agents cannot connect as agent, in 0.03 seconds) subtest: metrics relay: must succeed: curl -sf http://127.0.0.1:7400/metrics relay: (finished: must succeed: curl -sf http://127.0.0.1:7400/metrics, in 0.01 seconds) (finished: subtest: metrics, in 0.01 seconds) subtest: agent link drop mid-run replays missed lines once agent: must succeed: jq '.services.app.prebuilt="/nix/store/lb9sxpw9lh5imz78256ba6b43w55pkww-flakelet-app-slow"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json agent: (finished: must succeed: jq '.services.app.prebuilt="/nix/store/lb9sxpw9lh5imz78256ba6b43w55pkww-flakelet-app-slow"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json, in 0.01 seconds) agent: must succeed: date +%s agent: (finished: must succeed: date +%s, in 0.01 seconds) client: must succeed: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem deploy agent/app > /tmp/slow.out 2>&1 & client: (finished: must succeed: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem deploy agent/app > /tmp/slow.out 2>&1 &, in 0.01 seconds) agent: waiting for success: journalctl -u flakelet-relay-job-app.service --since=@1789646354 | grep -q 'app-slow' agent # [514046.543681] agent flakelet-agent[1839]: {"timestamp":"2026-09-17T11:59:14.831123Z","level":"INFO","message":"start","id":"93cb20cd583da2e5dbe9c7835e7910f9","flakelet":"app","rule":"admins","relay":"relay","caller_name":"\"admin@example.org\"","target":"flakelet_relay::agent::conn"} agent # [514046.544307] agent flakelet-agent[1839]: {"timestamp":"2026-09-17T11:59:14.831801Z","level":"INFO","message":"update starting","flakelet":"app","ids":"[\"93cb20cd583da2e5dbe9c7835e7910f9\"]","target":"flakelet_relay::agent::jobs"} agent # [514046.580673] agent systemd[1]: Starting flakelet update app (via relay)... agent # [514046.609046] agent systemd[1]: Started flakelet update app (via relay). agent # [514046.613419] agent flakelet[2561]: app: using prebuilt artifact /nix/store/lb9sxpw9lh5imz78256ba6b43w55pkww-flakelet-app-slow agent # [514046.614073] agent flakelet[2561]: app: activating generation 7 agent # [514046.631988] agent systemd[1]: Stopping app v1... agent # [514046.632520] agent systemd[1]: app.service: Deactivated successfully. agent # [514046.632791] agent systemd[1]: Stopped app v1. agent # [514046.650215] agent systemd[1]: Reload requested from client PID 2568 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [514046.650298] agent systemd[1]: Reloading... relay # [514046.482635] relay flakelet-relay[400]: {"timestamp":"2026-09-17T11:59:14.770058Z","level":"INFO","message":"agent rejected: no agents entry matches","principals":"[\"x509:dns:mallory\"]","target":"flakelet_relay::relay::agent_conn"} relay # [514046.543293] relay flakelet-relay[400]: {"timestamp":"2026-09-17T11:59:14.830754Z","level":"INFO","message":"deploy accepted","job":"d939f04bf9c498a7e8cd1b740fb7d6b0","caller":"admin@example.org","targets":"[\"agent/app\"]","target":"flakelet_relay::relay::api"} agent # [514046.990583] agent systemd[1]: Reloading finished in 339 ms. agent # [514047.081240] agent systemd[1]: Reload requested from client PID 2598 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [514047.081276] agent systemd[1]: Reloading... agent: (finished: waiting for success: journalctl -u flakelet-relay-job-app.service --since=@1789646354 | grep -q 'app-slow', in 1.02 seconds) agent: must succeed: for a in $(getent ahosts relay | cut -d' ' -f1 | sort -u); do ip route add unreachable $a; done agent: (finished: must succeed: for a in $(getent ahosts relay | cut -d' ' -f1 | sort -u); do ip route add unreachable $a; done, in 0.02 seconds) relay: must succeed: for a in $(getent ahosts agent | cut -d' ' -f1 | sort -u); do ss -K dst "[$a]"; done relay: (finished: must succeed: for a in $(getent ahosts agent | cut -d' ' -f1 | sort -u); do ss -K dst "[$a]"; done, in 0.06 seconds) agent: must succeed: ss -K dport = 7443 agent # [514047.402774] agent systemd[1]: Reloading finished in 321 ms. agent # [514047.456598] agent systemd[1]: Reload requested from client PID 2627 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [514047.456635] agent systemd[1]: Reloading... agent: (finished: must succeed: ss -K dport = 7443, in 0.03 seconds) agent: waiting for success: journalctl -u flakelet-relay-job-app.service --since=@1789646354 | grep -q 'updated to generation' agent # [514047.663998] agent flakelet-agent[1839]: {"timestamp":"2026-09-17T11:59:15.951294Z","level":"INFO","message":"disconnected: IO error: Software caused connection abort (os error 103)","relay":"https://relay:7443","target":"flakelet_relay::agent::conn"} agent # [514047.786334] agent systemd[1]: Reloading finished in 329 ms. agent # [514047.867489] agent systemd[1]: Reload requested from client PID 2675 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [514047.867530] agent systemd[1]: Reloading... relay # [514047.631439] relay flakelet-relay[400]: {"timestamp":"2026-09-17T11:59:15.918845Z","level":"INFO","message":"agent disconnected: IO error: Software caused connection abort (os error 103)","host":"agent","target":"flakelet_relay::relay::agent_conn"} agent # [514048.191796] agent systemd[1]: Reloading finished in 323 ms. agent # [514048.249060] agent systemd[1]: Starting app slow... agent # [514048.666455] agent flakelet-agent[1839]: {"timestamp":"2026-09-17T11:59:16.953877Z","level":"WARN","message":"connect failed: connect relay:7443: No route to host (os error 113)","relay":"https://relay:7443","target":"flakelet_relay::agent::conn"} agent # [514050.669217] agent flakelet-agent[1839]: {"timestamp":"2026-09-17T11:59:18.956658Z","level":"WARN","message":"connect failed: connect relay:7443: No route to host (os error 113)","relay":"https://relay:7443","target":"flakelet_relay::agent::conn"} agent # [514054.320199] agent systemd[1]: Started app slow. agent # [514054.327893] agent flakelet[2561]: app: updated to generation 7 agent # [514054.331553] agent systemd[1]: flakelet-relay-job-app.service: Deactivated successfully. agent: (finished: waiting for success: journalctl -u flakelet-relay-job-app.service --since=@1789646354 | grep -q 'updated to generation', in 7.10 seconds) agent: must succeed: for a in $(getent ahosts relay | cut -d' ' -f1 | sort -u); do ip route del unreachable $a; done agent: (finished: must succeed: for a in $(getent ahosts relay | cut -d' ' -f1 | sort -u); do ip route del unreachable $a; done, in 0.02 seconds) client: waiting for success: grep -q 'agent/app: updated' /tmp/slow.out agent # [514054.671147] agent flakelet-agent[1839]: {"timestamp":"2026-09-17T11:59:22.958584Z","level":"WARN","message":"connect failed: connect relay:7443: No route to host (os error 113)","relay":"https://relay:7443","target":"flakelet_relay::agent::conn"} agent # [514055.447956] agent flakelet-agent[1839]: {"timestamp":"2026-09-17T11:59:23.735403Z","level":"INFO","message":"update finished","flakelet":"app","status":"Updated","generation":"Some(7)","target":"flakelet_relay::agent::jobs"} client: (finished: waiting for success: grep -q 'agent/app: updated' /tmp/slow.out, in 8.09 seconds) client: must succeed: cat /tmp/slow.out client: (finished: must succeed: cat /tmp/slow.out, in 0.01 seconds) (finished: subtest: agent link drop mid-run replays missed lines once, in 16.39 seconds) subtest: agent restart mid-run reattaches to the unit agent: must succeed: jq '.services.app.prebuilt="/nix/store/yqdz0mcr1b2ljpw9vli2zvbmm43j4gr2-flakelet-app-slow2"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json agent: (finished: must succeed: jq '.services.app.prebuilt="/nix/store/yqdz0mcr1b2ljpw9vli2zvbmm43j4gr2-flakelet-app-slow2"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json, in 0.01 seconds) client: must succeed: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem deploy agent/app > /tmp/restart.out 2>&1 & client: (finished: must succeed: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem deploy agent/app > /tmp/restart.out 2>&1 &, in 0.01 seconds) agent: waiting for success: journalctl -u flakelet-relay-job-app.service --since -20s | grep -q 'app-slow2' relay # [514062.674498] relay flakelet-relay[400]: {"timestamp":"2026-09-17T11:59:30.961934Z","level":"INFO","message":"agent connected","host":"agent","version":"0.1.0","flakelets":"[Named { name: \"app\", generation: Some(7), revision: Some(\"prebuilt:app\") }, Named { name: \"flakelet-agent\", generation: Some(1), revision: Some(\"prebuilt:flakelet-agent\") }, Named { name: \"other\", generation: Some(1), revision: Some(\"prebuilt:other\") }]","target":"flakelet_relay::relay::agent_conn"} agent # [514062.674404] agent flakelet-agent[1839]: {"timestamp":"2026-09-17T11:59:30.961840Z","level":"INFO","message":"connected","relay":"https://relay:7443","relay_name":"relay","relay_version":"0.1.0","host":"agent","target":"flakelet_relay::agent::conn"} relay # [514062.676249] relay flakelet-relay[400]: {"timestamp":"2026-09-17T11:59:30.963713Z","level":"INFO","message":"deploy finished","job":"d939f04bf9c498a7e8cd1b740fb7d6b0","ok":true,"target":"flakelet_relay::relay::api"} agent # [514062.675658] agent flakelet-agent[1839]: {"timestamp":"2026-09-17T11:59:30.963146Z","level":"INFO","message":"start","id":"93cb20cd583da2e5dbe9c7835e7910f9","flakelet":"app","rule":"admins","relay":"relay","caller_name":"\"admin@example.org\"","target":"flakelet_relay::agent::conn"} relay # [514062.925061] relay flakelet-relay[400]: {"timestamp":"2026-09-17T11:59:31.212389Z","level":"INFO","message":"deploy accepted","job":"69db184941613cf0c2a71635a4403f43","caller":"admin@example.org","targets":"[\"agent/app\"]","target":"flakelet_relay::relay::api"} agent # [514062.925468] agent flakelet-agent[1839]: {"timestamp":"2026-09-17T11:59:31.212778Z","level":"INFO","message":"start","id":"f5a2ba7205a6da65424802cf84c243c8","flakelet":"app","rule":"admins","relay":"relay","caller_name":"\"admin@example.org\"","target":"flakelet_relay::agent::conn"} agent # [514062.925874] agent flakelet-agent[1839]: {"timestamp":"2026-09-17T11:59:31.213342Z","level":"INFO","message":"update starting","flakelet":"app","ids":"[\"f5a2ba7205a6da65424802cf84c243c8\"]","target":"flakelet_relay::agent::jobs"} agent # [514062.953310] agent systemd[1]: Starting flakelet update app (via relay)... agent # [514062.984342] agent systemd[1]: Started flakelet update app (via relay). agent # [514062.989820] agent flakelet[2797]: app: using prebuilt artifact /nix/store/yqdz0mcr1b2ljpw9vli2zvbmm43j4gr2-flakelet-app-slow2 agent # [514062.990504] agent flakelet[2797]: app: activating generation 8 agent # [514063.006582] agent systemd[1]: Stopping app slow... agent # [514063.008551] agent systemd[1]: app.service: Deactivated successfully. agent # [514063.008946] agent systemd[1]: Stopped app slow. agent # [514063.024405] agent systemd[1]: Reload requested from client PID 2804 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [514063.024469] agent systemd[1]: Reloading... agent # [514063.361717] agent systemd[1]: Reloading finished in 336 ms. agent # [514063.428105] agent systemd[1]: Reload requested from client PID 2834 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [514063.428140] agent systemd[1]: Reloading... agent # [514063.761595] agent systemd[1]: Reloading finished in 333 ms. agent: (finished: waiting for success: journalctl -u flakelet-relay-job-app.service --since -20s | grep -q 'app-slow2', in 1.02 seconds) agent: must succeed: systemctl restart flakelet-agent agent # [514063.850686] agent systemd[1]: Reload requested from client PID 2866 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [514063.850724] agent systemd[1]: Reloading... agent # [514064.176596] agent systemd[1]: Reloading finished in 325 ms. agent # [514064.248887] agent systemd[1]: Stopping flakelet-agent... agent # [514064.256952] agent systemd[1]: Reload requested from client PID 2902 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [514064.256987] agent systemd[1]: Reloading... relay # [514064.249101] relay flakelet-relay[400]: {"timestamp":"2026-09-17T11:59:32.536558Z","level":"INFO","message":"agent disconnected: IO error: peer closed connection without sending TLS close_notify: https://docs.rs/rustls/latest/rustls/manual/_03_howto/index.html#unexpected-eof","host":"agent","target":"flakelet_relay::relay::agent_conn"} agent: (finished: must succeed: systemctl restart flakelet-agent, in 0.82 seconds) agent: must succeed: test -n "$(ls /var/lib/flakelet-agent/jobs)" agent: (finished: must succeed: test -n "$(ls /var/lib/flakelet-agent/jobs)", in 0.01 seconds) client: waiting for success: grep -q 'agent/app: updated' /tmp/restart.out agent # [514064.591831] agent systemd[1]: Reloading finished in 334 ms. agent # [514064.673601] agent systemd[1]: flakelet-agent.service: Deactivated successfully. agent # [514064.673791] agent systemd[1]: Stopped flakelet-agent. agent # [514064.673804] agent systemd[1]: flakelet-agent.service: Consumed 1.650s CPU time over 29.134s wall clock time. agent # [514064.692358] agent systemd[1]: Starting app slow2... agent # [514064.693162] agent systemd[1]: Starting flakelet-agent... agent # [514064.743279] agent flakelet-agent[2934]: {"timestamp":"2026-09-17T11:59:33.030752Z","level":"INFO","message":"reattaching to running unit","flakelet":"app","ids":"[\"f5a2ba7205a6da65424802cf84c243c8\"]","target":"flakelet_relay::agent::jobs"} agent # [514064.759641] agent systemd[1]: Started flakelet-agent. agent # [514064.768263] agent flakelet-agent[2934]: {"timestamp":"2026-09-17T11:59:33.055750Z","level":"INFO","message":"connected","relay":"https://relay:7443","relay_name":"relay","relay_version":"0.1.0","host":"agent","target":"flakelet_relay::agent::conn"} agent # [514064.769491] agent flakelet-agent[2934]: {"timestamp":"2026-09-17T11:59:33.056995Z","level":"INFO","message":"start","id":"f5a2ba7205a6da65424802cf84c243c8","flakelet":"app","rule":"admins","relay":"relay","caller_name":"\"admin@example.org\"","target":"flakelet_relay::agent::conn"} relay # [514064.768415] relay flakelet-relay[400]: {"timestamp":"2026-09-17T11:59:33.055840Z","level":"INFO","message":"agent connected","host":"agent","version":"0.1.0","flakelets":"[Named { name: \"app\", generation: Some(7), revision: Some(\"prebuilt:app\") }, Named { name: \"flakelet-agent\", generation: Some(1), revision: Some(\"prebuilt:flakelet-agent\") }, Named { name: \"other\", generation: Some(1), revision: Some(\"prebuilt:other\") }]","target":"flakelet_relay::relay::agent_conn"} agent # [514070.757202] agent systemd[1]: Started app slow2. agent # [514070.766714] agent flakelet[2797]: app: updated to generation 8 agent # [514070.771239] agent systemd[1]: flakelet-relay-job-app.service: Deactivated successfully. relay # [514071.326394] relay flakelet-relay[400]: {"timestamp":"2026-09-17T11:59:39.613835Z","level":"INFO","message":"deploy finished","job":"69db184941613cf0c2a71635a4403f43","ok":true,"target":"flakelet_relay::relay::api"} agent # [514071.325118] agent flakelet-agent[2934]: {"timestamp":"2026-09-17T11:59:39.612500Z","level":"INFO","message":"update finished","flakelet":"app","status":"Updated","generation":"Some(8)","target":"flakelet_relay::agent::jobs"} client: (finished: waiting for success: grep -q 'agent/app: updated' /tmp/restart.out, in 7.08 seconds) client: must succeed: cat /tmp/restart.out client: (finished: must succeed: cat /tmp/restart.out, in 0.01 seconds) agent: must succeed: systemctl is-active app.service agent: (finished: must succeed: systemctl is-active app.service, in 0.01 seconds) (finished: subtest: agent restart mid-run reattaches to the unit, in 8.98 seconds) subtest: relay restart mid-run, push resumes via /v1/jobs agent: must succeed: jq '.services.app.prebuilt="/nix/store/lb9sxpw9lh5imz78256ba6b43w55pkww-flakelet-app-slow"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json agent: (finished: must succeed: jq '.services.app.prebuilt="/nix/store/lb9sxpw9lh5imz78256ba6b43w55pkww-flakelet-app-slow"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json, in 0.01 seconds) agent: must succeed: date +%s agent: (finished: must succeed: date +%s, in 0.01 seconds) client: must succeed: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem deploy agent/app > /tmp/resume.out 2>&1 & client: (finished: must succeed: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem deploy agent/app > /tmp/resume.out 2>&1 &, in 0.01 seconds) agent: waiting for success: journalctl -u flakelet-relay-job-app.service --since=@1789646380 | grep -q 'app-slow' agent # [514071.913638] agent flakelet-agent[2934]: {"timestamp":"2026-09-17T11:59:40.201074Z","level":"INFO","message":"start","id":"a6bfbacb1642439cdbbbdec31ff82387","flakelet":"app","rule":"admins","relay":"relay","caller_name":"\"admin@example.org\"","target":"flakelet_relay::agent::conn"} agent # [514071.914219] agent flakelet-agent[2934]: {"timestamp":"2026-09-17T11:59:40.201710Z","level":"INFO","message":"update starting","flakelet":"app","ids":"[\"a6bfbacb1642439cdbbbdec31ff82387\"]","target":"flakelet_relay::agent::jobs"} agent # [514071.947901] agent systemd[1]: Starting flakelet update app (via relay)... agent # [514071.985181] agent systemd[1]: Started flakelet update app (via relay). agent # [514071.988753] agent flakelet[3379]: app: using prebuilt artifact /nix/store/lb9sxpw9lh5imz78256ba6b43w55pkww-flakelet-app-slow agent # [514071.989362] agent flakelet[3379]: app: activating generation 9 agent # [514072.007285] agent systemd[1]: Stopping app slow2... agent # [514072.007745] agent systemd[1]: app.service: Deactivated successfully. agent # [514072.008019] agent systemd[1]: Stopped app slow2. agent # [514072.024441] agent systemd[1]: Reload requested from client PID 3386 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [514072.024508] agent systemd[1]: Reloading... relay # [514071.913249] relay flakelet-relay[400]: {"timestamp":"2026-09-17T11:59:40.200692Z","level":"INFO","message":"deploy accepted","job":"c15795180f7e911f92a7c474e1a66cb4","caller":"admin@example.org","targets":"[\"agent/app\"]","target":"flakelet_relay::relay::api"} agent # [514072.358726] agent systemd[1]: Reloading finished in 333 ms. agent # [514072.423582] agent systemd[1]: Reload requested from client PID 3416 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [514072.423621] agent systemd[1]: Reloading... agent # [514072.749644] agent systemd[1]: Reloading finished in 325 ms. agent: (finished: waiting for success: journalctl -u flakelet-relay-job-app.service --since=@1789646380 | grep -q 'app-slow', in 1.03 seconds) relay: must succeed: systemctl restart flakelet-relay agent # [514072.847208] agent systemd[1]: Reload requested from client PID 3447 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [514072.847247] agent systemd[1]: Reloading... agent # [514072.939930] agent flakelet-agent[2934]: {"timestamp":"2026-09-17T11:59:41.227396Z","level":"INFO","message":"disconnected: IO error: peer closed connection without sending TLS close_notify: https://docs.rs/rustls/latest/rustls/manual/_03_howto/index.html#unexpected-eof","relay":"https://relay:7443","target":"flakelet_relay::agent::conn"} relay # [514072.939513] relay systemd[1]: Stopping flakelet-relay... relay # [514072.976025] relay systemd[1]: flakelet-relay.service: Deactivated successfully. relay # [514072.976317] relay systemd[1]: Stopped flakelet-relay. relay # [514072.976346] relay systemd[1]: flakelet-relay.service: Consumed 2.532s CPU time over 1min 8.300s wall clock time. relay # [514072.993490] relay systemd[1]: Starting flakelet-relay... agent # [514073.175605] agent systemd[1]: Reloading finished in 327 ms. agent # [514073.267948] agent systemd[1]: Reload requested from client PID 3483 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [514073.267987] agent systemd[1]: Reloading... agent # [514073.596461] agent systemd[1]: Reloading finished in 328 ms. agent # [514073.680988] agent systemd[1]: Starting app slow... agent # [514073.943097] agent flakelet-agent[2934]: {"timestamp":"2026-09-17T11:59:42.230334Z","level":"WARN","message":"connect failed: connect relay:7443: Connection refused (os error 111)","relay":"https://relay:7443","target":"flakelet_relay::agent::conn"} relay: (finished: must succeed: systemctl restart flakelet-relay, in 1.50 seconds) client: waiting for success: grep -q '» ok' /tmp/resume.out relay # [514074.422122] relay flakelet-relay[932]: {"timestamp":"2026-09-17T11:59:42.709538Z","level":"INFO","message":"listening on 127.0.0.1:7400","target":"flakelet_relay::relay::server"} relay # [514074.422837] relay flakelet-relay[932]: {"timestamp":"2026-09-17T11:59:42.710030Z","level":"INFO","message":"listening on [::]:7443","target":"flakelet_relay::relay::server"} relay # [514074.422601] relay systemd[1]: Started flakelet-relay. agent # [514075.955204] agent flakelet-agent[2934]: {"timestamp":"2026-09-17T11:59:44.242619Z","level":"INFO","message":"connected","relay":"https://relay:7443","relay_name":"relay","relay_version":"0.1.0","host":"agent","target":"flakelet_relay::agent::conn"} relay # [514075.955372] relay flakelet-relay[932]: {"timestamp":"2026-09-17T11:59:44.242811Z","level":"INFO","message":"agent connected","host":"agent","version":"0.1.0","flakelets":"[Named { name: \"app\", generation: Some(8), revision: Some(\"prebuilt:app\") }, Named { name: \"flakelet-agent\", generation: Some(1), revision: Some(\"prebuilt:flakelet-agent\") }, Named { name: \"other\", generation: Some(1), revision: Some(\"prebuilt:other\") }]","target":"flakelet_relay::relay::agent_conn"} agent # [514079.750111] agent systemd[1]: Started app slow. agent # [514079.758742] agent flakelet[3379]: app: updated to generation 9 agent # [514079.762481] agent systemd[1]: flakelet-relay-job-app.service: Deactivated successfully. relay # [514079.958537] relay flakelet-relay[932]: {"timestamp":"2026-09-17T11:59:48.245935Z","level":"INFO","message":"job reattached","job":"c15795180f7e911f92a7c474e1a66cb4","targets":"[\"agent/app\"]","target":"flakelet_relay::relay::api"} agent # [514080.834106] agent flakelet-agent[2934]: {"timestamp":"2026-09-17T11:59:49.121539Z","level":"INFO","message":"update finished","flakelet":"app","status":"Updated","generation":"Some(9)","target":"flakelet_relay::agent::jobs"} client: (finished: waiting for success: grep -q '» ok' /tmp/resume.out, in 7.08 seconds) client: must succeed: cat /tmp/resume.out client: (finished: must succeed: cat /tmp/resume.out, in 0.01 seconds) (finished: subtest: relay restart mid-run, push resumes via /v1/jobs, in 9.64 seconds) subtest: job history comes from the agents and survives a relay restart client: must succeed: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem deploy --id hist1 agent/other 2>&1 agent # [514081.536670] agent flakelet-agent[2934]: {"timestamp":"2026-09-17T11:59:49.824081Z","level":"INFO","message":"start","id":"5be20c693e5c0b60e3f85779794dfb86","flakelet":"other","rule":"admins","relay":"relay","caller_name":"\"admin@example.org\"","target":"flakelet_relay::agent::conn"} agent # [514081.537303] agent flakelet-agent[2934]: {"timestamp":"2026-09-17T11:59:49.824793Z","level":"INFO","message":"update starting","flakelet":"other","ids":"[\"5be20c693e5c0b60e3f85779794dfb86\"]","target":"flakelet_relay::agent::jobs"} agent # [514081.567111] agent systemd[1]: Starting flakelet update other (via relay)... agent # [514081.593049] agent systemd[1]: Started flakelet update other (via relay). agent # [514081.597155] agent flakelet[3557]: other: using prebuilt artifact /nix/store/h4k7d4dqcfa4h9d48c0rm4x08qziqfaf-flakelet-other-v1 agent # [514081.597704] agent flakelet[3557]: other: up to date agent # [514081.598224] agent systemd[1]: flakelet-relay-job-other.service: Deactivated successfully. relay # [514081.536193] relay flakelet-relay[932]: {"timestamp":"2026-09-17T11:59:49.823603Z","level":"INFO","message":"deploy accepted","job":"188b06c4636b65849b3ba8daac22dc7c","caller":"admin@example.org","targets":"[\"agent/other\"]","target":"flakelet_relay::relay::api"} client: (finished: must succeed: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem deploy --id hist1 agent/other 2>&1, in 0.62 seconds) relay: must succeed: systemctl restart flakelet-relay agent # [514082.136446] agent flakelet-agent[2934]: {"timestamp":"2026-09-17T11:59:50.423874Z","level":"INFO","message":"update finished","flakelet":"other","status":"Unchanged","generation":"Some(1)","target":"flakelet_relay::agent::jobs"} agent # [514082.156397] agent flakelet-agent[2934]: {"timestamp":"2026-09-17T11:59:50.443874Z","level":"INFO","message":"disconnected: IO error: peer closed connection without sending TLS close_notify: https://docs.rs/rustls/latest/rustls/manual/_03_howto/index.html#unexpected-eof","relay":"https://relay:7443","target":"flakelet_relay::agent::conn"} relay # [514082.137840] relay flakelet-relay[932]: {"timestamp":"2026-09-17T11:59:50.425248Z","level":"INFO","message":"deploy finished","job":"188b06c4636b65849b3ba8daac22dc7c","ok":true,"target":"flakelet_relay::relay::api"} relay # [514082.156350] relay systemd[1]: Stopping flakelet-relay... relay # [514082.192236] relay systemd[1]: flakelet-relay.service: Deactivated successfully. relay # [514082.192729] relay systemd[1]: Stopped flakelet-relay. relay # [514082.192777] relay systemd[1]: flakelet-relay.service: Consumed 2.873s CPU time over 9.198s wall clock time. relay # [514082.214563] relay systemd[1]: Starting flakelet-relay... agent # [514083.159099] agent flakelet-agent[2934]: {"timestamp":"2026-09-17T11:59:51.446541Z","level":"WARN","message":"connect failed: connect relay:7443: Connection refused (os error 111)","relay":"https://relay:7443","target":"flakelet_relay::agent::conn"} relay: (finished: must succeed: systemctl restart flakelet-relay, in 1.49 seconds) relay: waiting for success: curl -sf http://127.0.0.1:7400/metrics | grep -q 'agent_up{host="agent"} 1' relay # [514083.632049] relay flakelet-relay[1322]: {"timestamp":"2026-09-17T11:59:51.919495Z","level":"INFO","message":"listening on 127.0.0.1:7400","target":"flakelet_relay::relay::server"} relay # [514083.632794] relay flakelet-relay[1322]: {"timestamp":"2026-09-17T11:59:51.919989Z","level":"INFO","message":"listening on [::]:7443","target":"flakelet_relay::relay::server"} relay # [514083.632645] relay systemd[1]: Started flakelet-relay. agent # [514085.169609] agent flakelet-agent[2934]: {"timestamp":"2026-09-17T11:59:53.457057Z","level":"INFO","message":"connected","relay":"https://relay:7443","relay_name":"relay","relay_version":"0.1.0","host":"agent","target":"flakelet_relay::agent::conn"} relay # [514085.169760] relay flakelet-relay[1322]: {"timestamp":"2026-09-17T11:59:53.457204Z","level":"INFO","message":"agent connected","host":"agent","version":"0.1.0","flakelets":"[Named { name: \"app\", generation: Some(9), revision: Some(\"prebuilt:app\") }, Named { name: \"flakelet-agent\", generation: Some(1), revision: Some(\"prebuilt:flakelet-agent\") }, Named { name: \"other\", generation: Some(1), revision: Some(\"prebuilt:other\") }]","target":"flakelet_relay::relay::agent_conn"} relay: (finished: waiting for success: curl -sf http://127.0.0.1:7400/metrics | grep -q 'agent_up{host="agent"} 1', in 2.06 seconds) client: must succeed: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem jobs client: (finished: must succeed: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem jobs, in 0.02 seconds) client: must succeed: echo '{"groups": [], "repository": "example/app"}' | step crypto jwt sign --key /nix/store/4y8k1nvlafnzaadg5frncjjarv7aw3ry-flakelet-relay-test-issuer/jwk.json --iss https://relay:8443 --aud flakelet-relay --sub 'repo:github:example/app:ref:refs/heads/main' --exp $(( $(date +%s) + 300 )) --jti $(head -c8 /dev/urandom | base32) client: (finished: must succeed: echo '{"groups": [], "repository": "example/app"}' | step crypto jwt sign --key /nix/store/4y8k1nvlafnzaadg5frncjjarv7aw3ry-flakelet-relay-test-issuer/jwk.json --iss https://relay:8443 --aud flakelet-relay --sub 'repo:github:example/app:ref:refs/heads/main' --exp $(( $(date +%s) + 300 )) --jti $(head -c8 /dev/urandom | base32), in 0.06 seconds) client: must succeed: FLAKELET_RELAY_TOKEN_COMMAND='echo eyJhbGciOiJFZERTQSIsImtpZCI6ImsxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJmbGFrZWxldC1yZWxheSIsImV4cCI6MTc4OTY0NjY5NCwiZ3JvdXBzIjpbXSwiaWF0IjoxNzg5NjQ2Mzk0LCJpc3MiOiJodHRwczovL3JlbGF5Ojg0NDMiLCJqdGkiOiJaWURFVUJCSDVLRElBPT09IiwibmJmIjoxNzg5NjQ2Mzk0LCJyZXBvc2l0b3J5IjoiZXhhbXBsZS9hcHAiLCJzdWIiOiJyZXBvOmdpdGh1YjpleGFtcGxlL2FwcDpyZWY6cmVmcy9oZWFkcy9tYWluIn0.uvX139lrq9hPVPcnl2zHd14cMJUruRDKJ-NS2cI9xWfxrsCeVzO8imrZOmPPwL4I1yLMRLCFc-CPNgScPhE2Bw' push --relay https://relay:7443 deploy --id hist2 agent/app » job 6a239dbf87cfec796e49404c351d787f via relay 0.1.0 (agents: agent 0.1.0) » wave 0 agent/app | Starting flakelet update app (via relay)... agent/app | Started flakelet update app (via relay). agent/app | app: using prebuilt artifact /nix/store/lb9sxpw9lh5imz78256ba6b43w55pkww-flakelet-app-slow agent # [514085.788774] agent flakelet-agent[2934]: {"timestamp":"2026-09-17T11:59:54.076188Z","level":"INFO","message":"start","id":"dce6faec8455f85321e72ee634888126","flakelet":"app","rule":"ci","relay":"relay","caller_name":"\"example/app\"","target":"flakelet_relay::agent::conn"} agent # [514085.789314] agent flakelet-agent[2934]: {"timestamp":"2026-09-17T11:59:54.076808Z","level":"INFO","message":"update starting","flakelet":"app","ids":"[\"dce6faec8455f85321e72ee634888126\"]","target":"flakelet_relay::agent::jobs"} agent # [514085.822827] agent systemd[1]: Starting flakelet update app (via relay)... agent/appagent # [514085.854127] agent systemd[1]: Started flakelet update app (via relay). | agent # [514085.858579] agent flakelet[3585]: app: using prebuilt artifact /nix/store/lb9sxpw9lh5imz78256ba6b43w55pkww-flakelet-app-slow app: up to dateagent # [514085.859182] agent flakelet[3585]: app: up to date agent # [514085.861390] agent systemd[1]: flakelet-relay-job-app.service: Deactivated successfully. agent/app | flakelet-relay-job-app.service: Deactivated successfully. relay # [514085.788327] relay flakelet-relay[1322]: {"timestamp":"2026-09-17T11:59:54.075782Z","level":"INFO","message":"deploy accepted","job":"6a239dbf87cfec796e49404c351d787f","caller":"example/app","targets":"[\"agent/app\"]","target":"flakelet_relay::relay::api"} » agent/app: unchanged (generation 9) » ok client: (finished: must succeed: FLAKELET_RELAY_TOKEN_COMMAND='echo eyJhbGciOiJFZERTQSIsImtpZCI6ImsxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJmbGFrZWxldC1yZWxheSIsImV4cCI6MTc4OTY0NjY5NCwiZ3JvdXBzIjpbXSwiaWF0IjoxNzg5NjQ2Mzk0LCJpc3MiOiJodHRwczovL3JlbGF5Ojg0NDMiLCJqdGkiOiJaWURFVUJCSDVLRElBPT09IiwibmJmIjoxNzg5NjQ2Mzk0LCJyZXBvc2l0b3J5IjoiZXhhbXBsZS9hcHAiLCJzdWIiOiJyZXBvOmdpdGh1YjpleGFtcGxlL2FwcDpyZWY6cmVmcy9oZWFkcy9tYWluIn0.uvX139lrq9hPVPcnl2zHd14cMJUruRDKJ-NS2cI9xWfxrsCeVzO8imrZOmPPwL4I1yLMRLCFc-CPNgScPhE2Bw' push --relay https://relay:7443 deploy --id hist2 agent/app, in 0.63 seconds) client: must succeed: FLAKELET_RELAY_TOKEN_COMMAND='echo eyJhbGciOiJFZERTQSIsImtpZCI6ImsxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJmbGFrZWxldC1yZWxheSIsImV4cCI6MTc4OTY0NjY5NCwiZ3JvdXBzIjpbXSwiaWF0IjoxNzg5NjQ2Mzk0LCJpc3MiOiJodHRwczovL3JlbGF5Ojg0NDMiLCJqdGkiOiJaWURFVUJCSDVLRElBPT09IiwibmJmIjoxNzg5NjQ2Mzk0LCJyZXBvc2l0b3J5IjoiZXhhbXBsZS9hcHAiLCJzdWIiOiJyZXBvOmdpdGh1YjpleGFtcGxlL2FwcDpyZWY6cmVmcy9oZWFkcy9tYWluIn0.uvX139lrq9hPVPcnl2zHd14cMJUruRDKJ-NS2cI9xWfxrsCeVzO8imrZOmPPwL4I1yLMRLCFc-CPNgScPhE2Bw' push --relay https://relay:7443 jobs client: (finished: must succeed: FLAKELET_RELAY_TOKEN_COMMAND='echo eyJhbGciOiJFZERTQSIsImtpZCI6ImsxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJmbGFrZWxldC1yZWxheSIsImV4cCI6MTc4OTY0NjY5NCwiZ3JvdXBzIjpbXSwiaWF0IjoxNzg5NjQ2Mzk0LCJpc3MiOiJodHRwczovL3JlbGF5Ojg0NDMiLCJqdGkiOiJaWURFVUJCSDVLRElBPT09IiwibmJmIjoxNzg5NjQ2Mzk0LCJyZXBvc2l0b3J5IjoiZXhhbXBsZS9hcHAiLCJzdWIiOiJyZXBvOmdpdGh1YjpleGFtcGxlL2FwcDpyZWY6cmVmcy9oZWFkcy9tYWluIn0.uvX139lrq9hPVPcnl2zHd14cMJUruRDKJ-NS2cI9xWfxrsCeVzO8imrZOmPPwL4I1yLMRLCFc-CPNgScPhE2Bw' push --relay https://relay:7443 jobs, in 0.02 seconds) (finished: subtest: job history comes from the agents and survives a relay restart, in 4.91 seconds) subtest: dashboard login via authorization code flow client: must succeed: curl -sS --cacert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -o /dev/null -w '%{http_code} %{redirect_url}' https://relay:7443/ui/ | grep -q '303 https://relay:7443/ui/login' client: (finished: must succeed: curl -sS --cacert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -o /dev/null -w '%{http_code} %{redirect_url}' https://relay:7443/ui/ | grep -q '303 https://relay:7443/ui/login', in 0.02 seconds) client: must succeed: curl -sS --cacert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -L https://relay:7443/ui/login client: (finished: must succeed: curl -sS --cacert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -L https://relay:7443/ui/login, in 0.08 seconds) client: must succeed: curl -sS --cacert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar https://relay:7443/ui/jobs client: (finished: must succeed: curl -sS --cacert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar https://relay:7443/ui/jobs, in 0.02 seconds) !!! Test "dashboard login via authorization code flow" failed with error: "" cleanup kill NspawnMachine (pid 52) kill NspawnMachine (pid 54) Container agent terminated by signal KILL. kill NspawnMachine (pid 53) Container client terminated by signal KILL. (finished: cleanup, in 0.39 seconds) additionally exposed symbols: agent, client, relay, vlan1, start_all, test_script, machines, machines_qemu, machines_nspawn, vlans, driver, log, os, create_machine, subtest, run_tests, join_all, retry, serial_stdout_off, serial_stdout_on, polling_condition, BaseMachine, QemuMachine, NspawnMachine, t, debug, dump_machine_ssh » job 718651f3253803c42a213aff77f3b07c via relay 0.1.0 (agents: agent 0.1.0) » wave 0 agent/app | Starting flakelet update app (via relay)... agent/app | Started flakelet update app (via relay). agent/app | app: using prebuilt artifact /nix/store/1hhsfvlan5s6l308sr1yx2i20rkz1mdc-flakelet-app-v2 agent/app | app: activating generation 2 agent/app | app: updated to generation 2 agent/app | flakelet-relay-job-app.service: Deactivated successfully. » agent/app: updated (generation 2) » ok » job 7785a7578bc59044d2221ef3e1e57ec4 via relay 0.1.0 (agents: agent 0.1.0) » wave 0 agent/app | Starting flakelet update app (via relay)... agent/app | Started flakelet update app (via relay). agent/app | app: using prebuilt artifact /nix/store/qy8h9kkvfrxq0bc47vwjy6lfnk9i95ga-flakelet-app-broken agent/app | app: activating generation 4 agent/app | app: deploy failed, rolled back: systemctl start app.service failed: agent/app | failed units: app.service. See journalctl -u app.service agent/app | flakelet-relay-job-app.service: Main process exited, code=exited, status=1/FAILURE agent/app | flakelet-relay-job-app.service: Failed with result 'exit-code'. » agent/app: rolled-back (generation 3) agent/app │ Starting app v1... agent/app │ Started app v1. agent/app │ Stopping app v1... agent/app │ app.service: Deactivated successfully. agent/app │ Stopped app v1. agent/app │ Starting app v2... agent/app │ Started app v2. agent/app │ Stopping app v2... agent/app │ app.service: Deactivated successfully. agent/app │ Stopped app v2. agent/app │ Starting app v1... agent/app │ Started app v1. agent/app │ Stopping app v1... agent/app │ app.service: Deactivated successfully. agent/app │ Stopped app v1. agent/app │ Starting app broken... agent/app │ app.service: Unable to locate executable '/nonexistent': No such file or directory agent/app │ app.service: Failed at step EXEC spawning /nonexistent: No such file or directory agent/app │ app.service: Main process exited, code=exited, status=203/EXEC agent/app │ app.service: Failed with result 'exit-code'. agent/app │ Failed to start app broken. agent/app │ Starting app v1... agent/app │ Started app v1. » skipped: agent/other » failed » job d939f04bf9c498a7e8cd1b740fb7d6b0 via relay 0.1.0 (agents: agent 0.1.0) » wave 0 agent/app | Starting flakelet update app (via relay)... agent/app | Started flakelet update app (via relay). agent/app | app: using prebuilt artifact /nix/store/lb9sxpw9lh5imz78256ba6b43w55pkww-flakelet-app-slow agent/app | app: activating generation 7 agent/app | app: updated to generation 7 agent/app | flakelet-relay-job-app.service: Deactivated successfully. » agent/app: updated (generation 7) » ok » job 69db184941613cf0c2a71635a4403f43 via relay 0.1.0 (agents: agent 0.1.0) » wave 0 agent/app | Starting flakelet update app (via relay)... agent/app | Started flakelet update app (via relay). agent/app | app: using prebuilt artifact /nix/store/yqdz0mcr1b2ljpw9vli2zvbmm43j4gr2-flakelet-app-slow2 agent/app | app: activating generation 8 agent/app | app: updated to generation 8 agent/app | flakelet-relay-job-app.service: Deactivated successfully. » agent/app: updated (generation 8) » ok » job c15795180f7e911f92a7c474e1a66cb4 via relay 0.1.0 (agents: agent 0.1.0) » wave 0 agent/app | Starting flakelet update app (via relay)... agent/app | Started flakelet update app (via relay). agent/app | app: using prebuilt artifact /nix/store/lb9sxpw9lh5imz78256ba6b43w55pkww-flakelet-app-slow agent/app | app: activating generation 9 flakelet-push: stream: error reading a body from connection, resuming job 4082ea48118924f93dfd8cdcaaa1f47a flakelet-push: https://relay:7443: connect relay:7443: Connection refused (os error 111), retrying in 1s flakelet-push: https://relay:7443: connect relay:7443: Connection refused (os error 111), retrying in 2s flakelet-push: https://relay:7443: 404 Not Found: unknown_job: no connected agent knows this job, retrying in 4s » job c15795180f7e911f92a7c474e1a66cb4 via relay 0.1.0 (agents: agent 0.1.0) » wave 0 agent/app | app: updated to generation 9 agent/app | flakelet-relay-job-app.service: Deactivated successfully. » agent/app: updated (generation 9) » ok 1789646389 hist1 admin@example.org agent/other:unchanged 1789646380 4082ea48118924f93dfd8cdcaaa1f47a admin@example.org agent/app:updated 1789646371 4d96611fe01bd4a8dcfd73774911c009 admin@example.org agent/app:updated 1789646354 0142853d8680ae8fea861acda70164c0 admin@example.org agent/app:updated 1789646352 local-app-6-1789646352 root (shell) agent/app:updated 1789646348 local-app-5-1789646348 root (shell) agent/app:updated 1789646345 ad526be85bb12cb71e689e5a875a39f2 admin@example.org agent/other:unchanged 1789646334 29c80fb31ff0d1ce20cb6850f112a8c2 admin@example.org agent/app:unchanged 1789646334 ac37e0fc0cd119a7b862314a78f0f68a admin@example.org agent/app:updated 1789646333 492a81812dbdfa87b48d7db6a0dd7948 admin@example.org agent/app:unchanged 1789646332 b7162414b61f02ad5294a01b5d44424a admin@example.org agent/app:unchanged 1789646328 dbdb9f38d458f1fec1896ee8b11b7a8f dev@example.org agent/other:unchanged 1789646328 e356f69259812a4c426b4e7dfe398ac9 admin@example.org agent/app:rolled-back 1789646327 dd0b3b2048365378051bfcf6645f5ffd dev@example.org agent/other:unchanged 1789646323 0b30516e854691c530e2248d692f02a2 someone agent/other:unchanged 1789646320 e87c185910a57f76d635957715a05a16 repo:github:example/app:ref:refs/heads/main agent/app:updated 1789646317 aa289c58965ca17c93d0a2df7767414b admin@example.org agent/app:updated 1789646314 local-app-1-1789646314 host activation agent/app:updated 1789646314 local-flakelet-agent-1-1789646314 host activation agent/flakelet-agent:updated 1789646314 local-other-1-1789646314 host activation agent/other:updated Container relay terminated by signal KILL. Traceback (most recent call last): File "/nix/store/463h52kqv09byfnk9gw61bsa1h1qb8bh-nixos-test-driver-1.1/bin/.nixos-test-driver-wrapped", line 9, in sys.exit(main()) ~~~~^^ File "/nix/store/463h52kqv09byfnk9gw61bsa1h1qb8bh-nixos-test-driver-1.1/lib/python3.14/site-packages/test_driver/__init__.py", line 194, in main driver.run_tests() ~~~~~~~~~~~~~~~~^^ File "/nix/store/463h52kqv09byfnk9gw61bsa1h1qb8bh-nixos-test-driver-1.1/lib/python3.14/site-packages/test_driver/driver.py", line 437, in run_tests self.test_script() ~~~~~~~~~~~~~~~~^^ File "/nix/store/463h52kqv09byfnk9gw61bsa1h1qb8bh-nixos-test-driver-1.1/lib/python3.14/site-packages/test_driver/driver.py", line 395, in test_script exec(self.tests, symbols, None) ~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "", line 208, in AssertionError