tribuchet: building on jamie Machine state will be reset. To keep it, pass --keep-machine-state start all VLans (finished: start all VLans, in 0.00 seconds) Initializing machine ID from random generator. Test will time out and terminate in 3600.0 seconds run the VM test script start all VMs agent: systemd-nspawn running (pid 52) client: systemd-nspawn running (pid 53) agent: Waiting for journal at /build/vm-state-agent/var/log/journal... relay: systemd-nspawn running (pid 55) client: Waiting for journal at /build/vm-state-client/var/log/journal... relay: Waiting for journal at /build/vm-state-relay/var/log/journal... (finished: start all VMs, in 0.00 seconds) relay: waiting for unit flakelet-relay.service nixos-nspawn(agent): TAP vde-tap1 not found; container will be isolated from VDE nixos-nspawn(agent): A common reason for this is that /dev/net is not available in the Nix sandbox. Try adding /dev/net to extra-sandbox-paths. nixos-nspawn(relay): TAP vde-tap1 not found; container will be isolated from VDE nixos-nspawn(relay): A common reason for this is that /dev/net is not available in the Nix sandbox. Try adding /dev/net to extra-sandbox-paths. nixos-nspawn(client): TAP vde-tap1 not found; container will be isolated from VDE nixos-nspawn(client): A common reason for this is that /dev/net is not available in the Nix sandbox. Try adding /dev/net to extra-sandbox-paths. Note: in a future version of systemd-nspawn the default set of permitted socket address families will be restricted to AF_INET, AF_INET6 and AF_UNIX. Use --restrict-address-families= to configure the set of permitted socket address families, or set RestrictAddressFamilies= in a .nspawn file. Note: in a future version of systemd-nspawn the default set of permitted socket address families will be restricted to AF_INET, AF_INET6 and AF_UNIX. Use --restrict-address-families= to configure the set of permitted socket address families, or set RestrictAddressFamilies= in a .nspawn file. Note: in a future version of systemd-nspawn the default set of permitted socket address families will be restricted to AF_INET, AF_INET6 and AF_UNIX. Use --restrict-address-families= to configure the set of permitted socket address families, or set RestrictAddressFamilies= in a .nspawn file. ░ Spawning container client on /build/vm-state-client. ░ Spawning container relay on /build/vm-state-relay. ░ Spawning container agent on /build/vm-state-agent. agent # No journal files were found. agent # No journal boot entry found for the specified boot (+0). client # No journal files were found. client # No journal boot entry found for the specified boot (+0). relay # No journal files were found. relay # No journal boot entry found for the specified boot (+0). agent # [1118709.597855] agent systemd-journald[57]: Journal started agent # [1118709.597897] agent systemd-journald[57]: Runtime Journal (/run/log/journal/cb44bd480c074869afbbe15ede2b8472) is 8M, max 4G, 3.9G free. agent # [1118709.609136] agent systemd[1]: Finished Apply Kernel Variables. agent # [1118709.621792] agent systemd[1]: Finished Create Static Device Nodes in /dev gracefully. agent # [1118709.649672] agent systemd[1]: Starting Flush Journal to Persistent Storage... agent # [1118709.650406] agent systemd[1]: Starting Create Static Device Nodes in /dev... agent # [1118709.658822] agent systemd-journald[57]: Time spent on flushing to /var/log/journal/cb44bd480c074869afbbe15ede2b8472 is 1.522ms for 6 entries. agent # [1118709.658822] agent systemd-journald[57]: System Journal (/var/log/journal/cb44bd480c074869afbbe15ede2b8472) is 512B, max 4G, 3.9G free. agent # [1118709.668760] agent systemd[1]: Finished Create Static Device Nodes in /dev. agent # [1118709.669574] agent systemd[1]: Reached target Preparation for Local File Systems. agent # [1118709.669685] agent systemd[1]: Rule-based Manager for Device Events and Files skipped, unmet condition check ConditionPathIsReadWrite=/sys agent # [1118709.681159] agent systemd[1]: Finished Flush Journal to Persistent Storage. agent # [1118709.746782] agent systemd[1]: Finished Firewall. agent # [1118710.617704] agent systemd[1]: Mounting /run/wrappers... relay # [1118709.587693] relay systemd-journald[57]: Journal started client # [1118709.597498] client systemd-journald[57]: Journal started relay # [1118709.587736] relay systemd-journald[57]: Runtime Journal (/run/log/journal/6f516a1588764f53b614e5af80d9536b) is 8M, max 4G, 3.9G free. client # [1118709.597539] client systemd-journald[57]: Runtime Journal (/run/log/journal/acde72d991bc482c94abd3116c5163d9) is 8M, max 4G, 3.9G free. client # [1118709.608779] client systemd[1]: Finished Apply Kernel Variables. relay # [1118709.598242] relay systemd[1]: Finished Apply Kernel Variables. client # [1118709.621589] client systemd[1]: Finished Create Static Device Nodes in /dev gracefully. relay # [1118709.610512] relay systemd[1]: Finished Create Static Device Nodes in /dev gracefully. client # [1118709.649555] client systemd[1]: Starting Flush Journal to Persistent Storage... relay # [1118709.624975] relay systemd[1]: Starting Flush Journal to Persistent Storage... client # [1118709.650242] client systemd[1]: Starting Create Static Device Nodes in /dev... relay # [1118709.625632] relay systemd[1]: Starting Create Static Device Nodes in /dev... client # [1118709.658107] client systemd-journald[57]: Time spent on flushing to /var/log/journal/acde72d991bc482c94abd3116c5163d9 is 2.533ms for 6 entries. relay # [1118709.654832] relay systemd-journald[57]: Time spent on flushing to /var/log/journal/6f516a1588764f53b614e5af80d9536b is 1.381ms for 6 entries. client # [1118709.658107] client systemd-journald[57]: System Journal (/var/log/journal/acde72d991bc482c94abd3116c5163d9) is 512B, max 4G, 3.9G free. relay # [1118709.654832] relay systemd-journald[57]: System Journal (/var/log/journal/6f516a1588764f53b614e5af80d9536b) is 512B, max 4G, 3.9G free. client # [1118709.668699] client systemd[1]: Finished Create Static Device Nodes in /dev. relay # [1118709.663140] relay systemd[1]: Finished Create Static Device Nodes in /dev. client # [1118709.669477] client systemd[1]: Reached target Preparation for Local File Systems. relay # [1118709.663910] relay systemd[1]: Reached target Preparation for Local File Systems. client # [1118709.669619] client systemd[1]: Rule-based Manager for Device Events and Files skipped, unmet condition check ConditionPathIsReadWrite=/sys relay # [1118709.664027] relay systemd[1]: Rule-based Manager for Device Events and Files skipped, unmet condition check ConditionPathIsReadWrite=/sys client # [1118709.673266] client systemd[1]: Finished Flush Journal to Persistent Storage. relay # [1118709.676188] relay systemd[1]: Finished Flush Journal to Persistent Storage. client # [1118709.742758] client systemd[1]: Finished Firewall. client # [1118710.617881] client systemd[1]: Mounting /run/wrappers... relay # [1118709.758313] relay systemd[1]: Finished Firewall. relay # [1118710.586737] relay systemd[1]: Mounting /run/wrappers... agent # [1118710.639409] agent systemd[1]: Mounted /run/wrappers. agent # [1118710.640261] agent systemd[1]: Reached target Local File Systems. agent # [1118710.641081] agent systemd[1]: Listening on Boot Loader Control Service Socket. agent # [1118710.642254] agent systemd[1]: Starting Create SUID/SGID Wrappers... agent # [1118710.642281] agent systemd[1]: Update Boot Loader Random Seed skipped, unmet condition check ConditionVirtualization=!container agent # [1118710.643072] agent systemd[1]: Starting Save Transient machine-id to Disk... agent # [1118710.643803] agent systemd[1]: Starting Create System Files and Directories... agent # [1118710.661144] agent systemd-tmpfiles[161]: Cannot set file attributes for '/var/empty', value=0x00000010, mask=0x00000010, ignoring: Operation not permitted agent # [1118710.661570] agent systemd-tmpfiles[161]: fchmod() of /var/log/journal failed: Operation not permitted agent # [1118710.661883] agent systemd-tmpfiles[161]: fchmod() of /var/log/journal/cb44bd480c074869afbbe15ede2b8472 failed: Operation not permitted agent # [1118710.662240] agent systemd-tmpfiles[161]: fchmod() of /run/log/journal failed: Operation not permitted client # [1118710.639513] client systemd[1]: Mounted /run/wrappers. client # [1118710.640392] client systemd[1]: Reached target Local File Systems. client # [1118710.641253] client systemd[1]: Listening on Boot Loader Control Service Socket. client # [1118710.642215] client systemd[1]: Starting Create SUID/SGID Wrappers... client # [1118710.642244] client systemd[1]: Update Boot Loader Random Seed skipped, unmet condition check ConditionVirtualization=!container client # [1118710.643192] client systemd[1]: Starting Save Transient machine-id to Disk... client # [1118710.644137] client systemd[1]: Starting Create System Files and Directories... client # [1118710.663199] client systemd-tmpfiles[161]: Cannot set file attributes for '/var/empty', value=0x00000010, mask=0x00000010, ignoring: Operation not permitted client # [1118710.663607] client systemd-tmpfiles[161]: fchmod() of /var/log/journal failed: Operation not permitted client # [1118710.663938] client systemd-tmpfiles[161]: fchmod() of /var/log/journal/acde72d991bc482c94abd3116c5163d9 failed: Operation not permitted client # [1118710.664282] client systemd-tmpfiles[161]: fchmod() of /run/log/journal failed: Operation not permitted relay # [1118710.649992] relay systemd[1]: Mounted /run/wrappers. relay # [1118710.651300] relay systemd[1]: Reached target Local File Systems. relay # [1118710.652618] relay systemd[1]: Listening on Boot Loader Control Service Socket. relay # [1118710.653951] relay systemd[1]: Starting Create SUID/SGID Wrappers... relay # [1118710.654016] relay systemd[1]: Update Boot Loader Random Seed skipped, unmet condition check ConditionVirtualization=!container relay # [1118710.655462] relay systemd[1]: Starting Save Transient machine-id to Disk... relay # [1118710.656772] relay systemd[1]: Starting Create System Files and Directories... relay # [1118710.675046] relay suid-sgid-wrappers-start[176]: chmod: changing permissions of '/run/wrappers/wrappers.9wmImWYqep/chsh': Operation not permitted relay # [1118710.675446] relay systemd-tmpfiles[169]: Cannot set file attributes for '/var/empty', value=0x00000010, mask=0x00000010, ignoring: Operation not permitted relay # [1118710.675850] relay systemd-tmpfiles[169]: fchmod() of /var/log/journal failed: Operation not permitted relay # [1118710.676220] relay systemd-tmpfiles[169]: fchmod() of /var/log/journal/6f516a1588764f53b614e5af80d9536b failed: Operation not permitted relay # [1118710.676549] relay systemd-tmpfiles[169]: fchmod() of /run/log/journal failed: Operation not permitted relay # [1118710.684363] relay systemd[1]: suid-sgid-wrappers.service: Main process exited, code=exited, status=1/FAILURE relay # [1118710.684400] relay systemd[1]: suid-sgid-wrappers.service: Failed with result 'exit-code'. relay # [1118710.684451] relay systemd[1]: Failed to start Create SUID/SGID Wrappers. relay # [1118710.684963] relay systemd[1]: Finished Create System Files and Directories. relay # [1118710.698258] relay systemd[1]: Starting Rebuild Journal Catalog... relay # [1118710.699295] relay systemd[1]: Starting Record System Boot/Shutdown in UTMP... relay # [1118710.712769] relay systemd[1]: Finished Record System Boot/Shutdown in UTMP. relay # [1118710.716810] relay systemd[1]: Finished Rebuild Journal Catalog. relay # [1118710.717879] relay systemd[1]: Starting Update is Completed... relay # [1118710.727427] relay systemd[1]: Finished Update is Completed. relay # [1118710.728032] relay systemd[1]: Reached target System Initialization. relay # [1118710.728137] relay systemd[1]: Discard unused filesystem blocks once a week skipped, unmet condition check ConditionVirtualization=!container relay # [1118710.728175] relay systemd[1]: Started Daily Cleanup of Temporary Directories. relay # [1118710.728195] relay systemd[1]: Reached target Timer Units. relay # [1118710.728318] relay systemd[1]: Listening on D-Bus System Message Bus Socket. relay # [1118710.728481] relay systemd[1]: Listening on Nix Daemon Socket. relay # [1118710.728587] relay systemd[1]: Listening on Virtual Machine and Container Registration Service Socket. relay # [1118710.728600] relay systemd[1]: Reached target Socket Units. relay # [1118710.728628] relay systemd[1]: Reached target Basic System. relay # [1118710.729668] relay systemd[1]: Started device-mock.service. relay # [1118710.730575] relay systemd[1]: Starting Re-link flakelet services at boot... relay # [1118710.731271] relay systemd[1]: Starting Import lastlog data into lastlog2 database... relay # [1118710.732091] relay systemd[1]: Starting Name Service Cache Daemon (nsncd)... relay # [1118710.732789] relay systemd[1]: Starting resolvconf update... relay # [1118710.743712] relay systemd[1]: Finished Re-link flakelet services at boot. relay # [1118710.744571] relay systemd[1]: Starting Reconcile flakelet services with the host configuration... relay # [1118710.752366] relay systemd[1]: Finished Import lastlog data into lastlog2 database. relay # [1118710.755431] relay systemd[1]: Finished Reconcile flakelet services with the host configuration. agent # [1118710.664760] agent systemd[1]: Finished Create System Files and Directories. agent # [1118710.669041] agent suid-sgid-wrappers-start[169]: chmod: changing permissions of '/run/wrappers/wrappers.ufcljbCmnK/chsh': Operation not permitted agent # [1118710.678399] agent systemd[1]: suid-sgid-wrappers.service: Main process exited, code=exited, status=1/FAILURE agent # [1118710.678493] agent systemd[1]: suid-sgid-wrappers.service: Failed with result 'exit-code'. agent # [1118710.678641] agent systemd[1]: Failed to start Create SUID/SGID Wrappers. agent # [1118710.680456] agent systemd[1]: Starting Rebuild Journal Catalog... agent # [1118710.681259] agent systemd[1]: Starting Record System Boot/Shutdown in UTMP... agent # [1118710.693441] agent systemd[1]: Finished Record System Boot/Shutdown in UTMP. agent # [1118710.700195] agent systemd[1]: Finished Rebuild Journal Catalog. agent # [1118710.701329] agent systemd[1]: Starting Update is Completed... agent # [1118710.711322] agent systemd[1]: Finished Update is Completed. agent # [1118710.711422] agent systemd[1]: Reached target System Initialization. agent # [1118710.711497] agent systemd[1]: Discard unused filesystem blocks once a week skipped, unmet condition check ConditionVirtualization=!container agent # [1118710.711524] agent systemd[1]: Started Daily Cleanup of Temporary Directories. agent # [1118710.711539] agent systemd[1]: Reached target Timer Units. agent # [1118710.711641] agent systemd[1]: Listening on D-Bus System Message Bus Socket. agent # [1118710.711871] agent systemd[1]: Listening on Nix Daemon Socket. agent # [1118710.711973] agent systemd[1]: Listening on Virtual Machine and Container Registration Service Socket. agent # [1118710.711988] agent systemd[1]: Reached target Socket Units. agent # [1118710.712026] agent systemd[1]: Reached target Basic System. agent # [1118710.713378] agent systemd[1]: Starting Re-link flakelet services at boot... agent # [1118710.714272] agent systemd[1]: Starting Import lastlog data into lastlog2 database... agent # [1118710.715148] agent systemd[1]: Starting Name Service Cache Daemon (nsncd)... agent # [1118710.715928] agent systemd[1]: Starting resolvconf update... agent # [1118710.726140] agent systemd[1]: Finished Re-link flakelet services at boot. agent # [1118710.727188] agent systemd[1]: Starting Reconcile flakelet services with the host configuration... agent # [1118710.731353] agent systemd[1]: Finished Import lastlog data into lastlog2 database. agent # [1118710.740515] agent systemd[1]: Finished Reconcile flakelet services with the host configuration. agent # [1118710.801515] agent systemd[1]: Finished resolvconf update. agent # [1118710.801654] agent systemd[1]: Reached target Preparation for Network. agent # [1118710.803025] agent systemd[1]: Starting Address configuration of eth1... agent # [1118710.804029] agent systemd[1]: Starting Extra networking commands.... agent # [1118710.824826] agent network-addresses-eth1-start[236]: adding address 192.168.1.1/24... done agent # [1118710.828697] agent network-addresses-eth1-start[236]: adding address 2001:db8:1::1/64... done agent # [1118710.866471] agent systemd[1]: etc-machine\x2did.mount: Deactivated successfully. agent # [1118710.867265] agent systemd[1]: Finished Save Transient machine-id to Disk. agent # [1118710.867461] agent systemd[1]: nscd.service: Deactivated successfully. agent # [1118710.867590] agent systemd[1]: Stopped Name Service Cache Daemon (nsncd). agent # [1118710.869023] agent systemd[1]: Finished Address configuration of eth1. client # [1118710.666851] client systemd[1]: Finished Create System Files and Directories. client # [1118710.669217] client suid-sgid-wrappers-start[169]: chmod: changing permissions of '/run/wrappers/wrappers.GxFABcgksH/chsh': Operation not permitted client # [1118710.680157] client systemd[1]: suid-sgid-wrappers.service: Main process exited, code=exited, status=1/FAILURE client # [1118710.680194] client systemd[1]: suid-sgid-wrappers.service: Failed with result 'exit-code'. client # [1118710.680245] client systemd[1]: Failed to start Create SUID/SGID Wrappers. client # [1118710.681861] client systemd[1]: Starting Rebuild Journal Catalog... client # [1118710.682477] client systemd[1]: Starting Record System Boot/Shutdown in UTMP... client # [1118710.693496] client systemd[1]: Finished Record System Boot/Shutdown in UTMP. client # [1118710.700845] client systemd[1]: Finished Rebuild Journal Catalog. client # [1118710.701816] client systemd[1]: Starting Update is Completed... client # [1118710.713020] client systemd[1]: Finished Update is Completed. client # [1118710.713155] client systemd[1]: Reached target System Initialization. client # [1118710.713218] client systemd[1]: Discard unused filesystem blocks once a week skipped, unmet condition check ConditionVirtualization=!container client # [1118710.713243] client systemd[1]: Started Daily Cleanup of Temporary Directories. client # [1118710.713258] client systemd[1]: Reached target Timer Units. client # [1118710.713355] client systemd[1]: Listening on D-Bus System Message Bus Socket. client # [1118710.713575] client systemd[1]: Listening on Nix Daemon Socket. client # [1118710.713801] client systemd[1]: Listening on Virtual Machine and Container Registration Service Socket. client # [1118710.713817] client systemd[1]: Reached target Socket Units. client # [1118710.713843] client systemd[1]: Reached target Basic System. client # [1118710.714861] client systemd[1]: Starting Import lastlog data into lastlog2 database... client # [1118710.715813] client systemd[1]: Starting Name Service Cache Daemon (nsncd)... client # [1118710.716519] client systemd[1]: Starting resolvconf update... client # [1118710.734379] client systemd[1]: Finished Import lastlog data into lastlog2 database. client # [1118710.881559] client systemd[1]: etc-machine\x2did.mount: Deactivated successfully. client # [1118710.882915] client systemd[1]: Finished Save Transient machine-id to Disk. client # [1118710.909860] client systemd[1]: nscd.service: Deactivated successfully. client # [1118710.923500] client systemd[1]: Stopped Name Service Cache Daemon (nsncd). client # [1118710.926732] client systemd[1]: Finished resolvconf update. client # [1118710.928103] client systemd[1]: Reached target Preparation for Network. client # [1118710.929707] client systemd[1]: Starting Address configuration of eth1... client # [1118710.931283] client systemd[1]: Starting Extra networking commands.... client # [1118710.932955] client systemd[1]: Starting Name Service Cache Daemon (nsncd)... client # [1118710.951932] client network-addresses-eth1-start[236]: adding address 192.168.1.2/24... done client # [1118710.955531] client network-addresses-eth1-start[236]: adding address 2001:db8:1::2/64... done client # [1118710.960680] client systemd[1]: Finished Address configuration of eth1. client # [1118711.023413] client systemd[1]: Finished Extra networking commands.. client # [1118711.023695] client systemd[1]: Reached target Network. client # [1118711.029663] client systemd[1]: nscd.service: Deactivated successfully. client # [1118711.029946] client systemd[1]: Stopped Name Service Cache Daemon (nsncd). client # [1118711.034411] client systemd[1]: Starting Name Service Cache Daemon (nsncd)... agent # [1118710.881757] agent systemd[1]: Starting Name Service Cache Daemon (nsncd)... agent # [1118710.885193] agent systemd[1]: nscd.service: Deactivated successfully. agent # [1118710.885284] agent systemd[1]: Stopped Name Service Cache Daemon (nsncd). agent # [1118710.887407] agent systemd[1]: Starting Name Service Cache Daemon (nsncd)... agent # [1118710.923398] agent systemd[1]: Finished Extra networking commands.. agent # [1118710.924201] agent systemd[1]: Reached target Network. agent # [1118710.925508] agent systemd[1]: Starting Update flakelet service app... agent # [1118710.926466] agent systemd[1]: Starting Update flakelet service flakelet-agent... agent # [1118710.927524] agent systemd[1]: Starting Update flakelet service other... agent # [1118710.937169] agent flakelet[307]: app: using prebuilt artifact /nix/store/cqj0khcw75fy7djy7y554bcgxi50911r-flakelet-app-v1 agent # [1118710.937722] agent flakelet[307]: app: activating generation 1 agent # [1118710.938444] agent flakelet[308]: flakelet-agent: using prebuilt artifact /nix/store/flgya4ggb72hyvnnjcsjw6zbknb4w1s4-flakelet-flakelet-agent agent # [1118710.938976] agent flakelet[308]: flakelet-agent: activating generation 1 agent # [1118710.939299] agent flakelet[309]: other: using prebuilt artifact /nix/store/h4k7d4dqcfa4h9d48c0rm4x08qziqfaf-flakelet-other-v1 agent # [1118710.939832] agent flakelet[309]: other: activating generation 1 agent # [1118710.943348] agent systemd[1]: Reload requested from client PID 310 ('systemctl') (unit flakelet-app.service)... agent # [1118710.943383] agent systemd[1]: Reloading... relay # [1118710.866047] relay systemd[1]: etc-machine\x2did.mount: Deactivated successfully. relay # [1118710.866927] relay systemd[1]: Finished Save Transient machine-id to Disk. relay # [1118710.895302] relay systemd[1]: nscd.service: Deactivated successfully. relay # [1118710.900227] relay systemd[1]: Stopped Name Service Cache Daemon (nsncd). relay # [1118710.923832] relay systemd[1]: Starting Name Service Cache Daemon (nsncd)... relay # [1118710.924083] relay systemd[1]: Finished resolvconf update. relay # [1118710.924830] relay systemd[1]: Reached target Preparation for Network. relay # [1118710.926237] relay systemd[1]: Starting Address configuration of eth1... relay # [1118710.927565] relay systemd[1]: Starting Extra networking commands.... relay # [1118710.947892] relay network-addresses-eth1-start[250]: adding address 192.168.1.3/24... done relay # [1118710.951906] relay network-addresses-eth1-start[250]: adding address 2001:db8:1::3/64... done relay # [1118710.956029] relay systemd[1]: Finished Address configuration of eth1. relay # [1118710.992568] relay systemd[1]: Finished Extra networking commands.. relay # [1118710.992660] relay systemd[1]: Reached target Network. relay # [1118710.993939] relay systemd[1]: Starting Dnsmasq Daemon... relay # [1118710.994832] relay systemd[1]: Starting Nginx Web Server... relay # [1118710.995574] relay systemd[1]: Starting Update flakelet service flakelet-relay... relay # [1118711.033150] relay flakelet[311]: flakelet-relay: using prebuilt artifact /nix/store/55zvswj0bvimz7faqq19rhsjlvqasa00-flakelet-flakelet-relay relay # [1118711.033853] relay flakelet[311]: flakelet-relay: activating generation 1 relay # [1118711.038020] relay systemd[1]: Reload requested from client PID 316 ('systemctl') (unit flakelet-flakelet-relay.service)... relay # [1118711.038060] relay systemd[1]: Reloading... relay # [1118711.297663] relay nsncd[245]: Sep 24 11:56:59.584 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" relay # [1118711.405201] relay systemd[1]: Reloading finished in 366 ms. relay # [1118711.545948] relay systemd[1]: Started Name Service Cache Daemon (nsncd). relay # [1118711.546546] relay systemd[1]: Reached target Host and Network Name Lookups. relay # [1118711.546629] relay systemd[1]: Reached target User and Group Name Lookups. relay # [1118711.547863] relay systemd[1]: Starting User Login Management... client # [1118711.312572] client nsncd[303]: Sep 24 11:56:59.600 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" client # [1118711.312699] client systemd[1]: Started Name Service Cache Daemon (nsncd). client # [1118711.312799] client systemd[1]: Reached target Host and Network Name Lookups. client # [1118711.312902] client systemd[1]: Reached target User and Group Name Lookups. client # [1118711.315138] client systemd[1]: Starting User Login Management... client # [1118711.316436] client systemd[1]: Starting Permit User Sessions... client # [1118711.353576] client systemd[1]: Finished Permit User Sessions. client # [1118711.355648] client systemd[1]: Started Console Getty. client # [1118711.355698] client systemd[1]: Getty on tty1 skipped, unmet condition check ConditionPathExists=/dev/tty0 client # [1118711.355727] client systemd[1]: Reached target Login Prompts. agent # [1118711.318637] agent nsncd[297]: Sep 24 11:56:59.606 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" agent # [1118711.402644] agent systemd[1]: Reloading finished in 458 ms. agent # [1118711.549910] agent systemd[1]: Started Name Service Cache Daemon (nsncd). agent # [1118711.550581] agent systemd[1]: Reload requested from client PID 311 ('systemctl') (unit flakelet-flakelet-agent.service)... agent # [1118711.550633] agent systemd[1]: Reloading... relay # [1118711.548718] relay systemd[1]: Starting Permit User Sessions... relay # [1118711.570872] relay systemd[1]: Reload requested from client PID 357 ('systemctl') (unit flakelet-flakelet-relay.service)... relay # [1118711.570904] relay systemd[1]: Reloading... relay # [1118711.578826] relay dnsmasq-pre-start[360]: dnsmasq: syntax check OK. agent # [1118711.953032] agent systemd[1]: Reloading finished in 401 ms. agent # [1118712.032913] agent systemd[1]: Reload requested from client PID 312 ('systemctl') (unit flakelet-other.service)... agent # [1118712.032961] agent systemd[1]: Reloading... relay # [1118711.931427] relay systemd[1]: Reloading finished in 360 ms. relay # [1118712.005618] relay systemd[1]: Finished Permit User Sessions. relay # [1118712.031663] relay systemd[1]: Started Console Getty. relay # [1118712.031730] relay systemd[1]: Getty on tty1 skipped, unmet condition check ConditionPathExists=/dev/tty0 relay # [1118712.031798] relay systemd[1]: Reached target Login Prompts. relay # [1118712.033644] relay systemd[1]: Starting flakelet-relay... agent # [1118712.381724] agent systemd[1]: Reloading finished in 348 ms. agent # [1118712.458466] agent systemd[1]: Reached target Host and Network Name Lookups. agent # [1118712.458616] agent systemd[1]: Reached target User and Group Name Lookups. agent # [1118712.460347] agent systemd[1]: Starting User Login Management... agent # [1118712.461396] agent systemd[1]: Starting Permit User Sessions... agent # [1118712.466010] agent systemd[1]: Reload requested from client PID 404 ('systemctl') (unit flakelet-app.service)... agent # [1118712.466044] agent systemd[1]: Reloading... relay # [1118712.550929] relay nginx-pre-start[402]: nginx: the configuration file /nix/store/hwh65j4pm30lhgh5hab7anrqnnr1nfkx-nginx.conf syntax is ok relay # [1118712.551678] relay nginx-pre-start[402]: nginx: configuration file /nix/store/hwh65j4pm30lhgh5hab7anrqnnr1nfkx-nginx.conf test is successful relay # [1118712.571815] relay systemd[1]: Started Nginx Web Server. relay # [1118712.690126] relay systemd[1]: Starting D-Bus System Message Bus... client # [1118712.703816] client systemd-logind[315]: New seat seat0. client # [1118712.704385] client systemd[1]: Started User Login Management. client # [1118712.715773] client systemd[1]: Starting D-Bus System Message Bus... client # [1118712.717406] client systemd[1]: Starting linger-users.service... client # [1118712.756933] client systemd[1]: linger-users.service: Deactivated successfully. client # [1118712.757296] client systemd[1]: Finished linger-users.service. client # [1118712.757667] client systemd[1]: Reached target Multi-User System. relay # [1118713.027169] relay systemd-logind[354]: New seat seat0. relay # [1118713.027586] relay systemd[1]: Started User Login Management. relay # [1118713.029848] relay systemd[1]: Starting linger-users.service... relay # [1118713.075650] relay systemd[1]: linger-users.service: Deactivated successfully. relay # [1118713.075862] relay systemd[1]: Finished linger-users.service. relay # [1118713.271751] relay dbus-broker-launch[408]: Looking up NSS user entry for 'systemd-timesync'... relay # [1118713.273167] relay dbus-broker-launch[408]: NSS returned no entry for 'systemd-timesync' relay # [1118713.273167] relay dbus-broker-launch[408]: Invalid user-name in /nix/store/gzyya0rqnysc8n29w09licavwrfbk6c2-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync" relay # [1118713.274266] relay systemd[1]: Started D-Bus System Message Bus. agent # [1118712.867494] agent systemd[1]: Reloading finished in 401 ms. agent # [1118712.934912] agent systemd[1]: Finished Permit User Sessions. agent # [1118712.948576] agent systemd[1]: Started Console Getty. agent # [1118712.948689] agent systemd[1]: Getty on tty1 skipped, unmet condition check ConditionPathExists=/dev/tty0 agent # [1118712.948818] agent systemd[1]: Reached target Login Prompts. agent # [1118712.949872] agent systemd[1]: Reload requested from client PID 406 ('systemctl') (unit flakelet-flakelet-agent.service)... agent # [1118712.949903] agent systemd[1]: Reloading... client # [1118713.231094] client dbus-broker-launch[325]: Looking up NSS user entry for 'systemd-timesync'... client # [1118713.232987] client dbus-broker-launch[325]: NSS returned no entry for 'systemd-timesync' client # [1118713.232987] client dbus-broker-launch[325]: Invalid user-name in /nix/store/kin0dpm07f5g7ivsrf5g15f6w5jakfhl-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync" client # [1118713.234114] client systemd[1]: Started D-Bus System Message Bus. client # [1118713.234552] client systemd[1]: Startup finished in 4.096s. client # [1118713.239115] client dbus-broker-launch[325]: Ready relay # [1118713.280014] relay dbus-broker-launch[408]: Ready relay # [1118713.283366] relay systemd[1]: Started Dnsmasq Daemon. relay # [1118713.285332] relay dnsmasq[394]: started, version 2.93 cachesize 150 relay # [1118713.285339] relay dnsmasq[394]: compile time options: IPv6 GNU-getopt DBus no-UBus no-i18n IDN2 DHCP DHCPv6 no-Lua TFTP conntrack ipset nftset auth DNSSEC loop-detect inotify dumpfile relay # [1118713.285344] relay dnsmasq[394]: DBus support enabled: connected to system bus relay # [1118713.285351] relay dnsmasq[394]: no upstream servers configured - please set them from DBus relay # [1118713.285442] relay dnsmasq[394]: read /etc/hosts - 9 names agent # [1118713.299315] agent systemd[1]: Reloading finished in 348 ms. agent # [1118713.376870] agent systemd[1]: Starting app v1... agent # [1118713.378649] agent systemd[1]: Reload requested from client PID 443 ('systemctl') (unit flakelet-other.service)... agent # [1118713.378693] agent systemd[1]: Reloading... agent # [1118713.724357] agent systemd[1]: Reloading finished in 345 ms. agent # [1118713.798221] agent systemd[1]: Started app v1. agent # [1118713.802191] agent systemd[1]: Starting flakelet-agent... agent # [1118713.806115] agent flakelet[307]: app: updated to generation 1 agent # [1118713.833182] agent systemd[1]: Starting other v1... agent # [1118713.833678] agent systemd[1]: Finished Update flakelet service app. agent # [1118713.865230] agent systemd[1]: Started other v1. agent # [1118713.870230] agent flakelet[309]: other: updated to generation 1 agent # [1118713.872176] agent systemd[1]: Finished Update flakelet service other. agent # [1118713.881685] agent flakelet-agent[502]: {"timestamp":"2026-09-24T11:57:02.169170Z","level":"INFO","message":"local change","flakelet":"app","generation":1,"by":"host activation","target":"flakelet_relay::agent::jobs"} agent # [1118713.881907] agent flakelet-agent[502]: {"timestamp":"2026-09-24T11:57:02.169344Z","level":"INFO","message":"local change","flakelet":"other","generation":1,"by":"host activation","target":"flakelet_relay::agent::jobs"} agent # [1118713.882013] agent systemd[1]: Started flakelet-agent. agent # [1118713.884265] agent flakelet-agent[502]: {"timestamp":"2026-09-24T11:57:02.171748Z","level":"WARN","message":"connect failed: connect relay:7443: Connection refused (os error 111)","relay":"https://relay:7443","target":"flakelet_relay::agent::conn"} agent # [1118713.887040] agent flakelet[308]: flakelet-agent: updated to generation 1 agent # [1118713.888974] agent systemd[1]: Finished Update flakelet service flakelet-agent. agent # [1118713.889140] agent systemd[1]: Reached target flakelet managed services. agent # [1118713.906393] agent flakelet-agent[502]: {"timestamp":"2026-09-24T11:57:02.193890Z","level":"INFO","message":"local change","flakelet":"flakelet-agent","generation":1,"by":"host activation","target":"flakelet_relay::agent::jobs"} agent # [1118713.924691] agent systemd-logind[403]: New seat seat0. agent # [1118713.925707] agent systemd[1]: Starting D-Bus System Message Bus... agent # [1118713.925783] agent systemd[1]: Started User Login Management. agent # [1118713.926567] agent systemd[1]: Starting linger-users.service... agent # [1118713.965129] agent systemd[1]: linger-users.service: Deactivated successfully. agent # [1118713.965313] agent systemd[1]: Finished linger-users.service. agent # [1118713.965476] agent systemd[1]: Reached target Multi-User System. relay # [1118714.137928] relay flakelet-relay[400]: {"timestamp":"2026-09-24T11:57:02.425404Z","level":"INFO","message":"listening on 127.0.0.1:7400","target":"flakelet_relay::relay::server"} relay # [1118714.138431] relay flakelet-relay[400]: {"timestamp":"2026-09-24T11:57:02.425856Z","level":"INFO","message":"listening on [::]:7443","target":"flakelet_relay::relay::server"} relay # [1118714.138627] relay systemd[1]: Started flakelet-relay. relay # [1118714.148278] relay systemd[1]: Starting health for flakelet-relay... relay: (finished: waiting for unit flakelet-relay.service, in 6.16 seconds) relay: waiting for unit nginx.service agent # [1118714.417686] agent dbus-broker-launch[902]: Looking up NSS user entry for 'systemd-timesync'... agent # [1118714.420689] agent dbus-broker-launch[902]: NSS returned no entry for 'systemd-timesync' agent # [1118714.420689] agent dbus-broker-launch[902]: Invalid user-name in /nix/store/phv8q4bqjivzlnv4zxgvb2lcnrlk0i4a-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync" agent # [1118714.425828] agent systemd[1]: Started D-Bus System Message Bus. agent # [1118714.426450] agent systemd[1]: Startup finished in 5.285s. agent # [1118714.431911] agent dbus-broker-launch[902]: Ready relay: (finished: waiting for unit nginx.service, in 0.02 seconds) agent: waiting for unit flakelet.target agent: (finished: waiting for unit flakelet.target, in 0.02 seconds) agent: waiting for unit flakelet-agent.service agent: (finished: waiting for unit flakelet-agent.service, in 0.02 seconds) client: waiting for unit multi-user.target client: (finished: waiting for unit multi-user.target, in 0.02 seconds) relay: waiting for success: curl -sf http://127.0.0.1:7400/metrics | grep -q 'agent_up{host="agent"} 1' relay # [1118714.457927] relay systemd[1]: flakelet-relay-health.service: Deactivated successfully. relay # [1118714.458198] relay systemd[1]: Finished health for flakelet-relay. relay # [1118714.466053] relay flakelet[311]: flakelet-relay: updated to generation 1 relay # [1118714.467797] relay systemd[1]: Finished Update flakelet service flakelet-relay. relay # [1118714.468079] relay systemd[1]: Reached target flakelet managed services. relay # [1118714.468161] relay systemd[1]: Reached target Multi-User System. relay # [1118714.468322] relay systemd[1]: Startup finished in 5.323s. agent # [1118715.898599] agent flakelet-agent[502]: {"timestamp":"2026-09-24T11:57:04.186058Z","level":"INFO","message":"connected","relay":"https://relay:7443","relay_name":"relay","relay_version":"0.1.0","host":"agent","target":"flakelet_relay::agent::conn"} relay # [1118715.898674] relay flakelet-relay[400]: {"timestamp":"2026-09-24T11:57:04.186133Z","level":"INFO","message":"agent connected","host":"agent","version":"0.1.0","flakelets":"[Named { name: \"app\", generation: Some(1), revision: Some(\"prebuilt:app\") }, Named { name: \"flakelet-agent\", generation: Some(1), revision: Some(\"prebuilt:flakelet-agent\") }, Named { name: \"other\", generation: Some(1), revision: Some(\"prebuilt:other\") }]","target":"flakelet_relay::relay::agent_conn"} relay: (finished: waiting for success: curl -sf http://127.0.0.1:7400/metrics | grep -q 'agent_up{host="agent"} 1', in 2.05 seconds) agent: must succeed: systemctl is-active app.service other.service agent: (finished: must succeed: systemctl is-active app.service other.service, in 0.01 seconds) subtest: deploy with client certificate, relay found via SRV agent: must succeed: jq '.services.app.prebuilt="/nix/store/1hhsfvlan5s6l308sr1yx2i20rkz1mdc-flakelet-app-v2"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json agent: (finished: must succeed: jq '.services.app.prebuilt="/nix/store/1hhsfvlan5s6l308sr1yx2i20rkz1mdc-flakelet-app-v2"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json, in 0.01 seconds) client: must succeed: push --relay-srv test --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem deploy agent/app 2>&1 agent # [1118716.957535] agent flakelet-agent[502]: {"timestamp":"2026-09-24T11:57:05.244959Z","level":"INFO","message":"start","id":"a84e2e64de5b7f6a6e6aa7e0fe63bdc4","flakelet":"app","rule":"admins","relay":"relay","caller_name":"\"admin@example.org\"","target":"flakelet_relay::agent::conn"} agent # [1118716.958266] agent flakelet-agent[502]: {"timestamp":"2026-09-24T11:57:05.245758Z","level":"INFO","message":"update starting","flakelet":"app","ids":"[\"a84e2e64de5b7f6a6e6aa7e0fe63bdc4\"]","target":"flakelet_relay::agent::jobs"} agent # [1118716.999672] agent systemd[1]: Starting flakelet update app (via relay)... agent # [1118717.031394] agent systemd[1]: Started flakelet update app (via relay). agent # [1118717.036625] agent flakelet[924]: app: using prebuilt artifact /nix/store/1hhsfvlan5s6l308sr1yx2i20rkz1mdc-flakelet-app-v2 agent # [1118717.037332] agent flakelet[924]: app: activating generation 2 agent # [1118717.057423] agent systemd[1]: Stopping app v1... agent # [1118717.057870] agent systemd[1]: app.service: Deactivated successfully. agent # [1118717.058164] agent systemd[1]: Stopped app v1. agent # [1118717.076769] agent systemd[1]: Reload requested from client PID 931 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [1118717.076837] agent systemd[1]: Reloading... relay # [1118716.957052] relay flakelet-relay[400]: {"timestamp":"2026-09-24T11:57:05.244463Z","level":"INFO","message":"deploy accepted","job":"1871c3fd17edfdfcb76aed332435f41d","caller":"admin@example.org","targets":"[\"agent/app\"]","target":"flakelet_relay::relay::api"} agent # [1118717.424517] agent systemd[1]: Reloading finished in 346 ms. agent # [1118717.513900] agent systemd[1]: Reload requested from client PID 961 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [1118717.513941] agent systemd[1]: Reloading... agent # [1118717.850483] agent systemd[1]: Reloading finished in 336 ms. agent # [1118717.938022] agent systemd[1]: Reload requested from client PID 992 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [1118717.938062] agent systemd[1]: Reloading... agent # [1118718.278567] agent systemd[1]: Reloading finished in 340 ms. agent # [1118718.368973] agent systemd[1]: Reload requested from client PID 1025 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [1118718.369020] agent systemd[1]: Reloading... agent # [1118718.705275] agent systemd[1]: Reloading finished in 335 ms. agent # [1118718.793516] agent systemd[1]: Starting app v2... agent # [1118718.821877] agent systemd[1]: Started app v2. agent # [1118718.827625] agent flakelet[924]: app: updated to generation 2 agent # [1118718.829853] agent systemd[1]: flakelet-relay-job-app.service: Deactivated successfully. client: (finished: must succeed: push --relay-srv test --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem deploy agent/app 2>&1, in 2.98 seconds) agent: must succeed: systemctl show app.service -p Description | grep -q 'app v2' agent: (finished: must succeed: systemctl show app.service -p Description | grep -q 'app v2', in 0.02 seconds) agent: must succeed: journalctl -u flakelet-relay-job-app.service | grep -q 'generation 2' agent: (finished: must succeed: journalctl -u flakelet-relay-job-app.service | grep -q 'generation 2', in 0.01 seconds) agent: must succeed: flakelet status --json app | jq -c '.[0].changed.by' agent: (finished: must succeed: flakelet status --json app | jq -c '.[0].changed.by', in 0.03 seconds) (finished: subtest: deploy with client certificate, relay found via SRV, in 3.05 seconds) subtest: deploy with OIDC token, sub principal agent: must succeed: jq '.services.app.prebuilt="/nix/store/cqj0khcw75fy7djy7y554bcgxi50911r-flakelet-app-v1"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json agent: (finished: must succeed: jq '.services.app.prebuilt="/nix/store/cqj0khcw75fy7djy7y554bcgxi50911r-flakelet-app-v1"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json, in 0.01 seconds) client: must succeed: echo '{"groups": []}' | step crypto jwt sign --key /nix/store/4y8k1nvlafnzaadg5frncjjarv7aw3ry-flakelet-relay-test-issuer/jwk.json --iss https://relay:8443 --aud flakelet-relay --sub 'repo:github:example/app:ref:refs/heads/main' --exp $(( $(date +%s) + 300 )) --jti $(head -c8 /dev/urandom | base32) client: (finished: must succeed: echo '{"groups": []}' | step crypto jwt sign --key /nix/store/4y8k1nvlafnzaadg5frncjjarv7aw3ry-flakelet-relay-test-issuer/jwk.json --iss https://relay:8443 --aud flakelet-relay --sub 'repo:github:example/app:ref:refs/heads/main' --exp $(( $(date +%s) + 300 )) --jti $(head -c8 /dev/urandom | base32), in 0.06 seconds) client: must succeed: FLAKELET_RELAY_TOKEN_COMMAND='echo eyJhbGciOiJFZERTQSIsImtpZCI6ImsxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJmbGFrZWxldC1yZWxheSIsImV4cCI6MTc5MDI1MTMyOCwiZ3JvdXBzIjpbXSwiaWF0IjoxNzkwMjUxMDI4LCJpc3MiOiJodHRwczovL3JlbGF5Ojg0NDMiLCJqdGkiOiIzM1U2Ujc0TzI2NllNPT09IiwibmJmIjoxNzkwMjUxMDI4LCJzdWIiOiJyZXBvOmdpdGh1YjpleGFtcGxlL2FwcDpyZWY6cmVmcy9oZWFkcy9tYWluIn0.gDyVmzHwq2yEbHans97PhWwR-xObz7XnINUk1_WAXl6DUCbI-bForsGVISiA7urBrO-PHFTBA07As0-de6cBBQ' push --relay https://relay:7443 deploy agent/app 2>&1 agent # [1118719.908906] agent flakelet-agent[502]: {"timestamp":"2026-09-24T11:57:08.196351Z","level":"INFO","message":"update finished","flakelet":"app","status":"Updated","generation":"Some(2)","target":"flakelet_relay::agent::jobs"} agent # [1118720.082213] agent flakelet-agent[502]: {"timestamp":"2026-09-24T11:57:08.369605Z","level":"INFO","message":"start","id":"18198525fba59166df6f37096fdd6c86","flakelet":"app","rule":"ci","relay":"relay","caller_name":"\"repo:github:example/app:ref:refs/heads/main\"","target":"flakelet_relay::agent::conn"} agent # [1118720.082686] agent flakelet-agent[502]: {"timestamp":"2026-09-24T11:57:08.370179Z","level":"INFO","message":"update starting","flakelet":"app","ids":"[\"18198525fba59166df6f37096fdd6c86\"]","target":"flakelet_relay::agent::jobs"} agent # [1118720.123320] agent systemd[1]: Starting flakelet update app (via relay)... agent # [1118720.154286] agent systemd[1]: Started flakelet update app (via relay). agent # [1118720.158046] agent flakelet[1094]: app: using prebuilt artifact /nix/store/cqj0khcw75fy7djy7y554bcgxi50911r-flakelet-app-v1 agent # [1118720.158707] agent flakelet[1094]: app: activating generation 3 agent # [1118720.173187] agent systemd[1]: Stopping app v2... relay # [1118719.910634] relay flakelet-relay[400]: {"timestamp":"2026-09-24T11:57:08.198045Z","level":"INFO","message":"deploy finished","job":"1871c3fd17edfdfcb76aed332435f41d","ok":true,"target":"flakelet_relay::relay::api"} relay # [1118720.081843] relay flakelet-relay[400]: {"timestamp":"2026-09-24T11:57:08.369267Z","level":"INFO","message":"deploy accepted","job":"cdecf93ce00bc2523ddec4012c746c8c","caller":"repo:github:example/app:ref:refs/heads/main","targets":"[\"agent/app\"]","target":"flakelet_relay::relay::api"} agent # [1118720.174987] agent systemd[1]: app.service: Deactivated successfully. agent # [1118720.175411] agent systemd[1]: Stopped app v2. agent # [1118720.193182] agent systemd[1]: Reload requested from client PID 1101 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [1118720.193252] agent systemd[1]: Reloading... agent # [1118720.545015] agent systemd[1]: Reloading finished in 351 ms. agent # [1118720.644437] agent systemd[1]: Reload requested from client PID 1131 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [1118720.644483] agent systemd[1]: Reloading... agent # [1118720.979720] agent systemd[1]: Reloading finished in 334 ms. agent # [1118721.063177] agent systemd[1]: Reload requested from client PID 1160 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [1118721.063219] agent systemd[1]: Reloading... agent # [1118721.400399] agent systemd[1]: Reloading finished in 336 ms. agent # [1118721.500451] agent systemd[1]: Reload requested from client PID 1191 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [1118721.500493] agent systemd[1]: Reloading... agent # [1118721.835428] agent systemd[1]: Reloading finished in 334 ms. agent # [1118721.924591] agent systemd[1]: Starting app v1... agent # [1118721.960397] agent systemd[1]: Started app v1. agent # [1118721.966657] agent flakelet[1094]: app: updated to generation 3 agent # [1118721.968627] agent systemd[1]: flakelet-relay-job-app.service: Deactivated successfully. client: (finished: must succeed: FLAKELET_RELAY_TOKEN_COMMAND='echo eyJhbGciOiJFZERTQSIsImtpZCI6ImsxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJmbGFrZWxldC1yZWxheSIsImV4cCI6MTc5MDI1MTMyOCwiZ3JvdXBzIjpbXSwiaWF0IjoxNzkwMjUxMDI4LCJpc3MiOiJodHRwczovL3JlbGF5Ojg0NDMiLCJqdGkiOiIzM1U2Ujc0TzI2NllNPT09IiwibmJmIjoxNzkwMjUxMDI4LCJzdWIiOiJyZXBvOmdpdGh1YjpleGFtcGxlL2FwcDpyZWY6cmVmcy9oZWFkcy9tYWluIn0.gDyVmzHwq2yEbHans97PhWwR-xObz7XnINUk1_WAXl6DUCbI-bForsGVISiA7urBrO-PHFTBA07As0-de6cBBQ' push --relay https://relay:7443 deploy agent/app 2>&1, in 2.99 seconds) (finished: subtest: deploy with OIDC token, sub principal, in 3.06 seconds) subtest: policy denies uncovered target client: must fail: FLAKELET_RELAY_TOKEN_COMMAND='echo eyJhbGciOiJFZERTQSIsImtpZCI6ImsxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJmbGFrZWxldC1yZWxheSIsImV4cCI6MTc5MDI1MTMyOCwiZ3JvdXBzIjpbXSwiaWF0IjoxNzkwMjUxMDI4LCJpc3MiOiJodHRwczovL3JlbGF5Ojg0NDMiLCJqdGkiOiIzM1U2Ujc0TzI2NllNPT09IiwibmJmIjoxNzkwMjUxMDI4LCJzdWIiOiJyZXBvOmdpdGh1YjpleGFtcGxlL2FwcDpyZWY6cmVmcy9oZWFkcy9tYWluIn0.gDyVmzHwq2yEbHans97PhWwR-xObz7XnINUk1_WAXl6DUCbI-bForsGVISiA7urBrO-PHFTBA07As0-de6cBBQ' push --relay https://relay:7443 deploy agent/other 2>&1 client: (finished: must fail: FLAKELET_RELAY_TOKEN_COMMAND='echo eyJhbGciOiJFZERTQSIsImtpZCI6ImsxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJmbGFrZWxldC1yZWxheSIsImV4cCI6MTc5MDI1MTMyOCwiZ3JvdXBzIjpbXSwiaWF0IjoxNzkwMjUxMDI4LCJpc3MiOiJodHRwczovL3JlbGF5Ojg0NDMiLCJqdGkiOiIzM1U2Ujc0TzI2NllNPT09IiwibmJmIjoxNzkwMjUxMDI4LCJzdWIiOiJyZXBvOmdpdGh1YjpleGFtcGxlL2FwcDpyZWY6cmVmcy9oZWFkcy9tYWluIn0.gDyVmzHwq2yEbHans97PhWwR-xObz7XnINUk1_WAXl6DUCbI-bForsGVISiA7urBrO-PHFTBA07As0-de6cBBQ' push --relay https://relay:7443 deploy agent/other 2>&1, in 0.02 seconds) (finished: subtest: policy denies uncovered target, in 0.02 seconds) subtest: claim principal from groups, unchanged counts as success client: must succeed: echo '{"groups": ["wheel"]}' | step crypto jwt sign --key /nix/store/4y8k1nvlafnzaadg5frncjjarv7aw3ry-flakelet-relay-test-issuer/jwk.json --iss https://relay:8443 --aud flakelet-relay --sub 'someone' --exp $(( $(date +%s) + 300 )) --jti $(head -c8 /dev/urandom | base32) client: (finished: must succeed: echo '{"groups": ["wheel"]}' | step crypto jwt sign --key /nix/store/4y8k1nvlafnzaadg5frncjjarv7aw3ry-flakelet-relay-test-issuer/jwk.json --iss https://relay:8443 --aud flakelet-relay --sub 'someone' --exp $(( $(date +%s) + 300 )) --jti $(head -c8 /dev/urandom | base32), in 0.06 seconds) client: must succeed: FLAKELET_RELAY_TOKEN_COMMAND='echo eyJhbGciOiJFZERTQSIsImtpZCI6ImsxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJmbGFrZWxldC1yZWxheSIsImV4cCI6MTc5MDI1MTMzMSwiZ3JvdXBzIjpbIndoZWVsIl0sImlhdCI6MTc5MDI1MTAzMSwiaXNzIjoiaHR0cHM6Ly9yZWxheTo4NDQzIiwianRpIjoiSkhXS1AzNkpQVk9TTT09PSIsIm5iZiI6MTc5MDI1MTAzMSwic3ViIjoic29tZW9uZSJ9.tWWzOPufWyI09f2e9JtsbisPYfocg-9ysmkOYXzxmIeBb3bvYC0bjuKSBvE1lgHkiOT91GkzKV-K-guBKWaGCA' push --relay https://relay:7443 deploy agent/other 2>&1 relay # [1118723.038468] relay flakelet-relay[400]: {"timestamp":"2026-09-24T11:57:11.325890Z","level":"INFO","message":"deploy finished","job":"cdecf93ce00bc2523ddec4012c746c8c","ok":true,"target":"flakelet_relay::relay::api"} relay # [1118723.141336] relay flakelet-relay[400]: {"timestamp":"2026-09-24T11:57:11.428795Z","level":"INFO","message":"deploy accepted","job":"29978ed4b9de2615fac9e96a8a859654","caller":"someone","targets":"[\"agent/other\"]","target":"flakelet_relay::relay::api"} agent # [1118723.037235] agent flakelet-agent[502]: {"timestamp":"2026-09-24T11:57:11.324623Z","level":"INFO","message":"update finished","flakelet":"app","status":"Updated","generation":"Some(3)","target":"flakelet_relay::agent::jobs"} agent # [1118723.141692] agent flakelet-agent[502]: {"timestamp":"2026-09-24T11:57:11.429129Z","level":"INFO","message":"start","id":"f2a409b99d4deeab64d0445259ebca7b","flakelet":"other","rule":"admins","relay":"relay","caller_name":"\"someone\"","target":"flakelet_relay::agent::conn"} agent # [1118723.141974] agent flakelet-agent[502]: {"timestamp":"2026-09-24T11:57:11.429476Z","level":"INFO","message":"update starting","flakelet":"other","ids":"[\"f2a409b99d4deeab64d0445259ebca7b\"]","target":"flakelet_relay::agent::jobs"} agent # [1118723.181241] agent systemd[1]: Starting flakelet update other (via relay)... agent # [1118723.212406] agent systemd[1]: Started flakelet update other (via relay). agent # [1118723.216647] agent flakelet[1245]: other: using prebuilt artifact /nix/store/h4k7d4dqcfa4h9d48c0rm4x08qziqfaf-flakelet-other-v1 agent # [1118723.217255] agent flakelet[1245]: other: up to date agent # [1118723.219403] agent systemd[1]: flakelet-relay-job-other.service: Deactivated successfully. client: (finished: must succeed: FLAKELET_RELAY_TOKEN_COMMAND='echo eyJhbGciOiJFZERTQSIsImtpZCI6ImsxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJmbGFrZWxldC1yZWxheSIsImV4cCI6MTc5MDI1MTMzMSwiZ3JvdXBzIjpbIndoZWVsIl0sImlhdCI6MTc5MDI1MTAzMSwiaXNzIjoiaHR0cHM6Ly9yZWxheTo4NDQzIiwianRpIjoiSkhXS1AzNkpQVk9TTT09PSIsIm5iZiI6MTc5MDI1MTAzMSwic3ViIjoic29tZW9uZSJ9.tWWzOPufWyI09f2e9JtsbisPYfocg-9ysmkOYXzxmIeBb3bvYC0bjuKSBvE1lgHkiOT91GkzKV-K-guBKWaGCA' push --relay https://relay:7443 deploy agent/other 2>&1, in 0.64 seconds) (finished: subtest: claim principal from groups, unchanged counts as success, in 0.69 seconds) subtest: push login via device flow, cached token used afterwards client: must succeed: push login --issuer https://relay:8443 --client-id flakelet-relay 2>&1 agent # [1118723.755993] agent flakelet-agent[502]: {"timestamp":"2026-09-24T11:57:12.043442Z","level":"INFO","message":"update finished","flakelet":"other","status":"Unchanged","generation":"Some(1)","target":"flakelet_relay::agent::jobs"} relay # [1118723.757697] relay flakelet-relay[400]: {"timestamp":"2026-09-24T11:57:12.045108Z","level":"INFO","message":"deploy finished","job":"29978ed4b9de2615fac9e96a8a859654","ok":true,"target":"flakelet_relay::relay::api"} relay # [1118723.782326] relay 7gsx6vaqmkb2vf8rr0w4y1g018hi9d43-device-mock[185]: 127.0.0.1 - - [24/Sep/2026 11:57:12] "POST /device HTTP/1.1" 200 - relay # [1118724.787700] relay 7gsx6vaqmkb2vf8rr0w4y1g018hi9d43-device-mock[185]: 127.0.0.1 - - [24/Sep/2026 11:57:13] "POST /token HTTP/1.1" 400 - client: (finished: must succeed: push login --issuer https://relay:8443 --client-id flakelet-relay 2>&1, in 2.09 seconds) client: must succeed: test "$(stat -c %a /root/.local/state/flakelet-push/token.json)" = 600 client: (finished: must succeed: test "$(stat -c %a /root/.local/state/flakelet-push/token.json)" = 600, in 0.01 seconds) client: must succeed: push --relay https://relay:7443 deploy agent/other 2>&1 relay # [1118725.842874] relay 7gsx6vaqmkb2vf8rr0w4y1g018hi9d43-device-mock[185]: 127.0.0.1 - - [24/Sep/2026 11:57:14] "POST /token HTTP/1.1" 200 - relay # [1118725.873824] relay flakelet-relay[400]: {"timestamp":"2026-09-24T11:57:14.161296Z","level":"INFO","message":"deploy accepted","job":"939ee267879a39e6ee8d9d4eeb492bee","caller":"dev@example.org","targets":"[\"agent/other\"]","target":"flakelet_relay::relay::api"} agent # [1118725.874280] agent flakelet-agent[502]: {"timestamp":"2026-09-24T11:57:14.161705Z","level":"INFO","message":"start","id":"982a04f7251c9dc72fb8fa39009ed2de","flakelet":"other","rule":"admins","relay":"relay","caller_name":"\"dev@example.org\"","target":"flakelet_relay::agent::conn"} agent # [1118725.874887] agent flakelet-agent[502]: {"timestamp":"2026-09-24T11:57:14.162280Z","level":"INFO","message":"update starting","flakelet":"other","ids":"[\"982a04f7251c9dc72fb8fa39009ed2de\"]","target":"flakelet_relay::agent::jobs"} agent # [1118725.920632] agent systemd[1]: Starting flakelet update other (via relay)... agent # [1118725.957276] agent systemd[1]: Started flakelet update other (via relay). agent # [1118725.962092] agent flakelet[1273]: other: using prebuilt artifact /nix/store/h4k7d4dqcfa4h9d48c0rm4x08qziqfaf-flakelet-other-v1 agent # [1118725.962680] agent flakelet[1273]: other: up to date agent # [1118725.964559] agent systemd[1]: flakelet-relay-job-other.service: Deactivated successfully. client: (finished: must succeed: push --relay https://relay:7443 deploy agent/other 2>&1, in 0.65 seconds) client: must succeed: push --relay https://relay:7443 deploy agent/other 2>&1 agent # [1118726.505633] agent flakelet-agent[502]: {"timestamp":"2026-09-24T11:57:14.793038Z","level":"INFO","message":"update finished","flakelet":"other","status":"Unchanged","generation":"Some(1)","target":"flakelet_relay::agent::jobs"} agent # [1118726.529942] agent flakelet-agent[502]: {"timestamp":"2026-09-24T11:57:14.817418Z","level":"INFO","message":"start","id":"2dce3537d090579454593b6b28a2770b","flakelet":"other","rule":"admins","relay":"relay","caller_name":"\"dev@example.org\"","target":"flakelet_relay::agent::conn"} agent # [1118726.530327] agent flakelet-agent[502]: {"timestamp":"2026-09-24T11:57:14.817816Z","level":"INFO","message":"update starting","flakelet":"other","ids":"[\"2dce3537d090579454593b6b28a2770b\"]","target":"flakelet_relay::agent::jobs"} agent # [1118726.568365] agent systemd[1]: Starting flakelet update other (via relay)... agent # [1118726.600764] agent systemd[1]: Started flakelet update other (via relay). agent # [1118726.606015] agent flakelet[1293]: other: using prebuilt artifact /nix/store/h4k7d4dqcfa4h9d48c0rm4x08qziqfaf-flakelet-other-v1 agent # [1118726.606591] agent flakelet[1293]: other: up to date agent # [1118726.608391] agent systemd[1]: flakelet-relay-job-other.service: Deactivated successfully. relay # [1118726.507185] relay flakelet-relay[400]: {"timestamp":"2026-09-24T11:57:14.794601Z","level":"INFO","message":"deploy finished","job":"939ee267879a39e6ee8d9d4eeb492bee","ok":true,"target":"flakelet_relay::relay::api"} relay # [1118726.529720] relay flakelet-relay[400]: {"timestamp":"2026-09-24T11:57:14.817194Z","level":"INFO","message":"deploy accepted","job":"1c1582ab37d3f816b886c7ce7ba821e2","caller":"dev@example.org","targets":"[\"agent/other\"]","target":"flakelet_relay::relay::api"} client: (finished: must succeed: push --relay https://relay:7443 deploy agent/other 2>&1, in 0.65 seconds) (finished: subtest: push login via device flow, cached token used afterwards, in 3.39 seconds) subtest: waves: failing first wave skips the second agent: must succeed: jq '.services.app.prebuilt="/nix/store/qy8h9kkvfrxq0bc47vwjy6lfnk9i95ga-flakelet-app-broken"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json agent: (finished: must succeed: jq '.services.app.prebuilt="/nix/store/qy8h9kkvfrxq0bc47vwjy6lfnk9i95ga-flakelet-app-broken"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json, in 0.01 seconds) client: must fail: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem deploy agent/app --wave agent/other 2>&1 agent # [1118727.151590] agent flakelet-agent[502]: {"timestamp":"2026-09-24T11:57:15.439054Z","level":"INFO","message":"update finished","flakelet":"other","status":"Unchanged","generation":"Some(1)","target":"flakelet_relay::agent::jobs"} agent # [1118727.187238] agent flakelet-agent[502]: {"timestamp":"2026-09-24T11:57:15.474723Z","level":"INFO","message":"start","id":"b9bd08d121d8d401636379972fec5a73","flakelet":"app","rule":"admins","relay":"relay","caller_name":"\"admin@example.org\"","target":"flakelet_relay::agent::conn"} agent # [1118727.187443] agent flakelet-agent[502]: {"timestamp":"2026-09-24T11:57:15.474946Z","level":"INFO","message":"update starting","flakelet":"app","ids":"[\"b9bd08d121d8d401636379972fec5a73\"]","target":"flakelet_relay::agent::jobs"} agent # [1118727.225329] agent systemd[1]: Starting flakelet update app (via relay)... agent # [1118727.253235] agent systemd[1]: Started flakelet update app (via relay). agent # [1118727.258483] agent flakelet[1316]: app: using prebuilt artifact /nix/store/qy8h9kkvfrxq0bc47vwjy6lfnk9i95ga-flakelet-app-broken agent # [1118727.259186] agent flakelet[1316]: app: activating generation 4 agent # [1118727.275756] agent systemd[1]: Stopping app v1... agent # [1118727.277565] agent systemd[1]: app.service: Deactivated successfully. agent # [1118727.277944] agent systemd[1]: Stopped app v1. agent # [1118727.293318] agent systemd[1]: Reload requested from client PID 1323 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [1118727.293385] agent systemd[1]: Reloading... relay # [1118727.152693] relay flakelet-relay[400]: {"timestamp":"2026-09-24T11:57:15.440089Z","level":"INFO","message":"deploy finished","job":"1c1582ab37d3f816b886c7ce7ba821e2","ok":true,"target":"flakelet_relay::relay::api"} relay # [1118727.186902] relay flakelet-relay[400]: {"timestamp":"2026-09-24T11:57:15.474381Z","level":"INFO","message":"deploy accepted","job":"2feedd7e91a8f353c42fa3ee97c25d43","caller":"admin@example.org","targets":"[\"agent/app\", \"agent/other\"]","target":"flakelet_relay::relay::api"} agent # [1118727.641504] agent systemd[1]: Reloading finished in 347 ms. agent # [1118727.729603] agent systemd[1]: Reload requested from client PID 1353 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [1118727.729645] agent systemd[1]: Reloading... agent # [1118728.063989] agent systemd[1]: Reloading finished in 333 ms. agent # [1118728.148383] agent systemd[1]: Reload requested from client PID 1384 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [1118728.148422] agent systemd[1]: Reloading... agent # [1118728.488621] agent systemd[1]: Reloading finished in 339 ms. agent # [1118728.573937] agent systemd[1]: Reload requested from client PID 1417 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [1118728.573977] agent systemd[1]: Reloading... agent # [1118728.912256] agent systemd[1]: Reloading finished in 337 ms. agent # [1118729.002473] agent systemd[1]: Starting app broken... agent # [1118729.036658] agent (nonexistent)[1447]: app.service: Unable to locate executable '/nonexistent': No such file or directory agent # [1118729.036684] agent (nonexistent)[1447]: app.service: Failed at step EXEC spawning /nonexistent: No such file or directory agent # [1118729.038266] agent systemd[1]: app.service: Main process exited, code=exited, status=203/EXEC agent # [1118729.038380] agent systemd[1]: app.service: Failed with result 'exit-code'. agent # [1118729.038799] agent systemd[1]: Failed to start app broken. agent # [1118729.072363] agent systemd[1]: Reload requested from client PID 1453 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [1118729.072438] agent systemd[1]: Reloading... agent # [1118729.418251] agent systemd[1]: Reloading finished in 345 ms. agent # [1118729.506945] agent systemd[1]: Reload requested from client PID 1483 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [1118729.506989] agent systemd[1]: Reloading... agent # [1118729.842076] agent systemd[1]: Reloading finished in 334 ms. agent # [1118729.922449] agent systemd[1]: Reload requested from client PID 1515 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [1118729.922490] agent systemd[1]: Reloading... agent # [1118730.269694] agent systemd[1]: Reloading finished in 346 ms. agent # [1118730.361813] agent systemd[1]: Reload requested from client PID 1547 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [1118730.361853] agent systemd[1]: Reloading... agent # [1118730.705587] agent systemd[1]: Reloading finished in 343 ms. agent # [1118730.786666] agent systemd[1]: Starting app v1... agent # [1118730.813092] agent systemd[1]: Started app v1. agent # [1118730.814319] agent flakelet[1316]: app: deploy failed, rolled back: systemctl start app.service failed: agent # [1118730.814319] agent flakelet[1316]: failed units: app.service. See journalctl -u app.service agent # [1118730.816682] agent systemd[1]: flakelet-relay-job-app.service: Main process exited, code=exited, status=1/FAILURE agent # [1118730.816780] agent systemd[1]: flakelet-relay-job-app.service: Failed with result 'exit-code'. client: (finished: must fail: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem deploy agent/app --wave agent/other 2>&1, in 4.30 seconds) agent: must succeed: systemctl show app.service -p Description | grep -q 'app v1' agent: (finished: must succeed: systemctl show app.service -p Description | grep -q 'app v1', in 0.02 seconds) (finished: subtest: waves: failing first wave skips the second, in 4.34 seconds) subtest: host patterns expand on the relay agent: must succeed: jq '.services.app.prebuilt="/nix/store/cqj0khcw75fy7djy7y554bcgxi50911r-flakelet-app-v1"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json agent: (finished: must succeed: jq '.services.app.prebuilt="/nix/store/cqj0khcw75fy7djy7y554bcgxi50911r-flakelet-app-v1"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json, in 0.01 seconds) client: must succeed: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem deploy '@all/app' 2>&1 agent # [1118731.468455] agent flakelet-agent[502]: {"timestamp":"2026-09-24T11:57:19.755883Z","level":"INFO","message":"update finished","flakelet":"app","status":"RolledBack","generation":"Some(3)","target":"flakelet_relay::agent::jobs"} agent # [1118731.533466] agent flakelet-agent[502]: {"timestamp":"2026-09-24T11:57:19.820890Z","level":"INFO","message":"start","id":"2a09030c5fcb51fa7d467adc5538e813","flakelet":"app","rule":"admins","relay":"relay","caller_name":"\"admin@example.org\"","target":"flakelet_relay::agent::conn"} agent # [1118731.533875] agent flakelet-agent[502]: {"timestamp":"2026-09-24T11:57:19.821364Z","level":"INFO","message":"update starting","flakelet":"app","ids":"[\"2a09030c5fcb51fa7d467adc5538e813\"]","target":"flakelet_relay::agent::jobs"} agent # [1118731.573825] agent systemd[1]: Starting flakelet update app (via relay)... agent # [1118731.605267] agent systemd[1]: Started flakelet update app (via relay). agent # [1118731.609980] agent flakelet[1603]: app: using prebuilt artifact /nix/store/cqj0khcw75fy7djy7y554bcgxi50911r-flakelet-app-v1 agent # [1118731.610552] agent flakelet[1603]: app: up to date agent # [1118731.612984] agent systemd[1]: flakelet-relay-job-app.service: Deactivated successfully. relay # [1118731.469995] relay flakelet-relay[400]: {"timestamp":"2026-09-24T11:57:19.757430Z","level":"INFO","message":"deploy finished","job":"2feedd7e91a8f353c42fa3ee97c25d43","ok":false,"target":"flakelet_relay::relay::api"} relay # [1118731.533184] relay flakelet-relay[400]: {"timestamp":"2026-09-24T11:57:19.820610Z","level":"INFO","message":"deploy accepted","job":"c0dd401dca5a9884ec9bae3e704447a3","caller":"admin@example.org","targets":"[\"agent/app\"]","target":"flakelet_relay::relay::api"} client: (finished: must succeed: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem deploy '@all/app' 2>&1, in 0.65 seconds) client: must fail: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem deploy '*/app' 2>&1 agent # [1118732.155628] agent flakelet-agent[502]: {"timestamp":"2026-09-24T11:57:20.443080Z","level":"INFO","message":"update finished","flakelet":"app","status":"Unchanged","generation":"Some(3)","target":"flakelet_relay::agent::jobs"} agent # [1118732.180443] agent flakelet-agent[502]: {"timestamp":"2026-09-24T11:57:20.467924Z","level":"INFO","message":"start","id":"183e5ec56796d97e3c573c98e3819a8d","flakelet":"app","rule":"admins","relay":"relay","caller_name":"\"admin@example.org\"","target":"flakelet_relay::agent::conn"} agent # [1118732.180813] agent flakelet-agent[502]: {"timestamp":"2026-09-24T11:57:20.468308Z","level":"INFO","message":"update starting","flakelet":"app","ids":"[\"183e5ec56796d97e3c573c98e3819a8d\"]","target":"flakelet_relay::agent::jobs"} agent # [1118732.221067] agent systemd[1]: Starting flakelet update app (via relay)... agent # [1118732.250190] agent systemd[1]: Started flakelet update app (via relay). agent # [1118732.255668] agent flakelet[1627]: app: using prebuilt artifact /nix/store/cqj0khcw75fy7djy7y554bcgxi50911r-flakelet-app-v1 agent # [1118732.256273] agent flakelet[1627]: app: up to date agent # [1118732.258293] agent systemd[1]: flakelet-relay-job-app.service: Deactivated successfully. relay # [1118732.157145] relay flakelet-relay[400]: {"timestamp":"2026-09-24T11:57:20.444513Z","level":"INFO","message":"deploy finished","job":"c0dd401dca5a9884ec9bae3e704447a3","ok":true,"target":"flakelet_relay::relay::api"} relay # [1118732.180269] relay flakelet-relay[400]: {"timestamp":"2026-09-24T11:57:20.467742Z","level":"INFO","message":"deploy accepted","job":"d2bf329c361ecc21b1a9e6936fec7d0a","caller":"admin@example.org","targets":"[\"agent/app\", \"ghost/app\"]","target":"flakelet_relay::relay::api"} client: (finished: must fail: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem deploy '*/app' 2>&1, in 0.64 seconds) client: must fail: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem deploy agent/app --wave '*/app' 2>&1 relay # [1118732.801204] relay flakelet-relay[400]: {"timestamp":"2026-09-24T11:57:21.088614Z","level":"INFO","message":"deploy finished","job":"d2bf329c361ecc21b1a9e6936fec7d0a","ok":false,"target":"flakelet_relay::relay::api"} relay # [1118732.822357] relay flakelet-relay[400]: {"timestamp":"2026-09-24T11:57:21.109848Z","level":"INFO","message":"deploy accepted","job":"472d59ec9dafceeeb0186e474de41044","caller":"admin@example.org","targets":"[\"agent/app\", \"ghost/app\"]","target":"flakelet_relay::relay::api"} agent # [1118732.799565] agent flakelet-agent[502]: {"timestamp":"2026-09-24T11:57:21.087015Z","level":"INFO","message":"update finished","flakelet":"app","status":"Unchanged","generation":"Some(3)","target":"flakelet_relay::agent::jobs"} agent # [1118732.822618] agent flakelet-agent[502]: {"timestamp":"2026-09-24T11:57:21.110102Z","level":"INFO","message":"start","id":"962bc4eec0e73064ed8aa107903fd0e3","flakelet":"app","rule":"admins","relay":"relay","caller_name":"\"admin@example.org\"","target":"flakelet_relay::agent::conn"} agent # [1118732.822989] agent flakelet-agent[502]: {"timestamp":"2026-09-24T11:57:21.110483Z","level":"INFO","message":"update starting","flakelet":"app","ids":"[\"962bc4eec0e73064ed8aa107903fd0e3\"]","target":"flakelet_relay::agent::jobs"} agent # [1118732.860769] agent systemd[1]: Starting flakelet update app (via relay)... agent # [1118732.890291] agent systemd[1]: Started flakelet update app (via relay). agent # [1118732.895726] agent flakelet[1647]: app: using prebuilt artifact /nix/store/cqj0khcw75fy7djy7y554bcgxi50911r-flakelet-app-v1 agent # [1118732.896345] agent flakelet[1647]: app: up to date agent # [1118732.898554] agent systemd[1]: flakelet-relay-job-app.service: Deactivated successfully. client: (finished: must fail: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem deploy agent/app --wave '*/app' 2>&1, in 0.64 seconds) client: must fail: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem deploy '@all/nope' 2>&1 client: (finished: must fail: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem deploy '@all/nope' 2>&1, in 0.02 seconds) (finished: subtest: host patterns expand on the relay, in 1.97 seconds) subtest: detach returns once the job is accepted agent: must succeed: jq '.services.app.prebuilt="/nix/store/yqdz0mcr1b2ljpw9vli2zvbmm43j4gr2-flakelet-app-slow2"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json agent: (finished: must succeed: jq '.services.app.prebuilt="/nix/store/yqdz0mcr1b2ljpw9vli2zvbmm43j4gr2-flakelet-app-slow2"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json, in 0.01 seconds) client: must succeed: timeout 20 push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem deploy --detach agent/app 2>&1 client: (finished: must succeed: timeout 20 push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem deploy --detach agent/app 2>&1, in 0.02 seconds) agent: waiting for success: systemctl is-active flakelet-relay-job-app.service agent # [1118733.438770] agent flakelet-agent[502]: {"timestamp":"2026-09-24T11:57:21.726218Z","level":"INFO","message":"update finished","flakelet":"app","status":"Unchanged","generation":"Some(3)","target":"flakelet_relay::agent::jobs"} agent # [1118733.500137] agent flakelet-agent[502]: {"timestamp":"2026-09-24T11:57:21.787597Z","level":"INFO","message":"start","id":"8203995bd4cc07f2305145a76a28bdd5","flakelet":"app","rule":"admins","relay":"relay","caller_name":"\"admin@example.org\"","target":"flakelet_relay::agent::conn"} agent # [1118733.500585] agent flakelet-agent[502]: {"timestamp":"2026-09-24T11:57:21.788079Z","level":"INFO","message":"update starting","flakelet":"app","ids":"[\"8203995bd4cc07f2305145a76a28bdd5\"]","target":"flakelet_relay::agent::jobs"} agent # [1118733.543533] agent systemd[1]: Starting flakelet update app (via relay)... agent # [1118733.572349] agent systemd[1]: Started flakelet update app (via relay). agent # [1118733.577725] agent flakelet[1671]: app: using prebuilt artifact /nix/store/yqdz0mcr1b2ljpw9vli2zvbmm43j4gr2-flakelet-app-slow2 agent # [1118733.578427] agent flakelet[1671]: app: activating generation 4 agent # [1118733.591848] agent systemd[1]: Stopping app v1... agent # [1118733.592341] agent systemd[1]: app.service: Deactivated successfully. agent # [1118733.592602] agent systemd[1]: Stopped app v1. agent # [1118733.612218] agent systemd[1]: Reload requested from client PID 1678 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [1118733.612290] agent systemd[1]: Reloading... relay # [1118733.440520] relay flakelet-relay[400]: {"timestamp":"2026-09-24T11:57:21.727933Z","level":"INFO","message":"deploy finished","job":"472d59ec9dafceeeb0186e474de41044","ok":false,"target":"flakelet_relay::relay::api"} relay # [1118733.499922] relay flakelet-relay[400]: {"timestamp":"2026-09-24T11:57:21.787352Z","level":"INFO","message":"deploy accepted","job":"8fa8da58d62dacacb97fcd33499307c6","caller":"admin@example.org","targets":"[\"agent/app\"]","target":"flakelet_relay::relay::api"} agent # [1118733.962143] agent systemd[1]: Reloading finished in 349 ms. agent # [1118734.050239] agent systemd[1]: Reload requested from client PID 1712 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [1118734.050278] agent systemd[1]: Reloading... agent # [1118734.384345] agent systemd[1]: Reloading finished in 333 ms. agent # [1118734.460551] agent systemd[1]: Reload requested from client PID 1741 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [1118734.460594] agent systemd[1]: Reloading... agent: (finished: waiting for success: systemctl is-active flakelet-relay-job-app.service, in 1.37 seconds) ??? Warning (UserWarning): wait_until_fails(): passing a bare int/float as a duration is deprecated. Use datetime.timedelta instead. File "/nix/store/1pfwbz29g35j2afnmzzhnp99xjq1kqx9-nixos-test-driver-1.1/lib/python3.14/site-packages/test_driver/duration.py", line 39 agent: waiting for failure: systemctl is-active flakelet-relay-job-app.service ??? Warning (UserWarning): execute(): passing a bare int/float as a duration is deprecated. Use datetime.timedelta instead. File "/nix/store/1pfwbz29g35j2afnmzzhnp99xjq1kqx9-nixos-test-driver-1.1/lib/python3.14/site-packages/test_driver/duration.py", line 39 agent # [1118734.794257] agent systemd[1]: Reloading finished in 333 ms. agent # [1118734.884307] agent systemd[1]: Reload requested from client PID 1774 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [1118734.884347] agent systemd[1]: Reloading... agent # [1118735.217139] agent systemd[1]: Reloading finished in 332 ms. agent # [1118735.290703] agent systemd[1]: Starting app slow2... agent: (finished: waiting for failure: systemctl is-active flakelet-relay-job-app.service, in 6.50 seconds) (finished: subtest: detach returns once the job is accepted, in 7.91 seconds) subtest: unknown host is final, missing agent is retried, agents listing client: must fail: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem deploy nope/app 2>&1 client: (finished: must fail: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem deploy nope/app 2>&1, in 0.02 seconds) agent: must succeed: systemctl stop flakelet-agent agent: (finished: must succeed: systemctl stop flakelet-agent, in 0.08 seconds) relay: waiting for success: curl -sf http://127.0.0.1:7400/metrics | grep -q 'agent_up{host="agent"} 0' relay: (finished: waiting for success: curl -sf http://127.0.0.1:7400/metrics | grep -q 'agent_up{host="agent"} 0', in 0.02 seconds) client: must succeed: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem deploy agent/other > /tmp/retry.out 2>&1 & client: (finished: must succeed: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem deploy agent/other > /tmp/retry.out 2>&1 &, in 0.01 seconds) client: waiting for success: grep -q 'agent_unavailable.*retrying' /tmp/retry.out agent # [1118741.363337] agent systemd[1]: Started app slow2. agent # [1118741.370717] agent flakelet[1671]: app: updated to generation 4 agent # [1118741.372999] agent systemd[1]: flakelet-relay-job-app.service: Deactivated successfully. agent # [1118741.412199] agent systemd[1]: Stopping flakelet-agent... agent # [1118741.446386] agent systemd[1]: flakelet-agent.service: Deactivated successfully. agent # [1118741.446810] agent systemd[1]: Stopped flakelet-agent. agent # [1118741.447266] agent systemd[1]: flakelet-agent.service: Consumed 1.963s CPU time over 27.644s wall clock time. relay # [1118741.412331] relay flakelet-relay[400]: {"timestamp":"2026-09-24T11:57:29.699736Z","level":"INFO","message":"agent disconnected: IO error: peer closed connection without sending TLS close_notify: https://docs.rs/rustls/latest/rustls/manual/_03_howto/index.html#unexpected-eof","host":"agent","target":"flakelet_relay::relay::agent_conn"} client: (finished: waiting for success: grep -q 'agent_unavailable.*retrying' /tmp/retry.out, in 1.02 seconds) agent: must succeed: systemctl start flakelet-agent agent: (finished: must succeed: systemctl start flakelet-agent, in 0.10 seconds) ??? Warning (UserWarning): wait_until_succeeds(): passing a bare int/float as a duration is deprecated. Use datetime.timedelta instead. File "/nix/store/1pfwbz29g35j2afnmzzhnp99xjq1kqx9-nixos-test-driver-1.1/lib/python3.14/site-packages/test_driver/duration.py", line 39 client: waiting for success: grep -q 'agent/other: unchanged' /tmp/retry.out agent # [1118742.544460] agent systemd[1]: Starting flakelet-agent... agent # [1118742.604437] agent flakelet-agent[1836]: {"timestamp":"2026-09-24T11:57:30.891890Z","level":"INFO","message":"reattaching to running unit","flakelet":"app","ids":"[\"8203995bd4cc07f2305145a76a28bdd5\"]","target":"flakelet_relay::agent::jobs"} agent # [1118742.625653] agent systemd[1]: Started flakelet-agent. agent # [1118742.636351] agent flakelet-agent[1836]: {"timestamp":"2026-09-24T11:57:30.923837Z","level":"INFO","message":"connected","relay":"https://relay:7443","relay_name":"relay","relay_version":"0.1.0","host":"agent","target":"flakelet_relay::agent::conn"} relay # [1118742.636532] relay flakelet-relay[400]: {"timestamp":"2026-09-24T11:57:30.923939Z","level":"INFO","message":"agent connected","host":"agent","version":"0.1.0","flakelets":"[Named { name: \"app\", generation: Some(4), revision: Some(\"prebuilt:app\") }, Named { name: \"flakelet-agent\", generation: Some(1), revision: Some(\"prebuilt:flakelet-agent\") }, Named { name: \"other\", generation: Some(1), revision: Some(\"prebuilt:other\") }]","target":"flakelet_relay::relay::agent_conn"} agent # [1118743.141350] agent flakelet-agent[1836]: {"timestamp":"2026-09-24T11:57:31.428766Z","level":"INFO","message":"update finished","flakelet":"app","status":"Updated","generation":"Some(4)","target":"flakelet_relay::agent::jobs"} agent # [1118744.527788] agent flakelet-agent[1836]: {"timestamp":"2026-09-24T11:57:32.815220Z","level":"INFO","message":"start","id":"1dd06823958af220abf7708ecd0aa01e","flakelet":"other","rule":"admins","relay":"relay","caller_name":"\"admin@example.org\"","target":"flakelet_relay::agent::conn"} agent # [1118744.528791] agent flakelet-agent[1836]: {"timestamp":"2026-09-24T11:57:32.815972Z","level":"INFO","message":"update starting","flakelet":"other","ids":"[\"1dd06823958af220abf7708ecd0aa01e\"]","target":"flakelet_relay::agent::jobs"} agent # [1118744.561793] agent systemd[1]: Starting flakelet update other (via relay)... agent # [1118744.596248] agent systemd[1]: Started flakelet update other (via relay). agent # [1118744.601092] agent flakelet[2249]: other: using prebuilt artifact /nix/store/h4k7d4dqcfa4h9d48c0rm4x08qziqfaf-flakelet-other-v1 agent # [1118744.601693] agent flakelet[2249]: other: up to date agent # [1118744.602222] agent systemd[1]: flakelet-relay-job-other.service: Deactivated successfully. relay # [1118744.527448] relay flakelet-relay[400]: {"timestamp":"2026-09-24T11:57:32.814860Z","level":"INFO","message":"deploy accepted","job":"50b530bfa9fd7387c67181077e64bb80","caller":"admin@example.org","targets":"[\"agent/other\"]","target":"flakelet_relay::relay::api"} agent # [1118746.144717] agent flakelet-agent[1836]: {"timestamp":"2026-09-24T11:57:34.432169Z","level":"INFO","message":"update finished","flakelet":"other","status":"Unchanged","generation":"Some(1)","target":"flakelet_relay::agent::jobs"} relay # [1118746.146248] relay flakelet-relay[400]: {"timestamp":"2026-09-24T11:57:34.433658Z","level":"INFO","message":"deploy finished","job":"50b530bfa9fd7387c67181077e64bb80","ok":true,"target":"flakelet_relay::relay::api"} client: (finished: waiting for success: grep -q 'agent/other: unchanged' /tmp/retry.out, in 4.06 seconds) client: must succeed: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem agents client: (finished: must succeed: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem agents, in 0.02 seconds) client: must succeed: echo '{"groups": []}' | step crypto jwt sign --key /nix/store/4y8k1nvlafnzaadg5frncjjarv7aw3ry-flakelet-relay-test-issuer/jwk.json --iss https://relay:8443 --aud flakelet-relay --sub 'repo:github:example/app:ref:refs/heads/main' --exp $(( $(date +%s) + 300 )) --jti $(head -c8 /dev/urandom | base32) client: (finished: must succeed: echo '{"groups": []}' | step crypto jwt sign --key /nix/store/4y8k1nvlafnzaadg5frncjjarv7aw3ry-flakelet-relay-test-issuer/jwk.json --iss https://relay:8443 --aud flakelet-relay --sub 'repo:github:example/app:ref:refs/heads/main' --exp $(( $(date +%s) + 300 )) --jti $(head -c8 /dev/urandom | base32), in 0.06 seconds) client: must succeed: FLAKELET_RELAY_TOKEN_COMMAND='echo eyJhbGciOiJFZERTQSIsImtpZCI6ImsxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJmbGFrZWxldC1yZWxheSIsImV4cCI6MTc5MDI1MTM1NSwiZ3JvdXBzIjpbXSwiaWF0IjoxNzkwMjUxMDU1LCJpc3MiOiJodHRwczovL3JlbGF5Ojg0NDMiLCJqdGkiOiJVUk5SV1BQQlA2WERRPT09IiwibmJmIjoxNzkwMjUxMDU1LCJzdWIiOiJyZXBvOmdpdGh1YjpleGFtcGxlL2FwcDpyZWY6cmVmcy9oZWFkcy9tYWluIn0.UpmKgLfWrLzicAcKdlbRYvCRQkUdBWwCqbGxgXnCcaNzmTFCIGjPiZKd-Z1kyBXtQJqYyXsM9PAQYv6hHuKIAg' push --relay https://relay:7443 agents client: (finished: must succeed: FLAKELET_RELAY_TOKEN_COMMAND='echo eyJhbGciOiJFZERTQSIsImtpZCI6ImsxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJmbGFrZWxldC1yZWxheSIsImV4cCI6MTc5MDI1MTM1NSwiZ3JvdXBzIjpbXSwiaWF0IjoxNzkwMjUxMDU1LCJpc3MiOiJodHRwczovL3JlbGF5Ojg0NDMiLCJqdGkiOiJVUk5SV1BQQlA2WERRPT09IiwibmJmIjoxNzkwMjUxMDU1LCJzdWIiOiJyZXBvOmdpdGh1YjpleGFtcGxlL2FwcDpyZWY6cmVmcy9oZWFkcy9tYWluIn0.UpmKgLfWrLzicAcKdlbRYvCRQkUdBWwCqbGxgXnCcaNzmTFCIGjPiZKd-Z1kyBXtQJqYyXsM9PAQYv6hHuKIAg' push --relay https://relay:7443 agents, in 0.02 seconds) (finished: subtest: unknown host is final, missing agent is retried, agents listing, in 5.41 seconds) subtest: out-of-band flakelet update reaches the relay without reconnect agent: must succeed: flakelet status --json app | jq '.[0].generation' agent: (finished: must succeed: flakelet status --json app | jq '.[0].generation', in 0.03 seconds) agent: must succeed: journalctl -u flakelet-agent -o cat | grep -c '"connected"' || true agent: (finished: must succeed: journalctl -u flakelet-agent -o cat | grep -c '"connected"' || true, in 0.01 seconds) agent: must succeed: jq '.services.app.prebuilt="/nix/store/1hhsfvlan5s6l308sr1yx2i20rkz1mdc-flakelet-app-v2"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json agent: (finished: must succeed: jq '.services.app.prebuilt="/nix/store/1hhsfvlan5s6l308sr1yx2i20rkz1mdc-flakelet-app-v2"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json, in 0.01 seconds) agent: must succeed: flakelet update app app: using prebuilt artifact /nix/store/1hhsfvlan5s6l308sr1yx2i20rkz1mdc-flakelet-app-v2 app: activating generation 5 agent # [1118746.883130] agent systemd[1]: Stopping app slow2... agent # [1118746.884839] agent systemd[1]: app.service: Deactivated successfully. agent # [1118746.885132] agent systemd[1]: Stopped app slow2. agent # [1118746.901529] agent systemd[1]: Reload requested from client PID 2288 ('systemctl')... agent # [1118746.901600] agent systemd[1]: Reloading... agent # [1118747.251946] agent systemd[1]: Reloading finished in 349 ms. agent # [1118747.352754] agent systemd[1]: Reload requested from client PID 2317 ('systemctl')... agent # [1118747.352794] agent systemd[1]: Reloading... agent # [1118747.684392] agent systemd[1]: Reloading finished in 331 ms. agent # [1118747.770795] agent systemd[1]: Reload requested from client PID 2345 ('systemctl')... agent # [1118747.770837] agent systemd[1]: Reloading... agent # [1118748.108366] agent systemd[1]: Reloading finished in 337 ms. agent # [1118748.196741] agent systemd[1]: Reload requested from client PID 2374 ('systemctl')... agent # [1118748.196782] agent systemd[1]: Reloading... agent # [1118748.530126] agent systemd[1]: Reloading finished in 332 ms. agent: (finished: must succeed: flakelet update app, in 1.81 seconds) client: waiting for success: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem agents | grep -q 'app@5' agent # [1118748.617761] agent systemd[1]: Starting app v2... agent # [1118748.654301] agent systemd[1]: Started app v2. client: (finished: waiting for success: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem agents | grep -q 'app@5', in 2.08 seconds) agent: must succeed: journalctl -u flakelet-agent -o cat | grep -c '"connected"' || true agent: (finished: must succeed: journalctl -u flakelet-agent -o cat | grep -c '"connected"' || true, in 0.01 seconds) client: waiting for success: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem jobs | grep local-app-5- client: (finished: waiting for success: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem jobs | grep local-app-5-, in 0.02 seconds) client: must succeed: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem jobs | grep local-app- client: (finished: must succeed: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem jobs | grep local-app-, in 0.02 seconds) agent: must succeed: jq '.services.app.prebuilt="/nix/store/cqj0khcw75fy7djy7y554bcgxi50911r-flakelet-app-v1"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json agent: (finished: must succeed: jq '.services.app.prebuilt="/nix/store/cqj0khcw75fy7djy7y554bcgxi50911r-flakelet-app-v1"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json, in 0.01 seconds) agent: must succeed: flakelet update app app: using prebuilt artifact /nix/store/cqj0khcw75fy7djy7y554bcgxi50911r-flakelet-app-v1 app: activating generation 6 agent # [1118750.654548] agent flakelet-agent[1836]: {"timestamp":"2026-09-24T11:57:38.941959Z","level":"INFO","message":"local change","flakelet":"app","generation":5,"by":"root (shell)","target":"flakelet_relay::agent::jobs"} agent # [1118750.845050] agent systemd[1]: Stopping app v2... agent # [1118750.845586] agent systemd[1]: app.service: Deactivated successfully. agent # [1118750.845852] agent systemd[1]: Stopped app v2. agent # [1118750.860036] agent systemd[1]: Reload requested from client PID 2424 ('systemctl')... agent # [1118750.860077] agent systemd[1]: Reloading... agent # [1118751.196041] agent systemd[1]: Reloading finished in 335 ms. agent # [1118751.295934] agent systemd[1]: Reload requested from client PID 2453 ('systemctl')... agent # [1118751.295973] agent systemd[1]: Reloading... agent # [1118751.626403] agent systemd[1]: Reloading finished in 330 ms. agent # [1118751.713474] agent systemd[1]: Reload requested from client PID 2481 ('systemctl')... agent # [1118751.713514] agent systemd[1]: Reloading... agent # [1118752.044440] agent systemd[1]: Reloading finished in 330 ms. agent # [1118752.134376] agent systemd[1]: Reload requested from client PID 2510 ('systemctl')... agent # [1118752.134417] agent systemd[1]: Reloading... agent # [1118752.465747] agent systemd[1]: Reloading finished in 330 ms. agent: (finished: must succeed: flakelet update app, in 1.78 seconds) client: waiting for success: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem agents | grep -q 'app@6' agent # [1118752.544812] agent systemd[1]: Starting app v1... agent # [1118752.581202] agent systemd[1]: Started app v1. agent # [1118752.647237] agent flakelet-agent[1836]: {"timestamp":"2026-09-24T11:57:40.934667Z","level":"INFO","message":"local change","flakelet":"app","generation":6,"by":"root (shell)","target":"flakelet_relay::agent::jobs"} client: (finished: waiting for success: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem agents | grep -q 'app@6', in 1.05 seconds) (finished: subtest: out-of-band flakelet update reaches the relay without reconnect, in 6.85 seconds) subtest: certificate not listed under agents cannot connect as agent client: must succeed: curl -s -o /dev/null -w '%{http_code}' --cacert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/ca.pem --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/mallory/cert.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/mallory/key.pem -H 'Upgrade: websocket' -H 'Connection: Upgrade' -H 'Sec-WebSocket-Version: 13' -H 'Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==' https://relay:7443/v1/agent client: (finished: must succeed: curl -s -o /dev/null -w '%{http_code}' --cacert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/ca.pem --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/mallory/cert.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/mallory/key.pem -H 'Upgrade: websocket' -H 'Connection: Upgrade' -H 'Sec-WebSocket-Version: 13' -H 'Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==' https://relay:7443/v1/agent, in 0.03 seconds) (finished: subtest: certificate not listed under agents cannot connect as agent, in 0.03 seconds) subtest: metrics relay: must succeed: curl -sf http://127.0.0.1:7400/metrics relay: (finished: must succeed: curl -sf http://127.0.0.1:7400/metrics, in 0.02 seconds) (finished: subtest: metrics, in 0.02 seconds) subtest: agent link drop mid-run replays missed lines once agent: must succeed: jq '.services.app.prebuilt="/nix/store/lb9sxpw9lh5imz78256ba6b43w55pkww-flakelet-app-slow"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json agent: (finished: must succeed: jq '.services.app.prebuilt="/nix/store/lb9sxpw9lh5imz78256ba6b43w55pkww-flakelet-app-slow"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json, in 0.01 seconds) agent: must succeed: date +%s agent: (finished: must succeed: date +%s, in 0.01 seconds) client: must succeed: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem deploy agent/app > /tmp/slow.out 2>&1 & client: (finished: must succeed: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem deploy agent/app > /tmp/slow.out 2>&1 &, in 0.01 seconds) agent: waiting for success: journalctl -u flakelet-relay-job-app.service --since=@1790251061 | grep -q 'app-slow' agent # [1118753.719132] agent flakelet-agent[1836]: {"timestamp":"2026-09-24T11:57:42.006493Z","level":"INFO","message":"start","id":"ac91accd03e100102dd231a1a2b4cd95","flakelet":"app","rule":"admins","relay":"relay","caller_name":"\"admin@example.org\"","target":"flakelet_relay::agent::conn"} agent # [1118753.719986] agent flakelet-agent[1836]: {"timestamp":"2026-09-24T11:57:42.007342Z","level":"INFO","message":"update starting","flakelet":"app","ids":"[\"ac91accd03e100102dd231a1a2b4cd95\"]","target":"flakelet_relay::agent::jobs"} agent # [1118753.761595] agent systemd[1]: Starting flakelet update app (via relay)... agent # [1118753.798295] agent systemd[1]: Started flakelet update app (via relay). agent # [1118753.802910] agent flakelet[2559]: app: using prebuilt artifact /nix/store/lb9sxpw9lh5imz78256ba6b43w55pkww-flakelet-app-slow agent # [1118753.803556] agent flakelet[2559]: app: activating generation 7 agent # [1118753.822180] agent systemd[1]: Stopping app v1... agent # [1118753.822636] agent systemd[1]: app.service: Deactivated successfully. agent # [1118753.822900] agent systemd[1]: Stopped app v1. agent # [1118753.840990] agent systemd[1]: Reload requested from client PID 2566 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [1118753.841074] agent systemd[1]: Reloading... relay # [1118753.663111] relay flakelet-relay[400]: {"timestamp":"2026-09-24T11:57:41.950443Z","level":"INFO","message":"agent rejected: no agents entry matches","principals":"[\"x509:dns:mallory\"]","target":"flakelet_relay::relay::agent_conn"} relay # [1118753.718737] relay flakelet-relay[400]: {"timestamp":"2026-09-24T11:57:42.006175Z","level":"INFO","message":"deploy accepted","job":"11a3ef7c61da4a02f5bbff6b59d558a2","caller":"admin@example.org","targets":"[\"agent/app\"]","target":"flakelet_relay::relay::api"} agent # [1118754.186143] agent systemd[1]: Reloading finished in 344 ms. agent # [1118754.269472] agent systemd[1]: Reload requested from client PID 2596 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [1118754.269513] agent systemd[1]: Reloading... agent: (finished: waiting for success: journalctl -u flakelet-relay-job-app.service --since=@1790251061 | grep -q 'app-slow', in 1.02 seconds) agent: must succeed: for a in $(getent ahosts relay | cut -d' ' -f1 | sort -u); do ip route add unreachable $a; done agent: (finished: must succeed: for a in $(getent ahosts relay | cut -d' ' -f1 | sort -u); do ip route add unreachable $a; done, in 0.02 seconds) relay: must succeed: for a in $(getent ahosts agent | cut -d' ' -f1 | sort -u); do ss -K dst "[$a]"; done relay: (finished: must succeed: for a in $(getent ahosts agent | cut -d' ' -f1 | sort -u); do ss -K dst "[$a]"; done, in 0.06 seconds) agent: must succeed: ss -K dport = 7443 agent: (finished: must succeed: ss -K dport = 7443, in 0.03 seconds) agent: waiting for success: journalctl -u flakelet-relay-job-app.service --since=@1790251061 | grep -q 'updated to generation' agent # [1118754.602341] agent systemd[1]: Reloading finished in 332 ms. agent # [1118754.684727] agent systemd[1]: Reload requested from client PID 2628 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [1118754.684764] agent systemd[1]: Reloading... agent # [1118754.829922] agent flakelet-agent[1836]: {"timestamp":"2026-09-24T11:57:43.117356Z","level":"INFO","message":"disconnected: IO error: Software caused connection abort (os error 103)","relay":"https://relay:7443","target":"flakelet_relay::agent::conn"} relay # [1118754.799821] relay flakelet-relay[400]: {"timestamp":"2026-09-24T11:57:43.087236Z","level":"INFO","message":"agent disconnected: IO error: Software caused connection abort (os error 103)","host":"agent","target":"flakelet_relay::relay::agent_conn"} agent # [1118755.014840] agent systemd[1]: Reloading finished in 329 ms. agent # [1118755.106045] agent systemd[1]: Reload requested from client PID 2674 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [1118755.106083] agent systemd[1]: Reloading... agent # [1118755.437942] agent systemd[1]: Reloading finished in 331 ms. agent # [1118755.539328] agent systemd[1]: Starting app slow... agent # [1118755.832469] agent flakelet-agent[1836]: {"timestamp":"2026-09-24T11:57:44.119907Z","level":"WARN","message":"connect failed: connect relay:7443: No route to host (os error 113)","relay":"https://relay:7443","target":"flakelet_relay::agent::conn"} agent # [1118757.835824] agent flakelet-agent[1836]: {"timestamp":"2026-09-24T11:57:46.123179Z","level":"WARN","message":"connect failed: connect relay:7443: No route to host (os error 113)","relay":"https://relay:7443","target":"flakelet_relay::agent::conn"} agent: (finished: waiting for success: journalctl -u flakelet-relay-job-app.service --since=@1790251061 | grep -q 'updated to generation', in 7.11 seconds) agent: must succeed: for a in $(getent ahosts relay | cut -d' ' -f1 | sort -u); do ip route del unreachable $a; done agent: (finished: must succeed: for a in $(getent ahosts relay | cut -d' ' -f1 | sort -u); do ip route del unreachable $a; done, in 0.02 seconds) client: waiting for success: grep -q 'agent/app: updated' /tmp/slow.out agent # [1118761.635444] agent systemd[1]: Started app slow. agent # [1118761.642415] agent flakelet[2559]: app: updated to generation 7 agent # [1118761.644913] agent systemd[1]: flakelet-relay-job-app.service: Deactivated successfully. agent # [1118761.838250] agent flakelet-agent[1836]: {"timestamp":"2026-09-24T11:57:50.125645Z","level":"WARN","message":"connect failed: connect relay:7443: No route to host (os error 113)","relay":"https://relay:7443","target":"flakelet_relay::agent::conn"} agent # [1118762.675221] agent flakelet-agent[1836]: {"timestamp":"2026-09-24T11:57:50.962637Z","level":"INFO","message":"update finished","flakelet":"app","status":"Updated","generation":"Some(7)","target":"flakelet_relay::agent::jobs"} client: (finished: waiting for success: grep -q 'agent/app: updated' /tmp/slow.out, in 8.09 seconds) client: must succeed: cat /tmp/slow.out client: (finished: must succeed: cat /tmp/slow.out, in 0.01 seconds) (finished: subtest: agent link drop mid-run replays missed lines once, in 16.39 seconds) subtest: agent restart mid-run reattaches to the unit agent: must succeed: jq '.services.app.prebuilt="/nix/store/yqdz0mcr1b2ljpw9vli2zvbmm43j4gr2-flakelet-app-slow2"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json agent: (finished: must succeed: jq '.services.app.prebuilt="/nix/store/yqdz0mcr1b2ljpw9vli2zvbmm43j4gr2-flakelet-app-slow2"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json, in 0.02 seconds) client: must succeed: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem deploy agent/app > /tmp/restart.out 2>&1 & client: (finished: must succeed: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem deploy agent/app > /tmp/restart.out 2>&1 &, in 0.01 seconds) agent: waiting for success: journalctl -u flakelet-relay-job-app.service --since -20s | grep -q 'app-slow2' relay # [1118769.841795] relay flakelet-relay[400]: {"timestamp":"2026-09-24T11:57:58.129208Z","level":"INFO","message":"agent connected","host":"agent","version":"0.1.0","flakelets":"[Named { name: \"app\", generation: Some(7), revision: Some(\"prebuilt:app\") }, Named { name: \"flakelet-agent\", generation: Some(1), revision: Some(\"prebuilt:flakelet-agent\") }, Named { name: \"other\", generation: Some(1), revision: Some(\"prebuilt:other\") }]","target":"flakelet_relay::relay::agent_conn"} agent # [1118769.841695] agent flakelet-agent[1836]: {"timestamp":"2026-09-24T11:57:58.129103Z","level":"INFO","message":"connected","relay":"https://relay:7443","relay_name":"relay","relay_version":"0.1.0","host":"agent","target":"flakelet_relay::agent::conn"} relay # [1118769.843366] relay flakelet-relay[400]: {"timestamp":"2026-09-24T11:57:58.130846Z","level":"INFO","message":"deploy finished","job":"11a3ef7c61da4a02f5bbff6b59d558a2","ok":true,"target":"flakelet_relay::relay::api"} agent # [1118769.842921] agent flakelet-agent[1836]: {"timestamp":"2026-09-24T11:57:58.130409Z","level":"INFO","message":"start","id":"ac91accd03e100102dd231a1a2b4cd95","flakelet":"app","rule":"admins","relay":"relay","caller_name":"\"admin@example.org\"","target":"flakelet_relay::agent::conn"} relay # [1118770.101272] relay flakelet-relay[400]: {"timestamp":"2026-09-24T11:57:58.388593Z","level":"INFO","message":"deploy accepted","job":"a121d4f09d3479b4cb3db45ad5b65c0f","caller":"admin@example.org","targets":"[\"agent/app\"]","target":"flakelet_relay::relay::api"} agent # [1118770.101714] agent flakelet-agent[1836]: {"timestamp":"2026-09-24T11:57:58.389038Z","level":"INFO","message":"start","id":"94c5b24cec9eb2a22e4236d052c26186","flakelet":"app","rule":"admins","relay":"relay","caller_name":"\"admin@example.org\"","target":"flakelet_relay::agent::conn"} agent # [1118770.102321] agent flakelet-agent[1836]: {"timestamp":"2026-09-24T11:57:58.389811Z","level":"INFO","message":"update starting","flakelet":"app","ids":"[\"94c5b24cec9eb2a22e4236d052c26186\"]","target":"flakelet_relay::agent::jobs"} agent # [1118770.145543] agent systemd[1]: Starting flakelet update app (via relay)... agent # [1118770.173385] agent systemd[1]: Started flakelet update app (via relay). agent # [1118770.178999] agent flakelet[2795]: app: using prebuilt artifact /nix/store/yqdz0mcr1b2ljpw9vli2zvbmm43j4gr2-flakelet-app-slow2 agent # [1118770.179789] agent flakelet[2795]: app: activating generation 8 agent # [1118770.198540] agent systemd[1]: Stopping app slow... agent # [1118770.200217] agent systemd[1]: app.service: Deactivated successfully. agent # [1118770.200487] agent systemd[1]: Stopped app slow. agent # [1118770.217512] agent systemd[1]: Reload requested from client PID 2802 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [1118770.217583] agent systemd[1]: Reloading... agent # [1118770.569748] agent systemd[1]: Reloading finished in 351 ms. agent # [1118770.654681] agent systemd[1]: Reload requested from client PID 2836 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [1118770.654730] agent systemd[1]: Reloading... agent # [1118770.992584] agent systemd[1]: Reloading finished in 337 ms. agent: (finished: waiting for success: journalctl -u flakelet-relay-job-app.service --since -20s | grep -q 'app-slow2', in 1.02 seconds) agent: must succeed: systemctl restart flakelet-agent agent # [1118771.077329] agent systemd[1]: Reload requested from client PID 2865 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [1118771.077376] agent systemd[1]: Reloading... agent # [1118771.417365] agent systemd[1]: Reloading finished in 339 ms. agent # [1118771.500522] agent systemd[1]: Stopping flakelet-agent... agent # [1118771.508985] agent systemd[1]: Reload requested from client PID 2900 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [1118771.509032] agent systemd[1]: Reloading... relay # [1118771.500988] relay flakelet-relay[400]: {"timestamp":"2026-09-24T11:57:59.788337Z","level":"INFO","message":"agent disconnected: IO error: peer closed connection without sending TLS close_notify: https://docs.rs/rustls/latest/rustls/manual/_03_howto/index.html#unexpected-eof","host":"agent","target":"flakelet_relay::relay::agent_conn"} agent: (finished: must succeed: systemctl restart flakelet-agent, in 0.91 seconds) agent: must succeed: test -n "$(ls /var/lib/flakelet-agent/jobs)" agent: (finished: must succeed: test -n "$(ls /var/lib/flakelet-agent/jobs)", in 0.01 seconds) client: waiting for success: grep -q 'agent/app: updated' /tmp/restart.out agent # [1118771.856103] agent systemd[1]: Reloading finished in 346 ms. agent # [1118771.931383] agent systemd[1]: flakelet-agent.service: Deactivated successfully. agent # [1118771.931599] agent systemd[1]: Stopped flakelet-agent. agent # [1118771.931615] agent systemd[1]: flakelet-agent.service: Consumed 1.070s CPU time over 29.386s wall clock time. agent # [1118771.950690] agent systemd[1]: Starting app slow2... agent # [1118771.951525] agent systemd[1]: Starting flakelet-agent... agent # [1118772.002390] agent flakelet-agent[2932]: {"timestamp":"2026-09-24T11:58:00.289871Z","level":"INFO","message":"reattaching to running unit","flakelet":"app","ids":"[\"94c5b24cec9eb2a22e4236d052c26186\"]","target":"flakelet_relay::agent::jobs"} agent # [1118772.022102] agent systemd[1]: Started flakelet-agent. agent # [1118772.035618] agent flakelet-agent[2932]: {"timestamp":"2026-09-24T11:58:00.323099Z","level":"INFO","message":"connected","relay":"https://relay:7443","relay_name":"relay","relay_version":"0.1.0","host":"agent","target":"flakelet_relay::agent::conn"} agent # [1118772.036877] agent flakelet-agent[2932]: {"timestamp":"2026-09-24T11:58:00.324378Z","level":"INFO","message":"start","id":"94c5b24cec9eb2a22e4236d052c26186","flakelet":"app","rule":"admins","relay":"relay","caller_name":"\"admin@example.org\"","target":"flakelet_relay::agent::conn"} relay # [1118772.035787] relay flakelet-relay[400]: {"timestamp":"2026-09-24T11:58:00.323203Z","level":"INFO","message":"agent connected","host":"agent","version":"0.1.0","flakelets":"[Named { name: \"app\", generation: Some(7), revision: Some(\"prebuilt:app\") }, Named { name: \"flakelet-agent\", generation: Some(1), revision: Some(\"prebuilt:flakelet-agent\") }, Named { name: \"other\", generation: Some(1), revision: Some(\"prebuilt:other\") }]","target":"flakelet_relay::relay::agent_conn"} agent # [1118778.014158] agent systemd[1]: Started app slow2. agent # [1118778.023809] agent flakelet[2795]: app: updated to generation 8 agent # [1118778.027315] agent systemd[1]: flakelet-relay-job-app.service: Deactivated successfully. relay # [1118778.587758] relay flakelet-relay[400]: {"timestamp":"2026-09-24T11:58:06.875184Z","level":"INFO","message":"deploy finished","job":"a121d4f09d3479b4cb3db45ad5b65c0f","ok":true,"target":"flakelet_relay::relay::api"} agent # [1118778.586574] agent flakelet-agent[2932]: {"timestamp":"2026-09-24T11:58:06.874011Z","level":"INFO","message":"update finished","flakelet":"app","status":"Updated","generation":"Some(8)","target":"flakelet_relay::agent::jobs"} client: (finished: waiting for success: grep -q 'agent/app: updated' /tmp/restart.out, in 7.08 seconds) client: must succeed: cat /tmp/restart.out client: (finished: must succeed: cat /tmp/restart.out, in 0.01 seconds) agent: must succeed: systemctl is-active app.service agent: (finished: must succeed: systemctl is-active app.service, in 0.01 seconds) (finished: subtest: agent restart mid-run reattaches to the unit, in 9.07 seconds) subtest: relay restart mid-run, push resumes via /v1/jobs agent: must succeed: jq '.services.app.prebuilt="/nix/store/lb9sxpw9lh5imz78256ba6b43w55pkww-flakelet-app-slow"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json agent: (finished: must succeed: jq '.services.app.prebuilt="/nix/store/lb9sxpw9lh5imz78256ba6b43w55pkww-flakelet-app-slow"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json, in 0.01 seconds) agent: must succeed: date +%s agent: (finished: must succeed: date +%s, in 0.01 seconds) client: must succeed: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem deploy agent/app > /tmp/resume.out 2>&1 & client: (finished: must succeed: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem deploy agent/app > /tmp/resume.out 2>&1 &, in 0.01 seconds) agent: waiting for success: journalctl -u flakelet-relay-job-app.service --since=@1790251087 | grep -q 'app-slow' agent # [1118779.181337] agent flakelet-agent[2932]: {"timestamp":"2026-09-24T11:58:07.468744Z","level":"INFO","message":"start","id":"463c0da5a0baaf6443971e55beb90d08","flakelet":"app","rule":"admins","relay":"relay","caller_name":"\"admin@example.org\"","target":"flakelet_relay::agent::conn"} agent # [1118779.181955] agent flakelet-agent[2932]: {"timestamp":"2026-09-24T11:58:07.469441Z","level":"INFO","message":"update starting","flakelet":"app","ids":"[\"463c0da5a0baaf6443971e55beb90d08\"]","target":"flakelet_relay::agent::jobs"} agent # [1118779.221629] agent systemd[1]: Starting flakelet update app (via relay)... agent # [1118779.257361] agent systemd[1]: Started flakelet update app (via relay). agent # [1118779.261668] agent flakelet[3377]: app: using prebuilt artifact /nix/store/lb9sxpw9lh5imz78256ba6b43w55pkww-flakelet-app-slow agent # [1118779.262303] agent flakelet[3377]: app: activating generation 9 agent # [1118779.281462] agent systemd[1]: Stopping app slow2... agent # [1118779.281920] agent systemd[1]: app.service: Deactivated successfully. agent # [1118779.282201] agent systemd[1]: Stopped app slow2. agent # [1118779.300820] agent systemd[1]: Reload requested from client PID 3384 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [1118779.300892] agent systemd[1]: Reloading... relay # [1118779.180938] relay flakelet-relay[400]: {"timestamp":"2026-09-24T11:58:07.468373Z","level":"INFO","message":"deploy accepted","job":"750ee61ffac0c2f6a8d99d70c270513a","caller":"admin@example.org","targets":"[\"agent/app\"]","target":"flakelet_relay::relay::api"} agent # [1118779.651430] agent systemd[1]: Reloading finished in 349 ms. agent # [1118779.735335] agent systemd[1]: Reload requested from client PID 3414 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [1118779.735373] agent systemd[1]: Reloading... agent: (finished: waiting for success: journalctl -u flakelet-relay-job-app.service --since=@1790251087 | grep -q 'app-slow', in 1.02 seconds) relay: must succeed: systemctl restart flakelet-relay agent # [1118780.070570] agent systemd[1]: Reloading finished in 334 ms. agent # [1118780.154898] agent systemd[1]: Reload requested from client PID 3445 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [1118780.154935] agent systemd[1]: Reloading... agent # [1118780.207391] agent flakelet-agent[2932]: {"timestamp":"2026-09-24T11:58:08.494847Z","level":"INFO","message":"disconnected: IO error: peer closed connection without sending TLS close_notify: https://docs.rs/rustls/latest/rustls/manual/_03_howto/index.html#unexpected-eof","relay":"https://relay:7443","target":"flakelet_relay::agent::conn"} relay # [1118780.206888] relay systemd[1]: Stopping flakelet-relay... relay # [1118780.255764] relay systemd[1]: flakelet-relay.service: Deactivated successfully. relay # [1118780.256167] relay systemd[1]: Stopped flakelet-relay. relay # [1118780.256204] relay systemd[1]: flakelet-relay.service: Consumed 2.588s CPU time over 1min 8.222s wall clock time. relay # [1118780.292474] relay systemd[1]: Starting flakelet-relay... agent # [1118780.510111] agent systemd[1]: Reloading finished in 354 ms. agent # [1118780.599714] agent systemd[1]: Reload requested from client PID 3481 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [1118780.599751] agent systemd[1]: Reloading... agent # [1118780.940068] agent systemd[1]: Reloading finished in 339 ms. agent # [1118781.028522] agent systemd[1]: Starting app slow... agent # [1118781.210614] agent flakelet-agent[2932]: {"timestamp":"2026-09-24T11:58:09.497883Z","level":"WARN","message":"connect failed: connect relay:7443: Connection refused (os error 111)","relay":"https://relay:7443","target":"flakelet_relay::agent::conn"} relay: (finished: must succeed: systemctl restart flakelet-relay, in 1.54 seconds) client: waiting for success: grep -q '» ok' /tmp/resume.out relay # [1118781.732519] relay flakelet-relay[932]: {"timestamp":"2026-09-24T11:58:10.019934Z","level":"INFO","message":"listening on 127.0.0.1:7400","target":"flakelet_relay::relay::server"} relay # [1118781.733332] relay flakelet-relay[932]: {"timestamp":"2026-09-24T11:58:10.020375Z","level":"INFO","message":"listening on [::]:7443","target":"flakelet_relay::relay::server"} relay # [1118781.733080] relay systemd[1]: Started flakelet-relay. agent # [1118783.222810] agent flakelet-agent[2932]: {"timestamp":"2026-09-24T11:58:11.510223Z","level":"INFO","message":"connected","relay":"https://relay:7443","relay_name":"relay","relay_version":"0.1.0","host":"agent","target":"flakelet_relay::agent::conn"} relay # [1118783.223087] relay flakelet-relay[932]: {"timestamp":"2026-09-24T11:58:11.510465Z","level":"INFO","message":"agent connected","host":"agent","version":"0.1.0","flakelets":"[Named { name: \"app\", generation: Some(8), revision: Some(\"prebuilt:app\") }, Named { name: \"flakelet-agent\", generation: Some(1), revision: Some(\"prebuilt:flakelet-agent\") }, Named { name: \"other\", generation: Some(1), revision: Some(\"prebuilt:other\") }]","target":"flakelet_relay::relay::agent_conn"} agent # [1118787.105548] agent systemd[1]: Started app slow. agent # [1118787.115093] agent flakelet[3377]: app: updated to generation 9 agent # [1118787.118544] agent systemd[1]: flakelet-relay-job-app.service: Deactivated successfully. relay # [1118787.227118] relay flakelet-relay[932]: {"timestamp":"2026-09-24T11:58:15.514563Z","level":"INFO","message":"job reattached","job":"750ee61ffac0c2f6a8d99d70c270513a","targets":"[\"agent/app\"]","target":"flakelet_relay::relay::api"} agent # [1118788.186432] agent flakelet-agent[2932]: {"timestamp":"2026-09-24T11:58:16.473846Z","level":"INFO","message":"update finished","flakelet":"app","status":"Updated","generation":"Some(9)","target":"flakelet_relay::agent::jobs"} client: (finished: waiting for success: grep -q '» ok' /tmp/resume.out, in 7.09 seconds) client: must succeed: cat /tmp/resume.out client: (finished: must succeed: cat /tmp/resume.out, in 0.01 seconds) (finished: subtest: relay restart mid-run, push resumes via /v1/jobs, in 9.70 seconds) subtest: job history comes from the agents and survives a relay restart client: must succeed: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem deploy --id hist1 agent/other 2>&1 relay # [1118788.860043] relay flakelet-relay[932]: {"timestamp":"2026-09-24T11:58:17.147454Z","level":"INFO","message":"deploy accepted","job":"188b06c4636b65849b3ba8daac22dc7c","caller":"admin@example.org","targets":"[\"agent/other\"]","target":"flakelet_relay::relay::api"} agent # [1118788.860448] agent flakelet-agent[2932]: {"timestamp":"2026-09-24T11:58:17.147851Z","level":"INFO","message":"start","id":"5be20c693e5c0b60e3f85779794dfb86","flakelet":"other","rule":"admins","relay":"relay","caller_name":"\"admin@example.org\"","target":"flakelet_relay::agent::conn"} agent # [1118788.861186] agent flakelet-agent[2932]: {"timestamp":"2026-09-24T11:58:17.148669Z","level":"INFO","message":"update starting","flakelet":"other","ids":"[\"5be20c693e5c0b60e3f85779794dfb86\"]","target":"flakelet_relay::agent::jobs"} agent # [1118788.903997] agent systemd[1]: Starting flakelet update other (via relay)... agent # [1118788.941379] agent systemd[1]: Started flakelet update other (via relay). agent # [1118788.947474] agent flakelet[3555]: other: using prebuilt artifact /nix/store/h4k7d4dqcfa4h9d48c0rm4x08qziqfaf-flakelet-other-v1 agent # [1118788.948096] agent flakelet[3555]: other: up to date agent # [1118788.948983] agent systemd[1]: flakelet-relay-job-other.service: Deactivated successfully. client: (finished: must succeed: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem deploy --id hist1 agent/other 2>&1, in 0.65 seconds) relay: must succeed: systemctl restart flakelet-relay agent # [1118789.488344] agent flakelet-agent[2932]: {"timestamp":"2026-09-24T11:58:17.775759Z","level":"INFO","message":"update finished","flakelet":"other","status":"Unchanged","generation":"Some(1)","target":"flakelet_relay::agent::jobs"} agent # [1118789.507149] agent flakelet-agent[2932]: {"timestamp":"2026-09-24T11:58:17.794623Z","level":"INFO","message":"disconnected: IO error: peer closed connection without sending TLS close_notify: https://docs.rs/rustls/latest/rustls/manual/_03_howto/index.html#unexpected-eof","relay":"https://relay:7443","target":"flakelet_relay::agent::conn"} relay # [1118789.489794] relay flakelet-relay[932]: {"timestamp":"2026-09-24T11:58:17.777239Z","level":"INFO","message":"deploy finished","job":"188b06c4636b65849b3ba8daac22dc7c","ok":true,"target":"flakelet_relay::relay::api"} relay # [1118789.506870] relay systemd[1]: Stopping flakelet-relay... relay # [1118789.544862] relay systemd[1]: flakelet-relay.service: Deactivated successfully. relay # [1118789.545374] relay systemd[1]: Stopped flakelet-relay. relay # [1118789.545421] relay systemd[1]: flakelet-relay.service: Consumed 2.110s CPU time over 9.252s wall clock time. relay # [1118789.581093] relay systemd[1]: Starting flakelet-relay... agent # [1118790.509605] agent flakelet-agent[2932]: {"timestamp":"2026-09-24T11:58:18.797008Z","level":"WARN","message":"connect failed: connect relay:7443: Connection refused (os error 111)","relay":"https://relay:7443","target":"flakelet_relay::agent::conn"} relay: (finished: must succeed: systemctl restart flakelet-relay, in 1.52 seconds) relay: waiting for success: curl -sf http://127.0.0.1:7400/metrics | grep -q 'agent_up{host="agent"} 1' relay # [1118791.010944] relay flakelet-relay[1322]: {"timestamp":"2026-09-24T11:58:19.298356Z","level":"INFO","message":"listening on 127.0.0.1:7400","target":"flakelet_relay::relay::server"} relay # [1118791.011747] relay flakelet-relay[1322]: {"timestamp":"2026-09-24T11:58:19.298865Z","level":"INFO","message":"listening on [::]:7443","target":"flakelet_relay::relay::server"} relay # [1118791.011481] relay systemd[1]: Started flakelet-relay. agent # [1118792.524638] agent flakelet-agent[2932]: {"timestamp":"2026-09-24T11:58:20.812054Z","level":"INFO","message":"connected","relay":"https://relay:7443","relay_name":"relay","relay_version":"0.1.0","host":"agent","target":"flakelet_relay::agent::conn"} relay # [1118792.524779] relay flakelet-relay[1322]: {"timestamp":"2026-09-24T11:58:20.812182Z","level":"INFO","message":"agent connected","host":"agent","version":"0.1.0","flakelets":"[Named { name: \"app\", generation: Some(9), revision: Some(\"prebuilt:app\") }, Named { name: \"flakelet-agent\", generation: Some(1), revision: Some(\"prebuilt:flakelet-agent\") }, Named { name: \"other\", generation: Some(1), revision: Some(\"prebuilt:other\") }]","target":"flakelet_relay::relay::agent_conn"} relay: (finished: waiting for success: curl -sf http://127.0.0.1:7400/metrics | grep -q 'agent_up{host="agent"} 1', in 2.06 seconds) client: must succeed: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem jobs client: (finished: must succeed: push --relay https://relay:7443 --cert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin.pem --key /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/admin-key.pem jobs, in 0.02 seconds) client: must succeed: echo '{"groups": [], "repository": "example/app"}' | step crypto jwt sign --key /nix/store/4y8k1nvlafnzaadg5frncjjarv7aw3ry-flakelet-relay-test-issuer/jwk.json --iss https://relay:8443 --aud flakelet-relay --sub 'repo:github:example/app:ref:refs/heads/main' --exp $(( $(date +%s) + 300 )) --jti $(head -c8 /dev/urandom | base32) client: (finished: must succeed: echo '{"groups": [], "repository": "example/app"}' | step crypto jwt sign --key /nix/store/4y8k1nvlafnzaadg5frncjjarv7aw3ry-flakelet-relay-test-issuer/jwk.json --iss https://relay:8443 --aud flakelet-relay --sub 'repo:github:example/app:ref:refs/heads/main' --exp $(( $(date +%s) + 300 )) --jti $(head -c8 /dev/urandom | base32), in 0.06 seconds) client: must succeed: FLAKELET_RELAY_TOKEN_COMMAND='echo eyJhbGciOiJFZERTQSIsImtpZCI6ImsxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJmbGFrZWxldC1yZWxheSIsImV4cCI6MTc5MDI1MTQwMSwiZ3JvdXBzIjpbXSwiaWF0IjoxNzkwMjUxMTAxLCJpc3MiOiJodHRwczovL3JlbGF5Ojg0NDMiLCJqdGkiOiIzTVRKTlBJUzJWRFdNPT09IiwibmJmIjoxNzkwMjUxMTAxLCJyZXBvc2l0b3J5IjoiZXhhbXBsZS9hcHAiLCJzdWIiOiJyZXBvOmdpdGh1YjpleGFtcGxlL2FwcDpyZWY6cmVmcy9oZWFkcy9tYWluIn0.lINMgbb0MANNMmTT1tSzLs1P29Ai55_vuUBCOJ5YDCV6-xd6TdU0n84tST_WPYddDDx4i121ejlWtumQgO_vAg' push --relay https://relay:7443 deploy --id hist2 agent/app » job 6a239dbf87cfec796e49404c351d787f via relay 0.1.0 (agents: agent 0.1.0) » wave 0 agent/app | Starting flakelet update app (via relay)... agent/app | Started flakelet update app (via relay). agent/app | app: using prebuilt artifact /nix/store/lb9sxpw9lh5imz78256ba6b43w55pkww-flakelet-app-slow agent/app | app: up to date agent # [1118793.170757] agent flakelet-agent[2932]: {"timestamp":"2026-09-24T11:58:21.458195Z","level":"INFO","message":"start","id":"dce6faec8455f85321e72ee634888126","flakelet":"app","rule":"ci","relay":"relay","caller_name":"\"example/app\"","target":"flakelet_relay::agent::conn"} agent # [1118793.171388] agent flakelet-agent[2932]: {"timestamp":"2026-09-24T11:58:21.458880Z","level":"INFO","message":"update starting","flakelet":"app","ids":"[\"dce6faec8455f85321e72ee634888126\"]","target":"flakelet_relay::agent::jobs"} agent # [1118793.209944] agent systemd[1]: Starting flakelet update app (via relay)... agent # [1118793.237159] agent systemd[1]: Started flakelet update app (via relay). agent # [1118793.242240] agent flakelet[3583]: app: using prebuilt artifact /nix/store/lb9sxpw9lh5imz78256ba6b43w55pkww-flakelet-app-slow agent # [1118793.242898] agent flakelet[3583]: app: up to date agent # [1118793.245331] agent systemd[1]: flakelet-relay-job-app.service: Deactivated successfully. agent/app | flakelet-relay-job-app.service: Deactivated successfully. relay # [1118793.170301] relay flakelet-relay[1322]: {"timestamp":"2026-09-24T11:58:21.457727Z","level":"INFO","message":"deploy accepted","job":"6a239dbf87cfec796e49404c351d787f","caller":"example/app","targets":"[\"agent/app\"]","target":"flakelet_relay::relay::api"} » agent/app: unchanged (generation 9) » ok client: (finished: must succeed: FLAKELET_RELAY_TOKEN_COMMAND='echo eyJhbGciOiJFZERTQSIsImtpZCI6ImsxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJmbGFrZWxldC1yZWxheSIsImV4cCI6MTc5MDI1MTQwMSwiZ3JvdXBzIjpbXSwiaWF0IjoxNzkwMjUxMTAxLCJpc3MiOiJodHRwczovL3JlbGF5Ojg0NDMiLCJqdGkiOiIzTVRKTlBJUzJWRFdNPT09IiwibmJmIjoxNzkwMjUxMTAxLCJyZXBvc2l0b3J5IjoiZXhhbXBsZS9hcHAiLCJzdWIiOiJyZXBvOmdpdGh1YjpleGFtcGxlL2FwcDpyZWY6cmVmcy9oZWFkcy9tYWluIn0.lINMgbb0MANNMmTT1tSzLs1P29Ai55_vuUBCOJ5YDCV6-xd6TdU0n84tST_WPYddDDx4i121ejlWtumQgO_vAg' push --relay https://relay:7443 deploy --id hist2 agent/app, in 0.64 seconds) client: must succeed: FLAKELET_RELAY_TOKEN_COMMAND='echo eyJhbGciOiJFZERTQSIsImtpZCI6ImsxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJmbGFrZWxldC1yZWxheSIsImV4cCI6MTc5MDI1MTQwMSwiZ3JvdXBzIjpbXSwiaWF0IjoxNzkwMjUxMTAxLCJpc3MiOiJodHRwczovL3JlbGF5Ojg0NDMiLCJqdGkiOiIzTVRKTlBJUzJWRFdNPT09IiwibmJmIjoxNzkwMjUxMTAxLCJyZXBvc2l0b3J5IjoiZXhhbXBsZS9hcHAiLCJzdWIiOiJyZXBvOmdpdGh1YjpleGFtcGxlL2FwcDpyZWY6cmVmcy9oZWFkcy9tYWluIn0.lINMgbb0MANNMmTT1tSzLs1P29Ai55_vuUBCOJ5YDCV6-xd6TdU0n84tST_WPYddDDx4i121ejlWtumQgO_vAg' push --relay https://relay:7443 jobs client: (finished: must succeed: FLAKELET_RELAY_TOKEN_COMMAND='echo eyJhbGciOiJFZERTQSIsImtpZCI6ImsxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJmbGFrZWxldC1yZWxheSIsImV4cCI6MTc5MDI1MTQwMSwiZ3JvdXBzIjpbXSwiaWF0IjoxNzkwMjUxMTAxLCJpc3MiOiJodHRwczovL3JlbGF5Ojg0NDMiLCJqdGkiOiIzTVRKTlBJUzJWRFdNPT09IiwibmJmIjoxNzkwMjUxMTAxLCJyZXBvc2l0b3J5IjoiZXhhbXBsZS9hcHAiLCJzdWIiOiJyZXBvOmdpdGh1YjpleGFtcGxlL2FwcDpyZWY6cmVmcy9oZWFkcy9tYWluIn0.lINMgbb0MANNMmTT1tSzLs1P29Ai55_vuUBCOJ5YDCV6-xd6TdU0n84tST_WPYddDDx4i121ejlWtumQgO_vAg' push --relay https://relay:7443 jobs, in 0.02 seconds) (finished: subtest: job history comes from the agents and survives a relay restart, in 4.97 seconds) subtest: dashboard login via authorization code flow client: must succeed: curl -sS --cacert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -o /dev/null -w '%{http_code} %{redirect_url}' https://relay:7443/ui/ | grep -q '303 https://relay:7443/ui/login' client: (finished: must succeed: curl -sS --cacert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -o /dev/null -w '%{http_code} %{redirect_url}' https://relay:7443/ui/ | grep -q '303 https://relay:7443/ui/login', in 0.02 seconds) client: must succeed: curl -sS --cacert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -L https://relay:7443/ui/login client: (finished: must succeed: curl -sS --cacert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -L https://relay:7443/ui/login, in 0.09 seconds) client: must succeed: curl -sS --cacert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar https://relay:7443/ui/jobs client: (finished: must succeed: curl -sS --cacert /nix/store/1mmby7f9fvxprhygz7w9j1i4ks3vf5ms-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar https://relay:7443/ui/jobs, in 0.02 seconds) !!! Test "dashboard login via authorization code flow" failed with error: "" cleanup kill NspawnMachine (pid 52) kill NspawnMachine (pid 53) Container agent terminated by signal KILL. relay # [1118793.788480] relay flakelet-relay[1322]: {"timestamp":"2026-09-24T11:58:22.075915Z","level":"INFO","message":"deploy finished","job":"6a239dbf87cfec796e49404c351d787f","ok":true,"target":"flakelet_relay::relay::api"} relay # [1118793.864678] relay 7gsx6vaqmkb2vf8rr0w4y1g018hi9d43-device-mock[185]: 127.0.0.1 - - [24/Sep/2026 11:58:22] "GET /authorize?response_type=code&client_id=dashboard&redirect_uri=https%3A%2F%2Frelay%3A7443%2Fui%2Fcallback&scope=openid%20profile%20email%20groups&state=vWfvZLp8qwDvfD0IBDDJEn0YHMasVLtww-yDTP23VDQ&code_challenge=YW2Unp543V1T3NPLETFFxZhwb2lL9KAtlDxowwo2L8w&code_challenge_method=S256 HTTP/1.1" 302 - relay # [1118793.916767] relay 7gsx6vaqmkb2vf8rr0w4y1g018hi9d43-device-mock[185]: 127.0.0.1 - - [24/Sep/2026 11:58:22] "POST /token HTTP/1.1" 200 - relay # [1118793.917547] relay flakelet-relay[1322]: {"timestamp":"2026-09-24T11:58:22.205017Z","level":"INFO","message":"login","name":"dev@example.org","principals":"[\"oidc:mock:someone\", \"oidc:mock:groups:wheel\"]","target":"flakelet_relay::relay::login"} kill NspawnMachine (pid 55) Container client terminated by signal KILL. (finished: cleanup, in 0.34 seconds) additionally exposed symbols: agent, client, relay, vlan1, start_all, test_script, machines, machines_qemu, machines_nspawn, vlans, driver, log, os, create_machine, subtest, run_tests, join_all, retry, serial_stdout_off, serial_stdout_on, polling_condition, BaseMachine, QemuMachine, NspawnMachine, t, debug, dump_machine_ssh » job 1871c3fd17edfdfcb76aed332435f41d via relay 0.1.0 (agents: agent 0.1.0) » wave 0 agent/app | Starting flakelet update app (via relay)... agent/app | Started flakelet update app (via relay). agent/app | app: using prebuilt artifact /nix/store/1hhsfvlan5s6l308sr1yx2i20rkz1mdc-flakelet-app-v2 agent/app | app: activating generation 2 agent/app | app: updated to generation 2 agent/app | flakelet-relay-job-app.service: Deactivated successfully. » agent/app: updated (generation 2) » ok » job 2feedd7e91a8f353c42fa3ee97c25d43 via relay 0.1.0 (agents: agent 0.1.0) » wave 0 agent/app | Starting flakelet update app (via relay)... agent/app | Started flakelet update app (via relay). agent/app | app: using prebuilt artifact /nix/store/qy8h9kkvfrxq0bc47vwjy6lfnk9i95ga-flakelet-app-broken agent/app | app: activating generation 4 agent/app | app: deploy failed, rolled back: systemctl start app.service failed: agent/app | failed units: app.service. See journalctl -u app.service agent/app | flakelet-relay-job-app.service: Main process exited, code=exited, status=1/FAILURE agent/app | flakelet-relay-job-app.service: Failed with result 'exit-code'. » agent/app: rolled-back (generation 3) agent/app │ Starting app v1... agent/app │ Started app v1. agent/app │ Stopping app v1... agent/app │ app.service: Deactivated successfully. agent/app │ Stopped app v1. agent/app │ Starting app v2... agent/app │ Started app v2. agent/app │ Stopping app v2... agent/app │ app.service: Deactivated successfully. agent/app │ Stopped app v2. agent/app │ Starting app v1... agent/app │ Started app v1. agent/app │ Stopping app v1... agent/app │ app.service: Deactivated successfully. agent/app │ Stopped app v1. agent/app │ Starting app broken... agent/app │ app.service: Unable to locate executable '/nonexistent': No such file or directory agent/app │ app.service: Failed at step EXEC spawning /nonexistent: No such file or directory agent/app │ app.service: Main process exited, code=exited, status=203/EXEC agent/app │ app.service: Failed with result 'exit-code'. agent/app │ Failed to start app broken. agent/app │ Starting app v1... agent/app │ Started app v1. » skipped: agent/other » failed » job 11a3ef7c61da4a02f5bbff6b59d558a2 via relay 0.1.0 (agents: agent 0.1.0) » wave 0 agent/app | Starting flakelet update app (via relay)... agent/app | Started flakelet update app (via relay). agent/app | app: using prebuilt artifact /nix/store/lb9sxpw9lh5imz78256ba6b43w55pkww-flakelet-app-slow agent/app | app: activating generation 7 agent/app | app: updated to generation 7 agent/app | flakelet-relay-job-app.service: Deactivated successfully. » agent/app: updated (generation 7) » ok » job a121d4f09d3479b4cb3db45ad5b65c0f via relay 0.1.0 (agents: agent 0.1.0) » wave 0 agent/app | Starting flakelet update app (via relay)... agent/app | Started flakelet update app (via relay). agent/app | app: using prebuilt artifact /nix/store/yqdz0mcr1b2ljpw9vli2zvbmm43j4gr2-flakelet-app-slow2 agent/app | app: activating generation 8 agent/app | app: updated to generation 8 agent/app | flakelet-relay-job-app.service: Deactivated successfully. » agent/app: updated (generation 8) » ok » job 750ee61ffac0c2f6a8d99d70c270513a via relay 0.1.0 (agents: agent 0.1.0) » wave 0 agent/app | Starting flakelet update app (via relay)... agent/app | Started flakelet update app (via relay). agent/app | app: using prebuilt artifact /nix/store/lb9sxpw9lh5imz78256ba6b43w55pkww-flakelet-app-slow agent/app | app: activating generation 9 flakelet-push: stream: error reading a body from connection, resuming job ba06e9013ed242407d983748a5eb08fe flakelet-push: https://relay:7443: connect relay:7443: Connection refused (os error 111), retrying in 1s flakelet-push: https://relay:7443: connect relay:7443: Connection refused (os error 111), retrying in 2s flakelet-push: https://relay:7443: 404 Not Found: unknown_job: no connected agent knows this job, retrying in 4s » job 750ee61ffac0c2f6a8d99d70c270513a via relay 0.1.0 (agents: agent 0.1.0) » wave 0 agent/app | app: updated to generation 9 agent/app | flakelet-relay-job-app.service: Deactivated successfully. » agent/app: updated (generation 9) » ok 1790251097 hist1 admin@example.org agent/other:unchanged 1790251087 ba06e9013ed242407d983748a5eb08fe admin@example.org agent/app:updated 1790251078 6416dc9854ff34b089f74a6bb79fdc3a admin@example.org agent/app:updated 1790251062 19a8da5978cf2a08888bed115cdcd869 admin@example.org agent/app:updated 1790251060 local-app-6-1790251060 root (shell) agent/app:updated 1790251056 local-app-5-1790251056 root (shell) agent/app:updated 1790251052 e1526e36a54fa5b10badfed7a0340029 admin@example.org agent/other:unchanged 1790251041 2a1723a12e9a30fb0c0543aa8b50a943 admin@example.org agent/app:unchanged 1790251041 36adedcd03ad08b6ca3aebdb619f23b6 admin@example.org agent/app:updated 1790251040 574fd2a0e6c176b93b19ffcde65edf9b admin@example.org agent/app:unchanged 1790251039 0380202514fd490c030680d7ee2f2a83 admin@example.org agent/app:unchanged 1790251035 8fda3fab2737ba048367e777c6fffb3b admin@example.org agent/app:rolled-back 1790251034 70d9ed21da305f4b22d4267326a7352e dev@example.org agent/other:unchanged 1790251034 b1f1bda563c13fa1a0977dffff4a9f37 dev@example.org agent/other:unchanged 1790251031 1e80f73d1cbf36e77e9eb96bd80d74d4 someone agent/other:unchanged 1790251028 a88ee4187cc3aed4b9f43615d3569972 repo:github:example/app:ref:refs/heads/main agent/app:updated 1790251025 f46c37388dded28a93e329ce6c334a4a admin@example.org agent/app:updated 1790251022 local-app-1-1790251022 host activation agent/app:updated 1790251022 local-flakelet-agent-1-1790251022 host activation agent/flakelet-agent:updated 1790251022 local-other-1-1790251022 host activation agent/other:updated Traceback (most recent call last): File "/nix/store/1pfwbz29g35j2afnmzzhnp99xjq1kqx9-nixos-test-driver-1.1/bin/.nixos-test-driver-wrapped", line 9, in sys.exit(main()) ~~~~^^ File "/nix/store/1pfwbz29g35j2afnmzzhnp99xjq1kqx9-nixos-test-driver-1.1/lib/python3.14/site-packages/test_driver/__init__.py", line 194, in main driver.run_tests() ~~~~~~~~~~~~~~~~^^ File "/nix/store/1pfwbz29g35j2afnmzzhnp99xjq1kqx9-nixos-test-driver-1.1/lib/python3.14/site-packages/test_driver/driver.py", line 437, in run_tests self.test_script() ~~~~~~~~~~~~~~~~^^ File "/nix/store/1pfwbz29g35j2afnmzzhnp99xjq1kqx9-nixos-test-driver-1.1/lib/python3.14/site-packages/test_driver/driver.py", line 395, in test_script exec(self.tests, symbols, None) ~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "", line 208, in AssertionError Container relay terminated by signal KILL.