tribuchet: building on jamie Machine state will be reset. To keep it, pass --keep-machine-state start all VLans (finished: start all VLans, in 0.00 seconds) Initializing machine ID from random generator. Test will time out and terminate in 3600.0 seconds run the VM test script start all VMs agent: systemd-nspawn running (pid 52) agent: Waiting for journal at /build/vm-state-agent/var/log/journal... client: systemd-nspawn running (pid 55) relay: systemd-nspawn running (pid 56) client: Waiting for journal at /build/vm-state-client/var/log/journal... relay: Waiting for journal at /build/vm-state-relay/var/log/journal... (finished: start all VMs, in 0.00 seconds) relay: waiting for unit flakelet-relay.service nixos-nspawn(agent): TAP vde-tap1 not found; container will be isolated from VDE nixos-nspawn(agent): A common reason for this is that /dev/net is not available in the Nix sandbox. Try adding /dev/net to extra-sandbox-paths. nixos-nspawn(relay): TAP vde-tap1 not found; container will be isolated from VDE nixos-nspawn(relay): A common reason for this is that /dev/net is not available in the Nix sandbox. Try adding /dev/net to extra-sandbox-paths. nixos-nspawn(client): TAP vde-tap1 not found; container will be isolated from VDE nixos-nspawn(client): A common reason for this is that /dev/net is not available in the Nix sandbox. Try adding /dev/net to extra-sandbox-paths. Note: in a future version of systemd-nspawn the default set of permitted socket address families will be restricted to AF_INET, AF_INET6 and AF_UNIX. Use --restrict-address-families= to configure the set of permitted socket address families, or set RestrictAddressFamilies= in a .nspawn file. Note: in a future version of systemd-nspawn the default set of permitted socket address families will be restricted to AF_INET, AF_INET6 and AF_UNIX. Use --restrict-address-families= to configure the set of permitted socket address families, or set RestrictAddressFamilies= in a .nspawn file. Note: in a future version of systemd-nspawn the default set of permitted socket address families will be restricted to AF_INET, AF_INET6 and AF_UNIX. Use --restrict-address-families= to configure the set of permitted socket address families, or set RestrictAddressFamilies= in a .nspawn file. ░ Spawning container client on /build/vm-state-client. ░ Spawning container relay on /build/vm-state-relay. ░ Spawning container agent on /build/vm-state-agent. agent # No journal files were found. agent # No journal boot entry found for the specified boot (+0). client # No journal files were found. relay # No journal files were found. client # No journal boot entry found for the specified boot (+0). relay # No journal boot entry found for the specified boot (+0). agent # [ 1421.563467] agent systemd-journald[57]: Journal started agent # [ 1421.563510] agent systemd-journald[57]: Runtime Journal (/run/log/journal/5778cbac68d042d69481c0fd2feac4ed) is 8M, max 4G, 3.9G free. agent # [ 1421.573988] agent systemd[1]: Finished Apply Kernel Variables. agent # [ 1421.585811] agent systemd[1]: Finished Create Static Device Nodes in /dev gracefully. agent # [ 1421.600206] agent systemd[1]: Starting Flush Journal to Persistent Storage... agent # [ 1421.600940] agent systemd[1]: Starting Create Static Device Nodes in /dev... agent # [ 1421.625561] agent systemd-journald[57]: Time spent on flushing to /var/log/journal/5778cbac68d042d69481c0fd2feac4ed is 2.751ms for 6 entries. agent # [ 1421.625561] agent systemd-journald[57]: System Journal (/var/log/journal/5778cbac68d042d69481c0fd2feac4ed) is 512B, max 4G, 3.9G free. agent # [ 1421.636760] agent systemd[1]: Finished Create Static Device Nodes in /dev. agent # [ 1421.637701] agent systemd[1]: Reached target Preparation for Local File Systems. agent # [ 1421.637816] agent systemd[1]: Rule-based Manager for Device Events and Files skipped, unmet condition check ConditionPathIsReadWrite=/sys agent # [ 1421.651120] agent systemd[1]: Finished Flush Journal to Persistent Storage. agent # [ 1421.726437] agent systemd[1]: Finished Firewall. agent # [ 1422.579823] agent systemd[1]: Mounting /run/wrappers... agent # [ 1422.619478] agent systemd[1]: Mounted /run/wrappers. agent # [ 1422.620756] agent systemd[1]: Reached target Local File Systems. agent # [ 1422.622082] agent systemd[1]: Listening on Boot Loader Control Service Socket. client # [ 1421.562654] client systemd-journald[57]: Journal started client # [ 1421.562696] client systemd-journald[57]: Runtime Journal (/run/log/journal/b4b0c4d14f51492fa757adc936d654b6) is 8M, max 4G, 3.9G free. client # [ 1421.574349] client systemd[1]: Finished Apply Kernel Variables. client # [ 1421.586410] client systemd[1]: Finished Create Static Device Nodes in /dev gracefully. client # [ 1421.601166] client systemd[1]: Starting Flush Journal to Persistent Storage... client # [ 1421.601789] client systemd[1]: Starting Create Static Device Nodes in /dev... client # [ 1421.628141] client systemd-journald[57]: Time spent on flushing to /var/log/journal/b4b0c4d14f51492fa757adc936d654b6 is 2.345ms for 6 entries. client # [ 1421.628141] client systemd-journald[57]: System Journal (/var/log/journal/b4b0c4d14f51492fa757adc936d654b6) is 512B, max 4G, 3.9G free. client # [ 1421.636595] client systemd[1]: Finished Create Static Device Nodes in /dev. client # [ 1421.637435] client systemd[1]: Reached target Preparation for Local File Systems. client # [ 1421.637554] client systemd[1]: Rule-based Manager for Device Events and Files skipped, unmet condition check ConditionPathIsReadWrite=/sys client # [ 1421.654490] client systemd[1]: Finished Flush Journal to Persistent Storage. client # [ 1421.722063] client systemd[1]: Finished Firewall. client # [ 1422.579731] client systemd[1]: Mounting /run/wrappers... client # [ 1422.618906] client systemd[1]: Mounted /run/wrappers. client # [ 1422.620201] client systemd[1]: Reached target Local File Systems. client # [ 1422.621540] client systemd[1]: Listening on Boot Loader Control Service Socket. client # [ 1422.622800] client systemd[1]: Starting Create SUID/SGID Wrappers... client # [ 1422.622847] client systemd[1]: Update Boot Loader Random Seed skipped, unmet condition check ConditionVirtualization=!container client # [ 1422.624096] client systemd[1]: Starting Save Transient machine-id to Disk... client # [ 1422.625216] client systemd[1]: Starting Create System Files and Directories... relay # [ 1421.566202] relay systemd-journald[57]: Journal started relay # [ 1421.566242] relay systemd-journald[57]: Runtime Journal (/run/log/journal/315ef82c903f48909d52510bc4652d30) is 8M, max 4G, 3.9G free. relay # [ 1421.577749] relay systemd[1]: Finished Apply Kernel Variables. relay # [ 1421.589661] relay systemd[1]: Finished Create Static Device Nodes in /dev gracefully. relay # [ 1421.603670] relay systemd[1]: Starting Flush Journal to Persistent Storage... relay # [ 1421.604365] relay systemd[1]: Starting Create Static Device Nodes in /dev... relay # [ 1421.628153] relay systemd-journald[57]: Time spent on flushing to /var/log/journal/315ef82c903f48909d52510bc4652d30 is 2.292ms for 6 entries. relay # [ 1421.628153] relay systemd-journald[57]: System Journal (/var/log/journal/315ef82c903f48909d52510bc4652d30) is 512B, max 4G, 3.9G free. relay # [ 1421.635629] relay systemd[1]: Finished Create Static Device Nodes in /dev. relay # [ 1421.636356] relay systemd[1]: Reached target Preparation for Local File Systems. relay # [ 1421.636471] relay systemd[1]: Rule-based Manager for Device Events and Files skipped, unmet condition check ConditionPathIsReadWrite=/sys relay # [ 1421.639107] relay systemd[1]: Finished Flush Journal to Persistent Storage. relay # [ 1421.751590] relay systemd[1]: Finished Firewall. relay # [ 1422.567336] relay systemd[1]: Mounting /run/wrappers... relay # [ 1422.596491] relay systemd[1]: Mounted /run/wrappers. relay # [ 1422.597281] relay systemd[1]: Reached target Local File Systems. relay # [ 1422.598062] relay systemd[1]: Listening on Boot Loader Control Service Socket. relay # [ 1422.599189] relay systemd[1]: Starting Create SUID/SGID Wrappers... relay # [ 1422.599216] relay systemd[1]: Update Boot Loader Random Seed skipped, unmet condition check ConditionVirtualization=!container relay # [ 1422.600093] relay systemd[1]: Starting Save Transient machine-id to Disk... relay # [ 1422.600775] relay systemd[1]: Starting Create System Files and Directories... relay # [ 1422.622540] relay systemd-tmpfiles[169]: Cannot set file attributes for '/var/empty', value=0x00000010, mask=0x00000010, ignoring: Operation not permitted relay # [ 1422.622970] relay systemd-tmpfiles[169]: fchmod() of /var/log/journal failed: Operation not permitted relay # [ 1422.623297] relay systemd-tmpfiles[169]: fchmod() of /var/log/journal/315ef82c903f48909d52510bc4652d30 failed: Operation not permitted relay # [ 1422.623680] relay systemd-tmpfiles[169]: fchmod() of /run/log/journal failed: Operation not permitted relay # [ 1422.625680] relay suid-sgid-wrappers-start[176]: chmod: changing permissions of '/run/wrappers/wrappers.9KEcREW4pN/chsh': Operation not permitted relay # [ 1422.626370] relay systemd[1]: suid-sgid-wrappers.service: Main process exited, code=exited, status=1/FAILURE relay # [ 1422.626501] relay systemd[1]: suid-sgid-wrappers.service: Failed with result 'exit-code'. relay # [ 1422.626664] relay systemd[1]: Failed to start Create SUID/SGID Wrappers. agent # [ 1422.623260] agent systemd[1]: Starting Create SUID/SGID Wrappers... agent # [ 1422.623307] agent systemd[1]: Update Boot Loader Random Seed skipped, unmet condition check ConditionVirtualization=!container agent # [ 1422.624462] agent systemd[1]: Starting Save Transient machine-id to Disk... agent # [ 1422.625339] agent systemd[1]: Starting Create System Files and Directories... agent # [ 1422.645704] agent suid-sgid-wrappers-start[168]: chmod: changing permissions of '/run/wrappers/wrappers.vgQtiCG92E/chsh': Operation not permitted agent # [ 1422.646763] agent systemd-tmpfiles[161]: Cannot set file attributes for '/var/empty', value=0x00000010, mask=0x00000010, ignoring: Operation not permitted agent # [ 1422.647213] agent systemd-tmpfiles[161]: fchmod() of /var/log/journal failed: Operation not permitted agent # [ 1422.647569] agent systemd-tmpfiles[161]: fchmod() of /var/log/journal/5778cbac68d042d69481c0fd2feac4ed failed: Operation not permitted agent # [ 1422.647940] agent systemd-tmpfiles[161]: fchmod() of /run/log/journal failed: Operation not permitted client # [ 1422.648096] client systemd-tmpfiles[161]: Cannot set file attributes for '/var/empty', value=0x00000010, mask=0x00000010, ignoring: Operation not permitted client # [ 1422.648529] client systemd-tmpfiles[161]: fchmod() of /var/log/journal failed: Operation not permitted client # [ 1422.648854] client systemd-tmpfiles[161]: fchmod() of /var/log/journal/b4b0c4d14f51492fa757adc936d654b6 failed: Operation not permitted client # [ 1422.649024] client suid-sgid-wrappers-start[168]: chmod: changing permissions of '/run/wrappers/wrappers.IN4LbtUdkr/chsh': Operation not permitted relay # [ 1422.627164] relay systemd[1]: Finished Create System Files and Directories. relay # [ 1422.642050] relay systemd[1]: Starting Rebuild Journal Catalog... relay # [ 1422.642953] relay systemd[1]: Starting Record System Boot/Shutdown in UTMP... relay # [ 1422.656694] relay systemd[1]: Finished Record System Boot/Shutdown in UTMP. relay # [ 1422.664766] relay systemd[1]: Finished Rebuild Journal Catalog. relay # [ 1422.666053] relay systemd[1]: Starting Update is Completed... relay # [ 1422.675798] relay systemd[1]: Finished Update is Completed. relay # [ 1422.676468] relay systemd[1]: Reached target System Initialization. relay # [ 1422.676572] relay systemd[1]: Discard unused filesystem blocks once a week skipped, unmet condition check ConditionVirtualization=!container relay # [ 1422.676612] relay systemd[1]: Started Daily Cleanup of Temporary Directories. relay # [ 1422.676637] relay systemd[1]: Reached target Timer Units. relay # [ 1422.676761] relay systemd[1]: Listening on D-Bus System Message Bus Socket. relay # [ 1422.676931] relay systemd[1]: Listening on Nix Daemon Socket. relay # [ 1422.677055] relay systemd[1]: Listening on Virtual Machine and Container Registration Service Socket. relay # [ 1422.677071] relay systemd[1]: Reached target Socket Units. relay # [ 1422.677106] relay systemd[1]: Reached target Basic System. relay # [ 1422.678453] relay systemd[1]: Started device-mock.service. relay # [ 1422.679322] relay systemd[1]: Starting Re-link flakelet services at boot... relay # [ 1422.680107] relay systemd[1]: Starting Import lastlog data into lastlog2 database... relay # [ 1422.681202] relay systemd[1]: Starting Name Service Cache Daemon (nsncd)... relay # [ 1422.682145] relay systemd[1]: Starting resolvconf update... relay # [ 1422.693560] relay systemd[1]: Finished Re-link flakelet services at boot. relay # [ 1422.694601] relay systemd[1]: Starting Reconcile flakelet services with the host configuration... relay # [ 1422.701136] relay systemd[1]: Finished Import lastlog data into lastlog2 database. relay # [ 1422.708464] relay systemd[1]: Finished Reconcile flakelet services with the host configuration. relay # [ 1422.785156] relay systemd[1]: etc-machine\x2did.mount: Deactivated successfully. relay # [ 1422.786027] relay systemd[1]: Finished Save Transient machine-id to Disk. relay # [ 1422.786219] relay systemd[1]: nscd.service: Deactivated successfully. relay # [ 1422.786333] relay systemd[1]: Stopped Name Service Cache Daemon (nsncd). relay # [ 1422.787885] relay systemd[1]: Finished resolvconf update. relay # [ 1422.789774] relay systemd[1]: Reached target Preparation for Network. relay # [ 1422.790988] relay systemd[1]: Starting Address configuration of eth1... relay # [ 1422.791900] relay systemd[1]: Starting Extra networking commands.... relay # [ 1422.792761] relay systemd[1]: Starting Name Service Cache Daemon (nsncd)... client # [ 1422.649233] client systemd-tmpfiles[161]: fchmod() of /run/log/journal failed: Operation not permitted client # [ 1422.650190] client systemd[1]: suid-sgid-wrappers.service: Main process exited, code=exited, status=1/FAILURE client # [ 1422.650231] client systemd[1]: suid-sgid-wrappers.service: Failed with result 'exit-code'. client # [ 1422.650297] client systemd[1]: Failed to start Create SUID/SGID Wrappers. client # [ 1422.650857] client systemd[1]: Finished Create System Files and Directories. client # [ 1422.666566] client systemd[1]: Starting Rebuild Journal Catalog... client # [ 1422.667370] client systemd[1]: Starting Record System Boot/Shutdown in UTMP... client # [ 1422.680117] client systemd[1]: Finished Record System Boot/Shutdown in UTMP. client # [ 1422.687401] client systemd[1]: Finished Rebuild Journal Catalog. client # [ 1422.688683] client systemd[1]: Starting Update is Completed... client # [ 1422.699301] client systemd[1]: Finished Update is Completed. client # [ 1422.699405] client systemd[1]: Reached target System Initialization. client # [ 1422.699491] client systemd[1]: Discard unused filesystem blocks once a week skipped, unmet condition check ConditionVirtualization=!container client # [ 1422.699521] client systemd[1]: Started Daily Cleanup of Temporary Directories. client # [ 1422.699538] client systemd[1]: Reached target Timer Units. client # [ 1422.699648] client systemd[1]: Listening on D-Bus System Message Bus Socket. client # [ 1422.699812] client systemd[1]: Listening on Nix Daemon Socket. client # [ 1422.699914] client systemd[1]: Listening on Virtual Machine and Container Registration Service Socket. client # [ 1422.699930] client systemd[1]: Reached target Socket Units. client # [ 1422.699963] client systemd[1]: Reached target Basic System. client # [ 1422.701219] client systemd[1]: Starting Import lastlog data into lastlog2 database... client # [ 1422.702043] client systemd[1]: Starting Name Service Cache Daemon (nsncd)... client # [ 1422.702753] client systemd[1]: Starting resolvconf update... client # [ 1422.727183] client systemd[1]: Finished Import lastlog data into lastlog2 database. client # [ 1422.793372] client systemd[1]: etc-machine\x2did.mount: Deactivated successfully. client # [ 1422.794128] client systemd[1]: Finished Save Transient machine-id to Disk. client # [ 1422.817373] client systemd[1]: nscd.service: Deactivated successfully. client # [ 1422.817508] client systemd[1]: Stopped Name Service Cache Daemon (nsncd). client # [ 1422.820013] client systemd[1]: Starting Name Service Cache Daemon (nsncd)... client # [ 1422.829831] client systemd[1]: Finished resolvconf update. client # [ 1422.830108] client systemd[1]: Reached target Preparation for Network. agent # [ 1422.651807] agent systemd[1]: suid-sgid-wrappers.service: Main process exited, code=exited, status=1/FAILURE agent # [ 1422.651861] agent systemd[1]: suid-sgid-wrappers.service: Failed with result 'exit-code'. agent # [ 1422.651918] agent systemd[1]: Failed to start Create SUID/SGID Wrappers. agent # [ 1422.652783] agent systemd[1]: Finished Create System Files and Directories. agent # [ 1422.713921] agent systemd[1]: Starting Rebuild Journal Catalog... agent # [ 1422.715035] agent systemd[1]: Starting Record System Boot/Shutdown in UTMP... agent # [ 1422.731587] agent systemd[1]: Finished Record System Boot/Shutdown in UTMP. agent # [ 1422.738555] agent systemd[1]: Finished Rebuild Journal Catalog. agent # [ 1422.740320] agent systemd[1]: Starting Update is Completed... agent # [ 1422.750444] agent systemd[1]: Finished Update is Completed. agent # [ 1422.797714] agent systemd[1]: etc-machine\x2did.mount: Deactivated successfully. agent # [ 1422.806345] agent systemd[1]: Finished Save Transient machine-id to Disk. agent # [ 1422.808758] agent systemd[1]: Reached target System Initialization. agent # [ 1422.808876] agent systemd[1]: Discard unused filesystem blocks once a week skipped, unmet condition check ConditionVirtualization=!container agent # [ 1422.808917] agent systemd[1]: Started Daily Cleanup of Temporary Directories. agent # [ 1422.808942] agent systemd[1]: Reached target Timer Units. agent # [ 1422.809119] agent systemd[1]: Listening on D-Bus System Message Bus Socket. agent # [ 1422.809312] agent systemd[1]: Listening on Nix Daemon Socket. agent # [ 1422.809422] agent systemd[1]: Listening on Virtual Machine and Container Registration Service Socket. agent # [ 1422.809439] agent systemd[1]: Reached target Socket Units. agent # [ 1422.809470] agent systemd[1]: Reached target Basic System. agent # [ 1422.814106] agent systemd[1]: Starting Re-link flakelet services at boot... agent # [ 1422.815106] agent systemd[1]: Starting Import lastlog data into lastlog2 database... agent # [ 1422.815927] agent systemd[1]: Starting Name Service Cache Daemon (nsncd)... agent # [ 1422.816520] agent systemd[1]: Starting resolvconf update... agent # [ 1422.826839] agent systemd[1]: Finished Re-link flakelet services at boot. agent # [ 1422.828450] agent systemd[1]: Starting Reconcile flakelet services with the host configuration... relay # [ 1422.830957] relay network-addresses-eth1-start[249]: adding address 192.168.1.3/24... done relay # [ 1422.834193] relay network-addresses-eth1-start[249]: adding address 2001:db8:1::3/64... done relay # [ 1422.838662] relay systemd[1]: Finished Address configuration of eth1. relay # [ 1422.875654] relay systemd[1]: Finished Extra networking commands.. relay # [ 1422.876904] relay systemd[1]: Reached target Network. relay # [ 1422.879305] relay systemd[1]: Starting Dnsmasq Daemon... relay # [ 1422.881074] relay systemd[1]: Starting Nginx Web Server... relay # [ 1422.882507] relay systemd[1]: Starting Update flakelet service flakelet-relay... relay # [ 1422.895787] relay flakelet[311]: flakelet-relay: using prebuilt artifact /nix/store/ckflbc670cfwlgcf52fddj2l3gps7vdk-flakelet-flakelet-relay relay # [ 1422.896575] relay flakelet[311]: flakelet-relay: activating generation 1 relay # [ 1422.902267] relay systemd[1]: Reload requested from client PID 314 ('systemctl') (unit flakelet-flakelet-relay.service)... relay # [ 1422.902395] relay systemd[1]: Reloading... client # [ 1422.831442] client systemd[1]: Starting Address configuration of eth1... client # [ 1422.832704] client systemd[1]: Starting Extra networking commands.... client # [ 1422.851159] client network-addresses-eth1-start[238]: adding address 192.168.1.2/24... done client # [ 1422.854190] client network-addresses-eth1-start[238]: adding address 2001:db8:1::2/64... done client # [ 1422.859045] client systemd[1]: Finished Address configuration of eth1. client # [ 1422.917888] client systemd[1]: Finished Extra networking commands.. client # [ 1422.918191] client systemd[1]: Reached target Network. client # [ 1422.939894] client systemd[1]: nscd.service: Deactivated successfully. client # [ 1422.940217] client systemd[1]: Stopped Name Service Cache Daemon (nsncd). client # [ 1422.944876] client systemd[1]: Starting Name Service Cache Daemon (nsncd)... agent # [ 1422.837920] agent systemd[1]: Finished Import lastlog data into lastlog2 database. agent # [ 1422.842617] agent systemd[1]: Finished Reconcile flakelet services with the host configuration. agent # [ 1422.891008] agent systemd[1]: Finished resolvconf update. agent # [ 1422.891166] agent systemd[1]: Reached target Preparation for Network. agent # [ 1422.892207] agent systemd[1]: Starting Address configuration of eth1... agent # [ 1422.892917] agent systemd[1]: Starting Extra networking commands.... agent # [ 1422.912618] agent systemd[1]: nscd.service: Deactivated successfully. agent # [ 1422.912947] agent systemd[1]: Stopped Name Service Cache Daemon (nsncd). agent # [ 1422.913985] agent network-addresses-eth1-start[237]: adding address 192.168.1.1/24... done agent # [ 1422.917458] agent systemd[1]: Starting Name Service Cache Daemon (nsncd)... agent # [ 1422.917794] agent network-addresses-eth1-start[237]: adding address 2001:db8:1::1/64... done agent # [ 1422.927764] agent systemd[1]: Finished Address configuration of eth1. agent # [ 1422.979627] agent systemd[1]: Finished Extra networking commands.. agent # [ 1422.979918] agent systemd[1]: Reached target Network. agent # [ 1422.981778] agent systemd[1]: Starting Update flakelet service app... agent # [ 1422.983248] agent systemd[1]: Starting Update flakelet service flakelet-agent... agent # [ 1422.984668] agent systemd[1]: Starting Update flakelet service other... agent # [ 1422.996751] agent flakelet[304]: app: using prebuilt artifact /nix/store/hysydfr96ds875vr0jgw5rjxi4p39h2j-flakelet-app-v1 agent # [ 1422.997223] agent flakelet[304]: app: activating generation 1 agent # [ 1422.999197] agent flakelet[305]: flakelet-agent: using prebuilt artifact /nix/store/39lnml2vs1b977n7ms2bz22asf5kabfy-flakelet-flakelet-agent agent # [ 1422.999886] agent flakelet[305]: flakelet-agent: activating generation 1 agent # [ 1423.000991] agent systemd[1]: nscd.service: Deactivated successfully. agent # [ 1423.001236] agent flakelet[306]: other: using prebuilt artifact /nix/store/hmr990n0b4v6rpzrcpyyw738r9xi7z95-flakelet-other-v1 agent # [ 1423.001821] agent flakelet[306]: other: activating generation 1 agent # [ 1423.001320] agent systemd[1]: Stopped Name Service Cache Daemon (nsncd). agent # [ 1423.012151] agent systemd[1]: Reload requested from client PID 307 ('systemctl') (unit flakelet-app.service)... agent # [ 1423.012230] agent systemd[1]: Reloading... relay # [ 1423.295841] relay nsncd[251]: Sep 10 11:58:24.790 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" relay # [ 1423.348977] relay systemd[1]: Reloading finished in 444 ms. relay # [ 1423.404303] relay systemd[1]: Started Name Service Cache Daemon (nsncd). relay # [ 1423.404812] relay systemd[1]: Reached target Host and Network Name Lookups. relay # [ 1423.404902] relay systemd[1]: Reached target User and Group Name Lookups. relay # [ 1423.406208] relay systemd[1]: Starting User Login Management... relay # [ 1423.407040] relay systemd[1]: Starting Permit User Sessions... relay # [ 1423.418876] relay systemd[1]: Reload requested from client PID 357 ('systemctl') (unit flakelet-flakelet-relay.service)... relay # [ 1423.418922] relay systemd[1]: Reloading... relay # [ 1423.602065] relay dnsmasq-pre-start[393]: dnsmasq: syntax check OK. client # [ 1423.235141] client nsncd[303]: Sep 10 11:58:24.729 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" client # [ 1423.235165] client systemd[1]: Started Name Service Cache Daemon (nsncd). client # [ 1423.235270] client systemd[1]: Reached target Host and Network Name Lookups. client # [ 1423.235364] client systemd[1]: Reached target User and Group Name Lookups. client # [ 1423.237785] client systemd[1]: Starting User Login Management... client # [ 1423.239156] client systemd[1]: Starting Permit User Sessions... client # [ 1423.258248] client systemd[1]: Finished Permit User Sessions. client # [ 1423.259610] client systemd[1]: Started Console Getty. client # [ 1423.259634] client systemd[1]: Getty on tty1 skipped, unmet condition check ConditionPathExists=/dev/tty0 client # [ 1423.259648] client systemd[1]: Reached target Login Prompts. agent # [ 1423.387343] agent systemd[1]: Reloading finished in 374 ms. agent # [ 1423.456925] agent systemd[1]: Reload requested from client PID 308 ('systemctl') (unit flakelet-flakelet-agent.service)... agent # [ 1423.456982] agent systemd[1]: Reloading... relay # [ 1423.780179] relay systemd[1]: Reloading finished in 360 ms. relay # [ 1423.849807] relay systemd[1]: Finished Permit User Sessions. relay # [ 1423.853659] relay systemd[1]: Started Console Getty. relay # [ 1423.853685] relay systemd[1]: Getty on tty1 skipped, unmet condition check ConditionPathExists=/dev/tty0 relay # [ 1423.853696] relay systemd[1]: Reached target Login Prompts. agent # [ 1423.818154] agent systemd[1]: Reloading finished in 360 ms. agent # [ 1423.884619] agent systemd[1]: Reload requested from client PID 310 ('systemctl') (unit flakelet-other.service)... agent # [ 1423.884669] agent systemd[1]: Reloading... relay # [ 1423.865218] relay systemd[1]: Starting flakelet-relay... agent # [ 1424.225824] agent systemd[1]: Reloading finished in 340 ms. agent # [ 1424.288826] agent systemd[1]: Starting Name Service Cache Daemon (nsncd)... agent # [ 1424.294545] agent systemd[1]: Reload requested from client PID 390 ('systemctl') (unit flakelet-app.service)... agent # [ 1424.294586] agent systemd[1]: Reloading... relay # [ 1424.478768] relay systemd[1]: Starting D-Bus System Message Bus... relay # [ 1424.548033] relay nginx-pre-start[403]: nginx: the configuration file /nix/store/01avjj26wcihp926xgasszcj5s0k7cbs-nginx.conf syntax is ok relay # [ 1424.548719] relay nginx-pre-start[403]: nginx: configuration file /nix/store/01avjj26wcihp926xgasszcj5s0k7cbs-nginx.conf test is successful relay # [ 1424.563150] relay systemd[1]: Started Nginx Web Server. agent # [ 1424.594206] agent nsncd[391]: Sep 10 11:58:26.088 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" agent # [ 1424.663360] agent systemd[1]: Reloading finished in 368 ms. agent # [ 1424.731513] agent systemd[1]: Started Name Service Cache Daemon (nsncd). agent # [ 1424.735370] agent systemd[1]: Reload requested from client PID 392 ('systemctl') (unit flakelet-flakelet-agent.service)... agent # [ 1424.735410] agent systemd[1]: Reloading... client # [ 1424.708883] client systemd-logind[315]: New seat seat0. client # [ 1424.709243] client systemd[1]: Started User Login Management. client # [ 1424.711332] client systemd[1]: Starting D-Bus System Message Bus... client # [ 1424.713014] client systemd[1]: Starting linger-users.service... client # [ 1424.740427] client systemd[1]: linger-users.service: Deactivated successfully. client # [ 1424.740836] client systemd[1]: Finished linger-users.service. client # [ 1424.742097] client systemd[1]: Reached target Multi-User System. relay # [ 1424.951755] relay systemd-logind[354]: New seat seat0. relay # [ 1424.952119] relay systemd[1]: Started User Login Management. relay # [ 1424.954855] relay systemd[1]: Starting linger-users.service... relay # [ 1424.986348] relay systemd[1]: linger-users.service: Deactivated successfully. relay # [ 1424.986783] relay systemd[1]: Finished linger-users.service. relay # [ 1425.071433] relay dbus-broker-launch[402]: Looking up NSS user entry for 'systemd-timesync'... relay # [ 1425.072928] relay dbus-broker-launch[402]: NSS returned no entry for 'systemd-timesync' relay # [ 1425.072928] relay dbus-broker-launch[402]: Invalid user-name in /nix/store/d6qla8vgy057drw03ps2mdgny56bml50-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync" relay # [ 1425.074285] relay systemd[1]: Started D-Bus System Message Bus. relay # [ 1425.080326] relay dbus-broker-launch[402]: Ready relay # [ 1425.084330] relay systemd[1]: Started Dnsmasq Daemon. relay # [ 1425.086429] relay dnsmasq[394]: started, version 2.93 cachesize 150 relay # [ 1425.086437] relay dnsmasq[394]: compile time options: IPv6 GNU-getopt DBus no-UBus no-i18n IDN2 DHCP DHCPv6 no-Lua TFTP conntrack ipset nftset auth DNSSEC loop-detect inotify dumpfile relay # [ 1425.086444] relay dnsmasq[394]: DBus support enabled: connected to system bus relay # [ 1425.086452] relay dnsmasq[394]: no upstream servers configured - please set them from DBus relay # [ 1425.086537] relay dnsmasq[394]: read /etc/hosts - 9 names agent # [ 1425.092173] agent systemd[1]: Reloading finished in 356 ms. agent # [ 1425.167530] agent systemd[1]: Reached target Host and Network Name Lookups. agent # [ 1425.167672] agent systemd[1]: Reached target User and Group Name Lookups. agent # [ 1425.168927] agent systemd[1]: Starting app v1... agent # [ 1425.170232] agent systemd[1]: Starting User Login Management... agent # [ 1425.171504] agent systemd[1]: Starting Permit User Sessions... agent # [ 1425.174518] agent systemd[1]: Reload requested from client PID 431 ('systemctl') (unit flakelet-other.service)... agent # [ 1425.174549] agent systemd[1]: Reloading... agent # [ 1425.523388] agent systemd[1]: Reloading finished in 348 ms. client # [ 1425.195583] client dbus-broker-launch[325]: Looking up NSS user entry for 'systemd-timesync'... client # [ 1425.197502] client dbus-broker-launch[325]: NSS returned no entry for 'systemd-timesync' client # [ 1425.197502] client dbus-broker-launch[325]: Invalid user-name in /nix/store/1adggnjynlmbnp2w55shr3virbchwb8q-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync" client # [ 1425.198322] client systemd[1]: Started D-Bus System Message Bus. client # [ 1425.198866] client systemd[1]: Startup finished in 4.059s. client # [ 1425.204256] client dbus-broker-launch[325]: Ready agent # [ 1425.591818] agent systemd[1]: Started app v1. agent # [ 1425.592342] agent systemd[1]: Finished Permit User Sessions. agent # [ 1425.598099] agent systemd[1]: Started Console Getty. agent # [ 1425.599108] agent systemd[1]: Starting flakelet-agent... agent # [ 1425.599126] agent systemd[1]: Getty on tty1 skipped, unmet condition check ConditionPathExists=/dev/tty0 agent # [ 1425.599190] agent systemd[1]: Reached target Login Prompts. agent # [ 1425.602528] agent flakelet[304]: app: updated to generation 1 agent # [ 1425.602951] agent systemd[1]: Starting other v1... agent # [ 1425.603304] agent systemd[1]: Finished Update flakelet service app. agent # [ 1425.634934] agent systemd[1]: Started other v1. agent # [ 1425.641198] agent flakelet[306]: other: updated to generation 1 agent # [ 1425.643281] agent systemd[1]: Finished Update flakelet service other. agent # [ 1425.670742] agent flakelet-agent[498]: {"timestamp":"2026-09-10T11:58:27.165408Z","level":"INFO","message":"local change","flakelet":"app","generation":1,"by":"host activation","target":"flakelet_relay::agent::jobs"} agent # [ 1425.671194] agent flakelet-agent[498]: {"timestamp":"2026-09-10T11:58:27.165700Z","level":"INFO","message":"local change","flakelet":"other","generation":1,"by":"host activation","target":"flakelet_relay::agent::jobs"} agent # [ 1425.671401] agent systemd[1]: Started flakelet-agent. agent # [ 1425.673868] agent flakelet-agent[498]: {"timestamp":"2026-09-10T11:58:27.168584Z","level":"WARN","message":"connect failed: connect relay:7443: Connection refused (os error 111)","relay":"https://relay:7443","target":"flakelet_relay::agent::conn"} agent # [ 1425.678218] agent flakelet[305]: flakelet-agent: updated to generation 1 agent # [ 1425.680282] agent systemd[1]: Finished Update flakelet service flakelet-agent. agent # [ 1425.680439] agent systemd[1]: Reached target flakelet managed services. agent # [ 1425.696886] agent flakelet-agent[498]: {"timestamp":"2026-09-10T11:58:27.191535Z","level":"INFO","message":"local change","flakelet":"flakelet-agent","generation":1,"by":"host activation","target":"flakelet_relay::agent::jobs"} relay # [ 1425.883115] relay flakelet-relay[400]: {"timestamp":"2026-09-10T11:58:27.377679Z","level":"INFO","message":"listening on 127.0.0.1:7400","target":"flakelet_relay::relay::server"} relay # [ 1425.883765] relay flakelet-relay[400]: {"timestamp":"2026-09-10T11:58:27.378242Z","level":"INFO","message":"listening on [::]:7443","target":"flakelet_relay::relay::server"} relay # [ 1425.883585] relay systemd[1]: Started flakelet-relay. relay # [ 1425.894687] relay systemd[1]: Starting health for flakelet-relay... relay: (finished: waiting for unit flakelet-relay.service, in 5.66 seconds) relay: waiting for unit nginx.service relay: (finished: waiting for unit nginx.service, in 0.02 seconds) agent: waiting for unit flakelet.target relay # [ 1426.161451] relay systemd[1]: flakelet-relay-health.service: Deactivated successfully. relay # [ 1426.161746] relay systemd[1]: Finished health for flakelet-relay. relay # [ 1426.169554] relay flakelet[311]: flakelet-relay: updated to generation 1 relay # [ 1426.172758] relay systemd[1]: Finished Update flakelet service flakelet-relay. relay # [ 1426.173410] relay systemd[1]: Reached target flakelet managed services. relay # [ 1426.173594] relay systemd[1]: Reached target Multi-User System. relay # [ 1426.173845] relay systemd[1]: Startup finished in 5.031s. agent # [ 1426.334300] agent systemd-logind[460]: New seat seat0. agent # [ 1426.334616] agent systemd[1]: Started User Login Management. agent # [ 1426.336059] agent systemd[1]: Starting D-Bus System Message Bus... agent # [ 1426.337092] agent systemd[1]: Starting linger-users.service... agent # [ 1426.359122] agent systemd[1]: linger-users.service: Deactivated successfully. agent # [ 1426.359507] agent systemd[1]: Finished linger-users.service. agent # [ 1426.359877] agent systemd[1]: Reached target Multi-User System. agent: (finished: waiting for unit flakelet.target, in 0.65 seconds) agent: waiting for unit flakelet-agent.service agent: (finished: waiting for unit flakelet-agent.service, in 0.02 seconds) client: waiting for unit multi-user.target client: (finished: waiting for unit multi-user.target, in 0.02 seconds) relay: waiting for success: curl -sf http://127.0.0.1:7400/metrics | grep -q 'agent_up{host="agent"} 1' agent # [ 1426.693384] agent dbus-broker-launch[898]: Looking up NSS user entry for 'systemd-timesync'... agent # [ 1426.694899] agent dbus-broker-launch[898]: NSS returned no entry for 'systemd-timesync' agent # [ 1426.694899] agent dbus-broker-launch[898]: Invalid user-name in /nix/store/8mp15gfn2m81r8005nnkppky8976lhbb-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync" agent # [ 1426.696057] agent systemd[1]: Started D-Bus System Message Bus. agent # [ 1426.696640] agent systemd[1]: Startup finished in 5.561s. agent # [ 1426.701879] agent dbus-broker-launch[898]: Ready relay: (finished: waiting for success: curl -sf http://127.0.0.1:7400/metrics | grep -q 'agent_up{host="agent"} 1', in 1.04 seconds) agent: must succeed: systemctl is-active app.service other.service agent: (finished: must succeed: systemctl is-active app.service other.service, in 0.01 seconds) subtest: deploy with client certificate, relay found via SRV agent: must succeed: jq '.services.app.prebuilt="/nix/store/6ili1vxjlag9d7dy0bxrbxk6bpi5qh9p-flakelet-app-v2"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json agent: (finished: must succeed: jq '.services.app.prebuilt="/nix/store/6ili1vxjlag9d7dy0bxrbxk6bpi5qh9p-flakelet-app-v2"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json, in 0.02 seconds) client: must succeed: push --relay-srv test --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy agent/app 2>&1 relay # [ 1427.694628] relay flakelet-relay[400]: {"timestamp":"2026-09-10T11:58:29.189290Z","level":"INFO","message":"agent connected","host":"agent","version":"0.1.0","flakelets":"[Named { name: \"app\", generation: Some(1), revision: Some(\"prebuilt:app\") }, Named { name: \"flakelet-agent\", generation: Some(1), revision: Some(\"prebuilt:flakelet-agent\") }, Named { name: \"other\", generation: Some(1), revision: Some(\"prebuilt:other\") }]","target":"flakelet_relay::relay::agent_conn"} relay # [ 1428.080250] relay flakelet-relay[400]: {"timestamp":"2026-09-10T11:58:29.574818Z","level":"INFO","message":"deploy accepted","job":"d214d3b7566948f7ad37c0917026fbdb","caller":"admin@example.org","targets":"[\"agent/app\"]","target":"flakelet_relay::relay::api"} agent # [ 1427.694176] agent flakelet-agent[498]: {"timestamp":"2026-09-10T11:58:29.188870Z","level":"INFO","message":"connected","relay":"https://relay:7443","relay_name":"relay","relay_version":"0.1.0","host":"agent","target":"flakelet_relay::agent::conn"} agent # [ 1428.081261] agent flakelet-agent[498]: {"timestamp":"2026-09-10T11:58:29.575887Z","level":"INFO","message":"start","id":"dc428fca0538464a729854d0cc3fa405","flakelet":"app","rule":"admins","relay":"relay","caller_name":"\"admin@example.org\"","target":"flakelet_relay::agent::conn"} agent # [ 1428.081962] agent flakelet-agent[498]: {"timestamp":"2026-09-10T11:58:29.576704Z","level":"INFO","message":"update starting","flakelet":"app","ids":"[\"dc428fca0538464a729854d0cc3fa405\"]","target":"flakelet_relay::agent::jobs"} agent # [ 1428.129893] agent systemd[1]: Starting flakelet update app (via relay)... agent # [ 1428.152087] agent systemd[1]: Started flakelet update app (via relay). agent # [ 1428.157523] agent flakelet[920]: app: using prebuilt artifact /nix/store/6ili1vxjlag9d7dy0bxrbxk6bpi5qh9p-flakelet-app-v2 agent # [ 1428.158283] agent flakelet[920]: app: activating generation 2 agent # [ 1428.180020] agent systemd[1]: Stopping app v1... agent # [ 1428.180544] agent systemd[1]: app.service: Deactivated successfully. agent # [ 1428.180901] agent systemd[1]: Stopped app v1. agent # [ 1428.201353] agent systemd[1]: Reload requested from client PID 927 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [ 1428.201437] agent systemd[1]: Reloading... agent # [ 1428.559161] agent systemd[1]: Reloading finished in 356 ms. agent # [ 1428.611591] agent systemd[1]: Reload requested from client PID 957 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [ 1428.611636] agent systemd[1]: Reloading... agent # [ 1428.957303] agent systemd[1]: Reloading finished in 345 ms. agent # [ 1428.995894] agent systemd[1]: Reload requested from client PID 986 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [ 1428.995980] agent systemd[1]: Reloading... agent # [ 1429.342812] agent systemd[1]: Reloading finished in 346 ms. agent # [ 1429.404160] agent systemd[1]: Reload requested from client PID 1017 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [ 1429.404258] agent systemd[1]: Reloading... agent # [ 1429.751333] agent systemd[1]: Reloading finished in 346 ms. agent # [ 1429.792416] agent systemd[1]: Starting app v2... agent # [ 1429.808795] agent systemd[1]: Started app v2. agent # [ 1429.813956] agent flakelet[920]: app: updated to generation 2 agent # [ 1429.816939] agent systemd[1]: flakelet-relay-job-app.service: Deactivated successfully. client: (finished: must succeed: push --relay-srv test --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy agent/app 2>&1, in 2.89 seconds) agent: must succeed: systemctl show app.service -p Description | grep -q 'app v2' agent: (finished: must succeed: systemctl show app.service -p Description | grep -q 'app v2', in 0.02 seconds) agent: must succeed: journalctl -u flakelet-relay-job-app.service | grep -q 'generation 2' agent: (finished: must succeed: journalctl -u flakelet-relay-job-app.service | grep -q 'generation 2', in 0.01 seconds) agent: must succeed: flakelet status --json app | jq -c '.[0].changed.by' agent: (finished: must succeed: flakelet status --json app | jq -c '.[0].changed.by', in 0.03 seconds) (finished: subtest: deploy with client certificate, relay found via SRV, in 2.97 seconds) subtest: deploy with OIDC token, sub principal agent: must succeed: jq '.services.app.prebuilt="/nix/store/hysydfr96ds875vr0jgw5rjxi4p39h2j-flakelet-app-v1"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json agent: (finished: must succeed: jq '.services.app.prebuilt="/nix/store/hysydfr96ds875vr0jgw5rjxi4p39h2j-flakelet-app-v1"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json, in 0.01 seconds) client: must succeed: echo '{"groups": []}' | step crypto jwt sign --key /nix/store/r7sclamg09v434w8zjlfnrcrn1w79wh6-flakelet-relay-test-issuer/jwk.json --iss https://relay:8443 --aud flakelet-relay --sub 'repo:github:example/app:ref:refs/heads/main' --exp $(( $(date +%s) + 300 )) --jti $(head -c8 /dev/urandom | base32) client: (finished: must succeed: echo '{"groups": []}' | step crypto jwt sign --key /nix/store/r7sclamg09v434w8zjlfnrcrn1w79wh6-flakelet-relay-test-issuer/jwk.json --iss https://relay:8443 --aud flakelet-relay --sub 'repo:github:example/app:ref:refs/heads/main' --exp $(( $(date +%s) + 300 )) --jti $(head -c8 /dev/urandom | base32), in 0.06 seconds) client: must succeed: FLAKELET_RELAY_TOKEN_COMMAND='echo eyJhbGciOiJFZERTQSIsImtpZCI6ImsxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJmbGFrZWxldC1yZWxheSIsImV4cCI6MTc4OTA0MTgxMiwiZ3JvdXBzIjpbXSwiaWF0IjoxNzg5MDQxNTEyLCJpc3MiOiJodHRwczovL3JlbGF5Ojg0NDMiLCJqdGkiOiJXUlBFQTM0REZUM0FFPT09IiwibmJmIjoxNzg5MDQxNTEyLCJzdWIiOiJyZXBvOmdpdGh1YjpleGFtcGxlL2FwcDpyZWY6cmVmcy9oZWFkcy9tYWluIn0.ZaV5Zps0aBwDVjxpv0FZhqWRrbbNz22G1WIcRk8001tEPrpaIFaNO75ONUd0C-_kb9qWd1BaG5zqbKSXo4wJCw' push --relay https://relay:7443 deploy agent/app 2>&1 agent # [ 1430.936061] agent flakelet-agent[498]: {"timestamp":"2026-09-10T11:58:32.430693Z","level":"INFO","message":"update finished","flakelet":"app","status":"Updated","generation":"Some(2)","target":"flakelet_relay::agent::jobs"} agent # [ 1431.109865] agent flakelet-agent[498]: {"timestamp":"2026-09-10T11:58:32.604537Z","level":"INFO","message":"start","id":"d0a1b7e925475454697481bd0ef70eff","flakelet":"app","rule":"ci","relay":"relay","caller_name":"\"repo:github:example/app:ref:refs/heads/main\"","target":"flakelet_relay::agent::conn"} agent # [ 1431.110533] agent flakelet-agent[498]: {"timestamp":"2026-09-10T11:58:32.605298Z","level":"INFO","message":"update starting","flakelet":"app","ids":"[\"d0a1b7e925475454697481bd0ef70eff\"]","target":"flakelet_relay::agent::jobs"} agent # [ 1431.155900] agent systemd[1]: Starting flakelet update app (via relay)... agent # [ 1431.177639] agent systemd[1]: Started flakelet update app (via relay). agent # [ 1431.182743] agent flakelet[1086]: app: using prebuilt artifact /nix/store/hysydfr96ds875vr0jgw5rjxi4p39h2j-flakelet-app-v1 agent # [ 1431.183530] agent flakelet[1086]: app: activating generation 3 agent # [ 1431.198570] agent systemd[1]: Stopping app v2... relay # [ 1430.937865] relay flakelet-relay[400]: {"timestamp":"2026-09-10T11:58:32.432545Z","level":"INFO","message":"deploy finished","job":"d214d3b7566948f7ad37c0917026fbdb","ok":true,"target":"flakelet_relay::relay::api"} relay # [ 1431.109355] relay flakelet-relay[400]: {"timestamp":"2026-09-10T11:58:32.604086Z","level":"INFO","message":"deploy accepted","job":"7d555a2236c9902c5adf70d35c4ec742","caller":"repo:github:example/app:ref:refs/heads/main","targets":"[\"agent/app\"]","target":"flakelet_relay::relay::api"} agent # [ 1431.199099] agent systemd[1]: app.service: Deactivated successfully. agent # [ 1431.199434] agent systemd[1]: Stopped app v2. agent # [ 1431.219294] agent systemd[1]: Reload requested from client PID 1093 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [ 1431.219380] agent systemd[1]: Reloading... agent # [ 1431.576359] agent systemd[1]: Reloading finished in 356 ms. agent # [ 1431.655962] agent systemd[1]: Reload requested from client PID 1123 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [ 1431.656069] agent systemd[1]: Reloading... agent # [ 1432.005392] agent systemd[1]: Reloading finished in 348 ms. agent # [ 1432.082405] agent systemd[1]: Reload requested from client PID 1155 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [ 1432.082451] agent systemd[1]: Reloading... agent # [ 1432.432614] agent systemd[1]: Reloading finished in 349 ms. agent # [ 1432.475227] agent systemd[1]: Reload requested from client PID 1187 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [ 1432.475336] agent systemd[1]: Reloading... agent # [ 1432.823848] agent systemd[1]: Reloading finished in 347 ms. agent # [ 1432.858754] agent systemd[1]: Starting app v1... agent # [ 1432.878114] agent systemd[1]: Started app v1. agent # [ 1432.886294] agent flakelet[1086]: app: updated to generation 3 agent # [ 1432.888738] agent systemd[1]: flakelet-relay-job-app.service: Deactivated successfully. client: (finished: must succeed: FLAKELET_RELAY_TOKEN_COMMAND='echo eyJhbGciOiJFZERTQSIsImtpZCI6ImsxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJmbGFrZWxldC1yZWxheSIsImV4cCI6MTc4OTA0MTgxMiwiZ3JvdXBzIjpbXSwiaWF0IjoxNzg5MDQxNTEyLCJpc3MiOiJodHRwczovL3JlbGF5Ojg0NDMiLCJqdGkiOiJXUlBFQTM0REZUM0FFPT09IiwibmJmIjoxNzg5MDQxNTEyLCJzdWIiOiJyZXBvOmdpdGh1YjpleGFtcGxlL2FwcDpyZWY6cmVmcy9oZWFkcy9tYWluIn0.ZaV5Zps0aBwDVjxpv0FZhqWRrbbNz22G1WIcRk8001tEPrpaIFaNO75ONUd0C-_kb9qWd1BaG5zqbKSXo4wJCw' push --relay https://relay:7443 deploy agent/app 2>&1, in 2.94 seconds) (finished: subtest: deploy with OIDC token, sub principal, in 3.01 seconds) subtest: policy denies uncovered target client: must fail: FLAKELET_RELAY_TOKEN_COMMAND='echo eyJhbGciOiJFZERTQSIsImtpZCI6ImsxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJmbGFrZWxldC1yZWxheSIsImV4cCI6MTc4OTA0MTgxMiwiZ3JvdXBzIjpbXSwiaWF0IjoxNzg5MDQxNTEyLCJpc3MiOiJodHRwczovL3JlbGF5Ojg0NDMiLCJqdGkiOiJXUlBFQTM0REZUM0FFPT09IiwibmJmIjoxNzg5MDQxNTEyLCJzdWIiOiJyZXBvOmdpdGh1YjpleGFtcGxlL2FwcDpyZWY6cmVmcy9oZWFkcy9tYWluIn0.ZaV5Zps0aBwDVjxpv0FZhqWRrbbNz22G1WIcRk8001tEPrpaIFaNO75ONUd0C-_kb9qWd1BaG5zqbKSXo4wJCw' push --relay https://relay:7443 deploy agent/other 2>&1 client: (finished: must fail: FLAKELET_RELAY_TOKEN_COMMAND='echo eyJhbGciOiJFZERTQSIsImtpZCI6ImsxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJmbGFrZWxldC1yZWxheSIsImV4cCI6MTc4OTA0MTgxMiwiZ3JvdXBzIjpbXSwiaWF0IjoxNzg5MDQxNTEyLCJpc3MiOiJodHRwczovL3JlbGF5Ojg0NDMiLCJqdGkiOiJXUlBFQTM0REZUM0FFPT09IiwibmJmIjoxNzg5MDQxNTEyLCJzdWIiOiJyZXBvOmdpdGh1YjpleGFtcGxlL2FwcDpyZWY6cmVmcy9oZWFkcy9tYWluIn0.ZaV5Zps0aBwDVjxpv0FZhqWRrbbNz22G1WIcRk8001tEPrpaIFaNO75ONUd0C-_kb9qWd1BaG5zqbKSXo4wJCw' push --relay https://relay:7443 deploy agent/other 2>&1, in 0.02 seconds) (finished: subtest: policy denies uncovered target, in 0.02 seconds) subtest: claim principal from groups, unchanged counts as success client: must succeed: echo '{"groups": ["wheel"]}' | step crypto jwt sign --key /nix/store/r7sclamg09v434w8zjlfnrcrn1w79wh6-flakelet-relay-test-issuer/jwk.json --iss https://relay:8443 --aud flakelet-relay --sub 'someone' --exp $(( $(date +%s) + 300 )) --jti $(head -c8 /dev/urandom | base32) client: (finished: must succeed: echo '{"groups": ["wheel"]}' | step crypto jwt sign --key /nix/store/r7sclamg09v434w8zjlfnrcrn1w79wh6-flakelet-relay-test-issuer/jwk.json --iss https://relay:8443 --aud flakelet-relay --sub 'someone' --exp $(( $(date +%s) + 300 )) --jti $(head -c8 /dev/urandom | base32), in 0.05 seconds) client: must succeed: FLAKELET_RELAY_TOKEN_COMMAND='echo eyJhbGciOiJFZERTQSIsImtpZCI6ImsxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJmbGFrZWxldC1yZWxheSIsImV4cCI6MTc4OTA0MTgxNSwiZ3JvdXBzIjpbIndoZWVsIl0sImlhdCI6MTc4OTA0MTUxNSwiaXNzIjoiaHR0cHM6Ly9yZWxheTo4NDQzIiwianRpIjoiN1czQlBLWFgzV0xKRT09PSIsIm5iZiI6MTc4OTA0MTUxNSwic3ViIjoic29tZW9uZSJ9.ModybuWfco-FEpzkvaE4WDu-JdaFZ57WCRSFfBc0tA_yPAGWgFAGUQl0PAyDKP_JZVQvqdP9OSf6BVc6fBibAQ' push --relay https://relay:7443 deploy agent/other 2>&1 agent # [ 1434.011765] agent flakelet-agent[498]: {"timestamp":"2026-09-10T11:58:35.506380Z","level":"INFO","message":"update finished","flakelet":"app","status":"Updated","generation":"Some(3)","target":"flakelet_relay::agent::jobs"} agent # [ 1434.112787] agent flakelet-agent[498]: {"timestamp":"2026-09-10T11:58:35.607507Z","level":"INFO","message":"start","id":"6719e4cc81ab7dc4b4e16846848982ea","flakelet":"other","rule":"admins","relay":"relay","caller_name":"\"someone\"","target":"flakelet_relay::agent::conn"} agent # [ 1434.113563] agent flakelet-agent[498]: {"timestamp":"2026-09-10T11:58:35.608322Z","level":"INFO","message":"update starting","flakelet":"other","ids":"[\"6719e4cc81ab7dc4b4e16846848982ea\"]","target":"flakelet_relay::agent::jobs"} agent # [ 1434.162237] agent systemd[1]: Starting flakelet update other (via relay)... agent # [ 1434.178537] agent systemd[1]: Started flakelet update other (via relay). agent # [ 1434.182927] agent flakelet[1241]: other: using prebuilt artifact /nix/store/hmr990n0b4v6rpzrcpyyw738r9xi7z95-flakelet-other-v1 agent # [ 1434.183664] agent flakelet[1241]: other: up to date agent # [ 1434.186481] agent systemd[1]: flakelet-relay-job-other.service: Deactivated successfully. relay # [ 1434.013515] relay flakelet-relay[400]: {"timestamp":"2026-09-10T11:58:35.508190Z","level":"INFO","message":"deploy finished","job":"7d555a2236c9902c5adf70d35c4ec742","ok":true,"target":"flakelet_relay::relay::api"} relay # [ 1434.112414] relay flakelet-relay[400]: {"timestamp":"2026-09-10T11:58:35.607152Z","level":"INFO","message":"deploy accepted","job":"429fea359c12631f8e09901426b339f5","caller":"someone","targets":"[\"agent/other\"]","target":"flakelet_relay::relay::api"} client: (finished: must succeed: FLAKELET_RELAY_TOKEN_COMMAND='echo eyJhbGciOiJFZERTQSIsImtpZCI6ImsxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJmbGFrZWxldC1yZWxheSIsImV4cCI6MTc4OTA0MTgxNSwiZ3JvdXBzIjpbIndoZWVsIl0sImlhdCI6MTc4OTA0MTUxNSwiaXNzIjoiaHR0cHM6Ly9yZWxheTo4NDQzIiwianRpIjoiN1czQlBLWFgzV0xKRT09PSIsIm5iZiI6MTc4OTA0MTUxNSwic3ViIjoic29tZW9uZSJ9.ModybuWfco-FEpzkvaE4WDu-JdaFZ57WCRSFfBc0tA_yPAGWgFAGUQl0PAyDKP_JZVQvqdP9OSf6BVc6fBibAQ' push --relay https://relay:7443 deploy agent/other 2>&1, in 0.64 seconds) (finished: subtest: claim principal from groups, unchanged counts as success, in 0.69 seconds) subtest: push login via device flow, cached token used afterwards client: must succeed: push login --issuer https://relay:8443 --client-id flakelet-relay 2>&1 agent # [ 1434.728549] agent flakelet-agent[498]: {"timestamp":"2026-09-10T11:58:36.223256Z","level":"INFO","message":"update finished","flakelet":"other","status":"Unchanged","generation":"Some(1)","target":"flakelet_relay::agent::jobs"} relay # [ 1434.729899] relay flakelet-relay[400]: {"timestamp":"2026-09-10T11:58:36.224581Z","level":"INFO","message":"deploy finished","job":"429fea359c12631f8e09901426b339f5","ok":true,"target":"flakelet_relay::relay::api"} relay # [ 1434.754479] relay 93sn3sv24fj26ksccpppzwg6kbf10s4w-device-mock[185]: 127.0.0.1 - - [10/Sep/2026 11:58:36] "POST /device HTTP/1.1" 200 - relay # [ 1435.759989] relay 93sn3sv24fj26ksccpppzwg6kbf10s4w-device-mock[185]: 127.0.0.1 - - [10/Sep/2026 11:58:37] "POST /token HTTP/1.1" 400 - client: (finished: must succeed: push login --issuer https://relay:8443 --client-id flakelet-relay 2>&1, in 2.08 seconds) client: must succeed: test "$(stat -c %a /root/.local/state/flakelet-push/token.json)" = 600 client: (finished: must succeed: test "$(stat -c %a /root/.local/state/flakelet-push/token.json)" = 600, in 0.01 seconds) client: must succeed: push --relay https://relay:7443 deploy agent/other 2>&1 relay # [ 1436.807371] relay 93sn3sv24fj26ksccpppzwg6kbf10s4w-device-mock[185]: 127.0.0.1 - - [10/Sep/2026 11:58:38] "POST /token HTTP/1.1" 200 - relay # [ 1436.840936] relay flakelet-relay[400]: {"timestamp":"2026-09-10T11:58:38.335629Z","level":"INFO","message":"deploy accepted","job":"5d2049fad2e380b6afa2be3784bd3222","caller":"dev@example.org","targets":"[\"agent/other\"]","target":"flakelet_relay::relay::api"} agent # [ 1436.841384] agent flakelet-agent[498]: {"timestamp":"2026-09-10T11:58:38.336073Z","level":"INFO","message":"start","id":"fd348ff2622cb800c7a356001f50208f","flakelet":"other","rule":"admins","relay":"relay","caller_name":"\"dev@example.org\"","target":"flakelet_relay::agent::conn"} agent # [ 1436.842475] agent flakelet-agent[498]: {"timestamp":"2026-09-10T11:58:38.337198Z","level":"INFO","message":"update starting","flakelet":"other","ids":"[\"fd348ff2622cb800c7a356001f50208f\"]","target":"flakelet_relay::agent::jobs"} agent # [ 1436.888388] agent systemd[1]: Starting flakelet update other (via relay)... agent # [ 1436.904472] agent systemd[1]: Started flakelet update other (via relay). agent # [ 1436.909593] agent flakelet[1265]: other: using prebuilt artifact /nix/store/hmr990n0b4v6rpzrcpyyw738r9xi7z95-flakelet-other-v1 agent # [ 1436.910144] agent flakelet[1265]: other: up to date agent # [ 1436.912564] agent systemd[1]: flakelet-relay-job-other.service: Deactivated successfully. client: (finished: must succeed: push --relay https://relay:7443 deploy agent/other 2>&1, in 0.64 seconds) client: must succeed: push --relay https://relay:7443 deploy agent/other 2>&1 agent # [ 1437.457554] agent flakelet-agent[498]: {"timestamp":"2026-09-10T11:58:38.952296Z","level":"INFO","message":"update finished","flakelet":"other","status":"Unchanged","generation":"Some(1)","target":"flakelet_relay::agent::jobs"} agent # [ 1437.479676] agent flakelet-agent[498]: {"timestamp":"2026-09-10T11:58:38.974437Z","level":"INFO","message":"start","id":"c778f3180fc4bffef8f93bde57262fee","flakelet":"other","rule":"admins","relay":"relay","caller_name":"\"dev@example.org\"","target":"flakelet_relay::agent::conn"} agent # [ 1437.479879] agent flakelet-agent[498]: {"timestamp":"2026-09-10T11:58:38.974650Z","level":"INFO","message":"update starting","flakelet":"other","ids":"[\"c778f3180fc4bffef8f93bde57262fee\"]","target":"flakelet_relay::agent::jobs"} agent # [ 1437.520273] agent systemd[1]: Starting flakelet update other (via relay)... agent # [ 1437.538179] agent systemd[1]: Started flakelet update other (via relay). agent # [ 1437.542907] agent flakelet[1285]: other: using prebuilt artifact /nix/store/hmr990n0b4v6rpzrcpyyw738r9xi7z95-flakelet-other-v1 agent # [ 1437.543657] agent flakelet[1285]: other: up to date agent # [ 1437.546813] agent systemd[1]: flakelet-relay-job-other.service: Deactivated successfully. relay # [ 1437.458738] relay flakelet-relay[400]: {"timestamp":"2026-09-10T11:58:38.953433Z","level":"INFO","message":"deploy finished","job":"5d2049fad2e380b6afa2be3784bd3222","ok":true,"target":"flakelet_relay::relay::api"} relay # [ 1437.479456] relay flakelet-relay[400]: {"timestamp":"2026-09-10T11:58:38.974185Z","level":"INFO","message":"deploy accepted","job":"aea4ea7a69622e5f529afb758f8577b8","caller":"dev@example.org","targets":"[\"agent/other\"]","target":"flakelet_relay::relay::api"} client: (finished: must succeed: push --relay https://relay:7443 deploy agent/other 2>&1, in 0.63 seconds) (finished: subtest: push login via device flow, cached token used afterwards, in 3.36 seconds) subtest: waves: failing first wave skips the second agent: must succeed: jq '.services.app.prebuilt="/nix/store/8282rnbvyk847q4l4zby3zwcq3jpzrvw-flakelet-app-broken"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json agent: (finished: must succeed: jq '.services.app.prebuilt="/nix/store/8282rnbvyk847q4l4zby3zwcq3jpzrvw-flakelet-app-broken"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json, in 0.02 seconds) client: must fail: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy agent/app --wave agent/other 2>&1 relay # [ 1438.090344] relay flakelet-relay[400]: {"timestamp":"2026-09-10T11:58:39.585054Z","level":"INFO","message":"deploy finished","job":"aea4ea7a69622e5f529afb758f8577b8","ok":true,"target":"flakelet_relay::relay::api"} relay # [ 1438.129168] relay flakelet-relay[400]: {"timestamp":"2026-09-10T11:58:39.623864Z","level":"INFO","message":"deploy accepted","job":"cbe7d8d5edc7da1d1eb5c5828ea48613","caller":"admin@example.org","targets":"[\"agent/app\", \"agent/other\"]","target":"flakelet_relay::relay::api"} agent # [ 1438.089018] agent flakelet-agent[498]: {"timestamp":"2026-09-10T11:58:39.583705Z","level":"INFO","message":"update finished","flakelet":"other","status":"Unchanged","generation":"Some(1)","target":"flakelet_relay::agent::jobs"} agent # [ 1438.129576] agent flakelet-agent[498]: {"timestamp":"2026-09-10T11:58:39.624299Z","level":"INFO","message":"start","id":"4003703108b17adebbe08aaad70b216c","flakelet":"app","rule":"admins","relay":"relay","caller_name":"\"admin@example.org\"","target":"flakelet_relay::agent::conn"} agent # [ 1438.130224] agent flakelet-agent[498]: {"timestamp":"2026-09-10T11:58:39.624946Z","level":"INFO","message":"update starting","flakelet":"app","ids":"[\"4003703108b17adebbe08aaad70b216c\"]","target":"flakelet_relay::agent::jobs"} agent # [ 1438.174555] agent systemd[1]: Starting flakelet update app (via relay)... agent # [ 1438.195094] agent systemd[1]: Started flakelet update app (via relay). agent # [ 1438.199458] agent flakelet[1312]: app: using prebuilt artifact /nix/store/8282rnbvyk847q4l4zby3zwcq3jpzrvw-flakelet-app-broken agent # [ 1438.200108] agent flakelet[1312]: app: activating generation 4 agent # [ 1438.222429] agent systemd[1]: Stopping app v1... agent # [ 1438.222944] agent systemd[1]: app.service: Deactivated successfully. agent # [ 1438.223297] agent systemd[1]: Stopped app v1. agent # [ 1438.244759] agent systemd[1]: Reload requested from client PID 1319 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [ 1438.244808] agent systemd[1]: Reloading... agent # [ 1438.592865] agent systemd[1]: Reloading finished in 347 ms. agent # [ 1438.653760] agent systemd[1]: Reload requested from client PID 1349 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [ 1438.653809] agent systemd[1]: Reloading... agent # [ 1438.999825] agent systemd[1]: Reloading finished in 345 ms. agent # [ 1439.059907] agent systemd[1]: Reload requested from client PID 1378 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [ 1439.059956] agent systemd[1]: Reloading... agent # [ 1439.402037] agent systemd[1]: Reloading finished in 341 ms. agent # [ 1439.480861] agent systemd[1]: Reload requested from client PID 1409 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [ 1439.480946] agent systemd[1]: Reloading... agent # [ 1439.826588] agent systemd[1]: Reloading finished in 344 ms. agent # [ 1439.874535] agent systemd[1]: Starting app broken... agent # [ 1439.893380] agent (nonexistent)[1443]: app.service: Unable to locate executable '/nonexistent': No such file or directory agent # [ 1439.893407] agent (nonexistent)[1443]: app.service: Failed at step EXEC spawning /nonexistent: No such file or directory agent # [ 1439.896276] agent systemd[1]: app.service: Main process exited, code=exited, status=203/EXEC agent # [ 1439.896392] agent systemd[1]: app.service: Failed with result 'exit-code'. agent # [ 1439.896690] agent systemd[1]: Failed to start app broken. agent # [ 1439.928650] agent systemd[1]: Reload requested from client PID 1449 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [ 1439.928703] agent systemd[1]: Reloading... agent # [ 1440.274228] agent systemd[1]: Reloading finished in 345 ms. agent # [ 1440.318155] agent systemd[1]: Reload requested from client PID 1479 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [ 1440.318190] agent systemd[1]: Reloading... agent # [ 1440.666873] agent systemd[1]: Reloading finished in 348 ms. agent # [ 1440.706971] agent systemd[1]: Reload requested from client PID 1509 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [ 1440.707016] agent systemd[1]: Reloading... agent # [ 1441.055755] agent systemd[1]: Reloading finished in 348 ms. agent # [ 1441.100530] agent systemd[1]: Reload requested from client PID 1539 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [ 1441.100619] agent systemd[1]: Reloading... agent # [ 1441.451609] agent systemd[1]: Reloading finished in 350 ms. agent # [ 1441.493965] agent systemd[1]: Starting app v1... agent # [ 1441.508738] agent systemd[1]: Started app v1. agent # [ 1441.509830] agent flakelet[1312]: app: deploy failed, rolled back: systemctl start app.service failed: agent # [ 1441.509830] agent flakelet[1312]: failed units: app.service. See journalctl -u app.service agent # [ 1441.512298] agent systemd[1]: flakelet-relay-job-app.service: Main process exited, code=exited, status=1/FAILURE agent # [ 1441.512414] agent systemd[1]: flakelet-relay-job-app.service: Failed with result 'exit-code'. client: (finished: must fail: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy agent/app --wave agent/other 2>&1, in 4.15 seconds) agent: must succeed: systemctl show app.service -p Description | grep -q 'app v1' agent: (finished: must succeed: systemctl show app.service -p Description | grep -q 'app v1', in 0.02 seconds) (finished: subtest: waves: failing first wave skips the second, in 4.18 seconds) subtest: host patterns expand on the relay agent: must succeed: jq '.services.app.prebuilt="/nix/store/hysydfr96ds875vr0jgw5rjxi4p39h2j-flakelet-app-v1"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json agent: (finished: must succeed: jq '.services.app.prebuilt="/nix/store/hysydfr96ds875vr0jgw5rjxi4p39h2j-flakelet-app-v1"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json, in 0.02 seconds) client: must succeed: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy '@all/app' 2>&1 agent # [ 1442.249593] agent flakelet-agent[498]: {"timestamp":"2026-09-10T11:58:43.744250Z","level":"INFO","message":"update finished","flakelet":"app","status":"RolledBack","generation":"Some(3)","target":"flakelet_relay::agent::jobs"} agent # [ 1442.313045] agent flakelet-agent[498]: {"timestamp":"2026-09-10T11:58:43.807695Z","level":"INFO","message":"start","id":"a1bbebb0f0bd04c2469fedae62b58c91","flakelet":"app","rule":"admins","relay":"relay","caller_name":"\"admin@example.org\"","target":"flakelet_relay::agent::conn"} agent # [ 1442.313569] agent flakelet-agent[498]: {"timestamp":"2026-09-10T11:58:43.808330Z","level":"INFO","message":"update starting","flakelet":"app","ids":"[\"a1bbebb0f0bd04c2469fedae62b58c91\"]","target":"flakelet_relay::agent::jobs"} agent # [ 1442.360123] agent systemd[1]: Starting flakelet update app (via relay)... agent # [ 1442.385912] agent systemd[1]: Started flakelet update app (via relay). agent # [ 1442.390788] agent flakelet[1599]: app: using prebuilt artifact /nix/store/hysydfr96ds875vr0jgw5rjxi4p39h2j-flakelet-app-v1 agent # [ 1442.391347] agent flakelet[1599]: app: up to date agent # [ 1442.394238] agent systemd[1]: flakelet-relay-job-app.service: Deactivated successfully. relay # [ 1442.251384] relay flakelet-relay[400]: {"timestamp":"2026-09-10T11:58:43.746101Z","level":"INFO","message":"deploy finished","job":"cbe7d8d5edc7da1d1eb5c5828ea48613","ok":false,"target":"flakelet_relay::relay::api"} relay # [ 1442.312541] relay flakelet-relay[400]: {"timestamp":"2026-09-10T11:58:43.807224Z","level":"INFO","message":"deploy accepted","job":"67a4d00a6a0a59d1c2049712ebe25f0a","caller":"admin@example.org","targets":"[\"agent/app\"]","target":"flakelet_relay::relay::api"} client: (finished: must succeed: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy '@all/app' 2>&1, in 0.65 seconds) client: must fail: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy '*/app' 2>&1 agent # [ 1442.934498] agent flakelet-agent[498]: {"timestamp":"2026-09-10T11:58:44.429182Z","level":"INFO","message":"update finished","flakelet":"app","status":"Unchanged","generation":"Some(3)","target":"flakelet_relay::agent::jobs"} agent # [ 1442.959482] agent flakelet-agent[498]: {"timestamp":"2026-09-10T11:58:44.454163Z","level":"INFO","message":"start","id":"7ea0bb31d3862f7ca975b3fea78b9671","flakelet":"app","rule":"admins","relay":"relay","caller_name":"\"admin@example.org\"","target":"flakelet_relay::agent::conn"} agent # [ 1442.959949] agent flakelet-agent[498]: {"timestamp":"2026-09-10T11:58:44.454667Z","level":"INFO","message":"update starting","flakelet":"app","ids":"[\"7ea0bb31d3862f7ca975b3fea78b9671\"]","target":"flakelet_relay::agent::jobs"} agent # [ 1443.001433] agent systemd[1]: Starting flakelet update app (via relay)... agent # [ 1443.018706] agent systemd[1]: Started flakelet update app (via relay). agent # [ 1443.023622] agent flakelet[1619]: app: using prebuilt artifact /nix/store/hysydfr96ds875vr0jgw5rjxi4p39h2j-flakelet-app-v1 agent # [ 1443.024353] agent flakelet[1619]: app: up to date agent # [ 1443.025873] agent systemd[1]: flakelet-relay-job-app.service: Deactivated successfully. relay # [ 1442.936217] relay flakelet-relay[400]: {"timestamp":"2026-09-10T11:58:44.430939Z","level":"INFO","message":"deploy finished","job":"67a4d00a6a0a59d1c2049712ebe25f0a","ok":true,"target":"flakelet_relay::relay::api"} relay # [ 1442.958990] relay flakelet-relay[400]: {"timestamp":"2026-09-10T11:58:44.453750Z","level":"INFO","message":"deploy accepted","job":"132974fbed9ec8cd618b1f7991db7a7a","caller":"admin@example.org","targets":"[\"agent/app\", \"ghost/app\"]","target":"flakelet_relay::relay::api"} client: (finished: must fail: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy '*/app' 2>&1, in 0.63 seconds) client: must fail: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy agent/app --wave '*/app' 2>&1 agent # [ 1443.565288] agent flakelet-agent[498]: {"timestamp":"2026-09-10T11:58:45.059980Z","level":"INFO","message":"update finished","flakelet":"app","status":"Unchanged","generation":"Some(3)","target":"flakelet_relay::agent::jobs"} agent # [ 1443.591581] agent flakelet-agent[498]: {"timestamp":"2026-09-10T11:58:45.086271Z","level":"INFO","message":"start","id":"4b58b6ccbb6e39c17cb8ebafd3b15404","flakelet":"app","rule":"admins","relay":"relay","caller_name":"\"admin@example.org\"","target":"flakelet_relay::agent::conn"} agent # [ 1443.592076] agent flakelet-agent[498]: {"timestamp":"2026-09-10T11:58:45.086836Z","level":"INFO","message":"update starting","flakelet":"app","ids":"[\"4b58b6ccbb6e39c17cb8ebafd3b15404\"]","target":"flakelet_relay::agent::jobs"} agent # [ 1443.635758] agent systemd[1]: Starting flakelet update app (via relay)... agent # [ 1443.650886] agent systemd[1]: Started flakelet update app (via relay). agent # [ 1443.655533] agent flakelet[1639]: app: using prebuilt artifact /nix/store/hysydfr96ds875vr0jgw5rjxi4p39h2j-flakelet-app-v1 agent # [ 1443.655969] agent flakelet[1639]: app: up to date agent # [ 1443.656972] agent systemd[1]: flakelet-relay-job-app.service: Deactivated successfully. relay # [ 1443.566996] relay flakelet-relay[400]: {"timestamp":"2026-09-10T11:58:45.061686Z","level":"INFO","message":"deploy finished","job":"132974fbed9ec8cd618b1f7991db7a7a","ok":false,"target":"flakelet_relay::relay::api"} relay # [ 1443.591094] relay flakelet-relay[400]: {"timestamp":"2026-09-10T11:58:45.085771Z","level":"INFO","message":"deploy accepted","job":"cef1b11e59367cf3900cbd413c41be3c","caller":"admin@example.org","targets":"[\"agent/app\", \"ghost/app\"]","target":"flakelet_relay::relay::api"} client: (finished: must fail: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy agent/app --wave '*/app' 2>&1, in 0.63 seconds) client: must fail: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy '@all/nope' 2>&1 client: (finished: must fail: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy '@all/nope' 2>&1, in 0.02 seconds) (finished: subtest: host patterns expand on the relay, in 1.95 seconds) subtest: detach returns once the job is accepted agent: must succeed: jq '.services.app.prebuilt="/nix/store/ln47rsd5km8a04hd5zlkz4q1v352zqh3-flakelet-app-slow2"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json agent: (finished: must succeed: jq '.services.app.prebuilt="/nix/store/ln47rsd5km8a04hd5zlkz4q1v352zqh3-flakelet-app-slow2"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json, in 0.02 seconds) client: must succeed: timeout 20 push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy --detach agent/app 2>&1 client: (finished: must succeed: timeout 20 push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy --detach agent/app 2>&1, in 0.02 seconds) agent: waiting for success: systemctl is-active flakelet-relay-job-app.service agent # [ 1444.199248] agent flakelet-agent[498]: {"timestamp":"2026-09-10T11:58:45.693774Z","level":"INFO","message":"update finished","flakelet":"app","status":"Unchanged","generation":"Some(3)","target":"flakelet_relay::agent::jobs"} agent # [ 1444.263889] agent flakelet-agent[498]: {"timestamp":"2026-09-10T11:58:45.758511Z","level":"INFO","message":"start","id":"621eb1602073ca9adcac97df42dd5e24","flakelet":"app","rule":"admins","relay":"relay","caller_name":"\"admin@example.org\"","target":"flakelet_relay::agent::conn"} agent # [ 1444.264362] agent flakelet-agent[498]: {"timestamp":"2026-09-10T11:58:45.759121Z","level":"INFO","message":"update starting","flakelet":"app","ids":"[\"621eb1602073ca9adcac97df42dd5e24\"]","target":"flakelet_relay::agent::jobs"} agent # [ 1444.308852] agent systemd[1]: Starting flakelet update app (via relay)... agent # [ 1444.323849] agent systemd[1]: Started flakelet update app (via relay). agent # [ 1444.328590] agent flakelet[1667]: app: using prebuilt artifact /nix/store/ln47rsd5km8a04hd5zlkz4q1v352zqh3-flakelet-app-slow2 agent # [ 1444.329356] agent flakelet[1667]: app: activating generation 4 agent # [ 1444.345792] agent systemd[1]: Stopping app v1... agent # [ 1444.346336] agent systemd[1]: app.service: Deactivated successfully. agent # [ 1444.346675] agent systemd[1]: Stopped app v1. agent # [ 1444.368113] agent systemd[1]: Reload requested from client PID 1674 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [ 1444.368158] agent systemd[1]: Reloading... relay # [ 1444.200789] relay flakelet-relay[400]: {"timestamp":"2026-09-10T11:58:45.695489Z","level":"INFO","message":"deploy finished","job":"cef1b11e59367cf3900cbd413c41be3c","ok":false,"target":"flakelet_relay::relay::api"} relay # [ 1444.263420] relay flakelet-relay[400]: {"timestamp":"2026-09-10T11:58:45.758121Z","level":"INFO","message":"deploy accepted","job":"91ac22b914f34f659a0c67d80b32aafe","caller":"admin@example.org","targets":"[\"agent/app\"]","target":"flakelet_relay::relay::api"} agent # [ 1444.717128] agent systemd[1]: Reloading finished in 348 ms. agent # [ 1444.767954] agent systemd[1]: Reload requested from client PID 1704 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [ 1444.768015] agent systemd[1]: Reloading... agent # [ 1445.114903] agent systemd[1]: Reloading finished in 346 ms. agent # [ 1445.165941] agent systemd[1]: Reload requested from client PID 1733 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [ 1445.166056] agent systemd[1]: Reloading... agent # [ 1445.515112] agent systemd[1]: Reloading finished in 348 ms. agent: (finished: waiting for success: systemctl is-active flakelet-relay-job-app.service, in 1.29 seconds) ??? Warning (UserWarning): wait_until_fails(): passing a bare int/float as a duration is deprecated. Use datetime.timedelta instead. File "/nix/store/f3vbkhcnzmgjphqaarrzgjdpzrb02va5-nixos-test-driver-1.1/lib/python3.14/site-packages/test_driver/duration.py", line 39 agent: waiting for failure: systemctl is-active flakelet-relay-job-app.service ??? Warning (UserWarning): execute(): passing a bare int/float as a duration is deprecated. Use datetime.timedelta instead. File "/nix/store/f3vbkhcnzmgjphqaarrzgjdpzrb02va5-nixos-test-driver-1.1/lib/python3.14/site-packages/test_driver/duration.py", line 39 agent # [ 1445.565289] agent systemd[1]: Reload requested from client PID 1766 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [ 1445.565382] agent systemd[1]: Reloading... agent # [ 1445.913637] agent systemd[1]: Reloading finished in 347 ms. agent # [ 1445.961424] agent systemd[1]: Starting app slow2... agent: (finished: waiting for failure: systemctl is-active flakelet-relay-job-app.service, in 6.49 seconds) (finished: subtest: detach returns once the job is accepted, in 7.82 seconds) subtest: unknown host is final, missing agent is retried, agents listing client: must fail: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy nope/app 2>&1 client: (finished: must fail: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy nope/app 2>&1, in 0.02 seconds) agent: must succeed: systemctl stop flakelet-agent agent: (finished: must succeed: systemctl stop flakelet-agent, in 0.07 seconds) relay: waiting for success: curl -sf http://127.0.0.1:7400/metrics | grep -q 'agent_up{host="agent"} 0' relay: (finished: waiting for success: curl -sf http://127.0.0.1:7400/metrics | grep -q 'agent_up{host="agent"} 0', in 0.02 seconds) client: must succeed: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy agent/other > /tmp/retry.out 2>&1 & client: (finished: must succeed: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy agent/other > /tmp/retry.out 2>&1 &, in 0.01 seconds) client: waiting for success: grep -q 'agent_unavailable.*retrying' /tmp/retry.out agent # [ 1452.009694] agent systemd[1]: Started app slow2. agent # [ 1452.019243] agent flakelet[1667]: app: updated to generation 4 agent # [ 1452.023176] agent systemd[1]: flakelet-relay-job-app.service: Deactivated successfully. agent # [ 1452.080383] agent systemd[1]: Stopping flakelet-agent... agent # [ 1452.106056] agent systemd[1]: flakelet-agent.service: Deactivated successfully. agent # [ 1452.106591] agent systemd[1]: Stopped flakelet-agent. agent # [ 1452.107113] agent systemd[1]: flakelet-agent.service: Consumed 1.434s CPU time over 26.507s wall clock time. relay # [ 1452.080458] relay flakelet-relay[400]: {"timestamp":"2026-09-10T11:58:53.575193Z","level":"INFO","message":"agent disconnected: IO error: peer closed connection without sending TLS close_notify: https://docs.rs/rustls/latest/rustls/manual/_03_howto/index.html#unexpected-eof","host":"agent","target":"flakelet_relay::relay::agent_conn"} client: (finished: waiting for success: grep -q 'agent_unavailable.*retrying' /tmp/retry.out, in 1.02 seconds) agent: must succeed: systemctl start flakelet-agent agent: (finished: must succeed: systemctl start flakelet-agent, in 0.08 seconds) ??? Warning (UserWarning): wait_until_succeeds(): passing a bare int/float as a duration is deprecated. Use datetime.timedelta instead. File "/nix/store/f3vbkhcnzmgjphqaarrzgjdpzrb02va5-nixos-test-driver-1.1/lib/python3.14/site-packages/test_driver/duration.py", line 39 client: waiting for success: grep -q 'agent/other: unchanged' /tmp/retry.out agent # [ 1453.204727] agent systemd[1]: Starting flakelet-agent... agent # [ 1453.249711] agent flakelet-agent[1832]: {"timestamp":"2026-09-10T11:58:54.744383Z","level":"INFO","message":"reattaching to running unit","flakelet":"app","ids":"[\"621eb1602073ca9adcac97df42dd5e24\"]","target":"flakelet_relay::agent::jobs"} agent # [ 1453.272793] agent systemd[1]: Started flakelet-agent. agent # [ 1453.283408] agent flakelet-agent[1832]: {"timestamp":"2026-09-10T11:58:54.778146Z","level":"INFO","message":"connected","relay":"https://relay:7443","relay_name":"relay","relay_version":"0.1.0","host":"agent","target":"flakelet_relay::agent::conn"} relay # [ 1453.283438] relay flakelet-relay[400]: {"timestamp":"2026-09-10T11:58:54.778178Z","level":"INFO","message":"agent connected","host":"agent","version":"0.1.0","flakelets":"[Named { name: \"app\", generation: Some(4), revision: Some(\"prebuilt:app\") }, Named { name: \"flakelet-agent\", generation: Some(1), revision: Some(\"prebuilt:flakelet-agent\") }, Named { name: \"other\", generation: Some(1), revision: Some(\"prebuilt:other\") }]","target":"flakelet_relay::relay::agent_conn"} agent # [ 1453.797739] agent flakelet-agent[1832]: {"timestamp":"2026-09-10T11:58:55.292428Z","level":"INFO","message":"update finished","flakelet":"app","status":"Updated","generation":"Some(4)","target":"flakelet_relay::agent::jobs"} agent # [ 1455.190433] agent flakelet-agent[1832]: {"timestamp":"2026-09-10T11:58:56.685152Z","level":"INFO","message":"start","id":"70a1fcaf99dbe82248c64d3295362125","flakelet":"other","rule":"admins","relay":"relay","caller_name":"\"admin@example.org\"","target":"flakelet_relay::agent::conn"} agent # [ 1455.191349] agent flakelet-agent[1832]: {"timestamp":"2026-09-10T11:58:56.685704Z","level":"INFO","message":"update starting","flakelet":"other","ids":"[\"70a1fcaf99dbe82248c64d3295362125\"]","target":"flakelet_relay::agent::jobs"} agent # [ 1455.241883] agent systemd[1]: Starting flakelet update other (via relay)... agent # [ 1455.257489] agent systemd[1]: Started flakelet update other (via relay). agent # [ 1455.262979] agent flakelet[2245]: other: using prebuilt artifact /nix/store/hmr990n0b4v6rpzrcpyyw738r9xi7z95-flakelet-other-v1 agent # [ 1455.263760] agent flakelet[2245]: other: up to date agent # [ 1455.265847] agent systemd[1]: flakelet-relay-job-other.service: Deactivated successfully. relay # [ 1455.190083] relay flakelet-relay[400]: {"timestamp":"2026-09-10T11:58:56.684802Z","level":"INFO","message":"deploy accepted","job":"afdc8be7676e2ae1087fc97562dea061","caller":"admin@example.org","targets":"[\"agent/other\"]","target":"flakelet_relay::relay::api"} agent # [ 1455.808705] agent flakelet-agent[1832]: {"timestamp":"2026-09-10T11:58:57.303341Z","level":"INFO","message":"update finished","flakelet":"other","status":"Unchanged","generation":"Some(1)","target":"flakelet_relay::agent::jobs"} relay # [ 1455.810448] relay flakelet-relay[400]: {"timestamp":"2026-09-10T11:58:57.305179Z","level":"INFO","message":"deploy finished","job":"afdc8be7676e2ae1087fc97562dea061","ok":true,"target":"flakelet_relay::relay::api"} client: (finished: waiting for success: grep -q 'agent/other: unchanged' /tmp/retry.out, in 3.05 seconds) client: must succeed: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem agents client: (finished: must succeed: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem agents, in 0.02 seconds) client: must succeed: echo '{"groups": []}' | step crypto jwt sign --key /nix/store/r7sclamg09v434w8zjlfnrcrn1w79wh6-flakelet-relay-test-issuer/jwk.json --iss https://relay:8443 --aud flakelet-relay --sub 'repo:github:example/app:ref:refs/heads/main' --exp $(( $(date +%s) + 300 )) --jti $(head -c8 /dev/urandom | base32) client: (finished: must succeed: echo '{"groups": []}' | step crypto jwt sign --key /nix/store/r7sclamg09v434w8zjlfnrcrn1w79wh6-flakelet-relay-test-issuer/jwk.json --iss https://relay:8443 --aud flakelet-relay --sub 'repo:github:example/app:ref:refs/heads/main' --exp $(( $(date +%s) + 300 )) --jti $(head -c8 /dev/urandom | base32), in 0.05 seconds) client: must succeed: FLAKELET_RELAY_TOKEN_COMMAND='echo eyJhbGciOiJFZERTQSIsImtpZCI6ImsxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJmbGFrZWxldC1yZWxheSIsImV4cCI6MTc4OTA0MTgzNywiZ3JvdXBzIjpbXSwiaWF0IjoxNzg5MDQxNTM3LCJpc3MiOiJodHRwczovL3JlbGF5Ojg0NDMiLCJqdGkiOiI1WTdUVDJBR0RQV0pXPT09IiwibmJmIjoxNzg5MDQxNTM3LCJzdWIiOiJyZXBvOmdpdGh1YjpleGFtcGxlL2FwcDpyZWY6cmVmcy9oZWFkcy9tYWluIn0.jeA6OiECBQyWtj2Da1SSjFjWOQpNKVWVXzRFiIT7X7pi852oSwoQFUH7ZhBkrCZnyo_VZ0_NMvmUKssFOmypCA' push --relay https://relay:7443 agents client: (finished: must succeed: FLAKELET_RELAY_TOKEN_COMMAND='echo eyJhbGciOiJFZERTQSIsImtpZCI6ImsxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJmbGFrZWxldC1yZWxheSIsImV4cCI6MTc4OTA0MTgzNywiZ3JvdXBzIjpbXSwiaWF0IjoxNzg5MDQxNTM3LCJpc3MiOiJodHRwczovL3JlbGF5Ojg0NDMiLCJqdGkiOiI1WTdUVDJBR0RQV0pXPT09IiwibmJmIjoxNzg5MDQxNTM3LCJzdWIiOiJyZXBvOmdpdGh1YjpleGFtcGxlL2FwcDpyZWY6cmVmcy9oZWFkcy9tYWluIn0.jeA6OiECBQyWtj2Da1SSjFjWOQpNKVWVXzRFiIT7X7pi852oSwoQFUH7ZhBkrCZnyo_VZ0_NMvmUKssFOmypCA' push --relay https://relay:7443 agents, in 0.02 seconds) (finished: subtest: unknown host is final, missing agent is retried, agents listing, in 4.38 seconds) subtest: out-of-band flakelet update reaches the relay without reconnect agent: must succeed: flakelet status --json app | jq '.[0].generation' agent: (finished: must succeed: flakelet status --json app | jq '.[0].generation', in 0.04 seconds) agent: must succeed: journalctl -u flakelet-agent -o cat | grep -c '"connected"' || true agent: (finished: must succeed: journalctl -u flakelet-agent -o cat | grep -c '"connected"' || true, in 0.01 seconds) agent: must succeed: jq '.services.app.prebuilt="/nix/store/6ili1vxjlag9d7dy0bxrbxk6bpi5qh9p-flakelet-app-v2"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json agent: (finished: must succeed: jq '.services.app.prebuilt="/nix/store/6ili1vxjlag9d7dy0bxrbxk6bpi5qh9p-flakelet-app-v2"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json, in 0.02 seconds) agent: must succeed: flakelet update app app: using prebuilt artifact /nix/store/6ili1vxjlag9d7dy0bxrbxk6bpi5qh9p-flakelet-app-v2 app: activating generation 5 agent # [ 1456.518334] agent systemd[1]: Stopping app slow2... agent # [ 1456.518884] agent systemd[1]: app.service: Deactivated successfully. agent # [ 1456.519240] agent systemd[1]: Stopped app slow2. agent # [ 1456.542382] agent systemd[1]: Reload requested from client PID 2279 ('systemctl')... agent # [ 1456.542469] agent systemd[1]: Reloading... agent # [ 1456.904428] agent systemd[1]: Reloading finished in 361 ms. agent # [ 1456.956717] agent systemd[1]: Reload requested from client PID 2308 ('systemctl')... agent # [ 1456.956752] agent systemd[1]: Reloading... agent # [ 1457.300733] agent systemd[1]: Reloading finished in 343 ms. agent # [ 1457.346079] agent systemd[1]: Reload requested from client PID 2339 ('systemctl')... agent # [ 1457.346115] agent systemd[1]: Reloading... agent # [ 1457.686405] agent systemd[1]: Reloading finished in 339 ms. agent # [ 1457.743486] agent systemd[1]: Reload requested from client PID 2369 ('systemctl')... agent # [ 1457.743531] agent systemd[1]: Reloading... agent: (finished: must succeed: flakelet update app, in 1.67 seconds) client: waiting for success: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem agents | grep -q 'app@5' agent # [ 1458.087777] agent systemd[1]: Reloading finished in 343 ms. agent # [ 1458.128907] agent systemd[1]: Starting app v2... agent # [ 1458.144926] agent systemd[1]: Started app v2. agent # [ 1459.301807] agent flakelet-agent[1832]: {"timestamp":"2026-09-10T11:59:00.796500Z","level":"INFO","message":"local change","flakelet":"app","generation":5,"by":"root (shell)","target":"flakelet_relay::agent::jobs"} client: (finished: waiting for success: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem agents | grep -q 'app@5', in 2.07 seconds) agent: must succeed: journalctl -u flakelet-agent -o cat | grep -c '"connected"' || true agent: (finished: must succeed: journalctl -u flakelet-agent -o cat | grep -c '"connected"' || true, in 0.01 seconds) client: waiting for success: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem jobs | grep local-app-5- client: (finished: waiting for success: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem jobs | grep local-app-5-, in 0.02 seconds) client: must succeed: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem jobs | grep local-app- client: (finished: must succeed: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem jobs | grep local-app-, in 0.02 seconds) agent: must succeed: jq '.services.app.prebuilt="/nix/store/hysydfr96ds875vr0jgw5rjxi4p39h2j-flakelet-app-v1"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json agent: (finished: must succeed: jq '.services.app.prebuilt="/nix/store/hysydfr96ds875vr0jgw5rjxi4p39h2j-flakelet-app-v1"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json, in 0.01 seconds) agent: must succeed: flakelet update app app: using prebuilt artifact /nix/store/hysydfr96ds875vr0jgw5rjxi4p39h2j-flakelet-app-v1 app: activating generation 6 agent # [ 1460.327556] agent systemd[1]: Stopping app v2... agent # [ 1460.328075] agent systemd[1]: app.service: Deactivated successfully. agent # [ 1460.328407] agent systemd[1]: Stopped app v2. agent # [ 1460.350284] agent systemd[1]: Reload requested from client PID 2415 ('systemctl')... agent # [ 1460.350331] agent systemd[1]: Reloading... agent # [ 1460.717566] agent systemd[1]: Reloading finished in 366 ms. agent # [ 1460.771849] agent systemd[1]: Reload requested from client PID 2444 ('systemctl')... agent # [ 1460.771883] agent systemd[1]: Reloading... agent # [ 1461.166283] agent systemd[1]: Reloading finished in 393 ms. agent # [ 1461.233099] agent systemd[1]: Reload requested from client PID 2472 ('systemctl')... agent # [ 1461.233138] agent systemd[1]: Reloading... agent # [ 1461.594784] agent systemd[1]: Reloading finished in 361 ms. agent # [ 1461.673214] agent systemd[1]: Reload requested from client PID 2505 ('systemctl')... agent # [ 1461.673253] agent systemd[1]: Reloading... agent # [ 1462.017856] agent systemd[1]: Reloading finished in 344 ms. agent: (finished: must succeed: flakelet update app, in 1.82 seconds) client: waiting for success: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem agents | grep -q 'app@6' agent # [ 1462.075229] agent systemd[1]: Starting app v1... agent # [ 1462.103270] agent systemd[1]: Started app v1. agent # [ 1463.303809] agent flakelet-agent[1832]: {"timestamp":"2026-09-10T11:59:04.798483Z","level":"INFO","message":"local change","flakelet":"app","generation":6,"by":"root (shell)","target":"flakelet_relay::agent::jobs"} client: (finished: waiting for success: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem agents | grep -q 'app@6', in 2.07 seconds) (finished: subtest: out-of-band flakelet update reaches the relay without reconnect, in 7.76 seconds) subtest: certificate not listed under agents cannot connect as agent client: must succeed: curl -s -o /dev/null -w '%{http_code}' --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/mallory/cert.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/mallory/key.pem -H 'Upgrade: websocket' -H 'Connection: Upgrade' -H 'Sec-WebSocket-Version: 13' -H 'Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==' https://relay:7443/v1/agent client: (finished: must succeed: curl -s -o /dev/null -w '%{http_code}' --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/mallory/cert.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/mallory/key.pem -H 'Upgrade: websocket' -H 'Connection: Upgrade' -H 'Sec-WebSocket-Version: 13' -H 'Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==' https://relay:7443/v1/agent, in 0.02 seconds) (finished: subtest: certificate not listed under agents cannot connect as agent, in 0.03 seconds) subtest: metrics relay: must succeed: curl -sf http://127.0.0.1:7400/metrics relay: (finished: must succeed: curl -sf http://127.0.0.1:7400/metrics, in 0.02 seconds) (finished: subtest: metrics, in 0.02 seconds) subtest: agent link drop mid-run replays missed lines once agent: must succeed: jq '.services.app.prebuilt="/nix/store/c67jj8kx3m50nyj874aqmdqf21w5iznd-flakelet-app-slow"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json agent: (finished: must succeed: jq '.services.app.prebuilt="/nix/store/c67jj8kx3m50nyj874aqmdqf21w5iznd-flakelet-app-slow"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json, in 0.02 seconds) agent: must succeed: date +%s agent: (finished: must succeed: date +%s, in 0.01 seconds) client: must succeed: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy agent/app > /tmp/slow.out 2>&1 & client: (finished: must succeed: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy agent/app > /tmp/slow.out 2>&1 &, in 0.01 seconds) agent: waiting for success: journalctl -u flakelet-relay-job-app.service --since=@1789041545 | grep -q 'app-slow' agent # [ 1464.270789] agent flakelet-agent[1832]: {"timestamp":"2026-09-10T11:59:05.765490Z","level":"INFO","message":"start","id":"09ebd6d801013f590a28165672b4f5a7","flakelet":"app","rule":"admins","relay":"relay","caller_name":"\"admin@example.org\"","target":"flakelet_relay::agent::conn"} agent # [ 1464.271478] agent flakelet-agent[1832]: {"timestamp":"2026-09-10T11:59:05.766176Z","level":"INFO","message":"update starting","flakelet":"app","ids":"[\"09ebd6d801013f590a28165672b4f5a7\"]","target":"flakelet_relay::agent::jobs"} agent # [ 1464.319311] agent systemd[1]: Starting flakelet update app (via relay)... agent # [ 1464.335473] agent systemd[1]: Started flakelet update app (via relay). agent # [ 1464.340254] agent flakelet[2554]: app: using prebuilt artifact /nix/store/c67jj8kx3m50nyj874aqmdqf21w5iznd-flakelet-app-slow agent # [ 1464.340972] agent flakelet[2554]: app: activating generation 7 agent # [ 1464.363583] agent systemd[1]: Stopping app v1... agent # [ 1464.364069] agent systemd[1]: app.service: Deactivated successfully. agent # [ 1464.364424] agent systemd[1]: Stopped app v1. agent # [ 1464.385599] agent systemd[1]: Reload requested from client PID 2561 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [ 1464.385684] agent systemd[1]: Reloading... relay # [ 1464.204759] relay flakelet-relay[400]: {"timestamp":"2026-09-10T11:59:05.699460Z","level":"INFO","message":"agent rejected: no agents entry matches","principals":"[\"x509:dns:mallory\"]","target":"flakelet_relay::relay::agent_conn"} relay # [ 1464.270402] relay flakelet-relay[400]: {"timestamp":"2026-09-10T11:59:05.765039Z","level":"INFO","message":"deploy accepted","job":"268920a44215332a6af584190e3153f1","caller":"admin@example.org","targets":"[\"agent/app\"]","target":"flakelet_relay::relay::api"} agent # [ 1464.739437] agent systemd[1]: Reloading finished in 352 ms. agent # [ 1464.784586] agent systemd[1]: Reload requested from client PID 2591 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [ 1464.784674] agent systemd[1]: Reloading... agent: (finished: waiting for success: journalctl -u flakelet-relay-job-app.service --since=@1789041545 | grep -q 'app-slow', in 1.03 seconds) agent: must succeed: for a in $(getent ahosts relay | cut -d' ' -f1 | sort -u); do ip route add unreachable $a; done agent: (finished: must succeed: for a in $(getent ahosts relay | cut -d' ' -f1 | sort -u); do ip route add unreachable $a; done, in 0.02 seconds) relay: must succeed: for a in $(getent ahosts agent | cut -d' ' -f1 | sort -u); do ss -K dst "[$a]"; done relay: (finished: must succeed: for a in $(getent ahosts agent | cut -d' ' -f1 | sort -u); do ss -K dst "[$a]"; done, in 0.06 seconds) agent: must succeed: ss -K dport = 7443 agent # [ 1465.129140] agent systemd[1]: Reloading finished in 343 ms. agent # [ 1465.169863] agent systemd[1]: Reload requested from client PID 2620 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [ 1465.169911] agent systemd[1]: Reloading... agent: (finished: must succeed: ss -K dport = 7443, in 0.03 seconds) agent: waiting for success: journalctl -u flakelet-relay-job-app.service --since=@1789041545 | grep -q 'updated to generation' relay # [ 1465.355651] relay flakelet-relay[400]: {"timestamp":"2026-09-10T11:59:06.850346Z","level":"INFO","message":"agent disconnected: IO error: Software caused connection abort (os error 103)","host":"agent","target":"flakelet_relay::relay::agent_conn"} agent # [ 1465.386707] agent flakelet-agent[1832]: {"timestamp":"2026-09-10T11:59:06.881190Z","level":"INFO","message":"disconnected: IO error: Software caused connection abort (os error 103)","relay":"https://relay:7443","target":"flakelet_relay::agent::conn"} agent # [ 1465.512088] agent systemd[1]: Reloading finished in 341 ms. agent # [ 1465.557339] agent systemd[1]: Reload requested from client PID 2668 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [ 1465.557379] agent systemd[1]: Reloading... agent # [ 1465.903216] agent systemd[1]: Reloading finished in 345 ms. agent # [ 1465.949593] agent systemd[1]: Starting app slow... agent # [ 1466.389147] agent flakelet-agent[1832]: {"timestamp":"2026-09-10T11:59:07.883768Z","level":"WARN","message":"connect failed: connect relay:7443: No route to host (os error 113)","relay":"https://relay:7443","target":"flakelet_relay::agent::conn"} agent # [ 1468.392166] agent flakelet-agent[1832]: {"timestamp":"2026-09-10T11:59:09.886769Z","level":"WARN","message":"connect failed: connect relay:7443: No route to host (os error 113)","relay":"https://relay:7443","target":"flakelet_relay::agent::conn"} agent # [ 1471.990736] agent systemd[1]: Started app slow. agent # [ 1472.000477] agent flakelet[2554]: app: updated to generation 7 agent # [ 1472.004045] agent systemd[1]: flakelet-relay-job-app.service: Deactivated successfully. agent: (finished: waiting for success: journalctl -u flakelet-relay-job-app.service --since=@1789041545 | grep -q 'updated to generation', in 7.11 seconds) agent: must succeed: for a in $(getent ahosts relay | cut -d' ' -f1 | sort -u); do ip route del unreachable $a; done agent: (finished: must succeed: for a in $(getent ahosts relay | cut -d' ' -f1 | sort -u); do ip route del unreachable $a; done, in 0.02 seconds) client: waiting for success: grep -q 'agent/app: updated' /tmp/slow.out agent # [ 1472.394579] agent flakelet-agent[1832]: {"timestamp":"2026-09-10T11:59:13.889258Z","level":"WARN","message":"connect failed: connect relay:7443: No route to host (os error 113)","relay":"https://relay:7443","target":"flakelet_relay::agent::conn"} agent # [ 1473.131556] agent flakelet-agent[1832]: {"timestamp":"2026-09-10T11:59:14.626235Z","level":"INFO","message":"update finished","flakelet":"app","status":"Updated","generation":"Some(7)","target":"flakelet_relay::agent::jobs"} client: (finished: waiting for success: grep -q 'agent/app: updated' /tmp/slow.out, in 8.10 seconds) client: must succeed: cat /tmp/slow.out client: (finished: must succeed: cat /tmp/slow.out, in 0.01 seconds) (finished: subtest: agent link drop mid-run replays missed lines once, in 16.40 seconds) subtest: agent restart mid-run reattaches to the unit agent: must succeed: jq '.services.app.prebuilt="/nix/store/ln47rsd5km8a04hd5zlkz4q1v352zqh3-flakelet-app-slow2"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json agent: (finished: must succeed: jq '.services.app.prebuilt="/nix/store/ln47rsd5km8a04hd5zlkz4q1v352zqh3-flakelet-app-slow2"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json, in 0.01 seconds) client: must succeed: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy agent/app > /tmp/restart.out 2>&1 & client: (finished: must succeed: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy agent/app > /tmp/restart.out 2>&1 &, in 0.01 seconds) agent: waiting for success: journalctl -u flakelet-relay-job-app.service --since -20s | grep -q 'app-slow2' relay # [ 1480.399451] relay flakelet-relay[400]: {"timestamp":"2026-09-10T11:59:21.894069Z","level":"INFO","message":"agent connected","host":"agent","version":"0.1.0","flakelets":"[Named { name: \"app\", generation: Some(7), revision: Some(\"prebuilt:app\") }, Named { name: \"flakelet-agent\", generation: Some(1), revision: Some(\"prebuilt:flakelet-agent\") }, Named { name: \"other\", generation: Some(1), revision: Some(\"prebuilt:other\") }]","target":"flakelet_relay::relay::agent_conn"} relay # [ 1480.400788] relay flakelet-relay[400]: {"timestamp":"2026-09-10T11:59:21.895548Z","level":"INFO","message":"deploy finished","job":"268920a44215332a6af584190e3153f1","ok":true,"target":"flakelet_relay::relay::api"} relay # [ 1480.663580] relay flakelet-relay[400]: {"timestamp":"2026-09-10T11:59:22.158219Z","level":"INFO","message":"deploy accepted","job":"e4344d70047e670ef02f669aacb2f71f","caller":"admin@example.org","targets":"[\"agent/app\"]","target":"flakelet_relay::relay::api"} agent # [ 1480.399264] agent flakelet-agent[1832]: {"timestamp":"2026-09-10T11:59:21.893947Z","level":"INFO","message":"connected","relay":"https://relay:7443","relay_name":"relay","relay_version":"0.1.0","host":"agent","target":"flakelet_relay::agent::conn"} agent # [ 1480.400511] agent flakelet-agent[1832]: {"timestamp":"2026-09-10T11:59:21.895272Z","level":"INFO","message":"start","id":"09ebd6d801013f590a28165672b4f5a7","flakelet":"app","rule":"admins","relay":"relay","caller_name":"\"admin@example.org\"","target":"flakelet_relay::agent::conn"} agent # [ 1480.663926] agent flakelet-agent[1832]: {"timestamp":"2026-09-10T11:59:22.158553Z","level":"INFO","message":"start","id":"c9cfdf97a219aa07852fe7bcdcba1b44","flakelet":"app","rule":"admins","relay":"relay","caller_name":"\"admin@example.org\"","target":"flakelet_relay::agent::conn"} agent # [ 1480.664451] agent flakelet-agent[1832]: {"timestamp":"2026-09-10T11:59:22.159211Z","level":"INFO","message":"update starting","flakelet":"app","ids":"[\"c9cfdf97a219aa07852fe7bcdcba1b44\"]","target":"flakelet_relay::agent::jobs"} agent # [ 1480.708945] agent systemd[1]: Starting flakelet update app (via relay)... agent # [ 1480.726283] agent systemd[1]: Started flakelet update app (via relay). agent # [ 1480.730841] agent flakelet[2790]: app: using prebuilt artifact /nix/store/ln47rsd5km8a04hd5zlkz4q1v352zqh3-flakelet-app-slow2 agent # [ 1480.731635] agent flakelet[2790]: app: activating generation 8 agent # [ 1480.752393] agent systemd[1]: Stopping app slow... agent # [ 1480.752900] agent systemd[1]: app.service: Deactivated successfully. agent # [ 1480.753251] agent systemd[1]: Stopped app slow. agent # [ 1480.773722] agent systemd[1]: Reload requested from client PID 2797 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [ 1480.773767] agent systemd[1]: Reloading... agent # [ 1481.125269] agent systemd[1]: Reloading finished in 351 ms. agent # [ 1481.176670] agent systemd[1]: Reload requested from client PID 2827 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [ 1481.176704] agent systemd[1]: Reloading... agent # [ 1481.519423] agent systemd[1]: Reloading finished in 342 ms. agent: (finished: waiting for success: journalctl -u flakelet-relay-job-app.service --since -20s | grep -q 'app-slow2', in 1.03 seconds) agent: must succeed: systemctl restart flakelet-agent agent # [ 1481.565490] agent systemd[1]: Reload requested from client PID 2859 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [ 1481.565521] agent systemd[1]: Reloading... agent # [ 1481.912253] agent systemd[1]: Reloading finished in 346 ms. agent # [ 1481.960857] agent systemd[1]: Stopping flakelet-agent... agent # [ 1481.970678] agent systemd[1]: Reload requested from client PID 2895 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [ 1481.970765] agent systemd[1]: Reloading... agent # [ 1482.323591] agent systemd[1]: Reloading finished in 352 ms. relay # [ 1481.961474] relay flakelet-relay[400]: {"timestamp":"2026-09-10T11:59:23.456136Z","level":"INFO","message":"agent disconnected: IO error: peer closed connection without sending TLS close_notify: https://docs.rs/rustls/latest/rustls/manual/_03_howto/index.html#unexpected-eof","host":"agent","target":"flakelet_relay::relay::agent_conn"} agent: (finished: must succeed: systemctl restart flakelet-agent, in 0.77 seconds) agent: must succeed: test -n "$(ls /var/lib/flakelet-agent/jobs)" agent: (finished: must succeed: test -n "$(ls /var/lib/flakelet-agent/jobs)", in 0.01 seconds) client: waiting for success: grep -q 'agent/app: updated' /tmp/restart.out agent # [ 1482.358513] agent systemd[1]: flakelet-agent.service: Deactivated successfully. agent # [ 1482.358824] agent systemd[1]: Stopped flakelet-agent. agent # [ 1482.358839] agent systemd[1]: flakelet-agent.service: Consumed 1.242s CPU time over 29.153s wall clock time. agent # [ 1482.376804] agent systemd[1]: Starting flakelet-agent... agent # [ 1482.378212] agent systemd[1]: Starting app slow2... agent # [ 1482.421524] agent flakelet-agent[2926]: {"timestamp":"2026-09-10T11:59:23.916226Z","level":"INFO","message":"reattaching to running unit","flakelet":"app","ids":"[\"c9cfdf97a219aa07852fe7bcdcba1b44\"]","target":"flakelet_relay::agent::jobs"} agent # [ 1482.445502] agent systemd[1]: Started flakelet-agent. agent # [ 1482.458667] agent flakelet-agent[2926]: {"timestamp":"2026-09-10T11:59:23.953404Z","level":"INFO","message":"connected","relay":"https://relay:7443","relay_name":"relay","relay_version":"0.1.0","host":"agent","target":"flakelet_relay::agent::conn"} agent # [ 1482.459933] agent flakelet-agent[2926]: {"timestamp":"2026-09-10T11:59:23.954699Z","level":"INFO","message":"start","id":"c9cfdf97a219aa07852fe7bcdcba1b44","flakelet":"app","rule":"admins","relay":"relay","caller_name":"\"admin@example.org\"","target":"flakelet_relay::agent::conn"} relay # [ 1482.458831] relay flakelet-relay[400]: {"timestamp":"2026-09-10T11:59:23.953515Z","level":"INFO","message":"agent connected","host":"agent","version":"0.1.0","flakelets":"[Named { name: \"app\", generation: Some(7), revision: Some(\"prebuilt:app\") }, Named { name: \"flakelet-agent\", generation: Some(1), revision: Some(\"prebuilt:flakelet-agent\") }, Named { name: \"other\", generation: Some(1), revision: Some(\"prebuilt:other\") }]","target":"flakelet_relay::relay::agent_conn"} agent # [ 1488.425764] agent systemd[1]: Started app slow2. agent # [ 1488.433938] agent flakelet[2790]: app: updated to generation 8 agent # [ 1488.436686] agent systemd[1]: flakelet-relay-job-app.service: Deactivated successfully. agent # [ 1489.007922] agent flakelet-agent[2926]: {"timestamp":"2026-09-10T11:59:30.502568Z","level":"INFO","message":"update finished","flakelet":"app","status":"Updated","generation":"Some(8)","target":"flakelet_relay::agent::jobs"} relay # [ 1489.009636] relay flakelet-relay[400]: {"timestamp":"2026-09-10T11:59:30.504280Z","level":"INFO","message":"deploy finished","job":"e4344d70047e670ef02f669aacb2f71f","ok":true,"target":"flakelet_relay::relay::api"} client: (finished: waiting for success: grep -q 'agent/app: updated' /tmp/restart.out, in 7.09 seconds) client: must succeed: cat /tmp/restart.out client: (finished: must succeed: cat /tmp/restart.out, in 0.01 seconds) agent: must succeed: systemctl is-active app.service agent: (finished: must succeed: systemctl is-active app.service, in 0.01 seconds) (finished: subtest: agent restart mid-run reattaches to the unit, in 8.94 seconds) subtest: relay restart mid-run, push resumes via /v1/jobs agent: must succeed: jq '.services.app.prebuilt="/nix/store/c67jj8kx3m50nyj874aqmdqf21w5iznd-flakelet-app-slow"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json agent: (finished: must succeed: jq '.services.app.prebuilt="/nix/store/c67jj8kx3m50nyj874aqmdqf21w5iznd-flakelet-app-slow"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json, in 0.01 seconds) agent: must succeed: date +%s agent: (finished: must succeed: date +%s, in 0.01 seconds) client: must succeed: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy agent/app > /tmp/resume.out 2>&1 & client: (finished: must succeed: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy agent/app > /tmp/resume.out 2>&1 &, in 0.01 seconds) agent: waiting for success: journalctl -u flakelet-relay-job-app.service --since=@1789041571 | grep -q 'app-slow' relay # [ 1489.609132] relay flakelet-relay[400]: {"timestamp":"2026-09-10T11:59:31.103818Z","level":"INFO","message":"deploy accepted","job":"65371d9584bd2579bd2aabe594e6ce4c","caller":"admin@example.org","targets":"[\"agent/app\"]","target":"flakelet_relay::relay::api"} agent # [ 1489.609570] agent flakelet-agent[2926]: {"timestamp":"2026-09-10T11:59:31.104262Z","level":"INFO","message":"start","id":"55f95a60ef65eaac6f17504b6c7a5fc0","flakelet":"app","rule":"admins","relay":"relay","caller_name":"\"admin@example.org\"","target":"flakelet_relay::agent::conn"} agent # [ 1489.610234] agent flakelet-agent[2926]: {"timestamp":"2026-09-10T11:59:31.104993Z","level":"INFO","message":"update starting","flakelet":"app","ids":"[\"55f95a60ef65eaac6f17504b6c7a5fc0\"]","target":"flakelet_relay::agent::jobs"} agent # [ 1489.651634] agent systemd[1]: Starting flakelet update app (via relay)... agent # [ 1489.679397] agent systemd[1]: Started flakelet update app (via relay). agent # [ 1489.683871] agent flakelet[3372]: app: using prebuilt artifact /nix/store/c67jj8kx3m50nyj874aqmdqf21w5iznd-flakelet-app-slow agent # [ 1489.684559] agent flakelet[3372]: app: activating generation 9 agent # [ 1489.705850] agent systemd[1]: Stopping app slow2... agent # [ 1489.706388] agent systemd[1]: app.service: Deactivated successfully. agent # [ 1489.706734] agent systemd[1]: Stopped app slow2. agent # [ 1489.728456] agent systemd[1]: Reload requested from client PID 3379 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [ 1489.728540] agent systemd[1]: Reloading... agent # [ 1490.093256] agent systemd[1]: Reloading finished in 363 ms. agent # [ 1490.171521] agent systemd[1]: Reload requested from client PID 3409 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [ 1490.171574] agent systemd[1]: Reloading... agent: (finished: waiting for success: journalctl -u flakelet-relay-job-app.service --since=@1789041571 | grep -q 'app-slow', in 1.02 seconds) relay: must succeed: systemctl restart flakelet-relay agent # [ 1490.533794] agent systemd[1]: Reloading finished in 361 ms. agent # [ 1490.609104] agent systemd[1]: Reload requested from client PID 3440 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [ 1490.609143] agent systemd[1]: Reloading... agent # [ 1490.632795] agent flakelet-agent[2926]: {"timestamp":"2026-09-10T11:59:32.127478Z","level":"INFO","message":"disconnected: IO error: peer closed connection without sending TLS close_notify: https://docs.rs/rustls/latest/rustls/manual/_03_howto/index.html#unexpected-eof","relay":"https://relay:7443","target":"flakelet_relay::agent::conn"} relay # [ 1490.632147] relay systemd[1]: Stopping flakelet-relay... relay # [ 1490.663042] relay systemd[1]: flakelet-relay.service: Deactivated successfully. relay # [ 1490.663670] relay systemd[1]: Stopped flakelet-relay. relay # [ 1490.663713] relay systemd[1]: flakelet-relay.service: Consumed 2.192s CPU time over 1min 6.797s wall clock time. relay # [ 1490.711859] relay systemd[1]: Starting flakelet-relay... agent # [ 1490.968345] agent systemd[1]: Reloading finished in 358 ms. agent # [ 1491.060889] agent systemd[1]: Reload requested from client PID 3476 ('systemctl') (unit flakelet-relay-job-app.service)... agent # [ 1491.060932] agent systemd[1]: Reloading... agent # [ 1491.417032] agent systemd[1]: Reloading finished in 355 ms. agent # [ 1491.498055] agent systemd[1]: Starting app slow... agent # [ 1491.636416] agent flakelet-agent[2926]: {"timestamp":"2026-09-10T11:59:33.131124Z","level":"WARN","message":"connect failed: connect relay:7443: Connection refused (os error 111)","relay":"https://relay:7443","target":"flakelet_relay::agent::conn"} relay: (finished: must succeed: systemctl restart flakelet-relay, in 1.55 seconds) client: waiting for success: grep -q '» ok' /tmp/resume.out relay # [ 1492.167188] relay flakelet-relay[930]: {"timestamp":"2026-09-10T11:59:33.661927Z","level":"INFO","message":"listening on 127.0.0.1:7400","target":"flakelet_relay::relay::server"} relay # [ 1492.167677] relay flakelet-relay[930]: {"timestamp":"2026-09-10T11:59:33.662413Z","level":"INFO","message":"listening on [::]:7443","target":"flakelet_relay::relay::server"} relay # [ 1492.167802] relay systemd[1]: Started flakelet-relay. agent # [ 1493.646762] agent flakelet-agent[2926]: {"timestamp":"2026-09-10T11:59:35.141399Z","level":"INFO","message":"connected","relay":"https://relay:7443","relay_name":"relay","relay_version":"0.1.0","host":"agent","target":"flakelet_relay::agent::conn"} relay # [ 1493.646731] relay flakelet-relay[930]: {"timestamp":"2026-09-10T11:59:35.141443Z","level":"INFO","message":"agent connected","host":"agent","version":"0.1.0","flakelets":"[Named { name: \"app\", generation: Some(8), revision: Some(\"prebuilt:app\") }, Named { name: \"flakelet-agent\", generation: Some(1), revision: Some(\"prebuilt:flakelet-agent\") }, Named { name: \"other\", generation: Some(1), revision: Some(\"prebuilt:other\") }]","target":"flakelet_relay::relay::agent_conn"} agent # [ 1497.551287] agent systemd[1]: Started app slow. agent # [ 1497.560543] agent flakelet[3372]: app: updated to generation 9 agent # [ 1497.564149] agent systemd[1]: flakelet-relay-job-app.service: Deactivated successfully. relay # [ 1497.647830] relay flakelet-relay[930]: {"timestamp":"2026-09-10T11:59:39.142547Z","level":"INFO","message":"job reattached","job":"65371d9584bd2579bd2aabe594e6ce4c","targets":"[\"agent/app\"]","target":"flakelet_relay::relay::api"} agent # [ 1498.632566] agent flakelet-agent[2926]: {"timestamp":"2026-09-10T11:59:40.127258Z","level":"INFO","message":"update finished","flakelet":"app","status":"Updated","generation":"Some(9)","target":"flakelet_relay::agent::jobs"} client: (finished: waiting for success: grep -q '» ok' /tmp/resume.out, in 7.08 seconds) client: must succeed: cat /tmp/resume.out client: (finished: must succeed: cat /tmp/resume.out, in 0.01 seconds) (finished: subtest: relay restart mid-run, push resumes via /v1/jobs, in 9.69 seconds) subtest: job history comes from the agents and survives a relay restart client: must succeed: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy --id hist1 agent/other 2>&1 agent # [ 1499.282299] agent flakelet-agent[2926]: {"timestamp":"2026-09-10T11:59:40.777023Z","level":"INFO","message":"start","id":"5be20c693e5c0b60e3f85779794dfb86","flakelet":"other","rule":"admins","relay":"relay","caller_name":"\"admin@example.org\"","target":"flakelet_relay::agent::conn"} agent # [ 1499.282787] agent flakelet-agent[2926]: {"timestamp":"2026-09-10T11:59:40.777554Z","level":"INFO","message":"update starting","flakelet":"other","ids":"[\"5be20c693e5c0b60e3f85779794dfb86\"]","target":"flakelet_relay::agent::jobs"} agent # [ 1499.324688] agent systemd[1]: Starting flakelet update other (via relay)... agent # [ 1499.347843] agent systemd[1]: Started flakelet update other (via relay). agent # [ 1499.352237] agent flakelet[3550]: other: using prebuilt artifact /nix/store/hmr990n0b4v6rpzrcpyyw738r9xi7z95-flakelet-other-v1 agent # [ 1499.352775] agent flakelet[3550]: other: up to date agent # [ 1499.356086] agent systemd[1]: flakelet-relay-job-other.service: Deactivated successfully. relay # [ 1499.281947] relay flakelet-relay[930]: {"timestamp":"2026-09-10T11:59:40.776634Z","level":"INFO","message":"deploy accepted","job":"188b06c4636b65849b3ba8daac22dc7c","caller":"admin@example.org","targets":"[\"agent/other\"]","target":"flakelet_relay::relay::api"} client: (finished: must succeed: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy --id hist1 agent/other 2>&1, in 0.64 seconds) relay: must succeed: systemctl restart flakelet-relay agent # [ 1499.897791] agent flakelet-agent[2926]: {"timestamp":"2026-09-10T11:59:41.392488Z","level":"INFO","message":"update finished","flakelet":"other","status":"Unchanged","generation":"Some(1)","target":"flakelet_relay::agent::jobs"} agent # [ 1499.913174] agent flakelet-agent[2926]: {"timestamp":"2026-09-10T11:59:41.407921Z","level":"INFO","message":"disconnected: IO error: peer closed connection without sending TLS close_notify: https://docs.rs/rustls/latest/rustls/manual/_03_howto/index.html#unexpected-eof","relay":"https://relay:7443","target":"flakelet_relay::agent::conn"} relay # [ 1499.899208] relay flakelet-relay[930]: {"timestamp":"2026-09-10T11:59:41.393938Z","level":"INFO","message":"deploy finished","job":"188b06c4636b65849b3ba8daac22dc7c","ok":true,"target":"flakelet_relay::relay::api"} relay # [ 1499.912626] relay systemd[1]: Stopping flakelet-relay... relay # [ 1499.940517] relay systemd[1]: flakelet-relay.service: Deactivated successfully. relay # [ 1499.940856] relay systemd[1]: Stopped flakelet-relay. relay # [ 1499.940885] relay systemd[1]: flakelet-relay.service: Consumed 1.749s CPU time over 9.228s wall clock time. relay # [ 1499.958141] relay systemd[1]: Starting flakelet-relay... agent # [ 1500.915922] agent flakelet-agent[2926]: {"timestamp":"2026-09-10T11:59:42.410599Z","level":"WARN","message":"connect failed: connect relay:7443: Connection refused (os error 111)","relay":"https://relay:7443","target":"flakelet_relay::agent::conn"} relay: (finished: must succeed: systemctl restart flakelet-relay, in 1.46 seconds) relay: waiting for success: curl -sf http://127.0.0.1:7400/metrics | grep -q 'agent_up{host="agent"} 1' relay # [ 1501.355340] relay flakelet-relay[1320]: {"timestamp":"2026-09-10T11:59:42.849978Z","level":"INFO","message":"listening on 127.0.0.1:7400","target":"flakelet_relay::relay::server"} relay # [ 1501.356260] relay flakelet-relay[1320]: {"timestamp":"2026-09-10T11:59:42.850358Z","level":"INFO","message":"listening on [::]:7443","target":"flakelet_relay::relay::server"} relay # [ 1501.355816] relay systemd[1]: Started flakelet-relay. agent # [ 1502.929653] agent flakelet-agent[2926]: {"timestamp":"2026-09-10T11:59:44.424335Z","level":"INFO","message":"connected","relay":"https://relay:7443","relay_name":"relay","relay_version":"0.1.0","host":"agent","target":"flakelet_relay::agent::conn"} relay # [ 1502.929809] relay flakelet-relay[1320]: {"timestamp":"2026-09-10T11:59:44.424439Z","level":"INFO","message":"agent connected","host":"agent","version":"0.1.0","flakelets":"[Named { name: \"app\", generation: Some(9), revision: Some(\"prebuilt:app\") }, Named { name: \"flakelet-agent\", generation: Some(1), revision: Some(\"prebuilt:flakelet-agent\") }, Named { name: \"other\", generation: Some(1), revision: Some(\"prebuilt:other\") }]","target":"flakelet_relay::relay::agent_conn"} relay: (finished: waiting for success: curl -sf http://127.0.0.1:7400/metrics | grep -q 'agent_up{host="agent"} 1', in 2.06 seconds) client: must succeed: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem jobs client: (finished: must succeed: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem jobs, in 0.02 seconds) client: must succeed: echo '{"groups": [], "repository": "example/app"}' | step crypto jwt sign --key /nix/store/r7sclamg09v434w8zjlfnrcrn1w79wh6-flakelet-relay-test-issuer/jwk.json --iss https://relay:8443 --aud flakelet-relay --sub 'repo:github:example/app:ref:refs/heads/main' --exp $(( $(date +%s) + 300 )) --jti $(head -c8 /dev/urandom | base32) client: (finished: must succeed: echo '{"groups": [], "repository": "example/app"}' | step crypto jwt sign --key /nix/store/r7sclamg09v434w8zjlfnrcrn1w79wh6-flakelet-relay-test-issuer/jwk.json --iss https://relay:8443 --aud flakelet-relay --sub 'repo:github:example/app:ref:refs/heads/main' --exp $(( $(date +%s) + 300 )) --jti $(head -c8 /dev/urandom | base32), in 0.05 seconds) client: must succeed: FLAKELET_RELAY_TOKEN_COMMAND='echo eyJhbGciOiJFZERTQSIsImtpZCI6ImsxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJmbGFrZWxldC1yZWxheSIsImV4cCI6MTc4OTA0MTg4NCwiZ3JvdXBzIjpbXSwiaWF0IjoxNzg5MDQxNTg0LCJpc3MiOiJodHRwczovL3JlbGF5Ojg0NDMiLCJqdGkiOiI2M1VSTU5ENkdPQ0RHPT09IiwibmJmIjoxNzg5MDQxNTg0LCJyZXBvc2l0b3J5IjoiZXhhbXBsZS9hcHAiLCJzdWIiOiJyZXBvOmdpdGh1YjpleGFtcGxlL2FwcDpyZWY6cmVmcy9oZWFkcy9tYWluIn0.GkDyO3AWhoCVOrSMan_4sBnm0ZNcdtXCeX-UOtyum-fW3Q4Dt8VwI5JGcpc_eh5GPOu6mTL82WhPdhqvEBiKCA' push --relay https://relay:7443 deploy --id hist2 agent/app » job 6a239dbf87cfec796e49404c351d787f via relay 0.1.0 (agents: agent 0.1.0) » wave 0 agent/app | Starting flakelet update app (via relay)... agent/app | Started flakelet update app (via relay). agent # [ 1503.506699] agent flakelet-agent[2926]: {"timestamp":"2026-09-10T11:59:45.001419Z","level":"INFO","message":"start","id":"dce6faec8455f85321e72ee634888126","flakelet":"app","rule":"ci","relay":"relay","caller_name":"\"example/app\"","target":"flakelet_relay::agent::conn"} agent # [ 1503.507347] agent flakelet-agent[2926]: {"timestamp":"2026-09-10T11:59:45.002107Z","level":"INFO","message":"update starting","flakelet":"app","ids":"[\"dce6faec8455f85321e72ee634888126\"]","target":"flakelet_relay::agent::jobs"} agent # [ 1503.550564] agent systemd[1]: Starting flakelet update app (via relay)... agent # [ 1503.582295] agent systemd[1]: Started flakelet update app (via relay). agent # [ 1503.588833] agent flakelet[3578]: app: using prebuilt artifact /nix/store/c67jj8kx3m50nyj874aqmdqf21w5iznd-flakelet-app-slow agent # [ 1503.589690] agent flakelet[3578]: app: up to date agent # [ 1503.590587] agent systemd[1]: flakelet-relay-job-app.service: Deactivated successfully. agent/app | app: using prebuilt artifact /nix/store/c67jj8kx3m50nyj874aqmdqf21w5iznd-flakelet-app-slow agent/app | app: up to date agent/app | flakelet-relay-job-app.service: Deactivated successfully. relay # [ 1503.506371] relay flakelet-relay[1320]: {"timestamp":"2026-09-10T11:59:45.001079Z","level":"INFO","message":"deploy accepted","job":"6a239dbf87cfec796e49404c351d787f","caller":"example/app","targets":"[\"agent/app\"]","target":"flakelet_relay::relay::api"} » agent/app: unchanged (generation 9) » ok client: (finished: must succeed: FLAKELET_RELAY_TOKEN_COMMAND='echo eyJhbGciOiJFZERTQSIsImtpZCI6ImsxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJmbGFrZWxldC1yZWxheSIsImV4cCI6MTc4OTA0MTg4NCwiZ3JvdXBzIjpbXSwiaWF0IjoxNzg5MDQxNTg0LCJpc3MiOiJodHRwczovL3JlbGF5Ojg0NDMiLCJqdGkiOiI2M1VSTU5ENkdPQ0RHPT09IiwibmJmIjoxNzg5MDQxNTg0LCJyZXBvc2l0b3J5IjoiZXhhbXBsZS9hcHAiLCJzdWIiOiJyZXBvOmdpdGh1YjpleGFtcGxlL2FwcDpyZWY6cmVmcy9oZWFkcy9tYWluIn0.GkDyO3AWhoCVOrSMan_4sBnm0ZNcdtXCeX-UOtyum-fW3Q4Dt8VwI5JGcpc_eh5GPOu6mTL82WhPdhqvEBiKCA' push --relay https://relay:7443 deploy --id hist2 agent/app, in 0.65 seconds) client: must succeed: FLAKELET_RELAY_TOKEN_COMMAND='echo eyJhbGciOiJFZERTQSIsImtpZCI6ImsxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJmbGFrZWxldC1yZWxheSIsImV4cCI6MTc4OTA0MTg4NCwiZ3JvdXBzIjpbXSwiaWF0IjoxNzg5MDQxNTg0LCJpc3MiOiJodHRwczovL3JlbGF5Ojg0NDMiLCJqdGkiOiI2M1VSTU5ENkdPQ0RHPT09IiwibmJmIjoxNzg5MDQxNTg0LCJyZXBvc2l0b3J5IjoiZXhhbXBsZS9hcHAiLCJzdWIiOiJyZXBvOmdpdGh1YjpleGFtcGxlL2FwcDpyZWY6cmVmcy9oZWFkcy9tYWluIn0.GkDyO3AWhoCVOrSMan_4sBnm0ZNcdtXCeX-UOtyum-fW3Q4Dt8VwI5JGcpc_eh5GPOu6mTL82WhPdhqvEBiKCA' push --relay https://relay:7443 jobs client: (finished: must succeed: FLAKELET_RELAY_TOKEN_COMMAND='echo eyJhbGciOiJFZERTQSIsImtpZCI6ImsxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJmbGFrZWxldC1yZWxheSIsImV4cCI6MTc4OTA0MTg4NCwiZ3JvdXBzIjpbXSwiaWF0IjoxNzg5MDQxNTg0LCJpc3MiOiJodHRwczovL3JlbGF5Ojg0NDMiLCJqdGkiOiI2M1VSTU5ENkdPQ0RHPT09IiwibmJmIjoxNzg5MDQxNTg0LCJyZXBvc2l0b3J5IjoiZXhhbXBsZS9hcHAiLCJzdWIiOiJyZXBvOmdpdGh1YjpleGFtcGxlL2FwcDpyZWY6cmVmcy9oZWFkcy9tYWluIn0.GkDyO3AWhoCVOrSMan_4sBnm0ZNcdtXCeX-UOtyum-fW3Q4Dt8VwI5JGcpc_eh5GPOu6mTL82WhPdhqvEBiKCA' push --relay https://relay:7443 jobs, in 0.02 seconds) (finished: subtest: job history comes from the agents and survives a relay restart, in 4.90 seconds) subtest: dashboard login via authorization code flow client: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -o /dev/null -w '%{http_code} %{redirect_url}' https://relay:7443/ui/ | grep -q '303 https://relay:7443/ui/login' client: (finished: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -o /dev/null -w '%{http_code} %{redirect_url}' https://relay:7443/ui/ | grep -q '303 https://relay:7443/ui/login', in 0.02 seconds) client: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -L https://relay:7443/ui/login client: (finished: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -L https://relay:7443/ui/login, in 0.08 seconds) client: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar https://relay:7443/ui/jobs client: (finished: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar https://relay:7443/ui/jobs, in 0.02 seconds) client: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -f https://relay:7443/v1/jobs | grep -q hist1 client: (finished: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -f https://relay:7443/v1/jobs | grep -q hist1, in 0.02 seconds) client: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -N --max-time 10 https://relay:7443/ui/jobs/hist1/events || true client: (finished: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -N --max-time 10 https://relay:7443/ui/jobs/hist1/events || true, in 0.02 seconds) client: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -o /dev/null -w '%{http_code}' -X POST 'https://relay:7443/ui/deploy?arg=other' | grep -q 403 client: (finished: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -o /dev/null -w '%{http_code}' -X POST 'https://relay:7443/ui/deploy?arg=other' | grep -q 403, in 0.02 seconds) client: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -D - -o /dev/null -H 'HX-Request: true' -X POST 'https://relay:7443/ui/deploy?arg=other' client: (finished: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -D - -o /dev/null -H 'HX-Request: true' -X POST 'https://relay:7443/ui/deploy?arg=other', in 0.02 seconds) client: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -N --max-time 30 https://relay:7443/ui/jobs/409fa61c0dd5f5539c05f1b5f93d2d3c/events || true agent # [ 1504.131803] agent flakelet-agent[2926]: {"timestamp":"2026-09-10T11:59:45.626453Z","level":"INFO","message":"update finished","flakelet":"app","status":"Unchanged","generation":"Some(9)","target":"flakelet_relay::agent::jobs"} agent # [ 1504.356636] agent flakelet-agent[2926]: {"timestamp":"2026-09-10T11:59:45.851342Z","level":"INFO","message":"start","id":"1bb6826ef46d44444caefdf298260e8b","flakelet":"other","rule":"admins","relay":"relay","caller_name":"\"dev@example.org\"","target":"flakelet_relay::agent::conn"} agent # [ 1504.356945] agent flakelet-agent[2926]: {"timestamp":"2026-09-10T11:59:45.851713Z","level":"INFO","message":"update starting","flakelet":"other","ids":"[\"1bb6826ef46d44444caefdf298260e8b\"]","target":"flakelet_relay::agent::jobs"} agent # [ 1504.398400] agent systemd[1]: Starting flakelet update other (via relay)... relay # [ 1504.133573] relay flakelet-relay[1320]: {"timestamp":"2026-09-10T11:59:45.628264Z","level":"INFO","message":"deploy finished","job":"6a239dbf87cfec796e49404c351d787f","ok":true,"target":"flakelet_relay::relay::api"} relay # [ 1504.210944] relay 93sn3sv24fj26ksccpppzwg6kbf10s4w-device-mock[185]: 127.0.0.1 - - [10/Sep/2026 11:59:45] "GET /authorize?response_type=code&client_id=dashboard&redirect_uri=https%3A%2F%2Frelay%3A7443%2Fui%2Fcallback&scope=openid%20profile%20email%20groups&state=4-IkFaEe04dmhP0tDJsmuyMIcWa1dPv5VBmb0uo5o_g&code_challenge=TItANQlUxqsZOFZrNiYTKHdGglC7nRpSiT5wymyGETM&code_challenge_method=S256 HTTP/1.1" 302 - relay # [ 1504.255279] relay 93sn3sv24fj26ksccpppzwg6kbf10s4w-device-mock[185]: 127.0.0.1 - - [10/Sep/2026 11:59:45] "POST /token HTTP/1.1" 200 - relay # [ 1504.256443] relay flakelet-relay[1320]: {"timestamp":"2026-09-10T11:59:45.751186Z","level":"INFO","message":"login","name":"dev@example.org","principals":"[\"oidc:mock:someone\", \"oidc:mock:groups:wheel\"]","target":"flakelet_relay::relay::login"} relay # [ 1504.317315] relay flakelet-relay[1320]: {"timestamp":"2026-09-10T11:59:45.812059Z","level":"INFO","message":"job reattached","job":"188b06c4636b65849b3ba8daac22dc7c","targets":"[\"agent/other\"]","target":"flakelet_relay::relay::api"} relay # [ 1504.356346] relay flakelet-relay[1320]: {"timestamp":"2026-09-10T11:59:45.851058Z","level":"INFO","message":"deploy accepted","job":"acc10860c13e11179d515ecd80f28c62","caller":"dev@example.org","targets":"[\"agent/other\"]","target":"flakelet_relay::relay::api"} relay # [ 1504.376683] relay flakelet-relay[1320]: {"timestamp":"2026-09-10T11:59:45.871370Z","level":"INFO","message":"job reattached","job":"acc10860c13e11179d515ecd80f28c62","targets":"[\"agent/other\"]","target":"flakelet_relay::relay::api"} agent # [ 1504.424729] agent systemd[1]: Started flakelet update other (via relay). agent # [ 1504.429409] agent flakelet[3598]: other: using prebuilt artifact /nix/store/hmr990n0b4v6rpzrcpyyw738r9xi7z95-flakelet-other-v1 agent # [ 1504.429946] agent flakelet[3598]: other: up to date agent # [ 1504.433204] agent systemd[1]: flakelet-relay-job-other.service: Deactivated successfully. client: (finished: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -N --max-time 30 https://relay:7443/ui/jobs/409fa61c0dd5f5539c05f1b5f93d2d3c/events || true, in 0.63 seconds) client: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar https://relay:7443/ui/jobs/409fa61c0dd5f5539c05f1b5f93d2d3c client: (finished: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar https://relay:7443/ui/jobs/409fa61c0dd5f5539c05f1b5f93d2d3c, in 0.02 seconds) client: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -sf -o /dev/null https://relay:7443/ui/static/htmx.min.js client: (finished: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -sf -o /dev/null https://relay:7443/ui/static/htmx.min.js, in 0.02 seconds) client: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar 'https://relay:7443/ui/?q=status%3Ahealthy+oth' client: (finished: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar 'https://relay:7443/ui/?q=status%3Ahealthy+oth', in 0.02 seconds) client: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar https://relay:7443/ui/flakelets/other client: (finished: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar https://relay:7443/ui/flakelets/other, in 0.02 seconds) client: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar 'https://relay:7443/ui/hosts?q=status%3Adisconnected' client: (finished: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar 'https://relay:7443/ui/hosts?q=status%3Adisconnected', in 0.02 seconds) client: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -o /dev/null -w '%{http_code}' -H 'HX-Request: true' -X POST 'https://relay:7443/ui/retry?arg=409fa61c0dd5f5539c05f1b5f93d2d3c' | grep -q 400 client: (finished: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -o /dev/null -w '%{http_code}' -H 'HX-Request: true' -X POST 'https://relay:7443/ui/retry?arg=409fa61c0dd5f5539c05f1b5f93d2d3c' | grep -q 400, in 0.02 seconds) client: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -N --max-time 15 'https://relay:7443/ui/events?path=jobs' > /tmp/ev.out 2>&1 & client: (finished: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -N --max-time 15 'https://relay:7443/ui/events?path=jobs' > /tmp/ev.out 2>&1 &, in 0.01 seconds) client: must succeed: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy --id live1 agent/other » job 38796a7ba508eadfe82a1cd5def000bc via relay 0.1.0 (agents: agent 0.1.0) » wave 0 agent/other | Starting flakelet update other (via relay)... agent/other | Started flakelet update other (via relay). agent/other | other: using prebuilt artifact /nix/store/hmr990n0b4v6rpzrcpyyw738r9xi7z95-flakelet-other-v1 agent/other | other: up to date agent # [ 1504.977774] agent flakelet-agent[2926]: {"timestamp":"2026-09-10T11:59:46.472459Z","level":"INFO","message":"update finished","flakelet":"other","status":"Unchanged","generation":"Some(1)","target":"flakelet_relay::agent::jobs"} agent # [ 1505.124872] agent flakelet-agent[2926]: {"timestamp":"2026-09-10T11:59:46.619558Z","level":"INFO","message":"start","id":"5e317fc57863c2a7c03ebe1bdacd1356","flakelet":"other","rule":"admins","relay":"relay","caller_name":"\"admin@example.org\"","target":"flakelet_relay::agent::conn"} agent # [ 1505.125510] agent flakelet-agent[2926]: {"timestamp":"2026-09-10T11:59:46.620229Z","level":"INFO","message":"update starting","flakelet":"other","ids":"[\"5e317fc57863c2a7c03ebe1bdacd1356\"]","target":"flakelet_relay::agent::jobs"} agent # [ 1505.166904] agent systemd[1]: Starting flakelet update other (via relay)... agent # [ 1505.190594] agent systemd[1]: Started flakelet update other (via relay). agent # [ 1505.195138] agent flakelet[3622]: other: using prebuilt artifact /nix/store/hmr990n0b4v6rpzrcpyyw738r9xi7z95-flakelet-other-v1 agent # [ 1505.195859] agent flakelet[3622]: other: up to date agent # [ 1505.198672] agent systemd[1]: flakelet-relay-job-other.service: Deactivated successfully. agent/other | flakelet-relay-job-other.service: Deactivated successfully. relay # [ 1505.124574] relay flakelet-relay[1320]: {"timestamp":"2026-09-10T11:59:46.619261Z","level":"INFO","message":"deploy accepted","job":"38796a7ba508eadfe82a1cd5def000bc","caller":"admin@example.org","targets":"[\"agent/other\"]","target":"flakelet_relay::relay::api"} » agent/other: unchanged (generation 1) » ok client: (finished: must succeed: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy --id live1 agent/other, in 0.63 seconds) client: waiting for success: grep -q live1 /tmp/ev.out client: (finished: waiting for success: grep -q live1 /tmp/ev.out, in 0.01 seconds) client: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -X POST -o /dev/null https://relay:7443/ui/logout client: (finished: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -X POST -o /dev/null https://relay:7443/ui/logout, in 0.02 seconds) client: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -o /dev/null -w '%{http_code}' https://relay:7443/ui/ | grep -q 303 client: (finished: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -o /dev/null -w '%{http_code}' https://relay:7443/ui/ | grep -q 303, in 0.02 seconds) (finished: subtest: dashboard login via authorization code flow, in 1.63 seconds) (finished: run the VM test script, in 85.18 seconds) test script finished in 85.30s cleanup kill NspawnMachine (pid 52) kill NspawnMachine (pid 55) kill NspawnMachine (pid 56) Container agent terminated by signal KILL. Container client terminated by signal KILL. (finished: cleanup, in 0.24 seconds) additionally exposed symbols: agent, client, relay, vlan1, start_all, test_script, machines, machines_qemu, machines_nspawn, vlans, driver, log, os, create_machine, subtest, run_tests, join_all, retry, serial_stdout_off, serial_stdout_on, polling_condition, BaseMachine, QemuMachine, NspawnMachine, t, debug, dump_machine_ssh » job d214d3b7566948f7ad37c0917026fbdb via relay 0.1.0 (agents: agent 0.1.0) » wave 0 agent/app | Starting flakelet update app (via relay)... agent/app | Started flakelet update app (via relay). agent/app | app: using prebuilt artifact /nix/store/6ili1vxjlag9d7dy0bxrbxk6bpi5qh9p-flakelet-app-v2 agent/app | app: activating generation 2 agent/app | app: updated to generation 2 agent/app | flakelet-relay-job-app.service: Deactivated successfully. » agent/app: updated (generation 2) » ok » job cbe7d8d5edc7da1d1eb5c5828ea48613 via relay 0.1.0 (agents: agent 0.1.0) » wave 0 agent/app | Starting flakelet update app (via relay)... agent/app | Started flakelet update app (via relay). agent/app | app: using prebuilt artifact /nix/store/8282rnbvyk847q4l4zby3zwcq3jpzrvw-flakelet-app-broken agent/app | app: activating generation 4 agent/app | app: deploy failed, rolled back: systemctl start app.service failed: agent/app | failed units: app.service. See journalctl -u app.service agent/app | flakelet-relay-job-app.service: Main process exited, code=exited, status=1/FAILURE agent/app | flakelet-relay-job-app.service: Failed with result 'exit-code'. » agent/app: rolled-back (generation 3) agent/app │ Starting app v1... agent/app │ Started app v1. agent/app │ Stopping app v1... agent/app │ app.service: Deactivated successfully. agent/app │ Stopped app v1. agent/app │ Starting app v2... agent/app │ Started app v2. agent/app │ Stopping app v2... agent/app │ app.service: Deactivated successfully. agent/app │ Stopped app v2. agent/app │ Starting app v1... agent/app │ Started app v1. agent/app │ Stopping app v1... agent/app │ app.service: Deactivated successfully. agent/app │ Stopped app v1. agent/app │ Starting app broken... agent/app │ app.service: Unable to locate executable '/nonexistent': No such file or directory agent/app │ app.service: Failed at step EXEC spawning /nonexistent: No such file or directory agent/app │ app.service: Main process exited, code=exited, status=203/EXEC agent/app │ app.service: Failed with result 'exit-code'. agent/app │ Failed to start app broken. agent/app │ Starting app v1... agent/app │ Started app v1. » skipped: agent/other » failed » job 268920a44215332a6af584190e3153f1 via relay 0.1.0 (agents: agent 0.1.0) » wave 0 agent/app | Starting flakelet update app (via relay)... agent/app | Started flakelet update app (via relay). agent/app | app: using prebuilt artifact /nix/store/c67jj8kx3m50nyj874aqmdqf21w5iznd-flakelet-app-slow agent/app | app: activating generation 7 agent/app | app: updated to generation 7 agent/app | flakelet-relay-job-app.service: Deactivated successfully. » agent/app: updated (generation 7) » ok » job e4344d70047e670ef02f669aacb2f71f via relay 0.1.0 (agents: agent 0.1.0) » wave 0 agent/app | Starting flakelet update app (via relay)... agent/app | Started flakelet update app (via relay). agent/app | app: using prebuilt artifact /nix/store/ln47rsd5km8a04hd5zlkz4q1v352zqh3-flakelet-app-slow2 agent/app | app: activating generation 8 agent/app | app: updated to generation 8 agent/app | flakelet-relay-job-app.service: Deactivated successfully. » agent/app: updated (generation 8) » ok » job 65371d9584bd2579bd2aabe594e6ce4c via relay 0.1.0 (agents: agent 0.1.0) » wave 0 agent/app | Starting flakelet update app (via relay)... agent/app | Started flakelet update app (via relay). agent/app | app: using prebuilt artifact /nix/store/c67jj8kx3m50nyj874aqmdqf21w5iznd-flakelet-app-slow agent/app | app: activating generation 9 flakelet-push: stream: error reading a body from connection, resuming job d6f6eb7b78de971bb1c8ee7bb05ff730 flakelet-push: https://relay:7443: connect relay:7443: Connection refused (os error 111), retrying in 1s flakelet-push: https://relay:7443: connect relay:7443: Connection refused (os error 111), retrying in 2s flakelet-push: https://relay:7443: 404 Not Found: unknown_job: no connected agent knows this job, retrying in 4s » job 65371d9584bd2579bd2aabe594e6ce4c via relay 0.1.0 (agents: agent 0.1.0) » wave 0 agent/app | app: updated to generation 9 agent/app | flakelet-relay-job-app.service: Deactivated successfully. » agent/app: updated (generation 9) » ok 1789041580 hist1 admin@example.org agent/other:unchanged 1789041571 d6f6eb7b78de971bb1c8ee7bb05ff730 admin@example.org agent/app:updated 1789041562 f5847faf525d57e089d919acf419ba29 admin@example.org agent/app:updated 1789041545 108a5b96dc47817129b122bb5ec0c150 admin@example.org agent/app:updated 1789041543 local-app-6-1789041543 root (shell) agent/app:updated 1789041539 local-app-5-1789041539 root (shell) agent/app:updated 1789041536 46e23e8b1326b8604f870162864714c9 admin@example.org agent/other:unchanged 1789041525 2a54789fa285f12735169abd76fb0880 admin@example.org agent/app:unchanged 1789041525 c444ec8ff1d872439b4408886c79860e admin@example.org agent/app:updated 1789041524 663dc9809102e51e1d60f4233f48b9c9 admin@example.org agent/app:unchanged 1789041523 3b9425f81c8d03b1d4d6275ed8e92427 admin@example.org agent/app:unchanged 1789041519 3ec42230a912dd15c3f9e4175d94999e admin@example.org agent/app:rolled-back 1789041518 238d2aa85eca2498f5166b6c5d6bf3f0 dev@example.org agent/other:unchanged 1789041518 41ce0f7c8926cfbd2b1b9c9236df8dd8 dev@example.org agent/other:unchanged 1789041515 205e3cf1306f29c819635c72cde39a8e someone agent/other:unchanged 1789041512 852d17578acba49810ed6d153f9bd777 repo:github:example/app:ref:refs/heads/main agent/app:updated 1789041509 3a1502e96c68065c310cf3262eaa6ef5 admin@example.org agent/app:updated 1789041507 local-app-1-1789041507 host activation agent/app:updated 1789041507 local-flakelet-agent-1-1789041507 host activation agent/flakelet-agent:updated 1789041507 local-other-1-1789041507 host activation agent/other:updated Container relay terminated by signal KILL.