tribuchet: building on jamie
Machine state will be reset. To keep it, pass --keep-machine-state
start all VLans
(finished: start all VLans, in 0.00 seconds)
Initializing machine ID from random generator.
Test will time out and terminate in 3600.0 seconds
run the VM test script
start all VMs
agent: systemd-nspawn running (pid 52)
client: systemd-nspawn running (pid 53)
agent: Waiting for journal at /build/vm-state-agent/var/log/journal...
relay: systemd-nspawn running (pid 56)
client: Waiting for journal at /build/vm-state-client/var/log/journal...
relay: Waiting for journal at /build/vm-state-relay/var/log/journal...
(finished: start all VMs, in 0.00 seconds)
relay: waiting for unit flakelet-relay.service
nixos-nspawn(client): TAP vde-tap1 not found; container will be isolated from VDE
nixos-nspawn(client): A common reason for this is that /dev/net is not available in the Nix sandbox. Try adding /dev/net to extra-sandbox-paths.
nixos-nspawn(agent): TAP vde-tap1 not found; container will be isolated from VDE
nixos-nspawn(agent): A common reason for this is that /dev/net is not available in the Nix sandbox. Try adding /dev/net to extra-sandbox-paths.
nixos-nspawn(relay): TAP vde-tap1 not found; container will be isolated from VDE
nixos-nspawn(relay): A common reason for this is that /dev/net is not available in the Nix sandbox. Try adding /dev/net to extra-sandbox-paths.
Note: in a future version of systemd-nspawn the default set of permitted socket address families will be restricted to AF_INET, AF_INET6 and AF_UNIX. Use --restrict-address-families= to configure the set of permitted socket address families, or set RestrictAddressFamilies= in a .nspawn file.
Note: in a future version of systemd-nspawn the default set of permitted socket address families will be restricted to AF_INET, AF_INET6 and AF_UNIX. Use --restrict-address-families= to configure the set of permitted socket address families, or set RestrictAddressFamilies= in a .nspawn file.
░ Spawning container client on /build/vm-state-client.
░ Spawning container relay on /build/vm-state-relay.
Note: in a future version of systemd-nspawn the default set of permitted socket address families will be restricted to AF_INET, AF_INET6 and AF_UNIX. Use --restrict-address-families= to configure the set of permitted socket address families, or set RestrictAddressFamilies= in a .nspawn file.
░ Spawning container agent on /build/vm-state-agent.
agent # [ 4473.385397] agent systemd-journald[57]: Journal started
agent # [ 4473.385439] agent systemd-journald[57]: Runtime Journal (/run/log/journal/bbdc25a12fed41c4858accd83e668c4b) is 8M, max 4G, 3.9G free.
agent # [ 4473.395608] agent systemd[1]: Finished Apply Kernel Variables.
agent # [ 4473.407513] agent systemd[1]: Finished Create Static Device Nodes in /dev gracefully.
agent # [ 4473.443807] agent systemd[1]: Starting Flush Journal to Persistent Storage...
agent # [ 4473.444581] agent systemd[1]: Starting Create Static Device Nodes in /dev...
relay # [ 4473.383230] relay systemd-journald[57]: Journal started
relay # [ 4473.383272] relay systemd-journald[57]: Runtime Journal (/run/log/journal/ce1cc1ef91d649a5b86026ce32603edc) is 8M, max 4G, 3.9G free.
relay # [ 4473.395094] relay systemd[1]: Finished Apply Kernel Variables.
relay # [ 4473.407184] relay systemd[1]: Finished Create Static Device Nodes in /dev gracefully.
relay # [ 4473.443778] relay systemd[1]: Starting Flush Journal to Persistent Storage...
relay # [ 4473.444682] relay systemd[1]: Starting Create Static Device Nodes in /dev...
relay # [ 4473.452578] relay systemd-journald[57]: Time spent on flushing to /var/log/journal/ce1cc1ef91d649a5b86026ce32603edc is 1.394ms for 6 entries.
client # [ 4473.377224] client systemd-journald[57]: Journal started
client # [ 4473.377264] client systemd-journald[57]: Runtime Journal (/run/log/journal/083857f1ee3347c0b2c47f98b0ab6489) is 8M, max 4G, 3.9G free.
client # [ 4473.387857] client systemd[1]: Finished Apply Kernel Variables.
client # [ 4473.399110] client systemd[1]: Finished Create Static Device Nodes in /dev gracefully.
client # [ 4473.412742] client systemd[1]: Starting Flush Journal to Persistent Storage...
client # [ 4473.413462] client systemd[1]: Starting Create Static Device Nodes in /dev...
client # [ 4473.449074] client systemd-journald[57]: Time spent on flushing to /var/log/journal/083857f1ee3347c0b2c47f98b0ab6489 is 1.337ms for 6 entries.
client # [ 4473.449074] client systemd-journald[57]: System Journal (/var/log/journal/083857f1ee3347c0b2c47f98b0ab6489) is 512B, max 4G, 3.9G free.
client # [ 4473.456613] client systemd[1]: Finished Create Static Device Nodes in /dev.
client # [ 4473.457313] client systemd[1]: Reached target Preparation for Local File Systems.
client # [ 4473.457418] client systemd[1]: Rule-based Manager for Device Events and Files skipped, unmet condition check ConditionPathIsReadWrite=/sys
client # [ 4473.470506] client systemd[1]: Finished Flush Journal to Persistent Storage.
client # [ 4473.526140] client systemd[1]: Finished Firewall.
agent # [ 4473.452590] agent systemd-journald[57]: Time spent on flushing to /var/log/journal/bbdc25a12fed41c4858accd83e668c4b is 1.397ms for 6 entries.
agent # [ 4473.452590] agent systemd-journald[57]: System Journal (/var/log/journal/bbdc25a12fed41c4858accd83e668c4b) is 512B, max 4G, 3.9G free.
agent # [ 4473.461846] agent systemd[1]: Finished Create Static Device Nodes in /dev.
agent # [ 4473.462541] agent systemd[1]: Reached target Preparation for Local File Systems.
agent # [ 4473.462651] agent systemd[1]: Rule-based Manager for Device Events and Files skipped, unmet condition check ConditionPathIsReadWrite=/sys
agent # [ 4473.477174] agent systemd[1]: Finished Flush Journal to Persistent Storage.
agent # [ 4473.532058] agent systemd[1]: Finished Firewall.
relay # [ 4473.452578] relay systemd-journald[57]: System Journal (/var/log/journal/ce1cc1ef91d649a5b86026ce32603edc) is 512B, max 4G, 3.9G free.
relay # [ 4473.463009] relay systemd[1]: Finished Create Static Device Nodes in /dev.
relay # [ 4473.463748] relay systemd[1]: Reached target Preparation for Local File Systems.
relay # [ 4473.463870] relay systemd[1]: Rule-based Manager for Device Events and Files skipped, unmet condition check ConditionPathIsReadWrite=/sys
relay # [ 4473.474303] relay systemd[1]: Finished Flush Journal to Persistent Storage.
relay # [ 4473.555989] relay systemd[1]: Finished Firewall.
agent # [ 4474.385174] agent systemd[1]: Mounting /run/wrappers...
agent # [ 4474.432861] agent systemd[1]: Mounted /run/wrappers.
agent # [ 4474.433591] agent systemd[1]: Reached target Local File Systems.
agent # [ 4474.434389] agent systemd[1]: Listening on Boot Loader Control Service Socket.
agent # [ 4474.435571] agent systemd[1]: Starting Create SUID/SGID Wrappers...
agent # [ 4474.435600] agent systemd[1]: Update Boot Loader Random Seed skipped, unmet condition check ConditionVirtualization=!container
agent # [ 4474.436509] agent systemd[1]: Starting Save Transient machine-id to Disk...
agent # [ 4474.437318] agent systemd[1]: Starting Create System Files and Directories...
agent # [ 4474.454242] agent systemd-tmpfiles[161]: Cannot set file attributes for '/var/empty', value=0x00000010, mask=0x00000010, ignoring: Operation not permitted
agent # [ 4474.454649] agent systemd-tmpfiles[161]: fchmod() of /var/log/journal failed: Operation not permitted
agent # [ 4474.454944] agent systemd-tmpfiles[161]: fchmod() of /var/log/journal/bbdc25a12fed41c4858accd83e668c4b failed: Operation not permitted
agent # [ 4474.455287] agent systemd-tmpfiles[161]: fchmod() of /run/log/journal failed: Operation not permitted
client # [ 4474.389038] client systemd[1]: Mounting /run/wrappers...
client # [ 4474.442834] client systemd[1]: Mounted /run/wrappers.
client # [ 4474.443859] client systemd[1]: Reached target Local File Systems.
client # [ 4474.445207] client systemd[1]: Listening on Boot Loader Control Service Socket.
client # [ 4474.446593] client systemd[1]: Starting Create SUID/SGID Wrappers...
client # [ 4474.446643] client systemd[1]: Update Boot Loader Random Seed skipped, unmet condition check ConditionVirtualization=!container
client # [ 4474.447701] client systemd[1]: Starting Save Transient machine-id to Disk...
client # [ 4474.448825] client systemd[1]: Starting Create System Files and Directories...
client # [ 4474.466629] client systemd-tmpfiles[161]: Cannot set file attributes for '/var/empty', value=0x00000010, mask=0x00000010, ignoring: Operation not permitted
client # [ 4474.467047] client systemd-tmpfiles[161]: fchmod() of /var/log/journal failed: Operation not permitted
client # [ 4474.467360] client systemd-tmpfiles[161]: fchmod() of /var/log/journal/083857f1ee3347c0b2c47f98b0ab6489 failed: Operation not permitted
client # [ 4474.467704] client systemd-tmpfiles[161]: fchmod() of /run/log/journal failed: Operation not permitted
relay # [ 4474.384614] relay systemd[1]: Mounting /run/wrappers...
relay # [ 4474.447558] relay systemd[1]: Mounted /run/wrappers.
relay # [ 4474.448768] relay systemd[1]: Reached target Local File Systems.
relay # [ 4474.450176] relay systemd[1]: Listening on Boot Loader Control Service Socket.
relay # [ 4474.451373] relay systemd[1]: Starting Create SUID/SGID Wrappers...
relay # [ 4474.451422] relay systemd[1]: Update Boot Loader Random Seed skipped, unmet condition check ConditionVirtualization=!container
relay # [ 4474.452420] relay systemd[1]: Starting Save Transient machine-id to Disk...
relay # [ 4474.453444] relay systemd[1]: Starting Create System Files and Directories...
relay # [ 4474.471254] relay systemd-tmpfiles[169]: Cannot set file attributes for '/var/empty', value=0x00000010, mask=0x00000010, ignoring: Operation not permitted
relay # [ 4474.471652] relay systemd-tmpfiles[169]: fchmod() of /var/log/journal failed: Operation not permitted
relay # [ 4474.471949] relay systemd-tmpfiles[169]: fchmod() of /var/log/journal/ce1cc1ef91d649a5b86026ce32603edc failed: Operation not permitted
relay # [ 4474.472303] relay systemd-tmpfiles[169]: fchmod() of /run/log/journal failed: Operation not permitted
relay # [ 4474.472475] relay suid-sgid-wrappers-start[176]: chmod: changing permissions of '/run/wrappers/wrappers.rPcJe1jFKm/chsh': Operation not permitted
relay # [ 4474.479457] relay systemd[1]: suid-sgid-wrappers.service: Main process exited, code=exited, status=1/FAILURE
relay # [ 4474.479497] relay systemd[1]: suid-sgid-wrappers.service: Failed with result 'exit-code'.
relay # [ 4474.479548] relay systemd[1]: Failed to start Create SUID/SGID Wrappers.
relay # [ 4474.484966] relay systemd[1]: Finished Create System Files and Directories.
relay # [ 4474.562988] relay systemd[1]: etc-machine\x2did.mount: Deactivated successfully.
relay # [ 4474.582615] relay systemd[1]: Finished Save Transient machine-id to Disk.
relay # [ 4474.586511] relay systemd[1]: Starting Rebuild Journal Catalog...
relay # [ 4474.587341] relay systemd[1]: Starting Record System Boot/Shutdown in UTMP...
relay # [ 4474.634942] relay systemd[1]: Finished Record System Boot/Shutdown in UTMP.
agent # [ 4474.457532] agent systemd[1]: Finished Create System Files and Directories.
agent # [ 4474.460687] agent suid-sgid-wrappers-start[169]: chmod: changing permissions of '/run/wrappers/wrappers.kUkFjpisAc/chsh': Operation not permitted
agent # [ 4474.470315] agent systemd[1]: suid-sgid-wrappers.service: Main process exited, code=exited, status=1/FAILURE
agent # [ 4474.470531] agent systemd[1]: suid-sgid-wrappers.service: Failed with result 'exit-code'.
agent # [ 4474.470687] agent systemd[1]: Failed to start Create SUID/SGID Wrappers.
agent # [ 4474.472339] agent systemd[1]: Starting Rebuild Journal Catalog...
agent # [ 4474.473029] agent systemd[1]: Starting Record System Boot/Shutdown in UTMP...
agent # [ 4474.485213] agent systemd[1]: Finished Record System Boot/Shutdown in UTMP.
agent # [ 4474.489579] agent systemd[1]: Finished Rebuild Journal Catalog.
agent # [ 4474.490795] agent systemd[1]: Starting Update is Completed...
agent # [ 4474.499126] agent systemd[1]: Finished Update is Completed.
agent # [ 4474.499239] agent systemd[1]: Reached target System Initialization.
agent # [ 4474.499316] agent systemd[1]: Discard unused filesystem blocks once a week skipped, unmet condition check ConditionVirtualization=!container
agent # [ 4474.499341] agent systemd[1]: Started Daily Cleanup of Temporary Directories.
agent # [ 4474.499355] agent systemd[1]: Reached target Timer Units.
agent # [ 4474.499448] agent systemd[1]: Listening on D-Bus System Message Bus Socket.
agent # [ 4474.499638] agent systemd[1]: Listening on Nix Daemon Socket.
agent # [ 4474.499730] agent systemd[1]: Listening on Virtual Machine and Container Registration Service Socket.
agent # [ 4474.499744] agent systemd[1]: Reached target Socket Units.
agent # [ 4474.499771] agent systemd[1]: Reached target Basic System.
agent # [ 4474.500867] agent systemd[1]: Starting Re-link flakelet services at boot...
agent # [ 4474.501573] agent systemd[1]: Starting Import lastlog data into lastlog2 database...
agent # [ 4474.502477] agent systemd[1]: Starting Name Service Cache Daemon (nsncd)...
agent # [ 4474.503494] agent systemd[1]: Starting resolvconf update...
agent # [ 4474.511901] agent systemd[1]: Finished Re-link flakelet services at boot.
agent # [ 4474.521747] agent systemd[1]: Starting Reconcile flakelet services with the host configuration...
agent # [ 4474.565768] agent systemd[1]: etc-machine\x2did.mount: Deactivated successfully.
agent # [ 4474.566734] agent systemd[1]: Finished Save Transient machine-id to Disk.
agent # [ 4474.566925] agent systemd[1]: Finished Import lastlog data into lastlog2 database.
agent # [ 4474.567495] agent systemd[1]: Finished Reconcile flakelet services with the host configuration.
agent # [ 4474.592046] agent systemd[1]: nscd.service: Deactivated successfully.
agent # [ 4474.625430] agent systemd[1]: Stopped Name Service Cache Daemon (nsncd).
agent # [ 4474.627143] agent systemd[1]: Finished resolvconf update.
agent # [ 4474.627998] agent systemd[1]: Reached target Preparation for Network.
agent # [ 4474.629070] agent systemd[1]: Starting Address configuration of eth1...
agent # [ 4474.630259] agent systemd[1]: Starting Extra networking commands....
agent # [ 4474.631357] agent systemd[1]: Starting Name Service Cache Daemon (nsncd)...
client # [ 4474.469948] client suid-sgid-wrappers-start[168]: chmod: changing permissions of '/run/wrappers/wrappers.yjweE5IaPf/chsh': Operation not permitted
client # [ 4474.474606] client systemd[1]: suid-sgid-wrappers.service: Main process exited, code=exited, status=1/FAILURE
client # [ 4474.474647] client systemd[1]: suid-sgid-wrappers.service: Failed with result 'exit-code'.
client # [ 4474.474695] client systemd[1]: Failed to start Create SUID/SGID Wrappers.
client # [ 4474.509707] client systemd[1]: Finished Create System Files and Directories.
client # [ 4474.572217] client systemd[1]: etc-machine\x2did.mount: Deactivated successfully.
client # [ 4474.625581] client systemd[1]: Finished Save Transient machine-id to Disk.
client # [ 4474.629639] client systemd[1]: Starting Rebuild Journal Catalog...
client # [ 4474.630266] client systemd[1]: Starting Record System Boot/Shutdown in UTMP...
client # [ 4474.642783] client systemd[1]: Finished Record System Boot/Shutdown in UTMP.
client # [ 4474.650515] client systemd[1]: Finished Rebuild Journal Catalog.
client # [ 4474.652259] client systemd[1]: Starting Update is Completed...
client # [ 4474.662702] client systemd[1]: Finished Update is Completed.
client # [ 4474.662793] client systemd[1]: Reached target System Initialization.
client # [ 4474.662856] client systemd[1]: Discard unused filesystem blocks once a week skipped, unmet condition check ConditionVirtualization=!container
client # [ 4474.662880] client systemd[1]: Started Daily Cleanup of Temporary Directories.
client # [ 4474.662893] client systemd[1]: Reached target Timer Units.
client # [ 4474.662990] client systemd[1]: Listening on D-Bus System Message Bus Socket.
client # [ 4474.663184] client systemd[1]: Listening on Nix Daemon Socket.
client # [ 4474.663284] client systemd[1]: Listening on Virtual Machine and Container Registration Service Socket.
client # [ 4474.663297] client systemd[1]: Reached target Socket Units.
client # [ 4474.663325] client systemd[1]: Reached target Basic System.
client # [ 4474.664270] client systemd[1]: Starting Import lastlog data into lastlog2 database...
client # [ 4474.665069] client systemd[1]: Starting Name Service Cache Daemon (nsncd)...
client # [ 4474.665624] client systemd[1]: Starting resolvconf update...
client # [ 4474.679849] client systemd[1]: Finished Import lastlog data into lastlog2 database.
client # [ 4474.752144] client systemd[1]: nscd.service: Deactivated successfully.
client # [ 4474.752259] client systemd[1]: Stopped Name Service Cache Daemon (nsncd).
client # [ 4474.754847] client systemd[1]: Starting Name Service Cache Daemon (nsncd)...
client # [ 4474.757869] client systemd[1]: Finished resolvconf update.
client # [ 4474.757964] client systemd[1]: Reached target Preparation for Network.
client # [ 4474.758849] client systemd[1]: Starting Address configuration of eth1...
client # [ 4474.759742] client systemd[1]: Starting Extra networking commands....
client # [ 4474.779388] client network-addresses-eth1-start[237]: adding address 192.168.1.2/24... done
client # [ 4474.783234] client network-addresses-eth1-start[237]: adding address 2001:db8:1::2/64... done
client # [ 4474.788064] client systemd[1]: Finished Address configuration of eth1.
client # [ 4474.829835] client systemd[1]: Finished Extra networking commands..
client # [ 4474.829971] client systemd[1]: Reached target Network.
client # [ 4474.877149] client systemd[1]: nscd.service: Deactivated successfully.
client # [ 4474.877334] client systemd[1]: Stopped Name Service Cache Daemon (nsncd).
client # [ 4474.880220] client systemd[1]: Starting Name Service Cache Daemon (nsncd)...
agent # [ 4474.649769] agent network-addresses-eth1-start[240]: adding address 192.168.1.1/24... done
agent # [ 4474.653592] agent network-addresses-eth1-start[240]: adding address 2001:db8:1::1/64... done
agent # [ 4474.657547] agent systemd[1]: Finished Address configuration of eth1.
agent # [ 4474.707436] agent systemd[1]: Finished Extra networking commands..
agent # [ 4474.708111] agent systemd[1]: Reached target Network.
agent # [ 4474.709270] agent systemd[1]: Starting flakelet-agent...
agent # [ 4474.709977] agent systemd[1]: Starting Update flakelet service app...
agent # [ 4474.710639] agent systemd[1]: Starting Update flakelet service other...
agent # [ 4474.751768] agent systemd[1]: nscd.service: Deactivated successfully.
agent # [ 4474.751905] agent systemd[1]: Stopped Name Service Cache Daemon (nsncd).
agent # [ 4474.754589] agent systemd[1]: Starting Name Service Cache Daemon (nsncd)...
agent # [ 4474.759545] agent flakelet[305]: app: using prebuilt artifact /nix/store/hysydfr96ds875vr0jgw5rjxi4p39h2j-flakelet-app-v1
agent # [ 4474.759815] agent flakelet[306]: other: using prebuilt artifact /nix/store/hmr990n0b4v6rpzrcpyyw738r9xi7z95-flakelet-other-v1
agent # [ 4474.760121] agent flakelet[305]: app: activating generation 1
agent # [ 4474.760279] agent flakelet[306]: other: activating generation 1
agent # [ 4474.764987] agent systemd[1]: Reload requested from client PID 327 ('systemctl') (unit flakelet-other.service)...
agent # [ 4474.765028] agent systemd[1]: Reloading...
agent # [ 4474.793281] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:12.754929Z","level":"WARN","message":"SRV _flakelet-relay._tcp.test: lookup failed","target":"flakelet_agent"}
relay # [ 4474.642462] relay systemd[1]: Finished Rebuild Journal Catalog.
relay # [ 4474.643842] relay systemd[1]: Starting Update is Completed...
relay # [ 4474.656866] relay systemd[1]: Finished Update is Completed.
relay # [ 4474.656962] relay systemd[1]: Reached target System Initialization.
relay # [ 4474.657050] relay systemd[1]: Discard unused filesystem blocks once a week skipped, unmet condition check ConditionVirtualization=!container
relay # [ 4474.657070] relay systemd[1]: Started Daily Cleanup of Temporary Directories.
relay # [ 4474.657082] relay systemd[1]: Reached target Timer Units.
relay # [ 4474.657179] relay systemd[1]: Listening on D-Bus System Message Bus Socket.
relay # [ 4474.657347] relay systemd[1]: Listening on Nix Daemon Socket.
relay # [ 4474.657444] relay systemd[1]: Listening on Virtual Machine and Container Registration Service Socket.
relay # [ 4474.657458] relay systemd[1]: Reached target Socket Units.
relay # [ 4474.657487] relay systemd[1]: Reached target Basic System.
relay # [ 4474.658847] relay systemd[1]: Started device-mock.service.
relay # [ 4474.659631] relay systemd[1]: Starting Re-link flakelet services at boot...
relay # [ 4474.660406] relay systemd[1]: Starting Import lastlog data into lastlog2 database...
relay # [ 4474.661468] relay systemd[1]: Starting Name Service Cache Daemon (nsncd)...
relay # [ 4474.662191] relay systemd[1]: Starting resolvconf update...
relay # [ 4474.671103] relay systemd[1]: Finished Re-link flakelet services at boot.
relay # [ 4474.672105] relay systemd[1]: Starting Reconcile flakelet services with the host configuration...
relay # [ 4474.678095] relay systemd[1]: Finished Import lastlog data into lastlog2 database.
relay # [ 4474.683631] relay systemd[1]: Finished Reconcile flakelet services with the host configuration.
relay # [ 4474.761319] relay systemd[1]: nscd.service: Deactivated successfully.
relay # [ 4474.761513] relay systemd[1]: Stopped Name Service Cache Daemon (nsncd).
relay # [ 4474.763134] relay systemd[1]: Finished resolvconf update.
relay # [ 4474.763885] relay systemd[1]: Reached target Preparation for Network.
relay # [ 4474.764980] relay systemd[1]: Starting Address configuration of eth1...
relay # [ 4474.765641] relay systemd[1]: Starting Extra networking commands....
relay # [ 4474.766501] relay systemd[1]: Starting Name Service Cache Daemon (nsncd)...
relay # [ 4474.784154] relay network-addresses-eth1-start[249]: adding address 192.168.1.3/24... done
relay # [ 4474.787751] relay network-addresses-eth1-start[249]: adding address 2001:db8:1::3/64... done
relay # [ 4474.792169] relay systemd[1]: Finished Address configuration of eth1.
relay # [ 4474.831169] relay systemd[1]: Finished Extra networking commands..
relay # [ 4474.831260] relay systemd[1]: Reached target Network.
relay # [ 4474.832600] relay systemd[1]: Starting Dnsmasq Daemon...
relay # [ 4474.833523] relay systemd[1]: Starting Nginx Web Server...
relay # [ 4474.834357] relay systemd[1]: Starting Update flakelet service flakelet-relay...
relay # [ 4474.845099] relay flakelet[311]: flakelet-relay: using prebuilt artifact /nix/store/fg23fm1bd55b29023k2xz2vnvx84smyx-flakelet-flakelet-relay
relay # [ 4474.845734] relay flakelet[311]: flakelet-relay: activating generation 1
relay # [ 4474.849202] relay systemd[1]: Reload requested from client PID 314 ('systemctl') (unit flakelet-flakelet-relay.service)...
relay # [ 4474.849235] relay systemd[1]: Reloading...
relay # [ 4475.159457] relay nsncd[251]: Sep 06 08:19:13.121 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket"
relay # [ 4475.262189] relay systemd[1]: Reloading finished in 412 ms.
relay # [ 4475.404686] relay systemd[1]: Started Name Service Cache Daemon (nsncd).
relay # [ 4475.405797] relay systemd[1]: Reached target Host and Network Name Lookups.
relay # [ 4475.405904] relay systemd[1]: Reached target User and Group Name Lookups.
relay # [ 4475.407117] relay systemd[1]: Starting User Login Management...
relay # [ 4475.407997] relay systemd[1]: Starting Permit User Sessions...
relay # [ 4475.421349] relay systemd[1]: Reload requested from client PID 357 ('systemctl') (unit flakelet-flakelet-relay.service)...
relay # [ 4475.421386] relay systemd[1]: Reloading...
client # [ 4475.154606] client nsncd[303]: Sep 06 08:19:13.116 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket"
client # [ 4475.154626] client systemd[1]: Started Name Service Cache Daemon (nsncd).
client # [ 4475.154726] client systemd[1]: Reached target Host and Network Name Lookups.
client # [ 4475.154836] client systemd[1]: Reached target User and Group Name Lookups.
client # [ 4475.157402] client systemd[1]: Starting User Login Management...
client # [ 4475.158839] client systemd[1]: Starting Permit User Sessions...
client # [ 4475.194309] client systemd[1]: Finished Permit User Sessions.
client # [ 4475.196424] client systemd[1]: Started Console Getty.
client # [ 4475.196480] client systemd[1]: Getty on tty1 skipped, unmet condition check ConditionPathExists=/dev/tty0
client # [ 4475.196511] client systemd[1]: Reached target Login Prompts.
agent # [ 4475.194844] agent nsncd[311]: Sep 06 08:19:13.156 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket"
agent # [ 4475.275319] agent systemd[1]: Reloading finished in 509 ms.
agent # [ 4475.407202] agent systemd[1]: Started flakelet-agent.
agent # [ 4475.407383] agent systemd[1]: Started Name Service Cache Daemon (nsncd).
agent # [ 4475.407882] agent systemd[1]: Reload requested from client PID 325 ('systemctl') (unit flakelet-app.service)...
agent # [ 4475.407923] agent systemd[1]: Reloading...
relay # [ 4475.456229] relay dnsmasq-pre-start[372]: dnsmasq: syntax check OK.
relay # [ 4475.784769] relay systemd[1]: Reloading finished in 363 ms.
agent # [ 4475.777483] agent systemd[1]: Reloading finished in 369 ms.
agent # [ 4475.885503] agent systemd[1]: Reached target Host and Network Name Lookups.
agent # [ 4475.885624] agent systemd[1]: Reached target User and Group Name Lookups.
agent # [ 4475.887135] agent systemd[1]: Starting User Login Management...
agent # [ 4475.888022] agent systemd[1]: Starting Permit User Sessions...
agent # [ 4475.895522] agent systemd[1]: Reload requested from client PID 769 ('systemctl') (unit flakelet-other.service)...
agent # [ 4475.895553] agent systemd[1]: Reloading...
relay # [ 4475.886921] relay systemd[1]: Finished Permit User Sessions.
relay # [ 4475.921560] relay systemd[1]: Started Console Getty.
relay # [ 4475.921636] relay systemd[1]: Getty on tty1 skipped, unmet condition check ConditionPathExists=/dev/tty0
relay # [ 4475.921702] relay systemd[1]: Reached target Login Prompts.
relay # [ 4475.923462] relay systemd[1]: Starting flakelet-relay...
agent # [ 4476.280582] agent systemd[1]: Reloading finished in 384 ms.
agent # [ 4476.350377] agent systemd[1]: Finished Permit User Sessions.
agent # [ 4476.354566] agent systemd[1]: Started Console Getty.
agent # [ 4476.354628] agent systemd[1]: Getty on tty1 skipped, unmet condition check ConditionPathExists=/dev/tty0
agent # [ 4476.354695] agent systemd[1]: Reached target Login Prompts.
agent # [ 4476.358875] agent systemd[1]: Reload requested from client PID 805 ('systemctl') (unit flakelet-app.service)...
agent # [ 4476.358904] agent systemd[1]: Reloading...
client # [ 4476.487803] client systemd-logind[315]: New seat seat0.
client # [ 4476.488358] client systemd[1]: Started User Login Management.
client # [ 4476.490558] client systemd[1]: Starting D-Bus System Message Bus...
client # [ 4476.491858] client systemd[1]: Starting linger-users.service...
client # [ 4476.528081] client systemd[1]: linger-users.service: Deactivated successfully.
client # [ 4476.528451] client systemd[1]: Finished linger-users.service.
client # [ 4476.528792] client systemd[1]: Reached target Multi-User System.
relay # [ 4476.494466] relay nginx-pre-start[402]: nginx: the configuration file /nix/store/01avjj26wcihp926xgasszcj5s0k7cbs-nginx.conf syntax is ok
relay # [ 4476.495188] relay nginx-pre-start[402]: nginx: configuration file /nix/store/01avjj26wcihp926xgasszcj5s0k7cbs-nginx.conf test is successful
relay # [ 4476.518781] relay systemd[1]: Started Nginx Web Server.
relay # [ 4476.590709] relay systemd[1]: Starting D-Bus System Message Bus...
agent # [ 4476.709391] agent systemd[1]: Reloading finished in 350 ms.
agent # [ 4476.780368] agent systemd[1]: Starting other v1...
agent # [ 4476.805989] agent systemd[1]: Starting app v1...
agent # [ 4476.809253] agent systemd[1]: Started other v1.
agent # [ 4476.812877] agent systemd[1]: Started app v1.
agent # [ 4476.824995] agent flakelet[306]: other: updated to generation 1
agent # [ 4476.825615] agent systemd[1]: Finished Update flakelet service other.
agent # [ 4476.828602] agent flakelet[305]: app: updated to generation 1
agent # [ 4476.830093] agent systemd[1]: Finished Update flakelet service app.
agent # [ 4476.830203] agent systemd[1]: Reached target flakelet managed services.
relay # [ 4476.867498] relay systemd-logind[354]: New seat seat0.
relay # [ 4476.867815] relay systemd[1]: Started User Login Management.
relay # [ 4476.869899] relay systemd[1]: Starting linger-users.service...
relay # [ 4476.901778] relay systemd[1]: linger-users.service: Deactivated successfully.
relay # [ 4476.902073] relay systemd[1]: Finished linger-users.service.
relay # [ 4477.100132] relay dbus-broker-launch[408]: Looking up NSS user entry for 'systemd-timesync'...
relay # [ 4477.100932] relay dbus-broker-launch[408]: NSS returned no entry for 'systemd-timesync'
relay # [ 4477.100932] relay dbus-broker-launch[408]: Invalid user-name in /nix/store/27h4g2jdw18vrabfh6p1l0n8ss5dhbp5-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync"
relay # [ 4477.101558] relay systemd[1]: Started D-Bus System Message Bus.
relay # [ 4477.106993] relay dbus-broker-launch[408]: Ready
relay # [ 4477.110193] relay systemd[1]: Started Dnsmasq Daemon.
relay # [ 4477.112793] relay dnsmasq[394]: started, version 2.93 cachesize 150
relay # [ 4477.112811] relay dnsmasq[394]: compile time options: IPv6 GNU-getopt DBus no-UBus no-i18n IDN2 DHCP DHCPv6 no-Lua TFTP conntrack ipset nftset auth DNSSEC loop-detect inotify dumpfile
relay # [ 4477.112826] relay dnsmasq[394]: DBus support enabled: connected to system bus
relay # [ 4477.112840] relay dnsmasq[394]: no upstream servers configured - please set them from DBus
relay # [ 4477.112984] relay dnsmasq[394]: read /etc/hosts - 9 names
client # [ 4477.035700] client dbus-broker-launch[325]: Looking up NSS user entry for 'systemd-timesync'...
client # [ 4477.037065] client dbus-broker-launch[325]: NSS returned no entry for 'systemd-timesync'
client # [ 4477.037065] client dbus-broker-launch[325]: Invalid user-name in /nix/store/pgfa8wj70mb4y0q358afylir6nilgvmb-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync"
client # [ 4477.037880] client systemd[1]: Started D-Bus System Message Bus.
client # [ 4477.038367] client systemd[1]: Startup finished in 4.092s.
client # [ 4477.043839] client dbus-broker-launch[325]: Ready
agent # [ 4477.358599] agent systemd-logind[767]: New seat seat0.
agent # [ 4477.360143] agent systemd[1]: Starting D-Bus System Message Bus...
agent # [ 4477.360205] agent systemd[1]: Started User Login Management.
agent # [ 4477.361242] agent systemd[1]: Starting linger-users.service...
agent # [ 4477.398542] agent systemd[1]: linger-users.service: Deactivated successfully.
agent # [ 4477.398744] agent systemd[1]: Finished linger-users.service.
agent # [ 4477.398930] agent systemd[1]: Reached target Multi-User System.
agent # [ 4477.777347] agent dbus-broker-launch[845]: Looking up NSS user entry for 'systemd-timesync'...
agent # [ 4477.778527] agent dbus-broker-launch[845]: NSS returned no entry for 'systemd-timesync'
agent # [ 4477.778527] agent dbus-broker-launch[845]: Invalid user-name in /nix/store/q3c2gxfm178wqgzv8m87na4aa72wxc04-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync"
agent # [ 4477.779647] agent systemd[1]: Started D-Bus System Message Bus.
agent # [ 4477.780189] agent systemd[1]: Startup finished in 4.831s.
agent # [ 4477.786446] agent dbus-broker-launch[845]: Ready
relay # [ 4477.892555] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:15.854170Z","level":"INFO","message":"listening on 127.0.0.1:7400","target":"flakelet_relay::relay::server"}
relay # [ 4477.893457] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:15.854789Z","level":"INFO","message":"listening on [::]:7443","target":"flakelet_relay::relay::server"}
relay # [ 4477.893068] relay systemd[1]: Started flakelet-relay.
relay # [ 4477.900228] relay systemd[1]: Starting health for flakelet-relay...
relay: (finished: waiting for unit flakelet-relay.service, in 6.16 seconds)
relay: waiting for unit nginx.service
relay # [ 4478.149772] relay systemd[1]: flakelet-relay-health.service: Deactivated successfully.
relay # [ 4478.150123] relay systemd[1]: Finished health for flakelet-relay.
relay # [ 4478.192913] relay flakelet[311]: flakelet-relay: updated to generation 1
relay # [ 4478.195750] relay systemd[1]: Finished Update flakelet service flakelet-relay.
relay # [ 4478.196297] relay systemd[1]: Reached target flakelet managed services.
relay # [ 4478.196471] relay systemd[1]: Reached target Multi-User System.
relay # [ 4478.196726] relay systemd[1]: Startup finished in 5.250s.
relay: (finished: waiting for unit nginx.service, in 0.02 seconds)
agent: waiting for unit flakelet.target
agent: (finished: waiting for unit flakelet.target, in 0.02 seconds)
agent: waiting for unit flakelet-agent.service
agent: (finished: waiting for unit flakelet-agent.service, in 0.02 seconds)
client: waiting for unit multi-user.target
client: (finished: waiting for unit multi-user.target, in 0.02 seconds)
relay: waiting for success: curl -sf http://127.0.0.1:7400/metrics | grep -q 'agent_up{host="agent"} 1'
agent # [ 4479.814456] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:17.776218Z","level":"INFO","message":"connected","relay":"https://relay:7443","relay_name":"relay","relay_version":"0.1.0","host":"agent","target":"flakelet_relay::agent::conn"}
relay # [ 4479.814850] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:17.776577Z","level":"INFO","message":"agent connected","host":"agent","version":"0.1.0","flakelets":"[Named { name: \"app\", generation: Some(1), revision: Some(\"prebuilt:app\") }, Named { name: \"other\", generation: Some(1), revision: Some(\"prebuilt:other\") }]","target":"flakelet_relay::relay::agent_conn"}
relay: (finished: waiting for success: curl -sf http://127.0.0.1:7400/metrics | grep -q 'agent_up{host="agent"} 1', in 2.05 seconds)
agent: must succeed: systemctl is-active app.service other.service
agent: (finished: must succeed: systemctl is-active app.service other.service, in 0.01 seconds)
subtest: deploy with client certificate, relay found via SRV
agent: must succeed: jq '.services.app.prebuilt="/nix/store/6ili1vxjlag9d7dy0bxrbxk6bpi5qh9p-flakelet-app-v2"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json
agent: (finished: must succeed: jq '.services.app.prebuilt="/nix/store/6ili1vxjlag9d7dy0bxrbxk6bpi5qh9p-flakelet-app-v2"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json, in 0.01 seconds)
client: must succeed: push --relay-srv test --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy agent/app 2>&1
agent # [ 4480.792257] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:18.754018Z","level":"INFO","message":"start","id":"8ea3a03b211f684880093e2c3bcf81df","flakelet":"app","rule":"admins","relay":"relay","caller":"Some(\"x509:email:admin@example.org\")","target":"flakelet_relay::agent::conn"}
agent # [ 4480.793282] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:18.755085Z","level":"INFO","message":"update starting","flakelet":"app","ids":"[\"8ea3a03b211f684880093e2c3bcf81df\"]","target":"flakelet_relay::agent::jobs"}
agent # [ 4480.855227] agent systemd[1]: Starting flakelet update app (via relay)...
agent # [ 4480.861818] agent systemd[1]: Started flakelet update app (via relay).
agent # [ 4480.865820] agent flakelet[882]: app: using prebuilt artifact /nix/store/6ili1vxjlag9d7dy0bxrbxk6bpi5qh9p-flakelet-app-v2
agent # [ 4480.866468] agent flakelet[882]: app: activating generation 2
agent # [ 4480.879311] agent systemd[1]: Stopping app v1...
agent # [ 4480.879992] agent systemd[1]: app.service: Deactivated successfully.
agent # [ 4480.880115] agent systemd[1]: Stopped app v1.
agent # [ 4480.890791] agent systemd[1]: Reload requested from client PID 891 ('systemctl') (unit flakelet-relay-job-app.service)...
agent # [ 4480.890825] agent systemd[1]: Reloading...
relay # [ 4480.791718] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:18.753463Z","level":"INFO","message":"deploy accepted","job":"d30c8ccd9850f8cfcbe259628ad443a4","caller":"x509:email:admin@example.org","targets":"[\"agent/app\"]","target":"flakelet_relay::relay::api"}
agent # [ 4481.219637] agent systemd[1]: Reloading finished in 328 ms.
agent # [ 4481.352761] agent systemd[1]: Reload requested from client PID 921 ('systemctl') (unit flakelet-relay-job-app.service)...
agent # [ 4481.352798] agent systemd[1]: Reloading...
agent # [ 4481.679851] agent systemd[1]: Reloading finished in 326 ms.
agent # [ 4481.756985] agent systemd[1]: Reload requested from client PID 950 ('systemctl') (unit flakelet-relay-job-app.service)...
agent # [ 4481.757031] agent systemd[1]: Reloading...
agent # [ 4482.082633] agent systemd[1]: Reloading finished in 325 ms.
agent # [ 4482.168462] agent systemd[1]: Reload requested from client PID 981 ('systemctl') (unit flakelet-relay-job-app.service)...
agent # [ 4482.168496] agent systemd[1]: Reloading...
agent # [ 4482.493385] agent systemd[1]: Reloading finished in 324 ms.
agent # [ 4482.575716] agent systemd[1]: Starting app v2...
agent # [ 4482.604903] agent systemd[1]: Started app v2.
agent # [ 4482.619010] agent flakelet[882]: app: updated to generation 2
agent # [ 4482.620891] agent systemd[1]: flakelet-relay-job-app.service: Deactivated successfully.
client: (finished: must succeed: push --relay-srv test --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy agent/app 2>&1, in 2.95 seconds)
agent: must succeed: systemctl show app.service -p Description | grep -q 'app v2'
agent: (finished: must succeed: systemctl show app.service -p Description | grep -q 'app v2', in 0.02 seconds)
agent: must succeed: journalctl -u flakelet-relay-job-app.service | grep -q 'generation 2'
agent: (finished: must succeed: journalctl -u flakelet-relay-job-app.service | grep -q 'generation 2', in 0.01 seconds)
(finished: subtest: deploy with client certificate, relay found via SRV, in 3.00 seconds)
subtest: deploy with OIDC token, sub principal
agent: must succeed: jq '.services.app.prebuilt="/nix/store/hysydfr96ds875vr0jgw5rjxi4p39h2j-flakelet-app-v1"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json
agent: (finished: must succeed: jq '.services.app.prebuilt="/nix/store/hysydfr96ds875vr0jgw5rjxi4p39h2j-flakelet-app-v1"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json, in 0.01 seconds)
client: must succeed: echo '{"groups": []}' | step crypto jwt sign --key /nix/store/r7sclamg09v434w8zjlfnrcrn1w79wh6-flakelet-relay-test-issuer/jwk.json --iss https://relay:8443 --aud flakelet-relay --sub 'repo:github:example/app:ref:refs/heads/main' --exp $(( $(date +%s) + 300 )) --jti $(head -c8 /dev/urandom | base32)
client: (finished: must succeed: echo '{"groups": []}' | step crypto jwt sign --key /nix/store/r7sclamg09v434w8zjlfnrcrn1w79wh6-flakelet-relay-test-issuer/jwk.json --iss https://relay:8443 --aud flakelet-relay --sub 'repo:github:example/app:ref:refs/heads/main' --exp $(( $(date +%s) + 300 )) --jti $(head -c8 /dev/urandom | base32), in 0.05 seconds)
client: must succeed: FLAKELET_RELAY_TOKEN_COMMAND='echo eyJhbGciOiJFZERTQSIsImtpZCI6ImsxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJmbGFrZWxldC1yZWxheSIsImV4cCI6MTc4ODY4MzA2MSwiZ3JvdXBzIjpbXSwiaWF0IjoxNzg4NjgyNzYxLCJpc3MiOiJodHRwczovL3JlbGF5Ojg0NDMiLCJqdGkiOiI2M0g1SFIyUDNZMlZDPT09IiwibmJmIjoxNzg4NjgyNzYxLCJzdWIiOiJyZXBvOmdpdGh1YjpleGFtcGxlL2FwcDpyZWY6cmVmcy9oZWFkcy9tYWluIn0.2MyrB94AoGvWlHKiSoKZdNE_C_aooQVRkLvsjrTxFIrIMxgwf0JrFjYOi2BwISPgllRD3HBK9nBFaU-iotvTAQ' push --relay https://relay:7443 deploy agent/app 2>&1
agent # [ 4483.716687] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:21.678433Z","level":"INFO","message":"update finished","flakelet":"app","status":"Updated","generation":"Some(2)","target":"flakelet_relay::agent::jobs"}
agent # [ 4483.841090] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:21.802874Z","level":"INFO","message":"start","id":"90a362b4e152f6b56ec01f29e11d0b79","flakelet":"app","rule":"ci","relay":"relay","caller":"Some(\"oidc:mock:repo:github:example/app:ref:refs/heads/main\")","target":"flakelet_relay::agent::conn"}
agent # [ 4483.841430] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:21.803254Z","level":"INFO","message":"update starting","flakelet":"app","ids":"[\"90a362b4e152f6b56ec01f29e11d0b79\"]","target":"flakelet_relay::agent::jobs"}
agent # [ 4483.882237] agent systemd[1]: Starting flakelet update app (via relay)...
relay # [ 4483.717805] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:21.679553Z","level":"INFO","message":"deploy finished","job":"d30c8ccd9850f8cfcbe259628ad443a4","ok":true,"target":"flakelet_relay::relay::api"}
relay # [ 4483.840821] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:21.802625Z","level":"INFO","message":"deploy accepted","job":"5d5e8cac9596c0da9bc30c6abf2d63ec","caller":"oidc:mock:repo:github:example/app:ref:refs/heads/main","targets":"[\"agent/app\"]","target":"flakelet_relay::relay::api"}
agent # [ 4484.075405] agent systemd[1]: Started flakelet update app (via relay).
agent # [ 4484.080646] agent flakelet[1059]: app: using prebuilt artifact /nix/store/hysydfr96ds875vr0jgw5rjxi4p39h2j-flakelet-app-v1
agent # [ 4484.081654] agent flakelet[1059]: app: activating generation 3
agent # [ 4484.091714] agent systemd[1]: Stopping app v2...
agent # [ 4484.092410] agent systemd[1]: app.service: Deactivated successfully.
agent # [ 4484.092539] agent systemd[1]: Stopped app v2.
agent # [ 4484.104145] agent systemd[1]: Reload requested from client PID 1066 ('systemctl') (unit flakelet-relay-job-app.service)...
agent # [ 4484.104177] agent systemd[1]: Reloading...
agent # [ 4484.431744] agent systemd[1]: Reloading finished in 327 ms.
agent # [ 4484.495070] agent systemd[1]: Reload requested from client PID 1096 ('systemctl') (unit flakelet-relay-job-app.service)...
agent # [ 4484.495102] agent systemd[1]: Reloading...
agent # [ 4484.819728] agent systemd[1]: Reloading finished in 324 ms.
agent # [ 4484.877978] agent systemd[1]: Reload requested from client PID 1128 ('systemctl') (unit flakelet-relay-job-app.service)...
agent # [ 4484.878015] agent systemd[1]: Reloading...
agent # [ 4485.205872] agent systemd[1]: Reloading finished in 327 ms.
agent # [ 4485.283027] agent systemd[1]: Reload requested from client PID 1160 ('systemctl') (unit flakelet-relay-job-app.service)...
agent # [ 4485.283061] agent systemd[1]: Reloading...
agent # [ 4485.605594] agent systemd[1]: Reloading finished in 322 ms.
agent # [ 4485.675660] agent systemd[1]: Starting app v1...
agent # [ 4485.707103] agent systemd[1]: Started app v1.
agent # [ 4485.719454] agent flakelet[1059]: app: updated to generation 3
agent # [ 4485.721121] agent systemd[1]: flakelet-relay-job-app.service: Deactivated successfully.
client: (finished: must succeed: FLAKELET_RELAY_TOKEN_COMMAND='echo eyJhbGciOiJFZERTQSIsImtpZCI6ImsxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJmbGFrZWxldC1yZWxheSIsImV4cCI6MTc4ODY4MzA2MSwiZ3JvdXBzIjpbXSwiaWF0IjoxNzg4NjgyNzYxLCJpc3MiOiJodHRwczovL3JlbGF5Ojg0NDMiLCJqdGkiOiI2M0g1SFIyUDNZMlZDPT09IiwibmJmIjoxNzg4NjgyNzYxLCJzdWIiOiJyZXBvOmdpdGh1YjpleGFtcGxlL2FwcDpyZWY6cmVmcy9oZWFkcy9tYWluIn0.2MyrB94AoGvWlHKiSoKZdNE_C_aooQVRkLvsjrTxFIrIMxgwf0JrFjYOi2BwISPgllRD3HBK9nBFaU-iotvTAQ' push --relay https://relay:7443 deploy agent/app 2>&1, in 3.02 seconds)
(finished: subtest: deploy with OIDC token, sub principal, in 3.08 seconds)
subtest: policy denies uncovered target
client: must fail: FLAKELET_RELAY_TOKEN_COMMAND='echo eyJhbGciOiJFZERTQSIsImtpZCI6ImsxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJmbGFrZWxldC1yZWxheSIsImV4cCI6MTc4ODY4MzA2MSwiZ3JvdXBzIjpbXSwiaWF0IjoxNzg4NjgyNzYxLCJpc3MiOiJodHRwczovL3JlbGF5Ojg0NDMiLCJqdGkiOiI2M0g1SFIyUDNZMlZDPT09IiwibmJmIjoxNzg4NjgyNzYxLCJzdWIiOiJyZXBvOmdpdGh1YjpleGFtcGxlL2FwcDpyZWY6cmVmcy9oZWFkcy9tYWluIn0.2MyrB94AoGvWlHKiSoKZdNE_C_aooQVRkLvsjrTxFIrIMxgwf0JrFjYOi2BwISPgllRD3HBK9nBFaU-iotvTAQ' push --relay https://relay:7443 deploy agent/other 2>&1
client: (finished: must fail: FLAKELET_RELAY_TOKEN_COMMAND='echo eyJhbGciOiJFZERTQSIsImtpZCI6ImsxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJmbGFrZWxldC1yZWxheSIsImV4cCI6MTc4ODY4MzA2MSwiZ3JvdXBzIjpbXSwiaWF0IjoxNzg4NjgyNzYxLCJpc3MiOiJodHRwczovL3JlbGF5Ojg0NDMiLCJqdGkiOiI2M0g1SFIyUDNZMlZDPT09IiwibmJmIjoxNzg4NjgyNzYxLCJzdWIiOiJyZXBvOmdpdGh1YjpleGFtcGxlL2FwcDpyZWY6cmVmcy9oZWFkcy9tYWluIn0.2MyrB94AoGvWlHKiSoKZdNE_C_aooQVRkLvsjrTxFIrIMxgwf0JrFjYOi2BwISPgllRD3HBK9nBFaU-iotvTAQ' push --relay https://relay:7443 deploy agent/other 2>&1, in 0.02 seconds)
(finished: subtest: policy denies uncovered target, in 0.02 seconds)
subtest: claim principal from groups, unchanged counts as success
client: must succeed: echo '{"groups": ["wheel"]}' | step crypto jwt sign --key /nix/store/r7sclamg09v434w8zjlfnrcrn1w79wh6-flakelet-relay-test-issuer/jwk.json --iss https://relay:8443 --aud flakelet-relay --sub 'someone' --exp $(( $(date +%s) + 300 )) --jti $(head -c8 /dev/urandom | base32)
client: (finished: must succeed: echo '{"groups": ["wheel"]}' | step crypto jwt sign --key /nix/store/r7sclamg09v434w8zjlfnrcrn1w79wh6-flakelet-relay-test-issuer/jwk.json --iss https://relay:8443 --aud flakelet-relay --sub 'someone' --exp $(( $(date +%s) + 300 )) --jti $(head -c8 /dev/urandom | base32), in 0.05 seconds)
client: must succeed: FLAKELET_RELAY_TOKEN_COMMAND='echo eyJhbGciOiJFZERTQSIsImtpZCI6ImsxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJmbGFrZWxldC1yZWxheSIsImV4cCI6MTc4ODY4MzA2NCwiZ3JvdXBzIjpbIndoZWVsIl0sImlhdCI6MTc4ODY4Mjc2NCwiaXNzIjoiaHR0cHM6Ly9yZWxheTo4NDQzIiwianRpIjoiVEZURVQ2VjM0RVJCVT09PSIsIm5iZiI6MTc4ODY4Mjc2NCwic3ViIjoic29tZW9uZSJ9.E0pUXNcHewmb9QSQMPCQDlRdohKU5ezHOcSAqapusIqYOc5xPkAfdtFVeju0f3k1y51AwQdEF5Y2hLYPfGiVBw' push --relay https://relay:7443 deploy agent/other 2>&1
agent # [ 4486.825920] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:24.787696Z","level":"INFO","message":"update finished","flakelet":"app","status":"Updated","generation":"Some(3)","target":"flakelet_relay::agent::jobs"}
agent # [ 4486.919726] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:24.881482Z","level":"INFO","message":"start","id":"2fa777ab29b029159d6c3c0e355da55b","flakelet":"other","rule":"admins","relay":"relay","caller":"Some(\"oidc:mock:groups:wheel\\noidc:mock:someone\")","target":"flakelet_relay::agent::conn"}
agent # [ 4486.920110] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:24.881933Z","level":"INFO","message":"update starting","flakelet":"other","ids":"[\"2fa777ab29b029159d6c3c0e355da55b\"]","target":"flakelet_relay::agent::jobs"}
agent # [ 4486.960197] agent systemd[1]: Starting flakelet update other (via relay)...
agent # [ 4486.991020] agent systemd[1]: Started flakelet update other (via relay).
agent # [ 4486.995404] agent flakelet[1235]: other: using prebuilt artifact /nix/store/hmr990n0b4v6rpzrcpyyw738r9xi7z95-flakelet-other-v1
agent # [ 4486.995900] agent flakelet[1235]: other: up to date
agent # [ 4486.996428] agent systemd[1]: flakelet-relay-job-other.service: Deactivated successfully.
relay # [ 4486.827284] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:24.789032Z","level":"INFO","message":"deploy finished","job":"5d5e8cac9596c0da9bc30c6abf2d63ec","ok":true,"target":"flakelet_relay::relay::api"}
relay # [ 4486.919473] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:24.881197Z","level":"INFO","message":"deploy accepted","job":"9a2b7dd3805b54e0b139af73f1e9759b","caller":"oidc:mock:groups:wheel","targets":"[\"agent/other\"]","target":"flakelet_relay::relay::api"}
client: (finished: must succeed: FLAKELET_RELAY_TOKEN_COMMAND='echo eyJhbGciOiJFZERTQSIsImtpZCI6ImsxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJmbGFrZWxldC1yZWxheSIsImV4cCI6MTc4ODY4MzA2NCwiZ3JvdXBzIjpbIndoZWVsIl0sImlhdCI6MTc4ODY4Mjc2NCwiaXNzIjoiaHR0cHM6Ly9yZWxheTo4NDQzIiwianRpIjoiVEZURVQ2VjM0RVJCVT09PSIsIm5iZiI6MTc4ODY4Mjc2NCwic3ViIjoic29tZW9uZSJ9.E0pUXNcHewmb9QSQMPCQDlRdohKU5ezHOcSAqapusIqYOc5xPkAfdtFVeju0f3k1y51AwQdEF5Y2hLYPfGiVBw' push --relay https://relay:7443 deploy agent/other 2>&1, in 1.65 seconds)
(finished: subtest: claim principal from groups, unchanged counts as success, in 1.71 seconds)
subtest: push login via device flow, cached token used afterwards
client: must succeed: push login --issuer https://relay:8443 --client-id flakelet-relay 2>&1
agent # [ 4488.552643] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:26.514380Z","level":"INFO","message":"update finished","flakelet":"other","status":"Unchanged","generation":"Some(1)","target":"flakelet_relay::agent::jobs"}
relay # [ 4488.554139] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:26.515771Z","level":"INFO","message":"deploy finished","job":"9a2b7dd3805b54e0b139af73f1e9759b","ok":true,"target":"flakelet_relay::relay::api"}
relay # [ 4488.577730] relay 93sn3sv24fj26ksccpppzwg6kbf10s4w-device-mock[186]: 127.0.0.1 - - [06/Sep/2026 08:19:26] "POST /device HTTP/1.1" 200 -
relay # [ 4489.583825] relay 93sn3sv24fj26ksccpppzwg6kbf10s4w-device-mock[186]: 127.0.0.1 - - [06/Sep/2026 08:19:27] "POST /token HTTP/1.1" 400 -
client: (finished: must succeed: push login --issuer https://relay:8443 --client-id flakelet-relay 2>&1, in 2.08 seconds)
client: must succeed: test "$(stat -c %a /root/.local/state/flakelet-push/token.json)" = 600
client: (finished: must succeed: test "$(stat -c %a /root/.local/state/flakelet-push/token.json)" = 600, in 0.01 seconds)
client: must succeed: push --relay https://relay:7443 deploy agent/other 2>&1
agent # [ 4490.670283] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:28.632024Z","level":"INFO","message":"start","id":"b57846a3f70181fa8a4da9f8bf64f448","flakelet":"other","rule":"admins","relay":"relay","caller":"Some(\"oidc:mock:groups:wheel\\noidc:mock:someone\")","target":"flakelet_relay::agent::conn"}
agent # [ 4490.671233] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:28.632652Z","level":"INFO","message":"update starting","flakelet":"other","ids":"[\"b57846a3f70181fa8a4da9f8bf64f448\"]","target":"flakelet_relay::agent::jobs"}
agent # [ 4490.724992] agent systemd[1]: Starting flakelet update other (via relay)...
agent # [ 4490.756952] agent systemd[1]: Started flakelet update other (via relay).
agent # [ 4490.761693] agent flakelet[1274]: other: using prebuilt artifact /nix/store/hmr990n0b4v6rpzrcpyyw738r9xi7z95-flakelet-other-v1
agent # [ 4490.762173] agent flakelet[1274]: other: up to date
agent # [ 4490.764825] agent systemd[1]: flakelet-relay-job-other.service: Deactivated successfully.
relay # [ 4490.637189] relay 93sn3sv24fj26ksccpppzwg6kbf10s4w-device-mock[186]: 127.0.0.1 - - [06/Sep/2026 08:19:28] "POST /token HTTP/1.1" 200 -
relay # [ 4490.669912] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:28.631717Z","level":"INFO","message":"deploy accepted","job":"df334c4fa6725c4bc85aec269c56ca12","caller":"oidc:mock:groups:wheel","targets":"[\"agent/other\"]","target":"flakelet_relay::relay::api"}
client: (finished: must succeed: push --relay https://relay:7443 deploy agent/other 2>&1, in 0.66 seconds)
client: must succeed: push --relay https://relay:7443 deploy agent/other 2>&1
agent # [ 4491.308568] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:29.270333Z","level":"INFO","message":"update finished","flakelet":"other","status":"Unchanged","generation":"Some(1)","target":"flakelet_relay::agent::jobs"}
agent # [ 4491.329373] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:29.291179Z","level":"INFO","message":"start","id":"a5922f33efb0ab418d9afa0456e5b93e","flakelet":"other","rule":"admins","relay":"relay","caller":"Some(\"oidc:mock:groups:wheel\\noidc:mock:someone\")","target":"flakelet_relay::agent::conn"}
agent # [ 4491.329754] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:29.291579Z","level":"INFO","message":"update starting","flakelet":"other","ids":"[\"a5922f33efb0ab418d9afa0456e5b93e\"]","target":"flakelet_relay::agent::jobs"}
agent # [ 4491.366555] agent systemd[1]: Starting flakelet update other (via relay)...
agent # [ 4491.414122] agent systemd[1]: Started flakelet update other (via relay).
agent # [ 4491.418724] agent flakelet[1312]: other: using prebuilt artifact /nix/store/hmr990n0b4v6rpzrcpyyw738r9xi7z95-flakelet-other-v1
agent # [ 4491.419229] agent flakelet[1312]: other: up to date
agent # [ 4491.421026] agent systemd[1]: flakelet-relay-job-other.service: Deactivated successfully.
relay # [ 4491.310119] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:29.271834Z","level":"INFO","message":"deploy finished","job":"df334c4fa6725c4bc85aec269c56ca12","ok":true,"target":"flakelet_relay::relay::api"}
relay # [ 4491.328988] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:29.290803Z","level":"INFO","message":"deploy accepted","job":"c7858ad5d61ad5e6dd238530f01d5664","caller":"oidc:mock:groups:wheel","targets":"[\"agent/other\"]","target":"flakelet_relay::relay::api"}
client: (finished: must succeed: push --relay https://relay:7443 deploy agent/other 2>&1, in 0.65 seconds)
(finished: subtest: push login via device flow, cached token used afterwards, in 3.41 seconds)
subtest: waves: failing first wave skips the second
agent: must succeed: jq '.services.app.prebuilt="/nix/store/8282rnbvyk847q4l4zby3zwcq3jpzrvw-flakelet-app-broken"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json
agent: (finished: must succeed: jq '.services.app.prebuilt="/nix/store/8282rnbvyk847q4l4zby3zwcq3jpzrvw-flakelet-app-broken"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json, in 0.02 seconds)
client: must fail: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy agent/app --wave agent/other 2>&1
agent # [ 4491.961407] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:29.923179Z","level":"INFO","message":"update finished","flakelet":"other","status":"Unchanged","generation":"Some(1)","target":"flakelet_relay::agent::jobs"}
agent # [ 4491.997083] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:29.958884Z","level":"INFO","message":"start","id":"39f02f08c2eada80dc0e2d7433d8d7f5","flakelet":"app","rule":"admins","relay":"relay","caller":"Some(\"x509:email:admin@example.org\")","target":"flakelet_relay::agent::conn"}
agent # [ 4491.997465] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:29.959288Z","level":"INFO","message":"update starting","flakelet":"app","ids":"[\"39f02f08c2eada80dc0e2d7433d8d7f5\"]","target":"flakelet_relay::agent::jobs"}
agent # [ 4492.035789] agent systemd[1]: Starting flakelet update app (via relay)...
agent # [ 4492.085227] agent systemd[1]: Started flakelet update app (via relay).
agent # [ 4492.090041] agent flakelet[1343]: app: using prebuilt artifact /nix/store/8282rnbvyk847q4l4zby3zwcq3jpzrvw-flakelet-app-broken
agent # [ 4492.090693] agent flakelet[1343]: app: activating generation 4
agent # [ 4492.100381] agent systemd[1]: Stopping app v1...
agent # [ 4492.101093] agent systemd[1]: app.service: Deactivated successfully.
agent # [ 4492.101215] agent systemd[1]: Stopped app v1.
agent # [ 4492.110174] agent systemd[1]: Reload requested from client PID 1350 ('systemctl') (unit flakelet-relay-job-app.service)...
agent # [ 4492.110207] agent systemd[1]: Reloading...
relay # [ 4491.963075] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:29.924800Z","level":"INFO","message":"deploy finished","job":"c7858ad5d61ad5e6dd238530f01d5664","ok":true,"target":"flakelet_relay::relay::api"}
relay # [ 4491.996819] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:29.958623Z","level":"INFO","message":"deploy accepted","job":"75ef5f5be4a5fe966263e5925f6745cc","caller":"x509:email:admin@example.org","targets":"[\"agent/app\", \"agent/other\"]","target":"flakelet_relay::relay::api"}
agent # [ 4492.437420] agent systemd[1]: Reloading finished in 326 ms.
agent # [ 4492.505049] agent systemd[1]: Reload requested from client PID 1380 ('systemctl') (unit flakelet-relay-job-app.service)...
agent # [ 4492.505080] agent systemd[1]: Reloading...
agent # [ 4492.826174] agent systemd[1]: Reloading finished in 320 ms.
agent # [ 4492.894205] agent systemd[1]: Reload requested from client PID 1411 ('systemctl') (unit flakelet-relay-job-app.service)...
agent # [ 4492.894239] agent systemd[1]: Reloading...
agent # [ 4493.217369] agent systemd[1]: Reloading finished in 322 ms.
agent # [ 4493.296993] agent systemd[1]: Reload requested from client PID 1444 ('systemctl') (unit flakelet-relay-job-app.service)...
agent # [ 4493.297037] agent systemd[1]: Reloading...
agent # [ 4493.622935] agent systemd[1]: Reloading finished in 325 ms.
agent # [ 4493.691119] agent systemd[1]: Starting app broken...
agent # [ 4493.721473] agent (nonexistent)[1476]: app.service: Unable to locate executable '/nonexistent': No such file or directory
agent # [ 4493.721496] agent (nonexistent)[1476]: app.service: Failed at step EXEC spawning /nonexistent: No such file or directory
agent # [ 4493.723222] agent systemd[1]: app.service: Main process exited, code=exited, status=203/EXEC
agent # [ 4493.723315] agent systemd[1]: app.service: Failed with result 'exit-code'.
agent # [ 4493.723479] agent systemd[1]: Failed to start app broken.
agent # [ 4493.753688] agent systemd[1]: Reload requested from client PID 1487 ('systemctl') (unit flakelet-relay-job-app.service)...
agent # [ 4493.753723] agent systemd[1]: Reloading...
agent # [ 4494.080822] agent systemd[1]: Reloading finished in 326 ms.
agent # [ 4494.162429] agent systemd[1]: Reload requested from client PID 1517 ('systemctl') (unit flakelet-relay-job-app.service)...
agent # [ 4494.162465] agent systemd[1]: Reloading...
agent # [ 4494.487824] agent systemd[1]: Reloading finished in 324 ms.
agent # [ 4494.573830] agent systemd[1]: Reload requested from client PID 1547 ('systemctl') (unit flakelet-relay-job-app.service)...
agent # [ 4494.573866] agent systemd[1]: Reloading...
agent # [ 4494.896667] agent systemd[1]: Reloading finished in 322 ms.
agent # [ 4494.994058] agent systemd[1]: Reload requested from client PID 1580 ('systemctl') (unit flakelet-relay-job-app.service)...
agent # [ 4494.994096] agent systemd[1]: Reloading...
agent # [ 4495.318888] agent systemd[1]: Reloading finished in 324 ms.
agent # [ 4495.388693] agent systemd[1]: Starting app v1...
agent # [ 4495.432762] agent systemd[1]: Started app v1.
agent # [ 4495.434043] agent flakelet[1343]: app: deploy failed, rolled back: systemctl start app.service failed:
agent # [ 4495.434043] agent flakelet[1343]: failed units: app.service. See journalctl -u app.service
agent # [ 4495.435905] agent systemd[1]: flakelet-relay-job-app.service: Main process exited, code=exited, status=1/FAILURE
agent # [ 4495.435998] agent systemd[1]: flakelet-relay-job-app.service: Failed with result 'exit-code'.
client: (finished: must fail: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy agent/app --wave agent/other 2>&1, in 4.15 seconds)
agent: must succeed: systemctl show app.service -p Description | grep -q 'app v1'
agent: (finished: must succeed: systemctl show app.service -p Description | grep -q 'app v1', in 0.02 seconds)
(finished: subtest: waves: failing first wave skips the second, in 4.19 seconds)
subtest: host patterns expand on the relay
agent: must succeed: jq '.services.app.prebuilt="/nix/store/hysydfr96ds875vr0jgw5rjxi4p39h2j-flakelet-app-v1"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json
agent: (finished: must succeed: jq '.services.app.prebuilt="/nix/store/hysydfr96ds875vr0jgw5rjxi4p39h2j-flakelet-app-v1"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json, in 0.01 seconds)
client: must succeed: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy '@all/app' 2>&1
agent # [ 4496.127326] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:34.089093Z","level":"INFO","message":"update finished","flakelet":"app","status":"RolledBack","generation":"Some(3)","target":"flakelet_relay::agent::jobs"}
agent # [ 4496.184042] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:34.145810Z","level":"INFO","message":"start","id":"fbc2538011f662cb0a5de236a2ca6864","flakelet":"app","rule":"admins","relay":"relay","caller":"Some(\"x509:email:admin@example.org\")","target":"flakelet_relay::agent::conn"}
agent # [ 4496.184308] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:34.146134Z","level":"INFO","message":"update starting","flakelet":"app","ids":"[\"fbc2538011f662cb0a5de236a2ca6864\"]","target":"flakelet_relay::agent::jobs"}
agent # [ 4496.233470] agent systemd[1]: Starting flakelet update app (via relay)...
agent # [ 4496.262981] agent systemd[1]: Started flakelet update app (via relay).
agent # [ 4496.267505] agent flakelet[1651]: app: using prebuilt artifact /nix/store/hysydfr96ds875vr0jgw5rjxi4p39h2j-flakelet-app-v1
agent # [ 4496.268023] agent flakelet[1651]: app: up to date
agent # [ 4496.268614] agent systemd[1]: flakelet-relay-job-app.service: Deactivated successfully.
relay # [ 4496.128363] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:34.090085Z","level":"INFO","message":"deploy finished","job":"75ef5f5be4a5fe966263e5925f6745cc","ok":false,"target":"flakelet_relay::relay::api"}
relay # [ 4496.183842] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:34.145594Z","level":"INFO","message":"deploy accepted","job":"fbebd5b39d6f5f7be34a3217fa63fe1e","caller":"x509:email:admin@example.org","targets":"[\"agent/app\"]","target":"flakelet_relay::relay::api"}
client: (finished: must succeed: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy '@all/app' 2>&1, in 0.67 seconds)
client: must fail: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy '*/app' 2>&1
agent # [ 4496.832255] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:34.794049Z","level":"INFO","message":"update finished","flakelet":"app","status":"Unchanged","generation":"Some(3)","target":"flakelet_relay::agent::jobs"}
agent # [ 4496.855469] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:34.817279Z","level":"INFO","message":"start","id":"197bf481eef456c5177b789dd1e86911","flakelet":"app","rule":"admins","relay":"relay","caller":"Some(\"x509:email:admin@example.org\")","target":"flakelet_relay::agent::conn"}
agent # [ 4496.855716] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:34.817547Z","level":"INFO","message":"update starting","flakelet":"app","ids":"[\"197bf481eef456c5177b789dd1e86911\"]","target":"flakelet_relay::agent::jobs"}
agent # [ 4496.900379] agent systemd[1]: Starting flakelet update app (via relay)...
agent # [ 4496.924915] agent systemd[1]: Started flakelet update app (via relay).
agent # [ 4496.928407] agent flakelet[1689]: app: using prebuilt artifact /nix/store/hysydfr96ds875vr0jgw5rjxi4p39h2j-flakelet-app-v1
agent # [ 4496.928927] agent flakelet[1689]: app: up to date
agent # [ 4496.929862] agent systemd[1]: flakelet-relay-job-app.service: Deactivated successfully.
relay # [ 4496.833309] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:34.795044Z","level":"INFO","message":"deploy finished","job":"fbebd5b39d6f5f7be34a3217fa63fe1e","ok":true,"target":"flakelet_relay::relay::api"}
relay # [ 4496.855101] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:34.816902Z","level":"INFO","message":"deploy accepted","job":"9a4e7c1343036c053d36fb63e74435b2","caller":"x509:email:admin@example.org","targets":"[\"agent/app\", \"ghost/app\"]","target":"flakelet_relay::relay::api"}
client: (finished: must fail: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy '*/app' 2>&1, in 0.65 seconds)
client: must fail: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy agent/app --wave '*/app' 2>&1
agent # [ 4497.480056] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:35.441806Z","level":"INFO","message":"update finished","flakelet":"app","status":"Unchanged","generation":"Some(3)","target":"flakelet_relay::agent::jobs"}
agent # [ 4497.502783] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:35.464561Z","level":"INFO","message":"start","id":"515286f668c3a60c82318dafa8807ff1","flakelet":"app","rule":"admins","relay":"relay","caller":"Some(\"x509:email:admin@example.org\")","target":"flakelet_relay::agent::conn"}
agent # [ 4497.502974] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:35.464801Z","level":"INFO","message":"update starting","flakelet":"app","ids":"[\"515286f668c3a60c82318dafa8807ff1\"]","target":"flakelet_relay::agent::jobs"}
agent # [ 4497.560831] agent systemd[1]: Starting flakelet update app (via relay)...
agent # [ 4497.588995] agent systemd[1]: Started flakelet update app (via relay).
agent # [ 4497.593263] agent flakelet[1717]: app: using prebuilt artifact /nix/store/hysydfr96ds875vr0jgw5rjxi4p39h2j-flakelet-app-v1
agent # [ 4497.593735] agent flakelet[1717]: app: up to date
agent # [ 4497.595943] agent systemd[1]: flakelet-relay-job-app.service: Deactivated successfully.
relay # [ 4497.481260] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:35.443005Z","level":"INFO","message":"deploy finished","job":"9a4e7c1343036c053d36fb63e74435b2","ok":false,"target":"flakelet_relay::relay::api"}
relay # [ 4497.502610] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:35.464411Z","level":"INFO","message":"deploy accepted","job":"3ed0982d29273bb0fda943bbfccf3663","caller":"x509:email:admin@example.org","targets":"[\"agent/app\", \"ghost/app\"]","target":"flakelet_relay::relay::api"}
client: (finished: must fail: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy agent/app --wave '*/app' 2>&1, in 0.67 seconds)
client: must fail: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy '@all/nope' 2>&1
client: (finished: must fail: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy '@all/nope' 2>&1, in 0.02 seconds)
(finished: subtest: host patterns expand on the relay, in 2.02 seconds)
subtest: detach returns once the job is accepted
agent: must succeed: jq '.services.app.prebuilt="/nix/store/ln47rsd5km8a04hd5zlkz4q1v352zqh3-flakelet-app-slow2"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json
agent: (finished: must succeed: jq '.services.app.prebuilt="/nix/store/ln47rsd5km8a04hd5zlkz4q1v352zqh3-flakelet-app-slow2"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json, in 0.01 seconds)
client: must succeed: timeout 20 push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy --detach agent/app 2>&1
client: (finished: must succeed: timeout 20 push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy --detach agent/app 2>&1, in 0.02 seconds)
agent: waiting for success: systemctl is-active flakelet-relay-job-app.service
agent # [ 4498.147781] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:36.109531Z","level":"INFO","message":"update finished","flakelet":"app","status":"Unchanged","generation":"Some(3)","target":"flakelet_relay::agent::jobs"}
agent # [ 4498.201653] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:36.163418Z","level":"INFO","message":"start","id":"12bf059d3f3942e70f9508a8e804a7c0","flakelet":"app","rule":"admins","relay":"relay","caller":"Some(\"x509:email:admin@example.org\")","target":"flakelet_relay::agent::conn"}
agent # [ 4498.201878] agent flakelet-agent[304]: {"timestamp":"2026-09-06T08:19:36.163705Z","level":"INFO","message":"update starting","flakelet":"app","ids":"[\"12bf059d3f3942e70f9508a8e804a7c0\"]","target":"flakelet_relay::agent::jobs"}
agent # [ 4498.254887] agent systemd[1]: Starting flakelet update app (via relay)...
agent # [ 4498.284042] agent systemd[1]: Started flakelet update app (via relay).
agent # [ 4498.288448] agent flakelet[1749]: app: using prebuilt artifact /nix/store/ln47rsd5km8a04hd5zlkz4q1v352zqh3-flakelet-app-slow2
agent # [ 4498.289085] agent flakelet[1749]: app: activating generation 4
agent # [ 4498.301401] agent systemd[1]: Stopping app v1...
agent # [ 4498.301840] agent systemd[1]: app.service: Deactivated successfully.
agent # [ 4498.302116] agent systemd[1]: Stopped app v1.
agent # [ 4498.318233] agent systemd[1]: Reload requested from client PID 1756 ('systemctl') (unit flakelet-relay-job-app.service)...
agent # [ 4498.318297] agent systemd[1]: Reloading...
relay # [ 4498.148905] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:36.110655Z","level":"INFO","message":"deploy finished","job":"3ed0982d29273bb0fda943bbfccf3663","ok":false,"target":"flakelet_relay::relay::api"}
relay # [ 4498.201442] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:36.163201Z","level":"INFO","message":"deploy accepted","job":"6800e917a4af293f2eb524113736d111","caller":"x509:email:admin@example.org","targets":"[\"agent/app\"]","target":"flakelet_relay::relay::api"}
agent # [ 4498.655514] agent systemd[1]: Reloading finished in 336 ms.
agent # [ 4498.729471] agent systemd[1]: Reload requested from client PID 1786 ('systemctl') (unit flakelet-relay-job-app.service)...
agent # [ 4498.729508] agent systemd[1]: Reloading...
agent # [ 4499.056286] agent systemd[1]: Reloading finished in 326 ms.
agent # [ 4499.165043] agent systemd[1]: Reload requested from client PID 1817 ('systemctl') (unit flakelet-relay-job-app.service)...
agent # [ 4499.165079] agent systemd[1]: Reloading...
agent: (finished: waiting for success: systemctl is-active flakelet-relay-job-app.service, in 1.35 seconds)
??? Warning (UserWarning): wait_until_fails(): passing a bare int/float as a duration is deprecated. Use datetime.timedelta instead.
    File "/nix/store/fv4yr9c0f9kyrfc00100i4rlshiiidbd-nixos-test-driver-1.1/lib/python3.14/site-packages/test_driver/duration.py", line 39
agent: waiting for failure: systemctl is-active flakelet-relay-job-app.service
??? Warning (UserWarning): execute(): passing a bare int/float as a duration is deprecated. Use datetime.timedelta instead.
    File "/nix/store/fv4yr9c0f9kyrfc00100i4rlshiiidbd-nixos-test-driver-1.1/lib/python3.14/site-packages/test_driver/duration.py", line 39
agent # [ 4499.489476] agent systemd[1]: Reloading finished in 324 ms.
agent # [ 4499.562081] agent systemd[1]: Reload requested from client PID 1852 ('systemctl') (unit flakelet-relay-job-app.service)...
agent # [ 4499.562114] agent systemd[1]: Reloading...
agent # [ 4499.888826] agent systemd[1]: Reloading finished in 326 ms.
agent # [ 4499.965774] agent systemd[1]: Starting app slow2...
agent # [ 4506.044129] agent systemd[1]: Started app slow2.
agent # [ 4506.060217] agent flakelet[1749]: app: updated to generation 4
agent # [ 4506.062018] agent systemd[1]: flakelet-relay-job-app.service: Deactivated successfully.
agent: (finished: waiting for failure: systemctl is-active flakelet-relay-job-app.service, in 7.50 seconds)
(finished: subtest: detach returns once the job is accepted, in 8.89 seconds)
subtest: unknown host is final, missing agent is retried, agents listing
client: must fail: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy nope/app 2>&1
client: (finished: must fail: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy nope/app 2>&1, in 0.02 seconds)
agent: must succeed: systemctl stop flakelet-agent
agent: (finished: must succeed: systemctl stop flakelet-agent, in 0.04 seconds)
relay: waiting for success: curl -sf http://127.0.0.1:7400/metrics | grep -q 'agent_up{host="agent"} 0'
relay: (finished: waiting for success: curl -sf http://127.0.0.1:7400/metrics | grep -q 'agent_up{host="agent"} 0', in 0.02 seconds)
client: must succeed: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy agent/other > /tmp/retry.out 2>&1 &
client: (finished: must succeed: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy agent/other > /tmp/retry.out 2>&1 &, in 0.01 seconds)
client: waiting for success: grep -q 'agent_unavailable.*retrying' /tmp/retry.out
agent # [ 4507.096134] agent systemd[1]: Stopping flakelet-agent...
agent # [ 4507.124532] agent systemd[1]: flakelet-agent.service: Deactivated successfully.
agent # [ 4507.124922] agent systemd[1]: Stopped flakelet-agent.
agent # [ 4507.125366] agent systemd[1]: flakelet-agent.service: Consumed 2.988s CPU time over 32.415s wall clock time.
relay # [ 4507.096193] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:45.057934Z","level":"INFO","message":"agent disconnected: websocket closed","host":"agent","target":"flakelet_relay::relay::agent_conn"}
client: (finished: waiting for success: grep -q 'agent_unavailable.*retrying' /tmp/retry.out, in 1.02 seconds)
agent: must succeed: systemctl start flakelet-agent
agent: (finished: must succeed: systemctl start flakelet-agent, in 0.16 seconds)
??? Warning (UserWarning): wait_until_succeeds(): passing a bare int/float as a duration is deprecated. Use datetime.timedelta instead.
    File "/nix/store/fv4yr9c0f9kyrfc00100i4rlshiiidbd-nixos-test-driver-1.1/lib/python3.14/site-packages/test_driver/duration.py", line 39
client: waiting for success: grep -q 'agent/other: unchanged' /tmp/retry.out
agent # [ 4508.183264] agent systemd[1]: Starting flakelet-agent...
agent # [ 4508.238059] agent flakelet-agent[1952]: {"timestamp":"2026-09-06T08:19:46.199840Z","level":"INFO","message":"reattaching to running unit","flakelet":"app","ids":"[\"12bf059d3f3942e70f9508a8e804a7c0\"]","target":"flakelet_relay::agent::jobs"}
agent # [ 4508.324988] agent systemd[1]: Started flakelet-agent.
agent # [ 4508.335618] agent flakelet-agent[1952]: {"timestamp":"2026-09-06T08:19:46.297412Z","level":"INFO","message":"connected","relay":"https://relay:7443","relay_name":"relay","relay_version":"0.1.0","host":"agent","target":"flakelet_relay::agent::conn"}
relay # [ 4508.335686] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:46.297423Z","level":"INFO","message":"agent connected","host":"agent","version":"0.1.0","flakelets":"[Named { name: \"app\", generation: Some(4), revision: Some(\"prebuilt:app\") }, Named { name: \"other\", generation: Some(1), revision: Some(\"prebuilt:other\") }]","target":"flakelet_relay::relay::agent_conn"}
agent # [ 4508.795884] agent flakelet-agent[1952]: {"timestamp":"2026-09-06T08:19:46.757600Z","level":"INFO","message":"update finished","flakelet":"app","status":"Updated","generation":"Some(4)","target":"flakelet_relay::agent::jobs"}
agent # [ 4510.169735] agent flakelet-agent[1952]: {"timestamp":"2026-09-06T08:19:48.131441Z","level":"INFO","message":"start","id":"0e451e175fc58384e099ff87f303965c","flakelet":"other","rule":"admins","relay":"relay","caller":"Some(\"x509:email:admin@example.org\")","target":"flakelet_relay::agent::conn"}
agent # [ 4510.170636] agent flakelet-agent[1952]: {"timestamp":"2026-09-06T08:19:48.132280Z","level":"INFO","message":"update starting","flakelet":"other","ids":"[\"0e451e175fc58384e099ff87f303965c\"]","target":"flakelet_relay::agent::jobs"}
agent # [ 4510.224171] agent systemd[1]: Starting flakelet update other (via relay)...
agent # [ 4510.250946] agent systemd[1]: Started flakelet update other (via relay).
agent # [ 4510.254481] agent flakelet[2385]: other: using prebuilt artifact /nix/store/hmr990n0b4v6rpzrcpyyw738r9xi7z95-flakelet-other-v1
agent # [ 4510.254960] agent flakelet[2385]: other: up to date
agent # [ 4510.257875] agent systemd[1]: flakelet-relay-job-other.service: Deactivated successfully.
relay # [ 4510.169348] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:48.131079Z","level":"INFO","message":"deploy accepted","job":"9254837a90831217a89cc61211f155f5","caller":"x509:email:admin@example.org","targets":"[\"agent/other\"]","target":"flakelet_relay::relay::api"}
agent # [ 4510.809494] agent flakelet-agent[1952]: {"timestamp":"2026-09-06T08:19:48.771230Z","level":"INFO","message":"update finished","flakelet":"other","status":"Unchanged","generation":"Some(1)","target":"flakelet_relay::agent::jobs"}
relay # [ 4510.811191] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:48.772916Z","level":"INFO","message":"deploy finished","job":"9254837a90831217a89cc61211f155f5","ok":true,"target":"flakelet_relay::relay::api"}
client: (finished: waiting for success: grep -q 'agent/other: unchanged' /tmp/retry.out, in 3.04 seconds)
client: must succeed: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem agents
client: (finished: must succeed: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem agents, in 0.02 seconds)
client: must succeed: echo '{"groups": []}' | step crypto jwt sign --key /nix/store/r7sclamg09v434w8zjlfnrcrn1w79wh6-flakelet-relay-test-issuer/jwk.json --iss https://relay:8443 --aud flakelet-relay --sub 'repo:github:example/app:ref:refs/heads/main' --exp $(( $(date +%s) + 300 )) --jti $(head -c8 /dev/urandom | base32)
client: (finished: must succeed: echo '{"groups": []}' | step crypto jwt sign --key /nix/store/r7sclamg09v434w8zjlfnrcrn1w79wh6-flakelet-relay-test-issuer/jwk.json --iss https://relay:8443 --aud flakelet-relay --sub 'repo:github:example/app:ref:refs/heads/main' --exp $(( $(date +%s) + 300 )) --jti $(head -c8 /dev/urandom | base32), in 0.05 seconds)
client: must succeed: FLAKELET_RELAY_TOKEN_COMMAND='echo eyJhbGciOiJFZERTQSIsImtpZCI6ImsxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJmbGFrZWxldC1yZWxheSIsImV4cCI6MTc4ODY4MzA4OSwiZ3JvdXBzIjpbXSwiaWF0IjoxNzg4NjgyNzg5LCJpc3MiOiJodHRwczovL3JlbGF5Ojg0NDMiLCJqdGkiOiJNMzdZTDVGUkRISDU2PT09IiwibmJmIjoxNzg4NjgyNzg5LCJzdWIiOiJyZXBvOmdpdGh1YjpleGFtcGxlL2FwcDpyZWY6cmVmcy9oZWFkcy9tYWluIn0.JdQ3HIIPRLegYWQp8C-urvqDBmNPEwRtRXaXrBqlWbKRw4ZdWeTdYaZFqG7eX_umvn0nukiGYyXc-zQ-WiP1BA' push --relay https://relay:7443 agents
client: (finished: must succeed: FLAKELET_RELAY_TOKEN_COMMAND='echo eyJhbGciOiJFZERTQSIsImtpZCI6ImsxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJmbGFrZWxldC1yZWxheSIsImV4cCI6MTc4ODY4MzA4OSwiZ3JvdXBzIjpbXSwiaWF0IjoxNzg4NjgyNzg5LCJpc3MiOiJodHRwczovL3JlbGF5Ojg0NDMiLCJqdGkiOiJNMzdZTDVGUkRISDU2PT09IiwibmJmIjoxNzg4NjgyNzg5LCJzdWIiOiJyZXBvOmdpdGh1YjpleGFtcGxlL2FwcDpyZWY6cmVmcy9oZWFkcy9tYWluIn0.JdQ3HIIPRLegYWQp8C-urvqDBmNPEwRtRXaXrBqlWbKRw4ZdWeTdYaZFqG7eX_umvn0nukiGYyXc-zQ-WiP1BA' push --relay https://relay:7443 agents, in 0.02 seconds)
(finished: subtest: unknown host is final, missing agent is retried, agents listing, in 4.40 seconds)
subtest: out-of-band flakelet update reaches the relay without reconnect
agent: must succeed: flakelet status --json app | jq '.[0].generation'
agent: (finished: must succeed: flakelet status --json app | jq '.[0].generation', in 0.03 seconds)
agent: must succeed: journalctl -u flakelet-agent -o cat | grep -c '"connected"' || true
agent: (finished: must succeed: journalctl -u flakelet-agent -o cat | grep -c '"connected"' || true, in 0.01 seconds)
agent: must succeed: jq '.services.app.prebuilt="/nix/store/6ili1vxjlag9d7dy0bxrbxk6bpi5qh9p-flakelet-app-v2"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json
agent: (finished: must succeed: jq '.services.app.prebuilt="/nix/store/6ili1vxjlag9d7dy0bxrbxk6bpi5qh9p-flakelet-app-v2"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json, in 0.01 seconds)
agent: must succeed: flakelet update app
app: using prebuilt artifact /nix/store/6ili1vxjlag9d7dy0bxrbxk6bpi5qh9p-flakelet-app-v2
app: activating generation 5
agent # [ 4511.540657] agent systemd[1]: Stopping app slow2...
agent # [ 4511.541392] agent systemd[1]: app.service: Deactivated successfully.
agent # [ 4511.541529] agent systemd[1]: Stopped app slow2.
agent # [ 4511.553287] agent systemd[1]: Reload requested from client PID 2433 ('systemctl')...
agent # [ 4511.553324] agent systemd[1]: Reloading...
agent # [ 4511.882921] agent systemd[1]: Reloading finished in 329 ms.
agent # [ 4511.975443] agent systemd[1]: Reload requested from client PID 2462 ('systemctl')...
agent # [ 4511.975479] agent systemd[1]: Reloading...
agent # [ 4512.298636] agent systemd[1]: Reloading finished in 322 ms.
agent # [ 4512.384036] agent systemd[1]: Reload requested from client PID 2492 ('systemctl')...
agent # [ 4512.384079] agent systemd[1]: Reloading...
agent # [ 4512.708559] agent systemd[1]: Reloading finished in 324 ms.
agent # [ 4512.794051] agent systemd[1]: Reload requested from client PID 2524 ('systemctl')...
agent # [ 4512.794089] agent systemd[1]: Reloading...
agent: (finished: must succeed: flakelet update app, in 1.72 seconds)
client: waiting for success: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem agents | grep -q 'app@5'
agent # [ 4513.118825] agent systemd[1]: Reloading finished in 324 ms.
agent # [ 4513.187571] agent systemd[1]: Starting app v2...
agent # [ 4513.224077] agent systemd[1]: Started app v2.
client: (finished: waiting for success: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem agents | grep -q 'app@5', in 2.07 seconds)
agent: must succeed: journalctl -u flakelet-agent -o cat | grep -c '"connected"' || true
agent: (finished: must succeed: journalctl -u flakelet-agent -o cat | grep -c '"connected"' || true, in 0.01 seconds)
agent: must succeed: jq '.services.app.prebuilt="/nix/store/hysydfr96ds875vr0jgw5rjxi4p39h2j-flakelet-app-v1"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json
agent: (finished: must succeed: jq '.services.app.prebuilt="/nix/store/hysydfr96ds875vr0jgw5rjxi4p39h2j-flakelet-app-v1"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json, in 0.01 seconds)
agent: must succeed: flakelet update app
app: using prebuilt artifact /nix/store/hysydfr96ds875vr0jgw5rjxi4p39h2j-flakelet-app-v1
app: activating generation 6
agent # [ 4515.348610] agent systemd[1]: Stopping app v2...
agent # [ 4515.349348] agent systemd[1]: app.service: Deactivated successfully.
agent # [ 4515.349478] agent systemd[1]: Stopped app v2.
agent # [ 4515.359426] agent systemd[1]: Reload requested from client PID 2582 ('systemctl')...
agent # [ 4515.359463] agent systemd[1]: Reloading...
agent # [ 4515.686216] agent systemd[1]: Reloading finished in 326 ms.
agent # [ 4515.753803] agent systemd[1]: Reload requested from client PID 2611 ('systemctl')...
agent # [ 4515.753840] agent systemd[1]: Reloading...
agent # [ 4516.077352] agent systemd[1]: Reloading finished in 322 ms.
agent # [ 4516.142613] agent systemd[1]: Reload requested from client PID 2639 ('systemctl')...
agent # [ 4516.142649] agent systemd[1]: Reloading...
agent # [ 4516.466688] agent systemd[1]: Reloading finished in 323 ms.
agent # [ 4516.539211] agent systemd[1]: Reload requested from client PID 2671 ('systemctl')...
agent # [ 4516.539241] agent systemd[1]: Reloading...
agent: (finished: must succeed: flakelet update app, in 1.64 seconds)
client: waiting for success: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem agents | grep -q 'app@6'
agent # [ 4516.861402] agent systemd[1]: Reloading finished in 321 ms.
agent # [ 4516.932333] agent systemd[1]: Starting app v1...
agent # [ 4516.960005] agent systemd[1]: Started app v1.
client: (finished: waiting for success: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem agents | grep -q 'app@6', in 2.06 seconds)
(finished: subtest: out-of-band flakelet update reaches the relay without reconnect, in 7.57 seconds)
subtest: certificate not listed under agents cannot connect as agent
client: must succeed: curl -s -o /dev/null -w '%{http_code}' --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/mallory/cert.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/mallory/key.pem -H 'Upgrade: websocket' -H 'Connection: Upgrade' -H 'Sec-WebSocket-Version: 13' -H 'Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==' https://relay:7443/v1/agent
client: (finished: must succeed: curl -s -o /dev/null -w '%{http_code}' --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/mallory/cert.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/mallory/key.pem -H 'Upgrade: websocket' -H 'Connection: Upgrade' -H 'Sec-WebSocket-Version: 13' -H 'Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==' https://relay:7443/v1/agent, in 0.02 seconds)
(finished: subtest: certificate not listed under agents cannot connect as agent, in 0.03 seconds)
subtest: metrics
relay: must succeed: curl -sf http://127.0.0.1:7400/metrics
relay: (finished: must succeed: curl -sf http://127.0.0.1:7400/metrics, in 0.01 seconds)
(finished: subtest: metrics, in 0.01 seconds)
subtest: agent link drop mid-run replays missed lines once
agent: must succeed: jq '.services.app.prebuilt="/nix/store/c67jj8kx3m50nyj874aqmdqf21w5iznd-flakelet-app-slow"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json
agent: (finished: must succeed: jq '.services.app.prebuilt="/nix/store/c67jj8kx3m50nyj874aqmdqf21w5iznd-flakelet-app-slow"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json, in 0.01 seconds)
agent: must succeed: date +%s
agent: (finished: must succeed: date +%s, in 0.01 seconds)
client: must succeed: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy agent/app > /tmp/slow.out 2>&1 &
client: (finished: must succeed: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy agent/app > /tmp/slow.out 2>&1 &, in 0.01 seconds)
agent: waiting for success: journalctl -u flakelet-relay-job-app.service --since=@1788682797 | grep -q 'app-slow'
relay # [ 4519.059679] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:57.021407Z","level":"INFO","message":"agent rejected: no agents entry matches","principals":"[\"x509:dns:mallory\"]","target":"flakelet_relay::relay::agent_conn"}
relay # [ 4519.116493] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:57.078247Z","level":"INFO","message":"deploy accepted","job":"fc5d49a798d1b431af8fd0099b0d7754","caller":"x509:email:admin@example.org","targets":"[\"agent/app\"]","target":"flakelet_relay::relay::api"}
agent # [ 4519.116956] agent flakelet-agent[1952]: {"timestamp":"2026-09-06T08:19:57.078702Z","level":"INFO","message":"start","id":"a0aa3eb4699c49a364fd4d93a1df7cdf","flakelet":"app","rule":"admins","relay":"relay","caller":"Some(\"x509:email:admin@example.org\")","target":"flakelet_relay::agent::conn"}
agent # [ 4519.117800] agent flakelet-agent[1952]: {"timestamp":"2026-09-06T08:19:57.079403Z","level":"INFO","message":"update starting","flakelet":"app","ids":"[\"a0aa3eb4699c49a364fd4d93a1df7cdf\"]","target":"flakelet_relay::agent::jobs"}
agent # [ 4519.169893] agent systemd[1]: Starting flakelet update app (via relay)...
agent # [ 4519.203887] agent systemd[1]: Started flakelet update app (via relay).
agent # [ 4519.207991] agent flakelet[2735]: app: using prebuilt artifact /nix/store/c67jj8kx3m50nyj874aqmdqf21w5iznd-flakelet-app-slow
agent # [ 4519.208591] agent flakelet[2735]: app: activating generation 7
agent # [ 4519.221565] agent systemd[1]: Stopping app v1...
agent # [ 4519.222063] agent systemd[1]: app.service: Deactivated successfully.
agent # [ 4519.222324] agent systemd[1]: Stopped app v1.
agent # [ 4519.239562] agent systemd[1]: Reload requested from client PID 2742 ('systemctl') (unit flakelet-relay-job-app.service)...
agent # [ 4519.239628] agent systemd[1]: Reloading...
agent # [ 4519.577957] agent systemd[1]: Reloading finished in 337 ms.
agent # [ 4519.651551] agent systemd[1]: Reload requested from client PID 2772 ('systemctl') (unit flakelet-relay-job-app.service)...
agent # [ 4519.651589] agent systemd[1]: Reloading...
agent: (finished: waiting for success: journalctl -u flakelet-relay-job-app.service --since=@1788682797 | grep -q 'app-slow', in 1.02 seconds)
agent: must succeed: for a in $(getent ahosts relay | cut -d' ' -f1 | sort -u); do ip route add unreachable $a; done
agent: (finished: must succeed: for a in $(getent ahosts relay | cut -d' ' -f1 | sort -u); do ip route add unreachable $a; done, in 0.02 seconds)
relay: must succeed: for a in $(getent ahosts agent | cut -d' ' -f1 | sort -u); do ss -K dst "[$a]"; done
relay: (finished: must succeed: for a in $(getent ahosts agent | cut -d' ' -f1 | sort -u); do ss -K dst "[$a]"; done, in 0.06 seconds)
agent: must succeed: ss -K dport = 7443
agent # [ 4519.974709] agent systemd[1]: Reloading finished in 322 ms.
agent # [ 4520.053118] agent systemd[1]: Reload requested from client PID 2801 ('systemctl') (unit flakelet-relay-job-app.service)...
agent # [ 4520.053157] agent systemd[1]: Reloading...
agent: (finished: must succeed: ss -K dport = 7443, in 0.03 seconds)
agent: waiting for success: journalctl -u flakelet-relay-job-app.service --since=@1788682797 | grep -q 'updated to generation'
agent # [ 4520.231670] agent flakelet-agent[1952]: {"timestamp":"2026-09-06T08:19:58.193287Z","level":"INFO","message":"disconnected: Software caused connection abort (os error 103)","relay":"https://relay:7443","target":"flakelet_relay::agent::conn"}
agent # [ 4520.376518] agent systemd[1]: Reloading finished in 322 ms.
agent # [ 4520.453972] agent systemd[1]: Reload requested from client PID 2849 ('systemctl') (unit flakelet-relay-job-app.service)...
agent # [ 4520.454015] agent systemd[1]: Reloading...
relay # [ 4520.201286] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:19:58.163025Z","level":"INFO","message":"agent disconnected: Software caused connection abort (os error 103)","host":"agent","target":"flakelet_relay::relay::agent_conn"}
agent # [ 4520.779383] agent systemd[1]: Reloading finished in 325 ms.
agent # [ 4520.849444] agent systemd[1]: Starting app slow...
agent # [ 4521.234592] agent flakelet-agent[1952]: {"timestamp":"2026-09-06T08:19:59.196320Z","level":"WARN","message":"connect failed: connect relay:7443: No route to host (os error 113)","relay":"https://relay:7443","target":"flakelet_relay::agent::conn"}
agent # [ 4523.237832] agent flakelet-agent[1952]: {"timestamp":"2026-09-06T08:20:01.199546Z","level":"WARN","message":"connect failed: connect relay:7443: No route to host (os error 113)","relay":"https://relay:7443","target":"flakelet_relay::agent::conn"}
agent # [ 4526.911028] agent systemd[1]: Started app slow.
agent # [ 4526.928747] agent flakelet[2735]: app: updated to generation 7
agent # [ 4526.931230] agent systemd[1]: flakelet-relay-job-app.service: Deactivated successfully.
agent: (finished: waiting for success: journalctl -u flakelet-relay-job-app.service --since=@1788682797 | grep -q 'updated to generation', in 7.11 seconds)
agent: must succeed: for a in $(getent ahosts relay | cut -d' ' -f1 | sort -u); do ip route del unreachable $a; done
agent: (finished: must succeed: for a in $(getent ahosts relay | cut -d' ' -f1 | sort -u); do ip route del unreachable $a; done, in 0.01 seconds)
client: waiting for success: grep -q 'agent/app: updated' /tmp/slow.out
agent # [ 4527.240505] agent flakelet-agent[1952]: {"timestamp":"2026-09-06T08:20:05.202111Z","level":"WARN","message":"connect failed: connect relay:7443: No route to host (os error 113)","relay":"https://relay:7443","target":"flakelet_relay::agent::conn"}
agent # [ 4528.037225] agent flakelet-agent[1952]: {"timestamp":"2026-09-06T08:20:05.998965Z","level":"INFO","message":"update finished","flakelet":"app","status":"Updated","generation":"Some(7)","target":"flakelet_relay::agent::jobs"}
client: (finished: waiting for success: grep -q 'agent/app: updated' /tmp/slow.out, in 8.08 seconds)
client: must succeed: cat /tmp/slow.out
client: (finished: must succeed: cat /tmp/slow.out, in 0.01 seconds)
(finished: subtest: agent link drop mid-run replays missed lines once, in 16.37 seconds)
subtest: agent restart mid-run reattaches to the unit
agent: must succeed: jq '.services.app.prebuilt="/nix/store/ln47rsd5km8a04hd5zlkz4q1v352zqh3-flakelet-app-slow2"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json
agent: (finished: must succeed: jq '.services.app.prebuilt="/nix/store/ln47rsd5km8a04hd5zlkz4q1v352zqh3-flakelet-app-slow2"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json, in 0.01 seconds)
client: must succeed: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy agent/app > /tmp/restart.out 2>&1 &
client: (finished: must succeed: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy agent/app > /tmp/restart.out 2>&1 &, in 0.00 seconds)
agent: waiting for success: journalctl -u flakelet-relay-job-app.service --since -20s | grep -q 'app-slow2'
agent # [ 4535.244493] agent flakelet-agent[1952]: {"timestamp":"2026-09-06T08:20:13.206229Z","level":"INFO","message":"connected","relay":"https://relay:7443","relay_name":"relay","relay_version":"0.1.0","host":"agent","target":"flakelet_relay::agent::conn"}
agent # [ 4535.245524] agent flakelet-agent[1952]: {"timestamp":"2026-09-06T08:20:13.206413Z","level":"INFO","message":"start","id":"a0aa3eb4699c49a364fd4d93a1df7cdf","flakelet":"app","rule":"admins","relay":"relay","caller":"Some(\"x509:email:admin@example.org\")","target":"flakelet_relay::agent::conn"}
agent # [ 4535.480858] agent flakelet-agent[1952]: {"timestamp":"2026-09-06T08:20:13.442607Z","level":"INFO","message":"start","id":"abe982fc6094ddbd2ae474b1598d1f5c","flakelet":"app","rule":"admins","relay":"relay","caller":"Some(\"x509:email:admin@example.org\")","target":"flakelet_relay::agent::conn"}
agent # [ 4535.481239] agent flakelet-agent[1952]: {"timestamp":"2026-09-06T08:20:13.443066Z","level":"INFO","message":"update starting","flakelet":"app","ids":"[\"abe982fc6094ddbd2ae474b1598d1f5c\"]","target":"flakelet_relay::agent::jobs"}
relay # [ 4535.244633] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:20:13.206315Z","level":"INFO","message":"agent connected","host":"agent","version":"0.1.0","flakelets":"[Named { name: \"app\", generation: Some(7), revision: Some(\"prebuilt:app\") }, Named { name: \"other\", generation: Some(1), revision: Some(\"prebuilt:other\") }]","target":"flakelet_relay::relay::agent_conn"}
relay # [ 4535.245570] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:20:13.206669Z","level":"INFO","message":"deploy finished","job":"fc5d49a798d1b431af8fd0099b0d7754","ok":true,"target":"flakelet_relay::relay::api"}
relay # [ 4535.480626] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:20:13.442329Z","level":"INFO","message":"deploy accepted","job":"fcec88019960b634f620a5fb8ef1b38a","caller":"x509:email:admin@example.org","targets":"[\"agent/app\"]","target":"flakelet_relay::relay::api"}
agent # [ 4535.532139] agent systemd[1]: Starting flakelet update app (via relay)...
agent # [ 4535.562963] agent systemd[1]: Started flakelet update app (via relay).
agent # [ 4535.568135] agent flakelet[3028]: app: using prebuilt artifact /nix/store/ln47rsd5km8a04hd5zlkz4q1v352zqh3-flakelet-app-slow2
agent # [ 4535.568760] agent flakelet[3028]: app: activating generation 8
agent # [ 4535.582592] agent systemd[1]: Stopping app slow...
agent # [ 4535.583063] agent systemd[1]: app.service: Deactivated successfully.
agent # [ 4535.583334] agent systemd[1]: Stopped app slow.
agent # [ 4535.600043] agent systemd[1]: Reload requested from client PID 3035 ('systemctl') (unit flakelet-relay-job-app.service)...
agent # [ 4535.600109] agent systemd[1]: Reloading...
agent # [ 4535.937722] agent systemd[1]: Reloading finished in 336 ms.
agent # [ 4536.018375] agent systemd[1]: Reload requested from client PID 3065 ('systemctl') (unit flakelet-relay-job-app.service)...
agent # [ 4536.018413] agent systemd[1]: Reloading...
agent # [ 4536.342245] agent systemd[1]: Reloading finished in 323 ms.
agent: (finished: waiting for success: journalctl -u flakelet-relay-job-app.service --since -20s | grep -q 'app-slow2', in 1.02 seconds)
agent: must succeed: systemctl restart flakelet-agent
agent # [ 4536.420649] agent systemd[1]: Reload requested from client PID 3096 ('systemctl') (unit flakelet-relay-job-app.service)...
agent # [ 4536.420685] agent systemd[1]: Reloading...
agent # [ 4536.742702] agent systemd[1]: Reloading finished in 321 ms.
agent # [ 4536.809896] agent systemd[1]: Stopping flakelet-agent...
agent # [ 4536.815529] agent systemd[1]: Reload requested from client PID 3133 ('systemctl') (unit flakelet-relay-job-app.service)...
agent # [ 4536.815570] agent systemd[1]: Reloading...
relay # [ 4536.810352] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:20:14.772094Z","level":"INFO","message":"agent disconnected: websocket closed","host":"agent","target":"flakelet_relay::relay::agent_conn"}
agent: (finished: must succeed: systemctl restart flakelet-agent, in 0.88 seconds)
agent: must succeed: test -n "$(ls /var/lib/flakelet-agent/jobs)"
agent: (finished: must succeed: test -n "$(ls /var/lib/flakelet-agent/jobs)", in 0.01 seconds)
client: waiting for success: grep -q 'agent/app: updated' /tmp/restart.out
agent # [ 4537.156765] agent systemd[1]: Reloading finished in 340 ms.
agent # [ 4537.235727] agent systemd[1]: flakelet-agent.service: Deactivated successfully.
agent # [ 4537.235906] agent systemd[1]: Stopped flakelet-agent.
agent # [ 4537.235921] agent systemd[1]: flakelet-agent.service: Consumed 2.108s CPU time over 29.052s wall clock time.
agent # [ 4537.239635] agent systemd[1]: Starting flakelet-agent...
agent # [ 4537.242727] agent systemd[1]: Starting app slow2...
agent # [ 4537.291388] agent flakelet-agent[3164]: {"timestamp":"2026-09-06T08:20:15.253199Z","level":"INFO","message":"reattaching to running unit","flakelet":"app","ids":"[\"abe982fc6094ddbd2ae474b1598d1f5c\"]","target":"flakelet_relay::agent::jobs"}
agent # [ 4537.369509] agent systemd[1]: Started flakelet-agent.
agent # [ 4537.383877] agent flakelet-agent[3164]: {"timestamp":"2026-09-06T08:20:15.345627Z","level":"INFO","message":"connected","relay":"https://relay:7443","relay_name":"relay","relay_version":"0.1.0","host":"agent","target":"flakelet_relay::agent::conn"}
agent # [ 4537.384128] agent flakelet-agent[3164]: {"timestamp":"2026-09-06T08:20:15.345801Z","level":"INFO","message":"start","id":"abe982fc6094ddbd2ae474b1598d1f5c","flakelet":"app","rule":"admins","relay":"relay","caller":"Some(\"x509:email:admin@example.org\")","target":"flakelet_relay::agent::conn"}
relay # [ 4537.383946] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:20:15.345694Z","level":"INFO","message":"agent connected","host":"agent","version":"0.1.0","flakelets":"[Named { name: \"app\", generation: Some(7), revision: Some(\"prebuilt:app\") }, Named { name: \"other\", generation: Some(1), revision: Some(\"prebuilt:other\") }]","target":"flakelet_relay::relay::agent_conn"}
agent # [ 4543.315647] agent systemd[1]: Started app slow2.
agent # [ 4543.332803] agent flakelet[3028]: app: updated to generation 8
agent # [ 4543.335592] agent systemd[1]: flakelet-relay-job-app.service: Deactivated successfully.
agent # [ 4543.876891] agent flakelet-agent[3164]: {"timestamp":"2026-09-06T08:20:21.838634Z","level":"INFO","message":"update finished","flakelet":"app","status":"Updated","generation":"Some(8)","target":"flakelet_relay::agent::jobs"}
relay # [ 4543.878436] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:20:21.840177Z","level":"INFO","message":"deploy finished","job":"fcec88019960b634f620a5fb8ef1b38a","ok":true,"target":"flakelet_relay::relay::api"}
client: (finished: waiting for success: grep -q 'agent/app: updated' /tmp/restart.out, in 7.08 seconds)
client: must succeed: cat /tmp/restart.out
client: (finished: must succeed: cat /tmp/restart.out, in 0.01 seconds)
agent: must succeed: systemctl is-active app.service
agent: (finished: must succeed: systemctl is-active app.service, in 0.01 seconds)
(finished: subtest: agent restart mid-run reattaches to the unit, in 9.04 seconds)
subtest: relay restart mid-run, push resumes via /v1/jobs
agent: must succeed: jq '.services.app.prebuilt="/nix/store/c67jj8kx3m50nyj874aqmdqf21w5iznd-flakelet-app-slow"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json
agent: (finished: must succeed: jq '.services.app.prebuilt="/nix/store/c67jj8kx3m50nyj874aqmdqf21w5iznd-flakelet-app-slow"' /etc/flakelet/config.json > /tmp/config.json && cp --remove-destination /tmp/config.json /etc/flakelet/config.json, in 0.01 seconds)
agent: must succeed: date +%s
agent: (finished: must succeed: date +%s, in 0.01 seconds)
client: must succeed: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy agent/app > /tmp/resume.out 2>&1 &
client: (finished: must succeed: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy agent/app > /tmp/resume.out 2>&1 &, in 0.01 seconds)
agent: waiting for success: journalctl -u flakelet-relay-job-app.service --since=@1788682822 | grep -q 'app-slow'
agent # [ 4544.530497] agent flakelet-agent[3164]: {"timestamp":"2026-09-06T08:20:22.492248Z","level":"INFO","message":"start","id":"4a0077e6bf403a3622aaee6ccf6cc304","flakelet":"app","rule":"admins","relay":"relay","caller":"Some(\"x509:email:admin@example.org\")","target":"flakelet_relay::agent::conn"}
agent # [ 4544.531113] agent flakelet-agent[3164]: {"timestamp":"2026-09-06T08:20:22.492934Z","level":"INFO","message":"update starting","flakelet":"app","ids":"[\"4a0077e6bf403a3622aaee6ccf6cc304\"]","target":"flakelet_relay::agent::jobs"}
agent # [ 4544.582552] agent systemd[1]: Starting flakelet update app (via relay)...
agent # [ 4544.611044] agent systemd[1]: Started flakelet update app (via relay).
agent # [ 4544.614671] agent flakelet[3649]: app: using prebuilt artifact /nix/store/c67jj8kx3m50nyj874aqmdqf21w5iznd-flakelet-app-slow
agent # [ 4544.615262] agent flakelet[3649]: app: activating generation 9
agent # [ 4544.629113] agent systemd[1]: Stopping app slow2...
agent # [ 4544.630752] agent systemd[1]: app.service: Deactivated successfully.
agent # [ 4544.631090] agent systemd[1]: Stopped app slow2.
agent # [ 4544.646447] agent systemd[1]: Reload requested from client PID 3656 ('systemctl') (unit flakelet-relay-job-app.service)...
agent # [ 4544.646513] agent systemd[1]: Reloading...
relay # [ 4544.530061] relay flakelet-relay[400]: {"timestamp":"2026-09-06T08:20:22.491766Z","level":"INFO","message":"deploy accepted","job":"486e6ce5b02ce6df82a2d8eb7c4c4ecd","caller":"x509:email:admin@example.org","targets":"[\"agent/app\"]","target":"flakelet_relay::relay::api"}
agent # [ 4544.984364] agent systemd[1]: Reloading finished in 337 ms.
agent # [ 4545.075825] agent systemd[1]: Reload requested from client PID 3686 ('systemctl') (unit flakelet-relay-job-app.service)...
agent # [ 4545.075861] agent systemd[1]: Reloading...
agent: (finished: waiting for success: journalctl -u flakelet-relay-job-app.service --since=@1788682822 | grep -q 'app-slow', in 1.02 seconds)
relay: must succeed: systemctl restart flakelet-relay
agent # [ 4545.400218] agent systemd[1]: Reloading finished in 323 ms.
agent # [ 4545.468544] agent systemd[1]: Reload requested from client PID 3717 ('systemctl') (unit flakelet-relay-job-app.service)...
agent # [ 4545.468579] agent systemd[1]: Reloading...
agent # [ 4545.557873] agent flakelet-agent[3164]: {"timestamp":"2026-09-06T08:20:23.519557Z","level":"INFO","message":"disconnected: websocket closed","relay":"https://relay:7443","target":"flakelet_relay::agent::conn"}
relay # [ 4545.557338] relay systemd[1]: Stopping flakelet-relay...
relay # [ 4545.588599] relay systemd[1]: flakelet-relay.service: Deactivated successfully.
relay # [ 4545.588993] relay systemd[1]: Stopped flakelet-relay.
relay # [ 4545.589054] relay systemd[1]: flakelet-relay.service: Consumed 2.721s CPU time over 1min 9.665s wall clock time.
relay # [ 4545.625075] relay systemd[1]: Starting flakelet-relay...
agent # [ 4545.798579] agent systemd[1]: Reloading finished in 329 ms.
agent # [ 4545.890603] agent systemd[1]: Reload requested from client PID 3753 ('systemctl') (unit flakelet-relay-job-app.service)...
agent # [ 4545.890639] agent systemd[1]: Reloading...
agent # [ 4546.220379] agent systemd[1]: Reloading finished in 329 ms.
agent # [ 4546.304459] agent systemd[1]: Starting app slow...
agent # [ 4546.560167] agent flakelet-agent[3164]: {"timestamp":"2026-09-06T08:20:24.521776Z","level":"WARN","message":"connect failed: connect relay:7443: Connection refused (os error 111)","relay":"https://relay:7443","target":"flakelet_relay::agent::conn"}
relay: (finished: must succeed: systemctl restart flakelet-relay, in 1.48 seconds)
client: waiting for success: grep -q '» ok' /tmp/resume.out
relay # [ 4547.025283] relay flakelet-relay[934]: {"timestamp":"2026-09-06T08:20:24.987035Z","level":"INFO","message":"listening on 127.0.0.1:7400","target":"flakelet_relay::relay::server"}
relay # [ 4547.025785] relay flakelet-relay[934]: {"timestamp":"2026-09-06T08:20:24.987486Z","level":"INFO","message":"listening on [::]:7443","target":"flakelet_relay::relay::server"}
relay # [ 4547.025835] relay systemd[1]: Started flakelet-relay.
agent # [ 4548.574909] agent flakelet-agent[3164]: {"timestamp":"2026-09-06T08:20:26.536680Z","level":"INFO","message":"connected","relay":"https://relay:7443","relay_name":"relay","relay_version":"0.1.0","host":"agent","target":"flakelet_relay::agent::conn"}
relay # [ 4548.575049] relay flakelet-relay[934]: {"timestamp":"2026-09-06T08:20:26.536813Z","level":"INFO","message":"agent connected","host":"agent","version":"0.1.0","flakelets":"[Named { name: \"app\", generation: Some(8), revision: Some(\"prebuilt:app\") }, Named { name: \"other\", generation: Some(1), revision: Some(\"prebuilt:other\") }]","target":"flakelet_relay::relay::agent_conn"}
relay # [ 4548.577328] relay flakelet-relay[934]: {"timestamp":"2026-09-06T08:20:26.539092Z","level":"INFO","message":"job reattached","job":"486e6ce5b02ce6df82a2d8eb7c4c4ecd","targets":"[\"agent/app\"]","target":"flakelet_relay::relay::api"}
agent # [ 4552.378018] agent systemd[1]: Started app slow.
agent # [ 4552.397405] agent flakelet[3649]: app: updated to generation 9
agent # [ 4552.400011] agent systemd[1]: flakelet-relay-job-app.service: Deactivated successfully.
agent # [ 4553.479656] agent flakelet-agent[3164]: {"timestamp":"2026-09-06T08:20:31.441445Z","level":"INFO","message":"update finished","flakelet":"app","status":"Updated","generation":"Some(9)","target":"flakelet_relay::agent::jobs"}
client: (finished: waiting for success: grep -q '» ok' /tmp/resume.out, in 7.08 seconds)
client: must succeed: cat /tmp/resume.out
client: (finished: must succeed: cat /tmp/resume.out, in 0.01 seconds)
(finished: subtest: relay restart mid-run, push resumes via /v1/jobs, in 9.63 seconds)
subtest: job history comes from the agents and survives a relay restart
client: must succeed: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy --id hist1 agent/other 2>&1
agent # [ 4554.142705] agent flakelet-agent[3164]: {"timestamp":"2026-09-06T08:20:32.104457Z","level":"INFO","message":"start","id":"5be20c693e5c0b60e3f85779794dfb86","flakelet":"other","rule":"admins","relay":"relay","caller":"Some(\"x509:email:admin@example.org\")","target":"flakelet_relay::agent::conn"}
agent # [ 4554.143301] agent flakelet-agent[3164]: {"timestamp":"2026-09-06T08:20:32.105123Z","level":"INFO","message":"update starting","flakelet":"other","ids":"[\"5be20c693e5c0b60e3f85779794dfb86\"]","target":"flakelet_relay::agent::jobs"}
agent # [ 4554.193961] agent systemd[1]: Starting flakelet update other (via relay)...
agent # [ 4554.228977] agent systemd[1]: Started flakelet update other (via relay).
agent # [ 4554.233726] agent flakelet[3866]: other: using prebuilt artifact /nix/store/hmr990n0b4v6rpzrcpyyw738r9xi7z95-flakelet-other-v1
agent # [ 4554.234245] agent flakelet[3866]: other: up to date
agent # [ 4554.235625] agent systemd[1]: flakelet-relay-job-other.service: Deactivated successfully.
relay # [ 4554.142287] relay flakelet-relay[934]: {"timestamp":"2026-09-06T08:20:32.104038Z","level":"INFO","message":"deploy accepted","job":"188b06c4636b65849b3ba8daac22dc7c","caller":"x509:email:admin@example.org","targets":"[\"agent/other\"]","target":"flakelet_relay::relay::api"}
client: (finished: must succeed: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy --id hist1 agent/other 2>&1, in 0.67 seconds)
relay: must succeed: systemctl restart flakelet-relay
agent # [ 4554.783137] agent flakelet-agent[3164]: {"timestamp":"2026-09-06T08:20:32.744843Z","level":"INFO","message":"update finished","flakelet":"other","status":"Unchanged","generation":"Some(1)","target":"flakelet_relay::agent::jobs"}
agent # [ 4554.799272] agent flakelet-agent[3164]: {"timestamp":"2026-09-06T08:20:32.761082Z","level":"INFO","message":"disconnected: websocket closed","relay":"https://relay:7443","target":"flakelet_relay::agent::conn"}
relay # [ 4554.784295] relay flakelet-relay[934]: {"timestamp":"2026-09-06T08:20:32.746059Z","level":"INFO","message":"deploy finished","job":"188b06c4636b65849b3ba8daac22dc7c","ok":true,"target":"flakelet_relay::relay::api"}
relay # [ 4554.799009] relay systemd[1]: Stopping flakelet-relay...
relay # [ 4554.848547] relay systemd[1]: flakelet-relay.service: Deactivated successfully.
relay # [ 4554.848940] relay systemd[1]: Stopped flakelet-relay.
relay # [ 4554.848977] relay systemd[1]: flakelet-relay.service: Consumed 2.363s CPU time over 9.223s wall clock time.
relay # [ 4554.885233] relay systemd[1]: Starting flakelet-relay...
agent # [ 4555.801870] agent flakelet-agent[3164]: {"timestamp":"2026-09-06T08:20:33.763620Z","level":"WARN","message":"connect failed: connect relay:7443: Connection refused (os error 111)","relay":"https://relay:7443","target":"flakelet_relay::agent::conn"}
relay: (finished: must succeed: systemctl restart flakelet-relay, in 1.50 seconds)
relay: waiting for success: curl -sf http://127.0.0.1:7400/metrics | grep -q 'agent_up{host="agent"} 1'
relay # [ 4556.286999] relay flakelet-relay[1324]: {"timestamp":"2026-09-06T08:20:34.248753Z","level":"INFO","message":"listening on 127.0.0.1:7400","target":"flakelet_relay::relay::server"}
relay # [ 4556.287847] relay flakelet-relay[1324]: {"timestamp":"2026-09-06T08:20:34.249262Z","level":"INFO","message":"listening on [::]:7443","target":"flakelet_relay::relay::server"}
relay # [ 4556.287639] relay systemd[1]: Started flakelet-relay.
agent # [ 4557.812226] agent flakelet-agent[3164]: {"timestamp":"2026-09-06T08:20:35.773979Z","level":"INFO","message":"connected","relay":"https://relay:7443","relay_name":"relay","relay_version":"0.1.0","host":"agent","target":"flakelet_relay::agent::conn"}
relay # [ 4557.812363] relay flakelet-relay[1324]: {"timestamp":"2026-09-06T08:20:35.774100Z","level":"INFO","message":"agent connected","host":"agent","version":"0.1.0","flakelets":"[Named { name: \"app\", generation: Some(9), revision: Some(\"prebuilt:app\") }, Named { name: \"other\", generation: Some(1), revision: Some(\"prebuilt:other\") }]","target":"flakelet_relay::relay::agent_conn"}
relay: (finished: waiting for success: curl -sf http://127.0.0.1:7400/metrics | grep -q 'agent_up{host="agent"} 1', in 2.05 seconds)
client: must succeed: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem jobs
client: (finished: must succeed: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem jobs, in 0.02 seconds)
client: must succeed: echo '{"groups": []}' | step crypto jwt sign --key /nix/store/r7sclamg09v434w8zjlfnrcrn1w79wh6-flakelet-relay-test-issuer/jwk.json --iss https://relay:8443 --aud flakelet-relay --sub 'repo:github:example/app:ref:refs/heads/main' --exp $(( $(date +%s) + 300 )) --jti $(head -c8 /dev/urandom | base32)
client: (finished: must succeed: echo '{"groups": []}' | step crypto jwt sign --key /nix/store/r7sclamg09v434w8zjlfnrcrn1w79wh6-flakelet-relay-test-issuer/jwk.json --iss https://relay:8443 --aud flakelet-relay --sub 'repo:github:example/app:ref:refs/heads/main' --exp $(( $(date +%s) + 300 )) --jti $(head -c8 /dev/urandom | base32), in 0.05 seconds)
client: must succeed: FLAKELET_RELAY_TOKEN_COMMAND='echo eyJhbGciOiJFZERTQSIsImtpZCI6ImsxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJmbGFrZWxldC1yZWxheSIsImV4cCI6MTc4ODY4MzEzNiwiZ3JvdXBzIjpbXSwiaWF0IjoxNzg4NjgyODM2LCJpc3MiOiJodHRwczovL3JlbGF5Ojg0NDMiLCJqdGkiOiJHUVlXTEpTWDZMQ0RNPT09IiwibmJmIjoxNzg4NjgyODM2LCJzdWIiOiJyZXBvOmdpdGh1YjpleGFtcGxlL2FwcDpyZWY6cmVmcy9oZWFkcy9tYWluIn0.y6G6hWOfR5NacFu8RqsqKdsr7VlpZHUOzd8Tv3ZuBskcPfge7OCLZXeU9MokOr6XlNRFID-z6xfNUpLJ_GeCDw' push --relay https://relay:7443 jobs
client: (finished: must succeed: FLAKELET_RELAY_TOKEN_COMMAND='echo eyJhbGciOiJFZERTQSIsImtpZCI6ImsxIiwidHlwIjoiSldUIn0.eyJhdWQiOiJmbGFrZWxldC1yZWxheSIsImV4cCI6MTc4ODY4MzEzNiwiZ3JvdXBzIjpbXSwiaWF0IjoxNzg4NjgyODM2LCJpc3MiOiJodHRwczovL3JlbGF5Ojg0NDMiLCJqdGkiOiJHUVlXTEpTWDZMQ0RNPT09IiwibmJmIjoxNzg4NjgyODM2LCJzdWIiOiJyZXBvOmdpdGh1YjpleGFtcGxlL2FwcDpyZWY6cmVmcy9oZWFkcy9tYWluIn0.y6G6hWOfR5NacFu8RqsqKdsr7VlpZHUOzd8Tv3ZuBskcPfge7OCLZXeU9MokOr6XlNRFID-z6xfNUpLJ_GeCDw' push --relay https://relay:7443 jobs, in 0.02 seconds)
(finished: subtest: job history comes from the agents and survives a relay restart, in 4.31 seconds)
subtest: dashboard login via authorization code flow
client: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -o /dev/null -w '%{http_code} %{redirect_url}' https://relay:7443/ui/ | grep -q '303 https://relay:7443/ui/login'
client: (finished: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -o /dev/null -w '%{http_code} %{redirect_url}' https://relay:7443/ui/ | grep -q '303 https://relay:7443/ui/login', in 0.02 seconds)
client: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -L https://relay:7443/ui/login
client: (finished: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -L https://relay:7443/ui/login, in 0.08 seconds)
client: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar https://relay:7443/ui/jobs
client: (finished: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar https://relay:7443/ui/jobs, in 0.02 seconds)
client: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -f https://relay:7443/v1/jobs | grep -q hist1
client: (finished: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -f https://relay:7443/v1/jobs | grep -q hist1, in 0.02 seconds)
client: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -N --max-time 10 https://relay:7443/ui/jobs/hist1/events || true
client: (finished: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -N --max-time 10 https://relay:7443/ui/jobs/hist1/events || true, in 0.02 seconds)
client: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -o /dev/null -w '%{http_code}' -X POST 'https://relay:7443/ui/deploy?arg=other' | grep -q 403
client: (finished: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -o /dev/null -w '%{http_code}' -X POST 'https://relay:7443/ui/deploy?arg=other' | grep -q 403, in 0.02 seconds)
client: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -D - -o /dev/null -H 'HX-Request: true' -X POST 'https://relay:7443/ui/deploy?arg=other'
client: (finished: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -D - -o /dev/null -H 'HX-Request: true' -X POST 'https://relay:7443/ui/deploy?arg=other', in 0.02 seconds)
client: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -N --max-time 30 https://relay:7443/ui/jobs/d8ae15318f69e01f3f3f90b63ec13bad/events || true
relay # [ 4558.477044] relay 93sn3sv24fj26ksccpppzwg6kbf10s4w-device-mock[186]: 127.0.0.1 - - [06/Sep/2026 08:20:36] "GET /authorize?response_type=code&client_id=dashboard&redirect_uri=https%3A%2F%2Frelay%3A7443%2Fui%2Fcallback&scope=openid%20profile%20email%20groups&state=uVbdmHWfX2wb9BNRiArnya6u9aSOaTQvGeORzZ0U5gM&code_challenge=CXctGMkvkoGS0KpRTAP18SpOWr8xoT57DLeNlxBfVVw&code_challenge_method=S256 HTTP/1.1" 302 -
relay # [ 4558.528979] relay 93sn3sv24fj26ksccpppzwg6kbf10s4w-device-mock[186]: 127.0.0.1 - - [06/Sep/2026 08:20:36] "POST /token HTTP/1.1" 200 -
relay # [ 4558.529785] relay flakelet-relay[1324]: {"timestamp":"2026-09-06T08:20:36.491589Z","level":"INFO","message":"login","name":"someone","principals":"[\"oidc:mock:someone\", \"oidc:mock:groups:wheel\"]","target":"flakelet_relay::relay::login"}
relay # [ 4558.586046] relay flakelet-relay[1324]: {"timestamp":"2026-09-06T08:20:36.547810Z","level":"INFO","message":"job reattached","job":"188b06c4636b65849b3ba8daac22dc7c","targets":"[\"agent/other\"]","target":"flakelet_relay::relay::api"}
relay # [ 4558.629499] relay flakelet-relay[1324]: {"timestamp":"2026-09-06T08:20:36.591257Z","level":"INFO","message":"deploy accepted","job":"b41bdb4d57a8cae9358eb92ac75f95c3","caller":"oidc:mock:someone","targets":"[\"agent/other\"]","target":"flakelet_relay::relay::api"}
relay # [ 4558.650839] relay flakelet-relay[1324]: {"timestamp":"2026-09-06T08:20:36.612640Z","level":"INFO","message":"job reattached","job":"b41bdb4d57a8cae9358eb92ac75f95c3","targets":"[\"agent/other\"]","target":"flakelet_relay::relay::api"}
agent # [ 4558.629806] agent flakelet-agent[3164]: {"timestamp":"2026-09-06T08:20:36.591577Z","level":"INFO","message":"start","id":"c2047ca37729d2845c4cc26d801ee7ce","flakelet":"other","rule":"admins","relay":"relay","caller":"Some(\"oidc:mock:someone\\noidc:mock:groups:wheel\")","target":"flakelet_relay::agent::conn"}
agent # [ 4558.630175] agent flakelet-agent[3164]: {"timestamp":"2026-09-06T08:20:36.592008Z","level":"INFO","message":"update starting","flakelet":"other","ids":"[\"c2047ca37729d2845c4cc26d801ee7ce\"]","target":"flakelet_relay::agent::jobs"}
agent # [ 4558.673534] agent systemd[1]: Starting flakelet update other (via relay)...
agent # [ 4558.704457] agent systemd[1]: Started flakelet update other (via relay).
agent # [ 4558.708802] agent flakelet[3914]: other: using prebuilt artifact /nix/store/hmr990n0b4v6rpzrcpyyw738r9xi7z95-flakelet-other-v1
agent # [ 4558.709327] agent flakelet[3914]: other: up to date
agent # [ 4558.710022] agent systemd[1]: flakelet-relay-job-other.service: Deactivated successfully.
client: (finished: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -N --max-time 30 https://relay:7443/ui/jobs/d8ae15318f69e01f3f3f90b63ec13bad/events || true, in 0.63 seconds)
client: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar https://relay:7443/ui/jobs/d8ae15318f69e01f3f3f90b63ec13bad
client: (finished: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar https://relay:7443/ui/jobs/d8ae15318f69e01f3f3f90b63ec13bad, in 0.02 seconds)
client: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -sf -o /dev/null https://relay:7443/ui/static/htmx.min.js
client: (finished: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -sf -o /dev/null https://relay:7443/ui/static/htmx.min.js, in 0.02 seconds)
client: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar 'https://relay:7443/ui/?q=status%3Ahealthy+oth'
client: (finished: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar 'https://relay:7443/ui/?q=status%3Ahealthy+oth', in 0.02 seconds)
client: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar https://relay:7443/ui/flakelets/other
client: (finished: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar https://relay:7443/ui/flakelets/other, in 0.02 seconds)
client: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar 'https://relay:7443/ui/hosts?q=status%3Adisconnected'
client: (finished: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar 'https://relay:7443/ui/hosts?q=status%3Adisconnected', in 0.02 seconds)
client: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -o /dev/null -w '%{http_code}' -H 'HX-Request: true' -X POST 'https://relay:7443/ui/retry?arg=d8ae15318f69e01f3f3f90b63ec13bad' | grep -q 400
client: (finished: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -o /dev/null -w '%{http_code}' -H 'HX-Request: true' -X POST 'https://relay:7443/ui/retry?arg=d8ae15318f69e01f3f3f90b63ec13bad' | grep -q 400, in 0.02 seconds)
client: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -N --max-time 15 'https://relay:7443/ui/events?path=jobs' > /tmp/ev.out 2>&1 &
client: (finished: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -N --max-time 15 'https://relay:7443/ui/events?path=jobs' > /tmp/ev.out 2>&1 &, in 0.01 seconds)
client: must succeed: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy --id live1 agent/other
» job 38796a7ba508eadfe82a1cd5def000bc via relay 0.1.0 (agents: agent 0.1.0)
» wave 0
agent/other | Starting flakelet update other (via relay)...
agent/other | Started flakelet update other (via relay).
agent/other | other: using prebuilt artifact /nix/store/hmr990n0b4v6rpzrcpyyw738r9xi7z95-flakelet-other-v1
agent/other | other: up to date
agent # [ 4559.258432] agent flakelet-agent[3164]: {"timestamp":"2026-09-06T08:20:37.220192Z","level":"INFO","message":"update finished","flakelet":"other","status":"Unchanged","generation":"Some(1)","target":"flakelet_relay::agent::jobs"}
agent # [ 4559.400107] agent flakelet-agent[3164]: {"timestamp":"2026-09-06T08:20:37.361899Z","level":"INFO","message":"start","id":"5e317fc57863c2a7c03ebe1bdacd1356","flakelet":"other","rule":"admins","relay":"relay","caller":"Some(\"x509:email:admin@example.org\")","target":"flakelet_relay::agent::conn"}
agent/otheragent # [ 4559.400460] agent flakelet-agent[3164]: {"timestamp":"2026-09-06T08:20:37.362292Z","level":"INFO","message":"update starting","flakelet":"other","ids":"[\"5e317fc57863c2a7c03ebe1bdacd1356\"]","target":"flakelet_relay::agent::jobs"}
 | flakelet-relay-job-other.service: Deactivated successfully.agent # [ 4559.456745] agent systemd[1]: Starting flakelet update other (via relay)...

agent # [ 4559.488998] agent systemd[1]: Started flakelet update other (via relay).
agent # [ 4559.492937] agent flakelet[3952]: other: using prebuilt artifact /nix/store/hmr990n0b4v6rpzrcpyyw738r9xi7z95-flakelet-other-v1
agent # [ 4559.493395] agent flakelet[3952]: other: up to date
agent # [ 4559.494816] agent systemd[1]: flakelet-relay-job-other.service: Deactivated successfully.
relay # [ 4559.399836] relay flakelet-relay[1324]: {"timestamp":"2026-09-06T08:20:37.361584Z","level":"INFO","message":"deploy accepted","job":"38796a7ba508eadfe82a1cd5def000bc","caller":"x509:email:admin@example.org","targets":"[\"agent/other\"]","target":"flakelet_relay::relay::api"}
» agent/other: unchanged (generation 1)
» ok
client: (finished: must succeed: push --relay https://relay:7443 --cert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin.pem --key /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/admin-key.pem deploy --id live1 agent/other, in 0.66 seconds)
client: waiting for success: grep -q live1 /tmp/ev.out
client: (finished: waiting for success: grep -q live1 /tmp/ev.out, in 0.01 seconds)
client: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -X POST -o /dev/null https://relay:7443/ui/logout
client: (finished: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -X POST -o /dev/null https://relay:7443/ui/logout, in 0.02 seconds)
client: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -o /dev/null -w '%{http_code}' https://relay:7443/ui/ | grep -q 303
client: (finished: must succeed: curl -sS --cacert /nix/store/ydql3jgw8j7khsad67k605wlnywsd36n-flakelet-relay-test-certs/ca.pem -b /tmp/jar -c /tmp/jar -o /dev/null -w '%{http_code}' https://relay:7443/ui/ | grep -q 303, in 0.02 seconds)
(finished: subtest: dashboard login via authorization code flow, in 1.67 seconds)
(finished: run the VM test script, in 87.65 seconds)
test script finished in 87.79s
cleanup
kill NspawnMachine (pid 52)
kill NspawnMachine (pid 53)
Container agent terminated by signal KILL.
relay # [ 4560.045157] relay flakelet-relay[1324]: {"timestamp":"2026-09-06T08:20:38.006832Z","level":"INFO","message":"deploy finished","job":"38796a7ba508eadfe82a1cd5def000bc","ok":true,"target":"flakelet_relay::relay::api"}
kill NspawnMachine (pid 56)
Container client terminated by signal KILL.
(finished: cleanup, in 0.39 seconds)
additionally exposed symbols:
    agent, client, relay,
    vlan1,
    start_all, test_script, machines, machines_qemu, machines_nspawn, vlans, driver, log, os, create_machine, subtest, run_tests, join_all, retry, serial_stdout_off, serial_stdout_on, polling_condition, BaseMachine, QemuMachine, NspawnMachine, t, debug, dump_machine_ssh
» job d30c8ccd9850f8cfcbe259628ad443a4 via relay 0.1.0 (agents: agent 0.1.0)
» wave 0
agent/app | Starting flakelet update app (via relay)...
agent/app | Started flakelet update app (via relay).
agent/app | app: using prebuilt artifact /nix/store/6ili1vxjlag9d7dy0bxrbxk6bpi5qh9p-flakelet-app-v2
agent/app | app: activating generation 2
agent/app | app: updated to generation 2
agent/app | flakelet-relay-job-app.service: Deactivated successfully.
» agent/app: updated (generation 2)
» ok

» job 75ef5f5be4a5fe966263e5925f6745cc via relay 0.1.0 (agents: agent 0.1.0)
» wave 0
agent/app | Starting flakelet update app (via relay)...
agent/app | Started flakelet update app (via relay).
agent/app | app: using prebuilt artifact /nix/store/8282rnbvyk847q4l4zby3zwcq3jpzrvw-flakelet-app-broken
agent/app | app: activating generation 4
agent/app | app: deploy failed, rolled back: systemctl start app.service failed:
agent/app | failed units: app.service. See journalctl -u app.service
agent/app | flakelet-relay-job-app.service: Main process exited, code=exited, status=1/FAILURE
agent/app | flakelet-relay-job-app.service: Failed with result 'exit-code'.
» agent/app: rolled-back (generation 3)
agent/app │ Starting app v1...
agent/app │ Started app v1.
agent/app │ Stopping app v1...
agent/app │ app.service: Deactivated successfully.
agent/app │ Stopped app v1.
agent/app │ Starting app v2...
agent/app │ Started app v2.
agent/app │ Stopping app v2...
agent/app │ app.service: Deactivated successfully.
agent/app │ Stopped app v2.
agent/app │ Starting app v1...
agent/app │ Started app v1.
agent/app │ Stopping app v1...
agent/app │ app.service: Deactivated successfully.
agent/app │ Stopped app v1.
agent/app │ Starting app broken...
agent/app │ app.service: Unable to locate executable '/nonexistent': No such file or directory
agent/app │ app.service: Failed at step EXEC spawning /nonexistent: No such file or directory
agent/app │ app.service: Main process exited, code=exited, status=203/EXEC
agent/app │ app.service: Failed with result 'exit-code'.
agent/app │ Failed to start app broken.
agent/app │ Starting app v1...
agent/app │ Started app v1.
» skipped: agent/other
» failed

» job fc5d49a798d1b431af8fd0099b0d7754 via relay 0.1.0 (agents: agent 0.1.0)
» wave 0
agent/app | Starting flakelet update app (via relay)...
agent/app | Started flakelet update app (via relay).
agent/app | app: using prebuilt artifact /nix/store/c67jj8kx3m50nyj874aqmdqf21w5iznd-flakelet-app-slow
agent/app | app: activating generation 7
agent/app | app: updated to generation 7
agent/app | flakelet-relay-job-app.service: Deactivated successfully.
» agent/app: updated (generation 7)
» ok

» job fcec88019960b634f620a5fb8ef1b38a via relay 0.1.0 (agents: agent 0.1.0)
» wave 0
agent/app | Starting flakelet update app (via relay)...
agent/app | Started flakelet update app (via relay).
agent/app | app: using prebuilt artifact /nix/store/ln47rsd5km8a04hd5zlkz4q1v352zqh3-flakelet-app-slow2
agent/app | app: activating generation 8
agent/app | app: updated to generation 8
agent/app | flakelet-relay-job-app.service: Deactivated successfully.
» agent/app: updated (generation 8)
» ok

» job 486e6ce5b02ce6df82a2d8eb7c4c4ecd via relay 0.1.0 (agents: agent 0.1.0)
» wave 0
agent/app | Starting flakelet update app (via relay)...
agent/app | Started flakelet update app (via relay).
agent/app | app: using prebuilt artifact /nix/store/c67jj8kx3m50nyj874aqmdqf21w5iznd-flakelet-app-slow
agent/app | app: activating generation 9
flakelet-push: stream: error reading a body from connection, resuming job a70a44a7a5707cece21054c1e6c80319
flakelet-push: https://relay:7443: connect relay:7443: Connection refused (os error 111), retrying in 1s
flakelet-push: https://relay:7443: connect relay:7443: Connection refused (os error 111), retrying in 2s
» job 486e6ce5b02ce6df82a2d8eb7c4c4ecd via relay 0.1.0 (agents: agent 0.1.0)
» wave 0
agent/app | app: updated to generation 9
agent/app | flakelet-relay-job-app.service: Deactivated successfully.
» agent/app: updated (generation 9)
» ok

1788682832	hist1	x509:email:admin@example.org	agent/other:unchanged
1788682822	a70a44a7a5707cece21054c1e6c80319	x509:email:admin@example.org	agent/app:updated
1788682813	a7f19d977a022febd46d4cec67d6fcdb	x509:email:admin@example.org	agent/app:updated
1788682797	810d8512d83958f19d115bd4660409a3	x509:email:admin@example.org	agent/app:updated
1788682788	95c539f7cf0849fb0391973a03262a4d	x509:email:admin@example.org	agent/other:unchanged
1788682776	d88d64c0291e3d21bee391609fc3c7ac	x509:email:admin@example.org	agent/app:updated
1788682775	23529942aa3b7352cdd813ac71102efd	x509:email:admin@example.org	agent/app:unchanged
1788682774	2050bc877a2aeaf686b3f39e89a211e1	x509:email:admin@example.org	agent/app:unchanged
1788682774	aaad37000fcfecda421a60238df9df73	x509:email:admin@example.org	agent/app:unchanged
1788682769	cf02666942f1874edb95b0fc688392c2	x509:email:admin@example.org	agent/app:rolled-back
1788682769	ff51713f438e89d848c3cd5329261ac5	oidc:mock:groups:wheel	agent/other:unchanged
1788682768	e4b7ff0ca374863ec2d2462155978928	oidc:mock:groups:wheel	agent/other:unchanged
1788682764	863b961dd4e442f8b1e19b73c6592987	oidc:mock:groups:wheel	agent/other:unchanged
1788682761	3dbbed54b7d9639a680752b0826f2660	oidc:mock:repo:github:example/app:ref:refs/heads/main	agent/app:updated
1788682758	12d47d955f1de71c5aac940e24f6b47b	x509:email:admin@example.org	agent/app:updated

Container relay terminated by signal KILL.