niks3-go-unit-tests
checks.aarch64-linux.go-unit-tests
· build #149
· raw
1tribuchet: building on eliza2Running client tests...3=== RUN TestDoServerRequestAttachesToken4=== PAUSE TestDoServerRequestAttachesToken5=== RUN TestCaseHackSuffix6=== PAUSE TestCaseHackSuffix7=== RUN TestFilterOversizedClosures8=== PAUSE TestFilterOversizedClosures9=== RUN TestPartSizeForNAR10=== PAUSE TestPartSizeForNAR11=== RUN TestUploadMultipart_SupersededByPeer12=== PAUSE TestUploadMultipart_SupersededByPeer13=== RUN TestDumpPathMatchesNix14=== PAUSE TestDumpPathMatchesNix15=== RUN TestDumpPathSingleFile16=== PAUSE TestDumpPathSingleFile17=== RUN TestDumpPathWriterError18=== PAUSE TestDumpPathWriterError19=== RUN TestEncodeNixBase3220=== PAUSE TestEncodeNixBase3221=== RUN TestEncodeNixBase32WithRealHash22=== PAUSE TestEncodeNixBase32WithRealHash23=== RUN TestConvertHashToNix3224=== PAUSE TestConvertHashToNix3225=== RUN TestGetStorePathHash26=== PAUSE TestGetStorePathHash27=== RUN TestPathInfoHashCompatibility28=== PAUSE TestPathInfoHashCompatibility29=== RUN TestParsePathInfoJSON30=== PAUSE TestParsePathInfoJSON31=== RUN TestParsePathInfoJSONMultiplePaths32=== PAUSE TestParsePathInfoJSONMultiplePaths33=== RUN TestPathInfoCACompatibility34=== PAUSE TestPathInfoCACompatibility35=== RUN TestRateLimiterFeedback36=== PAUSE TestRateLimiterFeedback37=== RUN TestRateLimiterFeedback_400DoesNotCountAsSuccess38=== PAUSE TestRateLimiterFeedback_400DoesNotCountAsSuccess39=== RUN TestResolveStorePath40=== PAUSE TestResolveStorePath41=== RUN TestDoWithRetry_BodyReplayedViaGetBody42=== PAUSE TestDoWithRetry_BodyReplayedViaGetBody43=== RUN TestShellSplit44=== PAUSE TestShellSplit45=== RUN TestShellSplitErrors46=== PAUSE TestShellSplitErrors47=== RUN TestSetClientTLS48=== PAUSE TestSetClientTLS49=== RUN TestSetClientTLSDoesNotMutateDefaultTransport50=== PAUSE TestSetClientTLSDoesNotMutateDefaultTransport51=== RUN TestSetClientTLSErrors52=== PAUSE TestSetClientTLSErrors53=== RUN TestStaticToken54=== PAUSE TestStaticToken55=== RUN TestFileTokenReadsAndCaches56=== PAUSE TestFileTokenReadsAndCaches57=== RUN TestFileTokenMissing58=== PAUSE TestFileTokenMissing59=== RUN TestFileTokenEmpty60=== PAUSE TestFileTokenEmpty61=== RUN TestScriptTokenNoExpiryRerunsEveryCall62=== PAUSE TestScriptTokenNoExpiryRerunsEveryCall63=== RUN TestScriptTokenCachesUntilRefresh64=== PAUSE TestScriptTokenCachesUntilRefresh65=== RUN TestScriptTokenEmptyToken66=== PAUSE TestScriptTokenEmptyToken67=== RUN TestScriptTokenBadJSON68=== PAUSE TestScriptTokenBadJSON69=== RUN TestScriptTokenScriptFails70=== PAUSE TestScriptTokenScriptFails71=== RUN TestScriptTokenEmptyCommand72=== PAUSE TestScriptTokenEmptyCommand73=== CONT TestDoServerRequestAttachesToken74=== CONT TestGetStorePathHash75=== CONT TestDumpPathSingleFile76=== CONT TestScriptTokenNoExpiryRerunsEveryCall77=== CONT TestResolveStorePath78=== CONT TestEncodeNixBase32WithRealHash79--- PASS: TestEncodeNixBase32WithRealHash (0.00s)80=== CONT TestConvertHashToNix3281=== RUN TestConvertHashToNix32/SRI_format_to_Nix3282=== CONT TestEncodeNixBase3283=== RUN TestGetStorePathHash/valid_store_path84=== CONT TestDumpPathWriterError85=== PAUSE TestGetStorePathHash/valid_store_path86=== PAUSE TestConvertHashToNix32/SRI_format_to_Nix3287=== RUN TestConvertHashToNix32/already_Nix32_format88=== PAUSE TestConvertHashToNix32/already_Nix32_format89=== CONT TestFileTokenEmpty90=== RUN TestConvertHashToNix32/invalid_format91=== PAUSE TestConvertHashToNix32/invalid_format92=== CONT TestDumpPathMatchesNix93=== CONT TestFileTokenMissing94--- PASS: TestResolveStorePath (0.00s)95=== CONT TestUploadMultipart_SupersededByPeer96--- PASS: TestFileTokenEmpty (0.00s)97=== CONT TestFileTokenReadsAndCaches98--- PASS: TestFileTokenMissing (0.00s)99=== CONT TestCaseHackSuffix100=== RUN TestUploadMultipart_SupersededByPeer/exists101=== CONT TestStaticToken102=== PAUSE TestUploadMultipart_SupersededByPeer/exists103=== CONT TestParsePathInfoJSONMultiplePaths104=== RUN TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths105=== CONT TestSetClientTLSDoesNotMutateDefaultTransport106=== CONT TestScriptTokenBadJSON107=== CONT TestScriptTokenEmptyCommand108=== CONT TestRateLimiterFeedback_400DoesNotCountAsSuccess109=== CONT TestScriptTokenScriptFails110=== CONT TestRateLimiterFeedback111=== RUN TestRateLimiterFeedback/429_enables_limiter112=== PAUSE TestRateLimiterFeedback/429_enables_limiter113=== RUN TestRateLimiterFeedback/503_enables_limiter114=== PAUSE TestRateLimiterFeedback/503_enables_limiter115=== CONT TestScriptTokenEmptyToken116=== CONT TestScriptTokenCachesUntilRefresh117=== CONT TestSetClientTLS118=== CONT TestShellSplitErrors119=== CONT TestShellSplit120=== CONT TestDoWithRetry_BodyReplayedViaGetBody121=== CONT TestPartSizeForNAR122=== RUN TestEncodeNixBase32/test_string_hash123=== RUN TestGetStorePathHash/basename_without_hyphen_should_error124=== CONT TestFilterOversizedClosures125--- PASS: TestFileTokenReadsAndCaches (0.00s)126=== CONT TestSetClientTLSErrors127=== CONT TestParsePathInfoJSON1282026/08/27 09:46:00 WARN Rate limiter enabled after throttle name=server-test rate=5129--- PASS: TestStaticToken (0.00s)130--- PASS: TestScriptTokenEmptyCommand (0.00s)131--- PASS: TestDoServerRequestAttachesToken (0.01s)132--- PASS: TestShellSplit (0.00s)133--- PASS: TestShellSplitErrors (0.00s)134=== PAUSE TestGetStorePathHash/basename_without_hyphen_should_error135=== PAUSE TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths136=== CONT TestPathInfoHashCompatibility137=== RUN TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths138=== RUN TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)139=== RUN TestPartSizeForNAR/zero_stays_at_minimum140=== PAUSE TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths141=== CONT TestPathInfoCACompatibility142=== PAUSE TestPartSizeForNAR/zero_stays_at_minimum143=== RUN TestPartSizeForNAR/small_stays_at_minimum144=== PAUSE TestPartSizeForNAR/small_stays_at_minimum145=== RUN TestPartSizeForNAR/80_GiB_fits_at_minimum146=== PAUSE TestPartSizeForNAR/80_GiB_fits_at_minimum147=== RUN TestPartSizeForNAR/115_GiB_needs_larger_parts148=== PAUSE TestPartSizeForNAR/115_GiB_needs_larger_parts149=== RUN TestPartSizeForNAR/1_TiB150=== PAUSE TestPartSizeForNAR/1_TiB151=== RUN TestParsePathInfoJSON/Nix_format152=== PAUSE TestParsePathInfoJSON/Nix_format153=== RUN TestFilterOversizedClosures/no_limit_keeps_everything154=== PAUSE TestFilterOversizedClosures/no_limit_keeps_everything155=== RUN TestRateLimiterFeedback/200_does_not_enable_limiter156=== RUN TestParsePathInfoJSON/Lix_format157=== PAUSE TestRateLimiterFeedback/200_does_not_enable_limiter158=== RUN TestGetStorePathHash/hash_with_invalid_characters_should_error159=== PAUSE TestEncodeNixBase32/test_string_hash160=== RUN TestEncodeNixBase32/empty_input161=== RUN TestUploadMultipart_SupersededByPeer/missing162=== RUN TestPartSizeForNAR/5_TiB_S3_max_object163=== RUN TestPathInfoCACompatibility/null_ca_field164=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)165--- PASS: TestScriptTokenScriptFails (0.01s)166=== RUN TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped167=== PAUSE TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped168=== CONT TestConvertHashToNix32/already_Nix32_format169=== PAUSE TestParsePathInfoJSON/Lix_format1702026/08/27 09:46:00 WARN Rate limiter enabled after throttle name=server-test rate=5171=== RUN TestPathInfoHashCompatibility/old_string_format_with_colon1722026/08/27 09:46:00 WARN Request returned retryable status, retrying attempt=1 max_attempts=6 backoff=0s status=503 url=http://127.0.0.1:36457173=== RUN TestParsePathInfoJSON/empty_input174=== PAUSE TestParsePathInfoJSON/empty_input175=== RUN TestParsePathInfoJSON/whitespace_only176=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_colon1772026/08/27 09:46:00 WARN Rate limiter backed off name=server-test rate=51782026/08/27 09:46:00 WARN Request returned retryable status, retrying attempt=2 max_attempts=6 backoff=0s status=503 url=http://127.0.0.1:36457179=== CONT TestConvertHashToNix32/SRI_format_to_Nix32180=== PAUSE TestEncodeNixBase32/empty_input181=== PAUSE TestUploadMultipart_SupersededByPeer/missing182=== CONT TestUploadMultipart_SupersededByPeer/exists183=== PAUSE TestPartSizeForNAR/5_TiB_S3_max_object184=== RUN TestFilterOversizedClosures/all_closures_skipped185=== CONT TestConvertHashToNix32/invalid_format186=== RUN TestRateLimiterFeedback/400_does_not_enable_limiter187=== PAUSE TestGetStorePathHash/hash_with_invalid_characters_should_error188=== PAUSE TestParsePathInfoJSON/whitespace_only189=== PAUSE TestPathInfoCACompatibility/null_ca_field190=== RUN TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI191--- PASS: TestScriptTokenEmptyToken (0.01s)192=== RUN TestParsePathInfoJSON/invalid_JSON193=== PAUSE TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI194=== CONT TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths195=== CONT TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths196=== CONT TestEncodeNixBase32/test_string_hash197=== CONT TestEncodeNixBase32/empty_input198=== CONT TestUploadMultipart_SupersededByPeer/missing199=== RUN TestPartSizeForNAR/capped_at_5_GiB200=== PAUSE TestFilterOversizedClosures/all_closures_skipped201=== PAUSE TestPartSizeForNAR/capped_at_5_GiB202=== PAUSE TestRateLimiterFeedback/400_does_not_enable_limiter203=== RUN TestSetClientTLSErrors/missing_cert_file204=== RUN TestGetStorePathHash/hash_with_wrong_length_should_error205=== PAUSE TestSetClientTLSErrors/missing_cert_file206=== RUN TestSetClientTLSErrors/missing_key_file207=== PAUSE TestGetStorePathHash/hash_with_wrong_length_should_error208=== RUN TestPathInfoCACompatibility/old_string_format_-_text209=== PAUSE TestParsePathInfoJSON/invalid_JSON210=== RUN TestPathInfoHashCompatibility/new_structured_format_with_sha512211=== PAUSE TestPathInfoHashCompatibility/new_structured_format_with_sha512212=== CONT TestFilterOversizedClosures/no_limit_keeps_everything213=== CONT TestRateLimiterFeedback/503_enables_limiter214=== CONT TestRateLimiterFeedback/200_does_not_enable_limiter215=== CONT TestPartSizeForNAR/5_TiB_S3_max_object216=== RUN TestSetClientTLS/rejects_connection_without_client_cert217=== CONT TestPartSizeForNAR/small_stays_at_minimum218=== CONT TestFilterOversizedClosures/all_closures_skipped219=== CONT TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped220=== CONT TestPartSizeForNAR/1_TiB221=== CONT TestPartSizeForNAR/zero_stays_at_minimum222=== CONT TestPartSizeForNAR/115_GiB_needs_larger_parts223--- PASS: TestScriptTokenBadJSON (0.01s)224=== PAUSE TestSetClientTLSErrors/missing_key_file225=== CONT TestPartSizeForNAR/80_GiB_fits_at_minimum226=== CONT TestRateLimiterFeedback/429_enables_limiter227=== CONT TestRateLimiterFeedback/400_does_not_enable_limiter228=== PAUSE TestPathInfoCACompatibility/old_string_format_-_text229=== RUN TestPathInfoCACompatibility/old_string_format_-_fixed_recursive2302026/08/27 09:46:00 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper nar_size=100 max_nar_size=50231=== PAUSE TestPathInfoCACompatibility/old_string_format_-_fixed_recursive232=== RUN TestPathInfoCACompatibility/new_structured_format_-_text233=== CONT TestPartSizeForNAR/capped_at_5_GiB2342026/08/27 09:46:00 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb-vm-image nar_size=5000 max_nar_size=2000235=== CONT TestGetStorePathHash/valid_store_path2362026/08/27 09:46:00 WARN Rate limiter enabled after throttle name=server-test rate=5237=== CONT TestPathInfoHashCompatibility/new_structured_format_with_sha5122382026/08/27 09:46:00 WARN Request returned retryable status, retrying attempt=1 max_attempts=2 backoff=0s status=503 url=http://127.0.0.1:33355239=== CONT TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI240=== CONT TestPathInfoHashCompatibility/old_string_format_with_colon241=== CONT TestGetStorePathHash/hash_with_wrong_length_should_error242=== PAUSE TestSetClientTLS/rejects_connection_without_client_cert243=== RUN TestSetClientTLS/succeeds_with_client_cert_and_CA244--- PASS: TestScriptTokenNoExpiryRerunsEveryCall (0.02s)2452026/08/27 09:46:00 WARN Rate limiter enabled after throttle name=server-test rate=5246=== CONT TestParsePathInfoJSON/Nix_format2472026/08/27 09:46:00 WARN Request returned retryable status, retrying attempt=1 max_attempts=2 backoff=0s status=429 url=http://127.0.0.1:395352482026/08/27 09:46:00 WARN Rate limiter backed off name=server-test rate=5249=== CONT TestGetStorePathHash/basename_without_hyphen_should_error250=== RUN TestSetClientTLSErrors/missing_ca_file2512026/08/27 09:46:00 WARN Rate limiter backed off name=server-test rate=5252=== PAUSE TestSetClientTLSErrors/missing_ca_file253=== CONT TestParsePathInfoJSON/invalid_JSON254=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_text255=== RUN TestPathInfoCACompatibility/new_structured_format_-_nar_method256=== CONT TestParsePathInfoJSON/whitespace_only257=== CONT TestParsePathInfoJSON/empty_input258=== CONT TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)259=== CONT TestParsePathInfoJSON/Lix_format260=== CONT TestGetStorePathHash/hash_with_invalid_characters_should_error261=== PAUSE TestSetClientTLS/succeeds_with_client_cert_and_CA262--- PASS: TestSetClientTLSDoesNotMutateDefaultTransport (0.01s)263--- PASS: TestConvertHashToNix32 (0.00s)264 --- PASS: TestConvertHashToNix32/already_Nix32_format (0.00s)265 --- PASS: TestConvertHashToNix32/SRI_format_to_Nix32 (0.00s)266 --- PASS: TestConvertHashToNix32/invalid_format (0.00s)267--- PASS: TestDoWithRetry_BodyReplayedViaGetBody (0.01s)268--- PASS: TestParsePathInfoJSONMultiplePaths (0.01s)269 --- PASS: TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths (0.00s)270 --- PASS: TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths (0.00s)271=== RUN TestSetClientTLSErrors/invalid_ca_file272=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_nar_method273=== RUN TestSetClientTLS/preserves_debug_logging_transport274--- PASS: TestFilterOversizedClosures (0.01s)275 --- PASS: TestFilterOversizedClosures/no_limit_keeps_everything (0.00s)276 --- PASS: TestFilterOversizedClosures/all_closures_skipped (0.00s)277 --- PASS: TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped (0.00s)278=== PAUSE TestSetClientTLS/preserves_debug_logging_transport279=== CONT TestPathInfoCACompatibility/null_ca_field280=== CONT TestPathInfoCACompatibility/new_structured_format_-_nar_method281=== CONT TestPathInfoCACompatibility/new_structured_format_-_text282=== CONT TestPathInfoCACompatibility/old_string_format_-_fixed_recursive283=== CONT TestPathInfoCACompatibility/old_string_format_-_text284=== PAUSE TestSetClientTLSErrors/invalid_ca_file285=== CONT TestSetClientTLSErrors/missing_cert_file286=== CONT TestSetClientTLSErrors/invalid_ca_file287=== CONT TestSetClientTLSErrors/missing_key_file288=== CONT TestSetClientTLS/preserves_debug_logging_transport289=== CONT TestSetClientTLS/succeeds_with_client_cert_and_CA290--- PASS: TestScriptTokenCachesUntilRefresh (0.02s)291=== CONT TestSetClientTLSErrors/missing_ca_file292=== CONT TestSetClientTLS/rejects_connection_without_client_cert293--- PASS: TestRateLimiterFeedback (0.02s)294 --- PASS: TestRateLimiterFeedback/200_does_not_enable_limiter (0.00s)295 --- PASS: TestRateLimiterFeedback/400_does_not_enable_limiter (0.00s)296 --- PASS: TestRateLimiterFeedback/503_enables_limiter (0.00s)297 --- PASS: TestRateLimiterFeedback/429_enables_limiter (0.00s)298--- PASS: TestEncodeNixBase32 (0.02s)299 --- PASS: TestEncodeNixBase32/test_string_hash (0.00s)300 --- PASS: TestEncodeNixBase32/empty_input (0.00s)301--- PASS: TestGetStorePathHash (0.02s)302 --- PASS: TestGetStorePathHash/valid_store_path (0.00s)303 --- PASS: TestGetStorePathHash/hash_with_wrong_length_should_error (0.00s)304 --- PASS: TestGetStorePathHash/basename_without_hyphen_should_error (0.00s)305 --- PASS: TestGetStorePathHash/hash_with_invalid_characters_should_error (0.00s)306--- PASS: TestUploadMultipart_SupersededByPeer (0.02s)307 --- PASS: TestUploadMultipart_SupersededByPeer/exists (0.00s)308 --- PASS: TestUploadMultipart_SupersededByPeer/missing (0.00s)309--- PASS: TestPathInfoCACompatibility (0.01s)310 --- PASS: TestPathInfoCACompatibility/null_ca_field (0.00s)311 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_nar_method (0.00s)312 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_text (0.00s)313 --- PASS: TestPathInfoCACompatibility/old_string_format_-_fixed_recursive (0.00s)314 --- PASS: TestPathInfoCACompatibility/old_string_format_-_text (0.00s)315--- PASS: TestParsePathInfoJSON (0.01s)316 --- PASS: TestParsePathInfoJSON/Nix_format (0.00s)317 --- PASS: TestParsePathInfoJSON/invalid_JSON (0.00s)318 --- PASS: TestParsePathInfoJSON/whitespace_only (0.00s)319 --- PASS: TestParsePathInfoJSON/empty_input (0.00s)320 --- PASS: TestParsePathInfoJSON/Lix_format (0.00s)321--- PASS: TestPathInfoHashCompatibility (0.01s)322 --- PASS: TestPathInfoHashCompatibility/new_structured_format_with_sha512 (0.00s)323 --- PASS: TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI (0.00s)324 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_colon (0.00s)325 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI) (0.00s)326--- PASS: TestPartSizeForNAR (0.01s)327 --- PASS: TestPartSizeForNAR/5_TiB_S3_max_object (0.00s)328 --- PASS: TestPartSizeForNAR/small_stays_at_minimum (0.00s)329 --- PASS: TestPartSizeForNAR/1_TiB (0.00s)330 --- PASS: TestPartSizeForNAR/zero_stays_at_minimum (0.00s)331 --- PASS: TestPartSizeForNAR/115_GiB_needs_larger_parts (0.00s)332 --- PASS: TestPartSizeForNAR/80_GiB_fits_at_minimum (0.00s)333 --- PASS: TestPartSizeForNAR/capped_at_5_GiB (0.00s)334--- PASS: TestSetClientTLSErrors (0.01s)335 --- PASS: TestSetClientTLSErrors/missing_cert_file (0.00s)336 --- PASS: TestSetClientTLSErrors/missing_key_file (0.00s)337 --- PASS: TestSetClientTLSErrors/invalid_ca_file (0.00s)338 --- PASS: TestSetClientTLSErrors/missing_ca_file (0.00s)339--- PASS: TestDumpPathSingleFile (0.03s)340--- PASS: TestCaseHackSuffix (0.03s)3412026/08/27 09:46:00 http: TLS handshake error from 127.0.0.1:57264: remote error: tls: bad certificate342--- PASS: TestSetClientTLS (0.01s)343 --- PASS: TestSetClientTLS/succeeds_with_client_cert_and_CA (0.00s)344 --- PASS: TestSetClientTLS/preserves_debug_logging_transport (0.00s)345 --- PASS: TestSetClientTLS/rejects_connection_without_client_cert (0.01s)346--- PASS: TestDumpPathWriterError (0.05s)347--- PASS: TestDumpPathMatchesNix (0.08s)348--- PASS: TestRateLimiterFeedback_400DoesNotCountAsSuccess (1.00s)349PASS350Running server tests...351The files belonging to this database system will be owned by user "nixbld".352This user must also own the server process.353354The database cluster will be initialized with locale "C".355The default database encoding has accordingly been set to "SQL_ASCII".356The default text search configuration will be set to "english".357358Data page checksums are enabled.359360creating directory /build/postgres1121113842/data ... ok361creating subdirectories ... ok362selecting dynamic shared memory implementation ... posix363selecting default "max_connections" ... 100364selecting default "shared_buffers" ... 128MB365selecting default time zone ... UTC366creating configuration files ... ok367running bootstrap script ... ok368performing post-bootstrap initialization ... ok369syncing data to disk ... ok370371initdb: warning: enabling "trust" authentication for local connections372initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.373374Success. You can now start the database server using:375376 pg_ctl -D /build/postgres1121113842/data -l logfile start377378/build/postgres1121113842:5432 - no response3792026-08-27 09:46:02.183 UTC [110] LOG: starting PostgreSQL 18.4 on aarch64-unknown-linux-gnu, compiled by clang version 21.1.8, 64-bit3802026-08-27 09:46:02.183 UTC [110] LOG: listening on Unix socket "/build/postgres1121113842/.s.PGSQL.5432"3812026-08-27 09:46:02.190 UTC [117] LOG: database system was shut down at 2026-08-27 09:46:01 UTC3822026-08-27 09:46:02.195 UTC [110] LOG: database system is ready to accept connections383/build/postgres1121113842:5432 - accepting connections384=== RUN TestService_AuthMiddleware385=== PAUSE TestService_AuthMiddleware386=== RUN TestService_AuthMiddleware_MTLSProxyHeader387=== PAUSE TestService_AuthMiddleware_MTLSProxyHeader388=== RUN TestService_AuthMiddleware_MTLSBoundSubjects389=== PAUSE TestService_AuthMiddleware_MTLSBoundSubjects390=== RUN TestService_ReadAuthMiddleware391=== PAUSE TestService_ReadAuthMiddleware392=== RUN TestService_AuthMiddleware_OIDC393=== PAUSE TestService_AuthMiddleware_OIDC394=== RUN TestCacheConfigHandler395=== PAUSE TestCacheConfigHandler396=== RUN TestCacheStatsHandler397=== PAUSE TestCacheStatsHandler398=== RUN TestClientCADerivations399=== PAUSE TestClientCADerivations400=== RUN TestClientErrorHandling401=== PAUSE TestClientErrorHandling402=== RUN TestClientIntegration403=== PAUSE TestClientIntegration404=== RUN TestClientMultipleUploads405=== PAUSE TestClientMultipleUploads406=== RUN TestClientWithDependencies407=== PAUSE TestClientWithDependencies408=== RUN TestPinProtectsFromGC409=== PAUSE TestPinProtectsFromGC410=== RUN TestGCAdvisoryLockBlocksConcurrentRun4112026-08-27 09:46:05.796 UTC [520] ERROR: relation "goose_db_version" does not exist at character 364122026-08-27 09:46:05.796 UTC [520] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC4132026/08/27 09:46:05 OK 20241026095416_initial_model.sql (10.75ms)4142026/08/27 09:46:05 OK 20251210153512_drop_unused_gin_index.sql (1.91ms)4152026/08/27 09:46:05 OK 20251218171726_add_pins.sql (2.73ms)4162026/08/27 09:46:05 OK 20260628120000_add_object_size_and_stats.sql (2.52ms)4172026/08/27 09:46:05 goose: successfully migrated database to version: 202606281200004182026/08/27 09:46:05 OK 1_commit_pending_closure.sql (1.64ms)4192026/08/27 09:46:05 OK 2_object_stats_trigger.sql (704.99µs)4202026/08/27 09:46:05 goose: up to current file version: 2421--- PASS: TestGCAdvisoryLockBlocksConcurrentRun (0.67s)422=== RUN TestGCBugBareHashReferences423=== PAUSE TestGCBugBareHashReferences424=== RUN TestGCMetrics425=== PAUSE TestGCMetrics426=== RUN TestGCTaskStore_StartNew427=== PAUSE TestGCTaskStore_StartNew428=== RUN TestGCTaskStore_DeduplicateSameParams429=== PAUSE TestGCTaskStore_DeduplicateSameParams430=== RUN TestGCTaskStore_ConflictDifferentParams431=== PAUSE TestGCTaskStore_ConflictDifferentParams432=== RUN TestGCTaskStore_GetEmpty433=== PAUSE TestGCTaskStore_GetEmpty434=== RUN TestGCTaskStore_GetReturnsLatest435=== PAUSE TestGCTaskStore_GetReturnsLatest436=== RUN TestGCTaskStore_CompletedAllowsNewTask437=== PAUSE TestGCTaskStore_CompletedAllowsNewTask438=== RUN TestGCTaskStore_PhaseUpdates439=== PAUSE TestGCTaskStore_PhaseUpdates440=== RUN TestGCTaskStore_Fail441=== PAUSE TestGCTaskStore_Fail442=== RUN TestGracefulShutdownDrainsInflight443=== PAUSE TestGracefulShutdownDrainsInflight444=== RUN TestService_healthCheckHandler445=== PAUSE TestService_healthCheckHandler446=== RUN TestGenerateLandingPage447=== PAUSE TestGenerateLandingPage448=== RUN TestCacheConfigHandlerMaxNarSize449=== PAUSE TestCacheConfigHandlerMaxNarSize450=== RUN TestCreatePendingClosureRejectsOversizedNAR451=== PAUSE TestCreatePendingClosureRejectsOversizedNAR452=== RUN TestNARDeduplicationMetadataUploadBug453=== PAUSE TestNARDeduplicationMetadataUploadBug454=== RUN TestMetricsInventory455=== PAUSE TestMetricsInventory456=== RUN TestService_NativeMTLS457=== PAUSE TestService_NativeMTLS458=== RUN TestServerTLSConfig459=== PAUSE TestServerTLSConfig460=== RUN TestMultipartCleanup461=== PAUSE TestMultipartCleanup462=== RUN TestObjectStatsTrigger463=== PAUSE TestObjectStatsTrigger464=== RUN TestOrphanedObjectsGC465=== PAUSE TestOrphanedObjectsGC466=== RUN TestOrphanedObjectsGCStressTest467=== PAUSE TestOrphanedObjectsGCStressTest468=== RUN TestResurrectedObjectNotDeleted469=== PAUSE TestResurrectedObjectNotDeleted470=== RUN TestParseSingleRange471=== PAUSE TestParseSingleRange472=== RUN TestIsValidCachePath473=== PAUSE TestIsValidCachePath474=== RUN TestReadProxyNarinfo475=== PAUSE TestReadProxyNarinfo476=== RUN TestReadProxyNarinfoAlreadyDecompressed477=== PAUSE TestReadProxyNarinfoAlreadyDecompressed478=== RUN TestReadProxyNarStreaming479=== PAUSE TestReadProxyNarStreaming480=== RUN TestReadProxy404481=== PAUSE TestReadProxy404482=== RUN TestReadProxyInvalidPath483=== PAUSE TestReadProxyInvalidPath484=== RUN TestReadProxyHead485=== PAUSE TestReadProxyHead486=== RUN TestReadProxyConditionalGet487=== PAUSE TestReadProxyConditionalGet488=== RUN TestReadProxyRootRedirectsToIndexHTML489=== PAUSE TestReadProxyRootRedirectsToIndexHTML490=== RUN TestReadProxyDisabled491=== PAUSE TestReadProxyDisabled492=== RUN TestReadRedirectNar493=== PAUSE TestReadRedirectNar494=== RUN TestReadRedirectKeepsNarinfoProxied495=== PAUSE TestReadRedirectKeepsNarinfoProxied496=== RUN TestReadProxyRangeRequest497=== PAUSE TestReadProxyRangeRequest498=== RUN TestRedundantMultipartUpload499=== PAUSE TestRedundantMultipartUpload500=== RUN TestCompleteMultipartUpload_ErrorButObjectExists501=== PAUSE TestCompleteMultipartUpload_ErrorButObjectExists502=== RUN TestCompletedNarNotReofferedAcrossClosures503=== PAUSE TestCompletedNarNotReofferedAcrossClosures504=== RUN TestPresignedUploadRegisteredBeforeCommit505=== PAUSE TestPresignedUploadRegisteredBeforeCommit506=== RUN TestService_Rustfstest507=== PAUSE TestService_Rustfstest508=== RUN TestParseSize509=== PAUSE TestParseSize510=== RUN TestSkippedUploadsHandler511=== PAUSE TestSkippedUploadsHandler512=== RUN TestSystemdListenerNotActivated513--- PASS: TestSystemdListenerNotActivated (0.00s)514=== RUN TestWatchdogBeatsWhenHealthy515--- PASS: TestWatchdogBeatsWhenHealthy (0.02s)516=== RUN TestWatchdogSkipsWhenUnhealthy5172026/08/27 09:46:06 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"5182026/08/27 09:46:06 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"5192026/08/27 09:46:06 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"5202026/08/27 09:46:06 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"5212026/08/27 09:46:06 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"5222026/08/27 09:46:06 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"5232026/08/27 09:46:06 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"5242026/08/27 09:46:06 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"5252026/08/27 09:46:06 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"526--- PASS: TestWatchdogSkipsWhenUnhealthy (0.20s)527=== RUN TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle528=== PAUSE TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle529=== RUN TestProxyWriteTimeout530=== PAUSE TestProxyWriteTimeout531=== RUN TestIsValidUploadKey532=== PAUSE TestIsValidUploadKey533=== RUN TestUploadHandlersRejectInvalidKeys534=== PAUSE TestUploadHandlersRejectInvalidKeys535=== RUN TestUploadHandlersRejectOversizedBody536=== PAUSE TestUploadHandlersRejectOversizedBody537=== RUN TestService_cleanupPendingClosuresHandler538=== PAUSE TestService_cleanupPendingClosuresHandler539=== RUN TestService_createPendingClosureHandler540=== PAUSE TestService_createPendingClosureHandler541=== RUN TestService_verifyS3Integrity542=== PAUSE TestService_verifyS3Integrity543=== RUN TestCompleteMultipartUnregistered544=== PAUSE TestCompleteMultipartUnregistered545=== RUN TestCreatePendingClosure_SmallNARUsesSimplePUT546=== PAUSE TestCreatePendingClosure_SmallNARUsesSimplePUT547=== CONT TestSkippedUploadsHandler548=== CONT TestService_AuthMiddleware549=== CONT TestParseSize550=== CONT TestService_Rustfstest551=== CONT TestPresignedUploadRegisteredBeforeCommit552=== CONT TestCompletedNarNotReofferedAcrossClosures553=== CONT TestCompleteMultipartUpload_ErrorButObjectExists5542026/08/27 09:46:06 INFO Client skipped oversized paths paths=3 nar_bytes=5000000000555=== CONT TestRedundantMultipartUpload556=== CONT TestReadProxyRangeRequest557=== CONT TestReadRedirectKeepsNarinfoProxied558=== CONT TestReadRedirectNar559=== CONT TestReadProxyDisabled560=== CONT TestReadProxyRootRedirectsToIndexHTML561=== CONT TestService_cleanupPendingClosuresHandler562=== CONT TestCreatePendingClosure_SmallNARUsesSimplePUT563=== CONT TestReadProxyConditionalGet564=== CONT TestCompleteMultipartUnregistered565=== CONT TestReadProxyHead566=== CONT TestService_verifyS3Integrity567=== CONT TestReadProxyInvalidPath568=== CONT TestService_createPendingClosureHandler569=== CONT TestReadProxy404570=== CONT TestReadProxyNarStreaming571=== CONT TestIsValidUploadKey572--- PASS: TestParseSize (0.00s)573=== CONT TestReadProxyNarinfoAlreadyDecompressed574--- PASS: TestSkippedUploadsHandler (0.01s)575=== RUN TestIsValidUploadKey/narinfo576=== PAUSE TestIsValidUploadKey/narinfo577=== CONT TestUploadHandlersRejectOversizedBody578=== RUN TestIsValidUploadKey/nar_zst579=== PAUSE TestIsValidUploadKey/nar_zst580=== RUN TestIsValidUploadKey/nar_xz581=== PAUSE TestIsValidUploadKey/nar_xz582=== RUN TestIsValidUploadKey/nar_plain583=== PAUSE TestIsValidUploadKey/nar_plain584=== RUN TestIsValidUploadKey/listing585=== PAUSE TestIsValidUploadKey/listing586=== RUN TestIsValidUploadKey/build_log587=== PAUSE TestIsValidUploadKey/build_log588=== RUN TestIsValidUploadKey/build_log_home-manager_file589=== PAUSE TestIsValidUploadKey/build_log_home-manager_file590=== RUN TestIsValidUploadKey/build_log_plus_in_name591=== PAUSE TestIsValidUploadKey/build_log_plus_in_name592=== RUN TestIsValidUploadKey/build_log_question_mark593=== PAUSE TestIsValidUploadKey/build_log_question_mark594=== RUN TestIsValidUploadKey/build_log_equals595=== PAUSE TestIsValidUploadKey/build_log_equals596=== RUN TestIsValidUploadKey/realisation597=== PAUSE TestIsValidUploadKey/realisation598=== RUN TestIsValidUploadKey/realisation_plus_in_output599=== PAUSE TestIsValidUploadKey/realisation_plus_in_output600=== RUN TestIsValidUploadKey/nix-cache-info601=== PAUSE TestIsValidUploadKey/nix-cache-info602=== RUN TestIsValidUploadKey/index.html603=== PAUSE TestIsValidUploadKey/index.html604=== RUN TestIsValidUploadKey/narinfo_key,_nar_type605=== PAUSE TestIsValidUploadKey/narinfo_key,_nar_type606=== RUN TestIsValidUploadKey/nar_key,_narinfo_type607=== PAUSE TestIsValidUploadKey/nar_key,_narinfo_type608=== RUN TestIsValidUploadKey/listing_key,_narinfo_type609=== PAUSE TestIsValidUploadKey/listing_key,_narinfo_type610=== RUN TestIsValidUploadKey/traversal611=== PAUSE TestIsValidUploadKey/traversal612=== RUN TestIsValidUploadKey/traversal_nar613=== PAUSE TestIsValidUploadKey/traversal_nar614=== RUN TestIsValidUploadKey/absolute615=== PAUSE TestIsValidUploadKey/absolute616=== RUN TestIsValidUploadKey/empty_key617=== PAUSE TestIsValidUploadKey/empty_key618=== RUN TestIsValidUploadKey/unknown_type619=== PAUSE TestIsValidUploadKey/unknown_type620=== CONT TestReadProxyNarinfo6212026-08-27 09:46:06.712 UTC [596] ERROR: relation "goose_db_version" does not exist at character 366222026-08-27 09:46:06.712 UTC [596] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC6232026-08-27 09:46:06.714 UTC [597] ERROR: relation "goose_db_version" does not exist at character 366242026-08-27 09:46:06.714 UTC [597] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC6252026-08-27 09:46:06.737 UTC [598] ERROR: relation "goose_db_version" does not exist at character 366262026-08-27 09:46:06.737 UTC [598] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC627=== RUN TestUploadHandlersRejectOversizedBody/create_pending_closure628=== PAUSE TestUploadHandlersRejectOversizedBody/create_pending_closure629=== RUN TestUploadHandlersRejectOversizedBody/complete_multipart630=== PAUSE TestUploadHandlersRejectOversizedBody/complete_multipart631=== RUN TestUploadHandlersRejectOversizedBody/request_more_parts632=== PAUSE TestUploadHandlersRejectOversizedBody/request_more_parts633=== CONT TestIsValidCachePath634=== RUN TestIsValidCachePath/narinfo635=== PAUSE TestIsValidCachePath/narinfo636=== RUN TestIsValidCachePath/narinfo_all_nix_base32_chars637=== PAUSE TestIsValidCachePath/narinfo_all_nix_base32_chars638=== RUN TestIsValidCachePath/nar_zst639=== PAUSE TestIsValidCachePath/nar_zst640=== RUN TestIsValidCachePath/nar_xz641=== PAUSE TestIsValidCachePath/nar_xz642=== RUN TestIsValidCachePath/nar_bz2643=== PAUSE TestIsValidCachePath/nar_bz2644=== RUN TestIsValidCachePath/nar_uncompressed645=== PAUSE TestIsValidCachePath/nar_uncompressed646=== RUN TestIsValidCachePath/ls647=== PAUSE TestIsValidCachePath/ls648=== RUN TestIsValidCachePath/log649=== PAUSE TestIsValidCachePath/log650=== RUN TestIsValidCachePath/realisation651=== PAUSE TestIsValidCachePath/realisation652=== RUN TestIsValidCachePath/nix-cache-info653=== PAUSE TestIsValidCachePath/nix-cache-info654=== RUN TestIsValidCachePath/index.html655=== PAUSE TestIsValidCachePath/index.html656=== RUN TestIsValidCachePath/traversal_parent657=== PAUSE TestIsValidCachePath/traversal_parent658=== RUN TestIsValidCachePath/traversal_in_middle659=== PAUSE TestIsValidCachePath/traversal_in_middle660=== RUN TestIsValidCachePath/invalid_char_e661=== PAUSE TestIsValidCachePath/invalid_char_e662=== RUN TestIsValidCachePath/invalid_char_u663=== PAUSE TestIsValidCachePath/invalid_char_u664=== RUN TestIsValidCachePath/random_path665=== PAUSE TestIsValidCachePath/random_path666=== RUN TestIsValidCachePath/empty667=== PAUSE TestIsValidCachePath/empty668=== RUN TestIsValidCachePath/leading_slash669=== PAUSE TestIsValidCachePath/leading_slash670=== RUN TestIsValidCachePath/wrong_extension671=== PAUSE TestIsValidCachePath/wrong_extension672=== RUN TestIsValidCachePath/short_hash673=== PAUSE TestIsValidCachePath/short_hash674=== CONT TestProxyWriteTimeout675=== RUN TestProxyWriteTimeout/narinfo6762026-08-27 09:46:06.750 UTC [599] ERROR: relation "goose_db_version" does not exist at character 366772026-08-27 09:46:06.750 UTC [599] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC678=== PAUSE TestProxyWriteTimeout/narinfo679=== RUN TestProxyWriteTimeout/1_GiB_nar680=== PAUSE TestProxyWriteTimeout/1_GiB_nar681=== RUN TestProxyWriteTimeout/10_GiB_nar682=== PAUSE TestProxyWriteTimeout/10_GiB_nar683=== RUN TestProxyWriteTimeout/unknown_size684=== PAUSE TestProxyWriteTimeout/unknown_size685=== CONT TestParseSingleRange686=== RUN TestParseSingleRange/none687=== PAUSE TestParseSingleRange/none688=== RUN TestParseSingleRange/unknown_unit689=== PAUSE TestParseSingleRange/unknown_unit690=== RUN TestParseSingleRange/multi-range_ignored691=== PAUSE TestParseSingleRange/multi-range_ignored692=== RUN TestParseSingleRange/malformed_no_dash693=== PAUSE TestParseSingleRange/malformed_no_dash694=== RUN TestParseSingleRange/malformed_both_empty695=== PAUSE TestParseSingleRange/malformed_both_empty696=== RUN TestParseSingleRange/malformed_end_before_start697=== PAUSE TestParseSingleRange/malformed_end_before_start698=== RUN TestParseSingleRange/closed699=== PAUSE TestParseSingleRange/closed700=== RUN TestParseSingleRange/open-ended701=== PAUSE TestParseSingleRange/open-ended702=== RUN TestParseSingleRange/end_clamped_to_size703=== PAUSE TestParseSingleRange/end_clamped_to_size704=== RUN TestParseSingleRange/suffix705=== PAUSE TestParseSingleRange/suffix706=== RUN TestParseSingleRange/suffix_exceeds_size707=== PAUSE TestParseSingleRange/suffix_exceeds_size708=== RUN TestParseSingleRange/single_byte709=== PAUSE TestParseSingleRange/single_byte710=== RUN TestParseSingleRange/start_past_EOF711=== PAUSE TestParseSingleRange/start_past_EOF712=== RUN TestParseSingleRange/start_far_past_EOF713=== PAUSE TestParseSingleRange/start_far_past_EOF714=== CONT TestResurrectedObjectNotDeleted7152026-08-27 09:46:06.762 UTC [601] ERROR: relation "goose_db_version" does not exist at character 367162026-08-27 09:46:06.762 UTC [601] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC7172026-08-27 09:46:06.767 UTC [603] ERROR: relation "goose_db_version" does not exist at character 367182026-08-27 09:46:06.767 UTC [603] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC7192026/08/27 09:46:06 OK 20241026095416_initial_model.sql (15.68ms)7202026-08-27 09:46:06.781 UTC [604] ERROR: relation "goose_db_version" does not exist at character 367212026-08-27 09:46:06.781 UTC [604] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC7222026/08/27 09:46:06 OK 20241026095416_initial_model.sql (44.09ms)7232026/08/27 09:46:06 OK 20241026095416_initial_model.sql (39.45ms)7242026/08/27 09:46:06 OK 20241026095416_initial_model.sql (37.91ms)7252026/08/27 09:46:06 OK 20251210153512_drop_unused_gin_index.sql (11.99ms)7262026/08/27 09:46:06 OK 20251210153512_drop_unused_gin_index.sql (4.65ms)7272026/08/27 09:46:06 OK 20251210153512_drop_unused_gin_index.sql (3.57ms)7282026/08/27 09:46:06 OK 20251210153512_drop_unused_gin_index.sql (5.92ms)7292026/08/27 09:46:06 OK 20241026095416_initial_model.sql (28.07ms)7302026/08/27 09:46:06 OK 20251218171726_add_pins.sql (7.93ms)7312026/08/27 09:46:06 OK 20251210153512_drop_unused_gin_index.sql (3.13ms)7322026/08/27 09:46:06 OK 20251218171726_add_pins.sql (9.55ms)7332026/08/27 09:46:06 OK 20241026095416_initial_model.sql (30.63ms)7342026/08/27 09:46:06 OK 20251218171726_add_pins.sql (8.35ms)7352026/08/27 09:46:06 OK 20260628120000_add_object_size_and_stats.sql (5.54ms)7362026/08/27 09:46:06 goose: successfully migrated database to version: 202606281200007372026/08/27 09:46:06 OK 20251210153512_drop_unused_gin_index.sql (2.78ms)7382026/08/27 09:46:06 OK 20251218171726_add_pins.sql (9.03ms)7392026/08/27 09:46:06 OK 20251218171726_add_pins.sql (6.24ms)7402026/08/27 09:46:06 OK 1_commit_pending_closure.sql (3.68ms)7412026/08/27 09:46:06 OK 20260628120000_add_object_size_and_stats.sql (6.36ms)7422026/08/27 09:46:06 goose: successfully migrated database to version: 202606281200007432026/08/27 09:46:06 OK 20251218171726_add_pins.sql (5.47ms)7442026/08/27 09:46:06 OK 20260628120000_add_object_size_and_stats.sql (11.76ms)7452026/08/27 09:46:06 goose: successfully migrated database to version: 202606281200007462026/08/27 09:46:06 OK 2_object_stats_trigger.sql (9.32ms)7472026/08/27 09:46:06 goose: up to current file version: 27482026/08/27 09:46:06 OK 20260628120000_add_object_size_and_stats.sql (12.61ms)7492026/08/27 09:46:06 OK 1_commit_pending_closure.sql (11.43ms)7502026/08/27 09:46:06 goose: successfully migrated database to version: 202606281200007512026/08/27 09:46:06 OK 20260628120000_add_object_size_and_stats.sql (16.52ms)7522026/08/27 09:46:06 goose: successfully migrated database to version: 202606281200007532026/08/27 09:46:06 OK 20241026095416_initial_model.sql (22.35ms)7542026/08/27 09:46:06 OK 2_object_stats_trigger.sql (3.45ms)7552026/08/27 09:46:06 goose: up to current file version: 27562026/08/27 09:46:06 OK 1_commit_pending_closure.sql (7.27ms)7572026/08/27 09:46:06 OK 20260628120000_add_object_size_and_stats.sql (14.9ms)7582026/08/27 09:46:06 goose: successfully migrated database to version: 202606281200007592026/08/27 09:46:06 OK 20251210153512_drop_unused_gin_index.sql (4.46ms)7602026/08/27 09:46:06 OK 1_commit_pending_closure.sql (6.98ms)7612026/08/27 09:46:06 OK 1_commit_pending_closure.sql (6.76ms)7622026/08/27 09:46:06 OK 2_object_stats_trigger.sql (4.38ms)7632026/08/27 09:46:06 goose: up to current file version: 27642026/08/27 09:46:06 OK 2_object_stats_trigger.sql (4.07ms)7652026/08/27 09:46:06 goose: up to current file version: 27662026/08/27 09:46:06 OK 2_object_stats_trigger.sql (4.31ms)7672026/08/27 09:46:06 goose: up to current file version: 27682026/08/27 09:46:06 OK 1_commit_pending_closure.sql (7.41ms)7692026/08/27 09:46:06 OK 20251218171726_add_pins.sql (7.33ms)7702026-08-27 09:46:06.838 UTC [607] ERROR: relation "goose_db_version" does not exist at character 367712026-08-27 09:46:06.838 UTC [607] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC7722026-08-27 09:46:06.839 UTC [605] ERROR: relation "goose_db_version" does not exist at character 367732026-08-27 09:46:06.839 UTC [605] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC7742026-08-27 09:46:06.839 UTC [606] ERROR: relation "goose_db_version" does not exist at character 367752026-08-27 09:46:06.839 UTC [606] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC7762026/08/27 09:46:06 OK 2_object_stats_trigger.sql (3.78ms)7772026/08/27 09:46:06 goose: up to current file version: 27782026-08-27 09:46:06.843 UTC [608] ERROR: relation "goose_db_version" does not exist at character 367792026-08-27 09:46:06.843 UTC [608] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC7802026/08/27 09:46:06 OK 20260628120000_add_object_size_and_stats.sql (10.48ms)7812026/08/27 09:46:06 goose: successfully migrated database to version: 202606281200007822026-08-27 09:46:06.851 UTC [609] ERROR: relation "goose_db_version" does not exist at character 367832026-08-27 09:46:06.851 UTC [609] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC7842026-08-27 09:46:06.854 UTC [611] ERROR: relation "goose_db_version" does not exist at character 367852026-08-27 09:46:06.854 UTC [611] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC7862026-08-27 09:46:06.855 UTC [612] ERROR: relation "goose_db_version" does not exist at character 367872026-08-27 09:46:06.855 UTC [612] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC7882026-08-27 09:46:06.855 UTC [610] ERROR: relation "goose_db_version" does not exist at character 367892026-08-27 09:46:06.855 UTC [610] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC7902026-08-27 09:46:06.862 UTC [613] ERROR: relation "goose_db_version" does not exist at character 367912026-08-27 09:46:06.862 UTC [613] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC7922026-08-27 09:46:06.863 UTC [614] ERROR: relation "goose_db_version" does not exist at character 367932026-08-27 09:46:06.863 UTC [614] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC7942026/08/27 09:46:06 OK 1_commit_pending_closure.sql (15.28ms)7952026/08/27 09:46:06 OK 20241026095416_initial_model.sql (15.99ms)7962026/08/27 09:46:06 OK 20241026095416_initial_model.sql (15.86ms)7972026/08/27 09:46:06 OK 2_object_stats_trigger.sql (5.63ms)7982026/08/27 09:46:06 goose: up to current file version: 27992026/08/27 09:46:06 OK 20241026095416_initial_model.sql (19.41ms)8002026/08/27 09:46:06 OK 20251210153512_drop_unused_gin_index.sql (5.18ms)8012026/08/27 09:46:06 OK 20251210153512_drop_unused_gin_index.sql (6.51ms)8022026/08/27 09:46:06 OK 20251210153512_drop_unused_gin_index.sql (4.23ms)8032026/08/27 09:46:06 OK 20241026095416_initial_model.sql (14.97ms)8042026-08-27 09:46:06.876 UTC [616] ERROR: relation "goose_db_version" does not exist at character 368052026-08-27 09:46:06.876 UTC [616] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC8062026-08-27 09:46:06.877 UTC [617] ERROR: relation "goose_db_version" does not exist at character 368072026-08-27 09:46:06.877 UTC [617] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC8082026/08/27 09:46:06 OK 20251218171726_add_pins.sql (7.01ms)8092026-08-27 09:46:06.879 UTC [615] ERROR: relation "goose_db_version" does not exist at character 368102026-08-27 09:46:06.879 UTC [615] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC8112026/08/27 09:46:06 OK 20241026095416_initial_model.sql (12.74ms)8122026/08/27 09:46:06 OK 20251218171726_add_pins.sql (7.49ms)8132026/08/27 09:46:06 OK 20251210153512_drop_unused_gin_index.sql (4.23ms)8142026-08-27 09:46:06.881 UTC [618] ERROR: relation "goose_db_version" does not exist at character 368152026-08-27 09:46:06.881 UTC [618] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC8162026/08/27 09:46:06 OK 20251218171726_add_pins.sql (7.16ms)8172026/08/27 09:46:06 OK 20241026095416_initial_model.sql (14.45ms)8182026/08/27 09:46:06 OK 20241026095416_initial_model.sql (13.01ms)8192026/08/27 09:46:06 OK 20241026095416_initial_model.sql (11.23ms)8202026/08/27 09:46:06 OK 20251210153512_drop_unused_gin_index.sql (2.51ms)8212026-08-27 09:46:06.883 UTC [619] ERROR: relation "goose_db_version" does not exist at character 368222026-08-27 09:46:06.883 UTC [619] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC8232026/08/27 09:46:06 OK 20241026095416_initial_model.sql (15.14ms)8242026/08/27 09:46:06 OK 20260628120000_add_object_size_and_stats.sql (6.48ms)8252026/08/27 09:46:06 goose: successfully migrated database to version: 202606281200008262026-08-27 09:46:06.885 UTC [620] ERROR: relation "goose_db_version" does not exist at character 368272026-08-27 09:46:06.885 UTC [620] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC8282026/08/27 09:46:06 OK 20251210153512_drop_unused_gin_index.sql (3.63ms)8292026/08/27 09:46:06 OK 20251210153512_drop_unused_gin_index.sql (2.77ms)8302026/08/27 09:46:06 OK 20251210153512_drop_unused_gin_index.sql (3.79ms)8312026-08-27 09:46:06.886 UTC [621] ERROR: relation "goose_db_version" does not exist at character 368322026-08-27 09:46:06.886 UTC [621] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC8332026/08/27 09:46:06 OK 20241026095416_initial_model.sql (14.36ms)8342026/08/27 09:46:06 OK 20260628120000_add_object_size_and_stats.sql (6.15ms)8352026/08/27 09:46:06 goose: successfully migrated database to version: 202606281200008362026/08/27 09:46:06 OK 20251218171726_add_pins.sql (6.04ms)8372026/08/27 09:46:06 OK 20251218171726_add_pins.sql (4.65ms)8382026/08/27 09:46:06 OK 20260628120000_add_object_size_and_stats.sql (5.97ms)8392026/08/27 09:46:06 goose: successfully migrated database to version: 202606281200008402026/08/27 09:46:06 OK 20251210153512_drop_unused_gin_index.sql (3.82ms)8412026/08/27 09:46:06 OK 1_commit_pending_closure.sql (4.03ms)8422026/08/27 09:46:06 OK 20251218171726_add_pins.sql (3.67ms)8432026/08/27 09:46:06 OK 20251218171726_add_pins.sql (4.7ms)8442026/08/27 09:46:06 OK 20251210153512_drop_unused_gin_index.sql (2.87ms)8452026/08/27 09:46:06 OK 1_commit_pending_closure.sql (4.18ms)8462026/08/27 09:46:06 OK 1_commit_pending_closure.sql (3.76ms)8472026/08/27 09:46:06 OK 2_object_stats_trigger.sql (3.89ms)8482026/08/27 09:46:06 goose: up to current file version: 28492026/08/27 09:46:06 OK 2_object_stats_trigger.sql (2.46ms)8502026/08/27 09:46:06 goose: up to current file version: 28512026/08/27 09:46:06 OK 2_object_stats_trigger.sql (2.47ms)8522026/08/27 09:46:06 goose: up to current file version: 28532026/08/27 09:46:06 OK 20251218171726_add_pins.sql (6.29ms)8542026/08/27 09:46:06 OK 20251218171726_add_pins.sql (8.23ms)8552026/08/27 09:46:06 OK 20260628120000_add_object_size_and_stats.sql (6.47ms)8562026/08/27 09:46:06 goose: successfully migrated database to version: 202606281200008572026/08/27 09:46:06 OK 20260628120000_add_object_size_and_stats.sql (4.85ms)8582026/08/27 09:46:06 goose: successfully migrated database to version: 202606281200008592026/08/27 09:46:06 OK 20260628120000_add_object_size_and_stats.sql (6.72ms)8602026/08/27 09:46:06 goose: successfully migrated database to version: 202606281200008612026/08/27 09:46:06 OK 20251218171726_add_pins.sql (5.12ms)862{"timestamp":"2026-08-27T09:46:06.896000889Z","level":"ERROR","duration":"451.304µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(349)"}863{"timestamp":"2026-08-27T09:46:06.89609651Z","level":"ERROR","duration":"308.823µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(385)"}864{"timestamp":"2026-08-27T09:46:06.896198951Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"20fe2193-5068-4ab9-8e98-3543bb9db4ba","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket11/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(385)"}865{"timestamp":"2026-08-27T09:46:06.896716456Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"0994ce2e-ee6e-4603-b1c5-a3a24812de6d","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket10/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(349)"}8662026/08/27 09:46:06 OK 20260628120000_add_object_size_and_stats.sql (7.04ms)8672026/08/27 09:46:06 goose: successfully migrated database to version: 202606281200008682026/08/27 09:46:06 OK 20241026095416_initial_model.sql (12.38ms)8692026/08/27 09:46:06 INFO Received uploads request method=POST path=/api/pending_closures8702026/08/27 09:46:06 OK 1_commit_pending_closure.sql (4.2ms)8712026/08/27 09:46:06 OK 1_commit_pending_closure.sql (4.14ms)8722026/08/27 09:46:06 OK 1_commit_pending_closure.sql (4.27ms)8732026/08/27 09:46:06 OK 20241026095416_initial_model.sql (13.24ms)8742026/08/27 09:46:06 OK 20260628120000_add_object_size_and_stats.sql (5.44ms)8752026/08/27 09:46:06 goose: successfully migrated database to version: 202606281200008762026/08/27 09:46:06 OK 20241026095416_initial_model.sql (12.77ms)8772026/08/27 09:46:06 OK 20260628120000_add_object_size_and_stats.sql (5.97ms)8782026/08/27 09:46:06 goose: successfully migrated database to version: 202606281200008792026/08/27 09:46:06 OK 20251210153512_drop_unused_gin_index.sql (2.69ms)8802026/08/27 09:46:06 OK 20260628120000_add_object_size_and_stats.sql (6.49ms)8812026/08/27 09:46:06 OK 2_object_stats_trigger.sql (3.21ms)8822026/08/27 09:46:06 goose: up to current file version: 28832026/08/27 09:46:06 goose: successfully migrated database to version: 202606281200008842026/08/27 09:46:06 OK 20241026095416_initial_model.sql (12.31ms)8852026/08/27 09:46:06 OK 1_commit_pending_closure.sql (4.8ms)8862026/08/27 09:46:06 OK 2_object_stats_trigger.sql (3.27ms)8872026/08/27 09:46:06 goose: up to current file version: 28882026/08/27 09:46:06 OK 2_object_stats_trigger.sql (3.25ms)8892026/08/27 09:46:06 goose: up to current file version: 28902026/08/27 09:46:06 OK 20251210153512_drop_unused_gin_index.sql (3.2ms)8912026/08/27 09:46:06 OK 1_commit_pending_closure.sql (3.5ms)8922026/08/27 09:46:06 OK 1_commit_pending_closure.sql (3.38ms)8932026/08/27 09:46:06 OK 20241026095416_initial_model.sql (11.62ms)894{"timestamp":"2026-08-27T09:46:06.904345023Z","level":"ERROR","duration":"582.125µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(197)"}895{"timestamp":"2026-08-27T09:46:06.904410784Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"ecac2a59-cd8b-4f27-96f4-8d1ecfeab2a9","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket14/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(197)"}896{"timestamp":"2026-08-27T09:46:06.904641826Z","level":"ERROR","duration":"1.025589ms","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(375)"}897{"timestamp":"2026-08-27T09:46:06.904724547Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"bd5d5094-1f6a-4b2f-ba5f-3ec4cbb1b165","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket13/","status_code":503,"duration_ms":1,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(375)"}8982026/08/27 09:46:06 OK 20251210153512_drop_unused_gin_index.sql (5.21ms)8992026/08/27 09:46:06 OK 2_object_stats_trigger.sql (3.28ms)9002026/08/27 09:46:06 goose: up to current file version: 29012026/08/27 09:46:06 OK 20251210153512_drop_unused_gin_index.sql (3.46ms)9022026/08/27 09:46:06 OK 1_commit_pending_closure.sql (4.38ms)9032026/08/27 09:46:06 OK 2_object_stats_trigger.sql (2.84ms)9042026/08/27 09:46:06 goose: up to current file version: 29052026/08/27 09:46:06 OK 20241026095416_initial_model.sql (11.99ms)9062026/08/27 09:46:06 OK 20251210153512_drop_unused_gin_index.sql (3.02ms)9072026/08/27 09:46:06 OK 20251218171726_add_pins.sql (6.23ms)908{"timestamp":"2026-08-27T09:46:06.907163368Z","level":"ERROR","duration":"386.703µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(197)"}9092026/08/27 09:46:06 OK 2_object_stats_trigger.sql (2.82ms)910{"timestamp":"2026-08-27T09:46:06.907472531Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"582bbca4-9f55-4900-a2b6-ff1800659c55","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket15/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(197)"}9112026/08/27 09:46:06 goose: up to current file version: 29122026/08/27 09:46:06 OK 20241026095416_initial_model.sql (11.03ms)9132026/08/27 09:46:06 OK 20251218171726_add_pins.sql (5.06ms)914{"timestamp":"2026-08-27T09:46:06.908774683Z","level":"ERROR","duration":"338.723µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(375)"}915{"timestamp":"2026-08-27T09:46:06.908810723Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"29c97706-51e5-4e28-b91c-635acb481ded","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket16/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(375)"}916{"timestamp":"2026-08-27T09:46:06.908863444Z","level":"ERROR","duration":"264.123µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(197)"}917{"timestamp":"2026-08-27T09:46:06.908903444Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"226e70df-8d98-4c30-a1cc-6ec4f26d61d4","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket17/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(197)"}9182026/08/27 09:46:06 OK 2_object_stats_trigger.sql (2.93ms)9192026/08/27 09:46:06 goose: up to current file version: 29202026/08/27 09:46:06 OK 20251218171726_add_pins.sql (5.47ms)9212026/08/27 09:46:06 OK 20251218171726_add_pins.sql (5.4ms)9222026/08/27 09:46:06 OK 20251210153512_drop_unused_gin_index.sql (4.46ms)9232026/08/27 09:46:06 OK 20251210153512_drop_unused_gin_index.sql (3.56ms)9242026/08/27 09:46:06 OK 20251218171726_add_pins.sql (4.56ms)925{"timestamp":"2026-08-27T09:46:06.912265834Z","level":"ERROR","duration":"404.824µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(366)"}926{"timestamp":"2026-08-27T09:46:06.912324975Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"cc38586c-8398-4970-95d8-96c6c72e6a81","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket18/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(366)"}9272026/08/27 09:46:06 OK 20260628120000_add_object_size_and_stats.sql (6.08ms)9282026/08/27 09:46:06 goose: successfully migrated database to version: 202606281200009292026/08/27 09:46:06 OK 20260628120000_add_object_size_and_stats.sql (7.07ms)9302026/08/27 09:46:06 goose: successfully migrated database to version: 202606281200009312026/08/27 09:46:06 OK 1_commit_pending_closure.sql (3.4ms)9322026/08/27 09:46:06 OK 20260628120000_add_object_size_and_stats.sql (5.48ms)9332026/08/27 09:46:06 goose: successfully migrated database to version: 202606281200009342026/08/27 09:46:06 OK 20251218171726_add_pins.sql (5.42ms)9352026/08/27 09:46:06 OK 20260628120000_add_object_size_and_stats.sql (5.64ms)9362026/08/27 09:46:06 goose: successfully migrated database to version: 202606281200009372026/08/27 09:46:06 OK 20251218171726_add_pins.sql (5.36ms)9382026/08/27 09:46:06 OK 20260628120000_add_object_size_and_stats.sql (5.43ms)9392026/08/27 09:46:06 goose: successfully migrated database to version: 202606281200009402026/08/27 09:46:06 OK 1_commit_pending_closure.sql (2.15ms)9412026/08/27 09:46:06 OK 2_object_stats_trigger.sql (1.22ms)9422026/08/27 09:46:06 goose: up to current file version: 29432026/08/27 09:46:06 OK 1_commit_pending_closure.sql (2.09ms)944{"timestamp":"2026-08-27T09:46:06.920133044Z","level":"ERROR","duration":"431.043µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(375)"}945{"timestamp":"2026-08-27T09:46:06.920195245Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"d38a94f3-f829-46ba-a41e-3c1bd1d339d8","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket19/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(375)"}9462026/08/27 09:46:06 OK 1_commit_pending_closure.sql (2.92ms)9472026/08/27 09:46:06 OK 2_object_stats_trigger.sql (2.51ms)9482026/08/27 09:46:06 goose: up to current file version: 29492026/08/27 09:46:06 OK 1_commit_pending_closure.sql (2.74ms)9502026/08/27 09:46:06 OK 2_object_stats_trigger.sql (908.09µs)9512026/08/27 09:46:06 goose: up to current file version: 29522026/08/27 09:46:06 OK 20260628120000_add_object_size_and_stats.sql (4.02ms)9532026/08/27 09:46:06 goose: successfully migrated database to version: 20260628120000954{"timestamp":"2026-08-27T09:46:06.921500037Z","level":"ERROR","duration":"286.243µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(349)"}955{"timestamp":"2026-08-27T09:46:06.921533797Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"f438ae9d-24f4-42f5-83cd-4511dd0e8634","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket20/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(349)"}9562026/08/27 09:46:06 OK 2_object_stats_trigger.sql (1.3ms)9572026/08/27 09:46:06 goose: up to current file version: 29582026/08/27 09:46:06 OK 20260628120000_add_object_size_and_stats.sql (3.99ms)9592026/08/27 09:46:06 goose: successfully migrated database to version: 20260628120000960{"timestamp":"2026-08-27T09:46:06.922429985Z","level":"ERROR","duration":"264.082µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(349)"}961{"timestamp":"2026-08-27T09:46:06.922460425Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"3b98fc8e-720d-47ae-a3a5-5992af3b9e41","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket22/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(349)"}9622026/08/27 09:46:06 OK 2_object_stats_trigger.sql (2.01ms)963{"timestamp":"2026-08-27T09:46:06.922607007Z","level":"ERROR","duration":"438.404µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(356)"}964{"timestamp":"2026-08-27T09:46:06.922637487Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"ae655c6b-ada7-45d8-8d18-384f6b693a4d","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket21/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(356)"}9652026/08/27 09:46:06 goose: up to current file version: 29662026/08/27 09:46:06 INFO Registered completed upload object_key=nar/jjjjjjjjjjjjjjjjjjjjjjjjjjjjjj0100000000000000000000.nar.zst9672026/08/27 09:46:06 INFO Received uploads request method=POST path=/api/pending_closures9682026/08/27 09:46:06 OK 1_commit_pending_closure.sql (1.89ms)9692026/08/27 09:46:06 OK 1_commit_pending_closure.sql (1.92ms)970{"timestamp":"2026-08-27T09:46:06.924476083Z","level":"ERROR","duration":"791.947µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(197)"}971{"timestamp":"2026-08-27T09:46:06.924532524Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"cc58ba3e-8986-429b-84a9-61d1e9d1343d","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket23/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(197)"}9722026/08/27 09:46:06 OK 2_object_stats_trigger.sql (1.01ms)973--- PASS: TestPresignedUploadRegisteredBeforeCommit (0.32s)9742026/08/27 09:46:06 goose: up to current file version: 2975=== CONT TestGCTaskStore_GetEmpty976--- PASS: TestGCTaskStore_GetEmpty (0.00s)977=== CONT TestGCTaskStore_ConflictDifferentParams978--- PASS: TestGCTaskStore_ConflictDifferentParams (0.00s)979=== CONT TestOrphanedObjectsGCStressTest9802026/08/27 09:46:06 OK 2_object_stats_trigger.sql (1.65ms)9812026/08/27 09:46:06 goose: up to current file version: 2982{"timestamp":"2026-08-27T09:46:06.925615093Z","level":"ERROR","duration":"288.762µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(197)"}983{"timestamp":"2026-08-27T09:46:06.925657493Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"145b8b95-5c4b-44e8-a930-72db0e379579","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket24/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(197)"}984{"timestamp":"2026-08-27T09:46:06.926162398Z","level":"ERROR","duration":"198.062µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(197)"}985{"timestamp":"2026-08-27T09:46:06.926202478Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"3c0f0273-7631-49b2-b09c-522618bae9f2","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket25/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(197)"}9862026/08/27 09:46:06 INFO Received uploads request method=POST path=/api/pending_closures9872026/08/27 09:46:06 WARN Authentication failed token_preview=wrongtoken token_length=10 reason="static API token mismatch"988--- PASS: TestService_AuthMiddleware (0.35s)989=== CONT TestGCTaskStore_DeduplicateSameParams990--- PASS: TestGCTaskStore_DeduplicateSameParams (0.00s)991=== CONT TestOrphanedObjectsGC992--- PASS: TestCreatePendingClosure_SmallNARUsesSimplePUT (0.34s)993=== CONT TestGCTaskStore_StartNew994--- PASS: TestGCTaskStore_StartNew (0.00s)995=== CONT TestGCMetrics9962026/08/27 09:46:06 INFO Received uploads request method=POST path=/api/pending_closures9972026/08/27 09:46:06 INFO Received uploads request method=POST path=/api/pending_closures9982026-08-27 09:46:07.003 UTC [629] ERROR: relation "goose_db_version" does not exist at character 369992026-08-27 09:46:07.003 UTC [629] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC10002026/08/27 09:46:07 OK 20241026095416_initial_model.sql (11.6ms)10012026/08/27 09:46:07 OK 20251210153512_drop_unused_gin_index.sql (1.62ms)10022026/08/27 09:46:07 OK 20251218171726_add_pins.sql (4.49ms)10032026/08/27 09:46:07 OK 20260628120000_add_object_size_and_stats.sql (3.06ms)10042026/08/27 09:46:07 goose: successfully migrated database to version: 2026062812000010052026-08-27 09:46:07.047 UTC [630] ERROR: relation "goose_db_version" does not exist at character 3610062026-08-27 09:46:07.047 UTC [630] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC10072026/08/27 09:46:07 OK 1_commit_pending_closure.sql (2.45ms)10082026-08-27 09:46:07.048 UTC [631] ERROR: relation "goose_db_version" does not exist at character 3610092026-08-27 09:46:07.048 UTC [631] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC10102026/08/27 09:46:07 OK 2_object_stats_trigger.sql (1.03ms)10112026/08/27 09:46:07 goose: up to current file version: 210122026/08/27 09:46:07 OK 20241026095416_initial_model.sql (11.47ms)10132026/08/27 09:46:07 OK 20241026095416_initial_model.sql (11.33ms)10142026/08/27 09:46:07 OK 20251210153512_drop_unused_gin_index.sql (1.73ms)10152026/08/27 09:46:07 OK 20251210153512_drop_unused_gin_index.sql (1.64ms)10162026/08/27 09:46:07 OK 20251218171726_add_pins.sql (3.53ms)10172026/08/27 09:46:07 OK 20251218171726_add_pins.sql (3.7ms)10182026/08/27 09:46:07 OK 20260628120000_add_object_size_and_stats.sql (4.02ms)10192026/08/27 09:46:07 goose: successfully migrated database to version: 2026062812000010202026/08/27 09:46:07 OK 20260628120000_add_object_size_and_stats.sql (5.13ms)10212026/08/27 09:46:07 goose: successfully migrated database to version: 2026062812000010222026/08/27 09:46:07 OK 1_commit_pending_closure.sql (2.02ms)10232026/08/27 09:46:07 OK 2_object_stats_trigger.sql (5.05ms)10242026/08/27 09:46:07 goose: up to current file version: 210252026/08/27 09:46:07 OK 1_commit_pending_closure.sql (6.6ms)10262026/08/27 09:46:07 OK 2_object_stats_trigger.sql (1.13ms)10272026/08/27 09:46:07 goose: up to current file version: 21028{"timestamp":"2026-08-27T09:46:07.487293304Z","level":"ERROR","duration":"202.702µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(349)"}1029{"timestamp":"2026-08-27T09:46:07.487296384Z","level":"ERROR","duration":"114.761µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(356)"}1030{"timestamp":"2026-08-27T09:46:07.487349424Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"20b95fc7-8d0c-40d3-97a1-ec9455d77bed","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket11/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(349)"}1031{"timestamp":"2026-08-27T09:46:07.487374865Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"439fc243-b592-42b5-8b19-d299ee04773e","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket23/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(356)"}1032{"timestamp":"2026-08-27T09:46:07.487391205Z","level":"ERROR","duration":"195.102µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(375)"}1033{"timestamp":"2026-08-27T09:46:07.487395245Z","level":"ERROR","duration":"215.102µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(366)"}1034{"timestamp":"2026-08-27T09:46:07.487441125Z","level":"ERROR","duration":"122.021µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(342)"}1035{"timestamp":"2026-08-27T09:46:07.487451625Z","level":"ERROR","duration":"91.801µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(368)"}1036{"timestamp":"2026-08-27T09:46:07.487452445Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"aeb9ebdb-043d-427b-a86a-9295c33ec178","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket13/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(375)"}1037{"timestamp":"2026-08-27T09:46:07.487454985Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"f75bc964-08b7-4be6-b5c4-b1ffaad1bec3","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket10/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(366)"}1038{"timestamp":"2026-08-27T09:46:07.487468505Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"aac89195-7f8a-43c2-9030-ab5057fb5689","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket18/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(342)"}1039{"timestamp":"2026-08-27T09:46:07.487480385Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"bfb02fcd-b3ac-4b9d-9118-79685d48770b","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket16/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(368)"}1040{"timestamp":"2026-08-27T09:46:07.487478625Z","level":"ERROR","duration":"42.86µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(349)"}1041{"timestamp":"2026-08-27T09:46:07.487492346Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"736b599d-0325-4f3a-99a4-bc45c9c7d69b","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket25/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(349)"}1042{"timestamp":"2026-08-27T09:46:07.487568506Z","level":"ERROR","duration":"29.14µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(349)"}1043{"timestamp":"2026-08-27T09:46:07.487581126Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"a918952c-7169-4707-afbe-aba7a0b83d8d","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket15/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(349)"}1044{"timestamp":"2026-08-27T09:46:07.487626707Z","level":"ERROR","duration":"43.341µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(342)"}1045{"timestamp":"2026-08-27T09:46:07.487640887Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"1b9f3b04-2bab-421e-8904-afb1aa8a4e6a","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket19/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(342)"}1046{"timestamp":"2026-08-27T09:46:07.487648927Z","level":"ERROR","duration":"45.44µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(368)"}1047{"timestamp":"2026-08-27T09:46:07.487661967Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"449571f2-fb27-484e-baa2-416c8ffb77c7","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket24/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(368)"}1048{"timestamp":"2026-08-27T09:46:07.487727368Z","level":"ERROR","duration":"218.462µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(385)"}1049{"timestamp":"2026-08-27T09:46:07.487790128Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"4f1eccf6-ebd1-4f5b-8063-11b84a679cf9","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket21/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(385)"}1050{"timestamp":"2026-08-27T09:46:07.489170381Z","level":"ERROR","duration":"478.305µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(353)"}1051{"timestamp":"2026-08-27T09:46:07.489205321Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"03da013a-2b68-4ea0-8d1d-c47b955709a9","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket26/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(353)"}1052{"timestamp":"2026-08-27T09:46:07.489480243Z","level":"ERROR","duration":"533.184µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(368)"}1053{"timestamp":"2026-08-27T09:46:07.489513344Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"0a2c8b78-d479-4c51-a647-1d3742b4bca6","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket27/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(368)"}1054{"timestamp":"2026-08-27T09:46:07.490782975Z","level":"ERROR","duration":"1.06289ms","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(366)"}1055{"timestamp":"2026-08-27T09:46:07.490851916Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"135a7843-6bb5-41da-bb93-92a27f13dd88","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket28/","status_code":503,"duration_ms":1,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(366)"}1056--- PASS: TestReadProxyRangeRequest (0.89s)1057=== CONT TestUploadHandlersRejectInvalidKeys1058=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info1059=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info1060=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal1061=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal1062=== RUN TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key1063=== PAUSE TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key1064=== RUN TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key1065=== PAUSE TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key1066=== CONT TestGCBugBareHashReferences1067--- PASS: TestReadProxyConditionalGet (0.90s)1068=== CONT TestPinProtectsFromGC10692026/08/27 09:46:07 INFO Received uploads request method=POST path=/api/pending_closures1070--- PASS: TestReadRedirectNar (0.93s)1071=== CONT TestCacheConfigHandler1072=== RUN TestCacheConfigHandler/full_config,_no_issuer1073=== PAUSE TestCacheConfigHandler/full_config,_no_issuer1074=== RUN TestCacheConfigHandler/no_cache_url_configured1075=== PAUSE TestCacheConfigHandler/no_cache_url_configured1076=== RUN TestCacheConfigHandler/no_signing_keys1077=== PAUSE TestCacheConfigHandler/no_signing_keys1078=== RUN TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1079=== PAUSE TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1080=== CONT TestCacheStatsHandler10812026-08-27 09:46:07.565 UTC [638] ERROR: relation "goose_db_version" does not exist at character 3610822026-08-27 09:46:07.565 UTC [638] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC10832026-08-27 09:46:07.585 UTC [639] ERROR: relation "goose_db_version" does not exist at character 3610842026-08-27 09:46:07.585 UTC [639] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC10852026/08/27 09:46:07 OK 20241026095416_initial_model.sql (11.99ms)10862026/08/27 09:46:07 OK 20251210153512_drop_unused_gin_index.sql (2.96ms)10872026/08/27 09:46:07 OK 20251218171726_add_pins.sql (4.2ms)10882026/08/27 09:46:07 OK 20260628120000_add_object_size_and_stats.sql (10.13ms)10892026/08/27 09:46:07 goose: successfully migrated database to version: 2026062812000010902026/08/27 09:46:07 OK 20241026095416_initial_model.sql (11.92ms)10912026/08/27 09:46:07 OK 1_commit_pending_closure.sql (2.45ms)10922026/08/27 09:46:07 OK 20251210153512_drop_unused_gin_index.sql (1.93ms)10932026/08/27 09:46:07 OK 2_object_stats_trigger.sql (1.62ms)10942026/08/27 09:46:07 goose: up to current file version: 210952026/08/27 09:46:07 OK 20251218171726_add_pins.sql (3.17ms)10962026/08/27 09:46:07 OK 20260628120000_add_object_size_and_stats.sql (2.99ms)10972026/08/27 09:46:07 goose: successfully migrated database to version: 2026062812000010982026/08/27 09:46:07 OK 1_commit_pending_closure.sql (1.87ms)10992026/08/27 09:46:07 OK 2_object_stats_trigger.sql (1.15ms)11002026/08/27 09:46:07 goose: up to current file version: 211012026-08-27 09:46:07.617 UTC [640] ERROR: relation "goose_db_version" does not exist at character 3611022026-08-27 09:46:07.617 UTC [640] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC11032026/08/27 09:46:07 OK 20241026095416_initial_model.sql (10.23ms)11042026/08/27 09:46:07 OK 20251210153512_drop_unused_gin_index.sql (1.4ms)11052026/08/27 09:46:07 OK 20251218171726_add_pins.sql (3.58ms)11062026/08/27 09:46:07 OK 20260628120000_add_object_size_and_stats.sql (3.79ms)11072026/08/27 09:46:07 goose: successfully migrated database to version: 2026062812000011082026/08/27 09:46:07 OK 1_commit_pending_closure.sql (2.05ms)11092026/08/27 09:46:07 OK 2_object_stats_trigger.sql (884.63µs)11102026/08/27 09:46:07 goose: up to current file version: 21111{"timestamp":"2026-08-27T09:46:07.688326877Z","level":"ERROR","duration":"199.562µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(349)"}1112{"timestamp":"2026-08-27T09:46:07.688326737Z","level":"ERROR","duration":"141.641µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(366)"}1113{"timestamp":"2026-08-27T09:46:07.688367197Z","level":"ERROR","duration":"105.501µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(356)"}1114{"timestamp":"2026-08-27T09:46:07.688388557Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"958c835a-8e1d-41ce-81bc-ecd78f352cde","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket24/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(349)"}1115{"timestamp":"2026-08-27T09:46:07.688363737Z","level":"ERROR","duration":"155.501µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(375)"}1116{"timestamp":"2026-08-27T09:46:07.688395477Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"72e016e5-f49e-4dd8-b9a7-28eb0fe806e8","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket18/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(356)"}1117{"timestamp":"2026-08-27T09:46:07.688420658Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"058f5274-e4b6-4e03-852a-68118fa4f1ca","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket15/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(366)"}1118{"timestamp":"2026-08-27T09:46:07.688417278Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"cd728044-300c-49ec-8d73-e1b19367783c","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket19/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(375)"}1119{"timestamp":"2026-08-27T09:46:07.688582699Z","level":"ERROR","duration":"183.122µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(363)"}1120{"timestamp":"2026-08-27T09:46:07.6886464Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"b39f5b00-0ee9-4884-a643-cc109ff6ab7d","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket27/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(363)"}1121{"timestamp":"2026-08-27T09:46:07.689211525Z","level":"ERROR","duration":"155.102µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(356)"}1122{"timestamp":"2026-08-27T09:46:07.689243645Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"d7c9b1f3-69e9-41e5-9293-ceb9e1209396","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket26/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(356)"}1123{"timestamp":"2026-08-27T09:46:07.690201654Z","level":"ERROR","duration":"1.766396ms","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(375)"}1124{"timestamp":"2026-08-27T09:46:07.690253054Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"020cac10-8a87-499e-bb89-c0ee0a48ed99","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket31/","status_code":503,"duration_ms":1,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(375)"}1125{"timestamp":"2026-08-27T09:46:07.691281523Z","level":"ERROR","duration":"1.048729ms","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(375)"}1126{"timestamp":"2026-08-27T09:46:07.691328024Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"55c0d831-b0ae-4f54-b760-e8c5c3f86afe","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket29/","status_code":503,"duration_ms":1,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(375)"}1127{"timestamp":"2026-08-27T09:46:07.740809385Z","level":"ERROR","duration":"654.666µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(197)"}1128{"timestamp":"2026-08-27T09:46:07.740893366Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"384a9d2b-6d69-485e-afbf-7431dfc4b09d","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket23/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(197)"}1129{"timestamp":"2026-08-27T09:46:07.80973262Z","level":"ERROR","duration":"259.182µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(197)"}1130{"timestamp":"2026-08-27T09:46:07.809799841Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"00270a23-f7a3-478c-a8b2-9a3be071550f","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket29/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(197)"}1131{"timestamp":"2026-08-27T09:46:07.813678935Z","level":"ERROR","duration":"131.821µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(197)"}1132{"timestamp":"2026-08-27T09:46:07.813728016Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"f48eb2b5-4b53-4351-865d-eed4e4aa38fe","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket21/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(197)"}1133{"timestamp":"2026-08-27T09:46:07.822321372Z","level":"ERROR","duration":"114.721µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(197)"}1134{"timestamp":"2026-08-27T09:46:07.822368393Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"7cb4ea12-7e88-487e-b31e-ec9bf2b5d47c","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket25/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(197)"}1135{"timestamp":"2026-08-27T09:46:07.837135864Z","level":"ERROR","duration":"137.361µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(197)"}1136{"timestamp":"2026-08-27T09:46:07.837190005Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"d464f712-95c9-4765-8310-59eeb30b31a5","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket19/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(197)"}1137{"timestamp":"2026-08-27T09:46:07.84111604Z","level":"ERROR","duration":"97.421µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(197)"}1138{"timestamp":"2026-08-27T09:46:07.84115884Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"863ca636-690a-4801-bbc8-d44f1c2034ba","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket15/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(197)"}1139{"timestamp":"2026-08-27T09:46:07.859795567Z","level":"ERROR","duration":"128.002µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(197)"}1140{"timestamp":"2026-08-27T09:46:07.859846667Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"80c64f9d-3d06-40fd-8884-57a112fba4f3","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket31/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(197)"}1141{"timestamp":"2026-08-27T09:46:07.99583302Z","level":"ERROR","duration":"123.36448ms","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(351)"}1142{"timestamp":"2026-08-27T09:46:07.995902401Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"d2200fe2-c77e-4beb-be0e-390ff1c145fb","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket16/","status_code":503,"duration_ms":123,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(351)"}1143{"timestamp":"2026-08-27T09:46:07.995910001Z","level":"ERROR","duration":"139.122µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(197)"}1144{"timestamp":"2026-08-27T09:46:07.995956261Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"6d88b923-f96b-43bc-9dda-05306db721b6","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket26/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(197)"}1145{"timestamp":"2026-08-27T09:46:07.996530386Z","level":"ERROR","duration":"772.347µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(342)"}1146{"timestamp":"2026-08-27T09:46:07.996565086Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"e3f2b57a-4257-4a6c-8d5a-cbce027eeb5c","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket27/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(342)"}1147{"timestamp":"2026-08-27T09:46:07.997364674Z","level":"ERROR","duration":"306.386573ms","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(358)"}1148{"timestamp":"2026-08-27T09:46:07.997398794Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"2b1ddb72-8626-4429-a720-26166ab2fc71","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket30/","status_code":503,"duration_ms":306,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(358)"}1149{"timestamp":"2026-08-27T09:46:07.999477513Z","level":"ERROR","duration":"308.31531ms","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(373)"}1150{"timestamp":"2026-08-27T09:46:07.999532733Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"50022717-cc58-46a4-975b-4ef0c6c7ff27","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket11/","status_code":503,"duration_ms":310,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(373)"}1151--- PASS: TestReadProxyNarinfoAlreadyDecompressed (1.40s)1152=== CONT TestService_AuthMiddleware_OIDC11532026/08/27 09:46:08 INFO Received uploads request method=POST path=/api/pending_closures11542026/08/27 09:46:08 INFO OIDC provider initialized name=test11552026/08/27 09:46:08 INFO Received uploads request method=POST path=/api/pending_closures11562026/08/27 09:46:08 INFO Received complete multipart upload request method=POST path=/api/multipart/complete11572026/08/27 09:46:08 ERROR Failed to get multipart upload error="no rows in result set" upload_id=does-not-exist object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst1158--- PASS: TestCompleteMultipartUnregistered (1.42s)1159=== CONT TestClientWithDependencies11602026/08/27 09:46:08 INFO Received complete multipart upload request method=POST path=/api/multipart/complete11612026/08/27 09:46:08 WARN CompleteMultipartUpload errored but object exists; treating as success error="The specified key does not exist." object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=ZDc1NjU4ZjQtODU1My00NmQ5LWExMGItYjU3ZmVhODBhNTljLjAwMjhjMTQ4LWY2OGEtNGM4Yi04OTQ1LTE2NDU1MDg3YTQ4M3gxNzg3ODIzOTY4MDIzMDY0MjQz11622026/08/27 09:46:08 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=ZDc1NjU4ZjQtODU1My00NmQ5LWExMGItYjU3ZmVhODBhNTljLjAwMjhjMTQ4LWY2OGEtNGM4Yi04OTQ1LTE2NDU1MDg3YTQ4M3gxNzg3ODIzOTY4MDIzMDY0MjQz parts=11163--- PASS: TestCompleteMultipartUpload_ErrorButObjectExists (1.47s)1164=== CONT TestService_ReadAuthMiddleware11652026-08-27 09:46:08.089 UTC [648] ERROR: relation "goose_db_version" does not exist at character 3611662026-08-27 09:46:08.089 UTC [648] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC11672026-08-27 09:46:08.107 UTC [649] ERROR: relation "goose_db_version" does not exist at character 3611682026-08-27 09:46:08.107 UTC [649] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC11692026/08/27 09:46:08 OK 20241026095416_initial_model.sql (11.1ms)11702026/08/27 09:46:08 OK 20251210153512_drop_unused_gin_index.sql (4.45ms)1171--- PASS: TestReadProxyNarinfo (1.50s)1172=== CONT TestClientMultipleUploads11732026/08/27 09:46:08 OK 20251218171726_add_pins.sql (6.29ms)11742026/08/27 09:46:08 OK 20260628120000_add_object_size_and_stats.sql (7.09ms)11752026/08/27 09:46:08 goose: successfully migrated database to version: 2026062812000011762026/08/27 09:46:08 OK 1_commit_pending_closure.sql (7.31ms)11772026/08/27 09:46:08 OK 20241026095416_initial_model.sql (17.92ms)11782026/08/27 09:46:08 OK 2_object_stats_trigger.sql (1.44ms)11792026/08/27 09:46:08 goose: up to current file version: 211802026/08/27 09:46:08 OK 20251210153512_drop_unused_gin_index.sql (6.2ms)1181--- PASS: TestReadProxyRootRedirectsToIndexHTML (1.54s)1182=== CONT TestObjectStatsTrigger11832026-08-27 09:46:08.147 UTC [652] ERROR: relation "goose_db_version" does not exist at character 3611842026-08-27 09:46:08.147 UTC [652] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC11852026/08/27 09:46:08 OK 20251218171726_add_pins.sql (5.46ms)11862026/08/27 09:46:08 OK 20260628120000_add_object_size_and_stats.sql (6.86ms)11872026/08/27 09:46:08 goose: successfully migrated database to version: 2026062812000011882026/08/27 09:46:08 OK 1_commit_pending_closure.sql (3.28ms)11892026/08/27 09:46:08 INFO Received uploads request method=POST path=/api/pending_closures11902026/08/27 09:46:08 INFO Received uploads request method=POST path=/api/pending_closures11912026/08/27 09:46:08 INFO Received uploads request method=POST path=/api/pending_closures11922026/08/27 09:46:08 OK 2_object_stats_trigger.sql (2.38ms)11932026/08/27 09:46:08 goose: up to current file version: 211942026/08/27 09:46:08 OK 20241026095416_initial_model.sql (13.15ms)1195=== RUN TestService_AuthMiddleware_OIDC/valid_OIDC_token1196=== PAUSE TestService_AuthMiddleware_OIDC/valid_OIDC_token1197=== RUN TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1198=== PAUSE TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1199=== RUN TestService_AuthMiddleware_OIDC/malformed_token_rejected1200=== PAUSE TestService_AuthMiddleware_OIDC/malformed_token_rejected1201=== RUN TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1202=== PAUSE TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1203=== CONT TestService_AuthMiddleware_MTLSBoundSubjects12042026/08/27 09:46:08 OK 20251210153512_drop_unused_gin_index.sql (10.18ms)12052026/08/27 09:46:08 OK 20251218171726_add_pins.sql (5.92ms)12062026/08/27 09:46:08 OK 20260628120000_add_object_size_and_stats.sql (7.35ms)12072026/08/27 09:46:08 goose: successfully migrated database to version: 2026062812000012082026/08/27 09:46:08 OK 1_commit_pending_closure.sql (3.05ms)12092026/08/27 09:46:08 OK 2_object_stats_trigger.sql (2.23ms)12102026/08/27 09:46:08 goose: up to current file version: 212112026-08-27 09:46:08.204 UTC [659] ERROR: relation "goose_db_version" does not exist at character 3612122026-08-27 09:46:08.204 UTC [659] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC1213--- PASS: TestService_Rustfstest (1.61s)1214=== CONT TestClientIntegration1215--- PASS: TestCacheStatsHandler (0.68s)1216=== CONT TestMultipartCleanup12172026/08/27 09:46:08 OK 20241026095416_initial_model.sql (11.37ms)12182026/08/27 09:46:08 WARN mTLS auth: subject not in bound subjects subject="CN=writer"1219--- PASS: TestService_ReadAuthMiddleware (0.16s)1220=== CONT TestService_AuthMiddleware_MTLSProxyHeader12212026/08/27 09:46:08 OK 20251210153512_drop_unused_gin_index.sql (2.96ms)12222026/08/27 09:46:08 OK 20251218171726_add_pins.sql (5.24ms)12232026-08-27 09:46:08.236 UTC [682] ERROR: relation "goose_db_version" does not exist at character 3612242026-08-27 09:46:08.236 UTC [682] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC12252026/08/27 09:46:08 OK 20260628120000_add_object_size_and_stats.sql (10.82ms)12262026/08/27 09:46:08 goose: successfully migrated database to version: 2026062812000012272026/08/27 09:46:08 OK 1_commit_pending_closure.sql (4.39ms)12282026/08/27 09:46:08 OK 2_object_stats_trigger.sql (2.82ms)12292026/08/27 09:46:08 goose: up to current file version: 212302026-08-27 09:46:08.254 UTC [688] ERROR: relation "goose_db_version" does not exist at character 3612312026-08-27 09:46:08.254 UTC [688] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC12322026/08/27 09:46:08 OK 20241026095416_initial_model.sql (13.18ms)12332026/08/27 09:46:08 OK 20251210153512_drop_unused_gin_index.sql (3.15ms)1234=== NAME TestClientWithDependencies1235 client_integration_test.go:593: Built derivation: /build/TestClientWithDependencies1407706732/001/store/nzqwrxh360rs9x551m4jxqgcxhvfxxla-test-script1236--- PASS: TestReadRedirectKeepsNarinfoProxied (1.67s)1237=== CONT TestClientErrorHandling1238=== RUN TestClientErrorHandling/InvalidStorePath1239=== PAUSE TestClientErrorHandling/InvalidStorePath1240=== RUN TestClientErrorHandling/InvalidAuthToken1241=== PAUSE TestClientErrorHandling/InvalidAuthToken1242=== RUN TestClientErrorHandling/ServerNotAvailable1243=== PAUSE TestClientErrorHandling/ServerNotAvailable1244=== CONT TestServerTLSConfig1245=== RUN TestServerTLSConfig/no_client_CA1246=== PAUSE TestServerTLSConfig/no_client_CA1247=== RUN TestServerTLSConfig/missing_CA_file1248=== PAUSE TestServerTLSConfig/missing_CA_file1249=== RUN TestServerTLSConfig/not_a_PEM_file1250=== PAUSE TestServerTLSConfig/not_a_PEM_file1251=== CONT TestClientCADerivations12522026/08/27 09:46:08 OK 20251218171726_add_pins.sql (15.71ms)12532026/08/27 09:46:08 OK 20241026095416_initial_model.sql (20.74ms)12542026/08/27 09:46:08 OK 20260628120000_add_object_size_and_stats.sql (4.76ms)12552026/08/27 09:46:08 goose: successfully migrated database to version: 2026062812000012562026/08/27 09:46:08 OK 20251210153512_drop_unused_gin_index.sql (3.2ms)12572026/08/27 09:46:08 OK 1_commit_pending_closure.sql (3.52ms)12582026/08/27 09:46:08 OK 2_object_stats_trigger.sql (3.04ms)12592026/08/27 09:46:08 goose: up to current file version: 212602026/08/27 09:46:08 OK 20251218171726_add_pins.sql (7.81ms)12612026/08/27 09:46:08 OK 20260628120000_add_object_size_and_stats.sql (8.59ms)12622026/08/27 09:46:08 goose: successfully migrated database to version: 2026062812000012632026-08-27 09:46:08.304 UTC [725] ERROR: relation "goose_db_version" does not exist at character 3612642026-08-27 09:46:08.304 UTC [725] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC12652026/08/27 09:46:08 OK 1_commit_pending_closure.sql (3.86ms)1266=== NAME TestClientWithDependencies1267 client_integration_test.go:595: Found 1 dependencies (including self)12682026/08/27 09:46:08 OK 2_object_stats_trigger.sql (3.27ms)12692026/08/27 09:46:08 goose: up to current file version: 212702026-08-27 09:46:08.314 UTC [742] ERROR: relation "goose_db_version" does not exist at character 3612712026-08-27 09:46:08.314 UTC [742] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC12722026/08/27 09:46:08 OK 20241026095416_initial_model.sql (11.99ms)12732026/08/27 09:46:08 OK 20251210153512_drop_unused_gin_index.sql (2.11ms)12742026/08/27 09:46:08 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted,O=Other"12752026/08/27 09:46:08 WARN mTLS auth: bound subjects configured but subject DN unavailable12762026/08/27 09:46:08 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted"1277--- PASS: TestService_AuthMiddleware_MTLSBoundSubjects (0.16s)1278=== CONT TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle12792026-08-27 09:46:08.330 UTC [759] ERROR: relation "goose_db_version" does not exist at character 3612802026-08-27 09:46:08.330 UTC [759] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC1281=== NAME TestClientMultipleUploads1282 client_integration_test.go:338: Created store path 0: /build/TestClientMultipleUploads92594044/001/store/95h15v1a14lrg41ih843xsl2dkcz7kva-test-file-0.txt1283--- PASS: TestObjectStatsTrigger (0.19s)1284=== CONT TestService_NativeMTLS12852026/08/27 09:46:08 OK 20251218171726_add_pins.sql (4.52ms)1286=== NAME TestPinProtectsFromGC1287 client_integration_test.go:646: Pinned store path: /build/TestPinProtectsFromGC2135796416/001/store/inlmp7k7rippskqwbinpb5dxdkrrj6n4-pinned-file.txt1288 client_integration_test.go:647: Unpinned store path: /build/TestPinProtectsFromGC2135796416/001/store/acbfl0kgjyvbgaij6v7hgxwrm3qvg8g5-unpinned-file.txt12892026/08/27 09:46:08 OK 20241026095416_initial_model.sql (16.65ms)12902026/08/27 09:46:08 OK 20260628120000_add_object_size_and_stats.sql (7.77ms)12912026/08/27 09:46:08 goose: successfully migrated database to version: 2026062812000012922026/08/27 09:46:08 OK 20251210153512_drop_unused_gin_index.sql (1.77ms)12932026/08/27 09:46:08 OK 1_commit_pending_closure.sql (2.2ms)12942026/08/27 09:46:08 OK 2_object_stats_trigger.sql (1.12ms)12952026/08/27 09:46:08 goose: up to current file version: 212962026/08/27 09:46:08 OK 20251218171726_add_pins.sql (4.62ms)12972026/08/27 09:46:08 OK 20241026095416_initial_model.sql (10.71ms)12982026-08-27 09:46:08.352 UTC [799] ERROR: relation "goose_db_version" does not exist at character 3612992026-08-27 09:46:08.352 UTC [799] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC13002026/08/27 09:46:08 OK 20260628120000_add_object_size_and_stats.sql (5.64ms)13012026/08/27 09:46:08 goose: successfully migrated database to version: 2026062812000013022026/08/27 09:46:08 OK 20251210153512_drop_unused_gin_index.sql (3.99ms)13032026/08/27 09:46:08 OK 1_commit_pending_closure.sql (2.77ms)13042026/08/27 09:46:08 OK 2_object_stats_trigger.sql (2.76ms)13052026/08/27 09:46:08 goose: up to current file version: 213062026/08/27 09:46:08 OK 20251218171726_add_pins.sql (6.4ms)1307=== NAME TestClientMultipleUploads1308 client_integration_test.go:338: Created store path 1: /build/TestClientMultipleUploads92594044/001/store/35d8n1gji3anyh608df4l8nfgf2zqb5z-test-file-1.txt13092026/08/27 09:46:08 OK 20260628120000_add_object_size_and_stats.sql (7.12ms)13102026/08/27 09:46:08 goose: successfully migrated database to version: 2026062812000013112026/08/27 09:46:08 OK 1_commit_pending_closure.sql (4.63ms)13122026/08/27 09:46:08 OK 20241026095416_initial_model.sql (17.69ms)13132026/08/27 09:46:08 WARN Failed to fetch cache config, uploading without size limit error="fetching cache config: server returned 404: 404 page not found\n"13142026/08/27 09:46:08 INFO Received uploads request method=POST path=/api/pending_closures13152026/08/27 09:46:08 OK 2_object_stats_trigger.sql (10.86ms)13162026/08/27 09:46:08 goose: up to current file version: 213172026/08/27 09:46:08 OK 20251210153512_drop_unused_gin_index.sql (12.26ms)13182026/08/27 09:46:08 INFO Received uploads request method=POST path=/api/pending_closures13192026/08/27 09:46:08 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)13202026/08/27 09:46:08 INFO Uploading nzqwrxh360rs9x551m4jxqgcxhvfxxla-test-script (136B)13212026/08/27 09:46:08 OK 20251218171726_add_pins.sql (7.96ms)13222026/08/27 09:46:08 OK 20260628120000_add_object_size_and_stats.sql (7.56ms)13232026/08/27 09:46:08 goose: successfully migrated database to version: 2026062812000013242026/08/27 09:46:08 OK 1_commit_pending_closure.sql (4.76ms)13252026/08/27 09:46:08 OK 2_object_stats_trigger.sql (1.28ms)13262026/08/27 09:46:08 goose: up to current file version: 21327 client_integration_test.go:338: Created store path 2: /build/TestClientMultipleUploads92594044/001/store/dd80pb77va2pqaaqbz45pqawsvlakhkn-test-file-2.txt13282026-08-27 09:46:08.416 UTC [876] ERROR: relation "goose_db_version" does not exist at character 3613292026-08-27 09:46:08.416 UTC [876] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC13302026/08/27 09:46:08 WARN Failed to fetch cache config, uploading without size limit error="fetching cache config: server returned 404: 404 page not found\n"1331--- PASS: TestGCBugBareHashReferences (0.93s)1332=== CONT TestService_healthCheckHandler13332026-08-27 09:46:08.427 UTC [887] ERROR: relation "goose_db_version" does not exist at character 3613342026-08-27 09:46:08.427 UTC [887] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC13352026/08/27 09:46:08 OK 20241026095416_initial_model.sql (21.17ms)13362026/08/27 09:46:08 OK 20241026095416_initial_model.sql (13.5ms)13372026/08/27 09:46:08 OK 20251210153512_drop_unused_gin_index.sql (3.49ms)13382026/08/27 09:46:08 OK 20251210153512_drop_unused_gin_index.sql (3.13ms)13392026/08/27 09:46:08 OK 20251218171726_add_pins.sql (4.76ms)13402026/08/27 09:46:08 OK 20251218171726_add_pins.sql (4.75ms)13412026/08/27 09:46:08 INFO Received uploads request method=POST path=/api/pending_closures13422026/08/27 09:46:08 OK 20260628120000_add_object_size_and_stats.sql (4.77ms)13432026/08/27 09:46:08 goose: successfully migrated database to version: 2026062812000013442026/08/27 09:46:08 OK 20260628120000_add_object_size_and_stats.sql (4.73ms)13452026/08/27 09:46:08 goose: successfully migrated database to version: 2026062812000013462026/08/27 09:46:08 OK 1_commit_pending_closure.sql (6.55ms)13472026/08/27 09:46:08 OK 1_commit_pending_closure.sql (7.78ms)13482026/08/27 09:46:08 OK 2_object_stats_trigger.sql (3ms)13492026/08/27 09:46:08 goose: up to current file version: 213502026/08/27 09:46:08 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)13512026/08/27 09:46:08 INFO Uploading inlmp7k7rippskqwbinpb5dxdkrrj6n4-pinned-file.txt (128B)13522026/08/27 09:46:08 OK 2_object_stats_trigger.sql (3.32ms)13532026/08/27 09:46:08 goose: up to current file version: 213542026/08/27 09:46:08 WARN Failed to register uploaded object key=nar/1s7yghixzpinyqm7g59kyn3sav81pmnsdi5v63m6x4r2vacy907j.nar.zst error="server returned 404: 404 page not found\n"13552026/08/27 09:46:08 WARN Failed to register uploaded object key=log/ali3jffwdsplm9y9yfwbag0cqxghbhab-test-script.drv error="server returned 404: 404 page not found\n"13562026/08/27 09:46:08 WARN Failed to register uploaded object key=nar/0qmhsg42pqsz7x6hh02j8gznnac3yi3qpjhrqbwvz88i194jbb5s.nar.zst error="server returned 404: 404 page not found\n"1357--- PASS: TestReadProxyHead (1.88s)1358=== CONT TestGCTaskStore_CompletedAllowsNewTask1359--- PASS: TestGCTaskStore_CompletedAllowsNewTask (0.00s)13602026/08/27 09:46:08 WARN Failed to register uploaded object key=nzqwrxh360rs9x551m4jxqgcxhvfxxla.ls error="server returned 404: 404 page not found\n"1361=== CONT TestGCTaskStore_PhaseUpdates1362--- PASS: TestGCTaskStore_PhaseUpdates (0.00s)1363=== CONT TestMetricsInventory13642026/08/27 09:46:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign13652026/08/27 09:46:08 INFO Signed narinfos id=1 count=113662026/08/27 09:46:08 INFO Uploading 1 narinfos1367--- PASS: TestReadProxyNarStreaming (1.88s)13682026/08/27 09:46:08 WARN Failed to register uploaded object key=inlmp7k7rippskqwbinpb5dxdkrrj6n4.ls error="server returned 404: 404 page not found\n"1369=== CONT TestGracefulShutdownDrainsInflight13702026/08/27 09:46:08 INFO Starting HTTP server address=127.0.0.1:3279513712026/08/27 09:46:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign13722026/08/27 09:46:08 INFO Shutdown signal received, draining in-flight requests timeout=10s13732026/08/27 09:46:08 INFO Signed narinfos id=1 count=113742026/08/27 09:46:08 INFO Uploading 1 narinfos13752026/08/27 09:46:08 WARN Failed to register uploaded object key=nzqwrxh360rs9x551m4jxqgcxhvfxxla.narinfo error="server returned 404: 404 page not found\n"13762026/08/27 09:46:08 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete13772026/08/27 09:46:08 WARN Failed to register uploaded object key=inlmp7k7rippskqwbinpb5dxdkrrj6n4.narinfo error="server returned 404: 404 page not found\n"13782026/08/27 09:46:08 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete13792026-08-27 09:46:08.500 UTC [935] ERROR: relation "goose_db_version" does not exist at character 3613802026-08-27 09:46:08.500 UTC [935] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC13812026/08/27 09:46:08 INFO Completed upload id=113822026/08/27 09:46:08 INFO Upload complete. (157ms)1383=== NAME TestClientWithDependencies1384 client_integration_test.go:597: Skipping nix copy test - isolated store (/build/TestClientWithDependencies1407706732/001/store) requires matching store prefix13852026/08/27 09:46:08 INFO Completed upload id=113862026/08/27 09:46:08 INFO Upload complete. (130ms)13872026/08/27 09:46:08 INFO Received cleanup request method=DELETE path=/api/pending_closures1388--- PASS: TestClientWithDependencies (0.48s)1389=== CONT TestNARDeduplicationMetadataUploadBug13902026/08/27 09:46:08 INFO Aborted multipart uploads count=113912026/08/27 09:46:08 WARN Failed to fetch cache config, uploading without size limit error="fetching cache config: server returned 404: 404 page not found\n"13922026/08/27 09:46:08 OK 20241026095416_initial_model.sql (17.26ms)13932026/08/27 09:46:08 OK 20251210153512_drop_unused_gin_index.sql (5.22ms)1394--- PASS: TestMultipartCleanup (0.32s)1395=== CONT TestCacheConfigHandlerMaxNarSize1396--- PASS: TestCacheConfigHandlerMaxNarSize (0.00s)1397=== CONT TestGCTaskStore_Fail1398--- PASS: TestGCTaskStore_Fail (0.00s)1399=== CONT TestGenerateLandingPage14002026/08/27 09:46:08 INFO Aborted multipart uploads count=014012026/08/27 09:46:08 WARN mTLS auth: subject not in bound subjects subject="CN=reader"14022026/08/27 09:46:08 WARN mTLS auth: subject not in bound subjects subject="CN=writer"1403--- PASS: TestService_NativeMTLS (0.20s)1404=== CONT TestCreatePendingClosureRejectsOversizedNAR14052026/08/27 09:46:08 INFO Received uploads request method=POST path=/api/pending_closures1406--- PASS: TestCreatePendingClosureRejectsOversizedNAR (0.00s)1407=== CONT TestGCTaskStore_GetReturnsLatest1408--- PASS: TestGCTaskStore_GetReturnsLatest (0.00s)1409=== CONT TestIsValidUploadKey/narinfo1410=== CONT TestIsValidUploadKey/realisation_plus_in_output1411=== CONT TestIsValidUploadKey/unknown_type1412=== CONT TestIsValidUploadKey/empty_key1413=== CONT TestIsValidUploadKey/absolute1414=== CONT TestIsValidUploadKey/traversal_nar1415=== CONT TestIsValidUploadKey/traversal1416=== CONT TestIsValidUploadKey/listing_key,_narinfo_type1417=== CONT TestIsValidUploadKey/build_log_home-manager_file1418=== CONT TestIsValidUploadKey/nar_key,_narinfo_type1419=== CONT TestIsValidUploadKey/realisation1420=== CONT TestIsValidUploadKey/narinfo_key,_nar_type1421=== CONT TestIsValidUploadKey/build_log_equals1422=== CONT TestIsValidUploadKey/index.html1423=== CONT TestIsValidUploadKey/build_log_question_mark1424=== CONT TestIsValidUploadKey/nix-cache-info1425=== CONT TestIsValidUploadKey/build_log_plus_in_name1426=== CONT TestIsValidUploadKey/nar_plain1427=== CONT TestIsValidUploadKey/nar_xz1428=== CONT TestIsValidUploadKey/build_log1429=== CONT TestIsValidUploadKey/nar_zst1430=== CONT TestIsValidUploadKey/listing1431=== CONT TestUploadHandlersRejectOversizedBody/create_pending_closure14322026/08/27 09:46:08 INFO Received uploads request method=POST path=/1433--- PASS: TestIsValidUploadKey (0.00s)1434 --- PASS: TestIsValidUploadKey/narinfo (0.00s)1435 --- PASS: TestIsValidUploadKey/realisation_plus_in_output (0.00s)1436 --- PASS: TestIsValidUploadKey/unknown_type (0.00s)1437 --- PASS: TestIsValidUploadKey/empty_key (0.00s)1438 --- PASS: TestIsValidUploadKey/absolute (0.00s)1439 --- PASS: TestIsValidUploadKey/traversal_nar (0.00s)1440 --- PASS: TestIsValidUploadKey/traversal (0.00s)1441 --- PASS: TestIsValidUploadKey/listing_key,_narinfo_type (0.00s)1442 --- PASS: TestIsValidUploadKey/build_log_home-manager_file (0.00s)1443 --- PASS: TestIsValidUploadKey/nar_key,_narinfo_type (0.00s)1444 --- PASS: TestIsValidUploadKey/realisation (0.00s)1445 --- PASS: TestIsValidUploadKey/narinfo_key,_nar_type (0.00s)1446 --- PASS: TestIsValidUploadKey/build_log_equals (0.00s)1447 --- PASS: TestIsValidUploadKey/index.html (0.00s)1448 --- PASS: TestIsValidUploadKey/build_log_question_mark (0.00s)1449 --- PASS: TestIsValidUploadKey/nix-cache-info (0.00s)1450 --- PASS: TestIsValidUploadKey/build_log_plus_in_name (0.00s)1451 --- PASS: TestIsValidUploadKey/nar_plain (0.00s)1452 --- PASS: TestIsValidUploadKey/nar_xz (0.00s)1453 --- PASS: TestIsValidUploadKey/build_log (0.00s)1454 --- PASS: TestIsValidUploadKey/nar_zst (0.00s)1455 --- PASS: TestIsValidUploadKey/listing (0.00s)14562026/08/27 09:46:08 OK 20251218171726_add_pins.sql (4.14ms)14572026/08/27 09:46:08 WARN Force mode enabled - objects will be deleted immediately without grace period1458--- PASS: TestGenerateLandingPage (0.01s)1459=== CONT TestUploadHandlersRejectOversizedBody/request_more_parts14602026/08/27 09:46:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=014612026/08/27 09:46:08 INFO Received request for more parts method=POST path=/14622026/08/27 09:46:08 OK 20260628120000_add_object_size_and_stats.sql (6.28ms)14632026/08/27 09:46:08 goose: successfully migrated database to version: 2026062812000014642026/08/27 09:46:08 INFO Vacuumed table table=pending_closures14652026/08/27 09:46:08 INFO Vacuumed table table=pending_objects14662026/08/27 09:46:08 INFO Received uploads request method=POST path=/api/pending_closures14672026/08/27 09:46:08 INFO Vacuumed table table=multipart_uploads14682026/08/27 09:46:08 INFO Vacuumed table table=closures14692026/08/27 09:46:08 INFO Vacuumed table table=objects14702026/08/27 09:46:08 OK 1_commit_pending_closure.sql (5.09ms)14712026/08/27 09:46:08 OK 2_object_stats_trigger.sql (3.1ms)14722026/08/27 09:46:08 goose: up to current file version: 21473--- PASS: TestGCMetrics (1.60s)1474=== CONT TestUploadHandlersRejectOversizedBody/complete_multipart14752026/08/27 09:46:08 INFO Received complete multipart upload request method=POST path=/1476--- PASS: TestGracefulShutdownDrainsInflight (0.07s)1477=== CONT TestIsValidCachePath/narinfo1478=== CONT TestIsValidCachePath/index.html1479=== CONT TestIsValidCachePath/short_hash1480=== CONT TestIsValidCachePath/wrong_extension1481=== CONT TestIsValidCachePath/leading_slash1482=== CONT TestIsValidCachePath/empty1483=== CONT TestIsValidCachePath/random_path1484=== CONT TestIsValidCachePath/invalid_char_u1485=== CONT TestIsValidCachePath/invalid_char_e1486=== CONT TestIsValidCachePath/traversal_in_middle1487=== CONT TestIsValidCachePath/traversal_parent1488=== CONT TestIsValidCachePath/nar_uncompressed1489=== CONT TestIsValidCachePath/realisation1490=== CONT TestIsValidCachePath/log1491=== CONT TestIsValidCachePath/nix-cache-info1492=== CONT TestIsValidCachePath/ls1493=== CONT TestIsValidCachePath/nar_xz1494=== CONT TestIsValidCachePath/nar_bz21495=== CONT TestIsValidCachePath/narinfo_all_nix_base32_chars1496=== CONT TestIsValidCachePath/nar_zst1497--- PASS: TestIsValidCachePath (0.00s)1498 --- PASS: TestIsValidCachePath/narinfo (0.00s)1499 --- PASS: TestIsValidCachePath/index.html (0.00s)1500 --- PASS: TestIsValidCachePath/short_hash (0.00s)1501 --- PASS: TestIsValidCachePath/wrong_extension (0.00s)1502 --- PASS: TestIsValidCachePath/leading_slash (0.00s)1503 --- PASS: TestIsValidCachePath/empty (0.00s)1504 --- PASS: TestIsValidCachePath/random_path (0.00s)1505 --- PASS: TestIsValidCachePath/invalid_char_u (0.00s)1506 --- PASS: TestIsValidCachePath/invalid_char_e (0.00s)1507 --- PASS: TestIsValidCachePath/traversal_in_middle (0.00s)1508 --- PASS: TestIsValidCachePath/traversal_parent (0.00s)1509 --- PASS: TestIsValidCachePath/nar_uncompressed (0.00s)1510 --- PASS: TestIsValidCachePath/realisation (0.00s)1511 --- PASS: TestIsValidCachePath/log (0.00s)1512 --- PASS: TestIsValidCachePath/nix-cache-info (0.00s)1513 --- PASS: TestIsValidCachePath/ls (0.00s)1514 --- PASS: TestIsValidCachePath/nar_xz (0.00s)1515 --- PASS: TestIsValidCachePath/nar_bz2 (0.00s)1516 --- PASS: TestIsValidCachePath/narinfo_all_nix_base32_chars (0.00s)1517 --- PASS: TestIsValidCachePath/nar_zst (0.00s)1518=== CONT TestProxyWriteTimeout/narinfo1519=== CONT TestProxyWriteTimeout/10_GiB_nar1520=== CONT TestProxyWriteTimeout/unknown_size1521=== CONT TestProxyWriteTimeout/1_GiB_nar1522--- PASS: TestProxyWriteTimeout (0.00s)1523 --- PASS: TestProxyWriteTimeout/narinfo (0.00s)1524 --- PASS: TestProxyWriteTimeout/10_GiB_nar (0.00s)1525 --- PASS: TestProxyWriteTimeout/unknown_size (0.00s)1526 --- PASS: TestProxyWriteTimeout/1_GiB_nar (0.00s)1527=== CONT TestParseSingleRange/none1528=== CONT TestParseSingleRange/single_byte1529=== CONT TestParseSingleRange/suffix_exceeds_size1530=== CONT TestParseSingleRange/suffix1531=== CONT TestParseSingleRange/end_clamped_to_size1532=== CONT TestParseSingleRange/open-ended1533=== CONT TestParseSingleRange/closed1534=== CONT TestParseSingleRange/malformed_end_before_start1535=== CONT TestParseSingleRange/malformed_both_empty1536=== CONT TestParseSingleRange/malformed_no_dash1537=== CONT TestParseSingleRange/multi-range_ignored1538=== CONT TestParseSingleRange/unknown_unit1539=== CONT TestParseSingleRange/start_past_EOF1540=== CONT TestParseSingleRange/start_far_past_EOF1541--- PASS: TestParseSingleRange (0.00s)1542 --- PASS: TestParseSingleRange/none (0.00s)1543 --- PASS: TestParseSingleRange/single_byte (0.00s)1544 --- PASS: TestParseSingleRange/suffix_exceeds_size (0.00s)1545 --- PASS: TestParseSingleRange/suffix (0.00s)1546 --- PASS: TestParseSingleRange/end_clamped_to_size (0.00s)1547 --- PASS: TestParseSingleRange/open-ended (0.00s)1548 --- PASS: TestParseSingleRange/closed (0.00s)1549 --- PASS: TestParseSingleRange/malformed_end_before_start (0.00s)1550 --- PASS: TestParseSingleRange/malformed_both_empty (0.00s)1551 --- PASS: TestParseSingleRange/malformed_no_dash (0.00s)1552 --- PASS: TestParseSingleRange/multi-range_ignored (0.00s)1553 --- PASS: TestParseSingleRange/unknown_unit (0.00s)1554 --- PASS: TestParseSingleRange/start_past_EOF (0.00s)1555 --- PASS: TestParseSingleRange/start_far_past_EOF (0.00s)1556=== CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info15572026/08/27 09:46:08 INFO Received uploads request method=POST path=/1558=== CONT TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key15592026/08/27 09:46:08 INFO Received complete multipart upload request method=POST path=/1560=== CONT TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key15612026/08/27 09:46:08 INFO Received request for more parts method=POST path=/1562=== CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal15632026/08/27 09:46:08 INFO Received uploads request method=POST path=/1564--- PASS: TestUploadHandlersRejectInvalidKeys (0.00s)1565 --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info (0.00s)1566 --- PASS: TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key (0.00s)1567 --- PASS: TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key (0.00s)1568 --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal (0.00s)1569=== CONT TestCacheConfigHandler/full_config,_no_issuer1570=== CONT TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1571=== CONT TestCacheConfigHandler/no_signing_keys1572=== CONT TestCacheConfigHandler/no_cache_url_configured1573--- PASS: TestCacheConfigHandler (0.00s)1574 --- PASS: TestCacheConfigHandler/full_config,_no_issuer (0.00s)1575 --- PASS: TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator (0.00s)1576 --- PASS: TestCacheConfigHandler/no_signing_keys (0.00s)1577 --- PASS: TestCacheConfigHandler/no_cache_url_configured (0.00s)1578=== CONT TestService_AuthMiddleware_OIDC/valid_OIDC_token1579--- PASS: TestService_AuthMiddleware_MTLSProxyHeader (0.34s)1580=== CONT TestService_AuthMiddleware_OIDC/malformed_token_rejected15812026/08/27 09:46:08 INFO Received uploads request method=POST path=/api/pending_closures15822026/08/27 09:46:08 WARN Authentication failed token_preview=not-a-valid-jwt token_length=15 oidc_error="no provider could verify the token (signature or issuer mismatch)" oidc_provider="" tried_providers=[test]1583=== CONT TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1584=== CONT TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected15852026/08/27 09:46:08 INFO OIDC auth successful provider=test15862026/08/27 09:46:08 WARN Authentication failed token_preview=eyJhbGciOi...vj0njyVwzg token_length=702 oidc_error="bound claims validation failed: claim \"repository_owner\" value [otherorg] not in allowed patterns [myorg]" oidc_provider=test tried_providers=[test]1587=== CONT TestClientErrorHandling/InvalidStorePath1588=== CONT TestClientErrorHandling/ServerNotAvailable1589--- PASS: TestService_AuthMiddleware_OIDC (0.16s)1590 --- PASS: TestService_AuthMiddleware_OIDC/malformed_token_rejected (0.00s)1591 --- PASS: TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured (0.00s)1592 --- PASS: TestService_AuthMiddleware_OIDC/valid_OIDC_token (0.01s)1593 --- PASS: TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected (0.01s)1594--- PASS: TestService_healthCheckHandler (0.15s)1595=== CONT TestClientErrorHandling/InvalidAuthToken1596--- PASS: TestReadProxy404 (1.97s)1597=== CONT TestServerTLSConfig/no_client_CA1598=== CONT TestServerTLSConfig/not_a_PEM_file1599=== CONT TestServerTLSConfig/missing_CA_file1600--- PASS: TestServerTLSConfig (0.00s)1601 --- PASS: TestServerTLSConfig/no_client_CA (0.00s)1602 --- PASS: TestServerTLSConfig/not_a_PEM_file (0.00s)1603 --- PASS: TestServerTLSConfig/missing_CA_file (0.00s)16042026-08-27 09:46:08.580 UTC [1017] ERROR: relation "goose_db_version" does not exist at character 3616052026-08-27 09:46:08.580 UTC [1017] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC16062026/08/27 09:46:08 INFO Received uploads request method=POST path=/api/pending_closures16072026/08/27 09:46:08 INFO Received uploads request method=POST path=/api/pending_closures16082026/08/27 09:46:08 INFO Uploading 3 paths to 127.0.0.1 (0 already cached)16092026/08/27 09:46:08 INFO Uploading 35d8n1gji3anyh608df4l8nfgf2zqb5z-test-file-1.txt (160B)16102026/08/27 09:46:08 INFO Uploading dd80pb77va2pqaaqbz45pqawsvlakhkn-test-file-2.txt (160B)16112026/08/27 09:46:08 INFO Uploading 95h15v1a14lrg41ih843xsl2dkcz7kva-test-file-0.txt (160B)16122026/08/27 09:46:08 OK 20241026095416_initial_model.sql (13.14ms)16132026/08/27 09:46:08 WARN Failed to register uploaded object key=nar/1823g6vx4q5zqvr73m7lkq1wpk40fin7bsc792di430yk7sg4lka.nar.zst error="server returned 404: 404 page not found\n"16142026/08/27 09:46:08 WARN Failed to register uploaded object key=nar/13p9pqi0ga6fx2apcwls9hjnv7nmf740ggpn5zvfzdsp5vz2bbg4.nar.zst error="server returned 404: 404 page not found\n"16152026/08/27 09:46:08 WARN Failed to register uploaded object key=nar/0ahq3izw3h0b0c08p3xdfclh758fwxnp85ccvflasb4vnfsv50ri.nar.zst error="server returned 404: 404 page not found\n"16162026/08/27 09:46:08 WARN Failed to fetch cache config, uploading without size limit error="fetching cache config: server returned 404: 404 page not found\n"16172026-08-27 09:46:08.611 UTC [1057] ERROR: relation "goose_db_version" does not exist at character 3616182026-08-27 09:46:08.611 UTC [1057] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC16192026/08/27 09:46:08 OK 20251210153512_drop_unused_gin_index.sql (9.65ms)16202026/08/27 09:46:08 WARN Failed to register uploaded object key=dd80pb77va2pqaaqbz45pqawsvlakhkn.ls error="server returned 404: 404 page not found\n"16212026/08/27 09:46:08 WARN Failed to register uploaded object key=35d8n1gji3anyh608df4l8nfgf2zqb5z.ls error="server returned 404: 404 page not found\n"16222026/08/27 09:46:08 WARN Failed to register uploaded object key=95h15v1a14lrg41ih843xsl2dkcz7kva.ls error="server returned 404: 404 page not found\n"16232026/08/27 09:46:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/3/sign16242026/08/27 09:46:08 INFO Signed narinfos id=3 count=116252026/08/27 09:46:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign16262026/08/27 09:46:08 OK 20251218171726_add_pins.sql (8.19ms)16272026/08/27 09:46:08 INFO Signed narinfos id=1 count=116282026/08/27 09:46:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign16292026/08/27 09:46:08 INFO Signed narinfos id=2 count=116302026/08/27 09:46:08 INFO Uploading 3 narinfos1631=== NAME TestClientCADerivations1632 client_ca_test.go:136: Built CA derivation: /build/TestClientCADerivations1698391006/001/store/knhjj2575c7hmi6hgfnv3k59kk7nblf4-ca-test16332026/08/27 09:46:08 WARN Failed to register uploaded object key=95h15v1a14lrg41ih843xsl2dkcz7kva.narinfo error="server returned 404: 404 page not found\n"16342026/08/27 09:46:08 WARN Failed to register uploaded object key=dd80pb77va2pqaaqbz45pqawsvlakhkn.narinfo error="server returned 404: 404 page not found\n"16352026/08/27 09:46:08 WARN Failed to register uploaded object key=35d8n1gji3anyh608df4l8nfgf2zqb5z.narinfo error="server returned 404: 404 page not found\n"16362026/08/27 09:46:08 INFO Received complete upload request method=POST path=/api/pending_closures/3/complete16372026/08/27 09:46:08 OK 20260628120000_add_object_size_and_stats.sql (6.97ms)16382026/08/27 09:46:08 goose: successfully migrated database to version: 2026062812000016392026/08/27 09:46:08 OK 1_commit_pending_closure.sql (6.27ms)16402026/08/27 09:46:08 OK 20241026095416_initial_model.sql (14.27ms)1641=== NAME TestClientIntegration1642 client_integration_test.go:276: Created store path: /build/TestClientIntegration2601688988/002/store/pch9ip7cq0fyxipnkkqbr0f80h70rwv6-test-file.txt16432026/08/27 09:46:08 OK 2_object_stats_trigger.sql (5.13ms)16442026/08/27 09:46:08 goose: up to current file version: 216452026/08/27 09:46:08 INFO Completed upload id=316462026/08/27 09:46:08 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete16472026/08/27 09:46:08 OK 20251210153512_drop_unused_gin_index.sql (8.96ms)16482026/08/27 09:46:08 INFO Completed upload id=116492026/08/27 09:46:08 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete16502026/08/27 09:46:08 INFO Completed upload id=216512026/08/27 09:46:08 INFO Upload complete. (170ms)1652=== NAME TestClientMultipleUploads1653 client_integration_test.go:349: Uploaded 3 paths in 234.815674ms16542026/08/27 09:46:08 INFO Received uploads request method=POST path=/api/pending_closures16552026/08/27 09:46:08 OK 20251218171726_add_pins.sql (7.93ms)16562026/08/27 09:46:08 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)16572026/08/27 09:46:08 INFO Uploading acbfl0kgjyvbgaij6v7hgxwrm3qvg8g5-unpinned-file.txt (128B)16582026/08/27 09:46:08 OK 20260628120000_add_object_size_and_stats.sql (6.3ms)16592026/08/27 09:46:08 goose: successfully migrated database to version: 202606281200001660=== NAME TestClientCADerivations1661 client_ca_test.go:139: Found 1 dependencies (including self)16622026/08/27 09:46:08 OK 1_commit_pending_closure.sql (4.93ms)16632026/08/27 09:46:08 WARN Failed to register uploaded object key=nar/0gh0wdnr6gmk4r5sfzzh12isnl14rgmkrwy1ws41hpsrlndclg5h.nar.zst error="server returned 404: 404 page not found\n"1664--- PASS: TestClientMultipleUploads (0.55s)1665=== NAME TestOrphanedObjectsGC1666 orphaned_objects_gc_test.go:290: GC Test Summary:1667 orphaned_objects_gc_test.go:291: - Kept: 2 objects from closure A1668 orphaned_objects_gc_test.go:292: - Deleted: 2 objects from closure B1669 orphaned_objects_gc_test.go:293: - Deleted: 6 orphaned chain objects (X1->X2->X3)1670 orphaned_objects_gc_test.go:294: - Deleted: 2 orphaned single objects (Y)1671 orphaned_objects_gc_test.go:295: - Total deleted: 10 objects1672--- PASS: TestOrphanedObjectsGC (1.72s)16732026/08/27 09:46:08 OK 2_object_stats_trigger.sql (9.18ms)16742026/08/27 09:46:08 goose: up to current file version: 216752026/08/27 09:46:08 WARN Failed to register uploaded object key=acbfl0kgjyvbgaij6v7hgxwrm3qvg8g5.ls error="server returned 404: 404 page not found\n"16762026/08/27 09:46:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign1677--- PASS: TestResurrectedObjectNotDeleted (1.92s)16782026/08/27 09:46:08 INFO Signed narinfos id=2 count=116792026/08/27 09:46:08 INFO Uploading 1 narinfos16802026-08-27 09:46:08.677 UTC [1168] ERROR: relation "goose_db_version" does not exist at character 3616812026-08-27 09:46:08.677 UTC [1168] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC16822026/08/27 09:46:08 WARN Failed to register uploaded object key=acbfl0kgjyvbgaij6v7hgxwrm3qvg8g5.narinfo error="server returned 404: 404 page not found\n"16832026/08/27 09:46:08 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete16842026/08/27 09:46:08 INFO Received complete multipart upload request method=POST path=/api/multipart/complete16852026/08/27 09:46:08 INFO Completed upload id=216862026/08/27 09:46:08 INFO Upload complete. (134ms)1687--- PASS: TestMetricsInventory (0.20s)16882026-08-27 09:46:08.684 UTC [1170] ERROR: relation "goose_db_version" does not exist at character 3616892026-08-27 09:46:08.684 UTC [1170] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC16902026/08/27 09:46:08 OK 20241026095416_initial_model.sql (11.57ms)16912026/08/27 09:46:08 OK 20251210153512_drop_unused_gin_index.sql (4.83ms)16922026/08/27 09:46:08 OK 20241026095416_initial_model.sql (10.7ms)16932026/08/27 09:46:08 OK 20251210153512_drop_unused_gin_index.sql (1.8ms)16942026/08/27 09:46:08 OK 20251218171726_add_pins.sql (3.8ms)16952026/08/27 09:46:08 OK 20251218171726_add_pins.sql (5.2ms)16962026/08/27 09:46:08 OK 20260628120000_add_object_size_and_stats.sql (4.83ms)16972026/08/27 09:46:08 goose: successfully migrated database to version: 2026062812000016982026/08/27 09:46:08 WARN Failed to fetch cache config, uploading without size limit error="fetching cache config: server returned 404: 404 page not found\n"16992026/08/27 09:46:08 INFO Received create pin request method=POST path=/api/pins/myapp17002026/08/27 09:46:08 OK 1_commit_pending_closure.sql (2.93ms)17012026/08/27 09:46:08 OK 20260628120000_add_object_size_and_stats.sql (3.71ms)17022026/08/27 09:46:08 goose: successfully migrated database to version: 2026062812000017032026/08/27 09:46:08 WARN Request failed, retrying attempt=1 max_attempts=6 backoff=100ms error="Get \"http://localhost:19999/api/cache-config\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/cache-config17042026/08/27 09:46:08 OK 2_object_stats_trigger.sql (2.7ms)17052026/08/27 09:46:08 goose: up to current file version: 217062026/08/27 09:46:08 OK 1_commit_pending_closure.sql (2.91ms)17072026/08/27 09:46:08 OK 2_object_stats_trigger.sql (1.01ms)17082026/08/27 09:46:08 goose: up to current file version: 217092026/08/27 09:46:08 INFO Created/updated pin name=myapp store_path=/build/TestPinProtectsFromGC2135796416/001/store/inlmp7k7rippskqwbinpb5dxdkrrj6n4-pinned-file.txt narinfo_key=inlmp7k7rippskqwbinpb5dxdkrrj6n4.narinfo17102026/08/27 09:46:08 INFO Starting cleanup of old closures method=DELETE path=/api/closures17112026/08/27 09:46:08 INFO Garbage collection started1712=== NAME TestNARDeduplicationMetadataUploadBug1713 metadata_upload_test.go:48: First store path: /build/TestNARDeduplicationMetadataUploadBug215363896/001/store/r5ghlmpkzdnpfwkl6lpfp95ljpfbh3p9-file1.txt17142026/08/27 09:46:08 WARN Failed to fetch cache config, uploading without size limit error="fetching cache config: server returned 404: 404 page not found\n"17152026/08/27 09:46:08 INFO Aborted multipart uploads count=017162026/08/27 09:46:08 WARN Force mode enabled - objects will be deleted immediately without grace period17172026/08/27 09:46:08 INFO Received cleanup request method=DELETE path=/api/pending_closures17182026/08/27 09:46:08 INFO Aborted multipart uploads count=017192026/08/27 09:46:08 INFO Received uploads request method=POST path=/api/pending_closures17202026/08/27 09:46:08 INFO Received uploads request method=POST path=/api/pending_closures17212026/08/27 09:46:08 INFO Received cleanup request method=DELETE path=/api/pending_closures17222026/08/27 09:46:08 INFO Aborted multipart uploads count=117232026/08/27 09:46:08 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete17242026/08/27 09:46:08 INFO Received uploads request method=POST path=/api/pending_closures1725--- PASS: TestReadProxyInvalidPath (2.16s)17262026-08-27 09:46:08.769 UTC [609] ERROR: Closure does not exist: id=117272026-08-27 09:46:08.769 UTC [609] CONTEXT: PL/pgSQL function commit_pending_closure(bigint) line 16 at RAISE17282026-08-27 09:46:08.769 UTC [609] STATEMENT: -- name: CommitPendingClosure :exec1729 SELECT commit_pending_closure($1::bigint)1730 1731--- PASS: TestService_cleanupPendingClosuresHandler (2.17s)17322026/08/27 09:46:08 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)17332026/08/27 09:46:08 INFO Uploading pch9ip7cq0fyxipnkkqbr0f80h70rwv6-test-file.txt (152B)17342026/08/27 09:46:08 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)17352026/08/27 09:46:08 INFO Uploading knhjj2575c7hmi6hgfnv3k59kk7nblf4-ca-test (144B)17362026/08/27 09:46:08 WARN Failed to register uploaded object key=nar/11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1.nar.zst error="server returned 404: 404 page not found\n"17372026/08/27 09:46:08 WARN Failed to register uploaded object key=nar/04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n.nar.zst error="server returned 404: 404 page not found\n"17382026/08/27 09:46:08 WARN Failed to register uploaded object key=pch9ip7cq0fyxipnkkqbr0f80h70rwv6.ls error="server returned 404: 404 page not found\n"17392026/08/27 09:46:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign17402026/08/27 09:46:08 WARN Failed to register uploaded object key=log/nn07jwabakdyvs0kl4l204khv2shpv8d-ca-test.drv error="server returned 404: 404 page not found\n"17412026/08/27 09:46:08 INFO Signed narinfos id=1 count=117422026/08/27 09:46:08 INFO Uploading 1 narinfos17432026/08/27 09:46:08 WARN Failed to register uploaded object key=knhjj2575c7hmi6hgfnv3k59kk7nblf4.ls error="server returned 404: 404 page not found\n"17442026/08/27 09:46:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign17452026/08/27 09:46:08 INFO Signed narinfos id=1 count=117462026/08/27 09:46:08 INFO Uploading 1 narinfos17472026/08/27 09:46:08 WARN Failed to register uploaded object key=pch9ip7cq0fyxipnkkqbr0f80h70rwv6.narinfo error="server returned 404: 404 page not found\n"17482026/08/27 09:46:08 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete17492026/08/27 09:46:08 WARN Failed to register uploaded object key=knhjj2575c7hmi6hgfnv3k59kk7nblf4.narinfo error="server returned 404: 404 page not found\n"17502026/08/27 09:46:08 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete17512026/08/27 09:46:08 INFO Completed upload id=117522026/08/27 09:46:08 INFO Upload complete. (118ms)1753=== NAME TestClientIntegration1754 client_integration_test.go:292: Retrieved narinfo from S3:1755 StorePath: /build/TestClientIntegration2601688988/002/store/pch9ip7cq0fyxipnkkqbr0f80h70rwv6-test-file.txt1756 URL: nar/11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1.nar.zst1757 Compression: zstd1758 NarHash: sha256:11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk11759 NarSize: 1521760 References: 1761 CA: fixed:r:sha256:11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk117622026/08/27 09:46:08 INFO Completed upload id=117632026/08/27 09:46:08 INFO Upload complete. (97ms)17642026/08/27 09:46:08 WARN Failed to fetch cache config, uploading without size limit error="fetching cache config: server returned 404: 404 page not found\n"1765 client_integration_test.go:293: Retrieved .ls file from S3 (compressed size: 77 bytes)1766 client_integration_test.go:293: Decompressed .ls content (64 bytes):1767 {"version":1,"root":{"type":"regular","size":39,"narOffset":96}}1768 client_integration_test.go:296: Testing garbage collection...1769=== NAME TestClientCADerivations1770 client_ca_test.go:180: Narinfo contains CA field: StorePath: /build/TestClientCADerivations1698391006/001/store/knhjj2575c7hmi6hgfnv3k59kk7nblf4-ca-test1771 URL: nar/04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n.nar.zst1772 Compression: zstd1773 NarHash: sha256:04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n1774 NarSize: 1441775 References: 1776 Deriver: /build/TestClientCADerivations1698391006/001/store/nn07jwabakdyvs0kl4l204khv2shpv8d-ca-test.drv1777 CA: fixed:r:sha256:04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n1778 client_ca_test.go:185: Checking for realisation files in S3...1779 client_ca_test.go:207: No realisation files uploaded - this is expected for locally built CA derivations1780 client_ca_test.go:208: Locally built CA derivations don't have DrvOutput IDs until they're substituted from a cache17812026/08/27 09:46:08 WARN Request failed, retrying attempt=2 max_attempts=6 backoff=199.211992ms error="Get \"http://localhost:19999/api/cache-config\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/cache-config17822026/08/27 09:46:08 INFO Received uploads request method=POST path=/api/pending_closures17832026/08/27 09:46:08 INFO Starting cleanup of old closures method=DELETE path=/api/closures17842026/08/27 09:46:08 INFO Garbage collection started17852026/08/27 09:46:08 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)17862026/08/27 09:46:08 INFO Uploading r5ghlmpkzdnpfwkl6lpfp95ljpfbh3p9-file1.txt (160B)17872026/08/27 09:46:08 WARN Failed to register uploaded object key=nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst error="server returned 404: 404 page not found\n"17882026/08/27 09:46:08 WARN Failed to register uploaded object key=r5ghlmpkzdnpfwkl6lpfp95ljpfbh3p9.ls error="server returned 404: 404 page not found\n"17892026/08/27 09:46:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign17902026/08/27 09:46:08 INFO Aborted multipart uploads count=017912026/08/27 09:46:08 INFO Signed narinfos id=1 count=117922026/08/27 09:46:08 INFO Uploading 1 narinfos17932026/08/27 09:46:08 WARN Force mode enabled - objects will be deleted immediately without grace period17942026/08/27 09:46:08 WARN Failed to register uploaded object key=r5ghlmpkzdnpfwkl6lpfp95ljpfbh3p9.narinfo error="server returned 404: 404 page not found\n"17952026/08/27 09:46:08 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete17962026/08/27 09:46:08 INFO Completed upload id=117972026/08/27 09:46:08 INFO Upload complete. (91ms)1798=== NAME TestNARDeduplicationMetadataUploadBug1799 metadata_upload_test.go:54: Retrieved narinfo from S3:1800 StorePath: /build/TestNARDeduplicationMetadataUploadBug215363896/001/store/r5ghlmpkzdnpfwkl6lpfp95ljpfbh3p9-file1.txt1801 URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst1802 Compression: zstd1803 NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1804 NarSize: 1601805 References: 1806 CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1807 metadata_upload_test.go:55: Retrieved .ls file from S3 (compressed size: 77 bytes)1808 metadata_upload_test.go:55: Decompressed .ls content (64 bytes):1809 {"version":1,"root":{"type":"regular","size":44,"narOffset":96}}18102026/08/27 09:46:08 INFO Received complete multipart upload request method=POST path=/api/multipart/complete1811 metadata_upload_test.go:64: Second store path (same content): /build/TestNARDeduplicationMetadataUploadBug215363896/001/store/dc0khxyaj0vwm623m52anpl7hvjym7a0-file2.txt18122026/08/27 09:46:08 INFO Completed multipart upload object_key=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhh0100000000000000000000.nar.zst upload_id=ZDc1NjU4ZjQtODU1My00NmQ5LWExMGItYjU3ZmVhODBhNTljLjZhMmE3ZjViLTYxZGYtNDIzMy04ZjdlLWI2MDVhNjM2MGQ5NngxNzg3ODIzOTY4MDE3MDYzNjg5 parts=1218132026/08/27 09:46:08 INFO Received uploads request method=POST path=/api/pending_closures1814--- PASS: TestCompletedNarNotReofferedAcrossClosures (2.31s)18152026/08/27 09:46:08 WARN Failed to fetch cache config, uploading without size limit error="fetching cache config: server returned 404: 404 page not found\n"1816=== NAME TestClientCADerivations1817 client_ca_test.go:258: nix copy output: warning: you don't have Internet access; disabling some network-dependent features1818 warning: failed to create TLS context for AWS credential providers; SSO, STS WebIdentity, and ECS container authentication will be unavailable1819 error: binary cache 's3://bucket41?endpoint=http://localhost:33169®ion=eu-west-1' is for Nix stores with prefix '/nix/store', not '/build/TestClientCADerivations1698391006/001/store'1820 client_ca_test.go:259: nix copy failed (might be expected with isolated stores): exit status 11821--- PASS: TestClientCADerivations (0.70s)18222026/08/27 09:46:08 INFO Received uploads request method=POST path=/api/pending_closures18232026/08/27 09:46:09 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)18242026/08/27 09:46:09 WARN Failed to register uploaded object key=dc0khxyaj0vwm623m52anpl7hvjym7a0.ls error="server returned 404: 404 page not found\n"18252026/08/27 09:46:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign18262026/08/27 09:46:09 INFO Signed narinfos id=2 count=118272026/08/27 09:46:09 INFO Uploading 1 narinfos18282026/08/27 09:46:09 WARN Failed to register uploaded object key=dc0khxyaj0vwm623m52anpl7hvjym7a0.narinfo error="server returned 404: 404 page not found\n"18292026/08/27 09:46:09 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete18302026/08/27 09:46:09 INFO Completed upload id=218312026/08/27 09:46:09 INFO Upload complete. (80ms)1832=== NAME TestNARDeduplicationMetadataUploadBug1833 metadata_upload_test.go:76: Retrieved narinfo from S3:1834 StorePath: /build/TestNARDeduplicationMetadataUploadBug215363896/001/store/dc0khxyaj0vwm623m52anpl7hvjym7a0-file2.txt1835 URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst1836 Compression: zstd1837 NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1838 NarSize: 1601839 References: 1840 CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1841 metadata_upload_test.go:77: Retrieved .ls file from S3 (compressed size: 62 bytes)1842 metadata_upload_test.go:77: Decompressed .ls content (49 bytes):1843 {"version":1,"root":{"type":"regular","size":44}}18442026/08/27 09:46:09 WARN Request failed, retrying attempt=3 max_attempts=6 backoff=422.061186ms error="Get \"http://localhost:19999/api/cache-config\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/cache-config1845--- PASS: TestNARDeduplicationMetadataUploadBug (0.51s)1846--- PASS: TestReadProxyDisabled (2.61s)18472026/08/27 09:46:09 WARN Failed to fetch cache config, uploading without size limit error="fetching cache config: server returned 404: 404 page not found\n"18482026/08/27 09:46:09 WARN Authentication failed token_preview=invalid-token token_length=13 reason="static API token mismatch"18492026/08/27 09:46:09 INFO Received complete multipart upload request method=POST path=/api/multipart/complete18502026/08/27 09:46:09 WARN Request failed, retrying attempt=4 max_attempts=6 backoff=807.978461ms error="Get \"http://localhost:19999/api/cache-config\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/cache-config18512026/08/27 09:46:09 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst upload_id=ZDc1NjU4ZjQtODU1My00NmQ5LWExMGItYjU3ZmVhODBhNTljLmQyMTk5NDk5LTVlZjktNDgxOC1hZmNhLTg3NjYyMGRjMGRlNngxNzg3ODIzOTY3NTE5NTU4Njcx parts=1018522026/08/27 09:46:09 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete18532026/08/27 09:46:09 INFO Completed upload id=118542026/08/27 09:46:09 INFO Received uploads request method=POST path=/api/pending_closures18552026/08/27 09:46:09 INFO Received uploads request method=POST path=/api/pending_closures18562026/08/27 09:46:09 INFO Object in database but missing from S3 key=dadb44fdadb44fdadb44fdadb44f0000.narinfo18572026/08/27 09:46:09 WARN Found objects in DB but missing from S3, will re-upload count=11858--- PASS: TestService_verifyS3Integrity (2.85s)18592026/08/27 09:46:09 INFO Received complete multipart upload request method=POST path=/api/multipart/complete18602026/08/27 09:46:09 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0100000000000000000000.nar.zst upload_id=ZDc1NjU4ZjQtODU1My00NmQ5LWExMGItYjU3ZmVhODBhNTljLjkwOTc3YjA0LTBkMWMtNDhjYi05YzBlLTQwYjQ0MTMzZDRlYXgxNzg3ODIzOTY2OTYzNTU3NTEy parts=121861--- PASS: TestRedundantMultipartUpload (2.90s)1862--- PASS: TestUploadHandlersRejectOversizedBody (0.14s)1863 --- PASS: TestUploadHandlersRejectOversizedBody/request_more_parts (0.07s)1864 --- PASS: TestUploadHandlersRejectOversizedBody/complete_multipart (0.10s)1865 --- PASS: TestUploadHandlersRejectOversizedBody/create_pending_closure (1.37s)18662026/08/27 09:46:09 INFO Received complete multipart upload request method=POST path=/api/multipart/complete18672026/08/27 09:46:09 INFO Completed multipart upload object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst upload_id=ZDc1NjU4ZjQtODU1My00NmQ5LWExMGItYjU3ZmVhODBhNTljLmYyOWQ3Y2FjLWZjZDQtNDBhNy1hMjA4LWE5MjAzMzNjZGU3NXgxNzg3ODIzOTY4MTcwOTc2MDYy parts=1018682026/08/27 09:46:09 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete18692026/08/27 09:46:09 INFO Completed upload id=118702026/08/27 09:46:09 INFO Received get closure request method=GET path=/api/closures/0000000000000000000000000000000018712026/08/27 09:46:09 INFO Received uploads request method=POST path=/api/pending_closures18722026/08/27 09:46:09 INFO Starting cleanup of old closures method=DELETE path=/api/closures18732026/08/27 09:46:09 INFO Aborted multipart uploads count=018742026/08/27 09:46:09 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=2 objects-deleted-after-grace-period=0 objects-failed-to-delete=018752026/08/27 09:46:09 INFO Vacuumed table table=pending_closures18762026/08/27 09:46:09 INFO Vacuumed table table=pending_objects18772026/08/27 09:46:09 INFO Vacuumed table table=multipart_uploads18782026/08/27 09:46:09 INFO Vacuumed table table=closures18792026/08/27 09:46:09 INFO Vacuumed table table=objects18802026/08/27 09:46:09 INFO Received get closure request method=GET path=/api/closures/000000000000000000000000000000001881--- PASS: TestService_createPendingClosureHandler (3.38s)18822026/08/27 09:46:10 WARN Request failed, retrying attempt=5 max_attempts=6 backoff=1.566158153s error="Get \"http://localhost:19999/api/cache-config\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/cache-config18832026/08/27 09:46:10 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=2001 objects-failed-to-delete=018842026/08/27 09:46:10 INFO Vacuumed table table=pending_closures18852026/08/27 09:46:10 INFO Vacuumed table table=pending_objects18862026/08/27 09:46:10 INFO Vacuumed table table=multipart_uploads18872026/08/27 09:46:10 INFO Vacuumed table table=closures18882026/08/27 09:46:10 INFO Vacuumed table table=objects1889=== NAME TestOrphanedObjectsGCStressTest1890 orphaned_objects_gc_test.go:425: Created 10 active closures, 5 to-delete closures, 20 orphaned chains18912026/08/27 09:46:10 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=2001 objects-failed-to-delete=018922026/08/27 09:46:10 INFO Vacuumed table table=pending_closures18932026/08/27 09:46:10 INFO Vacuumed table table=pending_objects18942026/08/27 09:46:10 INFO Vacuumed table table=multipart_uploads18952026/08/27 09:46:10 INFO Vacuumed table table=closures18962026/08/27 09:46:10 INFO Vacuumed table table=objects1897 orphaned_objects_gc_test.go:446: Marked 210 objects for deletion18982026/08/27 09:46:10 INFO Garbage collection progress phase="" failed_uploads_deleted=0 old_closures_deleted=1 objects_marked=3 objects_deleted=2001 objects_failed=01899=== NAME TestPinProtectsFromGC1900 client_integration_test.go:709: Pin successfully protected closure from garbage collection1901--- PASS: TestPinProtectsFromGC (3.22s)19022026/08/27 09:46:10 INFO Garbage collection progress phase="" failed_uploads_deleted=0 old_closures_deleted=1 objects_marked=3 objects_deleted=2001 objects_failed=01903=== NAME TestClientIntegration1904 client_integration_test.go:303: Objects in database after GC:1905 client_integration_test.go:303: Successfully deleted all objects with GC --force1906--- PASS: TestClientIntegration (2.64s)1907=== NAME TestOrphanedObjectsGCStressTest1908 orphaned_objects_gc_test.go:509: Stress test completed successfully:1909 orphaned_objects_gc_test.go:510: - Active objects preserved: 201910 orphaned_objects_gc_test.go:511: - Objects deleted: 2101911 orphaned_objects_gc_test.go:512: - Total GC'd: 2101912--- PASS: TestOrphanedObjectsGCStressTest (4.62s)19132026/08/27 09:46:11 WARN Failed to fetch cache config, uploading without size limit error="fetching cache config: sending request: request failed after retries: Get \"http://localhost:19999/api/cache-config\": dial tcp [::1]:19999: connect: connection refused"19142026/08/27 09:46:11 WARN Request failed, retrying attempt=1 max_attempts=6 backoff=100ms error="Post \"http://localhost:19999/api/pending_closures\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/pending_closures19152026/08/27 09:46:11 WARN Request failed, retrying attempt=2 max_attempts=6 backoff=190.887622ms error="Post \"http://localhost:19999/api/pending_closures\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/pending_closures19162026/08/27 09:46:12 WARN Request failed, retrying attempt=3 max_attempts=6 backoff=362.025912ms error="Post \"http://localhost:19999/api/pending_closures\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/pending_closures19172026/08/27 09:46:12 WARN Request failed, retrying attempt=4 max_attempts=6 backoff=731.224584ms error="Post \"http://localhost:19999/api/pending_closures\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/pending_closures19182026/08/27 09:46:12 WARN Rate limiter enabled after throttle name=s3-test rate=519192026/08/27 09:46:12 WARN S3 rate limit hit operation="complete multipart upload" error="Please reduce your request rate."1920=== NAME TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle1921 throttle_test.go:213: Proxy stats: total=15, throttled=10, completeMultipart=101922 throttle_test.go:215: Rate limiter: enabled=true, rate=5.001923--- PASS: TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle (4.46s)19242026/08/27 09:46:13 WARN Request failed, retrying attempt=5 max_attempts=6 backoff=1.477659077s error="Post \"http://localhost:19999/api/pending_closures\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/pending_closures1925--- PASS: TestClientErrorHandling (0.00s)1926 --- PASS: TestClientErrorHandling/InvalidStorePath (0.21s)1927 --- PASS: TestClientErrorHandling/InvalidAuthToken (0.73s)1928 --- PASS: TestClientErrorHandling/ServerNotAvailable (6.16s)1929PASS1930{"timestamp":"2026-08-27T09:46:14.728485267Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:54744","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1836,"threadName":"rustfs-worker","threadId":"ThreadId(290)"}19312026-08-27 09:46:15.011 UTC [110] LOG: received smart shutdown request19322026-08-27 09:46:15.017 UTC [110] LOG: background worker "logical replication launcher" (PID 120) exited with exit code 119332026-08-27 09:46:15.027 UTC [115] LOG: shutting down19342026-08-27 09:46:15.027 UTC [115] LOG: checkpoint starting: shutdown immediate19352026-08-27 09:46:16.145 UTC [115] LOG: checkpoint complete: wrote 11572 buffers (70.6%), wrote 3 SLRU buffers; 0 WAL file(s) added, 0 removed, 13 recycled; write=0.225 s, sync=0.884 s, total=1.119 s; sync files=15825, longest=0.002 s, average=0.001 s; distance=217977 kB, estimate=217977 kB; lsn=0/EC422D8, redo lsn=0/EC422D819362026-08-27 09:46:16.242 UTC [110] LOG: database system is shut down1937Running OIDC tests...1938=== RUN TestGlobMatch1939=== PAUSE TestGlobMatch1940=== RUN TestAudienceForIssuer1941=== PAUSE TestAudienceForIssuer1942=== RUN TestValidateToken_ValidToken1943=== PAUSE TestValidateToken_ValidToken1944=== RUN TestValidateToken_WrongAudience1945=== PAUSE TestValidateToken_WrongAudience1946=== RUN TestValidateToken_Expired1947=== PAUSE TestValidateToken_Expired1948=== RUN TestValidateToken_BoundClaimsMismatch1949=== PAUSE TestValidateToken_BoundClaimsMismatch1950=== RUN TestValidateToken_BoundSubjectMismatch1951=== PAUSE TestValidateToken_BoundSubjectMismatch1952=== RUN TestValidateToken_MultipleProviders1953=== PAUSE TestValidateToken_MultipleProviders1954=== RUN TestValidateToken_NoMatchingProvider1955=== PAUSE TestValidateToken_NoMatchingProvider1956=== CONT TestGlobMatch1957=== RUN TestGlobMatch/foo_foo1958=== CONT TestValidateToken_BoundClaimsMismatch1959=== CONT TestValidateToken_MultipleProviders1960=== CONT TestValidateToken_WrongAudience1961=== PAUSE TestGlobMatch/foo_foo1962=== RUN TestGlobMatch/foo_bar1963=== PAUSE TestGlobMatch/foo_bar1964=== RUN TestGlobMatch/*_1965=== PAUSE TestGlobMatch/*_1966=== RUN TestGlobMatch/*_anything1967=== PAUSE TestGlobMatch/*_anything1968=== RUN TestGlobMatch/foo*_foo1969=== PAUSE TestGlobMatch/foo*_foo1970=== CONT TestValidateToken_ValidToken1971=== CONT TestAudienceForIssuer1972=== CONT TestValidateToken_Expired1973=== CONT TestValidateToken_BoundSubjectMismatch1974=== CONT TestValidateToken_NoMatchingProvider1975=== RUN TestGlobMatch/foo*_foobar1976=== PAUSE TestGlobMatch/foo*_foobar1977=== RUN TestGlobMatch/foo*_bar1978=== PAUSE TestGlobMatch/foo*_bar1979=== RUN TestGlobMatch/*bar_bar1980=== PAUSE TestGlobMatch/*bar_bar1981=== RUN TestGlobMatch/*bar_foobar1982=== PAUSE TestGlobMatch/*bar_foobar1983=== RUN TestGlobMatch/*bar_foo1984=== PAUSE TestGlobMatch/*bar_foo1985=== RUN TestGlobMatch/foo*bar_foobar1986--- PASS: TestAudienceForIssuer (0.00s)1987=== PAUSE TestGlobMatch/foo*bar_foobar1988=== RUN TestGlobMatch/foo*bar_foo123bar1989=== PAUSE TestGlobMatch/foo*bar_foo123bar1990=== RUN TestGlobMatch/foo*bar_foobarbaz1991=== PAUSE TestGlobMatch/foo*bar_foobarbaz1992=== RUN TestGlobMatch/*/*_foo/bar1993=== PAUSE TestGlobMatch/*/*_foo/bar1994=== RUN TestGlobMatch/*/*_foo1995=== PAUSE TestGlobMatch/*/*_foo1996=== RUN TestGlobMatch/refs/heads/*_refs/heads/main1997=== PAUSE TestGlobMatch/refs/heads/*_refs/heads/main1998=== RUN TestGlobMatch/refs/heads/*_refs/tags/v1.01999=== PAUSE TestGlobMatch/refs/heads/*_refs/tags/v1.02000=== RUN TestGlobMatch/refs/*/main_refs/heads/main2001=== PAUSE TestGlobMatch/refs/*/main_refs/heads/main2002=== RUN TestGlobMatch/fo?_foo2003=== PAUSE TestGlobMatch/fo?_foo2004=== RUN TestGlobMatch/fo?_fo2005=== PAUSE TestGlobMatch/fo?_fo2006=== RUN TestGlobMatch/fo?_fooo2007=== PAUSE TestGlobMatch/fo?_fooo2008=== RUN TestGlobMatch/?oo_foo2009=== PAUSE TestGlobMatch/?oo_foo2010=== RUN TestGlobMatch/?oo_boo2011=== PAUSE TestGlobMatch/?oo_boo2012=== RUN TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main2013=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main2014=== RUN TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main2015=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main2016=== CONT TestGlobMatch/foo_foo2017=== CONT TestGlobMatch/fo?_fooo2018=== CONT TestGlobMatch/foo*bar_foobar2019=== CONT TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main2020=== CONT TestGlobMatch/*_2021=== CONT TestGlobMatch/fo?_fo2022=== CONT TestGlobMatch/*/*_foo/bar2023=== CONT TestGlobMatch/*/*_foo2024=== CONT TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main2025=== CONT TestGlobMatch/foo*bar_foo123bar2026=== CONT TestGlobMatch/*bar_foo2027=== CONT TestGlobMatch/*bar_foobar2028=== CONT TestGlobMatch/?oo_boo2029=== CONT TestGlobMatch/*bar_bar2030=== CONT TestGlobMatch/foo*_bar2031=== CONT TestGlobMatch/foo*_foobar2032=== CONT TestGlobMatch/foo*_foo2033=== CONT TestGlobMatch/*_anything2034=== CONT TestGlobMatch/foo_bar2035=== CONT TestGlobMatch/?oo_foo2036=== CONT TestGlobMatch/refs/heads/*_refs/heads/main2037=== CONT TestGlobMatch/fo?_foo2038=== CONT TestGlobMatch/refs/*/main_refs/heads/main2039=== CONT TestGlobMatch/refs/heads/*_refs/tags/v1.02040=== CONT TestGlobMatch/foo*bar_foobarbaz2041--- PASS: TestGlobMatch (0.00s)2042 --- PASS: TestGlobMatch/foo_foo (0.00s)2043 --- PASS: TestGlobMatch/fo?_fooo (0.00s)2044 --- PASS: TestGlobMatch/foo*bar_foobar (0.00s)2045 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main (0.00s)2046 --- PASS: TestGlobMatch/*_ (0.00s)2047 --- PASS: TestGlobMatch/fo?_fo (0.00s)2048 --- PASS: TestGlobMatch/*/*_foo/bar (0.00s)2049 --- PASS: TestGlobMatch/*/*_foo (0.00s)2050 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main (0.00s)2051 --- PASS: TestGlobMatch/foo*bar_foo123bar (0.00s)2052 --- PASS: TestGlobMatch/*bar_foo (0.00s)2053 --- PASS: TestGlobMatch/*bar_foobar (0.00s)2054 --- PASS: TestGlobMatch/?oo_boo (0.00s)2055 --- PASS: TestGlobMatch/*bar_bar (0.00s)2056 --- PASS: TestGlobMatch/foo*_bar (0.00s)2057 --- PASS: TestGlobMatch/foo*_foobar (0.00s)2058 --- PASS: TestGlobMatch/foo*_foo (0.00s)2059 --- PASS: TestGlobMatch/*_anything (0.00s)2060 --- PASS: TestGlobMatch/foo_bar (0.00s)2061 --- PASS: TestGlobMatch/?oo_foo (0.00s)2062 --- PASS: TestGlobMatch/refs/heads/*_refs/heads/main (0.00s)2063 --- PASS: TestGlobMatch/fo?_foo (0.00s)2064 --- PASS: TestGlobMatch/refs/*/main_refs/heads/main (0.00s)2065 --- PASS: TestGlobMatch/refs/heads/*_refs/tags/v1.0 (0.00s)2066 --- PASS: TestGlobMatch/foo*bar_foobarbaz (0.00s)20672026/08/27 09:46:17 INFO OIDC provider initialized name=provider120682026/08/27 09:46:17 INFO OIDC provider initialized name=test20692026/08/27 09:46:17 INFO OIDC provider initialized name=provider120702026/08/27 09:46:17 INFO OIDC provider initialized name=test20712026/08/27 09:46:17 INFO OIDC provider initialized name=test20722026/08/27 09:46:17 INFO OIDC provider initialized name=test20732026/08/27 09:46:17 INFO OIDC provider initialized name=test20742026/08/27 09:46:17 INFO OIDC provider initialized name=provider22075--- PASS: TestValidateToken_BoundClaimsMismatch (0.01s)2076--- PASS: TestValidateToken_ValidToken (0.01s)2077--- PASS: TestValidateToken_MultipleProviders (0.01s)2078--- PASS: TestValidateToken_WrongAudience (0.01s)2079--- PASS: TestValidateToken_NoMatchingProvider (0.01s)2080--- PASS: TestValidateToken_BoundSubjectMismatch (0.01s)2081--- PASS: TestValidateToken_Expired (0.01s)2082PASS2083Running hook tests...2084=== RUN TestSendPathsEmpty2085=== PAUSE TestSendPathsEmpty2086=== RUN TestQueueEnqueueAndFetch2087=== PAUSE TestQueueEnqueueAndFetch2088=== RUN TestQueueDeduplication2089=== PAUSE TestQueueDeduplication2090=== RUN TestQueueRemove2091=== PAUSE TestQueueRemove2092=== RUN TestQueueFetchBatchLimit2093=== PAUSE TestQueueFetchBatchLimit2094=== RUN TestQueueRetryMovesToBack2095=== PAUSE TestQueueRetryMovesToBack2096=== RUN TestQueueFetchRemoveLifecycle2097=== PAUSE TestQueueFetchRemoveLifecycle2098=== RUN TestQueueConcurrentWriters2099=== PAUSE TestQueueConcurrentWriters2100=== RUN TestQueueRemoveLargeClosure2101=== PAUSE TestQueueRemoveLargeClosure2102=== RUN TestServerClientIntegration2103=== PAUSE TestServerClientIntegration2104=== RUN TestServerQueueError2105=== PAUSE TestServerQueueError2106=== RUN TestGetListenerSocketActivation2107 server_test.go:210: === RUN TestGetListenerSocketActivation2108 --- PASS: TestGetListenerSocketActivation (0.00s)2109 PASS2110 2111--- PASS: TestGetListenerSocketActivation (0.01s)2112=== RUN TestDrainIsolatesPoisonPath2113=== PAUSE TestDrainIsolatesPoisonPath2114=== RUN TestRunNotBlockedByPoisonHead2115=== PAUSE TestRunNotBlockedByPoisonHead2116=== RUN TestDrainGivesUpWhenServerDown2117=== PAUSE TestDrainGivesUpWhenServerDown2118=== RUN TestFailedPathPrunedByLaterClosure2119=== PAUSE TestFailedPathPrunedByLaterClosure2120=== RUN TestWorkerUploadsAndRemoves2121=== PAUSE TestWorkerUploadsAndRemoves2122=== RUN TestWorkerSkipsGCdPaths2123=== PAUSE TestWorkerSkipsGCdPaths2124=== RUN TestWorkerPrunesClosureDeps2125=== PAUSE TestWorkerPrunesClosureDeps2126=== CONT TestSendPathsEmpty2127=== CONT TestWorkerPrunesClosureDeps2128=== CONT TestServerClientIntegration2129=== CONT TestWorkerSkipsGCdPaths2130=== CONT TestWorkerUploadsAndRemoves2131=== CONT TestFailedPathPrunedByLaterClosure2132=== CONT TestDrainGivesUpWhenServerDown2133=== CONT TestRunNotBlockedByPoisonHead2134=== CONT TestDrainIsolatesPoisonPath2135=== CONT TestServerQueueError2136--- PASS: TestSendPathsEmpty (0.00s)2137=== CONT TestQueueRetryMovesToBack2138=== CONT TestQueueFetchBatchLimit2139=== CONT TestQueueRemoveLargeClosure2140=== CONT TestQueueRemove2141=== CONT TestQueueConcurrentWriters2142=== CONT TestQueueDeduplication2143=== CONT TestQueueFetchRemoveLifecycle21442026/08/27 09:46:17 ERROR Failed to queue paths error="permission denied" count=12145=== CONT TestQueueEnqueueAndFetch2146--- PASS: TestServerClientIntegration (0.00s)2147--- PASS: TestServerQueueError (0.00s)21482026/08/27 09:46:17 INFO Uploading batch count=221492026/08/27 09:46:17 ERROR Upload failed error="upload failed" count=221502026/08/27 09:46:17 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown3602516582/002/a21512026/08/27 09:46:17 INFO Uploading batch count=421522026/08/27 09:46:17 ERROR Upload failed error="upload failed" count=421532026/08/27 09:46:17 INFO Uploading batch count=121542026/08/27 09:46:17 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainIsolatesPoisonPath2098614473/002/bbb21552026/08/27 09:46:17 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown3602516582/002/b21562026/08/27 09:46:17 ERROR Upload failed error="upload failed" count=121572026/08/27 09:46:17 INFO Upload queue status pending=22158--- PASS: TestQueueFetchBatchLimit (0.02s)2159--- PASS: TestQueueEnqueueAndFetch (0.02s)21602026/08/27 09:46:17 INFO Uploading batch count=221612026/08/27 09:46:17 INFO Upload queue status pending=321622026/08/27 09:46:17 INFO Upload queue status pending=221632026/08/27 09:46:17 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerSkipsGCdPaths4030747242/002/nonexistent21642026/08/27 09:46:17 INFO Uploading batch count=221652026/08/27 09:46:17 ERROR Upload failed error="upload failed" count=221662026/08/27 09:46:17 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown3602516582/002/c21672026/08/27 09:46:17 INFO Uploading batch count=12168--- PASS: TestQueueRemove (0.02s)21692026/08/27 09:46:17 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown3602516582/002/d21702026/08/27 09:46:17 INFO Upload queue status pending=221712026/08/27 09:46:17 INFO Uploading batch count=221722026/08/27 09:46:17 ERROR Upload failed error="upload failed" count=221732026/08/27 09:46:17 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown3602516582/002/e21742026/08/27 09:46:17 INFO Uploading batch count=121752026/08/27 09:46:17 INFO Uploading batch count=121762026/08/27 09:46:17 ERROR Upload failed error="upload failed" count=121772026/08/27 09:46:17 INFO Uploading batch count=12178--- PASS: TestQueueDeduplication (0.02s)21792026/08/27 09:46:17 INFO Uploading batch count=121802026/08/27 09:46:17 ERROR Upload failed error="upload failed" count=121812026/08/27 09:46:17 INFO Uploading batch count=121822026/08/27 09:46:17 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown3602516582/002/f2183--- PASS: TestQueueRetryMovesToBack (0.02s)2184--- PASS: TestQueueFetchRemoveLifecycle (0.02s)21852026/08/27 09:46:17 INFO Uploading batch count=121862026/08/27 09:46:17 ERROR Upload failed error="upload failed" count=121872026/08/27 09:46:17 ERROR Drain finished with paths left in queue remaining=1021882026/08/27 09:46:17 INFO Uploading batch count=121892026/08/27 09:46:17 ERROR Upload failed error="upload failed" count=12190--- PASS: TestFailedPathPrunedByLaterClosure (0.03s)21912026/08/27 09:46:17 ERROR Drain finished with paths left in queue remaining=12192--- PASS: TestDrainGivesUpWhenServerDown (0.03s)2193--- PASS: TestDrainIsolatesPoisonPath (0.03s)2194--- PASS: TestWorkerUploadsAndRemoves (0.04s)2195--- PASS: TestWorkerSkipsGCdPaths (0.04s)2196--- PASS: TestWorkerPrunesClosureDeps (0.05s)2197--- PASS: TestQueueConcurrentWriters (0.22s)2198--- PASS: TestQueueRemoveLargeClosure (0.34s)21992026/08/27 09:46:18 INFO Uploading batch count=122002026/08/27 09:46:18 INFO Uploading batch count=122012026/08/27 09:46:18 INFO Uploading batch count=122022026/08/27 09:46:18 ERROR Upload failed error="upload failed" count=122032026/08/27 09:46:18 INFO Uploading batch count=122042026/08/27 09:46:18 ERROR Upload failed error="upload failed" count=122052026/08/27 09:46:18 INFO Uploading batch count=122062026/08/27 09:46:18 ERROR Upload failed error="upload failed" count=122072026/08/27 09:46:18 INFO Uploading batch count=122082026/08/27 09:46:18 ERROR Upload failed error="upload failed" count=122092026/08/27 09:46:18 ERROR Drain finished with paths left in queue remaining=12210--- PASS: TestRunNotBlockedByPoisonHead (1.04s)2211PASS