niks3-go-unit-tests
checks.aarch64-linux.go-unit-tests
· build #145
· raw
1tribuchet: building on eliza2Running client tests...3=== RUN TestDoServerRequestAttachesToken4=== PAUSE TestDoServerRequestAttachesToken5=== RUN TestCaseHackSuffix6=== PAUSE TestCaseHackSuffix7=== RUN TestFilterOversizedClosures8=== PAUSE TestFilterOversizedClosures9=== RUN TestPartSizeForNAR10=== PAUSE TestPartSizeForNAR11=== RUN TestUploadMultipart_SupersededByPeer12=== PAUSE TestUploadMultipart_SupersededByPeer13=== RUN TestDumpPathMatchesNix14=== PAUSE TestDumpPathMatchesNix15=== RUN TestDumpPathSingleFile16=== PAUSE TestDumpPathSingleFile17=== RUN TestDumpPathWriterError18=== PAUSE TestDumpPathWriterError19=== RUN TestEncodeNixBase3220=== PAUSE TestEncodeNixBase3221=== RUN TestEncodeNixBase32WithRealHash22=== PAUSE TestEncodeNixBase32WithRealHash23=== RUN TestConvertHashToNix3224=== PAUSE TestConvertHashToNix3225=== RUN TestGetStorePathHash26=== PAUSE TestGetStorePathHash27=== RUN TestPathInfoHashCompatibility28=== PAUSE TestPathInfoHashCompatibility29=== RUN TestParsePathInfoJSON30=== PAUSE TestParsePathInfoJSON31=== RUN TestParsePathInfoJSONMultiplePaths32=== PAUSE TestParsePathInfoJSONMultiplePaths33=== RUN TestPathInfoCACompatibility34=== PAUSE TestPathInfoCACompatibility35=== RUN TestRateLimiterFeedback36=== PAUSE TestRateLimiterFeedback37=== RUN TestRateLimiterFeedback_400DoesNotCountAsSuccess38=== PAUSE TestRateLimiterFeedback_400DoesNotCountAsSuccess39=== RUN TestResolveStorePath40=== PAUSE TestResolveStorePath41=== RUN TestDoWithRetry_BodyReplayedViaGetBody42=== PAUSE TestDoWithRetry_BodyReplayedViaGetBody43=== RUN TestShellSplit44=== PAUSE TestShellSplit45=== RUN TestShellSplitErrors46=== PAUSE TestShellSplitErrors47=== RUN TestSetClientTLS48=== PAUSE TestSetClientTLS49=== RUN TestSetClientTLSDoesNotMutateDefaultTransport50=== PAUSE TestSetClientTLSDoesNotMutateDefaultTransport51=== RUN TestSetClientTLSErrors52=== PAUSE TestSetClientTLSErrors53=== RUN TestStaticToken54=== PAUSE TestStaticToken55=== RUN TestFileTokenReadsAndCaches56=== PAUSE TestFileTokenReadsAndCaches57=== RUN TestFileTokenMissing58=== PAUSE TestFileTokenMissing59=== RUN TestFileTokenEmpty60=== PAUSE TestFileTokenEmpty61=== RUN TestScriptTokenNoExpiryRerunsEveryCall62=== PAUSE TestScriptTokenNoExpiryRerunsEveryCall63=== RUN TestScriptTokenCachesUntilRefresh64=== PAUSE TestScriptTokenCachesUntilRefresh65=== RUN TestScriptTokenEmptyToken66=== PAUSE TestScriptTokenEmptyToken67=== RUN TestScriptTokenBadJSON68=== PAUSE TestScriptTokenBadJSON69=== RUN TestScriptTokenScriptFails70=== PAUSE TestScriptTokenScriptFails71=== RUN TestScriptTokenEmptyCommand72=== PAUSE TestScriptTokenEmptyCommand73=== CONT TestDoServerRequestAttachesToken74=== CONT TestConvertHashToNix3275=== CONT TestDoWithRetry_BodyReplayedViaGetBody76=== CONT TestFileTokenReadsAndCaches77=== RUN TestConvertHashToNix32/SRI_format_to_Nix3278=== CONT TestScriptTokenScriptFails79=== CONT TestFileTokenMissing80=== CONT TestRateLimiterFeedback_400DoesNotCountAsSuccess81=== CONT TestScriptTokenEmptyCommand82=== CONT TestRateLimiterFeedback83=== RUN TestRateLimiterFeedback/429_enables_limiter84=== PAUSE TestRateLimiterFeedback/429_enables_limiter85=== RUN TestRateLimiterFeedback/503_enables_limiter86=== PAUSE TestRateLimiterFeedback/503_enables_limiter87=== RUN TestRateLimiterFeedback/200_does_not_enable_limiter88=== CONT TestPathInfoCACompatibility89=== RUN TestPathInfoCACompatibility/null_ca_field90=== PAUSE TestPathInfoCACompatibility/null_ca_field91=== CONT TestParsePathInfoJSONMultiplePaths92=== RUN TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths93=== PAUSE TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths94=== CONT TestScriptTokenCachesUntilRefresh95=== CONT TestScriptTokenBadJSON962026/08/27 09:37:20 WARN Rate limiter enabled after throttle name=server-test rate=597=== CONT TestPathInfoHashCompatibility98=== RUN TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)99=== CONT TestGetStorePathHash100=== CONT TestSetClientTLSDoesNotMutateDefaultTransport101=== CONT TestStaticToken102=== CONT TestSetClientTLSErrors103=== CONT TestShellSplitErrors104=== CONT TestSetClientTLS105=== CONT TestShellSplit106=== CONT TestPartSizeForNAR107=== CONT TestUploadMultipart_SupersededByPeer108=== CONT TestEncodeNixBase32109=== PAUSE TestConvertHashToNix32/SRI_format_to_Nix32110--- PASS: TestScriptTokenEmptyCommand (0.00s)111--- PASS: TestFileTokenReadsAndCaches (0.00s)112--- PASS: TestFileTokenMissing (0.00s)113--- PASS: TestShellSplitErrors (0.00s)114=== CONT TestScriptTokenEmptyToken115--- PASS: TestStaticToken (0.00s)116=== CONT TestFilterOversizedClosures117=== RUN TestGetStorePathHash/valid_store_path118=== RUN TestFilterOversizedClosures/no_limit_keeps_everything119=== PAUSE TestFilterOversizedClosures/no_limit_keeps_everything120=== RUN TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped121=== PAUSE TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped122=== RUN TestUploadMultipart_SupersededByPeer/exists123=== PAUSE TestUploadMultipart_SupersededByPeer/exists124=== PAUSE TestGetStorePathHash/valid_store_path125=== CONT TestResolveStorePath126=== PAUSE TestRateLimiterFeedback/200_does_not_enable_limiter127=== RUN TestRateLimiterFeedback/400_does_not_enable_limiter128=== CONT TestDumpPathSingleFile1292026/08/27 09:37:20 WARN Rate limiter enabled after throttle name=server-test rate=5130=== RUN TestPathInfoCACompatibility/old_string_format_-_text131=== CONT TestParsePathInfoJSON1322026/08/27 09:37:20 WARN Request returned retryable status, retrying attempt=1 max_attempts=6 backoff=0s status=503 url=http://127.0.0.1:46747133=== RUN TestParsePathInfoJSON/Nix_format134=== RUN TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths135=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)136=== CONT TestEncodeNixBase32WithRealHash137=== RUN TestFilterOversizedClosures/all_closures_skipped138=== RUN TestEncodeNixBase32/test_string_hash139=== RUN TestUploadMultipart_SupersededByPeer/missing140=== RUN TestConvertHashToNix32/already_Nix32_format141--- PASS: TestScriptTokenScriptFails (0.00s)142=== CONT TestCaseHackSuffix143=== RUN TestGetStorePathHash/basename_without_hyphen_should_error144=== CONT TestDumpPathWriterError145=== RUN TestPartSizeForNAR/zero_stays_at_minimum146=== PAUSE TestUploadMultipart_SupersededByPeer/missing147=== PAUSE TestPathInfoCACompatibility/old_string_format_-_text148=== RUN TestPathInfoCACompatibility/old_string_format_-_fixed_recursive149=== CONT TestDumpPathMatchesNix150=== PAUSE TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths151=== RUN TestPathInfoHashCompatibility/old_string_format_with_colon152=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_colon153=== PAUSE TestFilterOversizedClosures/all_closures_skipped154=== CONT TestUploadMultipart_SupersededByPeer/exists155=== CONT TestUploadMultipart_SupersededByPeer/missing156=== PAUSE TestEncodeNixBase32/test_string_hash157--- PASS: TestShellSplit (0.00s)158=== PAUSE TestGetStorePathHash/basename_without_hyphen_should_error159=== PAUSE TestPartSizeForNAR/zero_stays_at_minimum160=== CONT TestScriptTokenNoExpiryRerunsEveryCall161=== CONT TestFileTokenEmpty162=== RUN TestPartSizeForNAR/small_stays_at_minimum1632026/08/27 09:37:20 WARN Rate limiter backed off name=server-test rate=5164=== PAUSE TestRateLimiterFeedback/400_does_not_enable_limiter165=== PAUSE TestParsePathInfoJSON/Nix_format1662026/08/27 09:37:20 WARN Request returned retryable status, retrying attempt=2 max_attempts=6 backoff=0s status=503 url=http://127.0.0.1:46747167=== PAUSE TestPathInfoCACompatibility/old_string_format_-_fixed_recursive168=== RUN TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI169=== PAUSE TestConvertHashToNix32/already_Nix32_format170--- PASS: TestScriptTokenBadJSON (0.00s)171=== CONT TestFilterOversizedClosures/all_closures_skipped172=== CONT TestFilterOversizedClosures/no_limit_keeps_everything173=== RUN TestEncodeNixBase32/empty_input174=== CONT TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped1752026/08/27 09:37:20 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb-vm-image nar_size=5000 max_nar_size=50176=== RUN TestGetStorePathHash/hash_with_invalid_characters_should_error1772026/08/27 09:37:20 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb-vm-image nar_size=5000 max_nar_size=2000178=== RUN TestSetClientTLSErrors/missing_cert_file179=== RUN TestParsePathInfoJSON/Lix_format180=== CONT TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths181=== PAUSE TestParsePathInfoJSON/Lix_format182=== RUN TestParsePathInfoJSON/empty_input183=== PAUSE TestSetClientTLSErrors/missing_cert_file184=== RUN TestSetClientTLSErrors/missing_key_file185=== RUN TestPathInfoCACompatibility/new_structured_format_-_text186=== CONT TestRateLimiterFeedback/503_enables_limiter187=== PAUSE TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI188=== RUN TestConvertHashToNix32/invalid_format189--- PASS: TestResolveStorePath (0.00s)190=== CONT TestRateLimiterFeedback/429_enables_limiter191=== PAUSE TestEncodeNixBase32/empty_input192=== CONT TestRateLimiterFeedback/400_does_not_enable_limiter193=== PAUSE TestGetStorePathHash/hash_with_invalid_characters_should_error194=== RUN TestGetStorePathHash/hash_with_wrong_length_should_error195=== CONT TestRateLimiterFeedback/200_does_not_enable_limiter196=== PAUSE TestPartSizeForNAR/small_stays_at_minimum197=== CONT TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths1982026/08/27 09:37:20 WARN Rate limiter enabled after throttle name=server-test rate=5199=== RUN TestPartSizeForNAR/80_GiB_fits_at_minimum2002026/08/27 09:37:20 WARN Request returned retryable status, retrying attempt=1 max_attempts=2 backoff=0s status=503 url=http://127.0.0.1:40275201=== PAUSE TestParsePathInfoJSON/empty_input2022026/08/27 09:37:20 WARN Rate limiter enabled after throttle name=server-test rate=5203=== PAUSE TestPartSizeForNAR/80_GiB_fits_at_minimum2042026/08/27 09:37:20 WARN Request returned retryable status, retrying attempt=1 max_attempts=2 backoff=0s status=429 url=http://127.0.0.1:44729205=== RUN TestPartSizeForNAR/115_GiB_needs_larger_parts206=== PAUSE TestPartSizeForNAR/115_GiB_needs_larger_parts207=== PAUSE TestSetClientTLSErrors/missing_key_file208=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_text209=== RUN TestPathInfoHashCompatibility/new_structured_format_with_sha512210=== PAUSE TestPathInfoHashCompatibility/new_structured_format_with_sha5122112026/08/27 09:37:20 WARN Rate limiter backed off name=server-test rate=5212=== CONT TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)2132026/08/27 09:37:20 WARN Rate limiter backed off name=server-test rate=5214--- PASS: TestEncodeNixBase32WithRealHash (0.00s)215--- PASS: TestScriptTokenEmptyToken (0.00s)216--- PASS: TestDoServerRequestAttachesToken (0.01s)217=== PAUSE TestConvertHashToNix32/invalid_format218--- PASS: TestFileTokenEmpty (0.00s)219=== CONT TestConvertHashToNix32/SRI_format_to_Nix32220=== CONT TestEncodeNixBase32/empty_input221=== PAUSE TestGetStorePathHash/hash_with_wrong_length_should_error222=== CONT TestGetStorePathHash/valid_store_path223=== RUN TestParsePathInfoJSON/whitespace_only224=== RUN TestSetClientTLS/rejects_connection_without_client_cert225=== PAUSE TestSetClientTLS/rejects_connection_without_client_cert226=== RUN TestSetClientTLS/succeeds_with_client_cert_and_CA227=== PAUSE TestSetClientTLS/succeeds_with_client_cert_and_CA228=== RUN TestSetClientTLS/preserves_debug_logging_transport229=== PAUSE TestSetClientTLS/preserves_debug_logging_transport230=== CONT TestSetClientTLS/rejects_connection_without_client_cert231=== CONT TestSetClientTLS/preserves_debug_logging_transport232=== RUN TestPartSizeForNAR/1_TiB233=== RUN TestSetClientTLSErrors/missing_ca_file234=== PAUSE TestSetClientTLSErrors/missing_ca_file235=== RUN TestSetClientTLSErrors/invalid_ca_file236=== PAUSE TestSetClientTLSErrors/invalid_ca_file237=== RUN TestPathInfoCACompatibility/new_structured_format_-_nar_method238=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_nar_method239=== CONT TestPathInfoHashCompatibility/new_structured_format_with_sha512240=== CONT TestSetClientTLSErrors/missing_key_file241=== CONT TestPathInfoCACompatibility/null_ca_field242=== CONT TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI243=== CONT TestPathInfoCACompatibility/old_string_format_-_fixed_recursive244=== CONT TestPathInfoCACompatibility/old_string_format_-_text245=== CONT TestPathInfoCACompatibility/new_structured_format_-_nar_method246=== CONT TestPathInfoHashCompatibility/old_string_format_with_colon247=== CONT TestEncodeNixBase32/test_string_hash248=== CONT TestConvertHashToNix32/invalid_format249=== CONT TestConvertHashToNix32/already_Nix32_format250--- PASS: TestSetClientTLSDoesNotMutateDefaultTransport (0.01s)251=== CONT TestGetStorePathHash/hash_with_invalid_characters_should_error252=== CONT TestGetStorePathHash/basename_without_hyphen_should_error253--- PASS: TestFilterOversizedClosures (0.00s)254 --- PASS: TestFilterOversizedClosures/no_limit_keeps_everything (0.00s)255 --- PASS: TestFilterOversizedClosures/all_closures_skipped (0.00s)256 --- PASS: TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped (0.00s)257--- PASS: TestDoWithRetry_BodyReplayedViaGetBody (0.01s)258=== CONT TestGetStorePathHash/hash_with_wrong_length_should_error259=== PAUSE TestParsePathInfoJSON/whitespace_only260=== RUN TestParsePathInfoJSON/invalid_JSON261=== PAUSE TestParsePathInfoJSON/invalid_JSON262=== CONT TestParsePathInfoJSON/Nix_format263=== CONT TestParsePathInfoJSON/whitespace_only264=== CONT TestSetClientTLS/succeeds_with_client_cert_and_CA265=== CONT TestParsePathInfoJSON/empty_input266=== PAUSE TestPartSizeForNAR/1_TiB267=== RUN TestPartSizeForNAR/5_TiB_S3_max_object268=== PAUSE TestPartSizeForNAR/5_TiB_S3_max_object269=== CONT TestSetClientTLSErrors/missing_cert_file270=== CONT TestSetClientTLSErrors/invalid_ca_file271=== CONT TestSetClientTLSErrors/missing_ca_file272=== CONT TestPathInfoCACompatibility/new_structured_format_-_text273--- PASS: TestScriptTokenCachesUntilRefresh (0.01s)274=== CONT TestParsePathInfoJSON/Lix_format275=== CONT TestParsePathInfoJSON/invalid_JSON276=== RUN TestPartSizeForNAR/capped_at_5_GiB277=== PAUSE TestPartSizeForNAR/capped_at_5_GiB278--- PASS: TestUploadMultipart_SupersededByPeer (0.00s)279 --- PASS: TestUploadMultipart_SupersededByPeer/exists (0.00s)280 --- PASS: TestUploadMultipart_SupersededByPeer/missing (0.00s)281=== CONT TestPartSizeForNAR/5_TiB_S3_max_object282=== CONT TestPartSizeForNAR/115_GiB_needs_larger_parts283=== CONT TestPartSizeForNAR/1_TiB284=== CONT TestPartSizeForNAR/80_GiB_fits_at_minimum285=== CONT TestPartSizeForNAR/capped_at_5_GiB286=== CONT TestPartSizeForNAR/small_stays_at_minimum287=== CONT TestPartSizeForNAR/zero_stays_at_minimum288--- PASS: TestRateLimiterFeedback (0.01s)289 --- PASS: TestRateLimiterFeedback/400_does_not_enable_limiter (0.00s)290 --- PASS: TestRateLimiterFeedback/200_does_not_enable_limiter (0.00s)291 --- PASS: TestRateLimiterFeedback/503_enables_limiter (0.00s)292 --- PASS: TestRateLimiterFeedback/429_enables_limiter (0.00s)293--- PASS: TestPathInfoHashCompatibility (0.01s)294 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI) (0.00s)295 --- PASS: TestPathInfoHashCompatibility/new_structured_format_with_sha512 (0.00s)296 --- PASS: TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI (0.00s)297 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_colon (0.00s)298--- PASS: TestEncodeNixBase32 (0.01s)299 --- PASS: TestEncodeNixBase32/empty_input (0.00s)300 --- PASS: TestEncodeNixBase32/test_string_hash (0.00s)301--- PASS: TestConvertHashToNix32 (0.01s)302 --- PASS: TestConvertHashToNix32/SRI_format_to_Nix32 (0.00s)303 --- PASS: TestConvertHashToNix32/invalid_format (0.00s)304 --- PASS: TestConvertHashToNix32/already_Nix32_format (0.00s)305--- PASS: TestGetStorePathHash (0.01s)306 --- PASS: TestGetStorePathHash/valid_store_path (0.00s)307 --- PASS: TestGetStorePathHash/hash_with_invalid_characters_should_error (0.00s)308 --- PASS: TestGetStorePathHash/basename_without_hyphen_should_error (0.00s)309 --- PASS: TestGetStorePathHash/hash_with_wrong_length_should_error (0.00s)310--- PASS: TestParsePathInfoJSONMultiplePaths (0.00s)311 --- PASS: TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths (0.00s)312 --- PASS: TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths (0.00s)313--- PASS: TestParsePathInfoJSON (0.01s)314 --- PASS: TestParsePathInfoJSON/Nix_format (0.00s)315 --- PASS: TestParsePathInfoJSON/whitespace_only (0.00s)316 --- PASS: TestParsePathInfoJSON/empty_input (0.00s)317 --- PASS: TestParsePathInfoJSON/Lix_format (0.00s)318 --- PASS: TestParsePathInfoJSON/invalid_JSON (0.00s)319--- PASS: TestPathInfoCACompatibility (0.01s)320 --- PASS: TestPathInfoCACompatibility/null_ca_field (0.00s)321 --- PASS: TestPathInfoCACompatibility/old_string_format_-_fixed_recursive (0.00s)322 --- PASS: TestPathInfoCACompatibility/old_string_format_-_text (0.00s)323 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_nar_method (0.00s)324 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_text (0.00s)325--- PASS: TestPartSizeForNAR (0.01s)326 --- PASS: TestPartSizeForNAR/5_TiB_S3_max_object (0.00s)327 --- PASS: TestPartSizeForNAR/115_GiB_needs_larger_parts (0.00s)328 --- PASS: TestPartSizeForNAR/1_TiB (0.00s)329 --- PASS: TestPartSizeForNAR/80_GiB_fits_at_minimum (0.00s)330 --- PASS: TestPartSizeForNAR/capped_at_5_GiB (0.00s)331 --- PASS: TestPartSizeForNAR/small_stays_at_minimum (0.00s)332 --- PASS: TestPartSizeForNAR/zero_stays_at_minimum (0.00s)333--- PASS: TestSetClientTLSErrors (0.01s)334 --- PASS: TestSetClientTLSErrors/missing_key_file (0.00s)335 --- PASS: TestSetClientTLSErrors/missing_cert_file (0.00s)336 --- PASS: TestSetClientTLSErrors/missing_ca_file (0.00s)337 --- PASS: TestSetClientTLSErrors/invalid_ca_file (0.00s)338--- PASS: TestScriptTokenNoExpiryRerunsEveryCall (0.01s)3392026/08/27 09:37:20 http: TLS handshake error from 127.0.0.1:46998: remote error: tls: bad certificate340--- PASS: TestSetClientTLS (0.02s)341 --- PASS: TestSetClientTLS/preserves_debug_logging_transport (0.00s)342 --- PASS: TestSetClientTLS/succeeds_with_client_cert_and_CA (0.00s)343 --- PASS: TestSetClientTLS/rejects_connection_without_client_cert (0.01s)344--- PASS: TestCaseHackSuffix (0.03s)345--- PASS: TestDumpPathSingleFile (0.03s)346--- PASS: TestDumpPathWriterError (0.03s)347--- PASS: TestDumpPathMatchesNix (0.10s)348--- PASS: TestRateLimiterFeedback_400DoesNotCountAsSuccess (1.00s)349PASS350Running server tests...351The files belonging to this database system will be owned by user "nixbld".352This user must also own the server process.353354The database cluster will be initialized with locale "C".355The default database encoding has accordingly been set to "SQL_ASCII".356The default text search configuration will be set to "english".357358Data page checksums are enabled.359360creating directory /build/postgres3607394847/data ... ok361creating subdirectories ... ok362selecting dynamic shared memory implementation ... posix363selecting default "max_connections" ... 100364selecting default "shared_buffers" ... 128MB365selecting default time zone ... UTC366creating configuration files ... ok367running bootstrap script ... ok368performing post-bootstrap initialization ... ok369syncing data to disk ... ok370371initdb: warning: enabling "trust" authentication for local connections372initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.373374Success. You can now start the database server using:375376 pg_ctl -D /build/postgres3607394847/data -l logfile start377378/build/postgres3607394847:5432 - no response3792026-08-27 09:37:22.398 UTC [111] LOG: starting PostgreSQL 18.4 on aarch64-unknown-linux-gnu, compiled by clang version 21.1.8, 64-bit3802026-08-27 09:37:22.398 UTC [111] LOG: listening on Unix socket "/build/postgres3607394847/.s.PGSQL.5432"3812026-08-27 09:37:22.403 UTC [118] LOG: database system was shut down at 2026-08-27 09:37:22 UTC3822026-08-27 09:37:22.406 UTC [111] LOG: database system is ready to accept connections383/build/postgres3607394847:5432 - accepting connections384=== RUN TestService_AuthMiddleware385=== PAUSE TestService_AuthMiddleware386=== RUN TestService_AuthMiddleware_MTLSProxyHeader387=== PAUSE TestService_AuthMiddleware_MTLSProxyHeader388=== RUN TestService_AuthMiddleware_MTLSBoundSubjects389=== PAUSE TestService_AuthMiddleware_MTLSBoundSubjects390=== RUN TestService_ReadAuthMiddleware391=== PAUSE TestService_ReadAuthMiddleware392=== RUN TestService_AuthMiddleware_OIDC393=== PAUSE TestService_AuthMiddleware_OIDC394=== RUN TestCacheConfigHandler395=== PAUSE TestCacheConfigHandler396=== RUN TestCacheStatsHandler397=== PAUSE TestCacheStatsHandler398=== RUN TestClientCADerivations399=== PAUSE TestClientCADerivations400=== RUN TestClientErrorHandling401=== PAUSE TestClientErrorHandling402=== RUN TestClientIntegration403=== PAUSE TestClientIntegration404=== RUN TestClientMultipleUploads405=== PAUSE TestClientMultipleUploads406=== RUN TestClientWithDependencies407=== PAUSE TestClientWithDependencies408=== RUN TestPinProtectsFromGC409=== PAUSE TestPinProtectsFromGC410=== RUN TestGCAdvisoryLockBlocksConcurrentRun4112026-08-27 09:37:27.448 UTC [518] ERROR: relation "goose_db_version" does not exist at character 364122026-08-27 09:37:27.448 UTC [518] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC4132026/08/27 09:37:27 OK 20241026095416_initial_model.sql (12.01ms)4142026/08/27 09:37:27 OK 20251210153512_drop_unused_gin_index.sql (2.09ms)4152026/08/27 09:37:27 OK 20251218171726_add_pins.sql (3.39ms)4162026/08/27 09:37:27 OK 20260628120000_add_object_size_and_stats.sql (3.18ms)4172026/08/27 09:37:27 goose: successfully migrated database to version: 202606281200004182026/08/27 09:37:27 OK 1_commit_pending_closure.sql (2.04ms)4192026/08/27 09:37:27 OK 2_object_stats_trigger.sql (838.57µs)4202026/08/27 09:37:27 goose: up to current file version: 2421--- PASS: TestGCAdvisoryLockBlocksConcurrentRun (0.17s)422=== RUN TestGCBugBareHashReferences423=== PAUSE TestGCBugBareHashReferences424=== RUN TestGCMetrics425=== PAUSE TestGCMetrics426=== RUN TestGCTaskStore_StartNew427=== PAUSE TestGCTaskStore_StartNew428=== RUN TestGCTaskStore_DeduplicateSameParams429=== PAUSE TestGCTaskStore_DeduplicateSameParams430=== RUN TestGCTaskStore_ConflictDifferentParams431=== PAUSE TestGCTaskStore_ConflictDifferentParams432=== RUN TestGCTaskStore_GetEmpty433=== PAUSE TestGCTaskStore_GetEmpty434=== RUN TestGCTaskStore_GetReturnsLatest435=== PAUSE TestGCTaskStore_GetReturnsLatest436=== RUN TestGCTaskStore_CompletedAllowsNewTask437=== PAUSE TestGCTaskStore_CompletedAllowsNewTask438=== RUN TestGCTaskStore_PhaseUpdates439=== PAUSE TestGCTaskStore_PhaseUpdates440=== RUN TestGCTaskStore_Fail441=== PAUSE TestGCTaskStore_Fail442=== RUN TestGracefulShutdownDrainsInflight443=== PAUSE TestGracefulShutdownDrainsInflight444=== RUN TestService_healthCheckHandler445=== PAUSE TestService_healthCheckHandler446=== RUN TestGenerateLandingPage447=== PAUSE TestGenerateLandingPage448=== RUN TestCacheConfigHandlerMaxNarSize449=== PAUSE TestCacheConfigHandlerMaxNarSize450=== RUN TestCreatePendingClosureRejectsOversizedNAR451=== PAUSE TestCreatePendingClosureRejectsOversizedNAR452=== RUN TestNARDeduplicationMetadataUploadBug453=== PAUSE TestNARDeduplicationMetadataUploadBug454=== RUN TestMetricsInventory455=== PAUSE TestMetricsInventory456=== RUN TestService_NativeMTLS457=== PAUSE TestService_NativeMTLS458=== RUN TestServerTLSConfig459=== PAUSE TestServerTLSConfig460=== RUN TestMultipartCleanup461=== PAUSE TestMultipartCleanup462=== RUN TestObjectStatsTrigger463=== PAUSE TestObjectStatsTrigger464=== RUN TestOrphanedObjectsGC465=== PAUSE TestOrphanedObjectsGC466=== RUN TestOrphanedObjectsGCStressTest467=== PAUSE TestOrphanedObjectsGCStressTest468=== RUN TestResurrectedObjectNotDeleted469=== PAUSE TestResurrectedObjectNotDeleted470=== RUN TestParseSingleRange471=== PAUSE TestParseSingleRange472=== RUN TestIsValidCachePath473=== PAUSE TestIsValidCachePath474=== RUN TestReadProxyNarinfo475=== PAUSE TestReadProxyNarinfo476=== RUN TestReadProxyNarinfoAlreadyDecompressed477=== PAUSE TestReadProxyNarinfoAlreadyDecompressed478=== RUN TestReadProxyNarStreaming479=== PAUSE TestReadProxyNarStreaming480=== RUN TestReadProxy404481=== PAUSE TestReadProxy404482=== RUN TestReadProxyInvalidPath483=== PAUSE TestReadProxyInvalidPath484=== RUN TestReadProxyHead485=== PAUSE TestReadProxyHead486=== RUN TestReadProxyConditionalGet487=== PAUSE TestReadProxyConditionalGet488=== RUN TestReadProxyRootRedirectsToIndexHTML489=== PAUSE TestReadProxyRootRedirectsToIndexHTML490=== RUN TestReadProxyDisabled491=== PAUSE TestReadProxyDisabled492=== RUN TestReadProxyRangeRequest493=== PAUSE TestReadProxyRangeRequest494=== RUN TestRedundantMultipartUpload495=== PAUSE TestRedundantMultipartUpload496=== RUN TestCompleteMultipartUpload_ErrorButObjectExists497=== PAUSE TestCompleteMultipartUpload_ErrorButObjectExists498=== RUN TestCompletedNarNotReofferedAcrossClosures499=== PAUSE TestCompletedNarNotReofferedAcrossClosures500=== RUN TestPresignedUploadRegisteredBeforeCommit501=== PAUSE TestPresignedUploadRegisteredBeforeCommit502=== RUN TestService_Rustfstest503=== PAUSE TestService_Rustfstest504=== RUN TestParseSize505=== PAUSE TestParseSize506=== RUN TestSkippedUploadsHandler507=== PAUSE TestSkippedUploadsHandler508=== RUN TestSystemdListenerNotActivated509--- PASS: TestSystemdListenerNotActivated (0.00s)510=== RUN TestWatchdogBeatsWhenHealthy511--- PASS: TestWatchdogBeatsWhenHealthy (0.02s)512=== RUN TestWatchdogSkipsWhenUnhealthy5132026/08/27 09:37:27 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"5142026/08/27 09:37:27 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"5152026/08/27 09:37:27 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"5162026/08/27 09:37:27 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"5172026/08/27 09:37:27 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"5182026/08/27 09:37:27 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"5192026/08/27 09:37:27 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"5202026/08/27 09:37:27 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"5212026/08/27 09:37:27 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"5222026/08/27 09:37:27 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"523--- PASS: TestWatchdogSkipsWhenUnhealthy (0.20s)524=== RUN TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle525=== PAUSE TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle526=== RUN TestProxyWriteTimeout527=== PAUSE TestProxyWriteTimeout528=== RUN TestIsValidUploadKey529=== PAUSE TestIsValidUploadKey530=== RUN TestUploadHandlersRejectInvalidKeys531=== PAUSE TestUploadHandlersRejectInvalidKeys532=== RUN TestUploadHandlersRejectOversizedBody533=== PAUSE TestUploadHandlersRejectOversizedBody534=== RUN TestService_cleanupPendingClosuresHandler535=== PAUSE TestService_cleanupPendingClosuresHandler536=== RUN TestService_createPendingClosureHandler537=== PAUSE TestService_createPendingClosureHandler538=== RUN TestService_verifyS3Integrity539=== PAUSE TestService_verifyS3Integrity540=== RUN TestCompleteMultipartUnregistered541=== PAUSE TestCompleteMultipartUnregistered542=== RUN TestCreatePendingClosure_SmallNARUsesSimplePUT543=== PAUSE TestCreatePendingClosure_SmallNARUsesSimplePUT544=== CONT TestCompleteMultipartUnregistered545=== CONT TestGenerateLandingPage546=== CONT TestReadProxyRootRedirectsToIndexHTML547=== CONT TestNARDeduplicationMetadataUploadBug548=== CONT TestService_verifyS3Integrity549=== CONT TestService_createPendingClosureHandler550=== CONT TestService_cleanupPendingClosuresHandler551=== CONT TestUploadHandlersRejectOversizedBody552=== CONT TestUploadHandlersRejectInvalidKeys553=== CONT TestIsValidUploadKey554=== RUN TestIsValidUploadKey/narinfo555=== CONT TestProxyWriteTimeout556=== CONT TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle557=== CONT TestSkippedUploadsHandler558=== CONT TestCreatePendingClosureRejectsOversizedNAR559=== CONT TestCacheConfigHandlerMaxNarSize560=== CONT TestParseSize561=== CONT TestService_Rustfstest562=== CONT TestPresignedUploadRegisteredBeforeCommit563=== CONT TestCompletedNarNotReofferedAcrossClosures564=== CONT TestCompleteMultipartUpload_ErrorButObjectExists565=== CONT TestRedundantMultipartUpload566=== CONT TestReadProxyRangeRequest567=== CONT TestReadProxyDisabled568=== CONT TestService_AuthMiddleware569=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info570--- PASS: TestParseSize (0.00s)571=== CONT TestGCBugBareHashReferences572=== PAUSE TestIsValidUploadKey/narinfo573=== RUN TestIsValidUploadKey/nar_zst574=== PAUSE TestIsValidUploadKey/nar_zst575=== RUN TestIsValidUploadKey/nar_xz576=== PAUSE TestIsValidUploadKey/nar_xz577=== RUN TestIsValidUploadKey/nar_plain578=== PAUSE TestIsValidUploadKey/nar_plain5792026/08/27 09:37:27 INFO Received uploads request method=POST path=/api/pending_closures580=== RUN TestIsValidUploadKey/listing581--- PASS: TestCacheConfigHandlerMaxNarSize (0.00s)582=== CONT TestService_healthCheckHandler583--- PASS: TestCreatePendingClosureRejectsOversizedNAR (0.00s)5842026/08/27 09:37:27 INFO Client skipped oversized paths paths=3 nar_bytes=5000000000585=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info586=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal587=== CONT TestGracefulShutdownDrainsInflight588=== RUN TestProxyWriteTimeout/narinfo589=== PAUSE TestIsValidUploadKey/listing590=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal5912026/08/27 09:37:27 INFO Starting HTTP server address=127.0.0.1:40895592=== RUN TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key593=== RUN TestIsValidUploadKey/build_log594=== PAUSE TestProxyWriteTimeout/narinfo595=== RUN TestProxyWriteTimeout/1_GiB_nar596=== PAUSE TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key597=== RUN TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key598=== PAUSE TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key599=== CONT TestGCTaskStore_Fail600--- PASS: TestGCTaskStore_Fail (0.00s)601=== PAUSE TestIsValidUploadKey/build_log602=== RUN TestIsValidUploadKey/build_log_home-manager_file603=== PAUSE TestIsValidUploadKey/build_log_home-manager_file604=== RUN TestIsValidUploadKey/build_log_plus_in_name605=== PAUSE TestIsValidUploadKey/build_log_plus_in_name606=== RUN TestIsValidUploadKey/build_log_question_mark607=== PAUSE TestIsValidUploadKey/build_log_question_mark608=== RUN TestIsValidUploadKey/build_log_equals609=== PAUSE TestIsValidUploadKey/build_log_equals610=== RUN TestIsValidUploadKey/realisation611=== PAUSE TestIsValidUploadKey/realisation612=== RUN TestIsValidUploadKey/realisation_plus_in_output613=== PAUSE TestIsValidUploadKey/realisation_plus_in_output614=== RUN TestIsValidUploadKey/nix-cache-info615=== PAUSE TestIsValidUploadKey/nix-cache-info616=== RUN TestIsValidUploadKey/index.html617=== PAUSE TestIsValidUploadKey/index.html618=== RUN TestIsValidUploadKey/narinfo_key,_nar_type619=== PAUSE TestIsValidUploadKey/narinfo_key,_nar_type620=== RUN TestIsValidUploadKey/nar_key,_narinfo_type621=== PAUSE TestIsValidUploadKey/nar_key,_narinfo_type622=== RUN TestIsValidUploadKey/listing_key,_narinfo_type623=== PAUSE TestIsValidUploadKey/listing_key,_narinfo_type624=== RUN TestIsValidUploadKey/traversal625=== PAUSE TestIsValidUploadKey/traversal626=== RUN TestIsValidUploadKey/traversal_nar627=== PAUSE TestIsValidUploadKey/traversal_nar628=== RUN TestIsValidUploadKey/absolute629=== PAUSE TestIsValidUploadKey/absolute630=== RUN TestIsValidUploadKey/empty_key631=== PAUSE TestIsValidUploadKey/empty_key632=== RUN TestIsValidUploadKey/unknown_type633=== PAUSE TestIsValidUploadKey/unknown_type634=== CONT TestGCTaskStore_CompletedAllowsNewTask635=== CONT TestGCTaskStore_PhaseUpdates636=== CONT TestGCTaskStore_GetEmpty637=== CONT TestGCTaskStore_ConflictDifferentParams638=== CONT TestGCTaskStore_DeduplicateSameParams639=== CONT TestGCTaskStore_StartNew640=== CONT TestGCMetrics641--- PASS: TestGCTaskStore_CompletedAllowsNewTask (0.00s)642--- PASS: TestGCTaskStore_PhaseUpdates (0.00s)643--- PASS: TestGCTaskStore_GetEmpty (0.00s)644--- PASS: TestGCTaskStore_ConflictDifferentParams (0.00s)645--- PASS: TestGCTaskStore_DeduplicateSameParams (0.00s)646--- PASS: TestGCTaskStore_StartNew (0.00s)647=== CONT TestGCTaskStore_GetReturnsLatest648--- PASS: TestGCTaskStore_GetReturnsLatest (0.00s)649=== CONT TestClientCADerivations650=== PAUSE TestProxyWriteTimeout/1_GiB_nar651=== RUN TestProxyWriteTimeout/10_GiB_nar652=== PAUSE TestProxyWriteTimeout/10_GiB_nar653=== RUN TestProxyWriteTimeout/unknown_size654=== PAUSE TestProxyWriteTimeout/unknown_size655=== CONT TestPinProtectsFromGC6562026/08/27 09:37:27 INFO Shutdown signal received, draining in-flight requests timeout=10s657--- PASS: TestGenerateLandingPage (0.09s)658=== CONT TestClientWithDependencies659--- PASS: TestSkippedUploadsHandler (0.08s)660=== CONT TestClientMultipleUploads6612026-08-27 09:37:27.829 UTC [578] ERROR: relation "goose_db_version" does not exist at character 366622026-08-27 09:37:27.829 UTC [578] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC6632026-08-27 09:37:27.837 UTC [587] ERROR: relation "goose_db_version" does not exist at character 366642026-08-27 09:37:27.837 UTC [587] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC6652026-08-27 09:37:27.866 UTC [591] ERROR: relation "goose_db_version" does not exist at character 366662026-08-27 09:37:27.866 UTC [591] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC6672026-08-27 09:37:27.867 UTC [590] ERROR: relation "goose_db_version" does not exist at character 366682026-08-27 09:37:27.867 UTC [590] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC669--- PASS: TestGracefulShutdownDrainsInflight (0.15s)670=== CONT TestClientIntegration671=== RUN TestUploadHandlersRejectOversizedBody/request_more_parts672=== PAUSE TestUploadHandlersRejectOversizedBody/request_more_parts673=== RUN TestUploadHandlersRejectOversizedBody/create_pending_closure674=== PAUSE TestUploadHandlersRejectOversizedBody/create_pending_closure675=== RUN TestUploadHandlersRejectOversizedBody/complete_multipart676=== PAUSE TestUploadHandlersRejectOversizedBody/complete_multipart677=== CONT TestCreatePendingClosure_SmallNARUsesSimplePUT6782026/08/27 09:37:27 OK 20241026095416_initial_model.sql (96.62ms)6792026-08-27 09:37:27.939 UTC [594] ERROR: relation "goose_db_version" does not exist at character 366802026-08-27 09:37:27.939 UTC [594] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC6812026/08/27 09:37:27 OK 20241026095416_initial_model.sql (90.77ms)6822026/08/27 09:37:27 OK 20251210153512_drop_unused_gin_index.sql (6.59ms)6832026/08/27 09:37:27 OK 20251210153512_drop_unused_gin_index.sql (12.14ms)6842026-08-27 09:37:27.963 UTC [599] ERROR: relation "goose_db_version" does not exist at character 366852026-08-27 09:37:27.963 UTC [599] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC6862026/08/27 09:37:27 OK 20251218171726_add_pins.sql (19.08ms)6872026/08/27 09:37:27 OK 20241026095416_initial_model.sql (49.39ms)6882026/08/27 09:37:27 OK 20251218171726_add_pins.sql (8.16ms)6892026-08-27 09:37:27.970 UTC [600] ERROR: relation "goose_db_version" does not exist at character 366902026-08-27 09:37:27.970 UTC [600] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC6912026/08/27 09:37:27 OK 20251210153512_drop_unused_gin_index.sql (3.64ms)6922026/08/27 09:37:27 OK 20241026095416_initial_model.sql (36.36ms)6932026/08/27 09:37:27 OK 20260628120000_add_object_size_and_stats.sql (9.2ms)6942026/08/27 09:37:27 goose: successfully migrated database to version: 202606281200006952026/08/27 09:37:27 OK 20241026095416_initial_model.sql (16.54ms)6962026/08/27 09:37:27 OK 20260628120000_add_object_size_and_stats.sql (9.94ms)6972026/08/27 09:37:27 goose: successfully migrated database to version: 202606281200006982026/08/27 09:37:27 OK 20251210153512_drop_unused_gin_index.sql (4.96ms)6992026/08/27 09:37:27 OK 20251210153512_drop_unused_gin_index.sql (4.58ms)7002026/08/27 09:37:27 OK 1_commit_pending_closure.sql (7.42ms)7012026/08/27 09:37:27 OK 20251218171726_add_pins.sql (8.93ms)7022026/08/27 09:37:27 OK 1_commit_pending_closure.sql (6.24ms)7032026/08/27 09:37:27 OK 2_object_stats_trigger.sql (3ms)7042026/08/27 09:37:27 goose: up to current file version: 27052026/08/27 09:37:27 OK 2_object_stats_trigger.sql (2.52ms)7062026/08/27 09:37:27 goose: up to current file version: 27072026/08/27 09:37:27 OK 20251218171726_add_pins.sql (7.32ms)7082026/08/27 09:37:27 OK 20251218171726_add_pins.sql (8.72ms)7092026/08/27 09:37:27 OK 20241026095416_initial_model.sql (16.86ms)7102026-08-27 09:37:27.991 UTC [601] ERROR: relation "goose_db_version" does not exist at character 367112026-08-27 09:37:27.991 UTC [601] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC7122026-08-27 09:37:27.991 UTC [602] ERROR: relation "goose_db_version" does not exist at character 367132026-08-27 09:37:27.991 UTC [602] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC7142026-08-27 09:37:27.993 UTC [603] ERROR: relation "goose_db_version" does not exist at character 367152026-08-27 09:37:27.993 UTC [603] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC7162026-08-27 09:37:27.995 UTC [605] ERROR: relation "goose_db_version" does not exist at character 367172026-08-27 09:37:27.995 UTC [605] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC7182026-08-27 09:37:27.996 UTC [604] ERROR: relation "goose_db_version" does not exist at character 367192026-08-27 09:37:27.996 UTC [604] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC7202026/08/27 09:37:28 OK 20260628120000_add_object_size_and_stats.sql (19.88ms)7212026/08/27 09:37:28 goose: successfully migrated database to version: 202606281200007222026-08-27 09:37:28.003 UTC [606] ERROR: relation "goose_db_version" does not exist at character 367232026-08-27 09:37:28.003 UTC [606] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC7242026/08/27 09:37:28 OK 20241026095416_initial_model.sql (24.95ms)7252026/08/27 09:37:28 OK 20251210153512_drop_unused_gin_index.sql (14.2ms)7262026/08/27 09:37:28 OK 20260628120000_add_object_size_and_stats.sql (20.29ms)7272026/08/27 09:37:28 goose: successfully migrated database to version: 202606281200007282026/08/27 09:37:28 OK 20260628120000_add_object_size_and_stats.sql (20.27ms)7292026/08/27 09:37:28 goose: successfully migrated database to version: 20260628120000730--- PASS: TestReadProxyRootRedirectsToIndexHTML (0.27s)731=== CONT TestClientErrorHandling732=== RUN TestClientErrorHandling/InvalidStorePath733=== PAUSE TestClientErrorHandling/InvalidStorePath734=== RUN TestClientErrorHandling/InvalidAuthToken735=== PAUSE TestClientErrorHandling/InvalidAuthToken736=== RUN TestClientErrorHandling/ServerNotAvailable737=== PAUSE TestClientErrorHandling/ServerNotAvailable738=== CONT TestService_AuthMiddleware_OIDC7392026/08/27 09:37:28 OK 1_commit_pending_closure.sql (8.87ms)7402026/08/27 09:37:28 OK 20251210153512_drop_unused_gin_index.sql (6.16ms)7412026/08/27 09:37:28 INFO OIDC provider initialized name=test7422026/08/27 09:37:28 OK 1_commit_pending_closure.sql (6.23ms)7432026/08/27 09:37:28 OK 1_commit_pending_closure.sql (6.43ms)7442026/08/27 09:37:28 OK 20251218171726_add_pins.sql (9.42ms)7452026/08/27 09:37:28 OK 2_object_stats_trigger.sql (4.84ms)7462026/08/27 09:37:28 goose: up to current file version: 27472026/08/27 09:37:28 OK 2_object_stats_trigger.sql (3.57ms)7482026/08/27 09:37:28 goose: up to current file version: 27492026/08/27 09:37:28 OK 20251218171726_add_pins.sql (7.22ms)7502026/08/27 09:37:28 OK 2_object_stats_trigger.sql (3.5ms)7512026/08/27 09:37:28 goose: up to current file version: 27522026/08/27 09:37:28 OK 20260628120000_add_object_size_and_stats.sql (7.07ms)7532026/08/27 09:37:28 goose: successfully migrated database to version: 202606281200007542026/08/27 09:37:28 OK 20260628120000_add_object_size_and_stats.sql (9.26ms)7552026/08/27 09:37:28 goose: successfully migrated database to version: 202606281200007562026-08-27 09:37:28.029 UTC [608] ERROR: relation "goose_db_version" does not exist at character 367572026-08-27 09:37:28.029 UTC [608] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC7582026/08/27 09:37:28 OK 1_commit_pending_closure.sql (8.77ms)7592026/08/27 09:37:28 OK 20241026095416_initial_model.sql (17.33ms)7602026/08/27 09:37:28 OK 20241026095416_initial_model.sql (17.3ms)7612026-08-27 09:37:28.035 UTC [610] ERROR: relation "goose_db_version" does not exist at character 367622026-08-27 09:37:28.035 UTC [610] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC7632026/08/27 09:37:28 OK 1_commit_pending_closure.sql (8.54ms)7642026/08/27 09:37:28 OK 2_object_stats_trigger.sql (5.56ms)7652026/08/27 09:37:28 goose: up to current file version: 27662026/08/27 09:37:28 OK 20251210153512_drop_unused_gin_index.sql (5.73ms)7672026/08/27 09:37:28 OK 20251210153512_drop_unused_gin_index.sql (5.75ms)7682026/08/27 09:37:28 OK 20241026095416_initial_model.sql (23.63ms)7692026/08/27 09:37:28 OK 20241026095416_initial_model.sql (22.34ms)7702026/08/27 09:37:28 INFO Received uploads request method=POST path=/api/pending_closures7712026/08/27 09:37:28 OK 2_object_stats_trigger.sql (14.2ms)7722026/08/27 09:37:28 goose: up to current file version: 27732026/08/27 09:37:28 OK 20241026095416_initial_model.sql (32.99ms)7742026/08/27 09:37:28 OK 20251210153512_drop_unused_gin_index.sql (12.86ms)7752026/08/27 09:37:28 OK 20251210153512_drop_unused_gin_index.sql (12.75ms)7762026/08/27 09:37:28 OK 20251218171726_add_pins.sql (17.5ms)7772026/08/27 09:37:28 OK 20241026095416_initial_model.sql (37.56ms)7782026/08/27 09:37:28 OK 20251218171726_add_pins.sql (17.47ms)7792026/08/27 09:37:28 OK 20251218171726_add_pins.sql (5.72ms)780--- PASS: TestService_Rustfstest (0.32s)781=== CONT TestParseSingleRange782=== RUN TestParseSingleRange/none783=== PAUSE TestParseSingleRange/none784=== RUN TestParseSingleRange/unknown_unit785=== PAUSE TestParseSingleRange/unknown_unit786=== RUN TestParseSingleRange/multi-range_ignored787=== PAUSE TestParseSingleRange/multi-range_ignored788=== RUN TestParseSingleRange/malformed_no_dash789=== PAUSE TestParseSingleRange/malformed_no_dash790=== RUN TestParseSingleRange/malformed_both_empty791=== PAUSE TestParseSingleRange/malformed_both_empty792=== RUN TestParseSingleRange/malformed_end_before_start7932026/08/27 09:37:28 OK 20251210153512_drop_unused_gin_index.sql (9.35ms)794=== PAUSE TestParseSingleRange/malformed_end_before_start795=== RUN TestParseSingleRange/closed796=== PAUSE TestParseSingleRange/closed797=== RUN TestParseSingleRange/open-ended798=== PAUSE TestParseSingleRange/open-ended799=== RUN TestParseSingleRange/end_clamped_to_size800=== PAUSE TestParseSingleRange/end_clamped_to_size801=== RUN TestParseSingleRange/suffix802=== PAUSE TestParseSingleRange/suffix803=== RUN TestParseSingleRange/suffix_exceeds_size804=== PAUSE TestParseSingleRange/suffix_exceeds_size805=== RUN TestParseSingleRange/single_byte806=== PAUSE TestParseSingleRange/single_byte807=== RUN TestParseSingleRange/start_past_EOF808=== PAUSE TestParseSingleRange/start_past_EOF809=== RUN TestParseSingleRange/start_far_past_EOF810=== PAUSE TestParseSingleRange/start_far_past_EOF811=== CONT TestCacheStatsHandler8122026/08/27 09:37:28 OK 20251218171726_add_pins.sql (12.37ms)8132026/08/27 09:37:28 OK 20251210153512_drop_unused_gin_index.sql (8.89ms)8142026/08/27 09:37:28 OK 20241026095416_initial_model.sql (20.38ms)8152026/08/27 09:37:28 OK 20260628120000_add_object_size_and_stats.sql (10.31ms)8162026/08/27 09:37:28 goose: successfully migrated database to version: 202606281200008172026/08/27 09:37:28 OK 20260628120000_add_object_size_and_stats.sql (10.37ms)8182026/08/27 09:37:28 goose: successfully migrated database to version: 202606281200008192026/08/27 09:37:28 OK 20260628120000_add_object_size_and_stats.sql (12.42ms)8202026/08/27 09:37:28 goose: successfully migrated database to version: 202606281200008212026-08-27 09:37:28.073 UTC [614] ERROR: relation "goose_db_version" does not exist at character 368222026-08-27 09:37:28.073 UTC [614] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC8232026/08/27 09:37:28 OK 1_commit_pending_closure.sql (6.8ms)8242026/08/27 09:37:28 OK 20251218171726_add_pins.sql (8.59ms)8252026/08/27 09:37:28 OK 1_commit_pending_closure.sql (6.88ms)8262026/08/27 09:37:28 OK 20251210153512_drop_unused_gin_index.sql (7.32ms)8272026/08/27 09:37:28 OK 20251218171726_add_pins.sql (12.35ms)8282026-08-27 09:37:28.075 UTC [618] ERROR: relation "goose_db_version" does not exist at character 368292026-08-27 09:37:28.075 UTC [618] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC8302026/08/27 09:37:28 OK 20260628120000_add_object_size_and_stats.sql (10.7ms)8312026/08/27 09:37:28 goose: successfully migrated database to version: 202606281200008322026/08/27 09:37:28 OK 20241026095416_initial_model.sql (17.7ms)8332026-08-27 09:37:28.076 UTC [615] ERROR: relation "goose_db_version" does not exist at character 368342026-08-27 09:37:28.076 UTC [615] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC8352026-08-27 09:37:28.077 UTC [616] ERROR: relation "goose_db_version" does not exist at character 368362026-08-27 09:37:28.077 UTC [616] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC8372026/08/27 09:37:28 OK 2_object_stats_trigger.sql (4.05ms)8382026/08/27 09:37:28 goose: up to current file version: 28392026/08/27 09:37:28 OK 2_object_stats_trigger.sql (4.39ms)8402026/08/27 09:37:28 goose: up to current file version: 28412026/08/27 09:37:28 OK 1_commit_pending_closure.sql (7.44ms)8422026-08-27 09:37:28.078 UTC [619] ERROR: relation "goose_db_version" does not exist at character 368432026-08-27 09:37:28.078 UTC [619] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC8442026-08-27 09:37:28.078 UTC [617] ERROR: relation "goose_db_version" does not exist at character 368452026-08-27 09:37:28.078 UTC [617] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC8462026-08-27 09:37:28.079 UTC [620] ERROR: relation "goose_db_version" does not exist at character 368472026-08-27 09:37:28.079 UTC [620] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC8482026-08-27 09:37:28.080 UTC [622] ERROR: relation "goose_db_version" does not exist at character 368492026-08-27 09:37:28.080 UTC [622] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC8502026/08/27 09:37:28 OK 20251210153512_drop_unused_gin_index.sql (4.06ms)8512026/08/27 09:37:28 OK 1_commit_pending_closure.sql (5.33ms)8522026/08/27 09:37:28 OK 20251218171726_add_pins.sql (7.1ms)8532026/08/27 09:37:28 OK 2_object_stats_trigger.sql (3.98ms)8542026/08/27 09:37:28 OK 20260628120000_add_object_size_and_stats.sql (8.41ms)8552026/08/27 09:37:28 goose: successfully migrated database to version: 202606281200008562026/08/27 09:37:28 goose: up to current file version: 28572026/08/27 09:37:28 OK 20260628120000_add_object_size_and_stats.sql (8.15ms)8582026/08/27 09:37:28 goose: successfully migrated database to version: 202606281200008592026-08-27 09:37:28.097 UTC [623] ERROR: relation "goose_db_version" does not exist at character 368602026-08-27 09:37:28.097 UTC [623] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC8612026/08/27 09:37:28 OK 2_object_stats_trigger.sql (16.31ms)8622026/08/27 09:37:28 goose: up to current file version: 28632026/08/27 09:37:28 OK 1_commit_pending_closure.sql (21.05ms)8642026/08/27 09:37:28 OK 20241026095416_initial_model.sql (23.19ms)8652026/08/27 09:37:28 OK 1_commit_pending_closure.sql (21.47ms)8662026/08/27 09:37:28 OK 20251218171726_add_pins.sql (24.09ms)8672026/08/27 09:37:28 OK 20260628120000_add_object_size_and_stats.sql (22.68ms)8682026/08/27 09:37:28 goose: successfully migrated database to version: 202606281200008692026/08/27 09:37:28 OK 2_object_stats_trigger.sql (3.44ms)8702026/08/27 09:37:28 goose: up to current file version: 28712026/08/27 09:37:28 OK 2_object_stats_trigger.sql (4.39ms)8722026/08/27 09:37:28 goose: up to current file version: 28732026/08/27 09:37:28 OK 20251210153512_drop_unused_gin_index.sql (3.98ms)8742026/08/27 09:37:28 OK 20241026095416_initial_model.sql (14.55ms)8752026/08/27 09:37:28 OK 1_commit_pending_closure.sql (5.13ms)8762026/08/27 09:37:28 OK 20260628120000_add_object_size_and_stats.sql (7.54ms)8772026/08/27 09:37:28 goose: successfully migrated database to version: 202606281200008782026/08/27 09:37:28 OK 20241026095416_initial_model.sql (12.83ms)8792026/08/27 09:37:28 OK 20251210153512_drop_unused_gin_index.sql (4.09ms)8802026/08/27 09:37:28 OK 2_object_stats_trigger.sql (4.1ms)8812026/08/27 09:37:28 goose: up to current file version: 28822026/08/27 09:37:28 OK 20251218171726_add_pins.sql (6.25ms)8832026/08/27 09:37:28 OK 1_commit_pending_closure.sql (2.9ms)8842026/08/27 09:37:28 OK 20251210153512_drop_unused_gin_index.sql (3.51ms)8852026-08-27 09:37:28.118 UTC [624] ERROR: relation "goose_db_version" does not exist at character 368862026-08-27 09:37:28.118 UTC [624] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC8872026/08/27 09:37:28 OK 2_object_stats_trigger.sql (3.39ms)8882026/08/27 09:37:28 goose: up to current file version: 28892026/08/27 09:37:28 OK 20241026095416_initial_model.sql (12.77ms)8902026/08/27 09:37:28 OK 20251218171726_add_pins.sql (6.35ms)8912026/08/27 09:37:28 OK 20241026095416_initial_model.sql (14.71ms)8922026/08/27 09:37:28 OK 20241026095416_initial_model.sql (12.67ms)8932026/08/27 09:37:28 OK 20241026095416_initial_model.sql (13.12ms)8942026/08/27 09:37:28 OK 20241026095416_initial_model.sql (15.94ms)8952026/08/27 09:37:28 OK 20260628120000_add_object_size_and_stats.sql (5.91ms)8962026/08/27 09:37:28 goose: successfully migrated database to version: 202606281200008972026/08/27 09:37:28 OK 20251218171726_add_pins.sql (5.51ms)8982026/08/27 09:37:28 OK 20251210153512_drop_unused_gin_index.sql (3.89ms)8992026/08/27 09:37:28 OK 20241026095416_initial_model.sql (16.8ms)9002026/08/27 09:37:28 OK 20251210153512_drop_unused_gin_index.sql (3.89ms)9012026/08/27 09:37:28 OK 20251210153512_drop_unused_gin_index.sql (3.93ms)9022026/08/27 09:37:28 OK 20251210153512_drop_unused_gin_index.sql (4ms)9032026/08/27 09:37:28 OK 1_commit_pending_closure.sql (3.93ms)9042026/08/27 09:37:28 OK 20260628120000_add_object_size_and_stats.sql (5.23ms)9052026/08/27 09:37:28 goose: successfully migrated database to version: 202606281200009062026/08/27 09:37:28 OK 20251210153512_drop_unused_gin_index.sql (4.34ms)9072026/08/27 09:37:28 OK 20251210153512_drop_unused_gin_index.sql (4.18ms)9082026/08/27 09:37:28 OK 2_object_stats_trigger.sql (2.85ms)9092026/08/27 09:37:28 goose: up to current file version: 29102026/08/27 09:37:28 OK 20251218171726_add_pins.sql (5.72ms)9112026/08/27 09:37:28 OK 20260628120000_add_object_size_and_stats.sql (6.04ms)9122026/08/27 09:37:28 goose: successfully migrated database to version: 202606281200009132026/08/27 09:37:28 OK 1_commit_pending_closure.sql (4.17ms)9142026/08/27 09:37:28 OK 20251218171726_add_pins.sql (5.31ms)9152026/08/27 09:37:28 OK 20251218171726_add_pins.sql (5.37ms)9162026/08/27 09:37:28 OK 20251218171726_add_pins.sql (5.97ms)9172026/08/27 09:37:28 OK 20251218171726_add_pins.sql (7.26ms)9182026/08/27 09:37:28 OK 2_object_stats_trigger.sql (3.66ms)9192026/08/27 09:37:28 goose: up to current file version: 29202026/08/27 09:37:28 OK 20251218171726_add_pins.sql (6.18ms)9212026/08/27 09:37:28 OK 1_commit_pending_closure.sql (4.56ms)9222026/08/27 09:37:28 OK 20260628120000_add_object_size_and_stats.sql (5.62ms)9232026/08/27 09:37:28 goose: successfully migrated database to version: 202606281200009242026/08/27 09:37:28 OK 20260628120000_add_object_size_and_stats.sql (6.75ms)9252026/08/27 09:37:28 goose: successfully migrated database to version: 202606281200009262026/08/27 09:37:28 OK 20260628120000_add_object_size_and_stats.sql (5.7ms)9272026/08/27 09:37:28 goose: successfully migrated database to version: 202606281200009282026/08/27 09:37:28 OK 2_object_stats_trigger.sql (3.86ms)9292026/08/27 09:37:28 goose: up to current file version: 29302026/08/27 09:37:28 OK 20260628120000_add_object_size_and_stats.sql (6.02ms)9312026/08/27 09:37:28 goose: successfully migrated database to version: 202606281200009322026/08/27 09:37:28 OK 20260628120000_add_object_size_and_stats.sql (5.3ms)9332026/08/27 09:37:28 goose: successfully migrated database to version: 202606281200009342026/08/27 09:37:28 OK 20260628120000_add_object_size_and_stats.sql (6.98ms)9352026/08/27 09:37:28 goose: successfully migrated database to version: 202606281200009362026/08/27 09:37:28 OK 20241026095416_initial_model.sql (11.16ms)9372026/08/27 09:37:28 OK 1_commit_pending_closure.sql (3.47ms)9382026/08/27 09:37:28 OK 1_commit_pending_closure.sql (4.34ms)9392026/08/27 09:37:28 OK 1_commit_pending_closure.sql (4.19ms)9402026/08/27 09:37:28 OK 1_commit_pending_closure.sql (2.3ms)9412026/08/27 09:37:28 OK 2_object_stats_trigger.sql (1.01ms)9422026/08/27 09:37:28 goose: up to current file version: 29432026/08/27 09:37:28 OK 20251210153512_drop_unused_gin_index.sql (1.48ms)9442026/08/27 09:37:28 OK 1_commit_pending_closure.sql (1.87ms)9452026/08/27 09:37:28 OK 2_object_stats_trigger.sql (982.67µs)9462026/08/27 09:37:28 goose: up to current file version: 29472026/08/27 09:37:28 OK 2_object_stats_trigger.sql (1.04ms)9482026/08/27 09:37:28 goose: up to current file version: 29492026/08/27 09:37:28 OK 1_commit_pending_closure.sql (2.04ms)9502026/08/27 09:37:28 OK 2_object_stats_trigger.sql (796.35µs)9512026/08/27 09:37:28 goose: up to current file version: 29522026/08/27 09:37:28 OK 2_object_stats_trigger.sql (803.63µs)9532026/08/27 09:37:28 goose: up to current file version: 29542026/08/27 09:37:28 OK 2_object_stats_trigger.sql (1.34ms)9552026/08/27 09:37:28 goose: up to current file version: 29562026/08/27 09:37:28 OK 20251218171726_add_pins.sql (3ms)9572026/08/27 09:37:28 OK 20260628120000_add_object_size_and_stats.sql (2.64ms)9582026/08/27 09:37:28 goose: successfully migrated database to version: 202606281200009592026/08/27 09:37:28 OK 1_commit_pending_closure.sql (1.47ms)9602026/08/27 09:37:28 OK 2_object_stats_trigger.sql (753.07µs)9612026/08/27 09:37:28 goose: up to current file version: 29622026-08-27 09:37:28.150 UTC [625] ERROR: relation "goose_db_version" does not exist at character 369632026-08-27 09:37:28.150 UTC [625] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC9642026/08/27 09:37:28 OK 20241026095416_initial_model.sql (9.72ms)9652026/08/27 09:37:28 OK 20251210153512_drop_unused_gin_index.sql (1.37ms)9662026/08/27 09:37:28 OK 20251218171726_add_pins.sql (2.86ms)9672026/08/27 09:37:28 OK 20260628120000_add_object_size_and_stats.sql (3ms)9682026/08/27 09:37:28 goose: successfully migrated database to version: 202606281200009692026/08/27 09:37:28 OK 1_commit_pending_closure.sql (1.96ms)9702026/08/27 09:37:28 OK 2_object_stats_trigger.sql (792.87µs)9712026/08/27 09:37:28 goose: up to current file version: 2972{"timestamp":"2026-08-27T09:37:28.774067966Z","level":"ERROR","duration":"1.12891ms","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(308)"}973{"timestamp":"2026-08-27T09:37:28.774064906Z","level":"ERROR","duration":"1.495893ms","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(312)"}974{"timestamp":"2026-08-27T09:37:28.774281828Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"aedb7d21-29fe-4b90-bb41-a4ef4659721b","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket15/","status_code":503,"duration_ms":1,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(312)"}975{"timestamp":"2026-08-27T09:37:28.774279608Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"af73e614-172c-43d0-8352-aa7ca7f6c86c","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket17/","status_code":503,"duration_ms":1,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(308)"}976{"timestamp":"2026-08-27T09:37:28.779389752Z","level":"ERROR","duration":"5.719689ms","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(305)"}977{"timestamp":"2026-08-27T09:37:28.779494213Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"12a70009-29d6-4353-9bff-ca5d513c7e10","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket20/","status_code":503,"duration_ms":5,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(305)"}978{"timestamp":"2026-08-27T09:37:28.780869585Z","level":"ERROR","duration":"7.511845ms","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(286)"}979{"timestamp":"2026-08-27T09:37:28.780907105Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"a40dc6a0-cbbf-461f-8401-4470c622678e","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket16/","status_code":503,"duration_ms":8,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(286)"}980{"timestamp":"2026-08-27T09:37:28.78146805Z","level":"ERROR","duration":"7.889528ms","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(294)"}981{"timestamp":"2026-08-27T09:37:28.78150143Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"a88c7c17-cf19-4fc7-9f6b-e5659620d157","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket18/","status_code":503,"duration_ms":8,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(294)"}982{"timestamp":"2026-08-27T09:37:28.781721012Z","level":"ERROR","duration":"1.398892ms","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(311)"}983{"timestamp":"2026-08-27T09:37:28.781752013Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"bd703d5d-f324-49a6-911b-23e64abbc3bf","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket23/","status_code":503,"duration_ms":1,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(311)"}984{"timestamp":"2026-08-27T09:37:28.781912214Z","level":"ERROR","duration":"1.705735ms","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(307)"}985{"timestamp":"2026-08-27T09:37:28.781943094Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"1aab333a-d0a7-42b4-9c42-e79508a75072","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket21/","status_code":503,"duration_ms":8,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(307)"}986{"timestamp":"2026-08-27T09:37:28.781971754Z","level":"ERROR","duration":"1.747995ms","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(322)"}987{"timestamp":"2026-08-27T09:37:28.782266637Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"b8e64985-de6c-4575-b37c-8933fc49343c","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket19/","status_code":503,"duration_ms":2,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(322)"}988{"timestamp":"2026-08-27T09:37:28.782451459Z","level":"ERROR","duration":"2.219559ms","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(259)"}989{"timestamp":"2026-08-27T09:37:28.782545599Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"1a97c215-730a-4e0e-9213-176df165474c","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket22/","status_code":503,"duration_ms":2,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(259)"}990{"timestamp":"2026-08-27T09:37:28.783149485Z","level":"ERROR","duration":"917.648µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(272)"}991{"timestamp":"2026-08-27T09:37:28.783184985Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"ec3f082a-ccf8-4b8c-9f23-953f521dc98a","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket24/","status_code":503,"duration_ms":1,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(272)"}992{"timestamp":"2026-08-27T09:37:28.783701529Z","level":"ERROR","duration":"1.870096ms","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(263)"}993{"timestamp":"2026-08-27T09:37:28.78374131Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"61f44708-7085-41f8-b397-639a1141446d","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket26/","status_code":503,"duration_ms":9,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(263)"}994{"timestamp":"2026-08-27T09:37:28.792041042Z","level":"ERROR","duration":"791.967µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(272)"}995{"timestamp":"2026-08-27T09:37:28.792109742Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"ba02db96-95b7-4e9e-a4c2-5b64f5cdf3d9","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket19/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(272)"}996--- PASS: TestCacheStatsHandler (0.74s)997=== CONT TestReadProxyConditionalGet9982026/08/27 09:37:28 INFO Received uploads request method=POST path=/api/pending_closures9992026/08/27 09:37:28 INFO Received uploads request method=POST path=/api/pending_closures10002026/08/27 09:37:28 INFO Received uploads request method=POST path=/api/pending_closures1001{"timestamp":"2026-08-27T09:37:28.825301409Z","level":"ERROR","duration":"239.722µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(259)"}1002{"timestamp":"2026-08-27T09:37:28.82537963Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"07256032-bde3-4481-97be-e2c3fcfc09fe","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket23/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(259)"}1003{"timestamp":"2026-08-27T09:37:28.825731053Z","level":"ERROR","duration":"101.981µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(259)"}1004{"timestamp":"2026-08-27T09:37:28.825774073Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"e447acc6-8bee-4ed4-ac83-a77907696a84","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket21/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(259)"}10052026/08/27 09:37:28 INFO Received uploads request method=POST path=/api/pending_closures1006--- PASS: TestCreatePendingClosure_SmallNARUsesSimplePUT (0.90s)1007=== CONT TestCacheConfigHandler1008=== RUN TestCacheConfigHandler/full_config,_no_issuer1009=== PAUSE TestCacheConfigHandler/full_config,_no_issuer1010=== RUN TestCacheConfigHandler/no_cache_url_configured1011=== PAUSE TestCacheConfigHandler/no_cache_url_configured1012=== RUN TestCacheConfigHandler/no_signing_keys1013=== PAUSE TestCacheConfigHandler/no_signing_keys1014=== RUN TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1015=== PAUSE TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1016=== CONT TestReadProxyHead10172026-08-27 09:37:28.880 UTC [631] ERROR: relation "goose_db_version" does not exist at character 3610182026-08-27 09:37:28.880 UTC [631] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC10192026/08/27 09:37:28 OK 20241026095416_initial_model.sql (11.51ms)10202026/08/27 09:37:28 OK 20251210153512_drop_unused_gin_index.sql (1.37ms)10212026/08/27 09:37:28 OK 20251218171726_add_pins.sql (3.42ms)10222026/08/27 09:37:28 OK 20260628120000_add_object_size_and_stats.sql (2.98ms)10232026/08/27 09:37:28 goose: successfully migrated database to version: 2026062812000010242026/08/27 09:37:28 OK 1_commit_pending_closure.sql (1.97ms)10252026-08-27 09:37:28.912 UTC [632] ERROR: relation "goose_db_version" does not exist at character 3610262026-08-27 09:37:28.912 UTC [632] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC10272026/08/27 09:37:28 OK 2_object_stats_trigger.sql (1.27ms)10282026/08/27 09:37:28 goose: up to current file version: 210292026/08/27 09:37:28 OK 20241026095416_initial_model.sql (10.05ms)10302026/08/27 09:37:28 OK 20251210153512_drop_unused_gin_index.sql (1.31ms)10312026/08/27 09:37:28 OK 20251218171726_add_pins.sql (3.65ms)10322026/08/27 09:37:28 OK 20260628120000_add_object_size_and_stats.sql (3.51ms)10332026/08/27 09:37:28 goose: successfully migrated database to version: 2026062812000010342026/08/27 09:37:28 OK 1_commit_pending_closure.sql (1.86ms)10352026/08/27 09:37:28 OK 2_object_stats_trigger.sql (867.09µs)10362026/08/27 09:37:28 goose: up to current file version: 21037--- PASS: TestGCBugBareHashReferences (1.27s)1038=== CONT TestReadProxyNarStreaming10392026-08-27 09:37:29.081 UTC [635] ERROR: relation "goose_db_version" does not exist at character 3610402026-08-27 09:37:29.081 UTC [635] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC10412026/08/27 09:37:29 OK 20241026095416_initial_model.sql (9.82ms)10422026/08/27 09:37:29 OK 20251210153512_drop_unused_gin_index.sql (1.3ms)10432026/08/27 09:37:29 OK 20251218171726_add_pins.sql (4.31ms)10442026/08/27 09:37:29 OK 20260628120000_add_object_size_and_stats.sql (3.02ms)10452026/08/27 09:37:29 goose: successfully migrated database to version: 2026062812000010462026/08/27 09:37:29 OK 1_commit_pending_closure.sql (2.91ms)10472026/08/27 09:37:29 OK 2_object_stats_trigger.sql (887.47µs)10482026/08/27 09:37:29 goose: up to current file version: 21049{"timestamp":"2026-08-27T09:37:29.166353199Z","level":"ERROR","duration":"197.541µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(300)"}1050{"timestamp":"2026-08-27T09:37:29.166325899Z","level":"ERROR","duration":"173.501µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(294)"}1051{"timestamp":"2026-08-27T09:37:29.1663924Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"9dac45fc-deb3-4e09-a651-894408a7eebb","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket15/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(300)"}1052{"timestamp":"2026-08-27T09:37:29.16639714Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"c22d1801-a38f-4b67-84be-8eca1fb55499","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket20/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(294)"}1053{"timestamp":"2026-08-27T09:37:29.166515161Z","level":"ERROR","duration":"385.864µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(288)"}1054{"timestamp":"2026-08-27T09:37:29.166549361Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"17e7af55-4eeb-4e86-b8e5-61682185c2d8","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket19/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(288)"}1055{"timestamp":"2026-08-27T09:37:29.166859824Z","level":"ERROR","duration":"725.347µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(272)"}1056{"timestamp":"2026-08-27T09:37:29.166918684Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"50ef50bd-1873-43d8-ad90-c8823c1745d1","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket23/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(272)"}1057{"timestamp":"2026-08-27T09:37:29.166952845Z","level":"ERROR","duration":"456.324µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(286)"}1058{"timestamp":"2026-08-27T09:37:29.166990265Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"9f575d4f-f4a5-48e7-8de5-8a2a593ed056","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket30/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(286)"}1059{"timestamp":"2026-08-27T09:37:29.167247767Z","level":"ERROR","duration":"344.083µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(292)"}1060{"timestamp":"2026-08-27T09:37:29.167273087Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"ad12ef92-ac2a-4817-8f71-93788f7043c4","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket28/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(292)"}1061{"timestamp":"2026-08-27T09:37:29.16757385Z","level":"ERROR","duration":"909.908µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(266)"}1062{"timestamp":"2026-08-27T09:37:29.16757869Z","level":"ERROR","duration":"153.761µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(300)"}1063{"timestamp":"2026-08-27T09:37:29.16761083Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"5d9c1ff4-31d7-463e-b206-bc4001ddfbb1","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket16/","status_code":503,"duration_ms":1,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(266)"}1064{"timestamp":"2026-08-27T09:37:29.16761301Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"0c2eee5c-21ff-4cf7-b1ee-868ccf6abd67","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket21/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(300)"}1065{"timestamp":"2026-08-27T09:37:29.168031414Z","level":"ERROR","duration":"372.523µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(305)"}1066{"timestamp":"2026-08-27T09:37:29.168085534Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"61a77b6f-6ea0-4df1-9548-dafc5bd5b3ea","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket29/","status_code":503,"duration_ms":1,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(305)"}1067{"timestamp":"2026-08-27T09:37:29.168597179Z","level":"ERROR","duration":"1.026769ms","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(265)"}1068{"timestamp":"2026-08-27T09:37:29.168638039Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"183f0534-be65-4910-93f4-bd5b97dde137","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket3/","status_code":503,"duration_ms":1129,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(265)"}1069{"timestamp":"2026-08-27T09:37:29.168864861Z","level":"ERROR","duration":"261.302µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(265)"}1070{"timestamp":"2026-08-27T09:37:29.168883701Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"cab6ca45-c150-4f46-ab82-d847eda0d4f8","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket10/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(265)"}1071{"timestamp":"2026-08-27T09:37:29.171785586Z","level":"ERROR","duration":"121.621µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(307)"}1072{"timestamp":"2026-08-27T09:37:29.171822327Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"f952aef0-2578-444b-896e-99f6bc3babcb","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket15/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(307)"}1073{"timestamp":"2026-08-27T09:37:29.172014808Z","level":"ERROR","duration":"628.945µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(262)"}1074{"timestamp":"2026-08-27T09:37:29.172058169Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"b361dd46-318d-4d03-bbd9-a8eb79cd46d0","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket25/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(262)"}10752026/08/27 09:37:29 INFO Created nix-cache-info in bucket bucket=bucket610762026/08/27 09:37:29 INFO Received complete multipart upload request method=POST path=/api/multipart/complete10772026/08/27 09:37:29 ERROR Failed to get multipart upload error="no rows in result set" upload_id=does-not-exist object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst1078--- PASS: TestCompleteMultipartUnregistered (1.44s)1079=== CONT TestReadProxyNarinfoAlreadyDecompressed10802026/08/27 09:37:29 INFO Registered completed upload object_key=nar/jjjjjjjjjjjjjjjjjjjjjjjjjjjjjj0100000000000000000000.nar.zst10812026/08/27 09:37:29 INFO Received uploads request method=POST path=/api/pending_closures1082--- PASS: TestPresignedUploadRegisteredBeforeCommit (1.44s)1083=== CONT TestReadProxyInvalidPath10842026/08/27 09:37:29 INFO Created nix-cache-info in bucket bucket=bucket241085=== RUN TestService_AuthMiddleware_OIDC/valid_OIDC_token1086=== PAUSE TestService_AuthMiddleware_OIDC/valid_OIDC_token1087=== RUN TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1088=== PAUSE TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1089=== RUN TestService_AuthMiddleware_OIDC/malformed_token_rejected1090=== PAUSE TestService_AuthMiddleware_OIDC/malformed_token_rejected1091=== RUN TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1092=== PAUSE TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1093=== CONT TestReadProxyNarinfo1094--- PASS: TestService_healthCheckHandler (1.46s)1095=== CONT TestReadProxy4041096=== NAME TestNARDeduplicationMetadataUploadBug1097 metadata_upload_test.go:48: First store path: /build/TestNARDeduplicationMetadataUploadBug3130037694/001/store/f1hm85hxla58d8kd8di8sx7v41byw1x4-file1.txt1098=== NAME TestClientMultipleUploads1099 client_integration_test.go:338: Created store path 0: /build/TestClientMultipleUploads36095165/001/store/nkiivlqnj5r3xgcay3z3pz45126i4zxp-test-file-0.txt1100--- PASS: TestReadProxyHead (0.39s)1101=== CONT TestIsValidCachePath1102=== RUN TestIsValidCachePath/narinfo1103=== PAUSE TestIsValidCachePath/narinfo1104=== RUN TestIsValidCachePath/narinfo_all_nix_base32_chars1105=== PAUSE TestIsValidCachePath/narinfo_all_nix_base32_chars1106=== RUN TestIsValidCachePath/nar_zst1107=== PAUSE TestIsValidCachePath/nar_zst1108=== RUN TestIsValidCachePath/nar_xz1109=== PAUSE TestIsValidCachePath/nar_xz1110=== RUN TestIsValidCachePath/nar_bz21111=== PAUSE TestIsValidCachePath/nar_bz21112=== RUN TestIsValidCachePath/nar_uncompressed1113=== PAUSE TestIsValidCachePath/nar_uncompressed1114=== RUN TestIsValidCachePath/ls1115=== PAUSE TestIsValidCachePath/ls1116=== RUN TestIsValidCachePath/log1117=== PAUSE TestIsValidCachePath/log1118=== RUN TestIsValidCachePath/realisation1119=== PAUSE TestIsValidCachePath/realisation1120=== RUN TestIsValidCachePath/nix-cache-info1121=== PAUSE TestIsValidCachePath/nix-cache-info1122=== RUN TestIsValidCachePath/index.html1123=== PAUSE TestIsValidCachePath/index.html1124=== RUN TestIsValidCachePath/traversal_parent1125=== PAUSE TestIsValidCachePath/traversal_parent1126=== RUN TestIsValidCachePath/traversal_in_middle1127=== PAUSE TestIsValidCachePath/traversal_in_middle1128=== RUN TestIsValidCachePath/invalid_char_e1129=== PAUSE TestIsValidCachePath/invalid_char_e1130=== RUN TestIsValidCachePath/invalid_char_u1131=== PAUSE TestIsValidCachePath/invalid_char_u1132=== RUN TestIsValidCachePath/random_path1133=== PAUSE TestIsValidCachePath/random_path1134=== RUN TestIsValidCachePath/empty1135=== PAUSE TestIsValidCachePath/empty1136=== RUN TestIsValidCachePath/leading_slash1137=== PAUSE TestIsValidCachePath/leading_slash1138=== RUN TestIsValidCachePath/wrong_extension1139=== PAUSE TestIsValidCachePath/wrong_extension1140=== RUN TestIsValidCachePath/short_hash1141=== PAUSE TestIsValidCachePath/short_hash1142=== CONT TestServerTLSConfig1143=== RUN TestServerTLSConfig/no_client_CA1144=== PAUSE TestServerTLSConfig/no_client_CA1145=== RUN TestServerTLSConfig/missing_CA_file1146=== PAUSE TestServerTLSConfig/missing_CA_file1147=== RUN TestServerTLSConfig/not_a_PEM_file1148=== PAUSE TestServerTLSConfig/not_a_PEM_file1149=== CONT TestMultipartCleanup1150=== NAME TestClientMultipleUploads1151 client_integration_test.go:338: Created store path 1: /build/TestClientMultipleUploads36095165/001/store/nm9hfkswaxjnw9grpp487abkvnawc9h3-test-file-1.txt11522026-08-27 09:37:29.276 UTC [716] ERROR: relation "goose_db_version" does not exist at character 3611532026-08-27 09:37:29.276 UTC [716] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC11542026-08-27 09:37:29.280 UTC [718] ERROR: relation "goose_db_version" does not exist at character 3611552026-08-27 09:37:29.280 UTC [718] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC11562026/08/27 09:37:29 WARN Failed to fetch cache config, uploading without size limit error="fetching cache config: server returned 404: 404 page not found\n"11572026/08/27 09:37:29 OK 20241026095416_initial_model.sql (18.24ms)11582026/08/27 09:37:29 OK 20241026095416_initial_model.sql (15.21ms)11592026-08-27 09:37:29.306 UTC [738] ERROR: relation "goose_db_version" does not exist at character 3611602026-08-27 09:37:29.306 UTC [738] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC11612026/08/27 09:37:29 OK 20251210153512_drop_unused_gin_index.sql (3.5ms)11622026/08/27 09:37:29 OK 20251210153512_drop_unused_gin_index.sql (2.17ms)1163 client_integration_test.go:338: Created store path 2: /build/TestClientMultipleUploads36095165/001/store/986k8mrqw3di3kqjab88c9dskmyzgv3a-test-file-2.txt11642026/08/27 09:37:29 OK 20251218171726_add_pins.sql (4.37ms)11652026/08/27 09:37:29 OK 20251218171726_add_pins.sql (3.95ms)11662026/08/27 09:37:29 OK 20260628120000_add_object_size_and_stats.sql (4.34ms)11672026/08/27 09:37:29 goose: successfully migrated database to version: 2026062812000011682026/08/27 09:37:29 OK 20260628120000_add_object_size_and_stats.sql (4.32ms)11692026/08/27 09:37:29 goose: successfully migrated database to version: 2026062812000011702026-08-27 09:37:29.319 UTC [755] ERROR: relation "goose_db_version" does not exist at character 3611712026-08-27 09:37:29.319 UTC [755] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC11722026/08/27 09:37:29 OK 1_commit_pending_closure.sql (2.23ms)11732026/08/27 09:37:29 OK 1_commit_pending_closure.sql (4.17ms)11742026/08/27 09:37:29 OK 2_object_stats_trigger.sql (2.52ms)11752026/08/27 09:37:29 goose: up to current file version: 21176{"timestamp":"2026-08-27T09:37:29.32360472Z","level":"ERROR","duration":"188.862µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(266)"}1177{"timestamp":"2026-08-27T09:37:29.32366646Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"7f7fe648-7f73-4115-8a96-7ad89fd6225c","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket30/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(266)"}11782026/08/27 09:37:29 OK 2_object_stats_trigger.sql (1.94ms)11792026/08/27 09:37:29 goose: up to current file version: 21180{"timestamp":"2026-08-27T09:37:29.324343446Z","level":"ERROR","duration":"382.063µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(266)"}1181{"timestamp":"2026-08-27T09:37:29.324388026Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"dfca2af4-6045-456b-bd90-8131d9b2a2cc","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket31/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(266)"}11822026-08-27 09:37:29.325 UTC [757] ERROR: relation "goose_db_version" does not exist at character 3611832026-08-27 09:37:29.325 UTC [757] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC11842026/08/27 09:37:29 OK 20241026095416_initial_model.sql (11.79ms)1185{"timestamp":"2026-08-27T09:37:29.326147342Z","level":"ERROR","duration":"595.165µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(305)"}1186{"timestamp":"2026-08-27T09:37:29.326224462Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"befe8455-8aa9-4690-bdb4-074220d06cd6","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket32/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(305)"}11872026/08/27 09:37:29 OK 20251210153512_drop_unused_gin_index.sql (1.6ms)11882026/08/27 09:37:29 INFO Received uploads request method=POST path=/api/pending_closures11892026/08/27 09:37:29 OK 20251218171726_add_pins.sql (4.2ms)11902026/08/27 09:37:29 OK 20260628120000_add_object_size_and_stats.sql (3.6ms)11912026/08/27 09:37:29 goose: successfully migrated database to version: 2026062812000011922026/08/27 09:37:29 OK 20241026095416_initial_model.sql (11.47ms)11932026/08/27 09:37:29 OK 1_commit_pending_closure.sql (3ms)11942026/08/27 09:37:29 OK 2_object_stats_trigger.sql (989.49µs)11952026/08/27 09:37:29 goose: up to current file version: 211962026/08/27 09:37:29 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)11972026/08/27 09:37:29 OK 20251210153512_drop_unused_gin_index.sql (1.71ms)11982026/08/27 09:37:29 INFO Uploading f1hm85hxla58d8kd8di8sx7v41byw1x4-file1.txt (160B)1199{"timestamp":"2026-08-27T09:37:29.341292973Z","level":"ERROR","duration":"682.026µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(262)"}1200{"timestamp":"2026-08-27T09:37:29.341377574Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"ab71a3cd-a10e-4792-a583-b499ee80c90b","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket33/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(262)"}12012026/08/27 09:37:29 OK 20241026095416_initial_model.sql (9.99ms)12022026/08/27 09:37:29 OK 20251218171726_add_pins.sql (3.18ms)12032026/08/27 09:37:29 OK 20251210153512_drop_unused_gin_index.sql (1.55ms)12042026/08/27 09:37:29 WARN Failed to register uploaded object key=nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst error="server returned 404: 404 page not found\n"12052026/08/27 09:37:29 OK 20251218171726_add_pins.sql (3.71ms)12062026/08/27 09:37:29 OK 20260628120000_add_object_size_and_stats.sql (4ms)12072026/08/27 09:37:29 goose: successfully migrated database to version: 2026062812000012082026/08/27 09:37:29 WARN Failed to register uploaded object key=f1hm85hxla58d8kd8di8sx7v41byw1x4.ls error="server returned 404: 404 page not found\n"12092026/08/27 09:37:29 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign1210{"timestamp":"2026-08-27T09:37:29.34910416Z","level":"ERROR","duration":"219.322µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(233)"}1211{"timestamp":"2026-08-27T09:37:29.349180861Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"f1a8b197-c582-40a0-a48b-6106b34934fc","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket3/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(233)"}12122026/08/27 09:37:29 INFO Signed narinfos id=1 count=112132026/08/27 09:37:29 INFO Uploading 1 narinfos12142026/08/27 09:37:29 OK 1_commit_pending_closure.sql (2.63ms)12152026/08/27 09:37:29 OK 20260628120000_add_object_size_and_stats.sql (3.21ms)12162026/08/27 09:37:29 goose: successfully migrated database to version: 2026062812000012172026/08/27 09:37:29 OK 2_object_stats_trigger.sql (1.04ms)12182026/08/27 09:37:29 goose: up to current file version: 212192026/08/27 09:37:29 OK 1_commit_pending_closure.sql (1.62ms)12202026/08/27 09:37:29 OK 2_object_stats_trigger.sql (715.93µs)12212026/08/27 09:37:29 goose: up to current file version: 21222{"timestamp":"2026-08-27T09:37:29.353110815Z","level":"ERROR","duration":"501.104µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(305)"}1223{"timestamp":"2026-08-27T09:37:29.353181256Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"d545c79c-10fc-4bfe-aeb4-88d0c32d7d2d","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket34/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(305)"}12242026/08/27 09:37:29 WARN Failed to register uploaded object key=f1hm85hxla58d8kd8di8sx7v41byw1x4.narinfo error="server returned 404: 404 page not found\n"12252026/08/27 09:37:29 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete1226{"timestamp":"2026-08-27T09:37:29.354439927Z","level":"ERROR","duration":"437.904µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(233)"}1227{"timestamp":"2026-08-27T09:37:29.354515387Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"7f13be0e-c25d-440e-ade9-67b897ff1101","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket35/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(233)"}12282026/08/27 09:37:29 INFO Completed upload id=112292026/08/27 09:37:29 INFO Upload complete. (105ms)1230=== NAME TestNARDeduplicationMetadataUploadBug1231 metadata_upload_test.go:54: Retrieved narinfo from S3:1232 StorePath: /build/TestNARDeduplicationMetadataUploadBug3130037694/001/store/f1hm85hxla58d8kd8di8sx7v41byw1x4-file1.txt1233 URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst1234 Compression: zstd1235 NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1236 NarSize: 1601237 References: 1238 CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1239{"timestamp":"2026-08-27T09:37:29.365859745Z","level":"ERROR","duration":"169.841µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(233)"}1240{"timestamp":"2026-08-27T09:37:29.365916806Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"fde77f29-68d8-4e38-a28c-7a7b6a89d0ff","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket35/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(233)"}1241 metadata_upload_test.go:55: Retrieved .ls file from S3 (compressed size: 77 bytes)1242 metadata_upload_test.go:55: Decompressed .ls content (64 bytes):1243 {"version":1,"root":{"type":"regular","size":44,"narOffset":96}}12442026/08/27 09:37:29 WARN Failed to fetch cache config, uploading without size limit error="fetching cache config: server returned 404: 404 page not found\n"1245{"timestamp":"2026-08-27T09:37:29.402671064Z","level":"ERROR","duration":"191.722µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(262)"}1246 metadata_upload_test.go:64: Second store path (same content): /build/TestNARDeduplicationMetadataUploadBug3130037694/001/store/xzgbra8plmkd9qlxh9lxsylddszbj7w9-file2.txt1247{"timestamp":"2026-08-27T09:37:29.402740184Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"ee356950-d4cf-49f9-8e74-befcf8093dab","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket32/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(262)"}12482026/08/27 09:37:29 INFO Received uploads request method=POST path=/api/pending_closures1249{"timestamp":"2026-08-27T09:37:29.410934575Z","level":"ERROR","duration":"198.981µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(233)"}1250{"timestamp":"2026-08-27T09:37:29.411019176Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"e3667822-ce62-441f-9220-0c0d3db6e6d5","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket31/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(233)"}12512026/08/27 09:37:29 INFO Received uploads request method=POST path=/api/pending_closures12522026/08/27 09:37:29 INFO Received uploads request method=POST path=/api/pending_closures12532026/08/27 09:37:29 INFO Uploading 3 paths to 127.0.0.1 (0 already cached)12542026/08/27 09:37:29 INFO Uploading nm9hfkswaxjnw9grpp487abkvnawc9h3-test-file-1.txt (160B)12552026/08/27 09:37:29 INFO Uploading 986k8mrqw3di3kqjab88c9dskmyzgv3a-test-file-2.txt (160B)12562026/08/27 09:37:29 INFO Uploading nkiivlqnj5r3xgcay3z3pz45126i4zxp-test-file-0.txt (160B)12572026/08/27 09:37:29 WARN Failed to register uploaded object key=nar/0ahq3izw3h0b0c08p3xdfclh758fwxnp85ccvflasb4vnfsv50ri.nar.zst error="server returned 404: 404 page not found\n"1258{"timestamp":"2026-08-27T09:37:29.426035626Z","level":"ERROR","duration":"137.541µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(311)"}1259{"timestamp":"2026-08-27T09:37:29.426093166Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"ecffdcc8-8d18-46a9-9620-e7d44707e121","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket20/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(311)"}12602026/08/27 09:37:29 WARN Failed to register uploaded object key=nar/13p9pqi0ga6fx2apcwls9hjnv7nmf740ggpn5zvfzdsp5vz2bbg4.nar.zst error="server returned 404: 404 page not found\n"12612026/08/27 09:37:29 WARN Failed to register uploaded object key=nar/1823g6vx4q5zqvr73m7lkq1wpk40fin7bsc792di430yk7sg4lka.nar.zst error="server returned 404: 404 page not found\n"12622026/08/27 09:37:29 WARN Failed to register uploaded object key=nm9hfkswaxjnw9grpp487abkvnawc9h3.ls error="server returned 404: 404 page not found\n"12632026/08/27 09:37:29 WARN Failed to register uploaded object key=nkiivlqnj5r3xgcay3z3pz45126i4zxp.ls error="server returned 404: 404 page not found\n"12642026/08/27 09:37:29 WARN Failed to register uploaded object key=986k8mrqw3di3kqjab88c9dskmyzgv3a.ls error="server returned 404: 404 page not found\n"12652026/08/27 09:37:29 INFO Received sign narinfos request method=POST path=/api/pending_closures/3/sign12662026/08/27 09:37:29 INFO Signed narinfos id=3 count=112672026/08/27 09:37:29 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign12682026/08/27 09:37:29 INFO Signed narinfos id=1 count=112692026/08/27 09:37:29 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign12702026/08/27 09:37:29 INFO Signed narinfos id=2 count=112712026/08/27 09:37:29 INFO Uploading 3 narinfos12722026/08/27 09:37:29 WARN Failed to register uploaded object key=nkiivlqnj5r3xgcay3z3pz45126i4zxp.narinfo error="server returned 404: 404 page not found\n"12732026/08/27 09:37:29 WARN Failed to register uploaded object key=nm9hfkswaxjnw9grpp487abkvnawc9h3.narinfo error="server returned 404: 404 page not found\n"12742026/08/27 09:37:29 WARN Failed to register uploaded object key=986k8mrqw3di3kqjab88c9dskmyzgv3a.narinfo error="server returned 404: 404 page not found\n"12752026/08/27 09:37:29 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete12762026/08/27 09:37:29 INFO Completed upload id=112772026/08/27 09:37:29 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete12782026/08/27 09:37:29 INFO Completed upload id=212792026/08/27 09:37:29 INFO Received complete upload request method=POST path=/api/pending_closures/3/complete12802026/08/27 09:37:29 INFO Completed upload id=312812026/08/27 09:37:29 INFO Upload complete. (101ms)1282=== NAME TestClientMultipleUploads1283 client_integration_test.go:349: Uploaded 3 paths in 136.869804ms1284--- PASS: TestClientMultipleUploads (1.63s)1285=== CONT TestService_AuthMiddleware_MTLSBoundSubjects12862026/08/27 09:37:29 WARN Failed to fetch cache config, uploading without size limit error="fetching cache config: server returned 404: 404 page not found\n"1287{"timestamp":"2026-08-27T09:37:29.490529524Z","level":"ERROR","duration":"377.304µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(305)"}1288{"timestamp":"2026-08-27T09:37:29.490614344Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"32debaf9-914e-4d6c-8bc6-86d71c1d884d","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket33/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(305)"}12892026/08/27 09:37:29 INFO Received uploads request method=POST path=/api/pending_closures1290{"timestamp":"2026-08-27T09:37:29.504255443Z","level":"ERROR","duration":"156.282µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(305)"}1291{"timestamp":"2026-08-27T09:37:29.504317423Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"4e73cb49-f2a1-44fa-b640-4f12b19ba548","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket15/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(305)"}12922026/08/27 09:37:29 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)1293{"timestamp":"2026-08-27T09:37:29.51199941Z","level":"ERROR","duration":"183.042µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(305)"}1294{"timestamp":"2026-08-27T09:37:29.51207651Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"15b40ba2-87b0-4be9-95ed-75bbc9260f89","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket34/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(305)"}12952026-08-27 09:37:29.531 UTC [904] ERROR: relation "goose_db_version" does not exist at character 3612962026-08-27 09:37:29.531 UTC [904] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC12972026/08/27 09:37:29 OK 20241026095416_initial_model.sql (9.73ms)12982026/08/27 09:37:29 OK 20251210153512_drop_unused_gin_index.sql (1.33ms)12992026/08/27 09:37:29 OK 20251218171726_add_pins.sql (3.08ms)13002026/08/27 09:37:29 OK 20260628120000_add_object_size_and_stats.sql (2.83ms)13012026/08/27 09:37:29 goose: successfully migrated database to version: 2026062812000013022026/08/27 09:37:29 OK 1_commit_pending_closure.sql (2.05ms)13032026/08/27 09:37:29 OK 2_object_stats_trigger.sql (841.65µs)13042026/08/27 09:37:29 goose: up to current file version: 21305{"timestamp":"2026-08-27T09:37:29.559017316Z","level":"ERROR","duration":"482.904µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(305)"}1306{"timestamp":"2026-08-27T09:37:29.559073097Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"8a5998b7-2f07-48a8-bc90-cfbddb2f296b","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket36/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(305)"}1307{"timestamp":"2026-08-27T09:37:29.574172947Z","level":"ERROR","duration":"221.422µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(233)"}1308{"timestamp":"2026-08-27T09:37:29.574255988Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"7edaf387-dbb0-4307-8309-7453aa5d8b78","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket35/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(233)"}1309{"timestamp":"2026-08-27T09:37:29.574617451Z","level":"ERROR","duration":"104.021µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(233)"}1310{"timestamp":"2026-08-27T09:37:29.574662972Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"e2579a76-e53c-49b7-b587-a19d09d1afae","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket31/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(233)"}1311{"timestamp":"2026-08-27T09:37:29.580324441Z","level":"ERROR","duration":"113.601µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(233)"}1312{"timestamp":"2026-08-27T09:37:29.580369421Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"deafa07d-98e1-4033-a897-cd83ad664083","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket33/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(233)"}1313{"timestamp":"2026-08-27T09:37:29.582909523Z","level":"ERROR","duration":"97.081µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(233)"}1314{"timestamp":"2026-08-27T09:37:29.582951903Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"147b7084-bb86-4372-b2d2-e55c70ea8b1c","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket30/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(233)"}1315{"timestamp":"2026-08-27T09:37:29.590564169Z","level":"ERROR","duration":"159.421µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(233)"}1316{"timestamp":"2026-08-27T09:37:29.59061283Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"a949e70f-7758-4f69-8f33-fd2997601922","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket23/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(233)"}1317{"timestamp":"2026-08-27T09:37:29.59524609Z","level":"ERROR","duration":"110.441µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(233)"}1318{"timestamp":"2026-08-27T09:37:29.59529087Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"c826f11b-bba5-49e0-8a10-644c88e3f948","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket32/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(233)"}1319{"timestamp":"2026-08-27T09:37:29.618147868Z","level":"ERROR","duration":"143.181µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(233)"}1320{"timestamp":"2026-08-27T09:37:29.618202229Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"a0f35c9b-0772-43f7-9338-3fe600d52a99","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket3/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(233)"}1321{"timestamp":"2026-08-27T09:37:29.633889824Z","level":"ERROR","duration":"121.361µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(233)"}1322{"timestamp":"2026-08-27T09:37:29.633937285Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"8108a6c0-a87c-4504-bfaf-194dfc10cb3f","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket35/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(233)"}1323{"timestamp":"2026-08-27T09:37:29.666902389Z","level":"ERROR","duration":"199.681µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(233)"}1324{"timestamp":"2026-08-27T09:37:29.66698147Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"0e0b795a-e102-44ac-b077-67c4532c18be","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket3/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(233)"}13252026/08/27 09:37:29 WARN Failed to register uploaded object key=xzgbra8plmkd9qlxh9lxsylddszbj7w9.ls error="server returned 404: 404 page not found\n"13262026/08/27 09:37:29 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign13272026/08/27 09:37:29 INFO Signed narinfos id=2 count=113282026/08/27 09:37:29 INFO Uploading 1 narinfos1329{"timestamp":"2026-08-27T09:37:29.683044169Z","level":"ERROR","duration":"124.241µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(259)"}1330{"timestamp":"2026-08-27T09:37:29.68308811Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"a90882dc-301d-4f55-94ad-d331e3914c42","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket33/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(259)"}13312026/08/27 09:37:29 WARN Failed to register uploaded object key=xzgbra8plmkd9qlxh9lxsylddszbj7w9.narinfo error="server returned 404: 404 page not found\n"13322026/08/27 09:37:29 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete13332026/08/27 09:37:29 INFO Completed upload id=213342026/08/27 09:37:29 INFO Upload complete. (248ms)1335=== NAME TestNARDeduplicationMetadataUploadBug1336 metadata_upload_test.go:76: Retrieved narinfo from S3:1337 StorePath: /build/TestNARDeduplicationMetadataUploadBug3130037694/001/store/xzgbra8plmkd9qlxh9lxsylddszbj7w9-file2.txt1338 URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst1339 Compression: zstd1340 NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1341 NarSize: 1601342 References: 1343 CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1344 metadata_upload_test.go:77: Retrieved .ls file from S3 (compressed size: 62 bytes)1345 metadata_upload_test.go:77: Decompressed .ls content (49 bytes):1346 {"version":1,"root":{"type":"regular","size":44}}1347--- PASS: TestReadProxyConditionalGet (0.89s)1348=== CONT TestService_ReadAuthMiddleware1349--- PASS: TestNARDeduplicationMetadataUploadBug (1.95s)1350=== CONT TestService_NativeMTLS13512026/08/27 09:37:29 INFO Received uploads request method=POST path=/api/pending_closures13522026/08/27 09:37:29 INFO Aborted multipart uploads count=013532026/08/27 09:37:29 INFO Received uploads request method=POST path=/api/pending_closures13542026/08/27 09:37:29 WARN Force mode enabled - objects will be deleted immediately without grace period13552026/08/27 09:37:29 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=013562026/08/27 09:37:29 INFO Vacuumed table table=pending_closures13572026/08/27 09:37:29 INFO Vacuumed table table=pending_objects13582026/08/27 09:37:29 INFO Vacuumed table table=multipart_uploads13592026/08/27 09:37:29 INFO Vacuumed table table=closures13602026/08/27 09:37:29 INFO Received uploads request method=POST path=/api/pending_closures13612026/08/27 09:37:29 INFO Vacuumed table table=objects1362--- PASS: TestGCMetrics (1.89s)1363=== CONT TestService_AuthMiddleware_MTLSProxyHeader13642026/08/27 09:37:29 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted,O=Other"13652026/08/27 09:37:29 WARN mTLS auth: bound subjects configured but subject DN unavailable13662026/08/27 09:37:29 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted"1367--- PASS: TestService_AuthMiddleware_MTLSBoundSubjects (0.28s)1368=== CONT TestMetricsInventory13692026/08/27 09:37:29 INFO Received complete multipart upload request method=POST path=/api/multipart/complete1370--- PASS: TestReadProxy404 (0.54s)1371=== CONT TestOrphanedObjectsGCStressTest13722026/08/27 09:37:29 WARN CompleteMultipartUpload errored but object exists; treating as success error="The specified key does not exist." object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=NTYxNTAwNjUtNmMwNy00NDg5LWExZmYtZjY0YWQ2NzRkNGVjLjIzNWFlZmQ5LTBkMjgtNDk3ZS04NzQ1LTRhZGQyZDI4OGE5MHgxNzg3ODIzNDQ5NzExMzkzMjU113732026/08/27 09:37:29 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=NTYxNTAwNjUtNmMwNy00NDg5LWExZmYtZjY0YWQ2NzRkNGVjLjIzNWFlZmQ5LTBkMjgtNDk3ZS04NzQ1LTRhZGQyZDI4OGE5MHgxNzg3ODIzNDQ5NzExMzkzMjU1 parts=11374--- PASS: TestCompleteMultipartUpload_ErrorButObjectExists (2.01s)1375=== CONT TestResurrectedObjectNotDeleted13762026-08-27 09:37:29.799 UTC [923] ERROR: relation "goose_db_version" does not exist at character 3613772026-08-27 09:37:29.799 UTC [923] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC13782026-08-27 09:37:29.802 UTC [924] ERROR: relation "goose_db_version" does not exist at character 3613792026-08-27 09:37:29.802 UTC [924] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC13802026/08/27 09:37:29 OK 20241026095416_initial_model.sql (13.79ms)13812026/08/27 09:37:29 OK 20241026095416_initial_model.sql (12.72ms)13822026/08/27 09:37:29 OK 20251210153512_drop_unused_gin_index.sql (3.39ms)13832026-08-27 09:37:29.828 UTC [925] ERROR: relation "goose_db_version" does not exist at character 3613842026-08-27 09:37:29.828 UTC [925] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC13852026/08/27 09:37:29 OK 20251210153512_drop_unused_gin_index.sql (2.64ms)13862026/08/27 09:37:29 OK 20251218171726_add_pins.sql (5.76ms)13872026/08/27 09:37:29 OK 20251218171726_add_pins.sql (5.15ms)13882026/08/27 09:37:29 OK 20260628120000_add_object_size_and_stats.sql (5.03ms)13892026/08/27 09:37:29 goose: successfully migrated database to version: 2026062812000013902026/08/27 09:37:29 OK 20260628120000_add_object_size_and_stats.sql (5ms)13912026/08/27 09:37:29 goose: successfully migrated database to version: 2026062812000013922026/08/27 09:37:29 OK 1_commit_pending_closure.sql (3ms)13932026/08/27 09:37:29 OK 1_commit_pending_closure.sql (2.89ms)13942026-08-27 09:37:29.844 UTC [926] ERROR: relation "goose_db_version" does not exist at character 3613952026-08-27 09:37:29.844 UTC [926] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC13962026/08/27 09:37:29 OK 2_object_stats_trigger.sql (2.01ms)13972026/08/27 09:37:29 goose: up to current file version: 213982026/08/27 09:37:29 OK 2_object_stats_trigger.sql (2.25ms)13992026/08/27 09:37:29 goose: up to current file version: 214002026/08/27 09:37:29 OK 20241026095416_initial_model.sql (12.18ms)14012026/08/27 09:37:29 OK 20251210153512_drop_unused_gin_index.sql (2.27ms)14022026-08-27 09:37:29.854 UTC [927] ERROR: relation "goose_db_version" does not exist at character 3614032026-08-27 09:37:29.854 UTC [927] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC14042026/08/27 09:37:29 OK 20251218171726_add_pins.sql (11.23ms)14052026/08/27 09:37:29 OK 20241026095416_initial_model.sql (13.86ms)14062026/08/27 09:37:29 OK 20260628120000_add_object_size_and_stats.sql (5.17ms)14072026/08/27 09:37:29 goose: successfully migrated database to version: 2026062812000014082026/08/27 09:37:29 OK 20251210153512_drop_unused_gin_index.sql (4ms)14092026/08/27 09:37:29 OK 1_commit_pending_closure.sql (3.53ms)14102026/08/27 09:37:29 OK 20251218171726_add_pins.sql (4.86ms)14112026/08/27 09:37:29 OK 2_object_stats_trigger.sql (1.52ms)14122026/08/27 09:37:29 goose: up to current file version: 214132026/08/27 09:37:29 OK 20241026095416_initial_model.sql (11.05ms)14142026-08-27 09:37:29.876 UTC [928] ERROR: relation "goose_db_version" does not exist at character 3614152026-08-27 09:37:29.876 UTC [928] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC14162026/08/27 09:37:29 OK 20260628120000_add_object_size_and_stats.sql (4.71ms)14172026/08/27 09:37:29 goose: successfully migrated database to version: 2026062812000014182026/08/27 09:37:29 OK 20251210153512_drop_unused_gin_index.sql (1.44ms)14192026/08/27 09:37:29 OK 1_commit_pending_closure.sql (3.67ms)14202026/08/27 09:37:29 OK 20251218171726_add_pins.sql (3.43ms)14212026/08/27 09:37:29 OK 2_object_stats_trigger.sql (1.1ms)14222026/08/27 09:37:29 goose: up to current file version: 21423{"timestamp":"2026-08-27T09:37:29.886018745Z","level":"ERROR","duration":"1.718335ms","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(300)"}1424{"timestamp":"2026-08-27T09:37:29.886108246Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"bb36e1e3-b334-41ef-814d-7879d37d6b7c","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket40/","status_code":503,"duration_ms":1,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(300)"}14252026/08/27 09:37:29 OK 20260628120000_add_object_size_and_stats.sql (3.86ms)14262026/08/27 09:37:29 goose: successfully migrated database to version: 2026062812000014272026/08/27 09:37:29 OK 1_commit_pending_closure.sql (3.17ms)14282026/08/27 09:37:29 OK 2_object_stats_trigger.sql (1.48ms)14292026/08/27 09:37:29 goose: up to current file version: 214302026/08/27 09:37:29 OK 20241026095416_initial_model.sql (7.62ms)1431{"timestamp":"2026-08-27T09:37:29.892636602Z","level":"ERROR","duration":"561.144µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(311)"}1432{"timestamp":"2026-08-27T09:37:29.892715223Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"fc3bd627-5538-4286-819c-9e88327f9c09","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket41/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(311)"}14332026/08/27 09:37:29 OK 20251210153512_drop_unused_gin_index.sql (1.3ms)1434{"timestamp":"2026-08-27T09:37:29.895189045Z","level":"ERROR","duration":"159.362µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(311)"}1435{"timestamp":"2026-08-27T09:37:29.895239705Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"92c401de-9994-44c6-b5f7-81db636c6120","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket41/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(311)"}14362026/08/27 09:37:29 OK 20251218171726_add_pins.sql (3.2ms)14372026/08/27 09:37:29 OK 20260628120000_add_object_size_and_stats.sql (3.69ms)14382026/08/27 09:37:29 goose: successfully migrated database to version: 2026062812000014392026/08/27 09:37:29 OK 1_commit_pending_closure.sql (1.85ms)14402026/08/27 09:37:29 OK 2_object_stats_trigger.sql (794.55µs)14412026/08/27 09:37:29 goose: up to current file version: 21442{"timestamp":"2026-08-27T09:37:29.904275063Z","level":"ERROR","duration":"473.704µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(311)"}1443{"timestamp":"2026-08-27T09:37:29.904345764Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"71933d37-dc60-4695-b965-a20022d65329","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket42/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(311)"}1444{"timestamp":"2026-08-27T09:37:29.911229283Z","level":"ERROR","duration":"206.242µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(262)"}1445{"timestamp":"2026-08-27T09:37:29.911303024Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"922fcfa0-66ee-460b-b7a2-f4fb99208ab4","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket40/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(262)"}1446{"timestamp":"2026-08-27T09:37:29.91658729Z","level":"ERROR","duration":"219.002µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(294)"}1447{"timestamp":"2026-08-27T09:37:29.91666153Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"71a651fd-e12d-4666-96ba-361f7bf06a20","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket19/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(294)"}14482026/08/27 09:37:29 WARN Authentication failed token_preview=wrongtoken token_length=10 reason="static API token mismatch"1449--- PASS: TestService_AuthMiddleware (2.19s)1450=== CONT TestOrphanedObjectsGC14512026-08-27 09:37:30.012 UTC [931] ERROR: relation "goose_db_version" does not exist at character 3614522026-08-27 09:37:30.012 UTC [931] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC14532026/08/27 09:37:30 INFO Received complete multipart upload request method=POST path=/api/multipart/complete14542026/08/27 09:37:30 OK 20241026095416_initial_model.sql (7.75ms)14552026/08/27 09:37:30 OK 20251210153512_drop_unused_gin_index.sql (1.41ms)14562026/08/27 09:37:30 OK 20251218171726_add_pins.sql (3.26ms)14572026/08/27 09:37:30 OK 20260628120000_add_object_size_and_stats.sql (2.78ms)14582026/08/27 09:37:30 goose: successfully migrated database to version: 2026062812000014592026/08/27 09:37:30 OK 1_commit_pending_closure.sql (1.9ms)14602026/08/27 09:37:30 OK 2_object_stats_trigger.sql (780.81µs)14612026/08/27 09:37:30 goose: up to current file version: 21462{"timestamp":"2026-08-27T09:37:30.039723675Z","level":"ERROR","duration":"496.205µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(294)"}1463{"timestamp":"2026-08-27T09:37:30.039812036Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"dc2f62d6-15be-4b4a-a08e-26911c7c0ce5","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket43/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(294)"}1464{"timestamp":"2026-08-27T09:37:30.067938239Z","level":"ERROR","duration":"216.802µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(300)"}1465{"timestamp":"2026-08-27T09:37:30.06801996Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"f4973ed8-199f-42d2-ba8e-4ac7534064c1","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket40/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(300)"}1466{"timestamp":"2026-08-27T09:37:30.072879362Z","level":"ERROR","duration":"174.202µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(279)"}1467{"timestamp":"2026-08-27T09:37:30.072957262Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"316a619f-7b30-430e-820d-82e61f51366b","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket41/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(279)"}1468{"timestamp":"2026-08-27T09:37:30.074547356Z","level":"ERROR","duration":"129.761µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(279)"}1469{"timestamp":"2026-08-27T09:37:30.074590716Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"f74ee375-a670-44fd-b786-87554a8eaea1","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket42/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(279)"}1470{"timestamp":"2026-08-27T09:37:30.080541788Z","level":"ERROR","duration":"223.982µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(311)"}1471{"timestamp":"2026-08-27T09:37:30.080618689Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"54735446-a6d9-4142-83b8-51848080cecf","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket32/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(311)"}1472{"timestamp":"2026-08-27T09:37:30.083365233Z","level":"ERROR","duration":"202.202µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(262)"}1473{"timestamp":"2026-08-27T09:37:30.083440513Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"9956078c-ac11-4c7c-82b2-d9b8b950a971","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket43/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(262)"}1474{"timestamp":"2026-08-27T09:37:30.096266824Z","level":"ERROR","duration":"180.802µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(233)"}1475{"timestamp":"2026-08-27T09:37:30.096331065Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"6f72747c-0b72-4ce5-93df-53870372bc2b","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket40/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(233)"}1476{"timestamp":"2026-08-27T09:37:30.099123569Z","level":"ERROR","duration":"187.562µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(318)"}1477{"timestamp":"2026-08-27T09:37:30.099194669Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"096ddf8f-70db-4254-a82d-15ffe5daf2f4","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket31/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(318)"}1478{"timestamp":"2026-08-27T09:37:30.126052662Z","level":"ERROR","duration":"298.503µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(300)"}1479{"timestamp":"2026-08-27T09:37:30.126129942Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"4e61985a-ee25-4c61-976d-2be7e2906fa9","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket43/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(300)"}14802026/08/27 09:37:30 INFO Received complete multipart upload request method=POST path=/api/multipart/complete1481{"timestamp":"2026-08-27T09:37:30.194282652Z","level":"ERROR","duration":"196.342µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(294)"}1482{"timestamp":"2026-08-27T09:37:30.194358833Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"d11f4697-a620-4ffe-ade3-87615c3314c4","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket43/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(294)"}14832026/08/27 09:37:30 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst upload_id=NTYxNTAwNjUtNmMwNy00NDg5LWExZmYtZjY0YWQ2NzRkNGVjLmEyMGE1MjBkLTNlZjktNGRjZC05ZmI2LWQxN2U1ZjNlZDc3Y3gxNzg3ODIzNDQ4MDY3MTA0MTcx parts=1014842026/08/27 09:37:30 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete14852026/08/27 09:37:30 INFO Completed upload id=114862026/08/27 09:37:30 INFO Received uploads request method=POST path=/api/pending_closures14872026/08/27 09:37:30 INFO Received uploads request method=POST path=/api/pending_closures14882026/08/27 09:37:30 INFO Object in database but missing from S3 key=dadb44fdadb44fdadb44fdadb44f0000.narinfo14892026/08/27 09:37:30 WARN Found objects in DB but missing from S3, will re-upload count=11490--- PASS: TestService_verifyS3Integrity (2.47s)1491=== CONT TestObjectStatsTrigger14922026-08-27 09:37:30.279 UTC [945] ERROR: relation "goose_db_version" does not exist at character 3614932026-08-27 09:37:30.279 UTC [945] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC14942026/08/27 09:37:30 INFO Received complete multipart upload request method=POST path=/api/multipart/complete1495{"timestamp":"2026-08-27T09:37:30.290526744Z","level":"ERROR","duration":"228.402µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(300)"}1496{"timestamp":"2026-08-27T09:37:30.290611705Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"b719379e-22cd-4f2c-8b52-80955af9970a","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket42/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(300)"}14972026/08/27 09:37:30 OK 20241026095416_initial_model.sql (9.22ms)14982026/08/27 09:37:30 OK 20251210153512_drop_unused_gin_index.sql (1.29ms)14992026/08/27 09:37:30 OK 20251218171726_add_pins.sql (3.5ms)15002026/08/27 09:37:30 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0100000000000000000000.nar.zst upload_id=NTYxNTAwNjUtNmMwNy00NDg5LWExZmYtZjY0YWQ2NzRkNGVjLmE2ZmZhYTNjLTc0NWUtNDUxYS1hNmU0LTUwNGU1ODdiODA0YXgxNzg3ODIzNDQ4ODE0NTA1OTM2 parts=121501--- PASS: TestRedundantMultipartUpload (2.56s)1502=== CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info15032026/08/27 09:37:30 INFO Received uploads request method=POST path=/1504=== CONT TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key15052026/08/27 09:37:30 INFO Received complete multipart upload request method=POST path=/1506=== CONT TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key15072026/08/27 09:37:30 INFO Received request for more parts method=POST path=/1508=== CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal15092026/08/27 09:37:30 INFO Received uploads request method=POST path=/1510--- PASS: TestUploadHandlersRejectInvalidKeys (0.08s)1511 --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info (0.00s)1512 --- PASS: TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key (0.00s)1513 --- PASS: TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key (0.00s)1514 --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal (0.00s)1515=== CONT TestIsValidUploadKey/narinfo1516=== CONT TestIsValidUploadKey/unknown_type1517=== CONT TestIsValidUploadKey/empty_key1518=== CONT TestIsValidUploadKey/absolute1519=== CONT TestIsValidUploadKey/traversal_nar1520=== CONT TestIsValidUploadKey/traversal1521=== CONT TestIsValidUploadKey/listing_key,_narinfo_type1522=== CONT TestIsValidUploadKey/nar_key,_narinfo_type1523=== CONT TestIsValidUploadKey/narinfo_key,_nar_type1524=== CONT TestIsValidUploadKey/index.html1525=== CONT TestIsValidUploadKey/nix-cache-info1526=== CONT TestIsValidUploadKey/realisation_plus_in_output1527=== CONT TestIsValidUploadKey/realisation1528=== CONT TestIsValidUploadKey/build_log_equals1529=== CONT TestIsValidUploadKey/build_log_question_mark1530=== CONT TestIsValidUploadKey/build_log_plus_in_name1531=== CONT TestIsValidUploadKey/build_log_home-manager_file1532=== CONT TestIsValidUploadKey/build_log1533=== CONT TestIsValidUploadKey/listing1534=== CONT TestIsValidUploadKey/nar_plain1535=== CONT TestIsValidUploadKey/nar_xz1536=== CONT TestIsValidUploadKey/nar_zst1537=== CONT TestProxyWriteTimeout/narinfo1538--- PASS: TestIsValidUploadKey (0.08s)1539 --- PASS: TestIsValidUploadKey/narinfo (0.00s)1540 --- PASS: TestIsValidUploadKey/unknown_type (0.00s)1541 --- PASS: TestIsValidUploadKey/empty_key (0.00s)1542 --- PASS: TestIsValidUploadKey/absolute (0.00s)1543 --- PASS: TestIsValidUploadKey/traversal_nar (0.00s)1544 --- PASS: TestIsValidUploadKey/traversal (0.00s)1545 --- PASS: TestIsValidUploadKey/listing_key,_narinfo_type (0.00s)1546 --- PASS: TestIsValidUploadKey/nar_key,_narinfo_type (0.00s)1547 --- PASS: TestIsValidUploadKey/narinfo_key,_nar_type (0.00s)1548 --- PASS: TestIsValidUploadKey/index.html (0.00s)1549 --- PASS: TestIsValidUploadKey/nix-cache-info (0.00s)1550 --- PASS: TestIsValidUploadKey/realisation_plus_in_output (0.00s)1551 --- PASS: TestIsValidUploadKey/realisation (0.00s)1552 --- PASS: TestIsValidUploadKey/build_log_equals (0.00s)1553 --- PASS: TestIsValidUploadKey/build_log_question_mark (0.00s)1554 --- PASS: TestIsValidUploadKey/build_log_plus_in_name (0.00s)1555 --- PASS: TestIsValidUploadKey/build_log_home-manager_file (0.00s)1556 --- PASS: TestIsValidUploadKey/build_log (0.00s)1557 --- PASS: TestIsValidUploadKey/listing (0.00s)1558 --- PASS: TestIsValidUploadKey/nar_plain (0.00s)1559 --- PASS: TestIsValidUploadKey/nar_xz (0.00s)1560 --- PASS: TestIsValidUploadKey/nar_zst (0.00s)15612026/08/27 09:37:30 OK 20260628120000_add_object_size_and_stats.sql (3.03ms)1562=== CONT TestProxyWriteTimeout/10_GiB_nar15632026/08/27 09:37:30 goose: successfully migrated database to version: 202606281200001564=== CONT TestProxyWriteTimeout/unknown_size1565=== CONT TestProxyWriteTimeout/1_GiB_nar1566--- PASS: TestProxyWriteTimeout (0.08s)1567 --- PASS: TestProxyWriteTimeout/narinfo (0.00s)1568 --- PASS: TestProxyWriteTimeout/10_GiB_nar (0.00s)1569 --- PASS: TestProxyWriteTimeout/unknown_size (0.00s)1570 --- PASS: TestProxyWriteTimeout/1_GiB_nar (0.00s)1571=== CONT TestUploadHandlersRejectOversizedBody/request_more_parts15722026/08/27 09:37:30 INFO Received request for more parts method=POST path=/15732026/08/27 09:37:30 OK 1_commit_pending_closure.sql (2.16ms)15742026/08/27 09:37:30 OK 2_object_stats_trigger.sql (765.83µs)15752026/08/27 09:37:30 goose: up to current file version: 21576{"timestamp":"2026-08-27T09:37:30.306904966Z","level":"ERROR","duration":"545.144µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(233)"}1577{"timestamp":"2026-08-27T09:37:30.306986967Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"e126c531-074d-4249-9370-292d6b573bda","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket44/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(233)"}1578{"timestamp":"2026-08-27T09:37:30.316778832Z","level":"ERROR","duration":"190.222µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(311)"}1579{"timestamp":"2026-08-27T09:37:30.316852412Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"c7e1336c-6bc0-4680-aa40-4d4fe0064337","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket40/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(311)"}1580{"timestamp":"2026-08-27T09:37:30.378912509Z","level":"ERROR","duration":"213.722µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(294)"}1581{"timestamp":"2026-08-27T09:37:30.37899417Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"15f17f32-8b7a-4a41-ae69-a076433a1db7","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket44/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(294)"}1582=== CONT TestUploadHandlersRejectOversizedBody/complete_multipart15832026/08/27 09:37:30 INFO Received complete multipart upload request method=POST path=/1584{"timestamp":"2026-08-27T09:37:30.409923658Z","level":"ERROR","duration":"182.762µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(300)"}1585{"timestamp":"2026-08-27T09:37:30.409992658Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"b10d3ca4-0c7d-4976-a410-daea597f66cc","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket15/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(300)"}1586{"timestamp":"2026-08-27T09:37:30.418695454Z","level":"ERROR","duration":"136.702µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(233)"}1587{"timestamp":"2026-08-27T09:37:30.418746574Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"78d8ff86-c348-4e88-93c5-854737af02d1","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket35/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(233)"}1588{"timestamp":"2026-08-27T09:37:30.45069963Z","level":"ERROR","duration":"280.222µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(300)"}1589{"timestamp":"2026-08-27T09:37:30.450778291Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"ca156f10-8792-447b-b87f-52b763f2c4c2","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket23/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(300)"}1590=== CONT TestUploadHandlersRejectOversizedBody/create_pending_closure15912026/08/27 09:37:30 INFO Received uploads request method=POST path=/1592{"timestamp":"2026-08-27T09:37:30.624386433Z","level":"ERROR","duration":"199.002µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(233)"}1593{"timestamp":"2026-08-27T09:37:30.624386473Z","level":"ERROR","duration":"198.902µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(311)"}1594{"timestamp":"2026-08-27T09:37:30.624465474Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"7c521f62-6de2-41f9-879d-6c113458d1c9","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket42/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(233)"}1595{"timestamp":"2026-08-27T09:37:30.624465494Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"f835c8a7-8f35-4c7a-98e3-9b44ab0de439","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket3/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(311)"}1596{"timestamp":"2026-08-27T09:37:30.661444494Z","level":"ERROR","duration":"207.402µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(300)"}1597{"timestamp":"2026-08-27T09:37:30.661528134Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"d98f7c50-f90b-4b26-baf7-c689a88c435f","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket19/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(300)"}1598{"timestamp":"2026-08-27T09:37:30.765550134Z","level":"ERROR","duration":"271.322µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(233)"}1599{"timestamp":"2026-08-27T09:37:30.765626395Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"591d0f58-b781-4fc4-82d3-cb16fa498756","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket44/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(233)"}1600{"timestamp":"2026-08-27T09:37:30.767153208Z","level":"ERROR","duration":"98.441µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(233)"}1601{"timestamp":"2026-08-27T09:37:30.767194769Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"5663d264-5b2c-43c2-aaf1-5a62d9a3032b","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket41/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(233)"}1602{"timestamp":"2026-08-27T09:37:30.77202909Z","level":"ERROR","duration":"168.181µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(279)"}1603{"timestamp":"2026-08-27T09:37:30.772097971Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"f4959c00-cf14-4d48-9b39-5d75a078077a","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket41/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(279)"}1604{"timestamp":"2026-08-27T09:37:30.802468533Z","level":"ERROR","duration":"170.641µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(300)"}1605{"timestamp":"2026-08-27T09:37:30.802547454Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"3ad31048-1b8a-4ef4-8718-0a6ee5494286","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket41/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(300)"}1606{"timestamp":"2026-08-27T09:37:30.857428909Z","level":"ERROR","duration":"205.302µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(300)"}1607{"timestamp":"2026-08-27T09:37:30.85750937Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"8e0472d5-310a-42b2-ae44-f098d8f6bc99","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket23/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(300)"}1608{"timestamp":"2026-08-27T09:37:30.891280442Z","level":"ERROR","duration":"232.462µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(279)"}1609{"timestamp":"2026-08-27T09:37:30.891359942Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"e5d7a0de-efd9-47bf-a321-48a1afdedf10","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket44/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(279)"}1610{"timestamp":"2026-08-27T09:37:30.92922451Z","level":"ERROR","duration":"203.922µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(279)"}1611{"timestamp":"2026-08-27T09:37:30.92930245Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"68457940-89fb-4e70-a627-1248d8378fe8","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket31/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(279)"}1612{"timestamp":"2026-08-27T09:37:30.939086915Z","level":"ERROR","duration":"196.781µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(311)"}1613{"timestamp":"2026-08-27T09:37:30.939158296Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"dc401681-2620-4032-a536-c53f7569901e","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket32/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(311)"}1614{"timestamp":"2026-08-27T09:37:31.042990614Z","level":"ERROR","duration":"197.881µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(233)"}1615{"timestamp":"2026-08-27T09:37:31.043068775Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"73eaf687-4c18-4c5d-8243-be4e42087574","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket35/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(233)"}1616{"timestamp":"2026-08-27T09:37:31.078021537Z","level":"ERROR","duration":"273.842µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(294)"}1617{"timestamp":"2026-08-27T09:37:31.078105658Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"789f6496-9575-4cfb-becf-ac1d13b2c8de","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket40/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(294)"}1618{"timestamp":"2026-08-27T09:37:31.088815291Z","level":"ERROR","duration":"140.401µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(294)"}1619{"timestamp":"2026-08-27T09:37:31.088870051Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"29ea8924-eda1-4eb9-8c33-7958b48b0d81","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket44/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(294)"}1620{"timestamp":"2026-08-27T09:37:31.101665542Z","level":"ERROR","duration":"211.662µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(300)"}1621{"timestamp":"2026-08-27T09:37:31.101741042Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"7c4e3292-123b-4b78-906d-6b8fd85c1734","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket15/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(300)"}16222026/08/27 09:37:31 INFO Created nix-cache-info in bucket bucket=bucket251623{"timestamp":"2026-08-27T09:37:31.133004633Z","level":"ERROR","duration":"176.642µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(233)"}1624{"timestamp":"2026-08-27T09:37:31.133074634Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"47957fcb-4967-4c95-a64d-2806ad61f213","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket23/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(233)"}16252026/08/27 09:37:31 INFO Created nix-cache-info in bucket bucket=bucket1816262026/08/27 09:37:31 INFO Received cleanup request method=DELETE path=/api/pending_closures16272026/08/27 09:37:31 INFO Aborted multipart uploads count=01628--- PASS: TestReadProxyNarStreaming (2.14s)1629=== CONT TestClientErrorHandling/InvalidStorePath16302026/08/27 09:37:31 INFO Received uploads request method=POST path=/api/pending_closures16312026/08/27 09:37:31 INFO Received cleanup request method=DELETE path=/api/pending_closures16322026/08/27 09:37:31 INFO Aborted multipart uploads count=116332026/08/27 09:37:31 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete16342026-08-27 09:37:31.176 UTC [600] ERROR: Closure does not exist: id=116352026-08-27 09:37:31.176 UTC [600] CONTEXT: PL/pgSQL function commit_pending_closure(bigint) line 16 at RAISE16362026-08-27 09:37:31.176 UTC [600] STATEMENT: -- name: CommitPendingClosure :exec1637 SELECT commit_pending_closure($1::bigint)1638 1639--- PASS: TestService_cleanupPendingClosuresHandler (3.44s)1640=== CONT TestClientErrorHandling/ServerNotAvailable1641--- PASS: TestReadProxyNarinfo (1.98s)1642=== CONT TestClientErrorHandling/InvalidAuthToken1643=== NAME TestPinProtectsFromGC1644 client_integration_test.go:646: Pinned store path: /build/TestPinProtectsFromGC2160939581/001/store/xqs2jvk1vxd4qlsbwdnxcz86zhsdnfnj-pinned-file.txt1645 client_integration_test.go:647: Unpinned store path: /build/TestPinProtectsFromGC2160939581/001/store/slhbh1x6ngd8kxlz4hslx2caz5l140a1-unpinned-file.txt16462026/08/27 09:37:31 WARN mTLS auth: subject not in bound subjects subject="CN=writer"1647--- PASS: TestService_ReadAuthMiddleware (1.53s)1648=== CONT TestParseSingleRange/none1649=== CONT TestParseSingleRange/suffix_exceeds_size1650=== CONT TestParseSingleRange/suffix1651=== CONT TestParseSingleRange/end_clamped_to_size1652=== CONT TestParseSingleRange/open-ended1653=== CONT TestParseSingleRange/closed1654=== CONT TestParseSingleRange/malformed_end_before_start1655=== CONT TestParseSingleRange/malformed_both_empty1656=== CONT TestParseSingleRange/malformed_no_dash1657=== CONT TestParseSingleRange/single_byte1658=== CONT TestParseSingleRange/multi-range_ignored1659=== CONT TestParseSingleRange/unknown_unit1660=== CONT TestParseSingleRange/start_far_past_EOF1661=== CONT TestParseSingleRange/start_past_EOF1662--- PASS: TestParseSingleRange (0.00s)1663 --- PASS: TestParseSingleRange/none (0.00s)1664 --- PASS: TestParseSingleRange/suffix_exceeds_size (0.00s)1665 --- PASS: TestParseSingleRange/suffix (0.00s)1666 --- PASS: TestParseSingleRange/end_clamped_to_size (0.00s)1667 --- PASS: TestParseSingleRange/open-ended (0.00s)1668 --- PASS: TestParseSingleRange/closed (0.00s)1669 --- PASS: TestParseSingleRange/malformed_end_before_start (0.00s)1670 --- PASS: TestParseSingleRange/malformed_both_empty (0.00s)1671 --- PASS: TestParseSingleRange/malformed_no_dash (0.00s)1672 --- PASS: TestParseSingleRange/single_byte (0.00s)1673 --- PASS: TestParseSingleRange/multi-range_ignored (0.00s)1674 --- PASS: TestParseSingleRange/unknown_unit (0.00s)1675 --- PASS: TestParseSingleRange/start_far_past_EOF (0.00s)1676 --- PASS: TestParseSingleRange/start_past_EOF (0.00s)1677=== CONT TestCacheConfigHandler/full_config,_no_issuer1678=== CONT TestCacheConfigHandler/no_signing_keys1679=== CONT TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1680=== CONT TestCacheConfigHandler/no_cache_url_configured1681--- PASS: TestCacheConfigHandler (0.00s)1682 --- PASS: TestCacheConfigHandler/full_config,_no_issuer (0.00s)1683 --- PASS: TestCacheConfigHandler/no_signing_keys (0.00s)1684 --- PASS: TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator (0.00s)1685 --- PASS: TestCacheConfigHandler/no_cache_url_configured (0.00s)1686=== CONT TestService_AuthMiddleware_OIDC/valid_OIDC_token16872026/08/27 09:37:31 INFO OIDC auth successful provider=test1688=== CONT TestService_AuthMiddleware_OIDC/malformed_token_rejected16892026/08/27 09:37:31 WARN Authentication failed token_preview=not-a-valid-jwt token_length=15 oidc_error="no provider could verify the token (signature or issuer mismatch)" oidc_provider="" tried_providers=[test]1690=== CONT TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1691=== CONT TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected16922026-08-27 09:37:31.229 UTC [1037] ERROR: relation "goose_db_version" does not exist at character 3616932026-08-27 09:37:31.229 UTC [1037] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC16942026/08/27 09:37:31 WARN Authentication failed token_preview=eyJhbGciOi...aQ600Dtkyg token_length=702 oidc_error="bound claims validation failed: claim \"repository_owner\" value [otherorg] not in allowed patterns [myorg]" oidc_provider=test tried_providers=[test]1695=== CONT TestIsValidCachePath/wrong_extension1696=== CONT TestIsValidCachePath/narinfo1697=== CONT TestIsValidCachePath/leading_slash1698=== CONT TestIsValidCachePath/empty1699=== CONT TestIsValidCachePath/random_path1700=== CONT TestIsValidCachePath/invalid_char_u1701=== CONT TestIsValidCachePath/invalid_char_e1702=== CONT TestIsValidCachePath/traversal_in_middle1703=== CONT TestIsValidCachePath/traversal_parent1704=== CONT TestIsValidCachePath/index.html1705=== CONT TestIsValidCachePath/nix-cache-info1706=== CONT TestIsValidCachePath/realisation1707=== CONT TestIsValidCachePath/log1708=== CONT TestIsValidCachePath/nar_xz1709=== CONT TestIsValidCachePath/nar_zst1710=== CONT TestIsValidCachePath/narinfo_all_nix_base32_chars1711=== CONT TestIsValidCachePath/short_hash1712=== CONT TestIsValidCachePath/nar_bz21713=== CONT TestIsValidCachePath/ls1714=== CONT TestIsValidCachePath/nar_uncompressed1715--- PASS: TestIsValidCachePath (0.00s)1716 --- PASS: TestIsValidCachePath/wrong_extension (0.00s)1717 --- PASS: TestIsValidCachePath/narinfo (0.00s)1718 --- PASS: TestIsValidCachePath/leading_slash (0.00s)1719 --- PASS: TestIsValidCachePath/empty (0.00s)1720 --- PASS: TestIsValidCachePath/random_path (0.00s)1721 --- PASS: TestIsValidCachePath/invalid_char_u (0.00s)1722 --- PASS: TestIsValidCachePath/invalid_char_e (0.00s)1723 --- PASS: TestIsValidCachePath/traversal_in_middle (0.00s)1724 --- PASS: TestIsValidCachePath/traversal_parent (0.00s)1725 --- PASS: TestIsValidCachePath/index.html (0.00s)1726 --- PASS: TestIsValidCachePath/nix-cache-info (0.00s)1727 --- PASS: TestIsValidCachePath/realisation (0.00s)1728 --- PASS: TestIsValidCachePath/log (0.00s)1729 --- PASS: TestIsValidCachePath/nar_xz (0.00s)1730 --- PASS: TestIsValidCachePath/nar_zst (0.00s)1731 --- PASS: TestIsValidCachePath/narinfo_all_nix_base32_chars (0.00s)1732 --- PASS: TestIsValidCachePath/short_hash (0.00s)1733 --- PASS: TestIsValidCachePath/nar_bz2 (0.00s)1734 --- PASS: TestIsValidCachePath/ls (0.00s)1735 --- PASS: TestIsValidCachePath/nar_uncompressed (0.00s)1736=== CONT TestServerTLSConfig/no_client_CA1737=== CONT TestServerTLSConfig/not_a_PEM_file1738--- PASS: TestService_AuthMiddleware_OIDC (1.19s)1739 --- PASS: TestService_AuthMiddleware_OIDC/valid_OIDC_token (0.01s)1740 --- PASS: TestService_AuthMiddleware_OIDC/malformed_token_rejected (0.00s)1741 --- PASS: TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured (0.00s)1742 --- PASS: TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected (0.01s)1743=== NAME TestClientCADerivations1744 client_ca_test.go:136: Built CA derivation: /build/TestClientCADerivations1704596342/001/store/33pjna302bv5srgvqxv88ndslkf7p7a0-ca-test1745=== CONT TestServerTLSConfig/missing_CA_file1746--- PASS: TestServerTLSConfig (0.00s)1747 --- PASS: TestServerTLSConfig/no_client_CA (0.00s)1748 --- PASS: TestServerTLSConfig/not_a_PEM_file (0.00s)1749 --- PASS: TestServerTLSConfig/missing_CA_file (0.00s)17502026/08/27 09:37:31 OK 20241026095416_initial_model.sql (16.76ms)17512026/08/27 09:37:31 OK 20251210153512_drop_unused_gin_index.sql (2.99ms)17522026/08/27 09:37:31 OK 20251218171726_add_pins.sql (4.18ms)1753--- PASS: TestReadProxyDisabled (3.52s)17542026/08/27 09:37:31 OK 20260628120000_add_object_size_and_stats.sql (7.18ms)17552026/08/27 09:37:31 goose: successfully migrated database to version: 2026062812000017562026/08/27 09:37:31 OK 1_commit_pending_closure.sql (2.26ms)17572026/08/27 09:37:31 OK 2_object_stats_trigger.sql (1.23ms)17582026/08/27 09:37:31 goose: up to current file version: 21759=== NAME TestClientCADerivations1760 client_ca_test.go:139: Found 1 dependencies (including self)17612026-08-27 09:37:31.277 UTC [1093] ERROR: relation "goose_db_version" does not exist at character 3617622026-08-27 09:37:31.277 UTC [1093] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC17632026/08/27 09:37:31 WARN Failed to fetch cache config, uploading without size limit error="fetching cache config: server returned 404: 404 page not found\n"17642026/08/27 09:37:31 OK 20241026095416_initial_model.sql (10.46ms)17652026/08/27 09:37:31 INFO Created nix-cache-info in bucket bucket=bucket2017662026/08/27 09:37:31 OK 20251210153512_drop_unused_gin_index.sql (1.41ms)17672026/08/27 09:37:31 OK 20251218171726_add_pins.sql (2.57ms)17682026/08/27 09:37:31 WARN Request failed, retrying attempt=1 max_attempts=6 backoff=100ms error="Get \"http://localhost:19999/api/cache-config\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/cache-config17692026/08/27 09:37:31 OK 20260628120000_add_object_size_and_stats.sql (2.6ms)17702026/08/27 09:37:31 goose: successfully migrated database to version: 2026062812000017712026/08/27 09:37:31 OK 1_commit_pending_closure.sql (1.86ms)17722026/08/27 09:37:31 OK 2_object_stats_trigger.sql (1.14ms)17732026/08/27 09:37:31 goose: up to current file version: 21774--- PASS: TestReadProxyNarinfoAlreadyDecompressed (2.14s)17752026/08/27 09:37:31 INFO Received uploads request method=POST path=/api/pending_closures17762026/08/27 09:37:31 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)17772026/08/27 09:37:31 INFO Uploading xqs2jvk1vxd4qlsbwdnxcz86zhsdnfnj-pinned-file.txt (128B)1778=== NAME TestClientIntegration1779 client_integration_test.go:276: Created store path: /build/TestClientIntegration2571754508/002/store/iqq32z9a5dcwqmb8sb5hv9znz1yd5haf-test-file.txt17802026/08/27 09:37:31 WARN Failed to register uploaded object key=nar/0qmhsg42pqsz7x6hh02j8gznnac3yi3qpjhrqbwvz88i194jbb5s.nar.zst error="server returned 404: 404 page not found\n"17812026/08/27 09:37:31 WARN Failed to register uploaded object key=xqs2jvk1vxd4qlsbwdnxcz86zhsdnfnj.ls error="server returned 404: 404 page not found\n"17822026/08/27 09:37:31 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign1783--- PASS: TestReadProxyInvalidPath (2.16s)17842026/08/27 09:37:31 INFO Signed narinfos id=1 count=117852026/08/27 09:37:31 INFO Uploading 1 narinfos17862026/08/27 09:37:31 WARN Failed to fetch cache config, uploading without size limit error="fetching cache config: server returned 404: 404 page not found\n"17872026/08/27 09:37:31 WARN Failed to register uploaded object key=xqs2jvk1vxd4qlsbwdnxcz86zhsdnfnj.narinfo error="server returned 404: 404 page not found\n"17882026/08/27 09:37:31 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete17892026/08/27 09:37:31 INFO Completed upload id=117902026/08/27 09:37:31 INFO Upload complete. (99ms)17912026/08/27 09:37:31 INFO Received uploads request method=POST path=/api/pending_closures1792--- PASS: TestService_AuthMiddleware_MTLSProxyHeader (1.68s)17932026/08/27 09:37:31 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)17942026/08/27 09:37:31 INFO Uploading 33pjna302bv5srgvqxv88ndslkf7p7a0-ca-test (144B)17952026/08/27 09:37:31 WARN Failed to register uploaded object key=nar/04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n.nar.zst error="server returned 404: 404 page not found\n"17962026/08/27 09:37:31 WARN Failed to fetch cache config, uploading without size limit error="fetching cache config: server returned 404: 404 page not found\n"17972026/08/27 09:37:31 WARN Request failed, retrying attempt=2 max_attempts=6 backoff=200.501584ms error="Get \"http://localhost:19999/api/cache-config\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/cache-config17982026/08/27 09:37:31 WARN Failed to register uploaded object key=log/8fzh64vvz8j8iv6c944nnv9ldgzcfp4z-ca-test.drv error="server returned 404: 404 page not found\n"17992026/08/27 09:37:31 WARN Failed to register uploaded object key=33pjna302bv5srgvqxv88ndslkf7p7a0.ls error="server returned 404: 404 page not found\n"18002026/08/27 09:37:31 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign18012026/08/27 09:37:31 INFO Signed narinfos id=1 count=118022026/08/27 09:37:31 INFO Uploading 1 narinfos18032026/08/27 09:37:31 WARN Failed to register uploaded object key=33pjna302bv5srgvqxv88ndslkf7p7a0.narinfo error="server returned 404: 404 page not found\n"18042026/08/27 09:37:31 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete18052026/08/27 09:37:31 INFO Completed upload id=118062026/08/27 09:37:31 INFO Upload complete. (105ms)18072026/08/27 09:37:31 WARN Failed to fetch cache config, uploading without size limit error="fetching cache config: server returned 404: 404 page not found\n"1808=== NAME TestClientCADerivations1809 client_ca_test.go:180: Narinfo contains CA field: StorePath: /build/TestClientCADerivations1704596342/001/store/33pjna302bv5srgvqxv88ndslkf7p7a0-ca-test1810 URL: nar/04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n.nar.zst1811 Compression: zstd1812 NarHash: sha256:04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n1813 NarSize: 1441814 References: 1815 Deriver: /build/TestClientCADerivations1704596342/001/store/8fzh64vvz8j8iv6c944nnv9ldgzcfp4z-ca-test.drv1816 CA: fixed:r:sha256:04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n1817 client_ca_test.go:185: Checking for realisation files in S3...1818 client_ca_test.go:207: No realisation files uploaded - this is expected for locally built CA derivations1819 client_ca_test.go:208: Locally built CA derivations don't have DrvOutput IDs until they're substituted from a cache18202026/08/27 09:37:31 WARN Failed to fetch cache config, uploading without size limit error="fetching cache config: server returned 404: 404 page not found\n"18212026/08/27 09:37:31 INFO Received uploads request method=POST path=/api/pending_closures18222026/08/27 09:37:31 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)18232026/08/27 09:37:31 INFO Uploading iqq32z9a5dcwqmb8sb5hv9znz1yd5haf-test-file.txt (152B)18242026/08/27 09:37:31 INFO Received uploads request method=POST path=/api/pending_closures18252026/08/27 09:37:31 INFO Received uploads request method=POST path=/api/pending_closures18262026/08/27 09:37:31 INFO Received uploads request method=POST path=/api/pending_closures18272026/08/27 09:37:31 INFO Received uploads request method=POST path=/api/pending_closures18282026/08/27 09:37:31 WARN Failed to register uploaded object key=nar/11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1.nar.zst error="server returned 404: 404 page not found\n"18292026/08/27 09:37:31 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)18302026/08/27 09:37:31 INFO Uploading slhbh1x6ngd8kxlz4hslx2caz5l140a1-unpinned-file.txt (128B)18312026/08/27 09:37:31 WARN Failed to register uploaded object key=iqq32z9a5dcwqmb8sb5hv9znz1yd5haf.ls error="server returned 404: 404 page not found\n"18322026/08/27 09:37:31 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign18332026/08/27 09:37:31 INFO Signed narinfos id=1 count=118342026/08/27 09:37:31 INFO Uploading 1 narinfos18352026/08/27 09:37:31 WARN Failed to register uploaded object key=nar/0gh0wdnr6gmk4r5sfzzh12isnl14rgmkrwy1ws41hpsrlndclg5h.nar.zst error="server returned 404: 404 page not found\n"18362026/08/27 09:37:31 WARN Failed to register uploaded object key=iqq32z9a5dcwqmb8sb5hv9znz1yd5haf.narinfo error="server returned 404: 404 page not found\n"18372026/08/27 09:37:31 WARN Failed to register uploaded object key=slhbh1x6ngd8kxlz4hslx2caz5l140a1.ls error="server returned 404: 404 page not found\n"18382026/08/27 09:37:31 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete18392026/08/27 09:37:31 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign18402026/08/27 09:37:31 WARN Authentication failed token_preview=invalid-token token_length=13 reason="static API token mismatch"18412026/08/27 09:37:31 INFO Signed narinfos id=2 count=118422026/08/27 09:37:31 INFO Uploading 1 narinfos18432026/08/27 09:37:31 WARN Failed to register uploaded object key=slhbh1x6ngd8kxlz4hslx2caz5l140a1.narinfo error="server returned 404: 404 page not found\n"18442026/08/27 09:37:31 INFO Completed upload id=118452026/08/27 09:37:31 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete18462026/08/27 09:37:31 INFO Upload complete. (90ms)1847--- PASS: TestResurrectedObjectNotDeleted (1.70s)18482026/08/27 09:37:31 INFO Completed upload id=218492026/08/27 09:37:31 INFO Upload complete. (79ms)1850=== NAME TestClientIntegration1851 client_integration_test.go:292: Retrieved narinfo from S3:1852 StorePath: /build/TestClientIntegration2571754508/002/store/iqq32z9a5dcwqmb8sb5hv9znz1yd5haf-test-file.txt1853 URL: nar/11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1.nar.zst1854 Compression: zstd1855 NarHash: sha256:11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk11856 NarSize: 1521857 References: 1858 CA: fixed:r:sha256:11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk11859 client_integration_test.go:293: Retrieved .ls file from S3 (compressed size: 77 bytes)1860 client_integration_test.go:293: Decompressed .ls content (64 bytes):1861 {"version":1,"root":{"type":"regular","size":39,"narOffset":96}}1862 client_integration_test.go:296: Testing garbage collection...18632026/08/27 09:37:31 WARN mTLS auth: subject not in bound subjects subject="CN=reader"18642026/08/27 09:37:31 WARN mTLS auth: subject not in bound subjects subject="CN=writer"1865--- PASS: TestService_NativeMTLS (1.80s)18662026/08/27 09:37:31 INFO Received create pin request method=POST path=/api/pins/myapp18672026/08/27 09:37:31 INFO Starting cleanup of old closures method=DELETE path=/api/closures18682026/08/27 09:37:31 INFO Garbage collection started18692026/08/27 09:37:31 INFO Created/updated pin name=myapp store_path=/build/TestPinProtectsFromGC2160939581/001/store/xqs2jvk1vxd4qlsbwdnxcz86zhsdnfnj-pinned-file.txt narinfo_key=xqs2jvk1vxd4qlsbwdnxcz86zhsdnfnj.narinfo18702026/08/27 09:37:31 INFO Starting cleanup of old closures method=DELETE path=/api/closures18712026/08/27 09:37:31 INFO Garbage collection started18722026/08/27 09:37:31 INFO Aborted multipart uploads count=018732026/08/27 09:37:31 WARN Force mode enabled - objects will be deleted immediately without grace period18742026/08/27 09:37:31 INFO Aborted multipart uploads count=018752026/08/27 09:37:31 WARN Force mode enabled - objects will be deleted immediately without grace period1876=== NAME TestOrphanedObjectsGC1877 orphaned_objects_gc_test.go:290: GC Test Summary:1878 orphaned_objects_gc_test.go:291: - Kept: 2 objects from closure A1879 orphaned_objects_gc_test.go:292: - Deleted: 2 objects from closure B1880 orphaned_objects_gc_test.go:293: - Deleted: 6 orphaned chain objects (X1->X2->X3)1881 orphaned_objects_gc_test.go:294: - Deleted: 2 orphaned single objects (Y)1882 orphaned_objects_gc_test.go:295: - Total deleted: 10 objects1883--- PASS: TestOrphanedObjectsGC (1.59s)18842026/08/27 09:37:31 INFO Received uploads request method=POST path=/api/pending_closures1885=== NAME TestClientCADerivations1886 client_ca_test.go:258: nix copy output: warning: you don't have Internet access; disabling some network-dependent features1887 warning: failed to create TLS context for AWS credential providers; SSO, STS WebIdentity, and ECS container authentication will be unavailable1888 error: binary cache 's3://bucket18?endpoint=http://localhost:40405®ion=eu-west-1' is for Nix stores with prefix '/nix/store', not '/build/TestClientCADerivations1704596342/001/store'1889 client_ca_test.go:259: nix copy failed (might be expected with isolated stores): exit status 11890--- PASS: TestClientCADerivations (3.76s)18912026/08/27 09:37:31 WARN Request failed, retrying attempt=3 max_attempts=6 backoff=426.47408ms error="Get \"http://localhost:19999/api/cache-config\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/cache-config18922026/08/27 09:37:31 INFO Created nix-cache-info in bucket bucket=bucket1918932026/08/27 09:37:31 INFO Received complete multipart upload request method=POST path=/api/multipart/complete18942026/08/27 09:37:31 INFO Received cleanup request method=DELETE path=/api/pending_closures18952026/08/27 09:37:31 INFO Aborted multipart uploads count=11896--- PASS: TestReadProxyRangeRequest (3.86s)1897--- PASS: TestMultipartCleanup (2.45s)18982026/08/27 09:37:31 INFO Completed multipart upload object_key=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhh0100000000000000000000.nar.zst upload_id=NTYxNTAwNjUtNmMwNy00NDg5LWExZmYtZjY0YWQ2NzRkNGVjLmQ4ZTk4NzhmLTRiOTMtNGQ0MC05OGI2LWZiZjgyYjVlNjI3NHgxNzg3ODIzNDQ5NzE5NDU1MTg0 parts=1218992026/08/27 09:37:31 INFO Received uploads request method=POST path=/api/pending_closures1900--- PASS: TestCompletedNarNotReofferedAcrossClosures (3.87s)1901=== NAME TestClientWithDependencies1902 client_integration_test.go:593: Built derivation: /build/TestClientWithDependencies2802537714/001/store/mksxcxd6swh6a7lv312bykklnyx0zkgs-test-script1903 client_integration_test.go:595: Found 1 dependencies (including self)19042026/08/27 09:37:31 WARN Failed to fetch cache config, uploading without size limit error="fetching cache config: server returned 404: 404 page not found\n"19052026/08/27 09:37:31 INFO Received uploads request method=POST path=/api/pending_closures19062026/08/27 09:37:31 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)19072026/08/27 09:37:31 INFO Uploading mksxcxd6swh6a7lv312bykklnyx0zkgs-test-script (136B)19082026/08/27 09:37:31 WARN Failed to register uploaded object key=log/6nn5iizdhy83qnbg67604afx2cnbfn8x-test-script.drv error="server returned 404: 404 page not found\n"19092026/08/27 09:37:31 INFO Received complete multipart upload request method=POST path=/api/multipart/complete19102026/08/27 09:37:31 INFO Completed multipart upload object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst upload_id=NTYxNTAwNjUtNmMwNy00NDg5LWExZmYtZjY0YWQ2NzRkNGVjLjI4OWY3MjAwLTM2MTctNDM2MS04OTg0LWJjNmU0Mjc2MjdkZXgxNzg3ODIzNDUxNDUyNDg0MDc3 parts=1019112026/08/27 09:37:31 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete19122026/08/27 09:37:31 INFO Completed upload id=119132026/08/27 09:37:31 INFO Received get closure request method=GET path=/api/closures/0000000000000000000000000000000019142026/08/27 09:37:31 INFO Received uploads request method=POST path=/api/pending_closures19152026/08/27 09:37:31 INFO Starting cleanup of old closures method=DELETE path=/api/closures19162026/08/27 09:37:31 INFO Aborted multipart uploads count=019172026/08/27 09:37:31 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=2 objects-deleted-after-grace-period=0 objects-failed-to-delete=019182026/08/27 09:37:31 INFO Vacuumed table table=pending_closures19192026/08/27 09:37:31 INFO Vacuumed table table=pending_objects19202026/08/27 09:37:31 INFO Vacuumed table table=multipart_uploads19212026/08/27 09:37:31 INFO Vacuumed table table=closures19222026/08/27 09:37:31 INFO Vacuumed table table=objects19232026/08/27 09:37:31 INFO Received get closure request method=GET path=/api/closures/000000000000000000000000000000001924--- PASS: TestService_createPendingClosureHandler (4.23s)1925--- PASS: TestUploadHandlersRejectOversizedBody (0.20s)1926 --- PASS: TestUploadHandlersRejectOversizedBody/request_more_parts (0.11s)1927 --- PASS: TestUploadHandlersRejectOversizedBody/complete_multipart (0.10s)1928 --- PASS: TestUploadHandlersRejectOversizedBody/create_pending_closure (1.59s)19292026/08/27 09:37:32 WARN Request failed, retrying attempt=4 max_attempts=6 backoff=787.729644ms error="Get \"http://localhost:19999/api/cache-config\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/cache-config1930--- PASS: TestMetricsInventory (2.38s)19312026/08/27 09:37:32 WARN Failed to register uploaded object key=nar/1s7yghixzpinyqm7g59kyn3sav81pmnsdi5v63m6x4r2vacy907j.nar.zst error="server returned 404: 404 page not found\n"19322026/08/27 09:37:32 WARN Failed to register uploaded object key=mksxcxd6swh6a7lv312bykklnyx0zkgs.ls error="server returned 404: 404 page not found\n"19332026/08/27 09:37:32 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign19342026/08/27 09:37:32 INFO Signed narinfos id=1 count=119352026/08/27 09:37:32 INFO Uploading 1 narinfos19362026/08/27 09:37:32 WARN Failed to register uploaded object key=mksxcxd6swh6a7lv312bykklnyx0zkgs.narinfo error="server returned 404: 404 page not found\n"19372026/08/27 09:37:32 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete19382026/08/27 09:37:32 INFO Completed upload id=119392026/08/27 09:37:32 INFO Upload complete. (384ms)1940--- PASS: TestObjectStatsTrigger (1.97s)1941=== NAME TestClientWithDependencies1942 client_integration_test.go:597: Skipping nix copy test - isolated store (/build/TestClientWithDependencies2802537714/001/store) requires matching store prefix1943--- PASS: TestClientWithDependencies (4.37s)1944=== NAME TestOrphanedObjectsGCStressTest1945 orphaned_objects_gc_test.go:425: Created 10 active closures, 5 to-delete closures, 20 orphaned chains1946 orphaned_objects_gc_test.go:446: Marked 210 objects for deletion19472026/08/27 09:37:32 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=2001 objects-failed-to-delete=019482026/08/27 09:37:32 INFO Vacuumed table table=pending_closures19492026/08/27 09:37:32 INFO Vacuumed table table=pending_objects19502026/08/27 09:37:32 INFO Vacuumed table table=multipart_uploads19512026/08/27 09:37:32 INFO Vacuumed table table=closures19522026/08/27 09:37:32 INFO Vacuumed table table=objects1953 orphaned_objects_gc_test.go:509: Stress test completed successfully:1954 orphaned_objects_gc_test.go:510: - Active objects preserved: 201955 orphaned_objects_gc_test.go:511: - Objects deleted: 2101956 orphaned_objects_gc_test.go:512: - Total GC'd: 2101957--- PASS: TestOrphanedObjectsGCStressTest (3.13s)19582026/08/27 09:37:32 WARN Request failed, retrying attempt=5 max_attempts=6 backoff=1.675755226s error="Get \"http://localhost:19999/api/cache-config\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/cache-config19592026/08/27 09:37:33 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=2001 objects-failed-to-delete=019602026/08/27 09:37:33 INFO Vacuumed table table=pending_closures19612026/08/27 09:37:33 INFO Vacuumed table table=pending_objects19622026/08/27 09:37:33 INFO Vacuumed table table=multipart_uploads19632026/08/27 09:37:33 INFO Vacuumed table table=closures19642026/08/27 09:37:33 INFO Vacuumed table table=objects19652026/08/27 09:37:33 INFO Garbage collection progress phase="" failed_uploads_deleted=0 old_closures_deleted=1 objects_marked=3 objects_deleted=2001 objects_failed=01966=== NAME TestClientIntegration1967 client_integration_test.go:303: Objects in database after GC:1968 client_integration_test.go:303: Successfully deleted all objects with GC --force19692026/08/27 09:37:33 INFO Garbage collection progress phase="" failed_uploads_deleted=0 old_closures_deleted=1 objects_marked=3 objects_deleted=2001 objects_failed=01970=== NAME TestPinProtectsFromGC1971 client_integration_test.go:709: Pin successfully protected closure from garbage collection1972--- PASS: TestClientIntegration (5.62s)1973--- PASS: TestPinProtectsFromGC (5.69s)19742026/08/27 09:37:34 WARN Failed to fetch cache config, uploading without size limit error="fetching cache config: sending request: request failed after retries: Get \"http://localhost:19999/api/cache-config\": dial tcp [::1]:19999: connect: connection refused"19752026/08/27 09:37:34 WARN Request failed, retrying attempt=1 max_attempts=6 backoff=100ms error="Post \"http://localhost:19999/api/pending_closures\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/pending_closures19762026/08/27 09:37:34 WARN Request failed, retrying attempt=2 max_attempts=6 backoff=209.495367ms error="Post \"http://localhost:19999/api/pending_closures\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/pending_closures19772026/08/27 09:37:34 WARN Request failed, retrying attempt=3 max_attempts=6 backoff=418.422957ms error="Post \"http://localhost:19999/api/pending_closures\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/pending_closures19782026/08/27 09:37:35 WARN Rate limiter enabled after throttle name=s3-test rate=519792026/08/27 09:37:35 WARN S3 rate limit hit operation="complete multipart upload" error="Please reduce your request rate."1980=== NAME TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle1981 throttle_test.go:213: Proxy stats: total=15, throttled=10, completeMultipart=101982 throttle_test.go:215: Rate limiter: enabled=true, rate=5.001983--- PASS: TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle (7.54s)19842026/08/27 09:37:35 WARN Request failed, retrying attempt=4 max_attempts=6 backoff=779.098322ms error="Post \"http://localhost:19999/api/pending_closures\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/pending_closures19852026/08/27 09:37:36 WARN Request failed, retrying attempt=5 max_attempts=6 backoff=1.711779503s error="Post \"http://localhost:19999/api/pending_closures\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/pending_closures1986--- PASS: TestClientErrorHandling (0.00s)1987 --- PASS: TestClientErrorHandling/InvalidStorePath (0.16s)1988 --- PASS: TestClientErrorHandling/InvalidAuthToken (0.28s)1989 --- PASS: TestClientErrorHandling/ServerNotAvailable (6.66s)1990PASS1991{"timestamp":"2026-08-27T09:37:37.836826853Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:49140","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1836,"threadName":"rustfs-worker","threadId":"ThreadId(311)"}19922026-08-27 09:37:38.097 UTC [111] LOG: received smart shutdown request19932026-08-27 09:37:38.104 UTC [111] LOG: background worker "logical replication launcher" (PID 121) exited with exit code 119942026-08-27 09:37:38.115 UTC [116] LOG: shutting down19952026-08-27 09:37:38.116 UTC [116] LOG: checkpoint starting: shutdown immediate19962026-08-27 09:37:38.686 UTC [116] LOG: checkpoint complete: wrote 12051 buffers (73.6%), wrote 3 SLRU buffers; 0 WAL file(s) added, 0 removed, 13 recycled; write=0.228 s, sync=0.335 s, total=0.571 s; sync files=15167, longest=0.002 s, average=0.001 s; distance=208930 kB, estimate=208930 kB; lsn=0/E36C590, redo lsn=0/E36C59019972026-08-27 09:37:38.778 UTC [111] LOG: database system is shut down1998Running OIDC tests...1999=== RUN TestGlobMatch2000=== PAUSE TestGlobMatch2001=== RUN TestAudienceForIssuer2002=== PAUSE TestAudienceForIssuer2003=== RUN TestValidateToken_ValidToken2004=== PAUSE TestValidateToken_ValidToken2005=== RUN TestValidateToken_WrongAudience2006=== PAUSE TestValidateToken_WrongAudience2007=== RUN TestValidateToken_Expired2008=== PAUSE TestValidateToken_Expired2009=== RUN TestValidateToken_BoundClaimsMismatch2010=== PAUSE TestValidateToken_BoundClaimsMismatch2011=== RUN TestValidateToken_BoundSubjectMismatch2012=== PAUSE TestValidateToken_BoundSubjectMismatch2013=== RUN TestValidateToken_MultipleProviders2014=== PAUSE TestValidateToken_MultipleProviders2015=== RUN TestValidateToken_NoMatchingProvider2016=== PAUSE TestValidateToken_NoMatchingProvider2017=== CONT TestGlobMatch2018=== CONT TestValidateToken_BoundClaimsMismatch2019=== RUN TestGlobMatch/foo_foo2020=== PAUSE TestGlobMatch/foo_foo2021=== RUN TestGlobMatch/foo_bar2022=== PAUSE TestGlobMatch/foo_bar2023=== RUN TestGlobMatch/*_2024=== CONT TestValidateToken_MultipleProviders2025=== CONT TestValidateToken_NoMatchingProvider2026=== CONT TestValidateToken_BoundSubjectMismatch2027=== CONT TestValidateToken_Expired2028=== CONT TestValidateToken_ValidToken2029=== CONT TestAudienceForIssuer2030=== CONT TestValidateToken_WrongAudience2031=== PAUSE TestGlobMatch/*_2032=== RUN TestGlobMatch/*_anything2033--- PASS: TestAudienceForIssuer (0.00s)2034=== PAUSE TestGlobMatch/*_anything2035=== RUN TestGlobMatch/foo*_foo2036=== PAUSE TestGlobMatch/foo*_foo2037=== RUN TestGlobMatch/foo*_foobar2038=== PAUSE TestGlobMatch/foo*_foobar2039=== RUN TestGlobMatch/foo*_bar2040=== PAUSE TestGlobMatch/foo*_bar2041=== RUN TestGlobMatch/*bar_bar2042=== PAUSE TestGlobMatch/*bar_bar2043=== RUN TestGlobMatch/*bar_foobar2044=== PAUSE TestGlobMatch/*bar_foobar2045=== RUN TestGlobMatch/*bar_foo2046=== PAUSE TestGlobMatch/*bar_foo2047=== RUN TestGlobMatch/foo*bar_foobar2048=== PAUSE TestGlobMatch/foo*bar_foobar2049=== RUN TestGlobMatch/foo*bar_foo123bar2050=== PAUSE TestGlobMatch/foo*bar_foo123bar2051=== RUN TestGlobMatch/foo*bar_foobarbaz2052=== PAUSE TestGlobMatch/foo*bar_foobarbaz2053=== RUN TestGlobMatch/*/*_foo/bar2054=== PAUSE TestGlobMatch/*/*_foo/bar2055=== RUN TestGlobMatch/*/*_foo2056=== PAUSE TestGlobMatch/*/*_foo2057=== RUN TestGlobMatch/refs/heads/*_refs/heads/main2058=== PAUSE TestGlobMatch/refs/heads/*_refs/heads/main2059=== RUN TestGlobMatch/refs/heads/*_refs/tags/v1.02060=== PAUSE TestGlobMatch/refs/heads/*_refs/tags/v1.02061=== RUN TestGlobMatch/refs/*/main_refs/heads/main2062=== PAUSE TestGlobMatch/refs/*/main_refs/heads/main2063=== RUN TestGlobMatch/fo?_foo2064=== PAUSE TestGlobMatch/fo?_foo2065=== RUN TestGlobMatch/fo?_fo2066=== PAUSE TestGlobMatch/fo?_fo2067=== RUN TestGlobMatch/fo?_fooo2068=== PAUSE TestGlobMatch/fo?_fooo2069=== RUN TestGlobMatch/?oo_foo2070=== PAUSE TestGlobMatch/?oo_foo2071=== RUN TestGlobMatch/?oo_boo2072=== PAUSE TestGlobMatch/?oo_boo2073=== RUN TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main2074=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main2075=== RUN TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main2076=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main2077=== CONT TestGlobMatch/foo_foo2078=== CONT TestGlobMatch/*/*_foo/bar2079=== CONT TestGlobMatch/refs/*/main_refs/heads/main2080=== CONT TestGlobMatch/*bar_bar2081=== CONT TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main2082=== CONT TestGlobMatch/?oo_foo2083=== CONT TestGlobMatch/*/*_foo2084=== CONT TestGlobMatch/fo?_fooo2085=== CONT TestGlobMatch/foo*bar_foobarbaz2086=== CONT TestGlobMatch/foo*bar_foo123bar2087=== CONT TestGlobMatch/*_anything2088=== CONT TestGlobMatch/*bar_foobar2089=== CONT TestGlobMatch/*bar_foo2090=== CONT TestGlobMatch/*_2091=== CONT TestGlobMatch/foo*_foo2092=== CONT TestGlobMatch/foo*_bar2093=== CONT TestGlobMatch/foo*_foobar2094=== CONT TestGlobMatch/foo_bar2095=== CONT TestGlobMatch/foo*bar_foobar2096=== CONT TestGlobMatch/refs/heads/*_refs/tags/v1.02097=== CONT TestGlobMatch/fo?_foo2098=== CONT TestGlobMatch/fo?_fo2099=== CONT TestGlobMatch/?oo_boo2100=== CONT TestGlobMatch/refs/heads/*_refs/heads/main2101=== CONT TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main2102--- PASS: TestGlobMatch (0.00s)2103 --- PASS: TestGlobMatch/foo_foo (0.00s)2104 --- PASS: TestGlobMatch/*/*_foo/bar (0.00s)2105 --- PASS: TestGlobMatch/refs/*/main_refs/heads/main (0.00s)2106 --- PASS: TestGlobMatch/*bar_bar (0.00s)2107 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main (0.00s)2108 --- PASS: TestGlobMatch/?oo_foo (0.00s)2109 --- PASS: TestGlobMatch/*/*_foo (0.00s)2110 --- PASS: TestGlobMatch/fo?_fooo (0.00s)2111 --- PASS: TestGlobMatch/foo*bar_foobarbaz (0.00s)2112 --- PASS: TestGlobMatch/foo*bar_foo123bar (0.00s)2113 --- PASS: TestGlobMatch/*_anything (0.00s)2114 --- PASS: TestGlobMatch/*bar_foobar (0.00s)2115 --- PASS: TestGlobMatch/*bar_foo (0.00s)2116 --- PASS: TestGlobMatch/*_ (0.00s)2117 --- PASS: TestGlobMatch/foo*_foo (0.00s)2118 --- PASS: TestGlobMatch/foo*_bar (0.00s)2119 --- PASS: TestGlobMatch/foo*_foobar (0.00s)2120 --- PASS: TestGlobMatch/foo_bar (0.00s)2121 --- PASS: TestGlobMatch/foo*bar_foobar (0.00s)2122 --- PASS: TestGlobMatch/refs/heads/*_refs/tags/v1.0 (0.00s)2123 --- PASS: TestGlobMatch/fo?_foo (0.00s)2124 --- PASS: TestGlobMatch/fo?_fo (0.00s)2125 --- PASS: TestGlobMatch/?oo_boo (0.00s)2126 --- PASS: TestGlobMatch/refs/heads/*_refs/heads/main (0.00s)2127 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main (0.00s)21282026/08/27 09:37:39 INFO OIDC provider initialized name=provider121292026/08/27 09:37:39 INFO OIDC provider initialized name=test21302026/08/27 09:37:39 INFO OIDC provider initialized name=test21312026/08/27 09:37:39 INFO OIDC provider initialized name=test21322026/08/27 09:37:39 INFO OIDC provider initialized name=test21332026/08/27 09:37:39 INFO OIDC provider initialized name=provider121342026/08/27 09:37:39 INFO OIDC provider initialized name=test21352026/08/27 09:37:39 INFO OIDC provider initialized name=provider22136--- PASS: TestValidateToken_Expired (0.01s)2137--- PASS: TestValidateToken_NoMatchingProvider (0.01s)2138--- PASS: TestValidateToken_BoundSubjectMismatch (0.01s)2139--- PASS: TestValidateToken_ValidToken (0.01s)2140--- PASS: TestValidateToken_BoundClaimsMismatch (0.01s)2141--- PASS: TestValidateToken_WrongAudience (0.01s)2142--- PASS: TestValidateToken_MultipleProviders (0.01s)2143PASS2144Running hook tests...2145=== RUN TestSendPathsEmpty2146=== PAUSE TestSendPathsEmpty2147=== RUN TestQueueEnqueueAndFetch2148=== PAUSE TestQueueEnqueueAndFetch2149=== RUN TestQueueDeduplication2150=== PAUSE TestQueueDeduplication2151=== RUN TestQueueRemove2152=== PAUSE TestQueueRemove2153=== RUN TestQueueFetchBatchLimit2154=== PAUSE TestQueueFetchBatchLimit2155=== RUN TestQueueRetryMovesToBack2156=== PAUSE TestQueueRetryMovesToBack2157=== RUN TestQueueFetchRemoveLifecycle2158=== PAUSE TestQueueFetchRemoveLifecycle2159=== RUN TestQueueConcurrentWriters2160=== PAUSE TestQueueConcurrentWriters2161=== RUN TestQueueRemoveLargeClosure2162=== PAUSE TestQueueRemoveLargeClosure2163=== RUN TestServerClientIntegration2164=== PAUSE TestServerClientIntegration2165=== RUN TestServerQueueError2166=== PAUSE TestServerQueueError2167=== RUN TestGetListenerSocketActivation2168 server_test.go:210: === RUN TestGetListenerSocketActivation2169 --- PASS: TestGetListenerSocketActivation (0.00s)2170 PASS2171 2172--- PASS: TestGetListenerSocketActivation (0.01s)2173=== RUN TestDrainIsolatesPoisonPath2174=== PAUSE TestDrainIsolatesPoisonPath2175=== RUN TestRunNotBlockedByPoisonHead2176=== PAUSE TestRunNotBlockedByPoisonHead2177=== RUN TestDrainGivesUpWhenServerDown2178=== PAUSE TestDrainGivesUpWhenServerDown2179=== RUN TestFailedPathPrunedByLaterClosure2180=== PAUSE TestFailedPathPrunedByLaterClosure2181=== RUN TestWorkerUploadsAndRemoves2182=== PAUSE TestWorkerUploadsAndRemoves2183=== RUN TestWorkerSkipsGCdPaths2184=== PAUSE TestWorkerSkipsGCdPaths2185=== RUN TestWorkerPrunesClosureDeps2186=== PAUSE TestWorkerPrunesClosureDeps2187=== CONT TestSendPathsEmpty2188=== CONT TestWorkerSkipsGCdPaths2189=== CONT TestQueueRetryMovesToBack2190=== CONT TestDrainGivesUpWhenServerDown2191=== CONT TestWorkerPrunesClosureDeps2192=== CONT TestWorkerUploadsAndRemoves2193--- PASS: TestSendPathsEmpty (0.00s)2194=== CONT TestFailedPathPrunedByLaterClosure2195=== CONT TestServerQueueError2196=== CONT TestQueueFetchBatchLimit2197=== CONT TestQueueRemoveLargeClosure2198=== CONT TestServerClientIntegration2199=== CONT TestQueueRemove2200=== CONT TestRunNotBlockedByPoisonHead2201=== CONT TestQueueDeduplication2202=== CONT TestQueueEnqueueAndFetch2203=== CONT TestQueueConcurrentWriters22042026/08/27 09:37:39 ERROR Failed to queue paths error="permission denied" count=12205=== CONT TestQueueFetchRemoveLifecycle2206=== CONT TestDrainIsolatesPoisonPath2207--- PASS: TestServerQueueError (0.00s)2208--- PASS: TestServerClientIntegration (0.00s)22092026/08/27 09:37:39 INFO Uploading batch count=122102026/08/27 09:37:39 ERROR Upload failed error="upload failed" count=12211--- PASS: TestQueueDeduplication (0.01s)22122026/08/27 09:37:39 INFO Uploading batch count=422132026/08/27 09:37:39 ERROR Upload failed error="upload failed" count=422142026/08/27 09:37:39 INFO Upload queue status pending=222152026/08/27 09:37:39 INFO Uploading batch count=122162026/08/27 09:37:39 INFO Uploading batch count=22217--- PASS: TestQueueRemove (0.02s)22182026/08/27 09:37:39 INFO Upload queue status pending=222192026/08/27 09:37:39 INFO Uploading batch count=222202026/08/27 09:37:39 ERROR Upload failed error="upload failed" count=222212026/08/27 09:37:39 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown2904040908/002/a22222026/08/27 09:37:39 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainIsolatesPoisonPath1967153599/002/bbb22232026/08/27 09:37:39 INFO Uploading batch count=122242026/08/27 09:37:39 INFO Upload queue status pending=222252026/08/27 09:37:39 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerSkipsGCdPaths2123101486/002/nonexistent22262026/08/27 09:37:39 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown2904040908/002/b2227--- PASS: TestQueueFetchBatchLimit (0.02s)22282026/08/27 09:37:39 INFO Upload queue status pending=322292026/08/27 09:37:39 INFO Uploading batch count=122302026/08/27 09:37:39 ERROR Upload failed error="upload failed" count=12231--- PASS: TestQueueFetchRemoveLifecycle (0.02s)2232--- PASS: TestQueueRetryMovesToBack (0.02s)22332026/08/27 09:37:39 INFO Uploading batch count=222342026/08/27 09:37:39 ERROR Upload failed error="upload failed" count=222352026/08/27 09:37:39 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown2904040908/002/c22362026/08/27 09:37:39 INFO Uploading batch count=12237--- PASS: TestQueueEnqueueAndFetch (0.02s)22382026/08/27 09:37:39 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown2904040908/002/d22392026/08/27 09:37:39 INFO Uploading batch count=122402026/08/27 09:37:39 INFO Uploading batch count=122412026/08/27 09:37:39 ERROR Upload failed error="upload failed" count=122422026/08/27 09:37:39 INFO Uploading batch count=222432026/08/27 09:37:39 ERROR Upload failed error="upload failed" count=222442026/08/27 09:37:39 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown2904040908/002/e22452026/08/27 09:37:39 INFO Uploading batch count=122462026/08/27 09:37:39 ERROR Upload failed error="upload failed" count=122472026/08/27 09:37:39 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown2904040908/002/f22482026/08/27 09:37:39 INFO Uploading batch count=122492026/08/27 09:37:39 ERROR Upload failed error="upload failed" count=122502026/08/27 09:37:39 ERROR Drain finished with paths left in queue remaining=1022512026/08/27 09:37:39 ERROR Drain finished with paths left in queue remaining=12252--- PASS: TestFailedPathPrunedByLaterClosure (0.02s)2253--- PASS: TestDrainIsolatesPoisonPath (0.02s)2254--- PASS: TestDrainGivesUpWhenServerDown (0.02s)2255--- PASS: TestWorkerSkipsGCdPaths (0.04s)2256--- PASS: TestWorkerUploadsAndRemoves (0.04s)2257--- PASS: TestWorkerPrunesClosureDeps (0.04s)2258--- PASS: TestQueueConcurrentWriters (0.24s)2259--- PASS: TestQueueRemoveLargeClosure (0.33s)22602026/08/27 09:37:40 INFO Uploading batch count=122612026/08/27 09:37:40 INFO Uploading batch count=122622026/08/27 09:37:40 INFO Uploading batch count=122632026/08/27 09:37:40 ERROR Upload failed error="upload failed" count=122642026/08/27 09:37:40 INFO Uploading batch count=122652026/08/27 09:37:40 ERROR Upload failed error="upload failed" count=122662026/08/27 09:37:40 INFO Uploading batch count=122672026/08/27 09:37:40 ERROR Upload failed error="upload failed" count=122682026/08/27 09:37:40 INFO Uploading batch count=122692026/08/27 09:37:40 ERROR Upload failed error="upload failed" count=122702026/08/27 09:37:40 ERROR Drain finished with paths left in queue remaining=12271--- PASS: TestRunNotBlockedByPoisonHead (1.03s)2272PASS