this derivation will be built: /nix/store/rlqx83yh2793iy613vb943df26n5x616-niks3-go-unit-tests.drv Failed to find a machine for remote build! derivation: rlqx83yh2793iy613vb943df26n5x616-niks3-go-unit-tests.drv required (system, features): (aarch64-linux, []) 1 available machines: (systems, maxjobs, supportedFeatures, mandatoryFeatures) ([aarch64-darwin, x86_64-darwin], 8, [big-parallel, recursive-nix], []) building '/nix/store/rlqx83yh2793iy613vb943df26n5x616-niks3-go-unit-tests.drv' niks3-go-unit-tests> tribuchet: building on eliza niks3-go-unit-tests> Running client tests... niks3-go-unit-tests> === RUN TestDoServerRequestAttachesToken niks3-go-unit-tests> === PAUSE TestDoServerRequestAttachesToken niks3-go-unit-tests> === RUN TestCaseHackSuffix niks3-go-unit-tests> === PAUSE TestCaseHackSuffix niks3-go-unit-tests> === RUN TestFilterOversizedClosures niks3-go-unit-tests> === PAUSE TestFilterOversizedClosures niks3-go-unit-tests> === RUN TestPartSizeForNAR niks3-go-unit-tests> === PAUSE TestPartSizeForNAR niks3-go-unit-tests> === RUN TestUploadMultipart_SupersededByPeer niks3-go-unit-tests> === PAUSE TestUploadMultipart_SupersededByPeer niks3-go-unit-tests> === RUN TestDumpPathMatchesNix niks3-go-unit-tests> === PAUSE TestDumpPathMatchesNix niks3-go-unit-tests> === RUN TestDumpPathSingleFile niks3-go-unit-tests> === PAUSE TestDumpPathSingleFile niks3-go-unit-tests> === RUN TestDumpPathWriterError niks3-go-unit-tests> === PAUSE TestDumpPathWriterError niks3-go-unit-tests> === RUN TestEncodeNixBase32 niks3-go-unit-tests> === PAUSE TestEncodeNixBase32 niks3-go-unit-tests> === RUN TestEncodeNixBase32WithRealHash niks3-go-unit-tests> === PAUSE TestEncodeNixBase32WithRealHash niks3-go-unit-tests> === RUN TestConvertHashToNix32 niks3-go-unit-tests> === PAUSE TestConvertHashToNix32 niks3-go-unit-tests> === RUN TestGetStorePathHash niks3-go-unit-tests> === PAUSE TestGetStorePathHash niks3-go-unit-tests> === RUN TestPathInfoHashCompatibility niks3-go-unit-tests> === PAUSE TestPathInfoHashCompatibility niks3-go-unit-tests> === RUN TestParsePathInfoJSON niks3-go-unit-tests> === PAUSE TestParsePathInfoJSON niks3-go-unit-tests> === RUN TestParsePathInfoJSONMultiplePaths niks3-go-unit-tests> === PAUSE TestParsePathInfoJSONMultiplePaths niks3-go-unit-tests> === RUN TestPathInfoCACompatibility niks3-go-unit-tests> === PAUSE TestPathInfoCACompatibility niks3-go-unit-tests> === RUN TestRateLimiterFeedback niks3-go-unit-tests> === PAUSE TestRateLimiterFeedback niks3-go-unit-tests> === RUN TestRateLimiterFeedback_400DoesNotCountAsSuccess niks3-go-unit-tests> === PAUSE TestRateLimiterFeedback_400DoesNotCountAsSuccess niks3-go-unit-tests> === RUN TestResolveStorePath niks3-go-unit-tests> === PAUSE TestResolveStorePath niks3-go-unit-tests> === RUN TestDoWithRetry_BodyReplayedViaGetBody niks3-go-unit-tests> === PAUSE TestDoWithRetry_BodyReplayedViaGetBody niks3-go-unit-tests> === RUN TestShellSplit niks3-go-unit-tests> === PAUSE TestShellSplit niks3-go-unit-tests> === RUN TestShellSplitErrors niks3-go-unit-tests> === PAUSE TestShellSplitErrors niks3-go-unit-tests> === RUN TestSetClientTLS niks3-go-unit-tests> === PAUSE TestSetClientTLS niks3-go-unit-tests> === RUN TestSetClientTLSDoesNotMutateDefaultTransport niks3-go-unit-tests> === PAUSE TestSetClientTLSDoesNotMutateDefaultTransport niks3-go-unit-tests> === RUN TestSetClientTLSErrors niks3-go-unit-tests> === PAUSE TestSetClientTLSErrors niks3-go-unit-tests> === RUN TestStaticToken niks3-go-unit-tests> === PAUSE TestStaticToken niks3-go-unit-tests> === RUN TestFileTokenReadsAndCaches niks3-go-unit-tests> === PAUSE TestFileTokenReadsAndCaches niks3-go-unit-tests> === RUN TestFileTokenMissing niks3-go-unit-tests> === PAUSE TestFileTokenMissing niks3-go-unit-tests> === RUN TestFileTokenEmpty niks3-go-unit-tests> === PAUSE TestFileTokenEmpty niks3-go-unit-tests> === RUN TestScriptTokenNoExpiryRerunsEveryCall niks3-go-unit-tests> === PAUSE TestScriptTokenNoExpiryRerunsEveryCall niks3-go-unit-tests> === RUN TestScriptTokenCachesUntilRefresh niks3-go-unit-tests> === PAUSE TestScriptTokenCachesUntilRefresh niks3-go-unit-tests> === RUN TestScriptTokenEmptyToken niks3-go-unit-tests> === PAUSE TestScriptTokenEmptyToken niks3-go-unit-tests> === RUN TestScriptTokenBadJSON niks3-go-unit-tests> === PAUSE TestScriptTokenBadJSON niks3-go-unit-tests> === RUN TestScriptTokenScriptFails niks3-go-unit-tests> === PAUSE TestScriptTokenScriptFails niks3-go-unit-tests> === RUN TestScriptTokenEmptyCommand niks3-go-unit-tests> === PAUSE TestScriptTokenEmptyCommand niks3-go-unit-tests> === CONT TestDoServerRequestAttachesToken niks3-go-unit-tests> === CONT TestFileTokenEmpty niks3-go-unit-tests> === CONT TestFileTokenMissing niks3-go-unit-tests> === CONT TestGetStorePathHash niks3-go-unit-tests> === RUN TestGetStorePathHash/valid_store_path niks3-go-unit-tests> === CONT TestDumpPathMatchesNix niks3-go-unit-tests> === PAUSE TestGetStorePathHash/valid_store_path niks3-go-unit-tests> === CONT TestEncodeNixBase32WithRealHash niks3-go-unit-tests> --- PASS: TestEncodeNixBase32WithRealHash (0.00s) niks3-go-unit-tests> === CONT TestDumpPathSingleFile niks3-go-unit-tests> === RUN TestGetStorePathHash/basename_without_hyphen_should_error niks3-go-unit-tests> === PAUSE TestGetStorePathHash/basename_without_hyphen_should_error niks3-go-unit-tests> --- PASS: TestFileTokenEmpty (0.00s) niks3-go-unit-tests> === CONT TestSetClientTLSErrors niks3-go-unit-tests> === RUN TestGetStorePathHash/hash_with_invalid_characters_should_error niks3-go-unit-tests> === PAUSE TestGetStorePathHash/hash_with_invalid_characters_should_error niks3-go-unit-tests> === CONT TestFileTokenReadsAndCaches niks3-go-unit-tests> --- PASS: TestFileTokenMissing (0.00s) niks3-go-unit-tests> === RUN TestGetStorePathHash/hash_with_wrong_length_should_error niks3-go-unit-tests> === PAUSE TestGetStorePathHash/hash_with_wrong_length_should_error niks3-go-unit-tests> === CONT TestShellSplit niks3-go-unit-tests> --- PASS: TestShellSplit (0.00s) niks3-go-unit-tests> === CONT TestEncodeNixBase32 niks3-go-unit-tests> === RUN TestEncodeNixBase32/test_string_hash niks3-go-unit-tests> === CONT TestDoWithRetry_BodyReplayedViaGetBody niks3-go-unit-tests> --- PASS: TestFileTokenReadsAndCaches (0.00s) niks3-go-unit-tests> === CONT TestDumpPathWriterError niks3-go-unit-tests> === CONT TestResolveStorePath niks3-go-unit-tests> === CONT TestRateLimiterFeedback_400DoesNotCountAsSuccess niks3-go-unit-tests> === CONT TestRateLimiterFeedback niks3-go-unit-tests> === RUN TestRateLimiterFeedback/429_enables_limiter niks3-go-unit-tests> === PAUSE TestRateLimiterFeedback/429_enables_limiter niks3-go-unit-tests> === RUN TestRateLimiterFeedback/503_enables_limiter niks3-go-unit-tests> === PAUSE TestRateLimiterFeedback/503_enables_limiter niks3-go-unit-tests> 2026/08/27 09:15:55 WARN Rate limiter enabled after throttle name=server-test rate=5 niks3-go-unit-tests> === CONT TestPathInfoCACompatibility niks3-go-unit-tests> === RUN TestPathInfoCACompatibility/null_ca_field niks3-go-unit-tests> === PAUSE TestPathInfoCACompatibility/null_ca_field niks3-go-unit-tests> === CONT TestParsePathInfoJSONMultiplePaths niks3-go-unit-tests> === RUN TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths niks3-go-unit-tests> === PAUSE TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths niks3-go-unit-tests> === RUN TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths niks3-go-unit-tests> === CONT TestParsePathInfoJSON niks3-go-unit-tests> === CONT TestPathInfoHashCompatibility niks3-go-unit-tests> === CONT TestScriptTokenNoExpiryRerunsEveryCall niks3-go-unit-tests> === CONT TestScriptTokenCachesUntilRefresh niks3-go-unit-tests> === CONT TestScriptTokenScriptFails niks3-go-unit-tests> === CONT TestScriptTokenEmptyCommand niks3-go-unit-tests> === CONT TestPartSizeForNAR niks3-go-unit-tests> === CONT TestUploadMultipart_SupersededByPeer niks3-go-unit-tests> === CONT TestShellSplitErrors niks3-go-unit-tests> === CONT TestScriptTokenBadJSON niks3-go-unit-tests> === CONT TestConvertHashToNix32 niks3-go-unit-tests> === PAUSE TestEncodeNixBase32/test_string_hash niks3-go-unit-tests> === CONT TestStaticToken niks3-go-unit-tests> === RUN TestRateLimiterFeedback/200_does_not_enable_limiter niks3-go-unit-tests> --- PASS: TestResolveStorePath (0.00s) niks3-go-unit-tests> === CONT TestFilterOversizedClosures niks3-go-unit-tests> === RUN TestPathInfoCACompatibility/old_string_format_-_text niks3-go-unit-tests> === PAUSE TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths niks3-go-unit-tests> --- PASS: TestScriptTokenEmptyCommand (0.00s) niks3-go-unit-tests> === RUN TestParsePathInfoJSON/Nix_format niks3-go-unit-tests> === RUN TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI) niks3-go-unit-tests> === RUN TestUploadMultipart_SupersededByPeer/exists niks3-go-unit-tests> === CONT TestScriptTokenEmptyToken niks3-go-unit-tests> === CONT TestSetClientTLS niks3-go-unit-tests> === PAUSE TestRateLimiterFeedback/200_does_not_enable_limiter niks3-go-unit-tests> === PAUSE TestParsePathInfoJSON/Nix_format niks3-go-unit-tests> 2026/08/27 09:15:55 WARN Rate limiter enabled after throttle name=server-test rate=5 niks3-go-unit-tests> === RUN TestSetClientTLSErrors/missing_cert_file niks3-go-unit-tests> === CONT TestSetClientTLSDoesNotMutateDefaultTransport niks3-go-unit-tests> === RUN TestPartSizeForNAR/zero_stays_at_minimum niks3-go-unit-tests> --- PASS: TestShellSplitErrors (0.00s) niks3-go-unit-tests> === RUN TestFilterOversizedClosures/no_limit_keeps_everything niks3-go-unit-tests> === CONT TestCaseHackSuffix niks3-go-unit-tests> 2026/08/27 09:15:55 WARN Request returned retryable status, retrying attempt=1 max_attempts=6 backoff=0s status=503 url=http://127.0.0.1:37121 niks3-go-unit-tests> === RUN TestConvertHashToNix32/SRI_format_to_Nix32 niks3-go-unit-tests> === PAUSE TestPathInfoCACompatibility/old_string_format_-_text niks3-go-unit-tests> === PAUSE TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI) niks3-go-unit-tests> === RUN TestPathInfoHashCompatibility/old_string_format_with_colon niks3-go-unit-tests> === RUN TestEncodeNixBase32/empty_input niks3-go-unit-tests> === RUN TestRateLimiterFeedback/400_does_not_enable_limiter niks3-go-unit-tests> --- PASS: TestStaticToken (0.00s) niks3-go-unit-tests> === RUN TestParsePathInfoJSON/Lix_format niks3-go-unit-tests> === PAUSE TestParsePathInfoJSON/Lix_format niks3-go-unit-tests> --- PASS: TestDoServerRequestAttachesToken (0.02s) niks3-go-unit-tests> 2026/08/27 09:15:55 WARN Rate limiter backed off name=server-test rate=5 niks3-go-unit-tests> === CONT TestGetStorePathHash/valid_store_path niks3-go-unit-tests> 2026/08/27 09:15:55 WARN Request returned retryable status, retrying attempt=2 max_attempts=6 backoff=0s status=503 url=http://127.0.0.1:37121 niks3-go-unit-tests> === PAUSE TestSetClientTLSErrors/missing_cert_file niks3-go-unit-tests> === PAUSE TestPartSizeForNAR/zero_stays_at_minimum niks3-go-unit-tests> === RUN TestPartSizeForNAR/small_stays_at_minimum niks3-go-unit-tests> === PAUSE TestPartSizeForNAR/small_stays_at_minimum niks3-go-unit-tests> === PAUSE TestFilterOversizedClosures/no_limit_keeps_everything niks3-go-unit-tests> === PAUSE TestConvertHashToNix32/SRI_format_to_Nix32 niks3-go-unit-tests> === RUN TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped niks3-go-unit-tests> === PAUSE TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped niks3-go-unit-tests> === RUN TestConvertHashToNix32/already_Nix32_format niks3-go-unit-tests> === PAUSE TestConvertHashToNix32/already_Nix32_format niks3-go-unit-tests> === PAUSE TestPathInfoHashCompatibility/old_string_format_with_colon niks3-go-unit-tests> === PAUSE TestEncodeNixBase32/empty_input niks3-go-unit-tests> === PAUSE TestRateLimiterFeedback/400_does_not_enable_limiter niks3-go-unit-tests> === RUN TestConvertHashToNix32/invalid_format niks3-go-unit-tests> === CONT TestEncodeNixBase32/test_string_hash niks3-go-unit-tests> === PAUSE TestConvertHashToNix32/invalid_format niks3-go-unit-tests> === RUN TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI niks3-go-unit-tests> === PAUSE TestUploadMultipart_SupersededByPeer/exists niks3-go-unit-tests> === CONT TestGetStorePathHash/basename_without_hyphen_should_error niks3-go-unit-tests> === CONT TestRateLimiterFeedback/400_does_not_enable_limiter niks3-go-unit-tests> === RUN TestSetClientTLSErrors/missing_key_file niks3-go-unit-tests> === RUN TestPartSizeForNAR/80_GiB_fits_at_minimum niks3-go-unit-tests> --- PASS: TestDoWithRetry_BodyReplayedViaGetBody (0.02s) niks3-go-unit-tests> === CONT TestGetStorePathHash/hash_with_invalid_characters_should_error niks3-go-unit-tests> === RUN TestPathInfoCACompatibility/old_string_format_-_fixed_recursive niks3-go-unit-tests> === RUN TestFilterOversizedClosures/all_closures_skipped niks3-go-unit-tests> === PAUSE TestPartSizeForNAR/80_GiB_fits_at_minimum niks3-go-unit-tests> === CONT TestConvertHashToNix32/invalid_format niks3-go-unit-tests> === RUN TestPartSizeForNAR/115_GiB_needs_larger_parts niks3-go-unit-tests> === PAUSE TestPartSizeForNAR/115_GiB_needs_larger_parts niks3-go-unit-tests> === CONT TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths niks3-go-unit-tests> === RUN TestParsePathInfoJSON/empty_input niks3-go-unit-tests> === PAUSE TestParsePathInfoJSON/empty_input niks3-go-unit-tests> === RUN TestParsePathInfoJSON/whitespace_only niks3-go-unit-tests> === CONT TestEncodeNixBase32/empty_input niks3-go-unit-tests> === CONT TestRateLimiterFeedback/429_enables_limiter niks3-go-unit-tests> === RUN TestSetClientTLS/rejects_connection_without_client_cert niks3-go-unit-tests> === CONT TestRateLimiterFeedback/200_does_not_enable_limiter niks3-go-unit-tests> === PAUSE TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI niks3-go-unit-tests> === RUN TestPathInfoHashCompatibility/new_structured_format_with_sha512 niks3-go-unit-tests> === RUN TestUploadMultipart_SupersededByPeer/missing niks3-go-unit-tests> === CONT TestConvertHashToNix32/SRI_format_to_Nix32 niks3-go-unit-tests> === CONT TestRateLimiterFeedback/503_enables_limiter niks3-go-unit-tests> === PAUSE TestSetClientTLSErrors/missing_key_file niks3-go-unit-tests> --- PASS: TestScriptTokenEmptyToken (0.01s) niks3-go-unit-tests> === PAUSE TestFilterOversizedClosures/all_closures_skipped niks3-go-unit-tests> === CONT TestGetStorePathHash/hash_with_wrong_length_should_error niks3-go-unit-tests> === CONT TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths niks3-go-unit-tests> === CONT TestConvertHashToNix32/already_Nix32_format niks3-go-unit-tests> === PAUSE TestPathInfoCACompatibility/old_string_format_-_fixed_recursive niks3-go-unit-tests> 2026/08/27 09:15:55 WARN Rate limiter enabled after throttle name=server-test rate=5 niks3-go-unit-tests> 2026/08/27 09:15:55 WARN Request returned retryable status, retrying attempt=1 max_attempts=2 backoff=0s status=429 url=http://127.0.0.1:42637 niks3-go-unit-tests> === RUN TestPartSizeForNAR/1_TiB niks3-go-unit-tests> 2026/08/27 09:15:55 WARN Rate limiter backed off name=server-test rate=5 niks3-go-unit-tests> === PAUSE TestParsePathInfoJSON/whitespace_only niks3-go-unit-tests> === RUN TestParsePathInfoJSON/invalid_JSON niks3-go-unit-tests> === PAUSE TestSetClientTLS/rejects_connection_without_client_cert niks3-go-unit-tests> === PAUSE TestPathInfoHashCompatibility/new_structured_format_with_sha512 niks3-go-unit-tests> === PAUSE TestUploadMultipart_SupersededByPeer/missing niks3-go-unit-tests> --- PASS: TestScriptTokenBadJSON (0.01s) niks3-go-unit-tests> === CONT TestFilterOversizedClosures/all_closures_skipped niks3-go-unit-tests> === RUN TestSetClientTLSErrors/missing_ca_file niks3-go-unit-tests> === CONT TestUploadMultipart_SupersededByPeer/missing niks3-go-unit-tests> 2026/08/27 09:15:55 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb-vm-image nar_size=5000 max_nar_size=50 niks3-go-unit-tests> 2026/08/27 09:15:55 WARN Rate limiter enabled after throttle name=server-test rate=5 niks3-go-unit-tests> 2026/08/27 09:15:55 WARN Request returned retryable status, retrying attempt=1 max_attempts=2 backoff=0s status=503 url=http://127.0.0.1:45821 niks3-go-unit-tests> === CONT TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped niks3-go-unit-tests> 2026/08/27 09:15:55 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb-vm-image nar_size=5000 max_nar_size=2000 niks3-go-unit-tests> === RUN TestPathInfoCACompatibility/new_structured_format_-_text niks3-go-unit-tests> === PAUSE TestPathInfoCACompatibility/new_structured_format_-_text niks3-go-unit-tests> 2026/08/27 09:15:55 WARN Rate limiter backed off name=server-test rate=5 niks3-go-unit-tests> === PAUSE TestPartSizeForNAR/1_TiB niks3-go-unit-tests> === RUN TestPartSizeForNAR/5_TiB_S3_max_object niks3-go-unit-tests> === PAUSE TestPartSizeForNAR/5_TiB_S3_max_object niks3-go-unit-tests> === RUN TestPartSizeForNAR/capped_at_5_GiB niks3-go-unit-tests> === CONT TestUploadMultipart_SupersededByPeer/exists niks3-go-unit-tests> === CONT TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI) niks3-go-unit-tests> === CONT TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI niks3-go-unit-tests> === CONT TestPathInfoHashCompatibility/new_structured_format_with_sha512 niks3-go-unit-tests> === CONT TestPathInfoHashCompatibility/old_string_format_with_colon niks3-go-unit-tests> === PAUSE TestSetClientTLSErrors/missing_ca_file niks3-go-unit-tests> --- PASS: TestScriptTokenNoExpiryRerunsEveryCall (0.01s) niks3-go-unit-tests> --- PASS: TestScriptTokenScriptFails (0.01s) niks3-go-unit-tests> --- PASS: TestSetClientTLSDoesNotMutateDefaultTransport (0.00s) niks3-go-unit-tests> === PAUSE TestParsePathInfoJSON/invalid_JSON niks3-go-unit-tests> === RUN TestSetClientTLS/succeeds_with_client_cert_and_CA niks3-go-unit-tests> === CONT TestFilterOversizedClosures/no_limit_keeps_everything niks3-go-unit-tests> === CONT TestParsePathInfoJSON/whitespace_only niks3-go-unit-tests> === PAUSE TestPartSizeForNAR/capped_at_5_GiB niks3-go-unit-tests> --- PASS: TestEncodeNixBase32 (0.02s) niks3-go-unit-tests> --- PASS: TestEncodeNixBase32/test_string_hash (0.00s) niks3-go-unit-tests> --- PASS: TestEncodeNixBase32/empty_input (0.00s) niks3-go-unit-tests> === CONT TestPartSizeForNAR/zero_stays_at_minimum niks3-go-unit-tests> === CONT TestParsePathInfoJSON/invalid_JSON niks3-go-unit-tests> === CONT TestParsePathInfoJSON/empty_input niks3-go-unit-tests> === CONT TestParsePathInfoJSON/Nix_format niks3-go-unit-tests> === PAUSE TestSetClientTLS/succeeds_with_client_cert_and_CA niks3-go-unit-tests> === CONT TestParsePathInfoJSON/Lix_format niks3-go-unit-tests> === RUN TestSetClientTLS/preserves_debug_logging_transport niks3-go-unit-tests> === RUN TestPathInfoCACompatibility/new_structured_format_-_nar_method niks3-go-unit-tests> === CONT TestPartSizeForNAR/1_TiB niks3-go-unit-tests> === PAUSE TestPathInfoCACompatibility/new_structured_format_-_nar_method niks3-go-unit-tests> === CONT TestPathInfoCACompatibility/null_ca_field niks3-go-unit-tests> === CONT TestPathInfoCACompatibility/new_structured_format_-_text niks3-go-unit-tests> === CONT TestPartSizeForNAR/80_GiB_fits_at_minimum niks3-go-unit-tests> === CONT TestPathInfoCACompatibility/old_string_format_-_fixed_recursive niks3-go-unit-tests> === RUN TestSetClientTLSErrors/invalid_ca_file niks3-go-unit-tests> === PAUSE TestSetClientTLSErrors/invalid_ca_file niks3-go-unit-tests> === CONT TestPartSizeForNAR/capped_at_5_GiB niks3-go-unit-tests> === CONT TestPartSizeForNAR/5_TiB_S3_max_object niks3-go-unit-tests> === CONT TestPartSizeForNAR/115_GiB_needs_larger_parts niks3-go-unit-tests> === CONT TestPartSizeForNAR/small_stays_at_minimum niks3-go-unit-tests> --- PASS: TestGetStorePathHash (0.00s) niks3-go-unit-tests> --- PASS: TestGetStorePathHash/valid_store_path (0.00s) niks3-go-unit-tests> --- PASS: TestGetStorePathHash/basename_without_hyphen_should_error (0.00s) niks3-go-unit-tests> --- PASS: TestGetStorePathHash/hash_with_invalid_characters_should_error (0.00s) niks3-go-unit-tests> --- PASS: TestGetStorePathHash/hash_with_wrong_length_should_error (0.00s) niks3-go-unit-tests> === PAUSE TestSetClientTLS/preserves_debug_logging_transport niks3-go-unit-tests> === CONT TestSetClientTLS/rejects_connection_without_client_cert niks3-go-unit-tests> === CONT TestPathInfoCACompatibility/new_structured_format_-_nar_method niks3-go-unit-tests> === CONT TestPathInfoCACompatibility/old_string_format_-_text niks3-go-unit-tests> === CONT TestSetClientTLSErrors/invalid_ca_file niks3-go-unit-tests> === CONT TestSetClientTLSErrors/missing_cert_file niks3-go-unit-tests> === CONT TestSetClientTLSErrors/missing_key_file niks3-go-unit-tests> === CONT TestSetClientTLSErrors/missing_ca_file niks3-go-unit-tests> --- PASS: TestConvertHashToNix32 (0.01s) niks3-go-unit-tests> --- PASS: TestConvertHashToNix32/invalid_format (0.00s) niks3-go-unit-tests> --- PASS: TestConvertHashToNix32/SRI_format_to_Nix32 (0.00s) niks3-go-unit-tests> --- PASS: TestConvertHashToNix32/already_Nix32_format (0.00s) niks3-go-unit-tests> === CONT TestSetClientTLS/succeeds_with_client_cert_and_CA niks3-go-unit-tests> === CONT TestSetClientTLS/preserves_debug_logging_transport niks3-go-unit-tests> --- PASS: TestPathInfoHashCompatibility (0.02s) niks3-go-unit-tests> --- PASS: TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI) (0.00s) niks3-go-unit-tests> --- PASS: TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI (0.00s) niks3-go-unit-tests> --- PASS: TestPathInfoHashCompatibility/new_structured_format_with_sha512 (0.00s) niks3-go-unit-tests> --- PASS: TestPathInfoHashCompatibility/old_string_format_with_colon (0.00s) niks3-go-unit-tests> --- PASS: TestScriptTokenCachesUntilRefresh (0.02s) niks3-go-unit-tests> --- PASS: TestParsePathInfoJSONMultiplePaths (0.00s) niks3-go-unit-tests> --- PASS: TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths (0.00s) niks3-go-unit-tests> --- PASS: TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths (0.00s) niks3-go-unit-tests> --- PASS: TestFilterOversizedClosures (0.01s) niks3-go-unit-tests> --- PASS: TestFilterOversizedClosures/all_closures_skipped (0.00s) niks3-go-unit-tests> --- PASS: TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped (0.00s) niks3-go-unit-tests> --- PASS: TestFilterOversizedClosures/no_limit_keeps_everything (0.00s) niks3-go-unit-tests> --- PASS: TestParsePathInfoJSON (0.02s) niks3-go-unit-tests> --- PASS: TestParsePathInfoJSON/whitespace_only (0.00s) niks3-go-unit-tests> --- PASS: TestParsePathInfoJSON/invalid_JSON (0.00s) niks3-go-unit-tests> --- PASS: TestParsePathInfoJSON/empty_input (0.00s) niks3-go-unit-tests> --- PASS: TestParsePathInfoJSON/Nix_format (0.00s) niks3-go-unit-tests> --- PASS: TestParsePathInfoJSON/Lix_format (0.00s) niks3-go-unit-tests> --- PASS: TestPartSizeForNAR (0.02s) niks3-go-unit-tests> --- PASS: TestPartSizeForNAR/zero_stays_at_minimum (0.00s) niks3-go-unit-tests> --- PASS: TestPartSizeForNAR/1_TiB (0.00s) niks3-go-unit-tests> --- PASS: TestPartSizeForNAR/80_GiB_fits_at_minimum (0.00s) niks3-go-unit-tests> --- PASS: TestPartSizeForNAR/capped_at_5_GiB (0.00s) niks3-go-unit-tests> --- PASS: TestPartSizeForNAR/115_GiB_needs_larger_parts (0.00s) niks3-go-unit-tests> --- PASS: TestPartSizeForNAR/small_stays_at_minimum (0.00s) niks3-go-unit-tests> --- PASS: TestPartSizeForNAR/5_TiB_S3_max_object (0.00s) niks3-go-unit-tests> --- PASS: TestPathInfoCACompatibility (0.02s) niks3-go-unit-tests> --- PASS: TestPathInfoCACompatibility/null_ca_field (0.00s) niks3-go-unit-tests> --- PASS: TestPathInfoCACompatibility/new_structured_format_-_text (0.00s) niks3-go-unit-tests> --- PASS: TestPathInfoCACompatibility/old_string_format_-_fixed_recursive (0.00s) niks3-go-unit-tests> --- PASS: TestPathInfoCACompatibility/new_structured_format_-_nar_method (0.00s) niks3-go-unit-tests> --- PASS: TestPathInfoCACompatibility/old_string_format_-_text (0.00s) niks3-go-unit-tests> --- PASS: TestRateLimiterFeedback (0.02s) niks3-go-unit-tests> --- PASS: TestRateLimiterFeedback/400_does_not_enable_limiter (0.00s) niks3-go-unit-tests> --- PASS: TestRateLimiterFeedback/200_does_not_enable_limiter (0.00s) niks3-go-unit-tests> --- PASS: TestRateLimiterFeedback/429_enables_limiter (0.00s) niks3-go-unit-tests> --- PASS: TestRateLimiterFeedback/503_enables_limiter (0.00s) niks3-go-unit-tests> --- PASS: TestSetClientTLSErrors (0.02s) niks3-go-unit-tests> --- PASS: TestSetClientTLSErrors/missing_cert_file (0.00s) niks3-go-unit-tests> --- PASS: TestSetClientTLSErrors/missing_key_file (0.00s) niks3-go-unit-tests> --- PASS: TestSetClientTLSErrors/invalid_ca_file (0.00s) niks3-go-unit-tests> --- PASS: TestSetClientTLSErrors/missing_ca_file (0.00s) niks3-go-unit-tests> --- PASS: TestUploadMultipart_SupersededByPeer (0.01s) niks3-go-unit-tests> --- PASS: TestUploadMultipart_SupersededByPeer/exists (0.00s) niks3-go-unit-tests> --- PASS: TestUploadMultipart_SupersededByPeer/missing (0.01s) niks3-go-unit-tests> --- PASS: TestDumpPathSingleFile (0.03s) niks3-go-unit-tests> 2026/08/27 09:15:55 http: TLS handshake error from 127.0.0.1:59808: remote error: tls: bad certificate niks3-go-unit-tests> --- PASS: TestSetClientTLS (0.01s) niks3-go-unit-tests> --- PASS: TestSetClientTLS/succeeds_with_client_cert_and_CA (0.01s) niks3-go-unit-tests> --- PASS: TestSetClientTLS/preserves_debug_logging_transport (0.01s) niks3-go-unit-tests> --- PASS: TestSetClientTLS/rejects_connection_without_client_cert (0.01s) niks3-go-unit-tests> --- PASS: TestDumpPathWriterError (0.04s) niks3-go-unit-tests> --- PASS: TestCaseHackSuffix (0.03s) niks3-go-unit-tests> --- PASS: TestDumpPathMatchesNix (0.08s) niks3-go-unit-tests> --- PASS: TestRateLimiterFeedback_400DoesNotCountAsSuccess (1.00s) niks3-go-unit-tests> PASS niks3-go-unit-tests> Running server tests... niks3-go-unit-tests> The files belonging to this database system will be owned by user "nixbld". niks3-go-unit-tests> This user must also own the server process. niks3-go-unit-tests> niks3-go-unit-tests> The database cluster will be initialized with locale "C". niks3-go-unit-tests> The default database encoding has accordingly been set to "SQL_ASCII". niks3-go-unit-tests> The default text search configuration will be set to "english". niks3-go-unit-tests> niks3-go-unit-tests> Data page checksums are enabled. niks3-go-unit-tests> niks3-go-unit-tests> creating directory /build/postgres3650402394/data ... ok niks3-go-unit-tests> creating subdirectories ... ok niks3-go-unit-tests> selecting dynamic shared memory implementation ... posix niks3-go-unit-tests> selecting default "max_connections" ... 100 niks3-go-unit-tests> selecting default "shared_buffers" ... 128MB niks3-go-unit-tests> selecting default time zone ... UTC niks3-go-unit-tests> creating configuration files ... ok niks3-go-unit-tests> running bootstrap script ... ok niks3-go-unit-tests> performing post-bootstrap initialization ... ok niks3-go-unit-tests> syncing data to disk ... ok niks3-go-unit-tests> niks3-go-unit-tests> initdb: warning: enabling "trust" authentication for local connections niks3-go-unit-tests> initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb. niks3-go-unit-tests> niks3-go-unit-tests> Success. You can now start the database server using: niks3-go-unit-tests> niks3-go-unit-tests> pg_ctl -D /build/postgres3650402394/data -l logfile start niks3-go-unit-tests> niks3-go-unit-tests> /build/postgres3650402394:5432 - no response niks3-go-unit-tests> 2026-08-27 09:15:57.390 UTC [111] LOG: starting PostgreSQL 18.4 on aarch64-unknown-linux-gnu, compiled by clang version 21.1.8, 64-bit niks3-go-unit-tests> 2026-08-27 09:15:57.390 UTC [111] LOG: listening on Unix socket "/build/postgres3650402394/.s.PGSQL.5432" niks3-go-unit-tests> 2026-08-27 09:15:57.394 UTC [118] LOG: database system was shut down at 2026-08-27 09:15:57 UTC niks3-go-unit-tests> 2026-08-27 09:15:57.397 UTC [111] LOG: database system is ready to accept connections niks3-go-unit-tests> /build/postgres3650402394:5432 - accepting connections niks3-go-unit-tests> === RUN TestService_AuthMiddleware niks3-go-unit-tests> === PAUSE TestService_AuthMiddleware niks3-go-unit-tests> === RUN TestService_AuthMiddleware_MTLSProxyHeader niks3-go-unit-tests> === PAUSE TestService_AuthMiddleware_MTLSProxyHeader niks3-go-unit-tests> === RUN TestService_AuthMiddleware_MTLSBoundSubjects niks3-go-unit-tests> === PAUSE TestService_AuthMiddleware_MTLSBoundSubjects niks3-go-unit-tests> === RUN TestService_ReadAuthMiddleware niks3-go-unit-tests> === PAUSE TestService_ReadAuthMiddleware niks3-go-unit-tests> === RUN TestService_AuthMiddleware_OIDC niks3-go-unit-tests> === PAUSE TestService_AuthMiddleware_OIDC niks3-go-unit-tests> === RUN TestCacheConfigHandler niks3-go-unit-tests> === PAUSE TestCacheConfigHandler niks3-go-unit-tests> === RUN TestCacheStatsHandler niks3-go-unit-tests> === PAUSE TestCacheStatsHandler niks3-go-unit-tests> === RUN TestClientCADerivations niks3-go-unit-tests> === PAUSE TestClientCADerivations niks3-go-unit-tests> === RUN TestClientErrorHandling niks3-go-unit-tests> === PAUSE TestClientErrorHandling niks3-go-unit-tests> === RUN TestClientIntegration niks3-go-unit-tests> === PAUSE TestClientIntegration niks3-go-unit-tests> === RUN TestClientMultipleUploads niks3-go-unit-tests> === PAUSE TestClientMultipleUploads niks3-go-unit-tests> === RUN TestClientWithDependencies niks3-go-unit-tests> === PAUSE TestClientWithDependencies niks3-go-unit-tests> === RUN TestPinProtectsFromGC niks3-go-unit-tests> === PAUSE TestPinProtectsFromGC niks3-go-unit-tests> === RUN TestGCAdvisoryLockBlocksConcurrentRun niks3-go-unit-tests> 2026-08-27 09:16:03.095 UTC [521] ERROR: relation "goose_db_version" does not exist at character 36 niks3-go-unit-tests> 2026-08-27 09:16:03.095 UTC [521] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC niks3-go-unit-tests> 2026/08/27 09:16:03 OK 20241026095416_initial_model.sql (11.35ms) niks3-go-unit-tests> 2026/08/27 09:16:03 OK 20251210153512_drop_unused_gin_index.sql (1.26ms) niks3-go-unit-tests> 2026/08/27 09:16:03 OK 20251218171726_add_pins.sql (2.75ms) niks3-go-unit-tests> 2026/08/27 09:16:03 OK 20260628120000_add_object_size_and_stats.sql (2.6ms) niks3-go-unit-tests> 2026/08/27 09:16:03 goose: successfully migrated database to version: 20260628120000 niks3-go-unit-tests> 2026/08/27 09:16:03 OK 1_commit_pending_closure.sql (1.65ms) niks3-go-unit-tests> 2026/08/27 09:16:03 OK 2_object_stats_trigger.sql (757.53µs) niks3-go-unit-tests> 2026/08/27 09:16:03 goose: up to current file version: 2 niks3-go-unit-tests> --- PASS: TestGCAdvisoryLockBlocksConcurrentRun (1.43s) niks3-go-unit-tests> === RUN TestGCBugBareHashReferences niks3-go-unit-tests> === PAUSE TestGCBugBareHashReferences niks3-go-unit-tests> === RUN TestGCMetrics niks3-go-unit-tests> === PAUSE TestGCMetrics niks3-go-unit-tests> === RUN TestGCTaskStore_StartNew niks3-go-unit-tests> === PAUSE TestGCTaskStore_StartNew niks3-go-unit-tests> === RUN TestGCTaskStore_DeduplicateSameParams niks3-go-unit-tests> === PAUSE TestGCTaskStore_DeduplicateSameParams niks3-go-unit-tests> === RUN TestGCTaskStore_ConflictDifferentParams niks3-go-unit-tests> === PAUSE TestGCTaskStore_ConflictDifferentParams niks3-go-unit-tests> === RUN TestGCTaskStore_GetEmpty niks3-go-unit-tests> === PAUSE TestGCTaskStore_GetEmpty niks3-go-unit-tests> === RUN TestGCTaskStore_GetReturnsLatest niks3-go-unit-tests> === PAUSE TestGCTaskStore_GetReturnsLatest niks3-go-unit-tests> === RUN TestGCTaskStore_CompletedAllowsNewTask niks3-go-unit-tests> === PAUSE TestGCTaskStore_CompletedAllowsNewTask niks3-go-unit-tests> === RUN TestGCTaskStore_PhaseUpdates niks3-go-unit-tests> === PAUSE TestGCTaskStore_PhaseUpdates niks3-go-unit-tests> === RUN TestGCTaskStore_Fail niks3-go-unit-tests> === PAUSE TestGCTaskStore_Fail niks3-go-unit-tests> === RUN TestGracefulShutdownDrainsInflight niks3-go-unit-tests> === PAUSE TestGracefulShutdownDrainsInflight niks3-go-unit-tests> === RUN TestService_healthCheckHandler niks3-go-unit-tests> === PAUSE TestService_healthCheckHandler niks3-go-unit-tests> === RUN TestGenerateLandingPage niks3-go-unit-tests> === PAUSE TestGenerateLandingPage niks3-go-unit-tests> === RUN TestCacheConfigHandlerMaxNarSize niks3-go-unit-tests> === PAUSE TestCacheConfigHandlerMaxNarSize niks3-go-unit-tests> === RUN TestCreatePendingClosureRejectsOversizedNAR niks3-go-unit-tests> === PAUSE TestCreatePendingClosureRejectsOversizedNAR niks3-go-unit-tests> === RUN TestNARDeduplicationMetadataUploadBug niks3-go-unit-tests> === PAUSE TestNARDeduplicationMetadataUploadBug niks3-go-unit-tests> === RUN TestMetricsInventory niks3-go-unit-tests> === PAUSE TestMetricsInventory niks3-go-unit-tests> === RUN TestService_NativeMTLS niks3-go-unit-tests> === PAUSE TestService_NativeMTLS niks3-go-unit-tests> === RUN TestServerTLSConfig niks3-go-unit-tests> === PAUSE TestServerTLSConfig niks3-go-unit-tests> === RUN TestMultipartCleanup niks3-go-unit-tests> === PAUSE TestMultipartCleanup niks3-go-unit-tests> === RUN TestObjectStatsTrigger niks3-go-unit-tests> === PAUSE TestObjectStatsTrigger niks3-go-unit-tests> === RUN TestOrphanedObjectsGC niks3-go-unit-tests> === PAUSE TestOrphanedObjectsGC niks3-go-unit-tests> === RUN TestOrphanedObjectsGCStressTest niks3-go-unit-tests> === PAUSE TestOrphanedObjectsGCStressTest niks3-go-unit-tests> === RUN TestResurrectedObjectNotDeleted niks3-go-unit-tests> === PAUSE TestResurrectedObjectNotDeleted niks3-go-unit-tests> === RUN TestParseSingleRange niks3-go-unit-tests> === PAUSE TestParseSingleRange niks3-go-unit-tests> === RUN TestIsValidCachePath niks3-go-unit-tests> === PAUSE TestIsValidCachePath niks3-go-unit-tests> === RUN TestReadProxyNarinfo niks3-go-unit-tests> === PAUSE TestReadProxyNarinfo niks3-go-unit-tests> === RUN TestReadProxyNarinfoAlreadyDecompressed niks3-go-unit-tests> === PAUSE TestReadProxyNarinfoAlreadyDecompressed niks3-go-unit-tests> === RUN TestReadProxyNarStreaming niks3-go-unit-tests> === PAUSE TestReadProxyNarStreaming niks3-go-unit-tests> === RUN TestReadProxy404 niks3-go-unit-tests> === PAUSE TestReadProxy404 niks3-go-unit-tests> === RUN TestReadProxyInvalidPath niks3-go-unit-tests> === PAUSE TestReadProxyInvalidPath niks3-go-unit-tests> === RUN TestReadProxyHead niks3-go-unit-tests> === PAUSE TestReadProxyHead niks3-go-unit-tests> === RUN TestReadProxyConditionalGet niks3-go-unit-tests> === PAUSE TestReadProxyConditionalGet niks3-go-unit-tests> === RUN TestReadProxyRootRedirectsToIndexHTML niks3-go-unit-tests> === PAUSE TestReadProxyRootRedirectsToIndexHTML niks3-go-unit-tests> === RUN TestReadProxyDisabled niks3-go-unit-tests> === PAUSE TestReadProxyDisabled niks3-go-unit-tests> === RUN TestReadProxyRangeRequest niks3-go-unit-tests> === PAUSE TestReadProxyRangeRequest niks3-go-unit-tests> === RUN TestRedundantMultipartUpload niks3-go-unit-tests> === PAUSE TestRedundantMultipartUpload niks3-go-unit-tests> === RUN TestCompleteMultipartUpload_ErrorButObjectExists niks3-go-unit-tests> === PAUSE TestCompleteMultipartUpload_ErrorButObjectExists niks3-go-unit-tests> === RUN TestCompletedNarNotReofferedAcrossClosures niks3-go-unit-tests> === PAUSE TestCompletedNarNotReofferedAcrossClosures niks3-go-unit-tests> === RUN TestPresignedUploadRegisteredBeforeCommit niks3-go-unit-tests> === PAUSE TestPresignedUploadRegisteredBeforeCommit niks3-go-unit-tests> === RUN TestService_Rustfstest niks3-go-unit-tests> === PAUSE TestService_Rustfstest niks3-go-unit-tests> === RUN TestParseSize niks3-go-unit-tests> === PAUSE TestParseSize niks3-go-unit-tests> === RUN TestSkippedUploadsHandler niks3-go-unit-tests> === PAUSE TestSkippedUploadsHandler niks3-go-unit-tests> === RUN TestSystemdListenerNotActivated niks3-go-unit-tests> --- PASS: TestSystemdListenerNotActivated (0.00s) niks3-go-unit-tests> === RUN TestWatchdogBeatsWhenHealthy niks3-go-unit-tests> --- PASS: TestWatchdogBeatsWhenHealthy (0.02s) niks3-go-unit-tests> === RUN TestWatchdogSkipsWhenUnhealthy niks3-go-unit-tests> 2026/08/27 09:16:04 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded" niks3-go-unit-tests> 2026/08/27 09:16:04 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded" niks3-go-unit-tests> 2026/08/27 09:16:04 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded" niks3-go-unit-tests> 2026/08/27 09:16:04 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded" niks3-go-unit-tests> 2026/08/27 09:16:04 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded" niks3-go-unit-tests> 2026/08/27 09:16:04 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded" niks3-go-unit-tests> 2026/08/27 09:16:04 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded" niks3-go-unit-tests> 2026/08/27 09:16:04 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded" niks3-go-unit-tests> 2026/08/27 09:16:04 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded" niks3-go-unit-tests> 2026/08/27 09:16:04 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded" niks3-go-unit-tests> --- PASS: TestWatchdogSkipsWhenUnhealthy (0.20s) niks3-go-unit-tests> === RUN TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle niks3-go-unit-tests> === PAUSE TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle niks3-go-unit-tests> === RUN TestProxyWriteTimeout niks3-go-unit-tests> === PAUSE TestProxyWriteTimeout niks3-go-unit-tests> === RUN TestIsValidUploadKey niks3-go-unit-tests> === PAUSE TestIsValidUploadKey niks3-go-unit-tests> === RUN TestUploadHandlersRejectInvalidKeys niks3-go-unit-tests> === PAUSE TestUploadHandlersRejectInvalidKeys niks3-go-unit-tests> === RUN TestUploadHandlersRejectOversizedBody niks3-go-unit-tests> === PAUSE TestUploadHandlersRejectOversizedBody niks3-go-unit-tests> === RUN TestService_cleanupPendingClosuresHandler niks3-go-unit-tests> === PAUSE TestService_cleanupPendingClosuresHandler niks3-go-unit-tests> === RUN TestService_createPendingClosureHandler niks3-go-unit-tests> === PAUSE TestService_createPendingClosureHandler niks3-go-unit-tests> === RUN TestService_verifyS3Integrity niks3-go-unit-tests> === PAUSE TestService_verifyS3Integrity niks3-go-unit-tests> === RUN TestCompleteMultipartUnregistered niks3-go-unit-tests> === PAUSE TestCompleteMultipartUnregistered niks3-go-unit-tests> === RUN TestCreatePendingClosure_SmallNARUsesSimplePUT niks3-go-unit-tests> === PAUSE TestCreatePendingClosure_SmallNARUsesSimplePUT niks3-go-unit-tests> === CONT TestCompleteMultipartUnregistered niks3-go-unit-tests> === CONT TestIsValidUploadKey niks3-go-unit-tests> === CONT TestReadProxyInvalidPath niks3-go-unit-tests> === CONT TestCreatePendingClosure_SmallNARUsesSimplePUT niks3-go-unit-tests> === CONT TestReadProxyHead niks3-go-unit-tests> === CONT TestService_verifyS3Integrity niks3-go-unit-tests> === CONT TestService_createPendingClosureHandler niks3-go-unit-tests> === CONT TestService_cleanupPendingClosuresHandler niks3-go-unit-tests> === CONT TestUploadHandlersRejectOversizedBody niks3-go-unit-tests> === CONT TestUploadHandlersRejectInvalidKeys niks3-go-unit-tests> === RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info niks3-go-unit-tests> === PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info niks3-go-unit-tests> === RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal niks3-go-unit-tests> === PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal niks3-go-unit-tests> === RUN TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key niks3-go-unit-tests> === PAUSE TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key niks3-go-unit-tests> === RUN TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key niks3-go-unit-tests> === PAUSE TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key niks3-go-unit-tests> === CONT TestMetricsInventory niks3-go-unit-tests> === CONT TestProxyWriteTimeout niks3-go-unit-tests> === CONT TestReadProxyRootRedirectsToIndexHTML niks3-go-unit-tests> === CONT TestSkippedUploadsHandler niks3-go-unit-tests> === CONT TestParseSize niks3-go-unit-tests> === CONT TestService_Rustfstest niks3-go-unit-tests> === CONT TestPresignedUploadRegisteredBeforeCommit niks3-go-unit-tests> === CONT TestCompletedNarNotReofferedAcrossClosures niks3-go-unit-tests> === CONT TestCompleteMultipartUpload_ErrorButObjectExists niks3-go-unit-tests> === CONT TestRedundantMultipartUpload niks3-go-unit-tests> --- PASS: TestParseSize (0.00s) niks3-go-unit-tests> === CONT TestReadProxyConditionalGet niks3-go-unit-tests> === CONT TestReadProxyRangeRequest niks3-go-unit-tests> === CONT TestReadProxyDisabled niks3-go-unit-tests> 2026/08/27 09:16:04 INFO Client skipped oversized paths paths=3 nar_bytes=5000000000 niks3-go-unit-tests> === RUN TestIsValidUploadKey/narinfo niks3-go-unit-tests> === PAUSE TestIsValidUploadKey/narinfo niks3-go-unit-tests> === RUN TestIsValidUploadKey/nar_zst niks3-go-unit-tests> === PAUSE TestIsValidUploadKey/nar_zst niks3-go-unit-tests> === CONT TestService_AuthMiddleware niks3-go-unit-tests> === CONT TestGCTaskStore_StartNew niks3-go-unit-tests> === RUN TestProxyWriteTimeout/narinfo niks3-go-unit-tests> === CONT TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle niks3-go-unit-tests> === RUN TestIsValidUploadKey/nar_xz niks3-go-unit-tests> --- PASS: TestGCTaskStore_StartNew (0.00s) niks3-go-unit-tests> === CONT TestGCMetrics niks3-go-unit-tests> === PAUSE TestProxyWriteTimeout/narinfo niks3-go-unit-tests> === PAUSE TestIsValidUploadKey/nar_xz niks3-go-unit-tests> === RUN TestIsValidUploadKey/nar_plain niks3-go-unit-tests> === PAUSE TestIsValidUploadKey/nar_plain niks3-go-unit-tests> === RUN TestProxyWriteTimeout/1_GiB_nar niks3-go-unit-tests> === PAUSE TestProxyWriteTimeout/1_GiB_nar niks3-go-unit-tests> === RUN TestProxyWriteTimeout/10_GiB_nar niks3-go-unit-tests> === PAUSE TestProxyWriteTimeout/10_GiB_nar niks3-go-unit-tests> === RUN TestIsValidUploadKey/listing niks3-go-unit-tests> === PAUSE TestIsValidUploadKey/listing niks3-go-unit-tests> === RUN TestProxyWriteTimeout/unknown_size niks3-go-unit-tests> === PAUSE TestProxyWriteTimeout/unknown_size niks3-go-unit-tests> === RUN TestIsValidUploadKey/build_log niks3-go-unit-tests> === PAUSE TestIsValidUploadKey/build_log niks3-go-unit-tests> === RUN TestIsValidUploadKey/build_log_home-manager_file niks3-go-unit-tests> === PAUSE TestIsValidUploadKey/build_log_home-manager_file niks3-go-unit-tests> === CONT TestResurrectedObjectNotDeleted niks3-go-unit-tests> === RUN TestIsValidUploadKey/build_log_plus_in_name niks3-go-unit-tests> === PAUSE TestIsValidUploadKey/build_log_plus_in_name niks3-go-unit-tests> === RUN TestIsValidUploadKey/build_log_question_mark niks3-go-unit-tests> === PAUSE TestIsValidUploadKey/build_log_question_mark niks3-go-unit-tests> === RUN TestIsValidUploadKey/build_log_equals niks3-go-unit-tests> === PAUSE TestIsValidUploadKey/build_log_equals niks3-go-unit-tests> === RUN TestIsValidUploadKey/realisation niks3-go-unit-tests> === PAUSE TestIsValidUploadKey/realisation niks3-go-unit-tests> === RUN TestIsValidUploadKey/realisation_plus_in_output niks3-go-unit-tests> === PAUSE TestIsValidUploadKey/realisation_plus_in_output niks3-go-unit-tests> === RUN TestIsValidUploadKey/nix-cache-info niks3-go-unit-tests> === PAUSE TestIsValidUploadKey/nix-cache-info niks3-go-unit-tests> === RUN TestIsValidUploadKey/index.html niks3-go-unit-tests> === PAUSE TestIsValidUploadKey/index.html niks3-go-unit-tests> === RUN TestIsValidUploadKey/narinfo_key,_nar_type niks3-go-unit-tests> === PAUSE TestIsValidUploadKey/narinfo_key,_nar_type niks3-go-unit-tests> === RUN TestIsValidUploadKey/nar_key,_narinfo_type niks3-go-unit-tests> === PAUSE TestIsValidUploadKey/nar_key,_narinfo_type niks3-go-unit-tests> === RUN TestIsValidUploadKey/listing_key,_narinfo_type niks3-go-unit-tests> === PAUSE TestIsValidUploadKey/listing_key,_narinfo_type niks3-go-unit-tests> === RUN TestIsValidUploadKey/traversal niks3-go-unit-tests> === PAUSE TestIsValidUploadKey/traversal niks3-go-unit-tests> === RUN TestIsValidUploadKey/traversal_nar niks3-go-unit-tests> === PAUSE TestIsValidUploadKey/traversal_nar niks3-go-unit-tests> === RUN TestIsValidUploadKey/absolute niks3-go-unit-tests> === PAUSE TestIsValidUploadKey/absolute niks3-go-unit-tests> === RUN TestIsValidUploadKey/empty_key niks3-go-unit-tests> === PAUSE TestIsValidUploadKey/empty_key niks3-go-unit-tests> === RUN TestIsValidUploadKey/unknown_type niks3-go-unit-tests> === PAUSE TestIsValidUploadKey/unknown_type niks3-go-unit-tests> === CONT TestGCBugBareHashReferences niks3-go-unit-tests> === RUN TestUploadHandlersRejectOversizedBody/create_pending_closure niks3-go-unit-tests> === PAUSE TestUploadHandlersRejectOversizedBody/create_pending_closure niks3-go-unit-tests> === RUN TestUploadHandlersRejectOversizedBody/complete_multipart niks3-go-unit-tests> === PAUSE TestUploadHandlersRejectOversizedBody/complete_multipart niks3-go-unit-tests> === RUN TestUploadHandlersRejectOversizedBody/request_more_parts niks3-go-unit-tests> === PAUSE TestUploadHandlersRejectOversizedBody/request_more_parts niks3-go-unit-tests> === CONT TestReadProxy404 niks3-go-unit-tests> --- PASS: TestSkippedUploadsHandler (0.15s) niks3-go-unit-tests> === CONT TestPinProtectsFromGC niks3-go-unit-tests> 2026-08-27 09:16:04.827 UTC [596] ERROR: relation "goose_db_version" does not exist at character 36 niks3-go-unit-tests> 2026-08-27 09:16:04.827 UTC [596] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC niks3-go-unit-tests> 2026-08-27 09:16:04.844 UTC [601] ERROR: relation "goose_db_version" does not exist at character 36 niks3-go-unit-tests> 2026-08-27 09:16:04.844 UTC [601] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC niks3-go-unit-tests> 2026-08-27 09:16:04.844 UTC [600] ERROR: relation "goose_db_version" does not exist at character 36 niks3-go-unit-tests> 2026-08-27 09:16:04.844 UTC [600] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC niks3-go-unit-tests> 2026-08-27 09:16:04.845 UTC [602] ERROR: relation "goose_db_version" does not exist at character 36 niks3-go-unit-tests> 2026-08-27 09:16:04.845 UTC [602] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC niks3-go-unit-tests> 2026-08-27 09:16:04.878 UTC [603] ERROR: relation "goose_db_version" does not exist at character 36 niks3-go-unit-tests> 2026-08-27 09:16:04.878 UTC [603] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC niks3-go-unit-tests> 2026/08/27 09:16:04 OK 20241026095416_initial_model.sql (40.29ms) niks3-go-unit-tests> 2026/08/27 09:16:04 OK 20241026095416_initial_model.sql (50.69ms) niks3-go-unit-tests> 2026-08-27 09:16:04.905 UTC [604] ERROR: relation "goose_db_version" does not exist at character 36 niks3-go-unit-tests> 2026-08-27 09:16:04.905 UTC [604] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC niks3-go-unit-tests> 2026/08/27 09:16:04 OK 20251210153512_drop_unused_gin_index.sql (3.5ms) niks3-go-unit-tests> 2026/08/27 09:16:04 OK 20241026095416_initial_model.sql (43.17ms) niks3-go-unit-tests> 2026/08/27 09:16:04 OK 20251210153512_drop_unused_gin_index.sql (5.63ms) niks3-go-unit-tests> 2026/08/27 09:16:04 OK 20251218171726_add_pins.sql (6.25ms) niks3-go-unit-tests> 2026/08/27 09:16:04 OK 20241026095416_initial_model.sql (47.93ms) niks3-go-unit-tests> 2026/08/27 09:16:04 OK 20251210153512_drop_unused_gin_index.sql (5.2ms) niks3-go-unit-tests> 2026/08/27 09:16:04 OK 20251210153512_drop_unused_gin_index.sql (4.41ms) niks3-go-unit-tests> 2026/08/27 09:16:04 OK 20251218171726_add_pins.sql (8.88ms) niks3-go-unit-tests> 2026/08/27 09:16:04 OK 20260628120000_add_object_size_and_stats.sql (7.23ms) niks3-go-unit-tests> 2026/08/27 09:16:04 goose: successfully migrated database to version: 20260628120000 niks3-go-unit-tests> 2026/08/27 09:16:04 OK 20251218171726_add_pins.sql (8.37ms) niks3-go-unit-tests> 2026/08/27 09:16:04 OK 1_commit_pending_closure.sql (3.92ms) niks3-go-unit-tests> 2026-08-27 09:16:04.923 UTC [606] ERROR: relation "goose_db_version" does not exist at character 36 niks3-go-unit-tests> 2026-08-27 09:16:04.923 UTC [606] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC niks3-go-unit-tests> 2026-08-27 09:16:04.924 UTC [607] ERROR: relation "goose_db_version" does not exist at character 36 niks3-go-unit-tests> 2026-08-27 09:16:04.924 UTC [607] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC niks3-go-unit-tests> 2026/08/27 09:16:04 OK 20251218171726_add_pins.sql (8.94ms) niks3-go-unit-tests> 2026-08-27 09:16:04.926 UTC [605] ERROR: relation "goose_db_version" does not exist at character 36 niks3-go-unit-tests> 2026-08-27 09:16:04.926 UTC [605] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC niks3-go-unit-tests> 2026/08/27 09:16:04 OK 2_object_stats_trigger.sql (10.66ms) niks3-go-unit-tests> 2026/08/27 09:16:04 goose: up to current file version: 2 niks3-go-unit-tests> 2026/08/27 09:16:04 OK 20260628120000_add_object_size_and_stats.sql (18.63ms) niks3-go-unit-tests> 2026/08/27 09:16:04 goose: successfully migrated database to version: 20260628120000 niks3-go-unit-tests> 2026/08/27 09:16:04 OK 20241026095416_initial_model.sql (31.42ms) niks3-go-unit-tests> 2026/08/27 09:16:04 OK 20260628120000_add_object_size_and_stats.sql (15.8ms) niks3-go-unit-tests> 2026/08/27 09:16:04 goose: successfully migrated database to version: 20260628120000 niks3-go-unit-tests> 2026/08/27 09:16:04 OK 20251210153512_drop_unused_gin_index.sql (5.5ms) niks3-go-unit-tests> 2026/08/27 09:16:04 OK 1_commit_pending_closure.sql (7.73ms) niks3-go-unit-tests> 2026/08/27 09:16:04 OK 20260628120000_add_object_size_and_stats.sql (19.23ms) niks3-go-unit-tests> 2026/08/27 09:16:04 goose: successfully migrated database to version: 20260628120000 niks3-go-unit-tests> 2026/08/27 09:16:04 OK 20241026095416_initial_model.sql (29.65ms) niks3-go-unit-tests> 2026/08/27 09:16:04 OK 1_commit_pending_closure.sql (9.71ms) niks3-go-unit-tests> 2026/08/27 09:16:04 OK 2_object_stats_trigger.sql (4.38ms) niks3-go-unit-tests> 2026/08/27 09:16:04 goose: up to current file version: 2 niks3-go-unit-tests> 2026-08-27 09:16:04.951 UTC [608] ERROR: relation "goose_db_version" does not exist at character 36 niks3-go-unit-tests> 2026-08-27 09:16:04.951 UTC [608] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC niks3-go-unit-tests> 2026/08/27 09:16:04 OK 20251210153512_drop_unused_gin_index.sql (6.1ms) niks3-go-unit-tests> 2026/08/27 09:16:04 OK 1_commit_pending_closure.sql (8.36ms) niks3-go-unit-tests> 2026/08/27 09:16:04 OK 2_object_stats_trigger.sql (6.29ms) niks3-go-unit-tests> 2026/08/27 09:16:04 goose: up to current file version: 2 niks3-go-unit-tests> 2026/08/27 09:16:04 OK 20251218171726_add_pins.sql (10.67ms) niks3-go-unit-tests> 2026-08-27 09:16:04.956 UTC [609] ERROR: relation "goose_db_version" does not exist at character 36 niks3-go-unit-tests> 2026-08-27 09:16:04.956 UTC [609] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC niks3-go-unit-tests> 2026/08/27 09:16:04 OK 2_object_stats_trigger.sql (6.55ms) niks3-go-unit-tests> 2026/08/27 09:16:04 goose: up to current file version: 2 niks3-go-unit-tests> 2026-08-27 09:16:04.961 UTC [610] ERROR: relation "goose_db_version" does not exist at character 36 niks3-go-unit-tests> 2026-08-27 09:16:04.961 UTC [610] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC niks3-go-unit-tests> 2026-08-27 09:16:04.962 UTC [611] ERROR: relation "goose_db_version" does not exist at character 36 niks3-go-unit-tests> 2026-08-27 09:16:04.962 UTC [611] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC niks3-go-unit-tests> 2026/08/27 09:16:04 OK 20251218171726_add_pins.sql (11.15ms) niks3-go-unit-tests> 2026/08/27 09:16:04 OK 20241026095416_initial_model.sql (21.07ms) niks3-go-unit-tests> 2026/08/27 09:16:04 OK 20260628120000_add_object_size_and_stats.sql (11.83ms) niks3-go-unit-tests> 2026/08/27 09:16:04 OK 20241026095416_initial_model.sql (18.66ms) niks3-go-unit-tests> 2026/08/27 09:16:04 goose: successfully migrated database to version: 20260628120000 niks3-go-unit-tests> 2026-08-27 09:16:04.971 UTC [612] ERROR: relation "goose_db_version" does not exist at character 36 niks3-go-unit-tests> 2026-08-27 09:16:04.971 UTC [612] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC niks3-go-unit-tests> 2026/08/27 09:16:04 INFO Received cleanup request method=DELETE path=/api/pending_closures niks3-go-unit-tests> 2026/08/27 09:16:04 INFO Aborted multipart uploads count=0 niks3-go-unit-tests> 2026/08/27 09:16:04 OK 1_commit_pending_closure.sql (15.01ms) niks3-go-unit-tests> 2026-08-27 09:16:04.980 UTC [613] ERROR: relation "goose_db_version" does not exist at character 36 niks3-go-unit-tests> 2026-08-27 09:16:04.980 UTC [613] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC niks3-go-unit-tests> 2026/08/27 09:16:04 OK 20241026095416_initial_model.sql (28.17ms) niks3-go-unit-tests> 2026/08/27 09:16:04 OK 20251210153512_drop_unused_gin_index.sql (18.17ms) niks3-go-unit-tests> 2026/08/27 09:16:04 OK 20251210153512_drop_unused_gin_index.sql (15.57ms) niks3-go-unit-tests> 2026/08/27 09:16:04 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> 2026-08-27 09:16:04.981 UTC [614] ERROR: relation "goose_db_version" does not exist at character 36 niks3-go-unit-tests> 2026-08-27 09:16:04.981 UTC [614] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC niks3-go-unit-tests> 2026/08/27 09:16:04 OK 20260628120000_add_object_size_and_stats.sql (19.19ms) niks3-go-unit-tests> 2026/08/27 09:16:04 goose: successfully migrated database to version: 20260628120000 niks3-go-unit-tests> 2026/08/27 09:16:04 OK 20241026095416_initial_model.sql (18.94ms) niks3-go-unit-tests> 2026/08/27 09:16:04 OK 20251210153512_drop_unused_gin_index.sql (7.53ms) niks3-go-unit-tests> 2026/08/27 09:16:04 OK 2_object_stats_trigger.sql (7.96ms) niks3-go-unit-tests> 2026/08/27 09:16:04 goose: up to current file version: 2 niks3-go-unit-tests> 2026/08/27 09:16:04 OK 20251218171726_add_pins.sql (8.67ms) niks3-go-unit-tests> 2026/08/27 09:16:04 OK 1_commit_pending_closure.sql (7.5ms) niks3-go-unit-tests> 2026/08/27 09:16:04 OK 20251218171726_add_pins.sql (8.71ms) niks3-go-unit-tests> 2026/08/27 09:16:04 OK 20251210153512_drop_unused_gin_index.sql (3.83ms) niks3-go-unit-tests> 2026-08-27 09:16:04.993 UTC [616] ERROR: relation "goose_db_version" does not exist at character 36 niks3-go-unit-tests> 2026-08-27 09:16:04.993 UTC [616] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC niks3-go-unit-tests> 2026-08-27 09:16:04.993 UTC [615] ERROR: relation "goose_db_version" does not exist at character 36 niks3-go-unit-tests> 2026-08-27 09:16:04.993 UTC [615] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC niks3-go-unit-tests> 2026-08-27 09:16:04.994 UTC [617] ERROR: relation "goose_db_version" does not exist at character 36 niks3-go-unit-tests> 2026-08-27 09:16:04.994 UTC [617] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC niks3-go-unit-tests> 2026/08/27 09:16:04 OK 2_object_stats_trigger.sql (5.46ms) niks3-go-unit-tests> 2026/08/27 09:16:04 goose: up to current file version: 2 niks3-go-unit-tests> 2026/08/27 09:16:04 OK 20241026095416_initial_model.sql (16.93ms) niks3-go-unit-tests> 2026/08/27 09:16:04 OK 20251218171726_add_pins.sql (6.97ms) niks3-go-unit-tests> 2026-08-27 09:16:04.996 UTC [618] ERROR: relation "goose_db_version" does not exist at character 36 niks3-go-unit-tests> 2026-08-27 09:16:04.996 UTC [618] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC niks3-go-unit-tests> 2026/08/27 09:16:04 OK 20251218171726_add_pins.sql (6.44ms) niks3-go-unit-tests> 2026-08-27 09:16:04.997 UTC [619] ERROR: relation "goose_db_version" does not exist at character 36 niks3-go-unit-tests> 2026-08-27 09:16:04.997 UTC [619] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC niks3-go-unit-tests> 2026/08/27 09:16:04 OK 20251210153512_drop_unused_gin_index.sql (3.94ms) niks3-go-unit-tests> 2026/08/27 09:16:04 OK 20241026095416_initial_model.sql (15.16ms) niks3-go-unit-tests> 2026/08/27 09:16:04 OK 20260628120000_add_object_size_and_stats.sql (9.28ms) niks3-go-unit-tests> 2026/08/27 09:16:04 goose: successfully migrated database to version: 20260628120000 niks3-go-unit-tests> 2026/08/27 09:16:04 OK 20260628120000_add_object_size_and_stats.sql (9.51ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: successfully migrated database to version: 20260628120000 niks3-go-unit-tests> 2026-08-27 09:16:05.000 UTC [620] ERROR: relation "goose_db_version" does not exist at character 36 niks3-go-unit-tests> 2026-08-27 09:16:05.000 UTC [620] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC niks3-go-unit-tests> 2026/08/27 09:16:05 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> 2026-08-27 09:16:05.001 UTC [621] ERROR: relation "goose_db_version" does not exist at character 36 niks3-go-unit-tests> 2026-08-27 09:16:05.001 UTC [621] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC niks3-go-unit-tests> 2026/08/27 09:16:05 INFO Received cleanup request method=DELETE path=/api/pending_closures niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20241026095416_initial_model.sql (18.09ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20260628120000_add_object_size_and_stats.sql (6.59ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: successfully migrated database to version: 20260628120000 niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20241026095416_initial_model.sql (19.85ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20251210153512_drop_unused_gin_index.sql (3.8ms) niks3-go-unit-tests> 2026-08-27 09:16:05.003 UTC [622] ERROR: relation "goose_db_version" does not exist at character 36 niks3-go-unit-tests> 2026-08-27 09:16:05.003 UTC [622] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20260628120000_add_object_size_and_stats.sql (10.35ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: successfully migrated database to version: 20260628120000 niks3-go-unit-tests> 2026/08/27 09:16:05 INFO Aborted multipart uploads count=1 niks3-go-unit-tests> 2026/08/27 09:16:05 OK 1_commit_pending_closure.sql (6.18ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20251218171726_add_pins.sql (7.81ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 1_commit_pending_closure.sql (7.4ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20251210153512_drop_unused_gin_index.sql (6.03ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 2_object_stats_trigger.sql (3.28ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: up to current file version: 2 niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20241026095416_initial_model.sql (14.94ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20241026095416_initial_model.sql (17.88ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 1_commit_pending_closure.sql (5.95ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20251210153512_drop_unused_gin_index.sql (7.39ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20251218171726_add_pins.sql (7.05ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 2_object_stats_trigger.sql (3.71ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: up to current file version: 2 niks3-go-unit-tests> 2026/08/27 09:16:05 OK 1_commit_pending_closure.sql (4.99ms) niks3-go-unit-tests> 2026/08/27 09:16:05 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete niks3-go-unit-tests> 2026/08/27 09:16:05 OK 2_object_stats_trigger.sql (2.11ms) niks3-go-unit-tests> 2026-08-27 09:16:05.012 UTC [596] ERROR: Closure does not exist: id=1 niks3-go-unit-tests> 2026-08-27 09:16:05.012 UTC [596] CONTEXT: PL/pgSQL function commit_pending_closure(bigint) line 16 at RAISE niks3-go-unit-tests> 2026-08-27 09:16:05.012 UTC [596] STATEMENT: -- name: CommitPendingClosure :exec niks3-go-unit-tests> SELECT commit_pending_closure($1::bigint) niks3-go-unit-tests> niks3-go-unit-tests> 2026/08/27 09:16:05 goose: up to current file version: 2 niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20251210153512_drop_unused_gin_index.sql (2.37ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 2_object_stats_trigger.sql (3.18ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: up to current file version: 2 niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20251210153512_drop_unused_gin_index.sql (4.31ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20260628120000_add_object_size_and_stats.sql (8.63ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: successfully migrated database to version: 20260628120000 niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20251218171726_add_pins.sql (8.16ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20251218171726_add_pins.sql (5.06ms) niks3-go-unit-tests> --- PASS: TestService_cleanupPendingClosuresHandler (0.35s) niks3-go-unit-tests> === CONT TestGCTaskStore_Fail niks3-go-unit-tests> --- PASS: TestGCTaskStore_Fail (0.00s) niks3-go-unit-tests> === CONT TestReadProxyNarStreaming niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20260628120000_add_object_size_and_stats.sql (6.75ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: successfully migrated database to version: 20260628120000 niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20251218171726_add_pins.sql (7.34ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20251218171726_add_pins.sql (5.39ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 1_commit_pending_closure.sql (3.37ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20260628120000_add_object_size_and_stats.sql (5.19ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: successfully migrated database to version: 20260628120000 niks3-go-unit-tests> 2026/08/27 09:16:05 OK 1_commit_pending_closure.sql (5.22ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20260628120000_add_object_size_and_stats.sql (6.46ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20241026095416_initial_model.sql (18.41ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: successfully migrated database to version: 20260628120000 niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20241026095416_initial_model.sql (18.49ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 2_object_stats_trigger.sql (2.69ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: up to current file version: 2 niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20241026095416_initial_model.sql (11.34ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 1_commit_pending_closure.sql (4.04ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20241026095416_initial_model.sql (14.13ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20260628120000_add_object_size_and_stats.sql (5.54ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: successfully migrated database to version: 20260628120000 niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20241026095416_initial_model.sql (14.22ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20241026095416_initial_model.sql (17.37ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20260628120000_add_object_size_and_stats.sql (5.51ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: successfully migrated database to version: 20260628120000 niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20241026095416_initial_model.sql (14.35ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20251210153512_drop_unused_gin_index.sql (2.93ms) niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.026059741Z","level":"ERROR","duration":"420.363µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(302)"} niks3-go-unit-tests> 2026/08/27 09:16:05 OK 2_object_stats_trigger.sql (4.39ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: up to current file version: 2 niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.026486624Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"47330122-0afd-49ac-86d8-2b75b2b49320","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket12/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(302)"} niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20241026095416_initial_model.sql (16.63ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 1_commit_pending_closure.sql (5.06ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20251210153512_drop_unused_gin_index.sql (5.13ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20251210153512_drop_unused_gin_index.sql (3.74ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 1_commit_pending_closure.sql (2.69ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20251218171726_add_pins.sql (2.74ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 2_object_stats_trigger.sql (3.01ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: up to current file version: 2 niks3-go-unit-tests> 2026/08/27 09:16:05 OK 1_commit_pending_closure.sql (2.97ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20251210153512_drop_unused_gin_index.sql (3.23ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 2_object_stats_trigger.sql (1.42ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: up to current file version: 2 niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20251210153512_drop_unused_gin_index.sql (1.56ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20251210153512_drop_unused_gin_index.sql (3.59ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20251210153512_drop_unused_gin_index.sql (4.17ms) niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.030088657Z","level":"ERROR","duration":"526.704µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(209)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.030165878Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"1dd07a09-d680-4723-a069-5b360138975b","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket13/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(209)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.03032558Z","level":"ERROR","duration":"774.488µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(308)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.03038124Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"c3a71436-4479-4e8e-a082-9c9d4d435c6a","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket14/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(308)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.031877754Z","level":"ERROR","duration":"519.185µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(302)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.031946794Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"3a939ee4-7291-4f11-93cb-f9d972970e37","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket15/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(302)"} niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20251210153512_drop_unused_gin_index.sql (6.22ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 2_object_stats_trigger.sql (4.97ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 2_object_stats_trigger.sql (5.37ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: up to current file version: 2 niks3-go-unit-tests> 2026/08/27 09:16:05 goose: up to current file version: 2 niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20251218171726_add_pins.sql (5.66ms) niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.035528367Z","level":"ERROR","duration":"482.645µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(293)"} niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20251218171726_add_pins.sql (5.67ms) niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.035599688Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"95dbdb25-9196-45bc-a0d8-45a3a13fc7af","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket16/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(293)"} niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20251218171726_add_pins.sql (5.71ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20260628120000_add_object_size_and_stats.sql (6.65ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: successfully migrated database to version: 20260628120000 niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20251218171726_add_pins.sql (6.28ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20251218171726_add_pins.sql (5.46ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20251218171726_add_pins.sql (7.35ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20251218171726_add_pins.sql (3.54ms) niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.036707798Z","level":"ERROR","duration":"344.903µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(314)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.036754098Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"47c6a1a8-0f09-4132-ad7a-10e18c18790b","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket17/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(314)"} niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20260628120000_add_object_size_and_stats.sql (4.29ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: successfully migrated database to version: 20260628120000 niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20260628120000_add_object_size_and_stats.sql (4.65ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: successfully migrated database to version: 20260628120000 niks3-go-unit-tests> 2026/08/27 09:16:05 OK 1_commit_pending_closure.sql (4.56ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20260628120000_add_object_size_and_stats.sql (4.79ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: successfully migrated database to version: 20260628120000 niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20260628120000_add_object_size_and_stats.sql (4.97ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: successfully migrated database to version: 20260628120000 niks3-go-unit-tests> 2026/08/27 09:16:05 OK 1_commit_pending_closure.sql (1.79ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20260628120000_add_object_size_and_stats.sql (7.72ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: successfully migrated database to version: 20260628120000 niks3-go-unit-tests> 2026/08/27 09:16:05 OK 1_commit_pending_closure.sql (5.41ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20260628120000_add_object_size_and_stats.sql (11.05ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: successfully migrated database to version: 20260628120000 niks3-go-unit-tests> 2026/08/27 09:16:05 OK 1_commit_pending_closure.sql (5.98ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20260628120000_add_object_size_and_stats.sql (11.11ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: successfully migrated database to version: 20260628120000 niks3-go-unit-tests> 2026/08/27 09:16:05 OK 2_object_stats_trigger.sql (6.49ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: up to current file version: 2 niks3-go-unit-tests> 2026/08/27 09:16:05 OK 1_commit_pending_closure.sql (3.76ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 1_commit_pending_closure.sql (6.94ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 2_object_stats_trigger.sql (5.47ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: up to current file version: 2 niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.049332333Z","level":"ERROR","duration":"423.684µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(293)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.049395514Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"a8a95f2c-aedc-4120-9035-5439892f2107","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket18/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(293)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.049551235Z","level":"ERROR","duration":"573.045µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(302)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.049627796Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"589a42d5-bb58-475e-bbaf-ca56336270c2","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket19/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(302)"} niks3-go-unit-tests> 2026/08/27 09:16:05 OK 2_object_stats_trigger.sql (3.31ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: up to current file version: 2 niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.052388701Z","level":"ERROR","duration":"424.504µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(278)"} niks3-go-unit-tests> 2026/08/27 09:16:05 OK 2_object_stats_trigger.sql (4.89ms) niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.052439002Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"d1825fe2-fe10-40ee-92b0-f20ba1aed6eb","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket20/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(278)"} niks3-go-unit-tests> 2026/08/27 09:16:05 goose: up to current file version: 2 niks3-go-unit-tests> 2026/08/27 09:16:05 OK 1_commit_pending_closure.sql (5.57ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 2_object_stats_trigger.sql (5.96ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: up to current file version: 2 niks3-go-unit-tests> 2026/08/27 09:16:05 OK 1_commit_pending_closure.sql (5.76ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 2_object_stats_trigger.sql (5.7ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: up to current file version: 2 niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.054611822Z","level":"ERROR","duration":"502.125µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(278)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.054668842Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"7661969a-72ec-4861-bf24-ac3687384c8a","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket22/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(278)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.054756563Z","level":"ERROR","duration":"490.785µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(302)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.054831444Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"3ae257c2-9894-4f19-9e69-7514481fd915","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket21/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(302)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.054950365Z","level":"ERROR","duration":"385.724µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(308)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.055002265Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"24537c0f-ff4d-4134-b952-45376bdb96dc","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket23/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(308)"} niks3-go-unit-tests> 2026/08/27 09:16:05 OK 2_object_stats_trigger.sql (2.77ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: up to current file version: 2 niks3-go-unit-tests> 2026/08/27 09:16:05 OK 2_object_stats_trigger.sql (4.22ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: up to current file version: 2 niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.058490177Z","level":"ERROR","duration":"407.784µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(302)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.058541118Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"232a1969-3653-4adf-981b-03ccbf5f9a60","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket25/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(302)"} niks3-go-unit-tests> 2026/08/27 09:16:05 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> --- PASS: TestReadProxyRootRedirectsToIndexHTML (0.40s) niks3-go-unit-tests> === CONT TestNARDeduplicationMetadataUploadBug niks3-go-unit-tests> --- PASS: TestService_Rustfstest (0.40s) niks3-go-unit-tests> === CONT TestReadProxyNarinfoAlreadyDecompressed niks3-go-unit-tests> --- PASS: TestCreatePendingClosure_SmallNARUsesSimplePUT (0.42s) niks3-go-unit-tests> === CONT TestReadProxyNarinfo niks3-go-unit-tests> --- PASS: TestReadProxyInvalidPath (0.42s) niks3-go-unit-tests> === CONT TestIsValidCachePath niks3-go-unit-tests> === RUN TestIsValidCachePath/narinfo niks3-go-unit-tests> === PAUSE TestIsValidCachePath/narinfo niks3-go-unit-tests> === RUN TestIsValidCachePath/narinfo_all_nix_base32_chars niks3-go-unit-tests> === PAUSE TestIsValidCachePath/narinfo_all_nix_base32_chars niks3-go-unit-tests> === RUN TestIsValidCachePath/nar_zst niks3-go-unit-tests> === PAUSE TestIsValidCachePath/nar_zst niks3-go-unit-tests> === RUN TestIsValidCachePath/nar_xz niks3-go-unit-tests> === PAUSE TestIsValidCachePath/nar_xz niks3-go-unit-tests> === RUN TestIsValidCachePath/nar_bz2 niks3-go-unit-tests> === PAUSE TestIsValidCachePath/nar_bz2 niks3-go-unit-tests> === RUN TestIsValidCachePath/nar_uncompressed niks3-go-unit-tests> === PAUSE TestIsValidCachePath/nar_uncompressed niks3-go-unit-tests> === RUN TestIsValidCachePath/ls niks3-go-unit-tests> === PAUSE TestIsValidCachePath/ls niks3-go-unit-tests> === RUN TestIsValidCachePath/log niks3-go-unit-tests> === PAUSE TestIsValidCachePath/log niks3-go-unit-tests> === RUN TestIsValidCachePath/realisation niks3-go-unit-tests> === PAUSE TestIsValidCachePath/realisation niks3-go-unit-tests> === RUN TestIsValidCachePath/nix-cache-info niks3-go-unit-tests> === PAUSE TestIsValidCachePath/nix-cache-info niks3-go-unit-tests> === RUN TestIsValidCachePath/index.html niks3-go-unit-tests> === PAUSE TestIsValidCachePath/index.html niks3-go-unit-tests> === RUN TestIsValidCachePath/traversal_parent niks3-go-unit-tests> === PAUSE TestIsValidCachePath/traversal_parent niks3-go-unit-tests> === RUN TestIsValidCachePath/traversal_in_middle niks3-go-unit-tests> === PAUSE TestIsValidCachePath/traversal_in_middle niks3-go-unit-tests> === RUN TestIsValidCachePath/invalid_char_e niks3-go-unit-tests> === PAUSE TestIsValidCachePath/invalid_char_e niks3-go-unit-tests> === RUN TestIsValidCachePath/invalid_char_u niks3-go-unit-tests> === PAUSE TestIsValidCachePath/invalid_char_u niks3-go-unit-tests> === RUN TestIsValidCachePath/random_path niks3-go-unit-tests> === PAUSE TestIsValidCachePath/random_path niks3-go-unit-tests> === RUN TestIsValidCachePath/empty niks3-go-unit-tests> === PAUSE TestIsValidCachePath/empty niks3-go-unit-tests> === RUN TestIsValidCachePath/leading_slash niks3-go-unit-tests> === PAUSE TestIsValidCachePath/leading_slash niks3-go-unit-tests> === RUN TestIsValidCachePath/wrong_extension niks3-go-unit-tests> === PAUSE TestIsValidCachePath/wrong_extension niks3-go-unit-tests> === RUN TestIsValidCachePath/short_hash niks3-go-unit-tests> === PAUSE TestIsValidCachePath/short_hash niks3-go-unit-tests> === CONT TestParseSingleRange niks3-go-unit-tests> === RUN TestParseSingleRange/none niks3-go-unit-tests> === PAUSE TestParseSingleRange/none niks3-go-unit-tests> === RUN TestParseSingleRange/unknown_unit niks3-go-unit-tests> === PAUSE TestParseSingleRange/unknown_unit niks3-go-unit-tests> === RUN TestParseSingleRange/multi-range_ignored niks3-go-unit-tests> === PAUSE TestParseSingleRange/multi-range_ignored niks3-go-unit-tests> === RUN TestParseSingleRange/malformed_no_dash niks3-go-unit-tests> === PAUSE TestParseSingleRange/malformed_no_dash niks3-go-unit-tests> === RUN TestParseSingleRange/malformed_both_empty niks3-go-unit-tests> === PAUSE TestParseSingleRange/malformed_both_empty niks3-go-unit-tests> === RUN TestParseSingleRange/malformed_end_before_start niks3-go-unit-tests> === PAUSE TestParseSingleRange/malformed_end_before_start niks3-go-unit-tests> === RUN TestParseSingleRange/closed niks3-go-unit-tests> === PAUSE TestParseSingleRange/closed niks3-go-unit-tests> === RUN TestParseSingleRange/open-ended niks3-go-unit-tests> === PAUSE TestParseSingleRange/open-ended niks3-go-unit-tests> === RUN TestParseSingleRange/end_clamped_to_size niks3-go-unit-tests> === PAUSE TestParseSingleRange/end_clamped_to_size niks3-go-unit-tests> === RUN TestParseSingleRange/suffix niks3-go-unit-tests> === PAUSE TestParseSingleRange/suffix niks3-go-unit-tests> === RUN TestParseSingleRange/suffix_exceeds_size niks3-go-unit-tests> === PAUSE TestParseSingleRange/suffix_exceeds_size niks3-go-unit-tests> === RUN TestParseSingleRange/single_byte niks3-go-unit-tests> === PAUSE TestParseSingleRange/single_byte niks3-go-unit-tests> === RUN TestParseSingleRange/start_past_EOF niks3-go-unit-tests> === PAUSE TestParseSingleRange/start_past_EOF niks3-go-unit-tests> === RUN TestParseSingleRange/start_far_past_EOF niks3-go-unit-tests> === PAUSE TestParseSingleRange/start_far_past_EOF niks3-go-unit-tests> === CONT TestObjectStatsTrigger niks3-go-unit-tests> 2026-08-27 09:16:05.101 UTC [630] ERROR: relation "goose_db_version" does not exist at character 36 niks3-go-unit-tests> 2026-08-27 09:16:05.101 UTC [630] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.102764582Z","level":"ERROR","duration":"226.382µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(314)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.102855543Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"a759dc41-e611-4fa9-85f0-aa0d341c9658","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket18/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(314)"} niks3-go-unit-tests> 2026/08/27 09:16:05 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> 2026/08/27 09:16:05 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> 2026/08/27 09:16:05 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> 2026/08/27 09:16:05 WARN Authentication failed token_preview=wrongtoken token_length=10 reason="static API token mismatch" niks3-go-unit-tests> --- PASS: TestService_AuthMiddleware (0.40s) niks3-go-unit-tests> === CONT TestCreatePendingClosureRejectsOversizedNAR niks3-go-unit-tests> 2026/08/27 09:16:05 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> --- PASS: TestCreatePendingClosureRejectsOversizedNAR (0.00s) niks3-go-unit-tests> === CONT TestOrphanedObjectsGCStressTest niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20241026095416_initial_model.sql (21.38ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20251210153512_drop_unused_gin_index.sql (3.35ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20251218171726_add_pins.sql (4.39ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20260628120000_add_object_size_and_stats.sql (5.45ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: successfully migrated database to version: 20260628120000 niks3-go-unit-tests> 2026/08/27 09:16:05 OK 1_commit_pending_closure.sql (3.66ms) niks3-go-unit-tests> --- PASS: TestMetricsInventory (0.48s) niks3-go-unit-tests> === CONT TestCacheConfigHandlerMaxNarSize niks3-go-unit-tests> --- PASS: TestCacheConfigHandlerMaxNarSize (0.00s) niks3-go-unit-tests> === CONT TestCacheStatsHandler niks3-go-unit-tests> 2026/08/27 09:16:05 OK 2_object_stats_trigger.sql (2.6ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: up to current file version: 2 niks3-go-unit-tests> 2026-08-27 09:16:05.153 UTC [638] ERROR: relation "goose_db_version" does not exist at character 36 niks3-go-unit-tests> 2026-08-27 09:16:05.153 UTC [638] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC niks3-go-unit-tests> 2026-08-27 09:16:05.163 UTC [641] ERROR: relation "goose_db_version" does not exist at character 36 niks3-go-unit-tests> 2026-08-27 09:16:05.163 UTC [641] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC niks3-go-unit-tests> 2026/08/27 09:16:05 INFO Aborted multipart uploads count=0 niks3-go-unit-tests> --- PASS: TestReadProxyRangeRequest (0.49s) niks3-go-unit-tests> === CONT TestGenerateLandingPage niks3-go-unit-tests> 2026/08/27 09:16:05 WARN Force mode enabled - objects will be deleted immediately without grace period niks3-go-unit-tests> --- PASS: TestGenerateLandingPage (0.01s) niks3-go-unit-tests> === CONT TestClientMultipleUploads niks3-go-unit-tests> 2026/08/27 09:16:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=0 niks3-go-unit-tests> 2026/08/27 09:16:05 INFO Vacuumed table table=pending_closures niks3-go-unit-tests> 2026/08/27 09:16:05 INFO Vacuumed table table=pending_objects niks3-go-unit-tests> 2026/08/27 09:16:05 INFO Vacuumed table table=multipart_uploads niks3-go-unit-tests> 2026/08/27 09:16:05 INFO Vacuumed table table=closures niks3-go-unit-tests> 2026/08/27 09:16:05 INFO Vacuumed table table=objects niks3-go-unit-tests> 2026-08-27 09:16:05.182 UTC [643] ERROR: relation "goose_db_version" does not exist at character 36 niks3-go-unit-tests> 2026-08-27 09:16:05.182 UTC [643] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20241026095416_initial_model.sql (17.17ms) niks3-go-unit-tests> --- PASS: TestReadProxyConditionalGet (0.51s) niks3-go-unit-tests> === CONT TestService_healthCheckHandler niks3-go-unit-tests> 2026/08/27 09:16:05 INFO Received complete multipart upload request method=POST path=/api/multipart/complete niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20251210153512_drop_unused_gin_index.sql (3.73ms) niks3-go-unit-tests> --- PASS: TestGCMetrics (0.46s) niks3-go-unit-tests> === CONT TestClientIntegration niks3-go-unit-tests> 2026-08-27 09:16:05.190 UTC [646] ERROR: relation "goose_db_version" does not exist at character 36 niks3-go-unit-tests> 2026-08-27 09:16:05.190 UTC [646] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC niks3-go-unit-tests> 2026/08/27 09:16:05 ERROR Failed to get multipart upload error="no rows in result set" upload_id=does-not-exist object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst niks3-go-unit-tests> --- PASS: TestCompleteMultipartUnregistered (0.53s) niks3-go-unit-tests> === CONT TestGracefulShutdownDrainsInflight niks3-go-unit-tests> 2026/08/27 09:16:05 INFO Starting HTTP server address=127.0.0.1:35057 niks3-go-unit-tests> 2026/08/27 09:16:05 INFO Shutdown signal received, draining in-flight requests timeout=10s niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20251218171726_add_pins.sql (18.41ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20241026095416_initial_model.sql (30.09ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20251210153512_drop_unused_gin_index.sql (10.24ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20260628120000_add_object_size_and_stats.sql (12.63ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: successfully migrated database to version: 20260628120000 niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20241026095416_initial_model.sql (14.93ms) niks3-go-unit-tests> --- PASS: TestResurrectedObjectNotDeleted (0.49s) niks3-go-unit-tests> === CONT TestClientErrorHandling niks3-go-unit-tests> === RUN TestClientErrorHandling/InvalidStorePath niks3-go-unit-tests> === PAUSE TestClientErrorHandling/InvalidStorePath niks3-go-unit-tests> === RUN TestClientErrorHandling/InvalidAuthToken niks3-go-unit-tests> === PAUSE TestClientErrorHandling/InvalidAuthToken niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20251218171726_add_pins.sql (8.07ms) niks3-go-unit-tests> === RUN TestClientErrorHandling/ServerNotAvailable niks3-go-unit-tests> 2026/08/27 09:16:05 OK 1_commit_pending_closure.sql (5.89ms) niks3-go-unit-tests> === PAUSE TestClientErrorHandling/ServerNotAvailable niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20241026095416_initial_model.sql (15.23ms) niks3-go-unit-tests> === CONT TestClientCADerivations niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20251210153512_drop_unused_gin_index.sql (5.09ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 2_object_stats_trigger.sql (2.85ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: up to current file version: 2 niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20251210153512_drop_unused_gin_index.sql (3.74ms) niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.229747503Z","level":"ERROR","duration":"245.322µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(209)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.229844984Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"8245c7b5-7cf6-4646-add9-d8b6d10926e8","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket19/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(209)"} niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20260628120000_add_object_size_and_stats.sql (5.48ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: successfully migrated database to version: 20260628120000 niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.230910494Z","level":"ERROR","duration":"546.905µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(209)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.230980795Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"1a323e8e-78a2-4a4d-bc46-28c070e65805","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket27/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(209)"} niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20251218171726_add_pins.sql (5.76ms) niks3-go-unit-tests> 2026-08-27 09:16:05.234 UTC [651] ERROR: relation "goose_db_version" does not exist at character 36 niks3-go-unit-tests> 2026-08-27 09:16:05.234 UTC [651] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC niks3-go-unit-tests> 2026-08-27 09:16:05.235 UTC [653] ERROR: relation "goose_db_version" does not exist at character 36 niks3-go-unit-tests> 2026-08-27 09:16:05.235 UTC [653] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC niks3-go-unit-tests> 2026/08/27 09:16:05 OK 1_commit_pending_closure.sql (4.67ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20251218171726_add_pins.sql (6.25ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 2_object_stats_trigger.sql (4.04ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: up to current file version: 2 niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20260628120000_add_object_size_and_stats.sql (6.67ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: successfully migrated database to version: 20260628120000 niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20260628120000_add_object_size_and_stats.sql (5.88ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: successfully migrated database to version: 20260628120000 niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.241865734Z","level":"ERROR","duration":"667.326µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(314)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.241935695Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"b826e35f-7f3b-4d13-8028-a953b74b8ca6","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket28/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(314)"} niks3-go-unit-tests> 2026/08/27 09:16:05 OK 1_commit_pending_closure.sql (3.8ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 1_commit_pending_closure.sql (3.72ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 2_object_stats_trigger.sql (2.51ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: up to current file version: 2 niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.248822138Z","level":"ERROR","duration":"622.446µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(209)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.248898799Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"86866e68-46d3-44ed-8682-71ec033b8168","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket29/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(209)"} niks3-go-unit-tests> 2026/08/27 09:16:05 OK 2_object_stats_trigger.sql (3.47ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: up to current file version: 2 niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.251141059Z","level":"ERROR","duration":"534.065µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(279)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.25121338Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"4353087e-f277-4f0f-aa86-2abd4571062a","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket30/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(279)"} niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20241026095416_initial_model.sql (15.8ms) niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.260097841Z","level":"ERROR","duration":"193.142µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(314)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.260158642Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"6c431616-a138-4c0e-a6fc-6ac52573fa27","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket18/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(314)"} niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20241026095416_initial_model.sql (18.42ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20251210153512_drop_unused_gin_index.sql (3.21ms) niks3-go-unit-tests> --- PASS: TestGracefulShutdownDrainsInflight (0.07s) niks3-go-unit-tests> === CONT TestService_ReadAuthMiddleware niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20251210153512_drop_unused_gin_index.sql (2.95ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20251218171726_add_pins.sql (4.65ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20251218171726_add_pins.sql (4.97ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20260628120000_add_object_size_and_stats.sql (4.5ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: successfully migrated database to version: 20260628120000 niks3-go-unit-tests> 2026-08-27 09:16:05.282 UTC [657] ERROR: relation "goose_db_version" does not exist at character 36 niks3-go-unit-tests> 2026-08-27 09:16:05.282 UTC [657] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20260628120000_add_object_size_and_stats.sql (13.05ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: successfully migrated database to version: 20260628120000 niks3-go-unit-tests> 2026/08/27 09:16:05 OK 1_commit_pending_closure.sql (13.48ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 1_commit_pending_closure.sql (4.07ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 2_object_stats_trigger.sql (3.07ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: up to current file version: 2 niks3-go-unit-tests> 2026/08/27 09:16:05 OK 2_object_stats_trigger.sql (2.01ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: up to current file version: 2 niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.290994484Z","level":"ERROR","duration":"676.206µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(279)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.291010064Z","level":"ERROR","duration":"686.786µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(209)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.291086025Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"d0df395c-f970-46dd-abb8-e989a399b578","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket32/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(279)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.291087345Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"48685707-69fa-4f7b-a619-1a0310fb3bd6","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket31/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(209)"} niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20241026095416_initial_model.sql (13.35ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20251210153512_drop_unused_gin_index.sql (2.29ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20251218171726_add_pins.sql (3.91ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20260628120000_add_object_size_and_stats.sql (4.56ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: successfully migrated database to version: 20260628120000 niks3-go-unit-tests> 2026-08-27 09:16:05.314 UTC [659] ERROR: relation "goose_db_version" does not exist at character 36 niks3-go-unit-tests> 2026-08-27 09:16:05.314 UTC [659] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC niks3-go-unit-tests> 2026-08-27 09:16:05.315 UTC [658] ERROR: relation "goose_db_version" does not exist at character 36 niks3-go-unit-tests> 2026-08-27 09:16:05.315 UTC [658] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC niks3-go-unit-tests> 2026-08-27 09:16:05.316 UTC [660] ERROR: relation "goose_db_version" does not exist at character 36 niks3-go-unit-tests> 2026-08-27 09:16:05.316 UTC [660] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC niks3-go-unit-tests> 2026/08/27 09:16:05 OK 1_commit_pending_closure.sql (3.78ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 2_object_stats_trigger.sql (2ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: up to current file version: 2 niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.321387802Z","level":"ERROR","duration":"660.206µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(314)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.321474883Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"137ff5b1-a8e1-4c01-bad4-aaa5222ddf2e","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket33/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(314)"} niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20241026095416_initial_model.sql (10.44ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20241026095416_initial_model.sql (10.26ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20251210153512_drop_unused_gin_index.sql (1.48ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20251210153512_drop_unused_gin_index.sql (1.44ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20241026095416_initial_model.sql (12.6ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20251218171726_add_pins.sql (3.2ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20251218171726_add_pins.sql (2.5ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20251210153512_drop_unused_gin_index.sql (1.72ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20260628120000_add_object_size_and_stats.sql (3.25ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: successfully migrated database to version: 20260628120000 niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20260628120000_add_object_size_and_stats.sql (3.06ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: successfully migrated database to version: 20260628120000 niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20251218171726_add_pins.sql (3.34ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 1_commit_pending_closure.sql (2.12ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 1_commit_pending_closure.sql (2.28ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 2_object_stats_trigger.sql (1.04ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: up to current file version: 2 niks3-go-unit-tests> 2026-08-27 09:16:05.342 UTC [661] ERROR: relation "goose_db_version" does not exist at character 36 niks3-go-unit-tests> 2026-08-27 09:16:05.342 UTC [661] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC niks3-go-unit-tests> 2026/08/27 09:16:05 OK 2_object_stats_trigger.sql (1.07ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: up to current file version: 2 niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.343765487Z","level":"ERROR","duration":"156.962µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(279)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.343820447Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"8833ce7f-a46b-40f3-b110-480a6b4461d8","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket29/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(279)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.344498933Z","level":"ERROR","duration":"350.223µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(279)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.344531394Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"c2b9261e-a950-45f5-9091-73a8e94571fa","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket35/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(279)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.344654695Z","level":"ERROR","duration":"446.564µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(279)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.344674215Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"931e1a1d-96c1-487a-81a5-756b227fbf3a","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket34/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(279)"} niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20260628120000_add_object_size_and_stats.sql (3.68ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: successfully migrated database to version: 20260628120000 niks3-go-unit-tests> 2026/08/27 09:16:05 OK 1_commit_pending_closure.sql (2.4ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 2_object_stats_trigger.sql (953.39µs) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: up to current file version: 2 niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.350391367Z","level":"ERROR","duration":"564.485µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(308)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.350470628Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"82088948-4d84-4a3e-99dd-ab6f7e3b84a6","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket36/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(308)"} niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20241026095416_initial_model.sql (10.18ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20251210153512_drop_unused_gin_index.sql (1.32ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20251218171726_add_pins.sql (3.1ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20260628120000_add_object_size_and_stats.sql (3.82ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: successfully migrated database to version: 20260628120000 niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.367870247Z","level":"ERROR","duration":"192.342µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(314)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.367939068Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"78ac5463-732e-435a-b3b2-546dcbe5313f","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket32/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(314)"} niks3-go-unit-tests> 2026/08/27 09:16:05 OK 1_commit_pending_closure.sql (1.89ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 2_object_stats_trigger.sql (821.29µs) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: up to current file version: 2 niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.371193417Z","level":"ERROR","duration":"132.021µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(314)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.371244638Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"626c49d9-1626-4053-9e54-b970009235c0","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket27/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(314)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.372072605Z","level":"ERROR","duration":"476.084µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(314)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.372131166Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"4e4f98d8-c9d1-442a-8557-002af8d4793a","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket37/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(314)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.386070634Z","level":"ERROR","duration":"159.822µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(209)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.386127474Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"4ae2a01d-0ed2-4ddd-8411-cc331b225660","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket37/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(209)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.402819467Z","level":"ERROR","duration":"130.661µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(209)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.402868887Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"d5c8545f-b2a6-4e1a-92a7-a033ab4c83f6","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket33/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(209)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.409464727Z","level":"ERROR","duration":"101.441µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(209)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.409508248Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"fb2634a6-81c2-46ce-9ecc-6d81d9e528e0","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket28/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(209)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.432282996Z","level":"ERROR","duration":"129.921µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(209)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.432334017Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"901dcc6f-bf07-44c5-8e49-bdcba72c964c","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket19/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(209)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.434810439Z","level":"ERROR","duration":"93.381µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(209)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.434851479Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"141e787d-2935-479d-bbbd-24e869081f6e","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket30/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(209)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.444580169Z","level":"ERROR","duration":"107.681µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(209)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.444624189Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"8ea597da-35df-4b75-b475-572c7947dee6","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket34/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(209)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.449177851Z","level":"ERROR","duration":"95.001µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(209)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.449220611Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"1f4a8ca7-2724-4b16-a561-5b502ad955c7","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket33/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(209)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.453742892Z","level":"ERROR","duration":"98.161µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(209)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.453787093Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"26365cd3-80bc-4f29-a881-3aa463aa88d9","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket31/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(209)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.461340422Z","level":"ERROR","duration":"101.321µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(209)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.461384742Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"2880fc1b-1c84-4719-91ee-aa6e7404f957","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket35/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(209)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.471091271Z","level":"ERROR","duration":"107.661µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(209)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.471135512Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"3eb77fcf-ba93-4cbd-87a2-308bebf5b571","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket34/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(209)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.512817853Z","level":"ERROR","duration":"138.282µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(209)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.512870213Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"d1112c54-4cf3-41c9-842e-253f16f81e60","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket31/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(209)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.53661945Z","level":"ERROR","duration":"122.441µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(209)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.536668071Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"6d41ff33-af49-4bf2-8c07-4bb9b3d63787","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket31/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(209)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.541306353Z","level":"ERROR","duration":"90.021µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(209)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.541348634Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"c28ca1b4-b6a6-4114-941a-0f08b6351824","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket36/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(209)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.544998387Z","level":"ERROR","duration":"196.802µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(209)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.545040768Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"bdb4edd3-0836-45dc-a3fd-75968fff5800","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket30/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(209)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.596843601Z","level":"ERROR","duration":"1.828637ms","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(209)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.596898962Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"1b1a9792-2be7-44d7-8b04-6e90a8c0308e","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket29/","status_code":503,"duration_ms":1,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(209)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.614619324Z","level":"ERROR","duration":"170.522µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(209)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.614663544Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"2c12cfd9-864a-4c6f-8f29-af10cb5dec3f","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket35/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(209)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.621196164Z","level":"ERROR","duration":"84.401µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(209)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.621220124Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"81dda948-e569-4e78-b77f-cb2d7748f5a2","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket27/","status_code":503,"duration_ms":2,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(209)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.652034506Z","level":"ERROR","duration":"3.229469ms","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(209)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:05.652085107Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"7a0411ff-e16e-41de-84dd-5689bb043df3","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket37/","status_code":503,"duration_ms":3,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(209)"} niks3-go-unit-tests> --- PASS: TestReadProxyHead (1.00s) niks3-go-unit-tests> === CONT TestCacheConfigHandler niks3-go-unit-tests> === RUN TestCacheConfigHandler/full_config,_no_issuer niks3-go-unit-tests> === PAUSE TestCacheConfigHandler/full_config,_no_issuer niks3-go-unit-tests> === RUN TestCacheConfigHandler/no_cache_url_configured niks3-go-unit-tests> === PAUSE TestCacheConfigHandler/no_cache_url_configured niks3-go-unit-tests> === RUN TestCacheConfigHandler/no_signing_keys niks3-go-unit-tests> === PAUSE TestCacheConfigHandler/no_signing_keys niks3-go-unit-tests> === RUN TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator niks3-go-unit-tests> === PAUSE TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator niks3-go-unit-tests> === CONT TestOrphanedObjectsGC niks3-go-unit-tests> 2026/08/27 09:16:05 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> 2026/08/27 09:16:05 WARN mTLS auth: subject not in bound subjects subject="CN=writer" niks3-go-unit-tests> --- PASS: TestService_ReadAuthMiddleware (0.41s) niks3-go-unit-tests> === CONT TestService_AuthMiddleware_OIDC niks3-go-unit-tests> 2026/08/27 09:16:05 INFO OIDC provider initialized name=test niks3-go-unit-tests> 2026/08/27 09:16:05 INFO Created nix-cache-info in bucket bucket=bucket25 niks3-go-unit-tests> 2026/08/27 09:16:05 INFO Created nix-cache-info in bucket bucket=bucket33 niks3-go-unit-tests> 2026/08/27 09:16:05 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> 2026-08-27 09:16:05.752 UTC [685] ERROR: relation "goose_db_version" does not exist at character 36 niks3-go-unit-tests> 2026-08-27 09:16:05.752 UTC [685] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC niks3-go-unit-tests> === NAME TestClientMultipleUploads niks3-go-unit-tests> client_integration_test.go:338: Created store path 0: /build/TestClientMultipleUploads370382083/001/store/q61b497vmc5ddplki781695ymhwhzn0f-test-file-0.txt niks3-go-unit-tests> 2026-08-27 09:16:05.764 UTC [702] ERROR: relation "goose_db_version" does not exist at character 36 niks3-go-unit-tests> 2026-08-27 09:16:05.764 UTC [702] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC niks3-go-unit-tests> 2026/08/27 09:16:05 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20241026095416_initial_model.sql (8.39ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20251210153512_drop_unused_gin_index.sql (2.39ms) niks3-go-unit-tests> === NAME TestPinProtectsFromGC niks3-go-unit-tests> client_integration_test.go:646: Pinned store path: /build/TestPinProtectsFromGC1663763512/001/store/1sfvrsrhj4r4vqgmwaybv2a7xn2ypwqq-pinned-file.txt niks3-go-unit-tests> client_integration_test.go:647: Unpinned store path: /build/TestPinProtectsFromGC1663763512/001/store/j0ql421zwbjglvaczrb46z2wwdwz6kkp-unpinned-file.txt niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20251218171726_add_pins.sql (6.28ms) niks3-go-unit-tests> 2026/08/27 09:16:05 INFO Created nix-cache-info in bucket bucket=bucket36 niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20241026095416_initial_model.sql (14.74ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20260628120000_add_object_size_and_stats.sql (8.61ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: successfully migrated database to version: 20260628120000 niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20251210153512_drop_unused_gin_index.sql (1.51ms) niks3-go-unit-tests> 2026/08/27 09:16:05 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> 2026/08/27 09:16:05 OK 1_commit_pending_closure.sql (2.68ms) niks3-go-unit-tests> 2026/08/27 09:16:05 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20251218171726_add_pins.sql (3.04ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 2_object_stats_trigger.sql (1.14ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: up to current file version: 2 niks3-go-unit-tests> 2026/08/27 09:16:05 OK 20260628120000_add_object_size_and_stats.sql (3.85ms) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: successfully migrated database to version: 20260628120000 niks3-go-unit-tests> === NAME TestClientMultipleUploads niks3-go-unit-tests> client_integration_test.go:338: Created store path 1: /build/TestClientMultipleUploads370382083/001/store/gk7ndjg2yhpbjp4jkr20gbjkq1zlxmj2-test-file-1.txt niks3-go-unit-tests> 2026/08/27 09:16:05 OK 1_commit_pending_closure.sql (2.28ms) niks3-go-unit-tests> 2026/08/27 09:16:05 OK 2_object_stats_trigger.sql (918.13µs) niks3-go-unit-tests> 2026/08/27 09:16:05 goose: up to current file version: 2 niks3-go-unit-tests> client_integration_test.go:338: Created store path 2: /build/TestClientMultipleUploads370382083/001/store/m88l640s3xb2i04b5hrh644zz8yxnnn6-test-file-2.txt niks3-go-unit-tests> 2026/08/27 09:16:05 WARN Failed to fetch cache config, uploading without size limit error="fetching cache config: server returned 404: 404 page not found\n" niks3-go-unit-tests> === NAME TestClientCADerivations niks3-go-unit-tests> client_ca_test.go:136: Built CA derivation: /build/TestClientCADerivations2760299426/001/store/mm8mzf8nbmqxdcw04hq3g91h8xhipag5-ca-test niks3-go-unit-tests> 2026/08/27 09:16:05 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> 2026/08/27 09:16:05 INFO Uploading 1 paths to 127.0.0.1 (0 already cached) niks3-go-unit-tests> 2026/08/27 09:16:05 INFO Uploading 1sfvrsrhj4r4vqgmwaybv2a7xn2ypwqq-pinned-file.txt (128B) niks3-go-unit-tests> client_ca_test.go:139: Found 1 dependencies (including self) niks3-go-unit-tests> 2026/08/27 09:16:05 WARN Failed to fetch cache config, uploading without size limit error="fetching cache config: server returned 404: 404 page not found\n" niks3-go-unit-tests> --- PASS: TestGCBugBareHashReferences (1.23s) niks3-go-unit-tests> === CONT TestServerTLSConfig niks3-go-unit-tests> 2026/08/27 09:16:05 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> === RUN TestServerTLSConfig/no_client_CA niks3-go-unit-tests> === PAUSE TestServerTLSConfig/no_client_CA niks3-go-unit-tests> === RUN TestServerTLSConfig/missing_CA_file niks3-go-unit-tests> === PAUSE TestServerTLSConfig/missing_CA_file niks3-go-unit-tests> === RUN TestServerTLSConfig/not_a_PEM_file niks3-go-unit-tests> === PAUSE TestServerTLSConfig/not_a_PEM_file niks3-go-unit-tests> === CONT TestMultipartCleanup niks3-go-unit-tests> 2026/08/27 09:16:05 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> 2026/08/27 09:16:05 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> 2026/08/27 09:16:05 INFO Uploading 3 paths to 127.0.0.1 (0 already cached) niks3-go-unit-tests> 2026/08/27 09:16:05 INFO Uploading m88l640s3xb2i04b5hrh644zz8yxnnn6-test-file-2.txt (160B) niks3-go-unit-tests> 2026/08/27 09:16:05 INFO Uploading gk7ndjg2yhpbjp4jkr20gbjkq1zlxmj2-test-file-1.txt (160B) niks3-go-unit-tests> 2026/08/27 09:16:05 INFO Uploading q61b497vmc5ddplki781695ymhwhzn0f-test-file-0.txt (160B) niks3-go-unit-tests> 2026/08/27 09:16:05 WARN Failed to fetch cache config, uploading without size limit error="fetching cache config: server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026-08-27 09:16:06.027 UTC [957] ERROR: relation "goose_db_version" does not exist at character 36 niks3-go-unit-tests> 2026-08-27 09:16:06.027 UTC [957] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC niks3-go-unit-tests> 2026/08/27 09:16:06 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> 2026/08/27 09:16:06 INFO Uploading 1 paths to 127.0.0.1 (0 already cached) niks3-go-unit-tests> 2026/08/27 09:16:06 INFO Uploading mm8mzf8nbmqxdcw04hq3g91h8xhipag5-ca-test (144B) niks3-go-unit-tests> 2026/08/27 09:16:06 OK 20241026095416_initial_model.sql (8.07ms) niks3-go-unit-tests> 2026/08/27 09:16:06 OK 20251210153512_drop_unused_gin_index.sql (1.31ms) niks3-go-unit-tests> 2026/08/27 09:16:06 OK 20251218171726_add_pins.sql (6.29ms) niks3-go-unit-tests> 2026/08/27 09:16:06 OK 20260628120000_add_object_size_and_stats.sql (3ms) niks3-go-unit-tests> 2026/08/27 09:16:06 goose: successfully migrated database to version: 20260628120000 niks3-go-unit-tests> 2026/08/27 09:16:06 OK 1_commit_pending_closure.sql (2.99ms) niks3-go-unit-tests> 2026/08/27 09:16:06 OK 2_object_stats_trigger.sql (1.11ms) niks3-go-unit-tests> 2026/08/27 09:16:06 goose: up to current file version: 2 niks3-go-unit-tests> 2026/08/27 09:16:06 WARN Failed to register uploaded object key=nar/0ahq3izw3h0b0c08p3xdfclh758fwxnp85ccvflasb4vnfsv50ri.nar.zst error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/08/27 09:16:06 WARN Failed to register uploaded object key=nar/0qmhsg42pqsz7x6hh02j8gznnac3yi3qpjhrqbwvz88i194jbb5s.nar.zst error="server returned 404: 404 page not found\n" niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:06.278231614Z","level":"ERROR","duration":"2.448143ms","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(322)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:06.278314974Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"2ba896cf-fcda-414f-af4f-63a932390d08","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket30/","status_code":503,"duration_ms":2,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(322)"} niks3-go-unit-tests> 2026/08/27 09:16:06 WARN Failed to register uploaded object key=nar/13p9pqi0ga6fx2apcwls9hjnv7nmf740ggpn5zvfzdsp5vz2bbg4.nar.zst error="server returned 404: 404 page not found\n" niks3-go-unit-tests> --- PASS: TestService_healthCheckHandler (1.10s) niks3-go-unit-tests> === CONT TestService_AuthMiddleware_MTLSBoundSubjects niks3-go-unit-tests> 2026/08/27 09:16:06 WARN Failed to register uploaded object key=nar/1823g6vx4q5zqvr73m7lkq1wpk40fin7bsc792di430yk7sg4lka.nar.zst error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/08/27 09:16:06 WARN Failed to register uploaded object key=1sfvrsrhj4r4vqgmwaybv2a7xn2ypwqq.ls error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/08/27 09:16:06 WARN Failed to register uploaded object key=gk7ndjg2yhpbjp4jkr20gbjkq1zlxmj2.ls error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/08/27 09:16:06 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign niks3-go-unit-tests> 2026/08/27 09:16:06 WARN Failed to register uploaded object key=m88l640s3xb2i04b5hrh644zz8yxnnn6.ls error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/08/27 09:16:06 WARN Failed to register uploaded object key=q61b497vmc5ddplki781695ymhwhzn0f.ls error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/08/27 09:16:06 WARN Failed to register uploaded object key=log/439mlin36arpxfz71srhg7prjvcqwmpf-ca-test.drv error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/08/27 09:16:06 INFO Received sign narinfos request method=POST path=/api/pending_closures/3/sign niks3-go-unit-tests> 2026/08/27 09:16:06 WARN Failed to register uploaded object key=nar/04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n.nar.zst error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/08/27 09:16:06 INFO Signed narinfos id=1 count=1 niks3-go-unit-tests> 2026/08/27 09:16:06 INFO Uploading 1 narinfos niks3-go-unit-tests> 2026/08/27 09:16:06 INFO Signed narinfos id=3 count=1 niks3-go-unit-tests> 2026/08/27 09:16:06 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign niks3-go-unit-tests> 2026/08/27 09:16:06 INFO Signed narinfos id=1 count=1 niks3-go-unit-tests> 2026/08/27 09:16:06 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign niks3-go-unit-tests> 2026/08/27 09:16:06 INFO Signed narinfos id=2 count=1 niks3-go-unit-tests> 2026/08/27 09:16:06 INFO Uploading 3 narinfos niks3-go-unit-tests> 2026/08/27 09:16:06 WARN Failed to register uploaded object key=mm8mzf8nbmqxdcw04hq3g91h8xhipag5.ls error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/08/27 09:16:06 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> 2026/08/27 09:16:06 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign niks3-go-unit-tests> 2026/08/27 09:16:06 INFO Signed narinfos id=1 count=1 niks3-go-unit-tests> 2026/08/27 09:16:06 INFO Uploading 1 narinfos niks3-go-unit-tests> 2026/08/27 09:16:06 WARN Failed to register uploaded object key=1sfvrsrhj4r4vqgmwaybv2a7xn2ypwqq.narinfo error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/08/27 09:16:06 WARN Failed to register uploaded object key=gk7ndjg2yhpbjp4jkr20gbjkq1zlxmj2.narinfo error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/08/27 09:16:06 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete niks3-go-unit-tests> 2026/08/27 09:16:06 WARN Failed to register uploaded object key=m88l640s3xb2i04b5hrh644zz8yxnnn6.narinfo error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/08/27 09:16:06 WARN Failed to register uploaded object key=q61b497vmc5ddplki781695ymhwhzn0f.narinfo error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/08/27 09:16:06 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete niks3-go-unit-tests> 2026/08/27 09:16:06 WARN Failed to register uploaded object key=mm8mzf8nbmqxdcw04hq3g91h8xhipag5.narinfo error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/08/27 09:16:06 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete niks3-go-unit-tests> 2026/08/27 09:16:06 INFO Completed upload id=1 niks3-go-unit-tests> 2026/08/27 09:16:06 INFO Upload complete. (475ms) niks3-go-unit-tests> 2026/08/27 09:16:06 INFO Completed upload id=1 niks3-go-unit-tests> 2026/08/27 09:16:06 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete niks3-go-unit-tests> 2026/08/27 09:16:06 INFO Completed upload id=1 niks3-go-unit-tests> 2026/08/27 09:16:06 INFO Upload complete. (352ms) niks3-go-unit-tests> 2026/08/27 09:16:06 INFO Completed upload id=2 niks3-go-unit-tests> 2026/08/27 09:16:06 INFO Received complete upload request method=POST path=/api/pending_closures/3/complete niks3-go-unit-tests> --- PASS: TestReadProxy404 (1.49s) niks3-go-unit-tests> === CONT TestGCTaskStore_ConflictDifferentParams niks3-go-unit-tests> --- PASS: TestGCTaskStore_ConflictDifferentParams (0.00s) niks3-go-unit-tests> === CONT TestClientWithDependencies niks3-go-unit-tests> 2026/08/27 09:16:06 INFO Completed upload id=3 niks3-go-unit-tests> 2026/08/27 09:16:06 INFO Upload complete. (429ms) niks3-go-unit-tests> === NAME TestClientCADerivations niks3-go-unit-tests> client_ca_test.go:180: Narinfo contains CA field: StorePath: /build/TestClientCADerivations2760299426/001/store/mm8mzf8nbmqxdcw04hq3g91h8xhipag5-ca-test niks3-go-unit-tests> URL: nar/04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n.nar.zst niks3-go-unit-tests> Compression: zstd niks3-go-unit-tests> NarHash: sha256:04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n niks3-go-unit-tests> NarSize: 144 niks3-go-unit-tests> References: niks3-go-unit-tests> Deriver: /build/TestClientCADerivations2760299426/001/store/439mlin36arpxfz71srhg7prjvcqwmpf-ca-test.drv niks3-go-unit-tests> CA: fixed:r:sha256:04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n niks3-go-unit-tests> client_ca_test.go:185: Checking for realisation files in S3... niks3-go-unit-tests> === NAME TestClientMultipleUploads niks3-go-unit-tests> client_integration_test.go:349: Uploaded 3 paths in 463.49924ms niks3-go-unit-tests> === NAME TestClientCADerivations niks3-go-unit-tests> client_ca_test.go:207: No realisation files uploaded - this is expected for locally built CA derivations niks3-go-unit-tests> client_ca_test.go:208: Locally built CA derivations don't have DrvOutput IDs until they're substituted from a cache niks3-go-unit-tests> --- PASS: TestClientMultipleUploads (1.15s) niks3-go-unit-tests> === CONT TestGCTaskStore_GetEmpty niks3-go-unit-tests> --- PASS: TestGCTaskStore_GetEmpty (0.00s) niks3-go-unit-tests> === CONT TestService_NativeMTLS niks3-go-unit-tests> 2026-08-27 09:16:06.355 UTC [1019] ERROR: relation "goose_db_version" does not exist at character 36 niks3-go-unit-tests> 2026-08-27 09:16:06.355 UTC [1019] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC niks3-go-unit-tests> 2026/08/27 09:16:06 WARN Failed to fetch cache config, uploading without size limit error="fetching cache config: server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/08/27 09:16:06 OK 20241026095416_initial_model.sql (11.71ms) niks3-go-unit-tests> 2026/08/27 09:16:06 OK 20251210153512_drop_unused_gin_index.sql (2.89ms) niks3-go-unit-tests> 2026/08/27 09:16:06 OK 20251218171726_add_pins.sql (3.42ms) niks3-go-unit-tests> 2026-08-27 09:16:06.400 UTC [1098] ERROR: relation "goose_db_version" does not exist at character 36 niks3-go-unit-tests> 2026-08-27 09:16:06.400 UTC [1098] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC niks3-go-unit-tests> 2026/08/27 09:16:06 OK 20260628120000_add_object_size_and_stats.sql (4.43ms) niks3-go-unit-tests> 2026/08/27 09:16:06 goose: successfully migrated database to version: 20260628120000 niks3-go-unit-tests> 2026/08/27 09:16:06 OK 1_commit_pending_closure.sql (1.98ms) niks3-go-unit-tests> 2026/08/27 09:16:06 OK 2_object_stats_trigger.sql (925.81µs) niks3-go-unit-tests> 2026/08/27 09:16:06 goose: up to current file version: 2 niks3-go-unit-tests> 2026-08-27 09:16:06.409 UTC [1099] ERROR: relation "goose_db_version" does not exist at character 36 niks3-go-unit-tests> 2026-08-27 09:16:06.409 UTC [1099] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC niks3-go-unit-tests> 2026/08/27 09:16:06 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> 2026/08/27 09:16:06 OK 20241026095416_initial_model.sql (11.24ms) niks3-go-unit-tests> 2026/08/27 09:16:06 INFO Uploading 1 paths to 127.0.0.1 (0 already cached) niks3-go-unit-tests> 2026/08/27 09:16:06 INFO Uploading j0ql421zwbjglvaczrb46z2wwdwz6kkp-unpinned-file.txt (128B) niks3-go-unit-tests> 2026/08/27 09:16:06 OK 20251210153512_drop_unused_gin_index.sql (1.29ms) niks3-go-unit-tests> 2026/08/27 09:16:06 WARN Failed to register uploaded object key=nar/0gh0wdnr6gmk4r5sfzzh12isnl14rgmkrwy1ws41hpsrlndclg5h.nar.zst error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/08/27 09:16:06 OK 20251218171726_add_pins.sql (4.6ms) niks3-go-unit-tests> 2026/08/27 09:16:06 OK 20241026095416_initial_model.sql (8.91ms) niks3-go-unit-tests> 2026/08/27 09:16:06 OK 20251210153512_drop_unused_gin_index.sql (1.33ms) niks3-go-unit-tests> 2026/08/27 09:16:06 OK 20260628120000_add_object_size_and_stats.sql (2.72ms) niks3-go-unit-tests> 2026/08/27 09:16:06 goose: successfully migrated database to version: 20260628120000 niks3-go-unit-tests> 2026/08/27 09:16:06 OK 1_commit_pending_closure.sql (2.38ms) niks3-go-unit-tests> 2026/08/27 09:16:06 OK 20251218171726_add_pins.sql (2.86ms) niks3-go-unit-tests> 2026/08/27 09:16:06 OK 2_object_stats_trigger.sql (814.31µs) niks3-go-unit-tests> 2026/08/27 09:16:06 goose: up to current file version: 2 niks3-go-unit-tests> 2026/08/27 09:16:06 OK 20260628120000_add_object_size_and_stats.sql (3.19ms) niks3-go-unit-tests> 2026/08/27 09:16:06 goose: successfully migrated database to version: 20260628120000 niks3-go-unit-tests> 2026/08/27 09:16:06 OK 1_commit_pending_closure.sql (1.68ms) niks3-go-unit-tests> 2026/08/27 09:16:06 OK 2_object_stats_trigger.sql (851.83µs) niks3-go-unit-tests> 2026/08/27 09:16:06 goose: up to current file version: 2 niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:06.436689543Z","level":"ERROR","duration":"844.228µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(279)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:06.436782524Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"cf75f574-364d-4042-a790-004018940708","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket43/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(279)"} niks3-go-unit-tests> === NAME TestClientCADerivations niks3-go-unit-tests> client_ca_test.go:258: nix copy output: warning: you don't have Internet access; disabling some network-dependent features niks3-go-unit-tests> warning: failed to create TLS context for AWS credential providers; SSO, STS WebIdentity, and ECS container authentication will be unavailable niks3-go-unit-tests> error: binary cache 's3://bucket36?endpoint=http://localhost:37323®ion=eu-west-1' is for Nix stores with prefix '/nix/store', not '/build/TestClientCADerivations2760299426/001/store' niks3-go-unit-tests> client_ca_test.go:259: nix copy failed (might be expected with isolated stores): exit status 1 niks3-go-unit-tests> --- PASS: TestClientCADerivations (1.24s) niks3-go-unit-tests> === CONT TestService_AuthMiddleware_MTLSProxyHeader niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:06.482503902Z","level":"ERROR","duration":"272.802µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(308)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:06.482594203Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"db036093-0940-4053-992f-f5eb8544d671","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket31/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(308)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:06.514654196Z","level":"ERROR","duration":"269.402µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(279)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:06.514782157Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"007c7cf8-0262-49a2-b47d-69b9e98da66a","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket43/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(279)"} niks3-go-unit-tests> 2026-08-27 09:16:06.531 UTC [1156] ERROR: relation "goose_db_version" does not exist at character 36 niks3-go-unit-tests> 2026-08-27 09:16:06.531 UTC [1156] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC niks3-go-unit-tests> 2026/08/27 09:16:06 OK 20241026095416_initial_model.sql (9.75ms) niks3-go-unit-tests> 2026/08/27 09:16:06 OK 20251210153512_drop_unused_gin_index.sql (1.38ms) niks3-go-unit-tests> 2026/08/27 09:16:06 OK 20251218171726_add_pins.sql (3.07ms) niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:06.555800192Z","level":"ERROR","duration":"240.542µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(279)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:06.555891393Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"8a468ff7-8d1d-43c5-ba99-dfd8d0eda63e","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket31/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(279)"} niks3-go-unit-tests> 2026/08/27 09:16:06 OK 20260628120000_add_object_size_and_stats.sql (4.67ms) niks3-go-unit-tests> 2026/08/27 09:16:06 goose: successfully migrated database to version: 20260628120000 niks3-go-unit-tests> 2026/08/27 09:16:06 OK 1_commit_pending_closure.sql (1.99ms) niks3-go-unit-tests> 2026/08/27 09:16:06 OK 2_object_stats_trigger.sql (861.11µs) niks3-go-unit-tests> 2026/08/27 09:16:06 goose: up to current file version: 2 niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:06.56202127Z","level":"ERROR","duration":"606.605µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(282)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:06.56209939Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"f34a7e7e-e6e0-4500-8c79-77802b012cf8","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket44/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(282)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:06.627339067Z","level":"ERROR","duration":"376.463µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(279)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:06.627415908Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"0904423c-02ce-4470-9c71-6dc6b82d78e9","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket43/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(279)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:06.633928727Z","level":"ERROR","duration":"203.621102ms","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(261)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:06.633987608Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"8534ddcd-bf28-407b-b737-8640868b23de","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket42/","status_code":503,"duration_ms":203,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(261)"} niks3-go-unit-tests> 2026/08/27 09:16:06 WARN Failed to register uploaded object key=j0ql421zwbjglvaczrb46z2wwdwz6kkp.ls error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/08/27 09:16:06 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign niks3-go-unit-tests> 2026/08/27 09:16:06 INFO Signed narinfos id=2 count=1 niks3-go-unit-tests> 2026/08/27 09:16:06 INFO Uploading 1 narinfos niks3-go-unit-tests> 2026/08/27 09:16:06 WARN Failed to register uploaded object key=j0ql421zwbjglvaczrb46z2wwdwz6kkp.narinfo error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/08/27 09:16:06 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete niks3-go-unit-tests> 2026/08/27 09:16:06 INFO Completed upload id=2 niks3-go-unit-tests> 2026/08/27 09:16:06 INFO Upload complete. (303ms) niks3-go-unit-tests> 2026/08/27 09:16:06 INFO Received create pin request method=POST path=/api/pins/myapp niks3-go-unit-tests> 2026/08/27 09:16:06 INFO Created/updated pin name=myapp store_path=/build/TestPinProtectsFromGC1663763512/001/store/1sfvrsrhj4r4vqgmwaybv2a7xn2ypwqq-pinned-file.txt narinfo_key=1sfvrsrhj4r4vqgmwaybv2a7xn2ypwqq.narinfo niks3-go-unit-tests> 2026/08/27 09:16:06 INFO Starting cleanup of old closures method=DELETE path=/api/closures niks3-go-unit-tests> 2026/08/27 09:16:06 INFO Garbage collection started niks3-go-unit-tests> 2026/08/27 09:16:06 INFO Aborted multipart uploads count=0 niks3-go-unit-tests> 2026/08/27 09:16:06 WARN Force mode enabled - objects will be deleted immediately without grace period niks3-go-unit-tests> === NAME TestOrphanedObjectsGC niks3-go-unit-tests> orphaned_objects_gc_test.go:290: GC Test Summary: niks3-go-unit-tests> orphaned_objects_gc_test.go:291: - Kept: 2 objects from closure A niks3-go-unit-tests> orphaned_objects_gc_test.go:292: - Deleted: 2 objects from closure B niks3-go-unit-tests> orphaned_objects_gc_test.go:293: - Deleted: 6 orphaned chain objects (X1->X2->X3) niks3-go-unit-tests> orphaned_objects_gc_test.go:294: - Deleted: 2 orphaned single objects (Y) niks3-go-unit-tests> orphaned_objects_gc_test.go:295: - Total deleted: 10 objects niks3-go-unit-tests> --- PASS: TestOrphanedObjectsGC (1.05s) niks3-go-unit-tests> === CONT TestGCTaskStore_PhaseUpdates niks3-go-unit-tests> --- PASS: TestGCTaskStore_PhaseUpdates (0.00s) niks3-go-unit-tests> === CONT TestGCTaskStore_CompletedAllowsNewTask niks3-go-unit-tests> --- PASS: TestGCTaskStore_CompletedAllowsNewTask (0.00s) niks3-go-unit-tests> === CONT TestGCTaskStore_DeduplicateSameParams niks3-go-unit-tests> --- PASS: TestGCTaskStore_DeduplicateSameParams (0.00s) niks3-go-unit-tests> === CONT TestGCTaskStore_GetReturnsLatest niks3-go-unit-tests> --- PASS: TestGCTaskStore_GetReturnsLatest (0.00s) niks3-go-unit-tests> === CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info niks3-go-unit-tests> 2026/08/27 09:16:06 INFO Received uploads request method=POST path=/ niks3-go-unit-tests> === CONT TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key niks3-go-unit-tests> 2026/08/27 09:16:06 INFO Received request for more parts method=POST path=/ niks3-go-unit-tests> === CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal niks3-go-unit-tests> 2026/08/27 09:16:06 INFO Received uploads request method=POST path=/ niks3-go-unit-tests> === CONT TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key niks3-go-unit-tests> 2026/08/27 09:16:06 INFO Received complete multipart upload request method=POST path=/ niks3-go-unit-tests> --- PASS: TestUploadHandlersRejectInvalidKeys (0.00s) niks3-go-unit-tests> --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info (0.00s) niks3-go-unit-tests> --- PASS: TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key (0.00s) niks3-go-unit-tests> --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal (0.00s) niks3-go-unit-tests> --- PASS: TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key (0.00s) niks3-go-unit-tests> === CONT TestProxyWriteTimeout/narinfo niks3-go-unit-tests> === CONT TestProxyWriteTimeout/unknown_size niks3-go-unit-tests> === CONT TestProxyWriteTimeout/10_GiB_nar niks3-go-unit-tests> === CONT TestProxyWriteTimeout/1_GiB_nar niks3-go-unit-tests> --- PASS: TestProxyWriteTimeout (0.06s) niks3-go-unit-tests> --- PASS: TestProxyWriteTimeout/narinfo (0.00s) niks3-go-unit-tests> --- PASS: TestProxyWriteTimeout/unknown_size (0.00s) niks3-go-unit-tests> --- PASS: TestProxyWriteTimeout/10_GiB_nar (0.00s) niks3-go-unit-tests> --- PASS: TestProxyWriteTimeout/1_GiB_nar (0.00s) niks3-go-unit-tests> === CONT TestIsValidUploadKey/narinfo niks3-go-unit-tests> === CONT TestIsValidUploadKey/realisation_plus_in_output niks3-go-unit-tests> === CONT TestIsValidUploadKey/unknown_type niks3-go-unit-tests> === CONT TestIsValidUploadKey/empty_key niks3-go-unit-tests> === CONT TestIsValidUploadKey/absolute niks3-go-unit-tests> === CONT TestIsValidUploadKey/traversal_nar niks3-go-unit-tests> === CONT TestIsValidUploadKey/traversal niks3-go-unit-tests> === CONT TestIsValidUploadKey/index.html niks3-go-unit-tests> === CONT TestIsValidUploadKey/nix-cache-info niks3-go-unit-tests> === CONT TestIsValidUploadKey/narinfo_key,_nar_type niks3-go-unit-tests> === CONT TestIsValidUploadKey/listing_key,_narinfo_type niks3-go-unit-tests> === CONT TestIsValidUploadKey/nar_key,_narinfo_type niks3-go-unit-tests> === CONT TestIsValidUploadKey/build_log_home-manager_file niks3-go-unit-tests> === CONT TestIsValidUploadKey/build_log_equals niks3-go-unit-tests> === CONT TestIsValidUploadKey/build_log_question_mark niks3-go-unit-tests> === CONT TestIsValidUploadKey/realisation niks3-go-unit-tests> === CONT TestIsValidUploadKey/build_log_plus_in_name niks3-go-unit-tests> === CONT TestIsValidUploadKey/nar_zst niks3-go-unit-tests> === CONT TestIsValidUploadKey/nar_plain niks3-go-unit-tests> === CONT TestIsValidUploadKey/listing niks3-go-unit-tests> === CONT TestIsValidUploadKey/nar_xz niks3-go-unit-tests> === CONT TestIsValidUploadKey/build_log niks3-go-unit-tests> --- PASS: TestIsValidUploadKey (0.07s) niks3-go-unit-tests> --- PASS: TestIsValidUploadKey/narinfo (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidUploadKey/realisation_plus_in_output (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidUploadKey/unknown_type (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidUploadKey/empty_key (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidUploadKey/absolute (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidUploadKey/traversal_nar (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidUploadKey/traversal (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidUploadKey/index.html (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidUploadKey/nix-cache-info (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidUploadKey/narinfo_key,_nar_type (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidUploadKey/listing_key,_narinfo_type (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidUploadKey/nar_key,_narinfo_type (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidUploadKey/build_log_home-manager_file (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidUploadKey/build_log_equals (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidUploadKey/build_log_question_mark (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidUploadKey/realisation (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidUploadKey/build_log_plus_in_name (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidUploadKey/nar_zst (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidUploadKey/nar_plain (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidUploadKey/listing (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidUploadKey/nar_xz (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidUploadKey/build_log (0.00s) niks3-go-unit-tests> === CONT TestUploadHandlersRejectOversizedBody/create_pending_closure niks3-go-unit-tests> 2026/08/27 09:16:06 INFO Received uploads request method=POST path=/ niks3-go-unit-tests> 2026/08/27 09:16:06 INFO Received complete multipart upload request method=POST path=/api/multipart/complete niks3-go-unit-tests> 2026/08/27 09:16:06 WARN CompleteMultipartUpload errored but object exists; treating as success error="The specified key does not exist." object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=YmM0MGU2MDUtNGUxNy00MWM1LWEyOTItODkzM2EwMjFhODUwLjU3ZWFhMWUzLTRlOTUtNGEyZC04NDEyLTFkNzYxY2VhNTFkM3gxNzg3ODIyMTY1NzYyNjc1Nzk4 niks3-go-unit-tests> --- PASS: TestReadProxyNarinfoAlreadyDecompressed (1.66s) niks3-go-unit-tests> === CONT TestUploadHandlersRejectOversizedBody/request_more_parts niks3-go-unit-tests> 2026/08/27 09:16:06 INFO Received request for more parts method=POST path=/ niks3-go-unit-tests> 2026/08/27 09:16:06 INFO Registered completed upload object_key=nar/jjjjjjjjjjjjjjjjjjjjjjjjjjjjjj0100000000000000000000.nar.zst niks3-go-unit-tests> 2026/08/27 09:16:06 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> --- PASS: TestPresignedUploadRegisteredBeforeCommit (2.06s) niks3-go-unit-tests> === CONT TestUploadHandlersRejectOversizedBody/complete_multipart niks3-go-unit-tests> 2026/08/27 09:16:06 INFO Received complete multipart upload request method=POST path=/ niks3-go-unit-tests> 2026/08/27 09:16:06 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=YmM0MGU2MDUtNGUxNy00MWM1LWEyOTItODkzM2EwMjFhODUwLjU3ZWFhMWUzLTRlOTUtNGEyZC04NDEyLTFkNzYxY2VhNTFkM3gxNzg3ODIyMTY1NzYyNjc1Nzk4 parts=1 niks3-go-unit-tests> --- PASS: TestCompleteMultipartUpload_ErrorButObjectExists (2.06s) niks3-go-unit-tests> === CONT TestIsValidCachePath/narinfo niks3-go-unit-tests> === CONT TestIsValidCachePath/traversal_parent niks3-go-unit-tests> === CONT TestIsValidCachePath/index.html niks3-go-unit-tests> === CONT TestIsValidCachePath/nix-cache-info niks3-go-unit-tests> === CONT TestIsValidCachePath/realisation niks3-go-unit-tests> === CONT TestIsValidCachePath/log niks3-go-unit-tests> === CONT TestIsValidCachePath/ls niks3-go-unit-tests> === CONT TestIsValidCachePath/nar_uncompressed niks3-go-unit-tests> === CONT TestIsValidCachePath/nar_bz2 niks3-go-unit-tests> === CONT TestIsValidCachePath/nar_xz niks3-go-unit-tests> === CONT TestIsValidCachePath/nar_zst niks3-go-unit-tests> === CONT TestIsValidCachePath/narinfo_all_nix_base32_chars niks3-go-unit-tests> === CONT TestIsValidCachePath/leading_slash niks3-go-unit-tests> === CONT TestIsValidCachePath/traversal_in_middle niks3-go-unit-tests> === CONT TestIsValidCachePath/empty niks3-go-unit-tests> === CONT TestIsValidCachePath/random_path niks3-go-unit-tests> === CONT TestIsValidCachePath/invalid_char_u niks3-go-unit-tests> === CONT TestIsValidCachePath/invalid_char_e niks3-go-unit-tests> === CONT TestIsValidCachePath/short_hash niks3-go-unit-tests> === CONT TestIsValidCachePath/wrong_extension niks3-go-unit-tests> --- PASS: TestIsValidCachePath (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidCachePath/narinfo (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidCachePath/traversal_parent (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidCachePath/index.html (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidCachePath/nix-cache-info (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidCachePath/realisation (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidCachePath/log (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidCachePath/ls (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidCachePath/nar_uncompressed (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidCachePath/nar_bz2 (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidCachePath/nar_xz (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidCachePath/nar_zst (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidCachePath/narinfo_all_nix_base32_chars (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidCachePath/leading_slash (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidCachePath/traversal_in_middle (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidCachePath/empty (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidCachePath/random_path (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidCachePath/invalid_char_u (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidCachePath/invalid_char_e (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidCachePath/short_hash (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidCachePath/wrong_extension (0.00s) niks3-go-unit-tests> === CONT TestParseSingleRange/none niks3-go-unit-tests> === CONT TestParseSingleRange/open-ended niks3-go-unit-tests> === CONT TestParseSingleRange/start_far_past_EOF niks3-go-unit-tests> === CONT TestParseSingleRange/start_past_EOF niks3-go-unit-tests> === CONT TestParseSingleRange/single_byte niks3-go-unit-tests> === CONT TestParseSingleRange/suffix_exceeds_size niks3-go-unit-tests> === CONT TestParseSingleRange/end_clamped_to_size niks3-go-unit-tests> === CONT TestParseSingleRange/malformed_both_empty niks3-go-unit-tests> === CONT TestParseSingleRange/closed niks3-go-unit-tests> === CONT TestParseSingleRange/malformed_end_before_start niks3-go-unit-tests> === CONT TestParseSingleRange/suffix niks3-go-unit-tests> === CONT TestParseSingleRange/multi-range_ignored niks3-go-unit-tests> === CONT TestParseSingleRange/unknown_unit niks3-go-unit-tests> === CONT TestParseSingleRange/malformed_no_dash niks3-go-unit-tests> === CONT TestClientErrorHandling/InvalidStorePath niks3-go-unit-tests> --- PASS: TestParseSingleRange (0.00s) niks3-go-unit-tests> --- PASS: TestParseSingleRange/none (0.00s) niks3-go-unit-tests> --- PASS: TestParseSingleRange/open-ended (0.00s) niks3-go-unit-tests> --- PASS: TestParseSingleRange/start_far_past_EOF (0.00s) niks3-go-unit-tests> --- PASS: TestParseSingleRange/start_past_EOF (0.00s) niks3-go-unit-tests> --- PASS: TestParseSingleRange/single_byte (0.00s) niks3-go-unit-tests> --- PASS: TestParseSingleRange/suffix_exceeds_size (0.00s) niks3-go-unit-tests> --- PASS: TestParseSingleRange/end_clamped_to_size (0.00s) niks3-go-unit-tests> --- PASS: TestParseSingleRange/malformed_both_empty (0.00s) niks3-go-unit-tests> --- PASS: TestParseSingleRange/closed (0.00s) niks3-go-unit-tests> --- PASS: TestParseSingleRange/malformed_end_before_start (0.00s) niks3-go-unit-tests> --- PASS: TestParseSingleRange/suffix (0.00s) niks3-go-unit-tests> --- PASS: TestParseSingleRange/multi-range_ignored (0.00s) niks3-go-unit-tests> --- PASS: TestParseSingleRange/unknown_unit (0.00s) niks3-go-unit-tests> --- PASS: TestParseSingleRange/malformed_no_dash (0.00s) niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:06.737508555Z","level":"ERROR","duration":"458.265µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(292)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:06.737560435Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"1c67bdd4-7308-4bda-ab1a-ddd71ec5e6b7","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket44/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(292)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:06.75552662Z","level":"ERROR","duration":"703.186µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(239)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:06.7556005Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"03725dc6-72db-4f5b-ba89-d8fd9aa1b7af","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket31/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(239)"} niks3-go-unit-tests> === CONT TestClientErrorHandling/ServerNotAvailable niks3-go-unit-tests> === CONT TestClientErrorHandling/InvalidAuthToken niks3-go-unit-tests> 2026/08/27 09:16:06 INFO Received complete multipart upload request method=POST path=/api/multipart/complete niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:06.816685319Z","level":"ERROR","duration":"172.322µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(280)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:06.8167598Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"cc156bd9-027e-496c-bee2-48a78dc50fb7","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket42/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(280)"} niks3-go-unit-tests> 2026-08-27 09:16:06.828 UTC [1201] ERROR: relation "goose_db_version" does not exist at character 36 niks3-go-unit-tests> 2026-08-27 09:16:06.828 UTC [1201] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC niks3-go-unit-tests> 2026/08/27 09:16:06 OK 20241026095416_initial_model.sql (9.38ms) niks3-go-unit-tests> 2026/08/27 09:16:06 OK 20251210153512_drop_unused_gin_index.sql (2.74ms) niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:06.852822429Z","level":"ERROR","duration":"298.843µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(235)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:06.85290231Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"db5c9503-8e03-492d-b3c8-fc87f4441076","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket30/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(235)"} niks3-go-unit-tests> 2026/08/27 09:16:06 OK 20251218171726_add_pins.sql (15.83ms) niks3-go-unit-tests> 2026/08/27 09:16:06 OK 20260628120000_add_object_size_and_stats.sql (5.4ms) niks3-go-unit-tests> 2026/08/27 09:16:06 goose: successfully migrated database to version: 20260628120000 niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:06.870118248Z","level":"ERROR","duration":"217.642µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(280)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:06.870189988Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"85235aa1-40c5-4a6e-bb14-fa4a79c259e7","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket44/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(280)"} niks3-go-unit-tests> 2026/08/27 09:16:06 OK 1_commit_pending_closure.sql (8.67ms) niks3-go-unit-tests> 2026/08/27 09:16:06 OK 2_object_stats_trigger.sql (6.02ms) niks3-go-unit-tests> 2026/08/27 09:16:06 goose: up to current file version: 2 niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:06.886951422Z","level":"ERROR","duration":"551.605µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(239)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:06.887033102Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"7a6fa682-3a47-42cf-9619-463e511ba1fe","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket45/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(239)"} niks3-go-unit-tests> 2026-08-27 09:16:06.891 UTC [1221] ERROR: relation "goose_db_version" does not exist at character 36 niks3-go-unit-tests> 2026-08-27 09:16:06.891 UTC [1221] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC niks3-go-unit-tests> 2026/08/27 09:16:06 WARN Request failed, retrying attempt=1 max_attempts=6 backoff=100ms error="Get \"http://localhost:19999/api/cache-config\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/cache-config niks3-go-unit-tests> 2026/08/27 09:16:06 OK 20241026095416_initial_model.sql (10.8ms) niks3-go-unit-tests> 2026/08/27 09:16:06 OK 20251210153512_drop_unused_gin_index.sql (1.65ms) niks3-go-unit-tests> 2026/08/27 09:16:06 OK 20251218171726_add_pins.sql (4.17ms) niks3-go-unit-tests> 2026/08/27 09:16:06 OK 20260628120000_add_object_size_and_stats.sql (3ms) niks3-go-unit-tests> 2026/08/27 09:16:06 goose: successfully migrated database to version: 20260628120000 niks3-go-unit-tests> 2026/08/27 09:16:06 OK 1_commit_pending_closure.sql (2.97ms) niks3-go-unit-tests> 2026/08/27 09:16:06 OK 2_object_stats_trigger.sql (889.29µs) niks3-go-unit-tests> 2026/08/27 09:16:06 goose: up to current file version: 2 niks3-go-unit-tests> 2026/08/27 09:16:07 WARN Request failed, retrying attempt=2 max_attempts=6 backoff=215.698386ms error="Get \"http://localhost:19999/api/cache-config\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/cache-config niks3-go-unit-tests> 2026/08/27 09:16:07 WARN Request failed, retrying attempt=3 max_attempts=6 backoff=394.73896ms error="Get \"http://localhost:19999/api/cache-config\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/cache-config niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:07.415607497Z","level":"ERROR","duration":"202.322µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(261)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:07.415665738Z","level":"ERROR","duration":"155.562µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(308)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:07.415691538Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"23886ac7-f12b-47ce-98a7-922553904a39","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket44/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(261)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:07.415724358Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"da9fd9f5-9cf8-4055-936d-15f39b556e9b","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket42/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(308)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:07.41919609Z","level":"ERROR","duration":"3.800795ms","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(285)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:07.419273571Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"3256ebbd-fac3-4758-92f0-b6b330998f74","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket31/","status_code":503,"duration_ms":3,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(285)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:07.419516253Z","level":"ERROR","duration":"4.097718ms","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(241)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:07.419566173Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"b0f3584f-57cf-4413-bdd6-fe95614b6943","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket43/","status_code":503,"duration_ms":4,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(241)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:07.432926435Z","level":"ERROR","duration":"17.49974ms","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(302)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:07.432981856Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"a50cd4b7-aa84-439e-b497-fba2a8549ce3","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket45/","status_code":503,"duration_ms":17,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(302)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:07.434393069Z","level":"ERROR","duration":"504.831018ms","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(288)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:07.434439929Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"d00d01f5-4fe4-4acf-84ac-e11f9cea0834","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket46/","status_code":503,"duration_ms":504,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(288)"} niks3-go-unit-tests> 2026/08/27 09:16:07 INFO Received complete multipart upload request method=POST path=/api/multipart/complete niks3-go-unit-tests> 2026/08/27 09:16:07 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst upload_id=YmM0MGU2MDUtNGUxNy00MWM1LWEyOTItODkzM2EwMjFhODUwLjk2MWVjZTVjLTM0ZTAtNGRjOC1hMTJjLWNkOTZhM2JjMjU4NHgxNzg3ODIyMTY1MDEzNzc0OTQ4 parts=10 niks3-go-unit-tests> 2026/08/27 09:16:07 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete niks3-go-unit-tests> 2026/08/27 09:16:07 INFO Completed upload id=1 niks3-go-unit-tests> 2026/08/27 09:16:07 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> 2026/08/27 09:16:07 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> 2026/08/27 09:16:07 INFO Object in database but missing from S3 key=dadb44fdadb44fdadb44fdadb44f0000.narinfo niks3-go-unit-tests> 2026/08/27 09:16:07 WARN Found objects in DB but missing from S3, will re-upload count=1 niks3-go-unit-tests> --- PASS: TestService_verifyS3Integrity (2.86s) niks3-go-unit-tests> === CONT TestCacheConfigHandler/full_config,_no_issuer niks3-go-unit-tests> === CONT TestCacheConfigHandler/no_signing_keys niks3-go-unit-tests> === CONT TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator niks3-go-unit-tests> === CONT TestCacheConfigHandler/no_cache_url_configured niks3-go-unit-tests> --- PASS: TestCacheConfigHandler (0.00s) niks3-go-unit-tests> --- PASS: TestCacheConfigHandler/full_config,_no_issuer (0.00s) niks3-go-unit-tests> --- PASS: TestCacheConfigHandler/no_signing_keys (0.00s) niks3-go-unit-tests> --- PASS: TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator (0.00s) niks3-go-unit-tests> --- PASS: TestCacheConfigHandler/no_cache_url_configured (0.00s) niks3-go-unit-tests> === CONT TestServerTLSConfig/no_client_CA niks3-go-unit-tests> === CONT TestServerTLSConfig/not_a_PEM_file niks3-go-unit-tests> === CONT TestServerTLSConfig/missing_CA_file niks3-go-unit-tests> --- PASS: TestServerTLSConfig (0.00s) niks3-go-unit-tests> --- PASS: TestServerTLSConfig/no_client_CA (0.00s) niks3-go-unit-tests> --- PASS: TestServerTLSConfig/not_a_PEM_file (0.00s) niks3-go-unit-tests> --- PASS: TestServerTLSConfig/missing_CA_file (0.00s) niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:07.579627117Z","level":"ERROR","duration":"698.146µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(286)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:07.579717178Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"ecd72078-b832-4b7c-b617-4703040e4cf7","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket30/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(286)"} niks3-go-unit-tests> --- PASS: TestUploadHandlersRejectOversizedBody (0.15s) niks3-go-unit-tests> --- PASS: TestUploadHandlersRejectOversizedBody/request_more_parts (0.05s) niks3-go-unit-tests> --- PASS: TestUploadHandlersRejectOversizedBody/complete_multipart (0.07s) niks3-go-unit-tests> --- PASS: TestUploadHandlersRejectOversizedBody/create_pending_closure (0.89s) niks3-go-unit-tests> 2026/08/27 09:16:07 WARN Request failed, retrying attempt=4 max_attempts=6 backoff=735.413539ms error="Get \"http://localhost:19999/api/cache-config\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/cache-config niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:07.619356101Z","level":"ERROR","duration":"196.242µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(279)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:07.619430521Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"028a60d3-f8e3-41d5-a320-f9bad6ae6c4f","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket46/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(279)"} niks3-go-unit-tests> 2026/08/27 09:16:07 INFO Received complete multipart upload request method=POST path=/api/multipart/complete niks3-go-unit-tests> 2026/08/27 09:16:07 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0100000000000000000000.nar.zst upload_id=YmM0MGU2MDUtNGUxNy00MWM1LWEyOTItODkzM2EwMjFhODUwLmFkZGUyNmMyLTU3ZmItNDk1Zi04NzEzLWNiZGMxMDEyNzZlY3gxNzg3ODIyMTY1Nzg0MTMyNjM0 parts=12 niks3-go-unit-tests> --- PASS: TestRedundantMultipartUpload (3.00s) niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:07.698653886Z","level":"ERROR","duration":"210.642µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(302)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:07.698735847Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"70612c76-91de-4fca-838d-1f7195550491","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket42/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(302)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:07.786911393Z","level":"ERROR","duration":"323.083µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(302)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:07.786997354Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"5602c307-788b-4f61-a53e-41d840574490","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket45/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(302)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:07.831781124Z","level":"ERROR","duration":"222.362µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(282)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:07.831858564Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"6e2406dd-c3be-438c-ac48-5b0893f4c3f0","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket46/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(282)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:08.057122545Z","level":"ERROR","duration":"170.582µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(314)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:08.057197266Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"5b50fcc1-6784-4c47-9316-98d988f7f254","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket43/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(314)"} niks3-go-unit-tests> 2026/08/27 09:16:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=2001 objects-failed-to-delete=0 niks3-go-unit-tests> 2026/08/27 09:16:08 INFO Vacuumed table table=pending_closures niks3-go-unit-tests> 2026/08/27 09:16:08 INFO Vacuumed table table=pending_objects niks3-go-unit-tests> 2026/08/27 09:16:08 INFO Vacuumed table table=multipart_uploads niks3-go-unit-tests> 2026/08/27 09:16:08 INFO Vacuumed table table=closures niks3-go-unit-tests> 2026/08/27 09:16:08 INFO Vacuumed table table=objects niks3-go-unit-tests> 2026/08/27 09:16:08 INFO Received complete multipart upload request method=POST path=/api/multipart/complete niks3-go-unit-tests> 2026/08/27 09:16:08 INFO Completed multipart upload object_key=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhh0100000000000000000000.nar.zst upload_id=YmM0MGU2MDUtNGUxNy00MWM1LWEyOTItODkzM2EwMjFhODUwLjYwMWE3MTQzLWFlMGEtNDIxMi1hMmYzLThiOTk2NjYyMGI4N3gxNzg3ODIyMTY1NjcwNzYyMzk3 parts=12 niks3-go-unit-tests> 2026/08/27 09:16:08 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> --- PASS: TestCompletedNarNotReofferedAcrossClosures (3.49s) niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:08.168018359Z","level":"ERROR","duration":"277.642µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(280)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:08.16809478Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"900c3360-15b0-4b83-9dce-cbcde3026ba9","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket31/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(280)"} niks3-go-unit-tests> 2026/08/27 09:16:08 INFO Received complete multipart upload request method=POST path=/api/multipart/complete niks3-go-unit-tests> 2026/08/27 09:16:08 INFO Completed multipart upload object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst upload_id=YmM0MGU2MDUtNGUxNy00MWM1LWEyOTItODkzM2EwMjFhODUwLjFhMjhlMDljLTJhMWEtNGQwYi04ZTIzLWNhYzg0N2NjMWE3NngxNzg3ODIyMTY1MTI4MjUwNjc1 parts=10 niks3-go-unit-tests> 2026/08/27 09:16:08 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete niks3-go-unit-tests> 2026/08/27 09:16:08 INFO Completed upload id=1 niks3-go-unit-tests> 2026/08/27 09:16:08 INFO Received get closure request method=GET path=/api/closures/00000000000000000000000000000000 niks3-go-unit-tests> 2026/08/27 09:16:08 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> 2026/08/27 09:16:08 INFO Starting cleanup of old closures method=DELETE path=/api/closures niks3-go-unit-tests> 2026/08/27 09:16:08 INFO Aborted multipart uploads count=0 niks3-go-unit-tests> 2026/08/27 09:16:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=2 objects-deleted-after-grace-period=0 objects-failed-to-delete=0 niks3-go-unit-tests> 2026/08/27 09:16:08 INFO Vacuumed table table=pending_closures niks3-go-unit-tests> 2026/08/27 09:16:08 INFO Vacuumed table table=pending_objects niks3-go-unit-tests> 2026/08/27 09:16:08 INFO Vacuumed table table=multipart_uploads niks3-go-unit-tests> 2026/08/27 09:16:08 INFO Vacuumed table table=closures niks3-go-unit-tests> 2026/08/27 09:16:08 INFO Vacuumed table table=objects niks3-go-unit-tests> 2026/08/27 09:16:08 INFO Received get closure request method=GET path=/api/closures/00000000000000000000000000000000 niks3-go-unit-tests> --- PASS: TestService_createPendingClosureHandler (3.59s) niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:08.323356Z","level":"ERROR","duration":"190.082µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(280)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:08.323437061Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"23b5fdb2-4dfa-40d0-98bd-dd990888316d","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket30/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(280)"} niks3-go-unit-tests> 2026/08/27 09:16:08 WARN Request failed, retrying attempt=5 max_attempts=6 backoff=1.759352772s error="Get \"http://localhost:19999/api/cache-config\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/cache-config niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:08.402759566Z","level":"ERROR","duration":"192.762µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(280)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:08.402844287Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"6c020b61-659f-417d-adc3-1fe1b0de0ffb","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket44/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(280)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:08.425703796Z","level":"ERROR","duration":"267.982µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(241)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:08.425788037Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"b591d162-0861-452d-a03d-0822a174dd99","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket45/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(241)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:08.452227959Z","level":"ERROR","duration":"237.242µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(239)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:08.45230608Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"0aaa2c79-d467-46f7-be06-33201392f649","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket30/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(239)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:08.465985785Z","level":"ERROR","duration":"214.662µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(302)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:08.466068565Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"0b140d4e-5c83-48b7-9ec5-5e70427729d7","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket46/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(302)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:08.488604191Z","level":"ERROR","duration":"284.862µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(279)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:08.488679092Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"25b068ee-2b74-4ed7-ac3f-b8a80047dbee","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket46/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(279)"} niks3-go-unit-tests> === NAME TestOrphanedObjectsGCStressTest niks3-go-unit-tests> orphaned_objects_gc_test.go:425: Created 10 active closures, 5 to-delete closures, 20 orphaned chains niks3-go-unit-tests> orphaned_objects_gc_test.go:446: Marked 210 objects for deletion niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:08.670089191Z","level":"ERROR","duration":"211.022µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(279)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:08.670183932Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"9c690801-4a19-41b0-87fa-d9b7f6fae68b","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket42/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(279)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:08.674149689Z","level":"ERROR","duration":"219.242µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(235)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:08.674234329Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"1f2fe1f0-ad4b-422c-a978-895410b6d46f","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket45/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(235)"} niks3-go-unit-tests> 2026/08/27 09:16:08 INFO Garbage collection progress phase="" failed_uploads_deleted=0 old_closures_deleted=1 objects_marked=3 objects_deleted=2001 objects_failed=0 niks3-go-unit-tests> === NAME TestPinProtectsFromGC niks3-go-unit-tests> client_integration_test.go:709: Pin successfully protected closure from garbage collection niks3-go-unit-tests> --- PASS: TestPinProtectsFromGC (3.87s) niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:08.915022952Z","level":"ERROR","duration":"334.003µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(279)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:08.915109553Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"6c1b3548-1499-4a79-a79a-037ab3555e6f","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket31/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(279)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:08.974327394Z","level":"ERROR","duration":"15.29944ms","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(302)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:08.974397675Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"c7fb1e48-0869-43dd-8a51-fc6408367a62","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket43/","status_code":503,"duration_ms":15,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(302)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:09.066600158Z","level":"ERROR","duration":"285.863µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(302)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:09.066698659Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"d4272205-7a8f-480c-9e3d-7c1878222525","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket46/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(302)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:09.184797179Z","level":"ERROR","duration":"7.303466ms","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(286)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:09.18489382Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"c9a071cb-56a3-48f0-b237-1b58b2948e05","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket30/","status_code":503,"duration_ms":7,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(286)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:09.191842124Z","level":"ERROR","duration":"175.362µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(286)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:09.191894284Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"0ce06b54-a45a-4fae-9962-cbbeded57358","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket45/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(286)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:09.312929532Z","level":"ERROR","duration":"311.163µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(302)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:09.313006472Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"5d397c8c-47b3-4a86-a661-dad86ada225d","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket44/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(302)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:09.331923265Z","level":"ERROR","duration":"171.921µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(286)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:09.331984046Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"cb3c25c4-5b77-496c-a129-b54dda8be754","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket45/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(286)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:09.367863954Z","level":"ERROR","duration":"211.882µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(302)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:09.367943475Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"ea8a617e-4464-4eae-b02f-6ec80aeb6cc6","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket43/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(302)"} niks3-go-unit-tests> === NAME TestCacheStatsHandler niks3-go-unit-tests> cache_stats_test.go:17: unexpected error: bucket creation concurrency limit reached; retry later niks3-go-unit-tests> niks3-go-unit-tests> --- FAIL: TestCacheStatsHandler (4.22s) niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:09.483124708Z","level":"ERROR","duration":"267.622µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(241)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:09.483212509Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"f4269eb4-cb29-4eb3-8ea8-64ce699d7c01","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket42/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(241)"} niks3-go-unit-tests> === NAME TestOrphanedObjectsGCStressTest niks3-go-unit-tests> orphaned_objects_gc_test.go:509: Stress test completed successfully: niks3-go-unit-tests> orphaned_objects_gc_test.go:510: - Active objects preserved: 20 niks3-go-unit-tests> orphaned_objects_gc_test.go:511: - Objects deleted: 210 niks3-go-unit-tests> orphaned_objects_gc_test.go:512: - Total GC'd: 210 niks3-go-unit-tests> --- PASS: TestOrphanedObjectsGCStressTest (4.41s) niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:09.64619226Z","level":"ERROR","duration":"317.083µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(286)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:09.6462727Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"b04d3ab5-059f-4f15-bdff-2d40a5da1209","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket46/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(286)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:09.648108517Z","level":"ERROR","duration":"262.002µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(239)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:09.648180898Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"0fec1929-f965-4ae6-8950-9b8825d53ee8","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket42/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(239)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:09.833797376Z","level":"ERROR","duration":"169.761µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(286)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:09.833857996Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"db20a381-52c5-47be-bef8-7ba6a52614cc","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket43/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(286)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:09.89880253Z","level":"ERROR","duration":"234.702µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(279)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:09.898888351Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"0fdba4d7-e1b4-4983-8b2a-14a112bf6411","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket44/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(279)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:09.904110559Z","level":"ERROR","duration":"240.442µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(239)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:09.90419352Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"52abd09d-66a6-43a6-aaa1-3fc5bb2833f7","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket45/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(239)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:09.994386665Z","level":"ERROR","duration":"282.463µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(239)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:09.994459845Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"777420fb-520d-450c-8669-a4e77221f062","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket44/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(239)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:10.092598523Z","level":"ERROR","duration":"232.422µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(241)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:10.092687024Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"a686bc62-e5a5-478c-80cc-ff4eaddcb302","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket30/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(241)"} niks3-go-unit-tests> 2026/08/27 09:16:10 WARN Failed to fetch cache config, uploading without size limit error="fetching cache config: sending request: request failed after retries: Get \"http://localhost:19999/api/cache-config\": dial tcp [::1]:19999: connect: connection refused" niks3-go-unit-tests> 2026/08/27 09:16:10 WARN Request failed, retrying attempt=1 max_attempts=6 backoff=100ms error="Post \"http://localhost:19999/api/pending_closures\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/pending_closures niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:10.158803309Z","level":"ERROR","duration":"217.482µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(241)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:10.158887829Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"beae9bef-eb3c-4f21-94f0-9344acee5c3a","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket44/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(241)"} niks3-go-unit-tests> 2026/08/27 09:16:10 WARN Request failed, retrying attempt=2 max_attempts=6 backoff=186.62194ms error="Post \"http://localhost:19999/api/pending_closures\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/pending_closures niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:10.305866154Z","level":"ERROR","duration":"328.003µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(302)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:10.305957235Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"b707144e-726e-4bec-b7e5-81d178b2265c","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket46/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(302)"} niks3-go-unit-tests> === NAME TestReadProxyNarinfo niks3-go-unit-tests> proxy_test.go:140: unexpected error: bucket creation concurrency limit reached; retry later niks3-go-unit-tests> niks3-go-unit-tests> --- FAIL: TestReadProxyNarinfo (5.33s) niks3-go-unit-tests> 2026/08/27 09:16:10 WARN Request failed, retrying attempt=3 max_attempts=6 backoff=411.156204ms error="Post \"http://localhost:19999/api/pending_closures\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/pending_closures niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:10.454960398Z","level":"ERROR","duration":"229.622µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(279)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:10.455047558Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"0928e589-3a3f-4aa2-82d5-36a1d6e24f8c","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket46/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(279)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:10.514194619Z","level":"ERROR","duration":"216.822µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(239)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:10.51427652Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"9df8122a-62f9-4d46-9c1f-324a98085dcc","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket45/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(239)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:10.563287768Z","level":"ERROR","duration":"213.462µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(286)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:10.563371209Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"c9802a8f-9604-4a55-8902-1e4c315b43f6","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket42/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(286)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:10.636531598Z","level":"ERROR","duration":"319.963µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(239)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:10.636613479Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"77b63b89-c23b-4431-a11b-324f5977b4ee","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket43/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(239)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:10.841010568Z","level":"ERROR","duration":"214.222µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(239)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:10.841088889Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"1896680b-dac5-4804-a6b1-a2a57fdcb786","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket42/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(239)"} niks3-go-unit-tests> 2026/08/27 09:16:10 WARN Request failed, retrying attempt=4 max_attempts=6 backoff=855.655391ms error="Post \"http://localhost:19999/api/pending_closures\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/pending_closures niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:10.905382677Z","level":"ERROR","duration":"229.222µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(302)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:10.905470318Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"59384058-50e9-46cd-b7aa-f370f90c4f98","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket45/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(302)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:10.944980899Z","level":"ERROR","duration":"293.602µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(286)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:10.9450708Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"a7ea3081-b029-4a81-ab48-d2c5aac3e3ee","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket44/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(286)"} niks3-go-unit-tests> === NAME TestService_AuthMiddleware_MTLSProxyHeader niks3-go-unit-tests> auth_test.go:55: unexpected error: bucket creation concurrency limit reached; retry later niks3-go-unit-tests> niks3-go-unit-tests> --- FAIL: TestService_AuthMiddleware_MTLSProxyHeader (4.51s) niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:11.123811595Z","level":"ERROR","duration":"162.461µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(302)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:11.123864796Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"e176a58b-056c-4307-9975-f3cc7b6edf7d","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket46/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(302)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:11.209917923Z","level":"ERROR","duration":"227.562µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownbucket creation concurrency limit reached; retry later\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(286)"} niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:11.210006244Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"30d6d08c-88aa-48c5-92a7-36a04890e109","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket43/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(286)"} niks3-go-unit-tests> === NAME TestClientErrorHandling/InvalidStorePath niks3-go-unit-tests> client_errors_test.go:18: unexpected error: bucket creation concurrency limit reached; retry later niks3-go-unit-tests> niks3-go-unit-tests> 2026/08/27 09:16:11 INFO Created nix-cache-info in bucket bucket=bucket35 niks3-go-unit-tests> === NAME TestService_NativeMTLS niks3-go-unit-tests> mtls_test.go:21: unexpected error: bucket creation concurrency limit reached; retry later niks3-go-unit-tests> niks3-go-unit-tests> --- FAIL: TestService_NativeMTLS (4.99s) niks3-go-unit-tests> 2026/08/27 09:16:11 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> 2026/08/27 09:16:11 INFO Created nix-cache-info in bucket bucket=bucket27 niks3-go-unit-tests> === RUN TestService_AuthMiddleware_OIDC/valid_OIDC_token niks3-go-unit-tests> === PAUSE TestService_AuthMiddleware_OIDC/valid_OIDC_token niks3-go-unit-tests> === RUN TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected niks3-go-unit-tests> === PAUSE TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected niks3-go-unit-tests> === RUN TestService_AuthMiddleware_OIDC/malformed_token_rejected niks3-go-unit-tests> === PAUSE TestService_AuthMiddleware_OIDC/malformed_token_rejected niks3-go-unit-tests> === RUN TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured niks3-go-unit-tests> === PAUSE TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured niks3-go-unit-tests> === CONT TestService_AuthMiddleware_OIDC/valid_OIDC_token niks3-go-unit-tests> === CONT TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected niks3-go-unit-tests> === CONT TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured niks3-go-unit-tests> === CONT TestService_AuthMiddleware_OIDC/malformed_token_rejected niks3-go-unit-tests> 2026/08/27 09:16:11 WARN Authentication failed token_preview=not-a-valid-jwt token_length=15 oidc_error="no provider could verify the token (signature or issuer mismatch)" oidc_provider="" tried_providers=[test] niks3-go-unit-tests> --- PASS: TestReadProxyNarStreaming (6.33s) niks3-go-unit-tests> 2026/08/27 09:16:11 INFO OIDC auth successful provider=test niks3-go-unit-tests> 2026/08/27 09:16:11 WARN Authentication failed token_preview=eyJhbGciOi...8zTNbNoISQ token_length=702 oidc_error="bound claims validation failed: claim \"repository_owner\" value [otherorg] not in allowed patterns [myorg]" oidc_provider=test tried_providers=[test] niks3-go-unit-tests> --- PASS: TestService_AuthMiddleware_OIDC (5.67s) niks3-go-unit-tests> --- PASS: TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured (0.00s) niks3-go-unit-tests> --- PASS: TestService_AuthMiddleware_OIDC/malformed_token_rejected (0.00s) niks3-go-unit-tests> --- PASS: TestService_AuthMiddleware_OIDC/valid_OIDC_token (0.01s) niks3-go-unit-tests> --- PASS: TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected (0.01s) niks3-go-unit-tests> 2026/08/27 09:16:11 WARN Rate limiter enabled after throttle name=s3-test rate=5 niks3-go-unit-tests> 2026/08/27 09:16:11 WARN S3 rate limit hit operation="complete multipart upload" error="Please reduce your request rate." niks3-go-unit-tests> === NAME TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle niks3-go-unit-tests> throttle_test.go:213: Proxy stats: total=15, throttled=10, completeMultipart=10 niks3-go-unit-tests> 2026/08/27 09:16:11 INFO Created nix-cache-info in bucket bucket=bucket42 niks3-go-unit-tests> throttle_test.go:215: Rate limiter: enabled=true, rate=5.00 niks3-go-unit-tests> --- PASS: TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle (6.63s) niks3-go-unit-tests> === NAME TestClientIntegration niks3-go-unit-tests> client_integration_test.go:276: Created store path: /build/TestClientIntegration2268468003/002/store/hj3kdf33rcjklcwixlbz023l9x7r0mnw-test-file.txt niks3-go-unit-tests> === NAME TestNARDeduplicationMetadataUploadBug niks3-go-unit-tests> metadata_upload_test.go:48: First store path: /build/TestNARDeduplicationMetadataUploadBug778154564/001/store/9ki0kjxd9p1vd0hrm4vh2hq76lc6xvfg-file1.txt niks3-go-unit-tests> --- PASS: TestReadProxyDisabled (6.70s) niks3-go-unit-tests> --- PASS: TestObjectStatsTrigger (6.28s) niks3-go-unit-tests> 2026/08/27 09:16:11 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted,O=Other" niks3-go-unit-tests> 2026/08/27 09:16:11 WARN mTLS auth: bound subjects configured but subject DN unavailable niks3-go-unit-tests> 2026/08/27 09:16:11 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted" niks3-go-unit-tests> --- PASS: TestService_AuthMiddleware_MTLSBoundSubjects (5.11s) niks3-go-unit-tests> 2026/08/27 09:16:11 WARN Failed to fetch cache config, uploading without size limit error="fetching cache config: server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/08/27 09:16:11 WARN Failed to fetch cache config, uploading without size limit error="fetching cache config: server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/08/27 09:16:11 INFO Received cleanup request method=DELETE path=/api/pending_closures niks3-go-unit-tests> === NAME TestClientWithDependencies niks3-go-unit-tests> client_integration_test.go:593: Built derivation: /build/TestClientWithDependencies1449288560/001/store/8gd2bflh28169h5j5bl8wmhp1kxw9qwx-test-script niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:11.435309084Z","level":"ERROR","message":"Scanner cycle completed without a durable data usage snapshot","event":"scanner_persist_state","component":"scanner","subsystem":"runtime","cycle":7,"outcome":"Current","state":"usage_not_durable","target":"rustfs::scanner","filename":"crates/scanner/src/scanner.rs","line_number":2844,"threadName":"rustfs-worker","threadId":"ThreadId(279)"} niks3-go-unit-tests> 2026/08/27 09:16:11 INFO Aborted multipart uploads count=1 niks3-go-unit-tests> --- PASS: TestMultipartCleanup (5.49s) niks3-go-unit-tests> 2026/08/27 09:16:11 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> 2026/08/27 09:16:11 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> 2026/08/27 09:16:11 INFO Uploading 1 paths to 127.0.0.1 (0 already cached) niks3-go-unit-tests> 2026/08/27 09:16:11 INFO Uploading hj3kdf33rcjklcwixlbz023l9x7r0mnw-test-file.txt (152B) niks3-go-unit-tests> 2026/08/27 09:16:11 INFO Uploading 1 paths to 127.0.0.1 (0 already cached) niks3-go-unit-tests> 2026/08/27 09:16:11 INFO Uploading 9ki0kjxd9p1vd0hrm4vh2hq76lc6xvfg-file1.txt (160B) niks3-go-unit-tests> 2026/08/27 09:16:11 WARN Failed to register uploaded object key=nar/11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1.nar.zst error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/08/27 09:16:11 WARN Failed to register uploaded object key=hj3kdf33rcjklcwixlbz023l9x7r0mnw.ls error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/08/27 09:16:11 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign niks3-go-unit-tests> 2026/08/27 09:16:11 WARN Failed to register uploaded object key=nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/08/27 09:16:11 INFO Signed narinfos id=1 count=1 niks3-go-unit-tests> 2026/08/27 09:16:11 INFO Uploading 1 narinfos niks3-go-unit-tests> 2026/08/27 09:16:11 WARN Failed to register uploaded object key=9ki0kjxd9p1vd0hrm4vh2hq76lc6xvfg.ls error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/08/27 09:16:11 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign niks3-go-unit-tests> 2026/08/27 09:16:11 INFO Signed narinfos id=1 count=1 niks3-go-unit-tests> 2026/08/27 09:16:11 INFO Uploading 1 narinfos niks3-go-unit-tests> === NAME TestClientWithDependencies niks3-go-unit-tests> client_integration_test.go:595: Found 1 dependencies (including self) niks3-go-unit-tests> 2026/08/27 09:16:11 WARN Failed to fetch cache config, uploading without size limit error="fetching cache config: server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/08/27 09:16:11 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> 2026/08/27 09:16:11 INFO Uploading 1 paths to 127.0.0.1 (0 already cached) niks3-go-unit-tests> 2026/08/27 09:16:11 INFO Uploading 8gd2bflh28169h5j5bl8wmhp1kxw9qwx-test-script (136B) niks3-go-unit-tests> 2026/08/27 09:16:11 WARN Request failed, retrying attempt=5 max_attempts=6 backoff=1.645557306s error="Post \"http://localhost:19999/api/pending_closures\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/pending_closures niks3-go-unit-tests> === NAME TestClientErrorHandling/InvalidAuthToken niks3-go-unit-tests> client_errors_test.go:44: unexpected error: bucket creation concurrency limit reached; retry later niks3-go-unit-tests> niks3-go-unit-tests> 2026/08/27 09:16:11 WARN Failed to register uploaded object key=9ki0kjxd9p1vd0hrm4vh2hq76lc6xvfg.narinfo error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/08/27 09:16:11 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete niks3-go-unit-tests> 2026/08/27 09:16:11 WARN Failed to register uploaded object key=nar/1s7yghixzpinyqm7g59kyn3sav81pmnsdi5v63m6x4r2vacy907j.nar.zst error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/08/27 09:16:11 WARN Failed to register uploaded object key=8gd2bflh28169h5j5bl8wmhp1kxw9qwx.ls error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/08/27 09:16:11 INFO Completed upload id=1 niks3-go-unit-tests> 2026/08/27 09:16:11 WARN Failed to register uploaded object key=log/100485rdj93j7c031akg2mzphcd2rdqr-test-script.drv error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/08/27 09:16:11 INFO Upload complete. (538ms) niks3-go-unit-tests> 2026/08/27 09:16:11 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign niks3-go-unit-tests> 2026/08/27 09:16:11 INFO Signed narinfos id=1 count=1 niks3-go-unit-tests> 2026/08/27 09:16:11 INFO Uploading 1 narinfos niks3-go-unit-tests> === NAME TestNARDeduplicationMetadataUploadBug niks3-go-unit-tests> metadata_upload_test.go:54: Retrieved narinfo from S3: niks3-go-unit-tests> StorePath: /build/TestNARDeduplicationMetadataUploadBug778154564/001/store/9ki0kjxd9p1vd0hrm4vh2hq76lc6xvfg-file1.txt niks3-go-unit-tests> URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst niks3-go-unit-tests> Compression: zstd niks3-go-unit-tests> NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf niks3-go-unit-tests> NarSize: 160 niks3-go-unit-tests> References: niks3-go-unit-tests> CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf niks3-go-unit-tests> 2026/08/27 09:16:11 WARN Failed to register uploaded object key=8gd2bflh28169h5j5bl8wmhp1kxw9qwx.narinfo error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/08/27 09:16:11 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete niks3-go-unit-tests> metadata_upload_test.go:55: Retrieved .ls file from S3 (compressed size: 77 bytes) niks3-go-unit-tests> metadata_upload_test.go:55: Decompressed .ls content (64 bytes): niks3-go-unit-tests> {"version":1,"root":{"type":"regular","size":44,"narOffset":96}} niks3-go-unit-tests> 2026/08/27 09:16:11 INFO Completed upload id=1 niks3-go-unit-tests> 2026/08/27 09:16:11 INFO Upload complete. (434ms) niks3-go-unit-tests> === NAME TestClientWithDependencies niks3-go-unit-tests> client_integration_test.go:597: Skipping nix copy test - isolated store (/build/TestClientWithDependencies1449288560/001/store) requires matching store prefix niks3-go-unit-tests> --- PASS: TestClientWithDependencies (5.65s) niks3-go-unit-tests> === NAME TestNARDeduplicationMetadataUploadBug niks3-go-unit-tests> metadata_upload_test.go:64: Second store path (same content): /build/TestNARDeduplicationMetadataUploadBug778154564/001/store/27hah63f8adrgw42cvj3lvks84y845v8-file2.txt niks3-go-unit-tests> 2026/08/27 09:16:12 WARN Failed to fetch cache config, uploading without size limit error="fetching cache config: server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/08/27 09:16:12 WARN Failed to register uploaded object key=hj3kdf33rcjklcwixlbz023l9x7r0mnw.narinfo error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/08/27 09:16:12 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete niks3-go-unit-tests> 2026/08/27 09:16:12 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> 2026/08/27 09:16:12 INFO Completed upload id=1 niks3-go-unit-tests> 2026/08/27 09:16:12 INFO Upload complete. (693ms) niks3-go-unit-tests> === NAME TestClientIntegration niks3-go-unit-tests> client_integration_test.go:292: Retrieved narinfo from S3: niks3-go-unit-tests> StorePath: /build/TestClientIntegration2268468003/002/store/hj3kdf33rcjklcwixlbz023l9x7r0mnw-test-file.txt niks3-go-unit-tests> URL: nar/11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1.nar.zst niks3-go-unit-tests> Compression: zstd niks3-go-unit-tests> NarHash: sha256:11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1 niks3-go-unit-tests> NarSize: 152 niks3-go-unit-tests> References: niks3-go-unit-tests> CA: fixed:r:sha256:11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1 niks3-go-unit-tests> 2026/08/27 09:16:12 INFO Uploading 0 paths to 127.0.0.1 (1 already cached) niks3-go-unit-tests> client_integration_test.go:293: Retrieved .ls file from S3 (compressed size: 77 bytes) niks3-go-unit-tests> client_integration_test.go:293: Decompressed .ls content (64 bytes): niks3-go-unit-tests> {"version":1,"root":{"type":"regular","size":39,"narOffset":96}} niks3-go-unit-tests> client_integration_test.go:296: Testing garbage collection... niks3-go-unit-tests> 2026/08/27 09:16:12 WARN Failed to register uploaded object key=27hah63f8adrgw42cvj3lvks84y845v8.ls error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/08/27 09:16:12 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign niks3-go-unit-tests> 2026/08/27 09:16:12 INFO Signed narinfos id=2 count=1 niks3-go-unit-tests> 2026/08/27 09:16:12 INFO Uploading 1 narinfos niks3-go-unit-tests> 2026/08/27 09:16:12 WARN Failed to register uploaded object key=27hah63f8adrgw42cvj3lvks84y845v8.narinfo error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/08/27 09:16:12 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete niks3-go-unit-tests> 2026/08/27 09:16:12 INFO Completed upload id=2 niks3-go-unit-tests> 2026/08/27 09:16:12 INFO Upload complete. (82ms) niks3-go-unit-tests> === NAME TestNARDeduplicationMetadataUploadBug niks3-go-unit-tests> metadata_upload_test.go:76: Retrieved narinfo from S3: niks3-go-unit-tests> StorePath: /build/TestNARDeduplicationMetadataUploadBug778154564/001/store/27hah63f8adrgw42cvj3lvks84y845v8-file2.txt niks3-go-unit-tests> URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst niks3-go-unit-tests> Compression: zstd niks3-go-unit-tests> NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf niks3-go-unit-tests> NarSize: 160 niks3-go-unit-tests> References: niks3-go-unit-tests> CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf niks3-go-unit-tests> metadata_upload_test.go:77: Retrieved .ls file from S3 (compressed size: 62 bytes) niks3-go-unit-tests> metadata_upload_test.go:77: Decompressed .ls content (49 bytes): niks3-go-unit-tests> {"version":1,"root":{"type":"regular","size":44}} niks3-go-unit-tests> --- PASS: TestNARDeduplicationMetadataUploadBug (7.04s) niks3-go-unit-tests> 2026/08/27 09:16:12 INFO Starting cleanup of old closures method=DELETE path=/api/closures niks3-go-unit-tests> 2026/08/27 09:16:12 INFO Garbage collection started niks3-go-unit-tests> 2026/08/27 09:16:12 INFO Aborted multipart uploads count=0 niks3-go-unit-tests> 2026/08/27 09:16:12 WARN Force mode enabled - objects will be deleted immediately without grace period niks3-go-unit-tests> 2026/08/27 09:16:12 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=2001 objects-failed-to-delete=0 niks3-go-unit-tests> 2026/08/27 09:16:12 INFO Vacuumed table table=pending_closures niks3-go-unit-tests> 2026/08/27 09:16:12 INFO Vacuumed table table=pending_objects niks3-go-unit-tests> 2026/08/27 09:16:12 INFO Vacuumed table table=multipart_uploads niks3-go-unit-tests> 2026/08/27 09:16:12 INFO Vacuumed table table=closures niks3-go-unit-tests> 2026/08/27 09:16:12 INFO Vacuumed table table=objects niks3-go-unit-tests> --- FAIL: TestClientErrorHandling (0.00s) niks3-go-unit-tests> --- FAIL: TestClientErrorHandling/InvalidStorePath (4.49s) niks3-go-unit-tests> --- FAIL: TestClientErrorHandling/InvalidAuthToken (5.12s) niks3-go-unit-tests> --- PASS: TestClientErrorHandling/ServerNotAvailable (6.58s) niks3-go-unit-tests> 2026/08/27 09:16:14 INFO Garbage collection progress phase="" failed_uploads_deleted=0 old_closures_deleted=1 objects_marked=3 objects_deleted=2001 objects_failed=0 niks3-go-unit-tests> === NAME TestClientIntegration niks3-go-unit-tests> client_integration_test.go:303: Objects in database after GC: niks3-go-unit-tests> client_integration_test.go:303: Successfully deleted all objects with GC --force niks3-go-unit-tests> --- PASS: TestClientIntegration (8.94s) niks3-go-unit-tests> FAIL niks3-go-unit-tests> {"timestamp":"2026-08-27T09:16:14.127245287Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:50844","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1836,"threadName":"rustfs-worker","threadId":"ThreadId(241)"} niks3-go-unit-tests> 2026-08-27 09:16:14.474 UTC [111] LOG: received smart shutdown request niks3-go-unit-tests> 2026/08/27 09:16:24 ERROR failed to kill rustfs error="no such process" niks3-go-unit-tests> 2026/08/27 09:16:24 INFO killed rustfs niks3-go-unit-tests> 2026-08-27 09:16:24.478 UTC [1201] FATAL: terminating connection due to unexpected postmaster exit niks3-go-unit-tests> 2026-08-27 09:16:24.478 UTC [643] FATAL: terminating connection due to unexpected postmaster exit niks3-go-unit-tests> 2026-08-27 09:16:24.478 UTC [1221] FATAL: terminating connection due to unexpected postmaster exit niks3-go-unit-tests> 2026-08-27 09:16:24.478 UTC [1099] FATAL: terminating connection due to unexpected postmaster exit niks3-go-unit-tests> 2026-08-27 09:16:24.478 UTC [653] FATAL: terminating connection due to unexpected postmaster exit niks3-go-unit-tests> 2026-08-27 09:16:24.478 UTC [1156] FATAL: terminating connection due to unexpected postmaster exit niks3-go-unit-tests> 2026/08/27 09:16:24 ERROR failed to wait for rustfs error="signal: killed" error: Cannot build '/nix/store/rlqx83yh2793iy613vb943df26n5x616-niks3-go-unit-tests.drv'. Reason: builder failed with exit code 1. Output paths: /nix/store/0w25shygpmjiifdzkd0fif29lnhkzv07-niks3-go-unit-tests Last 25 log lines: > 2026/08/27 09:16:12 INFO Vacuumed table table=pending_objects > 2026/08/27 09:16:12 INFO Vacuumed table table=multipart_uploads > 2026/08/27 09:16:12 INFO Vacuumed table table=closures > 2026/08/27 09:16:12 INFO Vacuumed table table=objects > --- FAIL: TestClientErrorHandling (0.00s) > --- FAIL: TestClientErrorHandling/InvalidStorePath (4.49s) > --- FAIL: TestClientErrorHandling/InvalidAuthToken (5.12s) > --- PASS: TestClientErrorHandling/ServerNotAvailable (6.58s) > 2026/08/27 09:16:14 INFO Garbage collection progress phase="" failed_uploads_deleted=0 old_closures_deleted=1 objects_marked=3 objects_deleted=2001 objects_failed=0 > === NAME TestClientIntegration > client_integration_test.go:303: Objects in database after GC: > client_integration_test.go:303: Successfully deleted all objects with GC --force > --- PASS: TestClientIntegration (8.94s) > FAIL > {"timestamp":"2026-08-27T09:16:14.127245287Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:50844","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1836,"threadName":"rustfs-worker","threadId":"ThreadId(241)"} > 2026-08-27 09:16:14.474 UTC [111] LOG: received smart shutdown request > 2026/08/27 09:16:24 ERROR failed to kill rustfs error="no such process" > 2026/08/27 09:16:24 INFO killed rustfs > 2026-08-27 09:16:24.478 UTC [1201] FATAL: terminating connection due to unexpected postmaster exit > 2026-08-27 09:16:24.478 UTC [643] FATAL: terminating connection due to unexpected postmaster exit > 2026-08-27 09:16:24.478 UTC [1221] FATAL: terminating connection due to unexpected postmaster exit > 2026-08-27 09:16:24.478 UTC [1099] FATAL: terminating connection due to unexpected postmaster exit > 2026-08-27 09:16:24.478 UTC [653] FATAL: terminating connection due to unexpected postmaster exit > 2026-08-27 09:16:24.478 UTC [1156] FATAL: terminating connection due to unexpected postmaster exit > 2026/08/27 09:16:24 ERROR failed to wait for rustfs error="signal: killed" For full logs, run: nix log /nix/store/rlqx83yh2793iy613vb943df26n5x616-niks3-go-unit-tests.drv