tribuchet: building on eliza Running client tests... === RUN TestDoServerRequestAttachesToken === PAUSE TestDoServerRequestAttachesToken === RUN TestUploadBuildLog_FileBodyReplayedOnRetry === PAUSE TestUploadBuildLog_FileBodyReplayedOnRetry === RUN TestRegisterUploadedObjectReusesConnections === PAUSE TestRegisterUploadedObjectReusesConnections === RUN TestRunGarbageCollection_FinishedOnAnotherReplica === PAUSE TestRunGarbageCollection_FinishedOnAnotherReplica === RUN TestRunGarbageCollection_NotFoundAfterLocalRun === PAUSE TestRunGarbageCollection_NotFoundAfterLocalRun === RUN TestCaseHackSuffix === PAUSE TestCaseHackSuffix === RUN TestFilterOversizedClosures === PAUSE TestFilterOversizedClosures === RUN TestUploadMultipart_PartsInParallel === PAUSE TestUploadMultipart_PartsInParallel === RUN TestUploadMultipart_ReturnsAfterDroppedBody === PAUSE TestUploadMultipart_ReturnsAfterDroppedBody === RUN TestPartSizeForNAR === PAUSE TestPartSizeForNAR === RUN TestDumpPathCaseHackMatchesNix --- PASS: TestDumpPathCaseHackMatchesNix (0.26s) === RUN TestDumpPathCaseHackCollision --- PASS: TestDumpPathCaseHackCollision (0.00s) === RUN TestDumpPathMatchesNix === PAUSE TestDumpPathMatchesNix === RUN TestDumpPathSingleFile === PAUSE TestDumpPathSingleFile === RUN TestDumpPathWriterError === PAUSE TestDumpPathWriterError === RUN TestEncodeNixBase32 === PAUSE TestEncodeNixBase32 === RUN TestEncodeNixBase32WithRealHash === PAUSE TestEncodeNixBase32WithRealHash === RUN TestConvertHashToNix32 === PAUSE TestConvertHashToNix32 === RUN TestGetStorePathHash === PAUSE TestGetStorePathHash === RUN TestPathInfoHashCompatibility === PAUSE TestPathInfoHashCompatibility === RUN TestParsePathInfoJSON === PAUSE TestParsePathInfoJSON === RUN TestParsePathInfoJSONMultiplePaths === PAUSE TestParsePathInfoJSONMultiplePaths === RUN TestPathInfoCACompatibility === PAUSE TestPathInfoCACompatibility === RUN TestRateLimiterFeedback === PAUSE TestRateLimiterFeedback === RUN TestRateLimiterFeedback_400DoesNotCountAsSuccess === PAUSE TestRateLimiterFeedback_400DoesNotCountAsSuccess === RUN TestRegisterUploadedObject_BoundedAgainstSilentServer === PAUSE TestRegisterUploadedObject_BoundedAgainstSilentServer === RUN TestResolveStorePath === PAUSE TestResolveStorePath === RUN TestDoWithRetry_BodyReplayedViaGetBody === PAUSE TestDoWithRetry_BodyReplayedViaGetBody === RUN TestRetryTimeoutOutlastsMaxRetries === PAUSE TestRetryTimeoutOutlastsMaxRetries === RUN TestDoWithRetry_FinalResponseBodyReadable === PAUSE TestDoWithRetry_FinalResponseBodyReadable === RUN TestShellSplit === PAUSE TestShellSplit === RUN TestShellSplitErrors === PAUSE TestShellSplitErrors === RUN TestStreamPushReportsEveryPath === PAUSE TestStreamPushReportsEveryPath === RUN TestStreamPushBatchesUnderLoad === PAUSE TestStreamPushBatchesUnderLoad === RUN TestStreamPushIsolatesFailures === PAUSE TestStreamPushIsolatesFailures === RUN TestStreamPushGivesUpOnDeadServer === PAUSE TestStreamPushGivesUpOnDeadServer === RUN TestStreamPushRequestLine === PAUSE TestStreamPushRequestLine === RUN TestStreamPushReportsSignatures === PAUSE TestStreamPushReportsSignatures === RUN TestStreamPushReportsSkippedPaths === PAUSE TestStreamPushReportsSkippedPaths === RUN TestClientSignaturesByStorePath === PAUSE TestClientSignaturesByStorePath === RUN TestStreamPushStopsWhenCanceled === PAUSE TestStreamPushStopsWhenCanceled === RUN TestSetClientTLS === PAUSE TestSetClientTLS === RUN TestSetClientTLSDoesNotMutateDefaultTransport === PAUSE TestSetClientTLSDoesNotMutateDefaultTransport === RUN TestSetClientTLSErrors === PAUSE TestSetClientTLSErrors === RUN TestStaticToken === PAUSE TestStaticToken === RUN TestFileTokenReadsAndCaches === PAUSE TestFileTokenReadsAndCaches === RUN TestFileTokenMissing === PAUSE TestFileTokenMissing === RUN TestFileTokenEmpty === PAUSE TestFileTokenEmpty === RUN TestScriptTokenNoExpiryRerunsEveryCall === PAUSE TestScriptTokenNoExpiryRerunsEveryCall === RUN TestScriptTokenCachesUntilRefresh === PAUSE TestScriptTokenCachesUntilRefresh === RUN TestScriptTokenEmptyToken === PAUSE TestScriptTokenEmptyToken === RUN TestScriptTokenBadJSON === PAUSE TestScriptTokenBadJSON === RUN TestScriptTokenScriptFails === PAUSE TestScriptTokenScriptFails === RUN TestScriptTokenEmptyCommand === PAUSE TestScriptTokenEmptyCommand === CONT TestDoServerRequestAttachesToken === CONT TestRetryTimeoutOutlastsMaxRetries === CONT TestScriptTokenEmptyCommand --- PASS: TestScriptTokenEmptyCommand (0.00s) === CONT TestScriptTokenScriptFails === CONT TestScriptTokenBadJSON === CONT TestDoWithRetry_BodyReplayedViaGetBody === CONT TestResolveStorePath === CONT TestScriptTokenEmptyToken --- PASS: TestResolveStorePath (0.00s) === CONT TestRegisterUploadedObject_BoundedAgainstSilentServer === CONT TestRateLimiterFeedback_400DoesNotCountAsSuccess 2026/11/06 07:46:10 WARN Rate limiter enabled after throttle name=server-test rate=5 === CONT TestRateLimiterFeedback === RUN TestRateLimiterFeedback/429_enables_limiter === PAUSE TestRateLimiterFeedback/429_enables_limiter === RUN TestRateLimiterFeedback/503_enables_limiter === PAUSE TestRateLimiterFeedback/503_enables_limiter === RUN TestRateLimiterFeedback/200_does_not_enable_limiter === PAUSE TestRateLimiterFeedback/200_does_not_enable_limiter === RUN TestRateLimiterFeedback/400_does_not_enable_limiter === PAUSE TestRateLimiterFeedback/400_does_not_enable_limiter === CONT TestRateLimiterFeedback/429_enables_limiter === CONT TestPathInfoCACompatibility === RUN TestPathInfoCACompatibility/null_ca_field === PAUSE TestPathInfoCACompatibility/null_ca_field === RUN TestPathInfoCACompatibility/old_string_format_-_text === PAUSE TestPathInfoCACompatibility/old_string_format_-_text === RUN TestPathInfoCACompatibility/old_string_format_-_fixed_recursive === PAUSE TestPathInfoCACompatibility/old_string_format_-_fixed_recursive === RUN TestPathInfoCACompatibility/new_structured_format_-_text === PAUSE TestPathInfoCACompatibility/new_structured_format_-_text === RUN TestPathInfoCACompatibility/new_structured_format_-_nar_method === PAUSE TestPathInfoCACompatibility/new_structured_format_-_nar_method === CONT TestPathInfoCACompatibility/null_ca_field === CONT TestParsePathInfoJSONMultiplePaths === RUN TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths === PAUSE TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths === RUN TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths === PAUSE TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths === CONT TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths === CONT TestParsePathInfoJSON === RUN TestParsePathInfoJSON/Nix_format === PAUSE TestParsePathInfoJSON/Nix_format === RUN TestParsePathInfoJSON/Lix_format === PAUSE TestParsePathInfoJSON/Lix_format === RUN TestParsePathInfoJSON/empty_input === PAUSE TestParsePathInfoJSON/empty_input === RUN TestParsePathInfoJSON/whitespace_only === PAUSE TestParsePathInfoJSON/whitespace_only === RUN TestParsePathInfoJSON/invalid_JSON === PAUSE TestParsePathInfoJSON/invalid_JSON === CONT TestParsePathInfoJSON/Nix_format === CONT TestPathInfoHashCompatibility === RUN TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI) === PAUSE TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI) === RUN TestPathInfoHashCompatibility/old_string_format_with_colon === PAUSE TestPathInfoHashCompatibility/old_string_format_with_colon === RUN TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI === PAUSE TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI === RUN TestPathInfoHashCompatibility/new_structured_format_with_sha512 === PAUSE TestPathInfoHashCompatibility/new_structured_format_with_sha512 === CONT TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI) === CONT TestGetStorePathHash === RUN TestGetStorePathHash/valid_store_path === PAUSE TestGetStorePathHash/valid_store_path === RUN TestGetStorePathHash/basename_without_hyphen_should_error === PAUSE TestGetStorePathHash/basename_without_hyphen_should_error === RUN TestGetStorePathHash/hash_with_invalid_characters_should_error === PAUSE TestGetStorePathHash/hash_with_invalid_characters_should_error === RUN TestGetStorePathHash/hash_with_wrong_length_should_error === PAUSE TestGetStorePathHash/hash_with_wrong_length_should_error === CONT TestGetStorePathHash/valid_store_path === CONT TestConvertHashToNix32 === RUN TestConvertHashToNix32/SRI_format_to_Nix32 === PAUSE TestConvertHashToNix32/SRI_format_to_Nix32 === RUN TestConvertHashToNix32/already_Nix32_format === PAUSE TestConvertHashToNix32/already_Nix32_format === RUN TestConvertHashToNix32/invalid_format === PAUSE TestConvertHashToNix32/invalid_format === CONT TestConvertHashToNix32/SRI_format_to_Nix32 === CONT TestEncodeNixBase32WithRealHash --- PASS: TestEncodeNixBase32WithRealHash (0.00s) === CONT TestEncodeNixBase32 === RUN TestEncodeNixBase32/test_string_hash === PAUSE TestEncodeNixBase32/test_string_hash === RUN TestEncodeNixBase32/empty_input === PAUSE TestEncodeNixBase32/empty_input === CONT TestEncodeNixBase32/test_string_hash === CONT TestDumpPathWriterError === CONT TestScriptTokenCachesUntilRefresh === CONT TestDumpPathSingleFile === CONT TestDumpPathMatchesNix === CONT TestPartSizeForNAR === RUN TestPartSizeForNAR/zero_stays_at_minimum === PAUSE TestPartSizeForNAR/zero_stays_at_minimum === RUN TestPartSizeForNAR/small_stays_at_minimum === PAUSE TestPartSizeForNAR/small_stays_at_minimum === RUN TestPartSizeForNAR/80_GiB_fits_at_minimum === PAUSE TestPartSizeForNAR/80_GiB_fits_at_minimum === RUN TestPartSizeForNAR/115_GiB_needs_larger_parts === PAUSE TestPartSizeForNAR/115_GiB_needs_larger_parts === RUN TestPartSizeForNAR/1_TiB === PAUSE TestPartSizeForNAR/1_TiB === RUN TestPartSizeForNAR/5_TiB_S3_max_object === PAUSE TestPartSizeForNAR/5_TiB_S3_max_object === RUN TestPartSizeForNAR/capped_at_5_GiB === PAUSE TestPartSizeForNAR/capped_at_5_GiB === CONT TestPartSizeForNAR/zero_stays_at_minimum === CONT TestUploadMultipart_ReturnsAfterDroppedBody === CONT TestUploadMultipart_PartsInParallel === CONT TestFilterOversizedClosures === RUN TestFilterOversizedClosures/no_limit_keeps_everything === CONT TestCaseHackSuffix === CONT TestRunGarbageCollection_NotFoundAfterLocalRun === CONT TestRunGarbageCollection_FinishedOnAnotherReplica === CONT TestRegisterUploadedObjectReusesConnections === CONT TestUploadBuildLog_FileBodyReplayedOnRetry === CONT TestRateLimiterFeedback/400_does_not_enable_limiter === CONT TestRateLimiterFeedback/200_does_not_enable_limiter === CONT TestRateLimiterFeedback/503_enables_limiter --- PASS: TestScriptTokenScriptFails (0.31s) === CONT TestPathInfoCACompatibility/new_structured_format_-_nar_method === CONT TestPathInfoCACompatibility/new_structured_format_-_text === CONT TestPathInfoCACompatibility/old_string_format_-_fixed_recursive === CONT TestPathInfoCACompatibility/old_string_format_-_text --- PASS: TestPathInfoCACompatibility (0.00s) --- PASS: TestPathInfoCACompatibility/null_ca_field (0.00s) --- PASS: TestPathInfoCACompatibility/new_structured_format_-_nar_method (0.03s) --- PASS: TestPathInfoCACompatibility/new_structured_format_-_text (0.00s) --- PASS: TestPathInfoCACompatibility/old_string_format_-_fixed_recursive (0.00s) --- PASS: TestPathInfoCACompatibility/old_string_format_-_text (0.00s) === CONT TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths --- PASS: TestParsePathInfoJSONMultiplePaths (0.00s) --- PASS: TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths (0.00s) --- PASS: TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths (0.00s) === CONT TestParsePathInfoJSON/invalid_JSON === CONT TestParsePathInfoJSON/whitespace_only --- PASS: TestScriptTokenBadJSON (0.35s) --- PASS: TestScriptTokenEmptyToken (0.34s) === CONT TestPathInfoHashCompatibility/new_structured_format_with_sha512 === CONT TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI === CONT TestPathInfoHashCompatibility/old_string_format_with_colon --- PASS: TestPathInfoHashCompatibility (0.00s) --- PASS: TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI) (0.00s) --- PASS: TestPathInfoHashCompatibility/new_structured_format_with_sha512 (0.00s) --- PASS: TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI (0.00s) --- PASS: TestPathInfoHashCompatibility/old_string_format_with_colon (0.00s) === CONT TestParsePathInfoJSON/empty_input === CONT TestGetStorePathHash/hash_with_invalid_characters_should_error === PAUSE TestFilterOversizedClosures/no_limit_keeps_everything === RUN TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped === PAUSE TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped === RUN TestFilterOversizedClosures/all_closures_skipped === PAUSE TestFilterOversizedClosures/all_closures_skipped === CONT TestConvertHashToNix32/invalid_format === CONT TestConvertHashToNix32/already_Nix32_format --- PASS: TestConvertHashToNix32 (0.00s) --- PASS: TestConvertHashToNix32/SRI_format_to_Nix32 (0.00s) --- PASS: TestConvertHashToNix32/invalid_format (0.00s) --- PASS: TestConvertHashToNix32/already_Nix32_format (0.00s) === CONT TestGetStorePathHash/basename_without_hyphen_should_error === CONT TestEncodeNixBase32/empty_input --- PASS: TestEncodeNixBase32 (0.00s) --- PASS: TestEncodeNixBase32/test_string_hash (0.00s) --- PASS: TestEncodeNixBase32/empty_input (0.00s) === CONT TestParsePathInfoJSON/Lix_format --- PASS: TestParsePathInfoJSON (0.00s) --- PASS: TestParsePathInfoJSON/Nix_format (0.00s) --- PASS: TestParsePathInfoJSON/invalid_JSON (0.00s) --- PASS: TestParsePathInfoJSON/whitespace_only (0.00s) --- PASS: TestParsePathInfoJSON/empty_input (0.00s) --- PASS: TestParsePathInfoJSON/Lix_format (0.00s) === CONT TestScriptTokenNoExpiryRerunsEveryCall 2026/11/06 07:46:10 INFO Garbage collection started 2026/11/06 07:46:10 WARN Rate limiter enabled after throttle name=server-test rate=5 2026/11/06 07:46:10 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:37031 --- PASS: TestDoServerRequestAttachesToken (0.35s) 2026/11/06 07:46:10 WARN Rate limiter backed off name=server-test rate=5 === CONT TestFileTokenReadsAndCaches 2026/11/06 07:46:10 WARN Request returned retryable status, retrying attempt=2 backoff=0s status=503 url=http://127.0.0.1:37031 === CONT TestFileTokenEmpty --- PASS: TestFileTokenReadsAndCaches (0.05s) === CONT TestStaticToken --- PASS: TestStaticToken (0.00s) === CONT TestSetClientTLSErrors --- PASS: TestFileTokenEmpty (0.00s) === CONT TestSetClientTLSDoesNotMutateDefaultTransport === CONT TestGetStorePathHash/hash_with_wrong_length_should_error --- PASS: TestGetStorePathHash (0.00s) --- PASS: TestGetStorePathHash/valid_store_path (0.00s) --- PASS: TestGetStorePathHash/hash_with_invalid_characters_should_error (0.00s) --- PASS: TestGetStorePathHash/basename_without_hyphen_should_error (0.00s) --- PASS: TestGetStorePathHash/hash_with_wrong_length_should_error (0.00s) === CONT TestFileTokenMissing --- PASS: TestFileTokenMissing (0.00s) === CONT TestStreamPushStopsWhenCanceled 2026/11/06 07:46:10 INFO Garbage collection progress phase=running_on_another_replica failed_uploads_deleted=0 old_closures_deleted=0 objects_marked=0 objects_deleted=0 objects_failed=0 --- PASS: TestDoWithRetry_BodyReplayedViaGetBody (0.40s) === CONT TestClientSignaturesByStorePath --- PASS: TestClientSignaturesByStorePath (0.00s) === CONT TestStreamPushReportsSkippedPaths --- PASS: TestStreamPushStopsWhenCanceled (0.35s) === CONT TestStreamPushReportsSignatures --- PASS: TestStreamPushReportsSkippedPaths (0.44s) === CONT TestStreamPushRequestLine 2026/11/06 07:46:11 ERROR Upload failed error=boom count=1 --- PASS: TestStreamPushReportsSignatures (0.09s) === CONT TestStreamPushGivesUpOnDeadServer === CONT TestStreamPushIsolatesFailures --- PASS: TestScriptTokenCachesUntilRefresh (1.28s) === CONT TestStreamPushBatchesUnderLoad === RUN TestStreamPushBatchesUnderLoad/oneAtATime=false === PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=false === RUN TestStreamPushBatchesUnderLoad/oneAtATime=true === PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=true === CONT TestStreamPushReportsEveryPath 2026/11/06 07:46:12 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:43661/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:43661/api/uploads/complete 2026/11/06 07:46:12 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded" 2026/11/06 07:46:12 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:43661/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:43661/api/uploads/complete 2026/11/06 07:46:12 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded" --- PASS: TestScriptTokenNoExpiryRerunsEveryCall (1.56s) 2026/11/06 07:46:12 WARN Garbage collection finished on another replica. Its statistics are not available here === CONT TestShellSplitErrors --- PASS: TestShellSplitErrors (0.00s) === CONT TestShellSplit 2026/11/06 07:46:12 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:43661/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:43661/api/uploads/complete --- PASS: TestStreamPushReportsEveryPath (0.00s) 2026/11/06 07:46:12 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded" === CONT TestDoWithRetry_FinalResponseBodyReadable --- PASS: TestRunGarbageCollection_FinishedOnAnotherReplica (1.90s) === CONT TestPartSizeForNAR/capped_at_5_GiB === CONT TestPartSizeForNAR/5_TiB_S3_max_object === CONT TestPartSizeForNAR/1_TiB === CONT TestPartSizeForNAR/115_GiB_needs_larger_parts === CONT TestPartSizeForNAR/80_GiB_fits_at_minimum === CONT TestPartSizeForNAR/small_stays_at_minimum --- PASS: TestPartSizeForNAR (0.00s) --- PASS: TestPartSizeForNAR/zero_stays_at_minimum (0.00s) --- PASS: TestPartSizeForNAR/capped_at_5_GiB (0.00s) --- PASS: TestPartSizeForNAR/5_TiB_S3_max_object (0.00s) --- PASS: TestPartSizeForNAR/1_TiB (0.00s) --- PASS: TestPartSizeForNAR/115_GiB_needs_larger_parts (0.00s) --- PASS: TestPartSizeForNAR/80_GiB_fits_at_minimum (0.00s) --- PASS: TestPartSizeForNAR/small_stays_at_minimum (0.00s) === CONT TestFilterOversizedClosures/no_limit_keeps_everything === CONT TestFilterOversizedClosures/all_closures_skipped 2026/11/06 07:46:12 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-small nar_size=1000 max_nar_size=50 === CONT TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped 2026/11/06 07:46:12 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb-vm-image nar_size=5000 max_nar_size=2000 --- PASS: TestFilterOversizedClosures (0.33s) --- PASS: TestFilterOversizedClosures/no_limit_keeps_everything (0.00s) --- PASS: TestFilterOversizedClosures/all_closures_skipped (0.00s) --- PASS: TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped (0.00s) === CONT TestStreamPushBatchesUnderLoad/oneAtATime=false === CONT TestSetClientTLS --- PASS: TestCaseHackSuffix (1.90s) 2026/11/06 07:46:12 WARN Rate limiter enabled after throttle name=server-test rate=5 === CONT TestStreamPushBatchesUnderLoad/oneAtATime=true 2026/11/06 07:46:12 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:46241 2026/11/06 07:46:12 WARN Rate limiter backed off name=server-test rate=5 --- PASS: TestDoWithRetry_FinalResponseBodyReadable (0.00s) --- PASS: TestShellSplit (0.00s) 2026/11/06 07:46:12 WARN Rate limiter enabled after throttle name=server-test rate=5 2026/11/06 07:46:12 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=429 url=http://127.0.0.1:43091 2026/11/06 07:46:12 WARN Rate limiter enabled after throttle name=server-test rate=5 2026/11/06 07:46:12 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:34113 2026/11/06 07:46:12 ERROR Upload failed error=boom count=1 2026/11/06 07:46:12 WARN Rate limiter backed off name=server-test rate=5 2026/11/06 07:46:12 WARN Rate limiter backed off name=server-test rate=5 --- PASS: TestStreamPushBatchesUnderLoad (0.62s) --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=true (0.00s) --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=false (0.00s) 2026/11/06 07:46:12 ERROR Upload failed error="bad path" count=3 --- PASS: TestRateLimiterFeedback (0.00s) --- PASS: TestRateLimiterFeedback/200_does_not_enable_limiter (1.27s) --- PASS: TestRateLimiterFeedback/400_does_not_enable_limiter (1.90s) --- PASS: TestRateLimiterFeedback/503_enables_limiter (1.90s) --- PASS: TestRateLimiterFeedback/429_enables_limiter (1.91s) --- PASS: TestStreamPushIsolatesFailures (0.64s) 2026/11/06 07:46:12 ERROR Upload failed error="connection refused" count=20 2026/11/06 07:46:12 ERROR Server seems unavailable, giving up on batch untried=17 --- PASS: TestDumpPathSingleFile (1.91s) --- PASS: TestStreamPushGivesUpOnDeadServer (1.08s) 2026/11/06 07:46:12 INFO Garbage collection started 2026/11/06 07:46:12 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:41015 --- PASS: TestUploadMultipart_ReturnsAfterDroppedBody (1.91s) 2026/11/06 07:46:12 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:41015 --- PASS: TestRunGarbageCollection_NotFoundAfterLocalRun (1.91s) 2026/11/06 07:46:12 WARN Rate limiter enabled after throttle name=s3-test rate=5 2026/11/06 07:46:12 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:38195 === RUN TestSetClientTLSErrors/missing_cert_file --- PASS: TestUploadBuildLog_FileBodyReplayedOnRetry (1.95s) 2026/11/06 07:46:12 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:41015 --- PASS: TestSetClientTLSDoesNotMutateDefaultTransport (1.56s) === PAUSE TestSetClientTLSErrors/missing_cert_file === RUN TestSetClientTLSErrors/missing_key_file 2026/11/06 07:46:12 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:41015 === RUN TestSetClientTLS/rejects_connection_without_client_cert === PAUSE TestSetClientTLSErrors/missing_key_file === RUN TestSetClientTLSErrors/missing_ca_file === PAUSE TestSetClientTLSErrors/missing_ca_file === RUN TestSetClientTLSErrors/invalid_ca_file === PAUSE TestSetClientTLSErrors/invalid_ca_file === CONT TestSetClientTLSErrors/missing_cert_file === CONT TestSetClientTLSErrors/invalid_ca_file === CONT TestSetClientTLSErrors/missing_ca_file 2026/11/06 07:46:12 WARN Request returned retryable status, retrying attempt=3 backoff=1ms status=502 url=http://127.0.0.1:41015 === PAUSE TestSetClientTLS/rejects_connection_without_client_cert === RUN TestSetClientTLS/succeeds_with_client_cert_and_CA === PAUSE TestSetClientTLS/succeeds_with_client_cert_and_CA === RUN TestSetClientTLS/preserves_debug_logging_transport === PAUSE TestSetClientTLS/preserves_debug_logging_transport === CONT TestSetClientTLS/rejects_connection_without_client_cert === CONT TestSetClientTLS/preserves_debug_logging_transport === CONT TestSetClientTLSErrors/missing_key_file === CONT TestSetClientTLS/succeeds_with_client_cert_and_CA 2026/11/06 07:46:12 WARN Request returned retryable status, retrying attempt=4 backoff=1ms status=502 url=http://127.0.0.1:41015 --- PASS: TestSetClientTLSErrors (1.58s) --- PASS: TestSetClientTLSErrors/missing_cert_file (0.00s) --- PASS: TestSetClientTLSErrors/missing_ca_file (0.00s) --- PASS: TestSetClientTLSErrors/invalid_ca_file (0.00s) --- PASS: TestSetClientTLSErrors/missing_key_file (0.00s) 2026/11/06 07:46:12 WARN Request returned retryable status, retrying attempt=5 backoff=1ms status=502 url=http://127.0.0.1:41015 2026/11/06 07:46:12 WARN Request returned retryable status, retrying attempt=6 backoff=1ms status=502 url=http://127.0.0.1:41015 2026/11/06 07:46:12 WARN Request returned retryable status, retrying attempt=7 backoff=1ms status=502 url=http://127.0.0.1:41015 2026/11/06 07:46:12 WARN Request returned retryable status, retrying attempt=8 backoff=1ms status=502 url=http://127.0.0.1:41015 2026/11/06 07:46:12 WARN Request returned retryable status, retrying attempt=9 backoff=1ms status=502 url=http://127.0.0.1:41015 2026/11/06 07:46:12 WARN Request returned retryable status, retrying attempt=10 backoff=1ms status=502 url=http://127.0.0.1:41015 2026/11/06 07:46:12 WARN Request returned retryable status, retrying attempt=11 backoff=1ms status=502 url=http://127.0.0.1:41015 2026/11/06 07:46:12 WARN Request returned retryable status, retrying attempt=12 backoff=1ms status=502 url=http://127.0.0.1:41015 --- PASS: TestRegisterUploadedObjectReusesConnections (1.98s) 2026/11/06 07:46:12 http: TLS handshake error from 127.0.0.1:49216: remote error: tls: bad certificate --- PASS: TestDumpPathWriterError (1.98s) --- PASS: TestSetClientTLS (0.06s) --- PASS: TestSetClientTLS/succeeds_with_client_cert_and_CA (0.00s) --- PASS: TestSetClientTLS/preserves_debug_logging_transport (0.00s) --- PASS: TestSetClientTLS/rejects_connection_without_client_cert (0.01s) 2026/11/06 07:46:12 WARN Request returned retryable status, retrying attempt=13 backoff=1ms status=502 url=http://127.0.0.1:41015 2026/11/06 07:46:12 WARN Request returned retryable status, retrying attempt=14 backoff=1ms status=502 url=http://127.0.0.1:41015 2026/11/06 07:46:12 WARN Request returned retryable status, retrying attempt=15 backoff=1ms status=502 url=http://127.0.0.1:41015 2026/11/06 07:46:12 WARN Request returned retryable status, retrying attempt=16 backoff=1ms status=502 url=http://127.0.0.1:41015 --- PASS: TestStreamPushRequestLine (1.16s) 2026/11/06 07:46:12 WARN Request returned retryable status, retrying attempt=17 backoff=1ms status=502 url=http://127.0.0.1:41015 --- PASS: TestRetryTimeoutOutlastsMaxRetries (2.00s) --- PASS: TestDumpPathMatchesNix (2.16s) --- PASS: TestUploadMultipart_PartsInParallel (2.56s) --- PASS: TestRateLimiterFeedback_400DoesNotCountAsSuccess (2.71s) === NAME TestRegisterUploadedObject_BoundedAgainstSilentServer register_upload_test.go:52: server saw 2 registrations, want 3 --- FAIL: TestRegisterUploadedObject_BoundedAgainstSilentServer (10.01s) FAIL