niks3-go-unit-tests
checks.aarch64-darwin.go-unit-tests
· build #402
· raw
1tribuchet: building on mac022Running client tests...3=== RUN TestDoServerRequestAttachesToken4=== PAUSE TestDoServerRequestAttachesToken5=== RUN TestUploadBuildLog_FileBodyReplayedOnRetry6=== PAUSE TestUploadBuildLog_FileBodyReplayedOnRetry7=== RUN TestRegisterUploadedObjectReusesConnections8=== PAUSE TestRegisterUploadedObjectReusesConnections9=== RUN TestRunGarbageCollection_FinishedOnAnotherReplica10=== PAUSE TestRunGarbageCollection_FinishedOnAnotherReplica11=== RUN TestRunGarbageCollection_NotFoundAfterLocalRun12=== PAUSE TestRunGarbageCollection_NotFoundAfterLocalRun13=== RUN TestCaseHackSuffix14=== PAUSE TestCaseHackSuffix15=== RUN TestFilterOversizedClosures16=== PAUSE TestFilterOversizedClosures17=== RUN TestPushPathsReportsOnlyUnuploadablePathsOfSkippedClosure18=== PAUSE TestPushPathsReportsOnlyUnuploadablePathsOfSkippedClosure19=== RUN TestUploadMultipart_PartsInParallel20=== PAUSE TestUploadMultipart_PartsInParallel21=== RUN TestUploadMultipart_ProducerErrorIsNotEOF22=== PAUSE TestUploadMultipart_ProducerErrorIsNotEOF23=== RUN TestUploadMultipart_FailedPartBufferNotReused24=== PAUSE TestUploadMultipart_FailedPartBufferNotReused25=== RUN TestPartSizeForNAR26=== PAUSE TestPartSizeForNAR27=== RUN TestUploadMultipart_SupersededByPeer28=== PAUSE TestUploadMultipart_SupersededByPeer29=== RUN TestDumpPathCaseHackMatchesNix30--- PASS: TestDumpPathCaseHackMatchesNix (0.08s)31=== RUN TestDumpPathCaseHackCollision32--- PASS: TestDumpPathCaseHackCollision (0.00s)33=== RUN TestSupersededNARStillUploadsListing34=== PAUSE TestSupersededNARStillUploadsListing35=== RUN TestTruncatedNARDumpIsNotCompleted36=== PAUSE TestTruncatedNARDumpIsNotCompleted37=== RUN TestDumpPathMatchesNix38=== PAUSE TestDumpPathMatchesNix39=== RUN TestDumpPathSingleFile40=== PAUSE TestDumpPathSingleFile41=== RUN TestDumpPathWriterError42=== PAUSE TestDumpPathWriterError43=== RUN TestDumpPathWriterErrorStopsReading44 nar_test.go:280: no /proc/self/io: open /proc/self/io: no such file or directory45--- SKIP: TestDumpPathWriterErrorStopsReading (0.39s)46=== RUN TestEncodeNixBase3247=== PAUSE TestEncodeNixBase3248=== RUN TestEncodeNixBase32WithRealHash49=== PAUSE TestEncodeNixBase32WithRealHash50=== RUN TestConvertHashToNix3251=== PAUSE TestConvertHashToNix3252=== RUN TestGetStorePathHash53=== PAUSE TestGetStorePathHash54=== RUN TestPathInfoHashCompatibility55=== PAUSE TestPathInfoHashCompatibility56=== RUN TestParsePathInfoJSON57=== PAUSE TestParsePathInfoJSON58=== RUN TestParsePathInfoJSONMultiplePaths59=== PAUSE TestParsePathInfoJSONMultiplePaths60=== RUN TestPathInfoCACompatibility61=== PAUSE TestPathInfoCACompatibility62=== RUN TestUploadPendingObjectsStopsStartingAfterFailure63--- PASS: TestUploadPendingObjectsStopsStartingAfterFailure (0.01s)64=== RUN TestCompletePendingClosure_NotFoundIsSuccessWhenClosurePresent65=== PAUSE TestCompletePendingClosure_NotFoundIsSuccessWhenClosurePresent66=== RUN TestCompletePendingClosure_NotFoundWithoutKey67=== PAUSE TestCompletePendingClosure_NotFoundWithoutKey68=== RUN TestRateLimiterFeedback69=== PAUSE TestRateLimiterFeedback70=== RUN TestRateLimiterFeedback_400DoesNotCountAsSuccess71=== PAUSE TestRateLimiterFeedback_400DoesNotCountAsSuccess72=== RUN TestRegisterUploadedObject_BoundedAgainstSilentServer73=== PAUSE TestRegisterUploadedObject_BoundedAgainstSilentServer74=== RUN TestResolveStorePath75=== PAUSE TestResolveStorePath76=== RUN TestDoWithRetry_BodyReplayedViaGetBody77=== PAUSE TestDoWithRetry_BodyReplayedViaGetBody78=== RUN TestDoWithRetry_ClosesBodyWhenNoAttemptIsMade79=== PAUSE TestDoWithRetry_ClosesBodyWhenNoAttemptIsMade80=== RUN TestRetryTimeoutOutlastsMaxRetries81=== PAUSE TestRetryTimeoutOutlastsMaxRetries82=== RUN TestDoWithRetry_FinalResponseBodyReadable83=== PAUSE TestDoWithRetry_FinalResponseBodyReadable84=== RUN TestShellSplit85=== PAUSE TestShellSplit86=== RUN TestShellSplitErrors87=== PAUSE TestShellSplitErrors88=== RUN TestStreamPushReportsEveryPath89=== PAUSE TestStreamPushReportsEveryPath90=== RUN TestStreamPushBatchesUnderLoad91=== PAUSE TestStreamPushBatchesUnderLoad92=== RUN TestStreamPushIsolatesFailures93=== PAUSE TestStreamPushIsolatesFailures94=== RUN TestStreamPushGivesUpOnDeadServer95=== PAUSE TestStreamPushGivesUpOnDeadServer96=== RUN TestStreamPushRequestLine97=== PAUSE TestStreamPushRequestLine98=== RUN TestStreamPushReportsSignatures99=== PAUSE TestStreamPushReportsSignatures100=== RUN TestStreamPushReportsSkippedPaths101=== PAUSE TestStreamPushReportsSkippedPaths102=== RUN TestClientSignaturesByStorePath103=== PAUSE TestClientSignaturesByStorePath104=== RUN TestStreamPushStopsWhenCanceled105=== PAUSE TestStreamPushStopsWhenCanceled106=== RUN TestStreamPushStopsOnCancel107=== PAUSE TestStreamPushStopsOnCancel108=== RUN TestSetClientTLS109=== PAUSE TestSetClientTLS110=== RUN TestSetClientTLSDoesNotMutateDefaultTransport111=== PAUSE TestSetClientTLSDoesNotMutateDefaultTransport112=== RUN TestSetClientTLSErrors113=== PAUSE TestSetClientTLSErrors114=== RUN TestStaticToken115=== PAUSE TestStaticToken116=== RUN TestFileTokenReadsAndCaches117=== PAUSE TestFileTokenReadsAndCaches118=== RUN TestFileTokenMissing119=== PAUSE TestFileTokenMissing120=== RUN TestFileTokenEmpty121=== PAUSE TestFileTokenEmpty122=== RUN TestScriptTokenNoExpiryRerunsEveryCall123=== PAUSE TestScriptTokenNoExpiryRerunsEveryCall124=== RUN TestScriptTokenCachesUntilRefresh125=== PAUSE TestScriptTokenCachesUntilRefresh126=== RUN TestScriptTokenEmptyToken127=== PAUSE TestScriptTokenEmptyToken128=== RUN TestScriptTokenBadJSON129=== PAUSE TestScriptTokenBadJSON130=== RUN TestScriptTokenScriptFails131=== PAUSE TestScriptTokenScriptFails132=== RUN TestScriptTokenEmptyCommand133=== PAUSE TestScriptTokenEmptyCommand134=== RUN TestScriptTokenDoesNotWaitForItsChildren135=== PAUSE TestScriptTokenDoesNotWaitForItsChildren136=== CONT TestRegisterUploadedObjectReusesConnections137=== CONT TestScriptTokenEmptyCommand138=== CONT TestStreamPushReportsSignatures139=== CONT TestResolveStorePath140=== CONT TestScriptTokenScriptFails141=== CONT TestScriptTokenBadJSON142=== CONT TestScriptTokenEmptyToken143=== CONT TestScriptTokenCachesUntilRefresh144=== CONT TestFileTokenEmpty145=== CONT TestFileTokenMissing146=== CONT TestCompletePendingClosure_NotFoundIsSuccessWhenClosurePresent147=== RUN TestCompletePendingClosure_NotFoundIsSuccessWhenClosurePresent/committed_earlier148=== CONT TestPathInfoCACompatibility149=== RUN TestPathInfoCACompatibility/null_ca_field150=== CONT TestStaticToken151=== CONT TestSetClientTLSErrors152=== CONT TestScriptTokenNoExpiryRerunsEveryCall153=== CONT TestGetStorePathHash154=== CONT TestSetClientTLS155=== CONT TestSetClientTLSDoesNotMutateDefaultTransport156=== CONT TestStreamPushStopsWhenCanceled157=== CONT TestClientSignaturesByStorePath158=== PAUSE TestCompletePendingClosure_NotFoundIsSuccessWhenClosurePresent/committed_earlier159=== CONT TestStreamPushReportsSkippedPaths160=== CONT TestDumpPathMatchesNix161=== CONT TestRateLimiterFeedback_400DoesNotCountAsSuccess162=== CONT TestRegisterUploadedObject_BoundedAgainstSilentServer163=== CONT TestRateLimiterFeedback164=== CONT TestScriptTokenDoesNotWaitForItsChildren165--- PASS: TestScriptTokenEmptyCommand (0.00s)166=== CONT TestCompletePendingClosure_NotFoundWithoutKey167=== CONT TestFileTokenReadsAndCaches168=== CONT TestPathInfoHashCompatibility169=== CONT TestParsePathInfoJSONMultiplePaths170=== PAUSE TestPathInfoCACompatibility/null_ca_field171=== CONT TestParsePathInfoJSON172=== RUN TestGetStorePathHash/valid_store_path173=== CONT TestConvertHashToNix32174=== CONT TestEncodeNixBase32WithRealHash175=== RUN TestCompletePendingClosure_NotFoundIsSuccessWhenClosurePresent/cleaned_up176=== RUN TestSetClientTLSErrors/missing_cert_file177=== RUN TestRateLimiterFeedback/429_enables_limiter178--- PASS: TestStreamPushReportsSignatures (0.00s)179=== RUN TestPathInfoCACompatibility/old_string_format_-_text180--- PASS: TestResolveStorePath (0.00s)181=== RUN TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths182=== CONT TestStreamPushGivesUpOnDeadServer183=== CONT TestEncodeNixBase32184=== RUN TestParsePathInfoJSON/Nix_format185=== RUN TestScriptTokenDoesNotWaitForItsChildren/cancelled_while_a_child_runs186=== CONT TestDumpPathWriterError187=== PAUSE TestRateLimiterFeedback/429_enables_limiter188=== PAUSE TestPathInfoCACompatibility/old_string_format_-_text189=== CONT TestShellSplitErrors190=== CONT TestStreamPushStopsOnCancel191=== CONT TestStreamPushRequestLine192--- PASS: TestScriptTokenScriptFails (0.00s)193=== PAUSE TestSetClientTLSErrors/missing_cert_file194=== PAUSE TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths195=== PAUSE TestParsePathInfoJSON/Nix_format196=== PAUSE TestScriptTokenDoesNotWaitForItsChildren/cancelled_while_a_child_runs197=== RUN TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)198=== RUN TestSetClientTLS/rejects_connection_without_client_cert199=== RUN TestRateLimiterFeedback/503_enables_limiter200=== RUN TestPathInfoCACompatibility/old_string_format_-_fixed_recursive201=== RUN TestSetClientTLSErrors/missing_key_file202--- PASS: TestStaticToken (0.00s)203=== RUN TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths204=== RUN TestParsePathInfoJSON/Lix_format205=== RUN TestScriptTokenDoesNotWaitForItsChildren/child_left_holding_stdout206=== RUN TestConvertHashToNix32/SRI_format_to_Nix32207=== CONT TestStreamPushIsolatesFailures208=== PAUSE TestCompletePendingClosure_NotFoundIsSuccessWhenClosurePresent/cleaned_up209=== PAUSE TestGetStorePathHash/valid_store_path210--- PASS: TestFileTokenMissing (0.00s)211=== PAUSE TestRateLimiterFeedback/503_enables_limiter212=== PAUSE TestSetClientTLSErrors/missing_key_file213--- PASS: TestClientSignaturesByStorePath (0.00s)214=== PAUSE TestSetClientTLS/rejects_connection_without_client_cert215--- PASS: TestStreamPushStopsWhenCanceled (0.00s)216=== CONT TestStreamPushBatchesUnderLoad217--- PASS: TestFileTokenEmpty (0.00s)218--- PASS: TestSetClientTLSDoesNotMutateDefaultTransport (0.01s)219--- PASS: TestFileTokenReadsAndCaches (0.00s)220=== PAUSE TestScriptTokenDoesNotWaitForItsChildren/child_left_holding_stdout221=== PAUSE TestConvertHashToNix32/SRI_format_to_Nix32222=== PAUSE TestPathInfoCACompatibility/old_string_format_-_fixed_recursive223=== CONT TestPushPathsReportsOnlyUnuploadablePathsOfSkippedClosure224--- PASS: TestStreamPushReportsSkippedPaths (0.01s)225=== CONT TestTruncatedNARDumpIsNotCompleted226=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)227=== PAUSE TestParsePathInfoJSON/Lix_format228--- PASS: TestCompletePendingClosure_NotFoundWithoutKey (0.01s)229=== RUN TestEncodeNixBase32/test_string_hash230--- PASS: TestScriptTokenEmptyToken (0.02s)231=== RUN TestParsePathInfoJSON/empty_input232=== PAUSE TestParsePathInfoJSON/empty_input233=== RUN TestParsePathInfoJSON/whitespace_only234=== RUN TestPathInfoHashCompatibility/old_string_format_with_colon235--- PASS: TestScriptTokenBadJSON (0.02s)236--- PASS: TestEncodeNixBase32WithRealHash (0.00s)237--- PASS: TestStreamPushIsolatesFailures (0.00s)238=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_colon239=== RUN TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI240=== PAUSE TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI241=== RUN TestPathInfoHashCompatibility/new_structured_format_with_sha512242=== PAUSE TestParsePathInfoJSON/whitespace_only243=== RUN TestParsePathInfoJSON/invalid_JSON244=== PAUSE TestParsePathInfoJSON/invalid_JSON245=== PAUSE TestPathInfoHashCompatibility/new_structured_format_with_sha512246=== PAUSE TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths247=== RUN TestGetStorePathHash/basename_without_hyphen_should_error248=== RUN TestStreamPushStopsOnCancel/waiting_for_input249=== CONT TestSupersededNARStillUploadsListing250=== RUN TestRateLimiterFeedback/200_does_not_enable_limiter251=== RUN TestSupersededNARStillUploadsListing/small_NAR252=== PAUSE TestRateLimiterFeedback/200_does_not_enable_limiter253=== CONT TestUploadMultipart_ProducerErrorIsNotEOF254=== RUN TestSetClientTLS/succeeds_with_client_cert_and_CA255=== PAUSE TestSetClientTLS/succeeds_with_client_cert_and_CA256=== RUN TestConvertHashToNix32/already_Nix32_format257=== PAUSE TestConvertHashToNix32/already_Nix32_format258=== RUN TestPathInfoCACompatibility/new_structured_format_-_text259=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_text260=== CONT TestDumpPathSingleFile261=== RUN TestCompletePendingClosure_NotFoundIsSuccessWhenClosurePresent/push_committed_earlier262=== PAUSE TestEncodeNixBase32/test_string_hash263=== PAUSE TestCompletePendingClosure_NotFoundIsSuccessWhenClosurePresent/push_committed_earlier264=== RUN TestSetClientTLSErrors/missing_ca_file265=== PAUSE TestSetClientTLSErrors/missing_ca_file266=== CONT TestUploadMultipart_FailedPartBufferNotReused267--- PASS: TestShellSplitErrors (0.00s)268=== CONT TestUploadMultipart_SupersededByPeer269=== CONT TestPartSizeForNAR270=== PAUSE TestGetStorePathHash/basename_without_hyphen_should_error271=== PAUSE TestStreamPushStopsOnCancel/waiting_for_input272=== RUN TestStreamPushBatchesUnderLoad/together273=== PAUSE TestSupersededNARStillUploadsListing/small_NAR274=== RUN TestUploadMultipart_ProducerErrorIsNotEOF/short_read_inside_a_part275=== RUN TestStreamPushStopsOnCancel/partial_batch_waiting_for_a_slot276=== PAUSE TestStreamPushBatchesUnderLoad/together277=== PAUSE TestStreamPushStopsOnCancel/partial_batch_waiting_for_a_slot278=== RUN TestRateLimiterFeedback/400_does_not_enable_limiter279=== PAUSE TestRateLimiterFeedback/400_does_not_enable_limiter280--- PASS: TestStreamPushGivesUpOnDeadServer (0.01s)281=== RUN TestSetClientTLS/preserves_debug_logging_transport282=== PAUSE TestSetClientTLS/preserves_debug_logging_transport283=== PAUSE TestUploadMultipart_ProducerErrorIsNotEOF/short_read_inside_a_part284=== RUN TestConvertHashToNix32/invalid_format285=== RUN TestEncodeNixBase32/empty_input286=== RUN TestUploadMultipart_ProducerErrorIsNotEOF/empty_read_on_a_part_boundary287=== RUN TestCompletePendingClosure_NotFoundIsSuccessWhenClosurePresent/push_cleaned_up,_first_root_committed_by_another_client288=== RUN TestSetClientTLSErrors/invalid_ca_file289=== RUN TestUploadMultipart_SupersededByPeer/exists290=== RUN TestPartSizeForNAR/zero_stays_at_minimum291=== PAUSE TestPartSizeForNAR/zero_stays_at_minimum292=== RUN TestGetStorePathHash/hash_with_invalid_characters_should_error293=== PAUSE TestGetStorePathHash/hash_with_invalid_characters_should_error294=== PAUSE TestUploadMultipart_ProducerErrorIsNotEOF/empty_read_on_a_part_boundary295=== RUN TestGetStorePathHash/hash_with_wrong_length_should_error296=== CONT TestCaseHackSuffix297=== PAUSE TestConvertHashToNix32/invalid_format298=== RUN TestStreamPushBatchesUnderLoad/one_at_a_time299=== RUN TestPathInfoCACompatibility/new_structured_format_-_nar_method300=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_nar_method301--- PASS: TestScriptTokenNoExpiryRerunsEveryCall (0.03s)302=== CONT TestRetryTimeoutOutlastsMaxRetries303=== RUN TestStreamPushStopsOnCancel/input_closed,_last_batch_waiting_for_a_slot304--- PASS: TestScriptTokenCachesUntilRefresh (0.04s)305=== PAUSE TestStreamPushStopsOnCancel/input_closed,_last_batch_waiting_for_a_slot306=== RUN TestSupersededNARStillUploadsListing/dump_cut_short307=== CONT TestFilterOversizedClosures308=== PAUSE TestEncodeNixBase32/empty_input309=== PAUSE TestSupersededNARStillUploadsListing/dump_cut_short310=== CONT TestRunGarbageCollection_FinishedOnAnotherReplica311=== PAUSE TestCompletePendingClosure_NotFoundIsSuccessWhenClosurePresent/push_cleaned_up,_first_root_committed_by_another_client312=== PAUSE TestSetClientTLSErrors/invalid_ca_file313=== RUN TestPartSizeForNAR/small_stays_at_minimum314=== RUN TestSupersededNARStillUploadsListing/listing_upload_fails315=== PAUSE TestPartSizeForNAR/small_stays_at_minimum316=== CONT TestUploadBuildLog_FileBodyReplayedOnRetry317=== RUN TestPartSizeForNAR/80_GiB_fits_at_minimum318=== PAUSE TestStreamPushBatchesUnderLoad/one_at_a_time319=== PAUSE TestGetStorePathHash/hash_with_wrong_length_should_error320=== PAUSE TestUploadMultipart_SupersededByPeer/exists321=== RUN TestUploadMultipart_SupersededByPeer/missing322=== PAUSE TestUploadMultipart_SupersededByPeer/missing323=== RUN TestStreamPushStopsOnCancel/full_batch_waiting_for_a_slot324=== RUN TestFilterOversizedClosures/no_limit_keeps_everything325=== CONT TestShellSplit326=== PAUSE TestFilterOversizedClosures/no_limit_keeps_everything327=== RUN TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped328=== CONT TestDoWithRetry_FinalResponseBodyReadable329--- PASS: TestShellSplit (0.00s)330=== RUN TestCompletePendingClosure_NotFoundIsSuccessWhenClosurePresent/push_cleaned_up,_last_root_committed_by_another_client331=== PAUSE TestStreamPushStopsOnCancel/full_batch_waiting_for_a_slot332=== RUN TestStreamPushStopsOnCancel/request_line_waiting_for_a_slot333=== PAUSE TestStreamPushStopsOnCancel/request_line_waiting_for_a_slot334=== PAUSE TestCompletePendingClosure_NotFoundIsSuccessWhenClosurePresent/push_cleaned_up,_last_root_committed_by_another_client335=== PAUSE TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped336=== PAUSE TestSupersededNARStillUploadsListing/listing_upload_fails337=== CONT TestStreamPushReportsEveryPath338=== CONT TestRunGarbageCollection_NotFoundAfterLocalRun339=== PAUSE TestPartSizeForNAR/80_GiB_fits_at_minimum340=== CONT TestDoWithRetry_ClosesBodyWhenNoAttemptIsMade341=== CONT TestDoWithRetry_BodyReplayedViaGetBody342=== CONT TestUploadMultipart_PartsInParallel343--- PASS: TestStreamPushReportsEveryPath (0.00s)344=== RUN TestStreamPushStopsOnCancel/lines_read_but_not_taken345=== PAUSE TestStreamPushStopsOnCancel/lines_read_but_not_taken346=== CONT TestDoServerRequestAttachesToken347=== RUN TestPartSizeForNAR/115_GiB_needs_larger_parts348=== RUN TestDoWithRetry_ClosesBodyWhenNoAttemptIsMade/retries=0349=== RUN TestFilterOversizedClosures/all_closures_skipped350=== CONT TestScriptTokenDoesNotWaitForItsChildren/child_left_holding_stdout351--- PASS: TestDoWithRetry_FinalResponseBodyReadable (0.01s)352=== CONT TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI353=== PAUSE TestFilterOversizedClosures/all_closures_skipped354=== CONT TestPathInfoHashCompatibility/new_structured_format_with_sha512355=== CONT TestPathInfoHashCompatibility/old_string_format_with_colon356=== CONT TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths357--- PASS: TestRunGarbageCollection_NotFoundAfterLocalRun (0.01s)358=== CONT TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths359=== CONT TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)360=== CONT TestScriptTokenDoesNotWaitForItsChildren/cancelled_while_a_child_runs361=== PAUSE TestPartSizeForNAR/115_GiB_needs_larger_parts362=== RUN TestPartSizeForNAR/1_TiB363=== PAUSE TestPartSizeForNAR/1_TiB364=== RUN TestPartSizeForNAR/5_TiB_S3_max_object365=== PAUSE TestPartSizeForNAR/5_TiB_S3_max_object366=== RUN TestPartSizeForNAR/capped_at_5_GiB367=== PAUSE TestPartSizeForNAR/capped_at_5_GiB368=== PAUSE TestDoWithRetry_ClosesBodyWhenNoAttemptIsMade/retries=0369=== RUN TestDoWithRetry_ClosesBodyWhenNoAttemptIsMade/retries=5370=== PAUSE TestDoWithRetry_ClosesBodyWhenNoAttemptIsMade/retries=5371=== CONT TestParsePathInfoJSON/Nix_format372=== CONT TestParsePathInfoJSON/whitespace_only373--- PASS: TestDoServerRequestAttachesToken (0.02s)374=== CONT TestParsePathInfoJSON/invalid_JSON375=== CONT TestParsePathInfoJSON/empty_input376=== CONT TestParsePathInfoJSON/Lix_format377=== CONT TestRateLimiterFeedback/429_enables_limiter378--- PASS: TestUploadBuildLog_FileBodyReplayedOnRetry (0.02s)379=== CONT TestSetClientTLS/rejects_connection_without_client_cert380=== CONT TestRateLimiterFeedback/200_does_not_enable_limiter381=== CONT TestRateLimiterFeedback/400_does_not_enable_limiter382--- PASS: TestParsePathInfoJSON (0.02s)383 --- PASS: TestParsePathInfoJSON/whitespace_only (0.00s)384 --- PASS: TestParsePathInfoJSON/invalid_JSON (0.00s)385 --- PASS: TestParsePathInfoJSON/Nix_format (0.00s)386 --- PASS: TestParsePathInfoJSON/empty_input (0.00s)387 --- PASS: TestParsePathInfoJSON/Lix_format (0.00s)388=== CONT TestRateLimiterFeedback/503_enables_limiter389=== CONT TestSetClientTLS/preserves_debug_logging_transport390=== CONT TestSetClientTLS/succeeds_with_client_cert_and_CA391=== CONT TestUploadMultipart_ProducerErrorIsNotEOF/empty_read_on_a_part_boundary392--- PASS: TestDoWithRetry_BodyReplayedViaGetBody (0.01s)393--- PASS: TestRunGarbageCollection_FinishedOnAnotherReplica (0.01s)394--- PASS: TestPathInfoHashCompatibility (0.02s)395 --- PASS: TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI (0.00s)396 --- PASS: TestPathInfoHashCompatibility/new_structured_format_with_sha512 (0.00s)397 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_colon (0.00s)398 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI) (0.00s)399=== CONT TestUploadMultipart_ProducerErrorIsNotEOF/short_read_inside_a_part400--- PASS: TestParsePathInfoJSONMultiplePaths (0.02s)401 --- PASS: TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths (0.00s)402 --- PASS: TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths (0.00s)403=== CONT TestConvertHashToNix32/already_Nix32_format404=== CONT TestConvertHashToNix32/invalid_format405=== CONT TestConvertHashToNix32/SRI_format_to_Nix32406=== CONT TestPathInfoCACompatibility/null_ca_field407=== CONT TestPathInfoCACompatibility/new_structured_format_-_text408=== CONT TestPathInfoCACompatibility/new_structured_format_-_nar_method409=== CONT TestPathInfoCACompatibility/old_string_format_-_text410=== CONT TestPathInfoCACompatibility/old_string_format_-_fixed_recursive411=== CONT TestEncodeNixBase32/test_string_hash412--- PASS: TestPathInfoCACompatibility (0.04s)413 --- PASS: TestPathInfoCACompatibility/null_ca_field (0.00s)414 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_text (0.00s)415 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_nar_method (0.00s)416 --- PASS: TestPathInfoCACompatibility/old_string_format_-_text (0.00s)417 --- PASS: TestPathInfoCACompatibility/old_string_format_-_fixed_recursive (0.00s)418=== CONT TestEncodeNixBase32/empty_input419--- PASS: TestEncodeNixBase32 (0.02s)420 --- PASS: TestEncodeNixBase32/test_string_hash (0.00s)421 --- PASS: TestEncodeNixBase32/empty_input (0.00s)422=== CONT TestSetClientTLSErrors/missing_key_file423=== CONT TestSetClientTLSErrors/invalid_ca_file424=== CONT TestSetClientTLSErrors/missing_ca_file425=== CONT TestSetClientTLSErrors/missing_cert_file426=== CONT TestGetStorePathHash/valid_store_path427=== CONT TestStreamPushBatchesUnderLoad/one_at_a_time428--- PASS: TestConvertHashToNix32 (0.02s)429 --- PASS: TestConvertHashToNix32/already_Nix32_format (0.00s)430 --- PASS: TestConvertHashToNix32/invalid_format (0.00s)431 --- PASS: TestConvertHashToNix32/SRI_format_to_Nix32 (0.01s)432=== CONT TestGetStorePathHash/hash_with_wrong_length_should_error433=== CONT TestGetStorePathHash/hash_with_invalid_characters_should_error434=== CONT TestUploadMultipart_SupersededByPeer/missing435--- PASS: TestSetClientTLSErrors (0.04s)436 --- PASS: TestSetClientTLSErrors/missing_key_file (0.00s)437 --- PASS: TestSetClientTLSErrors/invalid_ca_file (0.00s)438 --- PASS: TestSetClientTLSErrors/missing_ca_file (0.00s)439 --- PASS: TestSetClientTLSErrors/missing_cert_file (0.00s)440=== CONT TestUploadMultipart_SupersededByPeer/exists441=== CONT TestGetStorePathHash/basename_without_hyphen_should_error442--- PASS: TestRateLimiterFeedback (0.03s)443 --- PASS: TestRateLimiterFeedback/400_does_not_enable_limiter (0.01s)444 --- PASS: TestRateLimiterFeedback/200_does_not_enable_limiter (0.01s)445 --- PASS: TestRateLimiterFeedback/503_enables_limiter (0.01s)446 --- PASS: TestRateLimiterFeedback/429_enables_limiter (0.03s)447=== CONT TestStreamPushBatchesUnderLoad/together448--- PASS: TestGetStorePathHash (0.04s)449 --- PASS: TestGetStorePathHash/valid_store_path (0.00s)450 --- PASS: TestGetStorePathHash/hash_with_wrong_length_should_error (0.00s)451 --- PASS: TestGetStorePathHash/hash_with_invalid_characters_should_error (0.00s)452 --- PASS: TestGetStorePathHash/basename_without_hyphen_should_error (0.00s)453=== CONT TestCompletePendingClosure_NotFoundIsSuccessWhenClosurePresent/push_cleaned_up,_first_root_committed_by_another_client454=== CONT TestCompletePendingClosure_NotFoundIsSuccessWhenClosurePresent/push_cleaned_up,_last_root_committed_by_another_client455=== CONT TestCompletePendingClosure_NotFoundIsSuccessWhenClosurePresent/push_committed_earlier456=== CONT TestCompletePendingClosure_NotFoundIsSuccessWhenClosurePresent/cleaned_up457--- PASS: TestStreamPushBatchesUnderLoad (0.02s)458 --- PASS: TestStreamPushBatchesUnderLoad/one_at_a_time (0.01s)459 --- PASS: TestStreamPushBatchesUnderLoad/together (0.01s)460=== CONT TestCompletePendingClosure_NotFoundIsSuccessWhenClosurePresent/committed_earlier461=== CONT TestSupersededNARStillUploadsListing/listing_upload_fails462--- PASS: TestUploadMultipart_SupersededByPeer (0.00s)463 --- PASS: TestUploadMultipart_SupersededByPeer/exists (0.01s)464 --- PASS: TestUploadMultipart_SupersededByPeer/missing (0.01s)465--- PASS: TestDumpPathSingleFile (0.09s)466=== CONT TestSupersededNARStillUploadsListing/dump_cut_short467=== CONT TestSupersededNARStillUploadsListing/small_NAR468=== CONT TestStreamPushStopsOnCancel/waiting_for_input469=== CONT TestStreamPushStopsOnCancel/lines_read_but_not_taken470=== CONT TestStreamPushStopsOnCancel/full_batch_waiting_for_a_slot471=== CONT TestStreamPushStopsOnCancel/input_closed,_last_batch_waiting_for_a_slot472=== CONT TestStreamPushStopsOnCancel/request_line_waiting_for_a_slot473=== CONT TestFilterOversizedClosures/all_closures_skipped474=== CONT TestFilterOversizedClosures/no_limit_keeps_everything475--- PASS: TestCompletePendingClosure_NotFoundIsSuccessWhenClosurePresent (0.04s)476 --- PASS: TestCompletePendingClosure_NotFoundIsSuccessWhenClosurePresent/push_cleaned_up,_last_root_committed_by_another_client (0.02s)477 --- PASS: TestCompletePendingClosure_NotFoundIsSuccessWhenClosurePresent/cleaned_up (0.02s)478 --- PASS: TestCompletePendingClosure_NotFoundIsSuccessWhenClosurePresent/push_cleaned_up,_first_root_committed_by_another_client (0.03s)479 --- PASS: TestCompletePendingClosure_NotFoundIsSuccessWhenClosurePresent/push_committed_earlier (0.04s)480 --- PASS: TestCompletePendingClosure_NotFoundIsSuccessWhenClosurePresent/committed_earlier (0.04s)481=== CONT TestDoWithRetry_ClosesBodyWhenNoAttemptIsMade/retries=0482--- PASS: TestUploadMultipart_ProducerErrorIsNotEOF (0.01s)483 --- PASS: TestUploadMultipart_ProducerErrorIsNotEOF/empty_read_on_a_part_boundary (0.06s)484 --- PASS: TestUploadMultipart_ProducerErrorIsNotEOF/short_read_inside_a_part (0.08s)485--- PASS: TestStreamPushRequestLine (0.14s)486--- PASS: TestCaseHackSuffix (0.12s)487=== CONT TestStreamPushStopsOnCancel/partial_batch_waiting_for_a_slot488=== CONT TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped489=== CONT TestDoWithRetry_ClosesBodyWhenNoAttemptIsMade/retries=5490=== CONT TestPartSizeForNAR/5_TiB_S3_max_object491=== CONT TestPartSizeForNAR/zero_stays_at_minimum492=== CONT TestPartSizeForNAR/1_TiB493=== CONT TestPartSizeForNAR/capped_at_5_GiB494=== CONT TestPartSizeForNAR/115_GiB_needs_larger_parts495=== CONT TestPartSizeForNAR/small_stays_at_minimum496--- PASS: TestFilterOversizedClosures (0.01s)497 --- PASS: TestFilterOversizedClosures/all_closures_skipped (0.00s)498 --- PASS: TestFilterOversizedClosures/no_limit_keeps_everything (0.00s)499 --- PASS: TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped (0.00s)500=== CONT TestPartSizeForNAR/80_GiB_fits_at_minimum501--- PASS: TestDoWithRetry_ClosesBodyWhenNoAttemptIsMade (0.02s)502 --- PASS: TestDoWithRetry_ClosesBodyWhenNoAttemptIsMade/retries=0 (0.00s)503 --- PASS: TestDoWithRetry_ClosesBodyWhenNoAttemptIsMade/retries=5 (0.00s)504--- PASS: TestPartSizeForNAR (0.02s)505 --- PASS: TestPartSizeForNAR/5_TiB_S3_max_object (0.00s)506 --- PASS: TestPartSizeForNAR/zero_stays_at_minimum (0.00s)507 --- PASS: TestPartSizeForNAR/115_GiB_needs_larger_parts (0.00s)508 --- PASS: TestPartSizeForNAR/1_TiB (0.00s)509 --- PASS: TestPartSizeForNAR/small_stays_at_minimum (0.00s)510 --- PASS: TestPartSizeForNAR/80_GiB_fits_at_minimum (0.00s)511 --- PASS: TestPartSizeForNAR/capped_at_5_GiB (0.00s)512--- PASS: TestDumpPathWriterError (0.15s)513--- PASS: TestRetryTimeoutOutlastsMaxRetries (0.13s)514--- PASS: TestSetClientTLS (0.04s)515 --- PASS: TestSetClientTLS/succeeds_with_client_cert_and_CA (0.03s)516 --- PASS: TestSetClientTLS/preserves_debug_logging_transport (0.04s)517 --- PASS: TestSetClientTLS/rejects_connection_without_client_cert (0.11s)518--- PASS: TestStreamPushStopsOnCancel (0.02s)519 --- PASS: TestStreamPushStopsOnCancel/waiting_for_input (0.05s)520 --- PASS: TestStreamPushStopsOnCancel/input_closed,_last_batch_waiting_for_a_slot (0.05s)521 --- PASS: TestStreamPushStopsOnCancel/lines_read_but_not_taken (0.05s)522 --- PASS: TestStreamPushStopsOnCancel/partial_batch_waiting_for_a_slot (0.05s)523 --- PASS: TestStreamPushStopsOnCancel/full_batch_waiting_for_a_slot (0.05s)524 --- PASS: TestStreamPushStopsOnCancel/request_line_waiting_for_a_slot (0.05s)525--- PASS: TestDumpPathMatchesNix (0.24s)526--- PASS: TestRegisterUploadedObjectReusesConnections (0.29s)527--- PASS: TestPushPathsReportsOnlyUnuploadablePathsOfSkippedClosure (0.27s)528--- PASS: TestTruncatedNARDumpIsNotCompleted (0.45s)529--- PASS: TestUploadMultipart_FailedPartBufferNotReused (0.67s)530--- PASS: TestUploadMultipart_PartsInParallel (0.68s)531--- PASS: TestSupersededNARStillUploadsListing (0.01s)532 --- PASS: TestSupersededNARStillUploadsListing/listing_upload_fails (0.06s)533 --- PASS: TestSupersededNARStillUploadsListing/small_NAR (0.04s)534 --- PASS: TestSupersededNARStillUploadsListing/dump_cut_short (0.65s)535--- PASS: TestRateLimiterFeedback_400DoesNotCountAsSuccess (1.00s)536--- PASS: TestRegisterUploadedObject_BoundedAgainstSilentServer (2.01s)537--- PASS: TestScriptTokenDoesNotWaitForItsChildren (0.01s)538 --- PASS: TestScriptTokenDoesNotWaitForItsChildren/child_left_holding_stdout (2.05s)539 --- PASS: TestScriptTokenDoesNotWaitForItsChildren/cancelled_while_a_child_runs (2.21s)540PASS541Running server tests...542The files belonging to this database system will be owned by user "_tribuchetbld5".543This user must also own the server process.544545The database cluster will be initialized with locale "C".546The default database encoding has accordingly been set to "SQL_ASCII".547The default text search configuration will be set to "english".548549Data page checksums are enabled.550551creating directory /private/var/tb/5/scratch/00f1341bdd59a71e/build/postgres1970240236/data ... ok552creating subdirectories ... ok553selecting dynamic shared memory implementation ... posix554selecting default "max_connections" ... 100555selecting default "shared_buffers" ... 128MB556selecting default time zone ... UTC557creating configuration files ... ok558running bootstrap script ... ok559performing post-bootstrap initialization ... ok560syncing data to disk ... ok561562initdb: warning: enabling "trust" authentication for local connections563initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.564565Success. You can now start the database server using:566567 pg_ctl -D /private/var/tb/5/scratch/00f1341bdd59a71e/build/postgres1970240236/data -l logfile start5685692026-10-09 09:57:30.815 UTC [65329] LOG: starting PostgreSQL 18.6 on aarch64-apple-darwin25.6.0, compiled by clang version 21.1.8, 64-bit5702026-10-09 09:57:30.815 UTC [65329] LOG: listening on Unix socket "/private/var/tb/5/scratch/00f1341bdd59a71e/build/postgres1970240236/.s.PGSQL.5432"5712026-10-09 09:57:30.817 UTC [65336] LOG: database system was shut down at 2026-10-09 09:57:30 UTC5722026-10-09 09:57:30.818 UTC [65329] LOG: database system is ready to accept connections573/private/var/tb/5/scratch/00f1341bdd59a71e/build/postgres1970240236:5432 - accepting connections574{"timestamp":"2026-10-09T09:57:33.008729Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"4d6c9634-721f-478f-ab55-734248db51ce","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"GET","uri":"/health/ready","status_code":503,"suppressed_errors":0,"duration_ms":1,"result":"server_error","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":501,"threadName":"rustfs-worker","threadId":"ThreadId(4)"}575=== RUN TestService_AuthMiddleware576=== PAUSE TestService_AuthMiddleware577=== RUN TestService_AuthMiddleware_MTLSProxyHeader578=== PAUSE TestService_AuthMiddleware_MTLSProxyHeader579=== RUN TestService_AuthMiddleware_MTLSBoundSubjects580=== PAUSE TestService_AuthMiddleware_MTLSBoundSubjects581=== RUN TestService_ReadAuthMiddleware582=== PAUSE TestService_ReadAuthMiddleware583=== RUN TestService_AuthMiddleware_OIDC584=== PAUSE TestService_AuthMiddleware_OIDC585=== RUN TestService_RequireScope_OIDC586=== PAUSE TestService_RequireScope_OIDC587=== RUN TestService_ReadScope_PublicByDefault588=== PAUSE TestService_ReadScope_PublicByDefault589=== RUN TestCacheConfigHandler590=== PAUSE TestCacheConfigHandler591=== RUN TestCacheStatsHandler592=== PAUSE TestCacheStatsHandler593=== RUN TestClientCADerivations594=== PAUSE TestClientCADerivations595=== RUN TestClientErrorHandling596=== PAUSE TestClientErrorHandling597=== RUN TestClientIntegration598=== PAUSE TestClientIntegration599=== RUN TestClientMultipleUploads600=== PAUSE TestClientMultipleUploads601=== RUN TestClientWithDependencies602=== PAUSE TestClientWithDependencies603=== RUN TestClientSharedPathCommittedMidPush604=== PAUSE TestClientSharedPathCommittedMidPush605=== RUN TestPinProtectsFromGC606=== PAUSE TestPinProtectsFromGC607=== RUN TestClientPushesUseOnePush608=== PAUSE TestClientPushesUseOnePush609=== RUN TestClientFallsBackToClosures610=== PAUSE TestClientFallsBackToClosures611=== RUN TestDeleteClosuresRacingPins612=== PAUSE TestDeleteClosuresRacingPins613=== RUN TestConcurrentCommitsSharingObjectsDoNotDeadlock614=== PAUSE TestConcurrentCommitsSharingObjectsDoNotDeadlock615=== RUN TestResolveDBConnectionString616=== PAUSE TestResolveDBConnectionString617=== RUN TestConnectWaitsForAPeerMigration618=== PAUSE TestConnectWaitsForAPeerMigration619=== RUN TestConnectSerialisesConcurrentMigrations620=== PAUSE TestConnectSerialisesConcurrentMigrations621=== RUN TestLeadElectsOneAndHandsOver622=== PAUSE TestLeadElectsOneAndHandsOver623=== RUN TestLeadIncumbentWinsAfterRestart6242026/10/09 09:57:33 INFO lead: acquired remote=192.0.2.1:12346252026/10/09 09:57:34 INFO lead: released remote=192.0.2.1:12346262026/10/09 09:57:34 INFO lead: acquired remote=192.0.2.1:12346272026/10/09 09:57:35 INFO lead: released remote=192.0.2.1:1234628--- PASS: TestLeadIncumbentWinsAfterRestart (2.08s)629=== RUN TestLeadIncumbentAnnouncesAtOnce630=== PAUSE TestLeadIncumbentAnnouncesAtOnce631=== RUN TestLeadEndsOnShutdown632=== PAUSE TestLeadEndsOnShutdown633=== RUN TestLeadEndsWhenItsConnectionHangs6342026/10/09 09:57:35 INFO lead: acquired remote=192.0.2.1:12346352026-10-09 09:57:36.548 UTC [65470] FATAL: terminating connection due to administrator command6362026/10/09 09:57:36 INFO lead: acquired remote=192.0.2.1:12346372026/10/09 09:57:37 WARN lead: lock connection lost error="timeout: context deadline exceeded"6382026/10/09 09:57:37 INFO lead: released remote=192.0.2.1:12346392026/10/09 09:57:37 INFO lead: released remote=192.0.2.1:1234640--- PASS: TestLeadEndsWhenItsConnectionHangs (2.76s)641=== RUN TestGCAdvisoryLockBlocksConcurrentRun642=== PAUSE TestGCAdvisoryLockBlocksConcurrentRun643=== RUN TestGCBugBareHashReferences644=== PAUSE TestGCBugBareHashReferences645=== RUN TestGCMetrics646=== PAUSE TestGCMetrics647=== RUN TestPushDedupSurvivesConcurrentGC648=== PAUSE TestPushDedupSurvivesConcurrentGC649=== RUN TestDeduplicatedObjectsRecordedAsPending650=== PAUSE TestDeduplicatedObjectsRecordedAsPending651=== RUN TestGCSweepSkipsPendingObjects652=== PAUSE TestGCSweepSkipsPendingObjects653=== RUN TestTombstonedObjectOfferedWithoutWaiting654=== PAUSE TestTombstonedObjectOfferedWithoutWaiting655=== RUN TestGCSweepDeliversEachKeyOnce656=== PAUSE TestGCSweepDeliversEachKeyOnce657=== RUN TestCreatePendingClosureVerifyS3FailureReleasesConnection658=== PAUSE TestCreatePendingClosureVerifyS3FailureReleasesConnection659=== RUN TestForceGCDuringPushOffersSweptObject660=== PAUSE TestForceGCDuringPushOffersSweptObject661=== RUN TestFailedS3DeleteKeepsTombstone662=== PAUSE TestFailedS3DeleteKeepsTombstone663=== RUN TestSweepSkipsRowsLockedByPush664=== PAUSE TestSweepSkipsRowsLockedByPush665=== RUN TestRegisterCompletedObjectNeedsPendingRow666=== PAUSE TestRegisterCompletedObjectNeedsPendingRow667=== RUN TestGetExistingObjectsLocksRows668=== PAUSE TestGetExistingObjectsLocksRows669=== RUN TestSweepRechecksPendingAfterLocking670=== PAUSE TestSweepRechecksPendingAfterLocking671=== RUN TestPushWaitsForSweepHoldingRow672=== PAUSE TestPushWaitsForSweepHoldingRow673=== RUN TestCommitRacingPendingCleanupKeepsObjects674=== PAUSE TestCommitRacingPendingCleanupKeepsObjects675=== RUN TestGCEndsOnShutdown676=== PAUSE TestGCEndsOnShutdown677=== RUN TestGCTaskStore_StartNew678=== PAUSE TestGCTaskStore_StartNew679=== RUN TestGCTaskStore_DeduplicateSameParams680=== PAUSE TestGCTaskStore_DeduplicateSameParams681=== RUN TestGCTaskStore_ConflictDifferentParams682=== PAUSE TestGCTaskStore_ConflictDifferentParams683=== RUN TestGCTaskStore_GetEmpty684=== PAUSE TestGCTaskStore_GetEmpty685=== RUN TestGCTaskStore_GetReturnsLatest686=== PAUSE TestGCTaskStore_GetReturnsLatest687=== RUN TestGCTaskStore_CompletedAllowsNewTask688=== PAUSE TestGCTaskStore_CompletedAllowsNewTask689=== RUN TestGCTaskStore_PhaseUpdates690=== PAUSE TestGCTaskStore_PhaseUpdates691=== RUN TestGCTaskStore_Fail692=== PAUSE TestGCTaskStore_Fail693=== RUN TestGracefulShutdownDrainsInflight694=== PAUSE TestGracefulShutdownDrainsInflight695=== RUN TestService_healthCheckHandler696=== PAUSE TestService_healthCheckHandler697=== RUN TestService_readinessHandler698=== PAUSE TestService_readinessHandler699=== RUN TestGenerateLandingPage700=== PAUSE TestGenerateLandingPage701=== RUN TestCacheConfigHandlerMaxNarSize702=== PAUSE TestCacheConfigHandlerMaxNarSize703=== RUN TestCreatePendingClosureRejectsOversizedNAR704=== PAUSE TestCreatePendingClosureRejectsOversizedNAR705=== RUN TestNARDeduplicationMetadataUploadBug706=== PAUSE TestNARDeduplicationMetadataUploadBug707=== RUN TestMetricsInventory708=== PAUSE TestMetricsInventory709=== RUN TestMetricsCommit710=== PAUSE TestMetricsCommit711=== RUN TestService_NativeMTLS712=== PAUSE TestService_NativeMTLS713=== RUN TestServerTLSConfig714=== PAUSE TestServerTLSConfig715=== RUN TestMultipartCleanup716=== PAUSE TestMultipartCleanup717=== RUN TestMultipartUploadAbortedWhenCancelledBeforeRecorded718=== PAUSE TestMultipartUploadAbortedWhenCancelledBeforeRecorded719=== RUN TestPendingCleanupUsesOneCutoff720=== PAUSE TestPendingCleanupUsesOneCutoff721=== RUN TestPendingClosureFailureTracksEveryUpload722=== PAUSE TestPendingClosureFailureTracksEveryUpload723=== RUN TestPendingClosureFailureAbortsItsUploads724=== PAUSE TestPendingClosureFailureAbortsItsUploads725=== RUN TestObjectStatsTrigger726=== PAUSE TestObjectStatsTrigger727=== RUN TestObjectStatsLargeCommit728=== PAUSE TestObjectStatsLargeCommit729=== RUN TestReconnectLeavesObjectsUnlocked730=== PAUSE TestReconnectLeavesObjectsUnlocked731=== RUN TestOrphanedObjectsGC732=== PAUSE TestOrphanedObjectsGC733=== RUN TestOrphanedObjectsGCStressTest734=== PAUSE TestOrphanedObjectsGCStressTest735=== RUN TestCreatePin_ReservedPins736=== PAUSE TestCreatePin_ReservedPins737=== RUN TestConcurrentPinUpdatesAgree738=== PAUSE TestConcurrentPinUpdatesAgree739=== RUN TestCreatePinRejectsBadInput740=== PAUSE TestCreatePinRejectsBadInput741=== RUN TestDeletePinKeepsRowWhenS3Fails742=== PAUSE TestDeletePinKeepsRowWhenS3Fails743=== RUN TestPresentReportsOnlyClosureRoots744=== PAUSE TestPresentReportsOnlyClosureRoots745=== RUN TestPresentRefreshesOnlyStaleClosures746=== PAUSE TestPresentRefreshesOnlyStaleClosures747=== RUN TestPresentNotReportedWhileGCDeletesClosure748=== PAUSE TestPresentNotReportedWhileGCDeletesClosure749=== RUN TestParseSingleRange750=== PAUSE TestParseSingleRange751=== RUN TestProxyHeadersOnlyTrustedOnSocket7522026/10/09 09:57:38 INFO Starting HTTP server address=127.0.0.1:611707532026/10/09 09:57:38 INFO Starting HTTP server address=proxy.sock7542026/10/09 09:57:38 WARN mTLS auth: subject not in bound subjects subject="CN=someone"7552026/10/09 09:57:38 INFO Shutdown signal received, draining in-flight requests timeout=10s756--- PASS: TestProxyHeadersOnlyTrustedOnSocket (0.15s)757=== RUN TestIsValidCachePath758=== PAUSE TestIsValidCachePath759=== RUN TestReadProxyNarinfo760=== PAUSE TestReadProxyNarinfo761=== RUN TestReadProxyNarinfoAlreadyDecompressed762=== PAUSE TestReadProxyNarinfoAlreadyDecompressed763=== RUN TestReadProxyContentNegotiation764=== PAUSE TestReadProxyContentNegotiation765=== RUN TestReadProxyNarStreaming766=== PAUSE TestReadProxyNarStreaming767=== RUN TestReadProxy404768=== PAUSE TestReadProxy404769=== RUN TestReadProxyInvalidPath770=== PAUSE TestReadProxyInvalidPath771=== RUN TestReadProxyHead772=== PAUSE TestReadProxyHead773=== RUN TestReadProxyOutlastsServerWriteTimeout774=== PAUSE TestReadProxyOutlastsServerWriteTimeout775=== RUN TestReadProxyConditionalGet776=== PAUSE TestReadProxyConditionalGet777=== RUN TestReadProxyRootRedirectsToIndexHTML778=== PAUSE TestReadProxyRootRedirectsToIndexHTML779=== RUN TestReadProxyDisabled780=== PAUSE TestReadProxyDisabled781=== RUN TestReadRedirectNar782=== PAUSE TestReadRedirectNar783=== RUN TestReadRedirectKeepsNarinfoProxied784=== PAUSE TestReadRedirectKeepsNarinfoProxied785=== RUN TestReadProxyRangeRequest786=== PAUSE TestReadProxyRangeRequest787=== RUN TestReadRedirectUsesPublicS3URL788=== PAUSE TestReadRedirectUsesPublicS3URL789=== RUN TestPush_OverlappingRootsStoreOneRowPerKey790=== PAUSE TestPush_OverlappingRootsStoreOneRowPerKey791=== RUN TestPush_CompleteCommitsEveryRoot792=== PAUSE TestPush_CompleteCommitsEveryRoot793=== RUN TestPush_RepeatedRootCommitsOnce794=== PAUSE TestPush_RepeatedRootCommitsOnce795=== RUN TestPush_CompleteWaitsForCleanupOfThePush796=== PAUSE TestPush_CompleteWaitsForCleanupOfThePush797=== RUN TestPush_RootsAreLockedInByteOrder798=== PAUSE TestPush_RootsAreLockedInByteOrder799=== RUN TestPush_SkippedKeySurvivesGCBeforeCommit800=== PAUSE TestPush_SkippedKeySurvivesGCBeforeCommit801=== RUN TestPush_RejectsBadRequests802=== PAUSE TestPush_RejectsBadRequests803=== RUN TestPush_SignsNarinfosOfItsPendingObjects804=== PAUSE TestPush_SignsNarinfosOfItsPendingObjects805=== RUN TestRedundantMultipartUpload806=== PAUSE TestRedundantMultipartUpload807=== RUN TestCompleteMultipartUpload_ErrorButObjectExists808=== PAUSE TestCompleteMultipartUpload_ErrorButObjectExists809=== RUN TestCompletedNarNotReofferedAcrossClosures810=== PAUSE TestCompletedNarNotReofferedAcrossClosures811=== RUN TestPresignedUploadRegisteredBeforeCommit812=== PAUSE TestPresignedUploadRegisteredBeforeCommit813=== RUN TestService_Rustfstest814=== PAUSE TestService_Rustfstest815=== RUN TestParseSize816=== PAUSE TestParseSize817=== RUN TestSkippedUploadsHandler818=== PAUSE TestSkippedUploadsHandler819=== RUN TestSystemdListenerNotActivated820--- PASS: TestSystemdListenerNotActivated (0.00s)821=== RUN TestWatchdogBeatsWhenHealthy822--- PASS: TestWatchdogBeatsWhenHealthy (0.02s)823=== RUN TestWatchdogSkipsWhenUnhealthy8242026/10/09 09:57:38 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"8252026/10/09 09:57:38 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"8262026/10/09 09:57:38 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"8272026/10/09 09:57:38 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"8282026/10/09 09:57:38 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"8292026/10/09 09:57:38 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"8302026/10/09 09:57:38 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"8312026/10/09 09:57:38 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"8322026/10/09 09:57:38 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"8332026/10/09 09:57:38 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"834--- PASS: TestWatchdogSkipsWhenUnhealthy (0.20s)835=== RUN TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle836=== PAUSE TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle837=== RUN TestProxyWriteTimeout838=== PAUSE TestProxyWriteTimeout839=== RUN TestPendingClosureWriteTimeout840=== PAUSE TestPendingClosureWriteTimeout841=== RUN TestIsValidUploadKey842=== PAUSE TestIsValidUploadKey843=== RUN TestUploadHandlersRejectInvalidKeys844=== PAUSE TestUploadHandlersRejectInvalidKeys845=== RUN TestUploadHandlersRejectOversizedBody846=== PAUSE TestUploadHandlersRejectOversizedBody847=== RUN TestService_cleanupPendingClosuresHandler848=== PAUSE TestService_cleanupPendingClosuresHandler849=== RUN TestService_createPendingClosureHandler850=== PAUSE TestService_createPendingClosureHandler851=== RUN TestService_verifyS3Integrity852=== PAUSE TestService_verifyS3Integrity853=== RUN TestCompleteMultipartUnregistered854=== PAUSE TestCompleteMultipartUnregistered855=== RUN TestCreatePendingClosure_SmallNARUsesSimplePUT856=== PAUSE TestCreatePendingClosure_SmallNARUsesSimplePUT857=== CONT TestService_AuthMiddleware858=== CONT TestService_createPendingClosureHandler859=== CONT TestGCTaskStore_CompletedAllowsNewTask860=== CONT TestPendingClosureFailureAbortsItsUploads861--- PASS: TestGCTaskStore_CompletedAllowsNewTask (0.00s)862=== CONT TestIsValidUploadKey863=== CONT TestOrphanedObjectsGC864=== CONT TestService_verifyS3Integrity865=== CONT TestService_readinessHandler866=== CONT TestFailedS3DeleteKeepsTombstone867=== CONT TestCompleteMultipartUnregistered868=== CONT TestService_cleanupPendingClosuresHandler869=== CONT TestGracefulShutdownDrainsInflight870=== CONT TestObjectStatsLargeCommit871=== CONT TestReconnectLeavesObjectsUnlocked872=== CONT TestConcurrentCommitsSharingObjectsDoNotDeadlock873=== CONT TestReadRedirectKeepsNarinfoProxied874=== CONT TestCreatePendingClosureVerifyS3FailureReleasesConnection875=== CONT TestService_healthCheckHandler876=== CONT TestForceGCDuringPushOffersSweptObject877=== RUN TestForceGCDuringPushOffersSweptObject/before_pending_rows878=== CONT TestPendingClosureFailureTracksEveryUpload879=== PAUSE TestForceGCDuringPushOffersSweptObject/before_pending_rows880=== CONT TestUploadHandlersRejectOversizedBody881=== CONT TestUploadHandlersRejectInvalidKeys882=== CONT TestPendingClosureWriteTimeout883=== CONT TestObjectStatsTrigger884=== RUN TestIsValidUploadKey/narinfo885=== CONT TestCreatePendingClosure_SmallNARUsesSimplePUT886=== RUN TestPendingClosureFailureTracksEveryUpload/refuse_aborts=false8872026/10/09 09:57:38 INFO Starting HTTP server address=127.0.0.1:61179888=== RUN TestForceGCDuringPushOffersSweptObject/after_presence_check889=== RUN TestPendingClosureWriteTimeout/empty890=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info891=== PAUSE TestIsValidUploadKey/narinfo892=== PAUSE TestPendingClosureWriteTimeout/empty893=== RUN TestIsValidUploadKey/nar_zst894=== RUN TestPendingClosureWriteTimeout/400_objects895=== PAUSE TestForceGCDuringPushOffersSweptObject/after_presence_check896=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info897=== PAUSE TestPendingClosureWriteTimeout/400_objects898=== PAUSE TestIsValidUploadKey/nar_zst899=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal900=== RUN TestPendingClosureWriteTimeout/670k_objects901=== RUN TestIsValidUploadKey/nar_xz902=== CONT TestGCSweepDeliversEachKeyOnce903=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal904=== RUN TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key905=== PAUSE TestPendingClosureWriteTimeout/670k_objects906=== RUN TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline907=== PAUSE TestIsValidUploadKey/nar_xz908=== PAUSE TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key909=== RUN TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key910=== PAUSE TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key911=== CONT TestSkippedUploadsHandler912=== PAUSE TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline913=== PAUSE TestPendingClosureFailureTracksEveryUpload/refuse_aborts=false914=== RUN TestPendingClosureFailureTracksEveryUpload/refuse_aborts=true915=== CONT TestPendingCleanupUsesOneCutoff916=== RUN TestIsValidUploadKey/nar_plain917=== PAUSE TestIsValidUploadKey/nar_plain918=== RUN TestIsValidUploadKey/listing919=== PAUSE TestIsValidUploadKey/listing920=== RUN TestIsValidUploadKey/build_log921=== PAUSE TestIsValidUploadKey/build_log922=== RUN TestIsValidUploadKey/build_log_home-manager_file923=== PAUSE TestIsValidUploadKey/build_log_home-manager_file924=== RUN TestIsValidUploadKey/build_log_plus_in_name925=== PAUSE TestIsValidUploadKey/build_log_plus_in_name926=== RUN TestIsValidUploadKey/build_log_question_mark927=== PAUSE TestIsValidUploadKey/build_log_question_mark928=== RUN TestIsValidUploadKey/build_log_equals929=== PAUSE TestIsValidUploadKey/build_log_equals930=== RUN TestIsValidUploadKey/realisation931=== PAUSE TestIsValidUploadKey/realisation932=== RUN TestIsValidUploadKey/realisation_plus_in_output933=== PAUSE TestIsValidUploadKey/realisation_plus_in_output934=== RUN TestIsValidUploadKey/nix-cache-info935=== PAUSE TestIsValidUploadKey/nix-cache-info936=== RUN TestIsValidUploadKey/index.html937=== PAUSE TestIsValidUploadKey/index.html938=== RUN TestIsValidUploadKey/narinfo_key,_nar_type939=== PAUSE TestIsValidUploadKey/narinfo_key,_nar_type940=== RUN TestIsValidUploadKey/nar_key,_narinfo_type941=== PAUSE TestIsValidUploadKey/nar_key,_narinfo_type942=== RUN TestIsValidUploadKey/listing_key,_narinfo_type943=== PAUSE TestIsValidUploadKey/listing_key,_narinfo_type944=== RUN TestIsValidUploadKey/traversal945=== PAUSE TestIsValidUploadKey/traversal946=== RUN TestIsValidUploadKey/traversal_nar947=== PAUSE TestIsValidUploadKey/traversal_nar948=== RUN TestIsValidUploadKey/absolute949=== PAUSE TestIsValidUploadKey/absolute950=== RUN TestIsValidUploadKey/empty_key951=== PAUSE TestIsValidUploadKey/empty_key952=== RUN TestIsValidUploadKey/unknown_type953=== PAUSE TestIsValidUploadKey/unknown_type954=== CONT TestService_Rustfstest955=== PAUSE TestPendingClosureFailureTracksEveryUpload/refuse_aborts=true9562026/10/09 09:57:38 INFO Shutdown signal received, draining in-flight requests timeout=10s957=== CONT TestTombstonedObjectOfferedWithoutWaiting9582026/10/09 09:57:38 INFO Client skipped oversized paths paths=3 nar_bytes=5000000000959--- PASS: TestSkippedUploadsHandler (0.04s)960=== CONT TestParseSize961--- PASS: TestParseSize (0.00s)962=== CONT TestPresignedUploadRegisteredBeforeCommit963--- PASS: TestGracefulShutdownDrainsInflight (0.07s)964=== CONT TestMultipartUploadAbortedWhenCancelledBeforeRecorded965=== RUN TestUploadHandlersRejectOversizedBody/create_pending_closure966=== PAUSE TestUploadHandlersRejectOversizedBody/create_pending_closure967=== RUN TestUploadHandlersRejectOversizedBody/complete_multipart968=== PAUSE TestUploadHandlersRejectOversizedBody/complete_multipart969=== RUN TestUploadHandlersRejectOversizedBody/request_more_parts970=== PAUSE TestUploadHandlersRejectOversizedBody/request_more_parts971=== CONT TestCompleteMultipartUpload_ErrorButObjectExists9722026/10/09 09:57:38 INFO Received uploads request method=POST path=/api/pending_closures973--- PASS: TestPendingClosureFailureAbortsItsUploads (0.74s)974=== CONT TestGCSweepSkipsPendingObjects975--- PASS: TestReadRedirectKeepsNarinfoProxied (1.68s)976=== CONT TestCompletedNarNotReofferedAcrossClosures9772026/10/09 09:57:40 INFO Received uploads request method=POST path=/api/pending_closures9782026/10/09 09:57:40 INFO Received uploads request method=POST path=/api/pending_closures9792026/10/09 09:57:40 INFO Received uploads request method=POST path=/api/pending_closures980=== NAME TestOrphanedObjectsGC981 orphaned_objects_gc_test.go:287: GC Test Summary:982 orphaned_objects_gc_test.go:288: - Kept: 2 objects from closure A983 orphaned_objects_gc_test.go:289: - Deleted: 2 objects from closure B984 orphaned_objects_gc_test.go:290: - Deleted: 6 orphaned chain objects (X1->X2->X3)985 orphaned_objects_gc_test.go:291: - Deleted: 2 orphaned single objects (Y)986 orphaned_objects_gc_test.go:292: - Total deleted: 10 objects987--- PASS: TestOrphanedObjectsGC (1.96s)988=== CONT TestRedundantMultipartUpload9892026/10/09 09:57:40 WARN readiness check failed error="closed pool"990--- PASS: TestService_readinessHandler (2.46s)991=== CONT TestPush_SkippedKeySurvivesGCBeforeCommit9922026/10/09 09:57:41 INFO Received uploads request method=POST path=/api/pending_closures9932026/10/09 09:57:41 INFO Aborted multipart uploads count=0 kept=09942026/10/09 09:57:41 WARN Force mode enabled - objects will be deleted immediately without grace period9952026/10/09 09:57:41 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."9962026/10/09 09:57:41 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."997--- PASS: TestFailedS3DeleteKeepsTombstone (3.03s)998=== CONT TestService_NativeMTLS9992026/10/09 09:57:41 INFO Received uploads request method=POST path=/api/pending_closures10002026/10/09 09:57:41 INFO Received uploads request method=POST path=/api/pending_closures10012026/10/09 09:57:41 INFO Received cleanup request method=DELETE path=/api/pending_closures10022026/10/09 09:57:41 INFO Received cleanup request method=DELETE path=/api/pending_closures10032026/10/09 09:57:41 INFO Aborted multipart uploads count=0 kept=010042026/10/09 09:57:41 INFO Received uploads request method=POST path=/api/pending_closures10052026/10/09 09:57:41 INFO Received cleanup request method=DELETE path=/api/pending_closures10062026/10/09 09:57:41 INFO Aborted multipart uploads count=1 kept=010072026/10/09 09:57:41 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete10082026/10/09 09:57:41 INFO Received complete multipart upload request method=POST path=/api/multipart/complete10092026-10-09 09:57:41.838 UTC [65645] ERROR: Closure does not exist: id=110102026-10-09 09:57:41.838 UTC [65645] CONTEXT: PL/pgSQL function commit_pending_closure(bigint) line 13 at RAISE10112026-10-09 09:57:41.838 UTC [65645] STATEMENT: -- name: CommitPendingClosure :one1012 SELECT commit_pending_closure($1::bigint)::bigint1013 1014=== CONT TestPush_CompleteWaitsForCleanupOfThePush1015--- PASS: TestService_cleanupPendingClosuresHandler (3.50s)10162026/10/09 09:57:41 INFO Completed multipart upload object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst upload_id=MTE5MjdlYzItNjE3OS00ZDA5LTljZjMtZGFiOThmMGQ2ZDQ0LjE5MDUyNzM0LWM0MTgtNDBkMC04OWY3LTA4Mjc4MGQ1MTVmOHgxNzkxNTM5ODYwMzc1MjkxMDAw parts=1010172026/10/09 09:57:41 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete10182026/10/09 09:57:41 INFO Completed upload id=1 objects=2 duration=16.561334ms10192026/10/09 09:57:41 INFO Received get closure request method=GET path=/api/closures/0000000000000000000000000000000010202026/10/09 09:57:41 INFO Received uploads request method=POST path=/api/pending_closures10212026/10/09 09:57:41 INFO Starting cleanup of old closures method=DELETE path=/api/closures10222026/10/09 09:57:41 INFO Aborted multipart uploads count=0 kept=01023--- PASS: TestService_Rustfstest (3.61s)10242026/10/09 09:57:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=01025=== CONT TestPush_RootsAreLockedInByteOrder10262026/10/09 09:57:42 INFO Vacuumed table table=pending_closures10272026/10/09 09:57:42 INFO Vacuumed table table=pending_objects10282026/10/09 09:57:42 INFO Vacuumed table table=multipart_uploads10292026/10/09 09:57:42 INFO Vacuumed table table=closures10302026/10/09 09:57:42 INFO Vacuumed table table=objects10312026/10/09 09:57:42 INFO Received get closure request method=GET path=/api/closures/000000000000000000000000000000001032--- PASS: TestService_createPendingClosureHandler (3.76s)1033=== CONT TestPushDedupSurvivesConcurrentGC10342026/10/09 09:57:42 WARN Authentication failed token_preview=wrongtoken token_length=10 reason="static API token mismatch"1035--- PASS: TestService_AuthMiddleware (3.84s)1036=== CONT TestPush_CompleteCommitsEveryRoot1037--- PASS: TestConcurrentCommitsSharingObjectsDoNotDeadlock (3.93s)1038=== CONT TestMetricsCommit1039--- PASS: TestObjectStatsTrigger (4.12s)1040=== CONT TestPush_RejectsBadRequests10412026/10/09 09:57:42 INFO Received uploads request method=POST path=/api/pending_closures1042--- PASS: TestTombstonedObjectOfferedWithoutWaiting (4.37s)1043=== CONT TestMultipartCleanup10442026/10/09 09:57:42 INFO Received uploads request method=POST path=/api/pending_closures10452026/10/09 09:57:43 INFO Received complete multipart upload request method=POST path=/api/multipart/complete10462026/10/09 09:57:43 ERROR Failed to get multipart upload error="no rows in result set" upload_id=does-not-exist object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst1047=== CONT TestReadRedirectUsesPublicS3URL1048--- PASS: TestCompleteMultipartUnregistered (4.89s)1049--- PASS: TestReconnectLeavesObjectsUnlocked (5.24s)1050=== CONT TestPush_SignsNarinfosOfItsPendingObjects10512026/10/09 09:57:44 INFO Received uploads request method=POST path=/api/pending_closures1052--- PASS: TestMultipartUploadAbortedWhenCancelledBeforeRecorded (5.82s)1053=== CONT TestDeduplicatedObjectsRecordedAsPending10542026/10/09 09:57:44 INFO Received complete multipart upload request method=POST path=/api/multipart/complete10552026/10/09 09:57:44 INFO Received uploads request method=POST path=/api/pending_closures10562026/10/09 09:57:44 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst upload_id=MTE5MjdlYzItNjE3OS00ZDA5LTljZjMtZGFiOThmMGQ2ZDQ0LjJlMTk3Y2M1LWQwMGItNDBhZS1hMmE3LTcwZmVkY2QzMzQyN3gxNzkxNTM5ODYyOTIzNzkzMDAw parts=1010572026/10/09 09:57:44 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete10582026/10/09 09:57:44 INFO Completed upload id=1 objects=2 duration=6.389083ms10592026/10/09 09:57:44 INFO Received uploads request method=POST path=/api/pending_closures10602026/10/09 09:57:44 INFO Registered completed upload object_key=nar/jjjjjjjjjjjjjjjjjjjjjjjjjjjjjj0100000000000000000000.nar.zst10612026/10/09 09:57:44 INFO Received uploads request method=POST path=/api/pending_closures10622026/10/09 09:57:44 INFO Received uploads request method=POST path=/api/pending_closures10632026/10/09 09:57:44 INFO Object in database but missing from S3 key=dadb44fdadb44fdadb44fdadb44f0000.narinfo10642026/10/09 09:57:44 WARN Found objects in DB but missing from S3, will re-upload count=11065--- PASS: TestPresignedUploadRegisteredBeforeCommit (6.06s)1066=== CONT TestGCBugBareHashReferences10672026/10/09 09:57:44 INFO Received complete upload request method=POST path=/api/pending_closures/3/complete10682026/10/09 09:57:44 INFO Completed upload id=3 objects=2 duration=618.541µs1069--- PASS: TestService_verifyS3Integrity (6.18s)1070=== CONT TestNARDeduplicationMetadataUploadBug10712026/10/09 09:57:44 INFO Received uploads request method=POST path=/api/pending_closures1072=== CONT TestLeadEndsOnShutdown1073--- PASS: TestCreatePendingClosureVerifyS3FailureReleasesConnection (6.59s)1074=== CONT TestGenerateLandingPage1075--- PASS: TestService_healthCheckHandler (6.60s)1076--- PASS: TestGenerateLandingPage (0.01s)1077=== CONT TestCacheConfigHandlerMaxNarSize1078--- PASS: TestCacheConfigHandlerMaxNarSize (0.00s)1079=== CONT TestGCAdvisoryLockBlocksConcurrentRun10802026/10/09 09:57:45 INFO Aborted multipart uploads count=0 kept=010812026/10/09 09:57:45 WARN Force mode enabled - objects will be deleted immediately without grace period10822026/10/09 09:57:45 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=010832026/10/09 09:57:45 INFO Vacuumed table table=pending_closures10842026/10/09 09:57:45 INFO Vacuumed table table=pending_objects10852026/10/09 09:57:45 INFO Vacuumed table table=multipart_uploads10862026/10/09 09:57:45 INFO Vacuumed table table=closures10872026/10/09 09:57:45 INFO Received uploads request method=POST path=/api/pending_closures10882026/10/09 09:57:45 INFO Vacuumed table table=objects1089--- PASS: TestGCSweepDeliversEachKeyOnce (7.38s)1090=== CONT TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle1091--- PASS: TestCreatePendingClosure_SmallNARUsesSimplePUT (7.48s)1092=== CONT TestClientPushesUseOnePush10932026/10/09 09:57:45 INFO Received uploads request method=POST path=/api/pending_closures10942026/10/09 09:57:46 INFO Received uploads request method=POST path=/api/pending_closures10952026/10/09 09:57:46 INFO Received complete multipart upload request method=POST path=/api/multipart/complete10962026/10/09 09:57:46 WARN CompleteMultipartUpload errored but object exists; treating as success error="The specified key does not exist." object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=MTE5MjdlYzItNjE3OS00ZDA5LTljZjMtZGFiOThmMGQ2ZDQ0Ljc0YWM3NjcxLWYyZTItNDc1ZC05YWMxLTRhYjEyMjA5YjgzM3gxNzkxNTM5ODY2MDYwNjc5MDAw10972026/10/09 09:57:46 INFO Aborted multipart uploads count=0 kept=010982026/10/09 09:57:46 WARN Force mode enabled - objects will be deleted immediately without grace period10992026/10/09 09:57:46 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=011002026/10/09 09:57:46 WARN Failed to abort multipart upload key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=MTE5MjdlYzItNjE3OS00ZDA5LTljZjMtZGFiOThmMGQ2ZDQ0Ljc0YWM3NjcxLWYyZTItNDc1ZC05YWMxLTRhYjEyMjA5YjgzM3gxNzkxNTM5ODY2MDYwNjc5MDAw error="Delete \"http://localhost:60965/bucket27/nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst?uploadId=MTE5MjdlYzItNjE3OS00ZDA5LTljZjMtZGFiOThmMGQ2ZDQ0Ljc0YWM3NjcxLWYyZTItNDc1ZC05YWMxLTRhYjEyMjA5YjgzM3gxNzkxNTM5ODY2MDYwNjc5MDAw\": injected: abort refused"11012026/10/09 09:57:46 INFO Received complete multipart upload request method=POST path=/api/multipart/complete11022026/10/09 09:57:46 INFO Vacuumed table table=pending_closures11032026/10/09 09:57:46 WARN CompleteMultipartUpload errored but object exists; treating as success error="The specified key does not exist." object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=MTE5MjdlYzItNjE3OS00ZDA5LTljZjMtZGFiOThmMGQ2ZDQ0Ljc0YWM3NjcxLWYyZTItNDc1ZC05YWMxLTRhYjEyMjA5YjgzM3gxNzkxNTM5ODY2MDYwNjc5MDAw11042026/10/09 09:57:46 INFO Vacuumed table table=pending_objects11052026/10/09 09:57:46 INFO Vacuumed table table=multipart_uploads11062026/10/09 09:57:46 INFO Vacuumed table table=closures11072026/10/09 09:57:46 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=MTE5MjdlYzItNjE3OS00ZDA5LTljZjMtZGFiOThmMGQ2ZDQ0Ljc0YWM3NjcxLWYyZTItNDc1ZC05YWMxLTRhYjEyMjA5YjgzM3gxNzkxNTM5ODY2MDYwNjc5MDAw parts=111082026/10/09 09:57:46 INFO Vacuumed table table=objects1109--- PASS: TestCompleteMultipartUpload_ErrorButObjectExists (8.16s)1110=== CONT TestService_ReadAuthMiddleware11112026/10/09 09:57:46 INFO Received uploads request method=POST path=/api/pending_closures1112--- PASS: TestGCSweepSkipsPendingObjects (7.69s)1113=== CONT TestCacheConfigHandler1114=== RUN TestCacheConfigHandler/full_config,_no_issuer1115=== PAUSE TestCacheConfigHandler/full_config,_no_issuer1116=== RUN TestCacheConfigHandler/no_cache_url_configured1117=== PAUSE TestCacheConfigHandler/no_cache_url_configured1118=== RUN TestCacheConfigHandler/no_signing_keys1119=== PAUSE TestCacheConfigHandler/no_signing_keys1120=== RUN TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1121=== PAUSE TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1122=== CONT TestClientWithDependencies11232026/10/09 09:57:46 INFO Received uploads request method=POST path=/api/pending_closures11242026/10/09 09:57:47 INFO Received uploads request method=POST path=/api/pending_closures11252026/10/09 09:57:47 INFO Received push request method=POST path=/api/pushes11262026/10/09 09:57:47 INFO Aborted multipart uploads count=1 kept=011272026/10/09 09:57:47 INFO Received cleanup request method=DELETE path=/api/pending_closures11282026/10/09 09:57:47 INFO Aborted multipart uploads count=1 kept=01129--- PASS: TestPendingCleanupUsesOneCutoff (9.40s)1130=== CONT TestService_RequireScope_OIDC11312026/10/09 09:57:47 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:61409/oidc11322026/10/09 09:57:47 WARN mTLS auth: subject not in bound subjects subject="CN=reader"11332026/10/09 09:57:47 WARN mTLS auth: subject not in bound subjects subject="CN=reader"1134--- PASS: TestService_NativeMTLS (6.65s)1135=== CONT TestService_AuthMiddleware_OIDC11362026/10/09 09:57:48 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:61413/oidc11372026/10/09 09:57:48 INFO Received push request method=POST path=/api/pushes11382026/10/09 09:57:48 INFO Received push request method=POST path=/api/pushes11392026/10/09 09:57:49 INFO Received complete multipart upload request method=POST path=/api/multipart/complete11402026/10/09 09:57:49 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0100000000000000000000.nar.zst upload_id=MTE5MjdlYzItNjE3OS00ZDA5LTljZjMtZGFiOThmMGQ2ZDQ0LmM3ZGFmNWU3LTMyNjgtNDkyZS05OGQzLTg0NTgyZGVmNmRmY3gxNzkxNTM5ODY2NzkxODcwMDAw parts=1211412026/10/09 09:57:49 INFO Received uploads request method=POST path=/api/pending_closures11422026/10/09 09:57:49 INFO Received uploads request method=POST path=/api/pending_closures11432026/10/09 09:57:49 INFO Received uploads request method=POST path=/api/pending_closures11442026/10/09 09:57:49 INFO Received complete multipart upload request method=POST path=/api/multipart/complete11452026/10/09 09:57:49 INFO Aborted multipart uploads count=0 kept=011462026/10/09 09:57:49 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=011472026/10/09 09:57:49 INFO Completed multipart upload object_key=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhh0100000000000000000000.nar.zst upload_id=MTE5MjdlYzItNjE3OS00ZDA5LTljZjMtZGFiOThmMGQ2ZDQ0LjBhMzRkMzQxLTdlNTgtNDcxMC1hMjc1LTM4ZDBjNGY4MmEzYXgxNzkxNTM5ODY3MTI1NDk1MDAw parts=1211482026/10/09 09:57:49 INFO Received uploads request method=POST path=/api/pending_closures11492026/10/09 09:57:49 INFO Received push request method=POST path=/api/pushes11502026/10/09 09:57:49 INFO Vacuumed table table=pending_closures11512026/10/09 09:57:49 INFO Received complete multipart upload request method=POST path=/api/multipart/complete1152--- PASS: TestCompletedNarNotReofferedAcrossClosures (9.90s)1153=== CONT TestService_ReadScope_PublicByDefault11542026/10/09 09:57:49 INFO Vacuumed table table=pending_objects11552026/10/09 09:57:49 INFO Vacuumed table table=multipart_uploads11562026/10/09 09:57:50 INFO Vacuumed table table=closures11572026/10/09 09:57:50 INFO Vacuumed table table=objects11582026/10/09 09:57:50 INFO Completed multipart upload object_key=nar/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa00000000000000000000.nar.zst upload_id=MTE5MjdlYzItNjE3OS00ZDA5LTljZjMtZGFiOThmMGQ2ZDQ0Ljk1OTdlNGI1LTg5YTEtNDFhMi1hODlkLTlmMzk1NDBlNTIxM3gxNzkxNTM5ODY3NTEzOTMwMDAw parts=1011592026/10/09 09:57:50 INFO Aborted multipart uploads count=0 kept=011602026/10/09 09:57:50 WARN Force mode enabled - objects will be deleted immediately without grace period11612026/10/09 09:57:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=011622026/10/09 09:57:50 INFO Vacuumed table table=pending_closures11632026/10/09 09:57:50 INFO Vacuumed table table=pending_objects11642026/10/09 09:57:50 INFO Vacuumed table table=multipart_uploads11652026/10/09 09:57:50 INFO Vacuumed table table=closures11662026/10/09 09:57:50 INFO Vacuumed table table=objects1167--- PASS: TestPushDedupSurvivesConcurrentGC (8.12s)1168=== CONT TestService_AuthMiddleware_MTLSBoundSubjects11692026/10/09 09:57:50 INFO Received push request method=POST path=/api/pushes1170=== RUN TestPush_RejectsBadRequests/root_not_in_objects1171=== PAUSE TestPush_RejectsBadRequests/root_not_in_objects1172=== RUN TestPush_RejectsBadRequests/no_roots1173=== PAUSE TestPush_RejectsBadRequests/no_roots1174=== RUN TestPush_RejectsBadRequests/no_objects1175=== PAUSE TestPush_RejectsBadRequests/no_objects1176=== RUN TestPush_RejectsBadRequests/bad_root1177=== PAUSE TestPush_RejectsBadRequests/bad_root1178=== CONT TestReadProxyNarinfo11792026/10/09 09:57:50 INFO Received complete multipart upload request method=POST path=/api/multipart/complete11802026/10/09 09:57:50 INFO Completed multipart upload object_key=nar/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb00000000000000000000.nar.zst upload_id=MTE5MjdlYzItNjE3OS00ZDA5LTljZjMtZGFiOThmMGQ2ZDQ0LmVjY2UyZDhiLThkMGEtNDRkYS1hNzc1LWI3YmI4ZjJhOWYyNXgxNzkxNTM5ODY4NTAxODIwMDAw parts=1011812026/10/09 09:57:51 INFO Received uploads request method=POST path=/api/pending_closures11822026-10-09 09:57:51.305 UTC [65892] ERROR: Push does not exist: id=111832026-10-09 09:57:51.305 UTC [65892] CONTEXT: PL/pgSQL function commit_push(bigint) line 11 at RAISE11842026-10-09 09:57:51.305 UTC [65892] STATEMENT: SELECT commit_push($1)1185--- PASS: TestPush_CompleteWaitsForCleanupOfThePush (9.46s)1186=== CONT TestReadProxyRangeRequest11872026/10/09 09:57:51 INFO Received complete multipart upload request method=POST path=/api/multipart/complete11882026/10/09 09:57:51 INFO Received cleanup request method=DELETE path=/api/pending_closures11892026/10/09 09:57:51 WARN Failed to abort upload, keeping its closure key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst error="Get \"http://127.0.0.1:1/bucket39/?location=\": dial tcp 127.0.0.1:1: connect: connection refused" code=""11902026/10/09 09:57:51 INFO Aborted multipart uploads count=0 kept=111912026/10/09 09:57:51 INFO Completed multipart upload object_key=nar/ffffffffffffffffffffffffffffffff00000000000000000000.nar.zst upload_id=MTE5MjdlYzItNjE3OS00ZDA5LTljZjMtZGFiOThmMGQ2ZDQ0LjBjNzg3Y2IzLWQ4YzctNGUzZi1hZDZlLThjMzViOTMxNDUzYngxNzkxNTM5ODY4OTQ4MTkzMDAw parts=1011922026/10/09 09:57:51 INFO Received cleanup request method=DELETE path=/api/pending_closures11932026/10/09 09:57:51 INFO Aborted multipart uploads count=1 kept=01194--- PASS: TestMultipartCleanup (8.94s)1195=== CONT TestReadRedirectNar1196--- PASS: TestReadRedirectUsesPublicS3URL (8.63s)1197=== CONT TestReadProxyRootRedirectsToIndexHTML11982026/10/09 09:57:52 INFO Received push request method=POST path=/api/pushes11992026/10/09 09:57:52 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign12002026/10/09 09:57:52 INFO Signed narinfos id=1 count=112012026/10/09 09:57:52 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign12022026/10/09 09:57:52 INFO Received complete multipart upload request method=POST path=/api/multipart/complete12032026/10/09 09:57:52 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign12042026/10/09 09:57:52 ERROR Failed to sign narinfos signer="index 1" error="signing key cannot be nil"12052026/10/09 09:57:52 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign12062026/10/09 09:57:52 ERROR Failed to sign narinfos signer=niks3-test-1 error="signer failed"12072026/10/09 09:57:52 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign12082026/10/09 09:57:52 ERROR Signer returned unexpected signature count signer=niks3-test-1 got=0 want=112092026/10/09 09:57:52 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign12102026/10/09 09:57:52 ERROR Signer returned unexpected object key signer=niks3-test-1 object_key=unexpected.narinfo12112026/10/09 09:57:52 INFO Received complete multipart upload request method=POST path=/api/multipart/complete1212--- PASS: TestPush_SignsNarinfosOfItsPendingObjects (8.93s)1213=== CONT TestReadProxyConditionalGet12142026/10/09 09:57:52 INFO Completed multipart upload object_key=nar/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa00000000000000000000.nar.zst upload_id=MTE5MjdlYzItNjE3OS00ZDA5LTljZjMtZGFiOThmMGQ2ZDQ0LjJmYmYwMWVmLTJjMzEtNDdlZi1iZmRmLTE4Yjk3Zjg4ZDc3YngxNzkxNTM5ODY5OTQ3NjIwMDAw parts=1012152026/10/09 09:57:52 INFO Received complete multipart upload request method=POST path=/api/multipart/complete12162026/10/09 09:57:52 INFO Completed multipart upload object_key=nar/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb00000000000000000000.nar.zst upload_id=MTE5MjdlYzItNjE3OS00ZDA5LTljZjMtZGFiOThmMGQ2ZDQ0LmU2ZTcxNTY2LWRiNzYtNDEwOC1hYzY3LThiN2JmYjdhZjRmMXgxNzkxNTM5ODY3NTEzNTM2MDAw parts=1012172026/10/09 09:57:52 INFO Received complete push request method=POST path=/api/pushes/1/complete12182026/10/09 09:57:52 WARN Failed to abort redundant multipart upload, keeping its row object_key=nar/gggggggggggggggggggggggggggggg0200000000000000000000.nar.zst upload_id=MTE5MjdlYzItNjE3OS00ZDA5LTljZjMtZGFiOThmMGQ2ZDQ0LjNiOTc0MTI4LThlZmMtNDU5My1iYTk0LWZiNWQxYzhhMDUxNngxNzkxNTM5ODY5NzAwNDE4MDAw error="Get \"http://127.0.0.1:1/bucket29/?location=\": dial tcp 127.0.0.1:1: connect: connection refused"12192026/10/09 09:57:52 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0200000000000000000000.nar.zst upload_id=MTE5MjdlYzItNjE3OS00ZDA5LTljZjMtZGFiOThmMGQ2ZDQ0LjYxMDllNGU4LWU1YzYtNDRlOS04MDYwLWNhN2VjMzkzNzlhMXgxNzkxNTM5ODY5NjAyNzc0MDAw parts=1212202026/10/09 09:57:52 INFO Received cleanup request method=DELETE path=/api/pending_closures12212026/10/09 09:57:52 INFO Completed push id=1 objects=4 duration=21.732ms12222026/10/09 09:57:52 INFO Received push request method=POST path=/api/pushes12232026/10/09 09:57:52 INFO Received uploads request method=POST path=/api/pending_closures12242026/10/09 09:57:52 INFO Aborted multipart uploads count=1 kept=01225--- PASS: TestRedundantMultipartUpload (12.53s)1226=== CONT TestReadProxyOutlastsServerWriteTimeout1227--- PASS: TestDeduplicatedObjectsRecordedAsPending (8.64s)1228=== CONT TestClientIntegration12292026/10/09 09:57:52 INFO Received complete multipart upload request method=POST path=/api/multipart/complete12302026/10/09 09:57:52 INFO Aborted multipart uploads count=0 kept=012312026/10/09 09:57:52 WARN Force mode enabled - objects will be deleted immediately without grace period12322026/10/09 09:57:53 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=2 objects-deleted-after-grace-period=2 objects-failed-to-delete=012332026/10/09 09:57:53 INFO Completed multipart upload object_key=nar/cccccccccccccccccccccccccccccccc00000000000000000000.nar.zst upload_id=MTE5MjdlYzItNjE3OS00ZDA5LTljZjMtZGFiOThmMGQ2ZDQ0LmIxODRmNDRkLWE4NDYtNGMwYi1iZDRiLWEyMzc4NjdlZjY2ZngxNzkxNTM5ODcwNDAyMDY5MDAw parts=1012342026/10/09 09:57:53 INFO Vacuumed table table=pending_closures12352026/10/09 09:57:53 INFO Vacuumed table table=pending_objects12362026/10/09 09:57:53 INFO Vacuumed table table=multipart_uploads12372026/10/09 09:57:53 INFO Vacuumed table table=closures12382026/10/09 09:57:53 INFO Vacuumed table table=objects1239--- PASS: TestGCBugBareHashReferences (9.08s)1240=== CONT TestReadProxyDisabled12412026/10/09 09:57:54 INFO Received complete multipart upload request method=POST path=/api/multipart/complete12422026/10/09 09:57:54 INFO Completed multipart upload object_key=nar/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa00000000000000000000.nar.zst upload_id=MTE5MjdlYzItNjE3OS00ZDA5LTljZjMtZGFiOThmMGQ2ZDQ0LjZhZDFmZDM0LWMyYTYtNDU0OS04OGM1LWNlMGIwNTNhMjM5ZngxNzkxNTM5ODY4OTQ3NTA4MDAw parts=1012432026/10/09 09:57:54 INFO lead: acquired remote=192.0.2.1:12341244=== NAME TestNARDeduplicationMetadataUploadBug1245 metadata_upload_test.go:48: First store path: /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestNARDeduplicationMetadataUploadBug3739036377/001/store/jzdkkpjzngbd8hh4z6ilczfxk39vaky0-file1.txt12462026/10/09 09:57:54 INFO Received push request method=POST path=/api/pushes12472026/10/09 09:57:54 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)12482026/10/09 09:57:54 INFO Uploading jzdkkpjzngbd8hh4z6ilczfxk39vaky0-file1.txt (160B)12492026/10/09 09:57:54 WARN Failed to register uploaded object key=jzdkkpjzngbd8hh4z6ilczfxk39vaky0.ls error="server returned 404: 404 page not found\n"12502026/10/09 09:57:54 WARN Failed to register uploaded object key=nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst error="server returned 404: 404 page not found\n"12512026/10/09 09:57:54 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign12522026/10/09 09:57:54 INFO Signed narinfos id=1 count=112532026/10/09 09:57:54 INFO Uploading 1 narinfos12542026/10/09 09:57:54 INFO Received complete push request method=POST path=/api/pushes/1/complete12552026/10/09 09:57:54 WARN Failed to register uploaded object key=jzdkkpjzngbd8hh4z6ilczfxk39vaky0.narinfo error="server returned 404: 404 page not found\n"12562026/10/09 09:57:54 INFO Received complete multipart upload request method=POST path=/api/multipart/complete12572026/10/09 09:57:54 INFO Completed push id=1 objects=3 duration=96.959333ms12582026/10/09 09:57:54 INFO Upload complete. (369ms)1259 metadata_upload_test.go:54: Retrieved narinfo from S3:1260 StorePath: /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestNARDeduplicationMetadataUploadBug3739036377/001/store/jzdkkpjzngbd8hh4z6ilczfxk39vaky0-file1.txt1261 URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst12622026/10/09 09:57:54 INFO Completed multipart upload object_key=nar/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb00000000000000000000.nar.zst upload_id=MTE5MjdlYzItNjE3OS00ZDA5LTljZjMtZGFiOThmMGQ2ZDQ0LmZiNjRiN2Q3LWJkMTAtNGViYi1iM2FkLWE3MjA4MjJjNWI5MXgxNzkxNTM5ODY5OTQ4MTIxMDAw parts=101263 Compression: zstd12642026/10/09 09:57:54 INFO Received complete push request method=POST path=/api/pushes/1/complete1265 NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1266 NarSize: 1601267 References: 1268 CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1269 metadata_upload_test.go:55: Retrieved .ls file from S3 (compressed size: 77 bytes)1270 metadata_upload_test.go:55: Decompressed .ls content (64 bytes):1271 {"version":1,"root":{"type":"regular","size":44,"narOffset":96}}12722026/10/09 09:57:54 INFO Completed push id=1 objects=4 duration=28.881167ms12732026/10/09 09:57:54 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete12742026/10/09 09:57:54 INFO Completed upload id=2 objects=2 duration=945.917µs1275--- PASS: TestMetricsCommit (12.56s)1276=== CONT TestCreatePendingClosureRejectsOversizedNAR12772026/10/09 09:57:54 INFO Received uploads request method=POST path=/api/pending_closures1278--- PASS: TestCreatePendingClosureRejectsOversizedNAR (0.01s)1279=== CONT TestClientSharedPathCommittedMidPush1280=== NAME TestNARDeduplicationMetadataUploadBug1281 metadata_upload_test.go:64: Second store path (same content): /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestNARDeduplicationMetadataUploadBug3739036377/001/store/n0kh57iv90pnvqsnx7idg3b3c4327ips-file2.txt12822026/10/09 09:57:55 INFO Received uploads request method=POST path=/api/pending_closures12832026/10/09 09:57:55 INFO Received push request method=POST path=/api/pushes12842026/10/09 09:57:55 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)12852026/10/09 09:57:55 WARN Failed to register uploaded object key=n0kh57iv90pnvqsnx7idg3b3c4327ips.ls error="server returned 404: 404 page not found\n"12862026/10/09 09:57:55 INFO Received sign narinfos request method=POST path=/api/pushes/2/sign12872026/10/09 09:57:55 INFO Signed narinfos id=2 count=112882026/10/09 09:57:55 INFO Uploading 1 narinfos12892026/10/09 09:57:55 INFO Received complete push request method=POST path=/api/pushes/2/complete12902026/10/09 09:57:55 WARN Failed to register uploaded object key=n0kh57iv90pnvqsnx7idg3b3c4327ips.narinfo error="server returned 404: 404 page not found\n"12912026/10/09 09:57:55 INFO Completed push id=2 objects=3 duration=833.5µs12922026/10/09 09:57:55 INFO Upload complete. (204ms)1293 metadata_upload_test.go:76: Retrieved narinfo from S3:1294 StorePath: /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestNARDeduplicationMetadataUploadBug3739036377/001/store/n0kh57iv90pnvqsnx7idg3b3c4327ips-file2.txt1295 URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst1296 Compression: zstd1297 NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1298 NarSize: 1601299 References: 1300 CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1301 metadata_upload_test.go:77: Retrieved .ls file from S3 (compressed size: 62 bytes)1302 metadata_upload_test.go:77: Decompressed .ls content (49 bytes):1303 {"version":1,"root":{"type":"regular","size":44}}13042026/10/09 09:57:55 INFO lead: released remote=192.0.2.1:12341305--- PASS: TestLeadEndsOnShutdown (10.40s)13062026/10/09 09:57:55 INFO Received complete multipart upload request method=POST path=/api/multipart/complete1307=== CONT TestReadProxyHead13082026/10/09 09:57:55 INFO Completed multipart upload object_key=nar/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa00000000000000000000.nar.zst upload_id=MTE5MjdlYzItNjE3OS00ZDA5LTljZjMtZGFiOThmMGQ2ZDQ0LmUwZWY4NjRiLTAyMGYtNDdhNC04NzY5LTVmOGExZjYyMWMxMXgxNzkxNTM5ODcwNDAyNDM1MDAw parts=101309--- PASS: TestNARDeduplicationMetadataUploadBug (10.98s)1310=== CONT TestReadProxyInvalidPath13112026/10/09 09:57:55 INFO Received complete multipart upload request method=POST path=/api/multipart/complete13122026/10/09 09:57:55 INFO Completed multipart upload object_key=nar/cccccccccccccccccccccccccccccccc00000000000000000000.nar.zst upload_id=MTE5MjdlYzItNjE3OS00ZDA5LTljZjMtZGFiOThmMGQ2ZDQ0LjdlZmZmMzBiLWI2MjgtNGY5ZC1hZjI0LTFlODY4MTMyOTFlOXgxNzkxNTM5ODcyNzk0MjAxMDAw parts=1013132026/10/09 09:57:55 INFO Received complete push request method=POST path=/api/pushes/2/complete13142026/10/09 09:57:55 INFO Received complete multipart upload request method=POST path=/api/multipart/complete13152026/10/09 09:57:55 INFO Completed push id=2 objects=4 duration=91.808417ms1316--- PASS: TestPush_SkippedKeySurvivesGCBeforeCommit (15.00s)1317=== CONT TestPinProtectsFromGC1318--- PASS: TestService_ReadAuthMiddleware (9.34s)1319=== CONT TestReadProxyContentNegotiation13202026/10/09 09:57:56 INFO Received complete multipart upload request method=POST path=/api/multipart/complete13212026/10/09 09:57:56 INFO Completed multipart upload object_key=nar/dddddddddddddddddddddddddddddddd00000000000000000000.nar.zst upload_id=MTE5MjdlYzItNjE3OS00ZDA5LTljZjMtZGFiOThmMGQ2ZDQ0LmVmOTViZWM0LTIzNzctNGFjYy1hMDdiLTE5MDk4ZmJiNDU2NXgxNzkxNTM5ODY4OTQ4MDg4MDAw parts=1013222026/10/09 09:57:56 INFO Received push request method=POST path=/api/pushes13232026/10/09 09:57:56 INFO Uploading 2 paths to 127.0.0.1 (1 already cached)13242026/10/09 09:57:56 INFO Uploading xbqmkgdshgk5k0xjml4m1g56lpmw4j8d-b (256B)13252026/10/09 09:57:56 INFO Uploading 53ks2w0ysjv96vmx4rpi1al5gp75yamj-shared-dep (136B)13262026/10/09 09:57:56 WARN Failed to register uploaded object key=vhryrfv773fksmia7iy65577br00fhfm.ls error="server returned 404: 404 page not found\n"13272026/10/09 09:57:56 WARN Failed to register uploaded object key=nar/0hqznlhggmdhn8q1378c9hj7qi8v35l5g0yayi0xbvp8zc6wn5xl.nar.zst error="server returned 404: 404 page not found\n"13282026/10/09 09:57:56 WARN Failed to register uploaded object key=xbqmkgdshgk5k0xjml4m1g56lpmw4j8d.ls error="server returned 404: 404 page not found\n"13292026/10/09 09:57:56 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"13302026/10/09 09:57:56 WARN Failed to register uploaded object key=53ks2w0ysjv96vmx4rpi1al5gp75yamj.ls error="server returned 404: 404 page not found\n"13312026/10/09 09:57:56 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign13322026/10/09 09:57:56 INFO Signed narinfos id=1 count=313332026/10/09 09:57:56 INFO Uploading 3 narinfos13342026/10/09 09:57:56 WARN Failed to register uploaded object key=vhryrfv773fksmia7iy65577br00fhfm.narinfo error="server returned 404: 404 page not found\n"13352026/10/09 09:57:56 WARN Failed to register uploaded object key=53ks2w0ysjv96vmx4rpi1al5gp75yamj.narinfo error="server returned 404: 404 page not found\n"13362026/10/09 09:57:56 INFO Received complete push request method=POST path=/api/pushes/1/complete13372026/10/09 09:57:56 WARN Failed to register uploaded object key=xbqmkgdshgk5k0xjml4m1g56lpmw4j8d.narinfo error="server returned 404: 404 page not found\n"13382026/10/09 09:57:56 INFO Completed push id=1 objects=8 duration=133.390625ms13392026/10/09 09:57:56 INFO Upload complete. (383ms)1340=== NAME TestClientPushesUseOnePush1341 client_pushes_test.go:97: Retrieved narinfo from S3:1342 StorePath: /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestClientPushesUseOnePush2694814707/001/store/53ks2w0ysjv96vmx4rpi1al5gp75yamj-shared-dep1343 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1344 Compression: zstd1345 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821346 NarSize: 1361347 References: 1348 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n1349 client_pushes_test.go:97: Retrieved narinfo from S3:1350 StorePath: /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestClientPushesUseOnePush2694814707/001/store/vhryrfv773fksmia7iy65577br00fhfm-a1351 URL: nar/0hqznlhggmdhn8q1378c9hj7qi8v35l5g0yayi0xbvp8zc6wn5xl.nar.zst1352 Compression: zstd1353 NarHash: sha256:0hqznlhggmdhn8q1378c9hj7qi8v35l5g0yayi0xbvp8zc6wn5xl1354 NarSize: 2561355 References: /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestClientPushesUseOnePush2694814707/001/store/53ks2w0ysjv96vmx4rpi1al5gp75yamj-shared-dep1356 CA: text:sha256:1yxkka82di6scx68lcanr9wjmx53w6lw2ssjns8j14j9hyjmw4qa1357 client_pushes_test.go:97: Retrieved narinfo from S3:1358 StorePath: /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestClientPushesUseOnePush2694814707/001/store/xbqmkgdshgk5k0xjml4m1g56lpmw4j8d-b1359 URL: nar/0hqznlhggmdhn8q1378c9hj7qi8v35l5g0yayi0xbvp8zc6wn5xl.nar.zst1360 Compression: zstd1361 NarHash: sha256:0hqznlhggmdhn8q1378c9hj7qi8v35l5g0yayi0xbvp8zc6wn5xl1362 NarSize: 2561363 References: /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestClientPushesUseOnePush2694814707/001/store/53ks2w0ysjv96vmx4rpi1al5gp75yamj-shared-dep1364 CA: text:sha256:1yxkka82di6scx68lcanr9wjmx53w6lw2ssjns8j14j9hyjmw4qa1365--- PASS: TestClientPushesUseOnePush (10.99s)1366=== CONT TestReadProxyNarinfoAlreadyDecompressed1367=== RUN TestService_AuthMiddleware_OIDC/valid_OIDC_token1368=== PAUSE TestService_AuthMiddleware_OIDC/valid_OIDC_token1369=== RUN TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1370=== PAUSE TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1371=== RUN TestService_AuthMiddleware_OIDC/malformed_token_rejected1372=== PAUSE TestService_AuthMiddleware_OIDC/malformed_token_rejected1373=== RUN TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1374=== PAUSE TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1375=== CONT TestGCMetrics1376=== NAME TestClientWithDependencies1377 client_integration_test.go:735: Built derivation: /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestClientWithDependencies4176572669/001/store/irf7apapgkmyry2z90ashdg40g27dhib-test-script1378 client_integration_test.go:737: Found 1 dependencies (including self)1379=== RUN TestService_RequireScope_OIDC/builder_may_write1380=== PAUSE TestService_RequireScope_OIDC/builder_may_write1381=== RUN TestService_RequireScope_OIDC/builder_may_not_admin1382=== PAUSE TestService_RequireScope_OIDC/builder_may_not_admin1383=== RUN TestService_RequireScope_OIDC/ops_may_admin1384=== PAUSE TestService_RequireScope_OIDC/ops_may_admin1385=== RUN TestService_RequireScope_OIDC/ops_may_not_write1386=== PAUSE TestService_RequireScope_OIDC/ops_may_not_write1387=== RUN TestService_RequireScope_OIDC/reader_may_not_write1388=== PAUSE TestService_RequireScope_OIDC/reader_may_not_write1389=== RUN TestService_RequireScope_OIDC/static_token_may_admin1390=== PAUSE TestService_RequireScope_OIDC/static_token_may_admin1391=== RUN TestService_RequireScope_OIDC/static_token_may_write1392=== PAUSE TestService_RequireScope_OIDC/static_token_may_write1393=== RUN TestService_RequireScope_OIDC/reader_may_read1394=== PAUSE TestService_RequireScope_OIDC/reader_may_read1395=== RUN TestService_RequireScope_OIDC/writer_implies_read1396=== PAUSE TestService_RequireScope_OIDC/writer_implies_read1397=== RUN TestService_RequireScope_OIDC/anonymous_may_not_read1398=== PAUSE TestService_RequireScope_OIDC/anonymous_may_not_read1399=== RUN TestService_RequireScope_OIDC/reader_may_read_with_netrc1400=== PAUSE TestService_RequireScope_OIDC/reader_may_read_with_netrc1401=== RUN TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1402=== PAUSE TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1403=== RUN TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read1404=== PAUSE TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read1405=== RUN TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1406=== PAUSE TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1407=== CONT TestClientMultipleUploads14082026/10/09 09:57:57 INFO Received push request method=POST path=/api/pushes14092026/10/09 09:57:57 INFO Received complete multipart upload request method=POST path=/api/multipart/complete14102026/10/09 09:57:57 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)14112026/10/09 09:57:57 INFO Uploading irf7apapgkmyry2z90ashdg40g27dhib-test-script (136B)14122026/10/09 09:57:57 INFO Completed multipart upload object_key=nar/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb00000000000000000000.nar.zst upload_id=MTE5MjdlYzItNjE3OS00ZDA5LTljZjMtZGFiOThmMGQ2ZDQ0LjIzNmU2NTc4LWVkN2UtNGY1Ni1hN2ZjLTU5ZTY0MzQ1NDUxM3gxNzkxNTM5ODcwNDAyNDc5MDAw parts=1014132026/10/09 09:57:57 INFO Received complete push request method=POST path=/api/pushes/1/complete14142026/10/09 09:57:57 INFO Completed push id=1 objects=6 duration=22.64675ms14152026/10/09 09:57:57 WARN Failed to register uploaded object key=nar/1s7yghixzpinyqm7g59kyn3sav81pmnsdi5v63m6x4r2vacy907j.nar.zst error="server returned 404: 404 page not found\n"14162026/10/09 09:57:57 WARN Failed to register uploaded object key=irf7apapgkmyry2z90ashdg40g27dhib.ls error="server returned 404: 404 page not found\n"14172026/10/09 09:57:57 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign14182026/10/09 09:57:57 WARN Failed to register uploaded object key=log/ny8v8b6b3fzlawc3gswr7kmdjs5svfxl-test-script.drv error="server returned 404: 404 page not found\n"14192026/10/09 09:57:57 INFO Signed narinfos id=1 count=114202026/10/09 09:57:57 INFO Uploading 1 narinfos1421=== CONT TestClientFallsBackToClosures1422--- PASS: TestPush_CompleteCommitsEveryRoot (15.24s)14232026/10/09 09:57:57 INFO Received complete push request method=POST path=/api/pushes/1/complete14242026/10/09 09:57:57 WARN Failed to register uploaded object key=irf7apapgkmyry2z90ashdg40g27dhib.narinfo error="server returned 404: 404 page not found\n"14252026/10/09 09:57:57 INFO Completed push id=1 objects=4 duration=52.092166ms14262026/10/09 09:57:57 INFO Upload complete. (291ms)14272026/10/09 09:57:57 INFO Aborted multipart uploads count=0 kept=014282026/10/09 09:57:57 WARN Force mode enabled - objects will be deleted immediately without grace period14292026/10/09 09:57:57 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=014302026/10/09 09:57:57 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted,O=Other"14312026/10/09 09:57:57 WARN mTLS auth: bound subjects configured but subject DN unavailable14322026/10/09 09:57:57 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted"14332026/10/09 09:57:57 INFO Vacuumed table table=pending_closures1434--- PASS: TestService_AuthMiddleware_MTLSBoundSubjects (7.40s)1435=== CONT TestReadProxy40414362026/10/09 09:57:57 INFO Vacuumed table table=pending_objects14372026/10/09 09:57:57 INFO Vacuumed table table=multipart_uploads14382026/10/09 09:57:57 INFO Vacuumed table table=closures14392026/10/09 09:57:57 INFO Vacuumed table table=objects1440=== NAME TestClientWithDependencies1441 client_integration_test.go:753: Skipping nix copy test - isolated store (/private/var/tb/5/scratch/00f1341bdd59a71e/build/TestClientWithDependencies4176572669/001/store) requires matching store prefix1442--- PASS: TestClientWithDependencies (11.04s)1443=== CONT TestGCTaskStore_Fail1444--- PASS: TestGCTaskStore_Fail (0.00s)1445=== CONT TestClientErrorHandling1446=== RUN TestClientErrorHandling/InvalidStorePath1447=== PAUSE TestClientErrorHandling/InvalidStorePath1448=== RUN TestClientErrorHandling/InvalidAuthToken1449=== PAUSE TestClientErrorHandling/InvalidAuthToken1450=== RUN TestClientErrorHandling/ServerNotAvailable1451=== PAUSE TestClientErrorHandling/ServerNotAvailable1452=== CONT TestLeadElectsOneAndHandsOver14532026/10/09 09:57:57 INFO Received complete multipart upload request method=POST path=/api/multipart/complete14542026/10/09 09:57:58 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggggg00000000000000000000.nar.zst upload_id=MTE5MjdlYzItNjE3OS00ZDA5LTljZjMtZGFiOThmMGQ2ZDQ0LmUzM2I5NjkwLTY0MTMtNDY3My04YTRmLTY0ZWI1OGVhNTExYngxNzkxNTM5ODY4OTQ4MTY2MDAw parts=101455--- PASS: TestService_ReadScope_PublicByDefault (8.10s)1456=== CONT TestPush_RepeatedRootCommitsOnce1457=== NAME TestObjectStatsLargeCommit1458 object_stats_test.go:101: committing 60000 objects took 8.843845166s14592026/10/09 09:57:58 ERROR Failed to decode narinfo key=3hcdxyjf9yiq7qf3i4548drb6sjmwa1v.narinfo limit=16777216 error=<nil>1460--- PASS: TestReadProxyNarinfo (7.87s)1461--- FAIL: TestObjectStatsLargeCommit (20.31s)1462=== CONT TestProxyWriteTimeout1463=== CONT TestServerTLSConfig1464=== RUN TestServerTLSConfig/no_client_CA1465=== RUN TestProxyWriteTimeout/narinfo1466=== PAUSE TestProxyWriteTimeout/narinfo1467=== RUN TestProxyWriteTimeout/1_GiB_nar1468=== PAUSE TestServerTLSConfig/no_client_CA1469=== PAUSE TestProxyWriteTimeout/1_GiB_nar1470=== RUN TestProxyWriteTimeout/10_GiB_nar1471=== PAUSE TestProxyWriteTimeout/10_GiB_nar1472=== RUN TestServerTLSConfig/missing_CA_file1473=== RUN TestProxyWriteTimeout/unknown_size1474=== PAUSE TestServerTLSConfig/missing_CA_file1475=== PAUSE TestProxyWriteTimeout/unknown_size1476=== RUN TestServerTLSConfig/not_a_PEM_file1477=== CONT TestConnectWaitsForAPeerMigration1478=== PAUSE TestServerTLSConfig/not_a_PEM_file1479=== CONT TestConnectSerialisesConcurrentMigrations1480--- PASS: TestReadProxyRangeRequest (7.46s)1481=== CONT TestMetricsInventory1482--- PASS: TestReadRedirectNar (7.53s)1483=== CONT TestPush_OverlappingRootsStoreOneRowPerKey14842026/10/09 09:57:59 WARN Rate limiter enabled after throttle name=s3-test rate=514852026/10/09 09:57:59 WARN S3 rate limit hit operation="complete multipart upload" error="Please reduce your request rate."1486=== NAME TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle1487 throttle_test.go:213: Proxy stats: total=13, throttled=10, completeMultipart=101488 throttle_test.go:215: Rate limiter: enabled=true, rate=5.001489--- PASS: TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle (13.77s)1490=== CONT TestPresentNotReportedWhileGCDeletesClosure1491=== RUN TestPresentNotReportedWhileGCDeletesClosure/stale1492=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/stale1493=== RUN TestPresentNotReportedWhileGCDeletesClosure/fresh1494=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/fresh1495=== CONT TestReadProxyNarStreaming1496--- PASS: TestReadProxyRootRedirectsToIndexHTML (7.76s)1497=== CONT TestParseSingleRange1498=== RUN TestParseSingleRange/none1499=== PAUSE TestParseSingleRange/none1500=== RUN TestParseSingleRange/unknown_unit1501=== PAUSE TestParseSingleRange/unknown_unit1502=== RUN TestParseSingleRange/multi-range_ignored1503=== PAUSE TestParseSingleRange/multi-range_ignored1504=== RUN TestParseSingleRange/malformed_no_dash1505=== PAUSE TestParseSingleRange/malformed_no_dash1506=== RUN TestParseSingleRange/malformed_both_empty1507=== PAUSE TestParseSingleRange/malformed_both_empty1508=== RUN TestParseSingleRange/malformed_end_before_start1509=== PAUSE TestParseSingleRange/malformed_end_before_start1510=== RUN TestParseSingleRange/closed1511=== PAUSE TestParseSingleRange/closed1512=== RUN TestParseSingleRange/open-ended1513=== PAUSE TestParseSingleRange/open-ended1514=== RUN TestParseSingleRange/end_clamped_to_size1515=== PAUSE TestParseSingleRange/end_clamped_to_size1516=== RUN TestParseSingleRange/suffix1517=== PAUSE TestParseSingleRange/suffix1518=== RUN TestParseSingleRange/suffix_exceeds_size1519=== PAUSE TestParseSingleRange/suffix_exceeds_size1520=== RUN TestParseSingleRange/single_byte1521=== PAUSE TestParseSingleRange/single_byte1522=== RUN TestParseSingleRange/start_past_EOF1523=== PAUSE TestParseSingleRange/start_past_EOF1524=== RUN TestParseSingleRange/start_far_past_EOF1525=== PAUSE TestParseSingleRange/start_far_past_EOF1526=== CONT TestIsValidCachePath1527=== RUN TestIsValidCachePath/narinfo1528=== PAUSE TestIsValidCachePath/narinfo1529=== RUN TestIsValidCachePath/narinfo_all_nix_base32_chars1530=== PAUSE TestIsValidCachePath/narinfo_all_nix_base32_chars1531=== RUN TestIsValidCachePath/nar_zst1532=== PAUSE TestIsValidCachePath/nar_zst1533=== RUN TestIsValidCachePath/nar_xz1534=== PAUSE TestIsValidCachePath/nar_xz1535=== RUN TestIsValidCachePath/nar_bz21536=== PAUSE TestIsValidCachePath/nar_bz21537=== RUN TestIsValidCachePath/nar_uncompressed1538=== PAUSE TestIsValidCachePath/nar_uncompressed1539=== RUN TestIsValidCachePath/ls1540=== PAUSE TestIsValidCachePath/ls1541=== RUN TestIsValidCachePath/log1542=== PAUSE TestIsValidCachePath/log1543=== RUN TestIsValidCachePath/realisation1544=== PAUSE TestIsValidCachePath/realisation1545=== RUN TestIsValidCachePath/nix-cache-info1546=== PAUSE TestIsValidCachePath/nix-cache-info1547=== RUN TestIsValidCachePath/index.html1548=== PAUSE TestIsValidCachePath/index.html1549=== RUN TestIsValidCachePath/traversal_parent1550=== PAUSE TestIsValidCachePath/traversal_parent1551=== RUN TestIsValidCachePath/traversal_in_middle1552=== PAUSE TestIsValidCachePath/traversal_in_middle1553=== RUN TestIsValidCachePath/invalid_char_e1554=== PAUSE TestIsValidCachePath/invalid_char_e1555=== RUN TestIsValidCachePath/invalid_char_u1556=== PAUSE TestIsValidCachePath/invalid_char_u1557=== RUN TestIsValidCachePath/random_path1558=== PAUSE TestIsValidCachePath/random_path1559=== RUN TestIsValidCachePath/empty1560=== PAUSE TestIsValidCachePath/empty1561=== RUN TestIsValidCachePath/leading_slash1562=== PAUSE TestIsValidCachePath/leading_slash1563=== RUN TestIsValidCachePath/wrong_extension1564=== PAUSE TestIsValidCachePath/wrong_extension1565=== RUN TestIsValidCachePath/short_hash1566=== PAUSE TestIsValidCachePath/short_hash1567=== CONT TestResolveDBConnectionString15682026/10/09 09:57:59 INFO Received complete multipart upload request method=POST path=/api/multipart/complete1569=== RUN TestResolveDBConnectionString/flag_wins1570=== PAUSE TestResolveDBConnectionString/flag_wins1571=== RUN TestResolveDBConnectionString/file_when_flag_empty1572=== PAUSE TestResolveDBConnectionString/file_when_flag_empty1573=== RUN TestResolveDBConnectionString/missing_file_is_an_error1574=== PAUSE TestResolveDBConnectionString/missing_file_is_an_error1575=== RUN TestResolveDBConnectionString/PGHOST_allows_empty1576=== PAUSE TestResolveDBConnectionString/PGHOST_allows_empty1577=== RUN TestResolveDBConnectionString/nothing_configured1578=== PAUSE TestResolveDBConnectionString/nothing_configured1579=== CONT TestDeleteClosuresRacingPins15802026/10/09 09:57:59 INFO Completed multipart upload object_key=nar/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb00000000000000000000.nar.zst upload_id=MTE5MjdlYzItNjE3OS00ZDA5LTljZjMtZGFiOThmMGQ2ZDQ0LjBjYTI4NWE2LTNkMDUtNGQ4OS04M2FhLTUxZTNmM2FiNGM0ZHgxNzkxNTM5ODY4OTQ4MTkxMDAw parts=101581--- PASS: TestReadProxyConditionalGet (7.49s)1582=== CONT TestClientCADerivations1583=== NAME TestClientIntegration1584 client_integration_test.go:334: Created store path: /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestClientIntegration2081764932/002/store/45b2lzrw4y0rlm7xrxp9n421q405wqbi-test-file.txt15852026/10/09 09:58:00 INFO Received push request method=POST path=/api/pushes15862026/10/09 09:58:00 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)15872026/10/09 09:58:00 INFO Uploading 45b2lzrw4y0rlm7xrxp9n421q405wqbi-test-file.txt (152B)15882026/10/09 09:58:00 WARN Failed to register uploaded object key=nar/11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1.nar.zst error="server returned 404: 404 page not found\n"15892026/10/09 09:58:00 WARN Failed to register uploaded object key=45b2lzrw4y0rlm7xrxp9n421q405wqbi.ls error="server returned 404: 404 page not found\n"15902026/10/09 09:58:00 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign15912026/10/09 09:58:00 INFO Signed narinfos id=1 count=115922026/10/09 09:58:01 INFO Uploading 1 narinfos15932026/10/09 09:58:01 INFO Received complete push request method=POST path=/api/pushes/1/complete15942026/10/09 09:58:01 WARN Failed to register uploaded object key=45b2lzrw4y0rlm7xrxp9n421q405wqbi.narinfo error="server returned 404: 404 page not found\n"15952026/10/09 09:58:01 INFO Completed push id=1 objects=3 duration=27.312958ms15962026/10/09 09:58:01 INFO Upload complete. (302ms)15972026/10/09 09:58:01 INFO All 1 paths already cached1598 client_integration_test.go:360: Retrieved narinfo from S3:1599 StorePath: /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestClientIntegration2081764932/002/store/45b2lzrw4y0rlm7xrxp9n421q405wqbi-test-file.txt1600 URL: nar/11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1.nar.zst1601 Compression: zstd1602 NarHash: sha256:11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk11603 NarSize: 1521604 References: 1605 CA: fixed:r:sha256:11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk11606 client_integration_test.go:361: Retrieved .ls file from S3 (compressed size: 77 bytes)1607 client_integration_test.go:361: Decompressed .ls content (64 bytes):1608 {"version":1,"root":{"type":"regular","size":39,"narOffset":96}}1609--- PASS: TestReadProxyDisabled (7.68s)1610=== CONT TestPresentRefreshesOnlyStaleClosures16112026/10/09 09:58:01 INFO Received push request method=POST path=/api/pushes16122026/10/09 09:58:01 INFO Object in database but missing from S3 key=45b2lzrw4y0rlm7xrxp9n421q405wqbi.ls16132026/10/09 09:58:01 WARN Found objects in DB but missing from S3, will re-upload count=116142026/10/09 09:58:01 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)16152026/10/09 09:58:01 INFO Received complete push request method=POST path=/api/pushes/2/complete16162026/10/09 09:58:01 WARN Failed to register uploaded object key=45b2lzrw4y0rlm7xrxp9n421q405wqbi.ls error="server returned 404: 404 page not found\n"16172026/10/09 09:58:01 INFO Completed push id=2 objects=3 duration=1.358917ms16182026/10/09 09:58:01 INFO Upload complete. (143ms)1619=== NAME TestClientIntegration1620 client_integration_test.go:389: Retrieved .ls file from S3 (compressed size: 62 bytes)1621 client_integration_test.go:389: Decompressed .ls content (49 bytes):1622 {"version":1,"root":{"type":"regular","size":39}}16232026/10/09 09:58:01 INFO Received push request method=POST path=/api/pushes16242026/10/09 09:58:01 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)16252026/10/09 09:58:01 INFO Received complete push request method=POST path=/api/pushes/3/complete16262026/10/09 09:58:01 WARN Failed to register uploaded object key=45b2lzrw4y0rlm7xrxp9n421q405wqbi.ls error="server returned 404: 404 page not found\n"16272026/10/09 09:58:01 INFO Completed push id=3 objects=3 duration=1.488167ms16282026/10/09 09:58:01 INFO Upload complete. (127ms)16292026/10/09 09:58:01 ERROR failed to check GC advisory lock error="query pg_locks: failed to connect to `user=_tribuchetbld5 database=niks3`: /nonexistent/.s.PGSQL.5432 (/nonexistent): dial error: dial unix /nonexistent/.s.PGSQL.5432: connect: no such file or directory"1630 client_integration_test.go:405: Retrieved .ls file from S3 (compressed size: 62 bytes)1631 client_integration_test.go:405: Decompressed .ls content (49 bytes):1632 {"version":1,"root":{"type":"regular","size":39}}1633--- PASS: TestGCAdvisoryLockBlocksConcurrentRun (16.76s)1634=== CONT TestPresentReportsOnlyClosureRoots16352026/10/09 09:58:01 INFO Received complete multipart upload request method=POST path=/api/multipart/complete16362026/10/09 09:58:01 INFO Completed multipart upload object_key=nar/cccccccccccccccccccccccccccccccc00000000000000000000.nar.zst upload_id=MTE5MjdlYzItNjE3OS00ZDA5LTljZjMtZGFiOThmMGQ2ZDQ0LmQwYjg4ZDkzLTIwZjItNGYxNC04ZGJlLWZhZWE1OTZhMGI4NHgxNzkxNTM5ODY4OTQ4MDg2MDAw parts=1016372026-10-09 09:58:01.922 UTC [65876] ERROR: could not obtain lock on row in relation "closures"16382026-10-09 09:58:01.922 UTC [65876] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT16392026/10/09 09:58:01 INFO Received push request method=POST path=/api/pushes16402026-10-09 09:58:01.935 UTC [65876] ERROR: could not obtain lock on row in relation "closures"16412026-10-09 09:58:01.935 UTC [65876] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT16422026/10/09 09:58:01 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)16432026/10/09 09:58:01 INFO Uploading wb7jix9zmfkmv1g4vdn34rxj89kz2pnh-lost-commit.txt (152B)16442026-10-09 09:58:01.943 UTC [65876] ERROR: could not obtain lock on row in relation "closures"16452026-10-09 09:58:01.943 UTC [65876] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT16462026-10-09 09:58:01.948 UTC [65876] ERROR: could not obtain lock on row in relation "closures"16472026-10-09 09:58:01.948 UTC [65876] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT16482026-10-09 09:58:01.948 UTC [65876] ERROR: could not obtain lock on row in relation "closures"16492026-10-09 09:58:01.948 UTC [65876] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT16502026/10/09 09:58:01 WARN Failed to register uploaded object key=wb7jix9zmfkmv1g4vdn34rxj89kz2pnh.ls error="server returned 404: 404 page not found\n"16512026/10/09 09:58:01 WARN Failed to register uploaded object key=nar/04d5klw9iix884nk82y7nlkps8afh2bg119y7zkvcf1pc1x2y2jl.nar.zst error="server returned 404: 404 page not found\n"16522026/10/09 09:58:01 INFO Received sign narinfos request method=POST path=/api/pushes/4/sign16532026/10/09 09:58:01 INFO Signed narinfos id=4 count=116542026/10/09 09:58:01 INFO Uploading 1 narinfos1655--- PASS: TestPush_RootsAreLockedInByteOrder (20.00s)1656=== CONT TestGCEndsOnShutdown1657=== RUN TestGCEndsOnShutdown/before_the_run1658=== PAUSE TestGCEndsOnShutdown/before_the_run1659=== RUN TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete1660=== PAUSE TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete16612026/10/09 09:58:02 INFO Received complete push request method=POST path=/api/pushes/4/complete1662=== RUN TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark16632026/10/09 09:58:02 WARN Failed to register uploaded object key=wb7jix9zmfkmv1g4vdn34rxj89kz2pnh.narinfo error="server returned 404: 404 page not found\n"16642026/10/09 09:58:02 INFO Completed push id=4 objects=3 duration=877.708µs1665=== PAUSE TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark1666=== CONT TestGCTaskStore_GetReturnsLatest16672026/10/09 09:58:02 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:61533/api/pushes/4/complete\": EOF" url=http://127.0.0.1:61533/api/pushes/4/complete1668--- PASS: TestGCTaskStore_GetReturnsLatest (0.00s)1669=== CONT TestGCTaskStore_DeduplicateSameParams1670--- PASS: TestGCTaskStore_DeduplicateSameParams (0.00s)1671=== CONT TestGCTaskStore_ConflictDifferentParams1672--- PASS: TestGCTaskStore_ConflictDifferentParams (0.00s)1673=== CONT TestGCTaskStore_StartNew1674--- PASS: TestGCTaskStore_StartNew (0.00s)1675=== CONT TestGCTaskStore_GetEmpty1676--- PASS: TestGCTaskStore_GetEmpty (0.00s)1677=== CONT TestCacheStatsHandler1678--- PASS: TestReadProxyHead (6.76s)16792026/10/09 09:58:02 INFO Received complete push request method=POST path=/api/pushes/4/complete1680=== CONT TestGCTaskStore_PhaseUpdates1681--- PASS: TestGCTaskStore_PhaseUpdates (0.00s)16822026-10-09 09:58:02.128 UTC [65985] ERROR: Push does not exist: id=416832026-10-09 09:58:02.128 UTC [65985] CONTEXT: PL/pgSQL function commit_push(bigint) line 11 at RAISE16842026-10-09 09:58:02.128 UTC [65985] STATEMENT: -- name: CommitPush :one1685 SELECT commit_push($1::bigint)::bigint1686 1687=== CONT TestPushWaitsForSweepHoldingRow16882026/10/09 09:58:02 WARN Push not found id=416892026/10/09 09:58:02 INFO Upload complete. (278ms)1690=== NAME TestClientIntegration1691 client_integration_test.go:435: Retrieved narinfo from S3:1692 StorePath: /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestClientIntegration2081764932/002/store/wb7jix9zmfkmv1g4vdn34rxj89kz2pnh-lost-commit.txt1693 URL: nar/04d5klw9iix884nk82y7nlkps8afh2bg119y7zkvcf1pc1x2y2jl.nar.zst1694 Compression: zstd1695 NarHash: sha256:04d5klw9iix884nk82y7nlkps8afh2bg119y7zkvcf1pc1x2y2jl1696 NarSize: 1521697 References: 1698 CA: fixed:r:sha256:04d5klw9iix884nk82y7nlkps8afh2bg119y7zkvcf1pc1x2y2jl1699 client_integration_test.go:438: Testing garbage collection...17002026/10/09 09:58:02 INFO Starting cleanup of old closures method=DELETE path=/api/closures17012026/10/09 09:58:02 INFO Garbage collection started17022026/10/09 09:58:02 INFO Aborted multipart uploads count=0 kept=017032026/10/09 09:58:02 WARN Force mode enabled - objects will be deleted immediately without grace period17042026-10-09 09:58:02.294 UTC [66074] ERROR: duplicate key value violates unique constraint "pg_type_typname_nsp_index"17052026-10-09 09:58:02.294 UTC [66074] DETAIL: Key (typname, typnamespace)=(goose_db_version, 2200) already exists.17062026-10-09 09:58:02.294 UTC [66074] STATEMENT: CREATE TABLE goose_db_version (1707 id integer PRIMARY KEY GENERATED BY DEFAULT AS IDENTITY,1708 version_id bigint NOT NULL,1709 is_applied boolean NOT NULL,1710 tstamp timestamp NOT NULL DEFAULT now()1711 )17122026/10/09 09:58:02 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=2 objects-marked-for-deletion=6 objects-deleted-after-grace-period=6 objects-failed-to-delete=017132026/10/09 09:58:02 INFO Vacuumed table table=pending_closures17142026/10/09 09:58:02 INFO Vacuumed table table=pending_objects17152026/10/09 09:58:02 INFO Vacuumed table table=multipart_uploads17162026/10/09 09:58:02 INFO Vacuumed table table=closures17172026/10/09 09:58:02 INFO Vacuumed table table=objects1718=== CONT TestService_AuthMiddleware_MTLSProxyHeader1719--- PASS: TestReadProxyInvalidPath (7.23s)17202026/10/09 09:58:02 INFO Received push request method=POST path=/api/pushes17212026/10/09 09:58:02 INFO Uploading 2 paths to 127.0.0.1 (0 already cached)17222026/10/09 09:58:02 INFO Uploading 5c5zi1r1y48bkjhw0c1ach12f9qvyfmp-shared-dep (136B)17232026/10/09 09:58:02 INFO Uploading fk1l0pgwnb8ykv2938ysa7h0fzilx3if-top (264B)17242026/10/09 09:58:02 WARN Failed to register uploaded object key=5c5zi1r1y48bkjhw0c1ach12f9qvyfmp.ls error="server returned 404: 404 page not found\n"17252026/10/09 09:58:02 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"17262026/10/09 09:58:02 WARN Failed to register uploaded object key=fk1l0pgwnb8ykv2938ysa7h0fzilx3if.ls error="server returned 404: 404 page not found\n"17272026/10/09 09:58:02 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign17282026/10/09 09:58:02 WARN Failed to register uploaded object key=nar/0iqpvzd3p6vfgig0mgnq65fg56hwhkl3fdfm8bygqh3q33wwj93v.nar.zst error="server returned 404: 404 page not found\n"17292026/10/09 09:58:02 INFO Signed narinfos id=1 count=217302026/10/09 09:58:02 INFO Uploading 2 narinfos17312026/10/09 09:58:03 WARN Failed to register uploaded object key=fk1l0pgwnb8ykv2938ysa7h0fzilx3if.narinfo error="server returned 404: 404 page not found\n"17322026/10/09 09:58:03 INFO Received complete push request method=POST path=/api/pushes/1/complete17332026/10/09 09:58:03 WARN Failed to register uploaded object key=5c5zi1r1y48bkjhw0c1ach12f9qvyfmp.narinfo error="server returned 404: 404 page not found\n"17342026/10/09 09:58:03 INFO Completed push id=1 objects=6 duration=32.0545ms17352026/10/09 09:58:03 INFO Upload complete. (237ms)1736=== NAME TestClientSharedPathCommittedMidPush1737 client_integration_test.go:816: Retrieved narinfo from S3:1738 StorePath: /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestClientSharedPathCommittedMidPush2098838153/001/store/5c5zi1r1y48bkjhw0c1ach12f9qvyfmp-shared-dep1739 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1740 Compression: zstd1741 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821742 NarSize: 1361743 References: 1744 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n1745 client_integration_test.go:816: Retrieved narinfo from S3:1746 StorePath: /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestClientSharedPathCommittedMidPush2098838153/001/store/fk1l0pgwnb8ykv2938ysa7h0fzilx3if-top1747 URL: nar/0iqpvzd3p6vfgig0mgnq65fg56hwhkl3fdfm8bygqh3q33wwj93v.nar.zst1748 Compression: zstd1749 NarHash: sha256:0iqpvzd3p6vfgig0mgnq65fg56hwhkl3fdfm8bygqh3q33wwj93v1750 NarSize: 2641751 References: /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestClientSharedPathCommittedMidPush2098838153/001/store/5c5zi1r1y48bkjhw0c1ach12f9qvyfmp-shared-dep1752 CA: text:sha256:0i3n7vcsklsr2aifnh5wr7yxqhwjg2k0b9crkj27p9bhqwn5rmin1753--- PASS: TestClientSharedPathCommittedMidPush (8.24s)1754=== CONT TestCommitRacingPendingCleanupKeepsObjects1755--- PASS: TestReadProxyContentNegotiation (7.14s)1756=== CONT TestSweepRechecksPendingAfterLocking1757--- PASS: TestConnectSerialisesConcurrentMigrations (4.64s)1758=== CONT TestDeletePinKeepsRowWhenS3Fails17592026/10/09 09:58:03 INFO Aborted multipart uploads count=0 kept=017602026/10/09 09:58:03 WARN Force mode enabled - objects will be deleted immediately without grace period17612026/10/09 09:58:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=017622026/10/09 09:58:03 INFO Vacuumed table table=pending_closures17632026/10/09 09:58:03 INFO Vacuumed table table=pending_objects17642026/10/09 09:58:03 INFO Vacuumed table table=multipart_uploads1765=== NAME TestPinProtectsFromGC1766 client_integration_test.go:867: Pinned store path: /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestPinProtectsFromGC2645952582/001/store/pdrcsd9i9ijjm0kq6kqvagrql0r3gshc-pinned-file.txt1767 client_integration_test.go:868: Unpinned store path: /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestPinProtectsFromGC2645952582/001/store/jn0wsxl6dny48rhxhky5arzyhg47sysv-unpinned-file.txt17682026/10/09 09:58:03 INFO Vacuumed table table=closures17692026/10/09 09:58:03 INFO Vacuumed table table=objects1770--- PASS: TestGCMetrics (6.89s)1771=== CONT TestCreatePinRejectsBadInput17722026/10/09 09:58:03 INFO Received push request method=POST path=/api/pushes17732026/10/09 09:58:03 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)17742026/10/09 09:58:03 INFO Uploading pdrcsd9i9ijjm0kq6kqvagrql0r3gshc-pinned-file.txt (128B)17752026/10/09 09:58:03 WARN Failed to register uploaded object key=nar/0qmhsg42pqsz7x6hh02j8gznnac3yi3qpjhrqbwvz88i194jbb5s.nar.zst error="server returned 404: 404 page not found\n"17762026/10/09 09:58:03 WARN Failed to register uploaded object key=pdrcsd9i9ijjm0kq6kqvagrql0r3gshc.ls error="server returned 404: 404 page not found\n"17772026/10/09 09:58:03 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign17782026/10/09 09:58:03 INFO Signed narinfos id=1 count=117792026/10/09 09:58:03 INFO Uploading 1 narinfos1780=== CONT TestConcurrentPinUpdatesAgree1781--- PASS: TestReadProxyNarinfoAlreadyDecompressed (7.11s)17822026/10/09 09:58:03 INFO Received complete push request method=POST path=/api/pushes/1/complete17832026/10/09 09:58:03 WARN Failed to register uploaded object key=pdrcsd9i9ijjm0kq6kqvagrql0r3gshc.narinfo error="server returned 404: 404 page not found\n"17842026/10/09 09:58:03 INFO Completed push id=1 objects=3 duration=23.210875ms17852026/10/09 09:58:03 INFO Upload complete. (187ms)17862026/10/09 09:58:04 INFO Received push request method=POST path=/api/pushes17872026/10/09 09:58:04 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)17882026/10/09 09:58:04 INFO Uploading jn0wsxl6dny48rhxhky5arzyhg47sysv-unpinned-file.txt (128B)17892026/10/09 09:58:04 WARN Failed to register uploaded object key=nar/0gh0wdnr6gmk4r5sfzzh12isnl14rgmkrwy1ws41hpsrlndclg5h.nar.zst error="server returned 404: 404 page not found\n"17902026/10/09 09:58:04 INFO Received sign narinfos request method=POST path=/api/pushes/2/sign17912026/10/09 09:58:04 WARN Failed to register uploaded object key=jn0wsxl6dny48rhxhky5arzyhg47sysv.ls error="server returned 404: 404 page not found\n"17922026/10/09 09:58:04 INFO Signed narinfos id=2 count=117932026/10/09 09:58:04 INFO Uploading 1 narinfos17942026/10/09 09:58:04 INFO Received complete push request method=POST path=/api/pushes/2/complete17952026/10/09 09:58:04 WARN Failed to register uploaded object key=jn0wsxl6dny48rhxhky5arzyhg47sysv.narinfo error="server returned 404: 404 page not found\n"17962026/10/09 09:58:04 INFO Completed push id=2 objects=3 duration=958.083µs17972026/10/09 09:58:04 INFO Upload complete. (121ms)17982026/10/09 09:58:04 INFO Starting cleanup of old closures method=DELETE path=/api/closures17992026/10/09 09:58:04 INFO Garbage collection started18002026/10/09 09:58:04 INFO Garbage collection progress phase="" failed_uploads_deleted=0 old_closures_deleted=2 objects_marked=6 objects_deleted=6 objects_failed=018012026/10/09 09:58:04 INFO Aborted multipart uploads count=0 kept=01802=== NAME TestClientIntegration1803 client_integration_test.go:445: Objects in database after GC:1804 client_integration_test.go:445: Successfully deleted all objects with GC --force18052026/10/09 09:58:04 WARN Force mode enabled - objects will be deleted immediately without grace period18062026/10/09 09:58:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=01807--- PASS: TestClientIntegration (11.41s)1808=== CONT TestOrphanedObjectsGCStressTest18092026/10/09 09:58:04 INFO Vacuumed table table=pending_closures18102026/10/09 09:58:04 INFO Vacuumed table table=pending_objects1811=== NAME TestClientMultipleUploads1812 client_integration_test.go:480: Created store path 0: /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestClientMultipleUploads1516919282/001/store/izqdx8ksj5dggrf9w7d6zw1w8dk0cphg-test-file-0.txt18132026/10/09 09:58:04 INFO Vacuumed table table=multipart_uploads18142026/10/09 09:58:04 INFO Vacuumed table table=closures18152026/10/09 09:58:04 INFO Vacuumed table table=objects1816 client_integration_test.go:480: Created store path 1: /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestClientMultipleUploads1516919282/001/store/6a6kq2kbllxn8sqpgq9lf9dhkkr0wh59-test-file-1.txt1817 client_integration_test.go:480: Created store path 2: /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestClientMultipleUploads1516919282/001/store/rqkifjy739cgi0bca3yw8mbjslfc5vms-test-file-2.txt18182026/10/09 09:58:04 INFO Received push request method=POST path=/api/pushes18192026/10/09 09:58:04 INFO Uploading 3 paths to 127.0.0.1 (0 already cached)18202026/10/09 09:58:04 INFO Uploading rqkifjy739cgi0bca3yw8mbjslfc5vms-test-file-2.txt (160B)18212026/10/09 09:58:04 INFO Uploading 6a6kq2kbllxn8sqpgq9lf9dhkkr0wh59-test-file-1.txt (160B)18222026/10/09 09:58:04 INFO Uploading izqdx8ksj5dggrf9w7d6zw1w8dk0cphg-test-file-0.txt (160B)18232026/10/09 09:58:04 WARN Failed to register uploaded object key=6a6kq2kbllxn8sqpgq9lf9dhkkr0wh59.ls error="server returned 404: 404 page not found\n"18242026/10/09 09:58:04 WARN Failed to register uploaded object key=rqkifjy739cgi0bca3yw8mbjslfc5vms.ls error="server returned 404: 404 page not found\n"18252026/10/09 09:58:04 WARN Failed to register uploaded object key=izqdx8ksj5dggrf9w7d6zw1w8dk0cphg.ls error="server returned 404: 404 page not found\n"18262026/10/09 09:58:04 WARN Failed to register uploaded object key=nar/0ahq3izw3h0b0c08p3xdfclh758fwxnp85ccvflasb4vnfsv50ri.nar.zst error="server returned 404: 404 page not found\n"18272026/10/09 09:58:04 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign18282026/10/09 09:58:04 WARN Failed to register uploaded object key=nar/1823g6vx4q5zqvr73m7lkq1wpk40fin7bsc792di430yk7sg4lka.nar.zst error="server returned 404: 404 page not found\n"18292026/10/09 09:58:04 WARN Failed to register uploaded object key=nar/13p9pqi0ga6fx2apcwls9hjnv7nmf740ggpn5zvfzdsp5vz2bbg4.nar.zst error="server returned 404: 404 page not found\n"18302026/10/09 09:58:04 INFO Signed narinfos id=1 count=318312026/10/09 09:58:04 INFO Uploading 3 narinfos18322026/10/09 09:58:04 INFO Received complete push request method=POST path=/api/pushes/1/complete18332026/10/09 09:58:04 WARN Failed to register uploaded object key=rqkifjy739cgi0bca3yw8mbjslfc5vms.narinfo error="server returned 404: 404 page not found\n"18342026/10/09 09:58:04 WARN Failed to register uploaded object key=6a6kq2kbllxn8sqpgq9lf9dhkkr0wh59.narinfo error="server returned 404: 404 page not found\n"18352026/10/09 09:58:04 WARN Failed to register uploaded object key=izqdx8ksj5dggrf9w7d6zw1w8dk0cphg.narinfo error="server returned 404: 404 page not found\n"18362026/10/09 09:58:04 INFO Completed push id=1 objects=9 duration=14.505792ms18372026/10/09 09:58:04 INFO Upload complete. (187ms)1838 client_integration_test.go:491: Uploaded 3 paths in 228.728375ms18392026/10/09 09:58:04 INFO Received uploads request method=POST path=/api/pending_closures1840--- PASS: TestClientMultipleUploads (7.72s)1841=== CONT TestCreatePin_ReservedPins18422026/10/09 09:58:05 INFO Received uploads request method=POST path=/api/pending_closures18432026/10/09 09:58:05 INFO Uploading 2 paths to 127.0.0.1 (1 already cached)18442026/10/09 09:58:05 INFO Uploading xxzxrcmv6wadq97qdac2cd9dk468xa1m-a (256B)18452026/10/09 09:58:05 INFO Uploading m0sn449c9cypp5v62g8n3vdq4vaffrcf-shared-dep (136B)18462026/10/09 09:58:05 WARN Rate limiter enabled after throttle name=s3-test rate=518472026/10/09 09:58:05 WARN S3 rate limit hit during proxy key=4hcdxyjf9yiq7qf3i4548drb6sjmwa1v.narinfo error="Please reduce your request rate."18482026/10/09 09:58:05 WARN Failed to register uploaded object key=3f3r9z4p2isbyqakgx82awx024gzkrcy.ls error="server returned 404: 404 page not found\n"18492026/10/09 09:58:05 WARN Failed to register uploaded object key=xxzxrcmv6wadq97qdac2cd9dk468xa1m.ls error="server returned 404: 404 page not found\n"18502026/10/09 09:58:05 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"18512026/10/09 09:58:05 WARN Failed to register uploaded object key=nar/1chzg4pqjcm1s2nm5fgdkry9dkrym3gd4qxjzpi272n979qlvdk0.nar.zst error="server returned 404: 404 page not found\n"18522026/10/09 09:58:05 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign18532026/10/09 09:58:05 WARN Failed to register uploaded object key=m0sn449c9cypp5v62g8n3vdq4vaffrcf.ls error="server returned 404: 404 page not found\n"18542026/10/09 09:58:05 INFO Signed narinfos id=1 count=218552026/10/09 09:58:05 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign18562026/10/09 09:58:05 INFO Signed narinfos id=2 count=218572026/10/09 09:58:05 INFO Uploading 4 narinfos18582026/10/09 09:58:05 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:61638/oidc1859--- PASS: TestReadProxyOutlastsServerWriteTimeout (12.23s)1860=== CONT TestSweepSkipsRowsLockedByPush18612026/10/09 09:58:05 WARN Failed to register uploaded object key=m0sn449c9cypp5v62g8n3vdq4vaffrcf.narinfo error="server returned 404: 404 page not found\n"18622026/10/09 09:58:05 WARN Failed to register uploaded object key=xxzxrcmv6wadq97qdac2cd9dk468xa1m.narinfo error="server returned 404: 404 page not found\n"18632026/10/09 09:58:05 WARN Failed to register uploaded object key=3f3r9z4p2isbyqakgx82awx024gzkrcy.narinfo error="server returned 404: 404 page not found\n"18642026/10/09 09:58:05 INFO lead: acquired remote=192.0.2.1:123418652026/10/09 09:58:05 WARN Rate limiter backed off name=s3-test rate=518662026/10/09 09:58:05 WARN S3 rate limit hit during proxy key=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Please reduce your request rate."18672026/10/09 09:58:05 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete18682026/10/09 09:58:05 WARN Failed to register uploaded object key=m0sn449c9cypp5v62g8n3vdq4vaffrcf.narinfo error="server returned 404: 404 page not found\n"1869--- PASS: TestReadProxy404 (7.53s)1870=== CONT TestGetExistingObjectsLocksRows18712026/10/09 09:58:05 INFO Completed upload id=1 objects=6 duration=33.109167ms18722026/10/09 09:58:05 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete18732026/10/09 09:58:05 INFO Completed upload id=2 objects=6 duration=679.959µs18742026/10/09 09:58:05 INFO Upload complete. (325ms)1875=== NAME TestClientFallsBackToClosures1876 client_pushes_test.go:112: Retrieved narinfo from S3:1877 StorePath: /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestClientFallsBackToClosures3382990515/001/store/m0sn449c9cypp5v62g8n3vdq4vaffrcf-shared-dep1878 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1879 Compression: zstd1880 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821881 NarSize: 1361882 References: 1883 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n1884 client_pushes_test.go:112: Retrieved narinfo from S3:1885 StorePath: /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestClientFallsBackToClosures3382990515/001/store/xxzxrcmv6wadq97qdac2cd9dk468xa1m-a1886 URL: nar/1chzg4pqjcm1s2nm5fgdkry9dkrym3gd4qxjzpi272n979qlvdk0.nar.zst1887 Compression: zstd1888 NarHash: sha256:1chzg4pqjcm1s2nm5fgdkry9dkrym3gd4qxjzpi272n979qlvdk01889 NarSize: 2561890 References: /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestClientFallsBackToClosures3382990515/001/store/m0sn449c9cypp5v62g8n3vdq4vaffrcf-shared-dep1891 CA: text:sha256:10n9icachmk13jydkmabgzy7qd58vn5ll960zscj088g1pnbwrq31892 client_pushes_test.go:112: Retrieved narinfo from S3:1893 StorePath: /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestClientFallsBackToClosures3382990515/001/store/3f3r9z4p2isbyqakgx82awx024gzkrcy-b1894 URL: nar/1chzg4pqjcm1s2nm5fgdkry9dkrym3gd4qxjzpi272n979qlvdk0.nar.zst1895 Compression: zstd1896 NarHash: sha256:1chzg4pqjcm1s2nm5fgdkry9dkrym3gd4qxjzpi272n979qlvdk01897 NarSize: 2561898 References: /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestClientFallsBackToClosures3382990515/001/store/m0sn449c9cypp5v62g8n3vdq4vaffrcf-shared-dep1899 CA: text:sha256:10n9icachmk13jydkmabgzy7qd58vn5ll960zscj088g1pnbwrq31900--- PASS: TestClientFallsBackToClosures (7.87s)1901=== CONT TestRegisterCompletedObjectNeedsPendingRow19022026/10/09 09:58:05 INFO Received push request method=POST path=/api/pushes1903--- PASS: TestMetricsInventory (7.12s)1904=== CONT TestLeadIncumbentAnnouncesAtOnce19052026/10/09 09:58:06 INFO Received push request method=POST path=/api/pushes19062026/10/09 09:58:06 INFO Garbage collection progress phase="" failed_uploads_deleted=0 old_closures_deleted=1 objects_marked=3 objects_deleted=3 objects_failed=019072026/10/09 09:58:06 INFO Received create pin request method=POST path=/api/pins/myapp1908--- PASS: TestPush_OverlappingRootsStoreOneRowPerKey (7.01s)1909=== CONT TestForceGCDuringPushOffersSweptObject/after_presence_check19102026/10/09 09:58:06 INFO Created/updated pin name=myapp store_path=/private/var/tb/5/scratch/00f1341bdd59a71e/build/TestPinProtectsFromGC2645952582/001/store/pdrcsd9i9ijjm0kq6kqvagrql0r3gshc-pinned-file.txt narinfo_key=pdrcsd9i9ijjm0kq6kqvagrql0r3gshc.narinfo19112026/10/09 09:58:06 INFO Starting cleanup of old closures method=DELETE path=/api/closures19122026/10/09 09:58:06 INFO Garbage collection started19132026/10/09 09:58:06 INFO Aborted multipart uploads count=0 kept=019142026/10/09 09:58:06 WARN Force mode enabled - objects will be deleted immediately without grace period19152026/10/09 09:58:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=019162026/10/09 09:58:06 INFO Vacuumed table table=pending_closures19172026/10/09 09:58:06 INFO Vacuumed table table=pending_objects19182026/10/09 09:58:06 INFO Vacuumed table table=multipart_uploads19192026/10/09 09:58:06 INFO Vacuumed table table=closures19202026/10/09 09:58:06 INFO Vacuumed table table=objects19212026/10/09 09:58:06 INFO lead: released remote=192.0.2.1:123419222026/10/09 09:58:06 INFO lead: acquired remote=192.0.2.1:12341923--- PASS: TestReadProxyNarStreaming (7.03s)1924=== CONT TestForceGCDuringPushOffersSweptObject/before_pending_rows19252026/10/09 09:58:07 INFO Received complete multipart upload request method=POST path=/api/multipart/complete19262026/10/09 09:58:07 INFO Completed multipart upload object_key=nar/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa00000000000000000000.nar.zst upload_id=MTE5MjdlYzItNjE3OS00ZDA5LTljZjMtZGFiOThmMGQ2ZDQ0LmIzMzU4OTY3LWRkNjgtNGNmZS04OGJhLTc4OTE4ZjAyN2E5NXgxNzkxNTM5ODg1NTUzODIxMDAw parts=1019272026/10/09 09:58:07 INFO lead: released remote=192.0.2.1:12341928--- PASS: TestLeadElectsOneAndHandsOver (9.62s)1929=== CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal19302026/10/09 09:58:07 INFO Received uploads request method=POST path=/1931=== CONT TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key19322026/10/09 09:58:07 INFO Received complete multipart upload request method=POST path=/1933=== CONT TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key19342026/10/09 09:58:07 INFO Received request for more parts method=POST path=/1935=== CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info19362026/10/09 09:58:07 INFO Received uploads request method=POST path=/1937=== CONT TestPendingClosureWriteTimeout/670k_objects1938=== CONT TestPendingClosureWriteTimeout/empty1939--- PASS: TestUploadHandlersRejectInvalidKeys (0.01s)1940 --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal (0.01s)1941 --- PASS: TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key (0.00s)1942 --- PASS: TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key (0.01s)1943 --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info (0.00s)1944=== CONT TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline1945--- PASS: TestPresentRefreshesOnlyStaleClosures (6.37s)1946=== CONT TestPendingClosureWriteTimeout/400_objects1947=== CONT TestIsValidUploadKey/build_log_home-manager_file1948=== CONT TestIsValidUploadKey/unknown_type1949=== CONT TestPendingClosureFailureTracksEveryUpload/refuse_aborts=false1950=== NAME TestClientCADerivations1951 client_ca_test.go:136: Built CA derivation: /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestClientCADerivations48090870/001/store/sbspf4zal7xybqarcb96sazpvn09dw1j-ca-test1952 client_ca_test.go:139: Found 1 dependencies (including self)1953--- PASS: TestPresentReportsOnlyClosureRoots (6.25s)1954=== CONT TestIsValidUploadKey/absolute1955=== CONT TestIsValidUploadKey/traversal_nar1956=== CONT TestIsValidUploadKey/traversal1957=== CONT TestIsValidUploadKey/listing_key,_narinfo_type1958=== CONT TestPendingClosureFailureTracksEveryUpload/refuse_aborts=true19592026/10/09 09:58:08 INFO Received push request method=POST path=/api/pushes19602026/10/09 09:58:08 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)19612026/10/09 09:58:08 INFO Uploading sbspf4zal7xybqarcb96sazpvn09dw1j-ca-test (144B)1962--- PASS: TestConnectWaitsForAPeerMigration (9.46s)1963=== CONT TestIsValidUploadKey/build_log_question_mark1964=== CONT TestIsValidUploadKey/index.html1965=== CONT TestIsValidUploadKey/build_log_plus_in_name1966=== CONT TestIsValidUploadKey/nix-cache-info1967=== CONT TestIsValidUploadKey/nar_key,_narinfo_type1968=== CONT TestIsValidUploadKey/realisation_plus_in_output1969=== CONT TestIsValidUploadKey/build_log1970=== CONT TestIsValidUploadKey/narinfo_key,_nar_type1971=== CONT TestIsValidUploadKey/build_log_equals1972=== CONT TestIsValidUploadKey/listing1973=== CONT TestIsValidUploadKey/empty_key1974=== CONT TestIsValidUploadKey/narinfo1975=== CONT TestIsValidUploadKey/nar_plain1976=== CONT TestIsValidUploadKey/realisation1977=== CONT TestIsValidUploadKey/nar_zst1978=== CONT TestIsValidUploadKey/nar_xz1979=== CONT TestUploadHandlersRejectOversizedBody/create_pending_closure1980--- PASS: TestIsValidUploadKey (0.02s)1981 --- PASS: TestIsValidUploadKey/build_log_home-manager_file (0.00s)1982 --- PASS: TestIsValidUploadKey/unknown_type (0.00s)1983 --- PASS: TestIsValidUploadKey/absolute (0.00s)1984 --- PASS: TestIsValidUploadKey/traversal_nar (0.00s)1985 --- PASS: TestIsValidUploadKey/traversal (0.00s)1986 --- PASS: TestIsValidUploadKey/listing_key,_narinfo_type (0.00s)1987 --- PASS: TestIsValidUploadKey/build_log_question_mark (0.00s)1988 --- PASS: TestIsValidUploadKey/index.html (0.00s)1989 --- PASS: TestIsValidUploadKey/build_log_plus_in_name (0.00s)1990 --- PASS: TestIsValidUploadKey/nix-cache-info (0.00s)1991 --- PASS: TestIsValidUploadKey/nar_key,_narinfo_type (0.00s)1992 --- PASS: TestIsValidUploadKey/realisation_plus_in_output (0.00s)1993 --- PASS: TestIsValidUploadKey/build_log (0.00s)1994 --- PASS: TestIsValidUploadKey/narinfo_key,_nar_type (0.00s)1995 --- PASS: TestIsValidUploadKey/build_log_equals (0.00s)1996 --- PASS: TestIsValidUploadKey/listing (0.00s)1997 --- PASS: TestIsValidUploadKey/empty_key (0.00s)1998 --- PASS: TestIsValidUploadKey/narinfo (0.00s)1999 --- PASS: TestIsValidUploadKey/nar_plain (0.00s)2000 --- PASS: TestIsValidUploadKey/realisation (0.00s)2001 --- PASS: TestIsValidUploadKey/nar_zst (0.00s)2002 --- PASS: TestIsValidUploadKey/nar_xz (0.00s)20032026/10/09 09:58:08 INFO Received uploads request method=POST path=/20042026/10/09 09:58:08 WARN Failed to register uploaded object key=nar/04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n.nar.zst error="server returned 404: 404 page not found\n"20052026/10/09 09:58:08 WARN Failed to register uploaded object key=sbspf4zal7xybqarcb96sazpvn09dw1j.ls error="server returned 404: 404 page not found\n"20062026/10/09 09:58:08 WARN Failed to register uploaded object key=log/bbk7vwwnbls3hamrmv5i6i7fpd2b0h8y-ca-test.drv error="server returned 404: 404 page not found\n"20072026/10/09 09:58:08 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign20082026/10/09 09:58:08 INFO Signed narinfos id=1 count=120092026/10/09 09:58:08 INFO Uploading 1 narinfos20102026/10/09 09:58:08 INFO Received complete push request method=POST path=/api/pushes/1/complete20112026/10/09 09:58:08 WARN Failed to register uploaded object key=sbspf4zal7xybqarcb96sazpvn09dw1j.narinfo error="server returned 404: 404 page not found\n"2012=== NAME TestPinProtectsFromGC2013 client_integration_test.go:976: Pin successfully protected closure from garbage collection20142026/10/09 09:58:08 INFO Completed push id=1 objects=4 duration=42.304709ms20152026/10/09 09:58:08 INFO Upload complete. (358ms)2016=== NAME TestClientCADerivations2017 client_ca_test.go:180: Narinfo contains CA field: StorePath: /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestClientCADerivations48090870/001/store/sbspf4zal7xybqarcb96sazpvn09dw1j-ca-test2018 URL: nar/04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n.nar.zst2019 Compression: zstd2020 NarHash: sha256:04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n2021 NarSize: 1442022 References: 2023 Deriver: /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestClientCADerivations48090870/001/store/bbk7vwwnbls3hamrmv5i6i7fpd2b0h8y-ca-test.drv2024 CA: fixed:r:sha256:04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n2025 client_ca_test.go:185: Checking for realisation files in S3...2026 client_ca_test.go:207: No realisation files uploaded - this is expected for locally built CA derivations2027 client_ca_test.go:208: Locally built CA derivations don't have DrvOutput IDs until they're substituted from a cache2028--- PASS: TestPinProtectsFromGC (12.54s)2029=== CONT TestUploadHandlersRejectOversizedBody/request_more_parts20302026/10/09 09:58:08 INFO Received request for more parts method=POST path=/2031--- PASS: TestCacheStatsHandler (6.43s)2032=== CONT TestUploadHandlersRejectOversizedBody/complete_multipart20332026/10/09 09:58:08 INFO Received complete multipart upload request method=POST path=/2034=== NAME TestClientCADerivations2035 client_ca_test.go:258: nix copy output: error: binary cache 's3://bucket80?endpoint=http://localhost:60965®ion=eu-west-1' is for Nix stores with prefix '/nix/store', not '/private/var/tb/5/scratch/00f1341bdd59a71e/build/TestClientCADerivations48090870/001/store'2036 client_ca_test.go:259: nix copy failed (might be expected with isolated stores): exit status 12037--- PASS: TestClientCADerivations (8.55s)2038=== CONT TestCacheConfigHandler/full_config,_no_issuer2039=== CONT TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator2040=== CONT TestCacheConfigHandler/no_signing_keys2041=== CONT TestCacheConfigHandler/no_cache_url_configured2042=== CONT TestPush_RejectsBadRequests/no_objects2043--- PASS: TestCacheConfigHandler (0.00s)2044 --- PASS: TestCacheConfigHandler/full_config,_no_issuer (0.00s)2045 --- PASS: TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator (0.00s)2046 --- PASS: TestCacheConfigHandler/no_signing_keys (0.00s)2047 --- PASS: TestCacheConfigHandler/no_cache_url_configured (0.00s)20482026/10/09 09:58:08 INFO Received push request method=POST path=/api/pushes2049=== CONT TestPush_RejectsBadRequests/bad_root20502026/10/09 09:58:08 INFO Received push request method=POST path=/api/pushes2051=== CONT TestPush_RejectsBadRequests/no_roots20522026/10/09 09:58:08 INFO Received push request method=POST path=/api/pushes2053=== CONT TestPush_RejectsBadRequests/root_not_in_objects20542026/10/09 09:58:08 INFO Received push request method=POST path=/api/pushes2055=== CONT TestService_AuthMiddleware_OIDC/malformed_token_rejected20562026/10/09 09:58:08 WARN Authentication failed token_preview=not-a-valid-jwt token_length=15 oidc_error="no provider could verify the token (signature or issuer mismatch)" oidc_provider="" tried_providers=[test]2057=== CONT TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured2058=== CONT TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected20592026/10/09 09:58:08 WARN Authentication failed token_preview=eyJhbGciOi...Z8feoM4R0Q token_length=701 oidc_error="no rule matched: claim \"repository_owner\" value [otherorg] not in allowed patterns [myorg]" oidc_provider=test tried_providers=[test]2060=== CONT TestService_AuthMiddleware_OIDC/valid_OIDC_token2061=== CONT TestService_RequireScope_OIDC/builder_may_write2062=== CONT TestService_RequireScope_OIDC/static_token_may_write2063=== CONT TestService_RequireScope_OIDC/empty_netrc_password_may_not_read2064=== CONT TestService_RequireScope_OIDC/static_token_may_admin_with_netrc2065=== CONT TestService_RequireScope_OIDC/reader_may_read_with_netrc2066=== CONT TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read2067=== CONT TestService_RequireScope_OIDC/writer_implies_read20682026/10/09 09:58:08 WARN Authentication failed token_preview=wrong token_length=5 oidc_error="no provider could verify the token (signature or issuer mismatch)" oidc_provider="" tried_providers=[test]2069=== CONT TestService_RequireScope_OIDC/anonymous_may_not_read2070=== CONT TestService_RequireScope_OIDC/ops_may_not_write2071--- PASS: TestPush_RejectsBadRequests (8.38s)2072 --- PASS: TestPush_RejectsBadRequests/no_objects (0.00s)2073 --- PASS: TestPush_RejectsBadRequests/bad_root (0.01s)2074 --- PASS: TestPush_RejectsBadRequests/no_roots (0.00s)2075 --- PASS: TestPush_RejectsBadRequests/root_not_in_objects (0.00s)2076=== CONT TestService_RequireScope_OIDC/static_token_may_admin2077=== CONT TestService_RequireScope_OIDC/reader_may_not_write2078=== CONT TestService_RequireScope_OIDC/builder_may_not_admin2079=== CONT TestService_RequireScope_OIDC/reader_may_read2080=== CONT TestService_RequireScope_OIDC/ops_may_admin2081=== CONT TestClientErrorHandling/InvalidStorePath2082--- PASS: TestService_AuthMiddleware_OIDC (8.91s)2083 --- PASS: TestService_AuthMiddleware_OIDC/malformed_token_rejected (0.01s)2084 --- PASS: TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured (0.00s)2085 --- PASS: TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected (0.01s)2086 --- PASS: TestService_AuthMiddleware_OIDC/valid_OIDC_token (0.00s)2087=== CONT TestClientErrorHandling/ServerNotAvailable20882026/10/09 09:58:08 INFO Received uploads request method=POST path=/api/pending_closures2089{"timestamp":"2026-10-09T09:58:08.779206Z","level":"ERROR","message":"save_config_with_opts: err: Io(Custom { kind: TooManyOpenFiles, error: \"Too many open files (os error 24)\" }), file: scanner/durable-dirty-producer-replay.json","target":"rustfs_ecstore::config::com","filename":"crates/ecstore/src/config/com.rs","line_number":813,"threadName":"rustfs-worker","threadId":"ThreadId(2)"}2090=== CONT TestClientErrorHandling/InvalidAuthToken2091--- PASS: TestService_RequireScope_OIDC (9.59s)2092 --- PASS: TestService_RequireScope_OIDC/builder_may_write (0.00s)2093 --- PASS: TestService_RequireScope_OIDC/static_token_may_write (0.00s)2094 --- PASS: TestService_RequireScope_OIDC/empty_netrc_password_may_not_read (0.00s)2095 --- PASS: TestService_RequireScope_OIDC/static_token_may_admin_with_netrc (0.00s)2096 --- PASS: TestService_RequireScope_OIDC/reader_may_read_with_netrc (0.00s)2097 --- PASS: TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read (0.01s)2098 --- PASS: TestService_RequireScope_OIDC/anonymous_may_not_read (0.00s)2099 --- PASS: TestService_RequireScope_OIDC/writer_implies_read (0.00s)2100 --- PASS: TestService_RequireScope_OIDC/static_token_may_admin (0.00s)2101 --- PASS: TestService_RequireScope_OIDC/ops_may_not_write (0.00s)2102 --- PASS: TestService_RequireScope_OIDC/reader_may_not_write (0.00s)2103 --- PASS: TestService_RequireScope_OIDC/builder_may_not_admin (0.00s)2104 --- PASS: TestService_RequireScope_OIDC/reader_may_read (0.00s)2105 --- PASS: TestService_RequireScope_OIDC/ops_may_admin (0.00s)2106{"timestamp":"2026-10-09T09:58:08.906349Z","level":"ERROR","duration":"163.301583ms","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}2107{"timestamp":"2026-10-09T09:58:08.906473Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"58176551-7cdd-498b-9ba8-72b6e2722e38","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket75/nar/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb00000000000000000000.nar.zst","status_code":500,"suppressed_errors":0,"duration_ms":163,"result":"server_error","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":501,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}2108=== NAME TestPush_RepeatedRootCommitsOnce2109 pushes_test.go:162: expected http status 200 for part 9, got 5002110 pushes_test.go:162: no ETag in response for part 921112026/10/09 09:58:08 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://localhost:19999/api/objects/present\": dial tcp 127.0.0.1:19999: connect: connection refused" url=http://localhost:19999/api/objects/present2112{"timestamp":"2026-10-09T09:58:08.996192Z","level":"ERROR","message":"Scanner bucket cache save failed","event":"scanner_cache_persist_state","component":"scanner","subsystem":"io","bucket":"bucket7","cache_name":"bucket7/.usage-cache.bin","state":"save_failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::scanner::io","filename":"crates/scanner/src/scanner_io/io_cache.rs","line_number":1373,"threadName":"rustfs-worker","threadId":"ThreadId(10)"}21132026/10/09 09:58:09 INFO Received complete multipart upload request method=POST path=/api/multipart/complete21142026/10/09 09:58:09 ERROR Failed to complete multipart upload error="One or more of the specified parts could not be found. The part may not have been uploaded, or the specified entity tag may not match the part's entity tag." object_key=nar/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb00000000000000000000.nar.zst upload_id=MTE5MjdlYzItNjE3OS00ZDA5LTljZjMtZGFiOThmMGQ2ZDQ0LmQ5NzhjZWJlLTVjYjItNDlmYy05MGI3LTQxMDUxZTlkOTUyOHgxNzkxNTM5ODg1NTU0Mzk0MDAw2115 pushes_test.go:162: unexpected http status=500 body=failed to complete multipart upload: One or more of the specified parts could not be found. The part may not have been uploaded, or the specified entity tag may not match the part's entity tag.2116 2117 21182026-10-09 09:58:09.040 UTC [65334] LOG: checkpoint starting: wal2119{"timestamp":"2026-10-09T09:58:09.065359Z","level":"ERROR","duration":"427.833µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(2)"}2120{"timestamp":"2026-10-09T09:58:09.066941Z","level":"ERROR","message":"Disk local scan failed","event":"disk_local_scan_failed","component":"ecstore","subsystem":"disk_local","path":"","operation":"list_dir","error":"Io(Os { code: 24, kind: TooManyOpenFiles, message: \"Too many open files\" })","target":"rustfs_ecstore::disk::local","filename":"crates/ecstore/src/disk/local.rs","line_number":7605,"threadName":"rustfs-worker","threadId":"ThreadId(6)"}2121{"timestamp":"2026-10-09T09:58:09.066989Z","level":"ERROR","message":"Metacache listing quorum failed","event":"metacache_listing","component":"ecstore","subsystem":"metacache","bucket":"bucket85","path":"","state":"quorum_failed","error":"drive /private/var/tb/5/scratch/00f1341bdd59a71e/build/rustfs694584765/data returned: io error Too many open files (os error 24)","target":"rustfs_ecstore::cache_value::metacache_set","filename":"crates/ecstore/src/cache_value/metacache_set.rs","line_number":909,"threadName":"rustfs-worker","threadId":"ThreadId(6)"}2122{"timestamp":"2026-10-09T09:58:09.067008Z","level":"ERROR","message":"Metacache merge job failed","event":"metacache_listing","component":"ecstore","subsystem":"metacache","bucket":"bucket85","path":"","state":"merge_job_failed","error":"ErasureReadQuorum","target":"rustfs_ecstore::cache_value::metacache_set","filename":"crates/ecstore/src/cache_value/metacache_set.rs","line_number":1002,"threadName":"rustfs-worker","threadId":"ThreadId(6)"}2123{"timestamp":"2026-10-09T09:58:09.067056Z","level":"ERROR","message":"list_path worker failed","component":"store","stage":"list_merged","bucket":"bucket85","prefix":"","base_dir":"","limit":1001,"marker":"","error":"erasure read quorum","target":"rustfs_ecstore::store::list_objects","filename":"crates/ecstore/src/store/list_objects.rs","line_number":242,"threadName":"rustfs-worker","threadId":"ThreadId(6)"}2124{"timestamp":"2026-10-09T09:58:09.06707Z","level":"ERROR","message":"list_path worker failed","component":"store","stage":"trailing_worker_error","bucket":"bucket85","prefix":"","base_dir":"","limit":1001,"marker":"","error":"erasure read quorum","target":"rustfs_ecstore::store::list_objects","filename":"crates/ecstore/src/store/list_objects.rs","line_number":242,"threadName":"rustfs-worker","threadId":"ThreadId(6)"}2125{"timestamp":"2026-10-09T09:58:09.067092Z","level":"ERROR","duration":"289.333µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDownRead</Code><Message>Resource requested is unreadable, please reduce your request rate</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(6)"}2126{"timestamp":"2026-10-09T09:58:09.067101Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"032970fa-32b0-41ab-9254-e9f2986812d9","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"GET","uri":"/bucket85/","status_code":503,"suppressed_errors":0,"duration_ms":0,"result":"server_error","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":501,"threadName":"rustfs-worker","threadId":"ThreadId(6)"}2127{"timestamp":"2026-10-09T09:58:09.07386Z","level":"ERROR","duration":"32.146625ms","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(6)"}2128{"timestamp":"2026-10-09T09:58:09.082839Z","level":"ERROR","message":"save_config_with_opts: err: Io(Custom { kind: TooManyOpenFiles, error: \"Too many open files (os error 24)\" }), file: scanner/durable-dirty-producer-replay.json","target":"rustfs_ecstore::config::com","filename":"crates/ecstore/src/config/com.rs","line_number":813,"threadName":"rustfs-worker","threadId":"ThreadId(10)"}21292026/10/09 09:58:09 WARN Request failed, retrying attempt=2 backoff=204.056967ms error="Post \"http://localhost:19999/api/objects/present\": dial tcp 127.0.0.1:19999: connect: connection refused" url=http://localhost:19999/api/objects/present2130{"timestamp":"2026-10-09T09:58:09.113487Z","level":"ERROR","duration":"634.583µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(10)"}2131{"timestamp":"2026-10-09T09:58:09.194901Z","level":"ERROR","message":"data usage snapshot load failed","event":"data_usage_snapshot_load_failed","component":"ecstore","subsystem":"data_usage","state":"read_failed","operation":"remove_bucket_from_observed","reason":"io","object":"buckets/.usage.observed.json","error":"Io error: Too many open files (os error 24)","target":"rustfs_ecstore::data_usage","filename":"crates/ecstore/src/data_usage/mod.rs","line_number":968,"threadName":"rustfs-worker","threadId":"ThreadId(10)"}2132--- FAIL: TestPush_RepeatedRootCommitsOnce (11.14s)2133=== CONT TestProxyWriteTimeout/narinfo2134=== CONT TestProxyWriteTimeout/unknown_size2135=== CONT TestProxyWriteTimeout/10_GiB_nar2136=== CONT TestProxyWriteTimeout/1_GiB_nar2137=== CONT TestServerTLSConfig/no_client_CA2138--- PASS: TestProxyWriteTimeout (0.01s)2139 --- PASS: TestProxyWriteTimeout/narinfo (0.00s)2140 --- PASS: TestProxyWriteTimeout/unknown_size (0.00s)2141 --- PASS: TestProxyWriteTimeout/10_GiB_nar (0.00s)2142 --- PASS: TestProxyWriteTimeout/1_GiB_nar (0.00s)2143=== CONT TestServerTLSConfig/missing_CA_file2144=== CONT TestServerTLSConfig/not_a_PEM_file2145--- PASS: TestServerTLSConfig (0.01s)2146 --- PASS: TestServerTLSConfig/no_client_CA (0.00s)2147 --- PASS: TestServerTLSConfig/missing_CA_file (0.00s)2148 --- PASS: TestServerTLSConfig/not_a_PEM_file (0.02s)2149=== CONT TestPresentNotReportedWhileGCDeletesClosure/stale2150{"timestamp":"2026-10-09T09:58:09.254267Z","level":"ERROR","duration":"465.041µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(10)"}21512026/10/09 09:58:09 WARN Request failed, retrying attempt=3 backoff=371.915341ms error="Post \"http://localhost:19999/api/objects/present\": dial tcp 127.0.0.1:19999: connect: connection refused" url=http://localhost:19999/api/objects/present2152{"timestamp":"2026-10-09T09:58:09.302251Z","level":"ERROR","message":"save_config_with_opts: err: Io(Custom { kind: TooManyOpenFiles, error: \"Too many open files (os error 24)\" }), file: scanner/durable-dirty-producer-replay.json","target":"rustfs_ecstore::config::com","filename":"crates/ecstore/src/config/com.rs","line_number":813,"threadName":"rustfs-worker","threadId":"ThreadId(2)"}2153{"timestamp":"2026-10-09T09:58:09.306119Z","level":"ERROR","duration":"19.640667ms","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(2)"}2154{"timestamp":"2026-10-09T09:58:09.306729Z","level":"ERROR","message":"Heal scheduler task failed","event":"heal_scheduler_state","component":"heal","subsystem":"manager","task_id":"bcca5f0b-fd84-47cd-9a6f-ad089df61ac7","heal_type":"object","set":"pool_0_set_0","state":"task_failed","error":"Heal task execution failed: Object not found: bucket86/nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst","target":"rustfs::heal::manager","filename":"crates/heal/src/heal/manager/scheduler.rs","line_number":269,"threadName":"rustfs-worker","threadId":"ThreadId(7)"}2155{"timestamp":"2026-10-09T09:58:09.315758Z","level":"ERROR","duration":"30.256041ms","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(2)"}2156{"timestamp":"2026-10-09T09:58:09.317222Z","level":"ERROR","duration":"1.66725ms","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(7)"}2157{"timestamp":"2026-10-09T09:58:09.359339Z","level":"ERROR","duration":"660µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(10)"}2158{"timestamp":"2026-10-09T09:58:09.40127Z","level":"ERROR","message":"Scanner bucket cache save failed","event":"scanner_cache_persist_state","component":"scanner","subsystem":"io","bucket":"bucket8","cache_name":"bucket8/.usage-cache.bin","state":"save_failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::scanner::io","filename":"crates/scanner/src/scanner_io/io_cache.rs","line_number":1373,"threadName":"rustfs-worker","threadId":"ThreadId(7)"}2159{"timestamp":"2026-10-09T09:58:09.417015Z","level":"ERROR","duration":"443.458µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(10)"}2160{"timestamp":"2026-10-09T09:58:09.442516Z","level":"ERROR","duration":"446.083µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(2)"}2161{"timestamp":"2026-10-09T09:58:09.446557Z","level":"ERROR","duration":"17.875875ms","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(10)"}2162=== NAME TestSweepRechecksPendingAfterLocking2163 gc_races_test.go:635: unexpected error: We encountered an internal error, please try again.2164 2165{"timestamp":"2026-10-09T09:58:09.458605Z","level":"ERROR","duration":"550.916µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(2)"}2166{"timestamp":"2026-10-09T09:58:09.482916Z","level":"ERROR","duration":"484.583µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}2167{"timestamp":"2026-10-09T09:58:09.641673Z","level":"ERROR","duration":"692.917µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}2168{"timestamp":"2026-10-09T09:58:09.660286Z","level":"ERROR","duration":"19.308125ms","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(2)"}21692026/10/09 09:58:09 WARN Request failed, retrying attempt=4 backoff=761.908619ms error="Post \"http://localhost:19999/api/objects/present\": dial tcp 127.0.0.1:19999: connect: connection refused" url=http://localhost:19999/api/objects/present2170{"timestamp":"2026-10-09T09:58:09.729025Z","level":"ERROR","duration":"378.583µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(10)"}2171{"timestamp":"2026-10-09T09:58:09.736939Z","level":"ERROR","message":"Scanner bucket cache save failed","event":"scanner_cache_persist_state","component":"scanner","subsystem":"io","bucket":"bucket9","cache_name":"bucket9/.usage-cache.bin","state":"save_failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::scanner::io","filename":"crates/scanner/src/scanner_io/io_cache.rs","line_number":1373,"threadName":"rustfs-worker","threadId":"ThreadId(10)"}2172{"timestamp":"2026-10-09T09:58:09.737048Z","level":"ERROR","message":"Scanner cycle completed without a durable data usage snapshot","event":"scanner_persist_state","component":"scanner","subsystem":"runtime","cycle":6,"outcome":"NoUpdate","state":"usage_not_durable","target":"rustfs::scanner","filename":"crates/scanner/src/scanner.rs","line_number":2350,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}2173{"timestamp":"2026-10-09T09:58:09.808186Z","level":"ERROR","duration":"358.5µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(10)"}2174{"timestamp":"2026-10-09T09:58:09.817506Z","level":"ERROR","duration":"297.292µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(10)"}2175{"timestamp":"2026-10-09T09:58:10.016772Z","level":"ERROR","duration":"373.458µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}2176{"timestamp":"2026-10-09T09:58:10.09051Z","level":"ERROR","duration":"442.5µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(6)"}2177{"timestamp":"2026-10-09T09:58:10.090629Z","level":"ERROR","duration":"561.875µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(7)"}2178{"timestamp":"2026-10-09T09:58:10.098924Z","level":"ERROR","message":"Scanner bucket cache save failed","event":"scanner_cache_persist_state","component":"scanner","subsystem":"io","bucket":"bucket1","cache_name":"bucket1/.usage-cache.bin","state":"save_failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::scanner::io","filename":"crates/scanner/src/scanner_io/io_cache.rs","line_number":1373,"threadName":"rustfs-worker","threadId":"ThreadId(7)"}2179{"timestamp":"2026-10-09T09:58:10.208967Z","level":"ERROR","duration":"406.875µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(2)"}2180{"timestamp":"2026-10-09T09:58:10.309578Z","level":"ERROR","duration":"14.953792ms","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(2)"}2181{"timestamp":"2026-10-09T09:58:10.34909Z","level":"ERROR","duration":"9.428292ms","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(6)"}2182{"timestamp":"2026-10-09T09:58:10.357989Z","level":"ERROR","message":"Scanner partial bucket cache save failed","event":"scanner_cache_persist_state","component":"scanner","subsystem":"io","bucket":"bucket10","cache_name":"bucket10/.usage-cache.bin","state":"partial_save_failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::scanner::io","filename":"crates/scanner/src/scanner_io/io_cache.rs","line_number":1297,"threadName":"rustfs-worker","threadId":"ThreadId(2)"}2183{"timestamp":"2026-10-09T09:58:10.413311Z","level":"ERROR","duration":"10.794ms","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(10)"}21842026/10/09 09:58:10 WARN Request failed, retrying attempt=5 backoff=1.629358619s error="Post \"http://localhost:19999/api/objects/present\": dial tcp 127.0.0.1:19999: connect: connection refused" url=http://localhost:19999/api/objects/present2185{"timestamp":"2026-10-09T09:58:10.651793Z","level":"ERROR","duration":"425.542µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}2186{"timestamp":"2026-10-09T09:58:10.656551Z","level":"ERROR","duration":"312.958µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(10)"}2187{"timestamp":"2026-10-09T09:58:10.666423Z","level":"ERROR","message":"Scanner bucket cache save failed","event":"scanner_cache_persist_state","component":"scanner","subsystem":"io","bucket":"bucket11","cache_name":"bucket11/.usage-cache.bin","state":"save_failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::scanner::io","filename":"crates/scanner/src/scanner_io/io_cache.rs","line_number":1373,"threadName":"rustfs-worker","threadId":"ThreadId(2)"}2188{"timestamp":"2026-10-09T09:58:10.707Z","level":"ERROR","duration":"313.25µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(6)"}2189{"timestamp":"2026-10-09T09:58:10.762209Z","level":"ERROR","duration":"10.598833ms","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(6)"}2190{"timestamp":"2026-10-09T09:58:10.827838Z","level":"ERROR","duration":"314.083µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}2191{"timestamp":"2026-10-09T09:58:10.935614Z","level":"ERROR","message":"Scanner bucket cache save failed","event":"scanner_cache_persist_state","component":"scanner","subsystem":"io","bucket":"bucket12","cache_name":"bucket12/.usage-cache.bin","state":"save_failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::scanner::io","filename":"crates/scanner/src/scanner_io/io_cache.rs","line_number":1373,"threadName":"rustfs-worker","threadId":"ThreadId(6)"}2192{"timestamp":"2026-10-09T09:58:10.983111Z","level":"ERROR","duration":"335.542µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}2193{"timestamp":"2026-10-09T09:58:11.102635Z","level":"ERROR","duration":"9.827084ms","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}2194{"timestamp":"2026-10-09T09:58:11.142343Z","level":"ERROR","duration":"345.167µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(10)"}2195{"timestamp":"2026-10-09T09:58:11.174448Z","level":"ERROR","duration":"298.375µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(10)"}2196{"timestamp":"2026-10-09T09:58:11.217036Z","level":"ERROR","duration":"303.958µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(2)"}2197{"timestamp":"2026-10-09T09:58:11.217125Z","level":"ERROR","message":"Scanner bucket cache save failed","event":"scanner_cache_persist_state","component":"scanner","subsystem":"io","bucket":"bucket13","cache_name":"bucket13/.usage-cache.bin","state":"save_failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::scanner::io","filename":"crates/scanner/src/scanner_io/io_cache.rs","line_number":1373,"threadName":"rustfs-worker","threadId":"ThreadId(2)"}2198{"timestamp":"2026-10-09T09:58:11.369277Z","level":"ERROR","duration":"362.875µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}2199{"timestamp":"2026-10-09T09:58:11.509662Z","level":"ERROR","message":"Scanner bucket cache save failed","event":"scanner_cache_persist_state","component":"scanner","subsystem":"io","bucket":"bucket14","cache_name":"bucket14/.usage-cache.bin","state":"save_failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::scanner::io","filename":"crates/scanner/src/scanner_io/io_cache.rs","line_number":1373,"threadName":"rustfs-worker","threadId":"ThreadId(10)"}2200{"timestamp":"2026-10-09T09:58:11.653783Z","level":"ERROR","duration":"415.25µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(6)"}2201{"timestamp":"2026-10-09T09:58:11.680759Z","level":"ERROR","duration":"336.208µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(10)"}2202{"timestamp":"2026-10-09T09:58:11.813764Z","level":"ERROR","message":"Scanner bucket cache save failed","event":"scanner_cache_persist_state","component":"scanner","subsystem":"io","bucket":"bucket15","cache_name":"bucket15/.usage-cache.bin","state":"save_failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::scanner::io","filename":"crates/scanner/src/scanner_io/io_cache.rs","line_number":1373,"threadName":"rustfs-worker","threadId":"ThreadId(2)"}2203{"timestamp":"2026-10-09T09:58:11.931235Z","level":"ERROR","duration":"333.083µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(2)"}2204{"timestamp":"2026-10-09T09:58:12.013798Z","level":"ERROR","duration":"10.014208ms","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(6)"}2205{"timestamp":"2026-10-09T09:58:12.023709Z","level":"ERROR","duration":"306.875µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(2)"}2206{"timestamp":"2026-10-09T09:58:12.024419Z","level":"ERROR","duration":"268.25µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(2)"}2207{"timestamp":"2026-10-09T09:58:12.087947Z","level":"ERROR","message":"Scanner bucket cache save failed","event":"scanner_cache_persist_state","component":"scanner","subsystem":"io","bucket":"bucket16","cache_name":"bucket16/.usage-cache.bin","state":"save_failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::scanner::io","filename":"crates/scanner/src/scanner_io/io_cache.rs","line_number":1373,"threadName":"rustfs-worker","threadId":"ThreadId(10)"}22082026/10/09 09:58:12 WARN Request failed, retrying attempt=1 backoff=100ms error="Get \"http://localhost:19999/api/cache-config\": dial tcp 127.0.0.1:19999: connect: connection refused" url=http://localhost:19999/api/cache-config2209{"timestamp":"2026-10-09T09:58:12.192889Z","level":"ERROR","duration":"370.125µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(2)"}22102026/10/09 09:58:12 WARN Request failed, retrying attempt=2 backoff=194.154309ms error="Get \"http://localhost:19999/api/cache-config\": dial tcp 127.0.0.1:19999: connect: connection refused" url=http://localhost:19999/api/cache-config22112026/10/09 09:58:12 WARN Request failed, retrying attempt=3 backoff=421.446163ms error="Get \"http://localhost:19999/api/cache-config\": dial tcp 127.0.0.1:19999: connect: connection refused" url=http://localhost:19999/api/cache-config2212{"timestamp":"2026-10-09T09:58:12.408925Z","level":"ERROR","message":"Scanner partial bucket cache save failed","event":"scanner_cache_persist_state","component":"scanner","subsystem":"io","bucket":"bucket17","cache_name":"bucket17/.usage-cache.bin","state":"partial_save_failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::scanner::io","filename":"crates/scanner/src/scanner_io/io_cache.rs","line_number":1297,"threadName":"rustfs-worker","threadId":"ThreadId(10)"}2213=== CONT TestPresentNotReportedWhileGCDeletesClosure/fresh2214--- PASS: TestUploadHandlersRejectOversizedBody (0.19s)2215 --- PASS: TestUploadHandlersRejectOversizedBody/request_more_parts (0.30s)2216 --- PASS: TestUploadHandlersRejectOversizedBody/complete_multipart (0.32s)2217 --- PASS: TestUploadHandlersRejectOversizedBody/create_pending_closure (4.35s)2218{"timestamp":"2026-10-09T09:58:12.589202Z","level":"ERROR","duration":"422.042µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(10)"}2219{"timestamp":"2026-10-09T09:58:12.669501Z","level":"ERROR","message":"Scanner bucket cache save failed","event":"scanner_cache_persist_state","component":"scanner","subsystem":"io","bucket":"bucket18","cache_name":"bucket18/.usage-cache.bin","state":"save_failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::scanner::io","filename":"crates/scanner/src/scanner_io/io_cache.rs","line_number":1373,"threadName":"rustfs-worker","threadId":"ThreadId(6)"}2220{"timestamp":"2026-10-09T09:58:12.669722Z","level":"ERROR","duration":"13.860125ms","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(2)"}22212026/10/09 09:58:12 WARN Request failed, retrying attempt=4 backoff=735.575204ms error="Get \"http://localhost:19999/api/cache-config\": dial tcp 127.0.0.1:19999: connect: connection refused" url=http://localhost:19999/api/cache-config2222{"timestamp":"2026-10-09T09:58:12.84792Z","level":"ERROR","duration":"1.9455ms","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(6)"}2223{"timestamp":"2026-10-09T09:58:12.873208Z","level":"ERROR","duration":"1.643708ms","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}2224{"timestamp":"2026-10-09T09:58:12.903528Z","level":"ERROR","duration":"1.385167ms","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(10)"}2225{"timestamp":"2026-10-09T09:58:12.920205Z","level":"ERROR","message":"Scanner bucket cache save failed","event":"scanner_cache_persist_state","component":"scanner","subsystem":"io","bucket":"bucket19","cache_name":"bucket19/.usage-cache.bin","state":"save_failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::scanner::io","filename":"crates/scanner/src/scanner_io/io_cache.rs","line_number":1373,"threadName":"rustfs-worker","threadId":"ThreadId(10)"}2226=== NAME TestService_AuthMiddleware_MTLSProxyHeader2227 request_test.go:70: store invariants: list multipart uploads: We encountered an internal error, please try again.2228--- FAIL: TestService_AuthMiddleware_MTLSProxyHeader (10.40s)2229=== CONT TestParseSingleRange/multi-range_ignored2230=== CONT TestParseSingleRange/closed2231=== CONT TestParseSingleRange/start_far_past_EOF2232=== CONT TestParseSingleRange/single_byte2233=== CONT TestParseSingleRange/suffix_exceeds_size2234=== CONT TestParseSingleRange/suffix2235=== CONT TestParseSingleRange/start_past_EOF2236=== CONT TestParseSingleRange/open-ended2237=== CONT TestParseSingleRange/malformed_no_dash2238=== CONT TestParseSingleRange/end_clamped_to_size2239=== CONT TestParseSingleRange/malformed_end_before_start2240=== CONT TestParseSingleRange/unknown_unit2241=== CONT TestParseSingleRange/malformed_both_empty2242=== CONT TestParseSingleRange/none2243=== CONT TestIsValidCachePath/nar_zst2244--- PASS: TestParseSingleRange (0.05s)2245 --- PASS: TestParseSingleRange/multi-range_ignored (0.00s)2246 --- PASS: TestParseSingleRange/closed (0.00s)2247 --- PASS: TestParseSingleRange/start_far_past_EOF (0.00s)2248 --- PASS: TestParseSingleRange/single_byte (0.00s)2249 --- PASS: TestParseSingleRange/suffix_exceeds_size (0.00s)2250 --- PASS: TestParseSingleRange/suffix (0.00s)2251 --- PASS: TestParseSingleRange/start_past_EOF (0.00s)2252 --- PASS: TestParseSingleRange/open-ended (0.00s)2253 --- PASS: TestParseSingleRange/malformed_no_dash (0.00s)2254 --- PASS: TestParseSingleRange/end_clamped_to_size (0.00s)2255 --- PASS: TestParseSingleRange/malformed_end_before_start (0.00s)2256 --- PASS: TestParseSingleRange/unknown_unit (0.00s)2257 --- PASS: TestParseSingleRange/malformed_both_empty (0.00s)2258 --- PASS: TestParseSingleRange/none (0.00s)2259=== CONT TestIsValidCachePath/invalid_char_u2260=== CONT TestIsValidCachePath/short_hash2261=== CONT TestIsValidCachePath/wrong_extension2262=== CONT TestIsValidCachePath/leading_slash2263=== CONT TestIsValidCachePath/random_path2264=== CONT TestIsValidCachePath/empty2265=== CONT TestIsValidCachePath/invalid_char_e2266=== CONT TestIsValidCachePath/traversal_in_middle2267=== CONT TestIsValidCachePath/traversal_parent2268=== CONT TestIsValidCachePath/index.html2269=== CONT TestIsValidCachePath/log2270=== CONT TestIsValidCachePath/realisation2271=== CONT TestIsValidCachePath/ls2272=== CONT TestIsValidCachePath/nar_uncompressed2273=== CONT TestIsValidCachePath/nar_xz2274=== CONT TestIsValidCachePath/nar_bz22275=== CONT TestIsValidCachePath/narinfo_all_nix_base32_chars2276=== CONT TestIsValidCachePath/narinfo2277=== CONT TestIsValidCachePath/nix-cache-info2278=== CONT TestResolveDBConnectionString/flag_wins2279--- PASS: TestIsValidCachePath (0.10s)2280 --- PASS: TestIsValidCachePath/nar_zst (0.00s)2281 --- PASS: TestIsValidCachePath/invalid_char_u (0.00s)2282 --- PASS: TestIsValidCachePath/short_hash (0.00s)2283 --- PASS: TestIsValidCachePath/wrong_extension (0.00s)2284 --- PASS: TestIsValidCachePath/leading_slash (0.00s)2285 --- PASS: TestIsValidCachePath/random_path (0.00s)2286 --- PASS: TestIsValidCachePath/empty (0.00s)2287 --- PASS: TestIsValidCachePath/invalid_char_e (0.00s)2288 --- PASS: TestIsValidCachePath/traversal_in_middle (0.00s)2289 --- PASS: TestIsValidCachePath/traversal_parent (0.00s)2290 --- PASS: TestIsValidCachePath/index.html (0.00s)2291 --- PASS: TestIsValidCachePath/log (0.00s)2292 --- PASS: TestIsValidCachePath/realisation (0.00s)2293 --- PASS: TestIsValidCachePath/ls (0.00s)2294 --- PASS: TestIsValidCachePath/nar_uncompressed (0.00s)2295 --- PASS: TestIsValidCachePath/nar_xz (0.00s)2296 --- PASS: TestIsValidCachePath/nar_bz2 (0.00s)2297 --- PASS: TestIsValidCachePath/narinfo_all_nix_base32_chars (0.00s)2298 --- PASS: TestIsValidCachePath/narinfo (0.00s)2299 --- PASS: TestIsValidCachePath/nix-cache-info (0.00s)2300=== CONT TestResolveDBConnectionString/PGHOST_allows_empty2301=== CONT TestResolveDBConnectionString/nothing_configured2302=== CONT TestResolveDBConnectionString/missing_file_is_an_error2303=== CONT TestResolveDBConnectionString/file_when_flag_empty2304=== CONT TestGCEndsOnShutdown/before_the_run2305--- PASS: TestResolveDBConnectionString (0.09s)2306 --- PASS: TestResolveDBConnectionString/flag_wins (0.00s)2307 --- PASS: TestResolveDBConnectionString/PGHOST_allows_empty (0.00s)2308 --- PASS: TestResolveDBConnectionString/nothing_configured (0.00s)2309 --- PASS: TestResolveDBConnectionString/missing_file_is_an_error (0.00s)2310 --- PASS: TestResolveDBConnectionString/file_when_flag_empty (0.00s)2311{"timestamp":"2026-10-09T09:58:13.220135Z","level":"ERROR","duration":"536.542µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}2312{"timestamp":"2026-10-09T09:58:13.226739Z","level":"ERROR","message":"Scanner bucket cache save failed","event":"scanner_cache_persist_state","component":"scanner","subsystem":"io","bucket":"bucket2","cache_name":"bucket2/.usage-cache.bin","state":"save_failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::scanner::io","filename":"crates/scanner/src/scanner_io/io_cache.rs","line_number":1373,"threadName":"rustfs-worker","threadId":"ThreadId(2)"}2313{"timestamp":"2026-10-09T09:58:13.300608Z","level":"ERROR","duration":"528.875µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(10)"}2314=== NAME TestPushWaitsForSweepHoldingRow2315 request_test.go:70: store invariants: list multipart uploads: We encountered an internal error, please try again.2316--- FAIL: TestPushWaitsForSweepHoldingRow (11.19s)2317=== CONT TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark2318{"timestamp":"2026-10-09T09:58:13.55611Z","level":"ERROR","message":"Scanner bucket cache save failed","event":"scanner_cache_persist_state","component":"scanner","subsystem":"io","bucket":"bucket20","cache_name":"bucket20/.usage-cache.bin","state":"save_failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::scanner::io","filename":"crates/scanner/src/scanner_io/io_cache.rs","line_number":1373,"threadName":"rustfs-worker","threadId":"ThreadId(2)"}23192026/10/09 09:58:13 WARN Request failed, retrying attempt=5 backoff=1.5382196s error="Get \"http://localhost:19999/api/cache-config\": dial tcp 127.0.0.1:19999: connect: connection refused" url=http://localhost:19999/api/cache-config2320=== NAME TestCommitRacingPendingCleanupKeepsObjects2321 gc_races_test.go:737: unexpected error: We encountered an internal error, please try again.2322 2323--- FAIL: TestCommitRacingPendingCleanupKeepsObjects (10.48s)2324=== CONT TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete2325{"timestamp":"2026-10-09T09:58:13.645351Z","level":"ERROR","duration":"12.946625ms","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}2326{"timestamp":"2026-10-09T09:58:13.676449Z","level":"ERROR","duration":"11.506333ms","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(10)"}2327{"timestamp":"2026-10-09T09:58:13.726377Z","level":"ERROR","duration":"13.235542ms","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(6)"}2328{"timestamp":"2026-10-09T09:58:13.727981Z","level":"ERROR","duration":"647.75µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(10)"}2329{"timestamp":"2026-10-09T09:58:13.797467Z","level":"ERROR","message":"save_config_with_opts: err: Io(Custom { kind: TooManyOpenFiles, error: \"Too many open files (os error 24)\" }), file: scanner/durable-dirty-producer-replay.json","target":"rustfs_ecstore::config::com","filename":"crates/ecstore/src/config/com.rs","line_number":813,"threadName":"rustfs-worker","threadId":"ThreadId(2)"}2330{"timestamp":"2026-10-09T09:58:13.816348Z","level":"ERROR","duration":"141.041µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(10)"}2331{"timestamp":"2026-10-09T09:58:13.82035Z","level":"ERROR","message":"Scanner bucket cache save failed","event":"scanner_cache_persist_state","component":"scanner","subsystem":"io","bucket":"bucket21","cache_name":"bucket21/.usage-cache.bin","state":"save_failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::scanner::io","filename":"crates/scanner/src/scanner_io/io_cache.rs","line_number":1373,"threadName":"rustfs-worker","threadId":"ThreadId(10)"}2332{"timestamp":"2026-10-09T09:58:14.06371Z","level":"ERROR","duration":"17.459167ms","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(2)"}2333{"timestamp":"2026-10-09T09:58:14.063794Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"5abe50b5-a709-4967-b608-91b657823c83","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket88/","status_code":500,"suppressed_errors":55,"duration_ms":17,"result":"server_error","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":501,"threadName":"rustfs-worker","threadId":"ThreadId(2)"}2334=== NAME TestSweepRechecksPendingAfterLocking2335 request_test.go:70: store invariants: list multipart uploads: We encountered an internal error, please try again.2336--- FAIL: TestSweepRechecksPendingAfterLocking (10.92s)2337{"timestamp":"2026-10-09T09:58:14.139687Z","level":"ERROR","message":"Scanner partial bucket cache save failed","event":"scanner_cache_persist_state","component":"scanner","subsystem":"io","bucket":"bucket22","cache_name":"bucket22/.usage-cache.bin","state":"partial_save_failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::scanner::io","filename":"crates/scanner/src/scanner_io/io_cache.rs","line_number":1297,"threadName":"rustfs-worker","threadId":"ThreadId(10)"}2338{"timestamp":"2026-10-09T09:58:14.182Z","level":"ERROR","duration":"15.743125ms","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(10)"}2339{"timestamp":"2026-10-09T09:58:14.28822Z","level":"ERROR","duration":"1.840833ms","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(10)"}2340{"timestamp":"2026-10-09T09:58:14.400103Z","level":"ERROR","message":"Scanner bucket cache save failed","event":"scanner_cache_persist_state","component":"scanner","subsystem":"io","bucket":"bucket23","cache_name":"bucket23/.usage-cache.bin","state":"save_failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::scanner::io","filename":"crates/scanner/src/scanner_io/io_cache.rs","line_number":1373,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}2341{"timestamp":"2026-10-09T09:58:14.774624Z","level":"ERROR","message":"Scanner bucket cache save failed","event":"scanner_cache_persist_state","component":"scanner","subsystem":"io","bucket":"bucket24","cache_name":"bucket24/.usage-cache.bin","state":"save_failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::scanner::io","filename":"crates/scanner/src/scanner_io/io_cache.rs","line_number":1373,"threadName":"rustfs-worker","threadId":"ThreadId(2)"}2342=== NAME TestDeleteClosuresRacingPins2343 request_test.go:70: store invariants: list multipart uploads: We encountered an internal error, please try again.2344--- FAIL: TestDeleteClosuresRacingPins (14.98s)2345{"timestamp":"2026-10-09T09:58:15.106985Z","level":"ERROR","message":"Scanner bucket cache save failed","event":"scanner_cache_persist_state","component":"scanner","subsystem":"io","bucket":"bucket25","cache_name":"bucket25/.usage-cache.bin","state":"save_failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::scanner::io","filename":"crates/scanner/src/scanner_io/io_cache.rs","line_number":1373,"threadName":"rustfs-worker","threadId":"ThreadId(10)"}23462026/10/09 09:58:15 WARN Failed to fetch cache config, uploading without size limit error="fetching cache config: sending request: request failed after retries: Get \"http://localhost:19999/api/cache-config\": dial tcp 127.0.0.1:19999: connect: connection refused"23472026/10/09 09:58:15 WARN Request failed, retrying attempt=1 backoff=100ms error="Get \"http://localhost:19999/api/cache-config\": dial tcp 127.0.0.1:19999: connect: connection refused" url=http://localhost:19999/api/cache-config2348{"timestamp":"2026-10-09T09:58:15.190028Z","level":"ERROR","duration":"15.831084ms","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(6)"}23492026/10/09 09:58:15 WARN Request failed, retrying attempt=2 backoff=196.408927ms error="Get \"http://localhost:19999/api/cache-config\": dial tcp 127.0.0.1:19999: connect: connection refused" url=http://localhost:19999/api/cache-config2350=== NAME TestDeletePinKeepsRowWhenS3Fails2351 pins_test.go:231: unexpected error: context deadline exceeded2352 2353--- FAIL: TestDeletePinKeepsRowWhenS3Fails (11.96s)2354{"timestamp":"2026-10-09T09:58:15.299942Z","level":"ERROR","duration":"10.468708ms","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(2)"}23552026/10/09 09:58:15 WARN Request failed, retrying attempt=3 backoff=400.780306ms error="Get \"http://localhost:19999/api/cache-config\": dial tcp 127.0.0.1:19999: connect: connection refused" url=http://localhost:19999/api/cache-config2356{"timestamp":"2026-10-09T09:58:15.431598Z","level":"ERROR","message":"Scanner bucket cache save failed","event":"scanner_cache_persist_state","component":"scanner","subsystem":"io","bucket":"bucket26","cache_name":"bucket26/.usage-cache.bin","state":"save_failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::scanner::io","filename":"crates/scanner/src/scanner_io/io_cache.rs","line_number":1373,"threadName":"rustfs-worker","threadId":"ThreadId(2)"}2357{"timestamp":"2026-10-09T09:58:15.431473Z","level":"ERROR","message":"Heal storage request failed","event":"heal_storage_object_io","component":"heal","subsystem":"storage","operation":"object_exists","bucket":".rustfs.sys","object":"buckets/bucket26/.usage-cache.bin","result":"failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::heal::storage","filename":"crates/heal/src/heal/storage.rs","line_number":1339,"threadName":"rustfs-worker","threadId":"ThreadId(6)"}2358{"timestamp":"2026-10-09T09:58:15.431736Z","level":"ERROR","message":"Heal scheduler task failed","event":"heal_scheduler_state","component":"heal","subsystem":"manager","task_id":"da92b888-c534-4274-aff8-bcc57e4e2dd5","heal_type":"object","set":"pool_0_set_0","state":"task_failed","error":"Other error: Io error: Too many open files (os error 24)","target":"rustfs::heal::manager","filename":"crates/heal/src/heal/manager/scheduler.rs","line_number":269,"threadName":"rustfs-worker","threadId":"ThreadId(6)"}2359{"timestamp":"2026-10-09T09:58:15.446414Z","level":"ERROR","duration":"644.708µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}2360{"timestamp":"2026-10-09T09:58:15.779011Z","level":"ERROR","message":"Heal storage request failed","event":"heal_storage_object_io","component":"heal","subsystem":"storage","operation":"object_exists","bucket":".rustfs.sys","object":"buckets/.usage.v2.json","result":"failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::heal::storage","filename":"crates/heal/src/heal/storage.rs","line_number":1339,"threadName":"rustfs-worker","threadId":"ThreadId(10)"}2361{"timestamp":"2026-10-09T09:58:15.779065Z","level":"ERROR","message":"Heal scheduler task failed","event":"heal_scheduler_state","component":"heal","subsystem":"manager","task_id":"cc604d50-0af9-4b1a-adae-e1db0cc7bcdc","heal_type":"object","set":"pool_0_set_0","state":"task_failed","error":"Other error: Io error: Too many open files (os error 24)","target":"rustfs::heal::manager","filename":"crates/heal/src/heal/manager/scheduler.rs","line_number":269,"threadName":"rustfs-worker","threadId":"ThreadId(10)"}2362{"timestamp":"2026-10-09T09:58:15.779375Z","level":"ERROR","duration":"12.142333ms","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(10)"}23632026/10/09 09:58:15 WARN Request failed, retrying attempt=4 backoff=826.192428ms error="Get \"http://localhost:19999/api/cache-config\": dial tcp 127.0.0.1:19999: connect: connection refused" url=http://localhost:19999/api/cache-config2364{"timestamp":"2026-10-09T09:58:15.87813Z","level":"ERROR","message":"Scanner partial bucket cache save failed","event":"scanner_cache_persist_state","component":"scanner","subsystem":"io","bucket":"bucket27","cache_name":"bucket27/.usage-cache.bin","state":"partial_save_failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::scanner::io","filename":"crates/scanner/src/scanner_io/io_cache.rs","line_number":1297,"threadName":"rustfs-worker","threadId":"ThreadId(2)"}2365=== NAME TestCreatePinRejectsBadInput2366 pins_test.go:189: unexpected error: context deadline exceeded2367 2368--- FAIL: TestCreatePinRejectsBadInput (12.17s)2369{"timestamp":"2026-10-09T09:58:15.944457Z","level":"ERROR","message":"op returns error","op":"CreateBucket","err":"S3Error(Inner { code: InternalError, message: Some(\"We encountered an internal error, please try again.\"), request_id: None, status_code: None, source: Some(Io(Custom { kind: TooManyOpenFiles, error: \"Too many open files (os error 24)\" })), headers: None })","target":"s3s::ops","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/ops/mod.rs","line_number":456,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}2370{"timestamp":"2026-10-09T09:58:15.944657Z","level":"ERROR","duration":"759.583µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}2371{"timestamp":"2026-10-09T09:58:16.001576Z","level":"ERROR","message":"op returns error","op":"CreateBucket","err":"S3Error(Inner { code: InternalError, message: Some(\"We encountered an internal error, please try again.\"), request_id: None, status_code: None, source: Some(Io(Custom { kind: TooManyOpenFiles, error: \"Too many open files (os error 24)\" })), headers: None })","target":"s3s::ops","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/ops/mod.rs","line_number":456,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}2372{"timestamp":"2026-10-09T09:58:16.001621Z","level":"ERROR","duration":"398.375µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}2373{"timestamp":"2026-10-09T09:58:16.037028Z","level":"ERROR","message":"op returns error","op":"CreateBucket","err":"S3Error(Inner { code: InternalError, message: Some(\"We encountered an internal error, please try again.\"), request_id: None, status_code: None, source: Some(Io(Custom { kind: TooManyOpenFiles, error: \"Too many open files (os error 24)\" })), headers: None })","target":"s3s::ops","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/ops/mod.rs","line_number":456,"threadName":"rustfs-worker","threadId":"ThreadId(2)"}2374{"timestamp":"2026-10-09T09:58:16.037076Z","level":"ERROR","duration":"438.25µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message></Error>\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(2)"}2375=== NAME TestConcurrentPinUpdatesAgree2376 pins_test.go:111: unexpected error: context deadline exceeded2377 2378--- FAIL: TestConcurrentPinUpdatesAgree (12.10s)2379{"timestamp":"2026-10-09T09:58:16.047155Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}2380{"timestamp":"2026-10-09T09:58:16.05199Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}2381{"timestamp":"2026-10-09T09:58:16.085236Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}2382{"timestamp":"2026-10-09T09:58:16.263117Z","level":"ERROR","message":"Scanner partial bucket cache save failed","event":"scanner_cache_persist_state","component":"scanner","subsystem":"io","bucket":"bucket28","cache_name":"bucket28/.usage-cache.bin","state":"partial_save_failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::scanner::io","filename":"crates/scanner/src/scanner_io/io_cache.rs","line_number":1297,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}23832026/10/09 09:58:16 WARN Request failed, retrying attempt=5 backoff=1.685071608s error="Get \"http://localhost:19999/api/cache-config\": dial tcp 127.0.0.1:19999: connect: connection refused" url=http://localhost:19999/api/cache-config2384{"timestamp":"2026-10-09T09:58:16.832392Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(2)"}2385=== NAME TestOrphanedObjectsGCStressTest2386 orphaned_objects_gc_test.go:304: unexpected error: context deadline exceeded2387 2388--- FAIL: TestOrphanedObjectsGCStressTest (12.78s)2389{"timestamp":"2026-10-09T09:58:17.10565Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(2)"}2390{"timestamp":"2026-10-09T09:58:17.153145Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(2)"}2391{"timestamp":"2026-10-09T09:58:17.549796Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(2)"}2392{"timestamp":"2026-10-09T09:58:17.655232Z","level":"ERROR","message":"Scanner bucket cache load or revision lookup failed","event":"scanner_cache_persist_state","component":"scanner","subsystem":"io","bucket":"bucket29","cache_name":"bucket29/.usage-cache.bin","state":"load_or_revision_lookup_failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::scanner::io","filename":"crates/scanner/src/scanner_io/io_cache.rs","line_number":988,"threadName":"rustfs-worker","threadId":"ThreadId(2)"}2393{"timestamp":"2026-10-09T09:58:17.972877Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}2394=== NAME TestCreatePin_ReservedPins2395 pins_acl_test.go:31: unexpected error: context deadline exceeded2396 2397{"timestamp":"2026-10-09T09:58:18.072599Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}2398--- FAIL: TestCreatePin_ReservedPins (13.07s)2399=== NAME TestSweepSkipsRowsLockedByPush2400 gc_races_test.go:514: unexpected error: Put "http://localhost:60965/bucket93/": context deadline exceeded2401 2402--- FAIL: TestSweepSkipsRowsLockedByPush (13.11s)2403{"timestamp":"2026-10-09T09:58:18.224835Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}2404{"timestamp":"2026-10-09T09:58:18.225872Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}2405{"timestamp":"2026-10-09T09:58:18.297935Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}24062026/10/09 09:58:18 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://localhost:19999/api/pending_closures\": dial tcp 127.0.0.1:19999: connect: connection refused" url=http://localhost:19999/api/pending_closures2407=== NAME TestGetExistingObjectsLocksRows2408 gc_races_test.go:592: unexpected error: context deadline exceeded2409 2410--- FAIL: TestGetExistingObjectsLocksRows (13.20s)2411{"timestamp":"2026-10-09T09:58:18.39833Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}2412{"timestamp":"2026-10-09T09:58:18.447294Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}24132026/10/09 09:58:18 WARN Request failed, retrying attempt=2 backoff=199.679109ms error="Post \"http://localhost:19999/api/pending_closures\": dial tcp 127.0.0.1:19999: connect: connection refused" url=http://localhost:19999/api/pending_closures2414=== NAME TestRegisterCompletedObjectNeedsPendingRow2415 gc_races_test.go:570: unexpected error: context deadline exceeded2416 2417--- FAIL: TestRegisterCompletedObjectNeedsPendingRow (13.16s)2418{"timestamp":"2026-10-09T09:58:18.492269Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}24192026/10/09 09:58:18 WARN Request failed, retrying attempt=3 backoff=384.31874ms error="Post \"http://localhost:19999/api/pending_closures\": dial tcp 127.0.0.1:19999: connect: connection refused" url=http://localhost:19999/api/pending_closures2420{"timestamp":"2026-10-09T09:58:18.664821Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(10)"}2421{"timestamp":"2026-10-09T09:58:18.784574Z","level":"ERROR","message":"save_config_with_opts: err: Io(Custom { kind: TooManyOpenFiles, error: \"Too many open files (os error 24)\" }), file: scanner/durable-dirty-producer-replay.json","target":"rustfs_ecstore::config::com","filename":"crates/ecstore/src/config/com.rs","line_number":813,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}2422{"timestamp":"2026-10-09T09:58:18.784731Z","level":"ERROR","message":"Heal storage request failed","event":"heal_storage_object_io","component":"heal","subsystem":"storage","operation":"object_exists","bucket":".rustfs.sys","object":"scanner/durable-dirty-producer-replay.json","result":"failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::heal::storage","filename":"crates/heal/src/heal/storage.rs","line_number":1339,"threadName":"rustfs-worker","threadId":"ThreadId(10)"}2423{"timestamp":"2026-10-09T09:58:18.784775Z","level":"ERROR","message":"Heal scheduler task failed","event":"heal_scheduler_state","component":"heal","subsystem":"manager","task_id":"4ff19040-fd82-440c-ac3d-b94a5a334121","heal_type":"object","set":"pool_0_set_0","state":"task_failed","error":"Other error: Io error: Too many open files (os error 24)","target":"rustfs::heal::manager","filename":"crates/heal/src/heal/manager/scheduler.rs","line_number":269,"threadName":"rustfs-worker","threadId":"ThreadId(10)"}2424=== NAME TestLeadIncumbentAnnouncesAtOnce2425 farm_test.go:198: unexpected error: context deadline exceeded2426 2427--- FAIL: TestLeadIncumbentAnnouncesAtOnce (13.06s)2428{"timestamp":"2026-10-09T09:58:18.981138Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}24292026/10/09 09:58:19 WARN Request failed, retrying attempt=4 backoff=745.097326ms error="Post \"http://localhost:19999/api/pending_closures\": dial tcp 127.0.0.1:19999: connect: connection refused" url=http://localhost:19999/api/pending_closures2430=== NAME TestForceGCDuringPushOffersSweptObject/after_presence_check2431 gc_races_test.go:387: unexpected error: context deadline exceeded2432 2433{"timestamp":"2026-10-09T09:58:19.087694Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}2434=== NAME TestForceGCDuringPushOffersSweptObject/before_pending_rows2435 gc_races_test.go:387: unexpected error: context deadline exceeded2436 2437--- FAIL: TestForceGCDuringPushOffersSweptObject (0.01s)2438 --- FAIL: TestForceGCDuringPushOffersSweptObject/after_presence_check (12.85s)2439 --- FAIL: TestForceGCDuringPushOffersSweptObject/before_pending_rows (12.62s)2440{"timestamp":"2026-10-09T09:58:19.181073Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}2441{"timestamp":"2026-10-09T09:58:19.30887Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}2442=== NAME TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline2443 timeout_test.go:74: unexpected error: context deadline exceeded2444 2445--- FAIL: TestPendingClosureWriteTimeout (0.01s)2446 --- PASS: TestPendingClosureWriteTimeout/670k_objects (0.00s)2447 --- PASS: TestPendingClosureWriteTimeout/empty (0.00s)2448 --- PASS: TestPendingClosureWriteTimeout/400_objects (0.00s)2449 --- FAIL: TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline (11.86s)2450{"timestamp":"2026-10-09T09:58:19.362912Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}2451=== NAME TestPendingClosureFailureTracksEveryUpload/refuse_aborts=false2452 multipart_cleanup_test.go:387: unexpected error: context deadline exceeded2453 2454=== NAME TestPendingClosureFailureTracksEveryUpload/refuse_aborts=true2455 multipart_cleanup_test.go:387: unexpected error: context deadline exceeded2456 2457--- FAIL: TestPendingClosureFailureTracksEveryUpload (0.02s)2458 --- FAIL: TestPendingClosureFailureTracksEveryUpload/refuse_aborts=false (11.73s)2459 --- FAIL: TestPendingClosureFailureTracksEveryUpload/refuse_aborts=true (11.36s)2460{"timestamp":"2026-10-09T09:58:19.378525Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}2461{"timestamp":"2026-10-09T09:58:19.44275Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}2462=== NAME TestClientErrorHandling/InvalidStorePath2463 client_errors_test.go:18: unexpected error: context deadline exceeded2464 2465{"timestamp":"2026-10-09T09:58:19.498531Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(2)"}2466=== NAME TestClientErrorHandling/InvalidAuthToken2467 client_errors_test.go:44: unexpected error: context deadline exceeded2468 2469{"timestamp":"2026-10-09T09:58:19.500409Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(2)"}2470=== NAME TestPresentNotReportedWhileGCDeletesClosure/stale2471 present_test.go:122: unexpected error: context deadline exceeded2472 2473{"timestamp":"2026-10-09T09:58:19.553647Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(2)"}2474{"timestamp":"2026-10-09T09:58:19.706857Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(2)"}24752026/10/09 09:58:19 WARN Request failed, retrying attempt=5 backoff=1.545215406s error="Post \"http://localhost:19999/api/pending_closures\": dial tcp 127.0.0.1:19999: connect: connection refused" url=http://localhost:19999/api/pending_closures2476{"timestamp":"2026-10-09T09:58:19.932769Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(2)"}2477{"timestamp":"2026-10-09T09:58:20.040717Z","level":"ERROR","message":"Scanner bucket cache load or revision lookup failed","event":"scanner_cache_persist_state","component":"scanner","subsystem":"io","bucket":"bucket3","cache_name":"bucket3/.usage-cache.bin","state":"load_or_revision_lookup_failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::scanner::io","filename":"crates/scanner/src/scanner_io/io_cache.rs","line_number":988,"threadName":"rustfs-worker","threadId":"ThreadId(2)"}2478{"timestamp":"2026-10-09T09:58:20.534385Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(2)"}2479--- FAIL: TestClientErrorHandling (0.00s)2480 --- FAIL: TestClientErrorHandling/InvalidStorePath (10.77s)2481 --- FAIL: TestClientErrorHandling/InvalidAuthToken (10.70s)2482 --- PASS: TestClientErrorHandling/ServerNotAvailable (12.61s)2483{"timestamp":"2026-10-09T09:58:21.445644Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(2)"}2484{"timestamp":"2026-10-09T09:58:21.772283Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(2)"}2485{"timestamp":"2026-10-09T09:58:21.805388Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(2)"}2486{"timestamp":"2026-10-09T09:58:21.837495Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(2)"}2487{"timestamp":"2026-10-09T09:58:22.159595Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}2488{"timestamp":"2026-10-09T09:58:22.461874Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}2489=== NAME TestPresentNotReportedWhileGCDeletesClosure/fresh2490 present_test.go:122: unexpected error: context deadline exceeded2491 2492--- FAIL: TestPresentNotReportedWhileGCDeletesClosure (0.01s)2493 --- FAIL: TestPresentNotReportedWhileGCDeletesClosure/stale (10.33s)2494 --- FAIL: TestPresentNotReportedWhileGCDeletesClosure/fresh (10.04s)2495{"timestamp":"2026-10-09T09:58:22.559652Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}2496{"timestamp":"2026-10-09T09:58:22.574026Z","level":"ERROR","message":"Scanner bucket cache load or revision lookup failed","event":"scanner_cache_persist_state","component":"scanner","subsystem":"io","bucket":"bucket30","cache_name":"bucket30/.usage-cache.bin","state":"load_or_revision_lookup_failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::scanner::io","filename":"crates/scanner/src/scanner_io/io_cache.rs","line_number":988,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}2497{"timestamp":"2026-10-09T09:58:22.667896Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(2)"}2498{"timestamp":"2026-10-09T09:58:22.915968Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(6)"}2499{"timestamp":"2026-10-09T09:58:22.977968Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(6)"}2500=== NAME TestGCEndsOnShutdown/before_the_run2501 gc_shutdown_test.go:34: unexpected error: context deadline exceeded2502 2503{"timestamp":"2026-10-09T09:58:23.27394Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}2504=== NAME TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark2505 gc_shutdown_test.go:34: unexpected error: context deadline exceeded2506 2507{"timestamp":"2026-10-09T09:58:23.406369Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}2508{"timestamp":"2026-10-09T09:58:23.552169Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}2509=== NAME TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete2510 gc_shutdown_test.go:34: unexpected error: context deadline exceeded2511 2512--- FAIL: TestGCEndsOnShutdown (0.02s)2513 --- FAIL: TestGCEndsOnShutdown/before_the_run (10.10s)2514 --- FAIL: TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark (10.08s)2515 --- FAIL: TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete (10.15s)2516FAIL2517{"timestamp":"2026-10-09T09:58:23.772481Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:61549","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2354,"threadName":"rustfs-worker","threadId":"ThreadId(7)"}25182026-10-09 09:58:23.945 UTC [65329] LOG: received smart shutdown request25192026/10/09 09:58:33 INFO killed process group pgid=6532925202026-10-09 09:58:33.953 UTC [66253] FATAL: terminating connection due to unexpected postmaster exit25212026-10-09 09:58:33.953 UTC [66249] FATAL: terminating connection due to unexpected postmaster exit25222026-10-09 09:58:33.953 UTC [66267] FATAL: terminating connection due to unexpected postmaster exit25232026-10-09 09:58:33.953 UTC [66244] FATAL: terminating connection due to unexpected postmaster exit25242026-10-09 09:58:33.953 UTC [66243] FATAL: terminating connection due to unexpected postmaster exit25252026-10-09 09:58:33.953 UTC [66218] FATAL: terminating connection due to unexpected postmaster exit25262026-10-09 09:58:33.953 UTC [66216] FATAL: terminating connection due to unexpected postmaster exit25272026-10-09 09:58:33.953 UTC [66212] FATAL: terminating connection due to unexpected postmaster exit25282026-10-09 09:58:33.953 UTC [66187] FATAL: terminating connection due to unexpected postmaster exit25292026-10-09 09:58:33.956 UTC [66168] FATAL: terminating connection due to unexpected postmaster exit25302026-10-09 09:58:33.957 UTC [66156] FATAL: terminating connection due to unexpected postmaster exit25312026-10-09 09:58:33.961 UTC [66270] FATAL: terminating connection due to unexpected postmaster exit25322026/10/09 09:58:33 ERROR failed to wait for process error="signal: killed"25332026-10-09 09:58:33.953 UTC [66252] FATAL: terminating connection due to unexpected postmaster exit25342026-10-09 09:58:33.953 UTC [66248] FATAL: terminating connection due to unexpected postmaster exit25352026-10-09 09:58:33.953 UTC [66240] FATAL: terminating connection due to unexpected postmaster exit25362026-10-09 09:58:33.953 UTC [66211] FATAL: terminating connection due to unexpected postmaster exit25372026-10-09 09:58:33.953 UTC [66177] FATAL: terminating connection due to unexpected postmaster exit25382026-10-09 09:58:33.953 UTC [66175] FATAL: terminating connection due to unexpected postmaster exit25392026-10-09 09:58:33.962 UTC [66264] FATAL: terminating connection due to unexpected postmaster exit25402026-10-09 09:58:33.953 UTC [66254] FATAL: terminating connection due to unexpected postmaster exit25412026-10-09 09:58:33.953 UTC [66250] FATAL: terminating connection due to unexpected postmaster exit25422026-10-09 09:58:33.962 UTC [66261] FATAL: terminating connection due to unexpected postmaster exit