this derivation will be built: /nix/store/5g7hv3a4slzymmk7v935r05pzhqwa9fv-niks3-go-unit-tests.drv building '/nix/store/5g7hv3a4slzymmk7v935r05pzhqwa9fv-niks3-go-unit-tests.drv' niks3-go-unit-tests> tribuchet: building on mac02 niks3-go-unit-tests> Running client tests... niks3-go-unit-tests> === RUN TestDoServerRequestAttachesToken niks3-go-unit-tests> === PAUSE TestDoServerRequestAttachesToken niks3-go-unit-tests> === RUN TestUploadBuildLog_FileBodyReplayedOnRetry niks3-go-unit-tests> === PAUSE TestUploadBuildLog_FileBodyReplayedOnRetry niks3-go-unit-tests> === RUN TestRegisterUploadedObjectReusesConnections niks3-go-unit-tests> === PAUSE TestRegisterUploadedObjectReusesConnections niks3-go-unit-tests> === RUN TestRunGarbageCollection_FinishedOnAnotherReplica niks3-go-unit-tests> === PAUSE TestRunGarbageCollection_FinishedOnAnotherReplica niks3-go-unit-tests> === RUN TestRunGarbageCollection_NotFoundAfterLocalRun niks3-go-unit-tests> === PAUSE TestRunGarbageCollection_NotFoundAfterLocalRun niks3-go-unit-tests> === RUN TestCaseHackSuffix niks3-go-unit-tests> === PAUSE TestCaseHackSuffix niks3-go-unit-tests> === RUN TestFilterOversizedClosures niks3-go-unit-tests> === PAUSE TestFilterOversizedClosures niks3-go-unit-tests> === RUN TestPushPathsReportsOnlyUnuploadablePathsOfSkippedClosure niks3-go-unit-tests> === PAUSE TestPushPathsReportsOnlyUnuploadablePathsOfSkippedClosure niks3-go-unit-tests> === RUN TestUploadMultipart_PartsInParallel niks3-go-unit-tests> === PAUSE TestUploadMultipart_PartsInParallel niks3-go-unit-tests> === RUN TestUploadMultipart_ProducerErrorIsNotEOF niks3-go-unit-tests> === PAUSE TestUploadMultipart_ProducerErrorIsNotEOF niks3-go-unit-tests> === RUN TestUploadMultipart_FailedPartBufferNotReused niks3-go-unit-tests> === PAUSE TestUploadMultipart_FailedPartBufferNotReused niks3-go-unit-tests> === RUN TestPartSizeForNAR niks3-go-unit-tests> === PAUSE TestPartSizeForNAR niks3-go-unit-tests> === RUN TestUploadMultipart_SupersededByPeer niks3-go-unit-tests> === PAUSE TestUploadMultipart_SupersededByPeer niks3-go-unit-tests> === RUN TestDumpPathCaseHackMatchesNix niks3-go-unit-tests> --- PASS: TestDumpPathCaseHackMatchesNix (0.08s) niks3-go-unit-tests> === RUN TestDumpPathCaseHackCollision niks3-go-unit-tests> --- PASS: TestDumpPathCaseHackCollision (0.00s) niks3-go-unit-tests> === RUN TestSupersededNARStillUploadsListing niks3-go-unit-tests> === PAUSE TestSupersededNARStillUploadsListing niks3-go-unit-tests> === RUN TestTruncatedNARDumpIsNotCompleted niks3-go-unit-tests> === PAUSE TestTruncatedNARDumpIsNotCompleted niks3-go-unit-tests> === RUN TestDumpPathMatchesNix niks3-go-unit-tests> === PAUSE TestDumpPathMatchesNix niks3-go-unit-tests> === RUN TestDumpPathSingleFile niks3-go-unit-tests> === PAUSE TestDumpPathSingleFile niks3-go-unit-tests> === RUN TestDumpPathWriterError niks3-go-unit-tests> === PAUSE TestDumpPathWriterError niks3-go-unit-tests> === RUN TestDumpPathWriterErrorStopsReading niks3-go-unit-tests> nar_test.go:280: no /proc/self/io: open /proc/self/io: no such file or directory niks3-go-unit-tests> --- SKIP: TestDumpPathWriterErrorStopsReading (0.39s) niks3-go-unit-tests> === RUN TestEncodeNixBase32 niks3-go-unit-tests> === PAUSE TestEncodeNixBase32 niks3-go-unit-tests> === RUN TestEncodeNixBase32WithRealHash niks3-go-unit-tests> === PAUSE TestEncodeNixBase32WithRealHash niks3-go-unit-tests> === RUN TestConvertHashToNix32 niks3-go-unit-tests> === PAUSE TestConvertHashToNix32 niks3-go-unit-tests> === RUN TestGetStorePathHash niks3-go-unit-tests> === PAUSE TestGetStorePathHash niks3-go-unit-tests> === RUN TestPathInfoHashCompatibility niks3-go-unit-tests> === PAUSE TestPathInfoHashCompatibility niks3-go-unit-tests> === RUN TestParsePathInfoJSON niks3-go-unit-tests> === PAUSE TestParsePathInfoJSON niks3-go-unit-tests> === RUN TestParsePathInfoJSONMultiplePaths niks3-go-unit-tests> === PAUSE TestParsePathInfoJSONMultiplePaths niks3-go-unit-tests> === RUN TestPathInfoCACompatibility niks3-go-unit-tests> === PAUSE TestPathInfoCACompatibility niks3-go-unit-tests> === RUN TestUploadPendingObjectsStopsStartingAfterFailure niks3-go-unit-tests> --- PASS: TestUploadPendingObjectsStopsStartingAfterFailure (0.01s) niks3-go-unit-tests> === RUN TestCompletePendingClosure_NotFoundIsSuccessWhenClosurePresent niks3-go-unit-tests> === PAUSE TestCompletePendingClosure_NotFoundIsSuccessWhenClosurePresent niks3-go-unit-tests> === RUN TestCompletePendingClosure_NotFoundWithoutKey niks3-go-unit-tests> === PAUSE TestCompletePendingClosure_NotFoundWithoutKey niks3-go-unit-tests> === RUN TestRateLimiterFeedback niks3-go-unit-tests> === PAUSE TestRateLimiterFeedback niks3-go-unit-tests> === RUN TestRateLimiterFeedback_400DoesNotCountAsSuccess niks3-go-unit-tests> === PAUSE TestRateLimiterFeedback_400DoesNotCountAsSuccess niks3-go-unit-tests> === RUN TestRegisterUploadedObject_BoundedAgainstSilentServer niks3-go-unit-tests> === PAUSE TestRegisterUploadedObject_BoundedAgainstSilentServer niks3-go-unit-tests> === RUN TestResolveStorePath niks3-go-unit-tests> === PAUSE TestResolveStorePath niks3-go-unit-tests> === RUN TestDoWithRetry_BodyReplayedViaGetBody niks3-go-unit-tests> === PAUSE TestDoWithRetry_BodyReplayedViaGetBody niks3-go-unit-tests> === RUN TestDoWithRetry_ClosesBodyWhenNoAttemptIsMade niks3-go-unit-tests> === PAUSE TestDoWithRetry_ClosesBodyWhenNoAttemptIsMade niks3-go-unit-tests> === RUN TestRetryTimeoutOutlastsMaxRetries niks3-go-unit-tests> === PAUSE TestRetryTimeoutOutlastsMaxRetries niks3-go-unit-tests> === RUN TestDoWithRetry_FinalResponseBodyReadable niks3-go-unit-tests> === PAUSE TestDoWithRetry_FinalResponseBodyReadable niks3-go-unit-tests> === RUN TestShellSplit niks3-go-unit-tests> === PAUSE TestShellSplit niks3-go-unit-tests> === RUN TestShellSplitErrors niks3-go-unit-tests> === PAUSE TestShellSplitErrors niks3-go-unit-tests> === RUN TestStreamPushReportsEveryPath niks3-go-unit-tests> === PAUSE TestStreamPushReportsEveryPath niks3-go-unit-tests> === RUN TestStreamPushBatchesUnderLoad niks3-go-unit-tests> === PAUSE TestStreamPushBatchesUnderLoad niks3-go-unit-tests> === RUN TestStreamPushIsolatesFailures niks3-go-unit-tests> === PAUSE TestStreamPushIsolatesFailures niks3-go-unit-tests> === RUN TestStreamPushGivesUpOnDeadServer niks3-go-unit-tests> === PAUSE TestStreamPushGivesUpOnDeadServer niks3-go-unit-tests> === RUN TestStreamPushRequestLine niks3-go-unit-tests> === PAUSE TestStreamPushRequestLine niks3-go-unit-tests> === RUN TestStreamPushReportsSignatures niks3-go-unit-tests> === PAUSE TestStreamPushReportsSignatures niks3-go-unit-tests> === RUN TestStreamPushReportsSkippedPaths niks3-go-unit-tests> === PAUSE TestStreamPushReportsSkippedPaths niks3-go-unit-tests> === RUN TestClientSignaturesByStorePath niks3-go-unit-tests> === PAUSE TestClientSignaturesByStorePath niks3-go-unit-tests> === RUN TestStreamPushStopsWhenCanceled niks3-go-unit-tests> === PAUSE TestStreamPushStopsWhenCanceled niks3-go-unit-tests> === RUN TestStreamPushStopsOnCancel niks3-go-unit-tests> === PAUSE TestStreamPushStopsOnCancel niks3-go-unit-tests> === RUN TestSetClientTLS niks3-go-unit-tests> === PAUSE TestSetClientTLS niks3-go-unit-tests> === RUN TestSetClientTLSDoesNotMutateDefaultTransport niks3-go-unit-tests> === PAUSE TestSetClientTLSDoesNotMutateDefaultTransport niks3-go-unit-tests> === RUN TestSetClientTLSErrors niks3-go-unit-tests> === PAUSE TestSetClientTLSErrors niks3-go-unit-tests> === RUN TestStaticToken niks3-go-unit-tests> === PAUSE TestStaticToken niks3-go-unit-tests> === RUN TestFileTokenReadsAndCaches niks3-go-unit-tests> === PAUSE TestFileTokenReadsAndCaches niks3-go-unit-tests> === RUN TestFileTokenMissing niks3-go-unit-tests> === PAUSE TestFileTokenMissing niks3-go-unit-tests> === RUN TestFileTokenEmpty niks3-go-unit-tests> === PAUSE TestFileTokenEmpty niks3-go-unit-tests> === RUN TestScriptTokenNoExpiryRerunsEveryCall niks3-go-unit-tests> === PAUSE TestScriptTokenNoExpiryRerunsEveryCall niks3-go-unit-tests> === RUN TestScriptTokenCachesUntilRefresh niks3-go-unit-tests> === PAUSE TestScriptTokenCachesUntilRefresh niks3-go-unit-tests> === RUN TestScriptTokenEmptyToken niks3-go-unit-tests> === PAUSE TestScriptTokenEmptyToken niks3-go-unit-tests> === RUN TestScriptTokenBadJSON niks3-go-unit-tests> === PAUSE TestScriptTokenBadJSON niks3-go-unit-tests> === RUN TestScriptTokenScriptFails niks3-go-unit-tests> === PAUSE TestScriptTokenScriptFails niks3-go-unit-tests> === RUN TestScriptTokenEmptyCommand niks3-go-unit-tests> === PAUSE TestScriptTokenEmptyCommand niks3-go-unit-tests> === RUN TestScriptTokenDoesNotWaitForItsChildren niks3-go-unit-tests> === PAUSE TestScriptTokenDoesNotWaitForItsChildren niks3-go-unit-tests> === CONT TestRegisterUploadedObjectReusesConnections niks3-go-unit-tests> === CONT TestScriptTokenEmptyCommand niks3-go-unit-tests> === CONT TestStreamPushReportsSignatures niks3-go-unit-tests> === CONT TestResolveStorePath niks3-go-unit-tests> === CONT TestScriptTokenScriptFails niks3-go-unit-tests> === CONT TestScriptTokenBadJSON niks3-go-unit-tests> === CONT TestScriptTokenEmptyToken niks3-go-unit-tests> === CONT TestScriptTokenCachesUntilRefresh niks3-go-unit-tests> === CONT TestFileTokenEmpty niks3-go-unit-tests> === CONT TestFileTokenMissing niks3-go-unit-tests> === CONT TestCompletePendingClosure_NotFoundIsSuccessWhenClosurePresent niks3-go-unit-tests> === RUN TestCompletePendingClosure_NotFoundIsSuccessWhenClosurePresent/committed_earlier niks3-go-unit-tests> === CONT TestPathInfoCACompatibility niks3-go-unit-tests> === RUN TestPathInfoCACompatibility/null_ca_field niks3-go-unit-tests> === CONT TestStaticToken niks3-go-unit-tests> === CONT TestSetClientTLSErrors niks3-go-unit-tests> === CONT TestScriptTokenNoExpiryRerunsEveryCall niks3-go-unit-tests> === CONT TestGetStorePathHash niks3-go-unit-tests> === CONT TestSetClientTLS niks3-go-unit-tests> === CONT TestSetClientTLSDoesNotMutateDefaultTransport niks3-go-unit-tests> === CONT TestStreamPushStopsWhenCanceled niks3-go-unit-tests> === CONT TestClientSignaturesByStorePath niks3-go-unit-tests> === PAUSE TestCompletePendingClosure_NotFoundIsSuccessWhenClosurePresent/committed_earlier niks3-go-unit-tests> === CONT TestStreamPushReportsSkippedPaths niks3-go-unit-tests> === CONT TestDumpPathMatchesNix niks3-go-unit-tests> === CONT TestRateLimiterFeedback_400DoesNotCountAsSuccess niks3-go-unit-tests> === CONT TestRegisterUploadedObject_BoundedAgainstSilentServer niks3-go-unit-tests> === CONT TestRateLimiterFeedback niks3-go-unit-tests> === CONT TestScriptTokenDoesNotWaitForItsChildren niks3-go-unit-tests> --- PASS: TestScriptTokenEmptyCommand (0.00s) niks3-go-unit-tests> === CONT TestCompletePendingClosure_NotFoundWithoutKey niks3-go-unit-tests> === CONT TestFileTokenReadsAndCaches niks3-go-unit-tests> === CONT TestPathInfoHashCompatibility niks3-go-unit-tests> === CONT TestParsePathInfoJSONMultiplePaths niks3-go-unit-tests> === PAUSE TestPathInfoCACompatibility/null_ca_field niks3-go-unit-tests> === CONT TestParsePathInfoJSON niks3-go-unit-tests> === RUN TestGetStorePathHash/valid_store_path niks3-go-unit-tests> === CONT TestConvertHashToNix32 niks3-go-unit-tests> === CONT TestEncodeNixBase32WithRealHash niks3-go-unit-tests> === RUN TestCompletePendingClosure_NotFoundIsSuccessWhenClosurePresent/cleaned_up niks3-go-unit-tests> === RUN TestSetClientTLSErrors/missing_cert_file niks3-go-unit-tests> === RUN TestRateLimiterFeedback/429_enables_limiter niks3-go-unit-tests> --- PASS: TestStreamPushReportsSignatures (0.00s) niks3-go-unit-tests> === RUN TestPathInfoCACompatibility/old_string_format_-_text niks3-go-unit-tests> --- PASS: TestResolveStorePath (0.00s) niks3-go-unit-tests> === RUN TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths niks3-go-unit-tests> === CONT TestStreamPushGivesUpOnDeadServer niks3-go-unit-tests> === CONT TestEncodeNixBase32 niks3-go-unit-tests> === RUN TestParsePathInfoJSON/Nix_format niks3-go-unit-tests> === RUN TestScriptTokenDoesNotWaitForItsChildren/cancelled_while_a_child_runs niks3-go-unit-tests> === CONT TestDumpPathWriterError niks3-go-unit-tests> === PAUSE TestRateLimiterFeedback/429_enables_limiter niks3-go-unit-tests> === PAUSE TestPathInfoCACompatibility/old_string_format_-_text niks3-go-unit-tests> === CONT TestShellSplitErrors niks3-go-unit-tests> === CONT TestStreamPushStopsOnCancel niks3-go-unit-tests> === CONT TestStreamPushRequestLine niks3-go-unit-tests> --- PASS: TestScriptTokenScriptFails (0.00s) niks3-go-unit-tests> === PAUSE TestSetClientTLSErrors/missing_cert_file niks3-go-unit-tests> === PAUSE TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths niks3-go-unit-tests> === PAUSE TestParsePathInfoJSON/Nix_format niks3-go-unit-tests> === PAUSE TestScriptTokenDoesNotWaitForItsChildren/cancelled_while_a_child_runs niks3-go-unit-tests> === RUN TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI) niks3-go-unit-tests> === RUN TestSetClientTLS/rejects_connection_without_client_cert niks3-go-unit-tests> === RUN TestRateLimiterFeedback/503_enables_limiter niks3-go-unit-tests> === RUN TestPathInfoCACompatibility/old_string_format_-_fixed_recursive niks3-go-unit-tests> === RUN TestSetClientTLSErrors/missing_key_file niks3-go-unit-tests> --- PASS: TestStaticToken (0.00s) niks3-go-unit-tests> === RUN TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths niks3-go-unit-tests> === RUN TestParsePathInfoJSON/Lix_format niks3-go-unit-tests> === RUN TestScriptTokenDoesNotWaitForItsChildren/child_left_holding_stdout niks3-go-unit-tests> === RUN TestConvertHashToNix32/SRI_format_to_Nix32 niks3-go-unit-tests> === CONT TestStreamPushIsolatesFailures niks3-go-unit-tests> === PAUSE TestCompletePendingClosure_NotFoundIsSuccessWhenClosurePresent/cleaned_up niks3-go-unit-tests> === PAUSE TestGetStorePathHash/valid_store_path niks3-go-unit-tests> --- PASS: TestFileTokenMissing (0.00s) niks3-go-unit-tests> === PAUSE TestRateLimiterFeedback/503_enables_limiter niks3-go-unit-tests> === PAUSE TestSetClientTLSErrors/missing_key_file niks3-go-unit-tests> --- PASS: TestClientSignaturesByStorePath (0.00s) niks3-go-unit-tests> === PAUSE TestSetClientTLS/rejects_connection_without_client_cert niks3-go-unit-tests> --- PASS: TestStreamPushStopsWhenCanceled (0.00s) niks3-go-unit-tests> === CONT TestStreamPushBatchesUnderLoad niks3-go-unit-tests> --- PASS: TestFileTokenEmpty (0.00s) niks3-go-unit-tests> --- PASS: TestSetClientTLSDoesNotMutateDefaultTransport (0.01s) niks3-go-unit-tests> --- PASS: TestFileTokenReadsAndCaches (0.00s) niks3-go-unit-tests> === PAUSE TestScriptTokenDoesNotWaitForItsChildren/child_left_holding_stdout niks3-go-unit-tests> === PAUSE TestConvertHashToNix32/SRI_format_to_Nix32 niks3-go-unit-tests> === PAUSE TestPathInfoCACompatibility/old_string_format_-_fixed_recursive niks3-go-unit-tests> === CONT TestPushPathsReportsOnlyUnuploadablePathsOfSkippedClosure niks3-go-unit-tests> --- PASS: TestStreamPushReportsSkippedPaths (0.01s) niks3-go-unit-tests> === CONT TestTruncatedNARDumpIsNotCompleted niks3-go-unit-tests> === PAUSE TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI) niks3-go-unit-tests> === PAUSE TestParsePathInfoJSON/Lix_format niks3-go-unit-tests> --- PASS: TestCompletePendingClosure_NotFoundWithoutKey (0.01s) niks3-go-unit-tests> === RUN TestEncodeNixBase32/test_string_hash niks3-go-unit-tests> --- PASS: TestScriptTokenEmptyToken (0.02s) niks3-go-unit-tests> === RUN TestParsePathInfoJSON/empty_input niks3-go-unit-tests> === PAUSE TestParsePathInfoJSON/empty_input niks3-go-unit-tests> === RUN TestParsePathInfoJSON/whitespace_only niks3-go-unit-tests> === RUN TestPathInfoHashCompatibility/old_string_format_with_colon niks3-go-unit-tests> --- PASS: TestScriptTokenBadJSON (0.02s) niks3-go-unit-tests> --- PASS: TestEncodeNixBase32WithRealHash (0.00s) niks3-go-unit-tests> --- PASS: TestStreamPushIsolatesFailures (0.00s) niks3-go-unit-tests> === PAUSE TestPathInfoHashCompatibility/old_string_format_with_colon niks3-go-unit-tests> === RUN TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI niks3-go-unit-tests> === PAUSE TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI niks3-go-unit-tests> === RUN TestPathInfoHashCompatibility/new_structured_format_with_sha512 niks3-go-unit-tests> === PAUSE TestParsePathInfoJSON/whitespace_only niks3-go-unit-tests> === RUN TestParsePathInfoJSON/invalid_JSON niks3-go-unit-tests> === PAUSE TestParsePathInfoJSON/invalid_JSON niks3-go-unit-tests> === PAUSE TestPathInfoHashCompatibility/new_structured_format_with_sha512 niks3-go-unit-tests> === PAUSE TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths niks3-go-unit-tests> === RUN TestGetStorePathHash/basename_without_hyphen_should_error niks3-go-unit-tests> === RUN TestStreamPushStopsOnCancel/waiting_for_input niks3-go-unit-tests> === CONT TestSupersededNARStillUploadsListing niks3-go-unit-tests> === RUN TestRateLimiterFeedback/200_does_not_enable_limiter niks3-go-unit-tests> === RUN TestSupersededNARStillUploadsListing/small_NAR niks3-go-unit-tests> === PAUSE TestRateLimiterFeedback/200_does_not_enable_limiter niks3-go-unit-tests> === CONT TestUploadMultipart_ProducerErrorIsNotEOF niks3-go-unit-tests> === RUN TestSetClientTLS/succeeds_with_client_cert_and_CA niks3-go-unit-tests> === PAUSE TestSetClientTLS/succeeds_with_client_cert_and_CA niks3-go-unit-tests> === RUN TestConvertHashToNix32/already_Nix32_format niks3-go-unit-tests> === PAUSE TestConvertHashToNix32/already_Nix32_format niks3-go-unit-tests> === RUN TestPathInfoCACompatibility/new_structured_format_-_text niks3-go-unit-tests> === PAUSE TestPathInfoCACompatibility/new_structured_format_-_text niks3-go-unit-tests> === CONT TestDumpPathSingleFile niks3-go-unit-tests> === RUN TestCompletePendingClosure_NotFoundIsSuccessWhenClosurePresent/push_committed_earlier niks3-go-unit-tests> === PAUSE TestEncodeNixBase32/test_string_hash niks3-go-unit-tests> === PAUSE TestCompletePendingClosure_NotFoundIsSuccessWhenClosurePresent/push_committed_earlier niks3-go-unit-tests> === RUN TestSetClientTLSErrors/missing_ca_file niks3-go-unit-tests> === PAUSE TestSetClientTLSErrors/missing_ca_file niks3-go-unit-tests> === CONT TestUploadMultipart_FailedPartBufferNotReused niks3-go-unit-tests> --- PASS: TestShellSplitErrors (0.00s) niks3-go-unit-tests> === CONT TestUploadMultipart_SupersededByPeer niks3-go-unit-tests> === CONT TestPartSizeForNAR niks3-go-unit-tests> === PAUSE TestGetStorePathHash/basename_without_hyphen_should_error niks3-go-unit-tests> === PAUSE TestStreamPushStopsOnCancel/waiting_for_input niks3-go-unit-tests> === RUN TestStreamPushBatchesUnderLoad/together niks3-go-unit-tests> === PAUSE TestSupersededNARStillUploadsListing/small_NAR niks3-go-unit-tests> === RUN TestUploadMultipart_ProducerErrorIsNotEOF/short_read_inside_a_part niks3-go-unit-tests> === RUN TestStreamPushStopsOnCancel/partial_batch_waiting_for_a_slot niks3-go-unit-tests> === PAUSE TestStreamPushBatchesUnderLoad/together niks3-go-unit-tests> === PAUSE TestStreamPushStopsOnCancel/partial_batch_waiting_for_a_slot niks3-go-unit-tests> === RUN TestRateLimiterFeedback/400_does_not_enable_limiter niks3-go-unit-tests> === PAUSE TestRateLimiterFeedback/400_does_not_enable_limiter niks3-go-unit-tests> --- PASS: TestStreamPushGivesUpOnDeadServer (0.01s) niks3-go-unit-tests> === RUN TestSetClientTLS/preserves_debug_logging_transport niks3-go-unit-tests> === PAUSE TestSetClientTLS/preserves_debug_logging_transport niks3-go-unit-tests> === PAUSE TestUploadMultipart_ProducerErrorIsNotEOF/short_read_inside_a_part niks3-go-unit-tests> === RUN TestConvertHashToNix32/invalid_format niks3-go-unit-tests> === RUN TestEncodeNixBase32/empty_input niks3-go-unit-tests> === RUN TestUploadMultipart_ProducerErrorIsNotEOF/empty_read_on_a_part_boundary niks3-go-unit-tests> === RUN TestCompletePendingClosure_NotFoundIsSuccessWhenClosurePresent/push_cleaned_up,_first_root_committed_by_another_client niks3-go-unit-tests> === RUN TestSetClientTLSErrors/invalid_ca_file niks3-go-unit-tests> === RUN TestUploadMultipart_SupersededByPeer/exists niks3-go-unit-tests> === RUN TestPartSizeForNAR/zero_stays_at_minimum niks3-go-unit-tests> === PAUSE TestPartSizeForNAR/zero_stays_at_minimum niks3-go-unit-tests> === RUN TestGetStorePathHash/hash_with_invalid_characters_should_error niks3-go-unit-tests> === PAUSE TestGetStorePathHash/hash_with_invalid_characters_should_error niks3-go-unit-tests> === PAUSE TestUploadMultipart_ProducerErrorIsNotEOF/empty_read_on_a_part_boundary niks3-go-unit-tests> === RUN TestGetStorePathHash/hash_with_wrong_length_should_error niks3-go-unit-tests> === CONT TestCaseHackSuffix niks3-go-unit-tests> === PAUSE TestConvertHashToNix32/invalid_format niks3-go-unit-tests> === RUN TestStreamPushBatchesUnderLoad/one_at_a_time niks3-go-unit-tests> === RUN TestPathInfoCACompatibility/new_structured_format_-_nar_method niks3-go-unit-tests> === PAUSE TestPathInfoCACompatibility/new_structured_format_-_nar_method niks3-go-unit-tests> --- PASS: TestScriptTokenNoExpiryRerunsEveryCall (0.03s) niks3-go-unit-tests> === CONT TestRetryTimeoutOutlastsMaxRetries niks3-go-unit-tests> === RUN TestStreamPushStopsOnCancel/input_closed,_last_batch_waiting_for_a_slot niks3-go-unit-tests> --- PASS: TestScriptTokenCachesUntilRefresh (0.04s) niks3-go-unit-tests> === PAUSE TestStreamPushStopsOnCancel/input_closed,_last_batch_waiting_for_a_slot niks3-go-unit-tests> === RUN TestSupersededNARStillUploadsListing/dump_cut_short niks3-go-unit-tests> === CONT TestFilterOversizedClosures niks3-go-unit-tests> === PAUSE TestEncodeNixBase32/empty_input niks3-go-unit-tests> === PAUSE TestSupersededNARStillUploadsListing/dump_cut_short niks3-go-unit-tests> === CONT TestRunGarbageCollection_FinishedOnAnotherReplica niks3-go-unit-tests> === PAUSE TestCompletePendingClosure_NotFoundIsSuccessWhenClosurePresent/push_cleaned_up,_first_root_committed_by_another_client niks3-go-unit-tests> === PAUSE TestSetClientTLSErrors/invalid_ca_file niks3-go-unit-tests> === RUN TestPartSizeForNAR/small_stays_at_minimum niks3-go-unit-tests> === RUN TestSupersededNARStillUploadsListing/listing_upload_fails niks3-go-unit-tests> === PAUSE TestPartSizeForNAR/small_stays_at_minimum niks3-go-unit-tests> === CONT TestUploadBuildLog_FileBodyReplayedOnRetry niks3-go-unit-tests> === RUN TestPartSizeForNAR/80_GiB_fits_at_minimum niks3-go-unit-tests> === PAUSE TestStreamPushBatchesUnderLoad/one_at_a_time niks3-go-unit-tests> === PAUSE TestGetStorePathHash/hash_with_wrong_length_should_error niks3-go-unit-tests> === PAUSE TestUploadMultipart_SupersededByPeer/exists niks3-go-unit-tests> === RUN TestUploadMultipart_SupersededByPeer/missing niks3-go-unit-tests> === PAUSE TestUploadMultipart_SupersededByPeer/missing niks3-go-unit-tests> === RUN TestStreamPushStopsOnCancel/full_batch_waiting_for_a_slot niks3-go-unit-tests> === RUN TestFilterOversizedClosures/no_limit_keeps_everything niks3-go-unit-tests> === CONT TestShellSplit niks3-go-unit-tests> === PAUSE TestFilterOversizedClosures/no_limit_keeps_everything niks3-go-unit-tests> === RUN TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped niks3-go-unit-tests> === CONT TestDoWithRetry_FinalResponseBodyReadable niks3-go-unit-tests> --- PASS: TestShellSplit (0.00s) niks3-go-unit-tests> === RUN TestCompletePendingClosure_NotFoundIsSuccessWhenClosurePresent/push_cleaned_up,_last_root_committed_by_another_client niks3-go-unit-tests> === PAUSE TestStreamPushStopsOnCancel/full_batch_waiting_for_a_slot niks3-go-unit-tests> === RUN TestStreamPushStopsOnCancel/request_line_waiting_for_a_slot niks3-go-unit-tests> === PAUSE TestStreamPushStopsOnCancel/request_line_waiting_for_a_slot niks3-go-unit-tests> === PAUSE TestCompletePendingClosure_NotFoundIsSuccessWhenClosurePresent/push_cleaned_up,_last_root_committed_by_another_client niks3-go-unit-tests> === PAUSE TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped niks3-go-unit-tests> === PAUSE TestSupersededNARStillUploadsListing/listing_upload_fails niks3-go-unit-tests> === CONT TestStreamPushReportsEveryPath niks3-go-unit-tests> === CONT TestRunGarbageCollection_NotFoundAfterLocalRun niks3-go-unit-tests> === PAUSE TestPartSizeForNAR/80_GiB_fits_at_minimum niks3-go-unit-tests> === CONT TestDoWithRetry_ClosesBodyWhenNoAttemptIsMade niks3-go-unit-tests> === CONT TestDoWithRetry_BodyReplayedViaGetBody niks3-go-unit-tests> === CONT TestUploadMultipart_PartsInParallel niks3-go-unit-tests> --- PASS: TestStreamPushReportsEveryPath (0.00s) niks3-go-unit-tests> === RUN TestStreamPushStopsOnCancel/lines_read_but_not_taken niks3-go-unit-tests> === PAUSE TestStreamPushStopsOnCancel/lines_read_but_not_taken niks3-go-unit-tests> === CONT TestDoServerRequestAttachesToken niks3-go-unit-tests> === RUN TestPartSizeForNAR/115_GiB_needs_larger_parts niks3-go-unit-tests> === RUN TestDoWithRetry_ClosesBodyWhenNoAttemptIsMade/retries=0 niks3-go-unit-tests> === RUN TestFilterOversizedClosures/all_closures_skipped niks3-go-unit-tests> === CONT TestScriptTokenDoesNotWaitForItsChildren/child_left_holding_stdout niks3-go-unit-tests> --- PASS: TestDoWithRetry_FinalResponseBodyReadable (0.01s) niks3-go-unit-tests> === CONT TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI niks3-go-unit-tests> === PAUSE TestFilterOversizedClosures/all_closures_skipped niks3-go-unit-tests> === CONT TestPathInfoHashCompatibility/new_structured_format_with_sha512 niks3-go-unit-tests> === CONT TestPathInfoHashCompatibility/old_string_format_with_colon niks3-go-unit-tests> === CONT TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths niks3-go-unit-tests> --- PASS: TestRunGarbageCollection_NotFoundAfterLocalRun (0.01s) niks3-go-unit-tests> === CONT TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths niks3-go-unit-tests> === CONT TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI) niks3-go-unit-tests> === CONT TestScriptTokenDoesNotWaitForItsChildren/cancelled_while_a_child_runs niks3-go-unit-tests> === PAUSE TestPartSizeForNAR/115_GiB_needs_larger_parts niks3-go-unit-tests> === RUN TestPartSizeForNAR/1_TiB niks3-go-unit-tests> === PAUSE TestPartSizeForNAR/1_TiB niks3-go-unit-tests> === RUN TestPartSizeForNAR/5_TiB_S3_max_object niks3-go-unit-tests> === PAUSE TestPartSizeForNAR/5_TiB_S3_max_object niks3-go-unit-tests> === RUN TestPartSizeForNAR/capped_at_5_GiB niks3-go-unit-tests> === PAUSE TestPartSizeForNAR/capped_at_5_GiB niks3-go-unit-tests> === PAUSE TestDoWithRetry_ClosesBodyWhenNoAttemptIsMade/retries=0 niks3-go-unit-tests> === RUN TestDoWithRetry_ClosesBodyWhenNoAttemptIsMade/retries=5 niks3-go-unit-tests> === PAUSE TestDoWithRetry_ClosesBodyWhenNoAttemptIsMade/retries=5 niks3-go-unit-tests> === CONT TestParsePathInfoJSON/Nix_format niks3-go-unit-tests> === CONT TestParsePathInfoJSON/whitespace_only niks3-go-unit-tests> --- PASS: TestDoServerRequestAttachesToken (0.02s) niks3-go-unit-tests> === CONT TestParsePathInfoJSON/invalid_JSON niks3-go-unit-tests> === CONT TestParsePathInfoJSON/empty_input niks3-go-unit-tests> === CONT TestParsePathInfoJSON/Lix_format niks3-go-unit-tests> === CONT TestRateLimiterFeedback/429_enables_limiter niks3-go-unit-tests> --- PASS: TestUploadBuildLog_FileBodyReplayedOnRetry (0.02s) niks3-go-unit-tests> === CONT TestSetClientTLS/rejects_connection_without_client_cert niks3-go-unit-tests> === CONT TestRateLimiterFeedback/200_does_not_enable_limiter niks3-go-unit-tests> === CONT TestRateLimiterFeedback/400_does_not_enable_limiter niks3-go-unit-tests> --- PASS: TestParsePathInfoJSON (0.02s) niks3-go-unit-tests> --- PASS: TestParsePathInfoJSON/whitespace_only (0.00s) niks3-go-unit-tests> --- PASS: TestParsePathInfoJSON/invalid_JSON (0.00s) niks3-go-unit-tests> --- PASS: TestParsePathInfoJSON/Nix_format (0.00s) niks3-go-unit-tests> --- PASS: TestParsePathInfoJSON/empty_input (0.00s) niks3-go-unit-tests> --- PASS: TestParsePathInfoJSON/Lix_format (0.00s) niks3-go-unit-tests> === CONT TestRateLimiterFeedback/503_enables_limiter niks3-go-unit-tests> === CONT TestSetClientTLS/preserves_debug_logging_transport niks3-go-unit-tests> === CONT TestSetClientTLS/succeeds_with_client_cert_and_CA niks3-go-unit-tests> === CONT TestUploadMultipart_ProducerErrorIsNotEOF/empty_read_on_a_part_boundary niks3-go-unit-tests> --- PASS: TestDoWithRetry_BodyReplayedViaGetBody (0.01s) niks3-go-unit-tests> --- PASS: TestRunGarbageCollection_FinishedOnAnotherReplica (0.01s) niks3-go-unit-tests> --- PASS: TestPathInfoHashCompatibility (0.02s) niks3-go-unit-tests> --- PASS: TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI (0.00s) niks3-go-unit-tests> --- PASS: TestPathInfoHashCompatibility/new_structured_format_with_sha512 (0.00s) niks3-go-unit-tests> --- PASS: TestPathInfoHashCompatibility/old_string_format_with_colon (0.00s) niks3-go-unit-tests> --- PASS: TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI) (0.00s) niks3-go-unit-tests> === CONT TestUploadMultipart_ProducerErrorIsNotEOF/short_read_inside_a_part niks3-go-unit-tests> --- PASS: TestParsePathInfoJSONMultiplePaths (0.02s) niks3-go-unit-tests> --- PASS: TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths (0.00s) niks3-go-unit-tests> --- PASS: TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths (0.00s) niks3-go-unit-tests> === CONT TestConvertHashToNix32/already_Nix32_format niks3-go-unit-tests> === CONT TestConvertHashToNix32/invalid_format niks3-go-unit-tests> === CONT TestConvertHashToNix32/SRI_format_to_Nix32 niks3-go-unit-tests> === CONT TestPathInfoCACompatibility/null_ca_field niks3-go-unit-tests> === CONT TestPathInfoCACompatibility/new_structured_format_-_text niks3-go-unit-tests> === CONT TestPathInfoCACompatibility/new_structured_format_-_nar_method niks3-go-unit-tests> === CONT TestPathInfoCACompatibility/old_string_format_-_text niks3-go-unit-tests> === CONT TestPathInfoCACompatibility/old_string_format_-_fixed_recursive niks3-go-unit-tests> === CONT TestEncodeNixBase32/test_string_hash niks3-go-unit-tests> --- PASS: TestPathInfoCACompatibility (0.04s) niks3-go-unit-tests> --- PASS: TestPathInfoCACompatibility/null_ca_field (0.00s) niks3-go-unit-tests> --- PASS: TestPathInfoCACompatibility/new_structured_format_-_text (0.00s) niks3-go-unit-tests> --- PASS: TestPathInfoCACompatibility/new_structured_format_-_nar_method (0.00s) niks3-go-unit-tests> --- PASS: TestPathInfoCACompatibility/old_string_format_-_text (0.00s) niks3-go-unit-tests> --- PASS: TestPathInfoCACompatibility/old_string_format_-_fixed_recursive (0.00s) niks3-go-unit-tests> === CONT TestEncodeNixBase32/empty_input niks3-go-unit-tests> --- PASS: TestEncodeNixBase32 (0.02s) niks3-go-unit-tests> --- PASS: TestEncodeNixBase32/test_string_hash (0.00s) niks3-go-unit-tests> --- PASS: TestEncodeNixBase32/empty_input (0.00s) niks3-go-unit-tests> === CONT TestSetClientTLSErrors/missing_key_file niks3-go-unit-tests> === CONT TestSetClientTLSErrors/invalid_ca_file niks3-go-unit-tests> === CONT TestSetClientTLSErrors/missing_ca_file niks3-go-unit-tests> === CONT TestSetClientTLSErrors/missing_cert_file niks3-go-unit-tests> === CONT TestGetStorePathHash/valid_store_path niks3-go-unit-tests> === CONT TestStreamPushBatchesUnderLoad/one_at_a_time niks3-go-unit-tests> --- PASS: TestConvertHashToNix32 (0.02s) niks3-go-unit-tests> --- PASS: TestConvertHashToNix32/already_Nix32_format (0.00s) niks3-go-unit-tests> --- PASS: TestConvertHashToNix32/invalid_format (0.00s) niks3-go-unit-tests> --- PASS: TestConvertHashToNix32/SRI_format_to_Nix32 (0.01s) niks3-go-unit-tests> === CONT TestGetStorePathHash/hash_with_wrong_length_should_error niks3-go-unit-tests> === CONT TestGetStorePathHash/hash_with_invalid_characters_should_error niks3-go-unit-tests> === CONT TestUploadMultipart_SupersededByPeer/missing niks3-go-unit-tests> --- PASS: TestSetClientTLSErrors (0.04s) niks3-go-unit-tests> --- PASS: TestSetClientTLSErrors/missing_key_file (0.00s) niks3-go-unit-tests> --- PASS: TestSetClientTLSErrors/invalid_ca_file (0.00s) niks3-go-unit-tests> --- PASS: TestSetClientTLSErrors/missing_ca_file (0.00s) niks3-go-unit-tests> --- PASS: TestSetClientTLSErrors/missing_cert_file (0.00s) niks3-go-unit-tests> === CONT TestUploadMultipart_SupersededByPeer/exists niks3-go-unit-tests> === CONT TestGetStorePathHash/basename_without_hyphen_should_error niks3-go-unit-tests> --- PASS: TestRateLimiterFeedback (0.03s) niks3-go-unit-tests> --- PASS: TestRateLimiterFeedback/400_does_not_enable_limiter (0.01s) niks3-go-unit-tests> --- PASS: TestRateLimiterFeedback/200_does_not_enable_limiter (0.01s) niks3-go-unit-tests> --- PASS: TestRateLimiterFeedback/503_enables_limiter (0.01s) niks3-go-unit-tests> --- PASS: TestRateLimiterFeedback/429_enables_limiter (0.03s) niks3-go-unit-tests> === CONT TestStreamPushBatchesUnderLoad/together niks3-go-unit-tests> --- PASS: TestGetStorePathHash (0.04s) niks3-go-unit-tests> --- PASS: TestGetStorePathHash/valid_store_path (0.00s) niks3-go-unit-tests> --- PASS: TestGetStorePathHash/hash_with_wrong_length_should_error (0.00s) niks3-go-unit-tests> --- PASS: TestGetStorePathHash/hash_with_invalid_characters_should_error (0.00s) niks3-go-unit-tests> --- PASS: TestGetStorePathHash/basename_without_hyphen_should_error (0.00s) niks3-go-unit-tests> === CONT TestCompletePendingClosure_NotFoundIsSuccessWhenClosurePresent/push_cleaned_up,_first_root_committed_by_another_client niks3-go-unit-tests> === CONT TestCompletePendingClosure_NotFoundIsSuccessWhenClosurePresent/push_cleaned_up,_last_root_committed_by_another_client niks3-go-unit-tests> === CONT TestCompletePendingClosure_NotFoundIsSuccessWhenClosurePresent/push_committed_earlier niks3-go-unit-tests> === CONT TestCompletePendingClosure_NotFoundIsSuccessWhenClosurePresent/cleaned_up niks3-go-unit-tests> --- PASS: TestStreamPushBatchesUnderLoad (0.02s) niks3-go-unit-tests> --- PASS: TestStreamPushBatchesUnderLoad/one_at_a_time (0.01s) niks3-go-unit-tests> --- PASS: TestStreamPushBatchesUnderLoad/together (0.01s) niks3-go-unit-tests> === CONT TestCompletePendingClosure_NotFoundIsSuccessWhenClosurePresent/committed_earlier niks3-go-unit-tests> === CONT TestSupersededNARStillUploadsListing/listing_upload_fails niks3-go-unit-tests> --- PASS: TestUploadMultipart_SupersededByPeer (0.00s) niks3-go-unit-tests> --- PASS: TestUploadMultipart_SupersededByPeer/exists (0.01s) niks3-go-unit-tests> --- PASS: TestUploadMultipart_SupersededByPeer/missing (0.01s) niks3-go-unit-tests> --- PASS: TestDumpPathSingleFile (0.09s) niks3-go-unit-tests> === CONT TestSupersededNARStillUploadsListing/dump_cut_short niks3-go-unit-tests> === CONT TestSupersededNARStillUploadsListing/small_NAR niks3-go-unit-tests> === CONT TestStreamPushStopsOnCancel/waiting_for_input niks3-go-unit-tests> === CONT TestStreamPushStopsOnCancel/lines_read_but_not_taken niks3-go-unit-tests> === CONT TestStreamPushStopsOnCancel/full_batch_waiting_for_a_slot niks3-go-unit-tests> === CONT TestStreamPushStopsOnCancel/input_closed,_last_batch_waiting_for_a_slot niks3-go-unit-tests> === CONT TestStreamPushStopsOnCancel/request_line_waiting_for_a_slot niks3-go-unit-tests> === CONT TestFilterOversizedClosures/all_closures_skipped niks3-go-unit-tests> === CONT TestFilterOversizedClosures/no_limit_keeps_everything niks3-go-unit-tests> --- PASS: TestCompletePendingClosure_NotFoundIsSuccessWhenClosurePresent (0.04s) niks3-go-unit-tests> --- PASS: TestCompletePendingClosure_NotFoundIsSuccessWhenClosurePresent/push_cleaned_up,_last_root_committed_by_another_client (0.02s) niks3-go-unit-tests> --- PASS: TestCompletePendingClosure_NotFoundIsSuccessWhenClosurePresent/cleaned_up (0.02s) niks3-go-unit-tests> --- PASS: TestCompletePendingClosure_NotFoundIsSuccessWhenClosurePresent/push_cleaned_up,_first_root_committed_by_another_client (0.03s) niks3-go-unit-tests> --- PASS: TestCompletePendingClosure_NotFoundIsSuccessWhenClosurePresent/push_committed_earlier (0.04s) niks3-go-unit-tests> --- PASS: TestCompletePendingClosure_NotFoundIsSuccessWhenClosurePresent/committed_earlier (0.04s) niks3-go-unit-tests> === CONT TestDoWithRetry_ClosesBodyWhenNoAttemptIsMade/retries=0 niks3-go-unit-tests> --- PASS: TestUploadMultipart_ProducerErrorIsNotEOF (0.01s) niks3-go-unit-tests> --- PASS: TestUploadMultipart_ProducerErrorIsNotEOF/empty_read_on_a_part_boundary (0.06s) niks3-go-unit-tests> --- PASS: TestUploadMultipart_ProducerErrorIsNotEOF/short_read_inside_a_part (0.08s) niks3-go-unit-tests> --- PASS: TestStreamPushRequestLine (0.14s) niks3-go-unit-tests> --- PASS: TestCaseHackSuffix (0.12s) niks3-go-unit-tests> === CONT TestStreamPushStopsOnCancel/partial_batch_waiting_for_a_slot niks3-go-unit-tests> === CONT TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped niks3-go-unit-tests> === CONT TestDoWithRetry_ClosesBodyWhenNoAttemptIsMade/retries=5 niks3-go-unit-tests> === CONT TestPartSizeForNAR/5_TiB_S3_max_object niks3-go-unit-tests> === CONT TestPartSizeForNAR/zero_stays_at_minimum niks3-go-unit-tests> === CONT TestPartSizeForNAR/1_TiB niks3-go-unit-tests> === CONT TestPartSizeForNAR/capped_at_5_GiB niks3-go-unit-tests> === CONT TestPartSizeForNAR/115_GiB_needs_larger_parts niks3-go-unit-tests> === CONT TestPartSizeForNAR/small_stays_at_minimum niks3-go-unit-tests> --- PASS: TestFilterOversizedClosures (0.01s) niks3-go-unit-tests> --- PASS: TestFilterOversizedClosures/all_closures_skipped (0.00s) niks3-go-unit-tests> --- PASS: TestFilterOversizedClosures/no_limit_keeps_everything (0.00s) niks3-go-unit-tests> --- PASS: TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped (0.00s) niks3-go-unit-tests> === CONT TestPartSizeForNAR/80_GiB_fits_at_minimum niks3-go-unit-tests> --- PASS: TestDoWithRetry_ClosesBodyWhenNoAttemptIsMade (0.02s) niks3-go-unit-tests> --- PASS: TestDoWithRetry_ClosesBodyWhenNoAttemptIsMade/retries=0 (0.00s) niks3-go-unit-tests> --- PASS: TestDoWithRetry_ClosesBodyWhenNoAttemptIsMade/retries=5 (0.00s) niks3-go-unit-tests> --- PASS: TestPartSizeForNAR (0.02s) niks3-go-unit-tests> --- PASS: TestPartSizeForNAR/5_TiB_S3_max_object (0.00s) niks3-go-unit-tests> --- PASS: TestPartSizeForNAR/zero_stays_at_minimum (0.00s) niks3-go-unit-tests> --- PASS: TestPartSizeForNAR/115_GiB_needs_larger_parts (0.00s) niks3-go-unit-tests> --- PASS: TestPartSizeForNAR/1_TiB (0.00s) niks3-go-unit-tests> --- PASS: TestPartSizeForNAR/small_stays_at_minimum (0.00s) niks3-go-unit-tests> --- PASS: TestPartSizeForNAR/80_GiB_fits_at_minimum (0.00s) niks3-go-unit-tests> --- PASS: TestPartSizeForNAR/capped_at_5_GiB (0.00s) niks3-go-unit-tests> --- PASS: TestDumpPathWriterError (0.15s) niks3-go-unit-tests> --- PASS: TestRetryTimeoutOutlastsMaxRetries (0.13s) niks3-go-unit-tests> --- PASS: TestSetClientTLS (0.04s) niks3-go-unit-tests> --- PASS: TestSetClientTLS/succeeds_with_client_cert_and_CA (0.03s) niks3-go-unit-tests> --- PASS: TestSetClientTLS/preserves_debug_logging_transport (0.04s) niks3-go-unit-tests> --- PASS: TestSetClientTLS/rejects_connection_without_client_cert (0.11s) niks3-go-unit-tests> --- PASS: TestStreamPushStopsOnCancel (0.02s) niks3-go-unit-tests> --- PASS: TestStreamPushStopsOnCancel/waiting_for_input (0.05s) niks3-go-unit-tests> --- PASS: TestStreamPushStopsOnCancel/input_closed,_last_batch_waiting_for_a_slot (0.05s) niks3-go-unit-tests> --- PASS: TestStreamPushStopsOnCancel/lines_read_but_not_taken (0.05s) niks3-go-unit-tests> --- PASS: TestStreamPushStopsOnCancel/partial_batch_waiting_for_a_slot (0.05s) niks3-go-unit-tests> --- PASS: TestStreamPushStopsOnCancel/full_batch_waiting_for_a_slot (0.05s) niks3-go-unit-tests> --- PASS: TestStreamPushStopsOnCancel/request_line_waiting_for_a_slot (0.05s) niks3-go-unit-tests> --- PASS: TestDumpPathMatchesNix (0.24s) niks3-go-unit-tests> --- PASS: TestRegisterUploadedObjectReusesConnections (0.29s) niks3-go-unit-tests> --- PASS: TestPushPathsReportsOnlyUnuploadablePathsOfSkippedClosure (0.27s) niks3-go-unit-tests> --- PASS: TestTruncatedNARDumpIsNotCompleted (0.45s) niks3-go-unit-tests> --- PASS: TestUploadMultipart_FailedPartBufferNotReused (0.67s) niks3-go-unit-tests> --- PASS: TestUploadMultipart_PartsInParallel (0.68s) niks3-go-unit-tests> --- PASS: TestSupersededNARStillUploadsListing (0.01s) niks3-go-unit-tests> --- PASS: TestSupersededNARStillUploadsListing/listing_upload_fails (0.06s) niks3-go-unit-tests> --- PASS: TestSupersededNARStillUploadsListing/small_NAR (0.04s) niks3-go-unit-tests> --- PASS: TestSupersededNARStillUploadsListing/dump_cut_short (0.65s) niks3-go-unit-tests> --- PASS: TestRateLimiterFeedback_400DoesNotCountAsSuccess (1.00s) niks3-go-unit-tests> --- PASS: TestRegisterUploadedObject_BoundedAgainstSilentServer (2.01s) niks3-go-unit-tests> --- PASS: TestScriptTokenDoesNotWaitForItsChildren (0.01s) niks3-go-unit-tests> --- PASS: TestScriptTokenDoesNotWaitForItsChildren/child_left_holding_stdout (2.05s) niks3-go-unit-tests> --- PASS: TestScriptTokenDoesNotWaitForItsChildren/cancelled_while_a_child_runs (2.21s) niks3-go-unit-tests> PASS niks3-go-unit-tests> Running server tests... niks3-go-unit-tests> The files belonging to this database system will be owned by user "_tribuchetbld5". niks3-go-unit-tests> This user must also own the server process. niks3-go-unit-tests> niks3-go-unit-tests> The database cluster will be initialized with locale "C". niks3-go-unit-tests> The default database encoding has accordingly been set to "SQL_ASCII". niks3-go-unit-tests> The default text search configuration will be set to "english". niks3-go-unit-tests> niks3-go-unit-tests> Data page checksums are enabled. niks3-go-unit-tests> niks3-go-unit-tests> creating directory /private/var/tb/5/scratch/00f1341bdd59a71e/build/postgres1970240236/data ... ok niks3-go-unit-tests> creating subdirectories ... ok niks3-go-unit-tests> selecting dynamic shared memory implementation ... posix niks3-go-unit-tests> selecting default "max_connections" ... 100 niks3-go-unit-tests> selecting default "shared_buffers" ... 128MB niks3-go-unit-tests> selecting default time zone ... UTC niks3-go-unit-tests> creating configuration files ... ok niks3-go-unit-tests> running bootstrap script ... ok niks3-go-unit-tests> performing post-bootstrap initialization ... ok niks3-go-unit-tests> syncing data to disk ... ok niks3-go-unit-tests> niks3-go-unit-tests> initdb: warning: enabling "trust" authentication for local connections niks3-go-unit-tests> initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb. niks3-go-unit-tests> niks3-go-unit-tests> Success. You can now start the database server using: niks3-go-unit-tests> niks3-go-unit-tests> pg_ctl -D /private/var/tb/5/scratch/00f1341bdd59a71e/build/postgres1970240236/data -l logfile start niks3-go-unit-tests> niks3-go-unit-tests> 2026-10-09 09:57:30.815 UTC [65329] LOG: starting PostgreSQL 18.6 on aarch64-apple-darwin25.6.0, compiled by clang version 21.1.8, 64-bit niks3-go-unit-tests> 2026-10-09 09:57:30.815 UTC [65329] LOG: listening on Unix socket "/private/var/tb/5/scratch/00f1341bdd59a71e/build/postgres1970240236/.s.PGSQL.5432" niks3-go-unit-tests> 2026-10-09 09:57:30.817 UTC [65336] LOG: database system was shut down at 2026-10-09 09:57:30 UTC niks3-go-unit-tests> 2026-10-09 09:57:30.818 UTC [65329] LOG: database system is ready to accept connections niks3-go-unit-tests> /private/var/tb/5/scratch/00f1341bdd59a71e/build/postgres1970240236:5432 - accepting connections niks3-go-unit-tests> {"timestamp":"2026-10-09T09:57:33.008729Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"4d6c9634-721f-478f-ab55-734248db51ce","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"GET","uri":"/health/ready","status_code":503,"suppressed_errors":0,"duration_ms":1,"result":"server_error","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":501,"threadName":"rustfs-worker","threadId":"ThreadId(4)"} niks3-go-unit-tests> === RUN TestService_AuthMiddleware niks3-go-unit-tests> === PAUSE TestService_AuthMiddleware niks3-go-unit-tests> === RUN TestService_AuthMiddleware_MTLSProxyHeader niks3-go-unit-tests> === PAUSE TestService_AuthMiddleware_MTLSProxyHeader niks3-go-unit-tests> === RUN TestService_AuthMiddleware_MTLSBoundSubjects niks3-go-unit-tests> === PAUSE TestService_AuthMiddleware_MTLSBoundSubjects niks3-go-unit-tests> === RUN TestService_ReadAuthMiddleware niks3-go-unit-tests> === PAUSE TestService_ReadAuthMiddleware niks3-go-unit-tests> === RUN TestService_AuthMiddleware_OIDC niks3-go-unit-tests> === PAUSE TestService_AuthMiddleware_OIDC niks3-go-unit-tests> === RUN TestService_RequireScope_OIDC niks3-go-unit-tests> === PAUSE TestService_RequireScope_OIDC niks3-go-unit-tests> === RUN TestService_ReadScope_PublicByDefault niks3-go-unit-tests> === PAUSE TestService_ReadScope_PublicByDefault niks3-go-unit-tests> === RUN TestCacheConfigHandler niks3-go-unit-tests> === PAUSE TestCacheConfigHandler niks3-go-unit-tests> === RUN TestCacheStatsHandler niks3-go-unit-tests> === PAUSE TestCacheStatsHandler niks3-go-unit-tests> === RUN TestClientCADerivations niks3-go-unit-tests> === PAUSE TestClientCADerivations niks3-go-unit-tests> === RUN TestClientErrorHandling niks3-go-unit-tests> === PAUSE TestClientErrorHandling niks3-go-unit-tests> === RUN TestClientIntegration niks3-go-unit-tests> === PAUSE TestClientIntegration niks3-go-unit-tests> === RUN TestClientMultipleUploads niks3-go-unit-tests> === PAUSE TestClientMultipleUploads niks3-go-unit-tests> === RUN TestClientWithDependencies niks3-go-unit-tests> === PAUSE TestClientWithDependencies niks3-go-unit-tests> === RUN TestClientSharedPathCommittedMidPush niks3-go-unit-tests> === PAUSE TestClientSharedPathCommittedMidPush niks3-go-unit-tests> === RUN TestPinProtectsFromGC niks3-go-unit-tests> === PAUSE TestPinProtectsFromGC niks3-go-unit-tests> === RUN TestClientPushesUseOnePush niks3-go-unit-tests> === PAUSE TestClientPushesUseOnePush niks3-go-unit-tests> === RUN TestClientFallsBackToClosures niks3-go-unit-tests> === PAUSE TestClientFallsBackToClosures niks3-go-unit-tests> === RUN TestDeleteClosuresRacingPins niks3-go-unit-tests> === PAUSE TestDeleteClosuresRacingPins niks3-go-unit-tests> === RUN TestConcurrentCommitsSharingObjectsDoNotDeadlock niks3-go-unit-tests> === PAUSE TestConcurrentCommitsSharingObjectsDoNotDeadlock niks3-go-unit-tests> === RUN TestResolveDBConnectionString niks3-go-unit-tests> === PAUSE TestResolveDBConnectionString niks3-go-unit-tests> === RUN TestConnectWaitsForAPeerMigration niks3-go-unit-tests> === PAUSE TestConnectWaitsForAPeerMigration niks3-go-unit-tests> === RUN TestConnectSerialisesConcurrentMigrations niks3-go-unit-tests> === PAUSE TestConnectSerialisesConcurrentMigrations niks3-go-unit-tests> === RUN TestLeadElectsOneAndHandsOver niks3-go-unit-tests> === PAUSE TestLeadElectsOneAndHandsOver niks3-go-unit-tests> === RUN TestLeadIncumbentWinsAfterRestart niks3-go-unit-tests> 2026/10/09 09:57:33 INFO lead: acquired remote=192.0.2.1:1234 niks3-go-unit-tests> 2026/10/09 09:57:34 INFO lead: released remote=192.0.2.1:1234 niks3-go-unit-tests> 2026/10/09 09:57:34 INFO lead: acquired remote=192.0.2.1:1234 niks3-go-unit-tests> 2026/10/09 09:57:35 INFO lead: released remote=192.0.2.1:1234 niks3-go-unit-tests> --- PASS: TestLeadIncumbentWinsAfterRestart (2.08s) niks3-go-unit-tests> === RUN TestLeadIncumbentAnnouncesAtOnce niks3-go-unit-tests> === PAUSE TestLeadIncumbentAnnouncesAtOnce niks3-go-unit-tests> === RUN TestLeadEndsOnShutdown niks3-go-unit-tests> === PAUSE TestLeadEndsOnShutdown niks3-go-unit-tests> === RUN TestLeadEndsWhenItsConnectionHangs niks3-go-unit-tests> 2026/10/09 09:57:35 INFO lead: acquired remote=192.0.2.1:1234 niks3-go-unit-tests> 2026-10-09 09:57:36.548 UTC [65470] FATAL: terminating connection due to administrator command niks3-go-unit-tests> 2026/10/09 09:57:36 INFO lead: acquired remote=192.0.2.1:1234 niks3-go-unit-tests> 2026/10/09 09:57:37 WARN lead: lock connection lost error="timeout: context deadline exceeded" niks3-go-unit-tests> 2026/10/09 09:57:37 INFO lead: released remote=192.0.2.1:1234 niks3-go-unit-tests> 2026/10/09 09:57:37 INFO lead: released remote=192.0.2.1:1234 niks3-go-unit-tests> --- PASS: TestLeadEndsWhenItsConnectionHangs (2.76s) niks3-go-unit-tests> === RUN TestGCAdvisoryLockBlocksConcurrentRun niks3-go-unit-tests> === PAUSE TestGCAdvisoryLockBlocksConcurrentRun niks3-go-unit-tests> === RUN TestGCBugBareHashReferences niks3-go-unit-tests> === PAUSE TestGCBugBareHashReferences niks3-go-unit-tests> === RUN TestGCMetrics niks3-go-unit-tests> === PAUSE TestGCMetrics niks3-go-unit-tests> === RUN TestPushDedupSurvivesConcurrentGC niks3-go-unit-tests> === PAUSE TestPushDedupSurvivesConcurrentGC niks3-go-unit-tests> === RUN TestDeduplicatedObjectsRecordedAsPending niks3-go-unit-tests> === PAUSE TestDeduplicatedObjectsRecordedAsPending niks3-go-unit-tests> === RUN TestGCSweepSkipsPendingObjects niks3-go-unit-tests> === PAUSE TestGCSweepSkipsPendingObjects niks3-go-unit-tests> === RUN TestTombstonedObjectOfferedWithoutWaiting niks3-go-unit-tests> === PAUSE TestTombstonedObjectOfferedWithoutWaiting niks3-go-unit-tests> === RUN TestGCSweepDeliversEachKeyOnce niks3-go-unit-tests> === PAUSE TestGCSweepDeliversEachKeyOnce niks3-go-unit-tests> === RUN TestCreatePendingClosureVerifyS3FailureReleasesConnection niks3-go-unit-tests> === PAUSE TestCreatePendingClosureVerifyS3FailureReleasesConnection niks3-go-unit-tests> === RUN TestForceGCDuringPushOffersSweptObject niks3-go-unit-tests> === PAUSE TestForceGCDuringPushOffersSweptObject niks3-go-unit-tests> === RUN TestFailedS3DeleteKeepsTombstone niks3-go-unit-tests> === PAUSE TestFailedS3DeleteKeepsTombstone niks3-go-unit-tests> === RUN TestSweepSkipsRowsLockedByPush niks3-go-unit-tests> === PAUSE TestSweepSkipsRowsLockedByPush niks3-go-unit-tests> === RUN TestRegisterCompletedObjectNeedsPendingRow niks3-go-unit-tests> === PAUSE TestRegisterCompletedObjectNeedsPendingRow niks3-go-unit-tests> === RUN TestGetExistingObjectsLocksRows niks3-go-unit-tests> === PAUSE TestGetExistingObjectsLocksRows niks3-go-unit-tests> === RUN TestSweepRechecksPendingAfterLocking niks3-go-unit-tests> === PAUSE TestSweepRechecksPendingAfterLocking niks3-go-unit-tests> === RUN TestPushWaitsForSweepHoldingRow niks3-go-unit-tests> === PAUSE TestPushWaitsForSweepHoldingRow niks3-go-unit-tests> === RUN TestCommitRacingPendingCleanupKeepsObjects niks3-go-unit-tests> === PAUSE TestCommitRacingPendingCleanupKeepsObjects niks3-go-unit-tests> === RUN TestGCEndsOnShutdown niks3-go-unit-tests> === PAUSE TestGCEndsOnShutdown niks3-go-unit-tests> === RUN TestGCTaskStore_StartNew niks3-go-unit-tests> === PAUSE TestGCTaskStore_StartNew niks3-go-unit-tests> === RUN TestGCTaskStore_DeduplicateSameParams niks3-go-unit-tests> === PAUSE TestGCTaskStore_DeduplicateSameParams niks3-go-unit-tests> === RUN TestGCTaskStore_ConflictDifferentParams niks3-go-unit-tests> === PAUSE TestGCTaskStore_ConflictDifferentParams niks3-go-unit-tests> === RUN TestGCTaskStore_GetEmpty niks3-go-unit-tests> === PAUSE TestGCTaskStore_GetEmpty niks3-go-unit-tests> === RUN TestGCTaskStore_GetReturnsLatest niks3-go-unit-tests> === PAUSE TestGCTaskStore_GetReturnsLatest niks3-go-unit-tests> === RUN TestGCTaskStore_CompletedAllowsNewTask niks3-go-unit-tests> === PAUSE TestGCTaskStore_CompletedAllowsNewTask niks3-go-unit-tests> === RUN TestGCTaskStore_PhaseUpdates niks3-go-unit-tests> === PAUSE TestGCTaskStore_PhaseUpdates niks3-go-unit-tests> === RUN TestGCTaskStore_Fail niks3-go-unit-tests> === PAUSE TestGCTaskStore_Fail niks3-go-unit-tests> === RUN TestGracefulShutdownDrainsInflight niks3-go-unit-tests> === PAUSE TestGracefulShutdownDrainsInflight niks3-go-unit-tests> === RUN TestService_healthCheckHandler niks3-go-unit-tests> === PAUSE TestService_healthCheckHandler niks3-go-unit-tests> === RUN TestService_readinessHandler niks3-go-unit-tests> === PAUSE TestService_readinessHandler niks3-go-unit-tests> === RUN TestGenerateLandingPage niks3-go-unit-tests> === PAUSE TestGenerateLandingPage niks3-go-unit-tests> === RUN TestCacheConfigHandlerMaxNarSize niks3-go-unit-tests> === PAUSE TestCacheConfigHandlerMaxNarSize niks3-go-unit-tests> === RUN TestCreatePendingClosureRejectsOversizedNAR niks3-go-unit-tests> === PAUSE TestCreatePendingClosureRejectsOversizedNAR niks3-go-unit-tests> === RUN TestNARDeduplicationMetadataUploadBug niks3-go-unit-tests> === PAUSE TestNARDeduplicationMetadataUploadBug niks3-go-unit-tests> === RUN TestMetricsInventory niks3-go-unit-tests> === PAUSE TestMetricsInventory niks3-go-unit-tests> === RUN TestMetricsCommit niks3-go-unit-tests> === PAUSE TestMetricsCommit niks3-go-unit-tests> === RUN TestService_NativeMTLS niks3-go-unit-tests> === PAUSE TestService_NativeMTLS niks3-go-unit-tests> === RUN TestServerTLSConfig niks3-go-unit-tests> === PAUSE TestServerTLSConfig niks3-go-unit-tests> === RUN TestMultipartCleanup niks3-go-unit-tests> === PAUSE TestMultipartCleanup niks3-go-unit-tests> === RUN TestMultipartUploadAbortedWhenCancelledBeforeRecorded niks3-go-unit-tests> === PAUSE TestMultipartUploadAbortedWhenCancelledBeforeRecorded niks3-go-unit-tests> === RUN TestPendingCleanupUsesOneCutoff niks3-go-unit-tests> === PAUSE TestPendingCleanupUsesOneCutoff niks3-go-unit-tests> === RUN TestPendingClosureFailureTracksEveryUpload niks3-go-unit-tests> === PAUSE TestPendingClosureFailureTracksEveryUpload niks3-go-unit-tests> === RUN TestPendingClosureFailureAbortsItsUploads niks3-go-unit-tests> === PAUSE TestPendingClosureFailureAbortsItsUploads niks3-go-unit-tests> === RUN TestObjectStatsTrigger niks3-go-unit-tests> === PAUSE TestObjectStatsTrigger niks3-go-unit-tests> === RUN TestObjectStatsLargeCommit niks3-go-unit-tests> === PAUSE TestObjectStatsLargeCommit niks3-go-unit-tests> === RUN TestReconnectLeavesObjectsUnlocked niks3-go-unit-tests> === PAUSE TestReconnectLeavesObjectsUnlocked niks3-go-unit-tests> === RUN TestOrphanedObjectsGC niks3-go-unit-tests> === PAUSE TestOrphanedObjectsGC niks3-go-unit-tests> === RUN TestOrphanedObjectsGCStressTest niks3-go-unit-tests> === PAUSE TestOrphanedObjectsGCStressTest niks3-go-unit-tests> === RUN TestCreatePin_ReservedPins niks3-go-unit-tests> === PAUSE TestCreatePin_ReservedPins niks3-go-unit-tests> === RUN TestConcurrentPinUpdatesAgree niks3-go-unit-tests> === PAUSE TestConcurrentPinUpdatesAgree niks3-go-unit-tests> === RUN TestCreatePinRejectsBadInput niks3-go-unit-tests> === PAUSE TestCreatePinRejectsBadInput niks3-go-unit-tests> === RUN TestDeletePinKeepsRowWhenS3Fails niks3-go-unit-tests> === PAUSE TestDeletePinKeepsRowWhenS3Fails niks3-go-unit-tests> === RUN TestPresentReportsOnlyClosureRoots niks3-go-unit-tests> === PAUSE TestPresentReportsOnlyClosureRoots niks3-go-unit-tests> === RUN TestPresentRefreshesOnlyStaleClosures niks3-go-unit-tests> === PAUSE TestPresentRefreshesOnlyStaleClosures niks3-go-unit-tests> === RUN TestPresentNotReportedWhileGCDeletesClosure niks3-go-unit-tests> === PAUSE TestPresentNotReportedWhileGCDeletesClosure niks3-go-unit-tests> === RUN TestParseSingleRange niks3-go-unit-tests> === PAUSE TestParseSingleRange niks3-go-unit-tests> === RUN TestProxyHeadersOnlyTrustedOnSocket niks3-go-unit-tests> 2026/10/09 09:57:38 INFO Starting HTTP server address=127.0.0.1:61170 niks3-go-unit-tests> 2026/10/09 09:57:38 INFO Starting HTTP server address=proxy.sock niks3-go-unit-tests> 2026/10/09 09:57:38 WARN mTLS auth: subject not in bound subjects subject="CN=someone" niks3-go-unit-tests> 2026/10/09 09:57:38 INFO Shutdown signal received, draining in-flight requests timeout=10s niks3-go-unit-tests> --- PASS: TestProxyHeadersOnlyTrustedOnSocket (0.15s) niks3-go-unit-tests> === RUN TestIsValidCachePath niks3-go-unit-tests> === PAUSE TestIsValidCachePath niks3-go-unit-tests> === RUN TestReadProxyNarinfo niks3-go-unit-tests> === PAUSE TestReadProxyNarinfo niks3-go-unit-tests> === RUN TestReadProxyNarinfoAlreadyDecompressed niks3-go-unit-tests> === PAUSE TestReadProxyNarinfoAlreadyDecompressed niks3-go-unit-tests> === RUN TestReadProxyContentNegotiation niks3-go-unit-tests> === PAUSE TestReadProxyContentNegotiation niks3-go-unit-tests> === RUN TestReadProxyNarStreaming niks3-go-unit-tests> === PAUSE TestReadProxyNarStreaming niks3-go-unit-tests> === RUN TestReadProxy404 niks3-go-unit-tests> === PAUSE TestReadProxy404 niks3-go-unit-tests> === RUN TestReadProxyInvalidPath niks3-go-unit-tests> === PAUSE TestReadProxyInvalidPath niks3-go-unit-tests> === RUN TestReadProxyHead niks3-go-unit-tests> === PAUSE TestReadProxyHead niks3-go-unit-tests> === RUN TestReadProxyOutlastsServerWriteTimeout niks3-go-unit-tests> === PAUSE TestReadProxyOutlastsServerWriteTimeout niks3-go-unit-tests> === RUN TestReadProxyConditionalGet niks3-go-unit-tests> === PAUSE TestReadProxyConditionalGet niks3-go-unit-tests> === RUN TestReadProxyRootRedirectsToIndexHTML niks3-go-unit-tests> === PAUSE TestReadProxyRootRedirectsToIndexHTML niks3-go-unit-tests> === RUN TestReadProxyDisabled niks3-go-unit-tests> === PAUSE TestReadProxyDisabled niks3-go-unit-tests> === RUN TestReadRedirectNar niks3-go-unit-tests> === PAUSE TestReadRedirectNar niks3-go-unit-tests> === RUN TestReadRedirectKeepsNarinfoProxied niks3-go-unit-tests> === PAUSE TestReadRedirectKeepsNarinfoProxied niks3-go-unit-tests> === RUN TestReadProxyRangeRequest niks3-go-unit-tests> === PAUSE TestReadProxyRangeRequest niks3-go-unit-tests> === RUN TestReadRedirectUsesPublicS3URL niks3-go-unit-tests> === PAUSE TestReadRedirectUsesPublicS3URL niks3-go-unit-tests> === RUN TestPush_OverlappingRootsStoreOneRowPerKey niks3-go-unit-tests> === PAUSE TestPush_OverlappingRootsStoreOneRowPerKey niks3-go-unit-tests> === RUN TestPush_CompleteCommitsEveryRoot niks3-go-unit-tests> === PAUSE TestPush_CompleteCommitsEveryRoot niks3-go-unit-tests> === RUN TestPush_RepeatedRootCommitsOnce niks3-go-unit-tests> === PAUSE TestPush_RepeatedRootCommitsOnce niks3-go-unit-tests> === RUN TestPush_CompleteWaitsForCleanupOfThePush niks3-go-unit-tests> === PAUSE TestPush_CompleteWaitsForCleanupOfThePush niks3-go-unit-tests> === RUN TestPush_RootsAreLockedInByteOrder niks3-go-unit-tests> === PAUSE TestPush_RootsAreLockedInByteOrder niks3-go-unit-tests> === RUN TestPush_SkippedKeySurvivesGCBeforeCommit niks3-go-unit-tests> === PAUSE TestPush_SkippedKeySurvivesGCBeforeCommit niks3-go-unit-tests> === RUN TestPush_RejectsBadRequests niks3-go-unit-tests> === PAUSE TestPush_RejectsBadRequests niks3-go-unit-tests> === RUN TestPush_SignsNarinfosOfItsPendingObjects niks3-go-unit-tests> === PAUSE TestPush_SignsNarinfosOfItsPendingObjects niks3-go-unit-tests> === RUN TestRedundantMultipartUpload niks3-go-unit-tests> === PAUSE TestRedundantMultipartUpload niks3-go-unit-tests> === RUN TestCompleteMultipartUpload_ErrorButObjectExists niks3-go-unit-tests> === PAUSE TestCompleteMultipartUpload_ErrorButObjectExists niks3-go-unit-tests> === RUN TestCompletedNarNotReofferedAcrossClosures niks3-go-unit-tests> === PAUSE TestCompletedNarNotReofferedAcrossClosures niks3-go-unit-tests> === RUN TestPresignedUploadRegisteredBeforeCommit niks3-go-unit-tests> === PAUSE TestPresignedUploadRegisteredBeforeCommit niks3-go-unit-tests> === RUN TestService_Rustfstest niks3-go-unit-tests> === PAUSE TestService_Rustfstest niks3-go-unit-tests> === RUN TestParseSize niks3-go-unit-tests> === PAUSE TestParseSize niks3-go-unit-tests> === RUN TestSkippedUploadsHandler niks3-go-unit-tests> === PAUSE TestSkippedUploadsHandler niks3-go-unit-tests> === RUN TestSystemdListenerNotActivated niks3-go-unit-tests> --- PASS: TestSystemdListenerNotActivated (0.00s) niks3-go-unit-tests> === RUN TestWatchdogBeatsWhenHealthy niks3-go-unit-tests> --- PASS: TestWatchdogBeatsWhenHealthy (0.02s) niks3-go-unit-tests> === RUN TestWatchdogSkipsWhenUnhealthy niks3-go-unit-tests> 2026/10/09 09:57:38 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded" niks3-go-unit-tests> 2026/10/09 09:57:38 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded" niks3-go-unit-tests> 2026/10/09 09:57:38 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded" niks3-go-unit-tests> 2026/10/09 09:57:38 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded" niks3-go-unit-tests> 2026/10/09 09:57:38 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded" niks3-go-unit-tests> 2026/10/09 09:57:38 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded" niks3-go-unit-tests> 2026/10/09 09:57:38 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded" niks3-go-unit-tests> 2026/10/09 09:57:38 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded" niks3-go-unit-tests> 2026/10/09 09:57:38 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded" niks3-go-unit-tests> 2026/10/09 09:57:38 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded" niks3-go-unit-tests> --- PASS: TestWatchdogSkipsWhenUnhealthy (0.20s) niks3-go-unit-tests> === RUN TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle niks3-go-unit-tests> === PAUSE TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle niks3-go-unit-tests> === RUN TestProxyWriteTimeout niks3-go-unit-tests> === PAUSE TestProxyWriteTimeout niks3-go-unit-tests> === RUN TestPendingClosureWriteTimeout niks3-go-unit-tests> === PAUSE TestPendingClosureWriteTimeout niks3-go-unit-tests> === RUN TestIsValidUploadKey niks3-go-unit-tests> === PAUSE TestIsValidUploadKey niks3-go-unit-tests> === RUN TestUploadHandlersRejectInvalidKeys niks3-go-unit-tests> === PAUSE TestUploadHandlersRejectInvalidKeys niks3-go-unit-tests> === RUN TestUploadHandlersRejectOversizedBody niks3-go-unit-tests> === PAUSE TestUploadHandlersRejectOversizedBody niks3-go-unit-tests> === RUN TestService_cleanupPendingClosuresHandler niks3-go-unit-tests> === PAUSE TestService_cleanupPendingClosuresHandler niks3-go-unit-tests> === RUN TestService_createPendingClosureHandler niks3-go-unit-tests> === PAUSE TestService_createPendingClosureHandler niks3-go-unit-tests> === RUN TestService_verifyS3Integrity niks3-go-unit-tests> === PAUSE TestService_verifyS3Integrity niks3-go-unit-tests> === RUN TestCompleteMultipartUnregistered niks3-go-unit-tests> === PAUSE TestCompleteMultipartUnregistered niks3-go-unit-tests> === RUN TestCreatePendingClosure_SmallNARUsesSimplePUT niks3-go-unit-tests> === PAUSE TestCreatePendingClosure_SmallNARUsesSimplePUT niks3-go-unit-tests> === CONT TestService_AuthMiddleware niks3-go-unit-tests> === CONT TestService_createPendingClosureHandler niks3-go-unit-tests> === CONT TestGCTaskStore_CompletedAllowsNewTask niks3-go-unit-tests> === CONT TestPendingClosureFailureAbortsItsUploads niks3-go-unit-tests> --- PASS: TestGCTaskStore_CompletedAllowsNewTask (0.00s) niks3-go-unit-tests> === CONT TestIsValidUploadKey niks3-go-unit-tests> === CONT TestOrphanedObjectsGC niks3-go-unit-tests> === CONT TestService_verifyS3Integrity niks3-go-unit-tests> === CONT TestService_readinessHandler niks3-go-unit-tests> === CONT TestFailedS3DeleteKeepsTombstone niks3-go-unit-tests> === CONT TestCompleteMultipartUnregistered niks3-go-unit-tests> === CONT TestService_cleanupPendingClosuresHandler niks3-go-unit-tests> === CONT TestGracefulShutdownDrainsInflight niks3-go-unit-tests> === CONT TestObjectStatsLargeCommit niks3-go-unit-tests> === CONT TestReconnectLeavesObjectsUnlocked niks3-go-unit-tests> === CONT TestConcurrentCommitsSharingObjectsDoNotDeadlock niks3-go-unit-tests> === CONT TestReadRedirectKeepsNarinfoProxied niks3-go-unit-tests> === CONT TestCreatePendingClosureVerifyS3FailureReleasesConnection niks3-go-unit-tests> === CONT TestService_healthCheckHandler niks3-go-unit-tests> === CONT TestForceGCDuringPushOffersSweptObject niks3-go-unit-tests> === RUN TestForceGCDuringPushOffersSweptObject/before_pending_rows niks3-go-unit-tests> === CONT TestPendingClosureFailureTracksEveryUpload niks3-go-unit-tests> === PAUSE TestForceGCDuringPushOffersSweptObject/before_pending_rows niks3-go-unit-tests> === CONT TestUploadHandlersRejectOversizedBody niks3-go-unit-tests> === CONT TestUploadHandlersRejectInvalidKeys niks3-go-unit-tests> === CONT TestPendingClosureWriteTimeout niks3-go-unit-tests> === CONT TestObjectStatsTrigger niks3-go-unit-tests> === RUN TestIsValidUploadKey/narinfo niks3-go-unit-tests> === CONT TestCreatePendingClosure_SmallNARUsesSimplePUT niks3-go-unit-tests> === RUN TestPendingClosureFailureTracksEveryUpload/refuse_aborts=false niks3-go-unit-tests> 2026/10/09 09:57:38 INFO Starting HTTP server address=127.0.0.1:61179 niks3-go-unit-tests> === RUN TestForceGCDuringPushOffersSweptObject/after_presence_check niks3-go-unit-tests> === RUN TestPendingClosureWriteTimeout/empty niks3-go-unit-tests> === RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info niks3-go-unit-tests> === PAUSE TestIsValidUploadKey/narinfo niks3-go-unit-tests> === PAUSE TestPendingClosureWriteTimeout/empty niks3-go-unit-tests> === RUN TestIsValidUploadKey/nar_zst niks3-go-unit-tests> === RUN TestPendingClosureWriteTimeout/400_objects niks3-go-unit-tests> === PAUSE TestForceGCDuringPushOffersSweptObject/after_presence_check niks3-go-unit-tests> === PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info niks3-go-unit-tests> === PAUSE TestPendingClosureWriteTimeout/400_objects niks3-go-unit-tests> === PAUSE TestIsValidUploadKey/nar_zst niks3-go-unit-tests> === RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal niks3-go-unit-tests> === RUN TestPendingClosureWriteTimeout/670k_objects niks3-go-unit-tests> === RUN TestIsValidUploadKey/nar_xz niks3-go-unit-tests> === CONT TestGCSweepDeliversEachKeyOnce niks3-go-unit-tests> === PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal niks3-go-unit-tests> === RUN TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key niks3-go-unit-tests> === PAUSE TestPendingClosureWriteTimeout/670k_objects niks3-go-unit-tests> === RUN TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline niks3-go-unit-tests> === PAUSE TestIsValidUploadKey/nar_xz niks3-go-unit-tests> === PAUSE TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key niks3-go-unit-tests> === RUN TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key niks3-go-unit-tests> === PAUSE TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key niks3-go-unit-tests> === CONT TestSkippedUploadsHandler niks3-go-unit-tests> === PAUSE TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline niks3-go-unit-tests> === PAUSE TestPendingClosureFailureTracksEveryUpload/refuse_aborts=false niks3-go-unit-tests> === RUN TestPendingClosureFailureTracksEveryUpload/refuse_aborts=true niks3-go-unit-tests> === CONT TestPendingCleanupUsesOneCutoff niks3-go-unit-tests> === RUN TestIsValidUploadKey/nar_plain niks3-go-unit-tests> === PAUSE TestIsValidUploadKey/nar_plain niks3-go-unit-tests> === RUN TestIsValidUploadKey/listing niks3-go-unit-tests> === PAUSE TestIsValidUploadKey/listing niks3-go-unit-tests> === RUN TestIsValidUploadKey/build_log niks3-go-unit-tests> === PAUSE TestIsValidUploadKey/build_log niks3-go-unit-tests> === RUN TestIsValidUploadKey/build_log_home-manager_file niks3-go-unit-tests> === PAUSE TestIsValidUploadKey/build_log_home-manager_file niks3-go-unit-tests> === RUN TestIsValidUploadKey/build_log_plus_in_name niks3-go-unit-tests> === PAUSE TestIsValidUploadKey/build_log_plus_in_name niks3-go-unit-tests> === RUN TestIsValidUploadKey/build_log_question_mark niks3-go-unit-tests> === PAUSE TestIsValidUploadKey/build_log_question_mark niks3-go-unit-tests> === RUN TestIsValidUploadKey/build_log_equals niks3-go-unit-tests> === PAUSE TestIsValidUploadKey/build_log_equals niks3-go-unit-tests> === RUN TestIsValidUploadKey/realisation niks3-go-unit-tests> === PAUSE TestIsValidUploadKey/realisation niks3-go-unit-tests> === RUN TestIsValidUploadKey/realisation_plus_in_output niks3-go-unit-tests> === PAUSE TestIsValidUploadKey/realisation_plus_in_output niks3-go-unit-tests> === RUN TestIsValidUploadKey/nix-cache-info niks3-go-unit-tests> === PAUSE TestIsValidUploadKey/nix-cache-info niks3-go-unit-tests> === RUN TestIsValidUploadKey/index.html niks3-go-unit-tests> === PAUSE TestIsValidUploadKey/index.html niks3-go-unit-tests> === RUN TestIsValidUploadKey/narinfo_key,_nar_type niks3-go-unit-tests> === PAUSE TestIsValidUploadKey/narinfo_key,_nar_type niks3-go-unit-tests> === RUN TestIsValidUploadKey/nar_key,_narinfo_type niks3-go-unit-tests> === PAUSE TestIsValidUploadKey/nar_key,_narinfo_type niks3-go-unit-tests> === RUN TestIsValidUploadKey/listing_key,_narinfo_type niks3-go-unit-tests> === PAUSE TestIsValidUploadKey/listing_key,_narinfo_type niks3-go-unit-tests> === RUN TestIsValidUploadKey/traversal niks3-go-unit-tests> === PAUSE TestIsValidUploadKey/traversal niks3-go-unit-tests> === RUN TestIsValidUploadKey/traversal_nar niks3-go-unit-tests> === PAUSE TestIsValidUploadKey/traversal_nar niks3-go-unit-tests> === RUN TestIsValidUploadKey/absolute niks3-go-unit-tests> === PAUSE TestIsValidUploadKey/absolute niks3-go-unit-tests> === RUN TestIsValidUploadKey/empty_key niks3-go-unit-tests> === PAUSE TestIsValidUploadKey/empty_key niks3-go-unit-tests> === RUN TestIsValidUploadKey/unknown_type niks3-go-unit-tests> === PAUSE TestIsValidUploadKey/unknown_type niks3-go-unit-tests> === CONT TestService_Rustfstest niks3-go-unit-tests> === PAUSE TestPendingClosureFailureTracksEveryUpload/refuse_aborts=true niks3-go-unit-tests> 2026/10/09 09:57:38 INFO Shutdown signal received, draining in-flight requests timeout=10s niks3-go-unit-tests> === CONT TestTombstonedObjectOfferedWithoutWaiting niks3-go-unit-tests> 2026/10/09 09:57:38 INFO Client skipped oversized paths paths=3 nar_bytes=5000000000 niks3-go-unit-tests> --- PASS: TestSkippedUploadsHandler (0.04s) niks3-go-unit-tests> === CONT TestParseSize niks3-go-unit-tests> --- PASS: TestParseSize (0.00s) niks3-go-unit-tests> === CONT TestPresignedUploadRegisteredBeforeCommit niks3-go-unit-tests> --- PASS: TestGracefulShutdownDrainsInflight (0.07s) niks3-go-unit-tests> === CONT TestMultipartUploadAbortedWhenCancelledBeforeRecorded niks3-go-unit-tests> === RUN TestUploadHandlersRejectOversizedBody/create_pending_closure niks3-go-unit-tests> === PAUSE TestUploadHandlersRejectOversizedBody/create_pending_closure niks3-go-unit-tests> === RUN TestUploadHandlersRejectOversizedBody/complete_multipart niks3-go-unit-tests> === PAUSE TestUploadHandlersRejectOversizedBody/complete_multipart niks3-go-unit-tests> === RUN TestUploadHandlersRejectOversizedBody/request_more_parts niks3-go-unit-tests> === PAUSE TestUploadHandlersRejectOversizedBody/request_more_parts niks3-go-unit-tests> === CONT TestCompleteMultipartUpload_ErrorButObjectExists niks3-go-unit-tests> 2026/10/09 09:57:38 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> --- PASS: TestPendingClosureFailureAbortsItsUploads (0.74s) niks3-go-unit-tests> === CONT TestGCSweepSkipsPendingObjects niks3-go-unit-tests> --- PASS: TestReadRedirectKeepsNarinfoProxied (1.68s) niks3-go-unit-tests> === CONT TestCompletedNarNotReofferedAcrossClosures niks3-go-unit-tests> 2026/10/09 09:57:40 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> 2026/10/09 09:57:40 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> 2026/10/09 09:57:40 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> === NAME TestOrphanedObjectsGC niks3-go-unit-tests> orphaned_objects_gc_test.go:287: GC Test Summary: niks3-go-unit-tests> orphaned_objects_gc_test.go:288: - Kept: 2 objects from closure A niks3-go-unit-tests> orphaned_objects_gc_test.go:289: - Deleted: 2 objects from closure B niks3-go-unit-tests> orphaned_objects_gc_test.go:290: - Deleted: 6 orphaned chain objects (X1->X2->X3) niks3-go-unit-tests> orphaned_objects_gc_test.go:291: - Deleted: 2 orphaned single objects (Y) niks3-go-unit-tests> orphaned_objects_gc_test.go:292: - Total deleted: 10 objects niks3-go-unit-tests> --- PASS: TestOrphanedObjectsGC (1.96s) niks3-go-unit-tests> === CONT TestRedundantMultipartUpload niks3-go-unit-tests> 2026/10/09 09:57:40 WARN readiness check failed error="closed pool" niks3-go-unit-tests> --- PASS: TestService_readinessHandler (2.46s) niks3-go-unit-tests> === CONT TestPush_SkippedKeySurvivesGCBeforeCommit niks3-go-unit-tests> 2026/10/09 09:57:41 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> 2026/10/09 09:57:41 INFO Aborted multipart uploads count=0 kept=0 niks3-go-unit-tests> 2026/10/09 09:57:41 WARN Force mode enabled - objects will be deleted immediately without grace period niks3-go-unit-tests> 2026/10/09 09:57:41 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied." niks3-go-unit-tests> 2026/10/09 09:57:41 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied." niks3-go-unit-tests> --- PASS: TestFailedS3DeleteKeepsTombstone (3.03s) niks3-go-unit-tests> === CONT TestService_NativeMTLS niks3-go-unit-tests> 2026/10/09 09:57:41 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> 2026/10/09 09:57:41 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> 2026/10/09 09:57:41 INFO Received cleanup request method=DELETE path=/api/pending_closures niks3-go-unit-tests> 2026/10/09 09:57:41 INFO Received cleanup request method=DELETE path=/api/pending_closures niks3-go-unit-tests> 2026/10/09 09:57:41 INFO Aborted multipart uploads count=0 kept=0 niks3-go-unit-tests> 2026/10/09 09:57:41 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> 2026/10/09 09:57:41 INFO Received cleanup request method=DELETE path=/api/pending_closures niks3-go-unit-tests> 2026/10/09 09:57:41 INFO Aborted multipart uploads count=1 kept=0 niks3-go-unit-tests> 2026/10/09 09:57:41 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete niks3-go-unit-tests> 2026/10/09 09:57:41 INFO Received complete multipart upload request method=POST path=/api/multipart/complete niks3-go-unit-tests> 2026-10-09 09:57:41.838 UTC [65645] ERROR: Closure does not exist: id=1 niks3-go-unit-tests> 2026-10-09 09:57:41.838 UTC [65645] CONTEXT: PL/pgSQL function commit_pending_closure(bigint) line 13 at RAISE niks3-go-unit-tests> 2026-10-09 09:57:41.838 UTC [65645] STATEMENT: -- name: CommitPendingClosure :one niks3-go-unit-tests> SELECT commit_pending_closure($1::bigint)::bigint niks3-go-unit-tests> niks3-go-unit-tests> === CONT TestPush_CompleteWaitsForCleanupOfThePush niks3-go-unit-tests> --- PASS: TestService_cleanupPendingClosuresHandler (3.50s) niks3-go-unit-tests> 2026/10/09 09:57:41 INFO Completed multipart upload object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst upload_id=MTE5MjdlYzItNjE3OS00ZDA5LTljZjMtZGFiOThmMGQ2ZDQ0LjE5MDUyNzM0LWM0MTgtNDBkMC04OWY3LTA4Mjc4MGQ1MTVmOHgxNzkxNTM5ODYwMzc1MjkxMDAw parts=10 niks3-go-unit-tests> 2026/10/09 09:57:41 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete niks3-go-unit-tests> 2026/10/09 09:57:41 INFO Completed upload id=1 objects=2 duration=16.561334ms niks3-go-unit-tests> 2026/10/09 09:57:41 INFO Received get closure request method=GET path=/api/closures/00000000000000000000000000000000 niks3-go-unit-tests> 2026/10/09 09:57:41 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> 2026/10/09 09:57:41 INFO Starting cleanup of old closures method=DELETE path=/api/closures niks3-go-unit-tests> 2026/10/09 09:57:41 INFO Aborted multipart uploads count=0 kept=0 niks3-go-unit-tests> --- PASS: TestService_Rustfstest (3.61s) niks3-go-unit-tests> 2026/10/09 09:57:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=0 niks3-go-unit-tests> === CONT TestPush_RootsAreLockedInByteOrder niks3-go-unit-tests> 2026/10/09 09:57:42 INFO Vacuumed table table=pending_closures niks3-go-unit-tests> 2026/10/09 09:57:42 INFO Vacuumed table table=pending_objects niks3-go-unit-tests> 2026/10/09 09:57:42 INFO Vacuumed table table=multipart_uploads niks3-go-unit-tests> 2026/10/09 09:57:42 INFO Vacuumed table table=closures niks3-go-unit-tests> 2026/10/09 09:57:42 INFO Vacuumed table table=objects niks3-go-unit-tests> 2026/10/09 09:57:42 INFO Received get closure request method=GET path=/api/closures/00000000000000000000000000000000 niks3-go-unit-tests> --- PASS: TestService_createPendingClosureHandler (3.76s) niks3-go-unit-tests> === CONT TestPushDedupSurvivesConcurrentGC niks3-go-unit-tests> 2026/10/09 09:57:42 WARN Authentication failed token_preview=wrongtoken token_length=10 reason="static API token mismatch" niks3-go-unit-tests> --- PASS: TestService_AuthMiddleware (3.84s) niks3-go-unit-tests> === CONT TestPush_CompleteCommitsEveryRoot niks3-go-unit-tests> --- PASS: TestConcurrentCommitsSharingObjectsDoNotDeadlock (3.93s) niks3-go-unit-tests> === CONT TestMetricsCommit niks3-go-unit-tests> --- PASS: TestObjectStatsTrigger (4.12s) niks3-go-unit-tests> === CONT TestPush_RejectsBadRequests niks3-go-unit-tests> 2026/10/09 09:57:42 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> --- PASS: TestTombstonedObjectOfferedWithoutWaiting (4.37s) niks3-go-unit-tests> === CONT TestMultipartCleanup niks3-go-unit-tests> 2026/10/09 09:57:42 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> 2026/10/09 09:57:43 INFO Received complete multipart upload request method=POST path=/api/multipart/complete niks3-go-unit-tests> 2026/10/09 09:57:43 ERROR Failed to get multipart upload error="no rows in result set" upload_id=does-not-exist object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst niks3-go-unit-tests> === CONT TestReadRedirectUsesPublicS3URL niks3-go-unit-tests> --- PASS: TestCompleteMultipartUnregistered (4.89s) niks3-go-unit-tests> --- PASS: TestReconnectLeavesObjectsUnlocked (5.24s) niks3-go-unit-tests> === CONT TestPush_SignsNarinfosOfItsPendingObjects niks3-go-unit-tests> 2026/10/09 09:57:44 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> --- PASS: TestMultipartUploadAbortedWhenCancelledBeforeRecorded (5.82s) niks3-go-unit-tests> === CONT TestDeduplicatedObjectsRecordedAsPending niks3-go-unit-tests> 2026/10/09 09:57:44 INFO Received complete multipart upload request method=POST path=/api/multipart/complete niks3-go-unit-tests> 2026/10/09 09:57:44 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> 2026/10/09 09:57:44 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst upload_id=MTE5MjdlYzItNjE3OS00ZDA5LTljZjMtZGFiOThmMGQ2ZDQ0LjJlMTk3Y2M1LWQwMGItNDBhZS1hMmE3LTcwZmVkY2QzMzQyN3gxNzkxNTM5ODYyOTIzNzkzMDAw parts=10 niks3-go-unit-tests> 2026/10/09 09:57:44 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete niks3-go-unit-tests> 2026/10/09 09:57:44 INFO Completed upload id=1 objects=2 duration=6.389083ms niks3-go-unit-tests> 2026/10/09 09:57:44 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> 2026/10/09 09:57:44 INFO Registered completed upload object_key=nar/jjjjjjjjjjjjjjjjjjjjjjjjjjjjjj0100000000000000000000.nar.zst niks3-go-unit-tests> 2026/10/09 09:57:44 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> 2026/10/09 09:57:44 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> 2026/10/09 09:57:44 INFO Object in database but missing from S3 key=dadb44fdadb44fdadb44fdadb44f0000.narinfo niks3-go-unit-tests> 2026/10/09 09:57:44 WARN Found objects in DB but missing from S3, will re-upload count=1 niks3-go-unit-tests> --- PASS: TestPresignedUploadRegisteredBeforeCommit (6.06s) niks3-go-unit-tests> === CONT TestGCBugBareHashReferences niks3-go-unit-tests> 2026/10/09 09:57:44 INFO Received complete upload request method=POST path=/api/pending_closures/3/complete niks3-go-unit-tests> 2026/10/09 09:57:44 INFO Completed upload id=3 objects=2 duration=618.541µs niks3-go-unit-tests> --- PASS: TestService_verifyS3Integrity (6.18s) niks3-go-unit-tests> === CONT TestNARDeduplicationMetadataUploadBug niks3-go-unit-tests> 2026/10/09 09:57:44 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> === CONT TestLeadEndsOnShutdown niks3-go-unit-tests> --- PASS: TestCreatePendingClosureVerifyS3FailureReleasesConnection (6.59s) niks3-go-unit-tests> === CONT TestGenerateLandingPage niks3-go-unit-tests> --- PASS: TestService_healthCheckHandler (6.60s) niks3-go-unit-tests> --- PASS: TestGenerateLandingPage (0.01s) niks3-go-unit-tests> === CONT TestCacheConfigHandlerMaxNarSize niks3-go-unit-tests> --- PASS: TestCacheConfigHandlerMaxNarSize (0.00s) niks3-go-unit-tests> === CONT TestGCAdvisoryLockBlocksConcurrentRun niks3-go-unit-tests> 2026/10/09 09:57:45 INFO Aborted multipart uploads count=0 kept=0 niks3-go-unit-tests> 2026/10/09 09:57:45 WARN Force mode enabled - objects will be deleted immediately without grace period niks3-go-unit-tests> 2026/10/09 09:57:45 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=0 niks3-go-unit-tests> 2026/10/09 09:57:45 INFO Vacuumed table table=pending_closures niks3-go-unit-tests> 2026/10/09 09:57:45 INFO Vacuumed table table=pending_objects niks3-go-unit-tests> 2026/10/09 09:57:45 INFO Vacuumed table table=multipart_uploads niks3-go-unit-tests> 2026/10/09 09:57:45 INFO Vacuumed table table=closures niks3-go-unit-tests> 2026/10/09 09:57:45 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> 2026/10/09 09:57:45 INFO Vacuumed table table=objects niks3-go-unit-tests> --- PASS: TestGCSweepDeliversEachKeyOnce (7.38s) niks3-go-unit-tests> === CONT TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle niks3-go-unit-tests> --- PASS: TestCreatePendingClosure_SmallNARUsesSimplePUT (7.48s) niks3-go-unit-tests> === CONT TestClientPushesUseOnePush niks3-go-unit-tests> 2026/10/09 09:57:45 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> 2026/10/09 09:57:46 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> 2026/10/09 09:57:46 INFO Received complete multipart upload request method=POST path=/api/multipart/complete niks3-go-unit-tests> 2026/10/09 09:57:46 WARN CompleteMultipartUpload errored but object exists; treating as success error="The specified key does not exist." object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=MTE5MjdlYzItNjE3OS00ZDA5LTljZjMtZGFiOThmMGQ2ZDQ0Ljc0YWM3NjcxLWYyZTItNDc1ZC05YWMxLTRhYjEyMjA5YjgzM3gxNzkxNTM5ODY2MDYwNjc5MDAw niks3-go-unit-tests> 2026/10/09 09:57:46 INFO Aborted multipart uploads count=0 kept=0 niks3-go-unit-tests> 2026/10/09 09:57:46 WARN Force mode enabled - objects will be deleted immediately without grace period niks3-go-unit-tests> 2026/10/09 09:57:46 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=0 niks3-go-unit-tests> 2026/10/09 09:57:46 WARN Failed to abort multipart upload key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=MTE5MjdlYzItNjE3OS00ZDA5LTljZjMtZGFiOThmMGQ2ZDQ0Ljc0YWM3NjcxLWYyZTItNDc1ZC05YWMxLTRhYjEyMjA5YjgzM3gxNzkxNTM5ODY2MDYwNjc5MDAw error="Delete \"http://localhost:60965/bucket27/nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst?uploadId=MTE5MjdlYzItNjE3OS00ZDA5LTljZjMtZGFiOThmMGQ2ZDQ0Ljc0YWM3NjcxLWYyZTItNDc1ZC05YWMxLTRhYjEyMjA5YjgzM3gxNzkxNTM5ODY2MDYwNjc5MDAw\": injected: abort refused" niks3-go-unit-tests> 2026/10/09 09:57:46 INFO Received complete multipart upload request method=POST path=/api/multipart/complete niks3-go-unit-tests> 2026/10/09 09:57:46 INFO Vacuumed table table=pending_closures niks3-go-unit-tests> 2026/10/09 09:57:46 WARN CompleteMultipartUpload errored but object exists; treating as success error="The specified key does not exist." object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=MTE5MjdlYzItNjE3OS00ZDA5LTljZjMtZGFiOThmMGQ2ZDQ0Ljc0YWM3NjcxLWYyZTItNDc1ZC05YWMxLTRhYjEyMjA5YjgzM3gxNzkxNTM5ODY2MDYwNjc5MDAw niks3-go-unit-tests> 2026/10/09 09:57:46 INFO Vacuumed table table=pending_objects niks3-go-unit-tests> 2026/10/09 09:57:46 INFO Vacuumed table table=multipart_uploads niks3-go-unit-tests> 2026/10/09 09:57:46 INFO Vacuumed table table=closures niks3-go-unit-tests> 2026/10/09 09:57:46 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=MTE5MjdlYzItNjE3OS00ZDA5LTljZjMtZGFiOThmMGQ2ZDQ0Ljc0YWM3NjcxLWYyZTItNDc1ZC05YWMxLTRhYjEyMjA5YjgzM3gxNzkxNTM5ODY2MDYwNjc5MDAw parts=1 niks3-go-unit-tests> 2026/10/09 09:57:46 INFO Vacuumed table table=objects niks3-go-unit-tests> --- PASS: TestCompleteMultipartUpload_ErrorButObjectExists (8.16s) niks3-go-unit-tests> === CONT TestService_ReadAuthMiddleware niks3-go-unit-tests> 2026/10/09 09:57:46 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> --- PASS: TestGCSweepSkipsPendingObjects (7.69s) niks3-go-unit-tests> === CONT TestCacheConfigHandler niks3-go-unit-tests> === RUN TestCacheConfigHandler/full_config,_no_issuer niks3-go-unit-tests> === PAUSE TestCacheConfigHandler/full_config,_no_issuer niks3-go-unit-tests> === RUN TestCacheConfigHandler/no_cache_url_configured niks3-go-unit-tests> === PAUSE TestCacheConfigHandler/no_cache_url_configured niks3-go-unit-tests> === RUN TestCacheConfigHandler/no_signing_keys niks3-go-unit-tests> === PAUSE TestCacheConfigHandler/no_signing_keys niks3-go-unit-tests> === RUN TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator niks3-go-unit-tests> === PAUSE TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator niks3-go-unit-tests> === CONT TestClientWithDependencies niks3-go-unit-tests> 2026/10/09 09:57:46 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> 2026/10/09 09:57:47 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> 2026/10/09 09:57:47 INFO Received push request method=POST path=/api/pushes niks3-go-unit-tests> 2026/10/09 09:57:47 INFO Aborted multipart uploads count=1 kept=0 niks3-go-unit-tests> 2026/10/09 09:57:47 INFO Received cleanup request method=DELETE path=/api/pending_closures niks3-go-unit-tests> 2026/10/09 09:57:47 INFO Aborted multipart uploads count=1 kept=0 niks3-go-unit-tests> --- PASS: TestPendingCleanupUsesOneCutoff (9.40s) niks3-go-unit-tests> === CONT TestService_RequireScope_OIDC niks3-go-unit-tests> 2026/10/09 09:57:47 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:61409/oidc niks3-go-unit-tests> 2026/10/09 09:57:47 WARN mTLS auth: subject not in bound subjects subject="CN=reader" niks3-go-unit-tests> 2026/10/09 09:57:47 WARN mTLS auth: subject not in bound subjects subject="CN=reader" niks3-go-unit-tests> --- PASS: TestService_NativeMTLS (6.65s) niks3-go-unit-tests> === CONT TestService_AuthMiddleware_OIDC niks3-go-unit-tests> 2026/10/09 09:57:48 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:61413/oidc niks3-go-unit-tests> 2026/10/09 09:57:48 INFO Received push request method=POST path=/api/pushes niks3-go-unit-tests> 2026/10/09 09:57:48 INFO Received push request method=POST path=/api/pushes niks3-go-unit-tests> 2026/10/09 09:57:49 INFO Received complete multipart upload request method=POST path=/api/multipart/complete niks3-go-unit-tests> 2026/10/09 09:57:49 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0100000000000000000000.nar.zst upload_id=MTE5MjdlYzItNjE3OS00ZDA5LTljZjMtZGFiOThmMGQ2ZDQ0LmM3ZGFmNWU3LTMyNjgtNDkyZS05OGQzLTg0NTgyZGVmNmRmY3gxNzkxNTM5ODY2NzkxODcwMDAw parts=12 niks3-go-unit-tests> 2026/10/09 09:57:49 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> 2026/10/09 09:57:49 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> 2026/10/09 09:57:49 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> 2026/10/09 09:57:49 INFO Received complete multipart upload request method=POST path=/api/multipart/complete niks3-go-unit-tests> 2026/10/09 09:57:49 INFO Aborted multipart uploads count=0 kept=0 niks3-go-unit-tests> 2026/10/09 09:57:49 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=0 niks3-go-unit-tests> 2026/10/09 09:57:49 INFO Completed multipart upload object_key=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhh0100000000000000000000.nar.zst upload_id=MTE5MjdlYzItNjE3OS00ZDA5LTljZjMtZGFiOThmMGQ2ZDQ0LjBhMzRkMzQxLTdlNTgtNDcxMC1hMjc1LTM4ZDBjNGY4MmEzYXgxNzkxNTM5ODY3MTI1NDk1MDAw parts=12 niks3-go-unit-tests> 2026/10/09 09:57:49 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> 2026/10/09 09:57:49 INFO Received push request method=POST path=/api/pushes niks3-go-unit-tests> 2026/10/09 09:57:49 INFO Vacuumed table table=pending_closures niks3-go-unit-tests> 2026/10/09 09:57:49 INFO Received complete multipart upload request method=POST path=/api/multipart/complete niks3-go-unit-tests> --- PASS: TestCompletedNarNotReofferedAcrossClosures (9.90s) niks3-go-unit-tests> === CONT TestService_ReadScope_PublicByDefault niks3-go-unit-tests> 2026/10/09 09:57:49 INFO Vacuumed table table=pending_objects niks3-go-unit-tests> 2026/10/09 09:57:49 INFO Vacuumed table table=multipart_uploads niks3-go-unit-tests> 2026/10/09 09:57:50 INFO Vacuumed table table=closures niks3-go-unit-tests> 2026/10/09 09:57:50 INFO Vacuumed table table=objects niks3-go-unit-tests> 2026/10/09 09:57:50 INFO Completed multipart upload object_key=nar/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa00000000000000000000.nar.zst upload_id=MTE5MjdlYzItNjE3OS00ZDA5LTljZjMtZGFiOThmMGQ2ZDQ0Ljk1OTdlNGI1LTg5YTEtNDFhMi1hODlkLTlmMzk1NDBlNTIxM3gxNzkxNTM5ODY3NTEzOTMwMDAw parts=10 niks3-go-unit-tests> 2026/10/09 09:57:50 INFO Aborted multipart uploads count=0 kept=0 niks3-go-unit-tests> 2026/10/09 09:57:50 WARN Force mode enabled - objects will be deleted immediately without grace period niks3-go-unit-tests> 2026/10/09 09:57:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=0 niks3-go-unit-tests> 2026/10/09 09:57:50 INFO Vacuumed table table=pending_closures niks3-go-unit-tests> 2026/10/09 09:57:50 INFO Vacuumed table table=pending_objects niks3-go-unit-tests> 2026/10/09 09:57:50 INFO Vacuumed table table=multipart_uploads niks3-go-unit-tests> 2026/10/09 09:57:50 INFO Vacuumed table table=closures niks3-go-unit-tests> 2026/10/09 09:57:50 INFO Vacuumed table table=objects niks3-go-unit-tests> --- PASS: TestPushDedupSurvivesConcurrentGC (8.12s) niks3-go-unit-tests> === CONT TestService_AuthMiddleware_MTLSBoundSubjects niks3-go-unit-tests> 2026/10/09 09:57:50 INFO Received push request method=POST path=/api/pushes niks3-go-unit-tests> === RUN TestPush_RejectsBadRequests/root_not_in_objects niks3-go-unit-tests> === PAUSE TestPush_RejectsBadRequests/root_not_in_objects niks3-go-unit-tests> === RUN TestPush_RejectsBadRequests/no_roots niks3-go-unit-tests> === PAUSE TestPush_RejectsBadRequests/no_roots niks3-go-unit-tests> === RUN TestPush_RejectsBadRequests/no_objects niks3-go-unit-tests> === PAUSE TestPush_RejectsBadRequests/no_objects niks3-go-unit-tests> === RUN TestPush_RejectsBadRequests/bad_root niks3-go-unit-tests> === PAUSE TestPush_RejectsBadRequests/bad_root niks3-go-unit-tests> === CONT TestReadProxyNarinfo niks3-go-unit-tests> 2026/10/09 09:57:50 INFO Received complete multipart upload request method=POST path=/api/multipart/complete niks3-go-unit-tests> 2026/10/09 09:57:50 INFO Completed multipart upload object_key=nar/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb00000000000000000000.nar.zst upload_id=MTE5MjdlYzItNjE3OS00ZDA5LTljZjMtZGFiOThmMGQ2ZDQ0LmVjY2UyZDhiLThkMGEtNDRkYS1hNzc1LWI3YmI4ZjJhOWYyNXgxNzkxNTM5ODY4NTAxODIwMDAw parts=10 niks3-go-unit-tests> 2026/10/09 09:57:51 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> 2026-10-09 09:57:51.305 UTC [65892] ERROR: Push does not exist: id=1 niks3-go-unit-tests> 2026-10-09 09:57:51.305 UTC [65892] CONTEXT: PL/pgSQL function commit_push(bigint) line 11 at RAISE niks3-go-unit-tests> 2026-10-09 09:57:51.305 UTC [65892] STATEMENT: SELECT commit_push($1) niks3-go-unit-tests> --- PASS: TestPush_CompleteWaitsForCleanupOfThePush (9.46s) niks3-go-unit-tests> === CONT TestReadProxyRangeRequest niks3-go-unit-tests> 2026/10/09 09:57:51 INFO Received complete multipart upload request method=POST path=/api/multipart/complete niks3-go-unit-tests> 2026/10/09 09:57:51 INFO Received cleanup request method=DELETE path=/api/pending_closures niks3-go-unit-tests> 2026/10/09 09:57:51 WARN Failed to abort upload, keeping its closure key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst error="Get \"http://127.0.0.1:1/bucket39/?location=\": dial tcp 127.0.0.1:1: connect: connection refused" code="" niks3-go-unit-tests> 2026/10/09 09:57:51 INFO Aborted multipart uploads count=0 kept=1 niks3-go-unit-tests> 2026/10/09 09:57:51 INFO Completed multipart upload object_key=nar/ffffffffffffffffffffffffffffffff00000000000000000000.nar.zst upload_id=MTE5MjdlYzItNjE3OS00ZDA5LTljZjMtZGFiOThmMGQ2ZDQ0LjBjNzg3Y2IzLWQ4YzctNGUzZi1hZDZlLThjMzViOTMxNDUzYngxNzkxNTM5ODY4OTQ4MTkzMDAw parts=10 niks3-go-unit-tests> 2026/10/09 09:57:51 INFO Received cleanup request method=DELETE path=/api/pending_closures niks3-go-unit-tests> 2026/10/09 09:57:51 INFO Aborted multipart uploads count=1 kept=0 niks3-go-unit-tests> --- PASS: TestMultipartCleanup (8.94s) niks3-go-unit-tests> === CONT TestReadRedirectNar niks3-go-unit-tests> --- PASS: TestReadRedirectUsesPublicS3URL (8.63s) niks3-go-unit-tests> === CONT TestReadProxyRootRedirectsToIndexHTML niks3-go-unit-tests> 2026/10/09 09:57:52 INFO Received push request method=POST path=/api/pushes niks3-go-unit-tests> 2026/10/09 09:57:52 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign niks3-go-unit-tests> 2026/10/09 09:57:52 INFO Signed narinfos id=1 count=1 niks3-go-unit-tests> 2026/10/09 09:57:52 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign niks3-go-unit-tests> 2026/10/09 09:57:52 INFO Received complete multipart upload request method=POST path=/api/multipart/complete niks3-go-unit-tests> 2026/10/09 09:57:52 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign niks3-go-unit-tests> 2026/10/09 09:57:52 ERROR Failed to sign narinfos signer="index 1" error="signing key cannot be nil" niks3-go-unit-tests> 2026/10/09 09:57:52 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign niks3-go-unit-tests> 2026/10/09 09:57:52 ERROR Failed to sign narinfos signer=niks3-test-1 error="signer failed" niks3-go-unit-tests> 2026/10/09 09:57:52 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign niks3-go-unit-tests> 2026/10/09 09:57:52 ERROR Signer returned unexpected signature count signer=niks3-test-1 got=0 want=1 niks3-go-unit-tests> 2026/10/09 09:57:52 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign niks3-go-unit-tests> 2026/10/09 09:57:52 ERROR Signer returned unexpected object key signer=niks3-test-1 object_key=unexpected.narinfo niks3-go-unit-tests> 2026/10/09 09:57:52 INFO Received complete multipart upload request method=POST path=/api/multipart/complete niks3-go-unit-tests> --- PASS: TestPush_SignsNarinfosOfItsPendingObjects (8.93s) niks3-go-unit-tests> === CONT TestReadProxyConditionalGet niks3-go-unit-tests> 2026/10/09 09:57:52 INFO Completed multipart upload object_key=nar/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa00000000000000000000.nar.zst upload_id=MTE5MjdlYzItNjE3OS00ZDA5LTljZjMtZGFiOThmMGQ2ZDQ0LjJmYmYwMWVmLTJjMzEtNDdlZi1iZmRmLTE4Yjk3Zjg4ZDc3YngxNzkxNTM5ODY5OTQ3NjIwMDAw parts=10 niks3-go-unit-tests> 2026/10/09 09:57:52 INFO Received complete multipart upload request method=POST path=/api/multipart/complete niks3-go-unit-tests> 2026/10/09 09:57:52 INFO Completed multipart upload object_key=nar/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb00000000000000000000.nar.zst upload_id=MTE5MjdlYzItNjE3OS00ZDA5LTljZjMtZGFiOThmMGQ2ZDQ0LmU2ZTcxNTY2LWRiNzYtNDEwOC1hYzY3LThiN2JmYjdhZjRmMXgxNzkxNTM5ODY3NTEzNTM2MDAw parts=10 niks3-go-unit-tests> 2026/10/09 09:57:52 INFO Received complete push request method=POST path=/api/pushes/1/complete niks3-go-unit-tests> 2026/10/09 09:57:52 WARN Failed to abort redundant multipart upload, keeping its row object_key=nar/gggggggggggggggggggggggggggggg0200000000000000000000.nar.zst upload_id=MTE5MjdlYzItNjE3OS00ZDA5LTljZjMtZGFiOThmMGQ2ZDQ0LjNiOTc0MTI4LThlZmMtNDU5My1iYTk0LWZiNWQxYzhhMDUxNngxNzkxNTM5ODY5NzAwNDE4MDAw error="Get \"http://127.0.0.1:1/bucket29/?location=\": dial tcp 127.0.0.1:1: connect: connection refused" niks3-go-unit-tests> 2026/10/09 09:57:52 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0200000000000000000000.nar.zst upload_id=MTE5MjdlYzItNjE3OS00ZDA5LTljZjMtZGFiOThmMGQ2ZDQ0LjYxMDllNGU4LWU1YzYtNDRlOS04MDYwLWNhN2VjMzkzNzlhMXgxNzkxNTM5ODY5NjAyNzc0MDAw parts=12 niks3-go-unit-tests> 2026/10/09 09:57:52 INFO Received cleanup request method=DELETE path=/api/pending_closures niks3-go-unit-tests> 2026/10/09 09:57:52 INFO Completed push id=1 objects=4 duration=21.732ms niks3-go-unit-tests> 2026/10/09 09:57:52 INFO Received push request method=POST path=/api/pushes niks3-go-unit-tests> 2026/10/09 09:57:52 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> 2026/10/09 09:57:52 INFO Aborted multipart uploads count=1 kept=0 niks3-go-unit-tests> --- PASS: TestRedundantMultipartUpload (12.53s) niks3-go-unit-tests> === CONT TestReadProxyOutlastsServerWriteTimeout niks3-go-unit-tests> --- PASS: TestDeduplicatedObjectsRecordedAsPending (8.64s) niks3-go-unit-tests> === CONT TestClientIntegration niks3-go-unit-tests> 2026/10/09 09:57:52 INFO Received complete multipart upload request method=POST path=/api/multipart/complete niks3-go-unit-tests> 2026/10/09 09:57:52 INFO Aborted multipart uploads count=0 kept=0 niks3-go-unit-tests> 2026/10/09 09:57:52 WARN Force mode enabled - objects will be deleted immediately without grace period niks3-go-unit-tests> 2026/10/09 09:57:53 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=2 objects-deleted-after-grace-period=2 objects-failed-to-delete=0 niks3-go-unit-tests> 2026/10/09 09:57:53 INFO Completed multipart upload object_key=nar/cccccccccccccccccccccccccccccccc00000000000000000000.nar.zst upload_id=MTE5MjdlYzItNjE3OS00ZDA5LTljZjMtZGFiOThmMGQ2ZDQ0LmIxODRmNDRkLWE4NDYtNGMwYi1iZDRiLWEyMzc4NjdlZjY2ZngxNzkxNTM5ODcwNDAyMDY5MDAw parts=10 niks3-go-unit-tests> 2026/10/09 09:57:53 INFO Vacuumed table table=pending_closures niks3-go-unit-tests> 2026/10/09 09:57:53 INFO Vacuumed table table=pending_objects niks3-go-unit-tests> 2026/10/09 09:57:53 INFO Vacuumed table table=multipart_uploads niks3-go-unit-tests> 2026/10/09 09:57:53 INFO Vacuumed table table=closures niks3-go-unit-tests> 2026/10/09 09:57:53 INFO Vacuumed table table=objects niks3-go-unit-tests> --- PASS: TestGCBugBareHashReferences (9.08s) niks3-go-unit-tests> === CONT TestReadProxyDisabled niks3-go-unit-tests> 2026/10/09 09:57:54 INFO Received complete multipart upload request method=POST path=/api/multipart/complete niks3-go-unit-tests> 2026/10/09 09:57:54 INFO Completed multipart upload object_key=nar/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa00000000000000000000.nar.zst upload_id=MTE5MjdlYzItNjE3OS00ZDA5LTljZjMtZGFiOThmMGQ2ZDQ0LjZhZDFmZDM0LWMyYTYtNDU0OS04OGM1LWNlMGIwNTNhMjM5ZngxNzkxNTM5ODY4OTQ3NTA4MDAw parts=10 niks3-go-unit-tests> 2026/10/09 09:57:54 INFO lead: acquired remote=192.0.2.1:1234 niks3-go-unit-tests> === NAME TestNARDeduplicationMetadataUploadBug niks3-go-unit-tests> metadata_upload_test.go:48: First store path: /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestNARDeduplicationMetadataUploadBug3739036377/001/store/jzdkkpjzngbd8hh4z6ilczfxk39vaky0-file1.txt niks3-go-unit-tests> 2026/10/09 09:57:54 INFO Received push request method=POST path=/api/pushes niks3-go-unit-tests> 2026/10/09 09:57:54 INFO Uploading 1 paths to 127.0.0.1 (0 already cached) niks3-go-unit-tests> 2026/10/09 09:57:54 INFO Uploading jzdkkpjzngbd8hh4z6ilczfxk39vaky0-file1.txt (160B) niks3-go-unit-tests> 2026/10/09 09:57:54 WARN Failed to register uploaded object key=jzdkkpjzngbd8hh4z6ilczfxk39vaky0.ls error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:57:54 WARN Failed to register uploaded object key=nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:57:54 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign niks3-go-unit-tests> 2026/10/09 09:57:54 INFO Signed narinfos id=1 count=1 niks3-go-unit-tests> 2026/10/09 09:57:54 INFO Uploading 1 narinfos niks3-go-unit-tests> 2026/10/09 09:57:54 INFO Received complete push request method=POST path=/api/pushes/1/complete niks3-go-unit-tests> 2026/10/09 09:57:54 WARN Failed to register uploaded object key=jzdkkpjzngbd8hh4z6ilczfxk39vaky0.narinfo error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:57:54 INFO Received complete multipart upload request method=POST path=/api/multipart/complete niks3-go-unit-tests> 2026/10/09 09:57:54 INFO Completed push id=1 objects=3 duration=96.959333ms niks3-go-unit-tests> 2026/10/09 09:57:54 INFO Upload complete. (369ms) niks3-go-unit-tests> metadata_upload_test.go:54: Retrieved narinfo from S3: niks3-go-unit-tests> StorePath: /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestNARDeduplicationMetadataUploadBug3739036377/001/store/jzdkkpjzngbd8hh4z6ilczfxk39vaky0-file1.txt niks3-go-unit-tests> URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst niks3-go-unit-tests> 2026/10/09 09:57:54 INFO Completed multipart upload object_key=nar/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb00000000000000000000.nar.zst upload_id=MTE5MjdlYzItNjE3OS00ZDA5LTljZjMtZGFiOThmMGQ2ZDQ0LmZiNjRiN2Q3LWJkMTAtNGViYi1iM2FkLWE3MjA4MjJjNWI5MXgxNzkxNTM5ODY5OTQ4MTIxMDAw parts=10 niks3-go-unit-tests> Compression: zstd niks3-go-unit-tests> 2026/10/09 09:57:54 INFO Received complete push request method=POST path=/api/pushes/1/complete niks3-go-unit-tests> NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf niks3-go-unit-tests> NarSize: 160 niks3-go-unit-tests> References: niks3-go-unit-tests> CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf niks3-go-unit-tests> metadata_upload_test.go:55: Retrieved .ls file from S3 (compressed size: 77 bytes) niks3-go-unit-tests> metadata_upload_test.go:55: Decompressed .ls content (64 bytes): niks3-go-unit-tests> {"version":1,"root":{"type":"regular","size":44,"narOffset":96}} niks3-go-unit-tests> 2026/10/09 09:57:54 INFO Completed push id=1 objects=4 duration=28.881167ms niks3-go-unit-tests> 2026/10/09 09:57:54 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete niks3-go-unit-tests> 2026/10/09 09:57:54 INFO Completed upload id=2 objects=2 duration=945.917µs niks3-go-unit-tests> --- PASS: TestMetricsCommit (12.56s) niks3-go-unit-tests> === CONT TestCreatePendingClosureRejectsOversizedNAR niks3-go-unit-tests> 2026/10/09 09:57:54 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> --- PASS: TestCreatePendingClosureRejectsOversizedNAR (0.01s) niks3-go-unit-tests> === CONT TestClientSharedPathCommittedMidPush niks3-go-unit-tests> === NAME TestNARDeduplicationMetadataUploadBug niks3-go-unit-tests> metadata_upload_test.go:64: Second store path (same content): /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestNARDeduplicationMetadataUploadBug3739036377/001/store/n0kh57iv90pnvqsnx7idg3b3c4327ips-file2.txt niks3-go-unit-tests> 2026/10/09 09:57:55 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> 2026/10/09 09:57:55 INFO Received push request method=POST path=/api/pushes niks3-go-unit-tests> 2026/10/09 09:57:55 INFO Uploading 0 paths to 127.0.0.1 (1 already cached) niks3-go-unit-tests> 2026/10/09 09:57:55 WARN Failed to register uploaded object key=n0kh57iv90pnvqsnx7idg3b3c4327ips.ls error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:57:55 INFO Received sign narinfos request method=POST path=/api/pushes/2/sign niks3-go-unit-tests> 2026/10/09 09:57:55 INFO Signed narinfos id=2 count=1 niks3-go-unit-tests> 2026/10/09 09:57:55 INFO Uploading 1 narinfos niks3-go-unit-tests> 2026/10/09 09:57:55 INFO Received complete push request method=POST path=/api/pushes/2/complete niks3-go-unit-tests> 2026/10/09 09:57:55 WARN Failed to register uploaded object key=n0kh57iv90pnvqsnx7idg3b3c4327ips.narinfo error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:57:55 INFO Completed push id=2 objects=3 duration=833.5µs niks3-go-unit-tests> 2026/10/09 09:57:55 INFO Upload complete. (204ms) niks3-go-unit-tests> metadata_upload_test.go:76: Retrieved narinfo from S3: niks3-go-unit-tests> StorePath: /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestNARDeduplicationMetadataUploadBug3739036377/001/store/n0kh57iv90pnvqsnx7idg3b3c4327ips-file2.txt niks3-go-unit-tests> URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst niks3-go-unit-tests> Compression: zstd niks3-go-unit-tests> NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf niks3-go-unit-tests> NarSize: 160 niks3-go-unit-tests> References: niks3-go-unit-tests> CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf niks3-go-unit-tests> metadata_upload_test.go:77: Retrieved .ls file from S3 (compressed size: 62 bytes) niks3-go-unit-tests> metadata_upload_test.go:77: Decompressed .ls content (49 bytes): niks3-go-unit-tests> {"version":1,"root":{"type":"regular","size":44}} niks3-go-unit-tests> 2026/10/09 09:57:55 INFO lead: released remote=192.0.2.1:1234 niks3-go-unit-tests> --- PASS: TestLeadEndsOnShutdown (10.40s) niks3-go-unit-tests> 2026/10/09 09:57:55 INFO Received complete multipart upload request method=POST path=/api/multipart/complete niks3-go-unit-tests> === CONT TestReadProxyHead niks3-go-unit-tests> 2026/10/09 09:57:55 INFO Completed multipart upload object_key=nar/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa00000000000000000000.nar.zst upload_id=MTE5MjdlYzItNjE3OS00ZDA5LTljZjMtZGFiOThmMGQ2ZDQ0LmUwZWY4NjRiLTAyMGYtNDdhNC04NzY5LTVmOGExZjYyMWMxMXgxNzkxNTM5ODcwNDAyNDM1MDAw parts=10 niks3-go-unit-tests> --- PASS: TestNARDeduplicationMetadataUploadBug (10.98s) niks3-go-unit-tests> === CONT TestReadProxyInvalidPath niks3-go-unit-tests> 2026/10/09 09:57:55 INFO Received complete multipart upload request method=POST path=/api/multipart/complete niks3-go-unit-tests> 2026/10/09 09:57:55 INFO Completed multipart upload object_key=nar/cccccccccccccccccccccccccccccccc00000000000000000000.nar.zst upload_id=MTE5MjdlYzItNjE3OS00ZDA5LTljZjMtZGFiOThmMGQ2ZDQ0LjdlZmZmMzBiLWI2MjgtNGY5ZC1hZjI0LTFlODY4MTMyOTFlOXgxNzkxNTM5ODcyNzk0MjAxMDAw parts=10 niks3-go-unit-tests> 2026/10/09 09:57:55 INFO Received complete push request method=POST path=/api/pushes/2/complete niks3-go-unit-tests> 2026/10/09 09:57:55 INFO Received complete multipart upload request method=POST path=/api/multipart/complete niks3-go-unit-tests> 2026/10/09 09:57:55 INFO Completed push id=2 objects=4 duration=91.808417ms niks3-go-unit-tests> --- PASS: TestPush_SkippedKeySurvivesGCBeforeCommit (15.00s) niks3-go-unit-tests> === CONT TestPinProtectsFromGC niks3-go-unit-tests> --- PASS: TestService_ReadAuthMiddleware (9.34s) niks3-go-unit-tests> === CONT TestReadProxyContentNegotiation niks3-go-unit-tests> 2026/10/09 09:57:56 INFO Received complete multipart upload request method=POST path=/api/multipart/complete niks3-go-unit-tests> 2026/10/09 09:57:56 INFO Completed multipart upload object_key=nar/dddddddddddddddddddddddddddddddd00000000000000000000.nar.zst upload_id=MTE5MjdlYzItNjE3OS00ZDA5LTljZjMtZGFiOThmMGQ2ZDQ0LmVmOTViZWM0LTIzNzctNGFjYy1hMDdiLTE5MDk4ZmJiNDU2NXgxNzkxNTM5ODY4OTQ4MDg4MDAw parts=10 niks3-go-unit-tests> 2026/10/09 09:57:56 INFO Received push request method=POST path=/api/pushes niks3-go-unit-tests> 2026/10/09 09:57:56 INFO Uploading 2 paths to 127.0.0.1 (1 already cached) niks3-go-unit-tests> 2026/10/09 09:57:56 INFO Uploading xbqmkgdshgk5k0xjml4m1g56lpmw4j8d-b (256B) niks3-go-unit-tests> 2026/10/09 09:57:56 INFO Uploading 53ks2w0ysjv96vmx4rpi1al5gp75yamj-shared-dep (136B) niks3-go-unit-tests> 2026/10/09 09:57:56 WARN Failed to register uploaded object key=vhryrfv773fksmia7iy65577br00fhfm.ls error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:57:56 WARN Failed to register uploaded object key=nar/0hqznlhggmdhn8q1378c9hj7qi8v35l5g0yayi0xbvp8zc6wn5xl.nar.zst error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:57:56 WARN Failed to register uploaded object key=xbqmkgdshgk5k0xjml4m1g56lpmw4j8d.ls error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:57:56 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:57:56 WARN Failed to register uploaded object key=53ks2w0ysjv96vmx4rpi1al5gp75yamj.ls error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:57:56 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign niks3-go-unit-tests> 2026/10/09 09:57:56 INFO Signed narinfos id=1 count=3 niks3-go-unit-tests> 2026/10/09 09:57:56 INFO Uploading 3 narinfos niks3-go-unit-tests> 2026/10/09 09:57:56 WARN Failed to register uploaded object key=vhryrfv773fksmia7iy65577br00fhfm.narinfo error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:57:56 WARN Failed to register uploaded object key=53ks2w0ysjv96vmx4rpi1al5gp75yamj.narinfo error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:57:56 INFO Received complete push request method=POST path=/api/pushes/1/complete niks3-go-unit-tests> 2026/10/09 09:57:56 WARN Failed to register uploaded object key=xbqmkgdshgk5k0xjml4m1g56lpmw4j8d.narinfo error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:57:56 INFO Completed push id=1 objects=8 duration=133.390625ms niks3-go-unit-tests> 2026/10/09 09:57:56 INFO Upload complete. (383ms) niks3-go-unit-tests> === NAME TestClientPushesUseOnePush niks3-go-unit-tests> client_pushes_test.go:97: Retrieved narinfo from S3: niks3-go-unit-tests> StorePath: /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestClientPushesUseOnePush2694814707/001/store/53ks2w0ysjv96vmx4rpi1al5gp75yamj-shared-dep niks3-go-unit-tests> URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst niks3-go-unit-tests> Compression: zstd niks3-go-unit-tests> NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82 niks3-go-unit-tests> NarSize: 136 niks3-go-unit-tests> References: niks3-go-unit-tests> CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n niks3-go-unit-tests> client_pushes_test.go:97: Retrieved narinfo from S3: niks3-go-unit-tests> StorePath: /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestClientPushesUseOnePush2694814707/001/store/vhryrfv773fksmia7iy65577br00fhfm-a niks3-go-unit-tests> URL: nar/0hqznlhggmdhn8q1378c9hj7qi8v35l5g0yayi0xbvp8zc6wn5xl.nar.zst niks3-go-unit-tests> Compression: zstd niks3-go-unit-tests> NarHash: sha256:0hqznlhggmdhn8q1378c9hj7qi8v35l5g0yayi0xbvp8zc6wn5xl niks3-go-unit-tests> NarSize: 256 niks3-go-unit-tests> References: /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestClientPushesUseOnePush2694814707/001/store/53ks2w0ysjv96vmx4rpi1al5gp75yamj-shared-dep niks3-go-unit-tests> CA: text:sha256:1yxkka82di6scx68lcanr9wjmx53w6lw2ssjns8j14j9hyjmw4qa niks3-go-unit-tests> client_pushes_test.go:97: Retrieved narinfo from S3: niks3-go-unit-tests> StorePath: /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestClientPushesUseOnePush2694814707/001/store/xbqmkgdshgk5k0xjml4m1g56lpmw4j8d-b niks3-go-unit-tests> URL: nar/0hqznlhggmdhn8q1378c9hj7qi8v35l5g0yayi0xbvp8zc6wn5xl.nar.zst niks3-go-unit-tests> Compression: zstd niks3-go-unit-tests> NarHash: sha256:0hqznlhggmdhn8q1378c9hj7qi8v35l5g0yayi0xbvp8zc6wn5xl niks3-go-unit-tests> NarSize: 256 niks3-go-unit-tests> References: /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestClientPushesUseOnePush2694814707/001/store/53ks2w0ysjv96vmx4rpi1al5gp75yamj-shared-dep niks3-go-unit-tests> CA: text:sha256:1yxkka82di6scx68lcanr9wjmx53w6lw2ssjns8j14j9hyjmw4qa niks3-go-unit-tests> --- PASS: TestClientPushesUseOnePush (10.99s) niks3-go-unit-tests> === CONT TestReadProxyNarinfoAlreadyDecompressed niks3-go-unit-tests> === RUN TestService_AuthMiddleware_OIDC/valid_OIDC_token niks3-go-unit-tests> === PAUSE TestService_AuthMiddleware_OIDC/valid_OIDC_token niks3-go-unit-tests> === RUN TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected niks3-go-unit-tests> === PAUSE TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected niks3-go-unit-tests> === RUN TestService_AuthMiddleware_OIDC/malformed_token_rejected niks3-go-unit-tests> === PAUSE TestService_AuthMiddleware_OIDC/malformed_token_rejected niks3-go-unit-tests> === RUN TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured niks3-go-unit-tests> === PAUSE TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured niks3-go-unit-tests> === CONT TestGCMetrics niks3-go-unit-tests> === NAME TestClientWithDependencies niks3-go-unit-tests> client_integration_test.go:735: Built derivation: /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestClientWithDependencies4176572669/001/store/irf7apapgkmyry2z90ashdg40g27dhib-test-script niks3-go-unit-tests> client_integration_test.go:737: Found 1 dependencies (including self) niks3-go-unit-tests> === RUN TestService_RequireScope_OIDC/builder_may_write niks3-go-unit-tests> === PAUSE TestService_RequireScope_OIDC/builder_may_write niks3-go-unit-tests> === RUN TestService_RequireScope_OIDC/builder_may_not_admin niks3-go-unit-tests> === PAUSE TestService_RequireScope_OIDC/builder_may_not_admin niks3-go-unit-tests> === RUN TestService_RequireScope_OIDC/ops_may_admin niks3-go-unit-tests> === PAUSE TestService_RequireScope_OIDC/ops_may_admin niks3-go-unit-tests> === RUN TestService_RequireScope_OIDC/ops_may_not_write niks3-go-unit-tests> === PAUSE TestService_RequireScope_OIDC/ops_may_not_write niks3-go-unit-tests> === RUN TestService_RequireScope_OIDC/reader_may_not_write niks3-go-unit-tests> === PAUSE TestService_RequireScope_OIDC/reader_may_not_write niks3-go-unit-tests> === RUN TestService_RequireScope_OIDC/static_token_may_admin niks3-go-unit-tests> === PAUSE TestService_RequireScope_OIDC/static_token_may_admin niks3-go-unit-tests> === RUN TestService_RequireScope_OIDC/static_token_may_write niks3-go-unit-tests> === PAUSE TestService_RequireScope_OIDC/static_token_may_write niks3-go-unit-tests> === RUN TestService_RequireScope_OIDC/reader_may_read niks3-go-unit-tests> === PAUSE TestService_RequireScope_OIDC/reader_may_read niks3-go-unit-tests> === RUN TestService_RequireScope_OIDC/writer_implies_read niks3-go-unit-tests> === PAUSE TestService_RequireScope_OIDC/writer_implies_read niks3-go-unit-tests> === RUN TestService_RequireScope_OIDC/anonymous_may_not_read niks3-go-unit-tests> === PAUSE TestService_RequireScope_OIDC/anonymous_may_not_read niks3-go-unit-tests> === RUN TestService_RequireScope_OIDC/reader_may_read_with_netrc niks3-go-unit-tests> === PAUSE TestService_RequireScope_OIDC/reader_may_read_with_netrc niks3-go-unit-tests> === RUN TestService_RequireScope_OIDC/static_token_may_admin_with_netrc niks3-go-unit-tests> === PAUSE TestService_RequireScope_OIDC/static_token_may_admin_with_netrc niks3-go-unit-tests> === RUN TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read niks3-go-unit-tests> === PAUSE TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read niks3-go-unit-tests> === RUN TestService_RequireScope_OIDC/empty_netrc_password_may_not_read niks3-go-unit-tests> === PAUSE TestService_RequireScope_OIDC/empty_netrc_password_may_not_read niks3-go-unit-tests> === CONT TestClientMultipleUploads niks3-go-unit-tests> 2026/10/09 09:57:57 INFO Received push request method=POST path=/api/pushes niks3-go-unit-tests> 2026/10/09 09:57:57 INFO Received complete multipart upload request method=POST path=/api/multipart/complete niks3-go-unit-tests> 2026/10/09 09:57:57 INFO Uploading 1 paths to 127.0.0.1 (0 already cached) niks3-go-unit-tests> 2026/10/09 09:57:57 INFO Uploading irf7apapgkmyry2z90ashdg40g27dhib-test-script (136B) niks3-go-unit-tests> 2026/10/09 09:57:57 INFO Completed multipart upload object_key=nar/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb00000000000000000000.nar.zst upload_id=MTE5MjdlYzItNjE3OS00ZDA5LTljZjMtZGFiOThmMGQ2ZDQ0LjIzNmU2NTc4LWVkN2UtNGY1Ni1hN2ZjLTU5ZTY0MzQ1NDUxM3gxNzkxNTM5ODcwNDAyNDc5MDAw parts=10 niks3-go-unit-tests> 2026/10/09 09:57:57 INFO Received complete push request method=POST path=/api/pushes/1/complete niks3-go-unit-tests> 2026/10/09 09:57:57 INFO Completed push id=1 objects=6 duration=22.64675ms niks3-go-unit-tests> 2026/10/09 09:57:57 WARN Failed to register uploaded object key=nar/1s7yghixzpinyqm7g59kyn3sav81pmnsdi5v63m6x4r2vacy907j.nar.zst error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:57:57 WARN Failed to register uploaded object key=irf7apapgkmyry2z90ashdg40g27dhib.ls error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:57:57 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign niks3-go-unit-tests> 2026/10/09 09:57:57 WARN Failed to register uploaded object key=log/ny8v8b6b3fzlawc3gswr7kmdjs5svfxl-test-script.drv error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:57:57 INFO Signed narinfos id=1 count=1 niks3-go-unit-tests> 2026/10/09 09:57:57 INFO Uploading 1 narinfos niks3-go-unit-tests> === CONT TestClientFallsBackToClosures niks3-go-unit-tests> --- PASS: TestPush_CompleteCommitsEveryRoot (15.24s) niks3-go-unit-tests> 2026/10/09 09:57:57 INFO Received complete push request method=POST path=/api/pushes/1/complete niks3-go-unit-tests> 2026/10/09 09:57:57 WARN Failed to register uploaded object key=irf7apapgkmyry2z90ashdg40g27dhib.narinfo error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:57:57 INFO Completed push id=1 objects=4 duration=52.092166ms niks3-go-unit-tests> 2026/10/09 09:57:57 INFO Upload complete. (291ms) niks3-go-unit-tests> 2026/10/09 09:57:57 INFO Aborted multipart uploads count=0 kept=0 niks3-go-unit-tests> 2026/10/09 09:57:57 WARN Force mode enabled - objects will be deleted immediately without grace period niks3-go-unit-tests> 2026/10/09 09:57:57 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=0 niks3-go-unit-tests> 2026/10/09 09:57:57 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted,O=Other" niks3-go-unit-tests> 2026/10/09 09:57:57 WARN mTLS auth: bound subjects configured but subject DN unavailable niks3-go-unit-tests> 2026/10/09 09:57:57 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted" niks3-go-unit-tests> 2026/10/09 09:57:57 INFO Vacuumed table table=pending_closures niks3-go-unit-tests> --- PASS: TestService_AuthMiddleware_MTLSBoundSubjects (7.40s) niks3-go-unit-tests> === CONT TestReadProxy404 niks3-go-unit-tests> 2026/10/09 09:57:57 INFO Vacuumed table table=pending_objects niks3-go-unit-tests> 2026/10/09 09:57:57 INFO Vacuumed table table=multipart_uploads niks3-go-unit-tests> 2026/10/09 09:57:57 INFO Vacuumed table table=closures niks3-go-unit-tests> 2026/10/09 09:57:57 INFO Vacuumed table table=objects niks3-go-unit-tests> === NAME TestClientWithDependencies niks3-go-unit-tests> client_integration_test.go:753: Skipping nix copy test - isolated store (/private/var/tb/5/scratch/00f1341bdd59a71e/build/TestClientWithDependencies4176572669/001/store) requires matching store prefix niks3-go-unit-tests> --- PASS: TestClientWithDependencies (11.04s) niks3-go-unit-tests> === CONT TestGCTaskStore_Fail niks3-go-unit-tests> --- PASS: TestGCTaskStore_Fail (0.00s) niks3-go-unit-tests> === CONT TestClientErrorHandling niks3-go-unit-tests> === RUN TestClientErrorHandling/InvalidStorePath niks3-go-unit-tests> === PAUSE TestClientErrorHandling/InvalidStorePath niks3-go-unit-tests> === RUN TestClientErrorHandling/InvalidAuthToken niks3-go-unit-tests> === PAUSE TestClientErrorHandling/InvalidAuthToken niks3-go-unit-tests> === RUN TestClientErrorHandling/ServerNotAvailable niks3-go-unit-tests> === PAUSE TestClientErrorHandling/ServerNotAvailable niks3-go-unit-tests> === CONT TestLeadElectsOneAndHandsOver niks3-go-unit-tests> 2026/10/09 09:57:57 INFO Received complete multipart upload request method=POST path=/api/multipart/complete niks3-go-unit-tests> 2026/10/09 09:57:58 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggggg00000000000000000000.nar.zst upload_id=MTE5MjdlYzItNjE3OS00ZDA5LTljZjMtZGFiOThmMGQ2ZDQ0LmUzM2I5NjkwLTY0MTMtNDY3My04YTRmLTY0ZWI1OGVhNTExYngxNzkxNTM5ODY4OTQ4MTY2MDAw parts=10 niks3-go-unit-tests> --- PASS: TestService_ReadScope_PublicByDefault (8.10s) niks3-go-unit-tests> === CONT TestPush_RepeatedRootCommitsOnce niks3-go-unit-tests> === NAME TestObjectStatsLargeCommit niks3-go-unit-tests> object_stats_test.go:101: committing 60000 objects took 8.843845166s niks3-go-unit-tests> 2026/10/09 09:57:58 ERROR Failed to decode narinfo key=3hcdxyjf9yiq7qf3i4548drb6sjmwa1v.narinfo limit=16777216 error= niks3-go-unit-tests> --- PASS: TestReadProxyNarinfo (7.87s) niks3-go-unit-tests> --- FAIL: TestObjectStatsLargeCommit (20.31s) niks3-go-unit-tests> === CONT TestProxyWriteTimeout niks3-go-unit-tests> === CONT TestServerTLSConfig niks3-go-unit-tests> === RUN TestServerTLSConfig/no_client_CA niks3-go-unit-tests> === RUN TestProxyWriteTimeout/narinfo niks3-go-unit-tests> === PAUSE TestProxyWriteTimeout/narinfo niks3-go-unit-tests> === RUN TestProxyWriteTimeout/1_GiB_nar niks3-go-unit-tests> === PAUSE TestServerTLSConfig/no_client_CA niks3-go-unit-tests> === PAUSE TestProxyWriteTimeout/1_GiB_nar niks3-go-unit-tests> === RUN TestProxyWriteTimeout/10_GiB_nar niks3-go-unit-tests> === PAUSE TestProxyWriteTimeout/10_GiB_nar niks3-go-unit-tests> === RUN TestServerTLSConfig/missing_CA_file niks3-go-unit-tests> === RUN TestProxyWriteTimeout/unknown_size niks3-go-unit-tests> === PAUSE TestServerTLSConfig/missing_CA_file niks3-go-unit-tests> === PAUSE TestProxyWriteTimeout/unknown_size niks3-go-unit-tests> === RUN TestServerTLSConfig/not_a_PEM_file niks3-go-unit-tests> === CONT TestConnectWaitsForAPeerMigration niks3-go-unit-tests> === PAUSE TestServerTLSConfig/not_a_PEM_file niks3-go-unit-tests> === CONT TestConnectSerialisesConcurrentMigrations niks3-go-unit-tests> --- PASS: TestReadProxyRangeRequest (7.46s) niks3-go-unit-tests> === CONT TestMetricsInventory niks3-go-unit-tests> --- PASS: TestReadRedirectNar (7.53s) niks3-go-unit-tests> === CONT TestPush_OverlappingRootsStoreOneRowPerKey niks3-go-unit-tests> 2026/10/09 09:57:59 WARN Rate limiter enabled after throttle name=s3-test rate=5 niks3-go-unit-tests> 2026/10/09 09:57:59 WARN S3 rate limit hit operation="complete multipart upload" error="Please reduce your request rate." niks3-go-unit-tests> === NAME TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle niks3-go-unit-tests> throttle_test.go:213: Proxy stats: total=13, throttled=10, completeMultipart=10 niks3-go-unit-tests> throttle_test.go:215: Rate limiter: enabled=true, rate=5.00 niks3-go-unit-tests> --- PASS: TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle (13.77s) niks3-go-unit-tests> === CONT TestPresentNotReportedWhileGCDeletesClosure niks3-go-unit-tests> === RUN TestPresentNotReportedWhileGCDeletesClosure/stale niks3-go-unit-tests> === PAUSE TestPresentNotReportedWhileGCDeletesClosure/stale niks3-go-unit-tests> === RUN TestPresentNotReportedWhileGCDeletesClosure/fresh niks3-go-unit-tests> === PAUSE TestPresentNotReportedWhileGCDeletesClosure/fresh niks3-go-unit-tests> === CONT TestReadProxyNarStreaming niks3-go-unit-tests> --- PASS: TestReadProxyRootRedirectsToIndexHTML (7.76s) niks3-go-unit-tests> === CONT TestParseSingleRange niks3-go-unit-tests> === RUN TestParseSingleRange/none niks3-go-unit-tests> === PAUSE TestParseSingleRange/none niks3-go-unit-tests> === RUN TestParseSingleRange/unknown_unit niks3-go-unit-tests> === PAUSE TestParseSingleRange/unknown_unit niks3-go-unit-tests> === RUN TestParseSingleRange/multi-range_ignored niks3-go-unit-tests> === PAUSE TestParseSingleRange/multi-range_ignored niks3-go-unit-tests> === RUN TestParseSingleRange/malformed_no_dash niks3-go-unit-tests> === PAUSE TestParseSingleRange/malformed_no_dash niks3-go-unit-tests> === RUN TestParseSingleRange/malformed_both_empty niks3-go-unit-tests> === PAUSE TestParseSingleRange/malformed_both_empty niks3-go-unit-tests> === RUN TestParseSingleRange/malformed_end_before_start niks3-go-unit-tests> === PAUSE TestParseSingleRange/malformed_end_before_start niks3-go-unit-tests> === RUN TestParseSingleRange/closed niks3-go-unit-tests> === PAUSE TestParseSingleRange/closed niks3-go-unit-tests> === RUN TestParseSingleRange/open-ended niks3-go-unit-tests> === PAUSE TestParseSingleRange/open-ended niks3-go-unit-tests> === RUN TestParseSingleRange/end_clamped_to_size niks3-go-unit-tests> === PAUSE TestParseSingleRange/end_clamped_to_size niks3-go-unit-tests> === RUN TestParseSingleRange/suffix niks3-go-unit-tests> === PAUSE TestParseSingleRange/suffix niks3-go-unit-tests> === RUN TestParseSingleRange/suffix_exceeds_size niks3-go-unit-tests> === PAUSE TestParseSingleRange/suffix_exceeds_size niks3-go-unit-tests> === RUN TestParseSingleRange/single_byte niks3-go-unit-tests> === PAUSE TestParseSingleRange/single_byte niks3-go-unit-tests> === RUN TestParseSingleRange/start_past_EOF niks3-go-unit-tests> === PAUSE TestParseSingleRange/start_past_EOF niks3-go-unit-tests> === RUN TestParseSingleRange/start_far_past_EOF niks3-go-unit-tests> === PAUSE TestParseSingleRange/start_far_past_EOF niks3-go-unit-tests> === CONT TestIsValidCachePath niks3-go-unit-tests> === RUN TestIsValidCachePath/narinfo niks3-go-unit-tests> === PAUSE TestIsValidCachePath/narinfo niks3-go-unit-tests> === RUN TestIsValidCachePath/narinfo_all_nix_base32_chars niks3-go-unit-tests> === PAUSE TestIsValidCachePath/narinfo_all_nix_base32_chars niks3-go-unit-tests> === RUN TestIsValidCachePath/nar_zst niks3-go-unit-tests> === PAUSE TestIsValidCachePath/nar_zst niks3-go-unit-tests> === RUN TestIsValidCachePath/nar_xz niks3-go-unit-tests> === PAUSE TestIsValidCachePath/nar_xz niks3-go-unit-tests> === RUN TestIsValidCachePath/nar_bz2 niks3-go-unit-tests> === PAUSE TestIsValidCachePath/nar_bz2 niks3-go-unit-tests> === RUN TestIsValidCachePath/nar_uncompressed niks3-go-unit-tests> === PAUSE TestIsValidCachePath/nar_uncompressed niks3-go-unit-tests> === RUN TestIsValidCachePath/ls niks3-go-unit-tests> === PAUSE TestIsValidCachePath/ls niks3-go-unit-tests> === RUN TestIsValidCachePath/log niks3-go-unit-tests> === PAUSE TestIsValidCachePath/log niks3-go-unit-tests> === RUN TestIsValidCachePath/realisation niks3-go-unit-tests> === PAUSE TestIsValidCachePath/realisation niks3-go-unit-tests> === RUN TestIsValidCachePath/nix-cache-info niks3-go-unit-tests> === PAUSE TestIsValidCachePath/nix-cache-info niks3-go-unit-tests> === RUN TestIsValidCachePath/index.html niks3-go-unit-tests> === PAUSE TestIsValidCachePath/index.html niks3-go-unit-tests> === RUN TestIsValidCachePath/traversal_parent niks3-go-unit-tests> === PAUSE TestIsValidCachePath/traversal_parent niks3-go-unit-tests> === RUN TestIsValidCachePath/traversal_in_middle niks3-go-unit-tests> === PAUSE TestIsValidCachePath/traversal_in_middle niks3-go-unit-tests> === RUN TestIsValidCachePath/invalid_char_e niks3-go-unit-tests> === PAUSE TestIsValidCachePath/invalid_char_e niks3-go-unit-tests> === RUN TestIsValidCachePath/invalid_char_u niks3-go-unit-tests> === PAUSE TestIsValidCachePath/invalid_char_u niks3-go-unit-tests> === RUN TestIsValidCachePath/random_path niks3-go-unit-tests> === PAUSE TestIsValidCachePath/random_path niks3-go-unit-tests> === RUN TestIsValidCachePath/empty niks3-go-unit-tests> === PAUSE TestIsValidCachePath/empty niks3-go-unit-tests> === RUN TestIsValidCachePath/leading_slash niks3-go-unit-tests> === PAUSE TestIsValidCachePath/leading_slash niks3-go-unit-tests> === RUN TestIsValidCachePath/wrong_extension niks3-go-unit-tests> === PAUSE TestIsValidCachePath/wrong_extension niks3-go-unit-tests> === RUN TestIsValidCachePath/short_hash niks3-go-unit-tests> === PAUSE TestIsValidCachePath/short_hash niks3-go-unit-tests> === CONT TestResolveDBConnectionString niks3-go-unit-tests> 2026/10/09 09:57:59 INFO Received complete multipart upload request method=POST path=/api/multipart/complete niks3-go-unit-tests> === RUN TestResolveDBConnectionString/flag_wins niks3-go-unit-tests> === PAUSE TestResolveDBConnectionString/flag_wins niks3-go-unit-tests> === RUN TestResolveDBConnectionString/file_when_flag_empty niks3-go-unit-tests> === PAUSE TestResolveDBConnectionString/file_when_flag_empty niks3-go-unit-tests> === RUN TestResolveDBConnectionString/missing_file_is_an_error niks3-go-unit-tests> === PAUSE TestResolveDBConnectionString/missing_file_is_an_error niks3-go-unit-tests> === RUN TestResolveDBConnectionString/PGHOST_allows_empty niks3-go-unit-tests> === PAUSE TestResolveDBConnectionString/PGHOST_allows_empty niks3-go-unit-tests> === RUN TestResolveDBConnectionString/nothing_configured niks3-go-unit-tests> === PAUSE TestResolveDBConnectionString/nothing_configured niks3-go-unit-tests> === CONT TestDeleteClosuresRacingPins niks3-go-unit-tests> 2026/10/09 09:57:59 INFO Completed multipart upload object_key=nar/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb00000000000000000000.nar.zst upload_id=MTE5MjdlYzItNjE3OS00ZDA5LTljZjMtZGFiOThmMGQ2ZDQ0LjBjYTI4NWE2LTNkMDUtNGQ4OS04M2FhLTUxZTNmM2FiNGM0ZHgxNzkxNTM5ODY4OTQ4MTkxMDAw parts=10 niks3-go-unit-tests> --- PASS: TestReadProxyConditionalGet (7.49s) niks3-go-unit-tests> === CONT TestClientCADerivations niks3-go-unit-tests> === NAME TestClientIntegration niks3-go-unit-tests> client_integration_test.go:334: Created store path: /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestClientIntegration2081764932/002/store/45b2lzrw4y0rlm7xrxp9n421q405wqbi-test-file.txt niks3-go-unit-tests> 2026/10/09 09:58:00 INFO Received push request method=POST path=/api/pushes niks3-go-unit-tests> 2026/10/09 09:58:00 INFO Uploading 1 paths to 127.0.0.1 (0 already cached) niks3-go-unit-tests> 2026/10/09 09:58:00 INFO Uploading 45b2lzrw4y0rlm7xrxp9n421q405wqbi-test-file.txt (152B) niks3-go-unit-tests> 2026/10/09 09:58:00 WARN Failed to register uploaded object key=nar/11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1.nar.zst error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:58:00 WARN Failed to register uploaded object key=45b2lzrw4y0rlm7xrxp9n421q405wqbi.ls error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:58:00 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign niks3-go-unit-tests> 2026/10/09 09:58:00 INFO Signed narinfos id=1 count=1 niks3-go-unit-tests> 2026/10/09 09:58:01 INFO Uploading 1 narinfos niks3-go-unit-tests> 2026/10/09 09:58:01 INFO Received complete push request method=POST path=/api/pushes/1/complete niks3-go-unit-tests> 2026/10/09 09:58:01 WARN Failed to register uploaded object key=45b2lzrw4y0rlm7xrxp9n421q405wqbi.narinfo error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:58:01 INFO Completed push id=1 objects=3 duration=27.312958ms niks3-go-unit-tests> 2026/10/09 09:58:01 INFO Upload complete. (302ms) niks3-go-unit-tests> 2026/10/09 09:58:01 INFO All 1 paths already cached niks3-go-unit-tests> client_integration_test.go:360: Retrieved narinfo from S3: niks3-go-unit-tests> StorePath: /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestClientIntegration2081764932/002/store/45b2lzrw4y0rlm7xrxp9n421q405wqbi-test-file.txt niks3-go-unit-tests> URL: nar/11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1.nar.zst niks3-go-unit-tests> Compression: zstd niks3-go-unit-tests> NarHash: sha256:11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1 niks3-go-unit-tests> NarSize: 152 niks3-go-unit-tests> References: niks3-go-unit-tests> CA: fixed:r:sha256:11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1 niks3-go-unit-tests> client_integration_test.go:361: Retrieved .ls file from S3 (compressed size: 77 bytes) niks3-go-unit-tests> client_integration_test.go:361: Decompressed .ls content (64 bytes): niks3-go-unit-tests> {"version":1,"root":{"type":"regular","size":39,"narOffset":96}} niks3-go-unit-tests> --- PASS: TestReadProxyDisabled (7.68s) niks3-go-unit-tests> === CONT TestPresentRefreshesOnlyStaleClosures niks3-go-unit-tests> 2026/10/09 09:58:01 INFO Received push request method=POST path=/api/pushes niks3-go-unit-tests> 2026/10/09 09:58:01 INFO Object in database but missing from S3 key=45b2lzrw4y0rlm7xrxp9n421q405wqbi.ls niks3-go-unit-tests> 2026/10/09 09:58:01 WARN Found objects in DB but missing from S3, will re-upload count=1 niks3-go-unit-tests> 2026/10/09 09:58:01 INFO Uploading 0 paths to 127.0.0.1 (1 already cached) niks3-go-unit-tests> 2026/10/09 09:58:01 INFO Received complete push request method=POST path=/api/pushes/2/complete niks3-go-unit-tests> 2026/10/09 09:58:01 WARN Failed to register uploaded object key=45b2lzrw4y0rlm7xrxp9n421q405wqbi.ls error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:58:01 INFO Completed push id=2 objects=3 duration=1.358917ms niks3-go-unit-tests> 2026/10/09 09:58:01 INFO Upload complete. (143ms) niks3-go-unit-tests> === NAME TestClientIntegration niks3-go-unit-tests> client_integration_test.go:389: Retrieved .ls file from S3 (compressed size: 62 bytes) niks3-go-unit-tests> client_integration_test.go:389: Decompressed .ls content (49 bytes): niks3-go-unit-tests> {"version":1,"root":{"type":"regular","size":39}} niks3-go-unit-tests> 2026/10/09 09:58:01 INFO Received push request method=POST path=/api/pushes niks3-go-unit-tests> 2026/10/09 09:58:01 INFO Uploading 0 paths to 127.0.0.1 (1 already cached) niks3-go-unit-tests> 2026/10/09 09:58:01 INFO Received complete push request method=POST path=/api/pushes/3/complete niks3-go-unit-tests> 2026/10/09 09:58:01 WARN Failed to register uploaded object key=45b2lzrw4y0rlm7xrxp9n421q405wqbi.ls error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:58:01 INFO Completed push id=3 objects=3 duration=1.488167ms niks3-go-unit-tests> 2026/10/09 09:58:01 INFO Upload complete. (127ms) niks3-go-unit-tests> 2026/10/09 09:58:01 ERROR failed to check GC advisory lock error="query pg_locks: failed to connect to `user=_tribuchetbld5 database=niks3`: /nonexistent/.s.PGSQL.5432 (/nonexistent): dial error: dial unix /nonexistent/.s.PGSQL.5432: connect: no such file or directory" niks3-go-unit-tests> client_integration_test.go:405: Retrieved .ls file from S3 (compressed size: 62 bytes) niks3-go-unit-tests> client_integration_test.go:405: Decompressed .ls content (49 bytes): niks3-go-unit-tests> {"version":1,"root":{"type":"regular","size":39}} niks3-go-unit-tests> --- PASS: TestGCAdvisoryLockBlocksConcurrentRun (16.76s) niks3-go-unit-tests> === CONT TestPresentReportsOnlyClosureRoots niks3-go-unit-tests> 2026/10/09 09:58:01 INFO Received complete multipart upload request method=POST path=/api/multipart/complete niks3-go-unit-tests> 2026/10/09 09:58:01 INFO Completed multipart upload object_key=nar/cccccccccccccccccccccccccccccccc00000000000000000000.nar.zst upload_id=MTE5MjdlYzItNjE3OS00ZDA5LTljZjMtZGFiOThmMGQ2ZDQ0LmQwYjg4ZDkzLTIwZjItNGYxNC04ZGJlLWZhZWE1OTZhMGI4NHgxNzkxNTM5ODY4OTQ4MDg2MDAw parts=10 niks3-go-unit-tests> 2026-10-09 09:58:01.922 UTC [65876] ERROR: could not obtain lock on row in relation "closures" niks3-go-unit-tests> 2026-10-09 09:58:01.922 UTC [65876] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT niks3-go-unit-tests> 2026/10/09 09:58:01 INFO Received push request method=POST path=/api/pushes niks3-go-unit-tests> 2026-10-09 09:58:01.935 UTC [65876] ERROR: could not obtain lock on row in relation "closures" niks3-go-unit-tests> 2026-10-09 09:58:01.935 UTC [65876] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT niks3-go-unit-tests> 2026/10/09 09:58:01 INFO Uploading 1 paths to 127.0.0.1 (0 already cached) niks3-go-unit-tests> 2026/10/09 09:58:01 INFO Uploading wb7jix9zmfkmv1g4vdn34rxj89kz2pnh-lost-commit.txt (152B) niks3-go-unit-tests> 2026-10-09 09:58:01.943 UTC [65876] ERROR: could not obtain lock on row in relation "closures" niks3-go-unit-tests> 2026-10-09 09:58:01.943 UTC [65876] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT niks3-go-unit-tests> 2026-10-09 09:58:01.948 UTC [65876] ERROR: could not obtain lock on row in relation "closures" niks3-go-unit-tests> 2026-10-09 09:58:01.948 UTC [65876] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT niks3-go-unit-tests> 2026-10-09 09:58:01.948 UTC [65876] ERROR: could not obtain lock on row in relation "closures" niks3-go-unit-tests> 2026-10-09 09:58:01.948 UTC [65876] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT niks3-go-unit-tests> 2026/10/09 09:58:01 WARN Failed to register uploaded object key=wb7jix9zmfkmv1g4vdn34rxj89kz2pnh.ls error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:58:01 WARN Failed to register uploaded object key=nar/04d5klw9iix884nk82y7nlkps8afh2bg119y7zkvcf1pc1x2y2jl.nar.zst error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:58:01 INFO Received sign narinfos request method=POST path=/api/pushes/4/sign niks3-go-unit-tests> 2026/10/09 09:58:01 INFO Signed narinfos id=4 count=1 niks3-go-unit-tests> 2026/10/09 09:58:01 INFO Uploading 1 narinfos niks3-go-unit-tests> --- PASS: TestPush_RootsAreLockedInByteOrder (20.00s) niks3-go-unit-tests> === CONT TestGCEndsOnShutdown niks3-go-unit-tests> === RUN TestGCEndsOnShutdown/before_the_run niks3-go-unit-tests> === PAUSE TestGCEndsOnShutdown/before_the_run niks3-go-unit-tests> === RUN TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete niks3-go-unit-tests> === PAUSE TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete niks3-go-unit-tests> 2026/10/09 09:58:02 INFO Received complete push request method=POST path=/api/pushes/4/complete niks3-go-unit-tests> === RUN TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark niks3-go-unit-tests> 2026/10/09 09:58:02 WARN Failed to register uploaded object key=wb7jix9zmfkmv1g4vdn34rxj89kz2pnh.narinfo error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:58:02 INFO Completed push id=4 objects=3 duration=877.708µs niks3-go-unit-tests> === PAUSE TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark niks3-go-unit-tests> === CONT TestGCTaskStore_GetReturnsLatest niks3-go-unit-tests> 2026/10/09 09:58:02 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:61533/api/pushes/4/complete\": EOF" url=http://127.0.0.1:61533/api/pushes/4/complete niks3-go-unit-tests> --- PASS: TestGCTaskStore_GetReturnsLatest (0.00s) niks3-go-unit-tests> === CONT TestGCTaskStore_DeduplicateSameParams niks3-go-unit-tests> --- PASS: TestGCTaskStore_DeduplicateSameParams (0.00s) niks3-go-unit-tests> === CONT TestGCTaskStore_ConflictDifferentParams niks3-go-unit-tests> --- PASS: TestGCTaskStore_ConflictDifferentParams (0.00s) niks3-go-unit-tests> === CONT TestGCTaskStore_StartNew niks3-go-unit-tests> --- PASS: TestGCTaskStore_StartNew (0.00s) niks3-go-unit-tests> === CONT TestGCTaskStore_GetEmpty niks3-go-unit-tests> --- PASS: TestGCTaskStore_GetEmpty (0.00s) niks3-go-unit-tests> === CONT TestCacheStatsHandler niks3-go-unit-tests> --- PASS: TestReadProxyHead (6.76s) niks3-go-unit-tests> 2026/10/09 09:58:02 INFO Received complete push request method=POST path=/api/pushes/4/complete niks3-go-unit-tests> === CONT TestGCTaskStore_PhaseUpdates niks3-go-unit-tests> --- PASS: TestGCTaskStore_PhaseUpdates (0.00s) niks3-go-unit-tests> 2026-10-09 09:58:02.128 UTC [65985] ERROR: Push does not exist: id=4 niks3-go-unit-tests> 2026-10-09 09:58:02.128 UTC [65985] CONTEXT: PL/pgSQL function commit_push(bigint) line 11 at RAISE niks3-go-unit-tests> 2026-10-09 09:58:02.128 UTC [65985] STATEMENT: -- name: CommitPush :one niks3-go-unit-tests> SELECT commit_push($1::bigint)::bigint niks3-go-unit-tests> niks3-go-unit-tests> === CONT TestPushWaitsForSweepHoldingRow niks3-go-unit-tests> 2026/10/09 09:58:02 WARN Push not found id=4 niks3-go-unit-tests> 2026/10/09 09:58:02 INFO Upload complete. (278ms) niks3-go-unit-tests> === NAME TestClientIntegration niks3-go-unit-tests> client_integration_test.go:435: Retrieved narinfo from S3: niks3-go-unit-tests> StorePath: /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestClientIntegration2081764932/002/store/wb7jix9zmfkmv1g4vdn34rxj89kz2pnh-lost-commit.txt niks3-go-unit-tests> URL: nar/04d5klw9iix884nk82y7nlkps8afh2bg119y7zkvcf1pc1x2y2jl.nar.zst niks3-go-unit-tests> Compression: zstd niks3-go-unit-tests> NarHash: sha256:04d5klw9iix884nk82y7nlkps8afh2bg119y7zkvcf1pc1x2y2jl niks3-go-unit-tests> NarSize: 152 niks3-go-unit-tests> References: niks3-go-unit-tests> CA: fixed:r:sha256:04d5klw9iix884nk82y7nlkps8afh2bg119y7zkvcf1pc1x2y2jl niks3-go-unit-tests> client_integration_test.go:438: Testing garbage collection... niks3-go-unit-tests> 2026/10/09 09:58:02 INFO Starting cleanup of old closures method=DELETE path=/api/closures niks3-go-unit-tests> 2026/10/09 09:58:02 INFO Garbage collection started niks3-go-unit-tests> 2026/10/09 09:58:02 INFO Aborted multipart uploads count=0 kept=0 niks3-go-unit-tests> 2026/10/09 09:58:02 WARN Force mode enabled - objects will be deleted immediately without grace period niks3-go-unit-tests> 2026-10-09 09:58:02.294 UTC [66074] ERROR: duplicate key value violates unique constraint "pg_type_typname_nsp_index" niks3-go-unit-tests> 2026-10-09 09:58:02.294 UTC [66074] DETAIL: Key (typname, typnamespace)=(goose_db_version, 2200) already exists. niks3-go-unit-tests> 2026-10-09 09:58:02.294 UTC [66074] STATEMENT: CREATE TABLE goose_db_version ( niks3-go-unit-tests> id integer PRIMARY KEY GENERATED BY DEFAULT AS IDENTITY, niks3-go-unit-tests> version_id bigint NOT NULL, niks3-go-unit-tests> is_applied boolean NOT NULL, niks3-go-unit-tests> tstamp timestamp NOT NULL DEFAULT now() niks3-go-unit-tests> ) niks3-go-unit-tests> 2026/10/09 09:58:02 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=2 objects-marked-for-deletion=6 objects-deleted-after-grace-period=6 objects-failed-to-delete=0 niks3-go-unit-tests> 2026/10/09 09:58:02 INFO Vacuumed table table=pending_closures niks3-go-unit-tests> 2026/10/09 09:58:02 INFO Vacuumed table table=pending_objects niks3-go-unit-tests> 2026/10/09 09:58:02 INFO Vacuumed table table=multipart_uploads niks3-go-unit-tests> 2026/10/09 09:58:02 INFO Vacuumed table table=closures niks3-go-unit-tests> 2026/10/09 09:58:02 INFO Vacuumed table table=objects niks3-go-unit-tests> === CONT TestService_AuthMiddleware_MTLSProxyHeader niks3-go-unit-tests> --- PASS: TestReadProxyInvalidPath (7.23s) niks3-go-unit-tests> 2026/10/09 09:58:02 INFO Received push request method=POST path=/api/pushes niks3-go-unit-tests> 2026/10/09 09:58:02 INFO Uploading 2 paths to 127.0.0.1 (0 already cached) niks3-go-unit-tests> 2026/10/09 09:58:02 INFO Uploading 5c5zi1r1y48bkjhw0c1ach12f9qvyfmp-shared-dep (136B) niks3-go-unit-tests> 2026/10/09 09:58:02 INFO Uploading fk1l0pgwnb8ykv2938ysa7h0fzilx3if-top (264B) niks3-go-unit-tests> 2026/10/09 09:58:02 WARN Failed to register uploaded object key=5c5zi1r1y48bkjhw0c1ach12f9qvyfmp.ls error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:58:02 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:58:02 WARN Failed to register uploaded object key=fk1l0pgwnb8ykv2938ysa7h0fzilx3if.ls error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:58:02 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign niks3-go-unit-tests> 2026/10/09 09:58:02 WARN Failed to register uploaded object key=nar/0iqpvzd3p6vfgig0mgnq65fg56hwhkl3fdfm8bygqh3q33wwj93v.nar.zst error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:58:02 INFO Signed narinfos id=1 count=2 niks3-go-unit-tests> 2026/10/09 09:58:02 INFO Uploading 2 narinfos niks3-go-unit-tests> 2026/10/09 09:58:03 WARN Failed to register uploaded object key=fk1l0pgwnb8ykv2938ysa7h0fzilx3if.narinfo error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:58:03 INFO Received complete push request method=POST path=/api/pushes/1/complete niks3-go-unit-tests> 2026/10/09 09:58:03 WARN Failed to register uploaded object key=5c5zi1r1y48bkjhw0c1ach12f9qvyfmp.narinfo error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:58:03 INFO Completed push id=1 objects=6 duration=32.0545ms niks3-go-unit-tests> 2026/10/09 09:58:03 INFO Upload complete. (237ms) niks3-go-unit-tests> === NAME TestClientSharedPathCommittedMidPush niks3-go-unit-tests> client_integration_test.go:816: Retrieved narinfo from S3: niks3-go-unit-tests> StorePath: /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestClientSharedPathCommittedMidPush2098838153/001/store/5c5zi1r1y48bkjhw0c1ach12f9qvyfmp-shared-dep niks3-go-unit-tests> URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst niks3-go-unit-tests> Compression: zstd niks3-go-unit-tests> NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82 niks3-go-unit-tests> NarSize: 136 niks3-go-unit-tests> References: niks3-go-unit-tests> CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n niks3-go-unit-tests> client_integration_test.go:816: Retrieved narinfo from S3: niks3-go-unit-tests> StorePath: /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestClientSharedPathCommittedMidPush2098838153/001/store/fk1l0pgwnb8ykv2938ysa7h0fzilx3if-top niks3-go-unit-tests> URL: nar/0iqpvzd3p6vfgig0mgnq65fg56hwhkl3fdfm8bygqh3q33wwj93v.nar.zst niks3-go-unit-tests> Compression: zstd niks3-go-unit-tests> NarHash: sha256:0iqpvzd3p6vfgig0mgnq65fg56hwhkl3fdfm8bygqh3q33wwj93v niks3-go-unit-tests> NarSize: 264 niks3-go-unit-tests> References: /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestClientSharedPathCommittedMidPush2098838153/001/store/5c5zi1r1y48bkjhw0c1ach12f9qvyfmp-shared-dep niks3-go-unit-tests> CA: text:sha256:0i3n7vcsklsr2aifnh5wr7yxqhwjg2k0b9crkj27p9bhqwn5rmin niks3-go-unit-tests> --- PASS: TestClientSharedPathCommittedMidPush (8.24s) niks3-go-unit-tests> === CONT TestCommitRacingPendingCleanupKeepsObjects niks3-go-unit-tests> --- PASS: TestReadProxyContentNegotiation (7.14s) niks3-go-unit-tests> === CONT TestSweepRechecksPendingAfterLocking niks3-go-unit-tests> --- PASS: TestConnectSerialisesConcurrentMigrations (4.64s) niks3-go-unit-tests> === CONT TestDeletePinKeepsRowWhenS3Fails niks3-go-unit-tests> 2026/10/09 09:58:03 INFO Aborted multipart uploads count=0 kept=0 niks3-go-unit-tests> 2026/10/09 09:58:03 WARN Force mode enabled - objects will be deleted immediately without grace period niks3-go-unit-tests> 2026/10/09 09:58:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=0 niks3-go-unit-tests> 2026/10/09 09:58:03 INFO Vacuumed table table=pending_closures niks3-go-unit-tests> 2026/10/09 09:58:03 INFO Vacuumed table table=pending_objects niks3-go-unit-tests> 2026/10/09 09:58:03 INFO Vacuumed table table=multipart_uploads niks3-go-unit-tests> === NAME TestPinProtectsFromGC niks3-go-unit-tests> client_integration_test.go:867: Pinned store path: /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestPinProtectsFromGC2645952582/001/store/pdrcsd9i9ijjm0kq6kqvagrql0r3gshc-pinned-file.txt niks3-go-unit-tests> client_integration_test.go:868: Unpinned store path: /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestPinProtectsFromGC2645952582/001/store/jn0wsxl6dny48rhxhky5arzyhg47sysv-unpinned-file.txt niks3-go-unit-tests> 2026/10/09 09:58:03 INFO Vacuumed table table=closures niks3-go-unit-tests> 2026/10/09 09:58:03 INFO Vacuumed table table=objects niks3-go-unit-tests> --- PASS: TestGCMetrics (6.89s) niks3-go-unit-tests> === CONT TestCreatePinRejectsBadInput niks3-go-unit-tests> 2026/10/09 09:58:03 INFO Received push request method=POST path=/api/pushes niks3-go-unit-tests> 2026/10/09 09:58:03 INFO Uploading 1 paths to 127.0.0.1 (0 already cached) niks3-go-unit-tests> 2026/10/09 09:58:03 INFO Uploading pdrcsd9i9ijjm0kq6kqvagrql0r3gshc-pinned-file.txt (128B) niks3-go-unit-tests> 2026/10/09 09:58:03 WARN Failed to register uploaded object key=nar/0qmhsg42pqsz7x6hh02j8gznnac3yi3qpjhrqbwvz88i194jbb5s.nar.zst error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:58:03 WARN Failed to register uploaded object key=pdrcsd9i9ijjm0kq6kqvagrql0r3gshc.ls error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:58:03 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign niks3-go-unit-tests> 2026/10/09 09:58:03 INFO Signed narinfos id=1 count=1 niks3-go-unit-tests> 2026/10/09 09:58:03 INFO Uploading 1 narinfos niks3-go-unit-tests> === CONT TestConcurrentPinUpdatesAgree niks3-go-unit-tests> --- PASS: TestReadProxyNarinfoAlreadyDecompressed (7.11s) niks3-go-unit-tests> 2026/10/09 09:58:03 INFO Received complete push request method=POST path=/api/pushes/1/complete niks3-go-unit-tests> 2026/10/09 09:58:03 WARN Failed to register uploaded object key=pdrcsd9i9ijjm0kq6kqvagrql0r3gshc.narinfo error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:58:03 INFO Completed push id=1 objects=3 duration=23.210875ms niks3-go-unit-tests> 2026/10/09 09:58:03 INFO Upload complete. (187ms) niks3-go-unit-tests> 2026/10/09 09:58:04 INFO Received push request method=POST path=/api/pushes niks3-go-unit-tests> 2026/10/09 09:58:04 INFO Uploading 1 paths to 127.0.0.1 (0 already cached) niks3-go-unit-tests> 2026/10/09 09:58:04 INFO Uploading jn0wsxl6dny48rhxhky5arzyhg47sysv-unpinned-file.txt (128B) niks3-go-unit-tests> 2026/10/09 09:58:04 WARN Failed to register uploaded object key=nar/0gh0wdnr6gmk4r5sfzzh12isnl14rgmkrwy1ws41hpsrlndclg5h.nar.zst error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:58:04 INFO Received sign narinfos request method=POST path=/api/pushes/2/sign niks3-go-unit-tests> 2026/10/09 09:58:04 WARN Failed to register uploaded object key=jn0wsxl6dny48rhxhky5arzyhg47sysv.ls error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:58:04 INFO Signed narinfos id=2 count=1 niks3-go-unit-tests> 2026/10/09 09:58:04 INFO Uploading 1 narinfos niks3-go-unit-tests> 2026/10/09 09:58:04 INFO Received complete push request method=POST path=/api/pushes/2/complete niks3-go-unit-tests> 2026/10/09 09:58:04 WARN Failed to register uploaded object key=jn0wsxl6dny48rhxhky5arzyhg47sysv.narinfo error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:58:04 INFO Completed push id=2 objects=3 duration=958.083µs niks3-go-unit-tests> 2026/10/09 09:58:04 INFO Upload complete. (121ms) niks3-go-unit-tests> 2026/10/09 09:58:04 INFO Starting cleanup of old closures method=DELETE path=/api/closures niks3-go-unit-tests> 2026/10/09 09:58:04 INFO Garbage collection started niks3-go-unit-tests> 2026/10/09 09:58:04 INFO Garbage collection progress phase="" failed_uploads_deleted=0 old_closures_deleted=2 objects_marked=6 objects_deleted=6 objects_failed=0 niks3-go-unit-tests> 2026/10/09 09:58:04 INFO Aborted multipart uploads count=0 kept=0 niks3-go-unit-tests> === NAME TestClientIntegration niks3-go-unit-tests> client_integration_test.go:445: Objects in database after GC: niks3-go-unit-tests> client_integration_test.go:445: Successfully deleted all objects with GC --force niks3-go-unit-tests> 2026/10/09 09:58:04 WARN Force mode enabled - objects will be deleted immediately without grace period niks3-go-unit-tests> 2026/10/09 09:58:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=0 niks3-go-unit-tests> --- PASS: TestClientIntegration (11.41s) niks3-go-unit-tests> === CONT TestOrphanedObjectsGCStressTest niks3-go-unit-tests> 2026/10/09 09:58:04 INFO Vacuumed table table=pending_closures niks3-go-unit-tests> 2026/10/09 09:58:04 INFO Vacuumed table table=pending_objects niks3-go-unit-tests> === NAME TestClientMultipleUploads niks3-go-unit-tests> client_integration_test.go:480: Created store path 0: /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestClientMultipleUploads1516919282/001/store/izqdx8ksj5dggrf9w7d6zw1w8dk0cphg-test-file-0.txt niks3-go-unit-tests> 2026/10/09 09:58:04 INFO Vacuumed table table=multipart_uploads niks3-go-unit-tests> 2026/10/09 09:58:04 INFO Vacuumed table table=closures niks3-go-unit-tests> 2026/10/09 09:58:04 INFO Vacuumed table table=objects niks3-go-unit-tests> client_integration_test.go:480: Created store path 1: /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestClientMultipleUploads1516919282/001/store/6a6kq2kbllxn8sqpgq9lf9dhkkr0wh59-test-file-1.txt niks3-go-unit-tests> client_integration_test.go:480: Created store path 2: /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestClientMultipleUploads1516919282/001/store/rqkifjy739cgi0bca3yw8mbjslfc5vms-test-file-2.txt niks3-go-unit-tests> 2026/10/09 09:58:04 INFO Received push request method=POST path=/api/pushes niks3-go-unit-tests> 2026/10/09 09:58:04 INFO Uploading 3 paths to 127.0.0.1 (0 already cached) niks3-go-unit-tests> 2026/10/09 09:58:04 INFO Uploading rqkifjy739cgi0bca3yw8mbjslfc5vms-test-file-2.txt (160B) niks3-go-unit-tests> 2026/10/09 09:58:04 INFO Uploading 6a6kq2kbllxn8sqpgq9lf9dhkkr0wh59-test-file-1.txt (160B) niks3-go-unit-tests> 2026/10/09 09:58:04 INFO Uploading izqdx8ksj5dggrf9w7d6zw1w8dk0cphg-test-file-0.txt (160B) niks3-go-unit-tests> 2026/10/09 09:58:04 WARN Failed to register uploaded object key=6a6kq2kbllxn8sqpgq9lf9dhkkr0wh59.ls error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:58:04 WARN Failed to register uploaded object key=rqkifjy739cgi0bca3yw8mbjslfc5vms.ls error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:58:04 WARN Failed to register uploaded object key=izqdx8ksj5dggrf9w7d6zw1w8dk0cphg.ls error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:58:04 WARN Failed to register uploaded object key=nar/0ahq3izw3h0b0c08p3xdfclh758fwxnp85ccvflasb4vnfsv50ri.nar.zst error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:58:04 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign niks3-go-unit-tests> 2026/10/09 09:58:04 WARN Failed to register uploaded object key=nar/1823g6vx4q5zqvr73m7lkq1wpk40fin7bsc792di430yk7sg4lka.nar.zst error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:58:04 WARN Failed to register uploaded object key=nar/13p9pqi0ga6fx2apcwls9hjnv7nmf740ggpn5zvfzdsp5vz2bbg4.nar.zst error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:58:04 INFO Signed narinfos id=1 count=3 niks3-go-unit-tests> 2026/10/09 09:58:04 INFO Uploading 3 narinfos niks3-go-unit-tests> 2026/10/09 09:58:04 INFO Received complete push request method=POST path=/api/pushes/1/complete niks3-go-unit-tests> 2026/10/09 09:58:04 WARN Failed to register uploaded object key=rqkifjy739cgi0bca3yw8mbjslfc5vms.narinfo error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:58:04 WARN Failed to register uploaded object key=6a6kq2kbllxn8sqpgq9lf9dhkkr0wh59.narinfo error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:58:04 WARN Failed to register uploaded object key=izqdx8ksj5dggrf9w7d6zw1w8dk0cphg.narinfo error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:58:04 INFO Completed push id=1 objects=9 duration=14.505792ms niks3-go-unit-tests> 2026/10/09 09:58:04 INFO Upload complete. (187ms) niks3-go-unit-tests> client_integration_test.go:491: Uploaded 3 paths in 228.728375ms niks3-go-unit-tests> 2026/10/09 09:58:04 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> --- PASS: TestClientMultipleUploads (7.72s) niks3-go-unit-tests> === CONT TestCreatePin_ReservedPins niks3-go-unit-tests> 2026/10/09 09:58:05 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> 2026/10/09 09:58:05 INFO Uploading 2 paths to 127.0.0.1 (1 already cached) niks3-go-unit-tests> 2026/10/09 09:58:05 INFO Uploading xxzxrcmv6wadq97qdac2cd9dk468xa1m-a (256B) niks3-go-unit-tests> 2026/10/09 09:58:05 INFO Uploading m0sn449c9cypp5v62g8n3vdq4vaffrcf-shared-dep (136B) niks3-go-unit-tests> 2026/10/09 09:58:05 WARN Rate limiter enabled after throttle name=s3-test rate=5 niks3-go-unit-tests> 2026/10/09 09:58:05 WARN S3 rate limit hit during proxy key=4hcdxyjf9yiq7qf3i4548drb6sjmwa1v.narinfo error="Please reduce your request rate." niks3-go-unit-tests> 2026/10/09 09:58:05 WARN Failed to register uploaded object key=3f3r9z4p2isbyqakgx82awx024gzkrcy.ls error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:58:05 WARN Failed to register uploaded object key=xxzxrcmv6wadq97qdac2cd9dk468xa1m.ls error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:58:05 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:58:05 WARN Failed to register uploaded object key=nar/1chzg4pqjcm1s2nm5fgdkry9dkrym3gd4qxjzpi272n979qlvdk0.nar.zst error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:58:05 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign niks3-go-unit-tests> 2026/10/09 09:58:05 WARN Failed to register uploaded object key=m0sn449c9cypp5v62g8n3vdq4vaffrcf.ls error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:58:05 INFO Signed narinfos id=1 count=2 niks3-go-unit-tests> 2026/10/09 09:58:05 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign niks3-go-unit-tests> 2026/10/09 09:58:05 INFO Signed narinfos id=2 count=2 niks3-go-unit-tests> 2026/10/09 09:58:05 INFO Uploading 4 narinfos niks3-go-unit-tests> 2026/10/09 09:58:05 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:61638/oidc niks3-go-unit-tests> --- PASS: TestReadProxyOutlastsServerWriteTimeout (12.23s) niks3-go-unit-tests> === CONT TestSweepSkipsRowsLockedByPush niks3-go-unit-tests> 2026/10/09 09:58:05 WARN Failed to register uploaded object key=m0sn449c9cypp5v62g8n3vdq4vaffrcf.narinfo error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:58:05 WARN Failed to register uploaded object key=xxzxrcmv6wadq97qdac2cd9dk468xa1m.narinfo error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:58:05 WARN Failed to register uploaded object key=3f3r9z4p2isbyqakgx82awx024gzkrcy.narinfo error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:58:05 INFO lead: acquired remote=192.0.2.1:1234 niks3-go-unit-tests> 2026/10/09 09:58:05 WARN Rate limiter backed off name=s3-test rate=5 niks3-go-unit-tests> 2026/10/09 09:58:05 WARN S3 rate limit hit during proxy key=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Please reduce your request rate." niks3-go-unit-tests> 2026/10/09 09:58:05 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete niks3-go-unit-tests> 2026/10/09 09:58:05 WARN Failed to register uploaded object key=m0sn449c9cypp5v62g8n3vdq4vaffrcf.narinfo error="server returned 404: 404 page not found\n" niks3-go-unit-tests> --- PASS: TestReadProxy404 (7.53s) niks3-go-unit-tests> === CONT TestGetExistingObjectsLocksRows niks3-go-unit-tests> 2026/10/09 09:58:05 INFO Completed upload id=1 objects=6 duration=33.109167ms niks3-go-unit-tests> 2026/10/09 09:58:05 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete niks3-go-unit-tests> 2026/10/09 09:58:05 INFO Completed upload id=2 objects=6 duration=679.959µs niks3-go-unit-tests> 2026/10/09 09:58:05 INFO Upload complete. (325ms) niks3-go-unit-tests> === NAME TestClientFallsBackToClosures niks3-go-unit-tests> client_pushes_test.go:112: Retrieved narinfo from S3: niks3-go-unit-tests> StorePath: /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestClientFallsBackToClosures3382990515/001/store/m0sn449c9cypp5v62g8n3vdq4vaffrcf-shared-dep niks3-go-unit-tests> URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst niks3-go-unit-tests> Compression: zstd niks3-go-unit-tests> NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82 niks3-go-unit-tests> NarSize: 136 niks3-go-unit-tests> References: niks3-go-unit-tests> CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n niks3-go-unit-tests> client_pushes_test.go:112: Retrieved narinfo from S3: niks3-go-unit-tests> StorePath: /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestClientFallsBackToClosures3382990515/001/store/xxzxrcmv6wadq97qdac2cd9dk468xa1m-a niks3-go-unit-tests> URL: nar/1chzg4pqjcm1s2nm5fgdkry9dkrym3gd4qxjzpi272n979qlvdk0.nar.zst niks3-go-unit-tests> Compression: zstd niks3-go-unit-tests> NarHash: sha256:1chzg4pqjcm1s2nm5fgdkry9dkrym3gd4qxjzpi272n979qlvdk0 niks3-go-unit-tests> NarSize: 256 niks3-go-unit-tests> References: /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestClientFallsBackToClosures3382990515/001/store/m0sn449c9cypp5v62g8n3vdq4vaffrcf-shared-dep niks3-go-unit-tests> CA: text:sha256:10n9icachmk13jydkmabgzy7qd58vn5ll960zscj088g1pnbwrq3 niks3-go-unit-tests> client_pushes_test.go:112: Retrieved narinfo from S3: niks3-go-unit-tests> StorePath: /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestClientFallsBackToClosures3382990515/001/store/3f3r9z4p2isbyqakgx82awx024gzkrcy-b niks3-go-unit-tests> URL: nar/1chzg4pqjcm1s2nm5fgdkry9dkrym3gd4qxjzpi272n979qlvdk0.nar.zst niks3-go-unit-tests> Compression: zstd niks3-go-unit-tests> NarHash: sha256:1chzg4pqjcm1s2nm5fgdkry9dkrym3gd4qxjzpi272n979qlvdk0 niks3-go-unit-tests> NarSize: 256 niks3-go-unit-tests> References: /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestClientFallsBackToClosures3382990515/001/store/m0sn449c9cypp5v62g8n3vdq4vaffrcf-shared-dep niks3-go-unit-tests> CA: text:sha256:10n9icachmk13jydkmabgzy7qd58vn5ll960zscj088g1pnbwrq3 niks3-go-unit-tests> --- PASS: TestClientFallsBackToClosures (7.87s) niks3-go-unit-tests> === CONT TestRegisterCompletedObjectNeedsPendingRow niks3-go-unit-tests> 2026/10/09 09:58:05 INFO Received push request method=POST path=/api/pushes niks3-go-unit-tests> --- PASS: TestMetricsInventory (7.12s) niks3-go-unit-tests> === CONT TestLeadIncumbentAnnouncesAtOnce niks3-go-unit-tests> 2026/10/09 09:58:06 INFO Received push request method=POST path=/api/pushes niks3-go-unit-tests> 2026/10/09 09:58:06 INFO Garbage collection progress phase="" failed_uploads_deleted=0 old_closures_deleted=1 objects_marked=3 objects_deleted=3 objects_failed=0 niks3-go-unit-tests> 2026/10/09 09:58:06 INFO Received create pin request method=POST path=/api/pins/myapp niks3-go-unit-tests> --- PASS: TestPush_OverlappingRootsStoreOneRowPerKey (7.01s) niks3-go-unit-tests> === CONT TestForceGCDuringPushOffersSweptObject/after_presence_check niks3-go-unit-tests> 2026/10/09 09:58:06 INFO Created/updated pin name=myapp store_path=/private/var/tb/5/scratch/00f1341bdd59a71e/build/TestPinProtectsFromGC2645952582/001/store/pdrcsd9i9ijjm0kq6kqvagrql0r3gshc-pinned-file.txt narinfo_key=pdrcsd9i9ijjm0kq6kqvagrql0r3gshc.narinfo niks3-go-unit-tests> 2026/10/09 09:58:06 INFO Starting cleanup of old closures method=DELETE path=/api/closures niks3-go-unit-tests> 2026/10/09 09:58:06 INFO Garbage collection started niks3-go-unit-tests> 2026/10/09 09:58:06 INFO Aborted multipart uploads count=0 kept=0 niks3-go-unit-tests> 2026/10/09 09:58:06 WARN Force mode enabled - objects will be deleted immediately without grace period niks3-go-unit-tests> 2026/10/09 09:58:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=0 niks3-go-unit-tests> 2026/10/09 09:58:06 INFO Vacuumed table table=pending_closures niks3-go-unit-tests> 2026/10/09 09:58:06 INFO Vacuumed table table=pending_objects niks3-go-unit-tests> 2026/10/09 09:58:06 INFO Vacuumed table table=multipart_uploads niks3-go-unit-tests> 2026/10/09 09:58:06 INFO Vacuumed table table=closures niks3-go-unit-tests> 2026/10/09 09:58:06 INFO Vacuumed table table=objects niks3-go-unit-tests> 2026/10/09 09:58:06 INFO lead: released remote=192.0.2.1:1234 niks3-go-unit-tests> 2026/10/09 09:58:06 INFO lead: acquired remote=192.0.2.1:1234 niks3-go-unit-tests> --- PASS: TestReadProxyNarStreaming (7.03s) niks3-go-unit-tests> === CONT TestForceGCDuringPushOffersSweptObject/before_pending_rows niks3-go-unit-tests> 2026/10/09 09:58:07 INFO Received complete multipart upload request method=POST path=/api/multipart/complete niks3-go-unit-tests> 2026/10/09 09:58:07 INFO Completed multipart upload object_key=nar/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa00000000000000000000.nar.zst upload_id=MTE5MjdlYzItNjE3OS00ZDA5LTljZjMtZGFiOThmMGQ2ZDQ0LmIzMzU4OTY3LWRkNjgtNGNmZS04OGJhLTc4OTE4ZjAyN2E5NXgxNzkxNTM5ODg1NTUzODIxMDAw parts=10 niks3-go-unit-tests> 2026/10/09 09:58:07 INFO lead: released remote=192.0.2.1:1234 niks3-go-unit-tests> --- PASS: TestLeadElectsOneAndHandsOver (9.62s) niks3-go-unit-tests> === CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal niks3-go-unit-tests> 2026/10/09 09:58:07 INFO Received uploads request method=POST path=/ niks3-go-unit-tests> === CONT TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key niks3-go-unit-tests> 2026/10/09 09:58:07 INFO Received complete multipart upload request method=POST path=/ niks3-go-unit-tests> === CONT TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key niks3-go-unit-tests> 2026/10/09 09:58:07 INFO Received request for more parts method=POST path=/ niks3-go-unit-tests> === CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info niks3-go-unit-tests> 2026/10/09 09:58:07 INFO Received uploads request method=POST path=/ niks3-go-unit-tests> === CONT TestPendingClosureWriteTimeout/670k_objects niks3-go-unit-tests> === CONT TestPendingClosureWriteTimeout/empty niks3-go-unit-tests> --- PASS: TestUploadHandlersRejectInvalidKeys (0.01s) niks3-go-unit-tests> --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal (0.01s) niks3-go-unit-tests> --- PASS: TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key (0.00s) niks3-go-unit-tests> --- PASS: TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key (0.01s) niks3-go-unit-tests> --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info (0.00s) niks3-go-unit-tests> === CONT TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline niks3-go-unit-tests> --- PASS: TestPresentRefreshesOnlyStaleClosures (6.37s) niks3-go-unit-tests> === CONT TestPendingClosureWriteTimeout/400_objects niks3-go-unit-tests> === CONT TestIsValidUploadKey/build_log_home-manager_file niks3-go-unit-tests> === CONT TestIsValidUploadKey/unknown_type niks3-go-unit-tests> === CONT TestPendingClosureFailureTracksEveryUpload/refuse_aborts=false niks3-go-unit-tests> === NAME TestClientCADerivations niks3-go-unit-tests> client_ca_test.go:136: Built CA derivation: /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestClientCADerivations48090870/001/store/sbspf4zal7xybqarcb96sazpvn09dw1j-ca-test niks3-go-unit-tests> client_ca_test.go:139: Found 1 dependencies (including self) niks3-go-unit-tests> --- PASS: TestPresentReportsOnlyClosureRoots (6.25s) niks3-go-unit-tests> === CONT TestIsValidUploadKey/absolute niks3-go-unit-tests> === CONT TestIsValidUploadKey/traversal_nar niks3-go-unit-tests> === CONT TestIsValidUploadKey/traversal niks3-go-unit-tests> === CONT TestIsValidUploadKey/listing_key,_narinfo_type niks3-go-unit-tests> === CONT TestPendingClosureFailureTracksEveryUpload/refuse_aborts=true niks3-go-unit-tests> 2026/10/09 09:58:08 INFO Received push request method=POST path=/api/pushes niks3-go-unit-tests> 2026/10/09 09:58:08 INFO Uploading 1 paths to 127.0.0.1 (0 already cached) niks3-go-unit-tests> 2026/10/09 09:58:08 INFO Uploading sbspf4zal7xybqarcb96sazpvn09dw1j-ca-test (144B) niks3-go-unit-tests> --- PASS: TestConnectWaitsForAPeerMigration (9.46s) niks3-go-unit-tests> === CONT TestIsValidUploadKey/build_log_question_mark niks3-go-unit-tests> === CONT TestIsValidUploadKey/index.html niks3-go-unit-tests> === CONT TestIsValidUploadKey/build_log_plus_in_name niks3-go-unit-tests> === CONT TestIsValidUploadKey/nix-cache-info niks3-go-unit-tests> === CONT TestIsValidUploadKey/nar_key,_narinfo_type niks3-go-unit-tests> === CONT TestIsValidUploadKey/realisation_plus_in_output niks3-go-unit-tests> === CONT TestIsValidUploadKey/build_log niks3-go-unit-tests> === CONT TestIsValidUploadKey/narinfo_key,_nar_type niks3-go-unit-tests> === CONT TestIsValidUploadKey/build_log_equals niks3-go-unit-tests> === CONT TestIsValidUploadKey/listing niks3-go-unit-tests> === CONT TestIsValidUploadKey/empty_key niks3-go-unit-tests> === CONT TestIsValidUploadKey/narinfo niks3-go-unit-tests> === CONT TestIsValidUploadKey/nar_plain niks3-go-unit-tests> === CONT TestIsValidUploadKey/realisation niks3-go-unit-tests> === CONT TestIsValidUploadKey/nar_zst niks3-go-unit-tests> === CONT TestIsValidUploadKey/nar_xz niks3-go-unit-tests> === CONT TestUploadHandlersRejectOversizedBody/create_pending_closure niks3-go-unit-tests> --- PASS: TestIsValidUploadKey (0.02s) niks3-go-unit-tests> --- PASS: TestIsValidUploadKey/build_log_home-manager_file (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidUploadKey/unknown_type (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidUploadKey/absolute (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidUploadKey/traversal_nar (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidUploadKey/traversal (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidUploadKey/listing_key,_narinfo_type (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidUploadKey/build_log_question_mark (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidUploadKey/index.html (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidUploadKey/build_log_plus_in_name (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidUploadKey/nix-cache-info (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidUploadKey/nar_key,_narinfo_type (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidUploadKey/realisation_plus_in_output (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidUploadKey/build_log (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidUploadKey/narinfo_key,_nar_type (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidUploadKey/build_log_equals (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidUploadKey/listing (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidUploadKey/empty_key (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidUploadKey/narinfo (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidUploadKey/nar_plain (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidUploadKey/realisation (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidUploadKey/nar_zst (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidUploadKey/nar_xz (0.00s) niks3-go-unit-tests> 2026/10/09 09:58:08 INFO Received uploads request method=POST path=/ niks3-go-unit-tests> 2026/10/09 09:58:08 WARN Failed to register uploaded object key=nar/04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n.nar.zst error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:58:08 WARN Failed to register uploaded object key=sbspf4zal7xybqarcb96sazpvn09dw1j.ls error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:58:08 WARN Failed to register uploaded object key=log/bbk7vwwnbls3hamrmv5i6i7fpd2b0h8y-ca-test.drv error="server returned 404: 404 page not found\n" niks3-go-unit-tests> 2026/10/09 09:58:08 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign niks3-go-unit-tests> 2026/10/09 09:58:08 INFO Signed narinfos id=1 count=1 niks3-go-unit-tests> 2026/10/09 09:58:08 INFO Uploading 1 narinfos niks3-go-unit-tests> 2026/10/09 09:58:08 INFO Received complete push request method=POST path=/api/pushes/1/complete niks3-go-unit-tests> 2026/10/09 09:58:08 WARN Failed to register uploaded object key=sbspf4zal7xybqarcb96sazpvn09dw1j.narinfo error="server returned 404: 404 page not found\n" niks3-go-unit-tests> === NAME TestPinProtectsFromGC niks3-go-unit-tests> client_integration_test.go:976: Pin successfully protected closure from garbage collection niks3-go-unit-tests> 2026/10/09 09:58:08 INFO Completed push id=1 objects=4 duration=42.304709ms niks3-go-unit-tests> 2026/10/09 09:58:08 INFO Upload complete. (358ms) niks3-go-unit-tests> === NAME TestClientCADerivations niks3-go-unit-tests> client_ca_test.go:180: Narinfo contains CA field: StorePath: /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestClientCADerivations48090870/001/store/sbspf4zal7xybqarcb96sazpvn09dw1j-ca-test niks3-go-unit-tests> URL: nar/04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n.nar.zst niks3-go-unit-tests> Compression: zstd niks3-go-unit-tests> NarHash: sha256:04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n niks3-go-unit-tests> NarSize: 144 niks3-go-unit-tests> References: niks3-go-unit-tests> Deriver: /private/var/tb/5/scratch/00f1341bdd59a71e/build/TestClientCADerivations48090870/001/store/bbk7vwwnbls3hamrmv5i6i7fpd2b0h8y-ca-test.drv niks3-go-unit-tests> CA: fixed:r:sha256:04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n niks3-go-unit-tests> client_ca_test.go:185: Checking for realisation files in S3... niks3-go-unit-tests> client_ca_test.go:207: No realisation files uploaded - this is expected for locally built CA derivations niks3-go-unit-tests> client_ca_test.go:208: Locally built CA derivations don't have DrvOutput IDs until they're substituted from a cache niks3-go-unit-tests> --- PASS: TestPinProtectsFromGC (12.54s) niks3-go-unit-tests> === CONT TestUploadHandlersRejectOversizedBody/request_more_parts niks3-go-unit-tests> 2026/10/09 09:58:08 INFO Received request for more parts method=POST path=/ niks3-go-unit-tests> --- PASS: TestCacheStatsHandler (6.43s) niks3-go-unit-tests> === CONT TestUploadHandlersRejectOversizedBody/complete_multipart niks3-go-unit-tests> 2026/10/09 09:58:08 INFO Received complete multipart upload request method=POST path=/ niks3-go-unit-tests> === NAME TestClientCADerivations niks3-go-unit-tests> client_ca_test.go:258: nix copy output: error: binary cache 's3://bucket80?endpoint=http://localhost:60965®ion=eu-west-1' is for Nix stores with prefix '/nix/store', not '/private/var/tb/5/scratch/00f1341bdd59a71e/build/TestClientCADerivations48090870/001/store' niks3-go-unit-tests> client_ca_test.go:259: nix copy failed (might be expected with isolated stores): exit status 1 niks3-go-unit-tests> --- PASS: TestClientCADerivations (8.55s) niks3-go-unit-tests> === CONT TestCacheConfigHandler/full_config,_no_issuer niks3-go-unit-tests> === CONT TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator niks3-go-unit-tests> === CONT TestCacheConfigHandler/no_signing_keys niks3-go-unit-tests> === CONT TestCacheConfigHandler/no_cache_url_configured niks3-go-unit-tests> === CONT TestPush_RejectsBadRequests/no_objects niks3-go-unit-tests> --- PASS: TestCacheConfigHandler (0.00s) niks3-go-unit-tests> --- PASS: TestCacheConfigHandler/full_config,_no_issuer (0.00s) niks3-go-unit-tests> --- PASS: TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator (0.00s) niks3-go-unit-tests> --- PASS: TestCacheConfigHandler/no_signing_keys (0.00s) niks3-go-unit-tests> --- PASS: TestCacheConfigHandler/no_cache_url_configured (0.00s) niks3-go-unit-tests> 2026/10/09 09:58:08 INFO Received push request method=POST path=/api/pushes niks3-go-unit-tests> === CONT TestPush_RejectsBadRequests/bad_root niks3-go-unit-tests> 2026/10/09 09:58:08 INFO Received push request method=POST path=/api/pushes niks3-go-unit-tests> === CONT TestPush_RejectsBadRequests/no_roots niks3-go-unit-tests> 2026/10/09 09:58:08 INFO Received push request method=POST path=/api/pushes niks3-go-unit-tests> === CONT TestPush_RejectsBadRequests/root_not_in_objects niks3-go-unit-tests> 2026/10/09 09:58:08 INFO Received push request method=POST path=/api/pushes niks3-go-unit-tests> === CONT TestService_AuthMiddleware_OIDC/malformed_token_rejected niks3-go-unit-tests> 2026/10/09 09:58:08 WARN Authentication failed token_preview=not-a-valid-jwt token_length=15 oidc_error="no provider could verify the token (signature or issuer mismatch)" oidc_provider="" tried_providers=[test] niks3-go-unit-tests> === CONT TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured niks3-go-unit-tests> === CONT TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected niks3-go-unit-tests> 2026/10/09 09:58:08 WARN Authentication failed token_preview=eyJhbGciOi...Z8feoM4R0Q token_length=701 oidc_error="no rule matched: claim \"repository_owner\" value [otherorg] not in allowed patterns [myorg]" oidc_provider=test tried_providers=[test] niks3-go-unit-tests> === CONT TestService_AuthMiddleware_OIDC/valid_OIDC_token niks3-go-unit-tests> === CONT TestService_RequireScope_OIDC/builder_may_write niks3-go-unit-tests> === CONT TestService_RequireScope_OIDC/static_token_may_write niks3-go-unit-tests> === CONT TestService_RequireScope_OIDC/empty_netrc_password_may_not_read niks3-go-unit-tests> === CONT TestService_RequireScope_OIDC/static_token_may_admin_with_netrc niks3-go-unit-tests> === CONT TestService_RequireScope_OIDC/reader_may_read_with_netrc niks3-go-unit-tests> === CONT TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read niks3-go-unit-tests> === CONT TestService_RequireScope_OIDC/writer_implies_read niks3-go-unit-tests> 2026/10/09 09:58:08 WARN Authentication failed token_preview=wrong token_length=5 oidc_error="no provider could verify the token (signature or issuer mismatch)" oidc_provider="" tried_providers=[test] niks3-go-unit-tests> === CONT TestService_RequireScope_OIDC/anonymous_may_not_read niks3-go-unit-tests> === CONT TestService_RequireScope_OIDC/ops_may_not_write niks3-go-unit-tests> --- PASS: TestPush_RejectsBadRequests (8.38s) niks3-go-unit-tests> --- PASS: TestPush_RejectsBadRequests/no_objects (0.00s) niks3-go-unit-tests> --- PASS: TestPush_RejectsBadRequests/bad_root (0.01s) niks3-go-unit-tests> --- PASS: TestPush_RejectsBadRequests/no_roots (0.00s) niks3-go-unit-tests> --- PASS: TestPush_RejectsBadRequests/root_not_in_objects (0.00s) niks3-go-unit-tests> === CONT TestService_RequireScope_OIDC/static_token_may_admin niks3-go-unit-tests> === CONT TestService_RequireScope_OIDC/reader_may_not_write niks3-go-unit-tests> === CONT TestService_RequireScope_OIDC/builder_may_not_admin niks3-go-unit-tests> === CONT TestService_RequireScope_OIDC/reader_may_read niks3-go-unit-tests> === CONT TestService_RequireScope_OIDC/ops_may_admin niks3-go-unit-tests> === CONT TestClientErrorHandling/InvalidStorePath niks3-go-unit-tests> --- PASS: TestService_AuthMiddleware_OIDC (8.91s) niks3-go-unit-tests> --- PASS: TestService_AuthMiddleware_OIDC/malformed_token_rejected (0.01s) niks3-go-unit-tests> --- PASS: TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured (0.00s) niks3-go-unit-tests> --- PASS: TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected (0.01s) niks3-go-unit-tests> --- PASS: TestService_AuthMiddleware_OIDC/valid_OIDC_token (0.00s) niks3-go-unit-tests> === CONT TestClientErrorHandling/ServerNotAvailable niks3-go-unit-tests> 2026/10/09 09:58:08 INFO Received uploads request method=POST path=/api/pending_closures niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:08.779206Z","level":"ERROR","message":"save_config_with_opts: err: Io(Custom { kind: TooManyOpenFiles, error: \"Too many open files (os error 24)\" }), file: scanner/durable-dirty-producer-replay.json","target":"rustfs_ecstore::config::com","filename":"crates/ecstore/src/config/com.rs","line_number":813,"threadName":"rustfs-worker","threadId":"ThreadId(2)"} niks3-go-unit-tests> === CONT TestClientErrorHandling/InvalidAuthToken niks3-go-unit-tests> --- PASS: TestService_RequireScope_OIDC (9.59s) niks3-go-unit-tests> --- PASS: TestService_RequireScope_OIDC/builder_may_write (0.00s) niks3-go-unit-tests> --- PASS: TestService_RequireScope_OIDC/static_token_may_write (0.00s) niks3-go-unit-tests> --- PASS: TestService_RequireScope_OIDC/empty_netrc_password_may_not_read (0.00s) niks3-go-unit-tests> --- PASS: TestService_RequireScope_OIDC/static_token_may_admin_with_netrc (0.00s) niks3-go-unit-tests> --- PASS: TestService_RequireScope_OIDC/reader_may_read_with_netrc (0.00s) niks3-go-unit-tests> --- PASS: TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read (0.01s) niks3-go-unit-tests> --- PASS: TestService_RequireScope_OIDC/anonymous_may_not_read (0.00s) niks3-go-unit-tests> --- PASS: TestService_RequireScope_OIDC/writer_implies_read (0.00s) niks3-go-unit-tests> --- PASS: TestService_RequireScope_OIDC/static_token_may_admin (0.00s) niks3-go-unit-tests> --- PASS: TestService_RequireScope_OIDC/ops_may_not_write (0.00s) niks3-go-unit-tests> --- PASS: TestService_RequireScope_OIDC/reader_may_not_write (0.00s) niks3-go-unit-tests> --- PASS: TestService_RequireScope_OIDC/builder_may_not_admin (0.00s) niks3-go-unit-tests> --- PASS: TestService_RequireScope_OIDC/reader_may_read (0.00s) niks3-go-unit-tests> --- PASS: TestService_RequireScope_OIDC/ops_may_admin (0.00s) niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:08.906349Z","level":"ERROR","duration":"163.301583ms","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(9)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:08.906473Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"58176551-7cdd-498b-9ba8-72b6e2722e38","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket75/nar/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb00000000000000000000.nar.zst","status_code":500,"suppressed_errors":0,"duration_ms":163,"result":"server_error","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":501,"threadName":"rustfs-worker","threadId":"ThreadId(9)"} niks3-go-unit-tests> === NAME TestPush_RepeatedRootCommitsOnce niks3-go-unit-tests> pushes_test.go:162: expected http status 200 for part 9, got 500 niks3-go-unit-tests> pushes_test.go:162: no ETag in response for part 9 niks3-go-unit-tests> 2026/10/09 09:58:08 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://localhost:19999/api/objects/present\": dial tcp 127.0.0.1:19999: connect: connection refused" url=http://localhost:19999/api/objects/present niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:08.996192Z","level":"ERROR","message":"Scanner bucket cache save failed","event":"scanner_cache_persist_state","component":"scanner","subsystem":"io","bucket":"bucket7","cache_name":"bucket7/.usage-cache.bin","state":"save_failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::scanner::io","filename":"crates/scanner/src/scanner_io/io_cache.rs","line_number":1373,"threadName":"rustfs-worker","threadId":"ThreadId(10)"} niks3-go-unit-tests> 2026/10/09 09:58:09 INFO Received complete multipart upload request method=POST path=/api/multipart/complete niks3-go-unit-tests> 2026/10/09 09:58:09 ERROR Failed to complete multipart upload error="One or more of the specified parts could not be found. The part may not have been uploaded, or the specified entity tag may not match the part's entity tag." object_key=nar/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb00000000000000000000.nar.zst upload_id=MTE5MjdlYzItNjE3OS00ZDA5LTljZjMtZGFiOThmMGQ2ZDQ0LmQ5NzhjZWJlLTVjYjItNDlmYy05MGI3LTQxMDUxZTlkOTUyOHgxNzkxNTM5ODg1NTU0Mzk0MDAw niks3-go-unit-tests> pushes_test.go:162: unexpected http status=500 body=failed to complete multipart upload: One or more of the specified parts could not be found. The part may not have been uploaded, or the specified entity tag may not match the part's entity tag. niks3-go-unit-tests> niks3-go-unit-tests> niks3-go-unit-tests> 2026-10-09 09:58:09.040 UTC [65334] LOG: checkpoint starting: wal niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:09.065359Z","level":"ERROR","duration":"427.833µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(2)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:09.066941Z","level":"ERROR","message":"Disk local scan failed","event":"disk_local_scan_failed","component":"ecstore","subsystem":"disk_local","path":"","operation":"list_dir","error":"Io(Os { code: 24, kind: TooManyOpenFiles, message: \"Too many open files\" })","target":"rustfs_ecstore::disk::local","filename":"crates/ecstore/src/disk/local.rs","line_number":7605,"threadName":"rustfs-worker","threadId":"ThreadId(6)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:09.066989Z","level":"ERROR","message":"Metacache listing quorum failed","event":"metacache_listing","component":"ecstore","subsystem":"metacache","bucket":"bucket85","path":"","state":"quorum_failed","error":"drive /private/var/tb/5/scratch/00f1341bdd59a71e/build/rustfs694584765/data returned: io error Too many open files (os error 24)","target":"rustfs_ecstore::cache_value::metacache_set","filename":"crates/ecstore/src/cache_value/metacache_set.rs","line_number":909,"threadName":"rustfs-worker","threadId":"ThreadId(6)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:09.067008Z","level":"ERROR","message":"Metacache merge job failed","event":"metacache_listing","component":"ecstore","subsystem":"metacache","bucket":"bucket85","path":"","state":"merge_job_failed","error":"ErasureReadQuorum","target":"rustfs_ecstore::cache_value::metacache_set","filename":"crates/ecstore/src/cache_value/metacache_set.rs","line_number":1002,"threadName":"rustfs-worker","threadId":"ThreadId(6)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:09.067056Z","level":"ERROR","message":"list_path worker failed","component":"store","stage":"list_merged","bucket":"bucket85","prefix":"","base_dir":"","limit":1001,"marker":"","error":"erasure read quorum","target":"rustfs_ecstore::store::list_objects","filename":"crates/ecstore/src/store/list_objects.rs","line_number":242,"threadName":"rustfs-worker","threadId":"ThreadId(6)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:09.06707Z","level":"ERROR","message":"list_path worker failed","component":"store","stage":"trailing_worker_error","bucket":"bucket85","prefix":"","base_dir":"","limit":1001,"marker":"","error":"erasure read quorum","target":"rustfs_ecstore::store::list_objects","filename":"crates/ecstore/src/store/list_objects.rs","line_number":242,"threadName":"rustfs-worker","threadId":"ThreadId(6)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:09.067092Z","level":"ERROR","duration":"289.333µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"SlowDownReadResource requested is unreadable, please reduce your request rate\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(6)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:09.067101Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"032970fa-32b0-41ab-9254-e9f2986812d9","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"GET","uri":"/bucket85/","status_code":503,"suppressed_errors":0,"duration_ms":0,"result":"server_error","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":501,"threadName":"rustfs-worker","threadId":"ThreadId(6)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:09.07386Z","level":"ERROR","duration":"32.146625ms","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(6)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:09.082839Z","level":"ERROR","message":"save_config_with_opts: err: Io(Custom { kind: TooManyOpenFiles, error: \"Too many open files (os error 24)\" }), file: scanner/durable-dirty-producer-replay.json","target":"rustfs_ecstore::config::com","filename":"crates/ecstore/src/config/com.rs","line_number":813,"threadName":"rustfs-worker","threadId":"ThreadId(10)"} niks3-go-unit-tests> 2026/10/09 09:58:09 WARN Request failed, retrying attempt=2 backoff=204.056967ms error="Post \"http://localhost:19999/api/objects/present\": dial tcp 127.0.0.1:19999: connect: connection refused" url=http://localhost:19999/api/objects/present niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:09.113487Z","level":"ERROR","duration":"634.583µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(10)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:09.194901Z","level":"ERROR","message":"data usage snapshot load failed","event":"data_usage_snapshot_load_failed","component":"ecstore","subsystem":"data_usage","state":"read_failed","operation":"remove_bucket_from_observed","reason":"io","object":"buckets/.usage.observed.json","error":"Io error: Too many open files (os error 24)","target":"rustfs_ecstore::data_usage","filename":"crates/ecstore/src/data_usage/mod.rs","line_number":968,"threadName":"rustfs-worker","threadId":"ThreadId(10)"} niks3-go-unit-tests> --- FAIL: TestPush_RepeatedRootCommitsOnce (11.14s) niks3-go-unit-tests> === CONT TestProxyWriteTimeout/narinfo niks3-go-unit-tests> === CONT TestProxyWriteTimeout/unknown_size niks3-go-unit-tests> === CONT TestProxyWriteTimeout/10_GiB_nar niks3-go-unit-tests> === CONT TestProxyWriteTimeout/1_GiB_nar niks3-go-unit-tests> === CONT TestServerTLSConfig/no_client_CA niks3-go-unit-tests> --- PASS: TestProxyWriteTimeout (0.01s) niks3-go-unit-tests> --- PASS: TestProxyWriteTimeout/narinfo (0.00s) niks3-go-unit-tests> --- PASS: TestProxyWriteTimeout/unknown_size (0.00s) niks3-go-unit-tests> --- PASS: TestProxyWriteTimeout/10_GiB_nar (0.00s) niks3-go-unit-tests> --- PASS: TestProxyWriteTimeout/1_GiB_nar (0.00s) niks3-go-unit-tests> === CONT TestServerTLSConfig/missing_CA_file niks3-go-unit-tests> === CONT TestServerTLSConfig/not_a_PEM_file niks3-go-unit-tests> --- PASS: TestServerTLSConfig (0.01s) niks3-go-unit-tests> --- PASS: TestServerTLSConfig/no_client_CA (0.00s) niks3-go-unit-tests> --- PASS: TestServerTLSConfig/missing_CA_file (0.00s) niks3-go-unit-tests> --- PASS: TestServerTLSConfig/not_a_PEM_file (0.02s) niks3-go-unit-tests> === CONT TestPresentNotReportedWhileGCDeletesClosure/stale niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:09.254267Z","level":"ERROR","duration":"465.041µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(10)"} niks3-go-unit-tests> 2026/10/09 09:58:09 WARN Request failed, retrying attempt=3 backoff=371.915341ms error="Post \"http://localhost:19999/api/objects/present\": dial tcp 127.0.0.1:19999: connect: connection refused" url=http://localhost:19999/api/objects/present niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:09.302251Z","level":"ERROR","message":"save_config_with_opts: err: Io(Custom { kind: TooManyOpenFiles, error: \"Too many open files (os error 24)\" }), file: scanner/durable-dirty-producer-replay.json","target":"rustfs_ecstore::config::com","filename":"crates/ecstore/src/config/com.rs","line_number":813,"threadName":"rustfs-worker","threadId":"ThreadId(2)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:09.306119Z","level":"ERROR","duration":"19.640667ms","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(2)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:09.306729Z","level":"ERROR","message":"Heal scheduler task failed","event":"heal_scheduler_state","component":"heal","subsystem":"manager","task_id":"bcca5f0b-fd84-47cd-9a6f-ad089df61ac7","heal_type":"object","set":"pool_0_set_0","state":"task_failed","error":"Heal task execution failed: Object not found: bucket86/nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst","target":"rustfs::heal::manager","filename":"crates/heal/src/heal/manager/scheduler.rs","line_number":269,"threadName":"rustfs-worker","threadId":"ThreadId(7)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:09.315758Z","level":"ERROR","duration":"30.256041ms","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(2)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:09.317222Z","level":"ERROR","duration":"1.66725ms","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(7)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:09.359339Z","level":"ERROR","duration":"660µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(10)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:09.40127Z","level":"ERROR","message":"Scanner bucket cache save failed","event":"scanner_cache_persist_state","component":"scanner","subsystem":"io","bucket":"bucket8","cache_name":"bucket8/.usage-cache.bin","state":"save_failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::scanner::io","filename":"crates/scanner/src/scanner_io/io_cache.rs","line_number":1373,"threadName":"rustfs-worker","threadId":"ThreadId(7)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:09.417015Z","level":"ERROR","duration":"443.458µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(10)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:09.442516Z","level":"ERROR","duration":"446.083µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(2)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:09.446557Z","level":"ERROR","duration":"17.875875ms","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(10)"} niks3-go-unit-tests> === NAME TestSweepRechecksPendingAfterLocking niks3-go-unit-tests> gc_races_test.go:635: unexpected error: We encountered an internal error, please try again. niks3-go-unit-tests> niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:09.458605Z","level":"ERROR","duration":"550.916µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(2)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:09.482916Z","level":"ERROR","duration":"484.583µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(9)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:09.641673Z","level":"ERROR","duration":"692.917µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(9)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:09.660286Z","level":"ERROR","duration":"19.308125ms","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(2)"} niks3-go-unit-tests> 2026/10/09 09:58:09 WARN Request failed, retrying attempt=4 backoff=761.908619ms error="Post \"http://localhost:19999/api/objects/present\": dial tcp 127.0.0.1:19999: connect: connection refused" url=http://localhost:19999/api/objects/present niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:09.729025Z","level":"ERROR","duration":"378.583µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(10)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:09.736939Z","level":"ERROR","message":"Scanner bucket cache save failed","event":"scanner_cache_persist_state","component":"scanner","subsystem":"io","bucket":"bucket9","cache_name":"bucket9/.usage-cache.bin","state":"save_failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::scanner::io","filename":"crates/scanner/src/scanner_io/io_cache.rs","line_number":1373,"threadName":"rustfs-worker","threadId":"ThreadId(10)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:09.737048Z","level":"ERROR","message":"Scanner cycle completed without a durable data usage snapshot","event":"scanner_persist_state","component":"scanner","subsystem":"runtime","cycle":6,"outcome":"NoUpdate","state":"usage_not_durable","target":"rustfs::scanner","filename":"crates/scanner/src/scanner.rs","line_number":2350,"threadName":"rustfs-worker","threadId":"ThreadId(9)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:09.808186Z","level":"ERROR","duration":"358.5µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(10)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:09.817506Z","level":"ERROR","duration":"297.292µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(10)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:10.016772Z","level":"ERROR","duration":"373.458µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(9)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:10.09051Z","level":"ERROR","duration":"442.5µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(6)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:10.090629Z","level":"ERROR","duration":"561.875µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(7)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:10.098924Z","level":"ERROR","message":"Scanner bucket cache save failed","event":"scanner_cache_persist_state","component":"scanner","subsystem":"io","bucket":"bucket1","cache_name":"bucket1/.usage-cache.bin","state":"save_failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::scanner::io","filename":"crates/scanner/src/scanner_io/io_cache.rs","line_number":1373,"threadName":"rustfs-worker","threadId":"ThreadId(7)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:10.208967Z","level":"ERROR","duration":"406.875µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(2)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:10.309578Z","level":"ERROR","duration":"14.953792ms","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(2)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:10.34909Z","level":"ERROR","duration":"9.428292ms","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(6)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:10.357989Z","level":"ERROR","message":"Scanner partial bucket cache save failed","event":"scanner_cache_persist_state","component":"scanner","subsystem":"io","bucket":"bucket10","cache_name":"bucket10/.usage-cache.bin","state":"partial_save_failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::scanner::io","filename":"crates/scanner/src/scanner_io/io_cache.rs","line_number":1297,"threadName":"rustfs-worker","threadId":"ThreadId(2)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:10.413311Z","level":"ERROR","duration":"10.794ms","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(10)"} niks3-go-unit-tests> 2026/10/09 09:58:10 WARN Request failed, retrying attempt=5 backoff=1.629358619s error="Post \"http://localhost:19999/api/objects/present\": dial tcp 127.0.0.1:19999: connect: connection refused" url=http://localhost:19999/api/objects/present niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:10.651793Z","level":"ERROR","duration":"425.542µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(9)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:10.656551Z","level":"ERROR","duration":"312.958µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(10)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:10.666423Z","level":"ERROR","message":"Scanner bucket cache save failed","event":"scanner_cache_persist_state","component":"scanner","subsystem":"io","bucket":"bucket11","cache_name":"bucket11/.usage-cache.bin","state":"save_failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::scanner::io","filename":"crates/scanner/src/scanner_io/io_cache.rs","line_number":1373,"threadName":"rustfs-worker","threadId":"ThreadId(2)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:10.707Z","level":"ERROR","duration":"313.25µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(6)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:10.762209Z","level":"ERROR","duration":"10.598833ms","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(6)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:10.827838Z","level":"ERROR","duration":"314.083µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(9)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:10.935614Z","level":"ERROR","message":"Scanner bucket cache save failed","event":"scanner_cache_persist_state","component":"scanner","subsystem":"io","bucket":"bucket12","cache_name":"bucket12/.usage-cache.bin","state":"save_failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::scanner::io","filename":"crates/scanner/src/scanner_io/io_cache.rs","line_number":1373,"threadName":"rustfs-worker","threadId":"ThreadId(6)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:10.983111Z","level":"ERROR","duration":"335.542µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(9)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:11.102635Z","level":"ERROR","duration":"9.827084ms","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(9)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:11.142343Z","level":"ERROR","duration":"345.167µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(10)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:11.174448Z","level":"ERROR","duration":"298.375µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(10)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:11.217036Z","level":"ERROR","duration":"303.958µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(2)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:11.217125Z","level":"ERROR","message":"Scanner bucket cache save failed","event":"scanner_cache_persist_state","component":"scanner","subsystem":"io","bucket":"bucket13","cache_name":"bucket13/.usage-cache.bin","state":"save_failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::scanner::io","filename":"crates/scanner/src/scanner_io/io_cache.rs","line_number":1373,"threadName":"rustfs-worker","threadId":"ThreadId(2)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:11.369277Z","level":"ERROR","duration":"362.875µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(9)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:11.509662Z","level":"ERROR","message":"Scanner bucket cache save failed","event":"scanner_cache_persist_state","component":"scanner","subsystem":"io","bucket":"bucket14","cache_name":"bucket14/.usage-cache.bin","state":"save_failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::scanner::io","filename":"crates/scanner/src/scanner_io/io_cache.rs","line_number":1373,"threadName":"rustfs-worker","threadId":"ThreadId(10)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:11.653783Z","level":"ERROR","duration":"415.25µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(6)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:11.680759Z","level":"ERROR","duration":"336.208µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(10)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:11.813764Z","level":"ERROR","message":"Scanner bucket cache save failed","event":"scanner_cache_persist_state","component":"scanner","subsystem":"io","bucket":"bucket15","cache_name":"bucket15/.usage-cache.bin","state":"save_failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::scanner::io","filename":"crates/scanner/src/scanner_io/io_cache.rs","line_number":1373,"threadName":"rustfs-worker","threadId":"ThreadId(2)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:11.931235Z","level":"ERROR","duration":"333.083µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(2)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:12.013798Z","level":"ERROR","duration":"10.014208ms","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(6)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:12.023709Z","level":"ERROR","duration":"306.875µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(2)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:12.024419Z","level":"ERROR","duration":"268.25µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(2)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:12.087947Z","level":"ERROR","message":"Scanner bucket cache save failed","event":"scanner_cache_persist_state","component":"scanner","subsystem":"io","bucket":"bucket16","cache_name":"bucket16/.usage-cache.bin","state":"save_failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::scanner::io","filename":"crates/scanner/src/scanner_io/io_cache.rs","line_number":1373,"threadName":"rustfs-worker","threadId":"ThreadId(10)"} niks3-go-unit-tests> 2026/10/09 09:58:12 WARN Request failed, retrying attempt=1 backoff=100ms error="Get \"http://localhost:19999/api/cache-config\": dial tcp 127.0.0.1:19999: connect: connection refused" url=http://localhost:19999/api/cache-config niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:12.192889Z","level":"ERROR","duration":"370.125µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(2)"} niks3-go-unit-tests> 2026/10/09 09:58:12 WARN Request failed, retrying attempt=2 backoff=194.154309ms error="Get \"http://localhost:19999/api/cache-config\": dial tcp 127.0.0.1:19999: connect: connection refused" url=http://localhost:19999/api/cache-config niks3-go-unit-tests> 2026/10/09 09:58:12 WARN Request failed, retrying attempt=3 backoff=421.446163ms error="Get \"http://localhost:19999/api/cache-config\": dial tcp 127.0.0.1:19999: connect: connection refused" url=http://localhost:19999/api/cache-config niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:12.408925Z","level":"ERROR","message":"Scanner partial bucket cache save failed","event":"scanner_cache_persist_state","component":"scanner","subsystem":"io","bucket":"bucket17","cache_name":"bucket17/.usage-cache.bin","state":"partial_save_failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::scanner::io","filename":"crates/scanner/src/scanner_io/io_cache.rs","line_number":1297,"threadName":"rustfs-worker","threadId":"ThreadId(10)"} niks3-go-unit-tests> === CONT TestPresentNotReportedWhileGCDeletesClosure/fresh niks3-go-unit-tests> --- PASS: TestUploadHandlersRejectOversizedBody (0.19s) niks3-go-unit-tests> --- PASS: TestUploadHandlersRejectOversizedBody/request_more_parts (0.30s) niks3-go-unit-tests> --- PASS: TestUploadHandlersRejectOversizedBody/complete_multipart (0.32s) niks3-go-unit-tests> --- PASS: TestUploadHandlersRejectOversizedBody/create_pending_closure (4.35s) niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:12.589202Z","level":"ERROR","duration":"422.042µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(10)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:12.669501Z","level":"ERROR","message":"Scanner bucket cache save failed","event":"scanner_cache_persist_state","component":"scanner","subsystem":"io","bucket":"bucket18","cache_name":"bucket18/.usage-cache.bin","state":"save_failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::scanner::io","filename":"crates/scanner/src/scanner_io/io_cache.rs","line_number":1373,"threadName":"rustfs-worker","threadId":"ThreadId(6)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:12.669722Z","level":"ERROR","duration":"13.860125ms","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(2)"} niks3-go-unit-tests> 2026/10/09 09:58:12 WARN Request failed, retrying attempt=4 backoff=735.575204ms error="Get \"http://localhost:19999/api/cache-config\": dial tcp 127.0.0.1:19999: connect: connection refused" url=http://localhost:19999/api/cache-config niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:12.84792Z","level":"ERROR","duration":"1.9455ms","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(6)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:12.873208Z","level":"ERROR","duration":"1.643708ms","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(9)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:12.903528Z","level":"ERROR","duration":"1.385167ms","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(10)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:12.920205Z","level":"ERROR","message":"Scanner bucket cache save failed","event":"scanner_cache_persist_state","component":"scanner","subsystem":"io","bucket":"bucket19","cache_name":"bucket19/.usage-cache.bin","state":"save_failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::scanner::io","filename":"crates/scanner/src/scanner_io/io_cache.rs","line_number":1373,"threadName":"rustfs-worker","threadId":"ThreadId(10)"} niks3-go-unit-tests> === NAME TestService_AuthMiddleware_MTLSProxyHeader niks3-go-unit-tests> request_test.go:70: store invariants: list multipart uploads: We encountered an internal error, please try again. niks3-go-unit-tests> --- FAIL: TestService_AuthMiddleware_MTLSProxyHeader (10.40s) niks3-go-unit-tests> === CONT TestParseSingleRange/multi-range_ignored niks3-go-unit-tests> === CONT TestParseSingleRange/closed niks3-go-unit-tests> === CONT TestParseSingleRange/start_far_past_EOF niks3-go-unit-tests> === CONT TestParseSingleRange/single_byte niks3-go-unit-tests> === CONT TestParseSingleRange/suffix_exceeds_size niks3-go-unit-tests> === CONT TestParseSingleRange/suffix niks3-go-unit-tests> === CONT TestParseSingleRange/start_past_EOF niks3-go-unit-tests> === CONT TestParseSingleRange/open-ended niks3-go-unit-tests> === CONT TestParseSingleRange/malformed_no_dash niks3-go-unit-tests> === CONT TestParseSingleRange/end_clamped_to_size niks3-go-unit-tests> === CONT TestParseSingleRange/malformed_end_before_start niks3-go-unit-tests> === CONT TestParseSingleRange/unknown_unit niks3-go-unit-tests> === CONT TestParseSingleRange/malformed_both_empty niks3-go-unit-tests> === CONT TestParseSingleRange/none niks3-go-unit-tests> === CONT TestIsValidCachePath/nar_zst niks3-go-unit-tests> --- PASS: TestParseSingleRange (0.05s) niks3-go-unit-tests> --- PASS: TestParseSingleRange/multi-range_ignored (0.00s) niks3-go-unit-tests> --- PASS: TestParseSingleRange/closed (0.00s) niks3-go-unit-tests> --- PASS: TestParseSingleRange/start_far_past_EOF (0.00s) niks3-go-unit-tests> --- PASS: TestParseSingleRange/single_byte (0.00s) niks3-go-unit-tests> --- PASS: TestParseSingleRange/suffix_exceeds_size (0.00s) niks3-go-unit-tests> --- PASS: TestParseSingleRange/suffix (0.00s) niks3-go-unit-tests> --- PASS: TestParseSingleRange/start_past_EOF (0.00s) niks3-go-unit-tests> --- PASS: TestParseSingleRange/open-ended (0.00s) niks3-go-unit-tests> --- PASS: TestParseSingleRange/malformed_no_dash (0.00s) niks3-go-unit-tests> --- PASS: TestParseSingleRange/end_clamped_to_size (0.00s) niks3-go-unit-tests> --- PASS: TestParseSingleRange/malformed_end_before_start (0.00s) niks3-go-unit-tests> --- PASS: TestParseSingleRange/unknown_unit (0.00s) niks3-go-unit-tests> --- PASS: TestParseSingleRange/malformed_both_empty (0.00s) niks3-go-unit-tests> --- PASS: TestParseSingleRange/none (0.00s) niks3-go-unit-tests> === CONT TestIsValidCachePath/invalid_char_u niks3-go-unit-tests> === CONT TestIsValidCachePath/short_hash niks3-go-unit-tests> === CONT TestIsValidCachePath/wrong_extension niks3-go-unit-tests> === CONT TestIsValidCachePath/leading_slash niks3-go-unit-tests> === CONT TestIsValidCachePath/random_path niks3-go-unit-tests> === CONT TestIsValidCachePath/empty niks3-go-unit-tests> === CONT TestIsValidCachePath/invalid_char_e niks3-go-unit-tests> === CONT TestIsValidCachePath/traversal_in_middle niks3-go-unit-tests> === CONT TestIsValidCachePath/traversal_parent niks3-go-unit-tests> === CONT TestIsValidCachePath/index.html niks3-go-unit-tests> === CONT TestIsValidCachePath/log niks3-go-unit-tests> === CONT TestIsValidCachePath/realisation niks3-go-unit-tests> === CONT TestIsValidCachePath/ls niks3-go-unit-tests> === CONT TestIsValidCachePath/nar_uncompressed niks3-go-unit-tests> === CONT TestIsValidCachePath/nar_xz niks3-go-unit-tests> === CONT TestIsValidCachePath/nar_bz2 niks3-go-unit-tests> === CONT TestIsValidCachePath/narinfo_all_nix_base32_chars niks3-go-unit-tests> === CONT TestIsValidCachePath/narinfo niks3-go-unit-tests> === CONT TestIsValidCachePath/nix-cache-info niks3-go-unit-tests> === CONT TestResolveDBConnectionString/flag_wins niks3-go-unit-tests> --- PASS: TestIsValidCachePath (0.10s) niks3-go-unit-tests> --- PASS: TestIsValidCachePath/nar_zst (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidCachePath/invalid_char_u (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidCachePath/short_hash (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidCachePath/wrong_extension (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidCachePath/leading_slash (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidCachePath/random_path (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidCachePath/empty (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidCachePath/invalid_char_e (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidCachePath/traversal_in_middle (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidCachePath/traversal_parent (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidCachePath/index.html (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidCachePath/log (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidCachePath/realisation (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidCachePath/ls (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidCachePath/nar_uncompressed (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidCachePath/nar_xz (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidCachePath/nar_bz2 (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidCachePath/narinfo_all_nix_base32_chars (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidCachePath/narinfo (0.00s) niks3-go-unit-tests> --- PASS: TestIsValidCachePath/nix-cache-info (0.00s) niks3-go-unit-tests> === CONT TestResolveDBConnectionString/PGHOST_allows_empty niks3-go-unit-tests> === CONT TestResolveDBConnectionString/nothing_configured niks3-go-unit-tests> === CONT TestResolveDBConnectionString/missing_file_is_an_error niks3-go-unit-tests> === CONT TestResolveDBConnectionString/file_when_flag_empty niks3-go-unit-tests> === CONT TestGCEndsOnShutdown/before_the_run niks3-go-unit-tests> --- PASS: TestResolveDBConnectionString (0.09s) niks3-go-unit-tests> --- PASS: TestResolveDBConnectionString/flag_wins (0.00s) niks3-go-unit-tests> --- PASS: TestResolveDBConnectionString/PGHOST_allows_empty (0.00s) niks3-go-unit-tests> --- PASS: TestResolveDBConnectionString/nothing_configured (0.00s) niks3-go-unit-tests> --- PASS: TestResolveDBConnectionString/missing_file_is_an_error (0.00s) niks3-go-unit-tests> --- PASS: TestResolveDBConnectionString/file_when_flag_empty (0.00s) niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:13.220135Z","level":"ERROR","duration":"536.542µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(9)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:13.226739Z","level":"ERROR","message":"Scanner bucket cache save failed","event":"scanner_cache_persist_state","component":"scanner","subsystem":"io","bucket":"bucket2","cache_name":"bucket2/.usage-cache.bin","state":"save_failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::scanner::io","filename":"crates/scanner/src/scanner_io/io_cache.rs","line_number":1373,"threadName":"rustfs-worker","threadId":"ThreadId(2)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:13.300608Z","level":"ERROR","duration":"528.875µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(10)"} niks3-go-unit-tests> === NAME TestPushWaitsForSweepHoldingRow niks3-go-unit-tests> request_test.go:70: store invariants: list multipart uploads: We encountered an internal error, please try again. niks3-go-unit-tests> --- FAIL: TestPushWaitsForSweepHoldingRow (11.19s) niks3-go-unit-tests> === CONT TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:13.55611Z","level":"ERROR","message":"Scanner bucket cache save failed","event":"scanner_cache_persist_state","component":"scanner","subsystem":"io","bucket":"bucket20","cache_name":"bucket20/.usage-cache.bin","state":"save_failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::scanner::io","filename":"crates/scanner/src/scanner_io/io_cache.rs","line_number":1373,"threadName":"rustfs-worker","threadId":"ThreadId(2)"} niks3-go-unit-tests> 2026/10/09 09:58:13 WARN Request failed, retrying attempt=5 backoff=1.5382196s error="Get \"http://localhost:19999/api/cache-config\": dial tcp 127.0.0.1:19999: connect: connection refused" url=http://localhost:19999/api/cache-config niks3-go-unit-tests> === NAME TestCommitRacingPendingCleanupKeepsObjects niks3-go-unit-tests> gc_races_test.go:737: unexpected error: We encountered an internal error, please try again. niks3-go-unit-tests> niks3-go-unit-tests> --- FAIL: TestCommitRacingPendingCleanupKeepsObjects (10.48s) niks3-go-unit-tests> === CONT TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:13.645351Z","level":"ERROR","duration":"12.946625ms","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(9)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:13.676449Z","level":"ERROR","duration":"11.506333ms","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(10)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:13.726377Z","level":"ERROR","duration":"13.235542ms","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(6)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:13.727981Z","level":"ERROR","duration":"647.75µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(10)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:13.797467Z","level":"ERROR","message":"save_config_with_opts: err: Io(Custom { kind: TooManyOpenFiles, error: \"Too many open files (os error 24)\" }), file: scanner/durable-dirty-producer-replay.json","target":"rustfs_ecstore::config::com","filename":"crates/ecstore/src/config/com.rs","line_number":813,"threadName":"rustfs-worker","threadId":"ThreadId(2)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:13.816348Z","level":"ERROR","duration":"141.041µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(10)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:13.82035Z","level":"ERROR","message":"Scanner bucket cache save failed","event":"scanner_cache_persist_state","component":"scanner","subsystem":"io","bucket":"bucket21","cache_name":"bucket21/.usage-cache.bin","state":"save_failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::scanner::io","filename":"crates/scanner/src/scanner_io/io_cache.rs","line_number":1373,"threadName":"rustfs-worker","threadId":"ThreadId(10)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:14.06371Z","level":"ERROR","duration":"17.459167ms","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(2)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:14.063794Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"5abe50b5-a709-4967-b608-91b657823c83","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket88/","status_code":500,"suppressed_errors":55,"duration_ms":17,"result":"server_error","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":501,"threadName":"rustfs-worker","threadId":"ThreadId(2)"} niks3-go-unit-tests> === NAME TestSweepRechecksPendingAfterLocking niks3-go-unit-tests> request_test.go:70: store invariants: list multipart uploads: We encountered an internal error, please try again. niks3-go-unit-tests> --- FAIL: TestSweepRechecksPendingAfterLocking (10.92s) niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:14.139687Z","level":"ERROR","message":"Scanner partial bucket cache save failed","event":"scanner_cache_persist_state","component":"scanner","subsystem":"io","bucket":"bucket22","cache_name":"bucket22/.usage-cache.bin","state":"partial_save_failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::scanner::io","filename":"crates/scanner/src/scanner_io/io_cache.rs","line_number":1297,"threadName":"rustfs-worker","threadId":"ThreadId(10)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:14.182Z","level":"ERROR","duration":"15.743125ms","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(10)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:14.28822Z","level":"ERROR","duration":"1.840833ms","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(10)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:14.400103Z","level":"ERROR","message":"Scanner bucket cache save failed","event":"scanner_cache_persist_state","component":"scanner","subsystem":"io","bucket":"bucket23","cache_name":"bucket23/.usage-cache.bin","state":"save_failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::scanner::io","filename":"crates/scanner/src/scanner_io/io_cache.rs","line_number":1373,"threadName":"rustfs-worker","threadId":"ThreadId(9)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:14.774624Z","level":"ERROR","message":"Scanner bucket cache save failed","event":"scanner_cache_persist_state","component":"scanner","subsystem":"io","bucket":"bucket24","cache_name":"bucket24/.usage-cache.bin","state":"save_failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::scanner::io","filename":"crates/scanner/src/scanner_io/io_cache.rs","line_number":1373,"threadName":"rustfs-worker","threadId":"ThreadId(2)"} niks3-go-unit-tests> === NAME TestDeleteClosuresRacingPins niks3-go-unit-tests> request_test.go:70: store invariants: list multipart uploads: We encountered an internal error, please try again. niks3-go-unit-tests> --- FAIL: TestDeleteClosuresRacingPins (14.98s) niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:15.106985Z","level":"ERROR","message":"Scanner bucket cache save failed","event":"scanner_cache_persist_state","component":"scanner","subsystem":"io","bucket":"bucket25","cache_name":"bucket25/.usage-cache.bin","state":"save_failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::scanner::io","filename":"crates/scanner/src/scanner_io/io_cache.rs","line_number":1373,"threadName":"rustfs-worker","threadId":"ThreadId(10)"} niks3-go-unit-tests> 2026/10/09 09:58:15 WARN Failed to fetch cache config, uploading without size limit error="fetching cache config: sending request: request failed after retries: Get \"http://localhost:19999/api/cache-config\": dial tcp 127.0.0.1:19999: connect: connection refused" niks3-go-unit-tests> 2026/10/09 09:58:15 WARN Request failed, retrying attempt=1 backoff=100ms error="Get \"http://localhost:19999/api/cache-config\": dial tcp 127.0.0.1:19999: connect: connection refused" url=http://localhost:19999/api/cache-config niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:15.190028Z","level":"ERROR","duration":"15.831084ms","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(6)"} niks3-go-unit-tests> 2026/10/09 09:58:15 WARN Request failed, retrying attempt=2 backoff=196.408927ms error="Get \"http://localhost:19999/api/cache-config\": dial tcp 127.0.0.1:19999: connect: connection refused" url=http://localhost:19999/api/cache-config niks3-go-unit-tests> === NAME TestDeletePinKeepsRowWhenS3Fails niks3-go-unit-tests> pins_test.go:231: unexpected error: context deadline exceeded niks3-go-unit-tests> niks3-go-unit-tests> --- FAIL: TestDeletePinKeepsRowWhenS3Fails (11.96s) niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:15.299942Z","level":"ERROR","duration":"10.468708ms","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(2)"} niks3-go-unit-tests> 2026/10/09 09:58:15 WARN Request failed, retrying attempt=3 backoff=400.780306ms error="Get \"http://localhost:19999/api/cache-config\": dial tcp 127.0.0.1:19999: connect: connection refused" url=http://localhost:19999/api/cache-config niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:15.431598Z","level":"ERROR","message":"Scanner bucket cache save failed","event":"scanner_cache_persist_state","component":"scanner","subsystem":"io","bucket":"bucket26","cache_name":"bucket26/.usage-cache.bin","state":"save_failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::scanner::io","filename":"crates/scanner/src/scanner_io/io_cache.rs","line_number":1373,"threadName":"rustfs-worker","threadId":"ThreadId(2)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:15.431473Z","level":"ERROR","message":"Heal storage request failed","event":"heal_storage_object_io","component":"heal","subsystem":"storage","operation":"object_exists","bucket":".rustfs.sys","object":"buckets/bucket26/.usage-cache.bin","result":"failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::heal::storage","filename":"crates/heal/src/heal/storage.rs","line_number":1339,"threadName":"rustfs-worker","threadId":"ThreadId(6)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:15.431736Z","level":"ERROR","message":"Heal scheduler task failed","event":"heal_scheduler_state","component":"heal","subsystem":"manager","task_id":"da92b888-c534-4274-aff8-bcc57e4e2dd5","heal_type":"object","set":"pool_0_set_0","state":"task_failed","error":"Other error: Io error: Too many open files (os error 24)","target":"rustfs::heal::manager","filename":"crates/heal/src/heal/manager/scheduler.rs","line_number":269,"threadName":"rustfs-worker","threadId":"ThreadId(6)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:15.446414Z","level":"ERROR","duration":"644.708µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(9)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:15.779011Z","level":"ERROR","message":"Heal storage request failed","event":"heal_storage_object_io","component":"heal","subsystem":"storage","operation":"object_exists","bucket":".rustfs.sys","object":"buckets/.usage.v2.json","result":"failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::heal::storage","filename":"crates/heal/src/heal/storage.rs","line_number":1339,"threadName":"rustfs-worker","threadId":"ThreadId(10)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:15.779065Z","level":"ERROR","message":"Heal scheduler task failed","event":"heal_scheduler_state","component":"heal","subsystem":"manager","task_id":"cc604d50-0af9-4b1a-adae-e1db0cc7bcdc","heal_type":"object","set":"pool_0_set_0","state":"task_failed","error":"Other error: Io error: Too many open files (os error 24)","target":"rustfs::heal::manager","filename":"crates/heal/src/heal/manager/scheduler.rs","line_number":269,"threadName":"rustfs-worker","threadId":"ThreadId(10)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:15.779375Z","level":"ERROR","duration":"12.142333ms","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(10)"} niks3-go-unit-tests> 2026/10/09 09:58:15 WARN Request failed, retrying attempt=4 backoff=826.192428ms error="Get \"http://localhost:19999/api/cache-config\": dial tcp 127.0.0.1:19999: connect: connection refused" url=http://localhost:19999/api/cache-config niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:15.87813Z","level":"ERROR","message":"Scanner partial bucket cache save failed","event":"scanner_cache_persist_state","component":"scanner","subsystem":"io","bucket":"bucket27","cache_name":"bucket27/.usage-cache.bin","state":"partial_save_failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::scanner::io","filename":"crates/scanner/src/scanner_io/io_cache.rs","line_number":1297,"threadName":"rustfs-worker","threadId":"ThreadId(2)"} niks3-go-unit-tests> === NAME TestCreatePinRejectsBadInput niks3-go-unit-tests> pins_test.go:189: unexpected error: context deadline exceeded niks3-go-unit-tests> niks3-go-unit-tests> --- FAIL: TestCreatePinRejectsBadInput (12.17s) niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:15.944457Z","level":"ERROR","message":"op returns error","op":"CreateBucket","err":"S3Error(Inner { code: InternalError, message: Some(\"We encountered an internal error, please try again.\"), request_id: None, status_code: None, source: Some(Io(Custom { kind: TooManyOpenFiles, error: \"Too many open files (os error 24)\" })), headers: None })","target":"s3s::ops","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/ops/mod.rs","line_number":456,"threadName":"rustfs-worker","threadId":"ThreadId(9)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:15.944657Z","level":"ERROR","duration":"759.583µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(9)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:16.001576Z","level":"ERROR","message":"op returns error","op":"CreateBucket","err":"S3Error(Inner { code: InternalError, message: Some(\"We encountered an internal error, please try again.\"), request_id: None, status_code: None, source: Some(Io(Custom { kind: TooManyOpenFiles, error: \"Too many open files (os error 24)\" })), headers: None })","target":"s3s::ops","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/ops/mod.rs","line_number":456,"threadName":"rustfs-worker","threadId":"ThreadId(9)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:16.001621Z","level":"ERROR","duration":"398.375µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(9)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:16.037028Z","level":"ERROR","message":"op returns error","op":"CreateBucket","err":"S3Error(Inner { code: InternalError, message: Some(\"We encountered an internal error, please try again.\"), request_id: None, status_code: None, source: Some(Io(Custom { kind: TooManyOpenFiles, error: \"Too many open files (os error 24)\" })), headers: None })","target":"s3s::ops","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/ops/mod.rs","line_number":456,"threadName":"rustfs-worker","threadId":"ThreadId(2)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:16.037076Z","level":"ERROR","duration":"438.25µs","resp":"Response { status: 500, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"InternalErrorWe encountered an internal error, please try again.\", remaining_limit: None } }","target":"s3s::service","filename":"/nix/var/nix/builds/nix-28079-3532540677/rustfs-1.0.0-vendor/source-registry-0/s3s-0.16.0/src/service.rs","line_number":680,"threadName":"rustfs-worker","threadId":"ThreadId(2)"} niks3-go-unit-tests> === NAME TestConcurrentPinUpdatesAgree niks3-go-unit-tests> pins_test.go:111: unexpected error: context deadline exceeded niks3-go-unit-tests> niks3-go-unit-tests> --- FAIL: TestConcurrentPinUpdatesAgree (12.10s) niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:16.047155Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(9)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:16.05199Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(9)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:16.085236Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(9)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:16.263117Z","level":"ERROR","message":"Scanner partial bucket cache save failed","event":"scanner_cache_persist_state","component":"scanner","subsystem":"io","bucket":"bucket28","cache_name":"bucket28/.usage-cache.bin","state":"partial_save_failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::scanner::io","filename":"crates/scanner/src/scanner_io/io_cache.rs","line_number":1297,"threadName":"rustfs-worker","threadId":"ThreadId(9)"} niks3-go-unit-tests> 2026/10/09 09:58:16 WARN Request failed, retrying attempt=5 backoff=1.685071608s error="Get \"http://localhost:19999/api/cache-config\": dial tcp 127.0.0.1:19999: connect: connection refused" url=http://localhost:19999/api/cache-config niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:16.832392Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(2)"} niks3-go-unit-tests> === NAME TestOrphanedObjectsGCStressTest niks3-go-unit-tests> orphaned_objects_gc_test.go:304: unexpected error: context deadline exceeded niks3-go-unit-tests> niks3-go-unit-tests> --- FAIL: TestOrphanedObjectsGCStressTest (12.78s) niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:17.10565Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(2)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:17.153145Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(2)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:17.549796Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(2)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:17.655232Z","level":"ERROR","message":"Scanner bucket cache load or revision lookup failed","event":"scanner_cache_persist_state","component":"scanner","subsystem":"io","bucket":"bucket29","cache_name":"bucket29/.usage-cache.bin","state":"load_or_revision_lookup_failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::scanner::io","filename":"crates/scanner/src/scanner_io/io_cache.rs","line_number":988,"threadName":"rustfs-worker","threadId":"ThreadId(2)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:17.972877Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(9)"} niks3-go-unit-tests> === NAME TestCreatePin_ReservedPins niks3-go-unit-tests> pins_acl_test.go:31: unexpected error: context deadline exceeded niks3-go-unit-tests> niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:18.072599Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(9)"} niks3-go-unit-tests> --- FAIL: TestCreatePin_ReservedPins (13.07s) niks3-go-unit-tests> === NAME TestSweepSkipsRowsLockedByPush niks3-go-unit-tests> gc_races_test.go:514: unexpected error: Put "http://localhost:60965/bucket93/": context deadline exceeded niks3-go-unit-tests> niks3-go-unit-tests> --- FAIL: TestSweepSkipsRowsLockedByPush (13.11s) niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:18.224835Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(9)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:18.225872Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(9)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:18.297935Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(9)"} niks3-go-unit-tests> 2026/10/09 09:58:18 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://localhost:19999/api/pending_closures\": dial tcp 127.0.0.1:19999: connect: connection refused" url=http://localhost:19999/api/pending_closures niks3-go-unit-tests> === NAME TestGetExistingObjectsLocksRows niks3-go-unit-tests> gc_races_test.go:592: unexpected error: context deadline exceeded niks3-go-unit-tests> niks3-go-unit-tests> --- FAIL: TestGetExistingObjectsLocksRows (13.20s) niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:18.39833Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(9)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:18.447294Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(9)"} niks3-go-unit-tests> 2026/10/09 09:58:18 WARN Request failed, retrying attempt=2 backoff=199.679109ms error="Post \"http://localhost:19999/api/pending_closures\": dial tcp 127.0.0.1:19999: connect: connection refused" url=http://localhost:19999/api/pending_closures niks3-go-unit-tests> === NAME TestRegisterCompletedObjectNeedsPendingRow niks3-go-unit-tests> gc_races_test.go:570: unexpected error: context deadline exceeded niks3-go-unit-tests> niks3-go-unit-tests> --- FAIL: TestRegisterCompletedObjectNeedsPendingRow (13.16s) niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:18.492269Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(9)"} niks3-go-unit-tests> 2026/10/09 09:58:18 WARN Request failed, retrying attempt=3 backoff=384.31874ms error="Post \"http://localhost:19999/api/pending_closures\": dial tcp 127.0.0.1:19999: connect: connection refused" url=http://localhost:19999/api/pending_closures niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:18.664821Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(10)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:18.784574Z","level":"ERROR","message":"save_config_with_opts: err: Io(Custom { kind: TooManyOpenFiles, error: \"Too many open files (os error 24)\" }), file: scanner/durable-dirty-producer-replay.json","target":"rustfs_ecstore::config::com","filename":"crates/ecstore/src/config/com.rs","line_number":813,"threadName":"rustfs-worker","threadId":"ThreadId(9)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:18.784731Z","level":"ERROR","message":"Heal storage request failed","event":"heal_storage_object_io","component":"heal","subsystem":"storage","operation":"object_exists","bucket":".rustfs.sys","object":"scanner/durable-dirty-producer-replay.json","result":"failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::heal::storage","filename":"crates/heal/src/heal/storage.rs","line_number":1339,"threadName":"rustfs-worker","threadId":"ThreadId(10)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:18.784775Z","level":"ERROR","message":"Heal scheduler task failed","event":"heal_scheduler_state","component":"heal","subsystem":"manager","task_id":"4ff19040-fd82-440c-ac3d-b94a5a334121","heal_type":"object","set":"pool_0_set_0","state":"task_failed","error":"Other error: Io error: Too many open files (os error 24)","target":"rustfs::heal::manager","filename":"crates/heal/src/heal/manager/scheduler.rs","line_number":269,"threadName":"rustfs-worker","threadId":"ThreadId(10)"} niks3-go-unit-tests> === NAME TestLeadIncumbentAnnouncesAtOnce niks3-go-unit-tests> farm_test.go:198: unexpected error: context deadline exceeded niks3-go-unit-tests> niks3-go-unit-tests> --- FAIL: TestLeadIncumbentAnnouncesAtOnce (13.06s) niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:18.981138Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(9)"} niks3-go-unit-tests> 2026/10/09 09:58:19 WARN Request failed, retrying attempt=4 backoff=745.097326ms error="Post \"http://localhost:19999/api/pending_closures\": dial tcp 127.0.0.1:19999: connect: connection refused" url=http://localhost:19999/api/pending_closures niks3-go-unit-tests> === NAME TestForceGCDuringPushOffersSweptObject/after_presence_check niks3-go-unit-tests> gc_races_test.go:387: unexpected error: context deadline exceeded niks3-go-unit-tests> niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:19.087694Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(9)"} niks3-go-unit-tests> === NAME TestForceGCDuringPushOffersSweptObject/before_pending_rows niks3-go-unit-tests> gc_races_test.go:387: unexpected error: context deadline exceeded niks3-go-unit-tests> niks3-go-unit-tests> --- FAIL: TestForceGCDuringPushOffersSweptObject (0.01s) niks3-go-unit-tests> --- FAIL: TestForceGCDuringPushOffersSweptObject/after_presence_check (12.85s) niks3-go-unit-tests> --- FAIL: TestForceGCDuringPushOffersSweptObject/before_pending_rows (12.62s) niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:19.181073Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(9)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:19.30887Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(9)"} niks3-go-unit-tests> === NAME TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline niks3-go-unit-tests> timeout_test.go:74: unexpected error: context deadline exceeded niks3-go-unit-tests> niks3-go-unit-tests> --- FAIL: TestPendingClosureWriteTimeout (0.01s) niks3-go-unit-tests> --- PASS: TestPendingClosureWriteTimeout/670k_objects (0.00s) niks3-go-unit-tests> --- PASS: TestPendingClosureWriteTimeout/empty (0.00s) niks3-go-unit-tests> --- PASS: TestPendingClosureWriteTimeout/400_objects (0.00s) niks3-go-unit-tests> --- FAIL: TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline (11.86s) niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:19.362912Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(9)"} niks3-go-unit-tests> === NAME TestPendingClosureFailureTracksEveryUpload/refuse_aborts=false niks3-go-unit-tests> multipart_cleanup_test.go:387: unexpected error: context deadline exceeded niks3-go-unit-tests> niks3-go-unit-tests> === NAME TestPendingClosureFailureTracksEveryUpload/refuse_aborts=true niks3-go-unit-tests> multipart_cleanup_test.go:387: unexpected error: context deadline exceeded niks3-go-unit-tests> niks3-go-unit-tests> --- FAIL: TestPendingClosureFailureTracksEveryUpload (0.02s) niks3-go-unit-tests> --- FAIL: TestPendingClosureFailureTracksEveryUpload/refuse_aborts=false (11.73s) niks3-go-unit-tests> --- FAIL: TestPendingClosureFailureTracksEveryUpload/refuse_aborts=true (11.36s) niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:19.378525Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(9)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:19.44275Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(9)"} niks3-go-unit-tests> === NAME TestClientErrorHandling/InvalidStorePath niks3-go-unit-tests> client_errors_test.go:18: unexpected error: context deadline exceeded niks3-go-unit-tests> niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:19.498531Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(2)"} niks3-go-unit-tests> === NAME TestClientErrorHandling/InvalidAuthToken niks3-go-unit-tests> client_errors_test.go:44: unexpected error: context deadline exceeded niks3-go-unit-tests> niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:19.500409Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(2)"} niks3-go-unit-tests> === NAME TestPresentNotReportedWhileGCDeletesClosure/stale niks3-go-unit-tests> present_test.go:122: unexpected error: context deadline exceeded niks3-go-unit-tests> niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:19.553647Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(2)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:19.706857Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(2)"} niks3-go-unit-tests> 2026/10/09 09:58:19 WARN Request failed, retrying attempt=5 backoff=1.545215406s error="Post \"http://localhost:19999/api/pending_closures\": dial tcp 127.0.0.1:19999: connect: connection refused" url=http://localhost:19999/api/pending_closures niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:19.932769Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(2)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:20.040717Z","level":"ERROR","message":"Scanner bucket cache load or revision lookup failed","event":"scanner_cache_persist_state","component":"scanner","subsystem":"io","bucket":"bucket3","cache_name":"bucket3/.usage-cache.bin","state":"load_or_revision_lookup_failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::scanner::io","filename":"crates/scanner/src/scanner_io/io_cache.rs","line_number":988,"threadName":"rustfs-worker","threadId":"ThreadId(2)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:20.534385Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(2)"} niks3-go-unit-tests> --- FAIL: TestClientErrorHandling (0.00s) niks3-go-unit-tests> --- FAIL: TestClientErrorHandling/InvalidStorePath (10.77s) niks3-go-unit-tests> --- FAIL: TestClientErrorHandling/InvalidAuthToken (10.70s) niks3-go-unit-tests> --- PASS: TestClientErrorHandling/ServerNotAvailable (12.61s) niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:21.445644Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(2)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:21.772283Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(2)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:21.805388Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(2)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:21.837495Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(2)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:22.159595Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(9)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:22.461874Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(9)"} niks3-go-unit-tests> === NAME TestPresentNotReportedWhileGCDeletesClosure/fresh niks3-go-unit-tests> present_test.go:122: unexpected error: context deadline exceeded niks3-go-unit-tests> niks3-go-unit-tests> --- FAIL: TestPresentNotReportedWhileGCDeletesClosure (0.01s) niks3-go-unit-tests> --- FAIL: TestPresentNotReportedWhileGCDeletesClosure/stale (10.33s) niks3-go-unit-tests> --- FAIL: TestPresentNotReportedWhileGCDeletesClosure/fresh (10.04s) niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:22.559652Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(9)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:22.574026Z","level":"ERROR","message":"Scanner bucket cache load or revision lookup failed","event":"scanner_cache_persist_state","component":"scanner","subsystem":"io","bucket":"bucket30","cache_name":"bucket30/.usage-cache.bin","state":"load_or_revision_lookup_failed","error":"Io error: Too many open files (os error 24)","target":"rustfs::scanner::io","filename":"crates/scanner/src/scanner_io/io_cache.rs","line_number":988,"threadName":"rustfs-worker","threadId":"ThreadId(9)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:22.667896Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(2)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:22.915968Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(6)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:22.977968Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(6)"} niks3-go-unit-tests> === NAME TestGCEndsOnShutdown/before_the_run niks3-go-unit-tests> gc_shutdown_test.go:34: unexpected error: context deadline exceeded niks3-go-unit-tests> niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:23.27394Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(9)"} niks3-go-unit-tests> === NAME TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark niks3-go-unit-tests> gc_shutdown_test.go:34: unexpected error: context deadline exceeded niks3-go-unit-tests> niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:23.406369Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(9)"} niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:23.552169Z","level":"ERROR","message":"HTTP accept loop state changed","event":"http_accept_loop_state","component":"server","subsystem":"transport","state":"accept_failed","error":"Too many open files (os error 24)","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":1628,"threadName":"rustfs-worker","threadId":"ThreadId(9)"} niks3-go-unit-tests> === NAME TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete niks3-go-unit-tests> gc_shutdown_test.go:34: unexpected error: context deadline exceeded niks3-go-unit-tests> niks3-go-unit-tests> --- FAIL: TestGCEndsOnShutdown (0.02s) niks3-go-unit-tests> --- FAIL: TestGCEndsOnShutdown/before_the_run (10.10s) niks3-go-unit-tests> --- FAIL: TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark (10.08s) niks3-go-unit-tests> --- FAIL: TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete (10.15s) niks3-go-unit-tests> FAIL niks3-go-unit-tests> {"timestamp":"2026-10-09T09:58:23.772481Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:61549","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2354,"threadName":"rustfs-worker","threadId":"ThreadId(7)"} niks3-go-unit-tests> 2026-10-09 09:58:23.945 UTC [65329] LOG: received smart shutdown request niks3-go-unit-tests> 2026/10/09 09:58:33 INFO killed process group pgid=65329 niks3-go-unit-tests> 2026-10-09 09:58:33.953 UTC [66253] FATAL: terminating connection due to unexpected postmaster exit niks3-go-unit-tests> 2026-10-09 09:58:33.953 UTC [66249] FATAL: terminating connection due to unexpected postmaster exit niks3-go-unit-tests> 2026-10-09 09:58:33.953 UTC [66267] FATAL: terminating connection due to unexpected postmaster exit niks3-go-unit-tests> 2026-10-09 09:58:33.953 UTC [66244] FATAL: terminating connection due to unexpected postmaster exit niks3-go-unit-tests> 2026-10-09 09:58:33.953 UTC [66243] FATAL: terminating connection due to unexpected postmaster exit niks3-go-unit-tests> 2026-10-09 09:58:33.953 UTC [66218] FATAL: terminating connection due to unexpected postmaster exit niks3-go-unit-tests> 2026-10-09 09:58:33.953 UTC [66216] FATAL: terminating connection due to unexpected postmaster exit niks3-go-unit-tests> 2026-10-09 09:58:33.953 UTC [66212] FATAL: terminating connection due to unexpected postmaster exit niks3-go-unit-tests> 2026-10-09 09:58:33.953 UTC [66187] FATAL: terminating connection due to unexpected postmaster exit niks3-go-unit-tests> 2026-10-09 09:58:33.956 UTC [66168] FATAL: terminating connection due to unexpected postmaster exit niks3-go-unit-tests> 2026-10-09 09:58:33.957 UTC [66156] FATAL: terminating connection due to unexpected postmaster exit niks3-go-unit-tests> 2026-10-09 09:58:33.961 UTC [66270] FATAL: terminating connection due to unexpected postmaster exit niks3-go-unit-tests> 2026/10/09 09:58:33 ERROR failed to wait for process error="signal: killed" niks3-go-unit-tests> 2026-10-09 09:58:33.953 UTC [66252] FATAL: terminating connection due to unexpected postmaster exit niks3-go-unit-tests> 2026-10-09 09:58:33.953 UTC [66248] FATAL: terminating connection due to unexpected postmaster exit niks3-go-unit-tests> 2026-10-09 09:58:33.953 UTC [66240] FATAL: terminating connection due to unexpected postmaster exit niks3-go-unit-tests> 2026-10-09 09:58:33.953 UTC [66211] FATAL: terminating connection due to unexpected postmaster exit niks3-go-unit-tests> 2026-10-09 09:58:33.953 UTC [66177] FATAL: terminating connection due to unexpected postmaster exit niks3-go-unit-tests> 2026-10-09 09:58:33.953 UTC [66175] FATAL: terminating connection due to unexpected postmaster exit niks3-go-unit-tests> 2026-10-09 09:58:33.962 UTC [66264] FATAL: terminating connection due to unexpected postmaster exit niks3-go-unit-tests> 2026-10-09 09:58:33.953 UTC [66254] FATAL: terminating connection due to unexpected postmaster exit niks3-go-unit-tests> 2026-10-09 09:58:33.953 UTC [66250] FATAL: terminating connection due to unexpected postmaster exit niks3-go-unit-tests> 2026-10-09 09:58:33.962 UTC [66261] FATAL: terminating connection due to unexpected postmaster exit error: Cannot build '/nix/store/5g7hv3a4slzymmk7v935r05pzhqwa9fv-niks3-go-unit-tests.drv'. Reason: builder failed with exit code 1. Output paths: /nix/store/s04si3d5kfh8jfiac41rbwm75z1fcv27-niks3-go-unit-tests Last 25 log lines: > 2026-10-09 09:58:23.945 UTC [65329] LOG: received smart shutdown request > 2026/10/09 09:58:33 INFO killed process group pgid=65329 > 2026-10-09 09:58:33.953 UTC [66253] FATAL: terminating connection due to unexpected postmaster exit > 2026-10-09 09:58:33.953 UTC [66249] FATAL: terminating connection due to unexpected postmaster exit > 2026-10-09 09:58:33.953 UTC [66267] FATAL: terminating connection due to unexpected postmaster exit > 2026-10-09 09:58:33.953 UTC [66244] FATAL: terminating connection due to unexpected postmaster exit > 2026-10-09 09:58:33.953 UTC [66243] FATAL: terminating connection due to unexpected postmaster exit > 2026-10-09 09:58:33.953 UTC [66218] FATAL: terminating connection due to unexpected postmaster exit > 2026-10-09 09:58:33.953 UTC [66216] FATAL: terminating connection due to unexpected postmaster exit > 2026-10-09 09:58:33.953 UTC [66212] FATAL: terminating connection due to unexpected postmaster exit > 2026-10-09 09:58:33.953 UTC [66187] FATAL: terminating connection due to unexpected postmaster exit > 2026-10-09 09:58:33.956 UTC [66168] FATAL: terminating connection due to unexpected postmaster exit > 2026-10-09 09:58:33.957 UTC [66156] FATAL: terminating connection due to unexpected postmaster exit > 2026-10-09 09:58:33.961 UTC [66270] FATAL: terminating connection due to unexpected postmaster exit > 2026/10/09 09:58:33 ERROR failed to wait for process error="signal: killed" > 2026-10-09 09:58:33.953 UTC [66252] FATAL: terminating connection due to unexpected postmaster exit > 2026-10-09 09:58:33.953 UTC [66248] FATAL: terminating connection due to unexpected postmaster exit > 2026-10-09 09:58:33.953 UTC [66240] FATAL: terminating connection due to unexpected postmaster exit > 2026-10-09 09:58:33.953 UTC [66211] FATAL: terminating connection due to unexpected postmaster exit > 2026-10-09 09:58:33.953 UTC [66177] FATAL: terminating connection due to unexpected postmaster exit > 2026-10-09 09:58:33.953 UTC [66175] FATAL: terminating connection due to unexpected postmaster exit > 2026-10-09 09:58:33.962 UTC [66264] FATAL: terminating connection due to unexpected postmaster exit > 2026-10-09 09:58:33.953 UTC [66254] FATAL: terminating connection due to unexpected postmaster exit > 2026-10-09 09:58:33.953 UTC [66250] FATAL: terminating connection due to unexpected postmaster exit > 2026-10-09 09:58:33.962 UTC [66261] FATAL: terminating connection due to unexpected postmaster exit For full logs, run: nix log /nix/store/5g7hv3a4slzymmk7v935r05pzhqwa9fv-niks3-go-unit-tests.drv