niks3-go-unit-tests
checks.x86_64-linux.go-unit-tests
· build #506
· raw
1tribuchet: building on jamie2Running client tests...3=== RUN TestDoServerRequestAttachesToken4=== PAUSE TestDoServerRequestAttachesToken5=== RUN TestUploadBuildLog_FileBodyReplayedOnRetry6=== PAUSE TestUploadBuildLog_FileBodyReplayedOnRetry7=== RUN TestRegisterUploadedObjectReusesConnections8=== PAUSE TestRegisterUploadedObjectReusesConnections9=== RUN TestRunGarbageCollection_FinishedOnAnotherReplica10=== PAUSE TestRunGarbageCollection_FinishedOnAnotherReplica11=== RUN TestRunGarbageCollection_NotFoundAfterLocalRun12=== PAUSE TestRunGarbageCollection_NotFoundAfterLocalRun13=== RUN TestCaseHackSuffix14=== PAUSE TestCaseHackSuffix15=== RUN TestFilterOversizedClosures16=== PAUSE TestFilterOversizedClosures17=== RUN TestUploadMultipart_PartsInParallel18=== PAUSE TestUploadMultipart_PartsInParallel19=== RUN TestUploadMultipart_ReturnsAfterDroppedBody20=== PAUSE TestUploadMultipart_ReturnsAfterDroppedBody21=== RUN TestUploadMultipart_FailedProducerIsNotCompleted22=== PAUSE TestUploadMultipart_FailedProducerIsNotCompleted23=== RUN TestPartSizeForNAR24=== PAUSE TestPartSizeForNAR25=== RUN TestDumpPathCaseHackMatchesNix26--- PASS: TestDumpPathCaseHackMatchesNix (0.03s)27=== RUN TestDumpPathCaseHackCollision28--- PASS: TestDumpPathCaseHackCollision (0.00s)29=== RUN TestDumpPathMatchesNix30=== PAUSE TestDumpPathMatchesNix31=== RUN TestDumpPathSingleFile32=== PAUSE TestDumpPathSingleFile33=== RUN TestDumpPathWriterError34=== PAUSE TestDumpPathWriterError35=== RUN TestEncodeNixBase3236=== PAUSE TestEncodeNixBase3237=== RUN TestEncodeNixBase32WithRealHash38=== PAUSE TestEncodeNixBase32WithRealHash39=== RUN TestConvertHashToNix3240=== PAUSE TestConvertHashToNix3241=== RUN TestGetStorePathHash42=== PAUSE TestGetStorePathHash43=== RUN TestPathInfoHashCompatibility44=== PAUSE TestPathInfoHashCompatibility45=== RUN TestParsePathInfoJSON46=== PAUSE TestParsePathInfoJSON47=== RUN TestParsePathInfoJSONMultiplePaths48=== PAUSE TestParsePathInfoJSONMultiplePaths49=== RUN TestPathInfoCACompatibility50=== PAUSE TestPathInfoCACompatibility51=== RUN TestRateLimiterFeedback52=== PAUSE TestRateLimiterFeedback53=== RUN TestRateLimiterFeedback_400DoesNotCountAsSuccess54=== PAUSE TestRateLimiterFeedback_400DoesNotCountAsSuccess55=== RUN TestRegisterUploadedObject_BoundedAgainstSilentServer56=== PAUSE TestRegisterUploadedObject_BoundedAgainstSilentServer57=== RUN TestRegisterUploadedObject_OutlivesPush58=== PAUSE TestRegisterUploadedObject_OutlivesPush59=== RUN TestResolveStorePath60=== PAUSE TestResolveStorePath61=== RUN TestDoWithRetry_BodyReplayedViaGetBody62=== PAUSE TestDoWithRetry_BodyReplayedViaGetBody63=== RUN TestRetryTimeoutOutlastsMaxRetries64=== PAUSE TestRetryTimeoutOutlastsMaxRetries65=== RUN TestDoWithRetry_FinalResponseBodyReadable66=== PAUSE TestDoWithRetry_FinalResponseBodyReadable67=== RUN TestShellSplit68=== PAUSE TestShellSplit69=== RUN TestShellSplitErrors70=== PAUSE TestShellSplitErrors71=== RUN TestStreamPushReportsEveryPath72=== PAUSE TestStreamPushReportsEveryPath73=== RUN TestStreamPushBatchesUnderLoad74=== PAUSE TestStreamPushBatchesUnderLoad75=== RUN TestStreamPushIsolatesFailures76=== PAUSE TestStreamPushIsolatesFailures77=== RUN TestStreamPushGivesUpOnDeadServer78=== PAUSE TestStreamPushGivesUpOnDeadServer79=== RUN TestStreamPushRequestLine80=== PAUSE TestStreamPushRequestLine81=== RUN TestStreamPushReportsSignatures82=== PAUSE TestStreamPushReportsSignatures83=== RUN TestStreamPushReportsSkippedPaths84=== PAUSE TestStreamPushReportsSkippedPaths85=== RUN TestClientSignaturesByStorePath86=== PAUSE TestClientSignaturesByStorePath87=== RUN TestStreamPushAnswersEveryLineWhenCanceled88=== PAUSE TestStreamPushAnswersEveryLineWhenCanceled89=== RUN TestStreamPushStopsWhenCanceled90=== PAUSE TestStreamPushStopsWhenCanceled91=== RUN TestSetClientTLS92=== PAUSE TestSetClientTLS93=== RUN TestSetClientTLSDoesNotMutateDefaultTransport94=== PAUSE TestSetClientTLSDoesNotMutateDefaultTransport95=== RUN TestSetClientTLSErrors96=== PAUSE TestSetClientTLSErrors97=== RUN TestStaticToken98=== PAUSE TestStaticToken99=== RUN TestFileTokenReadsAndCaches100=== PAUSE TestFileTokenReadsAndCaches101=== RUN TestFileTokenMissing102=== PAUSE TestFileTokenMissing103=== RUN TestFileTokenEmpty104=== PAUSE TestFileTokenEmpty105=== RUN TestScriptTokenNoExpiryRerunsEveryCall106=== PAUSE TestScriptTokenNoExpiryRerunsEveryCall107=== RUN TestScriptTokenCachesUntilRefresh108=== PAUSE TestScriptTokenCachesUntilRefresh109=== RUN TestScriptTokenEmptyToken110=== PAUSE TestScriptTokenEmptyToken111=== RUN TestScriptTokenBadJSON112=== PAUSE TestScriptTokenBadJSON113=== RUN TestScriptTokenScriptFails114=== PAUSE TestScriptTokenScriptFails115=== RUN TestScriptTokenEmptyCommand116=== PAUSE TestScriptTokenEmptyCommand117=== CONT TestDoServerRequestAttachesToken118=== CONT TestRetryTimeoutOutlastsMaxRetries119=== CONT TestEncodeNixBase32120=== CONT TestDoWithRetry_BodyReplayedViaGetBody121=== CONT TestResolveStorePath122=== CONT TestRegisterUploadedObject_OutlivesPush123=== CONT TestRegisterUploadedObject_BoundedAgainstSilentServer124=== CONT TestRateLimiterFeedback_400DoesNotCountAsSuccess125--- PASS: TestResolveStorePath (0.00s)126=== CONT TestStreamPushStopsWhenCanceled127=== CONT TestRateLimiterFeedback128=== CONT TestPathInfoCACompatibility129=== CONT TestParsePathInfoJSONMultiplePaths130=== RUN TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths131=== CONT TestParsePathInfoJSON132=== CONT TestPathInfoHashCompatibility133=== CONT TestGetStorePathHash134=== CONT TestConvertHashToNix32135=== CONT TestUploadMultipart_PartsInParallel136=== CONT TestDumpPathWriterError137=== CONT TestDumpPathSingleFile138=== CONT TestEncodeNixBase32WithRealHash139=== CONT TestDumpPathMatchesNix140=== CONT TestPartSizeForNAR1412026/11/12 00:43:00 WARN Rate limiter enabled after throttle name=server-test rate=5142=== CONT TestUploadMultipart_FailedProducerIsNotCompleted143=== CONT TestStreamPushGivesUpOnDeadServer144=== CONT TestUploadMultipart_ReturnsAfterDroppedBody145=== RUN TestEncodeNixBase32/test_string_hash146=== PAUSE TestEncodeNixBase32/test_string_hash147=== RUN TestEncodeNixBase32/empty_input148--- PASS: TestEncodeNixBase32WithRealHash (0.00s)149=== CONT TestClientSignaturesByStorePath150=== RUN TestConvertHashToNix32/SRI_format_to_Nix32151=== RUN TestPartSizeForNAR/zero_stays_at_minimum152=== PAUSE TestConvertHashToNix32/SRI_format_to_Nix32153=== PAUSE TestPartSizeForNAR/zero_stays_at_minimum154=== RUN TestPartSizeForNAR/small_stays_at_minimum155=== RUN TestGetStorePathHash/valid_store_path156=== PAUSE TestPartSizeForNAR/small_stays_at_minimum157=== RUN TestParsePathInfoJSON/Nix_format158=== PAUSE TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths159=== RUN TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part160=== CONT TestStreamPushAnswersEveryLineWhenCanceled161=== PAUSE TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part162=== RUN TestConvertHashToNix32/already_Nix32_format163=== PAUSE TestConvertHashToNix32/already_Nix32_format164=== RUN TestConvertHashToNix32/invalid_format165=== PAUSE TestConvertHashToNix32/invalid_format166=== RUN TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths167=== PAUSE TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths168=== CONT TestRunGarbageCollection_NotFoundAfterLocalRun169=== CONT TestStreamPushReportsSkippedPaths170=== PAUSE TestEncodeNixBase32/empty_input171=== RUN TestPartSizeForNAR/80_GiB_fits_at_minimum172=== PAUSE TestGetStorePathHash/valid_store_path173--- PASS: TestClientSignaturesByStorePath (0.00s)174=== PAUSE TestPartSizeForNAR/80_GiB_fits_at_minimum175=== RUN TestGetStorePathHash/basename_without_hyphen_should_error176=== RUN TestPathInfoCACompatibility/null_ca_field177=== RUN TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)1782026/11/12 00:43:00 WARN Rate limiter enabled after throttle name=server-test rate=5179=== RUN TestRateLimiterFeedback/429_enables_limiter180=== RUN TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary181=== CONT TestFilterOversizedClosures1822026/11/12 00:43:00 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:37029183=== CONT TestStreamPushReportsSignatures1842026/11/12 00:43:00 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:445171852026/11/12 00:43:00 ERROR Upload failed error="connection refused" count=20186=== PAUSE TestParsePathInfoJSON/Nix_format187=== RUN TestParsePathInfoJSON/Lix_format188--- PASS: TestStreamPushStopsWhenCanceled (0.00s)189--- PASS: TestDoServerRequestAttachesToken (0.01s)1902026/11/12 00:43:00 ERROR Server seems unavailable, giving up on batch untried=17191=== RUN TestPartSizeForNAR/115_GiB_needs_larger_parts1922000/01/01 00:00:00 ERROR Upload failed error="context canceled" count=2193=== PAUSE TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary194=== RUN TestFilterOversizedClosures/no_limit_keeps_everything195=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)196=== PAUSE TestGetStorePathHash/basename_without_hyphen_should_error197=== RUN TestGetStorePathHash/hash_with_invalid_characters_should_error1982026/11/12 00:43:00 ERROR Upload failed error=boom count=1199=== PAUSE TestGetStorePathHash/hash_with_invalid_characters_should_error200=== PAUSE TestPathInfoCACompatibility/null_ca_field201=== CONT TestCaseHackSuffix202=== PAUSE TestRateLimiterFeedback/429_enables_limiter2032026/11/12 00:43:00 WARN Rate limiter backed off name=server-test rate=5204=== PAUSE TestParsePathInfoJSON/Lix_format205=== PAUSE TestPartSizeForNAR/115_GiB_needs_larger_parts2062026/11/12 00:43:00 WARN Request returned retryable status, retrying attempt=2 backoff=0s status=503 url=http://127.0.0.1:37029207=== CONT TestStreamPushRequestLine208=== CONT TestUploadBuildLog_FileBodyReplayedOnRetry209=== CONT TestStreamPushReportsEveryPath2102026/11/12 00:43:00 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:44517211--- PASS: TestStreamPushReportsSignatures (0.03s)2122026/11/12 00:43:00 INFO Garbage collection started213=== CONT TestRunGarbageCollection_FinishedOnAnotherReplica2142026/11/12 00:43:00 ERROR Upload failed error=boom count=1215=== CONT TestStreamPushIsolatesFailures216=== RUN TestPathInfoCACompatibility/old_string_format_-_text217=== PAUSE TestPathInfoCACompatibility/old_string_format_-_text2182026/11/12 00:43:00 ERROR Upload failed error="bad path" count=3219=== CONT TestScriptTokenEmptyCommand220=== CONT TestStreamPushBatchesUnderLoad221=== RUN TestStreamPushBatchesUnderLoad/oneAtATime=false222=== PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=false223=== RUN TestStreamPushBatchesUnderLoad/oneAtATime=true224=== PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=true225=== CONT TestScriptTokenScriptFails226=== RUN TestGetStorePathHash/hash_with_wrong_length_should_error227=== RUN TestParsePathInfoJSON/empty_input228=== PAUSE TestParsePathInfoJSON/empty_input229=== RUN TestParsePathInfoJSON/whitespace_only230=== RUN TestPathInfoHashCompatibility/old_string_format_with_colon231=== RUN TestPartSizeForNAR/1_TiB232=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_colon233=== RUN TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI234=== CONT TestRegisterUploadedObjectReusesConnections235=== PAUSE TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI236=== RUN TestPathInfoHashCompatibility/new_structured_format_with_sha5122372026/11/12 00:43:00 INFO Garbage collection started238--- PASS: TestRegisterUploadedObject_OutlivesPush (0.04s)239--- PASS: TestStreamPushAnswersEveryLineWhenCanceled (0.03s)240=== CONT TestScriptTokenEmptyToken241=== CONT TestScriptTokenNoExpiryRerunsEveryCall242--- PASS: TestStreamPushGivesUpOnDeadServer (0.03s)243--- PASS: TestStreamPushReportsSkippedPaths (0.00s)244--- PASS: TestStreamPushReportsEveryPath (0.00s)245--- PASS: TestDoWithRetry_BodyReplayedViaGetBody (0.04s)246--- PASS: TestUploadMultipart_ReturnsAfterDroppedBody (0.03s)247--- PASS: TestScriptTokenEmptyCommand (0.00s)248--- PASS: TestStreamPushIsolatesFailures (0.00s)249--- PASS: TestScriptTokenScriptFails (0.00s)250=== CONT TestFileTokenReadsAndCaches2512026/11/12 00:43:00 INFO Garbage collection progress phase=running_on_another_replica failed_uploads_deleted=0 old_closures_deleted=0 objects_marked=0 objects_deleted=0 objects_failed=0252=== RUN TestRateLimiterFeedback/503_enables_limiter253=== PAUSE TestRateLimiterFeedback/503_enables_limiter2542026/11/12 00:43:00 WARN Rate limiter enabled after throttle name=s3-test rate=5255=== RUN TestRateLimiterFeedback/200_does_not_enable_limiter2562026/11/12 00:43:00 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:36923257=== PAUSE TestRateLimiterFeedback/200_does_not_enable_limiter258=== RUN TestRateLimiterFeedback/400_does_not_enable_limiter259=== PAUSE TestRateLimiterFeedback/400_does_not_enable_limiter260=== RUN TestPathInfoCACompatibility/old_string_format_-_fixed_recursive261=== PAUSE TestPathInfoCACompatibility/old_string_format_-_fixed_recursive262=== RUN TestPathInfoCACompatibility/new_structured_format_-_text263=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_text264=== RUN TestPathInfoCACompatibility/new_structured_format_-_nar_method265=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_nar_method266=== CONT TestScriptTokenCachesUntilRefresh267=== CONT TestScriptTokenBadJSON268=== CONT TestFileTokenMissing269=== CONT TestSetClientTLSErrors2702026/11/12 00:43:00 WARN Garbage collection finished on another replica. Its statistics are not available here271=== CONT TestStaticToken272=== CONT TestSetClientTLSDoesNotMutateDefaultTransport273=== CONT TestShellSplit274=== CONT TestShellSplitErrors275=== CONT TestSetClientTLS276=== CONT TestConvertHashToNix32/SRI_format_to_Nix32277=== CONT TestConvertHashToNix32/invalid_format278=== CONT TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths279=== CONT TestConvertHashToNix32/already_Nix32_format280=== CONT TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths281=== CONT TestEncodeNixBase32/test_string_hash282=== PAUSE TestParsePathInfoJSON/whitespace_only283=== PAUSE TestPartSizeForNAR/1_TiB284=== PAUSE TestPathInfoHashCompatibility/new_structured_format_with_sha512285=== PAUSE TestFilterOversizedClosures/no_limit_keeps_everything286--- PASS: TestFileTokenReadsAndCaches (0.00s)287=== CONT TestFileTokenEmpty288=== CONT TestDoWithRetry_FinalResponseBodyReadable289=== PAUSE TestGetStorePathHash/hash_with_wrong_length_should_error290=== RUN TestParsePathInfoJSON/invalid_JSON291=== PAUSE TestParsePathInfoJSON/invalid_JSON292=== CONT TestEncodeNixBase32/empty_input2932026/11/12 00:43:00 WARN Rate limiter enabled after throttle name=server-test rate=5294=== CONT TestStreamPushBatchesUnderLoad/oneAtATime=false295=== CONT TestStreamPushBatchesUnderLoad/oneAtATime=true2962026/11/12 00:43:00 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:35443297=== RUN TestSetClientTLSErrors/missing_cert_file298=== RUN TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped299=== PAUSE TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped300=== RUN TestFilterOversizedClosures/all_closures_skipped301=== CONT TestRateLimiterFeedback/200_does_not_enable_limiter3022026/11/12 00:43:00 WARN Rate limiter backed off name=server-test rate=5303=== CONT TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary304=== CONT TestRateLimiterFeedback/400_does_not_enable_limiter305=== PAUSE TestFilterOversizedClosures/all_closures_skipped306=== RUN TestSetClientTLS/rejects_connection_without_client_cert307=== CONT TestPathInfoCACompatibility/null_ca_field308=== CONT TestRateLimiterFeedback/503_enables_limiter309=== CONT TestPathInfoCACompatibility/new_structured_format_-_nar_method310=== PAUSE TestSetClientTLSErrors/missing_cert_file311=== RUN TestSetClientTLSErrors/missing_key_file312=== PAUSE TestSetClientTLSErrors/missing_key_file313=== RUN TestSetClientTLSErrors/missing_ca_file314=== PAUSE TestSetClientTLSErrors/missing_ca_file315=== RUN TestSetClientTLSErrors/invalid_ca_file316=== PAUSE TestSetClientTLSErrors/invalid_ca_file317=== PAUSE TestSetClientTLS/rejects_connection_without_client_cert318=== RUN TestSetClientTLS/succeeds_with_client_cert_and_CA319=== PAUSE TestSetClientTLS/succeeds_with_client_cert_and_CA320=== RUN TestSetClientTLS/preserves_debug_logging_transport321=== PAUSE TestSetClientTLS/preserves_debug_logging_transport3222026/11/12 00:43:00 WARN Rate limiter enabled after throttle name=server-test rate=5323=== CONT TestPathInfoCACompatibility/new_structured_format_-_text3242026/11/12 00:43:00 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:35775325=== CONT TestGetStorePathHash/hash_with_invalid_characters_should_error3262026/11/12 00:43:00 WARN Rate limiter backed off name=server-test rate=5327=== CONT TestGetStorePathHash/hash_with_wrong_length_should_error328=== CONT TestParsePathInfoJSON/whitespace_only329=== CONT TestPathInfoCACompatibility/old_string_format_-_fixed_recursive3302026/11/12 00:43:00 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:44517331=== CONT TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part332=== CONT TestRateLimiterFeedback/429_enables_limiter333--- PASS: TestRunGarbageCollection_NotFoundAfterLocalRun (0.03s)334=== CONT TestGetStorePathHash/valid_store_path335=== CONT TestPathInfoCACompatibility/old_string_format_-_text336=== CONT TestGetStorePathHash/basename_without_hyphen_should_error337=== CONT TestPathInfoHashCompatibility/new_structured_format_with_sha512338=== CONT TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI339=== CONT TestPathInfoHashCompatibility/old_string_format_with_colon340=== CONT TestFilterOversizedClosures/no_limit_keeps_everything341=== CONT TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped342=== CONT TestSetClientTLSErrors/invalid_ca_file3432026/11/12 00:43:00 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb-vm-image nar_size=5000 max_nar_size=20003442026/11/12 00:43:00 WARN Rate limiter enabled after throttle name=server-test rate=5345=== CONT TestFilterOversizedClosures/all_closures_skipped3462026/11/12 00:43:00 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=429 url=http://127.0.0.1:33873347=== CONT TestSetClientTLSErrors/missing_cert_file3482026/11/12 00:43:00 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:445173492026/11/12 00:43:00 WARN Rate limiter backed off name=server-test rate=5350=== CONT TestSetClientTLS/rejects_connection_without_client_cert3512026/11/12 00:43:00 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper nar_size=100 max_nar_size=50352=== CONT TestParsePathInfoJSON/Nix_format353=== CONT TestSetClientTLS/succeeds_with_client_cert_and_CA354=== CONT TestParsePathInfoJSON/invalid_JSON355=== CONT TestParsePathInfoJSON/empty_input356=== CONT TestParsePathInfoJSON/Lix_format357--- PASS: TestRunGarbageCollection_FinishedOnAnotherReplica (0.00s)358--- PASS: TestStaticToken (0.00s)359=== CONT TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)360=== RUN TestPartSizeForNAR/5_TiB_S3_max_object361=== CONT TestSetClientTLSErrors/missing_key_file362=== CONT TestSetClientTLSErrors/missing_ca_file363=== CONT TestSetClientTLS/preserves_debug_logging_transport364--- PASS: TestFileTokenMissing (0.00s)365--- PASS: TestUploadBuildLog_FileBodyReplayedOnRetry (0.00s)366=== PAUSE TestPartSizeForNAR/5_TiB_S3_max_object367=== RUN TestPartSizeForNAR/capped_at_5_GiB368=== PAUSE TestPartSizeForNAR/capped_at_5_GiB369=== CONT TestPartSizeForNAR/zero_stays_at_minimum370--- PASS: TestShellSplit (0.00s)371=== CONT TestPartSizeForNAR/5_TiB_S3_max_object3722026/11/12 00:43:00 WARN Request returned retryable status, retrying attempt=3 backoff=1ms status=502 url=http://127.0.0.1:44517373=== CONT TestPartSizeForNAR/1_TiB374=== CONT TestPartSizeForNAR/capped_at_5_GiB375=== CONT TestPartSizeForNAR/80_GiB_fits_at_minimum376=== CONT TestPartSizeForNAR/115_GiB_needs_larger_parts377--- PASS: TestShellSplitErrors (0.00s)378--- PASS: TestScriptTokenEmptyToken (0.00s)379=== CONT TestPartSizeForNAR/small_stays_at_minimum380--- PASS: TestConvertHashToNix32 (0.00s)381 --- PASS: TestConvertHashToNix32/SRI_format_to_Nix32 (0.00s)382 --- PASS: TestConvertHashToNix32/invalid_format (0.00s)383 --- PASS: TestConvertHashToNix32/already_Nix32_format (0.00s)384--- PASS: TestScriptTokenNoExpiryRerunsEveryCall (0.01s)385--- PASS: TestScriptTokenBadJSON (0.00s)386--- PASS: TestFileTokenEmpty (0.00s)387--- PASS: TestDoWithRetry_FinalResponseBodyReadable (0.00s)388--- PASS: TestSetClientTLSDoesNotMutateDefaultTransport (0.00s)389--- PASS: TestGetStorePathHash (0.04s)390 --- PASS: TestGetStorePathHash/hash_with_invalid_characters_should_error (0.00s)391 --- PASS: TestGetStorePathHash/hash_with_wrong_length_should_error (0.00s)392 --- PASS: TestGetStorePathHash/valid_store_path (0.00s)393 --- PASS: TestGetStorePathHash/basename_without_hyphen_should_error (0.00s)394--- PASS: TestPathInfoCACompatibility (0.04s)395 --- PASS: TestPathInfoCACompatibility/null_ca_field (0.00s)396 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_nar_method (0.00s)397 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_text (0.00s)398 --- PASS: TestPathInfoCACompatibility/old_string_format_-_fixed_recursive (0.00s)399 --- PASS: TestPathInfoCACompatibility/old_string_format_-_text (0.00s)400--- PASS: TestDumpPathSingleFile (0.04s)401--- PASS: TestUploadMultipart_FailedProducerIsNotCompleted (0.01s)402 --- PASS: TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary (0.00s)403 --- PASS: TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part (0.00s)404--- PASS: TestRateLimiterFeedback (0.04s)405 --- PASS: TestRateLimiterFeedback/200_does_not_enable_limiter (0.00s)406 --- PASS: TestRateLimiterFeedback/400_does_not_enable_limiter (0.00s)407 --- PASS: TestRateLimiterFeedback/503_enables_limiter (0.00s)408 --- PASS: TestRateLimiterFeedback/429_enables_limiter (0.00s)409--- PASS: TestFilterOversizedClosures (0.03s)410 --- PASS: TestFilterOversizedClosures/no_limit_keeps_everything (0.00s)411 --- PASS: TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped (0.00s)412 --- PASS: TestFilterOversizedClosures/all_closures_skipped (0.00s)413--- PASS: TestParsePathInfoJSON (0.04s)414 --- PASS: TestParsePathInfoJSON/whitespace_only (0.00s)415 --- PASS: TestParsePathInfoJSON/Nix_format (0.00s)416 --- PASS: TestParsePathInfoJSON/invalid_JSON (0.00s)417 --- PASS: TestParsePathInfoJSON/empty_input (0.00s)418 --- PASS: TestParsePathInfoJSON/Lix_format (0.00s)419--- PASS: TestPathInfoHashCompatibility (0.04s)420 --- PASS: TestPathInfoHashCompatibility/new_structured_format_with_sha512 (0.00s)421 --- PASS: TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI (0.00s)422 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_colon (0.00s)423 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI) (0.00s)424--- PASS: TestPartSizeForNAR (0.05s)425 --- PASS: TestPartSizeForNAR/zero_stays_at_minimum (0.00s)426 --- PASS: TestPartSizeForNAR/5_TiB_S3_max_object (0.00s)427 --- PASS: TestPartSizeForNAR/1_TiB (0.00s)428 --- PASS: TestPartSizeForNAR/capped_at_5_GiB (0.00s)429 --- PASS: TestPartSizeForNAR/80_GiB_fits_at_minimum (0.00s)430 --- PASS: TestPartSizeForNAR/115_GiB_needs_larger_parts (0.00s)431 --- PASS: TestPartSizeForNAR/small_stays_at_minimum (0.00s)432--- PASS: TestParsePathInfoJSONMultiplePaths (0.00s)433 --- PASS: TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths (0.00s)434 --- PASS: TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths (0.00s)435--- PASS: TestEncodeNixBase32 (0.01s)436 --- PASS: TestEncodeNixBase32/test_string_hash (0.00s)437 --- PASS: TestEncodeNixBase32/empty_input (0.00s)438--- PASS: TestStreamPushBatchesUnderLoad (0.00s)439 --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=false (0.01s)440 --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=true (0.01s)4412026/11/12 00:43:00 WARN Request returned retryable status, retrying attempt=4 backoff=1ms status=502 url=http://127.0.0.1:44517442--- PASS: TestScriptTokenCachesUntilRefresh (0.02s)443--- PASS: TestSetClientTLSErrors (0.01s)444 --- PASS: TestSetClientTLSErrors/invalid_ca_file (0.00s)445 --- PASS: TestSetClientTLSErrors/missing_cert_file (0.00s)446 --- PASS: TestSetClientTLSErrors/missing_key_file (0.00s)447 --- PASS: TestSetClientTLSErrors/missing_ca_file (0.00s)4482026/11/12 00:43:00 WARN Request returned retryable status, retrying attempt=5 backoff=1ms status=502 url=http://127.0.0.1:445174492026/11/12 00:43:00 WARN Request returned retryable status, retrying attempt=6 backoff=1ms status=502 url=http://127.0.0.1:445174502026/11/12 00:43:00 WARN Request returned retryable status, retrying attempt=7 backoff=1ms status=502 url=http://127.0.0.1:445174512026/11/12 00:43:00 WARN Request returned retryable status, retrying attempt=8 backoff=1ms status=502 url=http://127.0.0.1:445174522026/11/12 00:43:00 WARN Request returned retryable status, retrying attempt=9 backoff=1ms status=502 url=http://127.0.0.1:44517453--- PASS: TestStreamPushRequestLine (0.03s)4542026/11/12 00:43:00 http: TLS handshake error from 127.0.0.1:46558: remote error: tls: bad certificate4552026/11/12 00:43:00 WARN Request returned retryable status, retrying attempt=10 backoff=1ms status=502 url=http://127.0.0.1:44517456--- PASS: TestCaseHackSuffix (0.03s)457--- PASS: TestSetClientTLS (0.00s)458 --- PASS: TestSetClientTLS/succeeds_with_client_cert_and_CA (0.01s)459 --- PASS: TestSetClientTLS/preserves_debug_logging_transport (0.01s)460 --- PASS: TestSetClientTLS/rejects_connection_without_client_cert (0.02s)4612026/11/12 00:43:00 WARN Request returned retryable status, retrying attempt=11 backoff=1ms status=502 url=http://127.0.0.1:44517462--- PASS: TestRegisterUploadedObjectReusesConnections (0.03s)4632026/11/12 00:43:00 WARN Request returned retryable status, retrying attempt=12 backoff=1ms status=502 url=http://127.0.0.1:445174642026/11/12 00:43:00 WARN Request returned retryable status, retrying attempt=13 backoff=1ms status=502 url=http://127.0.0.1:445174652026/11/12 00:43:00 WARN Request returned retryable status, retrying attempt=14 backoff=1ms status=502 url=http://127.0.0.1:445174662026/11/12 00:43:00 WARN Request returned retryable status, retrying attempt=15 backoff=1ms status=502 url=http://127.0.0.1:44517467--- PASS: TestDumpPathWriterError (0.07s)4682026/11/12 00:43:00 WARN Request returned retryable status, retrying attempt=16 backoff=1ms status=502 url=http://127.0.0.1:445174692026/11/12 00:43:00 WARN Request returned retryable status, retrying attempt=17 backoff=1ms status=502 url=http://127.0.0.1:44517470--- PASS: TestRetryTimeoutOutlastsMaxRetries (0.08s)4712026/11/12 00:43:00 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:38485/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:38485/api/uploads/complete4722026/11/12 00:43:00 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"4732026/11/12 00:43:00 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:38485/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:38485/api/uploads/complete4742026/11/12 00:43:00 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"4752026/11/12 00:43:00 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:38485/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:38485/api/uploads/complete4762026/11/12 00:43:00 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"477--- PASS: TestRegisterUploadedObject_BoundedAgainstSilentServer (0.10s)478--- PASS: TestDumpPathMatchesNix (0.11s)479--- PASS: TestUploadMultipart_PartsInParallel (0.66s)480--- PASS: TestRateLimiterFeedback_400DoesNotCountAsSuccess (1.00s)481PASS482Running server tests...483The files belonging to this database system will be owned by user "nixbld".484This user must also own the server process.485486The database cluster will be initialized with locale "C".487The default database encoding has accordingly been set to "SQL_ASCII".488The default text search configuration will be set to "english".489490Data page checksums are enabled.491492creating directory /build/postgres118328248/data ... ok493creating subdirectories ... ok494selecting dynamic shared memory implementation ... posix495selecting default "max_connections" ... 100496selecting default "shared_buffers" ... 128MB497selecting default time zone ... UTC498creating configuration files ... ok499running bootstrap script ... ok500performing post-bootstrap initialization ... ok501syncing data to disk ... ok502503initdb: warning: enabling "trust" authentication for local connections504initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.505506Success. You can now start the database server using:507508 pg_ctl -D /build/postgres118328248/data -l logfile start509510/build/postgres118328248:5432 - no response5112026-11-12 00:43:02.429 UTC [130] LOG: starting PostgreSQL 18.6 on x86_64-pc-linux-gnu, compiled by clang version 21.1.8, 64-bit5122026-11-12 00:43:02.430 UTC [130] LOG: listening on Unix socket "/build/postgres118328248/.s.PGSQL.5432"5132026-11-12 00:43:02.435 UTC [137] LOG: database system was shut down at 2026-11-12 00:43:02 UTC5142026-11-12 00:43:02.439 UTC [130] LOG: database system is ready to accept connections515/build/postgres118328248:5432 - accepting connections516{"timestamp":"2026-11-12T00:43:02.739631593Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"04268eba-0430-4790-9d3b-c0c0053fd307","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"GET","uri":"/health/ready","status_code":503,"suppressed_errors":0,"duration_ms":0,"result":"server_error","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":502,"threadName":"rustfs-worker","threadId":"ThreadId(392)"}517=== RUN TestAbortSucceeded518=== PAUSE TestAbortSucceeded519=== RUN TestService_AuthMiddleware520=== PAUSE TestService_AuthMiddleware521=== RUN TestService_AuthMiddleware_MTLSProxyHeader522=== PAUSE TestService_AuthMiddleware_MTLSProxyHeader523=== RUN TestService_AuthMiddleware_MTLSBoundSubjects524=== PAUSE TestService_AuthMiddleware_MTLSBoundSubjects525=== RUN TestService_ReadAuthMiddleware526=== PAUSE TestService_ReadAuthMiddleware527=== RUN TestService_AuthMiddleware_OIDC528=== PAUSE TestService_AuthMiddleware_OIDC529=== RUN TestService_RequireScope_OIDC530=== PAUSE TestService_RequireScope_OIDC531=== RUN TestService_ReadScope_PublicByDefault532=== PAUSE TestService_ReadScope_PublicByDefault533=== RUN TestCacheConfigHandler534=== PAUSE TestCacheConfigHandler535=== RUN TestCacheStatsHandler536=== PAUSE TestCacheStatsHandler537=== RUN TestClientCADerivations538=== PAUSE TestClientCADerivations539=== RUN TestClientErrorHandling540=== PAUSE TestClientErrorHandling541=== RUN TestClientIntegration542=== PAUSE TestClientIntegration543=== RUN TestClientMultipleUploads544=== PAUSE TestClientMultipleUploads545=== RUN TestClientWithDependencies546=== PAUSE TestClientWithDependencies547=== RUN TestClientSharedPathCommittedMidPush548=== PAUSE TestClientSharedPathCommittedMidPush549=== RUN TestPinProtectsFromGC550=== PAUSE TestPinProtectsFromGC551=== RUN TestClientPushesUseOnePush552=== PAUSE TestClientPushesUseOnePush553=== RUN TestClientFallsBackToClosures554=== PAUSE TestClientFallsBackToClosures555=== RUN TestDeleteClosuresRacingPins556=== PAUSE TestDeleteClosuresRacingPins557=== RUN TestConcurrentCommitsSharingObjectsDoNotDeadlock558=== PAUSE TestConcurrentCommitsSharingObjectsDoNotDeadlock559=== RUN TestResolveDBConnectionString560=== PAUSE TestResolveDBConnectionString561=== RUN TestConnectWaitsForAPeerMigration562=== PAUSE TestConnectWaitsForAPeerMigration563=== RUN TestLeadElectsOneAndHandsOver564=== PAUSE TestLeadElectsOneAndHandsOver565=== RUN TestLeadIncumbentWinsAfterRestart5662026/11/12 00:43:02 INFO lead: acquired remote=192.0.2.1:1234 generation=15672026/11/12 00:43:03 INFO lead: released remote=192.0.2.1:1234 generation=15682026/11/12 00:43:03 INFO lead: acquired remote=192.0.2.1:1234 generation=25692026/11/12 00:43:04 INFO lead: released remote=192.0.2.1:1234 generation=2570--- PASS: TestLeadIncumbentWinsAfterRestart (1.89s)571=== RUN TestLeadUnannouncedStandbyYieldsToIncumbent572=== PAUSE TestLeadUnannouncedStandbyYieldsToIncumbent573=== RUN TestLeadIncumbentAnnouncesAtOnce574=== PAUSE TestLeadIncumbentAnnouncesAtOnce575=== RUN TestLeadGenerationGrowsWithEveryGrant576=== PAUSE TestLeadGenerationGrowsWithEveryGrant577=== RUN TestLeadEndsOnShutdown578=== PAUSE TestLeadEndsOnShutdown579=== RUN TestLeadEndsWhenItsConnectionHangs5802026/11/12 00:43:04 INFO lead: acquired remote=192.0.2.1:1234 generation=15812026-11-12 00:43:06.262 UTC [576] FATAL: terminating connection due to administrator command5822026/11/12 00:43:06 INFO lead: acquired remote=192.0.2.1:1234 generation=25832026/11/12 00:43:07 WARN lead: lock connection lost error="timeout: context deadline exceeded"5842026/11/12 00:43:07 INFO lead: released remote=192.0.2.1:1234 generation=15852026/11/12 00:43:07 INFO lead: released remote=192.0.2.1:1234 generation=2586--- PASS: TestLeadEndsWhenItsConnectionHangs (3.12s)587=== RUN TestGCAdvisoryLockBlocksConcurrentRun588=== PAUSE TestGCAdvisoryLockBlocksConcurrentRun589=== RUN TestGCBugBareHashReferences590=== PAUSE TestGCBugBareHashReferences591=== RUN TestGCMetrics592=== PAUSE TestGCMetrics593=== RUN TestPushDedupSurvivesConcurrentGC594=== PAUSE TestPushDedupSurvivesConcurrentGC595=== RUN TestDeduplicatedObjectsRecordedAsPending596=== PAUSE TestDeduplicatedObjectsRecordedAsPending597=== RUN TestGCSweepSkipsPendingObjects598=== PAUSE TestGCSweepSkipsPendingObjects599=== RUN TestTombstonedObjectOfferedWithoutWaiting600=== PAUSE TestTombstonedObjectOfferedWithoutWaiting601=== RUN TestGCSweepDeliversEachKeyOnce602=== PAUSE TestGCSweepDeliversEachKeyOnce603=== RUN TestCreatePendingClosureVerifyS3FailureReleasesConnection604=== PAUSE TestCreatePendingClosureVerifyS3FailureReleasesConnection605=== RUN TestForceGCDuringPushOffersSweptObject606=== PAUSE TestForceGCDuringPushOffersSweptObject607=== RUN TestFailedS3DeleteKeepsTombstone608=== PAUSE TestFailedS3DeleteKeepsTombstone609=== RUN TestSweepSkipsRowsLockedByPush610=== PAUSE TestSweepSkipsRowsLockedByPush611=== RUN TestRegisterCompletedObjectNeedsPendingRow612=== PAUSE TestRegisterCompletedObjectNeedsPendingRow613=== RUN TestGetExistingObjectsLocksRows614=== PAUSE TestGetExistingObjectsLocksRows615=== RUN TestSweepRechecksPendingAfterLocking616=== PAUSE TestSweepRechecksPendingAfterLocking617=== RUN TestPushWaitsForSweepHoldingRow618=== PAUSE TestPushWaitsForSweepHoldingRow619=== RUN TestCommitRacingPendingCleanupKeepsObjects620=== PAUSE TestCommitRacingPendingCleanupKeepsObjects621=== RUN TestGCEndsOnShutdown622=== PAUSE TestGCEndsOnShutdown623=== RUN TestGCTaskStore_StartNew624=== PAUSE TestGCTaskStore_StartNew625=== RUN TestGCTaskStore_DeduplicateSameParams626=== PAUSE TestGCTaskStore_DeduplicateSameParams627=== RUN TestGCTaskStore_ConflictDifferentParams628=== PAUSE TestGCTaskStore_ConflictDifferentParams629=== RUN TestGCTaskStore_GetEmpty630=== PAUSE TestGCTaskStore_GetEmpty631=== RUN TestGCTaskStore_GetReturnsLatest632=== PAUSE TestGCTaskStore_GetReturnsLatest633=== RUN TestGCTaskStore_CompletedAllowsNewTask634=== PAUSE TestGCTaskStore_CompletedAllowsNewTask635=== RUN TestGCTaskStore_PhaseUpdates636=== PAUSE TestGCTaskStore_PhaseUpdates637=== RUN TestGCTaskStore_Fail638=== PAUSE TestGCTaskStore_Fail639=== RUN TestGracefulShutdownDrainsInflight640=== PAUSE TestGracefulShutdownDrainsInflight641=== RUN TestService_healthCheckHandler642=== PAUSE TestService_healthCheckHandler643=== RUN TestService_readinessHandler644=== PAUSE TestService_readinessHandler645=== RUN TestGenerateLandingPage646=== PAUSE TestGenerateLandingPage647=== RUN TestEnsureAbortUploadsRule648=== PAUSE TestEnsureAbortUploadsRule649=== RUN TestCacheConfigHandlerMaxNarSize650=== PAUSE TestCacheConfigHandlerMaxNarSize651=== RUN TestCreatePendingClosureRejectsOversizedNAR652=== PAUSE TestCreatePendingClosureRejectsOversizedNAR653=== RUN TestNARDeduplicationMetadataUploadBug654=== PAUSE TestNARDeduplicationMetadataUploadBug655=== RUN TestMetricsInventory656=== PAUSE TestMetricsInventory657=== RUN TestMetricsCommit658=== PAUSE TestMetricsCommit659=== RUN TestService_NativeMTLS660=== PAUSE TestService_NativeMTLS661=== RUN TestServerTLSConfig662=== PAUSE TestServerTLSConfig663=== RUN TestMultipartCleanup664=== PAUSE TestMultipartCleanup665=== RUN TestMultipartUploadAbortedWhenCancelledBeforeRecorded666=== PAUSE TestMultipartUploadAbortedWhenCancelledBeforeRecorded667=== RUN TestPendingCleanupUsesOneCutoff668=== PAUSE TestPendingCleanupUsesOneCutoff669=== RUN TestObjectStatsTrigger670=== PAUSE TestObjectStatsTrigger671=== RUN TestObjectStatsUpdatedOncePerCommit672=== PAUSE TestObjectStatsUpdatedOncePerCommit673=== RUN TestOrphanedObjectsGC674=== PAUSE TestOrphanedObjectsGC675=== RUN TestOrphanedObjectsGCStressTest676=== PAUSE TestOrphanedObjectsGCStressTest677=== RUN TestCreatePin_ReservedPins678=== PAUSE TestCreatePin_ReservedPins679=== RUN TestPresentReportsOnlyClosureRoots680=== PAUSE TestPresentReportsOnlyClosureRoots681=== RUN TestPresentRefreshesOnlyStaleClosures682=== PAUSE TestPresentRefreshesOnlyStaleClosures683=== RUN TestPresentNotReportedWhileGCDeletesClosure684=== PAUSE TestPresentNotReportedWhileGCDeletesClosure685=== RUN TestParseSingleRange686=== PAUSE TestParseSingleRange687=== RUN TestProxyHeadersOnlyTrustedOnSocket6882026/11/12 00:43:07 INFO Starting HTTP server address=127.0.0.1:438616892026/11/12 00:43:07 INFO Starting HTTP server address=proxy.sock6902026/11/12 00:43:07 WARN mTLS auth: subject not in bound subjects subject="CN=someone"6912026/11/12 00:43:07 INFO Shutdown signal received, draining in-flight requests timeout=10s692--- PASS: TestProxyHeadersOnlyTrustedOnSocket (0.12s)693=== RUN TestIsValidCachePath694=== PAUSE TestIsValidCachePath695=== RUN TestReadProxyNarinfo696=== PAUSE TestReadProxyNarinfo697=== RUN TestReadProxyNarinfoAlreadyDecompressed698=== PAUSE TestReadProxyNarinfoAlreadyDecompressed699=== RUN TestReadProxyContentNegotiation700=== PAUSE TestReadProxyContentNegotiation701=== RUN TestReadProxyNarStreaming702=== PAUSE TestReadProxyNarStreaming703=== RUN TestReadProxy404704=== PAUSE TestReadProxy404705=== RUN TestReadProxyInvalidPath706=== PAUSE TestReadProxyInvalidPath707=== RUN TestReadProxyHead708=== PAUSE TestReadProxyHead709=== RUN TestReadProxyOutlastsServerWriteTimeout710=== PAUSE TestReadProxyOutlastsServerWriteTimeout711=== RUN TestReadProxyConditionalGet712=== PAUSE TestReadProxyConditionalGet713=== RUN TestReadProxyRootRedirectsToIndexHTML714=== PAUSE TestReadProxyRootRedirectsToIndexHTML715=== RUN TestReadProxyDisabled716=== PAUSE TestReadProxyDisabled717=== RUN TestReadRedirectNar718=== PAUSE TestReadRedirectNar719=== RUN TestReadRedirectKeepsNarinfoProxied720=== PAUSE TestReadRedirectKeepsNarinfoProxied721=== RUN TestReadProxyRangeRequest722=== PAUSE TestReadProxyRangeRequest723=== RUN TestReadRedirectUsesPublicS3URL724=== PAUSE TestReadRedirectUsesPublicS3URL725=== RUN TestPush_OverlappingRootsStoreOneRowPerKey726=== PAUSE TestPush_OverlappingRootsStoreOneRowPerKey727=== RUN TestPush_CompleteCommitsEveryRoot728=== PAUSE TestPush_CompleteCommitsEveryRoot729=== RUN TestPush_RepeatedRootCommitsOnce730=== PAUSE TestPush_RepeatedRootCommitsOnce731=== RUN TestPush_CompleteWaitsForCleanupOfThePush732=== PAUSE TestPush_CompleteWaitsForCleanupOfThePush733=== RUN TestPush_RootsAreLockedInByteOrder734=== PAUSE TestPush_RootsAreLockedInByteOrder735=== RUN TestPush_SkippedKeySurvivesGCBeforeCommit736=== PAUSE TestPush_SkippedKeySurvivesGCBeforeCommit737=== RUN TestPush_RejectsBadRequests738=== PAUSE TestPush_RejectsBadRequests739=== RUN TestPush_SignsNarinfosOfItsPendingObjects740=== PAUSE TestPush_SignsNarinfosOfItsPendingObjects741=== RUN TestCommitCanBeRepeatedUntilCleanup742=== PAUSE TestCommitCanBeRepeatedUntilCleanup743=== RUN TestSharedNarBothUploadsComplete744=== PAUSE TestSharedNarBothUploadsComplete745=== RUN TestCompleteMultipartUpload_ErrorButObjectExists746=== PAUSE TestCompleteMultipartUpload_ErrorButObjectExists747=== RUN TestCompletedNarNotReofferedAcrossClosures748=== PAUSE TestCompletedNarNotReofferedAcrossClosures749=== RUN TestPresignedUploadRegisteredBeforeCommit750=== PAUSE TestPresignedUploadRegisteredBeforeCommit751=== RUN TestService_Rustfstest752=== PAUSE TestService_Rustfstest753=== RUN TestParseSize754=== PAUSE TestParseSize755=== RUN TestSkippedUploadsHandler756=== PAUSE TestSkippedUploadsHandler757=== RUN TestSystemdListenerNotActivated758--- PASS: TestSystemdListenerNotActivated (0.00s)759=== RUN TestWatchdogBeatsWhenHealthy760--- PASS: TestWatchdogBeatsWhenHealthy (0.02s)761=== RUN TestWatchdogSkipsWhenUnhealthy7622026/11/12 00:43:08 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7632026/11/12 00:43:08 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7642026/11/12 00:43:08 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7652026/11/12 00:43:08 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7662026/11/12 00:43:08 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7672026/11/12 00:43:08 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7682026/11/12 00:43:08 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7692026/11/12 00:43:08 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7702026/11/12 00:43:08 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7712026/11/12 00:43:08 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"772--- PASS: TestWatchdogSkipsWhenUnhealthy (0.20s)773=== RUN TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle774=== PAUSE TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle775=== RUN TestProxyWriteTimeout776=== PAUSE TestProxyWriteTimeout777=== RUN TestPendingClosureWriteTimeout778=== PAUSE TestPendingClosureWriteTimeout779=== RUN TestIsValidUploadKey780=== PAUSE TestIsValidUploadKey781=== RUN TestUploadHandlersRejectInvalidKeys782=== PAUSE TestUploadHandlersRejectInvalidKeys783=== RUN TestUploadHandlersRejectOversizedBody784=== PAUSE TestUploadHandlersRejectOversizedBody785=== RUN TestService_cleanupPendingClosuresHandler786=== PAUSE TestService_cleanupPendingClosuresHandler787=== RUN TestService_createPendingClosureHandler788=== PAUSE TestService_createPendingClosureHandler789=== RUN TestService_verifyS3Integrity790=== PAUSE TestService_verifyS3Integrity791=== RUN TestService_verifyS3ReuploadsStaleNarinfo792=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo793=== RUN TestCompleteMultipartUnregistered794=== PAUSE TestCompleteMultipartUnregistered795=== RUN TestCreatePendingClosure_SmallNARUsesSimplePUT796=== PAUSE TestCreatePendingClosure_SmallNARUsesSimplePUT797=== CONT TestAbortSucceeded798=== CONT TestNARDeduplicationMetadataUploadBug799--- PASS: TestAbortSucceeded (0.00s)800=== CONT TestCreatePendingClosureRejectsOversizedNAR801=== CONT TestCacheConfigHandlerMaxNarSize802=== CONT TestEnsureAbortUploadsRule803--- PASS: TestCacheConfigHandlerMaxNarSize (0.00s)804=== CONT TestForceGCDuringPushOffersSweptObject8052026/11/12 00:43:08 INFO Received uploads request method=POST path=/api/pending_closures806=== RUN TestForceGCDuringPushOffersSweptObject/before_pending_rows807=== CONT TestService_readinessHandler808=== CONT TestService_healthCheckHandler809=== CONT TestGracefulShutdownDrainsInflight810=== CONT TestGCTaskStore_Fail811=== CONT TestGCTaskStore_PhaseUpdates812=== CONT TestGCTaskStore_CompletedAllowsNewTask813=== CONT TestGCTaskStore_GetReturnsLatest814=== CONT TestGCTaskStore_GetEmpty815=== CONT TestGCTaskStore_ConflictDifferentParams816=== CONT TestGCTaskStore_DeduplicateSameParams817=== CONT TestGCTaskStore_StartNew818=== CONT TestGCEndsOnShutdown819=== CONT TestCommitRacingPendingCleanupKeepsObjects820=== CONT TestPushWaitsForSweepHoldingRow821=== CONT TestSweepRechecksPendingAfterLocking822=== CONT TestGetExistingObjectsLocksRows823=== CONT TestRegisterCompletedObjectNeedsPendingRow824=== CONT TestSweepSkipsRowsLockedByPush825=== CONT TestFailedS3DeleteKeepsTombstone826=== CONT TestGenerateLandingPage827=== PAUSE TestForceGCDuringPushOffersSweptObject/before_pending_rows828=== RUN TestForceGCDuringPushOffersSweptObject/after_presence_check829=== RUN TestGCEndsOnShutdown/before_the_run830=== PAUSE TestGCEndsOnShutdown/before_the_run831=== RUN TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete832=== PAUSE TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete833=== RUN TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark834=== PAUSE TestForceGCDuringPushOffersSweptObject/after_presence_check8352026/11/12 00:43:08 INFO Starting HTTP server address=127.0.0.1:34369836=== CONT TestCreatePendingClosureVerifyS3FailureReleasesConnection837=== CONT TestGCSweepDeliversEachKeyOnce838--- PASS: TestCreatePendingClosureRejectsOversizedNAR (0.00s)839--- PASS: TestGCTaskStore_GetReturnsLatest (0.00s)840=== CONT TestGCSweepSkipsPendingObjects841=== CONT TestDeduplicatedObjectsRecordedAsPending8422026/11/12 00:43:08 INFO Shutdown signal received, draining in-flight requests timeout=10s843=== CONT TestPushDedupSurvivesConcurrentGC844=== CONT TestGCMetrics845=== CONT TestGCBugBareHashReferences846=== PAUSE TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark847=== CONT TestGCAdvisoryLockBlocksConcurrentRun848=== CONT TestLeadEndsOnShutdown849=== CONT TestTombstonedObjectOfferedWithoutWaiting850--- PASS: TestGCTaskStore_CompletedAllowsNewTask (0.00s)851--- PASS: TestGCTaskStore_Fail (0.00s)852--- PASS: TestGCTaskStore_DeduplicateSameParams (0.00s)853--- PASS: TestGCTaskStore_GetEmpty (0.00s)854--- PASS: TestGCTaskStore_ConflictDifferentParams (0.00s)855--- PASS: TestGCTaskStore_PhaseUpdates (0.00s)856--- PASS: TestGCTaskStore_StartNew (0.00s)857=== CONT TestLeadGenerationGrowsWithEveryGrant858--- PASS: TestGenerateLandingPage (0.01s)859=== CONT TestLeadIncumbentAnnouncesAtOnce860--- PASS: TestGracefulShutdownDrainsInflight (0.07s)861=== CONT TestLeadUnannouncedStandbyYieldsToIncumbent8622026/11/12 00:43:08 INFO Received uploads request method=POST path=/api/pending_closures8632026/11/12 00:43:08 INFO Aborted multipart uploads count=0 kept=08642026/11/12 00:43:08 WARN Force mode enabled - objects will be deleted immediately without grace period865--- PASS: TestService_healthCheckHandler (0.47s)866=== CONT TestLeadElectsOneAndHandsOver8672026/11/12 00:43:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=08682026/11/12 00:43:08 INFO Vacuumed table table=pending_closures8692026/11/12 00:43:08 INFO Vacuumed table table=commit_receipts8702026/11/12 00:43:08 INFO Vacuumed table table=pending_objects8712026/11/12 00:43:08 INFO Vacuumed table table=multipart_uploads8722026/11/12 00:43:08 INFO Vacuumed table table=closures8732026/11/12 00:43:08 INFO Vacuumed table table=objects874--- PASS: TestCommitRacingPendingCleanupKeepsObjects (0.49s)875=== CONT TestConnectWaitsForAPeerMigration8762026/11/12 00:43:08 WARN readiness check failed error="closed pool"877--- PASS: TestService_readinessHandler (0.50s)878=== CONT TestResolveDBConnectionString879=== RUN TestResolveDBConnectionString/flag_wins880=== PAUSE TestResolveDBConnectionString/flag_wins881=== RUN TestResolveDBConnectionString/file_when_flag_empty882=== PAUSE TestResolveDBConnectionString/file_when_flag_empty883=== RUN TestResolveDBConnectionString/missing_file_is_an_error884=== PAUSE TestResolveDBConnectionString/missing_file_is_an_error885=== RUN TestResolveDBConnectionString/PGHOST_allows_empty886=== PAUSE TestResolveDBConnectionString/PGHOST_allows_empty887=== RUN TestResolveDBConnectionString/nothing_configured888=== PAUSE TestResolveDBConnectionString/nothing_configured889=== CONT TestConcurrentCommitsSharingObjectsDoNotDeadlock890--- PASS: TestRegisterCompletedObjectNeedsPendingRow (0.54s)891=== CONT TestDeleteClosuresRacingPins8922026/11/12 00:43:08 INFO Aborted multipart uploads count=0 kept=08932026/11/12 00:43:08 WARN Force mode enabled - objects will be deleted immediately without grace period8942026/11/12 00:43:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=08952026/11/12 00:43:08 INFO Vacuumed table table=pending_closures8962026/11/12 00:43:08 INFO Vacuumed table table=commit_receipts8972026/11/12 00:43:08 INFO Vacuumed table table=pending_objects8982026/11/12 00:43:08 INFO Vacuumed table table=multipart_uploads8992026/11/12 00:43:08 INFO Vacuumed table table=closures9002026/11/12 00:43:08 INFO Vacuumed table table=objects901--- PASS: TestSweepRechecksPendingAfterLocking (0.63s)902=== CONT TestClientFallsBackToClosures9032026/11/12 00:43:08 INFO Aborted multipart uploads count=0 kept=09042026/11/12 00:43:08 INFO Installed lifecycle rule aborting incomplete NAR uploads bucket=bucket11 days=2905--- PASS: TestEnsureAbortUploadsRule (0.64s)906=== CONT TestClientPushesUseOnePush9072026/11/12 00:43:08 WARN Force mode enabled - objects will be deleted immediately without grace period908--- PASS: TestCreatePendingClosureVerifyS3FailureReleasesConnection (0.67s)909=== CONT TestPinProtectsFromGC9102026/11/12 00:43:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=09112026/11/12 00:43:08 INFO Vacuumed table table=pending_closures9122026/11/12 00:43:08 INFO Vacuumed table table=commit_receipts9132026/11/12 00:43:08 INFO Vacuumed table table=pending_objects9142026/11/12 00:43:08 INFO Vacuumed table table=multipart_uploads9152026/11/12 00:43:08 INFO Vacuumed table table=closures9162026/11/12 00:43:08 INFO Received uploads request method=POST path=/api/pending_closures9172026/11/12 00:43:08 INFO Vacuumed table table=objects918--- PASS: TestGCSweepDeliversEachKeyOnce (0.69s)919=== CONT TestClientSharedPathCommittedMidPush920--- PASS: TestTombstonedObjectOfferedWithoutWaiting (0.69s)921=== CONT TestClientWithDependencies922=== NAME TestNARDeduplicationMetadataUploadBug923 metadata_upload_test.go:48: First store path: /build/TestNARDeduplicationMetadataUploadBug2491940307/001/store/rycw9z7aryirx99s5yjz94fm3zk5bnzp-file1.txt9242026/11/12 00:43:08 INFO Received uploads request method=POST path=/api/pending_closures925--- PASS: TestDeduplicatedObjectsRecordedAsPending (0.74s)926=== CONT TestClientMultipleUploads9272026/11/12 00:43:08 INFO Aborted multipart uploads count=0 kept=09282026/11/12 00:43:08 WARN Force mode enabled - objects will be deleted immediately without grace period9292026/11/12 00:43:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=09302026/11/12 00:43:08 INFO Vacuumed table table=pending_closures9312026/11/12 00:43:08 INFO Vacuumed table table=commit_receipts9322026/11/12 00:43:08 INFO Vacuumed table table=pending_objects9332026/11/12 00:43:08 INFO Vacuumed table table=multipart_uploads9342026/11/12 00:43:08 INFO Vacuumed table table=closures9352026/11/12 00:43:08 INFO Vacuumed table table=objects936--- PASS: TestGCMetrics (0.76s)937=== CONT TestClientIntegration9382026/11/12 00:43:08 INFO Received push request method=POST path=/api/pushes9392026/11/12 00:43:09 INFO lead: acquired remote=192.0.2.1:1234 generation=19402026/11/12 00:43:09 INFO lead: released remote=192.0.2.1:1234 generation=19412026/11/12 00:43:09 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)942--- PASS: TestLeadIncumbentAnnouncesAtOnce (0.78s)943=== CONT TestClientErrorHandling944=== RUN TestClientErrorHandling/InvalidStorePath945=== PAUSE TestClientErrorHandling/InvalidStorePath9462026/11/12 00:43:09 INFO Uploading rycw9z7aryirx99s5yjz94fm3zk5bnzp-file1.txt (160B)947=== RUN TestClientErrorHandling/InvalidAuthToken948=== PAUSE TestClientErrorHandling/InvalidAuthToken949=== RUN TestClientErrorHandling/ServerNotAvailable950=== PAUSE TestClientErrorHandling/ServerNotAvailable951=== CONT TestClientCADerivations9522026/11/12 00:43:09 WARN Failed to register uploaded object key=rycw9z7aryirx99s5yjz94fm3zk5bnzp.ls error="server returned 404: 404 page not found\n"9532026/11/12 00:43:09 WARN Failed to register uploaded object key=nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst error="server returned 404: 404 page not found\n"9542026/11/12 00:43:09 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign9552026/11/12 00:43:09 INFO Signed narinfos id=1 count=19562026/11/12 00:43:09 INFO Uploading 1 narinfos9572026/11/12 00:43:09 INFO Received complete push request method=POST path=/api/pushes/1/complete9582026/11/12 00:43:09 WARN Failed to register uploaded object key=rycw9z7aryirx99s5yjz94fm3zk5bnzp.narinfo error="server returned 404: 404 page not found\n"9592026/11/12 00:43:09 INFO Received uploads request method=POST path=/api/pending_closures9602026/11/12 00:43:09 INFO Completed push id=1 objects=3 duration=19.274309ms9612026/11/12 00:43:09 INFO Upload complete. (90ms)962=== NAME TestNARDeduplicationMetadataUploadBug963 metadata_upload_test.go:54: Retrieved narinfo from S3:964 StorePath: /build/TestNARDeduplicationMetadataUploadBug2491940307/001/store/rycw9z7aryirx99s5yjz94fm3zk5bnzp-file1.txt965 URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst966 Compression: zstd967 NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf968 NarSize: 160969 References: 970 CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf971 metadata_upload_test.go:55: Retrieved .ls file from S3 (compressed size: 77 bytes)972 metadata_upload_test.go:55: Decompressed .ls content (64 bytes):973 {"version":1,"root":{"type":"regular","size":44,"narOffset":96}}9742026/11/12 00:43:09 INFO Received uploads request method=POST path=/api/pending_closures9752026/11/12 00:43:09 INFO Aborted multipart uploads count=0 kept=0976 metadata_upload_test.go:64: Second store path (same content): /build/TestNARDeduplicationMetadataUploadBug2491940307/001/store/ccggxnspcbqdg5g4xgd5sf586qv6v3pi-file2.txt9772026/11/12 00:43:09 WARN Force mode enabled - objects will be deleted immediately without grace period9782026/11/12 00:43:09 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=09792026/11/12 00:43:09 INFO Vacuumed table table=pending_closures9802026/11/12 00:43:09 INFO Vacuumed table table=commit_receipts9812026/11/12 00:43:09 INFO Vacuumed table table=pending_objects9822026/11/12 00:43:09 INFO Vacuumed table table=multipart_uploads9832026/11/12 00:43:09 INFO Vacuumed table table=closures9842026/11/12 00:43:09 INFO Vacuumed table table=objects9852026/11/12 00:43:09 INFO Aborted multipart uploads count=0 kept=09862026/11/12 00:43:09 WARN Force mode enabled - objects will be deleted immediately without grace period9872026/11/12 00:43:09 INFO lead: acquired remote=192.0.2.1:1234 generation=19882026/11/12 00:43:09 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=09892026/11/12 00:43:09 INFO Vacuumed table table=pending_closures9902026/11/12 00:43:09 INFO Vacuumed table table=commit_receipts9912026/11/12 00:43:09 INFO Vacuumed table table=pending_objects9922026/11/12 00:43:09 INFO Vacuumed table table=multipart_uploads9932026/11/12 00:43:09 INFO Vacuumed table table=closures9942026/11/12 00:43:09 INFO Vacuumed table table=objects995--- PASS: TestSweepSkipsRowsLockedByPush (0.91s)996=== CONT TestCacheStatsHandler9972026/11/12 00:43:09 ERROR failed to check GC advisory lock error="query pg_locks: failed to connect to `user=nixbld database=niks3`: /nonexistent/.s.PGSQL.5432 (/nonexistent): dial error: dial unix /nonexistent/.s.PGSQL.5432: connect: no such file or directory"998--- PASS: TestGCAdvisoryLockBlocksConcurrentRun (0.93s)999=== CONT TestCacheConfigHandler1000=== RUN TestCacheConfigHandler/full_config,_no_issuer1001=== PAUSE TestCacheConfigHandler/full_config,_no_issuer1002=== RUN TestCacheConfigHandler/no_cache_url_configured1003=== PAUSE TestCacheConfigHandler/no_cache_url_configured1004=== RUN TestCacheConfigHandler/no_signing_keys1005=== PAUSE TestCacheConfigHandler/no_signing_keys1006=== RUN TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1007=== PAUSE TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1008=== CONT TestService_ReadScope_PublicByDefault10092026/11/12 00:43:09 INFO Received push request method=POST path=/api/pushes10102026/11/12 00:43:09 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)10112026/11/12 00:43:09 INFO Received sign narinfos request method=POST path=/api/pushes/2/sign10122026/11/12 00:43:09 WARN Failed to register uploaded object key=ccggxnspcbqdg5g4xgd5sf586qv6v3pi.ls error="server returned 404: 404 page not found\n"10132026/11/12 00:43:09 INFO Signed narinfos id=2 count=110142026/11/12 00:43:09 INFO Uploading 1 narinfos10152026/11/12 00:43:09 INFO Aborted multipart uploads count=0 kept=010162026/11/12 00:43:09 INFO lead: acquired remote=192.0.2.1:1234 generation=110172026/11/12 00:43:09 WARN Force mode enabled - objects will be deleted immediately without grace period10182026/11/12 00:43:09 INFO Received complete push request method=POST path=/api/pushes/2/complete10192026/11/12 00:43:09 WARN Failed to register uploaded object key=ccggxnspcbqdg5g4xgd5sf586qv6v3pi.narinfo error="server returned 404: 404 page not found\n"10202026/11/12 00:43:09 INFO Completed push id=2 objects=3 duration=3.408189ms10212026/11/12 00:43:09 INFO Upload complete. (62ms)1022=== NAME TestNARDeduplicationMetadataUploadBug1023 metadata_upload_test.go:76: Retrieved narinfo from S3:1024 StorePath: /build/TestNARDeduplicationMetadataUploadBug2491940307/001/store/ccggxnspcbqdg5g4xgd5sf586qv6v3pi-file2.txt1025 URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst1026 Compression: zstd1027 NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1028 NarSize: 1601029 References: 1030 CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf10312026/11/12 00:43:09 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."10322026/11/12 00:43:09 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."1033 metadata_upload_test.go:77: Retrieved .ls file from S3 (compressed size: 62 bytes)1034 metadata_upload_test.go:77: Decompressed .ls content (49 bytes):1035 {"version":1,"root":{"type":"regular","size":44}}1036--- PASS: TestFailedS3DeleteKeepsTombstone (0.97s)1037=== CONT TestService_RequireScope_OIDC1038--- PASS: TestNARDeduplicationMetadataUploadBug (0.98s)1039=== CONT TestService_AuthMiddleware_OIDC10402026/11/12 00:43:09 INFO Received uploads request method=POST path=/api/pending_closures10412026/11/12 00:43:09 INFO Aborted multipart uploads count=0 kept=010422026/11/12 00:43:09 INFO lead: acquired remote=192.0.2.1:1234 generation=110432026/11/12 00:43:09 WARN Force mode enabled - objects will be deleted immediately without grace period10442026/11/12 00:43:09 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=010452026/11/12 00:43:09 INFO Vacuumed table table=pending_closures10462026/11/12 00:43:09 INFO Vacuumed table table=commit_receipts10472026/11/12 00:43:09 INFO Vacuumed table table=pending_objects10482026/11/12 00:43:09 INFO Vacuumed table table=multipart_uploads10492026/11/12 00:43:09 INFO Vacuumed table table=closures10502026/11/12 00:43:09 INFO Vacuumed table table=objects1051--- PASS: TestGCSweepSkipsPendingObjects (1.03s)1052=== CONT TestService_ReadAuthMiddleware10532026/11/12 00:43:09 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:39785/oidc1054--- PASS: TestGetExistingObjectsLocksRows (1.04s)1055=== CONT TestService_AuthMiddleware_MTLSBoundSubjects10562026/11/12 00:43:09 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:37323/oidc10572026/11/12 00:43:09 INFO lead: acquired remote=192.0.2.1:1234 generation=110582026/11/12 00:43:09 INFO Received uploads request method=POST path=/api/pending_closures10592026/11/12 00:43:09 INFO Aborted multipart uploads count=0 kept=010602026/11/12 00:43:09 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=010612026/11/12 00:43:09 INFO Vacuumed table table=pending_closures10622026/11/12 00:43:09 INFO Vacuumed table table=commit_receipts10632026/11/12 00:43:09 INFO Vacuumed table table=pending_objects10642026/11/12 00:43:09 INFO Vacuumed table table=multipart_uploads10652026/11/12 00:43:09 INFO Vacuumed table table=closures10662026/11/12 00:43:09 INFO Vacuumed table table=objects10672026/11/12 00:43:09 INFO Aborted multipart uploads count=0 kept=010682026/11/12 00:43:09 WARN Force mode enabled - objects will be deleted immediately without grace period10692026/11/12 00:43:09 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=010702026/11/12 00:43:09 INFO Vacuumed table table=pending_closures10712026/11/12 00:43:09 INFO Vacuumed table table=commit_receipts10722026/11/12 00:43:09 INFO Vacuumed table table=pending_objects10732026/11/12 00:43:09 INFO Vacuumed table table=multipart_uploads10742026/11/12 00:43:09 INFO Vacuumed table table=closures10752026/11/12 00:43:09 INFO Vacuumed table table=objects1076--- PASS: TestPushDedupSurvivesConcurrentGC (1.24s)1077=== CONT TestService_AuthMiddleware_MTLSProxyHeader1078--- PASS: TestGCBugBareHashReferences (1.31s)1079=== CONT TestService_AuthMiddleware1080--- PASS: TestPushWaitsForSweepHoldingRow (1.31s)1081=== CONT TestReadProxyRangeRequest1082=== NAME TestPinProtectsFromGC1083 client_integration_test.go:791: Pinned store path: /build/TestPinProtectsFromGC2388730259/001/store/r007nwc94di5hzmvcc3cfgg80d3yb5zk-pinned-file.txt1084 client_integration_test.go:792: Unpinned store path: /build/TestPinProtectsFromGC2388730259/001/store/9kifi76w2q5bl3d8q9i50jlzgjdh7nhl-unpinned-file.txt1085=== NAME TestClientMultipleUploads1086 client_integration_test.go:406: Created store path 0: /build/TestClientMultipleUploads3949002173/001/store/fvabcdgj6qy08rkkarndckwi7v7m7m14-test-file-0.txt1087--- PASS: TestConcurrentCommitsSharingObjectsDoNotDeadlock (0.90s)1088=== CONT TestCreatePendingClosure_SmallNARUsesSimplePUT1089=== NAME TestClientWithDependencies1090 client_integration_test.go:661: Built derivation: /build/TestClientWithDependencies2051169984/001/store/plnpaz1f24663dd3b11ql92j5ksf8a96-test-script10912026/11/12 00:43:09 INFO Received uploads request method=POST path=/api/pending_closures10922026/11/12 00:43:09 INFO Received uploads request method=POST path=/api/pending_closures10932026/11/12 00:43:09 INFO Received push request method=POST path=/api/pushes10942026/11/12 00:43:09 INFO Uploading 2 paths to 127.0.0.1 (1 already cached)10952026/11/12 00:43:09 INFO Uploading 2c26cvcrhazvv2g4qinfprr639lrn7xx-shared-dep (136B)10962026/11/12 00:43:09 INFO Uploading p9n1n425rzings6skg7xdij1hi2j314q-b (216B)1097=== NAME TestClientMultipleUploads1098 client_integration_test.go:406: Created store path 1: /build/TestClientMultipleUploads3949002173/001/store/i8lw4nsiprjliv7cq982k0j494agp4f8-test-file-1.txt10992026/11/12 00:43:09 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)11002026/11/12 00:43:09 INFO Uploading r007nwc94di5hzmvcc3cfgg80d3yb5zk-pinned-file.txt (128B)11012026/11/12 00:43:09 WARN Failed to register uploaded object key=k1c42x2zbwgsppyrvjvljq9fzzd5xcd3.ls error="server returned 404: 404 page not found\n"11022026/11/12 00:43:09 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"11032026/11/12 00:43:09 WARN Failed to register uploaded object key=2c26cvcrhazvv2g4qinfprr639lrn7xx.ls error="server returned 404: 404 page not found\n"11042026/11/12 00:43:09 WARN Failed to register uploaded object key=p9n1n425rzings6skg7xdij1hi2j314q.ls error="server returned 404: 404 page not found\n"11052026/11/12 00:43:09 WARN Failed to register uploaded object key=r007nwc94di5hzmvcc3cfgg80d3yb5zk.ls error="server returned 404: 404 page not found\n"11062026/11/12 00:43:09 WARN Failed to register uploaded object key=nar/00m7n3hrf3d38mn1pkh9sbfyv2hffw2jq4lx3rri0wj3fj6dyl0h.nar.zst error="server returned 404: 404 page not found\n"11072026/11/12 00:43:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign11082026/11/12 00:43:09 INFO Signed narinfos id=1 count=211092026/11/12 00:43:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign11102026/11/12 00:43:09 INFO Signed narinfos id=2 count=211112026/11/12 00:43:09 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign11122026/11/12 00:43:09 INFO Uploading 4 narinfos11132026/11/12 00:43:09 WARN Failed to register uploaded object key=nar/0qmhsg42pqsz7x6hh02j8gznnac3yi3qpjhrqbwvz88i194jbb5s.nar.zst error="server returned 404: 404 page not found\n"11142026/11/12 00:43:09 INFO Signed narinfos id=1 count=11115=== NAME TestClientIntegration1116 client_integration_test.go:292: Created store path: /build/TestClientIntegration687895362/002/store/j553qrz1k2h1d2z6jw96dkj8wa13dd92-test-file.txt11172026/11/12 00:43:09 INFO Uploading 1 narinfos11182026/11/12 00:43:09 WARN Failed to register uploaded object key=2c26cvcrhazvv2g4qinfprr639lrn7xx.narinfo error="server returned 404: 404 page not found\n"11192026/11/12 00:43:09 INFO Received complete push request method=POST path=/api/pushes/1/complete11202026/11/12 00:43:09 WARN Failed to register uploaded object key=r007nwc94di5hzmvcc3cfgg80d3yb5zk.narinfo error="server returned 404: 404 page not found\n"11212026/11/12 00:43:09 WARN Failed to register uploaded object key=k1c42x2zbwgsppyrvjvljq9fzzd5xcd3.narinfo error="server returned 404: 404 page not found\n"11222026/11/12 00:43:09 WARN Failed to register uploaded object key=p9n1n425rzings6skg7xdij1hi2j314q.narinfo error="server returned 404: 404 page not found\n"1123--- PASS: TestCacheStatsHandler (0.52s)1124=== CONT TestCompleteMultipartUnregistered11252026/11/12 00:43:09 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete11262026/11/12 00:43:09 WARN Failed to register uploaded object key=2c26cvcrhazvv2g4qinfprr639lrn7xx.narinfo error="server returned 404: 404 page not found\n"1127=== NAME TestClientWithDependencies11282026/11/12 00:43:09 INFO Completed push id=1 objects=3 duration=5.761429ms1129 client_integration_test.go:663: Found 1 dependencies (including self)11302026/11/12 00:43:09 INFO Upload complete. (66ms)11312026/11/12 00:43:09 INFO Received push request method=POST path=/api/pushes11322026/11/12 00:43:09 INFO Received push request method=POST path=/api/pushes11332026/11/12 00:43:09 INFO Completed upload id=2 objects=6 duration=9.8985ms11342026/11/12 00:43:09 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete11352026/11/12 00:43:09 INFO Uploading 2 paths to 127.0.0.1 (1 already cached)11362026/11/12 00:43:09 INFO Uploading nrb489ykmzxm7azvi0cpd00w40g6sqg8-shared-dep (136B)11372026/11/12 00:43:09 INFO Uploading sl6vp6h5baf05qgdnns2ixbw6s38si58-b (216B)11382026/11/12 00:43:09 INFO Completed upload id=1 objects=6 duration=2.36829ms11392026/11/12 00:43:09 INFO Upload complete. (85ms)11402026/11/12 00:43:09 INFO Uploading 2 paths to 127.0.0.1 (0 already cached)1141=== NAME TestClientMultipleUploads11422026/11/12 00:43:09 INFO Uploading m1abjscg92rg72wgz30c78sy4kzn7pax-shared-dep (136B)1143 client_integration_test.go:406: Created store path 2: /build/TestClientMultipleUploads3949002173/001/store/kf2a92lv73fbxmg5vc52driqp3jnx236-test-file-2.txt11442026/11/12 00:43:09 INFO Uploading rvk943nqh9dpsis85i6bsznqcg22sxq2-top (224B)1145--- PASS: TestService_ReadScope_PublicByDefault (0.52s)1146=== CONT TestService_verifyS3ReuploadsStaleNarinfo1147=== NAME TestClientFallsBackToClosures1148 client_pushes_test.go:112: Retrieved narinfo from S3:1149 StorePath: /build/TestClientFallsBackToClosures1989831607/001/store/2c26cvcrhazvv2g4qinfprr639lrn7xx-shared-dep11502026/11/12 00:43:09 WARN Failed to register uploaded object key=7lihwcjqwkv6kdmvmprm85lsjm1g6zry.ls error="server returned 404: 404 page not found\n"1151 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1152 Compression: zstd1153 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821154 NarSize: 1361155 References: 1156 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n1157=== RUN TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed1158=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed1159=== RUN TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed1160=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed1161=== RUN TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost1162=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost1163=== RUN TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed1164=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed1165=== RUN TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed1166=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed11672026/11/12 00:43:09 WARN Failed to register uploaded object key=nrb489ykmzxm7azvi0cpd00w40g6sqg8.ls error="server returned 404: 404 page not found\n"1168=== CONT TestService_verifyS3Integrity11692026/11/12 00:43:09 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"1170=== NAME TestClientFallsBackToClosures1171 client_pushes_test.go:112: Retrieved narinfo from S3:1172 StorePath: /build/TestClientFallsBackToClosures1989831607/001/store/k1c42x2zbwgsppyrvjvljq9fzzd5xcd3-a1173 URL: nar/00m7n3hrf3d38mn1pkh9sbfyv2hffw2jq4lx3rri0wj3fj6dyl0h.nar.zst1174 Compression: zstd11752026/11/12 00:43:09 WARN Failed to register uploaded object key=nar/1n37r5m7wvjlyzgqbcmqh7rzhxwhnzpx9kqfkzd4ax1nm0610n3z.nar.zst error="server returned 404: 404 page not found\n"1176 NarHash: sha256:00m7n3hrf3d38mn1pkh9sbfyv2hffw2jq4lx3rri0wj3fj6dyl0h1177 NarSize: 2161178 References: /build/TestClientFallsBackToClosures1989831607/001/store/2c26cvcrhazvv2g4qinfprr639lrn7xx-shared-dep1179 CA: text:sha256:03ak539b9l6znflm4lrkk7vzrfw01ybhw5kj1dzl2rw15w9by4l911802026/11/12 00:43:09 WARN Failed to register uploaded object key=m1abjscg92rg72wgz30c78sy4kzn7pax.ls error="server returned 404: 404 page not found\n"11812026/11/12 00:43:09 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"11822026/11/12 00:43:09 WARN Failed to register uploaded object key=nar/0a44nz5dq8qwghb792knl7i90fw28ln95dv3kifizxygxmcnfgkp.nar.zst error="server returned 404: 404 page not found\n"11832026/11/12 00:43:09 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign11842026/11/12 00:43:09 WARN Failed to register uploaded object key=rvk943nqh9dpsis85i6bsznqcg22sxq2.ls error="server returned 404: 404 page not found\n"11852026/11/12 00:43:09 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign11862026/11/12 00:43:09 INFO Signed narinfos id=1 count=21187 client_pushes_test.go:112: Retrieved narinfo from S3:1188 StorePath: /build/TestClientFallsBackToClosures1989831607/001/store/p9n1n425rzings6skg7xdij1hi2j314q-b11892026/11/12 00:43:09 WARN Failed to register uploaded object key=sl6vp6h5baf05qgdnns2ixbw6s38si58.ls error="server returned 404: 404 page not found\n"1190 URL: nar/00m7n3hrf3d38mn1pkh9sbfyv2hffw2jq4lx3rri0wj3fj6dyl0h.nar.zst11912026/11/12 00:43:09 INFO Uploading 2 narinfos1192 Compression: zstd11932026/11/12 00:43:09 INFO Signed narinfos id=1 count=31194 NarHash: sha256:00m7n3hrf3d38mn1pkh9sbfyv2hffw2jq4lx3rri0wj3fj6dyl0h1195 NarSize: 2161196 References: /build/TestClientFallsBackToClosures1989831607/001/store/2c26cvcrhazvv2g4qinfprr639lrn7xx-shared-dep11972026/11/12 00:43:09 INFO Uploading 3 narinfos1198 CA: text:sha256:03ak539b9l6znflm4lrkk7vzrfw01ybhw5kj1dzl2rw15w9by4l911992026/11/12 00:43:09 WARN Failed to register uploaded object key=7lihwcjqwkv6kdmvmprm85lsjm1g6zry.narinfo error="server returned 404: 404 page not found\n"12002026/11/12 00:43:09 WARN Failed to register uploaded object key=m1abjscg92rg72wgz30c78sy4kzn7pax.narinfo error="server returned 404: 404 page not found\n"12012026/11/12 00:43:09 INFO Received complete push request method=POST path=/api/pushes/1/complete12022026/11/12 00:43:09 WARN Failed to register uploaded object key=rvk943nqh9dpsis85i6bsznqcg22sxq2.narinfo error="server returned 404: 404 page not found\n"12032026/11/12 00:43:09 WARN Failed to register uploaded object key=sl6vp6h5baf05qgdnns2ixbw6s38si58.narinfo error="server returned 404: 404 page not found\n"12042026/11/12 00:43:09 INFO Received complete push request method=POST path=/api/pushes/1/complete12052026/11/12 00:43:09 WARN Failed to register uploaded object key=nrb489ykmzxm7azvi0cpd00w40g6sqg8.narinfo error="server returned 404: 404 page not found\n"1206--- PASS: TestClientFallsBackToClosures (0.84s)1207=== CONT TestService_createPendingClosureHandler12082026/11/12 00:43:09 INFO Completed push id=1 objects=8 duration=6.206429ms12092026/11/12 00:43:09 INFO Upload complete. (70ms)1210=== NAME TestClientPushesUseOnePush1211 client_pushes_test.go:97: Retrieved narinfo from S3:1212 StorePath: /build/TestClientPushesUseOnePush2129342650/001/store/nrb489ykmzxm7azvi0cpd00w40g6sqg8-shared-dep1213 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1214 Compression: zstd1215 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821216 NarSize: 1361217 References: 1218 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n1219 client_pushes_test.go:97: Retrieved narinfo from S3:1220 StorePath: /build/TestClientPushesUseOnePush2129342650/001/store/7lihwcjqwkv6kdmvmprm85lsjm1g6zry-a12212026/11/12 00:43:09 INFO Completed push id=1 objects=6 duration=10.232709ms1222 URL: nar/1n37r5m7wvjlyzgqbcmqh7rzhxwhnzpx9kqfkzd4ax1nm0610n3z.nar.zst1223 Compression: zstd1224 NarHash: sha256:1n37r5m7wvjlyzgqbcmqh7rzhxwhnzpx9kqfkzd4ax1nm0610n3z12252026/11/12 00:43:09 INFO Upload complete. (72ms)1226 NarSize: 2161227 References: /build/TestClientPushesUseOnePush2129342650/001/store/nrb489ykmzxm7azvi0cpd00w40g6sqg8-shared-dep1228 CA: text:sha256:0pj3sra3q68lh2q8r7s7yxx91rs1axmxkk4kg8hxiishgr1319491229=== NAME TestClientSharedPathCommittedMidPush1230 client_integration_test.go:740: Retrieved narinfo from S3:1231 StorePath: /build/TestClientSharedPathCommittedMidPush2417648185/001/store/m1abjscg92rg72wgz30c78sy4kzn7pax-shared-dep1232 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1233 Compression: zstd1234 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821235 NarSize: 1361236 References: 1237 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n1238=== NAME TestClientCADerivations1239 client_ca_test.go:136: Built CA derivation: /build/TestClientCADerivations1481204780/001/store/gs3vaxrbqbfsd6wkpnskqkcxvanv778f-ca-test1240=== NAME TestClientPushesUseOnePush1241 client_pushes_test.go:97: Retrieved narinfo from S3:1242 StorePath: /build/TestClientPushesUseOnePush2129342650/001/store/sl6vp6h5baf05qgdnns2ixbw6s38si58-b1243 URL: nar/1n37r5m7wvjlyzgqbcmqh7rzhxwhnzpx9kqfkzd4ax1nm0610n3z.nar.zst1244 Compression: zstd1245 NarHash: sha256:1n37r5m7wvjlyzgqbcmqh7rzhxwhnzpx9kqfkzd4ax1nm0610n3z1246 NarSize: 2161247 References: /build/TestClientPushesUseOnePush2129342650/001/store/nrb489ykmzxm7azvi0cpd00w40g6sqg8-shared-dep1248 CA: text:sha256:0pj3sra3q68lh2q8r7s7yxx91rs1axmxkk4kg8hxiishgr1319491249--- PASS: TestService_ReadAuthMiddleware (0.45s)1250=== CONT TestService_cleanupPendingClosuresHandler1251=== NAME TestClientSharedPathCommittedMidPush1252 client_integration_test.go:740: Retrieved narinfo from S3:1253 StorePath: /build/TestClientSharedPathCommittedMidPush2417648185/001/store/rvk943nqh9dpsis85i6bsznqcg22sxq2-top1254 URL: nar/0a44nz5dq8qwghb792knl7i90fw28ln95dv3kifizxygxmcnfgkp.nar.zst1255 Compression: zstd1256 NarHash: sha256:0a44nz5dq8qwghb792knl7i90fw28ln95dv3kifizxygxmcnfgkp1257 NarSize: 2241258 References: /build/TestClientSharedPathCommittedMidPush2417648185/001/store/m1abjscg92rg72wgz30c78sy4kzn7pax-shared-dep1259 CA: text:sha256:1isl3px0zjd6sdxfd8vbza4s3ra8a9fd8y1fzgpvm0zpvzr9mnay1260--- PASS: TestClientPushesUseOnePush (0.85s)1261=== CONT TestUploadHandlersRejectOversizedBody1262--- PASS: TestClientSharedPathCommittedMidPush (0.80s)1263=== CONT TestUploadHandlersRejectInvalidKeys1264=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info1265=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info1266=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal1267=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal1268=== RUN TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key1269=== PAUSE TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key1270=== RUN TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key1271=== PAUSE TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key1272=== CONT TestIsValidUploadKey1273=== RUN TestIsValidUploadKey/narinfo1274=== NAME TestClientCADerivations12752026/11/12 00:43:09 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted,O=Other"1276 client_ca_test.go:139: Found 1 dependencies (including self)12772026/11/12 00:43:09 WARN mTLS auth: bound subjects configured but subject DN unavailable1278=== RUN TestUploadHandlersRejectOversizedBody/create_pending_closure12792026/11/12 00:43:09 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted"1280=== PAUSE TestIsValidUploadKey/narinfo12812026/11/12 00:43:09 INFO Received push request method=POST path=/api/pushes1282=== PAUSE TestUploadHandlersRejectOversizedBody/create_pending_closure12832026/11/12 00:43:09 INFO Received push request method=POST path=/api/pushes1284--- PASS: TestService_AuthMiddleware_MTLSBoundSubjects (0.63s)1285=== CONT TestPendingClosureWriteTimeout12862026/11/12 00:43:09 INFO Received push request method=POST path=/api/pushes1287=== RUN TestPendingClosureWriteTimeout/empty1288=== RUN TestIsValidUploadKey/nar_zst12892026/11/12 00:43:09 INFO Received push request method=POST path=/api/pushes1290=== RUN TestUploadHandlersRejectOversizedBody/complete_multipart1291=== PAUSE TestUploadHandlersRejectOversizedBody/complete_multipart1292=== RUN TestUploadHandlersRejectOversizedBody/request_more_parts1293=== PAUSE TestPendingClosureWriteTimeout/empty1294=== PAUSE TestIsValidUploadKey/nar_zst1295=== RUN TestIsValidUploadKey/nar_xz1296=== PAUSE TestIsValidUploadKey/nar_xz1297=== RUN TestIsValidUploadKey/nar_plain1298=== PAUSE TestIsValidUploadKey/nar_plain1299=== PAUSE TestUploadHandlersRejectOversizedBody/request_more_parts1300=== RUN TestPendingClosureWriteTimeout/400_objects1301=== PAUSE TestPendingClosureWriteTimeout/400_objects1302=== RUN TestIsValidUploadKey/listing1303=== CONT TestProxyWriteTimeout1304=== RUN TestProxyWriteTimeout/narinfo1305=== PAUSE TestProxyWriteTimeout/narinfo1306=== RUN TestProxyWriteTimeout/1_GiB_nar1307=== PAUSE TestProxyWriteTimeout/1_GiB_nar13082026/11/12 00:43:09 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)1309=== RUN TestProxyWriteTimeout/10_GiB_nar1310=== PAUSE TestProxyWriteTimeout/10_GiB_nar13112026/11/12 00:43:09 INFO Uploading 9kifi76w2q5bl3d8q9i50jlzgjdh7nhl-unpinned-file.txt (128B)1312=== RUN TestPendingClosureWriteTimeout/670k_objects1313=== PAUSE TestIsValidUploadKey/listing1314=== RUN TestIsValidUploadKey/build_log13152026/11/12 00:43:09 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)1316=== PAUSE TestIsValidUploadKey/build_log13172026/11/12 00:43:09 INFO Uploading j553qrz1k2h1d2z6jw96dkj8wa13dd92-test-file.txt (152B)1318=== RUN TestIsValidUploadKey/build_log_home-manager_file1319=== PAUSE TestIsValidUploadKey/build_log_home-manager_file1320=== RUN TestProxyWriteTimeout/unknown_size1321=== PAUSE TestPendingClosureWriteTimeout/670k_objects1322=== RUN TestIsValidUploadKey/build_log_plus_in_name1323=== PAUSE TestIsValidUploadKey/build_log_plus_in_name13242026/11/12 00:43:09 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)1325=== RUN TestIsValidUploadKey/build_log_question_mark13262026/11/12 00:43:09 INFO Uploading plnpaz1f24663dd3b11ql92j5ksf8a96-test-script (136B)1327=== PAUSE TestIsValidUploadKey/build_log_question_mark1328=== PAUSE TestProxyWriteTimeout/unknown_size1329=== RUN TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline1330=== PAUSE TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline1331=== RUN TestIsValidUploadKey/build_log_equals1332=== CONT TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle1333=== CONT TestSkippedUploadsHandler1334=== PAUSE TestIsValidUploadKey/build_log_equals1335=== RUN TestIsValidUploadKey/realisation1336=== PAUSE TestIsValidUploadKey/realisation1337=== RUN TestIsValidUploadKey/realisation_plus_in_output1338=== PAUSE TestIsValidUploadKey/realisation_plus_in_output13392026/11/12 00:43:09 INFO Client skipped oversized paths paths=3 nar_bytes=50000000001340=== RUN TestIsValidUploadKey/nix-cache-info13412026/11/12 00:43:09 INFO Uploading 3 paths to 127.0.0.1 (0 already cached)1342=== PAUSE TestIsValidUploadKey/nix-cache-info13432026/11/12 00:43:09 INFO Uploading fvabcdgj6qy08rkkarndckwi7v7m7m14-test-file-0.txt (160B)1344=== RUN TestIsValidUploadKey/index.html13452026/11/12 00:43:09 INFO Uploading i8lw4nsiprjliv7cq982k0j494agp4f8-test-file-1.txt (160B)1346=== PAUSE TestIsValidUploadKey/index.html13472026/11/12 00:43:09 INFO Uploading kf2a92lv73fbxmg5vc52driqp3jnx236-test-file-2.txt (160B)1348=== RUN TestIsValidUploadKey/narinfo_key,_nar_type1349=== PAUSE TestIsValidUploadKey/narinfo_key,_nar_type1350=== RUN TestIsValidUploadKey/nar_key,_narinfo_type1351=== PAUSE TestIsValidUploadKey/nar_key,_narinfo_type1352=== RUN TestIsValidUploadKey/listing_key,_narinfo_type1353=== PAUSE TestIsValidUploadKey/listing_key,_narinfo_type1354=== RUN TestIsValidUploadKey/traversal1355=== PAUSE TestIsValidUploadKey/traversal1356=== RUN TestIsValidUploadKey/traversal_nar1357=== PAUSE TestIsValidUploadKey/traversal_nar1358=== RUN TestIsValidUploadKey/absolute1359=== PAUSE TestIsValidUploadKey/absolute1360=== RUN TestIsValidUploadKey/empty_key1361=== PAUSE TestIsValidUploadKey/empty_key1362=== RUN TestIsValidUploadKey/unknown_type1363=== PAUSE TestIsValidUploadKey/unknown_type1364=== CONT TestParseSize1365--- PASS: TestParseSize (0.00s)1366=== CONT TestService_Rustfstest1367--- PASS: TestSkippedUploadsHandler (0.00s)1368=== CONT TestPresignedUploadRegisteredBeforeCommit13692026/11/12 00:43:09 WARN Failed to register uploaded object key=j553qrz1k2h1d2z6jw96dkj8wa13dd92.ls error="server returned 404: 404 page not found\n"13702026/11/12 00:43:09 WARN Failed to register uploaded object key=nar/0gh0wdnr6gmk4r5sfzzh12isnl14rgmkrwy1ws41hpsrlndclg5h.nar.zst error="server returned 404: 404 page not found\n"13712026/11/12 00:43:09 INFO Received sign narinfos request method=POST path=/api/pushes/2/sign13722026/11/12 00:43:09 WARN Failed to register uploaded object key=9kifi76w2q5bl3d8q9i50jlzgjdh7nhl.ls error="server returned 404: 404 page not found\n"13732026/11/12 00:43:09 INFO Signed narinfos id=2 count=113742026/11/12 00:43:09 INFO Uploading 1 narinfos13752026/11/12 00:43:09 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign13762026/11/12 00:43:09 WARN Failed to register uploaded object key=plnpaz1f24663dd3b11ql92j5ksf8a96.ls error="server returned 404: 404 page not found\n"13772026/11/12 00:43:09 WARN Failed to register uploaded object key=nar/0ahq3izw3h0b0c08p3xdfclh758fwxnp85ccvflasb4vnfsv50ri.nar.zst error="server returned 404: 404 page not found\n"13782026/11/12 00:43:09 WARN Failed to register uploaded object key=nar/1s7yghixzpinyqm7g59kyn3sav81pmnsdi5v63m6x4r2vacy907j.nar.zst error="server returned 404: 404 page not found\n"13792026/11/12 00:43:09 INFO Signed narinfos id=1 count=113802026/11/12 00:43:09 WARN Failed to register uploaded object key=nar/11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1.nar.zst error="server returned 404: 404 page not found\n"13812026/11/12 00:43:09 INFO Uploading 1 narinfos13822026/11/12 00:43:09 WARN Failed to register uploaded object key=i8lw4nsiprjliv7cq982k0j494agp4f8.ls error="server returned 404: 404 page not found\n"13832026/11/12 00:43:09 WARN Failed to register uploaded object key=nar/13p9pqi0ga6fx2apcwls9hjnv7nmf740ggpn5zvfzdsp5vz2bbg4.nar.zst error="server returned 404: 404 page not found\n"13842026/11/12 00:43:09 WARN Failed to register uploaded object key=log/jj19rb4x4jds6yl25izy725c4gwxr2hb-test-script.drv error="server returned 404: 404 page not found\n"13852026/11/12 00:43:09 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign13862026/11/12 00:43:09 WARN Failed to register uploaded object key=fvabcdgj6qy08rkkarndckwi7v7m7m14.ls error="server returned 404: 404 page not found\n"13872026/11/12 00:43:09 WARN Failed to register uploaded object key=nar/1823g6vx4q5zqvr73m7lkq1wpk40fin7bsc792di430yk7sg4lka.nar.zst error="server returned 404: 404 page not found\n"13882026/11/12 00:43:09 INFO Signed narinfos id=1 count=113892026/11/12 00:43:09 INFO Uploading 1 narinfos13902026/11/12 00:43:09 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign13912026/11/12 00:43:09 WARN Failed to register uploaded object key=kf2a92lv73fbxmg5vc52driqp3jnx236.ls error="server returned 404: 404 page not found\n"13922026/11/12 00:43:09 INFO Signed narinfos id=1 count=313932026/11/12 00:43:09 INFO Uploading 3 narinfos13942026/11/12 00:43:09 INFO Received complete push request method=POST path=/api/pushes/2/complete13952026/11/12 00:43:09 WARN Failed to register uploaded object key=9kifi76w2q5bl3d8q9i50jlzgjdh7nhl.narinfo error="server returned 404: 404 page not found\n"13962026/11/12 00:43:09 INFO Received complete push request method=POST path=/api/pushes/1/complete13972026/11/12 00:43:09 WARN Failed to register uploaded object key=j553qrz1k2h1d2z6jw96dkj8wa13dd92.narinfo error="server returned 404: 404 page not found\n"13982026/11/12 00:43:09 INFO Received complete push request method=POST path=/api/pushes/1/complete13992026/11/12 00:43:09 WARN Failed to register uploaded object key=plnpaz1f24663dd3b11ql92j5ksf8a96.narinfo error="server returned 404: 404 page not found\n"14002026/11/12 00:43:09 INFO Completed push id=2 objects=3 duration=3.89604ms14012026/11/12 00:43:09 INFO Upload complete. (225ms)14022026/11/12 00:43:09 WARN Failed to register uploaded object key=kf2a92lv73fbxmg5vc52driqp3jnx236.narinfo error="server returned 404: 404 page not found\n"14032026/11/12 00:43:09 WARN Failed to register uploaded object key=fvabcdgj6qy08rkkarndckwi7v7m7m14.narinfo error="server returned 404: 404 page not found\n"14042026/11/12 00:43:09 INFO Completed push id=1 objects=3 duration=5.653829ms14052026/11/12 00:43:09 INFO Upload complete. (239ms)14062026/11/12 00:43:09 INFO Received complete push request method=POST path=/api/pushes/1/complete14072026/11/12 00:43:09 WARN Failed to register uploaded object key=i8lw4nsiprjliv7cq982k0j494agp4f8.narinfo error="server returned 404: 404 page not found\n"14082026/11/12 00:43:09 INFO Completed push id=1 objects=4 duration=10.476919ms14092026/11/12 00:43:09 INFO Upload complete. (233ms)1410=== RUN TestService_AuthMiddleware_OIDC/valid_OIDC_token1411=== PAUSE TestService_AuthMiddleware_OIDC/valid_OIDC_token1412=== RUN TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1413=== PAUSE TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1414=== RUN TestService_AuthMiddleware_OIDC/malformed_token_rejected1415=== PAUSE TestService_AuthMiddleware_OIDC/malformed_token_rejected1416=== RUN TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1417=== PAUSE TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1418=== CONT TestCompletedNarNotReofferedAcrossClosures14192026/11/12 00:43:09 INFO Completed push id=1 objects=9 duration=11.28226ms14202026/11/12 00:43:09 INFO Upload complete. (226ms)1421=== NAME TestClientMultipleUploads1422 client_integration_test.go:417: Uploaded 3 paths in 260.644966ms14232026/11/12 00:43:09 INFO Aborted multipart uploads count=0 kept=01424--- PASS: TestClientMultipleUploads (0.98s)1425=== CONT TestCompleteMultipartUpload_ErrorButObjectExists14262026/11/12 00:43:09 WARN Force mode enabled - objects will be deleted immediately without grace period14272026/11/12 00:43:09 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=014282026/11/12 00:43:09 INFO All 1 paths already cached1429=== NAME TestClientIntegration1430 client_integration_test.go:318: Retrieved narinfo from S3:1431 StorePath: /build/TestClientIntegration687895362/002/store/j553qrz1k2h1d2z6jw96dkj8wa13dd92-test-file.txt1432 URL: nar/11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1.nar.zst14332026/11/12 00:43:09 INFO Starting cleanup of old closures method=DELETE path=/api/closures1434 Compression: zstd1435 NarHash: sha256:11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk11436 NarSize: 1521437 References: 1438 CA: fixed:r:sha256:11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk114392026/11/12 00:43:09 INFO Garbage collection started14402026/11/12 00:43:09 INFO Vacuumed table table=pending_closures1441 client_integration_test.go:319: Retrieved .ls file from S3 (compressed size: 77 bytes)1442 client_integration_test.go:319: Decompressed .ls content (64 bytes):1443 {"version":1,"root":{"type":"regular","size":39,"narOffset":96}}14442026/11/12 00:43:09 INFO Vacuumed table table=commit_receipts14452026/11/12 00:43:09 INFO Vacuumed table table=pending_objects14462026/11/12 00:43:09 INFO Vacuumed table table=multipart_uploads14472026/11/12 00:43:09 INFO Vacuumed table table=closures14482026/11/12 00:43:09 INFO Vacuumed table table=objects14492026/11/12 00:43:09 INFO Aborted multipart uploads count=0 kept=01450=== NAME TestClientWithDependencies1451 client_integration_test.go:677: Skipping nix copy test - isolated store (/build/TestClientWithDependencies2051169984/001/store) requires matching store prefix1452=== RUN TestService_RequireScope_OIDC/builder_may_write1453=== PAUSE TestService_RequireScope_OIDC/builder_may_write1454=== RUN TestService_RequireScope_OIDC/builder_may_not_admin1455=== PAUSE TestService_RequireScope_OIDC/builder_may_not_admin1456=== RUN TestService_RequireScope_OIDC/ops_may_admin1457=== PAUSE TestService_RequireScope_OIDC/ops_may_admin1458=== RUN TestService_RequireScope_OIDC/ops_may_not_write1459=== PAUSE TestService_RequireScope_OIDC/ops_may_not_write1460=== RUN TestService_RequireScope_OIDC/reader_may_not_write1461=== PAUSE TestService_RequireScope_OIDC/reader_may_not_write1462=== RUN TestService_RequireScope_OIDC/static_token_may_admin1463=== PAUSE TestService_RequireScope_OIDC/static_token_may_admin1464=== RUN TestService_RequireScope_OIDC/static_token_may_write1465=== PAUSE TestService_RequireScope_OIDC/static_token_may_write1466=== RUN TestService_RequireScope_OIDC/reader_may_read1467=== PAUSE TestService_RequireScope_OIDC/reader_may_read1468=== RUN TestService_RequireScope_OIDC/writer_implies_read1469=== PAUSE TestService_RequireScope_OIDC/writer_implies_read1470=== RUN TestService_RequireScope_OIDC/anonymous_may_not_read1471=== PAUSE TestService_RequireScope_OIDC/anonymous_may_not_read1472=== RUN TestService_RequireScope_OIDC/reader_may_read_with_netrc1473=== PAUSE TestService_RequireScope_OIDC/reader_may_read_with_netrc1474=== RUN TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1475=== PAUSE TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1476=== RUN TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read1477=== PAUSE TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read1478=== RUN TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1479=== PAUSE TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1480=== CONT TestSharedNarBothUploadsComplete1481--- PASS: TestClientWithDependencies (1.06s)1482=== CONT TestCommitCanBeRepeatedUntilCleanup1483=== RUN TestCommitCanBeRepeatedUntilCleanup/pending_closure1484=== PAUSE TestCommitCanBeRepeatedUntilCleanup/pending_closure1485=== RUN TestCommitCanBeRepeatedUntilCleanup/push1486=== PAUSE TestCommitCanBeRepeatedUntilCleanup/push1487=== CONT TestPush_SignsNarinfosOfItsPendingObjects14882026/11/12 00:43:09 WARN Force mode enabled - objects will be deleted immediately without grace period1489--- PASS: TestService_AuthMiddleware_MTLSProxyHeader (0.53s)1490=== CONT TestPush_RejectsBadRequests14912026/11/12 00:43:09 INFO Received push request method=POST path=/api/pushes14922026/11/12 00:43:09 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)14932026/11/12 00:43:09 INFO Uploading gs3vaxrbqbfsd6wkpnskqkcxvanv778f-ca-test (144B)14942026/11/12 00:43:10 WARN Failed to register uploaded object key=log/1wi8fnzgi1scnldyg02zix41ai2rkcxv-ca-test.drv error="server returned 404: 404 page not found\n"14952026/11/12 00:43:10 WARN Failed to register uploaded object key=nar/04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n.nar.zst error="server returned 404: 404 page not found\n"14962026/11/12 00:43:10 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign14972026/11/12 00:43:10 WARN Failed to register uploaded object key=gs3vaxrbqbfsd6wkpnskqkcxvanv778f.ls error="server returned 404: 404 page not found\n"14982026/11/12 00:43:10 INFO Signed narinfos id=1 count=114992026/11/12 00:43:10 INFO Uploading 1 narinfos15002026/11/12 00:43:10 INFO Received complete push request method=POST path=/api/pushes/1/complete15012026/11/12 00:43:10 WARN Failed to register uploaded object key=gs3vaxrbqbfsd6wkpnskqkcxvanv778f.narinfo error="server returned 404: 404 page not found\n"15022026/11/12 00:43:10 INFO Completed push id=1 objects=4 duration=8.230959ms15032026/11/12 00:43:10 INFO Upload complete. (98ms)1504=== NAME TestClientCADerivations1505 client_ca_test.go:180: Narinfo contains CA field: StorePath: /build/TestClientCADerivations1481204780/001/store/gs3vaxrbqbfsd6wkpnskqkcxvanv778f-ca-test15062026/11/12 00:43:10 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=01507 URL: nar/04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n.nar.zst1508 Compression: zstd1509 NarHash: sha256:04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n1510 NarSize: 1441511 References: 1512 Deriver: /build/TestClientCADerivations1481204780/001/store/1wi8fnzgi1scnldyg02zix41ai2rkcxv-ca-test.drv1513 CA: fixed:r:sha256:04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n1514 client_ca_test.go:185: Checking for realisation files in S3...1515 client_ca_test.go:207: No realisation files uploaded - this is expected for locally built CA derivations1516 client_ca_test.go:208: Locally built CA derivations don't have DrvOutput IDs until they're substituted from a cache1517--- PASS: TestReadProxyRangeRequest (0.49s)1518=== CONT TestPush_SkippedKeySurvivesGCBeforeCommit15192026/11/12 00:43:10 INFO Vacuumed table table=pending_closures15202026/11/12 00:43:10 INFO Vacuumed table table=commit_receipts15212026/11/12 00:43:10 INFO Received push request method=POST path=/api/pushes15222026/11/12 00:43:10 WARN Authentication failed token_preview=wrongtoken token_length=10 reason="static API token mismatch"1523--- PASS: TestService_AuthMiddleware (0.52s)1524=== CONT TestPush_RootsAreLockedInByteOrder15252026/11/12 00:43:10 INFO Vacuumed table table=pending_objects15262026/11/12 00:43:10 INFO Object in database but missing from S3 key=j553qrz1k2h1d2z6jw96dkj8wa13dd92.ls15272026/11/12 00:43:10 WARN Found objects in DB but missing from S3, will re-upload count=115282026/11/12 00:43:10 INFO Vacuumed table table=multipart_uploads15292026/11/12 00:43:10 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)15302026/11/12 00:43:10 INFO Vacuumed table table=closures15312026/11/12 00:43:10 INFO Received complete push request method=POST path=/api/pushes/2/complete15322026/11/12 00:43:10 WARN Failed to register uploaded object key=j553qrz1k2h1d2z6jw96dkj8wa13dd92.ls error="server returned 404: 404 page not found\n"15332026/11/12 00:43:10 INFO Vacuumed table table=objects15342026/11/12 00:43:10 INFO Completed push id=2 objects=3 duration=3.15644ms15352026/11/12 00:43:10 INFO Upload complete. (55ms)1536=== NAME TestClientIntegration1537 client_integration_test.go:336: Retrieved .ls file from S3 (compressed size: 62 bytes)1538 client_integration_test.go:336: Decompressed .ls content (49 bytes):1539 {"version":1,"root":{"type":"regular","size":39}}15402026/11/12 00:43:10 INFO Received uploads request method=POST path=/api/pending_closures15412026/11/12 00:43:10 INFO Received uploads request method=POST path=/api/pending_closures15422026/11/12 00:43:10 INFO Received uploads request method=POST path=/api/pending_closures15432026/11/12 00:43:10 INFO Received uploads request method=POST path=/api/pending_closures15442026/11/12 00:43:10 INFO Received complete multipart upload request method=POST path=/api/multipart/complete15452026/11/12 00:43:10 ERROR Failed to get multipart upload error="no rows in result set" upload_id=does-not-exist object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst1546--- PASS: TestCompleteMultipartUnregistered (0.48s)1547=== CONT TestPush_OverlappingRootsStoreOneRowPerKey15482026/11/12 00:43:10 INFO Received cleanup request method=DELETE path=/api/pending_closures15492026/11/12 00:43:10 INFO Aborted multipart uploads count=0 kept=015502026/11/12 00:43:10 INFO Received uploads request method=POST path=/api/pending_closures1551=== NAME TestClientCADerivations1552 client_ca_test.go:258: nix copy output: warning: you don't have Internet access; disabling some network-dependent features1553 warning: failed to create TLS context for AWS credential providers; SSO, STS WebIdentity, and ECS container authentication will be unavailable1554 error: binary cache 's3://bucket38?endpoint=http://localhost:45911®ion=eu-west-1' is for Nix stores with prefix '/nix/store', not '/build/TestClientCADerivations1481204780/001/store'1555 client_ca_test.go:259: nix copy failed (might be expected with isolated stores): exit status 115562026/11/12 00:43:10 INFO Received cleanup request method=DELETE path=/api/pending_closures1557--- PASS: TestClientCADerivations (1.15s)1558=== CONT TestReadRedirectUsesPublicS3URL15592026/11/12 00:43:10 INFO Aborted multipart uploads count=1 kept=015602026/11/12 00:43:10 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete15612026-11-12 00:43:10.159 UTC [1530] ERROR: Closure does not exist: id=115622026-11-12 00:43:10.159 UTC [1530] CONTEXT: PL/pgSQL function commit_pending_closure(bigint) line 17 at RAISE15632026-11-12 00:43:10.159 UTC [1530] STATEMENT: -- name: CommitPendingClosure :one1564 SELECT commit_pending_closure($1::bigint)::bigint1565 1566--- PASS: TestService_cleanupPendingClosuresHandler (0.46s)1567=== CONT TestPush_CompleteWaitsForCleanupOfThePush15682026/11/12 00:43:10 INFO Received uploads request method=POST path=/api/pending_closures15692026/11/12 00:43:10 INFO Received push request method=POST path=/api/pushes15702026/11/12 00:43:10 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)1571--- PASS: TestCreatePendingClosure_SmallNARUsesSimplePUT (0.56s)1572=== CONT TestParseSingleRange1573=== RUN TestParseSingleRange/none1574=== PAUSE TestParseSingleRange/none1575=== RUN TestParseSingleRange/unknown_unit1576=== PAUSE TestParseSingleRange/unknown_unit1577=== RUN TestParseSingleRange/multi-range_ignored1578=== PAUSE TestParseSingleRange/multi-range_ignored1579=== RUN TestParseSingleRange/malformed_no_dash1580=== PAUSE TestParseSingleRange/malformed_no_dash1581=== RUN TestParseSingleRange/malformed_both_empty1582=== PAUSE TestParseSingleRange/malformed_both_empty1583=== RUN TestParseSingleRange/malformed_end_before_start1584=== PAUSE TestParseSingleRange/malformed_end_before_start1585=== RUN TestParseSingleRange/closed1586=== PAUSE TestParseSingleRange/closed1587=== RUN TestParseSingleRange/open-ended1588=== PAUSE TestParseSingleRange/open-ended1589=== RUN TestParseSingleRange/end_clamped_to_size1590=== PAUSE TestParseSingleRange/end_clamped_to_size1591=== RUN TestParseSingleRange/suffix1592=== PAUSE TestParseSingleRange/suffix1593=== RUN TestParseSingleRange/suffix_exceeds_size1594=== PAUSE TestParseSingleRange/suffix_exceeds_size1595=== RUN TestParseSingleRange/single_byte1596=== PAUSE TestParseSingleRange/single_byte1597=== RUN TestParseSingleRange/start_past_EOF1598=== PAUSE TestParseSingleRange/start_past_EOF1599=== RUN TestParseSingleRange/start_far_past_EOF1600=== PAUSE TestParseSingleRange/start_far_past_EOF1601=== CONT TestReadRedirectKeepsNarinfoProxied16022026/11/12 00:43:10 WARN Failed to register uploaded object key=j553qrz1k2h1d2z6jw96dkj8wa13dd92.ls error="server returned 404: 404 page not found\n"16032026/11/12 00:43:10 INFO Received complete push request method=POST path=/api/pushes/3/complete16042026/11/12 00:43:10 INFO Completed push id=3 objects=3 duration=1.8418ms16052026/11/12 00:43:10 INFO Upload complete. (42ms)1606=== NAME TestClientIntegration1607 client_integration_test.go:353: Retrieved .ls file from S3 (compressed size: 62 bytes)1608 client_integration_test.go:353: Decompressed .ls content (49 bytes):1609 {"version":1,"root":{"type":"regular","size":39}}1610 client_integration_test.go:364: Testing garbage collection...16112026/11/12 00:43:10 INFO Received uploads request method=POST path=/api/pending_closures16122026/11/12 00:43:10 INFO Starting cleanup of old closures method=DELETE path=/api/closures1613--- PASS: TestService_Rustfstest (0.31s)1614=== CONT TestReadRedirectNar16152026/11/12 00:43:10 INFO Garbage collection started16162026/11/12 00:43:10 INFO Aborted multipart uploads count=0 kept=016172026/11/12 00:43:10 INFO Received uploads request method=POST path=/api/pending_closures16182026/11/12 00:43:10 WARN Force mode enabled - objects will be deleted immediately without grace period16192026/11/12 00:43:10 INFO Registered completed upload object_key=nar/jjjjjjjjjjjjjjjjjjjjjjjjjjjjjj0100000000000000000000.nar.zst16202026/11/12 00:43:10 INFO Received uploads request method=POST path=/api/pending_closures1621--- PASS: TestPresignedUploadRegisteredBeforeCommit (0.35s)1622=== CONT TestReadProxyDisabled16232026/11/12 00:43:10 INFO Received uploads request method=POST path=/api/pending_closures16242026/11/12 00:43:10 INFO Received complete multipart upload request method=POST path=/api/multipart/complete16252026/11/12 00:43:10 INFO lead: released remote=192.0.2.1:1234 generation=116262026/11/12 00:43:10 INFO lead: acquired remote=192.0.2.1:1234 generation=216272026/11/12 00:43:10 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=016282026/11/12 00:43:10 INFO Received uploads request method=POST path=/api/pending_closures16292026/11/12 00:43:10 INFO Vacuumed table table=pending_closures16302026/11/12 00:43:10 INFO lead: released remote=192.0.2.1:1234 generation=116312026/11/12 00:43:10 INFO Vacuumed table table=commit_receipts16322026/11/12 00:43:10 INFO Vacuumed table table=pending_objects16332026/11/12 00:43:10 INFO Vacuumed table table=multipart_uploads16342026/11/12 00:43:10 INFO Vacuumed table table=closures16352026/11/12 00:43:10 INFO Vacuumed table table=objects16362026/11/12 00:43:10 INFO Received push request method=POST path=/api/pushes16372026/11/12 00:43:10 INFO Received complete multipart upload request method=POST path=/api/multipart/complete16382026/11/12 00:43:10 INFO lead: acquired remote=192.0.2.1:1234 generation=216392026/11/12 00:43:10 INFO lead: released remote=192.0.2.1:1234 generation=11640--- PASS: TestLeadEndsOnShutdown (2.10s)1641=== CONT TestReadProxyRootRedirectsToIndexHTML16422026/11/12 00:43:10 WARN CompleteMultipartUpload errored but object exists; treating as success error="The specified key does not exist." object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=NGQyMGJjMjAtNjY4ZC00YzhlLWE5ZTItOWIyYjMyZGM0NjkzLjgwZThiM2YzLWI4MjItNGYxZS1hMzY0LTVkYTNkZjAyZTdmMngxNzk0NDQ0MTkwMjg4MjM4Nzgx16432026/11/12 00:43:10 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign16442026/11/12 00:43:10 INFO Signed narinfos id=1 count=116452026/11/12 00:43:10 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign16462026/11/12 00:43:10 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign16472026/11/12 00:43:10 ERROR Failed to sign narinfos signer="index 1" error="signing key cannot be nil"16482026/11/12 00:43:10 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign16492026/11/12 00:43:10 ERROR Failed to sign narinfos signer=niks3-test-1 error="signer failed"16502026/11/12 00:43:10 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign16512026/11/12 00:43:10 ERROR Signer returned unexpected signature count signer=niks3-test-1 got=0 want=116522026/11/12 00:43:10 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign16532026/11/12 00:43:10 ERROR Signer returned unexpected object key signer=niks3-test-1 object_key=unexpected.narinfo1654--- PASS: TestPush_SignsNarinfosOfItsPendingObjects (0.35s)1655=== CONT TestReadProxyConditionalGet16562026/11/12 00:43:10 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=NGQyMGJjMjAtNjY4ZC00YzhlLWE5ZTItOWIyYjMyZGM0NjkzLjgwZThiM2YzLWI4MjItNGYxZS1hMzY0LTVkYTNkZjAyZTdmMngxNzk0NDQ0MTkwMjg4MjM4Nzgx parts=116572026/11/12 00:43:10 INFO Received uploads request method=POST path=/api/pending_closures1658--- PASS: TestCompleteMultipartUpload_ErrorButObjectExists (0.39s)1659=== CONT TestReadProxyOutlastsServerWriteTimeout16602026/11/12 00:43:10 INFO Received uploads request method=POST path=/api/pending_closures1661=== RUN TestPush_RejectsBadRequests/bad_root1662=== PAUSE TestPush_RejectsBadRequests/bad_root1663=== RUN TestPush_RejectsBadRequests/root_not_in_objects1664=== PAUSE TestPush_RejectsBadRequests/root_not_in_objects1665=== RUN TestPush_RejectsBadRequests/no_roots1666=== PAUSE TestPush_RejectsBadRequests/no_roots1667=== RUN TestPush_RejectsBadRequests/no_objects1668=== PAUSE TestPush_RejectsBadRequests/no_objects1669=== CONT TestReadProxyHead16702026/11/12 00:43:10 INFO lead: given back to the incumbent remote=192.0.2.1:1234 generation=216712026/11/12 00:43:10 INFO lead: acquired remote=192.0.2.1:1234 generation=316722026/11/12 00:43:10 INFO Received push request method=POST path=/api/pushes16732026/11/12 00:43:10 INFO Received complete push request method=POST path=/api/pushes/1/complete16742026/11/12 00:43:10 INFO Received push request method=POST path=/api/pushes16752026/11/12 00:43:10 INFO Received complete multipart upload request method=POST path=/api/multipart/complete16762026/11/12 00:43:10 INFO Completed push id=1 objects=4 duration=8.80579ms16772026/11/12 00:43:10 INFO Received push request method=POST path=/api/pushes16782026/11/12 00:43:10 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst upload_id=NGQyMGJjMjAtNjY4ZC00YzhlLWE5ZTItOWIyYjMyZGM0NjkzLmJmZmRjODMyLWVkYTUtNDdhYS1hNDg1LTAwZDcxZGEwY2ViYXgxNzk0NDQ0MTkwMDg0MzU1MjQz parts=1016792026/11/12 00:43:10 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete16802026/11/12 00:43:10 INFO Received complete multipart upload request method=POST path=/api/multipart/complete16812026/11/12 00:43:10 INFO Received push request method=POST path=/api/pushes16822026/11/12 00:43:10 INFO Completed upload id=1 objects=2 duration=7.276889ms16832026/11/12 00:43:10 INFO Received uploads request method=POST path=/api/pending_closures16842026/11/12 00:43:10 INFO Received uploads request method=POST path=/api/pending_closures16852026-11-12 00:43:10.451 UTC [2011] ERROR: could not obtain lock on row in relation "closures"16862026-11-12 00:43:10.451 UTC [2011] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT16872026-11-12 00:43:10.451 UTC [2011] ERROR: could not obtain lock on row in relation "closures"16882026-11-12 00:43:10.451 UTC [2011] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT16892026-11-12 00:43:10.452 UTC [2011] ERROR: could not obtain lock on row in relation "closures"16902026-11-12 00:43:10.452 UTC [2011] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT16912026-11-12 00:43:10.452 UTC [2011] ERROR: could not obtain lock on row in relation "closures"16922026-11-12 00:43:10.452 UTC [2011] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT16932026-11-12 00:43:10.452 UTC [2011] ERROR: could not obtain lock on row in relation "closures"16942026-11-12 00:43:10.452 UTC [2011] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT16952026/11/12 00:43:10 INFO Object in database but missing from S3 key=dadb44fdadb44fdadb44fdadb44f0000.narinfo16962026/11/12 00:43:10 WARN Found objects in DB but missing from S3, will re-upload count=11697--- PASS: TestService_verifyS3Integrity (0.78s)1698=== CONT TestReadProxyInvalidPath1699--- PASS: TestPush_RootsAreLockedInByteOrder (0.41s)1700=== CONT TestReadProxy40417012026/11/12 00:43:10 INFO Completed multipart upload object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst upload_id=NGQyMGJjMjAtNjY4ZC00YzhlLWE5ZTItOWIyYjMyZGM0NjkzLjg0YTdjODY3LWUyYmEtNDRlOC1hOTYyLWYwMTdiOGI2ODI3OXgxNzk0NDQ0MTkwMTEwODA2OTQx parts=1017022026/11/12 00:43:10 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete17032026/11/12 00:43:10 INFO Completed upload id=1 objects=2 duration=2.898319ms17042026/11/12 00:43:10 INFO Received get closure request method=GET path=/api/closures/0000000000000000000000000000000017052026/11/12 00:43:10 INFO Received uploads request method=POST path=/api/pending_closures17062026/11/12 00:43:10 INFO Starting cleanup of old closures method=DELETE path=/api/closures1707--- PASS: TestPush_OverlappingRootsStoreOneRowPerKey (0.33s)1708=== CONT TestReadProxyNarStreaming17092026/11/12 00:43:10 INFO Aborted multipart uploads count=0 kept=017102026/11/12 00:43:10 INFO Aborted multipart uploads count=0 kept=017112026/11/12 00:43:10 WARN Force mode enabled - objects will be deleted immediately without grace period17122026/11/12 00:43:10 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=017132026/11/12 00:43:10 INFO Vacuumed table table=pending_closures1714--- PASS: TestReadRedirectUsesPublicS3URL (0.34s)1715=== CONT TestReadProxyContentNegotiation17162026/11/12 00:43:10 INFO Vacuumed table table=commit_receipts17172026/11/12 00:43:10 INFO Vacuumed table table=pending_objects17182026/11/12 00:43:10 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=2 objects-deleted-after-grace-period=2 objects-failed-to-delete=017192026/11/12 00:43:10 INFO Received push request method=POST path=/api/pushes17202026/11/12 00:43:10 INFO Vacuumed table table=multipart_uploads17212026/11/12 00:43:10 INFO Vacuumed table table=pending_closures17222026/11/12 00:43:10 INFO Vacuumed table table=closures17232026/11/12 00:43:10 INFO Vacuumed table table=commit_receipts17242026/11/12 00:43:10 INFO Vacuumed table table=objects17252026/11/12 00:43:10 INFO Vacuumed table table=pending_objects17262026/11/12 00:43:10 INFO Vacuumed table table=multipart_uploads17272026/11/12 00:43:10 INFO Vacuumed table table=closures17282026/11/12 00:43:10 INFO Vacuumed table table=objects17292026/11/12 00:43:10 INFO Received complete push request method=POST path=/api/pushes/2/complete17302026/11/12 00:43:10 INFO Completed push id=2 objects=4 duration=5.43112ms1731--- PASS: TestPush_SkippedKeySurvivesGCBeforeCommit (0.51s)1732=== CONT TestReadProxyNarinfoAlreadyDecompressed1733--- PASS: TestReadRedirectKeepsNarinfoProxied (0.38s)1734=== CONT TestReadProxyNarinfo17352026/11/12 00:43:10 INFO Received get closure request method=GET path=/api/closures/000000000000000000000000000000001736--- PASS: TestService_createPendingClosureHandler (0.88s)1737=== CONT TestIsValidCachePath1738=== RUN TestIsValidCachePath/narinfo1739=== PAUSE TestIsValidCachePath/narinfo1740=== RUN TestIsValidCachePath/narinfo_all_nix_base32_chars1741=== PAUSE TestIsValidCachePath/narinfo_all_nix_base32_chars1742=== RUN TestIsValidCachePath/nar_zst1743=== PAUSE TestIsValidCachePath/nar_zst1744=== RUN TestIsValidCachePath/nar_xz1745=== PAUSE TestIsValidCachePath/nar_xz1746=== RUN TestIsValidCachePath/nar_bz21747=== PAUSE TestIsValidCachePath/nar_bz21748=== RUN TestIsValidCachePath/nar_uncompressed1749=== PAUSE TestIsValidCachePath/nar_uncompressed1750=== RUN TestIsValidCachePath/ls1751=== PAUSE TestIsValidCachePath/ls1752=== RUN TestIsValidCachePath/log1753=== PAUSE TestIsValidCachePath/log1754=== RUN TestIsValidCachePath/realisation1755=== PAUSE TestIsValidCachePath/realisation1756=== RUN TestIsValidCachePath/nix-cache-info1757=== PAUSE TestIsValidCachePath/nix-cache-info1758=== RUN TestIsValidCachePath/index.html1759=== PAUSE TestIsValidCachePath/index.html1760=== RUN TestIsValidCachePath/traversal_parent1761=== PAUSE TestIsValidCachePath/traversal_parent1762=== RUN TestIsValidCachePath/traversal_in_middle1763=== PAUSE TestIsValidCachePath/traversal_in_middle1764=== RUN TestIsValidCachePath/invalid_char_e1765=== PAUSE TestIsValidCachePath/invalid_char_e1766=== RUN TestIsValidCachePath/invalid_char_u1767=== PAUSE TestIsValidCachePath/invalid_char_u1768=== RUN TestIsValidCachePath/random_path1769=== PAUSE TestIsValidCachePath/random_path1770=== RUN TestIsValidCachePath/empty1771=== PAUSE TestIsValidCachePath/empty1772=== RUN TestIsValidCachePath/leading_slash1773=== PAUSE TestIsValidCachePath/leading_slash1774=== RUN TestIsValidCachePath/wrong_extension1775=== PAUSE TestIsValidCachePath/wrong_extension1776=== RUN TestIsValidCachePath/short_hash1777=== PAUSE TestIsValidCachePath/short_hash1778=== CONT TestPush_RepeatedRootCommitsOnce1779--- PASS: TestReadRedirectNar (0.37s)1780=== CONT TestObjectStatsTrigger17812026/11/12 00:43:10 INFO lead: released remote=192.0.2.1:1234 generation=11782--- PASS: TestReadProxyDisabled (0.35s)1783=== CONT TestPresentNotReportedWhileGCDeletesClosure1784=== RUN TestPresentNotReportedWhileGCDeletesClosure/stale1785=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/stale1786=== RUN TestPresentNotReportedWhileGCDeletesClosure/fresh1787=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/fresh1788=== CONT TestPresentRefreshesOnlyStaleClosures1789--- PASS: TestDeleteClosuresRacingPins (1.88s)1790=== CONT TestPresentReportsOnlyClosureRoots17912026/11/12 00:43:10 INFO lead: acquired remote=192.0.2.1:1234 generation=21792--- PASS: TestReadProxyRootRedirectsToIndexHTML (0.32s)1793=== CONT TestCreatePin_ReservedPins1794--- PASS: TestReadProxyConditionalGet (0.40s)1795=== CONT TestOrphanedObjectsGCStressTest1796--- PASS: TestReadProxyHead (0.40s)1797=== CONT TestOrphanedObjectsGC17982026/11/12 00:43:10 INFO Received complete multipart upload request method=POST path=/api/multipart/complete1799--- PASS: TestReadProxyInvalidPath (0.33s)1800=== CONT TestObjectStatsUpdatedOncePerCommit18012026/11/12 00:43:10 INFO Completed multipart upload object_key=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhh0100000000000000000000.nar.zst upload_id=NGQyMGJjMjAtNjY4ZC00YzhlLWE5ZTItOWIyYjMyZGM0NjkzLjQyMTFjZjE1LTk5NWQtNDcxMS05YTFkLTk1YTM1MTkyZjViZHgxNzk0NDQ0MTkwMjY4MDMzMzEy parts=1218022026/11/12 00:43:10 INFO Received uploads request method=POST path=/api/pending_closures18032026/11/12 00:43:10 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:39011/oidc1804--- PASS: TestCompletedNarNotReofferedAcrossClosures (0.89s)1805=== CONT TestServerTLSConfig1806=== RUN TestServerTLSConfig/no_client_CA1807=== PAUSE TestServerTLSConfig/no_client_CA1808=== RUN TestServerTLSConfig/missing_CA_file1809=== PAUSE TestServerTLSConfig/missing_CA_file1810=== RUN TestServerTLSConfig/not_a_PEM_file1811=== PAUSE TestServerTLSConfig/not_a_PEM_file1812=== CONT TestPendingCleanupUsesOneCutoff18132026-11-12 00:43:10.839 UTC [2027] ERROR: Push does not exist: id=118142026-11-12 00:43:10.839 UTC [2027] CONTEXT: PL/pgSQL function commit_push(bigint) line 15 at RAISE18152026-11-12 00:43:10.839 UTC [2027] STATEMENT: SELECT commit_push($1)1816--- PASS: TestPush_CompleteWaitsForCleanupOfThePush (0.68s)1817=== CONT TestMultipartUploadAbortedWhenCancelledBeforeRecorded1818--- PASS: TestReadProxyNarStreaming (0.39s)1819=== CONT TestMultipartCleanup18202026/11/12 00:43:10 INFO Received complete multipart upload request method=POST path=/api/multipart/complete1821--- PASS: TestReadProxy404 (0.43s)1822=== CONT TestMetricsCommit18232026/11/12 00:43:10 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0100000000000000000000.nar.zst upload_id=NGQyMGJjMjAtNjY4ZC00YzhlLWE5ZTItOWIyYjMyZGM0NjkzLmM5MjU3ODZjLTY0MTktNDUwMS1hNWZhLWE2MWUzNGM2ZDhhZXgxNzk0NDQ0MTkwMzQxODQ1NDg4 parts=121824--- PASS: TestReadProxyContentNegotiation (0.45s)1825=== CONT TestService_NativeMTLS1826--- PASS: TestReadProxyNarinfoAlreadyDecompressed (0.42s)1827=== CONT TestMetricsInventory18282026/11/12 00:43:10 INFO Received push request method=POST path=/api/pushes18292026/11/12 00:43:11 ERROR Failed to decode narinfo key=3hcdxyjf9yiq7qf3i4548drb6sjmwa1v.narinfo limit=16777216 error=<nil>1830--- PASS: TestReadProxyNarinfo (0.46s)1831=== CONT TestPush_CompleteCommitsEveryRoot18322026/11/12 00:43:11 INFO Received complete push request method=POST path=/api/pushes/1/complete18332026/11/12 00:43:11 INFO Completed push id=1 objects=4 duration=8.13044ms1834--- PASS: TestPush_RepeatedRootCommitsOnce (0.46s)1835=== CONT TestForceGCDuringPushOffersSweptObject/before_pending_rows1836--- PASS: TestObjectStatsTrigger (0.47s)1837=== CONT TestForceGCDuringPushOffersSweptObject/after_presence_check1838--- PASS: TestPresentRefreshesOnlyStaleClosures (0.47s)1839=== CONT TestGCEndsOnShutdown/before_the_run1840--- PASS: TestPresentReportsOnlyClosureRoots (0.49s)1841=== CONT TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete18422026/11/12 00:43:11 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux18432026/11/12 00:43:11 WARN Refused reserved pin name=worker-x86_64-linux18442026/11/12 00:43:11 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux18452026/11/12 00:43:11 INFO Received create pin request method=POST path=/api/pins/my-app18462026/11/12 00:43:11 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux1847--- PASS: TestCreatePin_ReservedPins (0.59s)1848=== CONT TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark1849--- PASS: TestObjectStatsUpdatedOncePerCommit (0.46s)1850=== CONT TestResolveDBConnectionString/flag_wins1851=== CONT TestResolveDBConnectionString/PGHOST_allows_empty1852=== CONT TestResolveDBConnectionString/nothing_configured1853=== CONT TestResolveDBConnectionString/missing_file_is_an_error1854=== CONT TestResolveDBConnectionString/file_when_flag_empty1855=== CONT TestClientErrorHandling/InvalidStorePath1856--- PASS: TestResolveDBConnectionString (0.00s)1857 --- PASS: TestResolveDBConnectionString/flag_wins (0.00s)1858 --- PASS: TestResolveDBConnectionString/PGHOST_allows_empty (0.00s)1859 --- PASS: TestResolveDBConnectionString/nothing_configured (0.00s)1860 --- PASS: TestResolveDBConnectionString/missing_file_is_an_error (0.00s)1861 --- PASS: TestResolveDBConnectionString/file_when_flag_empty (0.00s)18622026/11/12 00:43:11 INFO Received uploads request method=POST path=/api/pending_closures18632026/11/12 00:43:11 INFO Received uploads request method=POST path=/api/pending_closures18642026/11/12 00:43:11 INFO Received uploads request method=POST path=/api/pending_closures18652026/11/12 00:43:11 INFO Received cleanup request method=DELETE path=/api/pending_closures1866--- PASS: TestMultipartUploadAbortedWhenCancelledBeforeRecorded (0.47s)1867=== CONT TestClientErrorHandling/ServerNotAvailable18682026/11/12 00:43:11 INFO Received uploads request method=POST path=/api/pending_closures18692026/11/12 00:43:11 INFO Received push request method=POST path=/api/pushes18702026/11/12 00:43:11 INFO Received complete push request method=POST path=/api/pushes/1/complete18712026/11/12 00:43:11 INFO lead: released remote=192.0.2.1:1234 generation=218722026/11/12 00:43:11 INFO Completed push id=1 objects=4 duration=5.349149ms18732026/11/12 00:43:11 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete18742026/11/12 00:43:11 INFO Completed upload id=2 objects=2 duration=905.34µs18752026/11/12 00:43:11 WARN mTLS auth: subject not in bound subjects subject="CN=reader"18762026/11/12 00:43:11 WARN mTLS auth: subject not in bound subjects subject="CN=reader"1877--- PASS: TestService_NativeMTLS (0.43s)1878=== CONT TestClientErrorHandling/InvalidAuthToken1879--- PASS: TestMetricsCommit (0.49s)1880=== CONT TestCacheConfigHandler/full_config,_no_issuer1881=== CONT TestCacheConfigHandler/no_signing_keys1882=== CONT TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1883=== CONT TestCacheConfigHandler/no_cache_url_configured1884--- PASS: TestCacheConfigHandler (0.00s)1885 --- PASS: TestCacheConfigHandler/full_config,_no_issuer (0.00s)1886 --- PASS: TestCacheConfigHandler/no_signing_keys (0.00s)1887 --- PASS: TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator (0.00s)1888 --- PASS: TestCacheConfigHandler/no_cache_url_configured (0.00s)1889=== CONT TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed18902026/11/12 00:43:11 INFO lead: acquired remote=192.0.2.1:1234 generation=318912026/11/12 00:43:11 INFO lead: released remote=192.0.2.1:1234 generation=31892--- PASS: TestLeadGenerationGrowsWithEveryGrant (3.16s)1893=== CONT TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed18942026/11/12 00:43:11 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://localhost:19999/api/objects/present\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/objects/present1895--- PASS: TestMetricsInventory (0.46s)1896=== CONT TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed18972026/11/12 00:43:11 INFO Received cleanup request method=DELETE path=/api/pending_closures18982026/11/12 00:43:11 INFO Received push request method=POST path=/api/pushes18992026/11/12 00:43:11 WARN Failed to abort upload, keeping its closure key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst error="Get \"http://127.0.0.1:1/bucket89/?location=\": dial tcp 127.0.0.1:1: connect: connection refused" code=""19002026/11/12 00:43:11 INFO Aborted multipart uploads count=0 kept=119012026/11/12 00:43:11 INFO Received cleanup request method=DELETE path=/api/pending_closures19022026/11/12 00:43:11 INFO Aborted multipart uploads count=1 kept=019032026/11/12 00:43:11 INFO Received complete multipart upload request method=POST path=/api/multipart/complete1904--- PASS: TestMultipartCleanup (0.59s)1905=== CONT TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost19062026/11/12 00:43:11 INFO Received complete push request method=POST path=/api/pushes/1/complete19072026/11/12 00:43:11 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0100000000000000000000.nar.zst upload_id=NGQyMGJjMjAtNjY4ZC00YzhlLWE5ZTItOWIyYjMyZGM0NjkzLjc0MGUzYzg0LTM5MzctNDE0MS1hNGE4LTE1NGY2MDMwNjEzOHgxNzk0NDQ0MTkwMzUxMjg2MTE4 parts=121908--- PASS: TestSharedNarBothUploadsComplete (1.51s)1909=== CONT TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed19102026/11/12 00:43:11 INFO Completed push id=1 objects=6 duration=7.41327ms19112026/11/12 00:43:11 INFO Received uploads request method=POST path=/api/pending_closures1912--- PASS: TestPush_CompleteCommitsEveryRoot (0.47s)1913=== CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info19142026/11/12 00:43:11 INFO Received uploads request method=POST path=/1915=== CONT TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key19162026/11/12 00:43:11 INFO Received complete multipart upload request method=POST path=/1917=== CONT TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key19182026/11/12 00:43:11 INFO Received request for more parts method=POST path=/1919=== CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal19202026/11/12 00:43:11 INFO Received uploads request method=POST path=/1921--- PASS: TestUploadHandlersRejectInvalidKeys (0.09s)1922 --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info (0.00s)1923 --- PASS: TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key (0.00s)1924 --- PASS: TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key (0.00s)1925 --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal (0.00s)1926=== CONT TestUploadHandlersRejectOversizedBody/create_pending_closure19272026/11/12 00:43:11 INFO Received uploads request method=POST path=/19282026/11/12 00:43:11 WARN Request failed, retrying attempt=2 backoff=198.507046ms error="Post \"http://localhost:19999/api/objects/present\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/objects/present19292026/11/12 00:43:11 INFO Aborted multipart uploads count=0 kept=019302026/11/12 00:43:11 WARN Force mode enabled - objects will be deleted immediately without grace period19312026/11/12 00:43:11 INFO Received uploads request method=POST path=/api/pending_closures19322026/11/12 00:43:11 INFO Aborted multipart uploads count=0 kept=019332026/11/12 00:43:11 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=1 objects-deleted-after-grace-period=1 objects-failed-to-delete=019342026/11/12 00:43:11 INFO Vacuumed table table=pending_closures19352026/11/12 00:43:11 INFO Vacuumed table table=commit_receipts19362026/11/12 00:43:11 WARN Force mode enabled - objects will be deleted immediately without grace period19372026/11/12 00:43:11 INFO Vacuumed table table=pending_objects19382026/11/12 00:43:11 INFO Vacuumed table table=multipart_uploads19392026/11/12 00:43:11 INFO Vacuumed table table=closures19402026/11/12 00:43:11 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=01941=== CONT TestUploadHandlersRejectOversizedBody/request_more_parts19422026/11/12 00:43:11 INFO Received request for more parts method=POST path=/19432026/11/12 00:43:11 INFO Vacuumed table table=objects19442026/11/12 00:43:11 INFO Vacuumed table table=pending_closures19452026/11/12 00:43:11 INFO Vacuumed table table=commit_receipts19462026/11/12 00:43:11 INFO Vacuumed table table=pending_objects19472026/11/12 00:43:11 INFO Vacuumed table table=multipart_uploads19482026/11/12 00:43:11 INFO Vacuumed table table=closures19492026/11/12 00:43:11 INFO Vacuumed table table=objects1950=== CONT TestUploadHandlersRejectOversizedBody/complete_multipart19512026/11/12 00:43:11 INFO Received complete multipart upload request method=POST path=/19522026/11/12 00:43:11 INFO Aborted multipart uploads count=0 kept=019532026/11/12 00:43:11 WARN Force mode enabled - objects will be deleted immediately without grace period1954--- PASS: TestForceGCDuringPushOffersSweptObject (0.00s)1955 --- PASS: TestForceGCDuringPushOffersSweptObject/after_presence_check (0.53s)1956 --- PASS: TestForceGCDuringPushOffersSweptObject/before_pending_rows (0.56s)1957=== CONT TestProxyWriteTimeout/narinfo1958=== CONT TestProxyWriteTimeout/unknown_size1959=== CONT TestProxyWriteTimeout/10_GiB_nar1960=== CONT TestProxyWriteTimeout/1_GiB_nar1961--- PASS: TestProxyWriteTimeout (0.00s)1962 --- PASS: TestProxyWriteTimeout/narinfo (0.00s)1963 --- PASS: TestProxyWriteTimeout/unknown_size (0.00s)1964 --- PASS: TestProxyWriteTimeout/10_GiB_nar (0.00s)1965 --- PASS: TestProxyWriteTimeout/1_GiB_nar (0.00s)1966=== CONT TestPendingClosureWriteTimeout/empty1967=== CONT TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline1968=== NAME TestOrphanedObjectsGC1969 orphaned_objects_gc_test.go:287: GC Test Summary:1970 orphaned_objects_gc_test.go:288: - Kept: 2 objects from closure A1971 orphaned_objects_gc_test.go:289: - Deleted: 2 objects from closure B1972 orphaned_objects_gc_test.go:290: - Deleted: 6 orphaned chain objects (X1->X2->X3)1973 orphaned_objects_gc_test.go:291: - Deleted: 2 orphaned single objects (Y)1974 orphaned_objects_gc_test.go:292: - Total deleted: 10 objects1975--- PASS: TestOrphanedObjectsGC (0.83s)1976=== CONT TestPendingClosureWriteTimeout/670k_objects1977=== CONT TestPendingClosureWriteTimeout/400_objects1978=== CONT TestIsValidUploadKey/narinfo1979=== CONT TestIsValidUploadKey/realisation_plus_in_output1980=== CONT TestIsValidUploadKey/unknown_type1981=== CONT TestIsValidUploadKey/empty_key1982=== CONT TestIsValidUploadKey/absolute1983=== CONT TestIsValidUploadKey/traversal_nar1984=== CONT TestIsValidUploadKey/traversal1985=== CONT TestIsValidUploadKey/listing_key,_narinfo_type1986=== CONT TestIsValidUploadKey/index.html1987=== CONT TestIsValidUploadKey/nix-cache-info1988=== CONT TestIsValidUploadKey/nar_key,_narinfo_type1989=== CONT TestIsValidUploadKey/narinfo_key,_nar_type1990=== CONT TestIsValidUploadKey/build_log_home-manager_file1991=== CONT TestIsValidUploadKey/build_log_equals1992=== CONT TestIsValidUploadKey/build_log_question_mark1993=== CONT TestIsValidUploadKey/realisation1994=== CONT TestIsValidUploadKey/build_log_plus_in_name1995=== CONT TestIsValidUploadKey/nar_plain1996=== CONT TestIsValidUploadKey/nar_zst1997=== CONT TestIsValidUploadKey/build_log1998=== CONT TestIsValidUploadKey/nar_xz1999=== CONT TestIsValidUploadKey/listing2000--- PASS: TestIsValidUploadKey (0.10s)
loading 870 hidden lines…
28712026/11/12 00:43:24 INFO Received uploads request method=POST path=/28722026/11/12 00:43:24 INFO Received push request method=POST path=/28732026/11/12 00:43:24 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=028742026/11/12 00:43:24 INFO Aborted multipart uploads count=0 kept=028752026/11/12 00:43:24 INFO Vacuumed table table=pending_closures28762026/11/12 00:43:24 INFO Vacuumed table table=commit_receipts28772026/11/12 00:43:24 INFO Received push request method=POST path=/28782026/11/12 00:43:24 INFO Received complete upload request method=POST path=/28792026/11/12 00:43:24 INFO Received complete upload request method=POST path=/28802026/11/12 00:43:24 INFO Vacuumed table table=pending_objects28812026/11/12 00:43:24 WARN Force mode enabled - objects will be deleted immediately without grace period28822026/11/12 00:43:24 INFO Vacuumed table table=multipart_uploads28832026/11/12 00:43:24 INFO Completed upload id=11 objects=6 duration=2.12772ms28842026/11/12 00:43:24 INFO Vacuumed table table=closures28852026/11/12 00:43:24 INFO Completed upload id=5 objects=7 duration=3.70894ms28862026/11/12 00:43:24 INFO Vacuumed table table=objects28872026/11/12 00:43:24 INFO Received push request method=POST path=/28882026/11/12 00:43:24 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=4 objects-failed-to-delete=028892026/11/12 00:43:24 INFO Received push request method=POST path=/28902026/11/12 00:43:24 INFO Received uploads request method=POST path=/28912026/11/12 00:43:24 INFO Vacuumed table table=pending_closures28922026/11/12 00:43:24 INFO Vacuumed table table=commit_receipts28932026/11/12 00:43:24 INFO Vacuumed table table=pending_objects28942026/11/12 00:43:24 INFO Vacuumed table table=multipart_uploads28952026/11/12 00:43:24 INFO Vacuumed table table=closures28962026/11/12 00:43:24 INFO Vacuumed table table=objects28972026/11/12 00:43:24 INFO Received uploads request method=POST path=/28982026/11/12 00:43:24 INFO Aborted multipart uploads count=0 kept=028992026/11/12 00:43:24 INFO Received push request method=POST path=/29002026/11/12 00:43:24 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."29012026/11/12 00:43:24 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."29022026/11/12 00:43:24 INFO Received complete push request method=POST path=/29032026/11/12 00:43:24 INFO Received complete upload request method=POST path=/29042026/11/12 00:43:24 INFO Received push request method=POST path=/29052026/11/12 00:43:24 INFO Completed upload id=6 objects=9 duration=1.57702ms29062026/11/12 00:43:24 INFO Received complete upload request method=POST path=/29072026/11/12 00:43:24 INFO Completed push id=7 objects=20 duration=3.16942ms29082026/11/12 00:43:24 INFO Received uploads request method=POST path=/29092026/11/12 00:43:24 INFO Completed upload id=10 objects=6 duration=2.99488ms29102026/11/12 00:43:24 INFO Received complete push request method=POST path=/29112026/11/12 00:43:24 INFO Received complete push request method=POST path=/29122026/11/12 00:43:24 INFO Received complete push request method=POST path=/29132026/11/12 00:43:24 INFO Completed push id=3 objects=22 duration=4.902919ms29142026/11/12 00:43:24 INFO Completed push id=5 objects=3 duration=2.011ms29152026/11/12 00:43:24 INFO Aborted multipart uploads count=0 kept=029162026/11/12 00:43:24 INFO Completed push id=12 objects=3 duration=2.53272ms29172026/11/12 00:43:24 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."29182026/11/12 00:43:24 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."29192026/11/12 00:43:24 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."29202026/11/12 00:43:24 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."29212026/11/12 00:43:24 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."29222026/11/12 00:43:24 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."29232026/11/12 00:43:24 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."29242026/11/12 00:43:24 INFO Aborted multipart uploads count=0 kept=029252026/11/12 00:43:24 WARN Force mode enabled - objects will be deleted immediately without grace period29262026/11/12 00:43:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029272026/11/12 00:43:24 INFO Vacuumed table table=pending_closures29282026/11/12 00:43:24 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."29292026/11/12 00:43:24 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."29302026/11/12 00:43:24 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."29312026/11/12 00:43:24 INFO Vacuumed table table=commit_receipts29322026/11/12 00:43:24 INFO Vacuumed table table=pending_objects29332026/11/12 00:43:24 INFO Vacuumed table table=multipart_uploads29342026/11/12 00:43:24 INFO Vacuumed table table=closures29352026/11/12 00:43:24 INFO Vacuumed table table=objects29362026/11/12 00:43:24 INFO Received complete push request method=POST path=/29372026/11/12 00:43:24 INFO Received complete upload request method=POST path=/29382026/11/12 00:43:24 INFO Received push request method=POST path=/29392026/11/12 00:43:24 INFO Completed upload id=3 objects=8 duration=2.10295ms29402026/11/12 00:43:24 INFO Completed push id=12 objects=15 duration=2.70668ms29412026/11/12 00:43:24 INFO Received uploads request method=POST path=/29422026/11/12 00:43:24 INFO Aborted multipart uploads count=0 kept=029432026/11/12 00:43:24 INFO Received complete push request method=POST path=/29442026/11/12 00:43:25 INFO Completed push id=2 objects=25 duration=3.74935ms29452026/11/12 00:43:25 INFO Received uploads request method=POST path=/29462026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=029472026/11/12 00:43:25 INFO Received complete push request method=POST path=/29482026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=029492026/11/12 00:43:25 INFO Received push request method=POST path=/29502026/11/12 00:43:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029512026/11/12 00:43:25 INFO Vacuumed table table=pending_closures29522026/11/12 00:43:25 INFO Completed push id=11 objects=8 duration=4.20588ms29532026/11/12 00:43:25 INFO Vacuumed table table=commit_receipts29542026/11/12 00:43:25 INFO Vacuumed table table=pending_objects29552026/11/12 00:43:25 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029562026/11/12 00:43:25 INFO Vacuumed table table=multipart_uploads29572026/11/12 00:43:25 INFO Vacuumed table table=closures29582026/11/12 00:43:25 INFO Received push request method=POST path=/29592026/11/12 00:43:25 INFO Vacuumed table table=pending_closures29602026/11/12 00:43:25 INFO Vacuumed table table=objects29612026/11/12 00:43:25 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."29622026/11/12 00:43:25 INFO Vacuumed table table=commit_receipts29632026/11/12 00:43:25 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."29642026/11/12 00:43:25 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."29652026/11/12 00:43:25 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."29662026/11/12 00:43:25 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."29672026/11/12 00:43:25 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."29682026/11/12 00:43:25 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."29692026/11/12 00:43:25 INFO Received push request method=POST path=/29702026/11/12 00:43:25 INFO Received uploads request method=POST path=/29712026/11/12 00:43:25 INFO Vacuumed table table=pending_objects29722026/11/12 00:43:25 INFO Vacuumed table table=multipart_uploads29732026/11/12 00:43:25 INFO Vacuumed table table=closures29742026/11/12 00:43:25 INFO Received complete push request method=POST path=/29752026/11/12 00:43:25 INFO Received complete upload request method=POST path=/29762026/11/12 00:43:25 INFO Vacuumed table table=objects29772026/11/12 00:43:25 INFO Completed push id=8 objects=13 duration=2.12556ms29782026/11/12 00:43:25 INFO Completed upload id=4 objects=14 duration=2.66609ms29792026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=029802026/11/12 00:43:25 INFO Received uploads request method=POST path=/29812026/11/12 00:43:25 INFO Received uploads request method=POST path=/29822026/11/12 00:43:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029832026/11/12 00:43:25 INFO Vacuumed table table=pending_closures29842026/11/12 00:43:25 INFO Received complete upload request method=POST path=/29852026/11/12 00:43:25 INFO Vacuumed table table=commit_receipts29862026/11/12 00:43:25 INFO Completed upload id=3 objects=5 duration=1.50816ms29872026/11/12 00:43:25 INFO Received uploads request method=POST path=/29882026/11/12 00:43:25 INFO Vacuumed table table=pending_objects29892026/11/12 00:43:25 INFO Vacuumed table table=multipart_uploads29902026/11/12 00:43:25 INFO Vacuumed table table=closures29912026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=029922026/11/12 00:43:25 INFO Vacuumed table table=objects29932026/11/12 00:43:25 INFO Received uploads request method=POST path=/29942026/11/12 00:43:25 INFO Received push request method=POST path=/29952026/11/12 00:43:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=4 objects-deleted-after-grace-period=0 objects-failed-to-delete=029962026/11/12 00:43:25 INFO Vacuumed table table=pending_closures29972026/11/12 00:43:25 INFO Vacuumed table table=commit_receipts29982026/11/12 00:43:25 INFO Vacuumed table table=pending_objects29992026/11/12 00:43:25 INFO Vacuumed table table=multipart_uploads30002026/11/12 00:43:25 INFO Vacuumed table table=closures30012026/11/12 00:43:25 INFO Vacuumed table table=objects30022026/11/12 00:43:25 INFO Received push request method=POST path=/30032026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=030042026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=030052026/11/12 00:43:25 INFO Received complete push request method=POST path=/30062026/11/12 00:43:25 INFO Received push request method=POST path=/30072026/11/12 00:43:25 INFO Received push request method=POST path=/30082026/11/12 00:43:25 INFO Completed push id=5 objects=3 duration=4.259669ms30092026/11/12 00:43:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030102026/11/12 00:43:25 INFO Received complete push request method=POST path=/30112026/11/12 00:43:25 INFO Vacuumed table table=pending_closures30122026/11/12 00:43:25 INFO Received push request method=POST path=/30132026/11/12 00:43:25 INFO Vacuumed table table=commit_receipts30142026/11/12 00:43:25 INFO Completed push id=10 objects=2 duration=2.46194ms30152026/11/12 00:43:25 INFO Vacuumed table table=pending_objects30162026/11/12 00:43:25 INFO Vacuumed table table=multipart_uploads30172026/11/12 00:43:25 INFO Vacuumed table table=closures30182026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=030192026/11/12 00:43:25 INFO Vacuumed table table=objects30202026/11/12 00:43:25 INFO Received push request method=POST path=/30212026/11/12 00:43:25 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=030222026/11/12 00:43:25 WARN Force mode enabled - objects will be deleted immediately without grace period30232026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=030242026/11/12 00:43:25 INFO Vacuumed table table=pending_closures30252026/11/12 00:43:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030262026/11/12 00:43:25 INFO Vacuumed table table=commit_receipts30272026/11/12 00:43:25 INFO Received complete push request method=POST path=/30282026/11/12 00:43:25 INFO Vacuumed table table=pending_objects30292026/11/12 00:43:25 INFO Vacuumed table table=pending_closures30302026/11/12 00:43:25 INFO Vacuumed table table=multipart_uploads30312026/11/12 00:43:25 WARN Force mode enabled - objects will be deleted immediately without grace period30322026/11/12 00:43:25 INFO Vacuumed table table=commit_receipts30332026/11/12 00:43:25 INFO Vacuumed table table=closures30342026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=030352026/11/12 00:43:25 INFO Completed push id=13 objects=6 duration=3.28373ms30362026/11/12 00:43:25 INFO Vacuumed table table=objects30372026/11/12 00:43:25 INFO Vacuumed table table=pending_objects30382026/11/12 00:43:25 INFO Vacuumed table table=multipart_uploads30392026/11/12 00:43:25 INFO Vacuumed table table=closures30402026/11/12 00:43:25 INFO Vacuumed table table=objects30412026/11/12 00:43:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030422026/11/12 00:43:25 INFO Vacuumed table table=pending_closures30432026/11/12 00:43:25 INFO Vacuumed table table=commit_receipts30442026/11/12 00:43:25 INFO Vacuumed table table=pending_objects30452026/11/12 00:43:25 INFO Vacuumed table table=multipart_uploads30462026/11/12 00:43:25 INFO Vacuumed table table=closures30472026/11/12 00:43:25 INFO Received push request method=POST path=/30482026/11/12 00:43:25 INFO Vacuumed table table=objects30492026/11/12 00:43:25 INFO Received uploads request method=POST path=/30502026/11/12 00:43:25 INFO Received push request method=POST path=/30512026/11/12 00:43:25 INFO Received uploads request method=POST path=/30522026/11/12 00:43:25 INFO Received push request method=POST path=/30532026/11/12 00:43:25 INFO Received complete upload request method=POST path=/30542026/11/12 00:43:25 INFO Received complete upload request method=POST path=/30552026/11/12 00:43:25 INFO Received push request method=POST path=/30562026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=030572026/11/12 00:43:25 INFO Completed upload id=12 objects=21 duration=1.40783ms30582026/11/12 00:43:25 INFO Received complete upload request method=POST path=/30592026/11/12 00:43:25 INFO Completed upload id=2 objects=2 duration=2.76091ms30602026/11/12 00:43:25 INFO Completed upload id=16 objects=23 duration=1.73268ms30612026/11/12 00:43:25 INFO Received push request method=POST path=/30622026/11/12 00:43:25 INFO Received uploads request method=POST path=/30632026/11/12 00:43:25 INFO Received push request method=POST path=/30642026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=030652026/11/12 00:43:25 INFO Received push request method=POST path=/30662026/11/12 00:43:25 INFO Received push request method=POST path=/30672026/11/12 00:43:25 INFO Received complete upload request method=POST path=/30682026/11/12 00:43:25 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."30692026/11/12 00:43:25 INFO Completed upload id=15 objects=6 duration=1.25719ms30702026/11/12 00:43:25 INFO Received complete push request method=POST path=/30712026/11/12 00:43:25 INFO Completed push id=6 objects=5 duration=2.34614ms30722026/11/12 00:43:25 INFO Received complete upload request method=POST path=/30732026/11/12 00:43:25 INFO Received push request method=POST path=/30742026/11/12 00:43:25 INFO Received complete push request method=POST path=/30752026/11/12 00:43:25 INFO Completed upload id=10 objects=4 duration=3.10103ms30762026/11/12 00:43:25 INFO Completed push id=2 objects=10 duration=3.83968ms30772026/11/12 00:43:25 INFO Received complete upload request method=POST path=/30782026/11/12 00:43:25 INFO Received uploads request method=POST path=/30792026/11/12 00:43:25 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."30802026/11/12 00:43:25 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."30812026/11/12 00:43:25 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."30822026/11/12 00:43:25 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."30832026/11/12 00:43:25 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."30842026/11/12 00:43:25 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."30852026/11/12 00:43:25 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."30862026/11/12 00:43:25 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."30872026/11/12 00:43:25 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."30882026/11/12 00:43:25 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."30892026/11/12 00:43:25 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."30902026/11/12 00:43:25 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."30912026/11/12 00:43:25 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."30922026/11/12 00:43:25 ERROR failed to remove object object=realisations/sha256:aaaaaaaa!out.doi error="Access Denied."30932026/11/12 00:43:25 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."30942026/11/12 00:43:25 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."30952026/11/12 00:43:25 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."30962026/11/12 00:43:25 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."30972026/11/12 00:43:25 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."30982026/11/12 00:43:25 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."30992026/11/12 00:43:25 INFO Completed upload id=11 objects=6 duration=2.77575ms31002026/11/12 00:43:25 INFO Received uploads request method=POST path=/31012026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=031022026/11/12 00:43:25 INFO Received uploads request method=POST path=/31032026/11/12 00:43:25 INFO Received push request method=POST path=/31042026/11/12 00:43:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031052026/11/12 00:43:25 INFO Received complete push request method=POST path=/31062026/11/12 00:43:25 INFO Vacuumed table table=pending_closures31072026/11/12 00:43:25 INFO Completed push id=13 objects=23 duration=2.49676ms31082026/11/12 00:43:25 INFO Vacuumed table table=commit_receipts31092026/11/12 00:43:25 INFO Received push request method=POST path=/31102026/11/12 00:43:25 INFO Vacuumed table table=pending_objects31112026/11/12 00:43:25 INFO Vacuumed table table=multipart_uploads31122026/11/12 00:43:25 INFO Received uploads request method=POST path=/31132026/11/12 00:43:25 INFO Vacuumed table table=closures31142026/11/12 00:43:25 INFO Received complete push request method=POST path=/31152026/11/12 00:43:25 INFO Vacuumed table table=objects31162026/11/12 00:43:25 INFO Received push request method=POST path=/31172026/11/12 00:43:25 INFO Completed push id=8 objects=6 duration=1.99699ms31182026/11/12 00:43:25 INFO Received uploads request method=POST path=/31192026/11/12 00:43:25 INFO Received uploads request method=POST path=/31202026/11/12 00:43:25 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."31212026/11/12 00:43:25 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.narinfo error="Access Denied."31222026/11/12 00:43:25 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."31232026/11/12 00:43:25 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."31242026/11/12 00:43:25 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."31252026/11/12 00:43:25 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."31262026/11/12 00:43:25 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."31272026/11/12 00:43:25 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."31282026/11/12 00:43:25 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."31292026/11/12 00:43:25 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."31302026/11/12 00:43:25 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."31312026/11/12 00:43:25 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."31322026/11/12 00:43:25 ERROR failed to remove object object=nar/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.nar.zst error="Access Denied."31332026/11/12 00:43:25 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."31342026/11/12 00:43:25 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."31352026/11/12 00:43:25 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."31362026/11/12 00:43:25 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."31372026/11/12 00:43:25 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."31382026/11/12 00:43:25 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."31392026/11/12 00:43:25 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."31402026/11/12 00:43:25 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."31412026/11/12 00:43:25 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."31422026/11/12 00:43:25 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."31432026/11/12 00:43:25 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."31442026/11/12 00:43:25 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."31452026/11/12 00:43:25 INFO Received push request method=POST path=/31462026/11/12 00:43:25 INFO Received push request method=POST path=/31472026/11/12 00:43:25 INFO Received complete push request method=POST path=/31482026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=031492026/11/12 00:43:25 INFO Completed push id=17 objects=6 duration=2.61705ms31502026/11/12 00:43:25 INFO Received push request method=POST path=/31512026/11/12 00:43:25 INFO Received complete upload request method=POST path=/31522026/11/12 00:43:25 INFO Completed upload id=17 objects=7 duration=1.31259ms31532026/11/12 00:43:25 INFO Received uploads request method=POST path=/31542026/11/12 00:43:25 INFO Received uploads request method=POST path=/31552026/11/12 00:43:25 INFO Received push request method=POST path=/31562026/11/12 00:43:25 INFO Received push request method=POST path=/31572026/11/12 00:43:25 INFO Received uploads request method=POST path=/31582026/11/12 00:43:25 INFO Received push request method=POST path=/31592026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=031602026/11/12 00:43:25 INFO Received complete push request method=POST path=/31612026/11/12 00:43:25 INFO Completed push id=5 objects=8 duration=2.88155ms31622026/11/12 00:43:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031632026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=031642026/11/12 00:43:25 WARN Force mode enabled - objects will be deleted immediately without grace period31652026/11/12 00:43:25 INFO Received uploads request method=POST path=/31662026/11/12 00:43:25 INFO Vacuumed table table=pending_closures31672026/11/12 00:43:25 INFO Vacuumed table table=commit_receipts31682026/11/12 00:43:25 INFO Vacuumed table table=pending_objects31692026/11/12 00:43:25 INFO Vacuumed table table=multipart_uploads31702026/11/12 00:43:25 INFO Vacuumed table table=closures31712026/11/12 00:43:25 INFO Vacuumed table table=objects31722026/11/12 00:43:25 INFO Received push request method=POST path=/31732026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=031742026/11/12 00:43:25 INFO Received push request method=POST path=/31752026/11/12 00:43:25 INFO Received push request method=POST path=/31762026/11/12 00:43:25 INFO Received complete push request method=POST path=/31772026/11/12 00:43:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=031782026/11/12 00:43:25 INFO Completed push id=8 objects=7 duration=2.85884ms31792026/11/12 00:43:25 INFO Vacuumed table table=pending_closures31802026/11/12 00:43:25 INFO Received complete push request method=POST path=/31812026/11/12 00:43:25 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."31822026/11/12 00:43:25 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."31832026/11/12 00:43:25 INFO Received push request method=POST path=/31842026/11/12 00:43:25 INFO Vacuumed table table=commit_receipts31852026/11/12 00:43:25 INFO Completed push id=4 objects=7 duration=2.12673ms31862026/11/12 00:43:25 INFO Vacuumed table table=pending_objects31872026/11/12 00:43:25 INFO Vacuumed table table=multipart_uploads31882026/11/12 00:43:25 INFO Received push request method=POST path=/31892026/11/12 00:43:25 INFO Vacuumed table table=closures31902026/11/12 00:43:25 INFO Vacuumed table table=objects31912026/11/12 00:43:25 INFO Received complete push request method=POST path=/31922026/11/12 00:43:25 INFO Received push request method=POST path=/31932026/11/12 00:43:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=16 objects-failed-to-delete=031942026/11/12 00:43:25 INFO Completed push id=4 objects=4 duration=3.88654ms31952026/11/12 00:43:25 INFO Received uploads request method=POST path=/31962026/11/12 00:43:25 INFO Vacuumed table table=pending_closures31972026/11/12 00:43:25 INFO Received uploads request method=POST path=/31982026/11/12 00:43:25 INFO Vacuumed table table=commit_receipts31992026/11/12 00:43:25 INFO Vacuumed table table=pending_objects32002026/11/12 00:43:25 INFO Vacuumed table table=multipart_uploads32012026/11/12 00:43:25 INFO Vacuumed table table=closures32022026/11/12 00:43:25 INFO Vacuumed table table=objects32032026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=032042026/11/12 00:43:25 INFO Received push request method=POST path=/32052026/11/12 00:43:25 INFO Received uploads request method=POST path=/32062026/11/12 00:43:25 INFO Received uploads request method=POST path=/32072026/11/12 00:43:25 INFO Received uploads request method=POST path=/32082026/11/12 00:43:25 INFO Received complete push request method=POST path=/32092026/11/12 00:43:25 INFO Completed push id=9 objects=5 duration=2.5127ms32102026/11/12 00:43:25 INFO Received uploads request method=POST path=/32112026/11/12 00:43:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=032122026/11/12 00:43:25 INFO Received push request method=POST path=/32132026/11/12 00:43:25 INFO Vacuumed table table=pending_closures32142026/11/12 00:43:25 INFO Vacuumed table table=commit_receipts32152026/11/12 00:43:25 INFO Vacuumed table table=pending_objects32162026/11/12 00:43:25 INFO Vacuumed table table=multipart_uploads32172026/11/12 00:43:25 INFO Vacuumed table table=closures32182026/11/12 00:43:25 INFO Vacuumed table table=objects32192026/11/12 00:43:25 INFO Received push request method=POST path=/32202026/11/12 00:43:25 INFO Received uploads request method=POST path=/32212026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=032222026/11/12 00:43:25 WARN Force mode enabled - objects will be deleted immediately without grace period32232026/11/12 00:43:25 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032242026/11/12 00:43:25 INFO Received uploads request method=POST path=/32252026/11/12 00:43:25 INFO Received push request method=POST path=/32262026/11/12 00:43:25 INFO Vacuumed table table=pending_closures32272026/11/12 00:43:25 INFO Received complete push request method=POST path=/32282026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=032292026/11/12 00:43:25 INFO Received push request method=POST path=/32302026/11/12 00:43:25 INFO Vacuumed table table=commit_receipts32312026/11/12 00:43:25 INFO Completed push id=6 objects=6 duration=2.44891ms32322026/11/12 00:43:25 INFO Received complete upload request method=POST path=/32332026/11/12 00:43:25 INFO Vacuumed table table=pending_objects32342026/11/12 00:43:25 INFO Received uploads request method=POST path=/32352026/11/12 00:43:25 INFO Vacuumed table table=multipart_uploads32362026/11/12 00:43:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032372026/11/12 00:43:25 INFO Completed upload id=11 objects=14 duration=1.51737ms32382026/11/12 00:43:25 INFO Vacuumed table table=closures32392026/11/12 00:43:25 INFO Received complete upload request method=POST path=/32402026/11/12 00:43:25 INFO Vacuumed table table=pending_closures32412026/11/12 00:43:25 INFO Completed upload id=14 objects=6 duration=1.25121ms32422026/11/12 00:43:25 INFO Vacuumed table table=objects32432026/11/12 00:43:25 INFO Vacuumed table table=commit_receipts32442026/11/12 00:43:25 INFO Vacuumed table table=pending_objects32452026/11/12 00:43:25 INFO Received complete upload request method=POST path=/32462026/11/12 00:43:25 INFO Vacuumed table table=multipart_uploads32472026/11/12 00:43:25 INFO Vacuumed table table=closures32482026/11/12 00:43:25 INFO Completed upload id=9 objects=16 duration=1.648399ms32492026/11/12 00:43:25 INFO Vacuumed table table=objects32502026/11/12 00:43:25 INFO Received push request method=POST path=/32512026/11/12 00:43:25 INFO Received push request method=POST path=/32522026/11/12 00:43:25 INFO Received push request method=POST path=/32532026/11/12 00:43:25 INFO Received complete upload request method=POST path=/32542026/11/12 00:43:25 INFO Completed upload id=15 objects=6 duration=1.47305ms32552026/11/12 00:43:25 INFO Received uploads request method=POST path=/32562026/11/12 00:43:25 INFO Received uploads request method=POST path=/32572026/11/12 00:43:25 INFO Received push request method=POST path=/32582026/11/12 00:43:25 INFO Received push request method=POST path=/32592026/11/12 00:43:25 INFO Received push request method=POST path=/32602026/11/12 00:43:25 INFO Received push request method=POST path=/32612026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=032622026/11/12 00:43:25 INFO Received push request method=POST path=/32632026/11/12 00:43:25 WARN Force mode enabled - objects will be deleted immediately without grace period32642026/11/12 00:43:25 INFO Received complete push request method=POST path=/32652026/11/12 00:43:25 INFO Received push request method=POST path=/32662026/11/12 00:43:25 INFO Completed push id=5 objects=9 duration=5.208ms32672026/11/12 00:43:25 INFO Received uploads request method=POST path=/32682026/11/12 00:43:25 INFO Received push request method=POST path=/32692026/11/12 00:43:25 INFO Received push request method=POST path=/32702026/11/12 00:43:25 INFO Received complete upload request method=POST path=/32712026/11/12 00:43:25 INFO Received push request method=POST path=/32722026/11/12 00:43:25 INFO Completed upload id=10 objects=17 duration=1.826019ms32732026/11/12 00:43:25 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."32742026/11/12 00:43:25 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."32752026/11/12 00:43:25 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."32762026/11/12 00:43:25 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."32772026/11/12 00:43:25 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."32782026/11/12 00:43:25 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."32792026/11/12 00:43:25 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."32802026/11/12 00:43:25 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."32812026/11/12 00:43:25 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."32822026/11/12 00:43:25 INFO Received uploads request method=POST path=/32832026/11/12 00:43:25 INFO Received uploads request method=POST path=/32842026/11/12 00:43:25 INFO Received push request method=POST path=/32852026/11/12 00:43:25 INFO Received push request method=POST path=/32862026/11/12 00:43:25 INFO Received complete upload request method=POST path=/32872026/11/12 00:43:25 INFO Received push request method=POST path=/32882026/11/12 00:43:25 INFO Received complete push request method=POST path=/32892026/11/12 00:43:25 INFO Completed upload id=7 objects=3 duration=1.57271ms32902026/11/12 00:43:25 INFO Received uploads request method=POST path=/32912026/11/12 00:43:25 INFO Completed push id=7 objects=7 duration=2.30633ms32922026/11/12 00:43:25 INFO Received push request method=POST path=/32932026/11/12 00:43:25 INFO Received push request method=POST path=/32942026/11/12 00:43:25 INFO Received push request method=POST path=/32952026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=032962026/11/12 00:43:25 INFO Received complete push request method=POST path=/32972026/11/12 00:43:25 WARN Force mode enabled - objects will be deleted immediately without grace period32982026/11/12 00:43:25 INFO Completed push id=7 objects=27 duration=2.57178ms32992026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=033002026/11/12 00:43:25 INFO Received uploads request method=POST path=/33012026/11/12 00:43:25 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033022026/11/12 00:43:25 INFO Vacuumed table table=pending_closures33032026/11/12 00:43:25 INFO Received push request method=POST path=/33042026/11/12 00:43:25 INFO Vacuumed table table=commit_receipts33052026/11/12 00:43:25 INFO Vacuumed table table=pending_objects33062026/11/12 00:43:25 INFO Vacuumed table table=multipart_uploads33072026/11/12 00:43:25 INFO Vacuumed table table=closures33082026/11/12 00:43:25 INFO Received uploads request method=POST path=/33092026/11/12 00:43:25 INFO Received uploads request method=POST path=/33102026/11/12 00:43:25 INFO Received uploads request method=POST path=/33112026/11/12 00:43:25 INFO Received complete push request method=POST path=/33122026/11/12 00:43:25 INFO Vacuumed table table=objects33132026/11/12 00:43:25 INFO Completed push id=10 objects=3 duration=2.00698ms33142026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=033152026/11/12 00:43:25 WARN Force mode enabled - objects will be deleted immediately without grace period33162026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=033172026/11/12 00:43:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=9 objects-failed-to-delete=033182026/11/12 00:43:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033192026/11/12 00:43:25 INFO Vacuumed table table=pending_closures33202026/11/12 00:43:25 INFO Vacuumed table table=commit_receipts33212026/11/12 00:43:25 WARN Force mode enabled - objects will be deleted immediately without grace period33222026/11/12 00:43:25 INFO Vacuumed table table=pending_closures33232026/11/12 00:43:25 INFO Vacuumed table table=commit_receipts33242026/11/12 00:43:25 INFO Vacuumed table table=pending_objects33252026/11/12 00:43:25 INFO Vacuumed table table=multipart_uploads33262026/11/12 00:43:25 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033272026/11/12 00:43:25 INFO Vacuumed table table=pending_objects33282026/11/12 00:43:25 INFO Vacuumed table table=closures33292026/11/12 00:43:25 INFO Vacuumed table table=multipart_uploads33302026/11/12 00:43:25 INFO Vacuumed table table=pending_closures33312026/11/12 00:43:25 INFO Received complete push request method=POST path=/33322026/11/12 00:43:25 INFO Received push request method=POST path=/33332026/11/12 00:43:25 INFO Vacuumed table table=closures33342026/11/12 00:43:25 INFO Vacuumed table table=commit_receipts33352026/11/12 00:43:25 INFO Vacuumed table table=objects33362026/11/12 00:43:25 INFO Vacuumed table table=objects33372026/11/12 00:43:25 INFO Completed push id=18 objects=23 duration=2.33899ms33382026/11/12 00:43:25 INFO Vacuumed table table=pending_objects33392026/11/12 00:43:25 INFO Received uploads request method=POST path=/33402026/11/12 00:43:25 INFO Vacuumed table table=multipart_uploads33412026/11/12 00:43:25 INFO Received push request method=POST path=/33422026/11/12 00:43:25 INFO Vacuumed table table=closures33432026/11/12 00:43:25 INFO Received push request method=POST path=/33442026/11/12 00:43:25 INFO Vacuumed table table=objects33452026/11/12 00:43:25 INFO Received complete upload request method=POST path=/33462026/11/12 00:43:25 INFO Received uploads request method=POST path=/33472026/11/12 00:43:25 INFO Completed upload id=15 objects=4 duration=2.83751ms33482026/11/12 00:43:25 INFO Received push request method=POST path=/33492026/11/12 00:43:25 INFO Received complete push request method=POST path=/33502026/11/12 00:43:25 INFO Received complete upload request method=POST path=/33512026/11/12 00:43:25 INFO Completed upload id=3 objects=9 duration=1.95882ms33522026/11/12 00:43:25 INFO Completed push id=16 objects=27 duration=2.74576ms33532026/11/12 00:43:25 INFO Received push request method=POST path=/33542026/11/12 00:43:25 INFO Received push request method=POST path=/33552026/11/12 00:43:25 INFO Received uploads request method=POST path=/33562026/11/12 00:43:25 INFO Received complete upload request method=POST path=/33572026/11/12 00:43:25 INFO Received uploads request method=POST path=/33582026/11/12 00:43:25 INFO Completed upload id=8 objects=3 duration=2.11186ms33592026/11/12 00:43:25 INFO Received uploads request method=POST path=/33602026/11/12 00:43:25 INFO Received complete upload request method=POST path=/33612026/11/12 00:43:25 INFO Received complete upload request method=POST path=/33622026/11/12 00:43:25 INFO Completed upload id=18 objects=11 duration=2.18896ms33632026/11/12 00:43:25 INFO Received push request method=POST path=/33642026/11/12 00:43:25 INFO Completed upload id=10 objects=6 duration=3.09376ms33652026/11/12 00:43:25 INFO Received push request method=POST path=/33662026/11/12 00:43:25 INFO Received uploads request method=POST path=/33672026/11/12 00:43:25 INFO Received uploads request method=POST path=/33682026/11/12 00:43:25 INFO Received uploads request method=POST path=/33692026/11/12 00:43:25 INFO Received uploads request method=POST path=/33702026/11/12 00:43:25 INFO Received push request method=POST path=/33712026/11/12 00:43:25 INFO Received uploads request method=POST path=/33722026/11/12 00:43:25 INFO Received complete push request method=POST path=/33732026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=033742026/11/12 00:43:25 INFO Completed push id=14 objects=6 duration=1.93966ms33752026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=033762026/11/12 00:43:25 INFO Received uploads request method=POST path=/33772026/11/12 00:43:25 WARN Force mode enabled - objects will be deleted immediately without grace period33782026/11/12 00:43:25 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033792026/11/12 00:43:25 WARN Force mode enabled - objects will be deleted immediately without grace period33802026/11/12 00:43:25 INFO Vacuumed table table=pending_closures33812026/11/12 00:43:25 INFO Vacuumed table table=commit_receipts33822026/11/12 00:43:25 INFO Received uploads request method=POST path=/33832026/11/12 00:43:25 INFO Received push request method=POST path=/33842026/11/12 00:43:25 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033852026/11/12 00:43:25 INFO Vacuumed table table=pending_objects33862026/11/12 00:43:25 INFO Vacuumed table table=pending_closures33872026/11/12 00:43:25 INFO Vacuumed table table=multipart_uploads33882026/11/12 00:43:25 INFO Vacuumed table table=closures33892026/11/12 00:43:25 INFO Vacuumed table table=commit_receipts33902026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=033912026/11/12 00:43:25 INFO Received complete push request method=POST path=/33922026/11/12 00:43:25 INFO Vacuumed table table=objects33932026/11/12 00:43:25 INFO Vacuumed table table=pending_objects33942026/11/12 00:43:25 INFO Vacuumed table table=multipart_uploads33952026/11/12 00:43:25 INFO Completed push id=11 objects=9 duration=1.70223ms33962026/11/12 00:43:25 INFO Vacuumed table table=closures33972026/11/12 00:43:25 WARN Force mode enabled - objects will be deleted immediately without grace period33982026/11/12 00:43:25 INFO Vacuumed table table=objects33992026/11/12 00:43:25 INFO Received push request method=POST path=/34002026/11/12 00:43:25 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034012026/11/12 00:43:25 INFO Received complete push request method=POST path=/34022026/11/12 00:43:25 INFO Vacuumed table table=pending_closures34032026/11/12 00:43:25 INFO Received complete push request method=POST path=/34042026/11/12 00:43:25 INFO Vacuumed table table=commit_receipts34052026/11/12 00:43:25 INFO Received push request method=POST path=/34062026/11/12 00:43:25 INFO Completed push id=5 objects=6 duration=2.13734ms34072026/11/12 00:43:25 INFO Vacuumed table table=pending_objects34082026/11/12 00:43:25 INFO Completed push id=14 objects=3 duration=4.03495ms34092026/11/12 00:43:25 INFO Vacuumed table table=multipart_uploads34102026/11/12 00:43:25 INFO Vacuumed table table=closures34112026/11/12 00:43:25 INFO Received push request method=POST path=/34122026/11/12 00:43:25 INFO Vacuumed table table=objects34132026/11/12 00:43:25 INFO Received push request method=POST path=/34142026/11/12 00:43:25 INFO Received uploads request method=POST path=/34152026/11/12 00:43:25 INFO Received complete upload request method=POST path=/34162026/11/12 00:43:25 INFO Completed upload id=16 objects=3 duration=1.22866ms34172026/11/12 00:43:25 INFO Received push request method=POST path=/34182026/11/12 00:43:25 INFO Received push request method=POST path=/34192026/11/12 00:43:25 INFO Received push request method=POST path=/34202026/11/12 00:43:25 INFO Received push request method=POST path=/34212026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=034222026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=034232026/11/12 00:43:25 WARN Force mode enabled - objects will be deleted immediately without grace period34242026/11/12 00:43:25 WARN Force mode enabled - objects will be deleted immediately without grace period34252026/11/12 00:43:25 INFO Received push request method=POST path=/34262026/11/12 00:43:25 INFO Received uploads request method=POST path=/34272026/11/12 00:43:25 INFO Received push request method=POST path=/34282026/11/12 00:43:25 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034292026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=034302026/11/12 00:43:25 INFO Vacuumed table table=pending_closures34312026/11/12 00:43:25 INFO Received complete push request method=POST path=/34322026/11/12 00:43:25 INFO Vacuumed table table=commit_receipts34332026/11/12 00:43:25 INFO Vacuumed table table=pending_objects34342026/11/12 00:43:25 INFO Vacuumed table table=multipart_uploads34352026/11/12 00:43:25 INFO Completed push id=18 objects=10 duration=2.52206ms34362026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=034372026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=034382026/11/12 00:43:25 INFO Vacuumed table table=closures34392026/11/12 00:43:25 INFO Vacuumed table table=objects34402026/11/12 00:43:25 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=034412026/11/12 00:43:25 INFO Received complete push request method=POST path=/34422026/11/12 00:43:25 INFO Vacuumed table table=pending_closures34432026/11/12 00:43:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034442026/11/12 00:43:25 INFO Vacuumed table table=commit_receipts34452026/11/12 00:43:25 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034462026/11/12 00:43:25 INFO Vacuumed table table=pending_objects34472026/11/12 00:43:25 INFO Received complete push request method=POST path=/34482026/11/12 00:43:25 INFO Vacuumed table table=multipart_uploads34492026/11/12 00:43:25 INFO Received push request method=POST path=/34502026/11/12 00:43:25 INFO Received complete push request method=POST path=/34512026/11/12 00:43:25 INFO Vacuumed table table=pending_closures34522026/11/12 00:43:25 INFO Vacuumed table table=closures34532026/11/12 00:43:25 INFO Vacuumed table table=pending_closures34542026/11/12 00:43:25 INFO Vacuumed table table=commit_receipts34552026/11/12 00:43:25 INFO Vacuumed table table=objects34562026/11/12 00:43:25 INFO Completed push id=6 objects=6 duration=1.96185ms34572026/11/12 00:43:25 INFO Completed push id=15 objects=24 duration=4.67288ms34582026/11/12 00:43:25 INFO Vacuumed table table=commit_receipts34592026/11/12 00:43:25 INFO Completed push id=5 objects=6 duration=1.66214ms34602026/11/12 00:43:25 INFO Vacuumed table table=pending_objects34612026/11/12 00:43:25 INFO Vacuumed table table=multipart_uploads34622026/11/12 00:43:25 INFO Vacuumed table table=pending_objects34632026/11/12 00:43:25 INFO Vacuumed table table=multipart_uploads34642026/11/12 00:43:25 INFO Vacuumed table table=closures34652026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=034662026/11/12 00:43:25 INFO Vacuumed table table=closures34672026/11/12 00:43:25 INFO Vacuumed table table=objects34682026/11/12 00:43:25 INFO Received complete push request method=POST path=/34692026/11/12 00:43:25 INFO Vacuumed table table=objects34702026/11/12 00:43:25 WARN Force mode enabled - objects will be deleted immediately without grace period34712026/11/12 00:43:25 INFO Completed push id=8 objects=6 duration=2.07125ms34722026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=034732026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=034742026/11/12 00:43:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=1 objects-deleted-after-grace-period=0 objects-failed-to-delete=034752026/11/12 00:43:25 WARN Force mode enabled - objects will be deleted immediately without grace period34762026/11/12 00:43:25 INFO Vacuumed table table=pending_closures34772026/11/12 00:43:25 INFO Vacuumed table table=commit_receipts34782026/11/12 00:43:25 INFO Received push request method=POST path=/34792026/11/12 00:43:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034802026/11/12 00:43:25 INFO Vacuumed table table=pending_objects34812026/11/12 00:43:25 INFO Vacuumed table table=multipart_uploads34822026/11/12 00:43:25 INFO Vacuumed table table=pending_closures34832026/11/12 00:43:25 INFO Vacuumed table table=closures34842026/11/12 00:43:25 INFO Vacuumed table table=commit_receipts34852026/11/12 00:43:25 INFO Received push request method=POST path=/34862026/11/12 00:43:25 INFO Vacuumed table table=objects34872026/11/12 00:43:25 INFO Vacuumed table table=pending_objects34882026/11/12 00:43:25 INFO Vacuumed table table=multipart_uploads34892026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=034902026/11/12 00:43:25 INFO Vacuumed table table=closures34912026/11/12 00:43:25 INFO Received uploads request method=POST path=/34922026/11/12 00:43:25 INFO Vacuumed table table=objects34932026/11/12 00:43:25 WARN Force mode enabled - objects will be deleted immediately without grace period34942026/11/12 00:43:25 INFO Received uploads request method=POST path=/34952026/11/12 00:43:25 INFO Received push request method=POST path=/34962026/11/12 00:43:25 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034972026/11/12 00:43:25 INFO Vacuumed table table=pending_closures34982026/11/12 00:43:25 INFO Received uploads request method=POST path=/34992026/11/12 00:43:25 INFO Vacuumed table table=commit_receipts35002026/11/12 00:43:25 INFO Vacuumed table table=pending_objects35012026/11/12 00:43:25 INFO Received uploads request method=POST path=/35022026/11/12 00:43:25 INFO Vacuumed table table=multipart_uploads35032026/11/12 00:43:25 INFO Vacuumed table table=closures35042026/11/12 00:43:25 INFO Received push request method=POST path=/35052026/11/12 00:43:25 INFO Vacuumed table table=objects35062026/11/12 00:43:25 INFO Received complete push request method=POST path=/35072026/11/12 00:43:25 INFO Completed push id=12 objects=7 duration=1.96225ms35082026/11/12 00:43:25 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."35092026/11/12 00:43:25 INFO Received uploads request method=POST path=/35102026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=035112026/11/12 00:43:25 INFO Received uploads request method=POST path=/35122026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=035132026/11/12 00:43:25 INFO Received complete push request method=POST path=/35142026/11/12 00:43:25 WARN Force mode enabled - objects will be deleted immediately without grace period35152026/11/12 00:43:25 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035162026/11/12 00:43:25 INFO Vacuumed table table=pending_closures35172026/11/12 00:43:25 INFO Received complete push request method=POST path=/35182026/11/12 00:43:25 INFO Vacuumed table table=commit_receipts35192026/11/12 00:43:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035202026/11/12 00:43:25 INFO Vacuumed table table=pending_objects35212026/11/12 00:43:25 INFO Vacuumed table table=pending_closures35222026/11/12 00:43:25 INFO Completed push id=3 objects=21 duration=5.34765ms35232026/11/12 00:43:25 INFO Vacuumed table table=multipart_uploads35242026/11/12 00:43:25 INFO Vacuumed table table=commit_receipts35252026/11/12 00:43:25 INFO Vacuumed table table=closures35262026/11/12 00:43:25 INFO Vacuumed table table=objects35272026/11/12 00:43:25 INFO Vacuumed table table=pending_objects35282026/11/12 00:43:25 INFO Completed push id=13 objects=12 duration=4.69128ms35292026/11/12 00:43:25 INFO Vacuumed table table=multipart_uploads35302026/11/12 00:43:25 INFO Vacuumed table table=closures35312026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=035322026/11/12 00:43:25 INFO Vacuumed table table=objects35332026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=035342026/11/12 00:43:25 INFO Received uploads request method=POST path=/35352026/11/12 00:43:25 WARN Force mode enabled - objects will be deleted immediately without grace period35362026/11/12 00:43:25 INFO Received push request method=POST path=/35372026/11/12 00:43:25 INFO Received uploads request method=POST path=/35382026/11/12 00:43:25 INFO Garbage collection completed failed-uploads-deleted=5 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035392026/11/12 00:43:25 INFO Received complete push request method=POST path=/35402026/11/12 00:43:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=8 objects-deleted-after-grace-period=8 objects-failed-to-delete=035412026/11/12 00:43:25 INFO Completed push id=10 objects=15 duration=1.62068ms35422026/11/12 00:43:25 INFO Vacuumed table table=pending_closures35432026/11/12 00:43:25 INFO Vacuumed table table=pending_closures35442026/11/12 00:43:25 INFO Vacuumed table table=commit_receipts35452026/11/12 00:43:25 INFO Vacuumed table table=commit_receipts35462026/11/12 00:43:25 INFO Received complete upload request method=POST path=/35472026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=035482026/11/12 00:43:25 INFO Vacuumed table table=pending_objects35492026/11/12 00:43:25 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=035502026/11/12 00:43:25 INFO Vacuumed table table=pending_objects35512026/11/12 00:43:25 INFO Received push request method=POST path=/35522026/11/12 00:43:25 INFO Vacuumed table table=multipart_uploads35532026/11/12 00:43:25 INFO Vacuumed table table=multipart_uploads35542026/11/12 00:43:25 INFO Received uploads request method=POST path=/35552026/11/12 00:43:25 INFO Received push request method=POST path=/35562026/11/12 00:43:25 INFO Vacuumed table table=closures35572026/11/12 00:43:25 INFO Completed upload id=18 objects=4 duration=3.541989ms35582026/11/12 00:43:25 INFO Vacuumed table table=closures35592026/11/12 00:43:25 INFO Vacuumed table table=pending_closures35602026/11/12 00:43:25 INFO Vacuumed table table=objects35612026/11/12 00:43:25 INFO Received uploads request method=POST path=/35622026/11/12 00:43:25 INFO Vacuumed table table=commit_receipts35632026/11/12 00:43:25 INFO Vacuumed table table=objects35642026/11/12 00:43:25 INFO Vacuumed table table=pending_objects35652026/11/12 00:43:25 INFO Received complete push request method=POST path=/35662026/11/12 00:43:25 INFO Vacuumed table table=multipart_uploads35672026/11/12 00:43:25 INFO Vacuumed table table=closures35682026/11/12 00:43:25 INFO Vacuumed table table=objects35692026/11/12 00:43:25 INFO Received uploads request method=POST path=/35702026/11/12 00:43:25 INFO Completed push id=11 objects=11 duration=3.96927ms35712026/11/12 00:43:25 INFO Received complete push request method=POST path=/35722026/11/12 00:43:25 INFO Received uploads request method=POST path=/35732026/11/12 00:43:25 INFO Completed push id=15 objects=4 duration=1.43716ms35742026/11/12 00:43:25 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=035752026/11/12 00:43:25 INFO Vacuumed table table=pending_closures35762026/11/12 00:43:25 INFO Vacuumed table table=commit_receipts35772026/11/12 00:43:25 INFO Vacuumed table table=pending_objects35782026/11/12 00:43:25 INFO Vacuumed table table=multipart_uploads35792026/11/12 00:43:25 INFO Vacuumed table table=closures35802026/11/12 00:43:25 INFO Vacuumed table table=objects35812026/11/12 00:43:25 INFO Received uploads request method=POST path=/35822026/11/12 00:43:25 INFO Received push request method=POST path=/35832026/11/12 00:43:25 INFO Received push request method=POST path=/35842026/11/12 00:43:25 INFO Received uploads request method=POST path=/35852026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=035862026/11/12 00:43:25 WARN Force mode enabled - objects will be deleted immediately without grace period35872026/11/12 00:43:25 INFO Received complete upload request method=POST path=/35882026/11/12 00:43:25 INFO Completed upload id=6 objects=6 duration=1.51718ms35892026/11/12 00:43:25 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035902026/11/12 00:43:25 INFO Received complete push request method=POST path=/35912026/11/12 00:43:25 INFO Vacuumed table table=pending_closures35922026/11/12 00:43:25 INFO Vacuumed table table=commit_receipts35932026/11/12 00:43:25 INFO Completed push id=17 objects=6 duration=2.1561ms35942026/11/12 00:43:25 INFO Vacuumed table table=pending_objects35952026/11/12 00:43:25 INFO Vacuumed table table=multipart_uploads35962026/11/12 00:43:25 INFO Received uploads request method=POST path=/35972026/11/12 00:43:25 INFO Vacuumed table table=closures35982026/11/12 00:43:25 INFO Vacuumed table table=objects35992026/11/12 00:43:25 INFO Received uploads request method=POST path=/36002026/11/12 00:43:25 INFO Received uploads request method=POST path=/36012026/11/12 00:43:25 INFO Received push request method=POST path=/36022026/11/12 00:43:25 INFO Received push request method=POST path=/36032026/11/12 00:43:25 INFO Received complete push request method=POST path=/36042026/11/12 00:43:25 INFO Completed push id=12 objects=2 duration=3.51264ms36052026/11/12 00:43:25 INFO Received complete push request method=POST path=/36062026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=036072026/11/12 00:43:25 INFO Completed push id=4 objects=6 duration=2.56669ms36082026/11/12 00:43:25 WARN Force mode enabled - objects will be deleted immediately without grace period36092026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=036102026/11/12 00:43:25 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036112026/11/12 00:43:25 INFO Vacuumed table table=pending_closures36122026/11/12 00:43:25 INFO Vacuumed table table=commit_receipts36132026/11/12 00:43:25 INFO Vacuumed table table=pending_objects36142026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=036152026/11/12 00:43:25 INFO Vacuumed table table=multipart_uploads36162026/11/12 00:43:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036172026/11/12 00:43:25 INFO Vacuumed table table=closures36182026/11/12 00:43:25 INFO Vacuumed table table=pending_closures36192026/11/12 00:43:25 INFO Vacuumed table table=objects36202026/11/12 00:43:25 INFO Vacuumed table table=commit_receipts36212026/11/12 00:43:25 INFO Received uploads request method=POST path=/36222026/11/12 00:43:25 INFO Vacuumed table table=pending_objects36232026/11/12 00:43:25 INFO Vacuumed table table=multipart_uploads36242026/11/12 00:43:25 INFO Vacuumed table table=closures36252026/11/12 00:43:25 INFO Received push request method=POST path=/36262026/11/12 00:43:25 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036272026/11/12 00:43:25 INFO Vacuumed table table=objects36282026/11/12 00:43:25 INFO Received uploads request method=POST path=/36292026/11/12 00:43:25 INFO Vacuumed table table=pending_closures36302026/11/12 00:43:25 INFO Received complete upload request method=POST path=/36312026/11/12 00:43:25 INFO Vacuumed table table=commit_receipts36322026/11/12 00:43:25 INFO Vacuumed table table=pending_objects36332026/11/12 00:43:25 INFO Vacuumed table table=multipart_uploads36342026/11/12 00:43:25 INFO Completed upload id=23 objects=15 duration=2.66902ms36352026/11/12 00:43:25 INFO Vacuumed table table=closures36362026/11/12 00:43:25 INFO Vacuumed table table=objects36372026/11/12 00:43:25 INFO Received uploads request method=POST path=/36382026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=036392026/11/12 00:43:25 INFO Received push request method=POST path=/36402026/11/12 00:43:25 INFO Received push request method=POST path=/36412026/11/12 00:43:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036422026/11/12 00:43:25 INFO Vacuumed table table=pending_closures36432026/11/12 00:43:25 INFO Vacuumed table table=commit_receipts36442026/11/12 00:43:25 INFO Received push request method=POST path=/36452026/11/12 00:43:25 INFO Vacuumed table table=pending_objects36462026/11/12 00:43:25 INFO Received complete push request method=POST path=/36472026/11/12 00:43:25 INFO Vacuumed table table=multipart_uploads36482026/11/12 00:43:25 INFO Vacuumed table table=closures36492026/11/12 00:43:25 INFO Received uploads request method=POST path=/36502026/11/12 00:43:25 INFO Completed push id=14 objects=3 duration=1.64398ms36512026/11/12 00:43:25 INFO Vacuumed table table=objects36522026/11/12 00:43:25 INFO Received complete push request method=POST path=/36532026/11/12 00:43:25 INFO Completed push id=6 objects=26 duration=3.30468ms36542026/11/12 00:43:25 INFO Received complete upload request method=POST path=/36552026/11/12 00:43:25 INFO Received push request method=POST path=/36562026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=036572026/11/12 00:43:25 INFO Completed upload id=17 objects=6 duration=4.03677ms36582026/11/12 00:43:25 WARN Force mode enabled - objects will be deleted immediately without grace period36592026/11/12 00:43:25 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=036602026/11/12 00:43:25 INFO Received complete push request method=POST path=/36612026/11/12 00:43:25 INFO Received complete push request method=POST path=/36622026/11/12 00:43:25 INFO Vacuumed table table=pending_closures36632026/11/12 00:43:25 INFO Vacuumed table table=commit_receipts36642026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=036652026/11/12 00:43:25 INFO Received push request method=POST path=/36662026/11/12 00:43:25 INFO Vacuumed table table=pending_objects36672026/11/12 00:43:25 INFO Completed push id=19 objects=6 duration=3.92433ms36682026/11/12 00:43:25 INFO Vacuumed table table=multipart_uploads36692026/11/12 00:43:25 INFO Received uploads request method=POST path=/36702026/11/12 00:43:25 INFO Received uploads request method=POST path=/36712026/11/12 00:43:25 INFO Completed push id=9 objects=7 duration=4.46304ms36722026/11/12 00:43:25 INFO Received complete upload request method=POST path=/36732026/11/12 00:43:25 INFO Vacuumed table table=closures36742026/11/12 00:43:25 INFO Vacuumed table table=objects36752026/11/12 00:43:25 INFO Completed upload id=9 objects=2 duration=1.79543ms36762026/11/12 00:43:25 INFO Received complete push request method=POST path=/36772026/11/12 00:43:25 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036782026/11/12 00:43:25 INFO Vacuumed table table=pending_closures36792026/11/12 00:43:25 INFO Completed push id=15 objects=5 duration=2.43432ms36802026/11/12 00:43:25 INFO Vacuumed table table=commit_receipts36812026/11/12 00:43:25 INFO Vacuumed table table=pending_objects36822026/11/12 00:43:25 INFO Vacuumed table table=multipart_uploads36832026/11/12 00:43:25 INFO Vacuumed table table=closures36842026/11/12 00:43:25 INFO Received push request method=POST path=/36852026/11/12 00:43:25 INFO Vacuumed table table=objects36862026/11/12 00:43:25 INFO Received complete upload request method=POST path=/36872026/11/12 00:43:25 INFO Received uploads request method=POST path=/36882026/11/12 00:43:25 INFO Received complete push request method=POST path=/36892026/11/12 00:43:25 INFO Completed upload id=19 objects=6 duration=1.80134ms36902026/11/12 00:43:25 INFO Completed push id=9 objects=4 duration=3.53845ms36912026/11/12 00:43:25 INFO Received complete push request method=POST path=/36922026/11/12 00:43:25 INFO Received complete push request method=POST path=/36932026/11/12 00:43:25 INFO Received uploads request method=POST path=/36942026/11/12 00:43:25 INFO Completed push id=8 objects=3 duration=2.17121ms36952026/11/12 00:43:25 INFO Received push request method=POST path=/36962026/11/12 00:43:25 INFO Completed push id=8 objects=6 duration=3.98054ms36972026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=036982026/11/12 00:43:25 INFO Received uploads request method=POST path=/36992026/11/12 00:43:25 WARN Force mode enabled - objects will be deleted immediately without grace period37002026/11/12 00:43:25 INFO Received push request method=POST path=/37012026/11/12 00:43:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037022026/11/12 00:43:25 INFO Vacuumed table table=pending_closures37032026/11/12 00:43:25 INFO Vacuumed table table=commit_receipts37042026/11/12 00:43:25 INFO Vacuumed table table=pending_objects37052026/11/12 00:43:25 INFO Vacuumed table table=multipart_uploads37062026/11/12 00:43:25 INFO Vacuumed table table=closures37072026/11/12 00:43:25 INFO Vacuumed table table=objects37082026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=037092026/11/12 00:43:25 INFO Received complete upload request method=POST path=/37102026/11/12 00:43:25 INFO Received complete upload request method=POST path=/37112026/11/12 00:43:25 INFO Completed upload id=20 objects=4 duration=858.319µs37122026/11/12 00:43:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037132026/11/12 00:43:25 INFO Completed upload id=10 objects=7 duration=1.770829ms37142026/11/12 00:43:25 INFO Vacuumed table table=pending_closures37152026/11/12 00:43:25 INFO Vacuumed table table=commit_receipts37162026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=037172026/11/12 00:43:25 INFO Vacuumed table table=pending_objects37182026/11/12 00:43:25 INFO Vacuumed table table=multipart_uploads37192026/11/12 00:43:25 INFO Vacuumed table table=closures37202026/11/12 00:43:25 INFO Vacuumed table table=objects37212026/11/12 00:43:25 INFO Received push request method=POST path=/37222026/11/12 00:43:25 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037232026/11/12 00:43:25 INFO Received push request method=POST path=/37242026/11/12 00:43:25 INFO Vacuumed table table=pending_closures37252026/11/12 00:43:25 INFO Received complete upload request method=POST path=/37262026/11/12 00:43:25 INFO Vacuumed table table=commit_receipts37272026/11/12 00:43:25 INFO Completed upload id=22 objects=7 duration=1.198ms37282026/11/12 00:43:25 INFO Vacuumed table table=pending_objects37292026/11/12 00:43:25 INFO Vacuumed table table=multipart_uploads37302026/11/12 00:43:25 INFO Vacuumed table table=closures37312026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=037322026/11/12 00:43:25 INFO Vacuumed table table=objects37332026/11/12 00:43:25 WARN Force mode enabled - objects will be deleted immediately without grace period37342026/11/12 00:43:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037352026/11/12 00:43:25 INFO Vacuumed table table=pending_closures37362026/11/12 00:43:25 INFO Vacuumed table table=commit_receipts37372026/11/12 00:43:25 INFO Vacuumed table table=pending_objects37382026/11/12 00:43:25 INFO Vacuumed table table=multipart_uploads37392026/11/12 00:43:25 INFO Vacuumed table table=closures37402026/11/12 00:43:25 INFO Received uploads request method=POST path=/37412026/11/12 00:43:25 INFO Vacuumed table table=objects37422026/11/12 00:43:25 INFO Received uploads request method=POST path=/37432026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=037442026/11/12 00:43:25 INFO Received uploads request method=POST path=/37452026/11/12 00:43:25 WARN Force mode enabled - objects will be deleted immediately without grace period37462026/11/12 00:43:25 INFO Received uploads request method=POST path=/37472026/11/12 00:43:25 INFO Received uploads request method=POST path=/37482026/11/12 00:43:25 INFO Received push request method=POST path=/37492026/11/12 00:43:25 INFO Received uploads request method=POST path=/37502026/11/12 00:43:25 INFO Received push request method=POST path=/37512026/11/12 00:43:25 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."37522026/11/12 00:43:25 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."37532026/11/12 00:43:25 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."37542026/11/12 00:43:25 INFO Received push request method=POST path=/37552026/11/12 00:43:25 INFO Received push request method=POST path=/37562026/11/12 00:43:25 INFO Received complete push request method=POST path=/37572026/11/12 00:43:25 INFO Completed push id=16 objects=5 duration=1.48466ms37582026/11/12 00:43:25 INFO Received complete upload request method=POST path=/37592026/11/12 00:43:25 INFO Completed upload id=13 objects=10 duration=1.28642ms37602026/11/12 00:43:25 INFO Received push request method=POST path=/37612026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=037622026/11/12 00:43:25 INFO Received uploads request method=POST path=/37632026/11/12 00:43:25 WARN Force mode enabled - objects will be deleted immediately without grace period37642026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=037652026/11/12 00:43:25 INFO Received uploads request method=POST path=/37662026/11/12 00:43:25 WARN Force mode enabled - objects will be deleted immediately without grace period37672026/11/12 00:43:25 INFO Received uploads request method=POST path=/37682026/11/12 00:43:25 INFO Received push request method=POST path=/37692026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=037702026/11/12 00:43:25 INFO Received uploads request method=POST path=/37712026/11/12 00:43:25 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."37722026/11/12 00:43:25 WARN Force mode enabled - objects will be deleted immediately without grace period37732026/11/12 00:43:25 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."37742026/11/12 00:43:25 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."37752026/11/12 00:43:25 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."37762026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=037772026/11/12 00:43:25 WARN Force mode enabled - objects will be deleted immediately without grace period37782026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=037792026/11/12 00:43:25 WARN Force mode enabled - objects will be deleted immediately without grace period37802026/11/12 00:43:25 INFO Received uploads request method=POST path=/37812026/11/12 00:43:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037822026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=037832026/11/12 00:43:25 INFO Vacuumed table table=pending_closures37842026/11/12 00:43:25 INFO Vacuumed table table=commit_receipts37852026/11/12 00:43:25 INFO Vacuumed table table=pending_objects37862026/11/12 00:43:25 INFO Vacuumed table table=multipart_uploads37872026/11/12 00:43:25 INFO Vacuumed table table=closures37882026/11/12 00:43:25 INFO Vacuumed table table=objects37892026/11/12 00:43:25 INFO Received complete push request method=POST path=/37902026/11/12 00:43:25 INFO Completed push id=24 objects=9 duration=2.04906ms37912026/11/12 00:43:25 INFO Received uploads request method=POST path=/37922026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=037932026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=037942026/11/12 00:43:25 WARN Force mode enabled - objects will be deleted immediately without grace period37952026/11/12 00:43:25 INFO Received complete upload request method=POST path=/37962026/11/12 00:43:25 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=037972026/11/12 00:43:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037982026/11/12 00:43:25 INFO Vacuumed table table=pending_closures37992026/11/12 00:43:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038002026/11/12 00:43:25 INFO Completed upload id=18 objects=4 duration=1.59353ms38012026/11/12 00:43:25 INFO Vacuumed table table=pending_closures38022026/11/12 00:43:25 INFO Vacuumed table table=commit_receipts38032026/11/12 00:43:25 INFO Vacuumed table table=commit_receipts38042026/11/12 00:43:25 INFO Vacuumed table table=pending_closures38052026/11/12 00:43:25 INFO Vacuumed table table=pending_objects38062026/11/12 00:43:25 INFO Vacuumed table table=pending_objects38072026/11/12 00:43:25 INFO Vacuumed table table=multipart_uploads38082026/11/12 00:43:25 INFO Vacuumed table table=multipart_uploads38092026/11/12 00:43:25 INFO Vacuumed table table=commit_receipts38102026/11/12 00:43:25 INFO Vacuumed table table=closures38112026/11/12 00:43:25 INFO Vacuumed table table=closures38122026/11/12 00:43:25 INFO Vacuumed table table=pending_objects38132026/11/12 00:43:25 INFO Vacuumed table table=objects38142026/11/12 00:43:25 INFO Vacuumed table table=objects38152026/11/12 00:43:25 INFO Vacuumed table table=multipart_uploads38162026/11/12 00:43:25 INFO Vacuumed table table=closures38172026/11/12 00:43:25 INFO Vacuumed table table=objects38182026/11/12 00:43:25 INFO Received uploads request method=POST path=/38192026/11/12 00:43:25 INFO Received complete push request method=POST path=/38202026/11/12 00:43:25 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."38212026/11/12 00:43:25 INFO Completed push id=9 objects=24 duration=1.92783ms38222026/11/12 00:43:25 INFO Received push request method=POST path=/38232026/11/12 00:43:25 INFO Received push request method=POST path=/38242026/11/12 00:43:25 INFO Received push request method=POST path=/38252026/11/12 00:43:25 INFO Received complete upload request method=POST path=/38262026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=038272026/11/12 00:43:25 INFO Received complete push request method=POST path=/38282026/11/12 00:43:25 INFO Completed upload id=14 objects=8 duration=1.90003ms38292026/11/12 00:43:25 INFO Completed push id=19 objects=8 duration=2.79264ms38302026/11/12 00:43:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038312026/11/12 00:43:25 INFO Vacuumed table table=pending_closures38322026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=038332026/11/12 00:43:25 INFO Vacuumed table table=commit_receipts38342026/11/12 00:43:25 WARN Force mode enabled - objects will be deleted immediately without grace period38352026/11/12 00:43:25 INFO Received complete push request method=POST path=/38362026/11/12 00:43:25 INFO Vacuumed table table=pending_objects38372026/11/12 00:43:25 INFO Vacuumed table table=multipart_uploads38382026/11/12 00:43:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038392026/11/12 00:43:25 INFO Vacuumed table table=closures38402026/11/12 00:43:25 INFO Vacuumed table table=pending_closures38412026/11/12 00:43:25 INFO Vacuumed table table=objects38422026/11/12 00:43:25 INFO Completed push id=22 objects=3 duration=3.43631ms38432026/11/12 00:43:25 INFO Vacuumed table table=commit_receipts38442026/11/12 00:43:25 INFO Received uploads request method=POST path=/38452026/11/12 00:43:25 INFO Vacuumed table table=pending_objects38462026/11/12 00:43:25 INFO Vacuumed table table=multipart_uploads38472026/11/12 00:43:25 INFO Vacuumed table table=closures38482026/11/12 00:43:25 INFO Received uploads request method=POST path=/38492026/11/12 00:43:25 INFO Vacuumed table table=objects38502026/11/12 00:43:25 INFO Received push request method=POST path=/38512026/11/12 00:43:25 INFO Received complete push request method=POST path=/38522026/11/12 00:43:25 INFO Received uploads request method=POST path=/38532026/11/12 00:43:25 INFO Received complete push request method=POST path=/38542026/11/12 00:43:25 INFO Completed push id=15 objects=7 duration=2.38557ms38552026/11/12 00:43:25 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=11 objects-failed-to-delete=038562026/11/12 00:43:25 INFO Completed push id=17 objects=22 duration=3.49462ms38572026/11/12 00:43:25 INFO Received uploads request method=POST path=/38582026/11/12 00:43:25 INFO Vacuumed table table=pending_closures38592026/11/12 00:43:25 INFO Vacuumed table table=commit_receipts38602026/11/12 00:43:25 INFO Received uploads request method=POST path=/38612026/11/12 00:43:25 INFO Vacuumed table table=pending_objects38622026/11/12 00:43:25 INFO Vacuumed table table=multipart_uploads38632026/11/12 00:43:25 INFO Vacuumed table table=closures38642026/11/12 00:43:25 INFO Vacuumed table table=objects38652026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=038662026/11/12 00:43:25 WARN Force mode enabled - objects will be deleted immediately without grace period38672026/11/12 00:43:25 INFO Received complete upload request method=POST path=/38682026/11/12 00:43:25 INFO Received push request method=POST path=/38692026/11/12 00:43:25 INFO Completed upload id=23 objects=18 duration=2.87252ms38702026/11/12 00:43:25 INFO Received uploads request method=POST path=/38712026/11/12 00:43:25 INFO Received push request method=POST path=/38722026/11/12 00:43:25 INFO Received uploads request method=POST path=/38732026/11/12 00:43:25 INFO Received complete upload request method=POST path=/38742026/11/12 00:43:25 INFO Completed upload id=15 objects=9 duration=1.53573ms38752026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=038762026/11/12 00:43:25 INFO Received uploads request method=POST path=/38772026/11/12 00:43:25 INFO Received push request method=POST path=/38782026/11/12 00:43:25 INFO Received complete upload request method=POST path=/38792026/11/12 00:43:25 INFO Completed upload id=19 objects=25 duration=2.11818ms38802026/11/12 00:43:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038812026/11/12 00:43:25 INFO Vacuumed table table=pending_closures38822026/11/12 00:43:25 INFO Vacuumed table table=commit_receipts38832026/11/12 00:43:25 INFO Aborted multipart uploads count=0 kept=038842026/11/12 00:43:25 INFO Vacuumed table table=pending_objects38852026/11/12 00:43:25 INFO Vacuumed table table=multipart_uploads38862026/11/12 00:43:25 INFO Vacuumed table table=closures38872026/11/12 00:43:25 INFO Vacuumed table table=objects38882026/11/12 00:43:25 WARN Force mode enabled - objects will be deleted immediately without grace period38892026/11/12 00:43:25 INFO Received uploads request method=POST path=/38902026/11/12 00:43:25 INFO Received uploads request method=POST path=/38912026/11/12 00:43:25 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038922026/11/12 00:43:25 INFO Vacuumed table table=pending_closures38932026/11/12 00:43:25 INFO Received complete upload request method=POST path=/38942026/11/12 00:43:25 INFO Vacuumed table table=commit_receipts38952026/11/12 00:43:25 INFO Completed upload id=10 objects=10 duration=1.48284ms38962026/11/12 00:43:25 INFO Received uploads request method=POST path=/38972026/11/12 00:43:25 INFO Vacuumed table table=pending_objects38982026/11/12 00:43:25 INFO Received push request method=POST path=/38992026/11/12 00:43:25 INFO Vacuumed table table=multipart_uploads39002026/11/12 00:43:25 INFO Received uploads request method=POST path=/39012026/11/12 00:43:25 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."39022026/11/12 00:43:25 INFO Vacuumed table table=closures39032026/11/12 00:43:25 INFO Vacuumed table table=objects39042026/11/12 00:43:25 INFO Received push request method=POST path=/39052026/11/12 00:43:26 INFO Received push request method=POST path=/39062026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=039072026/11/12 00:43:26 WARN Force mode enabled - objects will be deleted immediately without grace period39082026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=039092026/11/12 00:43:26 WARN Force mode enabled - objects will be deleted immediately without grace period39102026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039112026/11/12 00:43:26 INFO Vacuumed table table=pending_closures39122026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=039132026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts39142026/11/12 00:43:26 INFO Vacuumed table table=pending_closures39152026/11/12 00:43:26 INFO Received push request method=POST path=/39162026/11/12 00:43:26 INFO Vacuumed table table=pending_objects39172026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts39182026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads39192026/11/12 00:43:26 INFO Vacuumed table table=closures39202026/11/12 00:43:26 INFO Vacuumed table table=pending_objects39212026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads39222026/11/12 00:43:26 INFO Vacuumed table table=objects39232026/11/12 00:43:26 INFO Vacuumed table table=closures39242026/11/12 00:43:26 INFO Vacuumed table table=objects39252026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=039262026/11/12 00:43:26 INFO Received uploads request method=POST path=/39272026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039282026/11/12 00:43:26 INFO Vacuumed table table=pending_closures39292026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts39302026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=039312026/11/12 00:43:26 INFO Vacuumed table table=pending_objects39322026/11/12 00:43:26 INFO Received complete push request method=POST path=/39332026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads39342026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039352026/11/12 00:43:26 INFO Vacuumed table table=closures39362026/11/12 00:43:26 INFO Received complete upload request method=POST path=/39372026/11/12 00:43:26 INFO Received complete push request method=POST path=/39382026/11/12 00:43:26 INFO Completed push id=21 objects=17 duration=2.12517ms39392026/11/12 00:43:26 INFO Vacuumed table table=objects39402026/11/12 00:43:26 INFO Vacuumed table table=pending_closures39412026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts39422026/11/12 00:43:26 INFO Vacuumed table table=pending_objects39432026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads39442026/11/12 00:43:26 INFO Completed upload id=23 objects=16 duration=3.20985ms39452026/11/12 00:43:26 INFO Completed push id=14 objects=4 duration=3.62348ms39462026/11/12 00:43:26 INFO Vacuumed table table=closures39472026/11/12 00:43:26 INFO Vacuumed table table=objects39482026/11/12 00:43:26 INFO Received push request method=POST path=/39492026/11/12 00:43:26 INFO Received complete upload request method=POST path=/39502026/11/12 00:43:26 INFO Received uploads request method=POST path=/39512026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=039522026/11/12 00:43:26 INFO Completed upload id=22 objects=22 duration=1.61495ms39532026/11/12 00:43:26 WARN Force mode enabled - objects will be deleted immediately without grace period39542026/11/12 00:43:26 INFO Received complete upload request method=POST path=/39552026/11/12 00:43:26 INFO Completed upload id=7 objects=10 duration=2.26791ms39562026/11/12 00:43:26 INFO Received complete push request method=POST path=/39572026/11/12 00:43:26 INFO Received uploads request method=POST path=/39582026/11/12 00:43:26 INFO Completed push id=1 objects=11 duration=2.3864ms39592026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=039602026/11/12 00:43:26 INFO Received push request method=POST path=/39612026/11/12 00:43:26 INFO Vacuumed table table=pending_closures39622026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts39632026/11/12 00:43:26 INFO Vacuumed table table=pending_objects39642026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads39652026/11/12 00:43:26 INFO Vacuumed table table=closures39662026/11/12 00:43:26 INFO Received push request method=POST path=/39672026/11/12 00:43:26 INFO Vacuumed table table=objects39682026/11/12 00:43:26 INFO Received complete push request method=POST path=/39692026/11/12 00:43:26 INFO Completed push id=19 objects=7 duration=1.61654ms39702026/11/12 00:43:26 INFO Received uploads request method=POST path=/39712026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=039722026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=039732026/11/12 00:43:26 WARN Force mode enabled - objects will be deleted immediately without grace period39742026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039752026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039762026/11/12 00:43:26 INFO Received push request method=POST path=/39772026/11/12 00:43:26 INFO Vacuumed table table=pending_closures39782026/11/12 00:43:26 INFO Vacuumed table table=pending_closures39792026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts39802026/11/12 00:43:26 INFO Received push request method=POST path=/39812026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts39822026/11/12 00:43:26 INFO Received push request method=POST path=/39832026/11/12 00:43:26 INFO Vacuumed table table=pending_objects39842026/11/12 00:43:26 INFO Vacuumed table table=pending_objects39852026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads39862026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads39872026/11/12 00:43:26 INFO Vacuumed table table=closures39882026/11/12 00:43:26 INFO Vacuumed table table=closures39892026/11/12 00:43:26 INFO Vacuumed table table=objects39902026/11/12 00:43:26 INFO Received uploads request method=POST path=/39912026/11/12 00:43:26 INFO Vacuumed table table=objects39922026/11/12 00:43:26 INFO Received complete upload request method=POST path=/39932026/11/12 00:43:26 INFO Completed upload id=22 objects=2 duration=1.38537ms39942026/11/12 00:43:26 INFO Received push request method=POST path=/39952026/11/12 00:43:26 INFO Received complete upload request method=POST path=/39962026/11/12 00:43:26 INFO Completed upload id=16 objects=3 duration=1.58432ms39972026/11/12 00:43:26 INFO Received complete push request method=POST path=/39982026/11/12 00:43:26 INFO Received uploads request method=POST path=/39992026/11/12 00:43:26 INFO Received push request method=POST path=/40002026/11/12 00:43:26 INFO Received uploads request method=POST path=/40012026/11/12 00:43:26 INFO Completed push id=14 objects=3 duration=3.603969ms40022026/11/12 00:43:26 INFO Received complete push request method=POST path=/40032026/11/12 00:43:26 INFO Completed push id=26 objects=5 duration=2.12379ms40042026/11/12 00:43:26 INFO Received push request method=POST path=/40052026/11/12 00:43:26 INFO Received uploads request method=POST path=/40062026/11/12 00:43:26 INFO Received complete push request method=POST path=/40072026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=040082026/11/12 00:43:26 INFO Received uploads request method=POST path=/40092026/11/12 00:43:26 INFO Completed push id=21 objects=3 duration=1.26791ms40102026/11/12 00:43:26 INFO Received uploads request method=POST path=/40112026/11/12 00:43:26 INFO Received uploads request method=POST path=/40122026/11/12 00:43:26 INFO Received push request method=POST path=/40132026/11/12 00:43:26 INFO Received push request method=POST path=/40142026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=040152026/11/12 00:43:26 INFO Vacuumed table table=pending_closures40162026/11/12 00:43:26 INFO Received uploads request method=POST path=/40172026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts40182026/11/12 00:43:26 INFO Vacuumed table table=pending_objects40192026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads40202026/11/12 00:43:26 INFO Vacuumed table table=closures40212026/11/12 00:43:26 INFO Vacuumed table table=objects40222026/11/12 00:43:26 INFO Received push request method=POST path=/40232026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=040242026/11/12 00:43:26 INFO Received complete upload request method=POST path=/40252026/11/12 00:43:26 INFO Completed upload id=24 objects=14 duration=1.4951ms40262026/11/12 00:43:26 INFO Received complete push request method=POST path=/40272026/11/12 00:43:26 WARN Force mode enabled - objects will be deleted immediately without grace period40282026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=040292026/11/12 00:43:26 INFO Completed push id=21 objects=9 duration=2.24743ms40302026/11/12 00:43:26 INFO Received uploads request method=POST path=/40312026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040322026/11/12 00:43:26 INFO Vacuumed table table=pending_closures40332026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts40342026/11/12 00:43:26 INFO Vacuumed table table=pending_objects40352026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads40362026/11/12 00:43:26 INFO Vacuumed table table=closures40372026/11/12 00:43:26 INFO Vacuumed table table=objects40382026/11/12 00:43:26 INFO Received push request method=POST path=/40392026/11/12 00:43:26 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."40402026/11/12 00:43:26 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."40412026/11/12 00:43:26 INFO Received complete push request method=POST path=/40422026/11/12 00:43:26 INFO Received push request method=POST path=/40432026/11/12 00:43:26 INFO Received uploads request method=POST path=/40442026/11/12 00:43:26 INFO Completed push id=26 objects=14 duration=2.97817ms40452026/11/12 00:43:26 INFO Received push request method=POST path=/40462026/11/12 00:43:26 INFO Received complete upload request method=POST path=/40472026/11/12 00:43:26 INFO Completed upload id=16 objects=17 duration=2.32708ms40482026/11/12 00:43:26 INFO Received complete push request method=POST path=/40492026/11/12 00:43:26 INFO Completed push id=19 objects=3 duration=1.63346ms40502026/11/12 00:43:26 INFO Received uploads request method=POST path=/40512026/11/12 00:43:26 INFO Received uploads request method=POST path=/40522026/11/12 00:43:26 INFO Received uploads request method=POST path=/40532026/11/12 00:43:26 INFO Received uploads request method=POST path=/40542026/11/12 00:43:26 INFO Received uploads request method=POST path=/40552026/11/12 00:43:26 INFO Received push request method=POST path=/40562026/11/12 00:43:26 INFO Received complete upload request method=POST path=/40572026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=040582026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=040592026/11/12 00:43:26 INFO Completed upload id=11 objects=7 duration=1.7461ms40602026/11/12 00:43:26 WARN Force mode enabled - objects will be deleted immediately without grace period40612026/11/12 00:43:26 WARN Force mode enabled - objects will be deleted immediately without grace period40622026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040632026/11/12 00:43:26 INFO Vacuumed table table=pending_closures40642026/11/12 00:43:26 INFO Received push request method=POST path=/40652026/11/12 00:43:26 INFO Received complete push request method=POST path=/40662026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts40672026/11/12 00:43:26 INFO Received uploads request method=POST path=/40682026/11/12 00:43:26 INFO Completed push id=14 objects=3 duration=1.636969ms40692026/11/12 00:43:26 INFO Vacuumed table table=pending_objects40702026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=040712026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads40722026/11/12 00:43:26 INFO Vacuumed table table=closures40732026/11/12 00:43:26 INFO Vacuumed table table=objects40742026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=3 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040752026/11/12 00:43:26 INFO Vacuumed table table=pending_closures40762026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts40772026/11/12 00:43:26 INFO Received push request method=POST path=/40782026/11/12 00:43:26 INFO Vacuumed table table=pending_objects40792026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads40802026/11/12 00:43:26 INFO Vacuumed table table=closures40812026/11/12 00:43:26 INFO Vacuumed table table=objects40822026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=040832026/11/12 00:43:26 INFO Received push request method=POST path=/40842026/11/12 00:43:26 WARN Force mode enabled - objects will be deleted immediately without grace period40852026/11/12 00:43:26 INFO Received uploads request method=POST path=/40862026/11/12 00:43:26 INFO Received uploads request method=POST path=/40872026/11/12 00:43:26 INFO Received uploads request method=POST path=/40882026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=3 objects-failed-to-delete=040892026/11/12 00:43:26 INFO Received push request method=POST path=/40902026/11/12 00:43:26 INFO Vacuumed table table=pending_closures40912026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts40922026/11/12 00:43:26 INFO Received complete upload request method=POST path=/40932026/11/12 00:43:26 INFO Vacuumed table table=pending_objects40942026/11/12 00:43:26 INFO Received push request method=POST path=/40952026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads40962026/11/12 00:43:26 INFO Vacuumed table table=closures40972026/11/12 00:43:26 INFO Completed upload id=12 objects=26 duration=3.05153ms40982026/11/12 00:43:26 INFO Vacuumed table table=objects40992026/11/12 00:43:26 INFO Received push request method=POST path=/41002026/11/12 00:43:26 INFO Received push request method=POST path=/41012026/11/12 00:43:26 INFO Received uploads request method=POST path=/41022026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=041032026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041042026/11/12 00:43:26 INFO Vacuumed table table=pending_closures41052026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts41062026/11/12 00:43:26 INFO Vacuumed table table=pending_objects41072026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads41082026/11/12 00:43:26 INFO Vacuumed table table=closures41092026/11/12 00:43:26 INFO Vacuumed table table=objects41102026/11/12 00:43:26 INFO Received complete upload request method=POST path=/41112026/11/12 00:43:26 INFO Completed upload id=21 objects=7 duration=2.77021ms41122026/11/12 00:43:26 INFO Received push request method=POST path=/41132026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=041142026/11/12 00:43:26 INFO Received complete push request method=POST path=/41152026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041162026/11/12 00:43:26 INFO Vacuumed table table=pending_closures41172026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts41182026/11/12 00:43:26 INFO Received push request method=POST path=/41192026/11/12 00:43:26 INFO Completed push id=7 objects=15 duration=3.44494ms41202026/11/12 00:43:26 INFO Vacuumed table table=pending_objects41212026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads41222026/11/12 00:43:26 INFO Received complete push request method=POST path=/41232026/11/12 00:43:26 INFO Vacuumed table table=closures41242026/11/12 00:43:26 INFO Vacuumed table table=objects41252026/11/12 00:43:26 INFO Completed push id=16 objects=16 duration=4.1568ms41262026/11/12 00:43:26 INFO Received uploads request method=POST path=/41272026/11/12 00:43:26 INFO Received uploads request method=POST path=/41282026/11/12 00:43:26 INFO Received push request method=POST path=/41292026/11/12 00:43:26 INFO Received complete push request method=POST path=/41302026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=041312026/11/12 00:43:26 INFO Received push request method=POST path=/41322026/11/12 00:43:26 INFO Completed push id=4 objects=6 duration=2.25263ms41332026/11/12 00:43:26 INFO Received complete upload request method=POST path=/41342026/11/12 00:43:26 INFO Received uploads request method=POST path=/41352026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=8 objects-deleted-after-grace-period=8 objects-failed-to-delete=041362026/11/12 00:43:26 INFO Completed upload id=14 objects=16 duration=2.19662ms41372026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041382026/11/12 00:43:26 INFO Vacuumed table table=pending_closures41392026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts41402026/11/12 00:43:26 INFO Vacuumed table table=pending_closures41412026/11/12 00:43:26 INFO Vacuumed table table=pending_objects41422026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts41432026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads41442026/11/12 00:43:26 INFO Received complete push request method=POST path=/41452026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=041462026/11/12 00:43:26 INFO Vacuumed table table=closures41472026/11/12 00:43:26 INFO Vacuumed table table=pending_objects41482026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads41492026/11/12 00:43:26 WARN Force mode enabled - objects will be deleted immediately without grace period41502026/11/12 00:43:26 INFO Vacuumed table table=objects41512026/11/12 00:43:26 INFO Vacuumed table table=closures41522026/11/12 00:43:26 INFO Completed push id=30 objects=4 duration=3.47681ms41532026/11/12 00:43:26 INFO Vacuumed table table=objects41542026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041552026/11/12 00:43:26 INFO Received complete upload request method=POST path=/41562026/11/12 00:43:26 INFO Vacuumed table table=pending_closures41572026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts41582026/11/12 00:43:26 INFO Completed upload id=25 objects=4 duration=1.36661ms41592026/11/12 00:43:26 INFO Vacuumed table table=pending_objects41602026/11/12 00:43:26 INFO Received complete push request method=POST path=/41612026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads41622026/11/12 00:43:26 INFO Vacuumed table table=closures41632026/11/12 00:43:26 INFO Received push request method=POST path=/41642026/11/12 00:43:26 INFO Vacuumed table table=objects41652026/11/12 00:43:26 INFO Completed push id=16 objects=4 duration=2.25506ms41662026/11/12 00:43:26 INFO Received push request method=POST path=/41672026/11/12 00:43:26 INFO Received push request method=POST path=/41682026/11/12 00:43:26 INFO Received uploads request method=POST path=/41692026/11/12 00:43:26 INFO Received complete push request method=POST path=/41702026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=041712026/11/12 00:43:26 INFO Received push request method=POST path=/41722026/11/12 00:43:26 INFO Completed push id=16 objects=8 duration=2.05968ms41732026/11/12 00:43:26 WARN Force mode enabled - objects will be deleted immediately without grace period41742026/11/12 00:43:26 INFO Received push request method=POST path=/41752026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=041762026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041772026/11/12 00:43:26 INFO Received uploads request method=POST path=/41782026/11/12 00:43:26 INFO Vacuumed table table=pending_closures41792026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts41802026/11/12 00:43:26 INFO Received complete upload request method=POST path=/41812026/11/12 00:43:26 INFO Vacuumed table table=pending_objects41822026/11/12 00:43:26 INFO Received push request method=POST path=/41832026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads41842026/11/12 00:43:26 INFO Vacuumed table table=closures41852026/11/12 00:43:26 INFO Completed upload id=19 objects=2 duration=2.51371ms41862026/11/12 00:43:26 INFO Vacuumed table table=objects41872026/11/12 00:43:26 INFO Received uploads request method=POST path=/41882026/11/12 00:43:26 INFO Received push request method=POST path=/41892026/11/12 00:43:26 INFO Received push request method=POST path=/41902026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=041912026/11/12 00:43:26 INFO Received push request method=POST path=/41922026/11/12 00:43:26 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."41932026/11/12 00:43:26 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."41942026/11/12 00:43:26 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."41952026/11/12 00:43:26 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."41962026/11/12 00:43:26 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."41972026/11/12 00:43:26 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."41982026/11/12 00:43:26 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."41992026/11/12 00:43:26 WARN Force mode enabled - objects will be deleted immediately without grace period42002026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=042012026/11/12 00:43:26 INFO Received push request method=POST path=/42022026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042032026/11/12 00:43:26 INFO Vacuumed table table=pending_closures42042026/11/12 00:43:26 WARN Force mode enabled - objects will be deleted immediately without grace period42052026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts42062026/11/12 00:43:26 INFO Received uploads request method=POST path=/42072026/11/12 00:43:26 INFO Received push request method=POST path=/42082026/11/12 00:43:26 INFO Vacuumed table table=pending_objects42092026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads42102026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042112026/11/12 00:43:26 INFO Vacuumed table table=closures42122026/11/12 00:43:26 INFO Vacuumed table table=pending_closures42132026/11/12 00:43:26 INFO Vacuumed table table=objects42142026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts42152026/11/12 00:43:26 INFO Vacuumed table table=pending_objects42162026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads42172026/11/12 00:43:26 INFO Received complete upload request method=POST path=/42182026/11/12 00:43:26 INFO Vacuumed table table=closures42192026/11/12 00:43:26 INFO Completed upload id=20 objects=5 duration=784.27µs42202026/11/12 00:43:26 INFO Vacuumed table table=objects42212026/11/12 00:43:26 INFO Received uploads request method=POST path=/42222026/11/12 00:43:26 INFO Received push request method=POST path=/42232026/11/12 00:43:26 INFO Received push request method=POST path=/42242026/11/12 00:43:26 INFO Received complete push request method=POST path=/42252026/11/12 00:43:26 INFO Completed push id=17 objects=11 duration=1.50138ms42262026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=042272026/11/12 00:43:26 INFO Received uploads request method=POST path=/42282026/11/12 00:43:26 WARN Force mode enabled - objects will be deleted immediately without grace period42292026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042302026/11/12 00:43:26 INFO Vacuumed table table=pending_closures42312026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts42322026/11/12 00:43:26 INFO Received complete upload request method=POST path=/42332026/11/12 00:43:26 INFO Received complete push request method=POST path=/42342026/11/12 00:43:26 INFO Vacuumed table table=pending_objects42352026/11/12 00:43:26 INFO Completed upload id=15 objects=3 duration=1.70283ms42362026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads42372026/11/12 00:43:26 INFO Completed push id=33 objects=7 duration=1.45156ms42382026/11/12 00:43:26 INFO Vacuumed table table=closures42392026/11/12 00:43:26 INFO Received push request method=POST path=/42402026/11/12 00:43:26 INFO Vacuumed table table=objects42412026/11/12 00:43:26 INFO Received complete push request method=POST path=/42422026/11/12 00:43:26 INFO Received complete push request method=POST path=/42432026/11/12 00:43:26 INFO Received uploads request method=POST path=/42442026/11/12 00:43:26 INFO Completed push id=20 objects=8 duration=1.49017ms42452026/11/12 00:43:26 INFO Completed push id=13 objects=15 duration=2.23502ms42462026/11/12 00:43:26 INFO Received uploads request method=POST path=/42472026/11/12 00:43:26 INFO Received uploads request method=POST path=/42482026/11/12 00:43:26 INFO Received push request method=POST path=/42492026/11/12 00:43:26 INFO Received uploads request method=POST path=/42502026/11/12 00:43:26 INFO Received complete push request method=POST path=/42512026/11/12 00:43:26 INFO Received uploads request method=POST path=/42522026/11/12 00:43:26 INFO Completed push id=11 objects=6 duration=3.91838ms42532026/11/12 00:43:26 INFO Received complete upload request method=POST path=/42542026/11/12 00:43:26 INFO Completed upload id=26 objects=3 duration=1.10972ms42552026/11/12 00:43:26 INFO Received uploads request method=POST path=/42562026/11/12 00:43:26 INFO Received complete push request method=POST path=/42572026/11/12 00:43:26 INFO Completed push id=32 objects=7 duration=1.38473ms42582026/11/12 00:43:26 INFO Received push request method=POST path=/42592026/11/12 00:43:26 INFO Received uploads request method=POST path=/42602026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=042612026/11/12 00:43:26 INFO Received complete upload request method=POST path=/42622026/11/12 00:43:26 INFO Completed upload id=27 objects=4 duration=1.321769ms42632026/11/12 00:43:26 INFO Received push request method=POST path=/42642026/11/12 00:43:26 INFO Received push request method=POST path=/42652026/11/12 00:43:26 INFO Received push request method=POST path=/42662026/11/12 00:43:26 INFO Received uploads request method=POST path=/42672026/11/12 00:43:26 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."42682026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=042692026/11/12 00:43:26 WARN Force mode enabled - objects will be deleted immediately without grace period42702026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042712026/11/12 00:43:26 INFO Vacuumed table table=pending_closures42722026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts42732026/11/12 00:43:26 INFO Received complete push request method=POST path=/42742026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=042752026/11/12 00:43:26 INFO Vacuumed table table=pending_objects42762026/11/12 00:43:26 INFO Completed push id=18 objects=6 duration=1.54255ms42772026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads42782026/11/12 00:43:26 INFO Vacuumed table table=closures42792026/11/12 00:43:26 WARN Force mode enabled - objects will be deleted immediately without grace period42802026/11/12 00:43:26 INFO Vacuumed table table=objects42812026/11/12 00:43:26 INFO Received complete push request method=POST path=/42822026/11/12 00:43:26 INFO Completed push id=26 objects=4 duration=1.24086ms42832026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042842026/11/12 00:43:26 INFO Vacuumed table table=pending_closures42852026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts42862026/11/12 00:43:26 INFO Vacuumed table table=pending_objects42872026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads42882026/11/12 00:43:26 INFO Vacuumed table table=closures42892026/11/12 00:43:26 INFO Received push request method=POST path=/42902026/11/12 00:43:26 INFO Received uploads request method=POST path=/42912026/11/12 00:43:26 INFO Received complete push request method=POST path=/42922026/11/12 00:43:26 INFO Vacuumed table table=objects42932026/11/12 00:43:26 INFO Received push request method=POST path=/42942026/11/12 00:43:26 INFO Completed push id=13 objects=11 duration=2.057389ms42952026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=042962026/11/12 00:43:26 WARN Force mode enabled - objects will be deleted immediately without grace period42972026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042982026/11/12 00:43:26 INFO Vacuumed table table=pending_closures42992026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts43002026/11/12 00:43:26 INFO Vacuumed table table=pending_objects43012026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads43022026/11/12 00:43:26 INFO Vacuumed table table=closures43032026/11/12 00:43:26 INFO Vacuumed table table=objects43042026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=043052026/11/12 00:43:26 INFO Received push request method=POST path=/43062026/11/12 00:43:26 INFO Received complete upload request method=POST path=/43072026/11/12 00:43:26 INFO Completed upload id=28 objects=13 duration=1.30847ms43082026/11/12 00:43:26 INFO Received complete upload request method=POST path=/43092026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043102026/11/12 00:43:26 INFO Completed upload id=26 objects=6 duration=816.67µs43112026/11/12 00:43:26 INFO Vacuumed table table=pending_closures43122026/11/12 00:43:26 INFO Received uploads request method=POST path=/43132026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=043142026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts43152026/11/12 00:43:26 INFO Received uploads request method=POST path=/43162026/11/12 00:43:26 INFO Received push request method=POST path=/43172026/11/12 00:43:26 INFO Received push request method=POST path=/43182026/11/12 00:43:26 INFO Received complete upload request method=POST path=/43192026/11/12 00:43:26 INFO Received complete push request method=POST path=/43202026/11/12 00:43:26 WARN Force mode enabled - objects will be deleted immediately without grace period43212026/11/12 00:43:26 INFO Vacuumed table table=pending_objects43222026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads43232026/11/12 00:43:26 INFO Received complete push request method=POST path=/43242026/11/12 00:43:26 INFO Completed upload id=13 objects=8 duration=1.61092ms43252026/11/12 00:43:26 INFO Vacuumed table table=closures43262026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043272026/11/12 00:43:26 INFO Completed push id=20 objects=3 duration=2.43261ms43282026/11/12 00:43:26 INFO Vacuumed table table=objects43292026/11/12 00:43:26 INFO Vacuumed table table=pending_closures43302026/11/12 00:43:26 INFO Completed push id=19 objects=14 duration=2.720569ms43312026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts43322026/11/12 00:43:26 INFO Vacuumed table table=pending_objects43332026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads43342026/11/12 00:43:26 INFO Vacuumed table table=closures43352026/11/12 00:43:26 INFO Vacuumed table table=objects43362026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=043372026/11/12 00:43:26 WARN Force mode enabled - objects will be deleted immediately without grace period43382026/11/12 00:43:26 INFO Received uploads request method=POST path=/43392026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043402026/11/12 00:43:26 INFO Received complete push request method=POST path=/43412026/11/12 00:43:26 INFO Received complete push request method=POST path=/43422026/11/12 00:43:26 INFO Vacuumed table table=pending_closures43432026/11/12 00:43:26 INFO Completed push id=29 objects=3 duration=1.33068ms43442026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts43452026/11/12 00:43:26 INFO Completed push id=17 objects=22 duration=1.909769ms43462026/11/12 00:43:26 INFO Vacuumed table table=pending_objects43472026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads43482026/11/12 00:43:26 INFO Vacuumed table table=closures43492026/11/12 00:43:26 INFO Vacuumed table table=objects43502026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=043512026/11/12 00:43:26 INFO Received complete push request method=POST path=/43522026/11/12 00:43:26 INFO Received uploads request method=POST path=/43532026/11/12 00:43:26 WARN Force mode enabled - objects will be deleted immediately without grace period43542026/11/12 00:43:26 INFO Completed push id=27 objects=11 duration=1.71919ms43552026/11/12 00:43:26 INFO Received complete upload request method=POST path=/43562026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043572026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=043582026/11/12 00:43:26 INFO Vacuumed table table=pending_closures43592026/11/12 00:43:26 INFO Received complete push request method=POST path=/43602026/11/12 00:43:26 INFO Completed upload id=20 objects=3 duration=2.82022ms43612026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts43622026/11/12 00:43:26 WARN Force mode enabled - objects will be deleted immediately without grace period43632026/11/12 00:43:26 INFO Vacuumed table table=pending_objects43642026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads43652026/11/12 00:43:26 INFO Completed push id=27 objects=25 duration=3.296619ms43662026/11/12 00:43:26 INFO Vacuumed table table=closures43672026/11/12 00:43:26 INFO Received complete push request method=POST path=/43682026/11/12 00:43:26 INFO Vacuumed table table=objects43692026/11/12 00:43:26 INFO Received complete upload request method=POST path=/43702026/11/12 00:43:26 INFO Received uploads request method=POST path=/43712026/11/12 00:43:26 INFO Completed upload id=29 objects=5 duration=951.76µs43722026/11/12 00:43:26 INFO Completed push id=9 objects=9 duration=3.34787ms43732026/11/12 00:43:26 INFO Received uploads request method=POST path=/43742026/11/12 00:43:26 INFO Received complete upload request method=POST path=/43752026/11/12 00:43:26 INFO Received complete upload request method=POST path=/43762026/11/12 00:43:26 INFO Completed upload id=31 objects=2 duration=1.12706ms43772026/11/12 00:43:26 INFO Completed upload id=17 objects=25 duration=3.31328ms43782026/11/12 00:43:26 INFO Received complete upload request method=POST path=/43792026/11/12 00:43:26 INFO Received uploads request method=POST path=/43802026/11/12 00:43:26 INFO Completed upload id=29 objects=26 duration=1.361ms43812026/11/12 00:43:26 INFO Received push request method=POST path=/43822026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=043832026/11/12 00:43:26 INFO Received uploads request method=POST path=/43842026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=043852026/11/12 00:43:26 WARN Force mode enabled - objects will be deleted immediately without grace period43862026/11/12 00:43:26 INFO Received uploads request method=POST path=/43872026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043882026/11/12 00:43:26 INFO Vacuumed table table=pending_closures43892026/11/12 00:43:26 INFO Received complete upload request method=POST path=/43902026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts43912026/11/12 00:43:26 INFO Completed upload id=21 objects=8 duration=1.21691ms43922026/11/12 00:43:26 INFO Vacuumed table table=pending_objects43932026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads43942026/11/12 00:43:26 INFO Received complete push request method=POST path=/43952026/11/12 00:43:26 INFO Vacuumed table table=closures43962026/11/12 00:43:26 INFO Received push request method=POST path=/43972026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=043982026/11/12 00:43:26 INFO Completed push id=33 objects=6 duration=1.11331ms43992026/11/12 00:43:26 INFO Vacuumed table table=objects44002026/11/12 00:43:26 INFO Received push request method=POST path=/44012026/11/12 00:43:26 WARN Force mode enabled - objects will be deleted immediately without grace period44022026/11/12 00:43:26 INFO Received complete push request method=POST path=/44032026/11/12 00:43:26 INFO Completed push id=30 objects=8 duration=1.26231ms44042026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044052026/11/12 00:43:26 INFO Vacuumed table table=pending_closures44062026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts44072026/11/12 00:43:26 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."44082026/11/12 00:43:26 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."44092026/11/12 00:43:26 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."44102026/11/12 00:43:26 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."44112026/11/12 00:43:26 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."44122026/11/12 00:43:26 INFO Vacuumed table table=pending_objects44132026/11/12 00:43:26 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."44142026/11/12 00:43:26 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."44152026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads44162026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=044172026/11/12 00:43:26 INFO Received uploads request method=POST path=/44182026/11/12 00:43:26 INFO Vacuumed table table=closures44192026/11/12 00:43:26 INFO Vacuumed table table=pending_closures44202026/11/12 00:43:26 INFO Vacuumed table table=objects44212026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts44222026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=044232026/11/12 00:43:26 INFO Vacuumed table table=pending_objects44242026/11/12 00:43:26 WARN Force mode enabled - objects will be deleted immediately without grace period44252026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads44262026/11/12 00:43:26 INFO Received complete push request method=POST path=/44272026/11/12 00:43:26 INFO Vacuumed table table=closures44282026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044292026/11/12 00:43:26 INFO Completed push id=22 objects=6 duration=1.28932ms44302026/11/12 00:43:26 INFO Vacuumed table table=objects44312026/11/12 00:43:26 INFO Vacuumed table table=pending_closures44322026/11/12 00:43:26 INFO Received uploads request method=POST path=/44332026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts44342026/11/12 00:43:26 INFO Vacuumed table table=pending_objects44352026/11/12 00:43:26 INFO Received push request method=POST path=/44362026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads44372026/11/12 00:43:26 INFO Vacuumed table table=closures44382026/11/12 00:43:26 INFO Vacuumed table table=objects44392026/11/12 00:43:26 INFO Received complete upload request method=POST path=/44402026/11/12 00:43:26 INFO Received complete push request method=POST path=/44412026/11/12 00:43:26 INFO Completed upload id=21 objects=2 duration=1.19391ms44422026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=044432026/11/12 00:43:26 INFO Received complete upload request method=POST path=/44442026/11/12 00:43:26 WARN Force mode enabled - objects will be deleted immediately without grace period44452026/11/12 00:43:26 INFO Completed push id=24 objects=5 duration=2.87431ms44462026/11/12 00:43:26 INFO Completed upload id=31 objects=5 duration=1.06502ms44472026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044482026/11/12 00:43:26 INFO Vacuumed table table=pending_closures44492026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts44502026/11/12 00:43:26 INFO Vacuumed table table=pending_objects44512026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads44522026/11/12 00:43:26 INFO Vacuumed table table=closures44532026/11/12 00:43:26 INFO Vacuumed table table=objects44542026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=044552026/11/12 00:43:26 INFO Received complete upload request method=POST path=/44562026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=044572026/11/12 00:43:26 INFO Completed upload id=24 objects=7 duration=1.70342ms44582026/11/12 00:43:26 INFO Received complete push request method=POST path=/44592026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044602026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044612026/11/12 00:43:26 INFO Completed push id=24 objects=8 duration=1.33647ms44622026/11/12 00:43:26 INFO Vacuumed table table=pending_closures44632026/11/12 00:43:26 INFO Received push request method=POST path=/44642026/11/12 00:43:26 INFO Vacuumed table table=pending_closures44652026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts44662026/11/12 00:43:26 INFO Received complete push request method=POST path=/44672026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts44682026/11/12 00:43:26 INFO Vacuumed table table=pending_objects44692026/11/12 00:43:26 INFO Vacuumed table table=pending_objects44702026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads44712026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads44722026/11/12 00:43:26 INFO Completed push id=16 objects=14 duration=2.87516ms44732026/11/12 00:43:26 INFO Vacuumed table table=closures44742026/11/12 00:43:26 INFO Vacuumed table table=closures44752026/11/12 00:43:26 INFO Vacuumed table table=objects44762026/11/12 00:43:26 INFO Vacuumed table table=objects44772026/11/12 00:43:26 INFO Received push request method=POST path=/44782026/11/12 00:43:26 INFO Received push request method=POST path=/44792026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=044802026/11/12 00:43:26 WARN Force mode enabled - objects will be deleted immediately without grace period44812026/11/12 00:43:26 INFO Received complete push request method=POST path=/44822026/11/12 00:43:26 INFO Received complete upload request method=POST path=/44832026/11/12 00:43:26 INFO Completed push id=32 objects=3 duration=1.24273ms44842026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044852026/11/12 00:43:26 INFO Completed upload id=21 objects=3 duration=1.37627ms44862026/11/12 00:43:26 INFO Vacuumed table table=pending_closures44872026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts44882026/11/12 00:43:26 INFO Vacuumed table table=pending_objects44892026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads44902026/11/12 00:43:26 INFO Vacuumed table table=closures44912026/11/12 00:43:26 INFO Received push request method=POST path=/44922026/11/12 00:43:26 INFO Received complete upload request method=POST path=/44932026/11/12 00:43:26 INFO Vacuumed table table=objects44942026/11/12 00:43:26 INFO Completed upload id=25 objects=9 duration=1.404699ms44952026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=044962026/11/12 00:43:26 INFO Received complete push request method=POST path=/44972026/11/12 00:43:26 INFO Completed push id=5 objects=28 duration=2.23959ms44982026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044992026/11/12 00:43:26 INFO Vacuumed table table=pending_closures45002026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts45012026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=045022026/11/12 00:43:26 INFO Vacuumed table table=pending_objects45032026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads45042026/11/12 00:43:26 INFO Received uploads request method=POST path=/45052026/11/12 00:43:26 INFO Vacuumed table table=closures45062026/11/12 00:43:26 INFO Received push request method=POST path=/45072026/11/12 00:43:26 INFO Vacuumed table table=objects45082026/11/12 00:43:26 WARN Force mode enabled - objects will be deleted immediately without grace period45092026/11/12 00:43:26 INFO Received uploads request method=POST path=/45102026/11/12 00:43:26 INFO Received push request method=POST path=/45112026/11/12 00:43:26 INFO Received uploads request method=POST path=/45122026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045132026/11/12 00:43:26 INFO Vacuumed table table=pending_closures45142026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=045152026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts45162026/11/12 00:43:26 WARN Force mode enabled - objects will be deleted immediately without grace period45172026/11/12 00:43:26 INFO Vacuumed table table=pending_objects45182026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=045192026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads45202026/11/12 00:43:26 INFO Vacuumed table table=closures45212026/11/12 00:43:26 INFO Received complete upload request method=POST path=/45222026/11/12 00:43:26 INFO Vacuumed table table=objects45232026/11/12 00:43:26 INFO Completed upload id=27 objects=3 duration=1.12968ms45242026/11/12 00:43:26 WARN Force mode enabled - objects will be deleted immediately without grace period45252026/11/12 00:43:26 INFO Received complete push request method=POST path=/45262026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045272026/11/12 00:43:26 INFO Completed push id=19 objects=14 duration=1.32945ms45282026/11/12 00:43:26 INFO Vacuumed table table=pending_closures45292026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts45302026/11/12 00:43:26 INFO Vacuumed table table=pending_objects45312026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads45322026/11/12 00:43:26 INFO Vacuumed table table=closures45332026/11/12 00:43:26 INFO Received complete push request method=POST path=/45342026/11/12 00:43:26 INFO Vacuumed table table=objects45352026/11/12 00:43:26 INFO Completed push id=10 objects=2 duration=1.75603ms45362026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=045372026/11/12 00:43:26 INFO Received uploads request method=POST path=/45382026/11/12 00:43:26 INFO Received uploads request method=POST path=/45392026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=7 objects-failed-to-delete=045402026/11/12 00:43:26 INFO Received complete upload request method=POST path=/45412026/11/12 00:43:26 INFO Received complete upload request method=POST path=/45422026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045432026/11/12 00:43:26 INFO Completed upload id=22 objects=6 duration=1.28009ms45442026/11/12 00:43:26 INFO Vacuumed table table=pending_closures45452026/11/12 00:43:26 INFO Vacuumed table table=pending_closures45462026/11/12 00:43:26 INFO Completed upload id=33 objects=5 duration=2.57993ms45472026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts45482026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts45492026/11/12 00:43:26 INFO Received complete upload request method=POST path=/45502026/11/12 00:43:26 INFO Vacuumed table table=pending_objects45512026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads45522026/11/12 00:43:26 INFO Vacuumed table table=pending_objects45532026/11/12 00:43:26 INFO Completed upload id=28 objects=6 duration=1.38559ms45542026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads45552026/11/12 00:43:26 INFO Vacuumed table table=closures45562026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=045572026/11/12 00:43:26 INFO Vacuumed table table=closures45582026/11/12 00:43:26 INFO Vacuumed table table=objects45592026/11/12 00:43:26 INFO Vacuumed table table=objects45602026/11/12 00:43:26 INFO Received uploads request method=POST path=/45612026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045622026/11/12 00:43:26 INFO Vacuumed table table=pending_closures45632026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts45642026/11/12 00:43:26 INFO Vacuumed table table=pending_objects45652026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads45662026/11/12 00:43:26 INFO Vacuumed table table=closures45672026/11/12 00:43:26 INFO Vacuumed table table=objects45682026/11/12 00:43:26 INFO Received complete push request method=POST path=/45692026/11/12 00:43:26 INFO Completed push id=21 objects=26 duration=1.70771ms45702026/11/12 00:43:26 INFO Received push request method=POST path=/45712026/11/12 00:43:26 INFO Received uploads request method=POST path=/45722026/11/12 00:43:26 INFO Received complete push request method=POST path=/45732026/11/12 00:43:26 INFO Received complete push request method=POST path=/45742026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=045752026/11/12 00:43:26 INFO Completed push id=30 objects=4 duration=1.32408ms45762026/11/12 00:43:26 INFO Received complete upload request method=POST path=/45772026/11/12 00:43:26 WARN Force mode enabled - objects will be deleted immediately without grace period45782026/11/12 00:43:26 INFO Completed push id=18 objects=6 duration=3.37275ms45792026/11/12 00:43:26 INFO Completed upload id=17 objects=5 duration=1.26744ms45802026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045812026/11/12 00:43:26 INFO Vacuumed table table=pending_closures45822026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts45832026/11/12 00:43:26 INFO Vacuumed table table=pending_objects45842026/11/12 00:43:26 INFO Received push request method=POST path=/45852026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads45862026/11/12 00:43:26 INFO Vacuumed table table=closures45872026/11/12 00:43:26 INFO Vacuumed table table=objects45882026/11/12 00:43:26 INFO Received complete upload request method=POST path=/45892026/11/12 00:43:26 INFO Received complete push request method=POST path=/45902026/11/12 00:43:26 INFO Completed upload id=31 objects=3 duration=1.72883ms45912026/11/12 00:43:26 INFO Received complete upload request method=POST path=/45922026/11/12 00:43:26 INFO Completed upload id=20 objects=6 duration=1.08763ms45932026/11/12 00:43:26 INFO Completed push id=7 objects=24 duration=4.16041ms45942026/11/12 00:43:26 INFO Received complete push request method=POST path=/45952026/11/12 00:43:26 INFO Completed push id=31 objects=7 duration=1.31052ms45962026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=045972026/11/12 00:43:26 INFO Received complete upload request method=POST path=/45982026/11/12 00:43:26 INFO Completed upload id=23 objects=9 duration=1.08769ms45992026/11/12 00:43:26 WARN Force mode enabled - objects will be deleted immediately without grace period46002026/11/12 00:43:26 INFO Received complete upload request method=POST path=/46012026/11/12 00:43:26 INFO Completed upload id=28 objects=17 duration=1.10645ms46022026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046032026/11/12 00:43:26 INFO Vacuumed table table=pending_closures46042026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts46052026/11/12 00:43:26 INFO Vacuumed table table=pending_objects46062026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads46072026/11/12 00:43:26 INFO Received complete push request method=POST path=/46082026/11/12 00:43:26 INFO Vacuumed table table=closures46092026/11/12 00:43:26 INFO Received uploads request method=POST path=/46102026/11/12 00:43:26 INFO Vacuumed table table=objects46112026/11/12 00:43:26 INFO Completed push id=19 objects=3 duration=1.37895ms46122026/11/12 00:43:26 INFO Received complete upload request method=POST path=/46132026/11/12 00:43:26 INFO Completed upload id=22 objects=2 duration=783.24µs46142026/11/12 00:43:26 INFO Received complete upload request method=POST path=/46152026/11/12 00:43:26 INFO Completed upload id=21 objects=25 duration=1.60042ms46162026/11/12 00:43:26 INFO Received complete upload request method=POST path=/46172026/11/12 00:43:26 INFO Received uploads request method=POST path=/46182026/11/12 00:43:26 INFO Completed upload id=32 objects=5 duration=867.02µs46192026/11/12 00:43:26 INFO Received complete push request method=POST path=/46202026/11/12 00:43:26 INFO Received uploads request method=POST path=/46212026/11/12 00:43:26 INFO Completed push id=23 objects=28 duration=3.487749ms46222026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=046232026/11/12 00:43:26 WARN Force mode enabled - objects will be deleted immediately without grace period46242026/11/12 00:43:26 INFO Received complete push request method=POST path=/46252026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046262026/11/12 00:43:26 INFO Completed push id=29 objects=2 duration=1.23871ms46272026/11/12 00:43:26 INFO Vacuumed table table=pending_closures46282026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts46292026/11/12 00:43:26 INFO Received push request method=POST path=/46302026/11/12 00:43:26 INFO Vacuumed table table=pending_objects46312026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads46322026/11/12 00:43:26 INFO Vacuumed table table=closures46332026/11/12 00:43:26 INFO Vacuumed table table=objects46342026/11/12 00:43:26 INFO Received complete upload request method=POST path=/46352026/11/12 00:43:26 INFO Completed upload id=25 objects=6 duration=1.90569ms46362026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=046372026/11/12 00:43:26 INFO Received uploads request method=POST path=/46382026/11/12 00:43:26 INFO Received uploads request method=POST path=/46392026/11/12 00:43:26 WARN Force mode enabled - objects will be deleted immediately without grace period46402026/11/12 00:43:26 INFO Received complete upload request method=POST path=/46412026/11/12 00:43:26 INFO Received uploads request method=POST path=/46422026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046432026/11/12 00:43:26 INFO Vacuumed table table=pending_closures46442026/11/12 00:43:26 INFO Completed upload id=13 objects=14 duration=3.3921ms46452026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts46462026/11/12 00:43:26 INFO Received complete push request method=POST path=/46472026/11/12 00:43:26 INFO Received push request method=POST path=/46482026/11/12 00:43:26 INFO Vacuumed table table=pending_objects46492026/11/12 00:43:26 INFO Completed push id=20 objects=4 duration=1.49666ms46502026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads46512026/11/12 00:43:26 INFO Received push request method=POST path=/46522026/11/12 00:43:26 INFO Received complete push request method=POST path=/46532026/11/12 00:43:26 INFO Vacuumed table table=closures46542026/11/12 00:43:26 INFO Completed push id=31 objects=6 duration=1.34543ms46552026/11/12 00:43:26 INFO Vacuumed table table=objects46562026/11/12 00:43:26 INFO Received complete push request method=POST path=/46572026/11/12 00:43:26 INFO Completed push id=22 objects=3 duration=3.048519ms46582026/11/12 00:43:26 INFO Received uploads request method=POST path=/46592026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=046602026/11/12 00:43:26 INFO Received complete upload request method=POST path=/46612026/11/12 00:43:26 INFO Received push request method=POST path=/46622026/11/12 00:43:26 INFO Completed upload id=27 objects=8 duration=1.350811ms46632026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046642026/11/12 00:43:26 INFO Received complete upload request method=POST path=/46652026/11/12 00:43:26 INFO Vacuumed table table=pending_closures46662026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts46672026/11/12 00:43:26 INFO Received complete push request method=POST path=/46682026/11/12 00:43:26 INFO Completed upload id=18 objects=2 duration=1.92772ms46692026/11/12 00:43:26 INFO Received uploads request method=POST path=/46702026/11/12 00:43:26 INFO Vacuumed table table=pending_objects46712026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads46722026/11/12 00:43:26 INFO Completed push id=32 objects=9 duration=1.28016ms46732026/11/12 00:43:26 INFO Received uploads request method=POST path=/46742026/11/12 00:43:26 INFO Vacuumed table table=closures46752026/11/12 00:43:26 INFO Received push request method=POST path=/46762026/11/12 00:43:26 INFO Vacuumed table table=objects46772026/11/12 00:43:26 INFO Received push request method=POST path=/46782026/11/12 00:43:26 INFO Received complete upload request method=POST path=/46792026/11/12 00:43:26 INFO Received complete upload request method=POST path=/46802026/11/12 00:43:26 INFO Completed upload id=21 objects=2 duration=858.63µs46812026/11/12 00:43:26 INFO Received complete upload request method=POST path=/46822026/11/12 00:43:26 INFO Completed upload id=34 objects=24 duration=1.88408ms46832026/11/12 00:43:26 INFO Completed upload id=22 objects=8 duration=1.05443ms46842026/11/12 00:43:26 INFO Received push request method=POST path=/46852026/11/12 00:43:26 INFO Received complete upload request method=POST path=/46862026/11/12 00:43:26 INFO Received uploads request method=POST path=/46872026/11/12 00:43:26 INFO Completed upload id=12 objects=6 duration=1.52842ms46882026/11/12 00:43:26 INFO Received push request method=POST path=/46892026/11/12 00:43:26 INFO Received push request method=POST path=/46902026/11/12 00:43:26 INFO Received uploads request method=POST path=/46912026/11/12 00:43:26 INFO Received complete upload request method=POST path=/46922026/11/12 00:43:26 INFO Received push request method=POST path=/46932026/11/12 00:43:26 INFO Completed upload id=33 objects=9 duration=1.21956ms46942026/11/12 00:43:26 INFO Received uploads request method=POST path=/46952026/11/12 00:43:26 INFO Received complete push request method=POST path=/46962026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=046972026/11/12 00:43:26 INFO Received complete push request method=POST path=/46982026/11/12 00:43:26 WARN Force mode enabled - objects will be deleted immediately without grace period46992026/11/12 00:43:26 INFO Completed push id=31 objects=6 duration=3.30956ms47002026/11/12 00:43:26 INFO Completed push id=23 objects=5 duration=1.39847ms47012026/11/12 00:43:26 INFO Received uploads request method=POST path=/47022026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047032026/11/12 00:43:26 INFO Received complete push request method=POST path=/47042026/11/12 00:43:26 INFO Vacuumed table table=pending_closures47052026/11/12 00:43:26 INFO Completed push id=17 objects=3 duration=1.95184ms47062026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts47072026/11/12 00:43:26 INFO Received uploads request method=POST path=/47082026/11/12 00:43:26 INFO Vacuumed table table=pending_objects47092026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads47102026/11/12 00:43:26 INFO Received push request method=POST path=/47112026/11/12 00:43:26 INFO Vacuumed table table=closures47122026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=047132026/11/12 00:43:26 INFO Vacuumed table table=objects47142026/11/12 00:43:26 INFO Received push request method=POST path=/47152026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047162026/11/12 00:43:26 INFO Vacuumed table table=pending_closures47172026/11/12 00:43:26 INFO Received complete push request method=POST path=/47182026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts47192026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=047202026/11/12 00:43:26 INFO Received complete push request method=POST path=/47212026/11/12 00:43:26 INFO Completed push id=11 objects=9 duration=1.99848ms47222026/11/12 00:43:26 INFO Vacuumed table table=pending_objects47232026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads47242026/11/12 00:43:26 WARN Force mode enabled - objects will be deleted immediately without grace period47252026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=047262026/11/12 00:43:26 INFO Completed push id=31 objects=14 duration=2.85423ms47272026/11/12 00:43:26 INFO Vacuumed table table=closures47282026/11/12 00:43:26 INFO Received uploads request method=POST path=/47292026/11/12 00:43:26 INFO Received complete upload request method=POST path=/47302026/11/12 00:43:26 INFO Vacuumed table table=objects47312026/11/12 00:43:26 INFO Received complete upload request method=POST path=/47322026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047332026/11/12 00:43:26 INFO Completed upload id=13 objects=7 duration=991.891µs47342026/11/12 00:43:26 WARN Force mode enabled - objects will be deleted immediately without grace period47352026/11/12 00:43:26 INFO Received complete push request method=POST path=/47362026/11/12 00:43:26 INFO Completed upload id=24 objects=8 duration=1.494769ms47372026/11/12 00:43:26 INFO Vacuumed table table=pending_closures47382026/11/12 00:43:26 INFO Completed push id=13 objects=2 duration=1.33513ms47392026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047402026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts47412026/11/12 00:43:26 INFO Vacuumed table table=pending_closures47422026/11/12 00:43:26 INFO Received uploads request method=POST path=/47432026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts47442026/11/12 00:43:26 INFO Vacuumed table table=pending_objects47452026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads47462026/11/12 00:43:26 INFO Vacuumed table table=pending_objects47472026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads47482026/11/12 00:43:26 INFO Vacuumed table table=closures47492026/11/12 00:43:26 INFO Received complete push request method=POST path=/47502026/11/12 00:43:26 INFO Vacuumed table table=closures47512026/11/12 00:43:26 INFO Vacuumed table table=objects47522026/11/12 00:43:26 INFO Completed push id=24 objects=3 duration=1.38416ms47532026/11/12 00:43:26 INFO Vacuumed table table=objects47542026/11/12 00:43:26 INFO Received complete push request method=POST path=/47552026/11/12 00:43:26 INFO Completed push id=23 objects=5 duration=1.34938ms47562026/11/12 00:43:26 INFO Received complete upload request method=POST path=/47572026/11/12 00:43:26 INFO Completed upload id=20 objects=4 duration=835.65µs47582026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=047592026/11/12 00:43:26 INFO Received complete push request method=POST path=/47602026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047612026/11/12 00:43:26 INFO Received complete upload request method=POST path=/47622026/11/12 00:43:26 INFO Completed push id=15 objects=24 duration=1.57491ms47632026/11/12 00:43:26 INFO Received complete upload request method=POST path=/47642026/11/12 00:43:26 INFO Vacuumed table table=pending_closures47652026/11/12 00:43:26 INFO Completed upload id=27 objects=13 duration=1.35735ms47662026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts47672026/11/12 00:43:26 INFO Completed upload id=28 objects=5 duration=1.13089ms47682026/11/12 00:43:26 INFO Received uploads request method=POST path=/47692026/11/12 00:43:26 INFO Vacuumed table table=pending_objects47702026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads47712026/11/12 00:43:26 INFO Received push request method=POST path=/47722026/11/12 00:43:26 INFO Vacuumed table table=closures47732026/11/12 00:43:26 INFO Vacuumed table table=objects47742026/11/12 00:43:26 INFO Received complete upload request method=POST path=/47752026/11/12 00:43:26 INFO Completed upload id=24 objects=3 duration=1.39639ms47762026/11/12 00:43:26 INFO Received complete push request method=POST path=/47772026/11/12 00:43:26 INFO Completed push id=16 objects=26 duration=1.55046ms47782026/11/12 00:43:26 INFO Received uploads request method=POST path=/47792026/11/12 00:43:26 INFO Received complete upload request method=POST path=/47802026/11/12 00:43:26 INFO Completed upload id=23 objects=7 duration=858.8µs47812026/11/12 00:43:26 INFO Received complete upload request method=POST path=/47822026/11/12 00:43:26 INFO Completed upload id=19 objects=2 duration=1.16002ms47832026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=047842026/11/12 00:43:26 INFO Received uploads request method=POST path=/47852026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047862026/11/12 00:43:26 INFO Received complete upload request method=POST path=/47872026/11/12 00:43:26 INFO Vacuumed table table=pending_closures47882026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts47892026/11/12 00:43:26 INFO Received uploads request method=POST path=/47902026/11/12 00:43:26 INFO Received complete push request method=POST path=/47912026/11/12 00:43:26 INFO Vacuumed table table=pending_objects47922026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads47932026/11/12 00:43:26 INFO Completed upload id=25 objects=16 duration=3.0228ms47942026/11/12 00:43:26 INFO Received complete upload request method=POST path=/47952026/11/12 00:43:26 INFO Completed push id=25 objects=4 duration=1.50371ms47962026/11/12 00:43:26 INFO Vacuumed table table=closures47972026/11/12 00:43:26 INFO Completed upload id=30 objects=5 duration=1.1429ms47982026/11/12 00:43:26 INFO Vacuumed table table=objects47992026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=048002026/11/12 00:43:26 WARN Force mode enabled - objects will be deleted immediately without grace period48012026/11/12 00:43:26 INFO Received complete upload request method=POST path=/48022026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048032026/11/12 00:43:26 INFO Completed upload id=24 objects=7 duration=1.20955ms48042026/11/12 00:43:26 INFO Vacuumed table table=pending_closures48052026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts48062026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=048072026/11/12 00:43:26 INFO Vacuumed table table=pending_objects48082026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads48092026/11/12 00:43:26 INFO Received push request method=POST path=/48102026/11/12 00:43:26 INFO Vacuumed table table=closures48112026/11/12 00:43:26 WARN Force mode enabled - objects will be deleted immediately without grace period48122026/11/12 00:43:26 INFO Vacuumed table table=objects48132026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=048142026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048152026/11/12 00:43:26 WARN Force mode enabled - objects will be deleted immediately without grace period48162026/11/12 00:43:26 INFO Vacuumed table table=pending_closures48172026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048182026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts48192026/11/12 00:43:26 INFO Received push request method=POST path=/48202026/11/12 00:43:26 INFO Vacuumed table table=pending_closures48212026/11/12 00:43:26 INFO Vacuumed table table=pending_objects48222026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads48232026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts48242026/11/12 00:43:26 INFO Received uploads request method=POST path=/48252026/11/12 00:43:26 INFO Vacuumed table table=closures48262026/11/12 00:43:26 INFO Vacuumed table table=pending_objects48272026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads48282026/11/12 00:43:26 INFO Vacuumed table table=objects48292026/11/12 00:43:26 INFO Vacuumed table table=closures48302026/11/12 00:43:26 INFO Vacuumed table table=objects48312026/11/12 00:43:26 INFO Received complete push request method=POST path=/48322026/11/12 00:43:26 INFO Received complete push request method=POST path=/48332026/11/12 00:43:26 INFO Completed push id=26 objects=3 duration=1.16621ms48342026/11/12 00:43:26 INFO Received complete push request method=POST path=/48352026/11/12 00:43:26 INFO Completed push id=31 objects=3 duration=986.71µs48362026/11/12 00:43:26 INFO Completed push id=1 objects=9 duration=1.71406ms48372026/11/12 00:43:26 INFO Received complete upload request method=POST path=/48382026/11/12 00:43:26 INFO Received complete push request method=POST path=/48392026/11/12 00:43:26 INFO Completed upload id=24 objects=18 duration=1.35564ms48402026/11/12 00:43:26 INFO Received complete push request method=POST path=/48412026/11/12 00:43:26 INFO Completed push id=7 objects=7 duration=1.25498ms48422026/11/12 00:43:26 INFO Completed push id=27 objects=6 duration=3.41292ms48432026/11/12 00:43:26 INFO Received complete push request method=POST path=/48442026/11/12 00:43:26 INFO Completed push id=24 objects=24 duration=2.237649ms48452026/11/12 00:43:26 INFO Received push request method=POST path=/48462026/11/12 00:43:26 INFO Received push request method=POST path=/48472026/11/12 00:43:26 INFO Received complete upload request method=POST path=/48482026/11/12 00:43:26 INFO Completed upload id=28 objects=8 duration=1.02233ms48492026/11/12 00:43:26 INFO Received complete push request method=POST path=/48502026/11/12 00:43:26 INFO Completed push id=25 objects=2 duration=1.5132ms48512026/11/12 00:43:26 INFO Received complete push request method=POST path=/48522026/11/12 00:43:26 INFO Received complete push request method=POST path=/48532026/11/12 00:43:26 INFO Completed push id=12 objects=6 duration=938.439µs48542026/11/12 00:43:26 INFO Completed push id=26 objects=23 duration=1.5695ms48552026/11/12 00:43:26 INFO Received complete push request method=POST path=/48562026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=048572026/11/12 00:43:26 INFO Completed push id=32 objects=6 duration=1.31146ms48582026/11/12 00:43:26 INFO Received push request method=POST path=/48592026/11/12 00:43:26 WARN Force mode enabled - objects will be deleted immediately without grace period48602026/11/12 00:43:26 INFO Received complete upload request method=POST path=/48612026/11/12 00:43:26 INFO Completed upload id=29 objects=10 duration=1.00435ms48622026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048632026/11/12 00:43:26 INFO Received push request method=POST path=/48642026/11/12 00:43:26 INFO Vacuumed table table=pending_closures48652026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts48662026/11/12 00:43:26 INFO Received complete push request method=POST path=/48672026/11/12 00:43:26 INFO Vacuumed table table=pending_objects48682026/11/12 00:43:26 INFO Completed push id=27 objects=4 duration=1.28598ms48692026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads48702026/11/12 00:43:26 INFO Vacuumed table table=closures48712026/11/12 00:43:26 INFO Vacuumed table table=objects48722026/11/12 00:43:26 INFO Received complete push request method=POST path=/48732026/11/12 00:43:26 INFO Aborted multipart uploads count=0 kept=048742026/11/12 00:43:26 INFO Completed push id=16 objects=7 duration=2.13537ms48752026/11/12 00:43:26 INFO Received complete upload request method=POST path=/48762026/11/12 00:43:26 INFO Completed upload id=15 objects=2 duration=1.08641ms48772026/11/12 00:43:26 INFO Received complete upload request method=POST path=/48782026/11/12 00:43:26 INFO Completed upload id=33 objects=7 duration=1.14955ms48792026/11/12 00:43:26 INFO Received uploads request method=POST path=/48802026/11/12 00:43:26 INFO Received complete upload request method=POST path=/48812026/11/12 00:43:26 INFO Completed upload id=30 objects=20 duration=1.31675ms48822026/11/12 00:43:26 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=048832026/11/12 00:43:26 INFO Vacuumed table table=pending_closures48842026/11/12 00:43:26 INFO Vacuumed table table=commit_receipts48852026/11/12 00:43:26 INFO Vacuumed table table=pending_objects48862026/11/12 00:43:26 INFO Vacuumed table table=multipart_uploads48872026/11/12 00:43:26 INFO Vacuumed table table=closures48882026/11/12 00:43:26 INFO Received uploads request method=POST path=/48892026/11/12 00:43:26 INFO Received complete upload request method=POST path=/48902026/11/12 00:43:26 INFO Vacuumed table table=objects48912026/11/12 00:43:26 INFO Received complete push request method=POST path=/48922026/11/12 00:43:26 INFO Completed upload id=2 objects=7 duration=2.02633ms48932026/11/12 00:43:26 INFO Completed push id=27 objects=15 duration=1.54756ms48942026/11/12 00:43:26 INFO Received complete upload request method=POST path=/48952026/11/12 00:43:26 INFO Completed upload id=34 objects=3 duration=1.02048ms48962026/11/12 00:43:26 INFO Received uploads request method=POST path=/48972026/11/12 00:43:26 INFO Received complete push request method=POST path=/48982026/11/12 00:43:26 INFO Completed push id=31 objects=11 duration=1.67394ms48992026/11/12 00:43:26 INFO Received complete push request method=POST path=/49002026/11/12 00:43:26 INFO Received complete push request method=POST path=/49012026/11/12 00:43:26 INFO Received complete push request method=POST path=/49022026/11/12 00:43:26 INFO Completed push id=13 objects=22 duration=1.30291ms49032026/11/12 00:43:26 INFO Completed push id=23 objects=6 duration=1.47584ms49042026/11/12 00:43:26 INFO Completed push id=28 objects=26 duration=1.71443ms49052026/11/12 00:43:26 INFO Received uploads request method=POST path=/49062026/11/12 00:43:27 INFO Received complete push request method=POST path=/49072026/11/12 00:43:27 INFO Completed push id=35 objects=11 duration=1.35922ms49082026/11/12 00:43:27 INFO Received complete upload request method=POST path=/49092026/11/12 00:43:27 INFO Completed upload id=32 objects=2 duration=1.04778ms49102026/11/12 00:43:27 INFO Received complete upload request method=POST path=/49112026/11/12 00:43:27 INFO Received uploads request method=POST path=/49122026/11/12 00:43:27 INFO Received uploads request method=POST path=/49132026/11/12 00:43:27 INFO Received complete push request method=POST path=/49142026/11/12 00:43:27 INFO Received complete push request method=POST path=/49152026/11/12 00:43:27 INFO Received complete push request method=POST path=/49162026/11/12 00:43:27 INFO Completed upload id=29 objects=4 duration=12.779549ms49172026/11/12 00:43:27 INFO Completed push id=30 objects=3 duration=6.130119ms49182026/11/12 00:43:27 INFO Completed push id=21 objects=6 duration=7.644589ms49192026/11/12 00:43:27 INFO Completed push id=24 objects=6 duration=7.768059ms49202026/11/12 00:43:27 INFO Aborted multipart uploads count=0 kept=049212026/11/12 00:43:27 INFO Received complete push request method=POST path=/49222026/11/12 00:43:27 INFO Completed push id=33 objects=3 duration=1.16444ms49232026/11/12 00:43:27 WARN Force mode enabled - objects will be deleted immediately without grace period49242026/11/12 00:43:27 INFO Garbage collection completed failed-uploads-deleted=6 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049252026/11/12 00:43:27 INFO Vacuumed table table=pending_closures49262026/11/12 00:43:27 INFO Vacuumed table table=commit_receipts49272026/11/12 00:43:27 INFO Vacuumed table table=pending_objects49282026/11/12 00:43:27 INFO Vacuumed table table=multipart_uploads49292026/11/12 00:43:27 INFO Received complete upload request method=POST path=/49302026/11/12 00:43:27 INFO Vacuumed table table=closures49312026/11/12 00:43:27 INFO Completed upload id=8 objects=6 duration=1.30045ms49322026/11/12 00:43:27 INFO Vacuumed table table=objects49332026/11/12 00:43:27 INFO Received complete upload request method=POST path=/49342026/11/12 00:43:27 INFO Received uploads request method=POST path=/49352026/11/12 00:43:27 INFO Completed upload id=26 objects=2 duration=2.06371ms49362026/11/12 00:43:27 INFO Received push request method=POST path=/49372026/11/12 00:43:27 INFO Received complete push request method=POST path=/49382026/11/12 00:43:27 INFO Received complete push request method=POST path=/49392026/11/12 00:43:27 INFO Received complete upload request method=POST path=/49402026/11/12 00:43:27 INFO Received complete upload request method=POST path=/49412026/11/12 00:43:27 INFO Received push request method=POST path=/49422026/11/12 00:43:27 INFO Completed upload id=22 objects=7 duration=953.36µs49432026/11/12 00:43:27 INFO Completed push id=32 objects=19 duration=1.12589ms49442026/11/12 00:43:27 INFO Received complete upload request method=POST path=/49452026/11/12 00:43:27 INFO Completed push id=28 objects=19 duration=1.68988ms49462026/11/12 00:43:27 INFO Completed upload id=22 objects=5 duration=2.1443ms49472026/11/12 00:43:27 INFO Completed upload id=34 objects=3 duration=1.36091ms49482026/11/12 00:43:27 INFO Received complete push request method=POST path=/49492026/11/12 00:43:27 INFO Completed push id=25 objects=7 duration=1.52465ms49502026/11/12 00:43:27 INFO Aborted multipart uploads count=0 kept=049512026/11/12 00:43:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049522026/11/12 00:43:27 INFO Received uploads request method=POST path=/49532026/11/12 00:43:27 INFO Received uploads request method=POST path=/49542026/11/12 00:43:27 INFO Vacuumed table table=pending_closures49552026/11/12 00:43:27 INFO Vacuumed table table=commit_receipts49562026/11/12 00:43:27 INFO Received complete push request method=POST path=/49572026/11/12 00:43:27 INFO Vacuumed table table=pending_objects49582026/11/12 00:43:27 INFO Vacuumed table table=multipart_uploads49592026/11/12 00:43:27 INFO Completed push id=24 objects=4 duration=1.00165ms49602026/11/12 00:43:27 INFO Aborted multipart uploads count=0 kept=049612026/11/12 00:43:27 INFO Vacuumed table table=closures49622026/11/12 00:43:27 INFO Received complete push request method=POST path=/49632026/11/12 00:43:27 INFO Aborted multipart uploads count=0 kept=049642026/11/12 00:43:27 INFO Vacuumed table table=objects49652026/11/12 00:43:27 INFO Received uploads request method=POST path=/49662026/11/12 00:43:27 WARN Force mode enabled - objects will be deleted immediately without grace period49672026/11/12 00:43:27 INFO Completed push id=28 objects=3 duration=1.86403ms49682026/11/12 00:43:27 WARN Force mode enabled - objects will be deleted immediately without grace period49692026/11/12 00:43:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049702026/11/12 00:43:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049712026/11/12 00:43:27 INFO Vacuumed table table=pending_closures49722026/11/12 00:43:27 INFO Vacuumed table table=pending_closures49732026/11/12 00:43:27 INFO Vacuumed table table=commit_receipts49742026/11/12 00:43:27 INFO Vacuumed table table=commit_receipts49752026/11/12 00:43:27 INFO Vacuumed table table=pending_objects49762026/11/12 00:43:27 INFO Vacuumed table table=pending_objects49772026/11/12 00:43:27 INFO Vacuumed table table=multipart_uploads49782026/11/12 00:43:27 INFO Vacuumed table table=multipart_uploads49792026/11/12 00:43:27 INFO Received complete push request method=POST path=/49802026/11/12 00:43:27 INFO Vacuumed table table=closures49812026/11/12 00:43:27 INFO Vacuumed table table=closures49822026/11/12 00:43:27 INFO Completed push id=29 objects=10 duration=1.17429ms49832026/11/12 00:43:27 INFO Aborted multipart uploads count=0 kept=049842026/11/12 00:43:27 INFO Vacuumed table table=objects49852026/11/12 00:43:27 INFO Vacuumed table table=objects49862026/11/12 00:43:27 WARN Force mode enabled - objects will be deleted immediately without grace period49872026/11/12 00:43:27 INFO Received complete push request method=POST path=/49882026/11/12 00:43:27 INFO Completed push id=25 objects=9 duration=997.34µs49892026/11/12 00:43:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049902026/11/12 00:43:27 INFO Vacuumed table table=pending_closures49912026/11/12 00:43:27 INFO Received complete push request method=POST path=/49922026/11/12 00:43:27 INFO Vacuumed table table=commit_receipts49932026/11/12 00:43:27 INFO Vacuumed table table=pending_objects49942026/11/12 00:43:27 INFO Received complete push request method=POST path=/49952026/11/12 00:43:27 INFO Received uploads request method=POST path=/49962026/11/12 00:43:27 INFO Completed push id=29 objects=15 duration=1.52733ms49972026/11/12 00:43:27 INFO Vacuumed table table=multipart_uploads49982026/11/12 00:43:27 INFO Vacuumed table table=closures49992026/11/12 00:43:27 INFO Completed push id=11 objects=6 duration=2.29371ms50002026/11/12 00:43:27 INFO Vacuumed table table=objects50012026/11/12 00:43:27 INFO Received complete upload request method=POST path=/50022026/11/12 00:43:27 INFO Received complete upload request method=POST path=/50032026/11/12 00:43:27 INFO Completed upload id=12 objects=9 duration=1.524169ms50042026/11/12 00:43:27 INFO Completed upload id=26 objects=24 duration=1.55456ms50052026/11/12 00:43:27 INFO Received push request method=POST path=/50062026/11/12 00:43:27 INFO Received complete upload request method=POST path=/50072026/11/12 00:43:27 INFO Completed upload id=26 objects=3 duration=1.18095ms50082026/11/12 00:43:27 INFO Received complete upload request method=POST path=/50092026/11/12 00:43:27 INFO Received complete push request method=POST path=/50102026/11/12 00:43:27 INFO Completed upload id=30 objects=6 duration=1.19924ms50112026/11/12 00:43:27 INFO Completed push id=30 objects=6 duration=1.6735ms50122026/11/12 00:43:27 INFO Received complete upload request method=POST path=/50132026/11/12 00:43:27 INFO Completed upload id=26 objects=8 duration=1.27203ms50142026/11/12 00:43:27 INFO Received complete push request method=POST path=/50152026/11/12 00:43:27 INFO Completed push id=20 objects=13 duration=1.59037ms50162026/11/12 00:43:27 INFO Received complete upload request method=POST path=/50172026/11/12 00:43:27 INFO Completed upload id=31 objects=5 duration=952.98µs50182026/11/12 00:43:27 INFO Received complete upload request method=POST path=/50192026/11/12 00:43:27 INFO Completed upload id=31 objects=3 duration=1.12124ms50202026/11/12 00:43:27 INFO Received push request method=POST path=/50212026/11/12 00:43:27 INFO Received complete upload request method=POST path=/50222026/11/12 00:43:27 INFO Received complete push request method=POST path=/50232026/11/12 00:43:27 INFO Completed upload id=27 objects=7 duration=1.389659ms50242026/11/12 00:43:27 INFO Completed push id=28 objects=14 duration=1.408569ms50252026/11/12 00:43:27 INFO Received complete push request method=POST path=/50262026/11/12 00:43:27 INFO Completed push id=22 objects=4 duration=1.21692ms50272026/11/12 00:43:27 INFO Aborted multipart uploads count=0 kept=050282026/11/12 00:43:27 INFO Received complete upload request method=POST path=/50292026/11/12 00:43:27 INFO Received complete upload request method=POST path=/50302026/11/12 00:43:27 INFO Completed upload id=27 objects=26 duration=1.27686ms50312026/11/12 00:43:27 INFO Completed upload id=34 objects=3 duration=1.04899ms50322026/11/12 00:43:27 INFO Aborted multipart uploads count=0 kept=050332026/11/12 00:43:27 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."50342026/11/12 00:43:27 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."50352026/11/12 00:43:27 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."50362026/11/12 00:43:27 WARN Force mode enabled - objects will be deleted immediately without grace period50372026/11/12 00:43:27 INFO Received complete upload request method=POST path=/50382026/11/12 00:43:27 INFO Completed upload id=28 objects=3 duration=1.2219ms50392026/11/12 00:43:27 INFO Received push request method=POST path=/50402026/11/12 00:43:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=050412026/11/12 00:43:27 INFO Vacuumed table table=pending_closures50422026/11/12 00:43:27 INFO Received complete upload request method=POST path=/50432026/11/12 00:43:27 INFO Vacuumed table table=commit_receipts50442026/11/12 00:43:27 INFO Vacuumed table table=pending_objects50452026/11/12 00:43:27 INFO Completed upload id=23 objects=7 duration=2.49028ms50462026/11/12 00:43:27 INFO Vacuumed table table=multipart_uploads50472026/11/12 00:43:27 INFO Vacuumed table table=closures50482026/11/12 00:43:27 INFO Received complete upload request method=POST path=/50492026/11/12 00:43:27 INFO Vacuumed table table=objects50502026/11/12 00:43:27 INFO Completed upload id=23 objects=7 duration=1.21877ms50512026/11/12 00:43:27 INFO Received complete push request method=POST path=/50522026/11/12 00:43:27 INFO Completed push id=28 objects=6 duration=1.14265ms50532026/11/12 00:43:27 INFO Aborted multipart uploads count=0 kept=050542026/11/12 00:43:27 INFO Received push request method=POST path=/50552026/11/12 00:43:27 WARN Force mode enabled - objects will be deleted immediately without grace period50562026/11/12 00:43:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=050572026/11/12 00:43:27 INFO Received uploads request method=POST path=/50582026/11/12 00:43:27 INFO Vacuumed table table=pending_closures50592026/11/12 00:43:27 INFO Vacuumed table table=commit_receipts50602026/11/12 00:43:27 INFO Received complete push request method=POST path=/50612026/11/12 00:43:27 INFO Vacuumed table table=pending_objects50622026/11/12 00:43:27 INFO Received complete upload request method=POST path=/50632026/11/12 00:43:27 INFO Vacuumed table table=multipart_uploads50642026/11/12 00:43:27 INFO Completed push id=29 objects=23 duration=1.19542ms50652026/11/12 00:43:27 INFO Completed upload id=26 objects=5 duration=847.37µs50662026/11/12 00:43:27 INFO Vacuumed table table=closures50672026/11/12 00:43:27 INFO Vacuumed table table=objects50682026/11/12 00:43:27 INFO Received complete push request method=POST path=/50692026/11/12 00:43:27 INFO Completed push id=24 objects=5 duration=1.38724ms50702026/11/12 00:43:27 INFO Received complete push request method=POST path=/50712026/11/12 00:43:27 INFO Completed push id=29 objects=6 duration=1.0296ms50722026/11/12 00:43:27 INFO Aborted multipart uploads count=0 kept=050732026/11/12 00:43:27 INFO Received complete push request method=POST path=/50742026/11/12 00:43:27 INFO Received complete push request method=POST path=/50752026/11/12 00:43:27 INFO Completed push id=30 objects=6 duration=892.88µs50762026/11/12 00:43:27 INFO Completed push id=27 objects=3 duration=2.39879ms50772026/11/12 00:43:27 WARN Force mode enabled - objects will be deleted immediately without grace period50782026/11/12 00:43:27 INFO Received complete upload request method=POST path=/50792026/11/12 00:43:27 INFO Received complete upload request method=POST path=/50802026/11/12 00:43:27 INFO Completed upload id=25 objects=2 duration=992.8µs50812026/11/12 00:43:27 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=050822026/11/12 00:43:27 INFO Received push request method=POST path=/50832026/11/12 00:43:27 INFO Vacuumed table table=pending_closures50842026/11/12 00:43:27 INFO Completed upload id=18 objects=22 duration=1.870869ms50852026/11/12 00:43:27 INFO Vacuumed table table=commit_receipts50862026/11/12 00:43:27 INFO Vacuumed table table=pending_objects50872026/11/12 00:43:27 INFO Vacuumed table table=multipart_uploads50882026/11/12 00:43:27 INFO Vacuumed table table=closures50892026/11/12 00:43:27 INFO Vacuumed table table=objects50902026/11/12 00:43:27 INFO Received complete upload request method=POST path=/50912026/11/12 00:43:27 INFO Completed upload id=28 objects=5 duration=810.05µs50922026/11/12 00:43:27 INFO Received complete upload request method=POST path=/50932026/11/12 00:43:27 INFO Completed upload id=26 objects=20 duration=1.05639ms50942026/11/12 00:43:27 INFO Received complete push request method=POST path=/50952026/11/12 00:43:27 INFO Received complete upload request method=POST path=/50962026/11/12 00:43:27 INFO Received complete upload request method=POST path=/50972026/11/12 00:43:27 INFO Completed push id=33 objects=3 duration=926.889µs50982026/11/12 00:43:27 INFO Completed upload id=21 objects=8 duration=1.306859ms50992026/11/12 00:43:27 INFO Completed upload id=30 objects=8 duration=1.21147ms51002026/11/12 00:43:27 INFO Received complete push request method=POST path=/51012026/11/12 00:43:27 INFO Completed push id=28 objects=11 duration=2.70829ms51022026/11/12 00:43:27 INFO Received uploads request method=POST path=/51032026/11/12 00:43:27 INFO Received complete push request method=POST path=/51042026/11/12 00:43:27 INFO Completed push id=29 objects=17 duration=925.27µs51052026/11/12 00:43:27 INFO Received complete push request method=POST path=/51062026/11/12 00:43:27 INFO Completed push id=34 objects=6 duration=919.08µs51072026/11/12 00:43:27 INFO Received complete push request method=POST path=/51082026/11/12 00:43:27 INFO Completed push id=28 objects=7 duration=1.25105ms51092026/11/12 00:43:27 INFO Received complete upload request method=POST path=/51102026/11/12 00:43:27 INFO Received complete upload request method=POST path=/51112026/11/12 00:43:27 INFO Completed upload id=22 objects=7 duration=958.88µs51122026/11/12 00:43:27 INFO Completed upload id=31 objects=6 duration=1.17512ms51132026/11/12 00:43:27 INFO Received complete upload request method=POST path=/51142026/11/12 00:43:27 INFO Completed upload id=21 objects=9 duration=2.1172ms51152026/11/12 00:43:27 INFO Aborted multipart uploads count=0 kept=051162026/11/12 00:43:27 WARN Force mode enabled - objects will be deleted immediately without grace period51172026/11/12 00:43:27 INFO Received complete push request method=POST path=/51182026/11/12 00:43:27 INFO Completed push id=35 objects=8 duration=782.46µs51192026/11/12 00:43:27 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=051202026/11/12 00:43:27 INFO Vacuumed table table=pending_closures51212026/11/12 00:43:27 INFO Vacuumed table table=commit_receipts51222026/11/12 00:43:27 INFO Received complete push request method=POST path=/51232026/11/12 00:43:27 INFO Vacuumed table table=pending_objects51242026/11/12 00:43:27 INFO Vacuumed table table=multipart_uploads51252026/11/12 00:43:27 INFO Completed push id=24 objects=5 duration=1.07098ms51262026/11/12 00:43:27 INFO Vacuumed table table=closures51272026/11/12 00:43:27 INFO Received complete upload request method=POST path=/51282026/11/12 00:43:27 INFO Vacuumed table table=objects51292026/11/12 00:43:27 INFO Completed upload id=32 objects=4 duration=901.95µs51302026/11/12 00:43:27 INFO Received complete push request method=POST path=/51312026/11/12 00:43:27 INFO Received complete push request method=POST path=/51322026/11/12 00:43:27 INFO Received complete upload request method=POST path=/51332026/11/12 00:43:27 INFO Completed push id=29 objects=24 duration=1.6796ms51342026/11/12 00:43:27 INFO Completed upload id=30 objects=6 duration=999.63µs51352026/11/12 00:43:27 INFO Completed push id=25 objects=10 duration=1.85465ms51362026/11/12 00:43:27 INFO Received complete upload request method=POST path=/51372026/11/12 00:43:27 INFO Completed upload id=36 objects=9 duration=1.21026ms51382026/11/12 00:43:27 INFO Aborted multipart uploads count=0 kept=051392026/11/12 00:43:27 INFO Received complete push request method=POST path=/51402026/11/12 00:43:27 WARN Force mode enabled - objects will be deleted immediately without grace period51412026/11/12 00:43:27 INFO Received complete upload request method=POST path=/51422026/11/12 00:43:27 INFO Completed push id=31 objects=5 duration=1.038999ms51432026/11/12 00:43:27 INFO Received complete upload request method=POST path=/51442026/11/12 00:43:27 INFO Completed upload id=25 objects=8 duration=938.37µs51452026/11/12 00:43:27 INFO Completed upload id=29 objects=5 duration=1.17937ms51462026/11/12 00:43:27 INFO Received complete upload request method=POST path=/51472026/11/12 00:43:27 INFO Completed upload id=33 objects=10 duration=1.13889ms51482026/11/12 00:43:27 INFO Aborted multipart uploads count=0 kept=051492026/11/12 00:43:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=051502026/11/12 00:43:27 WARN Force mode enabled - objects will be deleted immediately without grace period51512026/11/12 00:43:27 INFO Vacuumed table table=pending_closures51522026/11/12 00:43:27 INFO Vacuumed table table=commit_receipts51532026/11/12 00:43:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=051542026/11/12 00:43:27 INFO Vacuumed table table=pending_objects51552026/11/12 00:43:27 INFO Vacuumed table table=multipart_uploads51562026/11/12 00:43:27 INFO Vacuumed table table=pending_closures51572026/11/12 00:43:27 INFO Vacuumed table table=closures51582026/11/12 00:43:27 INFO Vacuumed table table=commit_receipts51592026/11/12 00:43:27 INFO Vacuumed table table=objects51602026/11/12 00:43:27 INFO Vacuumed table table=pending_objects51612026/11/12 00:43:27 INFO Vacuumed table table=multipart_uploads51622026/11/12 00:43:27 INFO Vacuumed table table=closures51632026/11/12 00:43:27 INFO Vacuumed table table=objects51642026/11/12 00:43:27 INFO Received complete push request method=POST path=/51652026/11/12 00:43:27 INFO Received complete upload request method=POST path=/51662026/11/12 00:43:27 INFO Completed push id=26 objects=4 duration=1.04419ms51672026/11/12 00:43:27 INFO Completed upload id=30 objects=2 duration=1.38256ms51682026/11/12 00:43:27 INFO Received complete push request method=POST path=/51692026/11/12 00:43:27 INFO Completed push id=32 objects=26 duration=1.51246ms51702026/11/12 00:43:27 INFO Received complete upload request method=POST path=/51712026/11/12 00:43:27 INFO Completed upload id=34 objects=23 duration=1.1762ms51722026/11/12 00:43:27 INFO Received complete push request method=POST path=/51732026/11/12 00:43:27 INFO Completed push id=30 objects=2 duration=1.24506ms51742026/11/12 00:43:27 INFO Received complete push request method=POST path=/51752026/11/12 00:43:27 INFO Completed push id=31 objects=20 duration=1.38282ms51762026/11/12 00:43:27 INFO Received complete upload request method=POST path=/51772026/11/12 00:43:27 INFO Completed upload id=33 objects=2 duration=892.75µs51782026/11/12 00:43:27 INFO Received complete upload request method=POST path=/51792026/11/12 00:43:27 INFO Received complete push request method=POST path=/51802026/11/12 00:43:27 INFO Completed upload id=32 objects=5 duration=1.1778ms51812026/11/12 00:43:27 INFO Completed push id=35 objects=6 duration=1.35783ms51822026/11/12 00:43:27 INFO Received complete upload request method=POST path=/51832026/11/12 00:43:27 INFO Received complete push request method=POST path=/51842026/11/12 00:43:27 INFO Completed upload id=34 objects=5 duration=1.10709ms51852026/11/12 00:43:27 INFO Completed push id=32 objects=7 duration=1.3877ms51862026/11/12 00:43:27 INFO Received complete upload request method=POST path=/51872026/11/12 00:43:27 INFO Aborted multipart uploads count=0 kept=051882026/11/12 00:43:27 INFO Completed upload id=33 objects=3 duration=1.04347ms51892026/11/12 00:43:27 WARN Force mode enabled - objects will be deleted immediately without grace period51902026/11/12 00:43:27 INFO Received complete upload request method=POST path=/51912026/11/12 00:43:27 INFO Completed upload id=35 objects=11 duration=909.62µs51922026/11/12 00:43:27 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=051932026/11/12 00:43:27 INFO Vacuumed table table=pending_closures51942026/11/12 00:43:27 INFO Vacuumed table table=commit_receipts51952026/11/12 00:43:27 INFO Received complete upload request method=POST path=/51962026/11/12 00:43:27 INFO Vacuumed table table=pending_objects51972026/11/12 00:43:27 INFO Vacuumed table table=multipart_uploads51982026/11/12 00:43:27 INFO Completed upload id=33 objects=6 duration=1.15716ms51992026/11/12 00:43:27 INFO Vacuumed table table=closures52002026/11/12 00:43:27 INFO Vacuumed table table=objects52012026/11/12 00:43:27 INFO Aborted multipart uploads count=0 kept=052022026/11/12 00:43:27 WARN Force mode enabled - objects will be deleted immediately without grace period52032026/11/12 00:43:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=052042026/11/12 00:43:27 INFO Vacuumed table table=pending_closures52052026/11/12 00:43:27 INFO Vacuumed table table=commit_receipts52062026/11/12 00:43:27 INFO Received complete upload request method=POST path=/52072026/11/12 00:43:27 INFO Completed upload id=36 objects=3 duration=1.08815ms52082026/11/12 00:43:27 INFO Vacuumed table table=pending_objects52092026/11/12 00:43:27 INFO Vacuumed table table=multipart_uploads52102026/11/12 00:43:27 INFO Vacuumed table table=closures52112026/11/12 00:43:27 INFO Vacuumed table table=objects52122026/11/12 00:43:27 INFO Received complete upload request method=POST path=/52132026/11/12 00:43:27 INFO Completed upload id=34 objects=3 duration=1.13779ms52142026/11/12 00:43:27 INFO Aborted multipart uploads count=0 kept=052152026/11/12 00:43:27 WARN Force mode enabled - objects will be deleted immediately without grace period52162026/11/12 00:43:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=052172026/11/12 00:43:27 INFO Vacuumed table table=pending_closures52182026/11/12 00:43:27 INFO Vacuumed table table=commit_receipts52192026/11/12 00:43:27 INFO Vacuumed table table=pending_objects52202026/11/12 00:43:27 INFO Vacuumed table table=multipart_uploads52212026/11/12 00:43:27 INFO Vacuumed table table=closures52222026/11/12 00:43:27 INFO Aborted multipart uploads count=0 kept=052232026/11/12 00:43:27 INFO Vacuumed table table=objects52242026/11/12 00:43:27 WARN Force mode enabled - objects will be deleted immediately without grace period52252026/11/12 00:43:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=052262026/11/12 00:43:27 INFO Vacuumed table table=pending_closures52272026/11/12 00:43:27 INFO Vacuumed table table=commit_receipts52282026/11/12 00:43:27 INFO Vacuumed table table=pending_objects52292026/11/12 00:43:27 INFO Vacuumed table table=multipart_uploads52302026/11/12 00:43:27 INFO Vacuumed table table=closures52312026/11/12 00:43:27 INFO Vacuumed table table=objects5232--- PASS: FuzzPushGC (0.00s)5233 --- PASS: FuzzPushGC/seed#14 (2.90s)5234 --- PASS: FuzzPushGC/seed#3 (2.96s)5235 --- PASS: FuzzPushGC/seed#9 (3.13s)5236 --- PASS: FuzzPushGC/seed#0 (3.16s)5237 --- PASS: FuzzPushGC/seed#8 (3.35s)5238 --- PASS: FuzzPushGC/seed#5 (3.39s)5239 --- PASS: FuzzPushGC/seed#7 (3.39s)5240 --- PASS: FuzzPushGC/seed#15 (3.47s)5241 --- PASS: FuzzPushGC/seed#13 (3.48s)5242 --- PASS: FuzzPushGC/seed#1 (3.56s)5243 --- PASS: FuzzPushGC/seed#2 (3.59s)5244 --- PASS: FuzzPushGC/seed#4 (3.60s)5245 --- PASS: FuzzPushGC/seed#12 (3.68s)5246 --- PASS: FuzzPushGC/seed#6 (3.70s)5247 --- PASS: FuzzPushGC/seed#11 (3.73s)5248 --- PASS: FuzzPushGC/seed#10 (3.74s)5249PASS5250{"timestamp":"2026-11-12T00:43:27.442258121Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:42522","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(397)"}5251{"timestamp":"2026-11-12T00:43:27.442949781Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:43454","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(391)"}5252{"timestamp":"2026-11-12T00:43:27.442972121Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:43462","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(398)"}52532026-11-12 00:43:27.857 UTC [130] LOG: received smart shutdown request52542026-11-12 00:43:27.862 UTC [130] LOG: background worker "logical replication launcher" (PID 140) exited with exit code 152552026-11-12 00:43:29.123 UTC [135] LOG: checkpoint complete: wrote 3374 buffers (20.6%), wrote 5 SLRU buffers; 0 WAL file(s) added, 0 removed, 33 recycled; write=16.234 s, sync=1.133 s, total=17.441 s; sync files=43470, longest=0.012 s, average=0.001 s; distance=535398 kB, estimate=535398 kB; lsn=0/27A39EA0, redo lsn=0/2223F79052562026-11-12 00:43:29.126 UTC [135] LOG: shutting down52572026-11-12 00:43:29.126 UTC [135] LOG: checkpoint starting: shutdown immediate52582026-11-12 00:43:30.338 UTC [135] LOG: checkpoint complete: wrote 10023 buffers (61.2%), wrote 2 SLRU buffers; 0 WAL file(s) added, 0 removed, 5 recycled; write=0.186 s, sync=1.005 s, total=1.212 s; sync files=3224, longest=0.004 s, average=0.001 s; distance=90089 kB, estimate=490867 kB; lsn=0/27A39F18, redo lsn=0/27A39F1852592026-11-12 00:43:30.436 UTC [130] LOG: database system is shut down5260Running OIDC tests...5261=== RUN TestAudienceForIssuer5262=== PAUSE TestAudienceForIssuer5263=== RUN TestProviderClientGivesUpOnSilentIssuer5264=== PAUSE TestProviderClientGivesUpOnSilentIssuer5265=== RUN TestGlobMatch5266=== PAUSE TestGlobMatch5267=== RUN TestValidateToken_ValidToken5268=== PAUSE TestValidateToken_ValidToken5269=== RUN TestValidateToken_WrongAudience5270=== PAUSE TestValidateToken_WrongAudience5271=== RUN TestValidateToken_Expired5272=== PAUSE TestValidateToken_Expired5273=== RUN TestValidateToken_BoundClaimsMismatch5274=== PAUSE TestValidateToken_BoundClaimsMismatch5275=== RUN TestValidateToken_BoundSubjectMismatch5276=== PAUSE TestValidateToken_BoundSubjectMismatch5277=== RUN TestValidateToken_MultipleProviders5278=== PAUSE TestValidateToken_MultipleProviders5279=== RUN TestValidateToken_UnknownIssuerFetchesNoKeys5280=== PAUSE TestValidateToken_UnknownIssuerFetchesNoKeys5281=== RUN TestValidateToken_GoogleSchemelessIssuer5282=== PAUSE TestValidateToken_GoogleSchemelessIssuer5283=== RUN TestValidateToken_KubernetesServiceAccount5284=== PAUSE TestValidateToken_KubernetesServiceAccount5285=== RUN TestNewValidator_KubernetesRequiresCA5286=== PAUSE TestNewValidator_KubernetesRequiresCA5287=== RUN TestValidateToken_KubernetesIssuerFromOwnToken5288=== PAUSE TestValidateToken_KubernetesIssuerFromOwnToken5289=== RUN TestPins_ReservedForMatchingRule5290=== PAUSE TestPins_ReservedForMatchingRule5291=== RUN TestPins_TopLevelShorthand5292=== PAUSE TestPins_TopLevelShorthand5293=== RUN TestPins_ConfigValidation5294=== PAUSE TestPins_ConfigValidation5295=== RUN TestScopes_LegacyProviderDefaultsToWrite5296=== PAUSE TestScopes_LegacyProviderDefaultsToWrite5297=== RUN TestScopes_Rules5298=== PAUSE TestScopes_Rules5299=== RUN TestScopes_ConfigValidation5300=== PAUSE TestScopes_ConfigValidation5301=== CONT TestAudienceForIssuer5302=== CONT TestValidateToken_GoogleSchemelessIssuer5303--- PASS: TestAudienceForIssuer (0.00s)5304=== CONT TestValidateToken_UnknownIssuerFetchesNoKeys5305=== CONT TestValidateToken_MultipleProviders5306=== CONT TestValidateToken_BoundSubjectMismatch5307=== CONT TestValidateToken_BoundClaimsMismatch53082026/11/12 00:43:32 INFO OIDC provider initialized name=known issuer=https://known.example5309=== CONT TestValidateToken_Expired5310=== CONT TestValidateToken_WrongAudience5311=== CONT TestValidateToken_ValidToken5312=== CONT TestGlobMatch5313=== CONT TestProviderClientGivesUpOnSilentIssuer5314=== CONT TestNewValidator_KubernetesRequiresCA5315=== CONT TestPins_ReservedForMatchingRule5316--- PASS: TestProviderClientGivesUpOnSilentIssuer (0.00s)5317=== CONT TestScopes_Rules5318=== CONT TestScopes_ConfigValidation5319=== CONT TestScopes_LegacyProviderDefaultsToWrite5320=== CONT TestPins_ConfigValidation5321=== CONT TestValidateToken_KubernetesServiceAccount5322=== CONT TestPins_TopLevelShorthand5323--- PASS: TestScopes_ConfigValidation (0.00s)5324=== CONT TestValidateToken_KubernetesIssuerFromOwnToken5325=== RUN TestGlobMatch/foo_foo5326=== PAUSE TestGlobMatch/foo_foo5327=== RUN TestGlobMatch/foo_bar5328=== PAUSE TestGlobMatch/foo_bar5329=== RUN TestGlobMatch/*_5330=== PAUSE TestGlobMatch/*_5331=== RUN TestGlobMatch/*_anything5332=== PAUSE TestGlobMatch/*_anything5333=== RUN TestGlobMatch/foo*_foo5334=== PAUSE TestGlobMatch/foo*_foo5335=== RUN TestGlobMatch/foo*_foobar5336=== PAUSE TestGlobMatch/foo*_foobar5337=== RUN TestGlobMatch/foo*_bar5338=== PAUSE TestGlobMatch/foo*_bar5339=== RUN TestGlobMatch/*bar_bar5340=== PAUSE TestGlobMatch/*bar_bar5341=== RUN TestGlobMatch/*bar_foobar5342=== PAUSE TestGlobMatch/*bar_foobar5343=== RUN TestGlobMatch/*bar_foo5344=== PAUSE TestGlobMatch/*bar_foo5345=== RUN TestGlobMatch/foo*bar_foobar5346=== PAUSE TestGlobMatch/foo*bar_foobar5347=== RUN TestGlobMatch/foo*bar_foo123bar5348--- PASS: TestPins_ConfigValidation (0.00s)5349=== PAUSE TestGlobMatch/foo*bar_foo123bar5350=== RUN TestGlobMatch/foo*bar_foobarbaz5351=== PAUSE TestGlobMatch/foo*bar_foobarbaz5352=== RUN TestGlobMatch/*/*_foo/bar5353=== PAUSE TestGlobMatch/*/*_foo/bar5354=== RUN TestGlobMatch/*/*_foo5355=== PAUSE TestGlobMatch/*/*_foo5356=== RUN TestGlobMatch/refs/heads/*_refs/heads/main5357=== PAUSE TestGlobMatch/refs/heads/*_refs/heads/main5358=== RUN TestGlobMatch/refs/heads/*_refs/tags/v1.05359=== PAUSE TestGlobMatch/refs/heads/*_refs/tags/v1.05360=== RUN TestGlobMatch/refs/*/main_refs/heads/main5361=== PAUSE TestGlobMatch/refs/*/main_refs/heads/main5362=== RUN TestGlobMatch/fo?_foo5363=== PAUSE TestGlobMatch/fo?_foo5364=== RUN TestGlobMatch/fo?_fo5365=== PAUSE TestGlobMatch/fo?_fo5366=== RUN TestGlobMatch/fo?_fooo5367=== PAUSE TestGlobMatch/fo?_fooo5368=== RUN TestGlobMatch/?oo_foo5369=== PAUSE TestGlobMatch/?oo_foo5370=== RUN TestGlobMatch/?oo_boo5371=== PAUSE TestGlobMatch/?oo_boo5372=== RUN TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5373=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5374=== RUN TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5375=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5376=== CONT TestGlobMatch/foo_foo5377=== CONT TestGlobMatch/*/*_foo/bar5378=== CONT TestGlobMatch/refs/heads/*_refs/tags/v1.05379=== CONT TestGlobMatch/refs/heads/*_refs/heads/main5380=== CONT TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5381=== CONT TestGlobMatch/foo*bar_foobar5382=== CONT TestGlobMatch/foo*bar_foobarbaz5383=== CONT TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5384=== CONT TestGlobMatch/foo*bar_foo123bar5385=== CONT TestGlobMatch/*_anything5386=== CONT TestGlobMatch/*bar_foo5387=== CONT TestGlobMatch/foo*_bar5388=== CONT TestGlobMatch/foo*_foo5389=== CONT TestGlobMatch/*bar_foobar5390=== CONT TestGlobMatch/*_5391=== CONT TestGlobMatch/foo*_foobar5392=== CONT TestGlobMatch/foo_bar5393=== CONT TestGlobMatch/?oo_boo5394=== CONT TestGlobMatch/?oo_foo5395=== CONT TestGlobMatch/fo?_fooo5396=== CONT TestGlobMatch/fo?_fo5397=== CONT TestGlobMatch/fo?_foo5398=== CONT TestGlobMatch/refs/*/main_refs/heads/main5399=== CONT TestGlobMatch/*/*_foo5400=== CONT TestGlobMatch/*bar_bar5401--- PASS: TestGlobMatch (0.01s)5402 --- PASS: TestGlobMatch/foo_foo (0.00s)5403 --- PASS: TestGlobMatch/*/*_foo/bar (0.00s)5404 --- PASS: TestGlobMatch/refs/heads/*_refs/tags/v1.0 (0.00s)5405 --- PASS: TestGlobMatch/refs/heads/*_refs/heads/main (0.00s)5406 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main (0.00s)5407 --- PASS: TestGlobMatch/foo*bar_foobar (0.00s)5408 --- PASS: TestGlobMatch/foo*bar_foobarbaz (0.00s)5409 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main (0.00s)5410 --- PASS: TestGlobMatch/foo*bar_foo123bar (0.00s)5411 --- PASS: TestGlobMatch/*_anything (0.00s)5412 --- PASS: TestGlobMatch/*bar_foo (0.00s)5413 --- PASS: TestGlobMatch/foo*_bar (0.00s)5414 --- PASS: TestGlobMatch/foo*_foo (0.00s)5415 --- PASS: TestGlobMatch/*bar_foobar (0.00s)5416 --- PASS: TestGlobMatch/*_ (0.00s)5417 --- PASS: TestGlobMatch/foo*_foobar (0.00s)5418 --- PASS: TestGlobMatch/foo_bar (0.00s)5419 --- PASS: TestGlobMatch/?oo_boo (0.00s)5420 --- PASS: TestGlobMatch/?oo_foo (0.00s)5421 --- PASS: TestGlobMatch/fo?_fooo (0.00s)5422 --- PASS: TestGlobMatch/fo?_fo (0.00s)5423 --- PASS: TestGlobMatch/fo?_foo (0.00s)5424 --- PASS: TestGlobMatch/refs/*/main_refs/heads/main (0.00s)5425 --- PASS: TestGlobMatch/*/*_foo (0.00s)5426 --- PASS: TestGlobMatch/*bar_bar (0.00s)54272026/11/12 00:43:32 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:36769/oidc5428--- PASS: TestValidateToken_Expired (0.02s)54292026/11/12 00:43:33 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:36801/oidc5430--- PASS: TestPins_TopLevelShorthand (0.02s)54312026/11/12 00:43:33 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:40071/oidc5432--- PASS: TestPins_ReservedForMatchingRule (0.03s)54332026/11/12 00:43:33 http: TLS handshake error from 127.0.0.1:40516: remote error: tls: bad certificate5434--- PASS: TestNewValidator_KubernetesRequiresCA (0.04s)54352026/11/12 00:43:33 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:43595/oidc54362026/11/12 00:43:33 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:32987/oidc5437--- PASS: TestValidateToken_BoundClaimsMismatch (0.05s)5438--- PASS: TestValidateToken_UnknownIssuerFetchesNoKeys (0.06s)5439--- PASS: TestScopes_Rules (0.05s)54402026/11/12 00:43:33 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:34373/oidc5441--- PASS: TestValidateToken_ValidToken (0.06s)54422026/11/12 00:43:33 INFO OIDC provider initialized name=kubernetes issuer=https://oidc.eks.invalid/id/ABC12354432026/11/12 00:43:33 INFO OIDC provider initialized name=provider1 issuer=http://127.0.0.1:35895/oidc5444--- PASS: TestValidateToken_KubernetesIssuerFromOwnToken (0.06s)54452026/11/12 00:43:33 INFO OIDC provider initialized name=provider2 issuer=http://127.0.0.1:40061/oidc5446--- PASS: TestValidateToken_MultipleProviders (0.07s)54472026/11/12 00:43:33 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:43463/oidc5448--- PASS: TestValidateToken_BoundSubjectMismatch (0.11s)54492026/11/12 00:43:33 INFO OIDC provider initialized name=google issuer=https://accounts.google.com5450--- PASS: TestValidateToken_GoogleSchemelessIssuer (0.11s)54512026/11/12 00:43:33 INFO OIDC provider initialized name=kubernetes issuer=https://127.0.0.1:338295452--- PASS: TestValidateToken_KubernetesServiceAccount (0.11s)54532026/11/12 00:43:33 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:44253/oidc5454--- PASS: TestValidateToken_WrongAudience (0.13s)54552026/11/12 00:43:33 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:35991/oidc5456--- PASS: TestScopes_LegacyProviderDefaultsToWrite (0.19s)5457PASS5458Running hook tests...5459=== RUN TestSendPathsEmpty5460=== PAUSE TestSendPathsEmpty5461=== RUN TestQueueEnqueueAndFetch5462=== PAUSE TestQueueEnqueueAndFetch5463=== RUN TestQueueDeduplication5464=== PAUSE TestQueueDeduplication5465=== RUN TestQueueRemove5466=== PAUSE TestQueueRemove5467=== RUN TestQueueFetchBatchLimit5468=== PAUSE TestQueueFetchBatchLimit5469=== RUN TestQueueRetryMovesToBack5470=== PAUSE TestQueueRetryMovesToBack5471=== RUN TestQueueFetchRemoveLifecycle5472=== PAUSE TestQueueFetchRemoveLifecycle5473=== RUN TestQueueConcurrentWriters5474=== PAUSE TestQueueConcurrentWriters5475=== RUN TestQueueRemoveLargeClosure5476=== PAUSE TestQueueRemoveLargeClosure5477=== RUN TestServerClientIntegration5478--- PASS: TestServerClientIntegration (0.00s)5479=== RUN TestServerQueueError54802026/11/12 00:43:33 ERROR Failed to queue paths error="permission denied" count=15481--- PASS: TestServerQueueError (0.00s)5482=== RUN TestGetListenerSocketActivation5483 server_test.go:220: === RUN TestGetListenerSocketActivation5484 --- PASS: TestGetListenerSocketActivation (0.00s)5485 PASS5486 5487--- PASS: TestGetListenerSocketActivation (0.01s)5488=== RUN TestServerStalledClientDoesNotBlockShutdown5489=== PAUSE TestServerStalledClientDoesNotBlockShutdown5490=== RUN TestServerBacksOffOnAcceptErrors5491=== PAUSE TestServerBacksOffOnAcceptErrors5492=== RUN TestDrainIsolatesPoisonPath5493=== PAUSE TestDrainIsolatesPoisonPath5494=== RUN TestRunNotBlockedByPoisonHead5495=== PAUSE TestRunNotBlockedByPoisonHead5496=== RUN TestDrainGivesUpWhenServerDown5497=== PAUSE TestDrainGivesUpWhenServerDown5498=== RUN TestFailedPathPrunedByLaterClosure5499=== PAUSE TestFailedPathPrunedByLaterClosure5500=== RUN TestWorkerUploadsAndRemoves5501=== PAUSE TestWorkerUploadsAndRemoves5502=== RUN TestWorkerSkipsGCdPaths5503=== PAUSE TestWorkerSkipsGCdPaths5504=== RUN TestWorkerPrunesClosureDeps5505=== PAUSE TestWorkerPrunesClosureDeps5506=== RUN TestDrainTimeout5507=== PAUSE TestDrainTimeout5508=== RUN TestWorkerRemoveFailureIsNotProgress5509=== PAUSE TestWorkerRemoveFailureIsNotProgress5510=== RUN TestWorkerKeepsPathItCannotStat5511=== PAUSE TestWorkerKeepsPathItCannotStat5512=== CONT TestSendPathsEmpty5513=== CONT TestFailedPathPrunedByLaterClosure5514--- PASS: TestSendPathsEmpty (0.00s)5515=== CONT TestWorkerKeepsPathItCannotStat5516=== CONT TestWorkerRemoveFailureIsNotProgress5517=== CONT TestDrainIsolatesPoisonPath5518=== CONT TestServerBacksOffOnAcceptErrors5519=== CONT TestServerStalledClientDoesNotBlockShutdown55202000/01/01 00:00:00 ERROR Accept failed error="too many open files"5521=== CONT TestQueueRemoveLargeClosure55222000/01/01 00:00:00 ERROR Accept failed error="too many open files"55232000/01/01 00:00:00 ERROR Accept failed error="too many open files"55242000/01/01 00:00:00 ERROR Accept failed error="too many open files"55252000/01/01 00:00:00 ERROR Accept failed error="too many open files"55262000/01/01 00:00:00 ERROR Accept failed error="too many open files"55272000/01/01 00:00:00 ERROR Accept failed error="too many open files"55282000/01/01 00:00:00 ERROR Accept failed error="too many open files"5529=== CONT TestQueueConcurrentWriters5530=== CONT TestQueueFetchRemoveLifecycle5531=== CONT TestQueueRetryMovesToBack5532=== CONT TestWorkerUploadsAndRemoves5533=== CONT TestQueueFetchBatchLimit5534=== CONT TestDrainTimeout5535=== CONT TestQueueRemove5536=== CONT TestWorkerPrunesClosureDeps5537=== CONT TestQueueDeduplication5538=== CONT TestWorkerSkipsGCdPaths5539=== CONT TestQueueEnqueueAndFetch5540=== CONT TestRunNotBlockedByPoisonHead5541=== CONT TestDrainGivesUpWhenServerDown5542--- PASS: TestServerBacksOffOnAcceptErrors (0.00s)55432026/11/12 00:43:33 INFO Upload queue status pending=255442026/11/12 00:43:33 INFO Upload queue status pending=255452026/11/12 00:43:33 INFO Upload queue status pending=355462026/11/12 00:43:33 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerSkipsGCdPaths387724517/002/nonexistent55472026/11/12 00:43:33 INFO Uploading batch count=155482026/11/12 00:43:33 ERROR Upload failed error="upload failed" count=155492026/11/12 00:43:33 INFO Uploading batch count=155502026/11/12 00:43:33 ERROR Upload failed error="upload failed" count=155512000/01/01 00:00:00 INFO Uploading batch count=255522026/11/12 00:43:33 INFO Uploading batch count=255532026/11/12 00:43:33 INFO Upload queue status pending=255542026/11/12 00:43:33 INFO Uploading batch count=255552026/11/12 00:43:33 ERROR Upload failed error="upload failed" count=255562026/11/12 00:43:33 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown4083131521/002/a55572000/01/01 00:00:00 ERROR Upload failed error="context deadline exceeded" count=255582026/11/12 00:43:33 INFO Uploading batch count=455592026/11/12 00:43:33 ERROR Upload failed error="upload failed" count=455602026/11/12 00:43:33 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat3597334735/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat3597334735/002/locked/aaa: permission denied"55612026/11/12 00:43:33 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown4083131521/002/b55622026/11/12 00:43:33 INFO Uploading batch count=155632026/11/12 00:43:33 INFO Uploading batch count=155642000/01/01 00:00:00 ERROR Drain finished with paths left in queue remaining=45565--- PASS: TestQueueRetryMovesToBack (0.02s)55662026/11/12 00:43:33 INFO Uploading batch count=155672026/11/12 00:43:33 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress3696480838/002/nonexistent55682026/11/12 00:43:33 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=15569--- PASS: TestQueueEnqueueAndFetch (0.02s)5570--- PASS: TestQueueFetchBatchLimit (0.02s)55712026/11/12 00:43:33 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainIsolatesPoisonPath3800279514/002/bbb55722026/11/12 00:43:33 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat3597334735/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat3597334735/002/locked/aaa: permission denied"55732026/11/12 00:43:33 INFO Uploading batch count=255742026/11/12 00:43:33 INFO Uploading batch count=155752026/11/12 00:43:33 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress3696480838/002/nonexistent55762026/11/12 00:43:33 ERROR Upload failed error="upload failed" count=255772026/11/12 00:43:33 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown4083131521/002/c55782026/11/12 00:43:33 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=15579--- PASS: TestQueueFetchRemoveLifecycle (0.02s)55802026/11/12 00:43:33 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress3696480838/002/nonexistent5581--- PASS: TestQueueDeduplication (0.02s)55822026/11/12 00:43:33 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=155832026/11/12 00:43:33 ERROR Drain finished with paths left in queue remaining=15584--- PASS: TestQueueRemove (0.02s)55852026/11/12 00:43:33 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat3597334735/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat3597334735/002/locked/aaa: permission denied"55862026/11/12 00:43:33 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown4083131521/002/d5587--- PASS: TestDrainTimeout (0.02s)55882026/11/12 00:43:33 ERROR Drain finished with paths left in queue remaining=15589--- PASS: TestWorkerRemoveFailureIsNotProgress (0.02s)55902026/11/12 00:43:33 INFO Uploading batch count=155912026/11/12 00:43:33 ERROR Upload failed error="upload failed" count=155922026/11/12 00:43:33 INFO Uploading batch count=255932026/11/12 00:43:33 ERROR Upload failed error="upload failed" count=255942026/11/12 00:43:33 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown4083131521/002/e55952026/11/12 00:43:33 INFO Uploading batch count=155962026/11/12 00:43:33 ERROR Upload failed error="upload failed" count=155972026/11/12 00:43:33 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown4083131521/002/f5598--- PASS: TestFailedPathPrunedByLaterClosure (0.02s)5599--- PASS: TestWorkerKeepsPathItCannotStat (0.02s)56002026/11/12 00:43:33 INFO Uploading batch count=156012026/11/12 00:43:33 ERROR Upload failed error="upload failed" count=156022026/11/12 00:43:33 ERROR Drain finished with paths left in queue remaining=1056032026/11/12 00:43:33 ERROR Drain finished with paths left in queue remaining=15604--- PASS: TestDrainIsolatesPoisonPath (0.03s)5605--- PASS: TestDrainGivesUpWhenServerDown (0.02s)5606--- PASS: TestWorkerUploadsAndRemoves (0.04s)5607--- PASS: TestWorkerSkipsGCdPaths (0.03s)5608--- PASS: TestWorkerPrunesClosureDeps (0.04s)5609--- PASS: TestQueueRemoveLargeClosure (0.06s)56102026/11/12 00:43:33 ERROR Failed to decode request error="read unix /build/hook3638941461/test.sock->@: i/o timeout"56112026/11/12 00:43:33 ERROR Failed to write response error="write unix /build/hook3638941461/test.sock->@: i/o timeout"5612--- PASS: TestServerStalledClientDoesNotBlockShutdown (0.20s)5613--- PASS: TestQueueConcurrentWriters (0.22s)56142026/11/12 00:43:34 INFO Uploading batch count=156152026/11/12 00:43:34 INFO Uploading batch count=156162026/11/12 00:43:34 INFO Uploading batch count=156172026/11/12 00:43:34 ERROR Upload failed error="upload failed" count=156182026/11/12 00:43:34 INFO Uploading batch count=156192026/11/12 00:43:34 ERROR Upload failed error="upload failed" count=156202026/11/12 00:43:34 INFO Uploading batch count=156212026/11/12 00:43:34 ERROR Upload failed error="upload failed" count=156222026/11/12 00:43:34 INFO Uploading batch count=156232026/11/12 00:43:34 ERROR Upload failed error="upload failed" count=156242026/11/12 00:43:34 ERROR Drain finished with paths left in queue remaining=15625--- PASS: TestRunNotBlockedByPoisonHead (1.04s)5626PASS5627Running signing tests...5628=== RUN TestExternalSigner5629=== PAUSE TestExternalSigner5630=== RUN TestExternalSignerWrongPublicKey5631=== PAUSE TestExternalSignerWrongPublicKey5632=== RUN TestExternalSignerErrors5633=== PAUSE TestExternalSignerErrors5634=== RUN TestExternalSignerRestartsAfterCrash5635=== PAUSE TestExternalSignerRestartsAfterCrash5636=== RUN TestExternalSignerProgramErrorKeepsProcess5637=== PAUSE TestExternalSignerProgramErrorKeepsProcess5638=== RUN TestExternalSignerClose5639=== PAUSE TestExternalSignerClose5640=== RUN TestExternalSignerRejectsInvalidPublicKey5641=== PAUSE TestExternalSignerRejectsInvalidPublicKey5642=== RUN TestExternalSignerCancellation5643=== PAUSE TestExternalSignerCancellation5644=== RUN TestGenerateFingerprint5645=== PAUSE TestGenerateFingerprint5646=== RUN TestParseSigningKey5647=== PAUSE TestParseSigningKey5648=== RUN TestSign5649=== PAUSE TestSign5650=== RUN TestSignBatch5651=== PAUSE TestSignBatch5652=== RUN TestNilKeyPublicKey5653=== PAUSE TestNilKeyPublicKey5654=== RUN TestSignErrors5655=== PAUSE TestSignErrors5656=== CONT TestExternalSigner5657=== CONT TestExternalSignerCancellation5658=== CONT TestExternalSignerProgramErrorKeepsProcess5659=== CONT TestExternalSignerClose5660=== CONT TestExternalSignerErrors5661=== CONT TestExternalSignerRestartsAfterCrash5662=== CONT TestExternalSignerWrongPublicKey5663=== CONT TestSignBatch5664=== CONT TestSignErrors5665=== CONT TestNilKeyPublicKey5666=== CONT TestParseSigningKey5667=== CONT TestSign5668=== CONT TestGenerateFingerprint5669=== RUN TestGenerateFingerprint/basic_with_references5670=== CONT TestExternalSignerRejectsInvalidPublicKey5671=== RUN TestExternalSignerRejectsInvalidPublicKey/#005672=== RUN TestExternalSignerErrors/null5673--- PASS: TestNilKeyPublicKey (0.00s)5674=== RUN TestParseSigningKey/valid_32-byte_key5675=== PAUSE TestGenerateFingerprint/basic_with_references5676=== PAUSE TestParseSigningKey/valid_32-byte_key5677=== RUN TestParseSigningKey/valid_32-byte_key_with_different_name5678=== PAUSE TestParseSigningKey/valid_32-byte_key_with_different_name5679=== RUN TestGenerateFingerprint/no_references5680=== PAUSE TestGenerateFingerprint/no_references5681=== PAUSE TestExternalSignerRejectsInvalidPublicKey/#005682=== PAUSE TestExternalSignerErrors/null5683=== RUN TestSignErrors/nil_key5684=== PAUSE TestSignErrors/nil_key5685=== RUN TestSignErrors/nil_narinfo5686=== PAUSE TestSignErrors/nil_narinfo5687=== RUN TestSignErrors/invalid_narinfo5688=== RUN TestParseSigningKey/no_colon5689=== PAUSE TestParseSigningKey/no_colon5690=== PAUSE TestSignErrors/invalid_narinfo5691=== RUN TestExternalSignerRejectsInvalidPublicKey/no-colon5692=== PAUSE TestExternalSignerRejectsInvalidPublicKey/no-colon5693=== RUN TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5694=== PAUSE TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5695=== RUN TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5696=== PAUSE TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5697=== RUN TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5698=== PAUSE TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5699=== CONT TestExternalSignerRejectsInvalidPublicKey/#005700=== CONT TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5701=== CONT TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5702=== RUN TestGenerateFingerprint/unsorted_references_get_sorted5703=== CONT TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5704=== RUN TestExternalSignerErrors/missing_signatures5705=== PAUSE TestExternalSignerErrors/missing_signatures5706--- PASS: TestSignBatch (0.00s)5707=== RUN TestParseSigningKey/empty_name5708=== PAUSE TestParseSigningKey/empty_name5709=== RUN TestParseSigningKey/invalid_base645710=== PAUSE TestParseSigningKey/invalid_base645711--- PASS: TestSign (0.00s)5712=== RUN TestParseSigningKey/wrong_length5713--- PASS: TestExternalSignerProgramErrorKeepsProcess (0.00s)5714=== PAUSE TestParseSigningKey/wrong_length5715=== CONT TestParseSigningKey/valid_32-byte_key5716=== CONT TestParseSigningKey/wrong_length5717=== PAUSE TestGenerateFingerprint/unsorted_references_get_sorted5718=== RUN TestGenerateFingerprint/invalid_nar_hash_prefix5719=== RUN TestSignErrors/canceled5720=== RUN TestExternalSignerErrors/missing_signature_name5721=== PAUSE TestExternalSignerErrors/missing_signature_name5722=== RUN TestExternalSignerErrors/wrong_id5723=== PAUSE TestExternalSignerErrors/wrong_id5724--- PASS: TestExternalSignerClose (0.00s)5725=== PAUSE TestSignErrors/canceled5726=== CONT TestSignErrors/nil_key5727=== CONT TestSignErrors/nil_narinfo5728=== RUN TestExternalSignerErrors/too_few_signatures5729=== CONT TestSignErrors/canceled5730=== PAUSE TestGenerateFingerprint/invalid_nar_hash_prefix5731=== CONT TestParseSigningKey/empty_name5732=== CONT TestParseSigningKey/invalid_base645733=== CONT TestParseSigningKey/no_colon5734=== CONT TestParseSigningKey/valid_32-byte_key_with_different_name5735--- PASS: TestExternalSignerWrongPublicKey (0.00s)5736=== CONT TestExternalSignerRejectsInvalidPublicKey/no-colon5737=== CONT TestSignErrors/invalid_narinfo5738--- PASS: TestParseSigningKey (0.00s)5739 --- PASS: TestParseSigningKey/valid_32-byte_key (0.00s)5740 --- PASS: TestParseSigningKey/wrong_length (0.00s)5741 --- PASS: TestParseSigningKey/empty_name (0.00s)5742 --- PASS: TestParseSigningKey/invalid_base64 (0.00s)5743 --- PASS: TestParseSigningKey/no_colon (0.00s)5744 --- PASS: TestParseSigningKey/valid_32-byte_key_with_different_name (0.00s)5745--- PASS: TestExternalSignerRejectsInvalidPublicKey (0.00s)5746 --- PASS: TestExternalSignerRejectsInvalidPublicKey/#00 (0.00s)5747 --- PASS: TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= (0.00s)5748 --- PASS: TestExternalSignerRejectsInvalidPublicKey/test-key:YQ== (0.00s)5749 --- PASS: TestExternalSignerRejectsInvalidPublicKey/test-key:invalid! (0.00s)5750 --- PASS: TestExternalSignerRejectsInvalidPublicKey/no-colon (0.00s)5751--- PASS: TestSignErrors (0.00s)5752 --- PASS: TestSignErrors/nil_key (0.00s)5753 --- PASS: TestSignErrors/nil_narinfo (0.00s)5754 --- PASS: TestSignErrors/canceled (0.00s)5755 --- PASS: TestSignErrors/invalid_narinfo (0.00s)5756=== PAUSE TestExternalSignerErrors/too_few_signatures5757=== RUN TestExternalSignerErrors/wrong_signature_name5758=== RUN TestGenerateFingerprint/invalid_nar_hash_length5759=== PAUSE TestGenerateFingerprint/invalid_nar_hash_length5760=== PAUSE TestExternalSignerErrors/wrong_signature_name5761=== RUN TestGenerateFingerprint/invalid_store_path_prefix5762=== PAUSE TestGenerateFingerprint/invalid_store_path_prefix5763=== RUN TestExternalSignerErrors/invalid_signature_base645764=== PAUSE TestExternalSignerErrors/invalid_signature_base645765=== RUN TestExternalSignerErrors/program_error5766=== PAUSE TestExternalSignerErrors/program_error5767=== RUN TestExternalSignerErrors/exit_at_startup5768=== PAUSE TestExternalSignerErrors/exit_at_startup5769=== RUN TestExternalSignerErrors/invalid_JSON5770=== PAUSE TestExternalSignerErrors/invalid_JSON5771=== RUN TestExternalSignerErrors/wrong_shape5772=== PAUSE TestExternalSignerErrors/wrong_shape5773=== RUN TestExternalSignerErrors/null_signatures5774=== PAUSE TestExternalSignerErrors/null_signatures5775=== RUN TestExternalSignerErrors/exit_on_request5776=== PAUSE TestExternalSignerErrors/exit_on_request5777=== RUN TestExternalSignerErrors/wrong_signatures_shape5778=== PAUSE TestExternalSignerErrors/wrong_signatures_shape5779=== RUN TestExternalSignerErrors/unrequested_key5780=== RUN TestGenerateFingerprint/invalid_reference_prefix5781=== PAUSE TestGenerateFingerprint/invalid_reference_prefix5782=== CONT TestGenerateFingerprint/basic_with_references5783=== CONT TestGenerateFingerprint/no_references5784=== PAUSE TestExternalSignerErrors/unrequested_key5785=== RUN TestExternalSignerErrors/wrong_signature_length5786=== PAUSE TestExternalSignerErrors/wrong_signature_length5787=== CONT TestExternalSignerErrors/null5788=== CONT TestExternalSignerErrors/invalid_JSON5789=== CONT TestExternalSignerErrors/exit_on_request5790=== CONT TestGenerateFingerprint/invalid_nar_hash_length5791=== CONT TestExternalSignerErrors/wrong_signature_name5792=== CONT TestGenerateFingerprint/invalid_reference_prefix5793=== CONT TestGenerateFingerprint/invalid_store_path_prefix5794=== CONT TestGenerateFingerprint/unsorted_references_get_sorted5795=== CONT TestGenerateFingerprint/invalid_nar_hash_prefix5796--- PASS: TestGenerateFingerprint (0.00s)5797 --- PASS: TestGenerateFingerprint/basic_with_references (0.00s)5798 --- PASS: TestGenerateFingerprint/no_references (0.00s)5799 --- PASS: TestGenerateFingerprint/invalid_nar_hash_length (0.00s)5800 --- PASS: TestGenerateFingerprint/invalid_reference_prefix (0.00s)5801 --- PASS: TestGenerateFingerprint/invalid_store_path_prefix (0.00s)5802 --- PASS: TestGenerateFingerprint/unsorted_references_get_sorted (0.00s)5803 --- PASS: TestGenerateFingerprint/invalid_nar_hash_prefix (0.00s)5804=== CONT TestExternalSignerErrors/exit_at_startup5805=== CONT TestExternalSignerErrors/program_error5806=== CONT TestExternalSignerErrors/invalid_signature_base645807=== CONT TestExternalSignerErrors/wrong_signatures_shape5808=== CONT TestExternalSignerErrors/wrong_signature_length5809=== CONT TestExternalSignerErrors/wrong_id5810=== CONT TestExternalSignerErrors/unrequested_key5811=== CONT TestExternalSignerErrors/too_few_signatures5812=== CONT TestExternalSignerErrors/null_signatures5813--- PASS: TestExternalSigner (0.01s)5814=== CONT TestExternalSignerErrors/missing_signature_name5815=== CONT TestExternalSignerErrors/wrong_shape5816=== CONT TestExternalSignerErrors/missing_signatures5817--- PASS: TestExternalSignerRestartsAfterCrash (0.01s)5818--- PASS: TestExternalSignerErrors (0.01s)5819 --- PASS: TestExternalSignerErrors/wrong_signature_name (0.00s)5820 --- PASS: TestExternalSignerErrors/exit_at_startup (0.00s)5821 --- PASS: TestExternalSignerErrors/invalid_JSON (0.00s)5822 --- PASS: TestExternalSignerErrors/null (0.00s)5823 --- PASS: TestExternalSignerErrors/wrong_signatures_shape (0.00s)5824 --- PASS: TestExternalSignerErrors/exit_on_request (0.01s)5825 --- PASS: TestExternalSignerErrors/wrong_signature_length (0.00s)5826 --- PASS: TestExternalSignerErrors/program_error (0.01s)5827 --- PASS: TestExternalSignerErrors/invalid_signature_base64 (0.01s)5828 --- PASS: TestExternalSignerErrors/wrong_id (0.01s)5829 --- PASS: TestExternalSignerErrors/missing_signature_name (0.00s)5830 --- PASS: TestExternalSignerErrors/too_few_signatures (0.00s)5831 --- PASS: TestExternalSignerErrors/missing_signatures (0.00s)5832 --- PASS: TestExternalSignerErrors/wrong_shape (0.00s)5833 --- PASS: TestExternalSignerErrors/null_signatures (0.00s)5834 --- PASS: TestExternalSignerErrors/unrequested_key (0.00s)5835--- PASS: TestExternalSignerCancellation (0.02s)5836PASS5837Running OpenBao signer tests...5838=== RUN TestSign5839=== PAUSE TestSign5840=== RUN TestSignErrors5841=== PAUSE TestSignErrors5842=== RUN TestServe5843=== PAUSE TestServe5844=== RUN TestServeRejectsGarbage5845=== PAUSE TestServeRejectsGarbage5846=== CONT TestSign5847=== CONT TestServe5848=== CONT TestSignErrors5849=== CONT TestServeRejectsGarbage5850=== RUN TestSignErrors/no_prefix5851--- PASS: TestServeRejectsGarbage (0.00s)5852=== PAUSE TestSignErrors/no_prefix5853=== RUN TestSignErrors/invalid_base645854--- PASS: TestServe (0.00s)5855=== PAUSE TestSignErrors/invalid_base645856=== RUN TestSignErrors/http_error5857=== PAUSE TestSignErrors/http_error5858=== RUN TestSignErrors/invalid_json5859=== PAUSE TestSignErrors/invalid_json5860=== CONT TestSignErrors/no_prefix5861=== CONT TestSignErrors/http_error5862=== CONT TestSignErrors/invalid_base645863=== CONT TestSignErrors/invalid_json5864--- PASS: TestSign (0.00s)5865--- PASS: TestSignErrors (0.00s)5866 --- PASS: TestSignErrors/http_error (0.00s)5867 --- PASS: TestSignErrors/no_prefix (0.00s)5868 --- PASS: TestSignErrors/invalid_json (0.00s)5869 --- PASS: TestSignErrors/invalid_base64 (0.00s)5870PASS