niks3-go-unit-tests
checks.x86_64-linux.go-unit-tests
· build #510
· raw
1tribuchet: building on jamie2Running client tests...3=== RUN TestDoServerRequestAttachesToken4=== PAUSE TestDoServerRequestAttachesToken5=== RUN TestUploadBuildLog_FileBodyReplayedOnRetry6=== PAUSE TestUploadBuildLog_FileBodyReplayedOnRetry7=== RUN TestRegisterUploadedObjectReusesConnections8=== PAUSE TestRegisterUploadedObjectReusesConnections9=== RUN TestRunGarbageCollection_FinishedOnAnotherReplica10=== PAUSE TestRunGarbageCollection_FinishedOnAnotherReplica11=== RUN TestRunGarbageCollection_NotFoundAfterLocalRun12=== PAUSE TestRunGarbageCollection_NotFoundAfterLocalRun13=== RUN TestCaseHackSuffix14=== PAUSE TestCaseHackSuffix15=== RUN TestFilterOversizedClosures16=== PAUSE TestFilterOversizedClosures17=== RUN TestUploadMultipart_PartsInParallel18=== PAUSE TestUploadMultipart_PartsInParallel19=== RUN TestUploadMultipart_ReturnsAfterDroppedBody20=== PAUSE TestUploadMultipart_ReturnsAfterDroppedBody21=== RUN TestUploadMultipart_FailedProducerIsNotCompleted22=== PAUSE TestUploadMultipart_FailedProducerIsNotCompleted23=== RUN TestPartSizeForNAR24=== PAUSE TestPartSizeForNAR25=== RUN TestDumpPathCaseHackMatchesNix26--- PASS: TestDumpPathCaseHackMatchesNix (0.03s)27=== RUN TestDumpPathCaseHackCollision28--- PASS: TestDumpPathCaseHackCollision (0.00s)29=== RUN TestDumpPathMatchesNix30=== PAUSE TestDumpPathMatchesNix31=== RUN TestDumpPathSingleFile32=== PAUSE TestDumpPathSingleFile33=== RUN TestDumpPathWriterError34=== PAUSE TestDumpPathWriterError35=== RUN TestEncodeNixBase3236=== PAUSE TestEncodeNixBase3237=== RUN TestEncodeNixBase32WithRealHash38=== PAUSE TestEncodeNixBase32WithRealHash39=== RUN TestConvertHashToNix3240=== PAUSE TestConvertHashToNix3241=== RUN TestGetStorePathHash42=== PAUSE TestGetStorePathHash43=== RUN TestPathInfoHashCompatibility44=== PAUSE TestPathInfoHashCompatibility45=== RUN TestParsePathInfoJSON46=== PAUSE TestParsePathInfoJSON47=== RUN TestParsePathInfoJSONMultiplePaths48=== PAUSE TestParsePathInfoJSONMultiplePaths49=== RUN TestPathInfoCACompatibility50=== PAUSE TestPathInfoCACompatibility51=== RUN TestRateLimiterFeedback52=== PAUSE TestRateLimiterFeedback53=== RUN TestRateLimiterFeedback_400DoesNotCountAsSuccess54=== PAUSE TestRateLimiterFeedback_400DoesNotCountAsSuccess55=== RUN TestRegisterUploadedObject_BoundedAgainstSilentServer56=== PAUSE TestRegisterUploadedObject_BoundedAgainstSilentServer57=== RUN TestRegisterUploadedObject_OutlivesPush58=== PAUSE TestRegisterUploadedObject_OutlivesPush59=== RUN TestResolveStorePath60=== PAUSE TestResolveStorePath61=== RUN TestDoWithRetry_BodyReplayedViaGetBody62=== PAUSE TestDoWithRetry_BodyReplayedViaGetBody63=== RUN TestRetryTimeoutOutlastsMaxRetries64=== PAUSE TestRetryTimeoutOutlastsMaxRetries65=== RUN TestDoWithRetry_FinalResponseBodyReadable66=== PAUSE TestDoWithRetry_FinalResponseBodyReadable67=== RUN TestShellSplit68=== PAUSE TestShellSplit69=== RUN TestShellSplitErrors70=== PAUSE TestShellSplitErrors71=== RUN TestStreamPushReportsEveryPath72=== PAUSE TestStreamPushReportsEveryPath73=== RUN TestStreamPushBatchesUnderLoad74=== PAUSE TestStreamPushBatchesUnderLoad75=== RUN TestStreamPushIsolatesFailures76=== PAUSE TestStreamPushIsolatesFailures77=== RUN TestStreamPushGivesUpOnDeadServer78=== PAUSE TestStreamPushGivesUpOnDeadServer79=== RUN TestStreamPushRequestLine80=== PAUSE TestStreamPushRequestLine81=== RUN TestStreamPushReportsSignatures82=== PAUSE TestStreamPushReportsSignatures83=== RUN TestStreamPushReportsSkippedPaths84=== PAUSE TestStreamPushReportsSkippedPaths85=== RUN TestClientSignaturesByStorePath86=== PAUSE TestClientSignaturesByStorePath87=== RUN TestStreamPushAnswersEveryLineWhenCanceled88=== PAUSE TestStreamPushAnswersEveryLineWhenCanceled89=== RUN TestStreamPushStopsWhenCanceled90=== PAUSE TestStreamPushStopsWhenCanceled91=== RUN TestSetClientTLS92=== PAUSE TestSetClientTLS93=== RUN TestSetClientTLSDoesNotMutateDefaultTransport94=== PAUSE TestSetClientTLSDoesNotMutateDefaultTransport95=== RUN TestSetClientTLSErrors96=== PAUSE TestSetClientTLSErrors97=== RUN TestStaticToken98=== PAUSE TestStaticToken99=== RUN TestFileTokenReadsAndCaches100=== PAUSE TestFileTokenReadsAndCaches101=== RUN TestFileTokenMissing102=== PAUSE TestFileTokenMissing103=== RUN TestFileTokenEmpty104=== PAUSE TestFileTokenEmpty105=== RUN TestScriptTokenNoExpiryRerunsEveryCall106=== PAUSE TestScriptTokenNoExpiryRerunsEveryCall107=== RUN TestScriptTokenCachesUntilRefresh108=== PAUSE TestScriptTokenCachesUntilRefresh109=== RUN TestScriptTokenEmptyToken110=== PAUSE TestScriptTokenEmptyToken111=== RUN TestScriptTokenBadJSON112=== PAUSE TestScriptTokenBadJSON113=== RUN TestScriptTokenScriptFails114=== PAUSE TestScriptTokenScriptFails115=== RUN TestScriptTokenEmptyCommand116=== PAUSE TestScriptTokenEmptyCommand117=== CONT TestDoServerRequestAttachesToken118=== CONT TestRetryTimeoutOutlastsMaxRetries119=== CONT TestSetClientTLS120=== CONT TestScriptTokenEmptyCommand121=== CONT TestScriptTokenScriptFails122=== CONT TestScriptTokenBadJSON123=== CONT TestScriptTokenEmptyToken124=== CONT TestScriptTokenCachesUntilRefresh125=== CONT TestScriptTokenNoExpiryRerunsEveryCall126=== CONT TestFileTokenEmpty127=== CONT TestFileTokenMissing128=== CONT TestFileTokenReadsAndCaches129=== CONT TestStreamPushGivesUpOnDeadServer130=== CONT TestShellSplitErrors131=== CONT TestStaticToken132=== CONT TestDoWithRetry_FinalResponseBodyReadable133=== CONT TestEncodeNixBase32134=== CONT TestSetClientTLSErrors135=== CONT TestStreamPushAnswersEveryLineWhenCanceled136=== CONT TestSetClientTLSDoesNotMutateDefaultTransport137=== CONT TestClientSignaturesByStorePath138=== CONT TestStreamPushReportsEveryPath139=== CONT TestStreamPushReportsSkippedPaths140=== CONT TestStreamPushIsolatesFailures141=== CONT TestStreamPushReportsSignatures142=== CONT TestStreamPushBatchesUnderLoad143--- PASS: TestScriptTokenEmptyCommand (0.00s)144--- PASS: TestFileTokenMissing (0.00s)145=== CONT TestStreamPushRequestLine146=== CONT TestShellSplit147=== RUN TestEncodeNixBase32/test_string_hash148=== CONT TestResolveStorePath149=== CONT TestStreamPushStopsWhenCanceled150=== CONT TestDoWithRetry_BodyReplayedViaGetBody1512026/11/12 16:57:00 ERROR Upload failed error="connection refused" count=201522026/11/12 16:57:00 ERROR Server seems unavailable, giving up on batch untried=17153--- PASS: TestFileTokenEmpty (0.00s)154--- PASS: TestScriptTokenBadJSON (0.00s)155--- PASS: TestStaticToken (0.00s)156--- PASS: TestShellSplitErrors (0.00s)157--- PASS: TestClientSignaturesByStorePath (0.00s)158--- PASS: TestScriptTokenScriptFails (0.00s)159--- PASS: TestScriptTokenEmptyToken (0.01s)160=== CONT TestUploadMultipart_PartsInParallel161=== PAUSE TestEncodeNixBase32/test_string_hash162=== RUN TestEncodeNixBase32/empty_input163=== RUN TestStreamPushBatchesUnderLoad/oneAtATime=false164=== PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=false165--- PASS: TestShellSplit (0.00s)166=== RUN TestStreamPushBatchesUnderLoad/oneAtATime=true167=== CONT TestRegisterUploadedObject_BoundedAgainstSilentServer1682026/11/12 16:57:00 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:41577169=== CONT TestRegisterUploadedObject_OutlivesPush1702000/01/01 00:00:00 ERROR Upload failed error="context canceled" count=2171=== PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=true1722026/11/12 16:57:00 ERROR Upload failed error=boom count=11732026/11/12 16:57:00 WARN Rate limiter enabled after throttle name=server-test rate=5174--- PASS: TestStreamPushStopsWhenCanceled (0.00s)1752026/11/12 16:57:00 ERROR Upload failed error="bad path" count=3176--- PASS: TestFileTokenReadsAndCaches (0.00s)177=== CONT TestRateLimiterFeedback_400DoesNotCountAsSuccess178--- PASS: TestStreamPushReportsSkippedPaths (0.00s)179--- PASS: TestStreamPushAnswersEveryLineWhenCanceled (0.00s)180=== CONT TestDumpPathSingleFile181=== CONT TestRateLimiterFeedback1822026/11/12 16:57:00 WARN Rate limiter enabled after throttle name=server-test rate=5183=== RUN TestRateLimiterFeedback/429_enables_limiter1842026/11/12 16:57:00 ERROR Upload failed error=boom count=1185--- PASS: TestStreamPushIsolatesFailures (0.00s)1862026/11/12 16:57:00 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:32847187=== PAUSE TestRateLimiterFeedback/429_enables_limiter188=== CONT TestPathInfoCACompatibility189=== RUN TestPathInfoCACompatibility/null_ca_field190=== PAUSE TestPathInfoCACompatibility/null_ca_field191=== CONT TestConvertHashToNix32192=== RUN TestPathInfoCACompatibility/old_string_format_-_text193=== CONT TestEncodeNixBase32WithRealHash194=== PAUSE TestPathInfoCACompatibility/old_string_format_-_text195--- PASS: TestStreamPushReportsSignatures (0.00s)196--- PASS: TestStreamPushReportsEveryPath (0.00s)197=== CONT TestDumpPathMatchesNix198=== RUN TestConvertHashToNix32/SRI_format_to_Nix32199=== CONT TestUploadMultipart_FailedProducerIsNotCompleted200=== RUN TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part2012026/11/12 16:57:00 WARN Rate limiter enabled after throttle name=server-test rate=5202=== CONT TestPartSizeForNAR2032026/11/12 16:57:00 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:36017204=== RUN TestPartSizeForNAR/zero_stays_at_minimum2052026/11/12 16:57:00 WARN Rate limiter backed off name=server-test rate=5206=== PAUSE TestPartSizeForNAR/zero_stays_at_minimum2072026/11/12 16:57:00 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:41577208=== PAUSE TestEncodeNixBase32/empty_input209=== CONT TestGetStorePathHash210=== CONT TestDumpPathWriterError211=== RUN TestSetClientTLS/rejects_connection_without_client_cert212=== RUN TestSetClientTLSErrors/missing_cert_file2132026/11/12 16:57:00 WARN Rate limiter backed off name=server-test rate=5214=== CONT TestParsePathInfoJSONMultiplePaths2152026/11/12 16:57:00 WARN Request returned retryable status, retrying attempt=2 backoff=0s status=503 url=http://127.0.0.1:36017216--- PASS: TestStreamPushGivesUpOnDeadServer (0.00s)217--- PASS: TestResolveStorePath (0.00s)218--- PASS: TestScriptTokenCachesUntilRefresh (0.01s)219--- PASS: TestEncodeNixBase32WithRealHash (0.00s)220--- PASS: TestDoServerRequestAttachesToken (0.04s)221=== CONT TestCaseHackSuffix222=== PAUSE TestConvertHashToNix32/SRI_format_to_Nix32223=== RUN TestConvertHashToNix32/already_Nix32_format224=== PAUSE TestConvertHashToNix32/already_Nix32_format225=== RUN TestConvertHashToNix32/invalid_format226=== RUN TestRateLimiterFeedback/503_enables_limiter227=== PAUSE TestRateLimiterFeedback/503_enables_limiter228=== CONT TestUploadBuildLog_FileBodyReplayedOnRetry229=== RUN TestRateLimiterFeedback/200_does_not_enable_limiter230=== PAUSE TestRateLimiterFeedback/200_does_not_enable_limiter231=== RUN TestRateLimiterFeedback/400_does_not_enable_limiter232=== RUN TestPathInfoCACompatibility/old_string_format_-_fixed_recursive233=== CONT TestUploadMultipart_ReturnsAfterDroppedBody234=== RUN TestPartSizeForNAR/small_stays_at_minimum235=== PAUSE TestPartSizeForNAR/small_stays_at_minimum236=== CONT TestRunGarbageCollection_NotFoundAfterLocalRun237=== CONT TestFilterOversizedClosures238=== RUN TestFilterOversizedClosures/no_limit_keeps_everything239=== PAUSE TestFilterOversizedClosures/no_limit_keeps_everything240=== RUN TestGetStorePathHash/valid_store_path241=== CONT TestParsePathInfoJSON242=== PAUSE TestSetClientTLS/rejects_connection_without_client_cert243=== PAUSE TestSetClientTLSErrors/missing_cert_file244=== RUN TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths245=== PAUSE TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths246=== RUN TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths247=== PAUSE TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths248--- PASS: TestDoWithRetry_FinalResponseBodyReadable (0.03s)249--- PASS: TestRegisterUploadedObject_OutlivesPush (0.03s)250=== PAUSE TestConvertHashToNix32/invalid_format2512026/11/12 16:57:00 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:41577252=== PAUSE TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part253=== PAUSE TestRateLimiterFeedback/400_does_not_enable_limiter254=== RUN TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary255=== PAUSE TestPathInfoCACompatibility/old_string_format_-_fixed_recursive256=== RUN TestPartSizeForNAR/80_GiB_fits_at_minimum257=== PAUSE TestPartSizeForNAR/80_GiB_fits_at_minimum258=== RUN TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped259=== RUN TestPartSizeForNAR/115_GiB_needs_larger_parts260=== PAUSE TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped261=== CONT TestRegisterUploadedObjectReusesConnections262=== RUN TestSetClientTLS/succeeds_with_client_cert_and_CA263=== PAUSE TestSetClientTLS/succeeds_with_client_cert_and_CA2642026/11/12 16:57:00 INFO Garbage collection started265=== RUN TestSetClientTLSErrors/missing_key_file266=== PAUSE TestSetClientTLSErrors/missing_key_file267=== CONT TestRunGarbageCollection_FinishedOnAnotherReplica268=== CONT TestStreamPushBatchesUnderLoad/oneAtATime=true269--- PASS: TestScriptTokenNoExpiryRerunsEveryCall (0.03s)2702026/11/12 16:57:00 WARN Rate limiter enabled after throttle name=s3-test rate=5271--- PASS: TestDoWithRetry_BodyReplayedViaGetBody (0.03s)2722026/11/12 16:57:00 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:45953273--- PASS: TestSetClientTLSDoesNotMutateDefaultTransport (0.03s)274--- PASS: TestUploadMultipart_ReturnsAfterDroppedBody (0.00s)275=== CONT TestPathInfoHashCompatibility276=== CONT TestEncodeNixBase32/test_string_hash277=== RUN TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)278=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)2792026/11/12 16:57:00 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:41577280=== RUN TestPathInfoHashCompatibility/old_string_format_with_colon2812026/11/12 16:57:00 INFO Garbage collection started282=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_colon283=== RUN TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI284=== PAUSE TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI285=== RUN TestPathInfoHashCompatibility/new_structured_format_with_sha512286=== PAUSE TestPathInfoHashCompatibility/new_structured_format_with_sha512287=== CONT TestStreamPushBatchesUnderLoad/oneAtATime=false288=== CONT TestEncodeNixBase32/empty_input289=== CONT TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths290=== CONT TestConvertHashToNix32/SRI_format_to_Nix32291=== CONT TestConvertHashToNix32/invalid_format292=== CONT TestConvertHashToNix32/already_Nix32_format293=== CONT TestRateLimiterFeedback/429_enables_limiter294=== CONT TestRateLimiterFeedback/400_does_not_enable_limiter295=== RUN TestPathInfoCACompatibility/new_structured_format_-_text296=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_text297=== RUN TestPathInfoCACompatibility/new_structured_format_-_nar_method298=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_nar_method299=== PAUSE TestPartSizeForNAR/115_GiB_needs_larger_parts300=== RUN TestPartSizeForNAR/1_TiB301=== PAUSE TestPartSizeForNAR/1_TiB302=== RUN TestPartSizeForNAR/5_TiB_S3_max_object303=== PAUSE TestPartSizeForNAR/5_TiB_S3_max_object304=== RUN TestPartSizeForNAR/capped_at_5_GiB305=== PAUSE TestPartSizeForNAR/capped_at_5_GiB3062026/11/12 16:57:00 WARN Rate limiter enabled after throttle name=server-test rate=5307=== RUN TestParsePathInfoJSON/Nix_format3082026/11/12 16:57:00 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=429 url=http://127.0.0.1:33537309=== PAUSE TestParsePathInfoJSON/Nix_format310=== RUN TestParsePathInfoJSON/Lix_format311=== PAUSE TestParsePathInfoJSON/Lix_format312=== RUN TestParsePathInfoJSON/empty_input313=== PAUSE TestParsePathInfoJSON/empty_input314=== RUN TestParsePathInfoJSON/whitespace_only3152026/11/12 16:57:00 WARN Rate limiter backed off name=server-test rate=5316=== PAUSE TestParsePathInfoJSON/whitespace_only317=== RUN TestParsePathInfoJSON/invalid_JSON318--- PASS: TestEncodeNixBase32 (0.03s)319 --- PASS: TestEncodeNixBase32/test_string_hash (0.00s)320 --- PASS: TestEncodeNixBase32/empty_input (0.00s)3212026/11/12 16:57:00 WARN Request returned retryable status, retrying attempt=3 backoff=1ms status=502 url=http://127.0.0.1:41577322--- PASS: TestRunGarbageCollection_NotFoundAfterLocalRun (0.00s)323=== PAUSE TestGetStorePathHash/valid_store_path324=== PAUSE TestParsePathInfoJSON/invalid_JSON3252026/11/12 16:57:00 INFO Garbage collection progress phase=running_on_another_replica failed_uploads_deleted=0 old_closures_deleted=0 objects_marked=0 objects_deleted=0 objects_failed=0326=== CONT TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)327=== CONT TestPathInfoCACompatibility/null_ca_field328=== CONT TestPathInfoCACompatibility/new_structured_format_-_text329=== RUN TestSetClientTLS/preserves_debug_logging_transport330=== RUN TestSetClientTLSErrors/missing_ca_file331=== PAUSE TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary332=== CONT TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths333=== CONT TestRateLimiterFeedback/200_does_not_enable_limiter334=== CONT TestRateLimiterFeedback/503_enables_limiter335=== RUN TestGetStorePathHash/basename_without_hyphen_should_error3362026/11/12 16:57:00 WARN Request returned retryable status, retrying attempt=4 backoff=1ms status=502 url=http://127.0.0.1:415773372026/11/12 16:57:00 WARN Garbage collection finished on another replica. Its statistics are not available here338--- PASS: TestConvertHashToNix32 (0.03s)339 --- PASS: TestConvertHashToNix32/SRI_format_to_Nix32 (0.00s)340 --- PASS: TestConvertHashToNix32/invalid_format (0.00s)341 --- PASS: TestConvertHashToNix32/already_Nix32_format (0.00s)3422026/11/12 16:57:00 WARN Rate limiter enabled after throttle name=server-test rate=5343=== RUN TestFilterOversizedClosures/all_closures_skipped3442026/11/12 16:57:00 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:36071345=== CONT TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI3462026/11/12 16:57:00 WARN Request returned retryable status, retrying attempt=5 backoff=1ms status=502 url=http://127.0.0.1:41577347=== CONT TestPathInfoHashCompatibility/old_string_format_with_colon3482026/11/12 16:57:00 WARN Rate limiter backed off name=server-test rate=5349=== CONT TestPartSizeForNAR/1_TiB350=== CONT TestPathInfoHashCompatibility/new_structured_format_with_sha512351=== PAUSE TestSetClientTLSErrors/missing_ca_file352=== PAUSE TestSetClientTLS/preserves_debug_logging_transport353=== CONT TestPathInfoCACompatibility/old_string_format_-_text354=== CONT TestPathInfoCACompatibility/new_structured_format_-_nar_method355=== CONT TestPathInfoCACompatibility/old_string_format_-_fixed_recursive356=== PAUSE TestGetStorePathHash/basename_without_hyphen_should_error357=== CONT TestPartSizeForNAR/zero_stays_at_minimum358=== CONT TestPartSizeForNAR/5_TiB_S3_max_object359--- PASS: TestStreamPushBatchesUnderLoad (0.00s)360 --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=true (0.00s)361 --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=false (0.02s)362=== CONT TestPartSizeForNAR/capped_at_5_GiB363=== PAUSE TestFilterOversizedClosures/all_closures_skipped364=== CONT TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary365=== CONT TestPartSizeForNAR/80_GiB_fits_at_minimum366=== CONT TestSetClientTLS/succeeds_with_client_cert_and_CA367=== CONT TestPartSizeForNAR/115_GiB_needs_larger_parts368=== CONT TestFilterOversizedClosures/no_limit_keeps_everything369=== CONT TestPartSizeForNAR/small_stays_at_minimum3702026/11/12 16:57:00 WARN Request returned retryable status, retrying attempt=6 backoff=1ms status=502 url=http://127.0.0.1:41577371=== RUN TestSetClientTLSErrors/invalid_ca_file372=== PAUSE TestSetClientTLSErrors/invalid_ca_file373=== CONT TestSetClientTLSErrors/missing_cert_file374=== CONT TestSetClientTLSErrors/missing_ca_file375=== CONT TestSetClientTLSErrors/invalid_ca_file376=== CONT TestParsePathInfoJSON/empty_input377=== CONT TestParsePathInfoJSON/invalid_JSON378=== RUN TestGetStorePathHash/hash_with_invalid_characters_should_error379=== PAUSE TestGetStorePathHash/hash_with_invalid_characters_should_error380=== RUN TestGetStorePathHash/hash_with_wrong_length_should_error381=== PAUSE TestGetStorePathHash/hash_with_wrong_length_should_error382=== CONT TestParsePathInfoJSON/whitespace_only383=== CONT TestParsePathInfoJSON/Lix_format384=== CONT TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part385--- PASS: TestUploadBuildLog_FileBodyReplayedOnRetry (0.00s)386=== CONT TestSetClientTLS/rejects_connection_without_client_cert387=== CONT TestSetClientTLS/preserves_debug_logging_transport388=== CONT TestFilterOversizedClosures/all_closures_skipped389=== CONT TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped390=== CONT TestParsePathInfoJSON/Nix_format391=== CONT TestSetClientTLSErrors/missing_key_file392=== CONT TestGetStorePathHash/valid_store_path393=== CONT TestGetStorePathHash/hash_with_wrong_length_should_error394=== CONT TestGetStorePathHash/hash_with_invalid_characters_should_error395=== CONT TestGetStorePathHash/basename_without_hyphen_should_error396--- PASS: TestRunGarbageCollection_FinishedOnAnotherReplica (0.02s)3972026/11/12 16:57:00 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper nar_size=100 max_nar_size=50398--- PASS: TestParsePathInfoJSONMultiplePaths (0.00s)399 --- PASS: TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths (0.00s)400 --- PASS: TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths (0.00s)4012026/11/12 16:57:00 WARN Request returned retryable status, retrying attempt=7 backoff=1ms status=502 url=http://127.0.0.1:41577402--- PASS: TestPartSizeForNAR (0.00s)403 --- PASS: TestPartSizeForNAR/1_TiB (0.00s)404 --- PASS: TestPartSizeForNAR/zero_stays_at_minimum (0.00s)405 --- PASS: TestPartSizeForNAR/5_TiB_S3_max_object (0.00s)406 --- PASS: TestPartSizeForNAR/capped_at_5_GiB (0.00s)407 --- PASS: TestPartSizeForNAR/80_GiB_fits_at_minimum (0.00s)408 --- PASS: TestPartSizeForNAR/115_GiB_needs_larger_parts (0.00s)409 --- PASS: TestPartSizeForNAR/small_stays_at_minimum (0.00s)410--- PASS: TestRateLimiterFeedback (0.03s)411 --- PASS: TestRateLimiterFeedback/400_does_not_enable_limiter (0.00s)412 --- PASS: TestRateLimiterFeedback/429_enables_limiter (0.00s)413 --- PASS: TestRateLimiterFeedback/200_does_not_enable_limiter (0.00s)414 --- PASS: TestRateLimiterFeedback/503_enables_limiter (0.00s)415--- PASS: TestGetStorePathHash (0.02s)416 --- PASS: TestGetStorePathHash/valid_store_path (0.00s)417 --- PASS: TestGetStorePathHash/hash_with_wrong_length_should_error (0.00s)418 --- PASS: TestGetStorePathHash/hash_with_invalid_characters_should_error (0.00s)419 --- PASS: TestGetStorePathHash/basename_without_hyphen_should_error (0.00s)4202026/11/12 16:57:00 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb-vm-image nar_size=5000 max_nar_size=2000421--- PASS: TestParsePathInfoJSON (0.00s)422 --- PASS: TestParsePathInfoJSON/empty_input (0.00s)423 --- PASS: TestParsePathInfoJSON/invalid_JSON (0.00s)424 --- PASS: TestParsePathInfoJSON/whitespace_only (0.00s)425 --- PASS: TestParsePathInfoJSON/Lix_format (0.00s)426 --- PASS: TestParsePathInfoJSON/Nix_format (0.00s)427--- PASS: TestFilterOversizedClosures (0.02s)428 --- PASS: TestFilterOversizedClosures/no_limit_keeps_everything (0.00s)429 --- PASS: TestFilterOversizedClosures/all_closures_skipped (0.00s)430 --- PASS: TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped (0.00s)431--- PASS: TestPathInfoHashCompatibility (0.00s)432 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI) (0.00s)433 --- PASS: TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI (0.00s)434 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_colon (0.00s)435 --- PASS: TestPathInfoHashCompatibility/new_structured_format_with_sha512 (0.00s)436--- PASS: TestUploadMultipart_FailedProducerIsNotCompleted (0.01s)437 --- PASS: TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary (0.00s)438 --- PASS: TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part (0.00s)439--- PASS: TestPathInfoCACompatibility (0.03s)440 --- PASS: TestPathInfoCACompatibility/null_ca_field (0.00s)441 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_text (0.00s)442 --- PASS: TestPathInfoCACompatibility/old_string_format_-_text (0.00s)443 --- PASS: TestPathInfoCACompatibility/old_string_format_-_fixed_recursive (0.00s)444 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_nar_method (0.00s)445--- PASS: TestSetClientTLSErrors (0.05s)446 --- PASS: TestSetClientTLSErrors/missing_cert_file (0.00s)447 --- PASS: TestSetClientTLSErrors/missing_ca_file (0.00s)448 --- PASS: TestSetClientTLSErrors/invalid_ca_file (0.00s)449 --- PASS: TestSetClientTLSErrors/missing_key_file (0.00s)4502026/11/12 16:57:00 WARN Request returned retryable status, retrying attempt=8 backoff=1ms status=502 url=http://127.0.0.1:415774512026/11/12 16:57:00 WARN Request returned retryable status, retrying attempt=9 backoff=1ms status=502 url=http://127.0.0.1:41577452--- PASS: TestDumpPathSingleFile (0.06s)4532026/11/12 16:57:00 WARN Request returned retryable status, retrying attempt=10 backoff=1ms status=502 url=http://127.0.0.1:41577454--- PASS: TestStreamPushRequestLine (0.06s)4552026/11/12 16:57:00 WARN Request returned retryable status, retrying attempt=11 backoff=1ms status=502 url=http://127.0.0.1:415774562026/11/12 16:57:00 WARN Request returned retryable status, retrying attempt=12 backoff=1ms status=502 url=http://127.0.0.1:415774572026/11/12 16:57:00 WARN Request returned retryable status, retrying attempt=13 backoff=1ms status=502 url=http://127.0.0.1:415774582026/11/12 16:57:00 http: TLS handshake error from 127.0.0.1:47106: remote error: tls: bad certificate459--- PASS: TestCaseHackSuffix (0.04s)4602026/11/12 16:57:00 WARN Request returned retryable status, retrying attempt=14 backoff=1ms status=502 url=http://127.0.0.1:41577461--- PASS: TestSetClientTLS (0.06s)462 --- PASS: TestSetClientTLS/succeeds_with_client_cert_and_CA (0.00s)463 --- PASS: TestSetClientTLS/preserves_debug_logging_transport (0.01s)464 --- PASS: TestSetClientTLS/rejects_connection_without_client_cert (0.01s)4652026/11/12 16:57:00 WARN Request returned retryable status, retrying attempt=15 backoff=1ms status=502 url=http://127.0.0.1:415774662026/11/12 16:57:00 WARN Request returned retryable status, retrying attempt=16 backoff=1ms status=502 url=http://127.0.0.1:415774672026/11/12 16:57:00 WARN Request returned retryable status, retrying attempt=17 backoff=1ms status=502 url=http://127.0.0.1:41577468--- PASS: TestRegisterUploadedObjectReusesConnections (0.04s)469--- PASS: TestRetryTimeoutOutlastsMaxRetries (0.08s)470--- PASS: TestDumpPathWriterError (0.05s)4712026/11/12 16:57:00 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:34359/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:34359/api/uploads/complete4722026/11/12 16:57:00 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"4732026/11/12 16:57:00 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:34359/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:34359/api/uploads/complete4742026/11/12 16:57:00 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"4752026/11/12 16:57:00 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:34359/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:34359/api/uploads/complete4762026/11/12 16:57:00 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"477--- PASS: TestRegisterUploadedObject_BoundedAgainstSilentServer (0.10s)478--- PASS: TestDumpPathMatchesNix (0.09s)479--- PASS: TestUploadMultipart_PartsInParallel (0.66s)480--- PASS: TestRateLimiterFeedback_400DoesNotCountAsSuccess (1.00s)481PASS482Running server tests...483The files belonging to this database system will be owned by user "nixbld".484This user must also own the server process.485486The database cluster will be initialized with locale "C".487The default database encoding has accordingly been set to "SQL_ASCII".488The default text search configuration will be set to "english".489490Data page checksums are enabled.491492creating directory /build/postgres439876340/data ... ok493creating subdirectories ... ok494selecting dynamic shared memory implementation ... posix495selecting default "max_connections" ... 100496selecting default "shared_buffers" ... 128MB497selecting default time zone ... UTC498creating configuration files ... ok499running bootstrap script ... ok500performing post-bootstrap initialization ... ok501syncing data to disk ... ok502503initdb: warning: enabling "trust" authentication for local connections504initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.505506Success. You can now start the database server using:507508 pg_ctl -D /build/postgres439876340/data -l logfile start509510/build/postgres439876340:5432 - no response5112026-11-12 16:57:02.566 UTC [129] LOG: starting PostgreSQL 18.6 on x86_64-pc-linux-gnu, compiled by clang version 21.1.8, 64-bit5122026-11-12 16:57:02.567 UTC [129] LOG: listening on Unix socket "/build/postgres439876340/.s.PGSQL.5432"5132026-11-12 16:57:02.572 UTC [136] LOG: database system was shut down at 2026-11-12 16:57:02 UTC5142026-11-12 16:57:02.576 UTC [129] LOG: database system is ready to accept connections515/build/postgres439876340:5432 - accepting connections516{"timestamp":"2026-11-12T16:57:02.873105537Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"bf752c74-5b3a-47b4-a1dd-444d99b9bca4","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"GET","uri":"/health/ready","status_code":503,"suppressed_errors":0,"duration_ms":0,"result":"server_error","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":502,"threadName":"rustfs-worker","threadId":"ThreadId(399)"}517=== RUN TestAbortSucceeded518=== PAUSE TestAbortSucceeded519=== RUN TestService_AuthMiddleware520=== PAUSE TestService_AuthMiddleware521=== RUN TestService_AuthMiddleware_MTLSProxyHeader522=== PAUSE TestService_AuthMiddleware_MTLSProxyHeader523=== RUN TestService_AuthMiddleware_MTLSBoundSubjects524=== PAUSE TestService_AuthMiddleware_MTLSBoundSubjects525=== RUN TestService_ReadAuthMiddleware526=== PAUSE TestService_ReadAuthMiddleware527=== RUN TestService_AuthMiddleware_OIDC528=== PAUSE TestService_AuthMiddleware_OIDC529=== RUN TestService_RequireScope_OIDC530=== PAUSE TestService_RequireScope_OIDC531=== RUN TestService_ReadScope_PublicByDefault532=== PAUSE TestService_ReadScope_PublicByDefault533=== RUN TestCacheConfigHandler534=== PAUSE TestCacheConfigHandler535=== RUN TestCacheStatsHandler536=== PAUSE TestCacheStatsHandler537=== RUN TestClientCADerivations538=== PAUSE TestClientCADerivations539=== RUN TestClientErrorHandling540=== PAUSE TestClientErrorHandling541=== RUN TestClientIntegration542=== PAUSE TestClientIntegration543=== RUN TestClientMultipleUploads544=== PAUSE TestClientMultipleUploads545=== RUN TestClientWithDependencies546=== PAUSE TestClientWithDependencies547=== RUN TestClientSharedPathCommittedMidPush548=== PAUSE TestClientSharedPathCommittedMidPush549=== RUN TestPinProtectsFromGC550=== PAUSE TestPinProtectsFromGC551=== RUN TestClientPushesUseOnePush552=== PAUSE TestClientPushesUseOnePush553=== RUN TestClientFallsBackToClosures554=== PAUSE TestClientFallsBackToClosures555=== RUN TestDeleteClosuresRacingPins556=== PAUSE TestDeleteClosuresRacingPins557=== RUN TestConcurrentCommitsSharingObjectsDoNotDeadlock558=== PAUSE TestConcurrentCommitsSharingObjectsDoNotDeadlock559=== RUN TestResolveDBConnectionString560=== PAUSE TestResolveDBConnectionString561=== RUN TestConnectWaitsForAPeerMigration562=== PAUSE TestConnectWaitsForAPeerMigration563=== RUN TestLeadElectsOneAndHandsOver564=== PAUSE TestLeadElectsOneAndHandsOver565=== RUN TestLeadIncumbentWinsAfterRestart5662026/11/12 16:57:03 INFO lead: acquired remote=192.0.2.1:12345672026/11/12 16:57:03 INFO lead: released remote=192.0.2.1:12345682026/11/12 16:57:03 INFO lead: acquired remote=192.0.2.1:12345692026/11/12 16:57:04 INFO lead: released remote=192.0.2.1:1234570--- PASS: TestLeadIncumbentWinsAfterRestart (1.91s)571=== RUN TestLeadUnannouncedStandbyYieldsToIncumbent572=== PAUSE TestLeadUnannouncedStandbyYieldsToIncumbent573=== RUN TestLeadIncumbentAnnouncesAtOnce574=== PAUSE TestLeadIncumbentAnnouncesAtOnce575=== RUN TestLeadEndsOnShutdown576=== PAUSE TestLeadEndsOnShutdown577=== RUN TestLeadEndsWhenItsConnectionHangs5782026/11/12 16:57:05 INFO lead: acquired remote=192.0.2.1:12345792026-11-12 16:57:06.417 UTC [576] FATAL: terminating connection due to administrator command5802026/11/12 16:57:06 INFO lead: acquired remote=192.0.2.1:12345812026/11/12 16:57:07 WARN lead: lock connection lost error="timeout: context deadline exceeded"5822026/11/12 16:57:07 INFO lead: released remote=192.0.2.1:12345832026/11/12 16:57:08 INFO lead: released remote=192.0.2.1:1234584--- PASS: TestLeadEndsWhenItsConnectionHangs (3.13s)585=== RUN TestGCAdvisoryLockBlocksConcurrentRun586=== PAUSE TestGCAdvisoryLockBlocksConcurrentRun587=== RUN TestGCBugBareHashReferences588=== PAUSE TestGCBugBareHashReferences589=== RUN TestGCMetrics590=== PAUSE TestGCMetrics591=== RUN TestPushDedupSurvivesConcurrentGC592=== PAUSE TestPushDedupSurvivesConcurrentGC593=== RUN TestDeduplicatedObjectsRecordedAsPending594=== PAUSE TestDeduplicatedObjectsRecordedAsPending595=== RUN TestGCSweepSkipsPendingObjects596=== PAUSE TestGCSweepSkipsPendingObjects597=== RUN TestTombstonedObjectOfferedWithoutWaiting598=== PAUSE TestTombstonedObjectOfferedWithoutWaiting599=== RUN TestGCSweepDeliversEachKeyOnce600=== PAUSE TestGCSweepDeliversEachKeyOnce601=== RUN TestCreatePendingClosureVerifyS3FailureReleasesConnection602=== PAUSE TestCreatePendingClosureVerifyS3FailureReleasesConnection603=== RUN TestForceGCDuringPushOffersSweptObject604=== PAUSE TestForceGCDuringPushOffersSweptObject605=== RUN TestFailedS3DeleteKeepsTombstone606=== PAUSE TestFailedS3DeleteKeepsTombstone607=== RUN TestSweepSkipsRowsLockedByPush608=== PAUSE TestSweepSkipsRowsLockedByPush609=== RUN TestRegisterCompletedObjectNeedsPendingRow610=== PAUSE TestRegisterCompletedObjectNeedsPendingRow611=== RUN TestGetExistingObjectsLocksRows612=== PAUSE TestGetExistingObjectsLocksRows613=== RUN TestSweepRechecksPendingAfterLocking614=== PAUSE TestSweepRechecksPendingAfterLocking615=== RUN TestPushWaitsForSweepHoldingRow616=== PAUSE TestPushWaitsForSweepHoldingRow617=== RUN TestCommitRacingPendingCleanupKeepsObjects618=== PAUSE TestCommitRacingPendingCleanupKeepsObjects619=== RUN TestGCEndsOnShutdown620=== PAUSE TestGCEndsOnShutdown621=== RUN TestGCTaskStore_StartNew622=== PAUSE TestGCTaskStore_StartNew623=== RUN TestGCTaskStore_DeduplicateSameParams624=== PAUSE TestGCTaskStore_DeduplicateSameParams625=== RUN TestGCTaskStore_ConflictDifferentParams626=== PAUSE TestGCTaskStore_ConflictDifferentParams627=== RUN TestGCTaskStore_GetEmpty628=== PAUSE TestGCTaskStore_GetEmpty629=== RUN TestGCTaskStore_GetReturnsLatest630=== PAUSE TestGCTaskStore_GetReturnsLatest631=== RUN TestGCTaskStore_CompletedAllowsNewTask632=== PAUSE TestGCTaskStore_CompletedAllowsNewTask633=== RUN TestGCTaskStore_PhaseUpdates634=== PAUSE TestGCTaskStore_PhaseUpdates635=== RUN TestGCTaskStore_Fail636=== PAUSE TestGCTaskStore_Fail637=== RUN TestGracefulShutdownDrainsInflight638=== PAUSE TestGracefulShutdownDrainsInflight639=== RUN TestService_healthCheckHandler640=== PAUSE TestService_healthCheckHandler641=== RUN TestService_readinessHandler642=== PAUSE TestService_readinessHandler643=== RUN TestGenerateLandingPage644=== PAUSE TestGenerateLandingPage645=== RUN TestEnsureAbortUploadsRule646=== PAUSE TestEnsureAbortUploadsRule647=== RUN TestCacheConfigHandlerMaxNarSize648=== PAUSE TestCacheConfigHandlerMaxNarSize649=== RUN TestCreatePendingClosureRejectsOversizedNAR650=== PAUSE TestCreatePendingClosureRejectsOversizedNAR651=== RUN TestNARDeduplicationMetadataUploadBug652=== PAUSE TestNARDeduplicationMetadataUploadBug653=== RUN TestMetricsInventory654=== PAUSE TestMetricsInventory655=== RUN TestMetricsCommit656=== PAUSE TestMetricsCommit657=== RUN TestService_NativeMTLS658=== PAUSE TestService_NativeMTLS659=== RUN TestServerTLSConfig660=== PAUSE TestServerTLSConfig661=== RUN TestMultipartCleanup662=== PAUSE TestMultipartCleanup663=== RUN TestMultipartUploadAbortedWhenCancelledBeforeRecorded664=== PAUSE TestMultipartUploadAbortedWhenCancelledBeforeRecorded665=== RUN TestPendingCleanupUsesOneCutoff666=== PAUSE TestPendingCleanupUsesOneCutoff667=== RUN TestObjectStatsTrigger668=== PAUSE TestObjectStatsTrigger669=== RUN TestObjectStatsUpdatedOncePerCommit670=== PAUSE TestObjectStatsUpdatedOncePerCommit671=== RUN TestOrphanedObjectsGC672=== PAUSE TestOrphanedObjectsGC673=== RUN TestOrphanedObjectsGCStressTest674=== PAUSE TestOrphanedObjectsGCStressTest675=== RUN TestCreatePin_ReservedPins676=== PAUSE TestCreatePin_ReservedPins677=== RUN TestPresentReportsOnlyClosureRoots678=== PAUSE TestPresentReportsOnlyClosureRoots679=== RUN TestPresentRefreshesOnlyStaleClosures680=== PAUSE TestPresentRefreshesOnlyStaleClosures681=== RUN TestPresentNotReportedWhileGCDeletesClosure682=== PAUSE TestPresentNotReportedWhileGCDeletesClosure683=== RUN TestParseSingleRange684=== PAUSE TestParseSingleRange685=== RUN TestProxyHeadersOnlyTrustedOnSocket6862026/11/12 16:57:08 INFO Starting HTTP server address=127.0.0.1:345056872026/11/12 16:57:08 INFO Starting HTTP server address=proxy.sock6882026/11/12 16:57:08 WARN mTLS auth: subject not in bound subjects subject="CN=someone"6892026/11/12 16:57:08 INFO Shutdown signal received, draining in-flight requests timeout=10s690--- PASS: TestProxyHeadersOnlyTrustedOnSocket (0.11s)691=== RUN TestIsValidCachePath692=== PAUSE TestIsValidCachePath693=== RUN TestReadProxyNarinfo694=== PAUSE TestReadProxyNarinfo695=== RUN TestReadProxyNarinfoAlreadyDecompressed696=== PAUSE TestReadProxyNarinfoAlreadyDecompressed697=== RUN TestReadProxyContentNegotiation698=== PAUSE TestReadProxyContentNegotiation699=== RUN TestReadProxyNarStreaming700=== PAUSE TestReadProxyNarStreaming701=== RUN TestReadProxy404702=== PAUSE TestReadProxy404703=== RUN TestReadProxyInvalidPath704=== PAUSE TestReadProxyInvalidPath705=== RUN TestReadProxyHead706=== PAUSE TestReadProxyHead707=== RUN TestReadProxyOutlastsServerWriteTimeout708=== PAUSE TestReadProxyOutlastsServerWriteTimeout709=== RUN TestReadProxyConditionalGet710=== PAUSE TestReadProxyConditionalGet711=== RUN TestReadProxyRootRedirectsToIndexHTML712=== PAUSE TestReadProxyRootRedirectsToIndexHTML713=== RUN TestReadProxyDisabled714=== PAUSE TestReadProxyDisabled715=== RUN TestReadRedirectNar716=== PAUSE TestReadRedirectNar717=== RUN TestReadRedirectKeepsNarinfoProxied718=== PAUSE TestReadRedirectKeepsNarinfoProxied719=== RUN TestReadProxyRangeRequest720=== PAUSE TestReadProxyRangeRequest721=== RUN TestReadRedirectUsesPublicS3URL722=== PAUSE TestReadRedirectUsesPublicS3URL723=== RUN TestPush_OverlappingRootsStoreOneRowPerKey724=== PAUSE TestPush_OverlappingRootsStoreOneRowPerKey725=== RUN TestPush_CompleteCommitsEveryRoot726=== PAUSE TestPush_CompleteCommitsEveryRoot727=== RUN TestPush_RepeatedRootCommitsOnce728=== PAUSE TestPush_RepeatedRootCommitsOnce729=== RUN TestPush_CompleteWaitsForCleanupOfThePush730=== PAUSE TestPush_CompleteWaitsForCleanupOfThePush731=== RUN TestPush_RootsAreLockedInByteOrder732=== PAUSE TestPush_RootsAreLockedInByteOrder733=== RUN TestPush_SkippedKeySurvivesGCBeforeCommit734=== PAUSE TestPush_SkippedKeySurvivesGCBeforeCommit735=== RUN TestPush_RejectsBadRequests736=== PAUSE TestPush_RejectsBadRequests737=== RUN TestPush_SignsNarinfosOfItsPendingObjects738=== PAUSE TestPush_SignsNarinfosOfItsPendingObjects739=== RUN TestCommitCanBeRepeatedUntilCleanup740=== PAUSE TestCommitCanBeRepeatedUntilCleanup741=== RUN TestSharedNarBothUploadsComplete742=== PAUSE TestSharedNarBothUploadsComplete743=== RUN TestCompleteMultipartUpload_ErrorButObjectExists744=== PAUSE TestCompleteMultipartUpload_ErrorButObjectExists745=== RUN TestCompletedNarNotReofferedAcrossClosures746=== PAUSE TestCompletedNarNotReofferedAcrossClosures747=== RUN TestPresignedUploadRegisteredBeforeCommit748=== PAUSE TestPresignedUploadRegisteredBeforeCommit749=== RUN TestService_Rustfstest750=== PAUSE TestService_Rustfstest751=== RUN TestParseSize752=== PAUSE TestParseSize753=== RUN TestSkippedUploadsHandler754=== PAUSE TestSkippedUploadsHandler755=== RUN TestSystemdListenerNotActivated756--- PASS: TestSystemdListenerNotActivated (0.00s)757=== RUN TestWatchdogBeatsWhenHealthy758--- PASS: TestWatchdogBeatsWhenHealthy (0.02s)759=== RUN TestWatchdogSkipsWhenUnhealthy7602026/11/12 16:57:08 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7612026/11/12 16:57:08 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7622026/11/12 16:57:08 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7632026/11/12 16:57:08 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7642026/11/12 16:57:08 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7652026/11/12 16:57:08 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7662026/11/12 16:57:08 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7672026/11/12 16:57:08 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7682026/11/12 16:57:08 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7692026/11/12 16:57:08 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"770--- PASS: TestWatchdogSkipsWhenUnhealthy (0.20s)771=== RUN TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle772=== PAUSE TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle773=== RUN TestProxyWriteTimeout774=== PAUSE TestProxyWriteTimeout775=== RUN TestPendingClosureWriteTimeout776=== PAUSE TestPendingClosureWriteTimeout777=== RUN TestIsValidUploadKey778=== PAUSE TestIsValidUploadKey779=== RUN TestUploadHandlersRejectInvalidKeys780=== PAUSE TestUploadHandlersRejectInvalidKeys781=== RUN TestUploadHandlersRejectOversizedBody782=== PAUSE TestUploadHandlersRejectOversizedBody783=== RUN TestService_cleanupPendingClosuresHandler784=== PAUSE TestService_cleanupPendingClosuresHandler785=== RUN TestService_createPendingClosureHandler786=== PAUSE TestService_createPendingClosureHandler787=== RUN TestService_verifyS3Integrity788=== PAUSE TestService_verifyS3Integrity789=== RUN TestService_verifyS3ReuploadsStaleNarinfo790=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo791=== RUN TestCompleteMultipartUnregistered792=== PAUSE TestCompleteMultipartUnregistered793=== RUN TestCreatePendingClosure_SmallNARUsesSimplePUT794=== PAUSE TestCreatePendingClosure_SmallNARUsesSimplePUT795=== CONT TestAbortSucceeded796--- PASS: TestAbortSucceeded (0.00s)797=== CONT TestReadProxyNarinfoAlreadyDecompressed798=== CONT TestCreatePendingClosure_SmallNARUsesSimplePUT799=== CONT TestCompleteMultipartUnregistered800=== CONT TestService_verifyS3ReuploadsStaleNarinfo801=== RUN TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed802=== CONT TestService_verifyS3Integrity803=== CONT TestService_createPendingClosureHandler804=== CONT TestService_cleanupPendingClosuresHandler805=== CONT TestUploadHandlersRejectOversizedBody806=== CONT TestUploadHandlersRejectInvalidKeys807=== CONT TestIsValidUploadKey808=== CONT TestPendingClosureWriteTimeout809=== CONT TestProxyWriteTimeout810=== CONT TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle811=== CONT TestSkippedUploadsHandler812=== CONT TestParseSize813=== CONT TestService_Rustfstest814=== CONT TestPresignedUploadRegisteredBeforeCommit815=== CONT TestCompletedNarNotReofferedAcrossClosures816=== CONT TestCompleteMultipartUpload_ErrorButObjectExists817=== CONT TestSharedNarBothUploadsComplete818=== CONT TestCommitCanBeRepeatedUntilCleanup819=== CONT TestPush_SignsNarinfosOfItsPendingObjects820=== CONT TestPush_RejectsBadRequests821=== CONT TestPush_SkippedKeySurvivesGCBeforeCommit822=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed823=== RUN TestPendingClosureWriteTimeout/empty824=== CONT TestGetExistingObjectsLocksRows825=== RUN TestProxyWriteTimeout/narinfo826=== RUN TestCommitCanBeRepeatedUntilCleanup/push827=== PAUSE TestProxyWriteTimeout/narinfo828--- PASS: TestParseSize (0.00s)829=== PAUSE TestPendingClosureWriteTimeout/empty830=== RUN TestIsValidUploadKey/narinfo831=== RUN TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed832=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info833=== RUN TestProxyWriteTimeout/1_GiB_nar834=== RUN TestPendingClosureWriteTimeout/400_objects835=== PAUSE TestCommitCanBeRepeatedUntilCleanup/push836=== RUN TestCommitCanBeRepeatedUntilCleanup/pending_closure8372026/11/12 16:57:08 INFO Client skipped oversized paths paths=3 nar_bytes=5000000000838=== PAUSE TestIsValidUploadKey/narinfo839=== PAUSE TestProxyWriteTimeout/1_GiB_nar840=== RUN TestProxyWriteTimeout/10_GiB_nar841=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed842=== PAUSE TestPendingClosureWriteTimeout/400_objects843=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info844=== RUN TestIsValidUploadKey/nar_zst845=== PAUSE TestIsValidUploadKey/nar_zst846=== RUN TestIsValidUploadKey/nar_xz847=== PAUSE TestIsValidUploadKey/nar_xz848=== PAUSE TestCommitCanBeRepeatedUntilCleanup/pending_closure849=== PAUSE TestProxyWriteTimeout/10_GiB_nar850=== RUN TestProxyWriteTimeout/unknown_size851=== PAUSE TestProxyWriteTimeout/unknown_size852=== RUN TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost853=== CONT TestPush_CompleteWaitsForCleanupOfThePush854=== RUN TestPendingClosureWriteTimeout/670k_objects855=== PAUSE TestPendingClosureWriteTimeout/670k_objects856=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal857=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal858=== RUN TestIsValidUploadKey/nar_plain859=== CONT TestPush_RootsAreLockedInByteOrder860=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost861=== RUN TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed862=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed863=== RUN TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed864=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed865=== RUN TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline866=== RUN TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key867=== PAUSE TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key868=== PAUSE TestIsValidUploadKey/nar_plain869=== CONT TestReadProxyNarinfo870=== PAUSE TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline871=== CONT TestPush_RepeatedRootCommitsOnce872=== RUN TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key873=== PAUSE TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key874=== RUN TestIsValidUploadKey/listing875=== PAUSE TestIsValidUploadKey/listing876=== RUN TestIsValidUploadKey/build_log877=== PAUSE TestIsValidUploadKey/build_log878=== RUN TestIsValidUploadKey/build_log_home-manager_file879=== PAUSE TestIsValidUploadKey/build_log_home-manager_file880=== RUN TestIsValidUploadKey/build_log_plus_in_name881=== PAUSE TestIsValidUploadKey/build_log_plus_in_name882=== RUN TestIsValidUploadKey/build_log_question_mark883=== PAUSE TestIsValidUploadKey/build_log_question_mark884=== CONT TestIsValidCachePath885=== RUN TestIsValidCachePath/narinfo886=== RUN TestIsValidUploadKey/build_log_equals887=== PAUSE TestIsValidUploadKey/build_log_equals888=== RUN TestIsValidUploadKey/realisation889=== PAUSE TestIsValidUploadKey/realisation890=== RUN TestIsValidUploadKey/realisation_plus_in_output891=== PAUSE TestIsValidUploadKey/realisation_plus_in_output892=== RUN TestIsValidUploadKey/nix-cache-info893=== PAUSE TestIsValidUploadKey/nix-cache-info894=== RUN TestIsValidUploadKey/index.html895=== PAUSE TestIsValidUploadKey/index.html896=== PAUSE TestIsValidCachePath/narinfo897=== RUN TestIsValidCachePath/narinfo_all_nix_base32_chars898=== PAUSE TestIsValidCachePath/narinfo_all_nix_base32_chars899=== RUN TestIsValidCachePath/nar_zst900=== PAUSE TestIsValidCachePath/nar_zst901=== RUN TestIsValidCachePath/nar_xz902=== PAUSE TestIsValidCachePath/nar_xz903=== RUN TestIsValidCachePath/nar_bz2904=== PAUSE TestIsValidCachePath/nar_bz2905=== RUN TestIsValidCachePath/nar_uncompressed906=== PAUSE TestIsValidCachePath/nar_uncompressed907=== RUN TestIsValidCachePath/ls908=== PAUSE TestIsValidCachePath/ls909=== RUN TestIsValidUploadKey/narinfo_key,_nar_type910=== PAUSE TestIsValidUploadKey/narinfo_key,_nar_type911=== RUN TestIsValidUploadKey/nar_key,_narinfo_type912=== PAUSE TestIsValidUploadKey/nar_key,_narinfo_type913=== RUN TestIsValidUploadKey/listing_key,_narinfo_type914=== PAUSE TestIsValidUploadKey/listing_key,_narinfo_type915=== RUN TestIsValidUploadKey/traversal916=== PAUSE TestIsValidUploadKey/traversal917=== RUN TestIsValidUploadKey/traversal_nar918=== PAUSE TestIsValidUploadKey/traversal_nar919=== RUN TestIsValidUploadKey/absolute920=== PAUSE TestIsValidUploadKey/absolute921=== RUN TestIsValidUploadKey/empty_key922=== PAUSE TestIsValidUploadKey/empty_key923=== RUN TestIsValidUploadKey/unknown_type924=== PAUSE TestIsValidUploadKey/unknown_type925=== RUN TestIsValidCachePath/log926=== PAUSE TestIsValidCachePath/log927=== RUN TestIsValidCachePath/realisation928=== PAUSE TestIsValidCachePath/realisation929=== RUN TestIsValidCachePath/nix-cache-info930=== PAUSE TestIsValidCachePath/nix-cache-info931=== RUN TestIsValidCachePath/index.html932=== PAUSE TestIsValidCachePath/index.html933=== RUN TestIsValidCachePath/pin934=== PAUSE TestIsValidCachePath/pin935=== RUN TestIsValidCachePath/pin_all_alphabet_chars936=== PAUSE TestIsValidCachePath/pin_all_alphabet_chars937=== RUN TestIsValidCachePath/traversal_parent938=== PAUSE TestIsValidCachePath/traversal_parent939=== RUN TestIsValidCachePath/traversal_in_middle940=== PAUSE TestIsValidCachePath/traversal_in_middle941=== CONT TestPush_CompleteCommitsEveryRoot942--- PASS: TestSkippedUploadsHandler (0.10s)943=== CONT TestPush_OverlappingRootsStoreOneRowPerKey944=== RUN TestIsValidCachePath/pin_traversal945=== PAUSE TestIsValidCachePath/pin_traversal946=== RUN TestIsValidCachePath/invalid_char_e947=== PAUSE TestIsValidCachePath/invalid_char_e948=== RUN TestIsValidCachePath/invalid_char_u949=== PAUSE TestIsValidCachePath/invalid_char_u950=== RUN TestIsValidCachePath/random_path951=== PAUSE TestIsValidCachePath/random_path952=== RUN TestIsValidCachePath/empty953=== PAUSE TestIsValidCachePath/empty954=== RUN TestIsValidCachePath/leading_slash955=== PAUSE TestIsValidCachePath/leading_slash956=== RUN TestIsValidCachePath/wrong_extension957=== PAUSE TestIsValidCachePath/wrong_extension958=== RUN TestIsValidCachePath/short_hash959=== PAUSE TestIsValidCachePath/short_hash960=== RUN TestIsValidCachePath/pin_empty_name961=== PAUSE TestIsValidCachePath/pin_empty_name962=== RUN TestIsValidCachePath/pin_nested963=== PAUSE TestIsValidCachePath/pin_nested964=== RUN TestIsValidCachePath/pin_invalid_char965=== PAUSE TestIsValidCachePath/pin_invalid_char966=== CONT TestParseSingleRange967=== RUN TestParseSingleRange/none968=== PAUSE TestParseSingleRange/none969=== RUN TestParseSingleRange/unknown_unit970=== PAUSE TestParseSingleRange/unknown_unit971=== RUN TestParseSingleRange/multi-range_ignored972=== PAUSE TestParseSingleRange/multi-range_ignored973=== RUN TestParseSingleRange/malformed_no_dash974=== PAUSE TestParseSingleRange/malformed_no_dash975=== RUN TestParseSingleRange/malformed_both_empty976=== PAUSE TestParseSingleRange/malformed_both_empty977=== RUN TestParseSingleRange/malformed_end_before_start978=== PAUSE TestParseSingleRange/malformed_end_before_start979=== RUN TestParseSingleRange/closed980=== PAUSE TestParseSingleRange/closed981=== RUN TestParseSingleRange/open-ended982=== PAUSE TestParseSingleRange/open-ended983=== RUN TestParseSingleRange/end_clamped_to_size984=== PAUSE TestParseSingleRange/end_clamped_to_size985=== RUN TestParseSingleRange/suffix986=== PAUSE TestParseSingleRange/suffix987=== RUN TestParseSingleRange/suffix_exceeds_size988=== PAUSE TestParseSingleRange/suffix_exceeds_size989=== RUN TestParseSingleRange/single_byte990=== PAUSE TestParseSingleRange/single_byte991=== RUN TestParseSingleRange/start_past_EOF992=== PAUSE TestParseSingleRange/start_past_EOF993=== RUN TestParseSingleRange/start_far_past_EOF994=== PAUSE TestParseSingleRange/start_far_past_EOF995=== CONT TestReadRedirectUsesPublicS3URL996=== RUN TestUploadHandlersRejectOversizedBody/complete_multipart997=== PAUSE TestUploadHandlersRejectOversizedBody/complete_multipart998=== RUN TestUploadHandlersRejectOversizedBody/request_more_parts999=== PAUSE TestUploadHandlersRejectOversizedBody/request_more_parts1000=== RUN TestUploadHandlersRejectOversizedBody/create_pending_closure1001=== PAUSE TestUploadHandlersRejectOversizedBody/create_pending_closure1002=== CONT TestPresentNotReportedWhileGCDeletesClosure1003=== RUN TestPresentNotReportedWhileGCDeletesClosure/stale1004=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/stale1005=== RUN TestPresentNotReportedWhileGCDeletesClosure/fresh1006=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/fresh1007=== CONT TestReadProxyRangeRequest10082026/11/12 16:57:08 INFO Received complete multipart upload request method=POST path=/api/multipart/complete10092026/11/12 16:57:08 ERROR Failed to get multipart upload error="no rows in result set" upload_id=does-not-exist object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst1010--- PASS: TestCompleteMultipartUnregistered (0.38s)1011=== CONT TestPresentRefreshesOnlyStaleClosures1012--- PASS: TestReadProxyNarinfoAlreadyDecompressed (0.42s)1013=== CONT TestReadRedirectKeepsNarinfoProxied10142026/11/12 16:57:08 INFO Received uploads request method=POST path=/api/pending_closures10152026/11/12 16:57:08 INFO Received uploads request method=POST path=/api/pending_closures10162026/11/12 16:57:08 INFO Received uploads request method=POST path=/api/pending_closures10172026/11/12 16:57:08 INFO Received uploads request method=POST path=/api/pending_closures1018--- PASS: TestGetExistingObjectsLocksRows (0.53s)1019=== CONT TestPresentReportsOnlyClosureRoots10202026/11/12 16:57:09 INFO Received uploads request method=POST path=/api/pending_closures1021=== RUN TestPush_RejectsBadRequests/no_roots1022=== PAUSE TestPush_RejectsBadRequests/no_roots1023=== RUN TestPush_RejectsBadRequests/no_objects1024=== PAUSE TestPush_RejectsBadRequests/no_objects1025=== RUN TestPush_RejectsBadRequests/bad_root1026=== PAUSE TestPush_RejectsBadRequests/bad_root1027=== RUN TestPush_RejectsBadRequests/root_not_in_objects1028=== PAUSE TestPush_RejectsBadRequests/root_not_in_objects1029=== CONT TestCreatePin_ReservedPins10302026/11/12 16:57:09 INFO Received complete multipart upload request method=POST path=/api/multipart/complete10312026/11/12 16:57:09 WARN CompleteMultipartUpload errored but object exists; treating as success error="The specified key does not exist." object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=NGUwZjljNTMtYjgxYy00ZDQxLTlhY2ItZDkwMzA0MzdiZmNmLjIyMDQ1ZmJjLTU2NmQtNDgyZS1hZDBiLWM3Y2VhMmRhYWFhOHgxNzk0NTAyNjI5MTAzMzgzODk210322026/11/12 16:57:09 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=NGUwZjljNTMtYjgxYy00ZDQxLTlhY2ItZDkwMzA0MzdiZmNmLjIyMDQ1ZmJjLTU2NmQtNDgyZS1hZDBiLWM3Y2VhMmRhYWFhOHgxNzk0NTAyNjI5MTAzMzgzODk2 parts=11033--- PASS: TestCompleteMultipartUpload_ErrorButObjectExists (0.79s)1034=== CONT TestReadRedirectNar10352026/11/12 16:57:09 INFO Received uploads request method=POST path=/api/pending_closures10362026/11/12 16:57:09 INFO Received uploads request method=POST path=/api/pending_closures10372026/11/12 16:57:09 INFO Received uploads request method=POST path=/api/pending_closures1038--- PASS: TestService_Rustfstest (0.82s)1039=== CONT TestOrphanedObjectsGCStressTest10402026/11/12 16:57:09 INFO Received uploads request method=POST path=/api/pending_closures10412026/11/12 16:57:09 INFO Received push request method=POST path=/api/pushes10422026/11/12 16:57:09 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:33475/oidc10432026/11/12 16:57:09 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign10442026/11/12 16:57:09 INFO Signed narinfos id=1 count=110452026/11/12 16:57:09 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign10462026/11/12 16:57:09 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign10472026/11/12 16:57:09 ERROR Failed to sign narinfos signer="index 1" error="signing key cannot be nil"10482026/11/12 16:57:09 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign10492026/11/12 16:57:09 ERROR Failed to sign narinfos signer=niks3-test-1 error="signer failed"10502026/11/12 16:57:09 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign10512026/11/12 16:57:09 ERROR Signer returned unexpected signature count signer=niks3-test-1 got=0 want=110522026/11/12 16:57:09 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign10532026/11/12 16:57:09 ERROR Signer returned unexpected object key signer=niks3-test-1 object_key=unexpected.narinfo1054--- PASS: TestPush_SignsNarinfosOfItsPendingObjects (0.93s)1055=== CONT TestReadProxyDisabled10562026/11/12 16:57:09 INFO Received push request method=POST path=/api/pushes10572026/11/12 16:57:09 INFO Received complete multipart upload request method=POST path=/api/multipart/complete10582026/11/12 16:57:09 ERROR Failed to decode narinfo key=3hcdxyjf9yiq7qf3i4548drb6sjmwa1v.narinfo limit=16777216 error=<nil>1059--- PASS: TestReadProxyNarinfo (0.87s)10602026/11/12 16:57:09 INFO Received complete push request method=POST path=/api/pushes/1/complete1061=== CONT TestOrphanedObjectsGC10622026/11/12 16:57:09 INFO Completed push id=1 objects=4 duration=9.003598ms10632026/11/12 16:57:09 INFO Received push request method=POST path=/api/pushes10642026/11/12 16:57:09 INFO Received uploads request method=POST path=/api/pending_closures1065--- PASS: TestCreatePendingClosure_SmallNARUsesSimplePUT (1.00s)1066=== CONT TestReadProxyRootRedirectsToIndexHTML10672026/11/12 16:57:09 INFO Received cleanup request method=DELETE path=/api/pending_closures10682026/11/12 16:57:09 INFO Aborted multipart uploads count=0 kept=010692026/11/12 16:57:09 INFO Received uploads request method=POST path=/api/pending_closures10702026/11/12 16:57:09 INFO Received cleanup request method=DELETE path=/api/pending_closures10712026/11/12 16:57:09 INFO Aborted multipart uploads count=1 kept=010722026/11/12 16:57:09 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete10732026-11-12 16:57:09.396 UTC [664] ERROR: Closure does not exist: id=110742026-11-12 16:57:09.396 UTC [664] CONTEXT: PL/pgSQL function commit_pending_closure(bigint) line 17 at RAISE10752026-11-12 16:57:09.396 UTC [664] STATEMENT: -- name: CommitPendingClosure :one1076 SELECT commit_pending_closure($1::bigint)::bigint1077 1078--- PASS: TestService_cleanupPendingClosuresHandler (1.03s)1079=== CONT TestObjectStatsUpdatedOncePerCommit10802026/11/12 16:57:09 INFO Aborted multipart uploads count=0 kept=010812026/11/12 16:57:09 INFO Received uploads request method=POST path=/api/pending_closures10822026/11/12 16:57:09 WARN Force mode enabled - objects will be deleted immediately without grace period10832026/11/12 16:57:09 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=2 objects-deleted-after-grace-period=2 objects-failed-to-delete=010842026/11/12 16:57:09 INFO Registered completed upload object_key=nar/jjjjjjjjjjjjjjjjjjjjjjjjjjjjjj0100000000000000000000.nar.zst10852026/11/12 16:57:09 INFO Received uploads request method=POST path=/api/pending_closures10862026/11/12 16:57:09 INFO Vacuumed table table=pending_closures10872026/11/12 16:57:09 INFO Received complete multipart upload request method=POST path=/api/multipart/complete1088--- PASS: TestPresignedUploadRegisteredBeforeCommit (0.97s)1089=== CONT TestReadProxyConditionalGet10902026/11/12 16:57:09 INFO Vacuumed table table=commit_receipts10912026/11/12 16:57:09 INFO Vacuumed table table=pending_objects10922026/11/12 16:57:09 INFO Vacuumed table table=multipart_uploads10932026/11/12 16:57:09 INFO Received push request method=POST path=/api/pushes10942026/11/12 16:57:09 INFO Vacuumed table table=closures10952026/11/12 16:57:09 INFO Vacuumed table table=objects10962026/11/12 16:57:09 INFO Received complete push request method=POST path=/api/pushes/2/complete10972026/11/12 16:57:09 INFO Completed push id=2 objects=4 duration=3.960919ms1098--- PASS: TestPush_SkippedKeySurvivesGCBeforeCommit (1.09s)1099=== CONT TestObjectStatsTrigger11002026/11/12 16:57:09 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst upload_id=NGUwZjljNTMtYjgxYy00ZDQxLTlhY2ItZDkwMzA0MzdiZmNmLjZiMjJmNDU4LTU0NjktNDAyNS05ODJhLTc0ZDRkMzkwMDQyN3gxNzk0NTAyNjI4ODA5OTc2Njc0 parts=1011012026/11/12 16:57:09 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete11022026/11/12 16:57:09 INFO Received complete push request method=POST path=/api/pushes/1/complete11032026/11/12 16:57:09 INFO Completed upload id=1 objects=2 duration=5.790039ms11042026/11/12 16:57:09 INFO Received uploads request method=POST path=/api/pending_closures11052026/11/12 16:57:09 INFO Completed push id=1 objects=4 duration=9.135618ms1106--- PASS: TestPush_RepeatedRootCommitsOnce (1.01s)1107=== CONT TestReadProxyOutlastsServerWriteTimeout11082026/11/12 16:57:09 INFO Received uploads request method=POST path=/api/pending_closures11092026/11/12 16:57:09 INFO Received push request method=POST path=/api/pushes11102026/11/12 16:57:09 INFO Object in database but missing from S3 key=dadb44fdadb44fdadb44fdadb44f0000.narinfo11112026/11/12 16:57:09 WARN Found objects in DB but missing from S3, will re-upload count=11112--- PASS: TestService_verifyS3Integrity (1.12s)1113=== CONT TestPendingCleanupUsesOneCutoff11142026/11/12 16:57:09 INFO Received push request method=POST path=/api/pushes11152026/11/12 16:57:09 INFO Received complete push request method=POST path=/api/pushes/1/complete11162026/11/12 16:57:09 INFO Completed push id=1 objects=6 duration=10.545868ms1117--- PASS: TestPush_CompleteCommitsEveryRoot (1.05s)1118=== CONT TestReadProxyHead11192026/11/12 16:57:09 INFO Received complete multipart upload request method=POST path=/api/multipart/complete11202026/11/12 16:57:09 INFO Completed multipart upload object_key=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhh0100000000000000000000.nar.zst upload_id=NGUwZjljNTMtYjgxYy00ZDQxLTlhY2ItZDkwMzA0MzdiZmNmLjE2MTNmNjQ5LWU4Y2YtNDEwYy04N2QyLWI3ZDAxMDM5ZjVkZngxNzk0NTAyNjI4ODMxMTg1MDgw parts=1211212026/11/12 16:57:09 INFO Received uploads request method=POST path=/api/pending_closures1122--- PASS: TestReadProxyRangeRequest (1.02s)1123=== CONT TestMultipartUploadAbortedWhenCancelledBeforeRecorded1124--- PASS: TestCompletedNarNotReofferedAcrossClosures (1.19s)1125=== CONT TestReadProxyInvalidPath11262026/11/12 16:57:09 INFO Received push request method=POST path=/api/pushes11272026/11/12 16:57:09 INFO Received complete multipart upload request method=POST path=/api/multipart/complete1128--- PASS: TestPush_OverlappingRootsStoreOneRowPerKey (1.15s)1129=== CONT TestGCTaskStore_Fail1130--- PASS: TestGCTaskStore_Fail (0.00s)1131=== CONT TestReadProxy40411322026/11/12 16:57:09 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0100000000000000000000.nar.zst upload_id=NGUwZjljNTMtYjgxYy00ZDQxLTlhY2ItZDkwMzA0MzdiZmNmLmMzMTIyNjBiLTRiNDMtNGZjMS1hNDFmLTNhOTMwZWQ2NmU4ZHgxNzk0NTAyNjI4ODYwMjgyNDk0 parts=1211332026/11/12 16:57:09 INFO Received complete multipart upload request method=POST path=/api/multipart/complete11342026/11/12 16:57:09 INFO Completed multipart upload object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst upload_id=NGUwZjljNTMtYjgxYy00ZDQxLTlhY2ItZDkwMzA0MzdiZmNmLjU1MTI0YmJlLWNmNDAtNDEwMC1hZWQ5LTUwNzVjODVlN2QwN3gxNzk0NTAyNjI5MTYxMTUwMDI1 parts=1011352026/11/12 16:57:09 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete1136--- PASS: TestReadRedirectUsesPublicS3URL (1.18s)1137=== CONT TestGCTaskStore_PhaseUpdates1138--- PASS: TestGCTaskStore_PhaseUpdates (0.00s)1139=== CONT TestReadProxyNarStreaming11402026/11/12 16:57:09 INFO Received push request method=POST path=/api/pushes11412026/11/12 16:57:09 INFO Completed upload id=1 objects=2 duration=5.794779ms11422026/11/12 16:57:09 INFO Received get closure request method=GET path=/api/closures/0000000000000000000000000000000011432026/11/12 16:57:09 INFO Received uploads request method=POST path=/api/pending_closures11442026/11/12 16:57:09 INFO Starting cleanup of old closures method=DELETE path=/api/closures11452026/11/12 16:57:09 INFO Aborted multipart uploads count=0 kept=011462026/11/12 16:57:09 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=011472026/11/12 16:57:09 INFO Vacuumed table table=pending_closures11482026/11/12 16:57:09 INFO Vacuumed table table=commit_receipts1149--- PASS: TestPresentRefreshesOnlyStaleClosures (1.03s)1150=== CONT TestGCTaskStore_CompletedAllowsNewTask11512026/11/12 16:57:09 INFO Vacuumed table table=pending_objects1152--- PASS: TestGCTaskStore_CompletedAllowsNewTask (0.00s)1153=== CONT TestGCTaskStore_GetReturnsLatest1154--- PASS: TestGCTaskStore_GetReturnsLatest (0.00s)1155=== CONT TestReadProxyContentNegotiation11562026/11/12 16:57:09 INFO Vacuumed table table=multipart_uploads11572026/11/12 16:57:09 INFO Vacuumed table table=closures11582026/11/12 16:57:09 INFO Vacuumed table table=objects11592026-11-12 16:57:09.776 UTC [762] ERROR: could not obtain lock on row in relation "closures"11602026-11-12 16:57:09.776 UTC [762] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT11612026-11-12 16:57:09.777 UTC [762] ERROR: could not obtain lock on row in relation "closures"11622026-11-12 16:57:09.777 UTC [762] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT11632026-11-12 16:57:09.777 UTC [762] ERROR: could not obtain lock on row in relation "closures"11642026-11-12 16:57:09.777 UTC [762] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT11652026-11-12 16:57:09.777 UTC [762] ERROR: could not obtain lock on row in relation "closures"11662026-11-12 16:57:09.777 UTC [762] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT11672026-11-12 16:57:09.778 UTC [762] ERROR: could not obtain lock on row in relation "closures"11682026-11-12 16:57:09.778 UTC [762] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT1169--- PASS: TestPush_RootsAreLockedInByteOrder (1.32s)1170=== CONT TestGCTaskStore_GetEmpty1171--- PASS: TestGCTaskStore_GetEmpty (0.00s)1172=== CONT TestCommitRacingPendingCleanupKeepsObjects11732026/11/12 16:57:09 INFO Received get closure request method=GET path=/api/closures/000000000000000000000000000000001174--- PASS: TestService_createPendingClosureHandler (1.42s)1175=== CONT TestGCEndsOnShutdown1176=== RUN TestGCEndsOnShutdown/before_the_run1177=== PAUSE TestGCEndsOnShutdown/before_the_run1178=== RUN TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete1179=== PAUSE TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete1180=== RUN TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark1181=== PAUSE TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark1182=== CONT TestPushWaitsForSweepHoldingRow1183--- PASS: TestReadRedirectKeepsNarinfoProxied (1.02s)1184=== CONT TestGCTaskStore_ConflictDifferentParams1185--- PASS: TestGCTaskStore_ConflictDifferentParams (0.00s)1186=== CONT TestSweepRechecksPendingAfterLocking11872026-11-12 16:57:09.879 UTC [738] ERROR: Push does not exist: id=111882026-11-12 16:57:09.879 UTC [738] CONTEXT: PL/pgSQL function commit_push(bigint) line 15 at RAISE11892026-11-12 16:57:09.879 UTC [738] STATEMENT: SELECT commit_push($1)1190--- PASS: TestPush_CompleteWaitsForCleanupOfThePush (1.42s)1191=== CONT TestGCTaskStore_DeduplicateSameParams1192--- PASS: TestGCTaskStore_DeduplicateSameParams (0.00s)1193=== CONT TestGCTaskStore_StartNew1194--- PASS: TestGCTaskStore_StartNew (0.00s)1195=== CONT TestConcurrentCommitsSharingObjectsDoNotDeadlock1196--- PASS: TestPresentReportsOnlyClosureRoots (1.01s)1197=== CONT TestClientCADerivations1198--- PASS: TestReadRedirectNar (0.77s)1199=== CONT TestRegisterCompletedObjectNeedsPendingRow12002026/11/12 16:57:09 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux12012026/11/12 16:57:09 WARN Refused reserved pin name=worker-x86_64-linux12022026/11/12 16:57:09 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux12032026/11/12 16:57:09 INFO Received create pin request method=POST path=/api/pins/my-app12042026/11/12 16:57:09 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux1205--- PASS: TestCreatePin_ReservedPins (0.87s)1206=== CONT TestDeleteClosuresRacingPins1207--- PASS: TestReadProxyDisabled (0.72s)1208=== CONT TestSweepSkipsRowsLockedByPush1209--- PASS: TestReadProxyRootRedirectsToIndexHTML (0.72s)1210=== CONT TestClientFallsBackToClosures1211--- PASS: TestReadProxyConditionalGet (0.71s)1212=== CONT TestFailedS3DeleteKeepsTombstone1213--- PASS: TestObjectStatsUpdatedOncePerCommit (0.76s)1214=== CONT TestClientPushesUseOnePush1215--- PASS: TestObjectStatsTrigger (0.74s)1216=== CONT TestForceGCDuringPushOffersSweptObject1217=== RUN TestForceGCDuringPushOffersSweptObject/before_pending_rows1218=== PAUSE TestForceGCDuringPushOffersSweptObject/before_pending_rows1219=== RUN TestForceGCDuringPushOffersSweptObject/after_presence_check1220=== PAUSE TestForceGCDuringPushOffersSweptObject/after_presence_check1221=== CONT TestPinProtectsFromGC12222026/11/12 16:57:10 INFO Received uploads request method=POST path=/api/pending_closures12232026/11/12 16:57:10 INFO Received uploads request method=POST path=/api/pending_closures12242026/11/12 16:57:10 INFO Received complete multipart upload request method=POST path=/api/multipart/complete12252026/11/12 16:57:10 INFO Received cleanup request method=DELETE path=/api/pending_closures1226--- PASS: TestReadProxyHead (0.75s)1227=== CONT TestGracefulShutdownDrainsInflight12282026/11/12 16:57:10 INFO Starting HTTP server address=127.0.0.1:4253512292026/11/12 16:57:10 INFO Shutdown signal received, draining in-flight requests timeout=10s12302026/11/12 16:57:10 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0100000000000000000000.nar.zst upload_id=NGUwZjljNTMtYjgxYy00ZDQxLTlhY2ItZDkwMzA0MzdiZmNmLjFkMjM5MTA4LTcyNmYtNDkzZC1iYzNiLTJiYTQyY2M4NGMyNngxNzk0NTAyNjI4ODY4NDE0MTYy parts=121231--- PASS: TestSharedNarBothUploadsComplete (1.91s)1232=== CONT TestClientSharedPathCommittedMidPush1233--- PASS: TestReadProxyInvalidPath (0.73s)1234=== CONT TestCreatePendingClosureVerifyS3FailureReleasesConnection1235--- PASS: TestGracefulShutdownDrainsInflight (0.07s)1236=== CONT TestClientWithDependencies12372026/11/12 16:57:10 INFO Received uploads request method=POST path=/api/pending_closures1238--- PASS: TestReadProxy404 (0.77s)1239=== CONT TestGCSweepDeliversEachKeyOnce1240--- PASS: TestMultipartUploadAbortedWhenCancelledBeforeRecorded (0.84s)1241=== CONT TestClientMultipleUploads1242--- PASS: TestReadProxyNarStreaming (0.76s)1243=== CONT TestTombstonedObjectOfferedWithoutWaiting1244=== NAME TestOrphanedObjectsGC1245 orphaned_objects_gc_test.go:287: GC Test Summary:1246 orphaned_objects_gc_test.go:288: - Kept: 2 objects from closure A1247 orphaned_objects_gc_test.go:289: - Deleted: 2 objects from closure B1248 orphaned_objects_gc_test.go:290: - Deleted: 6 orphaned chain objects (X1->X2->X3)1249 orphaned_objects_gc_test.go:291: - Deleted: 2 orphaned single objects (Y)1250 orphaned_objects_gc_test.go:292: - Total deleted: 10 objects1251--- PASS: TestOrphanedObjectsGC (1.14s)1252=== CONT TestClientIntegration1253--- PASS: TestReadProxyContentNegotiation (0.73s)1254=== CONT TestMultipartCleanup12552026/11/12 16:57:10 INFO Received uploads request method=POST path=/api/pending_closures12562026/11/12 16:57:10 INFO Aborted multipart uploads count=0 kept=012572026/11/12 16:57:10 WARN Force mode enabled - objects will be deleted immediately without grace period12582026/11/12 16:57:10 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=012592026/11/12 16:57:10 INFO Vacuumed table table=pending_closures12602026/11/12 16:57:10 INFO Vacuumed table table=commit_receipts12612026/11/12 16:57:10 INFO Vacuumed table table=pending_objects12622026/11/12 16:57:10 INFO Vacuumed table table=multipart_uploads12632026/11/12 16:57:10 INFO Vacuumed table table=closures12642026/11/12 16:57:10 INFO Vacuumed table table=objects1265--- PASS: TestCommitRacingPendingCleanupKeepsObjects (0.75s)1266=== CONT TestGCSweepSkipsPendingObjects12672026/11/12 16:57:10 INFO Aborted multipart uploads count=0 kept=012682026/11/12 16:57:10 WARN Force mode enabled - objects will be deleted immediately without grace period12692026/11/12 16:57:10 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=012702026/11/12 16:57:10 INFO Vacuumed table table=pending_closures12712026/11/12 16:57:10 INFO Vacuumed table table=commit_receipts12722026/11/12 16:57:10 INFO Vacuumed table table=pending_objects12732026/11/12 16:57:10 INFO Vacuumed table table=multipart_uploads12742026/11/12 16:57:10 INFO Vacuumed table table=closures12752026/11/12 16:57:10 INFO Vacuumed table table=objects1276--- PASS: TestSweepRechecksPendingAfterLocking (0.79s)1277=== CONT TestClientErrorHandling1278=== RUN TestClientErrorHandling/InvalidStorePath1279=== PAUSE TestClientErrorHandling/InvalidStorePath1280=== RUN TestClientErrorHandling/InvalidAuthToken1281=== PAUSE TestClientErrorHandling/InvalidAuthToken1282=== RUN TestClientErrorHandling/ServerNotAvailable1283=== PAUSE TestClientErrorHandling/ServerNotAvailable1284=== CONT TestDeduplicatedObjectsRecordedAsPending1285--- PASS: TestRegisterCompletedObjectNeedsPendingRow (0.73s)1286=== CONT TestServerTLSConfig1287=== RUN TestServerTLSConfig/no_client_CA1288=== PAUSE TestServerTLSConfig/no_client_CA1289=== RUN TestServerTLSConfig/missing_CA_file1290=== PAUSE TestServerTLSConfig/missing_CA_file1291=== RUN TestServerTLSConfig/not_a_PEM_file1292=== PAUSE TestServerTLSConfig/not_a_PEM_file1293=== CONT TestPushDedupSurvivesConcurrentGC1294=== NAME TestClientCADerivations1295 client_ca_test.go:136: Built CA derivation: /build/TestClientCADerivations3473618602/001/store/rnwpn4pjgsv92zsxcrhvlfx591f0f1lq-ca-test12962026/11/12 16:57:10 INFO Aborted multipart uploads count=0 kept=012972026/11/12 16:57:10 WARN Force mode enabled - objects will be deleted immediately without grace period12982026/11/12 16:57:10 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=01299 client_ca_test.go:139: Found 1 dependencies (including self)13002026/11/12 16:57:10 INFO Vacuumed table table=pending_closures13012026/11/12 16:57:10 INFO Vacuumed table table=commit_receipts13022026/11/12 16:57:10 INFO Vacuumed table table=pending_objects13032026/11/12 16:57:10 INFO Vacuumed table table=multipart_uploads13042026/11/12 16:57:10 INFO Vacuumed table table=closures13052026/11/12 16:57:10 INFO Vacuumed table table=objects13062026/11/12 16:57:10 INFO Aborted multipart uploads count=0 kept=013072026/11/12 16:57:10 WARN Force mode enabled - objects will be deleted immediately without grace period13082026/11/12 16:57:10 INFO Received uploads request method=POST path=/api/pending_closures13092026/11/12 16:57:10 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=013102026/11/12 16:57:10 INFO Vacuumed table table=pending_closures13112026/11/12 16:57:10 INFO Vacuumed table table=commit_receipts13122026/11/12 16:57:10 INFO Vacuumed table table=pending_objects13132026/11/12 16:57:10 INFO Vacuumed table table=multipart_uploads13142026/11/12 16:57:10 INFO Vacuumed table table=closures13152026/11/12 16:57:10 INFO Vacuumed table table=objects1316--- PASS: TestSweepSkipsRowsLockedByPush (0.78s)1317=== CONT TestService_NativeMTLS1318--- PASS: TestConcurrentCommitsSharingObjectsDoNotDeadlock (0.93s)1319=== CONT TestGCMetrics13202026/11/12 16:57:10 INFO Aborted multipart uploads count=0 kept=013212026/11/12 16:57:10 INFO Received push request method=POST path=/api/pushes13222026/11/12 16:57:10 WARN Force mode enabled - objects will be deleted immediately without grace period13232026/11/12 16:57:10 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)13242026/11/12 16:57:10 INFO Uploading rnwpn4pjgsv92zsxcrhvlfx591f0f1lq-ca-test (144B)13252026/11/12 16:57:10 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."13262026/11/12 16:57:10 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."13272026/11/12 16:57:10 WARN Failed to register uploaded object key=log/r6dzz6vc5pgyjlnlzc29c1cq0rvr335w-ca-test.drv error="server returned 404: 404 page not found\n"1328--- PASS: TestFailedS3DeleteKeepsTombstone (0.73s)1329=== CONT TestMetricsCommit13302026/11/12 16:57:10 WARN Failed to register uploaded object key=nar/04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n.nar.zst error="server returned 404: 404 page not found\n"13312026/11/12 16:57:10 WARN Failed to register uploaded object key=rnwpn4pjgsv92zsxcrhvlfx591f0f1lq.ls error="server returned 404: 404 page not found\n"13322026/11/12 16:57:10 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign13332026/11/12 16:57:10 INFO Signed narinfos id=1 count=113342026/11/12 16:57:10 INFO Uploading 1 narinfos13352026/11/12 16:57:10 INFO Received complete push request method=POST path=/api/pushes/1/complete13362026/11/12 16:57:10 WARN Failed to register uploaded object key=rnwpn4pjgsv92zsxcrhvlfx591f0f1lq.narinfo error="server returned 404: 404 page not found\n"13372026/11/12 16:57:10 INFO Completed push id=1 objects=4 duration=7.603138ms13382026/11/12 16:57:10 INFO Upload complete. (104ms)13392026/11/12 16:57:10 INFO Received uploads request method=POST path=/api/pending_closures1340=== NAME TestClientCADerivations1341 client_ca_test.go:180: Narinfo contains CA field: StorePath: /build/TestClientCADerivations3473618602/001/store/rnwpn4pjgsv92zsxcrhvlfx591f0f1lq-ca-test1342 URL: nar/04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n.nar.zst1343 Compression: zstd1344 NarHash: sha256:04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n1345 NarSize: 1441346 References: 1347 Deriver: /build/TestClientCADerivations3473618602/001/store/r6dzz6vc5pgyjlnlzc29c1cq0rvr335w-ca-test.drv1348 CA: fixed:r:sha256:04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n1349 client_ca_test.go:185: Checking for realisation files in S3...1350 client_ca_test.go:207: No realisation files uploaded - this is expected for locally built CA derivations1351 client_ca_test.go:208: Locally built CA derivations don't have DrvOutput IDs until they're substituted from a cache1352=== NAME TestPinProtectsFromGC1353 client_integration_test.go:791: Pinned store path: /build/TestPinProtectsFromGC3223720116/001/store/08j53iw9p9w2p2cj6yghrzcbj9bbx2nv-pinned-file.txt1354 client_integration_test.go:792: Unpinned store path: /build/TestPinProtectsFromGC3223720116/001/store/fdjxasvh4hdwiig8ndy4zm95bx0ml9hd-unpinned-file.txt13552026/11/12 16:57:10 INFO Received uploads request method=POST path=/api/pending_closures13562026/11/12 16:57:10 INFO Received uploads request method=POST path=/api/pending_closures13572026/11/12 16:57:10 INFO Uploading 2 paths to 127.0.0.1 (1 already cached)13582026/11/12 16:57:10 INFO Uploading 7ylq8bvx8nvaab45rpvxgwbcjlxp3l83-b (216B)13592026/11/12 16:57:10 INFO Uploading gl3jb2p72a6j2mz8bz7a0w2ygn40cx6p-shared-dep (136B)13602026/11/12 16:57:10 WARN Failed to register uploaded object key=7srm9kc87h3l1cjvzyf5dmiyi75f6llv.ls error="server returned 404: 404 page not found\n"13612026/11/12 16:57:10 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"13622026/11/12 16:57:10 WARN Failed to register uploaded object key=7ylq8bvx8nvaab45rpvxgwbcjlxp3l83.ls error="server returned 404: 404 page not found\n"13632026/11/12 16:57:10 WARN Failed to register uploaded object key=gl3jb2p72a6j2mz8bz7a0w2ygn40cx6p.ls error="server returned 404: 404 page not found\n"13642026/11/12 16:57:10 WARN Failed to register uploaded object key=nar/0finmh4rc793sjkbnmhih38z4xpn954c28fr5wyxd7c8dhb4qr84.nar.zst error="server returned 404: 404 page not found\n"13652026/11/12 16:57:10 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign13662026/11/12 16:57:10 INFO Signed narinfos id=1 count=213672026/11/12 16:57:10 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign13682026/11/12 16:57:10 INFO Signed narinfos id=2 count=213692026/11/12 16:57:10 INFO Uploading 4 narinfos13702026/11/12 16:57:10 WARN Failed to register uploaded object key=7srm9kc87h3l1cjvzyf5dmiyi75f6llv.narinfo error="server returned 404: 404 page not found\n"13712026/11/12 16:57:10 WARN Failed to register uploaded object key=gl3jb2p72a6j2mz8bz7a0w2ygn40cx6p.narinfo error="server returned 404: 404 page not found\n"13722026/11/12 16:57:10 WARN Failed to register uploaded object key=7ylq8bvx8nvaab45rpvxgwbcjlxp3l83.narinfo error="server returned 404: 404 page not found\n"13732026/11/12 16:57:10 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete13742026/11/12 16:57:10 WARN Failed to register uploaded object key=gl3jb2p72a6j2mz8bz7a0w2ygn40cx6p.narinfo error="server returned 404: 404 page not found\n"13752026/11/12 16:57:10 INFO Aborted multipart uploads count=0 kept=013762026/11/12 16:57:10 WARN Force mode enabled - objects will be deleted immediately without grace period13772026/11/12 16:57:10 INFO Completed upload id=1 objects=6 duration=8.028259ms13782026/11/12 16:57:10 INFO Received push request method=POST path=/api/pushes13792026/11/12 16:57:10 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete13802026/11/12 16:57:10 INFO Completed upload id=2 objects=6 duration=1.36302ms13812026/11/12 16:57:10 INFO Upload complete. (77ms)1382=== NAME TestClientFallsBackToClosures1383 client_pushes_test.go:112: Retrieved narinfo from S3:1384 StorePath: /build/TestClientFallsBackToClosures1989735148/001/store/gl3jb2p72a6j2mz8bz7a0w2ygn40cx6p-shared-dep1385 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1386 Compression: zstd1387 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821388 NarSize: 1361389 References: 1390 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n13912026/11/12 16:57:10 INFO Uploading 2 paths to 127.0.0.1 (1 already cached)1392 client_pushes_test.go:112: Retrieved narinfo from S3:13932026/11/12 16:57:10 INFO Uploading q8ajlw62wl6xdhl3bh5dkw0xk6y99wh1-shared-dep (136B)1394 StorePath: /build/TestClientFallsBackToClosures1989735148/001/store/7srm9kc87h3l1cjvzyf5dmiyi75f6llv-a13952026/11/12 16:57:10 INFO Uploading wwxqas1yyi1mfwx9b82pcnbw4fdc8ka6-b (216B)1396 URL: nar/0finmh4rc793sjkbnmhih38z4xpn954c28fr5wyxd7c8dhb4qr84.nar.zst1397 Compression: zstd1398 NarHash: sha256:0finmh4rc793sjkbnmhih38z4xpn954c28fr5wyxd7c8dhb4qr841399 NarSize: 2161400 References: /build/TestClientFallsBackToClosures1989735148/001/store/gl3jb2p72a6j2mz8bz7a0w2ygn40cx6p-shared-dep1401 CA: text:sha256:06y1ldw2fkr654lfq0i8ffnlwks6764hl6c3pwyf5yriq0kycsic1402 client_pushes_test.go:112: Retrieved narinfo from S3:1403 StorePath: /build/TestClientFallsBackToClosures1989735148/001/store/7ylq8bvx8nvaab45rpvxgwbcjlxp3l83-b1404 URL: nar/0finmh4rc793sjkbnmhih38z4xpn954c28fr5wyxd7c8dhb4qr84.nar.zst1405 Compression: zstd1406 NarHash: sha256:0finmh4rc793sjkbnmhih38z4xpn954c28fr5wyxd7c8dhb4qr841407 NarSize: 2161408 References: /build/TestClientFallsBackToClosures1989735148/001/store/gl3jb2p72a6j2mz8bz7a0w2ygn40cx6p-shared-dep1409 CA: text:sha256:06y1ldw2fkr654lfq0i8ffnlwks6764hl6c3pwyf5yriq0kycsic14102026/11/12 16:57:10 WARN Failed to register uploaded object key=gsfrxcz6s6h99pn7jqckynakrdy3ppjq.ls error="server returned 404: 404 page not found\n"14112026/11/12 16:57:10 WARN Failed to register uploaded object key=q8ajlw62wl6xdhl3bh5dkw0xk6y99wh1.ls error="server returned 404: 404 page not found\n"14122026/11/12 16:57:10 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"14132026/11/12 16:57:10 WARN Failed to register uploaded object key=nar/03ki3kxq477fh9xi0hbbaxpi4wdqn08r6r1px6vqzhixb9id7qiy.nar.zst error="server returned 404: 404 page not found\n"14142026/11/12 16:57:10 WARN Failed to register uploaded object key=wwxqas1yyi1mfwx9b82pcnbw4fdc8ka6.ls error="server returned 404: 404 page not found\n"14152026/11/12 16:57:10 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign14162026/11/12 16:57:10 INFO Signed narinfos id=1 count=314172026/11/12 16:57:10 INFO Uploading 3 narinfos1418--- PASS: TestClientFallsBackToClosures (0.92s)1419=== CONT TestGCBugBareHashReferences14202026/11/12 16:57:11 INFO Received uploads request method=POST path=/api/pending_closures14212026/11/12 16:57:11 WARN Failed to register uploaded object key=q8ajlw62wl6xdhl3bh5dkw0xk6y99wh1.narinfo error="server returned 404: 404 page not found\n"14222026/11/12 16:57:11 WARN Failed to register uploaded object key=wwxqas1yyi1mfwx9b82pcnbw4fdc8ka6.narinfo error="server returned 404: 404 page not found\n"14232026/11/12 16:57:11 INFO Received complete push request method=POST path=/api/pushes/1/complete14242026/11/12 16:57:11 WARN Failed to register uploaded object key=gsfrxcz6s6h99pn7jqckynakrdy3ppjq.narinfo error="server returned 404: 404 page not found\n"1425--- PASS: TestPushWaitsForSweepHoldingRow (1.22s)1426=== CONT TestMetricsInventory1427=== NAME TestClientWithDependencies1428 client_integration_test.go:661: Built derivation: /build/TestClientWithDependencies1916622802/001/store/0ds3av84imcmvrz265x46h5ink01vk2n-test-script1429--- PASS: TestTombstonedObjectOfferedWithoutWaiting (0.59s)1430=== CONT TestGCAdvisoryLockBlocksConcurrentRun14312026/11/12 16:57:11 INFO Received push request method=POST path=/api/pushes1432=== NAME TestClientCADerivations1433 client_ca_test.go:258: nix copy output: warning: you don't have Internet access; disabling some network-dependent features1434 warning: failed to create TLS context for AWS credential providers; SSO, STS WebIdentity, and ECS container authentication will be unavailable1435 error: binary cache 's3://bucket52?endpoint=http://localhost:41275®ion=eu-west-1' is for Nix stores with prefix '/nix/store', not '/build/TestClientCADerivations3473618602/001/store'1436 client_ca_test.go:259: nix copy failed (might be expected with isolated stores): exit status 114372026/11/12 16:57:11 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)14382026/11/12 16:57:11 INFO Uploading 08j53iw9p9w2p2cj6yghrzcbj9bbx2nv-pinned-file.txt (128B)14392026/11/12 16:57:11 INFO Completed push id=1 objects=8 duration=9.654049ms14402026/11/12 16:57:11 INFO Upload complete. (74ms)1441=== NAME TestClientMultipleUploads1442 client_integration_test.go:406: Created store path 0: /build/TestClientMultipleUploads1209940428/001/store/bjznj39l1mhmpiqklm5dm8xqgkbfmijm-test-file-0.txt1443=== NAME TestClientPushesUseOnePush1444 client_pushes_test.go:97: Retrieved narinfo from S3:1445 StorePath: /build/TestClientPushesUseOnePush3295204201/001/store/q8ajlw62wl6xdhl3bh5dkw0xk6y99wh1-shared-dep1446 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1447 Compression: zstd1448 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821449 NarSize: 1361450 References: 14512026/11/12 16:57:11 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=01452 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n1453--- PASS: TestClientCADerivations (1.11s)1454=== CONT TestService_AuthMiddleware_OIDC14552026/11/12 16:57:11 INFO Vacuumed table table=pending_closures14562026/11/12 16:57:11 INFO Vacuumed table table=commit_receipts14572026/11/12 16:57:11 INFO Vacuumed table table=pending_objects14582026/11/12 16:57:11 INFO Vacuumed table table=multipart_uploads14592026/11/12 16:57:11 INFO Vacuumed table table=closures1460=== NAME TestClientPushesUseOnePush1461 client_pushes_test.go:97: Retrieved narinfo from S3:1462 StorePath: /build/TestClientPushesUseOnePush3295204201/001/store/gsfrxcz6s6h99pn7jqckynakrdy3ppjq-a1463 URL: nar/03ki3kxq477fh9xi0hbbaxpi4wdqn08r6r1px6vqzhixb9id7qiy.nar.zst1464 Compression: zstd1465 NarHash: sha256:03ki3kxq477fh9xi0hbbaxpi4wdqn08r6r1px6vqzhixb9id7qiy1466 NarSize: 2161467 References: /build/TestClientPushesUseOnePush3295204201/001/store/q8ajlw62wl6xdhl3bh5dkw0xk6y99wh1-shared-dep1468 CA: text:sha256:0qw33r827yjbnsk4bwki40jblr1zsixv5bvqnid90xlxbsrnila014692026/11/12 16:57:11 WARN Failed to register uploaded object key=nar/0qmhsg42pqsz7x6hh02j8gznnac3yi3qpjhrqbwvz88i194jbb5s.nar.zst error="server returned 404: 404 page not found\n"14702026/11/12 16:57:11 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign14712026/11/12 16:57:11 WARN Failed to register uploaded object key=08j53iw9p9w2p2cj6yghrzcbj9bbx2nv.ls error="server returned 404: 404 page not found\n"14722026/11/12 16:57:11 INFO Signed narinfos id=1 count=114732026/11/12 16:57:11 INFO Uploading 1 narinfos14742026/11/12 16:57:11 INFO Received push request method=POST path=/api/pushes1475 client_pushes_test.go:97: Retrieved narinfo from S3:1476 StorePath: /build/TestClientPushesUseOnePush3295204201/001/store/wwxqas1yyi1mfwx9b82pcnbw4fdc8ka6-b1477 URL: nar/03ki3kxq477fh9xi0hbbaxpi4wdqn08r6r1px6vqzhixb9id7qiy.nar.zst1478 Compression: zstd1479 NarHash: sha256:03ki3kxq477fh9xi0hbbaxpi4wdqn08r6r1px6vqzhixb9id7qiy1480 NarSize: 2161481 References: /build/TestClientPushesUseOnePush3295204201/001/store/q8ajlw62wl6xdhl3bh5dkw0xk6y99wh1-shared-dep1482 CA: text:sha256:0qw33r827yjbnsk4bwki40jblr1zsixv5bvqnid90xlxbsrnila014832026/11/12 16:57:11 INFO Received complete push request method=POST path=/api/pushes/1/complete14842026/11/12 16:57:11 WARN Failed to register uploaded object key=08j53iw9p9w2p2cj6yghrzcbj9bbx2nv.narinfo error="server returned 404: 404 page not found\n"1485--- PASS: TestClientPushesUseOnePush (0.87s)1486=== CONT TestNARDeduplicationMetadataUploadBug14872026/11/12 16:57:11 INFO Uploading 2 paths to 127.0.0.1 (0 already cached)14882026/11/12 16:57:11 INFO Uploading 64zqg9qcgkf8yx8s0nrnrdl7wkb9jr47-top (224B)14892026/11/12 16:57:11 INFO Uploading 3zng1izmzj8hgqfsk4vxy2vzzmiga55y-shared-dep (136B)14902026/11/12 16:57:11 INFO Vacuumed table table=objects1491--- PASS: TestGCSweepDeliversEachKeyOnce (0.64s)1492=== CONT TestCacheStatsHandler14932026/11/12 16:57:11 WARN Failed to register uploaded object key=nar/1a69jppxis1p24lhzdv4qgngwjjs0g1wb9cqzl20mzck2md4z2vr.nar.zst error="server returned 404: 404 page not found\n"14942026/11/12 16:57:11 WARN Failed to register uploaded object key=3zng1izmzj8hgqfsk4vxy2vzzmiga55y.ls error="server returned 404: 404 page not found\n"14952026/11/12 16:57:11 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"14962026/11/12 16:57:11 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign14972026/11/12 16:57:11 WARN Failed to register uploaded object key=64zqg9qcgkf8yx8s0nrnrdl7wkb9jr47.ls error="server returned 404: 404 page not found\n"14982026/11/12 16:57:11 INFO Signed narinfos id=1 count=214992026/11/12 16:57:11 INFO Uploading 2 narinfos15002026/11/12 16:57:11 INFO Completed push id=1 objects=3 duration=11.122968ms15012026/11/12 16:57:11 INFO Upload complete. (68ms)1502=== NAME TestClientWithDependencies1503 client_integration_test.go:663: Found 1 dependencies (including self)15042026/11/12 16:57:11 WARN Failed to register uploaded object key=64zqg9qcgkf8yx8s0nrnrdl7wkb9jr47.narinfo error="server returned 404: 404 page not found\n"15052026/11/12 16:57:11 INFO Received complete push request method=POST path=/api/pushes/1/complete15062026/11/12 16:57:11 WARN Failed to register uploaded object key=3zng1izmzj8hgqfsk4vxy2vzzmiga55y.narinfo error="server returned 404: 404 page not found\n"1507=== NAME TestClientMultipleUploads1508 client_integration_test.go:406: Created store path 1: /build/TestClientMultipleUploads1209940428/001/store/w77v4dfx42ylc4zfvbcpdqza4hkfgizh-test-file-1.txt15092026/11/12 16:57:11 INFO Received uploads request method=POST path=/api/pending_closures15102026/11/12 16:57:11 INFO Completed push id=1 objects=6 duration=7.571568ms15112026/11/12 16:57:11 INFO Upload complete. (68ms)1512=== NAME TestClientSharedPathCommittedMidPush1513 client_integration_test.go:740: Retrieved narinfo from S3:1514 StorePath: /build/TestClientSharedPathCommittedMidPush1722459108/001/store/3zng1izmzj8hgqfsk4vxy2vzzmiga55y-shared-dep1515 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1516 Compression: zstd1517 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821518 NarSize: 1361519 References: 1520 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n1521 client_integration_test.go:740: Retrieved narinfo from S3:1522 StorePath: /build/TestClientSharedPathCommittedMidPush1722459108/001/store/64zqg9qcgkf8yx8s0nrnrdl7wkb9jr47-top1523 URL: nar/1a69jppxis1p24lhzdv4qgngwjjs0g1wb9cqzl20mzck2md4z2vr.nar.zst1524 Compression: zstd1525 NarHash: sha256:1a69jppxis1p24lhzdv4qgngwjjs0g1wb9cqzl20mzck2md4z2vr1526 NarSize: 2241527 References: /build/TestClientSharedPathCommittedMidPush1722459108/001/store/3zng1izmzj8hgqfsk4vxy2vzzmiga55y-shared-dep1528 CA: text:sha256:15yc4268zg8smsk9maanq46ja57b0ak5j1fc3f57mr38nzb9qxld1529--- PASS: TestClientSharedPathCommittedMidPush (0.79s)1530=== CONT TestCreatePendingClosureRejectsOversizedNAR15312026/11/12 16:57:11 INFO Received uploads request method=POST path=/api/pending_closures1532--- PASS: TestCreatePendingClosureRejectsOversizedNAR (0.00s)1533=== CONT TestLeadEndsOnShutdown1534=== NAME TestClientIntegration1535 client_integration_test.go:292: Created store path: /build/TestClientIntegration146189624/002/store/npac04ayvddmhrx4dsklyvmh955zhavk-test-file.txt1536=== NAME TestClientMultipleUploads1537 client_integration_test.go:406: Created store path 2: /build/TestClientMultipleUploads1209940428/001/store/v7m6zrazkpx6mnjrj7379sn51mzxrafq-test-file-2.txt1538--- PASS: TestCreatePendingClosureVerifyS3FailureReleasesConnection (0.81s)1539=== CONT TestCacheConfigHandler1540=== RUN TestCacheConfigHandler/full_config,_no_issuer1541=== PAUSE TestCacheConfigHandler/full_config,_no_issuer1542=== RUN TestCacheConfigHandler/no_cache_url_configured1543=== PAUSE TestCacheConfigHandler/no_cache_url_configured1544=== RUN TestCacheConfigHandler/no_signing_keys1545=== PAUSE TestCacheConfigHandler/no_signing_keys1546=== RUN TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1547=== PAUSE TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1548=== CONT TestCacheConfigHandlerMaxNarSize1549--- PASS: TestCacheConfigHandlerMaxNarSize (0.00s)1550=== CONT TestService_ReadScope_PublicByDefault15512026/11/12 16:57:11 INFO Received push request method=POST path=/api/pushes15522026/11/12 16:57:11 INFO Received uploads request method=POST path=/api/pending_closures15532026/11/12 16:57:11 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:41427/oidc15542026/11/12 16:57:11 INFO Received push request method=POST path=/api/pushes15552026/11/12 16:57:11 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)15562026/11/12 16:57:11 INFO Uploading fdjxasvh4hdwiig8ndy4zm95bx0ml9hd-unpinned-file.txt (128B)15572026/11/12 16:57:11 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)15582026/11/12 16:57:11 INFO Uploading 0ds3av84imcmvrz265x46h5ink01vk2n-test-script (136B)15592026/11/12 16:57:11 WARN Failed to register uploaded object key=fdjxasvh4hdwiig8ndy4zm95bx0ml9hd.ls error="server returned 404: 404 page not found\n"15602026/11/12 16:57:11 WARN Failed to register uploaded object key=nar/0gh0wdnr6gmk4r5sfzzh12isnl14rgmkrwy1ws41hpsrlndclg5h.nar.zst error="server returned 404: 404 page not found\n"15612026/11/12 16:57:11 INFO Received sign narinfos request method=POST path=/api/pushes/2/sign15622026/11/12 16:57:11 INFO Signed narinfos id=2 count=115632026/11/12 16:57:11 INFO Uploading 1 narinfos15642026/11/12 16:57:11 WARN Failed to register uploaded object key=log/waczk1xmp8vv0ab6f73x6mpa09v40qz8-test-script.drv error="server returned 404: 404 page not found\n"15652026/11/12 16:57:11 WARN Failed to register uploaded object key=0ds3av84imcmvrz265x46h5ink01vk2n.ls error="server returned 404: 404 page not found\n"15662026/11/12 16:57:11 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign15672026/11/12 16:57:11 WARN Failed to register uploaded object key=nar/1s7yghixzpinyqm7g59kyn3sav81pmnsdi5v63m6x4r2vacy907j.nar.zst error="server returned 404: 404 page not found\n"15682026/11/12 16:57:11 INFO Signed narinfos id=1 count=115692026/11/12 16:57:11 INFO Uploading 1 narinfos15702026/11/12 16:57:11 INFO Received complete push request method=POST path=/api/pushes/2/complete15712026/11/12 16:57:11 WARN Failed to register uploaded object key=fdjxasvh4hdwiig8ndy4zm95bx0ml9hd.narinfo error="server returned 404: 404 page not found\n"15722026/11/12 16:57:11 INFO Received complete push request method=POST path=/api/pushes/1/complete15732026/11/12 16:57:11 WARN Failed to register uploaded object key=0ds3av84imcmvrz265x46h5ink01vk2n.narinfo error="server returned 404: 404 page not found\n"15742026/11/12 16:57:11 INFO Completed push id=2 objects=3 duration=3.971319ms15752026/11/12 16:57:11 INFO Upload complete. (88ms)15762026/11/12 16:57:11 INFO Received uploads request method=POST path=/api/pending_closures15772026/11/12 16:57:11 INFO Aborted multipart uploads count=0 kept=015782026/11/12 16:57:11 INFO Completed push id=1 objects=4 duration=5.721189ms15792026/11/12 16:57:11 INFO Upload complete. (93ms)15802026/11/12 16:57:11 WARN Force mode enabled - objects will be deleted immediately without grace period1581--- PASS: TestDeduplicatedObjectsRecordedAsPending (0.58s)1582=== CONT TestEnsureAbortUploadsRule15832026/11/12 16:57:11 INFO Received cleanup request method=DELETE path=/api/pending_closures15842026/11/12 16:57:11 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=015852026/11/12 16:57:11 WARN Failed to abort upload, keeping its closure key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst error="Get \"http://127.0.0.1:1/bucket67/?location=\": dial tcp 127.0.0.1:1: connect: connection refused" code=""15862026/11/12 16:57:11 INFO Aborted multipart uploads count=0 kept=115872026/11/12 16:57:11 INFO Vacuumed table table=pending_closures15882026/11/12 16:57:11 INFO Vacuumed table table=commit_receipts15892026/11/12 16:57:11 INFO Aborted multipart uploads count=0 kept=015902026/11/12 16:57:11 INFO Vacuumed table table=pending_objects15912026/11/12 16:57:11 INFO Vacuumed table table=multipart_uploads15922026/11/12 16:57:11 INFO Vacuumed table table=closures15932026/11/12 16:57:11 INFO Received push request method=POST path=/api/pushes15942026/11/12 16:57:11 INFO Received cleanup request method=DELETE path=/api/pending_closures15952026/11/12 16:57:11 INFO Received push request method=POST path=/api/pushes15962026/11/12 16:57:11 INFO Aborted multipart uploads count=1 kept=015972026/11/12 16:57:11 INFO Vacuumed table table=objects15982026/11/12 16:57:11 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)15992026/11/12 16:57:11 INFO Uploading npac04ayvddmhrx4dsklyvmh955zhavk-test-file.txt (152B)16002026/11/12 16:57:11 INFO Uploading 3 paths to 127.0.0.1 (0 already cached)16012026/11/12 16:57:11 INFO Uploading bjznj39l1mhmpiqklm5dm8xqgkbfmijm-test-file-0.txt (160B)1602--- PASS: TestMultipartCleanup (0.69s)16032026/11/12 16:57:11 INFO Uploading w77v4dfx42ylc4zfvbcpdqza4hkfgizh-test-file-1.txt (160B)1604=== CONT TestService_RequireScope_OIDC16052026/11/12 16:57:11 INFO Uploading v7m6zrazkpx6mnjrj7379sn51mzxrafq-test-file-2.txt (160B)16062026/11/12 16:57:11 WARN Force mode enabled - objects will be deleted immediately without grace period16072026/11/12 16:57:11 INFO Starting cleanup of old closures method=DELETE path=/api/closures16082026/11/12 16:57:11 INFO Garbage collection started16092026/11/12 16:57:11 WARN Failed to register uploaded object key=nar/13p9pqi0ga6fx2apcwls9hjnv7nmf740ggpn5zvfzdsp5vz2bbg4.nar.zst error="server returned 404: 404 page not found\n"16102026/11/12 16:57:11 WARN Failed to register uploaded object key=nar/11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1.nar.zst error="server returned 404: 404 page not found\n"16112026/11/12 16:57:11 WARN Failed to register uploaded object key=bjznj39l1mhmpiqklm5dm8xqgkbfmijm.ls error="server returned 404: 404 page not found\n"16122026/11/12 16:57:11 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign16132026/11/12 16:57:11 WARN Failed to register uploaded object key=npac04ayvddmhrx4dsklyvmh955zhavk.ls error="server returned 404: 404 page not found\n"16142026/11/12 16:57:11 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=016152026/11/12 16:57:11 WARN Failed to register uploaded object key=nar/1823g6vx4q5zqvr73m7lkq1wpk40fin7bsc792di430yk7sg4lka.nar.zst error="server returned 404: 404 page not found\n"16162026/11/12 16:57:11 INFO Signed narinfos id=1 count=116172026/11/12 16:57:11 WARN Failed to register uploaded object key=w77v4dfx42ylc4zfvbcpdqza4hkfgizh.ls error="server returned 404: 404 page not found\n"1618--- PASS: TestGCSweepSkipsPendingObjects (0.66s)16192026/11/12 16:57:11 INFO Uploading 1 narinfos1620=== CONT TestGenerateLandingPage16212026/11/12 16:57:11 WARN Failed to register uploaded object key=nar/0ahq3izw3h0b0c08p3xdfclh758fwxnp85ccvflasb4vnfsv50ri.nar.zst error="server returned 404: 404 page not found\n"16222026/11/12 16:57:11 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign16232026/11/12 16:57:11 WARN Failed to register uploaded object key=v7m6zrazkpx6mnjrj7379sn51mzxrafq.ls error="server returned 404: 404 page not found\n"16242026/11/12 16:57:11 INFO Signed narinfos id=1 count=316252026/11/12 16:57:11 INFO Uploading 3 narinfos1626--- PASS: TestGenerateLandingPage (0.00s)1627=== CONT TestService_readinessHandler16282026/11/12 16:57:11 INFO Received complete push request method=POST path=/api/pushes/1/complete16292026/11/12 16:57:11 WARN Failed to register uploaded object key=npac04ayvddmhrx4dsklyvmh955zhavk.narinfo error="server returned 404: 404 page not found\n"16302026/11/12 16:57:11 INFO Vacuumed table table=pending_closures16312026/11/12 16:57:11 WARN Failed to register uploaded object key=bjznj39l1mhmpiqklm5dm8xqgkbfmijm.narinfo error="server returned 404: 404 page not found\n"16322026/11/12 16:57:11 WARN Failed to register uploaded object key=w77v4dfx42ylc4zfvbcpdqza4hkfgizh.narinfo error="server returned 404: 404 page not found\n"16332026/11/12 16:57:11 INFO Received complete push request method=POST path=/api/pushes/1/complete16342026/11/12 16:57:11 WARN Failed to register uploaded object key=v7m6zrazkpx6mnjrj7379sn51mzxrafq.narinfo error="server returned 404: 404 page not found\n"16352026/11/12 16:57:11 INFO Aborted multipart uploads count=0 kept=016362026/11/12 16:57:11 INFO Vacuumed table table=commit_receipts16372026/11/12 16:57:11 WARN mTLS auth: subject not in bound subjects subject="CN=reader"16382026/11/12 16:57:11 WARN mTLS auth: subject not in bound subjects subject="CN=reader"1639--- PASS: TestService_NativeMTLS (0.42s)1640=== CONT TestService_healthCheckHandler16412026/11/12 16:57:11 INFO Completed push id=1 objects=3 duration=11.246678ms16422026/11/12 16:57:11 INFO Upload complete. (82ms)16432026/11/12 16:57:11 INFO Vacuumed table table=pending_objects16442026/11/12 16:57:11 WARN Force mode enabled - objects will be deleted immediately without grace period16452026/11/12 16:57:11 INFO Received uploads request method=POST path=/api/pending_closures16462026/11/12 16:57:11 INFO Vacuumed table table=multipart_uploads16472026/11/12 16:57:11 INFO Completed push id=1 objects=9 duration=12.208677ms16482026/11/12 16:57:11 INFO Upload complete. (85ms)1649=== NAME TestClientMultipleUploads1650 client_integration_test.go:417: Uploaded 3 paths in 131.457794ms16512026/11/12 16:57:11 INFO Vacuumed table table=closures16522026/11/12 16:57:11 INFO Vacuumed table table=objects1653=== NAME TestClientWithDependencies1654 client_integration_test.go:677: Skipping nix copy test - isolated store (/build/TestClientWithDependencies1916622802/001/store) requires matching store prefix1655--- PASS: TestClientWithDependencies (0.90s)1656=== CONT TestLeadIncumbentAnnouncesAtOnce1657--- PASS: TestClientMultipleUploads (0.83s)1658=== CONT TestLeadElectsOneAndHandsOver16592026/11/12 16:57:11 INFO Aborted multipart uploads count=0 kept=016602026/11/12 16:57:11 WARN Force mode enabled - objects will be deleted immediately without grace period16612026/11/12 16:57:11 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=016622026/11/12 16:57:11 INFO Vacuumed table table=pending_closures16632026/11/12 16:57:11 INFO Vacuumed table table=commit_receipts16642026/11/12 16:57:11 INFO Vacuumed table table=pending_objects16652026/11/12 16:57:11 INFO Vacuumed table table=multipart_uploads16662026/11/12 16:57:11 INFO All 1 paths already cached16672026/11/12 16:57:11 INFO Vacuumed table table=closures16682026/11/12 16:57:11 INFO Vacuumed table table=objects1669=== NAME TestClientIntegration1670 client_integration_test.go:318: Retrieved narinfo from S3:1671 StorePath: /build/TestClientIntegration146189624/002/store/npac04ayvddmhrx4dsklyvmh955zhavk-test-file.txt1672 URL: nar/11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1.nar.zst1673 Compression: zstd1674 NarHash: sha256:11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk11675 NarSize: 1521676 References: 1677 CA: fixed:r:sha256:11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk116782026/11/12 16:57:11 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=01679 client_integration_test.go:319: Retrieved .ls file from S3 (compressed size: 77 bytes)1680 client_integration_test.go:319: Decompressed .ls content (64 bytes):1681 {"version":1,"root":{"type":"regular","size":39,"narOffset":96}}16822026/11/12 16:57:11 INFO Received push request method=POST path=/api/pushes16832026/11/12 16:57:11 INFO Vacuumed table table=pending_closures1684--- PASS: TestGCMetrics (0.46s)1685=== CONT TestConnectWaitsForAPeerMigration16862026/11/12 16:57:11 INFO Vacuumed table table=commit_receipts16872026/11/12 16:57:11 INFO Vacuumed table table=pending_objects16882026/11/12 16:57:11 INFO Vacuumed table table=multipart_uploads16892026/11/12 16:57:11 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:37993/oidc16902026/11/12 16:57:11 INFO Vacuumed table table=closures16912026/11/12 16:57:11 INFO Vacuumed table table=objects16922026/11/12 16:57:11 INFO Aborted multipart uploads count=0 kept=016932026/11/12 16:57:11 INFO Received complete push request method=POST path=/api/pushes/1/complete16942026/11/12 16:57:11 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=016952026/11/12 16:57:11 INFO Vacuumed table table=pending_closures16962026/11/12 16:57:11 INFO Vacuumed table table=commit_receipts16972026/11/12 16:57:11 INFO Completed push id=1 objects=4 duration=10.467068ms16982026/11/12 16:57:11 INFO Vacuumed table table=pending_objects16992026/11/12 16:57:11 INFO Vacuumed table table=multipart_uploads17002026/11/12 16:57:11 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete17012026/11/12 16:57:11 INFO Completed upload id=2 objects=2 duration=1.54355ms17022026/11/12 16:57:11 INFO Vacuumed table table=closures17032026/11/12 16:57:11 INFO Vacuumed table table=objects1704--- PASS: TestMetricsCommit (0.47s)1705=== CONT TestResolveDBConnectionString1706=== RUN TestResolveDBConnectionString/flag_wins1707=== PAUSE TestResolveDBConnectionString/flag_wins1708=== RUN TestResolveDBConnectionString/file_when_flag_empty1709=== PAUSE TestResolveDBConnectionString/file_when_flag_empty1710=== RUN TestResolveDBConnectionString/missing_file_is_an_error1711=== PAUSE TestResolveDBConnectionString/missing_file_is_an_error1712=== RUN TestResolveDBConnectionString/PGHOST_allows_empty1713=== PAUSE TestResolveDBConnectionString/PGHOST_allows_empty1714=== RUN TestResolveDBConnectionString/nothing_configured1715=== PAUSE TestResolveDBConnectionString/nothing_configured1716=== CONT TestLeadUnannouncedStandbyYieldsToIncumbent17172026/11/12 16:57:11 INFO Aborted multipart uploads count=0 kept=017182026/11/12 16:57:11 WARN Force mode enabled - objects will be deleted immediately without grace period17192026/11/12 16:57:11 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=017202026/11/12 16:57:11 INFO Vacuumed table table=pending_closures17212026/11/12 16:57:11 INFO Vacuumed table table=commit_receipts17222026/11/12 16:57:11 INFO Vacuumed table table=pending_objects17232026/11/12 16:57:11 INFO Vacuumed table table=multipart_uploads17242026/11/12 16:57:11 INFO Vacuumed table table=closures17252026/11/12 16:57:11 INFO Received push request method=POST path=/api/pushes1726--- PASS: TestMetricsInventory (0.35s)1727=== CONT TestService_AuthMiddleware_MTLSBoundSubjects17282026/11/12 16:57:11 INFO Vacuumed table table=objects1729--- PASS: TestPushDedupSurvivesConcurrentGC (0.70s)1730=== CONT TestService_AuthMiddleware17312026/11/12 16:57:11 INFO Object in database but missing from S3 key=npac04ayvddmhrx4dsklyvmh955zhavk.ls17322026/11/12 16:57:11 WARN Found objects in DB but missing from S3, will re-upload count=117332026/11/12 16:57:11 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)17342026/11/12 16:57:11 INFO Received complete push request method=POST path=/api/pushes/2/complete17352026/11/12 16:57:11 WARN Failed to register uploaded object key=npac04ayvddmhrx4dsklyvmh955zhavk.ls error="server returned 404: 404 page not found\n"17362026/11/12 16:57:11 INFO Completed push id=2 objects=3 duration=1.68057ms17372026/11/12 16:57:11 INFO Upload complete. (47ms)17382026/11/12 16:57:11 ERROR failed to check GC advisory lock error="query pg_locks: failed to connect to `user=nixbld database=niks3`: /nonexistent/.s.PGSQL.5432 (/nonexistent): dial error: dial unix /nonexistent/.s.PGSQL.5432: connect: no such file or directory"1739=== NAME TestClientIntegration1740 client_integration_test.go:336: Retrieved .ls file from S3 (compressed size: 62 bytes)1741 client_integration_test.go:336: Decompressed .ls content (49 bytes):1742 {"version":1,"root":{"type":"regular","size":39}}1743--- PASS: TestGCAdvisoryLockBlocksConcurrentRun (0.37s)1744=== CONT TestService_AuthMiddleware_MTLSProxyHeader1745--- PASS: TestCacheStatsHandler (0.39s)1746=== CONT TestService_ReadAuthMiddleware17472026/11/12 16:57:11 INFO lead: acquired remote=192.0.2.1:12341748=== NAME TestNARDeduplicationMetadataUploadBug1749 metadata_upload_test.go:48: First store path: /build/TestNARDeduplicationMetadataUploadBug4021936798/001/store/sg7mkmkad0cdnnls806i5xxjxmxj9hi5-file1.txt1750=== RUN TestService_AuthMiddleware_OIDC/valid_OIDC_token1751=== PAUSE TestService_AuthMiddleware_OIDC/valid_OIDC_token1752=== RUN TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1753=== PAUSE TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1754=== RUN TestService_AuthMiddleware_OIDC/malformed_token_rejected1755=== PAUSE TestService_AuthMiddleware_OIDC/malformed_token_rejected1756=== RUN TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1757=== PAUSE TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1758=== CONT TestCommitCanBeRepeatedUntilCleanup/push17592026/11/12 16:57:11 INFO Received push request method=POST path=/api/pushes17602026/11/12 16:57:11 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)17612026/11/12 16:57:11 INFO Received complete push request method=POST path=/api/pushes/3/complete17622026/11/12 16:57:11 WARN Failed to register uploaded object key=npac04ayvddmhrx4dsklyvmh955zhavk.ls error="server returned 404: 404 page not found\n"17632026/11/12 16:57:11 INFO Completed push id=3 objects=3 duration=2.241899ms17642026/11/12 16:57:11 INFO Upload complete. (49ms)1765=== NAME TestClientIntegration1766 client_integration_test.go:353: Retrieved .ls file from S3 (compressed size: 62 bytes)1767 client_integration_test.go:353: Decompressed .ls content (49 bytes):1768 {"version":1,"root":{"type":"regular","size":39}}1769 client_integration_test.go:364: Testing garbage collection...1770--- PASS: TestService_ReadScope_PublicByDefault (0.41s)1771=== CONT TestCommitCanBeRepeatedUntilCleanup/pending_closure17722026/11/12 16:57:11 INFO Starting cleanup of old closures method=DELETE path=/api/closures17732026/11/12 16:57:11 INFO Garbage collection started17742026/11/12 16:57:11 INFO Received push request method=POST path=/api/pushes1775--- PASS: TestGCBugBareHashReferences (0.55s)1776=== CONT TestProxyWriteTimeout/narinfo1777=== CONT TestProxyWriteTimeout/unknown_size1778=== CONT TestProxyWriteTimeout/10_GiB_nar1779=== CONT TestProxyWriteTimeout/1_GiB_nar1780--- PASS: TestProxyWriteTimeout (0.10s)1781 --- PASS: TestProxyWriteTimeout/narinfo (0.00s)1782 --- PASS: TestProxyWriteTimeout/unknown_size (0.00s)1783 --- PASS: TestProxyWriteTimeout/10_GiB_nar (0.00s)1784 --- PASS: TestProxyWriteTimeout/1_GiB_nar (0.00s)1785=== CONT TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed17862026/11/12 16:57:11 INFO Aborted multipart uploads count=0 kept=017872026/11/12 16:57:11 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)17882026/11/12 16:57:11 INFO Uploading sg7mkmkad0cdnnls806i5xxjxmxj9hi5-file1.txt (160B)17892026/11/12 16:57:11 INFO Installed lifecycle rule aborting incomplete NAR uploads bucket=bucket82 days=21790--- PASS: TestEnsureAbortUploadsRule (0.39s)1791=== CONT TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed17922026/11/12 16:57:11 WARN Force mode enabled - objects will be deleted immediately without grace period17932026/11/12 16:57:11 WARN Failed to register uploaded object key=nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst error="server returned 404: 404 page not found\n"17942026/11/12 16:57:11 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign17952026/11/12 16:57:11 WARN Failed to register uploaded object key=sg7mkmkad0cdnnls806i5xxjxmxj9hi5.ls error="server returned 404: 404 page not found\n"17962026/11/12 16:57:11 INFO Signed narinfos id=1 count=117972026/11/12 16:57:11 INFO Uploading 1 narinfos17982026/11/12 16:57:11 INFO Received complete push request method=POST path=/api/pushes/1/complete17992026/11/12 16:57:11 WARN Failed to register uploaded object key=sg7mkmkad0cdnnls806i5xxjxmxj9hi5.narinfo error="server returned 404: 404 page not found\n"1800--- PASS: TestService_healthCheckHandler (0.36s)1801=== CONT TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed18022026/11/12 16:57:11 INFO Completed push id=1 objects=3 duration=11.204337ms18032026/11/12 16:57:11 INFO Upload complete. (75ms)1804=== NAME TestNARDeduplicationMetadataUploadBug1805 metadata_upload_test.go:54: Retrieved narinfo from S3:1806 StorePath: /build/TestNARDeduplicationMetadataUploadBug4021936798/001/store/sg7mkmkad0cdnnls806i5xxjxmxj9hi5-file1.txt1807 URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst1808 Compression: zstd1809 NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1810 NarSize: 1601811 References: 1812 CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1813 metadata_upload_test.go:55: Retrieved .ls file from S3 (compressed size: 77 bytes)1814 metadata_upload_test.go:55: Decompressed .ls content (64 bytes):1815 {"version":1,"root":{"type":"regular","size":44,"narOffset":96}}18162026/11/12 16:57:11 WARN readiness check failed error="closed pool"1817--- PASS: TestService_readinessHandler (0.40s)1818=== CONT TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost18192026/11/12 16:57:11 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=018202026/11/12 16:57:11 INFO Vacuumed table table=pending_closures18212026/11/12 16:57:11 INFO Vacuumed table table=commit_receipts1822=== NAME TestNARDeduplicationMetadataUploadBug1823 metadata_upload_test.go:64: Second store path (same content): /build/TestNARDeduplicationMetadataUploadBug4021936798/001/store/1a67gp0af22w2wv2wfgja1svx1bqxcki-file2.txt18242026/11/12 16:57:11 INFO Vacuumed table table=pending_objects18252026/11/12 16:57:11 INFO Vacuumed table table=multipart_uploads18262026/11/12 16:57:11 INFO Vacuumed table table=closures18272026/11/12 16:57:11 INFO Vacuumed table table=objects18282026/11/12 16:57:11 INFO lead: acquired remote=192.0.2.1:123418292026/11/12 16:57:11 INFO lead: released remote=192.0.2.1:12341830--- PASS: TestLeadIncumbentAnnouncesAtOnce (0.41s)1831=== CONT TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed18322026/11/12 16:57:11 INFO lead: acquired remote=192.0.2.1:123418332026/11/12 16:57:11 INFO Received push request method=POST path=/api/pushes18342026/11/12 16:57:11 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)18352026/11/12 16:57:11 INFO Received sign narinfos request method=POST path=/api/pushes/2/sign18362026/11/12 16:57:11 WARN Failed to register uploaded object key=1a67gp0af22w2wv2wfgja1svx1bqxcki.ls error="server returned 404: 404 page not found\n"18372026/11/12 16:57:11 INFO Signed narinfos id=2 count=11838=== RUN TestService_RequireScope_OIDC/builder_may_write18392026/11/12 16:57:11 INFO Uploading 1 narinfos1840=== PAUSE TestService_RequireScope_OIDC/builder_may_write1841=== RUN TestService_RequireScope_OIDC/builder_may_not_admin1842=== PAUSE TestService_RequireScope_OIDC/builder_may_not_admin1843=== RUN TestService_RequireScope_OIDC/ops_may_admin1844=== PAUSE TestService_RequireScope_OIDC/ops_may_admin1845=== RUN TestService_RequireScope_OIDC/ops_may_not_write1846=== PAUSE TestService_RequireScope_OIDC/ops_may_not_write1847=== RUN TestService_RequireScope_OIDC/reader_may_not_write1848=== PAUSE TestService_RequireScope_OIDC/reader_may_not_write1849=== RUN TestService_RequireScope_OIDC/static_token_may_admin1850=== PAUSE TestService_RequireScope_OIDC/static_token_may_admin1851=== RUN TestService_RequireScope_OIDC/static_token_may_write1852=== PAUSE TestService_RequireScope_OIDC/static_token_may_write1853=== RUN TestService_RequireScope_OIDC/reader_may_read1854=== PAUSE TestService_RequireScope_OIDC/reader_may_read1855=== RUN TestService_RequireScope_OIDC/writer_implies_read1856=== PAUSE TestService_RequireScope_OIDC/writer_implies_read1857=== RUN TestService_RequireScope_OIDC/anonymous_may_not_read1858=== PAUSE TestService_RequireScope_OIDC/anonymous_may_not_read1859=== RUN TestService_RequireScope_OIDC/reader_may_read_with_netrc1860=== PAUSE TestService_RequireScope_OIDC/reader_may_read_with_netrc1861=== RUN TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1862=== PAUSE TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1863=== RUN TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read1864=== PAUSE TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read1865=== RUN TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1866=== PAUSE TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1867=== CONT TestPendingClosureWriteTimeout/empty1868=== CONT TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline18692026/11/12 16:57:11 INFO Received complete push request method=POST path=/api/pushes/2/complete18702026/11/12 16:57:11 WARN Failed to register uploaded object key=1a67gp0af22w2wv2wfgja1svx1bqxcki.narinfo error="server returned 404: 404 page not found\n"18712026/11/12 16:57:11 INFO Completed push id=2 objects=3 duration=3.393999ms18722026/11/12 16:57:11 INFO Upload complete. (55ms)1873=== NAME TestNARDeduplicationMetadataUploadBug1874 metadata_upload_test.go:76: Retrieved narinfo from S3:1875 StorePath: /build/TestNARDeduplicationMetadataUploadBug4021936798/001/store/1a67gp0af22w2wv2wfgja1svx1bqxcki-file2.txt1876 URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst1877 Compression: zstd1878 NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1879 NarSize: 1601880 References: 1881 CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1882 metadata_upload_test.go:77: Retrieved .ls file from S3 (compressed size: 62 bytes)1883 metadata_upload_test.go:77: Decompressed .ls content (49 bytes):1884 {"version":1,"root":{"type":"regular","size":44}}1885--- PASS: TestNARDeduplicationMetadataUploadBug (0.69s)1886=== CONT TestPendingClosureWriteTimeout/670k_objects1887=== CONT TestPendingClosureWriteTimeout/400_objects1888=== CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info18892026/11/12 16:57:11 INFO Received uploads request method=POST path=/1890=== CONT TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key18912026/11/12 16:57:11 INFO Received request for more parts method=POST path=/1892=== CONT TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key18932026/11/12 16:57:11 INFO Received complete multipart upload request method=POST path=/1894=== CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal18952026/11/12 16:57:11 INFO Received uploads request method=POST path=/1896--- PASS: TestUploadHandlersRejectInvalidKeys (0.10s)1897 --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info (0.00s)1898 --- PASS: TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key (0.00s)1899 --- PASS: TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key (0.00s)1900 --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal (0.00s)1901=== CONT TestIsValidUploadKey/narinfo1902=== CONT TestIsValidUploadKey/unknown_type1903=== CONT TestIsValidUploadKey/empty_key1904=== CONT TestIsValidUploadKey/absolute1905=== CONT TestIsValidUploadKey/traversal_nar1906=== CONT TestIsValidUploadKey/traversal1907=== CONT TestIsValidUploadKey/listing_key,_narinfo_type1908=== CONT TestIsValidUploadKey/nar_key,_narinfo_type1909=== CONT TestIsValidUploadKey/narinfo_key,_nar_type1910=== CONT TestIsValidUploadKey/index.html1911=== CONT TestIsValidUploadKey/nix-cache-info1912=== CONT TestIsValidUploadKey/realisation_plus_in_output1913=== CONT TestIsValidUploadKey/realisation1914=== CONT TestIsValidUploadKey/build_log_equals1915=== CONT TestIsValidUploadKey/build_log_question_mark1916=== CONT TestIsValidUploadKey/build_log_plus_in_name1917=== CONT TestIsValidUploadKey/build_log_home-manager_file1918=== CONT TestIsValidUploadKey/build_log1919=== CONT TestIsValidUploadKey/listing1920=== CONT TestIsValidUploadKey/nar_plain1921=== CONT TestIsValidUploadKey/nar_xz1922=== CONT TestIsValidUploadKey/nar_zst1923--- PASS: TestIsValidUploadKey (0.10s)1924 --- PASS: TestIsValidUploadKey/narinfo (0.00s)1925 --- PASS: TestIsValidUploadKey/unknown_type (0.00s)1926 --- PASS: TestIsValidUploadKey/empty_key (0.00s)1927 --- PASS: TestIsValidUploadKey/absolute (0.00s)1928 --- PASS: TestIsValidUploadKey/traversal_nar (0.00s)1929 --- PASS: TestIsValidUploadKey/traversal (0.00s)1930 --- PASS: TestIsValidUploadKey/listing_key,_narinfo_type (0.00s)1931 --- PASS: TestIsValidUploadKey/nar_key,_narinfo_type (0.00s)1932 --- PASS: TestIsValidUploadKey/narinfo_key,_nar_type (0.00s)1933 --- PASS: TestIsValidUploadKey/index.html (0.00s)1934 --- PASS: TestIsValidUploadKey/nix-cache-info (0.00s)1935 --- PASS: TestIsValidUploadKey/realisation_plus_in_output (0.00s)1936 --- PASS: TestIsValidUploadKey/realisation (0.00s)1937 --- PASS: TestIsValidUploadKey/build_log_equals (0.00s)1938 --- PASS: TestIsValidUploadKey/build_log_question_mark (0.00s)1939 --- PASS: TestIsValidUploadKey/build_log_plus_in_name (0.00s)1940 --- PASS: TestIsValidUploadKey/build_log_home-manager_file (0.00s)1941 --- PASS: TestIsValidUploadKey/build_log (0.00s)1942 --- PASS: TestIsValidUploadKey/listing (0.00s)1943 --- PASS: TestIsValidUploadKey/nar_plain (0.00s)1944 --- PASS: TestIsValidUploadKey/nar_xz (0.00s)1945 --- PASS: TestIsValidUploadKey/nar_zst (0.00s)1946=== CONT TestIsValidCachePath/narinfo1947=== CONT TestIsValidCachePath/traversal_parent1948=== CONT TestIsValidCachePath/pin_invalid_char1949=== CONT TestIsValidCachePath/pin_nested1950=== CONT TestIsValidCachePath/pin_empty_name1951=== CONT TestIsValidCachePath/short_hash1952=== CONT TestIsValidCachePath/wrong_extension1953=== CONT TestIsValidCachePath/leading_slash1954=== CONT TestIsValidCachePath/empty1955=== CONT TestIsValidCachePath/random_path1956=== CONT TestIsValidCachePath/invalid_char_u1957=== CONT TestIsValidCachePath/invalid_char_e1958=== CONT TestIsValidCachePath/pin_traversal1959=== CONT TestIsValidCachePath/traversal_in_middle1960=== CONT TestIsValidCachePath/log1961=== CONT TestIsValidCachePath/pin_all_alphabet_chars1962=== CONT TestIsValidCachePath/pin1963=== CONT TestIsValidCachePath/index.html1964=== CONT TestIsValidCachePath/nix-cache-info1965=== CONT TestIsValidCachePath/realisation1966=== CONT TestIsValidCachePath/nar_bz21967=== CONT TestIsValidCachePath/ls1968=== CONT TestIsValidCachePath/nar_uncompressed1969=== CONT TestIsValidCachePath/nar_zst1970=== CONT TestIsValidCachePath/nar_xz1971=== CONT TestIsValidCachePath/narinfo_all_nix_base32_chars1972--- PASS: TestIsValidCachePath (0.00s)1973 --- PASS: TestIsValidCachePath/narinfo (0.00s)1974 --- PASS: TestIsValidCachePath/traversal_parent (0.00s)1975 --- PASS: TestIsValidCachePath/pin_invalid_char (0.00s)1976 --- PASS: TestIsValidCachePath/pin_nested (0.00s)1977 --- PASS: TestIsValidCachePath/pin_empty_name (0.00s)1978 --- PASS: TestIsValidCachePath/short_hash (0.00s)1979 --- PASS: TestIsValidCachePath/wrong_extension (0.00s)1980 --- PASS: TestIsValidCachePath/leading_slash (0.00s)1981 --- PASS: TestIsValidCachePath/empty (0.00s)1982 --- PASS: TestIsValidCachePath/random_path (0.00s)1983 --- PASS: TestIsValidCachePath/invalid_char_u (0.00s)1984 --- PASS: TestIsValidCachePath/invalid_char_e (0.00s)1985 --- PASS: TestIsValidCachePath/pin_traversal (0.00s)1986 --- PASS: TestIsValidCachePath/traversal_in_middle (0.00s)1987 --- PASS: TestIsValidCachePath/log (0.00s)1988 --- PASS: TestIsValidCachePath/pin_all_alphabet_chars (0.00s)1989 --- PASS: TestIsValidCachePath/pin (0.00s)1990 --- PASS: TestIsValidCachePath/index.html (0.00s)1991 --- PASS: TestIsValidCachePath/nix-cache-info (0.00s)1992 --- PASS: TestIsValidCachePath/realisation (0.00s)1993 --- PASS: TestIsValidCachePath/nar_bz2 (0.00s)1994 --- PASS: TestIsValidCachePath/ls (0.00s)1995 --- PASS: TestIsValidCachePath/nar_uncompressed (0.00s)1996 --- PASS: TestIsValidCachePath/nar_zst (0.00s)1997 --- PASS: TestIsValidCachePath/nar_xz (0.00s)1998 --- PASS: TestIsValidCachePath/narinfo_all_nix_base32_chars (0.00s)1999=== CONT TestParseSingleRange/none2000=== CONT TestParseSingleRange/open-ended
loading 881 hidden lines…
28822026/11/12 16:57:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028832026/11/12 16:57:26 INFO Vacuumed table table=closures28842026/11/12 16:57:26 INFO Vacuumed table table=pending_closures28852026/11/12 16:57:26 INFO Aborted multipart uploads count=0 kept=028862026/11/12 16:57:26 INFO Vacuumed table table=pending_closures28872026/11/12 16:57:26 INFO Vacuumed table table=objects28882026/11/12 16:57:26 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."28892026/11/12 16:57:26 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."28902026/11/12 16:57:26 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."28912026/11/12 16:57:26 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."28922026/11/12 16:57:26 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."28932026/11/12 16:57:26 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."28942026/11/12 16:57:26 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."28952026/11/12 16:57:26 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."28962026/11/12 16:57:26 INFO Vacuumed table table=commit_receipts28972026/11/12 16:57:26 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."28982026/11/12 16:57:26 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."28992026/11/12 16:57:26 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."29002026/11/12 16:57:26 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."29012026/11/12 16:57:26 INFO Vacuumed table table=commit_receipts29022026/11/12 16:57:26 INFO Vacuumed table table=pending_objects29032026/11/12 16:57:26 WARN Force mode enabled - objects will be deleted immediately without grace period29042026/11/12 16:57:26 INFO Vacuumed table table=multipart_uploads29052026/11/12 16:57:26 INFO Vacuumed table table=pending_objects29062026/11/12 16:57:26 INFO Vacuumed table table=multipart_uploads29072026/11/12 16:57:26 INFO Vacuumed table table=closures29082026/11/12 16:57:26 INFO Vacuumed table table=closures29092026/11/12 16:57:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029102026/11/12 16:57:26 INFO Vacuumed table table=objects29112026/11/12 16:57:26 INFO Vacuumed table table=objects29122026/11/12 16:57:26 INFO Vacuumed table table=pending_closures29132026/11/12 16:57:26 INFO Vacuumed table table=commit_receipts29142026/11/12 16:57:26 INFO Vacuumed table table=pending_objects29152026/11/12 16:57:26 INFO Vacuumed table table=multipart_uploads29162026/11/12 16:57:26 INFO Vacuumed table table=closures29172026/11/12 16:57:26 INFO Received uploads request method=POST path=/29182026/11/12 16:57:26 INFO Vacuumed table table=objects29192026/11/12 16:57:26 INFO Received uploads request method=POST path=/29202026/11/12 16:57:26 INFO Received push request method=POST path=/29212026/11/12 16:57:26 INFO Received uploads request method=POST path=/29222026/11/12 16:57:26 INFO Received uploads request method=POST path=/29232026/11/12 16:57:26 INFO Received uploads request method=POST path=/29242026/11/12 16:57:26 INFO Received uploads request method=POST path=/29252026/11/12 16:57:26 INFO Received complete upload request method=POST path=/29262026/11/12 16:57:26 INFO Completed upload id=2 objects=2 duration=1.899679ms29272026/11/12 16:57:26 INFO Received complete upload request method=POST path=/29282026/11/12 16:57:26 INFO Aborted multipart uploads count=0 kept=029292026/11/12 16:57:26 INFO Completed upload id=3 objects=9 duration=1.26815ms29302026/11/12 16:57:26 INFO Received push request method=POST path=/29312026/11/12 16:57:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029322026/11/12 16:57:26 INFO Received uploads request method=POST path=/29332026/11/12 16:57:26 INFO Received push request method=POST path=/29342026/11/12 16:57:26 INFO Vacuumed table table=pending_closures29352026/11/12 16:57:26 INFO Vacuumed table table=commit_receipts29362026/11/12 16:57:26 INFO Vacuumed table table=pending_objects29372026/11/12 16:57:26 INFO Received push request method=POST path=/29382026/11/12 16:57:26 INFO Vacuumed table table=multipart_uploads29392026/11/12 16:57:26 INFO Vacuumed table table=closures29402026/11/12 16:57:26 INFO Vacuumed table table=objects29412026/11/12 16:57:26 INFO Aborted multipart uploads count=0 kept=029422026/11/12 16:57:26 INFO Received uploads request method=POST path=/29432026/11/12 16:57:26 INFO Received uploads request method=POST path=/29442026/11/12 16:57:26 WARN Force mode enabled - objects will be deleted immediately without grace period29452026/11/12 16:57:26 INFO Received complete upload request method=POST path=/29462026/11/12 16:57:26 INFO Received push request method=POST path=/29472026/11/12 16:57:26 INFO Completed upload id=7 objects=4 duration=1.6457ms29482026/11/12 16:57:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029492026/11/12 16:57:26 INFO Vacuumed table table=pending_closures29502026/11/12 16:57:26 INFO Vacuumed table table=commit_receipts29512026/11/12 16:57:26 INFO Vacuumed table table=pending_objects29522026/11/12 16:57:26 INFO Vacuumed table table=multipart_uploads29532026/11/12 16:57:26 INFO Vacuumed table table=closures29542026/11/12 16:57:26 INFO Vacuumed table table=objects29552026/11/12 16:57:26 INFO Received uploads request method=POST path=/29562026/11/12 16:57:26 INFO Received complete push request method=POST path=/29572026/11/12 16:57:26 INFO Received push request method=POST path=/29582026/11/12 16:57:26 INFO Completed push id=6 objects=5 duration=2.084589ms29592026/11/12 16:57:26 INFO Received uploads request method=POST path=/29602026/11/12 16:57:26 INFO Received complete upload request method=POST path=/29612026/11/12 16:57:26 INFO Completed upload id=11 objects=14 duration=1.7306ms29622026/11/12 16:57:26 INFO Received uploads request method=POST path=/29632026/11/12 16:57:26 INFO Received uploads request method=POST path=/29642026/11/12 16:57:26 INFO Received push request method=POST path=/29652026/11/12 16:57:26 INFO Received complete push request method=POST path=/29662026/11/12 16:57:26 INFO Received complete push request method=POST path=/29672026/11/12 16:57:26 INFO Completed push id=4 objects=4 duration=2.126599ms29682026/11/12 16:57:26 INFO Completed push id=12 objects=3 duration=2.14813ms29692026/11/12 16:57:26 INFO Aborted multipart uploads count=0 kept=029702026/11/12 16:57:26 WARN Force mode enabled - objects will be deleted immediately without grace period29712026/11/12 16:57:26 INFO Received uploads request method=POST path=/29722026/11/12 16:57:26 INFO Received push request method=POST path=/29732026/11/12 16:57:26 INFO Received push request method=POST path=/29742026/11/12 16:57:26 INFO Received push request method=POST path=/29752026/11/12 16:57:26 INFO Received push request method=POST path=/29762026/11/12 16:57:26 INFO Received uploads request method=POST path=/29772026/11/12 16:57:26 INFO Received complete push request method=POST path=/29782026/11/12 16:57:26 INFO Completed push id=5 objects=6 duration=1.571519ms29792026/11/12 16:57:26 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."29802026/11/12 16:57:26 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."29812026/11/12 16:57:26 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."29822026/11/12 16:57:26 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."29832026/11/12 16:57:26 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."29842026/11/12 16:57:26 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."29852026/11/12 16:57:26 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."29862026/11/12 16:57:26 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."29872026/11/12 16:57:26 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."29882026/11/12 16:57:26 INFO Aborted multipart uploads count=0 kept=029892026/11/12 16:57:26 INFO Received push request method=POST path=/29902026/11/12 16:57:26 INFO Received complete push request method=POST path=/29912026/11/12 16:57:26 INFO Aborted multipart uploads count=0 kept=029922026/11/12 16:57:26 INFO Received uploads request method=POST path=/29932026/11/12 16:57:26 WARN Force mode enabled - objects will be deleted immediately without grace period29942026/11/12 16:57:26 INFO Received uploads request method=POST path=/29952026/11/12 16:57:26 INFO Received push request method=POST path=/29962026/11/12 16:57:26 WARN Force mode enabled - objects will be deleted immediately without grace period29972026/11/12 16:57:26 INFO Received push request method=POST path=/29982026/11/12 16:57:26 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029992026/11/12 16:57:26 INFO Completed push id=5 objects=9 duration=5.566648ms30002026/11/12 16:57:26 INFO Received push request method=POST path=/30012026/11/12 16:57:26 INFO Vacuumed table table=pending_closures30022026/11/12 16:57:26 INFO Vacuumed table table=commit_receipts30032026/11/12 16:57:26 INFO Vacuumed table table=pending_objects30042026/11/12 16:57:26 INFO Vacuumed table table=multipart_uploads30052026/11/12 16:57:26 INFO Vacuumed table table=closures30062026/11/12 16:57:26 INFO Aborted multipart uploads count=0 kept=030072026/11/12 16:57:26 INFO Vacuumed table table=objects30082026/11/12 16:57:26 WARN Force mode enabled - objects will be deleted immediately without grace period30092026/11/12 16:57:26 INFO Received complete upload request method=POST path=/30102026/11/12 16:57:26 INFO Completed upload id=6 objects=7 duration=3.66828ms30112026/11/12 16:57:26 INFO Aborted multipart uploads count=0 kept=030122026/11/12 16:57:26 INFO Aborted multipart uploads count=0 kept=030132026/11/12 16:57:26 INFO Aborted multipart uploads count=0 kept=030142026/11/12 16:57:26 WARN Force mode enabled - objects will be deleted immediately without grace period30152026/11/12 16:57:26 WARN Force mode enabled - objects will be deleted immediately without grace period30162026/11/12 16:57:26 INFO Received push request method=POST path=/30172026/11/12 16:57:26 INFO Received uploads request method=POST path=/30182026/11/12 16:57:26 INFO Aborted multipart uploads count=0 kept=030192026/11/12 16:57:26 INFO Received push request method=POST path=/30202026/11/12 16:57:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=030212026/11/12 16:57:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030222026/11/12 16:57:26 INFO Received push request method=POST path=/30232026/11/12 16:57:26 INFO Received complete upload request method=POST path=/30242026/11/12 16:57:26 INFO Vacuumed table table=pending_closures30252026/11/12 16:57:26 INFO Vacuumed table table=commit_receipts30262026/11/12 16:57:26 INFO Vacuumed table table=pending_closures30272026/11/12 16:57:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=9 objects-failed-to-delete=030282026/11/12 16:57:26 INFO Vacuumed table table=commit_receipts30292026/11/12 16:57:26 INFO Completed upload id=5 objects=7 duration=3.6024ms30302026/11/12 16:57:26 INFO Vacuumed table table=pending_objects30312026/11/12 16:57:26 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=4 objects-failed-to-delete=030322026/11/12 16:57:26 INFO Vacuumed table table=multipart_uploads30332026/11/12 16:57:26 INFO Received uploads request method=POST path=/30342026/11/12 16:57:26 INFO Vacuumed table table=pending_closures30352026/11/12 16:57:26 INFO Vacuumed table table=pending_objects30362026/11/12 16:57:26 INFO Vacuumed table table=pending_closures30372026/11/12 16:57:26 INFO Vacuumed table table=commit_receipts30382026/11/12 16:57:26 INFO Vacuumed table table=multipart_uploads30392026/11/12 16:57:26 INFO Vacuumed table table=closures30402026/11/12 16:57:26 INFO Vacuumed table table=commit_receipts30412026/11/12 16:57:26 INFO Vacuumed table table=pending_objects30422026/11/12 16:57:26 INFO Vacuumed table table=closures30432026/11/12 16:57:26 INFO Vacuumed table table=multipart_uploads30442026/11/12 16:57:26 INFO Received push request method=POST path=/30452026/11/12 16:57:26 INFO Vacuumed table table=objects30462026/11/12 16:57:26 INFO Vacuumed table table=objects30472026/11/12 16:57:26 INFO Vacuumed table table=pending_objects30482026/11/12 16:57:26 INFO Vacuumed table table=closures30492026/11/12 16:57:26 INFO Vacuumed table table=multipart_uploads30502026/11/12 16:57:26 INFO Vacuumed table table=objects30512026/11/12 16:57:26 INFO Vacuumed table table=closures30522026/11/12 16:57:26 INFO Received push request method=POST path=/30532026/11/12 16:57:26 INFO Vacuumed table table=objects30542026/11/12 16:57:26 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."30552026/11/12 16:57:26 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."30562026/11/12 16:57:26 INFO Received complete upload request method=POST path=/30572026/11/12 16:57:26 INFO Received push request method=POST path=/30582026/11/12 16:57:26 INFO Received push request method=POST path=/30592026/11/12 16:57:26 INFO Received push request method=POST path=/30602026/11/12 16:57:26 INFO Received complete upload request method=POST path=/30612026/11/12 16:57:26 INFO Aborted multipart uploads count=0 kept=030622026/11/12 16:57:26 WARN Force mode enabled - objects will be deleted immediately without grace period30632026/11/12 16:57:26 INFO Received push request method=POST path=/30642026/11/12 16:57:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030652026/11/12 16:57:26 INFO Completed upload id=10 objects=23 duration=11.690898ms30662026/11/12 16:57:26 INFO Completed upload id=10 objects=6 duration=7.138009ms30672026/11/12 16:57:26 INFO Received complete upload request method=POST path=/30682026/11/12 16:57:26 INFO Vacuumed table table=pending_closures30692026/11/12 16:57:26 INFO Received complete upload request method=POST path=/30702026/11/12 16:57:26 INFO Received push request method=POST path=/30712026/11/12 16:57:26 INFO Received uploads request method=POST path=/30722026/11/12 16:57:26 INFO Vacuumed table table=commit_receipts30732026/11/12 16:57:26 INFO Completed upload id=6 objects=9 duration=1.96488ms30742026/11/12 16:57:26 INFO Received complete push request method=POST path=/30752026/11/12 16:57:26 INFO Completed upload id=9 objects=7 duration=3.30018ms30762026/11/12 16:57:26 INFO Vacuumed table table=pending_objects30772026/11/12 16:57:26 INFO Vacuumed table table=multipart_uploads30782026/11/12 16:57:26 INFO Vacuumed table table=closures30792026/11/12 16:57:26 INFO Vacuumed table table=objects30802026/11/12 16:57:26 INFO Completed push id=5 objects=8 duration=4.205479ms30812026/11/12 16:57:26 INFO Aborted multipart uploads count=0 kept=030822026/11/12 16:57:26 WARN Force mode enabled - objects will be deleted immediately without grace period30832026/11/12 16:57:26 INFO Received complete upload request method=POST path=/30842026/11/12 16:57:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030852026/11/12 16:57:26 INFO Vacuumed table table=pending_closures30862026/11/12 16:57:26 INFO Completed upload id=3 objects=8 duration=2.673469ms30872026/11/12 16:57:26 INFO Received uploads request method=POST path=/30882026/11/12 16:57:26 INFO Vacuumed table table=commit_receipts30892026/11/12 16:57:26 INFO Vacuumed table table=pending_objects30902026/11/12 16:57:26 INFO Vacuumed table table=multipart_uploads30912026/11/12 16:57:26 INFO Vacuumed table table=closures30922026/11/12 16:57:26 INFO Vacuumed table table=objects30932026/11/12 16:57:26 INFO Aborted multipart uploads count=0 kept=030942026/11/12 16:57:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=1 objects-deleted-after-grace-period=0 objects-failed-to-delete=030952026/11/12 16:57:26 INFO Received uploads request method=POST path=/30962026/11/12 16:57:26 INFO Vacuumed table table=pending_closures30972026/11/12 16:57:26 INFO Received push request method=POST path=/30982026/11/12 16:57:26 INFO Vacuumed table table=commit_receipts30992026/11/12 16:57:26 INFO Received push request method=POST path=/31002026/11/12 16:57:26 INFO Received complete upload request method=POST path=/31012026/11/12 16:57:26 INFO Vacuumed table table=pending_objects31022026/11/12 16:57:26 INFO Received push request method=POST path=/31032026/11/12 16:57:26 INFO Vacuumed table table=multipart_uploads31042026/11/12 16:57:26 INFO Aborted multipart uploads count=0 kept=031052026/11/12 16:57:26 INFO Completed upload id=10 objects=2 duration=2.85324ms31062026/11/12 16:57:26 INFO Vacuumed table table=closures31072026/11/12 16:57:26 INFO Received complete upload request method=POST path=/31082026/11/12 16:57:26 INFO Vacuumed table table=objects31092026/11/12 16:57:26 INFO Completed upload id=16 objects=23 duration=1.596989ms31102026/11/12 16:57:26 INFO Received uploads request method=POST path=/31112026/11/12 16:57:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031122026/11/12 16:57:26 INFO Vacuumed table table=pending_closures31132026/11/12 16:57:26 INFO Vacuumed table table=commit_receipts31142026/11/12 16:57:26 INFO Received uploads request method=POST path=/31152026/11/12 16:57:26 INFO Received complete push request method=POST path=/31162026/11/12 16:57:26 INFO Vacuumed table table=pending_objects31172026/11/12 16:57:26 INFO Vacuumed table table=multipart_uploads31182026/11/12 16:57:26 INFO Vacuumed table table=closures31192026/11/12 16:57:26 INFO Completed push id=5 objects=3 duration=2.342199ms31202026/11/12 16:57:26 INFO Vacuumed table table=objects31212026/11/12 16:57:26 INFO Received uploads request method=POST path=/31222026/11/12 16:57:26 INFO Received push request method=POST path=/31232026/11/12 16:57:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=8 objects-deleted-after-grace-period=8 objects-failed-to-delete=031242026/11/12 16:57:26 INFO Received uploads request method=POST path=/31252026/11/12 16:57:26 INFO Received uploads request method=POST path=/31262026/11/12 16:57:26 INFO Received uploads request method=POST path=/31272026/11/12 16:57:26 INFO Vacuumed table table=pending_closures31282026/11/12 16:57:26 INFO Vacuumed table table=commit_receipts31292026/11/12 16:57:26 INFO Vacuumed table table=pending_objects31302026/11/12 16:57:26 INFO Vacuumed table table=multipart_uploads31312026/11/12 16:57:26 INFO Vacuumed table table=closures31322026/11/12 16:57:26 INFO Received push request method=POST path=/31332026/11/12 16:57:26 INFO Vacuumed table table=objects31342026/11/12 16:57:26 INFO Received uploads request method=POST path=/31352026/11/12 16:57:26 INFO Received complete upload request method=POST path=/31362026/11/12 16:57:26 INFO Completed upload id=15 objects=6 duration=1.19991ms31372026/11/12 16:57:26 INFO Received push request method=POST path=/31382026/11/12 16:57:26 INFO Received uploads request method=POST path=/31392026/11/12 16:57:26 INFO Aborted multipart uploads count=0 kept=031402026/11/12 16:57:26 INFO Received complete upload request method=POST path=/31412026/11/12 16:57:26 INFO Completed upload id=11 objects=6 duration=1.52854ms31422026/11/12 16:57:26 INFO Received complete upload request method=POST path=/31432026/11/12 16:57:26 INFO Received push request method=POST path=/31442026/11/12 16:57:26 INFO Completed upload id=3 objects=5 duration=1.5554ms31452026/11/12 16:57:26 INFO Received complete push request method=POST path=/31462026/11/12 16:57:26 INFO Aborted multipart uploads count=0 kept=031472026/11/12 16:57:26 INFO Received push request method=POST path=/31482026/11/12 16:57:26 INFO Completed push id=14 objects=3 duration=3.68682ms31492026/11/12 16:57:26 INFO Received uploads request method=POST path=/31502026/11/12 16:57:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031512026/11/12 16:57:26 INFO Vacuumed table table=pending_closures31522026/11/12 16:57:26 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=031532026/11/12 16:57:26 INFO Received push request method=POST path=/31542026/11/12 16:57:26 INFO Vacuumed table table=commit_receipts31552026/11/12 16:57:26 INFO Vacuumed table table=pending_closures31562026/11/12 16:57:26 INFO Vacuumed table table=pending_objects31572026/11/12 16:57:26 INFO Vacuumed table table=commit_receipts31582026/11/12 16:57:26 INFO Vacuumed table table=multipart_uploads31592026/11/12 16:57:26 INFO Vacuumed table table=closures31602026/11/12 16:57:26 INFO Vacuumed table table=pending_objects31612026/11/12 16:57:26 INFO Vacuumed table table=multipart_uploads31622026/11/12 16:57:26 INFO Vacuumed table table=objects31632026/11/12 16:57:26 INFO Received push request method=POST path=/31642026/11/12 16:57:26 INFO Vacuumed table table=closures31652026/11/12 16:57:26 INFO Received push request method=POST path=/31662026/11/12 16:57:26 INFO Vacuumed table table=objects31672026/11/12 16:57:26 INFO Aborted multipart uploads count=0 kept=031682026/11/12 16:57:26 INFO Aborted multipart uploads count=0 kept=031692026/11/12 16:57:26 WARN Force mode enabled - objects will be deleted immediately without grace period31702026/11/12 16:57:26 INFO Aborted multipart uploads count=0 kept=031712026/11/12 16:57:26 INFO Aborted multipart uploads count=0 kept=031722026/11/12 16:57:26 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031732026/11/12 16:57:26 INFO Aborted multipart uploads count=0 kept=031742026/11/12 16:57:26 INFO Vacuumed table table=pending_closures31752026/11/12 16:57:26 INFO Vacuumed table table=commit_receipts31762026/11/12 16:57:26 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031772026/11/12 16:57:26 INFO Vacuumed table table=pending_objects31782026/11/12 16:57:26 INFO Vacuumed table table=multipart_uploads31792026/11/12 16:57:26 INFO Vacuumed table table=pending_closures31802026/11/12 16:57:26 INFO Received push request method=POST path=/31812026/11/12 16:57:26 INFO Vacuumed table table=closures31822026/11/12 16:57:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=4 objects-deleted-after-grace-period=0 objects-failed-to-delete=031832026/11/12 16:57:26 INFO Vacuumed table table=commit_receipts31842026/11/12 16:57:26 INFO Vacuumed table table=objects31852026/11/12 16:57:26 INFO Received uploads request method=POST path=/31862026/11/12 16:57:26 INFO Received complete upload request method=POST path=/31872026/11/12 16:57:26 INFO Vacuumed table table=pending_objects31882026/11/12 16:57:26 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."31892026/11/12 16:57:26 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."31902026/11/12 16:57:26 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."31912026/11/12 16:57:26 INFO Vacuumed table table=multipart_uploads31922026/11/12 16:57:26 INFO Vacuumed table table=pending_closures31932026/11/12 16:57:26 INFO Completed upload id=17 objects=7 duration=1.18068ms31942026/11/12 16:57:26 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=031952026/11/12 16:57:26 INFO Vacuumed table table=closures31962026/11/12 16:57:26 INFO Vacuumed table table=commit_receipts31972026/11/12 16:57:26 INFO Vacuumed table table=pending_closures31982026/11/12 16:57:26 INFO Received complete upload request method=POST path=/31992026/11/12 16:57:26 INFO Vacuumed table table=commit_receipts32002026/11/12 16:57:26 INFO Vacuumed table table=objects32012026/11/12 16:57:26 INFO Aborted multipart uploads count=0 kept=032022026/11/12 16:57:26 INFO Vacuumed table table=pending_objects32032026/11/12 16:57:26 INFO Vacuumed table table=pending_objects32042026/11/12 16:57:26 INFO Vacuumed table table=multipart_uploads32052026/11/12 16:57:26 INFO Vacuumed table table=multipart_uploads32062026/11/12 16:57:26 INFO Completed upload id=3 objects=24 duration=2.446939ms32072026/11/12 16:57:26 INFO Received push request method=POST path=/32082026/11/12 16:57:26 INFO Vacuumed table table=closures32092026/11/12 16:57:26 INFO Vacuumed table table=closures32102026/11/12 16:57:26 INFO Received complete push request method=POST path=/32112026/11/12 16:57:26 INFO Vacuumed table table=objects32122026/11/12 16:57:26 INFO Vacuumed table table=objects32132026/11/12 16:57:26 INFO Completed push id=5 objects=6 duration=1.70142ms32142026/11/12 16:57:26 INFO Received complete push request method=POST path=/32152026/11/12 16:57:26 INFO Aborted multipart uploads count=0 kept=032162026/11/12 16:57:26 INFO Completed push id=12 objects=15 duration=2.215479ms32172026/11/12 16:57:26 WARN Force mode enabled - objects will be deleted immediately without grace period32182026/11/12 16:57:26 INFO Received uploads request method=POST path=/32192026/11/12 16:57:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032202026/11/12 16:57:26 INFO Vacuumed table table=pending_closures32212026/11/12 16:57:26 INFO Vacuumed table table=commit_receipts32222026/11/12 16:57:26 INFO Vacuumed table table=pending_objects32232026/11/12 16:57:26 INFO Vacuumed table table=multipart_uploads32242026/11/12 16:57:26 INFO Vacuumed table table=closures32252026/11/12 16:57:26 INFO Vacuumed table table=objects32262026/11/12 16:57:26 INFO Aborted multipart uploads count=0 kept=032272026/11/12 16:57:26 INFO Aborted multipart uploads count=0 kept=032282026/11/12 16:57:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032292026/11/12 16:57:26 INFO Vacuumed table table=pending_closures32302026/11/12 16:57:26 INFO Aborted multipart uploads count=0 kept=032312026/11/12 16:57:26 INFO Vacuumed table table=commit_receipts32322026/11/12 16:57:26 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."32332026/11/12 16:57:26 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."32342026/11/12 16:57:26 INFO Vacuumed table table=pending_objects32352026/11/12 16:57:26 INFO Vacuumed table table=multipart_uploads32362026/11/12 16:57:26 INFO Vacuumed table table=closures32372026/11/12 16:57:26 WARN Force mode enabled - objects will be deleted immediately without grace period32382026/11/12 16:57:26 INFO Vacuumed table table=objects32392026/11/12 16:57:26 INFO Received complete upload request method=POST path=/32402026/11/12 16:57:26 INFO Received push request method=POST path=/32412026/11/12 16:57:26 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032422026/11/12 16:57:26 INFO Vacuumed table table=pending_closures32432026/11/12 16:57:26 INFO Received uploads request method=POST path=/32442026/11/12 16:57:26 INFO Completed upload id=11 objects=6 duration=3.00094ms32452026/11/12 16:57:26 INFO Vacuumed table table=commit_receipts32462026/11/12 16:57:26 INFO Received complete push request method=POST path=/32472026/11/12 16:57:26 INFO Vacuumed table table=pending_objects32482026/11/12 16:57:26 INFO Vacuumed table table=multipart_uploads32492026/11/12 16:57:26 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."32502026/11/12 16:57:26 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."32512026/11/12 16:57:26 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."32522026/11/12 16:57:26 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."32532026/11/12 16:57:26 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."32542026/11/12 16:57:26 INFO Vacuumed table table=closures32552026/11/12 16:57:26 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."32562026/11/12 16:57:26 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."32572026/11/12 16:57:26 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."32582026/11/12 16:57:26 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."32592026/11/12 16:57:26 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."32602026/11/12 16:57:26 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."32612026/11/12 16:57:26 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."32622026/11/12 16:57:26 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."32632026/11/12 16:57:26 ERROR failed to remove object object=realisations/sha256:aaaaaaaa!out.doi error="Access Denied."32642026/11/12 16:57:26 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."32652026/11/12 16:57:26 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."32662026/11/12 16:57:26 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."32672026/11/12 16:57:26 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."32682026/11/12 16:57:26 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."32692026/11/12 16:57:26 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."32702026/11/12 16:57:26 INFO Received uploads request method=POST path=/32712026/11/12 16:57:26 INFO Vacuumed table table=objects32722026/11/12 16:57:26 INFO Received uploads request method=POST path=/32732026/11/12 16:57:26 INFO Completed push id=13 objects=12 duration=3.84641ms32742026/11/12 16:57:26 INFO Received push request method=POST path=/32752026/11/12 16:57:26 INFO Received complete push request method=POST path=/32762026/11/12 16:57:26 INFO Completed push id=11 objects=8 duration=3.693109ms32772026/11/12 16:57:26 INFO Received uploads request method=POST path=/32782026/11/12 16:57:26 INFO Received push request method=POST path=/32792026/11/12 16:57:26 INFO Received push request method=POST path=/32802026/11/12 16:57:26 INFO Received complete push request method=POST path=/32812026/11/12 16:57:26 INFO Received push request method=POST path=/32822026/11/12 16:57:26 INFO Aborted multipart uploads count=0 kept=032832026/11/12 16:57:26 INFO Completed push id=2 objects=10 duration=4.126849ms32842026/11/12 16:57:26 INFO Aborted multipart uploads count=0 kept=032852026/11/12 16:57:26 WARN Force mode enabled - objects will be deleted immediately without grace period32862026/11/12 16:57:26 INFO Aborted multipart uploads count=0 kept=032872026/11/12 16:57:26 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032882026/11/12 16:57:26 INFO Vacuumed table table=pending_closures32892026/11/12 16:57:26 INFO Vacuumed table table=commit_receipts32902026/11/12 16:57:26 INFO Vacuumed table table=pending_objects32912026/11/12 16:57:26 INFO Vacuumed table table=multipart_uploads32922026/11/12 16:57:26 INFO Received complete push request method=POST path=/32932026/11/12 16:57:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032942026/11/12 16:57:26 INFO Vacuumed table table=closures32952026/11/12 16:57:26 INFO Received uploads request method=POST path=/32962026/11/12 16:57:26 INFO Received uploads request method=POST path=/32972026/11/12 16:57:26 INFO Vacuumed table table=pending_closures32982026/11/12 16:57:26 INFO Vacuumed table table=objects32992026/11/12 16:57:26 INFO Completed push id=10 objects=2 duration=2.563989ms33002026/11/12 16:57:26 INFO Vacuumed table table=commit_receipts33012026/11/12 16:57:26 INFO Vacuumed table table=pending_objects33022026/11/12 16:57:26 INFO Received complete push request method=POST path=/33032026/11/12 16:57:26 INFO Vacuumed table table=multipart_uploads33042026/11/12 16:57:26 INFO Received push request method=POST path=/33052026/11/12 16:57:26 INFO Received complete upload request method=POST path=/33062026/11/12 16:57:26 INFO Vacuumed table table=closures33072026/11/12 16:57:26 INFO Completed upload id=8 objects=3 duration=6.078749ms33082026/11/12 16:57:26 INFO Completed push id=8 objects=6 duration=10.667338ms33092026/11/12 16:57:26 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=033102026/11/12 16:57:26 INFO Vacuumed table table=objects33112026/11/12 16:57:26 INFO Received complete upload request method=POST path=/33122026/11/12 16:57:26 INFO Aborted multipart uploads count=0 kept=033132026/11/12 16:57:26 INFO Vacuumed table table=pending_closures33142026/11/12 16:57:26 WARN Force mode enabled - objects will be deleted immediately without grace period33152026/11/12 16:57:26 INFO Completed upload id=14 objects=6 duration=1.71073ms33162026/11/12 16:57:26 INFO Vacuumed table table=commit_receipts33172026/11/12 16:57:26 INFO Received push request method=POST path=/33182026/11/12 16:57:26 INFO Vacuumed table table=pending_objects33192026/11/12 16:57:26 INFO Vacuumed table table=multipart_uploads33202026/11/12 16:57:26 INFO Vacuumed table table=closures33212026/11/12 16:57:26 INFO Received complete push request method=POST path=/33222026/11/12 16:57:26 INFO Vacuumed table table=objects33232026/11/12 16:57:26 INFO Completed push id=11 objects=11 duration=3.82927ms33242026/11/12 16:57:26 INFO Received complete upload request method=POST path=/33252026/11/12 16:57:26 INFO Completed upload id=12 objects=21 duration=1.22443ms33262026/11/12 16:57:26 INFO Received complete push request method=POST path=/33272026/11/12 16:57:26 INFO Received push request method=POST path=/33282026/11/12 16:57:26 INFO Completed push id=7 objects=20 duration=3.026629ms33292026/11/12 16:57:26 INFO Received uploads request method=POST path=/33302026/11/12 16:57:26 INFO Received push request method=POST path=/33312026/11/12 16:57:26 INFO Received uploads request method=POST path=/33322026/11/12 16:57:26 INFO Received push request method=POST path=/33332026/11/12 16:57:26 INFO Received push request method=POST path=/33342026/11/12 16:57:26 INFO Aborted multipart uploads count=0 kept=033352026/11/12 16:57:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033362026/11/12 16:57:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=16 objects-failed-to-delete=033372026/11/12 16:57:26 INFO Received complete push request method=POST path=/33382026/11/12 16:57:26 INFO Vacuumed table table=pending_closures33392026/11/12 16:57:26 INFO Received push request method=POST path=/33402026/11/12 16:57:26 INFO Vacuumed table table=commit_receipts33412026/11/12 16:57:26 INFO Vacuumed table table=pending_closures33422026/11/12 16:57:26 INFO Received complete push request method=POST path=/33432026/11/12 16:57:26 INFO Completed push id=13 objects=6 duration=3.477439ms33442026/11/12 16:57:26 INFO Vacuumed table table=commit_receipts33452026/11/12 16:57:26 INFO Vacuumed table table=pending_objects33462026/11/12 16:57:26 INFO Vacuumed table table=multipart_uploads33472026/11/12 16:57:26 INFO Vacuumed table table=closures33482026/11/12 16:57:26 INFO Vacuumed table table=pending_objects33492026/11/12 16:57:26 INFO Vacuumed table table=objects33502026/11/12 16:57:26 INFO Vacuumed table table=multipart_uploads33512026/11/12 16:57:26 INFO Completed push id=9 objects=5 duration=3.030189ms33522026/11/12 16:57:26 INFO Vacuumed table table=closures33532026/11/12 16:57:26 INFO Received complete push request method=POST path=/33542026/11/12 16:57:26 INFO Vacuumed table table=objects33552026/11/12 16:57:26 INFO Completed push id=8 objects=7 duration=2.158309ms33562026/11/12 16:57:26 INFO Received push request method=POST path=/33572026/11/12 16:57:26 INFO Received push request method=POST path=/33582026/11/12 16:57:26 INFO Received push request method=POST path=/33592026/11/12 16:57:26 INFO Received push request method=POST path=/33602026/11/12 16:57:26 INFO Aborted multipart uploads count=0 kept=033612026/11/12 16:57:26 INFO Received push request method=POST path=/33622026/11/12 16:57:26 INFO Received complete upload request method=POST path=/33632026/11/12 16:57:26 WARN Force mode enabled - objects will be deleted immediately without grace period33642026/11/12 16:57:26 INFO Completed upload id=7 objects=3 duration=1.09916ms33652026/11/12 16:57:26 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033662026/11/12 16:57:26 INFO Received push request method=POST path=/33672026/11/12 16:57:26 INFO Vacuumed table table=pending_closures33682026/11/12 16:57:26 INFO Aborted multipart uploads count=0 kept=033692026/11/12 16:57:26 INFO Vacuumed table table=commit_receipts33702026/11/12 16:57:26 INFO Vacuumed table table=pending_objects33712026/11/12 16:57:26 INFO Vacuumed table table=multipart_uploads33722026/11/12 16:57:26 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033732026/11/12 16:57:26 INFO Vacuumed table table=closures33742026/11/12 16:57:26 INFO Received complete push request method=POST path=/33752026/11/12 16:57:26 INFO Received uploads request method=POST path=/33762026/11/12 16:57:26 INFO Vacuumed table table=pending_closures33772026/11/12 16:57:26 INFO Vacuumed table table=commit_receipts33782026/11/12 16:57:26 INFO Vacuumed table table=objects33792026/11/12 16:57:26 INFO Received complete push request method=POST path=/33802026/11/12 16:57:26 INFO Received push request method=POST path=/33812026/11/12 16:57:26 INFO Completed push id=4 objects=6 duration=3.012579ms33822026/11/12 16:57:26 INFO Vacuumed table table=pending_objects33832026/11/12 16:57:26 INFO Vacuumed table table=multipart_uploads33842026/11/12 16:57:26 INFO Completed push id=13 objects=23 duration=2.14332ms33852026/11/12 16:57:26 INFO Received uploads request method=POST path=/33862026/11/12 16:57:26 INFO Vacuumed table table=closures33872026/11/12 16:57:26 INFO Vacuumed table table=objects33882026/11/12 16:57:26 INFO Received uploads request method=POST path=/33892026/11/12 16:57:26 INFO Aborted multipart uploads count=0 kept=033902026/11/12 16:57:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033912026/11/12 16:57:26 INFO Received complete push request method=POST path=/33922026/11/12 16:57:26 INFO Vacuumed table table=pending_closures33932026/11/12 16:57:26 INFO Vacuumed table table=commit_receipts33942026/11/12 16:57:26 INFO Completed push id=6 objects=6 duration=2.354899ms33952026/11/12 16:57:26 INFO Vacuumed table table=pending_objects33962026/11/12 16:57:26 INFO Vacuumed table table=multipart_uploads33972026/11/12 16:57:26 INFO Vacuumed table table=closures33982026/11/12 16:57:26 INFO Vacuumed table table=objects33992026/11/12 16:57:26 INFO Received complete push request method=POST path=/34002026/11/12 16:57:26 INFO Received uploads request method=POST path=/34012026/11/12 16:57:26 INFO Completed push id=18 objects=23 duration=2.10727ms34022026/11/12 16:57:26 INFO Received uploads request method=POST path=/34032026/11/12 16:57:26 INFO Received push request method=POST path=/34042026/11/12 16:57:26 INFO Received complete upload request method=POST path=/34052026/11/12 16:57:26 INFO Received complete push request method=POST path=/34062026/11/12 16:57:26 INFO Completed upload id=9 objects=16 duration=1.551349ms34072026/11/12 16:57:26 INFO Received push request method=POST path=/34082026/11/12 16:57:26 INFO Received complete push request method=POST path=/34092026/11/12 16:57:26 INFO Received complete push request method=POST path=/34102026/11/12 16:57:26 INFO Completed push id=5 objects=3 duration=4.092849ms34112026/11/12 16:57:26 INFO Received push request method=POST path=/34122026/11/12 16:57:26 INFO Completed push id=6 objects=26 duration=4.633379ms34132026/11/12 16:57:26 INFO Completed push id=19 objects=6 duration=4.119869ms34142026/11/12 16:57:26 INFO Received push request method=POST path=/34152026/11/12 16:57:26 INFO Received complete push request method=POST path=/34162026/11/12 16:57:26 INFO Completed push id=9 objects=24 duration=1.86392ms34172026/11/12 16:57:26 INFO Received push request method=POST path=/34182026/11/12 16:57:26 INFO Received complete push request method=POST path=/34192026/11/12 16:57:26 INFO Received complete upload request method=POST path=/34202026/11/12 16:57:26 INFO Received push request method=POST path=/34212026/11/12 16:57:26 INFO Received uploads request method=POST path=/34222026/11/12 16:57:26 INFO Completed upload id=9 objects=2 duration=1.78944ms34232026/11/12 16:57:26 INFO Completed push id=17 objects=6 duration=2.4887ms34242026/11/12 16:57:26 INFO Received push request method=POST path=/34252026/11/12 16:57:26 INFO Received uploads request method=POST path=/34262026/11/12 16:57:26 INFO Received push request method=POST path=/34272026/11/12 16:57:26 INFO Received uploads request method=POST path=/34282026/11/12 16:57:26 INFO Received complete push request method=POST path=/34292026/11/12 16:57:26 INFO Received uploads request method=POST path=/34302026/11/12 16:57:26 INFO Received push request method=POST path=/34312026/11/12 16:57:26 INFO Aborted multipart uploads count=0 kept=034322026/11/12 16:57:26 INFO Completed push id=8 objects=3 duration=2.235699ms34332026/11/12 16:57:26 INFO Received complete push request method=POST path=/34342026/11/12 16:57:26 INFO Received push request method=POST path=/34352026/11/12 16:57:26 INFO Completed push id=11 objects=9 duration=1.847719ms34362026/11/12 16:57:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=034372026/11/12 16:57:26 INFO Received uploads request method=POST path=/34382026/11/12 16:57:26 INFO Vacuumed table table=pending_closures34392026/11/12 16:57:26 INFO Vacuumed table table=commit_receipts34402026/11/12 16:57:26 INFO Aborted multipart uploads count=0 kept=034412026/11/12 16:57:26 INFO Vacuumed table table=pending_objects34422026/11/12 16:57:26 INFO Vacuumed table table=multipart_uploads34432026/11/12 16:57:26 INFO Vacuumed table table=closures34442026/11/12 16:57:26 WARN Force mode enabled - objects will be deleted immediately without grace period34452026/11/12 16:57:26 INFO Vacuumed table table=objects34462026/11/12 16:57:26 INFO Received uploads request method=POST path=/34472026/11/12 16:57:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034482026/11/12 16:57:26 INFO Vacuumed table table=pending_closures34492026/11/12 16:57:26 INFO Received push request method=POST path=/34502026/11/12 16:57:26 INFO Vacuumed table table=commit_receipts34512026/11/12 16:57:26 INFO Received complete push request method=POST path=/34522026/11/12 16:57:26 INFO Vacuumed table table=pending_objects34532026/11/12 16:57:26 INFO Vacuumed table table=multipart_uploads34542026/11/12 16:57:26 INFO Received complete push request method=POST path=/34552026/11/12 16:57:26 INFO Vacuumed table table=closures34562026/11/12 16:57:26 INFO Completed push id=10 objects=15 duration=2.206549ms34572026/11/12 16:57:26 INFO Vacuumed table table=objects34582026/11/12 16:57:26 INFO Completed push id=4 objects=7 duration=2.19665ms34592026/11/12 16:57:26 INFO Received uploads request method=POST path=/34602026/11/12 16:57:26 INFO Received push request method=POST path=/34612026/11/12 16:57:26 INFO Aborted multipart uploads count=0 kept=034622026/11/12 16:57:26 INFO Received push request method=POST path=/34632026/11/12 16:57:26 WARN Force mode enabled - objects will be deleted immediately without grace period34642026/11/12 16:57:26 INFO Received push request method=POST path=/34652026/11/12 16:57:26 INFO Received complete push request method=POST path=/34662026/11/12 16:57:26 INFO Completed push id=10 objects=3 duration=1.94778ms34672026/11/12 16:57:26 INFO Aborted multipart uploads count=0 kept=034682026/11/12 16:57:26 INFO Received uploads request method=POST path=/34692026/11/12 16:57:26 INFO Received push request method=POST path=/34702026/11/12 16:57:26 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034712026/11/12 16:57:26 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."34722026/11/12 16:57:26 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."34732026/11/12 16:57:26 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."34742026/11/12 16:57:26 INFO Received uploads request method=POST path=/34752026/11/12 16:57:26 INFO Vacuumed table table=pending_closures34762026/11/12 16:57:26 INFO Vacuumed table table=commit_receipts34772026/11/12 16:57:26 INFO Received uploads request method=POST path=/34782026/11/12 16:57:26 INFO Vacuumed table table=pending_objects34792026/11/12 16:57:26 INFO Vacuumed table table=multipart_uploads34802026/11/12 16:57:26 INFO Vacuumed table table=closures34812026/11/12 16:57:26 INFO Received complete upload request method=POST path=/34822026/11/12 16:57:26 INFO Received push request method=POST path=/34832026/11/12 16:57:26 INFO Received push request method=POST path=/34842026/11/12 16:57:26 INFO Vacuumed table table=objects34852026/11/12 16:57:26 INFO Completed upload id=15 objects=6 duration=1.38805ms34862026/11/12 16:57:26 INFO Received uploads request method=POST path=/34872026/11/12 16:57:26 INFO Received complete push request method=POST path=/34882026/11/12 16:57:26 INFO Received complete upload request method=POST path=/34892026/11/12 16:57:26 INFO Completed push id=8 objects=6 duration=2.447399ms34902026/11/12 16:57:26 INFO Completed upload id=10 objects=10 duration=1.64244ms34912026/11/12 16:57:26 INFO Received uploads request method=POST path=/34922026/11/12 16:57:26 INFO Received push request method=POST path=/34932026/11/12 16:57:26 INFO Aborted multipart uploads count=0 kept=034942026/11/12 16:57:26 WARN Force mode enabled - objects will be deleted immediately without grace period34952026/11/12 16:57:26 INFO Received complete upload request method=POST path=/34962026/11/12 16:57:26 INFO Received uploads request method=POST path=/34972026/11/12 16:57:26 INFO Received uploads request method=POST path=/34982026/11/12 16:57:26 INFO Completed upload id=13 objects=10 duration=1.361139ms34992026/11/12 16:57:26 INFO Received uploads request method=POST path=/35002026/11/12 16:57:26 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."35012026/11/12 16:57:26 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."35022026/11/12 16:57:26 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."35032026/11/12 16:57:26 INFO Received complete push request method=POST path=/35042026/11/12 16:57:26 INFO Completed push id=9 objects=7 duration=3.999049ms35052026/11/12 16:57:26 INFO Received push request method=POST path=/35062026/11/12 16:57:26 INFO Received uploads request method=POST path=/35072026/11/12 16:57:26 INFO Received push request method=POST path=/35082026/11/12 16:57:26 INFO Aborted multipart uploads count=0 kept=035092026/11/12 16:57:26 INFO Received uploads request method=POST path=/35102026/11/12 16:57:26 WARN Force mode enabled - objects will be deleted immediately without grace period35112026/11/12 16:57:26 INFO Received uploads request method=POST path=/35122026/11/12 16:57:26 INFO Received push request method=POST path=/35132026/11/12 16:57:26 INFO Aborted multipart uploads count=0 kept=035142026/11/12 16:57:26 INFO Received push request method=POST path=/35152026/11/12 16:57:26 INFO Received uploads request method=POST path=/35162026/11/12 16:57:26 WARN Force mode enabled - objects will be deleted immediately without grace period35172026/11/12 16:57:26 INFO Received complete upload request method=POST path=/35182026/11/12 16:57:26 INFO Completed upload id=10 objects=6 duration=1.65829ms35192026/11/12 16:57:26 INFO Aborted multipart uploads count=0 kept=035202026/11/12 16:57:26 INFO Aborted multipart uploads count=0 kept=035212026/11/12 16:57:26 WARN Force mode enabled - objects will be deleted immediately without grace period35222026/11/12 16:57:26 WARN Force mode enabled - objects will be deleted immediately without grace period35232026/11/12 16:57:26 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035242026/11/12 16:57:26 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035252026/11/12 16:57:26 INFO Vacuumed table table=pending_closures35262026/11/12 16:57:26 INFO Vacuumed table table=pending_closures35272026/11/12 16:57:26 INFO Vacuumed table table=commit_receipts35282026/11/12 16:57:26 INFO Received complete upload request method=POST path=/35292026/11/12 16:57:26 INFO Vacuumed table table=commit_receipts35302026/11/12 16:57:26 INFO Received uploads request method=POST path=/35312026/11/12 16:57:26 INFO Received push request method=POST path=/35322026/11/12 16:57:26 INFO Vacuumed table table=pending_objects35332026/11/12 16:57:26 INFO Vacuumed table table=pending_objects35342026/11/12 16:57:26 INFO Vacuumed table table=multipart_uploads35352026/11/12 16:57:26 INFO Completed upload id=10 objects=7 duration=1.9372ms35362026/11/12 16:57:26 INFO Vacuumed table table=multipart_uploads35372026/11/12 16:57:26 INFO Vacuumed table table=closures35382026/11/12 16:57:26 INFO Vacuumed table table=closures35392026/11/12 16:57:26 INFO Received uploads request method=POST path=/35402026/11/12 16:57:26 INFO Vacuumed table table=objects35412026/11/12 16:57:26 INFO Vacuumed table table=objects35422026/11/12 16:57:26 INFO Received push request method=POST path=/35432026/11/12 16:57:26 INFO Received complete push request method=POST path=/35442026/11/12 16:57:26 INFO Aborted multipart uploads count=0 kept=035452026/11/12 16:57:26 INFO Received complete push request method=POST path=/35462026/11/12 16:57:26 INFO Received complete push request method=POST path=/35472026/11/12 16:57:26 INFO Aborted multipart uploads count=0 kept=035482026/11/12 16:57:26 INFO Completed push id=7 objects=7 duration=1.57469ms35492026/11/12 16:57:26 INFO Completed push id=15 objects=24 duration=4.816659ms35502026/11/12 16:57:26 INFO Completed push id=16 objects=27 duration=2.32727ms35512026/11/12 16:57:26 WARN Force mode enabled - objects will be deleted immediately without grace period35522026/11/12 16:57:26 INFO Received uploads request method=POST path=/35532026/11/12 16:57:26 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."35542026/11/12 16:57:26 INFO Received complete push request method=POST path=/35552026/11/12 16:57:26 INFO Received push request method=POST path=/35562026/11/12 16:57:26 INFO Received complete upload request method=POST path=/35572026/11/12 16:57:26 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=035582026/11/12 16:57:26 INFO Completed push id=15 objects=7 duration=2.377749ms35592026/11/12 16:57:26 INFO Vacuumed table table=pending_closures35602026/11/12 16:57:26 INFO Vacuumed table table=commit_receipts35612026/11/12 16:57:26 INFO Completed upload id=15 objects=4 duration=2.720409ms35622026/11/12 16:57:26 INFO Vacuumed table table=pending_objects35632026/11/12 16:57:26 INFO Received push request method=POST path=/35642026/11/12 16:57:26 INFO Vacuumed table table=multipart_uploads35652026/11/12 16:57:26 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=035662026/11/12 16:57:26 INFO Vacuumed table table=closures35672026/11/12 16:57:26 INFO Received uploads request method=POST path=/35682026/11/12 16:57:26 INFO Vacuumed table table=objects35692026/11/12 16:57:26 INFO Received uploads request method=POST path=/35702026/11/12 16:57:26 INFO Vacuumed table table=pending_closures35712026/11/12 16:57:26 INFO Vacuumed table table=commit_receipts35722026/11/12 16:57:26 INFO Vacuumed table table=pending_objects35732026/11/12 16:57:26 INFO Vacuumed table table=multipart_uploads35742026/11/12 16:57:26 INFO Vacuumed table table=closures35752026/11/12 16:57:26 INFO Vacuumed table table=objects35762026/11/12 16:57:26 INFO Received push request method=POST path=/35772026/11/12 16:57:26 INFO Received uploads request method=POST path=/35782026/11/12 16:57:26 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=11 objects-failed-to-delete=035792026/11/12 16:57:26 INFO Vacuumed table table=pending_closures35802026/11/12 16:57:26 INFO Received uploads request method=POST path=/35812026/11/12 16:57:26 INFO Vacuumed table table=commit_receipts35822026/11/12 16:57:26 INFO Received push request method=POST path=/35832026/11/12 16:57:26 INFO Vacuumed table table=pending_objects35842026/11/12 16:57:26 INFO Vacuumed table table=multipart_uploads35852026/11/12 16:57:26 INFO Received uploads request method=POST path=/35862026/11/12 16:57:26 INFO Vacuumed table table=closures35872026/11/12 16:57:26 INFO Received uploads request method=POST path=/35882026/11/12 16:57:26 INFO Vacuumed table table=objects35892026/11/12 16:57:26 INFO Received push request method=POST path=/35902026/11/12 16:57:26 INFO Aborted multipart uploads count=0 kept=035912026/11/12 16:57:26 INFO Received push request method=POST path=/35922026/11/12 16:57:26 INFO Received push request method=POST path=/35932026/11/12 16:57:26 WARN Force mode enabled - objects will be deleted immediately without grace period35942026/11/12 16:57:26 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035952026/11/12 16:57:26 INFO Vacuumed table table=pending_closures35962026/11/12 16:57:26 INFO Received push request method=POST path=/35972026/11/12 16:57:26 INFO Vacuumed table table=commit_receipts35982026/11/12 16:57:26 INFO Vacuumed table table=pending_objects35992026/11/12 16:57:26 INFO Vacuumed table table=multipart_uploads36002026/11/12 16:57:26 INFO Vacuumed table table=closures36012026/11/12 16:57:26 INFO Vacuumed table table=objects36022026/11/12 16:57:26 INFO Received push request method=POST path=/36032026/11/12 16:57:26 INFO Received push request method=POST path=/36042026/11/12 16:57:26 INFO Received uploads request method=POST path=/36052026/11/12 16:57:26 INFO Received uploads request method=POST path=/36062026/11/12 16:57:26 INFO Received uploads request method=POST path=/36072026/11/12 16:57:26 INFO Received push request method=POST path=/36082026/11/12 16:57:26 INFO Received complete push request method=POST path=/36092026/11/12 16:57:26 INFO Received uploads request method=POST path=/36102026/11/12 16:57:26 INFO Completed push id=8 objects=6 duration=1.721629ms36112026/11/12 16:57:26 INFO Received uploads request method=POST path=/36122026/11/12 16:57:26 INFO Received push request method=POST path=/36132026/11/12 16:57:26 INFO Received uploads request method=POST path=/36142026/11/12 16:57:27 INFO Received complete upload request method=POST path=/36152026/11/12 16:57:27 INFO Completed upload id=18 objects=11 duration=1.588789ms36162026/11/12 16:57:27 INFO Received complete upload request method=POST path=/36172026/11/12 16:57:27 INFO Received uploads request method=POST path=/36182026/11/12 16:57:27 INFO Completed upload id=10 objects=17 duration=1.89537ms36192026/11/12 16:57:27 INFO Received push request method=POST path=/36202026/11/12 16:57:27 INFO Received complete upload request method=POST path=/36212026/11/12 16:57:27 INFO Completed upload id=19 objects=25 duration=3.08819ms36222026/11/12 16:57:27 INFO Received complete upload request method=POST path=/36232026/11/12 16:57:27 INFO Completed upload id=18 objects=4 duration=1.5094ms36242026/11/12 16:57:27 INFO Received complete upload request method=POST path=/36252026/11/12 16:57:27 INFO Received uploads request method=POST path=/36262026/11/12 16:57:27 INFO Completed upload id=16 objects=3 duration=1.25254ms36272026/11/12 16:57:27 INFO Received complete push request method=POST path=/36282026/11/12 16:57:27 INFO Completed push id=12 objects=7 duration=1.4325ms36292026/11/12 16:57:27 INFO Received complete upload request method=POST path=/36302026/11/12 16:57:27 INFO Received uploads request method=POST path=/36312026/11/12 16:57:27 INFO Received uploads request method=POST path=/36322026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=036332026/11/12 16:57:27 INFO Completed upload id=23 objects=15 duration=2.906359ms36342026/11/12 16:57:27 INFO Received complete push request method=POST path=/36352026/11/12 16:57:27 WARN Force mode enabled - objects will be deleted immediately without grace period36362026/11/12 16:57:27 INFO Completed push id=18 objects=10 duration=2.281119ms36372026/11/12 16:57:27 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036382026/11/12 16:57:27 INFO Received complete push request method=POST path=/36392026/11/12 16:57:27 INFO Vacuumed table table=pending_closures36402026/11/12 16:57:27 INFO Completed push id=14 objects=3 duration=1.3678ms36412026/11/12 16:57:27 INFO Vacuumed table table=commit_receipts36422026/11/12 16:57:27 INFO Received uploads request method=POST path=/36432026/11/12 16:57:27 INFO Vacuumed table table=pending_objects36442026/11/12 16:57:27 INFO Received complete push request method=POST path=/36452026/11/12 16:57:27 INFO Vacuumed table table=multipart_uploads36462026/11/12 16:57:27 INFO Vacuumed table table=closures36472026/11/12 16:57:27 INFO Vacuumed table table=objects36482026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=036492026/11/12 16:57:27 INFO Completed push id=14 objects=4 duration=3.9494ms36502026/11/12 16:57:27 WARN Force mode enabled - objects will be deleted immediately without grace period36512026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=036522026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=036532026/11/12 16:57:27 INFO Received push request method=POST path=/36542026/11/12 16:57:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=036552026/11/12 16:57:27 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036562026/11/12 16:57:27 INFO Vacuumed table table=pending_closures36572026/11/12 16:57:27 INFO Vacuumed table table=commit_receipts36582026/11/12 16:57:27 INFO Vacuumed table table=pending_closures36592026/11/12 16:57:27 INFO Received push request method=POST path=/36602026/11/12 16:57:27 INFO Vacuumed table table=commit_receipts36612026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=036622026/11/12 16:57:27 INFO Vacuumed table table=pending_objects36632026/11/12 16:57:27 INFO Vacuumed table table=multipart_uploads36642026/11/12 16:57:27 INFO Received push request method=POST path=/36652026/11/12 16:57:27 INFO Vacuumed table table=closures36662026/11/12 16:57:27 INFO Vacuumed table table=pending_objects36672026/11/12 16:57:27 INFO Vacuumed table table=multipart_uploads36682026/11/12 16:57:27 INFO Received push request method=POST path=/36692026/11/12 16:57:27 INFO Vacuumed table table=closures36702026/11/12 16:57:27 INFO Vacuumed table table=objects36712026/11/12 16:57:27 WARN Force mode enabled - objects will be deleted immediately without grace period36722026/11/12 16:57:27 INFO Vacuumed table table=objects36732026/11/12 16:57:27 INFO Received push request method=POST path=/36742026/11/12 16:57:27 INFO Received push request method=POST path=/36752026/11/12 16:57:27 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."36762026/11/12 16:57:27 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=036772026/11/12 16:57:27 INFO Received complete upload request method=POST path=/36782026/11/12 16:57:27 INFO Vacuumed table table=pending_closures36792026/11/12 16:57:27 INFO Completed upload id=6 objects=6 duration=1.52226ms36802026/11/12 16:57:27 INFO Vacuumed table table=commit_receipts36812026/11/12 16:57:27 INFO Vacuumed table table=pending_objects36822026/11/12 16:57:27 INFO Vacuumed table table=multipart_uploads36832026/11/12 16:57:27 INFO Vacuumed table table=closures36842026/11/12 16:57:27 INFO Vacuumed table table=objects36852026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=036862026/11/12 16:57:27 INFO Received complete upload request method=POST path=/36872026/11/12 16:57:27 WARN Force mode enabled - objects will be deleted immediately without grace period36882026/11/12 16:57:27 INFO Completed upload id=22 objects=2 duration=1.394669ms36892026/11/12 16:57:27 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036902026/11/12 16:57:27 INFO Vacuumed table table=pending_closures36912026/11/12 16:57:27 INFO Vacuumed table table=commit_receipts36922026/11/12 16:57:27 INFO Vacuumed table table=pending_objects36932026/11/12 16:57:27 INFO Received push request method=POST path=/36942026/11/12 16:57:27 INFO Vacuumed table table=multipart_uploads36952026/11/12 16:57:27 INFO Vacuumed table table=closures36962026/11/12 16:57:27 INFO Received push request method=POST path=/36972026/11/12 16:57:27 INFO Vacuumed table table=objects36982026/11/12 16:57:27 INFO Received uploads request method=POST path=/36992026/11/12 16:57:27 INFO Received complete upload request method=POST path=/37002026/11/12 16:57:27 INFO Completed upload id=19 objects=6 duration=1.354509ms37012026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=037022026/11/12 16:57:27 WARN Force mode enabled - objects will be deleted immediately without grace period37032026/11/12 16:57:27 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037042026/11/12 16:57:27 INFO Vacuumed table table=pending_closures37052026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=037062026/11/12 16:57:27 INFO Vacuumed table table=commit_receipts37072026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=037082026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=037092026/11/12 16:57:27 INFO Received uploads request method=POST path=/37102026/11/12 16:57:27 INFO Received uploads request method=POST path=/37112026/11/12 16:57:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037122026/11/12 16:57:27 WARN Force mode enabled - objects will be deleted immediately without grace period37132026/11/12 16:57:27 INFO Vacuumed table table=pending_objects37142026/11/12 16:57:27 INFO Vacuumed table table=multipart_uploads37152026/11/12 16:57:27 INFO Vacuumed table table=pending_closures37162026/11/12 16:57:27 INFO Received push request method=POST path=/37172026/11/12 16:57:27 INFO Vacuumed table table=commit_receipts37182026/11/12 16:57:27 INFO Vacuumed table table=closures37192026/11/12 16:57:27 INFO Received complete push request method=POST path=/37202026/11/12 16:57:27 INFO Vacuumed table table=pending_objects37212026/11/12 16:57:27 INFO Vacuumed table table=objects37222026/11/12 16:57:27 INFO Vacuumed table table=multipart_uploads37232026/11/12 16:57:27 INFO Vacuumed table table=closures37242026/11/12 16:57:27 INFO Completed push id=14 objects=6 duration=2.1466ms37252026/11/12 16:57:27 INFO Vacuumed table table=objects37262026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=037272026/11/12 16:57:27 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=037282026/11/12 16:57:27 WARN Force mode enabled - objects will be deleted immediately without grace period37292026/11/12 16:57:27 INFO Received push request method=POST path=/37302026/11/12 16:57:27 INFO Vacuumed table table=pending_closures37312026/11/12 16:57:27 INFO Vacuumed table table=commit_receipts37322026/11/12 16:57:27 INFO Vacuumed table table=pending_objects37332026/11/12 16:57:27 INFO Vacuumed table table=multipart_uploads37342026/11/12 16:57:27 INFO Garbage collection completed failed-uploads-deleted=5 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037352026/11/12 16:57:27 INFO Vacuumed table table=closures37362026/11/12 16:57:27 INFO Vacuumed table table=objects37372026/11/12 16:57:27 INFO Received complete upload request method=POST path=/37382026/11/12 16:57:27 INFO Received complete upload request method=POST path=/37392026/11/12 16:57:27 INFO Vacuumed table table=pending_closures37402026/11/12 16:57:27 INFO Completed upload id=22 objects=7 duration=1.27881ms37412026/11/12 16:57:27 INFO Vacuumed table table=commit_receipts37422026/11/12 16:57:27 INFO Completed upload id=14 objects=8 duration=2.666499ms37432026/11/12 16:57:27 INFO Vacuumed table table=pending_objects37442026/11/12 16:57:27 INFO Vacuumed table table=multipart_uploads37452026/11/12 16:57:27 INFO Received push request method=POST path=/37462026/11/12 16:57:27 INFO Vacuumed table table=closures37472026/11/12 16:57:27 INFO Vacuumed table table=objects37482026/11/12 16:57:27 INFO Received push request method=POST path=/37492026/11/12 16:57:27 INFO Received uploads request method=POST path=/37502026/11/12 16:57:27 INFO Received push request method=POST path=/37512026/11/12 16:57:27 INFO Received complete upload request method=POST path=/37522026/11/12 16:57:27 INFO Completed upload id=18 objects=4 duration=2.48963ms37532026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=037542026/11/12 16:57:27 WARN Force mode enabled - objects will be deleted immediately without grace period37552026/11/12 16:57:27 INFO Received uploads request method=POST path=/37562026/11/12 16:57:27 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037572026/11/12 16:57:27 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."37582026/11/12 16:57:27 INFO Vacuumed table table=pending_closures37592026/11/12 16:57:27 INFO Vacuumed table table=commit_receipts37602026/11/12 16:57:27 INFO Vacuumed table table=pending_objects37612026/11/12 16:57:27 INFO Vacuumed table table=multipart_uploads37622026/11/12 16:57:27 INFO Vacuumed table table=closures37632026/11/12 16:57:27 INFO Received uploads request method=POST path=/37642026/11/12 16:57:27 INFO Vacuumed table table=objects37652026/11/12 16:57:27 INFO Received complete push request method=POST path=/37662026/11/12 16:57:27 INFO Received complete upload request method=POST path=/37672026/11/12 16:57:27 INFO Completed push id=3 objects=21 duration=4.716269ms37682026/11/12 16:57:27 INFO Received push request method=POST path=/37692026/11/12 16:57:27 INFO Received uploads request method=POST path=/37702026/11/12 16:57:27 INFO Completed upload id=17 objects=6 duration=3.17804ms37712026/11/12 16:57:27 INFO Received complete push request method=POST path=/37722026/11/12 16:57:27 INFO Completed push id=1 objects=11 duration=2.371739ms37732026/11/12 16:57:27 INFO Received uploads request method=POST path=/37742026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=037752026/11/12 16:57:27 INFO Received uploads request method=POST path=/37762026/11/12 16:57:27 WARN Force mode enabled - objects will be deleted immediately without grace period37772026/11/12 16:57:27 INFO Received complete push request method=POST path=/37782026/11/12 16:57:27 INFO Received push request method=POST path=/37792026/11/12 16:57:27 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037802026/11/12 16:57:27 INFO Vacuumed table table=pending_closures37812026/11/12 16:57:27 INFO Vacuumed table table=commit_receipts37822026/11/12 16:57:27 INFO Completed push id=9 objects=4 duration=4.305659ms37832026/11/12 16:57:27 INFO Vacuumed table table=pending_objects37842026/11/12 16:57:27 INFO Vacuumed table table=multipart_uploads37852026/11/12 16:57:27 INFO Vacuumed table table=closures37862026/11/12 16:57:27 INFO Vacuumed table table=objects37872026/11/12 16:57:27 INFO Received complete push request method=POST path=/37882026/11/12 16:57:27 INFO Received uploads request method=POST path=/37892026/11/12 16:57:27 INFO Completed push id=17 objects=6 duration=2.14719ms37902026/11/12 16:57:27 INFO Received uploads request method=POST path=/37912026/11/12 16:57:27 INFO Received push request method=POST path=/37922026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=037932026/11/12 16:57:27 INFO Received uploads request method=POST path=/37942026/11/12 16:57:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=1 objects-deleted-after-grace-period=0 objects-failed-to-delete=037952026/11/12 16:57:27 INFO Vacuumed table table=pending_closures37962026/11/12 16:57:27 INFO Vacuumed table table=commit_receipts37972026/11/12 16:57:27 INFO Received uploads request method=POST path=/37982026/11/12 16:57:27 INFO Vacuumed table table=pending_objects37992026/11/12 16:57:27 INFO Received complete push request method=POST path=/38002026/11/12 16:57:27 INFO Vacuumed table table=multipart_uploads38012026/11/12 16:57:27 INFO Received complete upload request method=POST path=/38022026/11/12 16:57:27 INFO Vacuumed table table=closures38032026/11/12 16:57:27 INFO Completed push id=15 objects=4 duration=1.441359ms38042026/11/12 16:57:27 INFO Completed upload id=16 objects=3 duration=1.23512ms38052026/11/12 16:57:27 INFO Received complete push request method=POST path=/38062026/11/12 16:57:27 INFO Vacuumed table table=objects38072026/11/12 16:57:27 INFO Received push request method=POST path=/38082026/11/12 16:57:27 INFO Completed push id=24 objects=9 duration=1.94608ms38092026/11/12 16:57:27 INFO Received uploads request method=POST path=/38102026/11/12 16:57:27 INFO Received push request method=POST path=/38112026/11/12 16:57:27 INFO Received push request method=POST path=/38122026/11/12 16:57:27 INFO Received push request method=POST path=/38132026/11/12 16:57:27 INFO Received push request method=POST path=/38142026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=038152026/11/12 16:57:27 WARN Force mode enabled - objects will be deleted immediately without grace period38162026/11/12 16:57:27 INFO Received uploads request method=POST path=/38172026/11/12 16:57:27 INFO Received push request method=POST path=/38182026/11/12 16:57:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038192026/11/12 16:57:27 INFO Received push request method=POST path=/38202026/11/12 16:57:27 INFO Vacuumed table table=pending_closures38212026/11/12 16:57:27 INFO Vacuumed table table=commit_receipts38222026/11/12 16:57:27 INFO Vacuumed table table=pending_objects38232026/11/12 16:57:27 INFO Vacuumed table table=multipart_uploads38242026/11/12 16:57:27 INFO Vacuumed table table=closures38252026/11/12 16:57:27 INFO Received push request method=POST path=/38262026/11/12 16:57:27 INFO Vacuumed table table=objects38272026/11/12 16:57:27 INFO Received push request method=POST path=/38282026/11/12 16:57:27 INFO Received uploads request method=POST path=/38292026/11/12 16:57:27 INFO Received push request method=POST path=/38302026/11/12 16:57:27 INFO Received uploads request method=POST path=/38312026/11/12 16:57:27 INFO Received push request method=POST path=/38322026/11/12 16:57:27 INFO Received push request method=POST path=/38332026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=038342026/11/12 16:57:27 WARN Force mode enabled - objects will be deleted immediately without grace period38352026/11/12 16:57:27 INFO Received complete push request method=POST path=/38362026/11/12 16:57:27 INFO Received push request method=POST path=/38372026/11/12 16:57:27 INFO Received uploads request method=POST path=/38382026/11/12 16:57:27 INFO Received push request method=POST path=/38392026/11/12 16:57:27 INFO Completed push id=16 objects=16 duration=4.551259ms38402026/11/12 16:57:27 INFO Received push request method=POST path=/38412026/11/12 16:57:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=038422026/11/12 16:57:27 INFO Received push request method=POST path=/38432026/11/12 16:57:27 INFO Vacuumed table table=pending_closures38442026/11/12 16:57:27 INFO Vacuumed table table=commit_receipts38452026/11/12 16:57:27 INFO Vacuumed table table=pending_objects38462026/11/12 16:57:27 INFO Received push request method=POST path=/38472026/11/12 16:57:27 INFO Vacuumed table table=multipart_uploads38482026/11/12 16:57:27 INFO Vacuumed table table=closures38492026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=038502026/11/12 16:57:27 INFO Vacuumed table table=objects38512026/11/12 16:57:27 INFO Received push request method=POST path=/38522026/11/12 16:57:27 WARN Force mode enabled - objects will be deleted immediately without grace period38532026/11/12 16:57:27 INFO Received uploads request method=POST path=/38542026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=038552026/11/12 16:57:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038562026/11/12 16:57:27 INFO Vacuumed table table=pending_closures38572026/11/12 16:57:27 INFO Vacuumed table table=commit_receipts38582026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=038592026/11/12 16:57:27 INFO Vacuumed table table=pending_objects38602026/11/12 16:57:27 INFO Vacuumed table table=multipart_uploads38612026/11/12 16:57:27 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038622026/11/12 16:57:27 INFO Vacuumed table table=closures38632026/11/12 16:57:27 INFO Vacuumed table table=objects38642026/11/12 16:57:27 INFO Received uploads request method=POST path=/38652026/11/12 16:57:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038662026/11/12 16:57:27 INFO Vacuumed table table=pending_closures38672026/11/12 16:57:27 INFO Vacuumed table table=commit_receipts38682026/11/12 16:57:27 INFO Vacuumed table table=pending_closures38692026/11/12 16:57:27 INFO Vacuumed table table=commit_receipts38702026/11/12 16:57:27 INFO Vacuumed table table=pending_objects38712026/11/12 16:57:27 INFO Vacuumed table table=multipart_uploads38722026/11/12 16:57:27 INFO Vacuumed table table=closures38732026/11/12 16:57:27 INFO Vacuumed table table=pending_objects38742026/11/12 16:57:27 INFO Vacuumed table table=multipart_uploads38752026/11/12 16:57:27 INFO Vacuumed table table=objects38762026/11/12 16:57:27 INFO Vacuumed table table=closures38772026/11/12 16:57:27 INFO Vacuumed table table=objects38782026/11/12 16:57:27 INFO Received uploads request method=POST path=/38792026/11/12 16:57:27 INFO Received uploads request method=POST path=/38802026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=038812026/11/12 16:57:27 INFO Received push request method=POST path=/38822026/11/12 16:57:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038832026/11/12 16:57:27 INFO Received complete push request method=POST path=/38842026/11/12 16:57:27 INFO Vacuumed table table=pending_closures38852026/11/12 16:57:27 INFO Vacuumed table table=commit_receipts38862026/11/12 16:57:27 INFO Completed push id=14 objects=3 duration=1.782839ms38872026/11/12 16:57:27 INFO Received uploads request method=POST path=/38882026/11/12 16:57:27 INFO Received push request method=POST path=/38892026/11/12 16:57:27 INFO Vacuumed table table=pending_objects38902026/11/12 16:57:27 INFO Vacuumed table table=multipart_uploads38912026/11/12 16:57:27 INFO Vacuumed table table=closures38922026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=038932026/11/12 16:57:27 INFO Vacuumed table table=objects38942026/11/12 16:57:27 INFO Received push request method=POST path=/38952026/11/12 16:57:27 INFO Received uploads request method=POST path=/38962026/11/12 16:57:27 INFO Received uploads request method=POST path=/38972026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=038982026/11/12 16:57:27 INFO Received uploads request method=POST path=/38992026/11/12 16:57:27 WARN Force mode enabled - objects will be deleted immediately without grace period39002026/11/12 16:57:27 INFO Received complete push request method=POST path=/39012026/11/12 16:57:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039022026/11/12 16:57:27 INFO Vacuumed table table=pending_closures39032026/11/12 16:57:27 INFO Completed push id=15 objects=5 duration=2.27187ms39042026/11/12 16:57:27 INFO Vacuumed table table=commit_receipts39052026/11/12 16:57:27 INFO Vacuumed table table=pending_objects39062026/11/12 16:57:27 INFO Vacuumed table table=multipart_uploads39072026/11/12 16:57:27 INFO Vacuumed table table=closures39082026/11/12 16:57:27 INFO Vacuumed table table=objects39092026/11/12 16:57:27 INFO Received uploads request method=POST path=/39102026/11/12 16:57:27 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."39112026/11/12 16:57:27 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."39122026/11/12 16:57:27 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."39132026/11/12 16:57:27 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."39142026/11/12 16:57:27 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."39152026/11/12 16:57:27 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."39162026/11/12 16:57:27 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."39172026/11/12 16:57:27 INFO Received uploads request method=POST path=/39182026/11/12 16:57:27 INFO Received complete push request method=POST path=/39192026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=039202026/11/12 16:57:27 INFO Received push request method=POST path=/39212026/11/12 16:57:27 WARN Force mode enabled - objects will be deleted immediately without grace period39222026/11/12 16:57:27 INFO Completed push id=16 objects=4 duration=3.321839ms39232026/11/12 16:57:27 INFO Received uploads request method=POST path=/39242026/11/12 16:57:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039252026/11/12 16:57:27 INFO Vacuumed table table=pending_closures39262026/11/12 16:57:27 INFO Vacuumed table table=commit_receipts39272026/11/12 16:57:27 INFO Received complete push request method=POST path=/39282026/11/12 16:57:27 INFO Vacuumed table table=pending_objects39292026/11/12 16:57:27 INFO Vacuumed table table=multipart_uploads39302026/11/12 16:57:27 INFO Vacuumed table table=closures39312026/11/12 16:57:27 INFO Completed push id=12 objects=2 duration=3.535039ms39322026/11/12 16:57:27 INFO Vacuumed table table=objects39332026/11/12 16:57:27 INFO Received uploads request method=POST path=/39342026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=039352026/11/12 16:57:27 WARN Force mode enabled - objects will be deleted immediately without grace period39362026/11/12 16:57:27 INFO Received complete upload request method=POST path=/39372026/11/12 16:57:27 INFO Received push request method=POST path=/39382026/11/12 16:57:27 INFO Received uploads request method=POST path=/39392026/11/12 16:57:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039402026/11/12 16:57:27 INFO Vacuumed table table=pending_closures39412026/11/12 16:57:27 INFO Completed upload id=16 objects=17 duration=3.18598ms39422026/11/12 16:57:27 INFO Vacuumed table table=commit_receipts39432026/11/12 16:57:27 INFO Received complete push request method=POST path=/39442026/11/12 16:57:27 INFO Received uploads request method=POST path=/39452026/11/12 16:57:27 INFO Vacuumed table table=pending_objects39462026/11/12 16:57:27 INFO Vacuumed table table=multipart_uploads39472026/11/12 16:57:27 INFO Received uploads request method=POST path=/39482026/11/12 16:57:27 INFO Vacuumed table table=closures39492026/11/12 16:57:27 INFO Completed push id=17 objects=22 duration=3.460809ms39502026/11/12 16:57:27 INFO Vacuumed table table=objects39512026/11/12 16:57:27 INFO Received uploads request method=POST path=/39522026/11/12 16:57:27 INFO Received complete upload request method=POST path=/39532026/11/12 16:57:27 INFO Completed upload id=20 objects=4 duration=1.08605ms39542026/11/12 16:57:27 INFO Received uploads request method=POST path=/39552026/11/12 16:57:27 INFO Received complete push request method=POST path=/39562026/11/12 16:57:27 INFO Completed push id=4 objects=6 duration=1.85709ms39572026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=039582026/11/12 16:57:27 INFO Received uploads request method=POST path=/39592026/11/12 16:57:27 INFO Received uploads request method=POST path=/39602026/11/12 16:57:27 INFO Received complete push request method=POST path=/39612026/11/12 16:57:27 WARN Force mode enabled - objects will be deleted immediately without grace period39622026/11/12 16:57:27 INFO Completed push id=19 objects=8 duration=3.420229ms39632026/11/12 16:57:27 INFO Received uploads request method=POST path=/39642026/11/12 16:57:27 INFO Received complete upload request method=POST path=/39652026/11/12 16:57:27 INFO Received push request method=POST path=/39662026/11/12 16:57:27 INFO Completed upload id=20 objects=5 duration=1.428999ms39672026/11/12 16:57:27 INFO Received uploads request method=POST path=/39682026/11/12 16:57:27 INFO Received push request method=POST path=/39692026/11/12 16:57:27 INFO Received uploads request method=POST path=/39702026/11/12 16:57:27 INFO Received push request method=POST path=/39712026/11/12 16:57:27 INFO Received uploads request method=POST path=/39722026/11/12 16:57:27 INFO Received complete upload request method=POST path=/39732026/11/12 16:57:27 INFO Completed upload id=15 objects=9 duration=1.432689ms39742026/11/12 16:57:27 INFO Received uploads request method=POST path=/39752026/11/12 16:57:27 INFO Received complete push request method=POST path=/39762026/11/12 16:57:27 INFO Completed push id=16 objects=5 duration=1.360209ms39772026/11/12 16:57:27 INFO Received uploads request method=POST path=/39782026/11/12 16:57:27 INFO Received uploads request method=POST path=/39792026/11/12 16:57:27 INFO Received complete push request method=POST path=/39802026/11/12 16:57:27 INFO Completed push id=21 objects=3 duration=1.46401ms39812026/11/12 16:57:27 INFO Received push request method=POST path=/39822026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=039832026/11/12 16:57:27 INFO Received push request method=POST path=/39842026/11/12 16:57:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039852026/11/12 16:57:27 INFO Vacuumed table table=pending_closures39862026/11/12 16:57:27 INFO Vacuumed table table=commit_receipts39872026/11/12 16:57:27 INFO Vacuumed table table=pending_objects39882026/11/12 16:57:27 INFO Received uploads request method=POST path=/39892026/11/12 16:57:27 INFO Vacuumed table table=multipart_uploads39902026/11/12 16:57:27 INFO Received push request method=POST path=/39912026/11/12 16:57:27 INFO Vacuumed table table=closures39922026/11/12 16:57:27 INFO Received uploads request method=POST path=/39932026/11/12 16:57:27 INFO Vacuumed table table=objects39942026/11/12 16:57:27 INFO Received uploads request method=POST path=/39952026/11/12 16:57:27 INFO Received push request method=POST path=/39962026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=039972026/11/12 16:57:27 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039982026/11/12 16:57:27 INFO Vacuumed table table=pending_closures39992026/11/12 16:57:27 INFO Received uploads request method=POST path=/40002026/11/12 16:57:27 INFO Vacuumed table table=commit_receipts40012026/11/12 16:57:27 INFO Vacuumed table table=pending_objects40022026/11/12 16:57:27 INFO Vacuumed table table=multipart_uploads40032026/11/12 16:57:27 INFO Received uploads request method=POST path=/40042026/11/12 16:57:27 INFO Vacuumed table table=closures40052026/11/12 16:57:27 INFO Vacuumed table table=objects40062026/11/12 16:57:27 INFO Received complete upload request method=POST path=/40072026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=040082026/11/12 16:57:27 INFO Completed upload id=7 objects=10 duration=1.91388ms40092026/11/12 16:57:27 INFO Received complete upload request method=POST path=/40102026/11/12 16:57:27 WARN Force mode enabled - objects will be deleted immediately without grace period40112026/11/12 16:57:27 INFO Completed upload id=23 objects=16 duration=2.920129ms40122026/11/12 16:57:27 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040132026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=040142026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=040152026/11/12 16:57:27 INFO Vacuumed table table=pending_closures40162026/11/12 16:57:27 INFO Vacuumed table table=commit_receipts40172026/11/12 16:57:27 WARN Force mode enabled - objects will be deleted immediately without grace period40182026/11/12 16:57:27 INFO Vacuumed table table=pending_objects40192026/11/12 16:57:27 INFO Received push request method=POST path=/40202026/11/12 16:57:27 INFO Vacuumed table table=multipart_uploads40212026/11/12 16:57:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040222026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=040232026/11/12 16:57:27 INFO Vacuumed table table=closures40242026/11/12 16:57:27 INFO Vacuumed table table=pending_closures40252026/11/12 16:57:27 INFO Vacuumed table table=commit_receipts40262026/11/12 16:57:27 INFO Received complete upload request method=POST path=/40272026/11/12 16:57:27 INFO Received complete push request method=POST path=/40282026/11/12 16:57:27 INFO Vacuumed table table=objects40292026/11/12 16:57:27 WARN Force mode enabled - objects will be deleted immediately without grace period40302026/11/12 16:57:27 INFO Vacuumed table table=pending_objects40312026/11/12 16:57:27 INFO Vacuumed table table=multipart_uploads40322026/11/12 16:57:27 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=040332026/11/12 16:57:27 INFO Vacuumed table table=closures40342026/11/12 16:57:27 INFO Completed push id=21 objects=17 duration=2.42935ms40352026/11/12 16:57:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040362026/11/12 16:57:27 INFO Completed upload id=23 objects=18 duration=3.00391ms40372026/11/12 16:57:27 INFO Vacuumed table table=objects40382026/11/12 16:57:27 INFO Vacuumed table table=pending_closures40392026/11/12 16:57:27 INFO Vacuumed table table=pending_closures40402026/11/12 16:57:27 INFO Vacuumed table table=commit_receipts40412026/11/12 16:57:27 INFO Vacuumed table table=commit_receipts40422026/11/12 16:57:27 INFO Vacuumed table table=pending_objects40432026/11/12 16:57:27 INFO Vacuumed table table=pending_objects40442026/11/12 16:57:27 INFO Vacuumed table table=multipart_uploads40452026/11/12 16:57:27 INFO Vacuumed table table=multipart_uploads40462026/11/12 16:57:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=8 objects-deleted-after-grace-period=8 objects-failed-to-delete=040472026/11/12 16:57:27 INFO Vacuumed table table=closures40482026/11/12 16:57:27 INFO Vacuumed table table=closures40492026/11/12 16:57:27 INFO Vacuumed table table=pending_closures40502026/11/12 16:57:27 INFO Received push request method=POST path=/40512026/11/12 16:57:27 INFO Vacuumed table table=objects40522026/11/12 16:57:27 INFO Vacuumed table table=objects40532026/11/12 16:57:27 INFO Vacuumed table table=commit_receipts40542026/11/12 16:57:27 INFO Received uploads request method=POST path=/40552026/11/12 16:57:27 INFO Vacuumed table table=pending_objects40562026/11/12 16:57:27 INFO Vacuumed table table=multipart_uploads40572026/11/12 16:57:27 INFO Vacuumed table table=closures40582026/11/12 16:57:27 INFO Vacuumed table table=objects40592026/11/12 16:57:27 INFO Received push request method=POST path=/40602026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=040612026/11/12 16:57:27 WARN Force mode enabled - objects will be deleted immediately without grace period40622026/11/12 16:57:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040632026/11/12 16:57:27 INFO Vacuumed table table=pending_closures40642026/11/12 16:57:27 INFO Received complete push request method=POST path=/40652026/11/12 16:57:27 INFO Vacuumed table table=commit_receipts40662026/11/12 16:57:27 INFO Vacuumed table table=pending_objects40672026/11/12 16:57:27 INFO Vacuumed table table=multipart_uploads40682026/11/12 16:57:27 INFO Vacuumed table table=closures40692026/11/12 16:57:27 INFO Received push request method=POST path=/40702026/11/12 16:57:27 INFO Completed push id=18 objects=6 duration=3.74469ms40712026/11/12 16:57:27 INFO Received uploads request method=POST path=/40722026/11/12 16:57:27 INFO Vacuumed table table=objects40732026/11/12 16:57:27 INFO Received push request method=POST path=/40742026/11/12 16:57:27 INFO Received complete push request method=POST path=/40752026/11/12 16:57:27 INFO Received uploads request method=POST path=/40762026/11/12 16:57:27 INFO Completed push id=19 objects=7 duration=2.308049ms40772026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=040782026/11/12 16:57:27 INFO Received uploads request method=POST path=/40792026/11/12 16:57:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040802026/11/12 16:57:27 INFO Vacuumed table table=pending_closures40812026/11/12 16:57:27 INFO Vacuumed table table=commit_receipts40822026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=040832026/11/12 16:57:27 INFO Vacuumed table table=pending_objects40842026/11/12 16:57:27 INFO Vacuumed table table=multipart_uploads40852026/11/12 16:57:27 INFO Vacuumed table table=closures40862026/11/12 16:57:27 INFO Received push request method=POST path=/40872026/11/12 16:57:27 INFO Vacuumed table table=objects40882026/11/12 16:57:27 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=3 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040892026/11/12 16:57:27 INFO Vacuumed table table=pending_closures40902026/11/12 16:57:27 INFO Vacuumed table table=commit_receipts40912026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=040922026/11/12 16:57:27 INFO Received complete upload request method=POST path=/40932026/11/12 16:57:27 INFO Vacuumed table table=pending_objects40942026/11/12 16:57:27 INFO Vacuumed table table=multipart_uploads40952026/11/12 16:57:27 WARN Force mode enabled - objects will be deleted immediately without grace period40962026/11/12 16:57:27 INFO Completed upload id=23 objects=9 duration=1.21691ms40972026/11/12 16:57:27 INFO Vacuumed table table=closures40982026/11/12 16:57:27 INFO Received uploads request method=POST path=/40992026/11/12 16:57:27 INFO Vacuumed table table=objects41002026/11/12 16:57:27 INFO Received uploads request method=POST path=/41012026/11/12 16:57:27 INFO Received push request method=POST path=/41022026/11/12 16:57:27 INFO Received uploads request method=POST path=/41032026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=041042026/11/12 16:57:27 INFO Received push request method=POST path=/41052026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=041062026/11/12 16:57:27 WARN Force mode enabled - objects will be deleted immediately without grace period41072026/11/12 16:57:27 WARN Force mode enabled - objects will be deleted immediately without grace period41082026/11/12 16:57:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041092026/11/12 16:57:27 INFO Vacuumed table table=pending_closures41102026/11/12 16:57:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041112026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=041122026/11/12 16:57:27 INFO Vacuumed table table=commit_receipts41132026/11/12 16:57:27 INFO Vacuumed table table=pending_closures41142026/11/12 16:57:27 INFO Vacuumed table table=pending_objects41152026/11/12 16:57:27 INFO Vacuumed table table=commit_receipts41162026/11/12 16:57:27 INFO Vacuumed table table=multipart_uploads41172026/11/12 16:57:27 INFO Vacuumed table table=pending_objects41182026/11/12 16:57:27 INFO Received uploads request method=POST path=/41192026/11/12 16:57:27 INFO Vacuumed table table=multipart_uploads41202026/11/12 16:57:27 INFO Vacuumed table table=closures41212026/11/12 16:57:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041222026/11/12 16:57:27 INFO Vacuumed table table=closures41232026/11/12 16:57:27 INFO Vacuumed table table=objects41242026/11/12 16:57:27 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."41252026/11/12 16:57:27 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."41262026/11/12 16:57:27 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."41272026/11/12 16:57:27 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."41282026/11/12 16:57:27 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."41292026/11/12 16:57:27 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."41302026/11/12 16:57:27 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."41312026/11/12 16:57:27 INFO Vacuumed table table=pending_closures41322026/11/12 16:57:27 INFO Vacuumed table table=objects41332026/11/12 16:57:27 INFO Vacuumed table table=commit_receipts41342026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=041352026/11/12 16:57:27 INFO Vacuumed table table=pending_objects41362026/11/12 16:57:27 INFO Vacuumed table table=multipart_uploads41372026/11/12 16:57:27 INFO Vacuumed table table=closures41382026/11/12 16:57:27 INFO Vacuumed table table=objects41392026/11/12 16:57:27 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041402026/11/12 16:57:27 INFO Received uploads request method=POST path=/41412026/11/12 16:57:27 INFO Vacuumed table table=pending_closures41422026/11/12 16:57:27 INFO Vacuumed table table=commit_receipts41432026/11/12 16:57:27 INFO Vacuumed table table=pending_objects41442026/11/12 16:57:27 INFO Received push request method=POST path=/41452026/11/12 16:57:27 INFO Vacuumed table table=multipart_uploads41462026/11/12 16:57:27 INFO Vacuumed table table=closures41472026/11/12 16:57:27 INFO Vacuumed table table=objects41482026/11/12 16:57:27 INFO Received complete upload request method=POST path=/41492026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=041502026/11/12 16:57:27 INFO Completed upload id=22 objects=22 duration=1.50973ms41512026/11/12 16:57:27 WARN Force mode enabled - objects will be deleted immediately without grace period41522026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=041532026/11/12 16:57:27 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041542026/11/12 16:57:27 INFO Received complete upload request method=POST path=/41552026/11/12 16:57:27 INFO Received uploads request method=POST path=/41562026/11/12 16:57:27 INFO Vacuumed table table=pending_closures41572026/11/12 16:57:27 INFO Completed upload id=15 objects=3 duration=2.329739ms41582026/11/12 16:57:27 WARN Force mode enabled - objects will be deleted immediately without grace period41592026/11/12 16:57:27 INFO Vacuumed table table=commit_receipts41602026/11/12 16:57:27 INFO Vacuumed table table=pending_objects41612026/11/12 16:57:27 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041622026/11/12 16:57:27 INFO Vacuumed table table=multipart_uploads41632026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=041642026/11/12 16:57:27 INFO Vacuumed table table=closures41652026/11/12 16:57:27 INFO Vacuumed table table=pending_closures41662026/11/12 16:57:27 WARN Force mode enabled - objects will be deleted immediately without grace period41672026/11/12 16:57:27 INFO Vacuumed table table=commit_receipts41682026/11/12 16:57:27 INFO Vacuumed table table=objects41692026/11/12 16:57:27 INFO Received uploads request method=POST path=/41702026/11/12 16:57:27 INFO Vacuumed table table=pending_objects41712026/11/12 16:57:27 INFO Received complete push request method=POST path=/41722026/11/12 16:57:27 INFO Vacuumed table table=multipart_uploads41732026/11/12 16:57:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041742026/11/12 16:57:27 INFO Vacuumed table table=closures41752026/11/12 16:57:27 INFO Vacuumed table table=pending_closures41762026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=041772026/11/12 16:57:27 INFO Vacuumed table table=commit_receipts41782026/11/12 16:57:27 INFO Vacuumed table table=objects41792026/11/12 16:57:27 INFO Completed push id=22 objects=3 duration=3.170209ms41802026/11/12 16:57:27 INFO Vacuumed table table=pending_objects41812026/11/12 16:57:27 INFO Vacuumed table table=multipart_uploads41822026/11/12 16:57:27 INFO Received push request method=POST path=/41832026/11/12 16:57:27 INFO Vacuumed table table=closures41842026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=041852026/11/12 16:57:27 INFO Vacuumed table table=objects41862026/11/12 16:57:27 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041872026/11/12 16:57:27 INFO Vacuumed table table=pending_closures41882026/11/12 16:57:27 INFO Received push request method=POST path=/41892026/11/12 16:57:27 INFO Received push request method=POST path=/41902026/11/12 16:57:27 WARN Force mode enabled - objects will be deleted immediately without grace period41912026/11/12 16:57:27 INFO Vacuumed table table=commit_receipts41922026/11/12 16:57:27 INFO Received push request method=POST path=/41932026/11/12 16:57:27 INFO Vacuumed table table=pending_objects41942026/11/12 16:57:27 INFO Vacuumed table table=multipart_uploads41952026/11/12 16:57:27 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041962026/11/12 16:57:27 INFO Vacuumed table table=closures41972026/11/12 16:57:27 INFO Vacuumed table table=pending_closures41982026/11/12 16:57:27 INFO Vacuumed table table=objects41992026/11/12 16:57:27 INFO Received complete upload request method=POST path=/42002026/11/12 16:57:27 INFO Vacuumed table table=commit_receipts42012026/11/12 16:57:27 INFO Received push request method=POST path=/42022026/11/12 16:57:27 INFO Completed upload id=11 objects=7 duration=1.46765ms42032026/11/12 16:57:27 INFO Received complete upload request method=POST path=/42042026/11/12 16:57:27 INFO Vacuumed table table=pending_objects42052026/11/12 16:57:27 INFO Vacuumed table table=multipart_uploads42062026/11/12 16:57:27 INFO Completed upload id=21 objects=3 duration=1.04598ms42072026/11/12 16:57:27 INFO Vacuumed table table=closures42082026/11/12 16:57:27 INFO Vacuumed table table=objects42092026/11/12 16:57:27 INFO Received uploads request method=POST path=/42102026/11/12 16:57:27 INFO Received push request method=POST path=/42112026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=042122026/11/12 16:57:27 WARN Force mode enabled - objects will be deleted immediately without grace period42132026/11/12 16:57:27 INFO Received uploads request method=POST path=/42142026/11/12 16:57:27 INFO Received push request method=POST path=/42152026/11/12 16:57:27 INFO Received uploads request method=POST path=/42162026/11/12 16:57:27 INFO Received complete push request method=POST path=/42172026/11/12 16:57:27 INFO Received complete push request method=POST path=/42182026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=042192026/11/12 16:57:27 INFO Received uploads request method=POST path=/42202026/11/12 16:57:27 INFO Completed push id=30 objects=4 duration=3.543419ms42212026/11/12 16:57:27 INFO Completed push id=26 objects=5 duration=2.491099ms42222026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=042232026/11/12 16:57:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042242026/11/12 16:57:27 WARN Force mode enabled - objects will be deleted immediately without grace period42252026/11/12 16:57:27 INFO Received complete upload request method=POST path=/42262026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=042272026/11/12 16:57:27 INFO Vacuumed table table=pending_closures42282026/11/12 16:57:27 INFO Vacuumed table table=commit_receipts42292026/11/12 16:57:27 INFO Vacuumed table table=pending_objects42302026/11/12 16:57:27 INFO Vacuumed table table=multipart_uploads42312026/11/12 16:57:27 INFO Completed upload id=17 objects=25 duration=3.074379ms42322026/11/12 16:57:27 INFO Vacuumed table table=closures42332026/11/12 16:57:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042342026/11/12 16:57:27 INFO Received uploads request method=POST path=/42352026/11/12 16:57:27 INFO Vacuumed table table=objects42362026/11/12 16:57:27 INFO Vacuumed table table=pending_closures42372026/11/12 16:57:27 INFO Vacuumed table table=commit_receipts42382026/11/12 16:57:27 INFO Received uploads request method=POST path=/42392026/11/12 16:57:27 INFO Vacuumed table table=pending_objects42402026/11/12 16:57:27 INFO Vacuumed table table=multipart_uploads42412026/11/12 16:57:27 INFO Vacuumed table table=closures42422026/11/12 16:57:27 INFO Vacuumed table table=objects42432026/11/12 16:57:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=7 objects-failed-to-delete=042442026/11/12 16:57:27 INFO Received complete upload request method=POST path=/42452026/11/12 16:57:27 INFO Vacuumed table table=pending_closures42462026/11/12 16:57:27 INFO Received push request method=POST path=/42472026/11/12 16:57:27 INFO Received complete upload request method=POST path=/42482026/11/12 16:57:27 INFO Vacuumed table table=commit_receipts42492026/11/12 16:57:27 INFO Received uploads request method=POST path=/42502026/11/12 16:57:27 INFO Vacuumed table table=pending_objects42512026/11/12 16:57:27 INFO Completed upload id=12 objects=26 duration=3.0101ms42522026/11/12 16:57:27 INFO Vacuumed table table=multipart_uploads42532026/11/12 16:57:27 INFO Completed upload id=33 objects=5 duration=2.95647ms42542026/11/12 16:57:27 INFO Vacuumed table table=closures42552026/11/12 16:57:27 INFO Received uploads request method=POST path=/42562026/11/12 16:57:27 INFO Received complete upload request method=POST path=/42572026/11/12 16:57:27 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."42582026/11/12 16:57:27 INFO Vacuumed table table=objects42592026/11/12 16:57:27 INFO Completed upload id=24 objects=14 duration=1.40437ms42602026/11/12 16:57:27 INFO Received push request method=POST path=/42612026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=042622026/11/12 16:57:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042632026/11/12 16:57:27 INFO Vacuumed table table=pending_closures42642026/11/12 16:57:27 INFO Vacuumed table table=commit_receipts42652026/11/12 16:57:27 INFO Vacuumed table table=pending_objects42662026/11/12 16:57:27 INFO Received push request method=POST path=/42672026/11/12 16:57:27 INFO Vacuumed table table=multipart_uploads42682026/11/12 16:57:27 INFO Vacuumed table table=closures42692026/11/12 16:57:27 INFO Vacuumed table table=objects42702026/11/12 16:57:27 INFO Received push request method=POST path=/42712026/11/12 16:57:27 INFO Received complete push request method=POST path=/42722026/11/12 16:57:27 INFO Completed push id=14 objects=3 duration=3.48837ms42732026/11/12 16:57:27 INFO Received complete push request method=POST path=/42742026/11/12 16:57:27 INFO Completed push id=26 objects=14 duration=3.28804ms42752026/11/12 16:57:27 INFO Received complete push request method=POST path=/42762026/11/12 16:57:27 INFO Received push request method=POST path=/42772026/11/12 16:57:27 INFO Received uploads request method=POST path=/42782026/11/12 16:57:27 INFO Completed push id=7 objects=15 duration=3.518109ms42792026/11/12 16:57:27 INFO Received complete push request method=POST path=/42802026/11/12 16:57:27 INFO Completed push id=19 objects=3 duration=1.45458ms42812026/11/12 16:57:27 INFO Received complete upload request method=POST path=/42822026/11/12 16:57:27 INFO Completed upload id=21 objects=2 duration=1.36812ms42832026/11/12 16:57:27 INFO Received complete push request method=POST path=/42842026/11/12 16:57:27 INFO Completed push id=17 objects=22 duration=2.46825ms42852026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=042862026/11/12 16:57:27 INFO Received push request method=POST path=/42872026/11/12 16:57:27 WARN Force mode enabled - objects will be deleted immediately without grace period42882026/11/12 16:57:27 INFO Received complete push request method=POST path=/42892026/11/12 16:57:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042902026/11/12 16:57:27 INFO Vacuumed table table=pending_closures42912026/11/12 16:57:27 INFO Received uploads request method=POST path=/42922026/11/12 16:57:27 INFO Vacuumed table table=commit_receipts42932026/11/12 16:57:27 INFO Vacuumed table table=pending_objects42942026/11/12 16:57:27 INFO Completed push id=21 objects=9 duration=2.14642ms42952026/11/12 16:57:27 INFO Vacuumed table table=multipart_uploads42962026/11/12 16:57:27 INFO Vacuumed table table=closures42972026/11/12 16:57:27 INFO Received push request method=POST path=/42982026/11/12 16:57:27 INFO Vacuumed table table=objects42992026/11/12 16:57:27 INFO Received push request method=POST path=/43002026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=043012026/11/12 16:57:27 WARN Force mode enabled - objects will be deleted immediately without grace period43022026/11/12 16:57:27 INFO Received uploads request method=POST path=/43032026/11/12 16:57:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043042026/11/12 16:57:27 INFO Vacuumed table table=pending_closures43052026/11/12 16:57:27 INFO Received complete upload request method=POST path=/43062026/11/12 16:57:27 INFO Received push request method=POST path=/43072026/11/12 16:57:27 INFO Vacuumed table table=commit_receipts43082026/11/12 16:57:27 INFO Completed upload id=21 objects=25 duration=1.76421ms43092026/11/12 16:57:27 INFO Vacuumed table table=pending_objects43102026/11/12 16:57:27 INFO Vacuumed table table=multipart_uploads43112026/11/12 16:57:27 INFO Received complete push request method=POST path=/43122026/11/12 16:57:27 INFO Vacuumed table table=closures43132026/11/12 16:57:27 INFO Vacuumed table table=objects43142026/11/12 16:57:27 INFO Completed push id=16 objects=26 duration=1.79823ms43152026/11/12 16:57:27 INFO Received uploads request method=POST path=/43162026/11/12 16:57:27 INFO Received complete upload request method=POST path=/43172026/11/12 16:57:27 INFO Completed upload id=14 objects=16 duration=2.328779ms43182026/11/12 16:57:27 INFO Received uploads request method=POST path=/43192026/11/12 16:57:27 INFO Received push request method=POST path=/43202026/11/12 16:57:27 INFO Received push request method=POST path=/43212026/11/12 16:57:27 INFO Received uploads request method=POST path=/43222026/11/12 16:57:27 INFO Received complete push request method=POST path=/43232026/11/12 16:57:27 INFO Received push request method=POST path=/43242026/11/12 16:57:27 INFO Received complete push request method=POST path=/43252026/11/12 16:57:27 INFO Completed push id=32 objects=7 duration=1.47417ms43262026/11/12 16:57:27 INFO Completed push id=16 objects=8 duration=2.27103ms43272026/11/12 16:57:27 INFO Received uploads request method=POST path=/43282026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=043292026/11/12 16:57:27 WARN Force mode enabled - objects will be deleted immediately without grace period43302026/11/12 16:57:27 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043312026/11/12 16:57:27 INFO Vacuumed table table=pending_closures43322026/11/12 16:57:27 INFO Vacuumed table table=commit_receipts43332026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=043342026/11/12 16:57:27 INFO Vacuumed table table=pending_objects43352026/11/12 16:57:27 INFO Vacuumed table table=multipart_uploads43362026/11/12 16:57:27 INFO Vacuumed table table=closures43372026/11/12 16:57:27 INFO Vacuumed table table=objects43382026/11/12 16:57:27 INFO Received push request method=POST path=/43392026/11/12 16:57:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043402026/11/12 16:57:27 INFO Received uploads request method=POST path=/43412026/11/12 16:57:27 INFO Received complete push request method=POST path=/43422026/11/12 16:57:27 INFO Vacuumed table table=pending_closures43432026/11/12 16:57:27 INFO Received uploads request method=POST path=/43442026/11/12 16:57:27 INFO Vacuumed table table=commit_receipts43452026/11/12 16:57:27 INFO Completed push id=22 objects=3 duration=3.314419ms43462026/11/12 16:57:27 INFO Vacuumed table table=pending_objects43472026/11/12 16:57:27 INFO Vacuumed table table=multipart_uploads43482026/11/12 16:57:27 INFO Vacuumed table table=closures43492026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=043502026/11/12 16:57:27 INFO Received complete push request method=POST path=/43512026/11/12 16:57:27 INFO Vacuumed table table=objects43522026/11/12 16:57:27 INFO Completed push id=26 objects=4 duration=1.35898ms43532026/11/12 16:57:27 WARN Force mode enabled - objects will be deleted immediately without grace period43542026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=043552026/11/12 16:57:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043562026/11/12 16:57:27 WARN Force mode enabled - objects will be deleted immediately without grace period43572026/11/12 16:57:27 INFO Vacuumed table table=pending_closures43582026/11/12 16:57:27 INFO Vacuumed table table=commit_receipts43592026/11/12 16:57:27 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043602026/11/12 16:57:27 INFO Vacuumed table table=pending_objects43612026/11/12 16:57:27 INFO Vacuumed table table=multipart_uploads43622026/11/12 16:57:27 INFO Vacuumed table table=pending_closures43632026/11/12 16:57:27 INFO Vacuumed table table=commit_receipts43642026/11/12 16:57:27 INFO Vacuumed table table=closures43652026/11/12 16:57:27 INFO Vacuumed table table=objects43662026/11/12 16:57:27 INFO Vacuumed table table=pending_objects43672026/11/12 16:57:27 INFO Vacuumed table table=multipart_uploads43682026/11/12 16:57:27 INFO Vacuumed table table=closures43692026/11/12 16:57:27 INFO Vacuumed table table=objects43702026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=043712026/11/12 16:57:27 INFO Received push request method=POST path=/43722026/11/12 16:57:27 INFO Received complete push request method=POST path=/43732026/11/12 16:57:27 INFO Received complete push request method=POST path=/43742026/11/12 16:57:27 INFO Completed push id=27 objects=6 duration=3.091029ms43752026/11/12 16:57:27 INFO Received uploads request method=POST path=/43762026/11/12 16:57:27 INFO Received uploads request method=POST path=/43772026/11/12 16:57:27 INFO Completed push id=13 objects=15 duration=2.326119ms43782026/11/12 16:57:27 INFO Received complete upload request method=POST path=/43792026/11/12 16:57:27 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."43802026/11/12 16:57:27 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."43812026/11/12 16:57:27 INFO Completed upload id=28 objects=13 duration=1.45572ms43822026/11/12 16:57:27 INFO Received complete upload request method=POST path=/43832026/11/12 16:57:27 INFO Completed upload id=12 objects=6 duration=1.42909ms43842026/11/12 16:57:27 INFO Received complete push request method=POST path=/43852026/11/12 16:57:27 INFO Received complete push request method=POST path=/43862026/11/12 16:57:27 INFO Completed push id=17 objects=11 duration=1.934069ms43872026/11/12 16:57:27 INFO Completed push id=7 objects=24 duration=4.184879ms43882026/11/12 16:57:27 INFO Received push request method=POST path=/43892026/11/12 16:57:27 INFO Received complete push request method=POST path=/43902026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=043912026/11/12 16:57:27 INFO Received push request method=POST path=/43922026/11/12 16:57:27 INFO Completed push id=11 objects=6 duration=3.27523ms43932026/11/12 16:57:27 INFO Received complete upload request method=POST path=/43942026/11/12 16:57:27 INFO Received complete push request method=POST path=/43952026/11/12 16:57:27 INFO Completed upload id=25 objects=4 duration=1.234369ms43962026/11/12 16:57:27 INFO Completed push id=29 objects=3 duration=1.307409ms43972026/11/12 16:57:27 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."43982026/11/12 16:57:27 INFO Received complete push request method=POST path=/43992026/11/12 16:57:27 INFO Completed push id=17 objects=3 duration=2.068269ms44002026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=044012026/11/12 16:57:27 INFO Received push request method=POST path=/44022026/11/12 16:57:27 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044032026/11/12 16:57:27 INFO Vacuumed table table=pending_closures44042026/11/12 16:57:27 INFO Vacuumed table table=commit_receipts44052026/11/12 16:57:27 INFO Vacuumed table table=pending_objects44062026/11/12 16:57:27 INFO Vacuumed table table=multipart_uploads44072026/11/12 16:57:27 INFO Received push request method=POST path=/44082026/11/12 16:57:27 INFO Vacuumed table table=closures44092026/11/12 16:57:27 INFO Vacuumed table table=objects44102026/11/12 16:57:27 INFO Received uploads request method=POST path=/44112026/11/12 16:57:27 INFO Received complete push request method=POST path=/44122026/11/12 16:57:27 INFO Received push request method=POST path=/44132026/11/12 16:57:27 INFO Completed push id=19 objects=14 duration=2.13351ms44142026/11/12 16:57:27 INFO Received push request method=POST path=/44152026/11/12 16:57:27 INFO Received complete upload request method=POST path=/44162026/11/12 16:57:27 INFO Received push request method=POST path=/44172026/11/12 16:57:27 INFO Completed upload id=15 objects=2 duration=1.68905ms44182026/11/12 16:57:27 INFO Received push request method=POST path=/44192026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=044202026/11/12 16:57:27 INFO Received complete upload request method=POST path=/44212026/11/12 16:57:27 WARN Force mode enabled - objects will be deleted immediately without grace period44222026/11/12 16:57:27 INFO Completed upload id=31 objects=2 duration=1.145269ms44232026/11/12 16:57:27 INFO Received complete push request method=POST path=/44242026/11/12 16:57:27 INFO Completed push id=18 objects=6 duration=1.40115ms44252026/11/12 16:57:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=3 objects-failed-to-delete=044262026/11/12 16:57:27 INFO Received complete push request method=POST path=/44272026/11/12 16:57:27 INFO Received push request method=POST path=/44282026/11/12 16:57:27 INFO Vacuumed table table=pending_closures44292026/11/12 16:57:27 INFO Completed push id=23 objects=5 duration=1.51674ms44302026/11/12 16:57:27 INFO Vacuumed table table=commit_receipts44312026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=044322026/11/12 16:57:27 INFO Vacuumed table table=pending_objects44332026/11/12 16:57:27 INFO Vacuumed table table=multipart_uploads44342026/11/12 16:57:27 INFO Vacuumed table table=closures44352026/11/12 16:57:27 WARN Force mode enabled - objects will be deleted immediately without grace period44362026/11/12 16:57:27 INFO Vacuumed table table=objects44372026/11/12 16:57:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044382026/11/12 16:57:27 INFO Received push request method=POST path=/44392026/11/12 16:57:27 INFO Vacuumed table table=pending_closures44402026/11/12 16:57:27 INFO Vacuumed table table=commit_receipts44412026/11/12 16:57:27 INFO Received uploads request method=POST path=/44422026/11/12 16:57:27 INFO Vacuumed table table=pending_objects44432026/11/12 16:57:27 INFO Vacuumed table table=multipart_uploads44442026/11/12 16:57:27 INFO Received complete push request method=POST path=/44452026/11/12 16:57:27 INFO Vacuumed table table=closures44462026/11/12 16:57:27 INFO Completed push id=24 objects=3 duration=1.486089ms44472026/11/12 16:57:27 INFO Vacuumed table table=objects44482026/11/12 16:57:27 INFO Received uploads request method=POST path=/44492026/11/12 16:57:27 INFO Received push request method=POST path=/44502026/11/12 16:57:27 INFO Received complete push request method=POST path=/44512026/11/12 16:57:27 INFO Received push request method=POST path=/44522026/11/12 16:57:27 INFO Completed push id=20 objects=3 duration=1.86025ms44532026/11/12 16:57:27 INFO Received complete push request method=POST path=/44542026/11/12 16:57:27 INFO Completed push id=33 objects=6 duration=1.156909ms44552026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=044562026/11/12 16:57:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044572026/11/12 16:57:27 INFO Vacuumed table table=pending_closures44582026/11/12 16:57:27 INFO Vacuumed table table=commit_receipts44592026/11/12 16:57:27 INFO Received complete upload request method=POST path=/44602026/11/12 16:57:27 INFO Vacuumed table table=pending_objects44612026/11/12 16:57:27 INFO Completed upload id=24 objects=3 duration=1.29268ms44622026/11/12 16:57:27 INFO Vacuumed table table=multipart_uploads44632026/11/12 16:57:27 INFO Vacuumed table table=closures44642026/11/12 16:57:27 INFO Received complete upload request method=POST path=/44652026/11/12 16:57:27 INFO Vacuumed table table=objects44662026/11/12 16:57:27 INFO Received push request method=POST path=/44672026/11/12 16:57:27 INFO Completed upload id=21 objects=7 duration=2.704369ms44682026/11/12 16:57:27 INFO Received complete upload request method=POST path=/44692026/11/12 16:57:27 INFO Received complete push request method=POST path=/44702026/11/12 16:57:27 INFO Received complete push request method=POST path=/44712026/11/12 16:57:27 INFO Completed upload id=19 objects=2 duration=2.815689ms44722026/11/12 16:57:27 INFO Completed push id=31 objects=14 duration=3.194499ms44732026/11/12 16:57:27 INFO Completed push id=19 objects=14 duration=2.93263ms44742026/11/12 16:57:27 INFO Received uploads request method=POST path=/44752026/11/12 16:57:27 INFO Received complete push request method=POST path=/44762026/11/12 16:57:27 INFO Completed push id=20 objects=8 duration=1.26857ms44772026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=044782026/11/12 16:57:27 WARN Force mode enabled - objects will be deleted immediately without grace period44792026/11/12 16:57:27 INFO Received complete upload request method=POST path=/44802026/11/12 16:57:27 INFO Completed upload id=20 objects=6 duration=1.34027ms44812026/11/12 16:57:27 INFO Received complete upload request method=POST path=/44822026/11/12 16:57:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044832026/11/12 16:57:27 INFO Completed upload id=26 objects=6 duration=1.535949ms44842026/11/12 16:57:27 INFO Vacuumed table table=pending_closures44852026/11/12 16:57:27 INFO Vacuumed table table=commit_receipts44862026/11/12 16:57:27 INFO Received complete push request method=POST path=/44872026/11/12 16:57:27 INFO Vacuumed table table=pending_objects44882026/11/12 16:57:27 INFO Completed push id=25 objects=4 duration=1.326589ms44892026/11/12 16:57:27 INFO Vacuumed table table=multipart_uploads44902026/11/12 16:57:27 INFO Vacuumed table table=closures44912026/11/12 16:57:27 INFO Vacuumed table table=objects44922026/11/12 16:57:27 INFO Received complete upload request method=POST path=/44932026/11/12 16:57:27 INFO Received uploads request method=POST path=/44942026/11/12 16:57:27 INFO Completed upload id=20 objects=3 duration=2.57981ms44952026/11/12 16:57:27 INFO Received push request method=POST path=/44962026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=044972026/11/12 16:57:27 INFO Received uploads request method=POST path=/44982026/11/12 16:57:27 WARN Force mode enabled - objects will be deleted immediately without grace period44992026/11/12 16:57:27 INFO Received uploads request method=POST path=/45002026/11/12 16:57:27 INFO Aborted multipart uploads count=0 kept=045012026/11/12 16:57:27 INFO Received push request method=POST path=/45022026/11/12 16:57:27 INFO Received complete push request method=POST path=/45032026/11/12 16:57:27 WARN Force mode enabled - objects will be deleted immediately without grace period45042026/11/12 16:57:27 INFO Received push request method=POST path=/45052026/11/12 16:57:27 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045062026/11/12 16:57:27 INFO Received uploads request method=POST path=/45072026/11/12 16:57:27 INFO Received push request method=POST path=/45082026/11/12 16:57:27 INFO Received uploads request method=POST path=/45092026/11/12 16:57:27 INFO Received push request method=POST path=/45102026/11/12 16:57:27 INFO Received uploads request method=POST path=/45112026/11/12 16:57:27 INFO Completed push id=33 objects=7 duration=9.579679ms45122026/11/12 16:57:27 INFO Vacuumed table table=pending_closures45132026/11/12 16:57:27 INFO Received complete upload request method=POST path=/45142026/11/12 16:57:27 INFO Vacuumed table table=commit_receipts45152026/11/12 16:57:27 INFO Completed upload id=29 objects=5 duration=1.24465ms45162026/11/12 16:57:27 INFO Vacuumed table table=pending_objects45172026/11/12 16:57:27 INFO Vacuumed table table=multipart_uploads45182026/11/12 16:57:27 INFO Vacuumed table table=closures45192026/11/12 16:57:27 INFO Vacuumed table table=objects45202026/11/12 16:57:27 INFO Received push request method=POST path=/45212026/11/12 16:57:27 INFO Received complete upload request method=POST path=/45222026/11/12 16:57:27 INFO Received push request method=POST path=/45232026/11/12 16:57:27 INFO Completed upload id=26 objects=3 duration=1.20391ms45242026/11/12 16:57:27 INFO Received complete push request method=POST path=/45252026/11/12 16:57:27 INFO Completed push id=13 objects=2 duration=1.3921ms45262026/11/12 16:57:27 INFO Received uploads request method=POST path=/45272026/11/12 16:57:28 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=045282026/11/12 16:57:28 INFO Received complete push request method=POST path=/45292026/11/12 16:57:28 INFO Vacuumed table table=pending_closures45302026/11/12 16:57:28 INFO Aborted multipart uploads count=0 kept=045312026/11/12 16:57:28 INFO Vacuumed table table=commit_receipts45322026/11/12 16:57:28 WARN Force mode enabled - objects will be deleted immediately without grace period45332026/11/12 16:57:28 INFO Completed push id=26 objects=23 duration=1.782149ms45342026/11/12 16:57:28 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045352026/11/12 16:57:28 INFO Vacuumed table table=pending_objects45362026/11/12 16:57:28 INFO Vacuumed table table=multipart_uploads45372026/11/12 16:57:28 INFO Vacuumed table table=pending_closures45382026/11/12 16:57:28 INFO Vacuumed table table=closures45392026/11/12 16:57:28 INFO Vacuumed table table=commit_receipts45402026/11/12 16:57:28 INFO Vacuumed table table=objects45412026/11/12 16:57:28 INFO Vacuumed table table=pending_objects45422026/11/12 16:57:28 INFO Vacuumed table table=multipart_uploads45432026/11/12 16:57:28 INFO Vacuumed table table=closures45442026/11/12 16:57:28 INFO Vacuumed table table=objects45452026/11/12 16:57:28 INFO Received uploads request method=POST path=/45462026/11/12 16:57:28 INFO Received push request method=POST path=/45472026/11/12 16:57:28 INFO Received complete upload request method=POST path=/45482026/11/12 16:57:28 INFO Received complete upload request method=POST path=/45492026/11/12 16:57:28 INFO Received complete push request method=POST path=/45502026/11/12 16:57:28 INFO Aborted multipart uploads count=0 kept=045512026/11/12 16:57:28 INFO Completed upload id=27 objects=13 duration=1.41319ms45522026/11/12 16:57:28 INFO Completed upload id=21 objects=2 duration=1.31552ms45532026/11/12 16:57:28 INFO Received uploads request method=POST path=/45542026/11/12 16:57:28 INFO Received complete upload request method=POST path=/45552026/11/12 16:57:28 INFO Completed push id=27 objects=11 duration=1.353939ms45562026/11/12 16:57:28 WARN Force mode enabled - objects will be deleted immediately without grace period45572026/11/12 16:57:28 INFO Completed upload id=24 objects=7 duration=1.41787ms45582026/11/12 16:57:28 INFO Received complete push request method=POST path=/45592026/11/12 16:57:28 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045602026/11/12 16:57:28 INFO Vacuumed table table=pending_closures45612026/11/12 16:57:28 INFO Vacuumed table table=commit_receipts45622026/11/12 16:57:28 INFO Completed push id=27 objects=25 duration=3.34288ms45632026/11/12 16:57:28 INFO Vacuumed table table=pending_objects45642026/11/12 16:57:28 INFO Vacuumed table table=multipart_uploads45652026/11/12 16:57:28 INFO Received complete push request method=POST path=/45662026/11/12 16:57:28 INFO Vacuumed table table=closures45672026/11/12 16:57:28 INFO Completed push id=7 objects=7 duration=1.12064ms45682026/11/12 16:57:28 INFO Vacuumed table table=objects45692026/11/12 16:57:28 INFO Received push request method=POST path=/45702026/11/12 16:57:28 INFO Received uploads request method=POST path=/45712026/11/12 16:57:28 INFO Aborted multipart uploads count=0 kept=045722026/11/12 16:57:28 INFO Received complete upload request method=POST path=/45732026/11/12 16:57:28 INFO Received uploads request method=POST path=/45742026/11/12 16:57:28 INFO Completed upload id=13 objects=8 duration=1.53753ms45752026/11/12 16:57:28 INFO Received push request method=POST path=/45762026/11/12 16:57:28 INFO Received complete upload request method=POST path=/45772026/11/12 16:57:28 INFO Aborted multipart uploads count=0 kept=045782026/11/12 16:57:28 INFO Completed upload id=29 objects=26 duration=1.320359ms45792026/11/12 16:57:28 INFO Received complete upload request method=POST path=/45802026/11/12 16:57:28 INFO Aborted multipart uploads count=0 kept=045812026/11/12 16:57:28 INFO Completed upload id=26 objects=8 duration=1.80616ms45822026/11/12 16:57:28 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045832026/11/12 16:57:28 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=045842026/11/12 16:57:28 INFO Aborted multipart uploads count=0 kept=045852026/11/12 16:57:28 INFO Vacuumed table table=pending_closures45862026/11/12 16:57:28 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045872026/11/12 16:57:28 INFO Vacuumed table table=commit_receipts45882026/11/12 16:57:28 INFO Vacuumed table table=pending_closures45892026/11/12 16:57:28 INFO Received complete push request method=POST path=/45902026/11/12 16:57:28 INFO Vacuumed table table=pending_objects45912026/11/12 16:57:28 INFO Vacuumed table table=commit_receipts45922026/11/12 16:57:28 INFO Vacuumed table table=pending_closures45932026/11/12 16:57:28 INFO Vacuumed table table=multipart_uploads45942026/11/12 16:57:28 INFO Completed push id=12 objects=6 duration=870.89µs45952026/11/12 16:57:28 INFO Received push request method=POST path=/45962026/11/12 16:57:28 INFO Received complete push request method=POST path=/45972026/11/12 16:57:28 INFO Vacuumed table table=closures45982026/11/12 16:57:28 INFO Vacuumed table table=commit_receipts45992026/11/12 16:57:28 INFO Vacuumed table table=pending_objects46002026/11/12 16:57:28 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046012026/11/12 16:57:28 INFO Vacuumed table table=multipart_uploads46022026/11/12 16:57:28 INFO Vacuumed table table=objects46032026/11/12 16:57:28 INFO Vacuumed table table=pending_objects46042026/11/12 16:57:28 INFO Completed push id=27 objects=15 duration=1.585659ms46052026/11/12 16:57:28 INFO Vacuumed table table=closures46062026/11/12 16:57:28 INFO Vacuumed table table=multipart_uploads46072026/11/12 16:57:28 INFO Received uploads request method=POST path=/46082026/11/12 16:57:28 INFO Vacuumed table table=pending_closures46092026/11/12 16:57:28 INFO Vacuumed table table=closures46102026/11/12 16:57:28 INFO Vacuumed table table=objects46112026/11/12 16:57:28 INFO Vacuumed table table=commit_receipts46122026/11/12 16:57:28 INFO Vacuumed table table=objects46132026/11/12 16:57:28 INFO Vacuumed table table=pending_objects46142026/11/12 16:57:28 INFO Received complete upload request method=POST path=/46152026/11/12 16:57:28 INFO Vacuumed table table=multipart_uploads46162026/11/12 16:57:28 INFO Vacuumed table table=closures46172026/11/12 16:57:28 INFO Completed upload id=27 objects=4 duration=1.44326ms46182026/11/12 16:57:28 INFO Vacuumed table table=objects46192026/11/12 16:57:28 INFO Received uploads request method=POST path=/46202026/11/12 16:57:28 INFO Received complete upload request method=POST path=/46212026/11/12 16:57:28 INFO Aborted multipart uploads count=0 kept=046222026/11/12 16:57:28 INFO Completed upload id=22 objects=6 duration=1.555649ms46232026/11/12 16:57:28 INFO Received complete push request method=POST path=/46242026/11/12 16:57:28 INFO Completed push id=30 objects=8 duration=1.27417ms46252026/11/12 16:57:28 WARN Force mode enabled - objects will be deleted immediately without grace period46262026/11/12 16:57:28 INFO Received complete push request method=POST path=/46272026/11/12 16:57:28 INFO Completed push id=28 objects=14 duration=1.401ms46282026/11/12 16:57:28 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046292026/11/12 16:57:28 INFO Vacuumed table table=pending_closures46302026/11/12 16:57:28 INFO Vacuumed table table=commit_receipts46312026/11/12 16:57:28 INFO Vacuumed table table=pending_objects46322026/11/12 16:57:28 INFO Vacuumed table table=multipart_uploads46332026/11/12 16:57:28 INFO Vacuumed table table=closures46342026/11/12 16:57:28 INFO Vacuumed table table=objects46352026/11/12 16:57:28 INFO Received complete push request method=POST path=/46362026/11/12 16:57:28 INFO Received uploads request method=POST path=/46372026/11/12 16:57:28 INFO Completed push id=9 objects=9 duration=4.06881ms46382026/11/12 16:57:28 INFO Aborted multipart uploads count=0 kept=046392026/11/12 16:57:28 INFO Received uploads request method=POST path=/46402026/11/12 16:57:28 INFO Received complete push request method=POST path=/46412026/11/12 16:57:28 INFO Received complete push request method=POST path=/46422026/11/12 16:57:28 INFO Completed push id=13 objects=11 duration=2.0699ms46432026/11/12 16:57:28 INFO Received uploads request method=POST path=/46442026/11/12 16:57:28 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046452026/11/12 16:57:28 INFO Received uploads request method=POST path=/46462026/11/12 16:57:28 INFO Vacuumed table table=pending_closures46472026/11/12 16:57:28 INFO Completed push id=24 objects=5 duration=3.529939ms46482026/11/12 16:57:28 INFO Vacuumed table table=commit_receipts46492026/11/12 16:57:28 INFO Received complete upload request method=POST path=/46502026/11/12 16:57:28 INFO Completed upload id=31 objects=5 duration=1.08016ms46512026/11/12 16:57:28 INFO Vacuumed table table=pending_objects46522026/11/12 16:57:28 INFO Vacuumed table table=multipart_uploads46532026/11/12 16:57:28 INFO Vacuumed table table=closures46542026/11/12 16:57:28 INFO Received push request method=POST path=/46552026/11/12 16:57:28 INFO Vacuumed table table=objects46562026/11/12 16:57:28 INFO Received uploads request method=POST path=/46572026/11/12 16:57:28 INFO Received uploads request method=POST path=/46582026/11/12 16:57:28 INFO Received push request method=POST path=/46592026/11/12 16:57:28 INFO Received complete upload request method=POST path=/46602026/11/12 16:57:28 INFO Received complete push request method=POST path=/46612026/11/12 16:57:28 INFO Received uploads request method=POST path=/46622026/11/12 16:57:28 INFO Completed upload id=21 objects=8 duration=1.75985ms46632026/11/12 16:57:28 INFO Completed push id=13 objects=22 duration=1.886259ms46642026/11/12 16:57:28 INFO Received push request method=POST path=/46652026/11/12 16:57:28 INFO Aborted multipart uploads count=0 kept=046662026/11/12 16:57:28 WARN Force mode enabled - objects will be deleted immediately without grace period46672026/11/12 16:57:28 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046682026/11/12 16:57:28 INFO Vacuumed table table=pending_closures46692026/11/12 16:57:28 INFO Received complete push request method=POST path=/46702026/11/12 16:57:28 INFO Received complete push request method=POST path=/46712026/11/12 16:57:28 INFO Vacuumed table table=commit_receipts46722026/11/12 16:57:28 INFO Completed push id=32 objects=3 duration=1.37104ms46732026/11/12 16:57:28 INFO Completed push id=28 objects=19 duration=1.87021ms46742026/11/12 16:57:28 INFO Vacuumed table table=pending_objects46752026/11/12 16:57:28 INFO Vacuumed table table=multipart_uploads46762026/11/12 16:57:28 INFO Vacuumed table table=closures46772026/11/12 16:57:28 INFO Received uploads request method=POST path=/46782026/11/12 16:57:28 INFO Vacuumed table table=objects46792026/11/12 16:57:28 INFO Received complete upload request method=POST path=/46802026/11/12 16:57:28 INFO Completed upload id=26 objects=2 duration=2.15559ms46812026/11/12 16:57:28 INFO Received uploads request method=POST path=/46822026/11/12 16:57:28 INFO Received complete push request method=POST path=/46832026/11/12 16:57:28 INFO Completed push id=22 objects=6 duration=1.4522ms46842026/11/12 16:57:28 INFO Received push request method=POST path=/46852026/11/12 16:57:28 INFO Received complete push request method=POST path=/46862026/11/12 16:57:28 INFO Aborted multipart uploads count=0 kept=046872026/11/12 16:57:28 INFO Completed push id=21 objects=6 duration=1.84111ms46882026/11/12 16:57:28 WARN Force mode enabled - objects will be deleted immediately without grace period46892026/11/12 16:57:28 INFO Received uploads request method=POST path=/46902026/11/12 16:57:28 INFO Received uploads request method=POST path=/46912026/11/12 16:57:28 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046922026/11/12 16:57:28 INFO Vacuumed table table=pending_closures46932026/11/12 16:57:28 INFO Aborted multipart uploads count=0 kept=046942026/11/12 16:57:28 INFO Vacuumed table table=commit_receipts46952026/11/12 16:57:28 INFO Vacuumed table table=pending_objects46962026/11/12 16:57:28 INFO Vacuumed table table=multipart_uploads46972026/11/12 16:57:28 INFO Vacuumed table table=closures46982026/11/12 16:57:28 WARN Force mode enabled - objects will be deleted immediately without grace period46992026/11/12 16:57:28 INFO Vacuumed table table=objects47002026/11/12 16:57:28 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047012026/11/12 16:57:28 INFO Aborted multipart uploads count=0 kept=047022026/11/12 16:57:28 INFO Aborted multipart uploads count=0 kept=047032026/11/12 16:57:28 INFO Vacuumed table table=pending_closures47042026/11/12 16:57:28 INFO Vacuumed table table=commit_receipts47052026/11/12 16:57:28 INFO Received complete push request method=POST path=/47062026/11/12 16:57:28 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047072026/11/12 16:57:28 INFO Received complete upload request method=POST path=/47082026/11/12 16:57:28 WARN Force mode enabled - objects will be deleted immediately without grace period47092026/11/12 16:57:28 INFO Vacuumed table table=pending_objects47102026/11/12 16:57:28 INFO Received complete push request method=POST path=/47112026/11/12 16:57:28 INFO Received push request method=POST path=/47122026/11/12 16:57:28 INFO Vacuumed table table=multipart_uploads47132026/11/12 16:57:28 INFO Completed push id=24 objects=8 duration=2.14856ms47142026/11/12 16:57:28 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047152026/11/12 16:57:28 INFO Vacuumed table table=closures47162026/11/12 16:57:28 INFO Vacuumed table table=pending_closures47172026/11/12 16:57:28 INFO Received complete push request method=POST path=/47182026/11/12 16:57:28 INFO Completed upload id=31 objects=3 duration=2.791159ms47192026/11/12 16:57:28 INFO Vacuumed table table=pending_closures47202026/11/12 16:57:28 INFO Vacuumed table table=objects47212026/11/12 16:57:28 INFO Vacuumed table table=commit_receipts47222026/11/12 16:57:28 INFO Completed push id=29 objects=10 duration=1.50699ms47232026/11/12 16:57:28 INFO Vacuumed table table=commit_receipts47242026/11/12 16:57:28 INFO Received complete upload request method=POST path=/47252026/11/12 16:57:28 INFO Completed push id=23 objects=28 duration=3.979839ms47262026/11/12 16:57:28 INFO Received uploads request method=POST path=/47272026/11/12 16:57:28 INFO Vacuumed table table=pending_objects47282026/11/12 16:57:28 INFO Completed upload id=22 objects=7 duration=1.16749ms47292026/11/12 16:57:28 INFO Vacuumed table table=pending_objects47302026/11/12 16:57:28 INFO Vacuumed table table=multipart_uploads47312026/11/12 16:57:28 INFO Vacuumed table table=multipart_uploads47322026/11/12 16:57:28 INFO Vacuumed table table=closures47332026/11/12 16:57:28 INFO Vacuumed table table=closures47342026/11/12 16:57:28 INFO Vacuumed table table=objects47352026/11/12 16:57:28 INFO Vacuumed table table=objects47362026/11/12 16:57:28 INFO Received push request method=POST path=/47372026/11/12 16:57:28 INFO Received complete push request method=POST path=/47382026/11/12 16:57:28 INFO Received complete upload request method=POST path=/47392026/11/12 16:57:28 INFO Completed push id=5 objects=28 duration=2.300639ms47402026/11/12 16:57:28 INFO Completed upload id=25 objects=9 duration=1.24231ms47412026/11/12 16:57:28 INFO Received complete upload request method=POST path=/47422026/11/12 16:57:28 INFO Completed upload id=22 objects=2 duration=1.42013ms47432026/11/12 16:57:28 INFO Received complete push request method=POST path=/47442026/11/12 16:57:28 INFO Received complete upload request method=POST path=/47452026/11/12 16:57:28 INFO Received complete push request method=POST path=/47462026/11/12 16:57:28 INFO Completed upload id=13 objects=14 duration=3.042119ms47472026/11/12 16:57:28 INFO Completed push id=24 objects=24 duration=3.657299ms47482026/11/12 16:57:28 INFO Completed push id=21 objects=26 duration=1.73355ms47492026/11/12 16:57:28 INFO Received complete push request method=POST path=/47502026/11/12 16:57:28 INFO Completed push id=24 objects=4 duration=963.38µs47512026/11/12 16:57:28 INFO Received uploads request method=POST path=/47522026/11/12 16:57:28 INFO Received uploads request method=POST path=/47532026/11/12 16:57:28 INFO Received push request method=POST path=/47542026/11/12 16:57:28 INFO Received complete upload request method=POST path=/47552026/11/12 16:57:28 INFO Received uploads request method=POST path=/47562026/11/12 16:57:28 INFO Completed upload id=30 objects=6 duration=1.27275ms47572026/11/12 16:57:28 INFO Received complete upload request method=POST path=/47582026/11/12 16:57:28 INFO Completed upload id=27 objects=3 duration=1.23601ms47592026/11/12 16:57:28 INFO Received complete push request method=POST path=/47602026/11/12 16:57:28 INFO Completed push id=10 objects=2 duration=1.83119ms47612026/11/12 16:57:28 INFO Aborted multipart uploads count=0 kept=047622026/11/12 16:57:28 INFO Received push request method=POST path=/47632026/11/12 16:57:28 INFO Aborted multipart uploads count=0 kept=047642026/11/12 16:57:28 INFO Aborted multipart uploads count=0 kept=047652026/11/12 16:57:28 WARN Force mode enabled - objects will be deleted immediately without grace period47662026/11/12 16:57:28 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047672026/11/12 16:57:28 INFO Received complete upload request method=POST path=/47682026/11/12 16:57:28 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047692026/11/12 16:57:28 WARN Force mode enabled - objects will be deleted immediately without grace period47702026/11/12 16:57:28 INFO Received complete push request method=POST path=/47712026/11/12 16:57:28 INFO Vacuumed table table=pending_closures47722026/11/12 16:57:28 INFO Completed upload id=27 objects=8 duration=1.52725ms47732026/11/12 16:57:28 INFO Vacuumed table table=pending_closures47742026/11/12 16:57:28 INFO Completed push id=25 objects=9 duration=1.16803ms47752026/11/12 16:57:28 INFO Received complete upload request method=POST path=/47762026/11/12 16:57:28 INFO Vacuumed table table=commit_receipts47772026/11/12 16:57:28 INFO Vacuumed table table=commit_receipts47782026/11/12 16:57:28 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047792026/11/12 16:57:28 INFO Received complete upload request method=POST path=/47802026/11/12 16:57:28 INFO Vacuumed table table=pending_objects47812026/11/12 16:57:28 INFO Received uploads request method=POST path=/47822026/11/12 16:57:28 INFO Vacuumed table table=multipart_uploads47832026/11/12 16:57:28 INFO Vacuumed table table=pending_closures47842026/11/12 16:57:28 INFO Completed upload id=29 objects=4 duration=2.671539ms47852026/11/12 16:57:28 INFO Vacuumed table table=pending_objects47862026/11/12 16:57:28 INFO Vacuumed table table=closures47872026/11/12 16:57:28 INFO Vacuumed table table=multipart_uploads47882026/11/12 16:57:28 INFO Completed upload id=21 objects=9 duration=2.39027ms47892026/11/12 16:57:28 INFO Received complete upload request method=POST path=/47902026/11/12 16:57:28 INFO Vacuumed table table=commit_receipts47912026/11/12 16:57:28 INFO Received complete upload request method=POST path=/47922026/11/12 16:57:28 INFO Vacuumed table table=objects47932026/11/12 16:57:28 INFO Vacuumed table table=closures47942026/11/12 16:57:28 INFO Completed upload id=31 objects=5 duration=1.05974ms47952026/11/12 16:57:28 INFO Completed upload id=28 objects=17 duration=1.69618ms47962026/11/12 16:57:28 INFO Received complete upload request method=POST path=/47972026/11/12 16:57:28 INFO Vacuumed table table=objects47982026/11/12 16:57:28 INFO Vacuumed table table=pending_objects47992026/11/12 16:57:28 INFO Completed upload id=28 objects=6 duration=1.07083ms48002026/11/12 16:57:28 INFO Vacuumed table table=multipart_uploads48012026/11/12 16:57:28 INFO Vacuumed table table=closures48022026/11/12 16:57:28 INFO Vacuumed table table=objects48032026/11/12 16:57:28 INFO Received complete push request method=POST path=/48042026/11/12 16:57:28 INFO Completed push id=16 objects=14 duration=2.89315ms48052026/11/12 16:57:28 INFO Received complete upload request method=POST path=/48062026/11/12 16:57:28 INFO Completed upload id=17 objects=5 duration=1.33913ms48072026/11/12 16:57:28 INFO Received push request method=POST path=/48082026/11/12 16:57:28 INFO Received complete upload request method=POST path=/48092026/11/12 16:57:28 INFO Completed upload id=26 objects=3 duration=793.34µs48102026/11/12 16:57:28 INFO Received uploads request method=POST path=/48112026/11/12 16:57:28 INFO Received complete push request method=POST path=/48122026/11/12 16:57:28 INFO Received complete push request method=POST path=/48132026/11/12 16:57:28 INFO Received complete push request method=POST path=/48142026/11/12 16:57:28 INFO Received uploads request method=POST path=/48152026/11/12 16:57:28 INFO Received complete upload request method=POST path=/48162026/11/12 16:57:28 INFO Received complete upload request method=POST path=/48172026/11/12 16:57:28 INFO Completed push id=25 objects=10 duration=2.154639ms48182026/11/12 16:57:28 INFO Completed push id=30 objects=4 duration=1.239549ms48192026/11/12 16:57:28 INFO Completed push id=29 objects=2 duration=1.526849ms48202026/11/12 16:57:28 INFO Completed upload id=34 objects=3 duration=1.07996ms48212026/11/12 16:57:28 INFO Received uploads request method=POST path=/48222026/11/12 16:57:28 INFO Completed upload id=18 objects=2 duration=2.889739ms48232026/11/12 16:57:28 INFO Received complete push request method=POST path=/48242026/11/12 16:57:28 INFO Aborted multipart uploads count=0 kept=048252026/11/12 16:57:28 INFO Completed push id=31 objects=6 duration=3.418839ms48262026/11/12 16:57:28 INFO Aborted multipart uploads count=0 kept=048272026/11/12 16:57:28 WARN Force mode enabled - objects will be deleted immediately without grace period48282026/11/12 16:57:28 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048292026/11/12 16:57:28 INFO Vacuumed table table=pending_closures48302026/11/12 16:57:28 INFO Vacuumed table table=commit_receipts48312026/11/12 16:57:28 INFO Vacuumed table table=pending_objects48322026/11/12 16:57:28 INFO Vacuumed table table=multipart_uploads48332026/11/12 16:57:28 INFO Aborted multipart uploads count=0 kept=048342026/11/12 16:57:28 INFO Vacuumed table table=closures48352026/11/12 16:57:28 INFO Received complete push request method=POST path=/48362026/11/12 16:57:28 INFO Vacuumed table table=objects48372026/11/12 16:57:28 INFO Received complete push request method=POST path=/48382026/11/12 16:57:28 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."48392026/11/12 16:57:28 INFO Received uploads request method=POST path=/48402026/11/12 16:57:28 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."48412026/11/12 16:57:28 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."48422026/11/12 16:57:28 INFO Completed push id=19 objects=3 duration=1.9362ms48432026/11/12 16:57:28 INFO Completed push id=31 objects=7 duration=1.60565ms48442026/11/12 16:57:28 INFO Received complete upload request method=POST path=/48452026/11/12 16:57:28 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048462026/11/12 16:57:28 INFO Received complete upload request method=POST path=/48472026/11/12 16:57:28 INFO Completed upload id=27 objects=7 duration=1.0448ms48482026/11/12 16:57:28 INFO Vacuumed table table=pending_closures48492026/11/12 16:57:28 INFO Completed upload id=29 objects=5 duration=1.0875ms48502026/11/12 16:57:28 INFO Vacuumed table table=commit_receipts48512026/11/12 16:57:28 INFO Received complete push request method=POST path=/48522026/11/12 16:57:28 INFO Completed push id=31 objects=6 duration=1.12758ms48532026/11/12 16:57:28 INFO Aborted multipart uploads count=0 kept=048542026/11/12 16:57:28 INFO Vacuumed table table=pending_objects48552026/11/12 16:57:28 INFO Received uploads request method=POST path=/48562026/11/12 16:57:28 INFO Vacuumed table table=multipart_uploads48572026/11/12 16:57:28 INFO Received uploads request method=POST path=/48582026/11/12 16:57:28 WARN Force mode enabled - objects will be deleted immediately without grace period48592026/11/12 16:57:28 INFO Received push request method=POST path=/48602026/11/12 16:57:28 INFO Vacuumed table table=closures48612026/11/12 16:57:28 INFO Vacuumed table table=objects48622026/11/12 16:57:28 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048632026/11/12 16:57:28 INFO Received uploads request method=POST path=/48642026/11/12 16:57:28 INFO Vacuumed table table=pending_closures48652026/11/12 16:57:28 INFO Vacuumed table table=commit_receipts48662026/11/12 16:57:28 INFO Vacuumed table table=pending_objects48672026/11/12 16:57:28 INFO Vacuumed table table=multipart_uploads48682026/11/12 16:57:28 INFO Vacuumed table table=closures48692026/11/12 16:57:28 INFO Vacuumed table table=objects48702026/11/12 16:57:28 INFO Received complete upload request method=POST path=/48712026/11/12 16:57:28 INFO Completed upload id=32 objects=5 duration=945.77µs48722026/11/12 16:57:28 INFO Received push request method=POST path=/48732026/11/12 16:57:28 INFO Aborted multipart uploads count=0 kept=048742026/11/12 16:57:28 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048752026/11/12 16:57:28 INFO Vacuumed table table=pending_closures48762026/11/12 16:57:28 INFO Vacuumed table table=commit_receipts48772026/11/12 16:57:28 INFO Received complete upload request method=POST path=/48782026/11/12 16:57:28 INFO Vacuumed table table=pending_objects48792026/11/12 16:57:28 INFO Received complete upload request method=POST path=/48802026/11/12 16:57:28 INFO Vacuumed table table=multipart_uploads48812026/11/12 16:57:28 INFO Completed upload id=28 objects=3 duration=905.179µs48822026/11/12 16:57:28 INFO Received complete push request method=POST path=/48832026/11/12 16:57:28 INFO Vacuumed table table=closures48842026/11/12 16:57:28 INFO Completed upload id=30 objects=2 duration=1.10447ms48852026/11/12 16:57:28 INFO Completed push id=32 objects=9 duration=1.40972ms48862026/11/12 16:57:28 INFO Vacuumed table table=objects48872026/11/12 16:57:28 INFO Received uploads request method=POST path=/48882026/11/12 16:57:28 INFO Received complete push request method=POST path=/48892026/11/12 16:57:28 INFO Aborted multipart uploads count=0 kept=048902026/11/12 16:57:28 INFO Completed push id=11 objects=6 duration=2.74594ms48912026/11/12 16:57:28 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048922026/11/12 16:57:28 INFO Vacuumed table table=pending_closures48932026/11/12 16:57:28 INFO Vacuumed table table=commit_receipts48942026/11/12 16:57:28 INFO Received complete push request method=POST path=/48952026/11/12 16:57:28 INFO Vacuumed table table=pending_objects48962026/11/12 16:57:28 INFO Vacuumed table table=multipart_uploads48972026/11/12 16:57:28 INFO Completed push id=20 objects=4 duration=1.52695ms48982026/11/12 16:57:28 INFO Vacuumed table table=closures48992026/11/12 16:57:28 INFO Vacuumed table table=objects49002026/11/12 16:57:28 INFO Received complete upload request method=POST path=/49012026/11/12 16:57:28 INFO Completed upload id=13 objects=7 duration=1.384149ms49022026/11/12 16:57:28 INFO Received push request method=POST path=/49032026/11/12 16:57:28 INFO Received complete push request method=POST path=/49042026/11/12 16:57:28 INFO Received complete push request method=POST path=/49052026/11/12 16:57:28 INFO Received uploads request method=POST path=/49062026/11/12 16:57:28 INFO Completed push id=29 objects=23 duration=1.3582ms49072026/11/12 16:57:28 INFO Completed push id=31 objects=20 duration=1.57374ms49082026/11/12 16:57:28 INFO Received push request method=POST path=/49092026/11/12 16:57:28 INFO Received complete upload request method=POST path=/49102026/11/12 16:57:28 INFO Completed upload id=33 objects=9 duration=1.03244ms49112026/11/12 16:57:28 INFO Received push request method=POST path=/49122026/11/12 16:57:28 INFO Received complete push request method=POST path=/49132026/11/12 16:57:28 INFO Received complete upload request method=POST path=/49142026/11/12 16:57:28 INFO Completed push id=20 objects=13 duration=1.62612ms49152026/11/12 16:57:28 INFO Completed upload id=25 objects=16 duration=2.6338ms49162026/11/12 16:57:28 INFO Received complete upload request method=POST path=/49172026/11/12 16:57:28 INFO Completed upload id=20 objects=4 duration=1.24124ms49182026/11/12 16:57:28 INFO Received complete upload request method=POST path=/49192026/11/12 16:57:28 INFO Received complete push request method=POST path=/49202026/11/12 16:57:28 INFO Completed upload id=22 objects=8 duration=1.18153ms49212026/11/12 16:57:28 INFO Completed push id=30 objects=6 duration=1.01935ms49222026/11/12 16:57:28 INFO Received push request method=POST path=/49232026/11/12 16:57:28 INFO Received uploads request method=POST path=/49242026/11/12 16:57:28 INFO Received complete upload request method=POST path=/49252026/11/12 16:57:28 INFO Received uploads request method=POST path=/49262026/11/12 16:57:28 INFO Completed upload id=32 objects=5 duration=1.962ms49272026/11/12 16:57:28 INFO Received complete push request method=POST path=/49282026/11/12 16:57:28 INFO Received complete upload request method=POST path=/49292026/11/12 16:57:28 INFO Aborted multipart uploads count=0 kept=049302026/11/12 16:57:28 INFO Completed upload id=34 objects=24 duration=1.416819ms49312026/11/12 16:57:28 WARN Force mode enabled - objects will be deleted immediately without grace period49322026/11/12 16:57:28 INFO Completed push id=28 objects=11 duration=3.301579ms49332026/11/12 16:57:28 INFO Received complete push request method=POST path=/49342026/11/12 16:57:28 INFO Completed push id=22 objects=4 duration=1.40521ms49352026/11/12 16:57:28 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049362026/11/12 16:57:28 INFO Vacuumed table table=pending_closures49372026/11/12 16:57:28 INFO Received complete upload request method=POST path=/49382026/11/12 16:57:28 INFO Aborted multipart uploads count=0 kept=049392026/11/12 16:57:28 INFO Vacuumed table table=commit_receipts49402026/11/12 16:57:28 WARN Force mode enabled - objects will be deleted immediately without grace period49412026/11/12 16:57:28 INFO Completed upload id=23 objects=7 duration=1.40443ms49422026/11/12 16:57:28 INFO Vacuumed table table=pending_objects49432026/11/12 16:57:28 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049442026/11/12 16:57:28 INFO Vacuumed table table=multipart_uploads49452026/11/12 16:57:28 INFO Vacuumed table table=closures49462026/11/12 16:57:28 INFO Vacuumed table table=pending_closures49472026/11/12 16:57:28 INFO Vacuumed table table=objects49482026/11/12 16:57:28 INFO Vacuumed table table=commit_receipts49492026/11/12 16:57:28 INFO Received complete push request method=POST path=/49502026/11/12 16:57:28 INFO Received complete push request method=POST path=/49512026/11/12 16:57:28 INFO Vacuumed table table=pending_objects49522026/11/12 16:57:28 INFO Vacuumed table table=multipart_uploads49532026/11/12 16:57:28 INFO Completed push id=23 objects=5 duration=1.14269ms49542026/11/12 16:57:28 INFO Vacuumed table table=closures49552026/11/12 16:57:28 INFO Completed push id=1 objects=9 duration=1.69191ms49562026/11/12 16:57:28 INFO Vacuumed table table=objects49572026/11/12 16:57:28 INFO Received complete upload request method=POST path=/49582026/11/12 16:57:28 INFO Received complete upload request method=POST path=/49592026/11/12 16:57:28 INFO Received complete push request method=POST path=/49602026/11/12 16:57:28 INFO Completed upload id=33 objects=3 duration=878.78µs49612026/11/12 16:57:28 INFO Aborted multipart uploads count=0 kept=049622026/11/12 16:57:28 INFO Completed upload id=25 objects=6 duration=1.75908ms49632026/11/12 16:57:28 INFO Completed push id=33 objects=3 duration=1.02187ms49642026/11/12 16:57:28 WARN Force mode enabled - objects will be deleted immediately without grace period49652026/11/12 16:57:28 INFO Aborted multipart uploads count=0 kept=049662026/11/12 16:57:28 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049672026/11/12 16:57:28 WARN Force mode enabled - objects will be deleted immediately without grace period49682026/11/12 16:57:28 INFO Vacuumed table table=pending_closures49692026/11/12 16:57:28 INFO Vacuumed table table=commit_receipts49702026/11/12 16:57:28 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049712026/11/12 16:57:28 INFO Vacuumed table table=pending_objects49722026/11/12 16:57:28 INFO Vacuumed table table=multipart_uploads49732026/11/12 16:57:28 INFO Vacuumed table table=pending_closures49742026/11/12 16:57:28 INFO Vacuumed table table=closures49752026/11/12 16:57:28 INFO Vacuumed table table=commit_receipts49762026/11/12 16:57:28 INFO Vacuumed table table=objects49772026/11/12 16:57:28 INFO Received complete upload request method=POST path=/49782026/11/12 16:57:28 INFO Vacuumed table table=pending_objects49792026/11/12 16:57:28 INFO Vacuumed table table=multipart_uploads49802026/11/12 16:57:28 INFO Completed upload id=23 objects=7 duration=1.60539ms49812026/11/12 16:57:28 INFO Vacuumed table table=closures49822026/11/12 16:57:28 INFO Vacuumed table table=objects49832026/11/12 16:57:28 INFO Received complete upload request method=POST path=/49842026/11/12 16:57:28 INFO Completed upload id=24 objects=8 duration=1.345369ms49852026/11/12 16:57:28 INFO Received push request method=POST path=/49862026/11/12 16:57:28 INFO Received complete push request method=POST path=/49872026/11/12 16:57:28 INFO Completed push id=34 objects=6 duration=1.096939ms49882026/11/12 16:57:28 INFO Aborted multipart uploads count=0 kept=049892026/11/12 16:57:28 WARN Force mode enabled - objects will be deleted immediately without grace period49902026/11/12 16:57:28 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049912026/11/12 16:57:28 INFO Vacuumed table table=pending_closures49922026/11/12 16:57:28 INFO Received complete push request method=POST path=/49932026/11/12 16:57:28 INFO Vacuumed table table=commit_receipts49942026/11/12 16:57:28 INFO Completed push id=29 objects=24 duration=1.88725ms49952026/11/12 16:57:28 INFO Vacuumed table table=pending_objects49962026/11/12 16:57:28 INFO Vacuumed table table=multipart_uploads49972026/11/12 16:57:28 INFO Received complete push request method=POST path=/49982026/11/12 16:57:28 INFO Vacuumed table table=closures49992026/11/12 16:57:28 INFO Completed push id=24 objects=5 duration=1.43706ms50002026/11/12 16:57:28 INFO Vacuumed table table=objects50012026/11/12 16:57:28 INFO Received complete upload request method=POST path=/50022026/11/12 16:57:28 INFO Received complete upload request method=POST path=/50032026/11/12 16:57:28 INFO Completed upload id=28 objects=5 duration=1.099449ms50042026/11/12 16:57:28 INFO Received complete push request method=POST path=/50052026/11/12 16:57:28 INFO Received push request method=POST path=/50062026/11/12 16:57:28 INFO Completed upload id=24 objects=18 duration=1.51842ms50072026/11/12 16:57:28 INFO Received push request method=POST path=/50082026/11/12 16:57:28 INFO Completed push id=16 objects=7 duration=2.24126ms50092026/11/12 16:57:28 INFO Received complete push request method=POST path=/50102026/11/12 16:57:28 INFO Completed push id=35 objects=8 duration=1.058699ms50112026/11/12 16:57:28 INFO Received push request method=POST path=/50122026/11/12 16:57:28 INFO Aborted multipart uploads count=0 kept=050132026/11/12 16:57:28 WARN Force mode enabled - objects will be deleted immediately without grace period50142026/11/12 16:57:28 INFO Received complete upload request method=POST path=/50152026/11/12 16:57:28 INFO Completed upload id=25 objects=2 duration=1.06233ms50162026/11/12 16:57:28 INFO Received complete upload request method=POST path=/50172026/11/12 16:57:28 INFO Completed upload id=2 objects=7 duration=1.358169ms50182026/11/12 16:57:28 INFO Received complete push request method=POST path=/50192026/11/12 16:57:28 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=050202026/11/12 16:57:28 INFO Received uploads request method=POST path=/50212026/11/12 16:57:28 INFO Completed push id=25 objects=2 duration=2.31371ms50222026/11/12 16:57:28 INFO Vacuumed table table=pending_closures50232026/11/12 16:57:28 INFO Received complete upload request method=POST path=/50242026/11/12 16:57:28 INFO Vacuumed table table=commit_receipts50252026/11/12 16:57:28 INFO Completed upload id=36 objects=9 duration=1.16327ms50262026/11/12 16:57:28 INFO Received complete upload request method=POST path=/50272026/11/12 16:57:28 INFO Vacuumed table table=pending_objects50282026/11/12 16:57:28 INFO Vacuumed table table=multipart_uploads50292026/11/12 16:57:28 INFO Completed upload id=30 objects=5 duration=1.44177ms50302026/11/12 16:57:28 INFO Vacuumed table table=closures50312026/11/12 16:57:28 INFO Vacuumed table table=objects50322026/11/12 16:57:28 INFO Received complete push request method=POST path=/50332026/11/12 16:57:28 INFO Completed push id=23 objects=6 duration=1.57195ms50342026/11/12 16:57:28 INFO Received complete upload request method=POST path=/50352026/11/12 16:57:28 INFO Completed upload id=26 objects=20 duration=1.24673ms50362026/11/12 16:57:28 INFO Received complete upload request method=POST path=/50372026/11/12 16:57:28 INFO Received complete push request method=POST path=/50382026/11/12 16:57:28 INFO Completed upload id=23 objects=7 duration=2.958319ms50392026/11/12 16:57:28 INFO Completed push id=27 objects=4 duration=1.583489ms50402026/11/12 16:57:28 INFO Received complete push request method=POST path=/50412026/11/12 16:57:28 INFO Completed push id=11 objects=9 duration=1.63038ms50422026/11/12 16:57:28 INFO Aborted multipart uploads count=0 kept=050432026/11/12 16:57:28 WARN Force mode enabled - objects will be deleted immediately without grace period50442026/11/12 16:57:28 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=050452026/11/12 16:57:28 INFO Vacuumed table table=pending_closures50462026/11/12 16:57:28 INFO Received complete push request method=POST path=/50472026/11/12 16:57:28 INFO Vacuumed table table=commit_receipts50482026/11/12 16:57:28 INFO Received complete push request method=POST path=/50492026/11/12 16:57:28 INFO Completed push id=24 objects=6 duration=1.50769ms50502026/11/12 16:57:28 INFO Vacuumed table table=pending_objects50512026/11/12 16:57:28 INFO Vacuumed table table=multipart_uploads50522026/11/12 16:57:28 INFO Completed push id=31 objects=3 duration=1.27858ms50532026/11/12 16:57:28 INFO Vacuumed table table=closures50542026/11/12 16:57:28 INFO Vacuumed table table=objects50552026/11/12 16:57:28 INFO Received complete upload request method=POST path=/50562026/11/12 16:57:28 INFO Completed upload id=26 objects=5 duration=1.07868ms50572026/11/12 16:57:28 INFO Received complete push request method=POST path=/50582026/11/12 16:57:28 INFO Received complete upload request method=POST path=/50592026/11/12 16:57:28 INFO Completed push id=28 objects=7 duration=1.324319ms50602026/11/12 16:57:28 INFO Received uploads request method=POST path=/50612026/11/12 16:57:28 INFO Completed upload id=8 objects=6 duration=1.30558ms50622026/11/12 16:57:28 INFO Received complete push request method=POST path=/50632026/11/12 16:57:28 INFO Completed push id=15 objects=24 duration=1.57135ms50642026/11/12 16:57:28 INFO Received complete push request method=POST path=/50652026/11/12 16:57:28 INFO Completed push id=28 objects=26 duration=1.68182ms50662026/11/12 16:57:28 INFO Received complete push request method=POST path=/50672026/11/12 16:57:28 INFO Completed push id=27 objects=3 duration=2.107619ms50682026/11/12 16:57:28 INFO Received complete push request method=POST path=/50692026/11/12 16:57:28 INFO Completed push id=25 objects=7 duration=1.48772ms50702026/11/12 16:57:28 INFO Received complete upload request method=POST path=/50712026/11/12 16:57:28 INFO Received complete upload request method=POST path=/50722026/11/12 16:57:28 INFO Received complete push request method=POST path=/50732026/11/12 16:57:28 INFO Received complete upload request method=POST path=/50742026/11/12 16:57:28 INFO Completed upload id=22 objects=5 duration=2.12059ms50752026/11/12 16:57:28 INFO Completed upload id=19 objects=2 duration=1.18616ms50762026/11/12 16:57:28 INFO Completed upload id=30 objects=6 duration=1.2099ms50772026/11/12 16:57:28 INFO Completed push id=32 objects=6 duration=1.83617ms50782026/11/12 16:57:28 INFO Received complete push request method=POST path=/50792026/11/12 16:57:28 INFO Completed push id=30 objects=3 duration=1.15933ms50802026/11/12 16:57:28 INFO Received complete upload request method=POST path=/50812026/11/12 16:57:28 INFO Completed upload id=28 objects=5 duration=987.1µs50822026/11/12 16:57:28 INFO Received complete push request method=POST path=/50832026/11/12 16:57:28 INFO Received complete upload request method=POST path=/50842026/11/12 16:57:28 INFO Completed push id=28 objects=3 duration=1.91068ms50852026/11/12 16:57:28 INFO Completed upload id=24 objects=7 duration=1.161309ms50862026/11/12 16:57:28 INFO Received complete push request method=POST path=/50872026/11/12 16:57:28 INFO Received complete upload request method=POST path=/50882026/11/12 16:57:28 INFO Completed push id=31 objects=5 duration=1.20548ms50892026/11/12 16:57:28 INFO Completed upload id=33 objects=7 duration=1.12504ms50902026/11/12 16:57:28 INFO Received complete push request method=POST path=/50912026/11/12 16:57:28 INFO Completed push id=32 objects=19 duration=1.186099ms50922026/11/12 16:57:28 INFO Received complete push request method=POST path=/50932026/11/12 16:57:28 INFO Received complete upload request method=POST path=/50942026/11/12 16:57:28 INFO Completed push id=29 objects=17 duration=1.244129ms50952026/11/12 16:57:28 INFO Completed upload id=12 objects=9 duration=1.531169ms50962026/11/12 16:57:28 INFO Received complete upload request method=POST path=/50972026/11/12 16:57:28 INFO Received complete push request method=POST path=/50982026/11/12 16:57:28 INFO Completed upload id=26 objects=24 duration=1.46645ms50992026/11/12 16:57:28 INFO Completed push id=26 objects=3 duration=1.65156ms51002026/11/12 16:57:28 INFO Received complete push request method=POST path=/51012026/11/12 16:57:28 INFO Completed push id=29 objects=15 duration=1.35836ms51022026/11/12 16:57:28 INFO Received complete upload request method=POST path=/51032026/11/12 16:57:28 INFO Completed upload id=34 objects=3 duration=1.107459ms51042026/11/12 16:57:28 INFO Aborted multipart uploads count=0 kept=051052026/11/12 16:57:28 WARN Force mode enabled - objects will be deleted immediately without grace period51062026/11/12 16:57:28 INFO Aborted multipart uploads count=0 kept=051072026/11/12 16:57:28 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=051082026/11/12 16:57:28 INFO Received complete push request method=POST path=/51092026/11/12 16:57:28 WARN Force mode enabled - objects will be deleted immediately without grace period51102026/11/12 16:57:28 INFO Vacuumed table table=pending_closures51112026/11/12 16:57:28 INFO Vacuumed table table=commit_receipts51122026/11/12 16:57:28 INFO Completed push id=32 objects=26 duration=1.615179ms51132026/11/12 16:57:28 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=051142026/11/12 16:57:28 INFO Received complete upload request method=POST path=/51152026/11/12 16:57:28 INFO Vacuumed table table=pending_objects51162026/11/12 16:57:28 INFO Vacuumed table table=multipart_uploads51172026/11/12 16:57:28 INFO Vacuumed table table=pending_closures51182026/11/12 16:57:28 INFO Completed upload id=28 objects=8 duration=1.33843ms51192026/11/12 16:57:28 INFO Vacuumed table table=closures51202026/11/12 16:57:28 INFO Vacuumed table table=commit_receipts51212026/11/12 16:57:28 INFO Received complete push request method=POST path=/51222026/11/12 16:57:28 INFO Vacuumed table table=objects51232026/11/12 16:57:28 INFO Vacuumed table table=pending_objects51242026/11/12 16:57:28 INFO Received complete push request method=POST path=/51252026/11/12 16:57:28 INFO Completed push id=30 objects=6 duration=1.538489ms51262026/11/12 16:57:28 INFO Vacuumed table table=multipart_uploads51272026/11/12 16:57:28 INFO Vacuumed table table=closures51282026/11/12 16:57:28 INFO Completed push id=35 objects=11 duration=1.57263ms51292026/11/12 16:57:28 INFO Vacuumed table table=objects51302026/11/12 16:57:28 INFO Received complete upload request method=POST path=/51312026/11/12 16:57:28 INFO Completed upload id=33 objects=2 duration=1.13991ms51322026/11/12 16:57:28 INFO Received complete upload request method=POST path=/51332026/11/12 16:57:28 INFO Completed upload id=29 objects=10 duration=985.37µs51342026/11/12 16:57:28 INFO Received complete upload request method=POST path=/51352026/11/12 16:57:28 INFO Received complete upload request method=POST path=/51362026/11/12 16:57:28 INFO Completed upload id=31 objects=3 duration=1.12197ms51372026/11/12 16:57:28 INFO Completed upload id=27 objects=26 duration=1.24238ms51382026/11/12 16:57:28 INFO Aborted multipart uploads count=0 kept=051392026/11/12 16:57:28 WARN Force mode enabled - objects will be deleted immediately without grace period51402026/11/12 16:57:28 INFO Garbage collection completed failed-uploads-deleted=6 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=051412026/11/12 16:57:28 INFO Received complete upload request method=POST path=/51422026/11/12 16:57:28 INFO Vacuumed table table=pending_closures51432026/11/12 16:57:28 INFO Received complete upload request method=POST path=/51442026/11/12 16:57:28 INFO Completed upload id=34 objects=5 duration=1.240819ms51452026/11/12 16:57:28 INFO Vacuumed table table=commit_receipts51462026/11/12 16:57:28 INFO Completed upload id=30 objects=20 duration=1.29433ms51472026/11/12 16:57:28 INFO Vacuumed table table=pending_objects51482026/11/12 16:57:28 INFO Vacuumed table table=multipart_uploads51492026/11/12 16:57:28 INFO Vacuumed table table=closures51502026/11/12 16:57:28 INFO Vacuumed table table=objects51512026/11/12 16:57:28 INFO Aborted multipart uploads count=0 kept=051522026/11/12 16:57:28 INFO Received complete push request method=POST path=/51532026/11/12 16:57:28 INFO Completed push id=28 objects=6 duration=1.42697ms51542026/11/12 16:57:28 WARN Force mode enabled - objects will be deleted immediately without grace period51552026/11/12 16:57:28 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=051562026/11/12 16:57:28 INFO Vacuumed table table=pending_closures51572026/11/12 16:57:28 INFO Vacuumed table table=commit_receipts51582026/11/12 16:57:28 INFO Vacuumed table table=pending_objects51592026/11/12 16:57:28 INFO Vacuumed table table=multipart_uploads51602026/11/12 16:57:28 INFO Vacuumed table table=closures51612026/11/12 16:57:28 INFO Vacuumed table table=objects51622026/11/12 16:57:28 INFO Received complete upload request method=POST path=/51632026/11/12 16:57:28 INFO Received complete push request method=POST path=/51642026/11/12 16:57:28 INFO Completed upload id=35 objects=11 duration=1.18593ms51652026/11/12 16:57:28 INFO Completed push id=31 objects=11 duration=1.29782ms51662026/11/12 16:57:28 INFO Received complete push request method=POST path=/51672026/11/12 16:57:28 INFO Completed push id=29 objects=6 duration=1.19702ms51682026/11/12 16:57:28 INFO Received complete upload request method=POST path=/51692026/11/12 16:57:28 INFO Received complete upload request method=POST path=/51702026/11/12 16:57:28 INFO Received complete upload request method=POST path=/51712026/11/12 16:57:28 INFO Completed upload id=18 objects=22 duration=1.958179ms51722026/11/12 16:57:28 INFO Completed upload id=36 objects=3 duration=1.10224ms51732026/11/12 16:57:28 INFO Completed upload id=32 objects=2 duration=1.00723ms51742026/11/12 16:57:28 INFO Received complete upload request method=POST path=/51752026/11/12 16:57:28 INFO Completed upload id=21 objects=8 duration=1.15848ms51762026/11/12 16:57:28 INFO Received complete push request method=POST path=/51772026/11/12 16:57:28 INFO Aborted multipart uploads count=0 kept=051782026/11/12 16:57:28 INFO Completed push id=33 objects=3 duration=1.16025ms51792026/11/12 16:57:28 WARN Force mode enabled - objects will be deleted immediately without grace period51802026/11/12 16:57:28 INFO Received complete upload request method=POST path=/51812026/11/12 16:57:28 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=051822026/11/12 16:57:28 INFO Completed upload id=30 objects=8 duration=1.30446ms51832026/11/12 16:57:28 INFO Vacuumed table table=pending_closures51842026/11/12 16:57:28 INFO Vacuumed table table=commit_receipts51852026/11/12 16:57:28 INFO Vacuumed table table=pending_objects51862026/11/12 16:57:28 INFO Vacuumed table table=multipart_uploads51872026/11/12 16:57:28 INFO Vacuumed table table=closures51882026/11/12 16:57:28 INFO Vacuumed table table=objects51892026/11/12 16:57:28 INFO Received complete upload request method=POST path=/51902026/11/12 16:57:28 INFO Completed upload id=34 objects=3 duration=1.04909ms51912026/11/12 16:57:28 INFO Received complete upload request method=POST path=/51922026/11/12 16:57:28 INFO Completed upload id=22 objects=7 duration=1.19888ms51932026/11/12 16:57:28 INFO Received complete upload request method=POST path=/51942026/11/12 16:57:28 INFO Completed upload id=31 objects=6 duration=1.25056ms51952026/11/12 16:57:28 INFO Aborted multipart uploads count=0 kept=051962026/11/12 16:57:28 INFO Received complete push request method=POST path=/51972026/11/12 16:57:28 WARN Force mode enabled - objects will be deleted immediately without grace period51982026/11/12 16:57:28 INFO Completed push id=24 objects=5 duration=1.749179ms51992026/11/12 16:57:28 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=052002026/11/12 16:57:28 INFO Vacuumed table table=pending_closures52012026/11/12 16:57:28 INFO Vacuumed table table=commit_receipts52022026/11/12 16:57:28 INFO Vacuumed table table=pending_objects52032026/11/12 16:57:28 INFO Vacuumed table table=multipart_uploads52042026/11/12 16:57:28 INFO Vacuumed table table=closures52052026/11/12 16:57:28 INFO Vacuumed table table=objects52062026/11/12 16:57:28 INFO Received complete upload request method=POST path=/52072026/11/12 16:57:28 INFO Completed upload id=32 objects=4 duration=1.21ms52082026/11/12 16:57:28 INFO Received complete upload request method=POST path=/52092026/11/12 16:57:28 INFO Completed upload id=25 objects=8 duration=958.37µs52102026/11/12 16:57:28 INFO Received complete upload request method=POST path=/52112026/11/12 16:57:28 INFO Completed upload id=33 objects=10 duration=1.161119ms52122026/11/12 16:57:28 INFO Received complete push request method=POST path=/52132026/11/12 16:57:28 INFO Completed push id=26 objects=4 duration=1.18577ms52142026/11/12 16:57:28 INFO Received complete push request method=POST path=/52152026/11/12 16:57:28 INFO Completed push id=30 objects=2 duration=1.22573ms52162026/11/12 16:57:28 INFO Received complete upload request method=POST path=/52172026/11/12 16:57:28 INFO Completed upload id=34 objects=23 duration=1.485599ms52182026/11/12 16:57:28 INFO Received complete push request method=POST path=/52192026/11/12 16:57:28 INFO Completed push id=32 objects=7 duration=1.480599ms52202026/11/12 16:57:28 INFO Received complete push request method=POST path=/52212026/11/12 16:57:28 INFO Completed push id=35 objects=6 duration=1.41105ms52222026/11/12 16:57:28 INFO Received complete upload request method=POST path=/52232026/11/12 16:57:28 INFO Aborted multipart uploads count=0 kept=052242026/11/12 16:57:28 INFO Completed upload id=33 objects=6 duration=1.3551ms52252026/11/12 16:57:28 WARN Force mode enabled - objects will be deleted immediately without grace period52262026/11/12 16:57:28 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=052272026/11/12 16:57:28 INFO Vacuumed table table=pending_closures52282026/11/12 16:57:28 INFO Vacuumed table table=commit_receipts52292026/11/12 16:57:28 INFO Vacuumed table table=pending_objects52302026/11/12 16:57:28 INFO Vacuumed table table=multipart_uploads52312026/11/12 16:57:28 INFO Vacuumed table table=closures52322026/11/12 16:57:28 INFO Vacuumed table table=objects52332026/11/12 16:57:28 INFO Received complete upload request method=POST path=/52342026/11/12 16:57:28 INFO Completed upload id=34 objects=3 duration=1.147519ms52352026/11/12 16:57:28 INFO Aborted multipart uploads count=0 kept=052362026/11/12 16:57:28 WARN Force mode enabled - objects will be deleted immediately without grace period52372026/11/12 16:57:28 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=052382026/11/12 16:57:28 INFO Vacuumed table table=pending_closures52392026/11/12 16:57:28 INFO Vacuumed table table=commit_receipts52402026/11/12 16:57:28 INFO Vacuumed table table=pending_objects52412026/11/12 16:57:28 INFO Vacuumed table table=multipart_uploads52422026/11/12 16:57:28 INFO Vacuumed table table=closures52432026/11/12 16:57:28 INFO Vacuumed table table=objects5244--- PASS: FuzzPushGC (0.00s)5245 --- PASS: FuzzPushGC/seed#14 (2.79s)5246 --- PASS: FuzzPushGC/seed#3 (3.02s)5247 --- PASS: FuzzPushGC/seed#13 (3.14s)5248 --- PASS: FuzzPushGC/seed#0 (3.21s)5249 --- PASS: FuzzPushGC/seed#9 (3.23s)5250 --- PASS: FuzzPushGC/seed#6 (3.26s)5251 --- PASS: FuzzPushGC/seed#2 (3.29s)5252 --- PASS: FuzzPushGC/seed#4 (3.31s)5253 --- PASS: FuzzPushGC/seed#7 (3.39s)5254 --- PASS: FuzzPushGC/seed#1 (3.39s)5255 --- PASS: FuzzPushGC/seed#8 (3.42s)5256 --- PASS: FuzzPushGC/seed#15 (3.44s)5257 --- PASS: FuzzPushGC/seed#11 (3.50s)5258 --- PASS: FuzzPushGC/seed#5 (3.53s)5259 --- PASS: FuzzPushGC/seed#12 (3.68s)5260 --- PASS: FuzzPushGC/seed#10 (3.73s)5261PASS5262{"timestamp":"2026-11-12T16:57:28.891442668Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:37402","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(392)"}5263{"timestamp":"2026-11-12T16:57:28.891555108Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:37870","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(390)"}52642026-11-12 16:57:29.332 UTC [129] LOG: received smart shutdown request52652026-11-12 16:57:29.336 UTC [129] LOG: background worker "logical replication launcher" (PID 139) exited with exit code 152662026-11-12 16:57:30.235 UTC [134] LOG: checkpoint complete: wrote 3758 buffers (22.9%), wrote 5 SLRU buffers; 0 WAL file(s) added, 0 removed, 33 recycled; write=16.764 s, sync=0.747 s, total=17.590 s; sync files=43130, longest=0.001 s, average=0.001 s; distance=534036 kB, estimate=534036 kB; lsn=0/274FF378, redo lsn=0/220EAF4852672026-11-12 16:57:30.238 UTC [134] LOG: shutting down52682026-11-12 16:57:30.238 UTC [134] LOG: checkpoint starting: shutdown immediate52692026-11-12 16:57:30.740 UTC [134] LOG: checkpoint complete: wrote 9851 buffers (60.1%), wrote 2 SLRU buffers; 0 WAL file(s) added, 0 removed, 5 recycled; write=0.204 s, sync=0.277 s, total=0.503 s; sync files=3080, longest=0.020 s, average=0.001 s; distance=86097 kB, estimate=489242 kB; lsn=0/274FF3F0, redo lsn=0/274FF3F052702026-11-12 16:57:30.829 UTC [129] LOG: database system is shut down5271Running OIDC tests...5272=== RUN TestAudienceForIssuer5273=== PAUSE TestAudienceForIssuer5274=== RUN TestProviderClientGivesUpOnSilentIssuer5275=== PAUSE TestProviderClientGivesUpOnSilentIssuer5276=== RUN TestGlobMatch5277=== PAUSE TestGlobMatch5278=== RUN TestValidateToken_ValidToken5279=== PAUSE TestValidateToken_ValidToken5280=== RUN TestValidateToken_WrongAudience5281=== PAUSE TestValidateToken_WrongAudience5282=== RUN TestValidateToken_Expired5283=== PAUSE TestValidateToken_Expired5284=== RUN TestValidateToken_BoundClaimsMismatch5285=== PAUSE TestValidateToken_BoundClaimsMismatch5286=== RUN TestValidateToken_BoundSubjectMismatch5287=== PAUSE TestValidateToken_BoundSubjectMismatch5288=== RUN TestValidateToken_MultipleProviders5289=== PAUSE TestValidateToken_MultipleProviders5290=== RUN TestValidateToken_UnknownIssuerFetchesNoKeys5291=== PAUSE TestValidateToken_UnknownIssuerFetchesNoKeys5292=== RUN TestValidateToken_GoogleSchemelessIssuer5293=== PAUSE TestValidateToken_GoogleSchemelessIssuer5294=== RUN TestValidateToken_KubernetesServiceAccount5295=== PAUSE TestValidateToken_KubernetesServiceAccount5296=== RUN TestNewValidator_KubernetesRequiresCA5297=== PAUSE TestNewValidator_KubernetesRequiresCA5298=== RUN TestValidateToken_KubernetesIssuerFromOwnToken5299=== PAUSE TestValidateToken_KubernetesIssuerFromOwnToken5300=== RUN TestPins_ReservedForMatchingRule5301=== PAUSE TestPins_ReservedForMatchingRule5302=== RUN TestPins_TopLevelShorthand5303=== PAUSE TestPins_TopLevelShorthand5304=== RUN TestPins_ConfigValidation5305=== PAUSE TestPins_ConfigValidation5306=== RUN TestScopes_LegacyProviderDefaultsToWrite5307=== PAUSE TestScopes_LegacyProviderDefaultsToWrite5308=== RUN TestScopes_Rules5309=== PAUSE TestScopes_Rules5310=== RUN TestScopes_ConfigValidation5311=== PAUSE TestScopes_ConfigValidation5312=== CONT TestAudienceForIssuer5313=== CONT TestValidateToken_GoogleSchemelessIssuer5314=== CONT TestScopes_Rules5315--- PASS: TestAudienceForIssuer (0.00s)5316=== CONT TestValidateToken_UnknownIssuerFetchesNoKeys5317=== CONT TestValidateToken_MultipleProviders5318=== CONT TestValidateToken_BoundSubjectMismatch5319=== CONT TestValidateToken_BoundClaimsMismatch53202026/11/12 16:57:33 INFO OIDC provider initialized name=known issuer=https://known.example5321=== CONT TestValidateToken_Expired5322=== CONT TestValidateToken_WrongAudience5323=== CONT TestValidateToken_ValidToken5324=== CONT TestGlobMatch5325=== CONT TestProviderClientGivesUpOnSilentIssuer5326=== CONT TestValidateToken_KubernetesIssuerFromOwnToken5327--- PASS: TestProviderClientGivesUpOnSilentIssuer (0.00s)5328=== CONT TestPins_ReservedForMatchingRule5329=== CONT TestNewValidator_KubernetesRequiresCA5330=== CONT TestScopes_ConfigValidation5331=== CONT TestValidateToken_KubernetesServiceAccount5332=== CONT TestScopes_LegacyProviderDefaultsToWrite5333=== CONT TestPins_ConfigValidation5334=== CONT TestPins_TopLevelShorthand5335=== RUN TestGlobMatch/foo_foo5336=== PAUSE TestGlobMatch/foo_foo5337=== RUN TestGlobMatch/foo_bar5338=== PAUSE TestGlobMatch/foo_bar5339=== RUN TestGlobMatch/*_5340=== PAUSE TestGlobMatch/*_5341=== RUN TestGlobMatch/*_anything5342=== PAUSE TestGlobMatch/*_anything5343=== RUN TestGlobMatch/foo*_foo5344=== PAUSE TestGlobMatch/foo*_foo5345=== RUN TestGlobMatch/foo*_foobar5346=== PAUSE TestGlobMatch/foo*_foobar5347=== RUN TestGlobMatch/foo*_bar5348=== PAUSE TestGlobMatch/foo*_bar5349=== RUN TestGlobMatch/*bar_bar5350=== PAUSE TestGlobMatch/*bar_bar5351=== RUN TestGlobMatch/*bar_foobar5352=== PAUSE TestGlobMatch/*bar_foobar5353=== RUN TestGlobMatch/*bar_foo5354=== PAUSE TestGlobMatch/*bar_foo5355=== RUN TestGlobMatch/foo*bar_foobar5356=== PAUSE TestGlobMatch/foo*bar_foobar5357=== RUN TestGlobMatch/foo*bar_foo123bar5358=== PAUSE TestGlobMatch/foo*bar_foo123bar5359=== RUN TestGlobMatch/foo*bar_foobarbaz5360=== PAUSE TestGlobMatch/foo*bar_foobarbaz5361=== RUN TestGlobMatch/*/*_foo/bar5362=== PAUSE TestGlobMatch/*/*_foo/bar5363=== RUN TestGlobMatch/*/*_foo5364=== PAUSE TestGlobMatch/*/*_foo5365=== RUN TestGlobMatch/refs/heads/*_refs/heads/main5366=== PAUSE TestGlobMatch/refs/heads/*_refs/heads/main5367=== RUN TestGlobMatch/refs/heads/*_refs/tags/v1.05368=== PAUSE TestGlobMatch/refs/heads/*_refs/tags/v1.05369=== RUN TestGlobMatch/refs/*/main_refs/heads/main5370=== PAUSE TestGlobMatch/refs/*/main_refs/heads/main5371=== RUN TestGlobMatch/fo?_foo5372=== PAUSE TestGlobMatch/fo?_foo5373=== RUN TestGlobMatch/fo?_fo5374=== PAUSE TestGlobMatch/fo?_fo5375=== RUN TestGlobMatch/fo?_fooo5376=== PAUSE TestGlobMatch/fo?_fooo5377=== RUN TestGlobMatch/?oo_foo5378=== PAUSE TestGlobMatch/?oo_foo5379=== RUN TestGlobMatch/?oo_boo5380=== PAUSE TestGlobMatch/?oo_boo5381=== RUN TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5382=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5383=== RUN TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5384=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5385=== CONT TestGlobMatch/foo_foo5386=== CONT TestGlobMatch/foo*bar_foobarbaz5387=== CONT TestGlobMatch/*_5388=== CONT TestGlobMatch/foo_bar5389=== CONT TestGlobMatch/?oo_boo5390=== CONT TestGlobMatch/?oo_foo5391=== CONT TestGlobMatch/fo?_fooo5392=== CONT TestGlobMatch/refs/heads/*_refs/tags/v1.05393=== CONT TestGlobMatch/foo*_bar5394=== CONT TestGlobMatch/fo?_foo5395=== CONT TestGlobMatch/foo*bar_foobar5396=== CONT TestGlobMatch/refs/*/main_refs/heads/main5397=== CONT TestGlobMatch/refs/heads/*_refs/heads/main5398=== CONT TestGlobMatch/*/*_foo5399=== CONT TestGlobMatch/*bar_foobar5400=== CONT TestGlobMatch/*bar_foo5401=== CONT TestGlobMatch/foo*_foo5402=== CONT TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5403=== CONT TestGlobMatch/foo*bar_foo123bar5404=== CONT TestGlobMatch/fo?_fo5405=== CONT TestGlobMatch/*_anything5406=== CONT TestGlobMatch/*bar_bar5407=== CONT TestGlobMatch/*/*_foo/bar5408--- PASS: TestPins_ConfigValidation (0.00s)5409--- PASS: TestScopes_ConfigValidation (0.00s)5410=== CONT TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5411=== CONT TestGlobMatch/foo*_foobar5412--- PASS: TestGlobMatch (0.01s)5413 --- PASS: TestGlobMatch/foo_foo (0.00s)5414 --- PASS: TestGlobMatch/foo*bar_foobarbaz (0.00s)5415 --- PASS: TestGlobMatch/*_ (0.00s)5416 --- PASS: TestGlobMatch/foo_bar (0.00s)5417 --- PASS: TestGlobMatch/?oo_boo (0.00s)5418 --- PASS: TestGlobMatch/?oo_foo (0.00s)5419 --- PASS: TestGlobMatch/fo?_fooo (0.00s)5420 --- PASS: TestGlobMatch/refs/heads/*_refs/tags/v1.0 (0.00s)5421 --- PASS: TestGlobMatch/foo*_bar (0.00s)5422 --- PASS: TestGlobMatch/fo?_foo (0.00s)5423 --- PASS: TestGlobMatch/foo*bar_foobar (0.00s)5424 --- PASS: TestGlobMatch/refs/*/main_refs/heads/main (0.00s)5425 --- PASS: TestGlobMatch/refs/heads/*_refs/heads/main (0.00s)5426 --- PASS: TestGlobMatch/*/*_foo (0.00s)5427 --- PASS: TestGlobMatch/*bar_foobar (0.00s)5428 --- PASS: TestGlobMatch/*bar_foo (0.00s)5429 --- PASS: TestGlobMatch/foo*_foo (0.00s)5430 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main (0.00s)5431 --- PASS: TestGlobMatch/foo*bar_foo123bar (0.00s)5432 --- PASS: TestGlobMatch/fo?_fo (0.00s)5433 --- PASS: TestGlobMatch/*_anything (0.00s)5434 --- PASS: TestGlobMatch/*bar_bar (0.00s)5435 --- PASS: TestGlobMatch/*/*_foo/bar (0.00s)5436 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main (0.00s)5437 --- PASS: TestGlobMatch/foo*_foobar (0.00s)54382026/11/12 16:57:33 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:40077/oidc5439--- PASS: TestPins_TopLevelShorthand (0.03s)54402026/11/12 16:57:33 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:35717/oidc54412026/11/12 16:57:33 INFO OIDC provider initialized name=kubernetes issuer=https://oidc.eks.invalid/id/ABC1235442--- PASS: TestPins_ReservedForMatchingRule (0.04s)5443--- PASS: TestValidateToken_KubernetesIssuerFromOwnToken (0.04s)5444--- PASS: TestValidateToken_UnknownIssuerFetchesNoKeys (0.05s)54452026/11/12 16:57:33 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:45015/oidc5446--- PASS: TestScopes_LegacyProviderDefaultsToWrite (0.06s)54472026/11/12 16:57:33 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:38571/oidc5448--- PASS: TestValidateToken_BoundClaimsMismatch (0.07s)54492026/11/12 16:57:33 INFO OIDC provider initialized name=google issuer=https://accounts.google.com5450--- PASS: TestValidateToken_GoogleSchemelessIssuer (0.09s)54512026/11/12 16:57:33 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:35055/oidc54522026/11/12 16:57:33 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:39599/oidc5453--- PASS: TestValidateToken_WrongAudience (0.09s)5454--- PASS: TestValidateToken_ValidToken (0.09s)54552026/11/12 16:57:33 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:38181/oidc5456--- PASS: TestScopes_Rules (0.11s)54572026/11/12 16:57:33 INFO OIDC provider initialized name=provider1 issuer=http://127.0.0.1:37899/oidc54582026/11/12 16:57:33 INFO OIDC provider initialized name=provider2 issuer=http://127.0.0.1:46031/oidc5459--- PASS: TestValidateToken_MultipleProviders (0.11s)54602026/11/12 16:57:33 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:41229/oidc5461--- PASS: TestValidateToken_Expired (0.11s)54622026/11/12 16:57:33 http: TLS handshake error from 127.0.0.1:60316: remote error: tls: bad certificate5463--- PASS: TestNewValidator_KubernetesRequiresCA (0.12s)54642026/11/12 16:57:33 INFO OIDC provider initialized name=kubernetes issuer=https://127.0.0.1:333335465--- PASS: TestValidateToken_KubernetesServiceAccount (0.12s)54662026/11/12 16:57:33 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:44925/oidc5467--- PASS: TestValidateToken_BoundSubjectMismatch (0.15s)5468PASS5469Running hook tests...5470=== RUN TestSendPathsEmpty5471=== PAUSE TestSendPathsEmpty5472=== RUN TestQueueEnqueueAndFetch5473=== PAUSE TestQueueEnqueueAndFetch5474=== RUN TestQueueDeduplication5475=== PAUSE TestQueueDeduplication5476=== RUN TestQueueRemove5477=== PAUSE TestQueueRemove5478=== RUN TestQueueFetchBatchLimit5479=== PAUSE TestQueueFetchBatchLimit5480=== RUN TestQueueRetryMovesToBack5481=== PAUSE TestQueueRetryMovesToBack5482=== RUN TestQueueFetchRemoveLifecycle5483=== PAUSE TestQueueFetchRemoveLifecycle5484=== RUN TestQueueConcurrentWriters5485=== PAUSE TestQueueConcurrentWriters5486=== RUN TestQueueRemoveLargeClosure5487=== PAUSE TestQueueRemoveLargeClosure5488=== RUN TestServerClientIntegration5489--- PASS: TestServerClientIntegration (0.00s)5490=== RUN TestServerQueueError54912026/11/12 16:57:33 ERROR Failed to queue paths error="permission denied" count=15492--- PASS: TestServerQueueError (0.00s)5493=== RUN TestGetListenerSocketActivation5494 server_test.go:220: === RUN TestGetListenerSocketActivation5495 --- PASS: TestGetListenerSocketActivation (0.00s)5496 PASS5497 5498--- PASS: TestGetListenerSocketActivation (0.01s)5499=== RUN TestServerStalledClientDoesNotBlockShutdown5500=== PAUSE TestServerStalledClientDoesNotBlockShutdown5501=== RUN TestServerBacksOffOnAcceptErrors5502=== PAUSE TestServerBacksOffOnAcceptErrors5503=== RUN TestDrainIsolatesPoisonPath5504=== PAUSE TestDrainIsolatesPoisonPath5505=== RUN TestRunNotBlockedByPoisonHead5506=== PAUSE TestRunNotBlockedByPoisonHead5507=== RUN TestDrainGivesUpWhenServerDown5508=== PAUSE TestDrainGivesUpWhenServerDown5509=== RUN TestFailedPathPrunedByLaterClosure5510=== PAUSE TestFailedPathPrunedByLaterClosure5511=== RUN TestWorkerUploadsAndRemoves5512=== PAUSE TestWorkerUploadsAndRemoves5513=== RUN TestWorkerSkipsGCdPaths5514=== PAUSE TestWorkerSkipsGCdPaths5515=== RUN TestWorkerPrunesClosureDeps5516=== PAUSE TestWorkerPrunesClosureDeps5517=== RUN TestDrainTimeout5518=== PAUSE TestDrainTimeout5519=== RUN TestWorkerRemoveFailureIsNotProgress5520=== PAUSE TestWorkerRemoveFailureIsNotProgress5521=== RUN TestWorkerKeepsPathItCannotStat5522=== PAUSE TestWorkerKeepsPathItCannotStat5523=== CONT TestSendPathsEmpty5524=== CONT TestDrainTimeout5525--- PASS: TestSendPathsEmpty (0.00s)5526=== CONT TestWorkerKeepsPathItCannotStat5527=== CONT TestWorkerRemoveFailureIsNotProgress5528=== CONT TestDrainIsolatesPoisonPath5529=== CONT TestServerBacksOffOnAcceptErrors5530=== CONT TestServerStalledClientDoesNotBlockShutdown55312000/01/01 00:00:00 ERROR Accept failed error="too many open files"55322000/01/01 00:00:00 ERROR Accept failed error="too many open files"55332000/01/01 00:00:00 ERROR Accept failed error="too many open files"55342000/01/01 00:00:00 ERROR Accept failed error="too many open files"55352000/01/01 00:00:00 ERROR Accept failed error="too many open files"55362000/01/01 00:00:00 ERROR Accept failed error="too many open files"55372000/01/01 00:00:00 ERROR Accept failed error="too many open files"55382000/01/01 00:00:00 ERROR Accept failed error="too many open files"5539=== CONT TestQueueRemoveLargeClosure5540=== CONT TestQueueConcurrentWriters5541=== CONT TestWorkerUploadsAndRemoves5542=== CONT TestQueueFetchRemoveLifecycle5543=== CONT TestWorkerPrunesClosureDeps5544=== CONT TestQueueRetryMovesToBack5545=== CONT TestQueueFetchBatchLimit5546=== CONT TestQueueRemove5547=== CONT TestQueueDeduplication5548=== CONT TestWorkerSkipsGCdPaths5549=== CONT TestQueueEnqueueAndFetch5550=== CONT TestDrainGivesUpWhenServerDown5551=== CONT TestRunNotBlockedByPoisonHead5552=== CONT TestFailedPathPrunedByLaterClosure5553--- PASS: TestServerBacksOffOnAcceptErrors (0.00s)55542026/11/12 16:57:33 INFO Upload queue status pending=255552026/11/12 16:57:33 INFO Upload queue status pending=255562026/11/12 16:57:33 INFO Uploading batch count=455572026/11/12 16:57:33 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat2044069685/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat2044069685/002/locked/aaa: permission denied"55582026/11/12 16:57:33 ERROR Upload failed error="upload failed" count=455592026/11/12 16:57:33 INFO Upload queue status pending=255602026/11/12 16:57:33 INFO Uploading batch count=255612026/11/12 16:57:33 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerSkipsGCdPaths974689981/002/nonexistent55622026/11/12 16:57:33 INFO Uploading batch count=255632026/11/12 16:57:33 ERROR Upload failed error="upload failed" count=255642026/11/12 16:57:33 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown1848269841/002/a55652026/11/12 16:57:33 INFO Uploading batch count=155662026/11/12 16:57:33 ERROR Upload failed error="upload failed" count=155672026/11/12 16:57:33 INFO Upload queue status pending=355682026/11/12 16:57:33 INFO Uploading batch count=155692026/11/12 16:57:33 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat2044069685/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat2044069685/002/locked/aaa: permission denied"55702026/11/12 16:57:33 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown1848269841/002/b55712026/11/12 16:57:33 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainIsolatesPoisonPath780603287/002/bbb5572--- PASS: TestQueueFetchBatchLimit (0.02s)55732026/11/12 16:57:33 INFO Uploading batch count=155742026/11/12 16:57:33 ERROR Upload failed error="upload failed" count=15575--- PASS: TestQueueDeduplication (0.02s)55762026/11/12 16:57:33 INFO Uploading batch count=15577--- PASS: TestQueueEnqueueAndFetch (0.02s)55782026/11/12 16:57:33 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat2044069685/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat2044069685/002/locked/aaa: permission denied"55792026/11/12 16:57:33 INFO Uploading batch count=255802026/11/12 16:57:33 ERROR Upload failed error="upload failed" count=255812026/11/12 16:57:33 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown1848269841/002/c55822026/11/12 16:57:33 INFO Uploading batch count=15583--- PASS: TestQueueFetchRemoveLifecycle (0.02s)55842026/11/12 16:57:33 ERROR Drain finished with paths left in queue remaining=155852000/01/01 00:00:00 INFO Uploading batch count=255862026/11/12 16:57:33 INFO Uploading batch count=155872000/01/01 00:00:00 ERROR Upload failed error="context deadline exceeded" count=255882026/11/12 16:57:33 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown1848269841/002/d55892026/11/12 16:57:33 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress1685448895/002/nonexistent5590--- PASS: TestQueueRetryMovesToBack (0.02s)55912000/01/01 00:00:00 ERROR Drain finished with paths left in queue remaining=455922026/11/12 16:57:33 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=15593--- PASS: TestQueueRemove (0.02s)55942026/11/12 16:57:33 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress1685448895/002/nonexistent55952026/11/12 16:57:33 INFO Uploading batch count=155962026/11/12 16:57:33 ERROR Upload failed error="upload failed" count=155972026/11/12 16:57:33 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=155982026/11/12 16:57:33 INFO Uploading batch count=255992026/11/12 16:57:33 ERROR Upload failed error="upload failed" count=256002026/11/12 16:57:33 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress1685448895/002/nonexistent56012026/11/12 16:57:33 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown1848269841/002/e56022026/11/12 16:57:33 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=156032026/11/12 16:57:33 INFO Uploading batch count=156042026/11/12 16:57:33 ERROR Upload failed error="upload failed" count=156052026/11/12 16:57:33 ERROR Drain finished with paths left in queue remaining=15606--- PASS: TestWorkerKeepsPathItCannotStat (0.02s)56072026/11/12 16:57:33 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown1848269841/002/f56082026/11/12 16:57:33 INFO Uploading batch count=156092026/11/12 16:57:33 ERROR Upload failed error="upload failed" count=15610--- PASS: TestDrainTimeout (0.02s)5611--- PASS: TestWorkerRemoveFailureIsNotProgress (0.02s)56122026/11/12 16:57:33 ERROR Drain finished with paths left in queue remaining=156132026/11/12 16:57:33 ERROR Drain finished with paths left in queue remaining=105614--- PASS: TestFailedPathPrunedByLaterClosure (0.02s)5615--- PASS: TestDrainIsolatesPoisonPath (0.03s)5616--- PASS: TestDrainGivesUpWhenServerDown (0.02s)5617--- PASS: TestWorkerPrunesClosureDeps (0.04s)5618--- PASS: TestWorkerUploadsAndRemoves (0.04s)5619--- PASS: TestWorkerSkipsGCdPaths (0.04s)5620--- PASS: TestQueueRemoveLargeClosure (0.06s)56212026/11/12 16:57:33 ERROR Failed to decode request error="read unix /build/hook1569228841/test.sock->@: i/o timeout"56222026/11/12 16:57:33 ERROR Failed to write response error="write unix /build/hook1569228841/test.sock->@: i/o timeout"5623--- PASS: TestServerStalledClientDoesNotBlockShutdown (0.20s)5624--- PASS: TestQueueConcurrentWriters (0.22s)56252026/11/12 16:57:34 INFO Uploading batch count=156262026/11/12 16:57:34 INFO Uploading batch count=156272026/11/12 16:57:34 INFO Uploading batch count=156282026/11/12 16:57:34 ERROR Upload failed error="upload failed" count=156292026/11/12 16:57:34 INFO Uploading batch count=156302026/11/12 16:57:34 ERROR Upload failed error="upload failed" count=156312026/11/12 16:57:34 INFO Uploading batch count=156322026/11/12 16:57:34 ERROR Upload failed error="upload failed" count=156332026/11/12 16:57:34 INFO Uploading batch count=156342026/11/12 16:57:34 ERROR Upload failed error="upload failed" count=156352026/11/12 16:57:34 ERROR Drain finished with paths left in queue remaining=15636--- PASS: TestRunNotBlockedByPoisonHead (1.03s)5637PASS5638Running signing tests...5639=== RUN TestExternalSigner5640=== PAUSE TestExternalSigner5641=== RUN TestExternalSignerWrongPublicKey5642=== PAUSE TestExternalSignerWrongPublicKey5643=== RUN TestExternalSignerErrors5644=== PAUSE TestExternalSignerErrors5645=== RUN TestExternalSignerRestartsAfterCrash5646=== PAUSE TestExternalSignerRestartsAfterCrash5647=== RUN TestExternalSignerProgramErrorKeepsProcess5648=== PAUSE TestExternalSignerProgramErrorKeepsProcess5649=== RUN TestExternalSignerClose5650=== PAUSE TestExternalSignerClose5651=== RUN TestExternalSignerRejectsInvalidPublicKey5652=== PAUSE TestExternalSignerRejectsInvalidPublicKey5653=== RUN TestExternalSignerCancellation5654=== PAUSE TestExternalSignerCancellation5655=== RUN TestGenerateFingerprint5656=== PAUSE TestGenerateFingerprint5657=== RUN TestParseSigningKey5658=== PAUSE TestParseSigningKey5659=== RUN TestSign5660=== PAUSE TestSign5661=== RUN TestSignBatch5662=== PAUSE TestSignBatch5663=== RUN TestNilKeyPublicKey5664=== PAUSE TestNilKeyPublicKey5665=== RUN TestSignErrors5666=== PAUSE TestSignErrors5667=== CONT TestExternalSigner5668=== CONT TestExternalSignerCancellation5669=== CONT TestExternalSignerProgramErrorKeepsProcess5670=== CONT TestExternalSignerClose5671=== CONT TestSignBatch5672=== CONT TestSignErrors5673=== CONT TestNilKeyPublicKey5674--- PASS: TestNilKeyPublicKey (0.00s)5675=== CONT TestParseSigningKey5676=== CONT TestSign5677=== CONT TestExternalSignerErrors5678=== CONT TestExternalSignerRestartsAfterCrash5679=== CONT TestGenerateFingerprint5680=== CONT TestExternalSignerWrongPublicKey5681=== CONT TestExternalSignerRejectsInvalidPublicKey5682=== RUN TestExternalSignerErrors/exit_on_request5683=== RUN TestExternalSignerRejectsInvalidPublicKey/#005684=== PAUSE TestExternalSignerErrors/exit_on_request5685=== PAUSE TestExternalSignerRejectsInvalidPublicKey/#005686=== RUN TestExternalSignerRejectsInvalidPublicKey/no-colon5687=== PAUSE TestExternalSignerRejectsInvalidPublicKey/no-colon5688=== RUN TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5689=== PAUSE TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5690=== RUN TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5691=== RUN TestExternalSignerErrors/wrong_shape5692=== RUN TestGenerateFingerprint/basic_with_references5693=== PAUSE TestExternalSignerErrors/wrong_shape5694=== RUN TestParseSigningKey/valid_32-byte_key5695=== PAUSE TestParseSigningKey/valid_32-byte_key5696=== RUN TestParseSigningKey/valid_32-byte_key_with_different_name5697=== RUN TestSignErrors/nil_key5698=== PAUSE TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5699=== RUN TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5700=== PAUSE TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5701=== PAUSE TestSignErrors/nil_key5702=== RUN TestSignErrors/nil_narinfo5703--- PASS: TestSign (0.00s)5704=== PAUSE TestSignErrors/nil_narinfo5705=== RUN TestSignErrors/invalid_narinfo5706=== PAUSE TestParseSigningKey/valid_32-byte_key_with_different_name5707=== RUN TestExternalSignerErrors/invalid_signature_base645708=== RUN TestParseSigningKey/no_colon5709=== PAUSE TestParseSigningKey/no_colon5710=== RUN TestParseSigningKey/empty_name5711=== PAUSE TestParseSigningKey/empty_name5712=== RUN TestParseSigningKey/invalid_base645713=== PAUSE TestParseSigningKey/invalid_base645714=== RUN TestParseSigningKey/wrong_length5715=== PAUSE TestExternalSignerErrors/invalid_signature_base645716=== CONT TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5717=== CONT TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5718=== CONT TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5719=== CONT TestExternalSignerRejectsInvalidPublicKey/no-colon5720--- PASS: TestSignBatch (0.00s)5721=== PAUSE TestGenerateFingerprint/basic_with_references5722=== RUN TestGenerateFingerprint/no_references5723=== PAUSE TestGenerateFingerprint/no_references5724=== RUN TestGenerateFingerprint/unsorted_references_get_sorted5725=== PAUSE TestGenerateFingerprint/unsorted_references_get_sorted5726=== RUN TestGenerateFingerprint/invalid_nar_hash_prefix5727=== PAUSE TestGenerateFingerprint/invalid_nar_hash_prefix5728=== RUN TestGenerateFingerprint/invalid_nar_hash_length5729=== PAUSE TestGenerateFingerprint/invalid_nar_hash_length5730=== PAUSE TestSignErrors/invalid_narinfo5731=== CONT TestExternalSignerRejectsInvalidPublicKey/#005732--- PASS: TestExternalSignerRejectsInvalidPublicKey (0.00s)5733 --- PASS: TestExternalSignerRejectsInvalidPublicKey/test-key:YQ== (0.00s)5734 --- PASS: TestExternalSignerRejectsInvalidPublicKey/test-key:invalid! (0.00s)5735 --- PASS: TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= (0.00s)5736 --- PASS: TestExternalSignerRejectsInvalidPublicKey/no-colon (0.00s)5737 --- PASS: TestExternalSignerRejectsInvalidPublicKey/#00 (0.00s)5738=== PAUSE TestParseSigningKey/wrong_length5739=== CONT TestParseSigningKey/valid_32-byte_key5740=== CONT TestParseSigningKey/invalid_base645741=== CONT TestParseSigningKey/no_colon5742=== RUN TestGenerateFingerprint/invalid_store_path_prefix5743=== PAUSE TestGenerateFingerprint/invalid_store_path_prefix5744=== RUN TestSignErrors/canceled5745=== RUN TestGenerateFingerprint/invalid_reference_prefix5746=== PAUSE TestGenerateFingerprint/invalid_reference_prefix5747=== CONT TestParseSigningKey/empty_name5748=== CONT TestParseSigningKey/valid_32-byte_key_with_different_name5749=== CONT TestParseSigningKey/wrong_length5750=== RUN TestExternalSignerErrors/wrong_signature_length5751=== PAUSE TestSignErrors/canceled5752=== CONT TestSignErrors/nil_key5753=== CONT TestGenerateFingerprint/basic_with_references5754=== CONT TestSignErrors/invalid_narinfo5755=== CONT TestGenerateFingerprint/invalid_store_path_prefix5756=== CONT TestGenerateFingerprint/invalid_nar_hash_length5757=== CONT TestGenerateFingerprint/invalid_nar_hash_prefix5758=== CONT TestGenerateFingerprint/invalid_reference_prefix5759=== CONT TestGenerateFingerprint/no_references5760=== CONT TestGenerateFingerprint/unsorted_references_get_sorted5761--- PASS: TestParseSigningKey (0.00s)5762 --- PASS: TestParseSigningKey/valid_32-byte_key (0.00s)5763 --- PASS: TestParseSigningKey/invalid_base64 (0.00s)5764 --- PASS: TestParseSigningKey/no_colon (0.00s)5765 --- PASS: TestParseSigningKey/empty_name (0.00s)5766 --- PASS: TestParseSigningKey/valid_32-byte_key_with_different_name (0.00s)5767 --- PASS: TestParseSigningKey/wrong_length (0.00s)5768--- PASS: TestExternalSignerClose (0.00s)5769=== PAUSE TestExternalSignerErrors/wrong_signature_length5770=== RUN TestExternalSignerErrors/invalid_JSON5771=== PAUSE TestExternalSignerErrors/invalid_JSON5772=== CONT TestSignErrors/nil_narinfo5773--- PASS: TestGenerateFingerprint (0.00s)5774 --- PASS: TestGenerateFingerprint/basic_with_references (0.00s)5775 --- PASS: TestGenerateFingerprint/invalid_store_path_prefix (0.00s)5776 --- PASS: TestGenerateFingerprint/invalid_nar_hash_length (0.00s)5777 --- PASS: TestGenerateFingerprint/invalid_nar_hash_prefix (0.00s)5778 --- PASS: TestGenerateFingerprint/invalid_reference_prefix (0.00s)5779 --- PASS: TestGenerateFingerprint/no_references (0.00s)5780 --- PASS: TestGenerateFingerprint/unsorted_references_get_sorted (0.00s)5781=== CONT TestSignErrors/canceled5782--- PASS: TestSignErrors (0.00s)5783 --- PASS: TestSignErrors/nil_key (0.00s)5784 --- PASS: TestSignErrors/invalid_narinfo (0.00s)5785 --- PASS: TestSignErrors/nil_narinfo (0.00s)5786 --- PASS: TestSignErrors/canceled (0.00s)5787=== RUN TestExternalSignerErrors/too_few_signatures5788=== PAUSE TestExternalSignerErrors/too_few_signatures5789=== RUN TestExternalSignerErrors/wrong_signature_name5790=== PAUSE TestExternalSignerErrors/wrong_signature_name5791=== RUN TestExternalSignerErrors/exit_at_startup5792=== PAUSE TestExternalSignerErrors/exit_at_startup5793=== RUN TestExternalSignerErrors/null5794=== PAUSE TestExternalSignerErrors/null5795=== RUN TestExternalSignerErrors/wrong_id5796=== PAUSE TestExternalSignerErrors/wrong_id5797=== RUN TestExternalSignerErrors/missing_signatures5798=== PAUSE TestExternalSignerErrors/missing_signatures5799=== RUN TestExternalSignerErrors/wrong_signatures_shape5800=== PAUSE TestExternalSignerErrors/wrong_signatures_shape5801=== RUN TestExternalSignerErrors/unrequested_key5802=== PAUSE TestExternalSignerErrors/unrequested_key5803=== RUN TestExternalSignerErrors/program_error5804=== PAUSE TestExternalSignerErrors/program_error5805=== RUN TestExternalSignerErrors/null_signatures5806=== PAUSE TestExternalSignerErrors/null_signatures5807=== RUN TestExternalSignerErrors/missing_signature_name5808=== PAUSE TestExternalSignerErrors/missing_signature_name5809=== CONT TestExternalSignerErrors/exit_on_request5810=== CONT TestExternalSignerErrors/null5811=== CONT TestExternalSignerErrors/unrequested_key5812=== CONT TestExternalSignerErrors/missing_signature_name5813=== CONT TestExternalSignerErrors/null_signatures5814=== CONT TestExternalSignerErrors/program_error5815=== CONT TestExternalSignerErrors/missing_signatures5816=== CONT TestExternalSignerErrors/wrong_id5817=== CONT TestExternalSignerErrors/wrong_signatures_shape5818=== CONT TestExternalSignerErrors/invalid_JSON5819=== CONT TestExternalSignerErrors/wrong_signature_name5820=== CONT TestExternalSignerErrors/too_few_signatures5821=== CONT TestExternalSignerErrors/exit_at_startup5822=== CONT TestExternalSignerErrors/wrong_shape5823=== CONT TestExternalSignerErrors/wrong_signature_length5824=== CONT TestExternalSignerErrors/invalid_signature_base645825--- PASS: TestExternalSignerWrongPublicKey (0.00s)5826--- PASS: TestExternalSigner (0.01s)5827--- PASS: TestExternalSignerProgramErrorKeepsProcess (0.01s)5828--- PASS: TestExternalSignerRestartsAfterCrash (0.01s)5829--- PASS: TestExternalSignerErrors (0.00s)5830 --- PASS: TestExternalSignerErrors/null (0.00s)5831 --- PASS: TestExternalSignerErrors/exit_on_request (0.00s)5832 --- PASS: TestExternalSignerErrors/program_error (0.00s)5833 --- PASS: TestExternalSignerErrors/missing_signature_name (0.01s)5834 --- PASS: TestExternalSignerErrors/wrong_signature_name (0.01s)5835 --- PASS: TestExternalSignerErrors/missing_signatures (0.01s)5836 --- PASS: TestExternalSignerErrors/exit_at_startup (0.01s)5837 --- PASS: TestExternalSignerErrors/wrong_id (0.01s)5838 --- PASS: TestExternalSignerErrors/unrequested_key (0.01s)5839 --- PASS: TestExternalSignerErrors/wrong_signatures_shape (0.01s)5840 --- PASS: TestExternalSignerErrors/wrong_signature_length (0.01s)5841 --- PASS: TestExternalSignerErrors/null_signatures (0.01s)5842 --- PASS: TestExternalSignerErrors/invalid_signature_base64 (0.01s)5843 --- PASS: TestExternalSignerErrors/wrong_shape (0.01s)5844 --- PASS: TestExternalSignerErrors/invalid_JSON (0.01s)5845 --- PASS: TestExternalSignerErrors/too_few_signatures (0.01s)5846--- PASS: TestExternalSignerCancellation (0.02s)5847PASS5848Running OpenBao signer tests...5849=== RUN TestSign5850=== PAUSE TestSign5851=== RUN TestSignErrors5852=== PAUSE TestSignErrors5853=== RUN TestServe5854=== PAUSE TestServe5855=== RUN TestServeRejectsGarbage5856=== PAUSE TestServeRejectsGarbage5857=== CONT TestSign5858=== CONT TestServeRejectsGarbage5859=== CONT TestServe5860--- PASS: TestServeRejectsGarbage (0.00s)5861=== CONT TestSignErrors5862=== RUN TestSignErrors/no_prefix5863=== PAUSE TestSignErrors/no_prefix5864=== RUN TestSignErrors/invalid_base645865=== PAUSE TestSignErrors/invalid_base645866=== RUN TestSignErrors/http_error5867=== PAUSE TestSignErrors/http_error5868=== RUN TestSignErrors/invalid_json5869=== PAUSE TestSignErrors/invalid_json5870=== CONT TestSignErrors/no_prefix5871=== CONT TestSignErrors/http_error5872=== CONT TestSignErrors/invalid_base645873=== CONT TestSignErrors/invalid_json5874--- PASS: TestServe (0.00s)5875--- PASS: TestSign (0.00s)5876--- PASS: TestSignErrors (0.00s)5877 --- PASS: TestSignErrors/invalid_json (0.00s)5878 --- PASS: TestSignErrors/no_prefix (0.00s)5879 --- PASS: TestSignErrors/invalid_base64 (0.00s)5880 --- PASS: TestSignErrors/http_error (0.00s)5881PASS