niks3-go-unit-tests
checks.x86_64-linux.go-unit-tests
· build #509
· raw
1tribuchet: building on jamie2Running client tests...3=== RUN TestDoServerRequestAttachesToken4=== PAUSE TestDoServerRequestAttachesToken5=== RUN TestUploadBuildLog_FileBodyReplayedOnRetry6=== PAUSE TestUploadBuildLog_FileBodyReplayedOnRetry7=== RUN TestRegisterUploadedObjectReusesConnections8=== PAUSE TestRegisterUploadedObjectReusesConnections9=== RUN TestRunGarbageCollection_FinishedOnAnotherReplica10=== PAUSE TestRunGarbageCollection_FinishedOnAnotherReplica11=== RUN TestRunGarbageCollection_NotFoundAfterLocalRun12=== PAUSE TestRunGarbageCollection_NotFoundAfterLocalRun13=== RUN TestCaseHackSuffix14=== PAUSE TestCaseHackSuffix15=== RUN TestFilterOversizedClosures16=== PAUSE TestFilterOversizedClosures17=== RUN TestUploadMultipart_PartsInParallel18=== PAUSE TestUploadMultipart_PartsInParallel19=== RUN TestUploadMultipart_ReturnsAfterDroppedBody20=== PAUSE TestUploadMultipart_ReturnsAfterDroppedBody21=== RUN TestUploadMultipart_FailedProducerIsNotCompleted22=== PAUSE TestUploadMultipart_FailedProducerIsNotCompleted23=== RUN TestPartSizeForNAR24=== PAUSE TestPartSizeForNAR25=== RUN TestDumpPathCaseHackMatchesNix26--- PASS: TestDumpPathCaseHackMatchesNix (0.04s)27=== RUN TestDumpPathCaseHackCollision28--- PASS: TestDumpPathCaseHackCollision (0.00s)29=== RUN TestDumpPathMatchesNix30=== PAUSE TestDumpPathMatchesNix31=== RUN TestDumpPathSingleFile32=== PAUSE TestDumpPathSingleFile33=== RUN TestDumpPathWriterError34=== PAUSE TestDumpPathWriterError35=== RUN TestEncodeNixBase3236=== PAUSE TestEncodeNixBase3237=== RUN TestEncodeNixBase32WithRealHash38=== PAUSE TestEncodeNixBase32WithRealHash39=== RUN TestConvertHashToNix3240=== PAUSE TestConvertHashToNix3241=== RUN TestGetStorePathHash42=== PAUSE TestGetStorePathHash43=== RUN TestPathInfoHashCompatibility44=== PAUSE TestPathInfoHashCompatibility45=== RUN TestParsePathInfoJSON46=== PAUSE TestParsePathInfoJSON47=== RUN TestParsePathInfoJSONMultiplePaths48=== PAUSE TestParsePathInfoJSONMultiplePaths49=== RUN TestPathInfoCACompatibility50=== PAUSE TestPathInfoCACompatibility51=== RUN TestRateLimiterFeedback52=== PAUSE TestRateLimiterFeedback53=== RUN TestRateLimiterFeedback_400DoesNotCountAsSuccess54=== PAUSE TestRateLimiterFeedback_400DoesNotCountAsSuccess55=== RUN TestRegisterUploadedObject_BoundedAgainstSilentServer56=== PAUSE TestRegisterUploadedObject_BoundedAgainstSilentServer57=== RUN TestRegisterUploadedObject_OutlivesPush58=== PAUSE TestRegisterUploadedObject_OutlivesPush59=== RUN TestResolveStorePath60=== PAUSE TestResolveStorePath61=== RUN TestDoWithRetry_BodyReplayedViaGetBody62=== PAUSE TestDoWithRetry_BodyReplayedViaGetBody63=== RUN TestRetryTimeoutOutlastsMaxRetries64=== PAUSE TestRetryTimeoutOutlastsMaxRetries65=== RUN TestDoWithRetry_FinalResponseBodyReadable66=== PAUSE TestDoWithRetry_FinalResponseBodyReadable67=== RUN TestShellSplit68=== PAUSE TestShellSplit69=== RUN TestShellSplitErrors70=== PAUSE TestShellSplitErrors71=== RUN TestStreamPushReportsEveryPath72=== PAUSE TestStreamPushReportsEveryPath73=== RUN TestStreamPushBatchesUnderLoad74=== PAUSE TestStreamPushBatchesUnderLoad75=== RUN TestStreamPushIsolatesFailures76=== PAUSE TestStreamPushIsolatesFailures77=== RUN TestStreamPushGivesUpOnDeadServer78=== PAUSE TestStreamPushGivesUpOnDeadServer79=== RUN TestStreamPushRequestLine80=== PAUSE TestStreamPushRequestLine81=== RUN TestStreamPushReportsSignatures82=== PAUSE TestStreamPushReportsSignatures83=== RUN TestStreamPushReportsSkippedPaths84=== PAUSE TestStreamPushReportsSkippedPaths85=== RUN TestClientSignaturesByStorePath86=== PAUSE TestClientSignaturesByStorePath87=== RUN TestStreamPushAnswersEveryLineWhenCanceled88=== PAUSE TestStreamPushAnswersEveryLineWhenCanceled89=== RUN TestStreamPushStopsWhenCanceled90=== PAUSE TestStreamPushStopsWhenCanceled91=== RUN TestSetClientTLS92=== PAUSE TestSetClientTLS93=== RUN TestSetClientTLSDoesNotMutateDefaultTransport94=== PAUSE TestSetClientTLSDoesNotMutateDefaultTransport95=== RUN TestSetClientTLSErrors96=== PAUSE TestSetClientTLSErrors97=== RUN TestStaticToken98=== PAUSE TestStaticToken99=== RUN TestFileTokenReadsAndCaches100=== PAUSE TestFileTokenReadsAndCaches101=== RUN TestFileTokenMissing102=== PAUSE TestFileTokenMissing103=== RUN TestFileTokenEmpty104=== PAUSE TestFileTokenEmpty105=== RUN TestScriptTokenNoExpiryRerunsEveryCall106=== PAUSE TestScriptTokenNoExpiryRerunsEveryCall107=== RUN TestScriptTokenCachesUntilRefresh108=== PAUSE TestScriptTokenCachesUntilRefresh109=== RUN TestScriptTokenEmptyToken110=== PAUSE TestScriptTokenEmptyToken111=== RUN TestScriptTokenBadJSON112=== PAUSE TestScriptTokenBadJSON113=== RUN TestScriptTokenScriptFails114=== PAUSE TestScriptTokenScriptFails115=== RUN TestScriptTokenEmptyCommand116=== PAUSE TestScriptTokenEmptyCommand117=== CONT TestDoServerRequestAttachesToken118=== CONT TestScriptTokenBadJSON119=== CONT TestScriptTokenNoExpiryRerunsEveryCall120=== CONT TestScriptTokenEmptyCommand121=== CONT TestScriptTokenEmptyToken122=== CONT TestScriptTokenCachesUntilRefresh123=== CONT TestRegisterUploadedObject_OutlivesPush124=== CONT TestFileTokenEmpty125=== CONT TestFileTokenMissing126=== CONT TestFileTokenReadsAndCaches127=== CONT TestStaticToken128=== CONT TestShellSplitErrors129=== CONT TestSetClientTLSErrors130=== CONT TestShellSplit131=== CONT TestSetClientTLSDoesNotMutateDefaultTransport132=== CONT TestSetClientTLS133=== CONT TestStreamPushStopsWhenCanceled134=== CONT TestStreamPushAnswersEveryLineWhenCanceled135=== CONT TestRetryTimeoutOutlastsMaxRetries136=== CONT TestClientSignaturesByStorePath137=== CONT TestResolveStorePath138=== CONT TestStreamPushReportsSkippedPaths139=== CONT TestStreamPushReportsSignatures140=== CONT TestStreamPushRequestLine141=== CONT TestStreamPushGivesUpOnDeadServer142=== CONT TestStreamPushIsolatesFailures143=== CONT TestStreamPushBatchesUnderLoad144=== CONT TestScriptTokenScriptFails1452026/11/12 11:49:42 ERROR Upload failed error=boom count=1146=== CONT TestRegisterUploadedObject_BoundedAgainstSilentServer147--- PASS: TestScriptTokenEmptyCommand (0.00s)1482000/01/01 00:00:00 ERROR Upload failed error="context canceled" count=2149=== CONT TestStreamPushReportsEveryPath150=== CONT TestDoWithRetry_FinalResponseBodyReadable151=== CONT TestDoWithRetry_BodyReplayedViaGetBody152=== CONT TestDumpPathWriterError153=== RUN TestStreamPushBatchesUnderLoad/oneAtATime=false154=== CONT TestGetStorePathHash155=== RUN TestGetStorePathHash/valid_store_path156=== PAUSE TestGetStorePathHash/valid_store_path157=== RUN TestGetStorePathHash/basename_without_hyphen_should_error158=== CONT TestParsePathInfoJSON159=== CONT TestEncodeNixBase32WithRealHash1602026/11/12 11:49:42 ERROR Upload failed error="bad path" count=31612026/11/12 11:49:42 ERROR Upload failed error="connection refused" count=201622026/11/12 11:49:42 ERROR Upload failed error=boom count=1163=== RUN TestParsePathInfoJSON/Nix_format1642026/11/12 11:49:42 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:44939165=== CONT TestRateLimiterFeedback_400DoesNotCountAsSuccess166=== PAUSE TestParsePathInfoJSON/Nix_format1672026/11/12 11:49:42 ERROR Server seems unavailable, giving up on batch untried=17168--- PASS: TestStaticToken (0.00s)169--- PASS: TestShellSplitErrors (0.00s)170=== CONT TestPathInfoHashCompatibility1712026/11/12 11:49:42 WARN Rate limiter enabled after throttle name=server-test rate=5172=== PAUSE TestGetStorePathHash/basename_without_hyphen_should_error1732026/11/12 11:49:42 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:44939174=== PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=false175=== RUN TestStreamPushBatchesUnderLoad/oneAtATime=true176=== CONT TestConvertHashToNix32177=== RUN TestConvertHashToNix32/SRI_format_to_Nix32178=== CONT TestRateLimiterFeedback179=== RUN TestParsePathInfoJSON/Lix_format180--- PASS: TestShellSplit (0.00s)181=== RUN TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)182=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)183=== CONT TestEncodeNixBase32184=== PAUSE TestParsePathInfoJSON/Lix_format185=== CONT TestParsePathInfoJSONMultiplePaths186=== RUN TestParsePathInfoJSON/empty_input187=== RUN TestRateLimiterFeedback/429_enables_limiter188=== RUN TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths189=== PAUSE TestRateLimiterFeedback/429_enables_limiter190=== CONT TestUploadMultipart_PartsInParallel191=== PAUSE TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths192=== PAUSE TestParsePathInfoJSON/empty_input193=== PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=true194=== RUN TestSetClientTLSErrors/missing_cert_file195=== PAUSE TestSetClientTLSErrors/missing_cert_file196=== RUN TestSetClientTLSErrors/missing_key_file197=== PAUSE TestSetClientTLSErrors/missing_key_file198=== CONT TestPathInfoCACompatibility199--- PASS: TestStreamPushStopsWhenCanceled (0.00s)200=== PAUSE TestConvertHashToNix32/SRI_format_to_Nix32201=== RUN TestPathInfoHashCompatibility/old_string_format_with_colon202=== RUN TestEncodeNixBase32/test_string_hash203=== RUN TestRateLimiterFeedback/503_enables_limiter204=== RUN TestGetStorePathHash/hash_with_invalid_characters_should_error205=== RUN TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths206=== RUN TestParsePathInfoJSON/whitespace_only207=== CONT TestDumpPathSingleFile208=== RUN TestPathInfoCACompatibility/null_ca_field209=== PAUSE TestRateLimiterFeedback/503_enables_limiter2102026/11/12 11:49:42 WARN Rate limiter enabled after throttle name=server-test rate=5211=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_colon2122026/11/12 11:49:42 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:41323213=== PAUSE TestPathInfoCACompatibility/null_ca_field214=== RUN TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI215--- PASS: TestFileTokenMissing (0.00s)216--- PASS: TestClientSignaturesByStorePath (0.00s)217--- PASS: TestScriptTokenBadJSON (0.01s)218--- PASS: TestStreamPushReportsSkippedPaths (0.00s)219--- PASS: TestFileTokenReadsAndCaches (0.00s)220--- PASS: TestFileTokenEmpty (0.01s)221--- PASS: TestScriptTokenEmptyToken (0.01s)222=== PAUSE TestEncodeNixBase32/test_string_hash223=== RUN TestEncodeNixBase32/empty_input224=== RUN TestConvertHashToNix32/already_Nix32_format225=== RUN TestSetClientTLSErrors/missing_ca_file226=== PAUSE TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths227=== CONT TestRunGarbageCollection_NotFoundAfterLocalRun2282026/11/12 11:49:42 WARN Rate limiter backed off name=server-test rate=5229=== RUN TestRateLimiterFeedback/200_does_not_enable_limiter230=== PAUSE TestRateLimiterFeedback/200_does_not_enable_limiter231=== PAUSE TestParsePathInfoJSON/whitespace_only232=== CONT TestUploadMultipart_ReturnsAfterDroppedBody233=== PAUSE TestGetStorePathHash/hash_with_invalid_characters_should_error234=== CONT TestFilterOversizedClosures235=== RUN TestFilterOversizedClosures/no_limit_keeps_everything2362026/11/12 11:49:42 WARN Rate limiter enabled after throttle name=server-test rate=5237=== PAUSE TestFilterOversizedClosures/no_limit_keeps_everything238--- PASS: TestStreamPushReportsSignatures (0.00s)239=== RUN TestPathInfoCACompatibility/old_string_format_-_text2402026/11/12 11:49:42 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:34501241=== PAUSE TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI242=== PAUSE TestEncodeNixBase32/empty_input2432026/11/12 11:49:42 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:44939244=== PAUSE TestPathInfoCACompatibility/old_string_format_-_text245=== CONT TestCaseHackSuffix246=== RUN TestPathInfoCACompatibility/old_string_format_-_fixed_recursive247=== CONT TestDumpPathMatchesNix248=== PAUSE TestSetClientTLSErrors/missing_ca_file249=== RUN TestRateLimiterFeedback/400_does_not_enable_limiter2502026/11/12 11:49:42 WARN Rate limiter backed off name=server-test rate=5251=== PAUSE TestRateLimiterFeedback/400_does_not_enable_limiter2522026/11/12 11:49:42 WARN Request returned retryable status, retrying attempt=2 backoff=0s status=503 url=http://127.0.0.1:34501253=== RUN TestSetClientTLSErrors/invalid_ca_file254=== PAUSE TestSetClientTLSErrors/invalid_ca_file2552026/11/12 11:49:42 INFO Garbage collection started256=== PAUSE TestConvertHashToNix32/already_Nix32_format257=== RUN TestConvertHashToNix32/invalid_format258=== PAUSE TestConvertHashToNix32/invalid_format259=== RUN TestParsePathInfoJSON/invalid_JSON260=== PAUSE TestParsePathInfoJSON/invalid_JSON261=== RUN TestGetStorePathHash/hash_with_wrong_length_should_error262--- PASS: TestResolveStorePath (0.00s)263=== RUN TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped264=== RUN TestPathInfoHashCompatibility/new_structured_format_with_sha512265=== PAUSE TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped266=== CONT TestPartSizeForNAR267=== RUN TestPartSizeForNAR/zero_stays_at_minimum268=== PAUSE TestPathInfoCACompatibility/old_string_format_-_fixed_recursive269=== CONT TestUploadMultipart_FailedProducerIsNotCompleted270=== RUN TestPathInfoCACompatibility/new_structured_format_-_text2712026/11/12 11:49:42 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:44939272=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_text273=== RUN TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part274=== PAUSE TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part275=== RUN TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary276=== PAUSE TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary277=== CONT TestUploadBuildLog_FileBodyReplayedOnRetry278=== CONT TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths279=== CONT TestRegisterUploadedObjectReusesConnections280=== CONT TestStreamPushBatchesUnderLoad/oneAtATime=false281=== PAUSE TestGetStorePathHash/hash_with_wrong_length_should_error282--- PASS: TestDoWithRetry_BodyReplayedViaGetBody (0.09s)283--- PASS: TestStreamPushReportsEveryPath (0.00s)284=== CONT TestEncodeNixBase32/empty_input285=== CONT TestRateLimiterFeedback/429_enables_limiter286=== CONT TestRateLimiterFeedback/503_enables_limiter287--- PASS: TestEncodeNixBase32WithRealHash (0.00s)288--- PASS: TestStreamPushIsolatesFailures (0.00s)289--- PASS: TestScriptTokenScriptFails (0.00s)290--- PASS: TestStreamPushGivesUpOnDeadServer (0.00s)291--- PASS: TestRegisterUploadedObject_OutlivesPush (0.01s)292--- PASS: TestScriptTokenNoExpiryRerunsEveryCall (0.01s)293--- PASS: TestSetClientTLSDoesNotMutateDefaultTransport (0.01s)294--- PASS: TestScriptTokenCachesUntilRefresh (0.01s)295--- PASS: TestDoWithRetry_FinalResponseBodyReadable (0.09s)296--- PASS: TestDoServerRequestAttachesToken (0.10s)297--- PASS: TestUploadMultipart_ReturnsAfterDroppedBody (0.00s)298--- PASS: TestStreamPushAnswersEveryLineWhenCanceled (0.10s)299=== CONT TestRateLimiterFeedback/400_does_not_enable_limiter300--- PASS: TestRunGarbageCollection_NotFoundAfterLocalRun (0.09s)301=== CONT TestRateLimiterFeedback/200_does_not_enable_limiter302=== PAUSE TestPathInfoHashCompatibility/new_structured_format_with_sha512303=== RUN TestSetClientTLS/rejects_connection_without_client_cert304=== PAUSE TestSetClientTLS/rejects_connection_without_client_cert305=== RUN TestSetClientTLS/succeeds_with_client_cert_and_CA306=== PAUSE TestSetClientTLS/succeeds_with_client_cert_and_CA307=== RUN TestSetClientTLS/preserves_debug_logging_transport308=== PAUSE TestSetClientTLS/preserves_debug_logging_transport309=== CONT TestSetClientTLSErrors/missing_cert_file3102026/11/12 11:49:42 WARN Rate limiter enabled after throttle name=server-test rate=5311=== RUN TestFilterOversizedClosures/all_closures_skipped3122026/11/12 11:49:42 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:41007313=== PAUSE TestFilterOversizedClosures/all_closures_skipped314=== CONT TestSetClientTLSErrors/missing_ca_file3152026/11/12 11:49:42 WARN Rate limiter enabled after throttle name=server-test rate=5316=== CONT TestSetClientTLSErrors/missing_key_file3172026/11/12 11:49:42 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=429 url=http://127.0.0.1:44941318=== CONT TestConvertHashToNix32/already_Nix32_format3192026/11/12 11:49:42 WARN Rate limiter backed off name=server-test rate=5320=== CONT TestConvertHashToNix32/SRI_format_to_Nix32321=== CONT TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths322=== CONT TestParsePathInfoJSON/invalid_JSON3232026/11/12 11:49:42 WARN Rate limiter backed off name=server-test rate=5324=== CONT TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part3252026/11/12 11:49:42 WARN Request returned retryable status, retrying attempt=3 backoff=1ms status=502 url=http://127.0.0.1:44939326=== CONT TestParsePathInfoJSON/whitespace_only3272026/11/12 11:49:42 WARN Rate limiter enabled after throttle name=s3-test rate=5328=== CONT TestGetStorePathHash/valid_store_path3292026/11/12 11:49:42 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:32839330=== RUN TestPathInfoCACompatibility/new_structured_format_-_nar_method331=== CONT TestEncodeNixBase32/test_string_hash332=== CONT TestStreamPushBatchesUnderLoad/oneAtATime=true333=== CONT TestSetClientTLSErrors/invalid_ca_file334=== PAUSE TestPartSizeForNAR/zero_stays_at_minimum335=== CONT TestConvertHashToNix32/invalid_format336=== CONT TestParsePathInfoJSON/Nix_format337--- PASS: TestParsePathInfoJSONMultiplePaths (0.00s)338 --- PASS: TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths (0.00s)339 --- PASS: TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths (0.00s)340=== CONT TestRunGarbageCollection_FinishedOnAnotherReplica341=== CONT TestParsePathInfoJSON/empty_input342=== CONT TestParsePathInfoJSON/Lix_format343=== CONT TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary344=== CONT TestPathInfoHashCompatibility/new_structured_format_with_sha5123452026/11/12 11:49:42 WARN Request returned retryable status, retrying attempt=4 backoff=1ms status=502 url=http://127.0.0.1:44939346=== CONT TestSetClientTLS/preserves_debug_logging_transport347=== CONT TestSetClientTLS/succeeds_with_client_cert_and_CA3482026/11/12 11:49:42 INFO Garbage collection started349--- PASS: TestRateLimiterFeedback (0.09s)350 --- PASS: TestRateLimiterFeedback/400_does_not_enable_limiter (0.00s)351 --- PASS: TestRateLimiterFeedback/200_does_not_enable_limiter (0.00s)352 --- PASS: TestRateLimiterFeedback/503_enables_limiter (0.00s)353 --- PASS: TestRateLimiterFeedback/429_enables_limiter (0.00s)354=== CONT TestGetStorePathHash/basename_without_hyphen_should_error355=== CONT TestFilterOversizedClosures/all_closures_skipped356=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_nar_method3572026/11/12 11:49:42 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-small nar_size=1000 max_nar_size=50358=== CONT TestGetStorePathHash/hash_with_wrong_length_should_error3592026/11/12 11:49:42 WARN Request returned retryable status, retrying attempt=5 backoff=1ms status=502 url=http://127.0.0.1:44939360=== CONT TestPathInfoCACompatibility/new_structured_format_-_text361=== RUN TestPartSizeForNAR/small_stays_at_minimum362=== PAUSE TestPartSizeForNAR/small_stays_at_minimum363=== CONT TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)3642026/11/12 11:49:42 INFO Garbage collection progress phase=running_on_another_replica failed_uploads_deleted=0 old_closures_deleted=0 objects_marked=0 objects_deleted=0 objects_failed=0365=== CONT TestPathInfoCACompatibility/old_string_format_-_fixed_recursive366=== CONT TestSetClientTLS/rejects_connection_without_client_cert367=== CONT TestGetStorePathHash/hash_with_invalid_characters_should_error368=== CONT TestPathInfoHashCompatibility/old_string_format_with_colon369=== CONT TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI370=== CONT TestFilterOversizedClosures/no_limit_keeps_everything371--- PASS: TestEncodeNixBase32 (0.09s)372 --- PASS: TestEncodeNixBase32/empty_input (0.00s)373 --- PASS: TestEncodeNixBase32/test_string_hash (0.00s)374=== CONT TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped375=== CONT TestPathInfoCACompatibility/null_ca_field376=== CONT TestPathInfoCACompatibility/new_structured_format_-_nar_method377--- PASS: TestConvertHashToNix32 (0.09s)378 --- PASS: TestConvertHashToNix32/already_Nix32_format (0.00s)379 --- PASS: TestConvertHashToNix32/SRI_format_to_Nix32 (0.00s)380 --- PASS: TestConvertHashToNix32/invalid_format (0.00s)3812026/11/12 11:49:42 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb-vm-image nar_size=5000 max_nar_size=2000382--- PASS: TestStreamPushBatchesUnderLoad (0.01s)383 --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=false (0.00s)384 --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=true (0.00s)3852026/11/12 11:49:42 WARN Request returned retryable status, retrying attempt=6 backoff=1ms status=502 url=http://127.0.0.1:44939386--- PASS: TestUploadMultipart_FailedProducerIsNotCompleted (0.00s)387 --- PASS: TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part (0.00s)388 --- PASS: TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary (0.00s)3892026/11/12 11:49:42 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:33279/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:33279/api/uploads/complete390=== RUN TestPartSizeForNAR/80_GiB_fits_at_minimum3912026/11/12 11:49:42 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"392=== PAUSE TestPartSizeForNAR/80_GiB_fits_at_minimum3932026/11/12 11:49:42 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:33279/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:33279/api/uploads/complete394=== RUN TestPartSizeForNAR/115_GiB_needs_larger_parts3952026/11/12 11:49:42 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"396=== PAUSE TestPartSizeForNAR/115_GiB_needs_larger_parts3972026/11/12 11:49:42 WARN Garbage collection finished on another replica. Its statistics are not available here398=== RUN TestPartSizeForNAR/1_TiB399=== PAUSE TestPartSizeForNAR/1_TiB400=== RUN TestPartSizeForNAR/5_TiB_S3_max_object401=== PAUSE TestPartSizeForNAR/5_TiB_S3_max_object402=== RUN TestPartSizeForNAR/capped_at_5_GiB403=== PAUSE TestPartSizeForNAR/capped_at_5_GiB404=== CONT TestPartSizeForNAR/zero_stays_at_minimum405=== CONT TestPartSizeForNAR/1_TiB406--- PASS: TestUploadBuildLog_FileBodyReplayedOnRetry (0.01s)407=== CONT TestPartSizeForNAR/small_stays_at_minimum4082026/11/12 11:49:42 WARN Request returned retryable status, retrying attempt=7 backoff=1ms status=502 url=http://127.0.0.1:44939409=== CONT TestPathInfoCACompatibility/old_string_format_-_text410=== CONT TestPartSizeForNAR/capped_at_5_GiB411=== CONT TestPartSizeForNAR/5_TiB_S3_max_object412=== CONT TestPartSizeForNAR/80_GiB_fits_at_minimum413=== CONT TestPartSizeForNAR/115_GiB_needs_larger_parts414--- PASS: TestRunGarbageCollection_FinishedOnAnotherReplica (0.00s)415--- PASS: TestSetClientTLSErrors (0.10s)416 --- PASS: TestSetClientTLSErrors/missing_cert_file (0.00s)417 --- PASS: TestSetClientTLSErrors/missing_key_file (0.00s)418 --- PASS: TestSetClientTLSErrors/missing_ca_file (0.00s)419 --- PASS: TestSetClientTLSErrors/invalid_ca_file (0.00s)4202026/11/12 11:49:42 WARN Request returned retryable status, retrying attempt=8 backoff=1ms status=502 url=http://127.0.0.1:44939421--- PASS: TestGetStorePathHash (0.09s)422 --- PASS: TestGetStorePathHash/valid_store_path (0.00s)423 --- PASS: TestGetStorePathHash/basename_without_hyphen_should_error (0.00s)424 --- PASS: TestGetStorePathHash/hash_with_wrong_length_should_error (0.00s)425 --- PASS: TestGetStorePathHash/hash_with_invalid_characters_should_error (0.00s)4262026/11/12 11:49:42 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:33279/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:33279/api/uploads/complete427--- PASS: TestPartSizeForNAR (0.01s)428 --- PASS: TestPartSizeForNAR/zero_stays_at_minimum (0.00s)429 --- PASS: TestPartSizeForNAR/1_TiB (0.00s)430 --- PASS: TestPartSizeForNAR/small_stays_at_minimum (0.00s)431 --- PASS: TestPartSizeForNAR/capped_at_5_GiB (0.00s)432 --- PASS: TestPartSizeForNAR/5_TiB_S3_max_object (0.00s)433 --- PASS: TestPartSizeForNAR/115_GiB_needs_larger_parts (0.00s)434 --- PASS: TestPartSizeForNAR/80_GiB_fits_at_minimum (0.00s)4352026/11/12 11:49:42 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"436--- PASS: TestFilterOversizedClosures (0.01s)437 --- PASS: TestFilterOversizedClosures/all_closures_skipped (0.00s)438 --- PASS: TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped (0.00s)439 --- PASS: TestFilterOversizedClosures/no_limit_keeps_everything (0.00s)440--- PASS: TestPathInfoCACompatibility (0.10s)441 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_text (0.00s)442 --- PASS: TestPathInfoCACompatibility/old_string_format_-_fixed_recursive (0.00s)443 --- PASS: TestPathInfoCACompatibility/null_ca_field (0.00s)444 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_nar_method (0.00s)445 --- PASS: TestPathInfoCACompatibility/old_string_format_-_text (0.00s)446--- PASS: TestPathInfoHashCompatibility (0.09s)447 --- PASS: TestPathInfoHashCompatibility/new_structured_format_with_sha512 (0.00s)448 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI) (0.00s)449 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_colon (0.00s)450 --- PASS: TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI (0.00s)451--- PASS: TestParsePathInfoJSON (0.10s)452 --- PASS: TestParsePathInfoJSON/invalid_JSON (0.00s)453 --- PASS: TestParsePathInfoJSON/whitespace_only (0.00s)454 --- PASS: TestParsePathInfoJSON/Nix_format (0.00s)455 --- PASS: TestParsePathInfoJSON/empty_input (0.00s)456 --- PASS: TestParsePathInfoJSON/Lix_format (0.00s)457--- PASS: TestRegisterUploadedObject_BoundedAgainstSilentServer (0.11s)4582026/11/12 11:49:42 WARN Request returned retryable status, retrying attempt=9 backoff=1ms status=502 url=http://127.0.0.1:449394592026/11/12 11:49:42 http: TLS handshake error from 127.0.0.1:50048: remote error: tls: bad certificate4602026/11/12 11:49:42 WARN Request returned retryable status, retrying attempt=10 backoff=1ms status=502 url=http://127.0.0.1:449394612026/11/12 11:49:42 WARN Request returned retryable status, retrying attempt=11 backoff=1ms status=502 url=http://127.0.0.1:44939462--- PASS: TestSetClientTLS (0.10s)463 --- PASS: TestSetClientTLS/succeeds_with_client_cert_and_CA (0.01s)464 --- PASS: TestSetClientTLS/preserves_debug_logging_transport (0.01s)465 --- PASS: TestSetClientTLS/rejects_connection_without_client_cert (0.01s)4662026/11/12 11:49:42 WARN Request returned retryable status, retrying attempt=12 backoff=1ms status=502 url=http://127.0.0.1:449394672026/11/12 11:49:42 WARN Request returned retryable status, retrying attempt=13 backoff=1ms status=502 url=http://127.0.0.1:449394682026/11/12 11:49:42 WARN Request returned retryable status, retrying attempt=14 backoff=1ms status=502 url=http://127.0.0.1:44939469--- PASS: TestStreamPushRequestLine (0.12s)4702026/11/12 11:49:42 WARN Request returned retryable status, retrying attempt=15 backoff=1ms status=502 url=http://127.0.0.1:449394712026/11/12 11:49:42 WARN Request returned retryable status, retrying attempt=16 backoff=1ms status=502 url=http://127.0.0.1:449394722026/11/12 11:49:42 WARN Request returned retryable status, retrying attempt=17 backoff=1ms status=502 url=http://127.0.0.1:44939473--- PASS: TestRetryTimeoutOutlastsMaxRetries (0.13s)474--- PASS: TestDumpPathSingleFile (0.12s)475--- PASS: TestRegisterUploadedObjectReusesConnections (0.03s)476--- PASS: TestDumpPathWriterError (0.13s)477--- PASS: TestCaseHackSuffix (0.04s)478--- PASS: TestDumpPathMatchesNix (0.09s)479--- PASS: TestUploadMultipart_PartsInParallel (0.72s)480--- PASS: TestRateLimiterFeedback_400DoesNotCountAsSuccess (1.00s)481PASS482Running server tests...483The files belonging to this database system will be owned by user "nixbld".484This user must also own the server process.485486The database cluster will be initialized with locale "C".487The default database encoding has accordingly been set to "SQL_ASCII".488The default text search configuration will be set to "english".489490Data page checksums are enabled.491492creating directory /build/postgres153287528/data ... ok493creating subdirectories ... ok494selecting dynamic shared memory implementation ... posix495selecting default "max_connections" ... 100496selecting default "shared_buffers" ... 128MB497selecting default time zone ... UTC498creating configuration files ... ok499running bootstrap script ... ok500performing post-bootstrap initialization ... ok501syncing data to disk ... ok502503initdb: warning: enabling "trust" authentication for local connections504initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.505506Success. You can now start the database server using:507508 pg_ctl -D /build/postgres153287528/data -l logfile start509510/build/postgres153287528:5432 - no response5112026-11-12 11:49:44.083 UTC [129] LOG: starting PostgreSQL 18.6 on x86_64-pc-linux-gnu, compiled by clang version 21.1.8, 64-bit5122026-11-12 11:49:44.083 UTC [129] LOG: listening on Unix socket "/build/postgres153287528/.s.PGSQL.5432"5132026-11-12 11:49:44.088 UTC [136] LOG: database system was shut down at 2026-11-12 11:49:43 UTC5142026-11-12 11:49:44.091 UTC [129] LOG: database system is ready to accept connections515/build/postgres153287528:5432 - accepting connections516{"timestamp":"2026-11-12T11:49:44.385135018Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"38759b9f-a57e-465e-baa6-9104be90ec65","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"GET","uri":"/health/ready","status_code":503,"suppressed_errors":0,"duration_ms":0,"result":"server_error","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":502,"threadName":"rustfs-worker","threadId":"ThreadId(386)"}517=== RUN TestAbortSucceeded518=== PAUSE TestAbortSucceeded519=== RUN TestService_AuthMiddleware520=== PAUSE TestService_AuthMiddleware521=== RUN TestService_AuthMiddleware_MTLSProxyHeader522=== PAUSE TestService_AuthMiddleware_MTLSProxyHeader523=== RUN TestService_AuthMiddleware_MTLSBoundSubjects524=== PAUSE TestService_AuthMiddleware_MTLSBoundSubjects525=== RUN TestService_ReadAuthMiddleware526=== PAUSE TestService_ReadAuthMiddleware527=== RUN TestService_AuthMiddleware_OIDC528=== PAUSE TestService_AuthMiddleware_OIDC529=== RUN TestService_RequireScope_OIDC530=== PAUSE TestService_RequireScope_OIDC531=== RUN TestService_ReadScope_PublicByDefault532=== PAUSE TestService_ReadScope_PublicByDefault533=== RUN TestCacheConfigHandler534=== PAUSE TestCacheConfigHandler535=== RUN TestCacheStatsHandler536=== PAUSE TestCacheStatsHandler537=== RUN TestClientCADerivations538=== PAUSE TestClientCADerivations539=== RUN TestClientErrorHandling540=== PAUSE TestClientErrorHandling541=== RUN TestClientIntegration542=== PAUSE TestClientIntegration543=== RUN TestClientMultipleUploads544=== PAUSE TestClientMultipleUploads545=== RUN TestClientWithDependencies546=== PAUSE TestClientWithDependencies547=== RUN TestClientSharedPathCommittedMidPush548=== PAUSE TestClientSharedPathCommittedMidPush549=== RUN TestPinProtectsFromGC550=== PAUSE TestPinProtectsFromGC551=== RUN TestClientPushesUseOnePush552=== PAUSE TestClientPushesUseOnePush553=== RUN TestClientFallsBackToClosures554=== PAUSE TestClientFallsBackToClosures555=== RUN TestDeleteClosuresRacingPins556=== PAUSE TestDeleteClosuresRacingPins557=== RUN TestConcurrentCommitsSharingObjectsDoNotDeadlock558=== PAUSE TestConcurrentCommitsSharingObjectsDoNotDeadlock559=== RUN TestResolveDBConnectionString560=== PAUSE TestResolveDBConnectionString561=== RUN TestConnectWaitsForAPeerMigration562=== PAUSE TestConnectWaitsForAPeerMigration563=== RUN TestLeadElectsOneAndHandsOver564=== PAUSE TestLeadElectsOneAndHandsOver565=== RUN TestLeadIncumbentWinsAfterRestart5662026/11/12 11:49:44 INFO lead: acquired remote=192.0.2.1:12345672026/11/12 11:49:45 INFO lead: released remote=192.0.2.1:12345682026/11/12 11:49:45 INFO lead: acquired remote=192.0.2.1:12345692026/11/12 11:49:46 INFO lead: released remote=192.0.2.1:1234570--- PASS: TestLeadIncumbentWinsAfterRestart (1.90s)571=== RUN TestLeadUnannouncedStandbyYieldsToIncumbent572=== PAUSE TestLeadUnannouncedStandbyYieldsToIncumbent573=== RUN TestLeadIncumbentAnnouncesAtOnce574=== PAUSE TestLeadIncumbentAnnouncesAtOnce575=== RUN TestLeadEndsOnShutdown576=== PAUSE TestLeadEndsOnShutdown577=== RUN TestLeadEndsWhenItsConnectionHangs5782026/11/12 11:49:46 INFO lead: acquired remote=192.0.2.1:12345792026-11-12 11:49:47.900 UTC [575] FATAL: terminating connection due to administrator command5802026/11/12 11:49:48 INFO lead: acquired remote=192.0.2.1:12345812026/11/12 11:49:49 WARN lead: lock connection lost error="timeout: context deadline exceeded"5822026/11/12 11:49:49 INFO lead: released remote=192.0.2.1:12345832026/11/12 11:49:49 INFO lead: released remote=192.0.2.1:1234584--- PASS: TestLeadEndsWhenItsConnectionHangs (3.11s)585=== RUN TestGCAdvisoryLockBlocksConcurrentRun586=== PAUSE TestGCAdvisoryLockBlocksConcurrentRun587=== RUN TestGCBugBareHashReferences588=== PAUSE TestGCBugBareHashReferences589=== RUN TestGCMetrics590=== PAUSE TestGCMetrics591=== RUN TestPushDedupSurvivesConcurrentGC592=== PAUSE TestPushDedupSurvivesConcurrentGC593=== RUN TestDeduplicatedObjectsRecordedAsPending594=== PAUSE TestDeduplicatedObjectsRecordedAsPending595=== RUN TestGCSweepSkipsPendingObjects596=== PAUSE TestGCSweepSkipsPendingObjects597=== RUN TestTombstonedObjectOfferedWithoutWaiting598=== PAUSE TestTombstonedObjectOfferedWithoutWaiting599=== RUN TestGCSweepDeliversEachKeyOnce600=== PAUSE TestGCSweepDeliversEachKeyOnce601=== RUN TestCreatePendingClosureVerifyS3FailureReleasesConnection602=== PAUSE TestCreatePendingClosureVerifyS3FailureReleasesConnection603=== RUN TestForceGCDuringPushOffersSweptObject604=== PAUSE TestForceGCDuringPushOffersSweptObject605=== RUN TestFailedS3DeleteKeepsTombstone606=== PAUSE TestFailedS3DeleteKeepsTombstone607=== RUN TestSweepSkipsRowsLockedByPush608=== PAUSE TestSweepSkipsRowsLockedByPush609=== RUN TestRegisterCompletedObjectNeedsPendingRow610=== PAUSE TestRegisterCompletedObjectNeedsPendingRow611=== RUN TestGetExistingObjectsLocksRows612=== PAUSE TestGetExistingObjectsLocksRows613=== RUN TestSweepRechecksPendingAfterLocking614=== PAUSE TestSweepRechecksPendingAfterLocking615=== RUN TestPushWaitsForSweepHoldingRow616=== PAUSE TestPushWaitsForSweepHoldingRow617=== RUN TestCommitRacingPendingCleanupKeepsObjects618=== PAUSE TestCommitRacingPendingCleanupKeepsObjects619=== RUN TestGCEndsOnShutdown620=== PAUSE TestGCEndsOnShutdown621=== RUN TestGCTaskStore_StartNew622=== PAUSE TestGCTaskStore_StartNew623=== RUN TestGCTaskStore_DeduplicateSameParams624=== PAUSE TestGCTaskStore_DeduplicateSameParams625=== RUN TestGCTaskStore_ConflictDifferentParams626=== PAUSE TestGCTaskStore_ConflictDifferentParams627=== RUN TestGCTaskStore_GetEmpty628=== PAUSE TestGCTaskStore_GetEmpty629=== RUN TestGCTaskStore_GetReturnsLatest630=== PAUSE TestGCTaskStore_GetReturnsLatest631=== RUN TestGCTaskStore_CompletedAllowsNewTask632=== PAUSE TestGCTaskStore_CompletedAllowsNewTask633=== RUN TestGCTaskStore_PhaseUpdates634=== PAUSE TestGCTaskStore_PhaseUpdates635=== RUN TestGCTaskStore_Fail636=== PAUSE TestGCTaskStore_Fail637=== RUN TestGracefulShutdownDrainsInflight638=== PAUSE TestGracefulShutdownDrainsInflight639=== RUN TestService_healthCheckHandler640=== PAUSE TestService_healthCheckHandler641=== RUN TestService_readinessHandler642=== PAUSE TestService_readinessHandler643=== RUN TestGenerateLandingPage644=== PAUSE TestGenerateLandingPage645=== RUN TestEnsureAbortUploadsRule646=== PAUSE TestEnsureAbortUploadsRule647=== RUN TestCacheConfigHandlerMaxNarSize648=== PAUSE TestCacheConfigHandlerMaxNarSize649=== RUN TestCreatePendingClosureRejectsOversizedNAR650=== PAUSE TestCreatePendingClosureRejectsOversizedNAR651=== RUN TestNARDeduplicationMetadataUploadBug652=== PAUSE TestNARDeduplicationMetadataUploadBug653=== RUN TestMetricsInventory654=== PAUSE TestMetricsInventory655=== RUN TestMetricsCommit656=== PAUSE TestMetricsCommit657=== RUN TestService_NativeMTLS658=== PAUSE TestService_NativeMTLS659=== RUN TestServerTLSConfig660=== PAUSE TestServerTLSConfig661=== RUN TestMultipartCleanup662=== PAUSE TestMultipartCleanup663=== RUN TestMultipartUploadAbortedWhenCancelledBeforeRecorded664=== PAUSE TestMultipartUploadAbortedWhenCancelledBeforeRecorded665=== RUN TestPendingCleanupUsesOneCutoff666=== PAUSE TestPendingCleanupUsesOneCutoff667=== RUN TestObjectStatsTrigger668=== PAUSE TestObjectStatsTrigger669=== RUN TestObjectStatsUpdatedOncePerCommit670=== PAUSE TestObjectStatsUpdatedOncePerCommit671=== RUN TestOrphanedObjectsGC672=== PAUSE TestOrphanedObjectsGC673=== RUN TestOrphanedObjectsGCStressTest674=== PAUSE TestOrphanedObjectsGCStressTest675=== RUN TestCreatePin_ReservedPins676=== PAUSE TestCreatePin_ReservedPins677=== RUN TestPresentReportsOnlyClosureRoots678=== PAUSE TestPresentReportsOnlyClosureRoots679=== RUN TestPresentRefreshesOnlyStaleClosures680=== PAUSE TestPresentRefreshesOnlyStaleClosures681=== RUN TestPresentNotReportedWhileGCDeletesClosure682=== PAUSE TestPresentNotReportedWhileGCDeletesClosure683=== RUN TestParseSingleRange684=== PAUSE TestParseSingleRange685=== RUN TestProxyHeadersOnlyTrustedOnSocket6862026/11/12 11:49:49 INFO Starting HTTP server address=127.0.0.1:341056872026/11/12 11:49:49 INFO Starting HTTP server address=proxy.sock6882026/11/12 11:49:49 WARN mTLS auth: subject not in bound subjects subject="CN=someone"6892026/11/12 11:49:49 INFO Shutdown signal received, draining in-flight requests timeout=10s690--- PASS: TestProxyHeadersOnlyTrustedOnSocket (0.11s)691=== RUN TestIsValidCachePath692=== PAUSE TestIsValidCachePath693=== RUN TestReadProxyNarinfo694=== PAUSE TestReadProxyNarinfo695=== RUN TestReadProxyNarinfoAlreadyDecompressed696=== PAUSE TestReadProxyNarinfoAlreadyDecompressed697=== RUN TestReadProxyContentNegotiation698=== PAUSE TestReadProxyContentNegotiation699=== RUN TestReadProxyNarStreaming700=== PAUSE TestReadProxyNarStreaming701=== RUN TestReadProxy404702=== PAUSE TestReadProxy404703=== RUN TestReadProxyInvalidPath704=== PAUSE TestReadProxyInvalidPath705=== RUN TestReadProxyHead706=== PAUSE TestReadProxyHead707=== RUN TestReadProxyOutlastsServerWriteTimeout708=== PAUSE TestReadProxyOutlastsServerWriteTimeout709=== RUN TestReadProxyConditionalGet710=== PAUSE TestReadProxyConditionalGet711=== RUN TestReadProxyRootRedirectsToIndexHTML712=== PAUSE TestReadProxyRootRedirectsToIndexHTML713=== RUN TestReadProxyDisabled714=== PAUSE TestReadProxyDisabled715=== RUN TestReadRedirectNar716=== PAUSE TestReadRedirectNar717=== RUN TestReadRedirectKeepsNarinfoProxied718=== PAUSE TestReadRedirectKeepsNarinfoProxied719=== RUN TestReadProxyRangeRequest720=== PAUSE TestReadProxyRangeRequest721=== RUN TestReadRedirectUsesPublicS3URL722=== PAUSE TestReadRedirectUsesPublicS3URL723=== RUN TestPush_OverlappingRootsStoreOneRowPerKey724=== PAUSE TestPush_OverlappingRootsStoreOneRowPerKey725=== RUN TestPush_CompleteCommitsEveryRoot726=== PAUSE TestPush_CompleteCommitsEveryRoot727=== RUN TestPush_RepeatedRootCommitsOnce728=== PAUSE TestPush_RepeatedRootCommitsOnce729=== RUN TestPush_CompleteWaitsForCleanupOfThePush730=== PAUSE TestPush_CompleteWaitsForCleanupOfThePush731=== RUN TestPush_RootsAreLockedInByteOrder732=== PAUSE TestPush_RootsAreLockedInByteOrder733=== RUN TestPush_SkippedKeySurvivesGCBeforeCommit734=== PAUSE TestPush_SkippedKeySurvivesGCBeforeCommit735=== RUN TestPush_RejectsBadRequests736=== PAUSE TestPush_RejectsBadRequests737=== RUN TestPush_SignsNarinfosOfItsPendingObjects738=== PAUSE TestPush_SignsNarinfosOfItsPendingObjects739=== RUN TestCommitCanBeRepeatedUntilCleanup740=== PAUSE TestCommitCanBeRepeatedUntilCleanup741=== RUN TestSharedNarBothUploadsComplete742=== PAUSE TestSharedNarBothUploadsComplete743=== RUN TestCompleteMultipartUpload_ErrorButObjectExists744=== PAUSE TestCompleteMultipartUpload_ErrorButObjectExists745=== RUN TestCompletedNarNotReofferedAcrossClosures746=== PAUSE TestCompletedNarNotReofferedAcrossClosures747=== RUN TestPresignedUploadRegisteredBeforeCommit748=== PAUSE TestPresignedUploadRegisteredBeforeCommit749=== RUN TestService_Rustfstest750=== PAUSE TestService_Rustfstest751=== RUN TestParseSize752=== PAUSE TestParseSize753=== RUN TestSkippedUploadsHandler754=== PAUSE TestSkippedUploadsHandler755=== RUN TestSystemdListenerNotActivated756--- PASS: TestSystemdListenerNotActivated (0.00s)757=== RUN TestWatchdogBeatsWhenHealthy758--- PASS: TestWatchdogBeatsWhenHealthy (0.02s)759=== RUN TestWatchdogSkipsWhenUnhealthy7602026/11/12 11:49:49 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7612026/11/12 11:49:49 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7622026/11/12 11:49:49 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7632026/11/12 11:49:49 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7642026/11/12 11:49:49 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7652026/11/12 11:49:49 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7662026/11/12 11:49:49 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7672026/11/12 11:49:49 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7682026/11/12 11:49:49 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7692026/11/12 11:49:49 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"770--- PASS: TestWatchdogSkipsWhenUnhealthy (0.20s)771=== RUN TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle772=== PAUSE TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle773=== RUN TestProxyWriteTimeout774=== PAUSE TestProxyWriteTimeout775=== RUN TestPendingClosureWriteTimeout776=== PAUSE TestPendingClosureWriteTimeout777=== RUN TestIsValidUploadKey778=== PAUSE TestIsValidUploadKey779=== RUN TestUploadHandlersRejectInvalidKeys780=== PAUSE TestUploadHandlersRejectInvalidKeys781=== RUN TestUploadHandlersRejectOversizedBody782=== PAUSE TestUploadHandlersRejectOversizedBody783=== RUN TestService_cleanupPendingClosuresHandler784=== PAUSE TestService_cleanupPendingClosuresHandler785=== RUN TestService_createPendingClosureHandler786=== PAUSE TestService_createPendingClosureHandler787=== RUN TestService_verifyS3Integrity788=== PAUSE TestService_verifyS3Integrity789=== RUN TestService_verifyS3ReuploadsStaleNarinfo790=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo791=== RUN TestCompleteMultipartUnregistered792=== PAUSE TestCompleteMultipartUnregistered793=== RUN TestCreatePendingClosure_SmallNARUsesSimplePUT794=== PAUSE TestCreatePendingClosure_SmallNARUsesSimplePUT795=== CONT TestPush_OverlappingRootsStoreOneRowPerKey796=== CONT TestAbortSucceeded797--- PASS: TestAbortSucceeded (0.00s)798=== CONT TestReadRedirectUsesPublicS3URL799=== CONT TestOrphanedObjectsGC800=== CONT TestReadProxyRangeRequest801=== CONT TestReadRedirectKeepsNarinfoProxied802=== CONT TestReadRedirectNar803=== CONT TestReadProxyDisabled804=== CONT TestReadProxyRootRedirectsToIndexHTML805=== CONT TestReadProxyConditionalGet806=== CONT TestReadProxyOutlastsServerWriteTimeout807=== CONT TestReadProxyHead808=== CONT TestReadProxyInvalidPath809=== CONT TestReadProxy404810=== CONT TestReadProxyNarStreaming811=== CONT TestReadProxyContentNegotiation812=== CONT TestReadProxyNarinfoAlreadyDecompressed813=== CONT TestReadProxyNarinfo814=== CONT TestIsValidCachePath815=== RUN TestIsValidCachePath/narinfo816=== PAUSE TestIsValidCachePath/narinfo817=== RUN TestIsValidCachePath/narinfo_all_nix_base32_chars818=== PAUSE TestIsValidCachePath/narinfo_all_nix_base32_chars819=== RUN TestIsValidCachePath/nar_zst820=== PAUSE TestIsValidCachePath/nar_zst821=== RUN TestIsValidCachePath/nar_xz822=== PAUSE TestIsValidCachePath/nar_xz823=== CONT TestParseSingleRange824=== CONT TestPresentNotReportedWhileGCDeletesClosure825=== CONT TestPresentRefreshesOnlyStaleClosures826=== CONT TestPresentReportsOnlyClosureRoots827=== CONT TestCreatePin_ReservedPins828=== CONT TestOrphanedObjectsGCStressTest829=== RUN TestIsValidCachePath/nar_bz2830=== PAUSE TestIsValidCachePath/nar_bz2831=== RUN TestParseSingleRange/none832=== PAUSE TestParseSingleRange/none833=== RUN TestParseSingleRange/unknown_unit834=== PAUSE TestParseSingleRange/unknown_unit835=== RUN TestParseSingleRange/multi-range_ignored836=== PAUSE TestParseSingleRange/multi-range_ignored837=== RUN TestParseSingleRange/malformed_no_dash838=== PAUSE TestParseSingleRange/malformed_no_dash839=== RUN TestParseSingleRange/malformed_both_empty840=== PAUSE TestParseSingleRange/malformed_both_empty841=== RUN TestParseSingleRange/malformed_end_before_start842=== PAUSE TestParseSingleRange/malformed_end_before_start843=== RUN TestParseSingleRange/closed844=== PAUSE TestParseSingleRange/closed845=== RUN TestParseSingleRange/open-ended846=== PAUSE TestParseSingleRange/open-ended847=== RUN TestParseSingleRange/end_clamped_to_size848=== RUN TestIsValidCachePath/nar_uncompressed849=== PAUSE TestIsValidCachePath/nar_uncompressed850=== RUN TestPresentNotReportedWhileGCDeletesClosure/stale851=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/stale852=== PAUSE TestParseSingleRange/end_clamped_to_size853=== RUN TestParseSingleRange/suffix854=== RUN TestIsValidCachePath/ls855=== PAUSE TestIsValidCachePath/ls856=== RUN TestIsValidCachePath/log857=== PAUSE TestIsValidCachePath/log858=== RUN TestIsValidCachePath/realisation859=== PAUSE TestIsValidCachePath/realisation860=== RUN TestPresentNotReportedWhileGCDeletesClosure/fresh861=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/fresh862=== PAUSE TestParseSingleRange/suffix863=== RUN TestIsValidCachePath/nix-cache-info864=== PAUSE TestIsValidCachePath/nix-cache-info865=== RUN TestIsValidCachePath/index.html866=== PAUSE TestIsValidCachePath/index.html867=== RUN TestIsValidCachePath/traversal_parent868=== PAUSE TestIsValidCachePath/traversal_parent869=== RUN TestIsValidCachePath/traversal_in_middle870=== PAUSE TestIsValidCachePath/traversal_in_middle871=== RUN TestIsValidCachePath/invalid_char_e872=== PAUSE TestIsValidCachePath/invalid_char_e873=== RUN TestIsValidCachePath/invalid_char_u874=== PAUSE TestIsValidCachePath/invalid_char_u875=== CONT TestObjectStatsUpdatedOncePerCommit876=== RUN TestParseSingleRange/suffix_exceeds_size877=== PAUSE TestParseSingleRange/suffix_exceeds_size878=== RUN TestParseSingleRange/single_byte879=== PAUSE TestParseSingleRange/single_byte880=== RUN TestParseSingleRange/start_past_EOF881=== PAUSE TestParseSingleRange/start_past_EOF882=== RUN TestParseSingleRange/start_far_past_EOF883=== PAUSE TestParseSingleRange/start_far_past_EOF884=== RUN TestIsValidCachePath/random_path885=== PAUSE TestIsValidCachePath/random_path886=== CONT TestObjectStatsTrigger887=== RUN TestIsValidCachePath/empty888=== PAUSE TestIsValidCachePath/empty889=== RUN TestIsValidCachePath/leading_slash890=== PAUSE TestIsValidCachePath/leading_slash891=== RUN TestIsValidCachePath/wrong_extension892=== PAUSE TestIsValidCachePath/wrong_extension893=== RUN TestIsValidCachePath/short_hash894=== PAUSE TestIsValidCachePath/short_hash895=== CONT TestPendingCleanupUsesOneCutoff8962026/11/12 11:49:49 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:38097/oidc897--- PASS: TestReadRedirectUsesPublicS3URL (0.45s)898=== CONT TestMultipartUploadAbortedWhenCancelledBeforeRecorded899--- PASS: TestReadRedirectNar (0.46s)900=== CONT TestMultipartCleanup901--- PASS: TestReadProxyRootRedirectsToIndexHTML (0.48s)902=== CONT TestServerTLSConfig903=== RUN TestServerTLSConfig/no_client_CA904=== PAUSE TestServerTLSConfig/no_client_CA905=== RUN TestServerTLSConfig/missing_CA_file906=== PAUSE TestServerTLSConfig/missing_CA_file907=== RUN TestServerTLSConfig/not_a_PEM_file908=== PAUSE TestServerTLSConfig/not_a_PEM_file909=== CONT TestService_NativeMTLS910--- PASS: TestReadProxyInvalidPath (0.50s)911=== CONT TestMetricsCommit912--- PASS: TestReadRedirectKeepsNarinfoProxied (0.54s)913=== CONT TestMetricsInventory9142026/11/12 11:49:50 INFO Received push request method=POST path=/api/pushes915--- PASS: TestReadProxyDisabled (0.58s)916=== CONT TestNARDeduplicationMetadataUploadBug917--- PASS: TestPush_OverlappingRootsStoreOneRowPerKey (0.60s)918=== CONT TestCreatePendingClosureRejectsOversizedNAR9192026/11/12 11:49:50 INFO Received uploads request method=POST path=/api/pending_closures920--- PASS: TestCreatePendingClosureRejectsOversizedNAR (0.00s)921=== CONT TestGCMetrics922--- PASS: TestReadProxyRangeRequest (0.62s)923=== CONT TestGCBugBareHashReferences924--- PASS: TestPresentRefreshesOnlyStaleClosures (0.73s)925=== CONT TestGCAdvisoryLockBlocksConcurrentRun926--- PASS: TestReadProxyContentNegotiation (0.74s)927=== CONT TestLeadEndsOnShutdown928--- PASS: TestReadProxy404 (0.77s)929=== CONT TestLeadIncumbentAnnouncesAtOnce930--- PASS: TestPresentReportsOnlyClosureRoots (0.80s)931=== CONT TestLeadUnannouncedStandbyYieldsToIncumbent932--- PASS: TestReadProxyHead (0.83s)933=== CONT TestLeadElectsOneAndHandsOver934--- PASS: TestObjectStatsTrigger (0.86s)935=== CONT TestConnectWaitsForAPeerMigration9362026/11/12 11:49:50 INFO Received uploads request method=POST path=/api/pending_closures9372026/11/12 11:49:50 INFO Received uploads request method=POST path=/api/pending_closures9382026/11/12 11:49:50 INFO Received cleanup request method=DELETE path=/api/pending_closures939--- PASS: TestReadProxyNarinfoAlreadyDecompressed (0.98s)940=== CONT TestResolveDBConnectionString941=== RUN TestResolveDBConnectionString/flag_wins942=== PAUSE TestResolveDBConnectionString/flag_wins943=== RUN TestResolveDBConnectionString/file_when_flag_empty944=== PAUSE TestResolveDBConnectionString/file_when_flag_empty945=== RUN TestResolveDBConnectionString/missing_file_is_an_error946=== PAUSE TestResolveDBConnectionString/missing_file_is_an_error947=== RUN TestResolveDBConnectionString/PGHOST_allows_empty948=== PAUSE TestResolveDBConnectionString/PGHOST_allows_empty949=== RUN TestResolveDBConnectionString/nothing_configured950=== PAUSE TestResolveDBConnectionString/nothing_configured951=== CONT TestConcurrentCommitsSharingObjectsDoNotDeadlock9522026/11/12 11:49:50 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux9532026/11/12 11:49:50 WARN Refused reserved pin name=worker-x86_64-linux9542026/11/12 11:49:50 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux9552026/11/12 11:49:50 INFO Received create pin request method=POST path=/api/pins/my-app9562026/11/12 11:49:50 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux957--- PASS: TestCreatePin_ReservedPins (1.01s)958=== CONT TestDeleteClosuresRacingPins959--- PASS: TestReadProxyNarStreaming (1.07s)960=== CONT TestClientFallsBackToClosures9612026/11/12 11:49:50 ERROR Failed to decode narinfo key=3hcdxyjf9yiq7qf3i4548drb6sjmwa1v.narinfo limit=16777216 error=<nil>962--- PASS: TestReadProxyNarinfo (1.12s)963=== CONT TestClientPushesUseOnePush964--- PASS: TestReadProxyConditionalGet (1.14s)965=== CONT TestPinProtectsFromGC966--- PASS: TestObjectStatsUpdatedOncePerCommit (1.15s)967=== CONT TestClientSharedPathCommittedMidPush9682026/11/12 11:49:51 INFO Received uploads request method=POST path=/api/pending_closures969=== NAME TestOrphanedObjectsGC970 orphaned_objects_gc_test.go:287: GC Test Summary:971 orphaned_objects_gc_test.go:288: - Kept: 2 objects from closure A972 orphaned_objects_gc_test.go:289: - Deleted: 2 objects from closure B973 orphaned_objects_gc_test.go:290: - Deleted: 6 orphaned chain objects (X1->X2->X3)974 orphaned_objects_gc_test.go:291: - Deleted: 2 orphaned single objects (Y)975 orphaned_objects_gc_test.go:292: - Total deleted: 10 objects976--- PASS: TestOrphanedObjectsGC (1.17s)977=== CONT TestPushDedupSurvivesConcurrentGC978--- PASS: TestMultipartUploadAbortedWhenCancelledBeforeRecorded (0.73s)979=== CONT TestClientWithDependencies9802026/11/12 11:49:51 INFO Received uploads request method=POST path=/api/pending_closures9812026/11/12 11:49:51 WARN mTLS auth: subject not in bound subjects subject="CN=reader"9822026/11/12 11:49:51 WARN mTLS auth: subject not in bound subjects subject="CN=reader"983--- PASS: TestService_NativeMTLS (0.74s)984=== CONT TestClientMultipleUploads9852026/11/12 11:49:51 INFO Received push request method=POST path=/api/pushes9862026/11/12 11:49:51 INFO Received complete push request method=POST path=/api/pushes/1/complete9872026/11/12 11:49:51 INFO Completed push id=1 objects=4 duration=10.87815ms9882026/11/12 11:49:51 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete9892026/11/12 11:49:51 INFO Completed upload id=2 objects=2 duration=1.67019ms9902026/11/12 11:49:51 INFO Received cleanup request method=DELETE path=/api/pending_closures991--- PASS: TestMetricsCommit (0.81s)992=== CONT TestCacheConfigHandlerMaxNarSize993--- PASS: TestCacheConfigHandlerMaxNarSize (0.00s)994=== CONT TestClientIntegration9952026/11/12 11:49:51 WARN Failed to abort upload, keeping its closure key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst error="Get \"http://127.0.0.1:1/bucket29/?location=\": dial tcp 127.0.0.1:1: connect: connection refused" code=""9962026/11/12 11:49:51 INFO Aborted multipart uploads count=0 kept=1997--- PASS: TestMetricsInventory (0.77s)998=== CONT TestEnsureAbortUploadsRule9992026/11/12 11:49:51 INFO Received cleanup request method=DELETE path=/api/pending_closures10002026/11/12 11:49:51 INFO Aborted multipart uploads count=1 kept=01001--- PASS: TestMultipartCleanup (0.87s)1002=== CONT TestClientErrorHandling1003=== RUN TestClientErrorHandling/InvalidStorePath1004=== PAUSE TestClientErrorHandling/InvalidStorePath1005=== RUN TestClientErrorHandling/InvalidAuthToken1006=== PAUSE TestClientErrorHandling/InvalidAuthToken1007=== RUN TestClientErrorHandling/ServerNotAvailable1008=== PAUSE TestClientErrorHandling/ServerNotAvailable1009=== CONT TestGenerateLandingPage1010--- PASS: TestGenerateLandingPage (0.01s)1011=== CONT TestClientCADerivations10122026/11/12 11:49:51 INFO Aborted multipart uploads count=0 kept=010132026/11/12 11:49:51 WARN Force mode enabled - objects will be deleted immediately without grace period1014=== NAME TestNARDeduplicationMetadataUploadBug1015 metadata_upload_test.go:48: First store path: /build/TestNARDeduplicationMetadataUploadBug3377268787/001/store/s1zzy9qs3gyzfrmbnbzvva9j76nmgz9w-file1.txt10162026/11/12 11:49:51 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=010172026/11/12 11:49:51 INFO Vacuumed table table=pending_closures10182026/11/12 11:49:51 INFO Vacuumed table table=commit_receipts10192026/11/12 11:49:51 INFO Vacuumed table table=pending_objects10202026/11/12 11:49:51 INFO Vacuumed table table=multipart_uploads10212026/11/12 11:49:51 INFO Vacuumed table table=closures10222026/11/12 11:49:51 INFO Vacuumed table table=objects1023--- PASS: TestGCMetrics (0.78s)1024=== CONT TestService_readinessHandler10252026/11/12 11:49:51 ERROR failed to check GC advisory lock error="query pg_locks: failed to connect to `user=nixbld database=niks3`: /nonexistent/.s.PGSQL.5432 (/nonexistent): dial error: dial unix /nonexistent/.s.PGSQL.5432: connect: no such file or directory"1026--- PASS: TestGCAdvisoryLockBlocksConcurrentRun (0.68s)1027=== CONT TestService_healthCheckHandler10282026/11/12 11:49:51 INFO lead: acquired remote=192.0.2.1:123410292026/11/12 11:49:51 INFO Received push request method=POST path=/api/pushes10302026/11/12 11:49:51 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)10312026/11/12 11:49:51 INFO Uploading s1zzy9qs3gyzfrmbnbzvva9j76nmgz9w-file1.txt (160B)10322026/11/12 11:49:51 WARN Failed to register uploaded object key=s1zzy9qs3gyzfrmbnbzvva9j76nmgz9w.ls error="server returned 404: 404 page not found\n"10332026/11/12 11:49:51 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign10342026/11/12 11:49:51 WARN Failed to register uploaded object key=nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst error="server returned 404: 404 page not found\n"10352026/11/12 11:49:51 INFO Signed narinfos id=1 count=110362026/11/12 11:49:51 INFO Uploading 1 narinfos10372026/11/12 11:49:51 INFO Received complete push request method=POST path=/api/pushes/1/complete10382026/11/12 11:49:51 WARN Failed to register uploaded object key=s1zzy9qs3gyzfrmbnbzvva9j76nmgz9w.narinfo error="server returned 404: 404 page not found\n"10392026/11/12 11:49:51 INFO lead: acquired remote=192.0.2.1:123410402026/11/12 11:49:51 INFO lead: released remote=192.0.2.1:12341041--- PASS: TestLeadIncumbentAnnouncesAtOnce (0.71s)1042=== CONT TestCacheStatsHandler10432026/11/12 11:49:51 INFO Completed push id=1 objects=3 duration=11.32059ms10442026/11/12 11:49:51 INFO Upload complete. (79ms)1045=== NAME TestNARDeduplicationMetadataUploadBug1046 metadata_upload_test.go:54: Retrieved narinfo from S3:1047 StorePath: /build/TestNARDeduplicationMetadataUploadBug3377268787/001/store/s1zzy9qs3gyzfrmbnbzvva9j76nmgz9w-file1.txt1048 URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst1049 Compression: zstd1050 NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1051 NarSize: 1601052 References: 1053 CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1054 metadata_upload_test.go:55: Retrieved .ls file from S3 (compressed size: 77 bytes)1055 metadata_upload_test.go:55: Decompressed .ls content (64 bytes):1056 {"version":1,"root":{"type":"regular","size":44,"narOffset":96}}1057 metadata_upload_test.go:64: Second store path (same content): /build/TestNARDeduplicationMetadataUploadBug3377268787/001/store/7m676z7y1dyk7mcsk53skiina50fr7gs-file2.txt10582026/11/12 11:49:51 INFO lead: acquired remote=192.0.2.1:123410592026/11/12 11:49:51 INFO lead: acquired remote=192.0.2.1:123410602026/11/12 11:49:51 INFO Received push request method=POST path=/api/pushes10612026/11/12 11:49:51 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)10622026/11/12 11:49:51 INFO Received sign narinfos request method=POST path=/api/pushes/2/sign10632026/11/12 11:49:51 WARN Failed to register uploaded object key=7m676z7y1dyk7mcsk53skiina50fr7gs.ls error="server returned 404: 404 page not found\n"10642026/11/12 11:49:51 INFO Signed narinfos id=2 count=110652026/11/12 11:49:51 INFO Uploading 1 narinfos10662026/11/12 11:49:51 INFO Received complete push request method=POST path=/api/pushes/2/complete10672026/11/12 11:49:51 WARN Failed to register uploaded object key=7m676z7y1dyk7mcsk53skiina50fr7gs.narinfo error="server returned 404: 404 page not found\n"10682026/11/12 11:49:51 INFO Completed push id=2 objects=3 duration=3.00343ms10692026/11/12 11:49:51 INFO Upload complete. (53ms)1070 metadata_upload_test.go:76: Retrieved narinfo from S3:1071 StorePath: /build/TestNARDeduplicationMetadataUploadBug3377268787/001/store/7m676z7y1dyk7mcsk53skiina50fr7gs-file2.txt1072 URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst1073 Compression: zstd1074 NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1075 NarSize: 1601076 References: 1077 CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1078 metadata_upload_test.go:77: Retrieved .ls file from S3 (compressed size: 62 bytes)1079 metadata_upload_test.go:77: Decompressed .ls content (49 bytes):1080 {"version":1,"root":{"type":"regular","size":44}}1081--- PASS: TestNARDeduplicationMetadataUploadBug (1.04s)1082=== CONT TestGracefulShutdownDrainsInflight10832026/11/12 11:49:51 INFO Starting HTTP server address=127.0.0.1:3813710842026/11/12 11:49:51 INFO Shutdown signal received, draining in-flight requests timeout=10s1085--- PASS: TestGCBugBareHashReferences (1.00s)1086=== CONT TestGCTaskStore_Fail1087--- PASS: TestGCTaskStore_Fail (0.00s)1088=== CONT TestCacheConfigHandler1089=== RUN TestCacheConfigHandler/full_config,_no_issuer1090=== PAUSE TestCacheConfigHandler/full_config,_no_issuer1091=== RUN TestCacheConfigHandler/no_cache_url_configured1092=== PAUSE TestCacheConfigHandler/no_cache_url_configured1093=== RUN TestCacheConfigHandler/no_signing_keys1094=== PAUSE TestCacheConfigHandler/no_signing_keys1095=== RUN TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1096=== PAUSE TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1097=== CONT TestGCTaskStore_PhaseUpdates1098--- PASS: TestGCTaskStore_PhaseUpdates (0.00s)1099=== CONT TestService_ReadScope_PublicByDefault1100--- PASS: TestGracefulShutdownDrainsInflight (0.07s)1101=== CONT TestGCTaskStore_CompletedAllowsNewTask1102--- PASS: TestGCTaskStore_CompletedAllowsNewTask (0.00s)1103=== CONT TestService_RequireScope_OIDC11042026/11/12 11:49:51 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:40919/oidc11052026/11/12 11:49:51 INFO Received uploads request method=POST path=/api/pending_closures1106--- PASS: TestConcurrentCommitsSharingObjectsDoNotDeadlock (0.78s)1107=== CONT TestGCTaskStore_GetReturnsLatest1108--- PASS: TestGCTaskStore_GetReturnsLatest (0.00s)1109=== CONT TestService_AuthMiddleware_OIDC1110=== NAME TestPinProtectsFromGC1111 client_integration_test.go:791: Pinned store path: /build/TestPinProtectsFromGC3417453621/001/store/5ifxh1v74vyxar6ykq92kfj02la5hvs4-pinned-file.txt1112 client_integration_test.go:792: Unpinned store path: /build/TestPinProtectsFromGC3417453621/001/store/j87s9vjygqyy4aygms43yd7lq8snnpz2-unpinned-file.txt11132026/11/12 11:49:51 INFO Aborted multipart uploads count=0 kept=011142026/11/12 11:49:51 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=011152026/11/12 11:49:51 INFO Vacuumed table table=pending_closures11162026/11/12 11:49:51 INFO Vacuumed table table=commit_receipts11172026/11/12 11:49:51 INFO Vacuumed table table=pending_objects11182026/11/12 11:49:51 INFO Vacuumed table table=multipart_uploads11192026/11/12 11:49:51 INFO Vacuumed table table=closures11202026/11/12 11:49:51 INFO Vacuumed table table=objects11212026/11/12 11:49:51 INFO Aborted multipart uploads count=0 kept=011222026/11/12 11:49:51 WARN Force mode enabled - objects will be deleted immediately without grace period11232026/11/12 11:49:51 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=011242026/11/12 11:49:51 INFO Vacuumed table table=pending_closures11252026/11/12 11:49:51 INFO Received push request method=POST path=/api/pushes11262026/11/12 11:49:51 INFO Vacuumed table table=commit_receipts11272026/11/12 11:49:51 INFO Vacuumed table table=pending_objects11282026/11/12 11:49:51 INFO Vacuumed table table=multipart_uploads1129=== NAME TestClientMultipleUploads1130 client_integration_test.go:406: Created store path 0: /build/TestClientMultipleUploads1859048027/001/store/9lf8bj3drwfpzkz6f82ddgzpgpgjfxwl-test-file-0.txt11312026/11/12 11:49:51 INFO Vacuumed table table=closures11322026/11/12 11:49:51 INFO Vacuumed table table=objects11332026/11/12 11:49:51 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)11342026/11/12 11:49:51 INFO Uploading 5ifxh1v74vyxar6ykq92kfj02la5hvs4-pinned-file.txt (128B)11352026/11/12 11:49:51 INFO Received push request method=POST path=/api/pushes1136--- PASS: TestPushDedupSurvivesConcurrentGC (0.68s)1137=== CONT TestGCTaskStore_GetEmpty1138--- PASS: TestGCTaskStore_GetEmpty (0.00s)1139=== CONT TestService_ReadAuthMiddleware11402026/11/12 11:49:51 INFO Uploading 2 paths to 127.0.0.1 (1 already cached)11412026/11/12 11:49:51 INFO Uploading 5dqv4hbc10f0i575zxl08g5l162bj90d-a (208B)11422026/11/12 11:49:51 INFO Uploading z7f8pmwyss69xc1di0a5r5k8bvryg42r-shared-dep (136B)11432026/11/12 11:49:51 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign11442026/11/12 11:49:51 INFO Signed narinfos id=1 count=111452026/11/12 11:49:51 WARN Failed to register uploaded object key=nar/0qmhsg42pqsz7x6hh02j8gznnac3yi3qpjhrqbwvz88i194jbb5s.nar.zst error="server returned 404: 404 page not found\n"1146=== NAME TestClientWithDependencies11472026/11/12 11:49:51 INFO Uploading 1 narinfos1148 client_integration_test.go:661: Built derivation: /build/TestClientWithDependencies50158575/001/store/1j36gkgkf3z34zbkiai82sg26kip2x5q-test-script11492026/11/12 11:49:51 WARN Failed to register uploaded object key=5ifxh1v74vyxar6ykq92kfj02la5hvs4.ls error="server returned 404: 404 page not found\n"11502026/11/12 11:49:51 INFO Received uploads request method=POST path=/api/pending_closures11512026/11/12 11:49:51 WARN Failed to register uploaded object key=57kzpa1mdpv0ladmy6rv6ss8m4ni4b8n.ls error="server returned 404: 404 page not found\n"11522026/11/12 11:49:51 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"11532026/11/12 11:49:51 WARN Failed to register uploaded object key=nar/1l0x6mmn5bzxidwmi38pvmb20xgypkgnj0qz5ysf170b817fnlh1.nar.zst error="server returned 404: 404 page not found\n"11542026/11/12 11:49:51 INFO Received uploads request method=POST path=/api/pending_closures11552026/11/12 11:49:51 INFO Uploading 2 paths to 127.0.0.1 (1 already cached)11562026/11/12 11:49:51 INFO Uploading dva0xi2qxansgr26w5pxgs9ry4pdn86p-shared-dep (136B)11572026/11/12 11:49:51 INFO Uploading sha2vgmvl28qz67l9k2kpcclx6j8n7jc-a (216B)11582026/11/12 11:49:51 WARN Failed to register uploaded object key=z7f8pmwyss69xc1di0a5r5k8bvryg42r.ls error="server returned 404: 404 page not found\n"11592026/11/12 11:49:51 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign11602026/11/12 11:49:51 WARN Failed to register uploaded object key=5dqv4hbc10f0i575zxl08g5l162bj90d.ls error="server returned 404: 404 page not found\n"11612026/11/12 11:49:51 INFO Installed lifecycle rule aborting incomplete NAR uploads bucket=bucket51 days=211622026/11/12 11:49:51 INFO Received complete push request method=POST path=/api/pushes/1/complete11632026/11/12 11:49:51 WARN Failed to register uploaded object key=5ifxh1v74vyxar6ykq92kfj02la5hvs4.narinfo error="server returned 404: 404 page not found\n"11642026/11/12 11:49:51 INFO Signed narinfos id=1 count=311652026/11/12 11:49:51 INFO Uploading 3 narinfos1166--- PASS: TestEnsureAbortUploadsRule (0.56s)1167=== CONT TestGCTaskStore_ConflictDifferentParams1168--- PASS: TestGCTaskStore_ConflictDifferentParams (0.00s)1169=== CONT TestService_AuthMiddleware_MTLSBoundSubjects1170=== NAME TestClientIntegration1171 client_integration_test.go:292: Created store path: /build/TestClientIntegration3822821240/002/store/08wc6q11rfp1smc6vq17fryhcqgci102-test-file.txt11722026/11/12 11:49:51 WARN Failed to register uploaded object key=pc58r1pkbl24768nby0m5l5sa187vf39.ls error="server returned 404: 404 page not found\n"11732026/11/12 11:49:51 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"11742026/11/12 11:49:51 WARN Failed to register uploaded object key=dva0xi2qxansgr26w5pxgs9ry4pdn86p.ls error="server returned 404: 404 page not found\n"11752026/11/12 11:49:51 WARN Failed to register uploaded object key=5dqv4hbc10f0i575zxl08g5l162bj90d.narinfo error="server returned 404: 404 page not found\n"11762026/11/12 11:49:51 WARN Failed to register uploaded object key=sha2vgmvl28qz67l9k2kpcclx6j8n7jc.ls error="server returned 404: 404 page not found\n"11772026/11/12 11:49:51 WARN Failed to register uploaded object key=57kzpa1mdpv0ladmy6rv6ss8m4ni4b8n.narinfo error="server returned 404: 404 page not found\n"11782026/11/12 11:49:51 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign11792026/11/12 11:49:51 WARN Failed to register uploaded object key=nar/0v5s7crg3hf89yjww565nrp6v5rbvalwgprdnq9xfpi2b5ihxbz0.nar.zst error="server returned 404: 404 page not found\n"11802026/11/12 11:49:51 INFO Received complete push request method=POST path=/api/pushes/1/complete11812026/11/12 11:49:51 WARN Failed to register uploaded object key=z7f8pmwyss69xc1di0a5r5k8bvryg42r.narinfo error="server returned 404: 404 page not found\n"11822026/11/12 11:49:51 INFO Signed narinfos id=1 count=211832026/11/12 11:49:51 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign11842026/11/12 11:49:51 INFO Signed narinfos id=2 count=211852026/11/12 11:49:51 INFO Uploading 4 narinfos11862026/11/12 11:49:51 INFO Completed push id=1 objects=3 duration=9.78175ms11872026/11/12 11:49:51 INFO Upload complete. (74ms)1188=== NAME TestClientMultipleUploads1189 client_integration_test.go:406: Created store path 1: /build/TestClientMultipleUploads1859048027/001/store/3h49annsbcv8ag9vjr2rsg7c5cv51znc-test-file-1.txt11902026/11/12 11:49:51 INFO Completed push id=1 objects=8 duration=5.48124ms11912026/11/12 11:49:51 INFO Upload complete. (72ms)11922026/11/12 11:49:51 WARN Failed to register uploaded object key=dva0xi2qxansgr26w5pxgs9ry4pdn86p.narinfo error="server returned 404: 404 page not found\n"11932026/11/12 11:49:51 WARN Failed to register uploaded object key=pc58r1pkbl24768nby0m5l5sa187vf39.narinfo error="server returned 404: 404 page not found\n"1194=== NAME TestClientPushesUseOnePush1195 client_pushes_test.go:97: Retrieved narinfo from S3:1196 StorePath: /build/TestClientPushesUseOnePush191403387/001/store/z7f8pmwyss69xc1di0a5r5k8bvryg42r-shared-dep11972026/11/12 11:49:51 WARN Failed to register uploaded object key=sha2vgmvl28qz67l9k2kpcclx6j8n7jc.narinfo error="server returned 404: 404 page not found\n"1198 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1199 Compression: zstd1200 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821201 NarSize: 1361202 References: 1203 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n1204=== NAME TestClientWithDependencies1205 client_integration_test.go:663: Found 1 dependencies (including self)1206=== NAME TestClientPushesUseOnePush1207 client_pushes_test.go:97: Retrieved narinfo from S3:1208 StorePath: /build/TestClientPushesUseOnePush191403387/001/store/5dqv4hbc10f0i575zxl08g5l162bj90d-a1209 URL: nar/1l0x6mmn5bzxidwmi38pvmb20xgypkgnj0qz5ysf170b817fnlh1.nar.zst1210 Compression: zstd1211 NarHash: sha256:1l0x6mmn5bzxidwmi38pvmb20xgypkgnj0qz5ysf170b817fnlh11212 NarSize: 2081213 References: /build/TestClientPushesUseOnePush191403387/001/store/z7f8pmwyss69xc1di0a5r5k8bvryg42r-shared-dep1214 CA: text:sha256:1w1i2s4nhpn45wwvy7zaj1phiqr6f4v6nn2bkiigxc36hyd1pp2n12152026/11/12 11:49:51 INFO Received push request method=POST path=/api/pushes12162026/11/12 11:49:51 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete12172026/11/12 11:49:51 WARN Failed to register uploaded object key=dva0xi2qxansgr26w5pxgs9ry4pdn86p.narinfo error="server returned 404: 404 page not found\n"1218 client_pushes_test.go:97: Retrieved narinfo from S3:1219 StorePath: /build/TestClientPushesUseOnePush191403387/001/store/57kzpa1mdpv0ladmy6rv6ss8m4ni4b8n-b1220 URL: nar/1l0x6mmn5bzxidwmi38pvmb20xgypkgnj0qz5ysf170b817fnlh1.nar.zst1221 Compression: zstd1222 NarHash: sha256:1l0x6mmn5bzxidwmi38pvmb20xgypkgnj0qz5ysf170b817fnlh11223 NarSize: 2081224 References: /build/TestClientPushesUseOnePush191403387/001/store/z7f8pmwyss69xc1di0a5r5k8bvryg42r-shared-dep1225 CA: text:sha256:1w1i2s4nhpn45wwvy7zaj1phiqr6f4v6nn2bkiigxc36hyd1pp2n12262026/11/12 11:49:51 INFO Uploading 2 paths to 127.0.0.1 (0 already cached)12272026/11/12 11:49:51 INFO Uploading 0f0d2pw4wrm0q8y5f1229xxq730ckkff-top (224B)12282026/11/12 11:49:51 INFO Uploading ikiyqpxnyrlis7xsbmgi0gmj4liqbdcl-shared-dep (136B)1229--- PASS: TestClientPushesUseOnePush (0.77s)1230=== CONT TestGCTaskStore_DeduplicateSameParams1231--- PASS: TestGCTaskStore_DeduplicateSameParams (0.00s)1232=== CONT TestGCTaskStore_StartNew1233--- PASS: TestGCTaskStore_StartNew (0.00s)1234=== CONT TestService_AuthMiddleware_MTLSProxyHeader12352026/11/12 11:49:51 INFO Completed upload id=1 objects=6 duration=9.354539ms12362026/11/12 11:49:51 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete12372026/11/12 11:49:51 INFO Completed upload id=2 objects=6 duration=2.10528ms12382026/11/12 11:49:51 INFO Upload complete. (82ms)12392026/11/12 11:49:51 WARN Failed to register uploaded object key=ikiyqpxnyrlis7xsbmgi0gmj4liqbdcl.ls error="server returned 404: 404 page not found\n"12402026/11/12 11:49:51 WARN Failed to register uploaded object key=0f0d2pw4wrm0q8y5f1229xxq730ckkff.ls error="server returned 404: 404 page not found\n"12412026/11/12 11:49:51 WARN Failed to register uploaded object key=nar/15kgb4bv3il5xjd3ldfjahjr11h5kafahihjarkzj9kpz28mfppd.nar.zst error="server returned 404: 404 page not found\n"1242=== NAME TestClientFallsBackToClosures1243 client_pushes_test.go:112: Retrieved narinfo from S3:1244 StorePath: /build/TestClientFallsBackToClosures2229345656/001/store/dva0xi2qxansgr26w5pxgs9ry4pdn86p-shared-dep1245 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1246 Compression: zstd1247 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821248 NarSize: 1361249 References: 1250 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n1251 client_pushes_test.go:112: Retrieved narinfo from S3:1252 StorePath: /build/TestClientFallsBackToClosures2229345656/001/store/sha2vgmvl28qz67l9k2kpcclx6j8n7jc-a1253 URL: nar/0v5s7crg3hf89yjww565nrp6v5rbvalwgprdnq9xfpi2b5ihxbz0.nar.zst1254 Compression: zstd1255 NarHash: sha256:0v5s7crg3hf89yjww565nrp6v5rbvalwgprdnq9xfpi2b5ihxbz01256 NarSize: 2161257 References: /build/TestClientFallsBackToClosures2229345656/001/store/dva0xi2qxansgr26w5pxgs9ry4pdn86p-shared-dep1258 CA: text:sha256:1jsv7f18rd1m7xzazcaig9dhi1ilsfnjki0dbs90yl6czxg6rg681259 client_pushes_test.go:112: Retrieved narinfo from S3:1260 StorePath: /build/TestClientFallsBackToClosures2229345656/001/store/pc58r1pkbl24768nby0m5l5sa187vf39-b1261 URL: nar/0v5s7crg3hf89yjww565nrp6v5rbvalwgprdnq9xfpi2b5ihxbz0.nar.zst1262 Compression: zstd1263 NarHash: sha256:0v5s7crg3hf89yjww565nrp6v5rbvalwgprdnq9xfpi2b5ihxbz01264 NarSize: 2161265 References: /build/TestClientFallsBackToClosures2229345656/001/store/dva0xi2qxansgr26w5pxgs9ry4pdn86p-shared-dep1266 CA: text:sha256:1jsv7f18rd1m7xzazcaig9dhi1ilsfnjki0dbs90yl6czxg6rg681267=== NAME TestClientMultipleUploads1268 client_integration_test.go:406: Created store path 2: /build/TestClientMultipleUploads1859048027/001/store/yc375kg8iadg80kdqxrbglg585y7cm7x-test-file-2.txt1269--- PASS: TestClientFallsBackToClosures (0.85s)12702026/11/12 11:49:51 WARN readiness check failed error="closed pool"1271=== CONT TestGCEndsOnShutdown1272=== RUN TestGCEndsOnShutdown/before_the_run1273=== PAUSE TestGCEndsOnShutdown/before_the_run1274=== RUN TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete1275=== PAUSE TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete1276=== RUN TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark1277=== PAUSE TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark1278--- PASS: TestService_readinessHandler (0.54s)1279=== CONT TestCommitRacingPendingCleanupKeepsObjects1280=== CONT TestService_AuthMiddleware12812026/11/12 11:49:51 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign12822026/11/12 11:49:51 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"12832026/11/12 11:49:51 INFO Received push request method=POST path=/api/pushes12842026/11/12 11:49:51 INFO Signed narinfos id=1 count=212852026/11/12 11:49:51 INFO Uploading 2 narinfos12862026/11/12 11:49:51 INFO Received push request method=POST path=/api/pushes12872026/11/12 11:49:51 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)12882026/11/12 11:49:51 INFO Uploading 08wc6q11rfp1smc6vq17fryhcqgci102-test-file.txt (152B)12892026/11/12 11:49:51 INFO Received complete push request method=POST path=/api/pushes/1/complete12902026/11/12 11:49:51 WARN Failed to register uploaded object key=0f0d2pw4wrm0q8y5f1229xxq730ckkff.narinfo error="server returned 404: 404 page not found\n"12912026/11/12 11:49:51 WARN Failed to register uploaded object key=ikiyqpxnyrlis7xsbmgi0gmj4liqbdcl.narinfo error="server returned 404: 404 page not found\n"12922026/11/12 11:49:51 INFO Received push request method=POST path=/api/pushes12932026/11/12 11:49:51 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)12942026/11/12 11:49:51 INFO Uploading j87s9vjygqyy4aygms43yd7lq8snnpz2-unpinned-file.txt (128B)1295--- PASS: TestService_healthCheckHandler (0.54s)1296=== CONT TestPushWaitsForSweepHoldingRow12972026/11/12 11:49:51 WARN Failed to register uploaded object key=08wc6q11rfp1smc6vq17fryhcqgci102.ls error="server returned 404: 404 page not found\n"12982026/11/12 11:49:51 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign12992026/11/12 11:49:51 WARN Failed to register uploaded object key=nar/11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1.nar.zst error="server returned 404: 404 page not found\n"13002026/11/12 11:49:51 INFO Signed narinfos id=1 count=113012026/11/12 11:49:51 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)13022026/11/12 11:49:51 WARN Failed to register uploaded object key=j87s9vjygqyy4aygms43yd7lq8snnpz2.ls error="server returned 404: 404 page not found\n"13032026/11/12 11:49:51 WARN Failed to register uploaded object key=nar/0gh0wdnr6gmk4r5sfzzh12isnl14rgmkrwy1ws41hpsrlndclg5h.nar.zst error="server returned 404: 404 page not found\n"13042026/11/12 11:49:51 INFO Uploading 1j36gkgkf3z34zbkiai82sg26kip2x5q-test-script (136B)13052026/11/12 11:49:51 INFO Received sign narinfos request method=POST path=/api/pushes/2/sign13062026/11/12 11:49:51 INFO Uploading 1 narinfos13072026/11/12 11:49:51 INFO Signed narinfos id=2 count=113082026/11/12 11:49:51 INFO Uploading 1 narinfos13092026/11/12 11:49:51 INFO Completed push id=1 objects=6 duration=11.338689ms13102026/11/12 11:49:51 INFO Upload complete. (109ms)1311=== NAME TestClientSharedPathCommittedMidPush1312 client_integration_test.go:740: Retrieved narinfo from S3:1313 StorePath: /build/TestClientSharedPathCommittedMidPush1864689298/001/store/ikiyqpxnyrlis7xsbmgi0gmj4liqbdcl-shared-dep1314 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1315 Compression: zstd1316 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821317 NarSize: 1361318 References: 1319 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n13202026/11/12 11:49:51 INFO Received complete push request method=POST path=/api/pushes/1/complete13212026/11/12 11:49:51 WARN Failed to register uploaded object key=08wc6q11rfp1smc6vq17fryhcqgci102.narinfo error="server returned 404: 404 page not found\n"13222026/11/12 11:49:51 INFO Received complete push request method=POST path=/api/pushes/2/complete13232026/11/12 11:49:51 WARN Failed to register uploaded object key=j87s9vjygqyy4aygms43yd7lq8snnpz2.narinfo error="server returned 404: 404 page not found\n"13242026/11/12 11:49:51 WARN Failed to register uploaded object key=log/11rdglnivc4s6pnmpqfajag2vbwhfzsg-test-script.drv error="server returned 404: 404 page not found\n"13252026/11/12 11:49:51 WARN Failed to register uploaded object key=nar/1s7yghixzpinyqm7g59kyn3sav81pmnsdi5v63m6x4r2vacy907j.nar.zst error="server returned 404: 404 page not found\n"13262026/11/12 11:49:51 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign1327 client_integration_test.go:740: Retrieved narinfo from S3:13282026/11/12 11:49:51 INFO Signed narinfos id=1 count=11329 StorePath: /build/TestClientSharedPathCommittedMidPush1864689298/001/store/0f0d2pw4wrm0q8y5f1229xxq730ckkff-top13302026/11/12 11:49:51 WARN Failed to register uploaded object key=1j36gkgkf3z34zbkiai82sg26kip2x5q.ls error="server returned 404: 404 page not found\n"1331 URL: nar/15kgb4bv3il5xjd3ldfjahjr11h5kafahihjarkzj9kpz28mfppd.nar.zst13322026/11/12 11:49:51 INFO Uploading 1 narinfos1333 Compression: zstd1334 NarHash: sha256:15kgb4bv3il5xjd3ldfjahjr11h5kafahihjarkzj9kpz28mfppd1335 NarSize: 2241336 References: /build/TestClientSharedPathCommittedMidPush1864689298/001/store/ikiyqpxnyrlis7xsbmgi0gmj4liqbdcl-shared-dep1337 CA: text:sha256:1cg0mxapxw98p5v6v0bmp6djlvjxlr2bm561lh57xck4xjprbkm513382026/11/12 11:49:51 INFO Completed push id=2 objects=3 duration=3.51714ms13392026/11/12 11:49:51 INFO Upload complete. (57ms)13402026/11/12 11:49:51 INFO Completed push id=1 objects=3 duration=5.97762ms13412026/11/12 11:49:51 INFO Upload complete. (65ms)13422026/11/12 11:49:51 INFO Received complete push request method=POST path=/api/pushes/1/complete13432026/11/12 11:49:51 WARN Failed to register uploaded object key=1j36gkgkf3z34zbkiai82sg26kip2x5q.narinfo error="server returned 404: 404 page not found\n"1344--- PASS: TestClientSharedPathCommittedMidPush (0.82s)1345=== CONT TestCreatePendingClosureVerifyS3FailureReleasesConnection13462026/11/12 11:49:51 INFO Completed push id=1 objects=4 duration=9.20589ms13472026/11/12 11:49:51 INFO Upload complete. (66ms)13482026/11/12 11:49:51 INFO Received push request method=POST path=/api/pushes1349=== NAME TestClientCADerivations1350 client_ca_test.go:136: Built CA derivation: /build/TestClientCADerivations3062362673/001/store/xm5pys9kwazrm1v6wz4qcqiyfjl0ac6i-ca-test13512026/11/12 11:49:51 INFO Aborted multipart uploads count=0 kept=013522026/11/12 11:49:51 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:43241/oidc13532026/11/12 11:49:51 INFO Starting cleanup of old closures method=DELETE path=/api/closures13542026/11/12 11:49:51 INFO Garbage collection started1355--- PASS: TestCacheStatsHandler (0.53s)1356=== CONT TestRegisterCompletedObjectNeedsPendingRow13572026/11/12 11:49:51 WARN Force mode enabled - objects will be deleted immediately without grace period13582026/11/12 11:49:51 INFO Uploading 3 paths to 127.0.0.1 (0 already cached)13592026/11/12 11:49:51 INFO Uploading 9lf8bj3drwfpzkz6f82ddgzpgpgjfxwl-test-file-0.txt (160B)13602026/11/12 11:49:51 INFO Uploading 3h49annsbcv8ag9vjr2rsg7c5cv51znc-test-file-1.txt (160B)13612026/11/12 11:49:51 INFO All 1 paths already cached13622026/11/12 11:49:51 INFO Uploading yc375kg8iadg80kdqxrbglg585y7cm7x-test-file-2.txt (160B)1363=== NAME TestClientIntegration1364 client_integration_test.go:318: Retrieved narinfo from S3:1365 StorePath: /build/TestClientIntegration3822821240/002/store/08wc6q11rfp1smc6vq17fryhcqgci102-test-file.txt1366 URL: nar/11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1.nar.zst1367 Compression: zstd1368 NarHash: sha256:11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk11369 NarSize: 1521370 References: 1371 CA: fixed:r:sha256:11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk11372--- PASS: TestService_ReadScope_PublicByDefault (0.39s)1373=== CONT TestGCSweepDeliversEachKeyOnce13742026/11/12 11:49:51 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=01375=== NAME TestClientIntegration1376 client_integration_test.go:319: Retrieved .ls file from S3 (compressed size: 77 bytes)1377 client_integration_test.go:319: Decompressed .ls content (64 bytes):1378 {"version":1,"root":{"type":"regular","size":39,"narOffset":96}}13792026/11/12 11:49:51 WARN Failed to register uploaded object key=3h49annsbcv8ag9vjr2rsg7c5cv51znc.ls error="server returned 404: 404 page not found\n"13802026/11/12 11:49:51 WARN Failed to register uploaded object key=9lf8bj3drwfpzkz6f82ddgzpgpgjfxwl.ls error="server returned 404: 404 page not found\n"13812026/11/12 11:49:51 INFO Vacuumed table table=pending_closures13822026/11/12 11:49:51 WARN Failed to register uploaded object key=yc375kg8iadg80kdqxrbglg585y7cm7x.ls error="server returned 404: 404 page not found\n"13832026/11/12 11:49:51 WARN Failed to register uploaded object key=nar/0ahq3izw3h0b0c08p3xdfclh758fwxnp85ccvflasb4vnfsv50ri.nar.zst error="server returned 404: 404 page not found\n"13842026/11/12 11:49:51 WARN Failed to register uploaded object key=nar/13p9pqi0ga6fx2apcwls9hjnv7nmf740ggpn5zvfzdsp5vz2bbg4.nar.zst error="server returned 404: 404 page not found\n"13852026/11/12 11:49:51 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign13862026/11/12 11:49:51 WARN Failed to register uploaded object key=nar/1823g6vx4q5zqvr73m7lkq1wpk40fin7bsc792di430yk7sg4lka.nar.zst error="server returned 404: 404 page not found\n"13872026/11/12 11:49:51 INFO Signed narinfos id=1 count=313882026/11/12 11:49:51 INFO Uploading 3 narinfos1389=== NAME TestClientCADerivations1390 client_ca_test.go:139: Found 1 dependencies (including self)13912026/11/12 11:49:51 INFO Aborted multipart uploads count=0 kept=013922026/11/12 11:49:51 INFO Vacuumed table table=commit_receipts13932026/11/12 11:49:51 WARN Failed to register uploaded object key=9lf8bj3drwfpzkz6f82ddgzpgpgjfxwl.narinfo error="server returned 404: 404 page not found\n"13942026/11/12 11:49:51 WARN Failed to register uploaded object key=3h49annsbcv8ag9vjr2rsg7c5cv51znc.narinfo error="server returned 404: 404 page not found\n"13952026/11/12 11:49:51 INFO Received complete push request method=POST path=/api/pushes/1/complete13962026/11/12 11:49:51 WARN Failed to register uploaded object key=yc375kg8iadg80kdqxrbglg585y7cm7x.narinfo error="server returned 404: 404 page not found\n"13972026/11/12 11:49:51 INFO Vacuumed table table=pending_objects13982026/11/12 11:49:51 INFO Vacuumed table table=multipart_uploads13992026/11/12 11:49:51 WARN Force mode enabled - objects will be deleted immediately without grace period14002026/11/12 11:49:51 INFO Vacuumed table table=closures14012026/11/12 11:49:51 INFO Vacuumed table table=objects1402=== NAME TestClientWithDependencies1403 client_integration_test.go:677: Skipping nix copy test - isolated store (/build/TestClientWithDependencies50158575/001/store) requires matching store prefix1404--- PASS: TestClientWithDependencies (0.86s)1405=== CONT TestSweepRechecksPendingAfterLocking14062026/11/12 11:49:51 INFO Completed push id=1 objects=9 duration=14.18596ms14072026/11/12 11:49:51 INFO Upload complete. (92ms)1408=== NAME TestClientMultipleUploads1409 client_integration_test.go:417: Uploaded 3 paths in 122.794548ms1410--- PASS: TestClientMultipleUploads (0.83s)1411=== CONT TestTombstonedObjectOfferedWithoutWaiting1412=== RUN TestService_RequireScope_OIDC/builder_may_write1413=== PAUSE TestService_RequireScope_OIDC/builder_may_write1414=== RUN TestService_RequireScope_OIDC/builder_may_not_admin1415=== PAUSE TestService_RequireScope_OIDC/builder_may_not_admin1416=== RUN TestService_RequireScope_OIDC/ops_may_admin1417=== PAUSE TestService_RequireScope_OIDC/ops_may_admin1418=== RUN TestService_RequireScope_OIDC/ops_may_not_write1419=== PAUSE TestService_RequireScope_OIDC/ops_may_not_write1420=== RUN TestService_RequireScope_OIDC/reader_may_not_write1421=== PAUSE TestService_RequireScope_OIDC/reader_may_not_write1422=== RUN TestService_RequireScope_OIDC/static_token_may_admin1423=== PAUSE TestService_RequireScope_OIDC/static_token_may_admin1424=== RUN TestService_RequireScope_OIDC/static_token_may_write1425=== PAUSE TestService_RequireScope_OIDC/static_token_may_write1426=== RUN TestService_RequireScope_OIDC/reader_may_read1427=== PAUSE TestService_RequireScope_OIDC/reader_may_read1428=== RUN TestService_RequireScope_OIDC/writer_implies_read1429=== PAUSE TestService_RequireScope_OIDC/writer_implies_read1430=== RUN TestService_RequireScope_OIDC/anonymous_may_not_read1431=== PAUSE TestService_RequireScope_OIDC/anonymous_may_not_read1432=== RUN TestService_RequireScope_OIDC/reader_may_read_with_netrc1433=== PAUSE TestService_RequireScope_OIDC/reader_may_read_with_netrc1434=== RUN TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1435=== PAUSE TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1436=== RUN TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read1437=== PAUSE TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read1438=== RUN TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1439=== PAUSE TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1440=== CONT TestGetExistingObjectsLocksRows14412026/11/12 11:49:51 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=01442--- PASS: TestService_ReadAuthMiddleware (0.22s)1443=== CONT TestGCSweepSkipsPendingObjects14442026/11/12 11:49:51 INFO Vacuumed table table=pending_closures14452026/11/12 11:49:51 INFO Vacuumed table table=commit_receipts14462026/11/12 11:49:51 INFO Vacuumed table table=pending_objects14472026/11/12 11:49:51 INFO Vacuumed table table=multipart_uploads14482026/11/12 11:49:51 INFO Vacuumed table table=closures14492026/11/12 11:49:51 INFO Vacuumed table table=objects14502026/11/12 11:49:51 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted,O=Other"14512026/11/12 11:49:51 WARN mTLS auth: bound subjects configured but subject DN unavailable14522026/11/12 11:49:51 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted"1453--- PASS: TestService_AuthMiddleware_MTLSBoundSubjects (0.24s)14542026/11/12 11:49:51 INFO Received push request method=POST path=/api/pushes1455=== CONT TestDeduplicatedObjectsRecordedAsPending14562026/11/12 11:49:51 INFO Object in database but missing from S3 key=08wc6q11rfp1smc6vq17fryhcqgci102.ls14572026/11/12 11:49:51 WARN Found objects in DB but missing from S3, will re-upload count=114582026/11/12 11:49:51 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)14592026/11/12 11:49:51 INFO Received push request method=POST path=/api/pushes14602026/11/12 11:49:51 INFO Received complete push request method=POST path=/api/pushes/2/complete14612026/11/12 11:49:51 WARN Failed to register uploaded object key=08wc6q11rfp1smc6vq17fryhcqgci102.ls error="server returned 404: 404 page not found\n"14622026/11/12 11:49:51 INFO Completed push id=2 objects=3 duration=2.99408ms14632026/11/12 11:49:51 INFO Upload complete. (60ms)14642026/11/12 11:49:51 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)14652026/11/12 11:49:51 INFO Uploading xm5pys9kwazrm1v6wz4qcqiyfjl0ac6i-ca-test (144B)1466=== NAME TestClientIntegration1467 client_integration_test.go:336: Retrieved .ls file from S3 (compressed size: 62 bytes)1468 client_integration_test.go:336: Decompressed .ls content (49 bytes):1469 {"version":1,"root":{"type":"regular","size":39}}14702026/11/12 11:49:51 WARN Failed to register uploaded object key=xm5pys9kwazrm1v6wz4qcqiyfjl0ac6i.ls error="server returned 404: 404 page not found\n"14712026/11/12 11:49:51 WARN Failed to register uploaded object key=nar/04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n.nar.zst error="server returned 404: 404 page not found\n"14722026/11/12 11:49:51 WARN Failed to register uploaded object key=log/vfaxavyb51q7jq8ldzqj4pirdsyi9dhn-ca-test.drv error="server returned 404: 404 page not found\n"14732026/11/12 11:49:51 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign1474--- PASS: TestService_AuthMiddleware_MTLSProxyHeader (0.25s)1475=== CONT TestSweepSkipsRowsLockedByPush14762026/11/12 11:49:51 INFO Signed narinfos id=1 count=114772026/11/12 11:49:51 INFO Uploading 1 narinfos14782026/11/12 11:49:52 INFO Received complete push request method=POST path=/api/pushes/1/complete14792026/11/12 11:49:52 WARN Failed to register uploaded object key=xm5pys9kwazrm1v6wz4qcqiyfjl0ac6i.narinfo error="server returned 404: 404 page not found\n"14802026/11/12 11:49:52 INFO Completed push id=1 objects=4 duration=15.43109ms14812026/11/12 11:49:52 INFO Upload complete. (112ms)1482=== NAME TestClientCADerivations1483 client_ca_test.go:180: Narinfo contains CA field: StorePath: /build/TestClientCADerivations3062362673/001/store/xm5pys9kwazrm1v6wz4qcqiyfjl0ac6i-ca-test1484 URL: nar/04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n.nar.zst1485 Compression: zstd1486 NarHash: sha256:04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n1487 NarSize: 1441488 References: 1489 Deriver: /build/TestClientCADerivations3062362673/001/store/vfaxavyb51q7jq8ldzqj4pirdsyi9dhn-ca-test.drv1490 CA: fixed:r:sha256:04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n1491 client_ca_test.go:185: Checking for realisation files in S3...1492 client_ca_test.go:207: No realisation files uploaded - this is expected for locally built CA derivations1493 client_ca_test.go:208: Locally built CA derivations don't have DrvOutput IDs until they're substituted from a cache14942026/11/12 11:49:52 WARN Authentication failed token_preview=wrongtoken token_length=10 reason="static API token mismatch"1495--- PASS: TestService_AuthMiddleware (0.29s)1496=== CONT TestParseSize1497--- PASS: TestParseSize (0.00s)1498=== CONT TestCreatePendingClosure_SmallNARUsesSimplePUT14992026/11/12 11:49:52 INFO Aborted multipart uploads count=0 kept=015002026/11/12 11:49:52 WARN Force mode enabled - objects will be deleted immediately without grace period15012026/11/12 11:49:52 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=015022026/11/12 11:49:52 INFO Vacuumed table table=pending_closures15032026/11/12 11:49:52 INFO Vacuumed table table=commit_receipts15042026/11/12 11:49:52 INFO Vacuumed table table=pending_objects15052026/11/12 11:49:52 INFO Vacuumed table table=multipart_uploads15062026/11/12 11:49:52 INFO Vacuumed table table=closures15072026/11/12 11:49:52 INFO Vacuumed table table=objects15082026/11/12 11:49:52 INFO Received push request method=POST path=/api/pushes1509--- PASS: TestCommitRacingPendingCleanupKeepsObjects (0.32s)1510=== CONT TestCompleteMultipartUnregistered15112026/11/12 11:49:52 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)15122026/11/12 11:49:52 INFO Received uploads request method=POST path=/api/pending_closures15132026/11/12 11:49:52 INFO Received complete push request method=POST path=/api/pushes/3/complete15142026/11/12 11:49:52 WARN Failed to register uploaded object key=08wc6q11rfp1smc6vq17fryhcqgci102.ls error="server returned 404: 404 page not found\n"15152026/11/12 11:49:52 INFO Completed push id=3 objects=3 duration=2.32701ms15162026/11/12 11:49:52 INFO Upload complete. (48ms)1517=== NAME TestClientIntegration1518 client_integration_test.go:353: Retrieved .ls file from S3 (compressed size: 62 bytes)1519 client_integration_test.go:353: Decompressed .ls content (49 bytes):1520 {"version":1,"root":{"type":"regular","size":39}}1521 client_integration_test.go:364: Testing garbage collection...15222026/11/12 11:49:52 INFO Received uploads request method=POST path=/api/pending_closures15232026/11/12 11:49:52 INFO Starting cleanup of old closures method=DELETE path=/api/closures15242026/11/12 11:49:52 INFO Garbage collection started1525=== NAME TestClientCADerivations1526 client_ca_test.go:258: nix copy output: warning: you don't have Internet access; disabling some network-dependent features1527 warning: failed to create TLS context for AWS credential providers; SSO, STS WebIdentity, and ECS container authentication will be unavailable1528 error: binary cache 's3://bucket52?endpoint=http://localhost:40429®ion=eu-west-1' is for Nix stores with prefix '/nix/store', not '/build/TestClientCADerivations3062362673/001/store'1529 client_ca_test.go:259: nix copy failed (might be expected with isolated stores): exit status 11530--- PASS: TestClientCADerivations (0.96s)1531=== CONT TestService_verifyS3ReuploadsStaleNarinfo1532=== RUN TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed1533=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed1534=== RUN TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed1535=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed1536=== RUN TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost1537=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost1538=== RUN TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed1539=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed1540=== RUN TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed1541=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed1542=== CONT TestFailedS3DeleteKeepsTombstone15432026/11/12 11:49:52 INFO Aborted multipart uploads count=0 kept=01544=== RUN TestService_AuthMiddleware_OIDC/valid_OIDC_token1545=== PAUSE TestService_AuthMiddleware_OIDC/valid_OIDC_token1546=== RUN TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1547=== PAUSE TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1548=== RUN TestService_AuthMiddleware_OIDC/malformed_token_rejected1549=== PAUSE TestService_AuthMiddleware_OIDC/malformed_token_rejected1550=== RUN TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1551=== PAUSE TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1552=== CONT TestForceGCDuringPushOffersSweptObject1553=== RUN TestForceGCDuringPushOffersSweptObject/before_pending_rows1554=== PAUSE TestForceGCDuringPushOffersSweptObject/before_pending_rows1555=== RUN TestForceGCDuringPushOffersSweptObject/after_presence_check1556=== PAUSE TestForceGCDuringPushOffersSweptObject/after_presence_check1557=== CONT TestIsValidUploadKey1558=== RUN TestIsValidUploadKey/narinfo1559=== PAUSE TestIsValidUploadKey/narinfo1560=== RUN TestIsValidUploadKey/nar_zst1561=== PAUSE TestIsValidUploadKey/nar_zst1562=== RUN TestIsValidUploadKey/nar_xz1563=== PAUSE TestIsValidUploadKey/nar_xz1564=== RUN TestIsValidUploadKey/nar_plain1565=== PAUSE TestIsValidUploadKey/nar_plain1566=== RUN TestIsValidUploadKey/listing1567=== PAUSE TestIsValidUploadKey/listing1568=== RUN TestIsValidUploadKey/build_log1569=== PAUSE TestIsValidUploadKey/build_log1570=== RUN TestIsValidUploadKey/build_log_home-manager_file1571=== PAUSE TestIsValidUploadKey/build_log_home-manager_file1572=== RUN TestIsValidUploadKey/build_log_plus_in_name1573=== PAUSE TestIsValidUploadKey/build_log_plus_in_name1574=== RUN TestIsValidUploadKey/build_log_question_mark1575=== PAUSE TestIsValidUploadKey/build_log_question_mark1576=== RUN TestIsValidUploadKey/build_log_equals1577=== PAUSE TestIsValidUploadKey/build_log_equals1578=== RUN TestIsValidUploadKey/realisation1579=== PAUSE TestIsValidUploadKey/realisation1580=== RUN TestIsValidUploadKey/realisation_plus_in_output1581=== PAUSE TestIsValidUploadKey/realisation_plus_in_output1582=== RUN TestIsValidUploadKey/nix-cache-info1583=== PAUSE TestIsValidUploadKey/nix-cache-info1584=== RUN TestIsValidUploadKey/index.html1585=== PAUSE TestIsValidUploadKey/index.html1586=== RUN TestIsValidUploadKey/narinfo_key,_nar_type1587=== PAUSE TestIsValidUploadKey/narinfo_key,_nar_type1588=== RUN TestIsValidUploadKey/nar_key,_narinfo_type1589=== PAUSE TestIsValidUploadKey/nar_key,_narinfo_type1590=== RUN TestIsValidUploadKey/listing_key,_narinfo_type1591=== PAUSE TestIsValidUploadKey/listing_key,_narinfo_type1592=== RUN TestIsValidUploadKey/traversal1593=== PAUSE TestIsValidUploadKey/traversal1594=== RUN TestIsValidUploadKey/traversal_nar1595=== PAUSE TestIsValidUploadKey/traversal_nar1596=== RUN TestIsValidUploadKey/absolute1597=== PAUSE TestIsValidUploadKey/absolute1598=== RUN TestIsValidUploadKey/empty_key1599=== PAUSE TestIsValidUploadKey/empty_key1600=== RUN TestIsValidUploadKey/unknown_type1601=== PAUSE TestIsValidUploadKey/unknown_type1602=== CONT TestService_verifyS3Integrity16032026/11/12 11:49:52 WARN Force mode enabled - objects will be deleted immediately without grace period1604--- PASS: TestRegisterCompletedObjectNeedsPendingRow (0.32s)1605=== CONT TestUploadHandlersRejectOversizedBody16062026/11/12 11:49:52 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=016072026/11/12 11:49:52 INFO Vacuumed table table=pending_closures16082026/11/12 11:49:52 INFO Vacuumed table table=commit_receipts16092026/11/12 11:49:52 INFO Vacuumed table table=pending_objects16102026/11/12 11:49:52 INFO Vacuumed table table=multipart_uploads16112026/11/12 11:49:52 INFO Vacuumed table table=closures16122026/11/12 11:49:52 INFO Vacuumed table table=objects16132026/11/12 11:49:52 INFO Aborted multipart uploads count=0 kept=016142026/11/12 11:49:52 WARN Force mode enabled - objects will be deleted immediately without grace period16152026/11/12 11:49:52 INFO Aborted multipart uploads count=0 kept=01616--- PASS: TestCreatePendingClosureVerifyS3FailureReleasesConnection (0.51s)1617=== CONT TestService_cleanupPendingClosuresHandler16182026/11/12 11:49:52 WARN Force mode enabled - objects will be deleted immediately without grace period1619=== RUN TestUploadHandlersRejectOversizedBody/create_pending_closure1620=== PAUSE TestUploadHandlersRejectOversizedBody/create_pending_closure16212026/11/12 11:49:52 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=01622=== RUN TestUploadHandlersRejectOversizedBody/complete_multipart1623=== PAUSE TestUploadHandlersRejectOversizedBody/complete_multipart1624=== RUN TestUploadHandlersRejectOversizedBody/request_more_parts1625=== PAUSE TestUploadHandlersRejectOversizedBody/request_more_parts1626=== CONT TestPendingClosureWriteTimeout1627=== RUN TestPendingClosureWriteTimeout/empty1628=== PAUSE TestPendingClosureWriteTimeout/empty1629=== RUN TestPendingClosureWriteTimeout/400_objects1630=== PAUSE TestPendingClosureWriteTimeout/400_objects16312026/11/12 11:49:52 INFO lead: released remote=192.0.2.1:12341632=== RUN TestPendingClosureWriteTimeout/670k_objects1633=== PAUSE TestPendingClosureWriteTimeout/670k_objects1634=== RUN TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline1635--- PASS: TestLeadEndsOnShutdown (1.82s)1636=== CONT TestService_createPendingClosureHandler1637=== PAUSE TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline1638=== CONT TestProxyWriteTimeout1639=== RUN TestProxyWriteTimeout/narinfo1640=== PAUSE TestProxyWriteTimeout/narinfo1641=== RUN TestProxyWriteTimeout/1_GiB_nar1642=== PAUSE TestProxyWriteTimeout/1_GiB_nar1643=== RUN TestProxyWriteTimeout/10_GiB_nar1644=== PAUSE TestProxyWriteTimeout/10_GiB_nar1645=== RUN TestProxyWriteTimeout/unknown_size1646=== PAUSE TestProxyWriteTimeout/unknown_size1647=== CONT TestUploadHandlersRejectInvalidKeys1648=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info1649=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info1650=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal16512026/11/12 11:49:52 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=01652=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal1653=== RUN TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key1654=== PAUSE TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key1655=== RUN TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key1656=== PAUSE TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key1657=== CONT TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle1658--- PASS: TestGetExistingObjectsLocksRows (0.51s)16592026/11/12 11:49:52 INFO Vacuumed table table=pending_closures1660=== CONT TestPush_SignsNarinfosOfItsPendingObjects16612026/11/12 11:49:52 INFO Vacuumed table table=commit_receipts16622026/11/12 11:49:52 INFO Received uploads request method=POST path=/api/pending_closures16632026/11/12 11:49:52 INFO Vacuumed table table=pending_closures16642026/11/12 11:49:52 INFO Vacuumed table table=commit_receipts16652026/11/12 11:49:52 INFO Vacuumed table table=pending_objects16662026/11/12 11:49:52 INFO Vacuumed table table=multipart_uploads16672026/11/12 11:49:52 INFO Vacuumed table table=closures16682026/11/12 11:49:52 INFO Vacuumed table table=pending_objects16692026/11/12 11:49:52 INFO Vacuumed table table=multipart_uploads16702026/11/12 11:49:52 INFO Vacuumed table table=closures16712026/11/12 11:49:52 INFO Vacuumed table table=objects16722026/11/12 11:49:52 INFO Vacuumed table table=objects1673--- PASS: TestTombstonedObjectOfferedWithoutWaiting (0.52s)1674--- PASS: TestGCSweepDeliversEachKeyOnce (0.56s)1675=== CONT TestPush_RootsAreLockedInByteOrder1676=== CONT TestSkippedUploadsHandler16772026/11/12 11:49:52 INFO Client skipped oversized paths paths=3 nar_bytes=50000000001678--- PASS: TestSweepRechecksPendingAfterLocking (0.54s)1679=== CONT TestService_Rustfstest1680--- PASS: TestSkippedUploadsHandler (0.01s)1681=== CONT TestPush_RejectsBadRequests16822026/11/12 11:49:52 INFO Received uploads request method=POST path=/api/pending_closures16832026/11/12 11:49:52 INFO Aborted multipart uploads count=0 kept=016842026/11/12 11:49:52 WARN Force mode enabled - objects will be deleted immediately without grace period16852026/11/12 11:49:52 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=016862026/11/12 11:49:52 INFO Received uploads request method=POST path=/api/pending_closures16872026/11/12 11:49:52 INFO Vacuumed table table=pending_closures16882026/11/12 11:49:52 INFO Vacuumed table table=commit_receipts16892026/11/12 11:49:52 INFO Vacuumed table table=pending_objects16902026/11/12 11:49:52 INFO Vacuumed table table=multipart_uploads16912026/11/12 11:49:52 INFO Vacuumed table table=closures1692--- PASS: TestDeduplicatedObjectsRecordedAsPending (0.54s)1693=== CONT TestCompleteMultipartUpload_ErrorButObjectExists16942026/11/12 11:49:52 INFO Vacuumed table table=objects16952026/11/12 11:49:52 INFO lead: released remote=192.0.2.1:12341696--- PASS: TestGCSweepSkipsPendingObjects (0.58s)1697=== CONT TestPresignedUploadRegisteredBeforeCommit16982026/11/12 11:49:52 INFO lead: acquired remote=192.0.2.1:123416992026/11/12 11:49:52 INFO Aborted multipart uploads count=0 kept=017002026/11/12 11:49:52 INFO Received uploads request method=POST path=/api/pending_closures17012026/11/12 11:49:52 WARN Force mode enabled - objects will be deleted immediately without grace period17022026/11/12 11:49:52 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=017032026/11/12 11:49:52 INFO Vacuumed table table=pending_closures17042026/11/12 11:49:52 INFO Vacuumed table table=commit_receipts17052026/11/12 11:49:52 INFO Vacuumed table table=pending_objects17062026/11/12 11:49:52 INFO Vacuumed table table=multipart_uploads17072026/11/12 11:49:52 INFO Vacuumed table table=closures1708--- PASS: TestCreatePendingClosure_SmallNARUsesSimplePUT (0.50s)1709=== CONT TestCompletedNarNotReofferedAcrossClosures17102026/11/12 11:49:52 INFO Vacuumed table table=objects17112026/11/12 11:49:52 INFO Aborted multipart uploads count=0 kept=017122026/11/12 11:49:52 WARN Force mode enabled - objects will be deleted immediately without grace period17132026/11/12 11:49:52 INFO lead: given back to the incumbent remote=192.0.2.1:123417142026/11/12 11:49:52 INFO lead: acquired remote=192.0.2.1:123417152026/11/12 11:49:52 INFO Received complete multipart upload request method=POST path=/api/multipart/complete1716--- PASS: TestDeleteClosuresRacingPins (1.73s)1717=== CONT TestPush_SkippedKeySurvivesGCBeforeCommit17182026/11/12 11:49:52 ERROR Failed to get multipart upload error="no rows in result set" upload_id=does-not-exist object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst1719--- PASS: TestCompleteMultipartUnregistered (0.50s)1720=== CONT TestCommitCanBeRepeatedUntilCleanup1721=== RUN TestCommitCanBeRepeatedUntilCleanup/push1722=== PAUSE TestCommitCanBeRepeatedUntilCleanup/push1723=== RUN TestCommitCanBeRepeatedUntilCleanup/pending_closure1724=== PAUSE TestCommitCanBeRepeatedUntilCleanup/pending_closure1725=== CONT TestPush_RepeatedRootCommitsOnce1726--- PASS: TestPushWaitsForSweepHoldingRow (0.79s)1727=== CONT TestPush_CompleteCommitsEveryRoot17282026/11/12 11:49:52 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=017292026/11/12 11:49:52 INFO Vacuumed table table=pending_closures17302026/11/12 11:49:52 INFO Vacuumed table table=commit_receipts17312026/11/12 11:49:52 INFO Vacuumed table table=pending_objects17322026/11/12 11:49:52 INFO Vacuumed table table=multipart_uploads17332026/11/12 11:49:52 INFO Vacuumed table table=closures17342026/11/12 11:49:52 INFO Vacuumed table table=objects1735--- PASS: TestSweepSkipsRowsLockedByPush (0.62s)1736=== CONT TestPush_CompleteWaitsForCleanupOfThePush17372026/11/12 11:49:52 INFO Received uploads request method=POST path=/api/pending_closures17382026/11/12 11:49:52 INFO lead: released remote=192.0.2.1:123417392026/11/12 11:49:52 INFO Received uploads request method=POST path=/api/pending_closures17402026/11/12 11:49:52 INFO Received cleanup request method=DELETE path=/api/pending_closures17412026/11/12 11:49:52 INFO Aborted multipart uploads count=0 kept=017422026/11/12 11:49:52 INFO Received uploads request method=POST path=/api/pending_closures17432026/11/12 11:49:52 INFO lead: acquired remote=192.0.2.1:123417442026/11/12 11:49:52 INFO Received cleanup request method=DELETE path=/api/pending_closures17452026/11/12 11:49:52 INFO Aborted multipart uploads count=1 kept=017462026/11/12 11:49:52 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete17472026-11-12 11:49:52.721 UTC [2009] ERROR: Closure does not exist: id=117482026-11-12 11:49:52.721 UTC [2009] CONTEXT: PL/pgSQL function commit_pending_closure(bigint) line 17 at RAISE17492026-11-12 11:49:52.721 UTC [2009] STATEMENT: -- name: CommitPendingClosure :one1750 SELECT commit_pending_closure($1::bigint)::bigint1751 1752--- PASS: TestService_cleanupPendingClosuresHandler (0.38s)1753=== CONT TestSharedNarBothUploadsComplete17542026/11/12 11:49:52 INFO Aborted multipart uploads count=0 kept=017552026/11/12 11:49:52 INFO Received push request method=POST path=/api/pushes17562026/11/12 11:49:52 WARN Force mode enabled - objects will be deleted immediately without grace period17572026/11/12 11:49:52 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."17582026/11/12 11:49:52 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."17592026/11/12 11:49:52 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign17602026/11/12 11:49:52 INFO Signed narinfos id=1 count=117612026/11/12 11:49:52 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign17622026/11/12 11:49:52 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign17632026/11/12 11:49:52 ERROR Failed to sign narinfos signer="index 1" error="signing key cannot be nil"17642026/11/12 11:49:52 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign17652026/11/12 11:49:52 ERROR Failed to sign narinfos signer=niks3-test-1 error="signer failed"17662026/11/12 11:49:52 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign1767--- PASS: TestFailedS3DeleteKeepsTombstone (0.59s)17682026/11/12 11:49:52 ERROR Signer returned unexpected signature count signer=niks3-test-1 got=0 want=11769=== CONT TestPresentNotReportedWhileGCDeletesClosure/stale17702026/11/12 11:49:52 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign17712026/11/12 11:49:52 ERROR Signer returned unexpected object key signer=niks3-test-1 object_key=unexpected.narinfo1772--- PASS: TestPush_SignsNarinfosOfItsPendingObjects (0.32s)1773=== CONT TestPresentNotReportedWhileGCDeletesClosure/fresh1774--- PASS: TestService_Rustfstest (0.33s)1775=== CONT TestParseSingleRange/none1776=== CONT TestParseSingleRange/start_far_past_EOF1777=== CONT TestParseSingleRange/start_past_EOF1778=== CONT TestParseSingleRange/single_byte1779=== CONT TestParseSingleRange/suffix_exceeds_size1780=== CONT TestParseSingleRange/suffix1781=== CONT TestParseSingleRange/end_clamped_to_size1782=== CONT TestParseSingleRange/open-ended1783=== CONT TestParseSingleRange/closed1784=== CONT TestParseSingleRange/malformed_end_before_start1785=== CONT TestParseSingleRange/malformed_both_empty1786=== CONT TestParseSingleRange/malformed_no_dash1787=== CONT TestParseSingleRange/multi-range_ignored1788=== CONT TestParseSingleRange/unknown_unit1789--- PASS: TestParseSingleRange (0.00s)1790 --- PASS: TestParseSingleRange/none (0.00s)1791 --- PASS: TestParseSingleRange/start_far_past_EOF (0.00s)1792 --- PASS: TestParseSingleRange/start_past_EOF (0.00s)1793 --- PASS: TestParseSingleRange/single_byte (0.00s)1794 --- PASS: TestParseSingleRange/suffix_exceeds_size (0.00s)1795 --- PASS: TestParseSingleRange/suffix (0.00s)1796 --- PASS: TestParseSingleRange/end_clamped_to_size (0.00s)1797 --- PASS: TestParseSingleRange/open-ended (0.00s)1798 --- PASS: TestParseSingleRange/closed (0.00s)1799 --- PASS: TestParseSingleRange/malformed_end_before_start (0.00s)1800 --- PASS: TestParseSingleRange/malformed_both_empty (0.00s)1801 --- PASS: TestParseSingleRange/malformed_no_dash (0.00s)1802 --- PASS: TestParseSingleRange/multi-range_ignored (0.00s)1803 --- PASS: TestParseSingleRange/unknown_unit (0.00s)1804=== CONT TestIsValidCachePath/narinfo1805=== CONT TestIsValidCachePath/index.html1806=== CONT TestIsValidCachePath/short_hash1807=== CONT TestIsValidCachePath/wrong_extension1808=== CONT TestIsValidCachePath/leading_slash1809=== CONT TestIsValidCachePath/empty1810=== CONT TestIsValidCachePath/random_path1811=== CONT TestIsValidCachePath/invalid_char_u1812=== CONT TestIsValidCachePath/invalid_char_e1813=== CONT TestIsValidCachePath/traversal_in_middle1814=== CONT TestIsValidCachePath/traversal_parent1815=== CONT TestIsValidCachePath/nar_uncompressed1816=== CONT TestIsValidCachePath/nix-cache-info1817=== CONT TestIsValidCachePath/realisation1818=== CONT TestIsValidCachePath/log1819=== CONT TestIsValidCachePath/ls1820=== CONT TestIsValidCachePath/nar_xz1821=== CONT TestIsValidCachePath/nar_bz21822=== CONT TestIsValidCachePath/nar_zst1823=== CONT TestIsValidCachePath/narinfo_all_nix_base32_chars1824=== CONT TestServerTLSConfig/no_client_CA1825=== CONT TestServerTLSConfig/not_a_PEM_file1826--- PASS: TestIsValidCachePath (0.00s)1827 --- PASS: TestIsValidCachePath/narinfo (0.00s)1828 --- PASS: TestIsValidCachePath/index.html (0.00s)1829 --- PASS: TestIsValidCachePath/short_hash (0.00s)1830 --- PASS: TestIsValidCachePath/wrong_extension (0.00s)1831 --- PASS: TestIsValidCachePath/leading_slash (0.00s)1832 --- PASS: TestIsValidCachePath/empty (0.00s)1833 --- PASS: TestIsValidCachePath/random_path (0.00s)1834 --- PASS: TestIsValidCachePath/invalid_char_u (0.00s)1835 --- PASS: TestIsValidCachePath/invalid_char_e (0.00s)1836 --- PASS: TestIsValidCachePath/traversal_in_middle (0.00s)1837 --- PASS: TestIsValidCachePath/traversal_parent (0.00s)1838 --- PASS: TestIsValidCachePath/nar_uncompressed (0.00s)1839 --- PASS: TestIsValidCachePath/nix-cache-info (0.00s)1840 --- PASS: TestIsValidCachePath/realisation (0.00s)1841 --- PASS: TestIsValidCachePath/log (0.00s)1842 --- PASS: TestIsValidCachePath/ls (0.00s)1843 --- PASS: TestIsValidCachePath/nar_xz (0.00s)1844 --- PASS: TestIsValidCachePath/nar_bz2 (0.00s)1845 --- PASS: TestIsValidCachePath/nar_zst (0.00s)1846 --- PASS: TestIsValidCachePath/narinfo_all_nix_base32_chars (0.00s)1847=== CONT TestServerTLSConfig/missing_CA_file1848=== CONT TestResolveDBConnectionString/flag_wins1849=== CONT TestResolveDBConnectionString/PGHOST_allows_empty1850=== CONT TestResolveDBConnectionString/nothing_configured1851=== CONT TestResolveDBConnectionString/missing_file_is_an_error1852--- PASS: TestServerTLSConfig (0.00s)1853 --- PASS: TestServerTLSConfig/no_client_CA (0.00s)1854 --- PASS: TestServerTLSConfig/not_a_PEM_file (0.00s)1855 --- PASS: TestServerTLSConfig/missing_CA_file (0.00s)1856=== CONT TestResolveDBConnectionString/file_when_flag_empty1857=== CONT TestClientErrorHandling/InvalidStorePath1858--- PASS: TestResolveDBConnectionString (0.00s)1859 --- PASS: TestResolveDBConnectionString/flag_wins (0.00s)1860 --- PASS: TestResolveDBConnectionString/PGHOST_allows_empty (0.00s)1861 --- PASS: TestResolveDBConnectionString/nothing_configured (0.00s)1862 --- PASS: TestResolveDBConnectionString/missing_file_is_an_error (0.00s)1863 --- PASS: TestResolveDBConnectionString/file_when_flag_empty (0.00s)18642026/11/12 11:49:52 INFO Received push request method=POST path=/api/pushes18652026/11/12 11:49:52 INFO Received uploads request method=POST path=/api/pending_closures18662026/11/12 11:49:52 INFO Received uploads request method=POST path=/api/pending_closures18672026/11/12 11:49:52 INFO Received uploads request method=POST path=/api/pending_closures18682026-11-12 11:49:52.848 UTC [2070] ERROR: could not obtain lock on row in relation "closures"18692026-11-12 11:49:52.848 UTC [2070] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT18702026-11-12 11:49:52.848 UTC [2070] ERROR: could not obtain lock on row in relation "closures"18712026-11-12 11:49:52.848 UTC [2070] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT18722026-11-12 11:49:52.848 UTC [2070] ERROR: could not obtain lock on row in relation "closures"18732026-11-12 11:49:52.848 UTC [2070] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT18742026-11-12 11:49:52.849 UTC [2070] ERROR: could not obtain lock on row in relation "closures"18752026-11-12 11:49:52.849 UTC [2070] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT18762026-11-12 11:49:52.849 UTC [2070] ERROR: could not obtain lock on row in relation "closures"18772026-11-12 11:49:52.849 UTC [2070] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT1878=== RUN TestPush_RejectsBadRequests/root_not_in_objects1879=== PAUSE TestPush_RejectsBadRequests/root_not_in_objects1880=== RUN TestPush_RejectsBadRequests/no_roots1881=== PAUSE TestPush_RejectsBadRequests/no_roots1882=== RUN TestPush_RejectsBadRequests/no_objects1883=== PAUSE TestPush_RejectsBadRequests/no_objects1884=== RUN TestPush_RejectsBadRequests/bad_root1885=== PAUSE TestPush_RejectsBadRequests/bad_root1886=== CONT TestClientErrorHandling/ServerNotAvailable1887--- PASS: TestPush_RootsAreLockedInByteOrder (0.42s)1888=== CONT TestClientErrorHandling/InvalidAuthToken18892026/11/12 11:49:52 INFO Received uploads request method=POST path=/api/pending_closures18902026/11/12 11:49:52 INFO Received uploads request method=POST path=/api/pending_closures18912026/11/12 11:49:52 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://localhost:19999/api/objects/present\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/objects/present18922026/11/12 11:49:52 INFO Registered completed upload object_key=nar/jjjjjjjjjjjjjjjjjjjjjjjjjjjjjj0100000000000000000000.nar.zst18932026/11/12 11:49:52 INFO Received uploads request method=POST path=/api/pending_closures1894--- PASS: TestPresignedUploadRegisteredBeforeCommit (0.42s)1895=== CONT TestCacheConfigHandler/full_config,_no_issuer1896=== CONT TestCacheConfigHandler/no_signing_keys1897=== CONT TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1898=== CONT TestCacheConfigHandler/no_cache_url_configured1899--- PASS: TestCacheConfigHandler (0.00s)1900 --- PASS: TestCacheConfigHandler/full_config,_no_issuer (0.00s)1901 --- PASS: TestCacheConfigHandler/no_signing_keys (0.00s)1902 --- PASS: TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator (0.00s)1903 --- PASS: TestCacheConfigHandler/no_cache_url_configured (0.00s)1904=== CONT TestGCEndsOnShutdown/before_the_run19052026/11/12 11:49:52 INFO Received uploads request method=POST path=/api/pending_closures19062026/11/12 11:49:52 INFO Received uploads request method=POST path=/api/pending_closures19072026/11/12 11:49:52 INFO Received complete multipart upload request method=POST path=/api/multipart/complete19082026/11/12 11:49:52 INFO Received complete multipart upload request method=POST path=/api/multipart/complete19092026/11/12 11:49:52 WARN CompleteMultipartUpload errored but object exists; treating as success error="The specified key does not exist." object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=MmNmNDVlZGQtM2E5MC00ODA2LWIxYWEtZGRjNjZlZTdjNzFlLjNiOGExN2NmLTE1Y2QtNGFmZi05MDAyLWY4YTE2MTc5OWFiOHgxNzk0NDg0MTkyOTU1NzA0Mzc519102026/11/12 11:49:53 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=MmNmNDVlZGQtM2E5MC00ODA2LWIxYWEtZGRjNjZlZTdjNzFlLjNiOGExN2NmLTE1Y2QtNGFmZi05MDAyLWY4YTE2MTc5OWFiOHgxNzk0NDg0MTkyOTU1NzA0Mzc5 parts=11911--- PASS: TestCompleteMultipartUpload_ErrorButObjectExists (0.51s)1912=== CONT TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark19132026/11/12 11:49:53 INFO Received push request method=POST path=/api/pushes19142026/11/12 11:49:53 WARN Request failed, retrying attempt=2 backoff=182.656978ms error="Post \"http://localhost:19999/api/objects/present\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/objects/present19152026/11/12 11:49:53 INFO Received complete push request method=POST path=/api/pushes/1/complete19162026/11/12 11:49:53 INFO Completed push id=1 objects=6 duration=6.84144ms1917--- PASS: TestPush_CompleteCommitsEveryRoot (0.44s)1918=== CONT TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete1919=== NAME TestOrphanedObjectsGCStressTest1920 orphaned_objects_gc_test.go:422: Created 10 active closures, 5 to-delete closures, 20 orphaned chains19212026/11/12 11:49:53 INFO Received push request method=POST path=/api/pushes19222026/11/12 11:49:53 INFO Received complete push request method=POST path=/api/pushes/1/complete19232026/11/12 11:49:53 INFO Received push request method=POST path=/api/pushes19242026/11/12 11:49:53 INFO Completed push id=1 objects=4 duration=11.5204ms19252026/11/12 11:49:53 INFO Received push request method=POST path=/api/pushes1926 orphaned_objects_gc_test.go:440: Marked 210 objects for deletion19272026/11/12 11:49:53 INFO Received complete push request method=POST path=/api/pushes/1/complete19282026/11/12 11:49:53 INFO Completed push id=1 objects=4 duration=8.48183ms1929--- PASS: TestPush_RepeatedRootCommitsOnce (0.51s)1930=== CONT TestService_RequireScope_OIDC/builder_may_write1931=== CONT TestService_RequireScope_OIDC/reader_may_read1932=== CONT TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1933=== CONT TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read19342026/11/12 11:49:53 WARN Authentication failed token_preview=wrong token_length=5 oidc_error="not a JWT" oidc_provider=""1935=== CONT TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1936=== CONT TestService_RequireScope_OIDC/reader_may_read_with_netrc1937=== CONT TestService_RequireScope_OIDC/anonymous_may_not_read1938=== CONT TestService_RequireScope_OIDC/reader_may_not_write1939=== CONT TestService_RequireScope_OIDC/static_token_may_write1940=== CONT TestService_RequireScope_OIDC/writer_implies_read1941=== CONT TestService_RequireScope_OIDC/static_token_may_admin1942=== CONT TestService_RequireScope_OIDC/ops_may_admin1943=== CONT TestService_RequireScope_OIDC/ops_may_not_write1944=== CONT TestService_RequireScope_OIDC/builder_may_not_admin1945=== CONT TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed19462026/11/12 11:49:53 INFO Received push request method=POST path=/api/pushes1947--- PASS: TestService_RequireScope_OIDC (0.37s)1948 --- PASS: TestService_RequireScope_OIDC/builder_may_write (0.00s)1949 --- PASS: TestService_RequireScope_OIDC/reader_may_read (0.00s)1950 --- PASS: TestService_RequireScope_OIDC/empty_netrc_password_may_not_read (0.00s)1951 --- PASS: TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read (0.00s)1952 --- PASS: TestService_RequireScope_OIDC/static_token_may_admin_with_netrc (0.00s)1953 --- PASS: TestService_RequireScope_OIDC/reader_may_read_with_netrc (0.00s)1954 --- PASS: TestService_RequireScope_OIDC/anonymous_may_not_read (0.00s)1955 --- PASS: TestService_RequireScope_OIDC/reader_may_not_write (0.00s)1956 --- PASS: TestService_RequireScope_OIDC/static_token_may_write (0.00s)1957 --- PASS: TestService_RequireScope_OIDC/writer_implies_read (0.00s)1958 --- PASS: TestService_RequireScope_OIDC/static_token_may_admin (0.00s)1959 --- PASS: TestService_RequireScope_OIDC/ops_may_admin (0.00s)1960 --- PASS: TestService_RequireScope_OIDC/ops_may_not_write (0.00s)1961 --- PASS: TestService_RequireScope_OIDC/builder_may_not_admin (0.00s)19622026/11/12 11:49:53 INFO Aborted multipart uploads count=0 kept=019632026/11/12 11:49:53 INFO Received uploads request method=POST path=/api/pending_closures19642026/11/12 11:49:53 INFO Received complete multipart upload request method=POST path=/api/multipart/complete19652026/11/12 11:49:53 WARN Force mode enabled - objects will be deleted immediately without grace period19662026/11/12 11:49:53 INFO Received uploads request method=POST path=/api/pending_closures19672026/11/12 11:49:53 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=2 objects-deleted-after-grace-period=2 objects-failed-to-delete=019682026/11/12 11:49:53 INFO Vacuumed table table=pending_closures19692026/11/12 11:49:53 INFO Vacuumed table table=commit_receipts19702026/11/12 11:49:53 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst upload_id=MmNmNDVlZGQtM2E5MC00ODA2LWIxYWEtZGRjNjZlZTdjNzFlLmVmYTA5MjI2LTcxNzMtNGNiYS04N2FmLTI1YTRmZGQxNGM3YXgxNzk0NDg0MTkyNjc3MTg5ODYz parts=1019712026/11/12 11:49:53 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete19722026/11/12 11:49:53 INFO Vacuumed table table=pending_objects19732026/11/12 11:49:53 INFO Vacuumed table table=multipart_uploads19742026/11/12 11:49:53 INFO Completed upload id=1 objects=2 duration=6.14261ms19752026/11/12 11:49:53 INFO Vacuumed table table=closures19762026/11/12 11:49:53 INFO Received uploads request method=POST path=/api/pending_closures19772026/11/12 11:49:53 INFO Vacuumed table table=objects19782026/11/12 11:49:53 INFO Received uploads request method=POST path=/api/pending_closures19792026/11/12 11:49:53 INFO Received complete push request method=POST path=/api/pushes/2/complete19802026/11/12 11:49:53 INFO Object in database but missing from S3 key=dadb44fdadb44fdadb44fdadb44f0000.narinfo19812026/11/12 11:49:53 WARN Found objects in DB but missing from S3, will re-upload count=11982--- PASS: TestService_verifyS3Integrity (1.04s)1983=== CONT TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed19842026/11/12 11:49:53 INFO Completed push id=2 objects=4 duration=4.79032ms1985--- PASS: TestPush_SkippedKeySurvivesGCBeforeCommit (0.60s)1986=== CONT TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed19872026/11/12 11:49:53 WARN Request failed, retrying attempt=3 backoff=403.066523ms error="Post \"http://localhost:19999/api/objects/present\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/objects/present19882026-11-12 11:49:53.443 UTC [2128] ERROR: Push does not exist: id=119892026-11-12 11:49:53.443 UTC [2128] CONTEXT: PL/pgSQL function commit_push(bigint) line 15 at RAISE19902026-11-12 11:49:53.443 UTC [2128] STATEMENT: SELECT commit_push($1)1991--- PASS: TestPush_CompleteWaitsForCleanupOfThePush (0.82s)1992=== CONT TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost19932026/11/12 11:49:53 INFO Received complete multipart upload request method=POST path=/api/multipart/complete19942026/11/12 11:49:53 INFO Completed multipart upload object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst upload_id=MmNmNDVlZGQtM2E5MC00ODA2LWIxYWEtZGRjNjZlZTdjNzFlLmNlYjVlM2E1LWM0YTctNDVjMS04MmQ0LTBhN2Y2MmNlYzJlNXgxNzk0NDg0MTkyODMzNTgxNDMx parts=1019952026/11/12 11:49:53 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete19962026/11/12 11:49:53 INFO Completed upload id=1 objects=2 duration=5.86685ms19972026/11/12 11:49:53 INFO Received get closure request method=GET path=/api/closures/0000000000000000000000000000000019982026/11/12 11:49:53 INFO Received uploads request method=POST path=/api/pending_closures19992026/11/12 11:49:53 INFO Starting cleanup of old closures method=DELETE path=/api/closures2000=== CONT TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed
loading 857 hidden lines…
28582026/11/12 11:50:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028592026/11/12 11:50:06 INFO Vacuumed table table=pending_closures28602026/11/12 11:50:06 INFO Vacuumed table table=commit_receipts28612026/11/12 11:50:06 INFO Vacuumed table table=pending_objects28622026/11/12 11:50:06 INFO Received complete push request method=POST path=/28632026/11/12 11:50:06 INFO Received complete push request method=POST path=/28642026/11/12 11:50:06 INFO Vacuumed table table=multipart_uploads28652026/11/12 11:50:06 INFO Vacuumed table table=closures28662026/11/12 11:50:06 INFO Received uploads request method=POST path=/28672026/11/12 11:50:06 INFO Vacuumed table table=objects28682026/11/12 11:50:06 INFO Completed push id=7 objects=20 duration=2.87488ms28692026/11/12 11:50:06 INFO Received push request method=POST path=/28702026/11/12 11:50:06 INFO Completed push id=3 objects=22 duration=7.49416ms28712026/11/12 11:50:06 INFO Aborted multipart uploads count=0 kept=028722026/11/12 11:50:06 INFO Received complete upload request method=POST path=/28732026/11/12 11:50:06 INFO Received push request method=POST path=/28742026/11/12 11:50:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028752026/11/12 11:50:06 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."28762026/11/12 11:50:06 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."28772026/11/12 11:50:06 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."28782026/11/12 11:50:06 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."28792026/11/12 11:50:06 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."28802026/11/12 11:50:06 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."28812026/11/12 11:50:06 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."28822026/11/12 11:50:06 INFO Vacuumed table table=pending_closures28832026/11/12 11:50:06 INFO Completed upload id=5 objects=7 duration=3.648029ms28842026/11/12 11:50:06 INFO Vacuumed table table=commit_receipts28852026/11/12 11:50:06 INFO Vacuumed table table=pending_objects28862026/11/12 11:50:06 INFO Vacuumed table table=multipart_uploads28872026/11/12 11:50:06 INFO Aborted multipart uploads count=0 kept=028882026/11/12 11:50:06 INFO Vacuumed table table=closures28892026/11/12 11:50:06 INFO Vacuumed table table=objects28902026/11/12 11:50:06 INFO Received complete push request method=POST path=/28912026/11/12 11:50:06 INFO Received push request method=POST path=/28922026/11/12 11:50:06 INFO Received uploads request method=POST path=/28932026/11/12 11:50:06 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."28942026/11/12 11:50:06 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."28952026/11/12 11:50:06 INFO Completed push id=4 objects=4 duration=3.73002ms28962026/11/12 11:50:06 INFO Aborted multipart uploads count=0 kept=028972026/11/12 11:50:06 INFO Aborted multipart uploads count=0 kept=028982026/11/12 11:50:06 INFO Received push request method=POST path=/28992026/11/12 11:50:06 INFO Received uploads request method=POST path=/29002026/11/12 11:50:06 INFO Received push request method=POST path=/29012026/11/12 11:50:06 WARN Force mode enabled - objects will be deleted immediately without grace period29022026/11/12 11:50:06 INFO Received uploads request method=POST path=/29032026/11/12 11:50:06 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029042026/11/12 11:50:06 INFO Vacuumed table table=pending_closures29052026/11/12 11:50:06 INFO Received complete upload request method=POST path=/29062026/11/12 11:50:06 INFO Vacuumed table table=commit_receipts29072026/11/12 11:50:06 INFO Vacuumed table table=pending_objects29082026/11/12 11:50:06 INFO Aborted multipart uploads count=0 kept=029092026/11/12 11:50:06 INFO Vacuumed table table=multipart_uploads29102026/11/12 11:50:06 INFO Completed upload id=6 objects=9 duration=2.02271ms29112026/11/12 11:50:06 INFO Vacuumed table table=closures29122026/11/12 11:50:06 INFO Vacuumed table table=objects29132026/11/12 11:50:06 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."29142026/11/12 11:50:06 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."29152026/11/12 11:50:06 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."29162026/11/12 11:50:06 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."29172026/11/12 11:50:06 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."29182026/11/12 11:50:06 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."29192026/11/12 11:50:06 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."29202026/11/12 11:50:06 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029212026/11/12 11:50:06 INFO Vacuumed table table=pending_closures29222026/11/12 11:50:06 INFO Vacuumed table table=commit_receipts29232026/11/12 11:50:06 INFO Aborted multipart uploads count=0 kept=029242026/11/12 11:50:06 INFO Received uploads request method=POST path=/29252026/11/12 11:50:06 INFO Vacuumed table table=pending_objects29262026/11/12 11:50:06 INFO Received uploads request method=POST path=/29272026/11/12 11:50:06 INFO Vacuumed table table=multipart_uploads29282026/11/12 11:50:06 INFO Vacuumed table table=closures29292026/11/12 11:50:06 INFO Received uploads request method=POST path=/29302026/11/12 11:50:06 INFO Vacuumed table table=objects29312026/11/12 11:50:06 INFO Received push request method=POST path=/29322026/11/12 11:50:06 INFO Aborted multipart uploads count=0 kept=029332026/11/12 11:50:06 INFO Aborted multipart uploads count=0 kept=029342026/11/12 11:50:06 INFO Received push request method=POST path=/29352026/11/12 11:50:06 WARN Force mode enabled - objects will be deleted immediately without grace period29362026/11/12 11:50:06 INFO Received complete push request method=POST path=/29372026/11/12 11:50:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029382026/11/12 11:50:06 INFO Vacuumed table table=pending_closures29392026/11/12 11:50:06 INFO Aborted multipart uploads count=0 kept=029402026/11/12 11:50:06 INFO Vacuumed table table=commit_receipts29412026/11/12 11:50:06 INFO Completed push id=14 objects=3 duration=3.56186ms29422026/11/12 11:50:06 INFO Vacuumed table table=pending_objects29432026/11/12 11:50:06 INFO Vacuumed table table=multipart_uploads29442026/11/12 11:50:06 INFO Received complete push request method=POST path=/29452026/11/12 11:50:06 INFO Vacuumed table table=closures29462026/11/12 11:50:06 INFO Vacuumed table table=objects29472026/11/12 11:50:06 WARN Force mode enabled - objects will be deleted immediately without grace period29482026/11/12 11:50:06 INFO Completed push id=6 objects=6 duration=2.35587ms29492026/11/12 11:50:06 INFO Received complete push request method=POST path=/29502026/11/12 11:50:06 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."29512026/11/12 11:50:06 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."29522026/11/12 11:50:06 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."29532026/11/12 11:50:06 INFO Received uploads request method=POST path=/29542026/11/12 11:50:06 INFO Completed push id=6 objects=5 duration=4.27651ms29552026/11/12 11:50:06 INFO Received push request method=POST path=/29562026/11/12 11:50:06 INFO Received uploads request method=POST path=/29572026/11/12 11:50:06 INFO Received complete push request method=POST path=/29582026/11/12 11:50:06 INFO Completed push id=5 objects=3 duration=3.61488ms29592026/11/12 11:50:06 INFO Received push request method=POST path=/29602026/11/12 11:50:06 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."29612026/11/12 11:50:06 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."29622026/11/12 11:50:06 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."29632026/11/12 11:50:06 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."29642026/11/12 11:50:06 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."29652026/11/12 11:50:06 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."29662026/11/12 11:50:06 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."29672026/11/12 11:50:06 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."29682026/11/12 11:50:06 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."29692026/11/12 11:50:06 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."29702026/11/12 11:50:06 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."29712026/11/12 11:50:06 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."29722026/11/12 11:50:06 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."29732026/11/12 11:50:06 ERROR failed to remove object object=realisations/sha256:aaaaaaaa!out.doi error="Access Denied."29742026/11/12 11:50:06 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."29752026/11/12 11:50:06 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."29762026/11/12 11:50:06 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."29772026/11/12 11:50:06 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."29782026/11/12 11:50:06 INFO Received push request method=POST path=/29792026/11/12 11:50:06 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."29802026/11/12 11:50:06 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."29812026/11/12 11:50:06 INFO Received push request method=POST path=/29822026/11/12 11:50:06 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."29832026/11/12 11:50:06 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."29842026/11/12 11:50:06 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."29852026/11/12 11:50:06 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."29862026/11/12 11:50:06 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."29872026/11/12 11:50:06 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."29882026/11/12 11:50:06 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."29892026/11/12 11:50:06 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."29902026/11/12 11:50:06 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."29912026/11/12 11:50:06 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."29922026/11/12 11:50:06 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."29932026/11/12 11:50:06 INFO Received complete upload request method=POST path=/29942026/11/12 11:50:06 INFO Received uploads request method=POST path=/29952026/11/12 11:50:06 INFO Received push request method=POST path=/29962026/11/12 11:50:06 INFO Completed upload id=11 objects=6 duration=3.092ms29972026/11/12 11:50:06 INFO Received complete push request method=POST path=/29982026/11/12 11:50:06 INFO Received push request method=POST path=/29992026/11/12 11:50:06 INFO Completed push id=5 objects=3 duration=1.85253ms30002026/11/12 11:50:06 INFO Received complete push request method=POST path=/30012026/11/12 11:50:06 INFO Completed push id=12 objects=3 duration=2.18609ms30022026/11/12 11:50:06 INFO Received push request method=POST path=/30032026/11/12 11:50:06 INFO Aborted multipart uploads count=0 kept=030042026/11/12 11:50:06 INFO Received push request method=POST path=/30052026/11/12 11:50:06 INFO Received complete push request method=POST path=/30062026/11/12 11:50:06 WARN Force mode enabled - objects will be deleted immediately without grace period30072026/11/12 11:50:06 INFO Completed push id=5 objects=6 duration=1.87509ms30082026/11/12 11:50:06 INFO Received uploads request method=POST path=/30092026/11/12 11:50:06 INFO Received uploads request method=POST path=/30102026/11/12 11:50:06 INFO Received complete upload request method=POST path=/30112026/11/12 11:50:06 INFO Completed upload id=3 objects=5 duration=1.737029ms30122026/11/12 11:50:06 INFO Received uploads request method=POST path=/30132026/11/12 11:50:06 INFO Received push request method=POST path=/30142026/11/12 11:50:06 INFO Received uploads request method=POST path=/30152026/11/12 11:50:06 INFO Received complete upload request method=POST path=/30162026/11/12 11:50:06 INFO Completed upload id=2 objects=2 duration=2.05054ms30172026/11/12 11:50:06 INFO Received uploads request method=POST path=/30182026/11/12 11:50:06 INFO Aborted multipart uploads count=0 kept=030192026/11/12 11:50:06 INFO Received complete push request method=POST path=/30202026/11/12 11:50:06 WARN Force mode enabled - objects will be deleted immediately without grace period30212026/11/12 11:50:06 INFO Completed push id=8 objects=2 duration=2.1853ms30222026/11/12 11:50:06 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030232026/11/12 11:50:06 INFO Vacuumed table table=pending_closures30242026/11/12 11:50:06 INFO Received complete push request method=POST path=/30252026/11/12 11:50:06 INFO Vacuumed table table=commit_receipts30262026/11/12 11:50:06 INFO Aborted multipart uploads count=0 kept=030272026/11/12 11:50:06 WARN Force mode enabled - objects will be deleted immediately without grace period30282026/11/12 11:50:06 INFO Completed push id=10 objects=2 duration=2.71764ms30292026/11/12 11:50:06 INFO Vacuumed table table=pending_objects30302026/11/12 11:50:06 INFO Vacuumed table table=multipart_uploads30312026/11/12 11:50:06 INFO Vacuumed table table=closures30322026/11/12 11:50:06 INFO Vacuumed table table=objects30332026/11/12 11:50:06 INFO Aborted multipart uploads count=0 kept=030342026/11/12 11:50:06 INFO Received complete push request method=POST path=/30352026/11/12 11:50:06 INFO Received push request method=POST path=/30362026/11/12 11:50:06 INFO Aborted multipart uploads count=0 kept=030372026/11/12 11:50:06 INFO Completed push id=7 objects=27 duration=2.69238ms30382026/11/12 11:50:06 INFO Received uploads request method=POST path=/30392026/11/12 11:50:06 INFO Received push request method=POST path=/30402026/11/12 11:50:06 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."30412026/11/12 11:50:06 INFO Received uploads request method=POST path=/30422026/11/12 11:50:06 INFO Received complete push request method=POST path=/30432026/11/12 11:50:06 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=030442026/11/12 11:50:06 INFO Completed push id=6 objects=5 duration=2.33813ms30452026/11/12 11:50:06 INFO Vacuumed table table=pending_closures30462026/11/12 11:50:06 INFO Vacuumed table table=commit_receipts30472026/11/12 11:50:06 INFO Vacuumed table table=pending_objects30482026/11/12 11:50:06 INFO Vacuumed table table=multipart_uploads30492026/11/12 11:50:06 INFO Vacuumed table table=closures30502026/11/12 11:50:06 INFO Aborted multipart uploads count=0 kept=030512026/11/12 11:50:06 INFO Vacuumed table table=objects30522026/11/12 11:50:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=4 objects-deleted-after-grace-period=0 objects-failed-to-delete=030532026/11/12 11:50:06 INFO Received uploads request method=POST path=/30542026/11/12 11:50:06 INFO Vacuumed table table=pending_closures30552026/11/12 11:50:06 INFO Received push request method=POST path=/30562026/11/12 11:50:06 INFO Vacuumed table table=commit_receipts30572026/11/12 11:50:06 INFO Received uploads request method=POST path=/30582026/11/12 11:50:06 INFO Received complete upload request method=POST path=/30592026/11/12 11:50:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=16 objects-failed-to-delete=030602026/11/12 11:50:06 INFO Received complete upload request method=POST path=/30612026/11/12 11:50:06 INFO Vacuumed table table=pending_closures30622026/11/12 11:50:06 INFO Vacuumed table table=pending_objects30632026/11/12 11:50:06 INFO Aborted multipart uploads count=0 kept=030642026/11/12 11:50:06 INFO Completed upload id=4 objects=14 duration=3.31147ms30652026/11/12 11:50:06 INFO Vacuumed table table=multipart_uploads30662026/11/12 11:50:06 INFO Completed upload id=12 objects=21 duration=1.4327ms30672026/11/12 11:50:06 INFO Vacuumed table table=commit_receipts30682026/11/12 11:50:06 INFO Received complete push request method=POST path=/30692026/11/12 11:50:06 INFO Vacuumed table table=closures30702026/11/12 11:50:06 INFO Received push request method=POST path=/30712026/11/12 11:50:06 INFO Completed push id=8 objects=6 duration=2.594241ms30722026/11/12 11:50:06 INFO Vacuumed table table=pending_objects30732026/11/12 11:50:06 INFO Vacuumed table table=multipart_uploads30742026/11/12 11:50:06 INFO Vacuumed table table=objects30752026/11/12 11:50:06 WARN Force mode enabled - objects will be deleted immediately without grace period30762026/11/12 11:50:06 INFO Vacuumed table table=closures30772026/11/12 11:50:06 INFO Vacuumed table table=objects30782026/11/12 11:50:06 INFO Received complete push request method=POST path=/30792026/11/12 11:50:06 INFO Received complete upload request method=POST path=/30802026/11/12 11:50:06 INFO Completed upload id=3 objects=9 duration=1.83719ms30812026/11/12 11:50:06 INFO Completed push id=9 objects=5 duration=2.91132ms30822026/11/12 11:50:06 INFO Received uploads request method=POST path=/30832026/11/12 11:50:06 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."30842026/11/12 11:50:06 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.narinfo error="Access Denied."30852026/11/12 11:50:06 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."30862026/11/12 11:50:06 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."30872026/11/12 11:50:06 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."30882026/11/12 11:50:06 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."30892026/11/12 11:50:06 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."30902026/11/12 11:50:06 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."30912026/11/12 11:50:06 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."30922026/11/12 11:50:06 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."30932026/11/12 11:50:06 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."30942026/11/12 11:50:06 INFO Aborted multipart uploads count=0 kept=030952026/11/12 11:50:06 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."30962026/11/12 11:50:06 ERROR failed to remove object object=nar/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.nar.zst error="Access Denied."30972026/11/12 11:50:06 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."30982026/11/12 11:50:06 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."30992026/11/12 11:50:06 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."31002026/11/12 11:50:06 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."31012026/11/12 11:50:06 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."31022026/11/12 11:50:06 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."31032026/11/12 11:50:06 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."31042026/11/12 11:50:06 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."31052026/11/12 11:50:06 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."31062026/11/12 11:50:06 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."31072026/11/12 11:50:06 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."31082026/11/12 11:50:06 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."31092026/11/12 11:50:06 INFO Received push request method=POST path=/31102026/11/12 11:50:06 INFO Received push request method=POST path=/31112026/11/12 11:50:06 INFO Received push request method=POST path=/31122026/11/12 11:50:06 INFO Aborted multipart uploads count=0 kept=031132026/11/12 11:50:06 WARN Force mode enabled - objects will be deleted immediately without grace period31142026/11/12 11:50:06 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031152026/11/12 11:50:06 INFO Vacuumed table table=pending_closures31162026/11/12 11:50:06 INFO Vacuumed table table=commit_receipts31172026/11/12 11:50:06 INFO Vacuumed table table=pending_objects31182026/11/12 11:50:06 INFO Vacuumed table table=multipart_uploads31192026/11/12 11:50:06 INFO Vacuumed table table=closures31202026/11/12 11:50:06 INFO Received complete push request method=POST path=/31212026/11/12 11:50:06 INFO Aborted multipart uploads count=0 kept=031222026/11/12 11:50:06 INFO Vacuumed table table=objects31232026/11/12 11:50:06 INFO Received push request method=POST path=/31242026/11/12 11:50:06 WARN Force mode enabled - objects will be deleted immediately without grace period31252026/11/12 11:50:06 INFO Completed push id=2 objects=10 duration=3.5392ms31262026/11/12 11:50:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031272026/11/12 11:50:06 INFO Received push request method=POST path=/31282026/11/12 11:50:06 INFO Vacuumed table table=pending_closures31292026/11/12 11:50:06 INFO Vacuumed table table=commit_receipts31302026/11/12 11:50:06 INFO Received complete upload request method=POST path=/31312026/11/12 11:50:06 INFO Received push request method=POST path=/31322026/11/12 11:50:06 INFO Completed upload id=16 objects=23 duration=1.99405ms31332026/11/12 11:50:06 INFO Vacuumed table table=pending_objects31342026/11/12 11:50:06 INFO Vacuumed table table=multipart_uploads31352026/11/12 11:50:06 INFO Received complete push request method=POST path=/31362026/11/12 11:50:06 INFO Vacuumed table table=closures31372026/11/12 11:50:06 INFO Vacuumed table table=objects31382026/11/12 11:50:06 INFO Completed push id=13 objects=23 duration=1.97445ms31392026/11/12 11:50:06 INFO Received push request method=POST path=/31402026/11/12 11:50:06 INFO Received uploads request method=POST path=/31412026/11/12 11:50:06 INFO Received uploads request method=POST path=/31422026/11/12 11:50:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=031432026/11/12 11:50:06 INFO Vacuumed table table=pending_closures31442026/11/12 11:50:06 INFO Vacuumed table table=commit_receipts31452026/11/12 11:50:06 INFO Received push request method=POST path=/31462026/11/12 11:50:06 INFO Vacuumed table table=pending_objects31472026/11/12 11:50:06 INFO Vacuumed table table=multipart_uploads31482026/11/12 11:50:06 INFO Vacuumed table table=closures31492026/11/12 11:50:06 INFO Vacuumed table table=objects31502026/11/12 11:50:06 INFO Received complete upload request method=POST path=/31512026/11/12 11:50:06 INFO Received push request method=POST path=/31522026/11/12 11:50:06 INFO Received uploads request method=POST path=/31532026/11/12 11:50:06 INFO Received complete push request method=POST path=/31542026/11/12 11:50:06 INFO Completed upload id=15 objects=6 duration=1.3445ms31552026/11/12 11:50:06 INFO Completed push id=5 objects=6 duration=1.95171ms31562026/11/12 11:50:06 INFO Received push request method=POST path=/31572026/11/12 11:50:06 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."31582026/11/12 11:50:06 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."31592026/11/12 11:50:06 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."31602026/11/12 11:50:06 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."31612026/11/12 11:50:06 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."31622026/11/12 11:50:06 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."31632026/11/12 11:50:06 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."31642026/11/12 11:50:06 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."31652026/11/12 11:50:06 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."31662026/11/12 11:50:06 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."31672026/11/12 11:50:06 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."31682026/11/12 11:50:06 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."31692026/11/12 11:50:06 INFO Received push request method=POST path=/31702026/11/12 11:50:06 INFO Received push request method=POST path=/31712026/11/12 11:50:06 INFO Received push request method=POST path=/31722026/11/12 11:50:06 INFO Aborted multipart uploads count=0 kept=031732026/11/12 11:50:06 INFO Received uploads request method=POST path=/31742026/11/12 11:50:06 INFO Received push request method=POST path=/31752026/11/12 11:50:06 WARN Force mode enabled - objects will be deleted immediately without grace period31762026/11/12 11:50:06 INFO Aborted multipart uploads count=0 kept=031772026/11/12 11:50:06 INFO Received uploads request method=POST path=/31782026/11/12 11:50:06 INFO Received uploads request method=POST path=/31792026/11/12 11:50:06 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031802026/11/12 11:50:06 WARN Force mode enabled - objects will be deleted immediately without grace period31812026/11/12 11:50:06 INFO Vacuumed table table=pending_closures31822026/11/12 11:50:06 INFO Vacuumed table table=commit_receipts31832026/11/12 11:50:06 INFO Vacuumed table table=pending_objects31842026/11/12 11:50:06 INFO Vacuumed table table=multipart_uploads31852026/11/12 11:50:06 INFO Received uploads request method=POST path=/31862026/11/12 11:50:06 INFO Vacuumed table table=closures31872026/11/12 11:50:06 INFO Received push request method=POST path=/31882026/11/12 11:50:06 INFO Vacuumed table table=objects31892026/11/12 11:50:06 INFO Received complete push request method=POST path=/31902026/11/12 11:50:06 INFO Aborted multipart uploads count=0 kept=031912026/11/12 11:50:06 INFO Received push request method=POST path=/31922026/11/12 11:50:06 INFO Completed push id=8 objects=7 duration=1.42251ms31932026/11/12 11:50:06 INFO Received push request method=POST path=/31942026/11/12 11:50:06 INFO Received uploads request method=POST path=/31952026/11/12 11:50:06 INFO Received complete push request method=POST path=/31962026/11/12 11:50:06 INFO Aborted multipart uploads count=0 kept=031972026/11/12 11:50:06 INFO Received push request method=POST path=/31982026/11/12 11:50:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031992026/11/12 11:50:06 INFO Vacuumed table table=pending_closures32002026/11/12 11:50:06 INFO Vacuumed table table=commit_receipts32012026/11/12 11:50:06 INFO Completed push id=6 objects=26 duration=4.65335ms32022026/11/12 11:50:06 INFO Received uploads request method=POST path=/32032026/11/12 11:50:06 INFO Vacuumed table table=pending_objects32042026/11/12 11:50:06 INFO Vacuumed table table=multipart_uploads32052026/11/12 11:50:06 INFO Vacuumed table table=closures32062026/11/12 11:50:06 INFO Vacuumed table table=objects32072026/11/12 11:50:06 INFO Received complete upload request method=POST path=/32082026/11/12 11:50:06 INFO Completed upload id=17 objects=7 duration=1.43938ms32092026/11/12 11:50:06 INFO Received push request method=POST path=/32102026/11/12 11:50:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=032112026/11/12 11:50:06 INFO Received uploads request method=POST path=/32122026/11/12 11:50:06 INFO Vacuumed table table=pending_closures32132026/11/12 11:50:06 INFO Vacuumed table table=commit_receipts32142026/11/12 11:50:06 INFO Received uploads request method=POST path=/32152026/11/12 11:50:06 INFO Vacuumed table table=pending_objects32162026/11/12 11:50:06 INFO Vacuumed table table=multipart_uploads32172026/11/12 11:50:06 INFO Vacuumed table table=closures32182026/11/12 11:50:06 INFO Vacuumed table table=objects32192026/11/12 11:50:06 INFO Received push request method=POST path=/32202026/11/12 11:50:06 INFO Aborted multipart uploads count=0 kept=032212026/11/12 11:50:06 INFO Received push request method=POST path=/32222026/11/12 11:50:06 WARN Force mode enabled - objects will be deleted immediately without grace period32232026/11/12 11:50:06 INFO Aborted multipart uploads count=0 kept=032242026/11/12 11:50:06 INFO Aborted multipart uploads count=0 kept=032252026/11/12 11:50:06 INFO Received complete push request method=POST path=/32262026/11/12 11:50:06 INFO Received complete push request method=POST path=/32272026/11/12 11:50:06 INFO Received complete upload request method=POST path=/32282026/11/12 11:50:06 WARN Force mode enabled - objects will be deleted immediately without grace period32292026/11/12 11:50:06 INFO Completed upload id=10 objects=17 duration=1.68687ms32302026/11/12 11:50:06 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032312026/11/12 11:50:06 INFO Vacuumed table table=pending_closures32322026/11/12 11:50:06 INFO Completed push id=5 objects=8 duration=4.31087ms32332026/11/12 11:50:06 INFO Vacuumed table table=commit_receipts32342026/11/12 11:50:06 INFO Completed push id=5 objects=9 duration=5.04706ms32352026/11/12 11:50:06 INFO Received push request method=POST path=/32362026/11/12 11:50:06 INFO Vacuumed table table=pending_objects32372026/11/12 11:50:06 INFO Vacuumed table table=multipart_uploads32382026/11/12 11:50:06 INFO Vacuumed table table=closures32392026/11/12 11:50:06 INFO Vacuumed table table=objects32402026/11/12 11:50:06 INFO Received push request method=POST path=/32412026/11/12 11:50:06 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."32422026/11/12 11:50:06 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."32432026/11/12 11:50:06 INFO Received push request method=POST path=/32442026/11/12 11:50:06 INFO Received push request method=POST path=/32452026/11/12 11:50:06 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."32462026/11/12 11:50:06 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."32472026/11/12 11:50:06 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."32482026/11/12 11:50:06 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."32492026/11/12 11:50:06 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."32502026/11/12 11:50:06 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."32512026/11/12 11:50:06 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."32522026/11/12 11:50:06 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."32532026/11/12 11:50:06 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."32542026/11/12 11:50:06 INFO Aborted multipart uploads count=0 kept=032552026/11/12 11:50:06 INFO Received complete upload request method=POST path=/32562026/11/12 11:50:06 INFO Completed upload id=9 objects=16 duration=1.26094ms32572026/11/12 11:50:06 WARN Force mode enabled - objects will be deleted immediately without grace period32582026/11/12 11:50:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032592026/11/12 11:50:06 INFO Received complete upload request method=POST path=/32602026/11/12 11:50:06 INFO Received uploads request method=POST path=/32612026/11/12 11:50:06 INFO Vacuumed table table=pending_closures32622026/11/12 11:50:06 INFO Received push request method=POST path=/32632026/11/12 11:50:06 INFO Vacuumed table table=commit_receipts32642026/11/12 11:50:06 INFO Completed upload id=10 objects=23 duration=2.98015ms32652026/11/12 11:50:06 INFO Vacuumed table table=pending_objects32662026/11/12 11:50:06 INFO Received push request method=POST path=/32672026/11/12 11:50:06 INFO Vacuumed table table=multipart_uploads32682026/11/12 11:50:06 INFO Vacuumed table table=closures32692026/11/12 11:50:06 INFO Aborted multipart uploads count=0 kept=032702026/11/12 11:50:06 INFO Vacuumed table table=objects32712026/11/12 11:50:06 INFO Received push request method=POST path=/32722026/11/12 11:50:06 INFO Aborted multipart uploads count=0 kept=032732026/11/12 11:50:06 INFO Aborted multipart uploads count=0 kept=032742026/11/12 11:50:06 WARN Force mode enabled - objects will be deleted immediately without grace period32752026/11/12 11:50:06 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032762026/11/12 11:50:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=032772026/11/12 11:50:06 INFO Vacuumed table table=pending_closures32782026/11/12 11:50:06 INFO Vacuumed table table=pending_closures32792026/11/12 11:50:06 INFO Vacuumed table table=commit_receipts32802026/11/12 11:50:06 INFO Vacuumed table table=commit_receipts32812026/11/12 11:50:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=8 objects-deleted-after-grace-period=8 objects-failed-to-delete=032822026/11/12 11:50:06 INFO Received push request method=POST path=/32832026/11/12 11:50:06 INFO Vacuumed table table=pending_objects32842026/11/12 11:50:06 INFO Vacuumed table table=multipart_uploads32852026/11/12 11:50:06 INFO Vacuumed table table=pending_objects32862026/11/12 11:50:06 INFO Vacuumed table table=multipart_uploads32872026/11/12 11:50:06 INFO Received push request method=POST path=/32882026/11/12 11:50:06 INFO Vacuumed table table=pending_closures32892026/11/12 11:50:06 INFO Vacuumed table table=closures32902026/11/12 11:50:06 INFO Vacuumed table table=commit_receipts32912026/11/12 11:50:06 INFO Vacuumed table table=closures32922026/11/12 11:50:06 INFO Vacuumed table table=objects32932026/11/12 11:50:06 INFO Vacuumed table table=pending_objects32942026/11/12 11:50:06 INFO Vacuumed table table=objects32952026/11/12 11:50:06 INFO Vacuumed table table=multipart_uploads32962026/11/12 11:50:06 INFO Received uploads request method=POST path=/32972026/11/12 11:50:06 INFO Vacuumed table table=closures32982026/11/12 11:50:06 INFO Vacuumed table table=objects32992026/11/12 11:50:06 INFO Received push request method=POST path=/33002026/11/12 11:50:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=9 objects-failed-to-delete=033012026/11/12 11:50:06 INFO Received push request method=POST path=/33022026/11/12 11:50:06 INFO Received uploads request method=POST path=/33032026/11/12 11:50:06 INFO Vacuumed table table=pending_closures33042026/11/12 11:50:06 INFO Vacuumed table table=commit_receipts33052026/11/12 11:50:06 INFO Vacuumed table table=pending_objects33062026/11/12 11:50:06 INFO Vacuumed table table=multipart_uploads33072026/11/12 11:50:06 INFO Received push request method=POST path=/33082026/11/12 11:50:06 INFO Vacuumed table table=closures33092026/11/12 11:50:06 INFO Received uploads request method=POST path=/33102026/11/12 11:50:06 INFO Vacuumed table table=objects33112026/11/12 11:50:06 INFO Received push request method=POST path=/33122026/11/12 11:50:06 INFO Received complete upload request method=POST path=/33132026/11/12 11:50:06 INFO Received uploads request method=POST path=/33142026/11/12 11:50:06 INFO Completed upload id=14 objects=6 duration=1.53853ms33152026/11/12 11:50:06 INFO Received uploads request method=POST path=/33162026/11/12 11:50:06 INFO Received push request method=POST path=/33172026/11/12 11:50:06 INFO Aborted multipart uploads count=0 kept=033182026/11/12 11:50:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033192026/11/12 11:50:06 INFO Received push request method=POST path=/33202026/11/12 11:50:06 INFO Vacuumed table table=pending_closures33212026/11/12 11:50:06 INFO Received uploads request method=POST path=/33222026/11/12 11:50:06 INFO Received push request method=POST path=/33232026/11/12 11:50:06 INFO Vacuumed table table=commit_receipts33242026/11/12 11:50:06 INFO Vacuumed table table=pending_objects33252026/11/12 11:50:06 INFO Vacuumed table table=multipart_uploads33262026/11/12 11:50:06 INFO Vacuumed table table=closures33272026/11/12 11:50:06 INFO Vacuumed table table=objects33282026/11/12 11:50:06 INFO Received uploads request method=POST path=/33292026/11/12 11:50:06 INFO Received push request method=POST path=/33302026/11/12 11:50:06 INFO Received complete push request method=POST path=/33312026/11/12 11:50:06 INFO Received complete push request method=POST path=/33322026/11/12 11:50:06 INFO Received uploads request method=POST path=/33332026/11/12 11:50:06 INFO Completed push id=10 objects=3 duration=1.3366ms33342026/11/12 11:50:06 INFO Received uploads request method=POST path=/33352026/11/12 11:50:06 INFO Received complete upload request method=POST path=/33362026/11/12 11:50:06 INFO Aborted multipart uploads count=0 kept=033372026/11/12 11:50:06 INFO Received complete upload request method=POST path=/33382026/11/12 11:50:06 INFO Completed upload id=11 objects=6 duration=1.99372ms33392026/11/12 11:50:06 INFO Completed push id=15 objects=24 duration=5.02226ms33402026/11/12 11:50:06 INFO Completed upload id=10 objects=2 duration=3.04981ms33412026/11/12 11:50:06 INFO Received uploads request method=POST path=/33422026/11/12 11:50:06 INFO Received complete push request method=POST path=/33432026/11/12 11:50:06 INFO Received uploads request method=POST path=/33442026/11/12 11:50:06 INFO Completed push id=14 objects=6 duration=2.45102ms33452026/11/12 11:50:06 INFO Aborted multipart uploads count=0 kept=033462026/11/12 11:50:06 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=033472026/11/12 11:50:06 INFO Received complete upload request method=POST path=/33482026/11/12 11:50:06 INFO Vacuumed table table=pending_closures33492026/11/12 11:50:06 WARN Force mode enabled - objects will be deleted immediately without grace period33502026/11/12 11:50:06 INFO Vacuumed table table=commit_receipts33512026/11/12 11:50:06 INFO Completed upload id=15 objects=4 duration=2.68262ms33522026/11/12 11:50:06 INFO Vacuumed table table=pending_objects33532026/11/12 11:50:06 INFO Vacuumed table table=multipart_uploads33542026/11/12 11:50:06 INFO Received push request method=POST path=/33552026/11/12 11:50:06 INFO Vacuumed table table=closures33562026/11/12 11:50:06 INFO Received push request method=POST path=/33572026/11/12 11:50:06 INFO Vacuumed table table=objects33582026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=033592026/11/12 11:50:07 INFO Received uploads request method=POST path=/33602026/11/12 11:50:07 INFO Received push request method=POST path=/33612026/11/12 11:50:07 INFO Received uploads request method=POST path=/33622026/11/12 11:50:07 WARN Force mode enabled - objects will be deleted immediately without grace period33632026/11/12 11:50:07 INFO Received uploads request method=POST path=/33642026/11/12 11:50:07 INFO Received complete push request method=POST path=/33652026/11/12 11:50:07 INFO Received complete upload request method=POST path=/33662026/11/12 11:50:07 INFO Received push request method=POST path=/33672026/11/12 11:50:07 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033682026/11/12 11:50:07 INFO Completed upload id=7 objects=3 duration=1.64315ms33692026/11/12 11:50:07 INFO Vacuumed table table=pending_closures33702026/11/12 11:50:07 INFO Vacuumed table table=commit_receipts33712026/11/12 11:50:07 INFO Vacuumed table table=pending_objects33722026/11/12 11:50:07 INFO Completed push id=3 objects=21 duration=5.51218ms33732026/11/12 11:50:07 INFO Vacuumed table table=multipart_uploads33742026/11/12 11:50:07 INFO Vacuumed table table=closures33752026/11/12 11:50:07 INFO Received uploads request method=POST path=/33762026/11/12 11:50:07 INFO Received push request method=POST path=/33772026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=033782026/11/12 11:50:07 INFO Vacuumed table table=objects33792026/11/12 11:50:07 INFO Received push request method=POST path=/33802026/11/12 11:50:07 INFO Received push request method=POST path=/33812026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=033822026/11/12 11:50:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033832026/11/12 11:50:07 INFO Vacuumed table table=pending_closures33842026/11/12 11:50:07 INFO Vacuumed table table=commit_receipts33852026/11/12 11:50:07 INFO Vacuumed table table=pending_objects33862026/11/12 11:50:07 INFO Vacuumed table table=multipart_uploads33872026/11/12 11:50:07 INFO Vacuumed table table=closures33882026/11/12 11:50:07 INFO Received uploads request method=POST path=/33892026/11/12 11:50:07 INFO Received push request method=POST path=/33902026/11/12 11:50:07 INFO Vacuumed table table=objects33912026/11/12 11:50:07 INFO Received uploads request method=POST path=/33922026/11/12 11:50:07 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."33932026/11/12 11:50:07 INFO Received push request method=POST path=/33942026/11/12 11:50:07 INFO Received uploads request method=POST path=/33952026/11/12 11:50:07 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=033962026/11/12 11:50:07 INFO Received push request method=POST path=/33972026/11/12 11:50:07 INFO Received uploads request method=POST path=/33982026/11/12 11:50:07 INFO Vacuumed table table=pending_closures33992026/11/12 11:50:07 INFO Vacuumed table table=commit_receipts34002026/11/12 11:50:07 INFO Vacuumed table table=pending_objects34012026/11/12 11:50:07 INFO Received push request method=POST path=/34022026/11/12 11:50:07 INFO Vacuumed table table=multipart_uploads34032026/11/12 11:50:07 INFO Received push request method=POST path=/34042026/11/12 11:50:07 INFO Received complete push request method=POST path=/34052026/11/12 11:50:07 INFO Vacuumed table table=closures34062026/11/12 11:50:07 INFO Vacuumed table table=objects34072026/11/12 11:50:07 INFO Completed push id=12 objects=15 duration=2.69742ms34082026/11/12 11:50:07 INFO Received uploads request method=POST path=/34092026/11/12 11:50:07 INFO Received complete push request method=POST path=/34102026/11/12 11:50:07 INFO Completed push id=18 objects=23 duration=1.98383ms34112026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=034122026/11/12 11:50:07 INFO Received push request method=POST path=/34132026/11/12 11:50:07 WARN Force mode enabled - objects will be deleted immediately without grace period34142026/11/12 11:50:07 INFO Received uploads request method=POST path=/34152026/11/12 11:50:07 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034162026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=034172026/11/12 11:50:07 INFO Vacuumed table table=pending_closures34182026/11/12 11:50:07 INFO Vacuumed table table=commit_receipts34192026/11/12 11:50:07 WARN Force mode enabled - objects will be deleted immediately without grace period34202026/11/12 11:50:07 INFO Vacuumed table table=pending_objects34212026/11/12 11:50:07 INFO Vacuumed table table=multipart_uploads34222026/11/12 11:50:07 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034232026/11/12 11:50:07 INFO Vacuumed table table=closures34242026/11/12 11:50:07 INFO Vacuumed table table=pending_closures34252026/11/12 11:50:07 INFO Vacuumed table table=objects34262026/11/12 11:50:07 INFO Vacuumed table table=commit_receipts34272026/11/12 11:50:07 INFO Received push request method=POST path=/34282026/11/12 11:50:07 INFO Received push request method=POST path=/34292026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=034302026/11/12 11:50:07 INFO Vacuumed table table=pending_objects34312026/11/12 11:50:07 INFO Vacuumed table table=multipart_uploads34322026/11/12 11:50:07 INFO Vacuumed table table=closures34332026/11/12 11:50:07 INFO Vacuumed table table=objects34342026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=034352026/11/12 11:50:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034362026/11/12 11:50:07 INFO Received push request method=POST path=/34372026/11/12 11:50:07 INFO Vacuumed table table=pending_closures34382026/11/12 11:50:07 INFO Received complete upload request method=POST path=/34392026/11/12 11:50:07 INFO Vacuumed table table=commit_receipts34402026/11/12 11:50:07 INFO Received uploads request method=POST path=/34412026/11/12 11:50:07 INFO Completed upload id=16 objects=3 duration=1.26027ms34422026/11/12 11:50:07 INFO Vacuumed table table=pending_objects34432026/11/12 11:50:07 INFO Vacuumed table table=multipart_uploads34442026/11/12 11:50:07 INFO Received uploads request method=POST path=/34452026/11/12 11:50:07 INFO Vacuumed table table=closures34462026/11/12 11:50:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=1 objects-deleted-after-grace-period=0 objects-failed-to-delete=034472026/11/12 11:50:07 INFO Received push request method=POST path=/34482026/11/12 11:50:07 INFO Received complete push request method=POST path=/34492026/11/12 11:50:07 INFO Vacuumed table table=objects34502026/11/12 11:50:07 INFO Vacuumed table table=pending_closures34512026/11/12 11:50:07 INFO Vacuumed table table=commit_receipts34522026/11/12 11:50:07 INFO Vacuumed table table=pending_objects34532026/11/12 11:50:07 INFO Received complete upload request method=POST path=/34542026/11/12 11:50:07 INFO Received uploads request method=POST path=/34552026/11/12 11:50:07 INFO Vacuumed table table=multipart_uploads34562026/11/12 11:50:07 INFO Completed push id=8 objects=6 duration=3.75519ms34572026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=034582026/11/12 11:50:07 INFO Vacuumed table table=closures34592026/11/12 11:50:07 INFO Completed upload id=10 objects=6 duration=3.06315ms34602026/11/12 11:50:07 INFO Vacuumed table table=objects34612026/11/12 11:50:07 INFO Received complete upload request method=POST path=/34622026/11/12 11:50:07 WARN Force mode enabled - objects will be deleted immediately without grace period34632026/11/12 11:50:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034642026/11/12 11:50:07 INFO Completed upload id=8 objects=3 duration=2.58134ms34652026/11/12 11:50:07 INFO Vacuumed table table=pending_closures34662026/11/12 11:50:07 INFO Vacuumed table table=commit_receipts34672026/11/12 11:50:07 INFO Vacuumed table table=pending_objects34682026/11/12 11:50:07 INFO Vacuumed table table=multipart_uploads34692026/11/12 11:50:07 INFO Received uploads request method=POST path=/34702026/11/12 11:50:07 INFO Vacuumed table table=closures34712026/11/12 11:50:07 INFO Vacuumed table table=objects34722026/11/12 11:50:07 INFO Received uploads request method=POST path=/34732026/11/12 11:50:07 INFO Received push request method=POST path=/34742026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=034752026/11/12 11:50:07 INFO Received complete upload request method=POST path=/34762026/11/12 11:50:07 INFO Completed upload id=19 objects=25 duration=2.01411ms34772026/11/12 11:50:07 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034782026/11/12 11:50:07 INFO Received complete push request method=POST path=/34792026/11/12 11:50:07 INFO Vacuumed table table=pending_closures34802026/11/12 11:50:07 INFO Vacuumed table table=commit_receipts34812026/11/12 11:50:07 INFO Completed push id=11 objects=9 duration=2.1155ms34822026/11/12 11:50:07 INFO Vacuumed table table=pending_objects34832026/11/12 11:50:07 INFO Vacuumed table table=multipart_uploads34842026/11/12 11:50:07 INFO Received uploads request method=POST path=/34852026/11/12 11:50:07 INFO Vacuumed table table=closures34862026/11/12 11:50:07 INFO Vacuumed table table=objects34872026/11/12 11:50:07 INFO Received complete upload request method=POST path=/34882026/11/12 11:50:07 INFO Received uploads request method=POST path=/34892026/11/12 11:50:07 INFO Completed upload id=6 objects=6 duration=1.71048ms34902026/11/12 11:50:07 INFO Received push request method=POST path=/34912026/11/12 11:50:07 INFO Received complete push request method=POST path=/34922026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=034932026/11/12 11:50:07 INFO Received complete push request method=POST path=/34942026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=034952026/11/12 11:50:07 INFO Completed push id=11 objects=8 duration=3.745259ms34962026/11/12 11:50:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034972026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=034982026/11/12 11:50:07 INFO Completed push id=13 objects=12 duration=4.409039ms34992026/11/12 11:50:07 WARN Force mode enabled - objects will be deleted immediately without grace period35002026/11/12 11:50:07 INFO Vacuumed table table=pending_closures35012026/11/12 11:50:07 INFO Vacuumed table table=commit_receipts35022026/11/12 11:50:07 WARN Force mode enabled - objects will be deleted immediately without grace period35032026/11/12 11:50:07 INFO Received push request method=POST path=/35042026/11/12 11:50:07 INFO Vacuumed table table=pending_objects35052026/11/12 11:50:07 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035062026/11/12 11:50:07 INFO Vacuumed table table=multipart_uploads35072026/11/12 11:50:07 INFO Vacuumed table table=closures35082026/11/12 11:50:07 INFO Vacuumed table table=pending_closures35092026/11/12 11:50:07 INFO Vacuumed table table=objects35102026/11/12 11:50:07 INFO Vacuumed table table=commit_receipts35112026/11/12 11:50:07 INFO Received push request method=POST path=/35122026/11/12 11:50:07 INFO Vacuumed table table=pending_objects35132026/11/12 11:50:07 INFO Received uploads request method=POST path=/35142026/11/12 11:50:07 INFO Vacuumed table table=multipart_uploads35152026/11/12 11:50:07 INFO Vacuumed table table=closures35162026/11/12 11:50:07 INFO Vacuumed table table=objects35172026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=035182026/11/12 11:50:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=035192026/11/12 11:50:07 INFO Vacuumed table table=pending_closures35202026/11/12 11:50:07 INFO Vacuumed table table=commit_receipts35212026/11/12 11:50:07 INFO Received uploads request method=POST path=/35222026/11/12 11:50:07 WARN Force mode enabled - objects will be deleted immediately without grace period35232026/11/12 11:50:07 INFO Vacuumed table table=pending_objects35242026/11/12 11:50:07 INFO Vacuumed table table=multipart_uploads35252026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=035262026/11/12 11:50:07 INFO Vacuumed table table=closures35272026/11/12 11:50:07 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035282026/11/12 11:50:07 INFO Vacuumed table table=objects35292026/11/12 11:50:07 INFO Vacuumed table table=pending_closures35302026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=035312026/11/12 11:50:07 INFO Vacuumed table table=commit_receipts35322026/11/12 11:50:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035332026/11/12 11:50:07 INFO Vacuumed table table=pending_objects35342026/11/12 11:50:07 INFO Received push request method=POST path=/35352026/11/12 11:50:07 INFO Vacuumed table table=multipart_uploads35362026/11/12 11:50:07 INFO Vacuumed table table=pending_closures35372026/11/12 11:50:07 INFO Vacuumed table table=closures35382026/11/12 11:50:07 INFO Vacuumed table table=commit_receipts35392026/11/12 11:50:07 INFO Vacuumed table table=objects35402026/11/12 11:50:07 INFO Vacuumed table table=pending_objects35412026/11/12 11:50:07 INFO Vacuumed table table=multipart_uploads35422026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=035432026/11/12 11:50:07 INFO Vacuumed table table=closures35442026/11/12 11:50:07 INFO Vacuumed table table=objects35452026/11/12 11:50:07 INFO Received push request method=POST path=/35462026/11/12 11:50:07 INFO Received push request method=POST path=/35472026/11/12 11:50:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035482026/11/12 11:50:07 INFO Vacuumed table table=pending_closures35492026/11/12 11:50:07 INFO Vacuumed table table=commit_receipts35502026/11/12 11:50:07 INFO Received uploads request method=POST path=/35512026/11/12 11:50:07 INFO Vacuumed table table=pending_objects35522026/11/12 11:50:07 INFO Received complete push request method=POST path=/35532026/11/12 11:50:07 INFO Vacuumed table table=multipart_uploads35542026/11/12 11:50:07 INFO Received complete push request method=POST path=/35552026/11/12 11:50:07 INFO Vacuumed table table=closures35562026/11/12 11:50:07 INFO Vacuumed table table=objects35572026/11/12 11:50:07 INFO Completed push id=15 objects=5 duration=2.56681ms35582026/11/12 11:50:07 INFO Completed push id=11 objects=11 duration=3.76339ms35592026/11/12 11:50:07 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=035602026/11/12 11:50:07 INFO Vacuumed table table=pending_closures35612026/11/12 11:50:07 INFO Vacuumed table table=commit_receipts35622026/11/12 11:50:07 INFO Vacuumed table table=pending_objects35632026/11/12 11:50:07 INFO Received complete push request method=POST path=/35642026/11/12 11:50:07 INFO Vacuumed table table=multipart_uploads35652026/11/12 11:50:07 INFO Vacuumed table table=closures35662026/11/12 11:50:07 INFO Completed push id=8 objects=6 duration=2.05737ms35672026/11/12 11:50:07 INFO Vacuumed table table=objects35682026/11/12 11:50:07 INFO Received complete push request method=POST path=/35692026/11/12 11:50:07 INFO Received push request method=POST path=/35702026/11/12 11:50:07 INFO Completed push id=6 objects=6 duration=2.20115ms35712026/11/12 11:50:07 INFO Received uploads request method=POST path=/35722026/11/12 11:50:07 INFO Received push request method=POST path=/35732026/11/12 11:50:07 INFO Received uploads request method=POST path=/35742026/11/12 11:50:07 INFO Received complete upload request method=POST path=/35752026/11/12 11:50:07 INFO Received uploads request method=POST path=/35762026/11/12 11:50:07 INFO Completed upload id=18 objects=4 duration=2.420019ms35772026/11/12 11:50:07 INFO Received uploads request method=POST path=/35782026/11/12 11:50:07 INFO Received push request method=POST path=/35792026/11/12 11:50:07 INFO Received complete push request method=POST path=/35802026/11/12 11:50:07 INFO Received push request method=POST path=/35812026/11/12 11:50:07 INFO Completed push id=13 objects=6 duration=3.34967ms35822026/11/12 11:50:07 INFO Received complete push request method=POST path=/35832026/11/12 11:50:07 INFO Received push request method=POST path=/35842026/11/12 11:50:07 INFO Completed push id=9 objects=24 duration=2.1044ms35852026/11/12 11:50:07 INFO Received complete push request method=POST path=/35862026/11/12 11:50:07 INFO Received uploads request method=POST path=/35872026/11/12 11:50:07 INFO Received uploads request method=POST path=/35882026/11/12 11:50:07 INFO Completed push id=12 objects=7 duration=2.14272ms35892026/11/12 11:50:07 INFO Received push request method=POST path=/35902026/11/12 11:50:07 INFO Received complete push request method=POST path=/35912026/11/12 11:50:07 INFO Received push request method=POST path=/35922026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=035932026/11/12 11:50:07 INFO Completed push id=12 objects=2 duration=3.5947ms35942026/11/12 11:50:07 INFO Received uploads request method=POST path=/35952026/11/12 11:50:07 WARN Force mode enabled - objects will be deleted immediately without grace period35962026/11/12 11:50:07 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035972026/11/12 11:50:07 INFO Vacuumed table table=pending_closures35982026/11/12 11:50:07 INFO Vacuumed table table=commit_receipts35992026/11/12 11:50:07 INFO Vacuumed table table=pending_objects36002026/11/12 11:50:07 INFO Vacuumed table table=multipart_uploads36012026/11/12 11:50:07 INFO Vacuumed table table=closures36022026/11/12 11:50:07 INFO Vacuumed table table=objects36032026/11/12 11:50:07 INFO Received complete push request method=POST path=/36042026/11/12 11:50:07 INFO Received uploads request method=POST path=/36052026/11/12 11:50:07 INFO Completed push id=4 objects=6 duration=2.37498ms36062026/11/12 11:50:07 INFO Received complete push request method=POST path=/36072026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=036082026/11/12 11:50:07 INFO Received push request method=POST path=/36092026/11/12 11:50:07 INFO Completed push id=9 objects=4 duration=2.86287ms36102026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=036112026/11/12 11:50:07 INFO Received uploads request method=POST path=/36122026/11/12 11:50:07 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036132026/11/12 11:50:07 INFO Vacuumed table table=pending_closures36142026/11/12 11:50:07 INFO Vacuumed table table=commit_receipts36152026/11/12 11:50:07 INFO Received complete push request method=POST path=/36162026/11/12 11:50:07 INFO Vacuumed table table=pending_objects36172026/11/12 11:50:07 INFO Vacuumed table table=multipart_uploads36182026/11/12 11:50:07 INFO Vacuumed table table=closures36192026/11/12 11:50:07 INFO Completed push id=17 objects=6 duration=2.16807ms36202026/11/12 11:50:07 INFO Vacuumed table table=objects36212026/11/12 11:50:07 INFO Received uploads request method=POST path=/36222026/11/12 11:50:07 INFO Received push request method=POST path=/36232026/11/12 11:50:07 INFO Received uploads request method=POST path=/36242026/11/12 11:50:07 INFO Received uploads request method=POST path=/36252026/11/12 11:50:07 INFO Received uploads request method=POST path=/36262026/11/12 11:50:07 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=036272026/11/12 11:50:07 INFO Vacuumed table table=pending_closures36282026/11/12 11:50:07 INFO Vacuumed table table=commit_receipts36292026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=036302026/11/12 11:50:07 INFO Vacuumed table table=pending_objects36312026/11/12 11:50:07 INFO Vacuumed table table=multipart_uploads36322026/11/12 11:50:07 INFO Received complete push request method=POST path=/36332026/11/12 11:50:07 INFO Vacuumed table table=closures36342026/11/12 11:50:07 WARN Force mode enabled - objects will be deleted immediately without grace period36352026/11/12 11:50:07 INFO Completed push id=10 objects=15 duration=2.31606ms36362026/11/12 11:50:07 INFO Vacuumed table table=objects36372026/11/12 11:50:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036382026/11/12 11:50:07 INFO Vacuumed table table=pending_closures36392026/11/12 11:50:07 INFO Vacuumed table table=commit_receipts36402026/11/12 11:50:07 INFO Vacuumed table table=pending_objects36412026/11/12 11:50:07 INFO Vacuumed table table=multipart_uploads36422026/11/12 11:50:07 INFO Received uploads request method=POST path=/36432026/11/12 11:50:07 INFO Vacuumed table table=closures36442026/11/12 11:50:07 INFO Vacuumed table table=objects36452026/11/12 11:50:07 INFO Received complete push request method=POST path=/36462026/11/12 11:50:07 INFO Received complete push request method=POST path=/36472026/11/12 11:50:07 INFO Completed push id=19 objects=6 duration=4.08812ms36482026/11/12 11:50:07 INFO Completed push id=9 objects=7 duration=4.77209ms36492026/11/12 11:50:07 INFO Received complete upload request method=POST path=/36502026/11/12 11:50:07 INFO Completed upload id=10 objects=10 duration=1.27676ms36512026/11/12 11:50:07 INFO Received push request method=POST path=/36522026/11/12 11:50:07 INFO Received complete upload request method=POST path=/36532026/11/12 11:50:07 INFO Completed upload id=9 objects=2 duration=1.58151ms36542026/11/12 11:50:07 INFO Received complete push request method=POST path=/36552026/11/12 11:50:07 INFO Completed push id=17 objects=6 duration=2.73109ms36562026/11/12 11:50:07 INFO Received uploads request method=POST path=/36572026/11/12 11:50:07 INFO Received push request method=POST path=/36582026/11/12 11:50:07 INFO Received push request method=POST path=/36592026/11/12 11:50:07 INFO Received complete upload request method=POST path=/36602026/11/12 11:50:07 INFO Received push request method=POST path=/36612026/11/12 11:50:07 INFO Completed upload id=17 objects=6 duration=2.96106ms36622026/11/12 11:50:07 INFO Received complete push request method=POST path=/36632026/11/12 11:50:07 INFO Completed push id=8 objects=3 duration=1.90714ms36642026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=036652026/11/12 11:50:07 INFO Received uploads request method=POST path=/36662026/11/12 11:50:07 INFO Received complete upload request method=POST path=/36672026/11/12 11:50:07 INFO Completed upload id=10 objects=7 duration=2.17014ms36682026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=036692026/11/12 11:50:07 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036702026/11/12 11:50:07 INFO Vacuumed table table=pending_closures36712026/11/12 11:50:07 INFO Vacuumed table table=commit_receipts36722026/11/12 11:50:07 INFO Vacuumed table table=pending_objects36732026/11/12 11:50:07 INFO Vacuumed table table=multipart_uploads36742026/11/12 11:50:07 INFO Received uploads request method=POST path=/36752026/11/12 11:50:07 INFO Received complete upload request method=POST path=/36762026/11/12 11:50:07 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036772026/11/12 11:50:07 INFO Received push request method=POST path=/36782026/11/12 11:50:07 INFO Vacuumed table table=closures36792026/11/12 11:50:07 INFO Completed upload id=15 objects=6 duration=2.01756ms36802026/11/12 11:50:07 INFO Vacuumed table table=pending_closures36812026/11/12 11:50:07 INFO Vacuumed table table=objects36822026/11/12 11:50:07 INFO Vacuumed table table=commit_receipts36832026/11/12 11:50:07 INFO Vacuumed table table=pending_objects36842026/11/12 11:50:07 INFO Vacuumed table table=multipart_uploads36852026/11/12 11:50:07 INFO Received complete upload request method=POST path=/36862026/11/12 11:50:07 INFO Received push request method=POST path=/36872026/11/12 11:50:07 INFO Vacuumed table table=closures36882026/11/12 11:50:07 INFO Vacuumed table table=objects36892026/11/12 11:50:07 INFO Received uploads request method=POST path=/36902026/11/12 11:50:07 INFO Completed upload id=23 objects=15 duration=2.65626ms36912026/11/12 11:50:07 INFO Received uploads request method=POST path=/36922026/11/12 11:50:07 INFO Received complete push request method=POST path=/36932026/11/12 11:50:07 INFO Completed push id=4 objects=7 duration=2.02085ms36942026/11/12 11:50:07 INFO Received uploads request method=POST path=/36952026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=036962026/11/12 11:50:07 WARN Force mode enabled - objects will be deleted immediately without grace period36972026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=036982026/11/12 11:50:07 WARN Force mode enabled - objects will be deleted immediately without grace period36992026/11/12 11:50:07 INFO Received push request method=POST path=/37002026/11/12 11:50:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037012026/11/12 11:50:07 INFO Received uploads request method=POST path=/37022026/11/12 11:50:07 INFO Vacuumed table table=pending_closures37032026/11/12 11:50:07 INFO Received push request method=POST path=/37042026/11/12 11:50:07 INFO Vacuumed table table=commit_receipts37052026/11/12 11:50:07 INFO Received uploads request method=POST path=/37062026/11/12 11:50:07 INFO Vacuumed table table=pending_objects37072026/11/12 11:50:07 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."37082026/11/12 11:50:07 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."37092026/11/12 11:50:07 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."37102026/11/12 11:50:07 INFO Vacuumed table table=multipart_uploads37112026/11/12 11:50:07 INFO Vacuumed table table=closures37122026/11/12 11:50:07 INFO Vacuumed table table=objects37132026/11/12 11:50:07 INFO Received push request method=POST path=/37142026/11/12 11:50:07 INFO Received uploads request method=POST path=/37152026/11/12 11:50:07 INFO Received push request method=POST path=/37162026/11/12 11:50:07 INFO Received complete upload request method=POST path=/37172026/11/12 11:50:07 INFO Completed upload id=13 objects=10 duration=1.12336ms37182026/11/12 11:50:07 INFO Received uploads request method=POST path=/37192026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=037202026/11/12 11:50:07 INFO Received uploads request method=POST path=/37212026/11/12 11:50:07 INFO Received uploads request method=POST path=/37222026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=037232026/11/12 11:50:07 WARN Force mode enabled - objects will be deleted immediately without grace period37242026/11/12 11:50:07 WARN Force mode enabled - objects will be deleted immediately without grace period37252026/11/12 11:50:07 INFO Received uploads request method=POST path=/37262026/11/12 11:50:07 INFO Received push request method=POST path=/37272026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=037282026/11/12 11:50:07 WARN Force mode enabled - objects will be deleted immediately without grace period37292026/11/12 11:50:07 INFO Received complete upload request method=POST path=/37302026/11/12 11:50:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037312026/11/12 11:50:07 INFO Completed upload id=19 objects=6 duration=1.49185ms37322026/11/12 11:50:07 INFO Vacuumed table table=pending_closures37332026/11/12 11:50:07 INFO Received uploads request method=POST path=/37342026/11/12 11:50:07 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=037352026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=037362026/11/12 11:50:07 INFO Vacuumed table table=commit_receipts37372026/11/12 11:50:07 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."37382026/11/12 11:50:07 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."37392026/11/12 11:50:07 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."37402026/11/12 11:50:07 INFO Vacuumed table table=pending_objects37412026/11/12 11:50:07 INFO Vacuumed table table=multipart_uploads37422026/11/12 11:50:07 INFO Vacuumed table table=pending_closures37432026/11/12 11:50:07 INFO Vacuumed table table=closures37442026/11/12 11:50:07 INFO Received complete push request method=POST path=/37452026/11/12 11:50:07 WARN Force mode enabled - objects will be deleted immediately without grace period37462026/11/12 11:50:07 INFO Vacuumed table table=commit_receipts37472026/11/12 11:50:07 INFO Vacuumed table table=objects37482026/11/12 11:50:07 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037492026/11/12 11:50:07 INFO Completed push id=17 objects=22 duration=2.44935ms37502026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=037512026/11/12 11:50:07 INFO Vacuumed table table=pending_objects37522026/11/12 11:50:07 INFO Vacuumed table table=multipart_uploads37532026/11/12 11:50:07 INFO Vacuumed table table=pending_closures37542026/11/12 11:50:07 INFO Vacuumed table table=closures37552026/11/12 11:50:07 WARN Force mode enabled - objects will be deleted immediately without grace period37562026/11/12 11:50:07 INFO Vacuumed table table=commit_receipts37572026/11/12 11:50:07 INFO Received complete push request method=POST path=/37582026/11/12 11:50:07 INFO Vacuumed table table=objects37592026/11/12 11:50:07 INFO Vacuumed table table=pending_objects37602026/11/12 11:50:07 INFO Vacuumed table table=multipart_uploads37612026/11/12 11:50:07 INFO Received uploads request method=POST path=/37622026/11/12 11:50:07 INFO Completed push id=16 objects=4 duration=2.23385ms37632026/11/12 11:50:07 INFO Vacuumed table table=closures37642026/11/12 11:50:07 INFO Vacuumed table table=objects37652026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=037662026/11/12 11:50:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037672026/11/12 11:50:07 INFO Vacuumed table table=pending_closures37682026/11/12 11:50:07 INFO Vacuumed table table=commit_receipts37692026/11/12 11:50:07 INFO Vacuumed table table=pending_objects37702026/11/12 11:50:07 INFO Vacuumed table table=multipart_uploads37712026/11/12 11:50:07 INFO Vacuumed table table=closures37722026/11/12 11:50:07 INFO Received complete upload request method=POST path=/37732026/11/12 11:50:07 INFO Vacuumed table table=objects37742026/11/12 11:50:07 INFO Completed upload id=18 objects=4 duration=1.83278ms37752026/11/12 11:50:07 INFO Received complete push request method=POST path=/37762026/11/12 11:50:07 INFO Received push request method=POST path=/37772026/11/12 11:50:07 INFO Completed push id=16 objects=27 duration=2.273359ms37782026/11/12 11:50:07 INFO Received complete upload request method=POST path=/37792026/11/12 11:50:07 INFO Completed upload id=22 objects=7 duration=1.21203ms37802026/11/12 11:50:07 INFO Received push request method=POST path=/37812026/11/12 11:50:07 INFO Received push request method=POST path=/37822026/11/12 11:50:07 INFO Received uploads request method=POST path=/37832026/11/12 11:50:07 INFO Received push request method=POST path=/37842026/11/12 11:50:07 INFO Received uploads request method=POST path=/37852026/11/12 11:50:07 INFO Received complete push request method=POST path=/37862026/11/12 11:50:07 INFO Completed push id=22 objects=3 duration=3.35466ms37872026/11/12 11:50:07 INFO Received uploads request method=POST path=/37882026/11/12 11:50:07 INFO Received uploads request method=POST path=/37892026/11/12 11:50:07 INFO Received uploads request method=POST path=/37902026/11/12 11:50:07 INFO Received complete upload request method=POST path=/37912026/11/12 11:50:07 INFO Completed upload id=20 objects=5 duration=895.53µs37922026/11/12 11:50:07 INFO Received uploads request method=POST path=/37932026/11/12 11:50:07 INFO Received complete push request method=POST path=/37942026/11/12 11:50:07 INFO Completed push id=7 objects=7 duration=1.68927ms37952026/11/12 11:50:07 INFO Received complete push request method=POST path=/37962026/11/12 11:50:07 INFO Completed push id=14 objects=3 duration=1.45867ms37972026/11/12 11:50:07 INFO Received complete push request method=POST path=/37982026/11/12 11:50:07 INFO Completed push id=15 objects=7 duration=2.30923ms37992026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=038002026/11/12 11:50:07 WARN Force mode enabled - objects will be deleted immediately without grace period38012026/11/12 11:50:07 INFO Received push request method=POST path=/38022026/11/12 11:50:07 INFO Received uploads request method=POST path=/38032026/11/12 11:50:07 INFO Received push request method=POST path=/38042026/11/12 11:50:07 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=11 objects-failed-to-delete=038052026/11/12 11:50:07 INFO Received push request method=POST path=/38062026/11/12 11:50:07 INFO Received push request method=POST path=/38072026/11/12 11:50:07 INFO Received uploads request method=POST path=/38082026/11/12 11:50:07 INFO Vacuumed table table=pending_closures38092026/11/12 11:50:07 INFO Vacuumed table table=commit_receipts38102026/11/12 11:50:07 INFO Received uploads request method=POST path=/38112026/11/12 11:50:07 INFO Vacuumed table table=pending_objects38122026/11/12 11:50:07 INFO Vacuumed table table=multipart_uploads38132026/11/12 11:50:07 INFO Vacuumed table table=closures38142026/11/12 11:50:07 INFO Vacuumed table table=objects38152026/11/12 11:50:07 INFO Received uploads request method=POST path=/38162026/11/12 11:50:07 INFO Received complete push request method=POST path=/38172026/11/12 11:50:07 INFO Received push request method=POST path=/38182026/11/12 11:50:07 INFO Completed push id=14 objects=3 duration=3.52989ms38192026/11/12 11:50:07 INFO Received complete upload request method=POST path=/38202026/11/12 11:50:07 INFO Received complete upload request method=POST path=/38212026/11/12 11:50:07 INFO Completed upload id=23 objects=16 duration=2.86328ms38222026/11/12 11:50:07 INFO Completed upload id=7 objects=10 duration=2.0796ms38232026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=038242026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=038252026/11/12 11:50:07 INFO Received complete push request method=POST path=/38262026/11/12 11:50:07 WARN Force mode enabled - objects will be deleted immediately without grace period38272026/11/12 11:50:07 WARN Force mode enabled - objects will be deleted immediately without grace period38282026/11/12 11:50:07 INFO Received push request method=POST path=/38292026/11/12 11:50:07 INFO Completed push id=24 objects=9 duration=2.11499ms38302026/11/12 11:50:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038312026/11/12 11:50:07 INFO Received uploads request method=POST path=/38322026/11/12 11:50:07 INFO Vacuumed table table=pending_closures38332026/11/12 11:50:07 INFO Received push request method=POST path=/38342026/11/12 11:50:07 INFO Vacuumed table table=commit_receipts38352026/11/12 11:50:07 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."38362026/11/12 11:50:07 INFO Vacuumed table table=pending_objects38372026/11/12 11:50:07 INFO Vacuumed table table=multipart_uploads38382026/11/12 11:50:07 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."38392026/11/12 11:50:07 INFO Received uploads request method=POST path=/38402026/11/12 11:50:07 INFO Vacuumed table table=closures38412026/11/12 11:50:07 INFO Vacuumed table table=objects38422026/11/12 11:50:07 INFO Received complete push request method=POST path=/38432026/11/12 11:50:07 INFO Completed push id=18 objects=10 duration=2.11533ms38442026/11/12 11:50:07 INFO Received complete push request method=POST path=/38452026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=038462026/11/12 11:50:07 INFO Received push request method=POST path=/38472026/11/12 11:50:07 WARN Force mode enabled - objects will be deleted immediately without grace period38482026/11/12 11:50:07 INFO Completed push id=21 objects=9 duration=2.35566ms38492026/11/12 11:50:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038502026/11/12 11:50:07 INFO Received push request method=POST path=/38512026/11/12 11:50:07 INFO Received push request method=POST path=/38522026/11/12 11:50:07 INFO Vacuumed table table=pending_closures38532026/11/12 11:50:07 INFO Received complete upload request method=POST path=/38542026/11/12 11:50:07 INFO Vacuumed table table=commit_receipts38552026/11/12 11:50:07 INFO Completed upload id=18 objects=11 duration=1.84745ms38562026/11/12 11:50:07 INFO Vacuumed table table=pending_objects38572026/11/12 11:50:07 INFO Received complete upload request method=POST path=/38582026/11/12 11:50:07 INFO Vacuumed table table=multipart_uploads38592026/11/12 11:50:07 INFO Received push request method=POST path=/38602026/11/12 11:50:07 INFO Vacuumed table table=closures38612026/11/12 11:50:07 INFO Completed upload id=15 objects=9 duration=1.38367ms38622026/11/12 11:50:07 INFO Vacuumed table table=objects38632026/11/12 11:50:07 INFO Received uploads request method=POST path=/38642026/11/12 11:50:07 INFO Received push request method=POST path=/38652026/11/12 11:50:07 INFO Received push request method=POST path=/38662026/11/12 11:50:07 INFO Received uploads request method=POST path=/38672026/11/12 11:50:07 INFO Received push request method=POST path=/38682026/11/12 11:50:07 INFO Received complete push request method=POST path=/38692026/11/12 11:50:07 INFO Received push request method=POST path=/38702026/11/12 11:50:07 INFO Completed push id=19 objects=8 duration=4.32045ms38712026/11/12 11:50:07 INFO Received uploads request method=POST path=/38722026/11/12 11:50:07 INFO Received complete push request method=POST path=/38732026/11/12 11:50:07 INFO Received complete upload request method=POST path=/38742026/11/12 11:50:07 INFO Completed push id=14 objects=4 duration=3.7096ms38752026/11/12 11:50:07 INFO Completed upload id=14 objects=8 duration=2.75487ms38762026/11/12 11:50:07 INFO Received uploads request method=POST path=/38772026/11/12 11:50:07 INFO Received complete push request method=POST path=/38782026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=038792026/11/12 11:50:07 INFO Completed push id=1 objects=11 duration=2.82795ms38802026/11/12 11:50:07 WARN Force mode enabled - objects will be deleted immediately without grace period38812026/11/12 11:50:07 INFO Received push request method=POST path=/38822026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=038832026/11/12 11:50:07 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038842026/11/12 11:50:07 INFO Received uploads request method=POST path=/38852026/11/12 11:50:07 INFO Received push request method=POST path=/38862026/11/12 11:50:07 INFO Vacuumed table table=pending_closures38872026/11/12 11:50:07 INFO Vacuumed table table=commit_receipts38882026/11/12 11:50:07 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038892026/11/12 11:50:07 INFO Vacuumed table table=pending_objects38902026/11/12 11:50:07 INFO Vacuumed table table=pending_closures38912026/11/12 11:50:07 INFO Vacuumed table table=multipart_uploads38922026/11/12 11:50:07 INFO Vacuumed table table=commit_receipts38932026/11/12 11:50:07 INFO Vacuumed table table=closures38942026/11/12 11:50:07 INFO Vacuumed table table=pending_objects38952026/11/12 11:50:07 INFO Vacuumed table table=multipart_uploads38962026/11/12 11:50:07 INFO Vacuumed table table=objects38972026/11/12 11:50:07 INFO Received uploads request method=POST path=/38982026/11/12 11:50:07 INFO Vacuumed table table=closures38992026/11/12 11:50:07 INFO Vacuumed table table=objects39002026/11/12 11:50:07 INFO Received complete push request method=POST path=/39012026/11/12 11:50:07 INFO Completed push id=21 objects=17 duration=2.09568ms39022026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=039032026/11/12 11:50:07 WARN Force mode enabled - objects will be deleted immediately without grace period39042026/11/12 11:50:07 INFO Garbage collection completed failed-uploads-deleted=5 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039052026/11/12 11:50:07 INFO Vacuumed table table=pending_closures39062026/11/12 11:50:07 INFO Received complete upload request method=POST path=/39072026/11/12 11:50:07 INFO Completed upload id=22 objects=2 duration=2.234909ms39082026/11/12 11:50:07 INFO Vacuumed table table=commit_receipts39092026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=039102026/11/12 11:50:07 INFO Received uploads request method=POST path=/39112026/11/12 11:50:07 INFO Received complete upload request method=POST path=/39122026/11/12 11:50:07 INFO Received push request method=POST path=/39132026/11/12 11:50:07 INFO Completed upload id=16 objects=3 duration=2.20632ms39142026/11/12 11:50:07 INFO Vacuumed table table=pending_objects39152026/11/12 11:50:07 INFO Vacuumed table table=multipart_uploads39162026/11/12 11:50:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039172026/11/12 11:50:07 INFO Vacuumed table table=pending_closures39182026/11/12 11:50:07 INFO Vacuumed table table=closures39192026/11/12 11:50:07 INFO Vacuumed table table=commit_receipts39202026/11/12 11:50:07 INFO Received push request method=POST path=/39212026/11/12 11:50:07 INFO Vacuumed table table=objects39222026/11/12 11:50:07 INFO Vacuumed table table=pending_objects39232026/11/12 11:50:07 INFO Received push request method=POST path=/39242026/11/12 11:50:07 INFO Vacuumed table table=multipart_uploads39252026/11/12 11:50:07 INFO Received complete push request method=POST path=/39262026/11/12 11:50:07 INFO Vacuumed table table=closures39272026/11/12 11:50:07 INFO Received complete upload request method=POST path=/39282026/11/12 11:50:07 INFO Completed push id=19 objects=7 duration=2.31122ms39292026/11/12 11:50:07 INFO Vacuumed table table=objects39302026/11/12 11:50:07 INFO Completed upload id=11 objects=7 duration=1.55114ms39312026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=039322026/11/12 11:50:07 WARN Force mode enabled - objects will be deleted immediately without grace period39332026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=039342026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=039352026/11/12 11:50:07 INFO Received uploads request method=POST path=/39362026/11/12 11:50:07 WARN Force mode enabled - objects will be deleted immediately without grace period39372026/11/12 11:50:07 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039382026/11/12 11:50:07 INFO Received push request method=POST path=/39392026/11/12 11:50:07 INFO Vacuumed table table=pending_closures39402026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=039412026/11/12 11:50:07 INFO Vacuumed table table=commit_receipts39422026/11/12 11:50:07 INFO Received uploads request method=POST path=/39432026/11/12 11:50:07 INFO Vacuumed table table=pending_objects39442026/11/12 11:50:07 INFO Vacuumed table table=multipart_uploads39452026/11/12 11:50:07 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=039462026/11/12 11:50:07 INFO Vacuumed table table=closures39472026/11/12 11:50:07 INFO Vacuumed table table=pending_closures39482026/11/12 11:50:07 INFO Vacuumed table table=objects39492026/11/12 11:50:07 INFO Vacuumed table table=commit_receipts39502026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=039512026/11/12 11:50:07 INFO Vacuumed table table=pending_objects39522026/11/12 11:50:07 INFO Vacuumed table table=multipart_uploads39532026/11/12 11:50:07 INFO Received uploads request method=POST path=/39542026/11/12 11:50:07 INFO Vacuumed table table=closures39552026/11/12 11:50:07 INFO Received uploads request method=POST path=/39562026/11/12 11:50:07 INFO Received push request method=POST path=/39572026/11/12 11:50:07 WARN Force mode enabled - objects will be deleted immediately without grace period39582026/11/12 11:50:07 INFO Vacuumed table table=objects39592026/11/12 11:50:07 INFO Received uploads request method=POST path=/39602026/11/12 11:50:07 INFO Received push request method=POST path=/39612026/11/12 11:50:07 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."39622026/11/12 11:50:07 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."39632026/11/12 11:50:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039642026/11/12 11:50:07 INFO Vacuumed table table=pending_closures39652026/11/12 11:50:07 INFO Vacuumed table table=commit_receipts39662026/11/12 11:50:07 INFO Vacuumed table table=pending_objects39672026/11/12 11:50:07 INFO Vacuumed table table=multipart_uploads39682026/11/12 11:50:07 INFO Vacuumed table table=closures39692026/11/12 11:50:07 INFO Vacuumed table table=objects39702026/11/12 11:50:07 INFO Received push request method=POST path=/39712026/11/12 11:50:07 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."39722026/11/12 11:50:07 INFO Received uploads request method=POST path=/39732026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=039742026/11/12 11:50:07 INFO Received uploads request method=POST path=/39752026/11/12 11:50:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039762026/11/12 11:50:07 INFO Vacuumed table table=pending_closures39772026/11/12 11:50:07 INFO Received push request method=POST path=/39782026/11/12 11:50:07 INFO Vacuumed table table=commit_receipts39792026/11/12 11:50:07 INFO Vacuumed table table=pending_objects39802026/11/12 11:50:07 INFO Vacuumed table table=multipart_uploads39812026/11/12 11:50:07 INFO Vacuumed table table=closures39822026/11/12 11:50:07 INFO Received uploads request method=POST path=/39832026/11/12 11:50:07 INFO Vacuumed table table=objects39842026/11/12 11:50:07 INFO Received push request method=POST path=/39852026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=039862026/11/12 11:50:07 INFO Received push request method=POST path=/39872026/11/12 11:50:07 WARN Force mode enabled - objects will be deleted immediately without grace period39882026/11/12 11:50:07 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039892026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=039902026/11/12 11:50:07 INFO Vacuumed table table=pending_closures39912026/11/12 11:50:07 INFO Received uploads request method=POST path=/39922026/11/12 11:50:07 INFO Vacuumed table table=commit_receipts39932026/11/12 11:50:07 WARN Force mode enabled - objects will be deleted immediately without grace period39942026/11/12 11:50:07 INFO Received push request method=POST path=/39952026/11/12 11:50:07 INFO Vacuumed table table=pending_objects39962026/11/12 11:50:07 INFO Vacuumed table table=multipart_uploads39972026/11/12 11:50:07 INFO Vacuumed table table=closures39982026/11/12 11:50:07 INFO Vacuumed table table=objects39992026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=040002026/11/12 11:50:07 INFO Received complete push request method=POST path=/40012026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=040022026/11/12 11:50:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=040032026/11/12 11:50:07 INFO Received complete push request method=POST path=/40042026/11/12 11:50:07 WARN Force mode enabled - objects will be deleted immediately without grace period40052026/11/12 11:50:07 INFO Received push request method=POST path=/40062026/11/12 11:50:07 INFO Completed push id=7 objects=15 duration=3.79941ms40072026/11/12 11:50:07 WARN Force mode enabled - objects will be deleted immediately without grace period40082026/11/12 11:50:07 INFO Vacuumed table table=pending_closures40092026/11/12 11:50:07 INFO Completed push id=26 objects=14 duration=3.27995ms40102026/11/12 11:50:07 INFO Vacuumed table table=commit_receipts40112026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=040122026/11/12 11:50:07 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040132026/11/12 11:50:07 INFO Vacuumed table table=pending_objects40142026/11/12 11:50:07 INFO Received uploads request method=POST path=/40152026/11/12 11:50:07 INFO Vacuumed table table=pending_closures40162026/11/12 11:50:07 INFO Vacuumed table table=multipart_uploads40172026/11/12 11:50:07 WARN Force mode enabled - objects will be deleted immediately without grace period40182026/11/12 11:50:07 INFO Vacuumed table table=closures40192026/11/12 11:50:07 INFO Vacuumed table table=commit_receipts40202026/11/12 11:50:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040212026/11/12 11:50:07 INFO Vacuumed table table=objects40222026/11/12 11:50:07 INFO Received complete push request method=POST path=/40232026/11/12 11:50:07 INFO Vacuumed table table=pending_closures40242026/11/12 11:50:07 INFO Vacuumed table table=pending_objects40252026/11/12 11:50:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=3 objects-failed-to-delete=040262026/11/12 11:50:07 INFO Vacuumed table table=multipart_uploads40272026/11/12 11:50:07 INFO Vacuumed table table=commit_receipts40282026/11/12 11:50:07 INFO Completed push id=21 objects=3 duration=1.66373ms40292026/11/12 11:50:07 INFO Vacuumed table table=closures40302026/11/12 11:50:07 INFO Received push request method=POST path=/40312026/11/12 11:50:07 INFO Vacuumed table table=pending_closures40322026/11/12 11:50:07 INFO Vacuumed table table=pending_objects40332026/11/12 11:50:07 INFO Vacuumed table table=objects40342026/11/12 11:50:07 INFO Vacuumed table table=multipart_uploads40352026/11/12 11:50:07 INFO Vacuumed table table=commit_receipts40362026/11/12 11:50:07 INFO Vacuumed table table=closures40372026/11/12 11:50:07 INFO Vacuumed table table=objects40382026/11/12 11:50:07 INFO Vacuumed table table=pending_objects40392026/11/12 11:50:07 INFO Vacuumed table table=multipart_uploads40402026/11/12 11:50:07 INFO Vacuumed table table=closures40412026/11/12 11:50:07 INFO Received push request method=POST path=/40422026/11/12 11:50:07 INFO Received uploads request method=POST path=/40432026/11/12 11:50:07 INFO Received complete push request method=POST path=/40442026/11/12 11:50:07 INFO Vacuumed table table=objects40452026/11/12 11:50:07 INFO Completed push id=15 objects=4 duration=1.54216ms40462026/11/12 11:50:07 INFO Received push request method=POST path=/40472026/11/12 11:50:07 INFO Received uploads request method=POST path=/40482026/11/12 11:50:07 INFO Received uploads request method=POST path=/40492026/11/12 11:50:07 INFO Received push request method=POST path=/40502026/11/12 11:50:07 INFO Received push request method=POST path=/40512026/11/12 11:50:07 INFO Received complete upload request method=POST path=/40522026/11/12 11:50:07 INFO Completed upload id=21 objects=7 duration=2.97832ms40532026/11/12 11:50:07 INFO Received push request method=POST path=/40542026/11/12 11:50:07 INFO Received push request method=POST path=/40552026/11/12 11:50:07 INFO Received push request method=POST path=/40562026/11/12 11:50:07 INFO Received uploads request method=POST path=/40572026/11/12 11:50:07 INFO Received uploads request method=POST path=/40582026/11/12 11:50:07 INFO Received uploads request method=POST path=/40592026/11/12 11:50:07 INFO Received push request method=POST path=/40602026/11/12 11:50:07 INFO Received complete upload request method=POST path=/40612026/11/12 11:50:07 INFO Received uploads request method=POST path=/40622026/11/12 11:50:07 INFO Completed upload id=12 objects=26 duration=3.07112ms40632026/11/12 11:50:07 INFO Received uploads request method=POST path=/40642026/11/12 11:50:07 INFO Received complete push request method=POST path=/40652026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=040662026/11/12 11:50:07 INFO Completed push id=16 objects=16 duration=4.07918ms40672026/11/12 11:50:07 INFO Received push request method=POST path=/40682026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=040692026/11/12 11:50:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040702026/11/12 11:50:07 INFO Received push request method=POST path=/40712026/11/12 11:50:07 INFO Vacuumed table table=pending_closures40722026/11/12 11:50:07 INFO Vacuumed table table=commit_receipts40732026/11/12 11:50:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040742026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=040752026/11/12 11:50:07 INFO Vacuumed table table=pending_objects40762026/11/12 11:50:07 INFO Vacuumed table table=multipart_uploads40772026/11/12 11:50:07 INFO Vacuumed table table=pending_closures40782026/11/12 11:50:07 INFO Received complete push request method=POST path=/40792026/11/12 11:50:07 INFO Vacuumed table table=closures40802026/11/12 11:50:07 INFO Vacuumed table table=commit_receipts40812026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=040822026/11/12 11:50:07 INFO Vacuumed table table=objects40832026/11/12 11:50:07 INFO Completed push id=4 objects=6 duration=2.40925ms40842026/11/12 11:50:07 INFO Vacuumed table table=pending_objects40852026/11/12 11:50:07 INFO Vacuumed table table=multipart_uploads40862026/11/12 11:50:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040872026/11/12 11:50:07 INFO Vacuumed table table=closures40882026/11/12 11:50:07 INFO Vacuumed table table=pending_closures40892026/11/12 11:50:07 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=3 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040902026/11/12 11:50:07 INFO Vacuumed table table=objects40912026/11/12 11:50:07 INFO Vacuumed table table=commit_receipts40922026/11/12 11:50:07 INFO Vacuumed table table=pending_closures40932026/11/12 11:50:07 INFO Received push request method=POST path=/40942026/11/12 11:50:07 INFO Vacuumed table table=commit_receipts40952026/11/12 11:50:07 INFO Vacuumed table table=pending_objects40962026/11/12 11:50:07 INFO Vacuumed table table=multipart_uploads40972026/11/12 11:50:07 INFO Received uploads request method=POST path=/40982026/11/12 11:50:07 INFO Vacuumed table table=closures40992026/11/12 11:50:07 INFO Received complete upload request method=POST path=/41002026/11/12 11:50:07 INFO Vacuumed table table=pending_objects41012026/11/12 11:50:07 INFO Vacuumed table table=multipart_uploads41022026/11/12 11:50:07 INFO Vacuumed table table=objects41032026/11/12 11:50:07 INFO Vacuumed table table=closures41042026/11/12 11:50:07 INFO Vacuumed table table=objects41052026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=041062026/11/12 11:50:07 INFO Completed upload id=16 objects=17 duration=3.178649ms41072026/11/12 11:50:07 WARN Force mode enabled - objects will be deleted immediately without grace period41082026/11/12 11:50:07 INFO Received uploads request method=POST path=/41092026/11/12 11:50:07 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041102026/11/12 11:50:07 INFO Vacuumed table table=pending_closures41112026/11/12 11:50:07 INFO Vacuumed table table=commit_receipts41122026/11/12 11:50:07 INFO Vacuumed table table=pending_objects41132026/11/12 11:50:07 INFO Vacuumed table table=multipart_uploads41142026/11/12 11:50:07 INFO Vacuumed table table=closures41152026/11/12 11:50:07 INFO Received complete push request method=POST path=/41162026/11/12 11:50:07 INFO Received uploads request method=POST path=/41172026/11/12 11:50:07 INFO Vacuumed table table=objects41182026/11/12 11:50:07 INFO Received uploads request method=POST path=/41192026/11/12 11:50:07 INFO Completed push id=13 objects=15 duration=2.20482ms41202026/11/12 11:50:07 INFO Received complete push request method=POST path=/41212026/11/12 11:50:07 INFO Completed push id=14 objects=3 duration=2.05325ms41222026/11/12 11:50:07 INFO Received uploads request method=POST path=/41232026/11/12 11:50:07 INFO Received push request method=POST path=/41242026/11/12 11:50:07 INFO Received uploads request method=POST path=/41252026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=041262026/11/12 11:50:07 INFO Received push request method=POST path=/41272026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=041282026/11/12 11:50:07 WARN Force mode enabled - objects will be deleted immediately without grace period41292026/11/12 11:50:07 INFO Received complete upload request method=POST path=/41302026/11/12 11:50:07 INFO Completed upload id=13 objects=8 duration=1.99799ms41312026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=041322026/11/12 11:50:07 INFO Received push request method=POST path=/41332026/11/12 11:50:07 INFO Received push request method=POST path=/41342026/11/12 11:50:07 INFO Received push request method=POST path=/41352026/11/12 11:50:07 INFO Received uploads request method=POST path=/41362026/11/12 11:50:07 INFO Received complete push request method=POST path=/41372026/11/12 11:50:07 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."41382026/11/12 11:50:07 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."41392026/11/12 11:50:07 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."41402026/11/12 11:50:07 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."41412026/11/12 11:50:07 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."41422026/11/12 11:50:07 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."41432026/11/12 11:50:07 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."41442026/11/12 11:50:07 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."41452026/11/12 11:50:07 INFO Completed push id=16 objects=8 duration=2.39283ms41462026/11/12 11:50:07 INFO Received push request method=POST path=/41472026/11/12 11:50:07 INFO Received push request method=POST path=/41482026/11/12 11:50:07 INFO Received uploads request method=POST path=/41492026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=041502026/11/12 11:50:07 INFO Received push request method=POST path=/41512026/11/12 11:50:07 WARN Force mode enabled - objects will be deleted immediately without grace period41522026/11/12 11:50:07 INFO Received uploads request method=POST path=/41532026/11/12 11:50:07 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041542026/11/12 11:50:07 INFO Received uploads request method=POST path=/41552026/11/12 11:50:07 INFO Vacuumed table table=pending_closures41562026/11/12 11:50:07 INFO Vacuumed table table=commit_receipts41572026/11/12 11:50:07 INFO Received uploads request method=POST path=/41582026/11/12 11:50:07 INFO Vacuumed table table=pending_objects41592026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=041602026/11/12 11:50:07 INFO Vacuumed table table=multipart_uploads41612026/11/12 11:50:07 INFO Received uploads request method=POST path=/41622026/11/12 11:50:07 INFO Vacuumed table table=closures41632026/11/12 11:50:07 INFO Received complete upload request method=POST path=/41642026/11/12 11:50:07 INFO Vacuumed table table=objects41652026/11/12 11:50:07 WARN Force mode enabled - objects will be deleted immediately without grace period41662026/11/12 11:50:07 INFO Completed upload id=20 objects=4 duration=1.13854ms41672026/11/12 11:50:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041682026/11/12 11:50:07 INFO Vacuumed table table=pending_closures41692026/11/12 11:50:07 INFO Received push request method=POST path=/41702026/11/12 11:50:07 INFO Vacuumed table table=commit_receipts41712026/11/12 11:50:07 INFO Vacuumed table table=pending_objects41722026/11/12 11:50:07 INFO Vacuumed table table=multipart_uploads41732026/11/12 11:50:07 INFO Received complete push request method=POST path=/41742026/11/12 11:50:07 INFO Vacuumed table table=closures41752026/11/12 11:50:07 INFO Received uploads request method=POST path=/41762026/11/12 11:50:07 INFO Vacuumed table table=objects41772026/11/12 11:50:07 INFO Completed push id=30 objects=4 duration=3.88973ms41782026/11/12 11:50:07 INFO Received uploads request method=POST path=/41792026/11/12 11:50:07 INFO Received uploads request method=POST path=/41802026/11/12 11:50:07 INFO Received push request method=POST path=/41812026/11/12 11:50:07 INFO Received complete push request method=POST path=/41822026/11/12 11:50:07 INFO Completed push id=20 objects=3 duration=2.074ms41832026/11/12 11:50:07 INFO Received complete upload request method=POST path=/41842026/11/12 11:50:07 INFO Received complete upload request method=POST path=/41852026/11/12 11:50:07 INFO Completed upload id=27 objects=4 duration=1.50804ms41862026/11/12 11:50:07 INFO Completed upload id=14 objects=16 duration=2.56934ms41872026/11/12 11:50:07 INFO Received complete push request method=POST path=/41882026/11/12 11:50:07 INFO Received uploads request method=POST path=/41892026/11/12 11:50:07 INFO Completed push id=19 objects=14 duration=2.81768ms41902026/11/12 11:50:07 INFO Received complete push request method=POST path=/41912026/11/12 11:50:07 INFO Completed push id=31 objects=14 duration=3.02487ms41922026/11/12 11:50:07 INFO Received push request method=POST path=/41932026/11/12 11:50:07 INFO Received uploads request method=POST path=/41942026/11/12 11:50:07 INFO Received push request method=POST path=/41952026/11/12 11:50:07 INFO Received complete push request method=POST path=/41962026/11/12 11:50:07 INFO Received complete push request method=POST path=/41972026/11/12 11:50:07 INFO Completed push id=11 objects=6 duration=4.0306ms41982026/11/12 11:50:07 INFO Completed push id=16 objects=5 duration=1.30523ms41992026/11/12 11:50:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=8 objects-deleted-after-grace-period=8 objects-failed-to-delete=042002026/11/12 11:50:07 INFO Received complete push request method=POST path=/42012026/11/12 11:50:07 INFO Received uploads request method=POST path=/42022026/11/12 11:50:07 INFO Vacuumed table table=pending_closures42032026/11/12 11:50:07 INFO Completed push id=13 objects=11 duration=1.94844ms42042026/11/12 11:50:07 INFO Vacuumed table table=commit_receipts42052026/11/12 11:50:07 INFO Vacuumed table table=pending_objects42062026/11/12 11:50:07 INFO Received complete push request method=POST path=/42072026/11/12 11:50:07 INFO Vacuumed table table=multipart_uploads42082026/11/12 11:50:07 INFO Vacuumed table table=closures42092026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=042102026/11/12 11:50:07 INFO Vacuumed table table=objects42112026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=042122026/11/12 11:50:07 INFO Completed push id=18 objects=6 duration=3.36098ms42132026/11/12 11:50:07 WARN Force mode enabled - objects will be deleted immediately without grace period42142026/11/12 11:50:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042152026/11/12 11:50:07 INFO Received uploads request method=POST path=/42162026/11/12 11:50:07 WARN Force mode enabled - objects will be deleted immediately without grace period42172026/11/12 11:50:07 INFO Vacuumed table table=pending_closures42182026/11/12 11:50:07 INFO Vacuumed table table=commit_receipts42192026/11/12 11:50:07 INFO Vacuumed table table=pending_objects42202026/11/12 11:50:07 INFO Vacuumed table table=multipart_uploads42212026/11/12 11:50:07 INFO Received push request method=POST path=/42222026/11/12 11:50:07 INFO Vacuumed table table=closures42232026/11/12 11:50:07 INFO Vacuumed table table=objects42242026/11/12 11:50:07 INFO Received uploads request method=POST path=/42252026/11/12 11:50:07 INFO Received complete upload request method=POST path=/42262026/11/12 11:50:07 INFO Received uploads request method=POST path=/42272026/11/12 11:50:07 INFO Received complete upload request method=POST path=/42282026/11/12 11:50:07 INFO Received complete push request method=POST path=/42292026/11/12 11:50:07 INFO Completed upload id=23 objects=18 duration=3.20068ms42302026/11/12 11:50:07 INFO Completed upload id=23 objects=9 duration=1.24009ms42312026/11/12 11:50:07 INFO Completed push id=18 objects=6 duration=1.44923ms42322026/11/12 11:50:07 INFO Received uploads request method=POST path=/42332026/11/12 11:50:07 INFO Received push request method=POST path=/42342026/11/12 11:50:07 INFO Received uploads request method=POST path=/42352026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=042362026/11/12 11:50:07 INFO Received push request method=POST path=/42372026/11/12 11:50:07 WARN Force mode enabled - objects will be deleted immediately without grace period42382026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=042392026/11/12 11:50:07 INFO Received complete push request method=POST path=/42402026/11/12 11:50:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042412026/11/12 11:50:07 INFO Vacuumed table table=pending_closures42422026/11/12 11:50:07 INFO Completed push id=17 objects=11 duration=1.885761ms42432026/11/12 11:50:07 INFO Vacuumed table table=commit_receipts42442026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=042452026/11/12 11:50:07 INFO Vacuumed table table=pending_objects42462026/11/12 11:50:07 INFO Vacuumed table table=multipart_uploads42472026/11/12 11:50:07 INFO Vacuumed table table=closures42482026/11/12 11:50:07 INFO Received complete push request method=POST path=/42492026/11/12 11:50:07 WARN Force mode enabled - objects will be deleted immediately without grace period42502026/11/12 11:50:07 INFO Vacuumed table table=objects42512026/11/12 11:50:07 INFO Aborted multipart uploads count=0 kept=042522026/11/12 11:50:07 INFO Completed push id=5 objects=28 duration=2.25565ms42532026/11/12 11:50:07 INFO Received uploads request method=POST path=/42542026/11/12 11:50:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042552026/11/12 11:50:07 WARN Force mode enabled - objects will be deleted immediately without grace period42562026/11/12 11:50:08 INFO Vacuumed table table=pending_closures42572026/11/12 11:50:08 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=042582026/11/12 11:50:08 INFO Vacuumed table table=commit_receipts42592026/11/12 11:50:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042602026/11/12 11:50:08 INFO Vacuumed table table=pending_objects42612026/11/12 11:50:08 INFO Vacuumed table table=pending_closures42622026/11/12 11:50:08 INFO Vacuumed table table=multipart_uploads42632026/11/12 11:50:08 INFO Vacuumed table table=pending_closures42642026/11/12 11:50:08 INFO Vacuumed table table=closures42652026/11/12 11:50:08 INFO Aborted multipart uploads count=0 kept=042662026/11/12 11:50:08 INFO Vacuumed table table=commit_receipts42672026/11/12 11:50:08 INFO Vacuumed table table=commit_receipts42682026/11/12 11:50:08 INFO Vacuumed table table=objects42692026/11/12 11:50:08 INFO Aborted multipart uploads count=0 kept=042702026/11/12 11:50:08 INFO Vacuumed table table=pending_objects42712026/11/12 11:50:08 INFO Vacuumed table table=multipart_uploads42722026/11/12 11:50:08 INFO Vacuumed table table=pending_objects42732026/11/12 11:50:08 INFO Vacuumed table table=multipart_uploads42742026/11/12 11:50:08 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=042752026/11/12 11:50:08 INFO Vacuumed table table=closures42762026/11/12 11:50:08 WARN Force mode enabled - objects will be deleted immediately without grace period42772026/11/12 11:50:08 INFO Vacuumed table table=closures42782026/11/12 11:50:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042792026/11/12 11:50:08 INFO Vacuumed table table=pending_closures42802026/11/12 11:50:08 INFO Vacuumed table table=objects42812026/11/12 11:50:08 INFO Vacuumed table table=objects42822026/11/12 11:50:08 INFO Vacuumed table table=pending_closures42832026/11/12 11:50:08 INFO Vacuumed table table=commit_receipts42842026/11/12 11:50:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042852026/11/12 11:50:08 INFO Vacuumed table table=commit_receipts42862026/11/12 11:50:08 INFO Vacuumed table table=pending_closures42872026/11/12 11:50:08 INFO Received push request method=POST path=/42882026/11/12 11:50:08 INFO Received complete push request method=POST path=/42892026/11/12 11:50:08 INFO Vacuumed table table=pending_objects42902026/11/12 11:50:08 INFO Vacuumed table table=commit_receipts42912026/11/12 11:50:08 INFO Vacuumed table table=multipart_uploads42922026/11/12 11:50:08 INFO Received push request method=POST path=/42932026/11/12 11:50:08 INFO Completed push id=32 objects=7 duration=2.14314ms42942026/11/12 11:50:08 INFO Vacuumed table table=closures42952026/11/12 11:50:08 INFO Vacuumed table table=pending_objects42962026/11/12 11:50:08 INFO Received push request method=POST path=/42972026/11/12 11:50:08 INFO Vacuumed table table=pending_objects42982026/11/12 11:50:08 INFO Vacuumed table table=multipart_uploads42992026/11/12 11:50:08 INFO Vacuumed table table=multipart_uploads43002026/11/12 11:50:08 INFO Vacuumed table table=objects43012026/11/12 11:50:08 INFO Vacuumed table table=closures43022026/11/12 11:50:08 INFO Vacuumed table table=closures43032026/11/12 11:50:08 INFO Received complete push request method=POST path=/43042026/11/12 11:50:08 INFO Vacuumed table table=objects43052026/11/12 11:50:08 INFO Vacuumed table table=objects43062026/11/12 11:50:08 INFO Completed push id=10 objects=2 duration=1.79231ms43072026/11/12 11:50:08 INFO Received complete upload request method=POST path=/43082026/11/12 11:50:08 INFO Received push request method=POST path=/43092026/11/12 11:50:08 INFO Completed upload id=15 objects=3 duration=2.29214ms43102026/11/12 11:50:08 INFO Received push request method=POST path=/43112026/11/12 11:50:08 INFO Received complete upload request method=POST path=/43122026/11/12 11:50:08 INFO Aborted multipart uploads count=0 kept=043132026/11/12 11:50:08 INFO Received complete push request method=POST path=/43142026/11/12 11:50:08 INFO Completed upload id=17 objects=25 duration=3.43488ms43152026/11/12 11:50:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043162026/11/12 11:50:08 INFO Completed push id=26 objects=4 duration=1.38192ms43172026/11/12 11:50:08 INFO Vacuumed table table=pending_closures43182026/11/12 11:50:08 INFO Vacuumed table table=commit_receipts43192026/11/12 11:50:08 INFO Vacuumed table table=pending_objects43202026/11/12 11:50:08 INFO Vacuumed table table=multipart_uploads43212026/11/12 11:50:08 INFO Vacuumed table table=closures43222026/11/12 11:50:08 INFO Received complete push request method=POST path=/43232026/11/12 11:50:08 INFO Vacuumed table table=objects43242026/11/12 11:50:08 INFO Received complete upload request method=POST path=/43252026/11/12 11:50:08 INFO Completed push id=7 objects=7 duration=1.84027ms43262026/11/12 11:50:08 INFO Received push request method=POST path=/43272026/11/12 11:50:08 INFO Received complete push request method=POST path=/43282026/11/12 11:50:08 INFO Completed upload id=20 objects=3 duration=2.65988ms43292026/11/12 11:50:08 INFO Received push request method=POST path=/43302026/11/12 11:50:08 INFO Completed push id=27 objects=25 duration=3.35143ms43312026/11/12 11:50:08 INFO Received push request method=POST path=/43322026/11/12 11:50:08 INFO Aborted multipart uploads count=0 kept=043332026/11/12 11:50:08 INFO Received uploads request method=POST path=/43342026/11/12 11:50:08 INFO Received complete upload request method=POST path=/43352026/11/12 11:50:08 WARN Force mode enabled - objects will be deleted immediately without grace period43362026/11/12 11:50:08 INFO Received complete upload request method=POST path=/43372026/11/12 11:50:08 INFO Completed upload id=22 objects=22 duration=1.4235ms43382026/11/12 11:50:08 INFO Received uploads request method=POST path=/43392026/11/12 11:50:08 INFO Completed upload id=17 objects=5 duration=1.69324ms43402026/11/12 11:50:08 INFO Received push request method=POST path=/43412026/11/12 11:50:08 INFO Received complete upload request method=POST path=/43422026/11/12 11:50:08 INFO Received uploads request method=POST path=/43432026/11/12 11:50:08 INFO Received uploads request method=POST path=/43442026/11/12 11:50:08 INFO Completed upload id=28 objects=13 duration=1.25888ms43452026/11/12 11:50:08 INFO Received push request method=POST path=/43462026/11/12 11:50:08 INFO Received complete push request method=POST path=/43472026/11/12 11:50:08 INFO Completed push id=12 objects=6 duration=1.28939ms43482026/11/12 11:50:08 INFO Received push request method=POST path=/43492026/11/12 11:50:08 INFO Received push request method=POST path=/43502026/11/12 11:50:08 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."43512026/11/12 11:50:08 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."43522026/11/12 11:50:08 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."43532026/11/12 11:50:08 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."43542026/11/12 11:50:08 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."43552026/11/12 11:50:08 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."43562026/11/12 11:50:08 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."43572026/11/12 11:50:08 INFO Aborted multipart uploads count=0 kept=043582026/11/12 11:50:08 INFO Received uploads request method=POST path=/43592026/11/12 11:50:08 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043602026/11/12 11:50:08 INFO Vacuumed table table=pending_closures43612026/11/12 11:50:08 INFO Vacuumed table table=commit_receipts43622026/11/12 11:50:08 INFO Received push request method=POST path=/43632026/11/12 11:50:08 INFO Vacuumed table table=pending_objects43642026/11/12 11:50:08 INFO Vacuumed table table=multipart_uploads43652026/11/12 11:50:08 INFO Vacuumed table table=closures43662026/11/12 11:50:08 INFO Vacuumed table table=objects43672026/11/12 11:50:08 INFO Received complete upload request method=POST path=/43682026/11/12 11:50:08 INFO Completed upload id=21 objects=2 duration=1.21206ms43692026/11/12 11:50:08 INFO Received complete push request method=POST path=/43702026/11/12 11:50:08 INFO Received uploads request method=POST path=/43712026/11/12 11:50:08 INFO Received complete push request method=POST path=/43722026/11/12 11:50:08 INFO Received complete push request method=POST path=/43732026/11/12 11:50:08 INFO Completed push id=29 objects=3 duration=1.53109ms43742026/11/12 11:50:08 INFO Completed push id=19 objects=3 duration=1.88261ms43752026/11/12 11:50:08 INFO Completed push id=16 objects=14 duration=3.33156ms43762026/11/12 11:50:08 INFO Received complete upload request method=POST path=/43772026/11/12 11:50:08 INFO Completed upload id=22 objects=6 duration=1.34839ms43782026/11/12 11:50:08 INFO Received complete upload request method=POST path=/43792026/11/12 11:50:08 INFO Completed upload id=26 objects=6 duration=1.33428ms43802026/11/12 11:50:08 INFO Aborted multipart uploads count=0 kept=043812026/11/12 11:50:08 INFO Received push request method=POST path=/43822026/11/12 11:50:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043832026/11/12 11:50:08 INFO Received complete push request method=POST path=/43842026/11/12 11:50:08 INFO Vacuumed table table=pending_closures43852026/11/12 11:50:08 INFO Vacuumed table table=commit_receipts43862026/11/12 11:50:08 INFO Completed push id=26 objects=5 duration=2.69057ms43872026/11/12 11:50:08 INFO Aborted multipart uploads count=0 kept=043882026/11/12 11:50:08 INFO Vacuumed table table=pending_objects43892026/11/12 11:50:08 INFO Received complete push request method=POST path=/43902026/11/12 11:50:08 INFO Vacuumed table table=multipart_uploads43912026/11/12 11:50:08 INFO Vacuumed table table=closures43922026/11/12 11:50:08 INFO Vacuumed table table=objects43932026/11/12 11:50:08 INFO Aborted multipart uploads count=0 kept=043942026/11/12 11:50:08 INFO Completed push id=24 objects=5 duration=2.97762ms43952026/11/12 11:50:08 INFO Received push request method=POST path=/43962026/11/12 11:50:08 WARN Force mode enabled - objects will be deleted immediately without grace period43972026/11/12 11:50:08 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043982026/11/12 11:50:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043992026/11/12 11:50:08 INFO Received complete upload request method=POST path=/44002026/11/12 11:50:08 INFO Vacuumed table table=pending_closures44012026/11/12 11:50:08 INFO Vacuumed table table=commit_receipts44022026/11/12 11:50:08 INFO Vacuumed table table=pending_closures44032026/11/12 11:50:08 INFO Completed upload id=21 objects=3 duration=1.59857ms44042026/11/12 11:50:08 INFO Received complete upload request method=POST path=/44052026/11/12 11:50:08 INFO Aborted multipart uploads count=0 kept=044062026/11/12 11:50:08 INFO Vacuumed table table=commit_receipts44072026/11/12 11:50:08 INFO Vacuumed table table=pending_objects44082026/11/12 11:50:08 INFO Completed upload id=31 objects=2 duration=1.34198ms44092026/11/12 11:50:08 INFO Vacuumed table table=multipart_uploads44102026/11/12 11:50:08 INFO Received uploads request method=POST path=/44112026/11/12 11:50:08 INFO Vacuumed table table=pending_objects44122026/11/12 11:50:08 INFO Vacuumed table table=multipart_uploads44132026/11/12 11:50:08 INFO Vacuumed table table=closures44142026/11/12 11:50:08 WARN Force mode enabled - objects will be deleted immediately without grace period44152026/11/12 11:50:08 INFO Vacuumed table table=closures44162026/11/12 11:50:08 INFO Received uploads request method=POST path=/44172026/11/12 11:50:08 INFO Vacuumed table table=objects44182026/11/12 11:50:08 INFO Received complete upload request method=POST path=/44192026/11/12 11:50:08 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044202026/11/12 11:50:08 INFO Vacuumed table table=objects44212026/11/12 11:50:08 INFO Completed upload id=24 objects=7 duration=1.27512ms44222026/11/12 11:50:08 INFO Vacuumed table table=pending_closures44232026/11/12 11:50:08 INFO Vacuumed table table=commit_receipts44242026/11/12 11:50:08 INFO Vacuumed table table=pending_objects44252026/11/12 11:50:08 INFO Vacuumed table table=multipart_uploads44262026/11/12 11:50:08 INFO Vacuumed table table=closures44272026/11/12 11:50:08 INFO Received complete push request method=POST path=/44282026/11/12 11:50:08 INFO Vacuumed table table=objects44292026/11/12 11:50:08 INFO Completed push id=13 objects=22 duration=1.82035ms44302026/11/12 11:50:08 INFO Received complete upload request method=POST path=/44312026/11/12 11:50:08 INFO Received complete push request method=POST path=/44322026/11/12 11:50:08 INFO Completed upload id=24 objects=14 duration=1.25937ms44332026/11/12 11:50:08 INFO Received complete push request method=POST path=/44342026/11/12 11:50:08 INFO Completed push id=20 objects=4 duration=1.69996ms44352026/11/12 11:50:08 INFO Completed push id=17 objects=22 duration=2.76301ms44362026/11/12 11:50:08 INFO Aborted multipart uploads count=0 kept=044372026/11/12 11:50:08 INFO Aborted multipart uploads count=0 kept=044382026/11/12 11:50:08 WARN Force mode enabled - objects will be deleted immediately without grace period44392026/11/12 11:50:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044402026/11/12 11:50:08 INFO Received complete push request method=POST path=/44412026/11/12 11:50:08 INFO Vacuumed table table=pending_closures44422026/11/12 11:50:08 INFO Completed push id=33 objects=6 duration=1.12083ms44432026/11/12 11:50:08 INFO Vacuumed table table=commit_receipts44442026/11/12 11:50:08 INFO Vacuumed table table=pending_objects44452026/11/12 11:50:08 INFO Vacuumed table table=multipart_uploads44462026/11/12 11:50:08 INFO Vacuumed table table=closures44472026/11/12 11:50:08 INFO Received complete upload request method=POST path=/44482026/11/12 11:50:08 INFO Vacuumed table table=objects44492026/11/12 11:50:08 INFO Completed upload id=29 objects=5 duration=1.46373ms44502026/11/12 11:50:08 INFO Aborted multipart uploads count=0 kept=044512026/11/12 11:50:08 INFO Aborted multipart uploads count=0 kept=044522026/11/12 11:50:08 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044532026/11/12 11:50:08 INFO Received complete push request method=POST path=/44542026/11/12 11:50:08 INFO Vacuumed table table=pending_closures44552026/11/12 11:50:08 INFO Vacuumed table table=commit_receipts44562026/11/12 11:50:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=7 objects-failed-to-delete=044572026/11/12 11:50:08 INFO Completed push id=16 objects=26 duration=2.05074ms44582026/11/12 11:50:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044592026/11/12 11:50:08 INFO Vacuumed table table=pending_objects44602026/11/12 11:50:08 INFO Vacuumed table table=multipart_uploads44612026/11/12 11:50:08 INFO Received complete push request method=POST path=/44622026/11/12 11:50:08 INFO Vacuumed table table=pending_closures44632026/11/12 11:50:08 INFO Vacuumed table table=pending_closures44642026/11/12 11:50:08 INFO Vacuumed table table=commit_receipts44652026/11/12 11:50:08 INFO Vacuumed table table=closures44662026/11/12 11:50:08 INFO Received complete push request method=POST path=/44672026/11/12 11:50:08 INFO Completed push id=19 objects=3 duration=1.37319ms44682026/11/12 11:50:08 INFO Received uploads request method=POST path=/44692026/11/12 11:50:08 INFO Vacuumed table table=commit_receipts44702026/11/12 11:50:08 INFO Received uploads request method=POST path=/44712026/11/12 11:50:08 INFO Vacuumed table table=objects44722026/11/12 11:50:08 INFO Vacuumed table table=pending_objects44732026/11/12 11:50:08 INFO Completed push id=21 objects=6 duration=1.31495ms44742026/11/12 11:50:08 INFO Vacuumed table table=multipart_uploads44752026/11/12 11:50:08 INFO Vacuumed table table=closures44762026/11/12 11:50:08 INFO Vacuumed table table=pending_objects44772026/11/12 11:50:08 INFO Vacuumed table table=multipart_uploads44782026/11/12 11:50:08 INFO Vacuumed table table=objects44792026/11/12 11:50:08 INFO Vacuumed table table=closures44802026/11/12 11:50:08 INFO Vacuumed table table=objects44812026/11/12 11:50:08 INFO Received uploads request method=POST path=/44822026/11/12 11:50:08 INFO Aborted multipart uploads count=0 kept=044832026/11/12 11:50:08 INFO Aborted multipart uploads count=0 kept=044842026/11/12 11:50:08 INFO Received complete upload request method=POST path=/44852026/11/12 11:50:08 WARN Force mode enabled - objects will be deleted immediately without grace period44862026/11/12 11:50:08 INFO Completed upload id=33 objects=5 duration=3.01539ms44872026/11/12 11:50:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044882026/11/12 11:50:08 INFO Received complete upload request method=POST path=/44892026/11/12 11:50:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044902026/11/12 11:50:08 INFO Vacuumed table table=pending_closures44912026/11/12 11:50:08 INFO Completed upload id=22 objects=8 duration=1.44228ms44922026/11/12 11:50:08 INFO Vacuumed table table=commit_receipts44932026/11/12 11:50:08 INFO Vacuumed table table=pending_closures44942026/11/12 11:50:08 INFO Vacuumed table table=pending_objects44952026/11/12 11:50:08 INFO Vacuumed table table=commit_receipts44962026/11/12 11:50:08 INFO Vacuumed table table=multipart_uploads44972026/11/12 11:50:08 INFO Vacuumed table table=closures44982026/11/12 11:50:08 INFO Vacuumed table table=pending_objects44992026/11/12 11:50:08 INFO Vacuumed table table=objects45002026/11/12 11:50:08 INFO Vacuumed table table=multipart_uploads45012026/11/12 11:50:08 INFO Vacuumed table table=closures45022026/11/12 11:50:08 INFO Aborted multipart uploads count=0 kept=045032026/11/12 11:50:08 INFO Vacuumed table table=objects45042026/11/12 11:50:08 WARN Force mode enabled - objects will be deleted immediately without grace period45052026/11/12 11:50:08 INFO Received uploads request method=POST path=/45062026/11/12 11:50:08 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045072026/11/12 11:50:08 INFO Received complete upload request method=POST path=/45082026/11/12 11:50:08 INFO Vacuumed table table=pending_closures45092026/11/12 11:50:08 INFO Completed upload id=22 objects=7 duration=1.43455ms45102026/11/12 11:50:08 INFO Vacuumed table table=commit_receipts45112026/11/12 11:50:08 INFO Received push request method=POST path=/45122026/11/12 11:50:08 INFO Vacuumed table table=pending_objects45132026/11/12 11:50:08 INFO Vacuumed table table=multipart_uploads45142026/11/12 11:50:08 INFO Vacuumed table table=closures45152026/11/12 11:50:08 INFO Received push request method=POST path=/45162026/11/12 11:50:08 INFO Vacuumed table table=objects45172026/11/12 11:50:08 INFO Aborted multipart uploads count=0 kept=045182026/11/12 11:50:08 WARN Force mode enabled - objects will be deleted immediately without grace period45192026/11/12 11:50:08 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045202026/11/12 11:50:08 INFO Vacuumed table table=pending_closures45212026/11/12 11:50:08 INFO Vacuumed table table=commit_receipts45222026/11/12 11:50:08 INFO Vacuumed table table=pending_objects45232026/11/12 11:50:08 INFO Received complete push request method=POST path=/45242026/11/12 11:50:08 INFO Received complete upload request method=POST path=/45252026/11/12 11:50:08 INFO Vacuumed table table=multipart_uploads45262026/11/12 11:50:08 INFO Received complete push request method=POST path=/45272026/11/12 11:50:08 INFO Vacuumed table table=closures45282026/11/12 11:50:08 INFO Completed push id=23 objects=5 duration=1.77244ms45292026/11/12 11:50:08 INFO Vacuumed table table=objects45302026/11/12 11:50:08 INFO Completed upload id=13 objects=14 duration=3.68668ms45312026/11/12 11:50:08 INFO Completed push id=7 objects=24 duration=4.46994ms45322026/11/12 11:50:08 INFO Received complete push request method=POST path=/45332026/11/12 11:50:08 INFO Completed push id=21 objects=26 duration=1.64798ms45342026/11/12 11:50:08 INFO Received uploads request method=POST path=/45352026/11/12 11:50:08 INFO Received complete push request method=POST path=/45362026/11/12 11:50:08 INFO Completed push id=24 objects=4 duration=1.16343ms45372026/11/12 11:50:08 INFO Aborted multipart uploads count=0 kept=045382026/11/12 11:50:08 INFO Aborted multipart uploads count=0 kept=045392026/11/12 11:50:08 INFO Received uploads request method=POST path=/45402026/11/12 11:50:08 WARN Force mode enabled - objects will be deleted immediately without grace period45412026/11/12 11:50:08 WARN Force mode enabled - objects will be deleted immediately without grace period45422026/11/12 11:50:08 INFO Received push request method=POST path=/45432026/11/12 11:50:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045442026/11/12 11:50:08 INFO Received complete push request method=POST path=/45452026/11/12 11:50:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045462026/11/12 11:50:08 INFO Vacuumed table table=pending_closures45472026/11/12 11:50:08 INFO Vacuumed table table=commit_receipts45482026/11/12 11:50:08 INFO Vacuumed table table=pending_closures45492026/11/12 11:50:08 INFO Vacuumed table table=pending_objects45502026/11/12 11:50:08 INFO Vacuumed table table=commit_receipts45512026/11/12 11:50:08 INFO Vacuumed table table=multipart_uploads45522026/11/12 11:50:08 INFO Received push request method=POST path=/45532026/11/12 11:50:08 INFO Completed push id=27 objects=6 duration=3.91865ms45542026/11/12 11:50:08 INFO Vacuumed table table=closures45552026/11/12 11:50:08 INFO Vacuumed table table=pending_objects45562026/11/12 11:50:08 INFO Vacuumed table table=multipart_uploads45572026/11/12 11:50:08 INFO Vacuumed table table=objects45582026/11/12 11:50:08 INFO Vacuumed table table=closures45592026/11/12 11:50:08 INFO Received complete upload request method=POST path=/45602026/11/12 11:50:08 INFO Vacuumed table table=objects45612026/11/12 11:50:08 INFO Completed upload id=24 objects=8 duration=1.23609ms45622026/11/12 11:50:08 INFO Received push request method=POST path=/45632026/11/12 11:50:08 INFO Received uploads request method=POST path=/45642026/11/12 11:50:08 INFO Aborted multipart uploads count=0 kept=045652026/11/12 11:50:08 INFO Received complete push request method=POST path=/45662026/11/12 11:50:08 INFO Completed push id=25 objects=9 duration=1.31839ms45672026/11/12 11:50:08 INFO Received complete upload request method=POST path=/45682026/11/12 11:50:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045692026/11/12 11:50:08 INFO Vacuumed table table=pending_closures45702026/11/12 11:50:08 INFO Completed upload id=25 objects=6 duration=2.16514ms45712026/11/12 11:50:08 INFO Vacuumed table table=commit_receipts45722026/11/12 11:50:08 INFO Received uploads request method=POST path=/45732026/11/12 11:50:08 INFO Received complete upload request method=POST path=/45742026/11/12 11:50:08 INFO Received complete upload request method=POST path=/45752026/11/12 11:50:08 INFO Vacuumed table table=pending_objects45762026/11/12 11:50:08 INFO Vacuumed table table=multipart_uploads45772026/11/12 11:50:08 INFO Completed upload id=21 objects=25 duration=2.15421ms45782026/11/12 11:50:08 INFO Received push request method=POST path=/45792026/11/12 11:50:08 INFO Vacuumed table table=closures45802026/11/12 11:50:08 INFO Completed upload id=28 objects=17 duration=1.69405ms45812026/11/12 11:50:08 INFO Vacuumed table table=objects45822026/11/12 11:50:08 INFO Received push request method=POST path=/45832026/11/12 11:50:08 INFO Received complete upload request method=POST path=/45842026/11/12 11:50:08 INFO Completed upload id=28 objects=5 duration=1.01497ms45852026/11/12 11:50:08 INFO Received push request method=POST path=/45862026/11/12 11:50:08 INFO Received uploads request method=POST path=/45872026/11/12 11:50:08 INFO Received complete push request method=POST path=/45882026/11/12 11:50:08 INFO Received uploads request method=POST path=/45892026/11/12 11:50:08 INFO Received complete upload request method=POST path=/45902026/11/12 11:50:08 INFO Completed push id=23 objects=28 duration=3.38791ms45912026/11/12 11:50:08 INFO Completed upload id=26 objects=3 duration=967.38µs45922026/11/12 11:50:08 INFO Received uploads request method=POST path=/45932026/11/12 11:50:08 INFO Received uploads request method=POST path=/45942026/11/12 11:50:08 INFO Received uploads request method=POST path=/45952026/11/12 11:50:08 INFO Received complete push request method=POST path=/45962026/11/12 11:50:08 INFO Completed push id=29 objects=2 duration=1.27255ms45972026/11/12 11:50:08 INFO Received complete push request method=POST path=/45982026/11/12 11:50:08 INFO Received complete upload request method=POST path=/45992026/11/12 11:50:08 INFO Completed upload id=15 objects=2 duration=1.30186ms46002026/11/12 11:50:08 INFO Completed push id=22 objects=3 duration=3.68555ms46012026/11/12 11:50:08 INFO Received complete push request method=POST path=/46022026/11/12 11:50:08 INFO Received uploads request method=POST path=/46032026/11/12 11:50:08 INFO Received complete push request method=POST path=/46042026/11/12 11:50:08 INFO Received uploads request method=POST path=/46052026/11/12 11:50:08 INFO Received complete upload request method=POST path=/46062026/11/12 11:50:08 INFO Received push request method=POST path=/46072026/11/12 11:50:08 INFO Received push request method=POST path=/46082026/11/12 11:50:08 INFO Received uploads request method=POST path=/46092026/11/12 11:50:08 INFO Received complete push request method=POST path=/46102026/11/12 11:50:08 INFO Received uploads request method=POST path=/46112026/11/12 11:50:08 INFO Completed upload id=25 objects=4 duration=164.408967ms46122026/11/12 11:50:08 INFO Completed push id=31 objects=6 duration=167.529797ms46132026/11/12 11:50:08 INFO Completed push id=19 objects=14 duration=167.082027ms46142026/11/12 11:50:08 INFO Completed push id=31 objects=6 duration=156.453118ms46152026/11/12 11:50:08 INFO Received complete upload request method=POST path=/46162026/11/12 11:50:08 INFO Received complete upload request method=POST path=/46172026/11/12 11:50:08 INFO Completed upload id=27 objects=7 duration=1.49616ms46182026/11/12 11:50:08 INFO Completed upload id=30 objects=5 duration=1.60111ms46192026/11/12 11:50:08 INFO Received complete push request method=POST path=/46202026/11/12 11:50:08 INFO Received uploads request method=POST path=/46212026/11/12 11:50:08 INFO Completed push id=32 objects=9 duration=1.8382ms46222026/11/12 11:50:08 INFO Received uploads request method=POST path=/46232026/11/12 11:50:08 INFO Received uploads request method=POST path=/46242026/11/12 11:50:08 INFO Received complete upload request method=POST path=/46252026/11/12 11:50:08 INFO Received uploads request method=POST path=/46262026/11/12 11:50:08 INFO Received complete upload request method=POST path=/46272026/11/12 11:50:08 INFO Received push request method=POST path=/46282026/11/12 11:50:08 INFO Received push request method=POST path=/46292026/11/12 11:50:08 INFO Completed upload id=28 objects=3 duration=1.24645ms46302026/11/12 11:50:08 INFO Completed upload id=12 objects=6 duration=2.11338ms46312026/11/12 11:50:08 INFO Received complete upload request method=POST path=/46322026/11/12 11:50:08 INFO Completed upload id=27 objects=8 duration=1.43944ms46332026/11/12 11:50:08 INFO Received complete push request method=POST path=/46342026/11/12 11:50:08 INFO Completed push id=31 objects=3 duration=1.84481ms46352026/11/12 11:50:08 INFO Received complete upload request method=POST path=/46362026/11/12 11:50:08 INFO Completed upload id=31 objects=3 duration=2.63432ms46372026/11/12 11:50:08 INFO Aborted multipart uploads count=0 kept=046382026/11/12 11:50:08 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046392026/11/12 11:50:08 INFO Vacuumed table table=pending_closures46402026/11/12 11:50:08 INFO Vacuumed table table=commit_receipts46412026/11/12 11:50:08 INFO Vacuumed table table=pending_objects46422026/11/12 11:50:08 INFO Vacuumed table table=multipart_uploads46432026/11/12 11:50:08 INFO Vacuumed table table=closures46442026/11/12 11:50:08 INFO Vacuumed table table=objects46452026/11/12 11:50:08 INFO Received push request method=POST path=/46462026/11/12 11:50:08 INFO Received uploads request method=POST path=/46472026/11/12 11:50:08 INFO Received complete push request method=POST path=/46482026/11/12 11:50:08 INFO Received complete push request method=POST path=/46492026/11/12 11:50:08 INFO Received push request method=POST path=/46502026/11/12 11:50:08 INFO Received uploads request method=POST path=/46512026/11/12 11:50:08 INFO Completed push id=17 objects=3 duration=2.57302ms46522026/11/12 11:50:08 INFO Completed push id=11 objects=9 duration=2.59914ms46532026/11/12 11:50:08 INFO Received complete upload request method=POST path=/46542026/11/12 11:50:08 INFO Completed upload id=33 objects=9 duration=1.15121ms46552026/11/12 11:50:08 INFO Received complete push request method=POST path=/46562026/11/12 11:50:08 INFO Completed push id=29 objects=23 duration=1.61608ms46572026/11/12 11:50:08 INFO Received complete upload request method=POST path=/46582026/11/12 11:50:08 INFO Received complete upload request method=POST path=/46592026/11/12 11:50:08 INFO Completed upload id=20 objects=6 duration=1.64956ms46602026/11/12 11:50:08 INFO Aborted multipart uploads count=0 kept=046612026/11/12 11:50:08 INFO Completed upload id=22 objects=2 duration=1.45253ms46622026/11/12 11:50:08 INFO Received complete push request method=POST path=/46632026/11/12 11:50:08 WARN Force mode enabled - objects will be deleted immediately without grace period46642026/11/12 11:50:08 INFO Completed push id=24 objects=3 duration=1.51588ms46652026/11/12 11:50:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046662026/11/12 11:50:08 INFO Vacuumed table table=pending_closures46672026/11/12 11:50:08 INFO Received complete push request method=POST path=/46682026/11/12 11:50:08 INFO Received push request method=POST path=/46692026/11/12 11:50:08 INFO Vacuumed table table=commit_receipts46702026/11/12 11:50:08 INFO Completed push id=20 objects=8 duration=1.29231ms46712026/11/12 11:50:08 INFO Vacuumed table table=pending_objects46722026/11/12 11:50:08 INFO Vacuumed table table=multipart_uploads46732026/11/12 11:50:08 INFO Vacuumed table table=closures46742026/11/12 11:50:08 INFO Vacuumed table table=objects46752026/11/12 11:50:08 INFO Received complete push request method=POST path=/46762026/11/12 11:50:08 INFO Completed push id=32 objects=6 duration=1.53276ms46772026/11/12 11:50:08 INFO Received push request method=POST path=/46782026/11/12 11:50:08 INFO Received complete push request method=POST path=/46792026/11/12 11:50:08 INFO Aborted multipart uploads count=0 kept=046802026/11/12 11:50:08 INFO Completed push id=13 objects=2 duration=1.50151ms46812026/11/12 11:50:08 INFO Received complete push request method=POST path=/46822026/11/12 11:50:08 INFO Aborted multipart uploads count=0 kept=046832026/11/12 11:50:08 INFO Completed push id=30 objects=6 duration=1.23378ms46842026/11/12 11:50:08 INFO Received complete push request method=POST path=/46852026/11/12 11:50:08 INFO Received complete push request method=POST path=/46862026/11/12 11:50:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046872026/11/12 11:50:08 WARN Force mode enabled - objects will be deleted immediately without grace period46882026/11/12 11:50:08 INFO Completed push id=23 objects=5 duration=1.591ms46892026/11/12 11:50:08 INFO Completed push id=15 objects=24 duration=2.35087ms46902026/11/12 11:50:08 INFO Vacuumed table table=pending_closures46912026/11/12 11:50:08 INFO Vacuumed table table=commit_receipts46922026/11/12 11:50:08 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046932026/11/12 11:50:08 INFO Received push request method=POST path=/46942026/11/12 11:50:08 INFO Vacuumed table table=pending_objects46952026/11/12 11:50:08 INFO Vacuumed table table=pending_closures46962026/11/12 11:50:08 INFO Aborted multipart uploads count=0 kept=046972026/11/12 11:50:08 INFO Vacuumed table table=multipart_uploads46982026/11/12 11:50:08 INFO Vacuumed table table=commit_receipts46992026/11/12 11:50:08 INFO Vacuumed table table=closures47002026/11/12 11:50:08 INFO Received complete upload request method=POST path=/47012026/11/12 11:50:08 INFO Vacuumed table table=objects47022026/11/12 11:50:08 INFO Received uploads request method=POST path=/47032026/11/12 11:50:08 INFO Vacuumed table table=pending_objects47042026/11/12 11:50:08 WARN Force mode enabled - objects will be deleted immediately without grace period47052026/11/12 11:50:08 INFO Vacuumed table table=multipart_uploads47062026/11/12 11:50:08 INFO Vacuumed table table=closures47072026/11/12 11:50:08 INFO Completed upload id=19 objects=2 duration=3.05914ms47082026/11/12 11:50:08 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047092026/11/12 11:50:08 INFO Vacuumed table table=objects47102026/11/12 11:50:08 INFO Received uploads request method=POST path=/47112026/11/12 11:50:08 INFO Vacuumed table table=pending_closures47122026/11/12 11:50:08 INFO Vacuumed table table=commit_receipts47132026/11/12 11:50:08 INFO Vacuumed table table=pending_objects47142026/11/12 11:50:08 INFO Received complete upload request method=POST path=/47152026/11/12 11:50:08 INFO Received complete upload request method=POST path=/47162026/11/12 11:50:08 INFO Vacuumed table table=multipart_uploads47172026/11/12 11:50:08 INFO Completed upload id=33 objects=7 duration=984.42µs47182026/11/12 11:50:08 INFO Completed upload id=27 objects=13 duration=1.28722ms47192026/11/12 11:50:08 INFO Vacuumed table table=closures47202026/11/12 11:50:08 INFO Vacuumed table table=objects47212026/11/12 11:50:08 INFO Received uploads request method=POST path=/47222026/11/12 11:50:08 INFO Received complete upload request method=POST path=/47232026/11/12 11:50:08 INFO Received complete upload request method=POST path=/47242026/11/12 11:50:08 INFO Completed upload id=24 objects=3 duration=1.18741ms47252026/11/12 11:50:08 INFO Completed upload id=19 objects=2 duration=1.32712ms47262026/11/12 11:50:08 INFO Received complete push request method=POST path=/47272026/11/12 11:50:08 INFO Received complete push request method=POST path=/47282026/11/12 11:50:08 INFO Completed push id=33 objects=3 duration=1.09063ms47292026/11/12 11:50:08 INFO Received uploads request method=POST path=/47302026/11/12 11:50:08 INFO Completed push id=1 objects=9 duration=1.86649ms47312026/11/12 11:50:08 INFO Aborted multipart uploads count=0 kept=047322026/11/12 11:50:08 INFO Received push request method=POST path=/47332026/11/12 11:50:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047342026/11/12 11:50:08 INFO Vacuumed table table=pending_closures47352026/11/12 11:50:08 INFO Vacuumed table table=commit_receipts47362026/11/12 11:50:08 INFO Received uploads request method=POST path=/47372026/11/12 11:50:08 INFO Vacuumed table table=pending_objects47382026/11/12 11:50:08 INFO Vacuumed table table=multipart_uploads47392026/11/12 11:50:08 INFO Received complete upload request method=POST path=/47402026/11/12 11:50:08 INFO Vacuumed table table=closures47412026/11/12 11:50:08 INFO Aborted multipart uploads count=0 kept=047422026/11/12 11:50:08 INFO Completed upload id=34 objects=3 duration=1.06647ms47432026/11/12 11:50:08 INFO Vacuumed table table=objects47442026/11/12 11:50:08 INFO Received complete upload request method=POST path=/47452026/11/12 11:50:08 INFO Completed upload id=18 objects=2 duration=2.37728ms47462026/11/12 11:50:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047472026/11/12 11:50:08 INFO Vacuumed table table=pending_closures47482026/11/12 11:50:08 INFO Received complete push request method=POST path=/47492026/11/12 11:50:08 INFO Received complete upload request method=POST path=/47502026/11/12 11:50:08 INFO Vacuumed table table=commit_receipts47512026/11/12 11:50:08 INFO Completed push id=25 objects=4 duration=1.28056ms47522026/11/12 11:50:08 INFO Aborted multipart uploads count=0 kept=047532026/11/12 11:50:08 INFO Completed upload id=24 objects=7 duration=1.68454ms47542026/11/12 11:50:08 INFO Vacuumed table table=pending_objects47552026/11/12 11:50:08 INFO Vacuumed table table=multipart_uploads47562026/11/12 11:50:08 INFO Vacuumed table table=closures47572026/11/12 11:50:08 WARN Force mode enabled - objects will be deleted immediately without grace period47582026/11/12 11:50:08 INFO Received complete push request method=POST path=/47592026/11/12 11:50:08 INFO Vacuumed table table=objects47602026/11/12 11:50:08 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047612026/11/12 11:50:08 INFO Completed push id=33 objects=7 duration=2.08953ms47622026/11/12 11:50:08 INFO Received complete push request method=POST path=/47632026/11/12 11:50:08 INFO Completed push id=34 objects=6 duration=779.54µs47642026/11/12 11:50:08 INFO Vacuumed table table=pending_closures47652026/11/12 11:50:08 INFO Vacuumed table table=commit_receipts47662026/11/12 11:50:08 INFO Received complete push request method=POST path=/47672026/11/12 11:50:08 INFO Vacuumed table table=pending_objects47682026/11/12 11:50:08 INFO Vacuumed table table=multipart_uploads47692026/11/12 11:50:08 INFO Completed push id=35 objects=11 duration=1.21221ms47702026/11/12 11:50:08 INFO Vacuumed table table=closures47712026/11/12 11:50:08 INFO Vacuumed table table=objects47722026/11/12 11:50:08 INFO Received complete push request method=POST path=/47732026/11/12 11:50:08 INFO Received complete upload request method=POST path=/47742026/11/12 11:50:08 INFO Received push request method=POST path=/47752026/11/12 11:50:08 INFO Received complete upload request method=POST path=/47762026/11/12 11:50:08 INFO Completed upload id=26 objects=8 duration=1.65733ms47772026/11/12 11:50:08 INFO Completed push id=16 objects=7 duration=2.52619ms47782026/11/12 11:50:08 INFO Received complete upload request method=POST path=/47792026/11/12 11:50:08 INFO Completed upload id=25 objects=16 duration=2.73033ms47802026/11/12 11:50:08 INFO Completed upload id=21 objects=2 duration=1.30421ms47812026/11/12 11:50:08 INFO Received complete push request method=POST path=/47822026/11/12 11:50:08 INFO Completed push id=26 objects=3 duration=1.44796ms47832026/11/12 11:50:08 INFO Received complete upload request method=POST path=/47842026/11/12 11:50:08 INFO Received uploads request method=POST path=/47852026/11/12 11:50:08 INFO Completed upload id=26 objects=3 duration=1.34043ms47862026/11/12 11:50:08 INFO Received complete push request method=POST path=/47872026/11/12 11:50:08 INFO Completed push id=35 objects=8 duration=862.53µs47882026/11/12 11:50:08 INFO Received complete push request method=POST path=/47892026/11/12 11:50:08 INFO Completed push id=27 objects=11 duration=1.6736ms47902026/11/12 11:50:08 INFO Aborted multipart uploads count=0 kept=047912026/11/12 11:50:08 WARN Force mode enabled - objects will be deleted immediately without grace period47922026/11/12 11:50:08 INFO Received complete push request method=POST path=/47932026/11/12 11:50:08 INFO Garbage collection completed failed-uploads-deleted=6 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047942026/11/12 11:50:08 INFO Completed push id=28 objects=14 duration=1.75387ms47952026/11/12 11:50:08 INFO Received uploads request method=POST path=/47962026/11/12 11:50:08 INFO Vacuumed table table=pending_closures47972026/11/12 11:50:08 INFO Vacuumed table table=commit_receipts47982026/11/12 11:50:08 INFO Received complete upload request method=POST path=/47992026/11/12 11:50:08 INFO Aborted multipart uploads count=0 kept=048002026/11/12 11:50:08 WARN Force mode enabled - objects will be deleted immediately without grace period48012026/11/12 11:50:08 INFO Completed upload id=28 objects=8 duration=1.18272ms48022026/11/12 11:50:08 INFO Vacuumed table table=pending_objects48032026/11/12 11:50:08 INFO Vacuumed table table=multipart_uploads48042026/11/12 11:50:08 INFO Aborted multipart uploads count=0 kept=048052026/11/12 11:50:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048062026/11/12 11:50:08 INFO Received push request method=POST path=/48072026/11/12 11:50:08 INFO Vacuumed table table=closures48082026/11/12 11:50:08 WARN Force mode enabled - objects will be deleted immediately without grace period48092026/11/12 11:50:08 INFO Vacuumed table table=pending_closures48102026/11/12 11:50:08 INFO Vacuumed table table=objects48112026/11/12 11:50:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048122026/11/12 11:50:08 INFO Vacuumed table table=commit_receipts48132026/11/12 11:50:08 INFO Received complete upload request method=POST path=/48142026/11/12 11:50:08 INFO Received complete push request method=POST path=/48152026/11/12 11:50:08 INFO Vacuumed table table=pending_closures48162026/11/12 11:50:08 INFO Completed upload id=36 objects=9 duration=1.32743ms48172026/11/12 11:50:08 INFO Vacuumed table table=pending_objects48182026/11/12 11:50:08 INFO Vacuumed table table=multipart_uploads48192026/11/12 11:50:08 INFO Completed push id=23 objects=6 duration=1.89116ms48202026/11/12 11:50:08 INFO Vacuumed table table=commit_receipts48212026/11/12 11:50:08 INFO Vacuumed table table=closures48222026/11/12 11:50:08 INFO Received complete upload request method=POST path=/48232026/11/12 11:50:08 INFO Vacuumed table table=pending_objects48242026/11/12 11:50:08 INFO Completed upload id=29 objects=26 duration=1.08642ms48252026/11/12 11:50:08 INFO Vacuumed table table=objects48262026/11/12 11:50:08 INFO Received complete upload request method=POST path=/48272026/11/12 11:50:08 INFO Vacuumed table table=multipart_uploads48282026/11/12 11:50:08 INFO Received complete upload request method=POST path=/48292026/11/12 11:50:08 INFO Vacuumed table table=closures48302026/11/12 11:50:08 INFO Completed upload id=13 objects=7 duration=1.351909ms48312026/11/12 11:50:08 INFO Vacuumed table table=objects48322026/11/12 11:50:08 INFO Completed upload id=2 objects=7 duration=1.647351ms48332026/11/12 11:50:08 INFO Received complete push request method=POST path=/48342026/11/12 11:50:08 INFO Received push request method=POST path=/48352026/11/12 11:50:08 INFO Completed push id=26 objects=23 duration=1.76328ms48362026/11/12 11:50:08 INFO Received complete upload request method=POST path=/48372026/11/12 11:50:08 INFO Completed upload id=29 objects=10 duration=1.04293ms48382026/11/12 11:50:08 INFO Aborted multipart uploads count=0 kept=048392026/11/12 11:50:08 WARN Force mode enabled - objects will be deleted immediately without grace period48402026/11/12 11:50:08 INFO Received push request method=POST path=/48412026/11/12 11:50:08 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048422026/11/12 11:50:08 INFO Vacuumed table table=pending_closures48432026/11/12 11:50:08 INFO Received complete push request method=POST path=/48442026/11/12 11:50:08 INFO Aborted multipart uploads count=0 kept=048452026/11/12 11:50:08 INFO Vacuumed table table=commit_receipts48462026/11/12 11:50:08 INFO Completed push id=24 objects=6 duration=1.28571ms48472026/11/12 11:50:08 INFO Vacuumed table table=pending_objects48482026/11/12 11:50:08 INFO Vacuumed table table=multipart_uploads48492026/11/12 11:50:08 INFO Aborted multipart uploads count=0 kept=048502026/11/12 11:50:08 INFO Vacuumed table table=closures48512026/11/12 11:50:08 INFO Vacuumed table table=objects48522026/11/12 11:50:08 WARN Force mode enabled - objects will be deleted immediately without grace period48532026/11/12 11:50:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048542026/11/12 11:50:08 INFO Received complete upload request method=POST path=/48552026/11/12 11:50:08 INFO Received push request method=POST path=/48562026/11/12 11:50:08 INFO Completed upload id=20 objects=4 duration=1.18978ms48572026/11/12 11:50:08 INFO Received complete push request method=POST path=/48582026/11/12 11:50:08 INFO Vacuumed table table=pending_closures48592026/11/12 11:50:08 INFO Completed push id=30 objects=8 duration=1.27642ms48602026/11/12 11:50:08 INFO Vacuumed table table=commit_receipts48612026/11/12 11:50:08 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=048622026/11/12 11:50:08 INFO Vacuumed table table=pending_objects48632026/11/12 11:50:08 INFO Vacuumed table table=multipart_uploads48642026/11/12 11:50:08 INFO Vacuumed table table=closures48652026/11/12 11:50:08 INFO Vacuumed table table=pending_closures48662026/11/12 11:50:08 INFO Received complete upload request method=POST path=/48672026/11/12 11:50:08 INFO Vacuumed table table=objects48682026/11/12 11:50:08 INFO Vacuumed table table=commit_receipts48692026/11/12 11:50:08 INFO Completed upload id=30 objects=20 duration=1.54208ms48702026/11/12 11:50:08 INFO Received uploads request method=POST path=/48712026/11/12 11:50:08 INFO Vacuumed table table=pending_objects48722026/11/12 11:50:08 INFO Vacuumed table table=multipart_uploads48732026/11/12 11:50:08 INFO Vacuumed table table=closures48742026/11/12 11:50:08 INFO Received uploads request method=POST path=/48752026/11/12 11:50:08 INFO Vacuumed table table=objects48762026/11/12 11:50:08 INFO Received complete upload request method=POST path=/48772026/11/12 11:50:08 INFO Completed upload id=23 objects=7 duration=1.42148ms48782026/11/12 11:50:08 INFO Received complete upload request method=POST path=/48792026/11/12 11:50:08 INFO Received push request method=POST path=/48802026/11/12 11:50:08 INFO Completed upload id=31 objects=5 duration=993.89µs48812026/11/12 11:50:08 INFO Received complete push request method=POST path=/48822026/11/12 11:50:08 INFO Received complete push request method=POST path=/48832026/11/12 11:50:08 INFO Completed push id=25 objects=7 duration=1.56131ms48842026/11/12 11:50:08 INFO Completed push id=27 objects=15 duration=1.7446ms48852026/11/12 11:50:08 INFO Received complete push request method=POST path=/48862026/11/12 11:50:08 INFO Received complete upload request method=POST path=/48872026/11/12 11:50:08 INFO Completed push id=31 objects=11 duration=1.20178ms48882026/11/12 11:50:08 INFO Completed upload id=8 objects=6 duration=1.69351ms48892026/11/12 11:50:08 INFO Received uploads request method=POST path=/48902026/11/12 11:50:08 INFO Aborted multipart uploads count=0 kept=048912026/11/12 11:50:08 INFO Received uploads request method=POST path=/48922026/11/12 11:50:08 WARN Force mode enabled - objects will be deleted immediately without grace period48932026/11/12 11:50:08 INFO Received complete push request method=POST path=/48942026/11/12 11:50:08 INFO Completed push id=32 objects=3 duration=1.14854ms48952026/11/12 11:50:08 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048962026/11/12 11:50:08 INFO Vacuumed table table=pending_closures48972026/11/12 11:50:08 INFO Received complete push request method=POST path=/48982026/11/12 11:50:08 INFO Vacuumed table table=commit_receipts48992026/11/12 11:50:08 INFO Vacuumed table table=pending_objects49002026/11/12 11:50:08 INFO Vacuumed table table=multipart_uploads49012026/11/12 11:50:08 INFO Received uploads request method=POST path=/49022026/11/12 11:50:08 INFO Received complete upload request method=POST path=/49032026/11/12 11:50:08 INFO Completed push id=9 objects=9 duration=3.92316ms49042026/11/12 11:50:08 INFO Vacuumed table table=closures49052026/11/12 11:50:08 INFO Completed upload id=32 objects=2 duration=1.62738ms49062026/11/12 11:50:08 INFO Vacuumed table table=objects49072026/11/12 11:50:08 INFO Received complete upload request method=POST path=/49082026/11/12 11:50:08 INFO Completed upload id=29 objects=4 duration=2.29546ms49092026/11/12 11:50:08 INFO Received push request method=POST path=/49102026/11/12 11:50:08 INFO Aborted multipart uploads count=0 kept=049112026/11/12 11:50:08 WARN Force mode enabled - objects will be deleted immediately without grace period49122026/11/12 11:50:08 INFO Received complete upload request method=POST path=/49132026/11/12 11:50:08 INFO Received complete push request method=POST path=/49142026/11/12 11:50:08 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049152026/11/12 11:50:08 INFO Completed push id=33 objects=3 duration=1.31865ms49162026/11/12 11:50:08 INFO Received uploads request method=POST path=/49172026/11/12 11:50:08 INFO Received complete upload request method=POST path=/49182026/11/12 11:50:08 INFO Completed upload id=26 objects=2 duration=2.09717ms49192026/11/12 11:50:08 INFO Received complete upload request method=POST path=/49202026/11/12 11:50:08 INFO Received complete upload request method=POST path=/49212026/11/12 11:50:08 INFO Vacuumed table table=pending_closures49222026/11/12 11:50:08 INFO Completed upload id=26 objects=24 duration=1.66892ms49232026/11/12 11:50:08 INFO Received uploads request method=POST path=/49242026/11/12 11:50:08 INFO Completed upload id=24 objects=18 duration=1.42228ms49252026/11/12 11:50:08 INFO Completed upload id=21 objects=8 duration=1.79441ms49262026/11/12 11:50:08 INFO Vacuumed table table=commit_receipts49272026/11/12 11:50:08 INFO Vacuumed table table=pending_objects49282026/11/12 11:50:08 INFO Vacuumed table table=multipart_uploads49292026/11/12 11:50:08 INFO Received complete push request method=POST path=/49302026/11/12 11:50:08 INFO Vacuumed table table=closures49312026/11/12 11:50:08 INFO Received complete push request method=POST path=/49322026/11/12 11:50:08 INFO Completed push id=28 objects=19 duration=1.55859ms49332026/11/12 11:50:08 INFO Vacuumed table table=objects49342026/11/12 11:50:08 INFO Completed push id=24 objects=24 duration=3.49374ms49352026/11/12 11:50:08 INFO Received uploads request method=POST path=/49362026/11/12 11:50:08 INFO Aborted multipart uploads count=0 kept=049372026/11/12 11:50:08 INFO Received complete upload request method=POST path=/49382026/11/12 11:50:08 INFO Received complete upload request method=POST path=/49392026/11/12 11:50:08 INFO Completed upload id=34 objects=3 duration=927.93µs49402026/11/12 11:50:08 INFO Completed upload id=12 objects=9 duration=1.38391ms49412026/11/12 11:50:08 INFO Received complete push request method=POST path=/49422026/11/12 11:50:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049432026/11/12 11:50:08 INFO Completed push id=22 objects=6 duration=1.50319ms49442026/11/12 11:50:08 INFO Received complete push request method=POST path=/49452026/11/12 11:50:08 INFO Vacuumed table table=pending_closures49462026/11/12 11:50:08 INFO Received complete upload request method=POST path=/49472026/11/12 11:50:08 INFO Vacuumed table table=commit_receipts49482026/11/12 11:50:08 INFO Received uploads request method=POST path=/49492026/11/12 11:50:08 INFO Completed push id=25 objects=2 duration=2.20292ms49502026/11/12 11:50:08 INFO Vacuumed table table=pending_objects49512026/11/12 11:50:08 INFO Completed upload id=21 objects=9 duration=2.57583ms49522026/11/12 11:50:08 INFO Vacuumed table table=multipart_uploads49532026/11/12 11:50:08 INFO Vacuumed table table=closures49542026/11/12 11:50:08 INFO Vacuumed table table=objects49552026/11/12 11:50:08 INFO Aborted multipart uploads count=0 kept=049562026/11/12 11:50:08 INFO Received complete upload request method=POST path=/49572026/11/12 11:50:08 WARN Force mode enabled - objects will be deleted immediately without grace period49582026/11/12 11:50:08 INFO Completed upload id=22 objects=5 duration=2.05868ms49592026/11/12 11:50:08 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049602026/11/12 11:50:08 INFO Aborted multipart uploads count=0 kept=049612026/11/12 11:50:08 INFO Received complete push request method=POST path=/49622026/11/12 11:50:08 INFO Vacuumed table table=pending_closures49632026/11/12 11:50:08 WARN Force mode enabled - objects will be deleted immediately without grace period49642026/11/12 11:50:08 INFO Vacuumed table table=commit_receipts49652026/11/12 11:50:08 INFO Completed push id=24 objects=8 duration=1.62305ms49662026/11/12 11:50:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049672026/11/12 11:50:08 INFO Received complete push request method=POST path=/49682026/11/12 11:50:08 INFO Vacuumed table table=pending_objects49692026/11/12 11:50:08 INFO Received complete push request method=POST path=/49702026/11/12 11:50:08 INFO Completed push id=29 objects=10 duration=1.43438ms49712026/11/12 11:50:08 INFO Vacuumed table table=multipart_uploads49722026/11/12 11:50:08 INFO Vacuumed table table=pending_closures49732026/11/12 11:50:08 INFO Completed push id=27 objects=4 duration=1.24362ms49742026/11/12 11:50:08 INFO Vacuumed table table=closures49752026/11/12 11:50:08 INFO Vacuumed table table=commit_receipts49762026/11/12 11:50:08 INFO Received complete push request method=POST path=/49772026/11/12 11:50:08 INFO Vacuumed table table=objects49782026/11/12 11:50:08 INFO Vacuumed table table=pending_objects49792026/11/12 11:50:08 INFO Vacuumed table table=multipart_uploads49802026/11/12 11:50:08 INFO Completed push id=25 objects=10 duration=2.1385ms49812026/11/12 11:50:08 INFO Received complete upload request method=POST path=/49822026/11/12 11:50:08 INFO Vacuumed table table=closures49832026/11/12 11:50:08 INFO Completed upload id=27 objects=26 duration=1.53645ms49842026/11/12 11:50:08 INFO Vacuumed table table=objects49852026/11/12 11:50:08 INFO Received complete push request method=POST path=/49862026/11/12 11:50:08 INFO Completed push id=11 objects=6 duration=2.11724ms49872026/11/12 11:50:08 INFO Received complete push request method=POST path=/49882026/11/12 11:50:08 INFO Received complete upload request method=POST path=/49892026/11/12 11:50:08 INFO Completed upload id=25 objects=9 duration=1.21707ms49902026/11/12 11:50:08 INFO Completed push id=28 objects=3 duration=2.3748ms49912026/11/12 11:50:08 INFO Received push request method=POST path=/49922026/11/12 11:50:08 INFO Received uploads request method=POST path=/49932026/11/12 11:50:08 INFO Received complete upload request method=POST path=/49942026/11/12 11:50:08 INFO Completed upload id=29 objects=5 duration=1.13848ms49952026/11/12 11:50:08 INFO Received complete upload request method=POST path=/49962026/11/12 11:50:08 INFO Received complete push request method=POST path=/49972026/11/12 11:50:08 INFO Completed upload id=30 objects=6 duration=1.14881ms49982026/11/12 11:50:08 INFO Completed push id=28 objects=6 duration=1.34948ms49992026/11/12 11:50:08 INFO Received complete upload request method=POST path=/50002026/11/12 11:50:08 INFO Completed upload id=27 objects=3 duration=1.19208ms50012026/11/12 11:50:08 INFO Received complete push request method=POST path=/50022026/11/12 11:50:08 INFO Completed push id=29 objects=15 duration=1.3686ms50032026/11/12 11:50:08 INFO Received complete push request method=POST path=/50042026/11/12 11:50:08 INFO Received complete push request method=POST path=/50052026/11/12 11:50:08 INFO Completed push id=28 objects=26 duration=1.73758ms50062026/11/12 11:50:08 INFO Completed push id=20 objects=13 duration=1.83446ms50072026/11/12 11:50:08 INFO Received uploads request method=POST path=/50082026/11/12 11:50:08 INFO Received complete upload request method=POST path=/50092026/11/12 11:50:08 INFO Completed upload id=30 objects=2 duration=1.07025ms50102026/11/12 11:50:08 INFO Received complete upload request method=POST path=/50112026/11/12 11:50:08 INFO Completed upload id=31 objects=5 duration=1.34005ms50122026/11/12 11:50:08 INFO Received complete push request method=POST path=/50132026/11/12 11:50:08 INFO Received complete upload request method=POST path=/50142026/11/12 11:50:08 INFO Completed upload id=28 objects=6 duration=1.09563ms50152026/11/12 11:50:08 INFO Completed push id=29 objects=6 duration=1.62781ms50162026/11/12 11:50:08 INFO Received complete push request method=POST path=/50172026/11/12 11:50:08 INFO Aborted multipart uploads count=0 kept=050182026/11/12 11:50:08 INFO Completed push id=30 objects=6 duration=1.27266ms50192026/11/12 11:50:08 INFO Received complete push request method=POST path=/50202026/11/12 11:50:08 INFO Completed push id=30 objects=3 duration=1.20033ms50212026/11/12 11:50:08 INFO Received complete push request method=POST path=/50222026/11/12 11:50:08 INFO Completed push id=22 objects=4 duration=1.37768ms50232026/11/12 11:50:08 INFO Received complete push request method=POST path=/50242026/11/12 11:50:08 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."50252026/11/12 11:50:08 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."50262026/11/12 11:50:08 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."50272026/11/12 11:50:08 INFO Completed push id=31 objects=20 duration=1.849ms50282026/11/12 11:50:08 INFO Received complete upload request method=POST path=/50292026/11/12 11:50:08 INFO Received complete push request method=POST path=/50302026/11/12 11:50:08 INFO Completed upload id=34 objects=3 duration=1.11307ms50312026/11/12 11:50:08 INFO Completed push id=30 objects=4 duration=1.57145ms50322026/11/12 11:50:08 INFO Received complete upload request method=POST path=/50332026/11/12 11:50:08 INFO Completed upload id=31 objects=3 duration=1.16665ms50342026/11/12 11:50:08 INFO Received complete push request method=POST path=/50352026/11/12 11:50:08 INFO Completed push id=32 objects=19 duration=1.2826ms50362026/11/12 11:50:08 INFO Received push request method=POST path=/50372026/11/12 11:50:08 INFO Received push request method=POST path=/50382026/11/12 11:50:08 INFO Received complete upload request method=POST path=/50392026/11/12 11:50:08 INFO Received complete push request method=POST path=/50402026/11/12 11:50:08 INFO Completed upload id=32 objects=5 duration=1.21271ms50412026/11/12 11:50:08 INFO Received complete upload request method=POST path=/50422026/11/12 11:50:08 INFO Received complete upload request method=POST path=/50432026/11/12 11:50:08 INFO Completed push id=31 objects=7 duration=1.12171ms50442026/11/12 11:50:08 INFO Aborted multipart uploads count=0 kept=050452026/11/12 11:50:08 INFO Completed upload id=23 objects=7 duration=1.33285ms50462026/11/12 11:50:08 INFO Completed upload id=18 objects=22 duration=2.03355ms50472026/11/12 11:50:08 WARN Force mode enabled - objects will be deleted immediately without grace period50482026/11/12 11:50:08 INFO Received complete upload request method=POST path=/50492026/11/12 11:50:08 INFO Completed upload id=30 objects=8 duration=1.22013ms50502026/11/12 11:50:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=050512026/11/12 11:50:08 INFO Vacuumed table table=pending_closures50522026/11/12 11:50:08 INFO Vacuumed table table=commit_receipts50532026/11/12 11:50:08 INFO Aborted multipart uploads count=0 kept=050542026/11/12 11:50:08 INFO Vacuumed table table=pending_objects50552026/11/12 11:50:08 INFO Vacuumed table table=multipart_uploads50562026/11/12 11:50:08 INFO Aborted multipart uploads count=0 kept=050572026/11/12 11:50:08 INFO Vacuumed table table=closures50582026/11/12 11:50:08 WARN Force mode enabled - objects will be deleted immediately without grace period50592026/11/12 11:50:08 INFO Vacuumed table table=objects50602026/11/12 11:50:08 WARN Force mode enabled - objects will be deleted immediately without grace period50612026/11/12 11:50:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=050622026/11/12 11:50:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=050632026/11/12 11:50:08 INFO Received complete upload request method=POST path=/50642026/11/12 11:50:08 INFO Vacuumed table table=pending_closures50652026/11/12 11:50:08 INFO Received uploads request method=POST path=/50662026/11/12 11:50:08 INFO Received complete upload request method=POST path=/50672026/11/12 11:50:08 INFO Completed upload id=32 objects=5 duration=1.02495ms50682026/11/12 11:50:08 INFO Vacuumed table table=commit_receipts50692026/11/12 11:50:08 INFO Vacuumed table table=pending_closures50702026/11/12 11:50:08 INFO Completed upload id=33 objects=3 duration=1.10565ms50712026/11/12 11:50:08 INFO Vacuumed table table=commit_receipts50722026/11/12 11:50:08 INFO Vacuumed table table=pending_objects50732026/11/12 11:50:08 INFO Received complete push request method=POST path=/50742026/11/12 11:50:08 INFO Received complete upload request method=POST path=/50752026/11/12 11:50:08 INFO Vacuumed table table=multipart_uploads50762026/11/12 11:50:08 INFO Vacuumed table table=pending_objects50772026/11/12 11:50:08 INFO Vacuumed table table=closures50782026/11/12 11:50:08 INFO Vacuumed table table=multipart_uploads50792026/11/12 11:50:08 INFO Completed push id=24 objects=5 duration=1.43979ms50802026/11/12 11:50:08 INFO Completed upload id=21 objects=8 duration=1.39037ms50812026/11/12 11:50:08 INFO Vacuumed table table=objects50822026/11/12 11:50:08 INFO Vacuumed table table=closures50832026/11/12 11:50:08 INFO Vacuumed table table=objects50842026/11/12 11:50:08 INFO Received complete upload request method=POST path=/50852026/11/12 11:50:08 INFO Received push request method=POST path=/50862026/11/12 11:50:08 INFO Completed upload id=31 objects=6 duration=1.26645ms50872026/11/12 11:50:08 INFO Aborted multipart uploads count=0 kept=050882026/11/12 11:50:08 INFO Aborted multipart uploads count=0 kept=050892026/11/12 11:50:08 INFO Received complete upload request method=POST path=/50902026/11/12 11:50:08 WARN Force mode enabled - objects will be deleted immediately without grace period50912026/11/12 11:50:08 WARN Force mode enabled - objects will be deleted immediately without grace period50922026/11/12 11:50:08 INFO Completed upload id=25 objects=2 duration=1.08011ms50932026/11/12 11:50:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=050942026/11/12 11:50:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=050952026/11/12 11:50:08 INFO Vacuumed table table=pending_closures50962026/11/12 11:50:08 INFO Vacuumed table table=pending_closures50972026/11/12 11:50:08 INFO Received complete upload request method=POST path=/50982026/11/12 11:50:08 INFO Vacuumed table table=commit_receipts50992026/11/12 11:50:08 INFO Vacuumed table table=commit_receipts51002026/11/12 11:50:08 INFO Completed upload id=22 objects=7 duration=1.43424ms51012026/11/12 11:50:08 INFO Vacuumed table table=pending_objects51022026/11/12 11:50:08 INFO Vacuumed table table=multipart_uploads51032026/11/12 11:50:08 INFO Received complete push request method=POST path=/51042026/11/12 11:50:08 INFO Vacuumed table table=pending_objects51052026/11/12 11:50:08 INFO Vacuumed table table=closures51062026/11/12 11:50:08 INFO Vacuumed table table=multipart_uploads51072026/11/12 11:50:08 INFO Vacuumed table table=objects51082026/11/12 11:50:08 INFO Received complete upload request method=POST path=/51092026/11/12 11:50:08 INFO Vacuumed table table=closures51102026/11/12 11:50:08 INFO Completed upload id=32 objects=4 duration=1.18828ms51112026/11/12 11:50:08 INFO Vacuumed table table=objects51122026/11/12 11:50:08 INFO Completed push id=28 objects=11 duration=3.02953ms51132026/11/12 11:50:08 INFO Received complete upload request method=POST path=/51142026/11/12 11:50:08 INFO Completed upload id=26 objects=20 duration=1.1797ms51152026/11/12 11:50:08 INFO Received complete push request method=POST path=/51162026/11/12 11:50:08 INFO Completed push id=24 objects=5 duration=1.15798ms51172026/11/12 11:50:08 INFO Received complete upload request method=POST path=/51182026/11/12 11:50:08 INFO Completed upload id=34 objects=24 duration=1.3981ms51192026/11/12 11:50:08 INFO Received complete upload request method=POST path=/51202026/11/12 11:50:08 INFO Completed upload id=33 objects=10 duration=1.2129ms51212026/11/12 11:50:08 INFO Received push request method=POST path=/51222026/11/12 11:50:08 INFO Received complete push request method=POST path=/51232026/11/12 11:50:08 INFO Completed push id=28 objects=7 duration=1.29964ms51242026/11/12 11:50:08 INFO Received complete upload request method=POST path=/51252026/11/12 11:50:08 INFO Completed upload id=25 objects=8 duration=1.08725ms51262026/11/12 11:50:08 INFO Aborted multipart uploads count=0 kept=051272026/11/12 11:50:08 WARN Force mode enabled - objects will be deleted immediately without grace period51282026/11/12 11:50:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=051292026/11/12 11:50:08 INFO Received complete push request method=POST path=/51302026/11/12 11:50:08 INFO Vacuumed table table=pending_closures51312026/11/12 11:50:08 INFO Completed push id=29 objects=24 duration=1.89233ms51322026/11/12 11:50:08 INFO Vacuumed table table=commit_receipts51332026/11/12 11:50:08 INFO Vacuumed table table=pending_objects51342026/11/12 11:50:08 INFO Vacuumed table table=multipart_uploads51352026/11/12 11:50:08 INFO Vacuumed table table=closures51362026/11/12 11:50:08 INFO Vacuumed table table=objects51372026/11/12 11:50:08 INFO Received complete push request method=POST path=/51382026/11/12 11:50:08 INFO Completed push id=26 objects=4 duration=1.22536ms51392026/11/12 11:50:08 INFO Received complete upload request method=POST path=/51402026/11/12 11:50:08 INFO Completed upload id=34 objects=23 duration=1.23605ms51412026/11/12 11:50:08 INFO Received complete upload request method=POST path=/51422026/11/12 11:50:08 INFO Received complete upload request method=POST path=/51432026/11/12 11:50:08 INFO Completed upload id=30 objects=6 duration=1.10265ms51442026/11/12 11:50:08 INFO Completed upload id=23 objects=7 duration=3.06084ms51452026/11/12 11:50:08 INFO Aborted multipart uploads count=0 kept=051462026/11/12 11:50:08 WARN Force mode enabled - objects will be deleted immediately without grace period51472026/11/12 11:50:08 INFO Received complete push request method=POST path=/51482026/11/12 11:50:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=051492026/11/12 11:50:08 INFO Completed push id=30 objects=2 duration=1.68312ms51502026/11/12 11:50:09 INFO Vacuumed table table=pending_closures51512026/11/12 11:50:09 INFO Vacuumed table table=commit_receipts51522026/11/12 11:50:09 INFO Received complete push request method=POST path=/51532026/11/12 11:50:09 INFO Completed push id=31 objects=5 duration=1.18143ms51542026/11/12 11:50:09 INFO Vacuumed table table=pending_objects51552026/11/12 11:50:09 INFO Vacuumed table table=multipart_uploads51562026/11/12 11:50:09 INFO Received complete upload request method=POST path=/51572026/11/12 11:50:09 INFO Vacuumed table table=closures51582026/11/12 11:50:09 INFO Completed upload id=26 objects=5 duration=1.18132ms51592026/11/12 11:50:09 INFO Vacuumed table table=objects51602026/11/12 11:50:09 INFO Received complete push request method=POST path=/51612026/11/12 11:50:09 INFO Completed push id=35 objects=6 duration=1.50611ms51622026/11/12 11:50:09 INFO Received complete push request method=POST path=/51632026/11/12 11:50:09 INFO Completed push id=27 objects=3 duration=1.86783ms51642026/11/12 11:50:09 INFO Received complete push request method=POST path=/51652026/11/12 11:50:09 INFO Aborted multipart uploads count=0 kept=051662026/11/12 11:50:09 INFO Completed push id=32 objects=7 duration=1.4221ms51672026/11/12 11:50:09 WARN Force mode enabled - objects will be deleted immediately without grace period51682026/11/12 11:50:09 INFO Received complete push request method=POST path=/51692026/11/12 11:50:09 INFO Completed push id=32 objects=26 duration=1.62863ms51702026/11/12 11:50:09 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=051712026/11/12 11:50:09 INFO Vacuumed table table=pending_closures51722026/11/12 11:50:09 INFO Vacuumed table table=commit_receipts51732026/11/12 11:50:09 INFO Vacuumed table table=pending_objects51742026/11/12 11:50:09 INFO Vacuumed table table=multipart_uploads51752026/11/12 11:50:09 INFO Received complete upload request method=POST path=/51762026/11/12 11:50:09 INFO Vacuumed table table=closures51772026/11/12 11:50:09 INFO Completed upload id=28 objects=5 duration=1.07829ms51782026/11/12 11:50:09 INFO Vacuumed table table=objects51792026/11/12 11:50:09 INFO Received complete upload request method=POST path=/51802026/11/12 11:50:09 INFO Received complete upload request method=POST path=/51812026/11/12 11:50:09 INFO Completed upload id=33 objects=6 duration=1.11523ms51822026/11/12 11:50:09 INFO Completed upload id=33 objects=2 duration=1.19388ms51832026/11/12 11:50:09 INFO Received complete push request method=POST path=/51842026/11/12 11:50:09 INFO Completed push id=29 objects=17 duration=1.44102ms51852026/11/12 11:50:09 INFO Received complete upload request method=POST path=/51862026/11/12 11:50:09 INFO Received complete upload request method=POST path=/51872026/11/12 11:50:09 INFO Completed upload id=34 objects=5 duration=950.87µs51882026/11/12 11:50:09 INFO Completed upload id=34 objects=3 duration=1.33199ms51892026/11/12 11:50:09 INFO Aborted multipart uploads count=0 kept=051902026/11/12 11:50:09 WARN Force mode enabled - objects will be deleted immediately without grace period51912026/11/12 11:50:09 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=051922026/11/12 11:50:09 INFO Vacuumed table table=pending_closures51932026/11/12 11:50:09 INFO Vacuumed table table=commit_receipts51942026/11/12 11:50:09 INFO Vacuumed table table=pending_objects51952026/11/12 11:50:09 INFO Vacuumed table table=multipart_uploads51962026/11/12 11:50:09 INFO Vacuumed table table=closures51972026/11/12 11:50:09 INFO Received complete upload request method=POST path=/51982026/11/12 11:50:09 INFO Vacuumed table table=objects51992026/11/12 11:50:09 INFO Aborted multipart uploads count=0 kept=052002026/11/12 11:50:09 INFO Completed upload id=35 objects=11 duration=1.1762ms52012026/11/12 11:50:09 WARN Force mode enabled - objects will be deleted immediately without grace period52022026/11/12 11:50:09 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=052032026/11/12 11:50:09 INFO Vacuumed table table=pending_closures52042026/11/12 11:50:09 INFO Vacuumed table table=commit_receipts52052026/11/12 11:50:09 INFO Vacuumed table table=pending_objects52062026/11/12 11:50:09 INFO Vacuumed table table=multipart_uploads52072026/11/12 11:50:09 INFO Vacuumed table table=closures52082026/11/12 11:50:09 INFO Vacuumed table table=objects52092026/11/12 11:50:09 INFO Received complete upload request method=POST path=/52102026/11/12 11:50:09 INFO Completed upload id=36 objects=3 duration=1.13497ms52112026/11/12 11:50:09 INFO Aborted multipart uploads count=0 kept=052122026/11/12 11:50:09 WARN Force mode enabled - objects will be deleted immediately without grace period52132026/11/12 11:50:09 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=052142026/11/12 11:50:09 INFO Vacuumed table table=pending_closures52152026/11/12 11:50:09 INFO Vacuumed table table=commit_receipts52162026/11/12 11:50:09 INFO Vacuumed table table=pending_objects52172026/11/12 11:50:09 INFO Vacuumed table table=multipart_uploads52182026/11/12 11:50:09 INFO Vacuumed table table=closures52192026/11/12 11:50:09 INFO Vacuumed table table=objects5220--- PASS: FuzzPushGC (0.00s)5221 --- PASS: FuzzPushGC/seed#14 (2.80s)5222 --- PASS: FuzzPushGC/seed#0 (3.19s)5223 --- PASS: FuzzPushGC/seed#8 (3.27s)5224 --- PASS: FuzzPushGC/seed#4 (3.31s)5225 --- PASS: FuzzPushGC/seed#3 (3.37s)5226 --- PASS: FuzzPushGC/seed#5 (3.41s)5227 --- PASS: FuzzPushGC/seed#13 (3.51s)5228 --- PASS: FuzzPushGC/seed#15 (3.51s)5229 --- PASS: FuzzPushGC/seed#7 (3.52s)5230 --- PASS: FuzzPushGC/seed#6 (3.54s)5231 --- PASS: FuzzPushGC/seed#9 (3.58s)5232 --- PASS: FuzzPushGC/seed#2 (3.61s)5233 --- PASS: FuzzPushGC/seed#12 (3.65s)5234 --- PASS: FuzzPushGC/seed#1 (3.70s)5235 --- PASS: FuzzPushGC/seed#10 (3.72s)5236 --- PASS: FuzzPushGC/seed#11 (3.76s)5237PASS5238{"timestamp":"2026-11-12T11:50:09.176578305Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:50010","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(396)"}5239{"timestamp":"2026-11-12T11:50:09.176912895Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:49462","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(401)"}52402026-11-12 11:50:09.570 UTC [129] LOG: received smart shutdown request52412026-11-12 11:50:09.575 UTC [129] LOG: background worker "logical replication launcher" (PID 139) exited with exit code 152422026-11-12 11:50:10.437 UTC [134] LOG: checkpoint complete: wrote 3352 buffers (20.5%), wrote 5 SLRU buffers; 0 WAL file(s) added, 0 removed, 33 recycled; write=15.874 s, sync=0.722 s, total=16.667 s; sync files=43130, longest=0.014 s, average=0.001 s; distance=535376 kB, estimate=535376 kB; lsn=0/275CAB30, redo lsn=0/22239F5052432026-11-12 11:50:10.440 UTC [134] LOG: shutting down52442026-11-12 11:50:10.441 UTC [134] LOG: checkpoint starting: shutdown immediate52452026-11-12 11:50:11.733 UTC [134] LOG: checkpoint complete: wrote 9995 buffers (61.0%), wrote 2 SLRU buffers; 0 WAL file(s) added, 0 removed, 5 recycled; write=0.186 s, sync=1.085 s, total=1.294 s; sync files=3207, longest=0.002 s, average=0.001 s; distance=85571 kB, estimate=490396 kB; lsn=0/275CABA8, redo lsn=0/275CABA852462026-11-12 11:50:11.830 UTC [129] LOG: database system is shut down5247Running OIDC tests...5248=== RUN TestAudienceForIssuer5249=== PAUSE TestAudienceForIssuer5250=== RUN TestProviderClientGivesUpOnSilentIssuer5251=== PAUSE TestProviderClientGivesUpOnSilentIssuer5252=== RUN TestGlobMatch5253=== PAUSE TestGlobMatch5254=== RUN TestValidateToken_ValidToken5255=== PAUSE TestValidateToken_ValidToken5256=== RUN TestValidateToken_WrongAudience5257=== PAUSE TestValidateToken_WrongAudience5258=== RUN TestValidateToken_Expired5259=== PAUSE TestValidateToken_Expired5260=== RUN TestValidateToken_BoundClaimsMismatch5261=== PAUSE TestValidateToken_BoundClaimsMismatch5262=== RUN TestValidateToken_BoundSubjectMismatch5263=== PAUSE TestValidateToken_BoundSubjectMismatch5264=== RUN TestValidateToken_MultipleProviders5265=== PAUSE TestValidateToken_MultipleProviders5266=== RUN TestValidateToken_UnknownIssuerFetchesNoKeys5267=== PAUSE TestValidateToken_UnknownIssuerFetchesNoKeys5268=== RUN TestValidateToken_GoogleSchemelessIssuer5269=== PAUSE TestValidateToken_GoogleSchemelessIssuer5270=== RUN TestValidateToken_KubernetesServiceAccount5271=== PAUSE TestValidateToken_KubernetesServiceAccount5272=== RUN TestNewValidator_KubernetesRequiresCA5273=== PAUSE TestNewValidator_KubernetesRequiresCA5274=== RUN TestValidateToken_KubernetesIssuerFromOwnToken5275=== PAUSE TestValidateToken_KubernetesIssuerFromOwnToken5276=== RUN TestPins_ReservedForMatchingRule5277=== PAUSE TestPins_ReservedForMatchingRule5278=== RUN TestPins_TopLevelShorthand5279=== PAUSE TestPins_TopLevelShorthand5280=== RUN TestPins_ConfigValidation5281=== PAUSE TestPins_ConfigValidation5282=== RUN TestScopes_LegacyProviderDefaultsToWrite5283=== PAUSE TestScopes_LegacyProviderDefaultsToWrite5284=== RUN TestScopes_Rules5285=== PAUSE TestScopes_Rules5286=== RUN TestScopes_ConfigValidation5287=== PAUSE TestScopes_ConfigValidation5288=== CONT TestAudienceForIssuer5289=== CONT TestValidateToken_GoogleSchemelessIssuer5290--- PASS: TestAudienceForIssuer (0.00s)5291=== CONT TestValidateToken_UnknownIssuerFetchesNoKeys5292=== CONT TestValidateToken_MultipleProviders5293=== CONT TestValidateToken_BoundSubjectMismatch5294=== CONT TestValidateToken_BoundClaimsMismatch5295=== CONT TestValidateToken_Expired52962026/11/12 11:50:14 INFO OIDC provider initialized name=known issuer=https://known.example5297=== CONT TestValidateToken_WrongAudience5298=== CONT TestValidateToken_ValidToken5299=== CONT TestGlobMatch5300=== CONT TestProviderClientGivesUpOnSilentIssuer5301=== CONT TestScopes_LegacyProviderDefaultsToWrite5302=== CONT TestPins_ConfigValidation5303--- PASS: TestProviderClientGivesUpOnSilentIssuer (0.00s)5304=== CONT TestValidateToken_KubernetesIssuerFromOwnToken5305=== CONT TestPins_ReservedForMatchingRule5306=== CONT TestScopes_ConfigValidation5307--- PASS: TestPins_ConfigValidation (0.00s)5308=== CONT TestNewValidator_KubernetesRequiresCA5309=== CONT TestValidateToken_KubernetesServiceAccount5310=== CONT TestPins_TopLevelShorthand5311=== CONT TestScopes_Rules5312=== RUN TestGlobMatch/foo_foo5313=== PAUSE TestGlobMatch/foo_foo5314=== RUN TestGlobMatch/foo_bar5315=== PAUSE TestGlobMatch/foo_bar5316=== RUN TestGlobMatch/*_5317=== PAUSE TestGlobMatch/*_5318=== RUN TestGlobMatch/*_anything5319=== PAUSE TestGlobMatch/*_anything5320=== RUN TestGlobMatch/foo*_foo5321=== PAUSE TestGlobMatch/foo*_foo5322=== RUN TestGlobMatch/foo*_foobar5323=== PAUSE TestGlobMatch/foo*_foobar5324=== RUN TestGlobMatch/foo*_bar5325=== PAUSE TestGlobMatch/foo*_bar5326=== RUN TestGlobMatch/*bar_bar5327=== PAUSE TestGlobMatch/*bar_bar5328=== RUN TestGlobMatch/*bar_foobar5329=== PAUSE TestGlobMatch/*bar_foobar5330=== RUN TestGlobMatch/*bar_foo5331=== PAUSE TestGlobMatch/*bar_foo5332=== RUN TestGlobMatch/foo*bar_foobar5333=== PAUSE TestGlobMatch/foo*bar_foobar5334--- PASS: TestScopes_ConfigValidation (0.00s)5335=== RUN TestGlobMatch/foo*bar_foo123bar5336=== PAUSE TestGlobMatch/foo*bar_foo123bar5337=== RUN TestGlobMatch/foo*bar_foobarbaz5338=== PAUSE TestGlobMatch/foo*bar_foobarbaz5339=== RUN TestGlobMatch/*/*_foo/bar5340=== PAUSE TestGlobMatch/*/*_foo/bar5341=== RUN TestGlobMatch/*/*_foo5342=== PAUSE TestGlobMatch/*/*_foo5343=== RUN TestGlobMatch/refs/heads/*_refs/heads/main5344=== PAUSE TestGlobMatch/refs/heads/*_refs/heads/main5345=== RUN TestGlobMatch/refs/heads/*_refs/tags/v1.05346=== PAUSE TestGlobMatch/refs/heads/*_refs/tags/v1.05347=== RUN TestGlobMatch/refs/*/main_refs/heads/main5348=== PAUSE TestGlobMatch/refs/*/main_refs/heads/main5349=== RUN TestGlobMatch/fo?_foo5350=== PAUSE TestGlobMatch/fo?_foo5351=== RUN TestGlobMatch/fo?_fo5352=== PAUSE TestGlobMatch/fo?_fo5353=== RUN TestGlobMatch/fo?_fooo5354=== PAUSE TestGlobMatch/fo?_fooo5355=== RUN TestGlobMatch/?oo_foo5356=== PAUSE TestGlobMatch/?oo_foo5357=== RUN TestGlobMatch/?oo_boo5358=== PAUSE TestGlobMatch/?oo_boo5359=== RUN TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5360=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5361=== RUN TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5362=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5363=== CONT TestGlobMatch/foo_foo5364=== CONT TestGlobMatch/foo*bar_foobarbaz5365=== CONT TestGlobMatch/foo*bar_foobar5366=== CONT TestGlobMatch/foo*_foobar5367=== CONT TestGlobMatch/?oo_boo5368=== CONT TestGlobMatch/foo*_bar5369=== CONT TestGlobMatch/refs/heads/*_refs/heads/main5370=== CONT TestGlobMatch/*_anything5371=== CONT TestGlobMatch/refs/*/main_refs/heads/main5372=== CONT TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5373=== CONT TestGlobMatch/fo?_fooo5374=== CONT TestGlobMatch/*_5375=== CONT TestGlobMatch/?oo_foo5376=== CONT TestGlobMatch/foo_bar5377=== CONT TestGlobMatch/foo*bar_foo123bar5378=== CONT TestGlobMatch/*bar_foobar5379=== CONT TestGlobMatch/refs/heads/*_refs/tags/v1.05380=== CONT TestGlobMatch/*bar_bar5381=== CONT TestGlobMatch/*bar_foo5382=== CONT TestGlobMatch/fo?_foo5383=== CONT TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5384=== CONT TestGlobMatch/foo*_foo5385=== CONT TestGlobMatch/fo?_fo5386=== CONT TestGlobMatch/*/*_foo5387=== CONT TestGlobMatch/*/*_foo/bar5388--- PASS: TestGlobMatch (0.01s)5389 --- PASS: TestGlobMatch/foo_foo (0.00s)5390 --- PASS: TestGlobMatch/foo*bar_foobarbaz (0.00s)5391 --- PASS: TestGlobMatch/foo*bar_foobar (0.00s)5392 --- PASS: TestGlobMatch/foo*_foobar (0.00s)5393 --- PASS: TestGlobMatch/?oo_boo (0.00s)5394 --- PASS: TestGlobMatch/foo*_bar (0.00s)5395 --- PASS: TestGlobMatch/refs/heads/*_refs/heads/main (0.00s)5396 --- PASS: TestGlobMatch/*_anything (0.00s)5397 --- PASS: TestGlobMatch/refs/*/main_refs/heads/main (0.00s)5398 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main (0.00s)5399 --- PASS: TestGlobMatch/fo?_fooo (0.00s)5400 --- PASS: TestGlobMatch/*_ (0.00s)5401 --- PASS: TestGlobMatch/?oo_foo (0.00s)5402 --- PASS: TestGlobMatch/foo_bar (0.00s)5403 --- PASS: TestGlobMatch/foo*bar_foo123bar (0.00s)5404 --- PASS: TestGlobMatch/*bar_foobar (0.00s)5405 --- PASS: TestGlobMatch/refs/heads/*_refs/tags/v1.0 (0.00s)5406 --- PASS: TestGlobMatch/*bar_bar (0.00s)5407 --- PASS: TestGlobMatch/*bar_foo (0.00s)5408 --- PASS: TestGlobMatch/fo?_foo (0.00s)5409 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main (0.00s)5410 --- PASS: TestGlobMatch/foo*_foo (0.00s)5411 --- PASS: TestGlobMatch/fo?_fo (0.00s)5412 --- PASS: TestGlobMatch/*/*_foo (0.00s)5413 --- PASS: TestGlobMatch/*/*_foo/bar (0.00s)54142026/11/12 11:50:14 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:34155/oidc5415--- PASS: TestValidateToken_ValidToken (0.01s)54162026/11/12 11:50:14 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:40811/oidc5417--- PASS: TestValidateToken_BoundSubjectMismatch (0.02s)54182026/11/12 11:50:14 INFO OIDC provider initialized name=kubernetes issuer=https://oidc.eks.invalid/id/ABC12354192026/11/12 11:50:14 INFO OIDC provider initialized name=google issuer=https://accounts.google.com5420--- PASS: TestValidateToken_KubernetesIssuerFromOwnToken (0.03s)5421--- PASS: TestValidateToken_GoogleSchemelessIssuer (0.04s)54222026/11/12 11:50:14 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:39139/oidc5423--- PASS: TestValidateToken_BoundClaimsMismatch (0.04s)54242026/11/12 11:50:14 INFO OIDC provider initialized name=kubernetes issuer=https://127.0.0.1:432955425--- PASS: TestValidateToken_UnknownIssuerFetchesNoKeys (0.05s)54262026/11/12 11:50:14 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:39581/oidc5427--- PASS: TestValidateToken_Expired (0.05s)5428--- PASS: TestValidateToken_KubernetesServiceAccount (0.04s)54292026/11/12 11:50:14 http: TLS handshake error from 127.0.0.1:39028: remote error: tls: bad certificate5430--- PASS: TestNewValidator_KubernetesRequiresCA (0.05s)54312026/11/12 11:50:14 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:41083/oidc54322026/11/12 11:50:14 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:37821/oidc5433--- PASS: TestPins_TopLevelShorthand (0.06s)5434--- PASS: TestValidateToken_WrongAudience (0.06s)54352026/11/12 11:50:14 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:35159/oidc5436--- PASS: TestPins_ReservedForMatchingRule (0.07s)54372026/11/12 11:50:14 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:39333/oidc5438--- PASS: TestScopes_LegacyProviderDefaultsToWrite (0.11s)54392026/11/12 11:50:14 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:41031/oidc5440--- PASS: TestScopes_Rules (0.12s)54412026/11/12 11:50:14 INFO OIDC provider initialized name=provider2 issuer=http://127.0.0.1:42171/oidc54422026/11/12 11:50:14 INFO OIDC provider initialized name=provider1 issuer=http://127.0.0.1:34435/oidc5443--- PASS: TestValidateToken_MultipleProviders (0.13s)5444PASS5445Running hook tests...5446=== RUN TestSendPathsEmpty5447=== PAUSE TestSendPathsEmpty5448=== RUN TestQueueEnqueueAndFetch5449=== PAUSE TestQueueEnqueueAndFetch5450=== RUN TestQueueDeduplication5451=== PAUSE TestQueueDeduplication5452=== RUN TestQueueRemove5453=== PAUSE TestQueueRemove5454=== RUN TestQueueFetchBatchLimit5455=== PAUSE TestQueueFetchBatchLimit5456=== RUN TestQueueRetryMovesToBack5457=== PAUSE TestQueueRetryMovesToBack5458=== RUN TestQueueFetchRemoveLifecycle5459=== PAUSE TestQueueFetchRemoveLifecycle5460=== RUN TestQueueConcurrentWriters5461=== PAUSE TestQueueConcurrentWriters5462=== RUN TestQueueRemoveLargeClosure5463=== PAUSE TestQueueRemoveLargeClosure5464=== RUN TestServerClientIntegration5465--- PASS: TestServerClientIntegration (0.00s)5466=== RUN TestServerQueueError54672026/11/12 11:50:14 ERROR Failed to queue paths error="permission denied" count=15468--- PASS: TestServerQueueError (0.00s)5469=== RUN TestGetListenerSocketActivation5470 server_test.go:220: === RUN TestGetListenerSocketActivation5471 --- PASS: TestGetListenerSocketActivation (0.00s)5472 PASS5473 5474--- PASS: TestGetListenerSocketActivation (0.01s)5475=== RUN TestServerStalledClientDoesNotBlockShutdown5476=== PAUSE TestServerStalledClientDoesNotBlockShutdown5477=== RUN TestServerBacksOffOnAcceptErrors5478=== PAUSE TestServerBacksOffOnAcceptErrors5479=== RUN TestDrainIsolatesPoisonPath5480=== PAUSE TestDrainIsolatesPoisonPath5481=== RUN TestRunNotBlockedByPoisonHead5482=== PAUSE TestRunNotBlockedByPoisonHead5483=== RUN TestDrainGivesUpWhenServerDown5484=== PAUSE TestDrainGivesUpWhenServerDown5485=== RUN TestFailedPathPrunedByLaterClosure5486=== PAUSE TestFailedPathPrunedByLaterClosure5487=== RUN TestWorkerUploadsAndRemoves5488=== PAUSE TestWorkerUploadsAndRemoves5489=== RUN TestWorkerSkipsGCdPaths5490=== PAUSE TestWorkerSkipsGCdPaths5491=== RUN TestWorkerPrunesClosureDeps5492=== PAUSE TestWorkerPrunesClosureDeps5493=== RUN TestDrainTimeout5494=== PAUSE TestDrainTimeout5495=== RUN TestWorkerRemoveFailureIsNotProgress5496=== PAUSE TestWorkerRemoveFailureIsNotProgress5497=== RUN TestWorkerKeepsPathItCannotStat5498=== PAUSE TestWorkerKeepsPathItCannotStat5499=== CONT TestSendPathsEmpty5500=== CONT TestDrainGivesUpWhenServerDown5501--- PASS: TestSendPathsEmpty (0.00s)5502=== CONT TestQueueFetchRemoveLifecycle5503=== CONT TestQueueConcurrentWriters5504=== CONT TestQueueRetryMovesToBack5505=== CONT TestQueueFetchBatchLimit5506=== CONT TestRunNotBlockedByPoisonHead5507=== CONT TestQueueRemove5508=== CONT TestDrainIsolatesPoisonPath5509=== CONT TestQueueDeduplication5510=== CONT TestServerBacksOffOnAcceptErrors5511=== CONT TestQueueEnqueueAndFetch55122000/01/01 00:00:00 ERROR Accept failed error="too many open files"5513=== CONT TestWorkerPrunesClosureDeps55142000/01/01 00:00:00 ERROR Accept failed error="too many open files"55152000/01/01 00:00:00 ERROR Accept failed error="too many open files"55162000/01/01 00:00:00 ERROR Accept failed error="too many open files"55172000/01/01 00:00:00 ERROR Accept failed error="too many open files"55182000/01/01 00:00:00 ERROR Accept failed error="too many open files"55192000/01/01 00:00:00 ERROR Accept failed error="too many open files"55202000/01/01 00:00:00 ERROR Accept failed error="too many open files"5521--- PASS: TestServerBacksOffOnAcceptErrors (0.00s)5522=== CONT TestWorkerKeepsPathItCannotStat5523=== CONT TestQueueRemoveLargeClosure5524=== CONT TestWorkerRemoveFailureIsNotProgress5525=== CONT TestWorkerUploadsAndRemoves5526=== CONT TestServerStalledClientDoesNotBlockShutdown5527=== CONT TestDrainTimeout5528=== CONT TestWorkerSkipsGCdPaths5529=== CONT TestFailedPathPrunedByLaterClosure55302026/11/12 11:50:14 INFO Uploading batch count=455312026/11/12 11:50:14 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat282228200/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat282228200/002/locked/aaa: permission denied"55322026/11/12 11:50:14 INFO Upload queue status pending=255332026/11/12 11:50:14 INFO Upload queue status pending=255342026/11/12 11:50:14 INFO Uploading batch count=255352026/11/12 11:50:14 ERROR Upload failed error="upload failed" count=255362026/11/12 11:50:14 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown1479471151/002/a55372026/11/12 11:50:14 INFO Upload queue status pending=255382026/11/12 11:50:14 INFO Upload queue status pending=355392000/01/01 00:00:00 INFO Uploading batch count=255402026/11/12 11:50:14 ERROR Upload failed error="upload failed" count=455412026/11/12 11:50:14 INFO Uploading batch count=155422026/11/12 11:50:14 ERROR Upload failed error="upload failed" count=155432026/11/12 11:50:14 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat282228200/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat282228200/002/locked/aaa: permission denied"55442026/11/12 11:50:14 INFO Uploading batch count=15545--- PASS: TestQueueFetchBatchLimit (0.02s)55462026/11/12 11:50:14 ERROR Upload failed error="upload failed" count=155472026/11/12 11:50:14 INFO Uploading batch count=15548--- PASS: TestQueueEnqueueAndFetch (0.02s)55492000/01/01 00:00:00 ERROR Upload failed error="context deadline exceeded" count=255502026/11/12 11:50:14 INFO Uploading batch count=255512026/11/12 11:50:14 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerSkipsGCdPaths1470255858/002/nonexistent55522026/11/12 11:50:14 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainIsolatesPoisonPath2353786292/002/bbb55532000/01/01 00:00:00 ERROR Drain finished with paths left in queue remaining=455542026/11/12 11:50:14 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown1479471151/002/b55552026/11/12 11:50:14 INFO Uploading batch count=155562026/11/12 11:50:14 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat282228200/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat282228200/002/locked/aaa: permission denied"55572026/11/12 11:50:14 INFO Uploading batch count=155582026/11/12 11:50:14 ERROR Drain finished with paths left in queue remaining=15559--- PASS: TestQueueRemove (0.02s)5560--- PASS: TestQueueFetchRemoveLifecycle (0.02s)55612026/11/12 11:50:14 INFO Uploading batch count=15562--- PASS: TestQueueDeduplication (0.02s)55632026/11/12 11:50:14 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress1318368781/002/nonexistent55642026/11/12 11:50:14 INFO Uploading batch count=255652026/11/12 11:50:14 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=155662026/11/12 11:50:14 ERROR Upload failed error="upload failed" count=255672026/11/12 11:50:14 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress1318368781/002/nonexistent55682026/11/12 11:50:14 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown1479471151/002/c55692026/11/12 11:50:14 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=15570--- PASS: TestQueueRetryMovesToBack (0.02s)55712026/11/12 11:50:14 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress1318368781/002/nonexistent5572--- PASS: TestDrainTimeout (0.02s)55732026/11/12 11:50:14 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=155742026/11/12 11:50:14 INFO Uploading batch count=155752026/11/12 11:50:14 ERROR Upload failed error="upload failed" count=155762026/11/12 11:50:14 ERROR Drain finished with paths left in queue remaining=155772026/11/12 11:50:14 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown1479471151/002/d55782026/11/12 11:50:14 INFO Uploading batch count=155792026/11/12 11:50:14 ERROR Upload failed error="upload failed" count=15580--- PASS: TestWorkerKeepsPathItCannotStat (0.02s)55812026/11/12 11:50:14 INFO Uploading batch count=155822026/11/12 11:50:14 ERROR Upload failed error="upload failed" count=155832026/11/12 11:50:14 INFO Uploading batch count=255842026/11/12 11:50:14 ERROR Upload failed error="upload failed" count=255852026/11/12 11:50:14 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown1479471151/002/e5586--- PASS: TestWorkerRemoveFailureIsNotProgress (0.02s)55872026/11/12 11:50:14 ERROR Drain finished with paths left in queue remaining=15588--- PASS: TestFailedPathPrunedByLaterClosure (0.02s)55892026/11/12 11:50:14 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown1479471151/002/f55902026/11/12 11:50:14 ERROR Drain finished with paths left in queue remaining=105591--- PASS: TestDrainIsolatesPoisonPath (0.03s)5592--- PASS: TestDrainGivesUpWhenServerDown (0.03s)5593--- PASS: TestWorkerSkipsGCdPaths (0.03s)5594--- PASS: TestWorkerUploadsAndRemoves (0.03s)5595--- PASS: TestWorkerPrunesClosureDeps (0.04s)5596--- PASS: TestQueueRemoveLargeClosure (0.06s)55972026/11/12 11:50:14 ERROR Failed to decode request error="read unix /build/hook3303574952/test.sock->@: i/o timeout"55982026/11/12 11:50:14 ERROR Failed to write response error="write unix /build/hook3303574952/test.sock->@: i/o timeout"5599--- PASS: TestServerStalledClientDoesNotBlockShutdown (0.20s)5600--- PASS: TestQueueConcurrentWriters (0.24s)56012026/11/12 11:50:15 INFO Uploading batch count=156022026/11/12 11:50:15 INFO Uploading batch count=156032026/11/12 11:50:15 INFO Uploading batch count=156042026/11/12 11:50:15 ERROR Upload failed error="upload failed" count=156052026/11/12 11:50:15 INFO Uploading batch count=156062026/11/12 11:50:15 ERROR Upload failed error="upload failed" count=156072026/11/12 11:50:15 INFO Uploading batch count=156082026/11/12 11:50:15 ERROR Upload failed error="upload failed" count=156092026/11/12 11:50:15 INFO Uploading batch count=156102026/11/12 11:50:15 ERROR Upload failed error="upload failed" count=156112026/11/12 11:50:15 ERROR Drain finished with paths left in queue remaining=15612--- PASS: TestRunNotBlockedByPoisonHead (1.04s)5613PASS5614Running signing tests...5615=== RUN TestExternalSigner5616=== PAUSE TestExternalSigner5617=== RUN TestExternalSignerWrongPublicKey5618=== PAUSE TestExternalSignerWrongPublicKey5619=== RUN TestExternalSignerErrors5620=== PAUSE TestExternalSignerErrors5621=== RUN TestExternalSignerRestartsAfterCrash5622=== PAUSE TestExternalSignerRestartsAfterCrash5623=== RUN TestExternalSignerProgramErrorKeepsProcess5624=== PAUSE TestExternalSignerProgramErrorKeepsProcess5625=== RUN TestExternalSignerClose5626=== PAUSE TestExternalSignerClose5627=== RUN TestExternalSignerRejectsInvalidPublicKey5628=== PAUSE TestExternalSignerRejectsInvalidPublicKey5629=== RUN TestExternalSignerCancellation5630=== PAUSE TestExternalSignerCancellation5631=== RUN TestGenerateFingerprint5632=== PAUSE TestGenerateFingerprint5633=== RUN TestParseSigningKey5634=== PAUSE TestParseSigningKey5635=== RUN TestSign5636=== PAUSE TestSign5637=== RUN TestSignBatch5638=== PAUSE TestSignBatch5639=== RUN TestNilKeyPublicKey5640=== PAUSE TestNilKeyPublicKey5641=== RUN TestSignErrors5642=== PAUSE TestSignErrors5643=== CONT TestExternalSigner5644=== CONT TestExternalSignerCancellation5645=== CONT TestExternalSignerProgramErrorKeepsProcess5646=== CONT TestExternalSignerClose5647=== CONT TestSignBatch5648=== CONT TestSignErrors5649=== CONT TestNilKeyPublicKey5650--- PASS: TestNilKeyPublicKey (0.00s)5651=== CONT TestParseSigningKey5652=== RUN TestParseSigningKey/valid_32-byte_key5653=== CONT TestSign5654=== CONT TestGenerateFingerprint5655=== CONT TestExternalSignerErrors5656=== CONT TestExternalSignerRestartsAfterCrash5657=== CONT TestExternalSignerWrongPublicKey5658=== CONT TestExternalSignerRejectsInvalidPublicKey5659=== PAUSE TestParseSigningKey/valid_32-byte_key5660=== RUN TestExternalSignerErrors/unrequested_key5661=== PAUSE TestExternalSignerErrors/unrequested_key5662=== RUN TestGenerateFingerprint/basic_with_references5663=== RUN TestParseSigningKey/valid_32-byte_key_with_different_name5664=== PAUSE TestGenerateFingerprint/basic_with_references5665=== PAUSE TestParseSigningKey/valid_32-byte_key_with_different_name5666--- PASS: TestSign (0.00s)5667--- PASS: TestSignBatch (0.00s)5668=== RUN TestExternalSignerRejectsInvalidPublicKey/#005669=== RUN TestExternalSignerErrors/invalid_signature_base645670=== PAUSE TestExternalSignerErrors/invalid_signature_base645671=== RUN TestSignErrors/nil_key5672=== RUN TestGenerateFingerprint/no_references5673=== PAUSE TestGenerateFingerprint/no_references5674=== PAUSE TestSignErrors/nil_key5675=== RUN TestSignErrors/nil_narinfo5676=== RUN TestExternalSignerErrors/program_error5677=== PAUSE TestExternalSignerErrors/program_error5678=== RUN TestExternalSignerErrors/null5679=== PAUSE TestExternalSignerErrors/null5680=== RUN TestExternalSignerErrors/missing_signatures5681=== PAUSE TestExternalSignerErrors/missing_signatures5682=== RUN TestExternalSignerErrors/missing_signature_name5683=== PAUSE TestExternalSignerErrors/missing_signature_name5684=== PAUSE TestExternalSignerRejectsInvalidPublicKey/#005685=== RUN TestParseSigningKey/no_colon5686=== PAUSE TestParseSigningKey/no_colon5687=== RUN TestGenerateFingerprint/unsorted_references_get_sorted5688=== PAUSE TestSignErrors/nil_narinfo5689=== RUN TestSignErrors/invalid_narinfo5690=== PAUSE TestSignErrors/invalid_narinfo5691=== RUN TestSignErrors/canceled5692=== RUN TestExternalSignerErrors/wrong_signature_name5693=== PAUSE TestExternalSignerErrors/wrong_signature_name5694=== RUN TestExternalSignerRejectsInvalidPublicKey/no-colon5695=== PAUSE TestExternalSignerRejectsInvalidPublicKey/no-colon5696--- PASS: TestExternalSignerClose (0.00s)5697=== RUN TestParseSigningKey/empty_name5698=== PAUSE TestParseSigningKey/empty_name5699=== RUN TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5700=== PAUSE TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5701=== RUN TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5702=== PAUSE TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5703=== RUN TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5704=== PAUSE TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5705=== PAUSE TestGenerateFingerprint/unsorted_references_get_sorted5706=== PAUSE TestSignErrors/canceled5707=== RUN TestExternalSignerErrors/wrong_signature_length5708=== RUN TestParseSigningKey/invalid_base645709=== PAUSE TestParseSigningKey/invalid_base645710=== RUN TestParseSigningKey/wrong_length5711=== PAUSE TestParseSigningKey/wrong_length5712=== CONT TestParseSigningKey/valid_32-byte_key5713=== CONT TestExternalSignerRejectsInvalidPublicKey/#005714=== CONT TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5715=== CONT TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5716=== CONT TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5717=== CONT TestExternalSignerRejectsInvalidPublicKey/no-colon5718=== RUN TestGenerateFingerprint/invalid_nar_hash_prefix5719=== PAUSE TestGenerateFingerprint/invalid_nar_hash_prefix5720=== RUN TestGenerateFingerprint/invalid_nar_hash_length5721=== PAUSE TestGenerateFingerprint/invalid_nar_hash_length5722=== RUN TestGenerateFingerprint/invalid_store_path_prefix5723=== PAUSE TestGenerateFingerprint/invalid_store_path_prefix5724=== RUN TestGenerateFingerprint/invalid_reference_prefix5725=== CONT TestSignErrors/nil_key5726=== CONT TestSignErrors/canceled5727=== CONT TestSignErrors/invalid_narinfo5728=== CONT TestSignErrors/nil_narinfo5729=== PAUSE TestExternalSignerErrors/wrong_signature_length5730=== RUN TestExternalSignerErrors/wrong_signatures_shape5731=== PAUSE TestExternalSignerErrors/wrong_signatures_shape5732=== RUN TestExternalSignerErrors/too_few_signatures5733=== PAUSE TestExternalSignerErrors/too_few_signatures5734=== RUN TestExternalSignerErrors/exit_at_startup5735=== PAUSE TestExternalSignerErrors/exit_at_startup5736=== RUN TestExternalSignerErrors/exit_on_request5737=== PAUSE TestExternalSignerErrors/exit_on_request5738=== CONT TestParseSigningKey/empty_name5739=== CONT TestParseSigningKey/invalid_base645740=== CONT TestParseSigningKey/wrong_length5741=== CONT TestParseSigningKey/no_colon5742=== CONT TestParseSigningKey/valid_32-byte_key_with_different_name5743--- PASS: TestExternalSignerRejectsInvalidPublicKey (0.00s)5744 --- PASS: TestExternalSignerRejectsInvalidPublicKey/#00 (0.00s)5745 --- PASS: TestExternalSignerRejectsInvalidPublicKey/test-key:YQ== (0.00s)5746 --- PASS: TestExternalSignerRejectsInvalidPublicKey/test-key:invalid! (0.00s)5747 --- PASS: TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= (0.00s)5748 --- PASS: TestExternalSignerRejectsInvalidPublicKey/no-colon (0.00s)5749=== PAUSE TestGenerateFingerprint/invalid_reference_prefix5750=== CONT TestGenerateFingerprint/basic_with_references5751=== CONT TestGenerateFingerprint/invalid_store_path_prefix5752=== CONT TestGenerateFingerprint/invalid_nar_hash_length5753--- PASS: TestSignErrors (0.00s)5754 --- PASS: TestSignErrors/nil_key (0.00s)5755 --- PASS: TestSignErrors/canceled (0.00s)5756 --- PASS: TestSignErrors/invalid_narinfo (0.00s)5757 --- PASS: TestSignErrors/nil_narinfo (0.00s)5758--- PASS: TestExternalSignerWrongPublicKey (0.00s)5759--- PASS: TestExternalSignerProgramErrorKeepsProcess (0.01s)5760=== RUN TestExternalSignerErrors/invalid_JSON5761=== CONT TestGenerateFingerprint/invalid_reference_prefix5762=== CONT TestGenerateFingerprint/unsorted_references_get_sorted5763=== CONT TestGenerateFingerprint/invalid_nar_hash_prefix5764=== CONT TestGenerateFingerprint/no_references5765--- PASS: TestParseSigningKey (0.00s)5766 --- PASS: TestParseSigningKey/valid_32-byte_key (0.00s)5767 --- PASS: TestParseSigningKey/empty_name (0.00s)5768 --- PASS: TestParseSigningKey/invalid_base64 (0.00s)5769 --- PASS: TestParseSigningKey/wrong_length (0.00s)5770 --- PASS: TestParseSigningKey/no_colon (0.00s)5771 --- PASS: TestParseSigningKey/valid_32-byte_key_with_different_name (0.00s)5772=== PAUSE TestExternalSignerErrors/invalid_JSON5773=== RUN TestExternalSignerErrors/wrong_shape5774--- PASS: TestGenerateFingerprint (0.00s)5775 --- PASS: TestGenerateFingerprint/basic_with_references (0.00s)5776 --- PASS: TestGenerateFingerprint/invalid_store_path_prefix (0.00s)5777 --- PASS: TestGenerateFingerprint/invalid_nar_hash_length (0.00s)5778 --- PASS: TestGenerateFingerprint/invalid_reference_prefix (0.00s)5779 --- PASS: TestGenerateFingerprint/unsorted_references_get_sorted (0.00s)5780 --- PASS: TestGenerateFingerprint/invalid_nar_hash_prefix (0.00s)5781 --- PASS: TestGenerateFingerprint/no_references (0.00s)5782=== PAUSE TestExternalSignerErrors/wrong_shape5783=== RUN TestExternalSignerErrors/wrong_id5784=== PAUSE TestExternalSignerErrors/wrong_id5785=== RUN TestExternalSignerErrors/null_signatures5786=== PAUSE TestExternalSignerErrors/null_signatures5787=== CONT TestExternalSignerErrors/unrequested_key5788=== CONT TestExternalSignerErrors/null5789=== CONT TestExternalSignerErrors/invalid_signature_base645790=== CONT TestExternalSignerErrors/null_signatures5791=== CONT TestExternalSignerErrors/wrong_id5792=== CONT TestExternalSignerErrors/wrong_shape5793=== CONT TestExternalSignerErrors/invalid_JSON5794=== CONT TestExternalSignerErrors/missing_signatures5795=== CONT TestExternalSignerErrors/exit_on_request5796=== CONT TestExternalSignerErrors/wrong_signature_length5797=== CONT TestExternalSignerErrors/exit_at_startup5798=== CONT TestExternalSignerErrors/wrong_signature_name5799=== CONT TestExternalSignerErrors/too_few_signatures5800=== CONT TestExternalSignerErrors/program_error5801=== CONT TestExternalSignerErrors/missing_signature_name5802=== CONT TestExternalSignerErrors/wrong_signatures_shape5803--- PASS: TestExternalSigner (0.01s)5804--- PASS: TestExternalSignerRestartsAfterCrash (0.01s)5805--- PASS: TestExternalSignerErrors (0.00s)5806 --- PASS: TestExternalSignerErrors/null (0.00s)5807 --- PASS: TestExternalSignerErrors/invalid_signature_base64 (0.00s)5808 --- PASS: TestExternalSignerErrors/wrong_shape (0.01s)5809 --- PASS: TestExternalSignerErrors/unrequested_key (0.01s)5810 --- PASS: TestExternalSignerErrors/null_signatures (0.01s)5811 --- PASS: TestExternalSignerErrors/invalid_JSON (0.01s)5812 --- PASS: TestExternalSignerErrors/wrong_signature_length (0.01s)5813 --- PASS: TestExternalSignerErrors/wrong_id (0.01s)5814 --- PASS: TestExternalSignerErrors/missing_signatures (0.01s)5815 --- PASS: TestExternalSignerErrors/too_few_signatures (0.01s)5816 --- PASS: TestExternalSignerErrors/wrong_signature_name (0.01s)5817 --- PASS: TestExternalSignerErrors/exit_at_startup (0.01s)5818 --- PASS: TestExternalSignerErrors/program_error (0.01s)5819 --- PASS: TestExternalSignerErrors/exit_on_request (0.01s)5820 --- PASS: TestExternalSignerErrors/wrong_signatures_shape (0.01s)5821 --- PASS: TestExternalSignerErrors/missing_signature_name (0.01s)5822--- PASS: TestExternalSignerCancellation (0.02s)5823PASS5824Running OpenBao signer tests...5825=== RUN TestSign5826=== PAUSE TestSign5827=== RUN TestSignErrors5828=== PAUSE TestSignErrors5829=== RUN TestServe5830=== PAUSE TestServe5831=== RUN TestServeRejectsGarbage5832=== PAUSE TestServeRejectsGarbage5833=== CONT TestSign5834=== CONT TestSignErrors5835=== RUN TestSignErrors/http_error5836=== PAUSE TestSignErrors/http_error5837=== CONT TestServeRejectsGarbage5838=== CONT TestServe5839=== RUN TestSignErrors/invalid_json5840=== PAUSE TestSignErrors/invalid_json5841=== RUN TestSignErrors/no_prefix5842=== PAUSE TestSignErrors/no_prefix5843=== RUN TestSignErrors/invalid_base645844=== PAUSE TestSignErrors/invalid_base645845=== CONT TestSignErrors/http_error5846--- PASS: TestServeRejectsGarbage (0.00s)5847=== CONT TestSignErrors/invalid_base645848--- PASS: TestServe (0.00s)5849=== CONT TestSignErrors/no_prefix5850=== CONT TestSignErrors/invalid_json5851--- PASS: TestSign (0.00s)5852--- PASS: TestSignErrors (0.00s)5853 --- PASS: TestSignErrors/http_error (0.00s)5854 --- PASS: TestSignErrors/invalid_json (0.00s)5855 --- PASS: TestSignErrors/invalid_base64 (0.00s)5856 --- PASS: TestSignErrors/no_prefix (0.00s)5857PASS