niks3-go-unit-tests
checks.aarch64-linux.go-unit-tests
· build #507
· raw
1tribuchet: building on eliza2Running client tests...3=== RUN TestDoServerRequestAttachesToken4=== PAUSE TestDoServerRequestAttachesToken5=== RUN TestUploadBuildLog_FileBodyReplayedOnRetry6=== PAUSE TestUploadBuildLog_FileBodyReplayedOnRetry7=== RUN TestRegisterUploadedObjectReusesConnections8=== PAUSE TestRegisterUploadedObjectReusesConnections9=== RUN TestRunGarbageCollection_FinishedOnAnotherReplica10=== PAUSE TestRunGarbageCollection_FinishedOnAnotherReplica11=== RUN TestRunGarbageCollection_NotFoundAfterLocalRun12=== PAUSE TestRunGarbageCollection_NotFoundAfterLocalRun13=== RUN TestCaseHackSuffix14=== PAUSE TestCaseHackSuffix15=== RUN TestFilterOversizedClosures16=== PAUSE TestFilterOversizedClosures17=== RUN TestUploadMultipart_PartsInParallel18=== PAUSE TestUploadMultipart_PartsInParallel19=== RUN TestUploadMultipart_ReturnsAfterDroppedBody20=== PAUSE TestUploadMultipart_ReturnsAfterDroppedBody21=== RUN TestUploadMultipart_FailedProducerIsNotCompleted22=== PAUSE TestUploadMultipart_FailedProducerIsNotCompleted23=== RUN TestPartSizeForNAR24=== PAUSE TestPartSizeForNAR25=== RUN TestDumpPathCaseHackMatchesNix26--- PASS: TestDumpPathCaseHackMatchesNix (1.44s)27=== RUN TestDumpPathCaseHackCollision28--- PASS: TestDumpPathCaseHackCollision (0.00s)29=== RUN TestDumpPathMatchesNix30=== PAUSE TestDumpPathMatchesNix31=== RUN TestDumpPathSingleFile32=== PAUSE TestDumpPathSingleFile33=== RUN TestDumpPathWriterError34=== PAUSE TestDumpPathWriterError35=== RUN TestEncodeNixBase3236=== PAUSE TestEncodeNixBase3237=== RUN TestEncodeNixBase32WithRealHash38=== PAUSE TestEncodeNixBase32WithRealHash39=== RUN TestConvertHashToNix3240=== PAUSE TestConvertHashToNix3241=== RUN TestGetStorePathHash42=== PAUSE TestGetStorePathHash43=== RUN TestPathInfoHashCompatibility44=== PAUSE TestPathInfoHashCompatibility45=== RUN TestParsePathInfoJSON46=== PAUSE TestParsePathInfoJSON47=== RUN TestParsePathInfoJSONMultiplePaths48=== PAUSE TestParsePathInfoJSONMultiplePaths49=== RUN TestPathInfoCACompatibility50=== PAUSE TestPathInfoCACompatibility51=== RUN TestRateLimiterFeedback52=== PAUSE TestRateLimiterFeedback53=== RUN TestRateLimiterFeedback_400DoesNotCountAsSuccess54=== PAUSE TestRateLimiterFeedback_400DoesNotCountAsSuccess55=== RUN TestRegisterUploadedObject_BoundedAgainstSilentServer56=== PAUSE TestRegisterUploadedObject_BoundedAgainstSilentServer57=== RUN TestRegisterUploadedObject_OutlivesPush58=== PAUSE TestRegisterUploadedObject_OutlivesPush59=== RUN TestResolveStorePath60=== PAUSE TestResolveStorePath61=== RUN TestDoWithRetry_BodyReplayedViaGetBody62=== PAUSE TestDoWithRetry_BodyReplayedViaGetBody63=== RUN TestRetryTimeoutOutlastsMaxRetries64=== PAUSE TestRetryTimeoutOutlastsMaxRetries65=== RUN TestDoWithRetry_FinalResponseBodyReadable66=== PAUSE TestDoWithRetry_FinalResponseBodyReadable67=== RUN TestShellSplit68=== PAUSE TestShellSplit69=== RUN TestShellSplitErrors70=== PAUSE TestShellSplitErrors71=== RUN TestStreamPushReportsEveryPath72=== PAUSE TestStreamPushReportsEveryPath73=== RUN TestStreamPushBatchesUnderLoad74=== PAUSE TestStreamPushBatchesUnderLoad75=== RUN TestStreamPushIsolatesFailures76=== PAUSE TestStreamPushIsolatesFailures77=== RUN TestStreamPushGivesUpOnDeadServer78=== PAUSE TestStreamPushGivesUpOnDeadServer79=== RUN TestStreamPushRequestLine80=== PAUSE TestStreamPushRequestLine81=== RUN TestStreamPushReportsSignatures82=== PAUSE TestStreamPushReportsSignatures83=== RUN TestStreamPushReportsSkippedPaths84=== PAUSE TestStreamPushReportsSkippedPaths85=== RUN TestClientSignaturesByStorePath86=== PAUSE TestClientSignaturesByStorePath87=== RUN TestStreamPushAnswersEveryLineWhenCanceled88=== PAUSE TestStreamPushAnswersEveryLineWhenCanceled89=== RUN TestStreamPushStopsWhenCanceled90=== PAUSE TestStreamPushStopsWhenCanceled91=== RUN TestSetClientTLS92=== PAUSE TestSetClientTLS93=== RUN TestSetClientTLSDoesNotMutateDefaultTransport94=== PAUSE TestSetClientTLSDoesNotMutateDefaultTransport95=== RUN TestSetClientTLSErrors96=== PAUSE TestSetClientTLSErrors97=== RUN TestStaticToken98=== PAUSE TestStaticToken99=== RUN TestFileTokenReadsAndCaches100=== PAUSE TestFileTokenReadsAndCaches101=== RUN TestFileTokenMissing102=== PAUSE TestFileTokenMissing103=== RUN TestFileTokenEmpty104=== PAUSE TestFileTokenEmpty105=== RUN TestScriptTokenNoExpiryRerunsEveryCall106=== PAUSE TestScriptTokenNoExpiryRerunsEveryCall107=== RUN TestScriptTokenCachesUntilRefresh108=== PAUSE TestScriptTokenCachesUntilRefresh109=== RUN TestScriptTokenEmptyToken110=== PAUSE TestScriptTokenEmptyToken111=== RUN TestScriptTokenBadJSON112=== PAUSE TestScriptTokenBadJSON113=== RUN TestScriptTokenScriptFails114=== PAUSE TestScriptTokenScriptFails115=== RUN TestScriptTokenEmptyCommand116=== PAUSE TestScriptTokenEmptyCommand117=== CONT TestDoServerRequestAttachesToken118=== CONT TestScriptTokenEmptyCommand119--- PASS: TestScriptTokenEmptyCommand (0.00s)120=== CONT TestStreamPushRequestLine121=== CONT TestScriptTokenScriptFails1222026/11/12 07:17:45 ERROR Upload failed error=boom count=1123=== CONT TestStreamPushAnswersEveryLineWhenCanceled124=== CONT TestScriptTokenBadJSON125--- PASS: TestScriptTokenScriptFails (0.51s)126=== CONT TestRateLimiterFeedback_400DoesNotCountAsSuccess1272026/11/12 07:17:46 WARN Rate limiter enabled after throttle name=server-test rate=5128=== CONT TestScriptTokenEmptyToken1292000/01/01 00:00:00 ERROR Upload failed error="context canceled" count=2130=== CONT TestScriptTokenCachesUntilRefresh131=== CONT TestScriptTokenNoExpiryRerunsEveryCall132=== CONT TestFileTokenEmpty133=== CONT TestFileTokenMissing134=== CONT TestFileTokenReadsAndCaches135=== CONT TestRateLimiterFeedback136=== CONT TestStreamPushIsolatesFailures137=== CONT TestStreamPushBatchesUnderLoad1382026/11/12 07:17:47 ERROR Upload failed error="bad path" count=3139=== CONT TestStaticToken140=== CONT TestUploadMultipart_PartsInParallel141=== CONT TestStreamPushReportsEveryPath142=== RUN TestRateLimiterFeedback/429_enables_limiter143=== PAUSE TestRateLimiterFeedback/429_enables_limiter144=== RUN TestRateLimiterFeedback/503_enables_limiter145=== PAUSE TestRateLimiterFeedback/503_enables_limiter146=== RUN TestRateLimiterFeedback/200_does_not_enable_limiter147=== PAUSE TestRateLimiterFeedback/200_does_not_enable_limiter148=== RUN TestRateLimiterFeedback/400_does_not_enable_limiter149=== PAUSE TestRateLimiterFeedback/400_does_not_enable_limiter150=== CONT TestSetClientTLSDoesNotMutateDefaultTransport151=== CONT TestDumpPathWriterError152=== CONT TestStreamPushStopsWhenCanceled153=== CONT TestDoWithRetry_BodyReplayedViaGetBody154=== CONT TestPathInfoCACompatibility155=== CONT TestClientSignaturesByStorePath156=== CONT TestDumpPathSingleFile157=== CONT TestShellSplitErrors158=== CONT TestResolveStorePath159=== CONT TestStreamPushReportsSkippedPaths160=== CONT TestRegisterUploadedObject_OutlivesPush161=== CONT TestStreamPushReportsSignatures162=== CONT TestRegisterUploadedObject_BoundedAgainstSilentServer163--- PASS: TestStreamPushAnswersEveryLineWhenCanceled (0.00s)164=== CONT TestStreamPushGivesUpOnDeadServer165=== RUN TestStreamPushBatchesUnderLoad/oneAtATime=false166=== CONT TestSetClientTLSErrors167=== CONT TestSetClientTLS168=== CONT TestShellSplit169--- PASS: TestScriptTokenBadJSON (1.50s)170--- PASS: TestFileTokenReadsAndCaches (0.00s)171--- PASS: TestFileTokenMissing (0.00s)172--- PASS: TestStaticToken (0.00s)173--- PASS: TestStreamPushIsolatesFailures (0.00s)174--- PASS: TestScriptTokenEmptyToken (1.50s)175--- PASS: TestClientSignaturesByStorePath (0.00s)176--- PASS: TestShellSplitErrors (0.00s)177--- PASS: TestFileTokenEmpty (0.00s)1782026/11/12 07:17:47 ERROR Upload failed error=boom count=1179--- PASS: TestStreamPushReportsSignatures (0.00s)180=== CONT TestDoWithRetry_FinalResponseBodyReadable181--- PASS: TestStreamPushReportsSkippedPaths (0.00s)182--- PASS: TestStreamPushReportsEveryPath (0.00s)183=== CONT TestPartSizeForNAR184=== CONT TestDumpPathMatchesNix185--- PASS: TestShellSplit (0.00s)186=== CONT TestUploadMultipart_FailedProducerIsNotCompleted187=== RUN TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part188=== PAUSE TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part189=== RUN TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary190--- PASS: TestResolveStorePath (0.00s)191=== PAUSE TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary192=== CONT TestRunGarbageCollection_NotFoundAfterLocalRun193=== PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=false194=== RUN TestStreamPushBatchesUnderLoad/oneAtATime=true195=== PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=true196=== CONT TestFilterOversizedClosures1972026/11/12 07:17:47 ERROR Upload failed error="connection refused" count=20198=== CONT TestRetryTimeoutOutlastsMaxRetries1992026/11/12 07:17:47 ERROR Server seems unavailable, giving up on batch untried=17200=== RUN TestFilterOversizedClosures/no_limit_keeps_everything201=== PAUSE TestFilterOversizedClosures/no_limit_keeps_everything202=== CONT TestUploadMultipart_ReturnsAfterDroppedBody203=== RUN TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped204=== PAUSE TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped205=== RUN TestFilterOversizedClosures/all_closures_skipped206=== PAUSE TestFilterOversizedClosures/all_closures_skipped207=== RUN TestPartSizeForNAR/zero_stays_at_minimum208=== PAUSE TestPartSizeForNAR/zero_stays_at_minimum209=== RUN TestPartSizeForNAR/small_stays_at_minimum210=== PAUSE TestPartSizeForNAR/small_stays_at_minimum211=== RUN TestPartSizeForNAR/80_GiB_fits_at_minimum212=== PAUSE TestPartSizeForNAR/80_GiB_fits_at_minimum213=== RUN TestPartSizeForNAR/115_GiB_needs_larger_parts214=== PAUSE TestPartSizeForNAR/115_GiB_needs_larger_parts215=== RUN TestPartSizeForNAR/1_TiB216=== PAUSE TestPartSizeForNAR/1_TiB217=== CONT TestCaseHackSuffix218=== CONT TestRunGarbageCollection_FinishedOnAnotherReplica219=== RUN TestPartSizeForNAR/5_TiB_S3_max_object220=== PAUSE TestPartSizeForNAR/5_TiB_S3_max_object221=== RUN TestPartSizeForNAR/capped_at_5_GiB222=== PAUSE TestPartSizeForNAR/capped_at_5_GiB223=== CONT TestRegisterUploadedObjectReusesConnections224--- PASS: TestStreamPushGivesUpOnDeadServer (0.00s)225--- PASS: TestStreamPushStopsWhenCanceled (0.00s)226=== RUN TestPathInfoCACompatibility/null_ca_field227=== PAUSE TestPathInfoCACompatibility/null_ca_field228=== RUN TestPathInfoCACompatibility/old_string_format_-_text229=== PAUSE TestPathInfoCACompatibility/old_string_format_-_text230=== RUN TestPathInfoCACompatibility/old_string_format_-_fixed_recursive231=== PAUSE TestPathInfoCACompatibility/old_string_format_-_fixed_recursive232=== RUN TestPathInfoCACompatibility/new_structured_format_-_text233=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_text234=== RUN TestPathInfoCACompatibility/new_structured_format_-_nar_method235=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_nar_method236=== CONT TestUploadBuildLog_FileBodyReplayedOnRetry237=== CONT TestPathInfoHashCompatibility238=== RUN TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)239=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)240=== RUN TestPathInfoHashCompatibility/old_string_format_with_colon241=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_colon242=== RUN TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI243=== PAUSE TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI244=== RUN TestPathInfoHashCompatibility/new_structured_format_with_sha512245=== PAUSE TestPathInfoHashCompatibility/new_structured_format_with_sha512246=== CONT TestParsePathInfoJSONMultiplePaths247=== RUN TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths248=== PAUSE TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths249=== RUN TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths250=== PAUSE TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths251=== CONT TestParsePathInfoJSON252=== RUN TestParsePathInfoJSON/Nix_format253=== PAUSE TestParsePathInfoJSON/Nix_format254=== RUN TestParsePathInfoJSON/Lix_format255=== PAUSE TestParsePathInfoJSON/Lix_format256=== RUN TestParsePathInfoJSON/empty_input257=== PAUSE TestParsePathInfoJSON/empty_input258=== RUN TestParsePathInfoJSON/whitespace_only259=== PAUSE TestParsePathInfoJSON/whitespace_only260=== RUN TestParsePathInfoJSON/invalid_JSON261=== PAUSE TestParsePathInfoJSON/invalid_JSON262=== CONT TestConvertHashToNix32263=== RUN TestConvertHashToNix32/SRI_format_to_Nix32264=== PAUSE TestConvertHashToNix32/SRI_format_to_Nix32265=== RUN TestConvertHashToNix32/already_Nix32_format266=== PAUSE TestConvertHashToNix32/already_Nix32_format267=== RUN TestConvertHashToNix32/invalid_format268=== PAUSE TestConvertHashToNix32/invalid_format269=== CONT TestGetStorePathHash270=== RUN TestGetStorePathHash/valid_store_path271=== PAUSE TestGetStorePathHash/valid_store_path272=== RUN TestGetStorePathHash/basename_without_hyphen_should_error273=== PAUSE TestGetStorePathHash/basename_without_hyphen_should_error274=== RUN TestGetStorePathHash/hash_with_invalid_characters_should_error275=== PAUSE TestGetStorePathHash/hash_with_invalid_characters_should_error276=== RUN TestGetStorePathHash/hash_with_wrong_length_should_error277=== PAUSE TestGetStorePathHash/hash_with_wrong_length_should_error278=== CONT TestEncodeNixBase32WithRealHash2792026/11/12 07:17:50 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:33327/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:33327/api/uploads/complete2802026/11/12 07:17:50 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"2812026/11/12 07:17:50 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:33327/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:33327/api/uploads/complete2822026/11/12 07:17:50 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"2832026/11/12 07:17:50 WARN Rate limiter enabled after throttle name=server-test rate=52842026/11/12 07:17:50 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:33327/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:33327/api/uploads/complete2852026/11/12 07:17:50 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"2862026/11/12 07:17:50 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:37877287--- PASS: TestRegisterUploadedObject_OutlivesPush (2.34s)2882026/11/12 07:17:50 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:37859289--- PASS: TestDoServerRequestAttachesToken (4.36s)2902026/11/12 07:17:50 WARN Rate limiter enabled after throttle name=server-test rate=5291=== CONT TestEncodeNixBase322922026/11/12 07:17:50 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:38165293=== RUN TestEncodeNixBase32/test_string_hash294=== PAUSE TestEncodeNixBase32/test_string_hash2952026/11/12 07:17:50 WARN Rate limiter backed off name=server-test rate=5296=== CONT TestRateLimiterFeedback/400_does_not_enable_limiter297=== CONT TestRateLimiterFeedback/503_enables_limiter2982026/11/12 07:17:50 WARN Rate limiter backed off name=server-test rate=52992026/11/12 07:17:50 WARN Request returned retryable status, retrying attempt=2 backoff=0s status=503 url=http://127.0.0.1:38165300=== CONT TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part301=== CONT TestRateLimiterFeedback/200_does_not_enable_limiter302=== CONT TestRateLimiterFeedback/429_enables_limiter3032026/11/12 07:17:50 INFO Garbage collection started3042026/11/12 07:17:50 INFO Garbage collection started305--- PASS: TestUploadMultipart_ReturnsAfterDroppedBody (2.34s)306--- PASS: TestRegisterUploadedObject_BoundedAgainstSilentServer (2.34s)307--- PASS: TestEncodeNixBase32WithRealHash (2.34s)308--- PASS: TestDoWithRetry_BodyReplayedViaGetBody (2.34s)309--- PASS: TestDoWithRetry_FinalResponseBodyReadable (2.34s)310=== CONT TestStreamPushBatchesUnderLoad/oneAtATime=false311=== RUN TestEncodeNixBase32/empty_input312=== PAUSE TestEncodeNixBase32/empty_input313=== CONT TestStreamPushBatchesUnderLoad/oneAtATime=true314=== CONT TestFilterOversizedClosures/no_limit_keeps_everything315--- PASS: TestStreamPushBatchesUnderLoad (0.00s)316 --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=false (0.00s)317 --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=true (0.00s)3182026/11/12 07:17:50 WARN Rate limiter enabled after throttle name=server-test rate=5319=== CONT TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary3202026/11/12 07:17:50 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:329173212026/11/12 07:17:50 WARN Rate limiter enabled after throttle name=server-test rate=5322=== CONT TestPartSizeForNAR/zero_stays_at_minimum3232026/11/12 07:17:50 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=429 url=http://127.0.0.1:38199324=== CONT TestFilterOversizedClosures/all_closures_skipped3252026/11/12 07:17:50 WARN Rate limiter backed off name=server-test rate=53262026/11/12 07:17:50 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper nar_size=100 max_nar_size=50327--- PASS: TestDumpPathSingleFile (2.35s)328=== CONT TestPartSizeForNAR/capped_at_5_GiB329=== CONT TestPartSizeForNAR/115_GiB_needs_larger_parts330=== CONT TestPartSizeForNAR/80_GiB_fits_at_minimum331=== CONT TestPartSizeForNAR/5_TiB_S3_max_object3322026/11/12 07:17:50 WARN Rate limiter backed off name=server-test rate=5333=== CONT TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)334=== CONT TestPartSizeForNAR/1_TiB335=== CONT TestPathInfoCACompatibility/old_string_format_-_fixed_recursive336=== CONT TestPathInfoCACompatibility/old_string_format_-_text337=== CONT TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths338=== CONT TestPathInfoHashCompatibility/new_structured_format_with_sha512339--- PASS: TestRateLimiterFeedback (0.00s)340 --- PASS: TestRateLimiterFeedback/400_does_not_enable_limiter (0.00s)341 --- PASS: TestRateLimiterFeedback/200_does_not_enable_limiter (0.00s)342 --- PASS: TestRateLimiterFeedback/429_enables_limiter (0.00s)343 --- PASS: TestRateLimiterFeedback/503_enables_limiter (0.00s)344=== CONT TestPartSizeForNAR/small_stays_at_minimum345--- PASS: TestPartSizeForNAR (0.00s)346 --- PASS: TestPartSizeForNAR/zero_stays_at_minimum (0.00s)347 --- PASS: TestPartSizeForNAR/capped_at_5_GiB (0.00s)348 --- PASS: TestPartSizeForNAR/115_GiB_needs_larger_parts (0.00s)349 --- PASS: TestPartSizeForNAR/80_GiB_fits_at_minimum (0.00s)350 --- PASS: TestPartSizeForNAR/5_TiB_S3_max_object (0.00s)351 --- PASS: TestPartSizeForNAR/1_TiB (0.00s)352 --- PASS: TestPartSizeForNAR/small_stays_at_minimum (0.00s)353=== CONT TestPathInfoCACompatibility/null_ca_field354=== CONT TestPathInfoHashCompatibility/old_string_format_with_colon355=== CONT TestPathInfoCACompatibility/new_structured_format_-_nar_method356=== CONT TestParsePathInfoJSON/Nix_format3572026/11/12 07:17:50 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:37877358=== CONT TestParsePathInfoJSON/whitespace_only359=== CONT TestParsePathInfoJSON/empty_input360=== CONT TestParsePathInfoJSON/Lix_format361=== CONT TestConvertHashToNix32/invalid_format362=== CONT TestConvertHashToNix32/already_Nix32_format363=== CONT TestGetStorePathHash/valid_store_path3642026/11/12 07:17:50 INFO Garbage collection progress phase=running_on_another_replica failed_uploads_deleted=0 old_closures_deleted=0 objects_marked=0 objects_deleted=0 objects_failed=0365=== CONT TestGetStorePathHash/hash_with_wrong_length_should_error366=== CONT TestGetStorePathHash/basename_without_hyphen_should_error367=== CONT TestEncodeNixBase32/test_string_hash368=== CONT TestEncodeNixBase32/empty_input369=== CONT TestGetStorePathHash/hash_with_invalid_characters_should_error370=== CONT TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI371=== CONT TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped3722026/11/12 07:17:50 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb-vm-image nar_size=5000 max_nar_size=2000373--- PASS: TestRunGarbageCollection_NotFoundAfterLocalRun (2.35s)374=== CONT TestParsePathInfoJSON/invalid_JSON375=== CONT TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths376=== CONT TestPathInfoCACompatibility/new_structured_format_-_text3772026/11/12 07:17:50 WARN Rate limiter enabled after throttle name=s3-test rate=53782026/11/12 07:17:50 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:42379379--- PASS: TestUploadMultipart_FailedProducerIsNotCompleted (0.00s)380 --- PASS: TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part (0.00s)381 --- PASS: TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary (0.00s)382=== CONT TestConvertHashToNix32/SRI_format_to_Nix32383--- PASS: TestScriptTokenCachesUntilRefresh (2.35s)384--- PASS: TestEncodeNixBase32 (0.00s)385 --- PASS: TestEncodeNixBase32/test_string_hash (0.00s)386 --- PASS: TestEncodeNixBase32/empty_input (0.00s)387--- PASS: TestGetStorePathHash (0.00s)388 --- PASS: TestGetStorePathHash/valid_store_path (0.00s)389 --- PASS: TestGetStorePathHash/hash_with_wrong_length_should_error (0.00s)390 --- PASS: TestGetStorePathHash/basename_without_hyphen_should_error (0.00s)391 --- PASS: TestGetStorePathHash/hash_with_invalid_characters_should_error (0.00s)392--- PASS: TestPathInfoHashCompatibility (0.00s)393 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI) (0.00s)394 --- PASS: TestPathInfoHashCompatibility/new_structured_format_with_sha512 (0.00s)395 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_colon (0.00s)396 --- PASS: TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI (0.00s)397--- PASS: TestFilterOversizedClosures (0.00s)398 --- PASS: TestFilterOversizedClosures/no_limit_keeps_everything (0.00s)399 --- PASS: TestFilterOversizedClosures/all_closures_skipped (0.00s)400 --- PASS: TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped (0.00s)401--- PASS: TestParsePathInfoJSON (0.00s)402 --- PASS: TestParsePathInfoJSON/whitespace_only (0.00s)403 --- PASS: TestParsePathInfoJSON/empty_input (0.00s)404 --- PASS: TestParsePathInfoJSON/Lix_format (0.00s)405 --- PASS: TestParsePathInfoJSON/Nix_format (0.00s)406 --- PASS: TestParsePathInfoJSON/invalid_JSON (0.00s)407--- PASS: TestParsePathInfoJSONMultiplePaths (0.00s)408 --- PASS: TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths (0.00s)409 --- PASS: TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths (0.00s)410--- PASS: TestPathInfoCACompatibility (0.00s)411 --- PASS: TestPathInfoCACompatibility/old_string_format_-_fixed_recursive (0.00s)412 --- PASS: TestPathInfoCACompatibility/old_string_format_-_text (0.00s)413 --- PASS: TestPathInfoCACompatibility/null_ca_field (0.00s)414 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_nar_method (0.00s)415 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_text (0.00s)416--- PASS: TestConvertHashToNix32 (0.00s)417 --- PASS: TestConvertHashToNix32/invalid_format (0.00s)418 --- PASS: TestConvertHashToNix32/already_Nix32_format (0.00s)419 --- PASS: TestConvertHashToNix32/SRI_format_to_Nix32 (0.00s)4202026/11/12 07:17:50 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:37877421--- PASS: TestUploadBuildLog_FileBodyReplayedOnRetry (2.34s)4222026/11/12 07:17:50 WARN Garbage collection finished on another replica. Its statistics are not available here423--- PASS: TestRunGarbageCollection_FinishedOnAnotherReplica (2.35s)424--- PASS: TestScriptTokenNoExpiryRerunsEveryCall (2.35s)4252026/11/12 07:17:50 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:37877426=== RUN TestSetClientTLSErrors/missing_cert_file427=== PAUSE TestSetClientTLSErrors/missing_cert_file428=== RUN TestSetClientTLSErrors/missing_key_file429=== PAUSE TestSetClientTLSErrors/missing_key_file430=== RUN TestSetClientTLSErrors/missing_ca_file431=== PAUSE TestSetClientTLSErrors/missing_ca_file432=== RUN TestSetClientTLSErrors/invalid_ca_file433=== PAUSE TestSetClientTLSErrors/invalid_ca_file434=== CONT TestSetClientTLSErrors/missing_cert_file435=== CONT TestSetClientTLSErrors/missing_ca_file436=== CONT TestSetClientTLSErrors/missing_key_file4372026/11/12 07:17:50 WARN Request returned retryable status, retrying attempt=3 backoff=1ms status=502 url=http://127.0.0.1:37877438=== CONT TestSetClientTLSErrors/invalid_ca_file439--- PASS: TestSetClientTLSDoesNotMutateDefaultTransport (2.35s)440--- PASS: TestSetClientTLSErrors (2.35s)441 --- PASS: TestSetClientTLSErrors/missing_cert_file (0.00s)442 --- PASS: TestSetClientTLSErrors/missing_key_file (0.00s)443 --- PASS: TestSetClientTLSErrors/missing_ca_file (0.00s)444 --- PASS: TestSetClientTLSErrors/invalid_ca_file (0.00s)4452026/11/12 07:17:50 WARN Request returned retryable status, retrying attempt=4 backoff=1ms status=502 url=http://127.0.0.1:37877446=== RUN TestSetClientTLS/rejects_connection_without_client_cert447=== PAUSE TestSetClientTLS/rejects_connection_without_client_cert448=== RUN TestSetClientTLS/succeeds_with_client_cert_and_CA449=== PAUSE TestSetClientTLS/succeeds_with_client_cert_and_CA450=== RUN TestSetClientTLS/preserves_debug_logging_transport451=== PAUSE TestSetClientTLS/preserves_debug_logging_transport452=== CONT TestSetClientTLS/rejects_connection_without_client_cert453=== CONT TestSetClientTLS/preserves_debug_logging_transport4542026/11/12 07:17:50 WARN Request returned retryable status, retrying attempt=5 backoff=1ms status=502 url=http://127.0.0.1:37877455=== CONT TestSetClientTLS/succeeds_with_client_cert_and_CA4562026/11/12 07:17:50 WARN Request returned retryable status, retrying attempt=6 backoff=1ms status=502 url=http://127.0.0.1:378774572026/11/12 07:17:50 WARN Request returned retryable status, retrying attempt=7 backoff=1ms status=502 url=http://127.0.0.1:378774582026/11/12 07:17:50 WARN Request returned retryable status, retrying attempt=8 backoff=1ms status=502 url=http://127.0.0.1:37877459--- PASS: TestRateLimiterFeedback_400DoesNotCountAsSuccess (5.48s)4602026/11/12 07:17:51 WARN Request returned retryable status, retrying attempt=9 backoff=1ms status=502 url=http://127.0.0.1:378774612026/11/12 07:17:51 WARN Request returned retryable status, retrying attempt=10 backoff=1ms status=502 url=http://127.0.0.1:37877462--- PASS: TestStreamPushRequestLine (6.00s)463--- PASS: TestRegisterUploadedObjectReusesConnections (3.99s)4642026/11/12 07:17:51 WARN Request returned retryable status, retrying attempt=11 backoff=1ms status=502 url=http://127.0.0.1:37877465--- PASS: TestCaseHackSuffix (4.00s)4662026/11/12 07:17:51 http: TLS handshake error from 127.0.0.1:53500: remote error: tls: bad certificate467--- PASS: TestSetClientTLS (2.35s)468 --- PASS: TestSetClientTLS/preserves_debug_logging_transport (1.62s)469 --- PASS: TestSetClientTLS/succeeds_with_client_cert_and_CA (1.63s)470 --- PASS: TestSetClientTLS/rejects_connection_without_client_cert (1.65s)4712026/11/12 07:17:51 WARN Request returned retryable status, retrying attempt=12 backoff=1ms status=502 url=http://127.0.0.1:378774722026/11/12 07:17:51 WARN Request returned retryable status, retrying attempt=13 backoff=1ms status=502 url=http://127.0.0.1:378774732026/11/12 07:17:51 WARN Request returned retryable status, retrying attempt=14 backoff=1ms status=502 url=http://127.0.0.1:378774742026/11/12 07:17:51 WARN Request returned retryable status, retrying attempt=15 backoff=1ms status=502 url=http://127.0.0.1:378774752026/11/12 07:17:51 WARN Request returned retryable status, retrying attempt=16 backoff=1ms status=502 url=http://127.0.0.1:378774762026/11/12 07:17:51 WARN Request returned retryable status, retrying attempt=17 backoff=1ms status=502 url=http://127.0.0.1:37877477--- PASS: TestRetryTimeoutOutlastsMaxRetries (4.01s)478--- PASS: TestDumpPathWriterError (4.02s)479--- PASS: TestUploadMultipart_PartsInParallel (4.59s)480--- PASS: TestDumpPathMatchesNix (5.28s)481PASS482Running server tests...483The files belonging to this database system will be owned by user "nixbld".484This user must also own the server process.485486The database cluster will be initialized with locale "C".487The default database encoding has accordingly been set to "SQL_ASCII".488The default text search configuration will be set to "english".489490Data page checksums are enabled.491492creating directory /build/postgres179743418/data ... ok493creating subdirectories ... ok494selecting dynamic shared memory implementation ... posix495selecting default "max_connections" ... 100496selecting default "shared_buffers" ... 128MB497selecting default time zone ... UTC498creating configuration files ... ok499running bootstrap script ... ok500performing post-bootstrap initialization ... ok501syncing data to disk ... ok502503initdb: warning: enabling "trust" authentication for local connections504initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.505506Success. You can now start the database server using:507508 pg_ctl -D /build/postgres179743418/data -l logfile start509510/build/postgres179743418:5432 - no response511/build/postgres179743418:5432 - no response512/build/postgres179743418:5432 - no response513/build/postgres179743418:5432 - no response514/build/postgres179743418:5432 - no response515/build/postgres179743418:5432 - no response516/build/postgres179743418:5432 - no response517/build/postgres179743418:5432 - no response518/build/postgres179743418:5432 - no response519/build/postgres179743418:5432 - no response520/build/postgres179743418:5432 - no response521/build/postgres179743418:5432 - no response5222026-11-12 07:18:18.168 UTC [131] LOG: starting PostgreSQL 18.6 on aarch64-unknown-linux-gnu, compiled by clang version 21.1.8, 64-bit5232026-11-12 07:18:18.168 UTC [131] LOG: listening on Unix socket "/build/postgres179743418/.s.PGSQL.5432"5242026-11-12 07:18:18.175 UTC [149] LOG: database system was shut down at 2026-11-12 07:18:16 UTC5252026-11-12 07:18:18.182 UTC [131] LOG: database system is ready to accept connections526/build/postgres179743418:5432 - accepting connections527{"timestamp":"2026-11-12T07:18:29.122355149Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"c6ab23a5-4031-4733-9679-2ee6c89f061b","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"GET","uri":"/health/ready","status_code":503,"suppressed_errors":0,"duration_ms":10,"result":"server_error","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":502,"threadName":"rustfs-worker","threadId":"ThreadId(196)"}528{"timestamp":"2026-11-12T07:18:29.239373247Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"970182dd-e1a6-4d65-806a-48a66f2a6783","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"GET","uri":"/health/ready","status_code":503,"suppressed_errors":0,"duration_ms":0,"result":"server_error","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":502,"threadName":"rustfs-worker","threadId":"ThreadId(199)"}529=== RUN TestAbortSucceeded530=== PAUSE TestAbortSucceeded531=== RUN TestService_AuthMiddleware532=== PAUSE TestService_AuthMiddleware533=== RUN TestService_AuthMiddleware_MTLSProxyHeader534=== PAUSE TestService_AuthMiddleware_MTLSProxyHeader535=== RUN TestService_AuthMiddleware_MTLSBoundSubjects536=== PAUSE TestService_AuthMiddleware_MTLSBoundSubjects537=== RUN TestService_ReadAuthMiddleware538=== PAUSE TestService_ReadAuthMiddleware539=== RUN TestService_AuthMiddleware_OIDC540=== PAUSE TestService_AuthMiddleware_OIDC541=== RUN TestService_RequireScope_OIDC542=== PAUSE TestService_RequireScope_OIDC543=== RUN TestService_ReadScope_PublicByDefault544=== PAUSE TestService_ReadScope_PublicByDefault545=== RUN TestCacheConfigHandler546=== PAUSE TestCacheConfigHandler547=== RUN TestCacheStatsHandler548=== PAUSE TestCacheStatsHandler549=== RUN TestClientCADerivations550=== PAUSE TestClientCADerivations551=== RUN TestClientErrorHandling552=== PAUSE TestClientErrorHandling553=== RUN TestClientIntegration554=== PAUSE TestClientIntegration555=== RUN TestClientMultipleUploads556=== PAUSE TestClientMultipleUploads557=== RUN TestClientWithDependencies558=== PAUSE TestClientWithDependencies559=== RUN TestClientSharedPathCommittedMidPush560=== PAUSE TestClientSharedPathCommittedMidPush561=== RUN TestPinProtectsFromGC562=== PAUSE TestPinProtectsFromGC563=== RUN TestClientPushesUseOnePush564=== PAUSE TestClientPushesUseOnePush565=== RUN TestClientFallsBackToClosures566=== PAUSE TestClientFallsBackToClosures567=== RUN TestDeleteClosuresRacingPins568=== PAUSE TestDeleteClosuresRacingPins569=== RUN TestConcurrentCommitsSharingObjectsDoNotDeadlock570=== PAUSE TestConcurrentCommitsSharingObjectsDoNotDeadlock571=== RUN TestResolveDBConnectionString572=== PAUSE TestResolveDBConnectionString573=== RUN TestConnectWaitsForAPeerMigration574=== PAUSE TestConnectWaitsForAPeerMigration575=== RUN TestLeadElectsOneAndHandsOver576=== PAUSE TestLeadElectsOneAndHandsOver577=== RUN TestLeadIncumbentWinsAfterRestart5782026/11/12 07:18:35 INFO lead: acquired remote=192.0.2.1:1234 generation=15792026/11/12 07:18:35 INFO lead: released remote=192.0.2.1:1234 generation=15802026/11/12 07:18:35 INFO lead: acquired remote=192.0.2.1:1234 generation=25812026/11/12 07:18:36 INFO lead: released remote=192.0.2.1:1234 generation=2582--- PASS: TestLeadIncumbentWinsAfterRestart (7.63s)583=== RUN TestLeadUnannouncedStandbyYieldsToIncumbent584=== PAUSE TestLeadUnannouncedStandbyYieldsToIncumbent585=== RUN TestLeadIncumbentAnnouncesAtOnce586=== PAUSE TestLeadIncumbentAnnouncesAtOnce587=== RUN TestLeadGenerationGrowsWithEveryGrant588=== PAUSE TestLeadGenerationGrowsWithEveryGrant589=== RUN TestLeadEndsOnShutdown590=== PAUSE TestLeadEndsOnShutdown591=== RUN TestLeadEndsWhenItsConnectionHangs5922026/11/12 07:18:42 INFO lead: acquired remote=192.0.2.1:1234 generation=15932026-11-12 07:18:43.735 UTC [400] FATAL: terminating connection due to administrator command5942026/11/12 07:18:43 INFO lead: acquired remote=192.0.2.1:1234 generation=25952026/11/12 07:18:44 WARN lead: lock connection lost error="timeout: context deadline exceeded"5962026/11/12 07:18:44 INFO lead: released remote=192.0.2.1:1234 generation=15972026/11/12 07:18:45 INFO lead: released remote=192.0.2.1:1234 generation=2598--- PASS: TestLeadEndsWhenItsConnectionHangs (8.32s)599=== RUN TestGCAdvisoryLockBlocksConcurrentRun600=== PAUSE TestGCAdvisoryLockBlocksConcurrentRun601=== RUN TestGCBugBareHashReferences602=== PAUSE TestGCBugBareHashReferences603=== RUN TestGCMetrics604=== PAUSE TestGCMetrics605=== RUN TestPushDedupSurvivesConcurrentGC606=== PAUSE TestPushDedupSurvivesConcurrentGC607=== RUN TestDeduplicatedObjectsRecordedAsPending608=== PAUSE TestDeduplicatedObjectsRecordedAsPending609=== RUN TestGCSweepSkipsPendingObjects610=== PAUSE TestGCSweepSkipsPendingObjects611=== RUN TestTombstonedObjectOfferedWithoutWaiting612=== PAUSE TestTombstonedObjectOfferedWithoutWaiting613=== RUN TestGCSweepDeliversEachKeyOnce614=== PAUSE TestGCSweepDeliversEachKeyOnce615=== RUN TestCreatePendingClosureVerifyS3FailureReleasesConnection616=== PAUSE TestCreatePendingClosureVerifyS3FailureReleasesConnection617=== RUN TestForceGCDuringPushOffersSweptObject618=== PAUSE TestForceGCDuringPushOffersSweptObject619=== RUN TestFailedS3DeleteKeepsTombstone620=== PAUSE TestFailedS3DeleteKeepsTombstone621=== RUN TestSweepSkipsRowsLockedByPush622=== PAUSE TestSweepSkipsRowsLockedByPush623=== RUN TestRegisterCompletedObjectNeedsPendingRow624=== PAUSE TestRegisterCompletedObjectNeedsPendingRow625=== RUN TestGetExistingObjectsLocksRows626=== PAUSE TestGetExistingObjectsLocksRows627=== RUN TestSweepRechecksPendingAfterLocking628=== PAUSE TestSweepRechecksPendingAfterLocking629=== RUN TestPushWaitsForSweepHoldingRow630=== PAUSE TestPushWaitsForSweepHoldingRow631=== RUN TestCommitRacingPendingCleanupKeepsObjects632=== PAUSE TestCommitRacingPendingCleanupKeepsObjects633=== RUN TestGCEndsOnShutdown634=== PAUSE TestGCEndsOnShutdown635=== RUN TestGCTaskStore_StartNew636=== PAUSE TestGCTaskStore_StartNew637=== RUN TestGCTaskStore_DeduplicateSameParams638=== PAUSE TestGCTaskStore_DeduplicateSameParams639=== RUN TestGCTaskStore_ConflictDifferentParams640=== PAUSE TestGCTaskStore_ConflictDifferentParams641=== RUN TestGCTaskStore_GetEmpty642=== PAUSE TestGCTaskStore_GetEmpty643=== RUN TestGCTaskStore_GetReturnsLatest644=== PAUSE TestGCTaskStore_GetReturnsLatest645=== RUN TestGCTaskStore_CompletedAllowsNewTask646=== PAUSE TestGCTaskStore_CompletedAllowsNewTask647=== RUN TestGCTaskStore_PhaseUpdates648=== PAUSE TestGCTaskStore_PhaseUpdates649=== RUN TestGCTaskStore_Fail650=== PAUSE TestGCTaskStore_Fail651=== RUN TestGracefulShutdownDrainsInflight652=== PAUSE TestGracefulShutdownDrainsInflight653=== RUN TestService_healthCheckHandler654=== PAUSE TestService_healthCheckHandler655=== RUN TestService_readinessHandler656=== PAUSE TestService_readinessHandler657=== RUN TestGenerateLandingPage658=== PAUSE TestGenerateLandingPage659=== RUN TestEnsureAbortUploadsRule660=== PAUSE TestEnsureAbortUploadsRule661=== RUN TestCacheConfigHandlerMaxNarSize662=== PAUSE TestCacheConfigHandlerMaxNarSize663=== RUN TestCreatePendingClosureRejectsOversizedNAR664=== PAUSE TestCreatePendingClosureRejectsOversizedNAR665=== RUN TestNARDeduplicationMetadataUploadBug666=== PAUSE TestNARDeduplicationMetadataUploadBug667=== RUN TestMetricsInventory668=== PAUSE TestMetricsInventory669=== RUN TestMetricsCommit670=== PAUSE TestMetricsCommit671=== RUN TestService_NativeMTLS672=== PAUSE TestService_NativeMTLS673=== RUN TestServerTLSConfig674=== PAUSE TestServerTLSConfig675=== RUN TestMultipartCleanup676=== PAUSE TestMultipartCleanup677=== RUN TestMultipartUploadAbortedWhenCancelledBeforeRecorded678=== PAUSE TestMultipartUploadAbortedWhenCancelledBeforeRecorded679=== RUN TestPendingCleanupUsesOneCutoff680=== PAUSE TestPendingCleanupUsesOneCutoff681=== RUN TestObjectStatsTrigger682=== PAUSE TestObjectStatsTrigger683=== RUN TestObjectStatsUpdatedOncePerCommit684=== PAUSE TestObjectStatsUpdatedOncePerCommit685=== RUN TestOrphanedObjectsGC686=== PAUSE TestOrphanedObjectsGC687=== RUN TestOrphanedObjectsGCStressTest688=== PAUSE TestOrphanedObjectsGCStressTest689=== RUN TestCreatePin_ReservedPins690=== PAUSE TestCreatePin_ReservedPins691=== RUN TestPresentReportsOnlyClosureRoots692=== PAUSE TestPresentReportsOnlyClosureRoots693=== RUN TestPresentRefreshesOnlyStaleClosures694=== PAUSE TestPresentRefreshesOnlyStaleClosures695=== RUN TestPresentNotReportedWhileGCDeletesClosure696=== PAUSE TestPresentNotReportedWhileGCDeletesClosure697=== RUN TestParseSingleRange698=== PAUSE TestParseSingleRange699=== RUN TestProxyHeadersOnlyTrustedOnSocket7002026/11/12 07:18:48 INFO Starting HTTP server address=127.0.0.1:465457012026/11/12 07:18:48 INFO Starting HTTP server address=proxy.sock7022026/11/12 07:18:48 WARN mTLS auth: subject not in bound subjects subject="CN=someone"7032026/11/12 07:18:48 INFO Shutdown signal received, draining in-flight requests timeout=10s704--- PASS: TestProxyHeadersOnlyTrustedOnSocket (3.10s)705=== RUN TestIsValidCachePath706=== PAUSE TestIsValidCachePath707=== RUN TestReadProxyNarinfo708=== PAUSE TestReadProxyNarinfo709=== RUN TestReadProxyNarinfoAlreadyDecompressed710=== PAUSE TestReadProxyNarinfoAlreadyDecompressed711=== RUN TestReadProxyContentNegotiation712=== PAUSE TestReadProxyContentNegotiation713=== RUN TestReadProxyNarStreaming714=== PAUSE TestReadProxyNarStreaming715=== RUN TestReadProxy404716=== PAUSE TestReadProxy404717=== RUN TestReadProxyInvalidPath718=== PAUSE TestReadProxyInvalidPath719=== RUN TestReadProxyHead720=== PAUSE TestReadProxyHead721=== RUN TestReadProxyOutlastsServerWriteTimeout722=== PAUSE TestReadProxyOutlastsServerWriteTimeout723=== RUN TestReadProxyConditionalGet724=== PAUSE TestReadProxyConditionalGet725=== RUN TestReadProxyRootRedirectsToIndexHTML726=== PAUSE TestReadProxyRootRedirectsToIndexHTML727=== RUN TestReadProxyDisabled728=== PAUSE TestReadProxyDisabled729=== RUN TestReadRedirectNar730=== PAUSE TestReadRedirectNar731=== RUN TestReadRedirectKeepsNarinfoProxied732=== PAUSE TestReadRedirectKeepsNarinfoProxied733=== RUN TestReadProxyRangeRequest734=== PAUSE TestReadProxyRangeRequest735=== RUN TestReadRedirectUsesPublicS3URL736=== PAUSE TestReadRedirectUsesPublicS3URL737=== RUN TestPush_OverlappingRootsStoreOneRowPerKey738=== PAUSE TestPush_OverlappingRootsStoreOneRowPerKey739=== RUN TestPush_CompleteCommitsEveryRoot740=== PAUSE TestPush_CompleteCommitsEveryRoot741=== RUN TestPush_RepeatedRootCommitsOnce742=== PAUSE TestPush_RepeatedRootCommitsOnce743=== RUN TestPush_CompleteWaitsForCleanupOfThePush744=== PAUSE TestPush_CompleteWaitsForCleanupOfThePush745=== RUN TestPush_RootsAreLockedInByteOrder746=== PAUSE TestPush_RootsAreLockedInByteOrder747=== RUN TestPush_SkippedKeySurvivesGCBeforeCommit748=== PAUSE TestPush_SkippedKeySurvivesGCBeforeCommit749=== RUN TestPush_RejectsBadRequests750=== PAUSE TestPush_RejectsBadRequests751=== RUN TestPush_SignsNarinfosOfItsPendingObjects752=== PAUSE TestPush_SignsNarinfosOfItsPendingObjects753=== RUN TestCommitCanBeRepeatedUntilCleanup754=== PAUSE TestCommitCanBeRepeatedUntilCleanup755=== RUN TestSharedNarBothUploadsComplete756=== PAUSE TestSharedNarBothUploadsComplete757=== RUN TestCompleteMultipartUpload_ErrorButObjectExists758=== PAUSE TestCompleteMultipartUpload_ErrorButObjectExists759=== RUN TestCompletedNarNotReofferedAcrossClosures760=== PAUSE TestCompletedNarNotReofferedAcrossClosures761=== RUN TestPresignedUploadRegisteredBeforeCommit762=== PAUSE TestPresignedUploadRegisteredBeforeCommit763=== RUN TestService_Rustfstest764=== PAUSE TestService_Rustfstest765=== RUN TestParseSize766=== PAUSE TestParseSize767=== RUN TestSkippedUploadsHandler768=== PAUSE TestSkippedUploadsHandler769=== RUN TestSystemdListenerNotActivated770--- PASS: TestSystemdListenerNotActivated (0.00s)771=== RUN TestWatchdogBeatsWhenHealthy772--- PASS: TestWatchdogBeatsWhenHealthy (0.02s)773=== RUN TestWatchdogSkipsWhenUnhealthy7742026/11/12 07:18:48 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7752026/11/12 07:18:48 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7762026/11/12 07:18:48 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7772026/11/12 07:18:48 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7782026/11/12 07:18:48 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7792026/11/12 07:18:48 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7802026/11/12 07:18:48 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7812026/11/12 07:18:48 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7822026/11/12 07:18:48 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7832026/11/12 07:18:48 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"784--- PASS: TestWatchdogSkipsWhenUnhealthy (0.20s)785=== RUN TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle786=== PAUSE TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle787=== RUN TestProxyWriteTimeout788=== PAUSE TestProxyWriteTimeout789=== RUN TestPendingClosureWriteTimeout790=== PAUSE TestPendingClosureWriteTimeout791=== RUN TestIsValidUploadKey792=== PAUSE TestIsValidUploadKey793=== RUN TestUploadHandlersRejectInvalidKeys794=== PAUSE TestUploadHandlersRejectInvalidKeys795=== RUN TestUploadHandlersRejectOversizedBody796=== PAUSE TestUploadHandlersRejectOversizedBody797=== RUN TestService_cleanupPendingClosuresHandler798=== PAUSE TestService_cleanupPendingClosuresHandler799=== RUN TestService_createPendingClosureHandler800=== PAUSE TestService_createPendingClosureHandler801=== RUN TestService_verifyS3Integrity802=== PAUSE TestService_verifyS3Integrity803=== RUN TestService_verifyS3ReuploadsStaleNarinfo804=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo805=== RUN TestCompleteMultipartUnregistered806=== PAUSE TestCompleteMultipartUnregistered807=== RUN TestCreatePendingClosure_SmallNARUsesSimplePUT808=== PAUSE TestCreatePendingClosure_SmallNARUsesSimplePUT809=== CONT TestAbortSucceeded810=== CONT TestCreatePendingClosure_SmallNARUsesSimplePUT811=== CONT TestCompleteMultipartUnregistered812=== CONT TestService_verifyS3ReuploadsStaleNarinfo813=== CONT TestService_verifyS3Integrity814=== CONT TestService_createPendingClosureHandler815=== CONT TestService_cleanupPendingClosuresHandler816=== CONT TestUploadHandlersRejectOversizedBody817=== CONT TestUploadHandlersRejectInvalidKeys818=== CONT TestIsValidUploadKey819=== CONT TestPendingClosureWriteTimeout820=== CONT TestProxyWriteTimeout821=== CONT TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle822=== CONT TestSkippedUploadsHandler823=== CONT TestParseSize824--- PASS: TestParseSize (0.00s)825=== CONT TestPush_RootsAreLockedInByteOrder826=== CONT TestService_Rustfstest827--- PASS: TestAbortSucceeded (0.11s)828=== CONT TestPresignedUploadRegisteredBeforeCommit829=== CONT TestCompletedNarNotReofferedAcrossClosures830=== CONT TestCompleteMultipartUpload_ErrorButObjectExists831=== CONT TestSharedNarBothUploadsComplete832=== CONT TestCommitCanBeRepeatedUntilCleanup833=== RUN TestCommitCanBeRepeatedUntilCleanup/push834=== PAUSE TestCommitCanBeRepeatedUntilCleanup/push835=== RUN TestCommitCanBeRepeatedUntilCleanup/pending_closure836=== PAUSE TestCommitCanBeRepeatedUntilCleanup/pending_closure837=== CONT TestPush_CompleteWaitsForCleanupOfThePush838=== CONT TestPush_SignsNarinfosOfItsPendingObjects839=== CONT TestPush_RejectsBadRequests840=== CONT TestPush_SkippedKeySurvivesGCBeforeCommit841=== RUN TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed842=== CONT TestPush_RepeatedRootCommitsOnce843=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info844=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info845=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal846=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal847=== RUN TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key848=== PAUSE TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key849=== RUN TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key850=== PAUSE TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key851=== RUN TestPendingClosureWriteTimeout/empty852=== PAUSE TestPendingClosureWriteTimeout/empty853=== RUN TestPendingClosureWriteTimeout/400_objects854=== PAUSE TestPendingClosureWriteTimeout/400_objects855=== RUN TestPendingClosureWriteTimeout/670k_objects856=== PAUSE TestPendingClosureWriteTimeout/670k_objects857=== RUN TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline858=== PAUSE TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline859=== CONT TestPush_CompleteCommitsEveryRoot860=== RUN TestProxyWriteTimeout/narinfo861=== PAUSE TestProxyWriteTimeout/narinfo862=== RUN TestProxyWriteTimeout/1_GiB_nar863=== PAUSE TestProxyWriteTimeout/1_GiB_nar864=== RUN TestProxyWriteTimeout/10_GiB_nar865=== PAUSE TestProxyWriteTimeout/10_GiB_nar866=== RUN TestProxyWriteTimeout/unknown_size867=== PAUSE TestProxyWriteTimeout/unknown_size868=== CONT TestPush_OverlappingRootsStoreOneRowPerKey869=== RUN TestIsValidUploadKey/narinfo870=== PAUSE TestIsValidUploadKey/narinfo871=== RUN TestIsValidUploadKey/nar_zst872=== PAUSE TestIsValidUploadKey/nar_zst873=== RUN TestIsValidUploadKey/nar_xz874=== PAUSE TestIsValidUploadKey/nar_xz875=== RUN TestIsValidUploadKey/nar_plain876=== PAUSE TestIsValidUploadKey/nar_plain877=== RUN TestIsValidUploadKey/listing878=== PAUSE TestIsValidUploadKey/listing879=== RUN TestIsValidUploadKey/build_log880=== PAUSE TestIsValidUploadKey/build_log881=== RUN TestIsValidUploadKey/build_log_home-manager_file882=== PAUSE TestIsValidUploadKey/build_log_home-manager_file883=== RUN TestIsValidUploadKey/build_log_plus_in_name884=== PAUSE TestIsValidUploadKey/build_log_plus_in_name885=== RUN TestIsValidUploadKey/build_log_question_mark886=== PAUSE TestIsValidUploadKey/build_log_question_mark887=== CONT TestReadRedirectUsesPublicS3URL888=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed889=== RUN TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed890=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed891=== RUN TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost892=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost893=== RUN TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed894=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed895=== RUN TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed896=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed897=== CONT TestReadProxyRangeRequest898=== RUN TestUploadHandlersRejectOversizedBody/complete_multipart899=== PAUSE TestUploadHandlersRejectOversizedBody/complete_multipart900=== RUN TestUploadHandlersRejectOversizedBody/request_more_parts901=== PAUSE TestUploadHandlersRejectOversizedBody/request_more_parts902=== RUN TestUploadHandlersRejectOversizedBody/create_pending_closure903=== PAUSE TestUploadHandlersRejectOversizedBody/create_pending_closure904=== CONT TestReadRedirectKeepsNarinfoProxied905=== RUN TestIsValidUploadKey/build_log_equals906=== PAUSE TestIsValidUploadKey/build_log_equals907=== RUN TestIsValidUploadKey/realisation908=== PAUSE TestIsValidUploadKey/realisation909=== RUN TestIsValidUploadKey/realisation_plus_in_output910=== PAUSE TestIsValidUploadKey/realisation_plus_in_output911=== RUN TestIsValidUploadKey/nix-cache-info912=== PAUSE TestIsValidUploadKey/nix-cache-info913=== RUN TestIsValidUploadKey/index.html914=== PAUSE TestIsValidUploadKey/index.html915=== RUN TestIsValidUploadKey/narinfo_key,_nar_type916=== PAUSE TestIsValidUploadKey/narinfo_key,_nar_type917=== RUN TestIsValidUploadKey/nar_key,_narinfo_type918=== PAUSE TestIsValidUploadKey/nar_key,_narinfo_type919=== RUN TestIsValidUploadKey/listing_key,_narinfo_type920=== PAUSE TestIsValidUploadKey/listing_key,_narinfo_type921=== RUN TestIsValidUploadKey/traversal922=== PAUSE TestIsValidUploadKey/traversal923=== RUN TestIsValidUploadKey/traversal_nar924=== PAUSE TestIsValidUploadKey/traversal_nar925=== RUN TestIsValidUploadKey/absolute926=== PAUSE TestIsValidUploadKey/absolute927=== RUN TestIsValidUploadKey/empty_key928=== PAUSE TestIsValidUploadKey/empty_key929=== RUN TestIsValidUploadKey/unknown_type930=== PAUSE TestIsValidUploadKey/unknown_type931=== CONT TestReadRedirectNar9322026/11/12 07:18:49 INFO Client skipped oversized paths paths=3 nar_bytes=5000000000933=== CONT TestReadProxyDisabled934--- PASS: TestSkippedUploadsHandler (1.98s)9352026/11/12 07:18:50 INFO Received push request method=POST path=/api/pushes936--- PASS: TestReadRedirectUsesPublicS3URL (2.19s)937=== CONT TestReadProxyRootRedirectsToIndexHTML938--- PASS: TestReadRedirectNar (2.03s)939=== CONT TestReadProxyConditionalGet940--- PASS: TestReadProxyRangeRequest (2.06s)941=== CONT TestReadProxyOutlastsServerWriteTimeout942--- PASS: TestReadRedirectKeepsNarinfoProxied (2.06s)943=== CONT TestReadProxyHead9442026/11/12 07:18:51 INFO Received uploads request method=POST path=/api/pending_closures9452026/11/12 07:18:51 INFO Received push request method=POST path=/api/pushes946--- PASS: TestPush_OverlappingRootsStoreOneRowPerKey (2.32s)947=== CONT TestReadProxyInvalidPath9482026/11/12 07:18:51 INFO Received push request method=POST path=/api/pushes9492026/11/12 07:18:51 INFO Received complete push request method=POST path=/api/pushes/1/complete9502026/11/12 07:18:51 INFO Received uploads request method=POST path=/api/pending_closures9512026/11/12 07:18:51 INFO Received complete multipart upload request method=POST path=/api/multipart/complete9522026/11/12 07:18:51 INFO Received uploads request method=POST path=/api/pending_closures9532026/11/12 07:18:51 INFO Received uploads request method=POST path=/api/pending_closures9542026/11/12 07:18:51 INFO Received uploads request method=POST path=/api/pending_closures9552026/11/12 07:18:51 INFO Received push request method=POST path=/api/pushes9562026/11/12 07:18:51 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign9572026/11/12 07:18:51 INFO Signed narinfos id=1 count=19582026/11/12 07:18:51 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign9592026/11/12 07:18:51 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign9602026/11/12 07:18:51 ERROR Failed to sign narinfos signer="index 1" error="signing key cannot be nil"9612026/11/12 07:18:51 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign9622026/11/12 07:18:51 ERROR Failed to sign narinfos signer=niks3-test-1 error="signer failed"9632026/11/12 07:18:51 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign9642026/11/12 07:18:51 ERROR Signer returned unexpected signature count signer=niks3-test-1 got=0 want=19652026/11/12 07:18:51 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign9662026/11/12 07:18:51 ERROR Signer returned unexpected object key signer=niks3-test-1 object_key=unexpected.narinfo967--- PASS: TestPush_SignsNarinfosOfItsPendingObjects (2.46s)968=== CONT TestReadProxy4049692026/11/12 07:18:51 INFO Received uploads request method=POST path=/api/pending_closures9702026/11/12 07:18:51 INFO Completed push id=1 objects=6 duration=107.991595ms971--- PASS: TestPush_CompleteCommitsEveryRoot (2.49s)972=== CONT TestReadProxyNarStreaming9732026-11-12 07:18:51.270 UTC [502] ERROR: could not obtain lock on row in relation "closures"9742026-11-12 07:18:51.270 UTC [502] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT9752026/11/12 07:18:51 INFO Received uploads request method=POST path=/api/pending_closures9762026-11-12 07:18:51.271 UTC [502] ERROR: could not obtain lock on row in relation "closures"9772026-11-12 07:18:51.271 UTC [502] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT9782026-11-12 07:18:51.271 UTC [502] ERROR: could not obtain lock on row in relation "closures"9792026-11-12 07:18:51.271 UTC [502] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT9802026-11-12 07:18:51.272 UTC [502] ERROR: could not obtain lock on row in relation "closures"9812026-11-12 07:18:51.272 UTC [502] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT9822026-11-12 07:18:51.272 UTC [502] ERROR: could not obtain lock on row in relation "closures"9832026-11-12 07:18:51.272 UTC [502] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT984--- PASS: TestPush_RootsAreLockedInByteOrder (2.51s)985=== CONT TestReadProxyContentNegotiation9862026/11/12 07:18:53 INFO Received uploads request method=POST path=/api/pending_closures9872026/11/12 07:18:53 INFO Received uploads request method=POST path=/api/pending_closures9882026/11/12 07:18:53 INFO Received complete multipart upload request method=POST path=/api/multipart/complete9892026/11/12 07:18:53 INFO Completed multipart upload object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst upload_id=MjRjOTUzOWEtOWQyOS00MWUwLWJkMDItODUyZjM4NjYzZjc5LmQzYzVmZmM0LTAzYWEtNDZmOS1iOGMwLTY4MTMxMTEyZWRiNXgxNzk0NDY3OTMxMTk3MzMzNjQ3 parts=109902026/11/12 07:18:53 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete9912026/11/12 07:18:53 INFO Completed upload id=1 objects=2 duration=9.559047ms9922026/11/12 07:18:53 INFO Received get closure request method=GET path=/api/closures/000000000000000000000000000000009932026/11/12 07:18:53 INFO Received uploads request method=POST path=/api/pending_closures9942026/11/12 07:18:53 INFO Starting cleanup of old closures method=DELETE path=/api/closures9952026/11/12 07:18:53 INFO Aborted multipart uploads count=0 kept=09962026/11/12 07:18:53 INFO Received complete multipart upload request method=POST path=/api/multipart/complete9972026/11/12 07:18:53 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0100000000000000000000.nar.zst upload_id=MjRjOTUzOWEtOWQyOS00MWUwLWJkMDItODUyZjM4NjYzZjc5LmYxNmFjNWYzLWI3NmUtNDQ0MC1iNTViLTA1MTQzMzQ5NzBiY3gxNzk0NDY3OTMzMTQ2MTMyMTI1 parts=129982026/11/12 07:18:53 INFO Received complete multipart upload request method=POST path=/api/multipart/complete9992026/11/12 07:18:53 WARN CompleteMultipartUpload errored but object exists; treating as success error="The specified key does not exist." object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=MjRjOTUzOWEtOWQyOS00MWUwLWJkMDItODUyZjM4NjYzZjc5LjVhYTY1YmU5LWE1NjMtNDFlYi05NjA2LWQ5Nzg3NjlkY2UzZHgxNzk0NDY3OTMxMjg3MjkyMzk210002026/11/12 07:18:53 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=MjRjOTUzOWEtOWQyOS00MWUwLWJkMDItODUyZjM4NjYzZjc5LjVhYTY1YmU5LWE1NjMtNDFlYi05NjA2LWQ5Nzg3NjlkY2UzZHgxNzk0NDY3OTMxMjg3MjkyMzk2 parts=11001=== CONT TestReadProxyNarinfoAlreadyDecompressed1002--- PASS: TestCompleteMultipartUpload_ErrorButObjectExists (5.14s)10032026/11/12 07:18:53 INFO Received push request method=POST path=/api/pushes1004--- PASS: TestService_Rustfstest (5.22s)10052026/11/12 07:18:53 INFO Received complete push request method=POST path=/api/pushes/1/complete1006=== CONT TestReadProxyNarinfo10072026/11/12 07:18:54 INFO Received uploads request method=POST path=/api/pending_closures10082026/11/12 07:18:54 INFO Received push request method=POST path=/api/pushes10092026/11/12 07:18:54 INFO Received complete push request method=POST path=/api/pushes/1/complete10102026/11/12 07:18:54 INFO Completed push id=1 objects=4 duration=11.499086ms1011--- PASS: TestPush_RepeatedRootCommitsOnce (5.31s)1012=== CONT TestIsValidCachePath1013=== RUN TestIsValidCachePath/narinfo1014=== PAUSE TestIsValidCachePath/narinfo1015=== RUN TestIsValidCachePath/narinfo_all_nix_base32_chars1016=== PAUSE TestIsValidCachePath/narinfo_all_nix_base32_chars1017=== RUN TestIsValidCachePath/nar_zst1018=== PAUSE TestIsValidCachePath/nar_zst1019=== RUN TestIsValidCachePath/nar_xz1020=== PAUSE TestIsValidCachePath/nar_xz1021=== RUN TestIsValidCachePath/nar_bz21022=== PAUSE TestIsValidCachePath/nar_bz21023=== RUN TestIsValidCachePath/nar_uncompressed1024=== PAUSE TestIsValidCachePath/nar_uncompressed1025=== RUN TestIsValidCachePath/ls1026=== PAUSE TestIsValidCachePath/ls1027=== RUN TestIsValidCachePath/log1028=== PAUSE TestIsValidCachePath/log1029=== RUN TestIsValidCachePath/realisation1030=== PAUSE TestIsValidCachePath/realisation1031=== RUN TestIsValidCachePath/nix-cache-info1032=== PAUSE TestIsValidCachePath/nix-cache-info1033=== RUN TestIsValidCachePath/index.html1034=== PAUSE TestIsValidCachePath/index.html1035=== RUN TestIsValidCachePath/traversal_parent1036=== PAUSE TestIsValidCachePath/traversal_parent1037=== RUN TestIsValidCachePath/traversal_in_middle1038=== PAUSE TestIsValidCachePath/traversal_in_middle1039=== RUN TestIsValidCachePath/invalid_char_e1040=== PAUSE TestIsValidCachePath/invalid_char_e1041=== RUN TestIsValidCachePath/invalid_char_u1042=== PAUSE TestIsValidCachePath/invalid_char_u1043=== RUN TestIsValidCachePath/random_path1044=== PAUSE TestIsValidCachePath/random_path1045=== RUN TestIsValidCachePath/empty1046=== PAUSE TestIsValidCachePath/empty1047=== RUN TestIsValidCachePath/leading_slash1048=== PAUSE TestIsValidCachePath/leading_slash1049=== RUN TestIsValidCachePath/wrong_extension1050=== PAUSE TestIsValidCachePath/wrong_extension1051=== RUN TestIsValidCachePath/short_hash1052=== PAUSE TestIsValidCachePath/short_hash1053=== CONT TestParseSingleRange1054=== RUN TestParseSingleRange/none1055=== PAUSE TestParseSingleRange/none1056=== RUN TestParseSingleRange/unknown_unit1057=== PAUSE TestParseSingleRange/unknown_unit1058=== RUN TestParseSingleRange/multi-range_ignored1059=== PAUSE TestParseSingleRange/multi-range_ignored1060=== RUN TestParseSingleRange/malformed_no_dash1061=== PAUSE TestParseSingleRange/malformed_no_dash1062=== RUN TestParseSingleRange/malformed_both_empty1063=== PAUSE TestParseSingleRange/malformed_both_empty1064=== RUN TestParseSingleRange/malformed_end_before_start1065=== PAUSE TestParseSingleRange/malformed_end_before_start1066=== RUN TestParseSingleRange/closed1067=== PAUSE TestParseSingleRange/closed1068=== RUN TestParseSingleRange/open-ended1069=== PAUSE TestParseSingleRange/open-ended1070=== RUN TestParseSingleRange/end_clamped_to_size1071=== PAUSE TestParseSingleRange/end_clamped_to_size1072=== RUN TestParseSingleRange/suffix1073=== PAUSE TestParseSingleRange/suffix1074=== RUN TestParseSingleRange/suffix_exceeds_size1075=== PAUSE TestParseSingleRange/suffix_exceeds_size1076=== RUN TestParseSingleRange/single_byte1077=== PAUSE TestParseSingleRange/single_byte1078=== RUN TestParseSingleRange/start_past_EOF1079=== PAUSE TestParseSingleRange/start_past_EOF1080=== RUN TestParseSingleRange/start_far_past_EOF1081=== PAUSE TestParseSingleRange/start_far_past_EOF1082=== CONT TestPresentNotReportedWhileGCDeletesClosure1083=== RUN TestPresentNotReportedWhileGCDeletesClosure/stale1084=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/stale1085=== RUN TestPresentNotReportedWhileGCDeletesClosure/fresh1086=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/fresh1087=== CONT TestPresentRefreshesOnlyStaleClosures10882026/11/12 07:18:54 INFO Registered completed upload object_key=nar/jjjjjjjjjjjjjjjjjjjjjjjjjjjjjj0100000000000000000000.nar.zst10892026/11/12 07:18:54 INFO Received uploads request method=POST path=/api/pending_closures1090--- PASS: TestPresignedUploadRegisteredBeforeCommit (5.33s)1091=== CONT TestPresentReportsOnlyClosureRoots10922026/11/12 07:18:54 INFO Received uploads request method=POST path=/api/pending_closures1093--- PASS: TestCreatePendingClosure_SmallNARUsesSimplePUT (5.46s)1094=== CONT TestCreatePin_ReservedPins10952026/11/12 07:18:54 INFO Received complete multipart upload request method=POST path=/api/multipart/complete10962026/11/12 07:18:54 ERROR Failed to get multipart upload error="no rows in result set" upload_id=does-not-exist object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst1097--- PASS: TestCompleteMultipartUnregistered (5.48s)1098=== CONT TestOrphanedObjectsGCStressTest10992026/11/12 07:18:54 INFO Received push request method=POST path=/api/pushes11002026/11/12 07:18:54 INFO Received complete multipart upload request method=POST path=/api/multipart/complete11012026/11/12 07:18:54 INFO Received complete multipart upload request method=POST path=/api/multipart/complete1102=== RUN TestPush_RejectsBadRequests/bad_root1103=== PAUSE TestPush_RejectsBadRequests/bad_root1104=== RUN TestPush_RejectsBadRequests/root_not_in_objects1105=== PAUSE TestPush_RejectsBadRequests/root_not_in_objects1106=== RUN TestPush_RejectsBadRequests/no_roots1107=== PAUSE TestPush_RejectsBadRequests/no_roots1108=== RUN TestPush_RejectsBadRequests/no_objects1109=== PAUSE TestPush_RejectsBadRequests/no_objects1110=== CONT TestOrphanedObjectsGC11112026/11/12 07:18:54 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:38081/oidc1112--- PASS: TestReadProxyDisabled (3.61s)1113=== CONT TestObjectStatsUpdatedOncePerCommit11142026/11/12 07:18:54 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst upload_id=MjRjOTUzOWEtOWQyOS00MWUwLWJkMDItODUyZjM4NjYzZjc5LmUzNWViNDVlLTA0MzUtNDM3NS05MTM5LWRkOGVmZjYxMGUxMHgxNzk0NDY3OTMxMjUyMTM2NDcy parts=1011152026/11/12 07:18:54 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete11162026/11/12 07:18:54 INFO Completed multipart upload object_key=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhh0100000000000000000000.nar.zst upload_id=MjRjOTUzOWEtOWQyOS00MWUwLWJkMDItODUyZjM4NjYzZjc5LjIyZjIxNmY2LWM4ZmUtNDM4ZC05MTg2LTg2YjQyYWVhN2E1NXgxNzk0NDY3OTMxMTY5OTMyMTM1 parts=1211172026/11/12 07:18:54 INFO Received uploads request method=POST path=/api/pending_closures11182026/11/12 07:18:54 INFO Completed upload id=1 objects=2 duration=7.128646ms1119--- PASS: TestCompletedNarNotReofferedAcrossClosures (5.52s)1120=== CONT TestObjectStatsTrigger11212026/11/12 07:18:54 INFO Received uploads request method=POST path=/api/pending_closures11222026/11/12 07:18:54 INFO Received uploads request method=POST path=/api/pending_closures11232026/11/12 07:18:54 INFO Object in database but missing from S3 key=dadb44fdadb44fdadb44fdadb44f0000.narinfo11242026/11/12 07:18:54 WARN Found objects in DB but missing from S3, will re-upload count=111252026/11/12 07:18:54 INFO Received complete multipart upload request method=POST path=/api/multipart/complete1126--- PASS: TestService_verifyS3Integrity (5.74s)1127=== CONT TestPendingCleanupUsesOneCutoff1128--- PASS: TestReadProxyRootRedirectsToIndexHTML (3.43s)1129=== CONT TestMultipartUploadAbortedWhenCancelledBeforeRecorded11302026/11/12 07:18:54 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0100000000000000000000.nar.zst upload_id=MjRjOTUzOWEtOWQyOS00MWUwLWJkMDItODUyZjM4NjYzZjc5LmE3ZjIzZGQ5LTNlYzEtNDYzNC05YmMxLTIwODYxNDYxZTYxOHgxNzk0NDY3OTMzMTY1NjY0OTQ1 parts=121131--- PASS: TestSharedNarBothUploadsComplete (5.64s)1132=== CONT TestMultipartCleanup11332026/11/12 07:18:54 INFO Received cleanup request method=DELETE path=/api/pending_closures11342026/11/12 07:18:54 INFO Aborted multipart uploads count=0 kept=011352026/11/12 07:18:54 INFO Received uploads request method=POST path=/api/pending_closures11362026/11/12 07:18:54 INFO Received cleanup request method=DELETE path=/api/pending_closures11372026/11/12 07:18:54 INFO Aborted multipart uploads count=1 kept=011382026/11/12 07:18:54 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete11392026-11-12 07:18:54.469 UTC [457] ERROR: Closure does not exist: id=111402026-11-12 07:18:54.469 UTC [457] CONTEXT: PL/pgSQL function commit_pending_closure(bigint) line 17 at RAISE11412026-11-12 07:18:54.469 UTC [457] STATEMENT: -- name: CommitPendingClosure :one1142 SELECT commit_pending_closure($1::bigint)::bigint1143 1144--- PASS: TestService_cleanupPendingClosuresHandler (5.80s)1145=== CONT TestServerTLSConfig1146=== RUN TestServerTLSConfig/no_client_CA1147=== PAUSE TestServerTLSConfig/no_client_CA1148=== RUN TestServerTLSConfig/missing_CA_file1149=== PAUSE TestServerTLSConfig/missing_CA_file1150=== RUN TestServerTLSConfig/not_a_PEM_file1151=== PAUSE TestServerTLSConfig/not_a_PEM_file1152=== CONT TestService_NativeMTLS1153--- PASS: TestReadProxyConditionalGet (3.51s)1154=== CONT TestMetricsCommit1155--- PASS: TestReadProxyHead (3.50s)1156=== CONT TestMetricsInventory1157--- PASS: TestReadProxyInvalidPath (3.41s)1158=== CONT TestNARDeduplicationMetadataUploadBug11592026-11-12 07:18:54.537 UTC [565] ERROR: Push does not exist: id=111602026-11-12 07:18:54.537 UTC [565] CONTEXT: PL/pgSQL function commit_push(bigint) line 15 at RAISE11612026-11-12 07:18:54.537 UTC [565] STATEMENT: SELECT commit_push($1)1162--- PASS: TestPush_CompleteWaitsForCleanupOfThePush (5.76s)1163=== CONT TestCreatePendingClosureRejectsOversizedNAR11642026/11/12 07:18:54 INFO Received uploads request method=POST path=/api/pending_closures1165--- PASS: TestCreatePendingClosureRejectsOversizedNAR (0.00s)1166=== CONT TestCacheConfigHandlerMaxNarSize1167--- PASS: TestCacheConfigHandlerMaxNarSize (0.00s)1168=== CONT TestEnsureAbortUploadsRule1169--- PASS: TestReadProxyNarinfoAlreadyDecompressed (0.74s)1170=== CONT TestGenerateLandingPage1171--- PASS: TestGenerateLandingPage (0.01s)1172=== CONT TestService_readinessHandler1173--- PASS: TestReadProxyContentNegotiation (3.41s)1174=== CONT TestService_healthCheckHandler11752026/11/12 07:18:54 INFO Received uploads request method=POST path=/api/pending_closures1176--- PASS: TestReadProxy404 (3.50s)1177=== CONT TestGracefulShutdownDrainsInflight11782026/11/12 07:18:54 INFO Starting HTTP server address=127.0.0.1:3382911792026/11/12 07:18:54 INFO Shutdown signal received, draining in-flight requests timeout=10s11802026/11/12 07:18:54 INFO Completed push id=1 objects=4 duration=761.385917ms11812026/11/12 07:18:54 INFO Received push request method=POST path=/api/pushes11822026/11/12 07:18:54 INFO Received uploads request method=POST path=/api/pending_closures11832026/11/12 07:18:54 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=011842026/11/12 07:18:54 INFO Received uploads request method=POST path=/api/pending_closures11852026/11/12 07:18:54 INFO Vacuumed table table=pending_closures11862026/11/12 07:18:54 INFO Vacuumed table table=commit_receipts11872026/11/12 07:18:54 ERROR Failed to decode narinfo key=3hcdxyjf9yiq7qf3i4548drb6sjmwa1v.narinfo limit=16777216 error=<nil>1188--- PASS: TestReadProxyNarinfo (0.79s)1189=== CONT TestGCTaskStore_Fail1190--- PASS: TestGCTaskStore_Fail (0.00s)1191=== CONT TestGCTaskStore_PhaseUpdates1192--- PASS: TestGCTaskStore_PhaseUpdates (0.00s)1193=== CONT TestGCTaskStore_CompletedAllowsNewTask1194--- PASS: TestGCTaskStore_CompletedAllowsNewTask (0.00s)1195=== CONT TestGCTaskStore_GetReturnsLatest1196--- PASS: TestGCTaskStore_GetReturnsLatest (0.00s)1197=== CONT TestGCTaskStore_GetEmpty1198--- PASS: TestGCTaskStore_GetEmpty (0.00s)1199=== CONT TestGCTaskStore_ConflictDifferentParams1200--- PASS: TestGCTaskStore_ConflictDifferentParams (0.00s)1201=== CONT TestGCTaskStore_DeduplicateSameParams1202--- PASS: TestGCTaskStore_DeduplicateSameParams (0.00s)1203=== CONT TestGCTaskStore_StartNew1204--- PASS: TestGCTaskStore_StartNew (0.00s)1205=== CONT TestGCEndsOnShutdown1206=== RUN TestGCEndsOnShutdown/before_the_run1207=== PAUSE TestGCEndsOnShutdown/before_the_run1208=== RUN TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete1209=== PAUSE TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete1210=== RUN TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark1211=== PAUSE TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark1212=== CONT TestCommitRacingPendingCleanupKeepsObjects12132026/11/12 07:18:54 INFO Vacuumed table table=pending_objects12142026/11/12 07:18:54 INFO Received cleanup request method=DELETE path=/api/pending_closures12152026/11/12 07:18:54 INFO Vacuumed table table=multipart_uploads12162026/11/12 07:18:54 INFO Vacuumed table table=closures12172026/11/12 07:18:54 INFO Vacuumed table table=objects12182026/11/12 07:18:54 WARN mTLS auth: subject not in bound subjects subject="CN=reader"12192026/11/12 07:18:54 WARN mTLS auth: subject not in bound subjects subject="CN=reader"1220--- PASS: TestService_NativeMTLS (0.33s)1221=== CONT TestPushWaitsForSweepHoldingRow1222--- PASS: TestGracefulShutdownDrainsInflight (0.07s)1223=== CONT TestSweepRechecksPendingAfterLocking12242026/11/12 07:18:54 INFO Received get closure request method=GET path=/api/closures/000000000000000000000000000000001225--- PASS: TestService_createPendingClosureHandler (6.14s)1226=== CONT TestGetExistingObjectsLocksRows12272026/11/12 07:18:54 INFO Aborted multipart uploads count=0 kept=012282026/11/12 07:18:54 WARN Force mode enabled - objects will be deleted immediately without grace period12292026/11/12 07:18:54 INFO Received cleanup request method=DELETE path=/api/pending_closures12302026/11/12 07:18:54 INFO Received uploads request method=POST path=/api/pending_closures12312026/11/12 07:18:54 WARN Failed to abort upload, keeping its closure key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst error="Get \"http://127.0.0.1:1/bucket37/?location=\": dial tcp 127.0.0.1:1: connect: connection refused" code=""12322026/11/12 07:18:54 INFO Aborted multipart uploads count=0 kept=112332026/11/12 07:18:54 INFO Received cleanup request method=DELETE path=/api/pending_closures12342026/11/12 07:18:54 INFO Aborted multipart uploads count=1 kept=01235--- PASS: TestMultipartCleanup (0.43s)1236=== CONT TestRegisterCompletedObjectNeedsPendingRow1237--- PASS: TestMultipartUploadAbortedWhenCancelledBeforeRecorded (0.46s)1238=== CONT TestSweepSkipsRowsLockedByPush12392026/11/12 07:18:54 INFO Received push request method=POST path=/api/pushes12402026/11/12 07:18:54 INFO Received complete push request method=POST path=/api/pushes/1/complete12412026/11/12 07:18:54 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=2 objects-deleted-after-grace-period=2 objects-failed-to-delete=012422026/11/12 07:18:54 INFO Vacuumed table table=pending_closures12432026/11/12 07:18:54 INFO Completed push id=1 objects=4 duration=32.825322ms12442026/11/12 07:18:54 INFO Vacuumed table table=commit_receipts1245--- PASS: TestMetricsInventory (0.45s)1246=== CONT TestFailedS3DeleteKeepsTombstone12472026/11/12 07:18:54 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete12482026/11/12 07:18:54 INFO Vacuumed table table=pending_objects12492026/11/12 07:18:54 INFO Completed upload id=2 objects=2 duration=2.274261ms12502026/11/12 07:18:54 INFO Vacuumed table table=multipart_uploads1251--- PASS: TestMetricsCommit (0.46s)1252=== CONT TestForceGCDuringPushOffersSweptObject1253=== RUN TestForceGCDuringPushOffersSweptObject/before_pending_rows1254=== PAUSE TestForceGCDuringPushOffersSweptObject/before_pending_rows1255=== RUN TestForceGCDuringPushOffersSweptObject/after_presence_check1256=== PAUSE TestForceGCDuringPushOffersSweptObject/after_presence_check1257=== CONT TestCreatePendingClosureVerifyS3FailureReleasesConnection12582026/11/12 07:18:54 INFO Vacuumed table table=closures12592026/11/12 07:18:54 INFO Vacuumed table table=objects12602026/11/12 07:18:54 INFO Received complete push request method=POST path=/api/pushes/2/complete12612026/11/12 07:18:54 INFO Completed push id=2 objects=4 duration=4.115558ms1262--- PASS: TestPush_SkippedKeySurvivesGCBeforeCommit (6.20s)1263=== CONT TestGCSweepDeliversEachKeyOnce12642026/11/12 07:18:54 INFO Installed lifecycle rule aborting incomplete NAR uploads bucket=bucket44 days=21265--- PASS: TestEnsureAbortUploadsRule (0.46s)1266=== CONT TestTombstonedObjectOfferedWithoutWaiting1267--- PASS: TestObjectStatsUpdatedOncePerCommit (0.79s)1268=== CONT TestGCSweepSkipsPendingObjects1269--- PASS: TestObjectStatsTrigger (0.80s)1270=== CONT TestDeduplicatedObjectsRecordedAsPending1271--- PASS: TestReadProxyNarStreaming (3.87s)1272=== CONT TestPushDedupSurvivesConcurrentGC12732026/11/12 07:18:55 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux12742026/11/12 07:18:55 WARN Refused reserved pin name=worker-x86_64-linux12752026/11/12 07:18:55 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux12762026/11/12 07:18:55 INFO Received create pin request method=POST path=/api/pins/my-app12772026/11/12 07:18:55 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux1278--- PASS: TestCreatePin_ReservedPins (1.04s)1279=== CONT TestGCMetrics1280=== NAME TestNARDeduplicationMetadataUploadBug1281 metadata_upload_test.go:48: First store path: /build/TestNARDeduplicationMetadataUploadBug2405511238/001/store/hm6gqx2g8gil3vw02a5kxrvs0bcdf717-file1.txt1282--- PASS: TestPresentRefreshesOnlyStaleClosures (1.11s)1283=== CONT TestGCBugBareHashReferences1284--- PASS: TestPresentReportsOnlyClosureRoots (1.14s)1285=== CONT TestGCAdvisoryLockBlocksConcurrentRun12862026/11/12 07:18:55 WARN readiness check failed error="closed pool"1287--- PASS: TestService_readinessHandler (0.62s)1288=== CONT TestLeadEndsOnShutdown1289--- PASS: TestService_healthCheckHandler (0.62s)1290=== CONT TestLeadGenerationGrowsWithEveryGrant12912026/11/12 07:18:55 INFO Aborted multipart uploads count=0 kept=012922026/11/12 07:18:55 INFO Received uploads request method=POST path=/api/pending_closures12932026/11/12 07:18:55 INFO Aborted multipart uploads count=0 kept=01294--- PASS: TestGetExistingObjectsLocksRows (0.63s)1295=== CONT TestLeadIncumbentAnnouncesAtOnce1296--- PASS: TestRegisterCompletedObjectNeedsPendingRow (0.59s)1297=== CONT TestLeadUnannouncedStandbyYieldsToIncumbent12982026/11/12 07:18:55 WARN Force mode enabled - objects will be deleted immediately without grace period12992026/11/12 07:18:55 WARN Force mode enabled - objects will be deleted immediately without grace period1300=== NAME TestOrphanedObjectsGC1301 orphaned_objects_gc_test.go:287: GC Test Summary:1302 orphaned_objects_gc_test.go:288: - Kept: 2 objects from closure A1303 orphaned_objects_gc_test.go:289: - Deleted: 2 objects from closure B1304 orphaned_objects_gc_test.go:290: - Deleted: 6 orphaned chain objects (X1->X2->X3)1305 orphaned_objects_gc_test.go:291: - Deleted: 2 orphaned single objects (Y)1306 orphaned_objects_gc_test.go:292: - Total deleted: 10 objects1307--- PASS: TestOrphanedObjectsGC (1.23s)1308=== CONT TestLeadElectsOneAndHandsOver13092026/11/12 07:18:55 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=013102026/11/12 07:18:55 INFO Received uploads request method=POST path=/api/pending_closures13112026/11/12 07:18:55 INFO Vacuumed table table=pending_closures13122026/11/12 07:18:55 INFO Vacuumed table table=commit_receipts13132026/11/12 07:18:55 INFO Vacuumed table table=pending_objects13142026/11/12 07:18:55 INFO Vacuumed table table=multipart_uploads13152026/11/12 07:18:55 INFO Received push request method=POST path=/api/pushes13162026/11/12 07:18:55 INFO Vacuumed table table=closures13172026/11/12 07:18:55 INFO Vacuumed table table=objects13182026/11/12 07:18:55 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)13192026/11/12 07:18:55 INFO Uploading hm6gqx2g8gil3vw02a5kxrvs0bcdf717-file1.txt (160B)13202026/11/12 07:18:55 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=013212026/11/12 07:18:55 INFO Vacuumed table table=pending_closures13222026/11/12 07:18:55 INFO Vacuumed table table=commit_receipts13232026/11/12 07:18:55 WARN Failed to register uploaded object key=hm6gqx2g8gil3vw02a5kxrvs0bcdf717.ls error="server returned 404: 404 page not found\n"13242026/11/12 07:18:55 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign13252026/11/12 07:18:55 WARN Failed to register uploaded object key=nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst error="server returned 404: 404 page not found\n"13262026/11/12 07:18:55 INFO Signed narinfos id=1 count=113272026/11/12 07:18:55 INFO Uploading 1 narinfos13282026/11/12 07:18:55 INFO Vacuumed table table=pending_objects13292026/11/12 07:18:55 INFO Vacuumed table table=multipart_uploads13302026/11/12 07:18:55 INFO Vacuumed table table=closures13312026/11/12 07:18:55 INFO Received complete push request method=POST path=/api/pushes/1/complete13322026/11/12 07:18:55 WARN Failed to register uploaded object key=hm6gqx2g8gil3vw02a5kxrvs0bcdf717.narinfo error="server returned 404: 404 page not found\n"13332026/11/12 07:18:55 INFO Vacuumed table table=objects1334--- PASS: TestSweepRechecksPendingAfterLocking (0.72s)1335=== CONT TestConnectWaitsForAPeerMigration13362026/11/12 07:18:55 INFO Completed push id=1 objects=3 duration=9.180485ms13372026/11/12 07:18:55 INFO Upload complete. (127ms)1338=== NAME TestNARDeduplicationMetadataUploadBug1339 metadata_upload_test.go:54: Retrieved narinfo from S3:1340 StorePath: /build/TestNARDeduplicationMetadataUploadBug2405511238/001/store/hm6gqx2g8gil3vw02a5kxrvs0bcdf717-file1.txt1341 URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst1342 Compression: zstd1343 NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1344 NarSize: 1601345 References: 1346 CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1347 metadata_upload_test.go:55: Retrieved .ls file from S3 (compressed size: 77 bytes)1348 metadata_upload_test.go:55: Decompressed .ls content (64 bytes):1349 {"version":1,"root":{"type":"regular","size":44,"narOffset":96}}13502026/11/12 07:18:55 INFO Received uploads request method=POST path=/api/pending_closures13512026/11/12 07:18:55 INFO Aborted multipart uploads count=0 kept=01352--- PASS: TestTombstonedObjectOfferedWithoutWaiting (0.57s)1353=== CONT TestResolveDBConnectionString1354=== RUN TestResolveDBConnectionString/flag_wins1355=== PAUSE TestResolveDBConnectionString/flag_wins1356=== RUN TestResolveDBConnectionString/file_when_flag_empty1357=== PAUSE TestResolveDBConnectionString/file_when_flag_empty1358=== RUN TestResolveDBConnectionString/missing_file_is_an_error1359=== PAUSE TestResolveDBConnectionString/missing_file_is_an_error1360=== RUN TestResolveDBConnectionString/PGHOST_allows_empty1361=== PAUSE TestResolveDBConnectionString/PGHOST_allows_empty1362=== RUN TestResolveDBConnectionString/nothing_configured1363=== PAUSE TestResolveDBConnectionString/nothing_configured1364=== CONT TestConcurrentCommitsSharingObjectsDoNotDeadlock13652026/11/12 07:18:55 WARN Force mode enabled - objects will be deleted immediately without grace period13662026/11/12 07:18:55 INFO Aborted multipart uploads count=0 kept=013672026/11/12 07:18:55 INFO Received uploads request method=POST path=/api/pending_closures1368--- PASS: TestCommitRacingPendingCleanupKeepsObjects (0.81s)1369=== CONT TestDeleteClosuresRacingPins1370=== NAME TestNARDeduplicationMetadataUploadBug1371 metadata_upload_test.go:64: Second store path (same content): /build/TestNARDeduplicationMetadataUploadBug2405511238/001/store/3hcchcnagxnz30fk5bq498frmaar0ni1-file2.txt13722026/11/12 07:18:55 WARN Force mode enabled - objects will be deleted immediately without grace period13732026/11/12 07:18:55 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=013742026/11/12 07:18:55 INFO Vacuumed table table=pending_closures13752026/11/12 07:18:55 INFO Vacuumed table table=commit_receipts13762026/11/12 07:18:55 INFO Vacuumed table table=pending_objects13772026/11/12 07:18:55 INFO Vacuumed table table=multipart_uploads13782026/11/12 07:18:55 INFO Vacuumed table table=closures13792026/11/12 07:18:55 INFO Vacuumed table table=objects1380--- PASS: TestGCSweepDeliversEachKeyOnce (0.68s)1381=== CONT TestClientFallsBackToClosures13822026/11/12 07:18:55 INFO Aborted multipart uploads count=0 kept=013832026/11/12 07:18:55 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."13842026/11/12 07:18:55 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."13852026/11/12 07:18:55 INFO Received uploads request method=POST path=/api/pending_closures1386--- PASS: TestFailedS3DeleteKeepsTombstone (0.71s)1387=== CONT TestClientPushesUseOnePush13882026/11/12 07:18:55 INFO Aborted multipart uploads count=0 kept=013892026/11/12 07:18:55 WARN Force mode enabled - objects will be deleted immediately without grace period13902026/11/12 07:18:55 WARN Force mode enabled - objects will be deleted immediately without grace period13912026/11/12 07:18:55 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=013922026/11/12 07:18:55 INFO Vacuumed table table=pending_closures13932026/11/12 07:18:55 INFO Vacuumed table table=commit_receipts13942026/11/12 07:18:55 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=013952026/11/12 07:18:55 INFO Vacuumed table table=pending_objects13962026/11/12 07:18:55 INFO Vacuumed table table=multipart_uploads13972026/11/12 07:18:55 INFO Vacuumed table table=closures13982026/11/12 07:18:55 INFO Vacuumed table table=objects13992026/11/12 07:18:55 INFO Vacuumed table table=pending_closures14002026/11/12 07:18:55 INFO Vacuumed table table=commit_receipts14012026/11/12 07:18:55 INFO Vacuumed table table=pending_objects14022026/11/12 07:18:55 INFO Vacuumed table table=multipart_uploads14032026/11/12 07:18:55 INFO Vacuumed table table=closures14042026/11/12 07:18:55 INFO Vacuumed table table=objects14052026/11/12 07:18:55 INFO Aborted multipart uploads count=0 kept=014062026/11/12 07:18:55 INFO Received uploads request method=POST path=/api/pending_closures1407--- PASS: TestGCSweepSkipsPendingObjects (0.67s)1408=== CONT TestPinProtectsFromGC1409--- PASS: TestDeduplicatedObjectsRecordedAsPending (0.63s)1410=== CONT TestClientSharedPathCommittedMidPush14112026/11/12 07:18:55 WARN Force mode enabled - objects will be deleted immediately without grace period14122026/11/12 07:18:55 INFO Received push request method=POST path=/api/pushes14132026/11/12 07:18:55 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=014142026/11/12 07:18:55 INFO Vacuumed table table=pending_closures14152026/11/12 07:18:55 INFO Vacuumed table table=commit_receipts14162026/11/12 07:18:55 INFO Vacuumed table table=pending_objects14172026/11/12 07:18:55 INFO Vacuumed table table=multipart_uploads14182026/11/12 07:18:55 INFO Vacuumed table table=closures14192026/11/12 07:18:55 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)14202026/11/12 07:18:55 INFO Received uploads request method=POST path=/api/pending_closures14212026/11/12 07:18:55 INFO Vacuumed table table=objects1422--- PASS: TestSweepSkipsRowsLockedByPush (0.91s)1423=== CONT TestClientWithDependencies14242026/11/12 07:18:55 INFO Received sign narinfos request method=POST path=/api/pushes/2/sign14252026/11/12 07:18:55 INFO Signed narinfos id=2 count=114262026/11/12 07:18:55 INFO Uploading 1 narinfos14272026/11/12 07:18:55 WARN Failed to register uploaded object key=3hcchcnagxnz30fk5bq498frmaar0ni1.ls error="server returned 404: 404 page not found\n"14282026/11/12 07:18:55 INFO Aborted multipart uploads count=0 kept=014292026/11/12 07:18:55 INFO Received complete push request method=POST path=/api/pushes/2/complete14302026/11/12 07:18:55 WARN Failed to register uploaded object key=3hcchcnagxnz30fk5bq498frmaar0ni1.narinfo error="server returned 404: 404 page not found\n"14312026/11/12 07:18:55 INFO Completed push id=2 objects=3 duration=3.773535ms14322026/11/12 07:18:55 INFO Upload complete. (101ms)1433=== NAME TestNARDeduplicationMetadataUploadBug1434 metadata_upload_test.go:76: Retrieved narinfo from S3:1435 StorePath: /build/TestNARDeduplicationMetadataUploadBug2405511238/001/store/3hcchcnagxnz30fk5bq498frmaar0ni1-file2.txt1436 URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst1437 Compression: zstd1438 NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1439 NarSize: 1601440 References: 1441 CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1442--- PASS: TestCreatePendingClosureVerifyS3FailureReleasesConnection (0.83s)1443=== CONT TestClientMultipleUploads1444=== NAME TestNARDeduplicationMetadataUploadBug1445 metadata_upload_test.go:77: Retrieved .ls file from S3 (compressed size: 62 bytes)1446 metadata_upload_test.go:77: Decompressed .ls content (49 bytes):1447 {"version":1,"root":{"type":"regular","size":44}}1448--- PASS: TestNARDeduplicationMetadataUploadBug (1.29s)1449=== CONT TestClientIntegration14502026/11/12 07:18:55 INFO lead: acquired remote=192.0.2.1:1234 generation=114512026/11/12 07:18:55 INFO Aborted multipart uploads count=0 kept=014522026/11/12 07:18:55 WARN Force mode enabled - objects will be deleted immediately without grace period14532026/11/12 07:18:55 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=014542026/11/12 07:18:55 INFO Vacuumed table table=pending_closures14552026/11/12 07:18:55 INFO Vacuumed table table=commit_receipts14562026/11/12 07:18:55 INFO Vacuumed table table=pending_objects14572026/11/12 07:18:55 INFO Vacuumed table table=multipart_uploads14582026/11/12 07:18:55 INFO Vacuumed table table=closures14592026/11/12 07:18:55 INFO Vacuumed table table=objects1460--- PASS: TestGCMetrics (0.72s)1461=== CONT TestClientErrorHandling1462=== RUN TestClientErrorHandling/InvalidStorePath1463=== PAUSE TestClientErrorHandling/InvalidStorePath1464=== RUN TestClientErrorHandling/InvalidAuthToken1465=== PAUSE TestClientErrorHandling/InvalidAuthToken1466=== RUN TestClientErrorHandling/ServerNotAvailable1467=== PAUSE TestClientErrorHandling/ServerNotAvailable1468=== CONT TestClientCADerivations14692026/11/12 07:18:55 ERROR failed to check GC advisory lock error="query pg_locks: failed to connect to `user=nixbld database=niks3`: /nonexistent/.s.PGSQL.5432 (/nonexistent): dial error: dial unix /nonexistent/.s.PGSQL.5432: connect: no such file or directory"1470--- PASS: TestGCAdvisoryLockBlocksConcurrentRun (0.66s)1471=== CONT TestCacheStatsHandler14722026/11/12 07:18:55 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=01473--- PASS: TestPushWaitsForSweepHoldingRow (1.11s)1474=== CONT TestCacheConfigHandler1475=== RUN TestCacheConfigHandler/full_config,_no_issuer1476=== PAUSE TestCacheConfigHandler/full_config,_no_issuer1477=== RUN TestCacheConfigHandler/no_cache_url_configured1478=== PAUSE TestCacheConfigHandler/no_cache_url_configured1479=== RUN TestCacheConfigHandler/no_signing_keys1480=== PAUSE TestCacheConfigHandler/no_signing_keys1481=== RUN TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1482=== PAUSE TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1483=== CONT TestService_ReadScope_PublicByDefault14842026/11/12 07:18:55 INFO Vacuumed table table=pending_closures14852026/11/12 07:18:55 INFO Vacuumed table table=commit_receipts14862026/11/12 07:18:55 INFO Vacuumed table table=pending_objects14872026/11/12 07:18:55 INFO Vacuumed table table=multipart_uploads14882026/11/12 07:18:55 INFO Vacuumed table table=closures14892026/11/12 07:18:55 INFO Vacuumed table table=objects14902026/11/12 07:18:55 INFO Aborted multipart uploads count=0 kept=014912026/11/12 07:18:55 WARN Force mode enabled - objects will be deleted immediately without grace period14922026/11/12 07:18:55 INFO lead: acquired remote=192.0.2.1:1234 generation=114932026/11/12 07:18:55 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=014942026/11/12 07:18:55 INFO Vacuumed table table=pending_closures14952026/11/12 07:18:55 INFO Vacuumed table table=commit_receipts14962026/11/12 07:18:55 INFO Vacuumed table table=pending_objects14972026/11/12 07:18:55 INFO Vacuumed table table=multipart_uploads14982026/11/12 07:18:55 INFO Vacuumed table table=closures14992026/11/12 07:18:55 INFO Vacuumed table table=objects1500--- PASS: TestPushDedupSurvivesConcurrentGC (0.83s)1501=== CONT TestService_RequireScope_OIDC15022026/11/12 07:18:55 INFO lead: acquired remote=192.0.2.1:1234 generation=115032026/11/12 07:18:56 INFO lead: acquired remote=192.0.2.1:1234 generation=115042026/11/12 07:18:56 INFO lead: released remote=192.0.2.1:1234 generation=11505--- PASS: TestLeadIncumbentAnnouncesAtOnce (0.56s)1506=== CONT TestService_AuthMiddleware_OIDC15072026/11/12 07:18:56 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:32867/oidc15082026/11/12 07:18:56 INFO lead: acquired remote=192.0.2.1:1234 generation=115092026/11/12 07:18:56 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:38647/oidc1510--- PASS: TestGCBugBareHashReferences (1.04s)1511=== CONT TestService_ReadAuthMiddleware1512--- PASS: TestService_ReadScope_PublicByDefault (0.46s)1513=== CONT TestService_AuthMiddleware_MTLSBoundSubjects1514=== RUN TestService_RequireScope_OIDC/builder_may_write1515=== PAUSE TestService_RequireScope_OIDC/builder_may_write1516=== RUN TestService_RequireScope_OIDC/builder_may_not_admin1517=== PAUSE TestService_RequireScope_OIDC/builder_may_not_admin1518=== RUN TestService_RequireScope_OIDC/ops_may_admin1519=== PAUSE TestService_RequireScope_OIDC/ops_may_admin1520=== RUN TestService_RequireScope_OIDC/ops_may_not_write1521=== PAUSE TestService_RequireScope_OIDC/ops_may_not_write1522=== RUN TestService_RequireScope_OIDC/reader_may_not_write1523=== PAUSE TestService_RequireScope_OIDC/reader_may_not_write1524=== RUN TestService_RequireScope_OIDC/static_token_may_admin1525=== PAUSE TestService_RequireScope_OIDC/static_token_may_admin1526=== RUN TestService_RequireScope_OIDC/static_token_may_write1527=== PAUSE TestService_RequireScope_OIDC/static_token_may_write1528=== RUN TestService_RequireScope_OIDC/reader_may_read1529=== PAUSE TestService_RequireScope_OIDC/reader_may_read1530=== RUN TestService_RequireScope_OIDC/writer_implies_read1531=== PAUSE TestService_RequireScope_OIDC/writer_implies_read1532=== RUN TestService_RequireScope_OIDC/anonymous_may_not_read1533=== PAUSE TestService_RequireScope_OIDC/anonymous_may_not_read1534=== RUN TestService_RequireScope_OIDC/reader_may_read_with_netrc1535=== PAUSE TestService_RequireScope_OIDC/reader_may_read_with_netrc1536=== RUN TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1537=== PAUSE TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1538=== RUN TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read1539=== PAUSE TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read1540=== RUN TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1541=== PAUSE TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1542=== CONT TestService_AuthMiddleware_MTLSProxyHeader1543--- PASS: TestCacheStatsHandler (0.51s)1544=== CONT TestService_AuthMiddleware1545=== RUN TestService_AuthMiddleware_OIDC/valid_OIDC_token1546=== PAUSE TestService_AuthMiddleware_OIDC/valid_OIDC_token1547=== RUN TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1548=== PAUSE TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1549=== RUN TestService_AuthMiddleware_OIDC/malformed_token_rejected1550=== PAUSE TestService_AuthMiddleware_OIDC/malformed_token_rejected1551=== RUN TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1552=== PAUSE TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1553=== CONT TestCommitCanBeRepeatedUntilCleanup/push1554=== NAME TestPinProtectsFromGC1555 client_integration_test.go:791: Pinned store path: /build/TestPinProtectsFromGC1016494134/001/store/mgmh2nrn4mb6w6j7rr2jsl4hzn6bpgkq-pinned-file.txt1556 client_integration_test.go:792: Unpinned store path: /build/TestPinProtectsFromGC1016494134/001/store/9iywi8winr0fmj0vqw2p62g2g9255flx-unpinned-file.txt1557=== NAME TestClientMultipleUploads1558 client_integration_test.go:406: Created store path 0: /build/TestClientMultipleUploads2370503498/001/store/0d0lzzzzj2x8q45mndw0x1yr1aa4k1n8-test-file-0.txt1559=== NAME TestClientWithDependencies1560 client_integration_test.go:661: Built derivation: /build/TestClientWithDependencies1875601408/001/store/v7dyl3xqa2yxiw8j63m84l45pf331kl7-test-script1561--- PASS: TestConcurrentCommitsSharingObjectsDoNotDeadlock (0.95s)1562=== CONT TestCommitCanBeRepeatedUntilCleanup/pending_closure1563=== NAME TestClientMultipleUploads1564 client_integration_test.go:406: Created store path 1: /build/TestClientMultipleUploads2370503498/001/store/1ypb215v8z0l5s575a4giv50xmxsjq8a-test-file-1.txt1565=== NAME TestClientIntegration1566 client_integration_test.go:292: Created store path: /build/TestClientIntegration1664238411/002/store/78zwjn970gj13fg6kwjh6mjqfv0wcfz7-test-file.txt1567=== NAME TestClientWithDependencies1568 client_integration_test.go:663: Found 1 dependencies (including self)1569=== NAME TestClientCADerivations1570 client_ca_test.go:136: Built CA derivation: /build/TestClientCADerivations2091547917/001/store/hx9hdnl50xphp4dirxd14v0xxzr7xmhn-ca-test1571--- PASS: TestService_ReadAuthMiddleware (0.32s)1572=== CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info15732026/11/12 07:18:56 INFO Received uploads request method=POST path=/1574=== CONT TestPendingClosureWriteTimeout/empty1575=== CONT TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key15762026/11/12 07:18:56 INFO Received request for more parts method=POST path=/1577=== CONT TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key15782026/11/12 07:18:56 INFO Received complete multipart upload request method=POST path=/1579=== CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal15802026/11/12 07:18:56 INFO Received uploads request method=POST path=/1581=== CONT TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline1582--- PASS: TestUploadHandlersRejectInvalidKeys (0.10s)1583 --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info (0.00s)1584 --- PASS: TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key (0.00s)1585 --- PASS: TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key (0.00s)1586 --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal (0.00s)1587--- PASS: TestService_AuthMiddleware_MTLSProxyHeader (0.24s)1588=== CONT TestPendingClosureWriteTimeout/670k_objects1589=== CONT TestPendingClosureWriteTimeout/400_objects1590=== CONT TestProxyWriteTimeout/narinfo1591=== CONT TestProxyWriteTimeout/unknown_size1592=== CONT TestProxyWriteTimeout/10_GiB_nar1593=== CONT TestProxyWriteTimeout/1_GiB_nar1594--- PASS: TestProxyWriteTimeout (0.12s)1595 --- PASS: TestProxyWriteTimeout/narinfo (0.00s)1596 --- PASS: TestProxyWriteTimeout/unknown_size (0.00s)1597 --- PASS: TestProxyWriteTimeout/10_GiB_nar (0.00s)1598 --- PASS: TestProxyWriteTimeout/1_GiB_nar (0.00s)1599=== CONT TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed16002026/11/12 07:18:56 WARN Authentication failed token_preview=wrongtoken token_length=10 reason="static API token mismatch"1601--- PASS: TestService_AuthMiddleware (0.22s)1602=== CONT TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed1603=== NAME TestClientCADerivations1604 client_ca_test.go:139: Found 1 dependencies (including self)1605=== NAME TestClientMultipleUploads1606 client_integration_test.go:406: Created store path 2: /build/TestClientMultipleUploads2370503498/001/store/1lvbyb9ziv5bawfkca8jpn8sid052h8j-test-file-2.txt16072026/11/12 07:18:56 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted,O=Other"16082026/11/12 07:18:56 WARN mTLS auth: bound subjects configured but subject DN unavailable16092026/11/12 07:18:56 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted"1610--- PASS: TestService_AuthMiddleware_MTLSBoundSubjects (0.30s)1611=== CONT TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed16122026/11/12 07:18:56 INFO Received uploads request method=POST path=/api/pending_closures16132026/11/12 07:18:56 INFO Received push request method=POST path=/api/pushes16142026/11/12 07:18:56 INFO Received complete push request method=POST path=/complete16152026/11/12 07:18:56 INFO Received complete upload request method=POST path=/complete16162026/11/12 07:18:56 INFO Completed upload id=1 objects=2 duration=4.810904ms16172026/11/12 07:18:56 INFO Received complete upload request method=POST path=/complete16182026/11/12 07:18:56 INFO Completed upload id=1 objects=0 duration=469.984µs16192026/11/12 07:18:56 INFO Received cleanup request method=DELETE path=/api/pending_closures16202026/11/12 07:18:56 INFO Aborted multipart uploads count=0 kept=016212026/11/12 07:18:56 INFO Received complete upload request method=POST path=/complete16222026-11-12 07:18:56.810 UTC [1226] ERROR: Closure does not exist: id=116232026-11-12 07:18:56.810 UTC [1226] CONTEXT: PL/pgSQL function commit_pending_closure(bigint) line 17 at RAISE16242026-11-12 07:18:56.810 UTC [1226] STATEMENT: -- name: CommitPendingClosure :one1625 SELECT commit_pending_closure($1::bigint)::bigint1626 16272026/11/12 07:18:56 INFO Completed push id=1 objects=2 duration=9.851771ms16282026/11/12 07:18:56 INFO Received complete push request method=POST path=/complete1629=== CONT TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost16302026/11/12 07:18:56 INFO Completed push id=1 objects=0 duration=456.424µs16312026/11/12 07:18:56 INFO Received cleanup request method=DELETE path=/api/pending_closures16322026/11/12 07:18:56 INFO Aborted multipart uploads count=0 kept=016332026/11/12 07:18:56 INFO Received complete push request method=POST path=/complete16342026-11-12 07:18:56.814 UTC [1223] ERROR: Push does not exist: id=116352026-11-12 07:18:56.814 UTC [1223] CONTEXT: PL/pgSQL function commit_push(bigint) line 15 at RAISE16362026-11-12 07:18:56.814 UTC [1223] STATEMENT: -- name: CommitPush :one1637 SELECT commit_push($1::bigint)::bigint1638 16392026/11/12 07:18:56 WARN Push not found id=11640--- PASS: TestCommitCanBeRepeatedUntilCleanup (0.00s)1641 --- PASS: TestCommitCanBeRepeatedUntilCleanup/pending_closure (0.30s)1642 --- PASS: TestCommitCanBeRepeatedUntilCleanup/push (0.39s)1643=== CONT TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed16442026/11/12 07:18:56 INFO Received uploads request method=POST path=/api/pending_closures16452026/11/12 07:18:56 INFO Received uploads request method=POST path=/api/pending_closures16462026/11/12 07:18:56 INFO Received push request method=POST path=/api/pushes16472026/11/12 07:18:56 INFO Received uploads request method=POST path=/api/pending_closures16482026/11/12 07:18:56 INFO lead: released remote=192.0.2.1:1234 generation=11649--- PASS: TestLeadEndsOnShutdown (1.68s)1650=== CONT TestUploadHandlersRejectOversizedBody/complete_multipart16512026/11/12 07:18:56 INFO Received complete multipart upload request method=POST path=/16522026/11/12 07:18:56 INFO Received push request method=POST path=/api/pushes16532026/11/12 07:18:56 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)16542026/11/12 07:18:56 INFO Uploading 78zwjn970gj13fg6kwjh6mjqfv0wcfz7-test-file.txt (152B)16552026/11/12 07:18:56 INFO Received uploads request method=POST path=/api/pending_closures16562026/11/12 07:18:56 INFO Received push request method=POST path=/api/pushes16572026/11/12 07:18:56 INFO Uploading 2 paths to 127.0.0.1 (0 already cached)16582026/11/12 07:18:56 INFO Uploading 05bndnih2ciihm56k0hlbzhmx2zz5ads-shared-dep (136B)16592026/11/12 07:18:56 INFO Uploading zqjhxz59s2ksg68akrd1y93cmfj7im0l-top (224B)16602026/11/12 07:18:56 INFO Uploading 2 paths to 127.0.0.1 (1 already cached)16612026/11/12 07:18:56 INFO Uploading b3dhicwrps7qk342hmwhc5lqskxgrw87-a (216B)16622026/11/12 07:18:56 INFO Uploading wkpnkxgwczdn4by44cxsfxcphchwcmmf-shared-dep (136B)16632026/11/12 07:18:56 WARN Failed to register uploaded object key=78zwjn970gj13fg6kwjh6mjqfv0wcfz7.ls error="server returned 404: 404 page not found\n"16642026/11/12 07:18:56 WARN Failed to register uploaded object key=nar/11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1.nar.zst error="server returned 404: 404 page not found\n"16652026/11/12 07:18:56 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign16662026/11/12 07:18:56 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)16672026/11/12 07:18:56 INFO Uploading mgmh2nrn4mb6w6j7rr2jsl4hzn6bpgkq-pinned-file.txt (128B)16682026/11/12 07:18:56 INFO Signed narinfos id=1 count=116692026/11/12 07:18:56 INFO Uploading 1 narinfos16702026/11/12 07:18:56 WARN Failed to register uploaded object key=05bndnih2ciihm56k0hlbzhmx2zz5ads.ls error="server returned 404: 404 page not found\n"16712026/11/12 07:18:56 WARN Failed to register uploaded object key=zqjhxz59s2ksg68akrd1y93cmfj7im0l.ls error="server returned 404: 404 page not found\n"16722026/11/12 07:18:56 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"16732026/11/12 07:18:56 WARN Failed to register uploaded object key=nar/0qmhsg42pqsz7x6hh02j8gznnac3yi3qpjhrqbwvz88i194jbb5s.nar.zst error="server returned 404: 404 page not found\n"16742026/11/12 07:18:56 INFO Received complete push request method=POST path=/api/pushes/1/complete16752026/11/12 07:18:56 WARN Failed to register uploaded object key=wkpnkxgwczdn4by44cxsfxcphchwcmmf.ls error="server returned 404: 404 page not found\n"16762026/11/12 07:18:56 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign16772026/11/12 07:18:56 WARN Failed to register uploaded object key=nar/17r79b7wlzql9s5q5gzwfznhk28n85if5n32n1x1m0fvpv5bv7mi.nar.zst error="server returned 404: 404 page not found\n"16782026/11/12 07:18:56 WARN Failed to register uploaded object key=nar/02m5na0pxqd1f0kd30ka4i7slwwk3r8b4xgbwik8aqaf4vl26xrh.nar.zst error="server returned 404: 404 page not found\n"16792026/11/12 07:18:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign16802026/11/12 07:18:56 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"16812026/11/12 07:18:56 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign16822026/11/12 07:18:56 INFO Signed narinfos id=1 count=216832026/11/12 07:18:56 WARN Failed to register uploaded object key=b3dhicwrps7qk342hmwhc5lqskxgrw87.ls error="server returned 404: 404 page not found\n"16842026/11/12 07:18:56 INFO Uploading 2 narinfos16852026/11/12 07:18:56 INFO Signed narinfos id=1 count=216862026/11/12 07:18:56 WARN Failed to register uploaded object key=911fpdbb9amsjip52srj7qws8nw82x58.ls error="server returned 404: 404 page not found\n"16872026/11/12 07:18:56 INFO Signed narinfos id=1 count=116882026/11/12 07:18:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign16892026/11/12 07:18:56 INFO Uploading 1 narinfos16902026/11/12 07:18:56 INFO Signed narinfos id=2 count=216912026/11/12 07:18:56 INFO Uploading 4 narinfos16922026/11/12 07:18:57 INFO Completed push id=1 objects=3 duration=7.483709ms16932026/11/12 07:18:57 INFO Upload complete. (316ms)16942026/11/12 07:18:57 WARN Failed to register uploaded object key=wkpnkxgwczdn4by44cxsfxcphchwcmmf.narinfo error="server returned 404: 404 page not found\n"16952026/11/12 07:18:57 WARN Failed to register uploaded object key=b3dhicwrps7qk342hmwhc5lqskxgrw87.narinfo error="server returned 404: 404 page not found\n"16962026/11/12 07:18:57 WARN Failed to register uploaded object key=05bndnih2ciihm56k0hlbzhmx2zz5ads.narinfo error="server returned 404: 404 page not found\n"16972026/11/12 07:18:57 INFO Received complete push request method=POST path=/api/pushes/1/complete16982026/11/12 07:18:57 WARN Failed to register uploaded object key=zqjhxz59s2ksg68akrd1y93cmfj7im0l.narinfo error="server returned 404: 404 page not found\n"16992026/11/12 07:18:57 INFO Completed push id=1 objects=6 duration=7.716891ms17002026/11/12 07:18:57 INFO Upload complete. (368ms)17012026/11/12 07:18:57 INFO Received push request method=POST path=/api/pushes17022026/11/12 07:18:57 INFO Received push request method=POST path=/api/pushes17032026/11/12 07:18:57 WARN Failed to register uploaded object key=mgmh2nrn4mb6w6j7rr2jsl4hzn6bpgkq.ls error="server returned 404: 404 page not found\n"17042026/11/12 07:18:57 WARN Failed to register uploaded object key=78zwjn970gj13fg6kwjh6mjqfv0wcfz7.narinfo error="server returned 404: 404 page not found\n"17052026/11/12 07:18:57 INFO Uploading 2 paths to 127.0.0.1 (1 already cached)17062026/11/12 07:18:57 INFO Uploading p6hqfvvpm65jhzi8h5kq550c8ab2im40-shared-dep (136B)17072026/11/12 07:18:57 INFO Received uploads request method=POST path=/api/pending_closures17082026/11/12 07:18:57 INFO Uploading xhgxjq2yb934llkaylcij1hc3vqwdg6p-a (216B)17092026/11/12 07:18:57 INFO Received push request method=POST path=/api/pushes17102026/11/12 07:18:57 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)17112026/11/12 07:18:57 INFO Uploading v7dyl3xqa2yxiw8j63m84l45pf331kl7-test-script (136B)17122026/11/12 07:18:57 WARN Failed to register uploaded object key=911fpdbb9amsjip52srj7qws8nw82x58.narinfo error="server returned 404: 404 page not found\n"17132026/11/12 07:18:57 INFO Received complete push request method=POST path=/api/pushes/1/complete17142026/11/12 07:18:57 WARN Failed to register uploaded object key=mgmh2nrn4mb6w6j7rr2jsl4hzn6bpgkq.narinfo error="server returned 404: 404 page not found\n"17152026/11/12 07:18:57 WARN Failed to register uploaded object key=9c31lcw8m34zh73dy72ck0312ncnm1pn.ls error="server returned 404: 404 page not found\n"17162026/11/12 07:18:57 WARN Failed to register uploaded object key=p6hqfvvpm65jhzi8h5kq550c8ab2im40.ls error="server returned 404: 404 page not found\n"17172026/11/12 07:18:57 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"1718=== CONT TestUploadHandlersRejectOversizedBody/create_pending_closure17192026/11/12 07:18:57 INFO Received uploads request method=POST path=/17202026/11/12 07:18:57 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete17212026/11/12 07:18:57 WARN Failed to register uploaded object key=wkpnkxgwczdn4by44cxsfxcphchwcmmf.narinfo error="server returned 404: 404 page not found\n"17222026/11/12 07:18:57 INFO Uploading 3 paths to 127.0.0.1 (0 already cached)17232026/11/12 07:18:57 INFO Uploading 1ypb215v8z0l5s575a4giv50xmxsjq8a-test-file-1.txt (160B)17242026/11/12 07:18:57 INFO Uploading 1lvbyb9ziv5bawfkca8jpn8sid052h8j-test-file-2.txt (160B)17252026/11/12 07:18:57 WARN Failed to register uploaded object key=xhgxjq2yb934llkaylcij1hc3vqwdg6p.ls error="server returned 404: 404 page not found\n"17262026/11/12 07:18:57 INFO Uploading 0d0lzzzzj2x8q45mndw0x1yr1aa4k1n8-test-file-0.txt (160B)17272026/11/12 07:18:57 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign17282026/11/12 07:18:57 WARN Failed to register uploaded object key=nar/1ixnzpidy5s44h0vgzvhr5fcymmg0g4wsfhh0z087dsz2yhmf0qa.nar.zst error="server returned 404: 404 page not found\n"17292026/11/12 07:18:57 WARN Failed to register uploaded object key=nar/1s7yghixzpinyqm7g59kyn3sav81pmnsdi5v63m6x4r2vacy907j.nar.zst error="server returned 404: 404 page not found\n"17302026/11/12 07:18:57 INFO Signed narinfos id=1 count=317312026/11/12 07:18:57 INFO Uploading 3 narinfos17322026/11/12 07:18:57 WARN Failed to register uploaded object key=v7dyl3xqa2yxiw8j63m84l45pf331kl7.ls error="server returned 404: 404 page not found\n"17332026/11/12 07:18:57 WARN Failed to register uploaded object key=log/qp2c22a4qpddlgxxkqdx9xac1y8glkqz-test-script.drv error="server returned 404: 404 page not found\n"17342026/11/12 07:18:57 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign17352026/11/12 07:18:57 INFO Signed narinfos id=1 count=117362026/11/12 07:18:57 INFO Uploading 1 narinfos17372026/11/12 07:18:57 INFO lead: released remote=192.0.2.1:1234 generation=11738=== NAME TestClientSharedPathCommittedMidPush1739 client_integration_test.go:740: Retrieved narinfo from S3:1740 StorePath: /build/TestClientSharedPathCommittedMidPush2068071447/001/store/05bndnih2ciihm56k0hlbzhmx2zz5ads-shared-dep1741 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1742 Compression: zstd1743 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821744 NarSize: 1361745 References: 1746 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n17472026/11/12 07:18:57 INFO lead: released remote=192.0.2.1:1234 generation=117482026/11/12 07:18:57 INFO Completed upload id=2 objects=6 duration=12.945399ms17492026/11/12 07:18:57 INFO Completed push id=1 objects=3 duration=15.829926ms17502026/11/12 07:18:57 INFO Upload complete. (488ms)17512026/11/12 07:18:57 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete17522026/11/12 07:18:57 WARN Failed to register uploaded object key=xhgxjq2yb934llkaylcij1hc3vqwdg6p.narinfo error="server returned 404: 404 page not found\n"17532026/11/12 07:18:57 WARN Failed to register uploaded object key=1ypb215v8z0l5s575a4giv50xmxsjq8a.ls error="server returned 404: 404 page not found\n"17542026/11/12 07:18:57 WARN Failed to register uploaded object key=9c31lcw8m34zh73dy72ck0312ncnm1pn.narinfo error="server returned 404: 404 page not found\n"17552026/11/12 07:18:57 INFO Received complete push request method=POST path=/api/pushes/1/complete17562026/11/12 07:18:57 WARN Failed to register uploaded object key=v7dyl3xqa2yxiw8j63m84l45pf331kl7.narinfo error="server returned 404: 404 page not found\n"17572026/11/12 07:18:57 WARN Failed to register uploaded object key=p6hqfvvpm65jhzi8h5kq550c8ab2im40.narinfo error="server returned 404: 404 page not found\n"17582026/11/12 07:18:57 INFO Received complete push request method=POST path=/api/pushes/1/complete17592026/11/12 07:18:57 WARN Failed to register uploaded object key=1lvbyb9ziv5bawfkca8jpn8sid052h8j.ls error="server returned 404: 404 page not found\n"17602026/11/12 07:18:57 INFO Completed upload id=1 objects=6 duration=1.616655ms17612026/11/12 07:18:57 WARN Failed to register uploaded object key=nar/0ahq3izw3h0b0c08p3xdfclh758fwxnp85ccvflasb4vnfsv50ri.nar.zst error="server returned 404: 404 page not found\n"17622026/11/12 07:18:57 INFO Upload complete. (453ms)17632026/11/12 07:18:57 WARN Failed to register uploaded object key=0d0lzzzzj2x8q45mndw0x1yr1aa4k1n8.ls error="server returned 404: 404 page not found\n"17642026/11/12 07:18:57 WARN Failed to register uploaded object key=nar/1823g6vx4q5zqvr73m7lkq1wpk40fin7bsc792di430yk7sg4lka.nar.zst error="server returned 404: 404 page not found\n"1765 client_integration_test.go:740: Retrieved narinfo from S3:1766 StorePath: /build/TestClientSharedPathCommittedMidPush2068071447/001/store/zqjhxz59s2ksg68akrd1y93cmfj7im0l-top1767 URL: nar/02m5na0pxqd1f0kd30ka4i7slwwk3r8b4xgbwik8aqaf4vl26xrh.nar.zst1768 Compression: zstd1769 NarHash: sha256:02m5na0pxqd1f0kd30ka4i7slwwk3r8b4xgbwik8aqaf4vl26xrh1770 NarSize: 2241771 References: /build/TestClientSharedPathCommittedMidPush2068071447/001/store/05bndnih2ciihm56k0hlbzhmx2zz5ads-shared-dep1772 CA: text:sha256:1d7lhg1813m2hb7abwym8plhr93ql8a8xg1my8x4s9jv9431dspi17732026/11/12 07:18:57 WARN Failed to register uploaded object key=nar/13p9pqi0ga6fx2apcwls9hjnv7nmf740ggpn5zvfzdsp5vz2bbg4.nar.zst error="server returned 404: 404 page not found\n"1774=== NAME TestClientFallsBackToClosures17752026/11/12 07:18:57 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign1776 client_pushes_test.go:112: Retrieved narinfo from S3:1777 StorePath: /build/TestClientFallsBackToClosures1179113469/001/store/wkpnkxgwczdn4by44cxsfxcphchwcmmf-shared-dep1778 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1779 Compression: zstd1780 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821781 NarSize: 1361782 References: 1783 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n17842026/11/12 07:18:57 INFO Signed narinfos id=1 count=317852026/11/12 07:18:57 INFO Uploading 3 narinfos1786 client_pushes_test.go:112: Retrieved narinfo from S3:1787 StorePath: /build/TestClientFallsBackToClosures1179113469/001/store/b3dhicwrps7qk342hmwhc5lqskxgrw87-a1788 URL: nar/17r79b7wlzql9s5q5gzwfznhk28n85if5n32n1x1m0fvpv5bv7mi.nar.zst1789 Compression: zstd1790 NarHash: sha256:17r79b7wlzql9s5q5gzwfznhk28n85if5n32n1x1m0fvpv5bv7mi1791 NarSize: 2161792 References: /build/TestClientFallsBackToClosures1179113469/001/store/wkpnkxgwczdn4by44cxsfxcphchwcmmf-shared-dep1793 CA: text:sha256:18l2fzp7vvrfnr8ibnzi2slwpdhrn6zdpcx70fga1nfxj4yy01cv17942026/11/12 07:18:57 INFO Completed push id=1 objects=4 duration=8.768161ms17952026/11/12 07:18:57 WARN Failed to register uploaded object key=1ypb215v8z0l5s575a4giv50xmxsjq8a.narinfo error="server returned 404: 404 page not found\n"17962026/11/12 07:18:57 INFO Upload complete. (287ms)17972026/11/12 07:18:57 WARN Failed to register uploaded object key=1lvbyb9ziv5bawfkca8jpn8sid052h8j.narinfo error="server returned 404: 404 page not found\n"17982026/11/12 07:18:57 WARN Failed to register uploaded object key=0d0lzzzzj2x8q45mndw0x1yr1aa4k1n8.narinfo error="server returned 404: 404 page not found\n"17992026/11/12 07:18:57 INFO Received complete push request method=POST path=/api/pushes/1/complete1800 client_pushes_test.go:112: Retrieved narinfo from S3:1801 StorePath: /build/TestClientFallsBackToClosures1179113469/001/store/911fpdbb9amsjip52srj7qws8nw82x58-b1802 URL: nar/17r79b7wlzql9s5q5gzwfznhk28n85if5n32n1x1m0fvpv5bv7mi.nar.zst18032026/11/12 07:18:57 INFO Completed push id=1 objects=8 duration=9.920331ms1804 Compression: zstd1805 NarHash: sha256:17r79b7wlzql9s5q5gzwfznhk28n85if5n32n1x1m0fvpv5bv7mi1806 NarSize: 2161807 References: /build/TestClientFallsBackToClosures1179113469/001/store/wkpnkxgwczdn4by44cxsfxcphchwcmmf-shared-dep1808 CA: text:sha256:18l2fzp7vvrfnr8ibnzi2slwpdhrn6zdpcx70fga1nfxj4yy01cv18092026/11/12 07:18:57 INFO Upload complete. (486ms)1810=== NAME TestClientPushesUseOnePush1811 client_pushes_test.go:97: Retrieved narinfo from S3:1812 StorePath: /build/TestClientPushesUseOnePush2487036571/001/store/p6hqfvvpm65jhzi8h5kq550c8ab2im40-shared-dep1813 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1814 Compression: zstd1815 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821816 NarSize: 1361817 References: 1818 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n18192026/11/12 07:18:57 INFO Received uploads request method=POST path=/api/pending_closures18202026/11/12 07:18:57 INFO Completed push id=1 objects=9 duration=40.227571ms18212026/11/12 07:18:57 INFO Upload complete. (236ms)1822=== NAME TestClientMultipleUploads1823 client_integration_test.go:417: Uploaded 3 paths in 538.516642ms18242026/11/12 07:18:57 INFO lead: acquired remote=192.0.2.1:1234 generation=21825--- PASS: TestClientSharedPathCommittedMidPush (1.46s)1826=== CONT TestUploadHandlersRejectOversizedBody/request_more_parts18272026/11/12 07:18:57 INFO Received request for more parts method=POST path=/1828=== NAME TestClientPushesUseOnePush1829 client_pushes_test.go:97: Retrieved narinfo from S3:1830 StorePath: /build/TestClientPushesUseOnePush2487036571/001/store/xhgxjq2yb934llkaylcij1hc3vqwdg6p-a1831 URL: nar/1ixnzpidy5s44h0vgzvhr5fcymmg0g4wsfhh0z087dsz2yhmf0qa.nar.zst1832 Compression: zstd1833 NarHash: sha256:1ixnzpidy5s44h0vgzvhr5fcymmg0g4wsfhh0z087dsz2yhmf0qa1834 NarSize: 2161835 References: /build/TestClientPushesUseOnePush2487036571/001/store/p6hqfvvpm65jhzi8h5kq550c8ab2im40-shared-dep1836 CA: text:sha256:03kkxmm2fjrirc71fx5phmxkrj1nq1xc8h2rzjcycip85shbsxyh1837--- PASS: TestClientFallsBackToClosures (1.54s)1838=== CONT TestIsValidUploadKey/narinfo1839=== CONT TestIsValidUploadKey/unknown_type1840=== CONT TestIsValidUploadKey/empty_key1841=== CONT TestIsValidUploadKey/absolute1842=== CONT TestIsValidUploadKey/traversal_nar1843=== CONT TestIsValidUploadKey/traversal1844=== CONT TestIsValidUploadKey/listing_key,_narinfo_type1845=== CONT TestIsValidUploadKey/nar_key,_narinfo_type1846=== CONT TestIsValidUploadKey/narinfo_key,_nar_type1847=== CONT TestIsValidUploadKey/index.html1848=== CONT TestIsValidUploadKey/nix-cache-info1849=== CONT TestIsValidUploadKey/realisation_plus_in_output1850=== CONT TestIsValidUploadKey/realisation1851=== CONT TestIsValidUploadKey/build_log_equals1852=== CONT TestIsValidUploadKey/build_log_question_mark1853=== CONT TestIsValidUploadKey/build_log_plus_in_name1854=== CONT TestIsValidUploadKey/build_log_home-manager_file1855=== CONT TestIsValidUploadKey/build_log1856=== CONT TestIsValidUploadKey/listing1857=== CONT TestIsValidUploadKey/nar_plain1858=== CONT TestIsValidUploadKey/nar_xz1859=== CONT TestIsValidUploadKey/nar_zst1860=== CONT TestIsValidCachePath/narinfo1861--- PASS: TestIsValidUploadKey (0.29s)1862 --- PASS: TestIsValidUploadKey/narinfo (0.00s)1863 --- PASS: TestIsValidUploadKey/unknown_type (0.00s)1864 --- PASS: TestIsValidUploadKey/empty_key (0.00s)1865 --- PASS: TestIsValidUploadKey/absolute (0.00s)1866 --- PASS: TestIsValidUploadKey/traversal_nar (0.00s)1867 --- PASS: TestIsValidUploadKey/traversal (0.00s)1868 --- PASS: TestIsValidUploadKey/listing_key,_narinfo_type (0.00s)1869 --- PASS: TestIsValidUploadKey/nar_key,_narinfo_type (0.00s)1870 --- PASS: TestIsValidUploadKey/narinfo_key,_nar_type (0.00s)1871 --- PASS: TestIsValidUploadKey/index.html (0.00s)1872 --- PASS: TestIsValidUploadKey/nix-cache-info (0.00s)1873 --- PASS: TestIsValidUploadKey/realisation_plus_in_output (0.00s)1874 --- PASS: TestIsValidUploadKey/realisation (0.00s)1875 --- PASS: TestIsValidUploadKey/build_log_equals (0.00s)1876 --- PASS: TestIsValidUploadKey/build_log_question_mark (0.00s)1877 --- PASS: TestIsValidUploadKey/build_log_plus_in_name (0.00s)1878 --- PASS: TestIsValidUploadKey/build_log_home-manager_file (0.00s)1879 --- PASS: TestIsValidUploadKey/build_log (0.00s)1880 --- PASS: TestIsValidUploadKey/listing (0.00s)1881 --- PASS: TestIsValidUploadKey/nar_plain (0.00s)1882 --- PASS: TestIsValidUploadKey/nar_xz (0.00s)1883 --- PASS: TestIsValidUploadKey/nar_zst (0.00s)1884=== CONT TestParseSingleRange/none1885=== CONT TestIsValidCachePath/short_hash1886=== CONT TestIsValidCachePath/wrong_extension1887=== CONT TestIsValidCachePath/leading_slash1888=== CONT TestIsValidCachePath/empty1889=== CONT TestIsValidCachePath/random_path1890=== CONT TestIsValidCachePath/invalid_char_u1891=== CONT TestIsValidCachePath/invalid_char_e1892=== CONT TestIsValidCachePath/traversal_in_middle1893=== CONT TestIsValidCachePath/traversal_parent1894=== CONT TestIsValidCachePath/index.html1895=== CONT TestIsValidCachePath/nix-cache-info1896=== CONT TestIsValidCachePath/realisation1897=== CONT TestIsValidCachePath/log1898=== CONT TestIsValidCachePath/ls1899=== CONT TestIsValidCachePath/nar_uncompressed1900=== CONT TestIsValidCachePath/nar_bz21901=== CONT TestIsValidCachePath/nar_xz1902=== CONT TestIsValidCachePath/nar_zst1903=== CONT TestIsValidCachePath/narinfo_all_nix_base32_chars1904--- PASS: TestIsValidCachePath (0.00s)1905 --- PASS: TestIsValidCachePath/narinfo (0.00s)1906 --- PASS: TestIsValidCachePath/short_hash (0.00s)1907 --- PASS: TestIsValidCachePath/wrong_extension (0.00s)1908 --- PASS: TestIsValidCachePath/leading_slash (0.00s)1909 --- PASS: TestIsValidCachePath/empty (0.00s)1910 --- PASS: TestIsValidCachePath/random_path (0.00s)1911 --- PASS: TestIsValidCachePath/invalid_char_u (0.00s)1912 --- PASS: TestIsValidCachePath/invalid_char_e (0.00s)1913 --- PASS: TestIsValidCachePath/traversal_in_middle (0.00s)1914 --- PASS: TestIsValidCachePath/traversal_parent (0.00s)1915 --- PASS: TestIsValidCachePath/index.html (0.00s)1916 --- PASS: TestIsValidCachePath/nix-cache-info (0.00s)1917 --- PASS: TestIsValidCachePath/realisation (0.00s)1918 --- PASS: TestIsValidCachePath/log (0.00s)1919 --- PASS: TestIsValidCachePath/ls (0.00s)1920 --- PASS: TestIsValidCachePath/nar_uncompressed (0.00s)1921 --- PASS: TestIsValidCachePath/nar_bz2 (0.00s)1922 --- PASS: TestIsValidCachePath/nar_xz (0.00s)1923 --- PASS: TestIsValidCachePath/nar_zst (0.00s)1924 --- PASS: TestIsValidCachePath/narinfo_all_nix_base32_chars (0.00s)1925=== CONT TestParseSingleRange/open-ended1926=== CONT TestParseSingleRange/start_far_past_EOF1927=== CONT TestParseSingleRange/start_past_EOF1928=== CONT TestParseSingleRange/single_byte1929=== CONT TestParseSingleRange/suffix_exceeds_size1930=== CONT TestParseSingleRange/suffix1931=== CONT TestParseSingleRange/end_clamped_to_size1932=== CONT TestParseSingleRange/malformed_both_empty1933=== CONT TestParseSingleRange/closed1934=== CONT TestParseSingleRange/malformed_end_before_start1935=== CONT TestParseSingleRange/multi-range_ignored1936=== CONT TestParseSingleRange/malformed_no_dash1937=== CONT TestParseSingleRange/unknown_unit1938--- PASS: TestParseSingleRange (0.00s)1939 --- PASS: TestParseSingleRange/none (0.00s)1940 --- PASS: TestParseSingleRange/open-ended (0.00s)1941 --- PASS: TestParseSingleRange/start_far_past_EOF (0.00s)1942 --- PASS: TestParseSingleRange/start_past_EOF (0.00s)1943 --- PASS: TestParseSingleRange/single_byte (0.00s)1944 --- PASS: TestParseSingleRange/suffix_exceeds_size (0.00s)1945 --- PASS: TestParseSingleRange/suffix (0.00s)1946 --- PASS: TestParseSingleRange/end_clamped_to_size (0.00s)1947 --- PASS: TestParseSingleRange/malformed_both_empty (0.00s)1948 --- PASS: TestParseSingleRange/closed (0.00s)1949 --- PASS: TestParseSingleRange/malformed_end_before_start (0.00s)1950 --- PASS: TestParseSingleRange/multi-range_ignored (0.00s)1951 --- PASS: TestParseSingleRange/malformed_no_dash (0.00s)1952 --- PASS: TestParseSingleRange/unknown_unit (0.00s)1953=== CONT TestPresentNotReportedWhileGCDeletesClosure/stale1954=== NAME TestClientPushesUseOnePush1955 client_pushes_test.go:97: Retrieved narinfo from S3:1956 StorePath: /build/TestClientPushesUseOnePush2487036571/001/store/9c31lcw8m34zh73dy72ck0312ncnm1pn-b1957 URL: nar/1ixnzpidy5s44h0vgzvhr5fcymmg0g4wsfhh0z087dsz2yhmf0qa.nar.zst1958 Compression: zstd1959 NarHash: sha256:1ixnzpidy5s44h0vgzvhr5fcymmg0g4wsfhh0z087dsz2yhmf0qa19602026/11/12 07:18:57 INFO lead: acquired remote=192.0.2.1:1234 generation=21961 NarSize: 2161962 References: /build/TestClientPushesUseOnePush2487036571/001/store/p6hqfvvpm65jhzi8h5kq550c8ab2im40-shared-dep1963 CA: text:sha256:03kkxmm2fjrirc71fx5phmxkrj1nq1xc8h2rzjcycip85shbsxyh19642026/11/12 07:18:57 INFO Aborted multipart uploads count=0 kept=01965--- PASS: TestClientMultipleUploads (1.40s)1966=== CONT TestPresentNotReportedWhileGCDeletesClosure/fresh1967--- PASS: TestClientPushesUseOnePush (1.54s)1968=== CONT TestPush_RejectsBadRequests/bad_root19692026/11/12 07:18:57 INFO Received push request method=POST path=/api/pushes1970=== CONT TestPush_RejectsBadRequests/no_roots19712026/11/12 07:18:57 INFO Received push request method=POST path=/api/pushes1972=== CONT TestPush_RejectsBadRequests/root_not_in_objects19732026/11/12 07:18:57 INFO Received push request method=POST path=/api/pushes1974=== CONT TestPush_RejectsBadRequests/no_objects19752026/11/12 07:18:57 INFO Received push request method=POST path=/api/pushes1976=== CONT TestServerTLSConfig/no_client_CA1977=== CONT TestServerTLSConfig/not_a_PEM_file1978--- PASS: TestPush_RejectsBadRequests (5.47s)1979 --- PASS: TestPush_RejectsBadRequests/bad_root (0.00s)1980 --- PASS: TestPush_RejectsBadRequests/no_roots (0.00s)1981 --- PASS: TestPush_RejectsBadRequests/root_not_in_objects (0.00s)1982 --- PASS: TestPush_RejectsBadRequests/no_objects (0.00s)1983=== CONT TestServerTLSConfig/missing_CA_file1984--- PASS: TestServerTLSConfig (0.00s)1985 --- PASS: TestServerTLSConfig/no_client_CA (0.00s)1986 --- PASS: TestServerTLSConfig/not_a_PEM_file (0.00s)1987 --- PASS: TestServerTLSConfig/missing_CA_file (0.00s)1988=== CONT TestGCEndsOnShutdown/before_the_run19892026/11/12 07:18:57 INFO lead: given back to the incumbent remote=192.0.2.1:1234 generation=219902026/11/12 07:18:57 INFO Received uploads request method=POST path=/api/pending_closures1991=== CONT TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark19922026/11/12 07:18:57 INFO Received uploads request method=POST path=/api/pending_closures19932026/11/12 07:18:57 INFO lead: acquired remote=192.0.2.1:1234 generation=319942026/11/12 07:18:57 INFO lead: released remote=192.0.2.1:1234 generation=119952026/11/12 07:18:57 INFO lead: acquired remote=192.0.2.1:1234 generation=219962026/11/12 07:18:57 WARN Force mode enabled - objects will be deleted immediately without grace period19972026/11/12 07:18:57 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=019982026/11/12 07:18:57 INFO Vacuumed table table=pending_closures19992026/11/12 07:18:57 INFO Received complete multipart upload request method=POST path=/api/multipart/complete20002026/11/12 07:18:57 INFO Vacuumed table table=commit_receipts
loading 936 hidden lines…
29372026/11/12 07:19:13 INFO Vacuumed table table=commit_receipts29382026/11/12 07:19:13 INFO Vacuumed table table=pending_objects29392026/11/12 07:19:13 INFO Vacuumed table table=multipart_uploads29402026/11/12 07:19:13 INFO Vacuumed table table=closures29412026/11/12 07:19:13 INFO Vacuumed table table=objects29422026/11/12 07:19:13 INFO Received uploads request method=POST path=/29432026/11/12 07:19:13 INFO Received complete push request method=POST path=/29442026/11/12 07:19:13 INFO Aborted multipart uploads count=0 kept=029452026/11/12 07:19:13 INFO Completed push id=2 objects=7 duration=4.103638ms29462026/11/12 07:19:13 INFO Received complete upload request method=POST path=/29472026/11/12 07:19:13 INFO Completed upload id=10 objects=23 duration=2.115779ms29482026/11/12 07:19:13 INFO Received uploads request method=POST path=/29492026/11/12 07:19:13 INFO Received uploads request method=POST path=/29502026/11/12 07:19:13 INFO Aborted multipart uploads count=0 kept=029512026/11/12 07:19:13 INFO Received push request method=POST path=/29522026/11/12 07:19:13 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."29532026/11/12 07:19:13 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."29542026/11/12 07:19:13 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."29552026/11/12 07:19:13 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."29562026/11/12 07:19:13 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."29572026/11/12 07:19:13 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."29582026/11/12 07:19:13 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."29592026/11/12 07:19:13 INFO Received uploads request method=POST path=/29602026/11/12 07:19:13 INFO Received uploads request method=POST path=/29612026/11/12 07:19:13 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=4 objects-deleted-after-grace-period=0 objects-failed-to-delete=029622026/11/12 07:19:13 WARN Force mode enabled - objects will be deleted immediately without grace period29632026/11/12 07:19:13 INFO Received complete upload request method=POST path=/29642026/11/12 07:19:13 WARN Force mode enabled - objects will be deleted immediately without grace period29652026/11/12 07:19:13 INFO Vacuumed table table=pending_closures29662026/11/12 07:19:13 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029672026/11/12 07:19:13 INFO Completed upload id=3 objects=5 duration=2.938307ms29682026/11/12 07:19:13 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029692026/11/12 07:19:13 INFO Vacuumed table table=commit_receipts29702026/11/12 07:19:13 INFO Received uploads request method=POST path=/29712026/11/12 07:19:13 INFO Aborted multipart uploads count=0 kept=029722026/11/12 07:19:13 INFO Vacuumed table table=pending_closures29732026/11/12 07:19:13 INFO Vacuumed table table=pending_objects29742026/11/12 07:19:13 INFO Received complete push request method=POST path=/29752026/11/12 07:19:13 INFO Vacuumed table table=multipart_uploads29762026/11/12 07:19:13 INFO Received complete push request method=POST path=/29772026/11/12 07:19:13 INFO Vacuumed table table=commit_receipts29782026/11/12 07:19:13 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029792026/11/12 07:19:13 INFO Completed push id=4 objects=6 duration=2.390323ms29802026/11/12 07:19:13 INFO Vacuumed table table=pending_objects29812026/11/12 07:19:13 INFO Vacuumed table table=closures29822026/11/12 07:19:13 INFO Completed push id=14 objects=3 duration=3.18539ms29832026/11/12 07:19:13 INFO Vacuumed table table=multipart_uploads29842026/11/12 07:19:13 INFO Vacuumed table table=pending_closures29852026/11/12 07:19:13 INFO Vacuumed table table=commit_receipts29862026/11/12 07:19:13 INFO Vacuumed table table=objects29872026/11/12 07:19:13 INFO Vacuumed table table=pending_objects29882026/11/12 07:19:13 INFO Vacuumed table table=multipart_uploads29892026/11/12 07:19:13 INFO Vacuumed table table=closures29902026/11/12 07:19:13 INFO Vacuumed table table=closures29912026/11/12 07:19:13 INFO Vacuumed table table=objects29922026/11/12 07:19:13 INFO Vacuumed table table=pending_closures29932026/11/12 07:19:13 INFO Vacuumed table table=commit_receipts29942026/11/12 07:19:13 INFO Vacuumed table table=objects29952026/11/12 07:19:13 INFO Received uploads request method=POST path=/29962026/11/12 07:19:13 INFO Received complete push request method=POST path=/29972026/11/12 07:19:13 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."29982026/11/12 07:19:13 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."29992026/11/12 07:19:13 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."30002026/11/12 07:19:13 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."30012026/11/12 07:19:13 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."30022026/11/12 07:19:13 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."30032026/11/12 07:19:13 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."30042026/11/12 07:19:13 INFO Vacuumed table table=pending_objects30052026/11/12 07:19:13 INFO Vacuumed table table=multipart_uploads30062026/11/12 07:19:13 INFO Vacuumed table table=closures30072026/11/12 07:19:13 INFO Completed push id=7 objects=20 duration=2.071739ms30082026/11/12 07:19:13 INFO Aborted multipart uploads count=0 kept=030092026/11/12 07:19:13 INFO Aborted multipart uploads count=0 kept=030102026/11/12 07:19:13 INFO Vacuumed table table=objects30112026/11/12 07:19:13 INFO Aborted multipart uploads count=0 kept=030122026/11/12 07:19:13 WARN Force mode enabled - objects will be deleted immediately without grace period30132026/11/12 07:19:13 WARN Force mode enabled - objects will be deleted immediately without grace period30142026/11/12 07:19:13 INFO Received uploads request method=POST path=/30152026/11/12 07:19:13 INFO Received complete upload request method=POST path=/30162026/11/12 07:19:13 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030172026/11/12 07:19:13 INFO Vacuumed table table=pending_closures30182026/11/12 07:19:13 INFO Vacuumed table table=commit_receipts30192026/11/12 07:19:13 INFO Completed upload id=11 objects=6 duration=2.277701ms30202026/11/12 07:19:13 INFO Vacuumed table table=pending_objects30212026/11/12 07:19:13 INFO Vacuumed table table=multipart_uploads30222026/11/12 07:19:13 INFO Vacuumed table table=closures30232026/11/12 07:19:13 INFO Vacuumed table table=objects30242026/11/12 07:19:13 INFO Received complete push request method=POST path=/30252026/11/12 07:19:13 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=030262026/11/12 07:19:13 INFO Vacuumed table table=pending_closures30272026/11/12 07:19:13 INFO Completed push id=6 objects=5 duration=3.455392ms30282026/11/12 07:19:13 INFO Vacuumed table table=commit_receipts30292026/11/12 07:19:13 INFO Vacuumed table table=pending_objects30302026/11/12 07:19:13 INFO Aborted multipart uploads count=0 kept=030312026/11/12 07:19:13 INFO Vacuumed table table=multipart_uploads30322026/11/12 07:19:13 INFO Vacuumed table table=closures30332026/11/12 07:19:13 WARN Force mode enabled - objects will be deleted immediately without grace period30342026/11/12 07:19:13 INFO Vacuumed table table=objects30352026/11/12 07:19:13 INFO Received push request method=POST path=/30362026/11/12 07:19:13 INFO Received complete upload request method=POST path=/30372026/11/12 07:19:13 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030382026/11/12 07:19:13 INFO Vacuumed table table=pending_closures30392026/11/12 07:19:13 INFO Vacuumed table table=commit_receipts30402026/11/12 07:19:13 INFO Completed upload id=6 objects=6 duration=1.328552ms30412026/11/12 07:19:13 INFO Received complete push request method=POST path=/30422026/11/12 07:19:13 INFO Vacuumed table table=pending_objects30432026/11/12 07:19:13 INFO Vacuumed table table=multipart_uploads30442026/11/12 07:19:13 INFO Received push request method=POST path=/30452026/11/12 07:19:13 INFO Vacuumed table table=closures30462026/11/12 07:19:13 INFO Vacuumed table table=objects30472026/11/12 07:19:13 INFO Received push request method=POST path=/30482026/11/12 07:19:13 INFO Received push request method=POST path=/30492026/11/12 07:19:13 INFO Aborted multipart uploads count=0 kept=030502026/11/12 07:19:13 INFO Received push request method=POST path=/30512026/11/12 07:19:13 INFO Completed push id=5 objects=6 duration=4.709424ms30522026/11/12 07:19:13 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030532026/11/12 07:19:13 INFO Vacuumed table table=pending_closures30542026/11/12 07:19:13 INFO Vacuumed table table=commit_receipts30552026/11/12 07:19:13 INFO Vacuumed table table=pending_objects30562026/11/12 07:19:13 INFO Vacuumed table table=multipart_uploads30572026/11/12 07:19:13 INFO Vacuumed table table=closures30582026/11/12 07:19:13 INFO Vacuumed table table=objects30592026/11/12 07:19:13 INFO Aborted multipart uploads count=0 kept=030602026/11/12 07:19:13 INFO Aborted multipart uploads count=0 kept=030612026/11/12 07:19:13 WARN Force mode enabled - objects will be deleted immediately without grace period30622026/11/12 07:19:13 WARN Force mode enabled - objects will be deleted immediately without grace period30632026/11/12 07:19:13 INFO Aborted multipart uploads count=0 kept=030642026/11/12 07:19:13 WARN Force mode enabled - objects will be deleted immediately without grace period30652026/11/12 07:19:13 INFO Received complete push request method=POST path=/30662026/11/12 07:19:13 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030672026/11/12 07:19:13 INFO Vacuumed table table=pending_closures30682026/11/12 07:19:13 INFO Received push request method=POST path=/30692026/11/12 07:19:13 INFO Completed push id=12 objects=15 duration=2.10986ms30702026/11/12 07:19:13 INFO Vacuumed table table=commit_receipts30712026/11/12 07:19:13 INFO Vacuumed table table=pending_objects30722026/11/12 07:19:13 INFO Vacuumed table table=multipart_uploads30732026/11/12 07:19:13 INFO Vacuumed table table=closures30742026/11/12 07:19:13 INFO Vacuumed table table=objects30752026/11/12 07:19:13 INFO Received complete push request method=POST path=/30762026/11/12 07:19:13 INFO Completed push id=2 objects=10 duration=3.482652ms30772026/11/12 07:19:13 INFO Received push request method=POST path=/30782026/11/12 07:19:13 INFO Aborted multipart uploads count=0 kept=030792026/11/12 07:19:13 INFO Received complete push request method=POST path=/30802026/11/12 07:19:13 INFO Received uploads request method=POST path=/30812026/11/12 07:19:14 INFO Completed push id=5 objects=3 duration=3.938836ms30822026/11/12 07:19:14 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."30832026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=030842026/11/12 07:19:14 INFO Received complete upload request method=POST path=/30852026/11/12 07:19:14 WARN Force mode enabled - objects will be deleted immediately without grace period30862026/11/12 07:19:14 INFO Completed upload id=2 objects=7 duration=6.803603ms30872026/11/12 07:19:14 INFO Received uploads request method=POST path=/30882026/11/12 07:19:14 INFO Received push request method=POST path=/30892026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=030902026/11/12 07:19:14 INFO Received complete push request method=POST path=/30912026/11/12 07:19:14 INFO Received complete push request method=POST path=/30922026/11/12 07:19:14 INFO Completed push id=8 objects=7 duration=1.706695ms30932026/11/12 07:19:14 INFO Received push request method=POST path=/30942026/11/12 07:19:14 INFO Completed push id=1 objects=2 duration=11.717848ms30952026/11/12 07:19:14 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."30962026/11/12 07:19:14 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."30972026/11/12 07:19:14 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."30982026/11/12 07:19:14 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."30992026/11/12 07:19:14 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."31002026/11/12 07:19:14 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."31012026/11/12 07:19:14 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."31022026/11/12 07:19:14 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."31032026/11/12 07:19:14 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."31042026/11/12 07:19:14 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."31052026/11/12 07:19:14 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."31062026/11/12 07:19:14 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."31072026/11/12 07:19:14 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=031082026/11/12 07:19:14 INFO Received uploads request method=POST path=/31092026/11/12 07:19:14 INFO Vacuumed table table=pending_closures31102026/11/12 07:19:14 INFO Vacuumed table table=commit_receipts31112026/11/12 07:19:14 INFO Received complete push request method=POST path=/31122026/11/12 07:19:14 INFO Vacuumed table table=pending_objects31132026/11/12 07:19:14 INFO Vacuumed table table=multipart_uploads31142026/11/12 07:19:14 INFO Received uploads request method=POST path=/31152026/11/12 07:19:14 INFO Vacuumed table table=closures31162026/11/12 07:19:14 INFO Received push request method=POST path=/31172026/11/12 07:19:14 INFO Vacuumed table table=objects31182026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=031192026/11/12 07:19:14 INFO Completed push id=8 objects=6 duration=8.966522ms31202026/11/12 07:19:14 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031212026/11/12 07:19:14 INFO Vacuumed table table=pending_closures31222026/11/12 07:19:14 INFO Received push request method=POST path=/31232026/11/12 07:19:14 INFO Vacuumed table table=commit_receipts31242026/11/12 07:19:14 INFO Vacuumed table table=pending_objects31252026/11/12 07:19:14 INFO Vacuumed table table=multipart_uploads31262026/11/12 07:19:14 INFO Received uploads request method=POST path=/31272026/11/12 07:19:14 INFO Vacuumed table table=closures31282026/11/12 07:19:14 INFO Received uploads request method=POST path=/31292026/11/12 07:19:14 INFO Vacuumed table table=objects31302026/11/12 07:19:14 INFO Received complete upload request method=POST path=/31312026/11/12 07:19:14 INFO Received uploads request method=POST path=/31322026/11/12 07:19:14 INFO Received complete upload request method=POST path=/31332026/11/12 07:19:14 INFO Completed upload id=9 objects=16 duration=1.230571ms31342026/11/12 07:19:14 INFO Completed upload id=2 objects=26 duration=7.791211ms31352026/11/12 07:19:14 INFO Received push request method=POST path=/31362026/11/12 07:19:14 INFO Received uploads request method=POST path=/31372026/11/12 07:19:14 INFO Received push request method=POST path=/31382026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=031392026/11/12 07:19:14 INFO Received complete push request method=POST path=/31402026/11/12 07:19:14 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031412026/11/12 07:19:14 INFO Received uploads request method=POST path=/31422026/11/12 07:19:14 INFO Vacuumed table table=pending_closures31432026/11/12 07:19:14 INFO Vacuumed table table=commit_receipts31442026/11/12 07:19:14 INFO Vacuumed table table=pending_objects31452026/11/12 07:19:14 INFO Vacuumed table table=multipart_uploads31462026/11/12 07:19:14 INFO Vacuumed table table=closures31472026/11/12 07:19:14 INFO Completed push id=6 objects=26 duration=4.425001ms31482026/11/12 07:19:14 INFO Vacuumed table table=objects31492026/11/12 07:19:14 INFO Received push request method=POST path=/31502026/11/12 07:19:14 INFO Received uploads request method=POST path=/31512026/11/12 07:19:14 INFO Received push request method=POST path=/31522026/11/12 07:19:14 INFO Received uploads request method=POST path=/31532026/11/12 07:19:14 INFO Received uploads request method=POST path=/31542026/11/12 07:19:14 INFO Received push request method=POST path=/31552026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=031562026/11/12 07:19:14 INFO Received push request method=POST path=/31572026/11/12 07:19:14 INFO Received complete push request method=POST path=/31582026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=031592026/11/12 07:19:14 INFO Completed push id=5 objects=9 duration=12.351853ms31602026/11/12 07:19:14 INFO Received complete push request method=POST path=/31612026/11/12 07:19:14 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=031622026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=031632026/11/12 07:19:14 WARN Force mode enabled - objects will be deleted immediately without grace period31642026/11/12 07:19:14 INFO Completed push id=10 objects=3 duration=1.425634ms31652026/11/12 07:19:14 INFO Vacuumed table table=pending_closures31662026/11/12 07:19:14 INFO Received push request method=POST path=/31672026/11/12 07:19:14 INFO Vacuumed table table=commit_receipts31682026/11/12 07:19:14 WARN Force mode enabled - objects will be deleted immediately without grace period31692026/11/12 07:19:14 INFO Vacuumed table table=pending_objects31702026/11/12 07:19:14 INFO Vacuumed table table=multipart_uploads31712026/11/12 07:19:14 INFO Received push request method=POST path=/31722026/11/12 07:19:14 INFO Vacuumed table table=closures31732026/11/12 07:19:14 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031742026/11/12 07:19:14 INFO Vacuumed table table=objects31752026/11/12 07:19:14 INFO Vacuumed table table=pending_closures31762026/11/12 07:19:14 INFO Vacuumed table table=commit_receipts31772026/11/12 07:19:14 INFO Received push request method=POST path=/31782026/11/12 07:19:14 INFO Vacuumed table table=pending_objects31792026/11/12 07:19:14 INFO Vacuumed table table=multipart_uploads31802026/11/12 07:19:14 INFO Vacuumed table table=closures31812026/11/12 07:19:14 INFO Vacuumed table table=objects31822026/11/12 07:19:14 INFO Received uploads request method=POST path=/31832026/11/12 07:19:14 INFO Received push request method=POST path=/31842026/11/12 07:19:14 INFO Received complete upload request method=POST path=/31852026/11/12 07:19:14 INFO Received push request method=POST path=/31862026/11/12 07:19:14 INFO Received uploads request method=POST path=/31872026/11/12 07:19:14 INFO Completed upload id=15 objects=6 duration=1.702116ms31882026/11/12 07:19:14 INFO Received complete upload request method=POST path=/31892026/11/12 07:19:14 INFO Completed upload id=10 objects=6 duration=2.477483ms31902026/11/12 07:19:14 INFO Received uploads request method=POST path=/31912026/11/12 07:19:14 INFO Received push request method=POST path=/31922026/11/12 07:19:14 INFO Received push request method=POST path=/31932026/11/12 07:19:14 INFO Received complete upload request method=POST path=/31942026/11/12 07:19:14 INFO Received complete upload request method=POST path=/31952026/11/12 07:19:14 INFO Completed upload id=10 objects=17 duration=10.174514ms31962026/11/12 07:19:14 INFO Received push request method=POST path=/31972026/11/12 07:19:14 INFO Completed upload id=10 objects=2 duration=2.706165ms31982026/11/12 07:19:14 INFO Received push request method=POST path=/31992026/11/12 07:19:14 INFO Received push request method=POST path=/32002026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=032012026/11/12 07:19:14 WARN Force mode enabled - objects will be deleted immediately without grace period32022026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=032032026/11/12 07:19:14 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032042026/11/12 07:19:14 INFO Received push request method=POST path=/32052026/11/12 07:19:14 INFO Vacuumed table table=pending_closures32062026/11/12 07:19:14 INFO Vacuumed table table=commit_receipts32072026/11/12 07:19:14 WARN Force mode enabled - objects will be deleted immediately without grace period32082026/11/12 07:19:14 INFO Received uploads request method=POST path=/32092026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=032102026/11/12 07:19:14 INFO Vacuumed table table=pending_objects32112026/11/12 07:19:14 INFO Vacuumed table table=multipart_uploads32122026/11/12 07:19:14 INFO Vacuumed table table=closures32132026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=032142026/11/12 07:19:14 INFO Vacuumed table table=objects32152026/11/12 07:19:14 INFO Received push request method=POST path=/32162026/11/12 07:19:14 INFO Received complete push request method=POST path=/32172026/11/12 07:19:14 INFO Received complete push request method=POST path=/32182026/11/12 07:19:14 INFO Completed push id=16 objects=27 duration=2.377942ms32192026/11/12 07:19:14 INFO Received push request method=POST path=/32202026/11/12 07:19:14 INFO Completed push id=8 objects=6 duration=3.167089ms32212026/11/12 07:19:14 INFO Received push request method=POST path=/32222026/11/12 07:19:14 INFO Received uploads request method=POST path=/32232026/11/12 07:19:14 INFO Received uploads request method=POST path=/32242026/11/12 07:19:14 INFO Received complete push request method=POST path=/32252026/11/12 07:19:14 INFO Completed push id=14 objects=6 duration=3.661994ms32262026/11/12 07:19:14 INFO Received complete push request method=POST path=/32272026/11/12 07:19:14 INFO Completed push id=12 objects=7 duration=1.820937ms32282026/11/12 07:19:14 INFO Received push request method=POST path=/32292026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=032302026/11/12 07:19:14 INFO Received push request method=POST path=/32312026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=032322026/11/12 07:19:14 WARN Force mode enabled - objects will be deleted immediately without grace period32332026/11/12 07:19:14 INFO Received uploads request method=POST path=/32342026/11/12 07:19:14 INFO Received push request method=POST path=/32352026/11/12 07:19:14 INFO Received complete push request method=POST path=/32362026/11/12 07:19:14 INFO Completed push id=18 objects=10 duration=1.826997ms32372026/11/12 07:19:14 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=032382026/11/12 07:19:14 INFO Vacuumed table table=pending_closures32392026/11/12 07:19:14 INFO Vacuumed table table=commit_receipts32402026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=032412026/11/12 07:19:14 INFO Vacuumed table table=pending_objects32422026/11/12 07:19:14 INFO Vacuumed table table=multipart_uploads32432026/11/12 07:19:14 INFO Vacuumed table table=closures32442026/11/12 07:19:14 INFO Vacuumed table table=objects32452026/11/12 07:19:14 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=1 objects-deleted-after-grace-period=0 objects-failed-to-delete=032462026/11/12 07:19:14 INFO Vacuumed table table=pending_closures32472026/11/12 07:19:14 INFO Vacuumed table table=commit_receipts32482026/11/12 07:19:14 INFO Vacuumed table table=pending_objects32492026/11/12 07:19:14 INFO Vacuumed table table=multipart_uploads32502026/11/12 07:19:14 INFO Vacuumed table table=closures32512026/11/12 07:19:14 INFO Vacuumed table table=objects32522026/11/12 07:19:14 INFO Received push request method=POST path=/32532026/11/12 07:19:14 INFO Received push request method=POST path=/32542026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=032552026/11/12 07:19:14 INFO Received uploads request method=POST path=/32562026/11/12 07:19:14 WARN Force mode enabled - objects will be deleted immediately without grace period32572026/11/12 07:19:14 INFO Received uploads request method=POST path=/32582026/11/12 07:19:14 INFO Received uploads request method=POST path=/32592026/11/12 07:19:14 INFO Received uploads request method=POST path=/32602026/11/12 07:19:14 INFO Received uploads request method=POST path=/32612026/11/12 07:19:14 INFO Received uploads request method=POST path=/32622026/11/12 07:19:14 INFO Received complete upload request method=POST path=/32632026/11/12 07:19:14 INFO Received uploads request method=POST path=/32642026/11/12 07:19:14 INFO Received complete push request method=POST path=/32652026/11/12 07:19:14 INFO Completed upload id=17 objects=6 duration=17.555461ms32662026/11/12 07:19:14 INFO Completed push id=12 objects=2 duration=10.92948ms32672026/11/12 07:19:14 INFO Received push request method=POST path=/32682026/11/12 07:19:14 INFO Received uploads request method=POST path=/32692026/11/12 07:19:14 INFO Received push request method=POST path=/32702026/11/12 07:19:14 INFO Received uploads request method=POST path=/32712026/11/12 07:19:14 INFO Received uploads request method=POST path=/32722026/11/12 07:19:14 INFO Received push request method=POST path=/32732026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=032742026/11/12 07:19:14 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032752026/11/12 07:19:14 INFO Vacuumed table table=pending_closures32762026/11/12 07:19:14 INFO Vacuumed table table=commit_receipts32772026/11/12 07:19:14 INFO Vacuumed table table=pending_objects32782026/11/12 07:19:14 INFO Vacuumed table table=multipart_uploads32792026/11/12 07:19:14 INFO Vacuumed table table=closures32802026/11/12 07:19:14 INFO Vacuumed table table=objects32812026/11/12 07:19:14 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032822026/11/12 07:19:14 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032832026/11/12 07:19:14 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032842026/11/12 07:19:14 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032852026/11/12 07:19:14 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032862026/11/12 07:19:14 INFO Vacuumed table table=pending_closures32872026/11/12 07:19:14 INFO Vacuumed table table=pending_closures32882026/11/12 07:19:14 INFO Vacuumed table table=pending_closures32892026/11/12 07:19:14 INFO Vacuumed table table=pending_closures32902026/11/12 07:19:14 INFO Vacuumed table table=commit_receipts32912026/11/12 07:19:14 INFO Vacuumed table table=commit_receipts32922026/11/12 07:19:14 INFO Vacuumed table table=commit_receipts32932026/11/12 07:19:14 INFO Vacuumed table table=pending_closures32942026/11/12 07:19:14 INFO Vacuumed table table=commit_receipts32952026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=032962026/11/12 07:19:14 INFO Vacuumed table table=pending_objects32972026/11/12 07:19:14 INFO Garbage collection completed failed-uploads-deleted=5 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032982026/11/12 07:19:14 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=1 objects-deleted-after-grace-period=0 objects-failed-to-delete=032992026/11/12 07:19:14 INFO Vacuumed table table=multipart_uploads33002026/11/12 07:19:14 INFO Vacuumed table table=pending_objects33012026/11/12 07:19:14 INFO Vacuumed table table=commit_receipts33022026/11/12 07:19:14 INFO Vacuumed table table=closures33032026/11/12 07:19:14 INFO Vacuumed table table=pending_objects33042026/11/12 07:19:14 INFO Vacuumed table table=pending_objects33052026/11/12 07:19:14 INFO Vacuumed table table=multipart_uploads33062026/11/12 07:19:14 INFO Vacuumed table table=pending_closures33072026/11/12 07:19:14 INFO Vacuumed table table=multipart_uploads33082026/11/12 07:19:14 INFO Vacuumed table table=multipart_uploads33092026/11/12 07:19:14 WARN Force mode enabled - objects will be deleted immediately without grace period33102026/11/12 07:19:14 INFO Vacuumed table table=commit_receipts33112026/11/12 07:19:14 INFO Vacuumed table table=closures33122026/11/12 07:19:14 INFO Vacuumed table table=objects33132026/11/12 07:19:14 INFO Vacuumed table table=closures33142026/11/12 07:19:14 INFO Vacuumed table table=closures33152026/11/12 07:19:14 INFO Vacuumed table table=pending_objects33162026/11/12 07:19:14 INFO Vacuumed table table=pending_objects33172026/11/12 07:19:14 INFO Vacuumed table table=pending_closures33182026/11/12 07:19:14 INFO Vacuumed table table=objects33192026/11/12 07:19:14 INFO Vacuumed table table=multipart_uploads33202026/11/12 07:19:14 INFO Vacuumed table table=objects33212026/11/12 07:19:14 INFO Vacuumed table table=objects33222026/11/12 07:19:14 INFO Vacuumed table table=multipart_uploads33232026/11/12 07:19:14 WARN Force mode enabled - objects will be deleted immediately without grace period33242026/11/12 07:19:14 WARN Force mode enabled - objects will be deleted immediately without grace period33252026/11/12 07:19:14 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033262026/11/12 07:19:14 INFO Vacuumed table table=closures33272026/11/12 07:19:14 INFO Vacuumed table table=objects33282026/11/12 07:19:14 INFO Vacuumed table table=pending_closures33292026/11/12 07:19:14 INFO Vacuumed table table=commit_receipts33302026/11/12 07:19:14 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=033312026/11/12 07:19:14 INFO Vacuumed table table=closures33322026/11/12 07:19:14 INFO Vacuumed table table=commit_receipts33332026/11/12 07:19:14 INFO Vacuumed table table=pending_closures33342026/11/12 07:19:14 INFO Received push request method=POST path=/33352026/11/12 07:19:14 INFO Vacuumed table table=pending_objects33362026/11/12 07:19:14 INFO Vacuumed table table=objects33372026/11/12 07:19:14 INFO Vacuumed table table=commit_receipts33382026/11/12 07:19:14 INFO Vacuumed table table=multipart_uploads33392026/11/12 07:19:14 INFO Vacuumed table table=pending_objects33402026/11/12 07:19:14 INFO Vacuumed table table=multipart_uploads33412026/11/12 07:19:14 INFO Vacuumed table table=pending_objects33422026/11/12 07:19:14 INFO Vacuumed table table=closures33432026/11/12 07:19:14 INFO Vacuumed table table=multipart_uploads33442026/11/12 07:19:14 INFO Vacuumed table table=objects33452026/11/12 07:19:14 INFO Vacuumed table table=closures33462026/11/12 07:19:14 INFO Vacuumed table table=closures33472026/11/12 07:19:14 INFO Received push request method=POST path=/33482026/11/12 07:19:14 INFO Vacuumed table table=objects33492026/11/12 07:19:14 INFO Vacuumed table table=objects33502026/11/12 07:19:14 INFO Received push request method=POST path=/33512026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=033522026/11/12 07:19:14 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=3 objects-failed-to-delete=033532026/11/12 07:19:14 INFO Received uploads request method=POST path=/33542026/11/12 07:19:14 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=5 objects-failed-to-delete=033552026/11/12 07:19:14 WARN Force mode enabled - objects will be deleted immediately without grace period33562026/11/12 07:19:14 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033572026/11/12 07:19:14 INFO Received uploads request method=POST path=/33582026/11/12 07:19:14 INFO Vacuumed table table=pending_closures33592026/11/12 07:19:14 INFO Vacuumed table table=commit_receipts33602026/11/12 07:19:14 INFO Vacuumed table table=pending_closures33612026/11/12 07:19:14 INFO Vacuumed table table=pending_closures33622026/11/12 07:19:14 INFO Vacuumed table table=commit_receipts33632026/11/12 07:19:14 INFO Vacuumed table table=pending_objects33642026/11/12 07:19:14 INFO Vacuumed table table=multipart_uploads33652026/11/12 07:19:14 INFO Received uploads request method=POST path=/33662026/11/12 07:19:14 INFO Vacuumed table table=commit_receipts33672026/11/12 07:19:14 INFO Received push request method=POST path=/33682026/11/12 07:19:14 INFO Vacuumed table table=closures33692026/11/12 07:19:14 INFO Vacuumed table table=pending_objects33702026/11/12 07:19:14 INFO Vacuumed table table=multipart_uploads33712026/11/12 07:19:14 INFO Received push request method=POST path=/33722026/11/12 07:19:14 INFO Vacuumed table table=closures33732026/11/12 07:19:14 INFO Vacuumed table table=objects33742026/11/12 07:19:14 INFO Received complete push request method=POST path=/33752026/11/12 07:19:14 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."33762026/11/12 07:19:14 INFO Vacuumed table table=pending_objects33772026/11/12 07:19:14 INFO Vacuumed table table=objects33782026/11/12 07:19:14 INFO Vacuumed table table=multipart_uploads33792026/11/12 07:19:14 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."33802026/11/12 07:19:14 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."33812026/11/12 07:19:14 INFO Vacuumed table table=closures33822026/11/12 07:19:14 INFO Completed push id=8 objects=6 duration=3.674874ms33832026/11/12 07:19:14 INFO Received push request method=POST path=/33842026/11/12 07:19:14 INFO Vacuumed table table=objects33852026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=033862026/11/12 07:19:14 INFO Received uploads request method=POST path=/33872026/11/12 07:19:14 WARN Force mode enabled - objects will be deleted immediately without grace period33882026/11/12 07:19:14 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033892026/11/12 07:19:14 INFO Received complete push request method=POST path=/33902026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=033912026/11/12 07:19:14 INFO Vacuumed table table=pending_closures33922026/11/12 07:19:14 INFO Vacuumed table table=commit_receipts33932026/11/12 07:19:14 INFO Vacuumed table table=pending_objects33942026/11/12 07:19:14 WARN Force mode enabled - objects will be deleted immediately without grace period33952026/11/12 07:19:14 INFO Vacuumed table table=multipart_uploads33962026/11/12 07:19:14 INFO Vacuumed table table=closures33972026/11/12 07:19:14 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."33982026/11/12 07:19:14 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.narinfo error="Access Denied."33992026/11/12 07:19:14 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."34002026/11/12 07:19:14 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."34012026/11/12 07:19:14 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."34022026/11/12 07:19:14 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."34032026/11/12 07:19:14 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."34042026/11/12 07:19:14 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."34052026/11/12 07:19:14 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."34062026/11/12 07:19:14 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."34072026/11/12 07:19:14 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."34082026/11/12 07:19:14 INFO Vacuumed table table=objects34092026/11/12 07:19:14 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."34102026/11/12 07:19:14 ERROR failed to remove object object=nar/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.nar.zst error="Access Denied."34112026/11/12 07:19:14 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."34122026/11/12 07:19:14 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."34132026/11/12 07:19:14 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."34142026/11/12 07:19:14 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034152026/11/12 07:19:14 INFO Completed push id=19 objects=8 duration=3.828756ms34162026/11/12 07:19:14 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."34172026/11/12 07:19:14 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."34182026/11/12 07:19:14 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."34192026/11/12 07:19:14 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."34202026/11/12 07:19:14 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."34212026/11/12 07:19:14 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."34222026/11/12 07:19:14 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."34232026/11/12 07:19:14 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."34242026/11/12 07:19:14 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."34252026/11/12 07:19:14 INFO Vacuumed table table=pending_closures34262026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=034272026/11/12 07:19:14 INFO Vacuumed table table=commit_receipts34282026/11/12 07:19:14 INFO Vacuumed table table=pending_objects34292026/11/12 07:19:14 INFO Received push request method=POST path=/34302026/11/12 07:19:14 INFO Vacuumed table table=multipart_uploads34312026/11/12 07:19:14 INFO Received uploads request method=POST path=/34322026/11/12 07:19:14 INFO Vacuumed table table=closures34332026/11/12 07:19:14 INFO Vacuumed table table=objects34342026/11/12 07:19:14 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034352026/11/12 07:19:14 INFO Received uploads request method=POST path=/34362026/11/12 07:19:14 INFO Vacuumed table table=pending_closures34372026/11/12 07:19:14 INFO Vacuumed table table=commit_receipts34382026/11/12 07:19:14 INFO Vacuumed table table=pending_objects34392026/11/12 07:19:14 INFO Vacuumed table table=multipart_uploads34402026/11/12 07:19:14 INFO Vacuumed table table=closures34412026/11/12 07:19:14 INFO Vacuumed table table=objects34422026/11/12 07:19:14 INFO Received uploads request method=POST path=/34432026/11/12 07:19:14 INFO Received complete upload request method=POST path=/34442026/11/12 07:19:14 INFO Received push request method=POST path=/34452026/11/12 07:19:14 INFO Completed upload id=3 objects=8 duration=2.339041ms34462026/11/12 07:19:14 INFO Received push request method=POST path=/34472026/11/12 07:19:14 INFO Received push request method=POST path=/34482026/11/12 07:19:14 INFO Received push request method=POST path=/34492026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=034502026/11/12 07:19:14 INFO Received uploads request method=POST path=/34512026/11/12 07:19:14 WARN Force mode enabled - objects will be deleted immediately without grace period34522026/11/12 07:19:14 INFO Received push request method=POST path=/34532026/11/12 07:19:14 INFO Received complete push request method=POST path=/34542026/11/12 07:19:14 INFO Received complete push request method=POST path=/34552026/11/12 07:19:14 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034562026/11/12 07:19:14 INFO Vacuumed table table=pending_closures34572026/11/12 07:19:14 INFO Vacuumed table table=commit_receipts34582026/11/12 07:19:14 INFO Completed push id=22 objects=3 duration=2.689105ms34592026/11/12 07:19:14 INFO Received push request method=POST path=/34602026/11/12 07:19:14 INFO Vacuumed table table=pending_objects34612026/11/12 07:19:14 INFO Vacuumed table table=multipart_uploads34622026/11/12 07:19:14 INFO Vacuumed table table=closures34632026/11/12 07:19:14 INFO Completed push id=11 objects=8 duration=3.471692ms34642026/11/12 07:19:14 INFO Vacuumed table table=objects34652026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=034662026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=034672026/11/12 07:19:14 INFO Received complete push request method=POST path=/34682026/11/12 07:19:14 WARN Force mode enabled - objects will be deleted immediately without grace period34692026/11/12 07:19:14 INFO Received complete upload request method=POST path=/34702026/11/12 07:19:14 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034712026/11/12 07:19:14 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034722026/11/12 07:19:14 INFO Received complete push request method=POST path=/34732026/11/12 07:19:14 INFO Vacuumed table table=pending_closures34742026/11/12 07:19:14 INFO Completed push id=11 objects=11 duration=3.388131ms34752026/11/12 07:19:14 INFO Vacuumed table table=pending_closures34762026/11/12 07:19:14 INFO Vacuumed table table=commit_receipts34772026/11/12 07:19:14 INFO Vacuumed table table=commit_receipts34782026/11/12 07:19:14 INFO Completed upload id=19 objects=25 duration=2.837206ms34792026/11/12 07:19:14 INFO Vacuumed table table=pending_objects34802026/11/12 07:19:14 INFO Received uploads request method=POST path=/34812026/11/12 07:19:14 INFO Vacuumed table table=multipart_uploads34822026/11/12 07:19:14 INFO Vacuumed table table=pending_objects34832026/11/12 07:19:14 INFO Vacuumed table table=multipart_uploads34842026/11/12 07:19:14 INFO Vacuumed table table=closures34852026/11/12 07:19:14 INFO Vacuumed table table=closures34862026/11/12 07:19:14 INFO Vacuumed table table=objects34872026/11/12 07:19:14 INFO Vacuumed table table=objects34882026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=034892026/11/12 07:19:14 INFO Received push request method=POST path=/34902026/11/12 07:19:14 INFO Received uploads request method=POST path=/34912026/11/12 07:19:14 INFO Received push request method=POST path=/34922026/11/12 07:19:14 INFO Completed push id=1 objects=5 duration=7.028765ms34932026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=034942026/11/12 07:19:14 WARN Force mode enabled - objects will be deleted immediately without grace period34952026/11/12 07:19:14 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034962026/11/12 07:19:14 INFO Vacuumed table table=pending_closures34972026/11/12 07:19:14 INFO Vacuumed table table=commit_receipts34982026/11/12 07:19:14 INFO Vacuumed table table=pending_objects34992026/11/12 07:19:14 INFO Vacuumed table table=multipart_uploads35002026/11/12 07:19:14 INFO Vacuumed table table=closures35012026/11/12 07:19:14 INFO Vacuumed table table=objects35022026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=035032026/11/12 07:19:14 WARN Force mode enabled - objects will be deleted immediately without grace period35042026/11/12 07:19:14 INFO Received uploads request method=POST path=/35052026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=035062026/11/12 07:19:14 INFO Received complete upload request method=POST path=/35072026/11/12 07:19:14 INFO Received push request method=POST path=/35082026/11/12 07:19:14 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035092026/11/12 07:19:14 INFO Completed upload id=15 objects=9 duration=1.321652ms35102026/11/12 07:19:14 WARN Force mode enabled - objects will be deleted immediately without grace period35112026/11/12 07:19:14 INFO Vacuumed table table=pending_closures35122026/11/12 07:19:14 INFO Received complete upload request method=POST path=/35132026/11/12 07:19:14 INFO Received push request method=POST path=/35142026/11/12 07:19:14 INFO Vacuumed table table=commit_receipts35152026/11/12 07:19:14 INFO Vacuumed table table=pending_objects35162026/11/12 07:19:14 INFO Received push request method=POST path=/35172026/11/12 07:19:14 INFO Vacuumed table table=multipart_uploads35182026/11/12 07:19:14 INFO Vacuumed table table=closures35192026/11/12 07:19:14 INFO Completed upload id=17 objects=25 duration=3.000948ms35202026/11/12 07:19:14 INFO Vacuumed table table=objects35212026/11/12 07:19:14 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=035222026/11/12 07:19:14 INFO Received push request method=POST path=/35232026/11/12 07:19:14 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."35242026/11/12 07:19:14 INFO Vacuumed table table=pending_closures35252026/11/12 07:19:14 INFO Vacuumed table table=commit_receipts35262026/11/12 07:19:14 INFO Vacuumed table table=pending_objects35272026/11/12 07:19:14 INFO Vacuumed table table=multipart_uploads35282026/11/12 07:19:14 INFO Vacuumed table table=closures35292026/11/12 07:19:14 INFO Received complete push request method=POST path=/35302026/11/12 07:19:14 INFO Vacuumed table table=objects35312026/11/12 07:19:14 INFO Completed push id=9 objects=24 duration=1.498134ms35322026/11/12 07:19:14 INFO Received uploads request method=POST path=/35332026/11/12 07:19:14 INFO Received complete push request method=POST path=/35342026/11/12 07:19:14 INFO Received complete push request method=POST path=/35352026/11/12 07:19:14 INFO Received uploads request method=POST path=/35362026/11/12 07:19:14 INFO Received push request method=POST path=/35372026/11/12 07:19:14 INFO Completed push id=13 objects=6 duration=2.956707ms35382026/11/12 07:19:14 INFO Received complete push request method=POST path=/35392026/11/12 07:19:14 INFO Completed push id=6 objects=8 duration=3.340611ms35402026/11/12 07:19:14 INFO Received push request method=POST path=/35412026/11/12 07:19:14 INFO Received complete upload request method=POST path=/35422026/11/12 07:19:14 INFO Received push request method=POST path=/35432026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=035442026/11/12 07:19:14 INFO Completed push id=4 objects=6 duration=1.832717ms35452026/11/12 07:19:14 INFO Received push request method=POST path=/35462026/11/12 07:19:14 INFO Completed upload id=4 objects=14 duration=3.327611ms35472026/11/12 07:19:14 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035482026/11/12 07:19:14 INFO Received uploads request method=POST path=/35492026/11/12 07:19:14 INFO Vacuumed table table=pending_closures35502026/11/12 07:19:14 INFO Vacuumed table table=commit_receipts35512026/11/12 07:19:14 INFO Received push request method=POST path=/35522026/11/12 07:19:14 INFO Vacuumed table table=pending_objects35532026/11/12 07:19:14 INFO Vacuumed table table=multipart_uploads35542026/11/12 07:19:14 INFO Vacuumed table table=closures35552026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=035562026/11/12 07:19:14 INFO Vacuumed table table=objects35572026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=035582026/11/12 07:19:14 INFO Received uploads request method=POST path=/35592026/11/12 07:19:14 INFO Received uploads request method=POST path=/35602026/11/12 07:19:14 WARN Force mode enabled - objects will be deleted immediately without grace period35612026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=035622026/11/12 07:19:14 INFO Received complete push request method=POST path=/35632026/11/12 07:19:14 WARN Force mode enabled - objects will be deleted immediately without grace period35642026/11/12 07:19:14 INFO Received uploads request method=POST path=/35652026/11/12 07:19:14 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035662026/11/12 07:19:14 INFO Completed push id=21 objects=17 duration=2.068419ms35672026/11/12 07:19:14 INFO Received push request method=POST path=/35682026/11/12 07:19:14 INFO Vacuumed table table=pending_closures35692026/11/12 07:19:14 INFO Vacuumed table table=commit_receipts35702026/11/12 07:19:14 INFO Received push request method=POST path=/35712026/11/12 07:19:14 INFO Received complete upload request method=POST path=/35722026/11/12 07:19:14 INFO Vacuumed table table=pending_objects35732026/11/12 07:19:14 INFO Vacuumed table table=multipart_uploads35742026/11/12 07:19:14 INFO Completed upload id=21 objects=2 duration=1.358473ms35752026/11/12 07:19:14 INFO Vacuumed table table=closures35762026/11/12 07:19:14 INFO Vacuumed table table=objects35772026/11/12 07:19:14 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."35782026/11/12 07:19:14 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."35792026/11/12 07:19:14 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."35802026/11/12 07:19:14 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."35812026/11/12 07:19:14 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."35822026/11/12 07:19:14 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."35832026/11/12 07:19:14 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."35842026/11/12 07:19:14 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."35852026/11/12 07:19:14 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."35862026/11/12 07:19:14 INFO Received uploads request method=POST path=/35872026/11/12 07:19:14 INFO Received push request method=POST path=/35882026/11/12 07:19:14 INFO Received uploads request method=POST path=/35892026/11/12 07:19:14 INFO Received uploads request method=POST path=/35902026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=035912026/11/12 07:19:14 INFO Received complete upload request method=POST path=/35922026/11/12 07:19:14 INFO Received uploads request method=POST path=/35932026/11/12 07:19:14 WARN Force mode enabled - objects will be deleted immediately without grace period35942026/11/12 07:19:14 INFO Completed upload id=5 objects=5 duration=1.795217ms35952026/11/12 07:19:14 INFO Received push request method=POST path=/35962026/11/12 07:19:14 INFO Received complete push request method=POST path=/35972026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=035982026/11/12 07:19:14 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035992026/11/12 07:19:14 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."36002026/11/12 07:19:14 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."36012026/11/12 07:19:14 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."36022026/11/12 07:19:14 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."36032026/11/12 07:19:14 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."36042026/11/12 07:19:14 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."36052026/11/12 07:19:14 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."36062026/11/12 07:19:14 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."36072026/11/12 07:19:14 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."36082026/11/12 07:19:14 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."36092026/11/12 07:19:14 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."36102026/11/12 07:19:14 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."36112026/11/12 07:19:14 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."36122026/11/12 07:19:14 ERROR failed to remove object object=realisations/sha256:aaaaaaaa!out.doi error="Access Denied."36132026/11/12 07:19:14 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."36142026/11/12 07:19:14 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."36152026/11/12 07:19:14 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."36162026/11/12 07:19:14 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."36172026/11/12 07:19:14 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."36182026/11/12 07:19:14 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."36192026/11/12 07:19:14 INFO Completed push id=19 objects=7 duration=1.714336ms36202026/11/12 07:19:14 WARN Force mode enabled - objects will be deleted immediately without grace period36212026/11/12 07:19:14 INFO Vacuumed table table=pending_closures36222026/11/12 07:19:14 INFO Vacuumed table table=commit_receipts36232026/11/12 07:19:14 INFO Received complete upload request method=POST path=/36242026/11/12 07:19:14 INFO Received uploads request method=POST path=/36252026/11/12 07:19:14 INFO Vacuumed table table=pending_objects36262026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=036272026/11/12 07:19:14 INFO Vacuumed table table=multipart_uploads36282026/11/12 07:19:14 INFO Completed upload id=9 objects=2 duration=1.331232ms36292026/11/12 07:19:14 WARN Force mode enabled - objects will be deleted immediately without grace period36302026/11/12 07:19:14 INFO Vacuumed table table=closures36312026/11/12 07:19:14 INFO Vacuumed table table=objects36322026/11/12 07:19:14 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036332026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=036342026/11/12 07:19:14 INFO Vacuumed table table=pending_closures36352026/11/12 07:19:14 INFO Received uploads request method=POST path=/36362026/11/12 07:19:14 WARN Force mode enabled - objects will be deleted immediately without grace period36372026/11/12 07:19:14 INFO Vacuumed table table=commit_receipts36382026/11/12 07:19:14 INFO Vacuumed table table=pending_objects36392026/11/12 07:19:14 INFO Vacuumed table table=multipart_uploads36402026/11/12 07:19:14 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036412026/11/12 07:19:14 INFO Vacuumed table table=closures36422026/11/12 07:19:14 INFO Received uploads request method=POST path=/36432026/11/12 07:19:14 INFO Received uploads request method=POST path=/36442026/11/12 07:19:14 INFO Vacuumed table table=pending_closures36452026/11/12 07:19:14 INFO Received complete upload request method=POST path=/36462026/11/12 07:19:14 INFO Vacuumed table table=objects36472026/11/12 07:19:14 INFO Received uploads request method=POST path=/36482026/11/12 07:19:14 INFO Vacuumed table table=commit_receipts36492026/11/12 07:19:14 INFO Completed upload id=10 objects=10 duration=911.949µs36502026/11/12 07:19:14 INFO Vacuumed table table=pending_objects36512026/11/12 07:19:14 INFO Received uploads request method=POST path=/36522026/11/12 07:19:14 INFO Vacuumed table table=multipart_uploads36532026/11/12 07:19:14 INFO Received push request method=POST path=/36542026/11/12 07:19:14 INFO Vacuumed table table=closures36552026/11/12 07:19:14 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=9 objects-failed-to-delete=036562026/11/12 07:19:14 INFO Vacuumed table table=objects36572026/11/12 07:19:14 INFO Received push request method=POST path=/36582026/11/12 07:19:14 INFO Received complete push request method=POST path=/36592026/11/12 07:19:14 INFO Vacuumed table table=pending_closures36602026/11/12 07:19:14 INFO Vacuumed table table=commit_receipts36612026/11/12 07:19:14 INFO Completed push id=8 objects=3 duration=1.783156ms36622026/11/12 07:19:14 INFO Vacuumed table table=pending_objects36632026/11/12 07:19:14 INFO Vacuumed table table=multipart_uploads36642026/11/12 07:19:14 INFO Vacuumed table table=closures36652026/11/12 07:19:14 INFO Received uploads request method=POST path=/36662026/11/12 07:19:14 INFO Vacuumed table table=objects36672026/11/12 07:19:14 INFO Received complete upload request method=POST path=/36682026/11/12 07:19:14 INFO Completed upload id=7 objects=4 duration=1.489194ms36692026/11/12 07:19:14 INFO Received uploads request method=POST path=/36702026/11/12 07:19:14 INFO Received complete push request method=POST path=/36712026/11/12 07:19:14 INFO Received push request method=POST path=/36722026/11/12 07:19:14 INFO Completed push id=17 objects=6 duration=1.984278ms36732026/11/12 07:19:14 INFO Received push request method=POST path=/36742026/11/12 07:19:14 INFO Received uploads request method=POST path=/36752026/11/12 07:19:14 INFO Received uploads request method=POST path=/36762026/11/12 07:19:14 INFO Received push request method=POST path=/36772026/11/12 07:19:14 INFO Received push request method=POST path=/36782026/11/12 07:19:14 INFO Received push request method=POST path=/36792026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=036802026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=036812026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=036822026/11/12 07:19:14 INFO Received push request method=POST path=/36832026/11/12 07:19:14 WARN Force mode enabled - objects will be deleted immediately without grace period36842026/11/12 07:19:14 WARN Force mode enabled - objects will be deleted immediately without grace period36852026/11/12 07:19:14 INFO Received push request method=POST path=/36862026/11/12 07:19:14 INFO Received uploads request method=POST path=/36872026/11/12 07:19:14 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."36882026/11/12 07:19:14 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."36892026/11/12 07:19:14 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036902026/11/12 07:19:14 INFO Received uploads request method=POST path=/36912026/11/12 07:19:14 INFO Received push request method=POST path=/36922026/11/12 07:19:14 INFO Vacuumed table table=pending_closures36932026/11/12 07:19:14 INFO Received uploads request method=POST path=/36942026/11/12 07:19:14 INFO Received uploads request method=POST path=/36952026/11/12 07:19:14 INFO Vacuumed table table=commit_receipts36962026/11/12 07:19:14 INFO Received uploads request method=POST path=/36972026/11/12 07:19:14 INFO Vacuumed table table=pending_objects36982026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=036992026/11/12 07:19:14 INFO Vacuumed table table=multipart_uploads37002026/11/12 07:19:14 INFO Vacuumed table table=closures37012026/11/12 07:19:14 INFO Vacuumed table table=objects37022026/11/12 07:19:14 INFO Received uploads request method=POST path=/37032026/11/12 07:19:14 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037042026/11/12 07:19:14 INFO Vacuumed table table=pending_closures37052026/11/12 07:19:14 INFO Vacuumed table table=commit_receipts37062026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=037072026/11/12 07:19:14 INFO Received uploads request method=POST path=/37082026/11/12 07:19:14 INFO Vacuumed table table=pending_objects37092026/11/12 07:19:14 INFO Vacuumed table table=multipart_uploads37102026/11/12 07:19:14 INFO Received push request method=POST path=/37112026/11/12 07:19:14 INFO Vacuumed table table=closures37122026/11/12 07:19:14 INFO Received push request method=POST path=/37132026/11/12 07:19:14 WARN Force mode enabled - objects will be deleted immediately without grace period37142026/11/12 07:19:14 INFO Vacuumed table table=objects37152026/11/12 07:19:14 INFO Received complete push request method=POST path=/37162026/11/12 07:19:14 INFO Received push request method=POST path=/37172026/11/12 07:19:14 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=16 objects-failed-to-delete=037182026/11/12 07:19:14 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037192026/11/12 07:19:14 INFO Completed push id=4 objects=7 duration=2.364982ms37202026/11/12 07:19:14 INFO Received complete upload request method=POST path=/37212026/11/12 07:19:14 INFO Vacuumed table table=pending_closures37222026/11/12 07:19:14 INFO Received uploads request method=POST path=/37232026/11/12 07:19:14 INFO Vacuumed table table=pending_closures37242026/11/12 07:19:14 INFO Vacuumed table table=commit_receipts37252026/11/12 07:19:14 INFO Completed upload id=13 objects=10 duration=1.007329ms37262026/11/12 07:19:14 INFO Vacuumed table table=commit_receipts37272026/11/12 07:19:14 INFO Received push request method=POST path=/37282026/11/12 07:19:14 INFO Received complete push request method=POST path=/37292026/11/12 07:19:14 INFO Vacuumed table table=pending_objects37302026/11/12 07:19:14 INFO Vacuumed table table=pending_objects37312026/11/12 07:19:14 INFO Vacuumed table table=multipart_uploads37322026/11/12 07:19:14 INFO Vacuumed table table=multipart_uploads37332026/11/12 07:19:14 INFO Received uploads request method=POST path=/37342026/11/12 07:19:14 INFO Vacuumed table table=closures37352026/11/12 07:19:14 INFO Vacuumed table table=closures37362026/11/12 07:19:14 INFO Vacuumed table table=objects37372026/11/12 07:19:14 INFO Vacuumed table table=objects37382026/11/12 07:19:14 INFO Completed push id=7 objects=24 duration=3.782874ms37392026/11/12 07:19:14 INFO Received complete upload request method=POST path=/37402026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=037412026/11/12 07:19:14 WARN Force mode enabled - objects will be deleted immediately without grace period37422026/11/12 07:19:14 INFO Received uploads request method=POST path=/37432026/11/12 07:19:14 INFO Completed upload id=23 objects=18 duration=2.606024ms37442026/11/12 07:19:14 INFO Received complete push request method=POST path=/37452026/11/12 07:19:14 INFO Received complete upload request method=POST path=/37462026/11/12 07:19:14 INFO Received uploads request method=POST path=/37472026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=037482026/11/12 07:19:14 INFO Completed upload id=23 objects=16 duration=2.197281ms37492026/11/12 07:19:14 WARN Force mode enabled - objects will be deleted immediately without grace period37502026/11/12 07:19:14 INFO Received push request method=POST path=/37512026/11/12 07:19:14 INFO Completed push id=5 objects=8 duration=3.932637ms37522026/11/12 07:19:14 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=3 objects-failed-to-delete=037532026/11/12 07:19:14 INFO Received push request method=POST path=/37542026/11/12 07:19:14 INFO Received push request method=POST path=/37552026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=037562026/11/12 07:19:14 INFO Received uploads request method=POST path=/37572026/11/12 07:19:14 INFO Vacuumed table table=pending_closures37582026/11/12 07:19:14 INFO Received uploads request method=POST path=/37592026/11/12 07:19:14 INFO Received push request method=POST path=/37602026/11/12 07:19:14 INFO Vacuumed table table=commit_receipts37612026/11/12 07:19:14 INFO Vacuumed table table=pending_objects37622026/11/12 07:19:14 INFO Vacuumed table table=multipart_uploads37632026/11/12 07:19:14 INFO Vacuumed table table=closures37642026/11/12 07:19:14 INFO Received uploads request method=POST path=/37652026/11/12 07:19:14 INFO Received push request method=POST path=/37662026/11/12 07:19:14 INFO Vacuumed table table=objects37672026/11/12 07:19:14 INFO Received uploads request method=POST path=/37682026/11/12 07:19:14 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=037692026/11/12 07:19:14 INFO Received push request method=POST path=/37702026/11/12 07:19:14 INFO Vacuumed table table=pending_closures37712026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=037722026/11/12 07:19:14 WARN Force mode enabled - objects will be deleted immediately without grace period37732026/11/12 07:19:14 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037742026/11/12 07:19:14 INFO Vacuumed table table=commit_receipts37752026/11/12 07:19:14 INFO Vacuumed table table=pending_closures37762026/11/12 07:19:14 INFO Received complete upload request method=POST path=/37772026/11/12 07:19:14 INFO Vacuumed table table=commit_receipts37782026/11/12 07:19:14 INFO Vacuumed table table=pending_objects37792026/11/12 07:19:14 INFO Received complete upload request method=POST path=/37802026/11/12 07:19:14 INFO Vacuumed table table=multipart_uploads37812026/11/12 07:19:14 INFO Vacuumed table table=pending_objects37822026/11/12 07:19:14 INFO Received uploads request method=POST path=/37832026/11/12 07:19:14 INFO Vacuumed table table=multipart_uploads37842026/11/12 07:19:14 INFO Received uploads request method=POST path=/37852026/11/12 07:19:14 INFO Received complete upload request method=POST path=/37862026/11/12 07:19:14 INFO Vacuumed table table=closures37872026/11/12 07:19:14 INFO Completed upload id=12 objects=26 duration=3.935477ms37882026/11/12 07:19:14 INFO Completed upload id=9 objects=7 duration=5.178388ms37892026/11/12 07:19:14 INFO Vacuumed table table=closures37902026/11/12 07:19:14 INFO Completed upload id=21 objects=7 duration=1.967998ms37912026/11/12 07:19:14 INFO Vacuumed table table=objects37922026/11/12 07:19:14 INFO Vacuumed table table=objects37932026/11/12 07:19:14 INFO Received complete upload request method=POST path=/37942026/11/12 07:19:14 INFO Received uploads request method=POST path=/37952026/11/12 07:19:14 INFO Completed upload id=14 objects=16 duration=2.487683ms37962026/11/12 07:19:14 INFO Received push request method=POST path=/37972026/11/12 07:19:14 INFO Received uploads request method=POST path=/37982026/11/12 07:19:14 INFO Received uploads request method=POST path=/37992026/11/12 07:19:14 INFO Received uploads request method=POST path=/38002026/11/12 07:19:14 INFO Received push request method=POST path=/38012026/11/12 07:19:14 INFO Received uploads request method=POST path=/38022026/11/12 07:19:14 INFO Received uploads request method=POST path=/38032026/11/12 07:19:14 INFO Received complete push request method=POST path=/38042026/11/12 07:19:14 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=11 objects-failed-to-delete=038052026/11/12 07:19:14 INFO Completed push id=14 objects=3 duration=1.00441ms38062026/11/12 07:19:14 INFO Vacuumed table table=pending_closures38072026/11/12 07:19:14 INFO Vacuumed table table=commit_receipts38082026/11/12 07:19:14 INFO Received complete upload request method=POST path=/38092026/11/12 07:19:14 INFO Vacuumed table table=pending_objects38102026/11/12 07:19:14 INFO Vacuumed table table=multipart_uploads38112026/11/12 07:19:14 INFO Vacuumed table table=closures38122026/11/12 07:19:14 INFO Completed upload id=22 objects=22 duration=1.199371ms38132026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=038142026/11/12 07:19:14 INFO Vacuumed table table=objects38152026/11/12 07:19:14 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038162026/11/12 07:19:14 INFO Vacuumed table table=pending_closures38172026/11/12 07:19:14 INFO Vacuumed table table=commit_receipts38182026/11/12 07:19:14 INFO Vacuumed table table=pending_objects38192026/11/12 07:19:14 INFO Vacuumed table table=multipart_uploads38202026/11/12 07:19:14 INFO Received push request method=POST path=/38212026/11/12 07:19:14 INFO Received push request method=POST path=/38222026/11/12 07:19:14 INFO Vacuumed table table=closures38232026/11/12 07:19:14 INFO Received complete push request method=POST path=/38242026/11/12 07:19:14 INFO Vacuumed table table=objects38252026/11/12 07:19:14 INFO Received uploads request method=POST path=/38262026/11/12 07:19:14 INFO Received push request method=POST path=/38272026/11/12 07:19:14 INFO Received push request method=POST path=/38282026/11/12 07:19:14 INFO Completed push id=16 objects=4 duration=2.691445ms38292026/11/12 07:19:14 INFO Received complete push request method=POST path=/38302026/11/12 07:19:14 INFO Received push request method=POST path=/38312026/11/12 07:19:14 INFO Received complete push request method=POST path=/38322026/11/12 07:19:14 INFO Received push request method=POST path=/38332026/11/12 07:19:14 INFO Completed push id=7 objects=7 duration=1.459933ms38342026/11/12 07:19:14 INFO Received push request method=POST path=/38352026/11/12 07:19:14 INFO Completed push id=13 objects=12 duration=3.602793ms38362026/11/12 07:19:14 INFO Received push request method=POST path=/38372026/11/12 07:19:14 INFO Received complete push request method=POST path=/38382026/11/12 07:19:14 INFO Received uploads request method=POST path=/38392026/11/12 07:19:14 INFO Completed push id=17 objects=11 duration=2.601244ms38402026/11/12 07:19:14 INFO Received uploads request method=POST path=/38412026/11/12 07:19:14 INFO Received complete push request method=POST path=/38422026/11/12 07:19:14 INFO Received uploads request method=POST path=/38432026/11/12 07:19:14 INFO Received push request method=POST path=/38442026/11/12 07:19:14 INFO Completed push id=16 objects=8 duration=1.820957ms38452026/11/12 07:19:14 INFO Received uploads request method=POST path=/38462026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=038472026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=038482026/11/12 07:19:14 INFO Received complete push request method=POST path=/38492026/11/12 07:19:14 INFO Received uploads request method=POST path=/38502026/11/12 07:19:14 INFO Completed push id=26 objects=5 duration=1.666775ms38512026/11/12 07:19:14 INFO Received uploads request method=POST path=/38522026/11/12 07:19:14 INFO Received push request method=POST path=/38532026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=038542026/11/12 07:19:14 INFO Received complete upload request method=POST path=/38552026/11/12 07:19:14 INFO Completed upload id=20 objects=5 duration=1.379013ms38562026/11/12 07:19:14 INFO Received push request method=POST path=/38572026/11/12 07:19:14 INFO Received complete push request method=POST path=/38582026/11/12 07:19:14 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."38592026/11/12 07:19:14 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."38602026/11/12 07:19:14 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."38612026/11/12 07:19:14 INFO Received push request method=POST path=/38622026/11/12 07:19:14 WARN Force mode enabled - objects will be deleted immediately without grace period38632026/11/12 07:19:14 INFO Completed push id=13 objects=2 duration=1.231592ms38642026/11/12 07:19:14 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=038652026/11/12 07:19:14 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038662026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=038672026/11/12 07:19:14 INFO Vacuumed table table=pending_closures38682026/11/12 07:19:14 INFO Vacuumed table table=pending_closures38692026/11/12 07:19:14 INFO Vacuumed table table=commit_receipts38702026/11/12 07:19:14 INFO Vacuumed table table=commit_receipts38712026/11/12 07:19:14 INFO Vacuumed table table=pending_objects38722026/11/12 07:19:14 INFO Received complete push request method=POST path=/38732026/11/12 07:19:14 INFO Vacuumed table table=multipart_uploads38742026/11/12 07:19:14 INFO Vacuumed table table=pending_objects38752026/11/12 07:19:14 INFO Received uploads request method=POST path=/38762026/11/12 07:19:14 INFO Vacuumed table table=closures38772026/11/12 07:19:14 WARN Force mode enabled - objects will be deleted immediately without grace period38782026/11/12 07:19:14 INFO Vacuumed table table=multipart_uploads38792026/11/12 07:19:14 INFO Vacuumed table table=closures38802026/11/12 07:19:14 INFO Vacuumed table table=objects38812026/11/12 07:19:14 INFO Vacuumed table table=objects38822026/11/12 07:19:14 INFO Received uploads request method=POST path=/38832026/11/12 07:19:14 INFO Received complete upload request method=POST path=/38842026/11/12 07:19:14 INFO Completed push id=14 objects=4 duration=3.433772ms38852026/11/12 07:19:14 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038862026/11/12 07:19:14 INFO Received push request method=POST path=/38872026/11/12 07:19:14 INFO Completed upload id=24 objects=14 duration=949.709µs38882026/11/12 07:19:14 INFO Received push request method=POST path=/38892026/11/12 07:19:14 INFO Vacuumed table table=pending_closures38902026/11/12 07:19:14 INFO Received uploads request method=POST path=/38912026/11/12 07:19:14 INFO Vacuumed table table=commit_receipts38922026/11/12 07:19:14 INFO Vacuumed table table=pending_objects38932026/11/12 07:19:14 INFO Vacuumed table table=multipart_uploads38942026/11/12 07:19:14 INFO Received complete upload request method=POST path=/38952026/11/12 07:19:14 INFO Vacuumed table table=closures38962026/11/12 07:19:14 INFO Received complete upload request method=POST path=/38972026/11/12 07:19:14 INFO Completed upload id=27 objects=13 duration=1.239952ms38982026/11/12 07:19:14 INFO Vacuumed table table=objects38992026/11/12 07:19:14 INFO Completed upload id=18 objects=11 duration=1.308352ms39002026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=039012026/11/12 07:19:14 WARN Force mode enabled - objects will be deleted immediately without grace period39022026/11/12 07:19:14 INFO Received push request method=POST path=/39032026/11/12 07:19:14 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039042026/11/12 07:19:14 INFO Received uploads request method=POST path=/39052026/11/12 07:19:14 INFO Vacuumed table table=pending_closures39062026/11/12 07:19:14 INFO Vacuumed table table=commit_receipts39072026/11/12 07:19:14 INFO Received complete push request method=POST path=/39082026/11/12 07:19:14 INFO Received complete upload request method=POST path=/39092026/11/12 07:19:14 INFO Vacuumed table table=pending_objects39102026/11/12 07:19:14 INFO Completed push id=19 objects=3 duration=986.709µs39112026/11/12 07:19:14 INFO Vacuumed table table=multipart_uploads39122026/11/12 07:19:14 INFO Completed upload id=27 objects=4 duration=1.04969ms39132026/11/12 07:19:14 INFO Vacuumed table table=closures39142026/11/12 07:19:14 INFO Received push request method=POST path=/39152026/11/12 07:19:14 INFO Vacuumed table table=objects39162026/11/12 07:19:14 INFO Received uploads request method=POST path=/39172026/11/12 07:19:14 INFO Received uploads request method=POST path=/39182026/11/12 07:19:14 INFO Received push request method=POST path=/39192026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=039202026/11/12 07:19:14 INFO Received complete upload request method=POST path=/39212026/11/12 07:19:14 INFO Received push request method=POST path=/39222026/11/12 07:19:14 INFO Received complete upload request method=POST path=/39232026/11/12 07:19:14 INFO Completed upload id=6 objects=24 duration=2.933467ms39242026/11/12 07:19:14 INFO Received complete push request method=POST path=/39252026/11/12 07:19:14 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039262026/11/12 07:19:14 INFO Received complete push request method=POST path=/39272026/11/12 07:19:14 INFO Vacuumed table table=pending_closures39282026/11/12 07:19:14 INFO Completed upload id=8 objects=3 duration=2.036039ms39292026/11/12 07:19:14 INFO Vacuumed table table=commit_receipts39302026/11/12 07:19:14 INFO Completed push id=7 objects=15 duration=2.725945ms39312026/11/12 07:19:14 INFO Vacuumed table table=pending_objects39322026/11/12 07:19:14 INFO Received complete push request method=POST path=/39332026/11/12 07:19:14 INFO Received complete upload request method=POST path=/39342026/11/12 07:19:14 INFO Completed push id=11 objects=6 duration=2.587044ms39352026/11/12 07:19:14 INFO Vacuumed table table=multipart_uploads39362026/11/12 07:19:14 INFO Vacuumed table table=closures39372026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=039382026/11/12 07:19:14 INFO Completed upload id=26 objects=6 duration=841.448µs39392026/11/12 07:19:14 INFO Vacuumed table table=objects39402026/11/12 07:19:14 INFO Completed push id=13 objects=11 duration=1.597835ms39412026/11/12 07:19:14 INFO Received push request method=POST path=/39422026/11/12 07:19:14 WARN Force mode enabled - objects will be deleted immediately without grace period39432026/11/12 07:19:14 INFO Received push request method=POST path=/39442026/11/12 07:19:14 INFO Received push request method=POST path=/39452026/11/12 07:19:14 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039462026/11/12 07:19:14 INFO Received push request method=POST path=/39472026/11/12 07:19:14 INFO Vacuumed table table=pending_closures39482026/11/12 07:19:14 INFO Received uploads request method=POST path=/39492026/11/12 07:19:14 INFO Vacuumed table table=commit_receipts39502026/11/12 07:19:14 INFO Received complete push request method=POST path=/39512026/11/12 07:19:14 INFO Vacuumed table table=pending_objects39522026/11/12 07:19:14 INFO Received push request method=POST path=/39532026/11/12 07:19:14 INFO Vacuumed table table=multipart_uploads39542026/11/12 07:19:14 INFO Vacuumed table table=closures39552026/11/12 07:19:14 INFO Completed push id=10 objects=2 duration=1.938018ms39562026/11/12 07:19:14 INFO Received uploads request method=POST path=/39572026/11/12 07:19:14 INFO Vacuumed table table=objects39582026/11/12 07:19:14 INFO Received uploads request method=POST path=/39592026/11/12 07:19:14 INFO Received uploads request method=POST path=/39602026/11/12 07:19:14 INFO Received uploads request method=POST path=/39612026/11/12 07:19:14 INFO Received uploads request method=POST path=/39622026/11/12 07:19:14 INFO Received uploads request method=POST path=/39632026/11/12 07:19:14 INFO Received complete push request method=POST path=/39642026/11/12 07:19:14 INFO Received push request method=POST path=/39652026/11/12 07:19:14 INFO Completed push id=18 objects=6 duration=1.525934ms39662026/11/12 07:19:14 INFO Received push request method=POST path=/39672026/11/12 07:19:14 INFO Received complete upload request method=POST path=/39682026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=039692026/11/12 07:19:14 INFO Completed upload id=29 objects=5 duration=923.848µs39702026/11/12 07:19:14 INFO Received complete upload request method=POST path=/39712026/11/12 07:19:14 INFO Received push request method=POST path=/39722026/11/12 07:19:14 INFO Completed upload id=12 objects=21 duration=1.288492ms39732026/11/12 07:19:14 INFO Received complete push request method=POST path=/39742026/11/12 07:19:14 INFO Received complete push request method=POST path=/39752026/11/12 07:19:14 INFO Received push request method=POST path=/39762026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=039772026/11/12 07:19:14 INFO Completed push id=8 objects=6 duration=1.942018ms39782026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=039792026/11/12 07:19:14 INFO Completed push id=19 objects=6 duration=3.967736ms39802026/11/12 07:19:14 WARN Force mode enabled - objects will be deleted immediately without grace period39812026/11/12 07:19:14 WARN Force mode enabled - objects will be deleted immediately without grace period39822026/11/12 07:19:14 INFO Received uploads request method=POST path=/39832026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=039842026/11/12 07:19:14 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=039852026/11/12 07:19:14 INFO Vacuumed table table=pending_closures39862026/11/12 07:19:14 INFO Vacuumed table table=commit_receipts39872026/11/12 07:19:14 INFO Vacuumed table table=pending_objects39882026/11/12 07:19:14 INFO Vacuumed table table=multipart_uploads39892026/11/12 07:19:14 INFO Vacuumed table table=closures39902026/11/12 07:19:14 INFO Vacuumed table table=objects39912026/11/12 07:19:14 INFO Received push request method=POST path=/39922026/11/12 07:19:14 INFO Received push request method=POST path=/39932026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=039942026/11/12 07:19:14 WARN Force mode enabled - objects will be deleted immediately without grace period39952026/11/12 07:19:14 INFO Received push request method=POST path=/39962026/11/12 07:19:14 INFO Received uploads request method=POST path=/39972026/11/12 07:19:14 INFO Received complete push request method=POST path=/39982026/11/12 07:19:14 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039992026/11/12 07:19:14 INFO Received complete upload request method=POST path=/40002026/11/12 07:19:14 INFO Completed push id=6 objects=6 duration=2.356182ms40012026/11/12 07:19:14 INFO Vacuumed table table=pending_closures40022026/11/12 07:19:14 INFO Completed upload id=9 objects=3 duration=1.213671ms40032026/11/12 07:19:14 INFO Vacuumed table table=commit_receipts40042026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=040052026/11/12 07:19:14 INFO Received push request method=POST path=/40062026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=040072026/11/12 07:19:14 WARN Force mode enabled - objects will be deleted immediately without grace period40082026/11/12 07:19:14 INFO Vacuumed table table=pending_objects40092026/11/12 07:19:14 INFO Vacuumed table table=multipart_uploads40102026/11/12 07:19:14 INFO Vacuumed table table=closures40112026/11/12 07:19:14 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."40122026/11/12 07:19:14 WARN Force mode enabled - objects will be deleted immediately without grace period40132026/11/12 07:19:14 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040142026/11/12 07:19:14 INFO Vacuumed table table=objects40152026/11/12 07:19:14 INFO Vacuumed table table=pending_closures40162026/11/12 07:19:14 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040172026/11/12 07:19:14 INFO Vacuumed table table=commit_receipts40182026/11/12 07:19:14 INFO Vacuumed table table=pending_closures40192026/11/12 07:19:14 INFO Vacuumed table table=pending_objects40202026/11/12 07:19:14 INFO Vacuumed table table=multipart_uploads40212026/11/12 07:19:14 INFO Vacuumed table table=commit_receipts40222026/11/12 07:19:14 INFO Vacuumed table table=closures40232026/11/12 07:19:14 INFO Vacuumed table table=objects40242026/11/12 07:19:14 INFO Received push request method=POST path=/40252026/11/12 07:19:14 INFO Vacuumed table table=pending_objects40262026/11/12 07:19:14 INFO Received complete push request method=POST path=/40272026/11/12 07:19:14 INFO Vacuumed table table=multipart_uploads40282026/11/12 07:19:14 INFO Received uploads request method=POST path=/40292026/11/12 07:19:14 INFO Vacuumed table table=closures40302026/11/12 07:19:14 INFO Received push request method=POST path=/40312026/11/12 07:19:14 INFO Received complete upload request method=POST path=/40322026/11/12 07:19:14 INFO Completed push id=13 objects=23 duration=2.17526ms40332026/11/12 07:19:14 INFO Vacuumed table table=objects40342026/11/12 07:19:14 INFO Received push request method=POST path=/40352026/11/12 07:19:14 INFO Completed upload id=25 objects=4 duration=1.359992ms40362026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=040372026/11/12 07:19:14 WARN Force mode enabled - objects will be deleted immediately without grace period40382026/11/12 07:19:14 INFO Received uploads request method=POST path=/40392026/11/12 07:19:14 INFO Aborted multipart uploads count=0 kept=040402026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040412026/11/12 07:19:15 INFO Vacuumed table table=pending_closures40422026/11/12 07:19:15 WARN Force mode enabled - objects will be deleted immediately without grace period40432026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts40442026/11/12 07:19:15 INFO Vacuumed table table=pending_objects40452026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads40462026/11/12 07:19:15 INFO Vacuumed table table=closures40472026/11/12 07:19:15 INFO Received uploads request method=POST path=/40482026/11/12 07:19:15 INFO Vacuumed table table=objects40492026/11/12 07:19:15 INFO Received uploads request method=POST path=/40502026/11/12 07:19:15 INFO Received complete push request method=POST path=/40512026/11/12 07:19:15 INFO Received push request method=POST path=/40522026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=040532026/11/12 07:19:15 INFO Received push request method=POST path=/40542026/11/12 07:19:15 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."40552026/11/12 07:19:15 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."40562026/11/12 07:19:15 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."40572026/11/12 07:19:15 INFO Completed push id=3 objects=21 duration=4.106038ms40582026/11/12 07:19:15 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."40592026/11/12 07:19:15 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."40602026/11/12 07:19:15 INFO Received uploads request method=POST path=/40612026/11/12 07:19:15 INFO Received uploads request method=POST path=/40622026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=040632026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=040642026/11/12 07:19:15 INFO Received complete push request method=POST path=/40652026/11/12 07:19:15 INFO Received complete upload request method=POST path=/40662026/11/12 07:19:15 WARN Force mode enabled - objects will be deleted immediately without grace period40672026/11/12 07:19:15 INFO Completed push id=15 objects=4 duration=1.738156ms40682026/11/12 07:19:15 INFO Completed upload id=29 objects=4 duration=1.723415ms40692026/11/12 07:19:15 INFO Received complete push request method=POST path=/40702026/11/12 07:19:15 INFO Received uploads request method=POST path=/40712026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040722026/11/12 07:19:15 INFO Vacuumed table table=pending_closures40732026/11/12 07:19:15 INFO Received complete push request method=POST path=/40742026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts40752026/11/12 07:19:15 INFO Completed push id=16 objects=16 duration=3.387531ms40762026/11/12 07:19:15 INFO Vacuumed table table=pending_objects40772026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads40782026/11/12 07:19:15 INFO Completed push id=10 objects=15 duration=2.16736ms40792026/11/12 07:19:15 INFO Vacuumed table table=closures40802026/11/12 07:19:15 INFO Vacuumed table table=objects40812026/11/12 07:19:15 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."40822026/11/12 07:19:15 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."40832026/11/12 07:19:15 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."40842026/11/12 07:19:15 INFO Received uploads request method=POST path=/40852026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=040862026/11/12 07:19:15 INFO Received push request method=POST path=/40872026/11/12 07:19:15 INFO Received uploads request method=POST path=/40882026/11/12 07:19:15 INFO Received uploads request method=POST path=/40892026/11/12 07:19:15 INFO Received uploads request method=POST path=/40902026/11/12 07:19:15 INFO Received push request method=POST path=/40912026/11/12 07:19:15 INFO Received complete push request method=POST path=/40922026/11/12 07:19:15 INFO Completed push id=20 objects=8 duration=1.403653ms40932026/11/12 07:19:15 INFO Received push request method=POST path=/40942026/11/12 07:19:15 INFO Received uploads request method=POST path=/40952026/11/12 07:19:15 INFO Received uploads request method=POST path=/40962026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=040972026/11/12 07:19:15 INFO Received uploads request method=POST path=/40982026/11/12 07:19:15 WARN Force mode enabled - objects will be deleted immediately without grace period40992026/11/12 07:19:15 INFO Received uploads request method=POST path=/41002026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=041012026/11/12 07:19:15 INFO Received complete push request method=POST path=/41022026/11/12 07:19:15 INFO Received uploads request method=POST path=/41032026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041042026/11/12 07:19:15 INFO Completed push id=19 objects=14 duration=2.390082ms41052026/11/12 07:19:15 INFO Vacuumed table table=pending_closures41062026/11/12 07:19:15 INFO Received uploads request method=POST path=/41072026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts41082026/11/12 07:19:15 INFO Vacuumed table table=pending_objects41092026/11/12 07:19:15 INFO Received complete push request method=POST path=/41102026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads41112026/11/12 07:19:15 INFO Vacuumed table table=closures41122026/11/12 07:19:15 INFO Completed push id=12 objects=3 duration=1.520294ms41132026/11/12 07:19:15 INFO Vacuumed table table=objects41142026/11/12 07:19:15 INFO Received complete upload request method=POST path=/41152026/11/12 07:19:15 INFO Completed upload id=31 objects=3 duration=2.11118ms41162026/11/12 07:19:15 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."41172026/11/12 07:19:15 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."41182026/11/12 07:19:15 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."41192026/11/12 07:19:15 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."41202026/11/12 07:19:15 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."41212026/11/12 07:19:15 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."41222026/11/12 07:19:15 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."41232026/11/12 07:19:15 INFO Received complete push request method=POST path=/41242026/11/12 07:19:15 INFO Received uploads request method=POST path=/41252026/11/12 07:19:15 INFO Completed push id=11 objects=6 duration=2.600864ms41262026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=041272026/11/12 07:19:15 INFO Received push request method=POST path=/41282026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=041292026/11/12 07:19:15 INFO Received uploads request method=POST path=/41302026/11/12 07:19:15 INFO Received push request method=POST path=/41312026/11/12 07:19:15 INFO Received complete push request method=POST path=/41322026/11/12 07:19:15 WARN Force mode enabled - objects will be deleted immediately without grace period41332026/11/12 07:19:15 INFO Completed push id=14 objects=3 duration=1.118911ms41342026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041352026/11/12 07:19:15 INFO Vacuumed table table=pending_closures41362026/11/12 07:19:15 INFO Received complete push request method=POST path=/41372026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts41382026/11/12 07:19:15 INFO Received complete upload request method=POST path=/41392026/11/12 07:19:15 INFO Vacuumed table table=pending_objects41402026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads41412026/11/12 07:19:15 INFO Completed upload id=22 objects=2 duration=1.11527ms41422026/11/12 07:19:15 INFO Completed push id=15 objects=7 duration=1.909078ms41432026/11/12 07:19:15 INFO Vacuumed table table=closures41442026/11/12 07:19:15 INFO Vacuumed table table=objects41452026/11/12 07:19:15 INFO Received uploads request method=POST path=/41462026/11/12 07:19:15 INFO Received uploads request method=POST path=/41472026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=041482026/11/12 07:19:15 INFO Received uploads request method=POST path=/41492026/11/12 07:19:15 INFO Received complete push request method=POST path=/41502026/11/12 07:19:15 INFO Received push request method=POST path=/41512026/11/12 07:19:15 INFO Completed push id=20 objects=13 duration=1.891317ms41522026/11/12 07:19:15 WARN Force mode enabled - objects will be deleted immediately without grace period41532026/11/12 07:19:15 INFO Received uploads request method=POST path=/41542026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=041552026/11/12 07:19:15 WARN Force mode enabled - objects will be deleted immediately without grace period41562026/11/12 07:19:15 INFO Received uploads request method=POST path=/41572026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=041582026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041592026/11/12 07:19:15 INFO Received complete push request method=POST path=/41602026/11/12 07:19:15 INFO Vacuumed table table=pending_closures41612026/11/12 07:19:15 INFO Vacuumed table table=pending_closures41622026/11/12 07:19:15 INFO Received complete push request method=POST path=/41632026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts41642026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts41652026/11/12 07:19:15 INFO Completed push id=15 objects=5 duration=2.493963ms41662026/11/12 07:19:15 INFO Completed push id=27 objects=11 duration=1.10203ms41672026/11/12 07:19:15 INFO Vacuumed table table=pending_objects41682026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads41692026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=041702026/11/12 07:19:15 INFO Vacuumed table table=closures41712026/11/12 07:19:15 INFO Vacuumed table table=pending_objects41722026/11/12 07:19:15 INFO Received complete push request method=POST path=/41732026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads41742026/11/12 07:19:15 INFO Vacuumed table table=objects41752026/11/12 07:19:15 INFO Vacuumed table table=closures41762026/11/12 07:19:15 INFO Received uploads request method=POST path=/41772026/11/12 07:19:15 INFO Completed push id=22 objects=4 duration=1.382812ms41782026/11/12 07:19:15 WARN Force mode enabled - objects will be deleted immediately without grace period41792026/11/12 07:19:15 INFO Vacuumed table table=objects41802026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=041812026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041822026/11/12 07:19:15 INFO Vacuumed table table=pending_closures41832026/11/12 07:19:15 INFO Vacuumed table table=pending_closures41842026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts41852026/11/12 07:19:15 INFO Received complete upload request method=POST path=/41862026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts41872026/11/12 07:19:15 INFO Completed upload id=20 objects=4 duration=855.888µs41882026/11/12 07:19:15 INFO Received push request method=POST path=/41892026/11/12 07:19:15 INFO Vacuumed table table=pending_objects41902026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads41912026/11/12 07:19:15 INFO Received complete push request method=POST path=/41922026/11/12 07:19:15 INFO Vacuumed table table=pending_objects41932026/11/12 07:19:15 INFO Vacuumed table table=closures41942026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads41952026/11/12 07:19:15 INFO Vacuumed table table=objects41962026/11/12 07:19:15 INFO Vacuumed table table=closures41972026/11/12 07:19:15 INFO Vacuumed table table=objects41982026/11/12 07:19:15 INFO Received uploads request method=POST path=/41992026/11/12 07:19:15 INFO Completed push id=18 objects=6 duration=2.835006ms42002026/11/12 07:19:15 INFO Received complete upload request method=POST path=/42012026/11/12 07:19:15 INFO Completed upload id=29 objects=26 duration=953.869µs42022026/11/12 07:19:15 INFO Received complete upload request method=POST path=/42032026/11/12 07:19:15 INFO Received push request method=POST path=/42042026/11/12 07:19:15 INFO Completed upload id=18 objects=2 duration=2.09098ms42052026/11/12 07:19:15 INFO Received complete push request method=POST path=/42062026/11/12 07:19:15 INFO Received complete upload request method=POST path=/42072026/11/12 07:19:15 INFO Completed upload id=23 objects=7 duration=1.172631ms42082026/11/12 07:19:15 INFO Completed push id=9 objects=4 duration=3.165589ms42092026/11/12 07:19:15 INFO Received push request method=POST path=/42102026/11/12 07:19:15 INFO Received complete upload request method=POST path=/42112026/11/12 07:19:15 INFO Completed upload id=23 objects=9 duration=1.288972ms42122026/11/12 07:19:15 INFO Received complete push request method=POST path=/42132026/11/12 07:19:15 INFO Completed push id=30 objects=8 duration=883.048µs42142026/11/12 07:19:15 INFO Received complete upload request method=POST path=/42152026/11/12 07:19:15 INFO Received push request method=POST path=/42162026/11/12 07:19:15 INFO Completed upload id=16 objects=23 duration=1.729816ms42172026/11/12 07:19:15 INFO Received complete push request method=POST path=/42182026/11/12 07:19:15 INFO Completed push id=16 objects=5 duration=1.117931ms42192026/11/12 07:19:15 INFO Received uploads request method=POST path=/42202026/11/12 07:19:15 INFO Received push request method=POST path=/42212026/11/12 07:19:15 INFO Received complete push request method=POST path=/42222026/11/12 07:19:15 INFO Completed push id=24 objects=5 duration=1.08869ms42232026/11/12 07:19:15 INFO Received push request method=POST path=/42242026/11/12 07:19:15 INFO Received complete upload request method=POST path=/42252026/11/12 07:19:15 INFO Completed upload id=31 objects=5 duration=706.986µs42262026/11/12 07:19:15 INFO Received uploads request method=POST path=/42272026/11/12 07:19:15 INFO Received push request method=POST path=/42282026/11/12 07:19:15 INFO Received uploads request method=POST path=/42292026/11/12 07:19:15 INFO Received uploads request method=POST path=/42302026/11/12 07:19:15 INFO Received complete upload request method=POST path=/42312026/11/12 07:19:15 INFO Completed upload id=15 objects=6 duration=911.529µs42322026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=042332026/11/12 07:19:15 INFO Vacuumed table table=pending_closures42342026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts42352026/11/12 07:19:15 INFO Received push request method=POST path=/42362026/11/12 07:19:15 INFO Vacuumed table table=pending_objects42372026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads42382026/11/12 07:19:15 INFO Received complete upload request method=POST path=/42392026/11/12 07:19:15 INFO Vacuumed table table=closures42402026/11/12 07:19:15 INFO Completed upload id=25 objects=2 duration=734.727µs42412026/11/12 07:19:15 INFO Vacuumed table table=objects42422026/11/12 07:19:15 INFO Received complete upload request method=POST path=/42432026/11/12 07:19:15 INFO Received complete upload request method=POST path=/42442026/11/12 07:19:15 INFO Received complete push request method=POST path=/42452026/11/12 07:19:15 INFO Completed upload id=13 objects=7 duration=1.774197ms42462026/11/12 07:19:15 INFO Completed upload id=22 objects=2 duration=1.332693ms42472026/11/12 07:19:15 INFO Completed push id=32 objects=3 duration=1.344672ms42482026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042492026/11/12 07:19:15 INFO Received push request method=POST path=/42502026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042512026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=042522026/11/12 07:19:15 INFO Vacuumed table table=pending_closures42532026/11/12 07:19:15 INFO Vacuumed table table=pending_closures42542026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts42552026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts42562026/11/12 07:19:15 INFO Received uploads request method=POST path=/42572026/11/12 07:19:15 INFO Vacuumed table table=pending_objects42582026/11/12 07:19:15 INFO Vacuumed table table=pending_objects42592026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads42602026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads42612026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=042622026/11/12 07:19:15 INFO Vacuumed table table=closures42632026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=042642026/11/12 07:19:15 INFO Vacuumed table table=closures42652026/11/12 07:19:15 INFO Received complete push request method=POST path=/42662026/11/12 07:19:15 INFO Vacuumed table table=objects42672026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042682026/11/12 07:19:15 INFO Vacuumed table table=objects42692026/11/12 07:19:15 INFO Vacuumed table table=pending_closures42702026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts42712026/11/12 07:19:15 WARN Force mode enabled - objects will be deleted immediately without grace period42722026/11/12 07:19:15 INFO Vacuumed table table=pending_objects42732026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads42742026/11/12 07:19:15 INFO Completed push id=31 objects=14 duration=2.596304ms42752026/11/12 07:19:15 INFO Vacuumed table table=closures42762026/11/12 07:19:15 INFO Vacuumed table table=objects42772026/11/12 07:19:15 INFO Received complete upload request method=POST path=/42782026/11/12 07:19:15 INFO Completed upload id=26 objects=20 duration=944.509µs42792026/11/12 07:19:15 INFO Received push request method=POST path=/42802026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=042812026/11/12 07:19:15 INFO Received complete upload request method=POST path=/42822026/11/12 07:19:15 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."42832026/11/12 07:19:15 INFO Completed upload id=20 objects=4 duration=939.329µs42842026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=042852026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=042862026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=042872026/11/12 07:19:15 INFO Vacuumed table table=pending_closures42882026/11/12 07:19:15 INFO Received uploads request method=POST path=/42892026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts42902026/11/12 07:19:15 INFO Vacuumed table table=pending_objects42912026/11/12 07:19:15 WARN Force mode enabled - objects will be deleted immediately without grace period42922026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads42932026/11/12 07:19:15 WARN Force mode enabled - objects will be deleted immediately without grace period42942026/11/12 07:19:15 INFO Vacuumed table table=closures42952026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=042962026/11/12 07:19:15 INFO Vacuumed table table=objects42972026/11/12 07:19:15 INFO Vacuumed table table=pending_closures42982026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042992026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts43002026/11/12 07:19:15 INFO Received complete upload request method=POST path=/43012026/11/12 07:19:15 INFO Received complete upload request method=POST path=/43022026/11/12 07:19:15 INFO Vacuumed table table=pending_objects43032026/11/12 07:19:15 INFO Vacuumed table table=pending_closures43042026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads43052026/11/12 07:19:15 INFO Completed upload id=17 objects=7 duration=1.169211ms43062026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts43072026/11/12 07:19:15 INFO Vacuumed table table=closures43082026/11/12 07:19:15 INFO Received uploads request method=POST path=/43092026/11/12 07:19:15 INFO Completed upload id=20 objects=3 duration=2.390062ms43102026/11/12 07:19:15 INFO Vacuumed table table=pending_objects43112026/11/12 07:19:15 INFO Vacuumed table table=objects43122026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads43132026/11/12 07:19:15 INFO Vacuumed table table=closures43142026/11/12 07:19:15 INFO Vacuumed table table=objects43152026/11/12 07:19:15 INFO Received complete upload request method=POST path=/43162026/11/12 07:19:15 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."43172026/11/12 07:19:15 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."43182026/11/12 07:19:15 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."43192026/11/12 07:19:15 INFO Completed upload id=23 objects=7 duration=1.453534ms43202026/11/12 07:19:15 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."43212026/11/12 07:19:15 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."43222026/11/12 07:19:15 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."43232026/11/12 07:19:15 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."43242026/11/12 07:19:15 INFO Received complete push request method=POST path=/43252026/11/12 07:19:15 INFO Completed push id=28 objects=7 duration=1.05167ms43262026/11/12 07:19:15 INFO Received uploads request method=POST path=/43272026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=043282026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=043292026/11/12 07:19:15 INFO Received push request method=POST path=/43302026/11/12 07:19:15 INFO Received uploads request method=POST path=/43312026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043322026/11/12 07:19:15 INFO Received uploads request method=POST path=/43332026/11/12 07:19:15 INFO Vacuumed table table=pending_closures43342026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts43352026/11/12 07:19:15 INFO Vacuumed table table=pending_objects43362026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads43372026/11/12 07:19:15 INFO Vacuumed table table=closures43382026/11/12 07:19:15 INFO Received uploads request method=POST path=/43392026/11/12 07:19:15 INFO Vacuumed table table=objects43402026/11/12 07:19:15 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."43412026/11/12 07:19:15 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."43422026/11/12 07:19:15 INFO Received complete upload request method=POST path=/43432026/11/12 07:19:15 INFO Received complete upload request method=POST path=/43442026/11/12 07:19:15 INFO Completed upload id=30 objects=6 duration=849.908µs43452026/11/12 07:19:15 INFO Completed upload id=14 objects=8 duration=1.822556ms43462026/11/12 07:19:15 INFO Received push request method=POST path=/43472026/11/12 07:19:15 INFO Received complete push request method=POST path=/43482026/11/12 07:19:15 INFO Received uploads request method=POST path=/43492026/11/12 07:19:15 INFO Received complete upload request method=POST path=/43502026/11/12 07:19:15 INFO Completed push id=16 objects=14 duration=3.158089ms43512026/11/12 07:19:15 INFO Completed upload id=21 objects=3 duration=853.068µs43522026/11/12 07:19:15 INFO Received uploads request method=POST path=/43532026/11/12 07:19:15 INFO Received push request method=POST path=/43542026/11/12 07:19:15 INFO Received complete push request method=POST path=/43552026/11/12 07:19:15 INFO Completed push id=31 objects=5 duration=1.02171ms43562026/11/12 07:19:15 INFO Received complete upload request method=POST path=/43572026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=043582026/11/12 07:19:15 INFO Completed upload id=24 objects=18 duration=1.262752ms43592026/11/12 07:19:15 INFO Received complete upload request method=POST path=/43602026/11/12 07:19:15 INFO Received complete upload request method=POST path=/43612026/11/12 07:19:15 INFO Received uploads request method=POST path=/43622026/11/12 07:19:15 WARN Force mode enabled - objects will be deleted immediately without grace period43632026/11/12 07:19:15 INFO Completed upload id=13 objects=14 duration=3.154629ms43642026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=043652026/11/12 07:19:15 INFO Completed upload id=24 objects=7 duration=1.767177ms43662026/11/12 07:19:15 INFO Received complete upload request method=POST path=/43672026/11/12 07:19:15 WARN Force mode enabled - objects will be deleted immediately without grace period43682026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=043692026/11/12 07:19:15 INFO Completed upload id=15 objects=4 duration=2.311122ms43702026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043712026/11/12 07:19:15 INFO Received push request method=POST path=/43722026/11/12 07:19:15 INFO Vacuumed table table=pending_closures43732026/11/12 07:19:15 WARN Force mode enabled - objects will be deleted immediately without grace period43742026/11/12 07:19:15 INFO Received complete upload request method=POST path=/43752026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts43762026/11/12 07:19:15 INFO Completed upload id=14 objects=6 duration=970.569µs43772026/11/12 07:19:15 INFO Vacuumed table table=pending_objects43782026/11/12 07:19:15 INFO Received complete push request method=POST path=/43792026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads43802026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043812026/11/12 07:19:15 INFO Received complete push request method=POST path=/43822026/11/12 07:19:15 INFO Vacuumed table table=closures43832026/11/12 07:19:15 INFO Completed push id=17 objects=22 duration=2.473943ms43842026/11/12 07:19:15 INFO Received complete push request method=POST path=/43852026/11/12 07:19:15 INFO Vacuumed table table=pending_closures43862026/11/12 07:19:15 INFO Completed push id=16 objects=26 duration=1.183591ms43872026/11/12 07:19:15 INFO Vacuumed table table=objects43882026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts43892026/11/12 07:19:15 INFO Received push request method=POST path=/43902026/11/12 07:19:15 INFO Completed push id=25 objects=2 duration=2.095659ms43912026/11/12 07:19:15 INFO Vacuumed table table=pending_objects43922026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads43932026/11/12 07:19:15 INFO Received complete push request method=POST path=/43942026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=043952026/11/12 07:19:15 INFO Vacuumed table table=closures43962026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=7 objects-failed-to-delete=043972026/11/12 07:19:15 INFO Vacuumed table table=objects43982026/11/12 07:19:15 INFO Completed push id=7 objects=7 duration=1.626435ms43992026/11/12 07:19:15 INFO Received uploads request method=POST path=/44002026/11/12 07:19:15 INFO Vacuumed table table=pending_closures44012026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044022026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=044032026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts44042026/11/12 07:19:15 INFO Received push request method=POST path=/44052026/11/12 07:19:15 INFO Vacuumed table table=pending_closures44062026/11/12 07:19:15 INFO Received push request method=POST path=/44072026/11/12 07:19:15 INFO Vacuumed table table=pending_objects44082026/11/12 07:19:15 INFO Received complete push request method=POST path=/44092026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads44102026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts44112026/11/12 07:19:15 INFO Vacuumed table table=closures44122026/11/12 07:19:15 INFO Completed push id=32 objects=26 duration=1.438453ms44132026/11/12 07:19:15 INFO Received push request method=POST path=/44142026/11/12 07:19:15 INFO Vacuumed table table=pending_objects44152026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044162026/11/12 07:19:15 INFO Vacuumed table table=objects44172026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads44182026/11/12 07:19:15 INFO Vacuumed table table=pending_closures44192026/11/12 07:19:15 INFO Vacuumed table table=closures44202026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts44212026/11/12 07:19:15 INFO Vacuumed table table=objects44222026/11/12 07:19:15 INFO Vacuumed table table=pending_objects44232026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads44242026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=044252026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=044262026/11/12 07:19:15 INFO Received complete push request method=POST path=/44272026/11/12 07:19:15 INFO Vacuumed table table=closures44282026/11/12 07:19:15 INFO Vacuumed table table=objects44292026/11/12 07:19:15 INFO Completed push id=27 objects=4 duration=1.635435ms44302026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044312026/11/12 07:19:15 INFO Received complete push request method=POST path=/44322026/11/12 07:19:15 INFO Vacuumed table table=pending_closures44332026/11/12 07:19:15 WARN Force mode enabled - objects will be deleted immediately without grace period44342026/11/12 07:19:15 INFO Completed push id=12 objects=6 duration=1.311152ms44352026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=044362026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts44372026/11/12 07:19:15 INFO Vacuumed table table=pending_objects44382026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads44392026/11/12 07:19:15 INFO Received uploads request method=POST path=/44402026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044412026/11/12 07:19:15 INFO Vacuumed table table=closures44422026/11/12 07:19:15 INFO Vacuumed table table=objects44432026/11/12 07:19:15 INFO Vacuumed table table=pending_closures44442026/11/12 07:19:15 INFO Received complete upload request method=POST path=/44452026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts44462026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=044472026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044482026/11/12 07:19:15 INFO Completed upload id=33 objects=2 duration=1.331392ms44492026/11/12 07:19:15 INFO Vacuumed table table=pending_objects44502026/11/12 07:19:15 INFO Vacuumed table table=pending_closures44512026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads44522026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts44532026/11/12 07:19:15 INFO Vacuumed table table=closures44542026/11/12 07:19:15 INFO Vacuumed table table=objects44552026/11/12 07:19:15 INFO Vacuumed table table=pending_objects44562026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads44572026/11/12 07:19:15 INFO Vacuumed table table=closures44582026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044592026/11/12 07:19:15 INFO Vacuumed table table=objects44602026/11/12 07:19:15 INFO Received complete upload request method=POST path=/44612026/11/12 07:19:15 INFO Received uploads request method=POST path=/44622026/11/12 07:19:15 INFO Received uploads request method=POST path=/44632026/11/12 07:19:15 INFO Vacuumed table table=pending_closures44642026/11/12 07:19:15 INFO Received complete push request method=POST path=/44652026/11/12 07:19:15 INFO Completed upload id=7 objects=3 duration=906.248µs44662026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts44672026/11/12 07:19:15 INFO Vacuumed table table=pending_objects44682026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads44692026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=044702026/11/12 07:19:15 INFO Vacuumed table table=closures44712026/11/12 07:19:15 INFO Completed push id=31 objects=6 duration=3.047768ms44722026/11/12 07:19:15 INFO Vacuumed table table=objects44732026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=044742026/11/12 07:19:15 WARN Force mode enabled - objects will be deleted immediately without grace period44752026/11/12 07:19:15 INFO Received complete upload request method=POST path=/44762026/11/12 07:19:15 INFO Completed upload id=16 objects=3 duration=1.127031ms44772026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044782026/11/12 07:19:15 INFO Received complete upload request method=POST path=/44792026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=044802026/11/12 07:19:15 INFO Vacuumed table table=pending_closures44812026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts44822026/11/12 07:19:15 INFO Completed upload id=34 objects=5 duration=925.949µs44832026/11/12 07:19:15 INFO Vacuumed table table=pending_objects44842026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads44852026/11/12 07:19:15 INFO Vacuumed table table=closures44862026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=044872026/11/12 07:19:15 WARN Force mode enabled - objects will be deleted immediately without grace period44882026/11/12 07:19:15 INFO Vacuumed table table=objects44892026/11/12 07:19:15 INFO Received complete push request method=POST path=/44902026/11/12 07:19:15 INFO Received push request method=POST path=/44912026/11/12 07:19:15 INFO Received complete push request method=POST path=/44922026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044932026/11/12 07:19:15 INFO Vacuumed table table=pending_closures44942026/11/12 07:19:15 INFO Vacuumed table table=pending_closures44952026/11/12 07:19:15 INFO Completed push id=27 objects=6 duration=2.470083ms44962026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts44972026/11/12 07:19:15 INFO Completed push id=28 objects=26 duration=1.928838ms44982026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts44992026/11/12 07:19:15 INFO Vacuumed table table=pending_objects45002026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads45012026/11/12 07:19:15 INFO Received push request method=POST path=/45022026/11/12 07:19:15 INFO Vacuumed table table=pending_objects45032026/11/12 07:19:15 INFO Vacuumed table table=closures45042026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads45052026/11/12 07:19:15 INFO Vacuumed table table=objects45062026/11/12 07:19:15 INFO Vacuumed table table=closures45072026/11/12 07:19:15 INFO Vacuumed table table=objects45082026/11/12 07:19:15 INFO Received uploads request method=POST path=/45092026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=045102026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=045112026/11/12 07:19:15 WARN Force mode enabled - objects will be deleted immediately without grace period45122026/11/12 07:19:15 INFO Received complete push request method=POST path=/45132026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045142026/11/12 07:19:15 INFO Received complete upload request method=POST path=/45152026/11/12 07:19:15 INFO Received complete push request method=POST path=/45162026/11/12 07:19:15 INFO Vacuumed table table=pending_closures45172026/11/12 07:19:15 INFO Completed upload id=35 objects=11 duration=1.09085ms45182026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts45192026/11/12 07:19:15 INFO Completed push id=14 objects=3 duration=2.575364ms45202026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045212026/11/12 07:19:15 INFO Vacuumed table table=pending_objects45222026/11/12 07:19:15 INFO Completed push id=18 objects=23 duration=1.774857ms45232026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads45242026/11/12 07:19:15 INFO Received uploads request method=POST path=/45252026/11/12 07:19:15 INFO Vacuumed table table=closures45262026/11/12 07:19:15 INFO Vacuumed table table=pending_closures45272026/11/12 07:19:15 INFO Received complete push request method=POST path=/45282026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts45292026/11/12 07:19:15 INFO Vacuumed table table=objects45302026/11/12 07:19:15 INFO Received complete push request method=POST path=/45312026/11/12 07:19:15 INFO Vacuumed table table=pending_objects45322026/11/12 07:19:15 INFO Completed push id=13 objects=22 duration=1.818277ms45332026/11/12 07:19:15 INFO Completed push id=30 objects=3 duration=1.712515ms45342026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads45352026/11/12 07:19:15 INFO Received push request method=POST path=/45362026/11/12 07:19:15 INFO Vacuumed table table=closures45372026/11/12 07:19:15 INFO Received push request method=POST path=/45382026/11/12 07:19:15 INFO Vacuumed table table=objects45392026/11/12 07:19:15 INFO Received complete upload request method=POST path=/45402026/11/12 07:19:15 INFO Received push request method=POST path=/45412026/11/12 07:19:15 INFO Completed upload id=7 objects=10 duration=1.970778ms45422026/11/12 07:19:15 INFO Received complete upload request method=POST path=/45432026/11/12 07:19:15 INFO Completed upload id=36 objects=3 duration=787.687µs45442026/11/12 07:19:15 INFO Received uploads request method=POST path=/45452026/11/12 07:19:15 INFO Received complete upload request method=POST path=/45462026/11/12 07:19:15 INFO Received complete push request method=POST path=/45472026/11/12 07:19:15 INFO Completed upload id=25 objects=6 duration=1.698016ms45482026/11/12 07:19:15 INFO Received complete upload request method=POST path=/45492026/11/12 07:19:15 INFO Received complete push request method=POST path=/45502026/11/12 07:19:15 INFO Received complete push request method=POST path=/45512026/11/12 07:19:15 INFO Completed push id=21 objects=6 duration=1.464193ms45522026/11/12 07:19:15 INFO Completed upload id=15 objects=2 duration=955.328µs45532026/11/12 07:19:15 INFO Completed push id=32 objects=19 duration=1.332093ms45542026/11/12 07:19:15 INFO Received uploads request method=POST path=/45552026/11/12 07:19:15 INFO Completed push id=21 objects=9 duration=1.799116ms45562026/11/12 07:19:15 INFO Received complete push request method=POST path=/45572026/11/12 07:19:15 INFO Received uploads request method=POST path=/45582026/11/12 07:19:15 INFO Received complete push request method=POST path=/45592026/11/12 07:19:15 INFO Received push request method=POST path=/45602026/11/12 07:19:15 INFO Completed push id=22 objects=3 duration=2.455243ms45612026/11/12 07:19:15 INFO Completed push id=23 objects=28 duration=2.933387ms45622026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=045632026/11/12 07:19:15 WARN Force mode enabled - objects will be deleted immediately without grace period45642026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045652026/11/12 07:19:15 INFO Vacuumed table table=pending_closures45662026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts45672026/11/12 07:19:15 INFO Received complete upload request method=POST path=/45682026/11/12 07:19:15 INFO Received uploads request method=POST path=/45692026/11/12 07:19:15 INFO Completed upload id=22 objects=7 duration=905.208µs45702026/11/12 07:19:15 INFO Vacuumed table table=pending_objects45712026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads45722026/11/12 07:19:15 INFO Received uploads request method=POST path=/45732026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=045742026/11/12 07:19:15 INFO Received uploads request method=POST path=/45752026/11/12 07:19:15 INFO Vacuumed table table=closures45762026/11/12 07:19:15 INFO Received complete push request method=POST path=/45772026/11/12 07:19:15 INFO Vacuumed table table=objects45782026/11/12 07:19:15 WARN Force mode enabled - objects will be deleted immediately without grace period45792026/11/12 07:19:15 INFO Completed push id=11 objects=9 duration=1.634515ms45802026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045812026/11/12 07:19:15 INFO Vacuumed table table=pending_closures45822026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts45832026/11/12 07:19:15 INFO Received push request method=POST path=/45842026/11/12 07:19:15 INFO Vacuumed table table=pending_objects45852026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads45862026/11/12 07:19:15 INFO Received complete upload request method=POST path=/45872026/11/12 07:19:15 INFO Vacuumed table table=closures45882026/11/12 07:19:15 INFO Vacuumed table table=objects45892026/11/12 07:19:15 INFO Received complete push request method=POST path=/45902026/11/12 07:19:15 INFO Completed upload id=16 objects=17 duration=2.630264ms45912026/11/12 07:19:15 INFO Received push request method=POST path=/45922026/11/12 07:19:15 INFO Completed push id=24 objects=4 duration=997.79µs45932026/11/12 07:19:15 INFO Received complete upload request method=POST path=/45942026/11/12 07:19:15 INFO Received uploads request method=POST path=/45952026/11/12 07:19:15 INFO Completed upload id=18 objects=4 duration=2.15692ms45962026/11/12 07:19:15 INFO Received push request method=POST path=/45972026/11/12 07:19:15 INFO Received complete upload request method=POST path=/45982026/11/12 07:19:15 INFO Completed upload id=27 objects=8 duration=890.049µs45992026/11/12 07:19:15 INFO Received complete push request method=POST path=/46002026/11/12 07:19:15 INFO Completed push id=1 objects=9 duration=1.07713ms46012026/11/12 07:19:15 INFO Received uploads request method=POST path=/46022026/11/12 07:19:15 INFO Received uploads request method=POST path=/46032026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=046042026/11/12 07:19:15 INFO Received complete push request method=POST path=/46052026/11/12 07:19:15 WARN Force mode enabled - objects will be deleted immediately without grace period46062026/11/12 07:19:15 INFO Received push request method=POST path=/46072026/11/12 07:19:15 INFO Completed push id=25 objects=9 duration=906.048µs46082026/11/12 07:19:15 INFO Received push request method=POST path=/46092026/11/12 07:19:15 INFO Received push request method=POST path=/46102026/11/12 07:19:15 INFO Received push request method=POST path=/46112026/11/12 07:19:15 INFO Received complete push request method=POST path=/46122026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=046132026/11/12 07:19:15 INFO Completed push id=11 objects=9 duration=1.320352ms46142026/11/12 07:19:15 INFO Received complete push request method=POST path=/46152026/11/12 07:19:15 INFO Received complete upload request method=POST path=/46162026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046172026/11/12 07:19:15 INFO Received push request method=POST path=/46182026/11/12 07:19:15 INFO Completed upload id=26 objects=3 duration=1.554814ms46192026/11/12 07:19:15 INFO Completed push id=24 objects=3 duration=1.845997ms46202026/11/12 07:19:15 INFO Vacuumed table table=pending_closures46212026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts46222026/11/12 07:19:15 INFO Received complete push request method=POST path=/46232026/11/12 07:19:15 INFO Vacuumed table table=pending_objects46242026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads46252026/11/12 07:19:15 INFO Vacuumed table table=closures46262026/11/12 07:19:15 INFO Received uploads request method=POST path=/46272026/11/12 07:19:15 INFO Vacuumed table table=objects46282026/11/12 07:19:15 INFO Completed push id=17 objects=6 duration=2.234481ms46292026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=046302026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=046312026/11/12 07:19:15 WARN Force mode enabled - objects will be deleted immediately without grace period46322026/11/12 07:19:15 INFO Received complete upload request method=POST path=/46332026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046342026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046352026/11/12 07:19:15 INFO Vacuumed table table=pending_closures46362026/11/12 07:19:15 INFO Completed upload id=11 objects=7 duration=1.183291ms46372026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts46382026/11/12 07:19:15 INFO Vacuumed table table=pending_closures46392026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts46402026/11/12 07:19:15 INFO Vacuumed table table=pending_objects46412026/11/12 07:19:15 INFO Received push request method=POST path=/46422026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads46432026/11/12 07:19:15 INFO Vacuumed table table=closures46442026/11/12 07:19:15 INFO Vacuumed table table=pending_objects46452026/11/12 07:19:15 INFO Received complete push request method=POST path=/46462026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads46472026/11/12 07:19:15 INFO Vacuumed table table=objects46482026/11/12 07:19:15 INFO Vacuumed table table=closures46492026/11/12 07:19:15 INFO Completed push id=15 objects=24 duration=1.322773ms46502026/11/12 07:19:15 INFO Vacuumed table table=objects46512026/11/12 07:19:15 INFO Received push request method=POST path=/46522026/11/12 07:19:15 INFO Received uploads request method=POST path=/46532026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=046542026/11/12 07:19:15 WARN Force mode enabled - objects will be deleted immediately without grace period46552026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046562026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=046572026/11/12 07:19:15 INFO Received complete upload request method=POST path=/46582026/11/12 07:19:15 INFO Vacuumed table table=pending_closures46592026/11/12 07:19:15 INFO Completed upload id=27 objects=7 duration=994.05µs46602026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts46612026/11/12 07:19:15 WARN Force mode enabled - objects will be deleted immediately without grace period46622026/11/12 07:19:15 INFO Received uploads request method=POST path=/46632026/11/12 07:19:15 INFO Vacuumed table table=pending_objects46642026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads46652026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046662026/11/12 07:19:15 INFO Vacuumed table table=closures46672026/11/12 07:19:15 INFO Vacuumed table table=objects46682026/11/12 07:19:15 INFO Vacuumed table table=pending_closures46692026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts46702026/11/12 07:19:15 INFO Vacuumed table table=pending_objects46712026/11/12 07:19:15 INFO Received complete upload request method=POST path=/46722026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads46732026/11/12 07:19:15 INFO Received complete upload request method=POST path=/46742026/11/12 07:19:15 INFO Vacuumed table table=closures46752026/11/12 07:19:15 INFO Completed upload id=2 objects=7 duration=1.151971ms46762026/11/12 07:19:15 INFO Completed upload id=19 objects=2 duration=1.019449ms46772026/11/12 07:19:15 INFO Vacuumed table table=objects46782026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=046792026/11/12 07:19:15 INFO Received push request method=POST path=/46802026/11/12 07:19:15 INFO Received uploads request method=POST path=/46812026/11/12 07:19:15 INFO Received push request method=POST path=/46822026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046832026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=8 objects-deleted-after-grace-period=8 objects-failed-to-delete=046842026/11/12 07:19:15 INFO Vacuumed table table=pending_closures46852026/11/12 07:19:15 INFO Received complete upload request method=POST path=/46862026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts46872026/11/12 07:19:15 INFO Vacuumed table table=pending_closures46882026/11/12 07:19:15 INFO Vacuumed table table=pending_objects46892026/11/12 07:19:15 INFO Completed upload id=28 objects=3 duration=841.068µs46902026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads46912026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts46922026/11/12 07:19:15 INFO Vacuumed table table=closures46932026/11/12 07:19:15 INFO Vacuumed table table=pending_objects46942026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads46952026/11/12 07:19:15 INFO Received complete upload request method=POST path=/46962026/11/12 07:19:15 INFO Vacuumed table table=objects46972026/11/12 07:19:15 INFO Received uploads request method=POST path=/46982026/11/12 07:19:15 INFO Vacuumed table table=closures46992026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=047002026/11/12 07:19:15 INFO Vacuumed table table=objects47012026/11/12 07:19:15 INFO Received uploads request method=POST path=/47022026/11/12 07:19:15 INFO Completed upload id=19 objects=2 duration=2.229941ms47032026/11/12 07:19:15 WARN Force mode enabled - objects will be deleted immediately without grace period47042026/11/12 07:19:15 INFO Received complete upload request method=POST path=/47052026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047062026/11/12 07:19:15 INFO Completed upload id=24 objects=7 duration=939.808µs47072026/11/12 07:19:15 INFO Received uploads request method=POST path=/47082026/11/12 07:19:15 INFO Vacuumed table table=pending_closures47092026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts47102026/11/12 07:19:15 INFO Vacuumed table table=pending_objects47112026/11/12 07:19:15 INFO Received complete upload request method=POST path=/47122026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads47132026/11/12 07:19:15 INFO Vacuumed table table=closures47142026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=047152026/11/12 07:19:15 INFO Completed upload id=26 objects=8 duration=1.03327ms47162026/11/12 07:19:15 INFO Vacuumed table table=objects47172026/11/12 07:19:15 INFO Received complete upload request method=POST path=/47182026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047192026/11/12 07:19:15 INFO Received uploads request method=POST path=/47202026/11/12 07:19:15 INFO Received complete push request method=POST path=/47212026/11/12 07:19:15 INFO Vacuumed table table=pending_closures47222026/11/12 07:19:15 INFO Completed upload id=25 objects=16 duration=2.032539ms47232026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts47242026/11/12 07:19:15 INFO Completed push id=29 objects=23 duration=1.573454ms47252026/11/12 07:19:15 INFO Vacuumed table table=pending_objects47262026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads47272026/11/12 07:19:15 INFO Received complete push request method=POST path=/47282026/11/12 07:19:15 INFO Vacuumed table table=closures47292026/11/12 07:19:15 INFO Vacuumed table table=objects47302026/11/12 07:19:15 INFO Completed push id=26 objects=3 duration=1.180551ms47312026/11/12 07:19:15 INFO Received complete upload request method=POST path=/47322026/11/12 07:19:15 INFO Completed upload id=8 objects=6 duration=1.196151ms47332026/11/12 07:19:15 INFO Received complete push request method=POST path=/47342026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=047352026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=047362026/11/12 07:19:15 INFO Completed push id=28 objects=14 duration=1.885598ms47372026/11/12 07:19:15 WARN Force mode enabled - objects will be deleted immediately without grace period47382026/11/12 07:19:15 INFO Received complete push request method=POST path=/47392026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047402026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=047412026/11/12 07:19:15 INFO Completed push id=33 objects=7 duration=1.721036ms47422026/11/12 07:19:15 WARN Force mode enabled - objects will be deleted immediately without grace period47432026/11/12 07:19:15 INFO Received complete push request method=POST path=/47442026/11/12 07:19:15 INFO Vacuumed table table=pending_closures47452026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047462026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts47472026/11/12 07:19:15 INFO Completed push id=30 objects=6 duration=1.151991ms47482026/11/12 07:19:15 INFO Vacuumed table table=pending_objects47492026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047502026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads47512026/11/12 07:19:15 INFO Vacuumed table table=pending_closures47522026/11/12 07:19:15 INFO Received complete upload request method=POST path=/47532026/11/12 07:19:15 INFO Vacuumed table table=closures47542026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts47552026/11/12 07:19:15 INFO Vacuumed table table=pending_closures47562026/11/12 07:19:15 INFO Vacuumed table table=objects47572026/11/12 07:19:15 INFO Completed upload id=28 objects=8 duration=1.557154ms47582026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts47592026/11/12 07:19:15 INFO Vacuumed table table=pending_objects47602026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads47612026/11/12 07:19:15 INFO Vacuumed table table=pending_objects47622026/11/12 07:19:15 INFO Vacuumed table table=closures47632026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads47642026/11/12 07:19:15 INFO Vacuumed table table=objects47652026/11/12 07:19:15 INFO Vacuumed table table=closures47662026/11/12 07:19:15 INFO Vacuumed table table=objects47672026/11/12 07:19:15 INFO Received push request method=POST path=/47682026/11/12 07:19:15 INFO Received push request method=POST path=/47692026/11/12 07:19:15 INFO Received complete upload request method=POST path=/47702026/11/12 07:19:15 INFO Completed upload id=26 objects=3 duration=964.289µs47712026/11/12 07:19:15 INFO Received complete upload request method=POST path=/47722026/11/12 07:19:15 INFO Received complete upload request method=POST path=/47732026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=047742026/11/12 07:19:15 INFO Completed upload id=15 objects=3 duration=1.884217ms47752026/11/12 07:19:15 INFO Received push request method=POST path=/47762026/11/12 07:19:15 INFO Completed upload id=29 objects=10 duration=1.308672ms47772026/11/12 07:19:15 INFO Received complete push request method=POST path=/47782026/11/12 07:19:15 INFO Completed push id=33 objects=3 duration=1.461073ms47792026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=047802026/11/12 07:19:15 INFO Received complete upload request method=POST path=/47812026/11/12 07:19:15 INFO Received uploads request method=POST path=/47822026/11/12 07:19:15 INFO Completed upload id=23 objects=15 duration=2.719905ms47832026/11/12 07:19:15 WARN Force mode enabled - objects will be deleted immediately without grace period47842026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=047852026/11/12 07:19:15 INFO Vacuumed table table=pending_closures47862026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts47872026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047882026/11/12 07:19:15 INFO Received complete upload request method=POST path=/47892026/11/12 07:19:15 INFO Vacuumed table table=pending_objects47902026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads47912026/11/12 07:19:15 INFO Vacuumed table table=pending_closures47922026/11/12 07:19:15 INFO Vacuumed table table=closures47932026/11/12 07:19:15 INFO Completed upload id=30 objects=20 duration=1.524014ms47942026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts47952026/11/12 07:19:15 INFO Vacuumed table table=objects47962026/11/12 07:19:15 INFO Received push request method=POST path=/47972026/11/12 07:19:15 INFO Received complete upload request method=POST path=/47982026/11/12 07:19:15 INFO Vacuumed table table=pending_objects47992026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads48002026/11/12 07:19:15 INFO Vacuumed table table=closures48012026/11/12 07:19:15 INFO Received complete push request method=POST path=/48022026/11/12 07:19:15 INFO Received push request method=POST path=/48032026/11/12 07:19:15 INFO Vacuumed table table=objects48042026/11/12 07:19:15 INFO Completed push id=34 objects=6 duration=1.503594ms48052026/11/12 07:19:15 INFO Completed upload id=33 objects=5 duration=3.26325ms48062026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=048072026/11/12 07:19:15 INFO Received complete upload request method=POST path=/48082026/11/12 07:19:15 WARN Force mode enabled - objects will be deleted immediately without grace period48092026/11/12 07:19:15 INFO Received complete push request method=POST path=/48102026/11/12 07:19:15 INFO Completed upload id=12 objects=9 duration=1.303052ms48112026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048122026/11/12 07:19:15 INFO Completed push id=31 objects=11 duration=1.203311ms48132026/11/12 07:19:15 INFO Vacuumed table table=pending_closures48142026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts48152026/11/12 07:19:15 INFO Received complete push request method=POST path=/48162026/11/12 07:19:15 INFO Vacuumed table table=pending_objects48172026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads48182026/11/12 07:19:15 INFO Completed push id=35 objects=8 duration=1.194731ms48192026/11/12 07:19:15 INFO Received complete upload request method=POST path=/48202026/11/12 07:19:15 INFO Vacuumed table table=closures48212026/11/12 07:19:15 INFO Vacuumed table table=objects48222026/11/12 07:19:15 INFO Completed upload id=13 objects=8 duration=1.603574ms48232026/11/12 07:19:15 INFO Received uploads request method=POST path=/48242026/11/12 07:19:15 INFO Received complete upload request method=POST path=/48252026/11/12 07:19:15 INFO Received complete upload request method=POST path=/48262026/11/12 07:19:15 INFO Completed upload id=32 objects=2 duration=755.806µs48272026/11/12 07:19:15 INFO Completed upload id=19 objects=6 duration=1.329832ms48282026/11/12 07:19:15 INFO Received complete push request method=POST path=/48292026/11/12 07:19:15 INFO Completed push id=13 objects=15 duration=1.813217ms48302026/11/12 07:19:15 INFO Received complete upload request method=POST path=/48312026/11/12 07:19:15 INFO Completed upload id=36 objects=9 duration=902.988µs48322026/11/12 07:19:15 INFO Received uploads request method=POST path=/48332026/11/12 07:19:15 INFO Received uploads request method=POST path=/48342026/11/12 07:19:15 INFO Received push request method=POST path=/48352026/11/12 07:19:15 INFO Received complete push request method=POST path=/48362026/11/12 07:19:15 INFO Completed push id=33 objects=3 duration=1.485153ms48372026/11/12 07:19:15 INFO Received complete upload request method=POST path=/48382026/11/12 07:19:15 INFO Completed upload id=26 objects=2 duration=1.745856ms48392026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=048402026/11/12 07:19:15 INFO Received complete upload request method=POST path=/48412026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=048422026/11/12 07:19:15 INFO Completed upload id=22 objects=7 duration=851.608µs48432026/11/12 07:19:15 WARN Force mode enabled - objects will be deleted immediately without grace period48442026/11/12 07:19:15 INFO Received complete push request method=POST path=/48452026/11/12 07:19:15 INFO Received uploads request method=POST path=/48462026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048472026/11/12 07:19:15 INFO Received complete upload request method=POST path=/48482026/11/12 07:19:15 INFO Received complete push request method=POST path=/48492026/11/12 07:19:15 INFO Received complete upload request method=POST path=/48502026/11/12 07:19:15 INFO Completed push id=9 objects=9 duration=2.735625ms48512026/11/12 07:19:15 INFO Received uploads request method=POST path=/48522026/11/12 07:19:15 INFO Vacuumed table table=pending_closures48532026/11/12 07:19:15 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."48542026/11/12 07:19:15 INFO Completed upload id=21 objects=25 duration=1.976238ms48552026/11/12 07:19:15 INFO Received complete upload request method=POST path=/48562026/11/12 07:19:15 INFO Completed push id=17 objects=22 duration=2.103079ms48572026/11/12 07:19:15 INFO Completed upload id=21 objects=9 duration=2.17512ms48582026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts48592026/11/12 07:19:15 INFO Completed upload id=34 objects=3 duration=1.03943ms48602026/11/12 07:19:15 INFO Vacuumed table table=pending_objects48612026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=048622026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads48632026/11/12 07:19:15 INFO Vacuumed table table=closures48642026/11/12 07:19:15 INFO Vacuumed table table=objects48652026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048662026/11/12 07:19:15 INFO Vacuumed table table=pending_closures48672026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts48682026/11/12 07:19:15 INFO Vacuumed table table=pending_objects48692026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads48702026/11/12 07:19:15 INFO Vacuumed table table=closures48712026/11/12 07:19:15 INFO Vacuumed table table=objects48722026/11/12 07:19:15 INFO Received uploads request method=POST path=/48732026/11/12 07:19:15 INFO Received push request method=POST path=/48742026/11/12 07:19:15 INFO Received uploads request method=POST path=/48752026/11/12 07:19:15 INFO Received complete push request method=POST path=/48762026/11/12 07:19:15 INFO Received complete upload request method=POST path=/48772026/11/12 07:19:15 INFO Received complete upload request method=POST path=/48782026/11/12 07:19:15 INFO Completed upload id=21 objects=8 duration=1.284411ms48792026/11/12 07:19:15 INFO Completed push id=25 objects=10 duration=2.033839ms48802026/11/12 07:19:15 INFO Completed upload id=22 objects=5 duration=1.722395ms48812026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=048822026/11/12 07:19:15 WARN Force mode enabled - objects will be deleted immediately without grace period48832026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048842026/11/12 07:19:15 INFO Vacuumed table table=pending_closures48852026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts48862026/11/12 07:19:15 INFO Vacuumed table table=pending_objects48872026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads48882026/11/12 07:19:15 INFO Vacuumed table table=closures48892026/11/12 07:19:15 INFO Vacuumed table table=objects48902026/11/12 07:19:15 INFO Received complete push request method=POST path=/48912026/11/12 07:19:15 INFO Received complete push request method=POST path=/48922026/11/12 07:19:15 INFO Received push request method=POST path=/48932026/11/12 07:19:15 INFO Completed push id=22 objects=6 duration=1.446873ms48942026/11/12 07:19:15 INFO Completed push id=20 objects=3 duration=1.585435ms48952026/11/12 07:19:15 INFO Received push request method=POST path=/48962026/11/12 07:19:15 INFO Received complete push request method=POST path=/48972026/11/12 07:19:15 INFO Received complete upload request method=POST path=/48982026/11/12 07:19:15 INFO Completed upload id=29 objects=5 duration=1.06085ms48992026/11/12 07:19:15 INFO Completed push id=28 objects=3 duration=1.762156ms49002026/11/12 07:19:15 INFO Received push request method=POST path=/49012026/11/12 07:19:15 INFO Received complete upload request method=POST path=/49022026/11/12 07:19:15 INFO Completed upload id=18 objects=22 duration=1.212591ms49032026/11/12 07:19:15 INFO Received complete push request method=POST path=/49042026/11/12 07:19:15 INFO Completed push id=24 objects=9 duration=1.711655ms49052026/11/12 07:19:15 INFO Received complete push request method=POST path=/49062026/11/12 07:19:15 INFO Completed push id=24 objects=8 duration=1.11923ms49072026/11/12 07:19:15 INFO Received complete upload request method=POST path=/49082026/11/12 07:19:15 INFO Completed upload id=12 objects=6 duration=1.12417ms49092026/11/12 07:19:15 INFO Received complete upload request method=POST path=/49102026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=049112026/11/12 07:19:15 INFO Completed upload id=30 objects=2 duration=820.987µs49122026/11/12 07:19:15 INFO Received complete push request method=POST path=/49132026/11/12 07:19:15 INFO Completed push id=29 objects=15 duration=958.809µs49142026/11/12 07:19:15 INFO Received complete upload request method=POST path=/49152026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049162026/11/12 07:19:15 INFO Completed upload id=21 objects=8 duration=824.168µs49172026/11/12 07:19:15 INFO Vacuumed table table=pending_closures49182026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts49192026/11/12 07:19:15 INFO Vacuumed table table=pending_objects49202026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads49212026/11/12 07:19:15 INFO Vacuumed table table=closures49222026/11/12 07:19:15 INFO Received complete upload request method=POST path=/49232026/11/12 07:19:15 INFO Vacuumed table table=objects49242026/11/12 07:19:15 INFO Completed upload id=25 objects=9 duration=824.267µs49252026/11/12 07:19:15 INFO Received push request method=POST path=/49262026/11/12 07:19:15 INFO Received complete push request method=POST path=/49272026/11/12 07:19:15 INFO Received complete push request method=POST path=/49282026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=049292026/11/12 07:19:15 INFO Completed push id=17 objects=3 duration=2.068099ms49302026/11/12 07:19:15 INFO Completed push id=31 objects=20 duration=1.11255ms49312026/11/12 07:19:15 INFO Received complete push request method=POST path=/49322026/11/12 07:19:15 INFO Completed push id=30 objects=6 duration=941.268µs49332026/11/12 07:19:15 INFO Received complete upload request method=POST path=/49342026/11/12 07:19:15 INFO Completed upload id=27 objects=3 duration=1.197531ms49352026/11/12 07:19:15 INFO Received complete push request method=POST path=/49362026/11/12 07:19:15 INFO Received push request method=POST path=/49372026/11/12 07:19:15 INFO Received push request method=POST path=/49382026/11/12 07:19:15 INFO Completed push id=5 objects=28 duration=1.787397ms49392026/11/12 07:19:15 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."49402026/11/12 07:19:15 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."49412026/11/12 07:19:15 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."49422026/11/12 07:19:15 INFO Received push request method=POST path=/49432026/11/12 07:19:15 INFO Received complete upload request method=POST path=/49442026/11/12 07:19:15 INFO Completed upload id=22 objects=7 duration=1.191311ms49452026/11/12 07:19:15 INFO Received complete upload request method=POST path=/49462026/11/12 07:19:15 INFO Completed upload id=32 objects=5 duration=1.351153ms49472026/11/12 07:19:15 INFO Received complete upload request method=POST path=/49482026/11/12 07:19:15 INFO Completed upload id=31 objects=3 duration=663.226µs49492026/11/12 07:19:15 INFO Received complete upload request method=POST path=/49502026/11/12 07:19:15 INFO Completed upload id=28 objects=6 duration=877.348µs49512026/11/12 07:19:15 INFO Received push request method=POST path=/49522026/11/12 07:19:15 INFO Received complete push request method=POST path=/49532026/11/12 07:19:15 INFO Completed push id=23 objects=5 duration=1.415553ms49542026/11/12 07:19:15 INFO Received complete push request method=POST path=/49552026/11/12 07:19:15 INFO Received uploads request method=POST path=/49562026/11/12 07:19:15 INFO Completed push id=10 objects=2 duration=1.750056ms49572026/11/12 07:19:15 INFO Received complete push request method=POST path=/49582026/11/12 07:19:15 INFO Received push request method=POST path=/49592026/11/12 07:19:15 INFO Received complete push request method=POST path=/49602026/11/12 07:19:15 INFO Completed push id=19 objects=14 duration=1.790856ms49612026/11/12 07:19:15 INFO Completed push id=24 objects=5 duration=1.024089ms49622026/11/12 07:19:15 INFO Received complete push request method=POST path=/49632026/11/12 07:19:15 INFO Completed push id=30 objects=4 duration=921.369µs49642026/11/12 07:19:15 INFO Received complete upload request method=POST path=/49652026/11/12 07:19:15 INFO Completed upload id=33 objects=3 duration=670.446µs49662026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=049672026/11/12 07:19:15 INFO Received push request method=POST path=/49682026/11/12 07:19:15 INFO Received complete upload request method=POST path=/49692026/11/12 07:19:15 INFO Completed upload id=24 objects=3 duration=713.646µs49702026/11/12 07:19:15 WARN Force mode enabled - objects will be deleted immediately without grace period49712026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049722026/11/12 07:19:15 INFO Received uploads request method=POST path=/49732026/11/12 07:19:15 INFO Vacuumed table table=pending_closures49742026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts49752026/11/12 07:19:15 INFO Received push request method=POST path=/49762026/11/12 07:19:15 INFO Received uploads request method=POST path=/49772026/11/12 07:19:15 INFO Vacuumed table table=pending_objects49782026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads49792026/11/12 07:19:15 INFO Received complete push request method=POST path=/49802026/11/12 07:19:15 INFO Received complete upload request method=POST path=/49812026/11/12 07:19:15 INFO Vacuumed table table=closures49822026/11/12 07:19:15 INFO Completed push id=31 objects=7 duration=833.848µs49832026/11/12 07:19:15 INFO Vacuumed table table=objects49842026/11/12 07:19:15 INFO Completed upload id=17 objects=5 duration=1.11601ms49852026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=049862026/11/12 07:19:15 WARN Force mode enabled - objects will be deleted immediately without grace period49872026/11/12 07:19:15 INFO Received complete upload request method=POST path=/49882026/11/12 07:19:15 INFO Completed upload id=25 objects=8 duration=812.308µs49892026/11/12 07:19:15 INFO Received push request method=POST path=/49902026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049912026/11/12 07:19:15 INFO Received complete push request method=POST path=/49922026/11/12 07:19:15 INFO Vacuumed table table=pending_closures49932026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts49942026/11/12 07:19:15 INFO Completed push id=25 objects=4 duration=893.868µs49952026/11/12 07:19:15 INFO Received push request method=POST path=/49962026/11/12 07:19:15 INFO Vacuumed table table=pending_objects49972026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads49982026/11/12 07:19:15 INFO Vacuumed table table=closures49992026/11/12 07:19:15 INFO Vacuumed table table=objects50002026/11/12 07:19:15 INFO Received complete push request method=POST path=/50012026/11/12 07:19:15 INFO Received complete upload request method=POST path=/50022026/11/12 07:19:15 INFO Completed upload id=32 objects=5 duration=1.07001ms50032026/11/12 07:19:15 INFO Completed push id=26 objects=14 duration=2.648264ms50042026/11/12 07:19:15 INFO Received complete push request method=POST path=/50052026/11/12 07:19:15 INFO Received complete upload request method=POST path=/50062026/11/12 07:19:15 INFO Completed push id=26 objects=4 duration=815.348µs50072026/11/12 07:19:15 INFO Received complete upload request method=POST path=/50082026/11/12 07:19:15 INFO Completed upload id=20 objects=6 duration=4.030157ms50092026/11/12 07:19:15 INFO Completed upload id=22 objects=6 duration=2.778046ms50102026/11/12 07:19:15 INFO Received complete push request method=POST path=/50112026/11/12 07:19:15 INFO Received complete push request method=POST path=/50122026/11/12 07:19:15 INFO Completed push id=19 objects=3 duration=1.242791ms50132026/11/12 07:19:15 INFO Completed push id=28 objects=11 duration=2.325041ms50142026/11/12 07:19:15 INFO Received push request method=POST path=/50152026/11/12 07:19:15 INFO Received complete push request method=POST path=/50162026/11/12 07:19:15 INFO Completed push id=30 objects=2 duration=873.388µs50172026/11/12 07:19:15 INFO Received uploads request method=POST path=/50182026/11/12 07:19:15 INFO Received uploads request method=POST path=/50192026/11/12 07:19:15 INFO Received complete push request method=POST path=/50202026/11/12 07:19:15 INFO Completed push id=20 objects=4 duration=1.208451ms50212026/11/12 07:19:15 INFO Received uploads request method=POST path=/50222026/11/12 07:19:15 INFO Received complete push request method=POST path=/50232026/11/12 07:19:15 INFO Received complete push request method=POST path=/50242026/11/12 07:19:15 INFO Received complete push request method=POST path=/50252026/11/12 07:19:15 INFO Completed push id=26 objects=23 duration=1.264911ms50262026/11/12 07:19:15 INFO Completed push id=32 objects=7 duration=1.04937ms50272026/11/12 07:19:15 INFO Completed push id=21 objects=3 duration=1.158171ms50282026/11/12 07:19:15 INFO Received complete upload request method=POST path=/50292026/11/12 07:19:15 INFO Completed upload id=34 objects=24 duration=1.08467ms50302026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=050312026/11/12 07:19:15 INFO Received complete push request method=POST path=/50322026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=050332026/11/12 07:19:15 INFO Completed push id=29 objects=24 duration=1.445853ms50342026/11/12 07:19:15 INFO Vacuumed table table=pending_closures50352026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts50362026/11/12 07:19:15 INFO Vacuumed table table=pending_objects50372026/11/12 07:19:15 INFO Received uploads request method=POST path=/50382026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads50392026/11/12 07:19:15 INFO Vacuumed table table=closures50402026/11/12 07:19:15 INFO Vacuumed table table=objects50412026/11/12 07:19:15 INFO Received uploads request method=POST path=/50422026/11/12 07:19:15 INFO Received complete upload request method=POST path=/50432026/11/12 07:19:15 INFO Received complete upload request method=POST path=/50442026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=050452026/11/12 07:19:15 INFO Completed upload id=22 objects=8 duration=1.03985ms50462026/11/12 07:19:15 INFO Completed upload id=33 objects=6 duration=957.769µs50472026/11/12 07:19:15 WARN Force mode enabled - objects will be deleted immediately without grace period50482026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=050492026/11/12 07:19:15 WARN Force mode enabled - objects will be deleted immediately without grace period50502026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=050512026/11/12 07:19:15 INFO Vacuumed table table=pending_closures50522026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts50532026/11/12 07:19:15 INFO Received complete upload request method=POST path=/50542026/11/12 07:19:15 INFO Received push request method=POST path=/50552026/11/12 07:19:15 INFO Vacuumed table table=pending_objects50562026/11/12 07:19:15 INFO Completed upload id=21 objects=2 duration=2.056058ms50572026/11/12 07:19:15 INFO Received complete push request method=POST path=/50582026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads50592026/11/12 07:19:15 INFO Vacuumed table table=closures50602026/11/12 07:19:15 INFO Completed push id=27 objects=15 duration=1.175691ms50612026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=050622026/11/12 07:19:15 INFO Vacuumed table table=objects50632026/11/12 07:19:15 INFO Vacuumed table table=pending_closures50642026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts50652026/11/12 07:19:15 INFO Vacuumed table table=pending_objects50662026/11/12 07:19:15 INFO Received complete push request method=POST path=/50672026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads50682026/11/12 07:19:15 INFO Vacuumed table table=closures50692026/11/12 07:19:15 INFO Completed push id=23 objects=5 duration=957.289µs50702026/11/12 07:19:15 INFO Vacuumed table table=objects50712026/11/12 07:19:15 INFO Received complete upload request method=POST path=/50722026/11/12 07:19:15 INFO Completed upload id=34 objects=3 duration=966.169µs50732026/11/12 07:19:15 INFO Received push request method=POST path=/50742026/11/12 07:19:15 INFO Received uploads request method=POST path=/50752026/11/12 07:19:15 INFO Received push request method=POST path=/50762026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=050772026/11/12 07:19:15 INFO Received complete upload request method=POST path=/50782026/11/12 07:19:15 INFO Completed upload id=24 objects=8 duration=830.727µs50792026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=050802026/11/12 07:19:15 WARN Force mode enabled - objects will be deleted immediately without grace period50812026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=3 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=050822026/11/12 07:19:15 INFO Vacuumed table table=pending_closures50832026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts50842026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=050852026/11/12 07:19:15 INFO Received push request method=POST path=/50862026/11/12 07:19:15 INFO Received uploads request method=POST path=/50872026/11/12 07:19:15 INFO Vacuumed table table=pending_objects50882026/11/12 07:19:15 INFO Vacuumed table table=pending_closures50892026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads50902026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts50912026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=050922026/11/12 07:19:15 INFO Vacuumed table table=closures50932026/11/12 07:19:15 INFO Vacuumed table table=objects50942026/11/12 07:19:15 WARN Force mode enabled - objects will be deleted immediately without grace period50952026/11/12 07:19:15 INFO Vacuumed table table=pending_objects50962026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads50972026/11/12 07:19:15 INFO Vacuumed table table=closures50982026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=050992026/11/12 07:19:15 INFO Received complete upload request method=POST path=/51002026/11/12 07:19:15 INFO Vacuumed table table=objects51012026/11/12 07:19:15 INFO Vacuumed table table=pending_closures51022026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts51032026/11/12 07:19:15 INFO Completed upload id=28 objects=5 duration=919.849µs51042026/11/12 07:19:15 INFO Vacuumed table table=pending_objects51052026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads51062026/11/12 07:19:15 INFO Vacuumed table table=closures51072026/11/12 07:19:15 INFO Vacuumed table table=objects51082026/11/12 07:19:15 INFO Received complete push request method=POST path=/51092026/11/12 07:19:15 INFO Received uploads request method=POST path=/51102026/11/12 07:19:15 INFO Received push request method=POST path=/51112026/11/12 07:19:15 INFO Completed push id=28 objects=19 duration=1.510234ms51122026/11/12 07:19:15 INFO Received push request method=POST path=/51132026/11/12 07:19:15 INFO Received complete upload request method=POST path=/51142026/11/12 07:19:15 INFO Completed upload id=30 objects=5 duration=803.908µs51152026/11/12 07:19:15 INFO Received uploads request method=POST path=/51162026/11/12 07:19:15 INFO Received push request method=POST path=/51172026/11/12 07:19:15 INFO Received uploads request method=POST path=/51182026/11/12 07:19:15 INFO Received complete push request method=POST path=/51192026/11/12 07:19:15 INFO Completed push id=29 objects=10 duration=1.055709ms51202026/11/12 07:19:15 INFO Received complete push request method=POST path=/51212026/11/12 07:19:15 INFO Received uploads request method=POST path=/51222026/11/12 07:19:15 INFO Completed push id=31 objects=3 duration=1.134771ms51232026/11/12 07:19:15 INFO Received complete push request method=POST path=/51242026/11/12 07:19:15 INFO Completed push id=30 objects=4 duration=2.980708ms51252026/11/12 07:19:15 INFO Received push request method=POST path=/51262026/11/12 07:19:15 INFO Received uploads request method=POST path=/51272026/11/12 07:19:15 INFO Received complete upload request method=POST path=/51282026/11/12 07:19:15 INFO Completed upload id=30 objects=6 duration=1.388373ms51292026/11/12 07:19:15 INFO Received uploads request method=POST path=/51302026/11/12 07:19:15 INFO Received complete push request method=POST path=/51312026/11/12 07:19:15 INFO Received complete push request method=POST path=/51322026/11/12 07:19:15 INFO Received push request method=POST path=/51332026/11/12 07:19:15 INFO Completed push id=32 objects=6 duration=1.195191ms51342026/11/12 07:19:15 INFO Completed push id=24 objects=24 duration=3.321211ms51352026/11/12 07:19:15 INFO Received push request method=POST path=/51362026/11/12 07:19:15 INFO Received complete upload request method=POST path=/51372026/11/12 07:19:15 INFO Completed upload id=31 objects=5 duration=840.088µs51382026/11/12 07:19:15 INFO Received push request method=POST path=/51392026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=051402026/11/12 07:19:15 INFO Received complete upload request method=POST path=/51412026/11/12 07:19:15 INFO Completed upload id=33 objects=7 duration=979.889µs51422026/11/12 07:19:15 WARN Force mode enabled - objects will be deleted immediately without grace period51432026/11/12 07:19:15 INFO Received complete upload request method=POST path=/51442026/11/12 07:19:15 INFO Completed upload id=34 objects=3 duration=817.848µs51452026/11/12 07:19:15 INFO Received complete push request method=POST path=/51462026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=051472026/11/12 07:19:15 INFO Vacuumed table table=pending_closures51482026/11/12 07:19:15 INFO Received push request method=POST path=/51492026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts51502026/11/12 07:19:15 INFO Completed push id=27 objects=25 duration=2.998728ms51512026/11/12 07:19:15 INFO Vacuumed table table=pending_objects51522026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads51532026/11/12 07:19:15 INFO Vacuumed table table=closures51542026/11/12 07:19:15 INFO Vacuumed table table=objects51552026/11/12 07:19:15 INFO Received complete upload request method=POST path=/51562026/11/12 07:19:15 INFO Completed upload id=34 objects=3 duration=744.887µs51572026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=051582026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=051592026/11/12 07:19:15 INFO Received complete push request method=POST path=/51602026/11/12 07:19:15 WARN Force mode enabled - objects will be deleted immediately without grace period51612026/11/12 07:19:15 INFO Completed push id=32 objects=7 duration=1.138931ms51622026/11/12 07:19:15 INFO Received uploads request method=POST path=/51632026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=051642026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=051652026/11/12 07:19:15 INFO Vacuumed table table=pending_closures51662026/11/12 07:19:15 INFO Vacuumed table table=pending_closures51672026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts51682026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts51692026/11/12 07:19:15 INFO Vacuumed table table=pending_objects51702026/11/12 07:19:15 INFO Vacuumed table table=pending_objects51712026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads51722026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads51732026/11/12 07:19:15 INFO Vacuumed table table=closures51742026/11/12 07:19:15 INFO Vacuumed table table=closures51752026/11/12 07:19:15 INFO Vacuumed table table=objects51762026/11/12 07:19:15 INFO Vacuumed table table=objects51772026/11/12 07:19:15 INFO Received complete push request method=POST path=/51782026/11/12 07:19:15 INFO Received complete push request method=POST path=/51792026/11/12 07:19:15 INFO Completed push id=26 objects=4 duration=1.15069ms51802026/11/12 07:19:15 INFO Received uploads request method=POST path=/51812026/11/12 07:19:15 INFO Completed push id=16 objects=7 duration=1.951518ms51822026/11/12 07:19:15 INFO Received complete push request method=POST path=/51832026/11/12 07:19:15 INFO Completed push id=24 objects=5 duration=3.046188ms51842026/11/12 07:19:15 INFO Received complete push request method=POST path=/51852026/11/12 07:19:15 INFO Completed push id=35 objects=11 duration=1.04229ms51862026/11/12 07:19:15 INFO Received complete upload request method=POST path=/51872026/11/12 07:19:15 INFO Completed upload id=28 objects=13 duration=905.908µs51882026/11/12 07:19:15 INFO Received complete push request method=POST path=/51892026/11/12 07:19:15 INFO Completed push id=23 objects=6 duration=1.12213ms51902026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=051912026/11/12 07:19:15 INFO Received complete push request method=POST path=/51922026/11/12 07:19:15 INFO Completed push id=29 objects=3 duration=791.708µs51932026/11/12 07:19:15 WARN Force mode enabled - objects will be deleted immediately without grace period51942026/11/12 07:19:15 INFO Received push request method=POST path=/51952026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=6 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=051962026/11/12 07:19:15 INFO Vacuumed table table=pending_closures51972026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts51982026/11/12 07:19:15 INFO Vacuumed table table=pending_objects51992026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads52002026/11/12 07:19:15 INFO Received uploads request method=POST path=/52012026/11/12 07:19:15 INFO Vacuumed table table=closures52022026/11/12 07:19:15 INFO Vacuumed table table=objects52032026/11/12 07:19:15 INFO Received complete push request method=POST path=/52042026/11/12 07:19:15 INFO Completed push id=24 objects=6 duration=1.06459ms52052026/11/12 07:19:15 INFO Received complete upload request method=POST path=/52062026/11/12 07:19:15 INFO Received uploads request method=POST path=/52072026/11/12 07:19:15 INFO Completed upload id=31 objects=2 duration=933.789µs52082026/11/12 07:19:15 INFO Received complete push request method=POST path=/52092026/11/12 07:19:15 INFO Completed push id=33 objects=6 duration=865.448µs52102026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=052112026/11/12 07:19:15 WARN Force mode enabled - objects will be deleted immediately without grace period52122026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=052132026/11/12 07:19:15 INFO Received complete push request method=POST path=/52142026/11/12 07:19:15 INFO Vacuumed table table=pending_closures52152026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts52162026/11/12 07:19:15 INFO Completed push id=25 objects=7 duration=1.503254ms52172026/11/12 07:19:15 INFO Vacuumed table table=pending_objects52182026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads52192026/11/12 07:19:15 INFO Vacuumed table table=closures52202026/11/12 07:19:15 INFO Received complete push request method=POST path=/52212026/11/12 07:19:15 INFO Vacuumed table table=objects52222026/11/12 07:19:15 INFO Completed push id=21 objects=26 duration=1.550755ms52232026/11/12 07:19:15 INFO Aborted multipart uploads count=0 kept=052242026/11/12 07:19:15 WARN Force mode enabled - objects will be deleted immediately without grace period52252026/11/12 07:19:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=052262026/11/12 07:19:15 INFO Vacuumed table table=pending_closures52272026/11/12 07:19:15 INFO Vacuumed table table=commit_receipts52282026/11/12 07:19:15 INFO Vacuumed table table=pending_objects52292026/11/12 07:19:15 INFO Vacuumed table table=multipart_uploads52302026/11/12 07:19:15 INFO Vacuumed table table=closures52312026/11/12 07:19:15 INFO Vacuumed table table=objects52322026/11/12 07:19:15 INFO Received complete upload request method=POST path=/52332026/11/12 07:19:15 INFO Completed upload id=28 objects=17 duration=919.669µs52342026/11/12 07:19:15 INFO Received push request method=POST path=/52352026/11/12 07:19:15 INFO Received complete upload request method=POST path=/52362026/11/12 07:19:15 INFO Completed upload id=26 objects=24 duration=1.286812ms52372026/11/12 07:19:15 INFO Received complete push request method=POST path=/52382026/11/12 07:19:15 INFO Completed push id=29 objects=2 duration=1.12461ms52392026/11/12 07:19:15 INFO Received complete upload request method=POST path=/52402026/11/12 07:19:15 INFO Received complete push request method=POST path=/52412026/11/12 07:19:15 INFO Completed upload id=23 objects=7 duration=3.370631ms52422026/11/12 07:19:15 INFO Completed push id=31 objects=6 duration=1.10967ms52432026/11/12 07:19:15 INFO Received complete upload request method=POST path=/52442026/11/12 07:19:15 INFO Received complete upload request method=POST path=/52452026/11/12 07:19:15 INFO Completed upload id=26 objects=5 duration=880.848µs52462026/11/12 07:19:15 INFO Completed upload id=27 objects=26 duration=1.11497ms52472026/11/12 07:19:15 INFO Received complete push request method=POST path=/52482026/11/12 07:19:15 INFO Completed push id=32 objects=9 duration=898.668µs52492026/11/12 07:19:16 INFO Received complete push request method=POST path=/52502026/11/12 07:19:16 INFO Received complete push request method=POST path=/52512026/11/12 07:19:16 INFO Completed push id=27 objects=3 duration=2.14516ms52522026/11/12 07:19:16 INFO Completed push id=28 objects=6 duration=1.141151ms52532026/11/12 07:19:16 INFO Received complete upload request method=POST path=/52542026/11/12 07:19:16 INFO Completed upload id=33 objects=9 duration=815.667µs52552026/11/12 07:19:16 INFO Received complete upload request method=POST path=/52562026/11/12 07:19:16 INFO Received complete push request method=POST path=/52572026/11/12 07:19:16 INFO Completed upload id=28 objects=5 duration=809.707µs52582026/11/12 07:19:16 INFO Completed push id=29 objects=6 duration=1.11761ms52592026/11/12 07:19:16 INFO Aborted multipart uploads count=0 kept=052602026/11/12 07:19:16 WARN Force mode enabled - objects will be deleted immediately without grace period52612026/11/12 07:19:16 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=052622026/11/12 07:19:16 INFO Vacuumed table table=pending_closures52632026/11/12 07:19:16 INFO Vacuumed table table=commit_receipts52642026/11/12 07:19:16 INFO Vacuumed table table=pending_objects52652026/11/12 07:19:16 INFO Vacuumed table table=multipart_uploads52662026/11/12 07:19:16 INFO Vacuumed table table=closures52672026/11/12 07:19:16 INFO Vacuumed table table=objects52682026/11/12 07:19:16 INFO Received complete push request method=POST path=/52692026/11/12 07:19:16 INFO Completed push id=29 objects=17 duration=1.963619ms52702026/11/12 07:19:16 INFO Aborted multipart uploads count=0 kept=052712026/11/12 07:19:16 WARN Force mode enabled - objects will be deleted immediately without grace period52722026/11/12 07:19:16 INFO Received complete upload request method=POST path=/52732026/11/12 07:19:16 INFO Completed upload id=30 objects=8 duration=1.011109ms52742026/11/12 07:19:16 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=052752026/11/12 07:19:16 INFO Vacuumed table table=pending_closures52762026/11/12 07:19:16 INFO Vacuumed table table=commit_receipts52772026/11/12 07:19:16 INFO Vacuumed table table=pending_objects52782026/11/12 07:19:16 INFO Vacuumed table table=multipart_uploads52792026/11/12 07:19:16 INFO Vacuumed table table=closures52802026/11/12 07:19:16 INFO Vacuumed table table=objects52812026/11/12 07:19:16 INFO Received complete upload request method=POST path=/52822026/11/12 07:19:16 INFO Completed upload id=31 objects=6 duration=871.448µs52832026/11/12 07:19:16 INFO Received complete upload request method=POST path=/52842026/11/12 07:19:16 INFO Completed upload id=32 objects=4 duration=565.366µs52852026/11/12 07:19:16 INFO Received complete upload request method=POST path=/52862026/11/12 07:19:16 INFO Completed upload id=33 objects=10 duration=710.066µs52872026/11/12 07:19:16 INFO Received complete upload request method=POST path=/52882026/11/12 07:19:16 INFO Completed upload id=34 objects=23 duration=720.206µs52892026/11/12 07:19:16 INFO Received complete push request method=POST path=/52902026/11/12 07:19:16 INFO Completed push id=35 objects=6 duration=778.748µs52912026/11/12 07:19:16 INFO Aborted multipart uploads count=0 kept=052922026/11/12 07:19:16 WARN Force mode enabled - objects will be deleted immediately without grace period52932026/11/12 07:19:16 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=052942026/11/12 07:19:16 INFO Vacuumed table table=pending_closures52952026/11/12 07:19:16 INFO Vacuumed table table=commit_receipts52962026/11/12 07:19:16 INFO Vacuumed table table=pending_objects52972026/11/12 07:19:16 INFO Vacuumed table table=multipart_uploads52982026/11/12 07:19:16 INFO Vacuumed table table=closures52992026/11/12 07:19:16 INFO Vacuumed table table=objects5300--- PASS: FuzzPushGC (0.02s)5301 --- PASS: FuzzPushGC/seed#3 (3.23s)5302 --- PASS: FuzzPushGC/seed#11 (3.37s)5303 --- PASS: FuzzPushGC/seed#7 (3.38s)5304 --- PASS: FuzzPushGC/seed#4 (3.57s)5305 --- PASS: FuzzPushGC/seed#5 (3.59s)5306 --- PASS: FuzzPushGC/seed#15 (3.67s)5307 --- PASS: FuzzPushGC/seed#6 (3.68s)5308 --- PASS: FuzzPushGC/seed#9 (3.75s)5309 --- PASS: FuzzPushGC/seed#2 (3.75s)5310 --- PASS: FuzzPushGC/seed#10 (3.78s)5311 --- PASS: FuzzPushGC/seed#13 (3.89s)5312 --- PASS: FuzzPushGC/seed#8 (3.94s)5313 --- PASS: FuzzPushGC/seed#14 (3.98s)5314 --- PASS: FuzzPushGC/seed#0 (4.08s)5315 --- PASS: FuzzPushGC/seed#1 (4.10s)5316 --- PASS: FuzzPushGC/seed#12 (4.25s)5317PASS5318{"timestamp":"2026-11-12T07:19:16.231049573Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:60656","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(203)"}53192026-11-12 07:19:16.737 UTC [131] LOG: received smart shutdown request53202026-11-12 07:19:16.746 UTC [131] LOG: background worker "logical replication launcher" (PID 152) exited with exit code 153212026-11-12 07:19:20.100 UTC [147] LOG: checkpoint complete: wrote 3732 buffers (22.8%), wrote 5 SLRU buffers; 0 WAL file(s) added, 0 removed, 33 recycled; write=18.979 s, sync=0.464 s, total=22.158 s; sync files=43470, longest=0.018 s, average=0.001 s; distance=533917 kB, estimate=533917 kB; lsn=0/27A71E88, redo lsn=0/220CD22053222026-11-12 07:19:20.107 UTC [147] LOG: shutting down53232026-11-12 07:19:20.107 UTC [147] LOG: checkpoint starting: shutdown immediate53242026-11-12 07:19:20.568 UTC [147] LOG: checkpoint complete: wrote 9688 buffers (59.1%), wrote 2 SLRU buffers; 0 WAL file(s) added, 0 removed, 5 recycled; write=0.219 s, sync=0.229 s, total=0.461 s; sync files=3043, longest=0.012 s, average=0.001 s; distance=91795 kB, estimate=489705 kB; lsn=0/27A71F00, redo lsn=0/27A71F0053252026-11-12 07:19:20.804 UTC [131] LOG: database system is shut down5326Running OIDC tests...5327=== RUN TestAudienceForIssuer5328=== PAUSE TestAudienceForIssuer5329=== RUN TestProviderClientGivesUpOnSilentIssuer5330=== PAUSE TestProviderClientGivesUpOnSilentIssuer5331=== RUN TestGlobMatch5332=== PAUSE TestGlobMatch5333=== RUN TestValidateToken_ValidToken5334=== PAUSE TestValidateToken_ValidToken5335=== RUN TestValidateToken_WrongAudience5336=== PAUSE TestValidateToken_WrongAudience5337=== RUN TestValidateToken_Expired5338=== PAUSE TestValidateToken_Expired5339=== RUN TestValidateToken_BoundClaimsMismatch5340=== PAUSE TestValidateToken_BoundClaimsMismatch5341=== RUN TestValidateToken_BoundSubjectMismatch5342=== PAUSE TestValidateToken_BoundSubjectMismatch5343=== RUN TestValidateToken_MultipleProviders5344=== PAUSE TestValidateToken_MultipleProviders5345=== RUN TestValidateToken_UnknownIssuerFetchesNoKeys5346=== PAUSE TestValidateToken_UnknownIssuerFetchesNoKeys5347=== RUN TestValidateToken_GoogleSchemelessIssuer5348=== PAUSE TestValidateToken_GoogleSchemelessIssuer5349=== RUN TestValidateToken_KubernetesServiceAccount5350=== PAUSE TestValidateToken_KubernetesServiceAccount5351=== RUN TestNewValidator_KubernetesRequiresCA5352=== PAUSE TestNewValidator_KubernetesRequiresCA5353=== RUN TestValidateToken_KubernetesIssuerFromOwnToken5354=== PAUSE TestValidateToken_KubernetesIssuerFromOwnToken5355=== RUN TestPins_ReservedForMatchingRule5356=== PAUSE TestPins_ReservedForMatchingRule5357=== RUN TestPins_TopLevelShorthand5358=== PAUSE TestPins_TopLevelShorthand5359=== RUN TestPins_ConfigValidation5360=== PAUSE TestPins_ConfigValidation5361=== RUN TestScopes_LegacyProviderDefaultsToWrite5362=== PAUSE TestScopes_LegacyProviderDefaultsToWrite5363=== RUN TestScopes_Rules5364=== PAUSE TestScopes_Rules5365=== RUN TestScopes_ConfigValidation5366=== PAUSE TestScopes_ConfigValidation5367=== CONT TestAudienceForIssuer5368--- PASS: TestAudienceForIssuer (0.00s)5369=== CONT TestScopes_ConfigValidation5370--- PASS: TestScopes_ConfigValidation (0.00s)5371=== CONT TestScopes_Rules5372=== CONT TestScopes_LegacyProviderDefaultsToWrite5373=== CONT TestPins_ConfigValidation5374--- PASS: TestPins_ConfigValidation (0.01s)53752026/11/12 07:19:25 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:45151/oidc5376--- PASS: TestScopes_LegacyProviderDefaultsToWrite (0.15s)5377=== CONT TestPins_TopLevelShorthand5378=== CONT TestPins_ReservedForMatchingRule5379=== CONT TestValidateToken_KubernetesIssuerFromOwnToken5380=== CONT TestNewValidator_KubernetesRequiresCA5381=== CONT TestValidateToken_KubernetesServiceAccount5382=== CONT TestValidateToken_GoogleSchemelessIssuer5383=== CONT TestValidateToken_UnknownIssuerFetchesNoKeys5384=== CONT TestValidateToken_MultipleProviders5385=== CONT TestValidateToken_BoundSubjectMismatch5386=== CONT TestValidateToken_BoundClaimsMismatch5387=== CONT TestValidateToken_Expired53882026/11/12 07:19:26 INFO OIDC provider initialized name=known issuer=https://known.example5389=== CONT TestValidateToken_WrongAudience5390=== CONT TestValidateToken_ValidToken5391=== CONT TestGlobMatch5392=== RUN TestGlobMatch/foo_foo5393=== PAUSE TestGlobMatch/foo_foo5394=== RUN TestGlobMatch/foo_bar5395=== PAUSE TestGlobMatch/foo_bar5396=== RUN TestGlobMatch/*_5397=== PAUSE TestGlobMatch/*_5398=== RUN TestGlobMatch/*_anything5399=== PAUSE TestGlobMatch/*_anything5400=== RUN TestGlobMatch/foo*_foo5401=== PAUSE TestGlobMatch/foo*_foo5402=== CONT TestProviderClientGivesUpOnSilentIssuer5403=== RUN TestGlobMatch/foo*_foobar5404=== PAUSE TestGlobMatch/foo*_foobar5405=== RUN TestGlobMatch/foo*_bar5406=== PAUSE TestGlobMatch/foo*_bar5407=== RUN TestGlobMatch/*bar_bar5408=== PAUSE TestGlobMatch/*bar_bar5409=== RUN TestGlobMatch/*bar_foobar5410=== PAUSE TestGlobMatch/*bar_foobar5411=== RUN TestGlobMatch/*bar_foo5412=== PAUSE TestGlobMatch/*bar_foo5413=== RUN TestGlobMatch/foo*bar_foobar5414=== PAUSE TestGlobMatch/foo*bar_foobar5415=== RUN TestGlobMatch/foo*bar_foo123bar5416=== PAUSE TestGlobMatch/foo*bar_foo123bar5417=== RUN TestGlobMatch/foo*bar_foobarbaz5418=== PAUSE TestGlobMatch/foo*bar_foobarbaz5419=== RUN TestGlobMatch/*/*_foo/bar5420=== PAUSE TestGlobMatch/*/*_foo/bar5421=== RUN TestGlobMatch/*/*_foo5422=== PAUSE TestGlobMatch/*/*_foo5423=== RUN TestGlobMatch/refs/heads/*_refs/heads/main5424--- PASS: TestProviderClientGivesUpOnSilentIssuer (0.00s)5425=== PAUSE TestGlobMatch/refs/heads/*_refs/heads/main5426=== RUN TestGlobMatch/refs/heads/*_refs/tags/v1.05427=== PAUSE TestGlobMatch/refs/heads/*_refs/tags/v1.05428=== RUN TestGlobMatch/refs/*/main_refs/heads/main5429=== PAUSE TestGlobMatch/refs/*/main_refs/heads/main5430=== RUN TestGlobMatch/fo?_foo5431=== PAUSE TestGlobMatch/fo?_foo5432=== RUN TestGlobMatch/fo?_fo5433=== PAUSE TestGlobMatch/fo?_fo5434=== RUN TestGlobMatch/fo?_fooo5435=== PAUSE TestGlobMatch/fo?_fooo5436=== RUN TestGlobMatch/?oo_foo5437=== PAUSE TestGlobMatch/?oo_foo5438=== RUN TestGlobMatch/?oo_boo5439=== PAUSE TestGlobMatch/?oo_boo5440=== RUN TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5441=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5442=== RUN TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5443=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5444=== CONT TestGlobMatch/foo_foo5445=== CONT TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5446=== CONT TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5447=== CONT TestGlobMatch/?oo_boo5448=== CONT TestGlobMatch/?oo_foo5449=== CONT TestGlobMatch/fo?_fooo5450=== CONT TestGlobMatch/fo?_fo5451=== CONT TestGlobMatch/fo?_foo5452=== CONT TestGlobMatch/refs/*/main_refs/heads/main5453=== CONT TestGlobMatch/refs/heads/*_refs/tags/v1.05454=== CONT TestGlobMatch/refs/heads/*_refs/heads/main5455=== CONT TestGlobMatch/*/*_foo5456=== CONT TestGlobMatch/*/*_foo/bar5457=== CONT TestGlobMatch/foo*bar_foobarbaz5458=== CONT TestGlobMatch/foo*bar_foo123bar5459=== CONT TestGlobMatch/foo*bar_foobar5460=== CONT TestGlobMatch/*bar_foo5461=== CONT TestGlobMatch/*bar_foobar5462=== CONT TestGlobMatch/*bar_bar5463=== CONT TestGlobMatch/foo*_bar5464=== CONT TestGlobMatch/foo*_foobar5465=== CONT TestGlobMatch/foo*_foo5466=== CONT TestGlobMatch/*_anything5467=== CONT TestGlobMatch/*_5468=== CONT TestGlobMatch/foo_bar5469--- PASS: TestGlobMatch (0.01s)5470 --- PASS: TestGlobMatch/foo_foo (0.00s)5471 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main (0.00s)5472 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main (0.00s)5473 --- PASS: TestGlobMatch/?oo_boo (0.00s)5474 --- PASS: TestGlobMatch/?oo_foo (0.00s)5475 --- PASS: TestGlobMatch/fo?_fooo (0.00s)5476 --- PASS: TestGlobMatch/fo?_fo (0.00s)5477 --- PASS: TestGlobMatch/fo?_foo (0.00s)5478 --- PASS: TestGlobMatch/refs/*/main_refs/heads/main (0.00s)5479 --- PASS: TestGlobMatch/refs/heads/*_refs/tags/v1.0 (0.00s)5480 --- PASS: TestGlobMatch/refs/heads/*_refs/heads/main (0.00s)5481 --- PASS: TestGlobMatch/*/*_foo (0.00s)5482 --- PASS: TestGlobMatch/*/*_foo/bar (0.00s)5483 --- PASS: TestGlobMatch/foo*bar_foobarbaz (0.00s)5484 --- PASS: TestGlobMatch/foo*bar_foo123bar (0.00s)5485 --- PASS: TestGlobMatch/foo*bar_foobar (0.00s)5486 --- PASS: TestGlobMatch/*bar_foo (0.00s)5487 --- PASS: TestGlobMatch/*bar_foobar (0.00s)5488 --- PASS: TestGlobMatch/*bar_bar (0.00s)5489 --- PASS: TestGlobMatch/foo*_bar (0.00s)5490 --- PASS: TestGlobMatch/foo*_foobar (0.00s)5491 --- PASS: TestGlobMatch/foo*_foo (0.00s)5492 --- PASS: TestGlobMatch/*_anything (0.00s)5493 --- PASS: TestGlobMatch/*_ (0.00s)5494 --- PASS: TestGlobMatch/foo_bar (0.00s)54952026/11/12 07:19:27 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:36667/oidc5496--- PASS: TestPins_ReservedForMatchingRule (2.03s)54972026/11/12 07:19:27 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:42341/oidc54982026/11/12 07:19:27 INFO OIDC provider initialized name=kubernetes issuer=https://oidc.eks.invalid/id/ABC12354992026/11/12 07:19:27 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:37659/oidc55002026/11/12 07:19:27 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:46297/oidc5501--- PASS: TestValidateToken_KubernetesIssuerFromOwnToken (0.64s)5502--- PASS: TestScopes_Rules (2.53s)5503--- PASS: TestValidateToken_BoundClaimsMismatch (0.76s)5504--- PASS: TestValidateToken_ValidToken (0.75s)55052026/11/12 07:19:27 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:39923/oidc55062026/11/12 07:19:27 INFO OIDC provider initialized name=kubernetes issuer=https://127.0.0.1:405655507--- PASS: TestValidateToken_Expired (0.77s)5508--- PASS: TestValidateToken_KubernetesServiceAccount (0.77s)55092026/11/12 07:19:27 http: TLS handshake error from 127.0.0.1:37010: read tcp 127.0.0.1:44645->127.0.0.1:37010: use of closed network connection5510--- PASS: TestNewValidator_KubernetesRequiresCA (0.77s)55112026/11/12 07:19:27 INFO OIDC provider initialized name=google issuer=https://accounts.google.com5512--- PASS: TestValidateToken_GoogleSchemelessIssuer (0.79s)55132026/11/12 07:19:27 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:44111/oidc5514--- PASS: TestValidateToken_WrongAudience (0.79s)55152026/11/12 07:19:27 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:44693/oidc5516--- PASS: TestPins_TopLevelShorthand (2.40s)55172026/11/12 07:19:27 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:43021/oidc5518--- PASS: TestValidateToken_BoundSubjectMismatch (0.84s)55192026/11/12 07:19:27 INFO OIDC provider initialized name=provider1 issuer=http://127.0.0.1:43155/oidc55202026/11/12 07:19:27 INFO OIDC provider initialized name=provider2 issuer=http://127.0.0.1:35277/oidc5521--- PASS: TestValidateToken_MultipleProviders (0.96s)5522--- PASS: TestValidateToken_UnknownIssuerFetchesNoKeys (1.00s)5523PASS5524Running hook tests...5525=== RUN TestSendPathsEmpty5526=== PAUSE TestSendPathsEmpty5527=== RUN TestQueueEnqueueAndFetch5528=== PAUSE TestQueueEnqueueAndFetch5529=== RUN TestQueueDeduplication5530=== PAUSE TestQueueDeduplication5531=== RUN TestQueueRemove5532=== PAUSE TestQueueRemove5533=== RUN TestQueueFetchBatchLimit5534=== PAUSE TestQueueFetchBatchLimit5535=== RUN TestQueueRetryMovesToBack5536=== PAUSE TestQueueRetryMovesToBack5537=== RUN TestQueueFetchRemoveLifecycle5538=== PAUSE TestQueueFetchRemoveLifecycle5539=== RUN TestQueueConcurrentWriters5540=== PAUSE TestQueueConcurrentWriters5541=== RUN TestQueueRemoveLargeClosure5542=== PAUSE TestQueueRemoveLargeClosure5543=== RUN TestServerClientIntegration5544--- PASS: TestServerClientIntegration (0.00s)5545=== RUN TestServerQueueError55462026/11/12 07:19:27 ERROR Failed to queue paths error="permission denied" count=15547--- PASS: TestServerQueueError (0.00s)5548=== RUN TestGetListenerSocketActivation5549 server_test.go:220: === RUN TestGetListenerSocketActivation5550 --- PASS: TestGetListenerSocketActivation (0.00s)5551 PASS5552 5553--- PASS: TestGetListenerSocketActivation (0.03s)5554=== RUN TestServerStalledClientDoesNotBlockShutdown5555=== PAUSE TestServerStalledClientDoesNotBlockShutdown5556=== RUN TestServerBacksOffOnAcceptErrors5557=== PAUSE TestServerBacksOffOnAcceptErrors5558=== RUN TestDrainIsolatesPoisonPath5559=== PAUSE TestDrainIsolatesPoisonPath5560=== RUN TestRunNotBlockedByPoisonHead5561=== PAUSE TestRunNotBlockedByPoisonHead5562=== RUN TestDrainGivesUpWhenServerDown5563=== PAUSE TestDrainGivesUpWhenServerDown5564=== RUN TestFailedPathPrunedByLaterClosure5565=== PAUSE TestFailedPathPrunedByLaterClosure5566=== RUN TestWorkerUploadsAndRemoves5567=== PAUSE TestWorkerUploadsAndRemoves5568=== RUN TestWorkerSkipsGCdPaths5569=== PAUSE TestWorkerSkipsGCdPaths5570=== RUN TestWorkerPrunesClosureDeps5571=== PAUSE TestWorkerPrunesClosureDeps5572=== RUN TestDrainTimeout5573=== PAUSE TestDrainTimeout5574=== RUN TestWorkerRemoveFailureIsNotProgress5575=== PAUSE TestWorkerRemoveFailureIsNotProgress5576=== RUN TestWorkerKeepsPathItCannotStat5577=== PAUSE TestWorkerKeepsPathItCannotStat5578=== CONT TestSendPathsEmpty5579--- PASS: TestSendPathsEmpty (0.00s)5580=== CONT TestWorkerKeepsPathItCannotStat5581=== CONT TestQueueRetryMovesToBack5582=== CONT TestQueueRemove5583=== CONT TestDrainGivesUpWhenServerDown5584=== CONT TestQueueFetchBatchLimit5585=== CONT TestWorkerUploadsAndRemoves5586=== CONT TestWorkerRemoveFailureIsNotProgress5587=== CONT TestDrainTimeout5588=== CONT TestWorkerPrunesClosureDeps5589=== CONT TestWorkerSkipsGCdPaths5590=== CONT TestQueueDeduplication5591=== CONT TestFailedPathPrunedByLaterClosure5592=== CONT TestRunNotBlockedByPoisonHead5593=== CONT TestQueueEnqueueAndFetch5594=== CONT TestQueueRemoveLargeClosure5595=== CONT TestServerStalledClientDoesNotBlockShutdown5596=== CONT TestDrainIsolatesPoisonPath5597=== CONT TestServerBacksOffOnAcceptErrors5598=== CONT TestQueueConcurrentWriters5599=== CONT TestQueueFetchRemoveLifecycle56002000/01/01 00:00:00 ERROR Accept failed error="too many open files"56012000/01/01 00:00:00 ERROR Accept failed error="too many open files"56022000/01/01 00:00:00 ERROR Accept failed error="too many open files"56032000/01/01 00:00:00 ERROR Accept failed error="too many open files"56042000/01/01 00:00:00 ERROR Accept failed error="too many open files"56052000/01/01 00:00:00 ERROR Accept failed error="too many open files"56062000/01/01 00:00:00 ERROR Accept failed error="too many open files"56072000/01/01 00:00:00 ERROR Accept failed error="too many open files"5608--- PASS: TestServerBacksOffOnAcceptErrors (0.00s)56092026/11/12 07:19:27 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat4213680809/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat4213680809/002/locked/aaa: permission denied"56102026/11/12 07:19:27 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat4213680809/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat4213680809/002/locked/aaa: permission denied"56112026/11/12 07:19:27 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat4213680809/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat4213680809/002/locked/aaa: permission denied"56122026/11/12 07:19:27 ERROR Drain finished with paths left in queue remaining=15613--- PASS: TestWorkerKeepsPathItCannotStat (0.02s)5614--- PASS: TestQueueFetchBatchLimit (0.03s)56152026/11/12 07:19:27 INFO Upload queue status pending=256162026/11/12 07:19:27 INFO Uploading batch count=256172026/11/12 07:19:27 INFO Uploading batch count=256182026/11/12 07:19:27 ERROR Upload failed error="upload failed" count=256192026/11/12 07:19:27 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown1901026541/002/a56202026/11/12 07:19:27 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown1901026541/002/b56212026/11/12 07:19:27 INFO Uploading batch count=256222026/11/12 07:19:27 ERROR Upload failed error="upload failed" count=256232026/11/12 07:19:27 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown1901026541/002/c56242026/11/12 07:19:27 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown1901026541/002/d56252026/11/12 07:19:27 INFO Uploading batch count=256262026/11/12 07:19:27 ERROR Upload failed error="upload failed" count=256272026/11/12 07:19:27 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown1901026541/002/e56282026/11/12 07:19:27 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown1901026541/002/f56292026/11/12 07:19:27 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress2298458726/002/nonexistent56302026/11/12 07:19:27 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=156312026/11/12 07:19:27 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress2298458726/002/nonexistent56322026/11/12 07:19:27 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=156332026/11/12 07:19:27 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress2298458726/002/nonexistent56342026/11/12 07:19:27 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=156352026/11/12 07:19:27 INFO Uploading batch count=456362026/11/12 07:19:27 ERROR Upload failed error="upload failed" count=456372026/11/12 07:19:27 ERROR Drain finished with paths left in queue remaining=156382026/11/12 07:19:27 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainIsolatesPoisonPath3031193300/002/bbb5639--- PASS: TestWorkerRemoveFailureIsNotProgress (0.04s)56402026/11/12 07:19:27 INFO Uploading batch count=156412026/11/12 07:19:27 ERROR Upload failed error="upload failed" count=156422026/11/12 07:19:27 INFO Uploading batch count=156432026/11/12 07:19:27 ERROR Upload failed error="upload failed" count=15644--- PASS: TestQueueRetryMovesToBack (0.04s)56452026/11/12 07:19:27 ERROR Drain finished with paths left in queue remaining=105646--- PASS: TestDrainGivesUpWhenServerDown (0.04s)56472026/11/12 07:19:27 INFO Uploading batch count=156482026/11/12 07:19:27 ERROR Upload failed error="upload failed" count=156492026/11/12 07:19:27 ERROR Drain finished with paths left in queue remaining=15650--- PASS: TestDrainIsolatesPoisonPath (0.04s)5651--- PASS: TestQueueEnqueueAndFetch (0.05s)5652--- PASS: TestWorkerUploadsAndRemoves (0.05s)56532026/11/12 07:19:27 INFO Upload queue status pending=256542026/11/12 07:19:27 INFO Uploading batch count=156552026/11/12 07:19:27 INFO Uploading batch count=156562026/11/12 07:19:27 ERROR Upload failed error="upload failed" count=156572026/11/12 07:19:27 INFO Uploading batch count=156582000/01/01 00:00:00 INFO Uploading batch count=256592000/01/01 00:00:00 ERROR Upload failed error="context deadline exceeded" count=256602000/01/01 00:00:00 ERROR Drain finished with paths left in queue remaining=45661--- PASS: TestQueueFetchRemoveLifecycle (0.06s)5662--- PASS: TestDrainTimeout (0.06s)5663--- PASS: TestQueueRemove (0.06s)56642026/11/12 07:19:27 INFO Uploading batch count=156652026/11/12 07:19:27 INFO Upload queue status pending=356662026/11/12 07:19:27 INFO Uploading batch count=156672026/11/12 07:19:27 ERROR Upload failed error="upload failed" count=15668--- PASS: TestQueueDeduplication (0.08s)5669--- PASS: TestWorkerPrunesClosureDeps (0.08s)5670--- PASS: TestFailedPathPrunedByLaterClosure (0.08s)56712026/11/12 07:19:27 INFO Upload queue status pending=256722026/11/12 07:19:27 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerSkipsGCdPaths1201230977/002/nonexistent56732026/11/12 07:19:27 INFO Uploading batch count=15674--- PASS: TestWorkerSkipsGCdPaths (0.10s)5675--- PASS: TestQueueConcurrentWriters (0.18s)5676--- PASS: TestQueueRemoveLargeClosure (0.18s)56772026/11/12 07:19:28 ERROR Failed to decode request error="read unix /build/hook2105518778/test.sock->@: i/o timeout"56782026/11/12 07:19:28 ERROR Failed to write response error="write unix /build/hook2105518778/test.sock->@: i/o timeout"5679--- PASS: TestServerStalledClientDoesNotBlockShutdown (0.20s)56802026/11/12 07:19:28 INFO Uploading batch count=156812026/11/12 07:19:28 INFO Uploading batch count=156822026/11/12 07:19:28 INFO Uploading batch count=156832026/11/12 07:19:28 ERROR Upload failed error="upload failed" count=156842026/11/12 07:19:28 INFO Uploading batch count=156852026/11/12 07:19:28 ERROR Upload failed error="upload failed" count=156862026/11/12 07:19:28 INFO Uploading batch count=156872026/11/12 07:19:28 ERROR Upload failed error="upload failed" count=156882026/11/12 07:19:28 INFO Uploading batch count=156892026/11/12 07:19:28 ERROR Upload failed error="upload failed" count=156902026/11/12 07:19:28 ERROR Drain finished with paths left in queue remaining=15691--- PASS: TestRunNotBlockedByPoisonHead (1.10s)5692PASS5693Running signing tests...5694=== RUN TestExternalSigner5695=== PAUSE TestExternalSigner5696=== RUN TestExternalSignerWrongPublicKey5697=== PAUSE TestExternalSignerWrongPublicKey5698=== RUN TestExternalSignerErrors5699=== PAUSE TestExternalSignerErrors5700=== RUN TestExternalSignerRestartsAfterCrash5701=== PAUSE TestExternalSignerRestartsAfterCrash5702=== RUN TestExternalSignerProgramErrorKeepsProcess5703=== PAUSE TestExternalSignerProgramErrorKeepsProcess5704=== RUN TestExternalSignerClose5705=== PAUSE TestExternalSignerClose5706=== RUN TestExternalSignerRejectsInvalidPublicKey5707=== PAUSE TestExternalSignerRejectsInvalidPublicKey5708=== RUN TestExternalSignerCancellation5709=== PAUSE TestExternalSignerCancellation5710=== RUN TestGenerateFingerprint5711=== PAUSE TestGenerateFingerprint5712=== RUN TestParseSigningKey5713=== PAUSE TestParseSigningKey5714=== RUN TestSign5715=== PAUSE TestSign5716=== RUN TestSignBatch5717=== PAUSE TestSignBatch5718=== RUN TestNilKeyPublicKey5719=== PAUSE TestNilKeyPublicKey5720=== RUN TestSignErrors5721=== PAUSE TestSignErrors5722=== CONT TestExternalSigner5723=== CONT TestExternalSignerCancellation5724=== CONT TestSignBatch5725=== CONT TestExternalSignerProgramErrorKeepsProcess5726=== CONT TestSignErrors5727=== CONT TestNilKeyPublicKey5728--- PASS: TestNilKeyPublicKey (0.00s)5729=== CONT TestExternalSignerRejectsInvalidPublicKey5730=== RUN TestExternalSignerRejectsInvalidPublicKey/#005731=== PAUSE TestExternalSignerRejectsInvalidPublicKey/#005732=== RUN TestExternalSignerRejectsInvalidPublicKey/no-colon5733=== PAUSE TestExternalSignerRejectsInvalidPublicKey/no-colon5734=== CONT TestParseSigningKey5735=== RUN TestParseSigningKey/valid_32-byte_key5736=== PAUSE TestParseSigningKey/valid_32-byte_key5737=== RUN TestParseSigningKey/valid_32-byte_key_with_different_name5738=== PAUSE TestParseSigningKey/valid_32-byte_key_with_different_name5739=== RUN TestParseSigningKey/no_colon5740=== PAUSE TestParseSigningKey/no_colon5741=== RUN TestParseSigningKey/empty_name5742=== PAUSE TestParseSigningKey/empty_name5743=== RUN TestParseSigningKey/invalid_base645744=== PAUSE TestParseSigningKey/invalid_base645745=== RUN TestParseSigningKey/wrong_length5746=== CONT TestSign5747=== CONT TestGenerateFingerprint5748=== CONT TestExternalSignerClose5749=== CONT TestExternalSignerErrors5750=== CONT TestExternalSignerRestartsAfterCrash5751=== CONT TestExternalSignerWrongPublicKey5752=== RUN TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5753=== PAUSE TestParseSigningKey/wrong_length5754=== CONT TestParseSigningKey/valid_32-byte_key5755=== RUN TestExternalSignerErrors/exit_at_startup5756=== PAUSE TestExternalSignerErrors/exit_at_startup5757=== RUN TestExternalSignerErrors/unrequested_key5758=== PAUSE TestExternalSignerErrors/unrequested_key5759=== RUN TestExternalSignerErrors/missing_signature_name5760=== PAUSE TestExternalSignerErrors/missing_signature_name5761=== CONT TestParseSigningKey/invalid_base645762=== CONT TestParseSigningKey/wrong_length5763=== CONT TestParseSigningKey/no_colon5764=== CONT TestParseSigningKey/valid_32-byte_key_with_different_name5765=== CONT TestParseSigningKey/empty_name5766=== PAUSE TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5767=== RUN TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5768=== RUN TestGenerateFingerprint/basic_with_references5769=== PAUSE TestGenerateFingerprint/basic_with_references5770=== RUN TestGenerateFingerprint/no_references5771=== PAUSE TestGenerateFingerprint/no_references5772=== RUN TestGenerateFingerprint/unsorted_references_get_sorted5773=== PAUSE TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5774=== RUN TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5775=== PAUSE TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5776=== CONT TestExternalSignerRejectsInvalidPublicKey/#005777=== CONT TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5778--- PASS: TestSign (0.00s)5779=== CONT TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5780=== RUN TestExternalSignerErrors/wrong_signature_name5781=== RUN TestSignErrors/nil_key5782=== PAUSE TestSignErrors/nil_key5783=== PAUSE TestGenerateFingerprint/unsorted_references_get_sorted5784=== CONT TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5785=== CONT TestExternalSignerRejectsInvalidPublicKey/no-colon5786--- PASS: TestParseSigningKey (0.00s)5787 --- PASS: TestParseSigningKey/wrong_length (0.00s)5788 --- PASS: TestParseSigningKey/no_colon (0.00s)5789 --- PASS: TestParseSigningKey/valid_32-byte_key_with_different_name (0.00s)5790 --- PASS: TestParseSigningKey/empty_name (0.00s)5791 --- PASS: TestParseSigningKey/valid_32-byte_key (0.00s)5792 --- PASS: TestParseSigningKey/invalid_base64 (0.00s)5793=== PAUSE TestExternalSignerErrors/wrong_signature_name5794=== RUN TestGenerateFingerprint/invalid_nar_hash_prefix5795=== PAUSE TestGenerateFingerprint/invalid_nar_hash_prefix5796=== RUN TestGenerateFingerprint/invalid_nar_hash_length5797=== PAUSE TestGenerateFingerprint/invalid_nar_hash_length5798=== RUN TestGenerateFingerprint/invalid_store_path_prefix5799=== PAUSE TestGenerateFingerprint/invalid_store_path_prefix5800=== RUN TestGenerateFingerprint/invalid_reference_prefix5801=== PAUSE TestGenerateFingerprint/invalid_reference_prefix5802=== CONT TestGenerateFingerprint/basic_with_references5803=== CONT TestGenerateFingerprint/invalid_nar_hash_length5804--- PASS: TestSignBatch (0.00s)5805=== CONT TestGenerateFingerprint/unsorted_references_get_sorted5806=== RUN TestSignErrors/nil_narinfo5807=== RUN TestExternalSignerErrors/program_error5808=== CONT TestGenerateFingerprint/invalid_nar_hash_prefix5809=== CONT TestGenerateFingerprint/invalid_reference_prefix5810=== CONT TestGenerateFingerprint/no_references5811=== CONT TestGenerateFingerprint/invalid_store_path_prefix5812--- PASS: TestExternalSignerRejectsInvalidPublicKey (0.00s)5813 --- PASS: TestExternalSignerRejectsInvalidPublicKey/#00 (0.00s)5814 --- PASS: TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= (0.00s)5815 --- PASS: TestExternalSignerRejectsInvalidPublicKey/test-key:invalid! (0.00s)5816 --- PASS: TestExternalSignerRejectsInvalidPublicKey/test-key:YQ== (0.00s)5817 --- PASS: TestExternalSignerRejectsInvalidPublicKey/no-colon (0.00s)5818=== PAUSE TestSignErrors/nil_narinfo5819=== RUN TestSignErrors/invalid_narinfo5820=== PAUSE TestSignErrors/invalid_narinfo5821=== RUN TestSignErrors/canceled5822=== PAUSE TestSignErrors/canceled5823=== CONT TestSignErrors/nil_key5824=== PAUSE TestExternalSignerErrors/program_error5825=== RUN TestExternalSignerErrors/exit_on_request5826=== PAUSE TestExternalSignerErrors/exit_on_request5827=== RUN TestExternalSignerErrors/invalid_JSON5828=== PAUSE TestExternalSignerErrors/invalid_JSON5829=== RUN TestExternalSignerErrors/wrong_shape5830=== PAUSE TestExternalSignerErrors/wrong_shape5831=== RUN TestExternalSignerErrors/wrong_id5832=== PAUSE TestExternalSignerErrors/wrong_id5833=== RUN TestExternalSignerErrors/missing_signatures5834--- PASS: TestGenerateFingerprint (0.00s)5835 --- PASS: TestGenerateFingerprint/basic_with_references (0.00s)5836 --- PASS: TestGenerateFingerprint/invalid_nar_hash_length (0.00s)5837 --- PASS: TestGenerateFingerprint/unsorted_references_get_sorted (0.00s)5838 --- PASS: TestGenerateFingerprint/invalid_nar_hash_prefix (0.00s)5839 --- PASS: TestGenerateFingerprint/invalid_reference_prefix (0.00s)5840 --- PASS: TestGenerateFingerprint/no_references (0.00s)5841 --- PASS: TestGenerateFingerprint/invalid_store_path_prefix (0.00s)5842=== CONT TestSignErrors/invalid_narinfo5843=== CONT TestSignErrors/canceled5844=== CONT TestSignErrors/nil_narinfo5845--- PASS: TestSignErrors (0.00s)5846 --- PASS: TestSignErrors/nil_key (0.00s)5847 --- PASS: TestSignErrors/invalid_narinfo (0.00s)5848 --- PASS: TestSignErrors/canceled (0.00s)5849 --- PASS: TestSignErrors/nil_narinfo (0.00s)5850=== PAUSE TestExternalSignerErrors/missing_signatures5851=== RUN TestExternalSignerErrors/null_signatures5852=== PAUSE TestExternalSignerErrors/null_signatures5853=== RUN TestExternalSignerErrors/too_few_signatures5854=== PAUSE TestExternalSignerErrors/too_few_signatures5855=== RUN TestExternalSignerErrors/wrong_signature_length5856=== PAUSE TestExternalSignerErrors/wrong_signature_length5857=== RUN TestExternalSignerErrors/wrong_signatures_shape5858=== PAUSE TestExternalSignerErrors/wrong_signatures_shape5859=== RUN TestExternalSignerErrors/invalid_signature_base645860=== PAUSE TestExternalSignerErrors/invalid_signature_base645861=== RUN TestExternalSignerErrors/null5862=== PAUSE TestExternalSignerErrors/null5863=== CONT TestExternalSignerErrors/exit_at_startup5864=== CONT TestExternalSignerErrors/wrong_id5865=== CONT TestExternalSignerErrors/wrong_signature_length5866=== CONT TestExternalSignerErrors/program_error5867=== CONT TestExternalSignerErrors/wrong_shape5868=== CONT TestExternalSignerErrors/invalid_JSON5869=== CONT TestExternalSignerErrors/exit_on_request5870=== CONT TestExternalSignerErrors/null5871=== CONT TestExternalSignerErrors/missing_signature_name5872=== CONT TestExternalSignerErrors/wrong_signature_name5873=== CONT TestExternalSignerErrors/null_signatures5874=== CONT TestExternalSignerErrors/too_few_signatures5875=== CONT TestExternalSignerErrors/unrequested_key5876=== CONT TestExternalSignerErrors/missing_signatures5877=== CONT TestExternalSignerErrors/wrong_signatures_shape5878=== CONT TestExternalSignerErrors/invalid_signature_base645879--- PASS: TestExternalSignerClose (0.01s)5880--- PASS: TestExternalSignerProgramErrorKeepsProcess (0.01s)5881--- PASS: TestExternalSignerWrongPublicKey (0.01s)5882--- PASS: TestExternalSignerRestartsAfterCrash (0.01s)5883--- PASS: TestExternalSigner (0.01s)5884--- PASS: TestExternalSignerErrors (0.00s)5885 --- PASS: TestExternalSignerErrors/exit_at_startup (0.00s)5886 --- PASS: TestExternalSignerErrors/wrong_id (0.01s)5887 --- PASS: TestExternalSignerErrors/wrong_signature_length (0.01s)5888 --- PASS: TestExternalSignerErrors/program_error (0.01s)5889 --- PASS: TestExternalSignerErrors/wrong_shape (0.01s)5890 --- PASS: TestExternalSignerErrors/missing_signatures (0.00s)5891 --- PASS: TestExternalSignerErrors/invalid_JSON (0.01s)5892 --- PASS: TestExternalSignerErrors/wrong_signature_name (0.01s)5893 --- PASS: TestExternalSignerErrors/unrequested_key (0.01s)5894 --- PASS: TestExternalSignerErrors/null_signatures (0.01s)5895 --- PASS: TestExternalSignerErrors/null (0.01s)5896 --- PASS: TestExternalSignerErrors/missing_signature_name (0.01s)5897 --- PASS: TestExternalSignerErrors/exit_on_request (0.01s)5898 --- PASS: TestExternalSignerErrors/too_few_signatures (0.01s)5899 --- PASS: TestExternalSignerErrors/invalid_signature_base64 (0.01s)5900 --- PASS: TestExternalSignerErrors/wrong_signatures_shape (0.01s)5901--- PASS: TestExternalSignerCancellation (0.02s)5902PASS5903Running OpenBao signer tests...5904=== RUN TestSign5905=== PAUSE TestSign5906=== RUN TestSignErrors5907=== PAUSE TestSignErrors5908=== RUN TestServe5909=== PAUSE TestServe5910=== RUN TestServeRejectsGarbage5911=== PAUSE TestServeRejectsGarbage5912=== CONT TestSign5913=== CONT TestServeRejectsGarbage5914--- PASS: TestServeRejectsGarbage (0.00s)5915=== CONT TestServe5916--- PASS: TestServe (0.00s)5917=== CONT TestSignErrors5918=== RUN TestSignErrors/http_error5919=== PAUSE TestSignErrors/http_error5920=== RUN TestSignErrors/invalid_json5921=== PAUSE TestSignErrors/invalid_json5922=== RUN TestSignErrors/no_prefix5923=== PAUSE TestSignErrors/no_prefix5924=== RUN TestSignErrors/invalid_base645925=== PAUSE TestSignErrors/invalid_base645926=== CONT TestSignErrors/http_error5927=== CONT TestSignErrors/invalid_base645928=== CONT TestSignErrors/no_prefix5929=== CONT TestSignErrors/invalid_json5930--- PASS: TestSign (1.17s)5931--- PASS: TestSignErrors (0.00s)5932 --- PASS: TestSignErrors/invalid_base64 (1.16s)5933 --- PASS: TestSignErrors/http_error (1.17s)5934 --- PASS: TestSignErrors/no_prefix (0.54s)5935 --- PASS: TestSignErrors/invalid_json (0.54s)5936PASS