niks3-go-unit-tests
checks.aarch64-linux.go-unit-tests
· build #489
· raw
1tribuchet: building on eliza2Running client tests...3=== RUN TestDoServerRequestAttachesToken4=== PAUSE TestDoServerRequestAttachesToken5=== RUN TestUploadBuildLog_FileBodyReplayedOnRetry6=== PAUSE TestUploadBuildLog_FileBodyReplayedOnRetry7=== RUN TestRegisterUploadedObjectReusesConnections8=== PAUSE TestRegisterUploadedObjectReusesConnections9=== RUN TestRunGarbageCollection_FinishedOnAnotherReplica10=== PAUSE TestRunGarbageCollection_FinishedOnAnotherReplica11=== RUN TestRunGarbageCollection_NotFoundAfterLocalRun12=== PAUSE TestRunGarbageCollection_NotFoundAfterLocalRun13=== RUN TestCaseHackSuffix14=== PAUSE TestCaseHackSuffix15=== RUN TestFilterOversizedClosures16=== PAUSE TestFilterOversizedClosures17=== RUN TestUploadMultipart_PartsInParallel18=== PAUSE TestUploadMultipart_PartsInParallel19=== RUN TestUploadMultipart_ReturnsAfterDroppedBody20=== PAUSE TestUploadMultipart_ReturnsAfterDroppedBody21=== RUN TestPartSizeForNAR22=== PAUSE TestPartSizeForNAR23=== RUN TestDumpPathCaseHackMatchesNix24--- PASS: TestDumpPathCaseHackMatchesNix (0.57s)25=== RUN TestDumpPathCaseHackCollision26--- PASS: TestDumpPathCaseHackCollision (0.00s)27=== RUN TestDumpPathMatchesNix28=== PAUSE TestDumpPathMatchesNix29=== RUN TestDumpPathSingleFile30=== PAUSE TestDumpPathSingleFile31=== RUN TestDumpPathWriterError32=== PAUSE TestDumpPathWriterError33=== RUN TestEncodeNixBase3234=== PAUSE TestEncodeNixBase3235=== RUN TestEncodeNixBase32WithRealHash36=== PAUSE TestEncodeNixBase32WithRealHash37=== RUN TestConvertHashToNix3238=== PAUSE TestConvertHashToNix3239=== RUN TestGetStorePathHash40=== PAUSE TestGetStorePathHash41=== RUN TestPathInfoHashCompatibility42=== PAUSE TestPathInfoHashCompatibility43=== RUN TestParsePathInfoJSON44=== PAUSE TestParsePathInfoJSON45=== RUN TestParsePathInfoJSONMultiplePaths46=== PAUSE TestParsePathInfoJSONMultiplePaths47=== RUN TestPathInfoCACompatibility48=== PAUSE TestPathInfoCACompatibility49=== RUN TestRateLimiterFeedback50=== PAUSE TestRateLimiterFeedback51=== RUN TestRateLimiterFeedback_400DoesNotCountAsSuccess52=== PAUSE TestRateLimiterFeedback_400DoesNotCountAsSuccess53=== RUN TestRegisterUploadedObject_BoundedAgainstSilentServer54=== PAUSE TestRegisterUploadedObject_BoundedAgainstSilentServer55=== RUN TestRegisterUploadedObject_OutlivesPush56=== PAUSE TestRegisterUploadedObject_OutlivesPush57=== RUN TestResolveStorePath58=== PAUSE TestResolveStorePath59=== RUN TestDoWithRetry_BodyReplayedViaGetBody60=== PAUSE TestDoWithRetry_BodyReplayedViaGetBody61=== RUN TestRetryTimeoutOutlastsMaxRetries62=== PAUSE TestRetryTimeoutOutlastsMaxRetries63=== RUN TestDoWithRetry_FinalResponseBodyReadable64=== PAUSE TestDoWithRetry_FinalResponseBodyReadable65=== RUN TestShellSplit66=== PAUSE TestShellSplit67=== RUN TestShellSplitErrors68=== PAUSE TestShellSplitErrors69=== RUN TestStreamPushReportsEveryPath70=== PAUSE TestStreamPushReportsEveryPath71=== RUN TestStreamPushBatchesUnderLoad72=== PAUSE TestStreamPushBatchesUnderLoad73=== RUN TestStreamPushIsolatesFailures74=== PAUSE TestStreamPushIsolatesFailures75=== RUN TestStreamPushGivesUpOnDeadServer76=== PAUSE TestStreamPushGivesUpOnDeadServer77=== RUN TestStreamPushRequestLine78=== PAUSE TestStreamPushRequestLine79=== RUN TestStreamPushReportsSignatures80=== PAUSE TestStreamPushReportsSignatures81=== RUN TestStreamPushReportsSkippedPaths82=== PAUSE TestStreamPushReportsSkippedPaths83=== RUN TestClientSignaturesByStorePath84=== PAUSE TestClientSignaturesByStorePath85=== RUN TestStreamPushStopsWhenCanceled86=== PAUSE TestStreamPushStopsWhenCanceled87=== RUN TestSetClientTLS88=== PAUSE TestSetClientTLS89=== RUN TestSetClientTLSDoesNotMutateDefaultTransport90=== PAUSE TestSetClientTLSDoesNotMutateDefaultTransport91=== RUN TestSetClientTLSErrors92=== PAUSE TestSetClientTLSErrors93=== RUN TestStaticToken94=== PAUSE TestStaticToken95=== RUN TestFileTokenReadsAndCaches96=== PAUSE TestFileTokenReadsAndCaches97=== RUN TestFileTokenMissing98=== PAUSE TestFileTokenMissing99=== RUN TestFileTokenEmpty100=== PAUSE TestFileTokenEmpty101=== RUN TestScriptTokenNoExpiryRerunsEveryCall102=== PAUSE TestScriptTokenNoExpiryRerunsEveryCall103=== RUN TestScriptTokenCachesUntilRefresh104=== PAUSE TestScriptTokenCachesUntilRefresh105=== RUN TestScriptTokenEmptyToken106=== PAUSE TestScriptTokenEmptyToken107=== RUN TestScriptTokenBadJSON108=== PAUSE TestScriptTokenBadJSON109=== RUN TestScriptTokenScriptFails110=== PAUSE TestScriptTokenScriptFails111=== RUN TestScriptTokenEmptyCommand112=== PAUSE TestScriptTokenEmptyCommand113=== CONT TestScriptTokenEmptyCommand114=== CONT TestDoServerRequestAttachesToken115=== CONT TestResolveStorePath116=== CONT TestRegisterUploadedObject_OutlivesPush117=== CONT TestRegisterUploadedObject_BoundedAgainstSilentServer118=== CONT TestRateLimiterFeedback_400DoesNotCountAsSuccess119=== CONT TestRateLimiterFeedback120=== CONT TestPathInfoCACompatibility121=== RUN TestRateLimiterFeedback/429_enables_limiter122=== RUN TestPathInfoCACompatibility/null_ca_field123=== PAUSE TestRateLimiterFeedback/429_enables_limiter124=== RUN TestRateLimiterFeedback/503_enables_limiter125=== CONT TestParsePathInfoJSON126=== CONT TestPathInfoHashCompatibility127=== CONT TestGetStorePathHash128=== CONT TestConvertHashToNix32129=== CONT TestEncodeNixBase32WithRealHash130=== CONT TestEncodeNixBase32131=== CONT TestDumpPathWriterError132=== CONT TestDumpPathSingleFile133=== CONT TestDumpPathMatchesNix134=== CONT TestPartSizeForNAR135=== CONT TestUploadMultipart_ReturnsAfterDroppedBody136=== CONT TestUploadMultipart_PartsInParallel137=== CONT TestFilterOversizedClosures138=== CONT TestDoWithRetry_BodyReplayedViaGetBody139=== PAUSE TestRateLimiterFeedback/503_enables_limiter140=== RUN TestRateLimiterFeedback/200_does_not_enable_limiter141=== RUN TestEncodeNixBase32/test_string_hash142=== RUN TestPartSizeForNAR/zero_stays_at_minimum143=== PAUSE TestPartSizeForNAR/zero_stays_at_minimum144=== RUN TestPartSizeForNAR/small_stays_at_minimum145=== PAUSE TestPartSizeForNAR/small_stays_at_minimum146=== RUN TestPartSizeForNAR/80_GiB_fits_at_minimum147=== PAUSE TestPartSizeForNAR/80_GiB_fits_at_minimum148=== RUN TestPartSizeForNAR/115_GiB_needs_larger_parts149=== PAUSE TestPartSizeForNAR/115_GiB_needs_larger_parts150=== RUN TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)151=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)152=== RUN TestPathInfoHashCompatibility/old_string_format_with_colon153=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_colon154=== RUN TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI155=== PAUSE TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI156=== RUN TestPathInfoHashCompatibility/new_structured_format_with_sha512157=== PAUSE TestPathInfoHashCompatibility/new_structured_format_with_sha512158=== RUN TestPartSizeForNAR/1_TiB159=== PAUSE TestPartSizeForNAR/1_TiB160=== RUN TestPartSizeForNAR/5_TiB_S3_max_object161=== PAUSE TestPartSizeForNAR/5_TiB_S3_max_object162=== RUN TestPartSizeForNAR/capped_at_5_GiB163=== PAUSE TestPartSizeForNAR/capped_at_5_GiB164=== CONT TestRunGarbageCollection_FinishedOnAnotherReplica165=== RUN TestFilterOversizedClosures/no_limit_keeps_everything166=== PAUSE TestFilterOversizedClosures/no_limit_keeps_everything167=== RUN TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped168=== PAUSE TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped169=== RUN TestFilterOversizedClosures/all_closures_skipped170=== PAUSE TestFilterOversizedClosures/all_closures_skipped171=== CONT TestRunGarbageCollection_NotFoundAfterLocalRun172=== CONT TestUploadBuildLog_FileBodyReplayedOnRetry173=== RUN TestGetStorePathHash/valid_store_path174=== PAUSE TestGetStorePathHash/valid_store_path175=== RUN TestGetStorePathHash/basename_without_hyphen_should_error176=== PAUSE TestGetStorePathHash/basename_without_hyphen_should_error177=== RUN TestGetStorePathHash/hash_with_invalid_characters_should_error178=== PAUSE TestGetStorePathHash/hash_with_invalid_characters_should_error179=== RUN TestGetStorePathHash/hash_with_wrong_length_should_error180=== PAUSE TestGetStorePathHash/hash_with_wrong_length_should_error181=== PAUSE TestRateLimiterFeedback/200_does_not_enable_limiter182=== RUN TestRateLimiterFeedback/400_does_not_enable_limiter183=== PAUSE TestRateLimiterFeedback/400_does_not_enable_limiter184=== PAUSE TestEncodeNixBase32/test_string_hash185=== CONT TestStreamPushStopsWhenCanceled186=== CONT TestParsePathInfoJSONMultiplePaths187=== PAUSE TestPathInfoCACompatibility/null_ca_field188=== CONT TestCaseHackSuffix189=== RUN TestConvertHashToNix32/SRI_format_to_Nix32190=== CONT TestRegisterUploadedObjectReusesConnections191--- PASS: TestScriptTokenEmptyCommand (0.00s)192=== RUN TestParsePathInfoJSON/Nix_format193=== RUN TestPathInfoCACompatibility/old_string_format_-_text194=== PAUSE TestPathInfoCACompatibility/old_string_format_-_text195=== RUN TestPathInfoCACompatibility/old_string_format_-_fixed_recursive196=== PAUSE TestPathInfoCACompatibility/old_string_format_-_fixed_recursive197=== RUN TestPathInfoCACompatibility/new_structured_format_-_text198=== PAUSE TestParsePathInfoJSON/Nix_format199=== CONT TestScriptTokenScriptFails2002026/11/11 23:28:18 WARN Rate limiter enabled after throttle name=server-test rate=5201=== CONT TestScriptTokenBadJSON202=== RUN TestEncodeNixBase32/empty_input203=== CONT TestFileTokenEmpty204=== CONT TestScriptTokenEmptyToken2052026/11/11 23:28:19 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:33091/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:33091/api/uploads/complete2062026/11/11 23:28:19 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"2072026/11/11 23:28:19 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:33091/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:33091/api/uploads/complete2082026/11/11 23:28:19 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"2092026/11/11 23:28:19 INFO Garbage collection started2102026/11/11 23:28:19 INFO Garbage collection progress phase=running_on_another_replica failed_uploads_deleted=0 old_closures_deleted=0 objects_marked=0 objects_deleted=0 objects_failed=02112026/11/11 23:28:19 WARN Garbage collection finished on another replica. Its statistics are not available here212=== PAUSE TestConvertHashToNix32/SRI_format_to_Nix32213=== RUN TestConvertHashToNix32/already_Nix32_format214=== PAUSE TestConvertHashToNix32/already_Nix32_format215=== RUN TestConvertHashToNix32/invalid_format216=== PAUSE TestConvertHashToNix32/invalid_format217--- PASS: TestEncodeNixBase32WithRealHash (0.00s)218--- PASS: TestResolveStorePath (0.00s)219--- PASS: TestStreamPushStopsWhenCanceled (0.00s)220--- PASS: TestScriptTokenScriptFails (1.42s)221--- PASS: TestCaseHackSuffix (1.93s)222--- PASS: TestDoServerRequestAttachesToken (3.59s)223=== CONT TestSetClientTLSErrors224--- PASS: TestFileTokenEmpty (0.00s)225=== RUN TestParsePathInfoJSON/Lix_format226=== RUN TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths227=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_text228=== CONT TestScriptTokenCachesUntilRefresh229=== CONT TestScriptTokenNoExpiryRerunsEveryCall230=== CONT TestStaticToken231=== CONT TestFileTokenMissing232=== CONT TestFileTokenReadsAndCaches233=== CONT TestSetClientTLSDoesNotMutateDefaultTransport234=== PAUSE TestEncodeNixBase32/empty_input235=== PAUSE TestParsePathInfoJSON/Lix_format236=== RUN TestParsePathInfoJSON/empty_input237=== PAUSE TestParsePathInfoJSON/empty_input238=== RUN TestParsePathInfoJSON/whitespace_only239=== PAUSE TestParsePathInfoJSON/whitespace_only240=== RUN TestParsePathInfoJSON/invalid_JSON241=== PAUSE TestParsePathInfoJSON/invalid_JSON242--- PASS: TestRunGarbageCollection_FinishedOnAnotherReplica (3.60s)243--- PASS: TestRegisterUploadedObject_OutlivesPush (3.60s)244--- PASS: TestDumpPathSingleFile (3.60s)245=== CONT TestStreamPushReportsSkippedPaths246=== PAUSE TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths247=== RUN TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths248=== PAUSE TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths249=== CONT TestStreamPushReportsSignatures250--- PASS: TestStreamPushReportsSkippedPaths (0.00s)251=== CONT TestStreamPushRequestLine252=== RUN TestPathInfoCACompatibility/new_structured_format_-_nar_method253=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_nar_method2542026/11/11 23:28:19 ERROR Upload failed error=boom count=1255=== CONT TestStreamPushGivesUpOnDeadServer256--- PASS: TestStreamPushReportsSignatures (0.00s)257=== CONT TestShellSplitErrors258--- PASS: TestShellSplitErrors (0.00s)259=== CONT TestStreamPushBatchesUnderLoad260=== RUN TestStreamPushBatchesUnderLoad/oneAtATime=false261=== PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=false262=== RUN TestStreamPushBatchesUnderLoad/oneAtATime=true263=== PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=true2642026/11/11 23:28:19 ERROR Upload failed error="connection refused" count=20265=== CONT TestStreamPushReportsEveryPath2662026/11/11 23:28:19 ERROR Server seems unavailable, giving up on batch untried=17267--- PASS: TestStreamPushReportsEveryPath (0.00s)268=== CONT TestDoWithRetry_FinalResponseBodyReadable2692026/11/11 23:28:19 ERROR Upload failed error=boom count=1270--- PASS: TestStreamPushGivesUpOnDeadServer (0.00s)271=== CONT TestShellSplit272--- PASS: TestShellSplit (0.00s)273=== CONT TestSetClientTLS2742026/11/11 23:28:19 WARN Rate limiter enabled after throttle name=server-test rate=52752026/11/11 23:28:19 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:33425276--- PASS: TestStaticToken (0.00s)277=== CONT TestRetryTimeoutOutlastsMaxRetries2782026/11/11 23:28:19 WARN Rate limiter backed off name=server-test rate=5279--- PASS: TestDoWithRetry_FinalResponseBodyReadable (0.00s)280=== CONT TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)281=== CONT TestPartSizeForNAR/zero_stays_at_minimum2822026/11/11 23:28:19 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:37639283=== CONT TestPathInfoHashCompatibility/new_structured_format_with_sha512284=== CONT TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI285=== CONT TestPathInfoHashCompatibility/old_string_format_with_colon286--- PASS: TestFileTokenMissing (0.00s)287--- PASS: TestPathInfoHashCompatibility (0.00s)288 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI) (0.00s)289 --- PASS: TestPathInfoHashCompatibility/new_structured_format_with_sha512 (0.00s)290 --- PASS: TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI (0.00s)291 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_colon (0.00s)292=== CONT TestPartSizeForNAR/capped_at_5_GiB293=== CONT TestPartSizeForNAR/1_TiB294=== CONT TestPartSizeForNAR/115_GiB_needs_larger_parts295=== CONT TestPartSizeForNAR/80_GiB_fits_at_minimum296=== CONT TestPartSizeForNAR/small_stays_at_minimum297=== CONT TestClientSignaturesByStorePath298=== CONT TestFilterOversizedClosures/no_limit_keeps_everything299=== CONT TestGetStorePathHash/hash_with_wrong_length_should_error300=== CONT TestGetStorePathHash/hash_with_invalid_characters_should_error301=== CONT TestGetStorePathHash/basename_without_hyphen_should_error302=== CONT TestRateLimiterFeedback/400_does_not_enable_limiter303=== CONT TestPartSizeForNAR/5_TiB_S3_max_object304=== CONT TestRateLimiterFeedback/200_does_not_enable_limiter305=== CONT TestRateLimiterFeedback/429_enables_limiter306=== CONT TestStreamPushIsolatesFailures307=== CONT TestRateLimiterFeedback/503_enables_limiter3082026/11/11 23:28:19 ERROR Upload failed error="bad path" count=3309--- PASS: TestFileTokenReadsAndCaches (0.00s)310--- PASS: TestClientSignaturesByStorePath (0.00s)311--- PASS: TestStreamPushIsolatesFailures (0.00s)312--- PASS: TestPartSizeForNAR (0.00s)313 --- PASS: TestPartSizeForNAR/zero_stays_at_minimum (0.00s)314 --- PASS: TestPartSizeForNAR/capped_at_5_GiB (0.00s)315 --- PASS: TestPartSizeForNAR/1_TiB (0.00s)316 --- PASS: TestPartSizeForNAR/115_GiB_needs_larger_parts (0.00s)317 --- PASS: TestPartSizeForNAR/80_GiB_fits_at_minimum (0.00s)318 --- PASS: TestPartSizeForNAR/small_stays_at_minimum (0.00s)319 --- PASS: TestPartSizeForNAR/5_TiB_S3_max_object (0.00s)320=== CONT TestFilterOversizedClosures/all_closures_skipped3212026/11/11 23:28:19 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper nar_size=100 max_nar_size=50322=== CONT TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped3232026/11/11 23:28:19 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb-vm-image nar_size=5000 max_nar_size=2000324--- PASS: TestFilterOversizedClosures (0.00s)325 --- PASS: TestFilterOversizedClosures/no_limit_keeps_everything (0.00s)326 --- PASS: TestFilterOversizedClosures/all_closures_skipped (0.00s)327 --- PASS: TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped (0.00s)328=== CONT TestGetStorePathHash/valid_store_path3292026/11/11 23:28:19 WARN Rate limiter enabled after throttle name=server-test rate=53302026/11/11 23:28:19 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=429 url=http://127.0.0.1:393233312026/11/11 23:28:19 WARN Rate limiter enabled after throttle name=server-test rate=53322026/11/11 23:28:19 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:378993332026/11/11 23:28:19 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:376393342026/11/11 23:28:19 WARN Rate limiter backed off name=server-test rate=5335--- PASS: TestGetStorePathHash (0.32s)336 --- PASS: TestGetStorePathHash/hash_with_wrong_length_should_error (0.00s)337 --- PASS: TestGetStorePathHash/hash_with_invalid_characters_should_error (0.00s)338 --- PASS: TestGetStorePathHash/basename_without_hyphen_should_error (0.00s)339 --- PASS: TestGetStorePathHash/valid_store_path (0.00s)340--- PASS: TestScriptTokenBadJSON (0.01s)341=== CONT TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths3422026/11/11 23:28:19 WARN Rate limiter backed off name=server-test rate=5343=== CONT TestConvertHashToNix32/SRI_format_to_Nix32344=== CONT TestParsePathInfoJSON/invalid_JSON3452026/11/11 23:28:19 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:33091/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:33091/api/uploads/complete346=== CONT TestParsePathInfoJSON/whitespace_only347=== CONT TestParsePathInfoJSON/empty_input348=== CONT TestParsePathInfoJSON/Lix_format3492026/11/11 23:28:19 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"350=== CONT TestConvertHashToNix32/invalid_format351=== CONT TestPathInfoCACompatibility/null_ca_field352=== CONT TestPathInfoCACompatibility/new_structured_format_-_text3532026/11/11 23:28:19 INFO Garbage collection started354=== CONT TestPathInfoCACompatibility/old_string_format_-_fixed_recursive3552026/11/11 23:28:19 WARN Rate limiter enabled after throttle name=server-test rate=5356=== CONT TestPathInfoCACompatibility/old_string_format_-_text3572026/11/11 23:28:19 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:35459358=== CONT TestStreamPushBatchesUnderLoad/oneAtATime=false359=== CONT TestStreamPushBatchesUnderLoad/oneAtATime=true360=== CONT TestEncodeNixBase32/test_string_hash361=== CONT TestEncodeNixBase32/empty_input362=== CONT TestPathInfoCACompatibility/new_structured_format_-_nar_method3632026/11/11 23:28:19 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:37639364--- PASS: TestUploadMultipart_ReturnsAfterDroppedBody (3.61s)365=== CONT TestParsePathInfoJSON/Nix_format366--- PASS: TestRegisterUploadedObject_BoundedAgainstSilentServer (3.62s)367--- PASS: TestRateLimiterFeedback (0.32s)368 --- PASS: TestRateLimiterFeedback/200_does_not_enable_limiter (0.00s)369 --- PASS: TestRateLimiterFeedback/400_does_not_enable_limiter (0.01s)370 --- PASS: TestRateLimiterFeedback/503_enables_limiter (0.01s)371 --- PASS: TestRateLimiterFeedback/429_enables_limiter (0.01s)372=== CONT TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths373--- PASS: TestEncodeNixBase32 (3.60s)374 --- PASS: TestEncodeNixBase32/test_string_hash (0.00s)375 --- PASS: TestEncodeNixBase32/empty_input (0.00s)376--- PASS: TestPathInfoCACompatibility (3.60s)377 --- PASS: TestPathInfoCACompatibility/null_ca_field (0.00s)378 --- PASS: TestPathInfoCACompatibility/old_string_format_-_fixed_recursive (0.00s)379 --- PASS: TestPathInfoCACompatibility/old_string_format_-_text (0.00s)380 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_text (0.00s)381 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_nar_method (0.00s)3822026/11/11 23:28:19 WARN Rate limiter backed off name=server-test rate=53832026/11/11 23:28:19 WARN Request returned retryable status, retrying attempt=2 backoff=0s status=503 url=http://127.0.0.1:35459384--- PASS: TestDoWithRetry_BodyReplayedViaGetBody (3.62s)385--- PASS: TestStreamPushBatchesUnderLoad (0.00s)386 --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=false (0.00s)387 --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=true (0.00s)388=== CONT TestConvertHashToNix32/already_Nix32_format389--- PASS: TestConvertHashToNix32 (3.60s)390 --- PASS: TestConvertHashToNix32/SRI_format_to_Nix32 (0.00s)391 --- PASS: TestConvertHashToNix32/invalid_format (0.00s)392 --- PASS: TestConvertHashToNix32/already_Nix32_format (0.00s)393--- PASS: TestScriptTokenEmptyToken (0.02s)394--- PASS: TestParsePathInfoJSON (3.60s)395 --- PASS: TestParsePathInfoJSON/invalid_JSON (0.00s)396 --- PASS: TestParsePathInfoJSON/whitespace_only (0.00s)397 --- PASS: TestParsePathInfoJSON/empty_input (0.00s)398 --- PASS: TestParsePathInfoJSON/Lix_format (0.00s)399 --- PASS: TestParsePathInfoJSON/Nix_format (0.00s)400--- PASS: TestParsePathInfoJSONMultiplePaths (1.48s)401 --- PASS: TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths (0.00s)402 --- PASS: TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths (0.01s)4032026/11/11 23:28:19 WARN Rate limiter enabled after throttle name=s3-test rate=54042026/11/11 23:28:19 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:44993405--- PASS: TestUploadBuildLog_FileBodyReplayedOnRetry (6.16s)406--- PASS: TestScriptTokenCachesUntilRefresh (2.56s)4072026/11/11 23:28:22 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:37639408--- PASS: TestRunGarbageCollection_NotFoundAfterLocalRun (6.16s)409--- PASS: TestScriptTokenNoExpiryRerunsEveryCall (2.56s)410--- PASS: TestRateLimiterFeedback_400DoesNotCountAsSuccess (6.17s)4112026/11/11 23:28:22 WARN Request returned retryable status, retrying attempt=3 backoff=1ms status=502 url=http://127.0.0.1:37639412=== RUN TestSetClientTLSErrors/missing_cert_file413=== PAUSE TestSetClientTLSErrors/missing_cert_file414=== RUN TestSetClientTLSErrors/missing_key_file415--- PASS: TestSetClientTLSDoesNotMutateDefaultTransport (2.57s)416=== PAUSE TestSetClientTLSErrors/missing_key_file417=== RUN TestSetClientTLSErrors/missing_ca_file418=== PAUSE TestSetClientTLSErrors/missing_ca_file419=== RUN TestSetClientTLSErrors/invalid_ca_file420=== PAUSE TestSetClientTLSErrors/invalid_ca_file421=== CONT TestSetClientTLSErrors/missing_cert_file422=== CONT TestSetClientTLSErrors/missing_ca_file4232026/11/11 23:28:22 WARN Request returned retryable status, retrying attempt=4 backoff=1ms status=502 url=http://127.0.0.1:37639424=== CONT TestSetClientTLSErrors/invalid_ca_file425=== CONT TestSetClientTLSErrors/missing_key_file426--- PASS: TestRegisterUploadedObjectReusesConnections (4.01s)4272026/11/11 23:28:22 WARN Request returned retryable status, retrying attempt=5 backoff=1ms status=502 url=http://127.0.0.1:37639428--- PASS: TestSetClientTLSErrors (2.57s)429 --- PASS: TestSetClientTLSErrors/missing_cert_file (0.00s)430 --- PASS: TestSetClientTLSErrors/missing_key_file (0.00s)431 --- PASS: TestSetClientTLSErrors/missing_ca_file (0.00s)432 --- PASS: TestSetClientTLSErrors/invalid_ca_file (0.00s)4332026/11/11 23:28:22 WARN Request returned retryable status, retrying attempt=6 backoff=1ms status=502 url=http://127.0.0.1:37639434--- PASS: TestStreamPushRequestLine (2.57s)435=== RUN TestSetClientTLS/rejects_connection_without_client_cert436=== PAUSE TestSetClientTLS/rejects_connection_without_client_cert437=== RUN TestSetClientTLS/succeeds_with_client_cert_and_CA438=== PAUSE TestSetClientTLS/succeeds_with_client_cert_and_CA439=== RUN TestSetClientTLS/preserves_debug_logging_transport440=== PAUSE TestSetClientTLS/preserves_debug_logging_transport441=== CONT TestSetClientTLS/rejects_connection_without_client_cert442=== CONT TestSetClientTLS/preserves_debug_logging_transport443=== CONT TestSetClientTLS/succeeds_with_client_cert_and_CA4442026/11/11 23:28:22 WARN Request returned retryable status, retrying attempt=7 backoff=1ms status=502 url=http://127.0.0.1:376394452026/11/11 23:28:22 WARN Request returned retryable status, retrying attempt=8 backoff=1ms status=502 url=http://127.0.0.1:376394462026/11/11 23:28:22 WARN Request returned retryable status, retrying attempt=9 backoff=1ms status=502 url=http://127.0.0.1:376394472026/11/11 23:28:22 WARN Request returned retryable status, retrying attempt=10 backoff=1ms status=502 url=http://127.0.0.1:376394482026/11/11 23:28:22 WARN Request returned retryable status, retrying attempt=11 backoff=1ms status=502 url=http://127.0.0.1:37639449--- PASS: TestDumpPathWriterError (6.18s)4502026/11/11 23:28:22 WARN Request returned retryable status, retrying attempt=12 backoff=1ms status=502 url=http://127.0.0.1:376394512026/11/11 23:28:22 WARN Request returned retryable status, retrying attempt=13 backoff=1ms status=502 url=http://127.0.0.1:376394522026/11/11 23:28:22 WARN Request returned retryable status, retrying attempt=14 backoff=1ms status=502 url=http://127.0.0.1:376394532026/11/11 23:28:22 WARN Request returned retryable status, retrying attempt=15 backoff=1ms status=502 url=http://127.0.0.1:376394542026/11/11 23:28:22 http: TLS handshake error from 127.0.0.1:55112: remote error: tls: bad certificate455--- PASS: TestSetClientTLS (2.57s)456 --- PASS: TestSetClientTLS/succeeds_with_client_cert_and_CA (0.01s)457 --- PASS: TestSetClientTLS/preserves_debug_logging_transport (0.01s)458 --- PASS: TestSetClientTLS/rejects_connection_without_client_cert (0.01s)4592026/11/11 23:28:22 WARN Request returned retryable status, retrying attempt=16 backoff=1ms status=502 url=http://127.0.0.1:376394602026/11/11 23:28:22 WARN Request returned retryable status, retrying attempt=17 backoff=1ms status=502 url=http://127.0.0.1:37639461--- PASS: TestRetryTimeoutOutlastsMaxRetries (2.58s)462--- PASS: TestUploadMultipart_PartsInParallel (6.73s)463--- PASS: TestDumpPathMatchesNix (7.01s)464PASS465Running server tests...466The files belonging to this database system will be owned by user "nixbld".467This user must also own the server process.468469The database cluster will be initialized with locale "C".470The default database encoding has accordingly been set to "SQL_ASCII".471The default text search configuration will be set to "english".472473Data page checksums are enabled.474475creating directory /build/postgres2263749791/data ... ok476creating subdirectories ... ok477selecting dynamic shared memory implementation ... posix478selecting default "max_connections" ... 100479selecting default "shared_buffers" ... 128MB480selecting default time zone ... UTC481creating configuration files ... ok482running bootstrap script ... ok483performing post-bootstrap initialization ... ok484syncing data to disk ... ok485486initdb: warning: enabling "trust" authentication for local connections487initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.488489Success. You can now start the database server using:490491 pg_ctl -D /build/postgres2263749791/data -l logfile start492493/build/postgres2263749791:5432 - no response494/build/postgres2263749791:5432 - no response495/build/postgres2263749791:5432 - no response496/build/postgres2263749791:5432 - no response497/build/postgres2263749791:5432 - no response498/build/postgres2263749791:5432 - no response499/build/postgres2263749791:5432 - no response500/build/postgres2263749791:5432 - no response501/build/postgres2263749791:5432 - no response5022026-11-11 23:28:38.105 UTC [131] LOG: starting PostgreSQL 18.6 on aarch64-unknown-linux-gnu, compiled by clang version 21.1.8, 64-bit5032026-11-11 23:28:38.105 UTC [131] LOG: listening on Unix socket "/build/postgres2263749791/.s.PGSQL.5432"5042026-11-11 23:28:38.112 UTC [146] LOG: database system was shut down at 2026-11-11 23:28:37 UTC5052026-11-11 23:28:38.118 UTC [131] LOG: database system is ready to accept connections506/build/postgres2263749791:5432 - accepting connections507{"timestamp":"2026-11-11T23:29:23.856657838Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"65cc3dd1-0d13-41bb-ae0d-bf253e64c9a5","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"GET","uri":"/health/ready","status_code":503,"suppressed_errors":0,"duration_ms":19,"result":"server_error","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":502,"threadName":"rustfs-worker","threadId":"ThreadId(200)"}508{"timestamp":"2026-11-11T23:29:23.974061738Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"f88ea05f-98d5-4c14-b744-4a3ebf79532f","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"GET","uri":"/health/ready","status_code":503,"suppressed_errors":0,"duration_ms":0,"result":"server_error","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":502,"threadName":"rustfs-worker","threadId":"ThreadId(198)"}509=== RUN TestAbortSucceeded510=== PAUSE TestAbortSucceeded511=== RUN TestService_AuthMiddleware512=== PAUSE TestService_AuthMiddleware513=== RUN TestService_AuthMiddleware_MTLSProxyHeader514=== PAUSE TestService_AuthMiddleware_MTLSProxyHeader515=== RUN TestService_AuthMiddleware_MTLSBoundSubjects516=== PAUSE TestService_AuthMiddleware_MTLSBoundSubjects517=== RUN TestService_ReadAuthMiddleware518=== PAUSE TestService_ReadAuthMiddleware519=== RUN TestService_AuthMiddleware_OIDC520=== PAUSE TestService_AuthMiddleware_OIDC521=== RUN TestService_RequireScope_OIDC522=== PAUSE TestService_RequireScope_OIDC523=== RUN TestService_ReadScope_PublicByDefault524=== PAUSE TestService_ReadScope_PublicByDefault525=== RUN TestCacheConfigHandler526=== PAUSE TestCacheConfigHandler527=== RUN TestCacheStatsHandler528=== PAUSE TestCacheStatsHandler529=== RUN TestClientCADerivations530=== PAUSE TestClientCADerivations531=== RUN TestClientErrorHandling532=== PAUSE TestClientErrorHandling533=== RUN TestClientIntegration534=== PAUSE TestClientIntegration535=== RUN TestClientMultipleUploads536=== PAUSE TestClientMultipleUploads537=== RUN TestClientWithDependencies538=== PAUSE TestClientWithDependencies539=== RUN TestClientSharedPathCommittedMidPush540=== PAUSE TestClientSharedPathCommittedMidPush541=== RUN TestPinProtectsFromGC542=== PAUSE TestPinProtectsFromGC543=== RUN TestClientPushesUseOnePush544=== PAUSE TestClientPushesUseOnePush545=== RUN TestClientFallsBackToClosures546=== PAUSE TestClientFallsBackToClosures547=== RUN TestDeleteClosuresRacingPins548=== PAUSE TestDeleteClosuresRacingPins549=== RUN TestConcurrentCommitsSharingObjectsDoNotDeadlock550=== PAUSE TestConcurrentCommitsSharingObjectsDoNotDeadlock551=== RUN TestResolveDBConnectionString552=== PAUSE TestResolveDBConnectionString553=== RUN TestConnectWaitsForAPeerMigration554=== PAUSE TestConnectWaitsForAPeerMigration555=== RUN TestLeadElectsOneAndHandsOver556=== PAUSE TestLeadElectsOneAndHandsOver557=== RUN TestLeadIncumbentWinsAfterRestart5582026/11/11 23:29:25 INFO lead: acquired remote=192.0.2.1:1234 generation=15592026/11/11 23:29:26 INFO lead: released remote=192.0.2.1:1234 generation=15602026/11/11 23:29:26 INFO lead: acquired remote=192.0.2.1:1234 generation=25612026/11/11 23:29:27 INFO lead: released remote=192.0.2.1:1234 generation=2562--- PASS: TestLeadIncumbentWinsAfterRestart (3.42s)563=== RUN TestLeadUnannouncedStandbyYieldsToIncumbent564=== PAUSE TestLeadUnannouncedStandbyYieldsToIncumbent565=== RUN TestLeadIncumbentAnnouncesAtOnce566=== PAUSE TestLeadIncumbentAnnouncesAtOnce567=== RUN TestLeadGenerationGrowsWithEveryGrant568=== PAUSE TestLeadGenerationGrowsWithEveryGrant569=== RUN TestLeadEndsOnShutdown570=== PAUSE TestLeadEndsOnShutdown571=== RUN TestLeadEndsWhenItsConnectionHangs5722026/11/11 23:29:28 INFO lead: acquired remote=192.0.2.1:1234 generation=15732026-11-11 23:29:29.889 UTC [398] FATAL: terminating connection due to administrator command5742026/11/11 23:29:30 INFO lead: acquired remote=192.0.2.1:1234 generation=25752026/11/11 23:29:31 WARN lead: lock connection lost error="timeout: context deadline exceeded"5762026/11/11 23:29:31 INFO lead: released remote=192.0.2.1:1234 generation=15772026/11/11 23:29:31 INFO lead: released remote=192.0.2.1:1234 generation=2578--- PASS: TestLeadEndsWhenItsConnectionHangs (3.97s)579=== RUN TestGCAdvisoryLockBlocksConcurrentRun580=== PAUSE TestGCAdvisoryLockBlocksConcurrentRun581=== RUN TestGCBugBareHashReferences582=== PAUSE TestGCBugBareHashReferences583=== RUN TestGCMetrics584=== PAUSE TestGCMetrics585=== RUN TestPushDedupSurvivesConcurrentGC586=== PAUSE TestPushDedupSurvivesConcurrentGC587=== RUN TestDeduplicatedObjectsRecordedAsPending588=== PAUSE TestDeduplicatedObjectsRecordedAsPending589=== RUN TestGCSweepSkipsPendingObjects590=== PAUSE TestGCSweepSkipsPendingObjects591=== RUN TestTombstonedObjectOfferedWithoutWaiting592=== PAUSE TestTombstonedObjectOfferedWithoutWaiting593=== RUN TestGCSweepDeliversEachKeyOnce594=== PAUSE TestGCSweepDeliversEachKeyOnce595=== RUN TestCreatePendingClosureVerifyS3FailureReleasesConnection596=== PAUSE TestCreatePendingClosureVerifyS3FailureReleasesConnection597=== RUN TestForceGCDuringPushOffersSweptObject598=== PAUSE TestForceGCDuringPushOffersSweptObject599=== RUN TestFailedS3DeleteKeepsTombstone600=== PAUSE TestFailedS3DeleteKeepsTombstone601=== RUN TestSweepSkipsRowsLockedByPush602=== PAUSE TestSweepSkipsRowsLockedByPush603=== RUN TestRegisterCompletedObjectNeedsPendingRow604=== PAUSE TestRegisterCompletedObjectNeedsPendingRow605=== RUN TestGetExistingObjectsLocksRows606=== PAUSE TestGetExistingObjectsLocksRows607=== RUN TestSweepRechecksPendingAfterLocking608=== PAUSE TestSweepRechecksPendingAfterLocking609=== RUN TestPushWaitsForSweepHoldingRow610=== PAUSE TestPushWaitsForSweepHoldingRow611=== RUN TestCommitRacingPendingCleanupKeepsObjects612=== PAUSE TestCommitRacingPendingCleanupKeepsObjects613=== RUN TestGCEndsOnShutdown614=== PAUSE TestGCEndsOnShutdown615=== RUN TestGCTaskStore_StartNew616=== PAUSE TestGCTaskStore_StartNew617=== RUN TestGCTaskStore_DeduplicateSameParams618=== PAUSE TestGCTaskStore_DeduplicateSameParams619=== RUN TestGCTaskStore_ConflictDifferentParams620=== PAUSE TestGCTaskStore_ConflictDifferentParams621=== RUN TestGCTaskStore_GetEmpty622=== PAUSE TestGCTaskStore_GetEmpty623=== RUN TestGCTaskStore_GetReturnsLatest624=== PAUSE TestGCTaskStore_GetReturnsLatest625=== RUN TestGCTaskStore_CompletedAllowsNewTask626=== PAUSE TestGCTaskStore_CompletedAllowsNewTask627=== RUN TestGCTaskStore_PhaseUpdates628=== PAUSE TestGCTaskStore_PhaseUpdates629=== RUN TestGCTaskStore_Fail630=== PAUSE TestGCTaskStore_Fail631=== RUN TestGracefulShutdownDrainsInflight632=== PAUSE TestGracefulShutdownDrainsInflight633=== RUN TestService_healthCheckHandler634=== PAUSE TestService_healthCheckHandler635=== RUN TestService_readinessHandler636=== PAUSE TestService_readinessHandler637=== RUN TestGenerateLandingPage638=== PAUSE TestGenerateLandingPage639=== RUN TestEnsureAbortUploadsRule640=== PAUSE TestEnsureAbortUploadsRule641=== RUN TestCacheConfigHandlerMaxNarSize642=== PAUSE TestCacheConfigHandlerMaxNarSize643=== RUN TestCreatePendingClosureRejectsOversizedNAR644=== PAUSE TestCreatePendingClosureRejectsOversizedNAR645=== RUN TestNARDeduplicationMetadataUploadBug646=== PAUSE TestNARDeduplicationMetadataUploadBug647=== RUN TestMetricsInventory648=== PAUSE TestMetricsInventory649=== RUN TestMetricsCommit650=== PAUSE TestMetricsCommit651=== RUN TestService_NativeMTLS652=== PAUSE TestService_NativeMTLS653=== RUN TestServerTLSConfig654=== PAUSE TestServerTLSConfig655=== RUN TestMultipartCleanup656=== PAUSE TestMultipartCleanup657=== RUN TestMultipartUploadAbortedWhenCancelledBeforeRecorded658=== PAUSE TestMultipartUploadAbortedWhenCancelledBeforeRecorded659=== RUN TestPendingCleanupUsesOneCutoff660=== PAUSE TestPendingCleanupUsesOneCutoff661=== RUN TestObjectStatsTrigger662=== PAUSE TestObjectStatsTrigger663=== RUN TestObjectStatsUpdatedOncePerCommit664=== PAUSE TestObjectStatsUpdatedOncePerCommit665=== RUN TestOrphanedObjectsGC666=== PAUSE TestOrphanedObjectsGC667=== RUN TestOrphanedObjectsGCStressTest668=== PAUSE TestOrphanedObjectsGCStressTest669=== RUN TestCreatePin_ReservedPins670=== PAUSE TestCreatePin_ReservedPins671=== RUN TestPresentReportsOnlyClosureRoots672=== PAUSE TestPresentReportsOnlyClosureRoots673=== RUN TestPresentRefreshesOnlyStaleClosures674=== PAUSE TestPresentRefreshesOnlyStaleClosures675=== RUN TestPresentNotReportedWhileGCDeletesClosure676=== PAUSE TestPresentNotReportedWhileGCDeletesClosure677=== RUN TestParseSingleRange678=== PAUSE TestParseSingleRange679=== RUN TestProxyHeadersOnlyTrustedOnSocket6802026/11/11 23:29:37 INFO Starting HTTP server address=127.0.0.1:343116812026/11/11 23:29:37 INFO Starting HTTP server address=proxy.sock6822026/11/11 23:29:37 WARN mTLS auth: subject not in bound subjects subject="CN=someone"6832026/11/11 23:29:37 INFO Shutdown signal received, draining in-flight requests timeout=10s684--- PASS: TestProxyHeadersOnlyTrustedOnSocket (6.27s)685=== RUN TestIsValidCachePath686=== PAUSE TestIsValidCachePath687=== RUN TestReadProxyNarinfo688=== PAUSE TestReadProxyNarinfo689=== RUN TestReadProxyNarinfoAlreadyDecompressed690=== PAUSE TestReadProxyNarinfoAlreadyDecompressed691=== RUN TestReadProxyContentNegotiation692=== PAUSE TestReadProxyContentNegotiation693=== RUN TestReadProxyNarStreaming694=== PAUSE TestReadProxyNarStreaming695=== RUN TestReadProxy404696=== PAUSE TestReadProxy404697=== RUN TestReadProxyInvalidPath698=== PAUSE TestReadProxyInvalidPath699=== RUN TestReadProxyHead700=== PAUSE TestReadProxyHead701=== RUN TestReadProxyOutlastsServerWriteTimeout702=== PAUSE TestReadProxyOutlastsServerWriteTimeout703=== RUN TestReadProxyConditionalGet704=== PAUSE TestReadProxyConditionalGet705=== RUN TestReadProxyRootRedirectsToIndexHTML706=== PAUSE TestReadProxyRootRedirectsToIndexHTML707=== RUN TestReadProxyDisabled708=== PAUSE TestReadProxyDisabled709=== RUN TestReadRedirectNar710=== PAUSE TestReadRedirectNar711=== RUN TestReadRedirectKeepsNarinfoProxied712=== PAUSE TestReadRedirectKeepsNarinfoProxied713=== RUN TestReadProxyRangeRequest714=== PAUSE TestReadProxyRangeRequest715=== RUN TestReadRedirectUsesPublicS3URL716=== PAUSE TestReadRedirectUsesPublicS3URL717=== RUN TestPush_OverlappingRootsStoreOneRowPerKey718=== PAUSE TestPush_OverlappingRootsStoreOneRowPerKey719=== RUN TestPush_CompleteCommitsEveryRoot720=== PAUSE TestPush_CompleteCommitsEveryRoot721=== RUN TestPush_RepeatedRootCommitsOnce722=== PAUSE TestPush_RepeatedRootCommitsOnce723=== RUN TestPush_CompleteWaitsForCleanupOfThePush724=== PAUSE TestPush_CompleteWaitsForCleanupOfThePush725=== RUN TestPush_RootsAreLockedInByteOrder726=== PAUSE TestPush_RootsAreLockedInByteOrder727=== RUN TestPush_SkippedKeySurvivesGCBeforeCommit728=== PAUSE TestPush_SkippedKeySurvivesGCBeforeCommit729=== RUN TestPush_RejectsBadRequests730=== PAUSE TestPush_RejectsBadRequests731=== RUN TestPush_SignsNarinfosOfItsPendingObjects732=== PAUSE TestPush_SignsNarinfosOfItsPendingObjects733=== RUN TestSharedNarBothUploadsComplete734=== PAUSE TestSharedNarBothUploadsComplete735=== RUN TestCompleteMultipartUpload_ErrorButObjectExists736=== PAUSE TestCompleteMultipartUpload_ErrorButObjectExists737=== RUN TestCompletedNarNotReofferedAcrossClosures738=== PAUSE TestCompletedNarNotReofferedAcrossClosures739=== RUN TestPresignedUploadRegisteredBeforeCommit740=== PAUSE TestPresignedUploadRegisteredBeforeCommit741=== RUN TestService_Rustfstest742=== PAUSE TestService_Rustfstest743=== RUN TestParseSize744=== PAUSE TestParseSize745=== RUN TestSkippedUploadsHandler746=== PAUSE TestSkippedUploadsHandler747=== RUN TestSystemdListenerNotActivated748--- PASS: TestSystemdListenerNotActivated (0.00s)749=== RUN TestWatchdogBeatsWhenHealthy750--- PASS: TestWatchdogBeatsWhenHealthy (0.02s)751=== RUN TestWatchdogSkipsWhenUnhealthy7522026/11/11 23:29:37 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7532026/11/11 23:29:37 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7542026/11/11 23:29:37 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7552026/11/11 23:29:37 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7562026/11/11 23:29:37 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7572026/11/11 23:29:37 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7582026/11/11 23:29:37 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7592026/11/11 23:29:37 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7602026/11/11 23:29:37 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7612026/11/11 23:29:37 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"762--- PASS: TestWatchdogSkipsWhenUnhealthy (0.20s)763=== RUN TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle764=== PAUSE TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle765=== RUN TestProxyWriteTimeout766=== PAUSE TestProxyWriteTimeout767=== RUN TestPendingClosureWriteTimeout768=== PAUSE TestPendingClosureWriteTimeout769=== RUN TestIsValidUploadKey770=== PAUSE TestIsValidUploadKey771=== RUN TestUploadHandlersRejectInvalidKeys772=== PAUSE TestUploadHandlersRejectInvalidKeys773=== RUN TestUploadHandlersRejectOversizedBody774=== PAUSE TestUploadHandlersRejectOversizedBody775=== RUN TestService_cleanupPendingClosuresHandler776=== PAUSE TestService_cleanupPendingClosuresHandler777=== RUN TestService_createPendingClosureHandler778=== PAUSE TestService_createPendingClosureHandler779=== RUN TestService_verifyS3Integrity780=== PAUSE TestService_verifyS3Integrity781=== RUN TestService_verifyS3ReuploadsStaleNarinfo782=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo783=== RUN TestCompleteMultipartUnregistered784=== PAUSE TestCompleteMultipartUnregistered785=== RUN TestCreatePendingClosure_SmallNARUsesSimplePUT786=== PAUSE TestCreatePendingClosure_SmallNARUsesSimplePUT787=== CONT TestIsValidUploadKey788=== CONT TestAbortSucceeded789=== CONT TestService_readinessHandler790--- PASS: TestAbortSucceeded (0.00s)791=== CONT TestReadRedirectKeepsNarinfoProxied792=== CONT TestService_verifyS3Integrity793=== CONT TestCreatePendingClosure_SmallNARUsesSimplePUT794=== CONT TestCompleteMultipartUnregistered795=== CONT TestService_verifyS3ReuploadsStaleNarinfo796=== RUN TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed797=== CONT TestService_cleanupPendingClosuresHandler798=== CONT TestService_createPendingClosureHandler799=== CONT TestReadProxyInvalidPath800=== CONT TestPendingClosureWriteTimeout801=== CONT TestProxyWriteTimeout802=== CONT TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle803=== CONT TestSkippedUploadsHandler804=== CONT TestParseSize805=== CONT TestService_Rustfstest806=== CONT TestPresignedUploadRegisteredBeforeCommit807=== CONT TestCompletedNarNotReofferedAcrossClosures808=== CONT TestCompleteMultipartUpload_ErrorButObjectExists809=== CONT TestSharedNarBothUploadsComplete810=== CONT TestPush_SignsNarinfosOfItsPendingObjects811=== CONT TestPush_RejectsBadRequests812=== CONT TestUploadHandlersRejectOversizedBody813=== CONT TestPush_SkippedKeySurvivesGCBeforeCommit814=== RUN TestIsValidUploadKey/narinfo815=== PAUSE TestIsValidUploadKey/narinfo816=== RUN TestIsValidUploadKey/nar_zst817=== PAUSE TestIsValidUploadKey/nar_zst818=== RUN TestIsValidUploadKey/nar_xz819=== PAUSE TestIsValidUploadKey/nar_xz820=== RUN TestIsValidUploadKey/nar_plain821=== PAUSE TestIsValidUploadKey/nar_plain822=== RUN TestIsValidUploadKey/listing823=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed824=== RUN TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed825=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed826=== RUN TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost827=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost828=== RUN TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed829=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed830=== RUN TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed831=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed832=== CONT TestReadRedirectNar833=== RUN TestPendingClosureWriteTimeout/empty834=== PAUSE TestPendingClosureWriteTimeout/empty835=== RUN TestPendingClosureWriteTimeout/400_objects836=== PAUSE TestPendingClosureWriteTimeout/400_objects837=== RUN TestPendingClosureWriteTimeout/670k_objects838=== PAUSE TestPendingClosureWriteTimeout/670k_objects839=== RUN TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline840=== PAUSE TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline841=== CONT TestReadProxyDisabled842=== PAUSE TestIsValidUploadKey/listing843=== RUN TestIsValidUploadKey/build_log844=== PAUSE TestIsValidUploadKey/build_log845=== RUN TestIsValidUploadKey/build_log_home-manager_file846=== PAUSE TestIsValidUploadKey/build_log_home-manager_file847=== RUN TestIsValidUploadKey/build_log_plus_in_name848=== PAUSE TestIsValidUploadKey/build_log_plus_in_name849=== RUN TestIsValidUploadKey/build_log_question_mark850=== PAUSE TestIsValidUploadKey/build_log_question_mark851=== RUN TestIsValidUploadKey/build_log_equals852=== PAUSE TestIsValidUploadKey/build_log_equals853=== RUN TestIsValidUploadKey/realisation854=== PAUSE TestIsValidUploadKey/realisation855=== RUN TestProxyWriteTimeout/narinfo856=== RUN TestIsValidUploadKey/realisation_plus_in_output857=== PAUSE TestProxyWriteTimeout/narinfo858=== RUN TestProxyWriteTimeout/1_GiB_nar859--- PASS: TestParseSize (0.00s)860=== PAUSE TestProxyWriteTimeout/1_GiB_nar861=== PAUSE TestIsValidUploadKey/realisation_plus_in_output862=== RUN TestIsValidUploadKey/nix-cache-info863=== RUN TestProxyWriteTimeout/10_GiB_nar864=== CONT TestReadProxyRootRedirectsToIndexHTML865=== PAUSE TestIsValidUploadKey/nix-cache-info866=== RUN TestIsValidUploadKey/index.html867=== PAUSE TestProxyWriteTimeout/10_GiB_nar868=== RUN TestProxyWriteTimeout/unknown_size869=== PAUSE TestProxyWriteTimeout/unknown_size870=== CONT TestReadProxyConditionalGet871=== PAUSE TestIsValidUploadKey/index.html872=== RUN TestIsValidUploadKey/narinfo_key,_nar_type873=== PAUSE TestIsValidUploadKey/narinfo_key,_nar_type874=== RUN TestIsValidUploadKey/nar_key,_narinfo_type875=== PAUSE TestIsValidUploadKey/nar_key,_narinfo_type876=== RUN TestIsValidUploadKey/listing_key,_narinfo_type877=== PAUSE TestIsValidUploadKey/listing_key,_narinfo_type878=== RUN TestIsValidUploadKey/traversal879=== PAUSE TestIsValidUploadKey/traversal880=== RUN TestIsValidUploadKey/traversal_nar881=== PAUSE TestIsValidUploadKey/traversal_nar882=== RUN TestIsValidUploadKey/absolute883=== PAUSE TestIsValidUploadKey/absolute884=== RUN TestIsValidUploadKey/empty_key885=== PAUSE TestIsValidUploadKey/empty_key886=== RUN TestIsValidUploadKey/unknown_type887=== PAUSE TestIsValidUploadKey/unknown_type888=== CONT TestReadProxyOutlastsServerWriteTimeout8892026/11/11 23:29:40 INFO Client skipped oversized paths paths=3 nar_bytes=5000000000890--- PASS: TestSkippedUploadsHandler (0.22s)891=== CONT TestReadProxyHead892=== RUN TestUploadHandlersRejectOversizedBody/create_pending_closure893=== PAUSE TestUploadHandlersRejectOversizedBody/create_pending_closure894=== RUN TestUploadHandlersRejectOversizedBody/complete_multipart895=== PAUSE TestUploadHandlersRejectOversizedBody/complete_multipart896=== RUN TestUploadHandlersRejectOversizedBody/request_more_parts897=== PAUSE TestUploadHandlersRejectOversizedBody/request_more_parts898=== CONT TestReadProxyRangeRequest8992026/11/11 23:29:40 INFO Received uploads request method=POST path=/api/pending_closures900--- PASS: TestReadRedirectKeepsNarinfoProxied (2.41s)901=== CONT TestUploadHandlersRejectInvalidKeys902=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info903=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info904=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal905=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal906=== RUN TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key907=== PAUSE TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key908=== RUN TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key909=== PAUSE TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key910=== CONT TestPush_RootsAreLockedInByteOrder9112026/11/11 23:29:40 INFO Received complete multipart upload request method=POST path=/api/multipart/complete9122026/11/11 23:29:40 INFO Received push request method=POST path=/api/pushes9132026/11/11 23:29:40 INFO Received uploads request method=POST path=/api/pending_closures9142026/11/11 23:29:40 INFO Received push request method=POST path=/api/pushes9152026/11/11 23:29:40 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign9162026/11/11 23:29:40 INFO Signed narinfos id=1 count=19172026/11/11 23:29:41 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign9182026/11/11 23:29:41 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign9192026/11/11 23:29:41 ERROR Failed to sign narinfos signer="index 1" error="signing key cannot be nil"9202026/11/11 23:29:41 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign9212026/11/11 23:29:41 ERROR Failed to sign narinfos signer=niks3-test-1 error="signer failed"9222026/11/11 23:29:41 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign9232026/11/11 23:29:41 ERROR Signer returned unexpected signature count signer=niks3-test-1 got=0 want=19242026/11/11 23:29:41 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign9252026/11/11 23:29:41 ERROR Signer returned unexpected object key signer=niks3-test-1 object_key=unexpected.narinfo926--- PASS: TestPush_SignsNarinfosOfItsPendingObjects (2.21s)927=== CONT TestObjectStatsUpdatedOncePerCommit9282026/11/11 23:29:41 INFO Received push request method=POST path=/api/pushes9292026/11/11 23:29:41 INFO Received complete push request method=POST path=/api/pushes/1/complete9302026/11/11 23:29:41 INFO Received uploads request method=POST path=/api/pending_closures9312026/11/11 23:29:41 INFO Received uploads request method=POST path=/api/pending_closures9322026/11/11 23:29:41 INFO Received uploads request method=POST path=/api/pending_closures9332026/11/11 23:29:41 INFO Received uploads request method=POST path=/api/pending_closures934--- PASS: TestReadProxyDisabled (3.10s)935=== CONT TestPush_CompleteWaitsForCleanupOfThePush936--- PASS: TestService_Rustfstest (1.35s)937=== CONT TestReadProxy4049382026/11/11 23:29:41 INFO Completed push id=1 objects=4 duration=139.092987ms9392026/11/11 23:29:41 INFO Received push request method=POST path=/api/pushes9402026/11/11 23:29:41 INFO Received uploads request method=POST path=/api/pending_closures9412026-11-11 23:29:41.222 UTC [496] ERROR: could not obtain lock on row in relation "closures"9422026-11-11 23:29:41.222 UTC [496] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT9432026-11-11 23:29:41.224 UTC [496] ERROR: could not obtain lock on row in relation "closures"9442026-11-11 23:29:41.224 UTC [496] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT9452026-11-11 23:29:41.225 UTC [496] ERROR: could not obtain lock on row in relation "closures"9462026-11-11 23:29:41.225 UTC [496] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT9472026-11-11 23:29:41.225 UTC [496] ERROR: could not obtain lock on row in relation "closures"9482026-11-11 23:29:41.225 UTC [496] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT9492026-11-11 23:29:41.225 UTC [496] ERROR: could not obtain lock on row in relation "closures"9502026-11-11 23:29:41.225 UTC [496] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT951--- PASS: TestReadProxyInvalidPath (3.24s)952=== CONT TestPush_RepeatedRootCommitsOnce953--- PASS: TestPush_RootsAreLockedInByteOrder (0.84s)954=== CONT TestReadProxyNarStreaming9552026/11/11 23:29:41 INFO Aborted multipart uploads count=0 kept=09562026/11/11 23:29:41 WARN Force mode enabled - objects will be deleted immediately without grace period9572026/11/11 23:29:41 INFO Received uploads request method=POST path=/api/pending_closures9582026/11/11 23:29:41 INFO Registered completed upload object_key=nar/jjjjjjjjjjjjjjjjjjjjjjjjjjjjjj0100000000000000000000.nar.zst9592026/11/11 23:29:41 INFO Received uploads request method=POST path=/api/pending_closures960--- PASS: TestPresignedUploadRegisteredBeforeCommit (1.49s)961=== CONT TestPush_CompleteCommitsEveryRoot9622026/11/11 23:29:41 INFO Received uploads request method=POST path=/api/pending_closures9632026/11/11 23:29:41 INFO Received uploads request method=POST path=/api/pending_closures964--- PASS: TestCreatePendingClosure_SmallNARUsesSimplePUT (3.38s)965=== CONT TestReadProxyContentNegotiation9662026/11/11 23:29:41 INFO Received complete multipart upload request method=POST path=/api/multipart/complete9672026/11/11 23:29:41 ERROR Failed to get multipart upload error="no rows in result set" upload_id=does-not-exist object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst968--- PASS: TestCompleteMultipartUnregistered (3.47s)969=== CONT TestPush_OverlappingRootsStoreOneRowPerKey9702026/11/11 23:29:41 INFO Received complete multipart upload request method=POST path=/api/multipart/complete971=== RUN TestPush_RejectsBadRequests/root_not_in_objects972=== PAUSE TestPush_RejectsBadRequests/root_not_in_objects973=== RUN TestPush_RejectsBadRequests/no_roots974=== PAUSE TestPush_RejectsBadRequests/no_roots975=== RUN TestPush_RejectsBadRequests/no_objects976=== PAUSE TestPush_RejectsBadRequests/no_objects977=== RUN TestPush_RejectsBadRequests/bad_root978=== PAUSE TestPush_RejectsBadRequests/bad_root979=== CONT TestReadProxyNarinfoAlreadyDecompressed980--- PASS: TestReadProxyHead (1.51s)981=== CONT TestReadRedirectUsesPublicS3URL9822026/11/11 23:29:41 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst upload_id=OTkxZDgwZGMtZjIyMi00YWYyLTg3M2YtYjY3NGY0ZWU0MWU0LjUyN2IyYmE2LWUyNDgtNDA4My05NzA2LTYyYzU1NzA4MDQwOXgxNzk0NDM5NzgwOTc3Mzc2ODQ3 parts=109832026/11/11 23:29:41 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete9842026/11/11 23:29:41 INFO Completed upload id=1 objects=2 duration=7.577112ms9852026/11/11 23:29:41 INFO Received complete multipart upload request method=POST path=/api/multipart/complete9862026/11/11 23:29:41 INFO Received complete multipart upload request method=POST path=/api/multipart/complete9872026/11/11 23:29:41 INFO Received uploads request method=POST path=/api/pending_closures9882026/11/11 23:29:41 INFO Received uploads request method=POST path=/api/pending_closures9892026/11/11 23:29:41 INFO Completed multipart upload object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst upload_id=OTkxZDgwZGMtZjIyMi00YWYyLTg3M2YtYjY3NGY0ZWU0MWU0LjkxOWM3NjQ3LTZjYjYtNDdiMi1hMjRlLTdmN2FlMzYxN2Y1NngxNzk0NDM5NzgxMDcxMjg3Mzgz parts=109902026/11/11 23:29:41 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete9912026/11/11 23:29:41 INFO Completed multipart upload object_key=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhh0100000000000000000000.nar.zst upload_id=OTkxZDgwZGMtZjIyMi00YWYyLTg3M2YtYjY3NGY0ZWU0MWU0LjY2OTNmZDRmLTcwYzUtNGY0ZS1iOGNhLTUyYjc4NmJhMjBiZHgxNzk0NDM5NzgxMDM4MDI4OTY1 parts=129922026/11/11 23:29:41 INFO Received uploads request method=POST path=/api/pending_closures9932026/11/11 23:29:41 INFO Received uploads request method=POST path=/api/pending_closures9942026/11/11 23:29:41 INFO Completed upload id=1 objects=2 duration=5.901057ms9952026/11/11 23:29:41 INFO Received get closure request method=GET path=/api/closures/000000000000000000000000000000009962026/11/11 23:29:41 INFO Object in database but missing from S3 key=dadb44fdadb44fdadb44fdadb44f0000.narinfo9972026/11/11 23:29:41 WARN Found objects in DB but missing from S3, will re-upload count=1998--- PASS: TestService_verifyS3Integrity (3.60s)999=== CONT TestReadProxyNarinfo1000--- PASS: TestCompletedNarNotReofferedAcrossClosures (1.79s)1001=== CONT TestIsValidCachePath1002=== RUN TestIsValidCachePath/narinfo1003=== PAUSE TestIsValidCachePath/narinfo1004=== RUN TestIsValidCachePath/narinfo_all_nix_base32_chars1005=== PAUSE TestIsValidCachePath/narinfo_all_nix_base32_chars1006=== RUN TestIsValidCachePath/nar_zst1007=== PAUSE TestIsValidCachePath/nar_zst1008=== RUN TestIsValidCachePath/nar_xz1009=== PAUSE TestIsValidCachePath/nar_xz1010=== RUN TestIsValidCachePath/nar_bz21011=== PAUSE TestIsValidCachePath/nar_bz21012=== RUN TestIsValidCachePath/nar_uncompressed1013=== PAUSE TestIsValidCachePath/nar_uncompressed1014=== RUN TestIsValidCachePath/ls1015=== PAUSE TestIsValidCachePath/ls1016=== RUN TestIsValidCachePath/log1017=== PAUSE TestIsValidCachePath/log1018=== RUN TestIsValidCachePath/realisation1019=== PAUSE TestIsValidCachePath/realisation1020=== RUN TestIsValidCachePath/nix-cache-info1021=== PAUSE TestIsValidCachePath/nix-cache-info1022=== RUN TestIsValidCachePath/index.html1023=== PAUSE TestIsValidCachePath/index.html1024=== RUN TestIsValidCachePath/traversal_parent1025=== PAUSE TestIsValidCachePath/traversal_parent1026=== RUN TestIsValidCachePath/traversal_in_middle1027=== PAUSE TestIsValidCachePath/traversal_in_middle1028=== RUN TestIsValidCachePath/invalid_char_e1029=== PAUSE TestIsValidCachePath/invalid_char_e1030=== RUN TestIsValidCachePath/invalid_char_u1031=== PAUSE TestIsValidCachePath/invalid_char_u1032=== RUN TestIsValidCachePath/random_path1033=== PAUSE TestIsValidCachePath/random_path1034=== RUN TestIsValidCachePath/empty1035=== PAUSE TestIsValidCachePath/empty1036=== RUN TestIsValidCachePath/leading_slash1037=== PAUSE TestIsValidCachePath/leading_slash1038=== RUN TestIsValidCachePath/wrong_extension1039=== PAUSE TestIsValidCachePath/wrong_extension1040=== RUN TestIsValidCachePath/short_hash1041=== PAUSE TestIsValidCachePath/short_hash1042=== CONT TestPresentReportsOnlyClosureRoots10432026/11/11 23:29:41 INFO Received uploads request method=POST path=/api/pending_closures10442026/11/11 23:29:41 INFO Starting cleanup of old closures method=DELETE path=/api/closures10452026/11/11 23:29:41 INFO Received complete multipart upload request method=POST path=/api/multipart/complete10462026/11/11 23:29:41 WARN CompleteMultipartUpload errored but object exists; treating as success error="The specified key does not exist." object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=OTkxZDgwZGMtZjIyMi00YWYyLTg3M2YtYjY3NGY0ZWU0MWU0LmQxMTYxODdhLTE0N2UtNGNkZS1iYzU3LTg4MmM4ZDZiYjIyMngxNzk0NDM5NzgxNTcwMzIyMjAy10472026/11/11 23:29:41 INFO Aborted multipart uploads count=0 kept=01048--- PASS: TestReadRedirectNar (3.62s)1049=== CONT TestCreatePin_ReservedPins10502026/11/11 23:29:41 WARN readiness check failed error="closed pool"1051--- PASS: TestService_readinessHandler (3.64s)1052=== CONT TestGCAdvisoryLockBlocksConcurrentRun10532026/11/11 23:29:41 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=OTkxZDgwZGMtZjIyMi00YWYyLTg3M2YtYjY3NGY0ZWU0MWU0LmQxMTYxODdhLTE0N2UtNGNkZS1iYzU3LTg4MmM4ZDZiYjIyMngxNzk0NDM5NzgxNTcwMzIyMjAy parts=11054--- PASS: TestCompleteMultipartUpload_ErrorButObjectExists (3.63s)1055=== CONT TestParseSingleRange1056=== RUN TestParseSingleRange/none1057=== PAUSE TestParseSingleRange/none1058=== RUN TestParseSingleRange/unknown_unit1059=== PAUSE TestParseSingleRange/unknown_unit1060=== RUN TestParseSingleRange/multi-range_ignored1061=== PAUSE TestParseSingleRange/multi-range_ignored1062=== RUN TestParseSingleRange/malformed_no_dash1063=== PAUSE TestParseSingleRange/malformed_no_dash1064=== RUN TestParseSingleRange/malformed_both_empty1065=== PAUSE TestParseSingleRange/malformed_both_empty1066=== RUN TestParseSingleRange/malformed_end_before_start1067=== PAUSE TestParseSingleRange/malformed_end_before_start1068=== RUN TestParseSingleRange/closed1069=== PAUSE TestParseSingleRange/closed1070=== RUN TestParseSingleRange/open-ended1071=== PAUSE TestParseSingleRange/open-ended1072=== RUN TestParseSingleRange/end_clamped_to_size1073=== PAUSE TestParseSingleRange/end_clamped_to_size1074=== RUN TestParseSingleRange/suffix1075=== PAUSE TestParseSingleRange/suffix1076=== RUN TestParseSingleRange/suffix_exceeds_size1077=== PAUSE TestParseSingleRange/suffix_exceeds_size1078=== RUN TestParseSingleRange/single_byte1079=== PAUSE TestParseSingleRange/single_byte1080=== RUN TestParseSingleRange/start_past_EOF1081=== PAUSE TestParseSingleRange/start_past_EOF1082=== RUN TestParseSingleRange/start_far_past_EOF1083=== PAUSE TestParseSingleRange/start_far_past_EOF1084=== CONT TestOrphanedObjectsGCStressTest1085--- PASS: TestReadProxyRangeRequest (1.59s)1086=== CONT TestPresentNotReportedWhileGCDeletesClosure1087=== RUN TestPresentNotReportedWhileGCDeletesClosure/fresh1088=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/fresh1089=== RUN TestPresentNotReportedWhileGCDeletesClosure/stale1090=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/stale1091=== CONT TestService_healthCheckHandler1092--- PASS: TestReadProxyRootRedirectsToIndexHTML (1.70s)1093=== CONT TestOrphanedObjectsGC10942026/11/11 23:29:41 INFO Received cleanup request method=DELETE path=/api/pending_closures10952026/11/11 23:29:41 INFO Aborted multipart uploads count=0 kept=010962026/11/11 23:29:41 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:46055/oidc10972026/11/11 23:29:41 INFO Received uploads request method=POST path=/api/pending_closures10982026/11/11 23:29:41 INFO Received cleanup request method=DELETE path=/api/pending_closures10992026/11/11 23:29:41 INFO Aborted multipart uploads count=1 kept=011002026/11/11 23:29:41 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete11012026-11-11 23:29:41.757 UTC [464] ERROR: Closure does not exist: id=111022026-11-11 23:29:41.757 UTC [464] CONTEXT: PL/pgSQL function commit_pending_closure(bigint) line 13 at RAISE11032026-11-11 23:29:41.757 UTC [464] STATEMENT: -- name: CommitPendingClosure :one1104 SELECT commit_pending_closure($1::bigint)::bigint1105 1106--- PASS: TestService_cleanupPendingClosuresHandler (3.77s)1107=== CONT TestGracefulShutdownDrainsInflight11082026/11/11 23:29:41 INFO Starting HTTP server address=127.0.0.1:4174311092026/11/11 23:29:41 INFO Shutdown signal received, draining in-flight requests timeout=10s11102026/11/11 23:29:41 INFO Received complete multipart upload request method=POST path=/api/multipart/complete11112026/11/11 23:29:41 INFO Received push request method=POST path=/api/pushes11122026/11/11 23:29:41 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0100000000000000000000.nar.zst upload_id=OTkxZDgwZGMtZjIyMi00YWYyLTg3M2YtYjY3NGY0ZWU0MWU0LjY1MDM0NDYyLTU0YWQtNDRhOS1iNzA4LWU3NDIxYjYxMzRlYXgxNzk0NDM5NzgxMjczNTc1ODUy parts=121113--- PASS: TestObjectStatsUpdatedOncePerCommit (0.80s)1114=== CONT TestPresentRefreshesOnlyStaleClosures1115--- PASS: TestGracefulShutdownDrainsInflight (0.07s)1116=== CONT TestGCTaskStore_Fail1117--- PASS: TestGCTaskStore_Fail (0.00s)1118=== CONT TestResolveDBConnectionString1119=== RUN TestResolveDBConnectionString/flag_wins1120=== PAUSE TestResolveDBConnectionString/flag_wins1121=== RUN TestResolveDBConnectionString/file_when_flag_empty1122=== PAUSE TestResolveDBConnectionString/file_when_flag_empty1123=== RUN TestResolveDBConnectionString/missing_file_is_an_error1124=== PAUSE TestResolveDBConnectionString/missing_file_is_an_error1125=== RUN TestResolveDBConnectionString/PGHOST_allows_empty1126=== PAUSE TestResolveDBConnectionString/PGHOST_allows_empty1127=== RUN TestResolveDBConnectionString/nothing_configured1128=== PAUSE TestResolveDBConnectionString/nothing_configured1129=== CONT TestGCTaskStore_PhaseUpdates1130--- PASS: TestGCTaskStore_PhaseUpdates (0.00s)1131=== CONT TestClientWithDependencies1132--- PASS: TestReadProxyConditionalGet (1.87s)1133=== CONT TestLeadEndsOnShutdown1134--- PASS: TestReadProxy404 (0.76s)1135=== CONT TestGCTaskStore_CompletedAllowsNewTask1136--- PASS: TestGCTaskStore_CompletedAllowsNewTask (0.00s)1137=== CONT TestLeadGenerationGrowsWithEveryGrant11382026/11/11 23:29:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=011392026/11/11 23:29:41 INFO Vacuumed table table=pending_closures11402026/11/11 23:29:41 INFO Vacuumed table table=pending_objects11412026/11/11 23:29:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=2 objects-deleted-after-grace-period=2 objects-failed-to-delete=011422026/11/11 23:29:42 INFO Vacuumed table table=multipart_uploads11432026/11/11 23:29:42 INFO Vacuumed table table=pending_closures11442026/11/11 23:29:42 INFO Vacuumed table table=pending_objects11452026/11/11 23:29:42 INFO Vacuumed table table=closures11462026/11/11 23:29:42 INFO Vacuumed table table=multipart_uploads11472026/11/11 23:29:42 INFO Vacuumed table table=objects11482026/11/11 23:29:42 INFO Vacuumed table table=closures11492026/11/11 23:29:42 INFO Vacuumed table table=objects11502026/11/11 23:29:42 INFO Received complete push request method=POST path=/api/pushes/2/complete11512026/11/11 23:29:42 INFO Received get closure request method=GET path=/api/closures/000000000000000000000000000000001152--- PASS: TestService_createPendingClosureHandler (4.06s)1153=== CONT TestConcurrentCommitsSharingObjectsDoNotDeadlock11542026/11/11 23:29:42 INFO Completed push id=2 objects=4 duration=12.57212ms1155--- PASS: TestPush_SkippedKeySurvivesGCBeforeCommit (2.27s)1156=== CONT TestGCTaskStore_GetReturnsLatest1157--- PASS: TestGCTaskStore_GetReturnsLatest (0.00s)1158=== CONT TestLeadIncumbentAnnouncesAtOnce11592026-11-11 23:29:42.110 UTC [572] ERROR: Push does not exist: id=111602026-11-11 23:29:42.110 UTC [572] CONTEXT: PL/pgSQL function commit_push(bigint) line 11 at RAISE11612026-11-11 23:29:42.110 UTC [572] STATEMENT: SELECT commit_push($1)1162--- PASS: TestPush_CompleteWaitsForCleanupOfThePush (1.02s)1163=== CONT TestDeleteClosuresRacingPins11642026/11/11 23:29:42 INFO Received push request method=POST path=/api/pushes1165--- PASS: TestPush_OverlappingRootsStoreOneRowPerKey (0.68s)1166=== CONT TestGCTaskStore_GetEmpty1167--- PASS: TestGCTaskStore_GetEmpty (0.00s)1168=== CONT TestLeadUnannouncedStandbyYieldsToIncumbent11692026/11/11 23:29:42 INFO Received push request method=POST path=/api/pushes11702026/11/11 23:29:42 INFO Received complete push request method=POST path=/api/pushes/1/complete11712026/11/11 23:29:42 INFO Completed push id=1 objects=4 duration=11.460849ms1172--- PASS: TestPush_RepeatedRootCommitsOnce (0.94s)1173=== CONT TestClientFallsBackToClosures1174--- PASS: TestReadProxyNarinfoAlreadyDecompressed (0.75s)1175=== CONT TestGCTaskStore_ConflictDifferentParams1176--- PASS: TestGCTaskStore_ConflictDifferentParams (0.00s)1177=== CONT TestLeadElectsOneAndHandsOver11782026/11/11 23:29:42 INFO Received complete multipart upload request method=POST path=/api/multipart/complete1179--- PASS: TestService_healthCheckHandler (0.61s)1180=== CONT TestClientPushesUseOnePush11812026/11/11 23:29:42 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0100000000000000000000.nar.zst upload_id=OTkxZDgwZGMtZjIyMi00YWYyLTg3M2YtYjY3NGY0ZWU0MWU0Ljg0Y2Q1ZmM2LTg1NzItNDM0My05MGExLTVlNjQwYWM5YzdmY3gxNzk0NDM5NzgxMjg1MzU4MjA1 parts=121182--- PASS: TestSharedNarBothUploadsComplete (3.51s)1183=== CONT TestForceGCDuringPushOffersSweptObject1184=== RUN TestForceGCDuringPushOffersSweptObject/before_pending_rows1185=== PAUSE TestForceGCDuringPushOffersSweptObject/before_pending_rows1186=== RUN TestForceGCDuringPushOffersSweptObject/after_presence_check1187=== PAUSE TestForceGCDuringPushOffersSweptObject/after_presence_check1188=== CONT TestConnectWaitsForAPeerMigration11892026/11/11 23:29:42 ERROR Failed to decode narinfo key=3hcdxyjf9yiq7qf3i4548drb6sjmwa1v.narinfo limit=16777216 error=<nil>1190--- PASS: TestReadProxyNarinfo (0.73s)1191=== CONT TestPinProtectsFromGC11922026/11/11 23:29:42 ERROR failed to check GC advisory lock error="query pg_locks: failed to connect to `user=nixbld database=niks3`: /nonexistent/.s.PGSQL.5432 (/nonexistent): dial error: dial unix /nonexistent/.s.PGSQL.5432: connect: no such file or directory"1193--- PASS: TestGCAdvisoryLockBlocksConcurrentRun (0.71s)1194=== CONT TestCreatePendingClosureVerifyS3FailureReleasesConnection1195--- PASS: TestPresentRefreshesOnlyStaleClosures (0.56s)1196=== CONT TestPushWaitsForSweepHoldingRow11972026/11/11 23:29:42 INFO Received push request method=POST path=/api/pushes1198--- PASS: TestReadRedirectUsesPublicS3URL (0.86s)1199=== CONT TestGCSweepDeliversEachKeyOnce12002026/11/11 23:29:42 INFO Received complete push request method=POST path=/api/pushes/1/complete1201--- PASS: TestPresentReportsOnlyClosureRoots (0.85s)1202=== CONT TestClientSharedPathCommittedMidPush12032026/11/11 23:29:42 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux12042026/11/11 23:29:42 WARN Refused reserved pin name=worker-x86_64-linux12052026/11/11 23:29:42 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux12062026/11/11 23:29:42 INFO Received create pin request method=POST path=/api/pins/my-app12072026/11/11 23:29:42 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux1208--- PASS: TestCreatePin_ReservedPins (0.86s)1209=== CONT TestTombstonedObjectOfferedWithoutWaiting1210--- PASS: TestReadProxyNarStreaming (1.27s)1211=== CONT TestGCTaskStore_DeduplicateSameParams1212--- PASS: TestGCTaskStore_DeduplicateSameParams (0.00s)1213=== CONT TestPushDedupSurvivesConcurrentGC1214--- PASS: TestReadProxyContentNegotiation (1.19s)1215=== CONT TestFailedS3DeleteKeepsTombstone12162026/11/11 23:29:42 INFO lead: acquired remote=192.0.2.1:1234 generation=112172026/11/11 23:29:43 INFO Completed push id=1 objects=6 duration=682.982194ms1218--- PASS: TestPush_CompleteCommitsEveryRoot (1.84s)1219=== CONT TestGCSweepSkipsPendingObjects12202026/11/11 23:29:43 INFO Received uploads request method=POST path=/api/pending_closures1221=== NAME TestOrphanedObjectsGC1222 orphaned_objects_gc_test.go:287: GC Test Summary:1223 orphaned_objects_gc_test.go:288: - Kept: 2 objects from closure A1224 orphaned_objects_gc_test.go:289: - Deleted: 2 objects from closure B1225 orphaned_objects_gc_test.go:290: - Deleted: 6 orphaned chain objects (X1->X2->X3)1226 orphaned_objects_gc_test.go:291: - Deleted: 2 orphaned single objects (Y)1227 orphaned_objects_gc_test.go:292: - Total deleted: 10 objects1228--- PASS: TestOrphanedObjectsGC (1.52s)1229=== CONT TestSweepRechecksPendingAfterLocking12302026/11/11 23:29:43 INFO Received uploads request method=POST path=/api/pending_closures1231=== NAME TestClientWithDependencies1232 client_integration_test.go:661: Built derivation: /build/TestClientWithDependencies659564880/001/store/502nmqzxwwp5ngplf98a9xzsi9aj51mw-test-script12332026/11/11 23:29:43 INFO Aborted multipart uploads count=0 kept=012342026/11/11 23:29:43 INFO Aborted multipart uploads count=0 kept=012352026/11/11 23:29:43 INFO Received uploads request method=POST path=/api/pending_closures1236--- PASS: TestTombstonedObjectOfferedWithoutWaiting (0.92s)1237=== CONT TestGetExistingObjectsLocksRows1238--- PASS: TestCreatePendingClosureVerifyS3FailureReleasesConnection (1.08s)1239=== CONT TestRegisterCompletedObjectNeedsPendingRow12402026/11/11 23:29:43 INFO lead: acquired remote=192.0.2.1:1234 generation=112412026/11/11 23:29:43 INFO lead: released remote=192.0.2.1:1234 generation=11242--- PASS: TestLeadIncumbentAnnouncesAtOnce (1.35s)1243=== CONT TestGCMetrics12442026/11/11 23:29:43 INFO Received uploads request method=POST path=/api/pending_closures12452026/11/11 23:29:43 INFO lead: acquired remote=192.0.2.1:1234 generation=112462026/11/11 23:29:43 INFO lead: acquired remote=192.0.2.1:1234 generation=112472026/11/11 23:29:43 INFO Aborted multipart uploads count=0 kept=012482026/11/11 23:29:43 INFO Received uploads request method=POST path=/api/pending_closures12492026/11/11 23:29:43 INFO lead: acquired remote=192.0.2.1:1234 generation=112502026/11/11 23:29:43 INFO Received uploads request method=POST path=/api/pending_closures12512026/11/11 23:29:43 INFO Aborted multipart uploads count=0 kept=012522026/11/11 23:29:43 INFO lead: released remote=192.0.2.1:1234 generation=112532026/11/11 23:29:43 INFO lead: acquired remote=192.0.2.1:1234 generation=212542026/11/11 23:29:43 WARN Force mode enabled - objects will be deleted immediately without grace period12552026/11/11 23:29:43 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=012562026/11/11 23:29:43 WARN Force mode enabled - objects will be deleted immediately without grace period12572026/11/11 23:29:43 INFO Vacuumed table table=pending_closures1258=== NAME TestClientWithDependencies1259 client_integration_test.go:663: Found 1 dependencies (including self)12602026/11/11 23:29:43 INFO Vacuumed table table=pending_objects12612026/11/11 23:29:43 INFO Vacuumed table table=multipart_uploads12622026/11/11 23:29:43 INFO Vacuumed table table=closures12632026/11/11 23:29:43 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=012642026/11/11 23:29:43 INFO Vacuumed table table=objects12652026/11/11 23:29:43 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."12662026/11/11 23:29:43 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."12672026/11/11 23:29:43 INFO Vacuumed table table=pending_closures1268--- PASS: TestFailedS3DeleteKeepsTombstone (1.18s)1269=== CONT TestSweepSkipsRowsLockedByPush12702026/11/11 23:29:43 INFO Vacuumed table table=pending_objects12712026/11/11 23:29:43 INFO Vacuumed table table=multipart_uploads1272--- PASS: TestGCSweepSkipsPendingObjects (0.64s)1273=== CONT TestGCBugBareHashReferences12742026/11/11 23:29:43 INFO Vacuumed table table=closures12752026/11/11 23:29:43 INFO Vacuumed table table=objects12762026/11/11 23:29:43 INFO Aborted multipart uploads count=0 kept=012772026/11/11 23:29:43 WARN Force mode enabled - objects will be deleted immediately without grace period12782026/11/11 23:29:43 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=012792026/11/11 23:29:43 INFO Vacuumed table table=pending_closures12802026/11/11 23:29:43 INFO Vacuumed table table=pending_objects12812026/11/11 23:29:43 INFO Vacuumed table table=multipart_uploads12822026/11/11 23:29:43 INFO Vacuumed table table=closures12832026/11/11 23:29:43 INFO Vacuumed table table=objects1284--- PASS: TestPushDedupSurvivesConcurrentGC (1.35s)1285=== CONT TestGCTaskStore_StartNew1286--- PASS: TestGCTaskStore_StartNew (0.00s)1287=== CONT TestGCEndsOnShutdown1288=== RUN TestGCEndsOnShutdown/before_the_run1289=== PAUSE TestGCEndsOnShutdown/before_the_run1290=== RUN TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete1291=== PAUSE TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete1292=== RUN TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark1293=== PAUSE TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark1294=== CONT TestService_ReadScope_PublicByDefault12952026/11/11 23:29:43 INFO Aborted multipart uploads count=0 kept=01296--- PASS: TestRegisterCompletedObjectNeedsPendingRow (0.56s)1297=== CONT TestCommitRacingPendingCleanupKeepsObjects12982026/11/11 23:29:44 INFO Aborted multipart uploads count=0 kept=01299--- PASS: TestPushWaitsForSweepHoldingRow (1.69s)1300=== CONT TestClientMultipleUploads13012026/11/11 23:29:44 WARN Force mode enabled - objects will be deleted immediately without grace period13022026/11/11 23:29:44 WARN Force mode enabled - objects will be deleted immediately without grace period1303--- PASS: TestConcurrentCommitsSharingObjectsDoNotDeadlock (2.21s)1304=== CONT TestService_ReadAuthMiddleware13052026/11/11 23:29:44 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=013062026/11/11 23:29:44 INFO Vacuumed table table=pending_closures13072026/11/11 23:29:44 INFO Vacuumed table table=pending_objects13082026/11/11 23:29:44 INFO Vacuumed table table=multipart_uploads13092026/11/11 23:29:44 INFO Vacuumed table table=closures13102026/11/11 23:29:44 INFO Vacuumed table table=objects1311--- PASS: TestSweepRechecksPendingAfterLocking (1.05s)1312=== CONT TestClientCADerivations13132026/11/11 23:29:44 WARN Force mode enabled - objects will be deleted immediately without grace period13142026/11/11 23:29:44 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=013152026/11/11 23:29:44 INFO Vacuumed table table=pending_closures13162026/11/11 23:29:44 INFO Vacuumed table table=pending_objects13172026/11/11 23:29:44 INFO Vacuumed table table=multipart_uploads13182026/11/11 23:29:44 INFO Vacuumed table table=closures13192026/11/11 23:29:44 INFO Vacuumed table table=objects13202026/11/11 23:29:44 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=01321--- PASS: TestGCMetrics (0.90s)1322=== CONT TestCacheStatsHandler13232026/11/11 23:29:44 INFO Vacuumed table table=pending_closures13242026/11/11 23:29:44 INFO Vacuumed table table=pending_objects13252026/11/11 23:29:44 INFO Vacuumed table table=multipart_uploads13262026/11/11 23:29:44 INFO Vacuumed table table=closures13272026/11/11 23:29:44 INFO Vacuumed table table=objects1328--- PASS: TestGCSweepDeliversEachKeyOnce (1.94s)1329=== CONT TestService_RequireScope_OIDC1330--- PASS: TestService_ReadScope_PublicByDefault (0.49s)1331=== CONT TestClientIntegration1332=== NAME TestPinProtectsFromGC1333 client_integration_test.go:779: Pinned store path: /build/TestPinProtectsFromGC4273941726/001/store/8w3hpsipsqzby221s37rpvyli0260jjx-pinned-file.txt1334 client_integration_test.go:780: Unpinned store path: /build/TestPinProtectsFromGC4273941726/001/store/asdp6l9ac6z0iakz0qvl291vchfad4qh-unpinned-file.txt13352026/11/11 23:29:44 INFO Aborted multipart uploads count=0 kept=01336--- PASS: TestGetExistingObjectsLocksRows (1.04s)1337=== CONT TestCacheConfigHandler1338=== RUN TestCacheConfigHandler/full_config,_no_issuer1339=== PAUSE TestCacheConfigHandler/full_config,_no_issuer1340=== RUN TestCacheConfigHandler/no_cache_url_configured1341=== PAUSE TestCacheConfigHandler/no_cache_url_configured1342=== RUN TestCacheConfigHandler/no_signing_keys1343=== PAUSE TestCacheConfigHandler/no_signing_keys1344=== RUN TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1345=== PAUSE TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1346=== CONT TestService_AuthMiddleware_OIDC13472026/11/11 23:29:44 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:44789/oidc13482026/11/11 23:29:44 INFO Aborted multipart uploads count=0 kept=01349=== NAME TestClientMultipleUploads1350 client_integration_test.go:406: Created store path 0: /build/TestClientMultipleUploads661552598/001/store/7y7qyyz9nqv1jcbllfk8lpxinw28zvh9-test-file-0.txt13512026/11/11 23:29:44 WARN Force mode enabled - objects will be deleted immediately without grace period13522026/11/11 23:29:44 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=013532026/11/11 23:29:44 INFO Vacuumed table table=pending_closures13542026/11/11 23:29:44 INFO Vacuumed table table=pending_objects13552026/11/11 23:29:44 INFO Vacuumed table table=multipart_uploads13562026/11/11 23:29:44 INFO Vacuumed table table=closures13572026/11/11 23:29:44 INFO Vacuumed table table=objects13582026/11/11 23:29:44 INFO Aborted multipart uploads count=0 kept=013592026/11/11 23:29:44 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:46639/oidc1360=== NAME TestOrphanedObjectsGCStressTest1361 orphaned_objects_gc_test.go:422: Created 10 active closures, 5 to-delete closures, 20 orphaned chains13622026/11/11 23:29:44 INFO lead: released remote=192.0.2.1:1234 generation=113632026/11/11 23:29:44 INFO lead: released remote=192.0.2.1:1234 generation=11364--- PASS: TestLeadEndsOnShutdown (2.76s)1365=== CONT TestClientErrorHandling1366=== RUN TestClientErrorHandling/InvalidStorePath1367=== PAUSE TestClientErrorHandling/InvalidStorePath1368=== RUN TestClientErrorHandling/InvalidAuthToken1369=== PAUSE TestClientErrorHandling/InvalidAuthToken1370=== RUN TestClientErrorHandling/ServerNotAvailable1371=== PAUSE TestClientErrorHandling/ServerNotAvailable1372=== CONT TestService_AuthMiddleware_MTLSProxyHeader13732026/11/11 23:29:44 INFO lead: acquired remote=192.0.2.1:1234 generation=213742026/11/11 23:29:44 INFO lead: given back to the incumbent remote=192.0.2.1:1234 generation=213752026/11/11 23:29:44 INFO lead: acquired remote=192.0.2.1:1234 generation=31376--- PASS: TestGCBugBareHashReferences (1.02s)1377=== CONT TestMultipartUploadAbortedWhenCancelledBeforeRecorded13782026/11/11 23:29:44 WARN Force mode enabled - objects will be deleted immediately without grace period13792026/11/11 23:29:44 INFO Received push request method=POST path=/api/pushes1380=== NAME TestOrphanedObjectsGCStressTest1381 orphaned_objects_gc_test.go:440: Marked 210 objects for deletion1382=== NAME TestClientMultipleUploads1383 client_integration_test.go:406: Created store path 1: /build/TestClientMultipleUploads661552598/001/store/sjv1wj6alikcl4kakxaqjnsy6400pbq0-test-file-1.txt13842026/11/11 23:29:44 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)1385=== NAME TestClientIntegration1386 client_integration_test.go:292: Created store path: /build/TestClientIntegration985285290/002/store/vzh3kvpfkmjy1rlw8rwkgjbfm0w83i7d-test-file.txt13872026/11/11 23:29:44 INFO Uploading 502nmqzxwwp5ngplf98a9xzsi9aj51mw-test-script (136B)13882026/11/11 23:29:44 WARN Failed to register uploaded object key=502nmqzxwwp5ngplf98a9xzsi9aj51mw.ls error="server returned 404: 404 page not found\n"13892026/11/11 23:29:44 WARN Failed to register uploaded object key=nar/1s7yghixzpinyqm7g59kyn3sav81pmnsdi5v63m6x4r2vacy907j.nar.zst error="server returned 404: 404 page not found\n"13902026/11/11 23:29:44 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign13912026/11/11 23:29:44 WARN Failed to register uploaded object key=log/kpxxx496b1kic474vsw953p5fsfn3kp6-test-script.drv error="server returned 404: 404 page not found\n"13922026/11/11 23:29:44 INFO Signed narinfos id=1 count=113932026/11/11 23:29:44 INFO Uploading 1 narinfos13942026/11/11 23:29:44 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=013952026/11/11 23:29:44 INFO Vacuumed table table=pending_closures13962026/11/11 23:29:44 INFO Vacuumed table table=pending_objects13972026/11/11 23:29:44 INFO Vacuumed table table=multipart_uploads13982026/11/11 23:29:44 INFO Vacuumed table table=closures13992026/11/11 23:29:44 INFO Received complete push request method=POST path=/api/pushes/1/complete14002026/11/11 23:29:44 INFO Vacuumed table table=objects14012026/11/11 23:29:44 WARN Failed to register uploaded object key=502nmqzxwwp5ngplf98a9xzsi9aj51mw.narinfo error="server returned 404: 404 page not found\n"1402--- PASS: TestSweepSkipsRowsLockedByPush (1.06s)1403=== CONT TestService_AuthMiddleware_MTLSBoundSubjects14042026/11/11 23:29:44 INFO lead: released remote=192.0.2.1:1234 generation=214052026/11/11 23:29:44 INFO lead: released remote=192.0.2.1:1234 generation=114062026/11/11 23:29:44 INFO Completed push id=1 objects=4 duration=11.845133ms14072026/11/11 23:29:44 INFO Upload complete. (505ms)14082026/11/11 23:29:44 WARN Force mode enabled - objects will be deleted immediately without grace period1409=== NAME TestClientWithDependencies1410 client_integration_test.go:665: Skipping nix copy test - isolated store (/build/TestClientWithDependencies659564880/001/store) requires matching store prefix1411--- PASS: TestClientWithDependencies (3.00s)1412=== CONT TestCreatePendingClosureRejectsOversizedNAR14132026/11/11 23:29:44 INFO Received uploads request method=POST path=/api/pending_closures1414--- PASS: TestCreatePendingClosureRejectsOversizedNAR (0.00s)1415=== CONT TestMetricsInventory14162026/11/11 23:29:44 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=014172026/11/11 23:29:44 WARN Rate limiter enabled after throttle name=s3-test rate=514182026/11/11 23:29:44 WARN S3 rate limit hit operation="complete multipart upload" error="Please reduce your request rate."1419=== NAME TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle1420 throttle_test.go:213: Proxy stats: total=13, throttled=10, completeMultipart=101421 throttle_test.go:215: Rate limiter: enabled=true, rate=5.001422--- PASS: TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle (6.84s)1423=== CONT TestPendingCleanupUsesOneCutoff14242026/11/11 23:29:44 INFO lead: acquired remote=192.0.2.1:1234 generation=314252026/11/11 23:29:44 INFO Vacuumed table table=pending_closures14262026/11/11 23:29:44 INFO lead: released remote=192.0.2.1:1234 generation=31427--- PASS: TestLeadGenerationGrowsWithEveryGrant (2.93s)1428=== CONT TestService_NativeMTLS14292026/11/11 23:29:44 INFO Vacuumed table table=pending_objects14302026/11/11 23:29:44 INFO Vacuumed table table=multipart_uploads14312026/11/11 23:29:44 INFO Vacuumed table table=closures14322026/11/11 23:29:44 INFO Vacuumed table table=objects14332026/11/11 23:29:44 INFO lead: acquired remote=192.0.2.1:1234 generation=21434--- PASS: TestCommitRacingPendingCleanupKeepsObjects (0.92s)1435=== CONT TestMetricsCommit1436=== NAME TestClientCADerivations1437 client_ca_test.go:136: Built CA derivation: /build/TestClientCADerivations988488692/001/store/v6gh39p5d5rffzaicaav8yn4vvbxhjx5-ca-test1438=== NAME TestClientMultipleUploads1439 client_integration_test.go:406: Created store path 2: /build/TestClientMultipleUploads661552598/001/store/fl8rj1snf0ialdcygydn1z0qhwdrw3v1-test-file-2.txt1440=== RUN TestService_RequireScope_OIDC/builder_may_write1441=== PAUSE TestService_RequireScope_OIDC/builder_may_write1442=== RUN TestService_RequireScope_OIDC/builder_may_not_admin1443=== PAUSE TestService_RequireScope_OIDC/builder_may_not_admin1444=== RUN TestService_RequireScope_OIDC/ops_may_admin1445=== PAUSE TestService_RequireScope_OIDC/ops_may_admin1446=== RUN TestService_RequireScope_OIDC/ops_may_not_write1447=== PAUSE TestService_RequireScope_OIDC/ops_may_not_write1448=== RUN TestService_RequireScope_OIDC/reader_may_not_write1449=== PAUSE TestService_RequireScope_OIDC/reader_may_not_write1450=== RUN TestService_RequireScope_OIDC/static_token_may_admin1451=== PAUSE TestService_RequireScope_OIDC/static_token_may_admin1452=== RUN TestService_RequireScope_OIDC/static_token_may_write1453=== PAUSE TestService_RequireScope_OIDC/static_token_may_write1454=== RUN TestService_RequireScope_OIDC/reader_may_read1455=== PAUSE TestService_RequireScope_OIDC/reader_may_read1456=== RUN TestService_RequireScope_OIDC/writer_implies_read1457=== PAUSE TestService_RequireScope_OIDC/writer_implies_read1458=== RUN TestService_RequireScope_OIDC/anonymous_may_not_read1459=== PAUSE TestService_RequireScope_OIDC/anonymous_may_not_read1460=== RUN TestService_RequireScope_OIDC/reader_may_read_with_netrc1461=== PAUSE TestService_RequireScope_OIDC/reader_may_read_with_netrc1462=== RUN TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1463=== PAUSE TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1464=== RUN TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read1465=== PAUSE TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read1466=== RUN TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1467=== PAUSE TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1468=== CONT TestEnsureAbortUploadsRule1469=== RUN TestService_AuthMiddleware_OIDC/valid_OIDC_token1470=== PAUSE TestService_AuthMiddleware_OIDC/valid_OIDC_token1471=== RUN TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1472=== PAUSE TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1473=== RUN TestService_AuthMiddleware_OIDC/malformed_token_rejected1474=== PAUSE TestService_AuthMiddleware_OIDC/malformed_token_rejected1475=== RUN TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1476=== PAUSE TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1477=== CONT TestGenerateLandingPage1478--- PASS: TestGenerateLandingPage (0.01s)1479=== CONT TestCacheConfigHandlerMaxNarSize1480--- PASS: TestCacheConfigHandlerMaxNarSize (0.00s)1481=== CONT TestMultipartCleanup1482--- PASS: TestCacheStatsHandler (0.67s)1483=== CONT TestNARDeduplicationMetadataUploadBug1484--- PASS: TestService_ReadAuthMiddleware (0.74s)1485=== CONT TestService_AuthMiddleware14862026/11/11 23:29:45 INFO Received uploads request method=POST path=/api/pending_closures1487--- PASS: TestMultipartUploadAbortedWhenCancelledBeforeRecorded (0.30s)1488=== CONT TestObjectStatsTrigger14892026/11/11 23:29:45 INFO Received push request method=POST path=/api/pushes14902026/11/11 23:29:45 INFO Uploading 2 paths to 127.0.0.1 (0 already cached)14912026/11/11 23:29:45 INFO Uploading c64vrk0nbdfavvgwbxsmysdwj9jrzq9q-shared-dep (136B)14922026/11/11 23:29:45 INFO Uploading dz13d3daj0ldhjxcjhyq5bmf5q7sqiiw-top (224B)14932026/11/11 23:29:45 WARN Failed to register uploaded object key=nar/1w57f6gyxvggi1ac4lxbglcw0xwp4rw26cjna1mfrvwyvxa9pjf0.nar.zst error="server returned 404: 404 page not found\n"14942026/11/11 23:29:45 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"14952026/11/11 23:29:45 WARN Failed to register uploaded object key=c64vrk0nbdfavvgwbxsmysdwj9jrzq9q.ls error="server returned 404: 404 page not found\n"14962026/11/11 23:29:45 WARN Failed to register uploaded object key=dz13d3daj0ldhjxcjhyq5bmf5q7sqiiw.ls error="server returned 404: 404 page not found\n"14972026/11/11 23:29:45 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign14982026/11/11 23:29:45 INFO Signed narinfos id=1 count=214992026/11/11 23:29:45 INFO Uploading 2 narinfos15002026/11/11 23:29:45 INFO Received complete push request method=POST path=/api/pushes/1/complete15012026/11/11 23:29:45 INFO Received push request method=POST path=/api/pushes15022026/11/11 23:29:45 WARN Failed to register uploaded object key=dz13d3daj0ldhjxcjhyq5bmf5q7sqiiw.narinfo error="server returned 404: 404 page not found\n"15032026/11/11 23:29:45 WARN Failed to register uploaded object key=c64vrk0nbdfavvgwbxsmysdwj9jrzq9q.narinfo error="server returned 404: 404 page not found\n"15042026/11/11 23:29:45 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)15052026/11/11 23:29:45 INFO Uploading vzh3kvpfkmjy1rlw8rwkgjbfm0w83i7d-test-file.txt (152B)15062026/11/11 23:29:45 INFO Completed push id=1 objects=6 duration=10.4635ms15072026/11/11 23:29:45 INFO Upload complete. (680ms)1508=== NAME TestClientSharedPathCommittedMidPush1509 client_integration_test.go:728: Retrieved narinfo from S3:1510 StorePath: /build/TestClientSharedPathCommittedMidPush4061244275/001/store/c64vrk0nbdfavvgwbxsmysdwj9jrzq9q-shared-dep1511 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1512 Compression: zstd15132026/11/11 23:29:45 WARN Failed to register uploaded object key=vzh3kvpfkmjy1rlw8rwkgjbfm0w83i7d.ls error="server returned 404: 404 page not found\n"1514 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821515 NarSize: 1361516 References: 1517 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n15182026/11/11 23:29:45 WARN Failed to register uploaded object key=nar/11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1.nar.zst error="server returned 404: 404 page not found\n"15192026/11/11 23:29:45 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign15202026/11/11 23:29:45 INFO Signed narinfos id=1 count=115212026/11/11 23:29:45 INFO Uploading 1 narinfos1522 client_integration_test.go:728: Retrieved narinfo from S3:1523 StorePath: /build/TestClientSharedPathCommittedMidPush4061244275/001/store/dz13d3daj0ldhjxcjhyq5bmf5q7sqiiw-top1524 URL: nar/1w57f6gyxvggi1ac4lxbglcw0xwp4rw26cjna1mfrvwyvxa9pjf0.nar.zst1525 Compression: zstd1526 NarHash: sha256:1w57f6gyxvggi1ac4lxbglcw0xwp4rw26cjna1mfrvwyvxa9pjf01527 NarSize: 2241528 References: /build/TestClientSharedPathCommittedMidPush4061244275/001/store/c64vrk0nbdfavvgwbxsmysdwj9jrzq9q-shared-dep1529 CA: text:sha256:1pw7mav5ya1ymxzzmnydw9k76iz6aca3f8lbz1718vp1vyv4i39715302026/11/11 23:29:45 INFO Received complete push request method=POST path=/api/pushes/1/complete15312026/11/11 23:29:45 WARN Failed to register uploaded object key=vzh3kvpfkmjy1rlw8rwkgjbfm0w83i7d.narinfo error="server returned 404: 404 page not found\n"15322026/11/11 23:29:45 INFO Completed push id=1 objects=3 duration=5.743375ms15332026/11/11 23:29:45 INFO Upload complete. (638ms)1534--- PASS: TestClientSharedPathCommittedMidPush (3.08s)1535=== CONT TestServerTLSConfig1536=== RUN TestServerTLSConfig/no_client_CA1537=== PAUSE TestServerTLSConfig/no_client_CA1538=== RUN TestServerTLSConfig/missing_CA_file1539=== PAUSE TestServerTLSConfig/missing_CA_file1540=== RUN TestServerTLSConfig/not_a_PEM_file1541=== PAUSE TestServerTLSConfig/not_a_PEM_file1542=== CONT TestDeduplicatedObjectsRecordedAsPending15432026/11/11 23:29:45 INFO Installed lifecycle rule aborting incomplete NAR uploads bucket=bucket82 days=21544--- PASS: TestEnsureAbortUploadsRule (0.61s)1545=== CONT TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed15462026/11/11 23:29:45 INFO Received push request method=POST path=/api/pushes1547=== NAME TestClientCADerivations1548 client_ca_test.go:139: Found 1 dependencies (including self)15492026/11/11 23:29:45 INFO Received push request method=POST path=/api/pushes15502026/11/11 23:29:45 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)1551--- PASS: TestObjectStatsTrigger (0.51s)15522026/11/11 23:29:45 INFO Uploading 8w3hpsipsqzby221s37rpvyli0260jjx-pinned-file.txt (128B)1553=== CONT TestPendingClosureWriteTimeout/empty1554=== CONT TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed15552026/11/11 23:29:45 WARN mTLS auth: subject not in bound subjects subject="CN=reader"15562026/11/11 23:29:45 WARN mTLS auth: subject not in bound subjects subject="CN=reader"1557--- PASS: TestService_NativeMTLS (0.75s)1558=== CONT TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed15592026/11/11 23:29:45 WARN Failed to register uploaded object key=8w3hpsipsqzby221s37rpvyli0260jjx.ls error="server returned 404: 404 page not found\n"15602026/11/11 23:29:45 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign15612026/11/11 23:29:45 INFO Signed narinfos id=1 count=115622026/11/11 23:29:45 INFO Uploading 1 narinfos15632026/11/11 23:29:45 WARN Failed to register uploaded object key=nar/0qmhsg42pqsz7x6hh02j8gznnac3yi3qpjhrqbwvz88i194jbb5s.nar.zst error="server returned 404: 404 page not found\n"15642026/11/11 23:29:45 INFO Received complete push request method=POST path=/api/pushes/1/complete15652026/11/11 23:29:45 INFO Completed push id=1 objects=3 duration=11.140266ms15662026/11/11 23:29:45 INFO Upload complete. (717ms)15672026/11/11 23:29:45 INFO Received uploads request method=POST path=/api/pending_closures15682026/11/11 23:29:45 INFO Uploading 2 paths to 127.0.0.1 (1 already cached)15692026/11/11 23:29:45 INFO Uploading aqcscjxqsdqplgf6w0qd5i7g5922c4s7-b (208B)15702026/11/11 23:29:45 INFO Uploading gcclw23xyqaznig5kqkqbhjv448hcdyd-shared-dep (136B)15712026/11/11 23:29:45 WARN Failed to register uploaded object key=8w3hpsipsqzby221s37rpvyli0260jjx.narinfo error="server returned 404: 404 page not found\n"1572--- PASS: TestReadProxyOutlastsServerWriteTimeout (5.66s)1573=== CONT TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost15742026/11/11 23:29:45 WARN Failed to register uploaded object key=gcclw23xyqaznig5kqkqbhjv448hcdyd.ls error="server returned 404: 404 page not found\n"15752026/11/11 23:29:45 WARN Failed to register uploaded object key=aqcscjxqsdqplgf6w0qd5i7g5922c4s7.ls error="server returned 404: 404 page not found\n"15762026/11/11 23:29:45 WARN Failed to register uploaded object key=k3cc70s7ph4aiw5wsrgg4qynabqxz7zb.ls error="server returned 404: 404 page not found\n"15772026/11/11 23:29:45 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign15782026/11/11 23:29:45 WARN Failed to register uploaded object key=nar/1iqlmnwq3ysnss3ybv3q1iyx10kwh17jcz94kcmb8mm7n3sxccnb.nar.zst error="server returned 404: 404 page not found\n"15792026/11/11 23:29:45 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"15802026/11/11 23:29:45 INFO Signed narinfos id=1 count=315812026/11/11 23:29:45 INFO Uploading 3 narinfos15822026/11/11 23:29:45 INFO Received complete push request method=POST path=/api/pushes/1/complete15832026/11/11 23:29:45 WARN Failed to register uploaded object key=k3cc70s7ph4aiw5wsrgg4qynabqxz7zb.narinfo error="server returned 404: 404 page not found\n"15842026/11/11 23:29:45 WARN Failed to register uploaded object key=aqcscjxqsdqplgf6w0qd5i7g5922c4s7.narinfo error="server returned 404: 404 page not found\n"15852026/11/11 23:29:45 WARN Failed to register uploaded object key=gcclw23xyqaznig5kqkqbhjv448hcdyd.narinfo error="server returned 404: 404 page not found\n"1586=== NAME TestOrphanedObjectsGCStressTest1587 orphaned_objects_gc_test.go:503: Stress test completed successfully:1588 orphaned_objects_gc_test.go:504: - Active objects preserved: 201589 orphaned_objects_gc_test.go:505: - Objects deleted: 2101590 orphaned_objects_gc_test.go:506: - Total GC'd: 2101591--- PASS: TestOrphanedObjectsGCStressTest (4.12s)1592=== CONT TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed15932026/11/11 23:29:45 INFO Completed push id=1 objects=8 duration=29.571222ms15942026/11/11 23:29:45 INFO Upload complete. (986ms)1595=== NAME TestClientPushesUseOnePush1596 client_pushes_test.go:97: Retrieved narinfo from S3:1597 StorePath: /build/TestClientPushesUseOnePush31561541/001/store/gcclw23xyqaznig5kqkqbhjv448hcdyd-shared-dep1598 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1599 Compression: zstd1600 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821601 NarSize: 1361602 References: 1603 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n1604 client_pushes_test.go:97: Retrieved narinfo from S3:1605 StorePath: /build/TestClientPushesUseOnePush31561541/001/store/k3cc70s7ph4aiw5wsrgg4qynabqxz7zb-a1606 URL: nar/1iqlmnwq3ysnss3ybv3q1iyx10kwh17jcz94kcmb8mm7n3sxccnb.nar.zst1607 Compression: zstd1608 NarHash: sha256:1iqlmnwq3ysnss3ybv3q1iyx10kwh17jcz94kcmb8mm7n3sxccnb1609 NarSize: 2081610 References: /build/TestClientPushesUseOnePush31561541/001/store/gcclw23xyqaznig5kqkqbhjv448hcdyd-shared-dep1611 CA: text:sha256:1pxcci0a3ni3y7lx1qy0jqwjb4w7lk3s34mlbhx80i9nvah57fbd1612 client_pushes_test.go:97: Retrieved narinfo from S3:1613 StorePath: /build/TestClientPushesUseOnePush31561541/001/store/aqcscjxqsdqplgf6w0qd5i7g5922c4s7-b1614 URL: nar/1iqlmnwq3ysnss3ybv3q1iyx10kwh17jcz94kcmb8mm7n3sxccnb.nar.zst1615 Compression: zstd1616 NarHash: sha256:1iqlmnwq3ysnss3ybv3q1iyx10kwh17jcz94kcmb8mm7n3sxccnb1617 NarSize: 2081618 References: /build/TestClientPushesUseOnePush31561541/001/store/gcclw23xyqaznig5kqkqbhjv448hcdyd-shared-dep1619 CA: text:sha256:1pxcci0a3ni3y7lx1qy0jqwjb4w7lk3s34mlbhx80i9nvah57fbd1620--- PASS: TestClientPushesUseOnePush (3.49s)1621=== CONT TestPendingClosureWriteTimeout/670k_objects1622=== CONT TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline1623--- PASS: TestService_AuthMiddleware_MTLSProxyHeader (1.18s)1624=== CONT TestPendingClosureWriteTimeout/400_objects1625=== CONT TestProxyWriteTimeout/narinfo1626=== CONT TestProxyWriteTimeout/unknown_size1627=== CONT TestProxyWriteTimeout/10_GiB_nar1628=== CONT TestProxyWriteTimeout/1_GiB_nar1629--- PASS: TestProxyWriteTimeout (0.20s)1630 --- PASS: TestProxyWriteTimeout/narinfo (0.00s)1631 --- PASS: TestProxyWriteTimeout/unknown_size (0.00s)1632 --- PASS: TestProxyWriteTimeout/10_GiB_nar (0.00s)1633 --- PASS: TestProxyWriteTimeout/1_GiB_nar (0.00s)1634=== CONT TestIsValidUploadKey/narinfo1635=== CONT TestIsValidUploadKey/realisation_plus_in_output1636=== CONT TestIsValidUploadKey/unknown_type1637=== CONT TestIsValidUploadKey/empty_key1638=== CONT TestIsValidUploadKey/absolute1639=== CONT TestIsValidUploadKey/traversal_nar1640=== CONT TestIsValidUploadKey/traversal1641=== CONT TestIsValidUploadKey/listing_key,_narinfo_type1642=== CONT TestIsValidUploadKey/nar_key,_narinfo_type1643=== CONT TestIsValidUploadKey/narinfo_key,_nar_type1644=== CONT TestIsValidUploadKey/build_log_home-manager_file1645=== CONT TestIsValidUploadKey/index.html1646=== CONT TestIsValidUploadKey/realisation1647=== CONT TestIsValidUploadKey/nix-cache-info1648=== CONT TestIsValidUploadKey/build_log_equals1649=== CONT TestIsValidUploadKey/build_log_question_mark1650=== CONT TestIsValidUploadKey/build_log_plus_in_name1651=== CONT TestIsValidUploadKey/nar_plain1652=== CONT TestIsValidUploadKey/build_log1653=== CONT TestIsValidUploadKey/listing1654=== CONT TestIsValidUploadKey/nar_xz1655=== CONT TestIsValidUploadKey/nar_zst1656=== CONT TestUploadHandlersRejectOversizedBody/create_pending_closure1657--- PASS: TestIsValidUploadKey (2.01s)1658 --- PASS: TestIsValidUploadKey/narinfo (0.00s)1659 --- PASS: TestIsValidUploadKey/realisation_plus_in_output (0.00s)1660 --- PASS: TestIsValidUploadKey/unknown_type (0.00s)1661 --- PASS: TestIsValidUploadKey/empty_key (0.00s)1662 --- PASS: TestIsValidUploadKey/absolute (0.00s)1663 --- PASS: TestIsValidUploadKey/traversal_nar (0.00s)1664 --- PASS: TestIsValidUploadKey/traversal (0.00s)1665 --- PASS: TestIsValidUploadKey/listing_key,_narinfo_type (0.00s)1666 --- PASS: TestIsValidUploadKey/nar_key,_narinfo_type (0.00s)1667 --- PASS: TestIsValidUploadKey/narinfo_key,_nar_type (0.00s)1668 --- PASS: TestIsValidUploadKey/build_log_home-manager_file (0.00s)1669 --- PASS: TestIsValidUploadKey/index.html (0.00s)1670 --- PASS: TestIsValidUploadKey/realisation (0.00s)1671 --- PASS: TestIsValidUploadKey/nix-cache-info (0.00s)1672 --- PASS: TestIsValidUploadKey/build_log_equals (0.00s)1673 --- PASS: TestIsValidUploadKey/build_log_question_mark (0.00s)1674 --- PASS: TestIsValidUploadKey/build_log_plus_in_name (0.00s)1675 --- PASS: TestIsValidUploadKey/nar_plain (0.00s)1676 --- PASS: TestIsValidUploadKey/build_log (0.00s)1677 --- PASS: TestIsValidUploadKey/listing (0.00s)1678 --- PASS: TestIsValidUploadKey/nar_xz (0.00s)1679 --- PASS: TestIsValidUploadKey/nar_zst (0.00s)16802026/11/11 23:29:45 INFO Received uploads request method=POST path=/16812026/11/11 23:29:45 WARN Authentication failed token_preview=wrongtoken token_length=10 reason="static API token mismatch"1682--- PASS: TestService_AuthMiddleware (0.84s)1683=== CONT TestUploadHandlersRejectOversizedBody/request_more_parts16842026/11/11 23:29:45 INFO Received request for more parts method=POST path=/16852026/11/11 23:29:45 INFO Received push request method=POST path=/api/pushes16862026/11/11 23:29:45 INFO Received cleanup request method=DELETE path=/api/pending_closures16872026/11/11 23:29:45 WARN Failed to abort upload, keeping its closure key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst error="Get \"http://127.0.0.1:1/bucket85/?location=\": dial tcp 127.0.0.1:1: connect: connection refused" code=""16882026/11/11 23:29:45 INFO Aborted multipart uploads count=0 kept=11689=== CONT TestUploadHandlersRejectOversizedBody/complete_multipart16902026/11/11 23:29:45 INFO Received complete multipart upload request method=POST path=/16912026/11/11 23:29:45 INFO Received cleanup request method=DELETE path=/api/pending_closures16922026/11/11 23:29:45 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted,O=Other"16932026/11/11 23:29:45 WARN mTLS auth: bound subjects configured but subject DN unavailable16942026/11/11 23:29:45 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted"1695--- PASS: TestService_AuthMiddleware_MTLSBoundSubjects (1.07s)1696=== CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info16972026/11/11 23:29:45 INFO Received uploads request method=POST path=/1698=== CONT TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key16992026/11/11 23:29:45 INFO Received complete multipart upload request method=POST path=/1700=== CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal17012026/11/11 23:29:45 INFO Received uploads request method=POST path=/1702=== CONT TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key17032026/11/11 23:29:45 INFO Received request for more parts method=POST path=/1704--- PASS: TestUploadHandlersRejectInvalidKeys (0.00s)1705 --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info (0.00s)1706 --- PASS: TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key (0.00s)1707 --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal (0.00s)1708 --- PASS: TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key (0.00s)1709=== CONT TestPush_RejectsBadRequests/root_not_in_objects17102026/11/11 23:29:45 INFO Received push request method=POST path=/api/pushes1711=== CONT TestPush_RejectsBadRequests/no_objects17122026/11/11 23:29:45 INFO Received push request method=POST path=/api/pushes1713=== CONT TestPush_RejectsBadRequests/bad_root17142026/11/11 23:29:45 INFO Received push request method=POST path=/api/pushes17152026/11/11 23:29:45 INFO Aborted multipart uploads count=1 kept=01716=== CONT TestPush_RejectsBadRequests/no_roots17172026/11/11 23:29:45 INFO Received push request method=POST path=/api/pushes1718=== CONT TestIsValidCachePath/narinfo1719--- PASS: TestPush_RejectsBadRequests (3.07s)1720 --- PASS: TestPush_RejectsBadRequests/root_not_in_objects (0.00s)1721 --- PASS: TestPush_RejectsBadRequests/no_objects (0.00s)1722 --- PASS: TestPush_RejectsBadRequests/bad_root (0.00s)1723 --- PASS: TestPush_RejectsBadRequests/no_roots (0.00s)1724=== CONT TestIsValidCachePath/index.html1725=== CONT TestIsValidCachePath/short_hash1726=== CONT TestIsValidCachePath/wrong_extension1727=== CONT TestIsValidCachePath/leading_slash1728=== CONT TestIsValidCachePath/empty1729=== CONT TestIsValidCachePath/random_path1730=== CONT TestIsValidCachePath/invalid_char_u1731=== CONT TestIsValidCachePath/invalid_char_e1732=== CONT TestIsValidCachePath/traversal_in_middle1733=== CONT TestIsValidCachePath/traversal_parent1734=== CONT TestIsValidCachePath/nar_uncompressed1735=== CONT TestIsValidCachePath/log1736=== CONT TestIsValidCachePath/realisation1737=== CONT TestIsValidCachePath/ls1738=== CONT TestIsValidCachePath/nix-cache-info1739=== CONT TestIsValidCachePath/nar_xz1740=== CONT TestIsValidCachePath/nar_bz21741=== CONT TestIsValidCachePath/nar_zst1742=== CONT TestIsValidCachePath/narinfo_all_nix_base32_chars1743--- PASS: TestIsValidCachePath (0.00s)1744 --- PASS: TestIsValidCachePath/narinfo (0.00s)1745 --- PASS: TestIsValidCachePath/index.html (0.00s)1746 --- PASS: TestIsValidCachePath/short_hash (0.00s)1747 --- PASS: TestIsValidCachePath/wrong_extension (0.00s)1748 --- PASS: TestIsValidCachePath/leading_slash (0.00s)1749 --- PASS: TestIsValidCachePath/empty (0.00s)1750 --- PASS: TestIsValidCachePath/random_path (0.00s)1751 --- PASS: TestIsValidCachePath/invalid_char_u (0.00s)1752 --- PASS: TestIsValidCachePath/invalid_char_e (0.00s)1753 --- PASS: TestIsValidCachePath/traversal_in_middle (0.00s)1754 --- PASS: TestIsValidCachePath/traversal_parent (0.00s)1755 --- PASS: TestIsValidCachePath/nar_uncompressed (0.00s)1756 --- PASS: TestIsValidCachePath/log (0.00s)1757 --- PASS: TestIsValidCachePath/realisation (0.00s)1758 --- PASS: TestIsValidCachePath/ls (0.00s)1759 --- PASS: TestIsValidCachePath/nix-cache-info (0.00s)1760 --- PASS: TestIsValidCachePath/nar_xz (0.00s)1761 --- PASS: TestIsValidCachePath/nar_bz2 (0.00s)1762 --- PASS: TestIsValidCachePath/nar_zst (0.00s)1763 --- PASS: TestIsValidCachePath/narinfo_all_nix_base32_chars (0.00s)1764=== CONT TestParseSingleRange/none1765=== CONT TestParseSingleRange/single_byte1766=== CONT TestParseSingleRange/suffix_exceeds_size1767=== CONT TestParseSingleRange/suffix1768=== CONT TestParseSingleRange/end_clamped_to_size1769=== CONT TestParseSingleRange/open-ended1770=== CONT TestParseSingleRange/closed1771=== CONT TestParseSingleRange/malformed_end_before_start1772=== CONT TestParseSingleRange/malformed_both_empty1773=== CONT TestParseSingleRange/malformed_no_dash1774=== CONT TestParseSingleRange/multi-range_ignored1775=== CONT TestParseSingleRange/start_past_EOF1776=== CONT TestParseSingleRange/unknown_unit1777=== CONT TestParseSingleRange/start_far_past_EOF1778--- PASS: TestParseSingleRange (0.00s)1779 --- PASS: TestParseSingleRange/none (0.00s)1780 --- PASS: TestParseSingleRange/single_byte (0.00s)1781 --- PASS: TestParseSingleRange/suffix_exceeds_size (0.00s)1782 --- PASS: TestParseSingleRange/suffix (0.00s)1783 --- PASS: TestParseSingleRange/end_clamped_to_size (0.00s)1784 --- PASS: TestParseSingleRange/open-ended (0.00s)1785 --- PASS: TestParseSingleRange/closed (0.00s)1786 --- PASS: TestParseSingleRange/malformed_end_before_start (0.00s)1787 --- PASS: TestParseSingleRange/malformed_both_empty (0.00s)1788 --- PASS: TestParseSingleRange/malformed_no_dash (0.00s)1789 --- PASS: TestParseSingleRange/multi-range_ignored (0.00s)1790 --- PASS: TestParseSingleRange/start_past_EOF (0.00s)1791 --- PASS: TestParseSingleRange/unknown_unit (0.00s)1792 --- PASS: TestParseSingleRange/start_far_past_EOF (0.00s)1793=== CONT TestPresentNotReportedWhileGCDeletesClosure/fresh1794--- PASS: TestMultipartCleanup (0.94s)1795=== CONT TestPresentNotReportedWhileGCDeletesClosure/stale17962026/11/11 23:29:45 INFO Received complete push request method=POST path=/api/pushes/1/complete1797=== CONT TestResolveDBConnectionString/flag_wins1798=== CONT TestResolveDBConnectionString/missing_file_is_an_error1799=== CONT TestResolveDBConnectionString/file_when_flag_empty1800=== CONT TestResolveDBConnectionString/PGHOST_allows_empty1801=== CONT TestResolveDBConnectionString/nothing_configured1802=== CONT TestForceGCDuringPushOffersSweptObject/before_pending_rows1803--- PASS: TestResolveDBConnectionString (0.00s)1804 --- PASS: TestResolveDBConnectionString/flag_wins (0.00s)1805 --- PASS: TestResolveDBConnectionString/missing_file_is_an_error (0.00s)1806 --- PASS: TestResolveDBConnectionString/file_when_flag_empty (0.00s)1807 --- PASS: TestResolveDBConnectionString/PGHOST_allows_empty (0.00s)1808 --- PASS: TestResolveDBConnectionString/nothing_configured (0.00s)18092026/11/11 23:29:45 INFO lead: released remote=192.0.2.1:1234 generation=21810--- PASS: TestLeadElectsOneAndHandsOver (3.74s)1811=== CONT TestForceGCDuringPushOffersSweptObject/after_presence_check18122026/11/11 23:29:46 INFO Completed push id=1 objects=4 duration=471.004052ms18132026/11/11 23:29:46 INFO All 1 paths already cached1814=== NAME TestClientIntegration1815 client_integration_test.go:318: Retrieved narinfo from S3:1816 StorePath: /build/TestClientIntegration985285290/002/store/vzh3kvpfkmjy1rlw8rwkgjbfm0w83i7d-test-file.txt1817 URL: nar/11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1.nar.zst1818 Compression: zstd1819 NarHash: sha256:11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk11820 NarSize: 1521821 References: 1822 CA: fixed:r:sha256:11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk11823 client_integration_test.go:319: Retrieved .ls file from S3 (compressed size: 77 bytes)18242026/11/11 23:29:46 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete1825 client_integration_test.go:319: Decompressed .ls content (64 bytes):1826 {"version":1,"root":{"type":"regular","size":39,"narOffset":96}}18272026/11/11 23:29:46 INFO Completed upload id=2 objects=2 duration=5.437292ms18282026/11/11 23:29:46 INFO Received uploads request method=POST path=/api/pending_closures1829--- PASS: TestMetricsCommit (1.50s)1830=== CONT TestGCEndsOnShutdown/before_the_run18312026/11/11 23:29:46 INFO Received uploads request method=POST path=/api/pending_closures18322026/11/11 23:29:46 INFO Uploading 2 paths to 127.0.0.1 (1 already cached)18332026/11/11 23:29:46 INFO Uploading i082vz0nknsxabbjlzl3w6gjmv8gi3rc-shared-dep (136B)18342026/11/11 23:29:46 INFO Uploading fw7vc9kz6glds5alar5z8bwchr69b8fx-b (216B)18352026/11/11 23:29:46 WARN Failed to register uploaded object key=qah3li0g7brqplcrw8yxn04q40b3w3c5.ls error="server returned 404: 404 page not found\n"18362026/11/11 23:29:46 WARN Failed to register uploaded object key=i082vz0nknsxabbjlzl3w6gjmv8gi3rc.ls error="server returned 404: 404 page not found\n"18372026/11/11 23:29:46 WARN Failed to register uploaded object key=fw7vc9kz6glds5alar5z8bwchr69b8fx.ls error="server returned 404: 404 page not found\n"18382026/11/11 23:29:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign18392026/11/11 23:29:46 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"18402026/11/11 23:29:46 WARN Failed to register uploaded object key=nar/14lkkk586z6872ydz70na0a3r9jyimsxvinzz24skzz2zlcdpsdh.nar.zst error="server returned 404: 404 page not found\n"18412026/11/11 23:29:46 INFO Signed narinfos id=2 count=218422026/11/11 23:29:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign18432026/11/11 23:29:46 INFO Signed narinfos id=1 count=218442026/11/11 23:29:46 INFO Uploading 4 narinfos18452026/11/11 23:29:46 WARN Failed to register uploaded object key=qah3li0g7brqplcrw8yxn04q40b3w3c5.narinfo error="server returned 404: 404 page not found\n"18462026/11/11 23:29:46 WARN Failed to register uploaded object key=fw7vc9kz6glds5alar5z8bwchr69b8fx.narinfo error="server returned 404: 404 page not found\n"18472026/11/11 23:29:46 WARN Failed to register uploaded object key=i082vz0nknsxabbjlzl3w6gjmv8gi3rc.narinfo error="server returned 404: 404 page not found\n"18482026/11/11 23:29:46 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete18492026/11/11 23:29:46 WARN Failed to register uploaded object key=i082vz0nknsxabbjlzl3w6gjmv8gi3rc.narinfo error="server returned 404: 404 page not found\n"18502026/11/11 23:29:46 INFO Completed upload id=1 objects=6 duration=10.999505ms18512026/11/11 23:29:46 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete18522026/11/11 23:29:46 INFO Completed upload id=2 objects=6 duration=2.162061ms18532026/11/11 23:29:46 INFO Upload complete. (1.551s)1854=== NAME TestClientFallsBackToClosures1855 client_pushes_test.go:112: Retrieved narinfo from S3:1856 StorePath: /build/TestClientFallsBackToClosures1148589486/001/store/i082vz0nknsxabbjlzl3w6gjmv8gi3rc-shared-dep1857 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1858 Compression: zstd1859 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821860 NarSize: 1361861 References: 1862 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n1863 client_pushes_test.go:112: Retrieved narinfo from S3:1864 StorePath: /build/TestClientFallsBackToClosures1148589486/001/store/qah3li0g7brqplcrw8yxn04q40b3w3c5-a1865 URL: nar/14lkkk586z6872ydz70na0a3r9jyimsxvinzz24skzz2zlcdpsdh.nar.zst1866 Compression: zstd1867 NarHash: sha256:14lkkk586z6872ydz70na0a3r9jyimsxvinzz24skzz2zlcdpsdh1868 NarSize: 2161869 References: /build/TestClientFallsBackToClosures1148589486/001/store/i082vz0nknsxabbjlzl3w6gjmv8gi3rc-shared-dep1870 CA: text:sha256:0haj0cvwfd7d286q9xxv05dzmbh6k5lddxvil0qa8s8ch1dhlv481871 client_pushes_test.go:112: Retrieved narinfo from S3:1872 StorePath: /build/TestClientFallsBackToClosures1148589486/001/store/fw7vc9kz6glds5alar5z8bwchr69b8fx-b1873 URL: nar/14lkkk586z6872ydz70na0a3r9jyimsxvinzz24skzz2zlcdpsdh.nar.zst1874 Compression: zstd1875 NarHash: sha256:14lkkk586z6872ydz70na0a3r9jyimsxvinzz24skzz2zlcdpsdh1876 NarSize: 2161877 References: /build/TestClientFallsBackToClosures1148589486/001/store/i082vz0nknsxabbjlzl3w6gjmv8gi3rc-shared-dep1878 CA: text:sha256:0haj0cvwfd7d286q9xxv05dzmbh6k5lddxvil0qa8s8ch1dhlv4818792026/11/11 23:29:46 INFO Received uploads request method=POST path=/api/pending_closures1880--- PASS: TestClientFallsBackToClosures (4.28s)1881=== CONT TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark18822026/11/11 23:29:46 INFO Received uploads request method=POST path=/api/pending_closures18832026/11/11 23:29:46 INFO Received uploads request method=POST path=/api/pending_closures18842026/11/11 23:29:46 INFO Received uploads request method=POST path=/api/pending_closures18852026/11/11 23:29:46 INFO Received uploads request method=POST path=/api/pending_closures18862026/11/11 23:29:46 INFO Received uploads request method=POST path=/api/pending_closures18872026/11/11 23:29:46 INFO Received uploads request method=POST path=/api/pending_closures18882026/11/11 23:29:46 INFO Received uploads request method=POST path=/api/pending_closures18892026/11/11 23:29:46 INFO Received cleanup request method=DELETE path=/api/pending_closures18902026/11/11 23:29:46 INFO Received uploads request method=POST path=/api/pending_closures18912026/11/11 23:29:46 INFO Aborted multipart uploads count=0 kept=018922026/11/11 23:29:46 INFO lead: released remote=192.0.2.1:1234 generation=31893--- PASS: TestLeadUnannouncedStandbyYieldsToIncumbent (4.67s)1894=== CONT TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete18952026/11/11 23:29:46 INFO Received complete multipart upload request method=POST path=/api/multipart/complete18962026/11/11 23:29:46 INFO Received complete multipart upload request method=POST path=/api/multipart/complete18972026/11/11 23:29:46 WARN Force mode enabled - objects will be deleted immediately without grace period1898--- PASS: TestUploadHandlersRejectOversizedBody (0.30s)1899 --- PASS: TestUploadHandlersRejectOversizedBody/request_more_parts (0.05s)1900 --- PASS: TestUploadHandlersRejectOversizedBody/complete_multipart (0.06s)1901 --- PASS: TestUploadHandlersRejectOversizedBody/create_pending_closure (1.42s)1902=== CONT TestCacheConfigHandler/full_config,_no_issuer1903=== CONT TestCacheConfigHandler/no_signing_keys1904=== CONT TestCacheConfigHandler/no_cache_url_configured1905=== CONT TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1906=== CONT TestClientErrorHandling/InvalidStorePath1907--- PASS: TestPresentNotReportedWhileGCDeletesClosure (0.00s)1908 --- PASS: TestPresentNotReportedWhileGCDeletesClosure/stale (1.34s)1909 --- PASS: TestPresentNotReportedWhileGCDeletesClosure/fresh (1.34s)1910--- PASS: TestCacheConfigHandler (0.00s)1911 --- PASS: TestCacheConfigHandler/full_config,_no_issuer (0.00s)1912 --- PASS: TestCacheConfigHandler/no_signing_keys (0.00s)1913 --- PASS: TestCacheConfigHandler/no_cache_url_configured (0.00s)1914 --- PASS: TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator (0.00s)1915=== CONT TestClientErrorHandling/ServerNotAvailable1916=== CONT TestClientErrorHandling/InvalidAuthToken19172026/11/11 23:29:47 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000200000000000000000000.nar.zst upload_id=OTkxZDgwZGMtZjIyMi00YWYyLTg3M2YtYjY3NGY0ZWU0MWU0LjVhN2E1Mzg0LTJhMzYtNGRlMi04NzY0LTc2MTIzMGVkMTJmOXgxNzk0NDM5Nzg2NTQxNzU0MjM4 parts=1019182026/11/11 23:29:47 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000500000000000000000000.nar.zst upload_id=OTkxZDgwZGMtZjIyMi00YWYyLTg3M2YtYjY3NGY0ZWU0MWU0LjFkOTU0YWZhLTRjNTUtNGFhNi04MTMyLTAxMmJkZTc0ZGQ2ZXgxNzk0NDM5Nzg2NTYxOTM2MDMw parts=1019192026/11/11 23:29:47 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete1920--- PASS: TestMetricsInventory (2.42s)1921=== CONT TestService_RequireScope_OIDC/builder_may_write19222026/11/11 23:29:47 INFO Completed upload id=1 objects=3 duration=5.413911ms19232026/11/11 23:29:47 INFO Received uploads request method=POST path=/api/pending_closures1924=== CONT TestService_RequireScope_OIDC/reader_may_read1925=== CONT TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1926=== CONT TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read19272026/11/11 23:29:47 WARN Authentication failed token_preview=wrong token_length=5 oidc_error="no provider could verify the token (signature or issuer mismatch)" oidc_provider="" tried_providers=[test]1928=== CONT TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1929=== CONT TestService_RequireScope_OIDC/reader_may_read_with_netrc1930=== CONT TestService_RequireScope_OIDC/anonymous_may_not_read1931=== CONT TestService_RequireScope_OIDC/writer_implies_read1932=== CONT TestService_RequireScope_OIDC/ops_may_admin1933=== NAME TestNARDeduplicationMetadataUploadBug1934 metadata_upload_test.go:48: First store path: /build/TestNARDeduplicationMetadataUploadBug1489346372/001/store/zlkbfgnnrzb83ayfh3cwhkma9637xbnj-file1.txt1935=== CONT TestService_RequireScope_OIDC/reader_may_not_write1936=== CONT TestService_RequireScope_OIDC/ops_may_not_write1937=== CONT TestService_RequireScope_OIDC/static_token_may_write1938=== CONT TestService_RequireScope_OIDC/builder_may_not_admin1939=== CONT TestService_RequireScope_OIDC/static_token_may_admin1940=== CONT TestService_AuthMiddleware_OIDC/valid_OIDC_token1941--- PASS: TestService_RequireScope_OIDC (0.61s)1942 --- PASS: TestService_RequireScope_OIDC/builder_may_write (0.01s)1943 --- PASS: TestService_RequireScope_OIDC/reader_may_read (0.00s)1944 --- PASS: TestService_RequireScope_OIDC/empty_netrc_password_may_not_read (0.00s)1945 --- PASS: TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read (0.00s)1946 --- PASS: TestService_RequireScope_OIDC/static_token_may_admin_with_netrc (0.00s)1947 --- PASS: TestService_RequireScope_OIDC/reader_may_read_with_netrc (0.00s)1948 --- PASS: TestService_RequireScope_OIDC/anonymous_may_not_read (0.00s)1949 --- PASS: TestService_RequireScope_OIDC/writer_implies_read (0.00s)1950 --- PASS: TestService_RequireScope_OIDC/ops_may_admin (0.00s)1951 --- PASS: TestService_RequireScope_OIDC/reader_may_not_write (0.00s)1952 --- PASS: TestService_RequireScope_OIDC/ops_may_not_write (0.00s)1953 --- PASS: TestService_RequireScope_OIDC/static_token_may_write (0.00s)1954 --- PASS: TestService_RequireScope_OIDC/builder_may_not_admin (0.00s)1955 --- PASS: TestService_RequireScope_OIDC/static_token_may_admin (0.00s)1956=== CONT TestService_AuthMiddleware_OIDC/malformed_token_rejected19572026/11/11 23:29:47 WARN Authentication failed token_preview=not-a-valid-jwt token_length=15 oidc_error="no provider could verify the token (signature or issuer mismatch)" oidc_provider="" tried_providers=[test]1958=== CONT TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1959--- PASS: TestPendingClosureWriteTimeout (0.00s)1960 --- PASS: TestPendingClosureWriteTimeout/empty (0.00s)1961 --- PASS: TestPendingClosureWriteTimeout/670k_objects (0.00s)1962 --- PASS: TestPendingClosureWriteTimeout/400_objects (0.00s)1963 --- PASS: TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline (1.49s)1964=== CONT TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1965=== CONT TestServerTLSConfig/no_client_CA1966=== CONT TestServerTLSConfig/not_a_PEM_file19672026/11/11 23:29:47 WARN Authentication failed token_preview=eyJhbGciOi...vtEA5gqzRQ token_length=701 oidc_error="no rule matched: claim \"repository_owner\" value [otherorg] not in allowed patterns [myorg]" oidc_provider=test tried_providers=[test]1968=== CONT TestServerTLSConfig/missing_CA_file1969--- PASS: TestService_AuthMiddleware_OIDC (0.51s)1970 --- PASS: TestService_AuthMiddleware_OIDC/valid_OIDC_token (0.01s)1971 --- PASS: TestService_AuthMiddleware_OIDC/malformed_token_rejected (0.00s)1972 --- PASS: TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured (0.00s)1973 --- PASS: TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected (0.00s)1974--- PASS: TestServerTLSConfig (0.00s)1975 --- PASS: TestServerTLSConfig/no_client_CA (0.00s)1976 --- PASS: TestServerTLSConfig/missing_CA_file (0.00s)1977 --- PASS: TestServerTLSConfig/not_a_PEM_file (0.00s)19782026/11/11 23:29:47 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete19792026/11/11 23:29:47 INFO Completed upload id=1 objects=3 duration=4.662064ms19802026/11/11 23:29:47 INFO Received uploads request method=POST path=/api/pending_closures19812026/11/11 23:29:47 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=1 objects-deleted-after-grace-period=1 objects-failed-to-delete=01982--- PASS: TestDeduplicatedObjectsRecordedAsPending (1.78s)19832026/11/11 23:29:47 INFO Vacuumed table table=pending_closures19842026/11/11 23:29:47 INFO Vacuumed table table=pending_objects19852026/11/11 23:29:47 INFO Vacuumed table table=multipart_uploads19862026/11/11 23:29:47 INFO Vacuumed table table=closures19872026/11/11 23:29:47 INFO Vacuumed table table=objects19882026/11/11 23:29:47 INFO Received uploads request method=POST path=/api/pending_closures19892026/11/11 23:29:47 INFO Received uploads request method=POST path=/api/pending_closures19902026/11/11 23:29:47 INFO Object in database but missing from S3 key=nar/dadb44fdadb44fdadb44fdadb44f000500000000000000000000.nar.zst19912026/11/11 23:29:47 WARN Found objects in DB but missing from S3, will re-upload count=119922026/11/11 23:29:47 INFO Aborted multipart uploads count=0 kept=019932026/11/11 23:29:47 INFO Received push request method=POST path=/api/pushes19942026/11/11 23:29:47 INFO Received complete multipart upload request method=POST path=/api/multipart/complete19952026/11/11 23:29:47 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)19962026/11/11 23:29:47 INFO Uploading asdp6l9ac6z0iakz0qvl291vchfad4qh-unpinned-file.txt (128B)19972026/11/11 23:29:47 WARN Failed to register uploaded object key=asdp6l9ac6z0iakz0qvl291vchfad4qh.ls error="server returned 404: 404 page not found\n"19982026/11/11 23:29:47 WARN Failed to register uploaded object key=nar/0gh0wdnr6gmk4r5sfzzh12isnl14rgmkrwy1ws41hpsrlndclg5h.nar.zst error="server returned 404: 404 page not found\n"19992026/11/11 23:29:47 INFO Received sign narinfos request method=POST path=/api/pushes/2/sign20002026/11/11 23:29:47 INFO Signed narinfos id=2 count=1
loading 429 hidden lines…
24302026/11/11 23:30:05 INFO Vacuumed table table=pending_closures24312026/11/11 23:30:05 INFO Received uploads request method=POST path=/24322026/11/11 23:30:05 INFO Received uploads request method=POST path=/24332026/11/11 23:30:05 INFO Vacuumed table table=multipart_uploads24342026/11/11 23:30:05 INFO Vacuumed table table=closures24352026/11/11 23:30:05 INFO Vacuumed table table=objects24362026/11/11 23:30:05 INFO Vacuumed table table=pending_objects24372026/11/11 23:30:05 INFO Vacuumed table table=multipart_uploads24382026/11/11 23:30:05 INFO Vacuumed table table=closures24392026/11/11 23:30:05 INFO Vacuumed table table=objects24402026/11/11 23:30:05 INFO Received push request method=POST path=/24412026/11/11 23:30:06 INFO Received uploads request method=POST path=/24422026/11/11 23:30:06 INFO Received uploads request method=POST path=/24432026/11/11 23:30:06 INFO Received push request method=POST path=/24442026/11/11 23:30:06 INFO Received complete upload request method=POST path=/24452026/11/11 23:30:06 INFO Received push request method=POST path=/24462026/11/11 23:30:06 INFO Completed upload id=6 objects=7 duration=2.838087ms24472026/11/11 23:30:06 INFO Received complete upload request method=POST path=/24482026/11/11 23:30:06 INFO Received push request method=POST path=/24492026/11/11 23:30:06 INFO Received complete push request method=POST path=/24502026/11/11 23:30:06 INFO Completed upload id=1 objects=9 duration=6.501642ms24512026/11/11 23:30:06 INFO Received complete push request method=POST path=/24522026/11/11 23:30:06 INFO Aborted multipart uploads count=0 kept=024532026/11/11 23:30:06 INFO Completed push id=1 objects=5 duration=8.274879ms24542026/11/11 23:30:06 INFO Completed push id=3 objects=5 duration=6.439662ms24552026/11/11 23:30:06 INFO Received push request method=POST path=/24562026/11/11 23:30:06 INFO Received uploads request method=POST path=/24572026/11/11 23:30:06 INFO Received push request method=POST path=/24582026/11/11 23:30:06 INFO Aborted multipart uploads count=0 kept=024592026/11/11 23:30:06 INFO Received uploads request method=POST path=/24602026/11/11 23:30:06 INFO Received push request method=POST path=/24612026/11/11 23:30:06 INFO Received push request method=POST path=/24622026/11/11 23:30:06 INFO Received push request method=POST path=/24632026/11/11 23:30:06 INFO Received uploads request method=POST path=/24642026/11/11 23:30:06 INFO Received push request method=POST path=/24652026/11/11 23:30:06 INFO Received uploads request method=POST path=/24662026/11/11 23:30:06 INFO Received complete push request method=POST path=/24672026/11/11 23:30:06 INFO Received uploads request method=POST path=/24682026/11/11 23:30:06 INFO Received complete upload request method=POST path=/24692026/11/11 23:30:06 INFO Completed push id=6 objects=8 duration=3.985858ms24702026/11/11 23:30:06 INFO Completed upload id=4 objects=7 duration=1.987419ms24712026/11/11 23:30:06 INFO Received push request method=POST path=/24722026/11/11 23:30:06 INFO Received push request method=POST path=/24732026/11/11 23:30:06 INFO Received complete upload request method=POST path=/24742026/11/11 23:30:06 INFO Aborted multipart uploads count=0 kept=024752026/11/11 23:30:06 INFO Received uploads request method=POST path=/24762026/11/11 23:30:06 INFO Received uploads request method=POST path=/24772026/11/11 23:30:06 INFO Completed upload id=2 objects=26 duration=6.037917ms24782026/11/11 23:30:06 INFO Received complete upload request method=POST path=/24792026/11/11 23:30:06 INFO Completed upload id=5 objects=5 duration=1.647576ms24802026/11/11 23:30:06 INFO Received uploads request method=POST path=/24812026/11/11 23:30:06 INFO Received uploads request method=POST path=/24822026/11/11 23:30:06 INFO Received uploads request method=POST path=/24832026/11/11 23:30:06 INFO Received uploads request method=POST path=/24842026/11/11 23:30:06 INFO Received push request method=POST path=/24852026/11/11 23:30:06 INFO Received push request method=POST path=/24862026/11/11 23:30:06 INFO Received uploads request method=POST path=/24872026/11/11 23:30:06 INFO Received complete upload request method=POST path=/24882026/11/11 23:30:06 INFO Completed upload id=3 objects=24 duration=1.672455ms24892026/11/11 23:30:06 INFO Received uploads request method=POST path=/24902026/11/11 23:30:06 INFO Received complete upload request method=POST path=/24912026/11/11 23:30:06 INFO Received complete push request method=POST path=/24922026/11/11 23:30:06 INFO Completed upload id=5 objects=7 duration=2.638605ms24932026/11/11 23:30:06 INFO Received push request method=POST path=/24942026/11/11 23:30:06 INFO Received push request method=POST path=/24952026/11/11 23:30:06 INFO Completed push id=1 objects=6 duration=8.191658ms24962026/11/11 23:30:06 INFO Received push request method=POST path=/24972026/11/11 23:30:06 INFO Received complete push request method=POST path=/24982026/11/11 23:30:06 INFO Received complete upload request method=POST path=/24992026/11/11 23:30:06 INFO Completed upload id=7 objects=4 duration=1.793617ms25002026/11/11 23:30:06 INFO Completed push id=2 objects=25 duration=4.692865ms25012026/11/11 23:30:06 INFO Received push request method=POST path=/25022026/11/11 23:30:06 INFO Aborted multipart uploads count=0 kept=025032026/11/11 23:30:06 INFO Received complete upload request method=POST path=/25042026/11/11 23:30:06 INFO Completed upload id=6 objects=9 duration=1.777517ms25052026/11/11 23:30:06 WARN Force mode enabled - objects will be deleted immediately without grace period25062026/11/11 23:30:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=025072026/11/11 23:30:06 INFO Vacuumed table table=pending_closures25082026/11/11 23:30:06 INFO Received push request method=POST path=/25092026/11/11 23:30:06 INFO Vacuumed table table=pending_objects25102026/11/11 23:30:06 INFO Vacuumed table table=multipart_uploads25112026/11/11 23:30:06 INFO Vacuumed table table=closures25122026/11/11 23:30:06 INFO Vacuumed table table=objects25132026/11/11 23:30:06 INFO Received complete push request method=POST path=/25142026/11/11 23:30:06 INFO Completed push id=2 objects=4 duration=2.160101ms25152026/11/11 23:30:06 INFO Received uploads request method=POST path=/25162026/11/11 23:30:06 INFO Received push request method=POST path=/25172026/11/11 23:30:06 INFO Received complete push request method=POST path=/25182026/11/11 23:30:06 INFO Aborted multipart uploads count=0 kept=025192026/11/11 23:30:06 INFO Aborted multipart uploads count=0 kept=025202026/11/11 23:30:06 INFO Completed push id=1 objects=7 duration=7.159389ms25212026/11/11 23:30:06 INFO Received uploads request method=POST path=/25222026/11/11 23:30:06 INFO Received uploads request method=POST path=/25232026/11/11 23:30:06 INFO Received uploads request method=POST path=/25242026/11/11 23:30:06 INFO Aborted multipart uploads count=0 kept=025252026/11/11 23:30:06 INFO Received uploads request method=POST path=/25262026/11/11 23:30:06 INFO Received complete push request method=POST path=/25272026/11/11 23:30:06 WARN Force mode enabled - objects will be deleted immediately without grace period25282026/11/11 23:30:06 INFO Completed push id=1 objects=10 duration=6.886226ms25292026/11/11 23:30:06 WARN Force mode enabled - objects will be deleted immediately without grace period25302026/11/11 23:30:06 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=4 objects-failed-to-delete=025312026/11/11 23:30:06 INFO Vacuumed table table=pending_closures25322026/11/11 23:30:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=025332026/11/11 23:30:06 INFO Vacuumed table table=pending_objects25342026/11/11 23:30:06 INFO Vacuumed table table=multipart_uploads25352026/11/11 23:30:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=025362026/11/11 23:30:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=025372026/11/11 23:30:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=025382026/11/11 23:30:06 INFO Vacuumed table table=closures25392026/11/11 23:30:06 INFO Vacuumed table table=pending_closures25402026/11/11 23:30:06 INFO Vacuumed table table=pending_closures25412026/11/11 23:30:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=025422026/11/11 23:30:06 INFO Vacuumed table table=pending_objects25432026/11/11 23:30:06 INFO Vacuumed table table=pending_objects25442026/11/11 23:30:06 INFO Vacuumed table table=multipart_uploads25452026/11/11 23:30:06 INFO Vacuumed table table=pending_closures25462026/11/11 23:30:06 INFO Vacuumed table table=multipart_uploads25472026/11/11 23:30:06 INFO Vacuumed table table=closures25482026/11/11 23:30:06 INFO Vacuumed table table=objects25492026/11/11 23:30:06 INFO Vacuumed table table=objects25502026/11/11 23:30:06 INFO Vacuumed table table=closures25512026/11/11 23:30:06 INFO Received uploads request method=POST path=/25522026/11/11 23:30:06 INFO Received push request method=POST path=/25532026/11/11 23:30:06 INFO Vacuumed table table=objects25542026/11/11 23:30:06 INFO Vacuumed table table=pending_objects25552026/11/11 23:30:06 INFO Vacuumed table table=pending_closures25562026/11/11 23:30:06 INFO Vacuumed table table=multipart_uploads25572026/11/11 23:30:06 INFO Vacuumed table table=pending_closures25582026/11/11 23:30:06 INFO Vacuumed table table=closures25592026/11/11 23:30:06 INFO Vacuumed table table=objects25602026/11/11 23:30:06 INFO Vacuumed table table=pending_objects25612026/11/11 23:30:06 INFO Vacuumed table table=multipart_uploads25622026/11/11 23:30:06 INFO Vacuumed table table=pending_objects25632026/11/11 23:30:06 INFO Vacuumed table table=closures25642026/11/11 23:30:06 INFO Vacuumed table table=multipart_uploads25652026/11/11 23:30:06 INFO Vacuumed table table=objects25662026/11/11 23:30:06 INFO Vacuumed table table=closures25672026/11/11 23:30:06 INFO Received push request method=POST path=/25682026/11/11 23:30:06 INFO Received push request method=POST path=/25692026/11/11 23:30:06 INFO Received complete upload request method=POST path=/25702026/11/11 23:30:06 INFO Vacuumed table table=objects25712026/11/11 23:30:06 INFO Received uploads request method=POST path=/25722026/11/11 23:30:06 INFO Aborted multipart uploads count=0 kept=025732026/11/11 23:30:06 INFO Completed upload id=10 objects=6 duration=2.934928ms25742026/11/11 23:30:06 INFO Received push request method=POST path=/25752026/11/11 23:30:06 WARN Force mode enabled - objects will be deleted immediately without grace period25762026/11/11 23:30:06 INFO Received push request method=POST path=/25772026/11/11 23:30:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=025782026/11/11 23:30:06 INFO Vacuumed table table=pending_closures25792026/11/11 23:30:06 INFO Received uploads request method=POST path=/25802026/11/11 23:30:06 INFO Vacuumed table table=pending_objects25812026/11/11 23:30:06 INFO Vacuumed table table=multipart_uploads25822026/11/11 23:30:06 INFO Vacuumed table table=closures25832026/11/11 23:30:06 INFO Vacuumed table table=objects25842026/11/11 23:30:06 INFO Aborted multipart uploads count=0 kept=025852026/11/11 23:30:06 INFO Received uploads request method=POST path=/25862026/11/11 23:30:06 INFO Received push request method=POST path=/25872026/11/11 23:30:06 WARN Force mode enabled - objects will be deleted immediately without grace period25882026/11/11 23:30:06 INFO Received push request method=POST path=/25892026/11/11 23:30:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=025902026/11/11 23:30:06 INFO Received push request method=POST path=/25912026/11/11 23:30:06 INFO Vacuumed table table=pending_closures25922026/11/11 23:30:06 INFO Received push request method=POST path=/25932026/11/11 23:30:06 INFO Received push request method=POST path=/25942026/11/11 23:30:06 INFO Vacuumed table table=pending_objects25952026/11/11 23:30:06 INFO Aborted multipart uploads count=0 kept=025962026/11/11 23:30:06 INFO Aborted multipart uploads count=0 kept=025972026/11/11 23:30:06 INFO Vacuumed table table=multipart_uploads25982026/11/11 23:30:06 INFO Vacuumed table table=closures25992026/11/11 23:30:06 INFO Aborted multipart uploads count=0 kept=026002026/11/11 23:30:06 INFO Vacuumed table table=objects26012026/11/11 23:30:06 WARN Force mode enabled - objects will be deleted immediately without grace period26022026/11/11 23:30:06 WARN Force mode enabled - objects will be deleted immediately without grace period26032026/11/11 23:30:06 WARN Force mode enabled - objects will be deleted immediately without grace period26042026/11/11 23:30:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=026052026/11/11 23:30:06 INFO Received uploads request method=POST path=/26062026/11/11 23:30:06 INFO Vacuumed table table=pending_closures26072026/11/11 23:30:06 INFO Received push request method=POST path=/26082026/11/11 23:30:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=026092026/11/11 23:30:06 INFO Vacuumed table table=pending_objects26102026/11/11 23:30:06 INFO Vacuumed table table=multipart_uploads26112026/11/11 23:30:06 INFO Vacuumed table table=closures26122026/11/11 23:30:06 INFO Vacuumed table table=pending_closures26132026/11/11 23:30:06 INFO Vacuumed table table=objects26142026/11/11 23:30:06 INFO Received complete push request method=POST path=/26152026/11/11 23:30:06 INFO Vacuumed table table=pending_objects26162026/11/11 23:30:06 INFO Vacuumed table table=multipart_uploads26172026/11/11 23:30:06 INFO Vacuumed table table=closures26182026/11/11 23:30:06 INFO Vacuumed table table=objects26192026/11/11 23:30:06 INFO Aborted multipart uploads count=0 kept=026202026/11/11 23:30:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=026212026/11/11 23:30:06 INFO Received push request method=POST path=/26222026/11/11 23:30:06 INFO Received uploads request method=POST path=/26232026/11/11 23:30:06 INFO Received push request method=POST path=/26242026/11/11 23:30:06 INFO Vacuumed table table=pending_closures26252026/11/11 23:30:06 INFO Completed push id=3 objects=8 duration=6.775684ms26262026/11/11 23:30:06 INFO Aborted multipart uploads count=0 kept=026272026/11/11 23:30:06 INFO Vacuumed table table=pending_objects26282026/11/11 23:30:06 INFO Vacuumed table table=multipart_uploads26292026/11/11 23:30:06 INFO Received complete push request method=POST path=/26302026/11/11 23:30:06 INFO Aborted multipart uploads count=0 kept=026312026/11/11 23:30:06 INFO Vacuumed table table=closures26322026/11/11 23:30:06 INFO Received push request method=POST path=/26332026/11/11 23:30:06 INFO Vacuumed table table=objects26342026/11/11 23:30:06 INFO Completed push id=7 objects=20 duration=2.227061ms26352026/11/11 23:30:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=026362026/11/11 23:30:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=026372026/11/11 23:30:06 INFO Vacuumed table table=pending_closures26382026/11/11 23:30:06 INFO Aborted multipart uploads count=0 kept=026392026/11/11 23:30:06 INFO Received push request method=POST path=/26402026/11/11 23:30:06 INFO Vacuumed table table=pending_objects26412026/11/11 23:30:06 INFO Received uploads request method=POST path=/26422026/11/11 23:30:06 INFO Vacuumed table table=multipart_uploads26432026/11/11 23:30:06 INFO Vacuumed table table=pending_closures26442026/11/11 23:30:06 INFO Vacuumed table table=closures26452026/11/11 23:30:06 INFO Received complete upload request method=POST path=/26462026/11/11 23:30:06 INFO Vacuumed table table=objects26472026/11/11 23:30:06 INFO Vacuumed table table=pending_objects26482026/11/11 23:30:06 INFO Vacuumed table table=multipart_uploads26492026/11/11 23:30:06 INFO Received push request method=POST path=/26502026/11/11 23:30:06 INFO Vacuumed table table=closures26512026/11/11 23:30:06 INFO Completed upload id=3 objects=5 duration=2.404662ms26522026/11/11 23:30:06 INFO Received complete upload request method=POST path=/26532026/11/11 23:30:06 INFO Received uploads request method=POST path=/26542026/11/11 23:30:06 INFO Completed upload id=3 objects=5 duration=2.866127ms26552026/11/11 23:30:06 INFO Vacuumed table table=objects26562026/11/11 23:30:06 INFO Received uploads request method=POST path=/26572026/11/11 23:30:06 INFO Received push request method=POST path=/26582026/11/11 23:30:06 INFO Aborted multipart uploads count=0 kept=026592026/11/11 23:30:06 INFO Received push request method=POST path=/26602026/11/11 23:30:06 INFO Received push request method=POST path=/26612026/11/11 23:30:06 INFO Aborted multipart uploads count=0 kept=026622026/11/11 23:30:06 INFO Aborted multipart uploads count=0 kept=026632026/11/11 23:30:06 INFO Received complete upload request method=POST path=/26642026/11/11 23:30:06 INFO Aborted multipart uploads count=0 kept=026652026/11/11 23:30:06 INFO Aborted multipart uploads count=0 kept=026662026/11/11 23:30:06 INFO Received complete push request method=POST path=/26672026/11/11 23:30:06 INFO Received push request method=POST path=/26682026/11/11 23:30:06 INFO Completed upload id=2 objects=7 duration=5.348131ms26692026/11/11 23:30:06 INFO Received push request method=POST path=/26702026/11/11 23:30:06 INFO Completed push id=4 objects=7 duration=1.679056ms26712026/11/11 23:30:06 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=026722026/11/11 23:30:06 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=026732026/11/11 23:30:06 INFO Received complete upload request method=POST path=/26742026/11/11 23:30:06 INFO Vacuumed table table=pending_closures26752026/11/11 23:30:06 INFO Vacuumed table table=pending_closures26762026/11/11 23:30:06 INFO Received complete push request method=POST path=/26772026/11/11 23:30:06 INFO Vacuumed table table=pending_objects26782026/11/11 23:30:06 INFO Vacuumed table table=multipart_uploads26792026/11/11 23:30:06 INFO Vacuumed table table=pending_objects26802026/11/11 23:30:06 INFO Vacuumed table table=closures26812026/11/11 23:30:06 INFO Vacuumed table table=multipart_uploads26822026/11/11 23:30:06 INFO Vacuumed table table=objects26832026/11/11 23:30:06 INFO Completed upload id=3 objects=6 duration=5.740455ms26842026/11/11 23:30:06 INFO Vacuumed table table=closures26852026/11/11 23:30:06 INFO Vacuumed table table=objects26862026/11/11 23:30:06 INFO Completed push id=1 objects=7 duration=6.186399ms26872026/11/11 23:30:06 INFO Received uploads request method=POST path=/26882026/11/11 23:30:06 INFO Received uploads request method=POST path=/26892026/11/11 23:30:06 INFO Received uploads request method=POST path=/26902026/11/11 23:30:06 INFO Received push request method=POST path=/26912026/11/11 23:30:06 INFO Aborted multipart uploads count=0 kept=026922026/11/11 23:30:06 INFO Received complete push request method=POST path=/26932026/11/11 23:30:06 INFO Received complete push request method=POST path=/26942026/11/11 23:30:06 INFO Completed push id=4 objects=3 duration=1.923298ms26952026/11/11 23:30:06 INFO Completed push id=1 objects=2 duration=3.366192ms26962026/11/11 23:30:06 INFO Received complete push request method=POST path=/26972026/11/11 23:30:06 INFO Received uploads request method=POST path=/26982026/11/11 23:30:06 INFO Completed push id=5 objects=3 duration=3.288771ms26992026/11/11 23:30:06 INFO Aborted multipart uploads count=0 kept=027002026/11/11 23:30:06 INFO Received uploads request method=POST path=/27012026/11/11 23:30:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027022026/11/11 23:30:06 INFO Received push request method=POST path=/27032026/11/11 23:30:06 INFO Received complete upload request method=POST path=/27042026/11/11 23:30:06 INFO Vacuumed table table=pending_closures27052026/11/11 23:30:06 INFO Vacuumed table table=pending_objects27062026/11/11 23:30:06 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.narinfo error="Access Denied."27072026/11/11 23:30:06 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."27082026/11/11 23:30:06 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.narinfo error="Access Denied."27092026/11/11 23:30:06 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."27102026/11/11 23:30:06 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."27112026/11/11 23:30:06 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."27122026/11/11 23:30:06 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."27132026/11/11 23:30:06 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."27142026/11/11 23:30:06 ERROR failed to remove object object=nar/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.nar.zst error="Access Denied."27152026/11/11 23:30:06 ERROR failed to remove object object=nar/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.nar.zst error="Access Denied."27162026/11/11 23:30:06 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."27172026/11/11 23:30:06 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."27182026/11/11 23:30:06 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."27192026/11/11 23:30:06 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."27202026/11/11 23:30:06 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."27212026/11/11 23:30:06 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."27222026/11/11 23:30:06 INFO Vacuumed table table=multipart_uploads27232026/11/11 23:30:06 INFO Completed upload id=11 objects=6 duration=2.248262ms27242026/11/11 23:30:06 INFO Vacuumed table table=closures27252026/11/11 23:30:06 INFO Aborted multipart uploads count=0 kept=027262026/11/11 23:30:06 INFO Vacuumed table table=objects27272026/11/11 23:30:06 INFO Received complete push request method=POST path=/27282026/11/11 23:30:06 INFO Received complete upload request method=POST path=/27292026/11/11 23:30:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027302026/11/11 23:30:06 INFO Completed push id=2 objects=10 duration=2.708626ms27312026/11/11 23:30:06 INFO Vacuumed table table=pending_closures27322026/11/11 23:30:06 INFO Received complete upload request method=POST path=/27332026/11/11 23:30:06 INFO Vacuumed table table=pending_objects27342026/11/11 23:30:06 INFO Vacuumed table table=multipart_uploads27352026/11/11 23:30:06 INFO Received uploads request method=POST path=/27362026/11/11 23:30:06 INFO Vacuumed table table=closures27372026/11/11 23:30:06 INFO Completed upload id=2 objects=10 duration=3.429273ms27382026/11/11 23:30:06 INFO Completed upload id=1 objects=15 duration=6.587923ms27392026/11/11 23:30:06 INFO Received push request method=POST path=/27402026/11/11 23:30:06 INFO Aborted multipart uploads count=0 kept=027412026/11/11 23:30:06 INFO Vacuumed table table=objects27422026/11/11 23:30:06 INFO Received complete push request method=POST path=/27432026/11/11 23:30:06 INFO Received complete push request method=POST path=/27442026/11/11 23:30:06 INFO Completed push id=4 objects=4 duration=3.346292ms27452026/11/11 23:30:06 INFO Completed push id=6 objects=7 duration=2.714086ms27462026/11/11 23:30:06 INFO Received push request method=POST path=/27472026/11/11 23:30:06 INFO Received uploads request method=POST path=/27482026/11/11 23:30:06 INFO Received push request method=POST path=/27492026/11/11 23:30:06 INFO Received uploads request method=POST path=/27502026/11/11 23:30:06 INFO Received push request method=POST path=/27512026/11/11 23:30:06 INFO Aborted multipart uploads count=0 kept=027522026/11/11 23:30:06 INFO Received push request method=POST path=/27532026/11/11 23:30:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027542026/11/11 23:30:06 INFO Vacuumed table table=pending_closures27552026/11/11 23:30:06 INFO Vacuumed table table=pending_objects27562026/11/11 23:30:06 INFO Vacuumed table table=multipart_uploads27572026/11/11 23:30:06 INFO Vacuumed table table=closures27582026/11/11 23:30:06 INFO Received push request method=POST path=/27592026/11/11 23:30:06 INFO Vacuumed table table=objects27602026/11/11 23:30:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=16 objects-failed-to-delete=027612026/11/11 23:30:06 INFO Vacuumed table table=pending_closures27622026/11/11 23:30:06 INFO Vacuumed table table=pending_objects27632026/11/11 23:30:06 INFO Vacuumed table table=multipart_uploads27642026/11/11 23:30:06 INFO Received push request method=POST path=/27652026/11/11 23:30:06 INFO Vacuumed table table=closures27662026/11/11 23:30:06 INFO Vacuumed table table=objects27672026/11/11 23:30:06 INFO Received complete push request method=POST path=/27682026/11/11 23:30:06 INFO Received uploads request method=POST path=/27692026/11/11 23:30:06 INFO Received complete push request method=POST path=/27702026/11/11 23:30:06 INFO Completed push id=8 objects=6 duration=1.717817ms27712026/11/11 23:30:06 INFO Received uploads request method=POST path=/27722026/11/11 23:30:06 INFO Completed push id=6 objects=6 duration=1.738737ms27732026/11/11 23:30:06 INFO Aborted multipart uploads count=0 kept=027742026/11/11 23:30:06 INFO Received push request method=POST path=/27752026/11/11 23:30:06 INFO Received complete push request method=POST path=/27762026/11/11 23:30:06 INFO Received uploads request method=POST path=/27772026/11/11 23:30:06 INFO Completed push id=8 objects=7 duration=1.718017ms27782026/11/11 23:30:06 INFO Received complete push request method=POST path=/27792026/11/11 23:30:06 INFO Received push request method=POST path=/27802026/11/11 23:30:06 INFO Aborted multipart uploads count=0 kept=027812026/11/11 23:30:06 INFO Received complete push request method=POST path=/27822026/11/11 23:30:06 INFO Completed push id=3 objects=5 duration=2.483163ms27832026/11/11 23:30:06 INFO Completed push id=9 objects=5 duration=1.973919ms27842026/11/11 23:30:06 INFO Received uploads request method=POST path=/27852026/11/11 23:30:06 INFO Aborted multipart uploads count=0 kept=027862026/11/11 23:30:06 WARN Force mode enabled - objects will be deleted immediately without grace period27872026/11/11 23:30:06 INFO Aborted multipart uploads count=0 kept=027882026/11/11 23:30:06 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."27892026/11/11 23:30:06 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."27902026/11/11 23:30:06 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."27912026/11/11 23:30:06 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."27922026/11/11 23:30:06 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."27932026/11/11 23:30:06 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."27942026/11/11 23:30:06 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."27952026/11/11 23:30:06 INFO Received push request method=POST path=/27962026/11/11 23:30:06 WARN Force mode enabled - objects will be deleted immediately without grace period27972026/11/11 23:30:06 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027982026/11/11 23:30:06 INFO Vacuumed table table=pending_closures27992026/11/11 23:30:06 INFO Vacuumed table table=pending_objects28002026/11/11 23:30:06 INFO Vacuumed table table=multipart_uploads28012026/11/11 23:30:06 INFO Vacuumed table table=closures28022026/11/11 23:30:06 INFO Received push request method=POST path=/28032026/11/11 23:30:06 INFO Vacuumed table table=objects28042026/11/11 23:30:06 INFO Received complete push request method=POST path=/28052026/11/11 23:30:06 INFO Aborted multipart uploads count=0 kept=028062026/11/11 23:30:06 INFO Completed push id=8 objects=13 duration=3.190171ms28072026/11/11 23:30:06 INFO Received uploads request method=POST path=/28082026/11/11 23:30:06 INFO Received push request method=POST path=/28092026/11/11 23:30:06 INFO Received uploads request method=POST path=/28102026/11/11 23:30:06 INFO Received complete upload request method=POST path=/28112026/11/11 23:30:06 INFO Received complete push request method=POST path=/28122026/11/11 23:30:06 INFO Completed upload id=9 objects=16 duration=1.272172ms28132026/11/11 23:30:06 INFO Received complete push request method=POST path=/28142026/11/11 23:30:06 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."28152026/11/11 23:30:06 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."28162026/11/11 23:30:06 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."28172026/11/11 23:30:06 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."28182026/11/11 23:30:06 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."28192026/11/11 23:30:06 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."28202026/11/11 23:30:06 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."28212026/11/11 23:30:06 INFO Completed push id=2 objects=7 duration=4.279601ms28222026/11/11 23:30:06 INFO Completed push id=7 objects=27 duration=2.11412ms28232026/11/11 23:30:06 INFO Received push request method=POST path=/28242026/11/11 23:30:06 INFO Received push request method=POST path=/28252026/11/11 23:30:06 INFO Aborted multipart uploads count=0 kept=028262026/11/11 23:30:06 INFO Received push request method=POST path=/28272026/11/11 23:30:06 INFO Aborted multipart uploads count=0 kept=028282026/11/11 23:30:06 INFO Aborted multipart uploads count=0 kept=028292026/11/11 23:30:06 INFO Received complete push request method=POST path=/28302026/11/11 23:30:06 INFO Completed push id=4 objects=4 duration=1.359873ms28312026/11/11 23:30:06 WARN Force mode enabled - objects will be deleted immediately without grace period28322026/11/11 23:30:06 INFO Received push request method=POST path=/28332026/11/11 23:30:06 INFO Received push request method=POST path=/28342026/11/11 23:30:06 INFO Received push request method=POST path=/28352026/11/11 23:30:06 INFO Received complete upload request method=POST path=/28362026/11/11 23:30:06 INFO Completed upload id=3 objects=9 duration=1.161632ms28372026/11/11 23:30:06 INFO Received complete push request method=POST path=/28382026/11/11 23:30:06 INFO Completed push id=10 objects=3 duration=1.418393ms28392026/11/11 23:30:06 INFO Received push request method=POST path=/28402026/11/11 23:30:06 INFO Received complete upload request method=POST path=/28412026/11/11 23:30:06 INFO Completed upload id=10 objects=17 duration=1.10081ms28422026/11/11 23:30:06 INFO Received uploads request method=POST path=/28432026/11/11 23:30:06 INFO Received push request method=POST path=/28442026/11/11 23:30:06 INFO Aborted multipart uploads count=0 kept=028452026/11/11 23:30:06 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."28462026/11/11 23:30:06 INFO Received complete push request method=POST path=/28472026/11/11 23:30:06 INFO Completed push id=5 objects=6 duration=1.763997ms28482026/11/11 23:30:06 INFO Received push request method=POST path=/28492026/11/11 23:30:06 INFO Received push request method=POST path=/28502026/11/11 23:30:06 INFO Received uploads request method=POST path=/28512026/11/11 23:30:06 INFO Received complete push request method=POST path=/28522026/11/11 23:30:06 INFO Aborted multipart uploads count=0 kept=028532026/11/11 23:30:06 INFO Completed push id=5 objects=3 duration=1.680616ms28542026/11/11 23:30:06 INFO Aborted multipart uploads count=0 kept=028552026/11/11 23:30:06 WARN Force mode enabled - objects will be deleted immediately without grace period28562026/11/11 23:30:06 WARN Force mode enabled - objects will be deleted immediately without grace period28572026/11/11 23:30:06 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028582026/11/11 23:30:06 INFO Vacuumed table table=pending_closures28592026/11/11 23:30:06 INFO Vacuumed table table=pending_objects28602026/11/11 23:30:06 INFO Vacuumed table table=multipart_uploads28612026/11/11 23:30:06 INFO Received push request method=POST path=/28622026/11/11 23:30:06 INFO Vacuumed table table=closures28632026/11/11 23:30:06 INFO Vacuumed table table=objects28642026/11/11 23:30:06 INFO Aborted multipart uploads count=0 kept=028652026/11/11 23:30:06 INFO Received complete push request method=POST path=/28662026/11/11 23:30:06 INFO Received uploads request method=POST path=/28672026/11/11 23:30:06 INFO Completed push id=14 objects=6 duration=2.272461ms28682026/11/11 23:30:06 INFO Aborted multipart uploads count=0 kept=028692026/11/11 23:30:06 INFO Received complete push request method=POST path=/28702026/11/11 23:30:06 INFO Received push request method=POST path=/28712026/11/11 23:30:06 INFO Completed push id=15 objects=24 duration=3.471793ms28722026/11/11 23:30:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=028732026/11/11 23:30:06 INFO Vacuumed table table=pending_closures28742026/11/11 23:30:06 INFO Vacuumed table table=pending_objects28752026/11/11 23:30:06 INFO Vacuumed table table=multipart_uploads28762026/11/11 23:30:06 INFO Vacuumed table table=closures28772026/11/11 23:30:06 INFO Received uploads request method=POST path=/28782026/11/11 23:30:06 INFO Received push request method=POST path=/28792026/11/11 23:30:06 INFO Received uploads request method=POST path=/28802026/11/11 23:30:06 INFO Vacuumed table table=objects28812026/11/11 23:30:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=8 objects-deleted-after-grace-period=8 objects-failed-to-delete=028822026/11/11 23:30:06 INFO Vacuumed table table=pending_closures28832026/11/11 23:30:06 INFO Vacuumed table table=pending_objects28842026/11/11 23:30:06 INFO Vacuumed table table=multipart_uploads28852026/11/11 23:30:06 INFO Vacuumed table table=closures28862026/11/11 23:30:06 INFO Vacuumed table table=objects28872026/11/11 23:30:06 INFO Aborted multipart uploads count=0 kept=028882026/11/11 23:30:06 INFO Received uploads request method=POST path=/28892026/11/11 23:30:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=1 objects-deleted-after-grace-period=0 objects-failed-to-delete=028902026/11/11 23:30:06 INFO Vacuumed table table=pending_closures28912026/11/11 23:30:06 INFO Vacuumed table table=pending_objects28922026/11/11 23:30:06 INFO Vacuumed table table=multipart_uploads28932026/11/11 23:30:06 INFO Vacuumed table table=closures28942026/11/11 23:30:06 INFO Vacuumed table table=objects28952026/11/11 23:30:06 INFO Received push request method=POST path=/28962026/11/11 23:30:06 INFO Received uploads request method=POST path=/28972026/11/11 23:30:06 INFO Received push request method=POST path=/28982026/11/11 23:30:06 INFO Aborted multipart uploads count=0 kept=028992026/11/11 23:30:06 INFO Received complete push request method=POST path=/29002026/11/11 23:30:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029012026/11/11 23:30:06 INFO Received uploads request method=POST path=/29022026/11/11 23:30:06 INFO Vacuumed table table=pending_closures29032026/11/11 23:30:06 INFO Vacuumed table table=pending_objects29042026/11/11 23:30:06 INFO Vacuumed table table=multipart_uploads29052026/11/11 23:30:06 INFO Vacuumed table table=closures29062026/11/11 23:30:06 INFO Vacuumed table table=objects29072026/11/11 23:30:06 INFO Completed push id=5 objects=9 duration=5.397291ms29082026/11/11 23:30:06 INFO Received complete upload request method=POST path=/29092026/11/11 23:30:06 INFO Completed upload id=6 objects=6 duration=1.240392ms29102026/11/11 23:30:06 INFO Received uploads request method=POST path=/29112026/11/11 23:30:06 INFO Received push request method=POST path=/29122026/11/11 23:30:06 INFO Received push request method=POST path=/29132026/11/11 23:30:06 INFO Aborted multipart uploads count=0 kept=029142026/11/11 23:30:06 WARN Force mode enabled - objects will be deleted immediately without grace period29152026/11/11 23:30:06 INFO Received uploads request method=POST path=/29162026/11/11 23:30:06 INFO Received push request method=POST path=/29172026/11/11 23:30:06 INFO Received complete push request method=POST path=/29182026/11/11 23:30:06 INFO Completed push id=8 objects=6 duration=2.724986ms29192026/11/11 23:30:06 INFO Received complete upload request method=POST path=/29202026/11/11 23:30:06 INFO Completed upload id=17 objects=6 duration=2.402863ms29212026/11/11 23:30:06 INFO Received uploads request method=POST path=/29222026/11/11 23:30:06 INFO Received complete push request method=POST path=/29232026/11/11 23:30:06 INFO Completed push id=12 objects=2 duration=3.10669ms29242026/11/11 23:30:06 INFO Received uploads request method=POST path=/29252026/11/11 23:30:06 INFO Received uploads request method=POST path=/29262026/11/11 23:30:06 INFO Received push request method=POST path=/29272026/11/11 23:30:06 INFO Received push request method=POST path=/29282026/11/11 23:30:06 INFO Received push request method=POST path=/29292026/11/11 23:30:06 INFO Received uploads request method=POST path=/29302026/11/11 23:30:06 INFO Aborted multipart uploads count=0 kept=029312026/11/11 23:30:06 WARN Force mode enabled - objects will be deleted immediately without grace period29322026/11/11 23:30:06 INFO Aborted multipart uploads count=0 kept=029332026/11/11 23:30:06 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029342026/11/11 23:30:06 INFO Vacuumed table table=pending_closures29352026/11/11 23:30:06 INFO Received complete push request method=POST path=/29362026/11/11 23:30:06 INFO Vacuumed table table=pending_objects29372026/11/11 23:30:06 INFO Vacuumed table table=multipart_uploads29382026/11/11 23:30:06 INFO Vacuumed table table=closures29392026/11/11 23:30:06 INFO Completed push id=9 objects=24 duration=1.979598ms29402026/11/11 23:30:06 INFO Vacuumed table table=objects29412026/11/11 23:30:06 INFO Received push request method=POST path=/29422026/11/11 23:30:06 INFO Received push request method=POST path=/29432026/11/11 23:30:06 INFO Received uploads request method=POST path=/29442026/11/11 23:30:06 INFO Received uploads request method=POST path=/29452026/11/11 23:30:06 INFO Received uploads request method=POST path=/29462026/11/11 23:30:06 INFO Received uploads request method=POST path=/29472026/11/11 23:30:06 INFO Received push request method=POST path=/29482026/11/11 23:30:06 INFO Aborted multipart uploads count=0 kept=029492026/11/11 23:30:06 INFO Received complete upload request method=POST path=/29502026/11/11 23:30:06 INFO Completed upload id=10 objects=10 duration=1.03883ms29512026/11/11 23:30:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029522026/11/11 23:30:06 INFO Received uploads request method=POST path=/29532026/11/11 23:30:06 INFO Vacuumed table table=pending_closures29542026/11/11 23:30:06 INFO Vacuumed table table=pending_objects29552026/11/11 23:30:06 INFO Vacuumed table table=multipart_uploads29562026/11/11 23:30:06 INFO Vacuumed table table=closures29572026/11/11 23:30:06 INFO Vacuumed table table=objects29582026/11/11 23:30:06 INFO Received complete upload request method=POST path=/29592026/11/11 23:30:06 INFO Completed upload id=15 objects=9 duration=957.589µs29602026/11/11 23:30:06 INFO Received complete push request method=POST path=/29612026/11/11 23:30:06 INFO Completed push id=22 objects=3 duration=2.333622ms29622026/11/11 23:30:06 INFO Received uploads request method=POST path=/29632026/11/11 23:30:06 INFO Aborted multipart uploads count=0 kept=029642026/11/11 23:30:06 WARN Force mode enabled - objects will be deleted immediately without grace period29652026/11/11 23:30:06 INFO Received push request method=POST path=/29662026/11/11 23:30:06 INFO Received complete push request method=POST path=/29672026/11/11 23:30:06 INFO Completed push id=21 objects=17 duration=1.960559ms29682026/11/11 23:30:06 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."29692026/11/11 23:30:06 INFO Received push request method=POST path=/29702026/11/11 23:30:06 INFO Received complete push request method=POST path=/29712026/11/11 23:30:06 INFO Completed push id=19 objects=7 duration=1.708077ms29722026/11/11 23:30:06 INFO Received uploads request method=POST path=/29732026/11/11 23:30:06 INFO Received push request method=POST path=/29742026/11/11 23:30:06 INFO Received uploads request method=POST path=/29752026/11/11 23:30:06 INFO Received uploads request method=POST path=/29762026/11/11 23:30:06 INFO Aborted multipart uploads count=0 kept=029772026/11/11 23:30:06 WARN Force mode enabled - objects will be deleted immediately without grace period29782026/11/11 23:30:06 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029792026/11/11 23:30:06 INFO Vacuumed table table=pending_closures29802026/11/11 23:30:06 INFO Vacuumed table table=pending_objects29812026/11/11 23:30:06 INFO Vacuumed table table=multipart_uploads29822026/11/11 23:30:06 INFO Vacuumed table table=closures29832026/11/11 23:30:06 INFO Received complete push request method=POST path=/29842026/11/11 23:30:06 INFO Vacuumed table table=objects29852026/11/11 23:30:06 INFO Completed push id=14 objects=3 duration=1.03439ms29862026/11/11 23:30:06 INFO Received uploads request method=POST path=/29872026/11/11 23:30:06 INFO Aborted multipart uploads count=0 kept=029882026/11/11 23:30:06 INFO Received push request method=POST path=/29892026/11/11 23:30:06 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."29902026/11/11 23:30:06 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."29912026/11/11 23:30:06 INFO Received complete upload request method=POST path=/29922026/11/11 23:30:07 INFO Completed upload id=12 objects=26 duration=3.000009ms29932026/11/11 23:30:07 INFO Received push request method=POST path=/29942026/11/11 23:30:07 INFO Aborted multipart uploads count=0 kept=029952026/11/11 23:30:07 WARN Force mode enabled - objects will be deleted immediately without grace period29962026/11/11 23:30:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=3 objects-failed-to-delete=029972026/11/11 23:30:07 INFO Vacuumed table table=pending_closures29982026/11/11 23:30:07 INFO Vacuumed table table=pending_objects29992026/11/11 23:30:07 INFO Vacuumed table table=multipart_uploads30002026/11/11 23:30:07 INFO Vacuumed table table=closures30012026/11/11 23:30:07 INFO Vacuumed table table=objects30022026/11/11 23:30:07 INFO Aborted multipart uploads count=0 kept=030032026/11/11 23:30:07 WARN Force mode enabled - objects will be deleted immediately without grace period30042026/11/11 23:30:07 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030052026/11/11 23:30:07 INFO Vacuumed table table=pending_closures30062026/11/11 23:30:07 INFO Vacuumed table table=pending_objects30072026/11/11 23:30:07 INFO Vacuumed table table=multipart_uploads30082026/11/11 23:30:07 INFO Vacuumed table table=closures30092026/11/11 23:30:07 INFO Vacuumed table table=objects30102026/11/11 23:30:07 INFO Received complete upload request method=POST path=/30112026/11/11 23:30:07 INFO Received uploads request method=POST path=/30122026/11/11 23:30:07 INFO Completed upload id=21 objects=7 duration=2.12986ms30132026/11/11 23:30:07 INFO Received push request method=POST path=/30142026/11/11 23:30:07 INFO Received complete push request method=POST path=/30152026/11/11 23:30:07 INFO Completed push id=16 objects=8 duration=2.03874ms30162026/11/11 23:30:07 INFO Received push request method=POST path=/30172026/11/11 23:30:07 INFO Received uploads request method=POST path=/30182026/11/11 23:30:07 INFO Aborted multipart uploads count=0 kept=030192026/11/11 23:30:07 WARN Force mode enabled - objects will be deleted immediately without grace period30202026/11/11 23:30:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030212026/11/11 23:30:07 INFO Vacuumed table table=pending_closures30222026/11/11 23:30:07 INFO Vacuumed table table=pending_objects30232026/11/11 23:30:07 INFO Vacuumed table table=multipart_uploads30242026/11/11 23:30:07 INFO Vacuumed table table=closures30252026/11/11 23:30:07 INFO Vacuumed table table=objects30262026/11/11 23:30:07 INFO Received uploads request method=POST path=/30272026/11/11 23:30:07 INFO Received uploads request method=POST path=/30282026/11/11 23:30:07 INFO Aborted multipart uploads count=0 kept=030292026/11/11 23:30:07 WARN Force mode enabled - objects will be deleted immediately without grace period30302026/11/11 23:30:07 INFO Received complete push request method=POST path=/30312026/11/11 23:30:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030322026/11/11 23:30:07 INFO Vacuumed table table=pending_closures30332026/11/11 23:30:07 INFO Completed push id=11 objects=6 duration=3.397652ms30342026/11/11 23:30:07 INFO Vacuumed table table=pending_objects30352026/11/11 23:30:07 INFO Vacuumed table table=multipart_uploads30362026/11/11 23:30:07 INFO Vacuumed table table=closures30372026/11/11 23:30:07 INFO Vacuumed table table=objects30382026/11/11 23:30:07 INFO Received uploads request method=POST path=/30392026/11/11 23:30:07 INFO Received complete upload request method=POST path=/30402026/11/11 23:30:07 INFO Received complete push request method=POST path=/30412026/11/11 23:30:07 INFO Completed upload id=27 objects=4 duration=972.829µs30422026/11/11 23:30:07 INFO Completed push id=18 objects=6 duration=1.269772ms30432026/11/11 23:30:07 INFO Received complete push request method=POST path=/30442026/11/11 23:30:07 INFO Received uploads request method=POST path=/30452026/11/11 23:30:07 INFO Completed push id=13 objects=11 duration=1.922259ms30462026/11/11 23:30:07 INFO Aborted multipart uploads count=0 kept=030472026/11/11 23:30:07 INFO Received uploads request method=POST path=/30482026/11/11 23:30:07 INFO Received uploads request method=POST path=/30492026/11/11 23:30:07 INFO Aborted multipart uploads count=0 kept=030502026/11/11 23:30:07 INFO Received uploads request method=POST path=/30512026/11/11 23:30:07 WARN Force mode enabled - objects will be deleted immediately without grace period30522026/11/11 23:30:07 INFO Received uploads request method=POST path=/30532026/11/11 23:30:07 INFO Received complete push request method=POST path=/30542026/11/11 23:30:07 INFO Completed push id=18 objects=6 duration=2.874227ms30552026/11/11 23:30:07 INFO Received complete upload request method=POST path=/30562026/11/11 23:30:07 INFO Completed upload id=23 objects=9 duration=1.863958ms30572026/11/11 23:30:07 INFO Received push request method=POST path=/30582026/11/11 23:30:07 INFO Received uploads request method=POST path=/30592026/11/11 23:30:07 INFO Received push request method=POST path=/30602026/11/11 23:30:07 INFO Received complete push request method=POST path=/30612026/11/11 23:30:07 INFO Completed push id=16 objects=26 duration=2.868667ms30622026/11/11 23:30:07 INFO Aborted multipart uploads count=0 kept=030632026/11/11 23:30:07 WARN Force mode enabled - objects will be deleted immediately without grace period30642026/11/11 23:30:07 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030652026/11/11 23:30:07 INFO Vacuumed table table=pending_closures30662026/11/11 23:30:07 INFO Vacuumed table table=pending_objects30672026/11/11 23:30:07 INFO Vacuumed table table=multipart_uploads30682026/11/11 23:30:07 INFO Vacuumed table table=closures30692026/11/11 23:30:07 INFO Vacuumed table table=objects30702026/11/11 23:30:07 INFO Received complete push request method=POST path=/30712026/11/11 23:30:07 INFO Completed push id=27 objects=6 duration=4.292081ms30722026/11/11 23:30:07 INFO Received complete upload request method=POST path=/30732026/11/11 23:30:07 INFO Completed upload id=15 objects=2 duration=1.286232ms30742026/11/11 23:30:07 INFO Received push request method=POST path=/30752026/11/11 23:30:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030762026/11/11 23:30:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=4 objects-deleted-after-grace-period=0 objects-failed-to-delete=030772026/11/11 23:30:07 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030782026/11/11 23:30:07 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030792026/11/11 23:30:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030802026/11/11 23:30:07 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030812026/11/11 23:30:07 INFO Vacuumed table table=pending_closures30822026/11/11 23:30:07 INFO Vacuumed table table=pending_closures30832026/11/11 23:30:07 INFO Vacuumed table table=pending_closures30842026/11/11 23:30:07 INFO Vacuumed table table=pending_closures30852026/11/11 23:30:07 INFO Vacuumed table table=pending_closures30862026/11/11 23:30:07 INFO Vacuumed table table=pending_objects30872026/11/11 23:30:07 INFO Vacuumed table table=pending_closures30882026/11/11 23:30:07 INFO Vacuumed table table=multipart_uploads30892026/11/11 23:30:07 INFO Vacuumed table table=pending_objects30902026/11/11 23:30:07 INFO Vacuumed table table=pending_objects30912026/11/11 23:30:07 INFO Vacuumed table table=multipart_uploads30922026/11/11 23:30:07 INFO Vacuumed table table=pending_objects30932026/11/11 23:30:07 INFO Vacuumed table table=closures30942026/11/11 23:30:07 INFO Vacuumed table table=multipart_uploads30952026/11/11 23:30:07 INFO Vacuumed table table=multipart_uploads30962026/11/11 23:30:07 INFO Vacuumed table table=pending_objects30972026/11/11 23:30:07 INFO Vacuumed table table=multipart_uploads30982026/11/11 23:30:07 INFO Vacuumed table table=closures30992026/11/11 23:30:07 INFO Vacuumed table table=closures31002026/11/11 23:30:07 INFO Vacuumed table table=closures31012026/11/11 23:30:07 INFO Vacuumed table table=objects31022026/11/11 23:30:07 INFO Vacuumed table table=closures31032026/11/11 23:30:07 INFO Vacuumed table table=objects31042026/11/11 23:30:07 INFO Vacuumed table table=pending_objects31052026/11/11 23:30:07 INFO Vacuumed table table=objects31062026/11/11 23:30:07 INFO Vacuumed table table=multipart_uploads31072026/11/11 23:30:07 INFO Vacuumed table table=objects31082026/11/11 23:30:07 INFO Vacuumed table table=objects31092026/11/11 23:30:07 INFO Vacuumed table table=closures31102026/11/11 23:30:07 INFO Vacuumed table table=objects31112026/11/11 23:30:07 WARN Force mode enabled - objects will be deleted immediately without grace period31122026/11/11 23:30:07 WARN Force mode enabled - objects will be deleted immediately without grace period31132026/11/11 23:30:07 INFO Received uploads request method=POST path=/31142026/11/11 23:30:07 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031152026/11/11 23:30:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031162026/11/11 23:30:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031172026/11/11 23:30:07 INFO Received uploads request method=POST path=/31182026/11/11 23:30:07 INFO Vacuumed table table=pending_closures31192026/11/11 23:30:07 INFO Vacuumed table table=pending_closures31202026/11/11 23:30:07 INFO Vacuumed table table=pending_closures31212026/11/11 23:30:07 INFO Vacuumed table table=pending_objects31222026/11/11 23:30:07 INFO Vacuumed table table=pending_objects31232026/11/11 23:30:07 INFO Vacuumed table table=multipart_uploads31242026/11/11 23:30:07 INFO Received uploads request method=POST path=/31252026/11/11 23:30:07 INFO Received push request method=POST path=/31262026/11/11 23:30:07 INFO Aborted multipart uploads count=0 kept=031272026/11/11 23:30:07 INFO Received complete upload request method=POST path=/31282026/11/11 23:30:07 INFO Vacuumed table table=multipart_uploads31292026/11/11 23:30:07 INFO Vacuumed table table=pending_objects31302026/11/11 23:30:07 INFO Vacuumed table table=closures31312026/11/11 23:30:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=1 objects-deleted-after-grace-period=0 objects-failed-to-delete=031322026/11/11 23:30:07 INFO Vacuumed table table=multipart_uploads31332026/11/11 23:30:07 INFO Vacuumed table table=closures31342026/11/11 23:30:07 INFO Vacuumed table table=closures31352026/11/11 23:30:07 WARN Force mode enabled - objects will be deleted immediately without grace period31362026/11/11 23:30:07 INFO Vacuumed table table=objects31372026/11/11 23:30:07 INFO Vacuumed table table=pending_closures31382026/11/11 23:30:07 INFO Completed upload id=20 objects=3 duration=2.974068ms31392026/11/11 23:30:07 INFO Vacuumed table table=objects31402026/11/11 23:30:07 WARN Force mode enabled - objects will be deleted immediately without grace period31412026/11/11 23:30:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031422026/11/11 23:30:07 INFO Vacuumed table table=objects31432026/11/11 23:30:07 INFO Vacuumed table table=pending_objects31442026/11/11 23:30:07 INFO Vacuumed table table=pending_closures31452026/11/11 23:30:07 INFO Vacuumed table table=multipart_uploads31462026/11/11 23:30:07 INFO Vacuumed table table=pending_objects31472026/11/11 23:30:07 INFO Vacuumed table table=multipart_uploads31482026/11/11 23:30:07 INFO Received push request method=POST path=/31492026/11/11 23:30:07 INFO Vacuumed table table=closures31502026/11/11 23:30:07 INFO Received uploads request method=POST path=/31512026/11/11 23:30:07 INFO Vacuumed table table=closures31522026/11/11 23:30:07 INFO Vacuumed table table=objects31532026/11/11 23:30:07 INFO Vacuumed table table=objects31542026/11/11 23:30:07 INFO Aborted multipart uploads count=0 kept=031552026/11/11 23:30:07 WARN Force mode enabled - objects will be deleted immediately without grace period31562026/11/11 23:30:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031572026/11/11 23:30:07 INFO Received push request method=POST path=/31582026/11/11 23:30:07 INFO Vacuumed table table=pending_closures31592026/11/11 23:30:07 INFO Vacuumed table table=pending_objects31602026/11/11 23:30:07 INFO Vacuumed table table=multipart_uploads31612026/11/11 23:30:07 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=031622026/11/11 23:30:07 INFO Vacuumed table table=closures31632026/11/11 23:30:07 INFO Received complete upload request method=POST path=/31642026/11/11 23:30:07 INFO Vacuumed table table=objects31652026/11/11 23:30:07 INFO Vacuumed table table=pending_closures31662026/11/11 23:30:07 INFO Vacuumed table table=pending_objects31672026/11/11 23:30:07 INFO Received push request method=POST path=/31682026/11/11 23:30:07 INFO Vacuumed table table=multipart_uploads31692026/11/11 23:30:07 INFO Received uploads request method=POST path=/31702026/11/11 23:30:07 INFO Received uploads request method=POST path=/31712026/11/11 23:30:07 INFO Completed upload id=9 objects=7 duration=2.364463ms31722026/11/11 23:30:07 INFO Vacuumed table table=closures31732026/11/11 23:30:07 INFO Vacuumed table table=objects31742026/11/11 23:30:07 INFO Received complete upload request method=POST path=/31752026/11/11 23:30:07 INFO Received push request method=POST path=/31762026/11/11 23:30:07 INFO Received push request method=POST path=/31772026/11/11 23:30:07 INFO Completed upload id=26 objects=8 duration=1.929658ms31782026/11/11 23:30:07 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."31792026/11/11 23:30:07 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."31802026/11/11 23:30:07 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."31812026/11/11 23:30:07 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."31822026/11/11 23:30:07 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."31832026/11/11 23:30:07 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."31842026/11/11 23:30:07 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."31852026/11/11 23:30:07 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."31862026/11/11 23:30:07 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."31872026/11/11 23:30:07 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."31882026/11/11 23:30:07 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."31892026/11/11 23:30:07 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."31902026/11/11 23:30:07 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."31912026/11/11 23:30:07 INFO Received push request method=POST path=/31922026/11/11 23:30:07 INFO Received complete push request method=POST path=/31932026/11/11 23:30:07 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."31942026/11/11 23:30:07 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.narinfo error="Access Denied."31952026/11/11 23:30:07 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."31962026/11/11 23:30:07 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."31972026/11/11 23:30:07 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."31982026/11/11 23:30:07 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."31992026/11/11 23:30:07 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."32002026/11/11 23:30:07 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."32012026/11/11 23:30:07 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."32022026/11/11 23:30:07 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."32032026/11/11 23:30:07 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."32042026/11/11 23:30:07 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."32052026/11/11 23:30:07 ERROR failed to remove object object=nar/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.nar.zst error="Access Denied."32062026/11/11 23:30:07 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."32072026/11/11 23:30:07 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."32082026/11/11 23:30:07 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."32092026/11/11 23:30:07 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."32102026/11/11 23:30:07 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."32112026/11/11 23:30:07 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."32122026/11/11 23:30:07 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."32132026/11/11 23:30:07 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."32142026/11/11 23:30:07 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."32152026/11/11 23:30:07 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."32162026/11/11 23:30:07 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."32172026/11/11 23:30:07 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."32182026/11/11 23:30:07 INFO Received complete push request method=POST path=/32192026/11/11 23:30:07 INFO Completed push id=28 objects=14 duration=1.734696ms32202026/11/11 23:30:07 INFO Received push request method=POST path=/32212026/11/11 23:30:07 INFO Completed push id=6 objects=5 duration=3.995698ms32222026/11/11 23:30:07 INFO Received complete upload request method=POST path=/32232026/11/11 23:30:07 INFO Received uploads request method=POST path=/32242026/11/11 23:30:07 INFO Completed upload id=10 objects=4 duration=2.764206ms32252026/11/11 23:30:07 INFO Received complete push request method=POST path=/32262026/11/11 23:30:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032272026/11/11 23:30:07 INFO Received complete upload request method=POST path=/32282026/11/11 23:30:07 INFO Vacuumed table table=pending_closures32292026/11/11 23:30:07 INFO Completed upload id=24 objects=7 duration=1.263572ms32302026/11/11 23:30:07 INFO Aborted multipart uploads count=0 kept=032312026/11/11 23:30:07 INFO Completed push id=11 objects=11 duration=4.076959ms32322026/11/11 23:30:07 INFO Vacuumed table table=pending_objects32332026/11/11 23:30:07 INFO Received push request method=POST path=/32342026/11/11 23:30:07 INFO Vacuumed table table=multipart_uploads32352026/11/11 23:30:07 INFO Vacuumed table table=closures32362026/11/11 23:30:07 INFO Vacuumed table table=objects32372026/11/11 23:30:07 WARN Force mode enabled - objects will be deleted immediately without grace period32382026/11/11 23:30:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032392026/11/11 23:30:07 INFO Received complete upload request method=POST path=/32402026/11/11 23:30:07 INFO Vacuumed table table=pending_closures32412026/11/11 23:30:07 INFO Received complete push request method=POST path=/32422026/11/11 23:30:07 INFO Received push request method=POST path=/32432026/11/11 23:30:07 INFO Vacuumed table table=pending_objects32442026/11/11 23:30:07 INFO Received push request method=POST path=/32452026/11/11 23:30:07 INFO Vacuumed table table=multipart_uploads32462026/11/11 23:30:07 INFO Completed upload id=3 objects=8 duration=2.593545ms32472026/11/11 23:30:07 INFO Received push request method=POST path=/32482026/11/11 23:30:07 INFO Vacuumed table table=closures32492026/11/11 23:30:07 INFO Completed push id=8 objects=2 duration=2.161021ms32502026/11/11 23:30:07 INFO Vacuumed table table=objects32512026/11/11 23:30:07 INFO Received uploads request method=POST path=/32522026/11/11 23:30:07 INFO Received push request method=POST path=/32532026/11/11 23:30:07 INFO Received uploads request method=POST path=/32542026/11/11 23:30:07 INFO Aborted multipart uploads count=0 kept=032552026/11/11 23:30:07 INFO Received push request method=POST path=/32562026/11/11 23:30:07 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032572026/11/11 23:30:07 INFO Vacuumed table table=pending_closures32582026/11/11 23:30:07 INFO Received push request method=POST path=/32592026/11/11 23:30:07 INFO Vacuumed table table=pending_objects32602026/11/11 23:30:07 INFO Received uploads request method=POST path=/32612026/11/11 23:30:07 INFO Vacuumed table table=multipart_uploads32622026/11/11 23:30:07 INFO Vacuumed table table=closures32632026/11/11 23:30:07 INFO Aborted multipart uploads count=0 kept=032642026/11/11 23:30:07 INFO Vacuumed table table=objects32652026/11/11 23:30:07 INFO Received push request method=POST path=/32662026/11/11 23:30:07 INFO Received uploads request method=POST path=/32672026/11/11 23:30:07 INFO Received uploads request method=POST path=/32682026/11/11 23:30:07 INFO Received push request method=POST path=/32692026/11/11 23:30:07 INFO Received uploads request method=POST path=/32702026/11/11 23:30:07 INFO Received complete push request method=POST path=/32712026/11/11 23:30:07 INFO Completed push id=4 objects=6 duration=2.11562ms32722026/11/11 23:30:07 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."32732026/11/11 23:30:07 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."32742026/11/11 23:30:07 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."32752026/11/11 23:30:07 INFO Received uploads request method=POST path=/32762026/11/11 23:30:07 INFO Received uploads request method=POST path=/32772026/11/11 23:30:07 INFO Aborted multipart uploads count=0 kept=032782026/11/11 23:30:07 INFO Received complete upload request method=POST path=/32792026/11/11 23:30:07 WARN Force mode enabled - objects will be deleted immediately without grace period32802026/11/11 23:30:07 INFO Received uploads request method=POST path=/32812026/11/11 23:30:07 INFO Received push request method=POST path=/32822026/11/11 23:30:07 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032832026/11/11 23:30:07 INFO Completed upload id=4 objects=14 duration=3.19279ms32842026/11/11 23:30:07 INFO Vacuumed table table=pending_closures32852026/11/11 23:30:07 INFO Vacuumed table table=pending_objects32862026/11/11 23:30:07 INFO Vacuumed table table=multipart_uploads32872026/11/11 23:30:07 INFO Vacuumed table table=closures32882026/11/11 23:30:07 INFO Aborted multipart uploads count=0 kept=032892026/11/11 23:30:07 INFO Vacuumed table table=objects32902026/11/11 23:30:07 INFO Received complete push request method=POST path=/32912026/11/11 23:30:07 INFO Received push request method=POST path=/32922026/11/11 23:30:07 WARN Force mode enabled - objects will be deleted immediately without grace period32932026/11/11 23:30:07 INFO Completed push id=16 objects=14 duration=2.804107ms32942026/11/11 23:30:07 INFO Received push request method=POST path=/32952026/11/11 23:30:07 INFO Aborted multipart uploads count=0 kept=032962026/11/11 23:30:07 INFO Aborted multipart uploads count=0 kept=032972026/11/11 23:30:07 WARN Force mode enabled - objects will be deleted immediately without grace period32982026/11/11 23:30:07 WARN Force mode enabled - objects will be deleted immediately without grace period32992026/11/11 23:30:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033002026/11/11 23:30:07 INFO Received push request method=POST path=/33012026/11/11 23:30:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033022026/11/11 23:30:07 INFO Vacuumed table table=pending_closures33032026/11/11 23:30:07 INFO Received complete upload request method=POST path=/33042026/11/11 23:30:07 INFO Vacuumed table table=pending_closures33052026/11/11 23:30:07 INFO Vacuumed table table=pending_objects33062026/11/11 23:30:07 INFO Vacuumed table table=multipart_uploads33072026/11/11 23:30:07 INFO Vacuumed table table=pending_objects33082026/11/11 23:30:07 INFO Vacuumed table table=closures33092026/11/11 23:30:07 INFO Vacuumed table table=multipart_uploads33102026/11/11 23:30:07 INFO Vacuumed table table=closures33112026/11/11 23:30:07 INFO Aborted multipart uploads count=0 kept=033122026/11/11 23:30:07 INFO Received uploads request method=POST path=/33132026/11/11 23:30:07 INFO Vacuumed table table=objects33142026/11/11 23:30:07 INFO Vacuumed table table=objects33152026/11/11 23:30:07 INFO Received uploads request method=POST path=/33162026/11/11 23:30:07 INFO Completed upload id=6 objects=24 duration=4.412522ms33172026/11/11 23:30:07 WARN Force mode enabled - objects will be deleted immediately without grace period33182026/11/11 23:30:07 INFO Received uploads request method=POST path=/33192026/11/11 23:30:07 INFO Aborted multipart uploads count=0 kept=033202026/11/11 23:30:07 INFO Received complete upload request method=POST path=/33212026/11/11 23:30:07 INFO Received uploads request method=POST path=/33222026/11/11 23:30:07 INFO Completed upload id=11 objects=14 duration=1.363093ms33232026/11/11 23:30:07 INFO Received complete upload request method=POST path=/33242026/11/11 23:30:07 INFO Aborted multipart uploads count=0 kept=033252026/11/11 23:30:07 INFO Completed upload id=9 objects=2 duration=1.182191ms33262026/11/11 23:30:07 INFO Aborted multipart uploads count=0 kept=033272026/11/11 23:30:07 WARN Force mode enabled - objects will be deleted immediately without grace period33282026/11/11 23:30:07 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033292026/11/11 23:30:07 INFO Received push request method=POST path=/33302026/11/11 23:30:07 INFO Vacuumed table table=pending_closures33312026/11/11 23:30:07 INFO Vacuumed table table=pending_objects33322026/11/11 23:30:07 INFO Vacuumed table table=multipart_uploads33332026/11/11 23:30:07 INFO Vacuumed table table=closures33342026/11/11 23:30:07 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."33352026/11/11 23:30:07 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."33362026/11/11 23:30:07 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."33372026/11/11 23:30:07 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."33382026/11/11 23:30:07 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."33392026/11/11 23:30:07 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."33402026/11/11 23:30:07 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."33412026/11/11 23:30:07 WARN Force mode enabled - objects will be deleted immediately without grace period33422026/11/11 23:30:07 INFO Aborted multipart uploads count=0 kept=033432026/11/11 23:30:07 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."33442026/11/11 23:30:07 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."33452026/11/11 23:30:07 INFO Received complete push request method=POST path=/33462026/11/11 23:30:07 INFO Vacuumed table table=objects33472026/11/11 23:30:07 INFO Received push request method=POST path=/33482026/11/11 23:30:07 INFO Received uploads request method=POST path=/33492026/11/11 23:30:07 WARN Force mode enabled - objects will be deleted immediately without grace period33502026/11/11 23:30:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033512026/11/11 23:30:07 INFO Received complete upload request method=POST path=/33522026/11/11 23:30:07 INFO Received push request method=POST path=/33532026/11/11 23:30:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033542026/11/11 23:30:07 INFO Vacuumed table table=pending_closures33552026/11/11 23:30:07 INFO Completed upload id=21 objects=9 duration=2.238861ms33562026/11/11 23:30:07 INFO Vacuumed table table=pending_closures33572026/11/11 23:30:07 INFO Received complete push request method=POST path=/33582026/11/11 23:30:07 INFO Vacuumed table table=pending_objects33592026/11/11 23:30:07 INFO Vacuumed table table=pending_objects33602026/11/11 23:30:07 INFO Vacuumed table table=multipart_uploads33612026/11/11 23:30:07 INFO Vacuumed table table=multipart_uploads33622026/11/11 23:30:07 INFO Vacuumed table table=closures33632026/11/11 23:30:07 INFO Vacuumed table table=closures33642026/11/11 23:30:07 INFO Completed push id=8 objects=3 duration=1.612915ms33652026/11/11 23:30:07 INFO Received complete push request method=POST path=/33662026/11/11 23:30:07 INFO Vacuumed table table=objects33672026/11/11 23:30:07 INFO Vacuumed table table=objects33682026/11/11 23:30:07 INFO Received complete push request method=POST path=/33692026/11/11 23:30:07 INFO Completed push id=10 objects=2 duration=2.06758ms33702026/11/11 23:30:07 INFO Completed push id=3 objects=22 duration=8.875964ms33712026/11/11 23:30:07 INFO Completed push id=23 objects=28 duration=2.673085ms33722026/11/11 23:30:07 INFO Aborted multipart uploads count=0 kept=033732026/11/11 23:30:07 INFO Received push request method=POST path=/33742026/11/11 23:30:07 WARN Force mode enabled - objects will be deleted immediately without grace period33752026/11/11 23:30:07 INFO Received uploads request method=POST path=/33762026/11/11 23:30:07 INFO Aborted multipart uploads count=0 kept=033772026/11/11 23:30:07 INFO Received push request method=POST path=/33782026/11/11 23:30:07 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."33792026/11/11 23:30:07 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."33802026/11/11 23:30:07 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."33812026/11/11 23:30:07 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."33822026/11/11 23:30:07 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."33832026/11/11 23:30:07 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."33842026/11/11 23:30:07 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."33852026/11/11 23:30:07 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."33862026/11/11 23:30:07 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."33872026/11/11 23:30:07 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."33882026/11/11 23:30:07 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."33892026/11/11 23:30:07 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."33902026/11/11 23:30:07 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."33912026/11/11 23:30:07 ERROR failed to remove object object=realisations/sha256:aaaaaaaa!out.doi error="Access Denied."33922026/11/11 23:30:07 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."33932026/11/11 23:30:07 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."33942026/11/11 23:30:07 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."33952026/11/11 23:30:07 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."33962026/11/11 23:30:07 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."33972026/11/11 23:30:07 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."33982026/11/11 23:30:07 INFO Received push request method=POST path=/33992026/11/11 23:30:07 INFO Received complete push request method=POST path=/34002026/11/11 23:30:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034012026/11/11 23:30:07 INFO Vacuumed table table=pending_closures34022026/11/11 23:30:07 INFO Received push request method=POST path=/34032026/11/11 23:30:07 INFO Completed push id=25 objects=10 duration=1.894518ms34042026/11/11 23:30:07 INFO Received complete push request method=POST path=/34052026/11/11 23:30:07 INFO Vacuumed table table=pending_objects34062026/11/11 23:30:07 INFO Received push request method=POST path=/34072026/11/11 23:30:07 INFO Vacuumed table table=multipart_uploads34082026/11/11 23:30:07 INFO Vacuumed table table=closures34092026/11/11 23:30:07 INFO Completed push id=8 objects=6 duration=2.334183ms34102026/11/11 23:30:07 INFO Vacuumed table table=objects34112026/11/11 23:30:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=9 objects-failed-to-delete=034122026/11/11 23:30:07 INFO Received uploads request method=POST path=/34132026/11/11 23:30:07 INFO Received complete upload request method=POST path=/34142026/11/11 23:30:07 INFO Vacuumed table table=pending_closures34152026/11/11 23:30:07 INFO Completed upload id=12 objects=21 duration=1.546715ms34162026/11/11 23:30:07 INFO Received push request method=POST path=/34172026/11/11 23:30:07 INFO Vacuumed table table=pending_objects34182026/11/11 23:30:07 INFO Received push request method=POST path=/34192026/11/11 23:30:07 INFO Vacuumed table table=multipart_uploads34202026/11/11 23:30:07 INFO Vacuumed table table=closures34212026/11/11 23:30:07 INFO Vacuumed table table=objects34222026/11/11 23:30:07 INFO Received push request method=POST path=/34232026/11/11 23:30:07 INFO Received uploads request method=POST path=/34242026/11/11 23:30:07 INFO Received complete upload request method=POST path=/34252026/11/11 23:30:07 INFO Completed upload id=29 objects=5 duration=1.10157ms34262026/11/11 23:30:07 INFO Received push request method=POST path=/34272026/11/11 23:30:07 INFO Received complete upload request method=POST path=/34282026/11/11 23:30:07 INFO Completed upload id=27 objects=8 duration=1.04839ms34292026/11/11 23:30:07 INFO Received push request method=POST path=/34302026/11/11 23:30:07 INFO Received complete upload request method=POST path=/34312026/11/11 23:30:07 INFO Received uploads request method=POST path=/34322026/11/11 23:30:07 INFO Received uploads request method=POST path=/34332026/11/11 23:30:07 INFO Received push request method=POST path=/34342026/11/11 23:30:07 INFO Completed upload id=9 objects=3 duration=1.13487ms34352026/11/11 23:30:07 INFO Aborted multipart uploads count=0 kept=034362026/11/11 23:30:07 WARN Force mode enabled - objects will be deleted immediately without grace period34372026/11/11 23:30:07 INFO Received complete upload request method=POST path=/34382026/11/11 23:30:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034392026/11/11 23:30:07 INFO Completed upload id=13 objects=10 duration=1.158891ms34402026/11/11 23:30:07 INFO Aborted multipart uploads count=0 kept=034412026/11/11 23:30:07 INFO Vacuumed table table=pending_closures34422026/11/11 23:30:07 INFO Vacuumed table table=pending_objects34432026/11/11 23:30:07 INFO Vacuumed table table=multipart_uploads34442026/11/11 23:30:07 INFO Vacuumed table table=closures34452026/11/11 23:30:07 INFO Received complete upload request method=POST path=/34462026/11/11 23:30:07 INFO Vacuumed table table=objects34472026/11/11 23:30:07 INFO Received uploads request method=POST path=/34482026/11/11 23:30:07 INFO Completed upload id=30 objects=2 duration=1.04227ms34492026/11/11 23:30:07 INFO Received uploads request method=POST path=/34502026/11/11 23:30:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034512026/11/11 23:30:07 INFO Received push request method=POST path=/34522026/11/11 23:30:07 INFO Received push request method=POST path=/34532026/11/11 23:30:07 INFO Vacuumed table table=pending_closures34542026/11/11 23:30:07 INFO Vacuumed table table=pending_objects34552026/11/11 23:30:07 INFO Aborted multipart uploads count=0 kept=034562026/11/11 23:30:07 INFO Vacuumed table table=multipart_uploads34572026/11/11 23:30:07 INFO Vacuumed table table=closures34582026/11/11 23:30:07 INFO Received push request method=POST path=/34592026/11/11 23:30:07 WARN Force mode enabled - objects will be deleted immediately without grace period34602026/11/11 23:30:07 INFO Vacuumed table table=objects34612026/11/11 23:30:07 INFO Aborted multipart uploads count=0 kept=034622026/11/11 23:30:07 INFO Received push request method=POST path=/34632026/11/11 23:30:07 INFO Received complete upload request method=POST path=/34642026/11/11 23:30:07 INFO Received complete upload request method=POST path=/34652026/11/11 23:30:07 WARN Force mode enabled - objects will be deleted immediately without grace period34662026/11/11 23:30:07 INFO Received complete push request method=POST path=/34672026/11/11 23:30:07 INFO Received complete upload request method=POST path=/34682026/11/11 23:30:07 INFO Completed upload id=23 objects=16 duration=2.12254ms34692026/11/11 23:30:07 INFO Completed upload id=10 objects=6 duration=3.20447ms34702026/11/11 23:30:07 INFO Completed push id=13 objects=23 duration=1.846938ms34712026/11/11 23:30:07 INFO Received push request method=POST path=/34722026/11/11 23:30:07 INFO Completed upload id=10 objects=23 duration=2.217221ms34732026/11/11 23:30:07 INFO Received complete push request method=POST path=/34742026/11/11 23:30:07 INFO Completed push id=31 objects=20 duration=1.066151ms34752026/11/11 23:30:07 INFO Received complete upload request method=POST path=/34762026/11/11 23:30:07 INFO Completed upload id=25 objects=6 duration=1.599995ms34772026/11/11 23:30:07 INFO Aborted multipart uploads count=0 kept=034782026/11/11 23:30:07 INFO Received push request method=POST path=/34792026/11/11 23:30:07 INFO Aborted multipart uploads count=0 kept=034802026/11/11 23:30:07 INFO Received complete push request method=POST path=/34812026/11/11 23:30:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=16 objects-failed-to-delete=034822026/11/11 23:30:07 INFO Vacuumed table table=pending_closures34832026/11/11 23:30:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034842026/11/11 23:30:07 INFO Vacuumed table table=pending_objects34852026/11/11 23:30:07 INFO Aborted multipart uploads count=0 kept=034862026/11/11 23:30:07 INFO Vacuumed table table=multipart_uploads34872026/11/11 23:30:07 WARN Force mode enabled - objects will be deleted immediately without grace period34882026/11/11 23:30:07 INFO Completed push id=5 objects=8 duration=3.850057ms34892026/11/11 23:30:07 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."34902026/11/11 23:30:07 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."34912026/11/11 23:30:07 INFO Vacuumed table table=closures34922026/11/11 23:30:07 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."34932026/11/11 23:30:07 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."34942026/11/11 23:30:07 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."34952026/11/11 23:30:07 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."34962026/11/11 23:30:07 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."34972026/11/11 23:30:07 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."34982026/11/11 23:30:07 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."34992026/11/11 23:30:07 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."35002026/11/11 23:30:07 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."35012026/11/11 23:30:07 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."35022026/11/11 23:30:07 INFO Vacuumed table table=pending_closures35032026/11/11 23:30:07 INFO Received complete upload request method=POST path=/35042026/11/11 23:30:07 INFO Received uploads request method=POST path=/35052026/11/11 23:30:07 INFO Vacuumed table table=objects35062026/11/11 23:30:07 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."35072026/11/11 23:30:07 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."35082026/11/11 23:30:07 INFO Completed upload id=32 objects=5 duration=852.869µs35092026/11/11 23:30:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035102026/11/11 23:30:07 INFO Vacuumed table table=pending_closures35112026/11/11 23:30:07 INFO Vacuumed table table=pending_objects35122026/11/11 23:30:07 INFO Vacuumed table table=multipart_uploads35132026/11/11 23:30:07 INFO Vacuumed table table=closures35142026/11/11 23:30:07 INFO Received uploads request method=POST path=/35152026/11/11 23:30:07 INFO Vacuumed table table=pending_objects35162026/11/11 23:30:07 INFO Vacuumed table table=multipart_uploads35172026/11/11 23:30:07 INFO Vacuumed table table=objects35182026/11/11 23:30:07 INFO Vacuumed table table=closures35192026/11/11 23:30:07 INFO Vacuumed table table=objects35202026/11/11 23:30:07 INFO Received complete upload request method=POST path=/35212026/11/11 23:30:07 INFO Received uploads request method=POST path=/35222026/11/11 23:30:07 INFO Received uploads request method=POST path=/35232026/11/11 23:30:07 INFO Received push request method=POST path=/35242026/11/11 23:30:07 INFO Completed upload id=2 objects=2 duration=2.361983ms35252026/11/11 23:30:07 INFO Received uploads request method=POST path=/35262026/11/11 23:30:07 INFO Received push request method=POST path=/35272026/11/11 23:30:07 INFO Received push request method=POST path=/35282026/11/11 23:30:07 INFO Received complete push request method=POST path=/35292026/11/11 23:30:07 INFO Received complete upload request method=POST path=/35302026/11/11 23:30:07 INFO Completed upload id=33 objects=3 duration=856.728µs35312026/11/11 23:30:07 INFO Received push request method=POST path=/35322026/11/11 23:30:07 INFO Completed push id=14 objects=3 duration=2.815607ms35332026/11/11 23:30:07 INFO Aborted multipart uploads count=0 kept=035342026/11/11 23:30:07 INFO Received uploads request method=POST path=/35352026/11/11 23:30:07 INFO Received push request method=POST path=/35362026/11/11 23:30:07 INFO Received push request method=POST path=/35372026/11/11 23:30:07 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=11 objects-failed-to-delete=035382026/11/11 23:30:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035392026/11/11 23:30:07 INFO Vacuumed table table=pending_closures35402026/11/11 23:30:07 INFO Vacuumed table table=pending_closures35412026/11/11 23:30:07 INFO Vacuumed table table=pending_objects35422026/11/11 23:30:07 INFO Vacuumed table table=multipart_uploads35432026/11/11 23:30:07 INFO Vacuumed table table=closures35442026/11/11 23:30:07 INFO Vacuumed table table=objects35452026/11/11 23:30:07 INFO Vacuumed table table=pending_objects35462026/11/11 23:30:07 INFO Vacuumed table table=multipart_uploads35472026/11/11 23:30:07 INFO Received push request method=POST path=/35482026/11/11 23:30:07 INFO Vacuumed table table=closures35492026/11/11 23:30:07 INFO Received uploads request method=POST path=/35502026/11/11 23:30:07 INFO Vacuumed table table=objects35512026/11/11 23:30:07 INFO Received complete upload request method=POST path=/35522026/11/11 23:30:07 INFO Received complete push request method=POST path=/35532026/11/11 23:30:07 INFO Aborted multipart uploads count=0 kept=035542026/11/11 23:30:07 INFO Received uploads request method=POST path=/35552026/11/11 23:30:07 INFO Completed upload id=11 objects=6 duration=1.168252ms35562026/11/11 23:30:07 INFO Received uploads request method=POST path=/35572026/11/11 23:30:07 INFO Completed push id=6 objects=5 duration=2.0729ms35582026/11/11 23:30:07 INFO Aborted multipart uploads count=0 kept=035592026/11/11 23:30:07 WARN Force mode enabled - objects will be deleted immediately without grace period35602026/11/11 23:30:07 INFO Received uploads request method=POST path=/35612026/11/11 23:30:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035622026/11/11 23:30:07 INFO Received uploads request method=POST path=/35632026/11/11 23:30:07 INFO Vacuumed table table=pending_closures35642026/11/11 23:30:07 INFO Received push request method=POST path=/35652026/11/11 23:30:07 INFO Received push request method=POST path=/35662026/11/11 23:30:07 INFO Vacuumed table table=pending_objects35672026/11/11 23:30:07 INFO Vacuumed table table=multipart_uploads35682026/11/11 23:30:07 INFO Received complete push request method=POST path=/35692026/11/11 23:30:07 INFO Vacuumed table table=closures35702026/11/11 23:30:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=035712026/11/11 23:30:07 INFO Vacuumed table table=objects35722026/11/11 23:30:07 INFO Received complete push request method=POST path=/35732026/11/11 23:30:07 INFO Vacuumed table table=pending_closures35742026/11/11 23:30:07 INFO Received push request method=POST path=/35752026/11/11 23:30:07 INFO Completed push id=8 objects=6 duration=2.273562ms35762026/11/11 23:30:07 INFO Received complete push request method=POST path=/35772026/11/11 23:30:07 INFO Received complete upload request method=POST path=/35782026/11/11 23:30:07 INFO Received push request method=POST path=/35792026/11/11 23:30:07 INFO Received push request method=POST path=/35802026/11/11 23:30:07 INFO Vacuumed table table=pending_objects35812026/11/11 23:30:07 INFO Completed push id=12 objects=3 duration=1.904778ms35822026/11/11 23:30:07 INFO Vacuumed table table=multipart_uploads35832026/11/11 23:30:07 INFO Vacuumed table table=closures35842026/11/11 23:30:07 INFO Completed push id=5 objects=6 duration=2.06402ms35852026/11/11 23:30:07 INFO Completed upload id=25 objects=16 duration=1.878358ms35862026/11/11 23:30:07 INFO Received complete push request method=POST path=/35872026/11/11 23:30:07 INFO Received push request method=POST path=/35882026/11/11 23:30:07 INFO Vacuumed table table=objects35892026/11/11 23:30:07 INFO Completed push id=11 objects=9 duration=1.465474ms35902026/11/11 23:30:07 INFO Aborted multipart uploads count=0 kept=035912026/11/11 23:30:07 INFO Received uploads request method=POST path=/35922026/11/11 23:30:07 INFO Received push request method=POST path=/35932026/11/11 23:30:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035942026/11/11 23:30:07 INFO Vacuumed table table=pending_closures35952026/11/11 23:30:07 INFO Vacuumed table table=pending_objects35962026/11/11 23:30:07 INFO Vacuumed table table=multipart_uploads35972026/11/11 23:30:07 INFO Aborted multipart uploads count=0 kept=035982026/11/11 23:30:07 INFO Vacuumed table table=closures35992026/11/11 23:30:07 INFO Vacuumed table table=objects36002026/11/11 23:30:07 INFO Aborted multipart uploads count=0 kept=036012026/11/11 23:30:07 INFO Received push request method=POST path=/36022026/11/11 23:30:07 INFO Received uploads request method=POST path=/36032026/11/11 23:30:07 INFO Received complete push request method=POST path=/36042026/11/11 23:30:07 INFO Aborted multipart uploads count=0 kept=036052026/11/11 23:30:07 WARN Force mode enabled - objects will be deleted immediately without grace period36062026/11/11 23:30:07 INFO Completed push id=12 objects=7 duration=1.734556ms36072026/11/11 23:30:07 INFO Received push request method=POST path=/36082026/11/11 23:30:07 WARN Force mode enabled - objects will be deleted immediately without grace period36092026/11/11 23:30:07 INFO Received push request method=POST path=/36102026/11/11 23:30:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036112026/11/11 23:30:07 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036122026/11/11 23:30:07 INFO Vacuumed table table=pending_closures36132026/11/11 23:30:07 INFO Received complete push request method=POST path=/36142026/11/11 23:30:07 INFO Vacuumed table table=pending_closures36152026/11/11 23:30:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=036162026/11/11 23:30:07 INFO Vacuumed table table=pending_objects36172026/11/11 23:30:07 INFO Vacuumed table table=multipart_uploads36182026/11/11 23:30:07 INFO Received push request method=POST path=/36192026/11/11 23:30:07 INFO Completed push id=15 objects=24 duration=1.561755ms36202026/11/11 23:30:07 INFO Vacuumed table table=closures36212026/11/11 23:30:07 INFO Vacuumed table table=pending_objects36222026/11/11 23:30:07 INFO Vacuumed table table=multipart_uploads36232026/11/11 23:30:07 INFO Vacuumed table table=objects36242026/11/11 23:30:07 INFO Vacuumed table table=pending_closures36252026/11/11 23:30:07 INFO Vacuumed table table=closures36262026/11/11 23:30:07 INFO Received complete push request method=POST path=/36272026/11/11 23:30:07 INFO Received complete push request method=POST path=/36282026/11/11 23:30:07 INFO Vacuumed table table=pending_objects36292026/11/11 23:30:07 INFO Vacuumed table table=objects36302026/11/11 23:30:07 INFO Vacuumed table table=multipart_uploads36312026/11/11 23:30:07 INFO Received complete push request method=POST path=/36322026/11/11 23:30:07 INFO Completed push id=12 objects=15 duration=2.046939ms36332026/11/11 23:30:07 INFO Vacuumed table table=closures36342026/11/11 23:30:07 INFO Received push request method=POST path=/36352026/11/11 23:30:07 INFO Received uploads request method=POST path=/36362026/11/11 23:30:07 INFO Vacuumed table table=objects36372026/11/11 23:30:07 INFO Completed push id=13 objects=12 duration=3.267371ms36382026/11/11 23:30:07 INFO Completed push id=14 objects=4 duration=2.670926ms36392026/11/11 23:30:07 INFO Received uploads request method=POST path=/36402026/11/11 23:30:07 INFO Aborted multipart uploads count=0 kept=036412026/11/11 23:30:07 INFO Received push request method=POST path=/36422026/11/11 23:30:07 INFO Received uploads request method=POST path=/36432026/11/11 23:30:07 INFO Received push request method=POST path=/36442026/11/11 23:30:07 INFO Received complete upload request method=POST path=/36452026/11/11 23:30:07 INFO Completed upload id=19 objects=2 duration=750.467µs36462026/11/11 23:30:07 INFO Received push request method=POST path=/36472026/11/11 23:30:07 INFO Received push request method=POST path=/36482026/11/11 23:30:07 INFO Received push request method=POST path=/36492026/11/11 23:30:07 INFO Aborted multipart uploads count=0 kept=036502026/11/11 23:30:07 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=036512026/11/11 23:30:07 INFO Received uploads request method=POST path=/36522026/11/11 23:30:07 INFO Vacuumed table table=pending_closures36532026/11/11 23:30:07 INFO Received push request method=POST path=/36542026/11/11 23:30:07 INFO Vacuumed table table=pending_objects36552026/11/11 23:30:07 INFO Vacuumed table table=multipart_uploads36562026/11/11 23:30:07 INFO Vacuumed table table=closures36572026/11/11 23:30:07 INFO Received push request method=POST path=/36582026/11/11 23:30:07 INFO Received uploads request method=POST path=/36592026/11/11 23:30:07 INFO Received complete upload request method=POST path=/36602026/11/11 23:30:07 INFO Vacuumed table table=objects36612026/11/11 23:30:07 INFO Received complete upload request method=POST path=/36622026/11/11 23:30:07 INFO Completed upload id=24 objects=7 duration=1.04111ms36632026/11/11 23:30:07 INFO Aborted multipart uploads count=0 kept=036642026/11/11 23:30:07 INFO Completed upload id=10 objects=2 duration=2.07062ms36652026/11/11 23:30:07 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=036662026/11/11 23:30:07 INFO Received complete push request method=POST path=/36672026/11/11 23:30:07 INFO Received push request method=POST path=/36682026/11/11 23:30:07 WARN Force mode enabled - objects will be deleted immediately without grace period36692026/11/11 23:30:07 INFO Vacuumed table table=pending_closures36702026/11/11 23:30:07 INFO Vacuumed table table=pending_objects36712026/11/11 23:30:07 INFO Completed push id=7 objects=15 duration=2.696906ms36722026/11/11 23:30:07 INFO Vacuumed table table=multipart_uploads36732026/11/11 23:30:07 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036742026/11/11 23:30:07 INFO Vacuumed table table=closures36752026/11/11 23:30:07 INFO Received uploads request method=POST path=/36762026/11/11 23:30:07 INFO Vacuumed table table=pending_closures36772026/11/11 23:30:07 INFO Vacuumed table table=objects36782026/11/11 23:30:07 INFO Vacuumed table table=pending_objects36792026/11/11 23:30:07 INFO Vacuumed table table=multipart_uploads36802026/11/11 23:30:07 INFO Vacuumed table table=closures36812026/11/11 23:30:07 INFO Received uploads request method=POST path=/36822026/11/11 23:30:07 INFO Received push request method=POST path=/36832026/11/11 23:30:07 INFO Vacuumed table table=objects36842026/11/11 23:30:07 INFO Received uploads request method=POST path=/36852026/11/11 23:30:07 INFO Received push request method=POST path=/36862026/11/11 23:30:07 INFO Received uploads request method=POST path=/36872026/11/11 23:30:07 INFO Received complete push request method=POST path=/36882026/11/11 23:30:07 INFO Completed push id=26 objects=3 duration=860.789µs36892026/11/11 23:30:07 INFO Aborted multipart uploads count=0 kept=036902026/11/11 23:30:07 INFO Received complete upload request method=POST path=/36912026/11/11 23:30:07 INFO Completed upload id=16 objects=23 duration=1.380673ms36922026/11/11 23:30:07 INFO Received complete upload request method=POST path=/36932026/11/11 23:30:07 INFO Completed upload id=8 objects=3 duration=1.878198ms36942026/11/11 23:30:07 INFO Received complete upload request method=POST path=/36952026/11/11 23:30:07 INFO Received complete upload request method=POST path=/36962026/11/11 23:30:07 INFO Completed upload id=15 objects=4 duration=2.07292ms36972026/11/11 23:30:07 INFO Received push request method=POST path=/36982026/11/11 23:30:07 INFO Completed upload id=28 objects=8 duration=1.02543ms36992026/11/11 23:30:07 INFO Received push request method=POST path=/37002026/11/11 23:30:07 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=037012026/11/11 23:30:07 INFO Received complete upload request method=POST path=/37022026/11/11 23:30:07 INFO Vacuumed table table=pending_closures37032026/11/11 23:30:07 INFO Completed upload id=15 objects=6 duration=866.728µs37042026/11/11 23:30:07 INFO Vacuumed table table=pending_objects37052026/11/11 23:30:07 INFO Received push request method=POST path=/37062026/11/11 23:30:07 INFO Vacuumed table table=multipart_uploads37072026/11/11 23:30:07 INFO Vacuumed table table=closures37082026/11/11 23:30:07 INFO Aborted multipart uploads count=0 kept=037092026/11/11 23:30:07 INFO Vacuumed table table=objects37102026/11/11 23:30:07 INFO Received uploads request method=POST path=/37112026/11/11 23:30:07 WARN Force mode enabled - objects will be deleted immediately without grace period37122026/11/11 23:30:07 INFO Received push request method=POST path=/37132026/11/11 23:30:07 INFO Received uploads request method=POST path=/37142026/11/11 23:30:07 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037152026/11/11 23:30:07 INFO Vacuumed table table=pending_closures37162026/11/11 23:30:07 INFO Received complete upload request method=POST path=/37172026/11/11 23:30:07 INFO Aborted multipart uploads count=0 kept=037182026/11/11 23:30:07 INFO Vacuumed table table=pending_objects37192026/11/11 23:30:07 INFO Completed upload id=29 objects=10 duration=717.646µs37202026/11/11 23:30:07 INFO Received push request method=POST path=/37212026/11/11 23:30:07 INFO Vacuumed table table=multipart_uploads37222026/11/11 23:30:07 INFO Vacuumed table table=closures37232026/11/11 23:30:07 INFO Vacuumed table table=objects37242026/11/11 23:30:07 INFO Received uploads request method=POST path=/37252026/11/11 23:30:07 WARN Force mode enabled - objects will be deleted immediately without grace period37262026/11/11 23:30:07 INFO Received uploads request method=POST path=/37272026/11/11 23:30:07 INFO Received push request method=POST path=/37282026/11/11 23:30:07 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037292026/11/11 23:30:07 INFO Vacuumed table table=pending_closures37302026/11/11 23:30:07 INFO Aborted multipart uploads count=0 kept=037312026/11/11 23:30:07 INFO Vacuumed table table=pending_objects37322026/11/11 23:30:07 INFO Vacuumed table table=multipart_uploads37332026/11/11 23:30:07 INFO Vacuumed table table=closures37342026/11/11 23:30:07 INFO Received push request method=POST path=/37352026/11/11 23:30:07 INFO Vacuumed table table=objects37362026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=037372026/11/11 23:30:08 INFO Received uploads request method=POST path=/37382026/11/11 23:30:08 INFO Received complete upload request method=POST path=/37392026/11/11 23:30:08 INFO Received uploads request method=POST path=/37402026/11/11 23:30:08 INFO Received push request method=POST path=/37412026/11/11 23:30:08 INFO Completed upload id=30 objects=20 duration=965.509µs37422026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=037432026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037442026/11/11 23:30:08 INFO Vacuumed table table=pending_closures37452026/11/11 23:30:08 INFO Received complete upload request method=POST path=/37462026/11/11 23:30:08 INFO Received complete upload request method=POST path=/37472026/11/11 23:30:08 INFO Vacuumed table table=pending_objects37482026/11/11 23:30:08 INFO Vacuumed table table=pending_closures37492026/11/11 23:30:08 INFO Completed upload id=17 objects=7 duration=1.132451ms37502026/11/11 23:30:08 INFO Completed upload id=22 objects=5 duration=1.877738ms37512026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads37522026/11/11 23:30:08 INFO Vacuumed table table=closures37532026/11/11 23:30:08 INFO Vacuumed table table=pending_objects37542026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads37552026/11/11 23:30:08 INFO Vacuumed table table=objects37562026/11/11 23:30:08 INFO Vacuumed table table=closures37572026/11/11 23:30:08 INFO Received uploads request method=POST path=/37582026/11/11 23:30:08 INFO Vacuumed table table=objects37592026/11/11 23:30:08 INFO Received complete push request method=POST path=/37602026/11/11 23:30:08 INFO Received complete push request method=POST path=/37612026/11/11 23:30:08 INFO Received uploads request method=POST path=/37622026/11/11 23:30:08 INFO Received complete upload request method=POST path=/37632026/11/11 23:30:08 INFO Completed push id=31 objects=11 duration=1.339093ms37642026/11/11 23:30:08 INFO Completed upload id=16 objects=3 duration=1.06463ms37652026/11/11 23:30:08 INFO Completed push id=19 objects=6 duration=3.283291ms37662026/11/11 23:30:08 INFO Received complete push request method=POST path=/37672026/11/11 23:30:08 INFO Received push request method=POST path=/37682026/11/11 23:30:08 INFO Received complete push request method=POST path=/37692026/11/11 23:30:08 INFO Completed push id=6 objects=6 duration=2.131281ms37702026/11/11 23:30:08 INFO Received complete push request method=POST path=/37712026/11/11 23:30:08 INFO Completed push id=6 objects=26 duration=3.405672ms37722026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=037732026/11/11 23:30:08 INFO Completed push id=28 objects=3 duration=1.551155ms37742026/11/11 23:30:08 INFO Received uploads request method=POST path=/37752026/11/11 23:30:08 INFO Received uploads request method=POST path=/37762026/11/11 23:30:08 INFO Received push request method=POST path=/37772026/11/11 23:30:08 INFO Received complete upload request method=POST path=/37782026/11/11 23:30:08 INFO Received complete push request method=POST path=/37792026/11/11 23:30:08 INFO Completed upload id=32 objects=2 duration=695.386µs37802026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=037812026/11/11 23:30:08 INFO Received push request method=POST path=/37822026/11/11 23:30:08 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."37832026/11/11 23:30:08 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."37842026/11/11 23:30:08 INFO Completed push id=11 objects=8 duration=3.253871ms37852026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037862026/11/11 23:30:08 INFO Received uploads request method=POST path=/37872026/11/11 23:30:08 INFO Vacuumed table table=pending_closures37882026/11/11 23:30:08 INFO Received complete push request method=POST path=/37892026/11/11 23:30:08 INFO Vacuumed table table=pending_objects37902026/11/11 23:30:08 INFO Received complete push request method=POST path=/37912026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads37922026/11/11 23:30:08 INFO Completed push id=29 objects=15 duration=1.165631ms37932026/11/11 23:30:08 INFO Vacuumed table table=closures37942026/11/11 23:30:08 INFO Vacuumed table table=objects37952026/11/11 23:30:08 INFO Received uploads request method=POST path=/37962026/11/11 23:30:08 INFO Completed push id=16 objects=16 duration=3.842296ms37972026/11/11 23:30:08 INFO Received push request method=POST path=/37982026/11/11 23:30:08 INFO Received complete push request method=POST path=/37992026/11/11 23:30:08 INFO Completed push id=33 objects=3 duration=1.04749ms38002026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=038012026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038022026/11/11 23:30:08 INFO Vacuumed table table=pending_closures38032026/11/11 23:30:08 INFO Vacuumed table table=pending_objects38042026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=038052026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads38062026/11/11 23:30:08 INFO Received uploads request method=POST path=/38072026/11/11 23:30:08 INFO Received complete push request method=POST path=/38082026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=038092026/11/11 23:30:08 INFO Vacuumed table table=closures38102026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=038112026/11/11 23:30:08 INFO Completed push id=30 objects=6 duration=1.10377ms38122026/11/11 23:30:08 INFO Received complete push request method=POST path=/38132026/11/11 23:30:08 INFO Vacuumed table table=objects38142026/11/11 23:30:08 INFO Received uploads request method=POST path=/38152026/11/11 23:30:08 WARN Force mode enabled - objects will be deleted immediately without grace period38162026/11/11 23:30:08 WARN Force mode enabled - objects will be deleted immediately without grace period38172026/11/11 23:30:08 WARN Force mode enabled - objects will be deleted immediately without grace period38182026/11/11 23:30:08 INFO Received uploads request method=POST path=/38192026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038202026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038212026/11/11 23:30:08 INFO Received complete upload request method=POST path=/38222026/11/11 23:30:08 INFO Vacuumed table table=pending_closures38232026/11/11 23:30:08 INFO Completed upload id=34 objects=3 duration=753.987µs38242026/11/11 23:30:08 INFO Vacuumed table table=pending_closures38252026/11/11 23:30:08 INFO Vacuumed table table=pending_objects38262026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads38272026/11/11 23:30:08 INFO Vacuumed table table=pending_objects38282026/11/11 23:30:08 INFO Vacuumed table table=closures38292026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads38302026/11/11 23:30:08 INFO Completed push id=3 objects=21 duration=7.756234ms38312026/11/11 23:30:08 INFO Vacuumed table table=closures38322026/11/11 23:30:08 INFO Vacuumed table table=objects38332026/11/11 23:30:08 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."38342026/11/11 23:30:08 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."38352026/11/11 23:30:08 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."38362026/11/11 23:30:08 INFO Received complete upload request method=POST path=/38372026/11/11 23:30:08 INFO Vacuumed table table=objects38382026/11/11 23:30:08 INFO Completed upload id=14 objects=6 duration=873.609µs38392026/11/11 23:30:08 INFO Received complete push request method=POST path=/38402026/11/11 23:30:08 INFO Received complete push request method=POST path=/38412026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=038422026/11/11 23:30:08 INFO Completed push id=10 objects=15 duration=2.176681ms38432026/11/11 23:30:08 INFO Received complete upload request method=POST path=/38442026/11/11 23:30:08 INFO Completed push id=19 objects=14 duration=1.994139ms38452026/11/11 23:30:08 INFO Completed upload id=31 objects=3 duration=983.73µs38462026/11/11 23:30:08 INFO Received push request method=POST path=/38472026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=038482026/11/11 23:30:08 INFO Received uploads request method=POST path=/38492026/11/11 23:30:08 WARN Force mode enabled - objects will be deleted immediately without grace period38502026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=038512026/11/11 23:30:08 INFO Received complete push request method=POST path=/38522026/11/11 23:30:08 WARN Force mode enabled - objects will be deleted immediately without grace period38532026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038542026/11/11 23:30:08 INFO Received uploads request method=POST path=/38552026/11/11 23:30:08 INFO Received push request method=POST path=/38562026/11/11 23:30:08 INFO Received push request method=POST path=/38572026/11/11 23:30:08 INFO Vacuumed table table=pending_closures38582026/11/11 23:30:08 INFO Completed push id=13 objects=6 duration=2.809246ms38592026/11/11 23:30:08 INFO Vacuumed table table=pending_objects38602026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=038612026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads38622026/11/11 23:30:08 INFO Vacuumed table table=closures38632026/11/11 23:30:08 INFO Received complete upload request method=POST path=/38642026/11/11 23:30:08 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."38652026/11/11 23:30:08 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."38662026/11/11 23:30:08 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."38672026/11/11 23:30:08 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."38682026/11/11 23:30:08 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."38692026/11/11 23:30:08 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."38702026/11/11 23:30:08 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."38712026/11/11 23:30:08 INFO Vacuumed table table=objects38722026/11/11 23:30:08 INFO Completed upload id=15 objects=6 duration=1.298332ms38732026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=038742026/11/11 23:30:08 INFO Received complete upload request method=POST path=/38752026/11/11 23:30:08 WARN Force mode enabled - objects will be deleted immediately without grace period38762026/11/11 23:30:08 INFO Completed upload id=18 objects=4 duration=1.978499ms38772026/11/11 23:30:08 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."38782026/11/11 23:30:08 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."38792026/11/11 23:30:08 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."38802026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=038812026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=038822026/11/11 23:30:08 WARN Force mode enabled - objects will be deleted immediately without grace period38832026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038842026/11/11 23:30:08 INFO Received push request method=POST path=/38852026/11/11 23:30:08 INFO Received push request method=POST path=/38862026/11/11 23:30:08 WARN Force mode enabled - objects will be deleted immediately without grace period38872026/11/11 23:30:08 WARN Force mode enabled - objects will be deleted immediately without grace period38882026/11/11 23:30:08 INFO Vacuumed table table=pending_closures38892026/11/11 23:30:08 INFO Vacuumed table table=pending_objects38902026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads38912026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038922026/11/11 23:30:08 INFO Vacuumed table table=closures38932026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=038942026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038952026/11/11 23:30:08 INFO Vacuumed table table=objects38962026/11/11 23:30:08 INFO Vacuumed table table=pending_closures38972026/11/11 23:30:08 INFO Vacuumed table table=pending_objects38982026/11/11 23:30:08 INFO Vacuumed table table=pending_closures38992026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads39002026/11/11 23:30:08 INFO Vacuumed table table=closures39012026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039022026/11/11 23:30:08 INFO Vacuumed table table=objects39032026/11/11 23:30:08 INFO Received uploads request method=POST path=/39042026/11/11 23:30:08 INFO Vacuumed table table=pending_objects39052026/11/11 23:30:08 INFO Vacuumed table table=pending_closures39062026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads39072026/11/11 23:30:08 INFO Vacuumed table table=closures39082026/11/11 23:30:08 INFO Vacuumed table table=pending_objects39092026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads39102026/11/11 23:30:08 INFO Vacuumed table table=objects39112026/11/11 23:30:08 INFO Received complete upload request method=POST path=/39122026/11/11 23:30:08 INFO Vacuumed table table=closures39132026/11/11 23:30:08 INFO Received push request method=POST path=/39142026/11/11 23:30:08 INFO Vacuumed table table=objects39152026/11/11 23:30:08 INFO Completed upload id=7 objects=3 duration=1.024329ms39162026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=039172026/11/11 23:30:08 INFO Received uploads request method=POST path=/39182026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039192026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=039202026/11/11 23:30:08 INFO Vacuumed table table=pending_closures39212026/11/11 23:30:08 INFO Received push request method=POST path=/39222026/11/11 23:30:08 INFO Vacuumed table table=pending_objects39232026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads39242026/11/11 23:30:08 INFO Received uploads request method=POST path=/39252026/11/11 23:30:08 INFO Vacuumed table table=closures39262026/11/11 23:30:08 INFO Vacuumed table table=objects39272026/11/11 23:30:08 INFO Received push request method=POST path=/39282026/11/11 23:30:08 WARN Force mode enabled - objects will be deleted immediately without grace period39292026/11/11 23:30:08 INFO Received complete push request method=POST path=/39302026/11/11 23:30:08 INFO Completed push id=17 objects=6 duration=1.581615ms39312026/11/11 23:30:08 INFO Received complete push request method=POST path=/39322026/11/11 23:30:08 INFO Completed push id=19 objects=8 duration=2.942608ms39332026/11/11 23:30:08 INFO Received complete push request method=POST path=/39342026/11/11 23:30:08 INFO Received push request method=POST path=/39352026/11/11 23:30:08 INFO Received uploads request method=POST path=/39362026/11/11 23:30:08 INFO Completed push id=18 objects=23 duration=2.256362ms39372026/11/11 23:30:08 INFO Received push request method=POST path=/39382026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=039392026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039402026/11/11 23:30:08 INFO Received uploads request method=POST path=/39412026/11/11 23:30:08 INFO Received complete push request method=POST path=/39422026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=039432026/11/11 23:30:08 INFO Vacuumed table table=pending_closures39442026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=039452026/11/11 23:30:08 INFO Vacuumed table table=pending_closures39462026/11/11 23:30:08 INFO Vacuumed table table=pending_objects39472026/11/11 23:30:08 INFO Completed push id=15 objects=7 duration=1.494774ms39482026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads39492026/11/11 23:30:08 INFO Vacuumed table table=closures39502026/11/11 23:30:08 INFO Received uploads request method=POST path=/39512026/11/11 23:30:08 INFO Received complete push request method=POST path=/39522026/11/11 23:30:08 WARN Force mode enabled - objects will be deleted immediately without grace period39532026/11/11 23:30:08 INFO Vacuumed table table=objects39542026/11/11 23:30:08 INFO Vacuumed table table=pending_objects39552026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads39562026/11/11 23:30:08 INFO Vacuumed table table=closures39572026/11/11 23:30:08 INFO Completed push id=17 objects=6 duration=1.968938ms39582026/11/11 23:30:08 INFO Vacuumed table table=objects39592026/11/11 23:30:08 INFO Received complete push request method=POST path=/39602026/11/11 23:30:08 INFO Received push request method=POST path=/39612026/11/11 23:30:08 INFO Received uploads request method=POST path=/39622026/11/11 23:30:08 INFO Completed push id=16 objects=27 duration=1.601876ms39632026/11/11 23:30:08 INFO Received push request method=POST path=/39642026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=039652026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=039662026/11/11 23:30:08 INFO Vacuumed table table=pending_closures39672026/11/11 23:30:08 INFO Received complete push request method=POST path=/39682026/11/11 23:30:08 INFO Vacuumed table table=pending_objects39692026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads39702026/11/11 23:30:08 INFO Vacuumed table table=closures39712026/11/11 23:30:08 INFO Received uploads request method=POST path=/39722026/11/11 23:30:08 INFO Received complete push request method=POST path=/39732026/11/11 23:30:08 INFO Vacuumed table table=objects39742026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039752026/11/11 23:30:08 INFO Vacuumed table table=pending_closures39762026/11/11 23:30:08 INFO Completed push id=9 objects=4 duration=3.041229ms39772026/11/11 23:30:08 INFO Received complete push request method=POST path=/39782026/11/11 23:30:08 INFO Completed push id=15 objects=5 duration=2.04296ms39792026/11/11 23:30:08 INFO Received push request method=POST path=/39802026/11/11 23:30:08 INFO Vacuumed table table=pending_objects39812026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads39822026/11/11 23:30:08 INFO Vacuumed table table=closures39832026/11/11 23:30:08 INFO Completed push id=11 objects=9 duration=1.374913ms39842026/11/11 23:30:08 INFO Vacuumed table table=objects39852026/11/11 23:30:08 INFO Received uploads request method=POST path=/39862026/11/11 23:30:08 INFO Received uploads request method=POST path=/39872026/11/11 23:30:08 INFO Received uploads request method=POST path=/39882026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=039892026/11/11 23:30:08 INFO Received complete push request method=POST path=/39902026/11/11 23:30:08 INFO Completed push id=4 objects=7 duration=1.257932ms39912026/11/11 23:30:08 INFO Received push request method=POST path=/39922026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039932026/11/11 23:30:08 INFO Vacuumed table table=pending_closures39942026/11/11 23:30:08 INFO Vacuumed table table=pending_objects39952026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads39962026/11/11 23:30:08 INFO Received uploads request method=POST path=/39972026/11/11 23:30:08 INFO Vacuumed table table=closures39982026/11/11 23:30:08 INFO Vacuumed table table=objects39992026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=040002026/11/11 23:30:08 INFO Received uploads request method=POST path=/40012026/11/11 23:30:08 INFO Received push request method=POST path=/40022026/11/11 23:30:08 WARN Force mode enabled - objects will be deleted immediately without grace period40032026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040042026/11/11 23:30:08 INFO Vacuumed table table=pending_closures40052026/11/11 23:30:08 INFO Received uploads request method=POST path=/40062026/11/11 23:30:08 INFO Vacuumed table table=pending_objects40072026/11/11 23:30:08 INFO Received push request method=POST path=/40082026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads40092026/11/11 23:30:08 INFO Vacuumed table table=closures40102026/11/11 23:30:08 INFO Vacuumed table table=objects40112026/11/11 23:30:08 INFO Received push request method=POST path=/40122026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=040132026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=040142026/11/11 23:30:08 INFO Received uploads request method=POST path=/40152026/11/11 23:30:08 INFO Received push request method=POST path=/40162026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040172026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=040182026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040192026/11/11 23:30:08 INFO Vacuumed table table=pending_closures40202026/11/11 23:30:08 INFO Vacuumed table table=pending_closures40212026/11/11 23:30:08 WARN Force mode enabled - objects will be deleted immediately without grace period40222026/11/11 23:30:08 INFO Received uploads request method=POST path=/40232026/11/11 23:30:08 INFO Vacuumed table table=pending_objects40242026/11/11 23:30:08 INFO Vacuumed table table=pending_objects40252026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads40262026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads40272026/11/11 23:30:08 INFO Vacuumed table table=closures40282026/11/11 23:30:08 INFO Vacuumed table table=closures40292026/11/11 23:30:08 INFO Vacuumed table table=objects40302026/11/11 23:30:08 INFO Received push request method=POST path=/40312026/11/11 23:30:08 INFO Vacuumed table table=objects40322026/11/11 23:30:08 INFO Received push request method=POST path=/40332026/11/11 23:30:08 INFO Received complete push request method=POST path=/40342026/11/11 23:30:08 INFO Received uploads request method=POST path=/40352026/11/11 23:30:08 INFO Completed push id=18 objects=10 duration=1.466214ms40362026/11/11 23:30:08 INFO Received uploads request method=POST path=/40372026/11/11 23:30:08 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."40382026/11/11 23:30:08 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."40392026/11/11 23:30:08 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."40402026/11/11 23:30:08 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."40412026/11/11 23:30:08 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."40422026/11/11 23:30:08 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."40432026/11/11 23:30:08 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."40442026/11/11 23:30:08 INFO Received complete upload request method=POST path=/40452026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=040462026/11/11 23:30:08 INFO Completed upload id=22 objects=2 duration=1.826258ms40472026/11/11 23:30:08 WARN Force mode enabled - objects will be deleted immediately without grace period40482026/11/11 23:30:08 INFO Received complete push request method=POST path=/40492026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=040502026/11/11 23:30:08 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."40512026/11/11 23:30:08 WARN Force mode enabled - objects will be deleted immediately without grace period40522026/11/11 23:30:08 INFO Received uploads request method=POST path=/40532026/11/11 23:30:08 INFO Completed push id=9 objects=7 duration=3.759056ms40542026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040552026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=040562026/11/11 23:30:08 INFO Vacuumed table table=pending_closures40572026/11/11 23:30:08 INFO Vacuumed table table=pending_objects40582026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads40592026/11/11 23:30:08 INFO Vacuumed table table=closures40602026/11/11 23:30:08 INFO Vacuumed table table=objects40612026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=040622026/11/11 23:30:08 INFO Vacuumed table table=pending_closures40632026/11/11 23:30:08 INFO Received complete upload request method=POST path=/40642026/11/11 23:30:08 INFO Vacuumed table table=pending_objects40652026/11/11 23:30:08 INFO Completed upload id=21 objects=3 duration=1.457833ms40662026/11/11 23:30:08 INFO Received complete push request method=POST path=/40672026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads40682026/11/11 23:30:08 INFO Vacuumed table table=closures40692026/11/11 23:30:08 INFO Completed push id=7 objects=7 duration=1.538795ms40702026/11/11 23:30:08 INFO Vacuumed table table=objects40712026/11/11 23:30:08 INFO Received uploads request method=POST path=/40722026/11/11 23:30:08 INFO Received uploads request method=POST path=/40732026/11/11 23:30:08 INFO Received uploads request method=POST path=/40742026/11/11 23:30:08 INFO Received complete upload request method=POST path=/40752026/11/11 23:30:08 INFO Completed upload id=10 objects=7 duration=1.750416ms40762026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=040772026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=040782026/11/11 23:30:08 WARN Force mode enabled - objects will be deleted immediately without grace period40792026/11/11 23:30:08 INFO Received uploads request method=POST path=/40802026/11/11 23:30:08 INFO Received uploads request method=POST path=/40812026/11/11 23:30:08 INFO Received complete upload request method=POST path=/40822026/11/11 23:30:08 INFO Received push request method=POST path=/40832026/11/11 23:30:08 WARN Force mode enabled - objects will be deleted immediately without grace period40842026/11/11 23:30:08 INFO Completed upload id=14 objects=8 duration=1.926458ms40852026/11/11 23:30:08 INFO Received complete upload request method=POST path=/40862026/11/11 23:30:08 INFO Received push request method=POST path=/40872026/11/11 23:30:08 INFO Completed upload id=13 objects=14 duration=2.765327ms40882026/11/11 23:30:08 INFO Received uploads request method=POST path=/40892026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=7 objects-failed-to-delete=040902026/11/11 23:30:08 INFO Vacuumed table table=pending_closures40912026/11/11 23:30:08 INFO Vacuumed table table=pending_objects40922026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads40932026/11/11 23:30:08 INFO Vacuumed table table=closures40942026/11/11 23:30:08 INFO Vacuumed table table=objects40952026/11/11 23:30:08 INFO Received uploads request method=POST path=/40962026/11/11 23:30:08 INFO Received complete push request method=POST path=/40972026/11/11 23:30:08 INFO Completed push id=17 objects=22 duration=2.11366ms40982026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=040992026/11/11 23:30:08 INFO Received push request method=POST path=/41002026/11/11 23:30:08 INFO Received push request method=POST path=/41012026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041022026/11/11 23:30:08 INFO Vacuumed table table=pending_closures41032026/11/11 23:30:08 INFO Vacuumed table table=pending_objects41042026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads41052026/11/11 23:30:08 INFO Received complete upload request method=POST path=/41062026/11/11 23:30:08 INFO Vacuumed table table=closures41072026/11/11 23:30:08 INFO Completed upload id=18 objects=11 duration=939.969µs41082026/11/11 23:30:08 INFO Vacuumed table table=objects41092026/11/11 23:30:08 INFO Received uploads request method=POST path=/41102026/11/11 23:30:08 INFO Received push request method=POST path=/41112026/11/11 23:30:08 INFO Received uploads request method=POST path=/41122026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=041132026/11/11 23:30:08 INFO Received push request method=POST path=/41142026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=041152026/11/11 23:30:08 INFO Received uploads request method=POST path=/41162026/11/11 23:30:08 WARN Force mode enabled - objects will be deleted immediately without grace period41172026/11/11 23:30:08 INFO Received complete upload request method=POST path=/41182026/11/11 23:30:08 WARN Force mode enabled - objects will be deleted immediately without grace period41192026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041202026/11/11 23:30:08 INFO Completed upload id=23 objects=15 duration=1.593955ms41212026/11/11 23:30:08 INFO Vacuumed table table=pending_closures41222026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041232026/11/11 23:30:08 INFO Vacuumed table table=pending_closures41242026/11/11 23:30:08 INFO Vacuumed table table=pending_objects41252026/11/11 23:30:08 INFO Vacuumed table table=pending_objects41262026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads41272026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads41282026/11/11 23:30:08 INFO Received complete push request method=POST path=/41292026/11/11 23:30:08 INFO Vacuumed table table=closures41302026/11/11 23:30:08 INFO Vacuumed table table=closures41312026/11/11 23:30:08 INFO Received complete upload request method=POST path=/41322026/11/11 23:30:08 INFO Vacuumed table table=objects41332026/11/11 23:30:08 INFO Vacuumed table table=objects41342026/11/11 23:30:08 INFO Completed upload id=18 objects=4 duration=1.566335ms41352026/11/11 23:30:08 INFO Completed push id=31 objects=6 duration=5.103429ms41362026/11/11 23:30:08 INFO Received push request method=POST path=/41372026/11/11 23:30:08 INFO Received push request method=POST path=/41382026/11/11 23:30:08 INFO Received push request method=POST path=/41392026/11/11 23:30:08 INFO Received push request method=POST path=/41402026/11/11 23:30:08 INFO Received uploads request method=POST path=/41412026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=041422026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=041432026/11/11 23:30:08 WARN Force mode enabled - objects will be deleted immediately without grace period41442026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041452026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=041462026/11/11 23:30:08 INFO Vacuumed table table=pending_closures41472026/11/11 23:30:08 INFO Vacuumed table table=pending_objects41482026/11/11 23:30:08 INFO Received complete push request method=POST path=/41492026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads41502026/11/11 23:30:08 INFO Vacuumed table table=closures41512026/11/11 23:30:08 INFO Vacuumed table table=objects41522026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041532026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=041542026/11/11 23:30:08 INFO Vacuumed table table=pending_closures41552026/11/11 23:30:08 INFO Vacuumed table table=pending_closures41562026/11/11 23:30:08 INFO Completed push id=22 objects=3 duration=2.989169ms41572026/11/11 23:30:08 INFO Received complete upload request method=POST path=/41582026/11/11 23:30:08 INFO Vacuumed table table=pending_objects41592026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads41602026/11/11 23:30:08 INFO Vacuumed table table=pending_objects41612026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads41622026/11/11 23:30:08 INFO Vacuumed table table=closures41632026/11/11 23:30:08 INFO Completed upload id=19 objects=6 duration=1.356913ms41642026/11/11 23:30:08 INFO Vacuumed table table=closures41652026/11/11 23:30:08 INFO Vacuumed table table=objects41662026/11/11 23:30:08 INFO Received complete upload request method=POST path=/41672026/11/11 23:30:08 INFO Received uploads request method=POST path=/41682026/11/11 23:30:08 INFO Vacuumed table table=objects41692026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=041702026/11/11 23:30:08 INFO Completed upload id=14 objects=16 duration=2.377263ms41712026/11/11 23:30:08 WARN Force mode enabled - objects will be deleted immediately without grace period41722026/11/11 23:30:08 INFO Received push request method=POST path=/41732026/11/11 23:30:08 INFO Received complete upload request method=POST path=/41742026/11/11 23:30:08 INFO Received uploads request method=POST path=/41752026/11/11 23:30:08 INFO Completed upload id=7 objects=10 duration=2.985728ms41762026/11/11 23:30:08 INFO Received push request method=POST path=/41772026/11/11 23:30:08 INFO Received complete upload request method=POST path=/41782026/11/11 23:30:08 INFO Completed upload id=22 objects=7 duration=1.04289ms41792026/11/11 23:30:08 INFO Received complete push request method=POST path=/41802026/11/11 23:30:08 INFO Completed push id=14 objects=3 duration=2.961068ms41812026/11/11 23:30:08 INFO Received push request method=POST path=/41822026/11/11 23:30:08 INFO Received uploads request method=POST path=/41832026/11/11 23:30:08 INFO Received uploads request method=POST path=/41842026/11/11 23:30:08 INFO Received push request method=POST path=/41852026/11/11 23:30:08 INFO Received complete push request method=POST path=/41862026/11/11 23:30:08 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."41872026/11/11 23:30:08 INFO Received complete push request method=POST path=/41882026/11/11 23:30:08 INFO Received push request method=POST path=/41892026/11/11 23:30:08 INFO Completed push id=17 objects=11 duration=2.695686ms41902026/11/11 23:30:08 INFO Completed push id=21 objects=9 duration=2.659886ms41912026/11/11 23:30:08 INFO Received complete push request method=POST path=/41922026/11/11 23:30:08 INFO Completed push id=1 objects=9 duration=1.313772ms41932026/11/11 23:30:08 INFO Received complete upload request method=POST path=/41942026/11/11 23:30:08 INFO Received complete push request method=POST path=/41952026/11/11 23:30:08 INFO Completed push id=24 objects=3 duration=1.089811ms41962026/11/11 23:30:08 INFO Received uploads request method=POST path=/41972026/11/11 23:30:08 INFO Completed upload id=16 objects=17 duration=2.730726ms41982026/11/11 23:30:08 INFO Received uploads request method=POST path=/41992026/11/11 23:30:08 INFO Received push request method=POST path=/42002026/11/11 23:30:08 INFO Received complete push request method=POST path=/42012026/11/11 23:30:08 INFO Received complete push request method=POST path=/42022026/11/11 23:30:08 INFO Received push request method=POST path=/42032026/11/11 23:30:08 INFO Completed push id=26 objects=14 duration=2.717806ms42042026/11/11 23:30:08 INFO Completed push id=24 objects=9 duration=2.10868ms42052026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=042062026/11/11 23:30:08 INFO Received complete push request method=POST path=/42072026/11/11 23:30:08 WARN Force mode enabled - objects will be deleted immediately without grace period42082026/11/11 23:30:08 INFO Completed push id=1 objects=11 duration=2.516824ms42092026/11/11 23:30:08 INFO Received push request method=POST path=/42102026/11/11 23:30:08 INFO Received complete push request method=POST path=/42112026/11/11 23:30:08 INFO Completed push id=15 objects=4 duration=1.921439ms42122026/11/11 23:30:08 INFO Received push request method=POST path=/42132026/11/11 23:30:08 INFO Received push request method=POST path=/42142026/11/11 23:30:08 INFO Received uploads request method=POST path=/42152026/11/11 23:30:08 INFO Received uploads request method=POST path=/42162026/11/11 23:30:08 INFO Received complete upload request method=POST path=/42172026/11/11 23:30:08 INFO Completed upload id=11 objects=7 duration=1.349893ms42182026/11/11 23:30:08 INFO Received push request method=POST path=/42192026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=042202026/11/11 23:30:08 INFO Received push request method=POST path=/42212026/11/11 23:30:08 WARN Force mode enabled - objects will be deleted immediately without grace period42222026/11/11 23:30:08 INFO Received complete upload request method=POST path=/42232026/11/11 23:30:08 INFO Received push request method=POST path=/42242026/11/11 23:30:08 INFO Completed upload id=16 objects=3 duration=1.102811ms42252026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042262026/11/11 23:30:08 INFO Received uploads request method=POST path=/42272026/11/11 23:30:08 INFO Received uploads request method=POST path=/42282026/11/11 23:30:08 INFO Vacuumed table table=pending_closures42292026/11/11 23:30:08 INFO Received uploads request method=POST path=/42302026/11/11 23:30:08 INFO Received complete upload request method=POST path=/42312026/11/11 23:30:08 INFO Vacuumed table table=pending_objects42322026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads42332026/11/11 23:30:08 INFO Received complete upload request method=POST path=/42342026/11/11 23:30:08 INFO Vacuumed table table=closures42352026/11/11 23:30:08 INFO Completed upload id=2 objects=7 duration=1.340653ms42362026/11/11 23:30:08 INFO Vacuumed table table=objects42372026/11/11 23:30:08 INFO Completed upload id=26 objects=6 duration=870.068µs42382026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=042392026/11/11 23:30:08 WARN Force mode enabled - objects will be deleted immediately without grace period42402026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042412026/11/11 23:30:08 INFO Vacuumed table table=pending_closures42422026/11/11 23:30:08 INFO Received push request method=POST path=/42432026/11/11 23:30:08 INFO Vacuumed table table=pending_objects42442026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads42452026/11/11 23:30:08 INFO Vacuumed table table=closures42462026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=042472026/11/11 23:30:08 INFO Received uploads request method=POST path=/42482026/11/11 23:30:08 INFO Vacuumed table table=objects42492026/11/11 23:30:08 INFO Received uploads request method=POST path=/42502026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=8 objects-deleted-after-grace-period=8 objects-failed-to-delete=042512026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042522026/11/11 23:30:08 INFO Vacuumed table table=pending_closures42532026/11/11 23:30:08 INFO Vacuumed table table=pending_objects42542026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads42552026/11/11 23:30:08 INFO Vacuumed table table=pending_closures42562026/11/11 23:30:08 INFO Vacuumed table table=closures42572026/11/11 23:30:08 INFO Vacuumed table table=objects42582026/11/11 23:30:08 INFO Vacuumed table table=pending_objects42592026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads42602026/11/11 23:30:08 INFO Vacuumed table table=closures42612026/11/11 23:30:08 INFO Vacuumed table table=objects42622026/11/11 23:30:08 INFO Received push request method=POST path=/42632026/11/11 23:30:08 INFO Received push request method=POST path=/42642026/11/11 23:30:08 INFO Received uploads request method=POST path=/42652026/11/11 23:30:08 INFO Received complete upload request method=POST path=/42662026/11/11 23:30:08 INFO Received complete push request method=POST path=/42672026/11/11 23:30:08 INFO Completed upload id=29 objects=5 duration=882.308µs42682026/11/11 23:30:08 INFO Completed push id=14 objects=3 duration=1.107551ms42692026/11/11 23:30:08 INFO Received complete upload request method=POST path=/42702026/11/11 23:30:08 INFO Completed upload id=8 objects=6 duration=1.12971ms42712026/11/11 23:30:08 INFO Received uploads request method=POST path=/42722026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=042732026/11/11 23:30:08 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."42742026/11/11 23:30:08 INFO Received push request method=POST path=/42752026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=042762026/11/11 23:30:08 WARN Force mode enabled - objects will be deleted immediately without grace period42772026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042782026/11/11 23:30:08 INFO Vacuumed table table=pending_closures42792026/11/11 23:30:08 INFO Vacuumed table table=pending_objects42802026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads42812026/11/11 23:30:08 INFO Vacuumed table table=closures42822026/11/11 23:30:08 INFO Received push request method=POST path=/42832026/11/11 23:30:08 INFO Vacuumed table table=objects42842026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=042852026/11/11 23:30:08 INFO Vacuumed table table=pending_closures42862026/11/11 23:30:08 INFO Vacuumed table table=pending_objects42872026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads42882026/11/11 23:30:08 INFO Vacuumed table table=closures42892026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=042902026/11/11 23:30:08 INFO Vacuumed table table=objects42912026/11/11 23:30:08 WARN Force mode enabled - objects will be deleted immediately without grace period42922026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042932026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=042942026/11/11 23:30:08 INFO Received complete upload request method=POST path=/42952026/11/11 23:30:08 INFO Received push request method=POST path=/42962026/11/11 23:30:08 INFO Vacuumed table table=pending_closures42972026/11/11 23:30:08 INFO Completed upload id=15 objects=3 duration=2.742526ms42982026/11/11 23:30:08 INFO Vacuumed table table=pending_objects42992026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads43002026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043012026/11/11 23:30:08 INFO Vacuumed table table=closures43022026/11/11 23:30:08 INFO Received complete upload request method=POST path=/43032026/11/11 23:30:08 INFO Vacuumed table table=pending_closures43042026/11/11 23:30:08 INFO Vacuumed table table=objects43052026/11/11 23:30:08 INFO Completed upload id=20 objects=4 duration=804.947µs43062026/11/11 23:30:08 INFO Vacuumed table table=pending_objects43072026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads43082026/11/11 23:30:08 INFO Vacuumed table table=closures43092026/11/11 23:30:08 INFO Vacuumed table table=objects43102026/11/11 23:30:08 INFO Received complete upload request method=POST path=/43112026/11/11 23:30:08 INFO Completed upload id=12 objects=9 duration=1.06535ms43122026/11/11 23:30:08 INFO Received push request method=POST path=/43132026/11/11 23:30:08 INFO Received uploads request method=POST path=/43142026/11/11 23:30:08 INFO Received push request method=POST path=/43152026/11/11 23:30:08 INFO Received complete push request method=POST path=/43162026/11/11 23:30:08 INFO Completed push id=21 objects=3 duration=1.184811ms43172026/11/11 23:30:08 INFO Received push request method=POST path=/43182026/11/11 23:30:08 INFO Received complete upload request method=POST path=/43192026/11/11 23:30:08 INFO Received complete push request method=POST path=/43202026/11/11 23:30:08 INFO Completed upload id=19 objects=25 duration=2.935168ms43212026/11/11 23:30:08 INFO Completed push id=16 objects=5 duration=1.09331ms43222026/11/11 23:30:08 INFO Received complete push request method=POST path=/43232026/11/11 23:30:08 INFO Completed push id=4 objects=6 duration=1.808637ms43242026/11/11 23:30:08 INFO Received complete upload request method=POST path=/43252026/11/11 23:30:08 INFO Received complete upload request method=POST path=/43262026/11/11 23:30:08 INFO Completed upload id=26 objects=2 duration=1.771796ms43272026/11/11 23:30:08 INFO Completed upload id=33 objects=5 duration=2.705346ms43282026/11/11 23:30:08 INFO Received uploads request method=POST path=/43292026/11/11 23:30:08 INFO Received complete push request method=POST path=/43302026/11/11 23:30:08 INFO Completed push id=13 objects=15 duration=1.482354ms43312026/11/11 23:30:08 INFO Received uploads request method=POST path=/43322026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=043332026/11/11 23:30:08 INFO Received uploads request method=POST path=/43342026/11/11 23:30:08 WARN Force mode enabled - objects will be deleted immediately without grace period43352026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=043362026/11/11 23:30:08 INFO Received uploads request method=POST path=/43372026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043382026/11/11 23:30:08 INFO Vacuumed table table=pending_closures43392026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=043402026/11/11 23:30:08 INFO Vacuumed table table=pending_objects43412026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads43422026/11/11 23:30:08 INFO Vacuumed table table=pending_closures43432026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=043442026/11/11 23:30:08 INFO Vacuumed table table=closures43452026/11/11 23:30:08 INFO Vacuumed table table=objects43462026/11/11 23:30:08 INFO Vacuumed table table=pending_objects43472026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=043482026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads43492026/11/11 23:30:08 INFO Vacuumed table table=closures43502026/11/11 23:30:08 INFO Vacuumed table table=objects43512026/11/11 23:30:08 INFO Received push request method=POST path=/43522026/11/11 23:30:08 INFO Received complete upload request method=POST path=/43532026/11/11 23:30:08 INFO Completed upload id=13 objects=8 duration=1.305872ms43542026/11/11 23:30:08 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."43552026/11/11 23:30:08 INFO Received complete push request method=POST path=/43562026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=043572026/11/11 23:30:08 INFO Vacuumed table table=pending_closures43582026/11/11 23:30:08 INFO Vacuumed table table=pending_objects43592026/11/11 23:30:08 INFO Completed push id=17 objects=22 duration=2.329622ms43602026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads43612026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=043622026/11/11 23:30:08 INFO Vacuumed table table=closures43632026/11/11 23:30:08 INFO Vacuumed table table=objects43642026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=043652026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=3 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043662026/11/11 23:30:08 INFO Vacuumed table table=pending_closures43672026/11/11 23:30:08 INFO Received push request method=POST path=/43682026/11/11 23:30:08 INFO Received complete upload request method=POST path=/43692026/11/11 23:30:08 INFO Received push request method=POST path=/43702026/11/11 23:30:08 INFO Vacuumed table table=pending_objects43712026/11/11 23:30:08 INFO Received uploads request method=POST path=/43722026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads43732026/11/11 23:30:08 WARN Force mode enabled - objects will be deleted immediately without grace period43742026/11/11 23:30:08 INFO Vacuumed table table=closures43752026/11/11 23:30:08 INFO Vacuumed table table=objects43762026/11/11 23:30:08 INFO Completed upload id=31 objects=3 duration=3.04669ms43772026/11/11 23:30:08 INFO Received uploads request method=POST path=/43782026/11/11 23:30:08 INFO Received complete upload request method=POST path=/43792026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=043802026/11/11 23:30:08 INFO Completed upload id=21 objects=25 duration=1.777737ms43812026/11/11 23:30:08 INFO Received push request method=POST path=/43822026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043832026/11/11 23:30:08 INFO Vacuumed table table=pending_closures43842026/11/11 23:30:08 INFO Vacuumed table table=pending_objects43852026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads43862026/11/11 23:30:08 INFO Vacuumed table table=closures43872026/11/11 23:30:08 INFO Vacuumed table table=objects43882026/11/11 23:30:08 INFO Received complete upload request method=POST path=/43892026/11/11 23:30:08 INFO Completed upload id=18 objects=22 duration=1.315452ms43902026/11/11 23:30:08 INFO Received complete upload request method=POST path=/43912026/11/11 23:30:08 INFO Received complete push request method=POST path=/43922026/11/11 23:30:08 INFO Completed upload id=22 objects=2 duration=913.869µs43932026/11/11 23:30:08 INFO Received uploads request method=POST path=/43942026/11/11 23:30:08 INFO Received complete push request method=POST path=/43952026/11/11 23:30:08 INFO Received push request method=POST path=/43962026/11/11 23:30:08 INFO Received uploads request method=POST path=/43972026/11/11 23:30:08 INFO Completed push id=27 objects=25 duration=3.566874ms43982026/11/11 23:30:08 INFO Completed push id=20 objects=3 duration=6.056158ms43992026/11/11 23:30:08 INFO Received uploads request method=POST path=/44002026/11/11 23:30:08 INFO Received uploads request method=POST path=/44012026/11/11 23:30:08 INFO Received complete upload request method=POST path=/44022026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=044032026/11/11 23:30:08 INFO Completed upload id=21 objects=8 duration=995.569µs44042026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=044052026/11/11 23:30:08 INFO Received push request method=POST path=/44062026/11/11 23:30:08 WARN Force mode enabled - objects will be deleted immediately without grace period44072026/11/11 23:30:08 INFO Received push request method=POST path=/44082026/11/11 23:30:08 INFO Received uploads request method=POST path=/44092026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=044102026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044112026/11/11 23:30:08 INFO Vacuumed table table=pending_closures44122026/11/11 23:30:08 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."44132026/11/11 23:30:08 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."44142026/11/11 23:30:08 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."44152026/11/11 23:30:08 INFO Vacuumed table table=pending_objects44162026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads44172026/11/11 23:30:08 INFO Vacuumed table table=closures44182026/11/11 23:30:08 INFO Vacuumed table table=objects44192026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044202026/11/11 23:30:08 INFO Vacuumed table table=pending_closures44212026/11/11 23:30:08 INFO Vacuumed table table=pending_objects44222026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads44232026/11/11 23:30:08 INFO Vacuumed table table=closures44242026/11/11 23:30:08 INFO Vacuumed table table=objects44252026/11/11 23:30:08 INFO Received complete upload request method=POST path=/44262026/11/11 23:30:08 INFO Completed upload id=22 objects=7 duration=911.709µs44272026/11/11 23:30:08 INFO Received push request method=POST path=/44282026/11/11 23:30:08 INFO Received complete push request method=POST path=/44292026/11/11 23:30:08 INFO Received push request method=POST path=/44302026/11/11 23:30:08 INFO Received uploads request method=POST path=/44312026/11/11 23:30:08 INFO Completed push id=30 objects=4 duration=2.915368ms44322026/11/11 23:30:08 INFO Received complete upload request method=POST path=/44332026/11/11 23:30:08 INFO Received push request method=POST path=/44342026/11/11 23:30:08 INFO Received complete upload request method=POST path=/44352026/11/11 23:30:08 INFO Completed upload id=12 objects=6 duration=1.360133ms44362026/11/11 23:30:08 INFO Completed upload id=18 objects=2 duration=2.597625ms44372026/11/11 23:30:08 INFO Received complete push request method=POST path=/44382026/11/11 23:30:08 INFO Completed push id=24 objects=5 duration=1.04523ms44392026/11/11 23:30:08 INFO Received complete push request method=POST path=/44402026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=044412026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044422026/11/11 23:30:08 INFO Completed push id=24 objects=5 duration=2.711886ms44432026/11/11 23:30:08 INFO Vacuumed table table=pending_closures44442026/11/11 23:30:08 INFO Received uploads request method=POST path=/44452026/11/11 23:30:08 INFO Vacuumed table table=pending_objects44462026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads44472026/11/11 23:30:08 INFO Vacuumed table table=closures44482026/11/11 23:30:08 INFO Received push request method=POST path=/44492026/11/11 23:30:08 INFO Received complete push request method=POST path=/44502026/11/11 23:30:08 INFO Vacuumed table table=objects44512026/11/11 23:30:08 INFO Received push request method=POST path=/44522026/11/11 23:30:08 INFO Received complete push request method=POST path=/44532026/11/11 23:30:08 INFO Completed push id=19 objects=14 duration=3.304132ms44542026/11/11 23:30:08 INFO Completed push id=17 objects=3 duration=2.553885ms44552026/11/11 23:30:08 INFO Received complete push request method=POST path=/44562026/11/11 23:30:08 INFO Received push request method=POST path=/44572026/11/11 23:30:08 INFO Completed push id=5 objects=28 duration=2.08274ms44582026/11/11 23:30:08 INFO Received push request method=POST path=/44592026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=044602026/11/11 23:30:08 INFO Received push request method=POST path=/44612026/11/11 23:30:08 INFO Received complete upload request method=POST path=/44622026/11/11 23:30:08 INFO Received uploads request method=POST path=/44632026/11/11 23:30:08 INFO Completed upload id=25 objects=8 duration=895.988µs44642026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044652026/11/11 23:30:08 INFO Vacuumed table table=pending_closures44662026/11/11 23:30:08 INFO Received push request method=POST path=/44672026/11/11 23:30:08 INFO Vacuumed table table=pending_objects44682026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads44692026/11/11 23:30:08 INFO Vacuumed table table=closures44702026/11/11 23:30:08 INFO Vacuumed table table=objects44712026/11/11 23:30:08 INFO Received push request method=POST path=/44722026/11/11 23:30:08 INFO Received complete push request method=POST path=/44732026/11/11 23:30:08 INFO Received complete push request method=POST path=/44742026/11/11 23:30:08 INFO Received complete push request method=POST path=/44752026/11/11 23:30:08 INFO Completed push id=23 objects=5 duration=1.263572ms44762026/11/11 23:30:08 INFO Completed push id=10 objects=2 duration=1.666336ms44772026/11/11 23:30:08 INFO Completed push id=28 objects=11 duration=1.838998ms44782026/11/11 23:30:08 INFO Received uploads request method=POST path=/44792026/11/11 23:30:08 INFO Received complete upload request method=POST path=/44802026/11/11 23:30:08 INFO Received push request method=POST path=/44812026/11/11 23:30:08 INFO Completed upload id=13 objects=7 duration=1.006369ms44822026/11/11 23:30:08 INFO Received complete push request method=POST path=/44832026/11/11 23:30:08 INFO Completed push id=26 objects=4 duration=815.868µs44842026/11/11 23:30:08 INFO Received push request method=POST path=/44852026/11/11 23:30:08 INFO Received complete upload request method=POST path=/44862026/11/11 23:30:08 INFO Completed upload id=24 objects=3 duration=883.609µs44872026/11/11 23:30:08 INFO Received complete upload request method=POST path=/44882026/11/11 23:30:08 INFO Completed upload id=20 objects=6 duration=1.08869ms44892026/11/11 23:30:08 INFO Received complete upload request method=POST path=/44902026/11/11 23:30:08 INFO Completed upload id=20 objects=4 duration=839.028µs44912026/11/11 23:30:08 INFO Received complete upload request method=POST path=/44922026/11/11 23:30:08 INFO Received complete push request method=POST path=/44932026/11/11 23:30:08 INFO Received push request method=POST path=/44942026/11/11 23:30:08 INFO Completed push id=30 objects=2 duration=884.269µs44952026/11/11 23:30:08 INFO Completed upload id=17 objects=5 duration=1.269033ms44962026/11/11 23:30:08 INFO Received complete push request method=POST path=/44972026/11/11 23:30:08 INFO Received uploads request method=POST path=/44982026/11/11 23:30:08 INFO Completed push id=32 objects=7 duration=1.163691ms44992026/11/11 23:30:08 INFO Received complete upload request method=POST path=/45002026/11/11 23:30:08 INFO Completed upload id=22 objects=6 duration=1.148031ms45012026/11/11 23:30:08 INFO Received complete push request method=POST path=/45022026/11/11 23:30:08 INFO Completed push id=25 objects=4 duration=937.409µs45032026/11/11 23:30:08 INFO Received complete upload request method=POST path=/45042026/11/11 23:30:08 INFO Completed upload id=23 objects=7 duration=908.529µs45052026/11/11 23:30:08 INFO Received complete push request method=POST path=/45062026/11/11 23:30:08 INFO Received push request method=POST path=/45072026/11/11 23:30:08 INFO Completed push id=21 objects=26 duration=1.475335ms45082026/11/11 23:30:08 INFO Received complete push request method=POST path=/45092026/11/11 23:30:08 INFO Completed push id=29 objects=24 duration=1.240452ms45102026/11/11 23:30:08 INFO Received complete push request method=POST path=/45112026/11/11 23:30:08 INFO Received complete push request method=POST path=/45122026/11/11 23:30:08 INFO Completed push id=26 objects=4 duration=926.909µs45132026/11/11 23:30:08 INFO Received uploads request method=POST path=/45142026/11/11 23:30:08 INFO Received complete push request method=POST path=/45152026/11/11 23:30:08 INFO Completed push id=19 objects=3 duration=1.157111ms45162026/11/11 23:30:08 INFO Completed push id=16 objects=4 duration=3.043629ms45172026/11/11 23:30:08 INFO Received uploads request method=POST path=/45182026/11/11 23:30:08 INFO Received complete push request method=POST path=/45192026/11/11 23:30:08 INFO Completed push id=32 objects=7 duration=1.05537ms45202026/11/11 23:30:08 INFO Received uploads request method=POST path=/45212026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=045222026/11/11 23:30:08 WARN Force mode enabled - objects will be deleted immediately without grace period45232026/11/11 23:30:08 INFO Received complete upload request method=POST path=/45242026/11/11 23:30:08 INFO Completed upload id=28 objects=13 duration=918.809µs45252026/11/11 23:30:08 INFO Received complete upload request method=POST path=/45262026/11/11 23:30:08 INFO Received uploads request method=POST path=/45272026/11/11 23:30:08 INFO Completed upload id=28 objects=17 duration=1.002369ms45282026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=045292026/11/11 23:30:08 INFO Vacuumed table table=pending_closures45302026/11/11 23:30:08 INFO Vacuumed table table=pending_objects45312026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads45322026/11/11 23:30:08 INFO Vacuumed table table=closures45332026/11/11 23:30:08 INFO Received push request method=POST path=/45342026/11/11 23:30:08 INFO Vacuumed table table=objects45352026/11/11 23:30:08 INFO Received complete upload request method=POST path=/45362026/11/11 23:30:08 INFO Received complete push request method=POST path=/45372026/11/11 23:30:08 INFO Received uploads request method=POST path=/45382026/11/11 23:30:08 INFO Completed upload id=33 objects=6 duration=932.969µs45392026/11/11 23:30:08 INFO Completed push id=20 objects=4 duration=1.173831ms45402026/11/11 23:30:08 INFO Received complete push request method=POST path=/45412026/11/11 23:30:08 INFO Received complete upload request method=POST path=/45422026/11/11 23:30:08 INFO Completed push id=29 objects=3 duration=1.135911ms45432026/11/11 23:30:08 INFO Received complete push request method=POST path=/45442026/11/11 23:30:08 INFO Completed upload id=24 objects=18 duration=1.145731ms45452026/11/11 23:30:08 INFO Received complete upload request method=POST path=/45462026/11/11 23:30:08 INFO Received complete push request method=POST path=/45472026/11/11 23:30:08 INFO Completed upload id=20 objects=5 duration=947.909µs45482026/11/11 23:30:08 INFO Completed push id=26 objects=23 duration=1.304732ms45492026/11/11 23:30:08 INFO Received complete upload request method=POST path=/45502026/11/11 23:30:08 INFO Completed push id=29 objects=2 duration=939.969µs45512026/11/11 23:30:08 INFO Completed upload id=21 objects=2 duration=792.608µs45522026/11/11 23:30:08 INFO Received push request method=POST path=/45532026/11/11 23:30:08 INFO Received complete upload request method=POST path=/45542026/11/11 23:30:08 INFO Received complete push request method=POST path=/45552026/11/11 23:30:08 INFO Received complete upload request method=POST path=/45562026/11/11 23:30:08 INFO Completed upload id=34 objects=3 duration=1.419774ms45572026/11/11 23:30:08 INFO Completed upload id=31 objects=2 duration=848.868µs45582026/11/11 23:30:08 INFO Received uploads request method=POST path=/45592026/11/11 23:30:08 INFO Completed push id=25 objects=2 duration=1.965919ms45602026/11/11 23:30:08 INFO Received complete push request method=POST path=/45612026/11/11 23:30:08 INFO Received uploads request method=POST path=/45622026/11/11 23:30:08 INFO Completed push id=31 objects=6 duration=958.33µs45632026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=045642026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045652026/11/11 23:30:08 INFO Vacuumed table table=pending_closures45662026/11/11 23:30:08 INFO Vacuumed table table=pending_objects45672026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads45682026/11/11 23:30:08 INFO Vacuumed table table=closures45692026/11/11 23:30:08 INFO Vacuumed table table=objects45702026/11/11 23:30:08 INFO Received push request method=POST path=/45712026/11/11 23:30:08 INFO Received complete upload request method=POST path=/45722026/11/11 23:30:08 INFO Completed upload id=22 objects=8 duration=1.187871ms45732026/11/11 23:30:08 INFO Received complete push request method=POST path=/45742026/11/11 23:30:08 INFO Received complete push request method=POST path=/45752026/11/11 23:30:08 INFO Completed push id=27 objects=4 duration=825.548µs45762026/11/11 23:30:08 INFO Completed push id=33 objects=6 duration=1.084851ms45772026/11/11 23:30:08 INFO Received push request method=POST path=/45782026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=045792026/11/11 23:30:08 INFO Received complete push request method=POST path=/45802026/11/11 23:30:08 INFO Received complete push request method=POST path=/45812026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=045822026/11/11 23:30:08 WARN Force mode enabled - objects will be deleted immediately without grace period45832026/11/11 23:30:08 INFO Completed push id=27 objects=15 duration=1.200191ms45842026/11/11 23:30:08 INFO Completed push id=32 objects=9 duration=1.164251ms45852026/11/11 23:30:08 WARN Force mode enabled - objects will be deleted immediately without grace period45862026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045872026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045882026/11/11 23:30:08 INFO Vacuumed table table=pending_closures45892026/11/11 23:30:08 INFO Vacuumed table table=pending_closures45902026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=045912026/11/11 23:30:08 INFO Received uploads request method=POST path=/45922026/11/11 23:30:08 INFO Vacuumed table table=pending_objects45932026/11/11 23:30:08 WARN Force mode enabled - objects will be deleted immediately without grace period45942026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads45952026/11/11 23:30:08 INFO Vacuumed table table=pending_objects45962026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads45972026/11/11 23:30:08 INFO Vacuumed table table=closures45982026/11/11 23:30:08 INFO Received complete push request method=POST path=/45992026/11/11 23:30:08 INFO Vacuumed table table=closures46002026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046012026/11/11 23:30:08 INFO Vacuumed table table=objects46022026/11/11 23:30:08 INFO Vacuumed table table=objects46032026/11/11 23:30:08 INFO Completed push id=23 objects=5 duration=1.102011ms46042026/11/11 23:30:08 INFO Vacuumed table table=pending_closures46052026/11/11 23:30:08 INFO Vacuumed table table=pending_objects46062026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads46072026/11/11 23:30:08 INFO Vacuumed table table=closures46082026/11/11 23:30:08 INFO Vacuumed table table=objects46092026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=046102026/11/11 23:30:08 INFO Received push request method=POST path=/46112026/11/11 23:30:08 WARN Force mode enabled - objects will be deleted immediately without grace period46122026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046132026/11/11 23:30:08 INFO Vacuumed table table=pending_closures46142026/11/11 23:30:08 INFO Vacuumed table table=pending_objects46152026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads46162026/11/11 23:30:08 INFO Received uploads request method=POST path=/46172026/11/11 23:30:08 INFO Vacuumed table table=closures46182026/11/11 23:30:08 INFO Received complete upload request method=POST path=/46192026/11/11 23:30:08 INFO Vacuumed table table=objects46202026/11/11 23:30:08 INFO Completed upload id=33 objects=9 duration=890.568µs46212026/11/11 23:30:08 INFO Received complete upload request method=POST path=/46222026/11/11 23:30:08 INFO Received complete push request method=POST path=/46232026/11/11 23:30:08 INFO Received complete upload request method=POST path=/46242026/11/11 23:30:08 INFO Completed upload id=24 objects=8 duration=887.988µs46252026/11/11 23:30:08 INFO Completed push id=28 objects=26 duration=1.151391ms46262026/11/11 23:30:08 INFO Completed upload id=19 objects=2 duration=1.842478ms46272026/11/11 23:30:08 INFO Received push request method=POST path=/46282026/11/11 23:30:08 INFO Received uploads request method=POST path=/46292026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=5 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046302026/11/11 23:30:08 INFO Received uploads request method=POST path=/46312026/11/11 23:30:08 INFO Received uploads request method=POST path=/46322026/11/11 23:30:08 INFO Received complete push request method=POST path=/46332026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=046342026/11/11 23:30:08 INFO Completed push id=30 objects=3 duration=984.129µs46352026/11/11 23:30:08 INFO Vacuumed table table=pending_closures46362026/11/11 23:30:08 INFO Received complete upload request method=POST path=/46372026/11/11 23:30:08 INFO Received complete push request method=POST path=/46382026/11/11 23:30:08 INFO Completed upload id=28 objects=5 duration=817.668µs46392026/11/11 23:30:08 INFO Vacuumed table table=pending_objects46402026/11/11 23:30:08 INFO Completed push id=28 objects=19 duration=959.729µs46412026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads46422026/11/11 23:30:08 WARN Force mode enabled - objects will be deleted immediately without grace period46432026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046442026/11/11 23:30:08 INFO Vacuumed table table=closures46452026/11/11 23:30:08 INFO Vacuumed table table=objects46462026/11/11 23:30:08 INFO Vacuumed table table=pending_closures46472026/11/11 23:30:08 INFO Vacuumed table table=pending_objects46482026/11/11 23:30:08 INFO Received uploads request method=POST path=/46492026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads46502026/11/11 23:30:08 INFO Vacuumed table table=closures46512026/11/11 23:30:08 INFO Vacuumed table table=objects46522026/11/11 23:30:08 INFO Received push request method=POST path=/46532026/11/11 23:30:08 INFO Received complete push request method=POST path=/46542026/11/11 23:30:08 INFO Completed push id=33 objects=7 duration=1.594015ms46552026/11/11 23:30:08 INFO Received complete push request method=POST path=/46562026/11/11 23:30:08 INFO Completed push id=32 objects=19 duration=988.689µs46572026/11/11 23:30:08 INFO Received complete upload request method=POST path=/46582026/11/11 23:30:08 INFO Received uploads request method=POST path=/46592026/11/11 23:30:08 INFO Completed upload id=30 objects=5 duration=973.829µs46602026/11/11 23:30:08 INFO Received complete upload request method=POST path=/46612026/11/11 23:30:08 INFO Received complete push request method=POST path=/46622026/11/11 23:30:08 INFO Completed upload id=26 objects=3 duration=884.768µs46632026/11/11 23:30:08 INFO Completed push id=29 objects=10 duration=877.128µs46642026/11/11 23:30:08 INFO Received complete push request method=POST path=/46652026/11/11 23:30:08 INFO Completed push id=24 objects=24 duration=2.780747ms46662026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=046672026/11/11 23:30:08 WARN Force mode enabled - objects will be deleted immediately without grace period46682026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046692026/11/11 23:30:08 INFO Vacuumed table table=pending_closures46702026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=046712026/11/11 23:30:08 INFO Vacuumed table table=pending_objects46722026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=046732026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads46742026/11/11 23:30:08 WARN Force mode enabled - objects will be deleted immediately without grace period46752026/11/11 23:30:08 INFO Vacuumed table table=closures46762026/11/11 23:30:08 INFO Vacuumed table table=objects46772026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046782026/11/11 23:30:08 INFO Received push request method=POST path=/46792026/11/11 23:30:08 WARN Force mode enabled - objects will be deleted immediately without grace period46802026/11/11 23:30:08 INFO Vacuumed table table=pending_closures46812026/11/11 23:30:08 INFO Vacuumed table table=pending_objects46822026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads46832026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046842026/11/11 23:30:08 INFO Vacuumed table table=closures46852026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=046862026/11/11 23:30:08 INFO Vacuumed table table=pending_closures46872026/11/11 23:30:08 INFO Vacuumed table table=pending_objects46882026/11/11 23:30:08 WARN Force mode enabled - objects will be deleted immediately without grace period46892026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads46902026/11/11 23:30:08 INFO Vacuumed table table=closures46912026/11/11 23:30:08 INFO Vacuumed table table=objects46922026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046932026/11/11 23:30:08 INFO Vacuumed table table=pending_closures46942026/11/11 23:30:08 INFO Vacuumed table table=pending_objects46952026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads46962026/11/11 23:30:08 INFO Vacuumed table table=closures46972026/11/11 23:30:08 INFO Vacuumed table table=objects46982026/11/11 23:30:08 INFO Vacuumed table table=objects46992026/11/11 23:30:08 INFO Received push request method=POST path=/47002026/11/11 23:30:08 INFO Received complete push request method=POST path=/47012026/11/11 23:30:08 INFO Received complete upload request method=POST path=/47022026/11/11 23:30:08 INFO Completed upload id=30 objects=6 duration=963.37µs47032026/11/11 23:30:08 INFO Completed push id=31 objects=3 duration=1.232992ms47042026/11/11 23:30:08 INFO Received uploads request method=POST path=/47052026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=047062026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047072026/11/11 23:30:08 INFO Vacuumed table table=pending_closures47082026/11/11 23:30:08 INFO Received complete upload request method=POST path=/47092026/11/11 23:30:08 INFO Vacuumed table table=pending_objects47102026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads47112026/11/11 23:30:08 INFO Vacuumed table table=closures47122026/11/11 23:30:08 INFO Received push request method=POST path=/47132026/11/11 23:30:08 INFO Completed upload id=31 objects=5 duration=954.809µs47142026/11/11 23:30:08 INFO Vacuumed table table=objects47152026/11/11 23:30:08 INFO Received uploads request method=POST path=/47162026/11/11 23:30:08 INFO Received push request method=POST path=/47172026/11/11 23:30:08 INFO Received complete upload request method=POST path=/47182026/11/11 23:30:08 INFO Completed upload id=34 objects=3 duration=673.927µs47192026/11/11 23:30:08 INFO Received push request method=POST path=/47202026/11/11 23:30:08 INFO Received complete push request method=POST path=/47212026/11/11 23:30:08 INFO Received complete push request method=POST path=/47222026/11/11 23:30:08 INFO Completed push id=16 objects=7 duration=1.637636ms47232026/11/11 23:30:08 INFO Completed push id=32 objects=6 duration=908.689µs47242026/11/11 23:30:08 INFO Received complete push request method=POST path=/47252026/11/11 23:30:08 INFO Completed push id=9 objects=9 duration=2.395223ms47262026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=047272026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=047282026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047292026/11/11 23:30:08 INFO Received complete upload request method=POST path=/47302026/11/11 23:30:08 INFO Vacuumed table table=pending_closures47312026/11/11 23:30:08 INFO Completed upload id=33 objects=7 duration=1.06415ms47322026/11/11 23:30:08 INFO Received push request method=POST path=/47332026/11/11 23:30:08 INFO Vacuumed table table=pending_objects47342026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads47352026/11/11 23:30:08 INFO Vacuumed table table=closures47362026/11/11 23:30:08 INFO Vacuumed table table=objects47372026/11/11 23:30:08 INFO Received complete upload request method=POST path=/47382026/11/11 23:30:08 WARN Force mode enabled - objects will be deleted immediately without grace period47392026/11/11 23:30:08 INFO Completed upload id=21 objects=8 duration=1.08051ms47402026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047412026/11/11 23:30:08 INFO Received push request method=POST path=/47422026/11/11 23:30:08 INFO Received complete push request method=POST path=/47432026/11/11 23:30:08 INFO Vacuumed table table=pending_closures47442026/11/11 23:30:08 INFO Completed push id=23 objects=6 duration=1.245952ms47452026/11/11 23:30:08 INFO Vacuumed table table=pending_objects47462026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads47472026/11/11 23:30:08 INFO Vacuumed table table=closures47482026/11/11 23:30:08 INFO Vacuumed table table=objects47492026/11/11 23:30:08 INFO Received complete push request method=POST path=/47502026/11/11 23:30:08 INFO Completed push id=22 objects=6 duration=1.00547ms47512026/11/11 23:30:08 INFO Received uploads request method=POST path=/47522026/11/11 23:30:08 INFO Received complete upload request method=POST path=/47532026/11/11 23:30:08 INFO Received uploads request method=POST path=/47542026/11/11 23:30:08 INFO Completed upload id=34 objects=3 duration=849.508µs47552026/11/11 23:30:08 INFO Received complete push request method=POST path=/47562026/11/11 23:30:08 INFO Completed push id=24 objects=6 duration=864.449µs47572026/11/11 23:30:08 INFO Received complete push request method=POST path=/47582026/11/11 23:30:08 INFO Completed push id=24 objects=8 duration=1.232851ms47592026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=047602026/11/11 23:30:08 INFO Received complete push request method=POST path=/47612026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=047622026/11/11 23:30:08 WARN Force mode enabled - objects will be deleted immediately without grace period47632026/11/11 23:30:08 INFO Completed push id=35 objects=11 duration=1.121131ms47642026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047652026/11/11 23:30:08 WARN Force mode enabled - objects will be deleted immediately without grace period47662026/11/11 23:30:08 INFO Vacuumed table table=pending_closures47672026/11/11 23:30:08 INFO Vacuumed table table=pending_objects47682026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads47692026/11/11 23:30:08 INFO Vacuumed table table=closures47702026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047712026/11/11 23:30:08 INFO Vacuumed table table=objects47722026/11/11 23:30:08 INFO Vacuumed table table=pending_closures47732026/11/11 23:30:08 INFO Vacuumed table table=pending_objects47742026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads47752026/11/11 23:30:08 INFO Vacuumed table table=closures47762026/11/11 23:30:08 INFO Vacuumed table table=objects47772026/11/11 23:30:08 INFO Received complete push request method=POST path=/47782026/11/11 23:30:08 INFO Completed push id=25 objects=7 duration=1.234791ms47792026/11/11 23:30:08 INFO Received uploads request method=POST path=/47802026/11/11 23:30:08 INFO Aborted multipart uploads count=0 kept=047812026/11/11 23:30:08 INFO Received complete upload request method=POST path=/47822026/11/11 23:30:08 INFO Completed upload id=25 objects=9 duration=911.429µs47832026/11/11 23:30:08 WARN Force mode enabled - objects will be deleted immediately without grace period47842026/11/11 23:30:08 INFO Garbage collection completed failed-uploads-deleted=6 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047852026/11/11 23:30:08 INFO Vacuumed table table=pending_closures47862026/11/11 23:30:08 INFO Vacuumed table table=pending_objects47872026/11/11 23:30:08 INFO Vacuumed table table=multipart_uploads47882026/11/11 23:30:08 INFO Vacuumed table table=closures47892026/11/11 23:30:08 INFO Vacuumed table table=objects47902026/11/11 23:30:08 INFO Received complete upload request method=POST path=/47912026/11/11 23:30:08 INFO Completed upload id=27 objects=3 duration=868.788µs47922026/11/11 23:30:08 INFO Received push request method=POST path=/47932026/11/11 23:30:08 INFO Received push request method=POST path=/47942026/11/11 23:30:08 INFO Received complete upload request method=POST path=/47952026/11/11 23:30:08 INFO Completed upload id=28 objects=6 duration=783.367µs47962026/11/11 23:30:08 INFO Received complete upload request method=POST path=/47972026/11/11 23:30:08 INFO Received uploads request method=POST path=/47982026/11/11 23:30:08 INFO Received push request method=POST path=/47992026/11/11 23:30:08 INFO Completed upload id=26 objects=24 duration=2.239281ms48002026/11/11 23:30:08 INFO Received complete upload request method=POST path=/48012026/11/11 23:30:08 INFO Received complete push request method=POST path=/48022026/11/11 23:30:08 INFO Completed push id=30 objects=4 duration=860.829µs48032026/11/11 23:30:08 INFO Completed upload id=23 objects=7 duration=2.358782ms48042026/11/11 23:30:08 INFO Received complete upload request method=POST path=/48052026/11/11 23:30:09 INFO Received complete push request method=POST path=/48062026/11/11 23:30:09 INFO Completed upload id=23 objects=18 duration=2.664826ms48072026/11/11 23:30:09 INFO Completed push id=31 objects=14 duration=4.920887ms48082026/11/11 23:30:09 INFO Received complete push request method=POST path=/48092026/11/11 23:30:09 INFO Completed push id=31 objects=7 duration=966.229µs48102026/11/11 23:30:09 INFO Received complete upload request method=POST path=/48112026/11/11 23:30:09 INFO Completed upload id=26 objects=5 duration=815.788µs48122026/11/11 23:30:09 INFO Received complete upload request method=POST path=/48132026/11/11 23:30:09 INFO Received complete upload request method=POST path=/48142026/11/11 23:30:09 INFO Completed upload id=27 objects=26 duration=1.101051ms48152026/11/11 23:30:09 INFO Received uploads request method=POST path=/48162026/11/11 23:30:09 INFO Completed upload id=32 objects=5 duration=746.407µs48172026/11/11 23:30:09 INFO Received complete push request method=POST path=/48182026/11/11 23:30:09 INFO Completed push id=27 objects=3 duration=1.518754ms48192026/11/11 23:30:09 INFO Received complete push request method=POST path=/48202026/11/11 23:30:09 INFO Completed push id=28 objects=6 duration=938.149µs48212026/11/11 23:30:09 INFO Received uploads request method=POST path=/48222026/11/11 23:30:09 INFO Received complete upload request method=POST path=/48232026/11/11 23:30:09 INFO Completed upload id=28 objects=5 duration=831.948µs48242026/11/11 23:30:09 INFO Received push request method=POST path=/48252026/11/11 23:30:09 INFO Received complete push request method=POST path=/48262026/11/11 23:30:09 INFO Received complete push request method=POST path=/48272026/11/11 23:30:09 INFO Completed push id=29 objects=6 duration=1.05643ms48282026/11/11 23:30:09 INFO Completed push id=29 objects=17 duration=1.261112ms48292026/11/11 23:30:09 INFO Received complete upload request method=POST path=/48302026/11/11 23:30:09 INFO Completed upload id=22 objects=22 duration=911.808µs48312026/11/11 23:30:09 INFO Received complete upload request method=POST path=/48322026/11/11 23:30:09 INFO Completed upload id=34 objects=24 duration=1.162571ms48332026/11/11 23:30:09 INFO Aborted multipart uploads count=0 kept=048342026/11/11 23:30:09 WARN Force mode enabled - objects will be deleted immediately without grace period48352026/11/11 23:30:09 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048362026/11/11 23:30:09 INFO Vacuumed table table=pending_closures48372026/11/11 23:30:09 INFO Received push request method=POST path=/48382026/11/11 23:30:09 INFO Vacuumed table table=pending_objects48392026/11/11 23:30:09 INFO Vacuumed table table=multipart_uploads48402026/11/11 23:30:09 INFO Vacuumed table table=closures48412026/11/11 23:30:09 INFO Vacuumed table table=objects48422026/11/11 23:30:09 INFO Aborted multipart uploads count=0 kept=048432026/11/11 23:30:09 INFO Received complete upload request method=POST path=/48442026/11/11 23:30:09 WARN Force mode enabled - objects will be deleted immediately without grace period48452026/11/11 23:30:09 INFO Received complete push request method=POST path=/48462026/11/11 23:30:09 INFO Completed upload id=30 objects=8 duration=1.04825ms48472026/11/11 23:30:09 INFO Completed push id=7 objects=7 duration=1.472854ms48482026/11/11 23:30:09 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048492026/11/11 23:30:09 INFO Vacuumed table table=pending_closures48502026/11/11 23:30:09 INFO Vacuumed table table=pending_objects48512026/11/11 23:30:09 INFO Vacuumed table table=multipart_uploads48522026/11/11 23:30:09 INFO Vacuumed table table=closures48532026/11/11 23:30:09 INFO Vacuumed table table=objects48542026/11/11 23:30:09 INFO Received complete push request method=POST path=/48552026/11/11 23:30:09 INFO Completed push id=26 objects=5 duration=1.717337ms48562026/11/11 23:30:09 INFO Received complete upload request method=POST path=/48572026/11/11 23:30:09 INFO Completed upload id=31 objects=6 duration=815.328µs48582026/11/11 23:30:09 INFO Received complete push request method=POST path=/48592026/11/11 23:30:09 INFO Completed push id=12 objects=6 duration=1.03735ms48602026/11/11 23:30:09 INFO Received complete upload request method=POST path=/48612026/11/11 23:30:09 INFO Completed upload id=24 objects=14 duration=989.709µs48622026/11/11 23:30:09 INFO Received complete upload request method=POST path=/48632026/11/11 23:30:09 INFO Completed upload id=32 objects=4 duration=717.487µs48642026/11/11 23:30:09 INFO Received complete push request method=POST path=/48652026/11/11 23:30:09 INFO Completed push id=19 objects=3 duration=892.728µs48662026/11/11 23:30:09 INFO Received complete upload request method=POST path=/48672026/11/11 23:30:09 INFO Completed upload id=33 objects=10 duration=900.848µs48682026/11/11 23:30:09 INFO Received complete push request method=POST path=/48692026/11/11 23:30:09 INFO Completed push id=13 objects=22 duration=1.212812ms48702026/11/11 23:30:09 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048712026/11/11 23:30:09 INFO Vacuumed table table=pending_closures48722026/11/11 23:30:09 INFO Vacuumed table table=pending_objects48732026/11/11 23:30:09 INFO Vacuumed table table=multipart_uploads48742026/11/11 23:30:09 INFO Vacuumed table table=closures48752026/11/11 23:30:09 INFO Vacuumed table table=objects48762026/11/11 23:30:09 INFO Received complete push request method=POST path=/48772026/11/11 23:30:09 INFO Received uploads request method=POST path=/48782026/11/11 23:30:09 INFO Completed push id=21 objects=6 duration=1.001069ms48792026/11/11 23:30:09 INFO Received complete upload request method=POST path=/48802026/11/11 23:30:09 INFO Completed upload id=34 objects=23 duration=1.07757ms48812026/11/11 23:30:09 INFO Received complete upload request method=POST path=/48822026/11/11 23:30:09 INFO Received uploads request method=POST path=/48832026/11/11 23:30:09 INFO Completed upload id=22 objects=7 duration=1.147391ms48842026/11/11 23:30:09 INFO Received complete push request method=POST path=/48852026/11/11 23:30:09 INFO Completed push id=35 objects=6 duration=1.062111ms48862026/11/11 23:30:09 INFO Received uploads request method=POST path=/48872026/11/11 23:30:09 INFO Received complete push request method=POST path=/48882026/11/11 23:30:09 INFO Completed push id=24 objects=4 duration=870.189µs48892026/11/11 23:30:09 INFO Aborted multipart uploads count=0 kept=048902026/11/11 23:30:09 WARN Force mode enabled - objects will be deleted immediately without grace period48912026/11/11 23:30:09 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048922026/11/11 23:30:09 INFO Vacuumed table table=pending_closures48932026/11/11 23:30:09 INFO Vacuumed table table=pending_objects48942026/11/11 23:30:09 INFO Vacuumed table table=multipart_uploads48952026/11/11 23:30:09 INFO Vacuumed table table=closures48962026/11/11 23:30:09 INFO Vacuumed table table=objects48972026/11/11 23:30:09 INFO Aborted multipart uploads count=0 kept=048982026/11/11 23:30:09 WARN Force mode enabled - objects will be deleted immediately without grace period48992026/11/11 23:30:09 INFO Received complete push request method=POST path=/49002026/11/11 23:30:09 INFO Received complete upload request method=POST path=/49012026/11/11 23:30:09 INFO Completed push id=25 objects=9 duration=1.096311ms49022026/11/11 23:30:09 INFO Completed upload id=25 objects=4 duration=1.13727ms49032026/11/11 23:30:09 INFO Aborted multipart uploads count=0 kept=049042026/11/11 23:30:09 WARN Force mode enabled - objects will be deleted immediately without grace period49052026/11/11 23:30:09 INFO Received complete upload request method=POST path=/49062026/11/11 23:30:09 INFO Completed upload id=26 objects=3 duration=736.627µs49072026/11/11 23:30:09 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049082026/11/11 23:30:09 INFO Received push request method=POST path=/49092026/11/11 23:30:09 INFO Vacuumed table table=pending_closures49102026/11/11 23:30:09 INFO Vacuumed table table=pending_objects49112026/11/11 23:30:09 INFO Vacuumed table table=multipart_uploads49122026/11/11 23:30:09 INFO Vacuumed table table=closures49132026/11/11 23:30:09 INFO Vacuumed table table=objects49142026/11/11 23:30:09 INFO Received complete upload request method=POST path=/49152026/11/11 23:30:09 INFO Completed upload id=27 objects=7 duration=792.787µs49162026/11/11 23:30:09 INFO Received complete upload request method=POST path=/49172026/11/11 23:30:09 INFO Completed upload id=17 objects=25 duration=2.895667ms49182026/11/11 23:30:09 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=049192026/11/11 23:30:09 INFO Received complete push request method=POST path=/49202026/11/11 23:30:09 INFO Received complete upload request method=POST path=/49212026/11/11 23:30:09 INFO Completed push id=20 objects=8 duration=1.196452ms49222026/11/11 23:30:09 INFO Vacuumed table table=pending_closures49232026/11/11 23:30:09 INFO Completed upload id=28 objects=3 duration=1.182292ms49242026/11/11 23:30:09 INFO Vacuumed table table=pending_objects49252026/11/11 23:30:09 INFO Vacuumed table table=multipart_uploads49262026/11/11 23:30:09 INFO Vacuumed table table=closures49272026/11/11 23:30:09 INFO Vacuumed table table=objects49282026/11/11 23:30:09 INFO Received push request method=POST path=/49292026/11/11 23:30:09 INFO Received uploads request method=POST path=/49302026/11/11 23:30:09 INFO Received complete push request method=POST path=/49312026/11/11 23:30:09 INFO Completed push id=29 objects=23 duration=1.347193ms49322026/11/11 23:30:09 INFO Received complete upload request method=POST path=/49332026/11/11 23:30:09 INFO Completed upload id=21 objects=2 duration=1.148331ms49342026/11/11 23:30:09 INFO Received push request method=POST path=/49352026/11/11 23:30:09 INFO Received complete push request method=POST path=/49362026/11/11 23:30:09 INFO Completed push id=30 objects=6 duration=874.008µs49372026/11/11 23:30:09 INFO Aborted multipart uploads count=0 kept=049382026/11/11 23:30:09 WARN Force mode enabled - objects will be deleted immediately without grace period49392026/11/11 23:30:09 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049402026/11/11 23:30:09 INFO Vacuumed table table=pending_closures49412026/11/11 23:30:09 INFO Vacuumed table table=pending_objects49422026/11/11 23:30:09 INFO Vacuumed table table=multipart_uploads49432026/11/11 23:30:09 INFO Vacuumed table table=closures49442026/11/11 23:30:09 INFO Vacuumed table table=objects49452026/11/11 23:30:09 INFO Received complete push request method=POST path=/49462026/11/11 23:30:09 INFO Received complete push request method=POST path=/49472026/11/11 23:30:09 INFO Completed push id=27 objects=11 duration=935.549µs49482026/11/11 23:30:09 INFO Completed push id=33 objects=3 duration=1.077971ms49492026/11/11 23:30:09 INFO Received complete upload request method=POST path=/49502026/11/11 23:30:09 INFO Completed upload id=29 objects=26 duration=723.007µs49512026/11/11 23:30:09 INFO Received complete push request method=POST path=/49522026/11/11 23:30:09 INFO Completed push id=34 objects=6 duration=756.847µs49532026/11/11 23:30:09 INFO Received complete push request method=POST path=/49542026/11/11 23:30:09 INFO Received complete push request method=POST path=/49552026/11/11 23:30:09 INFO Completed push id=30 objects=8 duration=1.457534ms49562026/11/11 23:30:09 INFO Completed push id=35 objects=8 duration=1.270632ms49572026/11/11 23:30:09 INFO Received complete push request method=POST path=/49582026/11/11 23:30:09 INFO Completed push id=7 objects=24 duration=4.910107ms49592026/11/11 23:30:09 INFO Received complete upload request method=POST path=/49602026/11/11 23:30:09 INFO Completed upload id=31 objects=5 duration=816.808µs49612026/11/11 23:30:09 INFO Received complete upload request method=POST path=/49622026/11/11 23:30:09 INFO Completed upload id=36 objects=9 duration=679.427µs49632026/11/11 23:30:09 INFO Received uploads request method=POST path=/49642026/11/11 23:30:09 INFO Received complete push request method=POST path=/49652026/11/11 23:30:09 INFO Completed push id=32 objects=3 duration=939.389µs49662026/11/11 23:30:09 INFO Aborted multipart uploads count=0 kept=049672026/11/11 23:30:09 WARN Force mode enabled - objects will be deleted immediately without grace period49682026/11/11 23:30:09 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049692026/11/11 23:30:09 INFO Vacuumed table table=pending_closures49702026/11/11 23:30:09 INFO Vacuumed table table=pending_objects49712026/11/11 23:30:09 INFO Vacuumed table table=multipart_uploads49722026/11/11 23:30:09 INFO Vacuumed table table=closures49732026/11/11 23:30:09 INFO Vacuumed table table=objects49742026/11/11 23:30:09 INFO Received uploads request method=POST path=/49752026/11/11 23:30:09 INFO Aborted multipart uploads count=0 kept=049762026/11/11 23:30:09 WARN Force mode enabled - objects will be deleted immediately without grace period49772026/11/11 23:30:09 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049782026/11/11 23:30:09 INFO Vacuumed table table=pending_closures49792026/11/11 23:30:09 INFO Vacuumed table table=pending_objects49802026/11/11 23:30:09 INFO Vacuumed table table=multipart_uploads49812026/11/11 23:30:09 INFO Vacuumed table table=closures49822026/11/11 23:30:09 INFO Vacuumed table table=objects49832026/11/11 23:30:09 INFO Received uploads request method=POST path=/49842026/11/11 23:30:09 INFO Received push request method=POST path=/49852026/11/11 23:30:09 INFO Received uploads request method=POST path=/49862026/11/11 23:30:09 INFO Received complete push request method=POST path=/49872026/11/11 23:30:09 INFO Completed push id=13 objects=2 duration=3.525114ms49882026/11/11 23:30:09 INFO Received complete upload request method=POST path=/49892026/11/11 23:30:09 INFO Completed upload id=27 objects=13 duration=1.00399ms49902026/11/11 23:30:09 INFO Aborted multipart uploads count=0 kept=049912026/11/11 23:30:09 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049922026/11/11 23:30:09 INFO Vacuumed table table=pending_closures49932026/11/11 23:30:09 INFO Vacuumed table table=pending_objects49942026/11/11 23:30:09 INFO Vacuumed table table=multipart_uploads49952026/11/11 23:30:09 INFO Vacuumed table table=closures49962026/11/11 23:30:09 INFO Vacuumed table table=objects49972026/11/11 23:30:09 INFO Received uploads request method=POST path=/49982026/11/11 23:30:09 INFO Received push request method=POST path=/49992026/11/11 23:30:09 INFO Received push request method=POST path=/50002026/11/11 23:30:09 INFO Received uploads request method=POST path=/50012026/11/11 23:30:09 INFO Received uploads request method=POST path=/50022026/11/11 23:30:09 INFO Received complete upload request method=POST path=/50032026/11/11 23:30:09 INFO Completed upload id=29 objects=4 duration=1.778137ms50042026/11/11 23:30:09 INFO Received uploads request method=POST path=/50052026/11/11 23:30:09 INFO Received uploads request method=POST path=/50062026/11/11 23:30:09 INFO Received complete push request method=POST path=/50072026/11/11 23:30:09 INFO Completed push id=11 objects=6 duration=1.931778ms50082026/11/11 23:30:09 INFO Received complete push request method=POST path=/50092026/11/11 23:30:09 INFO Completed push id=20 objects=13 duration=3.424792ms50102026/11/11 23:30:09 INFO Received complete push request method=POST path=/50112026/11/11 23:30:09 INFO Completed push id=22 objects=4 duration=1.393913ms50122026/11/11 23:30:09 INFO Received complete upload request method=POST path=/50132026/11/11 23:30:09 INFO Completed upload id=23 objects=7 duration=4.24816ms50142026/11/11 23:30:09 INFO Received complete push request method=POST path=/50152026/11/11 23:30:09 INFO Completed push id=24 objects=5 duration=3.274131ms50162026/11/11 23:30:09 INFO Received complete upload request method=POST path=/50172026/11/11 23:30:09 INFO Completed upload id=25 objects=2 duration=783.027µs50182026/11/11 23:30:09 INFO Received complete upload request method=POST path=/50192026/11/11 23:30:09 INFO Completed upload id=26 objects=20 duration=1.501095ms50202026/11/11 23:30:10 INFO Received complete push request method=POST path=/50212026/11/11 23:30:10 INFO Completed push id=28 objects=7 duration=14.69724ms50222026/11/11 23:30:10 INFO Received complete upload request method=POST path=/50232026/11/11 23:30:10 INFO Completed upload id=30 objects=6 duration=5.006887ms50242026/11/11 23:30:10 INFO Received complete push request method=POST path=/50252026/11/11 23:30:10 INFO Completed push id=31 objects=5 duration=3.503173ms50262026/11/11 23:30:10 INFO Received complete push request method=POST path=/50272026/11/11 23:30:10 INFO Completed push id=32 objects=26 duration=6.150059ms50282026/11/11 23:30:10 INFO Received complete upload request method=POST path=/50292026/11/11 23:30:10 INFO Completed upload id=33 objects=2 duration=843.548µs50302026/11/11 23:30:10 INFO Received complete upload request method=POST path=/50312026/11/11 23:30:10 INFO Completed upload id=34 objects=5 duration=567.506µs50322026/11/11 23:30:10 INFO Received complete upload request method=POST path=/50332026/11/11 23:30:10 INFO Completed upload id=35 objects=11 duration=604.046µs50342026/11/11 23:30:10 INFO Received complete upload request method=POST path=/50352026/11/11 23:30:10 INFO Completed upload id=36 objects=3 duration=541.485µs50362026/11/11 23:30:10 INFO Aborted multipart uploads count=0 kept=050372026/11/11 23:30:10 WARN Force mode enabled - objects will be deleted immediately without grace period50382026/11/11 23:30:10 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=050392026/11/11 23:30:10 INFO Vacuumed table table=pending_closures50402026/11/11 23:30:10 INFO Vacuumed table table=pending_objects50412026/11/11 23:30:10 INFO Vacuumed table table=multipart_uploads50422026/11/11 23:30:10 INFO Vacuumed table table=closures50432026/11/11 23:30:10 INFO Vacuumed table table=objects5044--- PASS: FuzzPushGC (0.01s)5045 --- PASS: FuzzPushGC/seed#6 (4.30s)5046 --- PASS: FuzzPushGC/seed#5 (4.49s)5047 --- PASS: FuzzPushGC/seed#15 (4.50s)5048 --- PASS: FuzzPushGC/seed#2 (5.10s)5049 --- PASS: FuzzPushGC/seed#10 (5.15s)5050 --- PASS: FuzzPushGC/seed#14 (5.16s)5051 --- PASS: FuzzPushGC/seed#0 (5.19s)5052 --- PASS: FuzzPushGC/seed#7 (5.23s)5053 --- PASS: FuzzPushGC/seed#13 (5.32s)5054 --- PASS: FuzzPushGC/seed#8 (5.37s)5055 --- PASS: FuzzPushGC/seed#1 (5.48s)5056 --- PASS: FuzzPushGC/seed#9 (5.50s)5057 --- PASS: FuzzPushGC/seed#12 (5.67s)5058 --- PASS: FuzzPushGC/seed#4 (5.80s)5059 --- PASS: FuzzPushGC/seed#3 (5.81s)5060 --- PASS: FuzzPushGC/seed#11 (6.77s)5061PASS5062{"timestamp":"2026-11-11T23:30:10.384583482Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:46720","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(199)"}50632026-11-11 23:30:10.757 UTC [131] LOG: received smart shutdown request50642026-11-11 23:30:10.764 UTC [131] LOG: background worker "logical replication launcher" (PID 149) exited with exit code 150652026-11-11 23:30:13.239 UTC [144] LOG: checkpoint complete: wrote 4082 buffers (24.9%), wrote 5 SLRU buffers; 0 WAL file(s) added, 0 removed, 33 recycled; write=7.217 s, sync=0.815 s, total=9.568 s; sync files=42363, longest=0.020 s, average=0.001 s; distance=536601 kB, estimate=536601 kB; lsn=0/26D90098, redo lsn=0/2236C26050662026-11-11 23:30:13.246 UTC [144] LOG: shutting down50672026-11-11 23:30:13.247 UTC [144] LOG: checkpoint starting: shutdown immediate50682026-11-11 23:30:13.721 UTC [144] LOG: checkpoint complete: wrote 7963 buffers (48.6%), wrote 2 SLRU buffers; 0 WAL file(s) added, 0 removed, 4 recycled; write=0.164 s, sync=0.301 s, total=0.475 s; sync files=2549, longest=0.017 s, average=0.001 s; distance=75919 kB, estimate=490533 kB; lsn=0/26D90110, redo lsn=0/26D9011050692026-11-11 23:30:13.908 UTC [131] LOG: database system is shut down5070Running OIDC tests...5071=== RUN TestAudienceForIssuer5072=== PAUSE TestAudienceForIssuer5073=== RUN TestGlobMatch5074=== PAUSE TestGlobMatch5075=== RUN TestValidateToken_ValidToken5076=== PAUSE TestValidateToken_ValidToken5077=== RUN TestValidateToken_WrongAudience5078=== PAUSE TestValidateToken_WrongAudience5079=== RUN TestValidateToken_Expired5080=== PAUSE TestValidateToken_Expired5081=== RUN TestValidateToken_BoundClaimsMismatch5082=== PAUSE TestValidateToken_BoundClaimsMismatch5083=== RUN TestValidateToken_BoundSubjectMismatch5084=== PAUSE TestValidateToken_BoundSubjectMismatch5085=== RUN TestValidateToken_MultipleProviders5086=== PAUSE TestValidateToken_MultipleProviders5087=== RUN TestValidateToken_NoMatchingProvider5088=== PAUSE TestValidateToken_NoMatchingProvider5089=== RUN TestValidateToken_KubernetesServiceAccount5090=== PAUSE TestValidateToken_KubernetesServiceAccount5091=== RUN TestNewValidator_KubernetesRequiresCA5092=== PAUSE TestNewValidator_KubernetesRequiresCA5093=== RUN TestValidateToken_KubernetesIssuerFromOwnToken5094=== PAUSE TestValidateToken_KubernetesIssuerFromOwnToken5095=== RUN TestPins_ReservedForMatchingRule5096=== PAUSE TestPins_ReservedForMatchingRule5097=== RUN TestPins_TopLevelShorthand5098=== PAUSE TestPins_TopLevelShorthand5099=== RUN TestPins_ConfigValidation5100=== PAUSE TestPins_ConfigValidation5101=== RUN TestScopes_LegacyProviderDefaultsToWrite5102=== PAUSE TestScopes_LegacyProviderDefaultsToWrite5103=== RUN TestScopes_Rules5104=== PAUSE TestScopes_Rules5105=== RUN TestScopes_ConfigValidation5106=== PAUSE TestScopes_ConfigValidation5107=== CONT TestAudienceForIssuer5108=== CONT TestValidateToken_KubernetesServiceAccount5109=== CONT TestScopes_Rules5110--- PASS: TestAudienceForIssuer (0.00s)5111=== CONT TestValidateToken_NoMatchingProvider5112=== CONT TestValidateToken_MultipleProviders5113=== CONT TestValidateToken_BoundSubjectMismatch5114=== CONT TestValidateToken_BoundClaimsMismatch5115=== CONT TestValidateToken_Expired5116=== CONT TestValidateToken_WrongAudience5117=== CONT TestValidateToken_ValidToken5118=== CONT TestGlobMatch5119=== RUN TestGlobMatch/foo_foo5120=== PAUSE TestGlobMatch/foo_foo5121=== RUN TestGlobMatch/foo_bar5122=== PAUSE TestGlobMatch/foo_bar5123=== RUN TestGlobMatch/*_5124=== PAUSE TestGlobMatch/*_5125=== RUN TestGlobMatch/*_anything5126=== PAUSE TestGlobMatch/*_anything5127=== RUN TestGlobMatch/foo*_foo5128=== CONT TestPins_ReservedForMatchingRule5129=== CONT TestPins_TopLevelShorthand5130=== CONT TestValidateToken_KubernetesIssuerFromOwnToken5131=== CONT TestScopes_ConfigValidation5132=== CONT TestScopes_LegacyProviderDefaultsToWrite5133=== CONT TestNewValidator_KubernetesRequiresCA5134=== CONT TestPins_ConfigValidation5135=== PAUSE TestGlobMatch/foo*_foo5136=== RUN TestGlobMatch/foo*_foobar5137=== PAUSE TestGlobMatch/foo*_foobar5138=== RUN TestGlobMatch/foo*_bar5139=== PAUSE TestGlobMatch/foo*_bar5140=== RUN TestGlobMatch/*bar_bar5141=== PAUSE TestGlobMatch/*bar_bar5142=== RUN TestGlobMatch/*bar_foobar5143=== PAUSE TestGlobMatch/*bar_foobar5144=== RUN TestGlobMatch/*bar_foo5145=== PAUSE TestGlobMatch/*bar_foo5146=== RUN TestGlobMatch/foo*bar_foobar5147=== PAUSE TestGlobMatch/foo*bar_foobar5148=== RUN TestGlobMatch/foo*bar_foo123bar5149=== PAUSE TestGlobMatch/foo*bar_foo123bar5150=== RUN TestGlobMatch/foo*bar_foobarbaz5151=== PAUSE TestGlobMatch/foo*bar_foobarbaz5152=== RUN TestGlobMatch/*/*_foo/bar5153=== PAUSE TestGlobMatch/*/*_foo/bar5154=== RUN TestGlobMatch/*/*_foo5155=== PAUSE TestGlobMatch/*/*_foo5156=== RUN TestGlobMatch/refs/heads/*_refs/heads/main5157=== PAUSE TestGlobMatch/refs/heads/*_refs/heads/main5158=== RUN TestGlobMatch/refs/heads/*_refs/tags/v1.05159=== PAUSE TestGlobMatch/refs/heads/*_refs/tags/v1.05160=== RUN TestGlobMatch/refs/*/main_refs/heads/main5161=== PAUSE TestGlobMatch/refs/*/main_refs/heads/main5162=== RUN TestGlobMatch/fo?_foo5163=== PAUSE TestGlobMatch/fo?_foo5164=== RUN TestGlobMatch/fo?_fo5165=== PAUSE TestGlobMatch/fo?_fo5166=== RUN TestGlobMatch/fo?_fooo5167=== PAUSE TestGlobMatch/fo?_fooo5168=== RUN TestGlobMatch/?oo_foo5169=== PAUSE TestGlobMatch/?oo_foo5170=== RUN TestGlobMatch/?oo_boo5171=== PAUSE TestGlobMatch/?oo_boo5172=== RUN TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5173=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5174=== RUN TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5175=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5176=== CONT TestGlobMatch/foo_foo5177=== CONT TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5178=== CONT TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5179=== CONT TestGlobMatch/foo*bar_foo123bar5180=== CONT TestGlobMatch/foo*bar_foobar5181=== CONT TestGlobMatch/*bar_foo5182=== CONT TestGlobMatch/*bar_foobar5183=== CONT TestGlobMatch/*bar_bar5184=== CONT TestGlobMatch/foo*_bar5185=== CONT TestGlobMatch/foo*_foobar5186=== CONT TestGlobMatch/foo*_foo5187=== CONT TestGlobMatch/*_anything5188=== CONT TestGlobMatch/*_5189=== CONT TestGlobMatch/foo_bar5190=== CONT TestGlobMatch/fo?_foo5191=== CONT TestGlobMatch/?oo_boo5192=== CONT TestGlobMatch/?oo_foo5193=== CONT TestGlobMatch/fo?_fooo5194=== CONT TestGlobMatch/fo?_fo5195=== CONT TestGlobMatch/refs/heads/*_refs/heads/main5196=== CONT TestGlobMatch/refs/*/main_refs/heads/main5197=== CONT TestGlobMatch/refs/heads/*_refs/tags/v1.05198=== CONT TestGlobMatch/*/*_foo5199=== CONT TestGlobMatch/*/*_foo/bar5200=== CONT TestGlobMatch/foo*bar_foobarbaz5201--- PASS: TestGlobMatch (0.06s)5202 --- PASS: TestGlobMatch/foo_foo (0.00s)5203 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main (0.00s)5204 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main (0.00s)5205 --- PASS: TestGlobMatch/foo*bar_foo123bar (0.00s)5206 --- PASS: TestGlobMatch/foo*bar_foobar (0.00s)5207 --- PASS: TestGlobMatch/*bar_foo (0.00s)5208 --- PASS: TestGlobMatch/*bar_foobar (0.00s)5209 --- PASS: TestGlobMatch/*bar_bar (0.00s)5210 --- PASS: TestGlobMatch/foo*_bar (0.00s)5211 --- PASS: TestGlobMatch/foo*_foobar (0.00s)5212 --- PASS: TestGlobMatch/foo*_foo (0.00s)5213 --- PASS: TestGlobMatch/*_anything (0.00s)5214 --- PASS: TestGlobMatch/*_ (0.00s)5215 --- PASS: TestGlobMatch/foo_bar (0.00s)5216 --- PASS: TestGlobMatch/fo?_foo (0.00s)5217 --- PASS: TestGlobMatch/?oo_boo (0.00s)5218 --- PASS: TestGlobMatch/?oo_foo (0.00s)5219 --- PASS: TestGlobMatch/fo?_fooo (0.00s)5220 --- PASS: TestGlobMatch/fo?_fo (0.00s)5221 --- PASS: TestGlobMatch/refs/heads/*_refs/heads/main (0.00s)5222 --- PASS: TestGlobMatch/refs/*/main_refs/heads/main (0.00s)5223 --- PASS: TestGlobMatch/refs/heads/*_refs/tags/v1.0 (0.00s)5224 --- PASS: TestGlobMatch/*/*_foo (0.00s)5225 --- PASS: TestGlobMatch/*/*_foo/bar (0.00s)5226 --- PASS: TestGlobMatch/foo*bar_foobarbaz (0.00s)5227--- PASS: TestScopes_ConfigValidation (0.00s)5228--- PASS: TestPins_ConfigValidation (0.00s)52292026/11/11 23:30:18 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:36439/oidc52302026/11/11 23:30:18 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:43935/oidc5231--- PASS: TestValidateToken_BoundSubjectMismatch (3.50s)52322026/11/11 23:30:21 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:33367/oidc52332026/11/11 23:30:21 INFO OIDC provider initialized name=kubernetes issuer=https://127.0.0.1:428255234--- PASS: TestScopes_Rules (3.54s)5235--- PASS: TestValidateToken_BoundClaimsMismatch (3.49s)5236--- PASS: TestValidateToken_KubernetesServiceAccount (3.55s)52372026/11/11 23:30:21 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:32905/oidc5238--- PASS: TestPins_ReservedForMatchingRule (2.95s)52392026/11/11 23:30:21 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:33663/oidc5240--- PASS: TestPins_TopLevelShorthand (2.97s)52412026/11/11 23:30:21 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:39783/oidc5242--- PASS: TestValidateToken_ValidToken (3.04s)52432026/11/11 23:30:21 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:41783/oidc5244--- PASS: TestValidateToken_WrongAudience (3.13s)52452026/11/11 23:30:22 INFO OIDC provider initialized name=kubernetes issuer=https://oidc.eks.invalid/id/ABC12352462026/11/11 23:30:22 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:41929/oidc5247--- PASS: TestValidateToken_KubernetesIssuerFromOwnToken (3.09s)5248--- PASS: TestValidateToken_Expired (3.66s)52492026/11/11 23:30:22 INFO OIDC provider initialized name=provider1 issuer=http://127.0.0.1:33971/oidc5250--- PASS: TestValidateToken_NoMatchingProvider (3.73s)52512026/11/11 23:30:22 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:34463/oidc5252--- PASS: TestScopes_LegacyProviderDefaultsToWrite (3.20s)52532026/11/11 23:30:22 INFO OIDC provider initialized name=provider1 issuer=http://127.0.0.1:41761/oidc52542026/11/11 23:30:22 INFO OIDC provider initialized name=provider2 issuer=http://127.0.0.1:43037/oidc5255--- PASS: TestValidateToken_MultipleProviders (3.82s)52562026/11/11 23:30:22 http: TLS handshake error from 127.0.0.1:44584: remote error: tls: bad certificate5257--- PASS: TestNewValidator_KubernetesRequiresCA (3.41s)5258PASS5259Running hook tests...5260=== RUN TestSendPathsEmpty5261=== PAUSE TestSendPathsEmpty5262=== RUN TestQueueEnqueueAndFetch5263=== PAUSE TestQueueEnqueueAndFetch5264=== RUN TestQueueDeduplication5265=== PAUSE TestQueueDeduplication5266=== RUN TestQueueRemove5267=== PAUSE TestQueueRemove5268=== RUN TestQueueFetchBatchLimit5269=== PAUSE TestQueueFetchBatchLimit5270=== RUN TestQueueRetryMovesToBack5271=== PAUSE TestQueueRetryMovesToBack5272=== RUN TestQueueFetchRemoveLifecycle5273=== PAUSE TestQueueFetchRemoveLifecycle5274=== RUN TestQueueConcurrentWriters5275=== PAUSE TestQueueConcurrentWriters5276=== RUN TestQueueRemoveLargeClosure5277=== PAUSE TestQueueRemoveLargeClosure5278=== RUN TestServerClientIntegration5279--- PASS: TestServerClientIntegration (0.00s)5280=== RUN TestServerQueueError52812026/11/11 23:30:22 ERROR Failed to queue paths error="permission denied" count=15282--- PASS: TestServerQueueError (0.00s)5283=== RUN TestGetListenerSocketActivation5284 server_test.go:219: === RUN TestGetListenerSocketActivation5285 --- PASS: TestGetListenerSocketActivation (0.00s)5286 PASS5287 5288--- PASS: TestGetListenerSocketActivation (0.05s)5289=== RUN TestServerStalledClientDoesNotBlockShutdown5290=== PAUSE TestServerStalledClientDoesNotBlockShutdown5291=== RUN TestServerBacksOffOnAcceptErrors5292=== PAUSE TestServerBacksOffOnAcceptErrors5293=== RUN TestDrainIsolatesPoisonPath5294=== PAUSE TestDrainIsolatesPoisonPath5295=== RUN TestRunNotBlockedByPoisonHead5296=== PAUSE TestRunNotBlockedByPoisonHead5297=== RUN TestDrainGivesUpWhenServerDown5298=== PAUSE TestDrainGivesUpWhenServerDown5299=== RUN TestFailedPathPrunedByLaterClosure5300=== PAUSE TestFailedPathPrunedByLaterClosure5301=== RUN TestWorkerUploadsAndRemoves5302=== PAUSE TestWorkerUploadsAndRemoves5303=== RUN TestWorkerSkipsGCdPaths5304=== PAUSE TestWorkerSkipsGCdPaths5305=== RUN TestWorkerPrunesClosureDeps5306=== PAUSE TestWorkerPrunesClosureDeps5307=== RUN TestDrainTimeout5308=== PAUSE TestDrainTimeout5309=== RUN TestWorkerRemoveFailureIsNotProgress5310=== PAUSE TestWorkerRemoveFailureIsNotProgress5311=== RUN TestWorkerKeepsPathItCannotStat5312=== PAUSE TestWorkerKeepsPathItCannotStat5313=== CONT TestSendPathsEmpty5314=== CONT TestWorkerRemoveFailureIsNotProgress5315--- PASS: TestSendPathsEmpty (0.00s)5316=== CONT TestDrainTimeout5317=== CONT TestWorkerPrunesClosureDeps5318=== CONT TestWorkerKeepsPathItCannotStat5319=== CONT TestWorkerSkipsGCdPaths5320=== CONT TestWorkerUploadsAndRemoves5321=== CONT TestFailedPathPrunedByLaterClosure5322=== CONT TestDrainGivesUpWhenServerDown5323=== CONT TestRunNotBlockedByPoisonHead5324=== CONT TestDrainIsolatesPoisonPath5325=== CONT TestServerBacksOffOnAcceptErrors5326=== CONT TestServerStalledClientDoesNotBlockShutdown5327=== CONT TestQueueRemoveLargeClosure5328=== CONT TestQueueConcurrentWriters5329=== CONT TestQueueFetchRemoveLifecycle5330=== CONT TestQueueRetryMovesToBack5331=== CONT TestQueueFetchBatchLimit5332=== CONT TestQueueRemove5333=== CONT TestQueueDeduplication5334=== CONT TestQueueEnqueueAndFetch53352026/11/11 23:30:22 ERROR Accept failed error="too many open files"53362026/11/11 23:30:22 ERROR Accept failed error="too many open files"53372026/11/11 23:30:23 ERROR Failed to decode request error="read unix /build/hook4037521238/test.sock->@: i/o timeout"53382026/11/11 23:30:23 ERROR Failed to write response error="write unix /build/hook4037521238/test.sock->@: i/o timeout"5339--- PASS: TestServerStalledClientDoesNotBlockShutdown (0.94s)53402026/11/11 23:30:23 ERROR Accept failed error="too many open files"53412026/11/11 23:30:23 ERROR Accept failed error="too many open files"53422026/11/11 23:30:23 ERROR Accept failed error="too many open files"53432026/11/11 23:30:23 ERROR Accept failed error="too many open files"53442026/11/11 23:30:24 ERROR Accept failed error="too many open files"53452026/11/11 23:30:24 ERROR Accept failed error="too many open files"5346--- PASS: TestServerBacksOffOnAcceptErrors (1.58s)53472026/11/11 23:30:24 INFO Upload queue status pending=253482026/11/11 23:30:24 INFO Uploading batch count=153492026/11/11 23:30:24 INFO Upload queue status pending=353502026/11/11 23:30:24 INFO Uploading batch count=153512026/11/11 23:30:24 ERROR Upload failed error="upload failed" count=15352--- PASS: TestWorkerPrunesClosureDeps (1.89s)53532026/11/11 23:30:24 INFO Uploading batch count=453542026/11/11 23:30:26 ERROR Upload failed error="upload failed" count=453552026/11/11 23:30:26 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainIsolatesPoisonPath1905593209/002/bbb5356--- PASS: TestQueueFetchBatchLimit (3.19s)53572026/11/11 23:30:26 INFO Uploading batch count=253582026/11/11 23:30:26 INFO Uploading batch count=153592026/11/11 23:30:26 INFO Upload queue status pending=253602026/11/11 23:30:26 INFO Upload queue status pending=253612026/11/11 23:30:26 ERROR Upload failed error="upload failed" count=153622026/11/11 23:30:26 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress3250699956/002/nonexistent53632026/11/11 23:30:26 INFO Uploading batch count=253642026/11/11 23:30:26 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=153652026/11/11 23:30:26 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat632966360/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat632966360/002/locked/aaa: permission denied"53662026/11/11 23:30:26 ERROR Upload failed error="upload failed" count=253672026/11/11 23:30:26 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown98855242/002/a5368--- PASS: TestQueueDeduplication (3.19s)53692026/11/11 23:30:26 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerSkipsGCdPaths2303481036/002/nonexistent53702026/11/11 23:30:26 INFO Uploading batch count=253712026/11/11 23:30:26 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress3250699956/002/nonexistent53722026/11/11 23:30:26 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=15373--- PASS: TestQueueRetryMovesToBack (4.06s)5374--- PASS: TestQueueRemove (4.06s)5375--- PASS: TestQueueFetchRemoveLifecycle (4.06s)53762026/11/11 23:30:26 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress3250699956/002/nonexistent53772026/11/11 23:30:26 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat632966360/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat632966360/002/locked/aaa: permission denied"53782026/11/11 23:30:26 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown98855242/002/b5379--- PASS: TestQueueEnqueueAndFetch (4.06s)53802026/11/11 23:30:26 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=153812026/11/11 23:30:26 ERROR Drain finished with paths left in queue remaining=153822026/11/11 23:30:26 INFO Uploading batch count=153832026/11/11 23:30:26 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat632966360/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat632966360/002/locked/aaa: permission denied"53842026/11/11 23:30:26 ERROR Upload failed error="upload failed" count=153852026/11/11 23:30:26 INFO Uploading batch count=153862026/11/11 23:30:26 INFO Uploading batch count=253872026/11/11 23:30:26 ERROR Drain finished with paths left in queue remaining=153882026/11/11 23:30:26 INFO Uploading batch count=153892026/11/11 23:30:26 ERROR Upload failed error="upload failed" count=253902026/11/11 23:30:26 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown98855242/002/c5391--- PASS: TestWorkerRemoveFailureIsNotProgress (4.07s)53922026/11/11 23:30:26 INFO Uploading batch count=153932026/11/11 23:30:26 ERROR Upload failed error="upload failed" count=153942026/11/11 23:30:26 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown98855242/002/d53952026/11/11 23:30:26 INFO Uploading batch count=153962026/11/11 23:30:26 INFO Uploading batch count=153972026/11/11 23:30:26 ERROR Upload failed error="upload failed" count=153982026/11/11 23:30:26 INFO Uploading batch count=253992026/11/11 23:30:26 ERROR Upload failed error="upload failed" count=254002026/11/11 23:30:26 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown98855242/002/e5401--- PASS: TestWorkerKeepsPathItCannotStat (4.07s)54022026/11/11 23:30:26 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown98855242/002/f54032026/11/11 23:30:26 ERROR Drain finished with paths left in queue remaining=154042026/11/11 23:30:26 ERROR Drain finished with paths left in queue remaining=105405--- PASS: TestFailedPathPrunedByLaterClosure (4.07s)5406--- PASS: TestDrainIsolatesPoisonPath (4.07s)5407--- PASS: TestDrainGivesUpWhenServerDown (4.07s)5408--- PASS: TestWorkerUploadsAndRemoves (4.08s)5409--- PASS: TestWorkerSkipsGCdPaths (4.08s)5410--- PASS: TestQueueRemoveLargeClosure (4.17s)5411--- PASS: TestQueueConcurrentWriters (4.17s)54122026/11/11 23:30:27 ERROR Upload failed error="context deadline exceeded" count=254132026/11/11 23:30:27 ERROR Drain finished with paths left in queue remaining=45414=== NAME TestDrainTimeout5415 worker_test.go:407: drain ignored its timeout, took 2.394129475s5416--- FAIL: TestDrainTimeout (4.26s)54172026/11/11 23:30:27 INFO Uploading batch count=154182026/11/11 23:30:27 INFO Uploading batch count=154192026/11/11 23:30:27 INFO Uploading batch count=154202026/11/11 23:30:27 ERROR Upload failed error="upload failed" count=154212026/11/11 23:30:27 INFO Uploading batch count=154222026/11/11 23:30:27 ERROR Upload failed error="upload failed" count=154232026/11/11 23:30:27 INFO Uploading batch count=154242026/11/11 23:30:27 ERROR Upload failed error="upload failed" count=154252026/11/11 23:30:27 INFO Uploading batch count=154262026/11/11 23:30:27 ERROR Upload failed error="upload failed" count=154272026/11/11 23:30:27 ERROR Drain finished with paths left in queue remaining=15428--- PASS: TestRunNotBlockedByPoisonHead (5.07s)5429FAIL