niks3-go-unit-tests
checks.aarch64-linux.go-unit-tests
· build #495
· raw
1tribuchet: building on eliza2Running client tests...3=== RUN TestDoServerRequestAttachesToken4=== PAUSE TestDoServerRequestAttachesToken5=== RUN TestUploadBuildLog_FileBodyReplayedOnRetry6=== PAUSE TestUploadBuildLog_FileBodyReplayedOnRetry7=== RUN TestRegisterUploadedObjectReusesConnections8=== PAUSE TestRegisterUploadedObjectReusesConnections9=== RUN TestRunGarbageCollection_FinishedOnAnotherReplica10=== PAUSE TestRunGarbageCollection_FinishedOnAnotherReplica11=== RUN TestRunGarbageCollection_NotFoundAfterLocalRun12=== PAUSE TestRunGarbageCollection_NotFoundAfterLocalRun13=== RUN TestCaseHackSuffix14=== PAUSE TestCaseHackSuffix15=== RUN TestFilterOversizedClosures16=== PAUSE TestFilterOversizedClosures17=== RUN TestUploadMultipart_PartsInParallel18=== PAUSE TestUploadMultipart_PartsInParallel19=== RUN TestUploadMultipart_ReturnsAfterDroppedBody20=== PAUSE TestUploadMultipart_ReturnsAfterDroppedBody21=== RUN TestPartSizeForNAR22=== PAUSE TestPartSizeForNAR23=== RUN TestDumpPathCaseHackMatchesNix24--- PASS: TestDumpPathCaseHackMatchesNix (0.40s)25=== RUN TestDumpPathCaseHackCollision26--- PASS: TestDumpPathCaseHackCollision (0.00s)27=== RUN TestDumpPathMatchesNix28=== PAUSE TestDumpPathMatchesNix29=== RUN TestDumpPathSingleFile30=== PAUSE TestDumpPathSingleFile31=== RUN TestDumpPathWriterError32=== PAUSE TestDumpPathWriterError33=== RUN TestEncodeNixBase3234=== PAUSE TestEncodeNixBase3235=== RUN TestEncodeNixBase32WithRealHash36=== PAUSE TestEncodeNixBase32WithRealHash37=== RUN TestConvertHashToNix3238=== PAUSE TestConvertHashToNix3239=== RUN TestGetStorePathHash40=== PAUSE TestGetStorePathHash41=== RUN TestPathInfoHashCompatibility42=== PAUSE TestPathInfoHashCompatibility43=== RUN TestParsePathInfoJSON44=== PAUSE TestParsePathInfoJSON45=== RUN TestParsePathInfoJSONMultiplePaths46=== PAUSE TestParsePathInfoJSONMultiplePaths47=== RUN TestPathInfoCACompatibility48=== PAUSE TestPathInfoCACompatibility49=== RUN TestRateLimiterFeedback50=== PAUSE TestRateLimiterFeedback51=== RUN TestRateLimiterFeedback_400DoesNotCountAsSuccess52=== PAUSE TestRateLimiterFeedback_400DoesNotCountAsSuccess53=== RUN TestRegisterUploadedObject_BoundedAgainstSilentServer54=== PAUSE TestRegisterUploadedObject_BoundedAgainstSilentServer55=== RUN TestRegisterUploadedObject_OutlivesPush56=== PAUSE TestRegisterUploadedObject_OutlivesPush57=== RUN TestResolveStorePath58=== PAUSE TestResolveStorePath59=== RUN TestDoWithRetry_BodyReplayedViaGetBody60=== PAUSE TestDoWithRetry_BodyReplayedViaGetBody61=== RUN TestRetryTimeoutOutlastsMaxRetries62=== PAUSE TestRetryTimeoutOutlastsMaxRetries63=== RUN TestDoWithRetry_FinalResponseBodyReadable64=== PAUSE TestDoWithRetry_FinalResponseBodyReadable65=== RUN TestShellSplit66=== PAUSE TestShellSplit67=== RUN TestShellSplitErrors68=== PAUSE TestShellSplitErrors69=== RUN TestStreamPushReportsEveryPath70=== PAUSE TestStreamPushReportsEveryPath71=== RUN TestStreamPushBatchesUnderLoad72=== PAUSE TestStreamPushBatchesUnderLoad73=== RUN TestStreamPushIsolatesFailures74=== PAUSE TestStreamPushIsolatesFailures75=== RUN TestStreamPushGivesUpOnDeadServer76=== PAUSE TestStreamPushGivesUpOnDeadServer77=== RUN TestStreamPushRequestLine78=== PAUSE TestStreamPushRequestLine79=== RUN TestStreamPushReportsSignatures80=== PAUSE TestStreamPushReportsSignatures81=== RUN TestStreamPushReportsSkippedPaths82=== PAUSE TestStreamPushReportsSkippedPaths83=== RUN TestClientSignaturesByStorePath84=== PAUSE TestClientSignaturesByStorePath85=== RUN TestStreamPushStopsWhenCanceled86=== PAUSE TestStreamPushStopsWhenCanceled87=== RUN TestSetClientTLS88=== PAUSE TestSetClientTLS89=== RUN TestSetClientTLSDoesNotMutateDefaultTransport90=== PAUSE TestSetClientTLSDoesNotMutateDefaultTransport91=== RUN TestSetClientTLSErrors92=== PAUSE TestSetClientTLSErrors93=== RUN TestStaticToken94=== PAUSE TestStaticToken95=== RUN TestFileTokenReadsAndCaches96=== PAUSE TestFileTokenReadsAndCaches97=== RUN TestFileTokenMissing98=== PAUSE TestFileTokenMissing99=== RUN TestFileTokenEmpty100=== PAUSE TestFileTokenEmpty101=== RUN TestScriptTokenNoExpiryRerunsEveryCall102=== PAUSE TestScriptTokenNoExpiryRerunsEveryCall103=== RUN TestScriptTokenCachesUntilRefresh104=== PAUSE TestScriptTokenCachesUntilRefresh105=== RUN TestScriptTokenEmptyToken106=== PAUSE TestScriptTokenEmptyToken107=== RUN TestScriptTokenBadJSON108=== PAUSE TestScriptTokenBadJSON109=== RUN TestScriptTokenScriptFails110=== PAUSE TestScriptTokenScriptFails111=== RUN TestScriptTokenEmptyCommand112=== PAUSE TestScriptTokenEmptyCommand113=== CONT TestDoServerRequestAttachesToken114=== CONT TestScriptTokenEmptyCommand115--- PASS: TestScriptTokenEmptyCommand (0.00s)116=== CONT TestStreamPushStopsWhenCanceled117=== CONT TestEncodeNixBase32118=== CONT TestDoWithRetry_BodyReplayedViaGetBody119=== CONT TestGetStorePathHash120=== RUN TestEncodeNixBase32/test_string_hash121=== CONT TestResolveStorePath122=== PAUSE TestEncodeNixBase32/test_string_hash123=== CONT TestRegisterUploadedObject_OutlivesPush124=== CONT TestRegisterUploadedObject_BoundedAgainstSilentServer125=== CONT TestRateLimiterFeedback_400DoesNotCountAsSuccess126=== CONT TestRateLimiterFeedback127=== RUN TestRateLimiterFeedback/429_enables_limiter128=== PAUSE TestRateLimiterFeedback/429_enables_limiter129=== RUN TestRateLimiterFeedback/503_enables_limiter130=== PAUSE TestRateLimiterFeedback/503_enables_limiter131=== RUN TestRateLimiterFeedback/200_does_not_enable_limiter132=== PAUSE TestRateLimiterFeedback/200_does_not_enable_limiter133=== RUN TestRateLimiterFeedback/400_does_not_enable_limiter134=== PAUSE TestRateLimiterFeedback/400_does_not_enable_limiter135=== CONT TestPathInfoCACompatibility136=== RUN TestPathInfoCACompatibility/null_ca_field137=== PAUSE TestPathInfoCACompatibility/null_ca_field138=== RUN TestPathInfoCACompatibility/old_string_format_-_text139=== PAUSE TestPathInfoCACompatibility/old_string_format_-_text140=== RUN TestPathInfoCACompatibility/old_string_format_-_fixed_recursive141=== PAUSE TestPathInfoCACompatibility/old_string_format_-_fixed_recursive142=== RUN TestPathInfoCACompatibility/new_structured_format_-_text143=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_text144=== RUN TestPathInfoCACompatibility/new_structured_format_-_nar_method145=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_nar_method146=== CONT TestStreamPushGivesUpOnDeadServer147=== CONT TestParsePathInfoJSONMultiplePaths148=== RUN TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths149=== PAUSE TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths150=== RUN TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths151=== PAUSE TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths152=== CONT TestPartSizeForNAR153=== RUN TestPartSizeForNAR/zero_stays_at_minimum154=== PAUSE TestPartSizeForNAR/zero_stays_at_minimum155=== RUN TestPartSizeForNAR/small_stays_at_minimum156=== PAUSE TestPartSizeForNAR/small_stays_at_minimum157=== RUN TestPartSizeForNAR/80_GiB_fits_at_minimum158=== PAUSE TestPartSizeForNAR/80_GiB_fits_at_minimum159=== RUN TestPartSizeForNAR/115_GiB_needs_larger_parts160=== PAUSE TestPartSizeForNAR/115_GiB_needs_larger_parts161=== RUN TestPartSizeForNAR/1_TiB162=== PAUSE TestPartSizeForNAR/1_TiB163=== RUN TestPartSizeForNAR/5_TiB_S3_max_object164=== PAUSE TestPartSizeForNAR/5_TiB_S3_max_object165=== RUN TestPartSizeForNAR/capped_at_5_GiB166=== PAUSE TestPartSizeForNAR/capped_at_5_GiB167=== CONT TestStreamPushReportsEveryPath1682026/11/12 00:13:24 ERROR Upload failed error="connection refused" count=201692026/11/12 00:13:24 ERROR Server seems unavailable, giving up on batch untried=17170=== CONT TestRunGarbageCollection_NotFoundAfterLocalRun171=== CONT TestFilterOversizedClosures172=== RUN TestFilterOversizedClosures/no_limit_keeps_everything173=== PAUSE TestFilterOversizedClosures/no_limit_keeps_everything174=== RUN TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped175=== PAUSE TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped176=== RUN TestFilterOversizedClosures/all_closures_skipped177=== PAUSE TestFilterOversizedClosures/all_closures_skipped178=== CONT TestParsePathInfoJSON179=== RUN TestParsePathInfoJSON/Nix_format180=== PAUSE TestParsePathInfoJSON/Nix_format181=== RUN TestParsePathInfoJSON/Lix_format182=== PAUSE TestParsePathInfoJSON/Lix_format183=== RUN TestParsePathInfoJSON/empty_input184=== PAUSE TestParsePathInfoJSON/empty_input185=== RUN TestParsePathInfoJSON/whitespace_only186=== PAUSE TestParsePathInfoJSON/whitespace_only187=== RUN TestParsePathInfoJSON/invalid_JSON188=== PAUSE TestParsePathInfoJSON/invalid_JSON189=== CONT TestUploadMultipart_ReturnsAfterDroppedBody190--- PASS: TestResolveStorePath (0.00s)191--- PASS: TestStreamPushStopsWhenCanceled (0.00s)192--- PASS: TestStreamPushReportsEveryPath (0.00s)193=== CONT TestPathInfoHashCompatibility194=== CONT TestConvertHashToNix32195=== CONT TestEncodeNixBase32WithRealHash196=== CONT TestDumpPathMatchesNix197=== CONT TestDumpPathWriterError198=== CONT TestDumpPathSingleFile199=== CONT TestStreamPushIsolatesFailures200=== CONT TestClientSignaturesByStorePath201=== CONT TestStreamPushReportsSkippedPaths202=== CONT TestStreamPushReportsSignatures203=== CONT TestStreamPushRequestLine204=== RUN TestEncodeNixBase32/empty_input205=== CONT TestShellSplitErrors206=== CONT TestDoWithRetry_FinalResponseBodyReadable207=== RUN TestGetStorePathHash/valid_store_path208--- PASS: TestStreamPushGivesUpOnDeadServer (0.00s)209=== CONT TestStreamPushBatchesUnderLoad210=== RUN TestStreamPushBatchesUnderLoad/oneAtATime=false211=== CONT TestCaseHackSuffix212=== PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=false213=== RUN TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)214=== RUN TestStreamPushBatchesUnderLoad/oneAtATime=true215=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)216=== PAUSE TestGetStorePathHash/valid_store_path217=== RUN TestConvertHashToNix32/SRI_format_to_Nix32218=== PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=true219=== RUN TestPathInfoHashCompatibility/old_string_format_with_colon220=== CONT TestRetryTimeoutOutlastsMaxRetries221=== CONT TestFileTokenEmpty2222026/11/12 00:13:24 ERROR Upload failed error="bad path" count=3223=== CONT TestScriptTokenScriptFails224=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_colon225=== RUN TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI226=== CONT TestScriptTokenBadJSON227=== CONT TestShellSplit228=== PAUSE TestConvertHashToNix32/SRI_format_to_Nix32229=== PAUSE TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI230=== RUN TestConvertHashToNix32/already_Nix32_format2312026/11/12 00:13:24 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:43047232=== CONT TestScriptTokenNoExpiryRerunsEveryCall233=== CONT TestScriptTokenCachesUntilRefresh2342026/11/12 00:13:24 ERROR Upload failed error=boom count=1235=== PAUSE TestConvertHashToNix32/already_Nix32_format236=== RUN TestConvertHashToNix32/invalid_format2372026/11/12 00:13:24 ERROR Upload failed error=boom count=1238=== PAUSE TestConvertHashToNix32/invalid_format239--- PASS: TestEncodeNixBase32WithRealHash (0.00s)2402026/11/12 00:13:24 INFO Garbage collection started241--- PASS: TestClientSignaturesByStorePath (0.00s)2422026/11/12 00:13:24 WARN Rate limiter enabled after throttle name=server-test rate=5243--- PASS: TestStreamPushReportsSkippedPaths (0.00s)244--- PASS: TestStreamPushIsolatesFailures (0.00s)245--- PASS: TestShellSplit (0.00s)246--- PASS: TestFileTokenEmpty (0.00s)247--- PASS: TestShellSplitErrors (0.00s)248--- PASS: TestStreamPushReportsSignatures (0.00s)249=== CONT TestUploadMultipart_PartsInParallel250=== CONT TestRegisterUploadedObjectReusesConnections251=== CONT TestScriptTokenEmptyToken2522026/11/12 00:13:25 WARN Rate limiter enabled after throttle name=server-test rate=52532026/11/12 00:13:25 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:46665254--- PASS: TestScriptTokenScriptFails (0.12s)255=== CONT TestRunGarbageCollection_FinishedOnAnotherReplica2562026/11/12 00:13:25 WARN Rate limiter enabled after throttle name=server-test rate=5257=== RUN TestGetStorePathHash/basename_without_hyphen_should_error258=== PAUSE TestEncodeNixBase32/empty_input259=== RUN TestPathInfoHashCompatibility/new_structured_format_with_sha512260--- PASS: TestDumpPathSingleFile (0.12s)261=== CONT TestFileTokenMissing262=== CONT TestStaticToken263--- PASS: TestRegisterUploadedObject_OutlivesPush (0.56s)264--- PASS: TestFileTokenMissing (0.00s)265=== CONT TestUploadBuildLog_FileBodyReplayedOnRetry266--- PASS: TestScriptTokenBadJSON (0.22s)267=== CONT TestSetClientTLSDoesNotMutateDefaultTransport268=== CONT TestFileTokenReadsAndCaches269=== PAUSE TestPathInfoHashCompatibility/new_structured_format_with_sha512270=== PAUSE TestGetStorePathHash/basename_without_hyphen_should_error271=== RUN TestGetStorePathHash/hash_with_invalid_characters_should_error272=== PAUSE TestGetStorePathHash/hash_with_invalid_characters_should_error273=== RUN TestGetStorePathHash/hash_with_wrong_length_should_error274=== PAUSE TestGetStorePathHash/hash_with_wrong_length_should_error275=== CONT TestRateLimiterFeedback/200_does_not_enable_limiter2762026/11/12 00:13:26 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:44245/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:44245/api/uploads/complete2772026/11/12 00:13:26 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"278=== CONT TestRateLimiterFeedback/503_enables_limiter2792026/11/12 00:13:26 INFO Garbage collection started280=== CONT TestPathInfoCACompatibility/null_ca_field2812026/11/12 00:13:26 WARN Rate limiter backed off name=server-test rate=52822026/11/12 00:13:26 WARN Request returned retryable status, retrying attempt=2 backoff=0s status=503 url=http://127.0.0.1:46665283=== CONT TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths284=== CONT TestRateLimiterFeedback/400_does_not_enable_limiter285--- PASS: TestCaseHackSuffix (0.22s)286=== CONT TestRateLimiterFeedback/429_enables_limiter287--- PASS: TestScriptTokenCachesUntilRefresh (0.22s)288--- PASS: TestStaticToken (0.00s)2892026/11/12 00:13:26 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:44837290--- PASS: TestFileTokenReadsAndCaches (0.00s)291--- PASS: TestRunGarbageCollection_NotFoundAfterLocalRun (1.83s)2922026/11/12 00:13:26 WARN Rate limiter enabled after throttle name=server-test rate=52932026/11/12 00:13:26 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=429 url=http://127.0.0.1:33377294--- PASS: TestDoWithRetry_BodyReplayedViaGetBody (1.83s)2952026/11/12 00:13:26 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:43047296=== CONT TestPartSizeForNAR/zero_stays_at_minimum297=== CONT TestPathInfoCACompatibility/new_structured_format_-_nar_method2982026/11/12 00:13:26 WARN Rate limiter backed off name=server-test rate=52992026/11/12 00:13:26 INFO Garbage collection progress phase=running_on_another_replica failed_uploads_deleted=0 old_closures_deleted=0 objects_marked=0 objects_deleted=0 objects_failed=03002026/11/12 00:13:26 WARN Rate limiter enabled after throttle name=server-test rate=5301=== CONT TestPathInfoCACompatibility/new_structured_format_-_text3022026/11/12 00:13:26 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:36721303=== CONT TestPathInfoCACompatibility/old_string_format_-_text304=== CONT TestFilterOversizedClosures/no_limit_keeps_everything305=== CONT TestParsePathInfoJSON/Nix_format3062026/11/12 00:13:26 WARN Rate limiter backed off name=server-test rate=53072026/11/12 00:13:26 WARN Rate limiter backed off name=server-test rate=5308=== CONT TestFilterOversizedClosures/all_closures_skipped309=== CONT TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths310=== CONT TestPartSizeForNAR/capped_at_5_GiB3112026/11/12 00:13:26 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-small nar_size=1000 max_nar_size=50312=== CONT TestPartSizeForNAR/80_GiB_fits_at_minimum313=== CONT TestPartSizeForNAR/small_stays_at_minimum314=== CONT TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped3152026/11/12 00:13:26 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb-vm-image nar_size=5000 max_nar_size=2000316=== CONT TestParsePathInfoJSON/whitespace_only317=== CONT TestPartSizeForNAR/115_GiB_needs_larger_parts318=== CONT TestParsePathInfoJSON/Lix_format319=== CONT TestPartSizeForNAR/5_TiB_S3_max_object320=== CONT TestStreamPushBatchesUnderLoad/oneAtATime=false321=== CONT TestStreamPushBatchesUnderLoad/oneAtATime=true322=== CONT TestParsePathInfoJSON/invalid_JSON323=== CONT TestConvertHashToNix32/SRI_format_to_Nix32324=== CONT TestConvertHashToNix32/already_Nix32_format3252026/11/12 00:13:26 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:43047326=== CONT TestConvertHashToNix32/invalid_format327=== CONT TestPathInfoCACompatibility/old_string_format_-_fixed_recursive328=== CONT TestPartSizeForNAR/1_TiB329=== CONT TestEncodeNixBase32/empty_input330=== CONT TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)331=== CONT TestGetStorePathHash/valid_store_path332=== CONT TestPathInfoHashCompatibility/old_string_format_with_colon333=== CONT TestPathInfoHashCompatibility/new_structured_format_with_sha512334=== CONT TestGetStorePathHash/basename_without_hyphen_should_error335=== CONT TestGetStorePathHash/hash_with_wrong_length_should_error3362026/11/12 00:13:26 WARN Garbage collection finished on another replica. Its statistics are not available here337=== CONT TestGetStorePathHash/hash_with_invalid_characters_should_error338=== CONT TestParsePathInfoJSON/empty_input339--- PASS: TestDoWithRetry_FinalResponseBodyReadable (1.50s)340=== CONT TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI341=== CONT TestEncodeNixBase32/test_string_hash342--- PASS: TestParsePathInfoJSONMultiplePaths (0.00s)343 --- PASS: TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths (0.00s)344 --- PASS: TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths (0.00s)345--- PASS: TestFilterOversizedClosures (0.00s)346 --- PASS: TestFilterOversizedClosures/no_limit_keeps_everything (0.00s)347 --- PASS: TestFilterOversizedClosures/all_closures_skipped (0.00s)348 --- PASS: TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped (0.00s)349--- PASS: TestRateLimiterFeedback (0.00s)350 --- PASS: TestRateLimiterFeedback/400_does_not_enable_limiter (0.00s)351 --- PASS: TestRateLimiterFeedback/503_enables_limiter (0.00s)352 --- PASS: TestRateLimiterFeedback/429_enables_limiter (0.00s)353 --- PASS: TestRateLimiterFeedback/200_does_not_enable_limiter (0.01s)354--- PASS: TestStreamPushBatchesUnderLoad (0.00s)355 --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=true (0.00s)356 --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=false (0.00s)357--- PASS: TestPartSizeForNAR (0.00s)358 --- PASS: TestPartSizeForNAR/zero_stays_at_minimum (0.00s)359 --- PASS: TestPartSizeForNAR/capped_at_5_GiB (0.00s)360 --- PASS: TestPartSizeForNAR/80_GiB_fits_at_minimum (0.00s)361 --- PASS: TestPartSizeForNAR/small_stays_at_minimum (0.00s)362 --- PASS: TestPartSizeForNAR/115_GiB_needs_larger_parts (0.00s)363 --- PASS: TestPartSizeForNAR/5_TiB_S3_max_object (0.00s)364 --- PASS: TestPartSizeForNAR/1_TiB (0.00s)365--- PASS: TestEncodeNixBase32 (0.56s)366 --- PASS: TestEncodeNixBase32/empty_input (0.00s)367 --- PASS: TestEncodeNixBase32/test_string_hash (0.00s)368--- PASS: TestPathInfoHashCompatibility (1.49s)369 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI) (0.00s)370 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_colon (0.00s)371 --- PASS: TestPathInfoHashCompatibility/new_structured_format_with_sha512 (0.00s)372 --- PASS: TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI (0.00s)373--- PASS: TestConvertHashToNix32 (0.00s)374 --- PASS: TestConvertHashToNix32/SRI_format_to_Nix32 (0.00s)375 --- PASS: TestConvertHashToNix32/already_Nix32_format (0.00s)376 --- PASS: TestConvertHashToNix32/invalid_format (0.00s)377--- PASS: TestUploadMultipart_ReturnsAfterDroppedBody (1.83s)378--- PASS: TestGetStorePathHash (1.83s)379 --- PASS: TestGetStorePathHash/valid_store_path (0.00s)380 --- PASS: TestGetStorePathHash/basename_without_hyphen_should_error (0.00s)381 --- PASS: TestGetStorePathHash/hash_with_wrong_length_should_error (0.00s)382 --- PASS: TestGetStorePathHash/hash_with_invalid_characters_should_error (0.00s)383--- PASS: TestPathInfoCACompatibility (0.00s)384 --- PASS: TestPathInfoCACompatibility/null_ca_field (0.00s)385 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_nar_method (0.00s)386 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_text (0.00s)387 --- PASS: TestPathInfoCACompatibility/old_string_format_-_text (0.00s)388 --- PASS: TestPathInfoCACompatibility/old_string_format_-_fixed_recursive (0.00s)389--- PASS: TestRunGarbageCollection_FinishedOnAnotherReplica (1.37s)390--- PASS: TestParsePathInfoJSON (0.00s)391 --- PASS: TestParsePathInfoJSON/Nix_format (0.00s)392 --- PASS: TestParsePathInfoJSON/whitespace_only (0.00s)393 --- PASS: TestParsePathInfoJSON/Lix_format (0.00s)394 --- PASS: TestParsePathInfoJSON/invalid_JSON (0.00s)395 --- PASS: TestParsePathInfoJSON/empty_input (0.00s)3962026/11/12 00:13:26 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:44245/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:44245/api/uploads/complete397--- PASS: TestDoServerRequestAttachesToken (1.84s)3982026/11/12 00:13:26 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"3992026/11/12 00:13:26 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:44245/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:44245/api/uploads/complete4002026/11/12 00:13:26 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"4012026/11/12 00:13:26 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:430474022026/11/12 00:13:26 WARN Rate limiter enabled after throttle name=s3-test rate=5403--- PASS: TestRegisterUploadedObject_BoundedAgainstSilentServer (1.84s)4042026/11/12 00:13:26 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:45837405=== CONT TestSetClientTLSErrors406--- PASS: TestScriptTokenEmptyToken (1.38s)407=== CONT TestSetClientTLS4082026/11/12 00:13:26 WARN Request returned retryable status, retrying attempt=3 backoff=1ms status=502 url=http://127.0.0.1:430474092026/11/12 00:13:26 WARN Request returned retryable status, retrying attempt=4 backoff=1ms status=502 url=http://127.0.0.1:43047410--- PASS: TestUploadBuildLog_FileBodyReplayedOnRetry (1.28s)411=== RUN TestSetClientTLSErrors/missing_cert_file412=== PAUSE TestSetClientTLSErrors/missing_cert_file413=== RUN TestSetClientTLSErrors/missing_key_file414=== PAUSE TestSetClientTLSErrors/missing_key_file415=== RUN TestSetClientTLSErrors/missing_ca_file416=== PAUSE TestSetClientTLSErrors/missing_ca_file417=== RUN TestSetClientTLSErrors/invalid_ca_file418=== PAUSE TestSetClientTLSErrors/invalid_ca_file419=== CONT TestSetClientTLSErrors/missing_cert_file4202026/11/12 00:13:26 WARN Request returned retryable status, retrying attempt=5 backoff=1ms status=502 url=http://127.0.0.1:43047421=== CONT TestSetClientTLSErrors/invalid_ca_file422--- PASS: TestSetClientTLSDoesNotMutateDefaultTransport (1.28s)423=== CONT TestSetClientTLSErrors/missing_key_file424=== CONT TestSetClientTLSErrors/missing_ca_file4252026/11/12 00:13:26 WARN Request returned retryable status, retrying attempt=6 backoff=1ms status=502 url=http://127.0.0.1:43047426=== RUN TestSetClientTLS/rejects_connection_without_client_cert427=== PAUSE TestSetClientTLS/rejects_connection_without_client_cert428=== RUN TestSetClientTLS/succeeds_with_client_cert_and_CA429=== PAUSE TestSetClientTLS/succeeds_with_client_cert_and_CA430=== RUN TestSetClientTLS/preserves_debug_logging_transport431=== PAUSE TestSetClientTLS/preserves_debug_logging_transport432=== CONT TestSetClientTLS/rejects_connection_without_client_cert433=== CONT TestSetClientTLS/preserves_debug_logging_transport434--- PASS: TestSetClientTLSErrors (0.00s)435 --- PASS: TestSetClientTLSErrors/missing_cert_file (0.00s)436 --- PASS: TestSetClientTLSErrors/missing_key_file (0.00s)437 --- PASS: TestSetClientTLSErrors/invalid_ca_file (0.00s)438 --- PASS: TestSetClientTLSErrors/missing_ca_file (0.00s)439=== CONT TestSetClientTLS/succeeds_with_client_cert_and_CA4402026/11/12 00:13:26 WARN Request returned retryable status, retrying attempt=7 backoff=1ms status=502 url=http://127.0.0.1:430474412026/11/12 00:13:26 WARN Request returned retryable status, retrying attempt=8 backoff=1ms status=502 url=http://127.0.0.1:43047442--- PASS: TestScriptTokenNoExpiryRerunsEveryCall (1.51s)4432026/11/12 00:13:26 WARN Request returned retryable status, retrying attempt=9 backoff=1ms status=502 url=http://127.0.0.1:430474442026/11/12 00:13:26 WARN Request returned retryable status, retrying attempt=10 backoff=1ms status=502 url=http://127.0.0.1:43047445--- PASS: TestRegisterUploadedObjectReusesConnections (1.52s)4462026/11/12 00:13:26 WARN Request returned retryable status, retrying attempt=11 backoff=1ms status=502 url=http://127.0.0.1:430474472026/11/12 00:13:26 WARN Request returned retryable status, retrying attempt=12 backoff=1ms status=502 url=http://127.0.0.1:43047448--- PASS: TestStreamPushRequestLine (1.52s)4492026/11/12 00:13:26 http: TLS handshake error from 127.0.0.1:59444: remote error: tls: bad certificate4502026/11/12 00:13:26 WARN Request returned retryable status, retrying attempt=13 backoff=1ms status=502 url=http://127.0.0.1:43047451--- PASS: TestSetClientTLS (0.01s)452 --- PASS: TestSetClientTLS/succeeds_with_client_cert_and_CA (0.01s)453 --- PASS: TestSetClientTLS/preserves_debug_logging_transport (0.02s)454 --- PASS: TestSetClientTLS/rejects_connection_without_client_cert (0.02s)4552026/11/12 00:13:26 WARN Request returned retryable status, retrying attempt=14 backoff=1ms status=502 url=http://127.0.0.1:430474562026/11/12 00:13:26 WARN Request returned retryable status, retrying attempt=15 backoff=1ms status=502 url=http://127.0.0.1:430474572026/11/12 00:13:26 WARN Request returned retryable status, retrying attempt=16 backoff=1ms status=502 url=http://127.0.0.1:430474582026/11/12 00:13:26 WARN Request returned retryable status, retrying attempt=17 backoff=1ms status=502 url=http://127.0.0.1:43047459--- PASS: TestRetryTimeoutOutlastsMaxRetries (1.53s)460--- PASS: TestDumpPathWriterError (1.53s)461--- PASS: TestDumpPathMatchesNix (1.76s)462--- PASS: TestRateLimiterFeedback_400DoesNotCountAsSuccess (2.43s)463--- PASS: TestUploadMultipart_PartsInParallel (2.13s)464PASS465Running server tests...466The files belonging to this database system will be owned by user "nixbld".467This user must also own the server process.468469The database cluster will be initialized with locale "C".470The default database encoding has accordingly been set to "SQL_ASCII".471The default text search configuration will be set to "english".472473Data page checksums are enabled.474475creating directory /build/postgres1650060844/data ... ok476creating subdirectories ... ok477selecting dynamic shared memory implementation ... posix478selecting default "max_connections" ... 100479selecting default "shared_buffers" ... 128MB480selecting default time zone ... UTC481creating configuration files ... ok482running bootstrap script ... ok483performing post-bootstrap initialization ... ok484syncing data to disk ... ok485486initdb: warning: enabling "trust" authentication for local connections487initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.488489Success. You can now start the database server using:490491 pg_ctl -D /build/postgres1650060844/data -l logfile start492493/build/postgres1650060844:5432 - no response494/build/postgres1650060844:5432 - no response495/build/postgres1650060844:5432 - no response496/build/postgres1650060844:5432 - no response497/build/postgres1650060844:5432 - no response498/build/postgres1650060844:5432 - no response499/build/postgres1650060844:5432 - no response500/build/postgres1650060844:5432 - no response501/build/postgres1650060844:5432 - no response502/build/postgres1650060844:5432 - no response503/build/postgres1650060844:5432 - no response504/build/postgres1650060844:5432 - no response505/build/postgres1650060844:5432 - no response506/build/postgres1650060844:5432 - no response507/build/postgres1650060844:5432 - no response508/build/postgres1650060844:5432 - no response509/build/postgres1650060844:5432 - no response510/build/postgres1650060844:5432 - no response511/build/postgres1650060844:5432 - no response512/build/postgres1650060844:5432 - no response513/build/postgres1650060844:5432 - no response514/build/postgres1650060844:5432 - no response515/build/postgres1650060844:5432 - no response516/build/postgres1650060844:5432 - no response517/build/postgres1650060844:5432 - no response518/build/postgres1650060844:5432 - no response519/build/postgres1650060844:5432 - no response520/build/postgres1650060844:5432 - no response521/build/postgres1650060844:5432 - no response522/build/postgres1650060844:5432 - no response523/build/postgres1650060844:5432 - no response524/build/postgres1650060844:5432 - no response525/build/postgres1650060844:5432 - no response526/build/postgres1650060844:5432 - no response527/build/postgres1650060844:5432 - no response528/build/postgres1650060844:5432 - no response529/build/postgres1650060844:5432 - no response530/build/postgres1650060844:5432 - no response5312026-11-12 00:13:50.862 UTC [128] LOG: starting PostgreSQL 18.6 on aarch64-unknown-linux-gnu, compiled by clang version 21.1.8, 64-bit5322026-11-12 00:13:50.864 UTC [128] LOG: listening on Unix socket "/build/postgres1650060844/.s.PGSQL.5432"5332026-11-12 00:13:50.871 UTC [172] LOG: database system was shut down at 2026-11-12 00:13:46 UTC5342026-11-12 00:13:50.877 UTC [128] LOG: database system is ready to accept connections535/build/postgres1650060844:5432 - accepting connections536{"timestamp":"2026-11-12T00:14:18.02231981Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"717e553d-eda5-4993-acb1-79bfef843074","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"GET","uri":"/health/ready","status_code":503,"suppressed_errors":0,"duration_ms":18,"result":"server_error","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":502,"threadName":"rustfs-worker","threadId":"ThreadId(199)"}537{"timestamp":"2026-11-12T00:14:18.134081948Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"c1cd01d5-8735-4556-9200-e9ce31b86d8d","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"GET","uri":"/health/ready","status_code":503,"suppressed_errors":0,"duration_ms":0,"result":"server_error","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":502,"threadName":"rustfs-worker","threadId":"ThreadId(199)"}538{"timestamp":"2026-11-12T00:14:18.235999998Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"2bfcfce3-e2d9-47bb-9695-8d7dd311385b","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"GET","uri":"/health/ready","status_code":503,"suppressed_errors":0,"duration_ms":0,"result":"server_error","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":502,"threadName":"rustfs-worker","threadId":"ThreadId(198)"}539{"timestamp":"2026-11-12T00:14:18.339631204Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"c544cc17-69d7-48e3-ac4a-007bdf392e10","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"GET","uri":"/health/ready","status_code":503,"suppressed_errors":0,"duration_ms":0,"result":"server_error","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":502,"threadName":"rustfs-worker","threadId":"ThreadId(201)"}540=== RUN TestAbortSucceeded541=== PAUSE TestAbortSucceeded542=== RUN TestService_AuthMiddleware543=== PAUSE TestService_AuthMiddleware544=== RUN TestService_AuthMiddleware_MTLSProxyHeader545=== PAUSE TestService_AuthMiddleware_MTLSProxyHeader546=== RUN TestService_AuthMiddleware_MTLSBoundSubjects547=== PAUSE TestService_AuthMiddleware_MTLSBoundSubjects548=== RUN TestService_ReadAuthMiddleware549=== PAUSE TestService_ReadAuthMiddleware550=== RUN TestService_AuthMiddleware_OIDC551=== PAUSE TestService_AuthMiddleware_OIDC552=== RUN TestService_RequireScope_OIDC553=== PAUSE TestService_RequireScope_OIDC554=== RUN TestService_ReadScope_PublicByDefault555=== PAUSE TestService_ReadScope_PublicByDefault556=== RUN TestCacheConfigHandler557=== PAUSE TestCacheConfigHandler558=== RUN TestCacheStatsHandler559=== PAUSE TestCacheStatsHandler560=== RUN TestClientCADerivations561=== PAUSE TestClientCADerivations562=== RUN TestClientErrorHandling563=== PAUSE TestClientErrorHandling564=== RUN TestClientIntegration565=== PAUSE TestClientIntegration566=== RUN TestClientMultipleUploads567=== PAUSE TestClientMultipleUploads568=== RUN TestClientWithDependencies569=== PAUSE TestClientWithDependencies570=== RUN TestClientSharedPathCommittedMidPush571=== PAUSE TestClientSharedPathCommittedMidPush572=== RUN TestPinProtectsFromGC573=== PAUSE TestPinProtectsFromGC574=== RUN TestClientPushesUseOnePush575=== PAUSE TestClientPushesUseOnePush576=== RUN TestClientFallsBackToClosures577=== PAUSE TestClientFallsBackToClosures578=== RUN TestDeleteClosuresRacingPins579=== PAUSE TestDeleteClosuresRacingPins580=== RUN TestConcurrentCommitsSharingObjectsDoNotDeadlock581=== PAUSE TestConcurrentCommitsSharingObjectsDoNotDeadlock582=== RUN TestResolveDBConnectionString583=== PAUSE TestResolveDBConnectionString584=== RUN TestConnectWaitsForAPeerMigration585=== PAUSE TestConnectWaitsForAPeerMigration586=== RUN TestLeadElectsOneAndHandsOver587=== PAUSE TestLeadElectsOneAndHandsOver588=== RUN TestLeadIncumbentWinsAfterRestart5892026/11/12 00:14:21 INFO lead: acquired remote=192.0.2.1:1234 generation=15902026/11/12 00:14:22 INFO lead: released remote=192.0.2.1:1234 generation=15912026/11/12 00:14:22 INFO lead: acquired remote=192.0.2.1:1234 generation=25922026/11/12 00:14:23 INFO lead: released remote=192.0.2.1:1234 generation=2593--- PASS: TestLeadIncumbentWinsAfterRestart (4.91s)594=== RUN TestLeadUnannouncedStandbyYieldsToIncumbent595=== PAUSE TestLeadUnannouncedStandbyYieldsToIncumbent596=== RUN TestLeadIncumbentAnnouncesAtOnce597=== PAUSE TestLeadIncumbentAnnouncesAtOnce598=== RUN TestLeadGenerationGrowsWithEveryGrant599=== PAUSE TestLeadGenerationGrowsWithEveryGrant600=== RUN TestLeadEndsOnShutdown601=== PAUSE TestLeadEndsOnShutdown602=== RUN TestLeadEndsWhenItsConnectionHangs6032026/11/12 00:14:26 INFO lead: acquired remote=192.0.2.1:1234 generation=16042026-11-12 00:14:28.216 UTC [420] FATAL: terminating connection due to administrator command6052026/11/12 00:14:28 INFO lead: acquired remote=192.0.2.1:1234 generation=26062026/11/12 00:14:29 WARN lead: lock connection lost error="timeout: context deadline exceeded"6072026/11/12 00:14:29 INFO lead: released remote=192.0.2.1:1234 generation=16082026/11/12 00:14:29 INFO lead: released remote=192.0.2.1:1234 generation=2609--- PASS: TestLeadEndsWhenItsConnectionHangs (6.44s)610=== RUN TestGCAdvisoryLockBlocksConcurrentRun611=== PAUSE TestGCAdvisoryLockBlocksConcurrentRun612=== RUN TestGCBugBareHashReferences613=== PAUSE TestGCBugBareHashReferences614=== RUN TestGCMetrics615=== PAUSE TestGCMetrics616=== RUN TestPushDedupSurvivesConcurrentGC617=== PAUSE TestPushDedupSurvivesConcurrentGC618=== RUN TestDeduplicatedObjectsRecordedAsPending619=== PAUSE TestDeduplicatedObjectsRecordedAsPending620=== RUN TestGCSweepSkipsPendingObjects621=== PAUSE TestGCSweepSkipsPendingObjects622=== RUN TestTombstonedObjectOfferedWithoutWaiting623=== PAUSE TestTombstonedObjectOfferedWithoutWaiting624=== RUN TestGCSweepDeliversEachKeyOnce625=== PAUSE TestGCSweepDeliversEachKeyOnce626=== RUN TestCreatePendingClosureVerifyS3FailureReleasesConnection627=== PAUSE TestCreatePendingClosureVerifyS3FailureReleasesConnection628=== RUN TestForceGCDuringPushOffersSweptObject629=== PAUSE TestForceGCDuringPushOffersSweptObject630=== RUN TestFailedS3DeleteKeepsTombstone631=== PAUSE TestFailedS3DeleteKeepsTombstone632=== RUN TestSweepSkipsRowsLockedByPush633=== PAUSE TestSweepSkipsRowsLockedByPush634=== RUN TestRegisterCompletedObjectNeedsPendingRow635=== PAUSE TestRegisterCompletedObjectNeedsPendingRow636=== RUN TestGetExistingObjectsLocksRows637=== PAUSE TestGetExistingObjectsLocksRows638=== RUN TestSweepRechecksPendingAfterLocking639=== PAUSE TestSweepRechecksPendingAfterLocking640=== RUN TestPushWaitsForSweepHoldingRow641=== PAUSE TestPushWaitsForSweepHoldingRow642=== RUN TestCommitRacingPendingCleanupKeepsObjects643=== PAUSE TestCommitRacingPendingCleanupKeepsObjects644=== RUN TestGCEndsOnShutdown645=== PAUSE TestGCEndsOnShutdown646=== RUN TestGCTaskStore_StartNew647=== PAUSE TestGCTaskStore_StartNew648=== RUN TestGCTaskStore_DeduplicateSameParams649=== PAUSE TestGCTaskStore_DeduplicateSameParams650=== RUN TestGCTaskStore_ConflictDifferentParams651=== PAUSE TestGCTaskStore_ConflictDifferentParams652=== RUN TestGCTaskStore_GetEmpty653=== PAUSE TestGCTaskStore_GetEmpty654=== RUN TestGCTaskStore_GetReturnsLatest655=== PAUSE TestGCTaskStore_GetReturnsLatest656=== RUN TestGCTaskStore_CompletedAllowsNewTask657=== PAUSE TestGCTaskStore_CompletedAllowsNewTask658=== RUN TestGCTaskStore_PhaseUpdates659=== PAUSE TestGCTaskStore_PhaseUpdates660=== RUN TestGCTaskStore_Fail661=== PAUSE TestGCTaskStore_Fail662=== RUN TestGracefulShutdownDrainsInflight663=== PAUSE TestGracefulShutdownDrainsInflight664=== RUN TestService_healthCheckHandler665=== PAUSE TestService_healthCheckHandler666=== RUN TestService_readinessHandler667=== PAUSE TestService_readinessHandler668=== RUN TestGenerateLandingPage669=== PAUSE TestGenerateLandingPage670=== RUN TestEnsureAbortUploadsRule671=== PAUSE TestEnsureAbortUploadsRule672=== RUN TestCacheConfigHandlerMaxNarSize673=== PAUSE TestCacheConfigHandlerMaxNarSize674=== RUN TestCreatePendingClosureRejectsOversizedNAR675=== PAUSE TestCreatePendingClosureRejectsOversizedNAR676=== RUN TestNARDeduplicationMetadataUploadBug677=== PAUSE TestNARDeduplicationMetadataUploadBug678=== RUN TestMetricsInventory679=== PAUSE TestMetricsInventory680=== RUN TestMetricsCommit681=== PAUSE TestMetricsCommit682=== RUN TestService_NativeMTLS683=== PAUSE TestService_NativeMTLS684=== RUN TestServerTLSConfig685=== PAUSE TestServerTLSConfig686=== RUN TestMultipartCleanup687=== PAUSE TestMultipartCleanup688=== RUN TestMultipartUploadAbortedWhenCancelledBeforeRecorded689=== PAUSE TestMultipartUploadAbortedWhenCancelledBeforeRecorded690=== RUN TestPendingCleanupUsesOneCutoff691=== PAUSE TestPendingCleanupUsesOneCutoff692=== RUN TestObjectStatsTrigger693=== PAUSE TestObjectStatsTrigger694=== RUN TestObjectStatsUpdatedOncePerCommit695=== PAUSE TestObjectStatsUpdatedOncePerCommit696=== RUN TestOrphanedObjectsGC697=== PAUSE TestOrphanedObjectsGC698=== RUN TestOrphanedObjectsGCStressTest699=== PAUSE TestOrphanedObjectsGCStressTest700=== RUN TestCreatePin_ReservedPins701=== PAUSE TestCreatePin_ReservedPins702=== RUN TestPresentReportsOnlyClosureRoots703=== PAUSE TestPresentReportsOnlyClosureRoots704=== RUN TestPresentRefreshesOnlyStaleClosures705=== PAUSE TestPresentRefreshesOnlyStaleClosures706=== RUN TestPresentNotReportedWhileGCDeletesClosure707=== PAUSE TestPresentNotReportedWhileGCDeletesClosure708=== RUN TestParseSingleRange709=== PAUSE TestParseSingleRange710=== RUN TestProxyHeadersOnlyTrustedOnSocket7112026/11/12 00:14:32 INFO Starting HTTP server address=127.0.0.1:423237122026/11/12 00:14:32 INFO Starting HTTP server address=proxy.sock7132026/11/12 00:14:32 WARN mTLS auth: subject not in bound subjects subject="CN=someone"7142026/11/12 00:14:32 INFO Shutdown signal received, draining in-flight requests timeout=10s715--- PASS: TestProxyHeadersOnlyTrustedOnSocket (2.68s)716=== RUN TestIsValidCachePath717=== PAUSE TestIsValidCachePath718=== RUN TestReadProxyNarinfo719=== PAUSE TestReadProxyNarinfo720=== RUN TestReadProxyNarinfoAlreadyDecompressed721=== PAUSE TestReadProxyNarinfoAlreadyDecompressed722=== RUN TestReadProxyContentNegotiation723=== PAUSE TestReadProxyContentNegotiation724=== RUN TestReadProxyNarStreaming725=== PAUSE TestReadProxyNarStreaming726=== RUN TestReadProxy404727=== PAUSE TestReadProxy404728=== RUN TestReadProxyInvalidPath729=== PAUSE TestReadProxyInvalidPath730=== RUN TestReadProxyHead731=== PAUSE TestReadProxyHead732=== RUN TestReadProxyOutlastsServerWriteTimeout733=== PAUSE TestReadProxyOutlastsServerWriteTimeout734=== RUN TestReadProxyConditionalGet735=== PAUSE TestReadProxyConditionalGet736=== RUN TestReadProxyRootRedirectsToIndexHTML737=== PAUSE TestReadProxyRootRedirectsToIndexHTML738=== RUN TestReadProxyDisabled739=== PAUSE TestReadProxyDisabled740=== RUN TestReadRedirectNar741=== PAUSE TestReadRedirectNar742=== RUN TestReadRedirectKeepsNarinfoProxied743=== PAUSE TestReadRedirectKeepsNarinfoProxied744=== RUN TestReadProxyRangeRequest745=== PAUSE TestReadProxyRangeRequest746=== RUN TestReadRedirectUsesPublicS3URL747=== PAUSE TestReadRedirectUsesPublicS3URL748=== RUN TestPush_OverlappingRootsStoreOneRowPerKey749=== PAUSE TestPush_OverlappingRootsStoreOneRowPerKey750=== RUN TestPush_CompleteCommitsEveryRoot751=== PAUSE TestPush_CompleteCommitsEveryRoot752=== RUN TestPush_RepeatedRootCommitsOnce753=== PAUSE TestPush_RepeatedRootCommitsOnce754=== RUN TestPush_CompleteWaitsForCleanupOfThePush755=== PAUSE TestPush_CompleteWaitsForCleanupOfThePush756=== RUN TestPush_RootsAreLockedInByteOrder757=== PAUSE TestPush_RootsAreLockedInByteOrder758=== RUN TestPush_SkippedKeySurvivesGCBeforeCommit759=== PAUSE TestPush_SkippedKeySurvivesGCBeforeCommit760=== RUN TestPush_RejectsBadRequests761=== PAUSE TestPush_RejectsBadRequests762=== RUN TestPush_SignsNarinfosOfItsPendingObjects763=== PAUSE TestPush_SignsNarinfosOfItsPendingObjects764=== RUN TestSharedNarBothUploadsComplete765=== PAUSE TestSharedNarBothUploadsComplete766=== RUN TestCompleteMultipartUpload_ErrorButObjectExists767=== PAUSE TestCompleteMultipartUpload_ErrorButObjectExists768=== RUN TestCompletedNarNotReofferedAcrossClosures769=== PAUSE TestCompletedNarNotReofferedAcrossClosures770=== RUN TestPresignedUploadRegisteredBeforeCommit771=== PAUSE TestPresignedUploadRegisteredBeforeCommit772=== RUN TestService_Rustfstest773=== PAUSE TestService_Rustfstest774=== RUN TestParseSize775=== PAUSE TestParseSize776=== RUN TestSkippedUploadsHandler777=== PAUSE TestSkippedUploadsHandler778=== RUN TestSystemdListenerNotActivated779--- PASS: TestSystemdListenerNotActivated (0.00s)780=== RUN TestWatchdogBeatsWhenHealthy781--- PASS: TestWatchdogBeatsWhenHealthy (0.02s)782=== RUN TestWatchdogSkipsWhenUnhealthy7832026/11/12 00:14:32 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7842026/11/12 00:14:32 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7852026/11/12 00:14:32 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7862026/11/12 00:14:32 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7872026/11/12 00:14:32 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7882026/11/12 00:14:32 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7892026/11/12 00:14:32 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7902026/11/12 00:14:32 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7912026/11/12 00:14:32 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7922026/11/12 00:14:32 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"793--- PASS: TestWatchdogSkipsWhenUnhealthy (0.20s)794=== RUN TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle795=== PAUSE TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle796=== RUN TestProxyWriteTimeout797=== PAUSE TestProxyWriteTimeout798=== RUN TestPendingClosureWriteTimeout799=== PAUSE TestPendingClosureWriteTimeout800=== RUN TestIsValidUploadKey801=== PAUSE TestIsValidUploadKey802=== RUN TestUploadHandlersRejectInvalidKeys803=== PAUSE TestUploadHandlersRejectInvalidKeys804=== RUN TestUploadHandlersRejectOversizedBody805=== PAUSE TestUploadHandlersRejectOversizedBody806=== RUN TestService_cleanupPendingClosuresHandler807=== PAUSE TestService_cleanupPendingClosuresHandler808=== RUN TestService_createPendingClosureHandler809=== PAUSE TestService_createPendingClosureHandler810=== RUN TestService_verifyS3Integrity811=== PAUSE TestService_verifyS3Integrity812=== RUN TestService_verifyS3ReuploadsStaleNarinfo813=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo814=== RUN TestCompleteMultipartUnregistered815=== PAUSE TestCompleteMultipartUnregistered816=== RUN TestCreatePendingClosure_SmallNARUsesSimplePUT817=== PAUSE TestCreatePendingClosure_SmallNARUsesSimplePUT818=== CONT TestSkippedUploadsHandler819=== CONT TestGCTaskStore_DeduplicateSameParams820=== CONT TestPresentNotReportedWhileGCDeletesClosure821=== CONT TestAbortSucceeded822=== CONT TestGCTaskStore_Fail823=== CONT TestPushWaitsForSweepHoldingRow824=== CONT TestReadProxyNarinfo825=== CONT TestParseSize826=== CONT TestPush_RootsAreLockedInByteOrder8272026/11/12 00:14:32 INFO Client skipped oversized paths paths=3 nar_bytes=5000000000828=== CONT TestService_Rustfstest829=== CONT TestGCTaskStore_PhaseUpdates830=== CONT TestService_cleanupPendingClosuresHandler831=== CONT TestGCTaskStore_CompletedAllowsNewTask832=== CONT TestSweepRechecksPendingAfterLocking833=== CONT TestGCTaskStore_GetReturnsLatest834=== CONT TestCreatePendingClosure_SmallNARUsesSimplePUT835=== CONT TestGCTaskStore_GetEmpty836=== CONT TestGetExistingObjectsLocksRows837=== CONT TestGCTaskStore_ConflictDifferentParams838=== CONT TestCompleteMultipartUnregistered839=== CONT TestPresignedUploadRegisteredBeforeCommit840=== CONT TestPush_RepeatedRootCommitsOnce841=== CONT TestIsValidCachePath842=== RUN TestIsValidCachePath/narinfo843=== PAUSE TestIsValidCachePath/narinfo844=== RUN TestIsValidCachePath/narinfo_all_nix_base32_chars845=== PAUSE TestIsValidCachePath/narinfo_all_nix_base32_chars846=== RUN TestIsValidCachePath/nar_zst847=== PAUSE TestIsValidCachePath/nar_zst848=== RUN TestIsValidCachePath/nar_xz849=== PAUSE TestIsValidCachePath/nar_xz850=== RUN TestIsValidCachePath/nar_bz2851=== PAUSE TestIsValidCachePath/nar_bz2852=== RUN TestIsValidCachePath/nar_uncompressed853=== PAUSE TestIsValidCachePath/nar_uncompressed854=== RUN TestIsValidCachePath/ls855=== PAUSE TestIsValidCachePath/ls856=== RUN TestIsValidCachePath/log857=== PAUSE TestIsValidCachePath/log858=== RUN TestIsValidCachePath/realisation859=== PAUSE TestIsValidCachePath/realisation860=== RUN TestIsValidCachePath/nix-cache-info861=== PAUSE TestIsValidCachePath/nix-cache-info862=== RUN TestIsValidCachePath/index.html863=== PAUSE TestIsValidCachePath/index.html864=== RUN TestIsValidCachePath/traversal_parent865=== PAUSE TestIsValidCachePath/traversal_parent866=== RUN TestIsValidCachePath/traversal_in_middle867=== PAUSE TestIsValidCachePath/traversal_in_middle868=== RUN TestIsValidCachePath/invalid_char_e869=== PAUSE TestIsValidCachePath/invalid_char_e870=== RUN TestIsValidCachePath/invalid_char_u871=== PAUSE TestIsValidCachePath/invalid_char_u872=== RUN TestIsValidCachePath/random_path873=== PAUSE TestIsValidCachePath/random_path874=== RUN TestIsValidCachePath/empty875=== PAUSE TestIsValidCachePath/empty876=== RUN TestIsValidCachePath/leading_slash877=== PAUSE TestIsValidCachePath/leading_slash878=== RUN TestIsValidCachePath/wrong_extension879=== PAUSE TestIsValidCachePath/wrong_extension880=== RUN TestIsValidCachePath/short_hash881=== PAUSE TestIsValidCachePath/short_hash882=== CONT TestParseSingleRange883=== RUN TestParseSingleRange/none884=== PAUSE TestParseSingleRange/none885=== RUN TestParseSingleRange/unknown_unit886=== PAUSE TestParseSingleRange/unknown_unit887=== RUN TestParseSingleRange/multi-range_ignored888=== PAUSE TestParseSingleRange/multi-range_ignored889=== RUN TestParseSingleRange/malformed_no_dash890=== PAUSE TestParseSingleRange/malformed_no_dash891=== RUN TestParseSingleRange/malformed_both_empty892=== PAUSE TestParseSingleRange/malformed_both_empty893=== RUN TestParseSingleRange/malformed_end_before_start894=== PAUSE TestParseSingleRange/malformed_end_before_start895=== RUN TestParseSingleRange/closed896=== PAUSE TestParseSingleRange/closed897=== RUN TestParseSingleRange/open-ended898=== PAUSE TestParseSingleRange/open-ended899=== RUN TestParseSingleRange/end_clamped_to_size900=== PAUSE TestParseSingleRange/end_clamped_to_size901=== RUN TestParseSingleRange/suffix902=== PAUSE TestParseSingleRange/suffix903=== RUN TestParseSingleRange/suffix_exceeds_size904=== PAUSE TestParseSingleRange/suffix_exceeds_size905=== RUN TestParseSingleRange/single_byte906=== PAUSE TestParseSingleRange/single_byte907=== RUN TestParseSingleRange/start_past_EOF908=== PAUSE TestParseSingleRange/start_past_EOF909=== RUN TestParseSingleRange/start_far_past_EOF910=== PAUSE TestParseSingleRange/start_far_past_EOF911=== CONT TestRegisterCompletedObjectNeedsPendingRow912--- PASS: TestGCTaskStore_DeduplicateSameParams (0.00s)913--- PASS: TestGCTaskStore_Fail (0.00s)914--- PASS: TestAbortSucceeded (0.00s)915=== CONT TestService_verifyS3ReuploadsStaleNarinfo916=== RUN TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed917=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed918=== RUN TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed919=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed920=== RUN TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost921=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost922=== RUN TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed923=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed924=== RUN TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed925=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed926=== CONT TestCompletedNarNotReofferedAcrossClosures927--- PASS: TestParseSize (0.00s)928--- PASS: TestGCTaskStore_PhaseUpdates (0.00s)929--- PASS: TestGCTaskStore_CompletedAllowsNewTask (0.00s)930--- PASS: TestGCTaskStore_GetReturnsLatest (0.00s)931--- PASS: TestGCTaskStore_GetEmpty (0.00s)932--- PASS: TestGCTaskStore_ConflictDifferentParams (0.00s)933--- PASS: TestSkippedUploadsHandler (0.01s)934=== CONT TestPush_CompleteCommitsEveryRoot935=== CONT TestCompleteMultipartUpload_ErrorButObjectExists936=== CONT TestGCTaskStore_StartNew937=== CONT TestSharedNarBothUploadsComplete938=== CONT TestGCEndsOnShutdown939=== RUN TestGCEndsOnShutdown/before_the_run940--- PASS: TestGCTaskStore_StartNew (0.00s)941=== PAUSE TestGCEndsOnShutdown/before_the_run942=== RUN TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete943=== PAUSE TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete944=== RUN TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark945=== PAUSE TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark946=== CONT TestService_verifyS3Integrity947=== CONT TestSweepSkipsRowsLockedByPush948=== CONT TestCommitRacingPendingCleanupKeepsObjects949=== CONT TestPush_RejectsBadRequests950=== CONT TestPush_SkippedKeySurvivesGCBeforeCommit951=== RUN TestPresentNotReportedWhileGCDeletesClosure/stale952=== CONT TestPush_CompleteWaitsForCleanupOfThePush953=== CONT TestPush_SignsNarinfosOfItsPendingObjects954=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/stale955=== RUN TestPresentNotReportedWhileGCDeletesClosure/fresh956=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/fresh957=== CONT TestPush_OverlappingRootsStoreOneRowPerKey9582026/11/12 00:14:40 INFO Received uploads request method=POST path=/api/pending_closures9592026/11/12 00:14:40 INFO Received push request method=POST path=/api/pushes9602026/11/12 00:14:40 INFO Received uploads request method=POST path=/api/pending_closures9612026/11/12 00:14:40 INFO Registered completed upload object_key=nar/jjjjjjjjjjjjjjjjjjjjjjjjjjjjjj0100000000000000000000.nar.zst9622026/11/12 00:14:40 INFO Received uploads request method=POST path=/api/pending_closures9632026/11/12 00:14:40 INFO Aborted multipart uploads count=0 kept=09642026/11/12 00:14:40 INFO Received complete multipart upload request method=POST path=/api/multipart/complete9652026/11/12 00:14:40 ERROR Failed to get multipart upload error="no rows in result set" upload_id=does-not-exist object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst966--- PASS: TestPresignedUploadRegisteredBeforeCommit (7.83s)967=== CONT TestService_createPendingClosureHandler968--- PASS: TestCompleteMultipartUnregistered (7.83s)969=== CONT TestFailedS3DeleteKeepsTombstone970--- PASS: TestPush_OverlappingRootsStoreOneRowPerKey (4.97s)971=== CONT TestReadRedirectUsesPublicS3URL9722026/11/12 00:14:40 INFO Received push request method=POST path=/api/pushes9732026/11/12 00:14:40 INFO Received complete multipart upload request method=POST path=/api/multipart/complete9742026/11/12 00:14:40 WARN CompleteMultipartUpload errored but object exists; treating as success error="The specified key does not exist." object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=Y2IyZGFmYTktOThkZS00Yzg4LTkwODQtNmM1YmU2YTZhMGY2LmJjNTkwMTdkLTEyOTYtNGM5Yy1hOTc5LTg3MDNjYWY1YjY1N3gxNzk0NDQyNDgwNTM4Nzg4NTc49752026/11/12 00:14:40 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=Y2IyZGFmYTktOThkZS00Yzg4LTkwODQtNmM1YmU2YTZhMGY2LmJjNTkwMTdkLTEyOTYtNGM5Yy1hOTc5LTg3MDNjYWY1YjY1N3gxNzk0NDQyNDgwNTM4Nzg4NTc4 parts=1976--- PASS: TestCompleteMultipartUpload_ErrorButObjectExists (5.05s)977=== CONT TestForceGCDuringPushOffersSweptObject978=== RUN TestForceGCDuringPushOffersSweptObject/before_pending_rows979=== PAUSE TestForceGCDuringPushOffersSweptObject/before_pending_rows980=== RUN TestForceGCDuringPushOffersSweptObject/after_presence_check981=== PAUSE TestForceGCDuringPushOffersSweptObject/after_presence_check982=== CONT TestReadProxyRangeRequest983--- PASS: TestGetExistingObjectsLocksRows (7.90s)984=== CONT TestDeleteClosuresRacingPins9852026/11/12 00:14:40 INFO Received cleanup request method=DELETE path=/api/pending_closures9862026/11/12 00:14:40 INFO Aborted multipart uploads count=0 kept=09872026/11/12 00:14:40 INFO Received uploads request method=POST path=/api/pending_closures9882026/11/12 00:14:40 INFO Received cleanup request method=DELETE path=/api/pending_closures9892026/11/12 00:14:40 INFO Aborted multipart uploads count=1 kept=09902026/11/12 00:14:40 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete9912026-11-12 00:14:40.662 UTC [481] ERROR: Closure does not exist: id=19922026-11-12 00:14:40.662 UTC [481] CONTEXT: PL/pgSQL function commit_pending_closure(bigint) line 13 at RAISE9932026-11-12 00:14:40.662 UTC [481] STATEMENT: -- name: CommitPendingClosure :one994 SELECT commit_pending_closure($1::bigint)::bigint995 996--- PASS: TestService_cleanupPendingClosuresHandler (7.94s)997=== CONT TestClientFallsBackToClosures9982026/11/12 00:14:40 INFO Received uploads request method=POST path=/api/pending_closures9992026/11/12 00:14:40 INFO Received push request method=POST path=/api/pushes10002026/11/12 00:14:40 INFO Received complete push request method=POST path=/api/pushes/1/complete10012026/11/12 00:14:40 INFO Received uploads request method=POST path=/api/pending_closures10022026/11/12 00:14:40 INFO Aborted multipart uploads count=0 kept=01003--- PASS: TestCreatePendingClosure_SmallNARUsesSimplePUT (8.04s)1004=== CONT TestReadRedirectKeepsNarinfoProxied10052026/11/12 00:14:40 INFO Received push request method=POST path=/api/pushes10062026/11/12 00:14:40 INFO Received complete push request method=POST path=/api/pushes/1/complete10072026/11/12 00:14:40 INFO Received uploads request method=POST path=/api/pending_closures10082026/11/12 00:14:40 INFO Received uploads request method=POST path=/api/pending_closures1009--- PASS: TestService_Rustfstest (8.21s)1010=== CONT TestCreatePendingClosureVerifyS3FailureReleasesConnection10112026/11/12 00:14:40 INFO Received uploads request method=POST path=/api/pending_closures10122026/11/12 00:14:41 INFO Received uploads request method=POST path=/api/pending_closures10132026/11/12 00:14:41 INFO Aborted multipart uploads count=0 kept=010142026/11/12 00:14:41 INFO Received push request method=POST path=/api/pushes10152026/11/12 00:14:41 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign10162026/11/12 00:14:41 INFO Signed narinfos id=1 count=110172026/11/12 00:14:41 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign10182026/11/12 00:14:41 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign10192026/11/12 00:14:41 ERROR Failed to sign narinfos signer="index 1" error="signing key cannot be nil"10202026/11/12 00:14:41 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign10212026/11/12 00:14:41 ERROR Failed to sign narinfos signer=niks3-test-1 error="signer failed"10222026/11/12 00:14:41 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign10232026/11/12 00:14:41 ERROR Signer returned unexpected signature count signer=niks3-test-1 got=0 want=110242026/11/12 00:14:41 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign10252026/11/12 00:14:41 ERROR Signer returned unexpected object key signer=niks3-test-1 object_key=unexpected.narinfo1026--- PASS: TestPush_SignsNarinfosOfItsPendingObjects (5.47s)1027=== CONT TestReadRedirectNar1028=== RUN TestPush_RejectsBadRequests/no_roots1029=== PAUSE TestPush_RejectsBadRequests/no_roots1030=== RUN TestPush_RejectsBadRequests/no_objects1031=== PAUSE TestPush_RejectsBadRequests/no_objects1032=== RUN TestPush_RejectsBadRequests/bad_root1033=== PAUSE TestPush_RejectsBadRequests/bad_root1034=== RUN TestPush_RejectsBadRequests/root_not_in_objects1035=== PAUSE TestPush_RejectsBadRequests/root_not_in_objects1036=== CONT TestGCSweepDeliversEachKeyOnce10372026/11/12 00:14:41 INFO Received push request method=POST path=/api/pushes10382026/11/12 00:14:41 INFO Received complete push request method=POST path=/api/pushes/1/complete1039--- PASS: TestRegisterCompletedObjectNeedsPendingRow (8.42s)1040=== CONT TestReadProxyDisabled1041--- PASS: TestPushWaitsForSweepHoldingRow (8.47s)1042=== CONT TestClientPushesUseOnePush10432026/11/12 00:14:41 ERROR Failed to decode narinfo key=3hcdxyjf9yiq7qf3i4548drb6sjmwa1v.narinfo limit=16777216 error=<nil>1044--- PASS: TestReadProxyNarinfo (8.51s)1045=== CONT TestTombstonedObjectOfferedWithoutWaiting10462026/11/12 00:14:41 INFO Received complete multipart upload request method=POST path=/api/multipart/complete10472026/11/12 00:14:41 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst upload_id=Y2IyZGFmYTktOThkZS00Yzg4LTkwODQtNmM1YmU2YTZhMGY2LjczYzAwNDZiLTM5YjktNGRhYS1iZTk0LTJkMDM5ZDg1NGFmZXgxNzk0NDQyNDgwOTEzNzkzNDc1 parts=1010482026/11/12 00:14:41 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete10492026/11/12 00:14:41 INFO Completed upload id=1 objects=2 duration=4.126137ms10502026/11/12 00:14:41 INFO Received uploads request method=POST path=/api/pending_closures10512026/11/12 00:14:41 INFO Received uploads request method=POST path=/api/pending_closures10522026/11/12 00:14:41 INFO Object in database but missing from S3 key=dadb44fdadb44fdadb44fdadb44f0000.narinfo10532026/11/12 00:14:41 WARN Found objects in DB but missing from S3, will re-upload count=11054--- PASS: TestService_verifyS3Integrity (5.83s)1055=== CONT TestReadProxyRootRedirectsToIndexHTML10562026/11/12 00:14:41 INFO Received complete multipart upload request method=POST path=/api/multipart/complete10572026/11/12 00:14:41 INFO Completed multipart upload object_key=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhh0100000000000000000000.nar.zst upload_id=Y2IyZGFmYTktOThkZS00Yzg4LTkwODQtNmM1YmU2YTZhMGY2LmE2NmE1ZDk2LWE3MDEtNGQ5MS1iODU5LTE1YWQ2ZjU1NDdhYngxNzk0NDQyNDgwODg2Njc4MTkz parts=1210582026/11/12 00:14:41 INFO Received uploads request method=POST path=/api/pending_closures1059--- PASS: TestCompletedNarNotReofferedAcrossClosures (8.57s)1060=== CONT TestPinProtectsFromGC10612026/11/12 00:14:41 INFO Received complete multipart upload request method=POST path=/api/multipart/complete10622026/11/12 00:14:41 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0100000000000000000000.nar.zst upload_id=Y2IyZGFmYTktOThkZS00Yzg4LTkwODQtNmM1YmU2YTZhMGY2LjZkYWZjYzIwLTI4ZjctNGMzOS04YTRiLWE2MWQ3NzVmMmIyN3gxNzk0NDQyNDgxMDAzMTQzNDU2 parts=1210632026/11/12 00:14:42 INFO Received complete multipart upload request method=POST path=/api/multipart/complete10642026/11/12 00:14:42 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0100000000000000000000.nar.zst upload_id=Y2IyZGFmYTktOThkZS00Yzg4LTkwODQtNmM1YmU2YTZhMGY2LjUwMmEyM2RjLTBmYmItNGYyZS1iYmRkLTcwYzM1MmM0Mzg0NXgxNzk0NDQyNDgxMDEwOTY3MDY2 parts=121065--- PASS: TestSharedNarBothUploadsComplete (6.50s)1066=== CONT TestGCSweepSkipsPendingObjects10672026/11/12 00:14:43 INFO Completed push id=1 objects=6 duration=2.3117161s1068--- PASS: TestPush_CompleteCommitsEveryRoot (10.20s)1069=== CONT TestReadProxyConditionalGet10702026/11/12 00:14:43 WARN Force mode enabled - objects will be deleted immediately without grace period10712026/11/12 00:14:43 WARN Force mode enabled - objects will be deleted immediately without grace period10722026-11-12 00:14:43.143 UTC [533] ERROR: could not obtain lock on row in relation "closures"10732026-11-12 00:14:43.143 UTC [533] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT10742026/11/12 00:14:43 WARN Force mode enabled - objects will be deleted immediately without grace period10752026/11/12 00:14:43 INFO Completed push id=1 objects=4 duration=2.420229472s10762026/11/12 00:14:43 INFO Completed push id=1 objects=4 duration=1.98674329s10772026-11-12 00:14:43.145 UTC [533] ERROR: could not obtain lock on row in relation "closures"10782026-11-12 00:14:43.145 UTC [533] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT10792026/11/12 00:14:43 INFO Received push request method=POST path=/api/pushes10802026-11-12 00:14:43.146 UTC [533] ERROR: could not obtain lock on row in relation "closures"10812026-11-12 00:14:43.146 UTC [533] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT1082--- PASS: TestPush_RepeatedRootCommitsOnce (10.42s)1083=== CONT TestClientSharedPathCommittedMidPush10842026-11-12 00:14:43.146 UTC [533] ERROR: could not obtain lock on row in relation "closures"10852026-11-12 00:14:43.146 UTC [533] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT10862026-11-12 00:14:43.147 UTC [533] ERROR: could not obtain lock on row in relation "closures"10872026-11-12 00:14:43.147 UTC [533] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT10882026/11/12 00:14:43 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=010892026/11/12 00:14:43 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=01090--- PASS: TestPush_RootsAreLockedInByteOrder (10.44s)1091=== CONT TestDeduplicatedObjectsRecordedAsPending10922026/11/12 00:14:43 INFO Vacuumed table table=pending_closures10932026/11/12 00:14:43 INFO Vacuumed table table=pending_objects10942026/11/12 00:14:43 INFO Vacuumed table table=multipart_uploads10952026/11/12 00:14:43 INFO Vacuumed table table=closures10962026/11/12 00:14:43 INFO Vacuumed table table=pending_closures10972026/11/12 00:14:43 INFO Vacuumed table table=objects10982026/11/12 00:14:43 INFO Vacuumed table table=pending_objects10992026/11/12 00:14:43 INFO Vacuumed table table=multipart_uploads11002026/11/12 00:14:43 INFO Vacuumed table table=closures11012026/11/12 00:14:43 INFO Vacuumed table table=objects11022026/11/12 00:14:43 INFO Aborted multipart uploads count=0 kept=011032026/11/12 00:14:43 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=011042026/11/12 00:14:43 INFO Vacuumed table table=pending_closures11052026/11/12 00:14:43 INFO Vacuumed table table=pending_objects11062026/11/12 00:14:43 INFO Vacuumed table table=multipart_uploads11072026/11/12 00:14:43 INFO Vacuumed table table=closures11082026/11/12 00:14:43 INFO Vacuumed table table=objects1109--- PASS: TestSweepRechecksPendingAfterLocking (10.48s)1110=== CONT TestReadProxyOutlastsServerWriteTimeout11112026/11/12 00:14:43 INFO Aborted multipart uploads count=0 kept=011122026/11/12 00:14:43 WARN Force mode enabled - objects will be deleted immediately without grace period11132026/11/12 00:14:43 INFO Received uploads request method=POST path=/api/pending_closures1114--- PASS: TestTombstonedObjectOfferedWithoutWaiting (2.04s)1115=== CONT TestClientWithDependencies1116--- PASS: TestReadProxyDisabled (2.12s)1117=== CONT TestReadProxyHead11182026/11/12 00:14:43 INFO Received uploads request method=POST path=/api/pending_closures11192026/11/12 00:14:43 INFO Aborted multipart uploads count=0 kept=01120--- PASS: TestReadRedirectNar (2.30s)1121=== CONT TestService_RequireScope_OIDC1122--- PASS: TestReadProxyRootRedirectsToIndexHTML (1.96s)1123=== CONT TestReadProxyInvalidPath11242026/11/12 00:14:43 INFO Received uploads request method=POST path=/api/pending_closures11252026/11/12 00:14:43 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:43273/oidc1126--- PASS: TestReadRedirectKeepsNarinfoProxied (2.68s)1127=== CONT TestPushDedupSurvivesConcurrentGC11282026/11/12 00:14:43 INFO Aborted multipart uploads count=0 kept=011292026/11/12 00:14:43 INFO Received uploads request method=POST path=/api/pending_closures11302026/11/12 00:14:43 INFO Received uploads request method=POST path=/api/pending_closures11312026/11/12 00:14:43 INFO Received uploads request method=POST path=/api/pending_closures1132--- PASS: TestCreatePendingClosureVerifyS3FailureReleasesConnection (2.67s)1133=== CONT TestService_AuthMiddleware_OIDC1134--- PASS: TestReadProxyRangeRequest (3.02s)1135=== CONT TestReadProxy40411362026/11/12 00:14:43 INFO Received push request method=POST path=/api/pushes11372026/11/12 00:14:43 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:33105/oidc1138--- PASS: TestReadRedirectUsesPublicS3URL (3.13s)1139=== CONT TestGCMetrics11402026/11/12 00:14:43 INFO Received uploads request method=POST path=/api/pending_closures11412026/11/12 00:14:43 INFO Aborted multipart uploads count=0 kept=011422026-11-12 00:14:44.159 UTC [650] ERROR: Push does not exist: id=111432026-11-12 00:14:44.159 UTC [650] CONTEXT: PL/pgSQL function commit_push(bigint) line 11 at RAISE11442026-11-12 00:14:44.159 UTC [650] STATEMENT: SELECT commit_push($1)1145--- PASS: TestPush_CompleteWaitsForCleanupOfThePush (8.58s)1146=== CONT TestReadProxyNarStreaming11472026/11/12 00:14:44 INFO Received complete multipart upload request method=POST path=/api/multipart/complete11482026/11/12 00:14:44 INFO Completed multipart upload object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst upload_id=Y2IyZGFmYTktOThkZS00Yzg4LTkwODQtNmM1YmU2YTZhMGY2LjI4ZGVhNTcyLWY0MzktNDI0NC04MzhmLWUzMjZjNmI0OTE1ZHgxNzk0NDQyNDgzNTg1MTU5MzU2 parts=1011492026/11/12 00:14:44 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete11502026/11/12 00:14:44 INFO Completed upload id=1 objects=2 duration=2.961806ms11512026/11/12 00:14:44 INFO Received get closure request method=GET path=/api/closures/0000000000000000000000000000000011522026/11/12 00:14:44 INFO Received uploads request method=POST path=/api/pending_closures11532026/11/12 00:14:44 INFO Starting cleanup of old closures method=DELETE path=/api/closures11542026/11/12 00:14:44 INFO Aborted multipart uploads count=0 kept=011552026/11/12 00:14:47 WARN Force mode enabled - objects will be deleted immediately without grace period11562026/11/12 00:14:47 WARN Force mode enabled - objects will be deleted immediately without grace period11572026/11/12 00:14:47 WARN Force mode enabled - objects will be deleted immediately without grace period11582026/11/12 00:14:47 WARN Force mode enabled - objects will be deleted immediately without grace period11592026/11/12 00:14:47 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=011602026/11/12 00:14:47 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=011612026/11/12 00:14:47 INFO Vacuumed table table=pending_closures11622026/11/12 00:14:47 INFO Vacuumed table table=pending_closures11632026/11/12 00:14:47 INFO Vacuumed table table=pending_objects11642026/11/12 00:14:47 INFO Vacuumed table table=pending_objects11652026/11/12 00:14:47 INFO Vacuumed table table=multipart_uploads11662026/11/12 00:14:47 INFO Vacuumed table table=closures11672026/11/12 00:14:47 INFO Vacuumed table table=objects11682026/11/12 00:14:47 INFO Vacuumed table table=multipart_uploads11692026/11/12 00:14:47 INFO Vacuumed table table=closures11702026/11/12 00:14:47 INFO Vacuumed table table=objects11712026/11/12 00:14:47 INFO Received get closure request method=GET path=/api/closures/000000000000000000000000000000001172--- PASS: TestService_createPendingClosureHandler (6.95s)1173=== CONT TestService_ReadAuthMiddleware11742026/11/12 00:14:47 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."11752026/11/12 00:14:47 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."1176--- PASS: TestCommitRacingPendingCleanupKeepsObjects (11.94s)1177=== CONT TestGCBugBareHashReferences1178--- PASS: TestFailedS3DeleteKeepsTombstone (6.96s)1179=== CONT TestReadProxyContentNegotiation1180--- PASS: TestGCSweepSkipsPendingObjects (5.44s)1181=== CONT TestService_AuthMiddleware_MTLSBoundSubjects11822026/11/12 00:14:47 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=2 objects-deleted-after-grace-period=2 objects-failed-to-delete=011832026/11/12 00:14:47 INFO Vacuumed table table=pending_closures11842026/11/12 00:14:47 INFO Vacuumed table table=pending_objects11852026/11/12 00:14:47 INFO Vacuumed table table=multipart_uploads11862026/11/12 00:14:47 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=011872026/11/12 00:14:47 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=011882026/11/12 00:14:47 INFO Vacuumed table table=pending_closures11892026/11/12 00:14:47 INFO Vacuumed table table=pending_objects11902026/11/12 00:14:47 INFO Aborted multipart uploads count=0 kept=011912026/11/12 00:14:47 INFO Vacuumed table table=closures11922026/11/12 00:14:47 INFO Vacuumed table table=multipart_uploads11932026/11/12 00:14:47 INFO Vacuumed table table=pending_closures11942026/11/12 00:14:47 INFO Vacuumed table table=closures11952026/11/12 00:14:47 INFO Vacuumed table table=pending_objects11962026/11/12 00:14:47 INFO Vacuumed table table=multipart_uploads11972026/11/12 00:14:47 INFO Vacuumed table table=closures11982026/11/12 00:14:47 INFO Vacuumed table table=objects11992026/11/12 00:14:47 INFO Received complete push request method=POST path=/api/pushes/2/complete12002026/11/12 00:14:47 INFO Vacuumed table table=objects12012026/11/12 00:14:47 INFO Vacuumed table table=objects1202--- PASS: TestGCSweepDeliversEachKeyOnce (6.52s)1203=== CONT TestGCAdvisoryLockBlocksConcurrentRun1204=== CONT TestReadProxyNarinfoAlreadyDecompressed1205--- PASS: TestSweepSkipsRowsLockedByPush (12.00s)12062026/11/12 00:14:47 INFO Completed push id=2 objects=4 duration=6.828481ms1207--- PASS: TestPush_SkippedKeySurvivesGCBeforeCommit (12.00s)1208=== CONT TestService_AuthMiddleware_MTLSProxyHeader1209--- PASS: TestReadProxyInvalidPath (4.22s)1210=== CONT TestService_AuthMiddleware1211--- PASS: TestReadProxyNarStreaming (3.49s)1212=== CONT TestLeadElectsOneAndHandsOver1213=== RUN TestService_RequireScope_OIDC/builder_may_write1214=== PAUSE TestService_RequireScope_OIDC/builder_may_write1215=== RUN TestService_RequireScope_OIDC/builder_may_not_admin1216=== PAUSE TestService_RequireScope_OIDC/builder_may_not_admin1217=== RUN TestService_RequireScope_OIDC/ops_may_admin1218=== PAUSE TestService_RequireScope_OIDC/ops_may_admin1219=== RUN TestService_RequireScope_OIDC/ops_may_not_write1220=== PAUSE TestService_RequireScope_OIDC/ops_may_not_write1221=== RUN TestService_RequireScope_OIDC/reader_may_not_write1222=== PAUSE TestService_RequireScope_OIDC/reader_may_not_write1223=== RUN TestService_RequireScope_OIDC/static_token_may_admin1224=== PAUSE TestService_RequireScope_OIDC/static_token_may_admin1225=== RUN TestService_RequireScope_OIDC/static_token_may_write1226=== PAUSE TestService_RequireScope_OIDC/static_token_may_write1227=== RUN TestService_RequireScope_OIDC/reader_may_read1228=== PAUSE TestService_RequireScope_OIDC/reader_may_read1229=== RUN TestService_RequireScope_OIDC/writer_implies_read1230=== PAUSE TestService_RequireScope_OIDC/writer_implies_read1231=== RUN TestService_RequireScope_OIDC/anonymous_may_not_read1232=== PAUSE TestService_RequireScope_OIDC/anonymous_may_not_read1233=== RUN TestService_RequireScope_OIDC/reader_may_read_with_netrc1234=== PAUSE TestService_RequireScope_OIDC/reader_may_read_with_netrc1235=== RUN TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1236=== PAUSE TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1237=== RUN TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read1238=== PAUSE TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read1239=== RUN TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1240=== PAUSE TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1241=== CONT TestLeadEndsOnShutdown1242--- PASS: TestReadProxy404 (4.08s)1243=== CONT TestLeadGenerationGrowsWithEveryGrant1244=== RUN TestService_AuthMiddleware_OIDC/valid_OIDC_token1245=== PAUSE TestService_AuthMiddleware_OIDC/valid_OIDC_token1246=== RUN TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1247=== PAUSE TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1248=== RUN TestService_AuthMiddleware_OIDC/malformed_token_rejected1249=== PAUSE TestService_AuthMiddleware_OIDC/malformed_token_rejected1250=== RUN TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1251=== PAUSE TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1252=== CONT TestResolveDBConnectionString1253=== RUN TestResolveDBConnectionString/flag_wins1254=== PAUSE TestResolveDBConnectionString/flag_wins1255=== RUN TestResolveDBConnectionString/file_when_flag_empty1256=== PAUSE TestResolveDBConnectionString/file_when_flag_empty1257=== RUN TestResolveDBConnectionString/missing_file_is_an_error1258=== PAUSE TestResolveDBConnectionString/missing_file_is_an_error1259=== RUN TestResolveDBConnectionString/PGHOST_allows_empty1260=== PAUSE TestResolveDBConnectionString/PGHOST_allows_empty1261=== RUN TestResolveDBConnectionString/nothing_configured1262=== PAUSE TestResolveDBConnectionString/nothing_configured1263=== CONT TestConnectWaitsForAPeerMigration1264--- PASS: TestReadProxyHead (4.50s)1265=== CONT TestLeadIncumbentAnnouncesAtOnce12662026/11/12 00:14:47 INFO Received uploads request method=POST path=/api/pending_closures12672026/11/12 00:14:47 INFO Received uploads request method=POST path=/api/pending_closures1268--- PASS: TestDeduplicatedObjectsRecordedAsPending (4.72s)1269=== CONT TestConcurrentCommitsSharingObjectsDoNotDeadlock1270--- PASS: TestReadProxyConditionalGet (4.77s)1271=== CONT TestService_ReadScope_PublicByDefault12722026/11/12 00:14:47 INFO Aborted multipart uploads count=0 kept=012732026/11/12 00:14:49 WARN Force mode enabled - objects will be deleted immediately without grace period12742026/11/12 00:14:49 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=012752026/11/12 00:14:49 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=012762026/11/12 00:14:49 INFO Vacuumed table table=pending_closures12772026/11/12 00:14:49 INFO Vacuumed table table=pending_objects12782026/11/12 00:14:49 INFO Vacuumed table table=multipart_uploads12792026/11/12 00:14:49 INFO Vacuumed table table=closures12802026/11/12 00:14:49 INFO Vacuumed table table=objects12812026/11/12 00:14:49 INFO Vacuumed table table=pending_closures12822026/11/12 00:14:49 INFO Vacuumed table table=pending_objects1283=== NAME TestPinProtectsFromGC1284 client_integration_test.go:779: Pinned store path: /build/TestPinProtectsFromGC4187367844/001/store/chxfxmc260wcafdq94lbypibilcjmc80-pinned-file.txt1285 client_integration_test.go:780: Unpinned store path: /build/TestPinProtectsFromGC4187367844/001/store/vdyfkh3xcp48fhjnyzz5aih52zfvf0mx-unpinned-file.txt12862026/11/12 00:14:49 INFO Vacuumed table table=multipart_uploads1287--- PASS: TestGCMetrics (5.38s)1288=== CONT TestServerTLSConfig1289=== RUN TestServerTLSConfig/no_client_CA1290=== PAUSE TestServerTLSConfig/no_client_CA1291=== RUN TestServerTLSConfig/missing_CA_file1292=== PAUSE TestServerTLSConfig/missing_CA_file1293=== RUN TestServerTLSConfig/not_a_PEM_file1294=== PAUSE TestServerTLSConfig/not_a_PEM_file1295=== CONT TestLeadUnannouncedStandbyYieldsToIncumbent12962026/11/12 00:14:49 INFO Vacuumed table table=closures12972026/11/12 00:14:49 INFO Vacuumed table table=objects12982026/11/12 00:14:49 INFO Aborted multipart uploads count=0 kept=012992026/11/12 00:14:49 WARN Force mode enabled - objects will be deleted immediately without grace period13002026/11/12 00:14:49 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=013012026/11/12 00:14:49 INFO Vacuumed table table=pending_closures13022026/11/12 00:14:49 INFO Vacuumed table table=pending_objects13032026/11/12 00:14:49 INFO Vacuumed table table=multipart_uploads13042026/11/12 00:14:49 INFO Vacuumed table table=closures13052026/11/12 00:14:49 INFO Vacuumed table table=objects1306--- PASS: TestPushDedupSurvivesConcurrentGC (5.66s)1307=== CONT TestPresentRefreshesOnlyStaleClosures13082026/11/12 00:14:49 INFO lead: acquired remote=192.0.2.1:1234 generation=11309--- PASS: TestReadProxyNarinfoAlreadyDecompressed (1.60s)1310=== CONT TestClientMultipleUploads1311--- PASS: TestService_ReadScope_PublicByDefault (1.30s)1312=== CONT TestPresentReportsOnlyClosureRoots13132026/11/12 00:14:49 INFO lead: acquired remote=192.0.2.1:1234 generation=113142026/11/12 00:14:49 ERROR failed to check GC advisory lock error="query pg_locks: failed to connect to `user=nixbld database=niks3`: /nonexistent/.s.PGSQL.5432 (/nonexistent): dial error: dial unix /nonexistent/.s.PGSQL.5432: connect: no such file or directory"1315--- PASS: TestGCAdvisoryLockBlocksConcurrentRun (1.70s)1316=== CONT TestClientIntegration13172026/11/12 00:14:49 INFO lead: acquired remote=192.0.2.1:1234 generation=113182026/11/12 00:14:49 INFO lead: released remote=192.0.2.1:1234 generation=11319--- PASS: TestLeadIncumbentAnnouncesAtOnce (1.53s)1320=== CONT TestCreatePin_ReservedPins13212026/11/12 00:14:49 INFO lead: acquired remote=192.0.2.1:1234 generation=113222026/11/12 00:14:49 WARN Authentication failed token_preview=wrongtoken token_length=10 reason="static API token mismatch"1323--- PASS: TestService_AuthMiddleware (1.76s)1324=== CONT TestClientErrorHandling1325=== RUN TestClientErrorHandling/InvalidStorePath1326=== PAUSE TestClientErrorHandling/InvalidStorePath1327=== RUN TestClientErrorHandling/InvalidAuthToken1328=== PAUSE TestClientErrorHandling/InvalidAuthToken1329=== RUN TestClientErrorHandling/ServerNotAvailable1330=== PAUSE TestClientErrorHandling/ServerNotAvailable1331=== CONT TestOrphanedObjectsGCStressTest1332--- PASS: TestService_AuthMiddleware_MTLSProxyHeader (1.79s)1333=== CONT TestClientCADerivations13342026/11/12 00:14:49 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted,O=Other"13352026/11/12 00:14:49 WARN mTLS auth: bound subjects configured but subject DN unavailable13362026/11/12 00:14:49 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted"1337--- PASS: TestService_AuthMiddleware_MTLSBoundSubjects (1.92s)1338=== CONT TestOrphanedObjectsGC1339--- PASS: TestReadProxyContentNegotiation (1.97s)1340=== CONT TestObjectStatsUpdatedOncePerCommit13412026/11/12 00:14:49 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:39783/oidc1342--- PASS: TestService_ReadAuthMiddleware (2.04s)1343=== CONT TestCacheStatsHandler13442026/11/12 00:14:50 INFO lead: released remote=192.0.2.1:1234 generation=113452026/11/12 00:14:50 INFO lead: acquired remote=192.0.2.1:1234 generation=213462026/11/12 00:14:50 INFO lead: released remote=192.0.2.1:1234 generation=11347--- PASS: TestLeadEndsOnShutdown (2.63s)1348=== CONT TestObjectStatsTrigger13492026/11/12 00:14:50 INFO lead: released remote=192.0.2.1:1234 generation=113502026/11/12 00:14:50 INFO lead: acquired remote=192.0.2.1:1234 generation=21351--- PASS: TestGCBugBareHashReferences (3.67s)1352=== CONT TestIsValidUploadKey1353=== RUN TestIsValidUploadKey/narinfo1354=== PAUSE TestIsValidUploadKey/narinfo1355=== RUN TestIsValidUploadKey/nar_zst1356=== PAUSE TestIsValidUploadKey/nar_zst1357=== RUN TestIsValidUploadKey/nar_xz1358=== PAUSE TestIsValidUploadKey/nar_xz1359=== RUN TestIsValidUploadKey/nar_plain1360=== PAUSE TestIsValidUploadKey/nar_plain1361=== RUN TestIsValidUploadKey/listing1362=== PAUSE TestIsValidUploadKey/listing1363=== RUN TestIsValidUploadKey/build_log1364=== PAUSE TestIsValidUploadKey/build_log1365=== RUN TestIsValidUploadKey/build_log_home-manager_file1366=== PAUSE TestIsValidUploadKey/build_log_home-manager_file1367=== RUN TestIsValidUploadKey/build_log_plus_in_name1368=== PAUSE TestIsValidUploadKey/build_log_plus_in_name1369=== RUN TestIsValidUploadKey/build_log_question_mark1370=== PAUSE TestIsValidUploadKey/build_log_question_mark1371=== RUN TestIsValidUploadKey/build_log_equals1372=== PAUSE TestIsValidUploadKey/build_log_equals1373=== RUN TestIsValidUploadKey/realisation1374=== PAUSE TestIsValidUploadKey/realisation1375=== RUN TestIsValidUploadKey/realisation_plus_in_output1376=== PAUSE TestIsValidUploadKey/realisation_plus_in_output1377=== RUN TestIsValidUploadKey/nix-cache-info1378=== PAUSE TestIsValidUploadKey/nix-cache-info1379=== RUN TestIsValidUploadKey/index.html1380=== PAUSE TestIsValidUploadKey/index.html1381=== RUN TestIsValidUploadKey/narinfo_key,_nar_type1382=== PAUSE TestIsValidUploadKey/narinfo_key,_nar_type1383=== RUN TestIsValidUploadKey/nar_key,_narinfo_type1384=== PAUSE TestIsValidUploadKey/nar_key,_narinfo_type1385=== RUN TestIsValidUploadKey/listing_key,_narinfo_type1386=== PAUSE TestIsValidUploadKey/listing_key,_narinfo_type1387=== RUN TestIsValidUploadKey/traversal1388=== PAUSE TestIsValidUploadKey/traversal1389=== RUN TestIsValidUploadKey/traversal_nar1390=== PAUSE TestIsValidUploadKey/traversal_nar1391=== RUN TestIsValidUploadKey/absolute1392=== PAUSE TestIsValidUploadKey/absolute1393=== RUN TestIsValidUploadKey/empty_key1394=== PAUSE TestIsValidUploadKey/empty_key1395=== RUN TestIsValidUploadKey/unknown_type1396=== PAUSE TestIsValidUploadKey/unknown_type1397=== CONT TestPendingCleanupUsesOneCutoff1398=== NAME TestClientWithDependencies1399 client_integration_test.go:661: Built derivation: /build/TestClientWithDependencies4200276846/001/store/qz81mcphw0cbhqnvy9j3lx2z4rfkahb8-test-script1400--- PASS: TestConcurrentCommitsSharingObjectsDoNotDeadlock (3.37s)1401=== CONT TestUploadHandlersRejectOversizedBody1402--- PASS: TestObjectStatsTrigger (0.97s)1403=== CONT TestMultipartUploadAbortedWhenCancelledBeforeRecorded14042026/11/12 00:14:51 INFO lead: released remote=192.0.2.1:1234 generation=21405--- PASS: TestConnectWaitsForAPeerMigration (5.82s)1406=== CONT TestUploadHandlersRejectInvalidKeys1407--- PASS: TestCacheStatsHandler (4.01s)1408=== CONT TestMultipartCleanup1409=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info1410=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info1411=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal1412=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal1413=== RUN TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key1414=== PAUSE TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key1415=== RUN TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key1416=== PAUSE TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key1417=== CONT TestCacheConfigHandler1418=== RUN TestCacheConfigHandler/full_config,_no_issuer1419=== PAUSE TestCacheConfigHandler/full_config,_no_issuer1420=== RUN TestCacheConfigHandler/no_cache_url_configured1421=== PAUSE TestCacheConfigHandler/no_cache_url_configured1422=== RUN TestCacheConfigHandler/no_signing_keys1423=== PAUSE TestCacheConfigHandler/no_signing_keys1424=== RUN TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1425=== PAUSE TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1426=== CONT TestProxyWriteTimeout1427=== RUN TestProxyWriteTimeout/narinfo1428=== PAUSE TestProxyWriteTimeout/narinfo1429=== RUN TestProxyWriteTimeout/1_GiB_nar1430=== PAUSE TestProxyWriteTimeout/1_GiB_nar1431=== RUN TestProxyWriteTimeout/10_GiB_nar1432=== PAUSE TestProxyWriteTimeout/10_GiB_nar1433=== RUN TestProxyWriteTimeout/unknown_size1434=== PAUSE TestProxyWriteTimeout/unknown_size1435=== CONT TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle14362026/11/12 00:14:53 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux14372026/11/12 00:14:53 WARN Refused reserved pin name=worker-x86_64-linux14382026/11/12 00:14:53 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux14392026/11/12 00:14:53 INFO Received create pin request method=POST path=/api/pins/my-app14402026/11/12 00:14:53 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux1441--- PASS: TestCreatePin_ReservedPins (4.27s)1442=== CONT TestPendingClosureWriteTimeout1443=== RUN TestPendingClosureWriteTimeout/empty1444=== PAUSE TestPendingClosureWriteTimeout/empty1445=== RUN TestPendingClosureWriteTimeout/400_objects1446=== PAUSE TestPendingClosureWriteTimeout/400_objects1447=== RUN TestPendingClosureWriteTimeout/670k_objects1448=== PAUSE TestPendingClosureWriteTimeout/670k_objects1449=== RUN TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline1450=== PAUSE TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline1451=== CONT TestService_NativeMTLS14522026/11/12 00:14:53 INFO lead: released remote=192.0.2.1:1234 generation=21453--- PASS: TestLeadElectsOneAndHandsOver (5.93s)1454=== CONT TestCacheConfigHandlerMaxNarSize1455--- PASS: TestCacheConfigHandlerMaxNarSize (0.00s)1456=== CONT TestNARDeduplicationMetadataUploadBug1457=== NAME TestClientWithDependencies1458 client_integration_test.go:663: Found 1 dependencies (including self)14592026/11/12 00:14:53 INFO Received push request method=POST path=/api/pushes14602026/11/12 00:14:53 INFO lead: acquired remote=192.0.2.1:1234 generation=314612026/11/12 00:14:53 INFO lead: released remote=192.0.2.1:1234 generation=31462--- PASS: TestLeadGenerationGrowsWithEveryGrant (5.88s)1463=== CONT TestMetricsCommit14642026/11/12 00:14:53 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)14652026/11/12 00:14:53 INFO Uploading chxfxmc260wcafdq94lbypibilcjmc80-pinned-file.txt (128B)14662026/11/12 00:14:53 WARN Failed to register uploaded object key=chxfxmc260wcafdq94lbypibilcjmc80.ls error="server returned 404: 404 page not found\n"14672026/11/12 00:14:53 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign14682026/11/12 00:14:53 WARN Failed to register uploaded object key=nar/0qmhsg42pqsz7x6hh02j8gznnac3yi3qpjhrqbwvz88i194jbb5s.nar.zst error="server returned 404: 404 page not found\n"14692026/11/12 00:14:53 INFO Signed narinfos id=1 count=114702026/11/12 00:14:53 INFO Uploading 1 narinfos14712026/11/12 00:14:53 INFO Received complete push request method=POST path=/api/pushes/1/complete14722026/11/12 00:14:53 WARN Failed to register uploaded object key=chxfxmc260wcafdq94lbypibilcjmc80.narinfo error="server returned 404: 404 page not found\n"1473=== RUN TestUploadHandlersRejectOversizedBody/request_more_parts1474=== PAUSE TestUploadHandlersRejectOversizedBody/request_more_parts1475=== RUN TestUploadHandlersRejectOversizedBody/create_pending_closure1476=== PAUSE TestUploadHandlersRejectOversizedBody/create_pending_closure1477=== RUN TestUploadHandlersRejectOversizedBody/complete_multipart1478=== PAUSE TestUploadHandlersRejectOversizedBody/complete_multipart1479=== CONT TestCreatePendingClosureRejectsOversizedNAR14802026/11/12 00:14:53 INFO Received uploads request method=POST path=/api/pending_closures1481--- PASS: TestCreatePendingClosureRejectsOversizedNAR (0.00s)1482=== CONT TestService_readinessHandler14832026/11/12 00:14:53 INFO Completed push id=1 objects=3 duration=18.825629ms14842026/11/12 00:14:53 INFO Upload complete. (2.473s)1485--- PASS: TestObjectStatsUpdatedOncePerCommit (4.24s)1486=== CONT TestEnsureAbortUploadsRule1487--- PASS: TestPresentReportsOnlyClosureRoots (4.52s)1488=== CONT TestGenerateLandingPage1489--- PASS: TestGenerateLandingPage (0.01s)1490=== CONT TestGracefulShutdownDrainsInflight14912026/11/12 00:14:53 INFO Starting HTTP server address=127.0.0.1:4112314922026/11/12 00:14:53 INFO Shutdown signal received, draining in-flight requests timeout=10s1493--- PASS: TestReadProxyOutlastsServerWriteTimeout (10.56s)1494=== CONT TestService_healthCheckHandler1495--- PASS: TestPresentRefreshesOnlyStaleClosures (4.68s)1496=== CONT TestMetricsInventory1497--- PASS: TestGracefulShutdownDrainsInflight (0.07s)1498=== CONT TestIsValidCachePath/narinfo1499=== CONT TestParseSingleRange/none1500=== CONT TestIsValidCachePath/nar_xz1501=== CONT TestIsValidCachePath/short_hash1502=== CONT TestIsValidCachePath/wrong_extension1503=== CONT TestIsValidCachePath/leading_slash1504=== CONT TestIsValidCachePath/empty1505=== CONT TestIsValidCachePath/random_path1506=== CONT TestIsValidCachePath/invalid_char_u1507=== CONT TestIsValidCachePath/invalid_char_e1508=== CONT TestIsValidCachePath/traversal_in_middle1509=== CONT TestIsValidCachePath/traversal_parent1510=== CONT TestIsValidCachePath/index.html1511=== CONT TestIsValidCachePath/nix-cache-info1512=== CONT TestIsValidCachePath/realisation1513=== CONT TestIsValidCachePath/log1514=== CONT TestIsValidCachePath/ls1515=== CONT TestIsValidCachePath/nar_uncompressed1516=== CONT TestIsValidCachePath/nar_bz21517=== CONT TestParseSingleRange/suffix1518=== CONT TestIsValidCachePath/nar_zst1519=== CONT TestIsValidCachePath/narinfo_all_nix_base32_chars1520--- PASS: TestIsValidCachePath (0.00s)1521 --- PASS: TestIsValidCachePath/narinfo (0.00s)1522 --- PASS: TestIsValidCachePath/nar_xz (0.00s)1523 --- PASS: TestIsValidCachePath/short_hash (0.00s)1524 --- PASS: TestIsValidCachePath/wrong_extension (0.00s)1525 --- PASS: TestIsValidCachePath/leading_slash (0.00s)1526 --- PASS: TestIsValidCachePath/empty (0.00s)1527 --- PASS: TestIsValidCachePath/random_path (0.00s)1528 --- PASS: TestIsValidCachePath/invalid_char_u (0.00s)1529 --- PASS: TestIsValidCachePath/invalid_char_e (0.00s)1530 --- PASS: TestIsValidCachePath/traversal_in_middle (0.00s)1531 --- PASS: TestIsValidCachePath/traversal_parent (0.00s)1532 --- PASS: TestIsValidCachePath/index.html (0.00s)1533 --- PASS: TestIsValidCachePath/nix-cache-info (0.00s)1534 --- PASS: TestIsValidCachePath/realisation (0.00s)1535 --- PASS: TestIsValidCachePath/log (0.00s)1536 --- PASS: TestIsValidCachePath/ls (0.00s)1537 --- PASS: TestIsValidCachePath/nar_uncompressed (0.00s)1538 --- PASS: TestIsValidCachePath/nar_bz2 (0.00s)1539 --- PASS: TestIsValidCachePath/nar_zst (0.00s)1540 --- PASS: TestIsValidCachePath/narinfo_all_nix_base32_chars (0.00s)1541=== CONT TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed15422026/11/12 00:14:53 INFO lead: acquired remote=192.0.2.1:1234 generation=115432026/11/12 00:14:53 INFO Received uploads request method=POST path=/api/pending_closures15442026/11/12 00:14:53 INFO Received uploads request method=POST path=/api/pending_closures15452026/11/12 00:14:53 INFO Received cleanup request method=DELETE path=/api/pending_closures15462026/11/12 00:14:54 INFO lead: released remote=192.0.2.1:1234 generation=115472026/11/12 00:14:54 INFO lead: acquired remote=192.0.2.1:1234 generation=215482026/11/12 00:14:55 INFO lead: given back to the incumbent remote=192.0.2.1:1234 generation=215492026/11/12 00:14:55 INFO lead: acquired remote=192.0.2.1:1234 generation=31550=== NAME TestOrphanedObjectsGCStressTest1551 orphaned_objects_gc_test.go:422: Created 10 active closures, 5 to-delete closures, 20 orphaned chains15522026/11/12 00:14:56 INFO Received push request method=POST path=/api/pushes1553=== NAME TestClientMultipleUploads1554 client_integration_test.go:406: Created store path 0: /build/TestClientMultipleUploads622667313/001/store/zw4y0p76a20w5lgwnv0z06j2rnc21fwr-test-file-0.txt15552026/11/12 00:14:56 INFO Received push request method=POST path=/api/pushes1556=== NAME TestOrphanedObjectsGCStressTest1557 orphaned_objects_gc_test.go:440: Marked 210 objects for deletion15582026/11/12 00:14:56 INFO Uploading 2 paths to 127.0.0.1 (0 already cached)15592026/11/12 00:14:56 INFO Uploading lv0d2b52p41kgbz20d863z2qap6b3088-shared-dep (136B)15602026/11/12 00:14:56 INFO Uploading 1ijc0l71rphsq1i3gqnh5hyyhvrx8wa0-top (224B)1561=== NAME TestClientIntegration1562 client_integration_test.go:292: Created store path: /build/TestClientIntegration2692129413/002/store/8bjchx871jjga3brc46y1nf7rarqqgmb-test-file.txt15632026/11/12 00:14:56 INFO Uploading 2 paths to 127.0.0.1 (1 already cached)15642026/11/12 00:14:56 INFO Uploading 6rvr32r8y0hb8fzd4wmhk9a0y4mz8bkf-b (216B)15652026/11/12 00:14:56 WARN Failed to register uploaded object key=lv0d2b52p41kgbz20d863z2qap6b3088.ls error="server returned 404: 404 page not found\n"15662026/11/12 00:14:56 INFO Uploading b72fcc0l0iazk3nxprfgzxc5xfnzdwy8-shared-dep (136B)15672026/11/12 00:14:56 WARN Failed to register uploaded object key=1ijc0l71rphsq1i3gqnh5hyyhvrx8wa0.ls error="server returned 404: 404 page not found\n"15682026/11/12 00:14:56 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"15692026/11/12 00:14:56 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign15702026/11/12 00:14:56 WARN Failed to register uploaded object key=nar/0kxvk80j6wvipa82a9yv74p58dyj4nrw4b8rvvv4f357954prqdv.nar.zst error="server returned 404: 404 page not found\n"15712026/11/12 00:14:56 INFO Signed narinfos id=1 count=215722026/11/12 00:14:56 INFO Uploading 2 narinfos15732026/11/12 00:14:56 INFO Received complete push request method=POST path=/api/pushes/1/complete15742026/11/12 00:14:56 WARN Failed to register uploaded object key=lv0d2b52p41kgbz20d863z2qap6b3088.narinfo error="server returned 404: 404 page not found\n"15752026/11/12 00:14:56 WARN Failed to register uploaded object key=1ijc0l71rphsq1i3gqnh5hyyhvrx8wa0.narinfo error="server returned 404: 404 page not found\n"15762026/11/12 00:14:56 WARN Failed to register uploaded object key=b72fcc0l0iazk3nxprfgzxc5xfnzdwy8.ls error="server returned 404: 404 page not found\n"15772026/11/12 00:14:56 WARN Failed to register uploaded object key=0yh0wf8a7mk08qzv7znza7fk6m5jiv47.ls error="server returned 404: 404 page not found\n"15782026/11/12 00:14:56 WARN Failed to register uploaded object key=6rvr32r8y0hb8fzd4wmhk9a0y4mz8bkf.ls error="server returned 404: 404 page not found\n"15792026/11/12 00:14:56 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"15802026/11/12 00:14:56 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign15812026/11/12 00:14:56 WARN Failed to register uploaded object key=nar/1ycng88imirz3qdn6gaa3rn4klfgwdaczgs89azhx21wwwj3kax5.nar.zst error="server returned 404: 404 page not found\n"15822026/11/12 00:14:56 INFO Signed narinfos id=1 count=315832026/11/12 00:14:56 INFO Uploading 3 narinfos15842026/11/12 00:14:56 WARN Failed to register uploaded object key=0yh0wf8a7mk08qzv7znza7fk6m5jiv47.narinfo error="server returned 404: 404 page not found\n"15852026/11/12 00:14:56 WARN Failed to register uploaded object key=6rvr32r8y0hb8fzd4wmhk9a0y4mz8bkf.narinfo error="server returned 404: 404 page not found\n"15862026/11/12 00:14:56 INFO Completed push id=1 objects=6 duration=16.180425ms15872026/11/12 00:14:56 INFO Upload complete. (2.699s)15882026/11/12 00:14:56 INFO Received complete push request method=POST path=/api/pushes/1/complete15892026/11/12 00:14:56 WARN Failed to register uploaded object key=b72fcc0l0iazk3nxprfgzxc5xfnzdwy8.narinfo error="server returned 404: 404 page not found\n"15902026/11/12 00:14:56 INFO Received uploads request method=POST path=/api/pending_closures1591=== NAME TestClientSharedPathCommittedMidPush1592 client_integration_test.go:728: Retrieved narinfo from S3:1593 StorePath: /build/TestClientSharedPathCommittedMidPush2011630636/001/store/lv0d2b52p41kgbz20d863z2qap6b3088-shared-dep1594 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1595 Compression: zstd1596 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821597 NarSize: 1361598 References: 1599 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n1600 client_integration_test.go:728: Retrieved narinfo from S3:1601 StorePath: /build/TestClientSharedPathCommittedMidPush2011630636/001/store/1ijc0l71rphsq1i3gqnh5hyyhvrx8wa0-top1602 URL: nar/0kxvk80j6wvipa82a9yv74p58dyj4nrw4b8rvvv4f357954prqdv.nar.zst1603 Compression: zstd1604 NarHash: sha256:0kxvk80j6wvipa82a9yv74p58dyj4nrw4b8rvvv4f357954prqdv1605 NarSize: 2241606 References: /build/TestClientSharedPathCommittedMidPush2011630636/001/store/lv0d2b52p41kgbz20d863z2qap6b3088-shared-dep1607 CA: text:sha256:159pvh9w0gzsxiq2hvxbyq0jjb2qvpww1w3i7mv6f2xhbh2x9xg116082026/11/12 00:14:56 INFO Received push request method=POST path=/api/pushes16092026/11/12 00:14:56 INFO Completed push id=1 objects=8 duration=11.12858ms16102026/11/12 00:14:56 INFO Upload complete. (2.712s)1611--- PASS: TestClientSharedPathCommittedMidPush (13.15s)1612=== CONT TestParseSingleRange/start_far_past_EOF1613=== NAME TestClientMultipleUploads1614 client_integration_test.go:406: Created store path 1: /build/TestClientMultipleUploads622667313/001/store/gjr7iv66gyvbq9b5ambrqj0x3in5k1j7-test-file-1.txt1615=== CONT TestParseSingleRange/start_past_EOF1616=== CONT TestParseSingleRange/single_byte1617=== CONT TestParseSingleRange/suffix_exceeds_size1618=== CONT TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed1619=== NAME TestClientPushesUseOnePush1620 client_pushes_test.go:97: Retrieved narinfo from S3:1621 StorePath: /build/TestClientPushesUseOnePush2090543746/001/store/b72fcc0l0iazk3nxprfgzxc5xfnzdwy8-shared-dep1622 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1623 Compression: zstd1624 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821625 NarSize: 1361626 References: 1627 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n16282026/11/12 00:14:56 INFO Received uploads request method=POST path=/api/pending_closures1629 client_pushes_test.go:97: Retrieved narinfo from S3:1630 StorePath: /build/TestClientPushesUseOnePush2090543746/001/store/0yh0wf8a7mk08qzv7znza7fk6m5jiv47-a1631 URL: nar/1ycng88imirz3qdn6gaa3rn4klfgwdaczgs89azhx21wwwj3kax5.nar.zst1632 Compression: zstd1633 NarHash: sha256:1ycng88imirz3qdn6gaa3rn4klfgwdaczgs89azhx21wwwj3kax51634 NarSize: 2161635 References: /build/TestClientPushesUseOnePush2090543746/001/store/b72fcc0l0iazk3nxprfgzxc5xfnzdwy8-shared-dep1636 CA: text:sha256:152pmss1m8p4cmqplnh3lapw5hibx16nf2f42d4wvqvybm9qkxpy16372026/11/12 00:14:56 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)16382026/11/12 00:14:56 INFO Uploading qz81mcphw0cbhqnvy9j3lx2z4rfkahb8-test-script (136B)16392026/11/12 00:14:56 INFO Uploading 2 paths to 127.0.0.1 (1 already cached)16402026/11/12 00:14:56 INFO Uploading 1ifdfwfidgs2y3y3lq8x8bq2qn7s004i-a (216B)16412026/11/12 00:14:56 INFO Uploading g02d28nl2cwsf2fi46hdkqsjmab7imy0-shared-dep (136B)1642 client_pushes_test.go:97: Retrieved narinfo from S3:1643 StorePath: /build/TestClientPushesUseOnePush2090543746/001/store/6rvr32r8y0hb8fzd4wmhk9a0y4mz8bkf-b1644 URL: nar/1ycng88imirz3qdn6gaa3rn4klfgwdaczgs89azhx21wwwj3kax5.nar.zst1645 Compression: zstd1646 NarHash: sha256:1ycng88imirz3qdn6gaa3rn4klfgwdaczgs89azhx21wwwj3kax51647 NarSize: 2161648 References: /build/TestClientPushesUseOnePush2090543746/001/store/b72fcc0l0iazk3nxprfgzxc5xfnzdwy8-shared-dep1649 CA: text:sha256:152pmss1m8p4cmqplnh3lapw5hibx16nf2f42d4wvqvybm9qkxpy16502026/11/12 00:14:56 WARN Failed to register uploaded object key=qz81mcphw0cbhqnvy9j3lx2z4rfkahb8.ls error="server returned 404: 404 page not found\n"16512026/11/12 00:14:56 WARN Failed to register uploaded object key=1ifdfwfidgs2y3y3lq8x8bq2qn7s004i.ls error="server returned 404: 404 page not found\n"16522026/11/12 00:14:56 WARN Failed to register uploaded object key=jr7caz6v2q2rgq80zfy5j8s28hn35zay.ls error="server returned 404: 404 page not found\n"16532026/11/12 00:14:56 WARN Failed to register uploaded object key=log/d5fmbl1vm2ah7yylh60xvn69ipcmy070-test-script.drv error="server returned 404: 404 page not found\n"16542026/11/12 00:14:56 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign16552026/11/12 00:14:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign16562026/11/12 00:14:56 WARN Failed to register uploaded object key=g02d28nl2cwsf2fi46hdkqsjmab7imy0.ls error="server returned 404: 404 page not found\n"16572026/11/12 00:14:56 WARN Failed to register uploaded object key=nar/1pn0bh4gcc39lypi37bkfvywbxg6npk30r642p1nzb4kkwk8jgis.nar.zst error="server returned 404: 404 page not found\n"16582026/11/12 00:14:56 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"16592026/11/12 00:14:56 WARN Failed to register uploaded object key=nar/1s7yghixzpinyqm7g59kyn3sav81pmnsdi5v63m6x4r2vacy907j.nar.zst error="server returned 404: 404 page not found\n"16602026/11/12 00:14:56 INFO Signed narinfos id=1 count=216612026/11/12 00:14:56 INFO Signed narinfos id=1 count=11662--- PASS: TestClientPushesUseOnePush (15.13s)1663=== CONT TestParseSingleRange/malformed_end_before_start1664=== CONT TestParseSingleRange/end_clamped_to_size1665=== CONT TestParseSingleRange/open-ended1666=== CONT TestParseSingleRange/closed1667=== CONT TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost16682026/11/12 00:14:56 INFO Uploading 1 narinfos16692026/11/12 00:14:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign16702026/11/12 00:14:56 INFO Signed narinfos id=2 count=216712026/11/12 00:14:56 INFO Uploading 4 narinfos1672=== NAME TestClientCADerivations1673 client_ca_test.go:136: Built CA derivation: /build/TestClientCADerivations187940657/001/store/m95jbpyjicxy05plqfxidd1w32sgyid1-ca-test16742026/11/12 00:14:56 WARN Failed to register uploaded object key=jr7caz6v2q2rgq80zfy5j8s28hn35zay.narinfo error="server returned 404: 404 page not found\n"16752026/11/12 00:14:56 INFO Received complete push request method=POST path=/api/pushes/1/complete16762026/11/12 00:14:56 WARN Failed to register uploaded object key=1ifdfwfidgs2y3y3lq8x8bq2qn7s004i.narinfo error="server returned 404: 404 page not found\n"16772026/11/12 00:14:56 WARN Failed to register uploaded object key=qz81mcphw0cbhqnvy9j3lx2z4rfkahb8.narinfo error="server returned 404: 404 page not found\n"16782026/11/12 00:14:56 WARN Failed to register uploaded object key=g02d28nl2cwsf2fi46hdkqsjmab7imy0.narinfo error="server returned 404: 404 page not found\n"16792026/11/12 00:14:56 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete16802026/11/12 00:14:56 WARN Failed to register uploaded object key=g02d28nl2cwsf2fi46hdkqsjmab7imy0.narinfo error="server returned 404: 404 page not found\n"16812026/11/12 00:14:56 INFO Received uploads request method=POST path=/api/pending_closures16822026/11/12 00:14:56 INFO Completed push id=1 objects=4 duration=15.771161ms16832026/11/12 00:14:56 INFO Upload complete. (110ms)1684=== NAME TestClientWithDependencies1685 client_integration_test.go:665: Skipping nix copy test - isolated store (/build/TestClientWithDependencies4200276846/001/store) requires matching store prefix16862026/11/12 00:14:56 INFO Completed upload id=1 objects=6 duration=18.645767ms1687--- PASS: TestClientWithDependencies (13.08s)1688=== CONT TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed16892026/11/12 00:14:56 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete16902026/11/12 00:14:56 INFO Completed upload id=2 objects=6 duration=2.766965ms16912026/11/12 00:14:56 INFO Upload complete. (2.767s)1692=== NAME TestClientFallsBackToClosures1693 client_pushes_test.go:112: Retrieved narinfo from S3:1694 StorePath: /build/TestClientFallsBackToClosures3760957678/001/store/g02d28nl2cwsf2fi46hdkqsjmab7imy0-shared-dep1695 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1696 Compression: zstd1697 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821698 NarSize: 1361699 References: 1700 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n1701 client_pushes_test.go:112: Retrieved narinfo from S3:1702 StorePath: /build/TestClientFallsBackToClosures3760957678/001/store/1ifdfwfidgs2y3y3lq8x8bq2qn7s004i-a1703 URL: nar/1pn0bh4gcc39lypi37bkfvywbxg6npk30r642p1nzb4kkwk8jgis.nar.zst1704 Compression: zstd1705 NarHash: sha256:1pn0bh4gcc39lypi37bkfvywbxg6npk30r642p1nzb4kkwk8jgis1706 NarSize: 2161707 References: /build/TestClientFallsBackToClosures3760957678/001/store/g02d28nl2cwsf2fi46hdkqsjmab7imy0-shared-dep1708 CA: text:sha256:1xl5i1xanbzsy3rccz0km8a8xa6f4knlygylzrafqjhml2y8j03v1709 client_pushes_test.go:112: Retrieved narinfo from S3:1710 StorePath: /build/TestClientFallsBackToClosures3760957678/001/store/jr7caz6v2q2rgq80zfy5j8s28hn35zay-b1711 URL: nar/1pn0bh4gcc39lypi37bkfvywbxg6npk30r642p1nzb4kkwk8jgis.nar.zst1712 Compression: zstd1713 NarHash: sha256:1pn0bh4gcc39lypi37bkfvywbxg6npk30r642p1nzb4kkwk8jgis1714 NarSize: 2161715 References: /build/TestClientFallsBackToClosures3760957678/001/store/g02d28nl2cwsf2fi46hdkqsjmab7imy0-shared-dep1716 CA: text:sha256:1xl5i1xanbzsy3rccz0km8a8xa6f4knlygylzrafqjhml2y8j03v1717--- PASS: TestClientFallsBackToClosures (15.71s)1718=== CONT TestParseSingleRange/malformed_no_dash1719=== CONT TestParseSingleRange/malformed_both_empty1720=== CONT TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed1721=== NAME TestOrphanedObjectsGC1722 orphaned_objects_gc_test.go:287: GC Test Summary:1723 orphaned_objects_gc_test.go:288: - Kept: 2 objects from closure A1724 orphaned_objects_gc_test.go:289: - Deleted: 2 objects from closure B1725 orphaned_objects_gc_test.go:290: - Deleted: 6 orphaned chain objects (X1->X2->X3)1726 orphaned_objects_gc_test.go:291: - Deleted: 2 orphaned single objects (Y)1727 orphaned_objects_gc_test.go:292: - Total deleted: 10 objects1728--- PASS: TestOrphanedObjectsGC (6.94s)1729=== CONT TestParseSingleRange/multi-range_ignored1730=== CONT TestParseSingleRange/unknown_unit1731=== CONT TestGCEndsOnShutdown/before_the_run1732--- PASS: TestParseSingleRange (0.00s)1733 --- PASS: TestParseSingleRange/none (0.00s)1734 --- PASS: TestParseSingleRange/suffix (0.00s)1735 --- PASS: TestParseSingleRange/start_far_past_EOF (0.00s)1736 --- PASS: TestParseSingleRange/start_past_EOF (0.00s)1737 --- PASS: TestParseSingleRange/single_byte (0.00s)1738 --- PASS: TestParseSingleRange/suffix_exceeds_size (0.00s)1739 --- PASS: TestParseSingleRange/malformed_end_before_start (0.00s)1740 --- PASS: TestParseSingleRange/end_clamped_to_size (0.00s)1741 --- PASS: TestParseSingleRange/open-ended (0.00s)1742 --- PASS: TestParseSingleRange/closed (0.00s)1743 --- PASS: TestParseSingleRange/malformed_no_dash (0.00s)1744 --- PASS: TestParseSingleRange/malformed_both_empty (0.00s)1745 --- PASS: TestParseSingleRange/multi-range_ignored (0.00s)1746 --- PASS: TestParseSingleRange/unknown_unit (0.00s)17472026/11/12 00:14:56 INFO Installed lifecycle rule aborting incomplete NAR uploads bucket=bucket85 days=21748--- PASS: TestEnsureAbortUploadsRule (2.68s)1749=== CONT TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark17502026/11/12 00:14:56 INFO Received uploads request method=POST path=/api/pending_closures17512026/11/12 00:14:56 INFO Received uploads request method=POST path=/api/pending_closures17522026/11/12 00:14:56 INFO Received cleanup request method=DELETE path=/api/pending_closures17532026/11/12 00:14:56 WARN Failed to abort upload, keeping its closure key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst error="Get \"http://127.0.0.1:1/bucket84/?location=\": dial tcp 127.0.0.1:1: connect: connection refused" code=""17542026/11/12 00:14:56 INFO Aborted multipart uploads count=0 kept=117552026/11/12 00:14:56 INFO Received cleanup request method=DELETE path=/api/pending_closures1756--- PASS: TestService_healthCheckHandler (2.72s)1757=== CONT TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete17582026/11/12 00:14:56 INFO Aborted multipart uploads count=1 kept=01759--- PASS: TestMultipartCleanup (2.94s)1760=== CONT TestPresentNotReportedWhileGCDeletesClosure/stale17612026/11/12 00:14:56 INFO Received complete multipart upload request method=POST path=/api/multipart/complete17622026/11/12 00:14:56 INFO Received push request method=POST path=/api/pushes17632026/11/12 00:14:56 INFO Received complete push request method=POST path=/api/pushes/1/complete17642026/11/12 00:14:56 INFO Received uploads request method=POST path=/api/pending_closures17652026/11/12 00:14:56 WARN readiness check failed error="closed pool"1766--- PASS: TestService_readinessHandler (2.97s)1767=== CONT TestPresentNotReportedWhileGCDeletesClosure/fresh1768--- PASS: TestMultipartUploadAbortedWhenCancelledBeforeRecorded (5.35s)1769=== CONT TestForceGCDuringPushOffersSweptObject/before_pending_rows17702026/11/12 00:14:56 WARN mTLS auth: subject not in bound subjects subject="CN=reader"17712026/11/12 00:14:56 WARN mTLS auth: subject not in bound subjects subject="CN=reader"1772--- PASS: TestService_NativeMTLS (3.10s)1773=== CONT TestForceGCDuringPushOffersSweptObject/after_presence_check1774--- PASS: TestMetricsInventory (2.97s)1775=== CONT TestPush_RejectsBadRequests/no_roots17762026/11/12 00:14:56 INFO Received push request method=POST path=/api/pushes1777=== CONT TestPush_RejectsBadRequests/root_not_in_objects17782026/11/12 00:14:56 INFO Received push request method=POST path=/api/pushes1779=== CONT TestPush_RejectsBadRequests/bad_root17802026/11/12 00:14:56 INFO Received push request method=POST path=/api/pushes1781=== CONT TestPush_RejectsBadRequests/no_objects17822026/11/12 00:14:56 INFO Received push request method=POST path=/api/pushes1783=== CONT TestService_RequireScope_OIDC/builder_may_write1784--- PASS: TestPush_RejectsBadRequests (5.48s)1785 --- PASS: TestPush_RejectsBadRequests/no_roots (0.00s)1786 --- PASS: TestPush_RejectsBadRequests/root_not_in_objects (0.00s)1787 --- PASS: TestPush_RejectsBadRequests/bad_root (0.00s)1788 --- PASS: TestPush_RejectsBadRequests/no_objects (0.00s)1789=== CONT TestService_RequireScope_OIDC/reader_may_read1790=== CONT TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1791=== CONT TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read17922026/11/12 00:14:56 WARN Authentication failed token_preview=wrong token_length=5 oidc_error="not a JWT" oidc_provider=""1793=== CONT TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1794=== CONT TestService_RequireScope_OIDC/reader_may_read_with_netrc1795=== CONT TestService_RequireScope_OIDC/anonymous_may_not_read1796=== CONT TestService_RequireScope_OIDC/writer_implies_read1797=== CONT TestService_RequireScope_OIDC/static_token_may_write1798=== CONT TestService_RequireScope_OIDC/reader_may_not_write1799=== CONT TestService_RequireScope_OIDC/ops_may_admin1800=== CONT TestService_RequireScope_OIDC/static_token_may_admin1801=== CONT TestService_RequireScope_OIDC/ops_may_not_write1802=== CONT TestService_RequireScope_OIDC/builder_may_not_admin1803=== CONT TestService_AuthMiddleware_OIDC/valid_OIDC_token1804--- PASS: TestService_RequireScope_OIDC (4.35s)1805 --- PASS: TestService_RequireScope_OIDC/builder_may_write (0.01s)1806 --- PASS: TestService_RequireScope_OIDC/reader_may_read (0.00s)1807 --- PASS: TestService_RequireScope_OIDC/empty_netrc_password_may_not_read (0.00s)1808 --- PASS: TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read (0.00s)1809 --- PASS: TestService_RequireScope_OIDC/static_token_may_admin_with_netrc (0.00s)1810 --- PASS: TestService_RequireScope_OIDC/reader_may_read_with_netrc (0.00s)1811 --- PASS: TestService_RequireScope_OIDC/anonymous_may_not_read (0.00s)1812 --- PASS: TestService_RequireScope_OIDC/writer_implies_read (0.00s)1813 --- PASS: TestService_RequireScope_OIDC/static_token_may_write (0.00s)1814 --- PASS: TestService_RequireScope_OIDC/reader_may_not_write (0.00s)1815 --- PASS: TestService_RequireScope_OIDC/ops_may_admin (0.00s)1816 --- PASS: TestService_RequireScope_OIDC/static_token_may_admin (0.00s)1817 --- PASS: TestService_RequireScope_OIDC/ops_may_not_write (0.00s)1818 --- PASS: TestService_RequireScope_OIDC/builder_may_not_admin (0.00s)1819=== CONT TestService_AuthMiddleware_OIDC/malformed_token_rejected18202026/11/12 00:14:56 WARN Authentication failed token_preview=not-a-valid-jwt token_length=15 oidc_error="not a JWT" oidc_provider=""1821=== CONT TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1822=== CONT TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected18232026/11/12 00:14:56 WARN Authentication failed token_preview=eyJhbGciOi...zJbLTDFOmw token_length=701 oidc_error="no rule matched: claim \"repository_owner\" value [otherorg] not in allowed patterns [myorg]" oidc_provider=test1824=== CONT TestResolveDBConnectionString/flag_wins1825=== CONT TestResolveDBConnectionString/PGHOST_allows_empty1826=== CONT TestResolveDBConnectionString/nothing_configured1827=== CONT TestResolveDBConnectionString/missing_file_is_an_error1828=== CONT TestResolveDBConnectionString/file_when_flag_empty1829=== CONT TestServerTLSConfig/no_client_CA1830=== CONT TestServerTLSConfig/missing_CA_file1831=== CONT TestServerTLSConfig/not_a_PEM_file1832--- PASS: TestService_AuthMiddleware_OIDC (4.14s)1833 --- PASS: TestService_AuthMiddleware_OIDC/valid_OIDC_token (0.01s)1834 --- PASS: TestService_AuthMiddleware_OIDC/malformed_token_rejected (0.00s)1835 --- PASS: TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured (0.00s)1836 --- PASS: TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected (0.00s)1837--- PASS: TestResolveDBConnectionString (0.00s)1838 --- PASS: TestResolveDBConnectionString/flag_wins (0.00s)1839 --- PASS: TestResolveDBConnectionString/PGHOST_allows_empty (0.00s)1840 --- PASS: TestResolveDBConnectionString/nothing_configured (0.00s)1841 --- PASS: TestResolveDBConnectionString/missing_file_is_an_error (0.00s)1842 --- PASS: TestResolveDBConnectionString/file_when_flag_empty (0.00s)1843--- PASS: TestServerTLSConfig (0.00s)1844 --- PASS: TestServerTLSConfig/no_client_CA (0.00s)1845 --- PASS: TestServerTLSConfig/missing_CA_file (0.00s)1846 --- PASS: TestServerTLSConfig/not_a_PEM_file (0.00s)1847=== CONT TestClientErrorHandling/InvalidStorePath18482026/11/12 00:14:56 INFO Received complete multipart upload request method=POST path=/api/multipart/complete18492026/11/12 00:14:56 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000100000000000000000000.nar.zst upload_id=Y2IyZGFmYTktOThkZS00Yzg4LTkwODQtNmM1YmU2YTZhMGY2LjU3YjgyZWM2LWJjYmItNDJkNS04NWQ1LWNjNGRiMjIzYWE4YngxNzk0NDQyNDk2NDE4NTE3NzI1 parts=1018502026/11/12 00:14:56 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete18512026/11/12 00:14:56 INFO Completed upload id=1 objects=3 duration=3.222789ms18522026/11/12 00:14:56 INFO lead: released remote=192.0.2.1:1234 generation=31853--- PASS: TestLeadUnannouncedStandbyYieldsToIncumbent (7.86s)1854=== CONT TestClientErrorHandling/ServerNotAvailable18552026/11/12 00:14:56 INFO Received uploads request method=POST path=/api/pending_closures18562026/11/12 00:14:56 INFO Object in database but missing from S3 key=nar/dadb44fdadb44fdadb44fdadb44f000100000000000000000000.nar.zst18572026/11/12 00:14:56 WARN Narinfo references a NAR missing from S3, will re-upload narinfo=dadb44fdadb44fdadb44fdadb44f0001.narinfo1858=== CONT TestClientErrorHandling/InvalidAuthToken1859=== NAME TestOrphanedObjectsGCStressTest1860 orphaned_objects_gc_test.go:503: Stress test completed successfully:1861 orphaned_objects_gc_test.go:504: - Active objects preserved: 201862 orphaned_objects_gc_test.go:505: - Objects deleted: 2101863 orphaned_objects_gc_test.go:506: - Total GC'd: 2101864--- PASS: TestOrphanedObjectsGCStressTest (8.04s)1865=== CONT TestIsValidUploadKey/narinfo1866=== CONT TestIsValidUploadKey/realisation_plus_in_output1867=== CONT TestIsValidUploadKey/unknown_type1868=== CONT TestIsValidUploadKey/empty_key1869=== CONT TestIsValidUploadKey/absolute1870=== CONT TestIsValidUploadKey/traversal_nar1871=== CONT TestIsValidUploadKey/traversal1872=== CONT TestIsValidUploadKey/listing_key,_narinfo_type1873=== CONT TestIsValidUploadKey/nar_key,_narinfo_type1874=== CONT TestIsValidUploadKey/narinfo_key,_nar_type1875=== CONT TestIsValidUploadKey/index.html1876=== CONT TestIsValidUploadKey/nix-cache-info1877=== CONT TestIsValidUploadKey/build_log_home-manager_file1878=== CONT TestIsValidUploadKey/realisation1879=== CONT TestIsValidUploadKey/build_log_equals1880=== CONT TestIsValidUploadKey/build_log_question_mark1881=== CONT TestIsValidUploadKey/build_log_plus_in_name1882=== CONT TestIsValidUploadKey/nar_plain1883=== CONT TestIsValidUploadKey/build_log1884=== CONT TestIsValidUploadKey/listing1885=== CONT TestIsValidUploadKey/nar_xz1886=== CONT TestIsValidUploadKey/nar_zst1887--- PASS: TestIsValidUploadKey (0.00s)1888 --- PASS: TestIsValidUploadKey/narinfo (0.00s)1889 --- PASS: TestIsValidUploadKey/realisation_plus_in_output (0.00s)1890 --- PASS: TestIsValidUploadKey/unknown_type (0.00s)1891 --- PASS: TestIsValidUploadKey/empty_key (0.00s)1892 --- PASS: TestIsValidUploadKey/absolute (0.00s)1893 --- PASS: TestIsValidUploadKey/traversal_nar (0.00s)1894 --- PASS: TestIsValidUploadKey/traversal (0.00s)1895 --- PASS: TestIsValidUploadKey/listing_key,_narinfo_type (0.00s)1896 --- PASS: TestIsValidUploadKey/nar_key,_narinfo_type (0.00s)1897 --- PASS: TestIsValidUploadKey/narinfo_key,_nar_type (0.00s)1898 --- PASS: TestIsValidUploadKey/index.html (0.00s)1899 --- PASS: TestIsValidUploadKey/nix-cache-info (0.00s)1900 --- PASS: TestIsValidUploadKey/build_log_home-manager_file (0.00s)1901 --- PASS: TestIsValidUploadKey/realisation (0.00s)1902 --- PASS: TestIsValidUploadKey/build_log_equals (0.00s)1903 --- PASS: TestIsValidUploadKey/build_log_question_mark (0.00s)1904 --- PASS: TestIsValidUploadKey/build_log_plus_in_name (0.00s)1905 --- PASS: TestIsValidUploadKey/nar_plain (0.00s)1906 --- PASS: TestIsValidUploadKey/build_log (0.00s)1907 --- PASS: TestIsValidUploadKey/listing (0.00s)1908 --- PASS: TestIsValidUploadKey/nar_xz (0.00s)1909 --- PASS: TestIsValidUploadKey/nar_zst (0.00s)1910=== CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info19112026/11/12 00:14:57 INFO Received uploads request method=POST path=/1912=== CONT TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key19132026/11/12 00:14:57 INFO Received complete multipart upload request method=POST path=/1914=== CONT TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key19152026/11/12 00:14:57 INFO Received request for more parts method=POST path=/1916=== CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal19172026/11/12 00:14:57 INFO Received uploads request method=POST path=/1918--- PASS: TestUploadHandlersRejectInvalidKeys (0.01s)1919 --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info (0.00s)1920 --- PASS: TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key (0.00s)1921 --- PASS: TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key (0.00s)1922 --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal (0.00s)1923=== CONT TestCacheConfigHandler/full_config,_no_issuer1924=== CONT TestCacheConfigHandler/no_signing_keys1925=== CONT TestCacheConfigHandler/no_cache_url_configured1926=== CONT TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1927--- PASS: TestCacheConfigHandler (0.00s)1928 --- PASS: TestCacheConfigHandler/full_config,_no_issuer (0.00s)1929 --- PASS: TestCacheConfigHandler/no_signing_keys (0.00s)1930 --- PASS: TestCacheConfigHandler/no_cache_url_configured (0.00s)1931 --- PASS: TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator (0.00s)1932=== CONT TestProxyWriteTimeout/narinfo1933=== CONT TestProxyWriteTimeout/10_GiB_nar1934=== CONT TestProxyWriteTimeout/unknown_size1935=== CONT TestProxyWriteTimeout/1_GiB_nar1936--- PASS: TestProxyWriteTimeout (0.00s)1937 --- PASS: TestProxyWriteTimeout/narinfo (0.00s)1938 --- PASS: TestProxyWriteTimeout/10_GiB_nar (0.00s)1939 --- PASS: TestProxyWriteTimeout/unknown_size (0.00s)1940 --- PASS: TestProxyWriteTimeout/1_GiB_nar (0.00s)1941=== CONT TestPendingClosureWriteTimeout/empty1942=== CONT TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline19432026/11/12 00:14:57 INFO Received uploads request method=POST path=/api/pending_closures19442026/11/12 00:14:57 INFO Received complete multipart upload request method=POST path=/api/multipart/complete19452026/11/12 00:14:58 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000200000000000000000000.nar.zst upload_id=Y2IyZGFmYTktOThkZS00Yzg4LTkwODQtNmM1YmU2YTZhMGY2LjM4MTFkYWU1LTU0MzgtNDc2My1hMjM2LTY5NzRmMzAzNTZiNngxNzk0NDQyNDk3NTI5MjcyODY2 parts=1019462026/11/12 00:14:58 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete19472026/11/12 00:14:58 INFO Completed upload id=1 objects=3 duration=2.739425ms19482026/11/12 00:14:58 INFO Received uploads request method=POST path=/api/pending_closures1949=== CONT TestPendingClosureWriteTimeout/670k_objects1950=== CONT TestPendingClosureWriteTimeout/400_objects1951=== CONT TestUploadHandlersRejectOversizedBody/request_more_parts19522026/11/12 00:14:58 INFO Received request for more parts method=POST path=/19532026/11/12 00:14:58 INFO Completed push id=1 objects=4 duration=1.450255266s19542026/11/12 00:14:58 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete19552026/11/12 00:14:58 INFO Completed upload id=2 objects=2 duration=2.156739ms1956--- PASS: TestMetricsCommit (4.49s)1957=== CONT TestUploadHandlersRejectOversizedBody/complete_multipart19582026/11/12 00:14:58 INFO Received complete multipart upload request method=POST path=/19592026/11/12 00:14:58 INFO Received uploads request method=POST path=/api/pending_closures1960=== CONT TestUploadHandlersRejectOversizedBody/create_pending_closure19612026/11/12 00:14:58 INFO Received uploads request method=POST path=/19622026/11/12 00:14:58 INFO Received push request method=POST path=/api/pushes19632026/11/12 00:14:58 INFO Received push request method=POST path=/api/pushes19642026/11/12 00:14:58 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)19652026/11/12 00:14:58 INFO Uploading vdyfkh3xcp48fhjnyzz5aih52zfvf0mx-unpinned-file.txt (128B)19662026/11/12 00:14:58 INFO Received sign narinfos request method=POST path=/api/pushes/2/sign19672026/11/12 00:14:58 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)19682026/11/12 00:14:58 WARN Failed to register uploaded object key=vdyfkh3xcp48fhjnyzz5aih52zfvf0mx.ls error="server returned 404: 404 page not found\n"19692026/11/12 00:14:58 INFO Uploading 8bjchx871jjga3brc46y1nf7rarqqgmb-test-file.txt (152B)19702026/11/12 00:14:58 WARN Failed to register uploaded object key=nar/0gh0wdnr6gmk4r5sfzzh12isnl14rgmkrwy1ws41hpsrlndclg5h.nar.zst error="server returned 404: 404 page not found\n"19712026/11/12 00:14:58 INFO Signed narinfos id=2 count=119722026/11/12 00:14:58 INFO Uploading 1 narinfos1973=== NAME TestClientMultipleUploads1974 client_integration_test.go:406: Created store path 2: /build/TestClientMultipleUploads622667313/001/store/v2c4z112cx2viw11fs5dr65qslzrnij3-test-file-2.txt1975=== NAME TestClientCADerivations1976 client_ca_test.go:139: Found 1 dependencies (including self)19772026/11/12 00:14:58 INFO Received complete push request method=POST path=/api/pushes/2/complete19782026/11/12 00:14:58 WARN Failed to register uploaded object key=vdyfkh3xcp48fhjnyzz5aih52zfvf0mx.narinfo error="server returned 404: 404 page not found\n"19792026/11/12 00:14:58 WARN Failed to register uploaded object key=8bjchx871jjga3brc46y1nf7rarqqgmb.ls error="server returned 404: 404 page not found\n"19802026/11/12 00:14:58 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign19812026/11/12 00:14:58 WARN Failed to register uploaded object key=nar/11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1.nar.zst error="server returned 404: 404 page not found\n"19822026/11/12 00:14:58 INFO Signed narinfos id=1 count=119832026/11/12 00:14:58 INFO Uploading 1 narinfos19842026/11/12 00:14:58 INFO Completed push id=2 objects=3 duration=2.911626ms19852026/11/12 00:14:58 INFO Upload complete. (2.146s)19862026/11/12 00:14:58 INFO Received complete push request method=POST path=/api/pushes/1/complete19872026/11/12 00:14:58 WARN Failed to register uploaded object key=8bjchx871jjga3brc46y1nf7rarqqgmb.narinfo error="server returned 404: 404 page not found\n"1988=== NAME TestNARDeduplicationMetadataUploadBug1989 metadata_upload_test.go:48: First store path: /build/TestNARDeduplicationMetadataUploadBug3449883100/001/store/lls2k5ry2pyy1i0f7qxh79f1wc2h4vy4-file1.txt19902026/11/12 00:14:58 INFO Completed push id=1 objects=3 duration=16.874471ms19912026/11/12 00:14:58 INFO Upload complete. (2.133s)19922026/11/12 00:14:58 INFO Received uploads request method=POST path=/api/pending_closures19932026/11/12 00:14:58 INFO Aborted multipart uploads count=0 kept=019942026/11/12 00:14:58 INFO Received uploads request method=POST path=/api/pending_closures19952026/11/12 00:14:58 INFO Aborted multipart uploads count=0 kept=019962026/11/12 00:14:58 INFO Received uploads request method=POST path=/api/pending_closures19972026/11/12 00:14:58 WARN Force mode enabled - objects will be deleted immediately without grace period19982026/11/12 00:14:58 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=019992026/11/12 00:14:58 INFO Vacuumed table table=pending_closures20002026/11/12 00:14:58 INFO Vacuumed table table=pending_objects
loading 666 hidden lines…
26672026/11/12 00:15:21 INFO Received uploads request method=POST path=/26682026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=026692026/11/12 00:15:21 INFO Received push request method=POST path=/26702026/11/12 00:15:21 INFO Received push request method=POST path=/26712026/11/12 00:15:21 INFO Received complete upload request method=POST path=/26722026/11/12 00:15:21 INFO Completed upload id=3 objects=24 duration=1.833216ms26732026/11/12 00:15:21 INFO Received complete push request method=POST path=/26742026/11/12 00:15:21 INFO Received complete push request method=POST path=/26752026/11/12 00:15:21 INFO Received complete upload request method=POST path=/26762026/11/12 00:15:21 INFO Received uploads request method=POST path=/26772026/11/12 00:15:21 INFO Completed push id=3 objects=5 duration=2.762845ms26782026/11/12 00:15:21 INFO Completed upload id=6 objects=9 duration=1.793316ms26792026/11/12 00:15:21 INFO Received uploads request method=POST path=/26802026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=026812026/11/12 00:15:21 INFO Completed push id=1 objects=8 duration=7.466687ms26822026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=026832026/11/12 00:15:21 INFO Received push request method=POST path=/26842026/11/12 00:15:21 WARN Force mode enabled - objects will be deleted immediately without grace period26852026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=026862026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=026872026/11/12 00:15:21 WARN Force mode enabled - objects will be deleted immediately without grace period26882026/11/12 00:15:21 INFO Received push request method=POST path=/26892026/11/12 00:15:21 INFO Vacuumed table table=pending_closures26902026/11/12 00:15:21 INFO Received push request method=POST path=/26912026/11/12 00:15:21 INFO Vacuumed table table=pending_objects26922026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads26932026/11/12 00:15:21 INFO Received uploads request method=POST path=/26942026/11/12 00:15:21 INFO Vacuumed table table=closures26952026/11/12 00:15:21 INFO Vacuumed table table=objects26962026/11/12 00:15:21 INFO Received push request method=POST path=/26972026/11/12 00:15:21 INFO Received push request method=POST path=/26982026/11/12 00:15:21 INFO Received complete upload request method=POST path=/26992026/11/12 00:15:21 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."27002026/11/12 00:15:21 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."27012026/11/12 00:15:21 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."27022026/11/12 00:15:21 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."27032026/11/12 00:15:21 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."27042026/11/12 00:15:21 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."27052026/11/12 00:15:21 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."27062026/11/12 00:15:21 INFO Received complete push request method=POST path=/27072026/11/12 00:15:21 INFO Completed upload id=10 objects=23 duration=1.998738ms27082026/11/12 00:15:21 INFO Received uploads request method=POST path=/27092026/11/12 00:15:21 INFO Completed push id=2 objects=7 duration=3.428911ms27102026/11/12 00:15:21 INFO Received uploads request method=POST path=/27112026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=027122026/11/12 00:15:21 INFO Received complete upload request method=POST path=/27132026/11/12 00:15:21 INFO Received push request method=POST path=/27142026/11/12 00:15:21 INFO Completed upload id=2 objects=2 duration=1.755216ms27152026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=027162026/11/12 00:15:21 INFO Received uploads request method=POST path=/27172026/11/12 00:15:21 INFO Received complete push request method=POST path=/27182026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=027192026/11/12 00:15:21 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."27202026/11/12 00:15:21 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."27212026/11/12 00:15:21 INFO Completed push id=2 objects=25 duration=3.658153ms27222026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=027232026/11/12 00:15:21 INFO Vacuumed table table=pending_closures27242026/11/12 00:15:21 INFO Vacuumed table table=pending_objects27252026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads27262026/11/12 00:15:21 INFO Received uploads request method=POST path=/27272026/11/12 00:15:21 INFO Received complete upload request method=POST path=/27282026/11/12 00:15:21 INFO Vacuumed table table=closures27292026/11/12 00:15:21 INFO Completed upload id=3 objects=5 duration=1.254891ms27302026/11/12 00:15:21 INFO Received uploads request method=POST path=/27312026/11/12 00:15:21 INFO Received push request method=POST path=/27322026/11/12 00:15:21 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."27332026/11/12 00:15:21 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."27342026/11/12 00:15:21 INFO Vacuumed table table=objects27352026/11/12 00:15:21 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."27362026/11/12 00:15:21 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."27372026/11/12 00:15:21 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."27382026/11/12 00:15:21 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."27392026/11/12 00:15:21 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."27402026/11/12 00:15:21 INFO Received complete push request method=POST path=/27412026/11/12 00:15:21 INFO Completed push id=6 objects=5 duration=1.776756ms27422026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=027432026/11/12 00:15:21 INFO Received complete push request method=POST path=/27442026/11/12 00:15:21 INFO Received push request method=POST path=/27452026/11/12 00:15:21 INFO Completed push id=4 objects=4 duration=2.31604ms27462026/11/12 00:15:21 INFO Received complete push request method=POST path=/27472026/11/12 00:15:21 INFO Received complete upload request method=POST path=/27482026/11/12 00:15:21 INFO Received complete push request method=POST path=/27492026/11/12 00:15:21 INFO Completed push id=7 objects=20 duration=2.21306ms27502026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=027512026/11/12 00:15:21 INFO Received push request method=POST path=/27522026/11/12 00:15:21 INFO Completed upload id=3 objects=8 duration=1.852237ms27532026/11/12 00:15:21 INFO Received push request method=POST path=/27542026/11/12 00:15:21 INFO Received push request method=POST path=/27552026/11/12 00:15:21 INFO Completed push id=14 objects=3 duration=3.568793ms27562026/11/12 00:15:21 INFO Received complete upload request method=POST path=/27572026/11/12 00:15:21 INFO Completed upload id=11 objects=6 duration=900.748µs27582026/11/12 00:15:21 INFO Received push request method=POST path=/27592026/11/12 00:15:21 INFO Received push request method=POST path=/27602026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=027612026/11/12 00:15:21 INFO Received complete upload request method=POST path=/27622026/11/12 00:15:21 INFO Vacuumed table table=pending_closures27632026/11/12 00:15:21 INFO Vacuumed table table=pending_objects27642026/11/12 00:15:21 INFO Completed upload id=10 objects=2 duration=2.262ms27652026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads27662026/11/12 00:15:21 INFO Vacuumed table table=closures27672026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=027682026/11/12 00:15:21 INFO Received push request method=POST path=/27692026/11/12 00:15:21 INFO Vacuumed table table=objects27702026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=027712026/11/12 00:15:21 INFO Received push request method=POST path=/27722026/11/12 00:15:21 INFO Received uploads request method=POST path=/27732026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027742026/11/12 00:15:21 INFO Received push request method=POST path=/27752026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=1 objects-deleted-after-grace-period=0 objects-failed-to-delete=027762026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=027772026/11/12 00:15:21 INFO Vacuumed table table=pending_closures27782026/11/12 00:15:21 INFO Vacuumed table table=pending_closures27792026/11/12 00:15:21 INFO Vacuumed table table=pending_objects27802026/11/12 00:15:21 INFO Vacuumed table table=pending_objects27812026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads27822026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads27832026/11/12 00:15:21 INFO Received push request method=POST path=/27842026/11/12 00:15:21 INFO Received complete push request method=POST path=/27852026/11/12 00:15:21 INFO Received push request method=POST path=/27862026/11/12 00:15:21 INFO Vacuumed table table=closures27872026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027882026/11/12 00:15:21 INFO Completed push id=5 objects=6 duration=2.754625ms27892026/11/12 00:15:21 INFO Vacuumed table table=closures27902026/11/12 00:15:21 INFO Vacuumed table table=pending_closures27912026/11/12 00:15:21 INFO Vacuumed table table=objects27922026/11/12 00:15:21 INFO Vacuumed table table=objects27932026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=027942026/11/12 00:15:21 INFO Vacuumed table table=pending_objects27952026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads27962026/11/12 00:15:21 INFO Vacuumed table table=closures27972026/11/12 00:15:21 INFO Vacuumed table table=objects27982026/11/12 00:15:21 INFO Received push request method=POST path=/27992026/11/12 00:15:21 INFO Received complete push request method=POST path=/28002026/11/12 00:15:21 INFO Completed push id=12 objects=15 duration=1.914158ms28012026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=028022026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=028032026/11/12 00:15:21 INFO Received push request method=POST path=/28042026/11/12 00:15:21 INFO Received push request method=POST path=/28052026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=028062026/11/12 00:15:21 WARN Force mode enabled - objects will be deleted immediately without grace period28072026/11/12 00:15:21 INFO Vacuumed table table=pending_closures28082026/11/12 00:15:21 INFO Received complete push request method=POST path=/28092026/11/12 00:15:21 INFO Vacuumed table table=pending_objects28102026/11/12 00:15:21 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."28112026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads28122026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028132026/11/12 00:15:21 INFO Vacuumed table table=closures28142026/11/12 00:15:21 INFO Vacuumed table table=pending_closures28152026/11/12 00:15:21 INFO Received push request method=POST path=/28162026/11/12 00:15:21 INFO Vacuumed table table=objects28172026/11/12 00:15:21 INFO Completed push id=2 objects=10 duration=2.756764ms28182026/11/12 00:15:21 INFO Vacuumed table table=pending_objects28192026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads28202026/11/12 00:15:21 INFO Vacuumed table table=closures28212026/11/12 00:15:21 INFO Vacuumed table table=objects28222026/11/12 00:15:21 INFO Received complete push request method=POST path=/28232026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=028242026/11/12 00:15:21 INFO Completed push id=5 objects=3 duration=3.182588ms28252026/11/12 00:15:21 INFO Received push request method=POST path=/28262026/11/12 00:15:21 INFO Received complete push request method=POST path=/28272026/11/12 00:15:21 INFO Received uploads request method=POST path=/28282026/11/12 00:15:21 INFO Completed push id=5 objects=3 duration=1.435433ms28292026/11/12 00:15:21 INFO Received uploads request method=POST path=/28302026/11/12 00:15:21 INFO Received push request method=POST path=/28312026/11/12 00:15:21 INFO Received push request method=POST path=/28322026/11/12 00:15:21 INFO Received push request method=POST path=/28332026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=028342026/11/12 00:15:21 INFO Received push request method=POST path=/28352026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=028362026/11/12 00:15:21 INFO Received complete upload request method=POST path=/28372026/11/12 00:15:21 INFO Completed upload id=3 objects=25 duration=2.15804ms28382026/11/12 00:15:21 INFO Received complete push request method=POST path=/28392026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=028402026/11/12 00:15:21 INFO Received push request method=POST path=/28412026/11/12 00:15:21 INFO Completed push id=8 objects=7 duration=1.676935ms28422026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028432026/11/12 00:15:21 INFO Received push request method=POST path=/28442026/11/12 00:15:21 INFO Vacuumed table table=pending_closures28452026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=028462026/11/12 00:15:21 INFO Vacuumed table table=pending_objects28472026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads28482026/11/12 00:15:21 INFO Received complete push request method=POST path=/28492026/11/12 00:15:21 INFO Vacuumed table table=pending_closures28502026/11/12 00:15:21 INFO Vacuumed table table=closures28512026/11/12 00:15:21 INFO Vacuumed table table=objects28522026/11/12 00:15:21 INFO Vacuumed table table=pending_objects28532026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads28542026/11/12 00:15:21 INFO Completed push id=11 objects=8 duration=2.830645ms28552026/11/12 00:15:21 INFO Vacuumed table table=closures28562026/11/12 00:15:21 INFO Vacuumed table table=objects28572026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=028582026/11/12 00:15:21 INFO Received uploads request method=POST path=/28592026/11/12 00:15:21 INFO Received complete push request method=POST path=/28602026/11/12 00:15:21 INFO Completed push id=8 objects=6 duration=1.590234ms28612026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=028622026/11/12 00:15:21 INFO Received complete push request method=POST path=/28632026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028642026/11/12 00:15:21 INFO Completed push id=4 objects=4 duration=3.049228ms28652026/11/12 00:15:21 INFO Vacuumed table table=pending_closures28662026/11/12 00:15:21 INFO Vacuumed table table=pending_objects28672026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads28682026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=028692026/11/12 00:15:21 INFO Vacuumed table table=closures28702026/11/12 00:15:21 INFO Vacuumed table table=objects28712026/11/12 00:15:21 INFO Received push request method=POST path=/28722026/11/12 00:15:21 INFO Received complete upload request method=POST path=/28732026/11/12 00:15:21 INFO Received uploads request method=POST path=/28742026/11/12 00:15:21 INFO Completed upload id=9 objects=16 duration=1.544514ms28752026/11/12 00:15:21 INFO Received push request method=POST path=/28762026/11/12 00:15:21 INFO Received complete push request method=POST path=/28772026/11/12 00:15:21 INFO Completed push id=6 objects=6 duration=2.009938ms28782026/11/12 00:15:21 INFO Received push request method=POST path=/28792026/11/12 00:15:21 INFO Received uploads request method=POST path=/28802026/11/12 00:15:21 INFO Received uploads request method=POST path=/28812026/11/12 00:15:21 INFO Received push request method=POST path=/28822026/11/12 00:15:21 INFO Received uploads request method=POST path=/28832026/11/12 00:15:21 INFO Received push request method=POST path=/28842026/11/12 00:15:21 INFO Received push request method=POST path=/28852026/11/12 00:15:21 INFO Received complete push request method=POST path=/28862026/11/12 00:15:21 INFO Received complete push request method=POST path=/28872026/11/12 00:15:21 INFO Received push request method=POST path=/28882026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=028892026/11/12 00:15:21 INFO Completed push id=6 objects=26 duration=4.538881ms28902026/11/12 00:15:21 INFO Completed push id=5 objects=9 duration=5.132106ms28912026/11/12 00:15:21 INFO Received push request method=POST path=/28922026/11/12 00:15:21 INFO Received uploads request method=POST path=/28932026/11/12 00:15:21 WARN Force mode enabled - objects will be deleted immediately without grace period28942026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028952026/11/12 00:15:21 INFO Vacuumed table table=pending_closures28962026/11/12 00:15:21 INFO Vacuumed table table=pending_objects28972026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads28982026/11/12 00:15:21 INFO Vacuumed table table=closures28992026/11/12 00:15:21 INFO Vacuumed table table=objects29002026/11/12 00:15:21 INFO Received push request method=POST path=/29012026/11/12 00:15:21 INFO Received complete push request method=POST path=/29022026/11/12 00:15:21 INFO Received uploads request method=POST path=/29032026/11/12 00:15:21 INFO Received push request method=POST path=/29042026/11/12 00:15:21 INFO Completed push id=10 objects=3 duration=2.084899ms29052026/11/12 00:15:21 INFO Received push request method=POST path=/29062026/11/12 00:15:21 INFO Received push request method=POST path=/29072026/11/12 00:15:21 INFO Received complete push request method=POST path=/29082026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=029092026/11/12 00:15:21 INFO Completed push id=7 objects=27 duration=2.564623ms29102026/11/12 00:15:21 WARN Force mode enabled - objects will be deleted immediately without grace period29112026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029122026/11/12 00:15:21 INFO Received uploads request method=POST path=/29132026/11/12 00:15:21 INFO Vacuumed table table=pending_closures29142026/11/12 00:15:21 INFO Vacuumed table table=pending_objects29152026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads29162026/11/12 00:15:21 INFO Vacuumed table table=closures29172026/11/12 00:15:21 INFO Vacuumed table table=objects29182026/11/12 00:15:21 INFO Received uploads request method=POST path=/29192026/11/12 00:15:21 INFO Received push request method=POST path=/29202026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=029212026/11/12 00:15:21 INFO Received complete upload request method=POST path=/29222026/11/12 00:15:21 INFO Completed upload id=15 objects=6 duration=762.727µs29232026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029242026/11/12 00:15:21 INFO Vacuumed table table=pending_closures29252026/11/12 00:15:21 INFO Vacuumed table table=pending_objects29262026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads29272026/11/12 00:15:21 INFO Vacuumed table table=closures29282026/11/12 00:15:21 INFO Vacuumed table table=objects29292026/11/12 00:15:21 INFO Received complete upload request method=POST path=/29302026/11/12 00:15:21 INFO Completed upload id=10 objects=17 duration=1.10177ms29312026/11/12 00:15:21 INFO Received push request method=POST path=/29322026/11/12 00:15:21 INFO Received uploads request method=POST path=/29332026/11/12 00:15:21 INFO Received push request method=POST path=/29342026/11/12 00:15:21 INFO Received complete upload request method=POST path=/29352026/11/12 00:15:21 INFO Completed upload id=3 objects=9 duration=1.228191ms29362026/11/12 00:15:21 INFO Received uploads request method=POST path=/29372026/11/12 00:15:21 INFO Received push request method=POST path=/29382026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=029392026/11/12 00:15:21 INFO Received push request method=POST path=/29402026/11/12 00:15:21 WARN Force mode enabled - objects will be deleted immediately without grace period29412026/11/12 00:15:21 INFO Received push request method=POST path=/29422026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029432026/11/12 00:15:21 INFO Vacuumed table table=pending_closures29442026/11/12 00:15:21 INFO Vacuumed table table=pending_objects29452026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads29462026/11/12 00:15:21 INFO Vacuumed table table=closures29472026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=029482026/11/12 00:15:21 INFO Received push request method=POST path=/29492026/11/12 00:15:21 INFO Vacuumed table table=objects29502026/11/12 00:15:21 INFO Received push request method=POST path=/29512026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029522026/11/12 00:15:21 INFO Vacuumed table table=pending_closures29532026/11/12 00:15:21 INFO Vacuumed table table=pending_objects29542026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads29552026/11/12 00:15:21 INFO Vacuumed table table=closures29562026/11/12 00:15:21 INFO Vacuumed table table=objects29572026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=029582026/11/12 00:15:21 WARN Force mode enabled - objects will be deleted immediately without grace period29592026/11/12 00:15:21 INFO Received complete push request method=POST path=/29602026/11/12 00:15:21 INFO Received push request method=POST path=/29612026/11/12 00:15:21 INFO Completed push id=5 objects=6 duration=2.065818ms29622026/11/12 00:15:21 INFO Received uploads request method=POST path=/29632026/11/12 00:15:21 INFO Received uploads request method=POST path=/29642026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=029652026/11/12 00:15:21 INFO Received push request method=POST path=/29662026/11/12 00:15:21 WARN Force mode enabled - objects will be deleted immediately without grace period29672026/11/12 00:15:21 INFO Received complete push request method=POST path=/29682026/11/12 00:15:21 INFO Completed push id=16 objects=27 duration=1.980677ms29692026/11/12 00:15:21 INFO Received complete push request method=POST path=/29702026/11/12 00:15:21 INFO Received uploads request method=POST path=/29712026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=029722026/11/12 00:15:21 INFO Completed push id=14 objects=6 duration=1.570575ms29732026/11/12 00:15:21 INFO Received uploads request method=POST path=/29742026/11/12 00:15:21 WARN Force mode enabled - objects will be deleted immediately without grace period29752026/11/12 00:15:21 INFO Received push request method=POST path=/29762026/11/12 00:15:21 INFO Received uploads request method=POST path=/29772026/11/12 00:15:21 INFO Received push request method=POST path=/29782026/11/12 00:15:21 INFO Received uploads request method=POST path=/29792026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029802026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=4 objects-deleted-after-grace-period=0 objects-failed-to-delete=029812026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029822026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029832026/11/12 00:15:21 INFO Vacuumed table table=pending_closures29842026/11/12 00:15:21 INFO Vacuumed table table=pending_closures29852026/11/12 00:15:21 INFO Vacuumed table table=pending_closures29862026/11/12 00:15:21 INFO Vacuumed table table=pending_closures29872026/11/12 00:15:21 INFO Vacuumed table table=pending_objects29882026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads29892026/11/12 00:15:21 INFO Vacuumed table table=closures29902026/11/12 00:15:21 INFO Vacuumed table table=pending_objects29912026/11/12 00:15:21 INFO Vacuumed table table=pending_objects29922026/11/12 00:15:21 INFO Vacuumed table table=pending_objects29932026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads29942026/11/12 00:15:21 INFO Vacuumed table table=objects29952026/11/12 00:15:21 INFO Received push request method=POST path=/29962026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads29972026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads29982026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=029992026/11/12 00:15:21 INFO Received push request method=POST path=/30002026/11/12 00:15:21 INFO Vacuumed table table=closures30012026/11/12 00:15:21 INFO Vacuumed table table=closures30022026/11/12 00:15:21 INFO Vacuumed table table=pending_closures30032026/11/12 00:15:21 INFO Vacuumed table table=closures30042026/11/12 00:15:21 INFO Vacuumed table table=objects30052026/11/12 00:15:21 INFO Vacuumed table table=pending_objects30062026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads30072026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=030082026/11/12 00:15:21 INFO Vacuumed table table=objects30092026/11/12 00:15:21 INFO Received push request method=POST path=/30102026/11/12 00:15:21 INFO Vacuumed table table=objects30112026/11/12 00:15:21 INFO Vacuumed table table=closures30122026/11/12 00:15:21 WARN Force mode enabled - objects will be deleted immediately without grace period30132026/11/12 00:15:21 WARN Force mode enabled - objects will be deleted immediately without grace period30142026/11/12 00:15:21 INFO Vacuumed table table=objects30152026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030162026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030172026/11/12 00:15:21 INFO Vacuumed table table=pending_closures30182026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030192026/11/12 00:15:21 INFO Vacuumed table table=pending_closures30202026/11/12 00:15:21 INFO Vacuumed table table=pending_objects30212026/11/12 00:15:21 INFO Vacuumed table table=pending_objects30222026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads30232026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads30242026/11/12 00:15:21 INFO Vacuumed table table=closures30252026/11/12 00:15:21 INFO Vacuumed table table=closures30262026/11/12 00:15:21 INFO Vacuumed table table=pending_closures30272026/11/12 00:15:21 INFO Vacuumed table table=objects30282026/11/12 00:15:21 INFO Vacuumed table table=objects30292026/11/12 00:15:21 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."30302026/11/12 00:15:21 INFO Received uploads request method=POST path=/30312026/11/12 00:15:21 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."30322026/11/12 00:15:21 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."30332026/11/12 00:15:21 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."30342026/11/12 00:15:21 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."30352026/11/12 00:15:21 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."30362026/11/12 00:15:21 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."30372026/11/12 00:15:21 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."30382026/11/12 00:15:21 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."30392026/11/12 00:15:21 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."30402026/11/12 00:15:21 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."30412026/11/12 00:15:21 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."30422026/11/12 00:15:21 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."30432026/11/12 00:15:21 ERROR failed to remove object object=realisations/sha256:aaaaaaaa!out.doi error="Access Denied."30442026/11/12 00:15:21 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."30452026/11/12 00:15:21 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."30462026/11/12 00:15:21 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."30472026/11/12 00:15:21 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."30482026/11/12 00:15:21 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."30492026/11/12 00:15:21 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."30502026/11/12 00:15:21 INFO Vacuumed table table=pending_objects30512026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads30522026/11/12 00:15:21 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."30532026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=030542026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=030552026/11/12 00:15:21 WARN Force mode enabled - objects will be deleted immediately without grace period30562026/11/12 00:15:21 INFO Vacuumed table table=closures30572026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030582026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=3 objects-failed-to-delete=030592026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=030602026/11/12 00:15:21 INFO Received push request method=POST path=/30612026/11/12 00:15:21 INFO Vacuumed table table=objects30622026/11/12 00:15:21 INFO Vacuumed table table=pending_closures30632026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=1 objects-deleted-after-grace-period=0 objects-failed-to-delete=030642026/11/12 00:15:21 INFO Vacuumed table table=pending_closures30652026/11/12 00:15:21 WARN Force mode enabled - objects will be deleted immediately without grace period30662026/11/12 00:15:21 INFO Vacuumed table table=pending_objects30672026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads30682026/11/12 00:15:21 INFO Received push request method=POST path=/30692026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=030702026/11/12 00:15:21 INFO Vacuumed table table=closures30712026/11/12 00:15:21 INFO Vacuumed table table=pending_closures30722026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030732026/11/12 00:15:21 INFO Vacuumed table table=objects30742026/11/12 00:15:21 WARN Force mode enabled - objects will be deleted immediately without grace period30752026/11/12 00:15:21 INFO Vacuumed table table=pending_objects30762026/11/12 00:15:21 INFO Received complete push request method=POST path=/30772026/11/12 00:15:21 INFO Vacuumed table table=pending_objects30782026/11/12 00:15:21 INFO Vacuumed table table=pending_closures30792026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads30802026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads30812026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030822026/11/12 00:15:21 INFO Vacuumed table table=closures30832026/11/12 00:15:21 INFO Received push request method=POST path=/30842026/11/12 00:15:21 INFO Vacuumed table table=pending_objects30852026/11/12 00:15:21 INFO Vacuumed table table=closures30862026/11/12 00:15:21 INFO Completed push id=13 objects=6 duration=2.884866ms30872026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads30882026/11/12 00:15:21 INFO Vacuumed table table=closures30892026/11/12 00:15:21 INFO Vacuumed table table=pending_closures30902026/11/12 00:15:21 INFO Vacuumed table table=objects30912026/11/12 00:15:21 INFO Vacuumed table table=objects30922026/11/12 00:15:21 INFO Vacuumed table table=objects30932026/11/12 00:15:21 INFO Received complete push request method=POST path=/30942026/11/12 00:15:21 INFO Vacuumed table table=pending_objects30952026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads30962026/11/12 00:15:21 INFO Vacuumed table table=closures30972026/11/12 00:15:21 INFO Completed push id=18 objects=10 duration=2.460502ms30982026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=8 objects-deleted-after-grace-period=8 objects-failed-to-delete=030992026/11/12 00:15:21 INFO Vacuumed table table=objects31002026/11/12 00:15:21 INFO Received push request method=POST path=/31012026/11/12 00:15:21 INFO Vacuumed table table=pending_closures31022026/11/12 00:15:21 INFO Vacuumed table table=pending_objects31032026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads31042026/11/12 00:15:21 INFO Received uploads request method=POST path=/31052026/11/12 00:15:21 INFO Received complete push request method=POST path=/31062026/11/12 00:15:21 INFO Vacuumed table table=closures31072026/11/12 00:15:21 INFO Vacuumed table table=objects31082026/11/12 00:15:21 INFO Received push request method=POST path=/31092026/11/12 00:15:21 INFO Completed push id=9 objects=7 duration=2.766285ms31102026/11/12 00:15:21 INFO Received uploads request method=POST path=/31112026/11/12 00:15:21 INFO Received uploads request method=POST path=/31122026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=031132026/11/12 00:15:21 INFO Received push request method=POST path=/31142026/11/12 00:15:21 WARN Force mode enabled - objects will be deleted immediately without grace period31152026/11/12 00:15:21 INFO Received uploads request method=POST path=/31162026/11/12 00:15:21 INFO Received push request method=POST path=/31172026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=031182026/11/12 00:15:21 WARN Force mode enabled - objects will be deleted immediately without grace period31192026/11/12 00:15:21 INFO Received push request method=POST path=/31202026/11/12 00:15:21 INFO Received complete upload request method=POST path=/31212026/11/12 00:15:21 INFO Received push request method=POST path=/31222026/11/12 00:15:21 INFO Received push request method=POST path=/31232026/11/12 00:15:21 INFO Completed upload id=10 objects=7 duration=2.319301ms31242026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031252026/11/12 00:15:21 INFO Vacuumed table table=pending_closures31262026/11/12 00:15:21 INFO Received complete upload request method=POST path=/31272026/11/12 00:15:21 INFO Vacuumed table table=pending_objects31282026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads31292026/11/12 00:15:21 INFO Vacuumed table table=closures31302026/11/12 00:15:21 INFO Vacuumed table table=objects31312026/11/12 00:15:21 INFO Received uploads request method=POST path=/31322026/11/12 00:15:21 INFO Completed upload id=19 objects=25 duration=2.778805ms31332026/11/12 00:15:21 INFO Received complete push request method=POST path=/31342026/11/12 00:15:21 INFO Received uploads request method=POST path=/31352026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=16 objects-failed-to-delete=031362026/11/12 00:15:21 INFO Vacuumed table table=pending_closures31372026/11/12 00:15:21 INFO Completed push id=11 objects=11 duration=4.166137ms31382026/11/12 00:15:21 INFO Vacuumed table table=pending_objects31392026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads31402026/11/12 00:15:21 INFO Vacuumed table table=closures31412026/11/12 00:15:21 INFO Vacuumed table table=objects31422026/11/12 00:15:21 INFO Received uploads request method=POST path=/31432026/11/12 00:15:21 INFO Received uploads request method=POST path=/31442026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=031452026/11/12 00:15:21 WARN Force mode enabled - objects will be deleted immediately without grace period31462026/11/12 00:15:21 INFO Received complete push request method=POST path=/31472026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=031482026/11/12 00:15:21 INFO Received push request method=POST path=/31492026/11/12 00:15:21 INFO Completed push id=17 objects=6 duration=1.947458ms31502026/11/12 00:15:21 WARN Force mode enabled - objects will be deleted immediately without grace period31512026/11/12 00:15:21 INFO Received push request method=POST path=/31522026/11/12 00:15:21 INFO Received push request method=POST path=/31532026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=031542026/11/12 00:15:21 INFO Received complete upload request method=POST path=/31552026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=031562026/11/12 00:15:21 INFO Completed upload id=17 objects=6 duration=2.712984ms31572026/11/12 00:15:21 INFO Vacuumed table table=pending_closures31582026/11/12 00:15:21 INFO Vacuumed table table=pending_objects31592026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads31602026/11/12 00:15:21 INFO Received uploads request method=POST path=/31612026/11/12 00:15:21 INFO Vacuumed table table=closures31622026/11/12 00:15:21 INFO Vacuumed table table=objects31632026/11/12 00:15:21 INFO Received complete upload request method=POST path=/31642026/11/12 00:15:21 INFO Received complete upload request method=POST path=/31652026/11/12 00:15:21 INFO Received complete push request method=POST path=/31662026/11/12 00:15:21 INFO Completed upload id=6 objects=24 duration=2.868586ms31672026/11/12 00:15:21 INFO Completed upload id=4 objects=14 duration=4.47666ms31682026/11/12 00:15:21 INFO Received uploads request method=POST path=/31692026/11/12 00:15:21 INFO Completed push id=3 objects=5 duration=6.299456ms31702026/11/12 00:15:21 INFO Received uploads request method=POST path=/31712026/11/12 00:15:21 INFO Received complete push request method=POST path=/31722026/11/12 00:15:21 INFO Received complete push request method=POST path=/31732026/11/12 00:15:21 INFO Received complete push request method=POST path=/31742026/11/12 00:15:21 INFO Completed push id=6 objects=8 duration=3.626092ms31752026/11/12 00:15:21 INFO Completed push id=12 objects=2 duration=3.466771ms31762026/11/12 00:15:21 INFO Received push request method=POST path=/31772026/11/12 00:15:21 INFO Completed push id=4 objects=6 duration=2.469282ms31782026/11/12 00:15:21 INFO Received complete push request method=POST path=/31792026/11/12 00:15:21 INFO Received push request method=POST path=/31802026/11/12 00:15:21 INFO Completed push id=4 objects=7 duration=2.117978ms31812026/11/12 00:15:21 INFO Received push request method=POST path=/31822026/11/12 00:15:21 INFO Received complete upload request method=POST path=/31832026/11/12 00:15:21 INFO Received push request method=POST path=/31842026/11/12 00:15:21 INFO Completed upload id=18 objects=4 duration=1.783376ms31852026/11/12 00:15:21 INFO Received push request method=POST path=/31862026/11/12 00:15:21 INFO Received uploads request method=POST path=/31872026/11/12 00:15:21 INFO Received uploads request method=POST path=/31882026/11/12 00:15:21 INFO Received uploads request method=POST path=/31892026/11/12 00:15:21 INFO Received push request method=POST path=/31902026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=031912026/11/12 00:15:21 WARN Force mode enabled - objects will be deleted immediately without grace period31922026/11/12 00:15:21 INFO Received complete push request method=POST path=/31932026/11/12 00:15:21 INFO Received complete push request method=POST path=/31942026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031952026/11/12 00:15:21 INFO Completed push id=8 objects=6 duration=2.381522ms31962026/11/12 00:15:21 INFO Completed push id=6 objects=7 duration=2.831926ms31972026/11/12 00:15:21 INFO Vacuumed table table=pending_closures31982026/11/12 00:15:21 INFO Received push request method=POST path=/31992026/11/12 00:15:21 INFO Vacuumed table table=pending_objects32002026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads32012026/11/12 00:15:21 INFO Vacuumed table table=closures32022026/11/12 00:15:21 INFO Received uploads request method=POST path=/32032026/11/12 00:15:21 INFO Vacuumed table table=objects32042026/11/12 00:15:21 INFO Received uploads request method=POST path=/32052026/11/12 00:15:21 INFO Received uploads request method=POST path=/32062026/11/12 00:15:21 INFO Received uploads request method=POST path=/32072026/11/12 00:15:21 INFO Received complete push request method=POST path=/32082026/11/12 00:15:21 INFO Received complete upload request method=POST path=/32092026/11/12 00:15:21 INFO Completed upload id=9 objects=2 duration=1.397812ms32102026/11/12 00:15:21 INFO Received complete upload request method=POST path=/32112026/11/12 00:15:21 INFO Received uploads request method=POST path=/32122026/11/12 00:15:21 INFO Completed push id=3 objects=21 duration=4.614761ms32132026/11/12 00:15:21 INFO Completed upload id=4 objects=7 duration=2.395262ms32142026/11/12 00:15:21 INFO Received complete upload request method=POST path=/32152026/11/12 00:15:21 INFO Received uploads request method=POST path=/32162026/11/12 00:15:21 INFO Completed upload id=9 objects=3 duration=1.315172ms32172026/11/12 00:15:21 INFO Received uploads request method=POST path=/32182026/11/12 00:15:21 INFO Received uploads request method=POST path=/32192026/11/12 00:15:21 INFO Received push request method=POST path=/32202026/11/12 00:15:21 INFO Received complete push request method=POST path=/32212026/11/12 00:15:21 INFO Received push request method=POST path=/32222026/11/12 00:15:21 INFO Completed push id=8 objects=3 duration=1.660955ms32232026/11/12 00:15:21 INFO Received uploads request method=POST path=/32242026/11/12 00:15:21 INFO Received uploads request method=POST path=/32252026/11/12 00:15:21 INFO Received push request method=POST path=/32262026/11/12 00:15:21 INFO Received push request method=POST path=/32272026/11/12 00:15:21 INFO Received push request method=POST path=/32282026/11/12 00:15:21 INFO Received complete upload request method=POST path=/32292026/11/12 00:15:21 INFO Completed upload id=7 objects=4 duration=1.242771ms32302026/11/12 00:15:21 INFO Received uploads request method=POST path=/32312026/11/12 00:15:21 INFO Received uploads request method=POST path=/32322026/11/12 00:15:21 INFO Received push request method=POST path=/32332026/11/12 00:15:21 INFO Received push request method=POST path=/32342026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=032352026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032362026/11/12 00:15:21 INFO Vacuumed table table=pending_closures32372026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=032382026/11/12 00:15:21 INFO Received uploads request method=POST path=/32392026/11/12 00:15:21 INFO Received push request method=POST path=/32402026/11/12 00:15:21 INFO Vacuumed table table=pending_objects32412026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads32422026/11/12 00:15:21 INFO Vacuumed table table=closures32432026/11/12 00:15:21 INFO Vacuumed table table=objects32442026/11/12 00:15:21 INFO Received push request method=POST path=/32452026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032462026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=032472026/11/12 00:15:21 INFO Received uploads request method=POST path=/32482026/11/12 00:15:21 INFO Vacuumed table table=pending_closures32492026/11/12 00:15:21 INFO Received uploads request method=POST path=/32502026/11/12 00:15:21 INFO Vacuumed table table=pending_objects32512026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads32522026/11/12 00:15:21 INFO Received uploads request method=POST path=/32532026/11/12 00:15:21 INFO Vacuumed table table=closures32542026/11/12 00:15:21 INFO Received push request method=POST path=/32552026/11/12 00:15:21 INFO Vacuumed table table=objects32562026/11/12 00:15:21 INFO Received complete push request method=POST path=/32572026/11/12 00:15:21 INFO Received push request method=POST path=/32582026/11/12 00:15:21 INFO Completed push id=7 objects=7 duration=1.355392ms32592026/11/12 00:15:21 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."32602026/11/12 00:15:21 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."32612026/11/12 00:15:21 INFO Received uploads request method=POST path=/32622026/11/12 00:15:21 INFO Received push request method=POST path=/32632026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=032642026/11/12 00:15:21 INFO Received push request method=POST path=/32652026/11/12 00:15:21 INFO Received uploads request method=POST path=/32662026/11/12 00:15:21 INFO Received complete upload request method=POST path=/32672026/11/12 00:15:21 WARN Force mode enabled - objects will be deleted immediately without grace period32682026/11/12 00:15:21 INFO Completed upload id=13 objects=10 duration=882.908µs32692026/11/12 00:15:21 INFO Received push request method=POST path=/32702026/11/12 00:15:21 INFO Received uploads request method=POST path=/32712026/11/12 00:15:21 INFO Received push request method=POST path=/32722026/11/12 00:15:21 INFO Received push request method=POST path=/32732026/11/12 00:15:21 INFO Received complete push request method=POST path=/32742026/11/12 00:15:21 INFO Received uploads request method=POST path=/32752026/11/12 00:15:21 INFO Received uploads request method=POST path=/32762026/11/12 00:15:21 INFO Completed push id=5 objects=8 duration=3.568912ms32772026/11/12 00:15:21 INFO Received uploads request method=POST path=/32782026/11/12 00:15:21 INFO Received uploads request method=POST path=/32792026/11/12 00:15:21 INFO Received push request method=POST path=/32802026/11/12 00:15:21 INFO Received complete upload request method=POST path=/32812026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=032822026/11/12 00:15:21 INFO Completed upload id=10 objects=6 duration=2.698644ms32832026/11/12 00:15:21 WARN Force mode enabled - objects will be deleted immediately without grace period32842026/11/12 00:15:21 INFO Received complete push request method=POST path=/32852026/11/12 00:15:21 INFO Received uploads request method=POST path=/32862026/11/12 00:15:21 INFO Received uploads request method=POST path=/32872026/11/12 00:15:21 INFO Completed push id=1 objects=11 duration=1.793556ms32882026/11/12 00:15:21 INFO Received push request method=POST path=/32892026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=032902026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=032912026/11/12 00:15:21 WARN Force mode enabled - objects will be deleted immediately without grace period32922026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=032932026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032942026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=032952026/11/12 00:15:21 INFO Vacuumed table table=pending_closures32962026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032972026/11/12 00:15:21 INFO Vacuumed table table=pending_objects32982026/11/12 00:15:21 INFO Vacuumed table table=pending_closures32992026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads33002026/11/12 00:15:21 INFO Received uploads request method=POST path=/33012026/11/12 00:15:21 WARN Force mode enabled - objects will be deleted immediately without grace period33022026/11/12 00:15:21 INFO Received complete push request method=POST path=/33032026/11/12 00:15:21 INFO Received complete upload request method=POST path=/33042026/11/12 00:15:21 INFO Vacuumed table table=closures33052026/11/12 00:15:21 INFO Received uploads request method=POST path=/33062026/11/12 00:15:21 INFO Received complete upload request method=POST path=/33072026/11/12 00:15:21 INFO Vacuumed table table=pending_objects33082026/11/12 00:15:21 INFO Vacuumed table table=objects33092026/11/12 00:15:21 INFO Completed push id=12 objects=3 duration=1.823877ms33102026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads33112026/11/12 00:15:21 INFO Completed upload id=15 objects=9 duration=1.831917ms33122026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=033132026/11/12 00:15:21 INFO Completed upload id=18 objects=11 duration=1.257852ms33142026/11/12 00:15:21 INFO Vacuumed table table=closures33152026/11/12 00:15:21 INFO Vacuumed table table=pending_closures33162026/11/12 00:15:21 INFO Vacuumed table table=objects33172026/11/12 00:15:21 INFO Received complete upload request method=POST path=/33182026/11/12 00:15:21 INFO Vacuumed table table=pending_objects33192026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads33202026/11/12 00:15:21 INFO Completed upload id=16 objects=3 duration=1.08091ms33212026/11/12 00:15:21 INFO Vacuumed table table=closures33222026/11/12 00:15:21 INFO Vacuumed table table=objects33232026/11/12 00:15:21 INFO Received complete push request method=POST path=/33242026/11/12 00:15:21 INFO Completed push id=15 objects=5 duration=2.018118ms33252026/11/12 00:15:21 INFO Received uploads request method=POST path=/33262026/11/12 00:15:21 INFO Received uploads request method=POST path=/33272026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=033282026/11/12 00:15:21 INFO Received uploads request method=POST path=/33292026/11/12 00:15:21 INFO Received complete push request method=POST path=/33302026/11/12 00:15:21 INFO Received push request method=POST path=/33312026/11/12 00:15:21 INFO Received push request method=POST path=/33322026/11/12 00:15:21 INFO Received push request method=POST path=/33332026/11/12 00:15:21 INFO Received complete push request method=POST path=/33342026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033352026/11/12 00:15:21 INFO Completed push id=16 objects=4 duration=2.672664ms33362026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=11 objects-failed-to-delete=033372026/11/12 00:15:21 INFO Vacuumed table table=pending_closures33382026/11/12 00:15:21 INFO Completed push id=22 objects=3 duration=2.662804ms33392026/11/12 00:15:21 INFO Vacuumed table table=pending_closures33402026/11/12 00:15:21 INFO Vacuumed table table=pending_objects33412026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads33422026/11/12 00:15:21 INFO Vacuumed table table=closures33432026/11/12 00:15:21 INFO Vacuumed table table=pending_objects33442026/11/12 00:15:21 INFO Vacuumed table table=objects33452026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads33462026/11/12 00:15:21 INFO Vacuumed table table=closures33472026/11/12 00:15:21 INFO Received uploads request method=POST path=/33482026/11/12 00:15:21 INFO Vacuumed table table=objects33492026/11/12 00:15:21 INFO Received uploads request method=POST path=/33502026/11/12 00:15:21 INFO Received push request method=POST path=/33512026/11/12 00:15:21 INFO Received push request method=POST path=/33522026/11/12 00:15:21 INFO Received uploads request method=POST path=/33532026/11/12 00:15:21 INFO Received complete push request method=POST path=/33542026/11/12 00:15:21 INFO Received uploads request method=POST path=/33552026/11/12 00:15:21 INFO Received push request method=POST path=/33562026/11/12 00:15:21 INFO Completed push id=8 objects=6 duration=2.006758ms33572026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=033582026/11/12 00:15:21 WARN Force mode enabled - objects will be deleted immediately without grace period33592026/11/12 00:15:21 INFO Received complete push request method=POST path=/33602026/11/12 00:15:21 INFO Received push request method=POST path=/33612026/11/12 00:15:21 INFO Completed push id=21 objects=17 duration=2.15004ms33622026/11/12 00:15:21 INFO Received uploads request method=POST path=/33632026/11/12 00:15:21 INFO Received uploads request method=POST path=/33642026/11/12 00:15:21 INFO Received complete upload request method=POST path=/33652026/11/12 00:15:21 INFO Received uploads request method=POST path=/33662026/11/12 00:15:21 INFO Received complete push request method=POST path=/33672026/11/12 00:15:21 INFO Completed upload id=20 objects=5 duration=1.11411ms33682026/11/12 00:15:21 INFO Received push request method=POST path=/33692026/11/12 00:15:21 INFO Completed push id=12 objects=7 duration=1.751835ms33702026/11/12 00:15:21 INFO Received uploads request method=POST path=/33712026/11/12 00:15:21 INFO Received uploads request method=POST path=/33722026/11/12 00:15:21 INFO Received complete push request method=POST path=/33732026/11/12 00:15:21 INFO Received push request method=POST path=/33742026/11/12 00:15:21 INFO Completed push id=19 objects=7 duration=1.545273ms33752026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=033762026/11/12 00:15:21 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."33772026/11/12 00:15:21 WARN Force mode enabled - objects will be deleted immediately without grace period33782026/11/12 00:15:21 INFO Received uploads request method=POST path=/33792026/11/12 00:15:21 INFO Received uploads request method=POST path=/33802026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=033812026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=033822026/11/12 00:15:21 WARN Force mode enabled - objects will be deleted immediately without grace period33832026/11/12 00:15:21 INFO Received complete push request method=POST path=/33842026/11/12 00:15:21 INFO Received push request method=POST path=/33852026/11/12 00:15:21 INFO Received complete upload request method=POST path=/33862026/11/12 00:15:21 INFO Received uploads request method=POST path=/33872026/11/12 00:15:21 INFO Completed push id=14 objects=4 duration=2.956327ms33882026/11/12 00:15:21 INFO Completed upload id=16 objects=23 duration=1.433893ms33892026/11/12 00:15:21 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."33902026/11/12 00:15:21 INFO Received push request method=POST path=/33912026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=4 objects-failed-to-delete=033922026/11/12 00:15:21 INFO Received push request method=POST path=/33932026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=033942026/11/12 00:15:21 INFO Vacuumed table table=pending_closures33952026/11/12 00:15:21 INFO Vacuumed table table=pending_objects33962026/11/12 00:15:21 INFO Vacuumed table table=pending_closures33972026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads33982026/11/12 00:15:21 INFO Vacuumed table table=closures33992026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=034002026/11/12 00:15:21 INFO Vacuumed table table=pending_objects34012026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads34022026/11/12 00:15:21 INFO Vacuumed table table=objects34032026/11/12 00:15:21 INFO Vacuumed table table=closures34042026/11/12 00:15:21 INFO Received complete upload request method=POST path=/34052026/11/12 00:15:21 INFO Vacuumed table table=objects34062026/11/12 00:15:21 WARN Force mode enabled - objects will be deleted immediately without grace period34072026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=034082026/11/12 00:15:21 INFO Completed upload id=15 objects=6 duration=2.022978ms34092026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034102026/11/12 00:15:21 INFO Vacuumed table table=pending_closures34112026/11/12 00:15:21 INFO Received uploads request method=POST path=/34122026/11/12 00:15:21 INFO Vacuumed table table=pending_objects34132026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads34142026/11/12 00:15:21 WARN Force mode enabled - objects will be deleted immediately without grace period34152026/11/12 00:15:21 INFO Received complete upload request method=POST path=/34162026/11/12 00:15:21 INFO Vacuumed table table=closures34172026/11/12 00:15:21 INFO Received push request method=POST path=/34182026/11/12 00:15:21 INFO Received complete upload request method=POST path=/34192026/11/12 00:15:21 INFO Vacuumed table table=objects34202026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034212026/11/12 00:15:21 INFO Received push request method=POST path=/34222026/11/12 00:15:21 INFO Vacuumed table table=pending_closures34232026/11/12 00:15:21 INFO Completed upload id=8 objects=3 duration=3.147908ms34242026/11/12 00:15:21 INFO Completed upload id=10 objects=6 duration=2.583483ms34252026/11/12 00:15:21 INFO Vacuumed table table=pending_objects34262026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads34272026/11/12 00:15:21 INFO Received uploads request method=POST path=/34282026/11/12 00:15:21 INFO Vacuumed table table=closures34292026/11/12 00:15:21 INFO Vacuumed table table=objects34302026/11/12 00:15:21 INFO Received uploads request method=POST path=/34312026/11/12 00:15:21 INFO Received uploads request method=POST path=/34322026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=034332026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=034342026/11/12 00:15:21 INFO Received uploads request method=POST path=/34352026/11/12 00:15:21 INFO Received uploads request method=POST path=/34362026/11/12 00:15:21 WARN Force mode enabled - objects will be deleted immediately without grace period34372026/11/12 00:15:21 INFO Received complete push request method=POST path=/34382026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034392026/11/12 00:15:21 INFO Received complete push request method=POST path=/34402026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034412026/11/12 00:15:21 INFO Received push request method=POST path=/34422026/11/12 00:15:21 INFO Vacuumed table table=pending_closures34432026/11/12 00:15:21 INFO Completed push id=15 objects=4 duration=1.485233ms34442026/11/12 00:15:21 INFO Vacuumed table table=pending_closures34452026/11/12 00:15:21 INFO Completed push id=4 objects=6 duration=1.880237ms34462026/11/12 00:15:21 INFO Vacuumed table table=pending_objects34472026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads34482026/11/12 00:15:21 INFO Vacuumed table table=pending_objects34492026/11/12 00:15:21 INFO Vacuumed table table=closures34502026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads34512026/11/12 00:15:21 INFO Vacuumed table table=objects34522026/11/12 00:15:21 INFO Vacuumed table table=closures34532026/11/12 00:15:21 INFO Received uploads request method=POST path=/34542026/11/12 00:15:21 INFO Vacuumed table table=objects34552026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034562026/11/12 00:15:21 INFO Vacuumed table table=pending_closures34572026/11/12 00:15:21 INFO Vacuumed table table=pending_objects34582026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads34592026/11/12 00:15:21 INFO Vacuumed table table=closures34602026/11/12 00:15:21 INFO Received complete upload request method=POST path=/34612026/11/12 00:15:21 INFO Vacuumed table table=objects34622026/11/12 00:15:21 INFO Completed upload id=17 objects=7 duration=1.172891ms34632026/11/12 00:15:21 INFO Received push request method=POST path=/34642026/11/12 00:15:21 INFO Received push request method=POST path=/34652026/11/12 00:15:21 INFO Received uploads request method=POST path=/34662026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=034672026/11/12 00:15:21 INFO Received uploads request method=POST path=/34682026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034692026/11/12 00:15:21 INFO Vacuumed table table=pending_closures34702026/11/12 00:15:21 INFO Vacuumed table table=pending_objects34712026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=034722026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads34732026/11/12 00:15:21 INFO Vacuumed table table=closures34742026/11/12 00:15:21 INFO Vacuumed table table=objects34752026/11/12 00:15:21 INFO Received uploads request method=POST path=/34762026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=034772026/11/12 00:15:21 INFO Received uploads request method=POST path=/34782026/11/12 00:15:21 INFO Received uploads request method=POST path=/34792026/11/12 00:15:21 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."34802026/11/12 00:15:21 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."34812026/11/12 00:15:21 INFO Received push request method=POST path=/34822026/11/12 00:15:21 INFO Received uploads request method=POST path=/34832026/11/12 00:15:21 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."34842026/11/12 00:15:21 INFO Received complete upload request method=POST path=/34852026/11/12 00:15:21 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."34862026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=034872026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=034882026/11/12 00:15:21 INFO Received complete push request method=POST path=/34892026/11/12 00:15:21 INFO Completed upload id=12 objects=26 duration=3.121788ms34902026/11/12 00:15:21 WARN Force mode enabled - objects will be deleted immediately without grace period34912026/11/12 00:15:21 WARN Force mode enabled - objects will be deleted immediately without grace period34922026/11/12 00:15:21 INFO Completed push id=6 objects=6 duration=1.953577ms34932026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034942026/11/12 00:15:21 INFO Received complete upload request method=POST path=/34952026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034962026/11/12 00:15:21 INFO Vacuumed table table=pending_closures34972026/11/12 00:15:21 INFO Vacuumed table table=pending_closures34982026/11/12 00:15:21 INFO Vacuumed table table=pending_objects34992026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads35002026/11/12 00:15:21 INFO Completed upload id=9 objects=7 duration=2.311421ms35012026/11/12 00:15:21 INFO Received uploads request method=POST path=/35022026/11/12 00:15:21 INFO Vacuumed table table=closures35032026/11/12 00:15:21 INFO Vacuumed table table=pending_objects35042026/11/12 00:15:21 INFO Vacuumed table table=objects35052026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads35062026/11/12 00:15:21 INFO Received complete push request method=POST path=/35072026/11/12 00:15:21 INFO Vacuumed table table=closures35082026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=035092026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=035102026/11/12 00:15:21 INFO Vacuumed table table=objects35112026/11/12 00:15:21 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."35122026/11/12 00:15:21 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.narinfo error="Access Denied."35132026/11/12 00:15:21 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."35142026/11/12 00:15:21 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."35152026/11/12 00:15:21 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."35162026/11/12 00:15:21 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."35172026/11/12 00:15:21 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."35182026/11/12 00:15:21 INFO Completed push id=9 objects=4 duration=3.913795ms35192026/11/12 00:15:21 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."35202026/11/12 00:15:21 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."35212026/11/12 00:15:21 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."35222026/11/12 00:15:21 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."35232026/11/12 00:15:21 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."35242026/11/12 00:15:21 ERROR failed to remove object object=nar/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.nar.zst error="Access Denied."35252026/11/12 00:15:21 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."35262026/11/12 00:15:21 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."35272026/11/12 00:15:21 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."35282026/11/12 00:15:21 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."35292026/11/12 00:15:21 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."35302026/11/12 00:15:21 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."35312026/11/12 00:15:21 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."35322026/11/12 00:15:21 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."35332026/11/12 00:15:21 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."35342026/11/12 00:15:21 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."35352026/11/12 00:15:21 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."35362026/11/12 00:15:21 INFO Received push request method=POST path=/35372026/11/12 00:15:21 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."35382026/11/12 00:15:21 WARN Force mode enabled - objects will be deleted immediately without grace period35392026/11/12 00:15:21 WARN Force mode enabled - objects will be deleted immediately without grace period35402026/11/12 00:15:21 INFO Received push request method=POST path=/35412026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035422026/11/12 00:15:21 INFO Vacuumed table table=pending_closures35432026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=035442026/11/12 00:15:21 INFO Vacuumed table table=pending_objects35452026/11/12 00:15:21 WARN Force mode enabled - objects will be deleted immediately without grace period35462026/11/12 00:15:21 INFO Received uploads request method=POST path=/35472026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads35482026/11/12 00:15:21 INFO Received complete push request method=POST path=/35492026/11/12 00:15:21 INFO Vacuumed table table=closures35502026/11/12 00:15:21 INFO Completed push id=14 objects=3 duration=2.778925ms35512026/11/12 00:15:21 INFO Vacuumed table table=objects35522026/11/12 00:15:21 INFO Received uploads request method=POST path=/35532026/11/12 00:15:21 INFO Received complete upload request method=POST path=/35542026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=3 objects-failed-to-delete=035552026/11/12 00:15:21 INFO Received uploads request method=POST path=/35562026/11/12 00:15:21 INFO Completed upload id=14 objects=6 duration=1.882077ms35572026/11/12 00:15:21 INFO Received uploads request method=POST path=/35582026/11/12 00:15:21 INFO Vacuumed table table=pending_closures35592026/11/12 00:15:21 INFO Vacuumed table table=pending_objects35602026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads35612026/11/12 00:15:21 INFO Vacuumed table table=closures35622026/11/12 00:15:21 INFO Vacuumed table table=objects35632026/11/12 00:15:21 INFO Received push request method=POST path=/35642026/11/12 00:15:21 INFO Received uploads request method=POST path=/35652026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=035662026/11/12 00:15:21 INFO Received push request method=POST path=/35672026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035682026/11/12 00:15:21 INFO Vacuumed table table=pending_closures35692026/11/12 00:15:21 INFO Vacuumed table table=pending_objects35702026/11/12 00:15:21 INFO Received complete push request method=POST path=/35712026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads35722026/11/12 00:15:21 INFO Vacuumed table table=closures35732026/11/12 00:15:21 INFO Received complete upload request method=POST path=/35742026/11/12 00:15:21 INFO Received complete push request method=POST path=/35752026/11/12 00:15:21 INFO Vacuumed table table=objects35762026/11/12 00:15:21 INFO Received push request method=POST path=/35772026/11/12 00:15:21 INFO Completed push id=10 objects=15 duration=2.17596ms35782026/11/12 00:15:21 INFO Completed push id=16 objects=8 duration=1.826837ms35792026/11/12 00:15:21 INFO Received uploads request method=POST path=/35802026/11/12 00:15:21 INFO Completed upload id=21 objects=7 duration=2.307301ms35812026/11/12 00:15:21 INFO Received complete push request method=POST path=/35822026/11/12 00:15:21 INFO Received complete upload request method=POST path=/35832026/11/12 00:15:21 INFO Completed upload id=11 objects=6 duration=2.083099ms35842026/11/12 00:15:21 INFO Completed push id=16 objects=16 duration=4.194357ms35852026/11/12 00:15:21 INFO Received complete upload request method=POST path=/35862026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=035872026/11/12 00:15:21 INFO Completed upload id=20 objects=4 duration=859.168µs35882026/11/12 00:15:21 INFO Received uploads request method=POST path=/35892026/11/12 00:15:21 INFO Received complete push request method=POST path=/35902026/11/12 00:15:21 INFO Received complete upload request method=POST path=/35912026/11/12 00:15:21 INFO Received push request method=POST path=/35922026/11/12 00:15:21 INFO Completed upload id=7 objects=3 duration=900.228µs35932026/11/12 00:15:21 INFO Received push request method=POST path=/35942026/11/12 00:15:21 INFO Completed push id=14 objects=3 duration=2.763365ms35952026/11/12 00:15:21 INFO Received uploads request method=POST path=/35962026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=035972026/11/12 00:15:21 WARN Force mode enabled - objects will be deleted immediately without grace period35982026/11/12 00:15:21 INFO Received push request method=POST path=/35992026/11/12 00:15:21 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."36002026/11/12 00:15:21 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."36012026/11/12 00:15:21 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."36022026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036032026/11/12 00:15:21 INFO Vacuumed table table=pending_closures36042026/11/12 00:15:21 INFO Vacuumed table table=pending_objects36052026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads36062026/11/12 00:15:21 INFO Vacuumed table table=closures36072026/11/12 00:15:21 INFO Vacuumed table table=objects36082026/11/12 00:15:21 INFO Received uploads request method=POST path=/36092026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=036102026/11/12 00:15:21 INFO Received push request method=POST path=/36112026/11/12 00:15:21 INFO Received push request method=POST path=/36122026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=036132026/11/12 00:15:21 WARN Force mode enabled - objects will be deleted immediately without grace period36142026/11/12 00:15:21 INFO Received complete push request method=POST path=/36152026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036162026/11/12 00:15:21 INFO Vacuumed table table=pending_closures36172026/11/12 00:15:21 INFO Completed push id=21 objects=9 duration=2.811305ms36182026/11/12 00:15:21 INFO Vacuumed table table=pending_objects36192026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads36202026/11/12 00:15:21 INFO Received complete push request method=POST path=/36212026/11/12 00:15:21 INFO Received complete push request method=POST path=/36222026/11/12 00:15:21 INFO Vacuumed table table=closures36232026/11/12 00:15:21 INFO Completed push id=16 objects=5 duration=1.049409ms36242026/11/12 00:15:21 INFO Vacuumed table table=objects36252026/11/12 00:15:21 INFO Completed push id=18 objects=23 duration=2.051119ms36262026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=036272026/11/12 00:15:21 WARN Force mode enabled - objects will be deleted immediately without grace period36282026/11/12 00:15:21 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."36292026/11/12 00:15:21 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."36302026/11/12 00:15:21 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."36312026/11/12 00:15:21 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."36322026/11/12 00:15:21 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."36332026/11/12 00:15:21 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."36342026/11/12 00:15:21 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."36352026/11/12 00:15:21 INFO Received push request method=POST path=/36362026/11/12 00:15:21 INFO Received complete push request method=POST path=/36372026/11/12 00:15:21 INFO Received push request method=POST path=/36382026/11/12 00:15:21 INFO Completed push id=11 objects=6 duration=2.483002ms36392026/11/12 00:15:21 INFO Received uploads request method=POST path=/36402026/11/12 00:15:21 INFO Received uploads request method=POST path=/36412026/11/12 00:15:21 INFO Received uploads request method=POST path=/36422026/11/12 00:15:21 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."36432026/11/12 00:15:21 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."36442026/11/12 00:15:21 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."36452026/11/12 00:15:21 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."36462026/11/12 00:15:21 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."36472026/11/12 00:15:21 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."36482026/11/12 00:15:21 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."36492026/11/12 00:15:21 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."36502026/11/12 00:15:21 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."36512026/11/12 00:15:21 INFO Received complete push request method=POST path=/36522026/11/12 00:15:21 INFO Received uploads request method=POST path=/36532026/11/12 00:15:21 INFO Completed push id=9 objects=5 duration=2.144599ms36542026/11/12 00:15:21 INFO Received uploads request method=POST path=/36552026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=036562026/11/12 00:15:21 INFO Received complete push request method=POST path=/36572026/11/12 00:15:21 INFO Received uploads request method=POST path=/36582026/11/12 00:15:21 INFO Received complete push request method=POST path=/36592026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=036602026/11/12 00:15:21 WARN Force mode enabled - objects will be deleted immediately without grace period36612026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=036622026/11/12 00:15:21 INFO Completed push id=19 objects=8 duration=3.640493ms36632026/11/12 00:15:21 INFO Completed push id=10 objects=2 duration=2.26406ms36642026/11/12 00:15:21 WARN Force mode enabled - objects will be deleted immediately without grace period36652026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=036662026/11/12 00:15:21 INFO Received complete push request method=POST path=/36672026/11/12 00:15:21 WARN Force mode enabled - objects will be deleted immediately without grace period36682026/11/12 00:15:21 INFO Received uploads request method=POST path=/36692026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=036702026/11/12 00:15:21 INFO Completed push id=18 objects=6 duration=1.609674ms36712026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=036722026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036732026/11/12 00:15:21 INFO Vacuumed table table=pending_closures36742026/11/12 00:15:21 INFO Received uploads request method=POST path=/36752026/11/12 00:15:21 INFO Vacuumed table table=pending_closures36762026/11/12 00:15:21 INFO Received complete push request method=POST path=/36772026/11/12 00:15:21 INFO Vacuumed table table=pending_closures36782026/11/12 00:15:21 INFO Vacuumed table table=pending_objects36792026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads36802026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=036812026/11/12 00:15:21 INFO Vacuumed table table=pending_objects36822026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads36832026/11/12 00:15:21 INFO Vacuumed table table=closures36842026/11/12 00:15:21 INFO Completed push id=11 objects=9 duration=1.927597ms36852026/11/12 00:15:21 INFO Vacuumed table table=pending_objects36862026/11/12 00:15:21 INFO Vacuumed table table=closures36872026/11/12 00:15:21 WARN Force mode enabled - objects will be deleted immediately without grace period36882026/11/12 00:15:21 INFO Received uploads request method=POST path=/36892026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads36902026/11/12 00:15:21 INFO Vacuumed table table=objects36912026/11/12 00:15:21 INFO Vacuumed table table=closures36922026/11/12 00:15:21 INFO Vacuumed table table=objects36932026/11/12 00:15:21 INFO Vacuumed table table=objects36942026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036952026/11/12 00:15:21 INFO Received complete push request method=POST path=/36962026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=9 objects-failed-to-delete=036972026/11/12 00:15:21 INFO Vacuumed table table=pending_closures36982026/11/12 00:15:21 INFO Vacuumed table table=pending_closures36992026/11/12 00:15:21 INFO Completed push id=17 objects=22 duration=2.529182ms37002026/11/12 00:15:21 INFO Vacuumed table table=pending_objects37012026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads37022026/11/12 00:15:21 INFO Vacuumed table table=closures37032026/11/12 00:15:21 INFO Vacuumed table table=pending_objects37042026/11/12 00:15:21 INFO Received complete upload request method=POST path=/37052026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads37062026/11/12 00:15:21 INFO Vacuumed table table=objects37072026/11/12 00:15:21 INFO Vacuumed table table=closures37082026/11/12 00:15:21 INFO Vacuumed table table=objects37092026/11/12 00:15:21 INFO Completed upload id=12 objects=21 duration=1.524613ms37102026/11/12 00:15:21 INFO Received push request method=POST path=/37112026/11/12 00:15:21 INFO Received complete upload request method=POST path=/37122026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=037132026/11/12 00:15:21 INFO Completed upload id=27 objects=4 duration=873.688µs37142026/11/12 00:15:21 INFO Received push request method=POST path=/37152026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=037162026/11/12 00:15:21 INFO Received push request method=POST path=/37172026/11/12 00:15:21 INFO Aborted multipart uploads count=0 kept=037182026/11/12 00:15:21 INFO Received push request method=POST path=/37192026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037202026/11/12 00:15:21 INFO Vacuumed table table=pending_closures37212026/11/12 00:15:21 INFO Vacuumed table table=pending_objects37222026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads37232026/11/12 00:15:21 INFO Vacuumed table table=closures37242026/11/12 00:15:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037252026/11/12 00:15:21 INFO Received push request method=POST path=/37262026/11/12 00:15:21 INFO Vacuumed table table=objects37272026/11/12 00:15:21 INFO Vacuumed table table=pending_closures37282026/11/12 00:15:21 INFO Received push request method=POST path=/37292026/11/12 00:15:21 INFO Vacuumed table table=pending_objects37302026/11/12 00:15:21 INFO Vacuumed table table=multipart_uploads37312026/11/12 00:15:21 INFO Vacuumed table table=closures37322026/11/12 00:15:21 INFO Received complete push request method=POST path=/37332026/11/12 00:15:21 INFO Vacuumed table table=objects37342026/11/12 00:15:21 INFO Completed push id=13 objects=11 duration=1.755856ms37352026/11/12 00:15:21 INFO Received uploads request method=POST path=/37362026/11/12 00:15:21 INFO Received push request method=POST path=/37372026/11/12 00:15:21 INFO Received push request method=POST path=/37382026/11/12 00:15:21 INFO Received uploads request method=POST path=/37392026/11/12 00:15:21 INFO Received uploads request method=POST path=/37402026/11/12 00:15:21 INFO Received push request method=POST path=/37412026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=037422026/11/12 00:15:22 INFO Received push request method=POST path=/37432026/11/12 00:15:22 WARN Force mode enabled - objects will be deleted immediately without grace period37442026/11/12 00:15:22 INFO Received uploads request method=POST path=/37452026/11/12 00:15:22 INFO Received complete push request method=POST path=/37462026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037472026/11/12 00:15:22 INFO Received push request method=POST path=/37482026/11/12 00:15:22 INFO Vacuumed table table=pending_closures37492026/11/12 00:15:22 INFO Completed push id=13 objects=23 duration=1.744676ms37502026/11/12 00:15:22 INFO Vacuumed table table=pending_objects37512026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads37522026/11/12 00:15:22 INFO Received uploads request method=POST path=/37532026/11/12 00:15:22 INFO Vacuumed table table=closures37542026/11/12 00:15:22 INFO Received uploads request method=POST path=/37552026/11/12 00:15:22 INFO Vacuumed table table=objects37562026/11/12 00:15:22 INFO Received uploads request method=POST path=/37572026/11/12 00:15:22 INFO Received push request method=POST path=/37582026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=037592026/11/12 00:15:22 INFO Received uploads request method=POST path=/37602026/11/12 00:15:22 INFO Received push request method=POST path=/37612026/11/12 00:15:22 WARN Force mode enabled - objects will be deleted immediately without grace period37622026/11/12 00:15:22 INFO Received push request method=POST path=/37632026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=037642026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=037652026/11/12 00:15:22 WARN Force mode enabled - objects will be deleted immediately without grace period37662026/11/12 00:15:22 WARN Force mode enabled - objects will be deleted immediately without grace period37672026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037682026/11/12 00:15:22 INFO Vacuumed table table=pending_closures37692026/11/12 00:15:22 INFO Vacuumed table table=pending_objects37702026/11/12 00:15:22 INFO Received push request method=POST path=/37712026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads37722026/11/12 00:15:22 INFO Vacuumed table table=closures37732026/11/12 00:15:22 INFO Vacuumed table table=objects37742026/11/12 00:15:22 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."37752026/11/12 00:15:22 INFO Received complete upload request method=POST path=/37762026/11/12 00:15:22 INFO Completed upload id=7 objects=10 duration=1.654615ms37772026/11/12 00:15:22 INFO Received complete push request method=POST path=/37782026/11/12 00:15:22 INFO Received uploads request method=POST path=/37792026/11/12 00:15:22 INFO Completed push id=31 objects=14 duration=2.500462ms37802026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=037812026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037822026/11/12 00:15:22 INFO Vacuumed table table=pending_closures37832026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=037842026/11/12 00:15:22 INFO Vacuumed table table=pending_objects37852026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads37862026/11/12 00:15:22 INFO Vacuumed table table=closures37872026/11/12 00:15:22 INFO Vacuumed table table=objects37882026/11/12 00:15:22 WARN Force mode enabled - objects will be deleted immediately without grace period37892026/11/12 00:15:22 INFO Received uploads request method=POST path=/37902026/11/12 00:15:22 INFO Received uploads request method=POST path=/37912026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=037922026/11/12 00:15:22 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."37932026/11/12 00:15:22 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."37942026/11/12 00:15:22 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."37952026/11/12 00:15:22 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."37962026/11/12 00:15:22 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."37972026/11/12 00:15:22 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."37982026/11/12 00:15:22 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."37992026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=038002026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=038012026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038022026/11/12 00:15:22 INFO Vacuumed table table=pending_closures38032026/11/12 00:15:22 INFO Vacuumed table table=pending_closures38042026/11/12 00:15:22 INFO Vacuumed table table=pending_objects38052026/11/12 00:15:22 INFO Vacuumed table table=pending_objects38062026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads38072026/11/12 00:15:22 INFO Received uploads request method=POST path=/38082026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads38092026/11/12 00:15:22 INFO Vacuumed table table=closures38102026/11/12 00:15:22 INFO Vacuumed table table=closures38112026/11/12 00:15:22 INFO Vacuumed table table=objects38122026/11/12 00:15:22 INFO Received complete upload request method=POST path=/38132026/11/12 00:15:22 INFO Vacuumed table table=objects38142026/11/12 00:15:22 INFO Received uploads request method=POST path=/38152026/11/12 00:15:22 INFO Completed upload id=14 objects=8 duration=1.888397ms38162026/11/12 00:15:22 INFO Received push request method=POST path=/38172026/11/12 00:15:22 INFO Received push request method=POST path=/38182026/11/12 00:15:22 INFO Received push request method=POST path=/38192026/11/12 00:15:22 INFO Received complete upload request method=POST path=/38202026/11/12 00:15:22 INFO Completed upload id=21 objects=3 duration=905.088µs38212026/11/12 00:15:22 INFO Received complete push request method=POST path=/38222026/11/12 00:15:22 INFO Completed push id=13 objects=12 duration=3.630293ms38232026/11/12 00:15:22 INFO Received push request method=POST path=/38242026/11/12 00:15:22 INFO Received uploads request method=POST path=/38252026/11/12 00:15:22 INFO Received complete upload request method=POST path=/38262026/11/12 00:15:22 INFO Completed upload id=13 objects=8 duration=1.404053ms38272026/11/12 00:15:22 INFO Received complete upload request method=POST path=/38282026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=038292026/11/12 00:15:22 WARN Force mode enabled - objects will be deleted immediately without grace period38302026/11/12 00:15:22 INFO Completed upload id=23 objects=15 duration=2.544263ms38312026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=038322026/11/12 00:15:22 INFO Received push request method=POST path=/38332026/11/12 00:15:22 INFO Received uploads request method=POST path=/38342026/11/12 00:15:22 INFO Received complete push request method=POST path=/38352026/11/12 00:15:22 INFO Received uploads request method=POST path=/38362026/11/12 00:15:22 INFO Received complete push request method=POST path=/38372026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=7 objects-failed-to-delete=038382026/11/12 00:15:22 INFO Received complete upload request method=POST path=/38392026/11/12 00:15:22 INFO Completed push id=7 objects=7 duration=1.518373ms38402026/11/12 00:15:22 INFO Completed push id=15 objects=24 duration=3.898595ms38412026/11/12 00:15:22 INFO Completed upload id=11 objects=7 duration=1.516994ms38422026/11/12 00:15:22 INFO Vacuumed table table=pending_closures38432026/11/12 00:15:22 INFO Received uploads request method=POST path=/38442026/11/12 00:15:22 INFO Vacuumed table table=pending_objects38452026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=038462026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads38472026/11/12 00:15:22 INFO Vacuumed table table=closures38482026/11/12 00:15:22 INFO Vacuumed table table=pending_closures38492026/11/12 00:15:22 INFO Received uploads request method=POST path=/38502026/11/12 00:15:22 INFO Vacuumed table table=objects38512026/11/12 00:15:22 INFO Vacuumed table table=pending_objects38522026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads38532026/11/12 00:15:22 INFO Vacuumed table table=closures38542026/11/12 00:15:22 INFO Vacuumed table table=objects38552026/11/12 00:15:22 INFO Received uploads request method=POST path=/38562026/11/12 00:15:22 INFO Received uploads request method=POST path=/38572026/11/12 00:15:22 INFO Received complete push request method=POST path=/38582026/11/12 00:15:22 INFO Received uploads request method=POST path=/38592026/11/12 00:15:22 INFO Completed push id=12 objects=6 duration=1.224492ms38602026/11/12 00:15:22 INFO Received uploads request method=POST path=/38612026/11/12 00:15:22 INFO Received push request method=POST path=/38622026/11/12 00:15:22 INFO Received complete upload request method=POST path=/38632026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=038642026/11/12 00:15:22 INFO Completed upload id=19 objects=6 duration=1.17211ms38652026/11/12 00:15:22 WARN Force mode enabled - objects will be deleted immediately without grace period38662026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=038672026/11/12 00:15:22 INFO Received push request method=POST path=/38682026/11/12 00:15:22 INFO Vacuumed table table=pending_closures38692026/11/12 00:15:22 INFO Received push request method=POST path=/38702026/11/12 00:15:22 INFO Vacuumed table table=pending_objects38712026/11/12 00:15:22 INFO Received complete upload request method=POST path=/38722026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads38732026/11/12 00:15:22 INFO Received uploads request method=POST path=/38742026/11/12 00:15:22 INFO Received uploads request method=POST path=/38752026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=038762026/11/12 00:15:22 INFO Vacuumed table table=closures38772026/11/12 00:15:22 INFO Vacuumed table table=objects38782026/11/12 00:15:22 INFO Completed upload id=15 objects=4 duration=2.391101ms38792026/11/12 00:15:22 WARN Force mode enabled - objects will be deleted immediately without grace period38802026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038812026/11/12 00:15:22 INFO Vacuumed table table=pending_closures38822026/11/12 00:15:22 INFO Vacuumed table table=pending_objects38832026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads38842026/11/12 00:15:22 INFO Vacuumed table table=closures38852026/11/12 00:15:22 INFO Received complete upload request method=POST path=/38862026/11/12 00:15:22 INFO Vacuumed table table=objects38872026/11/12 00:15:22 INFO Completed upload id=22 objects=7 duration=1.12431ms38882026/11/12 00:15:22 INFO Received complete upload request method=POST path=/38892026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=038902026/11/12 00:15:22 INFO Completed upload id=14 objects=16 duration=2.296261ms38912026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038922026/11/12 00:15:22 INFO Vacuumed table table=pending_closures38932026/11/12 00:15:22 INFO Received push request method=POST path=/38942026/11/12 00:15:22 INFO Vacuumed table table=pending_objects38952026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads38962026/11/12 00:15:22 INFO Vacuumed table table=closures38972026/11/12 00:15:22 INFO Received push request method=POST path=/38982026/11/12 00:15:22 INFO Vacuumed table table=objects38992026/11/12 00:15:22 INFO Received complete push request method=POST path=/39002026/11/12 00:15:22 INFO Completed push id=13 objects=22 duration=1.281171ms39012026/11/12 00:15:22 INFO Received uploads request method=POST path=/39022026/11/12 00:15:22 INFO Received uploads request method=POST path=/39032026/11/12 00:15:22 INFO Received uploads request method=POST path=/39042026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=039052026/11/12 00:15:22 WARN Force mode enabled - objects will be deleted immediately without grace period39062026/11/12 00:15:22 INFO Received complete push request method=POST path=/39072026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039082026/11/12 00:15:22 INFO Vacuumed table table=pending_closures39092026/11/12 00:15:22 INFO Completed push id=22 objects=3 duration=2.432442ms39102026/11/12 00:15:22 INFO Vacuumed table table=pending_objects39112026/11/12 00:15:22 INFO Received complete push request method=POST path=/39122026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads39132026/11/12 00:15:22 INFO Received complete push request method=POST path=/39142026/11/12 00:15:22 INFO Vacuumed table table=closures39152026/11/12 00:15:22 INFO Vacuumed table table=objects39162026/11/12 00:15:22 INFO Completed push id=5 objects=28 duration=2.085658ms39172026/11/12 00:15:22 INFO Completed push id=19 objects=6 duration=3.157568ms39182026/11/12 00:15:22 INFO Received complete push request method=POST path=/39192026/11/12 00:15:22 INFO Completed push id=21 objects=6 duration=1.325052ms39202026/11/12 00:15:22 INFO Received complete upload request method=POST path=/39212026/11/12 00:15:22 INFO Completed upload id=16 objects=3 duration=1.030269ms39222026/11/12 00:15:22 INFO Received uploads request method=POST path=/39232026/11/12 00:15:22 INFO Received complete upload request method=POST path=/39242026/11/12 00:15:22 INFO Received complete upload request method=POST path=/39252026/11/12 00:15:22 INFO Completed upload id=6 objects=6 duration=1.299892ms39262026/11/12 00:15:22 INFO Received complete push request method=POST path=/39272026/11/12 00:15:22 INFO Completed upload id=19 objects=2 duration=2.420782ms39282026/11/12 00:15:22 INFO Received complete push request method=POST path=/39292026/11/12 00:15:22 INFO Received complete upload request method=POST path=/39302026/11/12 00:15:22 INFO Received complete push request method=POST path=/39312026/11/12 00:15:22 INFO Completed push id=24 objects=9 duration=1.766656ms39322026/11/12 00:15:22 INFO Completed push id=10 objects=2 duration=1.514694ms39332026/11/12 00:15:22 INFO Completed push id=17 objects=11 duration=2.079539ms39342026/11/12 00:15:22 INFO Completed upload id=16 objects=17 duration=2.700525ms39352026/11/12 00:15:22 INFO Received complete upload request method=POST path=/39362026/11/12 00:15:22 INFO Completed upload id=22 objects=7 duration=1.000109ms39372026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=039382026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=039392026/11/12 00:15:22 INFO Received uploads request method=POST path=/39402026/11/12 00:15:22 WARN Force mode enabled - objects will be deleted immediately without grace period39412026/11/12 00:15:22 INFO Received push request method=POST path=/39422026/11/12 00:15:22 INFO Received push request method=POST path=/39432026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039442026/11/12 00:15:22 INFO Vacuumed table table=pending_closures39452026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=039462026/11/12 00:15:22 INFO Received complete push request method=POST path=/39472026/11/12 00:15:22 INFO Vacuumed table table=pending_objects39482026/11/12 00:15:22 INFO Completed push id=24 objects=4 duration=1.058749ms39492026/11/12 00:15:22 WARN Force mode enabled - objects will be deleted immediately without grace period39502026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads39512026/11/12 00:15:22 INFO Vacuumed table table=closures39522026/11/12 00:15:22 INFO Received push request method=POST path=/39532026/11/12 00:15:22 INFO Vacuumed table table=objects39542026/11/12 00:15:22 INFO Received complete upload request method=POST path=/39552026/11/12 00:15:22 INFO Completed upload id=17 objects=5 duration=1.202291ms39562026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=039572026/11/12 00:15:22 INFO Received complete push request method=POST path=/39582026/11/12 00:15:22 INFO Completed push id=33 objects=7 duration=1.639555ms39592026/11/12 00:15:22 INFO Received push request method=POST path=/39602026/11/12 00:15:22 INFO Received push request method=POST path=/39612026/11/12 00:15:22 WARN Force mode enabled - objects will be deleted immediately without grace period39622026/11/12 00:15:22 INFO Received complete push request method=POST path=/39632026/11/12 00:15:22 INFO Completed push id=25 objects=9 duration=1.049049ms39642026/11/12 00:15:22 INFO Received complete push request method=POST path=/39652026/11/12 00:15:22 INFO Received uploads request method=POST path=/39662026/11/12 00:15:22 INFO Completed push id=24 objects=3 duration=1.16077ms39672026/11/12 00:15:22 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."39682026/11/12 00:15:22 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."39692026/11/12 00:15:22 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."39702026/11/12 00:15:22 INFO Received complete push request method=POST path=/39712026/11/12 00:15:22 INFO Received complete upload request method=POST path=/39722026/11/12 00:15:22 INFO Completed push id=13 objects=15 duration=1.979237ms39732026/11/12 00:15:22 INFO Completed upload id=26 objects=3 duration=949.969µs39742026/11/12 00:15:22 INFO Received complete push request method=POST path=/39752026/11/12 00:15:22 INFO Completed push id=19 objects=3 duration=1.16223ms39762026/11/12 00:15:22 INFO Received push request method=POST path=/39772026/11/12 00:15:22 INFO Received complete upload request method=POST path=/39782026/11/12 00:15:22 INFO Completed upload id=26 objects=3 duration=859.348µs39792026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=039802026/11/12 00:15:22 WARN Force mode enabled - objects will be deleted immediately without grace period39812026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=039822026/11/12 00:15:22 WARN Force mode enabled - objects will be deleted immediately without grace period39832026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=039842026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039852026/11/12 00:15:22 INFO Vacuumed table table=pending_closures39862026/11/12 00:15:22 INFO Received complete upload request method=POST path=/39872026/11/12 00:15:22 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."39882026/11/12 00:15:22 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."39892026/11/12 00:15:22 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."39902026/11/12 00:15:22 INFO Completed upload id=26 objects=6 duration=1.343272ms39912026/11/12 00:15:22 INFO Vacuumed table table=pending_objects39922026/11/12 00:15:22 INFO Received uploads request method=POST path=/39932026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads39942026/11/12 00:15:22 INFO Vacuumed table table=closures39952026/11/12 00:15:22 INFO Vacuumed table table=objects39962026/11/12 00:15:22 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."39972026/11/12 00:15:22 INFO Received complete upload request method=POST path=/39982026/11/12 00:15:22 INFO Completed upload id=18 objects=4 duration=1.986638ms39992026/11/12 00:15:22 INFO Received complete upload request method=POST path=/40002026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=8 objects-deleted-after-grace-period=8 objects-failed-to-delete=040012026/11/12 00:15:22 INFO Completed upload id=27 objects=7 duration=1.11525ms40022026/11/12 00:15:22 INFO Vacuumed table table=pending_closures40032026/11/12 00:15:22 INFO Vacuumed table table=pending_objects40042026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads40052026/11/12 00:15:22 INFO Received uploads request method=POST path=/40062026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=040072026/11/12 00:15:22 INFO Received complete upload request method=POST path=/40082026/11/12 00:15:22 INFO Received complete push request method=POST path=/40092026/11/12 00:15:22 INFO Vacuumed table table=closures40102026/11/12 00:15:22 INFO Completed push id=17 objects=6 duration=32.311791ms40112026/11/12 00:15:22 INFO Completed upload id=28 objects=3 duration=43.278709ms40122026/11/12 00:15:22 WARN Force mode enabled - objects will be deleted immediately without grace period40132026/11/12 00:15:22 INFO Vacuumed table table=objects40142026/11/12 00:15:22 INFO Received complete push request method=POST path=/40152026/11/12 00:15:22 INFO Completed push id=20 objects=4 duration=1.321491ms40162026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040172026/11/12 00:15:22 INFO Vacuumed table table=pending_closures40182026/11/12 00:15:22 INFO Vacuumed table table=pending_objects40192026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads40202026/11/12 00:15:22 INFO Vacuumed table table=closures40212026/11/12 00:15:22 INFO Vacuumed table table=objects40222026/11/12 00:15:22 INFO Received complete push request method=POST path=/40232026/11/12 00:15:22 INFO Completed push id=20 objects=3 duration=1.781416ms40242026/11/12 00:15:22 INFO Received complete upload request method=POST path=/40252026/11/12 00:15:22 INFO Completed upload id=29 objects=5 duration=1.261652ms40262026/11/12 00:15:22 INFO Received complete push request method=POST path=/40272026/11/12 00:15:22 INFO Completed push id=29 objects=23 duration=1.287452ms40282026/11/12 00:15:22 INFO Received complete upload request method=POST path=/40292026/11/12 00:15:22 INFO Received push request method=POST path=/40302026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=040312026/11/12 00:15:22 INFO Completed upload id=22 objects=8 duration=2.357121ms40322026/11/12 00:15:22 WARN Force mode enabled - objects will be deleted immediately without grace period40332026/11/12 00:15:22 INFO Received complete push request method=POST path=/40342026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040352026/11/12 00:15:22 INFO Completed push id=15 objects=7 duration=2.26586ms40362026/11/12 00:15:22 INFO Received complete push request method=POST path=/40372026/11/12 00:15:22 INFO Vacuumed table table=pending_closures40382026/11/12 00:15:22 INFO Vacuumed table table=pending_objects40392026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads40402026/11/12 00:15:22 INFO Vacuumed table table=closures40412026/11/12 00:15:22 INFO Received complete push request method=POST path=/40422026/11/12 00:15:22 INFO Vacuumed table table=objects40432026/11/12 00:15:22 INFO Completed push id=9 objects=9 duration=2.934847ms40442026/11/12 00:15:22 INFO Completed push id=30 objects=6 duration=1.16663ms40452026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=040462026/11/12 00:15:22 WARN Force mode enabled - objects will be deleted immediately without grace period40472026/11/12 00:15:22 INFO Received complete push request method=POST path=/40482026/11/12 00:15:22 INFO Completed push id=23 objects=5 duration=952.868µs40492026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040502026/11/12 00:15:22 INFO Received uploads request method=POST path=/40512026/11/12 00:15:22 INFO Received complete push request method=POST path=/40522026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=040532026/11/12 00:15:22 INFO Vacuumed table table=pending_closures40542026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=040552026/11/12 00:15:22 INFO Received complete upload request method=POST path=/40562026/11/12 00:15:22 INFO Completed push id=21 objects=3 duration=1.165511ms40572026/11/12 00:15:22 INFO Vacuumed table table=pending_objects40582026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads40592026/11/12 00:15:22 INFO Vacuumed table table=closures40602026/11/12 00:15:22 INFO Completed upload id=15 objects=3 duration=2.066079ms40612026/11/12 00:15:22 INFO Vacuumed table table=objects40622026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040632026/11/12 00:15:22 INFO Vacuumed table table=pending_closures40642026/11/12 00:15:22 INFO Received complete upload request method=POST path=/40652026/11/12 00:15:22 INFO Vacuumed table table=pending_objects40662026/11/12 00:15:22 INFO Completed upload id=21 objects=8 duration=1.290332ms40672026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads40682026/11/12 00:15:22 INFO Vacuumed table table=closures40692026/11/12 00:15:22 INFO Received complete push request method=POST path=/40702026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=040712026/11/12 00:15:22 INFO Vacuumed table table=objects40722026/11/12 00:15:22 INFO Completed push id=33 objects=3 duration=1.267371ms40732026/11/12 00:15:22 INFO Vacuumed table table=pending_closures40742026/11/12 00:15:22 INFO Received complete upload request method=POST path=/40752026/11/12 00:15:22 INFO Completed upload id=24 objects=8 duration=975.408µs40762026/11/12 00:15:22 INFO Vacuumed table table=pending_objects40772026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads40782026/11/12 00:15:22 INFO Vacuumed table table=closures40792026/11/12 00:15:22 INFO Vacuumed table table=objects40802026/11/12 00:15:22 INFO Received uploads request method=POST path=/40812026/11/12 00:15:22 INFO Received uploads request method=POST path=/40822026/11/12 00:15:22 INFO Received push request method=POST path=/40832026/11/12 00:15:22 INFO Received push request method=POST path=/40842026/11/12 00:15:22 INFO Received complete push request method=POST path=/40852026/11/12 00:15:22 INFO Completed push id=22 objects=6 duration=1.322352ms40862026/11/12 00:15:22 INFO Received push request method=POST path=/40872026/11/12 00:15:22 INFO Received complete push request method=POST path=/40882026/11/12 00:15:22 INFO Completed push id=34 objects=6 duration=921.648µs40892026/11/12 00:15:22 INFO Received complete upload request method=POST path=/40902026/11/12 00:15:22 INFO Completed upload id=28 objects=5 duration=1.186731ms40912026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040922026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040932026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040942026/11/12 00:15:22 INFO Vacuumed table table=pending_closures40952026/11/12 00:15:22 INFO Vacuumed table table=pending_closures40962026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040972026/11/12 00:15:22 INFO Vacuumed table table=pending_closures40982026/11/12 00:15:22 INFO Vacuumed table table=pending_objects40992026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads41002026/11/12 00:15:22 INFO Vacuumed table table=pending_objects41012026/11/12 00:15:22 INFO Vacuumed table table=pending_objects41022026/11/12 00:15:22 INFO Vacuumed table table=pending_closures41032026/11/12 00:15:22 INFO Vacuumed table table=closures41042026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041052026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads41062026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads41072026/11/12 00:15:22 INFO Vacuumed table table=pending_objects41082026/11/12 00:15:22 INFO Vacuumed table table=objects41092026/11/12 00:15:22 INFO Vacuumed table table=pending_closures41102026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads41112026/11/12 00:15:22 INFO Vacuumed table table=closures41122026/11/12 00:15:22 INFO Vacuumed table table=closures41132026/11/12 00:15:22 INFO Vacuumed table table=pending_objects41142026/11/12 00:15:22 INFO Received complete push request method=POST path=/41152026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=5 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041162026/11/12 00:15:22 INFO Vacuumed table table=closures41172026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads41182026/11/12 00:15:22 INFO Received push request method=POST path=/41192026/11/12 00:15:22 INFO Vacuumed table table=objects41202026/11/12 00:15:22 INFO Vacuumed table table=objects41212026/11/12 00:15:22 INFO Vacuumed table table=closures41222026/11/12 00:15:22 INFO Vacuumed table table=objects41232026/11/12 00:15:22 INFO Vacuumed table table=objects41242026/11/12 00:15:22 INFO Completed push id=24 objects=8 duration=1.642694ms41252026/11/12 00:15:22 INFO Received uploads request method=POST path=/41262026/11/12 00:15:22 INFO Vacuumed table table=pending_closures41272026/11/12 00:15:22 INFO Vacuumed table table=pending_objects41282026/11/12 00:15:22 INFO Received complete push request method=POST path=/41292026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads41302026/11/12 00:15:22 INFO Completed push id=35 objects=8 duration=1.346772ms41312026/11/12 00:15:22 INFO Vacuumed table table=closures41322026/11/12 00:15:22 INFO Received uploads request method=POST path=/41332026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=041342026/11/12 00:15:22 INFO Vacuumed table table=objects41352026/11/12 00:15:22 INFO Received complete upload request method=POST path=/41362026/11/12 00:15:22 INFO Completed upload id=30 objects=5 duration=1.08969ms41372026/11/12 00:15:22 INFO Received complete upload request method=POST path=/41382026/11/12 00:15:22 INFO Received push request method=POST path=/41392026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=3 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041402026/11/12 00:15:22 INFO Completed upload id=25 objects=9 duration=1.304392ms41412026/11/12 00:15:22 INFO Received push request method=POST path=/41422026/11/12 00:15:22 INFO Received complete upload request method=POST path=/41432026/11/12 00:15:22 INFO Received uploads request method=POST path=/41442026/11/12 00:15:22 INFO Received complete upload request method=POST path=/41452026/11/12 00:15:22 INFO Vacuumed table table=pending_closures41462026/11/12 00:15:22 INFO Received push request method=POST path=/41472026/11/12 00:15:22 INFO Received push request method=POST path=/41482026/11/12 00:15:22 INFO Completed upload id=26 objects=2 duration=2.172959ms41492026/11/12 00:15:22 INFO Completed upload id=20 objects=3 duration=2.308361ms41502026/11/12 00:15:22 INFO Vacuumed table table=pending_objects41512026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads41522026/11/12 00:15:22 INFO Vacuumed table table=closures41532026/11/12 00:15:22 INFO Received complete upload request method=POST path=/41542026/11/12 00:15:22 INFO Vacuumed table table=objects41552026/11/12 00:15:22 INFO Completed upload id=36 objects=9 duration=984.988µs41562026/11/12 00:15:22 INFO Received complete upload request method=POST path=/41572026/11/12 00:15:22 INFO Completed upload id=27 objects=3 duration=937.908µs41582026/11/12 00:15:22 INFO Received uploads request method=POST path=/41592026/11/12 00:15:22 INFO Received uploads request method=POST path=/41602026/11/12 00:15:22 INFO Received complete push request method=POST path=/41612026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=041622026/11/12 00:15:22 INFO Received complete push request method=POST path=/41632026/11/12 00:15:22 INFO Received complete upload request method=POST path=/41642026/11/12 00:15:22 INFO Received complete upload request method=POST path=/41652026/11/12 00:15:22 INFO Completed push id=8 objects=6 duration=3.273429ms41662026/11/12 00:15:22 INFO Completed push id=31 objects=3 duration=1.509053ms41672026/11/12 00:15:22 INFO Completed upload id=22 objects=6 duration=1.15167ms41682026/11/12 00:15:22 INFO Received push request method=POST path=/41692026/11/12 00:15:22 INFO Completed upload id=22 objects=2 duration=1.498893ms41702026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041712026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=041722026/11/12 00:15:22 INFO Received push request method=POST path=/41732026/11/12 00:15:22 INFO Vacuumed table table=pending_closures41742026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=041752026/11/12 00:15:22 WARN Force mode enabled - objects will be deleted immediately without grace period41762026/11/12 00:15:22 INFO Received complete push request method=POST path=/41772026/11/12 00:15:22 INFO Vacuumed table table=pending_objects41782026/11/12 00:15:22 WARN Force mode enabled - objects will be deleted immediately without grace period41792026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads41802026/11/12 00:15:22 INFO Vacuumed table table=closures41812026/11/12 00:15:22 INFO Completed push id=17 objects=22 duration=1.871617ms41822026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041832026/11/12 00:15:22 INFO Vacuumed table table=objects41842026/11/12 00:15:22 INFO Received complete upload request method=POST path=/41852026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041862026/11/12 00:15:22 INFO Vacuumed table table=pending_closures41872026/11/12 00:15:22 INFO Received complete upload request method=POST path=/41882026/11/12 00:15:22 INFO Vacuumed table table=pending_objects41892026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads41902026/11/12 00:15:22 INFO Completed upload id=28 objects=6 duration=827.348µs41912026/11/12 00:15:22 INFO Vacuumed table table=pending_closures41922026/11/12 00:15:22 INFO Vacuumed table table=closures41932026/11/12 00:15:22 INFO Completed upload id=31 objects=3 duration=2.442602ms41942026/11/12 00:15:22 INFO Vacuumed table table=objects41952026/11/12 00:15:22 INFO Vacuumed table table=pending_objects41962026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads41972026/11/12 00:15:22 INFO Vacuumed table table=closures41982026/11/12 00:15:22 INFO Vacuumed table table=objects41992026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=042002026/11/12 00:15:22 INFO Received uploads request method=POST path=/42012026/11/12 00:15:22 INFO Received complete push request method=POST path=/42022026/11/12 00:15:22 INFO Received uploads request method=POST path=/42032026/11/12 00:15:22 INFO Completed push id=16 objects=14 duration=2.684404ms42042026/11/12 00:15:22 INFO Received push request method=POST path=/42052026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=042062026/11/12 00:15:22 INFO Received complete push request method=POST path=/42072026/11/12 00:15:22 INFO Vacuumed table table=pending_closures42082026/11/12 00:15:22 INFO Completed push id=30 objects=4 duration=1.044249ms42092026/11/12 00:15:22 INFO Vacuumed table table=pending_objects42102026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads42112026/11/12 00:15:22 INFO Received complete upload request method=POST path=/42122026/11/12 00:15:22 INFO Vacuumed table table=closures42132026/11/12 00:15:22 INFO Received push request method=POST path=/42142026/11/12 00:15:22 INFO Received push request method=POST path=/42152026/11/12 00:15:22 INFO Vacuumed table table=objects42162026/11/12 00:15:22 INFO Completed upload id=22 objects=2 duration=1.06343ms42172026/11/12 00:15:22 INFO Received push request method=POST path=/42182026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=042192026/11/12 00:15:22 INFO Received complete push request method=POST path=/42202026/11/12 00:15:22 INFO Completed push id=32 objects=6 duration=952.348µs42212026/11/12 00:15:22 WARN Force mode enabled - objects will be deleted immediately without grace period42222026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=042232026/11/12 00:15:22 INFO Received push request method=POST path=/42242026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042252026/11/12 00:15:22 INFO Vacuumed table table=pending_closures42262026/11/12 00:15:22 WARN Force mode enabled - objects will be deleted immediately without grace period42272026/11/12 00:15:22 INFO Vacuumed table table=pending_objects42282026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads42292026/11/12 00:15:22 INFO Vacuumed table table=closures42302026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042312026/11/12 00:15:22 INFO Vacuumed table table=objects42322026/11/12 00:15:22 INFO Vacuumed table table=pending_closures42332026/11/12 00:15:22 INFO Received complete push request method=POST path=/42342026/11/12 00:15:22 INFO Received complete upload request method=POST path=/42352026/11/12 00:15:22 INFO Completed upload id=24 objects=7 duration=775.127µs42362026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=042372026/11/12 00:15:22 INFO Vacuumed table table=pending_objects42382026/11/12 00:15:22 INFO Completed push id=31 objects=7 duration=1.369612ms42392026/11/12 00:15:22 INFO Received uploads request method=POST path=/42402026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads42412026/11/12 00:15:22 INFO Received complete push request method=POST path=/42422026/11/12 00:15:22 INFO Vacuumed table table=closures42432026/11/12 00:15:22 WARN Force mode enabled - objects will be deleted immediately without grace period42442026/11/12 00:15:22 INFO Vacuumed table table=objects42452026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=042462026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042472026/11/12 00:15:22 INFO Completed push id=30 objects=4 duration=2.699524ms42482026/11/12 00:15:22 INFO Vacuumed table table=pending_closures42492026/11/12 00:15:22 INFO Received push request method=POST path=/42502026/11/12 00:15:22 INFO Vacuumed table table=pending_objects42512026/11/12 00:15:22 INFO Received complete upload request method=POST path=/42522026/11/12 00:15:22 WARN Force mode enabled - objects will be deleted immediately without grace period42532026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads42542026/11/12 00:15:22 INFO Vacuumed table table=closures42552026/11/12 00:15:22 INFO Received uploads request method=POST path=/42562026/11/12 00:15:22 INFO Completed upload id=33 objects=7 duration=1.09065ms42572026/11/12 00:15:22 INFO Vacuumed table table=objects42582026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042592026/11/12 00:15:22 INFO Vacuumed table table=pending_closures42602026/11/12 00:15:22 INFO Vacuumed table table=pending_objects42612026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads42622026/11/12 00:15:22 INFO Vacuumed table table=closures42632026/11/12 00:15:22 INFO Received complete upload request method=POST path=/42642026/11/12 00:15:22 INFO Vacuumed table table=objects42652026/11/12 00:15:22 INFO Completed upload id=32 objects=5 duration=739.467µs42662026/11/12 00:15:22 INFO Received push request method=POST path=/42672026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=042682026/11/12 00:15:22 INFO Received uploads request method=POST path=/42692026/11/12 00:15:22 INFO Received push request method=POST path=/42702026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042712026/11/12 00:15:22 INFO Received complete upload request method=POST path=/42722026/11/12 00:15:22 INFO Received complete upload request method=POST path=/42732026/11/12 00:15:22 INFO Vacuumed table table=pending_closures42742026/11/12 00:15:22 INFO Completed upload id=34 objects=3 duration=1.031929ms42752026/11/12 00:15:22 INFO Received push request method=POST path=/42762026/11/12 00:15:22 INFO Received push request method=POST path=/42772026/11/12 00:15:22 INFO Vacuumed table table=pending_objects42782026/11/12 00:15:22 INFO Completed upload id=17 objects=25 duration=2.324821ms42792026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=042802026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads42812026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=042822026/11/12 00:15:22 INFO Vacuumed table table=closures42832026/11/12 00:15:22 INFO Vacuumed table table=objects42842026/11/12 00:15:22 INFO Received complete upload request method=POST path=/42852026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042862026/11/12 00:15:22 INFO Received push request method=POST path=/42872026/11/12 00:15:22 INFO Vacuumed table table=pending_closures42882026/11/12 00:15:22 INFO Vacuumed table table=pending_objects42892026/11/12 00:15:22 INFO Completed upload id=18 objects=2 duration=2.097358ms42902026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads42912026/11/12 00:15:22 INFO Vacuumed table table=closures42922026/11/12 00:15:22 INFO Vacuumed table table=objects42932026/11/12 00:15:22 INFO Received push request method=POST path=/42942026/11/12 00:15:22 INFO Received push request method=POST path=/42952026/11/12 00:15:22 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."42962026/11/12 00:15:22 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."42972026/11/12 00:15:22 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."42982026/11/12 00:15:22 INFO Received uploads request method=POST path=/42992026/11/12 00:15:22 INFO Received complete push request method=POST path=/43002026/11/12 00:15:22 INFO Received push request method=POST path=/43012026/11/12 00:15:22 INFO Completed push id=35 objects=11 duration=1.839556ms43022026/11/12 00:15:22 INFO Received push request method=POST path=/43032026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=043042026/11/12 00:15:22 INFO Received complete push request method=POST path=/43052026/11/12 00:15:22 INFO Received push request method=POST path=/43062026/11/12 00:15:22 INFO Received uploads request method=POST path=/43072026/11/12 00:15:22 INFO Completed push id=9 objects=24 duration=1.757796ms43082026/11/12 00:15:22 INFO Received push request method=POST path=/43092026/11/12 00:15:22 INFO Received push request method=POST path=/43102026/11/12 00:15:22 INFO Received push request method=POST path=/43112026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043122026/11/12 00:15:22 INFO Vacuumed table table=pending_closures43132026/11/12 00:15:22 INFO Received push request method=POST path=/43142026/11/12 00:15:22 INFO Vacuumed table table=pending_objects43152026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads43162026/11/12 00:15:22 INFO Vacuumed table table=closures43172026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=043182026/11/12 00:15:22 INFO Vacuumed table table=objects43192026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=043202026/11/12 00:15:22 INFO Received push request method=POST path=/43212026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=043222026/11/12 00:15:22 WARN Force mode enabled - objects will be deleted immediately without grace period43232026/11/12 00:15:22 INFO Received uploads request method=POST path=/43242026/11/12 00:15:22 INFO Received complete upload request method=POST path=/43252026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043262026/11/12 00:15:22 INFO Completed upload id=21 objects=2 duration=939.649µs43272026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043282026/11/12 00:15:22 INFO Vacuumed table table=pending_closures43292026/11/12 00:15:22 WARN Force mode enabled - objects will be deleted immediately without grace period43302026/11/12 00:15:22 INFO Received uploads request method=POST path=/43312026/11/12 00:15:22 INFO Vacuumed table table=pending_closures43322026/11/12 00:15:22 INFO Received complete push request method=POST path=/43332026/11/12 00:15:22 INFO Vacuumed table table=pending_objects43342026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads43352026/11/12 00:15:22 INFO Vacuumed table table=pending_objects43362026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads43372026/11/12 00:15:22 INFO Completed push id=19 objects=14 duration=1.718176ms43382026/11/12 00:15:22 INFO Received push request method=POST path=/43392026/11/12 00:15:22 INFO Vacuumed table table=closures43402026/11/12 00:15:22 INFO Vacuumed table table=closures43412026/11/12 00:15:22 INFO Received complete upload request method=POST path=/43422026/11/12 00:15:22 INFO Vacuumed table table=objects43432026/11/12 00:15:22 INFO Vacuumed table table=objects43442026/11/12 00:15:22 INFO Received uploads request method=POST path=/43452026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=043462026/11/12 00:15:22 INFO Completed upload id=34 objects=24 duration=2.177059ms43472026/11/12 00:15:22 WARN Force mode enabled - objects will be deleted immediately without grace period43482026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043492026/11/12 00:15:22 INFO Received uploads request method=POST path=/43502026/11/12 00:15:22 INFO Vacuumed table table=pending_closures43512026/11/12 00:15:22 INFO Vacuumed table table=pending_objects43522026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads43532026/11/12 00:15:22 INFO Received uploads request method=POST path=/43542026/11/12 00:15:22 INFO Vacuumed table table=closures43552026/11/12 00:15:22 INFO Received complete push request method=POST path=/43562026/11/12 00:15:22 INFO Received complete upload request method=POST path=/43572026/11/12 00:15:22 INFO Vacuumed table table=objects43582026/11/12 00:15:22 INFO Received uploads request method=POST path=/43592026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=043602026/11/12 00:15:22 INFO Received uploads request method=POST path=/43612026/11/12 00:15:22 INFO Completed upload id=13 objects=7 duration=1.318872ms43622026/11/12 00:15:22 INFO Completed push id=32 objects=7 duration=2.149679ms43632026/11/12 00:15:22 WARN Force mode enabled - objects will be deleted immediately without grace period43642026/11/12 00:15:22 INFO Received push request method=POST path=/43652026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=043662026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043672026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=043682026/11/12 00:15:22 INFO Received complete upload request method=POST path=/43692026/11/12 00:15:22 INFO Vacuumed table table=pending_closures43702026/11/12 00:15:22 WARN Force mode enabled - objects will be deleted immediately without grace period43712026/11/12 00:15:22 INFO Completed upload id=25 objects=6 duration=1.357553ms43722026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=043732026/11/12 00:15:22 INFO Vacuumed table table=pending_objects43742026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads43752026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043762026/11/12 00:15:22 WARN Force mode enabled - objects will be deleted immediately without grace period43772026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043782026/11/12 00:15:22 INFO Vacuumed table table=closures43792026/11/12 00:15:22 INFO Vacuumed table table=pending_closures43802026/11/12 00:15:22 INFO Vacuumed table table=pending_closures43812026/11/12 00:15:22 INFO Vacuumed table table=objects43822026/11/12 00:15:22 INFO Vacuumed table table=pending_objects43832026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043842026/11/12 00:15:22 INFO Vacuumed table table=pending_objects43852026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads43862026/11/12 00:15:22 INFO Received push request method=POST path=/43872026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads43882026/11/12 00:15:22 INFO Vacuumed table table=closures43892026/11/12 00:15:22 INFO Vacuumed table table=closures43902026/11/12 00:15:22 INFO Vacuumed table table=pending_closures43912026/11/12 00:15:22 INFO Vacuumed table table=objects43922026/11/12 00:15:22 INFO Vacuumed table table=objects43932026/11/12 00:15:22 INFO Vacuumed table table=pending_objects43942026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads43952026/11/12 00:15:22 INFO Received uploads request method=POST path=/43962026/11/12 00:15:22 INFO Received complete upload request method=POST path=/43972026/11/12 00:15:22 INFO Vacuumed table table=closures43982026/11/12 00:15:22 INFO Completed upload id=20 objects=4 duration=1.17629ms43992026/11/12 00:15:22 INFO Vacuumed table table=objects44002026/11/12 00:15:22 INFO Received complete push request method=POST path=/44012026/11/12 00:15:22 INFO Received complete upload request method=POST path=/44022026/11/12 00:15:22 INFO Completed push id=26 objects=4 duration=937.068µs44032026/11/12 00:15:22 INFO Received uploads request method=POST path=/44042026/11/12 00:15:22 INFO Completed upload id=10 objects=10 duration=976.269µs44052026/11/12 00:15:22 INFO Received complete push request method=POST path=/44062026/11/12 00:15:22 INFO Received push request method=POST path=/44072026/11/12 00:15:22 INFO Completed push id=23 objects=28 duration=3.393371ms44082026/11/12 00:15:22 INFO Received complete upload request method=POST path=/44092026/11/12 00:15:22 INFO Received complete push request method=POST path=/44102026/11/12 00:15:22 INFO Received push request method=POST path=/44112026/11/12 00:15:22 INFO Completed upload id=20 objects=6 duration=1.19485ms44122026/11/12 00:15:22 INFO Received uploads request method=POST path=/44132026/11/12 00:15:22 INFO Received complete upload request method=POST path=/44142026/11/12 00:15:22 INFO Completed push id=28 objects=11 duration=2.271441ms44152026/11/12 00:15:22 INFO Received uploads request method=POST path=/44162026/11/12 00:15:22 INFO Completed upload id=23 objects=7 duration=1.13807ms44172026/11/12 00:15:22 INFO Received complete upload request method=POST path=/44182026/11/12 00:15:22 INFO Received push request method=POST path=/44192026/11/12 00:15:22 INFO Completed upload id=28 objects=13 duration=1.07093ms44202026/11/12 00:15:22 INFO Received push request method=POST path=/44212026/11/12 00:15:22 INFO Received push request method=POST path=/44222026/11/12 00:15:22 INFO Received uploads request method=POST path=/44232026/11/12 00:15:22 INFO Received complete push request method=POST path=/44242026/11/12 00:15:22 INFO Completed push id=29 objects=3 duration=1.04199ms44252026/11/12 00:15:22 INFO Received uploads request method=POST path=/44262026/11/12 00:15:22 INFO Received uploads request method=POST path=/44272026/11/12 00:15:22 INFO Received complete push request method=POST path=/44282026/11/12 00:15:22 INFO Completed push id=26 objects=14 duration=2.655084ms44292026/11/12 00:15:22 INFO Received uploads request method=POST path=/44302026/11/12 00:15:22 INFO Received complete upload request method=POST path=/44312026/11/12 00:15:22 INFO Received uploads request method=POST path=/44322026/11/12 00:15:22 INFO Received push request method=POST path=/44332026/11/12 00:15:22 INFO Received complete push request method=POST path=/44342026/11/12 00:15:22 INFO Completed upload id=27 objects=8 duration=975.229µs44352026/11/12 00:15:22 INFO Received complete upload request method=POST path=/44362026/11/12 00:15:22 INFO Completed upload id=31 objects=2 duration=1.264271ms44372026/11/12 00:15:22 INFO Received push request method=POST path=/44382026/11/12 00:15:22 INFO Completed push id=7 objects=24 duration=3.595413ms44392026/11/12 00:15:22 INFO Received complete push request method=POST path=/44402026/11/12 00:15:22 INFO Received complete push request method=POST path=/44412026/11/12 00:15:22 INFO Completed push id=29 objects=24 duration=1.548314ms44422026/11/12 00:15:22 INFO Completed push id=11 objects=9 duration=1.489733ms44432026/11/12 00:15:22 INFO Received uploads request method=POST path=/44442026/11/12 00:15:22 INFO Received push request method=POST path=/44452026/11/12 00:15:22 INFO Received complete upload request method=POST path=/44462026/11/12 00:15:22 INFO Received complete upload request method=POST path=/44472026/11/12 00:15:22 INFO Completed upload id=24 objects=18 duration=1.324232ms44482026/11/12 00:15:22 INFO Completed upload id=23 objects=16 duration=2.220741ms44492026/11/12 00:15:22 INFO Received push request method=POST path=/44502026/11/12 00:15:22 INFO Received complete upload request method=POST path=/44512026/11/12 00:15:22 INFO Received complete push request method=POST path=/44522026/11/12 00:15:22 INFO Completed push id=33 objects=6 duration=708.847µs44532026/11/12 00:15:22 INFO Received uploads request method=POST path=/44542026/11/12 00:15:22 INFO Completed upload id=23 objects=18 duration=2.655604ms44552026/11/12 00:15:22 INFO Received push request method=POST path=/44562026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=044572026/11/12 00:15:22 WARN Force mode enabled - objects will be deleted immediately without grace period44582026/11/12 00:15:22 INFO Received complete upload request method=POST path=/44592026/11/12 00:15:22 INFO Received complete push request method=POST path=/44602026/11/12 00:15:22 INFO Completed upload id=21 objects=2 duration=956.149µs44612026/11/12 00:15:22 INFO Completed push id=15 objects=24 duration=1.383112ms44622026/11/12 00:15:22 INFO Received complete push request method=POST path=/44632026/11/12 00:15:22 INFO Received uploads request method=POST path=/44642026/11/12 00:15:22 INFO Completed push id=25 objects=2 duration=2.179639ms44652026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=044662026/11/12 00:15:22 WARN Force mode enabled - objects will be deleted immediately without grace period44672026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=044682026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044692026/11/12 00:15:22 INFO Vacuumed table table=pending_closures44702026/11/12 00:15:22 INFO Vacuumed table table=pending_closures44712026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=044722026/11/12 00:15:22 INFO Vacuumed table table=pending_objects44732026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=044742026/11/12 00:15:22 INFO Received uploads request method=POST path=/44752026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads44762026/11/12 00:15:22 INFO Vacuumed table table=pending_objects44772026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads44782026/11/12 00:15:22 INFO Received uploads request method=POST path=/44792026/11/12 00:15:22 INFO Vacuumed table table=closures44802026/11/12 00:15:22 INFO Received uploads request method=POST path=/44812026/11/12 00:15:22 INFO Vacuumed table table=closures44822026/11/12 00:15:22 WARN Force mode enabled - objects will be deleted immediately without grace period44832026/11/12 00:15:22 INFO Vacuumed table table=objects44842026/11/12 00:15:22 INFO Vacuumed table table=objects44852026/11/12 00:15:22 INFO Received complete upload request method=POST path=/44862026/11/12 00:15:22 INFO Received uploads request method=POST path=/44872026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044882026/11/12 00:15:22 INFO Completed upload id=19 objects=2 duration=1.326472ms44892026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044902026/11/12 00:15:22 INFO Received complete push request method=POST path=/44912026/11/12 00:15:22 INFO Vacuumed table table=pending_closures44922026/11/12 00:15:22 INFO Vacuumed table table=pending_closures44932026/11/12 00:15:22 INFO Completed push id=27 objects=4 duration=936.528µs44942026/11/12 00:15:22 INFO Vacuumed table table=pending_objects44952026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads44962026/11/12 00:15:22 INFO Vacuumed table table=closures44972026/11/12 00:15:22 INFO Vacuumed table table=pending_objects44982026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads44992026/11/12 00:15:22 INFO Vacuumed table table=objects45002026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=045012026/11/12 00:15:22 INFO Vacuumed table table=closures45022026/11/12 00:15:22 INFO Vacuumed table table=objects45032026/11/12 00:15:22 INFO Received uploads request method=POST path=/45042026/11/12 00:15:22 INFO Received uploads request method=POST path=/45052026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045062026/11/12 00:15:22 INFO Received push request method=POST path=/45072026/11/12 00:15:22 INFO Received push request method=POST path=/45082026/11/12 00:15:22 INFO Received uploads request method=POST path=/45092026/11/12 00:15:22 INFO Vacuumed table table=pending_closures45102026/11/12 00:15:22 INFO Received complete push request method=POST path=/45112026/11/12 00:15:22 INFO Received complete upload request method=POST path=/45122026/11/12 00:15:22 INFO Vacuumed table table=pending_objects45132026/11/12 00:15:22 INFO Completed push id=16 objects=7 duration=1.679635ms45142026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=045152026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads45162026/11/12 00:15:22 INFO Completed upload id=24 objects=7 duration=939.488µs45172026/11/12 00:15:22 INFO Vacuumed table table=closures45182026/11/12 00:15:22 INFO Vacuumed table table=objects45192026/11/12 00:15:22 WARN Force mode enabled - objects will be deleted immediately without grace period45202026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045212026/11/12 00:15:22 INFO Vacuumed table table=pending_closures45222026/11/12 00:15:22 INFO Vacuumed table table=pending_objects45232026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads45242026/11/12 00:15:22 INFO Vacuumed table table=closures45252026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=045262026/11/12 00:15:22 INFO Vacuumed table table=objects45272026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=045282026/11/12 00:15:22 INFO Received complete push request method=POST path=/45292026/11/12 00:15:22 INFO Received complete upload request method=POST path=/45302026/11/12 00:15:22 INFO Received push request method=POST path=/45312026/11/12 00:15:22 INFO Completed push id=14 objects=3 duration=1.002309ms45322026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045332026/11/12 00:15:22 INFO Completed upload id=22 objects=22 duration=1.130511ms45342026/11/12 00:15:22 WARN Force mode enabled - objects will be deleted immediately without grace period45352026/11/12 00:15:22 INFO Vacuumed table table=pending_closures45362026/11/12 00:15:22 INFO Received complete push request method=POST path=/45372026/11/12 00:15:22 INFO Vacuumed table table=pending_objects45382026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads45392026/11/12 00:15:22 INFO Completed push id=26 objects=3 duration=992.249µs45402026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045412026/11/12 00:15:22 INFO Vacuumed table table=closures45422026/11/12 00:15:22 INFO Received complete push request method=POST path=/45432026/11/12 00:15:22 INFO Vacuumed table table=objects45442026/11/12 00:15:22 INFO Vacuumed table table=pending_closures45452026/11/12 00:15:22 INFO Completed push id=28 objects=26 duration=1.215851ms45462026/11/12 00:15:22 INFO Vacuumed table table=pending_objects45472026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads45482026/11/12 00:15:22 INFO Vacuumed table table=closures45492026/11/12 00:15:22 INFO Vacuumed table table=objects45502026/11/12 00:15:22 INFO Received complete push request method=POST path=/45512026/11/12 00:15:22 INFO Received push request method=POST path=/45522026/11/12 00:15:22 INFO Received push request method=POST path=/45532026/11/12 00:15:22 INFO Received uploads request method=POST path=/45542026/11/12 00:15:22 INFO Completed push id=23 objects=6 duration=1.205911ms45552026/11/12 00:15:22 INFO Received complete upload request method=POST path=/45562026/11/12 00:15:22 INFO Received push request method=POST path=/45572026/11/12 00:15:22 INFO Completed upload id=28 objects=8 duration=1.009269ms45582026/11/12 00:15:22 INFO Received complete push request method=POST path=/45592026/11/12 00:15:22 INFO Completed push id=30 objects=3 duration=974.149µs45602026/11/12 00:15:22 INFO Received complete upload request method=POST path=/45612026/11/12 00:15:22 INFO Completed upload id=25 objects=16 duration=1.928078ms45622026/11/12 00:15:22 INFO Received uploads request method=POST path=/45632026/11/12 00:15:22 INFO Received complete push request method=POST path=/45642026/11/12 00:15:22 INFO Completed push id=24 objects=6 duration=1.630235ms45652026/11/12 00:15:22 INFO Received complete push request method=POST path=/45662026/11/12 00:15:22 INFO Received complete push request method=POST path=/45672026/11/12 00:15:22 INFO Completed push id=26 objects=5 duration=1.841476ms45682026/11/12 00:15:22 INFO Received complete upload request method=POST path=/45692026/11/12 00:15:22 INFO Completed push id=13 objects=2 duration=1.245091ms45702026/11/12 00:15:22 INFO Received uploads request method=POST path=/45712026/11/12 00:15:22 INFO Completed upload id=29 objects=10 duration=953.609µs45722026/11/12 00:15:22 INFO Received push request method=POST path=/45732026/11/12 00:15:22 INFO Received complete push request method=POST path=/45742026/11/12 00:15:22 INFO Completed push id=32 objects=19 duration=1.284651ms45752026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=045762026/11/12 00:15:22 WARN Force mode enabled - objects will be deleted immediately without grace period45772026/11/12 00:15:22 INFO Received push request method=POST path=/45782026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045792026/11/12 00:15:22 INFO Vacuumed table table=pending_closures45802026/11/12 00:15:22 INFO Received complete upload request method=POST path=/45812026/11/12 00:15:22 INFO Vacuumed table table=pending_objects45822026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads45832026/11/12 00:15:22 INFO Received complete upload request method=POST path=/45842026/11/12 00:15:22 INFO Received push request method=POST path=/45852026/11/12 00:15:22 INFO Completed upload id=27 objects=13 duration=1.075309ms45862026/11/12 00:15:22 INFO Vacuumed table table=closures45872026/11/12 00:15:22 INFO Completed upload id=24 objects=14 duration=1.17103ms45882026/11/12 00:15:22 INFO Received complete upload request method=POST path=/45892026/11/12 00:15:22 INFO Vacuumed table table=objects45902026/11/12 00:15:22 INFO Completed upload id=30 objects=20 duration=833.908µs45912026/11/12 00:15:22 INFO Received push request method=POST path=/45922026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=045932026/11/12 00:15:22 WARN Force mode enabled - objects will be deleted immediately without grace period45942026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045952026/11/12 00:15:22 INFO Received complete push request method=POST path=/45962026/11/12 00:15:22 INFO Received complete push request method=POST path=/45972026/11/12 00:15:22 INFO Completed push id=25 objects=7 duration=1.447613ms45982026/11/12 00:15:22 INFO Vacuumed table table=pending_closures45992026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=046002026/11/12 00:15:22 INFO Vacuumed table table=pending_objects46012026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads46022026/11/12 00:15:22 INFO Received complete push request method=POST path=/46032026/11/12 00:15:22 INFO Received complete push request method=POST path=/46042026/11/12 00:15:22 INFO Completed push id=24 objects=24 duration=4.711943ms46052026/11/12 00:15:22 INFO Vacuumed table table=closures46062026/11/12 00:15:22 INFO Completed push id=19 objects=3 duration=1.672796ms46072026/11/12 00:15:22 INFO Completed push id=31 objects=11 duration=1.632375ms46082026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=046092026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=046102026/11/12 00:15:22 INFO Vacuumed table table=objects46112026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046122026/11/12 00:15:22 INFO Vacuumed table table=pending_closures46132026/11/12 00:15:22 WARN Force mode enabled - objects will be deleted immediately without grace period46142026/11/12 00:15:22 INFO Vacuumed table table=pending_objects46152026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads46162026/11/12 00:15:22 INFO Received push request method=POST path=/46172026/11/12 00:15:22 INFO Vacuumed table table=closures46182026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046192026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046202026/11/12 00:15:22 INFO Vacuumed table table=objects46212026/11/12 00:15:22 INFO Vacuumed table table=pending_closures46222026/11/12 00:15:22 INFO Vacuumed table table=pending_closures46232026/11/12 00:15:22 INFO Received complete push request method=POST path=/46242026/11/12 00:15:22 INFO Vacuumed table table=pending_objects46252026/11/12 00:15:22 INFO Vacuumed table table=pending_objects46262026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads46272026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads46282026/11/12 00:15:22 INFO Vacuumed table table=closures46292026/11/12 00:15:22 INFO Vacuumed table table=closures46302026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=046312026/11/12 00:15:22 INFO Vacuumed table table=objects46322026/11/12 00:15:22 INFO Vacuumed table table=objects46332026/11/12 00:15:22 INFO Completed push id=7 objects=15 duration=2.832986ms46342026/11/12 00:15:22 INFO Received complete upload request method=POST path=/46352026/11/12 00:15:22 WARN Force mode enabled - objects will be deleted immediately without grace period46362026/11/12 00:15:22 INFO Completed upload id=32 objects=2 duration=618.725µs46372026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046382026/11/12 00:15:22 INFO Vacuumed table table=pending_closures46392026/11/12 00:15:22 INFO Vacuumed table table=pending_objects46402026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads46412026/11/12 00:15:22 INFO Vacuumed table table=closures46422026/11/12 00:15:22 INFO Vacuumed table table=objects46432026/11/12 00:15:22 INFO Received push request method=POST path=/46442026/11/12 00:15:22 INFO Received uploads request method=POST path=/46452026/11/12 00:15:22 INFO Received complete push request method=POST path=/46462026/11/12 00:15:22 INFO Received push request method=POST path=/46472026/11/12 00:15:22 INFO Received uploads request method=POST path=/46482026/11/12 00:15:22 INFO Received complete upload request method=POST path=/46492026/11/12 00:15:22 INFO Completed push id=33 objects=3 duration=8.273374ms46502026/11/12 00:15:22 INFO Completed upload id=26 objects=24 duration=1.11883ms46512026/11/12 00:15:22 INFO Received complete upload request method=POST path=/46522026/11/12 00:15:22 INFO Completed upload id=33 objects=5 duration=2.407662ms46532026/11/12 00:15:22 INFO Received push request method=POST path=/46542026/11/12 00:15:22 INFO Received push request method=POST path=/46552026/11/12 00:15:22 INFO Received complete upload request method=POST path=/46562026/11/12 00:15:22 INFO Received uploads request method=POST path=/46572026/11/12 00:15:22 INFO Received push request method=POST path=/46582026/11/12 00:15:22 INFO Received uploads request method=POST path=/46592026/11/12 00:15:22 INFO Completed upload id=34 objects=3 duration=777.407µs46602026/11/12 00:15:22 INFO Received push request method=POST path=/46612026/11/12 00:15:22 INFO Received push request method=POST path=/46622026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=046632026/11/12 00:15:22 INFO Received uploads request method=POST path=/46642026/11/12 00:15:22 INFO Received uploads request method=POST path=/46652026/11/12 00:15:22 INFO Received uploads request method=POST path=/46662026/11/12 00:15:22 WARN Force mode enabled - objects will be deleted immediately without grace period46672026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046682026/11/12 00:15:22 INFO Received complete upload request method=POST path=/46692026/11/12 00:15:22 INFO Vacuumed table table=pending_closures46702026/11/12 00:15:22 INFO Received complete push request method=POST path=/46712026/11/12 00:15:22 INFO Completed upload id=27 objects=26 duration=1.10059ms46722026/11/12 00:15:22 INFO Vacuumed table table=pending_objects46732026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads46742026/11/12 00:15:22 INFO Vacuumed table table=closures46752026/11/12 00:15:22 INFO Completed push id=27 objects=25 duration=2.792105ms46762026/11/12 00:15:22 INFO Vacuumed table table=objects46772026/11/12 00:15:22 INFO Received uploads request method=POST path=/46782026/11/12 00:15:22 INFO Received complete upload request method=POST path=/46792026/11/12 00:15:22 INFO Received uploads request method=POST path=/46802026/11/12 00:15:22 INFO Received complete upload request method=POST path=/46812026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=046822026/11/12 00:15:22 INFO Received complete push request method=POST path=/46832026/11/12 00:15:22 INFO Completed upload id=25 objects=4 duration=1.023969ms46842026/11/12 00:15:22 INFO Completed upload id=21 objects=25 duration=2.280381ms46852026/11/12 00:15:22 WARN Force mode enabled - objects will be deleted immediately without grace period46862026/11/12 00:15:22 INFO Completed push id=28 objects=6 duration=1.15801ms46872026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046882026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=046892026/11/12 00:15:22 INFO Vacuumed table table=pending_closures46902026/11/12 00:15:22 INFO Vacuumed table table=pending_objects46912026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads46922026/11/12 00:15:22 INFO Vacuumed table table=closures46932026/11/12 00:15:22 INFO Vacuumed table table=objects46942026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046952026/11/12 00:15:22 INFO Vacuumed table table=pending_closures46962026/11/12 00:15:22 INFO Received complete upload request method=POST path=/46972026/11/12 00:15:22 INFO Vacuumed table table=pending_objects46982026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads46992026/11/12 00:15:22 INFO Completed upload id=22 objects=5 duration=1.396072ms47002026/11/12 00:15:22 INFO Vacuumed table table=closures47012026/11/12 00:15:22 INFO Vacuumed table table=objects47022026/11/12 00:15:22 INFO Received push request method=POST path=/47032026/11/12 00:15:22 INFO Received push request method=POST path=/47042026/11/12 00:15:22 INFO Received uploads request method=POST path=/47052026/11/12 00:15:22 INFO Received uploads request method=POST path=/47062026/11/12 00:15:22 INFO Received complete push request method=POST path=/47072026/11/12 00:15:22 INFO Received complete push request method=POST path=/47082026/11/12 00:15:22 INFO Completed push id=28 objects=3 duration=1.843856ms47092026/11/12 00:15:22 INFO Received uploads request method=POST path=/47102026/11/12 00:15:22 INFO Completed push id=24 objects=5 duration=2.338761ms47112026/11/12 00:15:22 INFO Received complete upload request method=POST path=/47122026/11/12 00:15:22 INFO Completed upload id=29 objects=4 duration=1.540233ms47132026/11/12 00:15:22 INFO Received complete push request method=POST path=/47142026/11/12 00:15:22 INFO Completed push id=29 objects=6 duration=1.040649ms47152026/11/12 00:15:22 INFO Received uploads request method=POST path=/47162026/11/12 00:15:22 INFO Received complete push request method=POST path=/47172026/11/12 00:15:22 INFO Completed push id=29 objects=15 duration=1.15625ms47182026/11/12 00:15:22 INFO Received complete push request method=POST path=/47192026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=047202026/11/12 00:15:22 INFO Received uploads request method=POST path=/47212026/11/12 00:15:22 INFO Completed push id=20 objects=8 duration=889.948µs47222026/11/12 00:15:22 WARN Force mode enabled - objects will be deleted immediately without grace period47232026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047242026/11/12 00:15:22 INFO Vacuumed table table=pending_closures47252026/11/12 00:15:22 INFO Vacuumed table table=pending_objects47262026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads47272026/11/12 00:15:22 INFO Vacuumed table table=closures47282026/11/12 00:15:22 INFO Vacuumed table table=objects47292026/11/12 00:15:22 INFO Received uploads request method=POST path=/47302026/11/12 00:15:22 INFO Received complete upload request method=POST path=/47312026/11/12 00:15:22 INFO Completed upload id=12 objects=6 duration=1.135551ms47322026/11/12 00:15:22 INFO Received complete push request method=POST path=/47332026/11/12 00:15:22 INFO Completed push id=30 objects=6 duration=979.269µs47342026/11/12 00:15:22 INFO Received uploads request method=POST path=/47352026/11/12 00:15:22 INFO Received uploads request method=POST path=/47362026/11/12 00:15:22 INFO Received uploads request method=POST path=/47372026/11/12 00:15:22 INFO Received complete upload request method=POST path=/47382026/11/12 00:15:22 INFO Completed upload id=30 objects=8 duration=963.808µs47392026/11/12 00:15:22 INFO Received complete push request method=POST path=/47402026/11/12 00:15:22 INFO Received complete push request method=POST path=/47412026/11/12 00:15:22 INFO Received complete upload request method=POST path=/47422026/11/12 00:15:22 INFO Completed upload id=31 objects=3 duration=1.457553ms47432026/11/12 00:15:22 INFO Completed push id=19 objects=14 duration=6.854562ms47442026/11/12 00:15:22 INFO Completed push id=17 objects=3 duration=2.817006ms47452026/11/12 00:15:22 INFO Received uploads request method=POST path=/47462026/11/12 00:15:22 INFO Received push request method=POST path=/47472026/11/12 00:15:22 INFO Received complete push request method=POST path=/47482026/11/12 00:15:22 INFO Completed push id=11 objects=6 duration=1.579854ms47492026/11/12 00:15:22 INFO Received push request method=POST path=/47502026/11/12 00:15:22 INFO Received complete upload request method=POST path=/47512026/11/12 00:15:22 INFO Received complete push request method=POST path=/47522026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=047532026/11/12 00:15:22 INFO Completed upload id=31 objects=6 duration=927.629µs47542026/11/12 00:15:22 INFO Completed push id=21 objects=26 duration=1.389072ms47552026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=047562026/11/12 00:15:22 WARN Force mode enabled - objects will be deleted immediately without grace period47572026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047582026/11/12 00:15:22 WARN Force mode enabled - objects will be deleted immediately without grace period47592026/11/12 00:15:22 INFO Received complete push request method=POST path=/47602026/11/12 00:15:22 INFO Vacuumed table table=pending_closures47612026/11/12 00:15:22 INFO Completed push id=23 objects=5 duration=1.219611ms47622026/11/12 00:15:22 INFO Vacuumed table table=pending_objects47632026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads47642026/11/12 00:15:22 INFO Vacuumed table table=closures47652026/11/12 00:15:22 INFO Vacuumed table table=objects47662026/11/12 00:15:22 INFO Received complete push request method=POST path=/47672026/11/12 00:15:22 INFO Completed push id=20 objects=13 duration=1.15441ms47682026/11/12 00:15:22 INFO Received complete upload request method=POST path=/47692026/11/12 00:15:22 INFO Completed upload id=32 objects=4 duration=711.487µs47702026/11/12 00:15:22 INFO Received uploads request method=POST path=/47712026/11/12 00:15:22 INFO Received complete upload request method=POST path=/47722026/11/12 00:15:22 INFO Received complete push request method=POST path=/47732026/11/12 00:15:22 INFO Completed push id=27 objects=11 duration=928.568µs47742026/11/12 00:15:22 INFO Received complete upload request method=POST path=/47752026/11/12 00:15:22 INFO Received complete upload request method=POST path=/47762026/11/12 00:15:22 INFO Completed upload id=23 objects=7 duration=3.27499ms47772026/11/12 00:15:22 INFO Completed upload id=24 objects=3 duration=1.13315ms47782026/11/12 00:15:22 INFO Completed upload id=28 objects=17 duration=1.12719ms47792026/11/12 00:15:22 INFO Received complete push request method=POST path=/47802026/11/12 00:15:22 INFO Completed push id=22 objects=4 duration=1.17721ms47812026/11/12 00:15:22 INFO Received complete upload request method=POST path=/47822026/11/12 00:15:22 INFO Completed upload id=33 objects=10 duration=929.168µs47832026/11/12 00:15:22 INFO Received complete upload request method=POST path=/47842026/11/12 00:15:22 INFO Received complete upload request method=POST path=/47852026/11/12 00:15:22 INFO Completed upload id=29 objects=26 duration=923.048µs47862026/11/12 00:15:22 INFO Received complete push request method=POST path=/47872026/11/12 00:15:22 INFO Completed upload id=26 objects=5 duration=999.208µs47882026/11/12 00:15:22 INFO Completed push id=25 objects=4 duration=1.029769ms47892026/11/12 00:15:22 INFO Received complete push request method=POST path=/47902026/11/12 00:15:22 INFO Completed push id=29 objects=2 duration=1.003089ms47912026/11/12 00:15:22 INFO Received complete push request method=POST path=/47922026/11/12 00:15:22 INFO Received complete upload request method=POST path=/47932026/11/12 00:15:22 INFO Completed push id=18 objects=6 duration=3.117248ms47942026/11/12 00:15:22 INFO Received complete push request method=POST path=/47952026/11/12 00:15:22 INFO Completed upload id=23 objects=7 duration=869.387µs47962026/11/12 00:15:22 INFO Received complete push request method=POST path=/47972026/11/12 00:15:22 INFO Completed push id=30 objects=8 duration=922.388µs47982026/11/12 00:15:22 INFO Completed push id=27 objects=3 duration=1.660095ms47992026/11/12 00:15:22 INFO Received complete push request method=POST path=/48002026/11/12 00:15:22 INFO Completed push id=31 objects=6 duration=1.049229ms48012026/11/12 00:15:22 INFO Received complete upload request method=POST path=/48022026/11/12 00:15:22 INFO Completed upload id=34 objects=23 duration=1.032449ms48032026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=048042026/11/12 00:15:22 INFO Vacuumed table table=pending_closures48052026/11/12 00:15:22 INFO Vacuumed table table=pending_objects48062026/11/12 00:15:22 INFO Received complete upload request method=POST path=/48072026/11/12 00:15:22 INFO Vacuumed table table=multipart_uploads48082026/11/12 00:15:22 INFO Vacuumed table table=closures48092026/11/12 00:15:22 INFO Completed upload id=23 objects=9 duration=989.309µs48102026/11/12 00:15:22 INFO Vacuumed table table=objects48112026/11/12 00:15:22 INFO Received complete upload request method=POST path=/48122026/11/12 00:15:22 INFO Received complete push request method=POST path=/48132026/11/12 00:15:22 INFO Completed upload id=28 objects=5 duration=770.707µs48142026/11/12 00:15:22 INFO Received complete upload request method=POST path=/48152026/11/12 00:15:22 INFO Completed push id=24 objects=5 duration=1.10825ms48162026/11/12 00:15:22 INFO Completed upload id=31 objects=5 duration=858.848µs48172026/11/12 00:15:22 INFO Received complete push request method=POST path=/48182026/11/12 00:15:22 INFO Completed push id=32 objects=9 duration=949.269µs48192026/11/12 00:15:22 INFO Received complete push request method=POST path=/48202026/11/12 00:15:22 INFO Received push request method=POST path=/48212026/11/12 00:15:22 INFO Received complete push request method=POST path=/48222026/11/12 00:15:22 INFO Completed push id=29 objects=17 duration=1.013789ms48232026/11/12 00:15:22 INFO Received complete upload request method=POST path=/48242026/11/12 00:15:22 INFO Received complete push request method=POST path=/48252026/11/12 00:15:22 INFO Received complete push request method=POST path=/48262026/11/12 00:15:22 INFO Completed push id=35 objects=6 duration=1.16785ms48272026/11/12 00:15:22 INFO Completed upload id=25 objects=2 duration=897.048µs48282026/11/12 00:15:22 INFO Completed push id=32 objects=3 duration=951.969µs48292026/11/12 00:15:22 INFO Completed push id=26 objects=23 duration=1.369432ms48302026/11/12 00:15:22 INFO Received complete upload request method=POST path=/48312026/11/12 00:15:22 INFO Completed upload id=33 objects=9 duration=855.308µs48322026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=048332026/11/12 00:15:22 INFO Received push request method=POST path=/48342026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=048352026/11/12 00:15:22 WARN Force mode enabled - objects will be deleted immediately without grace period48362026/11/12 00:15:22 INFO Received uploads request method=POST path=/48372026/11/12 00:15:22 INFO Received complete upload request method=POST path=/48382026/11/12 00:15:22 INFO Completed upload id=26 objects=20 duration=829.467µs48392026/11/12 00:15:22 INFO Aborted multipart uploads count=0 kept=048402026/11/12 00:15:22 WARN Force mode enabled - objects will be deleted immediately without grace period48412026/11/12 00:15:22 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048422026/11/12 00:15:23 INFO Vacuumed table table=pending_closures48432026/11/12 00:15:23 WARN Force mode enabled - objects will be deleted immediately without grace period48442026/11/12 00:15:23 INFO Vacuumed table table=pending_objects48452026/11/12 00:15:23 INFO Vacuumed table table=multipart_uploads48462026/11/12 00:15:23 INFO Vacuumed table table=closures48472026/11/12 00:15:23 INFO Aborted multipart uploads count=0 kept=048482026/11/12 00:15:23 INFO Vacuumed table table=objects48492026/11/12 00:15:23 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048502026/11/12 00:15:23 INFO Vacuumed table table=pending_closures48512026/11/12 00:15:23 WARN Force mode enabled - objects will be deleted immediately without grace period48522026/11/12 00:15:23 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=048532026/11/12 00:15:23 INFO Vacuumed table table=pending_objects48542026/11/12 00:15:23 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048552026/11/12 00:15:23 INFO Vacuumed table table=pending_closures48562026/11/12 00:15:23 INFO Vacuumed table table=multipart_uploads48572026/11/12 00:15:23 INFO Vacuumed table table=closures48582026/11/12 00:15:23 INFO Vacuumed table table=pending_closures48592026/11/12 00:15:23 INFO Vacuumed table table=objects48602026/11/12 00:15:23 INFO Vacuumed table table=pending_objects48612026/11/12 00:15:23 INFO Vacuumed table table=multipart_uploads48622026/11/12 00:15:23 INFO Vacuumed table table=closures48632026/11/12 00:15:23 INFO Received uploads request method=POST path=/48642026/11/12 00:15:23 INFO Vacuumed table table=pending_objects48652026/11/12 00:15:23 INFO Vacuumed table table=objects48662026/11/12 00:15:23 INFO Vacuumed table table=multipart_uploads48672026/11/12 00:15:23 INFO Vacuumed table table=closures48682026/11/12 00:15:23 INFO Vacuumed table table=objects48692026/11/12 00:15:23 INFO Received complete push request method=POST path=/48702026/11/12 00:15:23 INFO Aborted multipart uploads count=0 kept=048712026/11/12 00:15:23 INFO Completed push id=27 objects=15 duration=1.284091ms48722026/11/12 00:15:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048732026/11/12 00:15:23 INFO Received complete push request method=POST path=/48742026/11/12 00:15:23 INFO Vacuumed table table=pending_closures48752026/11/12 00:15:23 INFO Completed push id=28 objects=7 duration=1.007989ms48762026/11/12 00:15:23 INFO Vacuumed table table=pending_objects48772026/11/12 00:15:23 INFO Vacuumed table table=multipart_uploads48782026/11/12 00:15:23 INFO Vacuumed table table=closures48792026/11/12 00:15:23 INFO Vacuumed table table=objects48802026/11/12 00:15:23 INFO Received complete upload request method=POST path=/48812026/11/12 00:15:23 INFO Received push request method=POST path=/48822026/11/12 00:15:23 INFO Completed upload id=30 objects=6 duration=801.447µs48832026/11/12 00:15:23 INFO Garbage collection completed failed-uploads-deleted=6 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048842026/11/12 00:15:23 INFO Received complete push request method=POST path=/48852026/11/12 00:15:23 INFO Vacuumed table table=pending_closures48862026/11/12 00:15:23 INFO Completed push id=28 objects=19 duration=1.436033ms48872026/11/12 00:15:23 INFO Vacuumed table table=pending_objects48882026/11/12 00:15:23 INFO Vacuumed table table=multipart_uploads48892026/11/12 00:15:23 INFO Received complete push request method=POST path=/48902026/11/12 00:15:23 INFO Vacuumed table table=closures48912026/11/12 00:15:23 INFO Completed push id=31 objects=5 duration=966.389µs48922026/11/12 00:15:23 INFO Vacuumed table table=objects48932026/11/12 00:15:23 INFO Received complete push request method=POST path=/48942026/11/12 00:15:23 INFO Completed push id=16 objects=26 duration=1.290191ms48952026/11/12 00:15:23 INFO Received complete push request method=POST path=/48962026/11/12 00:15:23 INFO Received complete push request method=POST path=/48972026/11/12 00:15:23 INFO Completed push id=29 objects=10 duration=1.13709ms48982026/11/12 00:15:23 INFO Completed push id=32 objects=26 duration=1.413372ms48992026/11/12 00:15:23 INFO Received complete upload request method=POST path=/49002026/11/12 00:15:23 INFO Completed upload id=13 objects=14 duration=3.250709ms49012026/11/12 00:15:23 INFO Aborted multipart uploads count=0 kept=049022026/11/12 00:15:23 WARN Force mode enabled - objects will be deleted immediately without grace period49032026/11/12 00:15:23 INFO Received complete upload request method=POST path=/49042026/11/12 00:15:23 INFO Completed upload id=33 objects=2 duration=697.007µs49052026/11/12 00:15:23 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049062026/11/12 00:15:23 INFO Vacuumed table table=pending_closures49072026/11/12 00:15:23 INFO Received complete upload request method=POST path=/49082026/11/12 00:15:23 INFO Completed upload id=30 objects=6 duration=830.067µs49092026/11/12 00:15:23 INFO Vacuumed table table=pending_objects49102026/11/12 00:15:23 INFO Vacuumed table table=multipart_uploads49112026/11/12 00:15:23 INFO Vacuumed table table=closures49122026/11/12 00:15:23 INFO Vacuumed table table=objects49132026/11/12 00:15:23 INFO Aborted multipart uploads count=0 kept=049142026/11/12 00:15:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049152026/11/12 00:15:23 INFO Vacuumed table table=pending_closures49162026/11/12 00:15:23 INFO Vacuumed table table=pending_objects49172026/11/12 00:15:23 INFO Vacuumed table table=multipart_uploads49182026/11/12 00:15:23 INFO Vacuumed table table=closures49192026/11/12 00:15:23 INFO Vacuumed table table=objects49202026/11/12 00:15:23 INFO Received complete upload request method=POST path=/49212026/11/12 00:15:23 INFO Completed upload id=34 objects=5 duration=834.647µs49222026/11/12 00:15:23 INFO Received complete upload request method=POST path=/49232026/11/12 00:15:23 INFO Completed upload id=31 objects=5 duration=737.006µs49242026/11/12 00:15:23 INFO Received complete push request method=POST path=/49252026/11/12 00:15:23 INFO Completed push id=27 objects=6 duration=2.679484ms49262026/11/12 00:15:23 INFO Received complete upload request method=POST path=/49272026/11/12 00:15:23 INFO Completed upload id=35 objects=11 duration=691.206µs49282026/11/12 00:15:23 INFO Received complete upload request method=POST path=/49292026/11/12 00:15:23 INFO Received complete push request method=POST path=/49302026/11/12 00:15:23 INFO Completed upload id=34 objects=3 duration=543.745µs49312026/11/12 00:15:23 INFO Completed push id=31 objects=6 duration=3.431771ms49322026/11/12 00:15:23 INFO Received complete upload request method=POST path=/49332026/11/12 00:15:23 INFO Aborted multipart uploads count=0 kept=049342026/11/12 00:15:23 INFO Completed upload id=36 objects=3 duration=836.368µs49352026/11/12 00:15:23 WARN Force mode enabled - objects will be deleted immediately without grace period49362026/11/12 00:15:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049372026/11/12 00:15:23 INFO Vacuumed table table=pending_closures49382026/11/12 00:15:23 INFO Received uploads request method=POST path=/49392026/11/12 00:15:23 INFO Vacuumed table table=pending_objects49402026/11/12 00:15:23 INFO Vacuumed table table=multipart_uploads49412026/11/12 00:15:23 INFO Vacuumed table table=closures49422026/11/12 00:15:23 INFO Received complete upload request method=POST path=/49432026/11/12 00:15:23 INFO Vacuumed table table=objects49442026/11/12 00:15:23 INFO Completed upload id=15 objects=2 duration=845.307µs49452026/11/12 00:15:23 INFO Aborted multipart uploads count=0 kept=049462026/11/12 00:15:23 WARN Force mode enabled - objects will be deleted immediately without grace period49472026/11/12 00:15:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049482026/11/12 00:15:23 INFO Vacuumed table table=pending_closures49492026/11/12 00:15:23 INFO Vacuumed table table=pending_objects49502026/11/12 00:15:23 INFO Vacuumed table table=multipart_uploads49512026/11/12 00:15:23 INFO Vacuumed table table=closures49522026/11/12 00:15:23 INFO Vacuumed table table=objects49532026/11/12 00:15:23 INFO Received complete push request method=POST path=/49542026/11/12 00:15:23 INFO Received push request method=POST path=/49552026/11/12 00:15:23 INFO Completed push id=1 objects=9 duration=1.376853ms49562026/11/12 00:15:23 INFO Received uploads request method=POST path=/49572026/11/12 00:15:23 INFO Received uploads request method=POST path=/49582026/11/12 00:15:23 INFO Received complete upload request method=POST path=/49592026/11/12 00:15:23 INFO Completed upload id=26 objects=8 duration=960.629µs49602026/11/12 00:15:23 INFO Received complete upload request method=POST path=/49612026/11/12 00:15:23 INFO Completed upload id=2 objects=7 duration=1.015969ms49622026/11/12 00:15:23 INFO Received complete push request method=POST path=/49632026/11/12 00:15:23 INFO Completed push id=28 objects=14 duration=1.165551ms49642026/11/12 00:15:23 INFO Received push request method=POST path=/49652026/11/12 00:15:23 INFO Received complete upload request method=POST path=/49662026/11/12 00:15:23 INFO Completed upload id=8 objects=6 duration=1.192831ms49672026/11/12 00:15:23 INFO Received uploads request method=POST path=/49682026/11/12 00:15:23 INFO Aborted multipart uploads count=0 kept=049692026/11/12 00:15:23 WARN Force mode enabled - objects will be deleted immediately without grace period49702026/11/12 00:15:23 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049712026/11/12 00:15:23 INFO Vacuumed table table=pending_closures49722026/11/12 00:15:23 INFO Vacuumed table table=pending_objects49732026/11/12 00:15:23 INFO Vacuumed table table=multipart_uploads49742026/11/12 00:15:23 INFO Vacuumed table table=closures49752026/11/12 00:15:23 INFO Vacuumed table table=objects49762026/11/12 00:15:23 INFO Received complete upload request method=POST path=/49772026/11/12 00:15:23 INFO Completed upload id=12 objects=9 duration=1.15755ms49782026/11/12 00:15:23 INFO Received uploads request method=POST path=/49792026/11/12 00:15:23 INFO Received complete upload request method=POST path=/49802026/11/12 00:15:23 INFO Completed upload id=21 objects=9 duration=1.573314ms49812026/11/12 00:15:23 INFO Received complete push request method=POST path=/49822026/11/12 00:15:23 INFO Completed push id=25 objects=10 duration=2.091659ms49832026/11/12 00:15:23 INFO Received complete upload request method=POST path=/49842026/11/12 00:15:23 INFO Completed upload id=29 objects=5 duration=887.668µs49852026/11/12 00:15:23 INFO Received complete upload request method=POST path=/49862026/11/12 00:15:23 INFO Completed upload id=30 objects=2 duration=930.848µs49872026/11/12 00:15:23 INFO Received complete upload request method=POST path=/49882026/11/12 00:15:23 INFO Completed upload id=18 objects=22 duration=1.12867ms49892026/11/12 00:15:23 INFO Received complete push request method=POST path=/49902026/11/12 00:15:23 INFO Completed push id=31 objects=20 duration=1.146451ms49912026/11/12 00:15:23 INFO Received complete upload request method=POST path=/49922026/11/12 00:15:23 INFO Completed upload id=21 objects=8 duration=738.827µs49932026/11/12 00:15:23 INFO Received complete upload request method=POST path=/49942026/11/12 00:15:23 INFO Completed upload id=32 objects=5 duration=817.187µs49952026/11/12 00:15:23 INFO Received complete upload request method=POST path=/49962026/11/12 00:15:23 INFO Completed upload id=22 objects=7 duration=6.094934ms49972026/11/12 00:15:23 INFO Received complete upload request method=POST path=/49982026/11/12 00:15:23 INFO Completed upload id=33 objects=3 duration=603.945µs49992026/11/12 00:15:23 INFO Received complete push request method=POST path=/50002026/11/12 00:15:23 INFO Completed push id=24 objects=5 duration=1.220671ms50012026/11/12 00:15:23 INFO Aborted multipart uploads count=0 kept=050022026/11/12 00:15:23 WARN Force mode enabled - objects will be deleted immediately without grace period50032026/11/12 00:15:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=050042026/11/12 00:15:23 INFO Vacuumed table table=pending_closures50052026/11/12 00:15:23 INFO Vacuumed table table=pending_objects50062026/11/12 00:15:23 INFO Vacuumed table table=multipart_uploads50072026/11/12 00:15:23 INFO Vacuumed table table=closures50082026/11/12 00:15:23 INFO Vacuumed table table=objects50092026/11/12 00:15:23 INFO Received complete upload request method=POST path=/50102026/11/12 00:15:23 INFO Completed upload id=25 objects=8 duration=851.367µs50112026/11/12 00:15:23 INFO Received complete push request method=POST path=/50122026/11/12 00:15:23 INFO Completed push id=26 objects=4 duration=888.768µs50132026/11/12 00:15:23 INFO Received complete push request method=POST path=/50142026/11/12 00:15:23 INFO Completed push id=30 objects=2 duration=757.007µs50152026/11/12 00:15:23 INFO Received complete push request method=POST path=/50162026/11/12 00:15:23 INFO Completed push id=32 objects=7 duration=1.432313ms50172026/11/12 00:15:23 INFO Received complete upload request method=POST path=/50182026/11/12 00:15:23 INFO Completed upload id=33 objects=6 duration=597.886µs50192026/11/12 00:15:23 INFO Received complete upload request method=POST path=/50202026/11/12 00:15:23 INFO Completed upload id=34 objects=3 duration=775.147µs50212026/11/12 00:15:23 INFO Aborted multipart uploads count=0 kept=050222026/11/12 00:15:23 WARN Force mode enabled - objects will be deleted immediately without grace period50232026/11/12 00:15:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=050242026/11/12 00:15:23 INFO Vacuumed table table=pending_closures50252026/11/12 00:15:23 INFO Vacuumed table table=pending_objects50262026/11/12 00:15:23 INFO Vacuumed table table=multipart_uploads50272026/11/12 00:15:23 INFO Vacuumed table table=closures50282026/11/12 00:15:23 INFO Vacuumed table table=objects5029--- PASS: FuzzPushGC (0.01s)5030 --- PASS: FuzzPushGC/seed#4 (7.64s)5031 --- PASS: FuzzPushGC/seed#9 (7.74s)5032 --- PASS: FuzzPushGC/seed#2 (7.82s)5033 --- PASS: FuzzPushGC/seed#14 (7.82s)5034 --- PASS: FuzzPushGC/seed#7 (7.91s)5035 --- PASS: FuzzPushGC/seed#5 (7.99s)5036 --- PASS: FuzzPushGC/seed#15 (8.10s)5037 --- PASS: FuzzPushGC/seed#1 (8.18s)5038 --- PASS: FuzzPushGC/seed#3 (8.19s)5039 --- PASS: FuzzPushGC/seed#12 (8.20s)5040 --- PASS: FuzzPushGC/seed#0 (8.20s)5041 --- PASS: FuzzPushGC/seed#8 (8.24s)5042 --- PASS: FuzzPushGC/seed#13 (8.33s)5043 --- PASS: FuzzPushGC/seed#11 (8.35s)5044 --- PASS: FuzzPushGC/seed#6 (8.71s)5045 --- PASS: FuzzPushGC/seed#10 (8.82s)5046PASS5047{"timestamp":"2026-11-12T00:15:23.664961187Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:48388","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(207)"}5048{"timestamp":"2026-11-12T00:15:23.664962607Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:48364","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(208)"}50492026-11-12 00:15:24.162 UTC [128] LOG: received smart shutdown request50502026-11-12 00:15:24.169 UTC [128] LOG: background worker "logical replication launcher" (PID 175) exited with exit code 150512026-11-12 00:15:29.827 UTC [170] LOG: checkpoint complete: wrote 3671 buffers (22.4%), wrote 8 SLRU buffers; 0 WAL file(s) added, 0 removed, 33 recycled; write=18.041 s, sync=0.578 s, total=23.550 s; sync files=42371, longest=0.015 s, average=0.001 s; distance=533119 kB, estimate=533119 kB; lsn=0/26F71CC8, redo lsn=0/22005B9850522026-11-12 00:15:29.834 UTC [170] LOG: shutting down50532026-11-12 00:15:29.834 UTC [170] LOG: checkpoint starting: shutdown immediate50542026-11-12 00:15:30.362 UTC [170] LOG: checkpoint complete: wrote 8278 buffers (50.5%), wrote 2 SLRU buffers; 0 WAL file(s) added, 0 removed, 4 recycled; write=0.182 s, sync=0.335 s, total=0.529 s; sync files=2600, longest=0.075 s, average=0.001 s; distance=81328 kB, estimate=487940 kB; lsn=0/26F71D40, redo lsn=0/26F71D4050552026-11-12 00:15:30.549 UTC [128] LOG: database system is shut down5056Running OIDC tests...5057=== RUN TestAudienceForIssuer5058=== PAUSE TestAudienceForIssuer5059=== RUN TestProviderClientGivesUpOnSilentIssuer5060=== PAUSE TestProviderClientGivesUpOnSilentIssuer5061=== RUN TestGlobMatch5062=== PAUSE TestGlobMatch5063=== RUN TestValidateToken_ValidToken5064=== PAUSE TestValidateToken_ValidToken5065=== RUN TestValidateToken_WrongAudience5066=== PAUSE TestValidateToken_WrongAudience5067=== RUN TestValidateToken_Expired5068=== PAUSE TestValidateToken_Expired5069=== RUN TestValidateToken_BoundClaimsMismatch5070=== PAUSE TestValidateToken_BoundClaimsMismatch5071=== RUN TestValidateToken_BoundSubjectMismatch5072=== PAUSE TestValidateToken_BoundSubjectMismatch5073=== RUN TestValidateToken_MultipleProviders5074=== PAUSE TestValidateToken_MultipleProviders5075=== RUN TestValidateToken_UnknownIssuerFetchesNoKeys5076=== PAUSE TestValidateToken_UnknownIssuerFetchesNoKeys5077=== RUN TestValidateToken_GoogleSchemelessIssuer5078=== PAUSE TestValidateToken_GoogleSchemelessIssuer5079=== RUN TestValidateToken_KubernetesServiceAccount5080=== PAUSE TestValidateToken_KubernetesServiceAccount5081=== RUN TestNewValidator_KubernetesRequiresCA5082=== PAUSE TestNewValidator_KubernetesRequiresCA5083=== RUN TestValidateToken_KubernetesIssuerFromOwnToken5084=== PAUSE TestValidateToken_KubernetesIssuerFromOwnToken5085=== RUN TestPins_ReservedForMatchingRule5086=== PAUSE TestPins_ReservedForMatchingRule5087=== RUN TestPins_TopLevelShorthand5088=== PAUSE TestPins_TopLevelShorthand5089=== RUN TestPins_ConfigValidation5090=== PAUSE TestPins_ConfigValidation5091=== RUN TestScopes_LegacyProviderDefaultsToWrite5092=== PAUSE TestScopes_LegacyProviderDefaultsToWrite5093=== RUN TestScopes_Rules5094=== PAUSE TestScopes_Rules5095=== RUN TestScopes_ConfigValidation5096=== PAUSE TestScopes_ConfigValidation5097=== CONT TestAudienceForIssuer5098=== CONT TestPins_TopLevelShorthand5099--- PASS: TestAudienceForIssuer (0.00s)5100=== CONT TestValidateToken_UnknownIssuerFetchesNoKeys5101=== CONT TestValidateToken_MultipleProviders5102=== CONT TestValidateToken_BoundSubjectMismatch5103=== CONT TestValidateToken_BoundClaimsMismatch5104=== CONT TestValidateToken_Expired5105=== CONT TestValidateToken_WrongAudience5106=== CONT TestValidateToken_ValidToken5107=== CONT TestGlobMatch5108=== CONT TestProviderClientGivesUpOnSilentIssuer5109=== CONT TestValidateToken_KubernetesIssuerFromOwnToken5110=== CONT TestPins_ReservedForMatchingRule5111=== CONT TestScopes_Rules5112=== CONT TestScopes_ConfigValidation5113=== RUN TestGlobMatch/foo_foo5114=== PAUSE TestGlobMatch/foo_foo5115=== RUN TestGlobMatch/foo_bar5116=== PAUSE TestGlobMatch/foo_bar5117=== RUN TestGlobMatch/*_5118=== PAUSE TestGlobMatch/*_5119=== RUN TestGlobMatch/*_anything5120=== PAUSE TestGlobMatch/*_anything5121=== RUN TestGlobMatch/foo*_foo5122=== PAUSE TestGlobMatch/foo*_foo5123=== RUN TestGlobMatch/foo*_foobar5124=== PAUSE TestGlobMatch/foo*_foobar5125=== RUN TestGlobMatch/foo*_bar5126=== PAUSE TestGlobMatch/foo*_bar5127=== RUN TestGlobMatch/*bar_bar5128=== PAUSE TestGlobMatch/*bar_bar5129=== RUN TestGlobMatch/*bar_foobar5130=== PAUSE TestGlobMatch/*bar_foobar5131=== RUN TestGlobMatch/*bar_foo5132=== PAUSE TestGlobMatch/*bar_foo5133=== RUN TestGlobMatch/foo*bar_foobar5134=== PAUSE TestGlobMatch/foo*bar_foobar5135=== RUN TestGlobMatch/foo*bar_foo123bar5136=== PAUSE TestGlobMatch/foo*bar_foo123bar5137=== RUN TestGlobMatch/foo*bar_foobarbaz5138=== PAUSE TestGlobMatch/foo*bar_foobarbaz5139=== RUN TestGlobMatch/*/*_foo/bar5140=== PAUSE TestGlobMatch/*/*_foo/bar5141=== RUN TestGlobMatch/*/*_foo5142=== PAUSE TestGlobMatch/*/*_foo5143=== RUN TestGlobMatch/refs/heads/*_refs/heads/main5144=== PAUSE TestGlobMatch/refs/heads/*_refs/heads/main5145=== RUN TestGlobMatch/refs/heads/*_refs/tags/v1.05146=== PAUSE TestGlobMatch/refs/heads/*_refs/tags/v1.05147=== RUN TestGlobMatch/refs/*/main_refs/heads/main5148=== PAUSE TestGlobMatch/refs/*/main_refs/heads/main5149=== RUN TestGlobMatch/fo?_foo5150=== PAUSE TestGlobMatch/fo?_foo5151=== RUN TestGlobMatch/fo?_fo5152=== PAUSE TestGlobMatch/fo?_fo5153=== RUN TestGlobMatch/fo?_fooo5154=== PAUSE TestGlobMatch/fo?_fooo5155=== RUN TestGlobMatch/?oo_foo5156=== PAUSE TestGlobMatch/?oo_foo5157=== RUN TestGlobMatch/?oo_boo5158=== PAUSE TestGlobMatch/?oo_boo5159=== RUN TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5160=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5161=== RUN TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5162=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5163=== CONT TestGlobMatch/foo_foo5164=== CONT TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5165=== CONT TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5166=== CONT TestGlobMatch/?oo_boo5167=== CONT TestGlobMatch/?oo_foo5168=== CONT TestGlobMatch/fo?_fooo5169=== CONT TestGlobMatch/fo?_fo5170=== CONT TestGlobMatch/fo?_foo5171=== CONT TestGlobMatch/refs/*/main_refs/heads/main5172=== CONT TestGlobMatch/refs/heads/*_refs/tags/v1.05173=== CONT TestGlobMatch/refs/heads/*_refs/heads/main5174=== CONT TestGlobMatch/*/*_foo5175=== CONT TestGlobMatch/*/*_foo/bar5176=== CONT TestGlobMatch/foo*bar_foobarbaz5177=== CONT TestGlobMatch/foo*bar_foo123bar5178=== CONT TestGlobMatch/foo*bar_foobar5179=== CONT TestGlobMatch/*bar_foo5180=== CONT TestGlobMatch/*bar_foobar5181=== CONT TestGlobMatch/*bar_bar5182=== CONT TestGlobMatch/foo*_bar5183=== CONT TestGlobMatch/foo*_foobar5184=== CONT TestGlobMatch/foo*_foo5185=== CONT TestGlobMatch/*_anything5186=== CONT TestGlobMatch/*_5187=== CONT TestGlobMatch/foo_bar5188--- PASS: TestGlobMatch (0.02s)5189 --- PASS: TestGlobMatch/foo_foo (0.00s)5190 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main (0.00s)5191 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main (0.00s)5192 --- PASS: TestGlobMatch/?oo_boo (0.00s)5193 --- PASS: TestGlobMatch/?oo_foo (0.00s)5194 --- PASS: TestGlobMatch/fo?_fooo (0.00s)5195 --- PASS: TestGlobMatch/fo?_fo (0.00s)5196 --- PASS: TestGlobMatch/fo?_foo (0.00s)5197 --- PASS: TestGlobMatch/refs/*/main_refs/heads/main (0.00s)5198 --- PASS: TestGlobMatch/refs/heads/*_refs/tags/v1.0 (0.00s)5199 --- PASS: TestGlobMatch/refs/heads/*_refs/heads/main (0.00s)5200 --- PASS: TestGlobMatch/*/*_foo (0.00s)5201 --- PASS: TestGlobMatch/*/*_foo/bar (0.00s)5202 --- PASS: TestGlobMatch/foo*bar_foobarbaz (0.00s)5203 --- PASS: TestGlobMatch/foo*bar_foo123bar (0.00s)5204 --- PASS: TestGlobMatch/foo*bar_foobar (0.00s)5205 --- PASS: TestGlobMatch/*bar_foo (0.00s)5206 --- PASS: TestGlobMatch/*bar_foobar (0.00s)5207 --- PASS: TestGlobMatch/*bar_bar (0.00s)5208 --- PASS: TestGlobMatch/foo*_bar (0.00s)5209 --- PASS: TestGlobMatch/foo*_foobar (0.00s)5210 --- PASS: TestGlobMatch/foo*_foo (0.00s)5211 --- PASS: TestGlobMatch/*_anything (0.00s)5212 --- PASS: TestGlobMatch/*_ (0.00s)5213 --- PASS: TestGlobMatch/foo_bar (0.00s)52142026/11/12 00:15:36 INFO OIDC provider initialized name=known issuer=https://known.example5215--- PASS: TestScopes_ConfigValidation (0.04s)5216=== CONT TestNewValidator_KubernetesRequiresCA5217=== CONT TestValidateToken_KubernetesServiceAccount5218=== CONT TestScopes_LegacyProviderDefaultsToWrite5219=== CONT TestPins_ConfigValidation5220=== CONT TestValidateToken_GoogleSchemelessIssuer5221--- PASS: TestValidateToken_UnknownIssuerFetchesNoKeys (2.86s)5222--- PASS: TestProviderClientGivesUpOnSilentIssuer (0.02s)5223--- PASS: TestPins_ConfigValidation (0.00s)52242026/11/12 00:15:39 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:44297/oidc5225--- PASS: TestPins_ReservedForMatchingRule (2.88s)52262026/11/12 00:15:41 INFO OIDC provider initialized name=kubernetes issuer=https://oidc.eks.invalid/id/ABC12352272026/11/12 00:15:41 INFO OIDC provider initialized name=google issuer=https://accounts.google.com52282026/11/12 00:15:41 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:45667/oidc52292026/11/12 00:15:41 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:41839/oidc52302026/11/12 00:15:41 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:37545/oidc52312026/11/12 00:15:41 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:40979/oidc52322026/11/12 00:15:41 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:44935/oidc5233--- PASS: TestValidateToken_GoogleSchemelessIssuer (2.15s)52342026/11/12 00:15:41 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:33873/oidc52352026/11/12 00:15:41 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:36693/oidc5236--- PASS: TestPins_TopLevelShorthand (6.44s)5237--- PASS: TestValidateToken_BoundClaimsMismatch (6.43s)52382026/11/12 00:15:42 INFO OIDC provider initialized name=provider1 issuer=http://127.0.0.1:45999/oidc52392026/11/12 00:15:42 INFO OIDC provider initialized name=provider2 issuer=http://127.0.0.1:39103/oidc5240--- PASS: TestValidateToken_BoundSubjectMismatch (6.43s)5241--- PASS: TestValidateToken_WrongAudience (6.43s)5242--- PASS: TestScopes_LegacyProviderDefaultsToWrite (3.57s)5243--- PASS: TestValidateToken_ValidToken (6.43s)5244--- PASS: TestValidateToken_KubernetesIssuerFromOwnToken (3.58s)52452026/11/12 00:15:42 INFO OIDC provider initialized name=kubernetes issuer=https://127.0.0.1:467735246--- PASS: TestValidateToken_MultipleProviders (6.45s)5247--- PASS: TestScopes_Rules (6.45s)5248--- PASS: TestValidateToken_KubernetesServiceAccount (3.59s)52492026/11/12 00:15:43 http: TLS handshake error from 127.0.0.1:42890: remote error: tls: bad certificate5250--- PASS: TestNewValidator_KubernetesRequiresCA (3.59s)52512026/11/12 00:15:43 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:35997/oidc5252--- PASS: TestValidateToken_Expired (6.45s)5253PASS5254Running hook tests...5255=== RUN TestSendPathsEmpty5256=== PAUSE TestSendPathsEmpty5257=== RUN TestQueueEnqueueAndFetch5258=== PAUSE TestQueueEnqueueAndFetch5259=== RUN TestQueueDeduplication5260=== PAUSE TestQueueDeduplication5261=== RUN TestQueueRemove5262=== PAUSE TestQueueRemove5263=== RUN TestQueueFetchBatchLimit5264=== PAUSE TestQueueFetchBatchLimit5265=== RUN TestQueueRetryMovesToBack5266=== PAUSE TestQueueRetryMovesToBack5267=== RUN TestQueueFetchRemoveLifecycle5268=== PAUSE TestQueueFetchRemoveLifecycle5269=== RUN TestQueueConcurrentWriters5270=== PAUSE TestQueueConcurrentWriters5271=== RUN TestQueueRemoveLargeClosure5272=== PAUSE TestQueueRemoveLargeClosure5273=== RUN TestServerClientIntegration5274--- PASS: TestServerClientIntegration (1.45s)5275=== RUN TestServerQueueError52762026/11/12 00:15:44 ERROR Failed to queue paths error="permission denied" count=15277--- PASS: TestServerQueueError (0.00s)5278=== RUN TestGetListenerSocketActivation5279 server_test.go:220: === RUN TestGetListenerSocketActivation5280 --- PASS: TestGetListenerSocketActivation (0.00s)5281 PASS5282 5283--- PASS: TestGetListenerSocketActivation (0.02s)5284=== RUN TestServerStalledClientDoesNotBlockShutdown5285=== PAUSE TestServerStalledClientDoesNotBlockShutdown5286=== RUN TestServerBacksOffOnAcceptErrors5287=== PAUSE TestServerBacksOffOnAcceptErrors5288=== RUN TestDrainIsolatesPoisonPath5289=== PAUSE TestDrainIsolatesPoisonPath5290=== RUN TestRunNotBlockedByPoisonHead5291=== PAUSE TestRunNotBlockedByPoisonHead5292=== RUN TestDrainGivesUpWhenServerDown5293=== PAUSE TestDrainGivesUpWhenServerDown5294=== RUN TestFailedPathPrunedByLaterClosure5295=== PAUSE TestFailedPathPrunedByLaterClosure5296=== RUN TestWorkerUploadsAndRemoves5297=== PAUSE TestWorkerUploadsAndRemoves5298=== RUN TestWorkerSkipsGCdPaths5299=== PAUSE TestWorkerSkipsGCdPaths5300=== RUN TestWorkerPrunesClosureDeps5301=== PAUSE TestWorkerPrunesClosureDeps5302=== RUN TestDrainTimeout5303=== PAUSE TestDrainTimeout5304=== RUN TestWorkerRemoveFailureIsNotProgress5305=== PAUSE TestWorkerRemoveFailureIsNotProgress5306=== RUN TestWorkerKeepsPathItCannotStat5307=== PAUSE TestWorkerKeepsPathItCannotStat5308=== CONT TestSendPathsEmpty5309--- PASS: TestSendPathsEmpty (0.00s)5310=== CONT TestServerStalledClientDoesNotBlockShutdown5311=== CONT TestServerBacksOffOnAcceptErrors5312=== CONT TestWorkerUploadsAndRemoves53132000/01/01 00:00:00 ERROR Accept failed error="too many open files"5314=== CONT TestWorkerKeepsPathItCannotStat53152000/01/01 00:00:00 ERROR Accept failed error="too many open files"53162000/01/01 00:00:00 ERROR Accept failed error="too many open files"53172000/01/01 00:00:00 ERROR Accept failed error="too many open files"53182000/01/01 00:00:00 ERROR Accept failed error="too many open files"53192000/01/01 00:00:00 ERROR Accept failed error="too many open files"53202000/01/01 00:00:00 ERROR Accept failed error="too many open files"53212000/01/01 00:00:00 ERROR Accept failed error="too many open files"5322=== CONT TestWorkerRemoveFailureIsNotProgress5323=== CONT TestDrainTimeout5324=== CONT TestWorkerPrunesClosureDeps5325=== CONT TestWorkerSkipsGCdPaths5326=== CONT TestQueueRetryMovesToBack5327=== CONT TestQueueRemoveLargeClosure5328=== CONT TestQueueConcurrentWriters5329=== CONT TestDrainGivesUpWhenServerDown5330=== CONT TestFailedPathPrunedByLaterClosure5331=== CONT TestQueueRemove5332=== CONT TestQueueFetchBatchLimit5333=== CONT TestRunNotBlockedByPoisonHead5334=== CONT TestDrainIsolatesPoisonPath5335=== CONT TestQueueDeduplication5336=== CONT TestQueueEnqueueAndFetch5337=== CONT TestQueueFetchRemoveLifecycle5338--- PASS: TestServerBacksOffOnAcceptErrors (0.00s)53392026/11/12 00:15:44 INFO Uploading batch count=253402026/11/12 00:15:44 ERROR Upload failed error="upload failed" count=253412026/11/12 00:15:44 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown3365279796/002/a53422026/11/12 00:15:44 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown3365279796/002/b53432026/11/12 00:15:44 INFO Uploading batch count=253442026/11/12 00:15:44 ERROR Upload failed error="upload failed" count=253452026/11/12 00:15:44 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown3365279796/002/c53462026/11/12 00:15:44 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown3365279796/002/d53472026/11/12 00:15:44 INFO Uploading batch count=253482026/11/12 00:15:44 ERROR Upload failed error="upload failed" count=253492026/11/12 00:15:44 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown3365279796/002/e53502026/11/12 00:15:44 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown3365279796/002/f53512026/11/12 00:15:44 ERROR Drain finished with paths left in queue remaining=105352--- PASS: TestDrainGivesUpWhenServerDown (0.02s)5353--- PASS: TestQueueFetchBatchLimit (0.02s)53542026/11/12 00:15:44 INFO Upload queue status pending=353552026/11/12 00:15:44 INFO Uploading batch count=153562026/11/12 00:15:44 ERROR Upload failed error="upload failed" count=15357--- PASS: TestQueueFetchRemoveLifecycle (0.03s)53582026/11/12 00:15:44 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress3748091490/002/nonexistent53592026/11/12 00:15:44 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=153602026/11/12 00:15:44 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress3748091490/002/nonexistent53612026/11/12 00:15:44 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=153622026/11/12 00:15:44 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress3748091490/002/nonexistent53632026/11/12 00:15:44 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=153642026/11/12 00:15:44 ERROR Drain finished with paths left in queue remaining=15365--- PASS: TestWorkerRemoveFailureIsNotProgress (0.04s)53662026/11/12 00:15:44 INFO Uploading batch count=453672026/11/12 00:15:44 ERROR Upload failed error="upload failed" count=453682026/11/12 00:15:44 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainIsolatesPoisonPath267680109/002/bbb53692026/11/12 00:15:44 INFO Uploading batch count=153702026/11/12 00:15:44 ERROR Upload failed error="upload failed" count=153712026/11/12 00:15:44 INFO Uploading batch count=153722026/11/12 00:15:44 ERROR Upload failed error="upload failed" count=153732026/11/12 00:15:44 INFO Uploading batch count=153742026/11/12 00:15:44 ERROR Upload failed error="upload failed" count=153752026/11/12 00:15:44 ERROR Drain finished with paths left in queue remaining=15376--- PASS: TestDrainIsolatesPoisonPath (0.04s)5377--- PASS: TestQueueRemove (0.04s)5378--- PASS: TestQueueRetryMovesToBack (0.05s)53792026/11/12 00:15:44 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat3066493007/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat3066493007/002/locked/aaa: permission denied"53802026/11/12 00:15:44 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat3066493007/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat3066493007/002/locked/aaa: permission denied"53812026/11/12 00:15:44 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat3066493007/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat3066493007/002/locked/aaa: permission denied"53822026/11/12 00:15:44 INFO Upload queue status pending=253832026/11/12 00:15:44 INFO Uploading batch count=253842000/01/01 00:00:00 INFO Uploading batch count=253852026/11/12 00:15:44 ERROR Drain finished with paths left in queue remaining=15386--- PASS: TestWorkerKeepsPathItCannotStat (0.06s)53872026/11/12 00:15:44 INFO Upload queue status pending=253882026/11/12 00:15:44 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerSkipsGCdPaths1116340317/002/nonexistent53892026/11/12 00:15:44 INFO Uploading batch count=153902026/11/12 00:15:44 INFO Upload queue status pending=253912026/11/12 00:15:44 INFO Uploading batch count=153922000/01/01 00:00:00 ERROR Upload failed error="context deadline exceeded" count=253932000/01/01 00:00:00 ERROR Drain finished with paths left in queue remaining=45394--- PASS: TestDrainTimeout (1.00s)5395--- PASS: TestWorkerUploadsAndRemoves (1.00s)53962026/11/12 00:15:45 ERROR Failed to decode request error="read unix /build/hook1427713040/test.sock->@: i/o timeout"53972026/11/12 00:15:45 ERROR Failed to write response error="write unix /build/hook1427713040/test.sock->@: i/o timeout"5398--- PASS: TestServerStalledClientDoesNotBlockShutdown (1.00s)5399--- PASS: TestWorkerSkipsGCdPaths (1.00s)5400--- PASS: TestWorkerPrunesClosureDeps (1.01s)5401--- PASS: TestQueueEnqueueAndFetch (1.01s)54022026/11/12 00:15:45 INFO Uploading batch count=15403--- PASS: TestQueueDeduplication (1.01s)54042026/11/12 00:15:45 ERROR Upload failed error="upload failed" count=154052026/11/12 00:15:45 INFO Uploading batch count=154062026/11/12 00:15:45 INFO Uploading batch count=15407--- PASS: TestFailedPathPrunedByLaterClosure (1.01s)54082026/11/12 00:15:46 INFO Uploading batch count=154092026/11/12 00:15:46 INFO Uploading batch count=154102026/11/12 00:15:46 INFO Uploading batch count=154112026/11/12 00:15:46 ERROR Upload failed error="upload failed" count=154122026/11/12 00:15:46 INFO Uploading batch count=154132026/11/12 00:15:46 ERROR Upload failed error="upload failed" count=154142026/11/12 00:15:46 INFO Uploading batch count=154152026/11/12 00:15:46 ERROR Upload failed error="upload failed" count=154162026/11/12 00:15:46 INFO Uploading batch count=154172026/11/12 00:15:46 ERROR Upload failed error="upload failed" count=154182026/11/12 00:15:46 ERROR Drain finished with paths left in queue remaining=15419--- PASS: TestRunNotBlockedByPoisonHead (1.49s)5420--- PASS: TestQueueRemoveLargeClosure (1.55s)5421--- PASS: TestQueueConcurrentWriters (1.57s)5422PASS5423Running signing tests...5424=== RUN TestExternalSigner5425=== PAUSE TestExternalSigner5426=== RUN TestExternalSignerWrongPublicKey5427=== PAUSE TestExternalSignerWrongPublicKey5428=== RUN TestExternalSignerErrors5429=== PAUSE TestExternalSignerErrors5430=== RUN TestExternalSignerRestartsAfterCrash5431=== PAUSE TestExternalSignerRestartsAfterCrash5432=== RUN TestExternalSignerProgramErrorKeepsProcess5433=== PAUSE TestExternalSignerProgramErrorKeepsProcess5434=== RUN TestExternalSignerClose5435=== PAUSE TestExternalSignerClose5436=== RUN TestExternalSignerRejectsInvalidPublicKey5437=== PAUSE TestExternalSignerRejectsInvalidPublicKey5438=== RUN TestExternalSignerCancellation5439=== PAUSE TestExternalSignerCancellation5440=== RUN TestGenerateFingerprint5441=== PAUSE TestGenerateFingerprint5442=== RUN TestParseSigningKey5443=== PAUSE TestParseSigningKey5444=== RUN TestSign5445=== PAUSE TestSign5446=== RUN TestSignBatch5447=== PAUSE TestSignBatch5448=== RUN TestNilKeyPublicKey5449=== PAUSE TestNilKeyPublicKey5450=== RUN TestSignErrors5451=== PAUSE TestSignErrors5452=== CONT TestExternalSigner5453=== CONT TestExternalSignerCancellation5454=== CONT TestSignErrors5455=== CONT TestExternalSignerProgramErrorKeepsProcess5456=== CONT TestSignBatch5457=== CONT TestNilKeyPublicKey5458--- PASS: TestNilKeyPublicKey (0.00s)5459=== CONT TestExternalSignerRejectsInvalidPublicKey5460=== RUN TestExternalSignerRejectsInvalidPublicKey/#005461=== PAUSE TestExternalSignerRejectsInvalidPublicKey/#005462=== RUN TestExternalSignerRejectsInvalidPublicKey/no-colon5463=== CONT TestExternalSignerClose5464=== CONT TestParseSigningKey5465=== RUN TestParseSigningKey/valid_32-byte_key5466=== PAUSE TestParseSigningKey/valid_32-byte_key5467=== RUN TestParseSigningKey/valid_32-byte_key_with_different_name5468=== CONT TestSign5469=== CONT TestExternalSignerErrors5470=== CONT TestExternalSignerRestartsAfterCrash5471=== CONT TestGenerateFingerprint5472=== CONT TestExternalSignerWrongPublicKey5473=== PAUSE TestExternalSignerRejectsInvalidPublicKey/no-colon5474=== RUN TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5475=== PAUSE TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5476=== RUN TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5477=== PAUSE TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5478=== RUN TestGenerateFingerprint/basic_with_references5479=== RUN TestExternalSignerErrors/too_few_signatures5480=== PAUSE TestExternalSignerErrors/too_few_signatures5481=== PAUSE TestParseSigningKey/valid_32-byte_key_with_different_name5482=== PAUSE TestGenerateFingerprint/basic_with_references5483=== RUN TestGenerateFingerprint/no_references5484=== PAUSE TestGenerateFingerprint/no_references5485=== RUN TestGenerateFingerprint/unsorted_references_get_sorted5486=== PAUSE TestGenerateFingerprint/unsorted_references_get_sorted5487=== RUN TestExternalSignerErrors/invalid_signature_base645488=== PAUSE TestExternalSignerErrors/invalid_signature_base645489=== RUN TestExternalSignerErrors/wrong_signature_length5490=== PAUSE TestExternalSignerErrors/wrong_signature_length5491=== RUN TestExternalSignerErrors/program_error5492=== RUN TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5493=== PAUSE TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5494=== CONT TestExternalSignerRejectsInvalidPublicKey/#005495=== RUN TestGenerateFingerprint/invalid_nar_hash_prefix5496=== PAUSE TestGenerateFingerprint/invalid_nar_hash_prefix5497=== RUN TestGenerateFingerprint/invalid_nar_hash_length5498=== PAUSE TestGenerateFingerprint/invalid_nar_hash_length5499=== RUN TestGenerateFingerprint/invalid_store_path_prefix5500=== PAUSE TestGenerateFingerprint/invalid_store_path_prefix5501=== RUN TestGenerateFingerprint/invalid_reference_prefix5502=== PAUSE TestGenerateFingerprint/invalid_reference_prefix5503=== CONT TestGenerateFingerprint/basic_with_references5504=== CONT TestGenerateFingerprint/invalid_reference_prefix5505=== CONT TestGenerateFingerprint/invalid_store_path_prefix5506=== CONT TestGenerateFingerprint/invalid_nar_hash_length5507=== CONT TestGenerateFingerprint/invalid_nar_hash_prefix5508=== CONT TestGenerateFingerprint/unsorted_references_get_sorted5509=== CONT TestGenerateFingerprint/no_references5510=== CONT TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5511=== CONT TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5512=== CONT TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5513=== RUN TestParseSigningKey/no_colon5514=== PAUSE TestParseSigningKey/no_colon5515=== CONT TestExternalSignerRejectsInvalidPublicKey/no-colon5516=== PAUSE TestExternalSignerErrors/program_error5517--- PASS: TestSignBatch (0.00s)5518--- PASS: TestGenerateFingerprint (0.00s)5519 --- PASS: TestGenerateFingerprint/basic_with_references (0.00s)5520 --- PASS: TestGenerateFingerprint/invalid_reference_prefix (0.00s)5521 --- PASS: TestGenerateFingerprint/invalid_store_path_prefix (0.00s)5522 --- PASS: TestGenerateFingerprint/invalid_nar_hash_length (0.00s)5523 --- PASS: TestGenerateFingerprint/invalid_nar_hash_prefix (0.00s)5524 --- PASS: TestGenerateFingerprint/unsorted_references_get_sorted (0.00s)5525 --- PASS: TestGenerateFingerprint/no_references (0.00s)5526=== RUN TestParseSigningKey/empty_name5527=== RUN TestExternalSignerErrors/exit_on_request5528=== PAUSE TestParseSigningKey/empty_name5529=== RUN TestParseSigningKey/invalid_base645530=== PAUSE TestParseSigningKey/invalid_base645531=== RUN TestParseSigningKey/wrong_length5532=== PAUSE TestParseSigningKey/wrong_length5533=== CONT TestParseSigningKey/valid_32-byte_key5534=== PAUSE TestExternalSignerErrors/exit_on_request5535=== RUN TestExternalSignerErrors/null5536=== PAUSE TestExternalSignerErrors/null5537=== RUN TestExternalSignerErrors/missing_signatures5538=== PAUSE TestExternalSignerErrors/missing_signatures5539=== RUN TestExternalSignerErrors/unrequested_key5540=== PAUSE TestExternalSignerErrors/unrequested_key5541=== RUN TestExternalSignerErrors/missing_signature_name5542=== PAUSE TestExternalSignerErrors/missing_signature_name5543=== RUN TestExternalSignerErrors/wrong_signature_name5544=== PAUSE TestExternalSignerErrors/wrong_signature_name5545=== RUN TestExternalSignerErrors/exit_at_startup5546=== PAUSE TestExternalSignerErrors/exit_at_startup5547=== RUN TestExternalSignerErrors/invalid_JSON5548=== PAUSE TestExternalSignerErrors/invalid_JSON5549=== RUN TestExternalSignerErrors/wrong_shape5550=== PAUSE TestExternalSignerErrors/wrong_shape5551=== RUN TestExternalSignerErrors/wrong_id5552=== PAUSE TestExternalSignerErrors/wrong_id5553=== RUN TestExternalSignerErrors/null_signatures5554=== PAUSE TestExternalSignerErrors/null_signatures5555=== RUN TestExternalSignerErrors/wrong_signatures_shape5556=== PAUSE TestExternalSignerErrors/wrong_signatures_shape5557=== CONT TestExternalSignerErrors/too_few_signatures5558=== RUN TestSignErrors/nil_key5559=== PAUSE TestSignErrors/nil_key5560=== CONT TestExternalSignerErrors/invalid_JSON5561=== RUN TestSignErrors/nil_narinfo5562=== PAUSE TestSignErrors/nil_narinfo5563=== RUN TestSignErrors/invalid_narinfo5564=== PAUSE TestSignErrors/invalid_narinfo5565=== CONT TestExternalSignerErrors/invalid_signature_base645566=== CONT TestParseSigningKey/invalid_base645567=== CONT TestExternalSignerErrors/wrong_id5568=== CONT TestParseSigningKey/empty_name5569=== CONT TestExternalSignerErrors/missing_signatures5570=== CONT TestParseSigningKey/no_colon5571=== CONT TestExternalSignerErrors/exit_at_startup5572=== CONT TestExternalSignerErrors/program_error5573=== CONT TestExternalSignerErrors/wrong_signature_name5574=== CONT TestExternalSignerErrors/null5575=== CONT TestExternalSignerErrors/missing_signature_name5576=== CONT TestExternalSignerErrors/exit_on_request5577=== CONT TestExternalSignerErrors/unrequested_key5578=== CONT TestExternalSignerErrors/wrong_signature_length5579--- PASS: TestExternalSignerRejectsInvalidPublicKey (0.00s)5580 --- PASS: TestExternalSignerRejectsInvalidPublicKey/#00 (0.00s)5581 --- PASS: TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= (0.00s)5582 --- PASS: TestExternalSignerRejectsInvalidPublicKey/test-key:YQ== (0.00s)5583 --- PASS: TestExternalSignerRejectsInvalidPublicKey/test-key:invalid! (0.00s)5584 --- PASS: TestExternalSignerRejectsInvalidPublicKey/no-colon (0.00s)5585=== CONT TestParseSigningKey/valid_32-byte_key_with_different_name5586--- PASS: TestSign (0.00s)5587=== CONT TestParseSigningKey/wrong_length5588=== RUN TestSignErrors/canceled5589=== CONT TestExternalSignerErrors/wrong_shape5590=== CONT TestExternalSignerErrors/wrong_signatures_shape5591=== CONT TestExternalSignerErrors/null_signatures5592--- PASS: TestParseSigningKey (0.00s)5593 --- PASS: TestParseSigningKey/valid_32-byte_key (0.00s)5594 --- PASS: TestParseSigningKey/invalid_base64 (0.00s)5595 --- PASS: TestParseSigningKey/empty_name (0.00s)5596 --- PASS: TestParseSigningKey/no_colon (0.00s)5597 --- PASS: TestParseSigningKey/valid_32-byte_key_with_different_name (0.00s)5598 --- PASS: TestParseSigningKey/wrong_length (0.00s)5599--- PASS: TestExternalSignerProgramErrorKeepsProcess (0.01s)5600--- PASS: TestExternalSignerClose (0.01s)5601=== PAUSE TestSignErrors/canceled5602=== CONT TestSignErrors/nil_key5603=== CONT TestSignErrors/invalid_narinfo5604=== CONT TestSignErrors/nil_narinfo5605=== CONT TestSignErrors/canceled5606--- PASS: TestSignErrors (0.01s)5607 --- PASS: TestSignErrors/nil_key (0.00s)5608 --- PASS: TestSignErrors/invalid_narinfo (0.00s)5609 --- PASS: TestSignErrors/nil_narinfo (0.00s)5610 --- PASS: TestSignErrors/canceled (0.00s)5611--- PASS: TestExternalSignerWrongPublicKey (0.01s)5612--- PASS: TestExternalSignerRestartsAfterCrash (0.02s)5613--- PASS: TestExternalSigner (0.02s)5614--- PASS: TestExternalSignerErrors (0.00s)5615 --- PASS: TestExternalSignerErrors/too_few_signatures (0.01s)5616 --- PASS: TestExternalSignerErrors/exit_at_startup (0.01s)5617 --- PASS: TestExternalSignerErrors/wrong_id (0.01s)5618 --- PASS: TestExternalSignerErrors/invalid_JSON (0.01s)5619 --- PASS: TestExternalSignerErrors/invalid_signature_base64 (0.01s)5620 --- PASS: TestExternalSignerErrors/program_error (0.01s)5621 --- PASS: TestExternalSignerErrors/null (0.01s)5622 --- PASS: TestExternalSignerErrors/missing_signature_name (0.01s)5623 --- PASS: TestExternalSignerErrors/wrong_signature_name (0.01s)5624 --- PASS: TestExternalSignerErrors/missing_signatures (0.01s)5625 --- PASS: TestExternalSignerErrors/null_signatures (0.01s)5626 --- PASS: TestExternalSignerErrors/exit_on_request (0.01s)5627 --- PASS: TestExternalSignerErrors/wrong_signatures_shape (0.01s)5628 --- PASS: TestExternalSignerErrors/wrong_signature_length (0.01s)5629 --- PASS: TestExternalSignerErrors/unrequested_key (0.01s)5630 --- PASS: TestExternalSignerErrors/wrong_shape (0.01s)5631--- PASS: TestExternalSignerCancellation (0.02s)5632PASS5633Running OpenBao signer tests...5634=== RUN TestSign5635=== PAUSE TestSign5636=== RUN TestSignErrors5637=== PAUSE TestSignErrors5638=== RUN TestServe5639=== PAUSE TestServe5640=== RUN TestServeRejectsGarbage5641=== PAUSE TestServeRejectsGarbage5642=== CONT TestSign5643=== CONT TestServeRejectsGarbage5644--- PASS: TestServeRejectsGarbage (0.00s)5645=== CONT TestServe5646--- PASS: TestServe (0.00s)5647=== CONT TestSignErrors5648=== RUN TestSignErrors/http_error5649=== PAUSE TestSignErrors/http_error5650=== RUN TestSignErrors/invalid_json5651=== PAUSE TestSignErrors/invalid_json5652=== RUN TestSignErrors/no_prefix5653=== PAUSE TestSignErrors/no_prefix5654=== RUN TestSignErrors/invalid_base645655=== PAUSE TestSignErrors/invalid_base645656=== CONT TestSignErrors/http_error5657=== CONT TestSignErrors/invalid_base645658=== CONT TestSignErrors/no_prefix5659=== CONT TestSignErrors/invalid_json5660--- PASS: TestSign (0.58s)5661--- PASS: TestSignErrors (0.00s)5662 --- PASS: TestSignErrors/invalid_base64 (0.58s)5663 --- PASS: TestSignErrors/no_prefix (0.58s)5664 --- PASS: TestSignErrors/http_error (0.58s)5665 --- PASS: TestSignErrors/invalid_json (0.58s)5666PASS