niks3-go-unit-tests
checks.aarch64-darwin.go-unit-tests
· build #502
· raw
1tribuchet: building on mac022Running client tests...3=== RUN TestDoServerRequestAttachesToken4=== PAUSE TestDoServerRequestAttachesToken5=== RUN TestUploadBuildLog_FileBodyReplayedOnRetry6=== PAUSE TestUploadBuildLog_FileBodyReplayedOnRetry7=== RUN TestRegisterUploadedObjectReusesConnections8=== PAUSE TestRegisterUploadedObjectReusesConnections9=== RUN TestRunGarbageCollection_FinishedOnAnotherReplica10=== PAUSE TestRunGarbageCollection_FinishedOnAnotherReplica11=== RUN TestRunGarbageCollection_NotFoundAfterLocalRun12=== PAUSE TestRunGarbageCollection_NotFoundAfterLocalRun13=== RUN TestCaseHackSuffix14=== PAUSE TestCaseHackSuffix15=== RUN TestFilterOversizedClosures16=== PAUSE TestFilterOversizedClosures17=== RUN TestUploadMultipart_PartsInParallel18=== PAUSE TestUploadMultipart_PartsInParallel19=== RUN TestUploadMultipart_ReturnsAfterDroppedBody20=== PAUSE TestUploadMultipart_ReturnsAfterDroppedBody21=== RUN TestUploadMultipart_FailedProducerIsNotCompleted22=== PAUSE TestUploadMultipart_FailedProducerIsNotCompleted23=== RUN TestPartSizeForNAR24=== PAUSE TestPartSizeForNAR25=== RUN TestDumpPathCaseHackMatchesNix26--- PASS: TestDumpPathCaseHackMatchesNix (0.09s)27=== RUN TestDumpPathCaseHackCollision28--- PASS: TestDumpPathCaseHackCollision (0.00s)29=== RUN TestDumpPathMatchesNix30=== PAUSE TestDumpPathMatchesNix31=== RUN TestDumpPathSingleFile32=== PAUSE TestDumpPathSingleFile33=== RUN TestDumpPathWriterError34=== PAUSE TestDumpPathWriterError35=== RUN TestEncodeNixBase3236=== PAUSE TestEncodeNixBase3237=== RUN TestEncodeNixBase32WithRealHash38=== PAUSE TestEncodeNixBase32WithRealHash39=== RUN TestConvertHashToNix3240=== PAUSE TestConvertHashToNix3241=== RUN TestGetStorePathHash42=== PAUSE TestGetStorePathHash43=== RUN TestPathInfoHashCompatibility44=== PAUSE TestPathInfoHashCompatibility45=== RUN TestParsePathInfoJSON46=== PAUSE TestParsePathInfoJSON47=== RUN TestParsePathInfoJSONMultiplePaths48=== PAUSE TestParsePathInfoJSONMultiplePaths49=== RUN TestPathInfoCACompatibility50=== PAUSE TestPathInfoCACompatibility51=== RUN TestRateLimiterFeedback52=== PAUSE TestRateLimiterFeedback53=== RUN TestRateLimiterFeedback_400DoesNotCountAsSuccess54=== PAUSE TestRateLimiterFeedback_400DoesNotCountAsSuccess55=== RUN TestRegisterUploadedObject_BoundedAgainstSilentServer56=== PAUSE TestRegisterUploadedObject_BoundedAgainstSilentServer57=== RUN TestRegisterUploadedObject_OutlivesPush58=== PAUSE TestRegisterUploadedObject_OutlivesPush59=== RUN TestResolveStorePath60=== PAUSE TestResolveStorePath61=== RUN TestDoWithRetry_BodyReplayedViaGetBody62=== PAUSE TestDoWithRetry_BodyReplayedViaGetBody63=== RUN TestRetryTimeoutOutlastsMaxRetries64=== PAUSE TestRetryTimeoutOutlastsMaxRetries65=== RUN TestDoWithRetry_FinalResponseBodyReadable66=== PAUSE TestDoWithRetry_FinalResponseBodyReadable67=== RUN TestShellSplit68=== PAUSE TestShellSplit69=== RUN TestShellSplitErrors70=== PAUSE TestShellSplitErrors71=== RUN TestStreamPushReportsEveryPath72=== PAUSE TestStreamPushReportsEveryPath73=== RUN TestStreamPushBatchesUnderLoad74=== PAUSE TestStreamPushBatchesUnderLoad75=== RUN TestStreamPushIsolatesFailures76=== PAUSE TestStreamPushIsolatesFailures77=== RUN TestStreamPushGivesUpOnDeadServer78=== PAUSE TestStreamPushGivesUpOnDeadServer79=== RUN TestStreamPushRequestLine80=== PAUSE TestStreamPushRequestLine81=== RUN TestStreamPushReportsSignatures82=== PAUSE TestStreamPushReportsSignatures83=== RUN TestStreamPushReportsSkippedPaths84=== PAUSE TestStreamPushReportsSkippedPaths85=== RUN TestClientSignaturesByStorePath86=== PAUSE TestClientSignaturesByStorePath87=== RUN TestStreamPushAnswersEveryLineWhenCanceled88=== PAUSE TestStreamPushAnswersEveryLineWhenCanceled89=== RUN TestStreamPushStopsWhenCanceled90=== PAUSE TestStreamPushStopsWhenCanceled91=== RUN TestSetClientTLS92=== PAUSE TestSetClientTLS93=== RUN TestSetClientTLSDoesNotMutateDefaultTransport94=== PAUSE TestSetClientTLSDoesNotMutateDefaultTransport95=== RUN TestSetClientTLSErrors96=== PAUSE TestSetClientTLSErrors97=== RUN TestStaticToken98=== PAUSE TestStaticToken99=== RUN TestFileTokenReadsAndCaches100=== PAUSE TestFileTokenReadsAndCaches101=== RUN TestFileTokenMissing102=== PAUSE TestFileTokenMissing103=== RUN TestFileTokenEmpty104=== PAUSE TestFileTokenEmpty105=== RUN TestScriptTokenNoExpiryRerunsEveryCall106=== PAUSE TestScriptTokenNoExpiryRerunsEveryCall107=== RUN TestScriptTokenCachesUntilRefresh108=== PAUSE TestScriptTokenCachesUntilRefresh109=== RUN TestScriptTokenEmptyToken110=== PAUSE TestScriptTokenEmptyToken111=== RUN TestScriptTokenBadJSON112=== PAUSE TestScriptTokenBadJSON113=== RUN TestScriptTokenScriptFails114=== PAUSE TestScriptTokenScriptFails115=== RUN TestScriptTokenEmptyCommand116=== PAUSE TestScriptTokenEmptyCommand117=== CONT TestDoServerRequestAttachesToken118=== CONT TestEncodeNixBase32119=== CONT TestStreamPushStopsWhenCanceled120=== RUN TestEncodeNixBase32/test_string_hash121=== PAUSE TestEncodeNixBase32/test_string_hash122=== RUN TestEncodeNixBase32/empty_input123=== PAUSE TestEncodeNixBase32/empty_input124=== CONT TestScriptTokenEmptyCommand125--- PASS: TestScriptTokenEmptyCommand (0.00s)126=== CONT TestParsePathInfoJSON127=== RUN TestParsePathInfoJSON/Nix_format128=== PAUSE TestParsePathInfoJSON/Nix_format129=== RUN TestParsePathInfoJSON/Lix_format130=== PAUSE TestParsePathInfoJSON/Lix_format131=== RUN TestParsePathInfoJSON/empty_input132=== PAUSE TestParsePathInfoJSON/empty_input133=== RUN TestParsePathInfoJSON/whitespace_only134=== PAUSE TestParsePathInfoJSON/whitespace_only135=== RUN TestParsePathInfoJSON/invalid_JSON136=== PAUSE TestParsePathInfoJSON/invalid_JSON137--- PASS: TestStreamPushStopsWhenCanceled (0.00s)138=== CONT TestParsePathInfoJSONMultiplePaths139=== RUN TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths140=== PAUSE TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths141=== RUN TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths142=== PAUSE TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths143=== CONT TestScriptTokenBadJSON144=== CONT TestConvertHashToNix32145=== CONT TestScriptTokenEmptyToken146=== CONT TestScriptTokenCachesUntilRefresh147=== CONT TestScriptTokenNoExpiryRerunsEveryCall148=== CONT TestFileTokenEmpty149=== CONT TestFileTokenMissing150=== CONT TestFileTokenReadsAndCaches151=== CONT TestStaticToken152--- PASS: TestStaticToken (0.00s)153=== CONT TestEncodeNixBase32WithRealHash154=== CONT TestSetClientTLSErrors155=== CONT TestSetClientTLSDoesNotMutateDefaultTransport156=== CONT TestSetClientTLS157=== CONT TestRetryTimeoutOutlastsMaxRetries158=== CONT TestDoWithRetry_BodyReplayedViaGetBody159=== CONT TestDumpPathMatchesNix160=== CONT TestRegisterUploadedObject_OutlivesPush161=== CONT TestRegisterUploadedObject_BoundedAgainstSilentServer162=== CONT TestRateLimiterFeedback_400DoesNotCountAsSuccess163=== CONT TestRateLimiterFeedback164=== RUN TestRateLimiterFeedback/429_enables_limiter165=== PAUSE TestRateLimiterFeedback/429_enables_limiter166=== RUN TestRateLimiterFeedback/503_enables_limiter167=== PAUSE TestRateLimiterFeedback/503_enables_limiter168=== CONT TestPathInfoCACompatibility169=== RUN TestPathInfoCACompatibility/null_ca_field170=== CONT TestScriptTokenScriptFails171=== CONT TestGetStorePathHash172=== CONT TestPathInfoHashCompatibility173=== RUN TestConvertHashToNix32/SRI_format_to_Nix32174--- PASS: TestEncodeNixBase32WithRealHash (0.00s)175=== CONT TestUploadMultipart_PartsInParallel176=== CONT TestDumpPathSingleFile177=== CONT TestDumpPathWriterError178=== CONT TestResolveStorePath179=== CONT TestPartSizeForNAR180=== RUN TestPartSizeForNAR/zero_stays_at_minimum181=== PAUSE TestPartSizeForNAR/zero_stays_at_minimum182=== RUN TestPartSizeForNAR/small_stays_at_minimum183=== PAUSE TestPartSizeForNAR/small_stays_at_minimum184=== RUN TestPartSizeForNAR/80_GiB_fits_at_minimum185=== PAUSE TestPartSizeForNAR/80_GiB_fits_at_minimum186=== RUN TestPartSizeForNAR/115_GiB_needs_larger_parts187=== PAUSE TestPartSizeForNAR/115_GiB_needs_larger_parts188=== RUN TestPartSizeForNAR/1_TiB189=== PAUSE TestPartSizeForNAR/1_TiB190=== RUN TestPartSizeForNAR/5_TiB_S3_max_object191=== PAUSE TestPartSizeForNAR/5_TiB_S3_max_object192=== RUN TestPartSizeForNAR/capped_at_5_GiB193=== PAUSE TestPartSizeForNAR/capped_at_5_GiB194=== PAUSE TestPathInfoCACompatibility/null_ca_field195=== RUN TestPathInfoCACompatibility/old_string_format_-_text1962026/11/12 00:33:22 WARN Rate limiter enabled after throttle name=server-test rate=5197=== PAUSE TestPathInfoCACompatibility/old_string_format_-_text198=== RUN TestPathInfoCACompatibility/old_string_format_-_fixed_recursive199=== PAUSE TestPathInfoCACompatibility/old_string_format_-_fixed_recursive200=== RUN TestPathInfoCACompatibility/new_structured_format_-_text201=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_text202=== RUN TestPathInfoCACompatibility/new_structured_format_-_nar_method203=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_nar_method204=== RUN TestGetStorePathHash/valid_store_path205=== PAUSE TestGetStorePathHash/valid_store_path206=== RUN TestGetStorePathHash/basename_without_hyphen_should_error207=== PAUSE TestGetStorePathHash/basename_without_hyphen_should_error208=== RUN TestGetStorePathHash/hash_with_invalid_characters_should_error209=== PAUSE TestGetStorePathHash/hash_with_invalid_characters_should_error210=== RUN TestGetStorePathHash/hash_with_wrong_length_should_error211=== PAUSE TestGetStorePathHash/hash_with_wrong_length_should_error212=== CONT TestUploadMultipart_ReturnsAfterDroppedBody213=== RUN TestRateLimiterFeedback/200_does_not_enable_limiter214--- PASS: TestDoServerRequestAttachesToken (0.00s)215--- PASS: TestFileTokenMissing (0.00s)216=== PAUSE TestConvertHashToNix32/SRI_format_to_Nix32217--- PASS: TestFileTokenEmpty (0.00s)218--- PASS: TestFileTokenReadsAndCaches (0.00s)219=== CONT TestRunGarbageCollection_NotFoundAfterLocalRun220=== PAUSE TestRateLimiterFeedback/200_does_not_enable_limiter221=== CONT TestFilterOversizedClosures222=== CONT TestUploadMultipart_FailedProducerIsNotCompleted223--- PASS: TestSetClientTLSDoesNotMutateDefaultTransport (0.00s)2242026/11/12 00:33:22 WARN Rate limiter enabled after throttle name=server-test rate=52252026/11/12 00:33:22 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:54331226--- PASS: TestResolveStorePath (0.00s)227=== CONT TestCaseHackSuffix228=== RUN TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)229=== RUN TestFilterOversizedClosures/no_limit_keeps_everything2302026/11/12 00:33:22 WARN Rate limiter backed off name=server-test rate=52312026/11/12 00:33:22 WARN Request returned retryable status, retrying attempt=2 backoff=0s status=503 url=http://127.0.0.1:54331232--- PASS: TestScriptTokenScriptFails (0.01s)233=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)234=== PAUSE TestFilterOversizedClosures/no_limit_keeps_everything235=== RUN TestSetClientTLSErrors/missing_cert_file236=== PAUSE TestSetClientTLSErrors/missing_cert_file237=== RUN TestSetClientTLSErrors/missing_key_file238=== PAUSE TestSetClientTLSErrors/missing_key_file239=== RUN TestSetClientTLSErrors/missing_ca_file240=== PAUSE TestSetClientTLSErrors/missing_ca_file241=== RUN TestSetClientTLSErrors/invalid_ca_file242=== PAUSE TestSetClientTLSErrors/invalid_ca_file243=== RUN TestSetClientTLS/rejects_connection_without_client_cert244=== PAUSE TestSetClientTLS/rejects_connection_without_client_cert245=== RUN TestSetClientTLS/succeeds_with_client_cert_and_CA246=== PAUSE TestSetClientTLS/succeeds_with_client_cert_and_CA247=== RUN TestSetClientTLS/preserves_debug_logging_transport248=== CONT TestStreamPushAnswersEveryLineWhenCanceled249=== RUN TestRateLimiterFeedback/400_does_not_enable_limiter250=== PAUSE TestRateLimiterFeedback/400_does_not_enable_limiter251=== RUN TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part252=== PAUSE TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part253=== RUN TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary254=== PAUSE TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary255=== CONT TestClientSignaturesByStorePath256--- PASS: TestClientSignaturesByStorePath (0.00s)257=== PAUSE TestSetClientTLS/preserves_debug_logging_transport258=== RUN TestConvertHashToNix32/already_Nix32_format259=== CONT TestStreamPushReportsSignatures260=== RUN TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped261=== CONT TestStreamPushRequestLine2622026/11/12 00:33:22 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:54330263=== PAUSE TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped264=== CONT TestStreamPushReportsEveryPath265=== RUN TestFilterOversizedClosures/all_closures_skipped266=== PAUSE TestFilterOversizedClosures/all_closures_skipped267=== PAUSE TestConvertHashToNix32/already_Nix32_format2682000/01/01 00:00:00 ERROR Upload failed error="context canceled" count=22692026/11/12 00:33:22 ERROR Upload failed error=boom count=1270=== RUN TestConvertHashToNix32/invalid_format271=== CONT TestStreamPushIsolatesFailures272=== RUN TestPathInfoHashCompatibility/old_string_format_with_colon273--- PASS: TestDoWithRetry_BodyReplayedViaGetBody (0.01s)274=== CONT TestStreamPushReportsSkippedPaths275--- PASS: TestStreamPushAnswersEveryLineWhenCanceled (0.00s)276=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_colon277=== CONT TestShellSplitErrors278=== RUN TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI279=== CONT TestRegisterUploadedObjectReusesConnections280=== PAUSE TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI281=== RUN TestPathInfoHashCompatibility/new_structured_format_with_sha5122822026/11/12 00:33:22 ERROR Upload failed error=boom count=12832026/11/12 00:33:22 INFO Garbage collection started2842026/11/12 00:33:22 ERROR Upload failed error="bad path" count=3285=== PAUSE TestPathInfoHashCompatibility/new_structured_format_with_sha512286=== CONT TestStreamPushBatchesUnderLoad287=== RUN TestStreamPushBatchesUnderLoad/oneAtATime=false288=== PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=false289=== RUN TestStreamPushBatchesUnderLoad/oneAtATime=true290=== PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=true291=== CONT TestParsePathInfoJSON/Nix_format292=== CONT TestParsePathInfoJSON/invalid_JSON293=== CONT TestParsePathInfoJSON/whitespace_only294=== CONT TestParsePathInfoJSON/empty_input295=== CONT TestParsePathInfoJSON/Lix_format296=== CONT TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths297=== CONT TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths298=== CONT TestPartSizeForNAR/zero_stays_at_minimum299=== CONT TestPathInfoCACompatibility/null_ca_field300=== CONT TestPartSizeForNAR/capped_at_5_GiB301=== CONT TestPartSizeForNAR/small_stays_at_minimum302=== CONT TestPartSizeForNAR/1_TiB303=== CONT TestPartSizeForNAR/115_GiB_needs_larger_parts304=== CONT TestPartSizeForNAR/80_GiB_fits_at_minimum305=== CONT TestPathInfoCACompatibility/new_structured_format_-_text306=== CONT TestPathInfoCACompatibility/new_structured_format_-_nar_method307=== CONT TestPathInfoCACompatibility/old_string_format_-_fixed_recursive308=== CONT TestPathInfoCACompatibility/old_string_format_-_text309=== CONT TestGetStorePathHash/hash_with_invalid_characters_should_error310=== CONT TestGetStorePathHash/hash_with_wrong_length_should_error311=== CONT TestGetStorePathHash/basename_without_hyphen_should_error312=== CONT TestGetStorePathHash/valid_store_path313=== CONT TestSetClientTLSErrors/missing_cert_file314=== CONT TestSetClientTLSErrors/missing_ca_file315=== CONT TestSetClientTLSErrors/missing_key_file316=== CONT TestRateLimiterFeedback/429_enables_limiter317=== CONT TestRunGarbageCollection_FinishedOnAnotherReplica318=== CONT TestPartSizeForNAR/5_TiB_S3_max_object319=== CONT TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part320=== CONT TestStreamPushGivesUpOnDeadServer321--- PASS: TestStreamPushReportsEveryPath (0.00s)322--- PASS: TestStreamPushReportsSkippedPaths (0.00s)323--- PASS: TestShellSplitErrors (0.00s)324--- PASS: TestStreamPushReportsSignatures (0.00s)325--- PASS: TestRegisterUploadedObject_OutlivesPush (0.01s)326--- PASS: TestStreamPushIsolatesFailures (0.00s)327--- PASS: TestScriptTokenEmptyToken (0.02s)328=== CONT TestShellSplit329--- PASS: TestShellSplit (0.00s)330=== CONT TestSetClientTLSErrors/invalid_ca_file331=== CONT TestRateLimiterFeedback/400_does_not_enable_limiter332--- PASS: TestParsePathInfoJSON (0.00s)333 --- PASS: TestParsePathInfoJSON/Nix_format (0.00s)334 --- PASS: TestParsePathInfoJSON/invalid_JSON (0.00s)335 --- PASS: TestParsePathInfoJSON/whitespace_only (0.00s)336 --- PASS: TestParsePathInfoJSON/empty_input (0.00s)337 --- PASS: TestParsePathInfoJSON/Lix_format (0.00s)3382026/11/12 00:33:22 ERROR Upload failed error="connection refused" count=203392026/11/12 00:33:22 ERROR Server seems unavailable, giving up on batch untried=17340--- PASS: TestStreamPushGivesUpOnDeadServer (0.00s)341=== CONT TestRateLimiterFeedback/200_does_not_enable_limiter342=== PAUSE TestConvertHashToNix32/invalid_format343=== CONT TestDoWithRetry_FinalResponseBodyReadable344=== CONT TestRateLimiterFeedback/503_enables_limiter345=== CONT TestUploadBuildLog_FileBodyReplayedOnRetry346=== CONT TestEncodeNixBase32/test_string_hash347=== CONT TestSetClientTLS/rejects_connection_without_client_cert348=== CONT TestEncodeNixBase32/empty_input349=== CONT TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary350--- PASS: TestScriptTokenBadJSON (0.02s)351--- PASS: TestParsePathInfoJSONMultiplePaths (0.00s)352 --- PASS: TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths (0.00s)353 --- PASS: TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths (0.00s)354--- PASS: TestUploadMultipart_ReturnsAfterDroppedBody (0.01s)355--- PASS: TestPathInfoCACompatibility (0.00s)356 --- PASS: TestPathInfoCACompatibility/null_ca_field (0.00s)357 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_text (0.00s)358 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_nar_method (0.00s)359 --- PASS: TestPathInfoCACompatibility/old_string_format_-_fixed_recursive (0.00s)360 --- PASS: TestPathInfoCACompatibility/old_string_format_-_text (0.00s)361--- PASS: TestGetStorePathHash (0.00s)362 --- PASS: TestGetStorePathHash/hash_with_invalid_characters_should_error (0.00s)363 --- PASS: TestGetStorePathHash/hash_with_wrong_length_should_error (0.00s)364 --- PASS: TestGetStorePathHash/basename_without_hyphen_should_error (0.00s)365 --- PASS: TestGetStorePathHash/valid_store_path (0.00s)366--- PASS: TestPartSizeForNAR (0.00s)367 --- PASS: TestPartSizeForNAR/zero_stays_at_minimum (0.00s)368 --- PASS: TestPartSizeForNAR/capped_at_5_GiB (0.00s)369 --- PASS: TestPartSizeForNAR/small_stays_at_minimum (0.00s)370 --- PASS: TestPartSizeForNAR/1_TiB (0.00s)371 --- PASS: TestPartSizeForNAR/115_GiB_needs_larger_parts (0.00s)372 --- PASS: TestPartSizeForNAR/80_GiB_fits_at_minimum (0.00s)373 --- PASS: TestPartSizeForNAR/5_TiB_S3_max_object (0.00s)374--- PASS: TestEncodeNixBase32 (0.00s)375 --- PASS: TestEncodeNixBase32/test_string_hash (0.00s)376 --- PASS: TestEncodeNixBase32/empty_input (0.00s)3772026/11/12 00:33:22 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:543303782026/11/12 00:33:22 INFO Garbage collection started379=== CONT TestSetClientTLS/preserves_debug_logging_transport380--- PASS: TestUploadMultipart_FailedProducerIsNotCompleted (0.01s)381 --- PASS: TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part (0.00s)382 --- PASS: TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary (0.00s)383=== CONT TestSetClientTLS/succeeds_with_client_cert_and_CA3842026/11/12 00:33:22 WARN Rate limiter enabled after throttle name=server-test rate=53852026/11/12 00:33:22 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:544233862026/11/12 00:33:22 INFO Garbage collection progress phase=running_on_another_replica failed_uploads_deleted=0 old_closures_deleted=0 objects_marked=0 objects_deleted=0 objects_failed=03872026/11/12 00:33:22 WARN Rate limiter backed off name=server-test rate=5388--- PASS: TestDoWithRetry_FinalResponseBodyReadable (0.00s)389=== CONT TestFilterOversizedClosures/no_limit_keeps_everything390=== CONT TestFilterOversizedClosures/all_closures_skipped3912026/11/12 00:33:22 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper nar_size=100 max_nar_size=50392=== CONT TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped3932026/11/12 00:33:22 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb-vm-image nar_size=5000 max_nar_size=2000394--- PASS: TestFilterOversizedClosures (0.01s)395 --- PASS: TestFilterOversizedClosures/no_limit_keeps_everything (0.00s)396 --- PASS: TestFilterOversizedClosures/all_closures_skipped (0.00s)397 --- PASS: TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped (0.00s)398=== CONT TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)399=== CONT TestStreamPushBatchesUnderLoad/oneAtATime=false400=== CONT TestPathInfoHashCompatibility/new_structured_format_with_sha512401=== CONT TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI402=== CONT TestPathInfoHashCompatibility/old_string_format_with_colon403--- PASS: TestPathInfoHashCompatibility (0.01s)404 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI) (0.00s)405 --- PASS: TestPathInfoHashCompatibility/new_structured_format_with_sha512 (0.00s)406 --- PASS: TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI (0.00s)407 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_colon (0.00s)408=== CONT TestStreamPushBatchesUnderLoad/oneAtATime=true409--- PASS: TestStreamPushBatchesUnderLoad (0.00s)410 --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=false (0.00s)411 --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=true (0.00s)412=== CONT TestConvertHashToNix32/SRI_format_to_Nix32413=== CONT TestConvertHashToNix32/invalid_format414=== CONT TestConvertHashToNix32/already_Nix32_format415--- PASS: TestConvertHashToNix32 (0.02s)416 --- PASS: TestConvertHashToNix32/SRI_format_to_Nix32 (0.00s)417 --- PASS: TestConvertHashToNix32/invalid_format (0.00s)418 --- PASS: TestConvertHashToNix32/already_Nix32_format (0.00s)419--- PASS: TestRunGarbageCollection_NotFoundAfterLocalRun (0.02s)4202026/11/12 00:33:22 WARN Rate limiter enabled after throttle name=server-test rate=54212026/11/12 00:33:22 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:544244222026/11/12 00:33:22 WARN Rate limiter backed off name=server-test rate=5423--- PASS: TestSetClientTLSErrors (0.01s)424 --- PASS: TestSetClientTLSErrors/missing_cert_file (0.00s)425 --- PASS: TestSetClientTLSErrors/missing_key_file (0.00s)426 --- PASS: TestSetClientTLSErrors/invalid_ca_file (0.00s)427 --- PASS: TestSetClientTLSErrors/missing_ca_file (0.01s)4282026/11/12 00:33:22 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:543304292026/11/12 00:33:22 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:543304302026/11/12 00:33:22 WARN Garbage collection finished on another replica. Its statistics are not available here431--- PASS: TestRunGarbageCollection_FinishedOnAnotherReplica (0.01s)4322026/11/12 00:33:22 WARN Rate limiter enabled after throttle name=s3-test rate=54332026/11/12 00:33:22 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:54435434--- PASS: TestStreamPushRequestLine (0.03s)4352026/11/12 00:33:22 WARN Rate limiter enabled after throttle name=server-test rate=54362026/11/12 00:33:22 WARN Request returned retryable status, retrying attempt=3 backoff=1ms status=502 url=http://127.0.0.1:543304372026/11/12 00:33:22 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=429 url=http://127.0.0.1:54379438--- PASS: TestUploadBuildLog_FileBodyReplayedOnRetry (0.02s)4392026/11/12 00:33:22 WARN Rate limiter backed off name=server-test rate=5440--- PASS: TestRateLimiterFeedback (0.01s)441 --- PASS: TestRateLimiterFeedback/503_enables_limiter (0.01s)442 --- PASS: TestRateLimiterFeedback/200_does_not_enable_limiter (0.01s)443 --- PASS: TestRateLimiterFeedback/400_does_not_enable_limiter (0.01s)444 --- PASS: TestRateLimiterFeedback/429_enables_limiter (0.03s)4452026/11/12 00:33:22 WARN Request returned retryable status, retrying attempt=4 backoff=1ms status=502 url=http://127.0.0.1:543304462026/11/12 00:33:22 WARN Request returned retryable status, retrying attempt=5 backoff=1ms status=502 url=http://127.0.0.1:543304472026/11/12 00:33:22 WARN Request returned retryable status, retrying attempt=6 backoff=1ms status=502 url=http://127.0.0.1:543304482026/11/12 00:33:22 WARN Request returned retryable status, retrying attempt=7 backoff=1ms status=502 url=http://127.0.0.1:543304492026/11/12 00:33:22 WARN Request returned retryable status, retrying attempt=8 backoff=1ms status=502 url=http://127.0.0.1:54330450--- PASS: TestRegisterUploadedObjectReusesConnections (0.04s)4512026/11/12 00:33:22 WARN Request returned retryable status, retrying attempt=9 backoff=1ms status=502 url=http://127.0.0.1:543304522026/11/12 00:33:22 WARN Request returned retryable status, retrying attempt=10 backoff=1ms status=502 url=http://127.0.0.1:543304532026/11/12 00:33:22 WARN Request returned retryable status, retrying attempt=11 backoff=1ms status=502 url=http://127.0.0.1:543304542026/11/12 00:33:22 http: TLS handshake error from 127.0.0.1:54427: remote error: tls: bad certificate455--- PASS: TestSetClientTLS (0.01s)456 --- PASS: TestSetClientTLS/succeeds_with_client_cert_and_CA (0.00s)457 --- PASS: TestSetClientTLS/preserves_debug_logging_transport (0.00s)458 --- PASS: TestSetClientTLS/rejects_connection_without_client_cert (0.04s)4592026/11/12 00:33:22 WARN Request returned retryable status, retrying attempt=12 backoff=1ms status=502 url=http://127.0.0.1:543304602026/11/12 00:33:22 WARN Request returned retryable status, retrying attempt=13 backoff=1ms status=502 url=http://127.0.0.1:543304612026/11/12 00:33:22 WARN Request returned retryable status, retrying attempt=14 backoff=1ms status=502 url=http://127.0.0.1:54330462--- PASS: TestScriptTokenNoExpiryRerunsEveryCall (0.08s)4632026/11/12 00:33:22 WARN Request returned retryable status, retrying attempt=15 backoff=1ms status=502 url=http://127.0.0.1:543304642026/11/12 00:33:22 WARN Request returned retryable status, retrying attempt=16 backoff=1ms status=502 url=http://127.0.0.1:54330465--- PASS: TestScriptTokenCachesUntilRefresh (0.09s)4662026/11/12 00:33:22 WARN Request returned retryable status, retrying attempt=17 backoff=1ms status=502 url=http://127.0.0.1:54330467--- PASS: TestRetryTimeoutOutlastsMaxRetries (0.09s)468--- PASS: TestDumpPathWriterError (0.09s)4692026/11/12 00:33:22 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:54332/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:54332/api/uploads/complete4702026/11/12 00:33:22 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:54332/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:54332/api/uploads/complete4712026/11/12 00:33:22 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"4722026/11/12 00:33:22 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"4732026/11/12 00:33:22 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:54332/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:54332/api/uploads/complete4742026/11/12 00:33:22 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"475--- PASS: TestRegisterUploadedObject_BoundedAgainstSilentServer (0.10s)476--- PASS: TestDumpPathSingleFile (0.11s)477--- PASS: TestCaseHackSuffix (0.11s)478--- PASS: TestDumpPathMatchesNix (0.20s)479--- PASS: TestUploadMultipart_PartsInParallel (0.63s)480--- PASS: TestRateLimiterFeedback_400DoesNotCountAsSuccess (1.00s)481PASS482Running server tests...483The files belonging to this database system will be owned by user "_tribuchetbld5".484This user must also own the server process.485486The database cluster will be initialized with locale "C".487The default database encoding has accordingly been set to "SQL_ASCII".488The default text search configuration will be set to "english".489490Data page checksums are enabled.491492creating directory /private/var/tb/5/scratch/2e3e4b48f97c7cfb/build/postgres1326800213/data ... ok493creating subdirectories ... ok494selecting dynamic shared memory implementation ... posix495selecting default "max_connections" ... 100496selecting default "shared_buffers" ... 128MB497selecting default time zone ... UTC498creating configuration files ... ok499running bootstrap script ... ok500performing post-bootstrap initialization ... ok501syncing data to disk ... ok502503initdb: warning: enabling "trust" authentication for local connections504initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.505506Success. You can now start the database server using:507508 pg_ctl -D /private/var/tb/5/scratch/2e3e4b48f97c7cfb/build/postgres1326800213/data -l logfile start509510/private/var/tb/5/scratch/2e3e4b48f97c7cfb/build/postgres1326800213:5432 - no response5112026-11-12 00:33:24.654 UTC [4104] LOG: starting PostgreSQL 18.6 on aarch64-apple-darwin25.6.0, compiled by clang version 21.1.8, 64-bit5122026-11-12 00:33:24.654 UTC [4104] LOG: listening on Unix socket "/private/var/tb/5/scratch/2e3e4b48f97c7cfb/build/postgres1326800213/.s.PGSQL.5432"5132026-11-12 00:33:24.664 UTC [4112] LOG: database system was shut down at 2026-11-12 00:33:24 UTC5142026-11-12 00:33:24.665 UTC [4104] LOG: database system is ready to accept connections515/private/var/tb/5/scratch/2e3e4b48f97c7cfb/build/postgres1326800213:5432 - accepting connections516{"timestamp":"2026-11-12T00:33:24.898356Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"6938f78f-97df-466e-abc2-39744e1f1c1f","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"GET","uri":"/health/ready","status_code":503,"suppressed_errors":0,"duration_ms":10,"result":"server_error","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":502,"threadName":"rustfs-worker","threadId":"ThreadId(4)"}517{"timestamp":"2026-11-12T00:33:25.01288Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"4869b286-1d74-4283-835b-7483f071666f","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"GET","uri":"/health/ready","status_code":503,"suppressed_errors":0,"duration_ms":0,"result":"server_error","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":502,"threadName":"rustfs-worker","threadId":"ThreadId(4)"}518{"timestamp":"2026-11-12T00:33:25.115895Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"9f3181e2-aafd-4d07-a413-1a0605cbe68c","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"GET","uri":"/health/ready","status_code":503,"suppressed_errors":0,"duration_ms":0,"result":"server_error","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":502,"threadName":"rustfs-worker","threadId":"ThreadId(5)"}519=== RUN TestAbortSucceeded520=== PAUSE TestAbortSucceeded521=== RUN TestService_AuthMiddleware522=== PAUSE TestService_AuthMiddleware523=== RUN TestService_AuthMiddleware_MTLSProxyHeader524=== PAUSE TestService_AuthMiddleware_MTLSProxyHeader525=== RUN TestService_AuthMiddleware_MTLSBoundSubjects526=== PAUSE TestService_AuthMiddleware_MTLSBoundSubjects527=== RUN TestService_ReadAuthMiddleware528=== PAUSE TestService_ReadAuthMiddleware529=== RUN TestService_AuthMiddleware_OIDC530=== PAUSE TestService_AuthMiddleware_OIDC531=== RUN TestService_RequireScope_OIDC532=== PAUSE TestService_RequireScope_OIDC533=== RUN TestService_ReadScope_PublicByDefault534=== PAUSE TestService_ReadScope_PublicByDefault535=== RUN TestCacheConfigHandler536=== PAUSE TestCacheConfigHandler537=== RUN TestCacheStatsHandler538=== PAUSE TestCacheStatsHandler539=== RUN TestClientCADerivations540=== PAUSE TestClientCADerivations541=== RUN TestClientErrorHandling542=== PAUSE TestClientErrorHandling543=== RUN TestClientIntegration544=== PAUSE TestClientIntegration545=== RUN TestClientMultipleUploads546=== PAUSE TestClientMultipleUploads547=== RUN TestClientWithDependencies548=== PAUSE TestClientWithDependencies549=== RUN TestClientSharedPathCommittedMidPush550=== PAUSE TestClientSharedPathCommittedMidPush551=== RUN TestPinProtectsFromGC552=== PAUSE TestPinProtectsFromGC553=== RUN TestClientPushesUseOnePush554=== PAUSE TestClientPushesUseOnePush555=== RUN TestClientFallsBackToClosures556=== PAUSE TestClientFallsBackToClosures557=== RUN TestDeleteClosuresRacingPins558=== PAUSE TestDeleteClosuresRacingPins559=== RUN TestConcurrentCommitsSharingObjectsDoNotDeadlock560=== PAUSE TestConcurrentCommitsSharingObjectsDoNotDeadlock561=== RUN TestResolveDBConnectionString562=== PAUSE TestResolveDBConnectionString563=== RUN TestConnectWaitsForAPeerMigration564=== PAUSE TestConnectWaitsForAPeerMigration565=== RUN TestLeadElectsOneAndHandsOver566=== PAUSE TestLeadElectsOneAndHandsOver567=== RUN TestLeadIncumbentWinsAfterRestart5682026/11/12 00:33:25 INFO lead: acquired remote=192.0.2.1:1234 generation=15692026/11/12 00:33:26 INFO lead: released remote=192.0.2.1:1234 generation=15702026/11/12 00:33:26 INFO lead: acquired remote=192.0.2.1:1234 generation=25712026/11/12 00:33:27 INFO lead: released remote=192.0.2.1:1234 generation=2572--- PASS: TestLeadIncumbentWinsAfterRestart (2.04s)573=== RUN TestLeadUnannouncedStandbyYieldsToIncumbent574=== PAUSE TestLeadUnannouncedStandbyYieldsToIncumbent575=== RUN TestLeadIncumbentAnnouncesAtOnce576=== PAUSE TestLeadIncumbentAnnouncesAtOnce577=== RUN TestLeadGenerationGrowsWithEveryGrant578=== PAUSE TestLeadGenerationGrowsWithEveryGrant579=== RUN TestLeadEndsOnShutdown580=== PAUSE TestLeadEndsOnShutdown581=== RUN TestLeadEndsWhenItsConnectionHangs5822026/11/12 00:33:27 INFO lead: acquired remote=192.0.2.1:1234 generation=15832026-11-12 00:33:28.867 UTC [4187] FATAL: terminating connection due to administrator command5842026/11/12 00:33:29 INFO lead: acquired remote=192.0.2.1:1234 generation=25852026/11/12 00:33:30 WARN lead: lock connection lost error="timeout: context deadline exceeded"5862026/11/12 00:33:30 INFO lead: released remote=192.0.2.1:1234 generation=15872026/11/12 00:33:30 INFO lead: released remote=192.0.2.1:1234 generation=2588--- PASS: TestLeadEndsWhenItsConnectionHangs (2.99s)589=== RUN TestGCAdvisoryLockBlocksConcurrentRun590=== PAUSE TestGCAdvisoryLockBlocksConcurrentRun591=== RUN TestGCBugBareHashReferences592=== PAUSE TestGCBugBareHashReferences593=== RUN TestGCMetrics594=== PAUSE TestGCMetrics595=== RUN TestPushDedupSurvivesConcurrentGC596=== PAUSE TestPushDedupSurvivesConcurrentGC597=== RUN TestDeduplicatedObjectsRecordedAsPending598=== PAUSE TestDeduplicatedObjectsRecordedAsPending599=== RUN TestGCSweepSkipsPendingObjects600=== PAUSE TestGCSweepSkipsPendingObjects601=== RUN TestTombstonedObjectOfferedWithoutWaiting602=== PAUSE TestTombstonedObjectOfferedWithoutWaiting603=== RUN TestGCSweepDeliversEachKeyOnce604=== PAUSE TestGCSweepDeliversEachKeyOnce605=== RUN TestCreatePendingClosureVerifyS3FailureReleasesConnection606=== PAUSE TestCreatePendingClosureVerifyS3FailureReleasesConnection607=== RUN TestForceGCDuringPushOffersSweptObject608=== PAUSE TestForceGCDuringPushOffersSweptObject609=== RUN TestFailedS3DeleteKeepsTombstone610=== PAUSE TestFailedS3DeleteKeepsTombstone611=== RUN TestSweepSkipsRowsLockedByPush612=== PAUSE TestSweepSkipsRowsLockedByPush613=== RUN TestRegisterCompletedObjectNeedsPendingRow614=== PAUSE TestRegisterCompletedObjectNeedsPendingRow615=== RUN TestGetExistingObjectsLocksRows616=== PAUSE TestGetExistingObjectsLocksRows617=== RUN TestSweepRechecksPendingAfterLocking618=== PAUSE TestSweepRechecksPendingAfterLocking619=== RUN TestPushWaitsForSweepHoldingRow620=== PAUSE TestPushWaitsForSweepHoldingRow621=== RUN TestCommitRacingPendingCleanupKeepsObjects622=== PAUSE TestCommitRacingPendingCleanupKeepsObjects623=== RUN TestGCEndsOnShutdown624=== PAUSE TestGCEndsOnShutdown625=== RUN TestGCTaskStore_StartNew626=== PAUSE TestGCTaskStore_StartNew627=== RUN TestGCTaskStore_DeduplicateSameParams628=== PAUSE TestGCTaskStore_DeduplicateSameParams629=== RUN TestGCTaskStore_ConflictDifferentParams630=== PAUSE TestGCTaskStore_ConflictDifferentParams631=== RUN TestGCTaskStore_GetEmpty632=== PAUSE TestGCTaskStore_GetEmpty633=== RUN TestGCTaskStore_GetReturnsLatest634=== PAUSE TestGCTaskStore_GetReturnsLatest635=== RUN TestGCTaskStore_CompletedAllowsNewTask636=== PAUSE TestGCTaskStore_CompletedAllowsNewTask637=== RUN TestGCTaskStore_PhaseUpdates638=== PAUSE TestGCTaskStore_PhaseUpdates639=== RUN TestGCTaskStore_Fail640=== PAUSE TestGCTaskStore_Fail641=== RUN TestGracefulShutdownDrainsInflight642=== PAUSE TestGracefulShutdownDrainsInflight643=== RUN TestService_healthCheckHandler644=== PAUSE TestService_healthCheckHandler645=== RUN TestService_readinessHandler646=== PAUSE TestService_readinessHandler647=== RUN TestGenerateLandingPage648=== PAUSE TestGenerateLandingPage649=== RUN TestEnsureAbortUploadsRule650=== PAUSE TestEnsureAbortUploadsRule651=== RUN TestCacheConfigHandlerMaxNarSize652=== PAUSE TestCacheConfigHandlerMaxNarSize653=== RUN TestCreatePendingClosureRejectsOversizedNAR654=== PAUSE TestCreatePendingClosureRejectsOversizedNAR655=== RUN TestNARDeduplicationMetadataUploadBug656=== PAUSE TestNARDeduplicationMetadataUploadBug657=== RUN TestMetricsInventory658=== PAUSE TestMetricsInventory659=== RUN TestMetricsCommit660=== PAUSE TestMetricsCommit661=== RUN TestService_NativeMTLS662=== PAUSE TestService_NativeMTLS663=== RUN TestServerTLSConfig664=== PAUSE TestServerTLSConfig665=== RUN TestMultipartCleanup666=== PAUSE TestMultipartCleanup667=== RUN TestMultipartUploadAbortedWhenCancelledBeforeRecorded668=== PAUSE TestMultipartUploadAbortedWhenCancelledBeforeRecorded669=== RUN TestPendingCleanupUsesOneCutoff670=== PAUSE TestPendingCleanupUsesOneCutoff671=== RUN TestObjectStatsTrigger672=== PAUSE TestObjectStatsTrigger673=== RUN TestObjectStatsUpdatedOncePerCommit674=== PAUSE TestObjectStatsUpdatedOncePerCommit675=== RUN TestOrphanedObjectsGC676=== PAUSE TestOrphanedObjectsGC677=== RUN TestOrphanedObjectsGCStressTest678=== PAUSE TestOrphanedObjectsGCStressTest679=== RUN TestCreatePin_ReservedPins680=== PAUSE TestCreatePin_ReservedPins681=== RUN TestPresentReportsOnlyClosureRoots682=== PAUSE TestPresentReportsOnlyClosureRoots683=== RUN TestPresentRefreshesOnlyStaleClosures684=== PAUSE TestPresentRefreshesOnlyStaleClosures685=== RUN TestPresentNotReportedWhileGCDeletesClosure686=== PAUSE TestPresentNotReportedWhileGCDeletesClosure687=== RUN TestParseSingleRange688=== PAUSE TestParseSingleRange689=== RUN TestProxyHeadersOnlyTrustedOnSocket6902026/11/12 00:33:30 INFO Starting HTTP server address=127.0.0.1:544906912026/11/12 00:33:30 INFO Starting HTTP server address=proxy.sock6922026/11/12 00:33:30 WARN mTLS auth: subject not in bound subjects subject="CN=someone"6932026/11/12 00:33:30 INFO Shutdown signal received, draining in-flight requests timeout=10s694--- PASS: TestProxyHeadersOnlyTrustedOnSocket (0.25s)695=== RUN TestIsValidCachePath696=== PAUSE TestIsValidCachePath697=== RUN TestReadProxyNarinfo698=== PAUSE TestReadProxyNarinfo699=== RUN TestReadProxyNarinfoAlreadyDecompressed700=== PAUSE TestReadProxyNarinfoAlreadyDecompressed701=== RUN TestReadProxyContentNegotiation702=== PAUSE TestReadProxyContentNegotiation703=== RUN TestReadProxyNarStreaming704=== PAUSE TestReadProxyNarStreaming705=== RUN TestReadProxy404706=== PAUSE TestReadProxy404707=== RUN TestReadProxyInvalidPath708=== PAUSE TestReadProxyInvalidPath709=== RUN TestReadProxyHead710=== PAUSE TestReadProxyHead711=== RUN TestReadProxyOutlastsServerWriteTimeout712=== PAUSE TestReadProxyOutlastsServerWriteTimeout713=== RUN TestReadProxyConditionalGet714=== PAUSE TestReadProxyConditionalGet715=== RUN TestReadProxyRootRedirectsToIndexHTML716=== PAUSE TestReadProxyRootRedirectsToIndexHTML717=== RUN TestReadProxyDisabled718=== PAUSE TestReadProxyDisabled719=== RUN TestReadRedirectNar720=== PAUSE TestReadRedirectNar721=== RUN TestReadRedirectKeepsNarinfoProxied722=== PAUSE TestReadRedirectKeepsNarinfoProxied723=== RUN TestReadProxyRangeRequest724=== PAUSE TestReadProxyRangeRequest725=== RUN TestReadRedirectUsesPublicS3URL726=== PAUSE TestReadRedirectUsesPublicS3URL727=== RUN TestPush_OverlappingRootsStoreOneRowPerKey728=== PAUSE TestPush_OverlappingRootsStoreOneRowPerKey729=== RUN TestPush_CompleteCommitsEveryRoot730=== PAUSE TestPush_CompleteCommitsEveryRoot731=== RUN TestPush_RepeatedRootCommitsOnce732=== PAUSE TestPush_RepeatedRootCommitsOnce733=== RUN TestPush_CompleteWaitsForCleanupOfThePush734=== PAUSE TestPush_CompleteWaitsForCleanupOfThePush735=== RUN TestPush_RootsAreLockedInByteOrder736=== PAUSE TestPush_RootsAreLockedInByteOrder737=== RUN TestPush_SkippedKeySurvivesGCBeforeCommit738=== PAUSE TestPush_SkippedKeySurvivesGCBeforeCommit739=== RUN TestPush_RejectsBadRequests740=== PAUSE TestPush_RejectsBadRequests741=== RUN TestPush_SignsNarinfosOfItsPendingObjects742=== PAUSE TestPush_SignsNarinfosOfItsPendingObjects743=== RUN TestSharedNarBothUploadsComplete744=== PAUSE TestSharedNarBothUploadsComplete745=== RUN TestCompleteMultipartUpload_ErrorButObjectExists746=== PAUSE TestCompleteMultipartUpload_ErrorButObjectExists747=== RUN TestCompletedNarNotReofferedAcrossClosures748=== PAUSE TestCompletedNarNotReofferedAcrossClosures749=== RUN TestPresignedUploadRegisteredBeforeCommit750=== PAUSE TestPresignedUploadRegisteredBeforeCommit751=== RUN TestService_Rustfstest752=== PAUSE TestService_Rustfstest753=== RUN TestParseSize754=== PAUSE TestParseSize755=== RUN TestSkippedUploadsHandler756=== PAUSE TestSkippedUploadsHandler757=== RUN TestSystemdListenerNotActivated758--- PASS: TestSystemdListenerNotActivated (0.00s)759=== RUN TestWatchdogBeatsWhenHealthy760--- PASS: TestWatchdogBeatsWhenHealthy (0.03s)761=== RUN TestWatchdogSkipsWhenUnhealthy7622026/11/12 00:33:30 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7632026/11/12 00:33:30 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7642026/11/12 00:33:30 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7652026/11/12 00:33:30 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7662026/11/12 00:33:30 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7672026/11/12 00:33:30 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7682026/11/12 00:33:30 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7692026/11/12 00:33:30 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7702026/11/12 00:33:30 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7712026/11/12 00:33:30 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"772--- PASS: TestWatchdogSkipsWhenUnhealthy (0.21s)773=== RUN TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle774=== PAUSE TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle775=== RUN TestProxyWriteTimeout776=== PAUSE TestProxyWriteTimeout777=== RUN TestPendingClosureWriteTimeout778=== PAUSE TestPendingClosureWriteTimeout779=== RUN TestIsValidUploadKey780=== PAUSE TestIsValidUploadKey781=== RUN TestUploadHandlersRejectInvalidKeys782=== PAUSE TestUploadHandlersRejectInvalidKeys783=== RUN TestUploadHandlersRejectOversizedBody784=== PAUSE TestUploadHandlersRejectOversizedBody785=== RUN TestService_cleanupPendingClosuresHandler786=== PAUSE TestService_cleanupPendingClosuresHandler787=== RUN TestService_createPendingClosureHandler788=== PAUSE TestService_createPendingClosureHandler789=== RUN TestService_verifyS3Integrity790=== PAUSE TestService_verifyS3Integrity791=== RUN TestService_verifyS3ReuploadsStaleNarinfo792=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo793=== RUN TestCompleteMultipartUnregistered794=== PAUSE TestCompleteMultipartUnregistered795=== RUN TestCreatePendingClosure_SmallNARUsesSimplePUT796=== PAUSE TestCreatePendingClosure_SmallNARUsesSimplePUT797=== CONT TestPush_RejectsBadRequests798=== CONT TestPushWaitsForSweepHoldingRow799=== CONT TestAbortSucceeded800--- PASS: TestAbortSucceeded (0.00s)801=== CONT TestObjectStatsTrigger802=== CONT TestGCTaskStore_GetEmpty803--- PASS: TestGCTaskStore_GetEmpty (0.00s)804=== CONT TestPush_OverlappingRootsStoreOneRowPerKey805=== CONT TestService_NativeMTLS806=== CONT TestGCTaskStore_ConflictDifferentParams807--- PASS: TestGCTaskStore_ConflictDifferentParams (0.00s)808=== CONT TestMetricsCommit809=== CONT TestGCTaskStore_DeduplicateSameParams810--- PASS: TestGCTaskStore_DeduplicateSameParams (0.00s)811=== CONT TestPush_SkippedKeySurvivesGCBeforeCommit812=== CONT TestReadRedirectUsesPublicS3URL813=== CONT TestPush_RootsAreLockedInByteOrder814=== CONT TestPush_CompleteWaitsForCleanupOfThePush815=== CONT TestPush_RepeatedRootCommitsOnce816=== CONT TestGCTaskStore_StartNew817=== CONT TestPush_CompleteCommitsEveryRoot818=== CONT TestGCEndsOnShutdown819=== RUN TestGCEndsOnShutdown/before_the_run820=== PAUSE TestGCEndsOnShutdown/before_the_run821=== CONT TestOrphanedObjectsGCStressTest822=== RUN TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete823=== PAUSE TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete824=== CONT TestOrphanedObjectsGC825=== CONT TestObjectStatsUpdatedOncePerCommit826=== CONT TestCommitRacingPendingCleanupKeepsObjects827=== CONT TestPendingCleanupUsesOneCutoff828=== CONT TestMultipartUploadAbortedWhenCancelledBeforeRecorded829=== CONT TestMultipartCleanup830=== CONT TestCreatePin_ReservedPins831=== CONT TestServerTLSConfig832=== RUN TestServerTLSConfig/no_client_CA833=== PAUSE TestServerTLSConfig/no_client_CA834=== CONT TestGCTaskStore_GetReturnsLatest835--- PASS: TestGCTaskStore_StartNew (0.00s)836=== RUN TestServerTLSConfig/missing_CA_file837=== PAUSE TestServerTLSConfig/missing_CA_file838=== RUN TestServerTLSConfig/not_a_PEM_file839=== PAUSE TestServerTLSConfig/not_a_PEM_file840=== CONT TestMetricsInventory841=== RUN TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark842=== PAUSE TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark843=== CONT TestReadProxyRangeRequest844=== CONT TestReadRedirectNar845--- PASS: TestGCTaskStore_GetReturnsLatest (0.00s)846=== CONT TestReadRedirectKeepsNarinfoProxied8472026/11/12 00:33:31 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:54492/oidc848=== RUN TestPush_RejectsBadRequests/no_objects849=== PAUSE TestPush_RejectsBadRequests/no_objects850=== RUN TestPush_RejectsBadRequests/bad_root851=== PAUSE TestPush_RejectsBadRequests/bad_root852=== RUN TestPush_RejectsBadRequests/root_not_in_objects853=== PAUSE TestPush_RejectsBadRequests/root_not_in_objects854=== RUN TestPush_RejectsBadRequests/no_roots855=== PAUSE TestPush_RejectsBadRequests/no_roots856=== CONT TestReadProxyDisabled857--- PASS: TestObjectStatsTrigger (1.40s)858=== CONT TestNARDeduplicationMetadataUploadBug8592026/11/12 00:33:32 INFO Received push request method=POST path=/api/pushes860--- PASS: TestObjectStatsUpdatedOncePerCommit (1.91s)861=== CONT TestReadProxyRootRedirectsToIndexHTML8622026/11/12 00:33:32 INFO Received complete push request method=POST path=/api/pushes/1/complete8632026/11/12 00:33:32 INFO Completed push id=1 objects=6 duration=59.758833ms864--- PASS: TestPush_CompleteCommitsEveryRoot (2.06s)865=== CONT TestCreatePendingClosureRejectsOversizedNAR8662026/11/12 00:33:32 INFO Received uploads request method=POST path=/api/pending_closures867--- PASS: TestCreatePendingClosureRejectsOversizedNAR (0.00s)868=== CONT TestReadProxyConditionalGet8692026/11/12 00:33:33 INFO Aborted multipart uploads count=0 kept=08702026/11/12 00:33:33 WARN Force mode enabled - objects will be deleted immediately without grace period8712026/11/12 00:33:33 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=08722026/11/12 00:33:33 INFO Vacuumed table table=pending_closures8732026/11/12 00:33:33 INFO Vacuumed table table=pending_objects8742026/11/12 00:33:33 INFO Vacuumed table table=multipart_uploads8752026/11/12 00:33:33 INFO Vacuumed table table=closures8762026/11/12 00:33:33 INFO Vacuumed table table=objects877--- PASS: TestCommitRacingPendingCleanupKeepsObjects (2.42s)878=== CONT TestCacheConfigHandlerMaxNarSize879--- PASS: TestCacheConfigHandlerMaxNarSize (0.00s)880=== CONT TestReadProxyOutlastsServerWriteTimeout8812026/11/12 00:33:33 INFO Received push request method=POST path=/api/pushes882--- PASS: TestReadRedirectUsesPublicS3URL (2.69s)883=== CONT TestReadProxyHead8842026-11-12 00:33:33.568 UTC [4320] ERROR: Push does not exist: id=18852026-11-12 00:33:33.568 UTC [4320] CONTEXT: PL/pgSQL function commit_push(bigint) line 11 at RAISE8862026-11-12 00:33:33.568 UTC [4320] STATEMENT: SELECT commit_push($1)887--- PASS: TestPush_CompleteWaitsForCleanupOfThePush (2.80s)888=== CONT TestReadProxyInvalidPath8892026/11/12 00:33:33 WARN mTLS auth: subject not in bound subjects subject="CN=reader"8902026/11/12 00:33:33 WARN mTLS auth: subject not in bound subjects subject="CN=reader"891--- PASS: TestService_NativeMTLS (2.83s)892=== CONT TestReadProxy4048932026/11/12 00:33:33 INFO Received push request method=POST path=/api/pushes8942026/11/12 00:33:33 INFO Received complete push request method=POST path=/api/pushes/1/complete8952026/11/12 00:33:33 INFO Completed push id=1 objects=4 duration=19.227375ms8962026/11/12 00:33:33 INFO Received push request method=POST path=/api/pushes8972026/11/12 00:33:33 INFO Aborted multipart uploads count=0 kept=08982026/11/12 00:33:33 WARN Force mode enabled - objects will be deleted immediately without grace period8992026/11/12 00:33:34 INFO Received uploads request method=POST path=/api/pending_closures9002026/11/12 00:33:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=2 objects-deleted-after-grace-period=2 objects-failed-to-delete=09012026/11/12 00:33:34 INFO Vacuumed table table=pending_closures9022026/11/12 00:33:34 INFO Vacuumed table table=pending_objects9032026/11/12 00:33:34 INFO Vacuumed table table=multipart_uploads9042026/11/12 00:33:34 INFO Vacuumed table table=closures905--- PASS: TestMultipartUploadAbortedWhenCancelledBeforeRecorded (3.35s)9062026/11/12 00:33:34 INFO Vacuumed table table=objects907=== CONT TestReadProxyNarStreaming9082026/11/12 00:33:34 INFO Received complete push request method=POST path=/api/pushes/2/complete9092026/11/12 00:33:34 INFO Completed push id=2 objects=4 duration=5.014333ms910--- PASS: TestPush_SkippedKeySurvivesGCBeforeCommit (3.36s)911=== CONT TestReadProxyContentNegotiation9122026/11/12 00:33:34 INFO Received push request method=POST path=/api/pushes9132026-11-12 00:33:34.582 UTC [4360] ERROR: could not obtain lock on row in relation "closures"9142026-11-12 00:33:34.582 UTC [4360] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT9152026-11-12 00:33:34.585 UTC [4360] ERROR: could not obtain lock on row in relation "closures"9162026-11-12 00:33:34.585 UTC [4360] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT9172026-11-12 00:33:34.587 UTC [4360] ERROR: could not obtain lock on row in relation "closures"9182026-11-12 00:33:34.587 UTC [4360] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT9192026-11-12 00:33:34.589 UTC [4360] ERROR: could not obtain lock on row in relation "closures"9202026-11-12 00:33:34.589 UTC [4360] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT9212026-11-12 00:33:34.594 UTC [4360] ERROR: could not obtain lock on row in relation "closures"9222026-11-12 00:33:34.594 UTC [4360] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT923--- PASS: TestPush_RootsAreLockedInByteOrder (3.84s)924=== CONT TestReadProxyNarinfoAlreadyDecompressed9252026/11/12 00:33:34 INFO Received uploads request method=POST path=/api/pending_closures926--- PASS: TestReadProxyRangeRequest (4.31s)927=== CONT TestReadProxyNarinfo9282026/11/12 00:33:35 INFO Received push request method=POST path=/api/pushes929--- PASS: TestPushWaitsForSweepHoldingRow (4.53s)930=== CONT TestIsValidCachePath931=== RUN TestIsValidCachePath/narinfo932=== PAUSE TestIsValidCachePath/narinfo933=== RUN TestIsValidCachePath/narinfo_all_nix_base32_chars934=== PAUSE TestIsValidCachePath/narinfo_all_nix_base32_chars935=== RUN TestIsValidCachePath/nar_zst936=== PAUSE TestIsValidCachePath/nar_zst937=== RUN TestIsValidCachePath/nar_xz938=== PAUSE TestIsValidCachePath/nar_xz939=== RUN TestIsValidCachePath/nar_bz2940=== PAUSE TestIsValidCachePath/nar_bz2941=== RUN TestIsValidCachePath/nar_uncompressed942=== PAUSE TestIsValidCachePath/nar_uncompressed943=== RUN TestIsValidCachePath/ls944=== PAUSE TestIsValidCachePath/ls945=== RUN TestIsValidCachePath/log946=== PAUSE TestIsValidCachePath/log947=== RUN TestIsValidCachePath/realisation948=== PAUSE TestIsValidCachePath/realisation949=== RUN TestIsValidCachePath/nix-cache-info950=== PAUSE TestIsValidCachePath/nix-cache-info951=== RUN TestIsValidCachePath/index.html952=== PAUSE TestIsValidCachePath/index.html953=== RUN TestIsValidCachePath/traversal_parent954=== PAUSE TestIsValidCachePath/traversal_parent955=== RUN TestIsValidCachePath/traversal_in_middle956=== PAUSE TestIsValidCachePath/traversal_in_middle957=== RUN TestIsValidCachePath/invalid_char_e958=== PAUSE TestIsValidCachePath/invalid_char_e959=== RUN TestIsValidCachePath/invalid_char_u960=== PAUSE TestIsValidCachePath/invalid_char_u961=== RUN TestIsValidCachePath/random_path962=== PAUSE TestIsValidCachePath/random_path963=== RUN TestIsValidCachePath/empty964=== PAUSE TestIsValidCachePath/empty965=== RUN TestIsValidCachePath/leading_slash966=== PAUSE TestIsValidCachePath/leading_slash967=== RUN TestIsValidCachePath/wrong_extension968=== PAUSE TestIsValidCachePath/wrong_extension969=== RUN TestIsValidCachePath/short_hash970=== PAUSE TestIsValidCachePath/short_hash971=== CONT TestParseSingleRange972=== RUN TestParseSingleRange/none973=== PAUSE TestParseSingleRange/none974=== RUN TestParseSingleRange/unknown_unit975=== PAUSE TestParseSingleRange/unknown_unit976=== RUN TestParseSingleRange/multi-range_ignored977=== PAUSE TestParseSingleRange/multi-range_ignored978=== RUN TestParseSingleRange/malformed_no_dash979=== PAUSE TestParseSingleRange/malformed_no_dash980=== RUN TestParseSingleRange/malformed_both_empty981=== PAUSE TestParseSingleRange/malformed_both_empty982=== RUN TestParseSingleRange/malformed_end_before_start983=== PAUSE TestParseSingleRange/malformed_end_before_start984=== RUN TestParseSingleRange/closed985=== PAUSE TestParseSingleRange/closed986=== RUN TestParseSingleRange/open-ended987=== PAUSE TestParseSingleRange/open-ended988=== RUN TestParseSingleRange/end_clamped_to_size989=== PAUSE TestParseSingleRange/end_clamped_to_size990=== RUN TestParseSingleRange/suffix991=== PAUSE TestParseSingleRange/suffix992=== RUN TestParseSingleRange/suffix_exceeds_size993=== PAUSE TestParseSingleRange/suffix_exceeds_size994=== RUN TestParseSingleRange/single_byte995=== PAUSE TestParseSingleRange/single_byte996=== RUN TestParseSingleRange/start_past_EOF997=== PAUSE TestParseSingleRange/start_past_EOF998=== RUN TestParseSingleRange/start_far_past_EOF999=== PAUSE TestParseSingleRange/start_far_past_EOF1000=== CONT TestPresentNotReportedWhileGCDeletesClosure1001=== RUN TestPresentNotReportedWhileGCDeletesClosure/stale1002=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/stale1003=== RUN TestPresentNotReportedWhileGCDeletesClosure/fresh1004=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/fresh1005=== CONT TestEnsureAbortUploadsRule10062026/11/12 00:33:35 INFO Received complete push request method=POST path=/api/pushes/1/complete10072026/11/12 00:33:35 INFO Completed push id=1 objects=4 duration=22.109459ms10082026/11/12 00:33:35 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete10092026/11/12 00:33:35 INFO Completed upload id=2 objects=2 duration=907.083µs1010--- PASS: TestMetricsCommit (4.67s)1011=== CONT TestPresentRefreshesOnlyStaleClosures1012=== NAME TestOrphanedObjectsGC1013 orphaned_objects_gc_test.go:287: GC Test Summary:1014 orphaned_objects_gc_test.go:288: - Kept: 2 objects from closure A1015 orphaned_objects_gc_test.go:289: - Deleted: 2 objects from closure B1016 orphaned_objects_gc_test.go:290: - Deleted: 6 orphaned chain objects (X1->X2->X3)1017 orphaned_objects_gc_test.go:291: - Deleted: 2 orphaned single objects (Y)1018 orphaned_objects_gc_test.go:292: - Total deleted: 10 objects1019--- PASS: TestOrphanedObjectsGC (5.19s)1020=== CONT TestGenerateLandingPage1021--- PASS: TestGenerateLandingPage (0.00s)1022=== CONT TestService_readinessHandler1023--- PASS: TestReadRedirectKeepsNarinfoProxied (5.37s)1024=== CONT TestService_healthCheckHandler10252026/11/12 00:33:36 INFO Received push request method=POST path=/api/pushes10262026/11/12 00:33:36 INFO Received complete push request method=POST path=/api/pushes/1/complete10272026/11/12 00:33:36 INFO Completed push id=1 objects=4 duration=55.425958ms1028--- PASS: TestPush_RepeatedRootCommitsOnce (5.71s)1029=== CONT TestGracefulShutdownDrainsInflight10302026/11/12 00:33:36 INFO Starting HTTP server address=127.0.0.1:5456310312026/11/12 00:33:36 INFO Shutdown signal received, draining in-flight requests timeout=10s1032--- PASS: TestGracefulShutdownDrainsInflight (0.07s)1033=== CONT TestGCTaskStore_Fail1034--- PASS: TestGCTaskStore_Fail (0.00s)1035=== CONT TestGCTaskStore_PhaseUpdates1036--- PASS: TestGCTaskStore_PhaseUpdates (0.00s)1037=== CONT TestGCTaskStore_CompletedAllowsNewTask1038--- PASS: TestGCTaskStore_CompletedAllowsNewTask (0.00s)1039=== CONT TestPresentReportsOnlyClosureRoots10402026/11/12 00:33:36 INFO Received uploads request method=POST path=/api/pending_closures10412026/11/12 00:33:36 INFO Received cleanup request method=DELETE path=/api/pending_closures10422026/11/12 00:33:36 WARN Failed to abort upload, keeping its closure key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst error="Get \"http://127.0.0.1:1/bucket22/?location=\": dial tcp 127.0.0.1:1: connect: connection refused" code=""10432026/11/12 00:33:36 INFO Aborted multipart uploads count=0 kept=110442026/11/12 00:33:36 INFO Received cleanup request method=DELETE path=/api/pending_closures10452026/11/12 00:33:36 INFO Aborted multipart uploads count=1 kept=01046--- PASS: TestMultipartCleanup (6.07s)1047=== CONT TestPendingClosureWriteTimeout1048=== RUN TestPendingClosureWriteTimeout/empty1049=== PAUSE TestPendingClosureWriteTimeout/empty1050=== RUN TestPendingClosureWriteTimeout/400_objects1051=== PAUSE TestPendingClosureWriteTimeout/400_objects1052=== RUN TestPendingClosureWriteTimeout/670k_objects1053=== PAUSE TestPendingClosureWriteTimeout/670k_objects1054=== RUN TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline1055=== PAUSE TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline1056=== CONT TestCreatePendingClosure_SmallNARUsesSimplePUT10572026/11/12 00:33:36 INFO Received push request method=POST path=/api/pushes1058--- PASS: TestPush_OverlappingRootsStoreOneRowPerKey (6.28s)1059=== CONT TestCompleteMultipartUnregistered1060--- PASS: TestMetricsInventory (6.59s)1061=== CONT TestService_verifyS3ReuploadsStaleNarinfo1062=== RUN TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed1063=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed1064=== RUN TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed1065=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed1066=== RUN TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost1067=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost1068=== RUN TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed1069=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed1070=== RUN TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed1071=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed1072=== CONT TestService_verifyS3Integrity1073--- PASS: TestReadRedirectNar (6.85s)1074=== CONT TestService_createPendingClosureHandler10752026/11/12 00:33:37 INFO Received uploads request method=POST path=/api/pending_closures10762026/11/12 00:33:38 INFO Received uploads request method=POST path=/api/pending_closures10772026/11/12 00:33:38 INFO Received cleanup request method=DELETE path=/api/pending_closures10782026/11/12 00:33:38 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux10792026/11/12 00:33:38 WARN Refused reserved pin name=worker-x86_64-linux10802026/11/12 00:33:38 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux10812026/11/12 00:33:38 INFO Received create pin request method=POST path=/api/pins/my-app10822026/11/12 00:33:38 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux1083--- PASS: TestCreatePin_ReservedPins (7.59s)1084=== CONT TestService_cleanupPendingClosuresHandler1085=== NAME TestNARDeduplicationMetadataUploadBug1086 metadata_upload_test.go:48: First store path: /private/var/tb/5/scratch/2e3e4b48f97c7cfb/build/TestNARDeduplicationMetadataUploadBug1916096006/001/store/pidsfhv2wfb7d0km6w575ndxw4rcq68j-file1.txt1087--- PASS: TestReadProxyDisabled (7.17s)1088=== CONT TestUploadHandlersRejectOversizedBody1089=== RUN TestUploadHandlersRejectOversizedBody/create_pending_closure1090=== PAUSE TestUploadHandlersRejectOversizedBody/create_pending_closure1091=== RUN TestUploadHandlersRejectOversizedBody/complete_multipart1092=== PAUSE TestUploadHandlersRejectOversizedBody/complete_multipart1093=== RUN TestUploadHandlersRejectOversizedBody/request_more_parts1094=== PAUSE TestUploadHandlersRejectOversizedBody/request_more_parts1095=== CONT TestUploadHandlersRejectInvalidKeys1096=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info1097=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info1098=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal1099=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal1100=== RUN TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key1101=== PAUSE TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key1102=== RUN TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key1103=== PAUSE TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key1104=== CONT TestIsValidUploadKey1105=== RUN TestIsValidUploadKey/narinfo1106=== PAUSE TestIsValidUploadKey/narinfo1107=== RUN TestIsValidUploadKey/nar_zst1108=== PAUSE TestIsValidUploadKey/nar_zst1109=== RUN TestIsValidUploadKey/nar_xz1110=== PAUSE TestIsValidUploadKey/nar_xz1111=== RUN TestIsValidUploadKey/nar_plain1112=== PAUSE TestIsValidUploadKey/nar_plain1113=== RUN TestIsValidUploadKey/listing1114=== PAUSE TestIsValidUploadKey/listing1115=== RUN TestIsValidUploadKey/build_log1116=== PAUSE TestIsValidUploadKey/build_log1117=== RUN TestIsValidUploadKey/build_log_home-manager_file1118=== PAUSE TestIsValidUploadKey/build_log_home-manager_file1119=== RUN TestIsValidUploadKey/build_log_plus_in_name1120=== PAUSE TestIsValidUploadKey/build_log_plus_in_name1121=== RUN TestIsValidUploadKey/build_log_question_mark1122=== PAUSE TestIsValidUploadKey/build_log_question_mark1123=== RUN TestIsValidUploadKey/build_log_equals1124=== PAUSE TestIsValidUploadKey/build_log_equals1125=== RUN TestIsValidUploadKey/realisation1126=== PAUSE TestIsValidUploadKey/realisation1127=== RUN TestIsValidUploadKey/realisation_plus_in_output1128=== PAUSE TestIsValidUploadKey/realisation_plus_in_output1129=== RUN TestIsValidUploadKey/nix-cache-info1130=== PAUSE TestIsValidUploadKey/nix-cache-info1131=== RUN TestIsValidUploadKey/index.html1132=== PAUSE TestIsValidUploadKey/index.html1133=== RUN TestIsValidUploadKey/narinfo_key,_nar_type1134=== PAUSE TestIsValidUploadKey/narinfo_key,_nar_type1135=== RUN TestIsValidUploadKey/nar_key,_narinfo_type1136=== PAUSE TestIsValidUploadKey/nar_key,_narinfo_type1137=== RUN TestIsValidUploadKey/listing_key,_narinfo_type1138=== PAUSE TestIsValidUploadKey/listing_key,_narinfo_type1139=== RUN TestIsValidUploadKey/traversal1140=== PAUSE TestIsValidUploadKey/traversal1141=== RUN TestIsValidUploadKey/traversal_nar1142=== PAUSE TestIsValidUploadKey/traversal_nar1143=== RUN TestIsValidUploadKey/absolute1144=== PAUSE TestIsValidUploadKey/absolute1145=== RUN TestIsValidUploadKey/empty_key1146=== PAUSE TestIsValidUploadKey/empty_key1147=== RUN TestIsValidUploadKey/unknown_type1148=== PAUSE TestIsValidUploadKey/unknown_type1149=== CONT TestConnectWaitsForAPeerMigration11502026/11/12 00:33:39 INFO Received push request method=POST path=/api/pushes11512026/11/12 00:33:39 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)11522026/11/12 00:33:39 INFO Uploading pidsfhv2wfb7d0km6w575ndxw4rcq68j-file1.txt (160B)11532026/11/12 00:33:39 WARN Failed to register uploaded object key=nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst error="server returned 404: 404 page not found\n"11542026/11/12 00:33:39 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign11552026/11/12 00:33:39 WARN Failed to register uploaded object key=pidsfhv2wfb7d0km6w575ndxw4rcq68j.ls error="server returned 404: 404 page not found\n"11562026/11/12 00:33:39 INFO Signed narinfos id=1 count=111572026/11/12 00:33:39 INFO Uploading 1 narinfos11582026/11/12 00:33:39 INFO Received complete push request method=POST path=/api/pushes/1/complete11592026/11/12 00:33:39 WARN Failed to register uploaded object key=pidsfhv2wfb7d0km6w575ndxw4rcq68j.narinfo error="server returned 404: 404 page not found\n"1160--- PASS: TestReadProxyRootRedirectsToIndexHTML (6.67s)1161=== CONT TestSweepRechecksPendingAfterLocking11622026/11/12 00:33:39 INFO Completed push id=1 objects=3 duration=32.290459ms11632026/11/12 00:33:39 INFO Upload complete. (382ms)1164=== NAME TestNARDeduplicationMetadataUploadBug1165 metadata_upload_test.go:54: Retrieved narinfo from S3:1166 StorePath: /private/var/tb/5/scratch/2e3e4b48f97c7cfb/build/TestNARDeduplicationMetadataUploadBug1916096006/001/store/pidsfhv2wfb7d0km6w575ndxw4rcq68j-file1.txt1167 URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst1168 Compression: zstd1169 NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1170 NarSize: 1601171 References: 1172 CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1173 metadata_upload_test.go:55: Retrieved .ls file from S3 (compressed size: 77 bytes)1174 metadata_upload_test.go:55: Decompressed .ls content (64 bytes):1175 {"version":1,"root":{"type":"regular","size":44,"narOffset":96}}1176 metadata_upload_test.go:64: Second store path (same content): /private/var/tb/5/scratch/2e3e4b48f97c7cfb/build/TestNARDeduplicationMetadataUploadBug1916096006/001/store/8wdlsa03wqm1r9k5dwb5nak83lki39xd-file2.txt1177--- PASS: TestReadProxyConditionalGet (6.82s)1178=== CONT TestGetExistingObjectsLocksRows11792026/11/12 00:33:39 INFO Received push request method=POST path=/api/pushes11802026/11/12 00:33:39 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)11812026/11/12 00:33:39 INFO Received sign narinfos request method=POST path=/api/pushes/2/sign11822026/11/12 00:33:39 WARN Failed to register uploaded object key=8wdlsa03wqm1r9k5dwb5nak83lki39xd.ls error="server returned 404: 404 page not found\n"11832026/11/12 00:33:39 INFO Signed narinfos id=2 count=111842026/11/12 00:33:39 INFO Uploading 1 narinfos11852026/11/12 00:33:39 INFO Received complete push request method=POST path=/api/pushes/2/complete11862026/11/12 00:33:39 WARN Failed to register uploaded object key=8wdlsa03wqm1r9k5dwb5nak83lki39xd.narinfo error="server returned 404: 404 page not found\n"11872026/11/12 00:33:39 INFO Completed push id=2 objects=3 duration=1.288708ms11882026/11/12 00:33:39 INFO Upload complete. (205ms)1189=== NAME TestNARDeduplicationMetadataUploadBug1190 metadata_upload_test.go:76: Retrieved narinfo from S3:1191 StorePath: /private/var/tb/5/scratch/2e3e4b48f97c7cfb/build/TestNARDeduplicationMetadataUploadBug1916096006/001/store/8wdlsa03wqm1r9k5dwb5nak83lki39xd-file2.txt1192 URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst1193 Compression: zstd1194 NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1195 NarSize: 1601196 References: 1197 CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1198 metadata_upload_test.go:77: Retrieved .ls file from S3 (compressed size: 62 bytes)1199 metadata_upload_test.go:77: Decompressed .ls content (49 bytes):1200 {"version":1,"root":{"type":"regular","size":44}}1201--- PASS: TestNARDeduplicationMetadataUploadBug (7.68s)1202=== CONT TestRegisterCompletedObjectNeedsPendingRow1203--- PASS: TestReadProxyHead (6.76s)1204=== CONT TestSweepSkipsRowsLockedByPush1205--- PASS: TestReadProxyInvalidPath (6.87s)1206=== CONT TestFailedS3DeleteKeepsTombstone1207--- PASS: TestReadProxyContentNegotiation (7.14s)1208=== CONT TestForceGCDuringPushOffersSweptObject1209=== RUN TestForceGCDuringPushOffersSweptObject/before_pending_rows1210=== PAUSE TestForceGCDuringPushOffersSweptObject/before_pending_rows1211=== RUN TestForceGCDuringPushOffersSweptObject/after_presence_check1212=== PAUSE TestForceGCDuringPushOffersSweptObject/after_presence_check1213=== CONT TestCreatePendingClosureVerifyS3FailureReleasesConnection1214--- PASS: TestReadProxy404 (7.77s)1215=== CONT TestGCSweepDeliversEachKeyOnce1216--- PASS: TestReadProxyNarStreaming (7.39s)1217=== CONT TestTombstonedObjectOfferedWithoutWaiting1218--- PASS: TestReadProxyNarinfoAlreadyDecompressed (7.19s)1219=== CONT TestGCSweepSkipsPendingObjects12202026/11/12 00:33:42 ERROR Failed to decode narinfo key=3hcdxyjf9yiq7qf3i4548drb6sjmwa1v.narinfo limit=16777216 error=<nil>1221--- PASS: TestReadProxyNarinfo (7.12s)1222=== CONT TestDeduplicatedObjectsRecordedAsPending1223--- PASS: TestPresentRefreshesOnlyStaleClosures (7.03s)1224=== CONT TestPushDedupSurvivesConcurrentGC12252026/11/12 00:33:42 INFO Installed lifecycle rule aborting incomplete NAR uploads bucket=bucket41 days=21226--- PASS: TestEnsureAbortUploadsRule (7.50s)1227=== CONT TestGCMetrics12282026/11/12 00:33:43 WARN readiness check failed error="closed pool"1229--- PASS: TestService_readinessHandler (7.09s)1230=== CONT TestGCBugBareHashReferences1231--- PASS: TestService_healthCheckHandler (7.32s)1232=== CONT TestGCAdvisoryLockBlocksConcurrentRun1233--- PASS: TestPresentReportsOnlyClosureRoots (7.34s)1234=== CONT TestLeadEndsOnShutdown1235--- PASS: TestReadProxyOutlastsServerWriteTimeout (10.98s)1236=== CONT TestLeadGenerationGrowsWithEveryGrant12372026/11/12 00:33:44 INFO Aborted multipart uploads count=1 kept=012382026/11/12 00:33:44 INFO Received complete multipart upload request method=POST path=/api/multipart/complete12392026/11/12 00:33:44 ERROR Failed to get multipart upload error="no rows in result set" upload_id=does-not-exist object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst1240--- PASS: TestCompleteMultipartUnregistered (7.14s)1241=== CONT TestLeadIncumbentAnnouncesAtOnce12422026/11/12 00:33:44 INFO Received cleanup request method=DELETE path=/api/pending_closures12432026/11/12 00:33:44 INFO Aborted multipart uploads count=1 kept=01244--- PASS: TestPendingCleanupUsesOneCutoff (13.46s)1245=== CONT TestLeadUnannouncedStandbyYieldsToIncumbent12462026/11/12 00:33:44 INFO Received uploads request method=POST path=/api/pending_closures1247--- PASS: TestCreatePendingClosure_SmallNARUsesSimplePUT (7.86s)1248=== CONT TestLeadElectsOneAndHandsOver12492026/11/12 00:33:44 INFO Received uploads request method=POST path=/api/pending_closures12502026/11/12 00:33:45 INFO Received uploads request method=POST path=/api/pending_closures12512026/11/12 00:33:45 INFO Received uploads request method=POST path=/api/pending_closures12522026/11/12 00:33:45 INFO Received uploads request method=POST path=/api/pending_closures12532026/11/12 00:33:45 INFO Received cleanup request method=DELETE path=/api/pending_closures12542026/11/12 00:33:45 INFO Aborted multipart uploads count=0 kept=012552026/11/12 00:33:45 INFO Received uploads request method=POST path=/api/pending_closures12562026/11/12 00:33:46 INFO Received cleanup request method=DELETE path=/api/pending_closures12572026/11/12 00:33:46 INFO Aborted multipart uploads count=1 kept=012582026/11/12 00:33:46 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete12592026-11-12 00:33:46.076 UTC [4556] ERROR: Closure does not exist: id=112602026-11-12 00:33:46.076 UTC [4556] CONTEXT: PL/pgSQL function commit_pending_closure(bigint) line 13 at RAISE12612026-11-12 00:33:46.076 UTC [4556] STATEMENT: -- name: CommitPendingClosure :one1262 SELECT commit_pending_closure($1::bigint)::bigint1263 1264--- PASS: TestService_cleanupPendingClosuresHandler (7.72s)1265=== CONT TestService_Rustfstest12662026/11/12 00:33:46 INFO Aborted multipart uploads count=0 kept=012672026/11/12 00:33:46 WARN Force mode enabled - objects will be deleted immediately without grace period12682026/11/12 00:33:46 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=012692026/11/12 00:33:46 INFO Vacuumed table table=pending_closures12702026/11/12 00:33:46 INFO Vacuumed table table=pending_objects12712026/11/12 00:33:46 INFO Vacuumed table table=multipart_uploads12722026/11/12 00:33:46 INFO Vacuumed table table=closures12732026/11/12 00:33:46 INFO Vacuumed table table=objects1274--- PASS: TestSweepRechecksPendingAfterLocking (7.39s)1275=== CONT TestProxyWriteTimeout1276=== RUN TestProxyWriteTimeout/narinfo1277=== PAUSE TestProxyWriteTimeout/narinfo1278=== RUN TestProxyWriteTimeout/1_GiB_nar1279=== PAUSE TestProxyWriteTimeout/1_GiB_nar1280=== RUN TestProxyWriteTimeout/10_GiB_nar1281=== PAUSE TestProxyWriteTimeout/10_GiB_nar1282=== RUN TestProxyWriteTimeout/unknown_size1283=== PAUSE TestProxyWriteTimeout/unknown_size1284=== CONT TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle1285--- PASS: TestGetExistingObjectsLocksRows (7.15s)1286=== CONT TestSkippedUploadsHandler12872026/11/12 00:33:46 INFO Client skipped oversized paths paths=3 nar_bytes=50000000001288--- PASS: TestSkippedUploadsHandler (0.00s)1289=== CONT TestClientErrorHandling1290=== RUN TestClientErrorHandling/InvalidStorePath1291=== PAUSE TestClientErrorHandling/InvalidStorePath1292=== RUN TestClientErrorHandling/InvalidAuthToken1293=== PAUSE TestClientErrorHandling/InvalidAuthToken1294=== RUN TestClientErrorHandling/ServerNotAvailable1295=== PAUSE TestClientErrorHandling/ServerNotAvailable1296=== CONT TestParseSize1297--- PASS: TestParseSize (0.00s)1298=== CONT TestPinProtectsFromGC1299--- PASS: TestConnectWaitsForAPeerMigration (8.14s)1300=== CONT TestClientWithDependencies13012026/11/12 00:33:47 INFO Received complete multipart upload request method=POST path=/api/multipart/complete1302--- PASS: TestRegisterCompletedObjectNeedsPendingRow (7.37s)1303=== CONT TestClientSharedPathCommittedMidPush13042026/11/12 00:33:47 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst upload_id=MDNiMjFmNDUtYWZkMC00ZDllLTk0M2UtNWYyMTE4ODJhZGZkLjg1ODNlZDE0LTFjY2QtNDBlZi1hYWMzLWQyMmZkZDJmODhkN3gxNzk0NDQzNjI1MDE4NzA4MDAw parts=1013052026/11/12 00:33:47 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete13062026/11/12 00:33:47 INFO Completed upload id=1 objects=2 duration=34.036291ms13072026/11/12 00:33:47 INFO Received uploads request method=POST path=/api/pending_closures13082026/11/12 00:33:47 INFO Received uploads request method=POST path=/api/pending_closures13092026/11/12 00:33:47 INFO Object in database but missing from S3 key=dadb44fdadb44fdadb44fdadb44f0000.narinfo13102026/11/12 00:33:47 WARN Found objects in DB but missing from S3, will re-upload count=11311--- PASS: TestService_verifyS3Integrity (10.02s)1312=== CONT TestClientPushesUseOnePush13132026/11/12 00:33:47 INFO Received complete multipart upload request method=POST path=/api/multipart/complete13142026/11/12 00:33:47 INFO Completed multipart upload object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst upload_id=MDNiMjFmNDUtYWZkMC00ZDllLTk0M2UtNWYyMTE4ODJhZGZkLmE5ZTAyMmUyLTg0NTAtNDhjYy04ODcxLTYxZWE5NzgwNTY4Y3gxNzk0NDQzNjI1NDM3ODgzMDAw parts=1013152026/11/12 00:33:47 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete13162026/11/12 00:33:47 INFO Aborted multipart uploads count=0 kept=013172026/11/12 00:33:47 INFO Completed upload id=1 objects=2 duration=30.515709ms13182026/11/12 00:33:47 INFO Received get closure request method=GET path=/api/closures/0000000000000000000000000000000013192026/11/12 00:33:47 WARN Force mode enabled - objects will be deleted immediately without grace period13202026/11/12 00:33:47 INFO Received uploads request method=POST path=/api/pending_closures13212026/11/12 00:33:47 INFO Starting cleanup of old closures method=DELETE path=/api/closures13222026/11/12 00:33:47 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=013232026/11/12 00:33:47 INFO Vacuumed table table=pending_closures13242026/11/12 00:33:47 INFO Vacuumed table table=pending_objects13252026/11/12 00:33:47 INFO Vacuumed table table=multipart_uploads13262026/11/12 00:33:47 INFO Vacuumed table table=closures13272026/11/12 00:33:47 INFO Aborted multipart uploads count=0 kept=013282026/11/12 00:33:47 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=013292026/11/12 00:33:47 INFO Vacuumed table table=objects13302026/11/12 00:33:47 INFO Aborted multipart uploads count=0 kept=013312026/11/12 00:33:47 WARN Force mode enabled - objects will be deleted immediately without grace period13322026/11/12 00:33:47 INFO Vacuumed table table=pending_closures13332026/11/12 00:33:47 INFO Vacuumed table table=pending_objects13342026/11/12 00:33:48 INFO Vacuumed table table=multipart_uploads13352026/11/12 00:33:48 INFO Received uploads request method=POST path=/api/pending_closures13362026/11/12 00:33:48 INFO Vacuumed table table=closures13372026/11/12 00:33:48 INFO Vacuumed table table=objects13382026/11/12 00:33:48 INFO Received get closure request method=GET path=/api/closures/000000000000000000000000000000001339--- PASS: TestService_createPendingClosureHandler (10.54s)1340=== CONT TestClientMultipleUploads13412026/11/12 00:33:48 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=013422026/11/12 00:33:48 INFO Vacuumed table table=pending_closures13432026/11/12 00:33:48 INFO Vacuumed table table=pending_objects13442026/11/12 00:33:48 INFO Vacuumed table table=multipart_uploads13452026/11/12 00:33:48 INFO Vacuumed table table=closures13462026/11/12 00:33:48 INFO Vacuumed table table=objects1347--- PASS: TestSweepSkipsRowsLockedByPush (8.10s)1348=== CONT TestResolveDBConnectionString1349=== RUN TestResolveDBConnectionString/flag_wins1350=== PAUSE TestResolveDBConnectionString/flag_wins1351=== RUN TestResolveDBConnectionString/file_when_flag_empty1352=== PAUSE TestResolveDBConnectionString/file_when_flag_empty1353=== RUN TestResolveDBConnectionString/missing_file_is_an_error1354=== PAUSE TestResolveDBConnectionString/missing_file_is_an_error1355=== RUN TestResolveDBConnectionString/PGHOST_allows_empty1356=== PAUSE TestResolveDBConnectionString/PGHOST_allows_empty1357=== RUN TestResolveDBConnectionString/nothing_configured1358=== PAUSE TestResolveDBConnectionString/nothing_configured1359=== CONT TestClientIntegration13602026/11/12 00:33:48 INFO Aborted multipart uploads count=0 kept=013612026/11/12 00:33:48 WARN Force mode enabled - objects will be deleted immediately without grace period13622026/11/12 00:33:48 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."13632026/11/12 00:33:48 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."1364--- PASS: TestFailedS3DeleteKeepsTombstone (8.01s)1365=== CONT TestService_RequireScope_OIDC13662026/11/12 00:33:48 INFO Received uploads request method=POST path=/api/pending_closures1367--- PASS: TestCreatePendingClosureVerifyS3FailureReleasesConnection (7.49s)1368=== CONT TestConcurrentCommitsSharingObjectsDoNotDeadlock13692026/11/12 00:33:48 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:54698/oidc1370=== NAME TestOrphanedObjectsGCStressTest1371 orphaned_objects_gc_test.go:422: Created 10 active closures, 5 to-delete closures, 20 orphaned chains13722026/11/12 00:33:49 INFO Aborted multipart uploads count=0 kept=01373 orphaned_objects_gc_test.go:440: Marked 210 objects for deletion13742026/11/12 00:33:49 WARN Force mode enabled - objects will be deleted immediately without grace period13752026/11/12 00:33:49 INFO Received uploads request method=POST path=/api/pending_closures1376--- PASS: TestTombstonedObjectOfferedWithoutWaiting (7.80s)1377=== CONT TestCompleteMultipartUpload_ErrorButObjectExists13782026/11/12 00:33:49 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=013792026/11/12 00:33:49 INFO Vacuumed table table=pending_closures13802026/11/12 00:33:49 INFO Vacuumed table table=pending_objects13812026/11/12 00:33:49 INFO Received uploads request method=POST path=/api/pending_closures13822026/11/12 00:33:49 INFO Vacuumed table table=multipart_uploads13832026/11/12 00:33:49 INFO Vacuumed table table=closures13842026/11/12 00:33:49 INFO Vacuumed table table=objects1385--- PASS: TestGCSweepDeliversEachKeyOnce (8.38s)1386=== CONT TestDeleteClosuresRacingPins13872026/11/12 00:33:49 INFO Aborted multipart uploads count=0 kept=013882026/11/12 00:33:49 WARN Force mode enabled - objects will be deleted immediately without grace period13892026/11/12 00:33:49 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=013902026/11/12 00:33:49 INFO Vacuumed table table=pending_closures13912026/11/12 00:33:49 INFO Vacuumed table table=pending_objects13922026/11/12 00:33:49 INFO Vacuumed table table=multipart_uploads13932026/11/12 00:33:50 INFO Vacuumed table table=closures13942026/11/12 00:33:50 INFO Vacuumed table table=objects13952026/11/12 00:33:50 INFO Received uploads request method=POST path=/api/pending_closures1396--- PASS: TestGCSweepSkipsPendingObjects (8.28s)1397=== CONT TestClientFallsBackToClosures1398--- PASS: TestDeduplicatedObjectsRecordedAsPending (7.93s)1399=== CONT TestPresignedUploadRegisteredBeforeCommit14002026/11/12 00:33:50 INFO Received uploads request method=POST path=/api/pending_closures14012026/11/12 00:33:50 INFO Aborted multipart uploads count=0 kept=014022026/11/12 00:33:50 INFO Aborted multipart uploads count=0 kept=014032026/11/12 00:33:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=014042026/11/12 00:33:50 WARN Force mode enabled - objects will be deleted immediately without grace period14052026/11/12 00:33:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=014062026/11/12 00:33:50 INFO Vacuumed table table=pending_closures14072026/11/12 00:33:50 INFO Vacuumed table table=pending_closures14082026/11/12 00:33:50 INFO Vacuumed table table=pending_objects14092026/11/12 00:33:50 INFO Vacuumed table table=multipart_uploads14102026/11/12 00:33:50 INFO Vacuumed table table=closures14112026/11/12 00:33:50 INFO Vacuumed table table=objects1412--- PASS: TestGCMetrics (8.07s)1413=== CONT TestCompletedNarNotReofferedAcrossClosures14142026/11/12 00:33:50 INFO Vacuumed table table=pending_objects14152026/11/12 00:33:50 INFO Vacuumed table table=multipart_uploads14162026/11/12 00:33:50 INFO Vacuumed table table=closures14172026/11/12 00:33:51 INFO Vacuumed table table=objects14182026/11/12 00:33:51 INFO Aborted multipart uploads count=0 kept=014192026/11/12 00:33:51 WARN Force mode enabled - objects will be deleted immediately without grace period14202026/11/12 00:33:51 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=014212026/11/12 00:33:51 INFO Vacuumed table table=pending_closures14222026/11/12 00:33:51 INFO Vacuumed table table=pending_objects14232026/11/12 00:33:51 INFO Vacuumed table table=multipart_uploads14242026/11/12 00:33:51 INFO Vacuumed table table=closures14252026/11/12 00:33:51 INFO Vacuumed table table=objects1426--- PASS: TestPushDedupSurvivesConcurrentGC (8.70s)1427=== CONT TestSharedNarBothUploadsComplete1428--- PASS: TestGCBugBareHashReferences (8.44s)1429=== CONT TestClientCADerivations14302026/11/12 00:33:51 ERROR failed to check GC advisory lock error="query pg_locks: failed to connect to `user=_tribuchetbld5 database=niks3`: /nonexistent/.s.PGSQL.5432 (/nonexistent): dial error: dial unix /nonexistent/.s.PGSQL.5432: connect: no such file or directory"1431--- PASS: TestGCAdvisoryLockBlocksConcurrentRun (8.19s)1432=== CONT TestCacheConfigHandler1433=== RUN TestCacheConfigHandler/full_config,_no_issuer1434=== PAUSE TestCacheConfigHandler/full_config,_no_issuer1435=== RUN TestCacheConfigHandler/no_cache_url_configured1436=== PAUSE TestCacheConfigHandler/no_cache_url_configured1437=== RUN TestCacheConfigHandler/no_signing_keys1438=== PAUSE TestCacheConfigHandler/no_signing_keys1439=== RUN TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1440=== PAUSE TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1441=== CONT TestService_ReadScope_PublicByDefault14422026/11/12 00:33:51 INFO lead: acquired remote=192.0.2.1:1234 generation=114432026/11/12 00:33:52 INFO lead: acquired remote=192.0.2.1:1234 generation=114442026/11/12 00:33:52 INFO lead: acquired remote=192.0.2.1:1234 generation=114452026/11/12 00:33:52 INFO lead: released remote=192.0.2.1:1234 generation=11446--- PASS: TestLeadIncumbentAnnouncesAtOnce (8.36s)1447=== CONT TestCacheStatsHandler14482026/11/12 00:33:52 INFO lead: acquired remote=192.0.2.1:1234 generation=114492026/11/12 00:33:53 INFO lead: released remote=192.0.2.1:1234 generation=11450--- PASS: TestLeadEndsOnShutdown (9.10s)1451=== CONT TestPush_SignsNarinfosOfItsPendingObjects14522026/11/12 00:33:53 INFO lead: acquired remote=192.0.2.1:1234 generation=114532026/11/12 00:33:53 INFO lead: released remote=192.0.2.1:1234 generation=114542026/11/12 00:33:53 INFO lead: acquired remote=192.0.2.1:1234 generation=21455--- PASS: TestService_Rustfstest (7.32s)1456=== CONT TestService_AuthMiddleware_MTLSBoundSubjects14572026/11/12 00:33:53 INFO Received uploads request method=POST path=/api/pending_closures14582026/11/12 00:33:53 INFO lead: released remote=192.0.2.1:1234 generation=114592026/11/12 00:33:53 INFO lead: acquired remote=192.0.2.1:1234 generation=214602026/11/12 00:33:54 INFO lead: given back to the incumbent remote=192.0.2.1:1234 generation=214612026/11/12 00:33:54 INFO lead: acquired remote=192.0.2.1:1234 generation=314622026/11/12 00:33:54 INFO Received complete multipart upload request method=POST path=/api/multipart/complete14632026/11/12 00:33:54 INFO lead: released remote=192.0.2.1:1234 generation=114642026/11/12 00:33:54 INFO lead: acquired remote=192.0.2.1:1234 generation=214652026/11/12 00:33:54 INFO lead: released remote=192.0.2.1:1234 generation=214662026/11/12 00:33:54 INFO lead: acquired remote=192.0.2.1:1234 generation=314672026/11/12 00:33:54 INFO lead: released remote=192.0.2.1:1234 generation=31468--- PASS: TestLeadGenerationGrowsWithEveryGrant (10.30s)1469=== CONT TestService_AuthMiddleware_MTLSProxyHeader1470=== NAME TestPinProtectsFromGC1471 client_integration_test.go:791: Pinned store path: /private/var/tb/5/scratch/2e3e4b48f97c7cfb/build/TestPinProtectsFromGC2593085542/001/store/ijgix19kdz184gc5ly26l7x7nznni2cm-pinned-file.txt1472 client_integration_test.go:792: Unpinned store path: /private/var/tb/5/scratch/2e3e4b48f97c7cfb/build/TestPinProtectsFromGC2593085542/001/store/lrs2mb4gz6vsbirjfrm80zy4slx1jh60-unpinned-file.txt14732026/11/12 00:33:54 INFO Received push request method=POST path=/api/pushes14742026/11/12 00:33:54 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)14752026/11/12 00:33:54 INFO Uploading ijgix19kdz184gc5ly26l7x7nznni2cm-pinned-file.txt (128B)14762026/11/12 00:33:54 WARN Failed to register uploaded object key=ijgix19kdz184gc5ly26l7x7nznni2cm.ls error="server returned 404: 404 page not found\n"14772026/11/12 00:33:54 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign14782026/11/12 00:33:54 WARN Failed to register uploaded object key=nar/0qmhsg42pqsz7x6hh02j8gznnac3yi3qpjhrqbwvz88i194jbb5s.nar.zst error="server returned 404: 404 page not found\n"14792026/11/12 00:33:54 INFO Signed narinfos id=1 count=114802026/11/12 00:33:54 INFO Uploading 1 narinfos14812026/11/12 00:33:54 INFO Received complete push request method=POST path=/api/pushes/1/complete14822026/11/12 00:33:54 WARN Failed to register uploaded object key=ijgix19kdz184gc5ly26l7x7nznni2cm.narinfo error="server returned 404: 404 page not found\n"14832026/11/12 00:33:54 INFO Completed push id=1 objects=3 duration=39.2755ms14842026/11/12 00:33:54 INFO Upload complete. (359ms)14852026/11/12 00:33:54 INFO Received push request method=POST path=/api/pushes14862026/11/12 00:33:54 INFO Uploading 2 paths to 127.0.0.1 (0 already cached)14872026/11/12 00:33:54 INFO Uploading w4jbjf9v5hdwaqi4dggcpw6n9yfd89l6-shared-dep (136B)14882026/11/12 00:33:54 INFO Uploading jk6qn6vpy6xm22djmyifrhqb7v8w6zid-top (264B)14892026/11/12 00:33:55 WARN Failed to register uploaded object key=w4jbjf9v5hdwaqi4dggcpw6n9yfd89l6.ls error="server returned 404: 404 page not found\n"14902026/11/12 00:33:55 WARN Failed to register uploaded object key=jk6qn6vpy6xm22djmyifrhqb7v8w6zid.ls error="server returned 404: 404 page not found\n"14912026/11/12 00:33:55 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"14922026/11/12 00:33:55 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign14932026/11/12 00:33:55 WARN Failed to register uploaded object key=nar/0sjcsy697ly183yk7gvc2nrg8vnpkp1flayyga4v4ildr0pbvrb8.nar.zst error="server returned 404: 404 page not found\n"14942026/11/12 00:33:55 INFO Signed narinfos id=1 count=214952026/11/12 00:33:55 INFO Uploading 2 narinfos14962026/11/12 00:33:55 INFO Received push request method=POST path=/api/pushes14972026/11/12 00:33:55 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)14982026/11/12 00:33:55 INFO Uploading lrs2mb4gz6vsbirjfrm80zy4slx1jh60-unpinned-file.txt (128B)14992026/11/12 00:33:55 WARN Failed to register uploaded object key=jk6qn6vpy6xm22djmyifrhqb7v8w6zid.narinfo error="server returned 404: 404 page not found\n"15002026/11/12 00:33:55 WARN Failed to register uploaded object key=w4jbjf9v5hdwaqi4dggcpw6n9yfd89l6.narinfo error="server returned 404: 404 page not found\n"15012026/11/12 00:33:55 INFO Received complete push request method=POST path=/api/pushes/1/complete15022026/11/12 00:33:55 WARN Failed to register uploaded object key=lrs2mb4gz6vsbirjfrm80zy4slx1jh60.ls error="server returned 404: 404 page not found\n"15032026/11/12 00:33:55 INFO Completed push id=1 objects=6 duration=22.975542ms15042026/11/12 00:33:55 INFO Upload complete. (329ms)1505=== NAME TestClientSharedPathCommittedMidPush15062026/11/12 00:33:55 INFO Received sign narinfos request method=POST path=/api/pushes/2/sign1507 client_integration_test.go:740: Retrieved narinfo from S3:15082026/11/12 00:33:55 WARN Failed to register uploaded object key=nar/0gh0wdnr6gmk4r5sfzzh12isnl14rgmkrwy1ws41hpsrlndclg5h.nar.zst error="server returned 404: 404 page not found\n"1509 StorePath: /private/var/tb/5/scratch/2e3e4b48f97c7cfb/build/TestClientSharedPathCommittedMidPush561803849/001/store/w4jbjf9v5hdwaqi4dggcpw6n9yfd89l6-shared-dep1510 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1511 Compression: zstd1512 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821513 NarSize: 1361514 References: 1515 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n15162026/11/12 00:33:55 INFO Signed narinfos id=2 count=115172026/11/12 00:33:55 INFO Uploading 1 narinfos1518 client_integration_test.go:740: Retrieved narinfo from S3:1519 StorePath: /private/var/tb/5/scratch/2e3e4b48f97c7cfb/build/TestClientSharedPathCommittedMidPush561803849/001/store/jk6qn6vpy6xm22djmyifrhqb7v8w6zid-top1520 URL: nar/0sjcsy697ly183yk7gvc2nrg8vnpkp1flayyga4v4ildr0pbvrb8.nar.zst1521 Compression: zstd1522 NarHash: sha256:0sjcsy697ly183yk7gvc2nrg8vnpkp1flayyga4v4ildr0pbvrb81523 NarSize: 2641524 References: /private/var/tb/5/scratch/2e3e4b48f97c7cfb/build/TestClientSharedPathCommittedMidPush561803849/001/store/w4jbjf9v5hdwaqi4dggcpw6n9yfd89l6-shared-dep1525 CA: text:sha256:01a09088rjkwh3v7rw2kb6f87z9bpmh6j7xby9zsrqnhbh0jzy1h15262026/11/12 00:33:55 INFO Received complete push request method=POST path=/api/pushes/2/complete15272026/11/12 00:33:55 WARN Failed to register uploaded object key=lrs2mb4gz6vsbirjfrm80zy4slx1jh60.narinfo error="server returned 404: 404 page not found\n"15282026/11/12 00:33:55 INFO Completed push id=2 objects=3 duration=1.478917ms15292026/11/12 00:33:55 INFO Upload complete. (213ms)1530--- PASS: TestClientSharedPathCommittedMidPush (7.99s)1531=== CONT TestService_AuthMiddleware1532=== NAME TestClientWithDependencies1533 client_integration_test.go:661: Built derivation: /private/var/tb/5/scratch/2e3e4b48f97c7cfb/build/TestClientWithDependencies863834502/001/store/q0gk296m70dqc3aabl8lh422gix9f79x-test-script15342026/11/12 00:33:55 INFO Starting cleanup of old closures method=DELETE path=/api/closures15352026/11/12 00:33:55 INFO Garbage collection started1536 client_integration_test.go:663: Found 1 dependencies (including self)15372026/11/12 00:33:55 INFO Aborted multipart uploads count=0 kept=015382026/11/12 00:33:55 WARN Force mode enabled - objects will be deleted immediately without grace period15392026/11/12 00:33:55 INFO lead: released remote=192.0.2.1:1234 generation=21540--- PASS: TestLeadElectsOneAndHandsOver (10.79s)1541=== CONT TestService_AuthMiddleware_OIDC15422026/11/12 00:33:55 INFO Received push request method=POST path=/api/pushes15432026/11/12 00:33:55 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)15442026/11/12 00:33:55 INFO Uploading q0gk296m70dqc3aabl8lh422gix9f79x-test-script (136B)1545=== NAME TestClientMultipleUploads1546 client_integration_test.go:406: Created store path 0: /private/var/tb/5/scratch/2e3e4b48f97c7cfb/build/TestClientMultipleUploads3855158667/001/store/gv306ifg3av94cvwprfsif9m3nz9zpdd-test-file-0.txt15472026/11/12 00:33:55 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:54777/oidc15482026/11/12 00:33:55 WARN Failed to register uploaded object key=q0gk296m70dqc3aabl8lh422gix9f79x.ls error="server returned 404: 404 page not found\n"15492026/11/12 00:33:55 WARN Failed to register uploaded object key=log/cbr1wk2yck6fl61c0xjf0cl96crmk494-test-script.drv error="server returned 404: 404 page not found\n"15502026/11/12 00:33:55 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign15512026/11/12 00:33:55 WARN Failed to register uploaded object key=nar/1s7yghixzpinyqm7g59kyn3sav81pmnsdi5v63m6x4r2vacy907j.nar.zst error="server returned 404: 404 page not found\n"15522026/11/12 00:33:55 INFO Signed narinfos id=1 count=115532026/11/12 00:33:55 INFO Uploading 1 narinfos15542026/11/12 00:33:55 INFO Received push request method=POST path=/api/pushes15552026/11/12 00:33:55 INFO Received complete push request method=POST path=/api/pushes/1/complete15562026/11/12 00:33:55 WARN Failed to register uploaded object key=q0gk296m70dqc3aabl8lh422gix9f79x.narinfo error="server returned 404: 404 page not found\n"15572026/11/12 00:33:55 INFO Completed push id=1 objects=4 duration=23.019917ms15582026/11/12 00:33:55 INFO Uploading 2 paths to 127.0.0.1 (1 already cached)15592026/11/12 00:33:55 INFO Uploading v6kly0cjagmdbg75jhy3ss503vr1dvpa-b (256B)15602026/11/12 00:33:55 INFO Uploading vrn333yqxc4pslcprjiqpjlra9p246vh-shared-dep (136B)15612026/11/12 00:33:55 INFO Upload complete. (369ms)1562 client_integration_test.go:406: Created store path 1: /private/var/tb/5/scratch/2e3e4b48f97c7cfb/build/TestClientMultipleUploads3855158667/001/store/1bizpl56dhymhbck7qbn9k43gz02msd0-test-file-1.txt15632026/11/12 00:33:55 INFO Aborted multipart uploads count=0 kept=015642026/11/12 00:33:55 WARN Force mode enabled - objects will be deleted immediately without grace period15652026/11/12 00:33:55 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=015662026/11/12 00:33:55 INFO lead: released remote=192.0.2.1:1234 generation=31567--- PASS: TestLeadUnannouncedStandbyYieldsToIncumbent (11.67s)1568=== CONT TestService_ReadAuthMiddleware15692026/11/12 00:33:55 WARN Failed to register uploaded object key=9q861ggdwib0yr32pzwgnr7swn3piajg.ls error="server returned 404: 404 page not found\n"15702026/11/12 00:33:55 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=015712026/11/12 00:33:55 WARN Failed to register uploaded object key=v6kly0cjagmdbg75jhy3ss503vr1dvpa.ls error="server returned 404: 404 page not found\n"15722026/11/12 00:33:55 WARN Failed to register uploaded object key=nar/0safks1b6q56614vg79jm5s48shv4dca664hy0mpn84ia5dr3prr.nar.zst error="server returned 404: 404 page not found\n"15732026/11/12 00:33:55 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"15742026/11/12 00:33:55 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign15752026/11/12 00:33:55 WARN Failed to register uploaded object key=vrn333yqxc4pslcprjiqpjlra9p246vh.ls error="server returned 404: 404 page not found\n"15762026/11/12 00:33:55 INFO Vacuumed table table=pending_closures15772026/11/12 00:33:55 INFO Signed narinfos id=1 count=315782026/11/12 00:33:55 INFO Uploading 3 narinfos15792026/11/12 00:33:55 INFO Vacuumed table table=pending_closures15802026/11/12 00:33:55 INFO Vacuumed table table=pending_objects15812026/11/12 00:33:55 INFO Vacuumed table table=multipart_uploads15822026/11/12 00:33:55 WARN Failed to register uploaded object key=v6kly0cjagmdbg75jhy3ss503vr1dvpa.narinfo error="server returned 404: 404 page not found\n"15832026/11/12 00:33:56 WARN Failed to register uploaded object key=9q861ggdwib0yr32pzwgnr7swn3piajg.narinfo error="server returned 404: 404 page not found\n"15842026/11/12 00:33:56 INFO Vacuumed table table=pending_objects15852026/11/12 00:33:56 INFO Vacuumed table table=multipart_uploads15862026/11/12 00:33:56 INFO Vacuumed table table=closures15872026/11/12 00:33:56 INFO Received complete push request method=POST path=/api/pushes/1/complete15882026/11/12 00:33:56 WARN Failed to register uploaded object key=vrn333yqxc4pslcprjiqpjlra9p246vh.narinfo error="server returned 404: 404 page not found\n"1589=== NAME TestClientMultipleUploads1590 client_integration_test.go:406: Created store path 2: /private/var/tb/5/scratch/2e3e4b48f97c7cfb/build/TestClientMultipleUploads3855158667/001/store/w9w7nf83xn3f0byycbrx9n2xb7y7sb2y-test-file-2.txt15912026/11/12 00:33:56 INFO Completed push id=1 objects=8 duration=19.66ms15922026/11/12 00:33:56 INFO Vacuumed table table=objects15932026/11/12 00:33:56 INFO Upload complete. (409ms)15942026/11/12 00:33:56 INFO Vacuumed table table=closures1595=== NAME TestClientIntegration1596 client_integration_test.go:292: Created store path: /private/var/tb/5/scratch/2e3e4b48f97c7cfb/build/TestClientIntegration1081111093/002/store/ppxgnvb27dhckgxyssp3ib55ijcmmh1l-test-file.txt1597=== NAME TestClientPushesUseOnePush1598 client_pushes_test.go:97: Retrieved narinfo from S3:1599 StorePath: /private/var/tb/5/scratch/2e3e4b48f97c7cfb/build/TestClientPushesUseOnePush3655011277/001/store/vrn333yqxc4pslcprjiqpjlra9p246vh-shared-dep1600 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1601 Compression: zstd1602 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821603 NarSize: 1361604 References: 1605 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n1606=== NAME TestClientWithDependencies1607 client_integration_test.go:677: Skipping nix copy test - isolated store (/private/var/tb/5/scratch/2e3e4b48f97c7cfb/build/TestClientWithDependencies863834502/001/store) requires matching store prefix16082026/11/12 00:33:56 INFO Vacuumed table table=objects1609=== NAME TestClientPushesUseOnePush1610 client_pushes_test.go:97: Retrieved narinfo from S3:1611 StorePath: /private/var/tb/5/scratch/2e3e4b48f97c7cfb/build/TestClientPushesUseOnePush3655011277/001/store/9q861ggdwib0yr32pzwgnr7swn3piajg-a1612 URL: nar/0safks1b6q56614vg79jm5s48shv4dca664hy0mpn84ia5dr3prr.nar.zst1613 Compression: zstd1614 NarHash: sha256:0safks1b6q56614vg79jm5s48shv4dca664hy0mpn84ia5dr3prr1615 NarSize: 2561616 References: /private/var/tb/5/scratch/2e3e4b48f97c7cfb/build/TestClientPushesUseOnePush3655011277/001/store/vrn333yqxc4pslcprjiqpjlra9p246vh-shared-dep1617 CA: text:sha256:0g6bn17w7mzgmqrhky57gpd9nd48117cw3lfdm4fi4lq6zp63nlk1618 client_pushes_test.go:97: Retrieved narinfo from S3:1619 StorePath: /private/var/tb/5/scratch/2e3e4b48f97c7cfb/build/TestClientPushesUseOnePush3655011277/001/store/v6kly0cjagmdbg75jhy3ss503vr1dvpa-b1620 URL: nar/0safks1b6q56614vg79jm5s48shv4dca664hy0mpn84ia5dr3prr.nar.zst1621 Compression: zstd1622 NarHash: sha256:0safks1b6q56614vg79jm5s48shv4dca664hy0mpn84ia5dr3prr1623 NarSize: 2561624 References: /private/var/tb/5/scratch/2e3e4b48f97c7cfb/build/TestClientPushesUseOnePush3655011277/001/store/vrn333yqxc4pslcprjiqpjlra9p246vh-shared-dep1625 CA: text:sha256:0g6bn17w7mzgmqrhky57gpd9nd48117cw3lfdm4fi4lq6zp63nlk1626--- PASS: TestClientWithDependencies (8.94s)1627=== CONT TestServerTLSConfig/no_client_CA1628=== CONT TestServerTLSConfig/not_a_PEM_file1629=== CONT TestServerTLSConfig/missing_CA_file1630--- PASS: TestServerTLSConfig (0.00s)1631 --- PASS: TestServerTLSConfig/no_client_CA (0.00s)1632 --- PASS: TestServerTLSConfig/not_a_PEM_file (0.00s)1633 --- PASS: TestServerTLSConfig/missing_CA_file (0.00s)1634=== CONT TestGCEndsOnShutdown/before_the_run1635--- PASS: TestClientPushesUseOnePush (8.71s)1636=== CONT TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark1637=== RUN TestService_RequireScope_OIDC/builder_may_write1638=== PAUSE TestService_RequireScope_OIDC/builder_may_write1639=== RUN TestService_RequireScope_OIDC/builder_may_not_admin1640=== PAUSE TestService_RequireScope_OIDC/builder_may_not_admin1641=== RUN TestService_RequireScope_OIDC/ops_may_admin1642=== PAUSE TestService_RequireScope_OIDC/ops_may_admin1643=== RUN TestService_RequireScope_OIDC/ops_may_not_write1644=== PAUSE TestService_RequireScope_OIDC/ops_may_not_write1645=== RUN TestService_RequireScope_OIDC/reader_may_not_write1646=== PAUSE TestService_RequireScope_OIDC/reader_may_not_write1647=== RUN TestService_RequireScope_OIDC/static_token_may_admin1648=== PAUSE TestService_RequireScope_OIDC/static_token_may_admin1649=== RUN TestService_RequireScope_OIDC/static_token_may_write1650=== PAUSE TestService_RequireScope_OIDC/static_token_may_write1651=== RUN TestService_RequireScope_OIDC/reader_may_read1652=== PAUSE TestService_RequireScope_OIDC/reader_may_read1653=== RUN TestService_RequireScope_OIDC/writer_implies_read1654=== PAUSE TestService_RequireScope_OIDC/writer_implies_read1655=== RUN TestService_RequireScope_OIDC/anonymous_may_not_read1656=== PAUSE TestService_RequireScope_OIDC/anonymous_may_not_read1657=== RUN TestService_RequireScope_OIDC/reader_may_read_with_netrc1658=== PAUSE TestService_RequireScope_OIDC/reader_may_read_with_netrc1659=== RUN TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1660=== PAUSE TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1661=== RUN TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read1662=== PAUSE TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read1663=== RUN TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1664=== PAUSE TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1665=== CONT TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete16662026/11/12 00:33:56 INFO Received push request method=POST path=/api/pushes16672026/11/12 00:33:56 INFO Received push request method=POST path=/api/pushes16682026/11/12 00:33:56 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)16692026/11/12 00:33:56 INFO Uploading ppxgnvb27dhckgxyssp3ib55ijcmmh1l-test-file.txt (152B)16702026/11/12 00:33:56 INFO Uploading 3 paths to 127.0.0.1 (0 already cached)16712026/11/12 00:33:56 INFO Uploading 1bizpl56dhymhbck7qbn9k43gz02msd0-test-file-1.txt (160B)16722026/11/12 00:33:56 INFO Uploading w9w7nf83xn3f0byycbrx9n2xb7y7sb2y-test-file-2.txt (160B)16732026/11/12 00:33:56 INFO Uploading gv306ifg3av94cvwprfsif9m3nz9zpdd-test-file-0.txt (160B)16742026/11/12 00:33:56 WARN Failed to register uploaded object key=1bizpl56dhymhbck7qbn9k43gz02msd0.ls error="server returned 404: 404 page not found\n"16752026/11/12 00:33:56 WARN Failed to register uploaded object key=ppxgnvb27dhckgxyssp3ib55ijcmmh1l.ls error="server returned 404: 404 page not found\n"16762026/11/12 00:33:56 WARN Failed to register uploaded object key=w9w7nf83xn3f0byycbrx9n2xb7y7sb2y.ls error="server returned 404: 404 page not found\n"16772026/11/12 00:33:56 WARN Failed to register uploaded object key=gv306ifg3av94cvwprfsif9m3nz9zpdd.ls error="server returned 404: 404 page not found\n"16782026/11/12 00:33:56 WARN Failed to register uploaded object key=nar/0ahq3izw3h0b0c08p3xdfclh758fwxnp85ccvflasb4vnfsv50ri.nar.zst error="server returned 404: 404 page not found\n"16792026/11/12 00:33:56 WARN Failed to register uploaded object key=nar/1823g6vx4q5zqvr73m7lkq1wpk40fin7bsc792di430yk7sg4lka.nar.zst error="server returned 404: 404 page not found\n"16802026/11/12 00:33:56 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign16812026/11/12 00:33:56 WARN Failed to register uploaded object key=nar/13p9pqi0ga6fx2apcwls9hjnv7nmf740ggpn5zvfzdsp5vz2bbg4.nar.zst error="server returned 404: 404 page not found\n"16822026/11/12 00:33:56 INFO Signed narinfos id=1 count=316832026/11/12 00:33:56 INFO Uploading 3 narinfos16842026/11/12 00:33:56 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign16852026/11/12 00:33:56 WARN Failed to register uploaded object key=nar/11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1.nar.zst error="server returned 404: 404 page not found\n"16862026/11/12 00:33:56 INFO Signed narinfos id=1 count=116872026/11/12 00:33:56 INFO Uploading 1 narinfos16882026/11/12 00:33:56 WARN Failed to register uploaded object key=1bizpl56dhymhbck7qbn9k43gz02msd0.narinfo error="server returned 404: 404 page not found\n"16892026/11/12 00:33:56 WARN Failed to register uploaded object key=gv306ifg3av94cvwprfsif9m3nz9zpdd.narinfo error="server returned 404: 404 page not found\n"16902026/11/12 00:33:56 INFO Received complete push request method=POST path=/api/pushes/1/complete16912026/11/12 00:33:56 WARN Failed to register uploaded object key=w9w7nf83xn3f0byycbrx9n2xb7y7sb2y.narinfo error="server returned 404: 404 page not found\n"16922026/11/12 00:33:56 INFO Received complete push request method=POST path=/api/pushes/1/complete16932026/11/12 00:33:56 WARN Failed to register uploaded object key=ppxgnvb27dhckgxyssp3ib55ijcmmh1l.narinfo error="server returned 404: 404 page not found\n"16942026/11/12 00:33:56 INFO Completed push id=1 objects=9 duration=25.367625ms16952026/11/12 00:33:56 INFO Upload complete. (476ms)1696=== NAME TestClientMultipleUploads1697 client_integration_test.go:417: Uploaded 3 paths in 574.034625ms16982026/11/12 00:33:56 INFO Completed push id=1 objects=3 duration=43.219708ms16992026/11/12 00:33:56 INFO Upload complete. (472ms)17002026/11/12 00:33:56 INFO All 1 paths already cached1701=== NAME TestClientIntegration1702 client_integration_test.go:318: Retrieved narinfo from S3:1703 StorePath: /private/var/tb/5/scratch/2e3e4b48f97c7cfb/build/TestClientIntegration1081111093/002/store/ppxgnvb27dhckgxyssp3ib55ijcmmh1l-test-file.txt1704 URL: nar/11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1.nar.zst1705 Compression: zstd1706 NarHash: sha256:11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk11707 NarSize: 1521708 References: 1709 CA: fixed:r:sha256:11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk11710--- PASS: TestClientMultipleUploads (8.55s)1711=== CONT TestPush_RejectsBadRequests/no_objects1712=== NAME TestClientIntegration1713 client_integration_test.go:319: Retrieved .ls file from S3 (compressed size: 77 bytes)17142026/11/12 00:33:56 INFO Received push request method=POST path=/api/pushes1715 client_integration_test.go:319: Decompressed .ls content (64 bytes):1716 {"version":1,"root":{"type":"regular","size":39,"narOffset":96}}1717=== CONT TestPush_RejectsBadRequests/root_not_in_objects17182026/11/12 00:33:56 INFO Received push request method=POST path=/api/pushes1719=== CONT TestPush_RejectsBadRequests/no_roots17202026/11/12 00:33:56 INFO Received push request method=POST path=/api/pushes1721=== CONT TestPush_RejectsBadRequests/bad_root17222026/11/12 00:33:56 INFO Received push request method=POST path=/api/pushes1723=== CONT TestIsValidCachePath/leading_slash1724=== CONT TestIsValidCachePath/narinfo1725=== CONT TestIsValidCachePath/empty1726--- PASS: TestPush_RejectsBadRequests (1.03s)1727 --- PASS: TestPush_RejectsBadRequests/no_objects (0.00s)1728 --- PASS: TestPush_RejectsBadRequests/root_not_in_objects (0.00s)1729 --- PASS: TestPush_RejectsBadRequests/no_roots (0.00s)1730 --- PASS: TestPush_RejectsBadRequests/bad_root (0.00s)1731=== CONT TestIsValidCachePath/random_path1732=== CONT TestIsValidCachePath/invalid_char_u1733=== CONT TestIsValidCachePath/invalid_char_e1734=== CONT TestIsValidCachePath/traversal_in_middle1735=== CONT TestIsValidCachePath/traversal_parent1736=== CONT TestIsValidCachePath/index.html1737=== CONT TestIsValidCachePath/nix-cache-info1738=== CONT TestIsValidCachePath/realisation1739=== CONT TestIsValidCachePath/log1740=== CONT TestIsValidCachePath/ls1741=== CONT TestIsValidCachePath/nar_uncompressed1742=== CONT TestIsValidCachePath/nar_bz21743=== CONT TestIsValidCachePath/nar_xz1744=== CONT TestIsValidCachePath/nar_zst1745=== CONT TestIsValidCachePath/narinfo_all_nix_base32_chars1746=== CONT TestIsValidCachePath/short_hash1747=== CONT TestIsValidCachePath/wrong_extension1748--- PASS: TestIsValidCachePath (0.02s)1749 --- PASS: TestIsValidCachePath/leading_slash (0.00s)1750 --- PASS: TestIsValidCachePath/narinfo (0.00s)1751 --- PASS: TestIsValidCachePath/empty (0.00s)1752 --- PASS: TestIsValidCachePath/random_path (0.00s)1753 --- PASS: TestIsValidCachePath/invalid_char_u (0.00s)1754 --- PASS: TestIsValidCachePath/invalid_char_e (0.00s)1755 --- PASS: TestIsValidCachePath/traversal_in_middle (0.00s)1756 --- PASS: TestIsValidCachePath/traversal_parent (0.00s)1757 --- PASS: TestIsValidCachePath/index.html (0.00s)1758 --- PASS: TestIsValidCachePath/nix-cache-info (0.00s)1759 --- PASS: TestIsValidCachePath/realisation (0.00s)1760 --- PASS: TestIsValidCachePath/log (0.00s)1761 --- PASS: TestIsValidCachePath/ls (0.00s)1762 --- PASS: TestIsValidCachePath/nar_uncompressed (0.00s)1763 --- PASS: TestIsValidCachePath/nar_bz2 (0.00s)1764 --- PASS: TestIsValidCachePath/nar_xz (0.00s)1765 --- PASS: TestIsValidCachePath/nar_zst (0.00s)1766 --- PASS: TestIsValidCachePath/narinfo_all_nix_base32_chars (0.00s)1767 --- PASS: TestIsValidCachePath/short_hash (0.00s)1768 --- PASS: TestIsValidCachePath/wrong_extension (0.00s)1769=== CONT TestParseSingleRange/none1770=== CONT TestPresentNotReportedWhileGCDeletesClosure/stale17712026/11/12 00:33:56 INFO Received uploads request method=POST path=/api/pending_closures17722026/11/12 00:33:57 INFO Received push request method=POST path=/api/pushes17732026/11/12 00:33:57 INFO Object in database but missing from S3 key=ppxgnvb27dhckgxyssp3ib55ijcmmh1l.ls17742026/11/12 00:33:57 WARN Found objects in DB but missing from S3, will re-upload count=117752026/11/12 00:33:57 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)17762026/11/12 00:33:57 INFO Received complete push request method=POST path=/api/pushes/2/complete17772026/11/12 00:33:57 WARN Failed to register uploaded object key=ppxgnvb27dhckgxyssp3ib55ijcmmh1l.ls error="server returned 404: 404 page not found\n"17782026/11/12 00:33:57 INFO Completed push id=2 objects=3 duration=1.060333ms17792026/11/12 00:33:57 INFO Upload complete. (159ms)1780=== NAME TestClientIntegration1781 client_integration_test.go:336: Retrieved .ls file from S3 (compressed size: 62 bytes)1782 client_integration_test.go:336: Decompressed .ls content (49 bytes):1783 {"version":1,"root":{"type":"regular","size":39}}17842026/11/12 00:33:57 INFO Received complete multipart upload request method=POST path=/api/multipart/complete17852026/11/12 00:33:57 WARN CompleteMultipartUpload errored but object exists; treating as success error="The specified key does not exist." object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=MDNiMjFmNDUtYWZkMC00ZDllLTk0M2UtNWYyMTE4ODJhZGZkLjJjM2UzYzY0LWQxNDctNGM1NS1iMTM3LTYyYzk5Y2NjNWZiYngxNzk0NDQzNjM2OTIwODI2MDAw17862026/11/12 00:33:57 INFO Garbage collection progress phase="" failed_uploads_deleted=0 old_closures_deleted=1 objects_marked=3 objects_deleted=3 objects_failed=017872026/11/12 00:33:57 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=MDNiMjFmNDUtYWZkMC00ZDllLTk0M2UtNWYyMTE4ODJhZGZkLjJjM2UzYzY0LWQxNDctNGM1NS1iMTM3LTYyYzk5Y2NjNWZiYngxNzk0NDQzNjM2OTIwODI2MDAw parts=11788--- PASS: TestCompleteMultipartUpload_ErrorButObjectExists (8.04s)1789=== CONT TestParseSingleRange/start_far_past_EOF1790=== CONT TestParseSingleRange/start_past_EOF1791=== CONT TestParseSingleRange/single_byte1792=== CONT TestParseSingleRange/suffix_exceeds_size1793=== CONT TestParseSingleRange/suffix1794=== CONT TestParseSingleRange/end_clamped_to_size1795=== CONT TestParseSingleRange/open-ended1796=== CONT TestParseSingleRange/closed1797=== CONT TestParseSingleRange/malformed_end_before_start1798=== CONT TestParseSingleRange/malformed_both_empty1799=== CONT TestParseSingleRange/malformed_no_dash1800=== CONT TestParseSingleRange/multi-range_ignored1801=== CONT TestParseSingleRange/unknown_unit1802--- PASS: TestParseSingleRange (0.03s)1803 --- PASS: TestParseSingleRange/none (0.00s)1804 --- PASS: TestParseSingleRange/start_far_past_EOF (0.00s)1805 --- PASS: TestParseSingleRange/start_past_EOF (0.00s)1806 --- PASS: TestParseSingleRange/single_byte (0.00s)1807 --- PASS: TestParseSingleRange/suffix_exceeds_size (0.00s)1808 --- PASS: TestParseSingleRange/suffix (0.00s)1809 --- PASS: TestParseSingleRange/end_clamped_to_size (0.00s)1810 --- PASS: TestParseSingleRange/open-ended (0.00s)1811 --- PASS: TestParseSingleRange/closed (0.00s)1812 --- PASS: TestParseSingleRange/malformed_end_before_start (0.00s)1813 --- PASS: TestParseSingleRange/malformed_both_empty (0.00s)1814 --- PASS: TestParseSingleRange/malformed_no_dash (0.00s)1815 --- PASS: TestParseSingleRange/multi-range_ignored (0.00s)1816 --- PASS: TestParseSingleRange/unknown_unit (0.00s)1817=== CONT TestPresentNotReportedWhileGCDeletesClosure/fresh18182026/11/12 00:33:57 INFO Received create pin request method=POST path=/api/pins/myapp18192026/11/12 00:33:57 INFO Created/updated pin name=myapp store_path=/private/var/tb/5/scratch/2e3e4b48f97c7cfb/build/TestPinProtectsFromGC2593085542/001/store/ijgix19kdz184gc5ly26l7x7nznni2cm-pinned-file.txt narinfo_key=ijgix19kdz184gc5ly26l7x7nznni2cm.narinfo18202026/11/12 00:33:57 INFO Starting cleanup of old closures method=DELETE path=/api/closures18212026/11/12 00:33:57 INFO Garbage collection started18222026/11/12 00:33:57 INFO Aborted multipart uploads count=0 kept=018232026/11/12 00:33:57 WARN Force mode enabled - objects will be deleted immediately without grace period18242026/11/12 00:33:57 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=018252026/11/12 00:33:57 INFO Vacuumed table table=pending_closures18262026/11/12 00:33:57 INFO Vacuumed table table=pending_objects18272026/11/12 00:33:57 INFO Vacuumed table table=multipart_uploads18282026/11/12 00:33:57 INFO Vacuumed table table=closures18292026/11/12 00:33:57 INFO Vacuumed table table=objects18302026/11/12 00:33:57 INFO Received push request method=POST path=/api/pushes18312026/11/12 00:33:57 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)18322026/11/12 00:33:57 INFO Received complete push request method=POST path=/api/pushes/3/complete18332026/11/12 00:33:57 WARN Failed to register uploaded object key=ppxgnvb27dhckgxyssp3ib55ijcmmh1l.ls error="server returned 404: 404 page not found\n"18342026/11/12 00:33:57 INFO Completed push id=3 objects=3 duration=2.405292ms18352026/11/12 00:33:57 INFO Upload complete. (172ms)1836=== NAME TestClientIntegration1837 client_integration_test.go:353: Retrieved .ls file from S3 (compressed size: 62 bytes)1838 client_integration_test.go:353: Decompressed .ls content (49 bytes):1839 {"version":1,"root":{"type":"regular","size":39}}1840 client_integration_test.go:364: Testing garbage collection...18412026/11/12 00:33:57 INFO Starting cleanup of old closures method=DELETE path=/api/closures18422026/11/12 00:33:57 INFO Garbage collection started18432026/11/12 00:33:57 INFO Aborted multipart uploads count=0 kept=018442026/11/12 00:33:57 WARN Force mode enabled - objects will be deleted immediately without grace period18452026/11/12 00:33:58 INFO Received uploads request method=POST path=/api/pending_closures18462026/11/12 00:33:58 INFO Registered completed upload object_key=nar/jjjjjjjjjjjjjjjjjjjjjjjjjjjjjj0100000000000000000000.nar.zst18472026/11/12 00:33:58 INFO Received uploads request method=POST path=/api/pending_closures1848--- PASS: TestPresignedUploadRegisteredBeforeCommit (8.12s)1849=== CONT TestPendingClosureWriteTimeout/empty1850=== CONT TestPendingClosureWriteTimeout/670k_objects1851=== CONT TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline1852--- PASS: TestConcurrentCommitsSharingObjectsDoNotDeadlock (9.52s)1853=== CONT TestPendingClosureWriteTimeout/400_objects1854=== CONT TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed18552026/11/12 00:33:58 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=018562026/11/12 00:33:58 INFO Received uploads request method=POST path=/api/pending_closures18572026/11/12 00:33:58 INFO Vacuumed table table=pending_closures18582026/11/12 00:33:58 INFO Vacuumed table table=pending_objects18592026/11/12 00:33:58 INFO Vacuumed table table=multipart_uploads18602026/11/12 00:33:58 INFO Vacuumed table table=closures18612026/11/12 00:33:58 INFO Vacuumed table table=objects18622026/11/12 00:33:58 INFO Received uploads request method=POST path=/api/pending_closures18632026/11/12 00:33:58 WARN Rate limiter enabled after throttle name=s3-test rate=518642026/11/12 00:33:58 WARN S3 rate limit hit operation="complete multipart upload" error="Please reduce your request rate."1865=== NAME TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle1866 throttle_test.go:213: Proxy stats: total=13, throttled=10, completeMultipart=101867 throttle_test.go:215: Rate limiter: enabled=true, rate=5.0018682026/11/12 00:33:58 INFO Received uploads request method=POST path=/api/pending_closures1869--- PASS: TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle (12.12s)1870=== CONT TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed18712026/11/12 00:33:58 INFO Uploading 2 paths to 127.0.0.1 (1 already cached)18722026/11/12 00:33:58 INFO Uploading wrcrqs1yspg3qxq392fz6r6pdlf8rs0m-a (256B)18732026/11/12 00:33:58 INFO Uploading bmysfzy9pbk1q5md46d7247nwjdxkbjw-shared-dep (136B)18742026/11/12 00:33:58 WARN Failed to register uploaded object key=257df5w0wgbji6wnc5i1vk51ndx5r07a.ls error="server returned 404: 404 page not found\n"18752026/11/12 00:33:58 WARN Failed to register uploaded object key=bmysfzy9pbk1q5md46d7247nwjdxkbjw.ls error="server returned 404: 404 page not found\n"18762026/11/12 00:33:58 INFO Received uploads request method=POST path=/api/pending_closures18772026/11/12 00:33:58 WARN Failed to register uploaded object key=wrcrqs1yspg3qxq392fz6r6pdlf8rs0m.ls error="server returned 404: 404 page not found\n"18782026/11/12 00:33:58 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"18792026/11/12 00:33:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign18802026/11/12 00:33:58 WARN Failed to register uploaded object key=nar/17pnch8wn3i57lv8yfb63jygx0rph3y9c62lkdh5gxmkhclqqvfx.nar.zst error="server returned 404: 404 page not found\n"18812026/11/12 00:33:58 INFO Signed narinfos id=2 count=218822026/11/12 00:33:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign18832026/11/12 00:33:59 INFO Signed narinfos id=1 count=218842026/11/12 00:33:59 INFO Uploading 4 narinfos18852026/11/12 00:33:59 WARN Failed to register uploaded object key=bmysfzy9pbk1q5md46d7247nwjdxkbjw.narinfo error="server returned 404: 404 page not found\n"18862026/11/12 00:33:59 WARN Failed to register uploaded object key=wrcrqs1yspg3qxq392fz6r6pdlf8rs0m.narinfo error="server returned 404: 404 page not found\n"18872026/11/12 00:33:59 INFO Received uploads request method=POST path=/api/pending_closures18882026/11/12 00:33:59 WARN Failed to register uploaded object key=257df5w0wgbji6wnc5i1vk51ndx5r07a.narinfo error="server returned 404: 404 page not found\n"18892026/11/12 00:33:59 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete18902026/11/12 00:33:59 WARN Failed to register uploaded object key=bmysfzy9pbk1q5md46d7247nwjdxkbjw.narinfo error="server returned 404: 404 page not found\n"18912026/11/12 00:33:59 INFO Completed upload id=1 objects=6 duration=61.832709ms18922026/11/12 00:33:59 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete18932026/11/12 00:33:59 INFO Completed upload id=2 objects=6 duration=745.042µs18942026/11/12 00:33:59 INFO Upload complete. (578ms)1895=== NAME TestClientFallsBackToClosures1896 client_pushes_test.go:112: Retrieved narinfo from S3:1897 StorePath: /private/var/tb/5/scratch/2e3e4b48f97c7cfb/build/TestClientFallsBackToClosures1082906393/001/store/bmysfzy9pbk1q5md46d7247nwjdxkbjw-shared-dep1898 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1899 Compression: zstd1900 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821901 NarSize: 1361902 References: 1903 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n1904 client_pushes_test.go:112: Retrieved narinfo from S3:1905 StorePath: /private/var/tb/5/scratch/2e3e4b48f97c7cfb/build/TestClientFallsBackToClosures1082906393/001/store/wrcrqs1yspg3qxq392fz6r6pdlf8rs0m-a1906 URL: nar/17pnch8wn3i57lv8yfb63jygx0rph3y9c62lkdh5gxmkhclqqvfx.nar.zst1907 Compression: zstd1908 NarHash: sha256:17pnch8wn3i57lv8yfb63jygx0rph3y9c62lkdh5gxmkhclqqvfx1909 NarSize: 2561910 References: /private/var/tb/5/scratch/2e3e4b48f97c7cfb/build/TestClientFallsBackToClosures1082906393/001/store/bmysfzy9pbk1q5md46d7247nwjdxkbjw-shared-dep1911 CA: text:sha256:0ydcaysfninbhph2sx16is7y44idj15mgcraknvi62whjlr090sa1912 client_pushes_test.go:112: Retrieved narinfo from S3:1913 StorePath: /private/var/tb/5/scratch/2e3e4b48f97c7cfb/build/TestClientFallsBackToClosures1082906393/001/store/257df5w0wgbji6wnc5i1vk51ndx5r07a-b1914 URL: nar/17pnch8wn3i57lv8yfb63jygx0rph3y9c62lkdh5gxmkhclqqvfx.nar.zst1915 Compression: zstd1916 NarHash: sha256:17pnch8wn3i57lv8yfb63jygx0rph3y9c62lkdh5gxmkhclqqvfx1917 NarSize: 2561918 References: /private/var/tb/5/scratch/2e3e4b48f97c7cfb/build/TestClientFallsBackToClosures1082906393/001/store/bmysfzy9pbk1q5md46d7247nwjdxkbjw-shared-dep1919 CA: text:sha256:0ydcaysfninbhph2sx16is7y44idj15mgcraknvi62whjlr090sa1920--- PASS: TestClientFallsBackToClosures (9.31s)1921=== CONT TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed1922=== NAME TestPinProtectsFromGC1923 client_integration_test.go:900: Pin successfully protected closure from garbage collection1924--- PASS: TestPinProtectsFromGC (12.68s)1925=== CONT TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost19262026/11/12 00:33:59 INFO Garbage collection progress phase="" failed_uploads_deleted=0 old_closures_deleted=1 objects_marked=3 objects_deleted=3 objects_failed=01927=== NAME TestClientIntegration1928 client_integration_test.go:371: Objects in database after GC:1929 client_integration_test.go:371: Successfully deleted all objects with GC --force1930--- PASS: TestService_ReadScope_PublicByDefault (8.36s)1931=== CONT TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed1932=== CONT TestUploadHandlersRejectOversizedBody/create_pending_closure19332026/11/12 00:34:00 INFO Received uploads request method=POST path=/1934--- PASS: TestClientIntegration (11.68s)1935=== NAME TestClientCADerivations1936 client_ca_test.go:136: Built CA derivation: /private/var/tb/5/scratch/2e3e4b48f97c7cfb/build/TestClientCADerivations4281092481/001/store/x1300hbgpq602nppffr477i3h43w3727-ca-test1937 client_ca_test.go:139: Found 1 dependencies (including self)1938=== CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info19392026/11/12 00:34:00 INFO Received uploads request method=POST path=/1940=== CONT TestUploadHandlersRejectOversizedBody/request_more_parts19412026/11/12 00:34:00 INFO Received request for more parts method=POST path=/1942=== CONT TestUploadHandlersRejectOversizedBody/complete_multipart19432026/11/12 00:34:00 INFO Received complete multipart upload request method=POST path=/1944--- PASS: TestCacheStatsHandler (8.11s)1945=== CONT TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key19462026/11/12 00:34:00 INFO Received complete multipart upload request method=POST path=/1947--- PASS: TestUploadHandlersRejectOversizedBody (0.04s)1948 --- PASS: TestUploadHandlersRejectOversizedBody/create_pending_closure (0.59s)1949 --- PASS: TestUploadHandlersRejectOversizedBody/request_more_parts (0.03s)1950 --- PASS: TestUploadHandlersRejectOversizedBody/complete_multipart (0.03s)1951=== CONT TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key19522026/11/12 00:34:00 INFO Received request for more parts method=POST path=/1953=== CONT TestIsValidUploadKey/narinfo1954=== CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal19552026/11/12 00:34:00 INFO Received uploads request method=POST path=/1956--- PASS: TestUploadHandlersRejectInvalidKeys (0.00s)1957 --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info (0.00s)1958 --- PASS: TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key (0.01s)1959 --- PASS: TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key (0.00s)1960 --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal (0.00s)1961=== CONT TestIsValidUploadKey/realisation_plus_in_output1962=== CONT TestIsValidUploadKey/empty_key1963=== CONT TestIsValidUploadKey/absolute1964=== CONT TestIsValidUploadKey/traversal_nar1965=== CONT TestIsValidUploadKey/traversal1966=== CONT TestIsValidUploadKey/listing_key,_narinfo_type1967=== CONT TestIsValidUploadKey/unknown_type1968=== CONT TestIsValidUploadKey/narinfo_key,_nar_type1969=== CONT TestIsValidUploadKey/index.html1970=== CONT TestIsValidUploadKey/nar_key,_narinfo_type1971=== CONT TestIsValidUploadKey/nix-cache-info1972=== CONT TestIsValidUploadKey/build_log_home-manager_file1973=== CONT TestIsValidUploadKey/realisation1974=== CONT TestIsValidUploadKey/build_log_equals1975=== CONT TestIsValidUploadKey/build_log_question_mark1976=== CONT TestIsValidUploadKey/build_log_plus_in_name1977=== CONT TestIsValidUploadKey/nar_plain1978=== CONT TestIsValidUploadKey/build_log1979=== CONT TestIsValidUploadKey/listing1980=== CONT TestIsValidUploadKey/nar_xz1981=== CONT TestIsValidUploadKey/nar_zst1982--- PASS: TestIsValidUploadKey (0.00s)1983 --- PASS: TestIsValidUploadKey/narinfo (0.00s)1984 --- PASS: TestIsValidUploadKey/realisation_plus_in_output (0.00s)1985 --- PASS: TestIsValidUploadKey/empty_key (0.00s)1986 --- PASS: TestIsValidUploadKey/absolute (0.00s)1987 --- PASS: TestIsValidUploadKey/traversal_nar (0.00s)1988 --- PASS: TestIsValidUploadKey/traversal (0.00s)1989 --- PASS: TestIsValidUploadKey/listing_key,_narinfo_type (0.00s)1990 --- PASS: TestIsValidUploadKey/unknown_type (0.00s)1991 --- PASS: TestIsValidUploadKey/narinfo_key,_nar_type (0.00s)1992 --- PASS: TestIsValidUploadKey/nar_key,_narinfo_type (0.00s)1993 --- PASS: TestIsValidUploadKey/nix-cache-info (0.00s)1994 --- PASS: TestIsValidUploadKey/build_log_home-manager_file (0.00s)1995 --- PASS: TestIsValidUploadKey/realisation (0.00s)1996 --- PASS: TestIsValidUploadKey/build_log_equals (0.00s)1997 --- PASS: TestIsValidUploadKey/build_log_question_mark (0.00s)1998 --- PASS: TestIsValidUploadKey/build_log_plus_in_name (0.00s)1999 --- PASS: TestIsValidUploadKey/nar_plain (0.00s)2000 --- PASS: TestIsValidUploadKey/build_log (0.00s)
loading 708 hidden lines…
27092026/11/12 00:34:20 INFO Received push request method=POST path=/27102026/11/12 00:34:20 WARN Force mode enabled - objects will be deleted immediately without grace period27112026/11/12 00:34:20 INFO Received uploads request method=POST path=/27122026/11/12 00:34:20 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027132026/11/12 00:34:20 INFO Received push request method=POST path=/27142026/11/12 00:34:20 INFO Aborted multipart uploads count=0 kept=027152026/11/12 00:34:20 INFO Vacuumed table table=pending_closures27162026/11/12 00:34:20 WARN Force mode enabled - objects will be deleted immediately without grace period27172026/11/12 00:34:20 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027182026/11/12 00:34:20 INFO Vacuumed table table=pending_objects27192026/11/12 00:34:20 INFO Received uploads request method=POST path=/27202026/11/12 00:34:20 INFO Vacuumed table table=pending_closures27212026/11/12 00:34:20 INFO Vacuumed table table=multipart_uploads27222026/11/12 00:34:20 INFO Received uploads request method=POST path=/27232026/11/12 00:34:20 INFO Vacuumed table table=pending_objects27242026/11/12 00:34:20 INFO Vacuumed table table=closures27252026/11/12 00:34:20 INFO Vacuumed table table=multipart_uploads27262026/11/12 00:34:20 INFO Vacuumed table table=objects27272026/11/12 00:34:20 INFO Received push request method=POST path=/27282026/11/12 00:34:20 INFO Received complete push request method=POST path=/27292026/11/12 00:34:20 INFO Vacuumed table table=closures27302026/11/12 00:34:20 INFO Received complete upload request method=POST path=/27312026/11/12 00:34:20 INFO Vacuumed table table=objects27322026/11/12 00:34:20 INFO Aborted multipart uploads count=0 kept=027332026/11/12 00:34:20 INFO Completed upload id=2 objects=7 duration=13.961958ms27342026/11/12 00:34:20 INFO Completed push id=1 objects=24 duration=26.550958ms27352026/11/12 00:34:20 INFO Aborted multipart uploads count=0 kept=027362026/11/12 00:34:20 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027372026/11/12 00:34:20 INFO Received push request method=POST path=/27382026/11/12 00:34:20 INFO Received complete push request method=POST path=/27392026/11/12 00:34:20 WARN Force mode enabled - objects will be deleted immediately without grace period27402026/11/12 00:34:20 INFO Vacuumed table table=pending_closures27412026/11/12 00:34:20 INFO Received uploads request method=POST path=/27422026/11/12 00:34:20 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027432026/11/12 00:34:20 INFO Completed push id=19 objects=8 duration=9.901708ms27442026/11/12 00:34:20 INFO Vacuumed table table=pending_objects27452026/11/12 00:34:20 INFO Vacuumed table table=pending_closures27462026/11/12 00:34:20 INFO Vacuumed table table=multipart_uploads27472026/11/12 00:34:20 INFO Vacuumed table table=pending_objects27482026/11/12 00:34:20 INFO Vacuumed table table=multipart_uploads27492026/11/12 00:34:20 INFO Vacuumed table table=closures27502026/11/12 00:34:20 INFO Vacuumed table table=objects27512026/11/12 00:34:20 INFO Received complete push request method=POST path=/27522026/11/12 00:34:20 INFO Received uploads request method=POST path=/27532026/11/12 00:34:20 INFO Received uploads request method=POST path=/27542026/11/12 00:34:20 INFO Vacuumed table table=closures27552026/11/12 00:34:20 INFO Completed push id=14 objects=3 duration=3.641166ms27562026/11/12 00:34:20 INFO Aborted multipart uploads count=0 kept=027572026/11/12 00:34:20 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027582026/11/12 00:34:20 INFO Received push request method=POST path=/27592026/11/12 00:34:20 INFO Aborted multipart uploads count=0 kept=027602026/11/12 00:34:20 INFO Received push request method=POST path=/27612026/11/12 00:34:20 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027622026/11/12 00:34:20 INFO Vacuumed table table=pending_closures27632026/11/12 00:34:20 INFO Vacuumed table table=objects27642026/11/12 00:34:20 INFO Received complete push request method=POST path=/27652026/11/12 00:34:20 INFO Received complete push request method=POST path=/27662026/11/12 00:34:20 INFO Vacuumed table table=pending_objects27672026/11/12 00:34:20 INFO Aborted multipart uploads count=0 kept=027682026/11/12 00:34:20 INFO Completed push id=5 objects=6 duration=1.225ms27692026/11/12 00:34:20 INFO Received complete push request method=POST path=/27702026/11/12 00:34:20 INFO Vacuumed table table=multipart_uploads27712026/11/12 00:34:20 INFO Vacuumed table table=pending_closures27722026/11/12 00:34:20 INFO Vacuumed table table=closures27732026/11/12 00:34:20 INFO Completed push id=1 objects=2 duration=1.149166ms27742026/11/12 00:34:20 INFO Vacuumed table table=objects27752026/11/12 00:34:20 INFO Completed push id=3 objects=5 duration=11.323917ms27762026/11/12 00:34:20 INFO Vacuumed table table=pending_objects27772026/11/12 00:34:20 INFO Vacuumed table table=multipart_uploads27782026/11/12 00:34:20 INFO Vacuumed table table=closures27792026/11/12 00:34:20 WARN Force mode enabled - objects will be deleted immediately without grace period27802026/11/12 00:34:20 INFO Aborted multipart uploads count=0 kept=027812026/11/12 00:34:20 WARN Force mode enabled - objects will be deleted immediately without grace period27822026/11/12 00:34:20 INFO Aborted multipart uploads count=0 kept=027832026/11/12 00:34:20 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=5 objects-failed-to-delete=027842026/11/12 00:34:20 INFO Received uploads request method=POST path=/27852026/11/12 00:34:20 INFO Vacuumed table table=objects27862026/11/12 00:34:20 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027872026/11/12 00:34:20 WARN Force mode enabled - objects will be deleted immediately without grace period27882026/11/12 00:34:20 INFO Received complete push request method=POST path=/27892026/11/12 00:34:20 INFO Vacuumed table table=pending_closures27902026/11/12 00:34:20 INFO Completed push id=19 objects=6 duration=3.044458ms27912026/11/12 00:34:20 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027922026/11/12 00:34:20 INFO Aborted multipart uploads count=0 kept=027932026/11/12 00:34:20 INFO Vacuumed table table=pending_objects27942026/11/12 00:34:20 INFO Aborted multipart uploads count=0 kept=027952026/11/12 00:34:20 INFO Vacuumed table table=pending_closures27962026/11/12 00:34:20 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027972026/11/12 00:34:20 INFO Vacuumed table table=multipart_uploads27982026/11/12 00:34:21 INFO Vacuumed table table=pending_closures27992026/11/12 00:34:21 INFO Vacuumed table table=pending_objects28002026/11/12 00:34:21 INFO Vacuumed table table=closures28012026/11/12 00:34:21 WARN Force mode enabled - objects will be deleted immediately without grace period28022026/11/12 00:34:21 INFO Vacuumed table table=objects28032026/11/12 00:34:21 INFO Vacuumed table table=pending_closures28042026/11/12 00:34:21 INFO Vacuumed table table=multipart_uploads28052026/11/12 00:34:21 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028062026/11/12 00:34:21 INFO Vacuumed table table=pending_objects28072026/11/12 00:34:21 INFO Vacuumed table table=pending_objects28082026/11/12 00:34:21 INFO Vacuumed table table=closures28092026/11/12 00:34:21 INFO Vacuumed table table=pending_closures28102026/11/12 00:34:21 INFO Vacuumed table table=multipart_uploads28112026/11/12 00:34:21 INFO Vacuumed table table=multipart_uploads28122026/11/12 00:34:21 INFO Vacuumed table table=closures28132026/11/12 00:34:21 INFO Vacuumed table table=objects28142026/11/12 00:34:21 INFO Vacuumed table table=pending_objects28152026/11/12 00:34:21 INFO Vacuumed table table=multipart_uploads28162026/11/12 00:34:21 INFO Vacuumed table table=closures28172026/11/12 00:34:21 INFO Aborted multipart uploads count=0 kept=028182026/11/12 00:34:21 INFO Vacuumed table table=objects28192026/11/12 00:34:21 WARN Force mode enabled - objects will be deleted immediately without grace period28202026/11/12 00:34:21 INFO Vacuumed table table=objects28212026/11/12 00:34:21 INFO Received push request method=POST path=/28222026/11/12 00:34:21 INFO Received push request method=POST path=/28232026/11/12 00:34:21 INFO Vacuumed table table=closures28242026/11/12 00:34:21 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."28252026/11/12 00:34:21 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."28262026/11/12 00:34:21 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."28272026/11/12 00:34:21 INFO Received push request method=POST path=/28282026/11/12 00:34:21 INFO Received uploads request method=POST path=/28292026/11/12 00:34:21 INFO Vacuumed table table=objects28302026/11/12 00:34:21 INFO Received uploads request method=POST path=/28312026/11/12 00:34:21 INFO Aborted multipart uploads count=0 kept=028322026/11/12 00:34:21 INFO Aborted multipart uploads count=0 kept=028332026/11/12 00:34:21 WARN Force mode enabled - objects will be deleted immediately without grace period28342026/11/12 00:34:21 INFO Received uploads request method=POST path=/28352026/11/12 00:34:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028362026/11/12 00:34:21 INFO Aborted multipart uploads count=0 kept=028372026/11/12 00:34:21 INFO Vacuumed table table=pending_closures28382026/11/12 00:34:21 INFO Vacuumed table table=pending_objects28392026/11/12 00:34:21 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."28402026/11/12 00:34:21 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."28412026/11/12 00:34:21 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."28422026/11/12 00:34:21 INFO Vacuumed table table=multipart_uploads28432026/11/12 00:34:21 INFO Received push request method=POST path=/28442026/11/12 00:34:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028452026/11/12 00:34:21 INFO Vacuumed table table=closures28462026/11/12 00:34:21 INFO Vacuumed table table=pending_closures28472026/11/12 00:34:21 INFO Received uploads request method=POST path=/28482026/11/12 00:34:21 INFO Vacuumed table table=objects28492026/11/12 00:34:21 INFO Received complete push request method=POST path=/28502026/11/12 00:34:21 INFO Received uploads request method=POST path=/28512026/11/12 00:34:21 INFO Vacuumed table table=pending_objects28522026/11/12 00:34:21 INFO Received push request method=POST path=/28532026/11/12 00:34:21 INFO Completed push id=3 objects=21 duration=6.725959ms28542026/11/12 00:34:21 INFO Aborted multipart uploads count=0 kept=028552026/11/12 00:34:21 INFO Received push request method=POST path=/28562026/11/12 00:34:21 INFO Vacuumed table table=multipart_uploads28572026/11/12 00:34:21 INFO Received uploads request method=POST path=/28582026/11/12 00:34:21 WARN Force mode enabled - objects will be deleted immediately without grace period28592026/11/12 00:34:21 INFO Received uploads request method=POST path=/28602026/11/12 00:34:21 INFO Received uploads request method=POST path=/28612026/11/12 00:34:21 INFO Vacuumed table table=closures28622026/11/12 00:34:21 INFO Aborted multipart uploads count=0 kept=028632026/11/12 00:34:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028642026/11/12 00:34:21 INFO Vacuumed table table=objects28652026/11/12 00:34:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028662026/11/12 00:34:21 INFO Vacuumed table table=pending_closures28672026/11/12 00:34:21 INFO Received complete upload request method=POST path=/28682026/11/12 00:34:21 INFO Vacuumed table table=pending_closures28692026/11/12 00:34:21 INFO Received push request method=POST path=/28702026/11/12 00:34:21 INFO Vacuumed table table=pending_objects28712026/11/12 00:34:21 INFO Completed upload id=3 objects=25 duration=1.744125ms28722026/11/12 00:34:21 INFO Received uploads request method=POST path=/28732026/11/12 00:34:21 INFO Vacuumed table table=multipart_uploads28742026/11/12 00:34:21 INFO Vacuumed table table=pending_objects28752026/11/12 00:34:21 INFO Received uploads request method=POST path=/28762026/11/12 00:34:21 INFO Vacuumed table table=closures28772026/11/12 00:34:21 INFO Vacuumed table table=multipart_uploads28782026/11/12 00:34:21 INFO Vacuumed table table=objects28792026/11/12 00:34:21 INFO Aborted multipart uploads count=0 kept=028802026/11/12 00:34:21 INFO Vacuumed table table=closures28812026/11/12 00:34:21 INFO Received uploads request method=POST path=/28822026/11/12 00:34:21 INFO Aborted multipart uploads count=0 kept=028832026/11/12 00:34:21 INFO Vacuumed table table=objects28842026/11/12 00:34:21 WARN Force mode enabled - objects will be deleted immediately without grace period28852026/11/12 00:34:21 WARN Force mode enabled - objects will be deleted immediately without grace period28862026/11/12 00:34:21 INFO Received uploads request method=POST path=/28872026/11/12 00:34:21 INFO Received push request method=POST path=/28882026/11/12 00:34:21 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=3 objects-failed-to-delete=028892026/11/12 00:34:21 INFO Received complete push request method=POST path=/28902026/11/12 00:34:21 INFO Completed push id=15 objects=5 duration=2.667833ms28912026/11/12 00:34:21 INFO Received uploads request method=POST path=/28922026/11/12 00:34:21 INFO Vacuumed table table=pending_closures28932026/11/12 00:34:21 INFO Received uploads request method=POST path=/28942026/11/12 00:34:21 INFO Received complete upload request method=POST path=/28952026/11/12 00:34:21 INFO Completed upload id=5 objects=7 duration=2.172958ms28962026/11/12 00:34:21 INFO Aborted multipart uploads count=0 kept=028972026/11/12 00:34:21 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028982026/11/12 00:34:21 INFO Vacuumed table table=pending_objects28992026/11/12 00:34:21 INFO Received push request method=POST path=/29002026/11/12 00:34:21 INFO Vacuumed table table=pending_closures29012026/11/12 00:34:21 INFO Vacuumed table table=multipart_uploads29022026/11/12 00:34:21 INFO Vacuumed table table=pending_objects29032026/11/12 00:34:21 INFO Vacuumed table table=multipart_uploads29042026/11/12 00:34:21 INFO Vacuumed table table=closures29052026/11/12 00:34:21 INFO Vacuumed table table=objects29062026/11/12 00:34:21 INFO Vacuumed table table=closures29072026/11/12 00:34:21 INFO Received complete push request method=POST path=/29082026/11/12 00:34:21 INFO Received complete upload request method=POST path=/29092026/11/12 00:34:21 INFO Completed upload id=4 objects=7 duration=4.692459ms29102026/11/12 00:34:21 INFO Vacuumed table table=objects29112026/11/12 00:34:21 INFO Received complete upload request method=POST path=/29122026/11/12 00:34:21 INFO Completed push id=1 objects=5 duration=23.567292ms29132026/11/12 00:34:21 INFO Completed upload id=6 objects=9 duration=2.511708ms29142026/11/12 00:34:21 INFO Received push request method=POST path=/29152026/11/12 00:34:21 INFO Received uploads request method=POST path=/29162026/11/12 00:34:21 INFO Received push request method=POST path=/29172026/11/12 00:34:21 INFO Received uploads request method=POST path=/29182026/11/12 00:34:21 INFO Received uploads request method=POST path=/29192026/11/12 00:34:21 INFO Received uploads request method=POST path=/29202026/11/12 00:34:21 INFO Received push request method=POST path=/29212026/11/12 00:34:21 INFO Received uploads request method=POST path=/29222026/11/12 00:34:21 INFO Received push request method=POST path=/29232026/11/12 00:34:21 INFO Aborted multipart uploads count=0 kept=029242026/11/12 00:34:21 WARN Force mode enabled - objects will be deleted immediately without grace period29252026/11/12 00:34:21 INFO Received uploads request method=POST path=/29262026/11/12 00:34:21 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029272026/11/12 00:34:21 INFO Received complete upload request method=POST path=/29282026/11/12 00:34:21 INFO Vacuumed table table=pending_closures29292026/11/12 00:34:21 INFO Completed upload id=3 objects=5 duration=1.159917ms29302026/11/12 00:34:21 INFO Received push request method=POST path=/29312026/11/12 00:34:21 INFO Vacuumed table table=pending_objects29322026/11/12 00:34:21 INFO Received push request method=POST path=/29332026/11/12 00:34:21 INFO Vacuumed table table=multipart_uploads29342026/11/12 00:34:21 INFO Received complete push request method=POST path=/29352026/11/12 00:34:21 INFO Aborted multipart uploads count=0 kept=029362026/11/12 00:34:21 INFO Aborted multipart uploads count=0 kept=029372026/11/12 00:34:21 INFO Completed push id=9 objects=7 duration=3.166833ms29382026/11/12 00:34:21 INFO Vacuumed table table=closures29392026/11/12 00:34:21 WARN Force mode enabled - objects will be deleted immediately without grace period29402026/11/12 00:34:21 INFO Vacuumed table table=objects29412026/11/12 00:34:21 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=029422026/11/12 00:34:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029432026/11/12 00:34:21 INFO Received complete push request method=POST path=/29442026/11/12 00:34:21 INFO Completed push id=6 objects=8 duration=2.062542ms29452026/11/12 00:34:21 INFO Vacuumed table table=pending_closures29462026/11/12 00:34:21 INFO Received uploads request method=POST path=/29472026/11/12 00:34:21 INFO Received complete upload request method=POST path=/29482026/11/12 00:34:21 INFO Vacuumed table table=pending_objects29492026/11/12 00:34:21 INFO Received uploads request method=POST path=/29502026/11/12 00:34:21 INFO Vacuumed table table=pending_closures29512026/11/12 00:34:21 INFO Completed upload id=10 objects=23 duration=1.91775ms29522026/11/12 00:34:21 INFO Vacuumed table table=multipart_uploads29532026/11/12 00:34:21 INFO Received uploads request method=POST path=/29542026/11/12 00:34:21 INFO Vacuumed table table=closures29552026/11/12 00:34:21 INFO Vacuumed table table=pending_objects29562026/11/12 00:34:22 INFO Vacuumed table table=multipart_uploads29572026/11/12 00:34:22 INFO Vacuumed table table=objects29582026/11/12 00:34:22 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."29592026/11/12 00:34:22 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."29602026/11/12 00:34:22 INFO Received uploads request method=POST path=/29612026/11/12 00:34:22 INFO Vacuumed table table=closures29622026/11/12 00:34:22 INFO Received uploads request method=POST path=/29632026/11/12 00:34:22 INFO Received complete upload request method=POST path=/29642026/11/12 00:34:22 INFO Received complete push request method=POST path=/29652026/11/12 00:34:22 INFO Completed upload id=7 objects=4 duration=1.306625ms29662026/11/12 00:34:22 INFO Completed push id=15 objects=7 duration=2.118084ms29672026/11/12 00:34:22 INFO Vacuumed table table=objects29682026/11/12 00:34:22 INFO Received complete upload request method=POST path=/29692026/11/12 00:34:22 INFO Completed upload id=10 objects=7 duration=5.839458ms29702026/11/12 00:34:22 INFO Received push request method=POST path=/29712026/11/12 00:34:22 INFO Received uploads request method=POST path=/29722026/11/12 00:34:22 INFO Received complete push request method=POST path=/29732026/11/12 00:34:22 INFO Completed push id=14 objects=3 duration=5.419833ms29742026/11/12 00:34:22 INFO Received uploads request method=POST path=/29752026/11/12 00:34:22 INFO Received complete upload request method=POST path=/29762026/11/12 00:34:22 INFO Received complete upload request method=POST path=/29772026/11/12 00:34:22 INFO Completed upload id=14 objects=16 duration=6.887167ms29782026/11/12 00:34:22 INFO Completed upload id=11 objects=6 duration=1.053917ms29792026/11/12 00:34:22 INFO Received uploads request method=POST path=/29802026/11/12 00:34:22 INFO Received uploads request method=POST path=/29812026/11/12 00:34:22 INFO Aborted multipart uploads count=0 kept=029822026/11/12 00:34:22 INFO Received complete push request method=POST path=/29832026/11/12 00:34:22 WARN Force mode enabled - objects will be deleted immediately without grace period29842026/11/12 00:34:22 INFO Completed push id=2 objects=10 duration=7.652958ms29852026/11/12 00:34:22 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029862026/11/12 00:34:22 INFO Received uploads request method=POST path=/29872026/11/12 00:34:22 INFO Vacuumed table table=pending_closures29882026/11/12 00:34:22 INFO Received push request method=POST path=/29892026/11/12 00:34:22 INFO Vacuumed table table=pending_objects29902026/11/12 00:34:22 INFO Vacuumed table table=multipart_uploads29912026/11/12 00:34:22 INFO Vacuumed table table=closures29922026/11/12 00:34:22 INFO Received complete upload request method=POST path=/29932026/11/12 00:34:22 INFO Received uploads request method=POST path=/29942026/11/12 00:34:22 INFO Vacuumed table table=objects29952026/11/12 00:34:22 INFO Received complete upload request method=POST path=/29962026/11/12 00:34:22 INFO Received uploads request method=POST path=/29972026/11/12 00:34:22 INFO Completed upload id=5 objects=5 duration=1.560042ms29982026/11/12 00:34:22 INFO Received push request method=POST path=/29992026/11/12 00:34:22 INFO Received complete push request method=POST path=/30002026/11/12 00:34:22 INFO Completed upload id=3 objects=8 duration=6.890541ms30012026/11/12 00:34:22 INFO Received uploads request method=POST path=/30022026/11/12 00:34:22 INFO Received push request method=POST path=/30032026/11/12 00:34:22 INFO Completed push id=21 objects=9 duration=1.508209ms30042026/11/12 00:34:22 INFO Received uploads request method=POST path=/30052026/11/12 00:34:22 INFO Received complete push request method=POST path=/30062026/11/12 00:34:22 INFO Received uploads request method=POST path=/30072026/11/12 00:34:22 INFO Received push request method=POST path=/30082026/11/12 00:34:22 INFO Received complete push request method=POST path=/30092026/11/12 00:34:22 INFO Aborted multipart uploads count=0 kept=030102026/11/12 00:34:22 INFO Completed push id=8 objects=7 duration=4.614875ms30112026/11/12 00:34:22 INFO Completed push id=1 objects=5 duration=29.27ms30122026/11/12 00:34:22 INFO Received complete upload request method=POST path=/30132026/11/12 00:34:22 INFO Completed upload id=22 objects=2 duration=4.514ms30142026/11/12 00:34:22 INFO Received complete upload request method=POST path=/30152026/11/12 00:34:22 WARN Force mode enabled - objects will be deleted immediately without grace period30162026/11/12 00:34:22 INFO Completed upload id=18 objects=4 duration=1.661917ms30172026/11/12 00:34:22 INFO Received push request method=POST path=/30182026/11/12 00:34:22 INFO Received uploads request method=POST path=/30192026/11/12 00:34:22 INFO Aborted multipart uploads count=0 kept=030202026/11/12 00:34:22 INFO Received uploads request method=POST path=/30212026/11/12 00:34:22 INFO Received complete upload request method=POST path=/30222026/11/12 00:34:22 INFO Received uploads request method=POST path=/30232026/11/12 00:34:22 INFO Aborted multipart uploads count=0 kept=030242026/11/12 00:34:22 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=4 objects-failed-to-delete=030252026/11/12 00:34:22 INFO Received uploads request method=POST path=/30262026/11/12 00:34:22 INFO Received complete upload request method=POST path=/30272026/11/12 00:34:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030282026/11/12 00:34:22 INFO Vacuumed table table=pending_closures30292026/11/12 00:34:22 INFO Completed upload id=9 objects=7 duration=1.771833ms30302026/11/12 00:34:22 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=030312026/11/12 00:34:22 INFO Vacuumed table table=pending_closures30322026/11/12 00:34:22 INFO Vacuumed table table=pending_objects30332026/11/12 00:34:22 INFO Completed upload id=1 objects=9 duration=35.529625ms30342026/11/12 00:34:22 INFO Received complete push request method=POST path=/30352026/11/12 00:34:22 INFO Vacuumed table table=pending_closures30362026/11/12 00:34:22 INFO Vacuumed table table=multipart_uploads30372026/11/12 00:34:22 INFO Received push request method=POST path=/30382026/11/12 00:34:22 INFO Vacuumed table table=pending_objects30392026/11/12 00:34:22 INFO Vacuumed table table=closures30402026/11/12 00:34:22 INFO Completed push id=12 objects=15 duration=2.004792ms30412026/11/12 00:34:22 INFO Vacuumed table table=multipart_uploads30422026/11/12 00:34:22 INFO Vacuumed table table=pending_objects30432026/11/12 00:34:22 INFO Vacuumed table table=objects30442026/11/12 00:34:22 INFO Vacuumed table table=closures30452026/11/12 00:34:22 INFO Received uploads request method=POST path=/30462026/11/12 00:34:22 INFO Vacuumed table table=multipart_uploads30472026/11/12 00:34:22 INFO Aborted multipart uploads count=0 kept=030482026/11/12 00:34:22 INFO Received push request method=POST path=/30492026/11/12 00:34:22 INFO Vacuumed table table=closures30502026/11/12 00:34:22 INFO Vacuumed table table=objects30512026/11/12 00:34:22 INFO Received push request method=POST path=/30522026/11/12 00:34:22 INFO Received push request method=POST path=/30532026/11/12 00:34:22 INFO Vacuumed table table=objects30542026/11/12 00:34:22 INFO Received complete upload request method=POST path=/30552026/11/12 00:34:22 INFO Received uploads request method=POST path=/30562026/11/12 00:34:22 INFO Received uploads request method=POST path=/30572026/11/12 00:34:22 INFO Completed upload id=10 objects=6 duration=4.201917ms30582026/11/12 00:34:22 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."30592026/11/12 00:34:22 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."30602026/11/12 00:34:22 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."30612026/11/12 00:34:22 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."30622026/11/12 00:34:22 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."30632026/11/12 00:34:22 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."30642026/11/12 00:34:22 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."30652026/11/12 00:34:22 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."30662026/11/12 00:34:22 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."30672026/11/12 00:34:22 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."30682026/11/12 00:34:22 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."30692026/11/12 00:34:22 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."30702026/11/12 00:34:22 INFO Aborted multipart uploads count=0 kept=030712026/11/12 00:34:22 INFO Received uploads request method=POST path=/30722026/11/12 00:34:22 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."30732026/11/12 00:34:22 ERROR failed to remove object object=realisations/sha256:aaaaaaaa!out.doi error="Access Denied."30742026/11/12 00:34:22 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."30752026/11/12 00:34:22 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."30762026/11/12 00:34:22 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."30772026/11/12 00:34:22 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."30782026/11/12 00:34:22 INFO Received complete push request method=POST path=/30792026/11/12 00:34:22 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."30802026/11/12 00:34:22 INFO Received uploads request method=POST path=/30812026/11/12 00:34:22 INFO Aborted multipart uploads count=0 kept=030822026/11/12 00:34:22 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."30832026/11/12 00:34:22 INFO Completed push id=17 objects=11 duration=3.041958ms30842026/11/12 00:34:22 INFO Received push request method=POST path=/30852026/11/12 00:34:22 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."30862026/11/12 00:34:22 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."30872026/11/12 00:34:22 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."30882026/11/12 00:34:22 WARN Force mode enabled - objects will be deleted immediately without grace period30892026/11/12 00:34:22 INFO Aborted multipart uploads count=0 kept=030902026/11/12 00:34:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030912026/11/12 00:34:22 INFO Vacuumed table table=pending_closures30922026/11/12 00:34:22 INFO Received push request method=POST path=/30932026/11/12 00:34:22 INFO Received push request method=POST path=/30942026/11/12 00:34:22 INFO Vacuumed table table=pending_objects30952026/11/12 00:34:22 INFO Received push request method=POST path=/30962026/11/12 00:34:22 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=030972026/11/12 00:34:22 INFO Vacuumed table table=multipart_uploads30982026/11/12 00:34:22 INFO Vacuumed table table=pending_closures30992026/11/12 00:34:22 INFO Aborted multipart uploads count=0 kept=031002026/11/12 00:34:22 INFO Vacuumed table table=closures31012026/11/12 00:34:22 INFO Received complete upload request method=POST path=/31022026/11/12 00:34:22 INFO Vacuumed table table=pending_objects31032026/11/12 00:34:22 INFO Received uploads request method=POST path=/31042026/11/12 00:34:22 INFO Vacuumed table table=objects31052026/11/12 00:34:22 INFO Completed upload id=9 objects=16 duration=2.837292ms31062026/11/12 00:34:22 INFO Vacuumed table table=multipart_uploads31072026/11/12 00:34:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031082026/11/12 00:34:22 INFO Received push request method=POST path=/31092026/11/12 00:34:22 INFO Received uploads request method=POST path=/31102026/11/12 00:34:22 INFO Vacuumed table table=pending_closures31112026/11/12 00:34:22 INFO Vacuumed table table=closures31122026/11/12 00:34:22 INFO Vacuumed table table=objects31132026/11/12 00:34:22 INFO Received push request method=POST path=/31142026/11/12 00:34:22 INFO Aborted multipart uploads count=0 kept=031152026/11/12 00:34:22 INFO Vacuumed table table=pending_objects31162026/11/12 00:34:22 INFO Aborted multipart uploads count=0 kept=031172026/11/12 00:34:22 INFO Vacuumed table table=multipart_uploads31182026/11/12 00:34:22 WARN Force mode enabled - objects will be deleted immediately without grace period31192026/11/12 00:34:22 INFO Received push request method=POST path=/31202026/11/12 00:34:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031212026/11/12 00:34:22 INFO Vacuumed table table=closures31222026/11/12 00:34:22 INFO Received push request method=POST path=/31232026/11/12 00:34:22 INFO Vacuumed table table=pending_closures31242026/11/12 00:34:22 INFO Vacuumed table table=objects31252026/11/12 00:34:22 INFO Received complete push request method=POST path=/31262026/11/12 00:34:22 INFO Vacuumed table table=pending_objects31272026/11/12 00:34:22 INFO Vacuumed table table=multipart_uploads31282026/11/12 00:34:22 INFO Completed push id=10 objects=2 duration=4.36ms31292026/11/12 00:34:22 INFO Vacuumed table table=closures31302026/11/12 00:34:22 INFO Received uploads request method=POST path=/31312026/11/12 00:34:22 INFO Received push request method=POST path=/31322026/11/12 00:34:22 INFO Received push request method=POST path=/31332026/11/12 00:34:22 INFO Vacuumed table table=objects31342026/11/12 00:34:22 INFO Received complete push request method=POST path=/31352026/11/12 00:34:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=16 objects-failed-to-delete=031362026/11/12 00:34:22 INFO Completed push id=2 objects=4 duration=1.971375ms31372026/11/12 00:34:22 INFO Received complete upload request method=POST path=/31382026/11/12 00:34:22 INFO Completed upload id=26 objects=6 duration=2.408167ms31392026/11/12 00:34:22 INFO Vacuumed table table=pending_closures31402026/11/12 00:34:22 INFO Received uploads request method=POST path=/31412026/11/12 00:34:22 INFO Vacuumed table table=pending_objects31422026/11/12 00:34:22 INFO Received uploads request method=POST path=/31432026/11/12 00:34:22 INFO Aborted multipart uploads count=0 kept=031442026/11/12 00:34:22 INFO Received complete push request method=POST path=/31452026/11/12 00:34:22 INFO Vacuumed table table=multipart_uploads31462026/11/12 00:34:22 INFO Received uploads request method=POST path=/31472026/11/12 00:34:22 INFO Completed push id=10 objects=3 duration=7.204833ms31482026/11/12 00:34:22 INFO Vacuumed table table=closures31492026/11/12 00:34:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031502026/11/12 00:34:22 INFO Vacuumed table table=pending_closures31512026/11/12 00:34:22 INFO Vacuumed table table=objects31522026/11/12 00:34:22 INFO Received complete upload request method=POST path=/31532026/11/12 00:34:23 INFO Vacuumed table table=pending_objects31542026/11/12 00:34:23 INFO Received complete upload request method=POST path=/31552026/11/12 00:34:23 INFO Received push request method=POST path=/31562026/11/12 00:34:23 INFO Received complete upload request method=POST path=/31572026/11/12 00:34:23 INFO Received push request method=POST path=/31582026/11/12 00:34:23 INFO Completed upload id=12 objects=21 duration=1.0715ms31592026/11/12 00:34:23 INFO Vacuumed table table=multipart_uploads31602026/11/12 00:34:23 INFO Completed upload id=13 objects=8 duration=1.661042ms31612026/11/12 00:34:23 INFO Completed upload id=29 objects=5 duration=1.275083ms31622026/11/12 00:34:23 INFO Vacuumed table table=closures31632026/11/12 00:34:23 INFO Received uploads request method=POST path=/31642026/11/12 00:34:23 INFO Vacuumed table table=objects31652026/11/12 00:34:23 INFO Received uploads request method=POST path=/31662026/11/12 00:34:23 INFO Received push request method=POST path=/31672026/11/12 00:34:23 INFO Aborted multipart uploads count=0 kept=031682026/11/12 00:34:23 INFO Received push request method=POST path=/31692026/11/12 00:34:23 INFO Received push request method=POST path=/31702026/11/12 00:34:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031712026/11/12 00:34:23 INFO Received push request method=POST path=/31722026/11/12 00:34:23 INFO Received uploads request method=POST path=/31732026/11/12 00:34:23 INFO Received complete push request method=POST path=/31742026/11/12 00:34:23 INFO Completed push id=6 objects=26 duration=2.452292ms31752026/11/12 00:34:23 INFO Vacuumed table table=pending_closures31762026/11/12 00:34:23 INFO Received push request method=POST path=/31772026/11/12 00:34:23 INFO Aborted multipart uploads count=0 kept=031782026/11/12 00:34:23 INFO Received complete upload request method=POST path=/31792026/11/12 00:34:23 INFO Received uploads request method=POST path=/31802026/11/12 00:34:23 INFO Vacuumed table table=pending_objects31812026/11/12 00:34:23 INFO Completed upload id=15 objects=6 duration=1.25ms31822026/11/12 00:34:23 WARN Force mode enabled - objects will be deleted immediately without grace period31832026/11/12 00:34:23 INFO Vacuumed table table=multipart_uploads31842026/11/12 00:34:23 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031852026/11/12 00:34:23 INFO Received uploads request method=POST path=/31862026/11/12 00:34:23 INFO Aborted multipart uploads count=0 kept=031872026/11/12 00:34:23 INFO Vacuumed table table=closures31882026/11/12 00:34:23 INFO Received push request method=POST path=/31892026/11/12 00:34:23 INFO Vacuumed table table=objects31902026/11/12 00:34:23 INFO Vacuumed table table=pending_closures31912026/11/12 00:34:23 WARN Force mode enabled - objects will be deleted immediately without grace period31922026/11/12 00:34:23 INFO Vacuumed table table=pending_objects31932026/11/12 00:34:23 INFO Received uploads request method=POST path=/31942026/11/12 00:34:23 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031952026/11/12 00:34:23 INFO Vacuumed table table=multipart_uploads31962026/11/12 00:34:23 INFO Received push request method=POST path=/31972026/11/12 00:34:23 INFO Received uploads request method=POST path=/31982026/11/12 00:34:23 INFO Received push request method=POST path=/31992026/11/12 00:34:23 INFO Vacuumed table table=pending_closures32002026/11/12 00:34:23 INFO Received push request method=POST path=/32012026/11/12 00:34:23 INFO Aborted multipart uploads count=0 kept=032022026/11/12 00:34:23 INFO Vacuumed table table=closures32032026/11/12 00:34:23 INFO Aborted multipart uploads count=0 kept=032042026/11/12 00:34:23 INFO Received complete push request method=POST path=/32052026/11/12 00:34:23 INFO Vacuumed table table=pending_objects32062026/11/12 00:34:23 INFO Received uploads request method=POST path=/32072026/11/12 00:34:23 WARN Force mode enabled - objects will be deleted immediately without grace period32082026/11/12 00:34:23 INFO Vacuumed table table=objects32092026/11/12 00:34:23 INFO Completed push id=13 objects=23 duration=1.2055ms32102026/11/12 00:34:23 WARN Force mode enabled - objects will be deleted immediately without grace period32112026/11/12 00:34:23 INFO Received push request method=POST path=/32122026/11/12 00:34:23 INFO Vacuumed table table=multipart_uploads32132026/11/12 00:34:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032142026/11/12 00:34:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032152026/11/12 00:34:23 INFO Vacuumed table table=closures32162026/11/12 00:34:23 INFO Vacuumed table table=objects32172026/11/12 00:34:23 INFO Vacuumed table table=pending_closures32182026/11/12 00:34:23 INFO Vacuumed table table=pending_objects32192026/11/12 00:34:23 INFO Received uploads request method=POST path=/32202026/11/12 00:34:23 INFO Received uploads request method=POST path=/32212026/11/12 00:34:23 INFO Received push request method=POST path=/32222026/11/12 00:34:23 INFO Vacuumed table table=multipart_uploads32232026/11/12 00:34:23 INFO Vacuumed table table=pending_closures32242026/11/12 00:34:23 INFO Vacuumed table table=closures32252026/11/12 00:34:23 INFO Received push request method=POST path=/32262026/11/12 00:34:23 INFO Vacuumed table table=objects32272026/11/12 00:34:23 INFO Vacuumed table table=pending_objects32282026/11/12 00:34:23 INFO Received push request method=POST path=/32292026/11/12 00:34:23 INFO Received uploads request method=POST path=/32302026/11/12 00:34:23 INFO Vacuumed table table=multipart_uploads32312026/11/12 00:34:23 INFO Received push request method=POST path=/32322026/11/12 00:34:23 INFO Received uploads request method=POST path=/32332026/11/12 00:34:23 INFO Aborted multipart uploads count=0 kept=032342026/11/12 00:34:23 INFO Received uploads request method=POST path=/32352026/11/12 00:34:23 INFO Aborted multipart uploads count=0 kept=032362026/11/12 00:34:23 INFO Received complete upload request method=POST path=/32372026/11/12 00:34:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=032382026/11/12 00:34:23 WARN Force mode enabled - objects will be deleted immediately without grace period32392026/11/12 00:34:23 INFO Completed upload id=11 objects=6 duration=4.285583ms32402026/11/12 00:34:23 INFO Received uploads request method=POST path=/32412026/11/12 00:34:23 INFO Received uploads request method=POST path=/32422026/11/12 00:34:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032432026/11/12 00:34:23 INFO Vacuumed table table=closures32442026/11/12 00:34:23 INFO Vacuumed table table=pending_closures32452026/11/12 00:34:23 INFO Received uploads request method=POST path=/32462026/11/12 00:34:23 INFO Received push request method=POST path=/32472026/11/12 00:34:23 INFO Vacuumed table table=pending_objects32482026/11/12 00:34:23 INFO Vacuumed table table=pending_closures32492026/11/12 00:34:23 INFO Vacuumed table table=multipart_uploads32502026/11/12 00:34:23 INFO Received push request method=POST path=/32512026/11/12 00:34:23 INFO Vacuumed table table=objects32522026/11/12 00:34:23 INFO Vacuumed table table=closures32532026/11/12 00:34:23 INFO Aborted multipart uploads count=0 kept=032542026/11/12 00:34:23 INFO Vacuumed table table=pending_objects32552026/11/12 00:34:23 INFO Vacuumed table table=objects32562026/11/12 00:34:23 INFO Received uploads request method=POST path=/32572026/11/12 00:34:23 INFO Received complete upload request method=POST path=/32582026/11/12 00:34:23 WARN Force mode enabled - objects will be deleted immediately without grace period32592026/11/12 00:34:23 INFO Vacuumed table table=multipart_uploads32602026/11/12 00:34:23 INFO Received push request method=POST path=/32612026/11/12 00:34:23 INFO Completed upload id=31 objects=3 duration=1.951334ms32622026/11/12 00:34:23 INFO Aborted multipart uploads count=0 kept=032632026/11/12 00:34:23 INFO Received push request method=POST path=/32642026/11/12 00:34:23 INFO Vacuumed table table=closures32652026/11/12 00:34:23 INFO Received push request method=POST path=/32662026/11/12 00:34:23 INFO Received uploads request method=POST path=/32672026/11/12 00:34:23 INFO Vacuumed table table=objects32682026/11/12 00:34:23 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=032692026/11/12 00:34:23 INFO Received complete upload request method=POST path=/32702026/11/12 00:34:23 INFO Received uploads request method=POST path=/32712026/11/12 00:34:23 INFO Vacuumed table table=pending_closures32722026/11/12 00:34:23 INFO Completed upload id=22 objects=2 duration=1.294792ms32732026/11/12 00:34:23 INFO Vacuumed table table=pending_objects32742026/11/12 00:34:23 INFO Received uploads request method=POST path=/32752026/11/12 00:34:23 INFO Vacuumed table table=multipart_uploads32762026/11/12 00:34:23 INFO Vacuumed table table=closures32772026/11/12 00:34:23 INFO Received complete push request method=POST path=/32782026/11/12 00:34:23 INFO Vacuumed table table=objects32792026/11/12 00:34:23 INFO Received push request method=POST path=/32802026/11/12 00:34:23 INFO Completed push id=9 objects=5 duration=2.230167ms32812026/11/12 00:34:23 INFO Aborted multipart uploads count=0 kept=032822026/11/12 00:34:23 WARN Force mode enabled - objects will be deleted immediately without grace period32832026/11/12 00:34:23 INFO Aborted multipart uploads count=0 kept=032842026/11/12 00:34:23 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032852026/11/12 00:34:23 INFO Received push request method=POST path=/32862026/11/12 00:34:23 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032872026/11/12 00:34:23 INFO Vacuumed table table=pending_closures32882026/11/12 00:34:23 INFO Vacuumed table table=pending_closures32892026/11/12 00:34:23 INFO Vacuumed table table=pending_objects32902026/11/12 00:34:23 INFO Received complete push request method=POST path=/32912026/11/12 00:34:23 INFO Vacuumed table table=pending_objects32922026/11/12 00:34:23 INFO Aborted multipart uploads count=0 kept=032932026/11/12 00:34:23 INFO Completed push id=16 objects=27 duration=1.482584ms32942026/11/12 00:34:23 INFO Vacuumed table table=multipart_uploads32952026/11/12 00:34:23 INFO Vacuumed table table=multipart_uploads32962026/11/12 00:34:23 WARN Force mode enabled - objects will be deleted immediately without grace period32972026/11/12 00:34:23 INFO Vacuumed table table=closures32982026/11/12 00:34:23 INFO Vacuumed table table=closures32992026/11/12 00:34:23 INFO Received complete push request method=POST path=/33002026/11/12 00:34:23 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033012026/11/12 00:34:23 INFO Received push request method=POST path=/33022026/11/12 00:34:23 INFO Vacuumed table table=objects33032026/11/12 00:34:23 INFO Vacuumed table table=objects33042026/11/12 00:34:23 INFO Completed push id=1 objects=11 duration=4.583458ms33052026/11/12 00:34:23 INFO Vacuumed table table=pending_closures33062026/11/12 00:34:23 INFO Received uploads request method=POST path=/33072026/11/12 00:34:23 INFO Received uploads request method=POST path=/33082026/11/12 00:34:23 INFO Vacuumed table table=pending_objects33092026/11/12 00:34:23 INFO Vacuumed table table=multipart_uploads33102026/11/12 00:34:23 INFO Received uploads request method=POST path=/33112026/11/12 00:34:23 INFO Received complete push request method=POST path=/33122026/11/12 00:34:23 INFO Vacuumed table table=closures33132026/11/12 00:34:23 INFO Received complete upload request method=POST path=/33142026/11/12 00:34:23 INFO Completed push id=5 objects=28 duration=1.939083ms33152026/11/12 00:34:23 INFO Vacuumed table table=objects33162026/11/12 00:34:23 INFO Received complete upload request method=POST path=/33172026/11/12 00:34:23 INFO Completed upload id=18 objects=2 duration=5.542292ms33182026/11/12 00:34:23 INFO Received push request method=POST path=/33192026/11/12 00:34:23 INFO Received uploads request method=POST path=/33202026/11/12 00:34:23 INFO Completed upload id=17 objects=6 duration=2.484333ms33212026/11/12 00:34:23 INFO Received uploads request method=POST path=/33222026/11/12 00:34:23 INFO Received complete push request method=POST path=/33232026/11/12 00:34:23 INFO Completed push id=10 objects=2 duration=1.563291ms33242026/11/12 00:34:23 INFO Received push request method=POST path=/33252026/11/12 00:34:23 INFO Received push request method=POST path=/33262026/11/12 00:34:23 INFO Received complete upload request method=POST path=/33272026/11/12 00:34:23 INFO Received complete upload request method=POST path=/33282026/11/12 00:34:23 INFO Completed upload id=16 objects=3 duration=1.108083ms33292026/11/12 00:34:24 INFO Received complete upload request method=POST path=/33302026/11/12 00:34:24 INFO Received complete upload request method=POST path=/33312026/11/12 00:34:24 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."33322026/11/12 00:34:24 INFO Completed upload id=15 objects=4 duration=4.708417ms33332026/11/12 00:34:24 INFO Completed upload id=33 objects=5 duration=8.052375ms33342026/11/12 00:34:24 INFO Completed upload id=3 objects=6 duration=23.379709ms33352026/11/12 00:34:24 INFO Aborted multipart uploads count=0 kept=033362026/11/12 00:34:24 INFO Received push request method=POST path=/33372026/11/12 00:34:24 WARN Force mode enabled - objects will be deleted immediately without grace period33382026/11/12 00:34:24 INFO Received push request method=POST path=/33392026/11/12 00:34:24 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033402026/11/12 00:34:24 INFO Aborted multipart uploads count=0 kept=033412026/11/12 00:34:24 INFO Received push request method=POST path=/33422026/11/12 00:34:24 INFO Vacuumed table table=pending_closures33432026/11/12 00:34:24 INFO Received complete upload request method=POST path=/33442026/11/12 00:34:24 INFO Received push request method=POST path=/33452026/11/12 00:34:24 INFO Vacuumed table table=pending_objects33462026/11/12 00:34:24 INFO Completed upload id=6 objects=7 duration=2.53475ms33472026/11/12 00:34:24 INFO Vacuumed table table=multipart_uploads33482026/11/12 00:34:24 INFO Received complete upload request method=POST path=/33492026/11/12 00:34:24 INFO Received uploads request method=POST path=/33502026/11/12 00:34:24 INFO Received complete push request method=POST path=/33512026/11/12 00:34:24 INFO Vacuumed table table=closures33522026/11/12 00:34:24 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.narinfo error="Access Denied."33532026/11/12 00:34:24 INFO Completed upload id=17 objects=5 duration=6.088083ms33542026/11/12 00:34:24 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."33552026/11/12 00:34:24 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.narinfo error="Access Denied."33562026/11/12 00:34:24 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."33572026/11/12 00:34:24 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."33582026/11/12 00:34:24 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."33592026/11/12 00:34:24 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."33602026/11/12 00:34:24 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."33612026/11/12 00:34:24 ERROR failed to remove object object=nar/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.nar.zst error="Access Denied."33622026/11/12 00:34:24 ERROR failed to remove object object=nar/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.nar.zst error="Access Denied."33632026/11/12 00:34:24 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."33642026/11/12 00:34:24 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."33652026/11/12 00:34:24 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."33662026/11/12 00:34:24 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."33672026/11/12 00:34:24 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."33682026/11/12 00:34:24 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."33692026/11/12 00:34:24 INFO Completed push id=18 objects=10 duration=902.958µs33702026/11/12 00:34:24 INFO Vacuumed table table=objects33712026/11/12 00:34:24 INFO Received push request method=POST path=/33722026/11/12 00:34:24 INFO Received push request method=POST path=/33732026/11/12 00:34:24 INFO Received complete upload request method=POST path=/33742026/11/12 00:34:24 INFO Aborted multipart uploads count=0 kept=033752026/11/12 00:34:24 INFO Aborted multipart uploads count=0 kept=033762026/11/12 00:34:24 INFO Received complete push request method=POST path=/33772026/11/12 00:34:24 INFO Completed upload id=16 objects=3 duration=7.20325ms33782026/11/12 00:34:24 INFO Completed push id=2 objects=25 duration=4.6795ms33792026/11/12 00:34:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033802026/11/12 00:34:24 INFO Vacuumed table table=pending_closures33812026/11/12 00:34:24 INFO Received uploads request method=POST path=/33822026/11/12 00:34:24 INFO Vacuumed table table=pending_objects33832026/11/12 00:34:24 INFO Aborted multipart uploads count=0 kept=033842026/11/12 00:34:24 INFO Vacuumed table table=multipart_uploads33852026/11/12 00:34:24 INFO Received complete upload request method=POST path=/33862026/11/12 00:34:24 INFO Vacuumed table table=closures33872026/11/12 00:34:24 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033882026/11/12 00:34:24 INFO Completed upload id=13 objects=7 duration=3.049875ms33892026/11/12 00:34:24 INFO Received complete upload request method=POST path=/33902026/11/12 00:34:24 INFO Vacuumed table table=objects33912026/11/12 00:34:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=16 objects-failed-to-delete=033922026/11/12 00:34:24 INFO Vacuumed table table=pending_closures33932026/11/12 00:34:24 INFO Vacuumed table table=pending_closures33942026/11/12 00:34:24 INFO Received complete push request method=POST path=/33952026/11/12 00:34:24 INFO Received complete push request method=POST path=/33962026/11/12 00:34:24 INFO Vacuumed table table=pending_objects33972026/11/12 00:34:24 INFO Completed upload id=1 objects=15 duration=26.975083ms33982026/11/12 00:34:24 INFO Vacuumed table table=pending_objects33992026/11/12 00:34:24 INFO Completed push id=19 objects=3 duration=1.821333ms34002026/11/12 00:34:24 INFO Vacuumed table table=multipart_uploads34012026/11/12 00:34:24 INFO Vacuumed table table=multipart_uploads34022026/11/12 00:34:24 INFO Completed push id=1 objects=7 duration=22.148333ms34032026/11/12 00:34:24 INFO Vacuumed table table=closures34042026/11/12 00:34:24 INFO Vacuumed table table=objects34052026/11/12 00:34:24 INFO Received complete upload request method=POST path=/34062026/11/12 00:34:24 INFO Vacuumed table table=closures34072026/11/12 00:34:24 INFO Received push request method=POST path=/34082026/11/12 00:34:24 INFO Received push request method=POST path=/34092026/11/12 00:34:24 INFO Vacuumed table table=objects34102026/11/12 00:34:24 INFO Completed upload id=6 objects=24 duration=2.069209ms34112026/11/12 00:34:24 INFO Aborted multipart uploads count=0 kept=034122026/11/12 00:34:24 INFO Received push request method=POST path=/34132026/11/12 00:34:24 INFO Received complete upload request method=POST path=/34142026/11/12 00:34:24 INFO Completed upload id=20 objects=4 duration=619.875µs34152026/11/12 00:34:24 WARN Force mode enabled - objects will be deleted immediately without grace period34162026/11/12 00:34:24 INFO Received uploads request method=POST path=/34172026/11/12 00:34:24 INFO Garbage collection completed failed-uploads-deleted=5 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034182026/11/12 00:34:24 INFO Vacuumed table table=pending_closures34192026/11/12 00:34:24 INFO Received uploads request method=POST path=/34202026/11/12 00:34:24 INFO Received uploads request method=POST path=/34212026/11/12 00:34:24 INFO Aborted multipart uploads count=0 kept=034222026/11/12 00:34:24 INFO Received complete upload request method=POST path=/34232026/11/12 00:34:24 INFO Received push request method=POST path=/34242026/11/12 00:34:24 INFO Completed upload id=23 objects=7 duration=2.886167ms34252026/11/12 00:34:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034262026/11/12 00:34:24 INFO Vacuumed table table=pending_objects34272026/11/12 00:34:24 INFO Vacuumed table table=multipart_uploads34282026/11/12 00:34:24 INFO Received complete upload request method=POST path=/34292026/11/12 00:34:24 INFO Completed upload id=15 objects=9 duration=1.034875ms34302026/11/12 00:34:24 INFO Received uploads request method=POST path=/34312026/11/12 00:34:24 INFO Vacuumed table table=pending_closures34322026/11/12 00:34:24 INFO Vacuumed table table=pending_objects34332026/11/12 00:34:24 INFO Vacuumed table table=closures34342026/11/12 00:34:24 INFO Vacuumed table table=multipart_uploads34352026/11/12 00:34:24 INFO Aborted multipart uploads count=0 kept=034362026/11/12 00:34:24 INFO Vacuumed table table=closures34372026/11/12 00:34:24 INFO Received complete upload request method=POST path=/34382026/11/12 00:34:24 INFO Received push request method=POST path=/34392026/11/12 00:34:24 INFO Completed upload id=18 objects=4 duration=1.5945ms34402026/11/12 00:34:24 INFO Vacuumed table table=objects34412026/11/12 00:34:24 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."34422026/11/12 00:34:24 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."34432026/11/12 00:34:24 INFO Vacuumed table table=objects34442026/11/12 00:34:24 INFO Received complete push request method=POST path=/34452026/11/12 00:34:24 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."34462026/11/12 00:34:24 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."34472026/11/12 00:34:24 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."34482026/11/12 00:34:24 INFO Completed push id=8 objects=6 duration=2.977583ms34492026/11/12 00:34:24 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."34502026/11/12 00:34:24 INFO Received push request method=POST path=/34512026/11/12 00:34:24 INFO Received complete push request method=POST path=/34522026/11/12 00:34:24 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."34532026/11/12 00:34:24 INFO Completed push id=20 objects=4 duration=1.766333ms34542026/11/12 00:34:24 INFO Received push request method=POST path=/34552026/11/12 00:34:24 INFO Received push request method=POST path=/34562026/11/12 00:34:24 INFO Aborted multipart uploads count=0 kept=034572026/11/12 00:34:24 INFO Received complete push request method=POST path=/34582026/11/12 00:34:24 INFO Completed push id=17 objects=6 duration=2.462709ms34592026/11/12 00:34:24 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."34602026/11/12 00:34:24 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."34612026/11/12 00:34:24 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."34622026/11/12 00:34:24 INFO Received complete upload request method=POST path=/34632026/11/12 00:34:24 INFO Received push request method=POST path=/34642026/11/12 00:34:24 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."34652026/11/12 00:34:24 INFO Completed upload id=19 objects=25 duration=4.41775ms34662026/11/12 00:34:24 INFO Received complete push request method=POST path=/34672026/11/12 00:34:24 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."34682026/11/12 00:34:24 INFO Received complete upload request method=POST path=/34692026/11/12 00:34:24 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."34702026/11/12 00:34:24 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."34712026/11/12 00:34:24 INFO Completed push id=21 objects=17 duration=6.210375ms34722026/11/12 00:34:24 INFO Completed upload id=21 objects=25 duration=735.542µs34732026/11/12 00:34:24 INFO Received complete upload request method=POST path=/34742026/11/12 00:34:24 INFO Received push request method=POST path=/34752026/11/12 00:34:24 INFO Received push request method=POST path=/34762026/11/12 00:34:24 INFO Completed upload id=9 objects=3 duration=4.245417ms34772026/11/12 00:34:24 INFO Aborted multipart uploads count=0 kept=034782026/11/12 00:34:24 INFO Received push request method=POST path=/34792026/11/12 00:34:24 INFO Received complete push request method=POST path=/34802026/11/12 00:34:24 WARN Force mode enabled - objects will be deleted immediately without grace period34812026/11/12 00:34:24 INFO Received uploads request method=POST path=/34822026/11/12 00:34:24 INFO Completed push id=19 objects=7 duration=1.028458ms34832026/11/12 00:34:24 INFO Received push request method=POST path=/34842026/11/12 00:34:24 INFO Received push request method=POST path=/34852026/11/12 00:34:24 INFO Aborted multipart uploads count=0 kept=034862026/11/12 00:34:24 INFO Received push request method=POST path=/34872026/11/12 00:34:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=034882026/11/12 00:34:24 INFO Aborted multipart uploads count=0 kept=034892026/11/12 00:34:24 INFO Vacuumed table table=pending_closures34902026/11/12 00:34:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034912026/11/12 00:34:24 INFO Received push request method=POST path=/34922026/11/12 00:34:24 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034932026/11/12 00:34:24 INFO Received complete push request method=POST path=/34942026/11/12 00:34:24 INFO Received push request method=POST path=/34952026/11/12 00:34:24 INFO Vacuumed table table=pending_objects34962026/11/12 00:34:24 INFO Vacuumed table table=pending_closures34972026/11/12 00:34:24 INFO Vacuumed table table=pending_closures34982026/11/12 00:34:24 INFO Aborted multipart uploads count=0 kept=034992026/11/12 00:34:24 INFO Completed push id=9 objects=4 duration=5.53925ms35002026/11/12 00:34:24 INFO Vacuumed table table=multipart_uploads35012026/11/12 00:34:24 INFO Vacuumed table table=pending_objects35022026/11/12 00:34:24 INFO Vacuumed table table=closures35032026/11/12 00:34:24 INFO Vacuumed table table=pending_objects35042026/11/12 00:34:24 INFO Vacuumed table table=objects35052026/11/12 00:34:24 INFO Vacuumed table table=multipart_uploads35062026/11/12 00:34:24 INFO Vacuumed table table=multipart_uploads35072026/11/12 00:34:24 INFO Vacuumed table table=closures35082026/11/12 00:34:24 INFO Vacuumed table table=closures35092026/11/12 00:34:24 INFO Received complete upload request method=POST path=/35102026/11/12 00:34:24 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."35112026/11/12 00:34:24 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."35122026/11/12 00:34:24 INFO Vacuumed table table=objects35132026/11/12 00:34:24 INFO Vacuumed table table=objects35142026/11/12 00:34:24 INFO Completed upload id=22 objects=8 duration=2.476667ms35152026/11/12 00:34:24 INFO Aborted multipart uploads count=0 kept=035162026/11/12 00:34:24 INFO Received complete push request method=POST path=/35172026/11/12 00:34:24 INFO Received push request method=POST path=/35182026/11/12 00:34:24 INFO Aborted multipart uploads count=0 kept=035192026/11/12 00:34:24 INFO Completed push id=4 objects=6 duration=1.488125ms35202026/11/12 00:34:24 INFO Aborted multipart uploads count=0 kept=035212026/11/12 00:34:24 WARN Force mode enabled - objects will be deleted immediately without grace period35222026/11/12 00:34:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035232026/11/12 00:34:24 WARN Force mode enabled - objects will be deleted immediately without grace period35242026/11/12 00:34:24 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035252026/11/12 00:34:24 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035262026/11/12 00:34:24 INFO Vacuumed table table=pending_closures35272026/11/12 00:34:24 INFO Vacuumed table table=pending_objects35282026/11/12 00:34:24 INFO Vacuumed table table=pending_closures35292026/11/12 00:34:24 INFO Vacuumed table table=multipart_uploads35302026/11/12 00:34:24 INFO Vacuumed table table=closures35312026/11/12 00:34:24 INFO Vacuumed table table=pending_closures35322026/11/12 00:34:24 INFO Received uploads request method=POST path=/35332026/11/12 00:34:24 INFO Received complete push request method=POST path=/35342026/11/12 00:34:24 INFO Aborted multipart uploads count=0 kept=035352026/11/12 00:34:25 INFO Vacuumed table table=objects35362026/11/12 00:34:25 INFO Vacuumed table table=pending_objects35372026/11/12 00:34:25 INFO Received complete upload request method=POST path=/35382026/11/12 00:34:25 INFO Completed push id=4 objects=4 duration=5.812709ms35392026/11/12 00:34:25 INFO Received uploads request method=POST path=/35402026/11/12 00:34:25 INFO Completed upload id=24 objects=18 duration=3.657834ms35412026/11/12 00:34:25 INFO Vacuumed table table=multipart_uploads35422026/11/12 00:34:25 INFO Received complete push request method=POST path=/35432026/11/12 00:34:25 INFO Vacuumed table table=pending_objects35442026/11/12 00:34:25 INFO Received push request method=POST path=/35452026/11/12 00:34:25 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."35462026/11/12 00:34:25 INFO Completed push id=23 objects=5 duration=2.825958ms35472026/11/12 00:34:25 INFO Vacuumed table table=multipart_uploads35482026/11/12 00:34:25 INFO Vacuumed table table=closures35492026/11/12 00:34:25 INFO Received uploads request method=POST path=/35502026/11/12 00:34:25 INFO Vacuumed table table=closures35512026/11/12 00:34:25 INFO Received complete push request method=POST path=/35522026/11/12 00:34:25 INFO Received uploads request method=POST path=/35532026/11/12 00:34:25 INFO Received push request method=POST path=/35542026/11/12 00:34:25 INFO Vacuumed table table=objects35552026/11/12 00:34:25 INFO Received uploads request method=POST path=/35562026/11/12 00:34:25 INFO Received uploads request method=POST path=/35572026/11/12 00:34:25 INFO Completed push id=25 objects=2 duration=1.181958ms35582026/11/12 00:34:25 INFO Received complete push request method=POST path=/35592026/11/12 00:34:25 INFO Vacuumed table table=objects35602026/11/12 00:34:25 INFO Received complete upload request method=POST path=/35612026/11/12 00:34:25 INFO Completed push id=2 objects=7 duration=3.699333ms35622026/11/12 00:34:25 INFO Completed upload id=24 objects=8 duration=1.887166ms35632026/11/12 00:34:25 INFO Aborted multipart uploads count=0 kept=035642026/11/12 00:34:25 INFO Received complete push request method=POST path=/35652026/11/12 00:34:25 INFO Completed push id=12 objects=3 duration=6.550958ms35662026/11/12 00:34:25 INFO Aborted multipart uploads count=0 kept=035672026/11/12 00:34:25 WARN Force mode enabled - objects will be deleted immediately without grace period35682026/11/12 00:34:25 INFO Received uploads request method=POST path=/35692026/11/12 00:34:25 INFO Received complete upload request method=POST path=/35702026/11/12 00:34:25 INFO Received complete push request method=POST path=/35712026/11/12 00:34:25 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035722026/11/12 00:34:25 INFO Aborted multipart uploads count=0 kept=035732026/11/12 00:34:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035742026/11/12 00:34:25 INFO Completed upload id=28 objects=5 duration=1.528542ms35752026/11/12 00:34:25 INFO Completed push id=27 objects=4 duration=3.850875ms35762026/11/12 00:34:25 INFO Vacuumed table table=pending_closures35772026/11/12 00:34:25 INFO Vacuumed table table=pending_closures35782026/11/12 00:34:25 INFO Vacuumed table table=pending_objects35792026/11/12 00:34:25 INFO Received uploads request method=POST path=/35802026/11/12 00:34:25 INFO Received uploads request method=POST path=/35812026/11/12 00:34:25 INFO Vacuumed table table=pending_objects35822026/11/12 00:34:25 INFO Received complete push request method=POST path=/35832026/11/12 00:34:25 INFO Vacuumed table table=multipart_uploads35842026/11/12 00:34:25 INFO Received complete push request method=POST path=/35852026/11/12 00:34:25 INFO Vacuumed table table=multipart_uploads35862026/11/12 00:34:25 INFO Completed push id=6 objects=6 duration=995.208µs35872026/11/12 00:34:25 INFO Vacuumed table table=closures35882026/11/12 00:34:25 INFO Completed push id=6 objects=5 duration=2.899416ms35892026/11/12 00:34:25 INFO Received complete upload request method=POST path=/35902026/11/12 00:34:25 INFO Vacuumed table table=closures35912026/11/12 00:34:25 INFO Vacuumed table table=objects35922026/11/12 00:34:25 INFO Received push request method=POST path=/35932026/11/12 00:34:25 INFO Completed upload id=12 objects=6 duration=3.781417ms35942026/11/12 00:34:25 INFO Received uploads request method=POST path=/35952026/11/12 00:34:25 INFO Received push request method=POST path=/35962026/11/12 00:34:25 INFO Received push request method=POST path=/35972026/11/12 00:34:25 INFO Vacuumed table table=objects35982026/11/12 00:34:25 INFO Received uploads request method=POST path=/35992026/11/12 00:34:25 INFO Received push request method=POST path=/36002026/11/12 00:34:25 INFO Received uploads request method=POST path=/36012026/11/12 00:34:25 INFO Received complete push request method=POST path=/36022026/11/12 00:34:25 INFO Completed push id=17 objects=3 duration=2.180875ms36032026/11/12 00:34:25 INFO Received push request method=POST path=/36042026/11/12 00:34:25 INFO Received uploads request method=POST path=/36052026/11/12 00:34:25 INFO Received push request method=POST path=/36062026/11/12 00:34:25 INFO Received complete upload request method=POST path=/36072026/11/12 00:34:25 INFO Received complete upload request method=POST path=/36082026/11/12 00:34:25 INFO Completed upload id=12 objects=26 duration=2.305833ms36092026/11/12 00:34:25 INFO Aborted multipart uploads count=0 kept=036102026/11/12 00:34:25 INFO Completed upload id=30 objects=5 duration=680.458µs36112026/11/12 00:34:25 WARN Force mode enabled - objects will be deleted immediately without grace period36122026/11/12 00:34:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036132026/11/12 00:34:25 INFO Vacuumed table table=pending_closures36142026/11/12 00:34:25 INFO Vacuumed table table=pending_objects36152026/11/12 00:34:25 INFO Vacuumed table table=multipart_uploads36162026/11/12 00:34:25 INFO Received push request method=POST path=/36172026/11/12 00:34:25 INFO Vacuumed table table=closures36182026/11/12 00:34:25 INFO Aborted multipart uploads count=0 kept=036192026/11/12 00:34:25 INFO Received uploads request method=POST path=/36202026/11/12 00:34:25 INFO Vacuumed table table=objects36212026/11/12 00:34:25 INFO Received complete push request method=POST path=/36222026/11/12 00:34:25 WARN Force mode enabled - objects will be deleted immediately without grace period36232026/11/12 00:34:25 INFO Completed push id=23 objects=5 duration=699.625µs36242026/11/12 00:34:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036252026/11/12 00:34:25 INFO Aborted multipart uploads count=0 kept=036262026/11/12 00:34:25 INFO Aborted multipart uploads count=0 kept=036272026/11/12 00:34:25 WARN Force mode enabled - objects will be deleted immediately without grace period36282026/11/12 00:34:25 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036292026/11/12 00:34:25 INFO Received uploads request method=POST path=/36302026/11/12 00:34:25 INFO Vacuumed table table=pending_closures36312026/11/12 00:34:25 INFO Received complete upload request method=POST path=/36322026/11/12 00:34:25 INFO Received uploads request method=POST path=/36332026/11/12 00:34:25 INFO Vacuumed table table=pending_closures36342026/11/12 00:34:25 INFO Received complete push request method=POST path=/36352026/11/12 00:34:25 INFO Received complete upload request method=POST path=/36362026/11/12 00:34:25 INFO Received complete upload request method=POST path=/36372026/11/12 00:34:25 INFO Completed upload id=23 objects=18 duration=2.012208ms36382026/11/12 00:34:25 INFO Aborted multipart uploads count=0 kept=036392026/11/12 00:34:25 INFO Vacuumed table table=pending_objects36402026/11/12 00:34:25 INFO Completed push id=28 objects=26 duration=1.08675ms36412026/11/12 00:34:25 INFO Completed upload id=24 objects=3 duration=1.922667ms36422026/11/12 00:34:25 INFO Completed upload id=16 objects=23 duration=2.268833ms36432026/11/12 00:34:25 INFO Vacuumed table table=pending_objects36442026/11/12 00:34:25 INFO Received complete push request method=POST path=/36452026/11/12 00:34:25 INFO Received complete push request method=POST path=/36462026/11/12 00:34:25 WARN Force mode enabled - objects will be deleted immediately without grace period36472026/11/12 00:34:25 INFO Vacuumed table table=multipart_uploads36482026/11/12 00:34:25 INFO Received complete push request method=POST path=/36492026/11/12 00:34:25 INFO Vacuumed table table=multipart_uploads36502026/11/12 00:34:25 INFO Completed push id=16 objects=8 duration=1.605917ms36512026/11/12 00:34:25 INFO Completed push id=16 objects=4 duration=1.254458ms36522026/11/12 00:34:25 INFO Vacuumed table table=closures36532026/11/12 00:34:25 INFO Completed push id=31 objects=3 duration=2.874125ms36542026/11/12 00:34:25 INFO Vacuumed table table=closures36552026/11/12 00:34:25 INFO Received uploads request method=POST path=/36562026/11/12 00:34:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036572026/11/12 00:34:25 INFO Received complete push request method=POST path=/36582026/11/12 00:34:25 INFO Vacuumed table table=objects36592026/11/12 00:34:25 INFO Received complete push request method=POST path=/36602026/11/12 00:34:25 INFO Received complete upload request method=POST path=/36612026/11/12 00:34:25 INFO Received uploads request method=POST path=/36622026/11/12 00:34:25 INFO Vacuumed table table=objects36632026/11/12 00:34:25 INFO Aborted multipart uploads count=0 kept=036642026/11/12 00:34:25 INFO Vacuumed table table=pending_closures36652026/11/12 00:34:25 INFO Completed push id=30 objects=3 duration=3.061875ms36662026/11/12 00:34:25 INFO Completed push id=25 objects=4 duration=932.834µs36672026/11/12 00:34:25 INFO Received complete upload request method=POST path=/36682026/11/12 00:34:25 INFO Completed upload id=15 objects=6 duration=2.183917ms36692026/11/12 00:34:25 INFO Vacuumed table table=pending_objects36702026/11/12 00:34:25 INFO Completed upload id=3 objects=24 duration=1.379209ms36712026/11/12 00:34:25 WARN Force mode enabled - objects will be deleted immediately without grace period36722026/11/12 00:34:25 INFO Vacuumed table table=multipart_uploads36732026/11/12 00:34:25 INFO Vacuumed table table=closures36742026/11/12 00:34:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036752026/11/12 00:34:25 INFO Received push request method=POST path=/36762026/11/12 00:34:25 INFO Received push request method=POST path=/36772026/11/12 00:34:25 INFO Vacuumed table table=pending_closures36782026/11/12 00:34:25 INFO Vacuumed table table=objects36792026/11/12 00:34:25 INFO Vacuumed table table=pending_objects36802026/11/12 00:34:25 INFO Vacuumed table table=multipart_uploads36812026/11/12 00:34:25 INFO Vacuumed table table=closures36822026/11/12 00:34:25 INFO Received uploads request method=POST path=/36832026/11/12 00:34:25 INFO Received complete push request method=POST path=/36842026/11/12 00:34:25 INFO Received complete upload request method=POST path=/36852026/11/12 00:34:25 INFO Vacuumed table table=objects36862026/11/12 00:34:25 INFO Completed push id=32 objects=19 duration=1.267833ms36872026/11/12 00:34:25 INFO Completed upload id=20 objects=5 duration=1.551459ms36882026/11/12 00:34:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=036892026/11/12 00:34:25 INFO Vacuumed table table=pending_closures36902026/11/12 00:34:25 INFO Vacuumed table table=pending_objects36912026/11/12 00:34:25 INFO Received complete push request method=POST path=/36922026/11/12 00:34:25 INFO Received complete upload request method=POST path=/36932026/11/12 00:34:25 INFO Vacuumed table table=multipart_uploads36942026/11/12 00:34:25 INFO Received uploads request method=POST path=/36952026/11/12 00:34:25 INFO Completed upload id=22 objects=22 duration=848.375µs36962026/11/12 00:34:25 INFO Completed push id=7 objects=27 duration=4.463333ms36972026/11/12 00:34:25 INFO Received uploads request method=POST path=/36982026/11/12 00:34:25 INFO Aborted multipart uploads count=0 kept=036992026/11/12 00:34:25 INFO Received complete push request method=POST path=/37002026/11/12 00:34:25 WARN Force mode enabled - objects will be deleted immediately without grace period37012026/11/12 00:34:25 INFO Completed push id=15 objects=24 duration=7.879834ms37022026/11/12 00:34:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037032026/11/12 00:34:25 INFO Received push request method=POST path=/37042026/11/12 00:34:25 INFO Vacuumed table table=closures37052026/11/12 00:34:25 INFO Received push request method=POST path=/37062026/11/12 00:34:25 INFO Received push request method=POST path=/37072026/11/12 00:34:25 INFO Vacuumed table table=pending_closures37082026/11/12 00:34:25 INFO Received uploads request method=POST path=/37092026/11/12 00:34:25 INFO Received complete push request method=POST path=/37102026/11/12 00:34:25 INFO Received complete upload request method=POST path=/37112026/11/12 00:34:25 INFO Vacuumed table table=objects37122026/11/12 00:34:25 INFO Completed upload id=17 objects=7 duration=1.466459ms37132026/11/12 00:34:25 INFO Received complete push request method=POST path=/37142026/11/12 00:34:25 INFO Completed push id=32 objects=6 duration=1.713541ms37152026/11/12 00:34:25 INFO Vacuumed table table=pending_objects37162026/11/12 00:34:25 INFO Received complete push request method=POST path=/37172026/11/12 00:34:25 INFO Completed push id=11 objects=6 duration=2.354875ms37182026/11/12 00:34:25 INFO Vacuumed table table=multipart_uploads37192026/11/12 00:34:25 INFO Completed push id=26 objects=14 duration=2.860041ms37202026/11/12 00:34:25 INFO Received complete push request method=POST path=/37212026/11/12 00:34:25 INFO Vacuumed table table=closures37222026/11/12 00:34:25 INFO Received uploads request method=POST path=/37232026/11/12 00:34:25 INFO Received push request method=POST path=/37242026/11/12 00:34:25 INFO Completed push id=26 objects=5 duration=2.3455ms37252026/11/12 00:34:25 INFO Aborted multipart uploads count=0 kept=037262026/11/12 00:34:25 INFO Vacuumed table table=objects37272026/11/12 00:34:25 INFO Received complete upload request method=POST path=/37282026/11/12 00:34:25 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."37292026/11/12 00:34:25 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."37302026/11/12 00:34:25 INFO Completed upload id=33 objects=7 duration=3.039667ms37312026/11/12 00:34:25 INFO Received complete upload request method=POST path=/37322026/11/12 00:34:25 INFO Received push request method=POST path=/37332026/11/12 00:34:25 INFO Completed upload id=17 objects=25 duration=2.614792ms37342026/11/12 00:34:25 INFO Received push request method=POST path=/37352026/11/12 00:34:25 INFO Received complete upload request method=POST path=/37362026/11/12 00:34:25 INFO Completed upload id=24 objects=14 duration=2.445042ms37372026/11/12 00:34:25 INFO Received uploads request method=POST path=/37382026/11/12 00:34:25 INFO Received complete upload request method=POST path=/37392026/11/12 00:34:25 INFO Completed upload id=34 objects=3 duration=452.292µs37402026/11/12 00:34:25 INFO Received complete upload request method=POST path=/37412026/11/12 00:34:25 INFO Completed upload id=3 objects=9 duration=1.077917ms37422026/11/12 00:34:26 INFO Received push request method=POST path=/37432026/11/12 00:34:26 INFO Received uploads request method=POST path=/37442026/11/12 00:34:26 INFO Received complete push request method=POST path=/37452026/11/12 00:34:26 INFO Received complete push request method=POST path=/37462026/11/12 00:34:26 INFO Completed push id=19 objects=3 duration=689.25µs37472026/11/12 00:34:26 INFO Completed push id=18 objects=6 duration=821.042µs37482026/11/12 00:34:26 INFO Received complete push request method=POST path=/37492026/11/12 00:34:26 INFO Completed push id=35 objects=11 duration=1.326041ms37502026/11/12 00:34:26 INFO Received complete upload request method=POST path=/37512026/11/12 00:34:26 INFO Completed upload id=21 objects=2 duration=3.193083ms37522026/11/12 00:34:26 INFO Aborted multipart uploads count=0 kept=037532026/11/12 00:34:26 INFO Aborted multipart uploads count=0 kept=037542026/11/12 00:34:26 INFO Aborted multipart uploads count=0 kept=037552026/11/12 00:34:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037562026/11/12 00:34:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037572026/11/12 00:34:26 WARN Force mode enabled - objects will be deleted immediately without grace period37582026/11/12 00:34:26 INFO Vacuumed table table=pending_closures37592026/11/12 00:34:26 INFO Vacuumed table table=pending_closures37602026/11/12 00:34:26 INFO Garbage collection completed failed-uploads-deleted=6 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037612026/11/12 00:34:26 INFO Received complete upload request method=POST path=/37622026/11/12 00:34:26 INFO Received complete upload request method=POST path=/37632026/11/12 00:34:26 INFO Aborted multipart uploads count=0 kept=037642026/11/12 00:34:26 INFO Vacuumed table table=pending_objects37652026/11/12 00:34:26 INFO Vacuumed table table=pending_objects37662026/11/12 00:34:26 INFO Vacuumed table table=multipart_uploads37672026/11/12 00:34:26 INFO Vacuumed table table=multipart_uploads37682026/11/12 00:34:26 INFO Completed upload id=14 objects=6 duration=6.324709ms37692026/11/12 00:34:26 INFO Completed upload id=6 objects=6 duration=6.345333ms37702026/11/12 00:34:26 INFO Vacuumed table table=closures37712026/11/12 00:34:26 WARN Force mode enabled - objects will be deleted immediately without grace period37722026/11/12 00:34:26 INFO Vacuumed table table=closures37732026/11/12 00:34:26 INFO Aborted multipart uploads count=0 kept=037742026/11/12 00:34:26 INFO Vacuumed table table=pending_closures37752026/11/12 00:34:26 INFO Received uploads request method=POST path=/37762026/11/12 00:34:26 INFO Vacuumed table table=objects37772026/11/12 00:34:26 INFO Vacuumed table table=objects37782026/11/12 00:34:26 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037792026/11/12 00:34:26 WARN Force mode enabled - objects will be deleted immediately without grace period37802026/11/12 00:34:26 INFO Received complete push request method=POST path=/37812026/11/12 00:34:26 INFO Vacuumed table table=pending_objects37822026/11/12 00:34:26 INFO Vacuumed table table=pending_closures37832026/11/12 00:34:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037842026/11/12 00:34:26 INFO Aborted multipart uploads count=0 kept=037852026/11/12 00:34:26 INFO Received push request method=POST path=/37862026/11/12 00:34:26 INFO Completed push id=5 objects=6 duration=824.542µs37872026/11/12 00:34:26 INFO Received complete upload request method=POST path=/37882026/11/12 00:34:26 INFO Vacuumed table table=multipart_uploads37892026/11/12 00:34:26 INFO Vacuumed table table=pending_objects37902026/11/12 00:34:26 INFO Vacuumed table table=pending_closures37912026/11/12 00:34:26 INFO Received push request method=POST path=/37922026/11/12 00:34:26 INFO Received uploads request method=POST path=/37932026/11/12 00:34:26 WARN Force mode enabled - objects will be deleted immediately without grace period37942026/11/12 00:34:26 INFO Received complete push request method=POST path=/37952026/11/12 00:34:26 INFO Completed upload id=20 objects=3 duration=2.746417ms37962026/11/12 00:34:26 INFO Vacuumed table table=closures37972026/11/12 00:34:26 INFO Vacuumed table table=multipart_uploads37982026/11/12 00:34:26 INFO Vacuumed table table=pending_objects37992026/11/12 00:34:26 INFO Vacuumed table table=multipart_uploads38002026/11/12 00:34:26 INFO Completed push id=26 objects=23 duration=1.105167ms38012026/11/12 00:34:26 INFO Vacuumed table table=closures38022026/11/12 00:34:26 INFO Vacuumed table table=objects38032026/11/12 00:34:26 INFO Aborted multipart uploads count=0 kept=038042026/11/12 00:34:26 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038052026/11/12 00:34:26 INFO Vacuumed table table=closures38062026/11/12 00:34:26 INFO Vacuumed table table=pending_closures38072026/11/12 00:34:26 WARN Force mode enabled - objects will be deleted immediately without grace period38082026/11/12 00:34:26 INFO Vacuumed table table=objects38092026/11/12 00:34:26 INFO Vacuumed table table=objects38102026/11/12 00:34:26 INFO Vacuumed table table=pending_objects38112026/11/12 00:34:26 INFO Vacuumed table table=multipart_uploads38122026/11/12 00:34:26 INFO Vacuumed table table=closures38132026/11/12 00:34:26 INFO Vacuumed table table=objects38142026/11/12 00:34:26 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."38152026/11/12 00:34:26 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."38162026/11/12 00:34:26 INFO Received push request method=POST path=/38172026/11/12 00:34:26 INFO Received push request method=POST path=/38182026/11/12 00:34:26 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."38192026/11/12 00:34:26 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."38202026/11/12 00:34:26 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."38212026/11/12 00:34:26 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."38222026/11/12 00:34:26 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."38232026/11/12 00:34:26 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."38242026/11/12 00:34:26 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."38252026/11/12 00:34:26 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."38262026/11/12 00:34:26 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."38272026/11/12 00:34:26 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."38282026/11/12 00:34:26 INFO Received complete upload request method=POST path=/38292026/11/12 00:34:26 INFO Completed upload id=25 objects=4 duration=842.834µs38302026/11/12 00:34:26 INFO Received complete upload request method=POST path=/38312026/11/12 00:34:26 INFO Completed upload id=7 objects=3 duration=441.958µs38322026/11/12 00:34:26 INFO Received push request method=POST path=/38332026/11/12 00:34:26 INFO Received push request method=POST path=/38342026/11/12 00:34:26 INFO Received uploads request method=POST path=/38352026/11/12 00:34:26 INFO Aborted multipart uploads count=0 kept=038362026/11/12 00:34:26 INFO Received push request method=POST path=/38372026/11/12 00:34:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038382026/11/12 00:34:26 INFO Vacuumed table table=pending_closures38392026/11/12 00:34:26 INFO Vacuumed table table=pending_objects38402026/11/12 00:34:26 INFO Received push request method=POST path=/38412026/11/12 00:34:26 INFO Vacuumed table table=multipart_uploads38422026/11/12 00:34:26 INFO Received complete upload request method=POST path=/38432026/11/12 00:34:26 INFO Vacuumed table table=closures38442026/11/12 00:34:26 INFO Completed upload id=24 objects=7 duration=1.123791ms38452026/11/12 00:34:26 INFO Received uploads request method=POST path=/38462026/11/12 00:34:26 INFO Vacuumed table table=objects38472026/11/12 00:34:26 INFO Received uploads request method=POST path=/38482026/11/12 00:34:26 INFO Received complete push request method=POST path=/38492026/11/12 00:34:26 INFO Aborted multipart uploads count=0 kept=038502026/11/12 00:34:26 INFO Completed push id=20 objects=8 duration=1.1705ms38512026/11/12 00:34:26 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038522026/11/12 00:34:26 INFO Received push request method=POST path=/38532026/11/12 00:34:26 INFO Vacuumed table table=pending_closures38542026/11/12 00:34:26 INFO Received push request method=POST path=/38552026/11/12 00:34:26 INFO Received push request method=POST path=/38562026/11/12 00:34:26 INFO Aborted multipart uploads count=0 kept=038572026/11/12 00:34:26 INFO Received uploads request method=POST path=/38582026/11/12 00:34:26 INFO Vacuumed table table=pending_objects38592026/11/12 00:34:26 INFO Received push request method=POST path=/38602026/11/12 00:34:26 WARN Force mode enabled - objects will be deleted immediately without grace period38612026/11/12 00:34:26 INFO Vacuumed table table=multipart_uploads38622026/11/12 00:34:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038632026/11/12 00:34:26 INFO Vacuumed table table=closures38642026/11/12 00:34:26 INFO Vacuumed table table=pending_closures38652026/11/12 00:34:26 INFO Vacuumed table table=objects38662026/11/12 00:34:26 INFO Aborted multipart uploads count=0 kept=038672026/11/12 00:34:26 INFO Received push request method=POST path=/38682026/11/12 00:34:26 INFO Received push request method=POST path=/38692026/11/12 00:34:26 INFO Vacuumed table table=pending_objects38702026/11/12 00:34:26 INFO Received complete push request method=POST path=/38712026/11/12 00:34:26 INFO Vacuumed table table=multipart_uploads38722026/11/12 00:34:26 INFO Received complete push request method=POST path=/38732026/11/12 00:34:26 INFO Completed push id=18 objects=23 duration=1.572166ms38742026/11/12 00:34:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=038752026/11/12 00:34:26 INFO Vacuumed table table=closures38762026/11/12 00:34:26 INFO Received uploads request method=POST path=/38772026/11/12 00:34:26 INFO Received uploads request method=POST path=/38782026/11/12 00:34:26 INFO Completed push id=27 objects=25 duration=3.122834ms38792026/11/12 00:34:26 INFO Vacuumed table table=objects38802026/11/12 00:34:26 INFO Vacuumed table table=pending_closures38812026/11/12 00:34:26 INFO Received complete push request method=POST path=/38822026/11/12 00:34:26 INFO Aborted multipart uploads count=0 kept=038832026/11/12 00:34:26 INFO Aborted multipart uploads count=0 kept=038842026/11/12 00:34:26 INFO Vacuumed table table=pending_objects38852026/11/12 00:34:26 INFO Completed push id=27 objects=11 duration=886.75µs38862026/11/12 00:34:26 INFO Aborted multipart uploads count=0 kept=038872026/11/12 00:34:26 WARN Force mode enabled - objects will be deleted immediately without grace period38882026/11/12 00:34:26 WARN Force mode enabled - objects will be deleted immediately without grace period38892026/11/12 00:34:26 INFO Received push request method=POST path=/38902026/11/12 00:34:26 INFO Vacuumed table table=multipart_uploads38912026/11/12 00:34:26 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038922026/11/12 00:34:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038932026/11/12 00:34:26 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038942026/11/12 00:34:26 INFO Received complete push request method=POST path=/38952026/11/12 00:34:26 INFO Vacuumed table table=pending_closures38962026/11/12 00:34:26 INFO Received complete upload request method=POST path=/38972026/11/12 00:34:26 INFO Received uploads request method=POST path=/38982026/11/12 00:34:26 INFO Vacuumed table table=pending_closures38992026/11/12 00:34:26 INFO Vacuumed table table=pending_closures39002026/11/12 00:34:26 INFO Completed push id=27 objects=15 duration=3.454ms39012026/11/12 00:34:26 INFO Completed upload id=29 objects=26 duration=4.068792ms39022026/11/12 00:34:26 INFO Vacuumed table table=closures39032026/11/12 00:34:26 INFO Vacuumed table table=pending_objects39042026/11/12 00:34:26 INFO Vacuumed table table=pending_objects39052026/11/12 00:34:26 INFO Vacuumed table table=pending_objects39062026/11/12 00:34:26 INFO Vacuumed table table=multipart_uploads39072026/11/12 00:34:26 INFO Vacuumed table table=objects39082026/11/12 00:34:26 INFO Vacuumed table table=multipart_uploads39092026/11/12 00:34:26 INFO Aborted multipart uploads count=0 kept=039102026/11/12 00:34:26 INFO Vacuumed table table=multipart_uploads39112026/11/12 00:34:26 INFO Vacuumed table table=closures39122026/11/12 00:34:26 INFO Vacuumed table table=closures39132026/11/12 00:34:26 INFO Vacuumed table table=closures39142026/11/12 00:34:26 INFO Vacuumed table table=objects39152026/11/12 00:34:26 INFO Vacuumed table table=objects39162026/11/12 00:34:26 WARN Force mode enabled - objects will be deleted immediately without grace period39172026/11/12 00:34:26 INFO Received push request method=POST path=/39182026/11/12 00:34:26 INFO Received complete push request method=POST path=/39192026/11/12 00:34:26 INFO Vacuumed table table=objects39202026/11/12 00:34:26 INFO Received complete push request method=POST path=/39212026/11/12 00:34:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039222026/11/12 00:34:26 INFO Completed push id=30 objects=8 duration=931.166µs39232026/11/12 00:34:26 INFO Completed push id=11 objects=9 duration=1.006875ms39242026/11/12 00:34:26 INFO Vacuumed table table=pending_closures39252026/11/12 00:34:26 INFO Received complete push request method=POST path=/39262026/11/12 00:34:26 INFO Received push request method=POST path=/39272026/11/12 00:34:26 INFO Received complete push request method=POST path=/39282026/11/12 00:34:26 INFO Received complete push request method=POST path=/39292026/11/12 00:34:26 INFO Vacuumed table table=pending_objects39302026/11/12 00:34:26 INFO Received complete upload request method=POST path=/39312026/11/12 00:34:26 INFO Completed push id=7 objects=20 duration=2.194458ms39322026/11/12 00:34:26 INFO Received complete push request method=POST path=/39332026/11/12 00:34:26 INFO Completed push id=24 objects=5 duration=2.40825ms39342026/11/12 00:34:26 INFO Completed push id=17 objects=22 duration=1.9085ms39352026/11/12 00:34:26 INFO Received complete upload request method=POST path=/39362026/11/12 00:34:26 INFO Vacuumed table table=multipart_uploads39372026/11/12 00:34:26 INFO Received push request method=POST path=/39382026/11/12 00:34:26 INFO Completed upload id=31 objects=5 duration=894.25µs39392026/11/12 00:34:26 INFO Completed upload id=10 objects=2 duration=6.638375ms39402026/11/12 00:34:26 INFO Vacuumed table table=closures39412026/11/12 00:34:26 INFO Received push request method=POST path=/39422026/11/12 00:34:26 INFO Completed push id=5 objects=9 duration=8.31325ms39432026/11/12 00:34:26 INFO Vacuumed table table=objects39442026/11/12 00:34:26 INFO Received push request method=POST path=/39452026/11/12 00:34:26 INFO Received complete push request method=POST path=/39462026/11/12 00:34:26 INFO Received complete upload request method=POST path=/39472026/11/12 00:34:26 INFO Received push request method=POST path=/39482026/11/12 00:34:26 INFO Received complete push request method=POST path=/39492026/11/12 00:34:26 INFO Completed push id=7 objects=24 duration=4.778708ms39502026/11/12 00:34:26 INFO Received push request method=POST path=/39512026/11/12 00:34:26 INFO Completed upload id=4 objects=14 duration=2.471584ms39522026/11/12 00:34:26 INFO Completed push id=32 objects=3 duration=2.237417ms39532026/11/12 00:34:26 INFO Received complete push request method=POST path=/39542026/11/12 00:34:26 INFO Received complete push request method=POST path=/39552026/11/12 00:34:26 INFO Completed push id=31 objects=14 duration=2.266625ms39562026/11/12 00:34:26 INFO Aborted multipart uploads count=0 kept=039572026/11/12 00:34:26 INFO Received uploads request method=POST path=/39582026/11/12 00:34:26 INFO Aborted multipart uploads count=0 kept=039592026/11/12 00:34:26 INFO Completed push id=23 objects=28 duration=7.697042ms39602026/11/12 00:34:26 WARN Force mode enabled - objects will be deleted immediately without grace period39612026/11/12 00:34:26 INFO Aborted multipart uploads count=0 kept=039622026/11/12 00:34:26 INFO Received push request method=POST path=/39632026/11/12 00:34:26 INFO Received uploads request method=POST path=/39642026/11/12 00:34:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039652026/11/12 00:34:26 INFO Aborted multipart uploads count=0 kept=039662026/11/12 00:34:26 INFO Received uploads request method=POST path=/39672026/11/12 00:34:26 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039682026/11/12 00:34:26 INFO Vacuumed table table=pending_closures39692026/11/12 00:34:26 INFO Vacuumed table table=pending_closures39702026/11/12 00:34:26 WARN Force mode enabled - objects will be deleted immediately without grace period39712026/11/12 00:34:26 INFO Vacuumed table table=pending_objects39722026/11/12 00:34:26 INFO Vacuumed table table=pending_objects39732026/11/12 00:34:26 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039742026/11/12 00:34:26 INFO Vacuumed table table=multipart_uploads39752026/11/12 00:34:26 INFO Vacuumed table table=multipart_uploads39762026/11/12 00:34:26 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=039772026/11/12 00:34:26 INFO Vacuumed table table=closures39782026/11/12 00:34:26 INFO Vacuumed table table=pending_closures39792026/11/12 00:34:26 INFO Received uploads request method=POST path=/39802026/11/12 00:34:26 INFO Vacuumed table table=closures39812026/11/12 00:34:26 INFO Vacuumed table table=pending_closures39822026/11/12 00:34:26 INFO Vacuumed table table=objects39832026/11/12 00:34:26 INFO Vacuumed table table=pending_objects39842026/11/12 00:34:26 INFO Aborted multipart uploads count=0 kept=039852026/11/12 00:34:26 INFO Vacuumed table table=objects39862026/11/12 00:34:26 INFO Vacuumed table table=multipart_uploads39872026/11/12 00:34:26 INFO Vacuumed table table=pending_objects39882026/11/12 00:34:26 WARN Force mode enabled - objects will be deleted immediately without grace period39892026/11/12 00:34:26 INFO Vacuumed table table=multipart_uploads39902026/11/12 00:34:27 INFO Vacuumed table table=closures39912026/11/12 00:34:27 INFO Received uploads request method=POST path=/39922026/11/12 00:34:27 INFO Vacuumed table table=closures39932026/11/12 00:34:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039942026/11/12 00:34:27 INFO Vacuumed table table=objects39952026/11/12 00:34:27 INFO Received uploads request method=POST path=/39962026/11/12 00:34:27 INFO Vacuumed table table=objects39972026/11/12 00:34:27 INFO Received push request method=POST path=/39982026/11/12 00:34:27 INFO Received complete upload request method=POST path=/39992026/11/12 00:34:27 INFO Aborted multipart uploads count=0 kept=040002026/11/12 00:34:27 INFO Completed upload id=27 objects=8 duration=1.115375ms40012026/11/12 00:34:27 INFO Vacuumed table table=pending_closures40022026/11/12 00:34:27 INFO Received uploads request method=POST path=/40032026/11/12 00:34:27 WARN Force mode enabled - objects will be deleted immediately without grace period40042026/11/12 00:34:27 INFO Received uploads request method=POST path=/40052026/11/12 00:34:27 INFO Vacuumed table table=pending_objects40062026/11/12 00:34:27 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040072026/11/12 00:34:27 INFO Vacuumed table table=multipart_uploads40082026/11/12 00:34:27 INFO Vacuumed table table=pending_closures40092026/11/12 00:34:27 INFO Received push request method=POST path=/40102026/11/12 00:34:27 INFO Vacuumed table table=closures40112026/11/12 00:34:27 INFO Vacuumed table table=objects40122026/11/12 00:34:27 INFO Vacuumed table table=pending_objects40132026/11/12 00:34:27 INFO Received push request method=POST path=/40142026/11/12 00:34:27 INFO Vacuumed table table=multipart_uploads40152026/11/12 00:34:27 INFO Received push request method=POST path=/40162026/11/12 00:34:27 INFO Vacuumed table table=closures40172026/11/12 00:34:27 INFO Received push request method=POST path=/40182026/11/12 00:34:27 INFO Vacuumed table table=objects40192026/11/12 00:34:27 INFO Received complete push request method=POST path=/40202026/11/12 00:34:27 INFO Completed push id=5 objects=3 duration=3.512208ms40212026/11/12 00:34:27 INFO Received push request method=POST path=/40222026/11/12 00:34:27 INFO Received uploads request method=POST path=/40232026/11/12 00:34:27 INFO Aborted multipart uploads count=0 kept=040242026/11/12 00:34:27 INFO Aborted multipart uploads count=0 kept=040252026/11/12 00:34:27 WARN Force mode enabled - objects will be deleted immediately without grace period40262026/11/12 00:34:27 INFO Received uploads request method=POST path=/40272026/11/12 00:34:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040282026/11/12 00:34:27 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040292026/11/12 00:34:27 INFO Vacuumed table table=pending_closures40302026/11/12 00:34:27 INFO Vacuumed table table=pending_closures40312026/11/12 00:34:27 INFO Received complete push request method=POST path=/40322026/11/12 00:34:27 INFO Received complete push request method=POST path=/40332026/11/12 00:34:27 INFO Aborted multipart uploads count=0 kept=040342026/11/12 00:34:27 INFO Vacuumed table table=pending_objects40352026/11/12 00:34:27 INFO Completed push id=13 objects=2 duration=1.117458ms40362026/11/12 00:34:27 INFO Vacuumed table table=pending_objects40372026/11/12 00:34:27 INFO Completed push id=28 objects=19 duration=1.171167ms40382026/11/12 00:34:27 INFO Received complete push request method=POST path=/40392026/11/12 00:34:27 INFO Vacuumed table table=multipart_uploads40402026/11/12 00:34:27 INFO Received push request method=POST path=/40412026/11/12 00:34:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040422026/11/12 00:34:27 INFO Completed push id=21 objects=26 duration=1.805042ms40432026/11/12 00:34:27 INFO Vacuumed table table=multipart_uploads40442026/11/12 00:34:27 INFO Vacuumed table table=closures40452026/11/12 00:34:27 INFO Vacuumed table table=closures40462026/11/12 00:34:27 INFO Vacuumed table table=pending_closures40472026/11/12 00:34:27 INFO Vacuumed table table=objects40482026/11/12 00:34:27 INFO Received push request method=POST path=/40492026/11/12 00:34:27 INFO Vacuumed table table=objects40502026/11/12 00:34:27 INFO Received complete push request method=POST path=/40512026/11/12 00:34:27 INFO Received complete upload request method=POST path=/40522026/11/12 00:34:27 INFO Completed upload id=27 objects=13 duration=1.413916ms40532026/11/12 00:34:27 INFO Completed push id=8 objects=6 duration=1.568167ms40542026/11/12 00:34:27 INFO Vacuumed table table=pending_objects40552026/11/12 00:34:27 INFO Vacuumed table table=multipart_uploads40562026/11/12 00:34:27 INFO Vacuumed table table=closures40572026/11/12 00:34:27 INFO Received complete upload request method=POST path=/40582026/11/12 00:34:27 INFO Vacuumed table table=objects40592026/11/12 00:34:27 INFO Aborted multipart uploads count=0 kept=040602026/11/12 00:34:27 INFO Aborted multipart uploads count=0 kept=040612026/11/12 00:34:27 INFO Aborted multipart uploads count=0 kept=040622026/11/12 00:34:27 INFO Completed upload id=23 objects=15 duration=1.689292ms40632026/11/12 00:34:27 WARN Force mode enabled - objects will be deleted immediately without grace period40642026/11/12 00:34:27 INFO Received push request method=POST path=/40652026/11/12 00:34:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040662026/11/12 00:34:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040672026/11/12 00:34:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040682026/11/12 00:34:27 INFO Vacuumed table table=pending_closures40692026/11/12 00:34:27 INFO Vacuumed table table=pending_closures40702026/11/12 00:34:27 INFO Received complete upload request method=POST path=/40712026/11/12 00:34:27 INFO Vacuumed table table=pending_objects40722026/11/12 00:34:27 INFO Vacuumed table table=pending_objects40732026/11/12 00:34:27 INFO Received uploads request method=POST path=/40742026/11/12 00:34:27 INFO Vacuumed table table=pending_closures40752026/11/12 00:34:27 INFO Completed upload id=28 objects=17 duration=1.420375ms40762026/11/12 00:34:27 INFO Vacuumed table table=multipart_uploads40772026/11/12 00:34:27 INFO Vacuumed table table=multipart_uploads40782026/11/12 00:34:27 INFO Vacuumed table table=pending_objects40792026/11/12 00:34:27 INFO Vacuumed table table=closures40802026/11/12 00:34:27 INFO Received complete upload request method=POST path=/40812026/11/12 00:34:27 INFO Vacuumed table table=closures40822026/11/12 00:34:27 INFO Vacuumed table table=multipart_uploads40832026/11/12 00:34:27 INFO Vacuumed table table=objects40842026/11/12 00:34:27 INFO Completed upload id=23 objects=16 duration=2.394792ms40852026/11/12 00:34:27 INFO Received push request method=POST path=/40862026/11/12 00:34:27 INFO Received complete push request method=POST path=/40872026/11/12 00:34:27 INFO Vacuumed table table=closures40882026/11/12 00:34:27 INFO Vacuumed table table=objects40892026/11/12 00:34:27 INFO Received complete push request method=POST path=/40902026/11/12 00:34:27 INFO Completed push id=29 objects=2 duration=764.334µs40912026/11/12 00:34:27 INFO Vacuumed table table=objects40922026/11/12 00:34:27 INFO Received complete upload request method=POST path=/40932026/11/12 00:34:27 INFO Received uploads request method=POST path=/40942026/11/12 00:34:27 INFO Aborted multipart uploads count=0 kept=040952026/11/12 00:34:27 INFO Received push request method=POST path=/40962026/11/12 00:34:27 INFO Completed push id=5 objects=8 duration=3.518333ms40972026/11/12 00:34:27 INFO Completed upload id=25 objects=16 duration=3.048791ms40982026/11/12 00:34:27 INFO Received complete push request method=POST path=/40992026/11/12 00:34:27 WARN Force mode enabled - objects will be deleted immediately without grace period41002026/11/12 00:34:27 INFO Received complete upload request method=POST path=/41012026/11/12 00:34:27 INFO Received push request method=POST path=/41022026/11/12 00:34:27 INFO Completed push id=31 objects=6 duration=1.694167ms41032026/11/12 00:34:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041042026/11/12 00:34:27 INFO Vacuumed table table=pending_closures41052026/11/12 00:34:27 INFO Completed upload id=19 objects=6 duration=914.583µs41062026/11/12 00:34:27 INFO Received push request method=POST path=/41072026/11/12 00:34:27 INFO Vacuumed table table=pending_objects41082026/11/12 00:34:27 INFO Received complete push request method=POST path=/41092026/11/12 00:34:27 INFO Vacuumed table table=multipart_uploads41102026/11/12 00:34:27 INFO Received complete push request method=POST path=/41112026/11/12 00:34:27 INFO Aborted multipart uploads count=0 kept=041122026/11/12 00:34:27 INFO Completed push id=29 objects=10 duration=1.321042ms41132026/11/12 00:34:27 INFO Received push request method=POST path=/41142026/11/12 00:34:27 INFO Received uploads request method=POST path=/41152026/11/12 00:34:27 INFO Vacuumed table table=closures41162026/11/12 00:34:27 INFO Completed push id=32 objects=9 duration=896.458µs41172026/11/12 00:34:27 WARN Force mode enabled - objects will be deleted immediately without grace period41182026/11/12 00:34:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041192026/11/12 00:34:27 INFO Vacuumed table table=objects41202026/11/12 00:34:27 INFO Vacuumed table table=pending_closures41212026/11/12 00:34:27 INFO Received complete push request method=POST path=/41222026/11/12 00:34:27 INFO Vacuumed table table=pending_objects41232026/11/12 00:34:27 INFO Completed push id=11 objects=8 duration=6.235375ms41242026/11/12 00:34:27 INFO Received uploads request method=POST path=/41252026/11/12 00:34:27 INFO Vacuumed table table=multipart_uploads41262026/11/12 00:34:27 INFO Vacuumed table table=closures41272026/11/12 00:34:27 INFO Aborted multipart uploads count=0 kept=041282026/11/12 00:34:27 INFO Vacuumed table table=objects41292026/11/12 00:34:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041302026/11/12 00:34:27 INFO Vacuumed table table=pending_closures41312026/11/12 00:34:27 INFO Vacuumed table table=pending_objects41322026/11/12 00:34:27 INFO Vacuumed table table=multipart_uploads41332026/11/12 00:34:27 INFO Aborted multipart uploads count=0 kept=041342026/11/12 00:34:27 INFO Vacuumed table table=closures41352026/11/12 00:34:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=8 objects-deleted-after-grace-period=8 objects-failed-to-delete=041362026/11/12 00:34:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041372026/11/12 00:34:27 INFO Received uploads request method=POST path=/41382026/11/12 00:34:27 INFO Received uploads request method=POST path=/41392026/11/12 00:34:27 INFO Vacuumed table table=objects41402026/11/12 00:34:27 INFO Vacuumed table table=pending_closures41412026/11/12 00:34:27 INFO Received complete upload request method=POST path=/41422026/11/12 00:34:27 INFO Received push request method=POST path=/41432026/11/12 00:34:27 INFO Vacuumed table table=pending_objects41442026/11/12 00:34:27 INFO Completed upload id=22 objects=7 duration=600.084µs41452026/11/12 00:34:27 INFO Received complete upload request method=POST path=/41462026/11/12 00:34:27 INFO Received complete upload request method=POST path=/41472026/11/12 00:34:27 INFO Completed upload id=10 objects=17 duration=1.564791ms41482026/11/12 00:34:27 INFO Vacuumed table table=multipart_uploads41492026/11/12 00:34:27 INFO Vacuumed table table=pending_closures41502026/11/12 00:34:27 INFO Completed upload id=29 objects=4 duration=2.546ms41512026/11/12 00:34:27 INFO Received uploads request method=POST path=/41522026/11/12 00:34:27 INFO Vacuumed table table=closures41532026/11/12 00:34:27 INFO Received complete upload request method=POST path=/41542026/11/12 00:34:27 INFO Received complete push request method=POST path=/41552026/11/12 00:34:27 INFO Vacuumed table table=pending_objects41562026/11/12 00:34:27 INFO Completed push id=7 objects=7 duration=2.86525ms41572026/11/12 00:34:27 INFO Received complete push request method=POST path=/41582026/11/12 00:34:27 INFO Completed upload id=30 objects=6 duration=2.978875ms41592026/11/12 00:34:27 INFO Vacuumed table table=multipart_uploads41602026/11/12 00:34:27 INFO Vacuumed table table=objects41612026/11/12 00:34:27 INFO Received push request method=POST path=/41622026/11/12 00:34:27 INFO Vacuumed table table=closures41632026/11/12 00:34:27 INFO Vacuumed table table=objects41642026/11/12 00:34:27 INFO Completed push id=11 objects=11 duration=3.238875ms41652026/11/12 00:34:27 INFO Received complete upload request method=POST path=/41662026/11/12 00:34:27 INFO Completed upload id=33 objects=9 duration=2.962583ms41672026/11/12 00:34:27 INFO Received uploads request method=POST path=/41682026/11/12 00:34:27 INFO Received uploads request method=POST path=/41692026/11/12 00:34:27 INFO Received complete upload request method=POST path=/41702026/11/12 00:34:27 INFO Completed upload id=31 objects=5 duration=776.917µs41712026/11/12 00:34:27 INFO Received complete push request method=POST path=/41722026/11/12 00:34:27 INFO Completed push id=12 objects=6 duration=1.671667ms41732026/11/12 00:34:27 INFO Received complete upload request method=POST path=/41742026/11/12 00:34:27 INFO Completed upload id=7 objects=10 duration=4.951917ms41752026/11/12 00:34:27 INFO Received push request method=POST path=/41762026/11/12 00:34:27 INFO Aborted multipart uploads count=0 kept=041772026/11/12 00:34:27 WARN Force mode enabled - objects will be deleted immediately without grace period41782026/11/12 00:34:27 INFO Received push request method=POST path=/41792026/11/12 00:34:27 INFO Received uploads request method=POST path=/41802026/11/12 00:34:27 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041812026/11/12 00:34:27 INFO Received complete upload request method=POST path=/41822026/11/12 00:34:27 INFO Completed upload id=34 objects=3 duration=412.083µs41832026/11/12 00:34:27 INFO Vacuumed table table=pending_closures41842026/11/12 00:34:27 INFO Vacuumed table table=pending_objects41852026/11/12 00:34:27 INFO Vacuumed table table=multipart_uploads41862026/11/12 00:34:27 INFO Vacuumed table table=closures41872026/11/12 00:34:27 INFO Vacuumed table table=objects41882026/11/12 00:34:27 INFO Received push request method=POST path=/41892026/11/12 00:34:27 INFO Received complete upload request method=POST path=/41902026/11/12 00:34:27 INFO Completed upload id=8 objects=3 duration=1.405417ms41912026/11/12 00:34:27 INFO Aborted multipart uploads count=0 kept=041922026/11/12 00:34:27 WARN Force mode enabled - objects will be deleted immediately without grace period41932026/11/12 00:34:27 INFO Received uploads request method=POST path=/41942026/11/12 00:34:27 INFO Received complete push request method=POST path=/41952026/11/12 00:34:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041962026/11/12 00:34:27 INFO Completed push id=24 objects=9 duration=796.125µs41972026/11/12 00:34:27 INFO Received complete push request method=POST path=/41982026/11/12 00:34:27 INFO Vacuumed table table=pending_closures41992026/11/12 00:34:27 INFO Completed push id=14 objects=6 duration=1.60575ms42002026/11/12 00:34:27 INFO Vacuumed table table=pending_objects42012026/11/12 00:34:27 INFO Received uploads request method=POST path=/42022026/11/12 00:34:27 INFO Vacuumed table table=multipart_uploads42032026/11/12 00:34:27 INFO Aborted multipart uploads count=0 kept=042042026/11/12 00:34:27 INFO Received complete upload request method=POST path=/42052026/11/12 00:34:27 INFO Received push request method=POST path=/42062026/11/12 00:34:27 INFO Vacuumed table table=closures42072026/11/12 00:34:27 INFO Completed upload id=22 objects=5 duration=2.77475ms42082026/11/12 00:34:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042092026/11/12 00:34:27 INFO Vacuumed table table=objects42102026/11/12 00:34:27 INFO Received push request method=POST path=/42112026/11/12 00:34:28 INFO Vacuumed table table=pending_closures42122026/11/12 00:34:28 INFO Received push request method=POST path=/42132026/11/12 00:34:28 INFO Vacuumed table table=pending_objects42142026/11/12 00:34:28 INFO Vacuumed table table=multipart_uploads42152026/11/12 00:34:28 INFO Vacuumed table table=closures42162026/11/12 00:34:28 INFO Received complete push request method=POST path=/42172026/11/12 00:34:28 INFO Received complete push request method=POST path=/42182026/11/12 00:34:28 INFO Vacuumed table table=objects42192026/11/12 00:34:28 INFO Completed push id=11 objects=6 duration=1.018042ms42202026/11/12 00:34:28 INFO Completed push id=28 objects=3 duration=1.341417ms42212026/11/12 00:34:28 INFO Received push request method=POST path=/42222026/11/12 00:34:28 INFO Received complete push request method=POST path=/42232026/11/12 00:34:28 INFO Completed push id=13 objects=6 duration=3.64725ms42242026/11/12 00:34:28 INFO Received push request method=POST path=/42252026/11/12 00:34:28 INFO Received complete push request method=POST path=/42262026/11/12 00:34:28 INFO Aborted multipart uploads count=0 kept=042272026/11/12 00:34:28 INFO Completed push id=29 objects=15 duration=506.292µs42282026/11/12 00:34:28 INFO Received push request method=POST path=/42292026/11/12 00:34:28 INFO Received complete push request method=POST path=/42302026/11/12 00:34:28 INFO Received uploads request method=POST path=/42312026/11/12 00:34:28 INFO Completed push id=6 objects=6 duration=706.375µs42322026/11/12 00:34:28 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=1 objects-deleted-after-grace-period=0 objects-failed-to-delete=042332026/11/12 00:34:28 INFO Vacuumed table table=pending_closures42342026/11/12 00:34:28 INFO Vacuumed table table=pending_objects42352026/11/12 00:34:28 INFO Vacuumed table table=multipart_uploads42362026/11/12 00:34:28 INFO Vacuumed table table=closures42372026/11/12 00:34:28 INFO Vacuumed table table=objects42382026/11/12 00:34:28 INFO Received complete push request method=POST path=/42392026/11/12 00:34:28 INFO Completed push id=20 objects=13 duration=2.097375ms42402026/11/12 00:34:28 INFO Received push request method=POST path=/42412026/11/12 00:34:28 INFO Received complete push request method=POST path=/42422026/11/12 00:34:28 INFO Completed push id=4 objects=6 duration=1.322541ms42432026/11/12 00:34:28 INFO Received push request method=POST path=/42442026/11/12 00:34:28 INFO Received complete push request method=POST path=/42452026/11/12 00:34:28 INFO Completed push id=30 objects=6 duration=924.083µs42462026/11/12 00:34:28 INFO Received complete push request method=POST path=/42472026/11/12 00:34:28 INFO Completed push id=8 objects=6 duration=2.633959ms42482026/11/12 00:34:28 INFO Received complete push request method=POST path=/42492026/11/12 00:34:28 INFO Completed push id=22 objects=4 duration=1.613042ms42502026/11/12 00:34:28 INFO Received complete upload request method=POST path=/42512026/11/12 00:34:28 INFO Completed upload id=31 objects=3 duration=1.5365ms42522026/11/12 00:34:28 INFO Received push request method=POST path=/42532026/11/12 00:34:28 INFO Received push request method=POST path=/42542026/11/12 00:34:28 INFO Aborted multipart uploads count=0 kept=042552026/11/12 00:34:28 INFO Received uploads request method=POST path=/42562026/11/12 00:34:28 INFO Received complete push request method=POST path=/42572026/11/12 00:34:28 WARN Force mode enabled - objects will be deleted immediately without grace period42582026/11/12 00:34:28 INFO Completed push id=21 objects=3 duration=651.041µs42592026/11/12 00:34:28 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042602026/11/12 00:34:28 INFO Received complete push request method=POST path=/42612026/11/12 00:34:28 INFO Completed push id=13 objects=22 duration=4.650459ms42622026/11/12 00:34:28 INFO Received complete upload request method=POST path=/42632026/11/12 00:34:28 INFO Completed upload id=23 objects=7 duration=814.667µs42642026/11/12 00:34:28 INFO Vacuumed table table=pending_closures42652026/11/12 00:34:28 INFO Vacuumed table table=pending_objects42662026/11/12 00:34:28 INFO Received uploads request method=POST path=/42672026/11/12 00:34:28 INFO Vacuumed table table=multipart_uploads42682026/11/12 00:34:28 INFO Vacuumed table table=closures42692026/11/12 00:34:28 INFO Received complete push request method=POST path=/42702026/11/12 00:34:28 INFO Received uploads request method=POST path=/42712026/11/12 00:34:28 INFO Received complete push request method=POST path=/42722026/11/12 00:34:28 INFO Vacuumed table table=objects42732026/11/12 00:34:28 INFO Completed push id=24 objects=5 duration=3.05875ms42742026/11/12 00:34:28 INFO Completed push id=21 objects=6 duration=2.406ms42752026/11/12 00:34:28 INFO Received complete push request method=POST path=/42762026/11/12 00:34:28 INFO Received complete upload request method=POST path=/42772026/11/12 00:34:28 INFO Received complete push request method=POST path=/42782026/11/12 00:34:28 INFO Completed push id=10 objects=15 duration=2.056166ms42792026/11/12 00:34:28 INFO Completed upload id=9 objects=2 duration=4.83975ms42802026/11/12 00:34:28 INFO Completed push id=12 objects=2 duration=1.6845ms42812026/11/12 00:34:28 INFO Received complete push request method=POST path=/42822026/11/12 00:34:28 INFO Completed push id=7 objects=15 duration=3.27925ms42832026/11/12 00:34:28 INFO Received complete upload request method=POST path=/42842026/11/12 00:34:28 INFO Received complete upload request method=POST path=/42852026/11/12 00:34:28 INFO Completed upload id=25 objects=2 duration=3.257375ms42862026/11/12 00:34:28 INFO Completed upload id=22 objects=7 duration=3.875333ms42872026/11/12 00:34:28 INFO Received complete push request method=POST path=/42882026/11/12 00:34:28 INFO Completed push id=8 objects=3 duration=5.254917ms42892026/11/12 00:34:28 INFO Received uploads request method=POST path=/42902026/11/12 00:34:28 INFO Aborted multipart uploads count=0 kept=042912026/11/12 00:34:28 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=3 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042922026/11/12 00:34:28 INFO Vacuumed table table=pending_closures42932026/11/12 00:34:28 INFO Received complete upload request method=POST path=/42942026/11/12 00:34:28 INFO Completed upload id=11 objects=7 duration=1.215167ms42952026/11/12 00:34:28 INFO Vacuumed table table=pending_objects42962026/11/12 00:34:28 INFO Vacuumed table table=multipart_uploads42972026/11/12 00:34:28 INFO Received push request method=POST path=/42982026/11/12 00:34:28 INFO Received complete upload request method=POST path=/42992026/11/12 00:34:28 INFO Vacuumed table table=closures43002026/11/12 00:34:28 INFO Completed upload id=26 objects=20 duration=633.625µs43012026/11/12 00:34:28 INFO Received complete push request method=POST path=/43022026/11/12 00:34:28 INFO Completed push id=24 objects=4 duration=1.031375ms43032026/11/12 00:34:28 INFO Vacuumed table table=objects43042026/11/12 00:34:28 INFO Received complete push request method=POST path=/43052026/11/12 00:34:28 INFO Received push request method=POST path=/43062026/11/12 00:34:28 INFO Completed push id=17 objects=6 duration=2.123083ms43072026/11/12 00:34:28 INFO Received uploads request method=POST path=/43082026/11/12 00:34:28 INFO Received uploads request method=POST path=/43092026/11/12 00:34:28 INFO Received complete push request method=POST path=/43102026/11/12 00:34:28 INFO Received uploads request method=POST path=/43112026/11/12 00:34:28 INFO Received push request method=POST path=/43122026/11/12 00:34:28 INFO Completed push id=25 objects=9 duration=1.715167ms43132026/11/12 00:34:28 INFO Received uploads request method=POST path=/43142026/11/12 00:34:28 INFO Received complete upload request method=POST path=/43152026/11/12 00:34:28 INFO Completed upload id=13 objects=10 duration=2.779541ms43162026/11/12 00:34:28 INFO Received push request method=POST path=/43172026/11/12 00:34:28 INFO Received complete upload request method=POST path=/43182026/11/12 00:34:28 INFO Completed upload id=26 objects=3 duration=2.820417ms43192026/11/12 00:34:28 INFO Aborted multipart uploads count=0 kept=043202026/11/12 00:34:28 INFO Received push request method=POST path=/43212026/11/12 00:34:28 INFO Aborted multipart uploads count=0 kept=043222026/11/12 00:34:28 INFO Received complete push request method=POST path=/43232026/11/12 00:34:28 WARN Force mode enabled - objects will be deleted immediately without grace period43242026/11/12 00:34:28 INFO Completed push id=28 objects=7 duration=1.241333ms43252026/11/12 00:34:28 INFO Aborted multipart uploads count=0 kept=043262026/11/12 00:34:28 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043272026/11/12 00:34:28 INFO Vacuumed table table=pending_closures43282026/11/12 00:34:28 WARN Force mode enabled - objects will be deleted immediately without grace period43292026/11/12 00:34:28 INFO Vacuumed table table=pending_objects43302026/11/12 00:34:28 INFO Vacuumed table table=multipart_uploads43312026/11/12 00:34:28 INFO Vacuumed table table=closures43322026/11/12 00:34:28 INFO Received complete push request method=POST path=/43332026/11/12 00:34:28 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=043342026/11/12 00:34:28 INFO Vacuumed table table=objects43352026/11/12 00:34:28 INFO Completed push id=30 objects=4 duration=1.963958ms43362026/11/12 00:34:28 INFO Vacuumed table table=pending_closures43372026/11/12 00:34:28 INFO Vacuumed table table=pending_objects43382026/11/12 00:34:28 INFO Received push request method=POST path=/43392026/11/12 00:34:28 INFO Received uploads request method=POST path=/43402026/11/12 00:34:28 INFO Vacuumed table table=multipart_uploads43412026/11/12 00:34:28 INFO Vacuumed table table=closures43422026/11/12 00:34:28 INFO Vacuumed table table=objects43432026/11/12 00:34:28 INFO Received uploads request method=POST path=/43442026/11/12 00:34:28 INFO Received complete upload request method=POST path=/43452026/11/12 00:34:28 INFO Received complete push request method=POST path=/43462026/11/12 00:34:28 INFO Received push request method=POST path=/43472026/11/12 00:34:28 INFO Received complete upload request method=POST path=/43482026/11/12 00:34:28 INFO Completed upload id=27 objects=7 duration=798.875µs43492026/11/12 00:34:28 INFO Received push request method=POST path=/43502026/11/12 00:34:28 INFO Completed push id=9 objects=24 duration=2.120292ms43512026/11/12 00:34:28 INFO Completed upload id=30 objects=6 duration=1.565125ms43522026/11/12 00:34:28 INFO Aborted multipart uploads count=0 kept=043532026/11/12 00:34:28 INFO Received push request method=POST path=/43542026/11/12 00:34:28 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043552026/11/12 00:34:28 INFO Vacuumed table table=pending_closures43562026/11/12 00:34:28 INFO Received complete upload request method=POST path=/43572026/11/12 00:34:28 INFO Received uploads request method=POST path=/43582026/11/12 00:34:28 INFO Vacuumed table table=pending_objects43592026/11/12 00:34:28 INFO Completed upload id=28 objects=3 duration=774.875µs43602026/11/12 00:34:28 INFO Received complete push request method=POST path=/43612026/11/12 00:34:28 INFO Received complete push request method=POST path=/43622026/11/12 00:34:28 INFO Vacuumed table table=multipart_uploads43632026/11/12 00:34:28 INFO Completed push id=31 objects=5 duration=541.875µs43642026/11/12 00:34:28 INFO Completed push id=32 objects=7 duration=716.875µs43652026/11/12 00:34:28 INFO Received uploads request method=POST path=/43662026/11/12 00:34:28 INFO Vacuumed table table=closures43672026/11/12 00:34:28 INFO Received complete push request method=POST path=/43682026/11/12 00:34:28 INFO Received uploads request method=POST path=/43692026/11/12 00:34:28 INFO Vacuumed table table=objects43702026/11/12 00:34:28 INFO Completed push id=4 objects=7 duration=1.900792ms43712026/11/12 00:34:28 INFO Received complete push request method=POST path=/43722026/11/12 00:34:28 INFO Aborted multipart uploads count=0 kept=043732026/11/12 00:34:28 INFO Completed push id=26 objects=4 duration=1.151333ms43742026/11/12 00:34:28 WARN Force mode enabled - objects will be deleted immediately without grace period43752026/11/12 00:34:28 INFO Received uploads request method=POST path=/43762026/11/12 00:34:28 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."43772026/11/12 00:34:28 INFO Received complete upload request method=POST path=/43782026/11/12 00:34:28 INFO Received complete upload request method=POST path=/43792026/11/12 00:34:28 INFO Received complete push request method=POST path=/43802026/11/12 00:34:28 INFO Completed upload id=10 objects=10 duration=1.203209ms43812026/11/12 00:34:28 INFO Received complete push request method=POST path=/43822026/11/12 00:34:28 INFO Completed upload id=28 objects=13 duration=1.82825ms43832026/11/12 00:34:28 INFO Received uploads request method=POST path=/43842026/11/12 00:34:28 INFO Completed push id=22 objects=3 duration=3.150542ms43852026/11/12 00:34:28 INFO Completed push id=29 objects=23 duration=784.833µs43862026/11/12 00:34:28 INFO Aborted multipart uploads count=0 kept=043872026/11/12 00:34:28 INFO Received complete push request method=POST path=/43882026/11/12 00:34:28 WARN Force mode enabled - objects will be deleted immediately without grace period43892026/11/12 00:34:28 INFO Completed push id=29 objects=3 duration=2.068167ms43902026/11/12 00:34:28 INFO Received complete push request method=POST path=/43912026/11/12 00:34:28 INFO Completed push id=30 objects=6 duration=935.5µs43922026/11/12 00:34:29 INFO Received complete push request method=POST path=/43932026/11/12 00:34:29 INFO Completed push id=32 objects=26 duration=1.249208ms43942026/11/12 00:34:29 INFO Received uploads request method=POST path=/43952026/11/12 00:34:29 INFO Received complete upload request method=POST path=/43962026/11/12 00:34:29 INFO Completed upload id=31 objects=2 duration=744.5µs43972026/11/12 00:34:29 INFO Received complete upload request method=POST path=/43982026/11/12 00:34:29 INFO Completed upload id=14 objects=8 duration=3.432583ms43992026/11/12 00:34:29 INFO Received complete upload request method=POST path=/44002026/11/12 00:34:29 INFO Completed upload id=33 objects=2 duration=601.458µs44012026/11/12 00:34:29 INFO Received complete push request method=POST path=/44022026/11/12 00:34:29 INFO Received push request method=POST path=/44032026/11/12 00:34:29 INFO Completed push id=33 objects=6 duration=731.834µs44042026/11/12 00:34:29 INFO Received complete upload request method=POST path=/44052026/11/12 00:34:29 INFO Completed upload id=34 objects=5 duration=718.667µs44062026/11/12 00:34:29 INFO Received push request method=POST path=/44072026/11/12 00:34:29 INFO Received uploads request method=POST path=/44082026/11/12 00:34:29 INFO Aborted multipart uploads count=0 kept=044092026/11/12 00:34:29 WARN Force mode enabled - objects will be deleted immediately without grace period44102026/11/12 00:34:29 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044112026/11/12 00:34:29 INFO Received complete upload request method=POST path=/44122026/11/12 00:34:29 INFO Completed upload id=35 objects=11 duration=742.875µs44132026/11/12 00:34:29 INFO Received complete push request method=POST path=/44142026/11/12 00:34:29 INFO Vacuumed table table=pending_closures44152026/11/12 00:34:29 INFO Aborted multipart uploads count=0 kept=044162026/11/12 00:34:29 INFO Received uploads request method=POST path=/44172026/11/12 00:34:29 INFO Received complete upload request method=POST path=/44182026/11/12 00:34:29 INFO Completed push id=33 objects=3 duration=879.792µs44192026/11/12 00:34:29 WARN Force mode enabled - objects will be deleted immediately without grace period44202026/11/12 00:34:29 INFO Completed upload id=36 objects=3 duration=608.375µs44212026/11/12 00:34:29 INFO Vacuumed table table=pending_objects44222026/11/12 00:34:29 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=044232026/11/12 00:34:29 INFO Received complete push request method=POST path=/44242026/11/12 00:34:29 INFO Received complete push request method=POST path=/44252026/11/12 00:34:29 INFO Vacuumed table table=multipart_uploads44262026/11/12 00:34:29 INFO Completed push id=34 objects=6 duration=6.741125ms44272026/11/12 00:34:29 INFO Completed push id=14 objects=3 duration=6.843292ms44282026/11/12 00:34:29 INFO Vacuumed table table=closures44292026/11/12 00:34:29 INFO Aborted multipart uploads count=0 kept=044302026/11/12 00:34:29 INFO Vacuumed table table=objects44312026/11/12 00:34:29 WARN Force mode enabled - objects will be deleted immediately without grace period44322026/11/12 00:34:29 INFO Vacuumed table table=pending_closures44332026/11/12 00:34:29 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044342026/11/12 00:34:29 INFO Received complete push request method=POST path=/44352026/11/12 00:34:29 INFO Received push request method=POST path=/44362026/11/12 00:34:29 INFO Completed push id=35 objects=8 duration=773.875µs44372026/11/12 00:34:29 INFO Vacuumed table table=pending_objects44382026/11/12 00:34:29 INFO Vacuumed table table=multipart_uploads44392026/11/12 00:34:29 INFO Vacuumed table table=pending_closures44402026/11/12 00:34:29 INFO Vacuumed table table=closures44412026/11/12 00:34:29 INFO Received complete upload request method=POST path=/44422026/11/12 00:34:29 INFO Vacuumed table table=objects44432026/11/12 00:34:29 INFO Vacuumed table table=pending_objects44442026/11/12 00:34:29 INFO Completed upload id=36 objects=9 duration=3.084208ms44452026/11/12 00:34:29 INFO Received push request method=POST path=/44462026/11/12 00:34:29 INFO Vacuumed table table=multipart_uploads44472026/11/12 00:34:29 INFO Vacuumed table table=closures44482026/11/12 00:34:29 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=11 objects-failed-to-delete=044492026/11/12 00:34:29 INFO Vacuumed table table=objects44502026/11/12 00:34:29 INFO Vacuumed table table=pending_closures44512026/11/12 00:34:29 INFO Received complete push request method=POST path=/44522026/11/12 00:34:29 INFO Completed push id=19 objects=14 duration=1.183584ms44532026/11/12 00:34:29 INFO Vacuumed table table=pending_objects44542026/11/12 00:34:29 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."44552026/11/12 00:34:29 INFO Vacuumed table table=multipart_uploads44562026/11/12 00:34:29 INFO Aborted multipart uploads count=0 kept=044572026/11/12 00:34:29 INFO Aborted multipart uploads count=0 kept=044582026/11/12 00:34:29 INFO Vacuumed table table=closures44592026/11/12 00:34:29 WARN Force mode enabled - objects will be deleted immediately without grace period44602026/11/12 00:34:29 INFO Vacuumed table table=objects44612026/11/12 00:34:29 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044622026/11/12 00:34:29 WARN Force mode enabled - objects will be deleted immediately without grace period44632026/11/12 00:34:29 INFO Aborted multipart uploads count=0 kept=044642026/11/12 00:34:29 INFO Vacuumed table table=pending_closures44652026/11/12 00:34:29 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044662026/11/12 00:34:29 WARN Force mode enabled - objects will be deleted immediately without grace period44672026/11/12 00:34:29 INFO Received uploads request method=POST path=/44682026/11/12 00:34:29 INFO Received uploads request method=POST path=/44692026/11/12 00:34:29 INFO Vacuumed table table=pending_closures44702026/11/12 00:34:29 INFO Vacuumed table table=pending_objects44712026/11/12 00:34:29 INFO Vacuumed table table=multipart_uploads44722026/11/12 00:34:29 INFO Vacuumed table table=closures44732026/11/12 00:34:29 INFO Vacuumed table table=pending_objects44742026/11/12 00:34:29 INFO Vacuumed table table=objects44752026/11/12 00:34:29 INFO Vacuumed table table=multipart_uploads44762026/11/12 00:34:29 INFO Vacuumed table table=closures44772026/11/12 00:34:29 INFO Vacuumed table table=objects44782026/11/12 00:34:29 INFO Received push request method=POST path=/44792026/11/12 00:34:29 INFO Received uploads request method=POST path=/44802026/11/12 00:34:29 INFO Received push request method=POST path=/44812026/11/12 00:34:29 INFO Received push request method=POST path=/44822026/11/12 00:34:29 INFO Received uploads request method=POST path=/44832026/11/12 00:34:29 INFO Received complete upload request method=POST path=/44842026/11/12 00:34:29 INFO Aborted multipart uploads count=0 kept=044852026/11/12 00:34:29 INFO Completed upload id=16 objects=17 duration=6.606458ms44862026/11/12 00:34:29 INFO Received complete push request method=POST path=/44872026/11/12 00:34:29 INFO Completed push id=7 objects=7 duration=2.046834ms44882026/11/12 00:34:29 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."44892026/11/12 00:34:29 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."44902026/11/12 00:34:29 INFO Received uploads request method=POST path=/44912026/11/12 00:34:29 INFO Received uploads request method=POST path=/44922026/11/12 00:34:29 INFO Received complete push request method=POST path=/44932026/11/12 00:34:29 INFO Aborted multipart uploads count=0 kept=044942026/11/12 00:34:29 INFO Completed push id=14 objects=4 duration=4.264291ms44952026/11/12 00:34:29 WARN Force mode enabled - objects will be deleted immediately without grace period44962026/11/12 00:34:29 INFO Aborted multipart uploads count=0 kept=044972026/11/12 00:34:29 WARN Force mode enabled - objects will be deleted immediately without grace period44982026/11/12 00:34:29 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044992026/11/12 00:34:29 INFO Vacuumed table table=pending_closures45002026/11/12 00:34:29 INFO Received push request method=POST path=/45012026/11/12 00:34:29 INFO Vacuumed table table=pending_objects45022026/11/12 00:34:29 INFO Aborted multipart uploads count=0 kept=045032026/11/12 00:34:29 INFO Vacuumed table table=multipart_uploads45042026/11/12 00:34:29 WARN Force mode enabled - objects will be deleted immediately without grace period45052026/11/12 00:34:29 INFO Vacuumed table table=closures45062026/11/12 00:34:29 INFO Received uploads request method=POST path=/45072026/11/12 00:34:29 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=3 objects-failed-to-delete=045082026/11/12 00:34:29 INFO Vacuumed table table=objects45092026/11/12 00:34:29 INFO Vacuumed table table=pending_closures45102026/11/12 00:34:29 INFO Vacuumed table table=pending_objects45112026/11/12 00:34:29 INFO Vacuumed table table=multipart_uploads45122026/11/12 00:34:29 INFO Vacuumed table table=closures45132026/11/12 00:34:29 INFO Vacuumed table table=objects45142026/11/12 00:34:29 INFO Received complete upload request method=POST path=/45152026/11/12 00:34:29 INFO Received uploads request method=POST path=/45162026/11/12 00:34:29 INFO Completed upload id=21 objects=7 duration=4.498875ms45172026/11/12 00:34:29 INFO Aborted multipart uploads count=0 kept=045182026/11/12 00:34:29 WARN Force mode enabled - objects will be deleted immediately without grace period45192026/11/12 00:34:29 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045202026/11/12 00:34:29 INFO Vacuumed table table=pending_closures45212026/11/12 00:34:29 INFO Received uploads request method=POST path=/45222026/11/12 00:34:29 INFO Vacuumed table table=pending_objects45232026/11/12 00:34:29 INFO Vacuumed table table=multipart_uploads45242026/11/12 00:34:29 INFO Vacuumed table table=closures45252026/11/12 00:34:29 INFO Received push request method=POST path=/45262026/11/12 00:34:29 INFO Vacuumed table table=objects45272026/11/12 00:34:29 INFO Received uploads request method=POST path=/45282026/11/12 00:34:29 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=045292026/11/12 00:34:29 INFO Vacuumed table table=pending_closures45302026/11/12 00:34:29 INFO Vacuumed table table=pending_objects45312026/11/12 00:34:29 INFO Vacuumed table table=multipart_uploads45322026/11/12 00:34:29 INFO Vacuumed table table=closures45332026/11/12 00:34:29 INFO Vacuumed table table=objects45342026/11/12 00:34:29 INFO Received complete upload request method=POST path=/45352026/11/12 00:34:29 INFO Completed upload id=18 objects=11 duration=835.292µs45362026/11/12 00:34:29 INFO Received uploads request method=POST path=/45372026/11/12 00:34:29 INFO Received complete push request method=POST path=/45382026/11/12 00:34:29 INFO Received uploads request method=POST path=/45392026/11/12 00:34:29 INFO Received push request method=POST path=/45402026/11/12 00:34:29 INFO Completed push id=13 objects=15 duration=1.575208ms45412026/11/12 00:34:29 INFO Received uploads request method=POST path=/45422026/11/12 00:34:29 INFO Received push request method=POST path=/45432026/11/12 00:34:29 INFO Received uploads request method=POST path=/45442026/11/12 00:34:29 INFO Aborted multipart uploads count=0 kept=045452026/11/12 00:34:29 INFO Received uploads request method=POST path=/45462026/11/12 00:34:29 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."45472026/11/12 00:34:30 INFO Received complete upload request method=POST path=/45482026/11/12 00:34:30 INFO Completed upload id=27 objects=4 duration=4.4185ms45492026/11/12 00:34:30 INFO Received complete push request method=POST path=/45502026/11/12 00:34:30 INFO Aborted multipart uploads count=0 kept=045512026/11/12 00:34:30 INFO Completed push id=18 objects=6 duration=2.009042ms45522026/11/12 00:34:30 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045532026/11/12 00:34:30 INFO Aborted multipart uploads count=0 kept=045542026/11/12 00:34:30 INFO Vacuumed table table=pending_closures45552026/11/12 00:34:30 INFO Vacuumed table table=pending_objects45562026/11/12 00:34:30 WARN Force mode enabled - objects will be deleted immediately without grace period45572026/11/12 00:34:30 INFO Vacuumed table table=multipart_uploads45582026/11/12 00:34:30 INFO Received complete push request method=POST path=/45592026/11/12 00:34:30 INFO Vacuumed table table=closures45602026/11/12 00:34:30 INFO Completed push id=13 objects=11 duration=1.32625ms45612026/11/12 00:34:30 INFO Vacuumed table table=objects45622026/11/12 00:34:30 INFO Received complete upload request method=POST path=/45632026/11/12 00:34:30 INFO Completed upload id=23 objects=9 duration=829.084µs45642026/11/12 00:34:30 INFO Received uploads request method=POST path=/45652026/11/12 00:34:30 INFO Aborted multipart uploads count=0 kept=045662026/11/12 00:34:30 WARN Force mode enabled - objects will be deleted immediately without grace period45672026/11/12 00:34:30 INFO Received push request method=POST path=/45682026/11/12 00:34:30 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045692026/11/12 00:34:30 INFO Vacuumed table table=pending_closures45702026/11/12 00:34:30 INFO Received complete push request method=POST path=/45712026/11/12 00:34:30 INFO Vacuumed table table=pending_objects45722026/11/12 00:34:30 INFO Vacuumed table table=multipart_uploads45732026/11/12 00:34:30 INFO Completed push id=16 objects=16 duration=5.272917ms45742026/11/12 00:34:30 INFO Vacuumed table table=closures45752026/11/12 00:34:30 INFO Vacuumed table table=objects45762026/11/12 00:34:30 INFO Received uploads request method=POST path=/45772026/11/12 00:34:30 INFO Aborted multipart uploads count=0 kept=045782026/11/12 00:34:30 INFO Received uploads request method=POST path=/45792026/11/12 00:34:30 INFO Received push request method=POST path=/45802026/11/12 00:34:30 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."45812026/11/12 00:34:30 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."45822026/11/12 00:34:30 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."45832026/11/12 00:34:30 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."45842026/11/12 00:34:30 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."45852026/11/12 00:34:30 INFO Received complete push request method=POST path=/45862026/11/12 00:34:30 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."45872026/11/12 00:34:30 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."45882026/11/12 00:34:30 INFO Completed push id=20 objects=3 duration=1.590041ms45892026/11/12 00:34:30 INFO Received uploads request method=POST path=/45902026/11/12 00:34:30 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."45912026/11/12 00:34:30 INFO Received complete push request method=POST path=/45922026/11/12 00:34:30 INFO Completed push id=16 objects=26 duration=1.438167ms45932026/11/12 00:34:30 INFO Received complete push request method=POST path=/45942026/11/12 00:34:30 INFO Received uploads request method=POST path=/45952026/11/12 00:34:30 INFO Received uploads request method=POST path=/45962026/11/12 00:34:30 INFO Completed push id=16 objects=14 duration=3.431083ms45972026/11/12 00:34:30 INFO Aborted multipart uploads count=0 kept=045982026/11/12 00:34:30 WARN Force mode enabled - objects will be deleted immediately without grace period45992026/11/12 00:34:30 INFO Received push request method=POST path=/46002026/11/12 00:34:30 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046012026/11/12 00:34:30 INFO Received push request method=POST path=/46022026/11/12 00:34:30 INFO Vacuumed table table=pending_closures46032026/11/12 00:34:30 INFO Aborted multipart uploads count=0 kept=046042026/11/12 00:34:30 INFO Vacuumed table table=pending_objects46052026/11/12 00:34:30 INFO Vacuumed table table=multipart_uploads46062026/11/12 00:34:30 WARN Force mode enabled - objects will be deleted immediately without grace period46072026/11/12 00:34:30 INFO Received push request method=POST path=/46082026/11/12 00:34:30 INFO Vacuumed table table=closures46092026/11/12 00:34:30 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046102026/11/12 00:34:30 INFO Received uploads request method=POST path=/46112026/11/12 00:34:30 INFO Vacuumed table table=objects46122026/11/12 00:34:30 INFO Vacuumed table table=pending_closures46132026/11/12 00:34:30 INFO Received push request method=POST path=/46142026/11/12 00:34:30 INFO Vacuumed table table=pending_objects46152026/11/12 00:34:30 INFO Vacuumed table table=multipart_uploads46162026/11/12 00:34:30 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=8 objects-deleted-after-grace-period=8 objects-failed-to-delete=046172026/11/12 00:34:30 INFO Vacuumed table table=closures46182026/11/12 00:34:30 INFO Vacuumed table table=objects46192026/11/12 00:34:30 INFO Vacuumed table table=pending_closures46202026/11/12 00:34:30 INFO Vacuumed table table=pending_objects46212026/11/12 00:34:30 INFO Vacuumed table table=multipart_uploads46222026/11/12 00:34:30 INFO Vacuumed table table=closures46232026/11/12 00:34:30 INFO Received complete push request method=POST path=/46242026/11/12 00:34:30 INFO Completed push id=15 objects=4 duration=562.916µs46252026/11/12 00:34:30 INFO Vacuumed table table=objects46262026/11/12 00:34:30 INFO Received complete push request method=POST path=/46272026/11/12 00:34:30 INFO Completed push id=27 objects=6 duration=2.295417ms46282026/11/12 00:34:30 INFO Received complete upload request method=POST path=/46292026/11/12 00:34:30 INFO Completed upload id=22 objects=6 duration=512.458µs46302026/11/12 00:34:30 INFO Aborted multipart uploads count=0 kept=046312026/11/12 00:34:30 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046322026/11/12 00:34:30 INFO Vacuumed table table=pending_closures46332026/11/12 00:34:30 INFO Vacuumed table table=pending_objects46342026/11/12 00:34:30 INFO Vacuumed table table=multipart_uploads46352026/11/12 00:34:30 INFO Vacuumed table table=closures46362026/11/12 00:34:30 INFO Vacuumed table table=objects46372026/11/12 00:34:30 INFO Aborted multipart uploads count=0 kept=046382026/11/12 00:34:30 WARN Force mode enabled - objects will be deleted immediately without grace period46392026/11/12 00:34:30 INFO Received complete upload request method=POST path=/46402026/11/12 00:34:30 INFO Completed upload id=15 objects=2 duration=959.042µs46412026/11/12 00:34:30 INFO Received uploads request method=POST path=/46422026/11/12 00:34:30 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."46432026/11/12 00:34:30 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."46442026/11/12 00:34:30 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."46452026/11/12 00:34:30 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."46462026/11/12 00:34:30 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."46472026/11/12 00:34:30 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."46482026/11/12 00:34:30 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."46492026/11/12 00:34:30 INFO Aborted multipart uploads count=0 kept=046502026/11/12 00:34:30 INFO Received push request method=POST path=/46512026/11/12 00:34:30 INFO Received push request method=POST path=/46522026/11/12 00:34:30 WARN Force mode enabled - objects will be deleted immediately without grace period46532026/11/12 00:34:30 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046542026/11/12 00:34:30 INFO Vacuumed table table=pending_closures46552026/11/12 00:34:30 INFO Vacuumed table table=pending_objects46562026/11/12 00:34:30 INFO Vacuumed table table=multipart_uploads46572026/11/12 00:34:30 INFO Vacuumed table table=closures46582026/11/12 00:34:30 INFO Vacuumed table table=objects46592026/11/12 00:34:30 INFO Received complete upload request method=POST path=/46602026/11/12 00:34:30 INFO Received uploads request method=POST path=/46612026/11/12 00:34:30 INFO Received uploads request method=POST path=/46622026/11/12 00:34:30 INFO Completed upload id=13 objects=14 duration=4.27725ms46632026/11/12 00:34:30 INFO Received uploads request method=POST path=/46642026/11/12 00:34:30 INFO Received complete upload request method=POST path=/46652026/11/12 00:34:30 INFO Completed upload id=15 objects=3 duration=2.51175ms46662026/11/12 00:34:30 INFO Received complete upload request method=POST path=/46672026/11/12 00:34:30 INFO Completed upload id=21 objects=3 duration=616.333µs46682026/11/12 00:34:30 INFO Received complete upload request method=POST path=/46692026/11/12 00:34:30 INFO Completed upload id=25 objects=6 duration=1.908917ms46702026/11/12 00:34:30 INFO Aborted multipart uploads count=0 kept=046712026/11/12 00:34:30 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046722026/11/12 00:34:30 INFO Received push request method=POST path=/46732026/11/12 00:34:30 INFO Vacuumed table table=pending_closures46742026/11/12 00:34:30 INFO Aborted multipart uploads count=0 kept=046752026/11/12 00:34:30 INFO Vacuumed table table=pending_objects46762026/11/12 00:34:30 INFO Vacuumed table table=multipart_uploads46772026/11/12 00:34:30 WARN Force mode enabled - objects will be deleted immediately without grace period46782026/11/12 00:34:30 INFO Vacuumed table table=closures46792026/11/12 00:34:30 INFO Vacuumed table table=objects46802026/11/12 00:34:30 INFO Received push request method=POST path=/46812026/11/12 00:34:30 INFO Received push request method=POST path=/46822026/11/12 00:34:30 INFO Received push request method=POST path=/46832026/11/12 00:34:30 INFO Received complete push request method=POST path=/46842026/11/12 00:34:30 INFO Completed push id=14 objects=3 duration=432.666µs46852026/11/12 00:34:30 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=7 objects-failed-to-delete=046862026/11/12 00:34:30 INFO Vacuumed table table=pending_closures46872026/11/12 00:34:30 INFO Vacuumed table table=pending_objects46882026/11/12 00:34:30 INFO Vacuumed table table=multipart_uploads46892026/11/12 00:34:30 INFO Vacuumed table table=closures46902026/11/12 00:34:30 INFO Received push request method=POST path=/46912026/11/12 00:34:30 INFO Vacuumed table table=objects46922026/11/12 00:34:30 INFO Received complete push request method=POST path=/46932026/11/12 00:34:30 INFO Received uploads request method=POST path=/46942026/11/12 00:34:30 INFO Completed push id=31 objects=6 duration=1.791542ms46952026/11/12 00:34:30 INFO Received uploads request method=POST path=/46962026/11/12 00:34:31 INFO Received uploads request method=POST path=/46972026/11/12 00:34:31 INFO Received uploads request method=POST path=/46982026/11/12 00:34:31 INFO Received uploads request method=POST path=/46992026/11/12 00:34:31 INFO Aborted multipart uploads count=0 kept=047002026/11/12 00:34:31 INFO Received complete push request method=POST path=/47012026/11/12 00:34:31 INFO Completed push id=11 objects=9 duration=1.226875ms47022026/11/12 00:34:31 WARN Force mode enabled - objects will be deleted immediately without grace period47032026/11/12 00:34:31 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047042026/11/12 00:34:31 INFO Vacuumed table table=pending_closures47052026/11/12 00:34:31 INFO Received complete upload request method=POST path=/47062026/11/12 00:34:31 INFO Received uploads request method=POST path=/47072026/11/12 00:34:31 INFO Completed upload id=20 objects=4 duration=783.625µs47082026/11/12 00:34:31 INFO Vacuumed table table=pending_objects47092026/11/12 00:34:31 INFO Vacuumed table table=multipart_uploads47102026/11/12 00:34:31 INFO Vacuumed table table=closures47112026/11/12 00:34:31 INFO Vacuumed table table=objects47122026/11/12 00:34:31 INFO Received complete upload request method=POST path=/47132026/11/12 00:34:31 INFO Completed upload id=26 objects=8 duration=1.573ms47142026/11/12 00:34:31 INFO Received push request method=POST path=/47152026/11/12 00:34:31 INFO Received uploads request method=POST path=/47162026/11/12 00:34:31 INFO Received uploads request method=POST path=/47172026/11/12 00:34:31 INFO Received complete push request method=POST path=/47182026/11/12 00:34:31 INFO Completed push id=28 objects=14 duration=925.958µs47192026/11/12 00:34:31 INFO Received complete push request method=POST path=/47202026/11/12 00:34:31 INFO Received complete push request method=POST path=/47212026/11/12 00:34:31 INFO Received uploads request method=POST path=/47222026/11/12 00:34:31 INFO Completed push id=22 objects=3 duration=2.682ms47232026/11/12 00:34:31 INFO Completed push id=15 objects=24 duration=978.292µs47242026/11/12 00:34:31 INFO Received push request method=POST path=/47252026/11/12 00:34:31 INFO Received uploads request method=POST path=/47262026/11/12 00:34:31 INFO Received push request method=POST path=/47272026/11/12 00:34:31 INFO Received complete upload request method=POST path=/47282026/11/12 00:34:31 INFO Received uploads request method=POST path=/47292026/11/12 00:34:31 INFO Completed upload id=19 objects=2 duration=741.833µs47302026/11/12 00:34:31 INFO Received complete upload request method=POST path=/47312026/11/12 00:34:31 INFO Completed upload id=24 objects=7 duration=1.259917ms47322026/11/12 00:34:31 INFO Aborted multipart uploads count=0 kept=047332026/11/12 00:34:31 WARN Force mode enabled - objects will be deleted immediately without grace period47342026/11/12 00:34:31 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047352026/11/12 00:34:31 INFO Vacuumed table table=pending_closures47362026/11/12 00:34:31 INFO Vacuumed table table=pending_objects47372026/11/12 00:34:31 INFO Vacuumed table table=multipart_uploads47382026/11/12 00:34:31 INFO Vacuumed table table=closures47392026/11/12 00:34:31 INFO Vacuumed table table=objects47402026/11/12 00:34:31 INFO Received push request method=POST path=/47412026/11/12 00:34:31 INFO Received complete push request method=POST path=/47422026/11/12 00:34:31 INFO Completed push id=17 objects=22 duration=2.682125ms47432026/11/12 00:34:31 INFO Received complete push request method=POST path=/47442026/11/12 00:34:31 INFO Completed push id=16 objects=5 duration=926.5µs47452026/11/12 00:34:31 INFO Received complete push request method=POST path=/47462026/11/12 00:34:31 INFO Received complete push request method=POST path=/47472026/11/12 00:34:31 INFO Completed push id=24 objects=3 duration=2.278583ms47482026/11/12 00:34:31 INFO Received uploads request method=POST path=/47492026/11/12 00:34:31 INFO Completed push id=26 objects=3 duration=2.944917ms47502026/11/12 00:34:31 INFO Received uploads request method=POST path=/47512026/11/12 00:34:31 INFO Received uploads request method=POST path=/47522026/11/12 00:34:31 INFO Received push request method=POST path=/47532026/11/12 00:34:31 INFO Received complete upload request method=POST path=/47542026/11/12 00:34:31 INFO Received complete upload request method=POST path=/47552026/11/12 00:34:31 INFO Aborted multipart uploads count=0 kept=047562026/11/12 00:34:31 INFO Completed upload id=28 objects=8 duration=587.417µs47572026/11/12 00:34:31 INFO Completed upload id=21 objects=9 duration=1.480708ms47582026/11/12 00:34:31 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=047592026/11/12 00:34:31 INFO Vacuumed table table=pending_closures47602026/11/12 00:34:31 INFO Vacuumed table table=pending_objects47612026/11/12 00:34:31 INFO Vacuumed table table=multipart_uploads47622026/11/12 00:34:31 INFO Vacuumed table table=closures47632026/11/12 00:34:31 INFO Vacuumed table table=objects47642026/11/12 00:34:31 INFO Received complete push request method=POST path=/47652026/11/12 00:34:31 INFO Received complete upload request method=POST path=/47662026/11/12 00:34:31 INFO Completed push id=25 objects=10 duration=2.42375ms47672026/11/12 00:34:31 INFO Aborted multipart uploads count=0 kept=047682026/11/12 00:34:31 INFO Completed upload id=29 objects=10 duration=614.625µs47692026/11/12 00:34:31 WARN Force mode enabled - objects will be deleted immediately without grace period47702026/11/12 00:34:31 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047712026/11/12 00:34:31 INFO Vacuumed table table=pending_closures47722026/11/12 00:34:31 INFO Vacuumed table table=pending_objects47732026/11/12 00:34:31 INFO Vacuumed table table=multipart_uploads47742026/11/12 00:34:31 INFO Vacuumed table table=closures47752026/11/12 00:34:31 INFO Vacuumed table table=objects47762026/11/12 00:34:31 INFO Received push request method=POST path=/47772026/11/12 00:34:31 INFO Aborted multipart uploads count=0 kept=047782026/11/12 00:34:31 INFO Received complete upload request method=POST path=/47792026/11/12 00:34:31 INFO Completed upload id=30 objects=20 duration=1.467917ms47802026/11/12 00:34:31 INFO Received complete upload request method=POST path=/47812026/11/12 00:34:31 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047822026/11/12 00:34:31 INFO Completed upload id=29 objects=5 duration=740.083µs47832026/11/12 00:34:31 INFO Vacuumed table table=pending_closures47842026/11/12 00:34:31 INFO Vacuumed table table=pending_objects47852026/11/12 00:34:31 INFO Vacuumed table table=multipart_uploads47862026/11/12 00:34:31 INFO Vacuumed table table=closures47872026/11/12 00:34:31 INFO Vacuumed table table=objects47882026/11/12 00:34:31 INFO Received complete push request method=POST path=/47892026/11/12 00:34:31 INFO Received complete push request method=POST path=/47902026/11/12 00:34:31 INFO Completed push id=19 objects=14 duration=3.024ms47912026/11/12 00:34:31 INFO Completed push id=31 objects=11 duration=3.955542ms47922026/11/12 00:34:31 INFO Received complete upload request method=POST path=/47932026/11/12 00:34:31 INFO Completed upload id=30 objects=2 duration=2.502958ms47942026/11/12 00:34:31 INFO Received uploads request method=POST path=/47952026/11/12 00:34:31 INFO Received uploads request method=POST path=/47962026/11/12 00:34:31 INFO Received complete upload request method=POST path=/47972026/11/12 00:34:31 INFO Completed upload id=32 objects=2 duration=7.424292ms47982026/11/12 00:34:31 INFO Aborted multipart uploads count=0 kept=047992026/11/12 00:34:31 INFO Received complete push request method=POST path=/48002026/11/12 00:34:31 INFO Received uploads request method=POST path=/48012026/11/12 00:34:31 INFO Received complete push request method=POST path=/48022026/11/12 00:34:31 INFO Completed push id=31 objects=20 duration=826.375µs48032026/11/12 00:34:31 INFO Received complete push request method=POST path=/48042026/11/12 00:34:31 INFO Completed push id=16 objects=7 duration=1.466542ms48052026/11/12 00:34:31 INFO Completed push id=1 objects=9 duration=1.086209ms48062026/11/12 00:34:31 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=048072026/11/12 00:34:31 INFO Vacuumed table table=pending_closures48082026/11/12 00:34:31 INFO Vacuumed table table=pending_objects48092026/11/12 00:34:31 INFO Vacuumed table table=multipart_uploads48102026/11/12 00:34:31 INFO Vacuumed table table=closures48112026/11/12 00:34:31 INFO Received complete push request method=POST path=/48122026/11/12 00:34:31 INFO Received push request method=POST path=/48132026/11/12 00:34:31 INFO Completed push id=33 objects=3 duration=903.084µs48142026/11/12 00:34:31 INFO Vacuumed table table=objects48152026/11/12 00:34:31 INFO Aborted multipart uploads count=0 kept=048162026/11/12 00:34:31 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048172026/11/12 00:34:31 INFO Vacuumed table table=pending_closures48182026/11/12 00:34:31 INFO Received complete upload request method=POST path=/48192026/11/12 00:34:31 INFO Vacuumed table table=pending_objects48202026/11/12 00:34:31 INFO Completed upload id=34 objects=3 duration=565.25µs48212026/11/12 00:34:31 INFO Vacuumed table table=multipart_uploads48222026/11/12 00:34:31 INFO Vacuumed table table=closures48232026/11/12 00:34:31 INFO Received complete upload request method=POST path=/48242026/11/12 00:34:31 INFO Vacuumed table table=objects48252026/11/12 00:34:31 INFO Completed upload id=32 objects=5 duration=1.881209ms48262026/11/12 00:34:31 INFO Aborted multipart uploads count=0 kept=048272026/11/12 00:34:31 WARN Force mode enabled - objects will be deleted immediately without grace period48282026/11/12 00:34:31 INFO Aborted multipart uploads count=0 kept=048292026/11/12 00:34:31 INFO Received complete upload request method=POST path=/48302026/11/12 00:34:31 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048312026/11/12 00:34:31 INFO Completed upload id=33 objects=3 duration=6.261042ms48322026/11/12 00:34:31 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048332026/11/12 00:34:31 INFO Received uploads request method=POST path=/48342026/11/12 00:34:31 INFO Vacuumed table table=pending_closures48352026/11/12 00:34:31 INFO Vacuumed table table=pending_closures48362026/11/12 00:34:31 INFO Received complete push request method=POST path=/48372026/11/12 00:34:31 INFO Received complete upload request method=POST path=/48382026/11/12 00:34:31 INFO Completed upload id=20 objects=6 duration=1.304042ms48392026/11/12 00:34:31 INFO Vacuumed table table=pending_objects48402026/11/12 00:34:31 INFO Vacuumed table table=pending_objects48412026/11/12 00:34:31 INFO Vacuumed table table=multipart_uploads48422026/11/12 00:34:31 INFO Completed push id=23 objects=6 duration=2.356959ms48432026/11/12 00:34:31 INFO Vacuumed table table=multipart_uploads48442026/11/12 00:34:31 INFO Vacuumed table table=closures48452026/11/12 00:34:31 INFO Aborted multipart uploads count=0 kept=048462026/11/12 00:34:31 INFO Vacuumed table table=closures48472026/11/12 00:34:31 INFO Vacuumed table table=objects48482026/11/12 00:34:31 WARN Force mode enabled - objects will be deleted immediately without grace period48492026/11/12 00:34:31 INFO Vacuumed table table=objects48502026/11/12 00:34:31 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048512026/11/12 00:34:31 INFO Vacuumed table table=pending_closures48522026/11/12 00:34:31 INFO Vacuumed table table=pending_objects48532026/11/12 00:34:31 INFO Vacuumed table table=multipart_uploads48542026/11/12 00:34:31 INFO Vacuumed table table=closures48552026/11/12 00:34:31 INFO Vacuumed table table=objects48562026/11/12 00:34:31 INFO Received complete upload request method=POST path=/48572026/11/12 00:34:31 INFO Completed upload id=26 objects=2 duration=1.408916ms48582026/11/12 00:34:31 INFO Received complete push request method=POST path=/48592026/11/12 00:34:31 INFO Completed push id=24 objects=6 duration=1.255625ms48602026/11/12 00:34:31 INFO Received push request method=POST path=/48612026/11/12 00:34:31 INFO Aborted multipart uploads count=0 kept=048622026/11/12 00:34:31 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048632026/11/12 00:34:31 INFO Received uploads request method=POST path=/48642026/11/12 00:34:31 INFO Vacuumed table table=pending_closures48652026/11/12 00:34:31 INFO Vacuumed table table=pending_objects48662026/11/12 00:34:31 INFO Vacuumed table table=multipart_uploads48672026/11/12 00:34:31 INFO Vacuumed table table=closures48682026/11/12 00:34:31 INFO Vacuumed table table=objects48692026/11/12 00:34:31 INFO Received push request method=POST path=/48702026/11/12 00:34:32 INFO Received complete upload request method=POST path=/48712026/11/12 00:34:32 INFO Completed upload id=2 objects=7 duration=2.551708ms48722026/11/12 00:34:32 INFO Received complete push request method=POST path=/48732026/11/12 00:34:32 INFO Completed push id=25 objects=7 duration=1.771792ms48742026/11/12 00:34:32 INFO Received uploads request method=POST path=/48752026/11/12 00:34:32 INFO Received push request method=POST path=/48762026/11/12 00:34:32 INFO Received complete upload request method=POST path=/48772026/11/12 00:34:32 INFO Completed upload id=21 objects=2 duration=975.167µs48782026/11/12 00:34:32 INFO Received push request method=POST path=/48792026/11/12 00:34:32 INFO Received uploads request method=POST path=/48802026/11/12 00:34:32 INFO Aborted multipart uploads count=0 kept=048812026/11/12 00:34:32 INFO Aborted multipart uploads count=0 kept=048822026/11/12 00:34:32 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048832026/11/12 00:34:32 INFO Vacuumed table table=pending_closures48842026/11/12 00:34:32 INFO Vacuumed table table=pending_objects48852026/11/12 00:34:32 INFO Vacuumed table table=multipart_uploads48862026/11/12 00:34:32 INFO Vacuumed table table=closures48872026/11/12 00:34:32 INFO Vacuumed table table=objects48882026/11/12 00:34:32 INFO Aborted multipart uploads count=0 kept=048892026/11/12 00:34:32 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."48902026/11/12 00:34:32 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."48912026/11/12 00:34:32 WARN Force mode enabled - objects will be deleted immediately without grace period48922026/11/12 00:34:32 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."48932026/11/12 00:34:32 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048942026/11/12 00:34:32 INFO Vacuumed table table=pending_closures48952026/11/12 00:34:32 INFO Vacuumed table table=pending_objects48962026/11/12 00:34:32 INFO Vacuumed table table=multipart_uploads48972026/11/12 00:34:32 INFO Aborted multipart uploads count=0 kept=048982026/11/12 00:34:32 INFO Vacuumed table table=closures48992026/11/12 00:34:32 WARN Force mode enabled - objects will be deleted immediately without grace period49002026/11/12 00:34:32 INFO Vacuumed table table=objects49012026/11/12 00:34:32 INFO Received push request method=POST path=/49022026/11/12 00:34:32 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049032026/11/12 00:34:32 INFO Vacuumed table table=pending_closures49042026/11/12 00:34:32 INFO Vacuumed table table=pending_objects49052026/11/12 00:34:32 INFO Vacuumed table table=multipart_uploads49062026/11/12 00:34:32 INFO Vacuumed table table=closures49072026/11/12 00:34:32 INFO Vacuumed table table=objects49082026/11/12 00:34:32 INFO Received complete upload request method=POST path=/49092026/11/12 00:34:32 INFO Completed upload id=19 objects=2 duration=2.153125ms49102026/11/12 00:34:32 INFO Received complete upload request method=POST path=/49112026/11/12 00:34:32 INFO Completed upload id=26 objects=24 duration=1.667542ms49122026/11/12 00:34:32 INFO Received push request method=POST path=/49132026/11/12 00:34:32 INFO Received uploads request method=POST path=/49142026/11/12 00:34:32 INFO Received uploads request method=POST path=/49152026/11/12 00:34:32 INFO Received complete upload request method=POST path=/49162026/11/12 00:34:32 INFO Completed upload id=8 objects=6 duration=1.451667ms49172026/11/12 00:34:32 INFO Received complete push request method=POST path=/49182026/11/12 00:34:32 INFO Completed push id=28 objects=11 duration=4.831625ms49192026/11/12 00:34:32 INFO Received push request method=POST path=/49202026/11/12 00:34:32 INFO Received complete push request method=POST path=/49212026/11/12 00:34:32 INFO Completed push id=33 objects=7 duration=812.791µs49222026/11/12 00:34:32 INFO Received complete upload request method=POST path=/49232026/11/12 00:34:32 INFO Completed upload id=26 objects=3 duration=805.958µs49242026/11/12 00:34:32 INFO Received complete push request method=POST path=/49252026/11/12 00:34:32 INFO Completed push id=24 objects=24 duration=2.5995ms49262026/11/12 00:34:32 INFO Aborted multipart uploads count=0 kept=049272026/11/12 00:34:32 WARN Force mode enabled - objects will be deleted immediately without grace period49282026/11/12 00:34:32 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049292026/11/12 00:34:32 INFO Vacuumed table table=pending_closures49302026/11/12 00:34:32 INFO Aborted multipart uploads count=0 kept=049312026/11/12 00:34:32 INFO Vacuumed table table=pending_objects49322026/11/12 00:34:32 INFO Vacuumed table table=multipart_uploads49332026/11/12 00:34:32 WARN Force mode enabled - objects will be deleted immediately without grace period49342026/11/12 00:34:32 INFO Vacuumed table table=closures49352026/11/12 00:34:32 INFO Vacuumed table table=objects49362026/11/12 00:34:32 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049372026/11/12 00:34:32 INFO Vacuumed table table=pending_closures49382026/11/12 00:34:32 INFO Vacuumed table table=pending_objects49392026/11/12 00:34:32 INFO Vacuumed table table=multipart_uploads49402026/11/12 00:34:32 INFO Vacuumed table table=closures49412026/11/12 00:34:32 INFO Vacuumed table table=objects49422026/11/12 00:34:32 INFO Received uploads request method=POST path=/49432026/11/12 00:34:32 INFO Received complete upload request method=POST path=/49442026/11/12 00:34:32 INFO Completed upload id=27 objects=26 duration=864.417µs49452026/11/12 00:34:32 INFO Received uploads request method=POST path=/49462026/11/12 00:34:32 INFO Received complete push request method=POST path=/49472026/11/12 00:34:32 INFO Received complete push request method=POST path=/49482026/11/12 00:34:32 INFO Completed push id=28 objects=6 duration=2.203875ms49492026/11/12 00:34:32 INFO Completed push id=29 objects=24 duration=2.327ms49502026/11/12 00:34:32 INFO Aborted multipart uploads count=0 kept=049512026/11/12 00:34:32 INFO Received complete push request method=POST path=/49522026/11/12 00:34:32 INFO Completed push id=29 objects=6 duration=1.766166ms49532026/11/12 00:34:32 WARN Force mode enabled - objects will be deleted immediately without grace period49542026/11/12 00:34:32 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=049552026/11/12 00:34:32 INFO Received push request method=POST path=/49562026/11/12 00:34:32 INFO Vacuumed table table=pending_closures49572026/11/12 00:34:32 INFO Received complete upload request method=POST path=/49582026/11/12 00:34:32 INFO Vacuumed table table=pending_objects49592026/11/12 00:34:32 INFO Completed upload id=12 objects=9 duration=4.596167ms49602026/11/12 00:34:32 INFO Vacuumed table table=multipart_uploads49612026/11/12 00:34:32 INFO Vacuumed table table=closures49622026/11/12 00:34:32 INFO Vacuumed table table=objects49632026/11/12 00:34:32 INFO Received uploads request method=POST path=/49642026/11/12 00:34:32 INFO Aborted multipart uploads count=0 kept=049652026/11/12 00:34:32 WARN Force mode enabled - objects will be deleted immediately without grace period49662026/11/12 00:34:32 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049672026/11/12 00:34:32 INFO Vacuumed table table=pending_closures49682026/11/12 00:34:32 INFO Vacuumed table table=pending_objects49692026/11/12 00:34:32 INFO Vacuumed table table=multipart_uploads49702026/11/12 00:34:32 INFO Vacuumed table table=closures49712026/11/12 00:34:32 INFO Vacuumed table table=objects49722026/11/12 00:34:32 INFO Received push request method=POST path=/49732026/11/12 00:34:32 INFO Received complete push request method=POST path=/49742026/11/12 00:34:32 INFO Completed push id=9 objects=9 duration=2.854125ms49752026/11/12 00:34:32 INFO Received complete upload request method=POST path=/49762026/11/12 00:34:32 INFO Completed upload id=30 objects=8 duration=1.005417ms49772026/11/12 00:34:32 INFO Received complete upload request method=POST path=/49782026/11/12 00:34:32 INFO Completed upload id=23 objects=7 duration=1.724375ms49792026/11/12 00:34:32 INFO Received complete upload request method=POST path=/49802026/11/12 00:34:32 INFO Completed upload id=21 objects=8 duration=682.417µs49812026/11/12 00:34:32 INFO Received complete upload request method=POST path=/49822026/11/12 00:34:32 INFO Completed upload id=26 objects=5 duration=1.442292ms49832026/11/12 00:34:32 INFO Received complete upload request method=POST path=/49842026/11/12 00:34:32 INFO Completed upload id=31 objects=6 duration=6.51025ms49852026/11/12 00:34:32 INFO Received complete push request method=POST path=/49862026/11/12 00:34:33 INFO Completed push id=22 objects=6 duration=2.538709ms49872026/11/12 00:34:33 INFO Received complete push request method=POST path=/49882026/11/12 00:34:33 INFO Completed push id=27 objects=3 duration=1.074041ms49892026/11/12 00:34:33 INFO Received complete upload request method=POST path=/49902026/11/12 00:34:33 INFO Received complete push request method=POST path=/49912026/11/12 00:34:33 INFO Completed upload id=32 objects=4 duration=2.015375ms49922026/11/12 00:34:33 INFO Completed push id=24 objects=8 duration=2.086208ms49932026/11/12 00:34:33 INFO Received complete upload request method=POST path=/49942026/11/12 00:34:33 INFO Completed upload id=28 objects=5 duration=1.391292ms49952026/11/12 00:34:33 INFO Received complete upload request method=POST path=/49962026/11/12 00:34:33 INFO Completed upload id=25 objects=9 duration=1.069292ms49972026/11/12 00:34:33 INFO Received complete push request method=POST path=/49982026/11/12 00:34:33 INFO Completed push id=29 objects=17 duration=726.708µs49992026/11/12 00:34:33 INFO Received complete upload request method=POST path=/50002026/11/12 00:34:33 INFO Completed upload id=27 objects=3 duration=871.166µs50012026/11/12 00:34:33 INFO Aborted multipart uploads count=0 kept=050022026/11/12 00:34:33 INFO Received complete upload request method=POST path=/50032026/11/12 00:34:33 WARN Force mode enabled - objects will be deleted immediately without grace period50042026/11/12 00:34:33 INFO Completed upload id=33 objects=10 duration=2.032584ms50052026/11/12 00:34:33 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=050062026/11/12 00:34:33 INFO Received complete upload request method=POST path=/50072026/11/12 00:34:33 INFO Completed upload id=28 objects=6 duration=834.333µs50082026/11/12 00:34:33 INFO Vacuumed table table=pending_closures50092026/11/12 00:34:33 INFO Vacuumed table table=pending_objects50102026/11/12 00:34:33 INFO Vacuumed table table=multipart_uploads50112026/11/12 00:34:33 INFO Vacuumed table table=closures50122026/11/12 00:34:33 INFO Vacuumed table table=objects50132026/11/12 00:34:33 INFO Received complete push request method=POST path=/50142026/11/12 00:34:33 INFO Completed push id=30 objects=4 duration=785.041µs50152026/11/12 00:34:33 INFO Received complete push request method=POST path=/50162026/11/12 00:34:33 INFO Completed push id=31 objects=7 duration=1.154458ms50172026/11/12 00:34:33 INFO Received complete upload request method=POST path=/50182026/11/12 00:34:33 INFO Completed upload id=32 objects=5 duration=1.365417ms50192026/11/12 00:34:33 INFO Received complete upload request method=POST path=/50202026/11/12 00:34:33 INFO Completed upload id=34 objects=23 duration=892.042µs50212026/11/12 00:34:33 INFO Received complete upload request method=POST path=/50222026/11/12 00:34:33 INFO Completed upload id=18 objects=22 duration=762.584µs50232026/11/12 00:34:33 INFO Received complete push request method=POST path=/50242026/11/12 00:34:33 INFO Completed push id=35 objects=6 duration=769µs50252026/11/12 00:34:33 INFO Received complete upload request method=POST path=/50262026/11/12 00:34:33 INFO Completed upload id=21 objects=8 duration=450.458µs50272026/11/12 00:34:33 INFO Aborted multipart uploads count=0 kept=050282026/11/12 00:34:33 WARN Force mode enabled - objects will be deleted immediately without grace period50292026/11/12 00:34:33 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=050302026/11/12 00:34:33 INFO Vacuumed table table=pending_closures50312026/11/12 00:34:33 INFO Vacuumed table table=pending_objects50322026/11/12 00:34:33 INFO Vacuumed table table=multipart_uploads50332026/11/12 00:34:33 INFO Vacuumed table table=closures50342026/11/12 00:34:33 INFO Vacuumed table table=objects50352026/11/12 00:34:33 INFO Received complete upload request method=POST path=/50362026/11/12 00:34:33 INFO Completed upload id=22 objects=7 duration=576.25µs50372026/11/12 00:34:33 INFO Received complete push request method=POST path=/50382026/11/12 00:34:33 INFO Completed push id=24 objects=5 duration=666.541µs50392026/11/12 00:34:33 INFO Received complete upload request method=POST path=/50402026/11/12 00:34:33 INFO Completed upload id=34 objects=24 duration=691.25µs50412026/11/12 00:34:33 INFO Aborted multipart uploads count=0 kept=050422026/11/12 00:34:33 WARN Force mode enabled - objects will be deleted immediately without grace period50432026/11/12 00:34:33 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=050442026/11/12 00:34:33 INFO Received complete upload request method=POST path=/50452026/11/12 00:34:33 INFO Vacuumed table table=pending_closures50462026/11/12 00:34:33 INFO Completed upload id=25 objects=8 duration=616.208µs50472026/11/12 00:34:33 INFO Vacuumed table table=pending_objects50482026/11/12 00:34:33 INFO Vacuumed table table=multipart_uploads50492026/11/12 00:34:33 INFO Vacuumed table table=closures50502026/11/12 00:34:33 INFO Vacuumed table table=objects50512026/11/12 00:34:33 INFO Received complete push request method=POST path=/50522026/11/12 00:34:33 INFO Completed push id=26 objects=4 duration=689.833µs50532026/11/12 00:34:33 INFO Received complete push request method=POST path=/50542026/11/12 00:34:33 INFO Completed push id=30 objects=2 duration=591.542µs50552026/11/12 00:34:33 INFO Received complete push request method=POST path=/50562026/11/12 00:34:33 INFO Completed push id=32 objects=7 duration=692.166µs50572026/11/12 00:34:33 INFO Received complete upload request method=POST path=/50582026/11/12 00:34:33 INFO Completed upload id=33 objects=6 duration=513.042µs50592026/11/12 00:34:33 INFO Received complete upload request method=POST path=/50602026/11/12 00:34:33 INFO Completed upload id=34 objects=3 duration=470.666µs50612026/11/12 00:34:33 INFO Aborted multipart uploads count=0 kept=050622026/11/12 00:34:33 WARN Force mode enabled - objects will be deleted immediately without grace period50632026/11/12 00:34:33 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=050642026/11/12 00:34:33 INFO Vacuumed table table=pending_closures50652026/11/12 00:34:33 INFO Vacuumed table table=pending_objects50662026/11/12 00:34:33 INFO Vacuumed table table=multipart_uploads50672026/11/12 00:34:33 INFO Vacuumed table table=closures50682026/11/12 00:34:33 INFO Vacuumed table table=objects5069--- PASS: FuzzPushGC (0.00s)5070 --- PASS: FuzzPushGC/seed#7 (11.39s)5071 --- PASS: FuzzPushGC/seed#8 (11.82s)5072 --- PASS: FuzzPushGC/seed#3 (12.54s)5073 --- PASS: FuzzPushGC/seed#0 (13.39s)5074 --- PASS: FuzzPushGC/seed#13 (13.49s)5075 --- PASS: FuzzPushGC/seed#15 (13.88s)5076 --- PASS: FuzzPushGC/seed#14 (14.70s)5077 --- PASS: FuzzPushGC/seed#11 (14.81s)5078 --- PASS: FuzzPushGC/seed#4 (14.90s)5079 --- PASS: FuzzPushGC/seed#6 (17.31s)5080 --- PASS: FuzzPushGC/seed#5 (17.33s)5081 --- PASS: FuzzPushGC/seed#2 (18.18s)5082 --- PASS: FuzzPushGC/seed#1 (18.61s)5083 --- PASS: FuzzPushGC/seed#12 (18.88s)5084 --- PASS: FuzzPushGC/seed#9 (18.97s)5085 --- PASS: FuzzPushGC/seed#10 (19.11s)5086PASS5087{"timestamp":"2026-11-12T00:34:33.679098Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:54686","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(11)"}5088{"timestamp":"2026-11-12T00:34:33.679196Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:55017","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(6)"}5089{"timestamp":"2026-11-12T00:34:33.679275Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:55009","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(8)"}5090{"timestamp":"2026-11-12T00:34:33.67932Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:54834","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(10)"}5091{"timestamp":"2026-11-12T00:34:33.679414Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:55005","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(8)"}5092{"timestamp":"2026-11-12T00:34:33.679475Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:54833","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(6)"}50932026-11-12 00:34:33.920 UTC [4104] LOG: received smart shutdown request50942026-11-12 00:34:33.921 UTC [4104] LOG: background worker "logical replication launcher" (PID 4115) exited with exit code 150952026-11-12 00:34:35.605 UTC [4109] LOG: checkpoint complete: wrote 3326 buffers (20.3%), wrote 4 SLRU buffers; 0 WAL file(s) added, 0 removed, 33 recycled; write=30.828 s, sync=1.538 s, total=32.472 s; sync files=42374, longest=0.010 s, average=0.001 s; distance=533405 kB, estimate=533405 kB; lsn=0/27AA0768, redo lsn=0/2207461850962026-11-12 00:34:35.608 UTC [4109] LOG: shutting down50972026-11-12 00:34:35.608 UTC [4109] LOG: checkpoint starting: shutdown immediate50982026-11-12 00:34:36.222 UTC [4109] LOG: checkpoint complete: wrote 8606 buffers (52.5%), wrote 2 SLRU buffers; 0 WAL file(s) added, 0 removed, 5 recycled; write=0.466 s, sync=0.120 s, total=0.614 s; sync files=2874, longest=0.001 s, average=0.001 s; distance=92336 kB, estimate=489298 kB; lsn=0/27AA07E0, redo lsn=0/27AA07E050992026-11-12 00:34:36.238 UTC [4104] LOG: database system is shut down5100Running OIDC tests...5101=== RUN TestAudienceForIssuer5102=== PAUSE TestAudienceForIssuer5103=== RUN TestProviderClientGivesUpOnSilentIssuer5104=== PAUSE TestProviderClientGivesUpOnSilentIssuer5105=== RUN TestGlobMatch5106=== PAUSE TestGlobMatch5107=== RUN TestValidateToken_ValidToken5108=== PAUSE TestValidateToken_ValidToken5109=== RUN TestValidateToken_WrongAudience5110=== PAUSE TestValidateToken_WrongAudience5111=== RUN TestValidateToken_Expired5112=== PAUSE TestValidateToken_Expired5113=== RUN TestValidateToken_BoundClaimsMismatch5114=== PAUSE TestValidateToken_BoundClaimsMismatch5115=== RUN TestValidateToken_BoundSubjectMismatch5116=== PAUSE TestValidateToken_BoundSubjectMismatch5117=== RUN TestValidateToken_MultipleProviders5118=== PAUSE TestValidateToken_MultipleProviders5119=== RUN TestValidateToken_UnknownIssuerFetchesNoKeys5120=== PAUSE TestValidateToken_UnknownIssuerFetchesNoKeys5121=== RUN TestValidateToken_GoogleSchemelessIssuer5122=== PAUSE TestValidateToken_GoogleSchemelessIssuer5123=== RUN TestValidateToken_KubernetesServiceAccount5124=== PAUSE TestValidateToken_KubernetesServiceAccount5125=== RUN TestNewValidator_KubernetesRequiresCA5126=== PAUSE TestNewValidator_KubernetesRequiresCA5127=== RUN TestValidateToken_KubernetesIssuerFromOwnToken5128=== PAUSE TestValidateToken_KubernetesIssuerFromOwnToken5129=== RUN TestPins_ReservedForMatchingRule5130=== PAUSE TestPins_ReservedForMatchingRule5131=== RUN TestPins_TopLevelShorthand5132=== PAUSE TestPins_TopLevelShorthand5133=== RUN TestPins_ConfigValidation5134=== PAUSE TestPins_ConfigValidation5135=== RUN TestScopes_LegacyProviderDefaultsToWrite5136=== PAUSE TestScopes_LegacyProviderDefaultsToWrite5137=== RUN TestScopes_Rules5138=== PAUSE TestScopes_Rules5139=== RUN TestScopes_ConfigValidation5140=== PAUSE TestScopes_ConfigValidation5141=== CONT TestAudienceForIssuer5142--- PASS: TestAudienceForIssuer (0.00s)5143=== CONT TestValidateToken_WrongAudience5144=== CONT TestPins_TopLevelShorthand5145=== CONT TestScopes_LegacyProviderDefaultsToWrite5146=== CONT TestValidateToken_Expired5147=== CONT TestPins_ConfigValidation5148=== CONT TestValidateToken_GoogleSchemelessIssuer5149=== CONT TestPins_ReservedForMatchingRule5150=== CONT TestValidateToken_KubernetesIssuerFromOwnToken5151--- PASS: TestPins_ConfigValidation (0.00s)5152=== CONT TestNewValidator_KubernetesRequiresCA5153=== CONT TestValidateToken_KubernetesServiceAccount5154=== CONT TestValidateToken_MultipleProviders5155=== CONT TestValidateToken_UnknownIssuerFetchesNoKeys5156=== CONT TestGlobMatch51572026/11/12 00:34:39 INFO OIDC provider initialized name=known issuer=https://known.example5158=== RUN TestGlobMatch/foo_foo5159=== PAUSE TestGlobMatch/foo_foo5160=== RUN TestGlobMatch/foo_bar5161=== PAUSE TestGlobMatch/foo_bar5162=== RUN TestGlobMatch/*_5163=== PAUSE TestGlobMatch/*_5164=== RUN TestGlobMatch/*_anything5165=== PAUSE TestGlobMatch/*_anything5166=== RUN TestGlobMatch/foo*_foo5167=== PAUSE TestGlobMatch/foo*_foo5168=== RUN TestGlobMatch/foo*_foobar5169=== PAUSE TestGlobMatch/foo*_foobar5170=== RUN TestGlobMatch/foo*_bar5171=== PAUSE TestGlobMatch/foo*_bar5172=== RUN TestGlobMatch/*bar_bar5173=== PAUSE TestGlobMatch/*bar_bar5174=== RUN TestGlobMatch/*bar_foobar5175=== PAUSE TestGlobMatch/*bar_foobar5176=== RUN TestGlobMatch/*bar_foo5177=== PAUSE TestGlobMatch/*bar_foo5178=== RUN TestGlobMatch/foo*bar_foobar5179=== PAUSE TestGlobMatch/foo*bar_foobar5180=== RUN TestGlobMatch/foo*bar_foo123bar5181=== PAUSE TestGlobMatch/foo*bar_foo123bar5182=== RUN TestGlobMatch/foo*bar_foobarbaz5183=== PAUSE TestGlobMatch/foo*bar_foobarbaz5184=== RUN TestGlobMatch/*/*_foo/bar5185=== PAUSE TestGlobMatch/*/*_foo/bar5186=== RUN TestGlobMatch/*/*_foo5187=== PAUSE TestGlobMatch/*/*_foo5188=== RUN TestGlobMatch/refs/heads/*_refs/heads/main5189=== PAUSE TestGlobMatch/refs/heads/*_refs/heads/main5190=== RUN TestGlobMatch/refs/heads/*_refs/tags/v1.05191=== PAUSE TestGlobMatch/refs/heads/*_refs/tags/v1.05192=== RUN TestGlobMatch/refs/*/main_refs/heads/main5193=== PAUSE TestGlobMatch/refs/*/main_refs/heads/main5194=== RUN TestGlobMatch/fo?_foo5195=== PAUSE TestGlobMatch/fo?_foo5196=== RUN TestGlobMatch/fo?_fo5197=== PAUSE TestGlobMatch/fo?_fo5198=== RUN TestGlobMatch/fo?_fooo5199=== PAUSE TestGlobMatch/fo?_fooo5200=== RUN TestGlobMatch/?oo_foo5201=== PAUSE TestGlobMatch/?oo_foo5202=== RUN TestGlobMatch/?oo_boo5203=== PAUSE TestGlobMatch/?oo_boo5204=== RUN TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5205=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5206=== RUN TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5207=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5208=== CONT TestGlobMatch/foo_foo5209=== CONT TestValidateToken_ValidToken5210=== CONT TestGlobMatch/*/*_foo/bar5211=== CONT TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5212=== CONT TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5213=== CONT TestGlobMatch/?oo_boo5214=== CONT TestGlobMatch/?oo_foo5215=== CONT TestGlobMatch/fo?_fooo5216=== CONT TestGlobMatch/fo?_fo5217=== CONT TestGlobMatch/fo?_foo5218=== CONT TestGlobMatch/refs/*/main_refs/heads/main5219=== CONT TestGlobMatch/refs/heads/*_refs/tags/v1.05220=== CONT TestGlobMatch/refs/heads/*_refs/heads/main5221=== CONT TestGlobMatch/*/*_foo5222=== CONT TestGlobMatch/*bar_bar5223=== CONT TestGlobMatch/foo*bar_foobarbaz5224=== CONT TestGlobMatch/foo*bar_foo123bar5225=== CONT TestGlobMatch/foo*bar_foobar5226=== CONT TestGlobMatch/*bar_foo5227=== CONT TestGlobMatch/*bar_foobar5228=== CONT TestGlobMatch/foo*_foo5229=== CONT TestGlobMatch/foo*_bar5230=== CONT TestGlobMatch/foo*_foobar5231=== CONT TestGlobMatch/*_5232=== CONT TestGlobMatch/*_anything5233=== CONT TestGlobMatch/foo_bar5234--- PASS: TestGlobMatch (0.00s)5235 --- PASS: TestGlobMatch/foo_foo (0.00s)5236 --- PASS: TestGlobMatch/*/*_foo/bar (0.00s)5237 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main (0.00s)5238 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main (0.00s)5239 --- PASS: TestGlobMatch/?oo_boo (0.00s)5240 --- PASS: TestGlobMatch/?oo_foo (0.00s)5241 --- PASS: TestGlobMatch/fo?_fooo (0.00s)5242 --- PASS: TestGlobMatch/fo?_fo (0.00s)5243 --- PASS: TestGlobMatch/fo?_foo (0.00s)5244 --- PASS: TestGlobMatch/refs/*/main_refs/heads/main (0.00s)5245 --- PASS: TestGlobMatch/refs/heads/*_refs/tags/v1.0 (0.00s)5246 --- PASS: TestGlobMatch/refs/heads/*_refs/heads/main (0.00s)5247 --- PASS: TestGlobMatch/*/*_foo (0.00s)5248 --- PASS: TestGlobMatch/*bar_bar (0.00s)5249 --- PASS: TestGlobMatch/foo*bar_foobarbaz (0.00s)5250 --- PASS: TestGlobMatch/foo*bar_foo123bar (0.00s)5251 --- PASS: TestGlobMatch/foo*bar_foobar (0.00s)5252 --- PASS: TestGlobMatch/*bar_foo (0.00s)5253 --- PASS: TestGlobMatch/*bar_foobar (0.00s)5254 --- PASS: TestGlobMatch/foo*_foo (0.00s)5255 --- PASS: TestGlobMatch/foo*_bar (0.00s)5256 --- PASS: TestGlobMatch/foo*_foobar (0.00s)5257 --- PASS: TestGlobMatch/*_ (0.00s)5258 --- PASS: TestGlobMatch/*_anything (0.00s)5259 --- PASS: TestGlobMatch/foo_bar (0.00s)5260=== CONT TestValidateToken_BoundSubjectMismatch5261=== CONT TestValidateToken_BoundClaimsMismatch5262=== CONT TestScopes_Rules5263=== CONT TestScopes_ConfigValidation5264--- PASS: TestScopes_ConfigValidation (0.00s)5265=== CONT TestProviderClientGivesUpOnSilentIssuer5266--- PASS: TestProviderClientGivesUpOnSilentIssuer (0.00s)52672026/11/12 00:34:39 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:56221/oidc5268--- PASS: TestPins_TopLevelShorthand (0.05s)52692026/11/12 00:34:39 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:56223/oidc52702026/11/12 00:34:39 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:56225/oidc5271--- PASS: TestScopes_Rules (0.18s)5272--- PASS: TestValidateToken_BoundSubjectMismatch (0.18s)52732026/11/12 00:34:39 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:56227/oidc5274--- PASS: TestValidateToken_BoundClaimsMismatch (0.19s)52752026/11/12 00:34:39 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:56229/oidc52762026/11/12 00:34:39 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:56233/oidc5277--- PASS: TestValidateToken_Expired (0.29s)5278--- PASS: TestValidateToken_ValidToken (0.27s)52792026/11/12 00:34:39 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:56230/oidc5280--- PASS: TestValidateToken_WrongAudience (0.30s)52812026/11/12 00:34:39 INFO OIDC provider initialized name=kubernetes issuer=https://127.0.0.1:5623552822026/11/12 00:34:39 INFO OIDC provider initialized name=google issuer=https://accounts.google.com52832026/11/12 00:34:39 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:56239/oidc5284--- PASS: TestPins_ReservedForMatchingRule (0.33s)5285--- PASS: TestValidateToken_KubernetesServiceAccount (0.33s)52862026/11/12 00:34:39 INFO OIDC provider initialized name=kubernetes issuer=https://oidc.eks.invalid/id/ABC1235287--- PASS: TestValidateToken_GoogleSchemelessIssuer (0.35s)5288--- PASS: TestValidateToken_KubernetesIssuerFromOwnToken (0.35s)52892026/11/12 00:34:39 http: TLS handshake error from 127.0.0.1:56245: read tcp 127.0.0.1:56244->127.0.0.1:56245: use of closed network connection5290--- PASS: TestNewValidator_KubernetesRequiresCA (0.38s)5291--- PASS: TestValidateToken_UnknownIssuerFetchesNoKeys (0.39s)52922026/11/12 00:34:39 INFO OIDC provider initialized name=provider1 issuer=http://127.0.0.1:56236/oidc52932026/11/12 00:34:39 INFO OIDC provider initialized name=provider2 issuer=http://127.0.0.1:56247/oidc5294--- PASS: TestValidateToken_MultipleProviders (0.48s)52952026/11/12 00:34:40 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:56250/oidc5296--- PASS: TestScopes_LegacyProviderDefaultsToWrite (0.58s)5297PASS5298Running hook tests...5299=== RUN TestSendPathsEmpty5300=== PAUSE TestSendPathsEmpty5301=== RUN TestQueueEnqueueAndFetch5302=== PAUSE TestQueueEnqueueAndFetch5303=== RUN TestQueueDeduplication5304=== PAUSE TestQueueDeduplication5305=== RUN TestQueueRemove5306=== PAUSE TestQueueRemove5307=== RUN TestQueueFetchBatchLimit5308=== PAUSE TestQueueFetchBatchLimit5309=== RUN TestQueueRetryMovesToBack5310=== PAUSE TestQueueRetryMovesToBack5311=== RUN TestQueueFetchRemoveLifecycle5312=== PAUSE TestQueueFetchRemoveLifecycle5313=== RUN TestQueueConcurrentWriters5314=== PAUSE TestQueueConcurrentWriters5315=== RUN TestQueueRemoveLargeClosure5316=== PAUSE TestQueueRemoveLargeClosure5317=== RUN TestServerClientIntegration5318--- PASS: TestServerClientIntegration (0.00s)5319=== RUN TestServerQueueError53202026/11/12 00:34:40 ERROR Failed to queue paths error="permission denied" count=15321--- PASS: TestServerQueueError (0.00s)5322=== RUN TestGetListenerSocketActivation5323 server_test.go:147: systemd socket activation does not exist on darwin5324--- SKIP: TestGetListenerSocketActivation (0.00s)5325=== RUN TestServerStalledClientDoesNotBlockShutdown5326=== PAUSE TestServerStalledClientDoesNotBlockShutdown5327=== RUN TestServerBacksOffOnAcceptErrors5328=== PAUSE TestServerBacksOffOnAcceptErrors5329=== RUN TestDrainIsolatesPoisonPath5330=== PAUSE TestDrainIsolatesPoisonPath5331=== RUN TestRunNotBlockedByPoisonHead5332=== PAUSE TestRunNotBlockedByPoisonHead5333=== RUN TestDrainGivesUpWhenServerDown5334=== PAUSE TestDrainGivesUpWhenServerDown5335=== RUN TestFailedPathPrunedByLaterClosure5336=== PAUSE TestFailedPathPrunedByLaterClosure5337=== RUN TestWorkerUploadsAndRemoves5338=== PAUSE TestWorkerUploadsAndRemoves5339=== RUN TestWorkerSkipsGCdPaths5340=== PAUSE TestWorkerSkipsGCdPaths5341=== RUN TestWorkerPrunesClosureDeps5342=== PAUSE TestWorkerPrunesClosureDeps5343=== RUN TestDrainTimeout5344=== PAUSE TestDrainTimeout5345=== RUN TestWorkerRemoveFailureIsNotProgress5346=== PAUSE TestWorkerRemoveFailureIsNotProgress5347=== RUN TestWorkerKeepsPathItCannotStat5348=== PAUSE TestWorkerKeepsPathItCannotStat5349=== CONT TestSendPathsEmpty5350=== CONT TestFailedPathPrunedByLaterClosure5351--- PASS: TestSendPathsEmpty (0.00s)5352=== CONT TestWorkerKeepsPathItCannotStat5353=== CONT TestWorkerRemoveFailureIsNotProgress5354=== CONT TestDrainTimeout5355=== CONT TestWorkerPrunesClosureDeps5356=== CONT TestWorkerSkipsGCdPaths5357=== CONT TestWorkerUploadsAndRemoves5358=== CONT TestDrainIsolatesPoisonPath5359=== CONT TestServerBacksOffOnAcceptErrors5360=== CONT TestServerStalledClientDoesNotBlockShutdown53612000/01/01 00:00:00 ERROR Accept failed error="too many open files"5362=== CONT TestQueueRemoveLargeClosure53632000/01/01 00:00:00 ERROR Accept failed error="too many open files"53642000/01/01 00:00:00 ERROR Accept failed error="too many open files"53652000/01/01 00:00:00 ERROR Accept failed error="too many open files"53662000/01/01 00:00:00 ERROR Accept failed error="too many open files"53672000/01/01 00:00:00 ERROR Accept failed error="too many open files"53682000/01/01 00:00:00 ERROR Accept failed error="too many open files"53692000/01/01 00:00:00 ERROR Accept failed error="too many open files"5370--- PASS: TestServerBacksOffOnAcceptErrors (0.00s)5371=== CONT TestQueueConcurrentWriters5372=== CONT TestQueueFetchRemoveLifecycle5373=== CONT TestQueueDeduplication5374=== CONT TestQueueEnqueueAndFetch5375=== CONT TestQueueRetryMovesToBack5376=== CONT TestQueueRemove5377=== CONT TestQueueFetchBatchLimit5378=== CONT TestDrainGivesUpWhenServerDown5379=== CONT TestRunNotBlockedByPoisonHead53802026/11/12 00:34:40 INFO Upload queue status pending=253812026/11/12 00:34:40 INFO Upload queue status pending=353822026/11/12 00:34:40 WARN Store path no longer exists (garbage collected?), removing from queue path=/private/var/tb/5/scratch/2e3e4b48f97c7cfb/build/TestWorkerSkipsGCdPaths2969579874/002/nonexistent53832026/11/12 00:34:40 INFO Uploading batch count=253842026/11/12 00:34:40 ERROR Upload failed error="upload failed" count=253852026/11/12 00:34:40 ERROR Upload failed, will retry later error="upload failed" path=/private/var/tb/5/scratch/2e3e4b48f97c7cfb/build/TestDrainGivesUpWhenServerDown701676137/002/a5386--- PASS: TestQueueFetchBatchLimit (0.01s)53872026/11/12 00:34:40 WARN Cannot stat store path, will retry later path=/private/var/tb/5/scratch/2e3e4b48f97c7cfb/build/TestWorkerKeepsPathItCannotStat453111930/002/locked/aaa error="stat /private/var/tb/5/scratch/2e3e4b48f97c7cfb/build/TestWorkerKeepsPathItCannotStat453111930/002/locked/aaa: permission denied"53882026/11/12 00:34:40 INFO Uploading batch count=153892026/11/12 00:34:40 ERROR Upload failed error="upload failed" count=15390--- PASS: TestQueueEnqueueAndFetch (0.01s)53912026/11/12 00:34:40 INFO Uploading batch count=453922026/11/12 00:34:40 ERROR Upload failed error="upload failed" count=453932026/11/12 00:34:40 INFO Uploading batch count=153942026/11/12 00:34:40 WARN Cannot stat store path, will retry later path=/private/var/tb/5/scratch/2e3e4b48f97c7cfb/build/TestWorkerKeepsPathItCannotStat453111930/002/locked/aaa error="stat /private/var/tb/5/scratch/2e3e4b48f97c7cfb/build/TestWorkerKeepsPathItCannotStat453111930/002/locked/aaa: permission denied"53952026/11/12 00:34:40 ERROR Upload failed, will retry later error="upload failed" path=/private/var/tb/5/scratch/2e3e4b48f97c7cfb/build/TestDrainIsolatesPoisonPath1653465798/002/bbb53962026/11/12 00:34:40 INFO Uploading batch count=153972026/11/12 00:34:40 ERROR Upload failed error="upload failed" count=153982026/11/12 00:34:40 INFO Upload queue status pending=253992026/11/12 00:34:40 INFO Uploading batch count=15400--- PASS: TestQueueRemove (0.01s)54012026/11/12 00:34:40 ERROR Upload failed, will retry later error="upload failed" path=/private/var/tb/5/scratch/2e3e4b48f97c7cfb/build/TestDrainGivesUpWhenServerDown701676137/002/b54022026/11/12 00:34:40 INFO Uploading batch count=254032026/11/12 00:34:40 ERROR Upload failed error="upload failed" count=254042026/11/12 00:34:40 ERROR Upload failed, will retry later error="upload failed" path=/private/var/tb/5/scratch/2e3e4b48f97c7cfb/build/TestDrainGivesUpWhenServerDown701676137/002/c5405--- PASS: TestQueueDeduplication (0.01s)5406--- PASS: TestQueueRetryMovesToBack (0.01s)54072026/11/12 00:34:40 WARN Cannot stat store path, will retry later path=/private/var/tb/5/scratch/2e3e4b48f97c7cfb/build/TestWorkerKeepsPathItCannotStat453111930/002/locked/aaa error="stat /private/var/tb/5/scratch/2e3e4b48f97c7cfb/build/TestWorkerKeepsPathItCannotStat453111930/002/locked/aaa: permission denied"54082026/11/12 00:34:40 INFO Uploading batch count=154092026/11/12 00:34:40 INFO Upload queue status pending=254102026/11/12 00:34:40 INFO Uploading batch count=154112026/11/12 00:34:40 INFO Uploading batch count=254122026/11/12 00:34:40 ERROR Upload failed, will retry later error="upload failed" path=/private/var/tb/5/scratch/2e3e4b48f97c7cfb/build/TestDrainGivesUpWhenServerDown701676137/002/d54132026/11/12 00:34:40 ERROR Drain finished with paths left in queue remaining=15414--- PASS: TestQueueFetchRemoveLifecycle (0.01s)54152026/11/12 00:34:40 INFO Uploading batch count=154162026/11/12 00:34:40 ERROR Upload failed error="upload failed" count=154172026/11/12 00:34:40 WARN Store path no longer exists (garbage collected?), removing from queue path=/private/var/tb/5/scratch/2e3e4b48f97c7cfb/build/TestWorkerRemoveFailureIsNotProgress3398406643/002/nonexistent54182026/11/12 00:34:40 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=154192000/01/01 00:00:00 INFO Uploading batch count=254202000/01/01 00:00:00 ERROR Upload failed error="context deadline exceeded" count=254212026/11/12 00:34:40 WARN Store path no longer exists (garbage collected?), removing from queue path=/private/var/tb/5/scratch/2e3e4b48f97c7cfb/build/TestWorkerRemoveFailureIsNotProgress3398406643/002/nonexistent54222000/01/01 00:00:00 ERROR Drain finished with paths left in queue remaining=454232026/11/12 00:34:40 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=154242026/11/12 00:34:40 INFO Uploading batch count=154252026/11/12 00:34:40 ERROR Upload failed error="upload failed" count=154262026/11/12 00:34:40 WARN Store path no longer exists (garbage collected?), removing from queue path=/private/var/tb/5/scratch/2e3e4b48f97c7cfb/build/TestWorkerRemoveFailureIsNotProgress3398406643/002/nonexistent54272026/11/12 00:34:40 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=154282026/11/12 00:34:40 ERROR Drain finished with paths left in queue remaining=154292026/11/12 00:34:40 INFO Uploading batch count=154302026/11/12 00:34:40 ERROR Upload failed error="upload failed" count=154312026/11/12 00:34:40 INFO Uploading batch count=254322026/11/12 00:34:40 ERROR Upload failed error="upload failed" count=254332026/11/12 00:34:40 ERROR Upload failed, will retry later error="upload failed" path=/private/var/tb/5/scratch/2e3e4b48f97c7cfb/build/TestDrainGivesUpWhenServerDown701676137/002/e54342026/11/12 00:34:40 ERROR Drain finished with paths left in queue remaining=15435--- PASS: TestWorkerKeepsPathItCannotStat (0.02s)54362026/11/12 00:34:40 ERROR Upload failed, will retry later error="upload failed" path=/private/var/tb/5/scratch/2e3e4b48f97c7cfb/build/TestDrainGivesUpWhenServerDown701676137/002/f54372026/11/12 00:34:40 ERROR Drain finished with paths left in queue remaining=105438--- PASS: TestFailedPathPrunedByLaterClosure (0.02s)5439--- PASS: TestWorkerRemoveFailureIsNotProgress (0.02s)5440--- PASS: TestDrainTimeout (0.02s)5441--- PASS: TestDrainIsolatesPoisonPath (0.02s)5442--- PASS: TestDrainGivesUpWhenServerDown (0.02s)5443--- PASS: TestWorkerSkipsGCdPaths (0.03s)5444--- PASS: TestWorkerPrunesClosureDeps (0.03s)5445--- PASS: TestWorkerUploadsAndRemoves (0.03s)5446--- PASS: TestQueueRemoveLargeClosure (0.07s)5447--- PASS: TestQueueConcurrentWriters (0.07s)54482026/11/12 00:34:40 ERROR Failed to decode request error="read unix /private/var/tb/5/scratch/2e3e4b48f97c7cfb/build/hook3171258967/test.sock->: i/o timeout"54492026/11/12 00:34:40 ERROR Failed to write response error="write unix /private/var/tb/5/scratch/2e3e4b48f97c7cfb/build/hook3171258967/test.sock->: i/o timeout"5450--- PASS: TestServerStalledClientDoesNotBlockShutdown (0.20s)54512026/11/12 00:34:41 INFO Uploading batch count=154522026/11/12 00:34:41 INFO Uploading batch count=154532026/11/12 00:34:41 INFO Uploading batch count=154542026/11/12 00:34:41 ERROR Upload failed error="upload failed" count=154552026/11/12 00:34:41 INFO Uploading batch count=154562026/11/12 00:34:41 ERROR Upload failed error="upload failed" count=154572026/11/12 00:34:41 INFO Uploading batch count=154582026/11/12 00:34:41 ERROR Upload failed error="upload failed" count=154592026/11/12 00:34:41 INFO Uploading batch count=154602026/11/12 00:34:41 ERROR Upload failed error="upload failed" count=154612026/11/12 00:34:41 ERROR Drain finished with paths left in queue remaining=15462--- PASS: TestRunNotBlockedByPoisonHead (1.04s)5463PASS5464Running signing tests...5465=== RUN TestExternalSigner5466=== PAUSE TestExternalSigner5467=== RUN TestExternalSignerWrongPublicKey5468=== PAUSE TestExternalSignerWrongPublicKey5469=== RUN TestExternalSignerErrors5470=== PAUSE TestExternalSignerErrors5471=== RUN TestExternalSignerRestartsAfterCrash5472=== PAUSE TestExternalSignerRestartsAfterCrash5473=== RUN TestExternalSignerProgramErrorKeepsProcess5474=== PAUSE TestExternalSignerProgramErrorKeepsProcess5475=== RUN TestExternalSignerClose5476=== PAUSE TestExternalSignerClose5477=== RUN TestExternalSignerRejectsInvalidPublicKey5478=== PAUSE TestExternalSignerRejectsInvalidPublicKey5479=== RUN TestExternalSignerCancellation5480=== PAUSE TestExternalSignerCancellation5481=== RUN TestGenerateFingerprint5482=== PAUSE TestGenerateFingerprint5483=== RUN TestParseSigningKey5484=== PAUSE TestParseSigningKey5485=== RUN TestSign5486=== PAUSE TestSign5487=== RUN TestSignBatch5488=== PAUSE TestSignBatch5489=== RUN TestNilKeyPublicKey5490=== PAUSE TestNilKeyPublicKey5491=== RUN TestSignErrors5492=== PAUSE TestSignErrors5493=== CONT TestExternalSigner5494=== CONT TestSignBatch5495=== CONT TestExternalSignerProgramErrorKeepsProcess5496=== CONT TestExternalSignerCancellation5497=== CONT TestSign5498=== CONT TestParseSigningKey5499=== RUN TestParseSigningKey/valid_32-byte_key5500=== PAUSE TestParseSigningKey/valid_32-byte_key5501=== CONT TestGenerateFingerprint5502=== RUN TestGenerateFingerprint/basic_with_references5503=== PAUSE TestGenerateFingerprint/basic_with_references5504=== CONT TestExternalSignerRejectsInvalidPublicKey5505=== RUN TestExternalSignerRejectsInvalidPublicKey/#005506=== CONT TestExternalSignerClose5507=== CONT TestSignErrors5508=== CONT TestNilKeyPublicKey5509--- PASS: TestNilKeyPublicKey (0.00s)5510=== CONT TestExternalSignerErrors5511=== RUN TestExternalSignerErrors/exit_at_startup5512=== PAUSE TestExternalSignerErrors/exit_at_startup5513=== RUN TestExternalSignerErrors/missing_signature_name5514=== PAUSE TestExternalSignerErrors/missing_signature_name5515=== RUN TestExternalSignerErrors/invalid_signature_base645516=== PAUSE TestExternalSignerErrors/invalid_signature_base645517=== RUN TestExternalSignerErrors/exit_on_request5518=== PAUSE TestExternalSignerErrors/exit_on_request5519=== RUN TestExternalSignerErrors/invalid_JSON5520=== PAUSE TestExternalSignerErrors/invalid_JSON5521=== RUN TestExternalSignerErrors/null5522=== PAUSE TestExternalSignerErrors/null5523=== RUN TestExternalSignerErrors/too_few_signatures5524=== PAUSE TestExternalSignerErrors/too_few_signatures5525=== RUN TestExternalSignerErrors/wrong_signature_name5526=== PAUSE TestExternalSignerErrors/wrong_signature_name5527=== RUN TestExternalSignerErrors/program_error5528=== PAUSE TestExternalSignerErrors/program_error5529=== RUN TestExternalSignerErrors/wrong_id5530=== PAUSE TestExternalSignerErrors/wrong_id5531=== RUN TestExternalSignerErrors/null_signatures5532=== PAUSE TestExternalSignerErrors/null_signatures5533=== RUN TestExternalSignerErrors/wrong_signatures_shape5534=== PAUSE TestExternalSignerErrors/wrong_signatures_shape5535=== RUN TestExternalSignerErrors/wrong_shape5536=== PAUSE TestExternalSignerErrors/wrong_shape5537=== RUN TestExternalSignerErrors/missing_signatures5538=== PAUSE TestExternalSignerErrors/missing_signatures5539=== RUN TestExternalSignerErrors/unrequested_key5540=== PAUSE TestExternalSignerErrors/unrequested_key5541=== CONT TestExternalSignerRestartsAfterCrash5542=== RUN TestExternalSignerErrors/wrong_signature_length5543=== PAUSE TestExternalSignerErrors/wrong_signature_length5544=== CONT TestExternalSignerErrors/exit_at_startup5545=== CONT TestExternalSignerErrors/program_error5546=== CONT TestExternalSignerErrors/invalid_JSON5547=== CONT TestExternalSignerErrors/wrong_shape5548=== CONT TestExternalSignerWrongPublicKey5549=== CONT TestExternalSignerErrors/exit_on_request5550=== CONT TestExternalSignerErrors/wrong_signatures_shape5551=== CONT TestExternalSignerErrors/wrong_signature_name5552=== RUN TestParseSigningKey/valid_32-byte_key_with_different_name5553=== PAUSE TestParseSigningKey/valid_32-byte_key_with_different_name5554=== CONT TestExternalSignerErrors/missing_signature_name5555=== CONT TestExternalSignerErrors/invalid_signature_base645556=== RUN TestParseSigningKey/no_colon5557=== PAUSE TestParseSigningKey/no_colon5558=== RUN TestParseSigningKey/empty_name5559=== PAUSE TestParseSigningKey/empty_name5560=== RUN TestParseSigningKey/invalid_base645561=== CONT TestExternalSignerErrors/null5562=== PAUSE TestParseSigningKey/invalid_base645563=== RUN TestParseSigningKey/wrong_length5564=== PAUSE TestExternalSignerRejectsInvalidPublicKey/#005565--- PASS: TestSign (0.00s)5566=== RUN TestGenerateFingerprint/no_references5567=== CONT TestExternalSignerErrors/unrequested_key5568=== CONT TestExternalSignerErrors/wrong_signature_length5569=== CONT TestExternalSignerErrors/too_few_signatures5570=== PAUSE TestParseSigningKey/wrong_length5571=== RUN TestExternalSignerRejectsInvalidPublicKey/no-colon5572=== PAUSE TestExternalSignerRejectsInvalidPublicKey/no-colon5573=== PAUSE TestGenerateFingerprint/no_references5574--- PASS: TestSignBatch (0.00s)5575=== RUN TestSignErrors/nil_key5576=== RUN TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5577=== PAUSE TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5578=== PAUSE TestSignErrors/nil_key5579=== RUN TestSignErrors/nil_narinfo5580=== CONT TestExternalSignerErrors/null_signatures5581=== CONT TestExternalSignerErrors/missing_signatures5582=== RUN TestGenerateFingerprint/unsorted_references_get_sorted5583=== PAUSE TestGenerateFingerprint/unsorted_references_get_sorted5584=== RUN TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5585=== PAUSE TestSignErrors/nil_narinfo5586=== RUN TestGenerateFingerprint/invalid_nar_hash_prefix5587=== PAUSE TestGenerateFingerprint/invalid_nar_hash_prefix5588=== RUN TestGenerateFingerprint/invalid_nar_hash_length5589=== PAUSE TestGenerateFingerprint/invalid_nar_hash_length5590=== RUN TestGenerateFingerprint/invalid_store_path_prefix5591=== PAUSE TestGenerateFingerprint/invalid_store_path_prefix5592=== RUN TestGenerateFingerprint/invalid_reference_prefix5593=== PAUSE TestGenerateFingerprint/invalid_reference_prefix5594=== PAUSE TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5595=== RUN TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5596=== PAUSE TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5597=== RUN TestSignErrors/invalid_narinfo5598=== PAUSE TestSignErrors/invalid_narinfo5599=== RUN TestSignErrors/canceled5600=== CONT TestExternalSignerErrors/wrong_id5601=== PAUSE TestSignErrors/canceled5602=== CONT TestParseSigningKey/empty_name5603=== CONT TestParseSigningKey/valid_32-byte_key5604=== CONT TestParseSigningKey/invalid_base645605=== CONT TestParseSigningKey/no_colon5606=== CONT TestParseSigningKey/wrong_length5607=== CONT TestGenerateFingerprint/basic_with_references5608=== CONT TestParseSigningKey/valid_32-byte_key_with_different_name5609=== CONT TestGenerateFingerprint/invalid_nar_hash_length5610--- PASS: TestParseSigningKey (0.00s)5611 --- PASS: TestParseSigningKey/empty_name (0.00s)5612 --- PASS: TestParseSigningKey/valid_32-byte_key (0.00s)5613 --- PASS: TestParseSigningKey/invalid_base64 (0.00s)5614 --- PASS: TestParseSigningKey/no_colon (0.00s)5615 --- PASS: TestParseSigningKey/wrong_length (0.00s)5616 --- PASS: TestParseSigningKey/valid_32-byte_key_with_different_name (0.00s)5617=== CONT TestGenerateFingerprint/invalid_reference_prefix5618=== CONT TestGenerateFingerprint/unsorted_references_get_sorted5619=== CONT TestGenerateFingerprint/invalid_nar_hash_prefix5620=== CONT TestGenerateFingerprint/no_references5621=== CONT TestGenerateFingerprint/invalid_store_path_prefix5622--- PASS: TestGenerateFingerprint (0.00s)5623 --- PASS: TestGenerateFingerprint/basic_with_references (0.00s)5624 --- PASS: TestGenerateFingerprint/invalid_nar_hash_length (0.00s)5625 --- PASS: TestGenerateFingerprint/invalid_reference_prefix (0.00s)5626 --- PASS: TestGenerateFingerprint/unsorted_references_get_sorted (0.00s)5627 --- PASS: TestGenerateFingerprint/invalid_nar_hash_prefix (0.00s)5628 --- PASS: TestGenerateFingerprint/no_references (0.00s)5629 --- PASS: TestGenerateFingerprint/invalid_store_path_prefix (0.00s)5630=== CONT TestExternalSignerRejectsInvalidPublicKey/#005631=== CONT TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5632=== CONT TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5633=== CONT TestExternalSignerRejectsInvalidPublicKey/no-colon5634=== CONT TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5635--- PASS: TestExternalSignerRejectsInvalidPublicKey (0.00s)5636 --- PASS: TestExternalSignerRejectsInvalidPublicKey/#00 (0.00s)5637 --- PASS: TestExternalSignerRejectsInvalidPublicKey/test-key:YQ== (0.00s)5638 --- PASS: TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= (0.00s)5639 --- PASS: TestExternalSignerRejectsInvalidPublicKey/no-colon (0.00s)5640 --- PASS: TestExternalSignerRejectsInvalidPublicKey/test-key:invalid! (0.00s)5641=== CONT TestSignErrors/nil_key5642=== CONT TestSignErrors/invalid_narinfo5643=== CONT TestSignErrors/nil_narinfo5644=== CONT TestSignErrors/canceled5645--- PASS: TestSignErrors (0.00s)5646 --- PASS: TestSignErrors/nil_key (0.00s)5647 --- PASS: TestSignErrors/invalid_narinfo (0.00s)5648 --- PASS: TestSignErrors/nil_narinfo (0.00s)5649 --- PASS: TestSignErrors/canceled (0.00s)5650--- PASS: TestExternalSignerCancellation (0.24s)5651--- PASS: TestExternalSignerWrongPublicKey (0.41s)5652--- PASS: TestExternalSignerProgramErrorKeepsProcess (2.24s)5653--- PASS: TestExternalSignerClose (2.43s)5654--- PASS: TestExternalSigner (2.86s)5655--- PASS: TestExternalSignerRestartsAfterCrash (3.64s)5656/private/var/tb/5/scratch/2e3e4b48f97c7cfb/build/TestExternalSignerRestartsAfterCrash2755658694/001/sign program: line 7: /private/var/tb/5/scratch/2e3e4b48f97c7cfb/build/TestExternalSignerRestartsAfterCrash2755658694/001/sign program.starts: No such file or directory5657--- PASS: TestExternalSignerErrors (0.00s)5658 --- PASS: TestExternalSignerErrors/wrong_id (0.60s)5659 --- PASS: TestExternalSignerErrors/null (0.79s)5660 --- PASS: TestExternalSignerErrors/wrong_signatures_shape (1.00s)5661 --- PASS: TestExternalSignerErrors/wrong_signature_name (1.24s)5662 --- PASS: TestExternalSignerErrors/program_error (1.42s)5663 --- PASS: TestExternalSignerErrors/too_few_signatures (1.62s)5664 --- PASS: TestExternalSignerErrors/unrequested_key (1.82s)5665 --- PASS: TestExternalSignerErrors/wrong_signature_length (2.03s)5666 --- PASS: TestExternalSignerErrors/exit_at_startup (2.63s)5667 --- PASS: TestExternalSignerErrors/null_signatures (3.02s)5668 --- PASS: TestExternalSignerErrors/missing_signature_name (3.22s)5669 --- PASS: TestExternalSignerErrors/exit_on_request (3.42s)5670 --- PASS: TestExternalSignerErrors/wrong_shape (3.81s)5671 --- PASS: TestExternalSignerErrors/missing_signatures (4.00s)5672 --- PASS: TestExternalSignerErrors/invalid_JSON (4.19s)5673 --- PASS: TestExternalSignerErrors/invalid_signature_base64 (4.38s)5674PASS5675Running OpenBao signer tests...5676=== RUN TestSign5677=== PAUSE TestSign5678=== RUN TestSignErrors5679=== PAUSE TestSignErrors5680=== RUN TestServe5681=== PAUSE TestServe5682=== RUN TestServeRejectsGarbage5683=== PAUSE TestServeRejectsGarbage5684=== CONT TestSign5685=== CONT TestServe5686=== CONT TestServeRejectsGarbage5687=== CONT TestSignErrors5688=== RUN TestSignErrors/no_prefix5689=== PAUSE TestSignErrors/no_prefix5690=== RUN TestSignErrors/invalid_base645691=== PAUSE TestSignErrors/invalid_base645692=== RUN TestSignErrors/http_error5693=== PAUSE TestSignErrors/http_error5694=== RUN TestSignErrors/invalid_json5695=== PAUSE TestSignErrors/invalid_json5696--- PASS: TestServeRejectsGarbage (0.00s)5697=== CONT TestSignErrors/no_prefix5698=== CONT TestSignErrors/invalid_json5699=== CONT TestSignErrors/http_error5700--- PASS: TestServe (0.00s)5701=== CONT TestSignErrors/invalid_base645702--- PASS: TestSign (0.00s)5703--- PASS: TestSignErrors (0.00s)5704 --- PASS: TestSignErrors/invalid_json (0.00s)5705 --- PASS: TestSignErrors/http_error (0.00s)5706 --- PASS: TestSignErrors/invalid_base64 (0.00s)5707 --- PASS: TestSignErrors/no_prefix (0.00s)5708PASS