niks3-go-unit-tests
checks.aarch64-darwin.go-unit-tests
· build #493
· raw
1tribuchet: building on mac022Running client tests...3=== RUN TestDoServerRequestAttachesToken4=== PAUSE TestDoServerRequestAttachesToken5=== RUN TestUploadBuildLog_FileBodyReplayedOnRetry6=== PAUSE TestUploadBuildLog_FileBodyReplayedOnRetry7=== RUN TestRegisterUploadedObjectReusesConnections8=== PAUSE TestRegisterUploadedObjectReusesConnections9=== RUN TestRunGarbageCollection_FinishedOnAnotherReplica10=== PAUSE TestRunGarbageCollection_FinishedOnAnotherReplica11=== RUN TestRunGarbageCollection_NotFoundAfterLocalRun12=== PAUSE TestRunGarbageCollection_NotFoundAfterLocalRun13=== RUN TestCaseHackSuffix14=== PAUSE TestCaseHackSuffix15=== RUN TestFilterOversizedClosures16=== PAUSE TestFilterOversizedClosures17=== RUN TestUploadMultipart_PartsInParallel18=== PAUSE TestUploadMultipart_PartsInParallel19=== RUN TestUploadMultipart_ReturnsAfterDroppedBody20=== PAUSE TestUploadMultipart_ReturnsAfterDroppedBody21=== RUN TestUploadMultipart_FailedProducerIsNotCompleted22=== PAUSE TestUploadMultipart_FailedProducerIsNotCompleted23=== RUN TestPartSizeForNAR24=== PAUSE TestPartSizeForNAR25=== RUN TestDumpPathCaseHackMatchesNix26--- PASS: TestDumpPathCaseHackMatchesNix (0.05s)27=== RUN TestDumpPathCaseHackCollision28--- PASS: TestDumpPathCaseHackCollision (0.00s)29=== RUN TestDumpPathMatchesNix30=== PAUSE TestDumpPathMatchesNix31=== RUN TestDumpPathSingleFile32=== PAUSE TestDumpPathSingleFile33=== RUN TestDumpPathWriterError34=== PAUSE TestDumpPathWriterError35=== RUN TestEncodeNixBase3236=== PAUSE TestEncodeNixBase3237=== RUN TestEncodeNixBase32WithRealHash38=== PAUSE TestEncodeNixBase32WithRealHash39=== RUN TestConvertHashToNix3240=== PAUSE TestConvertHashToNix3241=== RUN TestGetStorePathHash42=== PAUSE TestGetStorePathHash43=== RUN TestPathInfoHashCompatibility44=== PAUSE TestPathInfoHashCompatibility45=== RUN TestParsePathInfoJSON46=== PAUSE TestParsePathInfoJSON47=== RUN TestParsePathInfoJSONMultiplePaths48=== PAUSE TestParsePathInfoJSONMultiplePaths49=== RUN TestPathInfoCACompatibility50=== PAUSE TestPathInfoCACompatibility51=== RUN TestRateLimiterFeedback52=== PAUSE TestRateLimiterFeedback53=== RUN TestRateLimiterFeedback_400DoesNotCountAsSuccess54=== PAUSE TestRateLimiterFeedback_400DoesNotCountAsSuccess55=== RUN TestRegisterUploadedObject_BoundedAgainstSilentServer56=== PAUSE TestRegisterUploadedObject_BoundedAgainstSilentServer57=== RUN TestRegisterUploadedObject_OutlivesPush58=== PAUSE TestRegisterUploadedObject_OutlivesPush59=== RUN TestResolveStorePath60=== PAUSE TestResolveStorePath61=== RUN TestDoWithRetry_BodyReplayedViaGetBody62=== PAUSE TestDoWithRetry_BodyReplayedViaGetBody63=== RUN TestRetryTimeoutOutlastsMaxRetries64=== PAUSE TestRetryTimeoutOutlastsMaxRetries65=== RUN TestDoWithRetry_FinalResponseBodyReadable66=== PAUSE TestDoWithRetry_FinalResponseBodyReadable67=== RUN TestShellSplit68=== PAUSE TestShellSplit69=== RUN TestShellSplitErrors70=== PAUSE TestShellSplitErrors71=== RUN TestStreamPushReportsEveryPath72=== PAUSE TestStreamPushReportsEveryPath73=== RUN TestStreamPushBatchesUnderLoad74=== PAUSE TestStreamPushBatchesUnderLoad75=== RUN TestStreamPushIsolatesFailures76=== PAUSE TestStreamPushIsolatesFailures77=== RUN TestStreamPushGivesUpOnDeadServer78=== PAUSE TestStreamPushGivesUpOnDeadServer79=== RUN TestStreamPushRequestLine80=== PAUSE TestStreamPushRequestLine81=== RUN TestStreamPushReportsSignatures82=== PAUSE TestStreamPushReportsSignatures83=== RUN TestStreamPushReportsSkippedPaths84=== PAUSE TestStreamPushReportsSkippedPaths85=== RUN TestClientSignaturesByStorePath86=== PAUSE TestClientSignaturesByStorePath87=== RUN TestStreamPushStopsWhenCanceled88=== PAUSE TestStreamPushStopsWhenCanceled89=== RUN TestSetClientTLS90=== PAUSE TestSetClientTLS91=== RUN TestSetClientTLSDoesNotMutateDefaultTransport92=== PAUSE TestSetClientTLSDoesNotMutateDefaultTransport93=== RUN TestSetClientTLSErrors94=== PAUSE TestSetClientTLSErrors95=== RUN TestStaticToken96=== PAUSE TestStaticToken97=== RUN TestFileTokenReadsAndCaches98=== PAUSE TestFileTokenReadsAndCaches99=== RUN TestFileTokenMissing100=== PAUSE TestFileTokenMissing101=== RUN TestFileTokenEmpty102=== PAUSE TestFileTokenEmpty103=== RUN TestScriptTokenNoExpiryRerunsEveryCall104=== PAUSE TestScriptTokenNoExpiryRerunsEveryCall105=== RUN TestScriptTokenCachesUntilRefresh106=== PAUSE TestScriptTokenCachesUntilRefresh107=== RUN TestScriptTokenEmptyToken108=== PAUSE TestScriptTokenEmptyToken109=== RUN TestScriptTokenBadJSON110=== PAUSE TestScriptTokenBadJSON111=== RUN TestScriptTokenScriptFails112=== PAUSE TestScriptTokenScriptFails113=== RUN TestScriptTokenEmptyCommand114=== PAUSE TestScriptTokenEmptyCommand115=== CONT TestDoServerRequestAttachesToken116=== CONT TestDoWithRetry_BodyReplayedViaGetBody117=== CONT TestStreamPushStopsWhenCanceled118=== CONT TestEncodeNixBase32119=== RUN TestEncodeNixBase32/test_string_hash120=== PAUSE TestEncodeNixBase32/test_string_hash121=== RUN TestEncodeNixBase32/empty_input122=== PAUSE TestEncodeNixBase32/empty_input123=== CONT TestEncodeNixBase32/test_string_hash124=== CONT TestDumpPathMatchesNix125=== CONT TestUploadMultipart_PartsInParallel126=== CONT TestResolveStorePath127=== CONT TestScriptTokenEmptyCommand128=== CONT TestRateLimiterFeedback129=== CONT TestScriptTokenScriptFails130=== CONT TestScriptTokenBadJSON131=== CONT TestScriptTokenEmptyToken132=== CONT TestScriptTokenCachesUntilRefresh133=== CONT TestScriptTokenNoExpiryRerunsEveryCall134=== CONT TestFileTokenEmpty135=== CONT TestFileTokenMissing136=== CONT TestFileTokenReadsAndCaches137=== CONT TestStaticToken138=== CONT TestSetClientTLSErrors139=== CONT TestSetClientTLSDoesNotMutateDefaultTransport140=== CONT TestStreamPushIsolatesFailures141=== CONT TestDumpPathWriterError142=== CONT TestDumpPathSingleFile143=== CONT TestRegisterUploadedObject_OutlivesPush144=== CONT TestRegisterUploadedObject_BoundedAgainstSilentServer145--- PASS: TestStreamPushStopsWhenCanceled (0.00s)146--- PASS: TestScriptTokenEmptyCommand (0.00s)147--- PASS: TestResolveStorePath (0.00s)148--- PASS: TestScriptTokenScriptFails (0.00s)149=== CONT TestRateLimiterFeedback_400DoesNotCountAsSuccess150=== CONT TestSetClientTLS151=== RUN TestRateLimiterFeedback/429_enables_limiter152=== PAUSE TestRateLimiterFeedback/429_enables_limiter153=== RUN TestRateLimiterFeedback/503_enables_limiter154=== PAUSE TestRateLimiterFeedback/503_enables_limiter155=== RUN TestRateLimiterFeedback/200_does_not_enable_limiter156=== PAUSE TestRateLimiterFeedback/200_does_not_enable_limiter157=== RUN TestRateLimiterFeedback/400_does_not_enable_limiter158=== PAUSE TestRateLimiterFeedback/400_does_not_enable_limiter159=== CONT TestPathInfoCACompatibility160=== RUN TestPathInfoCACompatibility/null_ca_field161=== PAUSE TestPathInfoCACompatibility/null_ca_field162=== RUN TestPathInfoCACompatibility/old_string_format_-_text163=== PAUSE TestPathInfoCACompatibility/old_string_format_-_text164=== RUN TestPathInfoCACompatibility/old_string_format_-_fixed_recursive165=== PAUSE TestPathInfoCACompatibility/old_string_format_-_fixed_recursive166=== RUN TestPathInfoCACompatibility/new_structured_format_-_text167=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_text168=== RUN TestPathInfoCACompatibility/new_structured_format_-_nar_method169=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_nar_method170=== CONT TestParsePathInfoJSON171=== RUN TestParsePathInfoJSON/Nix_format172=== PAUSE TestParsePathInfoJSON/Nix_format173=== RUN TestParsePathInfoJSON/Lix_format174=== PAUSE TestParsePathInfoJSON/Lix_format175=== RUN TestParsePathInfoJSON/empty_input176=== PAUSE TestParsePathInfoJSON/empty_input177=== RUN TestParsePathInfoJSON/whitespace_only178=== PAUSE TestParsePathInfoJSON/whitespace_only179=== RUN TestParsePathInfoJSON/invalid_JSON180=== PAUSE TestParsePathInfoJSON/invalid_JSON181=== CONT TestParsePathInfoJSONMultiplePaths182=== RUN TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths183=== PAUSE TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths184=== RUN TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths185=== PAUSE TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths186=== CONT TestGetStorePathHash187=== RUN TestGetStorePathHash/valid_store_path188=== PAUSE TestGetStorePathHash/valid_store_path189=== CONT TestPathInfoHashCompatibility190=== RUN TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)191=== RUN TestGetStorePathHash/basename_without_hyphen_should_error192=== PAUSE TestGetStorePathHash/basename_without_hyphen_should_error1932026/11/11 23:58:08 WARN Rate limiter enabled after throttle name=server-test rate=5194=== CONT TestConvertHashToNix32195=== RUN TestConvertHashToNix32/SRI_format_to_Nix32196=== PAUSE TestConvertHashToNix32/SRI_format_to_Nix32197=== RUN TestConvertHashToNix32/already_Nix32_format198=== PAUSE TestConvertHashToNix32/already_Nix32_format199=== RUN TestConvertHashToNix32/invalid_format200=== PAUSE TestConvertHashToNix32/invalid_format201=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)202=== RUN TestPathInfoHashCompatibility/old_string_format_with_colon203=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_colon204=== RUN TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI205=== PAUSE TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI206=== RUN TestPathInfoHashCompatibility/new_structured_format_with_sha512207=== PAUSE TestPathInfoHashCompatibility/new_structured_format_with_sha512208=== CONT TestEncodeNixBase32WithRealHash209--- PASS: TestEncodeNixBase32WithRealHash (0.00s)210=== CONT TestClientSignaturesByStorePath211--- PASS: TestClientSignaturesByStorePath (0.00s)212=== CONT TestUploadMultipart_FailedProducerIsNotCompleted213=== RUN TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part214=== PAUSE TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part215=== RUN TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary216=== PAUSE TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary217=== RUN TestGetStorePathHash/hash_with_invalid_characters_should_error218=== PAUSE TestGetStorePathHash/hash_with_invalid_characters_should_error219=== RUN TestGetStorePathHash/hash_with_wrong_length_should_error220=== PAUSE TestGetStorePathHash/hash_with_wrong_length_should_error221=== CONT TestEncodeNixBase32/empty_input222--- PASS: TestEncodeNixBase32 (0.00s)223 --- PASS: TestEncodeNixBase32/test_string_hash (0.00s)224 --- PASS: TestEncodeNixBase32/empty_input (0.00s)225=== CONT TestPartSizeForNAR226=== RUN TestPartSizeForNAR/zero_stays_at_minimum227=== PAUSE TestPartSizeForNAR/zero_stays_at_minimum228=== RUN TestPartSizeForNAR/small_stays_at_minimum229=== PAUSE TestPartSizeForNAR/small_stays_at_minimum230=== RUN TestPartSizeForNAR/80_GiB_fits_at_minimum231=== PAUSE TestPartSizeForNAR/80_GiB_fits_at_minimum232=== RUN TestPartSizeForNAR/115_GiB_needs_larger_parts233=== PAUSE TestPartSizeForNAR/115_GiB_needs_larger_parts234=== RUN TestPartSizeForNAR/1_TiB235=== PAUSE TestPartSizeForNAR/1_TiB236=== RUN TestPartSizeForNAR/5_TiB_S3_max_object237=== PAUSE TestPartSizeForNAR/5_TiB_S3_max_object238=== RUN TestPartSizeForNAR/capped_at_5_GiB239=== PAUSE TestPartSizeForNAR/capped_at_5_GiB240=== CONT TestStreamPushGivesUpOnDeadServer241=== CONT TestStreamPushRequestLine242=== CONT TestUploadMultipart_ReturnsAfterDroppedBody243--- PASS: TestFileTokenEmpty (0.00s)244=== CONT TestStreamPushReportsSkippedPaths245--- PASS: TestStreamPushReportsSkippedPaths (0.00s)246=== CONT TestShellSplitErrors247--- PASS: TestShellSplitErrors (0.00s)248=== CONT TestStreamPushBatchesUnderLoad249=== RUN TestStreamPushBatchesUnderLoad/oneAtATime=false250=== PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=false251=== CONT TestRunGarbageCollection_NotFoundAfterLocalRun252=== RUN TestStreamPushBatchesUnderLoad/oneAtATime=true253=== PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=true254=== CONT TestStreamPushReportsEveryPath255=== CONT TestStreamPushReportsSignatures256--- PASS: TestFileTokenReadsAndCaches (0.00s)257--- PASS: TestStaticToken (0.00s)258--- PASS: TestStreamPushReportsEveryPath (0.00s)259=== CONT TestFilterOversizedClosures260=== RUN TestFilterOversizedClosures/no_limit_keeps_everything261=== PAUSE TestFilterOversizedClosures/no_limit_keeps_everything262=== RUN TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped263=== PAUSE TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped264=== RUN TestFilterOversizedClosures/all_closures_skipped265=== PAUSE TestFilterOversizedClosures/all_closures_skipped266=== CONT TestCaseHackSuffix2672026/11/11 23:58:08 WARN Rate limiter enabled after throttle name=server-test rate=52682026/11/11 23:58:08 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:61125269--- PASS: TestFileTokenMissing (0.00s)270=== CONT TestDoWithRetry_FinalResponseBodyReadable2712026/11/11 23:58:08 ERROR Upload failed error="connection refused" count=202722026/11/11 23:58:08 ERROR Server seems unavailable, giving up on batch untried=172732026/11/11 23:58:08 ERROR Upload failed error=boom count=12742026/11/11 23:58:08 ERROR Upload failed error=boom count=12752026/11/11 23:58:08 ERROR Upload failed error="bad path" count=3276--- PASS: TestStreamPushIsolatesFailures (0.00s)277=== CONT TestShellSplit278--- PASS: TestShellSplit (0.00s)279=== CONT TestRegisterUploadedObjectReusesConnections280--- PASS: TestDoServerRequestAttachesToken (0.01s)281=== CONT TestUploadBuildLog_FileBodyReplayedOnRetry282--- PASS: TestStreamPushReportsSignatures (0.00s)283=== RUN TestSetClientTLS/rejects_connection_without_client_cert284=== PAUSE TestSetClientTLS/rejects_connection_without_client_cert285=== RUN TestSetClientTLS/succeeds_with_client_cert_and_CA286=== PAUSE TestSetClientTLS/succeeds_with_client_cert_and_CA287=== RUN TestSetClientTLS/preserves_debug_logging_transport288=== PAUSE TestSetClientTLS/preserves_debug_logging_transport289=== CONT TestRetryTimeoutOutlastsMaxRetries290--- PASS: TestUploadMultipart_ReturnsAfterDroppedBody (0.00s)291=== CONT TestRateLimiterFeedback/429_enables_limiter292=== CONT TestRunGarbageCollection_FinishedOnAnotherReplica2932026/11/11 23:58:08 WARN Rate limiter backed off name=server-test rate=52942026/11/11 23:58:08 WARN Request returned retryable status, retrying attempt=2 backoff=0s status=503 url=http://127.0.0.1:611252952026/11/11 23:58:08 INFO Garbage collection started296--- PASS: TestStreamPushGivesUpOnDeadServer (0.00s)297=== CONT TestPathInfoCACompatibility/null_ca_field298--- PASS: TestDoWithRetry_BodyReplayedViaGetBody (0.01s)299=== CONT TestParsePathInfoJSON/Nix_format300=== CONT TestRateLimiterFeedback/200_does_not_enable_limiter301=== CONT TestRateLimiterFeedback/400_does_not_enable_limiter3022026/11/11 23:58:08 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:611493032026/11/11 23:58:08 WARN Rate limiter enabled after throttle name=server-test rate=53042026/11/11 23:58:08 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=429 url=http://127.0.0.1:611523052026/11/11 23:58:08 WARN Rate limiter enabled after throttle name=server-test rate=53062026/11/11 23:58:08 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:611393072026/11/11 23:58:08 WARN Rate limiter enabled after throttle name=s3-test rate=53082026/11/11 23:58:08 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:611503092026/11/11 23:58:08 WARN Rate limiter backed off name=server-test rate=53102026/11/11 23:58:08 INFO Garbage collection started311=== CONT TestRateLimiterFeedback/503_enables_limiter3122026/11/11 23:58:08 WARN Rate limiter backed off name=server-test rate=5313--- PASS: TestDoWithRetry_FinalResponseBodyReadable (0.00s)314=== CONT TestPathInfoCACompatibility/old_string_format_-_text315=== CONT TestPathInfoCACompatibility/new_structured_format_-_text316=== CONT TestPathInfoCACompatibility/new_structured_format_-_nar_method317=== CONT TestPathInfoCACompatibility/old_string_format_-_fixed_recursive318--- PASS: TestPathInfoCACompatibility (0.00s)319 --- PASS: TestPathInfoCACompatibility/null_ca_field (0.00s)320 --- PASS: TestPathInfoCACompatibility/old_string_format_-_text (0.00s)321 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_text (0.00s)322 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_nar_method (0.00s)323 --- PASS: TestPathInfoCACompatibility/old_string_format_-_fixed_recursive (0.00s)324=== CONT TestParsePathInfoJSON/whitespace_only325=== CONT TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths326=== CONT TestParsePathInfoJSON/empty_input327=== CONT TestParsePathInfoJSON/Lix_format328--- PASS: TestRunGarbageCollection_NotFoundAfterLocalRun (0.01s)329=== CONT TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths330--- PASS: TestParsePathInfoJSONMultiplePaths (0.00s)331 --- PASS: TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths (0.00s)332 --- PASS: TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths (0.00s)333=== CONT TestParsePathInfoJSON/invalid_JSON334=== CONT TestConvertHashToNix32/SRI_format_to_Nix32335=== CONT TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part336--- PASS: TestParsePathInfoJSON (0.00s)337 --- PASS: TestParsePathInfoJSON/Nix_format (0.00s)338 --- PASS: TestParsePathInfoJSON/whitespace_only (0.00s)339 --- PASS: TestParsePathInfoJSON/empty_input (0.00s)340 --- PASS: TestParsePathInfoJSON/Lix_format (0.00s)341 --- PASS: TestParsePathInfoJSON/invalid_JSON (0.00s)342=== CONT TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)343=== CONT TestConvertHashToNix32/invalid_format344=== CONT TestConvertHashToNix32/already_Nix32_format345--- PASS: TestConvertHashToNix32 (0.00s)346 --- PASS: TestConvertHashToNix32/SRI_format_to_Nix32 (0.00s)347 --- PASS: TestConvertHashToNix32/invalid_format (0.00s)348 --- PASS: TestConvertHashToNix32/already_Nix32_format (0.00s)349=== CONT TestGetStorePathHash/valid_store_path350=== CONT TestPathInfoHashCompatibility/new_structured_format_with_sha512351=== CONT TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI352=== CONT TestPathInfoHashCompatibility/old_string_format_with_colon353--- PASS: TestPathInfoHashCompatibility (0.00s)354 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI) (0.00s)355 --- PASS: TestPathInfoHashCompatibility/new_structured_format_with_sha512 (0.00s)356 --- PASS: TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI (0.00s)357 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_colon (0.00s)358=== CONT TestPartSizeForNAR/zero_stays_at_minimum359=== CONT TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary3602026/11/11 23:58:08 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:61149361--- PASS: TestScriptTokenBadJSON (0.01s)362=== CONT TestGetStorePathHash/hash_with_wrong_length_should_error363=== CONT TestGetStorePathHash/hash_with_invalid_characters_should_error364=== CONT TestGetStorePathHash/basename_without_hyphen_should_error365--- PASS: TestGetStorePathHash (0.00s)366 --- PASS: TestGetStorePathHash/valid_store_path (0.00s)367 --- PASS: TestGetStorePathHash/hash_with_wrong_length_should_error (0.00s)368 --- PASS: TestGetStorePathHash/hash_with_invalid_characters_should_error (0.00s)369 --- PASS: TestGetStorePathHash/basename_without_hyphen_should_error (0.00s)370=== CONT TestPartSizeForNAR/capped_at_5_GiB371=== CONT TestPartSizeForNAR/5_TiB_S3_max_object372=== CONT TestPartSizeForNAR/1_TiB373=== CONT TestPartSizeForNAR/115_GiB_needs_larger_parts374=== CONT TestPartSizeForNAR/80_GiB_fits_at_minimum375=== CONT TestPartSizeForNAR/small_stays_at_minimum376--- PASS: TestPartSizeForNAR (0.00s)377 --- PASS: TestPartSizeForNAR/zero_stays_at_minimum (0.00s)378 --- PASS: TestPartSizeForNAR/capped_at_5_GiB (0.00s)379 --- PASS: TestPartSizeForNAR/5_TiB_S3_max_object (0.00s)380 --- PASS: TestPartSizeForNAR/1_TiB (0.00s)381 --- PASS: TestPartSizeForNAR/115_GiB_needs_larger_parts (0.00s)382 --- PASS: TestPartSizeForNAR/80_GiB_fits_at_minimum (0.00s)383 --- PASS: TestPartSizeForNAR/small_stays_at_minimum (0.00s)384=== CONT TestStreamPushBatchesUnderLoad/oneAtATime=false385=== CONT TestStreamPushBatchesUnderLoad/oneAtATime=true386--- PASS: TestStreamPushBatchesUnderLoad (0.00s)387 --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=false (0.00s)388 --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=true (0.00s)389=== CONT TestFilterOversizedClosures/no_limit_keeps_everything390=== CONT TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped3912026/11/11 23:58:08 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb-vm-image nar_size=5000 max_nar_size=2000392=== CONT TestFilterOversizedClosures/all_closures_skipped3932026/11/11 23:58:08 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper nar_size=100 max_nar_size=50394--- PASS: TestFilterOversizedClosures (0.00s)395 --- PASS: TestFilterOversizedClosures/no_limit_keeps_everything (0.00s)396 --- PASS: TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped (0.00s)397 --- PASS: TestFilterOversizedClosures/all_closures_skipped (0.00s)398=== CONT TestSetClientTLS/rejects_connection_without_client_cert3992026/11/11 23:58:08 INFO Garbage collection progress phase=running_on_another_replica failed_uploads_deleted=0 old_closures_deleted=0 objects_marked=0 objects_deleted=0 objects_failed=0400--- PASS: TestSetClientTLSDoesNotMutateDefaultTransport (0.01s)401=== CONT TestSetClientTLS/preserves_debug_logging_transport402--- PASS: TestScriptTokenEmptyToken (0.01s)403=== CONT TestSetClientTLS/succeeds_with_client_cert_and_CA404--- PASS: TestRegisterUploadedObject_OutlivesPush (0.01s)405--- PASS: TestUploadBuildLog_FileBodyReplayedOnRetry (0.01s)4062026/11/11 23:58:08 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:61149407=== RUN TestSetClientTLSErrors/missing_cert_file408=== PAUSE TestSetClientTLSErrors/missing_cert_file409=== RUN TestSetClientTLSErrors/missing_key_file410=== PAUSE TestSetClientTLSErrors/missing_key_file411=== RUN TestSetClientTLSErrors/missing_ca_file412=== PAUSE TestSetClientTLSErrors/missing_ca_file413=== RUN TestSetClientTLSErrors/invalid_ca_file414=== PAUSE TestSetClientTLSErrors/invalid_ca_file415=== CONT TestSetClientTLSErrors/missing_cert_file416=== CONT TestSetClientTLSErrors/invalid_ca_file417=== CONT TestSetClientTLSErrors/missing_ca_file418=== CONT TestSetClientTLSErrors/missing_key_file4192026/11/11 23:58:09 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:611494202026/11/11 23:58:09 WARN Rate limiter enabled after throttle name=server-test rate=54212026/11/11 23:58:09 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:612254222026/11/11 23:58:09 WARN Garbage collection finished on another replica. Its statistics are not available here423--- PASS: TestRunGarbageCollection_FinishedOnAnotherReplica (0.01s)4242026/11/11 23:58:09 WARN Rate limiter backed off name=server-test rate=5425--- PASS: TestUploadMultipart_FailedProducerIsNotCompleted (0.00s)426 --- PASS: TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part (0.00s)427 --- PASS: TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary (0.01s)428--- PASS: TestSetClientTLSErrors (0.01s)429 --- PASS: TestSetClientTLSErrors/missing_cert_file (0.00s)430 --- PASS: TestSetClientTLSErrors/invalid_ca_file (0.00s)431 --- PASS: TestSetClientTLSErrors/missing_key_file (0.00s)432 --- PASS: TestSetClientTLSErrors/missing_ca_file (0.00s)4332026/11/11 23:58:09 WARN Request returned retryable status, retrying attempt=3 backoff=1ms status=502 url=http://127.0.0.1:61149434--- PASS: TestRateLimiterFeedback (0.01s)435 --- PASS: TestRateLimiterFeedback/429_enables_limiter (0.00s)436 --- PASS: TestRateLimiterFeedback/200_does_not_enable_limiter (0.00s)437 --- PASS: TestRateLimiterFeedback/400_does_not_enable_limiter (0.00s)438 --- PASS: TestRateLimiterFeedback/503_enables_limiter (0.01s)4392026/11/11 23:58:09 WARN Request returned retryable status, retrying attempt=4 backoff=1ms status=502 url=http://127.0.0.1:61149440--- PASS: TestStreamPushRequestLine (0.02s)4412026/11/11 23:58:09 WARN Request returned retryable status, retrying attempt=5 backoff=1ms status=502 url=http://127.0.0.1:611494422026/11/11 23:58:09 WARN Request returned retryable status, retrying attempt=6 backoff=1ms status=502 url=http://127.0.0.1:611494432026/11/11 23:58:09 WARN Request returned retryable status, retrying attempt=7 backoff=1ms status=502 url=http://127.0.0.1:611494442026/11/11 23:58:09 WARN Request returned retryable status, retrying attempt=8 backoff=1ms status=502 url=http://127.0.0.1:611494452026/11/11 23:58:09 WARN Request returned retryable status, retrying attempt=9 backoff=1ms status=502 url=http://127.0.0.1:611494462026/11/11 23:58:09 WARN Request returned retryable status, retrying attempt=10 backoff=1ms status=502 url=http://127.0.0.1:611494472026/11/11 23:58:09 WARN Request returned retryable status, retrying attempt=11 backoff=1ms status=502 url=http://127.0.0.1:611494482026/11/11 23:58:09 http: TLS handshake error from 127.0.0.1:61229: remote error: tls: bad certificate449--- PASS: TestSetClientTLS (0.01s)450 --- PASS: TestSetClientTLS/preserves_debug_logging_transport (0.01s)451 --- PASS: TestSetClientTLS/succeeds_with_client_cert_and_CA (0.01s)452 --- PASS: TestSetClientTLS/rejects_connection_without_client_cert (0.02s)4532026/11/11 23:58:09 WARN Request returned retryable status, retrying attempt=12 backoff=1ms status=502 url=http://127.0.0.1:611494542026/11/11 23:58:09 WARN Request returned retryable status, retrying attempt=13 backoff=1ms status=502 url=http://127.0.0.1:611494552026/11/11 23:58:09 WARN Request returned retryable status, retrying attempt=14 backoff=1ms status=502 url=http://127.0.0.1:61149456--- PASS: TestRegisterUploadedObjectReusesConnections (0.04s)4572026/11/11 23:58:09 WARN Request returned retryable status, retrying attempt=15 backoff=1ms status=502 url=http://127.0.0.1:611494582026/11/11 23:58:09 WARN Request returned retryable status, retrying attempt=16 backoff=1ms status=502 url=http://127.0.0.1:611494592026/11/11 23:58:09 WARN Request returned retryable status, retrying attempt=17 backoff=1ms status=502 url=http://127.0.0.1:61149460--- PASS: TestRetryTimeoutOutlastsMaxRetries (0.04s)461--- PASS: TestDumpPathWriterError (0.05s)462--- PASS: TestScriptTokenNoExpiryRerunsEveryCall (0.05s)463--- PASS: TestScriptTokenCachesUntilRefresh (0.06s)464--- PASS: TestCaseHackSuffix (0.05s)465--- PASS: TestDumpPathSingleFile (0.06s)466--- PASS: TestDumpPathMatchesNix (0.08s)4672026/11/11 23:58:09 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:61128/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:61128/api/uploads/complete4682026/11/11 23:58:09 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"4692026/11/11 23:58:09 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:61128/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:61128/api/uploads/complete4702026/11/11 23:58:09 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"4712026/11/11 23:58:09 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:61128/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:61128/api/uploads/complete4722026/11/11 23:58:09 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"473--- PASS: TestRegisterUploadedObject_BoundedAgainstSilentServer (0.10s)474--- PASS: TestUploadMultipart_PartsInParallel (0.61s)475--- PASS: TestRateLimiterFeedback_400DoesNotCountAsSuccess (1.00s)476PASS477Running server tests...478The files belonging to this database system will be owned by user "_tribuchetbld5".479This user must also own the server process.480481The database cluster will be initialized with locale "C".482The default database encoding has accordingly been set to "SQL_ASCII".483The default text search configuration will be set to "english".484485Data page checksums are enabled.486487creating directory /private/var/tb/5/scratch/67d206d5acc7e9e5/build/postgres1860836163/data ... ok488creating subdirectories ... ok489selecting dynamic shared memory implementation ... posix490selecting default "max_connections" ... 100491selecting default "shared_buffers" ... 128MB492selecting default time zone ... UTC493creating configuration files ... ok494running bootstrap script ... ok495performing post-bootstrap initialization ... ok496syncing data to disk ... ok497498initdb: warning: enabling "trust" authentication for local connections499initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.500501Success. You can now start the database server using:502503 pg_ctl -D /private/var/tb/5/scratch/67d206d5acc7e9e5/build/postgres1860836163/data -l logfile start504505/private/var/tb/5/scratch/67d206d5acc7e9e5/build/postgres1860836163:5432 - no response5062026-11-11 23:58:10.746 UTC [99810] LOG: starting PostgreSQL 18.6 on aarch64-apple-darwin25.6.0, compiled by clang version 21.1.8, 64-bit5072026-11-11 23:58:10.747 UTC [99810] LOG: listening on Unix socket "/private/var/tb/5/scratch/67d206d5acc7e9e5/build/postgres1860836163/.s.PGSQL.5432"5082026-11-11 23:58:10.749 UTC [99817] LOG: database system was shut down at 2026-11-11 23:58:10 UTC5092026-11-11 23:58:10.750 UTC [99810] LOG: database system is ready to accept connections510/private/var/tb/5/scratch/67d206d5acc7e9e5/build/postgres1860836163:5432 - accepting connections511{"timestamp":"2026-11-11T23:58:10.965101Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"58954978-ef18-4300-867a-e2e0abf619c6","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"GET","uri":"/health/ready","status_code":503,"suppressed_errors":0,"duration_ms":0,"result":"server_error","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":502,"threadName":"rustfs-worker","threadId":"ThreadId(4)"}512=== RUN TestAbortSucceeded513=== PAUSE TestAbortSucceeded514=== RUN TestService_AuthMiddleware515=== PAUSE TestService_AuthMiddleware516=== RUN TestService_AuthMiddleware_MTLSProxyHeader517=== PAUSE TestService_AuthMiddleware_MTLSProxyHeader518=== RUN TestService_AuthMiddleware_MTLSBoundSubjects519=== PAUSE TestService_AuthMiddleware_MTLSBoundSubjects520=== RUN TestService_ReadAuthMiddleware521=== PAUSE TestService_ReadAuthMiddleware522=== RUN TestService_AuthMiddleware_OIDC523=== PAUSE TestService_AuthMiddleware_OIDC524=== RUN TestService_RequireScope_OIDC525=== PAUSE TestService_RequireScope_OIDC526=== RUN TestService_ReadScope_PublicByDefault527=== PAUSE TestService_ReadScope_PublicByDefault528=== RUN TestCacheConfigHandler529=== PAUSE TestCacheConfigHandler530=== RUN TestCacheStatsHandler531=== PAUSE TestCacheStatsHandler532=== RUN TestClientCADerivations533=== PAUSE TestClientCADerivations534=== RUN TestClientErrorHandling535=== PAUSE TestClientErrorHandling536=== RUN TestClientIntegration537=== PAUSE TestClientIntegration538=== RUN TestClientMultipleUploads539=== PAUSE TestClientMultipleUploads540=== RUN TestClientWithDependencies541=== PAUSE TestClientWithDependencies542=== RUN TestClientSharedPathCommittedMidPush543=== PAUSE TestClientSharedPathCommittedMidPush544=== RUN TestPinProtectsFromGC545=== PAUSE TestPinProtectsFromGC546=== RUN TestClientPushesUseOnePush547=== PAUSE TestClientPushesUseOnePush548=== RUN TestClientFallsBackToClosures549=== PAUSE TestClientFallsBackToClosures550=== RUN TestDeleteClosuresRacingPins551=== PAUSE TestDeleteClosuresRacingPins552=== RUN TestConcurrentCommitsSharingObjectsDoNotDeadlock553=== PAUSE TestConcurrentCommitsSharingObjectsDoNotDeadlock554=== RUN TestResolveDBConnectionString555=== PAUSE TestResolveDBConnectionString556=== RUN TestConnectWaitsForAPeerMigration557=== PAUSE TestConnectWaitsForAPeerMigration558=== RUN TestLeadElectsOneAndHandsOver559=== PAUSE TestLeadElectsOneAndHandsOver560=== RUN TestLeadIncumbentWinsAfterRestart5612026/11/11 23:58:11 INFO lead: acquired remote=192.0.2.1:1234 generation=15622026/11/11 23:58:11 INFO lead: released remote=192.0.2.1:1234 generation=15632026/11/11 23:58:11 INFO lead: acquired remote=192.0.2.1:1234 generation=25642026/11/11 23:58:13 INFO lead: released remote=192.0.2.1:1234 generation=2565--- PASS: TestLeadIncumbentWinsAfterRestart (1.95s)566=== RUN TestLeadUnannouncedStandbyYieldsToIncumbent567=== PAUSE TestLeadUnannouncedStandbyYieldsToIncumbent568=== RUN TestLeadIncumbentAnnouncesAtOnce569=== PAUSE TestLeadIncumbentAnnouncesAtOnce570=== RUN TestLeadGenerationGrowsWithEveryGrant571=== PAUSE TestLeadGenerationGrowsWithEveryGrant572=== RUN TestLeadEndsOnShutdown573=== PAUSE TestLeadEndsOnShutdown574=== RUN TestLeadEndsWhenItsConnectionHangs5752026/11/11 23:58:13 INFO lead: acquired remote=192.0.2.1:1234 generation=15762026-11-11 23:58:14.569 UTC [99868] FATAL: terminating connection due to administrator command5772026/11/11 23:58:14 INFO lead: acquired remote=192.0.2.1:1234 generation=25782026/11/11 23:58:15 WARN lead: lock connection lost error="timeout: context deadline exceeded"5792026/11/11 23:58:15 INFO lead: released remote=192.0.2.1:1234 generation=15802026/11/11 23:58:15 INFO lead: released remote=192.0.2.1:1234 generation=2581--- PASS: TestLeadEndsWhenItsConnectionHangs (2.94s)582=== RUN TestGCAdvisoryLockBlocksConcurrentRun583=== PAUSE TestGCAdvisoryLockBlocksConcurrentRun584=== RUN TestGCBugBareHashReferences585=== PAUSE TestGCBugBareHashReferences586=== RUN TestGCMetrics587=== PAUSE TestGCMetrics588=== RUN TestPushDedupSurvivesConcurrentGC589=== PAUSE TestPushDedupSurvivesConcurrentGC590=== RUN TestDeduplicatedObjectsRecordedAsPending591=== PAUSE TestDeduplicatedObjectsRecordedAsPending592=== RUN TestGCSweepSkipsPendingObjects593=== PAUSE TestGCSweepSkipsPendingObjects594=== RUN TestTombstonedObjectOfferedWithoutWaiting595=== PAUSE TestTombstonedObjectOfferedWithoutWaiting596=== RUN TestGCSweepDeliversEachKeyOnce597=== PAUSE TestGCSweepDeliversEachKeyOnce598=== RUN TestCreatePendingClosureVerifyS3FailureReleasesConnection599=== PAUSE TestCreatePendingClosureVerifyS3FailureReleasesConnection600=== RUN TestForceGCDuringPushOffersSweptObject601=== PAUSE TestForceGCDuringPushOffersSweptObject602=== RUN TestFailedS3DeleteKeepsTombstone603=== PAUSE TestFailedS3DeleteKeepsTombstone604=== RUN TestSweepSkipsRowsLockedByPush605=== PAUSE TestSweepSkipsRowsLockedByPush606=== RUN TestRegisterCompletedObjectNeedsPendingRow607=== PAUSE TestRegisterCompletedObjectNeedsPendingRow608=== RUN TestGetExistingObjectsLocksRows609=== PAUSE TestGetExistingObjectsLocksRows610=== RUN TestSweepRechecksPendingAfterLocking611=== PAUSE TestSweepRechecksPendingAfterLocking612=== RUN TestPushWaitsForSweepHoldingRow613=== PAUSE TestPushWaitsForSweepHoldingRow614=== RUN TestCommitRacingPendingCleanupKeepsObjects615=== PAUSE TestCommitRacingPendingCleanupKeepsObjects616=== RUN TestGCEndsOnShutdown617=== PAUSE TestGCEndsOnShutdown618=== RUN TestGCTaskStore_StartNew619=== PAUSE TestGCTaskStore_StartNew620=== RUN TestGCTaskStore_DeduplicateSameParams621=== PAUSE TestGCTaskStore_DeduplicateSameParams622=== RUN TestGCTaskStore_ConflictDifferentParams623=== PAUSE TestGCTaskStore_ConflictDifferentParams624=== RUN TestGCTaskStore_GetEmpty625=== PAUSE TestGCTaskStore_GetEmpty626=== RUN TestGCTaskStore_GetReturnsLatest627=== PAUSE TestGCTaskStore_GetReturnsLatest628=== RUN TestGCTaskStore_CompletedAllowsNewTask629=== PAUSE TestGCTaskStore_CompletedAllowsNewTask630=== RUN TestGCTaskStore_PhaseUpdates631=== PAUSE TestGCTaskStore_PhaseUpdates632=== RUN TestGCTaskStore_Fail633=== PAUSE TestGCTaskStore_Fail634=== RUN TestGracefulShutdownDrainsInflight635=== PAUSE TestGracefulShutdownDrainsInflight636=== RUN TestService_healthCheckHandler637=== PAUSE TestService_healthCheckHandler638=== RUN TestService_readinessHandler639=== PAUSE TestService_readinessHandler640=== RUN TestGenerateLandingPage641=== PAUSE TestGenerateLandingPage642=== RUN TestEnsureAbortUploadsRule643=== PAUSE TestEnsureAbortUploadsRule644=== RUN TestCacheConfigHandlerMaxNarSize645=== PAUSE TestCacheConfigHandlerMaxNarSize646=== RUN TestCreatePendingClosureRejectsOversizedNAR647=== PAUSE TestCreatePendingClosureRejectsOversizedNAR648=== RUN TestNARDeduplicationMetadataUploadBug649=== PAUSE TestNARDeduplicationMetadataUploadBug650=== RUN TestMetricsInventory651=== PAUSE TestMetricsInventory652=== RUN TestMetricsCommit653=== PAUSE TestMetricsCommit654=== RUN TestService_NativeMTLS655=== PAUSE TestService_NativeMTLS656=== RUN TestServerTLSConfig657=== PAUSE TestServerTLSConfig658=== RUN TestMultipartCleanup659=== PAUSE TestMultipartCleanup660=== RUN TestMultipartUploadAbortedWhenCancelledBeforeRecorded661=== PAUSE TestMultipartUploadAbortedWhenCancelledBeforeRecorded662=== RUN TestPendingCleanupUsesOneCutoff663=== PAUSE TestPendingCleanupUsesOneCutoff664=== RUN TestObjectStatsTrigger665=== PAUSE TestObjectStatsTrigger666=== RUN TestObjectStatsUpdatedOncePerCommit667=== PAUSE TestObjectStatsUpdatedOncePerCommit668=== RUN TestOrphanedObjectsGC669=== PAUSE TestOrphanedObjectsGC670=== RUN TestOrphanedObjectsGCStressTest671=== PAUSE TestOrphanedObjectsGCStressTest672=== RUN TestCreatePin_ReservedPins673=== PAUSE TestCreatePin_ReservedPins674=== RUN TestPresentReportsOnlyClosureRoots675=== PAUSE TestPresentReportsOnlyClosureRoots676=== RUN TestPresentRefreshesOnlyStaleClosures677=== PAUSE TestPresentRefreshesOnlyStaleClosures678=== RUN TestPresentNotReportedWhileGCDeletesClosure679=== PAUSE TestPresentNotReportedWhileGCDeletesClosure680=== RUN TestParseSingleRange681=== PAUSE TestParseSingleRange682=== RUN TestProxyHeadersOnlyTrustedOnSocket6832026/11/11 23:58:16 INFO Starting HTTP server address=127.0.0.1:612506842026/11/11 23:58:16 INFO Starting HTTP server address=proxy.sock6852026/11/11 23:58:16 WARN mTLS auth: subject not in bound subjects subject="CN=someone"6862026/11/11 23:58:16 INFO Shutdown signal received, draining in-flight requests timeout=10s687--- PASS: TestProxyHeadersOnlyTrustedOnSocket (0.17s)688=== RUN TestIsValidCachePath689=== PAUSE TestIsValidCachePath690=== RUN TestReadProxyNarinfo691=== PAUSE TestReadProxyNarinfo692=== RUN TestReadProxyNarinfoAlreadyDecompressed693=== PAUSE TestReadProxyNarinfoAlreadyDecompressed694=== RUN TestReadProxyContentNegotiation695=== PAUSE TestReadProxyContentNegotiation696=== RUN TestReadProxyNarStreaming697=== PAUSE TestReadProxyNarStreaming698=== RUN TestReadProxy404699=== PAUSE TestReadProxy404700=== RUN TestReadProxyInvalidPath701=== PAUSE TestReadProxyInvalidPath702=== RUN TestReadProxyHead703=== PAUSE TestReadProxyHead704=== RUN TestReadProxyOutlastsServerWriteTimeout705=== PAUSE TestReadProxyOutlastsServerWriteTimeout706=== RUN TestReadProxyConditionalGet707=== PAUSE TestReadProxyConditionalGet708=== RUN TestReadProxyRootRedirectsToIndexHTML709=== PAUSE TestReadProxyRootRedirectsToIndexHTML710=== RUN TestReadProxyDisabled711=== PAUSE TestReadProxyDisabled712=== RUN TestReadRedirectNar713=== PAUSE TestReadRedirectNar714=== RUN TestReadRedirectKeepsNarinfoProxied715=== PAUSE TestReadRedirectKeepsNarinfoProxied716=== RUN TestReadProxyRangeRequest717=== PAUSE TestReadProxyRangeRequest718=== RUN TestReadRedirectUsesPublicS3URL719=== PAUSE TestReadRedirectUsesPublicS3URL720=== RUN TestPush_OverlappingRootsStoreOneRowPerKey721=== PAUSE TestPush_OverlappingRootsStoreOneRowPerKey722=== RUN TestPush_CompleteCommitsEveryRoot723=== PAUSE TestPush_CompleteCommitsEveryRoot724=== RUN TestPush_RepeatedRootCommitsOnce725=== PAUSE TestPush_RepeatedRootCommitsOnce726=== RUN TestPush_CompleteWaitsForCleanupOfThePush727=== PAUSE TestPush_CompleteWaitsForCleanupOfThePush728=== RUN TestPush_RootsAreLockedInByteOrder729=== PAUSE TestPush_RootsAreLockedInByteOrder730=== RUN TestPush_SkippedKeySurvivesGCBeforeCommit731=== PAUSE TestPush_SkippedKeySurvivesGCBeforeCommit732=== RUN TestPush_RejectsBadRequests733=== PAUSE TestPush_RejectsBadRequests734=== RUN TestPush_SignsNarinfosOfItsPendingObjects735=== PAUSE TestPush_SignsNarinfosOfItsPendingObjects736=== RUN TestSharedNarBothUploadsComplete737=== PAUSE TestSharedNarBothUploadsComplete738=== RUN TestCompleteMultipartUpload_ErrorButObjectExists739=== PAUSE TestCompleteMultipartUpload_ErrorButObjectExists740=== RUN TestCompletedNarNotReofferedAcrossClosures741=== PAUSE TestCompletedNarNotReofferedAcrossClosures742=== RUN TestPresignedUploadRegisteredBeforeCommit743=== PAUSE TestPresignedUploadRegisteredBeforeCommit744=== RUN TestService_Rustfstest745=== PAUSE TestService_Rustfstest746=== RUN TestParseSize747=== PAUSE TestParseSize748=== RUN TestSkippedUploadsHandler749=== PAUSE TestSkippedUploadsHandler750=== RUN TestSystemdListenerNotActivated751--- PASS: TestSystemdListenerNotActivated (0.00s)752=== RUN TestWatchdogBeatsWhenHealthy753--- PASS: TestWatchdogBeatsWhenHealthy (0.03s)754=== RUN TestWatchdogSkipsWhenUnhealthy7552026/11/11 23:58:16 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7562026/11/11 23:58:16 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7572026/11/11 23:58:16 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7582026/11/11 23:58:16 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7592026/11/11 23:58:16 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7602026/11/11 23:58:16 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7612026/11/11 23:58:16 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7622026/11/11 23:58:16 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7632026/11/11 23:58:16 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7642026/11/11 23:58:16 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"765--- PASS: TestWatchdogSkipsWhenUnhealthy (0.20s)766=== RUN TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle767=== PAUSE TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle768=== RUN TestProxyWriteTimeout769=== PAUSE TestProxyWriteTimeout770=== RUN TestPendingClosureWriteTimeout771=== PAUSE TestPendingClosureWriteTimeout772=== RUN TestIsValidUploadKey773=== PAUSE TestIsValidUploadKey774=== RUN TestUploadHandlersRejectInvalidKeys775=== PAUSE TestUploadHandlersRejectInvalidKeys776=== RUN TestUploadHandlersRejectOversizedBody777=== PAUSE TestUploadHandlersRejectOversizedBody778=== RUN TestService_cleanupPendingClosuresHandler779=== PAUSE TestService_cleanupPendingClosuresHandler780=== RUN TestService_createPendingClosureHandler781=== PAUSE TestService_createPendingClosureHandler782=== RUN TestService_verifyS3Integrity783=== PAUSE TestService_verifyS3Integrity784=== RUN TestService_verifyS3ReuploadsStaleNarinfo785=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo786=== RUN TestCompleteMultipartUnregistered787=== PAUSE TestCompleteMultipartUnregistered788=== RUN TestCreatePendingClosure_SmallNARUsesSimplePUT789=== PAUSE TestCreatePendingClosure_SmallNARUsesSimplePUT790=== CONT TestPush_RootsAreLockedInByteOrder791=== CONT TestGCTaskStore_ConflictDifferentParams792=== CONT TestPush_CompleteWaitsForCleanupOfThePush793=== CONT TestProxyWriteTimeout794=== RUN TestProxyWriteTimeout/narinfo795=== CONT TestLeadElectsOneAndHandsOver796=== CONT TestGCTaskStore_GetEmpty797=== CONT TestPush_OverlappingRootsStoreOneRowPerKey798--- PASS: TestGCTaskStore_ConflictDifferentParams (0.00s)799--- PASS: TestGCTaskStore_GetEmpty (0.00s)800=== CONT TestLeadUnannouncedStandbyYieldsToIncumbent801=== CONT TestGCTaskStore_DeduplicateSameParams802--- PASS: TestGCTaskStore_DeduplicateSameParams (0.00s)803=== CONT TestReadRedirectUsesPublicS3URL804=== CONT TestConnectWaitsForAPeerMigration805=== CONT TestResolveDBConnectionString806=== CONT TestConcurrentCommitsSharingObjectsDoNotDeadlock807=== CONT TestService_createPendingClosureHandler808=== CONT TestDeleteClosuresRacingPins809=== CONT TestCreatePendingClosure_SmallNARUsesSimplePUT810=== CONT TestCompleteMultipartUnregistered811=== CONT TestClientFallsBackToClosures812=== CONT TestService_verifyS3ReuploadsStaleNarinfo813=== CONT TestClientPushesUseOnePush814=== RUN TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed815=== CONT TestService_verifyS3Integrity816=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed817=== CONT TestGCTaskStore_StartNew818=== RUN TestResolveDBConnectionString/flag_wins819=== CONT TestCreatePin_ReservedPins820=== CONT TestReadProxyRangeRequest821=== RUN TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed822=== CONT TestPush_RepeatedRootCommitsOnce823=== PAUSE TestResolveDBConnectionString/flag_wins824=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed825=== RUN TestResolveDBConnectionString/file_when_flag_empty826=== RUN TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost827=== PAUSE TestResolveDBConnectionString/file_when_flag_empty828=== RUN TestResolveDBConnectionString/missing_file_is_an_error829=== PAUSE TestResolveDBConnectionString/missing_file_is_an_error830=== RUN TestResolveDBConnectionString/PGHOST_allows_empty831=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost832=== CONT TestAbortSucceeded833=== CONT TestReadRedirectKeepsNarinfoProxied834=== RUN TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed835=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed836=== PAUSE TestProxyWriteTimeout/narinfo837--- PASS: TestGCTaskStore_StartNew (0.00s)838--- PASS: TestAbortSucceeded (0.00s)839=== CONT TestGCEndsOnShutdown840=== CONT TestPush_CompleteCommitsEveryRoot841=== PAUSE TestResolveDBConnectionString/PGHOST_allows_empty842=== RUN TestProxyWriteTimeout/1_GiB_nar843=== RUN TestResolveDBConnectionString/nothing_configured844=== RUN TestGCEndsOnShutdown/before_the_run845=== PAUSE TestProxyWriteTimeout/1_GiB_nar846=== RUN TestProxyWriteTimeout/10_GiB_nar847=== PAUSE TestResolveDBConnectionString/nothing_configured848=== RUN TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed849=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed850=== PAUSE TestGCEndsOnShutdown/before_the_run851=== PAUSE TestProxyWriteTimeout/10_GiB_nar852=== CONT TestReadRedirectNar853=== CONT TestCommitRacingPendingCleanupKeepsObjects854=== RUN TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete855=== PAUSE TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete856=== RUN TestProxyWriteTimeout/unknown_size857=== RUN TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark858=== PAUSE TestProxyWriteTimeout/unknown_size859=== PAUSE TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark860=== CONT TestReadProxyDisabled861=== CONT TestPinProtectsFromGC8622026/11/11 23:58:16 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:61252/oidc8632026/11/11 23:58:17 INFO Received push request method=POST path=/api/pushes8642026/11/11 23:58:17 INFO Received push request method=POST path=/api/pushes8652026-11-11 23:58:17.551 UTC [99932] ERROR: Push does not exist: id=18662026-11-11 23:58:17.551 UTC [99932] CONTEXT: PL/pgSQL function commit_push(bigint) line 11 at RAISE8672026-11-11 23:58:17.551 UTC [99932] STATEMENT: SELECT commit_push($1)868--- PASS: TestPush_CompleteWaitsForCleanupOfThePush (1.17s)869=== CONT TestReadProxyRootRedirectsToIndexHTML8702026-11-11 23:58:17.589 UTC [99939] ERROR: could not obtain lock on row in relation "closures"8712026-11-11 23:58:17.589 UTC [99939] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT8722026-11-11 23:58:17.594 UTC [99939] ERROR: could not obtain lock on row in relation "closures"8732026-11-11 23:58:17.594 UTC [99939] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT8742026-11-11 23:58:17.596 UTC [99939] ERROR: could not obtain lock on row in relation "closures"8752026-11-11 23:58:17.596 UTC [99939] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT8762026-11-11 23:58:17.603 UTC [99939] ERROR: could not obtain lock on row in relation "closures"8772026-11-11 23:58:17.603 UTC [99939] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT8782026-11-11 23:58:17.604 UTC [99939] ERROR: could not obtain lock on row in relation "closures"8792026-11-11 23:58:17.604 UTC [99939] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT880--- PASS: TestPush_RootsAreLockedInByteOrder (1.24s)881=== CONT TestClientSharedPathCommittedMidPush8822026/11/11 23:58:18 INFO lead: acquired remote=192.0.2.1:1234 generation=18832026/11/11 23:58:18 INFO Received push request method=POST path=/api/pushes8842026/11/11 23:58:18 INFO Uploading 2 paths to 127.0.0.1 (1 already cached)8852026/11/11 23:58:18 INFO Uploading i05zw0yrhvkzdi480bm8j0fvxl3gf70p-b (256B)8862026/11/11 23:58:18 INFO Uploading 3x6s17mq5l9d90xh5q1shdzw9w16x8sq-shared-dep (136B)8872026/11/11 23:58:18 INFO Received push request method=POST path=/api/pushes8882026/11/11 23:58:18 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"8892026/11/11 23:58:18 WARN Failed to register uploaded object key=i05zw0yrhvkzdi480bm8j0fvxl3gf70p.ls error="server returned 404: 404 page not found\n"8902026/11/11 23:58:18 WARN Failed to register uploaded object key=3pf7f7namnwn6b0xgav0cwfxjqn632g1.ls error="server returned 404: 404 page not found\n"8912026/11/11 23:58:18 WARN Failed to register uploaded object key=3x6s17mq5l9d90xh5q1shdzw9w16x8sq.ls error="server returned 404: 404 page not found\n"8922026/11/11 23:58:18 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign8932026/11/11 23:58:18 WARN Failed to register uploaded object key=nar/0askg83kccapb90rhr4ffvdkk2kjlz43ybh13g6pp8mymn9026wv.nar.zst error="server returned 404: 404 page not found\n"8942026/11/11 23:58:18 INFO Signed narinfos id=1 count=38952026/11/11 23:58:18 INFO Uploading 3 narinfos896--- PASS: TestPush_OverlappingRootsStoreOneRowPerKey (2.14s)8972026/11/11 23:58:18 WARN Failed to register uploaded object key=3x6s17mq5l9d90xh5q1shdzw9w16x8sq.narinfo error="server returned 404: 404 page not found\n"898=== CONT TestReadProxyConditionalGet8992026/11/11 23:58:18 INFO Received complete push request method=POST path=/api/pushes/1/complete9002026/11/11 23:58:18 WARN Failed to register uploaded object key=i05zw0yrhvkzdi480bm8j0fvxl3gf70p.narinfo error="server returned 404: 404 page not found\n"9012026/11/11 23:58:18 WARN Failed to register uploaded object key=3pf7f7namnwn6b0xgav0cwfxjqn632g1.narinfo error="server returned 404: 404 page not found\n"9022026/11/11 23:58:18 INFO Completed push id=1 objects=8 duration=14.268792ms9032026/11/11 23:58:18 INFO Upload complete. (214ms)904=== NAME TestClientPushesUseOnePush905 client_pushes_test.go:97: Retrieved narinfo from S3:906 StorePath: /private/var/tb/5/scratch/67d206d5acc7e9e5/build/TestClientPushesUseOnePush3194712162/001/store/3x6s17mq5l9d90xh5q1shdzw9w16x8sq-shared-dep907 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst908 Compression: zstd909 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82910 NarSize: 136911 References: 912 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n913 client_pushes_test.go:97: Retrieved narinfo from S3:914 StorePath: /private/var/tb/5/scratch/67d206d5acc7e9e5/build/TestClientPushesUseOnePush3194712162/001/store/3pf7f7namnwn6b0xgav0cwfxjqn632g1-a915 URL: nar/0askg83kccapb90rhr4ffvdkk2kjlz43ybh13g6pp8mymn9026wv.nar.zst916 Compression: zstd917 NarHash: sha256:0askg83kccapb90rhr4ffvdkk2kjlz43ybh13g6pp8mymn9026wv918 NarSize: 256919 References: /private/var/tb/5/scratch/67d206d5acc7e9e5/build/TestClientPushesUseOnePush3194712162/001/store/3x6s17mq5l9d90xh5q1shdzw9w16x8sq-shared-dep920 CA: text:sha256:03664cqcigkqd5nainz6qfx8jq75kkcrkj1rbimw52yzglyhi687921 client_pushes_test.go:97: Retrieved narinfo from S3:922 StorePath: /private/var/tb/5/scratch/67d206d5acc7e9e5/build/TestClientPushesUseOnePush3194712162/001/store/i05zw0yrhvkzdi480bm8j0fvxl3gf70p-b923 URL: nar/0askg83kccapb90rhr4ffvdkk2kjlz43ybh13g6pp8mymn9026wv.nar.zst924 Compression: zstd925 NarHash: sha256:0askg83kccapb90rhr4ffvdkk2kjlz43ybh13g6pp8mymn9026wv926 NarSize: 256927 References: /private/var/tb/5/scratch/67d206d5acc7e9e5/build/TestClientPushesUseOnePush3194712162/001/store/3x6s17mq5l9d90xh5q1shdzw9w16x8sq-shared-dep928 CA: text:sha256:03664cqcigkqd5nainz6qfx8jq75kkcrkj1rbimw52yzglyhi687929--- PASS: TestClientPushesUseOnePush (2.19s)930=== CONT TestReadProxyOutlastsServerWriteTimeout9312026/11/11 23:58:18 INFO lead: acquired remote=192.0.2.1:1234 generation=19322026/11/11 23:58:19 INFO Received uploads request method=POST path=/api/pending_closures9332026/11/11 23:58:19 INFO Received uploads request method=POST path=/api/pending_closures9342026/11/11 23:58:19 INFO Received uploads request method=POST path=/api/pending_closures935--- PASS: TestReadRedirectKeepsNarinfoProxied (2.95s)936=== CONT TestReadProxyHead9372026/11/11 23:58:19 INFO lead: released remote=192.0.2.1:1234 generation=19382026/11/11 23:58:19 INFO lead: acquired remote=192.0.2.1:1234 generation=29392026/11/11 23:58:19 INFO lead: given back to the incumbent remote=192.0.2.1:1234 generation=29402026/11/11 23:58:19 INFO lead: acquired remote=192.0.2.1:1234 generation=39412026/11/11 23:58:19 INFO Received push request method=POST path=/api/pushes9422026/11/11 23:58:19 INFO Received complete push request method=POST path=/api/pushes/1/complete9432026/11/11 23:58:19 INFO Completed push id=1 objects=4 duration=4.735459ms944--- PASS: TestPush_RepeatedRootCommitsOnce (3.19s)945=== CONT TestReadProxyInvalidPath946--- PASS: TestConcurrentCommitsSharingObjectsDoNotDeadlock (3.22s)947=== CONT TestReadProxy4049482026/11/11 23:58:19 INFO Received uploads request method=POST path=/api/pending_closures949--- PASS: TestCreatePendingClosure_SmallNARUsesSimplePUT (3.40s)950=== CONT TestReadProxyNarStreaming9512026/11/11 23:58:19 INFO lead: released remote=192.0.2.1:1234 generation=19522026/11/11 23:58:19 INFO lead: acquired remote=192.0.2.1:1234 generation=29532026/11/11 23:58:19 INFO Aborted multipart uploads count=0 kept=09542026/11/11 23:58:20 WARN Force mode enabled - objects will be deleted immediately without grace period9552026/11/11 23:58:20 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=09562026/11/11 23:58:20 INFO Vacuumed table table=pending_closures9572026/11/11 23:58:20 INFO Vacuumed table table=pending_objects9582026/11/11 23:58:20 INFO Vacuumed table table=multipart_uploads9592026/11/11 23:58:20 INFO Received complete multipart upload request method=POST path=/api/multipart/complete960--- PASS: TestReadRedirectUsesPublicS3URL (3.73s)961=== CONT TestReadProxyContentNegotiation9622026/11/11 23:58:20 INFO Vacuumed table table=closures9632026/11/11 23:58:20 INFO Vacuumed table table=objects964--- PASS: TestCommitRacingPendingCleanupKeepsObjects (3.78s)965=== CONT TestReadProxyNarinfoAlreadyDecompressed9662026/11/11 23:58:20 INFO Completed multipart upload object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst upload_id=ZDM5YmExODItZjViYi00ZTI4LWIwMTgtYWUzNjdkYzQxMGVhLjE3Njg3NzJjLTI0ZDctNDRjYy1hZDlhLWY3ZTUyNDU3YzI3NHgxNzk0NDQxNDk5MTIwNzkyMDAw parts=109672026/11/11 23:58:20 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete9682026/11/11 23:58:20 INFO Completed upload id=1 objects=2 duration=9.322292ms9692026/11/11 23:58:20 INFO Received get closure request method=GET path=/api/closures/000000000000000000000000000000009702026/11/11 23:58:20 INFO Received uploads request method=POST path=/api/pending_closures9712026/11/11 23:58:20 INFO Starting cleanup of old closures method=DELETE path=/api/closures9722026/11/11 23:58:20 INFO Aborted multipart uploads count=0 kept=09732026/11/11 23:58:20 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=09742026/11/11 23:58:20 INFO Vacuumed table table=pending_closures9752026/11/11 23:58:20 INFO Vacuumed table table=pending_objects9762026/11/11 23:58:20 INFO Vacuumed table table=multipart_uploads9772026/11/11 23:58:20 INFO Vacuumed table table=closures9782026/11/11 23:58:20 INFO Vacuumed table table=objects9792026/11/11 23:58:20 INFO Received get closure request method=GET path=/api/closures/00000000000000000000000000000000980--- PASS: TestService_createPendingClosureHandler (3.99s)981=== CONT TestReadProxyNarinfo9822026/11/11 23:58:20 INFO Received complete multipart upload request method=POST path=/api/multipart/complete9832026/11/11 23:58:20 ERROR Failed to get multipart upload error="no rows in result set" upload_id=does-not-exist object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst984--- PASS: TestCompleteMultipartUnregistered (4.08s)985=== CONT TestIsValidCachePath986=== RUN TestIsValidCachePath/narinfo987=== PAUSE TestIsValidCachePath/narinfo988=== RUN TestIsValidCachePath/narinfo_all_nix_base32_chars989=== PAUSE TestIsValidCachePath/narinfo_all_nix_base32_chars990=== RUN TestIsValidCachePath/nar_zst991=== PAUSE TestIsValidCachePath/nar_zst992=== RUN TestIsValidCachePath/nar_xz993=== PAUSE TestIsValidCachePath/nar_xz994=== RUN TestIsValidCachePath/nar_bz2995=== PAUSE TestIsValidCachePath/nar_bz2996=== RUN TestIsValidCachePath/nar_uncompressed997=== PAUSE TestIsValidCachePath/nar_uncompressed998=== RUN TestIsValidCachePath/ls999=== PAUSE TestIsValidCachePath/ls1000=== RUN TestIsValidCachePath/log1001=== PAUSE TestIsValidCachePath/log1002=== RUN TestIsValidCachePath/realisation1003=== PAUSE TestIsValidCachePath/realisation1004=== RUN TestIsValidCachePath/nix-cache-info1005=== PAUSE TestIsValidCachePath/nix-cache-info1006=== RUN TestIsValidCachePath/index.html1007=== PAUSE TestIsValidCachePath/index.html1008=== RUN TestIsValidCachePath/traversal_parent1009=== PAUSE TestIsValidCachePath/traversal_parent1010=== RUN TestIsValidCachePath/traversal_in_middle1011=== PAUSE TestIsValidCachePath/traversal_in_middle1012=== RUN TestIsValidCachePath/invalid_char_e1013=== PAUSE TestIsValidCachePath/invalid_char_e1014=== RUN TestIsValidCachePath/invalid_char_u1015=== PAUSE TestIsValidCachePath/invalid_char_u1016=== RUN TestIsValidCachePath/random_path1017=== PAUSE TestIsValidCachePath/random_path1018=== RUN TestIsValidCachePath/empty1019=== PAUSE TestIsValidCachePath/empty1020=== RUN TestIsValidCachePath/leading_slash1021=== PAUSE TestIsValidCachePath/leading_slash1022=== RUN TestIsValidCachePath/wrong_extension1023=== PAUSE TestIsValidCachePath/wrong_extension1024=== RUN TestIsValidCachePath/short_hash1025=== PAUSE TestIsValidCachePath/short_hash1026=== CONT TestParseSingleRange1027=== RUN TestParseSingleRange/none1028=== PAUSE TestParseSingleRange/none1029=== RUN TestParseSingleRange/unknown_unit1030=== PAUSE TestParseSingleRange/unknown_unit1031=== RUN TestParseSingleRange/multi-range_ignored1032=== PAUSE TestParseSingleRange/multi-range_ignored1033=== RUN TestParseSingleRange/malformed_no_dash1034=== PAUSE TestParseSingleRange/malformed_no_dash1035=== RUN TestParseSingleRange/malformed_both_empty1036=== PAUSE TestParseSingleRange/malformed_both_empty1037=== RUN TestParseSingleRange/malformed_end_before_start1038=== PAUSE TestParseSingleRange/malformed_end_before_start1039=== RUN TestParseSingleRange/closed1040=== PAUSE TestParseSingleRange/closed1041=== RUN TestParseSingleRange/open-ended1042=== PAUSE TestParseSingleRange/open-ended1043=== RUN TestParseSingleRange/end_clamped_to_size1044=== PAUSE TestParseSingleRange/end_clamped_to_size1045=== RUN TestParseSingleRange/suffix1046=== PAUSE TestParseSingleRange/suffix1047=== RUN TestParseSingleRange/suffix_exceeds_size1048=== PAUSE TestParseSingleRange/suffix_exceeds_size1049=== RUN TestParseSingleRange/single_byte1050=== PAUSE TestParseSingleRange/single_byte1051=== RUN TestParseSingleRange/start_past_EOF1052=== PAUSE TestParseSingleRange/start_past_EOF1053=== RUN TestParseSingleRange/start_far_past_EOF1054=== PAUSE TestParseSingleRange/start_far_past_EOF1055=== CONT TestClientWithDependencies1056=== NAME TestPinProtectsFromGC1057 client_integration_test.go:779: Pinned store path: /private/var/tb/5/scratch/67d206d5acc7e9e5/build/TestPinProtectsFromGC4007198935/001/store/g9pkgxl08jqpzxvc9r3ygdlgw7nc7zsv-pinned-file.txt1058 client_integration_test.go:780: Unpinned store path: /private/var/tb/5/scratch/67d206d5acc7e9e5/build/TestPinProtectsFromGC4007198935/001/store/2dw3qcpg62hcs2vvdxc079y42dsmnnmb-unpinned-file.txt10592026/11/11 23:58:20 INFO Received push request method=POST path=/api/pushes10602026/11/11 23:58:20 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)10612026/11/11 23:58:20 INFO Uploading g9pkgxl08jqpzxvc9r3ygdlgw7nc7zsv-pinned-file.txt (128B)1062--- PASS: TestReadProxyDisabled (4.25s)1063=== CONT TestClientMultipleUploads10642026/11/11 23:58:20 WARN Failed to register uploaded object key=g9pkgxl08jqpzxvc9r3ygdlgw7nc7zsv.ls error="server returned 404: 404 page not found\n"10652026/11/11 23:58:20 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign10662026/11/11 23:58:20 WARN Failed to register uploaded object key=nar/0qmhsg42pqsz7x6hh02j8gznnac3yi3qpjhrqbwvz88i194jbb5s.nar.zst error="server returned 404: 404 page not found\n"10672026/11/11 23:58:20 INFO Signed narinfos id=1 count=110682026/11/11 23:58:20 INFO Uploading 1 narinfos10692026/11/11 23:58:20 INFO Received complete push request method=POST path=/api/pushes/1/complete10702026/11/11 23:58:20 WARN Failed to register uploaded object key=g9pkgxl08jqpzxvc9r3ygdlgw7nc7zsv.narinfo error="server returned 404: 404 page not found\n"10712026/11/11 23:58:20 INFO Completed push id=1 objects=3 duration=18.418334ms10722026/11/11 23:58:20 INFO Upload complete. (184ms)10732026/11/11 23:58:20 INFO Received push request method=POST path=/api/pushes10742026/11/11 23:58:20 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)10752026/11/11 23:58:20 INFO Uploading 2dw3qcpg62hcs2vvdxc079y42dsmnnmb-unpinned-file.txt (128B)10762026/11/11 23:58:20 WARN Failed to register uploaded object key=2dw3qcpg62hcs2vvdxc079y42dsmnnmb.ls error="server returned 404: 404 page not found\n"10772026/11/11 23:58:20 INFO Received sign narinfos request method=POST path=/api/pushes/2/sign10782026/11/11 23:58:20 WARN Failed to register uploaded object key=nar/0gh0wdnr6gmk4r5sfzzh12isnl14rgmkrwy1ws41hpsrlndclg5h.nar.zst error="server returned 404: 404 page not found\n"10792026/11/11 23:58:20 INFO Signed narinfos id=2 count=110802026/11/11 23:58:20 INFO Uploading 1 narinfos1081--- PASS: TestReadProxyRangeRequest (4.54s)1082=== CONT TestClientIntegration10832026/11/11 23:58:20 INFO Received complete push request method=POST path=/api/pushes/2/complete10842026/11/11 23:58:20 WARN Failed to register uploaded object key=2dw3qcpg62hcs2vvdxc079y42dsmnnmb.narinfo error="server returned 404: 404 page not found\n"10852026/11/11 23:58:20 INFO Completed push id=2 objects=3 duration=1.103708ms10862026/11/11 23:58:20 INFO Upload complete. (161ms)10872026/11/11 23:58:20 INFO Starting cleanup of old closures method=DELETE path=/api/closures10882026/11/11 23:58:20 INFO Garbage collection started10892026/11/11 23:58:21 INFO Aborted multipart uploads count=0 kept=010902026/11/11 23:58:21 WARN Force mode enabled - objects will be deleted immediately without grace period10912026/11/11 23:58:21 INFO lead: released remote=192.0.2.1:1234 generation=21092--- PASS: TestLeadElectsOneAndHandsOver (4.65s)1093=== CONT TestClientErrorHandling1094=== RUN TestClientErrorHandling/InvalidStorePath1095=== PAUSE TestClientErrorHandling/InvalidStorePath1096=== RUN TestClientErrorHandling/InvalidAuthToken1097=== PAUSE TestClientErrorHandling/InvalidAuthToken1098=== RUN TestClientErrorHandling/ServerNotAvailable1099=== PAUSE TestClientErrorHandling/ServerNotAvailable1100=== CONT TestClientCADerivations1101--- PASS: TestReadRedirectNar (4.74s)1102=== CONT TestCacheStatsHandler11032026/11/11 23:58:21 INFO lead: released remote=192.0.2.1:1234 generation=31104--- PASS: TestLeadUnannouncedStandbyYieldsToIncumbent (4.97s)1105=== CONT TestCacheConfigHandler1106=== RUN TestCacheConfigHandler/full_config,_no_issuer1107=== PAUSE TestCacheConfigHandler/full_config,_no_issuer1108=== RUN TestCacheConfigHandler/no_cache_url_configured1109=== PAUSE TestCacheConfigHandler/no_cache_url_configured1110=== RUN TestCacheConfigHandler/no_signing_keys1111=== PAUSE TestCacheConfigHandler/no_signing_keys1112=== RUN TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1113=== PAUSE TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1114=== CONT TestPresentNotReportedWhileGCDeletesClosure1115=== RUN TestPresentNotReportedWhileGCDeletesClosure/stale1116=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/stale1117=== RUN TestPresentNotReportedWhileGCDeletesClosure/fresh1118=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/fresh1119=== CONT TestPresentRefreshesOnlyStaleClosures11202026/11/11 23:58:21 INFO Received uploads request method=POST path=/api/pending_closures11212026/11/11 23:58:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=011222026/11/11 23:58:21 INFO Vacuumed table table=pending_closures11232026/11/11 23:58:21 INFO Vacuumed table table=pending_objects11242026/11/11 23:58:21 INFO Vacuumed table table=multipart_uploads11252026/11/11 23:58:21 INFO Vacuumed table table=closures11262026/11/11 23:58:21 INFO Vacuumed table table=objects1127--- PASS: TestDeleteClosuresRacingPins (5.08s)1128=== CONT TestPresentReportsOnlyClosureRoots11292026/11/11 23:58:21 INFO Received push request method=POST path=/api/pushes11302026/11/11 23:58:21 INFO Received complete push request method=POST path=/api/pushes/1/complete11312026/11/11 23:58:21 INFO Completed push id=1 objects=6 duration=24.7745ms1132--- PASS: TestPush_CompleteCommitsEveryRoot (5.36s)1133=== CONT TestCompletedNarNotReofferedAcrossClosures11342026/11/11 23:58:22 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux11352026/11/11 23:58:22 WARN Refused reserved pin name=worker-x86_64-linux11362026/11/11 23:58:22 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux11372026/11/11 23:58:22 INFO Received create pin request method=POST path=/api/pins/my-app11382026/11/11 23:58:22 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux1139--- PASS: TestCreatePin_ReservedPins (5.72s)1140=== CONT TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle1141--- PASS: TestReadProxyRootRedirectsToIndexHTML (4.76s)1142=== CONT TestSkippedUploadsHandler11432026/11/11 23:58:22 INFO Client skipped oversized paths paths=3 nar_bytes=50000000001144--- PASS: TestSkippedUploadsHandler (0.00s)1145=== CONT TestParseSize1146--- PASS: TestParseSize (0.00s)1147=== CONT TestService_Rustfstest11482026/11/11 23:58:22 INFO Received uploads request method=POST path=/api/pending_closures11492026/11/11 23:58:22 INFO Received uploads request method=POST path=/api/pending_closures11502026/11/11 23:58:22 INFO Uploading 2 paths to 127.0.0.1 (1 already cached)11512026/11/11 23:58:22 INFO Uploading fv9x3zmdak7699vwmvcpkmxb2qnhn5dq-a (256B)11522026/11/11 23:58:22 INFO Uploading jfjwf4phb8jpn7bcq7bq8fdfg4sdsqii-shared-dep (136B)11532026/11/11 23:58:22 WARN Failed to register uploaded object key=c5lpjr3ckgpnflnc27pvf4m79hrmx40j.ls error="server returned 404: 404 page not found\n"11542026/11/11 23:58:22 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"11552026/11/11 23:58:22 WARN Failed to register uploaded object key=fv9x3zmdak7699vwmvcpkmxb2qnhn5dq.ls error="server returned 404: 404 page not found\n"11562026/11/11 23:58:22 WARN Failed to register uploaded object key=jfjwf4phb8jpn7bcq7bq8fdfg4sdsqii.ls error="server returned 404: 404 page not found\n"11572026/11/11 23:58:22 WARN Failed to register uploaded object key=nar/19m49mf31mmqvl7x0jzy0fkky1hfxv98xcn7kqhyrf7ddrlgd5c3.nar.zst error="server returned 404: 404 page not found\n"11582026/11/11 23:58:22 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign11592026/11/11 23:58:22 INFO Signed narinfos id=1 count=211602026/11/11 23:58:22 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign11612026/11/11 23:58:22 INFO Signed narinfos id=2 count=211622026/11/11 23:58:22 INFO Uploading 4 narinfos11632026/11/11 23:58:22 WARN Failed to register uploaded object key=jfjwf4phb8jpn7bcq7bq8fdfg4sdsqii.narinfo error="server returned 404: 404 page not found\n"11642026/11/11 23:58:22 WARN Failed to register uploaded object key=c5lpjr3ckgpnflnc27pvf4m79hrmx40j.narinfo error="server returned 404: 404 page not found\n"11652026/11/11 23:58:22 WARN Failed to register uploaded object key=fv9x3zmdak7699vwmvcpkmxb2qnhn5dq.narinfo error="server returned 404: 404 page not found\n"11662026/11/11 23:58:22 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete11672026/11/11 23:58:22 WARN Failed to register uploaded object key=jfjwf4phb8jpn7bcq7bq8fdfg4sdsqii.narinfo error="server returned 404: 404 page not found\n"11682026/11/11 23:58:22 INFO Received complete multipart upload request method=POST path=/api/multipart/complete11692026/11/11 23:58:22 INFO Completed upload id=1 objects=6 duration=42.034ms11702026/11/11 23:58:22 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete11712026/11/11 23:58:22 INFO Completed upload id=2 objects=6 duration=1.00975ms11722026/11/11 23:58:22 INFO Upload complete. (392ms)1173=== NAME TestClientFallsBackToClosures1174 client_pushes_test.go:112: Retrieved narinfo from S3:1175 StorePath: /private/var/tb/5/scratch/67d206d5acc7e9e5/build/TestClientFallsBackToClosures1492459366/001/store/jfjwf4phb8jpn7bcq7bq8fdfg4sdsqii-shared-dep1176 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1177 Compression: zstd1178 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821179 NarSize: 1361180 References: 1181 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n1182 client_pushes_test.go:112: Retrieved narinfo from S3:1183 StorePath: /private/var/tb/5/scratch/67d206d5acc7e9e5/build/TestClientFallsBackToClosures1492459366/001/store/fv9x3zmdak7699vwmvcpkmxb2qnhn5dq-a1184 URL: nar/19m49mf31mmqvl7x0jzy0fkky1hfxv98xcn7kqhyrf7ddrlgd5c3.nar.zst1185 Compression: zstd1186 NarHash: sha256:19m49mf31mmqvl7x0jzy0fkky1hfxv98xcn7kqhyrf7ddrlgd5c31187 NarSize: 2561188 References: /private/var/tb/5/scratch/67d206d5acc7e9e5/build/TestClientFallsBackToClosures1492459366/001/store/jfjwf4phb8jpn7bcq7bq8fdfg4sdsqii-shared-dep1189 CA: text:sha256:1imv87hv6rl4q1k8ckcrbp0q3wzhyz5fvr7aivqwhl9vhgjsi27d1190 client_pushes_test.go:112: Retrieved narinfo from S3:1191 StorePath: /private/var/tb/5/scratch/67d206d5acc7e9e5/build/TestClientFallsBackToClosures1492459366/001/store/c5lpjr3ckgpnflnc27pvf4m79hrmx40j-b1192 URL: nar/19m49mf31mmqvl7x0jzy0fkky1hfxv98xcn7kqhyrf7ddrlgd5c3.nar.zst1193 Compression: zstd1194 NarHash: sha256:19m49mf31mmqvl7x0jzy0fkky1hfxv98xcn7kqhyrf7ddrlgd5c31195 NarSize: 2561196 References: /private/var/tb/5/scratch/67d206d5acc7e9e5/build/TestClientFallsBackToClosures1492459366/001/store/jfjwf4phb8jpn7bcq7bq8fdfg4sdsqii-shared-dep1197 CA: text:sha256:1imv87hv6rl4q1k8ckcrbp0q3wzhyz5fvr7aivqwhl9vhgjsi27d11982026/11/11 23:58:22 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst upload_id=ZDM5YmExODItZjViYi00ZTI4LWIwMTgtYWUzNjdkYzQxMGVhLjBhYzU0OGNmLWUxMzctNDljNi1hNjkyLWUxNDMwOTM0MzQzOXgxNzk0NDQxNTAxNDA5MjI0MDAw parts=1011992026/11/11 23:58:22 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete1200--- PASS: TestClientFallsBackToClosures (6.31s)1201=== CONT TestService_ReadScope_PublicByDefault12022026/11/11 23:58:22 INFO Completed upload id=1 objects=2 duration=5.749709ms12032026/11/11 23:58:22 INFO Received uploads request method=POST path=/api/pending_closures12042026/11/11 23:58:22 INFO Received uploads request method=POST path=/api/pending_closures12052026/11/11 23:58:22 INFO Object in database but missing from S3 key=dadb44fdadb44fdadb44fdadb44f0000.narinfo12062026/11/11 23:58:22 WARN Found objects in DB but missing from S3, will re-upload count=11207--- PASS: TestService_verifyS3Integrity (6.34s)1208=== CONT TestService_RequireScope_OIDC12092026/11/11 23:58:22 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:61339/oidc1210--- PASS: TestConnectWaitsForAPeerMigration (6.50s)1211=== CONT TestPresignedUploadRegisteredBeforeCommit1212--- PASS: TestReadProxyConditionalGet (4.35s)1213=== CONT TestService_AuthMiddleware_OIDC12142026/11/11 23:58:22 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:61346/oidc12152026/11/11 23:58:22 INFO Received push request method=POST path=/api/pushes12162026/11/11 23:58:22 INFO Uploading 2 paths to 127.0.0.1 (0 already cached)12172026/11/11 23:58:22 INFO Uploading 34havhmh1pi6ynbdfmbnr4jnavqxn6q1-shared-dep (136B)12182026/11/11 23:58:22 INFO Uploading 4dl8ycfis8fmls3rshmc6mq116h84c2k-top (264B)12192026/11/11 23:58:22 INFO Garbage collection progress phase="" failed_uploads_deleted=0 old_closures_deleted=1 objects_marked=3 objects_deleted=3 objects_failed=012202026/11/11 23:58:23 INFO Received create pin request method=POST path=/api/pins/myapp12212026/11/11 23:58:23 WARN Failed to register uploaded object key=34havhmh1pi6ynbdfmbnr4jnavqxn6q1.ls error="server returned 404: 404 page not found\n"12222026/11/11 23:58:23 WARN Failed to register uploaded object key=4dl8ycfis8fmls3rshmc6mq116h84c2k.ls error="server returned 404: 404 page not found\n"12232026/11/11 23:58:23 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"12242026/11/11 23:58:23 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign12252026/11/11 23:58:23 WARN Failed to register uploaded object key=nar/0fdwk7mkq65sv7k5jvaimdd237zdjc33zwdrshxfzkypp09lhrwn.nar.zst error="server returned 404: 404 page not found\n"12262026/11/11 23:58:23 INFO Signed narinfos id=1 count=212272026/11/11 23:58:23 INFO Uploading 2 narinfos12282026/11/11 23:58:23 INFO Created/updated pin name=myapp store_path=/private/var/tb/5/scratch/67d206d5acc7e9e5/build/TestPinProtectsFromGC4007198935/001/store/g9pkgxl08jqpzxvc9r3ygdlgw7nc7zsv-pinned-file.txt narinfo_key=g9pkgxl08jqpzxvc9r3ygdlgw7nc7zsv.narinfo12292026/11/11 23:58:23 INFO Starting cleanup of old closures method=DELETE path=/api/closures12302026/11/11 23:58:23 INFO Garbage collection started12312026/11/11 23:58:23 WARN Failed to register uploaded object key=34havhmh1pi6ynbdfmbnr4jnavqxn6q1.narinfo error="server returned 404: 404 page not found\n"12322026/11/11 23:58:23 INFO Aborted multipart uploads count=0 kept=012332026/11/11 23:58:23 INFO Received complete push request method=POST path=/api/pushes/1/complete12342026/11/11 23:58:23 WARN Failed to register uploaded object key=4dl8ycfis8fmls3rshmc6mq116h84c2k.narinfo error="server returned 404: 404 page not found\n"12352026/11/11 23:58:23 WARN Force mode enabled - objects will be deleted immediately without grace period12362026/11/11 23:58:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=012372026/11/11 23:58:23 INFO Completed push id=1 objects=6 duration=19.411584ms12382026/11/11 23:58:23 INFO Upload complete. (236ms)12392026/11/11 23:58:23 INFO Vacuumed table table=pending_closures12402026/11/11 23:58:23 INFO Vacuumed table table=pending_objects1241=== NAME TestClientSharedPathCommittedMidPush12422026/11/11 23:58:23 INFO Vacuumed table table=multipart_uploads1243 client_integration_test.go:728: Retrieved narinfo from S3:1244 StorePath: /private/var/tb/5/scratch/67d206d5acc7e9e5/build/TestClientSharedPathCommittedMidPush2151515619/001/store/34havhmh1pi6ynbdfmbnr4jnavqxn6q1-shared-dep1245 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1246 Compression: zstd1247 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821248 NarSize: 1361249 References: 1250 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n12512026/11/11 23:58:23 INFO Vacuumed table table=closures1252 client_integration_test.go:728: Retrieved narinfo from S3:1253 StorePath: /private/var/tb/5/scratch/67d206d5acc7e9e5/build/TestClientSharedPathCommittedMidPush2151515619/001/store/4dl8ycfis8fmls3rshmc6mq116h84c2k-top1254 URL: nar/0fdwk7mkq65sv7k5jvaimdd237zdjc33zwdrshxfzkypp09lhrwn.nar.zst1255 Compression: zstd1256 NarHash: sha256:0fdwk7mkq65sv7k5jvaimdd237zdjc33zwdrshxfzkypp09lhrwn1257 NarSize: 2641258 References: /private/var/tb/5/scratch/67d206d5acc7e9e5/build/TestClientSharedPathCommittedMidPush2151515619/001/store/34havhmh1pi6ynbdfmbnr4jnavqxn6q1-shared-dep1259 CA: text:sha256:0rqighdmdxmvshj6c6ka24cd9g0nf37db9dbdj2gw5ya1cfl7sp912602026/11/11 23:58:23 INFO Vacuumed table table=objects1261--- PASS: TestClientSharedPathCommittedMidPush (5.53s)1262=== CONT TestNARDeduplicationMetadataUploadBug1263--- PASS: TestReadProxyHead (4.03s)1264=== CONT TestService_ReadAuthMiddleware1265--- PASS: TestReadProxyInvalidPath (4.24s)1266=== CONT TestMultipartUploadAbortedWhenCancelledBeforeRecorded1267--- PASS: TestReadProxy404 (4.51s)1268=== CONT TestMultipartCleanup1269--- PASS: TestReadProxyNarStreaming (4.44s)1270=== CONT TestService_AuthMiddleware_MTLSBoundSubjects1271--- PASS: TestReadProxyContentNegotiation (4.56s)1272=== CONT TestOrphanedObjectsGCStressTest1273--- PASS: TestReadProxyNarinfoAlreadyDecompressed (4.64s)1274=== CONT TestServerTLSConfig1275=== RUN TestServerTLSConfig/no_client_CA1276=== PAUSE TestServerTLSConfig/no_client_CA1277=== RUN TestServerTLSConfig/missing_CA_file1278=== PAUSE TestServerTLSConfig/missing_CA_file1279=== RUN TestServerTLSConfig/not_a_PEM_file1280=== PAUSE TestServerTLSConfig/not_a_PEM_file1281=== CONT TestService_AuthMiddleware_MTLSProxyHeader12822026/11/11 23:58:25 ERROR Failed to decode narinfo key=3hcdxyjf9yiq7qf3i4548drb6sjmwa1v.narinfo limit=16777216 error=<nil>1283=== NAME TestPinProtectsFromGC1284 client_integration_test.go:888: Pin successfully protected closure from garbage collection1285--- PASS: TestReadProxyNarinfo (4.71s)1286=== CONT TestOrphanedObjectsGC1287--- PASS: TestPinProtectsFromGC (8.72s)1288=== CONT TestService_NativeMTLS1289=== NAME TestClientWithDependencies1290 client_integration_test.go:661: Built derivation: /private/var/tb/5/scratch/67d206d5acc7e9e5/build/TestClientWithDependencies3599399429/001/store/37ri17n286cd8b0l9rzl7lihpkkc99gv-test-script1291=== NAME TestClientMultipleUploads1292 client_integration_test.go:406: Created store path 0: /private/var/tb/5/scratch/67d206d5acc7e9e5/build/TestClientMultipleUploads3138891812/001/store/q6ja4lgl78q38w2hgr6if6grz5mdrkiq-test-file-0.txt1293=== NAME TestClientWithDependencies1294 client_integration_test.go:663: Found 1 dependencies (including self)1295=== NAME TestClientMultipleUploads1296 client_integration_test.go:406: Created store path 1: /private/var/tb/5/scratch/67d206d5acc7e9e5/build/TestClientMultipleUploads3138891812/001/store/8jsgzbcprkx6pgjg2bx2fm8mdmnq7zrd-test-file-1.txt12972026/11/11 23:58:25 INFO Received push request method=POST path=/api/pushes12982026/11/11 23:58:25 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)12992026/11/11 23:58:25 INFO Uploading 37ri17n286cd8b0l9rzl7lihpkkc99gv-test-script (136B)1300 client_integration_test.go:406: Created store path 2: /private/var/tb/5/scratch/67d206d5acc7e9e5/build/TestClientMultipleUploads3138891812/001/store/nhyv0rgxw2d4h7fc4qqdrd5qlj2p2gpz-test-file-2.txt13012026/11/11 23:58:25 WARN Failed to register uploaded object key=37ri17n286cd8b0l9rzl7lihpkkc99gv.ls error="server returned 404: 404 page not found\n"13022026/11/11 23:58:25 WARN Failed to register uploaded object key=nar/1s7yghixzpinyqm7g59kyn3sav81pmnsdi5v63m6x4r2vacy907j.nar.zst error="server returned 404: 404 page not found\n"13032026/11/11 23:58:25 WARN Failed to register uploaded object key=log/xj8bmz8ba1155f8lnc5gaby0ld9a297c-test-script.drv error="server returned 404: 404 page not found\n"13042026/11/11 23:58:25 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign13052026/11/11 23:58:25 INFO Signed narinfos id=1 count=113062026/11/11 23:58:25 INFO Uploading 1 narinfos13072026/11/11 23:58:25 INFO Received complete push request method=POST path=/api/pushes/1/complete13082026/11/11 23:58:25 WARN Failed to register uploaded object key=37ri17n286cd8b0l9rzl7lihpkkc99gv.narinfo error="server returned 404: 404 page not found\n"1309=== NAME TestClientIntegration1310 client_integration_test.go:292: Created store path: /private/var/tb/5/scratch/67d206d5acc7e9e5/build/TestClientIntegration989275913/002/store/6b9p1jcwq0xfb833qny2nh5dn53jkqfi-test-file.txt13112026/11/11 23:58:25 INFO Received push request method=POST path=/api/pushes13122026/11/11 23:58:25 INFO Completed push id=1 objects=4 duration=16.660667ms13132026/11/11 23:58:25 INFO Upload complete. (280ms)1314=== NAME TestClientWithDependencies1315 client_integration_test.go:665: Skipping nix copy test - isolated store (/private/var/tb/5/scratch/67d206d5acc7e9e5/build/TestClientWithDependencies3599399429/001/store) requires matching store prefix13162026/11/11 23:58:25 INFO Uploading 3 paths to 127.0.0.1 (0 already cached)13172026/11/11 23:58:25 INFO Uploading nhyv0rgxw2d4h7fc4qqdrd5qlj2p2gpz-test-file-2.txt (160B)13182026/11/11 23:58:25 INFO Uploading 8jsgzbcprkx6pgjg2bx2fm8mdmnq7zrd-test-file-1.txt (160B)13192026/11/11 23:58:25 INFO Uploading q6ja4lgl78q38w2hgr6if6grz5mdrkiq-test-file-0.txt (160B)1320--- PASS: TestClientWithDependencies (5.48s)1321=== CONT TestObjectStatsUpdatedOncePerCommit13222026/11/11 23:58:25 WARN Failed to register uploaded object key=8jsgzbcprkx6pgjg2bx2fm8mdmnq7zrd.ls error="server returned 404: 404 page not found\n"13232026/11/11 23:58:25 WARN Failed to register uploaded object key=nar/0ahq3izw3h0b0c08p3xdfclh758fwxnp85ccvflasb4vnfsv50ri.nar.zst error="server returned 404: 404 page not found\n"13242026/11/11 23:58:25 WARN Failed to register uploaded object key=q6ja4lgl78q38w2hgr6if6grz5mdrkiq.ls error="server returned 404: 404 page not found\n"13252026/11/11 23:58:25 WARN Failed to register uploaded object key=nar/13p9pqi0ga6fx2apcwls9hjnv7nmf740ggpn5zvfzdsp5vz2bbg4.nar.zst error="server returned 404: 404 page not found\n"13262026/11/11 23:58:25 WARN Failed to register uploaded object key=nar/1823g6vx4q5zqvr73m7lkq1wpk40fin7bsc792di430yk7sg4lka.nar.zst error="server returned 404: 404 page not found\n"13272026/11/11 23:58:25 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign13282026/11/11 23:58:25 WARN Failed to register uploaded object key=nhyv0rgxw2d4h7fc4qqdrd5qlj2p2gpz.ls error="server returned 404: 404 page not found\n"13292026/11/11 23:58:26 INFO Signed narinfos id=1 count=313302026/11/11 23:58:26 INFO Uploading 3 narinfos13312026/11/11 23:58:26 INFO Received push request method=POST path=/api/pushes13322026/11/11 23:58:26 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)13332026/11/11 23:58:26 INFO Uploading 6b9p1jcwq0xfb833qny2nh5dn53jkqfi-test-file.txt (152B)13342026/11/11 23:58:26 WARN Failed to register uploaded object key=nhyv0rgxw2d4h7fc4qqdrd5qlj2p2gpz.narinfo error="server returned 404: 404 page not found\n"13352026/11/11 23:58:26 WARN Failed to register uploaded object key=q6ja4lgl78q38w2hgr6if6grz5mdrkiq.narinfo error="server returned 404: 404 page not found\n"13362026/11/11 23:58:26 INFO Received complete push request method=POST path=/api/pushes/1/complete13372026/11/11 23:58:26 WARN Failed to register uploaded object key=8jsgzbcprkx6pgjg2bx2fm8mdmnq7zrd.narinfo error="server returned 404: 404 page not found\n"13382026/11/11 23:58:26 WARN Failed to register uploaded object key=6b9p1jcwq0xfb833qny2nh5dn53jkqfi.ls error="server returned 404: 404 page not found\n"13392026/11/11 23:58:26 INFO Completed push id=1 objects=9 duration=34.994791ms13402026/11/11 23:58:26 INFO Upload complete. (288ms)1341=== NAME TestClientMultipleUploads1342 client_integration_test.go:417: Uploaded 3 paths in 330.285208ms13432026/11/11 23:58:26 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign13442026/11/11 23:58:26 WARN Failed to register uploaded object key=nar/11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1.nar.zst error="server returned 404: 404 page not found\n"13452026/11/11 23:58:26 INFO Signed narinfos id=1 count=113462026/11/11 23:58:26 INFO Uploading 1 narinfos13472026/11/11 23:58:26 INFO Received complete push request method=POST path=/api/pushes/1/complete13482026/11/11 23:58:26 WARN Failed to register uploaded object key=6b9p1jcwq0xfb833qny2nh5dn53jkqfi.narinfo error="server returned 404: 404 page not found\n"13492026/11/11 23:58:26 INFO Completed push id=1 objects=3 duration=24.983792ms13502026/11/11 23:58:26 INFO Upload complete. (267ms)1351--- PASS: TestClientMultipleUploads (5.58s)1352=== CONT TestMetricsCommit13532026/11/11 23:58:26 INFO All 1 paths already cached1354=== NAME TestClientIntegration1355 client_integration_test.go:318: Retrieved narinfo from S3:1356 StorePath: /private/var/tb/5/scratch/67d206d5acc7e9e5/build/TestClientIntegration989275913/002/store/6b9p1jcwq0xfb833qny2nh5dn53jkqfi-test-file.txt1357 URL: nar/11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1.nar.zst1358 Compression: zstd1359 NarHash: sha256:11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk11360 NarSize: 1521361 References: 1362 CA: fixed:r:sha256:11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk11363 client_integration_test.go:319: Retrieved .ls file from S3 (compressed size: 77 bytes)1364 client_integration_test.go:319: Decompressed .ls content (64 bytes):1365 {"version":1,"root":{"type":"regular","size":39,"narOffset":96}}1366--- PASS: TestCacheStatsHandler (5.20s)1367=== CONT TestMetricsInventory1368=== NAME TestClientCADerivations1369 client_ca_test.go:136: Built CA derivation: /private/var/tb/5/scratch/67d206d5acc7e9e5/build/TestClientCADerivations1909080877/001/store/h1x366fnyfi88pvg5rifi2ik3jmlxcgz-ca-test1370 client_ca_test.go:139: Found 1 dependencies (including self)13712026/11/11 23:58:26 INFO Received push request method=POST path=/api/pushes13722026/11/11 23:58:26 INFO Object in database but missing from S3 key=6b9p1jcwq0xfb833qny2nh5dn53jkqfi.ls13732026/11/11 23:58:26 WARN Found objects in DB but missing from S3, will re-upload count=113742026/11/11 23:58:26 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)1375--- PASS: TestPresentRefreshesOnlyStaleClosures (5.27s)1376=== CONT TestObjectStatsTrigger13772026/11/11 23:58:26 INFO Received complete push request method=POST path=/api/pushes/2/complete13782026/11/11 23:58:26 INFO Completed push id=2 objects=3 duration=1.454125ms13792026/11/11 23:58:26 WARN Failed to register uploaded object key=6b9p1jcwq0xfb833qny2nh5dn53jkqfi.ls error="server returned 404: 404 page not found\n"13802026/11/11 23:58:26 INFO Upload complete. (117ms)1381=== NAME TestClientIntegration1382 client_integration_test.go:336: Retrieved .ls file from S3 (compressed size: 62 bytes)1383 client_integration_test.go:336: Decompressed .ls content (49 bytes):1384 {"version":1,"root":{"type":"regular","size":39}}13852026/11/11 23:58:26 INFO Received push request method=POST path=/api/pushes13862026/11/11 23:58:26 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)13872026/11/11 23:58:26 INFO Uploading h1x366fnyfi88pvg5rifi2ik3jmlxcgz-ca-test (144B)13882026/11/11 23:58:26 WARN Failed to register uploaded object key=h1x366fnyfi88pvg5rifi2ik3jmlxcgz.ls error="server returned 404: 404 page not found\n"13892026/11/11 23:58:26 WARN Failed to register uploaded object key=nar/04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n.nar.zst error="server returned 404: 404 page not found\n"13902026/11/11 23:58:26 WARN Failed to register uploaded object key=log/bd33xf9yjrldxj8fj6zq981byfzq6ssl-ca-test.drv error="server returned 404: 404 page not found\n"13912026/11/11 23:58:26 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign13922026/11/11 23:58:26 INFO Signed narinfos id=1 count=113932026/11/11 23:58:26 INFO Uploading 1 narinfos13942026/11/11 23:58:26 INFO Received complete push request method=POST path=/api/pushes/1/complete13952026/11/11 23:58:26 WARN Failed to register uploaded object key=h1x366fnyfi88pvg5rifi2ik3jmlxcgz.narinfo error="server returned 404: 404 page not found\n"1396--- PASS: TestPresentReportsOnlyClosureRoots (5.50s)1397=== CONT TestPendingCleanupUsesOneCutoff13982026/11/11 23:58:26 INFO Completed push id=1 objects=4 duration=52.88025ms13992026/11/11 23:58:26 INFO Upload complete. (398ms)1400=== NAME TestClientCADerivations1401 client_ca_test.go:180: Narinfo contains CA field: StorePath: /private/var/tb/5/scratch/67d206d5acc7e9e5/build/TestClientCADerivations1909080877/001/store/h1x366fnyfi88pvg5rifi2ik3jmlxcgz-ca-test1402 URL: nar/04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n.nar.zst1403 Compression: zstd1404 NarHash: sha256:04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n1405 NarSize: 1441406 References: 1407 Deriver: /private/var/tb/5/scratch/67d206d5acc7e9e5/build/TestClientCADerivations1909080877/001/store/bd33xf9yjrldxj8fj6zq981byfzq6ssl-ca-test.drv1408 CA: fixed:r:sha256:04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n1409 client_ca_test.go:185: Checking for realisation files in S3...1410 client_ca_test.go:207: No realisation files uploaded - this is expected for locally built CA derivations1411 client_ca_test.go:208: Locally built CA derivations don't have DrvOutput IDs until they're substituted from a cache14122026/11/11 23:58:26 INFO Received push request method=POST path=/api/pushes14132026/11/11 23:58:27 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)14142026/11/11 23:58:27 INFO Received complete push request method=POST path=/api/pushes/3/complete14152026/11/11 23:58:27 WARN Failed to register uploaded object key=6b9p1jcwq0xfb833qny2nh5dn53jkqfi.ls error="server returned 404: 404 page not found\n"14162026/11/11 23:58:27 INFO Completed push id=3 objects=3 duration=1.084209ms14172026/11/11 23:58:27 INFO Upload complete. (92ms)1418=== NAME TestClientIntegration1419 client_integration_test.go:353: Retrieved .ls file from S3 (compressed size: 62 bytes)1420 client_integration_test.go:353: Decompressed .ls content (49 bytes):1421 {"version":1,"root":{"type":"regular","size":39}}1422 client_integration_test.go:364: Testing garbage collection...1423=== NAME TestClientCADerivations1424 client_ca_test.go:258: nix copy output: error: binary cache 's3://bucket41?endpoint=http://localhost:61235®ion=eu-west-1' is for Nix stores with prefix '/nix/store', not '/private/var/tb/5/scratch/67d206d5acc7e9e5/build/TestClientCADerivations1909080877/001/store'1425 client_ca_test.go:259: nix copy failed (might be expected with isolated stores): exit status 114262026/11/11 23:58:27 INFO Starting cleanup of old closures method=DELETE path=/api/closures14272026/11/11 23:58:27 INFO Garbage collection started14282026/11/11 23:58:27 INFO Aborted multipart uploads count=0 kept=014292026/11/11 23:58:27 INFO Received uploads request method=POST path=/api/pending_closures14302026/11/11 23:58:27 WARN Force mode enabled - objects will be deleted immediately without grace period1431--- PASS: TestClientCADerivations (6.11s)1432=== CONT TestSweepSkipsRowsLockedByPush1433--- PASS: TestReadProxyOutlastsServerWriteTimeout (8.83s)1434=== CONT TestSweepRechecksPendingAfterLocking14352026/11/11 23:58:27 INFO Received uploads request method=POST path=/api/pending_closures14362026/11/11 23:58:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=014372026/11/11 23:58:27 INFO Vacuumed table table=pending_closures14382026/11/11 23:58:27 INFO Vacuumed table table=pending_objects14392026/11/11 23:58:27 INFO Vacuumed table table=multipart_uploads14402026/11/11 23:58:27 INFO Vacuumed table table=closures14412026/11/11 23:58:27 INFO Vacuumed table table=objects1442--- PASS: TestService_Rustfstest (5.46s)1443=== CONT TestGetExistingObjectsLocksRows14442026/11/11 23:58:27 INFO Received complete multipart upload request method=POST path=/api/multipart/complete1445--- PASS: TestService_ReadScope_PublicByDefault (5.40s)1446=== CONT TestRegisterCompletedObjectNeedsPendingRow1447=== RUN TestService_RequireScope_OIDC/builder_may_write1448=== PAUSE TestService_RequireScope_OIDC/builder_may_write1449=== RUN TestService_RequireScope_OIDC/builder_may_not_admin1450=== PAUSE TestService_RequireScope_OIDC/builder_may_not_admin1451=== RUN TestService_RequireScope_OIDC/ops_may_admin1452=== PAUSE TestService_RequireScope_OIDC/ops_may_admin1453=== RUN TestService_RequireScope_OIDC/ops_may_not_write1454=== PAUSE TestService_RequireScope_OIDC/ops_may_not_write1455=== RUN TestService_RequireScope_OIDC/reader_may_not_write1456=== PAUSE TestService_RequireScope_OIDC/reader_may_not_write1457=== RUN TestService_RequireScope_OIDC/static_token_may_admin1458=== PAUSE TestService_RequireScope_OIDC/static_token_may_admin1459=== RUN TestService_RequireScope_OIDC/static_token_may_write1460=== PAUSE TestService_RequireScope_OIDC/static_token_may_write1461=== RUN TestService_RequireScope_OIDC/reader_may_read1462=== PAUSE TestService_RequireScope_OIDC/reader_may_read1463=== RUN TestService_RequireScope_OIDC/writer_implies_read1464=== PAUSE TestService_RequireScope_OIDC/writer_implies_read1465=== RUN TestService_RequireScope_OIDC/anonymous_may_not_read1466=== PAUSE TestService_RequireScope_OIDC/anonymous_may_not_read1467=== RUN TestService_RequireScope_OIDC/reader_may_read_with_netrc1468=== PAUSE TestService_RequireScope_OIDC/reader_may_read_with_netrc1469=== RUN TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1470=== PAUSE TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1471=== RUN TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read1472=== PAUSE TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read1473=== RUN TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1474=== PAUSE TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1475=== CONT TestPush_SignsNarinfosOfItsPendingObjects14762026/11/11 23:58:28 INFO Received uploads request method=POST path=/api/pending_closures14772026/11/11 23:58:28 INFO Registered completed upload object_key=nar/jjjjjjjjjjjjjjjjjjjjjjjjjjjjjj0100000000000000000000.nar.zst14782026/11/11 23:58:28 INFO Received uploads request method=POST path=/api/pending_closures1479--- PASS: TestPresignedUploadRegisteredBeforeCommit (5.92s)1480=== CONT TestCompleteMultipartUpload_ErrorButObjectExists1481=== RUN TestService_AuthMiddleware_OIDC/valid_OIDC_token1482=== PAUSE TestService_AuthMiddleware_OIDC/valid_OIDC_token1483=== RUN TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1484=== PAUSE TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1485=== RUN TestService_AuthMiddleware_OIDC/malformed_token_rejected1486=== PAUSE TestService_AuthMiddleware_OIDC/malformed_token_rejected1487=== RUN TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1488=== PAUSE TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1489=== CONT TestSharedNarBothUploadsComplete14902026/11/11 23:58:29 INFO Received complete multipart upload request method=POST path=/api/multipart/complete14912026/11/11 23:58:29 INFO Garbage collection progress phase="" failed_uploads_deleted=0 old_closures_deleted=1 objects_marked=3 objects_deleted=3 objects_failed=01492=== NAME TestClientIntegration1493 client_integration_test.go:371: Objects in database after GC:1494 client_integration_test.go:371: Successfully deleted all objects with GC --force14952026/11/11 23:58:29 INFO Completed multipart upload object_key=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhh0100000000000000000000.nar.zst upload_id=ZDM5YmExODItZjViYi00ZTI4LWIwMTgtYWUzNjdkYzQxMGVhLjY1ZWM0NjlhLTE2ZWEtNGFlNi1iZjFiLTU1MDMyNDRjYTczN3gxNzk0NDQxNTA3MTg1MDYyMDAw parts=1214962026/11/11 23:58:29 INFO Received uploads request method=POST path=/api/pending_closures1497--- PASS: TestCompletedNarNotReofferedAcrossClosures (7.37s)1498=== CONT TestForceGCDuringPushOffersSweptObject1499=== RUN TestForceGCDuringPushOffersSweptObject/before_pending_rows1500=== PAUSE TestForceGCDuringPushOffersSweptObject/before_pending_rows1501=== RUN TestForceGCDuringPushOffersSweptObject/after_presence_check1502=== PAUSE TestForceGCDuringPushOffersSweptObject/after_presence_check1503=== CONT TestPushWaitsForSweepHoldingRow1504--- PASS: TestClientIntegration (8.22s)1505=== CONT TestFailedS3DeleteKeepsTombstone1506=== NAME TestNARDeduplicationMetadataUploadBug1507 metadata_upload_test.go:48: First store path: /private/var/tb/5/scratch/67d206d5acc7e9e5/build/TestNARDeduplicationMetadataUploadBug2494630840/001/store/4s793gbs03isfy0b7fll2x3sv74vx103-file1.txt1508--- PASS: TestService_ReadAuthMiddleware (6.11s)1509=== CONT TestPush_RejectsBadRequests15102026/11/11 23:58:29 INFO Received push request method=POST path=/api/pushes15112026/11/11 23:58:29 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)15122026/11/11 23:58:29 INFO Uploading 4s793gbs03isfy0b7fll2x3sv74vx103-file1.txt (160B)15132026/11/11 23:58:29 WARN Failed to register uploaded object key=nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst error="server returned 404: 404 page not found\n"15142026/11/11 23:58:29 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign15152026/11/11 23:58:29 WARN Failed to register uploaded object key=4s793gbs03isfy0b7fll2x3sv74vx103.ls error="server returned 404: 404 page not found\n"15162026/11/11 23:58:29 INFO Signed narinfos id=1 count=115172026/11/11 23:58:29 INFO Uploading 1 narinfos15182026/11/11 23:58:29 INFO Received complete push request method=POST path=/api/pushes/1/complete15192026/11/11 23:58:29 WARN Failed to register uploaded object key=4s793gbs03isfy0b7fll2x3sv74vx103.narinfo error="server returned 404: 404 page not found\n"15202026/11/11 23:58:29 INFO Completed push id=1 objects=3 duration=24.1615ms15212026/11/11 23:58:29 INFO Upload complete. (217ms)15222026/11/11 23:58:29 INFO Received uploads request method=POST path=/api/pending_closures1523=== NAME TestNARDeduplicationMetadataUploadBug1524 metadata_upload_test.go:54: Retrieved narinfo from S3:1525 StorePath: /private/var/tb/5/scratch/67d206d5acc7e9e5/build/TestNARDeduplicationMetadataUploadBug2494630840/001/store/4s793gbs03isfy0b7fll2x3sv74vx103-file1.txt1526 URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst1527 Compression: zstd1528 NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1529 NarSize: 1601530 References: 1531 CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1532 metadata_upload_test.go:55: Retrieved .ls file from S3 (compressed size: 77 bytes)1533 metadata_upload_test.go:55: Decompressed .ls content (64 bytes):1534 {"version":1,"root":{"type":"regular","size":44,"narOffset":96}}1535--- PASS: TestMultipartUploadAbortedWhenCancelledBeforeRecorded (5.95s)1536=== CONT TestTombstonedObjectOfferedWithoutWaiting1537=== NAME TestNARDeduplicationMetadataUploadBug1538 metadata_upload_test.go:64: Second store path (same content): /private/var/tb/5/scratch/67d206d5acc7e9e5/build/TestNARDeduplicationMetadataUploadBug2494630840/001/store/vxnf89znfsnyq0wy299mzsf4llpd7v31-file2.txt15392026/11/11 23:58:29 INFO Received push request method=POST path=/api/pushes15402026/11/11 23:58:29 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)15412026/11/11 23:58:29 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted,O=Other"15422026/11/11 23:58:29 WARN mTLS auth: bound subjects configured but subject DN unavailable15432026/11/11 23:58:29 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted"1544--- PASS: TestService_AuthMiddleware_MTLSBoundSubjects (5.67s)1545=== CONT TestUploadHandlersRejectInvalidKeys1546=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info15472026/11/11 23:58:29 INFO Received sign narinfos request method=POST path=/api/pushes/2/sign1548=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info1549=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal1550=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal15512026/11/11 23:58:29 WARN Failed to register uploaded object key=vxnf89znfsnyq0wy299mzsf4llpd7v31.ls error="server returned 404: 404 page not found\n"1552=== RUN TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key1553=== PAUSE TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key1554=== RUN TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key1555=== PAUSE TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key1556=== CONT TestService_cleanupPendingClosuresHandler15572026/11/11 23:58:29 INFO Signed narinfos id=2 count=115582026/11/11 23:58:29 INFO Uploading 1 narinfos15592026/11/11 23:58:29 INFO Received complete push request method=POST path=/api/pushes/2/complete15602026/11/11 23:58:29 WARN Failed to register uploaded object key=vxnf89znfsnyq0wy299mzsf4llpd7v31.narinfo error="server returned 404: 404 page not found\n"15612026/11/11 23:58:29 INFO Completed push id=2 objects=3 duration=893.792µs15622026/11/11 23:58:29 INFO Upload complete. (155ms)1563=== NAME TestNARDeduplicationMetadataUploadBug1564 metadata_upload_test.go:76: Retrieved narinfo from S3:1565 StorePath: /private/var/tb/5/scratch/67d206d5acc7e9e5/build/TestNARDeduplicationMetadataUploadBug2494630840/001/store/vxnf89znfsnyq0wy299mzsf4llpd7v31-file2.txt1566 URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst1567 Compression: zstd1568 NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1569 NarSize: 1601570 References: 1571 CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1572 metadata_upload_test.go:77: Retrieved .ls file from S3 (compressed size: 62 bytes)1573 metadata_upload_test.go:77: Decompressed .ls content (49 bytes):1574 {"version":1,"root":{"type":"regular","size":44}}1575--- PASS: TestNARDeduplicationMetadataUploadBug (6.91s)1576=== CONT TestCreatePendingClosureVerifyS3FailureReleasesConnection15772026/11/11 23:58:30 INFO Received uploads request method=POST path=/api/pending_closures15782026/11/11 23:58:30 INFO Received cleanup request method=DELETE path=/api/pending_closures15792026/11/11 23:58:30 WARN Failed to abort upload, keeping its closure key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst error="Get \"http://127.0.0.1:1/bucket56/?location=\": dial tcp 127.0.0.1:1: connect: connection refused" code=""15802026/11/11 23:58:30 INFO Aborted multipart uploads count=0 kept=115812026/11/11 23:58:30 INFO Received cleanup request method=DELETE path=/api/pending_closures15822026/11/11 23:58:30 INFO Aborted multipart uploads count=1 kept=01583--- PASS: TestMultipartCleanup (6.39s)1584=== CONT TestUploadHandlersRejectOversizedBody1585=== RUN TestUploadHandlersRejectOversizedBody/request_more_parts1586=== PAUSE TestUploadHandlersRejectOversizedBody/request_more_parts1587=== RUN TestUploadHandlersRejectOversizedBody/create_pending_closure1588=== PAUSE TestUploadHandlersRejectOversizedBody/create_pending_closure1589=== RUN TestUploadHandlersRejectOversizedBody/complete_multipart1590=== PAUSE TestUploadHandlersRejectOversizedBody/complete_multipart1591=== CONT TestGCSweepSkipsPendingObjects15922026/11/11 23:58:30 WARN Rate limiter enabled after throttle name=s3-test rate=515932026/11/11 23:58:30 WARN S3 rate limit hit operation="complete multipart upload" error="Please reduce your request rate."1594=== NAME TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle1595 throttle_test.go:213: Proxy stats: total=13, throttled=10, completeMultipart=101596 throttle_test.go:215: Rate limiter: enabled=true, rate=5.001597--- PASS: TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle (8.57s)1598=== CONT TestGCSweepDeliversEachKeyOnce1599--- PASS: TestService_AuthMiddleware_MTLSProxyHeader (5.98s)1600=== CONT TestGCAdvisoryLockBlocksConcurrentRun16012026/11/11 23:58:31 WARN mTLS auth: subject not in bound subjects subject="CN=reader"16022026/11/11 23:58:31 WARN mTLS auth: subject not in bound subjects subject="CN=reader"1603--- PASS: TestService_NativeMTLS (6.26s)1604=== CONT TestGCMetrics1605--- PASS: TestObjectStatsUpdatedOncePerCommit (6.15s)1606=== CONT TestGCBugBareHashReferences16072026/11/11 23:58:32 INFO Received push request method=POST path=/api/pushes16082026/11/11 23:58:32 INFO Received complete push request method=POST path=/api/pushes/1/complete16092026/11/11 23:58:32 INFO Completed push id=1 objects=4 duration=60.544333ms16102026/11/11 23:58:32 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete16112026/11/11 23:58:32 INFO Completed upload id=2 objects=2 duration=1.3755ms1612--- PASS: TestMetricsCommit (6.18s)1613=== CONT TestIsValidUploadKey1614=== RUN TestIsValidUploadKey/narinfo1615=== PAUSE TestIsValidUploadKey/narinfo1616=== RUN TestIsValidUploadKey/nar_zst1617=== PAUSE TestIsValidUploadKey/nar_zst1618=== RUN TestIsValidUploadKey/nar_xz1619=== PAUSE TestIsValidUploadKey/nar_xz1620=== RUN TestIsValidUploadKey/nar_plain1621=== PAUSE TestIsValidUploadKey/nar_plain1622=== RUN TestIsValidUploadKey/listing1623=== PAUSE TestIsValidUploadKey/listing1624=== RUN TestIsValidUploadKey/build_log1625=== PAUSE TestIsValidUploadKey/build_log1626=== RUN TestIsValidUploadKey/build_log_home-manager_file1627=== PAUSE TestIsValidUploadKey/build_log_home-manager_file1628=== RUN TestIsValidUploadKey/build_log_plus_in_name1629=== PAUSE TestIsValidUploadKey/build_log_plus_in_name1630=== RUN TestIsValidUploadKey/build_log_question_mark1631=== PAUSE TestIsValidUploadKey/build_log_question_mark1632=== RUN TestIsValidUploadKey/build_log_equals1633=== PAUSE TestIsValidUploadKey/build_log_equals1634=== RUN TestIsValidUploadKey/realisation1635=== PAUSE TestIsValidUploadKey/realisation1636=== RUN TestIsValidUploadKey/realisation_plus_in_output1637=== PAUSE TestIsValidUploadKey/realisation_plus_in_output1638=== RUN TestIsValidUploadKey/nix-cache-info1639=== PAUSE TestIsValidUploadKey/nix-cache-info1640=== RUN TestIsValidUploadKey/index.html1641=== PAUSE TestIsValidUploadKey/index.html1642=== RUN TestIsValidUploadKey/narinfo_key,_nar_type1643=== PAUSE TestIsValidUploadKey/narinfo_key,_nar_type1644=== RUN TestIsValidUploadKey/nar_key,_narinfo_type1645=== PAUSE TestIsValidUploadKey/nar_key,_narinfo_type1646=== RUN TestIsValidUploadKey/listing_key,_narinfo_type1647=== PAUSE TestIsValidUploadKey/listing_key,_narinfo_type1648=== RUN TestIsValidUploadKey/traversal1649=== PAUSE TestIsValidUploadKey/traversal1650=== RUN TestIsValidUploadKey/traversal_nar1651=== PAUSE TestIsValidUploadKey/traversal_nar1652=== RUN TestIsValidUploadKey/absolute1653=== PAUSE TestIsValidUploadKey/absolute1654=== RUN TestIsValidUploadKey/empty_key1655=== PAUSE TestIsValidUploadKey/empty_key1656=== RUN TestIsValidUploadKey/unknown_type1657=== PAUSE TestIsValidUploadKey/unknown_type1658=== CONT TestPush_SkippedKeySurvivesGCBeforeCommit1659--- PASS: TestMetricsInventory (6.39s)1660=== CONT TestPendingClosureWriteTimeout1661=== RUN TestPendingClosureWriteTimeout/empty1662=== PAUSE TestPendingClosureWriteTimeout/empty1663=== RUN TestPendingClosureWriteTimeout/400_objects1664=== PAUSE TestPendingClosureWriteTimeout/400_objects1665=== RUN TestPendingClosureWriteTimeout/670k_objects1666=== PAUSE TestPendingClosureWriteTimeout/670k_objects1667=== RUN TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline1668=== PAUSE TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline1669=== CONT TestService_healthCheckHandler1670=== NAME TestOrphanedObjectsGC1671 orphaned_objects_gc_test.go:287: GC Test Summary:1672 orphaned_objects_gc_test.go:288: - Kept: 2 objects from closure A1673 orphaned_objects_gc_test.go:289: - Deleted: 2 objects from closure B1674 orphaned_objects_gc_test.go:290: - Deleted: 6 orphaned chain objects (X1->X2->X3)1675 orphaned_objects_gc_test.go:291: - Deleted: 2 orphaned single objects (Y)1676 orphaned_objects_gc_test.go:292: - Total deleted: 10 objects1677--- PASS: TestOrphanedObjectsGC (8.40s)1678=== CONT TestCreatePendingClosureRejectsOversizedNAR16792026/11/11 23:58:33 INFO Received uploads request method=POST path=/api/pending_closures1680--- PASS: TestCreatePendingClosureRejectsOversizedNAR (0.00s)1681=== CONT TestCacheConfigHandlerMaxNarSize1682--- PASS: TestCacheConfigHandlerMaxNarSize (0.00s)1683=== CONT TestEnsureAbortUploadsRule1684--- PASS: TestObjectStatsTrigger (6.87s)1685=== CONT TestGenerateLandingPage1686--- PASS: TestGenerateLandingPage (0.00s)1687=== CONT TestService_readinessHandler16882026/11/11 23:58:33 INFO Received uploads request method=POST path=/api/pending_closures16892026/11/11 23:58:33 INFO Received uploads request method=POST path=/api/pending_closures16902026/11/11 23:58:33 INFO Received cleanup request method=DELETE path=/api/pending_closures16912026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=016922026/11/11 23:58:34 WARN Force mode enabled - objects will be deleted immediately without grace period16932026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=016942026/11/11 23:58:34 INFO Vacuumed table table=pending_closures16952026/11/11 23:58:34 INFO Vacuumed table table=pending_objects16962026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads16972026/11/11 23:58:34 INFO Vacuumed table table=closures16982026/11/11 23:58:34 INFO Vacuumed table table=objects16992026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=017002026/11/11 23:58:34 WARN Force mode enabled - objects will be deleted immediately without grace period17012026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=017022026/11/11 23:58:34 WARN Force mode enabled - objects will be deleted immediately without grace period17032026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=017042026/11/11 23:58:34 INFO Vacuumed table table=pending_closures17052026/11/11 23:58:34 INFO Vacuumed table table=pending_objects17062026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads17072026/11/11 23:58:34 INFO Vacuumed table table=closures17082026/11/11 23:58:34 INFO Vacuumed table table=objects1709--- PASS: TestSweepSkipsRowsLockedByPush (7.50s)1710=== CONT TestDeduplicatedObjectsRecordedAsPending17112026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=017122026/11/11 23:58:34 INFO Vacuumed table table=pending_closures17132026/11/11 23:58:34 INFO Vacuumed table table=pending_objects17142026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads17152026/11/11 23:58:34 INFO Vacuumed table table=closures17162026/11/11 23:58:34 INFO Vacuumed table table=objects1717--- PASS: TestSweepRechecksPendingAfterLocking (7.36s)1718=== CONT TestLeadEndsOnShutdown1719--- PASS: TestGetExistingObjectsLocksRows (7.02s)1720=== CONT TestPushDedupSurvivesConcurrentGC1721--- PASS: TestRegisterCompletedObjectNeedsPendingRow (6.99s)1722=== CONT TestLeadGenerationGrowsWithEveryGrant17232026/11/11 23:58:35 INFO Received push request method=POST path=/api/pushes17242026/11/11 23:58:35 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign17252026/11/11 23:58:35 INFO Signed narinfos id=1 count=117262026/11/11 23:58:35 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign17272026/11/11 23:58:35 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign17282026/11/11 23:58:35 ERROR Failed to sign narinfos signer="index 1" error="signing key cannot be nil"17292026/11/11 23:58:35 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign17302026/11/11 23:58:35 ERROR Failed to sign narinfos signer=niks3-test-1 error="signer failed"17312026/11/11 23:58:35 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign17322026/11/11 23:58:35 ERROR Signer returned unexpected signature count signer=niks3-test-1 got=0 want=117332026/11/11 23:58:35 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign17342026/11/11 23:58:35 ERROR Signer returned unexpected object key signer=niks3-test-1 object_key=unexpected.narinfo1735--- PASS: TestPush_SignsNarinfosOfItsPendingObjects (7.18s)1736=== CONT TestLeadIncumbentAnnouncesAtOnce17372026/11/11 23:58:35 INFO Received uploads request method=POST path=/api/pending_closures17382026/11/11 23:58:36 INFO Received uploads request method=POST path=/api/pending_closures17392026/11/11 23:58:36 INFO Received complete multipart upload request method=POST path=/api/multipart/complete17402026/11/11 23:58:36 WARN CompleteMultipartUpload errored but object exists; treating as success error="The specified key does not exist." object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=ZDM5YmExODItZjViYi00ZTI4LWIwMTgtYWUzNjdkYzQxMGVhLjNhMzhmNTMzLTYzZDItNGRlNy1iNTFmLTQxZjJmZmNiY2RmNngxNzk0NDQxNTE1ODQ5Njk3MDAw17412026/11/11 23:58:36 INFO Received uploads request method=POST path=/api/pending_closures17422026/11/11 23:58:36 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=ZDM5YmExODItZjViYi00ZTI4LWIwMTgtYWUzNjdkYzQxMGVhLjNhMzhmNTMzLTYzZDItNGRlNy1iNTFmLTQxZjJmZmNiY2RmNngxNzk0NDQxNTE1ODQ5Njk3MDAw parts=11743--- PASS: TestCompleteMultipartUpload_ErrorButObjectExists (7.40s)1744=== CONT TestGCTaskStore_PhaseUpdates1745--- PASS: TestGCTaskStore_PhaseUpdates (0.00s)1746=== CONT TestGracefulShutdownDrainsInflight17472026/11/11 23:58:36 INFO Starting HTTP server address=127.0.0.1:6147317482026/11/11 23:58:36 INFO Shutdown signal received, draining in-flight requests timeout=10s1749--- PASS: TestGracefulShutdownDrainsInflight (0.07s)1750=== CONT TestGCTaskStore_Fail1751--- PASS: TestGCTaskStore_Fail (0.00s)1752=== CONT TestGCTaskStore_CompletedAllowsNewTask1753--- PASS: TestGCTaskStore_CompletedAllowsNewTask (0.00s)1754=== CONT TestGCTaskStore_GetReturnsLatest1755--- PASS: TestGCTaskStore_GetReturnsLatest (0.00s)1756=== CONT TestService_AuthMiddleware17572026/11/11 23:58:36 INFO Received uploads request method=POST path=/api/pending_closures17582026/11/11 23:58:36 INFO Received uploads request method=POST path=/api/pending_closures1759--- PASS: TestPushWaitsForSweepHoldingRow (7.83s)1760=== CONT TestResolveDBConnectionString/nothing_configured1761=== CONT TestResolveDBConnectionString/missing_file_is_an_error1762=== CONT TestResolveDBConnectionString/PGHOST_allows_empty1763=== CONT TestResolveDBConnectionString/file_when_flag_empty1764=== CONT TestResolveDBConnectionString/flag_wins1765=== CONT TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed1766--- PASS: TestResolveDBConnectionString (0.03s)1767 --- PASS: TestResolveDBConnectionString/nothing_configured (0.00s)1768 --- PASS: TestResolveDBConnectionString/missing_file_is_an_error (0.00s)1769 --- PASS: TestResolveDBConnectionString/PGHOST_allows_empty (0.00s)1770 --- PASS: TestResolveDBConnectionString/file_when_flag_empty (0.00s)1771 --- PASS: TestResolveDBConnectionString/flag_wins (0.00s)17722026/11/11 23:58:37 INFO Aborted multipart uploads count=0 kept=017732026/11/11 23:58:37 WARN Force mode enabled - objects will be deleted immediately without grace period17742026/11/11 23:58:37 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."17752026/11/11 23:58:37 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."1776--- PASS: TestFailedS3DeleteKeepsTombstone (8.00s)1777=== CONT TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed1778=== RUN TestPush_RejectsBadRequests/root_not_in_objects1779=== PAUSE TestPush_RejectsBadRequests/root_not_in_objects1780=== RUN TestPush_RejectsBadRequests/no_roots1781=== PAUSE TestPush_RejectsBadRequests/no_roots1782=== RUN TestPush_RejectsBadRequests/no_objects1783=== PAUSE TestPush_RejectsBadRequests/no_objects1784=== RUN TestPush_RejectsBadRequests/bad_root1785=== PAUSE TestPush_RejectsBadRequests/bad_root1786=== CONT TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed17872026/11/11 23:58:37 INFO Received uploads request method=POST path=/api/pending_closures1788--- PASS: TestTombstonedObjectOfferedWithoutWaiting (7.91s)1789=== CONT TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost17902026/11/11 23:58:37 INFO Received cleanup request method=DELETE path=/api/pending_closures17912026/11/11 23:58:37 INFO Aborted multipart uploads count=0 kept=017922026/11/11 23:58:37 INFO Received uploads request method=POST path=/api/pending_closures17932026/11/11 23:58:38 INFO Received cleanup request method=DELETE path=/api/pending_closures17942026/11/11 23:58:38 INFO Aborted multipart uploads count=1 kept=017952026/11/11 23:58:38 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete17962026-11-11 23:58:38.108 UTC [306] ERROR: Closure does not exist: id=117972026-11-11 23:58:38.108 UTC [306] CONTEXT: PL/pgSQL function commit_pending_closure(bigint) line 13 at RAISE17982026-11-11 23:58:38.108 UTC [306] STATEMENT: -- name: CommitPendingClosure :one1799 SELECT commit_pending_closure($1::bigint)::bigint1800 1801--- PASS: TestService_cleanupPendingClosuresHandler (8.21s)1802=== CONT TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed18032026/11/11 23:58:38 INFO Received uploads request method=POST path=/api/pending_closures18042026/11/11 23:58:38 INFO Received complete multipart upload request method=POST path=/api/multipart/complete18052026/11/11 23:58:38 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0100000000000000000000.nar.zst upload_id=ZDM5YmExODItZjViYi00ZTI4LWIwMTgtYWUzNjdkYzQxMGVhLmUxZGIxNTAzLTc3MTQtNDExMi05ZWI1LWRlZGRhOWRkNWNhY3gxNzk0NDQxNTE2MTQzNzY2MDAw parts=121806--- PASS: TestCreatePendingClosureVerifyS3FailureReleasesConnection (8.48s)1807=== CONT TestProxyWriteTimeout/narinfo1808=== CONT TestProxyWriteTimeout/unknown_size1809=== CONT TestGCEndsOnShutdown/before_the_run18102026/11/11 23:58:38 INFO Received uploads request method=POST path=/api/pending_closures18112026/11/11 23:58:38 INFO Aborted multipart uploads count=0 kept=018122026/11/11 23:58:38 WARN Force mode enabled - objects will be deleted immediately without grace period18132026/11/11 23:58:38 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=018142026/11/11 23:58:38 INFO Vacuumed table table=pending_closures18152026/11/11 23:58:38 INFO Vacuumed table table=pending_objects18162026/11/11 23:58:38 INFO Vacuumed table table=multipart_uploads18172026/11/11 23:58:39 INFO Vacuumed table table=closures18182026/11/11 23:58:39 INFO Vacuumed table table=objects1819--- PASS: TestGCSweepSkipsPendingObjects (8.51s)1820=== CONT TestProxyWriteTimeout/10_GiB_nar1821=== CONT TestProxyWriteTimeout/1_GiB_nar1822--- PASS: TestProxyWriteTimeout (0.00s)1823 --- PASS: TestProxyWriteTimeout/narinfo (0.00s)1824 --- PASS: TestProxyWriteTimeout/unknown_size (0.00s)1825 --- PASS: TestProxyWriteTimeout/10_GiB_nar (0.00s)1826 --- PASS: TestProxyWriteTimeout/1_GiB_nar (0.00s)1827=== CONT TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark18282026/11/11 23:58:39 INFO Aborted multipart uploads count=0 kept=018292026/11/11 23:58:39 WARN Force mode enabled - objects will be deleted immediately without grace period18302026/11/11 23:58:39 ERROR failed to check GC advisory lock error="query pg_locks: failed to connect to `user=_tribuchetbld5 database=niks3`: /nonexistent/.s.PGSQL.5432 (/nonexistent): dial error: dial unix /nonexistent/.s.PGSQL.5432: connect: no such file or directory"1831--- PASS: TestGCAdvisoryLockBlocksConcurrentRun (8.62s)1832=== CONT TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete18332026/11/11 23:58:39 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=018342026/11/11 23:58:39 INFO Vacuumed table table=pending_closures18352026/11/11 23:58:39 INFO Vacuumed table table=pending_objects18362026/11/11 23:58:39 INFO Aborted multipart uploads count=1 kept=018372026/11/11 23:58:39 INFO Vacuumed table table=multipart_uploads18382026/11/11 23:58:39 INFO Vacuumed table table=closures18392026/11/11 23:58:39 INFO Received cleanup request method=DELETE path=/api/pending_closures18402026/11/11 23:58:39 INFO Aborted multipart uploads count=0 kept=018412026/11/11 23:58:39 INFO Vacuumed table table=objects1842--- PASS: TestGCSweepDeliversEachKeyOnce (9.28s)18432026/11/11 23:58:39 INFO Aborted multipart uploads count=1 kept=01844=== CONT TestIsValidCachePath/narinfo18452026/11/11 23:58:39 WARN Force mode enabled - objects will be deleted immediately without grace period1846=== CONT TestIsValidCachePath/index.html1847=== CONT TestIsValidCachePath/traversal_parent1848=== CONT TestIsValidCachePath/nix-cache-info1849=== CONT TestIsValidCachePath/realisation1850=== CONT TestIsValidCachePath/log1851=== CONT TestIsValidCachePath/ls1852=== CONT TestIsValidCachePath/nar_uncompressed1853=== CONT TestIsValidCachePath/nar_bz21854=== CONT TestIsValidCachePath/nar_xz1855=== CONT TestIsValidCachePath/nar_zst1856=== CONT TestIsValidCachePath/narinfo_all_nix_base32_chars1857=== CONT TestIsValidCachePath/empty1858=== CONT TestIsValidCachePath/short_hash1859=== CONT TestIsValidCachePath/leading_slash1860--- PASS: TestPendingCleanupUsesOneCutoff (13.00s)1861=== CONT TestIsValidCachePath/wrong_extension1862=== CONT TestIsValidCachePath/random_path1863=== CONT TestIsValidCachePath/invalid_char_u1864=== CONT TestIsValidCachePath/invalid_char_e1865=== CONT TestIsValidCachePath/traversal_in_middle18662026/11/11 23:58:39 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=01867=== CONT TestParseSingleRange/open-ended1868--- PASS: TestIsValidCachePath (0.00s)1869 --- PASS: TestIsValidCachePath/narinfo (0.00s)1870 --- PASS: TestIsValidCachePath/index.html (0.00s)1871 --- PASS: TestIsValidCachePath/traversal_parent (0.00s)1872 --- PASS: TestIsValidCachePath/nix-cache-info (0.00s)1873 --- PASS: TestIsValidCachePath/realisation (0.00s)1874 --- PASS: TestIsValidCachePath/log (0.00s)1875 --- PASS: TestIsValidCachePath/ls (0.00s)1876 --- PASS: TestIsValidCachePath/nar_uncompressed (0.00s)1877 --- PASS: TestIsValidCachePath/nar_bz2 (0.00s)1878 --- PASS: TestIsValidCachePath/nar_xz (0.00s)1879 --- PASS: TestIsValidCachePath/nar_zst (0.00s)1880 --- PASS: TestIsValidCachePath/narinfo_all_nix_base32_chars (0.00s)1881 --- PASS: TestIsValidCachePath/empty (0.00s)1882 --- PASS: TestIsValidCachePath/short_hash (0.00s)1883 --- PASS: TestIsValidCachePath/leading_slash (0.00s)1884 --- PASS: TestIsValidCachePath/wrong_extension (0.00s)1885 --- PASS: TestIsValidCachePath/random_path (0.00s)1886 --- PASS: TestIsValidCachePath/invalid_char_u (0.00s)1887 --- PASS: TestIsValidCachePath/traversal_in_middle (0.00s)1888 --- PASS: TestIsValidCachePath/invalid_char_e (0.00s)1889=== CONT TestParseSingleRange/none1890=== CONT TestParseSingleRange/start_far_past_EOF1891=== CONT TestParseSingleRange/single_byte1892=== CONT TestParseSingleRange/start_past_EOF1893=== CONT TestParseSingleRange/suffix_exceeds_size18942026/11/11 23:58:40 INFO Vacuumed table table=pending_closures1895=== CONT TestParseSingleRange/end_clamped_to_size1896=== CONT TestParseSingleRange/malformed_both_empty1897=== CONT TestParseSingleRange/suffix1898=== CONT TestParseSingleRange/closed1899=== CONT TestParseSingleRange/malformed_end_before_start1900=== CONT TestParseSingleRange/multi-range_ignored1901=== CONT TestParseSingleRange/malformed_no_dash1902=== CONT TestParseSingleRange/unknown_unit1903=== CONT TestClientErrorHandling/InvalidStorePath1904=== CONT TestClientErrorHandling/ServerNotAvailable1905--- PASS: TestParseSingleRange (0.00s)1906 --- PASS: TestParseSingleRange/open-ended (0.00s)1907 --- PASS: TestParseSingleRange/none (0.00s)1908 --- PASS: TestParseSingleRange/start_far_past_EOF (0.00s)1909 --- PASS: TestParseSingleRange/start_past_EOF (0.00s)1910 --- PASS: TestParseSingleRange/single_byte (0.00s)1911 --- PASS: TestParseSingleRange/suffix_exceeds_size (0.00s)1912 --- PASS: TestParseSingleRange/end_clamped_to_size (0.00s)1913 --- PASS: TestParseSingleRange/suffix (0.00s)1914 --- PASS: TestParseSingleRange/closed (0.00s)1915 --- PASS: TestParseSingleRange/malformed_end_before_start (0.00s)1916 --- PASS: TestParseSingleRange/multi-range_ignored (0.00s)1917 --- PASS: TestParseSingleRange/malformed_both_empty (0.00s)1918 --- PASS: TestParseSingleRange/malformed_no_dash (0.00s)1919 --- PASS: TestParseSingleRange/unknown_unit (0.00s)19202026/11/11 23:58:40 INFO Vacuumed table table=pending_objects19212026/11/11 23:58:40 INFO Vacuumed table table=multipart_uploads19222026/11/11 23:58:40 INFO Vacuumed table table=closures19232026/11/11 23:58:40 INFO Vacuumed table table=objects1924--- PASS: TestGCMetrics (8.67s)1925=== CONT TestClientErrorHandling/InvalidAuthToken19262026/11/11 23:58:40 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://localhost:19999/api/objects/present\": dial tcp 127.0.0.1:19999: connect: connection refused" url=http://localhost:19999/api/objects/present19272026/11/11 23:58:40 WARN Request failed, retrying attempt=2 backoff=197.248106ms error="Post \"http://localhost:19999/api/objects/present\": dial tcp 127.0.0.1:19999: connect: connection refused" url=http://localhost:19999/api/objects/present1928--- PASS: TestGCBugBareHashReferences (8.46s)1929=== CONT TestCacheConfigHandler/full_config,_no_issuer1930=== CONT TestCacheConfigHandler/no_signing_keys1931=== CONT TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1932=== CONT TestCacheConfigHandler/no_cache_url_configured1933--- PASS: TestCacheConfigHandler (0.01s)1934 --- PASS: TestCacheConfigHandler/full_config,_no_issuer (0.00s)1935 --- PASS: TestCacheConfigHandler/no_signing_keys (0.00s)1936 --- PASS: TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator (0.00s)1937 --- PASS: TestCacheConfigHandler/no_cache_url_configured (0.00s)1938=== CONT TestPresentNotReportedWhileGCDeletesClosure/stale19392026/11/11 23:58:40 INFO Received push request method=POST path=/api/pushes19402026/11/11 23:58:40 INFO Received complete multipart upload request method=POST path=/api/multipart/complete19412026/11/11 23:58:40 WARN Request failed, retrying attempt=3 backoff=421.252543ms error="Post \"http://localhost:19999/api/objects/present\": dial tcp 127.0.0.1:19999: connect: connection refused" url=http://localhost:19999/api/objects/present19422026/11/11 23:58:40 INFO Received complete push request method=POST path=/api/pushes/1/complete19432026/11/11 23:58:40 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0100000000000000000000.nar.zst upload_id=ZDM5YmExODItZjViYi00ZTI4LWIwMTgtYWUzNjdkYzQxMGVhLjBjYjczOGRkLTZkNDgtNDk2Ny1hYzJhLWFjMGY0Nzc1MTY3YXgxNzk0NDQxNTE2MjAyNzU3MDAw parts=121944--- PASS: TestSharedNarBothUploadsComplete (11.83s)1945=== CONT TestPresentNotReportedWhileGCDeletesClosure/fresh19462026/11/11 23:58:40 INFO Completed push id=1 objects=4 duration=41.703625ms19472026/11/11 23:58:40 INFO Received push request method=POST path=/api/pushes19482026/11/11 23:58:40 INFO Aborted multipart uploads count=0 kept=019492026/11/11 23:58:40 WARN Force mode enabled - objects will be deleted immediately without grace period19502026/11/11 23:58:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=2 objects-deleted-after-grace-period=2 objects-failed-to-delete=01951--- PASS: TestService_healthCheckHandler (8.31s)1952=== CONT TestServerTLSConfig/no_client_CA1953=== CONT TestServerTLSConfig/not_a_PEM_file1954=== CONT TestServerTLSConfig/missing_CA_file1955--- PASS: TestServerTLSConfig (0.00s)1956 --- PASS: TestServerTLSConfig/no_client_CA (0.00s)1957 --- PASS: TestServerTLSConfig/not_a_PEM_file (0.00s)1958 --- PASS: TestServerTLSConfig/missing_CA_file (0.00s)1959=== CONT TestService_RequireScope_OIDC/builder_may_write1960=== CONT TestService_RequireScope_OIDC/reader_may_read1961=== CONT TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1962=== CONT TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read19632026/11/11 23:58:41 WARN Authentication failed token_preview=wrong token_length=5 oidc_error="no provider could verify the token (signature or issuer mismatch)" oidc_provider="" tried_providers=[test]1964=== CONT TestService_RequireScope_OIDC/static_token_may_admin_with_netrc19652026/11/11 23:58:41 INFO Vacuumed table table=pending_closures1966=== CONT TestService_RequireScope_OIDC/reader_may_read_with_netrc1967=== CONT TestService_RequireScope_OIDC/anonymous_may_not_read1968=== CONT TestService_RequireScope_OIDC/writer_implies_read1969=== CONT TestService_RequireScope_OIDC/reader_may_not_write1970=== CONT TestService_RequireScope_OIDC/static_token_may_write1971=== CONT TestService_RequireScope_OIDC/static_token_may_admin1972=== CONT TestService_RequireScope_OIDC/ops_may_admin1973=== CONT TestService_RequireScope_OIDC/ops_may_not_write1974=== CONT TestService_RequireScope_OIDC/builder_may_not_admin1975=== CONT TestService_AuthMiddleware_OIDC/valid_OIDC_token19762026/11/11 23:58:41 WARN Request failed, retrying attempt=4 backoff=780.132934ms error="Post \"http://localhost:19999/api/objects/present\": dial tcp 127.0.0.1:19999: connect: connection refused" url=http://localhost:19999/api/objects/present1977=== CONT TestService_AuthMiddleware_OIDC/malformed_token_rejected19782026/11/11 23:58:41 WARN Authentication failed token_preview=not-a-valid-jwt token_length=15 oidc_error="no provider could verify the token (signature or issuer mismatch)" oidc_provider="" tried_providers=[test]1979=== CONT TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1980=== CONT TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected19812026/11/11 23:58:41 INFO Vacuumed table table=pending_objects1982--- PASS: TestService_RequireScope_OIDC (5.69s)1983 --- PASS: TestService_RequireScope_OIDC/builder_may_write (0.00s)1984 --- PASS: TestService_RequireScope_OIDC/reader_may_read (0.00s)1985 --- PASS: TestService_RequireScope_OIDC/empty_netrc_password_may_not_read (0.00s)1986 --- PASS: TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read (0.00s)1987 --- PASS: TestService_RequireScope_OIDC/static_token_may_admin_with_netrc (0.00s)1988 --- PASS: TestService_RequireScope_OIDC/reader_may_read_with_netrc (0.00s)1989 --- PASS: TestService_RequireScope_OIDC/anonymous_may_not_read (0.00s)1990 --- PASS: TestService_RequireScope_OIDC/writer_implies_read (0.00s)1991 --- PASS: TestService_RequireScope_OIDC/reader_may_not_write (0.00s)1992 --- PASS: TestService_RequireScope_OIDC/static_token_may_write (0.00s)1993 --- PASS: TestService_RequireScope_OIDC/static_token_may_admin (0.00s)1994 --- PASS: TestService_RequireScope_OIDC/ops_may_admin (0.00s)1995 --- PASS: TestService_RequireScope_OIDC/ops_may_not_write (0.00s)1996 --- PASS: TestService_RequireScope_OIDC/builder_may_not_admin (0.00s)19972026/11/11 23:58:41 WARN Authentication failed token_preview=eyJhbGciOi...GOuw5ZLR1g token_length=701 oidc_error="no rule matched: claim \"repository_owner\" value [otherorg] not in allowed patterns [myorg]" oidc_provider=test tried_providers=[test]1998=== CONT TestForceGCDuringPushOffersSweptObject/before_pending_rows1999--- PASS: TestService_AuthMiddleware_OIDC (6.13s)2000 --- PASS: TestService_AuthMiddleware_OIDC/valid_OIDC_token (0.01s)
loading 667 hidden lines…
26682026/11/11 23:59:03 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."26692026/11/11 23:59:03 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."26702026/11/11 23:59:03 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."26712026/11/11 23:59:03 INFO Aborted multipart uploads count=0 kept=026722026/11/11 23:59:03 WARN Force mode enabled - objects will be deleted immediately without grace period26732026/11/11 23:59:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=9 objects-failed-to-delete=026742026/11/11 23:59:03 INFO Vacuumed table table=objects26752026/11/11 23:59:03 INFO Vacuumed table table=pending_closures26762026/11/11 23:59:03 INFO Vacuumed table table=pending_objects26772026/11/11 23:59:03 INFO Vacuumed table table=multipart_uploads26782026/11/11 23:59:03 INFO Vacuumed table table=closures26792026/11/11 23:59:03 INFO Received push request method=POST path=/26802026/11/11 23:59:03 INFO Vacuumed table table=objects26812026/11/11 23:59:03 INFO Received push request method=POST path=/26822026/11/11 23:59:03 INFO Received push request method=POST path=/26832026/11/11 23:59:03 INFO Received uploads request method=POST path=/26842026/11/11 23:59:03 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."26852026/11/11 23:59:03 INFO Received push request method=POST path=/26862026/11/11 23:59:03 INFO Received push request method=POST path=/26872026/11/11 23:59:03 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."26882026/11/11 23:59:03 INFO Received complete upload request method=POST path=/26892026/11/11 23:59:03 INFO Aborted multipart uploads count=0 kept=026902026/11/11 23:59:03 INFO Received uploads request method=POST path=/26912026/11/11 23:59:03 INFO Completed upload id=3 objects=25 duration=543.792µs26922026/11/11 23:59:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=026932026/11/11 23:59:03 INFO Received push request method=POST path=/26942026/11/11 23:59:03 INFO Aborted multipart uploads count=0 kept=026952026/11/11 23:59:03 WARN Force mode enabled - objects will be deleted immediately without grace period26962026/11/11 23:59:03 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=3 objects-failed-to-delete=026972026/11/11 23:59:03 INFO Vacuumed table table=pending_closures26982026/11/11 23:59:03 INFO Vacuumed table table=pending_closures26992026/11/11 23:59:03 INFO Vacuumed table table=pending_objects27002026/11/11 23:59:03 INFO Vacuumed table table=multipart_uploads27012026/11/11 23:59:03 INFO Vacuumed table table=closures27022026/11/11 23:59:03 INFO Vacuumed table table=objects27032026/11/11 23:59:03 INFO Vacuumed table table=pending_objects27042026/11/11 23:59:03 INFO Vacuumed table table=multipart_uploads27052026/11/11 23:59:03 INFO Received uploads request method=POST path=/27062026/11/11 23:59:03 INFO Vacuumed table table=closures27072026/11/11 23:59:03 INFO Received uploads request method=POST path=/27082026/11/11 23:59:03 INFO Received complete upload request method=POST path=/27092026/11/11 23:59:03 INFO Completed upload id=5 objects=5 duration=732.5µs27102026/11/11 23:59:03 INFO Aborted multipart uploads count=0 kept=027112026/11/11 23:59:03 INFO Received uploads request method=POST path=/27122026/11/11 23:59:03 INFO Vacuumed table table=objects27132026/11/11 23:59:03 INFO Received uploads request method=POST path=/27142026/11/11 23:59:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027152026/11/11 23:59:03 INFO Vacuumed table table=pending_closures27162026/11/11 23:59:03 INFO Received uploads request method=POST path=/27172026/11/11 23:59:03 INFO Vacuumed table table=pending_objects27182026/11/11 23:59:03 INFO Received complete upload request method=POST path=/27192026/11/11 23:59:03 INFO Vacuumed table table=multipart_uploads27202026/11/11 23:59:03 INFO Received push request method=POST path=/27212026/11/11 23:59:03 INFO Received uploads request method=POST path=/27222026/11/11 23:59:03 INFO Completed upload id=23 objects=18 duration=733.708µs27232026/11/11 23:59:03 INFO Vacuumed table table=closures27242026/11/11 23:59:03 INFO Received uploads request method=POST path=/27252026/11/11 23:59:03 INFO Received push request method=POST path=/27262026/11/11 23:59:03 INFO Vacuumed table table=objects27272026/11/11 23:59:03 INFO Received uploads request method=POST path=/27282026/11/11 23:59:03 INFO Received uploads request method=POST path=/27292026/11/11 23:59:03 INFO Aborted multipart uploads count=0 kept=027302026/11/11 23:59:03 INFO Received push request method=POST path=/27312026/11/11 23:59:03 INFO Received complete upload request method=POST path=/27322026/11/11 23:59:03 INFO Aborted multipart uploads count=0 kept=027332026/11/11 23:59:03 WARN Force mode enabled - objects will be deleted immediately without grace period27342026/11/11 23:59:03 INFO Received complete upload request method=POST path=/27352026/11/11 23:59:03 INFO Completed upload id=3 objects=8 duration=538.583µs27362026/11/11 23:59:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027372026/11/11 23:59:03 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=027382026/11/11 23:59:03 INFO Completed upload id=22 objects=22 duration=355.042µs27392026/11/11 23:59:03 INFO Received push request method=POST path=/27402026/11/11 23:59:03 INFO Vacuumed table table=pending_closures27412026/11/11 23:59:03 INFO Vacuumed table table=pending_closures27422026/11/11 23:59:03 INFO Received uploads request method=POST path=/27432026/11/11 23:59:03 INFO Received push request method=POST path=/27442026/11/11 23:59:03 INFO Vacuumed table table=pending_objects27452026/11/11 23:59:03 INFO Vacuumed table table=pending_objects27462026/11/11 23:59:03 INFO Aborted multipart uploads count=0 kept=027472026/11/11 23:59:03 INFO Received complete push request method=POST path=/27482026/11/11 23:59:03 INFO Vacuumed table table=multipart_uploads27492026/11/11 23:59:03 INFO Vacuumed table table=closures27502026/11/11 23:59:03 INFO Completed push id=26 objects=5 duration=8.3215ms27512026/11/11 23:59:03 INFO Vacuumed table table=multipart_uploads27522026/11/11 23:59:03 INFO Received complete push request method=POST path=/27532026/11/11 23:59:03 INFO Vacuumed table table=objects27542026/11/11 23:59:03 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."27552026/11/11 23:59:03 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."27562026/11/11 23:59:03 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."27572026/11/11 23:59:03 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."27582026/11/11 23:59:03 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."27592026/11/11 23:59:03 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."27602026/11/11 23:59:03 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."27612026/11/11 23:59:03 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."27622026/11/11 23:59:03 INFO Received complete upload request method=POST path=/27632026/11/11 23:59:03 INFO Completed push id=6 objects=8 duration=1.341292ms27642026/11/11 23:59:03 INFO Received push request method=POST path=/27652026/11/11 23:59:03 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."27662026/11/11 23:59:03 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."27672026/11/11 23:59:03 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."27682026/11/11 23:59:03 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."27692026/11/11 23:59:03 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."27702026/11/11 23:59:03 ERROR failed to remove object object=realisations/sha256:aaaaaaaa!out.doi error="Access Denied."27712026/11/11 23:59:03 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."27722026/11/11 23:59:03 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."27732026/11/11 23:59:03 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."27742026/11/11 23:59:03 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."27752026/11/11 23:59:03 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."27762026/11/11 23:59:03 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."27772026/11/11 23:59:03 INFO Completed upload id=24 objects=14 duration=339.791µs27782026/11/11 23:59:03 INFO Received push request method=POST path=/27792026/11/11 23:59:03 INFO Received uploads request method=POST path=/27802026/11/11 23:59:03 INFO Received complete push request method=POST path=/27812026/11/11 23:59:03 INFO Completed push id=19 objects=3 duration=592.208µs27822026/11/11 23:59:03 INFO Received uploads request method=POST path=/27832026/11/11 23:59:03 INFO Vacuumed table table=closures27842026/11/11 23:59:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=8 objects-deleted-after-grace-period=8 objects-failed-to-delete=027852026/11/11 23:59:03 INFO Received uploads request method=POST path=/27862026/11/11 23:59:03 INFO Received uploads request method=POST path=/27872026/11/11 23:59:03 INFO Vacuumed table table=objects27882026/11/11 23:59:03 INFO Aborted multipart uploads count=0 kept=027892026/11/11 23:59:03 INFO Received uploads request method=POST path=/27902026/11/11 23:59:03 WARN Force mode enabled - objects will be deleted immediately without grace period27912026/11/11 23:59:03 INFO Received uploads request method=POST path=/27922026/11/11 23:59:03 INFO Received complete upload request method=POST path=/27932026/11/11 23:59:03 INFO Vacuumed table table=pending_closures27942026/11/11 23:59:03 INFO Completed upload id=7 objects=4 duration=541.084µs27952026/11/11 23:59:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=16 objects-failed-to-delete=027962026/11/11 23:59:03 INFO Vacuumed table table=pending_objects27972026/11/11 23:59:03 INFO Received push request method=POST path=/27982026/11/11 23:59:03 INFO Received complete upload request method=POST path=/27992026/11/11 23:59:03 INFO Vacuumed table table=multipart_uploads28002026/11/11 23:59:03 INFO Completed upload id=25 objects=4 duration=336.959µs28012026/11/11 23:59:03 INFO Vacuumed table table=pending_closures28022026/11/11 23:59:03 INFO Aborted multipart uploads count=0 kept=028032026/11/11 23:59:03 INFO Vacuumed table table=closures28042026/11/11 23:59:03 INFO Vacuumed table table=objects28052026/11/11 23:59:03 INFO Received push request method=POST path=/28062026/11/11 23:59:03 WARN Force mode enabled - objects will be deleted immediately without grace period28072026/11/11 23:59:03 INFO Vacuumed table table=pending_objects28082026/11/11 23:59:03 INFO Vacuumed table table=multipart_uploads28092026/11/11 23:59:03 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028102026/11/11 23:59:03 INFO Aborted multipart uploads count=0 kept=028112026/11/11 23:59:03 INFO Vacuumed table table=closures28122026/11/11 23:59:03 INFO Vacuumed table table=pending_closures28132026/11/11 23:59:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028142026/11/11 23:59:03 INFO Vacuumed table table=pending_closures28152026/11/11 23:59:03 INFO Vacuumed table table=objects28162026/11/11 23:59:03 INFO Received complete push request method=POST path=/28172026/11/11 23:59:03 INFO Vacuumed table table=pending_objects28182026/11/11 23:59:03 INFO Vacuumed table table=pending_objects28192026/11/11 23:59:03 INFO Vacuumed table table=multipart_uploads28202026/11/11 23:59:03 INFO Completed push id=20 objects=8 duration=316µs28212026/11/11 23:59:03 INFO Vacuumed table table=closures28222026/11/11 23:59:03 INFO Vacuumed table table=multipart_uploads28232026/11/11 23:59:03 INFO Vacuumed table table=closures28242026/11/11 23:59:03 INFO Vacuumed table table=objects28252026/11/11 23:59:03 INFO Received uploads request method=POST path=/28262026/11/11 23:59:03 INFO Vacuumed table table=objects28272026/11/11 23:59:03 INFO Received push request method=POST path=/28282026/11/11 23:59:03 INFO Received uploads request method=POST path=/28292026/11/11 23:59:03 INFO Received push request method=POST path=/28302026/11/11 23:59:03 INFO Received push request method=POST path=/28312026/11/11 23:59:03 INFO Received complete upload request method=POST path=/28322026/11/11 23:59:03 INFO Completed upload id=9 objects=7 duration=1.083584ms28332026/11/11 23:59:03 INFO Received complete push request method=POST path=/28342026/11/11 23:59:03 INFO Completed push id=27 objects=11 duration=275.5µs28352026/11/11 23:59:03 INFO Received push request method=POST path=/28362026/11/11 23:59:03 INFO Received complete push request method=POST path=/28372026/11/11 23:59:03 INFO Completed push id=8 objects=6 duration=1.818958ms28382026/11/11 23:59:03 INFO Received complete upload request method=POST path=/28392026/11/11 23:59:03 INFO Aborted multipart uploads count=0 kept=028402026/11/11 23:59:03 INFO Received push request method=POST path=/28412026/11/11 23:59:03 INFO Completed upload id=29 objects=26 duration=393.208µs28422026/11/11 23:59:03 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."28432026/11/11 23:59:03 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."28442026/11/11 23:59:03 INFO Received uploads request method=POST path=/28452026/11/11 23:59:03 INFO Received push request method=POST path=/28462026/11/11 23:59:03 INFO Received complete push request method=POST path=/28472026/11/11 23:59:03 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."28482026/11/11 23:59:03 INFO Received push request method=POST path=/28492026/11/11 23:59:03 INFO Completed push id=30 objects=8 duration=430.917µs28502026/11/11 23:59:03 INFO Received push request method=POST path=/28512026/11/11 23:59:03 INFO Received uploads request method=POST path=/28522026/11/11 23:59:03 INFO Received complete upload request method=POST path=/28532026/11/11 23:59:03 INFO Received uploads request method=POST path=/28542026/11/11 23:59:03 INFO Completed upload id=31 objects=5 duration=388.25µs28552026/11/11 23:59:03 INFO Received complete push request method=POST path=/28562026/11/11 23:59:03 INFO Received complete push request method=POST path=/28572026/11/11 23:59:03 INFO Completed push id=10 objects=2 duration=612µs28582026/11/11 23:59:03 INFO Completed push id=32 objects=3 duration=264.625µs28592026/11/11 23:59:03 INFO Aborted multipart uploads count=0 kept=028602026/11/11 23:59:03 INFO Received push request method=POST path=/28612026/11/11 23:59:03 INFO Received complete push request method=POST path=/28622026/11/11 23:59:03 WARN Force mode enabled - objects will be deleted immediately without grace period28632026/11/11 23:59:03 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028642026/11/11 23:59:03 INFO Completed push id=1 objects=11 duration=6.66275ms28652026/11/11 23:59:03 INFO Aborted multipart uploads count=0 kept=028662026/11/11 23:59:03 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=028672026/11/11 23:59:03 INFO Vacuumed table table=pending_closures28682026/11/11 23:59:03 INFO Vacuumed table table=pending_objects28692026/11/11 23:59:03 INFO Vacuumed table table=pending_closures28702026/11/11 23:59:03 INFO Vacuumed table table=multipart_uploads28712026/11/11 23:59:04 INFO Received push request method=POST path=/28722026/11/11 23:59:04 INFO Vacuumed table table=closures28732026/11/11 23:59:04 INFO Received complete upload request method=POST path=/28742026/11/11 23:59:04 INFO Vacuumed table table=pending_objects28752026/11/11 23:59:04 INFO Vacuumed table table=objects28762026/11/11 23:59:04 INFO Completed upload id=12 objects=21 duration=371.459µs28772026/11/11 23:59:04 INFO Vacuumed table table=multipart_uploads28782026/11/11 23:59:04 INFO Received push request method=POST path=/28792026/11/11 23:59:04 INFO Vacuumed table table=closures28802026/11/11 23:59:04 INFO Received push request method=POST path=/28812026/11/11 23:59:04 INFO Received push request method=POST path=/28822026/11/11 23:59:04 INFO Vacuumed table table=objects28832026/11/11 23:59:04 INFO Received complete push request method=POST path=/28842026/11/11 23:59:04 INFO Received uploads request method=POST path=/28852026/11/11 23:59:04 INFO Completed push id=13 objects=12 duration=1.192416ms28862026/11/11 23:59:04 INFO Received push request method=POST path=/28872026/11/11 23:59:04 INFO Received uploads request method=POST path=/28882026/11/11 23:59:04 INFO Aborted multipart uploads count=0 kept=028892026/11/11 23:59:04 INFO Received complete upload request method=POST path=/28902026/11/11 23:59:04 INFO Completed upload id=16 objects=3 duration=360.875µs28912026/11/11 23:59:04 INFO Aborted multipart uploads count=0 kept=028922026/11/11 23:59:04 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=028932026/11/11 23:59:04 WARN Force mode enabled - objects will be deleted immediately without grace period28942026/11/11 23:59:04 INFO Received push request method=POST path=/28952026/11/11 23:59:04 INFO Vacuumed table table=pending_closures28962026/11/11 23:59:04 INFO Received complete upload request method=POST path=/28972026/11/11 23:59:04 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028982026/11/11 23:59:04 INFO Vacuumed table table=pending_objects28992026/11/11 23:59:04 INFO Received push request method=POST path=/29002026/11/11 23:59:04 INFO Vacuumed table table=pending_closures29012026/11/11 23:59:04 INFO Vacuumed table table=multipart_uploads29022026/11/11 23:59:04 INFO Received complete push request method=POST path=/29032026/11/11 23:59:04 INFO Completed upload id=2 objects=26 duration=13.781334ms29042026/11/11 23:59:04 INFO Vacuumed table table=closures29052026/11/11 23:59:04 INFO Vacuumed table table=pending_objects29062026/11/11 23:59:04 INFO Completed push id=13 objects=23 duration=765.042µs29072026/11/11 23:59:04 INFO Vacuumed table table=multipart_uploads29082026/11/11 23:59:04 INFO Vacuumed table table=objects29092026/11/11 23:59:04 INFO Vacuumed table table=closures29102026/11/11 23:59:04 INFO Vacuumed table table=objects29112026/11/11 23:59:04 INFO Received push request method=POST path=/29122026/11/11 23:59:04 INFO Received push request method=POST path=/29132026/11/11 23:59:04 INFO Aborted multipart uploads count=0 kept=029142026/11/11 23:59:04 INFO Received complete push request method=POST path=/29152026/11/11 23:59:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=029162026/11/11 23:59:04 INFO Aborted multipart uploads count=0 kept=029172026/11/11 23:59:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029182026/11/11 23:59:04 INFO Vacuumed table table=pending_closures29192026/11/11 23:59:04 INFO Received push request method=POST path=/29202026/11/11 23:59:04 INFO Vacuumed table table=pending_closures29212026/11/11 23:59:04 INFO Vacuumed table table=pending_objects29222026/11/11 23:59:04 INFO Vacuumed table table=multipart_uploads29232026/11/11 23:59:04 INFO Completed push id=1 objects=7 duration=37.263959ms29242026/11/11 23:59:04 INFO Vacuumed table table=closures29252026/11/11 23:59:04 INFO Received push request method=POST path=/29262026/11/11 23:59:04 INFO Received uploads request method=POST path=/29272026/11/11 23:59:04 INFO Vacuumed table table=objects29282026/11/11 23:59:04 INFO Vacuumed table table=pending_objects29292026/11/11 23:59:04 INFO Vacuumed table table=multipart_uploads29302026/11/11 23:59:04 INFO Aborted multipart uploads count=0 kept=029312026/11/11 23:59:04 INFO Received push request method=POST path=/29322026/11/11 23:59:04 INFO Vacuumed table table=closures29332026/11/11 23:59:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=1 objects-deleted-after-grace-period=0 objects-failed-to-delete=029342026/11/11 23:59:04 INFO Vacuumed table table=objects29352026/11/11 23:59:04 INFO Vacuumed table table=pending_closures29362026/11/11 23:59:04 INFO Received push request method=POST path=/29372026/11/11 23:59:04 INFO Vacuumed table table=pending_objects29382026/11/11 23:59:04 INFO Vacuumed table table=multipart_uploads29392026/11/11 23:59:04 INFO Received uploads request method=POST path=/29402026/11/11 23:59:04 INFO Received uploads request method=POST path=/29412026/11/11 23:59:04 INFO Received complete upload request method=POST path=/29422026/11/11 23:59:04 INFO Completed upload id=15 objects=4 duration=652.792µs29432026/11/11 23:59:04 INFO Received push request method=POST path=/29442026/11/11 23:59:04 INFO Received uploads request method=POST path=/29452026/11/11 23:59:04 INFO Received complete upload request method=POST path=/29462026/11/11 23:59:04 INFO Completed upload id=16 objects=3 duration=290.292µs29472026/11/11 23:59:04 INFO Vacuumed table table=closures29482026/11/11 23:59:04 INFO Aborted multipart uploads count=0 kept=029492026/11/11 23:59:04 INFO Vacuumed table table=objects29502026/11/11 23:59:04 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029512026/11/11 23:59:04 INFO Received push request method=POST path=/29522026/11/11 23:59:04 INFO Received push request method=POST path=/29532026/11/11 23:59:04 INFO Received push request method=POST path=/29542026/11/11 23:59:04 INFO Vacuumed table table=pending_closures29552026/11/11 23:59:04 INFO Received push request method=POST path=/29562026/11/11 23:59:04 INFO Vacuumed table table=pending_objects29572026/11/11 23:59:04 INFO Received push request method=POST path=/29582026/11/11 23:59:04 INFO Vacuumed table table=multipart_uploads29592026/11/11 23:59:04 INFO Received push request method=POST path=/29602026/11/11 23:59:04 INFO Vacuumed table table=closures29612026/11/11 23:59:04 INFO Vacuumed table table=objects29622026/11/11 23:59:04 INFO Received uploads request method=POST path=/29632026/11/11 23:59:04 INFO Received uploads request method=POST path=/29642026/11/11 23:59:04 INFO Received uploads request method=POST path=/29652026/11/11 23:59:04 INFO Received complete upload request method=POST path=/29662026/11/11 23:59:04 INFO Completed upload id=18 objects=4 duration=567.667µs29672026/11/11 23:59:04 INFO Received push request method=POST path=/29682026/11/11 23:59:04 INFO Received complete push request method=POST path=/29692026/11/11 23:59:04 INFO Completed push id=9 objects=24 duration=632.125µs29702026/11/11 23:59:04 INFO Received push request method=POST path=/29712026/11/11 23:59:04 INFO Received complete push request method=POST path=/29722026/11/11 23:59:04 INFO Received uploads request method=POST path=/29732026/11/11 23:59:04 INFO Completed push id=17 objects=6 duration=480.833µs29742026/11/11 23:59:04 INFO Received push request method=POST path=/29752026/11/11 23:59:04 INFO Received uploads request method=POST path=/29762026/11/11 23:59:04 INFO Received push request method=POST path=/29772026/11/11 23:59:04 INFO Received uploads request method=POST path=/29782026/11/11 23:59:04 INFO Received push request method=POST path=/29792026/11/11 23:59:04 INFO Received complete upload request method=POST path=/29802026/11/11 23:59:04 INFO Received uploads request method=POST path=/29812026/11/11 23:59:04 INFO Received complete push request method=POST path=/29822026/11/11 23:59:04 INFO Completed upload id=10 objects=10 duration=352.583µs29832026/11/11 23:59:04 INFO Aborted multipart uploads count=0 kept=029842026/11/11 23:59:04 INFO Completed push id=19 objects=6 duration=875.875µs29852026/11/11 23:59:04 INFO Received push request method=POST path=/29862026/11/11 23:59:04 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029872026/11/11 23:59:04 INFO Received uploads request method=POST path=/29882026/11/11 23:59:04 INFO Vacuumed table table=pending_closures29892026/11/11 23:59:04 INFO Vacuumed table table=pending_objects29902026/11/11 23:59:04 INFO Vacuumed table table=multipart_uploads29912026/11/11 23:59:04 INFO Received push request method=POST path=/29922026/11/11 23:59:04 INFO Received push request method=POST path=/29932026/11/11 23:59:04 INFO Vacuumed table table=closures29942026/11/11 23:59:04 INFO Aborted multipart uploads count=0 kept=029952026/11/11 23:59:04 INFO Aborted multipart uploads count=0 kept=029962026/11/11 23:59:04 INFO Vacuumed table table=objects29972026/11/11 23:59:04 WARN Force mode enabled - objects will be deleted immediately without grace period29982026/11/11 23:59:04 WARN Force mode enabled - objects will be deleted immediately without grace period29992026/11/11 23:59:04 INFO Received push request method=POST path=/30002026/11/11 23:59:04 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030012026/11/11 23:59:04 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."30022026/11/11 23:59:04 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."30032026/11/11 23:59:04 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."30042026/11/11 23:59:04 INFO Vacuumed table table=pending_closures30052026/11/11 23:59:04 INFO Aborted multipart uploads count=0 kept=030062026/11/11 23:59:04 INFO Received push request method=POST path=/30072026/11/11 23:59:04 INFO Received uploads request method=POST path=/30082026/11/11 23:59:04 WARN Force mode enabled - objects will be deleted immediately without grace period30092026/11/11 23:59:04 INFO Vacuumed table table=pending_objects30102026/11/11 23:59:04 INFO Vacuumed table table=multipart_uploads30112026/11/11 23:59:04 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."30122026/11/11 23:59:04 INFO Vacuumed table table=closures30132026/11/11 23:59:04 INFO Received complete push request method=POST path=/30142026/11/11 23:59:04 INFO Received push request method=POST path=/30152026/11/11 23:59:04 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."30162026/11/11 23:59:04 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."30172026/11/11 23:59:04 INFO Completed push id=14 objects=3 duration=428µs30182026/11/11 23:59:04 INFO Vacuumed table table=objects30192026/11/11 23:59:04 INFO Received push request method=POST path=/30202026/11/11 23:59:04 INFO Received push request method=POST path=/30212026/11/11 23:59:04 INFO Received complete upload request method=POST path=/30222026/11/11 23:59:04 INFO Aborted multipart uploads count=0 kept=030232026/11/11 23:59:04 INFO Aborted multipart uploads count=0 kept=030242026/11/11 23:59:04 INFO Received complete push request method=POST path=/30252026/11/11 23:59:04 INFO Received complete push request method=POST path=/30262026/11/11 23:59:04 INFO Received push request method=POST path=/30272026/11/11 23:59:04 INFO Completed upload id=2 objects=10 duration=510.541µs30282026/11/11 23:59:04 WARN Force mode enabled - objects will be deleted immediately without grace period30292026/11/11 23:59:04 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030302026/11/11 23:59:04 INFO Completed push id=9 objects=4 duration=710.833µs30312026/11/11 23:59:04 INFO Aborted multipart uploads count=0 kept=030322026/11/11 23:59:04 INFO Received push request method=POST path=/30332026/11/11 23:59:04 WARN Force mode enabled - objects will be deleted immediately without grace period30342026/11/11 23:59:04 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030352026/11/11 23:59:04 INFO Vacuumed table table=pending_closures30362026/11/11 23:59:04 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030372026/11/11 23:59:04 INFO Vacuumed table table=pending_closures30382026/11/11 23:59:04 INFO Completed push id=1 objects=24 duration=26.512459ms30392026/11/11 23:59:04 INFO Received complete push request method=POST path=/30402026/11/11 23:59:04 INFO Completed push id=4 objects=6 duration=487.541µs30412026/11/11 23:59:04 INFO Vacuumed table table=pending_objects30422026/11/11 23:59:04 INFO Vacuumed table table=pending_closures30432026/11/11 23:59:04 INFO Vacuumed table table=pending_objects30442026/11/11 23:59:04 INFO Received uploads request method=POST path=/30452026/11/11 23:59:04 INFO Aborted multipart uploads count=0 kept=030462026/11/11 23:59:04 INFO Vacuumed table table=multipart_uploads30472026/11/11 23:59:04 INFO Vacuumed table table=multipart_uploads30482026/11/11 23:59:04 INFO Received push request method=POST path=/30492026/11/11 23:59:04 INFO Vacuumed table table=closures30502026/11/11 23:59:04 INFO Vacuumed table table=closures30512026/11/11 23:59:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030522026/11/11 23:59:04 INFO Aborted multipart uploads count=0 kept=030532026/11/11 23:59:04 INFO Vacuumed table table=pending_objects30542026/11/11 23:59:04 INFO Vacuumed table table=objects30552026/11/11 23:59:04 INFO Vacuumed table table=multipart_uploads30562026/11/11 23:59:04 WARN Force mode enabled - objects will be deleted immediately without grace period30572026/11/11 23:59:04 INFO Vacuumed table table=objects30582026/11/11 23:59:04 INFO Vacuumed table table=closures30592026/11/11 23:59:04 INFO Received push request method=POST path=/30602026/11/11 23:59:04 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030612026/11/11 23:59:04 INFO Vacuumed table table=objects30622026/11/11 23:59:04 INFO Vacuumed table table=pending_closures30632026/11/11 23:59:04 INFO Received push request method=POST path=/30642026/11/11 23:59:04 INFO Received complete push request method=POST path=/30652026/11/11 23:59:04 INFO Vacuumed table table=pending_objects30662026/11/11 23:59:04 INFO Vacuumed table table=multipart_uploads30672026/11/11 23:59:04 INFO Vacuumed table table=closures30682026/11/11 23:59:04 INFO Received uploads request method=POST path=/30692026/11/11 23:59:04 INFO Vacuumed table table=objects30702026/11/11 23:59:04 INFO Received push request method=POST path=/30712026/11/11 23:59:04 INFO Vacuumed table table=pending_closures30722026/11/11 23:59:04 INFO Completed push id=1 objects=10 duration=23.139541ms30732026/11/11 23:59:04 INFO Received uploads request method=POST path=/30742026/11/11 23:59:04 INFO Received push request method=POST path=/30752026/11/11 23:59:04 INFO Received uploads request method=POST path=/30762026/11/11 23:59:04 INFO Received uploads request method=POST path=/30772026/11/11 23:59:04 INFO Received uploads request method=POST path=/30782026/11/11 23:59:04 INFO Received complete push request method=POST path=/30792026/11/11 23:59:04 INFO Aborted multipart uploads count=0 kept=030802026/11/11 23:59:04 INFO Received push request method=POST path=/30812026/11/11 23:59:04 INFO Aborted multipart uploads count=0 kept=030822026/11/11 23:59:04 WARN Force mode enabled - objects will be deleted immediately without grace period30832026/11/11 23:59:04 WARN Force mode enabled - objects will be deleted immediately without grace period30842026/11/11 23:59:04 INFO Completed push id=26 objects=14 duration=1.096625ms30852026/11/11 23:59:04 INFO Received push request method=POST path=/30862026/11/11 23:59:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030872026/11/11 23:59:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030882026/11/11 23:59:04 INFO Received push request method=POST path=/30892026/11/11 23:59:04 INFO Vacuumed table table=pending_closures30902026/11/11 23:59:04 INFO Vacuumed table table=pending_objects30912026/11/11 23:59:04 INFO Vacuumed table table=pending_closures30922026/11/11 23:59:04 INFO Received complete upload request method=POST path=/30932026/11/11 23:59:04 INFO Vacuumed table table=pending_objects30942026/11/11 23:59:04 INFO Vacuumed table table=multipart_uploads30952026/11/11 23:59:04 INFO Received complete upload request method=POST path=/30962026/11/11 23:59:04 INFO Completed upload id=3 objects=5 duration=723.5µs30972026/11/11 23:59:04 INFO Vacuumed table table=multipart_uploads30982026/11/11 23:59:04 INFO Vacuumed table table=pending_objects30992026/11/11 23:59:04 INFO Received uploads request method=POST path=/31002026/11/11 23:59:04 INFO Vacuumed table table=closures31012026/11/11 23:59:04 INFO Completed upload id=6 objects=24 duration=841.875µs31022026/11/11 23:59:04 INFO Received push request method=POST path=/31032026/11/11 23:59:04 INFO Vacuumed table table=multipart_uploads31042026/11/11 23:59:04 INFO Vacuumed table table=closures31052026/11/11 23:59:05 INFO Vacuumed table table=objects31062026/11/11 23:59:05 INFO Vacuumed table table=objects31072026/11/11 23:59:05 INFO Received push request method=POST path=/31082026/11/11 23:59:05 INFO Aborted multipart uploads count=0 kept=031092026/11/11 23:59:05 INFO Vacuumed table table=closures31102026/11/11 23:59:05 INFO Aborted multipart uploads count=0 kept=031112026/11/11 23:59:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031122026/11/11 23:59:05 INFO Vacuumed table table=pending_closures31132026/11/11 23:59:05 WARN Force mode enabled - objects will be deleted immediately without grace period31142026/11/11 23:59:05 INFO Received uploads request method=POST path=/31152026/11/11 23:59:05 INFO Vacuumed table table=pending_objects31162026/11/11 23:59:05 INFO Vacuumed table table=multipart_uploads31172026/11/11 23:59:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031182026/11/11 23:59:05 INFO Received uploads request method=POST path=/31192026/11/11 23:59:05 INFO Vacuumed table table=closures31202026/11/11 23:59:05 INFO Vacuumed table table=pending_closures31212026/11/11 23:59:05 INFO Vacuumed table table=objects31222026/11/11 23:59:05 INFO Received uploads request method=POST path=/31232026/11/11 23:59:05 INFO Vacuumed table table=pending_objects31242026/11/11 23:59:05 INFO Vacuumed table table=objects31252026/11/11 23:59:05 INFO Received push request method=POST path=/31262026/11/11 23:59:05 INFO Vacuumed table table=multipart_uploads31272026/11/11 23:59:05 INFO Received push request method=POST path=/31282026/11/11 23:59:05 INFO Received complete push request method=POST path=/31292026/11/11 23:59:05 INFO Vacuumed table table=closures31302026/11/11 23:59:05 INFO Received uploads request method=POST path=/31312026/11/11 23:59:05 INFO Received push request method=POST path=/31322026/11/11 23:59:05 INFO Vacuumed table table=objects31332026/11/11 23:59:05 INFO Completed push id=3 objects=21 duration=5.590083ms31342026/11/11 23:59:05 INFO Received push request method=POST path=/31352026/11/11 23:59:05 INFO Received complete push request method=POST path=/31362026/11/11 23:59:05 INFO Received uploads request method=POST path=/31372026/11/11 23:59:05 INFO Received push request method=POST path=/31382026/11/11 23:59:05 INFO Aborted multipart uploads count=0 kept=031392026/11/11 23:59:05 INFO Received complete push request method=POST path=/31402026/11/11 23:59:05 INFO Received complete push request method=POST path=/31412026/11/11 23:59:05 INFO Completed push id=18 objects=6 duration=956.875µs31422026/11/11 23:59:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031432026/11/11 23:59:05 INFO Received push request method=POST path=/31442026/11/11 23:59:05 INFO Received push request method=POST path=/31452026/11/11 23:59:05 INFO Received uploads request method=POST path=/31462026/11/11 23:59:05 INFO Completed push id=3 objects=5 duration=421.666µs31472026/11/11 23:59:05 INFO Completed push id=8 objects=6 duration=442.666µs31482026/11/11 23:59:05 INFO Received complete upload request method=POST path=/31492026/11/11 23:59:05 INFO Received uploads request method=POST path=/31502026/11/11 23:59:05 INFO Received complete push request method=POST path=/31512026/11/11 23:59:05 INFO Received push request method=POST path=/31522026/11/11 23:59:05 INFO Completed upload id=23 objects=9 duration=405.791µs31532026/11/11 23:59:05 INFO Vacuumed table table=pending_closures31542026/11/11 23:59:05 INFO Received uploads request method=POST path=/31552026/11/11 23:59:05 INFO Received uploads request method=POST path=/31562026/11/11 23:59:05 INFO Received push request method=POST path=/31572026/11/11 23:59:05 INFO Received push request method=POST path=/31582026/11/11 23:59:05 INFO Received uploads request method=POST path=/31592026/11/11 23:59:05 INFO Received complete push request method=POST path=/31602026/11/11 23:59:05 INFO Completed push id=1 objects=7 duration=29.478625ms31612026/11/11 23:59:05 INFO Received complete push request method=POST path=/31622026/11/11 23:59:05 INFO Vacuumed table table=pending_objects31632026/11/11 23:59:05 INFO Completed push id=15 objects=5 duration=599.083µs31642026/11/11 23:59:05 INFO Completed push id=4 objects=3 duration=708.917µs31652026/11/11 23:59:05 INFO Received uploads request method=POST path=/31662026/11/11 23:59:05 INFO Vacuumed table table=multipart_uploads31672026/11/11 23:59:05 INFO Received uploads request method=POST path=/31682026/11/11 23:59:05 INFO Received uploads request method=POST path=/31692026/11/11 23:59:05 INFO Received push request method=POST path=/31702026/11/11 23:59:05 INFO Received complete push request method=POST path=/31712026/11/11 23:59:05 INFO Vacuumed table table=closures31722026/11/11 23:59:05 INFO Received push request method=POST path=/31732026/11/11 23:59:05 INFO Received push request method=POST path=/31742026/11/11 23:59:05 INFO Vacuumed table table=objects31752026/11/11 23:59:05 INFO Completed push id=27 objects=25 duration=973.667µs31762026/11/11 23:59:05 INFO Received push request method=POST path=/31772026/11/11 23:59:05 INFO Received push request method=POST path=/31782026/11/11 23:59:05 INFO Received push request method=POST path=/31792026/11/11 23:59:05 INFO Received push request method=POST path=/31802026/11/11 23:59:05 INFO Received push request method=POST path=/31812026/11/11 23:59:05 INFO Received complete upload request method=POST path=/31822026/11/11 23:59:05 INFO Aborted multipart uploads count=0 kept=031832026/11/11 23:59:05 INFO Received complete upload request method=POST path=/31842026/11/11 23:59:05 INFO Received push request method=POST path=/31852026/11/11 23:59:05 INFO Received uploads request method=POST path=/31862026/11/11 23:59:05 INFO Completed upload id=17 objects=25 duration=1.374208ms31872026/11/11 23:59:05 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031882026/11/11 23:59:05 INFO Received push request method=POST path=/31892026/11/11 23:59:05 INFO Received complete push request method=POST path=/31902026/11/11 23:59:05 INFO Completed upload id=9 objects=3 duration=391.167µs31912026/11/11 23:59:05 INFO Vacuumed table table=pending_closures31922026/11/11 23:59:05 INFO Received uploads request method=POST path=/31932026/11/11 23:59:05 INFO Received uploads request method=POST path=/31942026/11/11 23:59:05 INFO Completed push id=16 objects=26 duration=585.333µs31952026/11/11 23:59:05 INFO Received push request method=POST path=/31962026/11/11 23:59:05 INFO Aborted multipart uploads count=0 kept=031972026/11/11 23:59:05 INFO Received complete push request method=POST path=/31982026/11/11 23:59:05 INFO Vacuumed table table=pending_objects31992026/11/11 23:59:05 INFO Vacuumed table table=multipart_uploads32002026/11/11 23:59:05 INFO Completed push id=4 objects=4 duration=5.718291ms32012026/11/11 23:59:05 WARN Force mode enabled - objects will be deleted immediately without grace period32022026/11/11 23:59:05 INFO Vacuumed table table=closures32032026/11/11 23:59:05 INFO Received complete upload request method=POST path=/32042026/11/11 23:59:05 INFO Received push request method=POST path=/32052026/11/11 23:59:05 INFO Aborted multipart uploads count=0 kept=032062026/11/11 23:59:05 INFO Completed upload id=21 objects=2 duration=383.167µs32072026/11/11 23:59:05 INFO Vacuumed table table=objects32082026/11/11 23:59:05 WARN Force mode enabled - objects will be deleted immediately without grace period32092026/11/11 23:59:05 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032102026/11/11 23:59:05 INFO Aborted multipart uploads count=0 kept=032112026/11/11 23:59:05 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032122026/11/11 23:59:05 INFO Aborted multipart uploads count=0 kept=032132026/11/11 23:59:05 INFO Aborted multipart uploads count=0 kept=032142026/11/11 23:59:05 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."32152026/11/11 23:59:05 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."32162026/11/11 23:59:05 INFO Received complete upload request method=POST path=/32172026/11/11 23:59:05 INFO Received push request method=POST path=/32182026/11/11 23:59:05 WARN Force mode enabled - objects will be deleted immediately without grace period32192026/11/11 23:59:05 INFO Vacuumed table table=pending_closures32202026/11/11 23:59:05 WARN Force mode enabled - objects will be deleted immediately without grace period32212026/11/11 23:59:05 INFO Completed upload id=5 objects=7 duration=774.75µs32222026/11/11 23:59:05 INFO Received uploads request method=POST path=/32232026/11/11 23:59:05 INFO Received complete push request method=POST path=/32242026/11/11 23:59:05 INFO Received complete push request method=POST path=/32252026/11/11 23:59:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032262026/11/11 23:59:05 INFO Vacuumed table table=pending_objects32272026/11/11 23:59:05 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032282026/11/11 23:59:05 INFO Vacuumed table table=multipart_uploads32292026/11/11 23:59:05 INFO Completed push id=5 objects=3 duration=475.166µs32302026/11/11 23:59:05 INFO Vacuumed table table=pending_closures32312026/11/11 23:59:05 INFO Vacuumed table table=pending_closures32322026/11/11 23:59:05 INFO Completed push id=24 objects=5 duration=1.112958ms32332026/11/11 23:59:05 INFO Received uploads request method=POST path=/32342026/11/11 23:59:05 INFO Vacuumed table table=pending_closures32352026/11/11 23:59:05 INFO Vacuumed table table=closures32362026/11/11 23:59:05 INFO Vacuumed table table=pending_objects32372026/11/11 23:59:05 INFO Vacuumed table table=pending_objects32382026/11/11 23:59:05 INFO Received complete push request method=POST path=/32392026/11/11 23:59:05 INFO Received uploads request method=POST path=/32402026/11/11 23:59:05 INFO Vacuumed table table=objects32412026/11/11 23:59:05 INFO Vacuumed table table=multipart_uploads32422026/11/11 23:59:05 INFO Vacuumed table table=multipart_uploads32432026/11/11 23:59:05 INFO Completed push id=12 objects=3 duration=567.542µs32442026/11/11 23:59:05 INFO Aborted multipart uploads count=0 kept=032452026/11/11 23:59:05 INFO Received complete push request method=POST path=/32462026/11/11 23:59:05 INFO Received complete push request method=POST path=/32472026/11/11 23:59:05 INFO Vacuumed table table=pending_objects32482026/11/11 23:59:05 INFO Received uploads request method=POST path=/32492026/11/11 23:59:05 INFO Vacuumed table table=multipart_uploads32502026/11/11 23:59:05 INFO Completed push id=15 objects=7 duration=6.227708ms32512026/11/11 23:59:05 INFO Completed push id=27 objects=6 duration=6.213708ms32522026/11/11 23:59:05 INFO Vacuumed table table=closures32532026/11/11 23:59:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=4 objects-deleted-after-grace-period=0 objects-failed-to-delete=032542026/11/11 23:59:05 INFO Vacuumed table table=closures32552026/11/11 23:59:05 INFO Received uploads request method=POST path=/32562026/11/11 23:59:05 INFO Received uploads request method=POST path=/32572026/11/11 23:59:05 INFO Vacuumed table table=objects32582026/11/11 23:59:05 INFO Received complete upload request method=POST path=/32592026/11/11 23:59:05 INFO Received complete upload request method=POST path=/32602026/11/11 23:59:05 INFO Vacuumed table table=objects32612026/11/11 23:59:05 INFO Received uploads request method=POST path=/32622026/11/11 23:59:05 INFO Vacuumed table table=closures32632026/11/11 23:59:05 INFO Completed upload id=15 objects=2 duration=546.625µs32642026/11/11 23:59:05 INFO Completed upload id=6 objects=9 duration=400.75µs32652026/11/11 23:59:05 INFO Received uploads request method=POST path=/32662026/11/11 23:59:05 INFO Vacuumed table table=pending_closures32672026/11/11 23:59:05 INFO Received uploads request method=POST path=/32682026/11/11 23:59:05 INFO Received push request method=POST path=/32692026/11/11 23:59:05 INFO Vacuumed table table=objects32702026/11/11 23:59:05 INFO Received push request method=POST path=/32712026/11/11 23:59:05 INFO Received complete upload request method=POST path=/32722026/11/11 23:59:05 INFO Received complete upload request method=POST path=/32732026/11/11 23:59:05 INFO Received uploads request method=POST path=/32742026/11/11 23:59:05 INFO Received complete upload request method=POST path=/32752026/11/11 23:59:05 INFO Completed upload id=22 objects=2 duration=462.833µs32762026/11/11 23:59:05 INFO Completed upload id=3 objects=5 duration=507.875µs32772026/11/11 23:59:05 INFO Received uploads request method=POST path=/32782026/11/11 23:59:05 INFO Completed upload id=16 objects=23 duration=286.791µs32792026/11/11 23:59:05 INFO Aborted multipart uploads count=0 kept=032802026/11/11 23:59:05 INFO Vacuumed table table=pending_objects32812026/11/11 23:59:05 INFO Aborted multipart uploads count=0 kept=032822026/11/11 23:59:05 INFO Received complete upload request method=POST path=/32832026/11/11 23:59:05 INFO Aborted multipart uploads count=0 kept=032842026/11/11 23:59:05 INFO Received complete upload request method=POST path=/32852026/11/11 23:59:05 INFO Vacuumed table table=multipart_uploads32862026/11/11 23:59:05 INFO Completed upload id=15 objects=6 duration=6.694ms32872026/11/11 23:59:05 INFO Completed upload id=26 objects=8 duration=6.651ms32882026/11/11 23:59:05 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=032892026/11/11 23:59:05 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=032902026/11/11 23:59:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032912026/11/11 23:59:05 INFO Vacuumed table table=pending_closures32922026/11/11 23:59:05 INFO Vacuumed table table=pending_closures32932026/11/11 23:59:05 INFO Received complete push request method=POST path=/32942026/11/11 23:59:05 INFO Received uploads request method=POST path=/32952026/11/11 23:59:05 INFO Completed push id=28 objects=14 duration=717.5µs32962026/11/11 23:59:05 INFO Vacuumed table table=pending_objects32972026/11/11 23:59:05 INFO Vacuumed table table=pending_objects32982026/11/11 23:59:05 INFO Vacuumed table table=pending_closures32992026/11/11 23:59:05 INFO Vacuumed table table=multipart_uploads33002026/11/11 23:59:05 INFO Vacuumed table table=multipart_uploads33012026/11/11 23:59:05 INFO Received push request method=POST path=/33022026/11/11 23:59:05 INFO Vacuumed table table=closures33032026/11/11 23:59:05 INFO Received complete push request method=POST path=/33042026/11/11 23:59:05 INFO Vacuumed table table=closures33052026/11/11 23:59:05 INFO Vacuumed table table=closures33062026/11/11 23:59:05 INFO Received uploads request method=POST path=/33072026/11/11 23:59:05 INFO Vacuumed table table=objects33082026/11/11 23:59:05 INFO Vacuumed table table=pending_objects33092026/11/11 23:59:05 INFO Vacuumed table table=objects33102026/11/11 23:59:05 INFO Completed push id=14 objects=3 duration=965.167µs33112026/11/11 23:59:05 INFO Received complete push request method=POST path=/33122026/11/11 23:59:05 INFO Aborted multipart uploads count=0 kept=033132026/11/11 23:59:05 INFO Received uploads request method=POST path=/33142026/11/11 23:59:05 INFO Vacuumed table table=multipart_uploads33152026/11/11 23:59:05 INFO Received uploads request method=POST path=/33162026/11/11 23:59:05 INFO Vacuumed table table=closures33172026/11/11 23:59:05 INFO Completed push id=6 objects=5 duration=6.228334ms33182026/11/11 23:59:05 INFO Received complete push request method=POST path=/33192026/11/11 23:59:05 WARN Force mode enabled - objects will be deleted immediately without grace period33202026/11/11 23:59:05 INFO Vacuumed table table=objects33212026/11/11 23:59:05 INFO Vacuumed table table=objects33222026/11/11 23:59:05 INFO Completed push id=21 objects=26 duration=388.166µs33232026/11/11 23:59:05 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033242026/11/11 23:59:05 INFO Received push request method=POST path=/33252026/11/11 23:59:05 INFO Received push request method=POST path=/33262026/11/11 23:59:05 INFO Received complete upload request method=POST path=/33272026/11/11 23:59:05 INFO Received complete push request method=POST path=/33282026/11/11 23:59:05 INFO Vacuumed table table=pending_closures33292026/11/11 23:59:05 INFO Completed upload id=17 objects=7 duration=531.375µs33302026/11/11 23:59:05 INFO Vacuumed table table=pending_objects33312026/11/11 23:59:05 INFO Completed push id=8 objects=13 duration=974.167µs33322026/11/11 23:59:05 INFO Received push request method=POST path=/33332026/11/11 23:59:05 INFO Received push request method=POST path=/33342026/11/11 23:59:05 INFO Vacuumed table table=multipart_uploads33352026/11/11 23:59:05 INFO Vacuumed table table=closures33362026/11/11 23:59:05 INFO Received push request method=POST path=/33372026/11/11 23:59:05 INFO Aborted multipart uploads count=0 kept=033382026/11/11 23:59:05 INFO Aborted multipart uploads count=0 kept=033392026/11/11 23:59:05 INFO Aborted multipart uploads count=0 kept=033402026/11/11 23:59:05 INFO Vacuumed table table=objects33412026/11/11 23:59:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033422026/11/11 23:59:05 INFO Received push request method=POST path=/33432026/11/11 23:59:05 INFO Aborted multipart uploads count=0 kept=033442026/11/11 23:59:05 INFO Received uploads request method=POST path=/33452026/11/11 23:59:05 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."33462026/11/11 23:59:05 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."33472026/11/11 23:59:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033482026/11/11 23:59:05 INFO Aborted multipart uploads count=0 kept=033492026/11/11 23:59:05 INFO Received uploads request method=POST path=/33502026/11/11 23:59:05 WARN Force mode enabled - objects will be deleted immediately without grace period33512026/11/11 23:59:05 INFO Received complete push request method=POST path=/33522026/11/11 23:59:05 INFO Received push request method=POST path=/33532026/11/11 23:59:05 INFO Vacuumed table table=pending_closures33542026/11/11 23:59:05 INFO Completed push id=21 objects=9 duration=804.625µs33552026/11/11 23:59:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033562026/11/11 23:59:05 INFO Received uploads request method=POST path=/33572026/11/11 23:59:05 INFO Received complete upload request method=POST path=/33582026/11/11 23:59:05 INFO Vacuumed table table=pending_closures33592026/11/11 23:59:05 INFO Received uploads request method=POST path=/33602026/11/11 23:59:05 INFO Received uploads request method=POST path=/33612026/11/11 23:59:05 INFO Received complete upload request method=POST path=/33622026/11/11 23:59:05 INFO Completed upload id=21 objects=9 duration=823.792µs33632026/11/11 23:59:05 INFO Received uploads request method=POST path=/33642026/11/11 23:59:05 INFO Received complete upload request method=POST path=/33652026/11/11 23:59:05 INFO Received push request method=POST path=/33662026/11/11 23:59:05 INFO Vacuumed table table=pending_objects33672026/11/11 23:59:05 INFO Received complete push request method=POST path=/33682026/11/11 23:59:05 INFO Vacuumed table table=pending_objects33692026/11/11 23:59:05 INFO Vacuumed table table=pending_closures33702026/11/11 23:59:05 INFO Completed upload id=28 objects=17 duration=317.084µs33712026/11/11 23:59:05 INFO Received push request method=POST path=/33722026/11/11 23:59:05 INFO Vacuumed table table=multipart_uploads33732026/11/11 23:59:05 INFO Received complete push request method=POST path=/33742026/11/11 23:59:05 INFO Vacuumed table table=multipart_uploads33752026/11/11 23:59:05 INFO Completed push id=25 objects=10 duration=1.054167ms33762026/11/11 23:59:05 INFO Received complete push request method=POST path=/33772026/11/11 23:59:05 INFO Vacuumed table table=closures33782026/11/11 23:59:05 INFO Completed upload id=3 objects=6 duration=45.907792ms33792026/11/11 23:59:05 INFO Completed push id=2 objects=10 duration=1.012125ms33802026/11/11 23:59:05 INFO Vacuumed table table=closures33812026/11/11 23:59:05 INFO Received complete upload request method=POST path=/33822026/11/11 23:59:05 INFO Vacuumed table table=pending_objects33832026/11/11 23:59:05 INFO Received complete upload request method=POST path=/33842026/11/11 23:59:05 INFO Vacuumed table table=objects33852026/11/11 23:59:05 INFO Vacuumed table table=objects33862026/11/11 23:59:05 INFO Completed push id=29 objects=2 duration=477.541µs33872026/11/11 23:59:05 INFO Completed upload id=14 objects=6 duration=454.875µs33882026/11/11 23:59:05 INFO Vacuumed table table=multipart_uploads33892026/11/11 23:59:05 INFO Aborted multipart uploads count=0 kept=033902026/11/11 23:59:05 INFO Completed upload id=29 objects=5 duration=278.834µs33912026/11/11 23:59:05 INFO Received complete upload request method=POST path=/33922026/11/11 23:59:05 INFO Received push request method=POST path=/33932026/11/11 23:59:05 INFO Received push request method=POST path=/33942026/11/11 23:59:05 INFO Received complete push request method=POST path=/33952026/11/11 23:59:05 INFO Received complete push request method=POST path=/33962026/11/11 23:59:05 INFO Completed upload id=13 objects=8 duration=779.125µs33972026/11/11 23:59:05 INFO Received push request method=POST path=/33982026/11/11 23:59:05 INFO Received complete push request method=POST path=/33992026/11/11 23:59:05 INFO Received complete upload request method=POST path=/34002026/11/11 23:59:05 INFO Completed push id=31 objects=6 duration=333.166µs34012026/11/11 23:59:05 INFO Completed push id=7 objects=24 duration=1.15175ms34022026/11/11 23:59:05 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.narinfo error="Access Denied."34032026/11/11 23:59:05 INFO Received push request method=POST path=/34042026/11/11 23:59:05 INFO Completed upload id=30 objects=2 duration=295.792µs34052026/11/11 23:59:05 INFO Received uploads request method=POST path=/34062026/11/11 23:59:05 INFO Completed push id=8 objects=7 duration=557.208µs34072026/11/11 23:59:05 INFO Received uploads request method=POST path=/34082026/11/11 23:59:05 INFO Received complete push request method=POST path=/34092026/11/11 23:59:05 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."34102026/11/11 23:59:05 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.narinfo error="Access Denied."34112026/11/11 23:59:05 INFO Vacuumed table table=closures34122026/11/11 23:59:05 INFO Received complete push request method=POST path=/34132026/11/11 23:59:05 INFO Received uploads request method=POST path=/34142026/11/11 23:59:05 INFO Completed push id=32 objects=9 duration=1.03075ms34152026/11/11 23:59:05 INFO Received uploads request method=POST path=/34162026/11/11 23:59:05 INFO Received uploads request method=POST path=/34172026/11/11 23:59:05 INFO Received uploads request method=POST path=/34182026/11/11 23:59:05 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."34192026/11/11 23:59:05 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."34202026/11/11 23:59:05 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."34212026/11/11 23:59:05 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."34222026/11/11 23:59:05 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."34232026/11/11 23:59:05 ERROR failed to remove object object=nar/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.nar.zst error="Access Denied."34242026/11/11 23:59:05 ERROR failed to remove object object=nar/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.nar.zst error="Access Denied."34252026/11/11 23:59:05 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."34262026/11/11 23:59:05 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."34272026/11/11 23:59:05 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."34282026/11/11 23:59:05 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."34292026/11/11 23:59:05 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."34302026/11/11 23:59:05 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."34312026/11/11 23:59:05 INFO Completed push id=31 objects=20 duration=373.833µs34322026/11/11 23:59:05 INFO Aborted multipart uploads count=0 kept=034332026/11/11 23:59:05 INFO Received push request method=POST path=/34342026/11/11 23:59:05 INFO Received uploads request method=POST path=/34352026/11/11 23:59:05 INFO Received uploads request method=POST path=/34362026/11/11 23:59:05 INFO Received complete upload request method=POST path=/34372026/11/11 23:59:05 INFO Vacuumed table table=objects34382026/11/11 23:59:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=16 objects-failed-to-delete=034392026/11/11 23:59:05 INFO Received push request method=POST path=/34402026/11/11 23:59:05 INFO Completed upload id=32 objects=5 duration=570.792µs34412026/11/11 23:59:05 INFO Vacuumed table table=pending_closures34422026/11/11 23:59:05 INFO Vacuumed table table=pending_objects34432026/11/11 23:59:05 INFO Received uploads request method=POST path=/34442026/11/11 23:59:05 INFO Received push request method=POST path=/34452026/11/11 23:59:05 INFO Received uploads request method=POST path=/34462026/11/11 23:59:05 INFO Received complete upload request method=POST path=/34472026/11/11 23:59:05 INFO Vacuumed table table=multipart_uploads34482026/11/11 23:59:05 INFO Received complete upload request method=POST path=/34492026/11/11 23:59:05 INFO Aborted multipart uploads count=0 kept=034502026/11/11 23:59:05 INFO Received complete push request method=POST path=/34512026/11/11 23:59:06 INFO Completed upload id=33 objects=3 duration=253µs34522026/11/11 23:59:06 INFO Completed upload id=7 objects=3 duration=239.584µs34532026/11/11 23:59:06 INFO Completed push id=13 objects=2 duration=282.958µs34542026/11/11 23:59:06 WARN Force mode enabled - objects will be deleted immediately without grace period34552026/11/11 23:59:06 INFO Aborted multipart uploads count=0 kept=034562026/11/11 23:59:06 INFO Received complete upload request method=POST path=/34572026/11/11 23:59:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034582026/11/11 23:59:06 WARN Force mode enabled - objects will be deleted immediately without grace period34592026/11/11 23:59:06 INFO Completed upload id=27 objects=13 duration=251.167µs34602026/11/11 23:59:06 INFO Received complete upload request method=POST path=/34612026/11/11 23:59:06 INFO Vacuumed table table=closures34622026/11/11 23:59:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034632026/11/11 23:59:06 INFO Vacuumed table table=pending_closures34642026/11/11 23:59:06 INFO Aborted multipart uploads count=0 kept=034652026/11/11 23:59:06 INFO Completed upload id=33 objects=9 duration=268.25µs34662026/11/11 23:59:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034672026/11/11 23:59:06 INFO Vacuumed table table=pending_objects34682026/11/11 23:59:06 INFO Vacuumed table table=pending_closures34692026/11/11 23:59:06 INFO Aborted multipart uploads count=0 kept=034702026/11/11 23:59:06 INFO Vacuumed table table=objects34712026/11/11 23:59:06 INFO Received complete upload request method=POST path=/34722026/11/11 23:59:06 INFO Vacuumed table table=multipart_uploads34732026/11/11 23:59:06 INFO Aborted multipart uploads count=0 kept=034742026/11/11 23:59:06 INFO Vacuumed table table=pending_closures34752026/11/11 23:59:06 INFO Completed upload id=9 objects=16 duration=521.375µs34762026/11/11 23:59:06 INFO Vacuumed table table=closures34772026/11/11 23:59:06 WARN Force mode enabled - objects will be deleted immediately without grace period34782026/11/11 23:59:06 INFO Vacuumed table table=pending_objects34792026/11/11 23:59:06 INFO Received complete upload request method=POST path=/34802026/11/11 23:59:06 WARN Force mode enabled - objects will be deleted immediately without grace period34812026/11/11 23:59:06 INFO Vacuumed table table=objects34822026/11/11 23:59:06 INFO Received push request method=POST path=/34832026/11/11 23:59:06 INFO Vacuumed table table=multipart_uploads34842026/11/11 23:59:06 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034852026/11/11 23:59:06 INFO Completed upload id=10 objects=6 duration=858.583µs34862026/11/11 23:59:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034872026/11/11 23:59:06 INFO Received complete upload request method=POST path=/34882026/11/11 23:59:06 INFO Vacuumed table table=closures34892026/11/11 23:59:06 INFO Received push request method=POST path=/34902026/11/11 23:59:06 INFO Vacuumed table table=pending_objects34912026/11/11 23:59:06 INFO Received uploads request method=POST path=/34922026/11/11 23:59:06 INFO Vacuumed table table=pending_closures34932026/11/11 23:59:06 INFO Completed upload id=10 objects=4 duration=749.333µs34942026/11/11 23:59:06 INFO Received push request method=POST path=/34952026/11/11 23:59:06 INFO Vacuumed table table=objects34962026/11/11 23:59:06 INFO Vacuumed table table=multipart_uploads34972026/11/11 23:59:06 INFO Vacuumed table table=pending_objects34982026/11/11 23:59:06 INFO Received complete push request method=POST path=/34992026/11/11 23:59:06 INFO Vacuumed table table=pending_closures35002026/11/11 23:59:06 INFO Vacuumed table table=closures35012026/11/11 23:59:06 INFO Completed push id=18 objects=23 duration=10.568625ms35022026/11/11 23:59:06 INFO Vacuumed table table=multipart_uploads35032026/11/11 23:59:06 INFO Vacuumed table table=objects35042026/11/11 23:59:06 INFO Received push request method=POST path=/35052026/11/11 23:59:06 INFO Received complete push request method=POST path=/35062026/11/11 23:59:06 INFO Vacuumed table table=pending_objects35072026/11/11 23:59:06 INFO Received complete push request method=POST path=/35082026/11/11 23:59:06 INFO Received uploads request method=POST path=/35092026/11/11 23:59:06 INFO Vacuumed table table=closures35102026/11/11 23:59:06 INFO Vacuumed table table=multipart_uploads35112026/11/11 23:59:06 INFO Vacuumed table table=objects35122026/11/11 23:59:06 INFO Completed push id=10 objects=3 duration=459.917µs35132026/11/11 23:59:06 INFO Completed push id=8 objects=6 duration=502.834µs35142026/11/11 23:59:06 INFO Received uploads request method=POST path=/35152026/11/11 23:59:06 INFO Aborted multipart uploads count=0 kept=035162026/11/11 23:59:06 INFO Received push request method=POST path=/35172026/11/11 23:59:06 INFO Vacuumed table table=closures35182026/11/11 23:59:06 INFO Received push request method=POST path=/35192026/11/11 23:59:06 INFO Received uploads request method=POST path=/35202026/11/11 23:59:06 INFO Received push request method=POST path=/35212026/11/11 23:59:06 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035222026/11/11 23:59:06 INFO Received push request method=POST path=/35232026/11/11 23:59:06 INFO Vacuumed table table=objects35242026/11/11 23:59:06 INFO Received push request method=POST path=/35252026/11/11 23:59:06 INFO Received push request method=POST path=/35262026/11/11 23:59:06 INFO Vacuumed table table=pending_closures35272026/11/11 23:59:06 INFO Received uploads request method=POST path=/35282026/11/11 23:59:06 INFO Received uploads request method=POST path=/35292026/11/11 23:59:06 INFO Aborted multipart uploads count=0 kept=035302026/11/11 23:59:06 INFO Vacuumed table table=pending_objects35312026/11/11 23:59:06 INFO Received uploads request method=POST path=/35322026/11/11 23:59:06 WARN Force mode enabled - objects will be deleted immediately without grace period35332026/11/11 23:59:06 INFO Vacuumed table table=multipart_uploads35342026/11/11 23:59:06 INFO Received complete upload request method=POST path=/35352026/11/11 23:59:06 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035362026/11/11 23:59:06 INFO Vacuumed table table=closures35372026/11/11 23:59:06 INFO Received uploads request method=POST path=/35382026/11/11 23:59:06 INFO Received complete push request method=POST path=/35392026/11/11 23:59:06 INFO Vacuumed table table=objects35402026/11/11 23:59:06 INFO Vacuumed table table=pending_closures35412026/11/11 23:59:06 INFO Completed upload id=29 objects=4 duration=1.301042ms35422026/11/11 23:59:06 INFO Received push request method=POST path=/35432026/11/11 23:59:06 INFO Completed push id=17 objects=22 duration=586.25µs35442026/11/11 23:59:06 INFO Received complete push request method=POST path=/35452026/11/11 23:59:06 INFO Vacuumed table table=pending_objects35462026/11/11 23:59:06 INFO Received uploads request method=POST path=/35472026/11/11 23:59:06 INFO Vacuumed table table=multipart_uploads35482026/11/11 23:59:06 INFO Completed push id=12 objects=7 duration=787.875µs35492026/11/11 23:59:06 INFO Received uploads request method=POST path=/35502026/11/11 23:59:06 INFO Vacuumed table table=closures35512026/11/11 23:59:06 INFO Aborted multipart uploads count=0 kept=035522026/11/11 23:59:06 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=035532026/11/11 23:59:06 INFO Vacuumed table table=objects35542026/11/11 23:59:06 INFO Received uploads request method=POST path=/35552026/11/11 23:59:06 INFO Received uploads request method=POST path=/35562026/11/11 23:59:06 INFO Received complete push request method=POST path=/35572026/11/11 23:59:06 INFO Received uploads request method=POST path=/35582026/11/11 23:59:06 INFO Completed push id=11 objects=9 duration=444.208µs35592026/11/11 23:59:06 INFO Received push request method=POST path=/35602026/11/11 23:59:06 INFO Received push request method=POST path=/35612026/11/11 23:59:06 INFO Received complete upload request method=POST path=/35622026/11/11 23:59:06 INFO Received push request method=POST path=/35632026/11/11 23:59:06 INFO Received uploads request method=POST path=/35642026/11/11 23:59:06 INFO Completed upload id=33 objects=5 duration=850.709µs35652026/11/11 23:59:06 INFO Vacuumed table table=pending_closures35662026/11/11 23:59:06 INFO Received complete push request method=POST path=/35672026/11/11 23:59:06 INFO Received uploads request method=POST path=/35682026/11/11 23:59:06 INFO Completed push id=5 objects=28 duration=736.542µs35692026/11/11 23:59:06 INFO Received complete push request method=POST path=/35702026/11/11 23:59:06 INFO Aborted multipart uploads count=0 kept=035712026/11/11 23:59:06 INFO Completed push id=10 objects=2 duration=9.471334ms35722026/11/11 23:59:06 INFO Vacuumed table table=pending_objects35732026/11/11 23:59:06 WARN Force mode enabled - objects will be deleted immediately without grace period35742026/11/11 23:59:06 INFO Vacuumed table table=multipart_uploads35752026/11/11 23:59:06 INFO Received complete upload request method=POST path=/35762026/11/11 23:59:06 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035772026/11/11 23:59:06 INFO Vacuumed table table=closures35782026/11/11 23:59:06 INFO Received complete push request method=POST path=/35792026/11/11 23:59:06 INFO Completed upload id=11 objects=14 duration=499.958µs35802026/11/11 23:59:06 INFO Vacuumed table table=pending_closures35812026/11/11 23:59:06 INFO Vacuumed table table=objects35822026/11/11 23:59:06 INFO Vacuumed table table=pending_objects35832026/11/11 23:59:06 INFO Completed push id=11 objects=6 duration=11.366667ms35842026/11/11 23:59:06 INFO Received push request method=POST path=/35852026/11/11 23:59:06 INFO Vacuumed table table=multipart_uploads35862026/11/11 23:59:06 INFO Received push request method=POST path=/35872026/11/11 23:59:06 INFO Received uploads request method=POST path=/35882026/11/11 23:59:06 INFO Vacuumed table table=closures35892026/11/11 23:59:06 INFO Received complete upload request method=POST path=/35902026/11/11 23:59:06 INFO Received push request method=POST path=/35912026/11/11 23:59:06 INFO Vacuumed table table=objects35922026/11/11 23:59:06 INFO Completed upload id=17 objects=6 duration=855.791µs35932026/11/11 23:59:06 INFO Aborted multipart uploads count=0 kept=035942026/11/11 23:59:06 INFO Received uploads request method=POST path=/35952026/11/11 23:59:06 INFO Received uploads request method=POST path=/35962026/11/11 23:59:06 WARN Force mode enabled - objects will be deleted immediately without grace period35972026/11/11 23:59:06 INFO Received uploads request method=POST path=/35982026/11/11 23:59:06 INFO Received push request method=POST path=/35992026/11/11 23:59:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036002026/11/11 23:59:06 INFO Received uploads request method=POST path=/36012026/11/11 23:59:06 INFO Received push request method=POST path=/36022026/11/11 23:59:06 INFO Vacuumed table table=pending_closures36032026/11/11 23:59:06 INFO Received push request method=POST path=/36042026/11/11 23:59:06 INFO Received complete upload request method=POST path=/36052026/11/11 23:59:06 INFO Received complete upload request method=POST path=/36062026/11/11 23:59:06 INFO Vacuumed table table=pending_objects36072026/11/11 23:59:06 INFO Received uploads request method=POST path=/36082026/11/11 23:59:06 INFO Completed upload id=17 objects=5 duration=284.208µs36092026/11/11 23:59:06 INFO Received push request method=POST path=/36102026/11/11 23:59:06 INFO Completed upload id=23 objects=15 duration=728.166µs36112026/11/11 23:59:06 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."36122026/11/11 23:59:06 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."36132026/11/11 23:59:06 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."36142026/11/11 23:59:06 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."36152026/11/11 23:59:06 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."36162026/11/11 23:59:06 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."36172026/11/11 23:59:06 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."36182026/11/11 23:59:06 INFO Vacuumed table table=multipart_uploads36192026/11/11 23:59:06 INFO Received complete push request method=POST path=/36202026/11/11 23:59:06 INFO Received uploads request method=POST path=/36212026/11/11 23:59:06 INFO Received push request method=POST path=/36222026/11/11 23:59:06 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."36232026/11/11 23:59:06 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."36242026/11/11 23:59:06 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."36252026/11/11 23:59:06 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."36262026/11/11 23:59:06 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."36272026/11/11 23:59:06 INFO Completed push id=20 objects=13 duration=544.166µs36282026/11/11 23:59:06 INFO Received uploads request method=POST path=/36292026/11/11 23:59:06 INFO Vacuumed table table=closures36302026/11/11 23:59:06 INFO Received push request method=POST path=/36312026/11/11 23:59:06 INFO Received complete push request method=POST path=/36322026/11/11 23:59:06 INFO Received uploads request method=POST path=/36332026/11/11 23:59:06 INFO Vacuumed table table=objects36342026/11/11 23:59:06 INFO Received push request method=POST path=/36352026/11/11 23:59:06 INFO Aborted multipart uploads count=0 kept=036362026/11/11 23:59:06 INFO Received push request method=POST path=/36372026/11/11 23:59:06 INFO Completed push id=22 objects=4 duration=492.25µs36382026/11/11 23:59:06 INFO Received complete upload request method=POST path=/36392026/11/11 23:59:06 INFO Received uploads request method=POST path=/36402026/11/11 23:59:06 INFO Received uploads request method=POST path=/36412026/11/11 23:59:06 INFO Completed upload id=15 objects=9 duration=319.791µs36422026/11/11 23:59:06 WARN Force mode enabled - objects will be deleted immediately without grace period36432026/11/11 23:59:06 INFO Received push request method=POST path=/36442026/11/11 23:59:06 INFO Received complete push request method=POST path=/36452026/11/11 23:59:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036462026/11/11 23:59:06 INFO Completed push id=14 objects=3 duration=853.583µs36472026/11/11 23:59:06 INFO Received push request method=POST path=/36482026/11/11 23:59:06 INFO Received complete upload request method=POST path=/36492026/11/11 23:59:06 INFO Aborted multipart uploads count=0 kept=036502026/11/11 23:59:06 INFO Received complete push request method=POST path=/36512026/11/11 23:59:06 INFO Completed upload id=19 objects=6 duration=299.458µs36522026/11/11 23:59:06 INFO Completed push id=5 objects=6 duration=558.042µs36532026/11/11 23:59:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=036542026/11/11 23:59:06 INFO Vacuumed table table=pending_closures36552026/11/11 23:59:06 INFO Aborted multipart uploads count=0 kept=036562026/11/11 23:59:06 INFO Received complete push request method=POST path=/36572026/11/11 23:59:06 INFO Vacuumed table table=pending_closures36582026/11/11 23:59:06 INFO Completed push id=19 objects=3 duration=498.709µs36592026/11/11 23:59:06 WARN Force mode enabled - objects will be deleted immediately without grace period36602026/11/11 23:59:06 INFO Vacuumed table table=pending_objects36612026/11/11 23:59:06 INFO Vacuumed table table=pending_objects36622026/11/11 23:59:06 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036632026/11/11 23:59:06 INFO Vacuumed table table=multipart_uploads36642026/11/11 23:59:06 INFO Vacuumed table table=multipart_uploads36652026/11/11 23:59:06 INFO Received complete push request method=POST path=/36662026/11/11 23:59:06 INFO Vacuumed table table=closures36672026/11/11 23:59:06 INFO Vacuumed table table=pending_closures36682026/11/11 23:59:06 INFO Completed push id=21 objects=17 duration=613.708µs36692026/11/11 23:59:06 INFO Vacuumed table table=objects36702026/11/11 23:59:06 INFO Received complete upload request method=POST path=/36712026/11/11 23:59:06 INFO Received complete push request method=POST path=/36722026/11/11 23:59:06 INFO Vacuumed table table=pending_objects36732026/11/11 23:59:06 INFO Vacuumed table table=closures36742026/11/11 23:59:06 INFO Received complete push request method=POST path=/36752026/11/11 23:59:06 INFO Completed upload id=23 objects=7 duration=282.958µs36762026/11/11 23:59:06 INFO Completed push id=19 objects=7 duration=6.223ms36772026/11/11 23:59:06 INFO Vacuumed table table=objects36782026/11/11 23:59:06 INFO Vacuumed table table=multipart_uploads36792026/11/11 23:59:06 INFO Received complete push request method=POST path=/36802026/11/11 23:59:06 INFO Completed push id=3 objects=5 duration=19.084042ms36812026/11/11 23:59:06 INFO Received push request method=POST path=/36822026/11/11 23:59:06 INFO Received push request method=POST path=/36832026/11/11 23:59:06 INFO Received complete upload request method=POST path=/36842026/11/11 23:59:06 INFO Vacuumed table table=closures36852026/11/11 23:59:06 INFO Completed push id=24 objects=5 duration=264.75µs36862026/11/11 23:59:06 INFO Received complete push request method=POST path=/36872026/11/11 23:59:06 INFO Received complete upload request method=POST path=/36882026/11/11 23:59:06 INFO Completed upload id=22 objects=7 duration=318.625µs36892026/11/11 23:59:06 INFO Vacuumed table table=objects36902026/11/11 23:59:06 INFO Received complete upload request method=POST path=/36912026/11/11 23:59:06 INFO Completed push id=2 objects=7 duration=1.115416ms36922026/11/11 23:59:06 INFO Completed upload id=10 objects=2 duration=6.807834ms36932026/11/11 23:59:06 INFO Received complete push request method=POST path=/36942026/11/11 23:59:06 INFO Aborted multipart uploads count=0 kept=036952026/11/11 23:59:06 INFO Completed upload id=25 objects=2 duration=250.791µs36962026/11/11 23:59:06 INFO Completed push id=24 objects=9 duration=588.958µs36972026/11/11 23:59:06 INFO Received push request method=POST path=/36982026/11/11 23:59:06 INFO Aborted multipart uploads count=0 kept=036992026/11/11 23:59:06 WARN Force mode enabled - objects will be deleted immediately without grace period37002026/11/11 23:59:06 INFO Received complete upload request method=POST path=/37012026/11/11 23:59:06 INFO Aborted multipart uploads count=0 kept=037022026/11/11 23:59:06 INFO Received push request method=POST path=/37032026/11/11 23:59:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037042026/11/11 23:59:06 INFO Completed upload id=26 objects=20 duration=274.917µs37052026/11/11 23:59:06 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037062026/11/11 23:59:06 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=037072026/11/11 23:59:06 INFO Received push request method=POST path=/37082026/11/11 23:59:06 INFO Vacuumed table table=pending_closures37092026/11/11 23:59:06 INFO Vacuumed table table=pending_closures37102026/11/11 23:59:06 INFO Vacuumed table table=pending_objects37112026/11/11 23:59:06 INFO Vacuumed table table=multipart_uploads37122026/11/11 23:59:06 INFO Vacuumed table table=pending_closures37132026/11/11 23:59:06 INFO Received uploads request method=POST path=/37142026/11/11 23:59:06 INFO Vacuumed table table=pending_objects37152026/11/11 23:59:06 INFO Vacuumed table table=closures37162026/11/11 23:59:06 INFO Received complete push request method=POST path=/37172026/11/11 23:59:06 INFO Vacuumed table table=multipart_uploads37182026/11/11 23:59:06 INFO Completed push id=20 objects=4 duration=323.084µs37192026/11/11 23:59:06 INFO Vacuumed table table=objects37202026/11/11 23:59:06 INFO Received uploads request method=POST path=/37212026/11/11 23:59:06 INFO Vacuumed table table=closures37222026/11/11 23:59:07 INFO Vacuumed table table=pending_objects37232026/11/11 23:59:07 INFO Received complete push request method=POST path=/37242026/11/11 23:59:07 INFO Vacuumed table table=objects37252026/11/11 23:59:07 INFO Vacuumed table table=multipart_uploads37262026/11/11 23:59:07 INFO Completed push id=21 objects=3 duration=6.138375ms37272026/11/11 23:59:07 INFO Received push request method=POST path=/37282026/11/11 23:59:07 INFO Vacuumed table table=closures37292026/11/11 23:59:07 INFO Received uploads request method=POST path=/37302026/11/11 23:59:07 INFO Received complete upload request method=POST path=/37312026/11/11 23:59:07 INFO Vacuumed table table=objects37322026/11/11 23:59:07 INFO Completed upload id=21 objects=25 duration=427.958µs37332026/11/11 23:59:07 INFO Aborted multipart uploads count=0 kept=037342026/11/11 23:59:07 INFO Received uploads request method=POST path=/37352026/11/11 23:59:07 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=3 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037362026/11/11 23:59:07 INFO Received uploads request method=POST path=/37372026/11/11 23:59:07 INFO Vacuumed table table=pending_closures37382026/11/11 23:59:07 INFO Received complete push request method=POST path=/37392026/11/11 23:59:07 INFO Received complete push request method=POST path=/37402026/11/11 23:59:07 INFO Aborted multipart uploads count=0 kept=037412026/11/11 23:59:07 INFO Completed push id=19 objects=8 duration=1.229417ms37422026/11/11 23:59:07 INFO Completed push id=28 objects=7 duration=314.084µs37432026/11/11 23:59:07 INFO Vacuumed table table=pending_objects37442026/11/11 23:59:07 WARN Force mode enabled - objects will be deleted immediately without grace period37452026/11/11 23:59:07 INFO Vacuumed table table=multipart_uploads37462026/11/11 23:59:07 INFO Received uploads request method=POST path=/37472026/11/11 23:59:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037482026/11/11 23:59:07 INFO Vacuumed table table=closures37492026/11/11 23:59:07 INFO Vacuumed table table=objects37502026/11/11 23:59:07 INFO Aborted multipart uploads count=0 kept=037512026/11/11 23:59:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037522026/11/11 23:59:07 INFO Vacuumed table table=pending_closures37532026/11/11 23:59:07 INFO Received complete push request method=POST path=/37542026/11/11 23:59:07 INFO Vacuumed table table=pending_closures37552026/11/11 23:59:07 INFO Received push request method=POST path=/37562026/11/11 23:59:07 INFO Vacuumed table table=pending_objects37572026/11/11 23:59:07 INFO Vacuumed table table=multipart_uploads37582026/11/11 23:59:07 INFO Received push request method=POST path=/37592026/11/11 23:59:07 INFO Received complete push request method=POST path=/37602026/11/11 23:59:07 INFO Received complete upload request method=POST path=/37612026/11/11 23:59:07 INFO Vacuumed table table=pending_objects37622026/11/11 23:59:07 INFO Completed push id=30 objects=4 duration=991.416µs37632026/11/11 23:59:07 INFO Vacuumed table table=multipart_uploads37642026/11/11 23:59:07 INFO Vacuumed table table=closures37652026/11/11 23:59:07 INFO Vacuumed table table=closures37662026/11/11 23:59:07 INFO Vacuumed table table=objects37672026/11/11 23:59:07 INFO Completed push id=1 objects=5 duration=38.047125ms37682026/11/11 23:59:07 INFO Received complete upload request method=POST path=/37692026/11/11 23:59:07 INFO Received uploads request method=POST path=/37702026/11/11 23:59:07 INFO Vacuumed table table=objects37712026/11/11 23:59:07 INFO Aborted multipart uploads count=0 kept=037722026/11/11 23:59:07 INFO Received complete upload request method=POST path=/37732026/11/11 23:59:07 INFO Completed upload id=22 objects=8 duration=297.5µs37742026/11/11 23:59:07 INFO Received push request method=POST path=/37752026/11/11 23:59:07 INFO Received complete upload request method=POST path=/37762026/11/11 23:59:07 INFO Completed upload id=1 objects=15 duration=29.948625ms37772026/11/11 23:59:07 INFO Received push request method=POST path=/37782026/11/11 23:59:07 WARN Force mode enabled - objects will be deleted immediately without grace period37792026/11/11 23:59:07 INFO Completed upload id=30 objects=6 duration=296.875µs37802026/11/11 23:59:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037812026/11/11 23:59:07 INFO Completed upload id=12 objects=26 duration=994.916µs37822026/11/11 23:59:07 INFO Received uploads request method=POST path=/37832026/11/11 23:59:07 INFO Received complete push request method=POST path=/37842026/11/11 23:59:07 INFO Vacuumed table table=pending_closures37852026/11/11 23:59:07 INFO Received push request method=POST path=/37862026/11/11 23:59:07 INFO Received complete push request method=POST path=/37872026/11/11 23:59:07 INFO Completed push id=31 objects=5 duration=452.625µs37882026/11/11 23:59:07 INFO Vacuumed table table=pending_objects37892026/11/11 23:59:07 INFO Completed push id=23 objects=5 duration=6.027291ms37902026/11/11 23:59:07 INFO Vacuumed table table=multipart_uploads37912026/11/11 23:59:07 INFO Received uploads request method=POST path=/37922026/11/11 23:59:07 INFO Vacuumed table table=closures37932026/11/11 23:59:07 INFO Received complete push request method=POST path=/37942026/11/11 23:59:07 INFO Received complete upload request method=POST path=/37952026/11/11 23:59:07 INFO Vacuumed table table=objects37962026/11/11 23:59:07 INFO Aborted multipart uploads count=0 kept=037972026/11/11 23:59:07 INFO Received complete push request method=POST path=/37982026/11/11 23:59:07 INFO Completed push id=32 objects=7 duration=265.708µs37992026/11/11 23:59:07 INFO Completed push id=16 objects=8 duration=6.080167ms38002026/11/11 23:59:07 INFO Completed upload id=24 objects=8 duration=6.155708ms38012026/11/11 23:59:07 INFO Received push request method=POST path=/38022026/11/11 23:59:07 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."38032026/11/11 23:59:07 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."38042026/11/11 23:59:07 INFO Received complete push request method=POST path=/38052026/11/11 23:59:07 INFO Aborted multipart uploads count=0 kept=038062026/11/11 23:59:07 INFO Received complete upload request method=POST path=/38072026/11/11 23:59:07 INFO Received uploads request method=POST path=/38082026/11/11 23:59:07 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."38092026/11/11 23:59:07 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."38102026/11/11 23:59:07 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."38112026/11/11 23:59:07 INFO Completed upload id=28 objects=5 duration=273.459µs38122026/11/11 23:59:07 INFO Completed push id=26 objects=4 duration=483.333µs38132026/11/11 23:59:07 WARN Force mode enabled - objects will be deleted immediately without grace period38142026/11/11 23:59:07 INFO Received uploads request method=POST path=/38152026/11/11 23:59:07 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."38162026/11/11 23:59:07 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."38172026/11/11 23:59:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038182026/11/11 23:59:07 INFO Aborted multipart uploads count=0 kept=038192026/11/11 23:59:07 INFO Received uploads request method=POST path=/38202026/11/11 23:59:07 INFO Vacuumed table table=pending_closures38212026/11/11 23:59:07 INFO Received complete upload request method=POST path=/38222026/11/11 23:59:07 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."38232026/11/11 23:59:07 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."38242026/11/11 23:59:07 INFO Received push request method=POST path=/38252026/11/11 23:59:07 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."38262026/11/11 23:59:07 INFO Completed upload id=28 objects=13 duration=287.708µs38272026/11/11 23:59:07 INFO Vacuumed table table=pending_objects38282026/11/11 23:59:07 INFO Received uploads request method=POST path=/38292026/11/11 23:59:07 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."38302026/11/11 23:59:07 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."38312026/11/11 23:59:07 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."38322026/11/11 23:59:07 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."38332026/11/11 23:59:07 INFO Vacuumed table table=multipart_uploads38342026/11/11 23:59:07 INFO Received complete push request method=POST path=/38352026/11/11 23:59:07 INFO Completed push id=29 objects=3 duration=379.334µs38362026/11/11 23:59:07 INFO Vacuumed table table=closures38372026/11/11 23:59:07 INFO Received push request method=POST path=/38382026/11/11 23:59:07 INFO Received complete upload request method=POST path=/38392026/11/11 23:59:07 INFO Vacuumed table table=objects38402026/11/11 23:59:07 INFO Completed upload id=31 objects=2 duration=356.5µs38412026/11/11 23:59:07 INFO Received push request method=POST path=/38422026/11/11 23:59:07 INFO Received complete push request method=POST path=/38432026/11/11 23:59:07 INFO Received complete push request method=POST path=/38442026/11/11 23:59:07 INFO Received complete upload request method=POST path=/38452026/11/11 23:59:07 INFO Completed push id=33 objects=6 duration=297.208µs38462026/11/11 23:59:07 INFO Aborted multipart uploads count=0 kept=038472026/11/11 23:59:07 INFO Completed push id=32 objects=26 duration=391.708µs38482026/11/11 23:59:07 INFO Completed upload id=30 objects=5 duration=341.333µs38492026/11/11 23:59:07 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."38502026/11/11 23:59:07 INFO Aborted multipart uploads count=0 kept=038512026/11/11 23:59:07 INFO Received complete push request method=POST path=/38522026/11/11 23:59:07 WARN Force mode enabled - objects will be deleted immediately without grace period38532026/11/11 23:59:07 INFO Received complete upload request method=POST path=/38542026/11/11 23:59:07 INFO Received uploads request method=POST path=/38552026/11/11 23:59:07 INFO Completed push id=11 objects=8 duration=1.122583ms38562026/11/11 23:59:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038572026/11/11 23:59:07 INFO Completed upload id=33 objects=2 duration=391.041µs38582026/11/11 23:59:07 INFO Received complete upload request method=POST path=/38592026/11/11 23:59:07 INFO Completed upload id=7 objects=10 duration=657.167µs38602026/11/11 23:59:07 INFO Received complete upload request method=POST path=/38612026/11/11 23:59:07 INFO Completed upload id=34 objects=5 duration=237.375µs38622026/11/11 23:59:07 INFO Aborted multipart uploads count=0 kept=038632026/11/11 23:59:07 INFO Vacuumed table table=pending_closures38642026/11/11 23:59:07 INFO Aborted multipart uploads count=0 kept=038652026/11/11 23:59:07 INFO Received complete upload request method=POST path=/38662026/11/11 23:59:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038672026/11/11 23:59:07 INFO Received complete upload request method=POST path=/38682026/11/11 23:59:07 INFO Vacuumed table table=pending_closures38692026/11/11 23:59:07 INFO Completed upload id=12 objects=6 duration=345.042µs38702026/11/11 23:59:07 INFO Completed upload id=35 objects=11 duration=239.917µs38712026/11/11 23:59:07 INFO Vacuumed table table=pending_objects38722026/11/11 23:59:07 INFO Vacuumed table table=pending_objects38732026/11/11 23:59:07 INFO Received complete push request method=POST path=/38742026/11/11 23:59:07 INFO Received complete push request method=POST path=/38752026/11/11 23:59:07 INFO Vacuumed table table=multipart_uploads38762026/11/11 23:59:07 INFO Vacuumed table table=multipart_uploads38772026/11/11 23:59:07 INFO Received complete push request method=POST path=/38782026/11/11 23:59:07 INFO Received complete upload request method=POST path=/38792026/11/11 23:59:07 INFO Received complete push request method=POST path=/38802026/11/11 23:59:07 INFO Completed push id=2 objects=4 duration=618.125µs38812026/11/11 23:59:07 INFO Completed upload id=36 objects=3 duration=430.958µs38822026/11/11 23:59:07 INFO Vacuumed table table=closures38832026/11/11 23:59:07 INFO Completed push id=31 objects=3 duration=462.667µs38842026/11/11 23:59:07 INFO Vacuumed table table=closures38852026/11/11 23:59:07 INFO Completed push id=17 objects=3 duration=778.5µs38862026/11/11 23:59:07 INFO Completed push id=11 objects=6 duration=1.261959ms38872026/11/11 23:59:07 INFO Received complete upload request method=POST path=/38882026/11/11 23:59:07 INFO Vacuumed table table=objects38892026/11/11 23:59:07 INFO Aborted multipart uploads count=0 kept=038902026/11/11 23:59:07 INFO Vacuumed table table=objects38912026/11/11 23:59:07 INFO Received uploads request method=POST path=/38922026/11/11 23:59:07 INFO Completed upload id=4 objects=7 duration=1.138375ms38932026/11/11 23:59:07 WARN Force mode enabled - objects will be deleted immediately without grace period38942026/11/11 23:59:07 INFO Received uploads request method=POST path=/38952026/11/11 23:59:07 INFO Received push request method=POST path=/38962026/11/11 23:59:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038972026/11/11 23:59:07 INFO Aborted multipart uploads count=0 kept=038982026/11/11 23:59:07 INFO Received uploads request method=POST path=/38992026/11/11 23:59:07 WARN Force mode enabled - objects will be deleted immediately without grace period39002026/11/11 23:59:07 INFO Received push request method=POST path=/39012026/11/11 23:59:07 INFO Vacuumed table table=pending_closures39022026/11/11 23:59:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039032026/11/11 23:59:07 INFO Received uploads request method=POST path=/39042026/11/11 23:59:07 INFO Received complete push request method=POST path=/39052026/11/11 23:59:07 INFO Completed push id=23 objects=5 duration=344.042µs39062026/11/11 23:59:07 INFO Vacuumed table table=pending_objects39072026/11/11 23:59:07 INFO Vacuumed table table=multipart_uploads39082026/11/11 23:59:07 INFO Received complete upload request method=POST path=/39092026/11/11 23:59:07 INFO Received push request method=POST path=/39102026/11/11 23:59:07 INFO Completed upload id=24 objects=3 duration=312.833µs39112026/11/11 23:59:07 INFO Vacuumed table table=closures39122026/11/11 23:59:07 INFO Aborted multipart uploads count=0 kept=039132026/11/11 23:59:07 INFO Vacuumed table table=objects39142026/11/11 23:59:07 INFO Received complete push request method=POST path=/39152026/11/11 23:59:07 WARN Force mode enabled - objects will be deleted immediately without grace period39162026/11/11 23:59:07 INFO Completed push id=25 objects=4 duration=270.125µs39172026/11/11 23:59:07 INFO Vacuumed table table=pending_closures39182026/11/11 23:59:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039192026/11/11 23:59:07 INFO Received complete push request method=POST path=/39202026/11/11 23:59:07 INFO Completed push id=18 objects=6 duration=532.083µs39212026/11/11 23:59:07 INFO Aborted multipart uploads count=0 kept=039222026/11/11 23:59:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039232026/11/11 23:59:07 INFO Vacuumed table table=pending_closures39242026/11/11 23:59:07 INFO Vacuumed table table=pending_objects39252026/11/11 23:59:07 INFO Vacuumed table table=pending_objects39262026/11/11 23:59:07 INFO Vacuumed table table=multipart_uploads39272026/11/11 23:59:07 INFO Vacuumed table table=multipart_uploads39282026/11/11 23:59:07 INFO Vacuumed table table=closures39292026/11/11 23:59:07 INFO Vacuumed table table=objects39302026/11/11 23:59:07 INFO Vacuumed table table=pending_closures39312026/11/11 23:59:07 INFO Received complete push request method=POST path=/39322026/11/11 23:59:07 INFO Received complete upload request method=POST path=/39332026/11/11 23:59:07 INFO Vacuumed table table=closures39342026/11/11 23:59:07 INFO Completed push id=32 objects=6 duration=423.25µs39352026/11/11 23:59:07 INFO Vacuumed table table=pending_objects39362026/11/11 23:59:07 INFO Completed upload id=20 objects=3 duration=1.114625ms39372026/11/11 23:59:07 INFO Vacuumed table table=multipart_uploads39382026/11/11 23:59:07 INFO Received complete upload request method=POST path=/39392026/11/11 23:59:07 INFO Completed upload id=33 objects=7 duration=492.334µs39402026/11/11 23:59:07 INFO Received push request method=POST path=/39412026/11/11 23:59:07 INFO Received complete upload request method=POST path=/39422026/11/11 23:59:07 INFO Completed upload id=34 objects=3 duration=349.5µs39432026/11/11 23:59:07 INFO Received complete upload request method=POST path=/39442026/11/11 23:59:07 INFO Completed upload id=24 objects=7 duration=516.334µs39452026/11/11 23:59:07 INFO Vacuumed table table=closures39462026/11/11 23:59:07 INFO Vacuumed table table=objects39472026/11/11 23:59:07 INFO Received complete push request method=POST path=/39482026/11/11 23:59:07 INFO Aborted multipart uploads count=0 kept=039492026/11/11 23:59:07 INFO Completed push id=35 objects=11 duration=366.375µs39502026/11/11 23:59:07 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039512026/11/11 23:59:07 INFO Received push request method=POST path=/39522026/11/11 23:59:07 INFO Vacuumed table table=pending_closures39532026/11/11 23:59:07 INFO Vacuumed table table=objects39542026/11/11 23:59:07 INFO Aborted multipart uploads count=0 kept=039552026/11/11 23:59:07 INFO Received uploads request method=POST path=/39562026/11/11 23:59:07 INFO Vacuumed table table=pending_objects39572026/11/11 23:59:07 INFO Vacuumed table table=multipart_uploads39582026/11/11 23:59:07 WARN Force mode enabled - objects will be deleted immediately without grace period39592026/11/11 23:59:07 INFO Vacuumed table table=closures39602026/11/11 23:59:07 INFO Garbage collection completed failed-uploads-deleted=6 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039612026/11/11 23:59:07 INFO Received uploads request method=POST path=/39622026/11/11 23:59:07 INFO Vacuumed table table=objects39632026/11/11 23:59:07 INFO Received uploads request method=POST path=/39642026/11/11 23:59:07 INFO Received uploads request method=POST path=/39652026/11/11 23:59:07 INFO Received complete push request method=POST path=/39662026/11/11 23:59:07 INFO Received uploads request method=POST path=/39672026/11/11 23:59:07 INFO Vacuumed table table=pending_closures39682026/11/11 23:59:07 INFO Aborted multipart uploads count=0 kept=039692026/11/11 23:59:07 INFO Completed push id=13 objects=6 duration=944.166µs39702026/11/11 23:59:07 INFO Received uploads request method=POST path=/39712026/11/11 23:59:07 WARN Force mode enabled - objects will be deleted immediately without grace period39722026/11/11 23:59:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039732026/11/11 23:59:07 INFO Received push request method=POST path=/39742026/11/11 23:59:07 INFO Aborted multipart uploads count=0 kept=039752026/11/11 23:59:07 INFO Vacuumed table table=pending_closures39762026/11/11 23:59:07 INFO Vacuumed table table=pending_objects39772026/11/11 23:59:07 INFO Received push request method=POST path=/39782026/11/11 23:59:07 INFO Vacuumed table table=multipart_uploads39792026/11/11 23:59:07 WARN Force mode enabled - objects will be deleted immediately without grace period39802026/11/11 23:59:07 INFO Vacuumed table table=closures39812026/11/11 23:59:08 INFO Vacuumed table table=pending_objects39822026/11/11 23:59:08 INFO Vacuumed table table=objects39832026/11/11 23:59:08 INFO Vacuumed table table=multipart_uploads39842026/11/11 23:59:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=039852026/11/11 23:59:08 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=4 objects-failed-to-delete=039862026/11/11 23:59:08 INFO Vacuumed table table=closures39872026/11/11 23:59:08 INFO Vacuumed table table=pending_closures39882026/11/11 23:59:08 INFO Vacuumed table table=pending_closures39892026/11/11 23:59:08 INFO Vacuumed table table=objects39902026/11/11 23:59:08 INFO Vacuumed table table=pending_objects39912026/11/11 23:59:08 INFO Vacuumed table table=multipart_uploads39922026/11/11 23:59:08 INFO Vacuumed table table=pending_objects39932026/11/11 23:59:08 INFO Vacuumed table table=multipart_uploads39942026/11/11 23:59:08 INFO Vacuumed table table=closures39952026/11/11 23:59:08 INFO Vacuumed table table=objects39962026/11/11 23:59:08 INFO Received complete upload request method=POST path=/39972026/11/11 23:59:08 INFO Completed upload id=10 objects=6 duration=923.792µs39982026/11/11 23:59:08 INFO Aborted multipart uploads count=0 kept=039992026/11/11 23:59:08 INFO Received complete upload request method=POST path=/40002026/11/11 23:59:08 WARN Force mode enabled - objects will be deleted immediately without grace period40012026/11/11 23:59:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040022026/11/11 23:59:08 INFO Vacuumed table table=closures40032026/11/11 23:59:08 INFO Completed upload id=14 objects=16 duration=6.263458ms40042026/11/11 23:59:08 INFO Vacuumed table table=pending_closures40052026/11/11 23:59:08 INFO Vacuumed table table=pending_objects40062026/11/11 23:59:08 INFO Vacuumed table table=multipart_uploads40072026/11/11 23:59:08 INFO Vacuumed table table=closures40082026/11/11 23:59:08 INFO Vacuumed table table=objects40092026/11/11 23:59:08 INFO Vacuumed table table=objects40102026/11/11 23:59:08 INFO Aborted multipart uploads count=0 kept=040112026/11/11 23:59:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040122026/11/11 23:59:08 INFO Received complete upload request method=POST path=/40132026/11/11 23:59:08 INFO Vacuumed table table=pending_closures40142026/11/11 23:59:08 INFO Received complete push request method=POST path=/40152026/11/11 23:59:08 INFO Completed upload id=11 objects=7 duration=436.75µs40162026/11/11 23:59:08 INFO Vacuumed table table=pending_objects40172026/11/11 23:59:08 INFO Completed push id=23 objects=28 duration=5.941833ms40182026/11/11 23:59:08 INFO Vacuumed table table=multipart_uploads40192026/11/11 23:59:08 INFO Received uploads request method=POST path=/40202026/11/11 23:59:08 INFO Vacuumed table table=closures40212026/11/11 23:59:08 INFO Vacuumed table table=objects40222026/11/11 23:59:08 INFO Received uploads request method=POST path=/40232026/11/11 23:59:08 INFO Received complete upload request method=POST path=/40242026/11/11 23:59:08 INFO Received uploads request method=POST path=/40252026/11/11 23:59:08 INFO Received complete upload request method=POST path=/40262026/11/11 23:59:08 INFO Completed upload id=27 objects=8 duration=523.791µs40272026/11/11 23:59:08 INFO Completed upload id=6 objects=7 duration=1.275292ms40282026/11/11 23:59:08 INFO Received push request method=POST path=/40292026/11/11 23:59:08 INFO Received push request method=POST path=/40302026/11/11 23:59:08 INFO Received push request method=POST path=/40312026/11/11 23:59:08 INFO Aborted multipart uploads count=0 kept=040322026/11/11 23:59:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040332026/11/11 23:59:08 INFO Vacuumed table table=pending_closures40342026/11/11 23:59:08 INFO Received uploads request method=POST path=/40352026/11/11 23:59:08 INFO Vacuumed table table=pending_objects40362026/11/11 23:59:08 INFO Vacuumed table table=multipart_uploads40372026/11/11 23:59:08 INFO Vacuumed table table=closures40382026/11/11 23:59:08 INFO Vacuumed table table=objects40392026/11/11 23:59:08 INFO Received complete upload request method=POST path=/40402026/11/11 23:59:08 INFO Aborted multipart uploads count=0 kept=040412026/11/11 23:59:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040422026/11/11 23:59:08 INFO Completed upload id=4 objects=14 duration=5.577917ms40432026/11/11 23:59:08 INFO Vacuumed table table=pending_closures40442026/11/11 23:59:08 INFO Received complete push request method=POST path=/40452026/11/11 23:59:08 INFO Vacuumed table table=pending_objects40462026/11/11 23:59:08 INFO Vacuumed table table=multipart_uploads40472026/11/11 23:59:08 INFO Completed push id=26 objects=23 duration=433.833µs40482026/11/11 23:59:08 INFO Aborted multipart uploads count=0 kept=040492026/11/11 23:59:08 INFO Vacuumed table table=closures40502026/11/11 23:59:08 INFO Vacuumed table table=objects40512026/11/11 23:59:08 WARN Force mode enabled - objects will be deleted immediately without grace period40522026/11/11 23:59:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040532026/11/11 23:59:08 INFO Received complete upload request method=POST path=/40542026/11/11 23:59:08 INFO Completed upload id=25 objects=16 duration=967.208µs40552026/11/11 23:59:08 INFO Aborted multipart uploads count=0 kept=040562026/11/11 23:59:08 WARN Force mode enabled - objects will be deleted immediately without grace period40572026/11/11 23:59:08 INFO Vacuumed table table=pending_closures40582026/11/11 23:59:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040592026/11/11 23:59:08 INFO Vacuumed table table=pending_closures40602026/11/11 23:59:08 INFO Vacuumed table table=pending_objects40612026/11/11 23:59:08 INFO Vacuumed table table=multipart_uploads40622026/11/11 23:59:08 INFO Vacuumed table table=pending_objects40632026/11/11 23:59:08 INFO Vacuumed table table=multipart_uploads40642026/11/11 23:59:08 INFO Vacuumed table table=closures40652026/11/11 23:59:08 INFO Received complete push request method=POST path=/40662026/11/11 23:59:08 INFO Vacuumed table table=closures40672026/11/11 23:59:08 INFO Vacuumed table table=objects40682026/11/11 23:59:08 INFO Received complete push request method=POST path=/40692026/11/11 23:59:08 INFO Completed push id=17 objects=6 duration=771.334µs40702026/11/11 23:59:08 INFO Vacuumed table table=objects40712026/11/11 23:59:08 INFO Completed push id=17 objects=11 duration=770.5µs40722026/11/11 23:59:08 INFO Received push request method=POST path=/40732026/11/11 23:59:08 INFO Received push request method=POST path=/40742026/11/11 23:59:08 INFO Received push request method=POST path=/40752026/11/11 23:59:08 INFO Received push request method=POST path=/40762026/11/11 23:59:08 INFO Aborted multipart uploads count=0 kept=040772026/11/11 23:59:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040782026/11/11 23:59:08 INFO Received push request method=POST path=/40792026/11/11 23:59:08 INFO Received uploads request method=POST path=/40802026/11/11 23:59:08 INFO Vacuumed table table=pending_closures40812026/11/11 23:59:08 INFO Received complete upload request method=POST path=/40822026/11/11 23:59:08 INFO Received complete upload request method=POST path=/40832026/11/11 23:59:08 INFO Completed upload id=26 objects=6 duration=302.167µs40842026/11/11 23:59:08 INFO Completed upload id=22 objects=5 duration=495.583µs40852026/11/11 23:59:08 INFO Received uploads request method=POST path=/40862026/11/11 23:59:08 INFO Received complete upload request method=POST path=/40872026/11/11 23:59:08 INFO Vacuumed table table=pending_objects40882026/11/11 23:59:08 INFO Received complete push request method=POST path=/40892026/11/11 23:59:08 INFO Received complete upload request method=POST path=/40902026/11/11 23:59:08 INFO Completed upload id=11 objects=6 duration=582.209µs40912026/11/11 23:59:08 INFO Completed upload id=29 objects=5 duration=230.333µs40922026/11/11 23:59:08 INFO Vacuumed table table=multipart_uploads40932026/11/11 23:59:08 INFO Completed push id=28 objects=3 duration=623.25µs40942026/11/11 23:59:08 INFO Received uploads request method=POST path=/40952026/11/11 23:59:08 INFO Vacuumed table table=closures40962026/11/11 23:59:08 INFO Received complete push request method=POST path=/40972026/11/11 23:59:08 INFO Vacuumed table table=objects40982026/11/11 23:59:08 INFO Completed push id=29 objects=15 duration=499.958µs40992026/11/11 23:59:08 INFO Aborted multipart uploads count=0 kept=041002026/11/11 23:59:08 INFO Received uploads request method=POST path=/41012026/11/11 23:59:08 WARN Force mode enabled - objects will be deleted immediately without grace period41022026/11/11 23:59:08 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041032026/11/11 23:59:08 INFO Received complete push request method=POST path=/41042026/11/11 23:59:08 INFO Vacuumed table table=pending_closures41052026/11/11 23:59:08 INFO Received push request method=POST path=/41062026/11/11 23:59:08 INFO Completed push id=30 objects=6 duration=268.375µs41072026/11/11 23:59:08 INFO Vacuumed table table=pending_objects41082026/11/11 23:59:08 INFO Vacuumed table table=multipart_uploads41092026/11/11 23:59:08 INFO Received complete upload request method=POST path=/41102026/11/11 23:59:08 INFO Received complete push request method=POST path=/41112026/11/11 23:59:08 INFO Vacuumed table table=closures41122026/11/11 23:59:08 INFO Completed upload id=31 objects=3 duration=301.083µs41132026/11/11 23:59:08 INFO Completed push id=9 objects=5 duration=487.417µs41142026/11/11 23:59:08 INFO Vacuumed table table=objects41152026/11/11 23:59:08 INFO Aborted multipart uploads count=0 kept=041162026/11/11 23:59:08 INFO Aborted multipart uploads count=0 kept=041172026/11/11 23:59:08 INFO Received push request method=POST path=/41182026/11/11 23:59:08 WARN Force mode enabled - objects will be deleted immediately without grace period41192026/11/11 23:59:08 WARN Force mode enabled - objects will be deleted immediately without grace period41202026/11/11 23:59:08 INFO Received complete push request method=POST path=/41212026/11/11 23:59:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041222026/11/11 23:59:08 INFO Received uploads request method=POST path=/41232026/11/11 23:59:08 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041242026/11/11 23:59:08 INFO Completed push id=5 objects=8 duration=1.143042ms41252026/11/11 23:59:08 INFO Vacuumed table table=pending_closures41262026/11/11 23:59:08 INFO Received complete upload request method=POST path=/41272026/11/11 23:59:08 INFO Received complete push request method=POST path=/41282026/11/11 23:59:08 INFO Vacuumed table table=pending_objects41292026/11/11 23:59:08 INFO Completed upload id=31 objects=3 duration=790.042µs41302026/11/11 23:59:08 INFO Received push request method=POST path=/41312026/11/11 23:59:08 INFO Vacuumed table table=pending_closures41322026/11/11 23:59:08 INFO Vacuumed table table=multipart_uploads41332026/11/11 23:59:08 INFO Completed push id=6 objects=26 duration=6.556875ms41342026/11/11 23:59:08 INFO Received complete upload request method=POST path=/41352026/11/11 23:59:08 INFO Received uploads request method=POST path=/41362026/11/11 23:59:08 INFO Received complete push request method=POST path=/41372026/11/11 23:59:08 INFO Vacuumed table table=closures41382026/11/11 23:59:08 INFO Received complete push request method=POST path=/41392026/11/11 23:59:08 INFO Completed upload id=22 objects=2 duration=253.958µs41402026/11/11 23:59:08 INFO Aborted multipart uploads count=0 kept=041412026/11/11 23:59:08 INFO Aborted multipart uploads count=0 kept=041422026/11/11 23:59:08 INFO Completed push id=4 objects=7 duration=417.459µs41432026/11/11 23:59:08 INFO Completed push id=27 objects=15 duration=6.534042ms41442026/11/11 23:59:08 INFO Vacuumed table table=objects41452026/11/11 23:59:08 INFO Vacuumed table table=pending_objects41462026/11/11 23:59:08 WARN Force mode enabled - objects will be deleted immediately without grace period41472026/11/11 23:59:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041482026/11/11 23:59:08 INFO Aborted multipart uploads count=0 kept=041492026/11/11 23:59:08 INFO Vacuumed table table=multipart_uploads41502026/11/11 23:59:08 INFO Vacuumed table table=pending_closures41512026/11/11 23:59:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041522026/11/11 23:59:08 WARN Force mode enabled - objects will be deleted immediately without grace period41532026/11/11 23:59:08 INFO Received uploads request method=POST path=/41542026/11/11 23:59:08 INFO Vacuumed table table=closures41552026/11/11 23:59:08 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041562026/11/11 23:59:08 INFO Vacuumed table table=pending_closures41572026/11/11 23:59:08 INFO Vacuumed table table=pending_objects41582026/11/11 23:59:08 INFO Vacuumed table table=objects41592026/11/11 23:59:08 INFO Vacuumed table table=multipart_uploads41602026/11/11 23:59:08 INFO Vacuumed table table=pending_objects41612026/11/11 23:59:08 INFO Vacuumed table table=pending_closures41622026/11/11 23:59:08 INFO Received uploads request method=POST path=/41632026/11/11 23:59:08 INFO Vacuumed table table=multipart_uploads41642026/11/11 23:59:08 INFO Vacuumed table table=closures41652026/11/11 23:59:08 INFO Vacuumed table table=pending_objects41662026/11/11 23:59:08 INFO Received push request method=POST path=/41672026/11/11 23:59:08 INFO Vacuumed table table=closures41682026/11/11 23:59:08 INFO Vacuumed table table=objects41692026/11/11 23:59:08 INFO Vacuumed table table=multipart_uploads41702026/11/11 23:59:08 INFO Vacuumed table table=objects41712026/11/11 23:59:08 INFO Vacuumed table table=closures41722026/11/11 23:59:08 INFO Received uploads request method=POST path=/41732026/11/11 23:59:08 INFO Received uploads request method=POST path=/41742026/11/11 23:59:08 INFO Vacuumed table table=objects41752026/11/11 23:59:08 INFO Received uploads request method=POST path=/41762026/11/11 23:59:08 INFO Received complete upload request method=POST path=/41772026/11/11 23:59:08 INFO Aborted multipart uploads count=0 kept=041782026/11/11 23:59:08 WARN Force mode enabled - objects will be deleted immediately without grace period41792026/11/11 23:59:08 INFO Completed upload id=18 objects=2 duration=6.086584ms41802026/11/11 23:59:08 INFO Received push request method=POST path=/41812026/11/11 23:59:08 INFO Received push request method=POST path=/41822026/11/11 23:59:08 INFO Received complete upload request method=POST path=/41832026/11/11 23:59:08 INFO Completed upload id=8 objects=3 duration=722.125µs41842026/11/11 23:59:08 INFO Received uploads request method=POST path=/41852026/11/11 23:59:08 INFO Received complete push request method=POST path=/41862026/11/11 23:59:08 INFO Completed push id=6 objects=6 duration=571.834µs41872026/11/11 23:59:08 INFO Received push request method=POST path=/41882026/11/11 23:59:08 INFO Received complete push request method=POST path=/41892026/11/11 23:59:08 INFO Received complete upload request method=POST path=/41902026/11/11 23:59:08 INFO Completed push id=10 objects=15 duration=847.458µs41912026/11/11 23:59:09 INFO Completed upload id=13 objects=7 duration=344.042µs41922026/11/11 23:59:09 INFO Received complete upload request method=POST path=/41932026/11/11 23:59:09 INFO Completed upload id=20 objects=4 duration=280.167µs41942026/11/11 23:59:09 INFO Received complete push request method=POST path=/41952026/11/11 23:59:09 INFO Received complete upload request method=POST path=/41962026/11/11 23:59:09 INFO Completed upload id=23 objects=7 duration=932.375µs41972026/11/11 23:59:09 INFO Completed push id=5 objects=9 duration=7.149ms41982026/11/11 23:59:09 INFO Aborted multipart uploads count=0 kept=041992026/11/11 23:59:09 WARN Force mode enabled - objects will be deleted immediately without grace period42002026/11/11 23:59:09 INFO Received push request method=POST path=/42012026/11/11 23:59:09 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=042022026/11/11 23:59:09 INFO Received push request method=POST path=/42032026/11/11 23:59:09 INFO Received uploads request method=POST path=/42042026/11/11 23:59:09 INFO Received uploads request method=POST path=/42052026/11/11 23:59:09 INFO Received push request method=POST path=/42062026/11/11 23:59:09 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."42072026/11/11 23:59:09 INFO Vacuumed table table=pending_closures42082026/11/11 23:59:09 INFO Aborted multipart uploads count=0 kept=042092026/11/11 23:59:09 INFO Received complete push request method=POST path=/42102026/11/11 23:59:09 WARN Force mode enabled - objects will be deleted immediately without grace period42112026/11/11 23:59:09 INFO Completed push id=28 objects=19 duration=344.208µs42122026/11/11 23:59:09 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042132026/11/11 23:59:09 INFO Vacuumed table table=pending_objects42142026/11/11 23:59:09 INFO Vacuumed table table=multipart_uploads42152026/11/11 23:59:09 INFO Vacuumed table table=pending_closures42162026/11/11 23:59:09 INFO Vacuumed table table=closures42172026/11/11 23:59:09 INFO Vacuumed table table=objects42182026/11/11 23:59:09 INFO Vacuumed table table=pending_objects42192026/11/11 23:59:09 INFO Vacuumed table table=multipart_uploads42202026/11/11 23:59:09 INFO Received push request method=POST path=/42212026/11/11 23:59:09 INFO Received uploads request method=POST path=/42222026/11/11 23:59:09 INFO Aborted multipart uploads count=0 kept=042232026/11/11 23:59:09 INFO Vacuumed table table=closures42242026/11/11 23:59:09 WARN Force mode enabled - objects will be deleted immediately without grace period42252026/11/11 23:59:09 INFO Vacuumed table table=objects42262026/11/11 23:59:09 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."42272026/11/11 23:59:09 INFO Aborted multipart uploads count=0 kept=042282026/11/11 23:59:09 WARN Force mode enabled - objects will be deleted immediately without grace period42292026/11/11 23:59:09 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042302026/11/11 23:59:09 INFO Vacuumed table table=pending_closures42312026/11/11 23:59:09 INFO Vacuumed table table=pending_objects42322026/11/11 23:59:09 INFO Vacuumed table table=multipart_uploads42332026/11/11 23:59:09 INFO Vacuumed table table=closures42342026/11/11 23:59:09 INFO Vacuumed table table=objects42352026/11/11 23:59:09 INFO Received complete upload request method=POST path=/42362026/11/11 23:59:09 INFO Completed upload id=14 objects=8 duration=767.25µs42372026/11/11 23:59:09 INFO Received complete upload request method=POST path=/42382026/11/11 23:59:09 INFO Completed upload id=3 objects=24 duration=532.458µs42392026/11/11 23:59:09 INFO Received complete push request method=POST path=/42402026/11/11 23:59:09 INFO Completed push id=29 objects=10 duration=367.167µs42412026/11/11 23:59:09 INFO Received uploads request method=POST path=/42422026/11/11 23:59:09 INFO Aborted multipart uploads count=0 kept=042432026/11/11 23:59:09 WARN Force mode enabled - objects will be deleted immediately without grace period42442026/11/11 23:59:09 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=042452026/11/11 23:59:09 INFO Vacuumed table table=pending_closures42462026/11/11 23:59:09 INFO Received complete push request method=POST path=/42472026/11/11 23:59:09 INFO Completed push id=7 objects=7 duration=472.791µs42482026/11/11 23:59:09 INFO Vacuumed table table=pending_objects42492026/11/11 23:59:09 INFO Vacuumed table table=multipart_uploads42502026/11/11 23:59:09 INFO Received uploads request method=POST path=/42512026/11/11 23:59:09 INFO Vacuumed table table=closures42522026/11/11 23:59:09 INFO Vacuumed table table=objects42532026/11/11 23:59:09 INFO Received complete upload request method=POST path=/42542026/11/11 23:59:09 INFO Completed upload id=30 objects=6 duration=425.417µs42552026/11/11 23:59:09 INFO Received complete push request method=POST path=/42562026/11/11 23:59:09 INFO Received uploads request method=POST path=/42572026/11/11 23:59:09 INFO Received complete upload request method=POST path=/42582026/11/11 23:59:09 INFO Completed upload id=31 objects=5 duration=380.458µs42592026/11/11 23:59:09 INFO Completed push id=11 objects=11 duration=5.784708ms42602026/11/11 23:59:09 INFO Received complete upload request method=POST path=/42612026/11/11 23:59:09 INFO Completed upload id=34 objects=3 duration=507.375µs42622026/11/11 23:59:09 INFO Aborted multipart uploads count=0 kept=042632026/11/11 23:59:09 WARN Force mode enabled - objects will be deleted immediately without grace period42642026/11/11 23:59:09 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042652026/11/11 23:59:09 INFO Received complete upload request method=POST path=/42662026/11/11 23:59:09 INFO Completed upload id=24 objects=18 duration=335.959µs42672026/11/11 23:59:09 INFO Received complete push request method=POST path=/42682026/11/11 23:59:09 INFO Completed push id=25 objects=2 duration=438.833µs42692026/11/11 23:59:09 INFO Vacuumed table table=pending_closures42702026/11/11 23:59:09 INFO Received complete push request method=POST path=/42712026/11/11 23:59:09 INFO Completed push id=27 objects=4 duration=245.916µs42722026/11/11 23:59:09 INFO Vacuumed table table=pending_objects42732026/11/11 23:59:09 INFO Vacuumed table table=multipart_uploads42742026/11/11 23:59:09 INFO Vacuumed table table=closures42752026/11/11 23:59:09 INFO Received complete push request method=POST path=/42762026/11/11 23:59:09 INFO Vacuumed table table=objects42772026/11/11 23:59:09 INFO Completed push id=13 objects=15 duration=518.417µs42782026/11/11 23:59:09 INFO Received complete upload request method=POST path=/42792026/11/11 23:59:09 INFO Completed upload id=16 objects=17 duration=799.666µs42802026/11/11 23:59:09 INFO Aborted multipart uploads count=0 kept=042812026/11/11 23:59:09 INFO Aborted multipart uploads count=0 kept=042822026/11/11 23:59:09 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."42832026/11/11 23:59:09 WARN Force mode enabled - objects will be deleted immediately without grace period42842026/11/11 23:59:09 INFO Received complete upload request method=POST path=/42852026/11/11 23:59:09 INFO Completed upload id=19 objects=25 duration=822.125µs42862026/11/11 23:59:09 INFO Aborted multipart uploads count=0 kept=042872026/11/11 23:59:09 WARN Force mode enabled - objects will be deleted immediately without grace period42882026/11/11 23:59:09 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=042892026/11/11 23:59:09 INFO Received complete upload request method=POST path=/42902026/11/11 23:59:09 INFO Vacuumed table table=pending_closures42912026/11/11 23:59:09 INFO Completed upload id=18 objects=11 duration=367.25µs42922026/11/11 23:59:09 INFO Vacuumed table table=pending_objects42932026/11/11 23:59:09 INFO Vacuumed table table=multipart_uploads42942026/11/11 23:59:09 INFO Received uploads request method=POST path=/42952026/11/11 23:59:09 INFO Vacuumed table table=closures42962026/11/11 23:59:09 INFO Received push request method=POST path=/42972026/11/11 23:59:09 INFO Vacuumed table table=objects42982026/11/11 23:59:09 INFO Aborted multipart uploads count=0 kept=042992026/11/11 23:59:09 INFO Received push request method=POST path=/43002026/11/11 23:59:09 WARN Force mode enabled - objects will be deleted immediately without grace period43012026/11/11 23:59:09 INFO Received uploads request method=POST path=/43022026/11/11 23:59:09 INFO Received complete push request method=POST path=/43032026/11/11 23:59:09 INFO Received uploads request method=POST path=/43042026/11/11 23:59:09 INFO Completed push id=28 objects=26 duration=539.166µs43052026/11/11 23:59:09 INFO Received push request method=POST path=/43062026/11/11 23:59:09 INFO Received complete push request method=POST path=/43072026/11/11 23:59:09 INFO Received push request method=POST path=/43082026/11/11 23:59:09 INFO Completed push id=30 objects=3 duration=250.166µs43092026/11/11 23:59:09 INFO Received uploads request method=POST path=/43102026/11/11 23:59:09 INFO Received complete push request method=POST path=/43112026/11/11 23:59:09 INFO Completed push id=32 objects=19 duration=232.125µs43122026/11/11 23:59:09 INFO Received complete push request method=POST path=/43132026/11/11 23:59:09 INFO Completed push id=16 objects=4 duration=809.542µs43142026/11/11 23:59:09 INFO Aborted multipart uploads count=0 kept=043152026/11/11 23:59:09 WARN Force mode enabled - objects will be deleted immediately without grace period43162026/11/11 23:59:09 INFO Received uploads request method=POST path=/43172026/11/11 23:59:09 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043182026/11/11 23:59:09 INFO Received complete upload request method=POST path=/43192026/11/11 23:59:09 INFO Completed upload id=20 objects=5 duration=307.166µs43202026/11/11 23:59:09 INFO Received uploads request method=POST path=/43212026/11/11 23:59:09 INFO Vacuumed table table=pending_closures43222026/11/11 23:59:09 INFO Received push request method=POST path=/43232026/11/11 23:59:09 INFO Received push request method=POST path=/43242026/11/11 23:59:09 INFO Received complete push request method=POST path=/43252026/11/11 23:59:09 INFO Received push request method=POST path=/43262026/11/11 23:59:09 INFO Vacuumed table table=pending_objects43272026/11/11 23:59:09 INFO Completed push id=4 objects=6 duration=6.60675ms43282026/11/11 23:59:09 INFO Vacuumed table table=multipart_uploads43292026/11/11 23:59:09 INFO Vacuumed table table=closures43302026/11/11 23:59:09 INFO Received complete push request method=POST path=/43312026/11/11 23:59:09 INFO Aborted multipart uploads count=0 kept=043322026/11/11 23:59:09 INFO Vacuumed table table=objects43332026/11/11 23:59:09 INFO Completed push id=20 objects=3 duration=485.958µs43342026/11/11 23:59:09 WARN Force mode enabled - objects will be deleted immediately without grace period43352026/11/11 23:59:09 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043362026/11/11 23:59:09 INFO Vacuumed table table=pending_closures43372026/11/11 23:59:09 INFO Vacuumed table table=pending_objects43382026/11/11 23:59:09 INFO Vacuumed table table=multipart_uploads43392026/11/11 23:59:09 INFO Received push request method=POST path=/43402026/11/11 23:59:09 INFO Vacuumed table table=closures43412026/11/11 23:59:09 INFO Vacuumed table table=objects43422026/11/11 23:59:09 INFO Received push request method=POST path=/43432026/11/11 23:59:09 INFO Aborted multipart uploads count=0 kept=043442026/11/11 23:59:09 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043452026/11/11 23:59:09 INFO Received push request method=POST path=/43462026/11/11 23:59:09 INFO Vacuumed table table=pending_closures43472026/11/11 23:59:09 INFO Received complete upload request method=POST path=/43482026/11/11 23:59:09 INFO Vacuumed table table=pending_objects43492026/11/11 23:59:09 INFO Completed upload id=22 objects=6 duration=232.083µs43502026/11/11 23:59:09 INFO Vacuumed table table=multipart_uploads43512026/11/11 23:59:09 INFO Vacuumed table table=closures43522026/11/11 23:59:09 INFO Received uploads request method=POST path=/43532026/11/11 23:59:09 INFO Vacuumed table table=objects43542026/11/11 23:59:09 INFO Received push request method=POST path=/43552026/11/11 23:59:10 INFO Received uploads request method=POST path=/43562026/11/11 23:59:10 INFO Received push request method=POST path=/43572026/11/11 23:59:10 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."43582026/11/11 23:59:10 INFO Received push request method=POST path=/43592026/11/11 23:59:10 INFO Received uploads request method=POST path=/43602026/11/11 23:59:10 INFO Aborted multipart uploads count=0 kept=043612026/11/11 23:59:10 WARN Force mode enabled - objects will be deleted immediately without grace period43622026/11/11 23:59:10 INFO Received push request method=POST path=/43632026/11/11 23:59:10 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043642026/11/11 23:59:10 INFO Received complete upload request method=POST path=/43652026/11/11 23:59:10 INFO Received uploads request method=POST path=/43662026/11/11 23:59:10 INFO Vacuumed table table=pending_closures43672026/11/11 23:59:10 INFO Completed upload id=9 objects=2 duration=332.834µs43682026/11/11 23:59:10 INFO Vacuumed table table=pending_objects43692026/11/11 23:59:10 INFO Received complete push request method=POST path=/43702026/11/11 23:59:10 INFO Received uploads request method=POST path=/43712026/11/11 23:59:10 INFO Vacuumed table table=multipart_uploads43722026/11/11 23:59:10 INFO Completed push id=8 objects=3 duration=349.917µs43732026/11/11 23:59:10 INFO Vacuumed table table=closures43742026/11/11 23:59:10 INFO Received complete push request method=POST path=/43752026/11/11 23:59:10 INFO Vacuumed table table=objects43762026/11/11 23:59:10 INFO Received push request method=POST path=/43772026/11/11 23:59:10 INFO Received uploads request method=POST path=/43782026/11/11 23:59:10 INFO Completed push id=15 objects=24 duration=1.780541ms43792026/11/11 23:59:10 INFO Received uploads request method=POST path=/43802026/11/11 23:59:10 INFO Received push request method=POST path=/43812026/11/11 23:59:10 INFO Received uploads request method=POST path=/43822026/11/11 23:59:10 INFO Received complete upload request method=POST path=/43832026/11/11 23:59:10 INFO Received uploads request method=POST path=/43842026/11/11 23:59:10 INFO Received push request method=POST path=/43852026/11/11 23:59:10 INFO Received uploads request method=POST path=/43862026/11/11 23:59:10 INFO Received complete push request method=POST path=/43872026/11/11 23:59:10 INFO Completed upload id=13 objects=10 duration=309.375µs43882026/11/11 23:59:10 INFO Completed push id=15 objects=4 duration=723.333µs43892026/11/11 23:59:10 INFO Received complete push request method=POST path=/43902026/11/11 23:59:10 INFO Completed push id=31 objects=14 duration=717.708µs43912026/11/11 23:59:10 INFO Received push request method=POST path=/43922026/11/11 23:59:10 INFO Received uploads request method=POST path=/43932026/11/11 23:59:10 INFO Received uploads request method=POST path=/43942026/11/11 23:59:10 INFO Received complete push request method=POST path=/43952026/11/11 23:59:10 INFO Aborted multipart uploads count=0 kept=043962026/11/11 23:59:10 INFO Completed push id=7 objects=20 duration=1.152416ms43972026/11/11 23:59:10 INFO Received uploads request method=POST path=/43982026/11/11 23:59:10 WARN Force mode enabled - objects will be deleted immediately without grace period43992026/11/11 23:59:10 INFO Received push request method=POST path=/44002026/11/11 23:59:10 INFO Received uploads request method=POST path=/44012026/11/11 23:59:10 INFO Aborted multipart uploads count=0 kept=044022026/11/11 23:59:10 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044032026/11/11 23:59:10 INFO Vacuumed table table=pending_closures44042026/11/11 23:59:10 INFO Vacuumed table table=pending_objects44052026/11/11 23:59:10 INFO Vacuumed table table=multipart_uploads44062026/11/11 23:59:10 INFO Received complete upload request method=POST path=/44072026/11/11 23:59:10 INFO Vacuumed table table=closures44082026/11/11 23:59:10 INFO Completed upload id=6 objects=6 duration=328µs44092026/11/11 23:59:10 INFO Vacuumed table table=objects44102026/11/11 23:59:10 INFO Aborted multipart uploads count=0 kept=044112026/11/11 23:59:10 WARN Force mode enabled - objects will be deleted immediately without grace period44122026/11/11 23:59:10 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044132026/11/11 23:59:10 INFO Vacuumed table table=pending_closures44142026/11/11 23:59:10 INFO Vacuumed table table=pending_objects44152026/11/11 23:59:10 INFO Vacuumed table table=multipart_uploads44162026/11/11 23:59:10 INFO Vacuumed table table=closures44172026/11/11 23:59:10 INFO Vacuumed table table=objects44182026/11/11 23:59:10 INFO Received push request method=POST path=/44192026/11/11 23:59:10 INFO Received uploads request method=POST path=/44202026/11/11 23:59:10 INFO Aborted multipart uploads count=0 kept=044212026/11/11 23:59:10 INFO Received complete push request method=POST path=/44222026/11/11 23:59:10 WARN Force mode enabled - objects will be deleted immediately without grace period44232026/11/11 23:59:10 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044242026/11/11 23:59:10 INFO Completed push id=5 objects=3 duration=8.973083ms44252026/11/11 23:59:10 INFO Vacuumed table table=pending_closures44262026/11/11 23:59:10 INFO Received push request method=POST path=/44272026/11/11 23:59:10 INFO Received complete push request method=POST path=/44282026/11/11 23:59:10 INFO Vacuumed table table=pending_objects44292026/11/11 23:59:10 INFO Completed push id=8 objects=6 duration=361.834µs44302026/11/11 23:59:10 INFO Vacuumed table table=multipart_uploads44312026/11/11 23:59:10 INFO Received push request method=POST path=/44322026/11/11 23:59:10 INFO Vacuumed table table=closures44332026/11/11 23:59:10 INFO Received uploads request method=POST path=/44342026/11/11 23:59:10 INFO Vacuumed table table=objects44352026/11/11 23:59:10 INFO Received push request method=POST path=/44362026/11/11 23:59:10 INFO Received push request method=POST path=/44372026/11/11 23:59:10 INFO Received push request method=POST path=/44382026/11/11 23:59:10 INFO Received uploads request method=POST path=/44392026/11/11 23:59:10 INFO Aborted multipart uploads count=0 kept=044402026/11/11 23:59:10 WARN Force mode enabled - objects will be deleted immediately without grace period44412026/11/11 23:59:10 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044422026/11/11 23:59:10 INFO Vacuumed table table=pending_closures44432026/11/11 23:59:10 INFO Vacuumed table table=pending_objects44442026/11/11 23:59:10 INFO Vacuumed table table=multipart_uploads44452026/11/11 23:59:10 INFO Received complete push request method=POST path=/44462026/11/11 23:59:10 INFO Vacuumed table table=closures44472026/11/11 23:59:10 INFO Completed push id=14 objects=3 duration=308.042µs44482026/11/11 23:59:10 INFO Vacuumed table table=objects44492026/11/11 23:59:10 INFO Aborted multipart uploads count=0 kept=044502026/11/11 23:59:10 WARN Force mode enabled - objects will be deleted immediately without grace period44512026/11/11 23:59:10 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044522026/11/11 23:59:10 INFO Received complete upload request method=POST path=/44532026/11/11 23:59:10 INFO Vacuumed table table=pending_closures44542026/11/11 23:59:10 INFO Completed upload id=20 objects=4 duration=396.75µs44552026/11/11 23:59:10 INFO Vacuumed table table=pending_objects44562026/11/11 23:59:10 INFO Vacuumed table table=multipart_uploads44572026/11/11 23:59:10 INFO Vacuumed table table=closures44582026/11/11 23:59:10 INFO Vacuumed table table=objects44592026/11/11 23:59:10 INFO Received push request method=POST path=/44602026/11/11 23:59:10 INFO Received complete upload request method=POST path=/44612026/11/11 23:59:10 INFO Completed upload id=10 objects=17 duration=400.542µs44622026/11/11 23:59:10 INFO Received push request method=POST path=/44632026/11/11 23:59:10 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=8 objects-deleted-after-grace-period=8 objects-failed-to-delete=044642026/11/11 23:59:10 INFO Vacuumed table table=pending_closures44652026/11/11 23:59:10 INFO Vacuumed table table=pending_objects44662026/11/11 23:59:10 INFO Vacuumed table table=multipart_uploads44672026/11/11 23:59:10 INFO Received push request method=POST path=/44682026/11/11 23:59:10 INFO Vacuumed table table=closures44692026/11/11 23:59:10 INFO Vacuumed table table=objects44702026/11/11 23:59:10 INFO Received complete push request method=POST path=/44712026/11/11 23:59:10 INFO Completed push id=14 objects=6 duration=467.75µs44722026/11/11 23:59:10 INFO Aborted multipart uploads count=0 kept=044732026/11/11 23:59:10 WARN Force mode enabled - objects will be deleted immediately without grace period44742026/11/11 23:59:10 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044752026/11/11 23:59:10 INFO Received push request method=POST path=/44762026/11/11 23:59:10 INFO Vacuumed table table=pending_closures44772026/11/11 23:59:10 INFO Vacuumed table table=pending_objects44782026/11/11 23:59:10 INFO Vacuumed table table=multipart_uploads44792026/11/11 23:59:10 INFO Vacuumed table table=closures44802026/11/11 23:59:10 INFO Received complete push request method=POST path=/44812026/11/11 23:59:10 INFO Received push request method=POST path=/44822026/11/11 23:59:10 INFO Vacuumed table table=objects44832026/11/11 23:59:10 INFO Completed push id=7 objects=7 duration=427.167µs44842026/11/11 23:59:10 INFO Aborted multipart uploads count=0 kept=044852026/11/11 23:59:10 INFO Received complete upload request method=POST path=/44862026/11/11 23:59:10 INFO Completed upload id=15 objects=3 duration=908.75µs44872026/11/11 23:59:10 INFO Received complete push request method=POST path=/44882026/11/11 23:59:10 INFO Completed push id=12 objects=6 duration=297.583µs44892026/11/11 23:59:10 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=1 objects-deleted-after-grace-period=0 objects-failed-to-delete=044902026/11/11 23:59:10 INFO Vacuumed table table=pending_closures44912026/11/11 23:59:10 INFO Vacuumed table table=pending_objects44922026/11/11 23:59:10 INFO Received push request method=POST path=/44932026/11/11 23:59:10 INFO Vacuumed table table=multipart_uploads44942026/11/11 23:59:10 INFO Vacuumed table table=closures44952026/11/11 23:59:10 INFO Vacuumed table table=objects44962026/11/11 23:59:10 INFO Received push request method=POST path=/44972026/11/11 23:59:10 INFO Received complete upload request method=POST path=/44982026/11/11 23:59:10 INFO Received uploads request method=POST path=/44992026/11/11 23:59:10 INFO Completed upload id=23 objects=16 duration=6.151625ms45002026/11/11 23:59:10 INFO Received uploads request method=POST path=/45012026/11/11 23:59:10 INFO Aborted multipart uploads count=0 kept=045022026/11/11 23:59:10 INFO Received complete push request method=POST path=/45032026/11/11 23:59:10 WARN Force mode enabled - objects will be deleted immediately without grace period45042026/11/11 23:59:10 INFO Completed push id=12 objects=2 duration=5.682333ms45052026/11/11 23:59:10 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045062026/11/11 23:59:10 INFO Vacuumed table table=pending_closures45072026/11/11 23:59:10 INFO Received uploads request method=POST path=/45082026/11/11 23:59:10 INFO Vacuumed table table=pending_objects45092026/11/11 23:59:10 INFO Vacuumed table table=multipart_uploads45102026/11/11 23:59:10 INFO Received complete push request method=POST path=/45112026/11/11 23:59:10 INFO Vacuumed table table=closures45122026/11/11 23:59:10 INFO Completed push id=16 objects=5 duration=6.280167ms45132026/11/11 23:59:10 INFO Vacuumed table table=objects45142026/11/11 23:59:10 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=11 objects-failed-to-delete=045152026/11/11 23:59:10 INFO Received uploads request method=POST path=/45162026/11/11 23:59:10 INFO Vacuumed table table=pending_closures45172026/11/11 23:59:10 INFO Received uploads request method=POST path=/45182026/11/11 23:59:10 INFO Aborted multipart uploads count=0 kept=045192026/11/11 23:59:10 INFO Received uploads request method=POST path=/45202026/11/11 23:59:10 INFO Vacuumed table table=pending_objects45212026/11/11 23:59:10 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=045222026/11/11 23:59:10 INFO Aborted multipart uploads count=0 kept=045232026/11/11 23:59:10 INFO Vacuumed table table=multipart_uploads45242026/11/11 23:59:10 INFO Vacuumed table table=pending_closures45252026/11/11 23:59:10 INFO Vacuumed table table=closures45262026/11/11 23:59:10 INFO Vacuumed table table=pending_objects45272026/11/11 23:59:10 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045282026/11/11 23:59:10 INFO Vacuumed table table=multipart_uploads45292026/11/11 23:59:10 INFO Vacuumed table table=pending_closures45302026/11/11 23:59:10 INFO Received complete push request method=POST path=/45312026/11/11 23:59:10 INFO Vacuumed table table=objects45322026/11/11 23:59:10 INFO Vacuumed table table=closures45332026/11/11 23:59:10 INFO Completed push id=16 objects=7 duration=5.479791ms45342026/11/11 23:59:10 INFO Vacuumed table table=pending_objects45352026/11/11 23:59:10 INFO Vacuumed table table=objects45362026/11/11 23:59:10 INFO Vacuumed table table=multipart_uploads45372026/11/11 23:59:10 INFO Vacuumed table table=closures45382026/11/11 23:59:10 INFO Vacuumed table table=objects45392026/11/11 23:59:10 INFO Received push request method=POST path=/45402026/11/11 23:59:10 INFO Aborted multipart uploads count=0 kept=045412026/11/11 23:59:10 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045422026/11/11 23:59:10 INFO Received push request method=POST path=/45432026/11/11 23:59:10 INFO Vacuumed table table=pending_closures45442026/11/11 23:59:10 INFO Received push request method=POST path=/45452026/11/11 23:59:10 INFO Received uploads request method=POST path=/45462026/11/11 23:59:10 INFO Vacuumed table table=pending_objects45472026/11/11 23:59:10 INFO Aborted multipart uploads count=0 kept=045482026/11/11 23:59:10 INFO Vacuumed table table=multipart_uploads45492026/11/11 23:59:10 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045502026/11/11 23:59:10 INFO Vacuumed table table=closures45512026/11/11 23:59:10 INFO Vacuumed table table=pending_closures45522026/11/11 23:59:10 INFO Vacuumed table table=objects45532026/11/11 23:59:10 INFO Received uploads request method=POST path=/45542026/11/11 23:59:10 INFO Received uploads request method=POST path=/45552026/11/11 23:59:10 INFO Received push request method=POST path=/45562026/11/11 23:59:10 INFO Vacuumed table table=pending_objects45572026/11/11 23:59:10 INFO Vacuumed table table=multipart_uploads45582026/11/11 23:59:10 INFO Aborted multipart uploads count=0 kept=045592026/11/11 23:59:10 INFO Vacuumed table table=closures45602026/11/11 23:59:10 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045612026/11/11 23:59:10 INFO Vacuumed table table=pending_closures45622026/11/11 23:59:10 INFO Received complete push request method=POST path=/45632026/11/11 23:59:10 INFO Vacuumed table table=objects45642026/11/11 23:59:10 INFO Completed push id=14 objects=4 duration=993µs45652026/11/11 23:59:11 INFO Received push request method=POST path=/45662026/11/11 23:59:11 INFO Vacuumed table table=pending_objects45672026/11/11 23:59:11 INFO Received uploads request method=POST path=/45682026/11/11 23:59:11 INFO Received complete push request method=POST path=/45692026/11/11 23:59:11 INFO Vacuumed table table=multipart_uploads45702026/11/11 23:59:11 INFO Completed push id=22 objects=3 duration=6.046375ms45712026/11/11 23:59:11 INFO Vacuumed table table=closures45722026/11/11 23:59:11 INFO Vacuumed table table=objects45732026/11/11 23:59:11 INFO Aborted multipart uploads count=0 kept=045742026/11/11 23:59:11 WARN Force mode enabled - objects will be deleted immediately without grace period45752026/11/11 23:59:11 INFO Aborted multipart uploads count=0 kept=045762026/11/11 23:59:11 WARN Force mode enabled - objects will be deleted immediately without grace period45772026/11/11 23:59:11 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045782026/11/11 23:59:11 INFO Received complete push request method=POST path=/45792026/11/11 23:59:11 INFO Aborted multipart uploads count=0 kept=045802026/11/11 23:59:11 INFO Vacuumed table table=pending_closures45812026/11/11 23:59:11 INFO Completed push id=23 objects=6 duration=317.25µs45822026/11/11 23:59:11 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045832026/11/11 23:59:11 INFO Vacuumed table table=pending_closures45842026/11/11 23:59:11 INFO Received complete push request method=POST path=/45852026/11/11 23:59:11 INFO Vacuumed table table=pending_objects45862026/11/11 23:59:11 INFO Vacuumed table table=pending_objects45872026/11/11 23:59:11 INFO Vacuumed table table=multipart_uploads45882026/11/11 23:59:11 INFO Vacuumed table table=multipart_uploads45892026/11/11 23:59:11 INFO Completed push id=24 objects=6 duration=291.542µs45902026/11/11 23:59:11 INFO Vacuumed table table=closures45912026/11/11 23:59:11 INFO Vacuumed table table=closures45922026/11/11 23:59:11 INFO Vacuumed table table=objects45932026/11/11 23:59:11 INFO Vacuumed table table=objects45942026/11/11 23:59:11 INFO Received push request method=POST path=/45952026/11/11 23:59:11 INFO Received push request method=POST path=/45962026/11/11 23:59:11 INFO Received uploads request method=POST path=/45972026/11/11 23:59:11 INFO Received complete push request method=POST path=/45982026/11/11 23:59:11 INFO Received push request method=POST path=/45992026/11/11 23:59:11 INFO Completed push id=13 objects=22 duration=400.708µs46002026/11/11 23:59:11 INFO Received complete push request method=POST path=/46012026/11/11 23:59:11 INFO Completed push id=21 objects=6 duration=277.25µs46022026/11/11 23:59:11 INFO Aborted multipart uploads count=0 kept=046032026/11/11 23:59:11 INFO Received complete push request method=POST path=/46042026/11/11 23:59:11 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046052026/11/11 23:59:11 INFO Received complete upload request method=POST path=/46062026/11/11 23:59:11 INFO Completed push id=17 objects=22 duration=571.791µs46072026/11/11 23:59:11 INFO Completed upload id=22 objects=7 duration=332.334µs46082026/11/11 23:59:11 INFO Vacuumed table table=pending_closures46092026/11/11 23:59:11 INFO Received uploads request method=POST path=/46102026/11/11 23:59:11 INFO Vacuumed table table=pending_objects46112026/11/11 23:59:11 INFO Vacuumed table table=multipart_uploads46122026/11/11 23:59:11 INFO Received complete push request method=POST path=/46132026/11/11 23:59:11 INFO Vacuumed table table=closures46142026/11/11 23:59:11 INFO Completed push id=24 objects=4 duration=5.710583ms46152026/11/11 23:59:11 INFO Vacuumed table table=objects46162026/11/11 23:59:11 INFO Aborted multipart uploads count=0 kept=046172026/11/11 23:59:11 INFO Received complete push request method=POST path=/46182026/11/11 23:59:11 INFO Received complete push request method=POST path=/46192026/11/11 23:59:11 INFO Completed push id=25 objects=9 duration=256.458µs46202026/11/11 23:59:11 WARN Force mode enabled - objects will be deleted immediately without grace period46212026/11/11 23:59:11 INFO Completed push id=25 objects=7 duration=304.541µs46222026/11/11 23:59:11 INFO Received push request method=POST path=/46232026/11/11 23:59:11 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046242026/11/11 23:59:11 INFO Vacuumed table table=pending_closures46252026/11/11 23:59:11 INFO Received complete upload request method=POST path=/46262026/11/11 23:59:11 INFO Completed upload id=26 objects=3 duration=170.417µs46272026/11/11 23:59:11 INFO Vacuumed table table=pending_objects46282026/11/11 23:59:11 INFO Received complete push request method=POST path=/46292026/11/11 23:59:11 INFO Vacuumed table table=multipart_uploads46302026/11/11 23:59:11 INFO Completed push id=19 objects=14 duration=399.083µs46312026/11/11 23:59:11 INFO Vacuumed table table=closures46322026/11/11 23:59:11 INFO Vacuumed table table=objects46332026/11/11 23:59:11 INFO Received uploads request method=POST path=/46342026/11/11 23:59:11 INFO Received complete upload request method=POST path=/46352026/11/11 23:59:11 INFO Completed upload id=19 objects=2 duration=780.708µs46362026/11/11 23:59:11 INFO Received uploads request method=POST path=/46372026/11/11 23:59:11 INFO Received complete push request method=POST path=/46382026/11/11 23:59:11 INFO Completed push id=33 objects=7 duration=454.083µs46392026/11/11 23:59:11 INFO Received complete upload request method=POST path=/46402026/11/11 23:59:11 INFO Completed upload id=26 objects=3 duration=145.583µs46412026/11/11 23:59:11 INFO Aborted multipart uploads count=0 kept=046422026/11/11 23:59:11 WARN Force mode enabled - objects will be deleted immediately without grace period46432026/11/11 23:59:11 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046442026/11/11 23:59:11 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."46452026/11/11 23:59:11 INFO Vacuumed table table=pending_closures46462026/11/11 23:59:11 INFO Received uploads request method=POST path=/46472026/11/11 23:59:11 INFO Received complete upload request method=POST path=/46482026/11/11 23:59:11 INFO Vacuumed table table=pending_objects46492026/11/11 23:59:11 INFO Completed upload id=27 objects=7 duration=194.167µs46502026/11/11 23:59:11 INFO Received uploads request method=POST path=/46512026/11/11 23:59:11 INFO Vacuumed table table=multipart_uploads46522026/11/11 23:59:11 INFO Vacuumed table table=closures46532026/11/11 23:59:11 INFO Received complete upload request method=POST path=/46542026/11/11 23:59:11 INFO Completed upload id=28 objects=3 duration=373.458µs46552026/11/11 23:59:11 INFO Vacuumed table table=objects46562026/11/11 23:59:11 INFO Aborted multipart uploads count=0 kept=046572026/11/11 23:59:11 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."46582026/11/11 23:59:11 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."46592026/11/11 23:59:11 INFO Received complete upload request method=POST path=/46602026/11/11 23:59:11 INFO Completed upload id=20 objects=6 duration=284.5µs46612026/11/11 23:59:11 INFO Aborted multipart uploads count=0 kept=046622026/11/11 23:59:11 WARN Force mode enabled - objects will be deleted immediately without grace period46632026/11/11 23:59:11 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=3 objects-failed-to-delete=046642026/11/11 23:59:11 INFO Vacuumed table table=pending_closures46652026/11/11 23:59:11 INFO Received uploads request method=POST path=/46662026/11/11 23:59:11 INFO Vacuumed table table=pending_objects46672026/11/11 23:59:11 INFO Vacuumed table table=multipart_uploads46682026/11/11 23:59:11 INFO Received complete push request method=POST path=/46692026/11/11 23:59:11 INFO Vacuumed table table=closures46702026/11/11 23:59:11 INFO Completed push id=29 objects=23 duration=305.208µs46712026/11/11 23:59:11 INFO Vacuumed table table=objects46722026/11/11 23:59:11 INFO Received complete upload request method=POST path=/46732026/11/11 23:59:11 INFO Completed upload id=21 objects=2 duration=380.166µs46742026/11/11 23:59:11 INFO Received complete push request method=POST path=/46752026/11/11 23:59:11 INFO Received complete upload request method=POST path=/46762026/11/11 23:59:11 INFO Received complete push request method=POST path=/46772026/11/11 23:59:11 INFO Completed push id=30 objects=6 duration=223µs46782026/11/11 23:59:11 INFO Completed upload id=21 objects=7 duration=856.25µs46792026/11/11 23:59:11 INFO Received uploads request method=POST path=/46802026/11/11 23:59:11 INFO Completed push id=16 objects=16 duration=1.404208ms46812026/11/11 23:59:11 INFO Aborted multipart uploads count=0 kept=046822026/11/11 23:59:11 WARN Force mode enabled - objects will be deleted immediately without grace period46832026/11/11 23:59:11 INFO Received push request method=POST path=/46842026/11/11 23:59:11 INFO Aborted multipart uploads count=0 kept=046852026/11/11 23:59:11 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046862026/11/11 23:59:11 INFO Vacuumed table table=pending_closures46872026/11/11 23:59:11 INFO Vacuumed table table=pending_objects46882026/11/11 23:59:11 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."46892026/11/11 23:59:11 INFO Vacuumed table table=multipart_uploads46902026/11/11 23:59:11 INFO Received uploads request method=POST path=/46912026/11/11 23:59:11 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."46922026/11/11 23:59:11 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."46932026/11/11 23:59:11 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."46942026/11/11 23:59:11 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."46952026/11/11 23:59:11 INFO Vacuumed table table=closures46962026/11/11 23:59:11 INFO Received uploads request method=POST path=/46972026/11/11 23:59:11 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."46982026/11/11 23:59:11 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."46992026/11/11 23:59:11 INFO Vacuumed table table=objects47002026/11/11 23:59:11 INFO Received complete upload request method=POST path=/47012026/11/11 23:59:11 INFO Completed upload id=27 objects=4 duration=277.708µs47022026/11/11 23:59:11 INFO Received complete upload request method=POST path=/47032026/11/11 23:59:11 INFO Received complete push request method=POST path=/47042026/11/11 23:59:11 INFO Completed upload id=26 objects=24 duration=1.0255ms47052026/11/11 23:59:11 INFO Received uploads request method=POST path=/47062026/11/11 23:59:11 INFO Received complete push request method=POST path=/47072026/11/11 23:59:11 INFO Completed push id=13 objects=11 duration=375.125µs47082026/11/11 23:59:11 INFO Completed push id=7 objects=15 duration=6.438958ms47092026/11/11 23:59:11 INFO Received complete push request method=POST path=/47102026/11/11 23:59:11 INFO Completed push id=33 objects=3 duration=649.042µs47112026/11/11 23:59:11 INFO Received uploads request method=POST path=/47122026/11/11 23:59:11 INFO Received uploads request method=POST path=/47132026/11/11 23:59:11 INFO Received uploads request method=POST path=/47142026/11/11 23:59:11 INFO Received complete push request method=POST path=/47152026/11/11 23:59:11 INFO Received push request method=POST path=/47162026/11/11 23:59:11 INFO Completed push id=34 objects=6 duration=306.083µs47172026/11/11 23:59:11 INFO Aborted multipart uploads count=0 kept=047182026/11/11 23:59:11 INFO Received complete push request method=POST path=/47192026/11/11 23:59:11 INFO Received push request method=POST path=/47202026/11/11 23:59:11 INFO Completed push id=35 objects=8 duration=302µs47212026/11/11 23:59:11 WARN Force mode enabled - objects will be deleted immediately without grace period47222026/11/11 23:59:11 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047232026/11/11 23:59:11 INFO Received push request method=POST path=/47242026/11/11 23:59:11 INFO Vacuumed table table=pending_closures47252026/11/11 23:59:11 INFO Received push request method=POST path=/47262026/11/11 23:59:11 INFO Received complete upload request method=POST path=/47272026/11/11 23:59:11 INFO Completed upload id=36 objects=9 duration=326.083µs47282026/11/11 23:59:11 INFO Vacuumed table table=pending_objects47292026/11/11 23:59:11 INFO Vacuumed table table=multipart_uploads47302026/11/11 23:59:11 INFO Received uploads request method=POST path=/47312026/11/11 23:59:11 INFO Vacuumed table table=closures47322026/11/11 23:59:11 INFO Vacuumed table table=objects47332026/11/11 23:59:11 INFO Received complete push request method=POST path=/47342026/11/11 23:59:11 INFO Aborted multipart uploads count=0 kept=047352026/11/11 23:59:11 WARN Force mode enabled - objects will be deleted immediately without grace period47362026/11/11 23:59:11 INFO Completed push id=24 objects=24 duration=6.29575ms47372026/11/11 23:59:11 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047382026/11/11 23:59:11 INFO Aborted multipart uploads count=0 kept=047392026/11/11 23:59:11 INFO Vacuumed table table=pending_closures47402026/11/11 23:59:11 WARN Force mode enabled - objects will be deleted immediately without grace period47412026/11/11 23:59:11 INFO Received uploads request method=POST path=/47422026/11/11 23:59:11 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047432026/11/11 23:59:11 INFO Vacuumed table table=pending_objects47442026/11/11 23:59:11 INFO Vacuumed table table=pending_closures47452026/11/11 23:59:11 INFO Vacuumed table table=multipart_uploads47462026/11/11 23:59:11 INFO Aborted multipart uploads count=0 kept=047472026/11/11 23:59:11 INFO Vacuumed table table=closures47482026/11/11 23:59:11 WARN Force mode enabled - objects will be deleted immediately without grace period47492026/11/11 23:59:11 INFO Vacuumed table table=objects47502026/11/11 23:59:11 INFO Vacuumed table table=pending_objects47512026/11/11 23:59:11 INFO Vacuumed table table=multipart_uploads47522026/11/11 23:59:11 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."47532026/11/11 23:59:11 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."47542026/11/11 23:59:11 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."47552026/11/11 23:59:11 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."47562026/11/11 23:59:11 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."47572026/11/11 23:59:11 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."47582026/11/11 23:59:11 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."47592026/11/11 23:59:11 INFO Vacuumed table table=closures47602026/11/11 23:59:11 INFO Received complete upload request method=POST path=/47612026/11/11 23:59:11 INFO Vacuumed table table=objects47622026/11/11 23:59:11 INFO Completed upload id=21 objects=3 duration=275.042µs47632026/11/11 23:59:11 INFO Aborted multipart uploads count=0 kept=047642026/11/11 23:59:11 WARN Force mode enabled - objects will be deleted immediately without grace period47652026/11/11 23:59:11 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=7 objects-failed-to-delete=047662026/11/11 23:59:11 INFO Received uploads request method=POST path=/47672026/11/11 23:59:11 INFO Received complete push request method=POST path=/47682026/11/11 23:59:11 INFO Received uploads request method=POST path=/47692026/11/11 23:59:11 INFO Vacuumed table table=pending_closures47702026/11/11 23:59:11 INFO Completed push id=16 objects=14 duration=754.625µs47712026/11/11 23:59:11 INFO Received push request method=POST path=/47722026/11/11 23:59:11 INFO Aborted multipart uploads count=0 kept=047732026/11/11 23:59:11 WARN Force mode enabled - objects will be deleted immediately without grace period47742026/11/11 23:59:11 INFO Received uploads request method=POST path=/47752026/11/11 23:59:11 INFO Vacuumed table table=pending_objects47762026/11/11 23:59:11 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047772026/11/11 23:59:11 INFO Vacuumed table table=multipart_uploads47782026/11/11 23:59:11 INFO Aborted multipart uploads count=0 kept=047792026/11/11 23:59:11 INFO Vacuumed table table=closures47802026/11/11 23:59:11 INFO Vacuumed table table=pending_closures47812026/11/11 23:59:11 WARN Force mode enabled - objects will be deleted immediately without grace period47822026/11/11 23:59:11 INFO Vacuumed table table=objects47832026/11/11 23:59:11 INFO Vacuumed table table=pending_objects47842026/11/11 23:59:11 INFO Vacuumed table table=multipart_uploads47852026/11/11 23:59:11 INFO Vacuumed table table=closures47862026/11/11 23:59:11 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047872026/11/11 23:59:11 INFO Vacuumed table table=pending_closures47882026/11/11 23:59:11 INFO Vacuumed table table=objects47892026/11/11 23:59:11 INFO Vacuumed table table=pending_objects47902026/11/11 23:59:11 INFO Aborted multipart uploads count=0 kept=047912026/11/11 23:59:11 INFO Vacuumed table table=multipart_uploads47922026/11/11 23:59:11 WARN Force mode enabled - objects will be deleted immediately without grace period47932026/11/11 23:59:11 INFO Vacuumed table table=closures47942026/11/11 23:59:11 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047952026/11/11 23:59:11 INFO Aborted multipart uploads count=0 kept=047962026/11/11 23:59:11 INFO Vacuumed table table=pending_closures47972026/11/11 23:59:11 INFO Vacuumed table table=objects47982026/11/11 23:59:11 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047992026/11/11 23:59:11 INFO Vacuumed table table=pending_objects48002026/11/11 23:59:11 INFO Vacuumed table table=pending_closures48012026/11/11 23:59:11 INFO Vacuumed table table=pending_objects48022026/11/11 23:59:11 INFO Vacuumed table table=multipart_uploads48032026/11/11 23:59:11 INFO Received complete upload request method=POST path=/48042026/11/11 23:59:11 INFO Vacuumed table table=multipart_uploads48052026/11/11 23:59:11 INFO Vacuumed table table=closures48062026/11/11 23:59:11 INFO Vacuumed table table=closures48072026/11/11 23:59:11 INFO Completed upload id=27 objects=26 duration=383.833µs48082026/11/11 23:59:11 INFO Vacuumed table table=objects48092026/11/11 23:59:11 INFO Vacuumed table table=objects48102026/11/11 23:59:11 INFO Received push request method=POST path=/48112026/11/11 23:59:11 INFO Received push request method=POST path=/48122026/11/11 23:59:11 INFO Received complete push request method=POST path=/48132026/11/11 23:59:11 INFO Received push request method=POST path=/48142026/11/11 23:59:11 INFO Completed push id=28 objects=6 duration=492.708µs48152026/11/11 23:59:11 INFO Received uploads request method=POST path=/48162026/11/11 23:59:11 INFO Received complete push request method=POST path=/48172026/11/11 23:59:11 INFO Received complete push request method=POST path=/48182026/11/11 23:59:11 INFO Received complete push request method=POST path=/48192026/11/11 23:59:11 INFO Completed push id=29 objects=6 duration=421µs48202026/11/11 23:59:11 INFO Completed push id=22 objects=3 duration=938.042µs48212026/11/11 23:59:11 INFO Completed push id=9 objects=9 duration=6.812459ms48222026/11/11 23:59:11 INFO Received complete upload request method=POST path=/48232026/11/11 23:59:11 INFO Completed upload id=25 objects=6 duration=520.708µs48242026/11/11 23:59:11 INFO Received uploads request method=POST path=/48252026/11/11 23:59:11 INFO Received complete upload request method=POST path=/48262026/11/11 23:59:11 INFO Completed upload id=23 objects=7 duration=932.958µs48272026/11/11 23:59:11 INFO Received push request method=POST path=/48282026/11/11 23:59:11 INFO Received complete upload request method=POST path=/48292026/11/11 23:59:11 INFO Received push request method=POST path=/48302026/11/11 23:59:11 INFO Completed upload id=26 objects=5 duration=206.708µs48312026/11/11 23:59:11 INFO Received complete push request method=POST path=/48322026/11/11 23:59:11 INFO Received uploads request method=POST path=/48332026/11/11 23:59:11 INFO Completed push id=24 objects=3 duration=375.375µs48342026/11/11 23:59:11 INFO Received push request method=POST path=/48352026/11/11 23:59:11 INFO Received complete push request method=POST path=/48362026/11/11 23:59:11 INFO Received complete push request method=POST path=/48372026/11/11 23:59:11 INFO Completed push id=11 objects=9 duration=311.709µs48382026/11/11 23:59:11 INFO Completed push id=27 objects=3 duration=428.25µs48392026/11/11 23:59:11 INFO Received complete upload request method=POST path=/48402026/11/11 23:59:11 INFO Aborted multipart uploads count=0 kept=048412026/11/11 23:59:11 INFO Completed upload id=21 objects=8 duration=272.25µs48422026/11/11 23:59:11 WARN Force mode enabled - objects will be deleted immediately without grace period48432026/11/11 23:59:11 INFO Received complete upload request method=POST path=/48442026/11/11 23:59:11 INFO Completed upload id=28 objects=5 duration=379.083µs48452026/11/11 23:59:11 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048462026/11/11 23:59:11 INFO Received complete push request method=POST path=/48472026/11/11 23:59:11 INFO Received complete push request method=POST path=/48482026/11/11 23:59:11 INFO Completed push id=22 objects=6 duration=244.083µs48492026/11/11 23:59:11 INFO Vacuumed table table=pending_closures48502026/11/11 23:59:11 INFO Completed push id=29 objects=17 duration=6.107417ms48512026/11/11 23:59:11 INFO Received complete push request method=POST path=/48522026/11/11 23:59:11 INFO Vacuumed table table=pending_objects48532026/11/11 23:59:11 INFO Completed push id=24 objects=8 duration=296.5µs48542026/11/11 23:59:11 INFO Vacuumed table table=multipart_uploads48552026/11/11 23:59:11 INFO Received complete push request method=POST path=/48562026/11/11 23:59:11 INFO Aborted multipart uploads count=0 kept=048572026/11/11 23:59:11 INFO Completed push id=15 objects=24 duration=431.916µs48582026/11/11 23:59:11 INFO Vacuumed table table=closures48592026/11/11 23:59:11 WARN Force mode enabled - objects will be deleted immediately without grace period48602026/11/11 23:59:11 INFO Vacuumed table table=objects48612026/11/11 23:59:11 INFO Received complete upload request method=POST path=/48622026/11/11 23:59:11 INFO Received complete upload request method=POST path=/48632026/11/11 23:59:11 INFO Completed upload id=25 objects=9 duration=169.375µs48642026/11/11 23:59:11 INFO Completed upload id=19 objects=2 duration=3.696083ms48652026/11/11 23:59:11 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048662026/11/11 23:59:11 INFO Vacuumed table table=pending_closures48672026/11/11 23:59:11 INFO Vacuumed table table=pending_objects48682026/11/11 23:59:11 INFO Vacuumed table table=multipart_uploads48692026/11/11 23:59:11 INFO Received complete upload request method=POST path=/48702026/11/11 23:59:11 INFO Received complete upload request method=POST path=/48712026/11/11 23:59:11 INFO Vacuumed table table=closures48722026/11/11 23:59:11 INFO Completed upload id=27 objects=3 duration=217.083µs48732026/11/11 23:59:11 INFO Vacuumed table table=objects48742026/11/11 23:59:11 INFO Completed upload id=24 objects=7 duration=4.398875ms48752026/11/11 23:59:11 INFO Aborted multipart uploads count=0 kept=048762026/11/11 23:59:11 INFO Received complete upload request method=POST path=/48772026/11/11 23:59:11 INFO Completed upload id=28 objects=6 duration=669.542µs48782026/11/11 23:59:11 INFO Received complete push request method=POST path=/48792026/11/11 23:59:11 INFO Completed push id=26 objects=3 duration=546.834µs48802026/11/11 23:59:11 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=048812026/11/11 23:59:11 INFO Received complete push request method=POST path=/48822026/11/11 23:59:11 INFO Completed push id=30 objects=4 duration=211.667µs48832026/11/11 23:59:11 INFO Received complete push request method=POST path=/48842026/11/11 23:59:11 INFO Completed push id=31 objects=7 duration=315.708µs48852026/11/11 23:59:11 INFO Vacuumed table table=pending_closures48862026/11/11 23:59:11 INFO Received complete upload request method=POST path=/48872026/11/11 23:59:11 INFO Received complete upload request method=POST path=/48882026/11/11 23:59:11 INFO Completed upload id=28 objects=8 duration=242.041µs48892026/11/11 23:59:11 INFO Completed upload id=30 objects=8 duration=417.5µs48902026/11/11 23:59:11 INFO Vacuumed table table=pending_objects48912026/11/11 23:59:11 INFO Vacuumed table table=multipart_uploads48922026/11/11 23:59:11 INFO Vacuumed table table=closures48932026/11/11 23:59:11 INFO Vacuumed table table=objects48942026/11/11 23:59:11 INFO Received complete upload request method=POST path=/48952026/11/11 23:59:11 INFO Received complete upload request method=POST path=/48962026/11/11 23:59:11 INFO Completed upload id=32 objects=5 duration=158.959µs48972026/11/11 23:59:11 INFO Completed upload id=29 objects=10 duration=156.083µs48982026/11/11 23:59:11 INFO Received complete push request method=POST path=/48992026/11/11 23:59:11 INFO Completed push id=19 objects=14 duration=790.333µs49002026/11/11 23:59:11 INFO Received complete upload request method=POST path=/49012026/11/11 23:59:11 INFO Completed upload id=30 objects=20 duration=211.084µs49022026/11/11 23:59:11 INFO Received uploads request method=POST path=/49032026/11/11 23:59:11 INFO Aborted multipart uploads count=0 kept=049042026/11/11 23:59:11 WARN Force mode enabled - objects will be deleted immediately without grace period49052026/11/11 23:59:11 INFO Received complete push request method=POST path=/49062026/11/11 23:59:11 INFO Completed push id=31 objects=11 duration=215.708µs49072026/11/11 23:59:11 INFO Received complete upload request method=POST path=/49082026/11/11 23:59:11 INFO Received complete upload request method=POST path=/49092026/11/11 23:59:11 INFO Received complete upload request method=POST path=/49102026/11/11 23:59:11 INFO Completed upload id=32 objects=2 duration=234.416µs49112026/11/11 23:59:11 INFO Completed upload id=31 objects=6 duration=301.542µs49122026/11/11 23:59:11 INFO Completed upload id=26 objects=2 duration=679.917µs49132026/11/11 23:59:11 INFO Received complete push request method=POST path=/49142026/11/11 23:59:11 INFO Completed push id=33 objects=3 duration=281.75µs49152026/11/11 23:59:11 INFO Received complete upload request method=POST path=/49162026/11/11 23:59:11 INFO Completed upload id=32 objects=4 duration=156.542µs49172026/11/11 23:59:11 INFO Aborted multipart uploads count=0 kept=049182026/11/11 23:59:11 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049192026/11/11 23:59:11 INFO Received complete upload request method=POST path=/49202026/11/11 23:59:11 INFO Completed upload id=34 objects=3 duration=185.25µs49212026/11/11 23:59:11 INFO Aborted multipart uploads count=0 kept=049222026/11/11 23:59:11 WARN Force mode enabled - objects will be deleted immediately without grace period49232026/11/11 23:59:11 INFO Vacuumed table table=pending_closures49242026/11/11 23:59:11 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049252026/11/11 23:59:11 INFO Vacuumed table table=pending_objects49262026/11/11 23:59:11 INFO Vacuumed table table=multipart_uploads49272026/11/11 23:59:11 INFO Vacuumed table table=closures49282026/11/11 23:59:11 INFO Vacuumed table table=pending_closures49292026/11/11 23:59:11 INFO Vacuumed table table=objects49302026/11/11 23:59:11 INFO Vacuumed table table=pending_objects49312026/11/11 23:59:11 INFO Vacuumed table table=multipart_uploads49322026/11/11 23:59:11 INFO Vacuumed table table=closures49332026/11/11 23:59:11 INFO Received push request method=POST path=/49342026/11/11 23:59:11 INFO Vacuumed table table=objects49352026/11/11 23:59:12 INFO Received complete upload request method=POST path=/49362026/11/11 23:59:12 INFO Completed upload id=33 objects=10 duration=982.916µs49372026/11/11 23:59:12 INFO Aborted multipart uploads count=0 kept=049382026/11/11 23:59:12 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."49392026/11/11 23:59:12 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."49402026/11/11 23:59:12 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."49412026/11/11 23:59:12 INFO Received push request method=POST path=/49422026/11/11 23:59:12 INFO Received push request method=POST path=/49432026/11/11 23:59:12 INFO Received complete push request method=POST path=/49442026/11/11 23:59:12 INFO Completed push id=28 objects=11 duration=523.167µs49452026/11/11 23:59:12 INFO Received complete upload request method=POST path=/49462026/11/11 23:59:12 INFO Completed upload id=34 objects=23 duration=349.875µs49472026/11/11 23:59:12 INFO Received complete upload request method=POST path=/49482026/11/11 23:59:12 INFO Completed upload id=34 objects=24 duration=289.375µs49492026/11/11 23:59:12 INFO Aborted multipart uploads count=0 kept=049502026/11/11 23:59:12 WARN Force mode enabled - objects will be deleted immediately without grace period49512026/11/11 23:59:12 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049522026/11/11 23:59:12 INFO Vacuumed table table=pending_closures49532026/11/11 23:59:12 INFO Vacuumed table table=pending_objects49542026/11/11 23:59:12 INFO Vacuumed table table=multipart_uploads49552026/11/11 23:59:12 INFO Vacuumed table table=closures49562026/11/11 23:59:12 INFO Vacuumed table table=objects49572026/11/11 23:59:12 INFO Received complete push request method=POST path=/49582026/11/11 23:59:12 INFO Completed push id=35 objects=6 duration=421.375µs49592026/11/11 23:59:12 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=049602026/11/11 23:59:12 INFO Vacuumed table table=pending_closures49612026/11/11 23:59:12 INFO Aborted multipart uploads count=0 kept=049622026/11/11 23:59:12 WARN Force mode enabled - objects will be deleted immediately without grace period49632026/11/11 23:59:12 INFO Received complete push request method=POST path=/49642026/11/11 23:59:12 INFO Vacuumed table table=pending_objects49652026/11/11 23:59:12 INFO Completed push id=29 objects=24 duration=443.208µs49662026/11/11 23:59:12 INFO Vacuumed table table=multipart_uploads49672026/11/11 23:59:12 INFO Vacuumed table table=closures49682026/11/11 23:59:12 INFO Vacuumed table table=objects49692026/11/11 23:59:12 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=049702026/11/11 23:59:12 INFO Vacuumed table table=pending_closures49712026/11/11 23:59:12 INFO Vacuumed table table=pending_objects49722026/11/11 23:59:12 INFO Vacuumed table table=multipart_uploads49732026/11/11 23:59:12 INFO Vacuumed table table=closures49742026/11/11 23:59:12 INFO Vacuumed table table=objects49752026/11/11 23:59:12 INFO Aborted multipart uploads count=0 kept=049762026/11/11 23:59:12 WARN Force mode enabled - objects will be deleted immediately without grace period49772026/11/11 23:59:12 INFO Received push request method=POST path=/49782026/11/11 23:59:12 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=049792026/11/11 23:59:12 INFO Vacuumed table table=pending_closures49802026/11/11 23:59:12 INFO Vacuumed table table=pending_objects49812026/11/11 23:59:12 INFO Vacuumed table table=multipart_uploads49822026/11/11 23:59:12 INFO Vacuumed table table=closures49832026/11/11 23:59:12 INFO Vacuumed table table=objects49842026/11/11 23:59:12 INFO Received uploads request method=POST path=/49852026/11/11 23:59:12 INFO Aborted multipart uploads count=0 kept=049862026/11/11 23:59:12 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049872026/11/11 23:59:12 INFO Vacuumed table table=pending_closures49882026/11/11 23:59:12 INFO Vacuumed table table=pending_objects49892026/11/11 23:59:12 INFO Vacuumed table table=multipart_uploads49902026/11/11 23:59:12 INFO Vacuumed table table=closures49912026/11/11 23:59:12 INFO Vacuumed table table=objects49922026/11/11 23:59:12 INFO Received complete upload request method=POST path=/49932026/11/11 23:59:12 INFO Completed upload id=13 objects=14 duration=2.537792ms49942026/11/11 23:59:12 INFO Aborted multipart uploads count=0 kept=049952026/11/11 23:59:12 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049962026/11/11 23:59:12 INFO Vacuumed table table=pending_closures49972026/11/11 23:59:12 INFO Vacuumed table table=pending_objects49982026/11/11 23:59:12 INFO Vacuumed table table=multipart_uploads49992026/11/11 23:59:12 INFO Vacuumed table table=closures50002026/11/11 23:59:12 INFO Vacuumed table table=objects50012026/11/11 23:59:12 INFO Received complete push request method=POST path=/50022026/11/11 23:59:12 INFO Completed push id=31 objects=6 duration=1.617042ms50032026/11/11 23:59:12 INFO Received uploads request method=POST path=/50042026/11/11 23:59:12 INFO Received complete push request method=POST path=/50052026/11/11 23:59:12 INFO Completed push id=1 objects=9 duration=1.601666ms50062026/11/11 23:59:12 INFO Received complete upload request method=POST path=/50072026/11/11 23:59:12 INFO Completed upload id=2 objects=7 duration=1.373333ms50082026/11/11 23:59:13 INFO Received complete upload request method=POST path=/50092026/11/11 23:59:13 INFO Completed upload id=8 objects=6 duration=1.365042ms50102026/11/11 23:59:13 INFO Received complete upload request method=POST path=/50112026/11/11 23:59:13 INFO Completed upload id=12 objects=9 duration=1.380375ms50122026/11/11 23:59:13 INFO Received complete upload request method=POST path=/50132026/11/11 23:59:13 INFO Completed upload id=18 objects=22 duration=1.42875ms50142026/11/11 23:59:13 INFO Received complete upload request method=POST path=/50152026/11/11 23:59:13 INFO Completed upload id=21 objects=8 duration=668.708µs50162026/11/11 23:59:13 INFO Received complete upload request method=POST path=/50172026/11/11 23:59:13 INFO Completed upload id=22 objects=7 duration=823.166µs50182026/11/11 23:59:13 INFO Received complete push request method=POST path=/50192026/11/11 23:59:13 INFO Completed push id=24 objects=5 duration=489.75µs50202026/11/11 23:59:13 INFO Received complete upload request method=POST path=/50212026/11/11 23:59:13 INFO Completed upload id=25 objects=8 duration=344.083µs50222026/11/11 23:59:13 INFO Received complete push request method=POST path=/50232026/11/11 23:59:13 INFO Completed push id=26 objects=4 duration=394.458µs50242026/11/11 23:59:13 INFO Received complete push request method=POST path=/50252026/11/11 23:59:13 INFO Completed push id=30 objects=2 duration=284.291µs50262026/11/11 23:59:13 INFO Received complete push request method=POST path=/50272026/11/11 23:59:13 INFO Completed push id=32 objects=7 duration=305.459µs50282026/11/11 23:59:13 INFO Received complete upload request method=POST path=/50292026/11/11 23:59:13 INFO Completed upload id=33 objects=6 duration=209.5µs50302026/11/11 23:59:13 INFO Received complete upload request method=POST path=/50312026/11/11 23:59:13 INFO Completed upload id=34 objects=3 duration=157.792µs50322026/11/11 23:59:13 INFO Aborted multipart uploads count=0 kept=050332026/11/11 23:59:13 WARN Force mode enabled - objects will be deleted immediately without grace period50342026/11/11 23:59:13 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=050352026/11/11 23:59:13 INFO Vacuumed table table=pending_closures50362026/11/11 23:59:13 INFO Vacuumed table table=pending_objects50372026/11/11 23:59:13 INFO Vacuumed table table=multipart_uploads50382026/11/11 23:59:13 INFO Vacuumed table table=closures50392026/11/11 23:59:13 INFO Vacuumed table table=objects5040--- PASS: FuzzPushGC (0.01s)5041 --- PASS: FuzzPushGC/seed#3 (6.41s)5042 --- PASS: FuzzPushGC/seed#6 (8.53s)5043 --- PASS: FuzzPushGC/seed#0 (8.58s)5044 --- PASS: FuzzPushGC/seed#14 (9.92s)5045 --- PASS: FuzzPushGC/seed#11 (10.02s)5046 --- PASS: FuzzPushGC/seed#8 (10.37s)5047 --- PASS: FuzzPushGC/seed#15 (11.08s)5048 --- PASS: FuzzPushGC/seed#13 (11.94s)5049 --- PASS: FuzzPushGC/seed#7 (12.27s)5050 --- PASS: FuzzPushGC/seed#4 (13.95s)5051 --- PASS: FuzzPushGC/seed#1 (14.23s)5052 --- PASS: FuzzPushGC/seed#5 (14.36s)5053 --- PASS: FuzzPushGC/seed#9 (14.58s)5054 --- PASS: FuzzPushGC/seed#12 (14.62s)5055 --- PASS: FuzzPushGC/seed#2 (14.62s)5056 --- PASS: FuzzPushGC/seed#10 (16.03s)5057PASS5058{"timestamp":"2026-11-11T23:59:13.689074Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:61412","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(11)"}5059{"timestamp":"2026-11-11T23:59:13.689337Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:61559","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(6)"}5060{"timestamp":"2026-11-11T23:59:13.689345Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:61338","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(2)"}50612026-11-11 23:59:13.831 UTC [99810] LOG: received smart shutdown request50622026-11-11 23:59:13.832 UTC [99810] LOG: background worker "logical replication launcher" (PID 99820) exited with exit code 150632026-11-11 23:59:14.783 UTC [99815] LOG: checkpoint complete: wrote 3466 buffers (21.2%), wrote 5 SLRU buffers; 0 WAL file(s) added, 0 removed, 33 recycled; write=31.055 s, sync=0.816 s, total=32.023 s; sync files=42366, longest=0.001 s, average=0.001 s; distance=533072 kB, estimate=533072 kB; lsn=0/27993320, redo lsn=0/22020F7850642026-11-11 23:59:14.785 UTC [99815] LOG: shutting down50652026-11-11 23:59:14.785 UTC [99815] LOG: checkpoint starting: shutdown immediate50662026-11-11 23:59:15.304 UTC [99815] LOG: checkpoint complete: wrote 9188 buffers (56.1%), wrote 2 SLRU buffers; 0 WAL file(s) added, 0 removed, 5 recycled; write=0.467 s, sync=0.050 s, total=0.520 s; sync files=3026, longest=0.001 s, average=0.001 s; distance=91593 kB, estimate=488924 kB; lsn=0/27993398, redo lsn=0/2799339850672026-11-11 23:59:15.311 UTC [99810] LOG: database system is shut down5068Running OIDC tests...5069=== RUN TestAudienceForIssuer5070=== PAUSE TestAudienceForIssuer5071=== RUN TestGlobMatch5072=== PAUSE TestGlobMatch5073=== RUN TestValidateToken_ValidToken5074=== PAUSE TestValidateToken_ValidToken5075=== RUN TestValidateToken_WrongAudience5076=== PAUSE TestValidateToken_WrongAudience5077=== RUN TestValidateToken_Expired5078=== PAUSE TestValidateToken_Expired5079=== RUN TestValidateToken_BoundClaimsMismatch5080=== PAUSE TestValidateToken_BoundClaimsMismatch5081=== RUN TestValidateToken_BoundSubjectMismatch5082=== PAUSE TestValidateToken_BoundSubjectMismatch5083=== RUN TestValidateToken_MultipleProviders5084=== PAUSE TestValidateToken_MultipleProviders5085=== RUN TestValidateToken_NoMatchingProvider5086=== PAUSE TestValidateToken_NoMatchingProvider5087=== RUN TestValidateToken_KubernetesServiceAccount5088=== PAUSE TestValidateToken_KubernetesServiceAccount5089=== RUN TestNewValidator_KubernetesRequiresCA5090=== PAUSE TestNewValidator_KubernetesRequiresCA5091=== RUN TestValidateToken_KubernetesIssuerFromOwnToken5092=== PAUSE TestValidateToken_KubernetesIssuerFromOwnToken5093=== RUN TestPins_ReservedForMatchingRule5094=== PAUSE TestPins_ReservedForMatchingRule5095=== RUN TestPins_TopLevelShorthand5096=== PAUSE TestPins_TopLevelShorthand5097=== RUN TestPins_ConfigValidation5098=== PAUSE TestPins_ConfigValidation5099=== RUN TestScopes_LegacyProviderDefaultsToWrite5100=== PAUSE TestScopes_LegacyProviderDefaultsToWrite5101=== RUN TestScopes_Rules5102=== PAUSE TestScopes_Rules5103=== RUN TestScopes_ConfigValidation5104=== PAUSE TestScopes_ConfigValidation5105=== CONT TestAudienceForIssuer5106=== CONT TestValidateToken_KubernetesServiceAccount5107=== CONT TestScopes_Rules5108--- PASS: TestAudienceForIssuer (0.00s)5109=== CONT TestValidateToken_BoundClaimsMismatch5110=== CONT TestValidateToken_Expired5111=== CONT TestValidateToken_WrongAudience5112=== CONT TestValidateToken_NoMatchingProvider5113=== CONT TestValidateToken_ValidToken5114=== CONT TestValidateToken_MultipleProviders5115=== CONT TestGlobMatch5116=== RUN TestGlobMatch/foo_foo5117=== PAUSE TestGlobMatch/foo_foo5118=== RUN TestGlobMatch/foo_bar5119=== PAUSE TestGlobMatch/foo_bar5120=== RUN TestGlobMatch/*_5121=== PAUSE TestGlobMatch/*_5122=== RUN TestGlobMatch/*_anything5123=== PAUSE TestGlobMatch/*_anything5124=== RUN TestGlobMatch/foo*_foo5125=== CONT TestValidateToken_BoundSubjectMismatch5126=== CONT TestPins_ConfigValidation5127=== CONT TestScopes_LegacyProviderDefaultsToWrite5128--- PASS: TestPins_ConfigValidation (0.00s)5129=== CONT TestScopes_ConfigValidation5130=== CONT TestPins_ReservedForMatchingRule5131--- PASS: TestScopes_ConfigValidation (0.00s)5132=== CONT TestPins_TopLevelShorthand5133=== CONT TestValidateToken_KubernetesIssuerFromOwnToken5134=== CONT TestNewValidator_KubernetesRequiresCA5135=== PAUSE TestGlobMatch/foo*_foo5136=== RUN TestGlobMatch/foo*_foobar5137=== PAUSE TestGlobMatch/foo*_foobar5138=== RUN TestGlobMatch/foo*_bar5139=== PAUSE TestGlobMatch/foo*_bar5140=== RUN TestGlobMatch/*bar_bar5141=== PAUSE TestGlobMatch/*bar_bar5142=== RUN TestGlobMatch/*bar_foobar5143=== PAUSE TestGlobMatch/*bar_foobar5144=== RUN TestGlobMatch/*bar_foo5145=== PAUSE TestGlobMatch/*bar_foo5146=== RUN TestGlobMatch/foo*bar_foobar5147=== PAUSE TestGlobMatch/foo*bar_foobar5148=== RUN TestGlobMatch/foo*bar_foo123bar5149=== PAUSE TestGlobMatch/foo*bar_foo123bar5150=== RUN TestGlobMatch/foo*bar_foobarbaz5151=== PAUSE TestGlobMatch/foo*bar_foobarbaz5152=== RUN TestGlobMatch/*/*_foo/bar5153=== PAUSE TestGlobMatch/*/*_foo/bar5154=== RUN TestGlobMatch/*/*_foo5155=== PAUSE TestGlobMatch/*/*_foo5156=== RUN TestGlobMatch/refs/heads/*_refs/heads/main5157=== PAUSE TestGlobMatch/refs/heads/*_refs/heads/main5158=== RUN TestGlobMatch/refs/heads/*_refs/tags/v1.05159=== PAUSE TestGlobMatch/refs/heads/*_refs/tags/v1.05160=== RUN TestGlobMatch/refs/*/main_refs/heads/main5161=== PAUSE TestGlobMatch/refs/*/main_refs/heads/main5162=== RUN TestGlobMatch/fo?_foo5163=== PAUSE TestGlobMatch/fo?_foo5164=== RUN TestGlobMatch/fo?_fo5165=== PAUSE TestGlobMatch/fo?_fo5166=== RUN TestGlobMatch/fo?_fooo5167=== PAUSE TestGlobMatch/fo?_fooo5168=== RUN TestGlobMatch/?oo_foo5169=== PAUSE TestGlobMatch/?oo_foo5170=== RUN TestGlobMatch/?oo_boo5171=== PAUSE TestGlobMatch/?oo_boo5172=== RUN TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5173=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5174=== RUN TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5175=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5176=== CONT TestGlobMatch/foo_foo5177=== CONT TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5178=== CONT TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5179=== CONT TestGlobMatch/?oo_boo5180=== CONT TestGlobMatch/?oo_foo5181=== CONT TestGlobMatch/fo?_fooo5182=== CONT TestGlobMatch/fo?_fo5183=== CONT TestGlobMatch/fo?_foo5184=== CONT TestGlobMatch/refs/*/main_refs/heads/main5185=== CONT TestGlobMatch/refs/heads/*_refs/tags/v1.05186=== CONT TestGlobMatch/refs/heads/*_refs/heads/main5187=== CONT TestGlobMatch/*/*_foo5188=== CONT TestGlobMatch/*/*_foo/bar5189=== CONT TestGlobMatch/foo*bar_foobarbaz5190=== CONT TestGlobMatch/foo*bar_foo123bar5191=== CONT TestGlobMatch/foo*bar_foobar5192=== CONT TestGlobMatch/*bar_foo5193=== CONT TestGlobMatch/*bar_foobar5194=== CONT TestGlobMatch/*bar_bar5195=== CONT TestGlobMatch/foo*_bar5196=== CONT TestGlobMatch/foo*_foobar5197=== CONT TestGlobMatch/foo*_foo5198=== CONT TestGlobMatch/*_anything5199=== CONT TestGlobMatch/*_5200=== CONT TestGlobMatch/foo_bar5201--- PASS: TestGlobMatch (0.01s)5202 --- PASS: TestGlobMatch/foo_foo (0.00s)5203 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main (0.00s)5204 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main (0.00s)5205 --- PASS: TestGlobMatch/?oo_boo (0.00s)5206 --- PASS: TestGlobMatch/?oo_foo (0.00s)5207 --- PASS: TestGlobMatch/fo?_fooo (0.00s)5208 --- PASS: TestGlobMatch/fo?_fo (0.00s)5209 --- PASS: TestGlobMatch/fo?_foo (0.00s)5210 --- PASS: TestGlobMatch/refs/*/main_refs/heads/main (0.00s)5211 --- PASS: TestGlobMatch/refs/heads/*_refs/tags/v1.0 (0.00s)5212 --- PASS: TestGlobMatch/refs/heads/*_refs/heads/main (0.00s)5213 --- PASS: TestGlobMatch/*/*_foo (0.00s)5214 --- PASS: TestGlobMatch/*/*_foo/bar (0.00s)5215 --- PASS: TestGlobMatch/foo*bar_foobarbaz (0.00s)5216 --- PASS: TestGlobMatch/foo*bar_foo123bar (0.00s)5217 --- PASS: TestGlobMatch/foo*bar_foobar (0.00s)5218 --- PASS: TestGlobMatch/*bar_foo (0.00s)5219 --- PASS: TestGlobMatch/*bar_foobar (0.00s)5220 --- PASS: TestGlobMatch/*bar_bar (0.00s)5221 --- PASS: TestGlobMatch/foo*_bar (0.00s)5222 --- PASS: TestGlobMatch/foo*_foobar (0.00s)5223 --- PASS: TestGlobMatch/foo*_foo (0.00s)5224 --- PASS: TestGlobMatch/*_anything (0.00s)5225 --- PASS: TestGlobMatch/*_ (0.00s)5226 --- PASS: TestGlobMatch/foo_bar (0.00s)52272026/11/11 23:59:17 INFO OIDC provider initialized name=kubernetes issuer=https://127.0.0.1:6198452282026/11/11 23:59:17 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:61986/oidc52292026/11/11 23:59:17 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:61988/oidc5230--- PASS: TestValidateToken_KubernetesServiceAccount (0.07s)5231--- PASS: TestScopes_LegacyProviderDefaultsToWrite (0.06s)5232--- PASS: TestValidateToken_WrongAudience (0.07s)52332026/11/11 23:59:17 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:61991/oidc5234--- PASS: TestPins_TopLevelShorthand (0.07s)52352026/11/11 23:59:17 http: TLS handshake error from 127.0.0.1:61994: read tcp 127.0.0.1:61993->127.0.0.1:61994: use of closed network connection5236--- PASS: TestNewValidator_KubernetesRequiresCA (0.10s)52372026/11/11 23:59:17 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:61995/oidc5238--- PASS: TestValidateToken_BoundSubjectMismatch (0.11s)52392026/11/11 23:59:17 INFO OIDC provider initialized name=kubernetes issuer=https://oidc.eks.invalid/id/ABC12352402026/11/11 23:59:17 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:61998/oidc52412026/11/11 23:59:17 INFO OIDC provider initialized name=provider1 issuer=http://127.0.0.1:61990/oidc52422026/11/11 23:59:17 INFO OIDC provider initialized name=provider2 issuer=http://127.0.0.1:62001/oidc5243--- PASS: TestValidateToken_KubernetesIssuerFromOwnToken (0.11s)5244--- PASS: TestValidateToken_MultipleProviders (0.13s)52452026/11/11 23:59:17 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:62006/oidc5246--- PASS: TestValidateToken_Expired (0.13s)5247--- PASS: TestScopes_Rules (0.13s)52482026/11/11 23:59:17 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:62009/oidc5249--- PASS: TestValidateToken_BoundClaimsMismatch (0.16s)52502026/11/11 23:59:17 INFO OIDC provider initialized name=provider1 issuer=http://127.0.0.1:62008/oidc5251--- PASS: TestValidateToken_NoMatchingProvider (0.17s)52522026/11/11 23:59:17 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:62013/oidc5253--- PASS: TestValidateToken_ValidToken (0.19s)52542026/11/11 23:59:17 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:62015/oidc5255--- PASS: TestPins_ReservedForMatchingRule (0.19s)5256PASS5257Running hook tests...5258=== RUN TestSendPathsEmpty5259=== PAUSE TestSendPathsEmpty5260=== RUN TestQueueEnqueueAndFetch5261=== PAUSE TestQueueEnqueueAndFetch5262=== RUN TestQueueDeduplication5263=== PAUSE TestQueueDeduplication5264=== RUN TestQueueRemove5265=== PAUSE TestQueueRemove5266=== RUN TestQueueFetchBatchLimit5267=== PAUSE TestQueueFetchBatchLimit5268=== RUN TestQueueRetryMovesToBack5269=== PAUSE TestQueueRetryMovesToBack5270=== RUN TestQueueFetchRemoveLifecycle5271=== PAUSE TestQueueFetchRemoveLifecycle5272=== RUN TestQueueConcurrentWriters5273=== PAUSE TestQueueConcurrentWriters5274=== RUN TestQueueRemoveLargeClosure5275=== PAUSE TestQueueRemoveLargeClosure5276=== RUN TestServerClientIntegration5277--- PASS: TestServerClientIntegration (0.00s)5278=== RUN TestServerQueueError52792026/11/11 23:59:17 ERROR Failed to queue paths error="permission denied" count=15280--- PASS: TestServerQueueError (0.00s)5281=== RUN TestGetListenerSocketActivation5282 server_test.go:147: systemd socket activation does not exist on darwin5283--- SKIP: TestGetListenerSocketActivation (0.00s)5284=== RUN TestServerStalledClientDoesNotBlockShutdown5285=== PAUSE TestServerStalledClientDoesNotBlockShutdown5286=== RUN TestServerBacksOffOnAcceptErrors5287=== PAUSE TestServerBacksOffOnAcceptErrors5288=== RUN TestDrainIsolatesPoisonPath5289=== PAUSE TestDrainIsolatesPoisonPath5290=== RUN TestRunNotBlockedByPoisonHead5291=== PAUSE TestRunNotBlockedByPoisonHead5292=== RUN TestDrainGivesUpWhenServerDown5293=== PAUSE TestDrainGivesUpWhenServerDown5294=== RUN TestFailedPathPrunedByLaterClosure5295=== PAUSE TestFailedPathPrunedByLaterClosure5296=== RUN TestWorkerUploadsAndRemoves5297=== PAUSE TestWorkerUploadsAndRemoves5298=== RUN TestWorkerSkipsGCdPaths5299=== PAUSE TestWorkerSkipsGCdPaths5300=== RUN TestWorkerPrunesClosureDeps5301=== PAUSE TestWorkerPrunesClosureDeps5302=== RUN TestDrainTimeout5303=== PAUSE TestDrainTimeout5304=== RUN TestWorkerRemoveFailureIsNotProgress5305=== PAUSE TestWorkerRemoveFailureIsNotProgress5306=== RUN TestWorkerKeepsPathItCannotStat5307=== PAUSE TestWorkerKeepsPathItCannotStat5308=== CONT TestSendPathsEmpty5309=== CONT TestDrainIsolatesPoisonPath5310=== CONT TestWorkerSkipsGCdPaths5311=== CONT TestWorkerRemoveFailureIsNotProgress5312=== CONT TestQueueRemove5313=== CONT TestDrainTimeout5314=== CONT TestServerBacksOffOnAcceptErrors53152000/01/01 00:00:00 ERROR Accept failed error="too many open files"53162000/01/01 00:00:00 ERROR Accept failed error="too many open files"53172000/01/01 00:00:00 ERROR Accept failed error="too many open files"53182000/01/01 00:00:00 ERROR Accept failed error="too many open files"53192000/01/01 00:00:00 ERROR Accept failed error="too many open files"53202000/01/01 00:00:00 ERROR Accept failed error="too many open files"53212000/01/01 00:00:00 ERROR Accept failed error="too many open files"53222000/01/01 00:00:00 ERROR Accept failed error="too many open files"5323=== CONT TestServerStalledClientDoesNotBlockShutdown5324=== CONT TestQueueRemoveLargeClosure5325=== CONT TestQueueConcurrentWriters5326=== CONT TestQueueFetchRemoveLifecycle5327=== CONT TestQueueDeduplication5328=== CONT TestQueueEnqueueAndFetch5329=== CONT TestQueueRetryMovesToBack5330=== CONT TestQueueFetchBatchLimit5331=== CONT TestFailedPathPrunedByLaterClosure5332=== CONT TestWorkerUploadsAndRemoves5333=== CONT TestDrainGivesUpWhenServerDown5334=== CONT TestWorkerKeepsPathItCannotStat5335=== CONT TestWorkerPrunesClosureDeps5336=== CONT TestRunNotBlockedByPoisonHead5337--- PASS: TestSendPathsEmpty (0.00s)5338--- PASS: TestServerBacksOffOnAcceptErrors (0.00s)53392026/11/11 23:59:17 WARN Cannot stat store path, will retry later path=/private/var/tb/5/scratch/67d206d5acc7e9e5/build/TestWorkerKeepsPathItCannotStat1828381411/002/locked/aaa error="stat /private/var/tb/5/scratch/67d206d5acc7e9e5/build/TestWorkerKeepsPathItCannotStat1828381411/002/locked/aaa: permission denied"53402026/11/11 23:59:17 INFO Upload queue status pending=253412026/11/11 23:59:17 INFO Uploading batch count=15342--- PASS: TestQueueDeduplication (0.01s)53432026/11/11 23:59:17 INFO Uploading batch count=153442026/11/11 23:59:17 ERROR Upload failed error="upload failed" count=153452026/11/11 23:59:17 WARN Cannot stat store path, will retry later path=/private/var/tb/5/scratch/67d206d5acc7e9e5/build/TestWorkerKeepsPathItCannotStat1828381411/002/locked/aaa error="stat /private/var/tb/5/scratch/67d206d5acc7e9e5/build/TestWorkerKeepsPathItCannotStat1828381411/002/locked/aaa: permission denied"53462026/11/11 23:59:17 INFO Upload queue status pending=253472026/11/11 23:59:17 INFO Uploading batch count=253482026/11/11 23:59:17 WARN Cannot stat store path, will retry later path=/private/var/tb/5/scratch/67d206d5acc7e9e5/build/TestWorkerKeepsPathItCannotStat1828381411/002/locked/aaa error="stat /private/var/tb/5/scratch/67d206d5acc7e9e5/build/TestWorkerKeepsPathItCannotStat1828381411/002/locked/aaa: permission denied"53492026/11/11 23:59:17 INFO Upload queue status pending=253502026/11/11 23:59:17 WARN Store path no longer exists (garbage collected?), removing from queue path=/private/var/tb/5/scratch/67d206d5acc7e9e5/build/TestWorkerSkipsGCdPaths2745057308/002/nonexistent5351--- PASS: TestQueueFetchBatchLimit (0.01s)53522026/11/11 23:59:17 ERROR Drain finished with paths left in queue remaining=153532026/11/11 23:59:17 INFO Uploading batch count=15354--- PASS: TestQueueRetryMovesToBack (0.01s)53552026/11/11 23:59:17 INFO Uploading batch count=253562026/11/11 23:59:17 ERROR Upload failed error="upload failed" count=253572026/11/11 23:59:17 ERROR Upload failed, will retry later error="upload failed" path=/private/var/tb/5/scratch/67d206d5acc7e9e5/build/TestDrainGivesUpWhenServerDown3323693909/002/a53582026/11/11 23:59:17 INFO Upload queue status pending=353592026/11/11 23:59:17 INFO Uploading batch count=153602026/11/11 23:59:17 INFO Uploading batch count=153612026/11/11 23:59:17 ERROR Upload failed error="upload failed" count=153622026/11/11 23:59:17 INFO Uploading batch count=153632026/11/11 23:59:17 INFO Uploading batch count=453642026/11/11 23:59:17 ERROR Upload failed error="upload failed" count=45365--- PASS: TestQueueFetchRemoveLifecycle (0.01s)53662000/01/01 00:00:00 INFO Uploading batch count=253672000/01/01 00:00:00 ERROR Upload failed error="context deadline exceeded" count=253682000/01/01 00:00:00 ERROR Drain finished with paths left in queue remaining=453692026/11/11 23:59:17 ERROR Upload failed, will retry later error="upload failed" path=/private/var/tb/5/scratch/67d206d5acc7e9e5/build/TestDrainGivesUpWhenServerDown3323693909/002/b53702026/11/11 23:59:17 ERROR Upload failed, will retry later error="upload failed" path=/private/var/tb/5/scratch/67d206d5acc7e9e5/build/TestDrainIsolatesPoisonPath1159102614/002/bbb5371--- PASS: TestQueueRemove (0.02s)5372--- PASS: TestWorkerKeepsPathItCannotStat (0.01s)53732026/11/11 23:59:17 INFO Uploading batch count=253742026/11/11 23:59:17 ERROR Upload failed error="upload failed" count=253752026/11/11 23:59:17 ERROR Upload failed, will retry later error="upload failed" path=/private/var/tb/5/scratch/67d206d5acc7e9e5/build/TestDrainGivesUpWhenServerDown3323693909/002/c53762026/11/11 23:59:17 ERROR Upload failed, will retry later error="upload failed" path=/private/var/tb/5/scratch/67d206d5acc7e9e5/build/TestDrainGivesUpWhenServerDown3323693909/002/d5377--- PASS: TestFailedPathPrunedByLaterClosure (0.02s)53782026/11/11 23:59:17 WARN Store path no longer exists (garbage collected?), removing from queue path=/private/var/tb/5/scratch/67d206d5acc7e9e5/build/TestWorkerRemoveFailureIsNotProgress413627751/002/nonexistent53792026/11/11 23:59:17 INFO Uploading batch count=253802026/11/11 23:59:17 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=153812026/11/11 23:59:17 WARN Store path no longer exists (garbage collected?), removing from queue path=/private/var/tb/5/scratch/67d206d5acc7e9e5/build/TestWorkerRemoveFailureIsNotProgress413627751/002/nonexistent53822026/11/11 23:59:17 ERROR Upload failed error="upload failed" count=253832026/11/11 23:59:17 ERROR Upload failed, will retry later error="upload failed" path=/private/var/tb/5/scratch/67d206d5acc7e9e5/build/TestDrainGivesUpWhenServerDown3323693909/002/e53842026/11/11 23:59:17 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=15385--- PASS: TestQueueEnqueueAndFetch (0.02s)53862026/11/11 23:59:17 WARN Store path no longer exists (garbage collected?), removing from queue path=/private/var/tb/5/scratch/67d206d5acc7e9e5/build/TestWorkerRemoveFailureIsNotProgress413627751/002/nonexistent53872026/11/11 23:59:17 INFO Uploading batch count=153882026/11/11 23:59:17 ERROR Upload failed error="upload failed" count=153892026/11/11 23:59:17 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=153902026/11/11 23:59:17 ERROR Upload failed, will retry later error="upload failed" path=/private/var/tb/5/scratch/67d206d5acc7e9e5/build/TestDrainGivesUpWhenServerDown3323693909/002/f53912026/11/11 23:59:17 ERROR Drain finished with paths left in queue remaining=153922026/11/11 23:59:17 INFO Uploading batch count=153932026/11/11 23:59:17 ERROR Upload failed error="upload failed" count=15394--- PASS: TestDrainTimeout (0.02s)53952026/11/11 23:59:17 ERROR Drain finished with paths left in queue remaining=1053962026/11/11 23:59:17 INFO Uploading batch count=153972026/11/11 23:59:17 ERROR Upload failed error="upload failed" count=153982026/11/11 23:59:17 ERROR Drain finished with paths left in queue remaining=15399--- PASS: TestWorkerRemoveFailureIsNotProgress (0.02s)5400--- PASS: TestDrainGivesUpWhenServerDown (0.02s)5401--- PASS: TestDrainIsolatesPoisonPath (0.02s)5402--- PASS: TestWorkerPrunesClosureDeps (0.03s)5403--- PASS: TestWorkerUploadsAndRemoves (0.03s)5404--- PASS: TestWorkerSkipsGCdPaths (0.03s)5405--- PASS: TestQueueRemoveLargeClosure (0.05s)5406--- PASS: TestQueueConcurrentWriters (0.06s)54072026/11/11 23:59:17 ERROR Failed to decode request error="read unix /private/var/tb/5/scratch/67d206d5acc7e9e5/build/hook2713211356/test.sock->: i/o timeout"54082026/11/11 23:59:17 ERROR Failed to write response error="write unix /private/var/tb/5/scratch/67d206d5acc7e9e5/build/hook2713211356/test.sock->: i/o timeout"5409--- PASS: TestServerStalledClientDoesNotBlockShutdown (0.20s)54102026/11/11 23:59:18 INFO Uploading batch count=154112026/11/11 23:59:18 INFO Uploading batch count=154122026/11/11 23:59:18 INFO Uploading batch count=154132026/11/11 23:59:18 ERROR Upload failed error="upload failed" count=154142026/11/11 23:59:18 INFO Uploading batch count=154152026/11/11 23:59:18 ERROR Upload failed error="upload failed" count=154162026/11/11 23:59:18 INFO Uploading batch count=154172026/11/11 23:59:18 ERROR Upload failed error="upload failed" count=154182026/11/11 23:59:18 INFO Uploading batch count=154192026/11/11 23:59:18 ERROR Upload failed error="upload failed" count=154202026/11/11 23:59:18 ERROR Drain finished with paths left in queue remaining=15421--- PASS: TestRunNotBlockedByPoisonHead (1.04s)5422PASS5423Running signing tests...5424=== RUN TestExternalSigner5425=== PAUSE TestExternalSigner5426=== RUN TestExternalSignerWrongPublicKey5427=== PAUSE TestExternalSignerWrongPublicKey5428=== RUN TestExternalSignerErrors5429=== PAUSE TestExternalSignerErrors5430=== RUN TestExternalSignerRestartsAfterCrash5431=== PAUSE TestExternalSignerRestartsAfterCrash5432=== RUN TestExternalSignerProgramErrorKeepsProcess5433=== PAUSE TestExternalSignerProgramErrorKeepsProcess5434=== RUN TestExternalSignerClose5435=== PAUSE TestExternalSignerClose5436=== RUN TestExternalSignerRejectsInvalidPublicKey5437=== PAUSE TestExternalSignerRejectsInvalidPublicKey5438=== RUN TestExternalSignerCancellation5439=== PAUSE TestExternalSignerCancellation5440=== RUN TestGenerateFingerprint5441=== PAUSE TestGenerateFingerprint5442=== RUN TestParseSigningKey5443=== PAUSE TestParseSigningKey5444=== RUN TestSign5445=== PAUSE TestSign5446=== RUN TestSignBatch5447=== PAUSE TestSignBatch5448=== RUN TestNilKeyPublicKey5449=== PAUSE TestNilKeyPublicKey5450=== RUN TestSignErrors5451=== PAUSE TestSignErrors5452=== CONT TestExternalSigner5453=== CONT TestSignBatch5454=== CONT TestExternalSignerCancellation5455=== CONT TestSignErrors5456=== CONT TestNilKeyPublicKey5457--- PASS: TestNilKeyPublicKey (0.00s)5458=== CONT TestExternalSignerRejectsInvalidPublicKey5459=== RUN TestExternalSignerRejectsInvalidPublicKey/#005460=== PAUSE TestExternalSignerRejectsInvalidPublicKey/#005461=== RUN TestExternalSignerRejectsInvalidPublicKey/no-colon5462=== CONT TestExternalSignerProgramErrorKeepsProcess5463=== CONT TestExternalSignerErrors5464=== RUN TestExternalSignerErrors/program_error5465=== PAUSE TestExternalSignerErrors/program_error5466=== RUN TestExternalSignerErrors/missing_signatures5467=== PAUSE TestExternalSignerErrors/missing_signatures5468=== RUN TestExternalSignerErrors/too_few_signatures5469=== PAUSE TestExternalSignerErrors/too_few_signatures5470=== RUN TestExternalSignerErrors/wrong_signature_length5471=== PAUSE TestExternalSignerErrors/wrong_signature_length5472=== RUN TestExternalSignerErrors/invalid_JSON5473=== PAUSE TestExternalSignerErrors/invalid_JSON5474=== RUN TestExternalSignerErrors/wrong_id5475=== PAUSE TestExternalSignerRejectsInvalidPublicKey/no-colon5476=== RUN TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5477=== PAUSE TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5478=== RUN TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5479=== PAUSE TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5480=== RUN TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5481=== PAUSE TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5482=== CONT TestExternalSignerRejectsInvalidPublicKey/#005483=== CONT TestExternalSignerRestartsAfterCrash5484=== CONT TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5485=== CONT TestExternalSignerRejectsInvalidPublicKey/no-colon5486=== CONT TestExternalSignerWrongPublicKey5487=== CONT TestExternalSignerClose5488=== CONT TestParseSigningKey5489=== RUN TestParseSigningKey/valid_32-byte_key5490=== CONT TestSign5491=== CONT TestGenerateFingerprint5492=== RUN TestGenerateFingerprint/basic_with_references5493=== PAUSE TestExternalSignerErrors/wrong_id5494=== PAUSE TestGenerateFingerprint/basic_with_references5495=== RUN TestExternalSignerErrors/exit_on_request5496=== CONT TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5497=== CONT TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5498=== PAUSE TestParseSigningKey/valid_32-byte_key5499=== RUN TestParseSigningKey/valid_32-byte_key_with_different_name5500=== PAUSE TestExternalSignerErrors/exit_on_request5501=== RUN TestGenerateFingerprint/no_references5502=== RUN TestExternalSignerErrors/null_signatures5503=== PAUSE TestParseSigningKey/valid_32-byte_key_with_different_name5504=== RUN TestParseSigningKey/no_colon5505=== PAUSE TestParseSigningKey/no_colon5506=== RUN TestParseSigningKey/empty_name5507=== PAUSE TestParseSigningKey/empty_name5508=== RUN TestParseSigningKey/invalid_base645509=== PAUSE TestParseSigningKey/invalid_base645510=== RUN TestParseSigningKey/wrong_length5511=== PAUSE TestParseSigningKey/wrong_length5512=== CONT TestParseSigningKey/valid_32-byte_key5513=== PAUSE TestGenerateFingerprint/no_references5514=== RUN TestGenerateFingerprint/unsorted_references_get_sorted5515=== PAUSE TestGenerateFingerprint/unsorted_references_get_sorted5516=== RUN TestGenerateFingerprint/invalid_nar_hash_prefix5517=== PAUSE TestGenerateFingerprint/invalid_nar_hash_prefix5518=== RUN TestGenerateFingerprint/invalid_nar_hash_length5519=== PAUSE TestGenerateFingerprint/invalid_nar_hash_length5520=== RUN TestGenerateFingerprint/invalid_store_path_prefix5521=== PAUSE TestGenerateFingerprint/invalid_store_path_prefix5522=== RUN TestGenerateFingerprint/invalid_reference_prefix5523=== PAUSE TestGenerateFingerprint/invalid_reference_prefix5524=== CONT TestGenerateFingerprint/basic_with_references5525=== CONT TestParseSigningKey/no_colon5526=== CONT TestParseSigningKey/valid_32-byte_key_with_different_name5527=== CONT TestParseSigningKey/wrong_length5528=== CONT TestGenerateFingerprint/invalid_nar_hash_length5529--- PASS: TestExternalSignerRejectsInvalidPublicKey (0.00s)5530 --- PASS: TestExternalSignerRejectsInvalidPublicKey/#00 (0.00s)5531 --- PASS: TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= (0.00s)5532 --- PASS: TestExternalSignerRejectsInvalidPublicKey/no-colon (0.00s)5533 --- PASS: TestExternalSignerRejectsInvalidPublicKey/test-key:YQ== (0.00s)5534 --- PASS: TestExternalSignerRejectsInvalidPublicKey/test-key:invalid! (0.00s)5535=== CONT TestGenerateFingerprint/invalid_store_path_prefix5536=== PAUSE TestExternalSignerErrors/null_signatures5537=== CONT TestGenerateFingerprint/invalid_nar_hash_prefix5538=== CONT TestParseSigningKey/invalid_base645539=== CONT TestGenerateFingerprint/no_references5540=== CONT TestGenerateFingerprint/invalid_reference_prefix5541=== CONT TestParseSigningKey/empty_name5542=== RUN TestExternalSignerErrors/wrong_signatures_shape5543=== CONT TestGenerateFingerprint/unsorted_references_get_sorted5544=== PAUSE TestExternalSignerErrors/wrong_signatures_shape5545--- PASS: TestGenerateFingerprint (0.00s)5546 --- PASS: TestGenerateFingerprint/basic_with_references (0.00s)5547 --- PASS: TestGenerateFingerprint/invalid_nar_hash_length (0.00s)5548 --- PASS: TestGenerateFingerprint/invalid_store_path_prefix (0.00s)5549 --- PASS: TestGenerateFingerprint/invalid_nar_hash_prefix (0.00s)5550 --- PASS: TestGenerateFingerprint/no_references (0.00s)5551 --- PASS: TestGenerateFingerprint/invalid_reference_prefix (0.00s)5552 --- PASS: TestGenerateFingerprint/unsorted_references_get_sorted (0.00s)5553=== RUN TestExternalSignerErrors/missing_signature_name5554=== PAUSE TestExternalSignerErrors/missing_signature_name5555=== RUN TestExternalSignerErrors/wrong_signature_name5556=== PAUSE TestExternalSignerErrors/wrong_signature_name5557=== RUN TestExternalSignerErrors/invalid_signature_base645558=== PAUSE TestExternalSignerErrors/invalid_signature_base645559=== RUN TestExternalSignerErrors/exit_at_startup5560=== PAUSE TestExternalSignerErrors/exit_at_startup5561=== RUN TestExternalSignerErrors/wrong_shape5562=== PAUSE TestExternalSignerErrors/wrong_shape5563=== RUN TestExternalSignerErrors/null5564=== PAUSE TestExternalSignerErrors/null5565=== RUN TestExternalSignerErrors/unrequested_key5566=== PAUSE TestExternalSignerErrors/unrequested_key5567=== CONT TestExternalSignerErrors/program_error5568--- PASS: TestSign (0.00s)5569=== CONT TestExternalSignerErrors/wrong_shape5570=== CONT TestExternalSignerErrors/exit_at_startup5571--- PASS: TestParseSigningKey (0.00s)5572 --- PASS: TestParseSigningKey/no_colon (0.00s)5573 --- PASS: TestParseSigningKey/wrong_length (0.00s)5574 --- PASS: TestParseSigningKey/invalid_base64 (0.00s)5575 --- PASS: TestParseSigningKey/empty_name (0.00s)5576 --- PASS: TestParseSigningKey/valid_32-byte_key (0.00s)5577 --- PASS: TestParseSigningKey/valid_32-byte_key_with_different_name (0.00s)5578=== CONT TestExternalSignerErrors/invalid_signature_base645579=== CONT TestExternalSignerErrors/unrequested_key5580=== CONT TestExternalSignerErrors/missing_signature_name5581=== CONT TestExternalSignerErrors/wrong_signature_name5582--- PASS: TestSignBatch (0.00s)5583=== CONT TestExternalSignerErrors/null5584=== CONT TestExternalSignerErrors/wrong_id5585=== RUN TestSignErrors/nil_key5586=== CONT TestExternalSignerErrors/exit_on_request5587=== CONT TestExternalSignerErrors/null_signatures5588=== CONT TestExternalSignerErrors/invalid_JSON5589=== CONT TestExternalSignerErrors/too_few_signatures5590=== CONT TestExternalSignerErrors/wrong_signature_length5591=== CONT TestExternalSignerErrors/missing_signatures5592=== CONT TestExternalSignerErrors/wrong_signatures_shape5593=== PAUSE TestSignErrors/nil_key5594=== RUN TestSignErrors/nil_narinfo5595=== PAUSE TestSignErrors/nil_narinfo5596=== RUN TestSignErrors/invalid_narinfo5597=== PAUSE TestSignErrors/invalid_narinfo5598=== RUN TestSignErrors/canceled5599=== PAUSE TestSignErrors/canceled5600=== CONT TestSignErrors/nil_key5601=== CONT TestSignErrors/canceled5602=== CONT TestSignErrors/invalid_narinfo5603=== CONT TestSignErrors/nil_narinfo5604--- PASS: TestSignErrors (0.01s)5605 --- PASS: TestSignErrors/nil_key (0.00s)5606 --- PASS: TestSignErrors/canceled (0.00s)5607 --- PASS: TestSignErrors/invalid_narinfo (0.00s)5608 --- PASS: TestSignErrors/nil_narinfo (0.00s)5609--- PASS: TestExternalSigner (0.11s)5610--- PASS: TestExternalSignerCancellation (0.19s)5611--- PASS: TestExternalSignerProgramErrorKeepsProcess (0.24s)5612--- PASS: TestExternalSignerRestartsAfterCrash (0.41s)5613/private/var/tb/5/scratch/67d206d5acc7e9e5/build/TestExternalSignerRestartsAfterCrash258177842/001/sign program: line 7: /private/var/tb/5/scratch/67d206d5acc7e9e5/build/TestExternalSignerRestartsAfterCrash258177842/001/sign program.starts: No such file or directory5614--- PASS: TestExternalSignerClose (1.18s)5615--- PASS: TestExternalSignerWrongPublicKey (1.26s)5616--- PASS: TestExternalSignerErrors (0.00s)5617 --- PASS: TestExternalSignerErrors/missing_signatures (0.31s)5618 --- PASS: TestExternalSignerErrors/too_few_signatures (0.46s)5619 --- PASS: TestExternalSignerErrors/null (0.54s)5620 --- PASS: TestExternalSignerErrors/wrong_signatures_shape (0.61s)5621 --- PASS: TestExternalSignerErrors/invalid_JSON (0.69s)5622 --- PASS: TestExternalSignerErrors/exit_at_startup (0.89s)5623 --- PASS: TestExternalSignerErrors/wrong_signature_name (0.96s)5624 --- PASS: TestExternalSignerErrors/missing_signature_name (1.03s)5625 --- PASS: TestExternalSignerErrors/null_signatures (1.11s)5626 --- PASS: TestExternalSignerErrors/program_error (1.33s)5627 --- PASS: TestExternalSignerErrors/wrong_id (1.40s)5628 --- PASS: TestExternalSignerErrors/wrong_shape (1.48s)5629 --- PASS: TestExternalSignerErrors/exit_on_request (1.55s)5630 --- PASS: TestExternalSignerErrors/wrong_signature_length (1.67s)5631 --- PASS: TestExternalSignerErrors/invalid_signature_base64 (1.71s)5632 --- PASS: TestExternalSignerErrors/unrequested_key (1.79s)5633PASS5634Running OpenBao signer tests...5635=== RUN TestSign5636=== PAUSE TestSign5637=== RUN TestSignErrors5638=== PAUSE TestSignErrors5639=== RUN TestServe5640=== PAUSE TestServe5641=== RUN TestServeRejectsGarbage5642=== PAUSE TestServeRejectsGarbage5643=== CONT TestSign5644=== CONT TestSignErrors5645=== RUN TestSignErrors/http_error5646=== PAUSE TestSignErrors/http_error5647=== RUN TestSignErrors/invalid_json5648=== PAUSE TestSignErrors/invalid_json5649=== RUN TestSignErrors/no_prefix5650=== PAUSE TestSignErrors/no_prefix5651=== RUN TestSignErrors/invalid_base645652=== PAUSE TestSignErrors/invalid_base645653=== CONT TestSignErrors/http_error5654=== CONT TestSignErrors/no_prefix5655=== CONT TestSignErrors/invalid_json5656=== CONT TestServeRejectsGarbage5657--- PASS: TestServeRejectsGarbage (0.00s)5658=== CONT TestSignErrors/invalid_base645659=== CONT TestServe5660--- PASS: TestServe (0.00s)5661--- PASS: TestSign (0.00s)5662--- PASS: TestSignErrors (0.00s)5663 --- PASS: TestSignErrors/http_error (0.00s)5664 --- PASS: TestSignErrors/invalid_base64 (0.00s)5665 --- PASS: TestSignErrors/invalid_json (0.00s)5666 --- PASS: TestSignErrors/no_prefix (0.00s)5667PASS