niks3-go-unit-tests
checks.aarch64-darwin.go-unit-tests
· build #510
· raw
1tribuchet: building on mac022Running client tests...3=== RUN TestDoServerRequestAttachesToken4=== PAUSE TestDoServerRequestAttachesToken5=== RUN TestUploadBuildLog_FileBodyReplayedOnRetry6=== PAUSE TestUploadBuildLog_FileBodyReplayedOnRetry7=== RUN TestRegisterUploadedObjectReusesConnections8=== PAUSE TestRegisterUploadedObjectReusesConnections9=== RUN TestRunGarbageCollection_FinishedOnAnotherReplica10=== PAUSE TestRunGarbageCollection_FinishedOnAnotherReplica11=== RUN TestRunGarbageCollection_NotFoundAfterLocalRun12=== PAUSE TestRunGarbageCollection_NotFoundAfterLocalRun13=== RUN TestCaseHackSuffix14=== PAUSE TestCaseHackSuffix15=== RUN TestFilterOversizedClosures16=== PAUSE TestFilterOversizedClosures17=== RUN TestUploadMultipart_PartsInParallel18=== PAUSE TestUploadMultipart_PartsInParallel19=== RUN TestUploadMultipart_ReturnsAfterDroppedBody20=== PAUSE TestUploadMultipart_ReturnsAfterDroppedBody21=== RUN TestUploadMultipart_FailedProducerIsNotCompleted22=== PAUSE TestUploadMultipart_FailedProducerIsNotCompleted23=== RUN TestPartSizeForNAR24=== PAUSE TestPartSizeForNAR25=== RUN TestDumpPathCaseHackMatchesNix26--- PASS: TestDumpPathCaseHackMatchesNix (0.09s)27=== RUN TestDumpPathCaseHackCollision28--- PASS: TestDumpPathCaseHackCollision (0.00s)29=== RUN TestDumpPathMatchesNix30=== PAUSE TestDumpPathMatchesNix31=== RUN TestDumpPathSingleFile32=== PAUSE TestDumpPathSingleFile33=== RUN TestDumpPathWriterError34=== PAUSE TestDumpPathWriterError35=== RUN TestEncodeNixBase3236=== PAUSE TestEncodeNixBase3237=== RUN TestEncodeNixBase32WithRealHash38=== PAUSE TestEncodeNixBase32WithRealHash39=== RUN TestConvertHashToNix3240=== PAUSE TestConvertHashToNix3241=== RUN TestGetStorePathHash42=== PAUSE TestGetStorePathHash43=== RUN TestPathInfoHashCompatibility44=== PAUSE TestPathInfoHashCompatibility45=== RUN TestParsePathInfoJSON46=== PAUSE TestParsePathInfoJSON47=== RUN TestParsePathInfoJSONMultiplePaths48=== PAUSE TestParsePathInfoJSONMultiplePaths49=== RUN TestPathInfoCACompatibility50=== PAUSE TestPathInfoCACompatibility51=== RUN TestRateLimiterFeedback52=== PAUSE TestRateLimiterFeedback53=== RUN TestRateLimiterFeedback_400DoesNotCountAsSuccess54=== PAUSE TestRateLimiterFeedback_400DoesNotCountAsSuccess55=== RUN TestRegisterUploadedObject_BoundedAgainstSilentServer56=== PAUSE TestRegisterUploadedObject_BoundedAgainstSilentServer57=== RUN TestRegisterUploadedObject_OutlivesPush58=== PAUSE TestRegisterUploadedObject_OutlivesPush59=== RUN TestResolveStorePath60=== PAUSE TestResolveStorePath61=== RUN TestDoWithRetry_BodyReplayedViaGetBody62=== PAUSE TestDoWithRetry_BodyReplayedViaGetBody63=== RUN TestRetryTimeoutOutlastsMaxRetries64=== PAUSE TestRetryTimeoutOutlastsMaxRetries65=== RUN TestDoWithRetry_FinalResponseBodyReadable66=== PAUSE TestDoWithRetry_FinalResponseBodyReadable67=== RUN TestShellSplit68=== PAUSE TestShellSplit69=== RUN TestShellSplitErrors70=== PAUSE TestShellSplitErrors71=== RUN TestStreamPushReportsEveryPath72=== PAUSE TestStreamPushReportsEveryPath73=== RUN TestStreamPushBatchesUnderLoad74=== PAUSE TestStreamPushBatchesUnderLoad75=== RUN TestStreamPushIsolatesFailures76=== PAUSE TestStreamPushIsolatesFailures77=== RUN TestStreamPushGivesUpOnDeadServer78=== PAUSE TestStreamPushGivesUpOnDeadServer79=== RUN TestStreamPushRequestLine80=== PAUSE TestStreamPushRequestLine81=== RUN TestStreamPushReportsSignatures82=== PAUSE TestStreamPushReportsSignatures83=== RUN TestStreamPushReportsSkippedPaths84=== PAUSE TestStreamPushReportsSkippedPaths85=== RUN TestClientSignaturesByStorePath86=== PAUSE TestClientSignaturesByStorePath87=== RUN TestStreamPushAnswersEveryLineWhenCanceled88=== PAUSE TestStreamPushAnswersEveryLineWhenCanceled89=== RUN TestStreamPushStopsWhenCanceled90=== PAUSE TestStreamPushStopsWhenCanceled91=== RUN TestSetClientTLS92=== PAUSE TestSetClientTLS93=== RUN TestSetClientTLSDoesNotMutateDefaultTransport94=== PAUSE TestSetClientTLSDoesNotMutateDefaultTransport95=== RUN TestSetClientTLSErrors96=== PAUSE TestSetClientTLSErrors97=== RUN TestStaticToken98=== PAUSE TestStaticToken99=== RUN TestFileTokenReadsAndCaches100=== PAUSE TestFileTokenReadsAndCaches101=== RUN TestFileTokenMissing102=== PAUSE TestFileTokenMissing103=== RUN TestFileTokenEmpty104=== PAUSE TestFileTokenEmpty105=== RUN TestScriptTokenNoExpiryRerunsEveryCall106=== PAUSE TestScriptTokenNoExpiryRerunsEveryCall107=== RUN TestScriptTokenCachesUntilRefresh108=== PAUSE TestScriptTokenCachesUntilRefresh109=== RUN TestScriptTokenEmptyToken110=== PAUSE TestScriptTokenEmptyToken111=== RUN TestScriptTokenBadJSON112=== PAUSE TestScriptTokenBadJSON113=== RUN TestScriptTokenScriptFails114=== PAUSE TestScriptTokenScriptFails115=== RUN TestScriptTokenEmptyCommand116=== PAUSE TestScriptTokenEmptyCommand117=== CONT TestDoServerRequestAttachesToken118=== CONT TestRetryTimeoutOutlastsMaxRetries119=== CONT TestEncodeNixBase32WithRealHash120=== CONT TestEncodeNixBase32121=== RUN TestEncodeNixBase32/test_string_hash122=== PAUSE TestEncodeNixBase32/test_string_hash123=== CONT TestDoWithRetry_BodyReplayedViaGetBody124=== CONT TestResolveStorePath125=== CONT TestRegisterUploadedObject_OutlivesPush126=== CONT TestRegisterUploadedObject_BoundedAgainstSilentServer127=== CONT TestRateLimiterFeedback_400DoesNotCountAsSuccess128=== CONT TestRateLimiterFeedback129=== CONT TestPathInfoCACompatibility130=== CONT TestParsePathInfoJSONMultiplePaths131=== CONT TestParsePathInfoJSON132=== CONT TestPathInfoHashCompatibility133=== CONT TestGetStorePathHash134=== CONT TestConvertHashToNix32135=== CONT TestSetClientTLS136=== CONT TestScriptTokenEmptyCommand137=== CONT TestScriptTokenScriptFails138=== CONT TestScriptTokenBadJSON139=== CONT TestScriptTokenEmptyToken140=== CONT TestScriptTokenCachesUntilRefresh1412026/11/12 16:56:59 WARN Rate limiter enabled after throttle name=server-test rate=5142=== CONT TestScriptTokenNoExpiryRerunsEveryCall143=== CONT TestFileTokenEmpty1442026/11/12 16:56:59 WARN Rate limiter enabled after throttle name=server-test rate=51452026/11/12 16:56:59 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:642311462026/11/12 16:56:59 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:642301472026/11/12 16:56:59 WARN Rate limiter backed off name=server-test rate=5148--- PASS: TestEncodeNixBase32WithRealHash (0.00s)149--- PASS: TestResolveStorePath (0.00s)1502026/11/12 16:56:59 WARN Request returned retryable status, retrying attempt=2 backoff=0s status=503 url=http://127.0.0.1:64231151=== CONT TestFileTokenMissing152=== RUN TestEncodeNixBase32/empty_input153=== PAUSE TestEncodeNixBase32/empty_input154=== RUN TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths155=== RUN TestParsePathInfoJSON/Nix_format156=== RUN TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)157=== RUN TestGetStorePathHash/valid_store_path158=== RUN TestConvertHashToNix32/SRI_format_to_Nix32159=== CONT TestFileTokenReadsAndCaches160=== CONT TestStaticToken161=== RUN TestRateLimiterFeedback/429_enables_limiter162=== PAUSE TestGetStorePathHash/valid_store_path163=== CONT TestStreamPushGivesUpOnDeadServer164=== PAUSE TestParsePathInfoJSON/Nix_format165=== CONT TestStreamPushAnswersEveryLineWhenCanceled166=== PAUSE TestRateLimiterFeedback/429_enables_limiter167=== RUN TestParsePathInfoJSON/Lix_format1682026/11/12 16:56:59 ERROR Upload failed error="connection refused" count=201692026/11/12 16:56:59 ERROR Server seems unavailable, giving up on batch untried=17170=== RUN TestGetStorePathHash/basename_without_hyphen_should_error171=== CONT TestSetClientTLSErrors172--- PASS: TestScriptTokenEmptyCommand (0.00s)173=== CONT TestSetClientTLSDoesNotMutateDefaultTransport174=== PAUSE TestGetStorePathHash/basename_without_hyphen_should_error175=== PAUSE TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths176=== RUN TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths177=== PAUSE TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths178=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)179=== CONT TestUploadMultipart_PartsInParallel180=== RUN TestPathInfoHashCompatibility/old_string_format_with_colon181=== CONT TestStreamPushRequestLine182=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_colon183=== RUN TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI1842026/11/12 16:56:59 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:642301852000/01/01 00:00:00 ERROR Upload failed error="context canceled" count=2186=== PAUSE TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI187=== CONT TestStreamPushStopsWhenCanceled1882026/11/12 16:56:59 ERROR Upload failed error=boom count=1189=== RUN TestPathInfoCACompatibility/null_ca_field190=== PAUSE TestConvertHashToNix32/SRI_format_to_Nix32191=== RUN TestRateLimiterFeedback/503_enables_limiter192=== PAUSE TestParsePathInfoJSON/Lix_format193--- PASS: TestRegisterUploadedObject_OutlivesPush (0.00s)194=== RUN TestGetStorePathHash/hash_with_invalid_characters_should_error195=== CONT TestClientSignaturesByStorePath196=== CONT TestStreamPushReportsSkippedPaths197=== PAUSE TestPathInfoCACompatibility/null_ca_field198=== PAUSE TestRateLimiterFeedback/503_enables_limiter199=== CONT TestDumpPathSingleFile200=== RUN TestConvertHashToNix32/already_Nix32_format201=== RUN TestRateLimiterFeedback/200_does_not_enable_limiter202=== PAUSE TestRateLimiterFeedback/200_does_not_enable_limiter203=== RUN TestRateLimiterFeedback/400_does_not_enable_limiter204=== PAUSE TestRateLimiterFeedback/400_does_not_enable_limiter205=== RUN TestParsePathInfoJSON/empty_input206=== CONT TestPartSizeForNAR207=== RUN TestPartSizeForNAR/zero_stays_at_minimum208=== PAUSE TestPartSizeForNAR/zero_stays_at_minimum209=== RUN TestPartSizeForNAR/small_stays_at_minimum210=== PAUSE TestPartSizeForNAR/small_stays_at_minimum211=== RUN TestPartSizeForNAR/80_GiB_fits_at_minimum212--- PASS: TestStaticToken (0.00s)213--- PASS: TestDoServerRequestAttachesToken (0.00s)214=== PAUSE TestParsePathInfoJSON/empty_input215=== RUN TestParsePathInfoJSON/whitespace_only216=== PAUSE TestParsePathInfoJSON/whitespace_only217=== RUN TestParsePathInfoJSON/invalid_JSON218=== PAUSE TestGetStorePathHash/hash_with_invalid_characters_should_error219=== RUN TestGetStorePathHash/hash_with_wrong_length_should_error220--- PASS: TestDoWithRetry_BodyReplayedViaGetBody (0.00s)221--- PASS: TestStreamPushGivesUpOnDeadServer (0.00s)222--- PASS: TestFileTokenMissing (0.00s)223=== CONT TestUploadMultipart_FailedProducerIsNotCompleted224=== RUN TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part225=== PAUSE TestConvertHashToNix32/already_Nix32_format226--- PASS: TestFileTokenEmpty (0.00s)227=== RUN TestPathInfoCACompatibility/old_string_format_-_text228=== PAUSE TestPathInfoCACompatibility/old_string_format_-_text229=== RUN TestPathInfoHashCompatibility/new_structured_format_with_sha512230=== CONT TestDumpPathWriterError231=== CONT TestStreamPushReportsSignatures232=== CONT TestDumpPathMatchesNix233=== PAUSE TestPartSizeForNAR/80_GiB_fits_at_minimum234=== PAUSE TestParsePathInfoJSON/invalid_JSON235=== PAUSE TestGetStorePathHash/hash_with_wrong_length_should_error236--- PASS: TestFileTokenReadsAndCaches (0.00s)237=== CONT TestUploadMultipart_ReturnsAfterDroppedBody238=== RUN TestSetClientTLSErrors/missing_cert_file2392026/11/12 16:56:59 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:64230240=== PAUSE TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part241=== RUN TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary242=== RUN TestConvertHashToNix32/invalid_format2432026/11/12 16:56:59 ERROR Upload failed error=boom count=1244=== PAUSE TestConvertHashToNix32/invalid_format245=== RUN TestPathInfoCACompatibility/old_string_format_-_fixed_recursive246=== PAUSE TestPathInfoHashCompatibility/new_structured_format_with_sha512247--- PASS: TestStreamPushAnswersEveryLineWhenCanceled (0.01s)248=== RUN TestSetClientTLS/rejects_connection_without_client_cert249=== PAUSE TestSetClientTLS/rejects_connection_without_client_cert250=== CONT TestStreamPushReportsEveryPath251=== RUN TestPartSizeForNAR/115_GiB_needs_larger_parts252=== PAUSE TestPartSizeForNAR/115_GiB_needs_larger_parts253=== CONT TestStreamPushIsolatesFailures254=== PAUSE TestSetClientTLSErrors/missing_cert_file2552026/11/12 16:56:59 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:642302562026/11/12 16:56:59 ERROR Upload failed error="bad path" count=3257=== CONT TestCaseHackSuffix258=== RUN TestSetClientTLSErrors/missing_key_file259=== PAUSE TestSetClientTLSErrors/missing_key_file260=== CONT TestRunGarbageCollection_NotFoundAfterLocalRun261=== CONT TestStreamPushBatchesUnderLoad262=== RUN TestStreamPushBatchesUnderLoad/oneAtATime=false263=== PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=false264=== RUN TestStreamPushBatchesUnderLoad/oneAtATime=true265=== PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=true266=== PAUSE TestPathInfoCACompatibility/old_string_format_-_fixed_recursive267=== RUN TestPathInfoCACompatibility/new_structured_format_-_text268=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_text269=== RUN TestPathInfoCACompatibility/new_structured_format_-_nar_method270=== CONT TestShellSplit271=== CONT TestShellSplitErrors272=== CONT TestDoWithRetry_FinalResponseBodyReadable273--- PASS: TestStreamPushStopsWhenCanceled (0.00s)274--- PASS: TestClientSignaturesByStorePath (0.00s)275--- PASS: TestStreamPushReportsSkippedPaths (0.00s)276--- PASS: TestScriptTokenScriptFails (0.01s)277--- PASS: TestStreamPushReportsSignatures (0.00s)278--- PASS: TestStreamPushReportsEveryPath (0.00s)279--- PASS: TestScriptTokenEmptyToken (0.01s)280=== RUN TestSetClientTLS/succeeds_with_client_cert_and_CA281=== RUN TestPartSizeForNAR/1_TiB282=== PAUSE TestSetClientTLS/succeeds_with_client_cert_and_CA283=== PAUSE TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary284=== CONT TestRegisterUploadedObjectReusesConnections285=== RUN TestSetClientTLSErrors/missing_ca_file286=== RUN TestSetClientTLS/preserves_debug_logging_transport287=== CONT TestFilterOversizedClosures288=== RUN TestFilterOversizedClosures/no_limit_keeps_everything289=== PAUSE TestFilterOversizedClosures/no_limit_keeps_everything290=== RUN TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped291=== PAUSE TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped292=== RUN TestFilterOversizedClosures/all_closures_skipped293--- PASS: TestStreamPushIsolatesFailures (0.00s)294--- PASS: TestShellSplit (0.00s)295--- PASS: TestShellSplitErrors (0.00s)296=== PAUSE TestPartSizeForNAR/1_TiB297--- PASS: TestUploadMultipart_ReturnsAfterDroppedBody (0.01s)298=== CONT TestRunGarbageCollection_FinishedOnAnotherReplica299=== RUN TestPartSizeForNAR/5_TiB_S3_max_object300=== PAUSE TestPartSizeForNAR/5_TiB_S3_max_object301=== CONT TestUploadBuildLog_FileBodyReplayedOnRetry302=== PAUSE TestSetClientTLSErrors/missing_ca_file303=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_nar_method304=== PAUSE TestSetClientTLS/preserves_debug_logging_transport305=== CONT TestEncodeNixBase32/test_string_hash306=== CONT TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths307=== PAUSE TestFilterOversizedClosures/all_closures_skipped308=== RUN TestPartSizeForNAR/capped_at_5_GiB309=== CONT TestRateLimiterFeedback/429_enables_limiter310=== RUN TestSetClientTLSErrors/invalid_ca_file311=== PAUSE TestPartSizeForNAR/capped_at_5_GiB312=== PAUSE TestSetClientTLSErrors/invalid_ca_file313=== CONT TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths314=== CONT TestRateLimiterFeedback/503_enables_limiter315--- PASS: TestParsePathInfoJSONMultiplePaths (0.00s)316 --- PASS: TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths (0.00s)317 --- PASS: TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths (0.00s)318=== CONT TestRateLimiterFeedback/400_does_not_enable_limiter319=== CONT TestRateLimiterFeedback/200_does_not_enable_limiter320=== CONT TestEncodeNixBase32/empty_input321--- PASS: TestEncodeNixBase32 (0.00s)322 --- PASS: TestEncodeNixBase32/test_string_hash (0.00s)323 --- PASS: TestEncodeNixBase32/empty_input (0.00s)324=== CONT TestGetStorePathHash/valid_store_path325=== CONT TestGetStorePathHash/hash_with_invalid_characters_should_error3262026/11/12 16:56:59 WARN Request returned retryable status, retrying attempt=3 backoff=1ms status=502 url=http://127.0.0.1:64230327=== CONT TestGetStorePathHash/hash_with_wrong_length_should_error328=== CONT TestGetStorePathHash/basename_without_hyphen_should_error329--- PASS: TestGetStorePathHash (0.01s)330 --- PASS: TestGetStorePathHash/valid_store_path (0.00s)331 --- PASS: TestGetStorePathHash/hash_with_invalid_characters_should_error (0.00s)332 --- PASS: TestGetStorePathHash/hash_with_wrong_length_should_error (0.00s)333 --- PASS: TestGetStorePathHash/basename_without_hyphen_should_error (0.00s)334=== CONT TestParsePathInfoJSON/Nix_format335=== CONT TestParsePathInfoJSON/invalid_JSON336=== CONT TestParsePathInfoJSON/empty_input337=== CONT TestParsePathInfoJSON/Lix_format338=== CONT TestParsePathInfoJSON/whitespace_only339=== CONT TestConvertHashToNix32/SRI_format_to_Nix32340--- PASS: TestParsePathInfoJSON (0.01s)341 --- PASS: TestParsePathInfoJSON/Nix_format (0.00s)342 --- PASS: TestParsePathInfoJSON/invalid_JSON (0.00s)343 --- PASS: TestParsePathInfoJSON/empty_input (0.00s)344 --- PASS: TestParsePathInfoJSON/whitespace_only (0.00s)345 --- PASS: TestParsePathInfoJSON/Lix_format (0.00s)346=== CONT TestConvertHashToNix32/already_Nix32_format347=== CONT TestConvertHashToNix32/invalid_format348--- PASS: TestConvertHashToNix32 (0.01s)349 --- PASS: TestConvertHashToNix32/SRI_format_to_Nix32 (0.00s)350 --- PASS: TestConvertHashToNix32/already_Nix32_format (0.00s)351 --- PASS: TestConvertHashToNix32/invalid_format (0.00s)352=== CONT TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)353=== CONT TestPathInfoHashCompatibility/new_structured_format_with_sha512354=== CONT TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI355=== CONT TestStreamPushBatchesUnderLoad/oneAtATime=false356=== CONT TestPathInfoHashCompatibility/old_string_format_with_colon357--- PASS: TestPathInfoHashCompatibility (0.01s)358 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI) (0.00s)359 --- PASS: TestPathInfoHashCompatibility/new_structured_format_with_sha512 (0.00s)360 --- PASS: TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI (0.00s)361 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_colon (0.00s)362=== CONT TestStreamPushBatchesUnderLoad/oneAtATime=true363--- PASS: TestSetClientTLSDoesNotMutateDefaultTransport (0.01s)364=== CONT TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part365--- PASS: TestStreamPushBatchesUnderLoad (0.00s)366 --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=false (0.00s)367 --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=true (0.00s)368=== CONT TestPathInfoCACompatibility/new_structured_format_-_nar_method369=== CONT TestPathInfoCACompatibility/null_ca_field370=== CONT TestPathInfoCACompatibility/new_structured_format_-_text371=== CONT TestPathInfoCACompatibility/old_string_format_-_fixed_recursive372=== CONT TestPathInfoCACompatibility/old_string_format_-_text373--- PASS: TestPathInfoCACompatibility (0.01s)374 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_nar_method (0.00s)375 --- PASS: TestPathInfoCACompatibility/null_ca_field (0.00s)376 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_text (0.00s)377 --- PASS: TestPathInfoCACompatibility/old_string_format_-_fixed_recursive (0.00s)378 --- PASS: TestPathInfoCACompatibility/old_string_format_-_text (0.00s)379=== CONT TestSetClientTLS/rejects_connection_without_client_cert380=== CONT TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary381--- PASS: TestScriptTokenBadJSON (0.01s)382=== CONT TestSetClientTLS/preserves_debug_logging_transport3832026/11/12 16:56:59 WARN Rate limiter enabled after throttle name=server-test rate=53842026/11/12 16:56:59 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:642603852026/11/12 16:56:59 WARN Rate limiter backed off name=server-test rate=53862026/11/12 16:56:59 INFO Garbage collection started387=== CONT TestSetClientTLS/succeeds_with_client_cert_and_CA3882026/11/12 16:56:59 WARN Request returned retryable status, retrying attempt=4 backoff=1ms status=502 url=http://127.0.0.1:642303892026/11/12 16:56:59 WARN Rate limiter enabled after throttle name=server-test rate=53902026/11/12 16:56:59 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=429 url=http://127.0.0.1:64261391=== CONT TestFilterOversizedClosures/no_limit_keeps_everything392=== CONT TestFilterOversizedClosures/all_closures_skipped3932026/11/12 16:56:59 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper nar_size=100 max_nar_size=50394=== CONT TestPartSizeForNAR/zero_stays_at_minimum395=== CONT TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped3962026/11/12 16:56:59 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb-vm-image nar_size=5000 max_nar_size=2000397--- PASS: TestFilterOversizedClosures (0.00s)398 --- PASS: TestFilterOversizedClosures/no_limit_keeps_everything (0.00s)399 --- PASS: TestFilterOversizedClosures/all_closures_skipped (0.00s)400 --- PASS: TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped (0.00s)401=== CONT TestPartSizeForNAR/capped_at_5_GiB4022026/11/12 16:56:59 WARN Rate limiter backed off name=server-test rate=5403=== CONT TestPartSizeForNAR/80_GiB_fits_at_minimum404=== CONT TestSetClientTLSErrors/missing_cert_file405=== CONT TestPartSizeForNAR/115_GiB_needs_larger_parts406=== CONT TestSetClientTLSErrors/missing_ca_file407=== CONT TestSetClientTLSErrors/missing_key_file408=== CONT TestSetClientTLSErrors/invalid_ca_file4092026/11/12 16:56:59 INFO Garbage collection started410--- PASS: TestUploadMultipart_FailedProducerIsNotCompleted (0.01s)411 --- PASS: TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary (0.00s)412 --- PASS: TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part (0.00s)413--- PASS: TestRateLimiterFeedback (0.00s)414 --- PASS: TestRateLimiterFeedback/503_enables_limiter (0.00s)415 --- PASS: TestRateLimiterFeedback/200_does_not_enable_limiter (0.01s)416 --- PASS: TestRateLimiterFeedback/429_enables_limiter (0.01s)417 --- PASS: TestRateLimiterFeedback/400_does_not_enable_limiter (0.01s)418=== CONT TestPartSizeForNAR/1_TiB419=== CONT TestPartSizeForNAR/small_stays_at_minimum420=== CONT TestPartSizeForNAR/5_TiB_S3_max_object421--- PASS: TestPartSizeForNAR (0.01s)422 --- PASS: TestPartSizeForNAR/zero_stays_at_minimum (0.00s)423 --- PASS: TestPartSizeForNAR/capped_at_5_GiB (0.00s)424 --- PASS: TestPartSizeForNAR/80_GiB_fits_at_minimum (0.00s)425 --- PASS: TestPartSizeForNAR/115_GiB_needs_larger_parts (0.00s)426 --- PASS: TestPartSizeForNAR/1_TiB (0.00s)427 --- PASS: TestPartSizeForNAR/small_stays_at_minimum (0.00s)428 --- PASS: TestPartSizeForNAR/5_TiB_S3_max_object (0.00s)4292026/11/12 16:56:59 WARN Rate limiter enabled after throttle name=server-test rate=54302026/11/12 16:56:59 WARN Rate limiter enabled after throttle name=s3-test rate=54312026/11/12 16:56:59 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:642474322026/11/12 16:56:59 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:64253433--- PASS: TestSetClientTLSErrors (0.01s)434 --- PASS: TestSetClientTLSErrors/missing_cert_file (0.00s)435 --- PASS: TestSetClientTLSErrors/missing_key_file (0.00s)436 --- PASS: TestSetClientTLSErrors/invalid_ca_file (0.00s)437 --- PASS: TestSetClientTLSErrors/missing_ca_file (0.00s)4382026/11/12 16:56:59 WARN Request returned retryable status, retrying attempt=5 backoff=1ms status=502 url=http://127.0.0.1:64230439--- PASS: TestRunGarbageCollection_NotFoundAfterLocalRun (0.01s)4402026/11/12 16:56:59 INFO Garbage collection progress phase=running_on_another_replica failed_uploads_deleted=0 old_closures_deleted=0 objects_marked=0 objects_deleted=0 objects_failed=04412026/11/12 16:56:59 WARN Rate limiter backed off name=server-test rate=5442--- PASS: TestDoWithRetry_FinalResponseBodyReadable (0.01s)443--- PASS: TestUploadBuildLog_FileBodyReplayedOnRetry (0.01s)4442026/11/12 16:56:59 WARN Garbage collection finished on another replica. Its statistics are not available here445--- PASS: TestRunGarbageCollection_FinishedOnAnotherReplica (0.01s)4462026/11/12 16:56:59 WARN Request returned retryable status, retrying attempt=6 backoff=1ms status=502 url=http://127.0.0.1:642304472026/11/12 16:56:59 WARN Request returned retryable status, retrying attempt=7 backoff=1ms status=502 url=http://127.0.0.1:64230448--- PASS: TestStreamPushRequestLine (0.02s)4492026/11/12 16:56:59 WARN Request returned retryable status, retrying attempt=8 backoff=1ms status=502 url=http://127.0.0.1:642304502026/11/12 16:56:59 WARN Request returned retryable status, retrying attempt=9 backoff=1ms status=502 url=http://127.0.0.1:642304512026/11/12 16:56:59 WARN Request returned retryable status, retrying attempt=10 backoff=1ms status=502 url=http://127.0.0.1:642304522026/11/12 16:56:59 WARN Request returned retryable status, retrying attempt=11 backoff=1ms status=502 url=http://127.0.0.1:642304532026/11/12 16:56:59 WARN Request returned retryable status, retrying attempt=12 backoff=1ms status=502 url=http://127.0.0.1:642304542026/11/12 16:56:59 WARN Request returned retryable status, retrying attempt=13 backoff=1ms status=502 url=http://127.0.0.1:64230455--- PASS: TestRegisterUploadedObjectReusesConnections (0.03s)4562026/11/12 16:56:59 WARN Request returned retryable status, retrying attempt=14 backoff=1ms status=502 url=http://127.0.0.1:642304572026/11/12 16:56:59 http: TLS handshake error from 127.0.0.1:64255: remote error: tls: bad certificate4582026/11/12 16:56:59 WARN Request returned retryable status, retrying attempt=15 backoff=1ms status=502 url=http://127.0.0.1:64230459--- PASS: TestSetClientTLS (0.01s)460 --- PASS: TestSetClientTLS/preserves_debug_logging_transport (0.01s)461 --- PASS: TestSetClientTLS/succeeds_with_client_cert_and_CA (0.01s)462 --- PASS: TestSetClientTLS/rejects_connection_without_client_cert (0.03s)4632026/11/12 16:56:59 WARN Request returned retryable status, retrying attempt=16 backoff=1ms status=502 url=http://127.0.0.1:642304642026/11/12 16:56:59 WARN Request returned retryable status, retrying attempt=17 backoff=1ms status=502 url=http://127.0.0.1:64230465--- PASS: TestRetryTimeoutOutlastsMaxRetries (0.05s)466--- PASS: TestScriptTokenNoExpiryRerunsEveryCall (0.05s)467--- PASS: TestScriptTokenCachesUntilRefresh (0.05s)468--- PASS: TestDumpPathWriterError (0.05s)469--- PASS: TestDumpPathSingleFile (0.05s)470--- PASS: TestCaseHackSuffix (0.05s)471--- PASS: TestDumpPathMatchesNix (0.08s)4722026/11/12 16:56:59 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:64233/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:64233/api/uploads/complete4732026/11/12 16:56:59 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"4742026/11/12 16:56:59 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:64233/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:64233/api/uploads/complete4752026/11/12 16:56:59 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"4762026/11/12 16:56:59 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:64233/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:64233/api/uploads/complete4772026/11/12 16:56:59 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"478--- PASS: TestRegisterUploadedObject_BoundedAgainstSilentServer (0.10s)479--- PASS: TestUploadMultipart_PartsInParallel (0.62s)480--- PASS: TestRateLimiterFeedback_400DoesNotCountAsSuccess (1.00s)481PASS482Running server tests...483The files belonging to this database system will be owned by user "_tribuchetbld2".484This user must also own the server process.485486The database cluster will be initialized with locale "C".487The default database encoding has accordingly been set to "SQL_ASCII".488The default text search configuration will be set to "english".489490Data page checksums are enabled.491492creating directory /private/var/tb/2/scratch/777606c6228c2b4e/build/postgres865372459/data ... ok493creating subdirectories ... ok494selecting dynamic shared memory implementation ... posix495selecting default "max_connections" ... 100496selecting default "shared_buffers" ... 128MB497selecting default time zone ... UTC498creating configuration files ... ok499running bootstrap script ... ok500performing post-bootstrap initialization ... ok501syncing data to disk ... ok502503initdb: warning: enabling "trust" authentication for local connections504initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.505506Success. You can now start the database server using:507508 pg_ctl -D /private/var/tb/2/scratch/777606c6228c2b4e/build/postgres865372459/data -l logfile start5095102026-11-12 16:57:01.351 UTC [8743] LOG: starting PostgreSQL 18.6 on aarch64-apple-darwin25.6.0, compiled by clang version 21.1.8, 64-bit5112026-11-12 16:57:01.352 UTC [8743] LOG: listening on Unix socket "/private/var/tb/2/scratch/777606c6228c2b4e/build/postgres865372459/.s.PGSQL.5432"5122026-11-12 16:57:01.354 UTC [8750] LOG: database system was shut down at 2026-11-12 16:57:01 UTC5132026-11-12 16:57:01.354 UTC [8751] FATAL: the database system is starting up514/private/var/tb/2/scratch/777606c6228c2b4e/build/postgres865372459:5432 - rejecting connections5152026-11-12 16:57:01.355 UTC [8743] LOG: database system is ready to accept connections516/private/var/tb/2/scratch/777606c6228c2b4e/build/postgres865372459:5432 - accepting connections517{"timestamp":"2026-11-12T16:57:01.597704Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"239ac427-3f2d-44cf-8329-9ec08f2f5e88","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"GET","uri":"/health/ready","status_code":503,"suppressed_errors":0,"duration_ms":9,"result":"server_error","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":502,"threadName":"rustfs-worker","threadId":"ThreadId(8)"}518{"timestamp":"2026-11-12T16:57:01.705172Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"0e399e29-7cbe-42d0-bf18-eb850010fbc9","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"GET","uri":"/health/ready","status_code":503,"suppressed_errors":0,"duration_ms":0,"result":"server_error","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":502,"threadName":"rustfs-worker","threadId":"ThreadId(6)"}519=== RUN TestAbortSucceeded520=== PAUSE TestAbortSucceeded521=== RUN TestService_AuthMiddleware522=== PAUSE TestService_AuthMiddleware523=== RUN TestService_AuthMiddleware_MTLSProxyHeader524=== PAUSE TestService_AuthMiddleware_MTLSProxyHeader525=== RUN TestService_AuthMiddleware_MTLSBoundSubjects526=== PAUSE TestService_AuthMiddleware_MTLSBoundSubjects527=== RUN TestService_ReadAuthMiddleware528=== PAUSE TestService_ReadAuthMiddleware529=== RUN TestService_AuthMiddleware_OIDC530=== PAUSE TestService_AuthMiddleware_OIDC531=== RUN TestService_RequireScope_OIDC532=== PAUSE TestService_RequireScope_OIDC533=== RUN TestService_ReadScope_PublicByDefault534=== PAUSE TestService_ReadScope_PublicByDefault535=== RUN TestCacheConfigHandler536=== PAUSE TestCacheConfigHandler537=== RUN TestCacheStatsHandler538=== PAUSE TestCacheStatsHandler539=== RUN TestClientCADerivations540=== PAUSE TestClientCADerivations541=== RUN TestClientErrorHandling542=== PAUSE TestClientErrorHandling543=== RUN TestClientIntegration544=== PAUSE TestClientIntegration545=== RUN TestClientMultipleUploads546=== PAUSE TestClientMultipleUploads547=== RUN TestClientWithDependencies548=== PAUSE TestClientWithDependencies549=== RUN TestClientSharedPathCommittedMidPush550=== PAUSE TestClientSharedPathCommittedMidPush551=== RUN TestPinProtectsFromGC552=== PAUSE TestPinProtectsFromGC553=== RUN TestClientPushesUseOnePush554=== PAUSE TestClientPushesUseOnePush555=== RUN TestClientFallsBackToClosures556=== PAUSE TestClientFallsBackToClosures557=== RUN TestDeleteClosuresRacingPins558=== PAUSE TestDeleteClosuresRacingPins559=== RUN TestConcurrentCommitsSharingObjectsDoNotDeadlock560=== PAUSE TestConcurrentCommitsSharingObjectsDoNotDeadlock561=== RUN TestResolveDBConnectionString562=== PAUSE TestResolveDBConnectionString563=== RUN TestConnectWaitsForAPeerMigration564=== PAUSE TestConnectWaitsForAPeerMigration565=== RUN TestLeadElectsOneAndHandsOver566=== PAUSE TestLeadElectsOneAndHandsOver567=== RUN TestLeadIncumbentWinsAfterRestart5682026/11/12 16:57:02 INFO lead: acquired remote=192.0.2.1:12345692026/11/12 16:57:02 INFO lead: released remote=192.0.2.1:12345702026/11/12 16:57:02 INFO lead: acquired remote=192.0.2.1:12345712026/11/12 16:57:03 INFO lead: released remote=192.0.2.1:1234572--- PASS: TestLeadIncumbentWinsAfterRestart (2.05s)573=== RUN TestLeadUnannouncedStandbyYieldsToIncumbent574=== PAUSE TestLeadUnannouncedStandbyYieldsToIncumbent575=== RUN TestLeadIncumbentAnnouncesAtOnce576=== PAUSE TestLeadIncumbentAnnouncesAtOnce577=== RUN TestLeadEndsOnShutdown578=== PAUSE TestLeadEndsOnShutdown579=== RUN TestLeadEndsWhenItsConnectionHangs5802026/11/12 16:57:03 INFO lead: acquired remote=192.0.2.1:12345812026-11-12 16:57:05.383 UTC [8809] FATAL: terminating connection due to administrator command5822026/11/12 16:57:05 INFO lead: acquired remote=192.0.2.1:12345832026/11/12 16:57:06 WARN lead: lock connection lost error="timeout: context deadline exceeded"5842026/11/12 16:57:06 INFO lead: released remote=192.0.2.1:12345852026/11/12 16:57:06 INFO lead: released remote=192.0.2.1:1234586--- PASS: TestLeadEndsWhenItsConnectionHangs (2.92s)587=== RUN TestGCAdvisoryLockBlocksConcurrentRun588=== PAUSE TestGCAdvisoryLockBlocksConcurrentRun589=== RUN TestGCBugBareHashReferences590=== PAUSE TestGCBugBareHashReferences591=== RUN TestGCMetrics592=== PAUSE TestGCMetrics593=== RUN TestPushDedupSurvivesConcurrentGC594=== PAUSE TestPushDedupSurvivesConcurrentGC595=== RUN TestDeduplicatedObjectsRecordedAsPending596=== PAUSE TestDeduplicatedObjectsRecordedAsPending597=== RUN TestGCSweepSkipsPendingObjects598=== PAUSE TestGCSweepSkipsPendingObjects599=== RUN TestTombstonedObjectOfferedWithoutWaiting600=== PAUSE TestTombstonedObjectOfferedWithoutWaiting601=== RUN TestGCSweepDeliversEachKeyOnce602=== PAUSE TestGCSweepDeliversEachKeyOnce603=== RUN TestCreatePendingClosureVerifyS3FailureReleasesConnection604=== PAUSE TestCreatePendingClosureVerifyS3FailureReleasesConnection605=== RUN TestForceGCDuringPushOffersSweptObject606=== PAUSE TestForceGCDuringPushOffersSweptObject607=== RUN TestFailedS3DeleteKeepsTombstone608=== PAUSE TestFailedS3DeleteKeepsTombstone609=== RUN TestSweepSkipsRowsLockedByPush610=== PAUSE TestSweepSkipsRowsLockedByPush611=== RUN TestRegisterCompletedObjectNeedsPendingRow612=== PAUSE TestRegisterCompletedObjectNeedsPendingRow613=== RUN TestGetExistingObjectsLocksRows614=== PAUSE TestGetExistingObjectsLocksRows615=== RUN TestSweepRechecksPendingAfterLocking616=== PAUSE TestSweepRechecksPendingAfterLocking617=== RUN TestPushWaitsForSweepHoldingRow618=== PAUSE TestPushWaitsForSweepHoldingRow619=== RUN TestCommitRacingPendingCleanupKeepsObjects620=== PAUSE TestCommitRacingPendingCleanupKeepsObjects621=== RUN TestGCEndsOnShutdown622=== PAUSE TestGCEndsOnShutdown623=== RUN TestGCTaskStore_StartNew624=== PAUSE TestGCTaskStore_StartNew625=== RUN TestGCTaskStore_DeduplicateSameParams626=== PAUSE TestGCTaskStore_DeduplicateSameParams627=== RUN TestGCTaskStore_ConflictDifferentParams628=== PAUSE TestGCTaskStore_ConflictDifferentParams629=== RUN TestGCTaskStore_GetEmpty630=== PAUSE TestGCTaskStore_GetEmpty631=== RUN TestGCTaskStore_GetReturnsLatest632=== PAUSE TestGCTaskStore_GetReturnsLatest633=== RUN TestGCTaskStore_CompletedAllowsNewTask634=== PAUSE TestGCTaskStore_CompletedAllowsNewTask635=== RUN TestGCTaskStore_PhaseUpdates636=== PAUSE TestGCTaskStore_PhaseUpdates637=== RUN TestGCTaskStore_Fail638=== PAUSE TestGCTaskStore_Fail639=== RUN TestGracefulShutdownDrainsInflight640=== PAUSE TestGracefulShutdownDrainsInflight641=== RUN TestService_healthCheckHandler642=== PAUSE TestService_healthCheckHandler643=== RUN TestService_readinessHandler644=== PAUSE TestService_readinessHandler645=== RUN TestGenerateLandingPage646=== PAUSE TestGenerateLandingPage647=== RUN TestEnsureAbortUploadsRule648=== PAUSE TestEnsureAbortUploadsRule649=== RUN TestCacheConfigHandlerMaxNarSize650=== PAUSE TestCacheConfigHandlerMaxNarSize651=== RUN TestCreatePendingClosureRejectsOversizedNAR652=== PAUSE TestCreatePendingClosureRejectsOversizedNAR653=== RUN TestNARDeduplicationMetadataUploadBug654=== PAUSE TestNARDeduplicationMetadataUploadBug655=== RUN TestMetricsInventory656=== PAUSE TestMetricsInventory657=== RUN TestMetricsCommit658=== PAUSE TestMetricsCommit659=== RUN TestService_NativeMTLS660=== PAUSE TestService_NativeMTLS661=== RUN TestServerTLSConfig662=== PAUSE TestServerTLSConfig663=== RUN TestMultipartCleanup664=== PAUSE TestMultipartCleanup665=== RUN TestMultipartUploadAbortedWhenCancelledBeforeRecorded666=== PAUSE TestMultipartUploadAbortedWhenCancelledBeforeRecorded667=== RUN TestPendingCleanupUsesOneCutoff668=== PAUSE TestPendingCleanupUsesOneCutoff669=== RUN TestObjectStatsTrigger670=== PAUSE TestObjectStatsTrigger671=== RUN TestObjectStatsUpdatedOncePerCommit672=== PAUSE TestObjectStatsUpdatedOncePerCommit673=== RUN TestOrphanedObjectsGC674=== PAUSE TestOrphanedObjectsGC675=== RUN TestOrphanedObjectsGCStressTest676=== PAUSE TestOrphanedObjectsGCStressTest677=== RUN TestCreatePin_ReservedPins678=== PAUSE TestCreatePin_ReservedPins679=== RUN TestPresentReportsOnlyClosureRoots680=== PAUSE TestPresentReportsOnlyClosureRoots681=== RUN TestPresentRefreshesOnlyStaleClosures682=== PAUSE TestPresentRefreshesOnlyStaleClosures683=== RUN TestPresentNotReportedWhileGCDeletesClosure684=== PAUSE TestPresentNotReportedWhileGCDeletesClosure685=== RUN TestParseSingleRange686=== PAUSE TestParseSingleRange687=== RUN TestProxyHeadersOnlyTrustedOnSocket6882026/11/12 16:57:06 INFO Starting HTTP server address=127.0.0.1:643636892026/11/12 16:57:06 INFO Starting HTTP server address=proxy.sock6902026/11/12 16:57:06 WARN mTLS auth: subject not in bound subjects subject="CN=someone"6912026/11/12 16:57:06 INFO Shutdown signal received, draining in-flight requests timeout=10s692--- PASS: TestProxyHeadersOnlyTrustedOnSocket (0.17s)693=== RUN TestIsValidCachePath694=== PAUSE TestIsValidCachePath695=== RUN TestReadProxyNarinfo696=== PAUSE TestReadProxyNarinfo697=== RUN TestReadProxyNarinfoAlreadyDecompressed698=== PAUSE TestReadProxyNarinfoAlreadyDecompressed699=== RUN TestReadProxyContentNegotiation700=== PAUSE TestReadProxyContentNegotiation701=== RUN TestReadProxyNarStreaming702=== PAUSE TestReadProxyNarStreaming703=== RUN TestReadProxy404704=== PAUSE TestReadProxy404705=== RUN TestReadProxyInvalidPath706=== PAUSE TestReadProxyInvalidPath707=== RUN TestReadProxyHead708=== PAUSE TestReadProxyHead709=== RUN TestReadProxyOutlastsServerWriteTimeout710=== PAUSE TestReadProxyOutlastsServerWriteTimeout711=== RUN TestReadProxyConditionalGet712=== PAUSE TestReadProxyConditionalGet713=== RUN TestReadProxyRootRedirectsToIndexHTML714=== PAUSE TestReadProxyRootRedirectsToIndexHTML715=== RUN TestReadProxyDisabled716=== PAUSE TestReadProxyDisabled717=== RUN TestReadRedirectNar718=== PAUSE TestReadRedirectNar719=== RUN TestReadRedirectKeepsNarinfoProxied720=== PAUSE TestReadRedirectKeepsNarinfoProxied721=== RUN TestReadProxyRangeRequest722=== PAUSE TestReadProxyRangeRequest723=== RUN TestReadRedirectUsesPublicS3URL724=== PAUSE TestReadRedirectUsesPublicS3URL725=== RUN TestPush_OverlappingRootsStoreOneRowPerKey726=== PAUSE TestPush_OverlappingRootsStoreOneRowPerKey727=== RUN TestPush_CompleteCommitsEveryRoot728=== PAUSE TestPush_CompleteCommitsEveryRoot729=== RUN TestPush_RepeatedRootCommitsOnce730=== PAUSE TestPush_RepeatedRootCommitsOnce731=== RUN TestPush_CompleteWaitsForCleanupOfThePush732=== PAUSE TestPush_CompleteWaitsForCleanupOfThePush733=== RUN TestPush_RootsAreLockedInByteOrder734=== PAUSE TestPush_RootsAreLockedInByteOrder735=== RUN TestPush_SkippedKeySurvivesGCBeforeCommit736=== PAUSE TestPush_SkippedKeySurvivesGCBeforeCommit737=== RUN TestPush_RejectsBadRequests738=== PAUSE TestPush_RejectsBadRequests739=== RUN TestPush_SignsNarinfosOfItsPendingObjects740=== PAUSE TestPush_SignsNarinfosOfItsPendingObjects741=== RUN TestCommitCanBeRepeatedUntilCleanup742=== PAUSE TestCommitCanBeRepeatedUntilCleanup743=== RUN TestSharedNarBothUploadsComplete744=== PAUSE TestSharedNarBothUploadsComplete745=== RUN TestCompleteMultipartUpload_ErrorButObjectExists746=== PAUSE TestCompleteMultipartUpload_ErrorButObjectExists747=== RUN TestCompletedNarNotReofferedAcrossClosures748=== PAUSE TestCompletedNarNotReofferedAcrossClosures749=== RUN TestPresignedUploadRegisteredBeforeCommit750=== PAUSE TestPresignedUploadRegisteredBeforeCommit751=== RUN TestService_Rustfstest752=== PAUSE TestService_Rustfstest753=== RUN TestParseSize754=== PAUSE TestParseSize755=== RUN TestSkippedUploadsHandler756=== PAUSE TestSkippedUploadsHandler757=== RUN TestSystemdListenerNotActivated758--- PASS: TestSystemdListenerNotActivated (0.00s)759=== RUN TestWatchdogBeatsWhenHealthy760--- PASS: TestWatchdogBeatsWhenHealthy (0.03s)761=== RUN TestWatchdogSkipsWhenUnhealthy7622026/11/12 16:57:07 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7632026/11/12 16:57:07 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7642026/11/12 16:57:07 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7652026/11/12 16:57:07 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7662026/11/12 16:57:07 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7672026/11/12 16:57:07 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7682026/11/12 16:57:07 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7692026/11/12 16:57:07 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7702026/11/12 16:57:07 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7712026/11/12 16:57:07 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"772--- PASS: TestWatchdogSkipsWhenUnhealthy (0.21s)773=== RUN TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle774=== PAUSE TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle775=== RUN TestProxyWriteTimeout776=== PAUSE TestProxyWriteTimeout777=== RUN TestPendingClosureWriteTimeout778=== PAUSE TestPendingClosureWriteTimeout779=== RUN TestIsValidUploadKey780=== PAUSE TestIsValidUploadKey781=== RUN TestUploadHandlersRejectInvalidKeys782=== PAUSE TestUploadHandlersRejectInvalidKeys783=== RUN TestUploadHandlersRejectOversizedBody784=== PAUSE TestUploadHandlersRejectOversizedBody785=== RUN TestService_cleanupPendingClosuresHandler786=== PAUSE TestService_cleanupPendingClosuresHandler787=== RUN TestService_createPendingClosureHandler788=== PAUSE TestService_createPendingClosureHandler789=== RUN TestService_verifyS3Integrity790=== PAUSE TestService_verifyS3Integrity791=== RUN TestService_verifyS3ReuploadsStaleNarinfo792=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo793=== RUN TestCompleteMultipartUnregistered794=== PAUSE TestCompleteMultipartUnregistered795=== RUN TestCreatePendingClosure_SmallNARUsesSimplePUT796=== PAUSE TestCreatePendingClosure_SmallNARUsesSimplePUT797=== CONT TestPush_CompleteWaitsForCleanupOfThePush798=== CONT TestGCTaskStore_GetEmpty799=== CONT TestFailedS3DeleteKeepsTombstone800--- PASS: TestGCTaskStore_GetEmpty (0.00s)801=== CONT TestReadProxyConditionalGet802=== CONT TestGCTaskStore_ConflictDifferentParams803=== CONT TestPush_RepeatedRootCommitsOnce804--- PASS: TestGCTaskStore_ConflictDifferentParams (0.00s)805=== CONT TestReadProxyOutlastsServerWriteTimeout806=== CONT TestGCTaskStore_StartNew807--- PASS: TestGCTaskStore_StartNew (0.00s)808=== CONT TestReadProxyHead809=== CONT TestGCEndsOnShutdown810=== CONT TestCommitRacingPendingCleanupKeepsObjects811=== RUN TestGCEndsOnShutdown/before_the_run812=== CONT TestPushWaitsForSweepHoldingRow813=== PAUSE TestGCEndsOnShutdown/before_the_run814=== CONT TestSweepRechecksPendingAfterLocking815=== CONT TestGetExistingObjectsLocksRows816=== CONT TestRegisterCompletedObjectNeedsPendingRow817=== CONT TestSweepSkipsRowsLockedByPush818=== CONT TestCreatePin_ReservedPins819=== CONT TestPush_CompleteCommitsEveryRoot820=== CONT TestPush_OverlappingRootsStoreOneRowPerKey821=== CONT TestReadRedirectUsesPublicS3URL822=== CONT TestReadProxyRangeRequest823=== CONT TestReadRedirectKeepsNarinfoProxied824=== CONT TestReadRedirectNar825=== CONT TestReadProxyDisabled826=== CONT TestReadProxyRootRedirectsToIndexHTML827=== CONT TestAbortSucceeded828--- PASS: TestAbortSucceeded (0.00s)829=== CONT TestGCTaskStore_DeduplicateSameParams830=== RUN TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete831=== CONT TestReadProxyInvalidPath832--- PASS: TestGCTaskStore_DeduplicateSameParams (0.00s)833=== CONT TestReadProxy404834=== PAUSE TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete835=== RUN TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark836=== PAUSE TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark837=== CONT TestReadProxyNarStreaming8382026/11/12 16:57:07 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:64365/oidc8392026/11/12 16:57:08 INFO Received push request method=POST path=/api/pushes840--- PASS: TestPush_OverlappingRootsStoreOneRowPerKey (1.35s)841=== CONT TestReadProxyContentNegotiation842--- PASS: TestReadProxyNarStreaming (1.53s)843=== CONT TestReadProxyNarinfoAlreadyDecompressed8442026/11/12 16:57:08 INFO Received uploads request method=POST path=/api/pending_closures8452026/11/12 16:57:09 INFO Aborted multipart uploads count=0 kept=08462026/11/12 16:57:09 WARN Force mode enabled - objects will be deleted immediately without grace period8472026/11/12 16:57:09 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."8482026/11/12 16:57:09 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."849--- PASS: TestReadProxyConditionalGet (1.88s)850=== CONT TestReadProxyNarinfo851--- PASS: TestFailedS3DeleteKeepsTombstone (1.88s)852=== CONT TestIsValidCachePath853=== RUN TestIsValidCachePath/narinfo854=== PAUSE TestIsValidCachePath/narinfo855=== RUN TestIsValidCachePath/narinfo_all_nix_base32_chars856=== PAUSE TestIsValidCachePath/narinfo_all_nix_base32_chars857=== RUN TestIsValidCachePath/nar_zst858=== PAUSE TestIsValidCachePath/nar_zst859=== RUN TestIsValidCachePath/nar_xz860=== PAUSE TestIsValidCachePath/nar_xz861=== RUN TestIsValidCachePath/nar_bz2862=== PAUSE TestIsValidCachePath/nar_bz2863=== RUN TestIsValidCachePath/nar_uncompressed864=== PAUSE TestIsValidCachePath/nar_uncompressed865=== RUN TestIsValidCachePath/ls866=== PAUSE TestIsValidCachePath/ls867=== RUN TestIsValidCachePath/log868=== PAUSE TestIsValidCachePath/log869=== RUN TestIsValidCachePath/realisation870=== PAUSE TestIsValidCachePath/realisation871=== RUN TestIsValidCachePath/nix-cache-info872=== PAUSE TestIsValidCachePath/nix-cache-info873=== RUN TestIsValidCachePath/index.html874=== PAUSE TestIsValidCachePath/index.html875=== RUN TestIsValidCachePath/pin876=== PAUSE TestIsValidCachePath/pin877=== RUN TestIsValidCachePath/pin_all_alphabet_chars878=== PAUSE TestIsValidCachePath/pin_all_alphabet_chars879=== RUN TestIsValidCachePath/traversal_parent880=== PAUSE TestIsValidCachePath/traversal_parent881=== RUN TestIsValidCachePath/traversal_in_middle882=== PAUSE TestIsValidCachePath/traversal_in_middle883=== RUN TestIsValidCachePath/pin_traversal884=== PAUSE TestIsValidCachePath/pin_traversal885=== RUN TestIsValidCachePath/invalid_char_e886=== PAUSE TestIsValidCachePath/invalid_char_e887=== RUN TestIsValidCachePath/invalid_char_u888=== PAUSE TestIsValidCachePath/invalid_char_u889=== RUN TestIsValidCachePath/random_path890=== PAUSE TestIsValidCachePath/random_path891=== RUN TestIsValidCachePath/empty892=== PAUSE TestIsValidCachePath/empty893=== RUN TestIsValidCachePath/leading_slash894=== PAUSE TestIsValidCachePath/leading_slash895=== RUN TestIsValidCachePath/wrong_extension896=== PAUSE TestIsValidCachePath/wrong_extension897=== RUN TestIsValidCachePath/short_hash898=== PAUSE TestIsValidCachePath/short_hash899=== RUN TestIsValidCachePath/pin_empty_name900=== PAUSE TestIsValidCachePath/pin_empty_name901=== RUN TestIsValidCachePath/pin_nested902=== PAUSE TestIsValidCachePath/pin_nested903=== RUN TestIsValidCachePath/pin_invalid_char904=== PAUSE TestIsValidCachePath/pin_invalid_char905=== CONT TestParseSingleRange906=== RUN TestParseSingleRange/none907=== PAUSE TestParseSingleRange/none908=== RUN TestParseSingleRange/unknown_unit909=== PAUSE TestParseSingleRange/unknown_unit910=== RUN TestParseSingleRange/multi-range_ignored911=== PAUSE TestParseSingleRange/multi-range_ignored912=== RUN TestParseSingleRange/malformed_no_dash913=== PAUSE TestParseSingleRange/malformed_no_dash914=== RUN TestParseSingleRange/malformed_both_empty915=== PAUSE TestParseSingleRange/malformed_both_empty916=== RUN TestParseSingleRange/malformed_end_before_start917=== PAUSE TestParseSingleRange/malformed_end_before_start918=== RUN TestParseSingleRange/closed919=== PAUSE TestParseSingleRange/closed920=== RUN TestParseSingleRange/open-ended921=== PAUSE TestParseSingleRange/open-ended922=== RUN TestParseSingleRange/end_clamped_to_size923=== PAUSE TestParseSingleRange/end_clamped_to_size924=== RUN TestParseSingleRange/suffix925=== PAUSE TestParseSingleRange/suffix926=== RUN TestParseSingleRange/suffix_exceeds_size927=== PAUSE TestParseSingleRange/suffix_exceeds_size928=== RUN TestParseSingleRange/single_byte929=== PAUSE TestParseSingleRange/single_byte930=== RUN TestParseSingleRange/start_past_EOF931=== PAUSE TestParseSingleRange/start_past_EOF932=== RUN TestParseSingleRange/start_far_past_EOF933=== PAUSE TestParseSingleRange/start_far_past_EOF934=== CONT TestPresentNotReportedWhileGCDeletesClosure935=== RUN TestPresentNotReportedWhileGCDeletesClosure/stale936=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/stale937=== RUN TestPresentNotReportedWhileGCDeletesClosure/fresh938=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/fresh939=== CONT TestPresentRefreshesOnlyStaleClosures940--- PASS: TestReadProxyRootRedirectsToIndexHTML (2.01s)941=== CONT TestPresentReportsOnlyClosureRoots9422026/11/12 16:57:09 INFO Received push request method=POST path=/api/pushes9432026/11/12 16:57:09 INFO Received complete push request method=POST path=/api/pushes/1/complete9442026/11/12 16:57:09 INFO Completed push id=1 objects=4 duration=26.524625ms945--- PASS: TestPush_RepeatedRootCommitsOnce (2.21s)946=== CONT TestClientCADerivations947--- PASS: TestReadProxyHead (2.31s)948=== CONT TestClientFallsBackToClosures949--- PASS: TestReadRedirectKeepsNarinfoProxied (2.80s)950=== CONT TestClientPushesUseOnePush951--- PASS: TestGetExistingObjectsLocksRows (3.03s)952=== CONT TestDeleteClosuresRacingPins953--- PASS: TestReadProxyRangeRequest (3.27s)954=== CONT TestForceGCDuringPushOffersSweptObject955=== RUN TestForceGCDuringPushOffersSweptObject/before_pending_rows956=== PAUSE TestForceGCDuringPushOffersSweptObject/before_pending_rows957=== RUN TestForceGCDuringPushOffersSweptObject/after_presence_check958=== PAUSE TestForceGCDuringPushOffersSweptObject/after_presence_check959=== CONT TestCreatePendingClosureVerifyS3FailureReleasesConnection9602026/11/12 16:57:10 INFO Aborted multipart uploads count=0 kept=09612026/11/12 16:57:10 WARN Force mode enabled - objects will be deleted immediately without grace period9622026/11/12 16:57:10 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=09632026/11/12 16:57:10 INFO Vacuumed table table=pending_closures9642026/11/12 16:57:10 INFO Vacuumed table table=commit_receipts9652026/11/12 16:57:10 INFO Vacuumed table table=pending_objects9662026/11/12 16:57:10 INFO Vacuumed table table=multipart_uploads9672026/11/12 16:57:10 INFO Vacuumed table table=closures9682026/11/12 16:57:10 INFO Vacuumed table table=objects969--- PASS: TestCommitRacingPendingCleanupKeepsObjects (3.57s)970=== CONT TestPinProtectsFromGC9712026/11/12 16:57:10 INFO Aborted multipart uploads count=0 kept=09722026/11/12 16:57:10 WARN Force mode enabled - objects will be deleted immediately without grace period9732026/11/12 16:57:10 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=09742026/11/12 16:57:10 INFO Vacuumed table table=pending_closures9752026/11/12 16:57:10 INFO Vacuumed table table=commit_receipts9762026/11/12 16:57:10 INFO Vacuumed table table=pending_objects9772026/11/12 16:57:10 INFO Vacuumed table table=multipart_uploads9782026/11/12 16:57:10 INFO Vacuumed table table=closures9792026/11/12 16:57:10 INFO Vacuumed table table=objects9802026/11/12 16:57:10 INFO Aborted multipart uploads count=0 kept=09812026/11/12 16:57:10 WARN Force mode enabled - objects will be deleted immediately without grace period9822026/11/12 16:57:11 INFO Received uploads request method=POST path=/api/pending_closures9832026/11/12 16:57:11 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=09842026/11/12 16:57:11 INFO Vacuumed table table=pending_closures9852026/11/12 16:57:11 INFO Vacuumed table table=commit_receipts9862026/11/12 16:57:11 INFO Vacuumed table table=pending_objects9872026/11/12 16:57:11 INFO Vacuumed table table=multipart_uploads9882026/11/12 16:57:11 INFO Vacuumed table table=closures9892026/11/12 16:57:11 INFO Vacuumed table table=objects990--- PASS: TestSweepSkipsRowsLockedByPush (3.99s)991=== CONT TestGCSweepDeliversEachKeyOnce992--- PASS: TestReadRedirectUsesPublicS3URL (4.08s)993=== CONT TestClientSharedPathCommittedMidPush994--- PASS: TestReadRedirectNar (4.23s)995=== CONT TestClientWithDependencies9962026/11/12 16:57:11 INFO Received push request method=POST path=/api/pushes997--- PASS: TestPushWaitsForSweepHoldingRow (4.39s)998=== CONT TestClientMultipleUploads9992026/11/12 16:57:11 INFO Received complete push request method=POST path=/api/pushes/1/complete10002026/11/12 16:57:11 INFO Completed push id=1 objects=6 duration=28.245125ms1001--- PASS: TestPush_CompleteCommitsEveryRoot (4.43s)1002=== CONT TestClientIntegration1003--- PASS: TestRegisterCompletedObjectNeedsPendingRow (4.53s)1004=== CONT TestClientErrorHandling1005=== RUN TestClientErrorHandling/InvalidStorePath1006=== PAUSE TestClientErrorHandling/InvalidStorePath1007=== RUN TestClientErrorHandling/InvalidAuthToken1008=== PAUSE TestClientErrorHandling/InvalidAuthToken1009=== RUN TestClientErrorHandling/ServerNotAvailable1010=== PAUSE TestClientErrorHandling/ServerNotAvailable1011=== CONT TestTombstonedObjectOfferedWithoutWaiting1012--- PASS: TestReadProxyDisabled (4.67s)1013=== CONT TestGCSweepSkipsPendingObjects10142026/11/12 16:57:12 INFO Aborted multipart uploads count=0 kept=010152026/11/12 16:57:12 WARN Force mode enabled - objects will be deleted immediately without grace period10162026/11/12 16:57:12 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=010172026/11/12 16:57:12 INFO Vacuumed table table=pending_closures10182026/11/12 16:57:12 INFO Vacuumed table table=commit_receipts10192026/11/12 16:57:12 INFO Vacuumed table table=pending_objects10202026/11/12 16:57:12 INFO Vacuumed table table=multipart_uploads10212026/11/12 16:57:12 INFO Vacuumed table table=closures10222026/11/12 16:57:12 INFO Vacuumed table table=objects1023--- PASS: TestSweepRechecksPendingAfterLocking (5.06s)1024=== CONT TestDeduplicatedObjectsRecordedAsPending10252026/11/12 16:57:12 INFO Received push request method=POST path=/api/pushes1026--- PASS: TestReadProxy404 (5.61s)1027=== CONT TestPushDedupSurvivesConcurrentGC10282026/11/12 16:57:12 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux10292026/11/12 16:57:12 WARN Refused reserved pin name=worker-x86_64-linux10302026/11/12 16:57:12 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux10312026/11/12 16:57:12 INFO Received create pin request method=POST path=/api/pins/my-app10322026/11/12 16:57:12 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux1033--- PASS: TestCreatePin_ReservedPins (5.68s)1034=== CONT TestNARDeduplicationMetadataUploadBug10352026-11-12 16:57:12.967 UTC [8986] ERROR: Push does not exist: id=110362026-11-12 16:57:12.967 UTC [8986] CONTEXT: PL/pgSQL function commit_push(bigint) line 15 at RAISE10372026-11-12 16:57:12.967 UTC [8986] STATEMENT: SELECT commit_push($1)1038--- PASS: TestPush_CompleteWaitsForCleanupOfThePush (5.77s)1039=== CONT TestOrphanedObjectsGCStressTest1040--- PASS: TestReadProxyInvalidPath (5.87s)1041=== CONT TestOrphanedObjectsGC1042--- PASS: TestReadProxyNarinfoAlreadyDecompressed (4.57s)1043=== CONT TestGCMetrics1044--- PASS: TestReadProxyContentNegotiation (5.03s)1045=== CONT TestServerTLSConfig1046=== RUN TestServerTLSConfig/no_client_CA1047=== PAUSE TestServerTLSConfig/no_client_CA1048=== RUN TestServerTLSConfig/missing_CA_file1049=== PAUSE TestServerTLSConfig/missing_CA_file1050=== RUN TestServerTLSConfig/not_a_PEM_file1051=== PAUSE TestServerTLSConfig/not_a_PEM_file1052=== CONT TestGCBugBareHashReferences10532026/11/12 16:57:13 ERROR Failed to decode narinfo key=3hcdxyjf9yiq7qf3i4548drb6sjmwa1v.narinfo limit=16777216 error=<nil>1054--- PASS: TestReadProxyNarinfo (4.69s)1055=== CONT TestService_NativeMTLS1056--- PASS: TestPresentRefreshesOnlyStaleClosures (4.84s)1057=== CONT TestGCAdvisoryLockBlocksConcurrentRun1058--- PASS: TestReadProxyOutlastsServerWriteTimeout (6.78s)1059=== CONT TestMetricsCommit1060--- PASS: TestPresentReportsOnlyClosureRoots (4.90s)1061=== CONT TestMetricsInventory1062=== NAME TestClientCADerivations1063 client_ca_test.go:136: Built CA derivation: /private/var/tb/2/scratch/777606c6228c2b4e/build/TestClientCADerivations2299807744/001/store/9xqpf0jan7fi5v2ws502chqww3awdi1b-ca-test1064 client_ca_test.go:139: Found 1 dependencies (including self)10652026/11/12 16:57:15 INFO Received push request method=POST path=/api/pushes10662026/11/12 16:57:15 INFO Received uploads request method=POST path=/api/pending_closures10672026/11/12 16:57:15 INFO Received uploads request method=POST path=/api/pending_closures10682026/11/12 16:57:15 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)10692026/11/12 16:57:15 INFO Uploading 2 paths to 127.0.0.1 (1 already cached)10702026/11/12 16:57:15 INFO Uploading 9xqpf0jan7fi5v2ws502chqww3awdi1b-ca-test (144B)10712026/11/12 16:57:15 INFO Uploading v64pw693qkbgmcsyckm6kbzhp3ik504x-a (256B)10722026/11/12 16:57:15 INFO Uploading 27liq9vs126mjvd4hfmfkb9bh6w5nclc-shared-dep (136B)10732026/11/12 16:57:15 WARN Failed to register uploaded object key=9xqpf0jan7fi5v2ws502chqww3awdi1b.ls error="server returned 404: 404 page not found\n"10742026/11/12 16:57:15 WARN Failed to register uploaded object key=6k1j67xbramh41yc8pyl1hv0gppk35m1.ls error="server returned 404: 404 page not found\n"10752026/11/12 16:57:15 WARN Failed to register uploaded object key=nar/04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n.nar.zst error="server returned 404: 404 page not found\n"10762026/11/12 16:57:15 WARN Failed to register uploaded object key=27liq9vs126mjvd4hfmfkb9bh6w5nclc.ls error="server returned 404: 404 page not found\n"10772026/11/12 16:57:15 WARN Failed to register uploaded object key=nar/0lidgkdgl09nj2yla2aknr5fy7klipqzvqxbfv41clmvjw4wszx9.nar.zst error="server returned 404: 404 page not found\n"10782026/11/12 16:57:15 WARN Failed to register uploaded object key=log/qnfsdb7lxpq88ayrmhyjca5xr84w7ljk-ca-test.drv error="server returned 404: 404 page not found\n"10792026/11/12 16:57:15 WARN Failed to register uploaded object key=v64pw693qkbgmcsyckm6kbzhp3ik504x.ls error="server returned 404: 404 page not found\n"10802026/11/12 16:57:15 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign10812026/11/12 16:57:15 INFO Signed narinfos id=1 count=110822026/11/12 16:57:15 INFO Uploading 1 narinfos10832026/11/12 16:57:15 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign10842026/11/12 16:57:15 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"10852026/11/12 16:57:15 INFO Signed narinfos id=2 count=210862026/11/12 16:57:15 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign10872026/11/12 16:57:15 INFO Signed narinfos id=1 count=210882026/11/12 16:57:15 INFO Uploading 4 narinfos10892026/11/12 16:57:15 INFO Received complete push request method=POST path=/api/pushes/1/complete10902026/11/12 16:57:15 WARN Failed to register uploaded object key=9xqpf0jan7fi5v2ws502chqww3awdi1b.narinfo error="server returned 404: 404 page not found\n"10912026/11/12 16:57:15 WARN Failed to register uploaded object key=6k1j67xbramh41yc8pyl1hv0gppk35m1.narinfo error="server returned 404: 404 page not found\n"10922026/11/12 16:57:15 WARN Failed to register uploaded object key=27liq9vs126mjvd4hfmfkb9bh6w5nclc.narinfo error="server returned 404: 404 page not found\n"10932026/11/12 16:57:15 WARN Failed to register uploaded object key=v64pw693qkbgmcsyckm6kbzhp3ik504x.narinfo error="server returned 404: 404 page not found\n"10942026/11/12 16:57:15 INFO Completed push id=1 objects=4 duration=54.013208ms10952026/11/12 16:57:15 INFO Upload complete. (405ms)1096 client_ca_test.go:180: Narinfo contains CA field: StorePath: /private/var/tb/2/scratch/777606c6228c2b4e/build/TestClientCADerivations2299807744/001/store/9xqpf0jan7fi5v2ws502chqww3awdi1b-ca-test1097 URL: nar/04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n.nar.zst1098 Compression: zstd1099 NarHash: sha256:04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n1100 NarSize: 1441101 References: 1102 Deriver: /private/var/tb/2/scratch/777606c6228c2b4e/build/TestClientCADerivations2299807744/001/store/qnfsdb7lxpq88ayrmhyjca5xr84w7ljk-ca-test.drv1103 CA: fixed:r:sha256:04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n1104 client_ca_test.go:185: Checking for realisation files in S3...1105 client_ca_test.go:207: No realisation files uploaded - this is expected for locally built CA derivations1106 client_ca_test.go:208: Locally built CA derivations don't have DrvOutput IDs until they're substituted from a cache11072026/11/12 16:57:15 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete11082026/11/12 16:57:15 WARN Failed to register uploaded object key=27liq9vs126mjvd4hfmfkb9bh6w5nclc.narinfo error="server returned 404: 404 page not found\n"11092026/11/12 16:57:15 INFO Completed upload id=2 objects=6 duration=63.339792ms11102026/11/12 16:57:15 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete11112026/11/12 16:57:15 INFO Completed upload id=1 objects=6 duration=708.292µs11122026/11/12 16:57:15 INFO Upload complete. (452ms)1113=== NAME TestClientFallsBackToClosures1114 client_pushes_test.go:112: Retrieved narinfo from S3:1115 StorePath: /private/var/tb/2/scratch/777606c6228c2b4e/build/TestClientFallsBackToClosures3804076751/001/store/27liq9vs126mjvd4hfmfkb9bh6w5nclc-shared-dep1116 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1117 Compression: zstd1118 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821119 NarSize: 1361120 References: 1121 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n1122 client_pushes_test.go:112: Retrieved narinfo from S3:1123 StorePath: /private/var/tb/2/scratch/777606c6228c2b4e/build/TestClientFallsBackToClosures3804076751/001/store/v64pw693qkbgmcsyckm6kbzhp3ik504x-a11242026/11/12 16:57:15 INFO Received push request method=POST path=/api/pushes1125 URL: nar/0lidgkdgl09nj2yla2aknr5fy7klipqzvqxbfv41clmvjw4wszx9.nar.zst1126 Compression: zstd1127 NarHash: sha256:0lidgkdgl09nj2yla2aknr5fy7klipqzvqxbfv41clmvjw4wszx91128 NarSize: 2561129 References: /private/var/tb/2/scratch/777606c6228c2b4e/build/TestClientFallsBackToClosures3804076751/001/store/27liq9vs126mjvd4hfmfkb9bh6w5nclc-shared-dep1130 CA: text:sha256:1fnixcvb0wb3vl3wy1sq28k4y7rslv3zprh2n3m4mf4k60c9qznh1131 client_pushes_test.go:112: Retrieved narinfo from S3:1132 StorePath: /private/var/tb/2/scratch/777606c6228c2b4e/build/TestClientFallsBackToClosures3804076751/001/store/6k1j67xbramh41yc8pyl1hv0gppk35m1-b1133 URL: nar/0lidgkdgl09nj2yla2aknr5fy7klipqzvqxbfv41clmvjw4wszx9.nar.zst1134 Compression: zstd1135 NarHash: sha256:0lidgkdgl09nj2yla2aknr5fy7klipqzvqxbfv41clmvjw4wszx91136 NarSize: 2561137 References: /private/var/tb/2/scratch/777606c6228c2b4e/build/TestClientFallsBackToClosures3804076751/001/store/27liq9vs126mjvd4hfmfkb9bh6w5nclc-shared-dep1138 CA: text:sha256:1fnixcvb0wb3vl3wy1sq28k4y7rslv3zprh2n3m4mf4k60c9qznh11392026/11/12 16:57:15 INFO Uploading 2 paths to 127.0.0.1 (1 already cached)1140=== NAME TestClientCADerivations1141 client_ca_test.go:258: nix copy output: error: binary cache 's3://bucket33?endpoint=http://localhost:64341®ion=eu-west-1' is for Nix stores with prefix '/nix/store', not '/private/var/tb/2/scratch/777606c6228c2b4e/build/TestClientCADerivations2299807744/001/store'1142 client_ca_test.go:259: nix copy failed (might be expected with isolated stores): exit status 111432026/11/12 16:57:15 INFO Uploading psdj0wg38wpbmlssgac7fibb0lzf601r-shared-dep (136B)11442026/11/12 16:57:15 INFO Uploading qlwsmk0gi5wlsz93jjir8ma914b1vg1j-b (256B)11452026/11/12 16:57:15 WARN Failed to register uploaded object key=qlwsmk0gi5wlsz93jjir8ma914b1vg1j.ls error="server returned 404: 404 page not found\n"11462026/11/12 16:57:15 WARN Failed to register uploaded object key=nar/02vhp8mb3819mhnikk4i7ap73bpwap36rmnb2xiln98qqhcgvfig.nar.zst error="server returned 404: 404 page not found\n"11472026/11/12 16:57:15 WARN Failed to register uploaded object key=psdj0wg38wpbmlssgac7fibb0lzf601r.ls error="server returned 404: 404 page not found\n"1148--- PASS: TestClientFallsBackToClosures (6.04s)11492026/11/12 16:57:15 WARN Failed to register uploaded object key=bzvwf56li1dvzm9gqci00m19gwch7iab.ls error="server returned 404: 404 page not found\n"1150=== CONT TestLeadEndsOnShutdown11512026/11/12 16:57:15 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign11522026/11/12 16:57:15 INFO Signed narinfos id=1 count=311532026/11/12 16:57:15 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"11542026/11/12 16:57:15 INFO Uploading 3 narinfos1155--- PASS: TestClientCADerivations (6.15s)1156=== CONT TestLeadElectsOneAndHandsOver11572026/11/12 16:57:15 WARN Failed to register uploaded object key=qlwsmk0gi5wlsz93jjir8ma914b1vg1j.narinfo error="server returned 404: 404 page not found\n"11582026/11/12 16:57:15 INFO Received uploads request method=POST path=/api/pending_closures11592026/11/12 16:57:15 WARN Failed to register uploaded object key=psdj0wg38wpbmlssgac7fibb0lzf601r.narinfo error="server returned 404: 404 page not found\n"11602026/11/12 16:57:15 INFO Received complete push request method=POST path=/api/pushes/1/complete11612026/11/12 16:57:15 WARN Failed to register uploaded object key=bzvwf56li1dvzm9gqci00m19gwch7iab.narinfo error="server returned 404: 404 page not found\n"11622026/11/12 16:57:15 INFO Completed push id=1 objects=8 duration=36.022041ms11632026/11/12 16:57:15 INFO Upload complete. (292ms)1164=== NAME TestClientPushesUseOnePush1165 client_pushes_test.go:97: Retrieved narinfo from S3:1166 StorePath: /private/var/tb/2/scratch/777606c6228c2b4e/build/TestClientPushesUseOnePush977250347/001/store/psdj0wg38wpbmlssgac7fibb0lzf601r-shared-dep1167 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1168 Compression: zstd1169 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821170 NarSize: 1361171 References: 1172 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n1173 client_pushes_test.go:97: Retrieved narinfo from S3:1174 StorePath: /private/var/tb/2/scratch/777606c6228c2b4e/build/TestClientPushesUseOnePush977250347/001/store/bzvwf56li1dvzm9gqci00m19gwch7iab-a1175 URL: nar/02vhp8mb3819mhnikk4i7ap73bpwap36rmnb2xiln98qqhcgvfig.nar.zst1176 Compression: zstd1177 NarHash: sha256:02vhp8mb3819mhnikk4i7ap73bpwap36rmnb2xiln98qqhcgvfig1178 NarSize: 2561179 References: /private/var/tb/2/scratch/777606c6228c2b4e/build/TestClientPushesUseOnePush977250347/001/store/psdj0wg38wpbmlssgac7fibb0lzf601r-shared-dep1180 CA: text:sha256:0p56w3rwidcsjxzyqrma83pqa3jr4vq8krrkrvgr5ckxg73b3iak1181 client_pushes_test.go:97: Retrieved narinfo from S3:1182 StorePath: /private/var/tb/2/scratch/777606c6228c2b4e/build/TestClientPushesUseOnePush977250347/001/store/qlwsmk0gi5wlsz93jjir8ma914b1vg1j-b1183 URL: nar/02vhp8mb3819mhnikk4i7ap73bpwap36rmnb2xiln98qqhcgvfig.nar.zst1184 Compression: zstd1185 NarHash: sha256:02vhp8mb3819mhnikk4i7ap73bpwap36rmnb2xiln98qqhcgvfig1186 NarSize: 2561187 References: /private/var/tb/2/scratch/777606c6228c2b4e/build/TestClientPushesUseOnePush977250347/001/store/psdj0wg38wpbmlssgac7fibb0lzf601r-shared-dep1188 CA: text:sha256:0p56w3rwidcsjxzyqrma83pqa3jr4vq8krrkrvgr5ckxg73b3iak1189--- PASS: TestClientPushesUseOnePush (5.75s)1190=== CONT TestConnectWaitsForAPeerMigration1191--- PASS: TestCreatePendingClosureVerifyS3FailureReleasesConnection (5.38s)1192=== CONT TestResolveDBConnectionString1193=== RUN TestResolveDBConnectionString/flag_wins1194=== PAUSE TestResolveDBConnectionString/flag_wins1195=== RUN TestResolveDBConnectionString/file_when_flag_empty1196=== PAUSE TestResolveDBConnectionString/file_when_flag_empty1197=== RUN TestResolveDBConnectionString/missing_file_is_an_error1198=== PAUSE TestResolveDBConnectionString/missing_file_is_an_error1199=== RUN TestResolveDBConnectionString/PGHOST_allows_empty1200=== PAUSE TestResolveDBConnectionString/PGHOST_allows_empty1201=== RUN TestResolveDBConnectionString/nothing_configured1202=== PAUSE TestResolveDBConnectionString/nothing_configured1203=== CONT TestLeadIncumbentAnnouncesAtOnce1204=== NAME TestPinProtectsFromGC1205 client_integration_test.go:791: Pinned store path: /private/var/tb/2/scratch/777606c6228c2b4e/build/TestPinProtectsFromGC1790194730/001/store/5jgxy6bz9v78nr1qwr9brv1caf1xhxc1-pinned-file.txt1206 client_integration_test.go:792: Unpinned store path: /private/var/tb/2/scratch/777606c6228c2b4e/build/TestPinProtectsFromGC1790194730/001/store/v3b27z0bik67za61s88wz46pmi8yzdja-unpinned-file.txt12072026/11/12 16:57:16 INFO Aborted multipart uploads count=0 kept=012082026/11/12 16:57:16 INFO Received push request method=POST path=/api/pushes12092026/11/12 16:57:16 WARN Force mode enabled - objects will be deleted immediately without grace period12102026/11/12 16:57:16 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)12112026/11/12 16:57:16 INFO Uploading 5jgxy6bz9v78nr1qwr9brv1caf1xhxc1-pinned-file.txt (128B)12122026/11/12 16:57:16 WARN Failed to register uploaded object key=5jgxy6bz9v78nr1qwr9brv1caf1xhxc1.ls error="server returned 404: 404 page not found\n"12132026/11/12 16:57:16 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign12142026/11/12 16:57:16 WARN Failed to register uploaded object key=nar/0qmhsg42pqsz7x6hh02j8gznnac3yi3qpjhrqbwvz88i194jbb5s.nar.zst error="server returned 404: 404 page not found\n"12152026/11/12 16:57:16 INFO Signed narinfos id=1 count=112162026/11/12 16:57:16 INFO Uploading 1 narinfos12172026/11/12 16:57:16 INFO Received complete push request method=POST path=/api/pushes/1/complete12182026/11/12 16:57:16 WARN Failed to register uploaded object key=5jgxy6bz9v78nr1qwr9brv1caf1xhxc1.narinfo error="server returned 404: 404 page not found\n"12192026/11/12 16:57:16 INFO Completed push id=1 objects=3 duration=27.031167ms12202026/11/12 16:57:16 INFO Upload complete. (226ms)12212026/11/12 16:57:16 INFO Received push request method=POST path=/api/pushes12222026/11/12 16:57:16 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)12232026/11/12 16:57:16 INFO Uploading v3b27z0bik67za61s88wz46pmi8yzdja-unpinned-file.txt (128B)12242026/11/12 16:57:16 WARN Failed to register uploaded object key=v3b27z0bik67za61s88wz46pmi8yzdja.ls error="server returned 404: 404 page not found\n"12252026/11/12 16:57:16 INFO Received sign narinfos request method=POST path=/api/pushes/2/sign12262026/11/12 16:57:16 WARN Failed to register uploaded object key=nar/0gh0wdnr6gmk4r5sfzzh12isnl14rgmkrwy1ws41hpsrlndclg5h.nar.zst error="server returned 404: 404 page not found\n"12272026/11/12 16:57:16 INFO Signed narinfos id=2 count=112282026/11/12 16:57:16 INFO Uploading 1 narinfos12292026/11/12 16:57:16 INFO Received complete push request method=POST path=/api/pushes/2/complete12302026/11/12 16:57:16 WARN Failed to register uploaded object key=v3b27z0bik67za61s88wz46pmi8yzdja.narinfo error="server returned 404: 404 page not found\n"12312026/11/12 16:57:16 INFO Completed push id=2 objects=3 duration=1.702ms12322026/11/12 16:57:16 INFO Upload complete. (244ms)12332026/11/12 16:57:16 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=012342026/11/12 16:57:16 INFO Vacuumed table table=pending_closures12352026/11/12 16:57:16 INFO Vacuumed table table=commit_receipts12362026/11/12 16:57:16 INFO Vacuumed table table=pending_objects12372026/11/12 16:57:16 INFO Vacuumed table table=multipart_uploads12382026/11/12 16:57:16 INFO Vacuumed table table=closures12392026/11/12 16:57:16 INFO Starting cleanup of old closures method=DELETE path=/api/closures12402026/11/12 16:57:16 INFO Garbage collection started12412026/11/12 16:57:16 INFO Aborted multipart uploads count=0 kept=012422026/11/12 16:57:16 INFO Vacuumed table table=objects1243--- PASS: TestGCSweepDeliversEachKeyOnce (5.59s)1244=== CONT TestConcurrentCommitsSharingObjectsDoNotDeadlock12452026/11/12 16:57:16 WARN Force mode enabled - objects will be deleted immediately without grace period12462026/11/12 16:57:16 INFO Received push request method=POST path=/api/pushes12472026/11/12 16:57:16 INFO Uploading 2 paths to 127.0.0.1 (0 already cached)12482026/11/12 16:57:16 INFO Uploading r6by6qnyqjmv98ypvvv22f69gnph07w4-top (264B)12492026/11/12 16:57:16 INFO Uploading 3nvaf2rffj93mr9jm58acvjlwfmg1xv5-shared-dep (136B)12502026/11/12 16:57:17 WARN Failed to register uploaded object key=r6by6qnyqjmv98ypvvv22f69gnph07w4.ls error="server returned 404: 404 page not found\n"12512026/11/12 16:57:17 WARN Failed to register uploaded object key=nar/0asi2bla5girh5rmn108vrhc0am0l5z48l3xpdzrdk4zh6qmqy1n.nar.zst error="server returned 404: 404 page not found\n"12522026/11/12 16:57:17 WARN Failed to register uploaded object key=3nvaf2rffj93mr9jm58acvjlwfmg1xv5.ls error="server returned 404: 404 page not found\n"12532026/11/12 16:57:17 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign12542026/11/12 16:57:17 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"12552026/11/12 16:57:17 INFO Signed narinfos id=1 count=212562026/11/12 16:57:17 INFO Uploading 2 narinfos12572026/11/12 16:57:17 WARN Failed to register uploaded object key=3nvaf2rffj93mr9jm58acvjlwfmg1xv5.narinfo error="server returned 404: 404 page not found\n"12582026/11/12 16:57:17 INFO Received complete push request method=POST path=/api/pushes/1/complete12592026/11/12 16:57:17 WARN Failed to register uploaded object key=r6by6qnyqjmv98ypvvv22f69gnph07w4.narinfo error="server returned 404: 404 page not found\n"12602026/11/12 16:57:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=01261=== NAME TestClientWithDependencies1262 client_integration_test.go:661: Built derivation: /private/var/tb/2/scratch/777606c6228c2b4e/build/TestClientWithDependencies320519898/001/store/6svvkd3zgk7jmkjw62355l0ff4vcrmv0-test-script12632026/11/12 16:57:17 INFO Completed push id=1 objects=6 duration=32.133834ms12642026/11/12 16:57:17 INFO Upload complete. (324ms)1265=== NAME TestClientSharedPathCommittedMidPush1266 client_integration_test.go:740: Retrieved narinfo from S3:1267 StorePath: /private/var/tb/2/scratch/777606c6228c2b4e/build/TestClientSharedPathCommittedMidPush194340037/001/store/3nvaf2rffj93mr9jm58acvjlwfmg1xv5-shared-dep1268 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1269 Compression: zstd1270 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821271 NarSize: 1361272 References: 1273 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n1274 client_integration_test.go:740: Retrieved narinfo from S3:1275 StorePath: /private/var/tb/2/scratch/777606c6228c2b4e/build/TestClientSharedPathCommittedMidPush194340037/001/store/r6by6qnyqjmv98ypvvv22f69gnph07w4-top1276 URL: nar/0asi2bla5girh5rmn108vrhc0am0l5z48l3xpdzrdk4zh6qmqy1n.nar.zst1277 Compression: zstd1278 NarHash: sha256:0asi2bla5girh5rmn108vrhc0am0l5z48l3xpdzrdk4zh6qmqy1n1279 NarSize: 2641280 References: /private/var/tb/2/scratch/777606c6228c2b4e/build/TestClientSharedPathCommittedMidPush194340037/001/store/3nvaf2rffj93mr9jm58acvjlwfmg1xv5-shared-dep1281 CA: text:sha256:1riw4b2r8iyrpy4rkq5w8j67jgh4wschn3hkzqsgigdjyif919sd12822026/11/12 16:57:17 INFO Vacuumed table table=pending_closures1283=== NAME TestClientWithDependencies1284 client_integration_test.go:663: Found 1 dependencies (including self)12852026/11/12 16:57:17 INFO Vacuumed table table=commit_receipts12862026/11/12 16:57:17 INFO Vacuumed table table=pending_objects12872026/11/12 16:57:17 INFO Vacuumed table table=multipart_uploads1288--- PASS: TestClientSharedPathCommittedMidPush (6.03s)1289=== CONT TestLeadUnannouncedStandbyYieldsToIncumbent12902026/11/12 16:57:17 INFO Vacuumed table table=closures1291=== NAME TestClientMultipleUploads1292 client_integration_test.go:406: Created store path 0: /private/var/tb/2/scratch/777606c6228c2b4e/build/TestClientMultipleUploads1250708395/001/store/ivvs5i4n7c87zlqricnjdaqdrh0948r5-test-file-0.txt12932026/11/12 16:57:17 INFO Vacuumed table table=objects12942026/11/12 16:57:17 INFO Received push request method=POST path=/api/pushes12952026/11/12 16:57:17 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)12962026/11/12 16:57:17 INFO Uploading 6svvkd3zgk7jmkjw62355l0ff4vcrmv0-test-script (136B)1297 client_integration_test.go:406: Created store path 1: /private/var/tb/2/scratch/777606c6228c2b4e/build/TestClientMultipleUploads1250708395/001/store/csck42n61h2yqg0r6dqv99dnndmcr3ix-test-file-1.txt12982026/11/12 16:57:17 WARN Failed to register uploaded object key=nar/1s7yghixzpinyqm7g59kyn3sav81pmnsdi5v63m6x4r2vacy907j.nar.zst error="server returned 404: 404 page not found\n"12992026/11/12 16:57:17 WARN Failed to register uploaded object key=6svvkd3zgk7jmkjw62355l0ff4vcrmv0.ls error="server returned 404: 404 page not found\n"13002026/11/12 16:57:17 WARN Failed to register uploaded object key=log/53n54avssmhvmj328v2imkybl5ajvhir-test-script.drv error="server returned 404: 404 page not found\n"13012026/11/12 16:57:17 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign13022026/11/12 16:57:17 INFO Signed narinfos id=1 count=113032026/11/12 16:57:17 INFO Uploading 1 narinfos13042026/11/12 16:57:17 INFO Received complete push request method=POST path=/api/pushes/1/complete13052026/11/12 16:57:17 WARN Failed to register uploaded object key=6svvkd3zgk7jmkjw62355l0ff4vcrmv0.narinfo error="server returned 404: 404 page not found\n"13062026/11/12 16:57:17 INFO Completed push id=1 objects=4 duration=36.073208ms13072026/11/12 16:57:17 INFO Upload complete. (299ms)1308 client_integration_test.go:406: Created store path 2: /private/var/tb/2/scratch/777606c6228c2b4e/build/TestClientMultipleUploads1250708395/001/store/x7sny6ynldl764689rf75f7wl11wvxnf-test-file-2.txt13092026/11/12 16:57:17 INFO Aborted multipart uploads count=0 kept=013102026/11/12 16:57:17 WARN Force mode enabled - objects will be deleted immediately without grace period13112026/11/12 16:57:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=013122026/11/12 16:57:17 INFO Vacuumed table table=pending_closures1313=== NAME TestClientIntegration1314 client_integration_test.go:292: Created store path: /private/var/tb/2/scratch/777606c6228c2b4e/build/TestClientIntegration3937436297/002/store/ng4rngms2cgdqcx09rk7wafn1mjsr7m0-test-file.txt13152026/11/12 16:57:17 INFO Vacuumed table table=commit_receipts13162026/11/12 16:57:17 INFO Received uploads request method=POST path=/api/pending_closures13172026/11/12 16:57:17 INFO Received push request method=POST path=/api/pushes13182026/11/12 16:57:17 INFO Vacuumed table table=pending_objects13192026/11/12 16:57:17 INFO Vacuumed table table=multipart_uploads13202026/11/12 16:57:17 INFO Vacuumed table table=closures13212026/11/12 16:57:17 INFO Uploading 3 paths to 127.0.0.1 (0 already cached)1322--- PASS: TestTombstonedObjectOfferedWithoutWaiting (6.05s)1323=== CONT TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle13242026/11/12 16:57:17 INFO Uploading csck42n61h2yqg0r6dqv99dnndmcr3ix-test-file-1.txt (160B)13252026/11/12 16:57:17 INFO Uploading ivvs5i4n7c87zlqricnjdaqdrh0948r5-test-file-0.txt (160B)13262026/11/12 16:57:17 INFO Uploading x7sny6ynldl764689rf75f7wl11wvxnf-test-file-2.txt (160B)13272026/11/12 16:57:17 INFO Vacuumed table table=objects13282026/11/12 16:57:17 INFO Received push request method=POST path=/api/pushes1329=== NAME TestClientWithDependencies1330 client_integration_test.go:677: Skipping nix copy test - isolated store (/private/var/tb/2/scratch/777606c6228c2b4e/build/TestClientWithDependencies320519898/001/store) requires matching store prefix13312026/11/12 16:57:17 WARN Failed to register uploaded object key=x7sny6ynldl764689rf75f7wl11wvxnf.ls error="server returned 404: 404 page not found\n"13322026/11/12 16:57:17 WARN Failed to register uploaded object key=csck42n61h2yqg0r6dqv99dnndmcr3ix.ls error="server returned 404: 404 page not found\n"13332026/11/12 16:57:17 WARN Failed to register uploaded object key=nar/13p9pqi0ga6fx2apcwls9hjnv7nmf740ggpn5zvfzdsp5vz2bbg4.nar.zst error="server returned 404: 404 page not found\n"13342026/11/12 16:57:17 WARN Failed to register uploaded object key=nar/0ahq3izw3h0b0c08p3xdfclh758fwxnp85ccvflasb4vnfsv50ri.nar.zst error="server returned 404: 404 page not found\n"13352026/11/12 16:57:17 WARN Failed to register uploaded object key=ivvs5i4n7c87zlqricnjdaqdrh0948r5.ls error="server returned 404: 404 page not found\n"13362026/11/12 16:57:17 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)13372026/11/12 16:57:17 INFO Uploading ng4rngms2cgdqcx09rk7wafn1mjsr7m0-test-file.txt (152B)13382026/11/12 16:57:17 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign13392026/11/12 16:57:17 WARN Failed to register uploaded object key=nar/1823g6vx4q5zqvr73m7lkq1wpk40fin7bsc792di430yk7sg4lka.nar.zst error="server returned 404: 404 page not found\n"13402026/11/12 16:57:17 INFO Signed narinfos id=1 count=313412026/11/12 16:57:17 INFO Uploading 3 narinfos13422026/11/12 16:57:17 WARN Failed to register uploaded object key=nar/11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1.nar.zst error="server returned 404: 404 page not found\n"13432026/11/12 16:57:17 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign13442026/11/12 16:57:17 WARN Failed to register uploaded object key=ng4rngms2cgdqcx09rk7wafn1mjsr7m0.ls error="server returned 404: 404 page not found\n"13452026/11/12 16:57:17 INFO Signed narinfos id=1 count=113462026/11/12 16:57:17 INFO Uploading 1 narinfos1347--- PASS: TestClientWithDependencies (6.53s)1348=== CONT TestCompleteMultipartUpload_ErrorButObjectExists13492026/11/12 16:57:17 WARN Failed to register uploaded object key=csck42n61h2yqg0r6dqv99dnndmcr3ix.narinfo error="server returned 404: 404 page not found\n"13502026/11/12 16:57:17 WARN Failed to register uploaded object key=ivvs5i4n7c87zlqricnjdaqdrh0948r5.narinfo error="server returned 404: 404 page not found\n"13512026/11/12 16:57:17 INFO Received complete push request method=POST path=/api/pushes/1/complete13522026/11/12 16:57:17 WARN Failed to register uploaded object key=x7sny6ynldl764689rf75f7wl11wvxnf.narinfo error="server returned 404: 404 page not found\n"13532026/11/12 16:57:18 INFO Received complete push request method=POST path=/api/pushes/1/complete13542026/11/12 16:57:18 WARN Failed to register uploaded object key=ng4rngms2cgdqcx09rk7wafn1mjsr7m0.narinfo error="server returned 404: 404 page not found\n"13552026/11/12 16:57:18 INFO Completed push id=1 objects=9 duration=28.46125ms13562026/11/12 16:57:18 INFO Upload complete. (360ms)1357=== NAME TestClientMultipleUploads1358 client_integration_test.go:417: Uploaded 3 paths in 393.898209ms13592026/11/12 16:57:18 INFO Completed push id=1 objects=3 duration=24.666333ms13602026/11/12 16:57:18 INFO Upload complete. (298ms)1361--- PASS: TestClientMultipleUploads (6.49s)1362=== CONT TestCreatePendingClosure_SmallNARUsesSimplePUT13632026/11/12 16:57:18 INFO Received uploads request method=POST path=/api/pending_closures13642026/11/12 16:57:18 INFO All 1 paths already cached1365=== NAME TestClientIntegration1366 client_integration_test.go:318: Retrieved narinfo from S3:1367 StorePath: /private/var/tb/2/scratch/777606c6228c2b4e/build/TestClientIntegration3937436297/002/store/ng4rngms2cgdqcx09rk7wafn1mjsr7m0-test-file.txt1368 URL: nar/11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1.nar.zst1369 Compression: zstd1370 NarHash: sha256:11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk11371 NarSize: 1521372 References: 1373 CA: fixed:r:sha256:11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk11374 client_integration_test.go:319: Retrieved .ls file from S3 (compressed size: 77 bytes)1375 client_integration_test.go:319: Decompressed .ls content (64 bytes):1376 {"version":1,"root":{"type":"regular","size":39,"narOffset":96}}13772026/11/12 16:57:18 INFO Aborted multipart uploads count=0 kept=013782026/11/12 16:57:18 WARN Force mode enabled - objects will be deleted immediately without grace period13792026/11/12 16:57:18 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=013802026/11/12 16:57:18 INFO Vacuumed table table=pending_closures13812026/11/12 16:57:18 INFO Vacuumed table table=commit_receipts13822026/11/12 16:57:18 INFO Vacuumed table table=pending_objects13832026/11/12 16:57:18 INFO Vacuumed table table=multipart_uploads13842026/11/12 16:57:18 INFO Vacuumed table table=closures13852026/11/12 16:57:18 INFO Vacuumed table table=objects1386--- PASS: TestGCSweepSkipsPendingObjects (6.52s)1387=== CONT TestCompleteMultipartUnregistered13882026/11/12 16:57:18 INFO Received uploads request method=POST path=/api/pending_closures13892026/11/12 16:57:18 INFO Received push request method=POST path=/api/pushes13902026/11/12 16:57:18 INFO Object in database but missing from S3 key=ng4rngms2cgdqcx09rk7wafn1mjsr7m0.ls13912026/11/12 16:57:18 WARN Found objects in DB but missing from S3, will re-upload count=113922026/11/12 16:57:18 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)1393--- PASS: TestDeduplicatedObjectsRecordedAsPending (6.16s)1394=== CONT TestService_verifyS3ReuploadsStaleNarinfo1395=== RUN TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed1396=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed1397=== RUN TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed1398=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed1399=== RUN TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost1400=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost1401=== RUN TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed1402=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed1403=== RUN TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed1404=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed1405=== CONT TestSkippedUploadsHandler14062026/11/12 16:57:18 INFO Client skipped oversized paths paths=3 nar_bytes=50000000001407--- PASS: TestSkippedUploadsHandler (0.00s)1408=== CONT TestService_verifyS3Integrity14092026/11/12 16:57:18 INFO Received complete push request method=POST path=/api/pushes/2/complete14102026/11/12 16:57:18 WARN Failed to register uploaded object key=ng4rngms2cgdqcx09rk7wafn1mjsr7m0.ls error="server returned 404: 404 page not found\n"14112026/11/12 16:57:18 INFO Completed push id=2 objects=3 duration=1.137375ms14122026/11/12 16:57:18 INFO Upload complete. (103ms)1413=== NAME TestClientIntegration1414 client_integration_test.go:336: Retrieved .ls file from S3 (compressed size: 62 bytes)1415 client_integration_test.go:336: Decompressed .ls content (49 bytes):1416 {"version":1,"root":{"type":"regular","size":39}}14172026/11/12 16:57:18 INFO Received push request method=POST path=/api/pushes14182026/11/12 16:57:18 INFO Garbage collection progress phase="" failed_uploads_deleted=0 old_closures_deleted=1 objects_marked=3 objects_deleted=3 objects_failed=014192026/11/12 16:57:18 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)14202026/11/12 16:57:18 INFO Received create pin request method=POST path=/api/pins/myapp14212026/11/12 16:57:18 INFO Received uploads request method=POST path=/api/pending_closures14222026/11/12 16:57:18 INFO Received complete push request method=POST path=/api/pushes/3/complete14232026/11/12 16:57:18 WARN Failed to register uploaded object key=ng4rngms2cgdqcx09rk7wafn1mjsr7m0.ls error="server returned 404: 404 page not found\n"14242026/11/12 16:57:18 INFO Completed push id=3 objects=3 duration=580.417µs14252026/11/12 16:57:18 INFO Upload complete. (133ms)1426 client_integration_test.go:353: Retrieved .ls file from S3 (compressed size: 62 bytes)1427 client_integration_test.go:353: Decompressed .ls content (49 bytes):1428 {"version":1,"root":{"type":"regular","size":39}}14292026/11/12 16:57:18 INFO Created/updated pin name=myapp store_path=/private/var/tb/2/scratch/777606c6228c2b4e/build/TestPinProtectsFromGC1790194730/001/store/5jgxy6bz9v78nr1qwr9brv1caf1xhxc1-pinned-file.txt narinfo_key=5jgxy6bz9v78nr1qwr9brv1caf1xhxc1.narinfo1430 client_integration_test.go:364: Testing garbage collection...14312026/11/12 16:57:18 INFO Starting cleanup of old closures method=DELETE path=/api/closures14322026/11/12 16:57:18 INFO Garbage collection started14332026/11/12 16:57:18 INFO Aborted multipart uploads count=0 kept=014342026/11/12 16:57:18 WARN Force mode enabled - objects will be deleted immediately without grace period14352026/11/12 16:57:18 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=014362026/11/12 16:57:18 INFO Vacuumed table table=pending_closures14372026/11/12 16:57:18 INFO Vacuumed table table=commit_receipts14382026/11/12 16:57:18 INFO Vacuumed table table=pending_objects14392026/11/12 16:57:18 INFO Vacuumed table table=multipart_uploads14402026/11/12 16:57:18 INFO Starting cleanup of old closures method=DELETE path=/api/closures14412026/11/12 16:57:18 INFO Garbage collection started14422026/11/12 16:57:18 INFO Vacuumed table table=closures14432026/11/12 16:57:18 INFO Vacuumed table table=objects14442026/11/12 16:57:18 INFO Aborted multipart uploads count=0 kept=014452026/11/12 16:57:18 WARN Force mode enabled - objects will be deleted immediately without grace period14462026/11/12 16:57:18 INFO Aborted multipart uploads count=0 kept=014472026/11/12 16:57:19 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=014482026/11/12 16:57:19 INFO Vacuumed table table=pending_closures14492026/11/12 16:57:19 INFO Vacuumed table table=commit_receipts14502026/11/12 16:57:19 INFO Vacuumed table table=pending_objects14512026/11/12 16:57:19 INFO Vacuumed table table=multipart_uploads14522026/11/12 16:57:19 INFO Vacuumed table table=closures14532026/11/12 16:57:19 INFO Vacuumed table table=objects14542026/11/12 16:57:19 INFO Aborted multipart uploads count=0 kept=014552026/11/12 16:57:19 WARN Force mode enabled - objects will be deleted immediately without grace period14562026/11/12 16:57:19 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=014572026/11/12 16:57:19 INFO Vacuumed table table=pending_closures14582026/11/12 16:57:19 INFO Vacuumed table table=commit_receipts14592026/11/12 16:57:19 INFO Vacuumed table table=pending_objects14602026/11/12 16:57:19 INFO Vacuumed table table=multipart_uploads14612026/11/12 16:57:19 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=014622026/11/12 16:57:19 INFO Vacuumed table table=closures14632026/11/12 16:57:19 INFO Vacuumed table table=objects1464--- PASS: TestPushDedupSurvivesConcurrentGC (6.61s)1465=== CONT TestService_createPendingClosureHandler1466=== NAME TestNARDeduplicationMetadataUploadBug1467 metadata_upload_test.go:48: First store path: /private/var/tb/2/scratch/777606c6228c2b4e/build/TestNARDeduplicationMetadataUploadBug3254163285/001/store/qcijqxhvvdvgn2ja4h2hak70dfg8wam2-file1.txt14682026/11/12 16:57:19 INFO Vacuumed table table=pending_closures14692026/11/12 16:57:19 INFO Vacuumed table table=commit_receipts14702026/11/12 16:57:19 INFO Vacuumed table table=pending_objects14712026/11/12 16:57:19 INFO Vacuumed table table=multipart_uploads14722026/11/12 16:57:19 INFO Received push request method=POST path=/api/pushes14732026/11/12 16:57:19 INFO Vacuumed table table=closures14742026/11/12 16:57:19 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)14752026/11/12 16:57:19 INFO Uploading qcijqxhvvdvgn2ja4h2hak70dfg8wam2-file1.txt (160B)14762026/11/12 16:57:19 INFO Vacuumed table table=objects14772026/11/12 16:57:19 WARN Failed to register uploaded object key=nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst error="server returned 404: 404 page not found\n"14782026/11/12 16:57:19 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign14792026/11/12 16:57:19 WARN Failed to register uploaded object key=qcijqxhvvdvgn2ja4h2hak70dfg8wam2.ls error="server returned 404: 404 page not found\n"14802026/11/12 16:57:19 INFO Signed narinfos id=1 count=114812026/11/12 16:57:19 INFO Uploading 1 narinfos14822026/11/12 16:57:19 INFO Received complete push request method=POST path=/api/pushes/1/complete14832026/11/12 16:57:19 WARN Failed to register uploaded object key=qcijqxhvvdvgn2ja4h2hak70dfg8wam2.narinfo error="server returned 404: 404 page not found\n"14842026/11/12 16:57:19 INFO Completed push id=1 objects=3 duration=60.295209ms14852026/11/12 16:57:19 INFO Upload complete. (345ms)1486 metadata_upload_test.go:54: Retrieved narinfo from S3:1487 StorePath: /private/var/tb/2/scratch/777606c6228c2b4e/build/TestNARDeduplicationMetadataUploadBug3254163285/001/store/qcijqxhvvdvgn2ja4h2hak70dfg8wam2-file1.txt1488 URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst1489 Compression: zstd1490 NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1491 NarSize: 1601492 References: 1493 CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1494 metadata_upload_test.go:55: Retrieved .ls file from S3 (compressed size: 77 bytes)1495 metadata_upload_test.go:55: Decompressed .ls content (64 bytes):1496 {"version":1,"root":{"type":"regular","size":44,"narOffset":96}}1497 metadata_upload_test.go:64: Second store path (same content): /private/var/tb/2/scratch/777606c6228c2b4e/build/TestNARDeduplicationMetadataUploadBug3254163285/001/store/nng88b0ng5hj3f7mzwim9x2k60rcz1kr-file2.txt1498--- PASS: TestDeleteClosuresRacingPins (9.74s)1499=== CONT TestParseSize1500--- PASS: TestParseSize (0.00s)1501=== CONT TestService_cleanupPendingClosuresHandler15022026/11/12 16:57:20 INFO Received push request method=POST path=/api/pushes15032026/11/12 16:57:20 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)15042026/11/12 16:57:20 INFO Received sign narinfos request method=POST path=/api/pushes/2/sign15052026/11/12 16:57:20 WARN Failed to register uploaded object key=nng88b0ng5hj3f7mzwim9x2k60rcz1kr.ls error="server returned 404: 404 page not found\n"15062026/11/12 16:57:20 INFO Aborted multipart uploads count=0 kept=015072026/11/12 16:57:20 INFO Signed narinfos id=2 count=115082026/11/12 16:57:20 INFO Uploading 1 narinfos15092026/11/12 16:57:20 WARN Force mode enabled - objects will be deleted immediately without grace period15102026/11/12 16:57:20 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=015112026/11/12 16:57:20 INFO Vacuumed table table=pending_closures15122026/11/12 16:57:20 INFO Vacuumed table table=commit_receipts15132026/11/12 16:57:20 INFO Vacuumed table table=pending_objects15142026/11/12 16:57:20 INFO Vacuumed table table=multipart_uploads15152026/11/12 16:57:20 INFO Vacuumed table table=closures15162026/11/12 16:57:20 INFO Vacuumed table table=objects15172026/11/12 16:57:20 INFO Received complete push request method=POST path=/api/pushes/2/complete15182026/11/12 16:57:20 WARN Failed to register uploaded object key=nng88b0ng5hj3f7mzwim9x2k60rcz1kr.narinfo error="server returned 404: 404 page not found\n"1519--- PASS: TestGCMetrics (6.82s)1520=== CONT TestUploadHandlersRejectOversizedBody15212026/11/12 16:57:20 INFO Completed push id=2 objects=3 duration=3.758375ms15222026/11/12 16:57:20 INFO Upload complete. (134ms)1523=== NAME TestNARDeduplicationMetadataUploadBug1524 metadata_upload_test.go:76: Retrieved narinfo from S3:1525 StorePath: /private/var/tb/2/scratch/777606c6228c2b4e/build/TestNARDeduplicationMetadataUploadBug3254163285/001/store/nng88b0ng5hj3f7mzwim9x2k60rcz1kr-file2.txt1526 URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst1527 Compression: zstd1528 NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1529 NarSize: 1601530 References: 1531 CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1532 metadata_upload_test.go:77: Retrieved .ls file from S3 (compressed size: 62 bytes)1533 metadata_upload_test.go:77: Decompressed .ls content (49 bytes):1534 {"version":1,"root":{"type":"regular","size":44}}1535=== RUN TestUploadHandlersRejectOversizedBody/create_pending_closure1536=== PAUSE TestUploadHandlersRejectOversizedBody/create_pending_closure1537=== RUN TestUploadHandlersRejectOversizedBody/complete_multipart1538=== PAUSE TestUploadHandlersRejectOversizedBody/complete_multipart1539=== RUN TestUploadHandlersRejectOversizedBody/request_more_parts1540=== PAUSE TestUploadHandlersRejectOversizedBody/request_more_parts1541=== CONT TestService_Rustfstest1542--- PASS: TestNARDeduplicationMetadataUploadBug (7.29s)1543=== CONT TestUploadHandlersRejectInvalidKeys1544=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info1545=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info1546=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal1547=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal1548=== RUN TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key1549=== PAUSE TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key1550=== RUN TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key1551=== PAUSE TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key1552=== CONT TestPresignedUploadRegisteredBeforeCommit1553--- PASS: TestGCBugBareHashReferences (7.11s)1554=== CONT TestIsValidUploadKey1555=== RUN TestIsValidUploadKey/narinfo1556=== PAUSE TestIsValidUploadKey/narinfo1557=== RUN TestIsValidUploadKey/nar_zst1558=== PAUSE TestIsValidUploadKey/nar_zst1559=== RUN TestIsValidUploadKey/nar_xz1560=== PAUSE TestIsValidUploadKey/nar_xz1561=== RUN TestIsValidUploadKey/nar_plain1562=== PAUSE TestIsValidUploadKey/nar_plain1563=== RUN TestIsValidUploadKey/listing1564=== PAUSE TestIsValidUploadKey/listing1565=== RUN TestIsValidUploadKey/build_log1566=== PAUSE TestIsValidUploadKey/build_log1567=== RUN TestIsValidUploadKey/build_log_home-manager_file1568=== PAUSE TestIsValidUploadKey/build_log_home-manager_file1569=== RUN TestIsValidUploadKey/build_log_plus_in_name1570=== PAUSE TestIsValidUploadKey/build_log_plus_in_name1571=== RUN TestIsValidUploadKey/build_log_question_mark1572=== PAUSE TestIsValidUploadKey/build_log_question_mark1573=== RUN TestIsValidUploadKey/build_log_equals1574=== PAUSE TestIsValidUploadKey/build_log_equals1575=== RUN TestIsValidUploadKey/realisation1576=== PAUSE TestIsValidUploadKey/realisation1577=== RUN TestIsValidUploadKey/realisation_plus_in_output1578=== PAUSE TestIsValidUploadKey/realisation_plus_in_output1579=== RUN TestIsValidUploadKey/nix-cache-info1580=== PAUSE TestIsValidUploadKey/nix-cache-info1581=== RUN TestIsValidUploadKey/index.html1582=== PAUSE TestIsValidUploadKey/index.html1583=== RUN TestIsValidUploadKey/narinfo_key,_nar_type1584=== PAUSE TestIsValidUploadKey/narinfo_key,_nar_type1585=== RUN TestIsValidUploadKey/nar_key,_narinfo_type1586=== PAUSE TestIsValidUploadKey/nar_key,_narinfo_type1587=== RUN TestIsValidUploadKey/listing_key,_narinfo_type1588=== PAUSE TestIsValidUploadKey/listing_key,_narinfo_type1589=== RUN TestIsValidUploadKey/traversal1590=== PAUSE TestIsValidUploadKey/traversal1591=== RUN TestIsValidUploadKey/traversal_nar1592=== PAUSE TestIsValidUploadKey/traversal_nar1593=== RUN TestIsValidUploadKey/absolute1594=== PAUSE TestIsValidUploadKey/absolute1595=== RUN TestIsValidUploadKey/empty_key1596=== PAUSE TestIsValidUploadKey/empty_key1597=== RUN TestIsValidUploadKey/unknown_type1598=== PAUSE TestIsValidUploadKey/unknown_type1599=== CONT TestPendingClosureWriteTimeout1600=== RUN TestPendingClosureWriteTimeout/empty1601=== PAUSE TestPendingClosureWriteTimeout/empty1602=== RUN TestPendingClosureWriteTimeout/400_objects1603=== PAUSE TestPendingClosureWriteTimeout/400_objects1604=== RUN TestPendingClosureWriteTimeout/670k_objects1605=== PAUSE TestPendingClosureWriteTimeout/670k_objects1606=== RUN TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline1607=== PAUSE TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline1608=== CONT TestCompletedNarNotReofferedAcrossClosures16092026/11/12 16:57:20 WARN mTLS auth: subject not in bound subjects subject="CN=reader"16102026/11/12 16:57:20 WARN mTLS auth: subject not in bound subjects subject="CN=reader"1611--- PASS: TestService_NativeMTLS (7.02s)1612=== CONT TestProxyWriteTimeout1613=== RUN TestProxyWriteTimeout/narinfo1614=== PAUSE TestProxyWriteTimeout/narinfo1615=== RUN TestProxyWriteTimeout/1_GiB_nar1616=== PAUSE TestProxyWriteTimeout/1_GiB_nar1617=== RUN TestProxyWriteTimeout/10_GiB_nar1618=== PAUSE TestProxyWriteTimeout/10_GiB_nar1619=== RUN TestProxyWriteTimeout/unknown_size1620=== PAUSE TestProxyWriteTimeout/unknown_size1621=== CONT TestPush_SignsNarinfosOfItsPendingObjects1622=== NAME TestPinProtectsFromGC1623 client_integration_test.go:900: Pin successfully protected closure from garbage collection16242026/11/12 16:57:20 INFO Garbage collection progress phase="" failed_uploads_deleted=0 old_closures_deleted=1 objects_marked=3 objects_deleted=3 objects_failed=01625=== NAME TestClientIntegration1626 client_integration_test.go:371: Objects in database after GC:1627 client_integration_test.go:371: Successfully deleted all objects with GC --force1628--- PASS: TestPinProtectsFromGC (10.22s)1629=== CONT TestObjectStatsTrigger1630--- PASS: TestClientIntegration (9.41s)1631=== CONT TestMultipartCleanup16322026/11/12 16:57:21 ERROR failed to check GC advisory lock error="query pg_locks: failed to connect to `user=_tribuchetbld2 database=niks3`: /nonexistent/.s.PGSQL.5432 (/nonexistent): dial error: dial unix /nonexistent/.s.PGSQL.5432: connect: no such file or directory"1633--- PASS: TestGCAdvisoryLockBlocksConcurrentRun (7.27s)1634=== CONT TestObjectStatsUpdatedOncePerCommit16352026/11/12 16:57:21 INFO Received push request method=POST path=/api/pushes16362026/11/12 16:57:21 INFO Received complete push request method=POST path=/api/pushes/1/complete16372026/11/12 16:57:22 INFO Completed push id=1 objects=4 duration=179.5405ms16382026/11/12 16:57:22 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete16392026/11/12 16:57:22 INFO Completed upload id=2 objects=2 duration=2.258ms1640--- PASS: TestMetricsCommit (8.18s)1641=== CONT TestPendingCleanupUsesOneCutoff1642=== NAME TestOrphanedObjectsGC1643 orphaned_objects_gc_test.go:287: GC Test Summary:1644 orphaned_objects_gc_test.go:288: - Kept: 2 objects from closure A1645 orphaned_objects_gc_test.go:289: - Deleted: 2 objects from closure B1646 orphaned_objects_gc_test.go:290: - Deleted: 6 orphaned chain objects (X1->X2->X3)1647 orphaned_objects_gc_test.go:291: - Deleted: 2 orphaned single objects (Y)1648 orphaned_objects_gc_test.go:292: - Total deleted: 10 objects1649=== CONT TestMultipartUploadAbortedWhenCancelledBeforeRecorded1650--- PASS: TestOrphanedObjectsGC (9.19s)1651--- PASS: TestMetricsInventory (8.29s)1652=== CONT TestService_AuthMiddleware_OIDC16532026/11/12 16:57:22 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:64650/oidc16542026/11/12 16:57:22 INFO lead: acquired remote=192.0.2.1:123416552026/11/12 16:57:22 INFO lead: acquired remote=192.0.2.1:12341656--- PASS: TestConnectWaitsForAPeerMigration (7.43s)1657=== CONT TestService_AuthMiddleware_MTLSBoundSubjects16582026/11/12 16:57:23 INFO lead: acquired remote=192.0.2.1:123416592026/11/12 16:57:23 INFO lead: released remote=192.0.2.1:12341660--- PASS: TestLeadIncumbentAnnouncesAtOnce (7.39s)1661=== CONT TestCacheStatsHandler16622026/11/12 16:57:23 INFO lead: released remote=192.0.2.1:123416632026/11/12 16:57:23 INFO lead: acquired remote=192.0.2.1:123416642026/11/12 16:57:23 INFO lead: acquired remote=192.0.2.1:123416652026/11/12 16:57:23 INFO lead: released remote=192.0.2.1:12341666--- PASS: TestLeadEndsOnShutdown (8.41s)1667=== CONT TestService_ReadAuthMiddleware16682026/11/12 16:57:24 INFO Received uploads request method=POST path=/api/pending_closures16692026/11/12 16:57:24 INFO Received uploads request method=POST path=/api/pending_closures16702026/11/12 16:57:24 INFO Received complete multipart upload request method=POST path=/api/multipart/complete16712026/11/12 16:57:24 INFO lead: released remote=192.0.2.1:12341672--- PASS: TestLeadElectsOneAndHandsOver (9.38s)1673=== CONT TestCacheConfigHandler1674=== RUN TestCacheConfigHandler/full_config,_no_issuer1675=== PAUSE TestCacheConfigHandler/full_config,_no_issuer1676=== RUN TestCacheConfigHandler/no_cache_url_configured1677=== PAUSE TestCacheConfigHandler/no_cache_url_configured1678=== RUN TestCacheConfigHandler/no_signing_keys1679=== PAUSE TestCacheConfigHandler/no_signing_keys1680=== RUN TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1681=== PAUSE TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1682=== CONT TestService_ReadScope_PublicByDefault16832026/11/12 16:57:24 INFO Received complete multipart upload request method=POST path=/api/multipart/complete16842026/11/12 16:57:24 WARN CompleteMultipartUpload errored but object exists; treating as success error="The specified key does not exist." object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=MmY2ODA0YWEtNzBmZS00ZTBiLTk5ZGMtNzU0MmM1OGFlMzZmLmI0NjYzMTdhLTYwZjMtNGVjNC1hMmZlLTY0NGI4ZmMzYjA4MHgxNzk0NTAyNjQ0NjQ1OTU5MDAw16852026/11/12 16:57:24 INFO Received uploads request method=POST path=/api/pending_closures16862026/11/12 16:57:25 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=MmY2ODA0YWEtNzBmZS00ZTBiLTk5ZGMtNzU0MmM1OGFlMzZmLmI0NjYzMTdhLTYwZjMtNGVjNC1hMmZlLTY0NGI4ZmMzYjA4MHgxNzk0NTAyNjQ0NjQ1OTU5MDAw parts=11687--- PASS: TestConcurrentCommitsSharingObjectsDoNotDeadlock (8.23s)1688=== CONT TestService_RequireScope_OIDC1689=== CONT TestPush_SkippedKeySurvivesGCBeforeCommit1690--- PASS: TestCompleteMultipartUpload_ErrorButObjectExists (7.03s)16912026/11/12 16:57:25 INFO lead: released remote=192.0.2.1:123416922026/11/12 16:57:25 INFO lead: acquired remote=192.0.2.1:12341693--- PASS: TestCreatePendingClosure_SmallNARUsesSimplePUT (7.01s)16942026/11/12 16:57:25 INFO lead: given back to the incumbent remote=192.0.2.1:12341695=== CONT TestCommitCanBeRepeatedUntilCleanup1696=== RUN TestCommitCanBeRepeatedUntilCleanup/push1697=== PAUSE TestCommitCanBeRepeatedUntilCleanup/push1698=== RUN TestCommitCanBeRepeatedUntilCleanup/pending_closure1699=== PAUSE TestCommitCanBeRepeatedUntilCleanup/pending_closure1700=== CONT TestPush_RejectsBadRequests17012026/11/12 16:57:25 INFO lead: acquired remote=192.0.2.1:123417022026/11/12 16:57:25 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:64671/oidc17032026/11/12 16:57:25 INFO Received complete multipart upload request method=POST path=/api/multipart/complete17042026/11/12 16:57:25 ERROR Failed to get multipart upload error="no rows in result set" upload_id=does-not-exist object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst1705--- PASS: TestCompleteMultipartUnregistered (6.93s)1706=== CONT TestPush_RootsAreLockedInByteOrder17072026/11/12 16:57:25 INFO Received uploads request method=POST path=/api/pending_closures17082026/11/12 16:57:25 INFO Received uploads request method=POST path=/api/pending_closures17092026/11/12 16:57:25 INFO Received uploads request method=POST path=/api/pending_closures17102026/11/12 16:57:25 INFO Received uploads request method=POST path=/api/pending_closures17112026/11/12 16:57:26 INFO Received cleanup request method=DELETE path=/api/pending_closures17122026/11/12 16:57:26 INFO Aborted multipart uploads count=0 kept=017132026/11/12 16:57:26 INFO Received uploads request method=POST path=/api/pending_closures17142026/11/12 16:57:26 INFO Received cleanup request method=DELETE path=/api/pending_closures17152026/11/12 16:57:26 INFO Aborted multipart uploads count=1 kept=017162026/11/12 16:57:26 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete17172026-11-12 16:57:26.429 UTC [9159] ERROR: Closure does not exist: id=117182026-11-12 16:57:26.429 UTC [9159] CONTEXT: PL/pgSQL function commit_pending_closure(bigint) line 17 at RAISE17192026-11-12 16:57:26.429 UTC [9159] STATEMENT: -- name: CommitPendingClosure :one1720 SELECT commit_pending_closure($1::bigint)::bigint1721 1722--- PASS: TestService_cleanupPendingClosuresHandler (6.47s)1723=== CONT TestEnsureAbortUploadsRule1724--- PASS: TestService_Rustfstest (6.46s)1725=== CONT TestCreatePendingClosureRejectsOversizedNAR17262026/11/12 16:57:26 INFO Received uploads request method=POST path=/api/pending_closures1727--- PASS: TestCreatePendingClosureRejectsOversizedNAR (0.00s)1728=== CONT TestService_healthCheckHandler17292026/11/12 16:57:26 INFO Received uploads request method=POST path=/api/pending_closures17302026/11/12 16:57:26 INFO lead: released remote=192.0.2.1:12341731--- PASS: TestLeadUnannouncedStandbyYieldsToIncumbent (9.69s)1732=== CONT TestGenerateLandingPage1733--- PASS: TestGenerateLandingPage (0.00s)1734=== CONT TestService_readinessHandler17352026/11/12 16:57:27 INFO Registered completed upload object_key=nar/jjjjjjjjjjjjjjjjjjjjjjjjjjjjjj0100000000000000000000.nar.zst17362026/11/12 16:57:27 INFO Received uploads request method=POST path=/api/pending_closures1737--- PASS: TestPresignedUploadRegisteredBeforeCommit (6.93s)1738=== CONT TestCacheConfigHandlerMaxNarSize1739--- PASS: TestCacheConfigHandlerMaxNarSize (0.00s)1740=== CONT TestService_AuthMiddleware_MTLSProxyHeader17412026/11/12 16:57:27 WARN Rate limiter enabled after throttle name=s3-test rate=517422026/11/12 16:57:27 WARN S3 rate limit hit operation="complete multipart upload" error="Please reduce your request rate."1743=== NAME TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle1744 throttle_test.go:213: Proxy stats: total=13, throttled=10, completeMultipart=101745 throttle_test.go:215: Rate limiter: enabled=true, rate=5.001746--- PASS: TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle (9.52s)1747=== CONT TestService_AuthMiddleware17482026/11/12 16:57:27 INFO Received uploads request method=POST path=/api/pending_closures17492026/11/12 16:57:27 INFO Received complete multipart upload request method=POST path=/api/multipart/complete17502026/11/12 16:57:27 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst upload_id=MmY2ODA0YWEtNzBmZS00ZTBiLTk5ZGMtNzU0MmM1OGFlMzZmLjNhMjc1ZjdlLTBmOTEtNGNmNC1iMTcwLWQ0MmQ2OGFkMzI3OXgxNzk0NTAyNjQ1NjU4MTE0MDAw parts=1017512026/11/12 16:57:27 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete17522026/11/12 16:57:27 INFO Completed upload id=1 objects=2 duration=43.336417ms17532026/11/12 16:57:27 INFO Received uploads request method=POST path=/api/pending_closures17542026/11/12 16:57:27 INFO Received uploads request method=POST path=/api/pending_closures17552026/11/12 16:57:27 INFO Object in database but missing from S3 key=dadb44fdadb44fdadb44fdadb44f0000.narinfo17562026/11/12 16:57:27 WARN Found objects in DB but missing from S3, will re-upload count=11757--- PASS: TestService_verifyS3Integrity (9.27s)1758=== CONT TestGCTaskStore_PhaseUpdates1759--- PASS: TestGCTaskStore_PhaseUpdates (0.00s)1760=== CONT TestGCTaskStore_GetReturnsLatest1761--- PASS: TestGCTaskStore_GetReturnsLatest (0.00s)1762=== CONT TestGracefulShutdownDrainsInflight17632026/11/12 16:57:27 INFO Starting HTTP server address=127.0.0.1:6470117642026/11/12 16:57:27 INFO Shutdown signal received, draining in-flight requests timeout=10s1765--- PASS: TestGracefulShutdownDrainsInflight (0.07s)1766=== CONT TestGCTaskStore_CompletedAllowsNewTask1767--- PASS: TestGCTaskStore_CompletedAllowsNewTask (0.00s)1768=== CONT TestGCTaskStore_Fail1769--- PASS: TestGCTaskStore_Fail (0.00s)1770=== CONT TestSharedNarBothUploadsComplete17712026/11/12 16:57:27 INFO Received complete multipart upload request method=POST path=/api/multipart/complete1772--- PASS: TestObjectStatsTrigger (7.02s)1773=== CONT TestGCEndsOnShutdown/before_the_run17742026/11/12 16:57:28 INFO Completed multipart upload object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst upload_id=MmY2ODA0YWEtNzBmZS00ZTBiLTk5ZGMtNzU0MmM1OGFlMzZmLjZmOTk1MzI5LWMzMzQtNDgyNi04YzIwLWZjZDlkNGRlNzUzYngxNzk0NTAyNjQ1OTQ3OTA3MDAw parts=1017752026/11/12 16:57:28 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete17762026/11/12 16:57:28 INFO Completed upload id=1 objects=2 duration=39.900208ms17772026/11/12 16:57:28 INFO Received get closure request method=GET path=/api/closures/0000000000000000000000000000000017782026/11/12 16:57:28 INFO Received uploads request method=POST path=/api/pending_closures17792026/11/12 16:57:28 INFO Starting cleanup of old closures method=DELETE path=/api/closures17802026/11/12 16:57:28 INFO Aborted multipart uploads count=0 kept=017812026/11/12 16:57:28 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=017822026/11/12 16:57:28 INFO Vacuumed table table=pending_closures17832026/11/12 16:57:28 INFO Vacuumed table table=commit_receipts17842026/11/12 16:57:28 INFO Vacuumed table table=pending_objects17852026/11/12 16:57:28 INFO Received push request method=POST path=/api/pushes17862026/11/12 16:57:28 INFO Vacuumed table table=multipart_uploads17872026/11/12 16:57:28 INFO Vacuumed table table=closures17882026/11/12 16:57:28 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign17892026/11/12 16:57:28 INFO Vacuumed table table=objects17902026/11/12 16:57:28 INFO Signed narinfos id=1 count=117912026/11/12 16:57:28 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign17922026/11/12 16:57:28 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign17932026/11/12 16:57:28 ERROR Failed to sign narinfos signer="index 1" error="signing key cannot be nil"17942026/11/12 16:57:28 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign17952026/11/12 16:57:28 INFO Received get closure request method=GET path=/api/closures/0000000000000000000000000000000017962026/11/12 16:57:28 ERROR Failed to sign narinfos signer=niks3-test-1 error="signer failed"17972026/11/12 16:57:28 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign1798--- PASS: TestService_createPendingClosureHandler (9.02s)17992026/11/12 16:57:28 ERROR Signer returned unexpected signature count signer=niks3-test-1 got=0 want=11800=== CONT TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark18012026/11/12 16:57:28 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign18022026/11/12 16:57:28 ERROR Signer returned unexpected object key signer=niks3-test-1 object_key=unexpected.narinfo1803--- PASS: TestPush_SignsNarinfosOfItsPendingObjects (7.67s)1804=== CONT TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete18052026/11/12 16:57:28 INFO Received uploads request method=POST path=/api/pending_closures18062026/11/12 16:57:28 INFO Received cleanup request method=DELETE path=/api/pending_closures18072026/11/12 16:57:28 WARN Failed to abort upload, keeping its closure key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst error="Get \"http://127.0.0.1:1/bucket74/?location=\": dial tcp 127.0.0.1:1: connect: connection refused" code=""18082026/11/12 16:57:28 INFO Aborted multipart uploads count=0 kept=118092026/11/12 16:57:28 INFO Received cleanup request method=DELETE path=/api/pending_closures18102026/11/12 16:57:28 INFO Aborted multipart uploads count=1 kept=01811--- PASS: TestMultipartCleanup (7.90s)1812=== CONT TestIsValidCachePath/narinfo1813=== CONT TestIsValidCachePath/traversal_parent1814=== CONT TestIsValidCachePath/pin_invalid_char1815=== CONT TestIsValidCachePath/pin_nested1816=== CONT TestIsValidCachePath/pin_empty_name1817=== CONT TestIsValidCachePath/short_hash1818=== CONT TestIsValidCachePath/wrong_extension1819=== CONT TestIsValidCachePath/leading_slash1820=== CONT TestIsValidCachePath/empty1821=== CONT TestIsValidCachePath/random_path1822=== CONT TestIsValidCachePath/invalid_char_u1823=== CONT TestIsValidCachePath/invalid_char_e1824=== CONT TestIsValidCachePath/pin_traversal1825=== CONT TestIsValidCachePath/traversal_in_middle1826=== CONT TestIsValidCachePath/log1827=== CONT TestIsValidCachePath/pin_all_alphabet_chars1828=== CONT TestIsValidCachePath/pin1829=== CONT TestIsValidCachePath/index.html1830=== CONT TestIsValidCachePath/nix-cache-info1831=== CONT TestIsValidCachePath/realisation1832=== CONT TestIsValidCachePath/nar_bz21833=== CONT TestIsValidCachePath/ls1834=== CONT TestIsValidCachePath/nar_uncompressed1835=== CONT TestIsValidCachePath/nar_zst1836=== CONT TestIsValidCachePath/nar_xz1837=== CONT TestIsValidCachePath/narinfo_all_nix_base32_chars1838--- PASS: TestIsValidCachePath (0.01s)1839 --- PASS: TestIsValidCachePath/narinfo (0.00s)1840 --- PASS: TestIsValidCachePath/traversal_parent (0.00s)1841 --- PASS: TestIsValidCachePath/pin_invalid_char (0.00s)1842 --- PASS: TestIsValidCachePath/pin_nested (0.00s)1843 --- PASS: TestIsValidCachePath/pin_empty_name (0.00s)1844 --- PASS: TestIsValidCachePath/short_hash (0.00s)1845 --- PASS: TestIsValidCachePath/wrong_extension (0.00s)1846 --- PASS: TestIsValidCachePath/leading_slash (0.00s)1847 --- PASS: TestIsValidCachePath/empty (0.00s)1848 --- PASS: TestIsValidCachePath/random_path (0.00s)1849 --- PASS: TestIsValidCachePath/invalid_char_u (0.00s)1850 --- PASS: TestIsValidCachePath/invalid_char_e (0.00s)1851 --- PASS: TestIsValidCachePath/pin_traversal (0.00s)1852 --- PASS: TestIsValidCachePath/traversal_in_middle (0.00s)1853 --- PASS: TestIsValidCachePath/log (0.00s)1854 --- PASS: TestIsValidCachePath/pin_all_alphabet_chars (0.00s)1855 --- PASS: TestIsValidCachePath/pin (0.00s)1856 --- PASS: TestIsValidCachePath/index.html (0.00s)1857 --- PASS: TestIsValidCachePath/nix-cache-info (0.00s)1858 --- PASS: TestIsValidCachePath/realisation (0.00s)1859 --- PASS: TestIsValidCachePath/nar_bz2 (0.00s)1860 --- PASS: TestIsValidCachePath/ls (0.00s)1861 --- PASS: TestIsValidCachePath/nar_uncompressed (0.00s)1862 --- PASS: TestIsValidCachePath/nar_zst (0.00s)1863 --- PASS: TestIsValidCachePath/nar_xz (0.00s)1864 --- PASS: TestIsValidCachePath/narinfo_all_nix_base32_chars (0.00s)1865=== CONT TestParseSingleRange/none1866=== CONT TestPresentNotReportedWhileGCDeletesClosure/stale1867--- PASS: TestObjectStatsUpdatedOncePerCommit (8.12s)1868=== CONT TestParseSingleRange/closed1869=== CONT TestParseSingleRange/malformed_end_before_start1870=== CONT TestParseSingleRange/malformed_both_empty1871=== CONT TestParseSingleRange/malformed_no_dash1872=== CONT TestParseSingleRange/multi-range_ignored1873=== CONT TestParseSingleRange/unknown_unit1874=== CONT TestParseSingleRange/open-ended1875=== CONT TestParseSingleRange/start_far_past_EOF1876=== CONT TestParseSingleRange/start_past_EOF1877=== CONT TestParseSingleRange/single_byte1878=== CONT TestParseSingleRange/suffix_exceeds_size1879=== CONT TestParseSingleRange/suffix1880=== CONT TestParseSingleRange/end_clamped_to_size1881=== CONT TestPresentNotReportedWhileGCDeletesClosure/fresh1882--- PASS: TestParseSingleRange (0.00s)1883 --- PASS: TestParseSingleRange/none (0.00s)1884 --- PASS: TestParseSingleRange/closed (0.00s)1885 --- PASS: TestParseSingleRange/malformed_end_before_start (0.00s)1886 --- PASS: TestParseSingleRange/malformed_both_empty (0.00s)1887 --- PASS: TestParseSingleRange/malformed_no_dash (0.00s)1888 --- PASS: TestParseSingleRange/multi-range_ignored (0.00s)1889 --- PASS: TestParseSingleRange/unknown_unit (0.00s)1890 --- PASS: TestParseSingleRange/open-ended (0.00s)1891 --- PASS: TestParseSingleRange/start_far_past_EOF (0.00s)1892 --- PASS: TestParseSingleRange/start_past_EOF (0.00s)1893 --- PASS: TestParseSingleRange/single_byte (0.00s)1894 --- PASS: TestParseSingleRange/suffix_exceeds_size (0.00s)1895 --- PASS: TestParseSingleRange/suffix (0.00s)1896 --- PASS: TestParseSingleRange/end_clamped_to_size (0.00s)18972026/11/12 16:57:29 INFO Received uploads request method=POST path=/api/pending_closures1898--- PASS: TestMultipartUploadAbortedWhenCancelledBeforeRecorded (7.39s)1899=== CONT TestForceGCDuringPushOffersSweptObject/before_pending_rows19002026/11/12 16:57:29 INFO Received complete multipart upload request method=POST path=/api/multipart/complete19012026/11/12 16:57:29 INFO Received uploads request method=POST path=/api/pending_closures19022026/11/12 16:57:30 INFO Completed multipart upload object_key=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhh0100000000000000000000.nar.zst upload_id=MmY2ODA0YWEtNzBmZS00ZTBiLTk5ZGMtNzU0MmM1OGFlMzZmLjdmOGVhNTZjLTRlOTAtNGY4ZS1hNTQ2LTNmZTIyZGMyMzQxN3gxNzk0NTAyNjQ3NDY5MTY2MDAw parts=1219032026/11/12 16:57:30 INFO Received uploads request method=POST path=/api/pending_closures1904--- PASS: TestCompletedNarNotReofferedAcrossClosures (9.35s)1905=== CONT TestForceGCDuringPushOffersSweptObject/after_presence_check19062026/11/12 16:57:30 INFO Received uploads request method=POST path=/api/pending_closures19072026/11/12 16:57:30 INFO Received cleanup request method=DELETE path=/api/pending_closures1908=== RUN TestService_AuthMiddleware_OIDC/valid_OIDC_token1909=== PAUSE TestService_AuthMiddleware_OIDC/valid_OIDC_token1910=== RUN TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1911=== PAUSE TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1912=== RUN TestService_AuthMiddleware_OIDC/malformed_token_rejected1913=== PAUSE TestService_AuthMiddleware_OIDC/malformed_token_rejected1914=== RUN TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1915=== PAUSE TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1916=== CONT TestClientErrorHandling/InvalidStorePath19172026/11/12 16:57:30 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted,O=Other"19182026/11/12 16:57:30 WARN mTLS auth: bound subjects configured but subject DN unavailable19192026/11/12 16:57:30 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted"1920--- PASS: TestService_AuthMiddleware_MTLSBoundSubjects (7.63s)1921=== CONT TestClientErrorHandling/ServerNotAvailable19222026/11/12 16:57:31 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://localhost:19999/api/objects/present\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/objects/present1923--- PASS: TestCacheStatsHandler (8.09s)1924=== CONT TestClientErrorHandling/InvalidAuthToken19252026/11/12 16:57:31 WARN Request failed, retrying attempt=2 backoff=193.527997ms error="Post \"http://localhost:19999/api/objects/present\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/objects/present1926--- PASS: TestService_ReadAuthMiddleware (7.59s)1927=== CONT TestServerTLSConfig/no_client_CA1928=== CONT TestServerTLSConfig/not_a_PEM_file19292026/11/12 16:57:31 WARN Request failed, retrying attempt=3 backoff=404.264336ms error="Post \"http://localhost:19999/api/objects/present\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/objects/present1930=== CONT TestServerTLSConfig/missing_CA_file1931--- PASS: TestServerTLSConfig (0.00s)1932 --- PASS: TestServerTLSConfig/no_client_CA (0.00s)1933 --- PASS: TestServerTLSConfig/not_a_PEM_file (0.02s)1934 --- PASS: TestServerTLSConfig/missing_CA_file (0.00s)1935=== CONT TestResolveDBConnectionString/flag_wins1936=== CONT TestResolveDBConnectionString/PGHOST_allows_empty1937=== CONT TestResolveDBConnectionString/nothing_configured1938=== CONT TestResolveDBConnectionString/missing_file_is_an_error1939=== CONT TestResolveDBConnectionString/file_when_flag_empty1940=== CONT TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed1941--- PASS: TestResolveDBConnectionString (0.03s)1942 --- PASS: TestResolveDBConnectionString/flag_wins (0.00s)1943 --- PASS: TestResolveDBConnectionString/PGHOST_allows_empty (0.00s)1944 --- PASS: TestResolveDBConnectionString/nothing_configured (0.00s)1945 --- PASS: TestResolveDBConnectionString/missing_file_is_an_error (0.00s)1946 --- PASS: TestResolveDBConnectionString/file_when_flag_empty (0.00s)19472026/11/12 16:57:31 WARN Request failed, retrying attempt=4 backoff=866.278639ms error="Post \"http://localhost:19999/api/objects/present\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/objects/present1948=== RUN TestPush_RejectsBadRequests/no_roots1949=== PAUSE TestPush_RejectsBadRequests/no_roots1950=== RUN TestPush_RejectsBadRequests/no_objects1951=== PAUSE TestPush_RejectsBadRequests/no_objects1952=== RUN TestPush_RejectsBadRequests/bad_root1953=== PAUSE TestPush_RejectsBadRequests/bad_root1954=== RUN TestPush_RejectsBadRequests/root_not_in_objects1955=== PAUSE TestPush_RejectsBadRequests/root_not_in_objects1956=== CONT TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed19572026/11/12 16:57:32 INFO Received push request method=POST path=/api/pushes19582026/11/12 16:57:32 INFO Received complete push request method=POST path=/api/pushes/1/complete19592026/11/12 16:57:32 INFO Completed push id=1 objects=4 duration=76.872875ms19602026/11/12 16:57:32 INFO Received push request method=POST path=/api/pushes1961--- PASS: TestService_ReadScope_PublicByDefault (7.86s)1962=== CONT TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed19632026/11/12 16:57:32 INFO Aborted multipart uploads count=0 kept=019642026/11/12 16:57:32 WARN Force mode enabled - objects will be deleted immediately without grace period19652026/11/12 16:57:32 WARN Request failed, retrying attempt=5 backoff=1.758159374s error="Post \"http://localhost:19999/api/objects/present\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/objects/present19662026/11/12 16:57:32 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=2 objects-deleted-after-grace-period=2 objects-failed-to-delete=019672026/11/12 16:57:32 INFO Vacuumed table table=pending_closures19682026/11/12 16:57:32 INFO Vacuumed table table=commit_receipts19692026/11/12 16:57:32 INFO Vacuumed table table=pending_objects19702026/11/12 16:57:33 INFO Vacuumed table table=multipart_uploads19712026/11/12 16:57:33 INFO Vacuumed table table=closures19722026/11/12 16:57:33 INFO Vacuumed table table=objects19732026/11/12 16:57:33 INFO Received complete push request method=POST path=/api/pushes/2/complete19742026/11/12 16:57:33 INFO Completed push id=2 objects=4 duration=14.054792ms1975--- PASS: TestPush_SkippedKeySurvivesGCBeforeCommit (8.08s)1976=== CONT TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost1977=== RUN TestService_RequireScope_OIDC/builder_may_write1978=== PAUSE TestService_RequireScope_OIDC/builder_may_write1979=== RUN TestService_RequireScope_OIDC/builder_may_not_admin1980=== PAUSE TestService_RequireScope_OIDC/builder_may_not_admin1981=== RUN TestService_RequireScope_OIDC/ops_may_admin1982=== PAUSE TestService_RequireScope_OIDC/ops_may_admin1983=== RUN TestService_RequireScope_OIDC/ops_may_not_write1984=== PAUSE TestService_RequireScope_OIDC/ops_may_not_write1985=== RUN TestService_RequireScope_OIDC/reader_may_not_write1986=== PAUSE TestService_RequireScope_OIDC/reader_may_not_write1987=== RUN TestService_RequireScope_OIDC/static_token_may_admin1988=== PAUSE TestService_RequireScope_OIDC/static_token_may_admin1989=== RUN TestService_RequireScope_OIDC/static_token_may_write1990=== PAUSE TestService_RequireScope_OIDC/static_token_may_write1991=== RUN TestService_RequireScope_OIDC/reader_may_read1992=== PAUSE TestService_RequireScope_OIDC/reader_may_read1993=== RUN TestService_RequireScope_OIDC/writer_implies_read1994=== PAUSE TestService_RequireScope_OIDC/writer_implies_read1995=== RUN TestService_RequireScope_OIDC/anonymous_may_not_read1996=== PAUSE TestService_RequireScope_OIDC/anonymous_may_not_read1997=== RUN TestService_RequireScope_OIDC/reader_may_read_with_netrc1998=== PAUSE TestService_RequireScope_OIDC/reader_may_read_with_netrc1999=== RUN TestService_RequireScope_OIDC/static_token_may_admin_with_netrc2000=== PAUSE TestService_RequireScope_OIDC/static_token_may_admin_with_netrc
loading 871 hidden lines…
28722026/11/12 16:57:57 INFO Received push request method=POST path=/28732026/11/12 16:57:57 INFO Aborted multipart uploads count=0 kept=028742026/11/12 16:57:57 INFO Received uploads request method=POST path=/28752026/11/12 16:57:57 INFO Vacuumed table table=closures28762026/11/12 16:57:57 INFO Vacuumed table table=pending_closures28772026/11/12 16:57:57 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028782026/11/12 16:57:57 INFO Vacuumed table table=commit_receipts28792026/11/12 16:57:57 INFO Vacuumed table table=pending_closures28802026/11/12 16:57:57 INFO Aborted multipart uploads count=0 kept=028812026/11/12 16:57:57 WARN Force mode enabled - objects will be deleted immediately without grace period28822026/11/12 16:57:57 INFO Vacuumed table table=commit_receipts28832026/11/12 16:57:57 INFO Vacuumed table table=pending_objects28842026/11/12 16:57:57 INFO Vacuumed table table=multipart_uploads28852026/11/12 16:57:57 INFO Vacuumed table table=pending_objects28862026/11/12 16:57:57 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=16 objects-failed-to-delete=028872026/11/12 16:57:57 INFO Vacuumed table table=closures28882026/11/12 16:57:57 INFO Vacuumed table table=objects28892026/11/12 16:57:57 INFO Vacuumed table table=multipart_uploads28902026/11/12 16:57:57 INFO Received complete upload request method=POST path=/28912026/11/12 16:57:57 INFO Vacuumed table table=closures28922026/11/12 16:57:57 INFO Vacuumed table table=objects28932026/11/12 16:57:57 INFO Vacuumed table table=pending_closures28942026/11/12 16:57:57 INFO Completed upload id=18 objects=11 duration=839.916µs28952026/11/12 16:57:57 INFO Vacuumed table table=commit_receipts28962026/11/12 16:57:57 INFO Vacuumed table table=objects28972026/11/12 16:57:57 INFO Received uploads request method=POST path=/28982026/11/12 16:57:57 INFO Received push request method=POST path=/28992026/11/12 16:57:57 INFO Received push request method=POST path=/29002026/11/12 16:57:57 INFO Vacuumed table table=pending_objects29012026/11/12 16:57:57 INFO Aborted multipart uploads count=0 kept=029022026/11/12 16:57:57 INFO Vacuumed table table=multipart_uploads29032026/11/12 16:57:57 INFO Received push request method=POST path=/29042026/11/12 16:57:57 WARN Force mode enabled - objects will be deleted immediately without grace period29052026/11/12 16:57:57 INFO Received complete push request method=POST path=/29062026/11/12 16:57:57 INFO Vacuumed table table=closures29072026/11/12 16:57:57 INFO Received push request method=POST path=/29082026/11/12 16:57:57 INFO Received push request method=POST path=/29092026/11/12 16:57:57 INFO Vacuumed table table=objects29102026/11/12 16:57:57 INFO Received push request method=POST path=/29112026/11/12 16:57:57 INFO Received push request method=POST path=/29122026/11/12 16:57:57 INFO Received push request method=POST path=/29132026/11/12 16:57:57 INFO Received push request method=POST path=/29142026/11/12 16:57:57 INFO Aborted multipart uploads count=0 kept=029152026/11/12 16:57:57 INFO Aborted multipart uploads count=0 kept=029162026/11/12 16:57:57 INFO Received push request method=POST path=/29172026/11/12 16:57:57 INFO Completed push id=1 objects=6 duration=34.760208ms29182026/11/12 16:57:57 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029192026/11/12 16:57:57 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029202026/11/12 16:57:57 INFO Vacuumed table table=pending_closures29212026/11/12 16:57:57 INFO Vacuumed table table=commit_receipts29222026/11/12 16:57:57 INFO Vacuumed table table=pending_closures29232026/11/12 16:57:57 INFO Vacuumed table table=commit_receipts29242026/11/12 16:57:57 INFO Received push request method=POST path=/29252026/11/12 16:57:58 INFO Vacuumed table table=pending_objects29262026/11/12 16:57:58 INFO Received uploads request method=POST path=/29272026/11/12 16:57:58 INFO Vacuumed table table=multipart_uploads29282026/11/12 16:57:58 INFO Received push request method=POST path=/29292026/11/12 16:57:58 INFO Vacuumed table table=closures29302026/11/12 16:57:58 INFO Received push request method=POST path=/29312026/11/12 16:57:58 INFO Vacuumed table table=objects29322026/11/12 16:57:58 INFO Received uploads request method=POST path=/29332026/11/12 16:57:58 INFO Received push request method=POST path=/29342026/11/12 16:57:58 INFO Received uploads request method=POST path=/29352026/11/12 16:57:58 INFO Vacuumed table table=pending_objects29362026/11/12 16:57:58 INFO Received push request method=POST path=/29372026/11/12 16:57:58 INFO Received push request method=POST path=/29382026/11/12 16:57:58 INFO Vacuumed table table=multipart_uploads29392026/11/12 16:57:58 INFO Received complete push request method=POST path=/29402026/11/12 16:57:58 INFO Aborted multipart uploads count=0 kept=029412026/11/12 16:57:58 INFO Vacuumed table table=closures29422026/11/12 16:57:58 INFO Completed push id=26 objects=14 duration=892.208µs29432026/11/12 16:57:58 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=029442026/11/12 16:57:58 INFO Vacuumed table table=objects29452026/11/12 16:57:58 INFO Received push request method=POST path=/29462026/11/12 16:57:58 INFO Vacuumed table table=pending_closures29472026/11/12 16:57:58 INFO Vacuumed table table=commit_receipts29482026/11/12 16:57:58 INFO Vacuumed table table=pending_objects29492026/11/12 16:57:58 INFO Vacuumed table table=multipart_uploads29502026/11/12 16:57:58 INFO Received uploads request method=POST path=/29512026/11/12 16:57:58 INFO Received uploads request method=POST path=/29522026/11/12 16:57:58 INFO Vacuumed table table=closures29532026/11/12 16:57:58 INFO Aborted multipart uploads count=0 kept=029542026/11/12 16:57:58 INFO Received uploads request method=POST path=/29552026/11/12 16:57:58 INFO Vacuumed table table=objects29562026/11/12 16:57:58 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029572026/11/12 16:57:58 INFO Vacuumed table table=pending_closures29582026/11/12 16:57:58 INFO Received push request method=POST path=/29592026/11/12 16:57:58 INFO Vacuumed table table=commit_receipts29602026/11/12 16:57:58 INFO Vacuumed table table=pending_objects29612026/11/12 16:57:58 INFO Vacuumed table table=multipart_uploads29622026/11/12 16:57:58 INFO Received uploads request method=POST path=/29632026/11/12 16:57:58 INFO Received push request method=POST path=/29642026/11/12 16:57:58 INFO Vacuumed table table=closures29652026/11/12 16:57:58 INFO Received complete upload request method=POST path=/29662026/11/12 16:57:58 INFO Vacuumed table table=objects29672026/11/12 16:57:58 INFO Received uploads request method=POST path=/29682026/11/12 16:57:58 INFO Completed upload id=23 objects=18 duration=1.078833ms29692026/11/12 16:57:58 INFO Received uploads request method=POST path=/29702026/11/12 16:57:58 INFO Received push request method=POST path=/29712026/11/12 16:57:58 INFO Received uploads request method=POST path=/29722026/11/12 16:57:58 INFO Received push request method=POST path=/29732026/11/12 16:57:58 INFO Aborted multipart uploads count=0 kept=029742026/11/12 16:57:58 WARN Force mode enabled - objects will be deleted immediately without grace period29752026/11/12 16:57:58 INFO Received push request method=POST path=/29762026/11/12 16:57:58 INFO Received push request method=POST path=/29772026/11/12 16:57:58 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029782026/11/12 16:57:58 INFO Vacuumed table table=pending_closures29792026/11/12 16:57:58 INFO Received complete upload request method=POST path=/29802026/11/12 16:57:58 INFO Vacuumed table table=commit_receipts29812026/11/12 16:57:58 INFO Completed upload id=22 objects=22 duration=5.828917ms29822026/11/12 16:57:58 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."29832026/11/12 16:57:58 INFO Received push request method=POST path=/29842026/11/12 16:57:58 INFO Received push request method=POST path=/29852026/11/12 16:57:58 INFO Vacuumed table table=pending_objects29862026/11/12 16:57:58 INFO Received uploads request method=POST path=/29872026/11/12 16:57:58 INFO Vacuumed table table=multipart_uploads29882026/11/12 16:57:58 INFO Received complete push request method=POST path=/29892026/11/12 16:57:58 INFO Vacuumed table table=closures29902026/11/12 16:57:58 INFO Completed push id=26 objects=5 duration=6.221041ms29912026/11/12 16:57:58 INFO Vacuumed table table=objects29922026/11/12 16:57:58 INFO Received complete upload request method=POST path=/29932026/11/12 16:57:58 INFO Completed upload id=24 objects=14 duration=508µs29942026/11/12 16:57:58 INFO Aborted multipart uploads count=0 kept=029952026/11/12 16:57:58 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029962026/11/12 16:57:58 INFO Received complete push request method=POST path=/29972026/11/12 16:57:58 INFO Vacuumed table table=pending_closures29982026/11/12 16:57:58 INFO Completed push id=19 objects=3 duration=304.791µs29992026/11/12 16:57:58 INFO Vacuumed table table=commit_receipts30002026/11/12 16:57:58 INFO Received uploads request method=POST path=/30012026/11/12 16:57:58 INFO Received complete push request method=POST path=/30022026/11/12 16:57:58 INFO Vacuumed table table=pending_objects30032026/11/12 16:57:58 INFO Vacuumed table table=multipart_uploads30042026/11/12 16:57:58 INFO Completed push id=27 objects=25 duration=10.818834ms30052026/11/12 16:57:58 INFO Vacuumed table table=closures30062026/11/12 16:57:58 INFO Received uploads request method=POST path=/30072026/11/12 16:57:58 INFO Received complete push request method=POST path=/30082026/11/12 16:57:58 INFO Aborted multipart uploads count=0 kept=030092026/11/12 16:57:58 INFO Received complete upload request method=POST path=/30102026/11/12 16:57:58 INFO Vacuumed table table=objects30112026/11/12 16:57:58 INFO Completed push id=2 objects=4 duration=505.209µs30122026/11/12 16:57:58 INFO Completed upload id=25 objects=4 duration=391.125µs30132026/11/12 16:57:58 INFO Received complete push request method=POST path=/30142026/11/12 16:57:58 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030152026/11/12 16:57:58 INFO Received push request method=POST path=/30162026/11/12 16:57:58 INFO Aborted multipart uploads count=0 kept=030172026/11/12 16:57:58 INFO Completed push id=15 objects=4 duration=954.917µs30182026/11/12 16:57:58 INFO Vacuumed table table=pending_closures30192026/11/12 16:57:58 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030202026/11/12 16:57:58 INFO Received push request method=POST path=/30212026/11/12 16:57:58 INFO Received push request method=POST path=/30222026/11/12 16:57:58 INFO Vacuumed table table=commit_receipts30232026/11/12 16:57:58 INFO Received complete push request method=POST path=/30242026/11/12 16:57:58 INFO Received uploads request method=POST path=/30252026/11/12 16:57:58 INFO Completed push id=20 objects=8 duration=301.083µs30262026/11/12 16:57:58 INFO Vacuumed table table=pending_objects30272026/11/12 16:57:58 INFO Received uploads request method=POST path=/30282026/11/12 16:57:58 INFO Vacuumed table table=multipart_uploads30292026/11/12 16:57:58 INFO Received push request method=POST path=/30302026/11/12 16:57:58 INFO Vacuumed table table=closures30312026/11/12 16:57:58 INFO Received push request method=POST path=/30322026/11/12 16:57:58 INFO Vacuumed table table=objects30332026/11/12 16:57:58 INFO Received complete push request method=POST path=/30342026/11/12 16:57:58 INFO Completed push id=27 objects=11 duration=387.166µs30352026/11/12 16:57:58 INFO Vacuumed table table=pending_closures30362026/11/12 16:57:58 INFO Received complete push request method=POST path=/30372026/11/12 16:57:58 INFO Received complete upload request method=POST path=/30382026/11/12 16:57:58 INFO Received push request method=POST path=/30392026/11/12 16:57:58 INFO Completed push id=24 objects=5 duration=1.048459ms30402026/11/12 16:57:58 INFO Completed upload id=29 objects=26 duration=237µs30412026/11/12 16:57:58 INFO Received uploads request method=POST path=/30422026/11/12 16:57:58 INFO Vacuumed table table=commit_receipts30432026/11/12 16:57:58 INFO Received complete push request method=POST path=/30442026/11/12 16:57:58 INFO Completed push id=30 objects=8 duration=310.459µs30452026/11/12 16:57:58 INFO Received complete upload request method=POST path=/30462026/11/12 16:57:58 INFO Completed upload id=31 objects=5 duration=260.416µs30472026/11/12 16:57:58 INFO Received complete push request method=POST path=/30482026/11/12 16:57:58 INFO Completed push id=32 objects=3 duration=251.667µs30492026/11/12 16:57:58 INFO Aborted multipart uploads count=0 kept=030502026/11/12 16:57:58 INFO Received complete push request method=POST path=/30512026/11/12 16:57:58 INFO Vacuumed table table=pending_objects30522026/11/12 16:57:58 WARN Force mode enabled - objects will be deleted immediately without grace period30532026/11/12 16:57:58 INFO Vacuumed table table=multipart_uploads30542026/11/12 16:57:58 INFO Completed push id=1 objects=11 duration=7.407958ms30552026/11/12 16:57:58 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030562026/11/12 16:57:58 INFO Received complete push request method=POST path=/30572026/11/12 16:57:58 INFO Vacuumed table table=closures30582026/11/12 16:57:58 INFO Completed push id=2 objects=25 duration=6.080458ms30592026/11/12 16:57:58 INFO Vacuumed table table=pending_closures30602026/11/12 16:57:58 INFO Vacuumed table table=commit_receipts30612026/11/12 16:57:58 INFO Received push request method=POST path=/30622026/11/12 16:57:58 INFO Aborted multipart uploads count=0 kept=030632026/11/12 16:57:58 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030642026/11/12 16:57:58 INFO Vacuumed table table=pending_objects30652026/11/12 16:57:58 INFO Vacuumed table table=multipart_uploads30662026/11/12 16:57:58 INFO Vacuumed table table=objects30672026/11/12 16:57:58 INFO Vacuumed table table=closures30682026/11/12 16:57:58 INFO Vacuumed table table=objects30692026/11/12 16:57:58 INFO Vacuumed table table=pending_closures30702026/11/12 16:57:58 INFO Received push request method=POST path=/30712026/11/12 16:57:58 INFO Received complete upload request method=POST path=/30722026/11/12 16:57:58 INFO Completed upload id=16 objects=3 duration=385.25µs30732026/11/12 16:57:58 INFO Vacuumed table table=commit_receipts30742026/11/12 16:57:58 INFO Received push request method=POST path=/30752026/11/12 16:57:58 INFO Received push request method=POST path=/30762026/11/12 16:57:58 INFO Received push request method=POST path=/30772026/11/12 16:57:58 INFO Received complete push request method=POST path=/30782026/11/12 16:57:58 INFO Received uploads request method=POST path=/30792026/11/12 16:57:58 INFO Completed push id=14 objects=3 duration=398.208µs30802026/11/12 16:57:58 INFO Vacuumed table table=pending_objects30812026/11/12 16:57:58 INFO Received push request method=POST path=/30822026/11/12 16:57:58 INFO Vacuumed table table=multipart_uploads30832026/11/12 16:57:58 INFO Received complete push request method=POST path=/30842026/11/12 16:57:58 INFO Completed push id=6 objects=7 duration=1.223583ms30852026/11/12 16:57:58 INFO Received complete upload request method=POST path=/30862026/11/12 16:57:58 INFO Received complete push request method=POST path=/30872026/11/12 16:57:58 INFO Received uploads request method=POST path=/30882026/11/12 16:57:58 INFO Received complete push request method=POST path=/30892026/11/12 16:57:58 INFO Completed upload id=20 objects=4 duration=362.25µs30902026/11/12 16:57:58 INFO Completed push id=7 objects=20 duration=631.792µs30912026/11/12 16:57:58 INFO Completed push id=9 objects=4 duration=1.018709ms30922026/11/12 16:57:58 INFO Received push request method=POST path=/30932026/11/12 16:57:58 INFO Vacuumed table table=closures30942026/11/12 16:57:58 INFO Received uploads request method=POST path=/30952026/11/12 16:57:58 INFO Aborted multipart uploads count=0 kept=030962026/11/12 16:57:59 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030972026/11/12 16:57:59 INFO Vacuumed table table=pending_closures30982026/11/12 16:57:59 INFO Aborted multipart uploads count=0 kept=030992026/11/12 16:57:59 INFO Vacuumed table table=commit_receipts31002026/11/12 16:57:59 INFO Received complete push request method=POST path=/31012026/11/12 16:57:59 INFO Vacuumed table table=objects31022026/11/12 16:57:59 INFO Completed push id=21 objects=26 duration=7.694291ms31032026/11/12 16:57:59 WARN Force mode enabled - objects will be deleted immediately without grace period31042026/11/12 16:57:59 INFO Vacuumed table table=pending_objects31052026/11/12 16:57:59 INFO Vacuumed table table=multipart_uploads31062026/11/12 16:57:59 INFO Vacuumed table table=closures31072026/11/12 16:57:59 INFO Vacuumed table table=objects31082026/11/12 16:57:59 INFO Received push request method=POST path=/31092026/11/12 16:57:59 INFO Received push request method=POST path=/31102026/11/12 16:57:59 INFO Aborted multipart uploads count=0 kept=031112026/11/12 16:57:59 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031122026/11/12 16:57:59 INFO Vacuumed table table=pending_closures31132026/11/12 16:57:59 INFO Vacuumed table table=commit_receipts31142026/11/12 16:57:59 INFO Vacuumed table table=pending_objects31152026/11/12 16:57:59 INFO Vacuumed table table=multipart_uploads31162026/11/12 16:57:59 INFO Vacuumed table table=closures31172026/11/12 16:57:59 INFO Vacuumed table table=objects31182026/11/12 16:57:59 INFO Received complete upload request method=POST path=/31192026/11/12 16:57:59 INFO Received complete push request method=POST path=/31202026/11/12 16:57:59 INFO Completed upload id=28 objects=17 duration=324.583µs31212026/11/12 16:57:59 INFO Received complete upload request method=POST path=/31222026/11/12 16:57:59 INFO Completed push id=4 objects=4 duration=2.056375ms31232026/11/12 16:57:59 INFO Received complete push request method=POST path=/31242026/11/12 16:57:59 INFO Completed upload id=4 objects=14 duration=11.534459ms31252026/11/12 16:57:59 INFO Completed push id=29 objects=2 duration=408.75µs31262026/11/12 16:57:59 INFO Received complete push request method=POST path=/31272026/11/12 16:57:59 INFO Completed push id=5 objects=3 duration=2.446583ms31282026/11/12 16:57:59 INFO Aborted multipart uploads count=0 kept=031292026/11/12 16:57:59 INFO Received complete push request method=POST path=/31302026/11/12 16:57:59 INFO Aborted multipart uploads count=0 kept=031312026/11/12 16:57:59 INFO Completed push id=31 objects=6 duration=367.167µs31322026/11/12 16:57:59 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031332026/11/12 16:57:59 INFO Received push request method=POST path=/31342026/11/12 16:57:59 INFO Vacuumed table table=pending_closures31352026/11/12 16:57:59 INFO Vacuumed table table=commit_receipts31362026/11/12 16:57:59 WARN Force mode enabled - objects will be deleted immediately without grace period31372026/11/12 16:57:59 INFO Received complete push request method=POST path=/31382026/11/12 16:57:59 INFO Received complete push request method=POST path=/31392026/11/12 16:57:59 INFO Vacuumed table table=pending_objects31402026/11/12 16:57:59 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031412026/11/12 16:57:59 INFO Completed push id=32 objects=9 duration=322.167µs31422026/11/12 16:57:59 INFO Vacuumed table table=multipart_uploads31432026/11/12 16:57:59 INFO Completed push id=8 objects=6 duration=6.6165ms31442026/11/12 16:57:59 INFO Vacuumed table table=closures31452026/11/12 16:57:59 INFO Vacuumed table table=pending_closures31462026/11/12 16:57:59 INFO Vacuumed table table=objects31472026/11/12 16:57:59 INFO Received complete push request method=POST path=/31482026/11/12 16:57:59 INFO Received push request method=POST path=/31492026/11/12 16:57:59 INFO Vacuumed table table=commit_receipts31502026/11/12 16:57:59 INFO Completed push id=16 objects=5 duration=332.916µs31512026/11/12 16:57:59 INFO Aborted multipart uploads count=0 kept=031522026/11/12 16:57:59 WARN Force mode enabled - objects will be deleted immediately without grace period31532026/11/12 16:57:59 INFO Received uploads request method=POST path=/31542026/11/12 16:57:59 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031552026/11/12 16:57:59 INFO Received uploads request method=POST path=/31562026/11/12 16:57:59 INFO Received complete upload request method=POST path=/31572026/11/12 16:57:59 INFO Aborted multipart uploads count=0 kept=031582026/11/12 16:57:59 INFO Vacuumed table table=pending_closures31592026/11/12 16:57:59 INFO Received complete push request method=POST path=/31602026/11/12 16:57:59 INFO Vacuumed table table=pending_objects31612026/11/12 16:57:59 INFO Received push request method=POST path=/31622026/11/12 16:57:59 INFO Vacuumed table table=multipart_uploads31632026/11/12 16:57:59 INFO Vacuumed table table=commit_receipts31642026/11/12 16:57:59 INFO Vacuumed table table=pending_objects31652026/11/12 16:57:59 INFO Vacuumed table table=closures31662026/11/12 16:57:59 INFO Completed push id=6 objects=6 duration=887.125µs31672026/11/12 16:57:59 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=031682026/11/12 16:57:59 INFO Received push request method=POST path=/31692026/11/12 16:57:59 INFO Vacuumed table table=multipart_uploads31702026/11/12 16:57:59 INFO Vacuumed table table=objects31712026/11/12 16:57:59 INFO Received push request method=POST path=/31722026/11/12 16:57:59 INFO Vacuumed table table=closures31732026/11/12 16:57:59 INFO Vacuumed table table=pending_closures31742026/11/12 16:57:59 INFO Received push request method=POST path=/31752026/11/12 16:57:59 INFO Completed upload id=2 objects=26 duration=38.801875ms31762026/11/12 16:57:59 INFO Vacuumed table table=commit_receipts31772026/11/12 16:57:59 INFO Vacuumed table table=objects31782026/11/12 16:57:59 INFO Received complete upload request method=POST path=/31792026/11/12 16:57:59 INFO Completed upload id=33 objects=9 duration=357.459µs31802026/11/12 16:57:59 INFO Vacuumed table table=pending_objects31812026/11/12 16:57:59 INFO Received push request method=POST path=/31822026/11/12 16:57:59 INFO Vacuumed table table=multipart_uploads31832026/11/12 16:57:59 INFO Vacuumed table table=closures31842026/11/12 16:57:59 INFO Aborted multipart uploads count=0 kept=031852026/11/12 16:57:59 INFO Vacuumed table table=objects31862026/11/12 16:57:59 WARN Force mode enabled - objects will be deleted immediately without grace period31872026/11/12 16:57:59 INFO Aborted multipart uploads count=0 kept=031882026/11/12 16:57:59 INFO Aborted multipart uploads count=0 kept=031892026/11/12 16:57:59 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031902026/11/12 16:57:59 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031912026/11/12 16:57:59 INFO Received push request method=POST path=/31922026/11/12 16:57:59 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031932026/11/12 16:57:59 INFO Vacuumed table table=pending_closures31942026/11/12 16:57:59 INFO Vacuumed table table=pending_closures31952026/11/12 16:57:59 INFO Vacuumed table table=commit_receipts31962026/11/12 16:57:59 INFO Received push request method=POST path=/31972026/11/12 16:57:59 INFO Vacuumed table table=commit_receipts31982026/11/12 16:57:59 INFO Vacuumed table table=pending_closures31992026/11/12 16:57:59 INFO Vacuumed table table=pending_objects32002026/11/12 16:57:59 INFO Received complete push request method=POST path=/32012026/11/12 16:57:59 INFO Vacuumed table table=multipart_uploads32022026/11/12 16:57:59 INFO Vacuumed table table=commit_receipts32032026/11/12 16:57:59 INFO Vacuumed table table=closures32042026/11/12 16:57:59 INFO Vacuumed table table=objects32052026/11/12 16:57:59 INFO Vacuumed table table=pending_objects32062026/11/12 16:57:59 INFO Received push request method=POST path=/32072026/11/12 16:57:59 INFO Received uploads request method=POST path=/32082026/11/12 16:57:59 INFO Vacuumed table table=multipart_uploads32092026/11/12 16:57:59 INFO Aborted multipart uploads count=0 kept=032102026/11/12 16:57:59 INFO Received complete upload request method=POST path=/32112026/11/12 16:57:59 INFO Received uploads request method=POST path=/32122026/11/12 16:57:59 INFO Vacuumed table table=closures32132026/11/12 16:57:59 INFO Vacuumed table table=pending_objects32142026/11/12 16:57:59 INFO Completed upload id=10 objects=17 duration=624.875µs32152026/11/12 16:57:59 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=1 objects-deleted-after-grace-period=0 objects-failed-to-delete=032162026/11/12 16:57:59 INFO Received push request method=POST path=/32172026/11/12 16:57:59 INFO Vacuumed table table=multipart_uploads32182026/11/12 16:57:59 INFO Vacuumed table table=objects32192026/11/12 16:57:59 INFO Vacuumed table table=closures32202026/11/12 16:57:59 INFO Vacuumed table table=pending_closures32212026/11/12 16:57:59 INFO Aborted multipart uploads count=0 kept=032222026/11/12 16:57:59 INFO Received push request method=POST path=/32232026/11/12 16:57:59 INFO Vacuumed table table=objects32242026/11/12 16:57:59 INFO Completed push id=1 objects=10 duration=76.990875ms32252026/11/12 16:57:59 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032262026/11/12 16:57:59 INFO Vacuumed table table=pending_closures32272026/11/12 16:57:59 INFO Received push request method=POST path=/32282026/11/12 16:57:59 INFO Received uploads request method=POST path=/32292026/11/12 16:57:59 INFO Vacuumed table table=commit_receipts32302026/11/12 16:57:59 INFO Received complete push request method=POST path=/32312026/11/12 16:57:59 INFO Received push request method=POST path=/32322026/11/12 16:57:59 INFO Vacuumed table table=commit_receipts32332026/11/12 16:57:59 INFO Completed push id=14 objects=6 duration=7.0875ms32342026/11/12 16:57:59 INFO Vacuumed table table=pending_objects32352026/11/12 16:57:59 INFO Received push request method=POST path=/32362026/11/12 16:57:59 INFO Vacuumed table table=multipart_uploads32372026/11/12 16:57:59 INFO Vacuumed table table=pending_objects32382026/11/12 16:57:59 INFO Vacuumed table table=closures32392026/11/12 16:57:59 INFO Received push request method=POST path=/32402026/11/12 16:57:59 INFO Vacuumed table table=multipart_uploads32412026/11/12 16:57:59 INFO Received complete push request method=POST path=/32422026/11/12 16:57:59 INFO Received complete upload request method=POST path=/32432026/11/12 16:57:59 INFO Received push request method=POST path=/32442026/11/12 16:57:59 INFO Vacuumed table table=objects32452026/11/12 16:57:59 INFO Completed push id=4 objects=6 duration=530.458µs32462026/11/12 16:57:59 INFO Completed upload id=3 objects=5 duration=426.125µs32472026/11/12 16:57:59 INFO Received push request method=POST path=/32482026/11/12 16:57:59 INFO Received uploads request method=POST path=/32492026/11/12 16:57:59 INFO Aborted multipart uploads count=0 kept=032502026/11/12 16:57:59 INFO Aborted multipart uploads count=0 kept=032512026/11/12 16:57:59 INFO Received push request method=POST path=/32522026/11/12 16:57:59 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=1 objects-deleted-after-grace-period=0 objects-failed-to-delete=032532026/11/12 16:57:59 INFO Vacuumed table table=pending_closures32542026/11/12 16:57:59 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032552026/11/12 16:57:59 INFO Aborted multipart uploads count=0 kept=032562026/11/12 16:57:59 INFO Vacuumed table table=pending_closures32572026/11/12 16:57:59 INFO Vacuumed table table=commit_receipts32582026/11/12 16:57:59 INFO Received complete push request method=POST path=/32592026/11/12 16:57:59 INFO Vacuumed table table=commit_receipts32602026/11/12 16:57:59 WARN Force mode enabled - objects will be deleted immediately without grace period32612026/11/12 16:57:59 INFO Vacuumed table table=closures32622026/11/12 16:57:59 INFO Vacuumed table table=pending_objects32632026/11/12 16:57:59 INFO Vacuumed table table=multipart_uploads32642026/11/12 16:57:59 INFO Completed push id=5 objects=8 duration=5.293709ms32652026/11/12 16:57:59 INFO Vacuumed table table=objects32662026/11/12 16:57:59 INFO Vacuumed table table=pending_objects32672026/11/12 16:57:59 INFO Vacuumed table table=multipart_uploads32682026/11/12 16:57:59 INFO Vacuumed table table=closures32692026/11/12 16:57:59 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032702026/11/12 16:57:59 INFO Vacuumed table table=closures32712026/11/12 16:57:59 INFO Received push request method=POST path=/32722026/11/12 16:57:59 INFO Received push request method=POST path=/32732026/11/12 16:57:59 INFO Vacuumed table table=objects32742026/11/12 16:57:59 INFO Vacuumed table table=pending_closures32752026/11/12 16:57:59 INFO Vacuumed table table=objects32762026/11/12 16:57:59 INFO Received uploads request method=POST path=/32772026/11/12 16:57:59 INFO Received push request method=POST path=/32782026/11/12 16:57:59 INFO Received complete push request method=POST path=/32792026/11/12 16:57:59 INFO Vacuumed table table=commit_receipts32802026/11/12 16:57:59 INFO Received push request method=POST path=/32812026/11/12 16:57:59 INFO Completed push id=7 objects=27 duration=5.453125ms32822026/11/12 16:57:59 INFO Vacuumed table table=pending_objects32832026/11/12 16:57:59 INFO Vacuumed table table=multipart_uploads32842026/11/12 16:57:59 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."32852026/11/12 16:57:59 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.narinfo error="Access Denied."32862026/11/12 16:57:59 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."32872026/11/12 16:57:59 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."32882026/11/12 16:57:59 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."32892026/11/12 16:57:59 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."32902026/11/12 16:57:59 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."32912026/11/12 16:57:59 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."32922026/11/12 16:57:59 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."32932026/11/12 16:57:59 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."32942026/11/12 16:57:59 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."32952026/11/12 16:57:59 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."32962026/11/12 16:57:59 ERROR failed to remove object object=nar/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.nar.zst error="Access Denied."32972026/11/12 16:57:59 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."32982026/11/12 16:57:59 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."32992026/11/12 16:57:59 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."33002026/11/12 16:57:59 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."33012026/11/12 16:57:59 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."33022026/11/12 16:57:59 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."33032026/11/12 16:57:59 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."33042026/11/12 16:57:59 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."33052026/11/12 16:57:59 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."33062026/11/12 16:57:59 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."33072026/11/12 16:57:59 INFO Vacuumed table table=closures33082026/11/12 16:57:59 INFO Aborted multipart uploads count=0 kept=033092026/11/12 16:57:59 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."33102026/11/12 16:57:59 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."33112026/11/12 16:57:59 INFO Vacuumed table table=objects33122026/11/12 16:57:59 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033132026/11/12 16:57:59 INFO Received push request method=POST path=/33142026/11/12 16:57:59 INFO Received push request method=POST path=/33152026/11/12 16:57:59 INFO Received uploads request method=POST path=/33162026/11/12 16:57:59 INFO Vacuumed table table=pending_closures33172026/11/12 16:57:59 INFO Received uploads request method=POST path=/33182026/11/12 16:57:59 INFO Vacuumed table table=commit_receipts33192026/11/12 16:57:59 INFO Received uploads request method=POST path=/33202026/11/12 16:57:59 INFO Vacuumed table table=pending_objects33212026/11/12 16:57:59 INFO Received push request method=POST path=/33222026/11/12 16:57:59 INFO Received push request method=POST path=/33232026/11/12 16:57:59 INFO Received uploads request method=POST path=/33242026/11/12 16:57:59 INFO Vacuumed table table=multipart_uploads33252026/11/12 16:57:59 INFO Received uploads request method=POST path=/33262026/11/12 16:57:59 INFO Aborted multipart uploads count=0 kept=033272026/11/12 16:57:59 INFO Vacuumed table table=closures33282026/11/12 16:57:59 WARN Force mode enabled - objects will be deleted immediately without grace period33292026/11/12 16:57:59 INFO Received push request method=POST path=/33302026/11/12 16:57:59 INFO Received uploads request method=POST path=/33312026/11/12 16:57:59 INFO Received push request method=POST path=/33322026/11/12 16:57:59 INFO Received complete push request method=POST path=/33332026/11/12 16:57:59 INFO Vacuumed table table=objects33342026/11/12 16:57:59 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033352026/11/12 16:57:59 INFO Completed push id=12 objects=2 duration=929.5µs33362026/11/12 16:57:59 INFO Received push request method=POST path=/33372026/11/12 16:57:59 INFO Vacuumed table table=pending_closures33382026/11/12 16:57:59 INFO Received push request method=POST path=/33392026/11/12 16:57:59 INFO Received push request method=POST path=/33402026/11/12 16:57:59 INFO Vacuumed table table=commit_receipts33412026/11/12 16:57:59 INFO Received uploads request method=POST path=/33422026/11/12 16:57:59 INFO Aborted multipart uploads count=0 kept=033432026/11/12 16:57:59 INFO Received push request method=POST path=/33442026/11/12 16:57:59 INFO Vacuumed table table=pending_objects33452026/11/12 16:57:59 INFO Received uploads request method=POST path=/33462026/11/12 16:57:59 INFO Aborted multipart uploads count=0 kept=033472026/11/12 16:57:59 INFO Received push request method=POST path=/33482026/11/12 16:57:59 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033492026/11/12 16:57:59 INFO Vacuumed table table=multipart_uploads33502026/11/12 16:57:59 INFO Received uploads request method=POST path=/33512026/11/12 16:57:59 WARN Force mode enabled - objects will be deleted immediately without grace period33522026/11/12 16:57:59 INFO Vacuumed table table=closures33532026/11/12 16:57:59 INFO Received uploads request method=POST path=/33542026/11/12 16:57:59 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."33552026/11/12 16:57:59 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."33562026/11/12 16:57:59 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."33572026/11/12 16:57:59 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."33582026/11/12 16:57:59 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."33592026/11/12 16:57:59 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."33602026/11/12 16:57:59 INFO Received push request method=POST path=/33612026/11/12 16:57:59 INFO Vacuumed table table=objects33622026/11/12 16:57:59 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."33632026/11/12 16:57:59 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."33642026/11/12 16:57:59 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."33652026/11/12 16:57:59 INFO Vacuumed table table=pending_closures33662026/11/12 16:57:59 INFO Received complete upload request method=POST path=/33672026/11/12 16:57:59 INFO Aborted multipart uploads count=0 kept=033682026/11/12 16:57:59 INFO Aborted multipart uploads count=0 kept=033692026/11/12 16:57:59 INFO Aborted multipart uploads count=0 kept=033702026/11/12 16:57:59 INFO Vacuumed table table=commit_receipts33712026/11/12 16:57:59 INFO Completed upload id=8 objects=3 duration=6.110916ms33722026/11/12 16:57:59 WARN Force mode enabled - objects will be deleted immediately without grace period33732026/11/12 16:57:59 WARN Force mode enabled - objects will be deleted immediately without grace period33742026/11/12 16:57:59 INFO Vacuumed table table=pending_objects33752026/11/12 16:57:59 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033762026/11/12 16:57:59 INFO Received uploads request method=POST path=/33772026/11/12 16:57:59 INFO Received uploads request method=POST path=/33782026/11/12 16:57:59 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033792026/11/12 16:57:59 INFO Received uploads request method=POST path=/33802026/11/12 16:57:59 INFO Vacuumed table table=multipart_uploads33812026/11/12 16:57:59 INFO Vacuumed table table=pending_closures33822026/11/12 16:57:59 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=9 objects-failed-to-delete=033832026/11/12 16:57:59 INFO Received push request method=POST path=/33842026/11/12 16:57:59 INFO Received uploads request method=POST path=/33852026/11/12 16:57:59 INFO Vacuumed table table=pending_closures33862026/11/12 16:57:59 INFO Vacuumed table table=closures33872026/11/12 16:57:59 INFO Received complete push request method=POST path=/33882026/11/12 16:58:00 INFO Received complete push request method=POST path=/33892026/11/12 16:58:00 INFO Vacuumed table table=commit_receipts33902026/11/12 16:58:00 INFO Vacuumed table table=commit_receipts33912026/11/12 16:58:00 INFO Vacuumed table table=objects33922026/11/12 16:58:00 INFO Completed push id=6 objects=6 duration=676.958µs33932026/11/12 16:58:00 INFO Completed push id=4 objects=3 duration=723.625µs33942026/11/12 16:58:00 INFO Aborted multipart uploads count=0 kept=033952026/11/12 16:58:00 INFO Vacuumed table table=pending_objects33962026/11/12 16:58:00 INFO Vacuumed table table=pending_closures33972026/11/12 16:58:00 INFO Vacuumed table table=pending_objects33982026/11/12 16:58:00 INFO Vacuumed table table=multipart_uploads33992026/11/12 16:58:00 INFO Vacuumed table table=commit_receipts34002026/11/12 16:58:00 INFO Vacuumed table table=multipart_uploads34012026/11/12 16:58:00 INFO Aborted multipart uploads count=0 kept=034022026/11/12 16:58:00 INFO Vacuumed table table=closures34032026/11/12 16:58:00 INFO Vacuumed table table=closures34042026/11/12 16:58:00 WARN Force mode enabled - objects will be deleted immediately without grace period34052026/11/12 16:58:00 INFO Received push request method=POST path=/34062026/11/12 16:58:00 INFO Received push request method=POST path=/34072026/11/12 16:58:00 INFO Received push request method=POST path=/34082026/11/12 16:58:00 INFO Vacuumed table table=objects34092026/11/12 16:58:00 INFO Vacuumed table table=objects34102026/11/12 16:58:00 INFO Vacuumed table table=pending_objects34112026/11/12 16:58:00 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034122026/11/12 16:58:00 WARN Force mode enabled - objects will be deleted immediately without grace period34132026/11/12 16:58:00 INFO Received complete upload request method=POST path=/34142026/11/12 16:58:00 INFO Received push request method=POST path=/34152026/11/12 16:58:00 INFO Vacuumed table table=multipart_uploads34162026/11/12 16:58:00 INFO Vacuumed table table=pending_closures34172026/11/12 16:58:00 INFO Completed upload id=3 objects=9 duration=1.25775ms34182026/11/12 16:58:00 INFO Received push request method=POST path=/34192026/11/12 16:58:00 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=5 objects-failed-to-delete=034202026/11/12 16:58:00 INFO Received push request method=POST path=/34212026/11/12 16:58:00 INFO Vacuumed table table=commit_receipts34222026/11/12 16:58:00 INFO Received push request method=POST path=/34232026/11/12 16:58:00 INFO Vacuumed table table=closures34242026/11/12 16:58:00 INFO Received uploads request method=POST path=/34252026/11/12 16:58:00 INFO Vacuumed table table=pending_objects34262026/11/12 16:58:00 INFO Received push request method=POST path=/34272026/11/12 16:58:00 INFO Vacuumed table table=multipart_uploads34282026/11/12 16:58:00 INFO Received uploads request method=POST path=/34292026/11/12 16:58:00 INFO Vacuumed table table=objects34302026/11/12 16:58:00 INFO Received uploads request method=POST path=/34312026/11/12 16:58:00 INFO Vacuumed table table=closures34322026/11/12 16:58:00 INFO Received complete push request method=POST path=/34332026/11/12 16:58:00 INFO Aborted multipart uploads count=0 kept=034342026/11/12 16:58:00 INFO Received complete push request method=POST path=/34352026/11/12 16:58:00 INFO Vacuumed table table=pending_closures34362026/11/12 16:58:00 INFO Received uploads request method=POST path=/34372026/11/12 16:58:00 INFO Vacuumed table table=objects34382026/11/12 16:58:00 INFO Vacuumed table table=commit_receipts34392026/11/12 16:58:00 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034402026/11/12 16:58:00 INFO Received push request method=POST path=/34412026/11/12 16:58:00 INFO Received complete upload request method=POST path=/34422026/11/12 16:58:00 INFO Vacuumed table table=pending_closures34432026/11/12 16:58:00 INFO Received uploads request method=POST path=/34442026/11/12 16:58:00 INFO Received push request method=POST path=/34452026/11/12 16:58:00 INFO Completed upload id=19 objects=2 duration=1.060875ms34462026/11/12 16:58:00 INFO Vacuumed table table=commit_receipts34472026/11/12 16:58:00 INFO Received push request method=POST path=/34482026/11/12 16:58:00 INFO Received push request method=POST path=/34492026/11/12 16:58:00 INFO Received uploads request method=POST path=/34502026/11/12 16:58:00 INFO Completed push id=1 objects=7 duration=45.736542ms34512026/11/12 16:58:00 INFO Completed push id=1 objects=24 duration=45.849166ms34522026/11/12 16:58:00 INFO Vacuumed table table=pending_objects34532026/11/12 16:58:00 INFO Received complete push request method=POST path=/34542026/11/12 16:58:00 INFO Aborted multipart uploads count=0 kept=034552026/11/12 16:58:00 INFO Received uploads request method=POST path=/34562026/11/12 16:58:00 INFO Received uploads request method=POST path=/34572026/11/12 16:58:00 INFO Received complete push request method=POST path=/34582026/11/12 16:58:00 INFO Vacuumed table table=pending_objects34592026/11/12 16:58:00 INFO Vacuumed table table=multipart_uploads34602026/11/12 16:58:00 INFO Completed push id=33 objects=7 duration=600.375µs34612026/11/12 16:58:00 INFO Completed push id=5 objects=6 duration=615.5µs34622026/11/12 16:58:00 WARN Force mode enabled - objects will be deleted immediately without grace period34632026/11/12 16:58:00 INFO Received push request method=POST path=/34642026/11/12 16:58:00 INFO Vacuumed table table=multipart_uploads34652026/11/12 16:58:00 INFO Received complete push request method=POST path=/34662026/11/12 16:58:00 INFO Vacuumed table table=closures34672026/11/12 16:58:00 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034682026/11/12 16:58:00 INFO Received complete upload request method=POST path=/34692026/11/12 16:58:00 INFO Aborted multipart uploads count=0 kept=034702026/11/12 16:58:00 INFO Vacuumed table table=closures34712026/11/12 16:58:00 INFO Completed upload id=26 objects=3 duration=7.104416ms34722026/11/12 16:58:00 INFO Vacuumed table table=pending_closures34732026/11/12 16:58:00 INFO Completed push id=10 objects=15 duration=7.148666ms34742026/11/12 16:58:00 WARN Force mode enabled - objects will be deleted immediately without grace period34752026/11/12 16:58:00 INFO Vacuumed table table=objects34762026/11/12 16:58:00 INFO Aborted multipart uploads count=0 kept=034772026/11/12 16:58:00 INFO Vacuumed table table=commit_receipts34782026/11/12 16:58:00 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034792026/11/12 16:58:00 INFO Aborted multipart uploads count=0 kept=034802026/11/12 16:58:00 WARN Force mode enabled - objects will be deleted immediately without grace period34812026/11/12 16:58:00 INFO Received complete upload request method=POST path=/34822026/11/12 16:58:00 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034832026/11/12 16:58:00 INFO Received complete push request method=POST path=/34842026/11/12 16:58:00 INFO Completed upload id=17 objects=25 duration=1.05775ms34852026/11/12 16:58:00 INFO Vacuumed table table=pending_objects34862026/11/12 16:58:00 INFO Vacuumed table table=pending_closures34872026/11/12 16:58:00 INFO Vacuumed table table=commit_receipts34882026/11/12 16:58:00 INFO Completed push id=22 objects=3 duration=9.064834ms34892026/11/12 16:58:00 INFO Vacuumed table table=multipart_uploads34902026/11/12 16:58:00 INFO Vacuumed table table=pending_objects34912026/11/12 16:58:00 INFO Vacuumed table table=closures34922026/11/12 16:58:00 INFO Vacuumed table table=objects34932026/11/12 16:58:00 INFO Received push request method=POST path=/34942026/11/12 16:58:00 INFO Vacuumed table table=multipart_uploads34952026/11/12 16:58:00 INFO Aborted multipart uploads count=0 kept=034962026/11/12 16:58:00 INFO Aborted multipart uploads count=0 kept=034972026/11/12 16:58:00 INFO Vacuumed table table=objects34982026/11/12 16:58:00 INFO Vacuumed table table=pending_closures34992026/11/12 16:58:00 INFO Vacuumed table table=closures35002026/11/12 16:58:00 INFO Received complete upload request method=POST path=/35012026/11/12 16:58:00 INFO Vacuumed table table=objects35022026/11/12 16:58:00 WARN Force mode enabled - objects will be deleted immediately without grace period35032026/11/12 16:58:00 INFO Completed upload id=21 objects=2 duration=519.542µs35042026/11/12 16:58:00 WARN Force mode enabled - objects will be deleted immediately without grace period35052026/11/12 16:58:00 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=035062026/11/12 16:58:00 INFO Aborted multipart uploads count=0 kept=035072026/11/12 16:58:00 INFO Aborted multipart uploads count=0 kept=035082026/11/12 16:58:00 INFO Vacuumed table table=commit_receipts35092026/11/12 16:58:00 WARN Force mode enabled - objects will be deleted immediately without grace period35102026/11/12 16:58:00 WARN Force mode enabled - objects will be deleted immediately without grace period35112026/11/12 16:58:00 INFO Vacuumed table table=pending_closures35122026/11/12 16:58:00 INFO Vacuumed table table=commit_receipts35132026/11/12 16:58:00 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035142026/11/12 16:58:00 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035152026/11/12 16:58:00 INFO Vacuumed table table=pending_closures35162026/11/12 16:58:00 INFO Vacuumed table table=pending_closures35172026/11/12 16:58:00 INFO Vacuumed table table=commit_receipts35182026/11/12 16:58:00 INFO Vacuumed table table=commit_receipts35192026/11/12 16:58:00 INFO Vacuumed table table=pending_objects35202026/11/12 16:58:00 INFO Vacuumed table table=multipart_uploads35212026/11/12 16:58:00 INFO Vacuumed table table=pending_objects35222026/11/12 16:58:00 INFO Vacuumed table table=multipart_uploads35232026/11/12 16:58:00 INFO Vacuumed table table=pending_objects35242026/11/12 16:58:00 INFO Vacuumed table table=closures35252026/11/12 16:58:00 INFO Vacuumed table table=pending_objects35262026/11/12 16:58:00 INFO Vacuumed table table=multipart_uploads35272026/11/12 16:58:00 INFO Vacuumed table table=closures35282026/11/12 16:58:00 INFO Vacuumed table table=objects35292026/11/12 16:58:00 INFO Aborted multipart uploads count=0 kept=035302026/11/12 16:58:00 INFO Vacuumed table table=closures35312026/11/12 16:58:00 INFO Vacuumed table table=objects35322026/11/12 16:58:00 INFO Vacuumed table table=multipart_uploads35332026/11/12 16:58:00 INFO Vacuumed table table=objects35342026/11/12 16:58:00 WARN Force mode enabled - objects will be deleted immediately without grace period35352026/11/12 16:58:00 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035362026/11/12 16:58:00 INFO Received push request method=POST path=/35372026/11/12 16:58:00 INFO Vacuumed table table=pending_closures35382026/11/12 16:58:00 INFO Received push request method=POST path=/35392026/11/12 16:58:00 INFO Vacuumed table table=commit_receipts35402026/11/12 16:58:00 INFO Vacuumed table table=pending_objects35412026/11/12 16:58:00 INFO Vacuumed table table=multipart_uploads35422026/11/12 16:58:00 INFO Vacuumed table table=closures35432026/11/12 16:58:00 INFO Vacuumed table table=closures35442026/11/12 16:58:00 INFO Vacuumed table table=objects35452026/11/12 16:58:00 INFO Received push request method=POST path=/35462026/11/12 16:58:00 INFO Received uploads request method=POST path=/35472026/11/12 16:58:00 INFO Aborted multipart uploads count=0 kept=035482026/11/12 16:58:00 INFO Received complete push request method=POST path=/35492026/11/12 16:58:00 WARN Force mode enabled - objects will be deleted immediately without grace period35502026/11/12 16:58:00 INFO Completed push id=8 objects=13 duration=607.833µs35512026/11/12 16:58:00 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."35522026/11/12 16:58:00 INFO Vacuumed table table=objects35532026/11/12 16:58:00 INFO Received uploads request method=POST path=/35542026/11/12 16:58:00 INFO Aborted multipart uploads count=0 kept=035552026/11/12 16:58:00 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035562026/11/12 16:58:00 INFO Received push request method=POST path=/35572026/11/12 16:58:00 INFO Received uploads request method=POST path=/35582026/11/12 16:58:00 INFO Received push request method=POST path=/35592026/11/12 16:58:00 INFO Received uploads request method=POST path=/35602026/11/12 16:58:00 INFO Vacuumed table table=pending_closures35612026/11/12 16:58:00 INFO Received push request method=POST path=/35622026/11/12 16:58:00 INFO Received uploads request method=POST path=/35632026/11/12 16:58:00 INFO Received complete upload request method=POST path=/35642026/11/12 16:58:00 INFO Received uploads request method=POST path=/35652026/11/12 16:58:00 INFO Completed upload id=14 objects=8 duration=863µs35662026/11/12 16:58:00 INFO Received complete upload request method=POST path=/35672026/11/12 16:58:00 INFO Vacuumed table table=commit_receipts35682026/11/12 16:58:00 INFO Completed upload id=2 objects=10 duration=627.166µs35692026/11/12 16:58:00 INFO Received uploads request method=POST path=/35702026/11/12 16:58:00 INFO Received push request method=POST path=/35712026/11/12 16:58:00 INFO Aborted multipart uploads count=0 kept=035722026/11/12 16:58:00 WARN Force mode enabled - objects will be deleted immediately without grace period35732026/11/12 16:58:00 INFO Vacuumed table table=pending_objects35742026/11/12 16:58:00 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=035752026/11/12 16:58:00 INFO Vacuumed table table=multipart_uploads35762026/11/12 16:58:00 INFO Vacuumed table table=pending_closures35772026/11/12 16:58:00 INFO Vacuumed table table=commit_receipts35782026/11/12 16:58:00 INFO Received complete push request method=POST path=/35792026/11/12 16:58:00 INFO Vacuumed table table=closures35802026/11/12 16:58:00 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."35812026/11/12 16:58:00 INFO Completed push id=3 objects=21 duration=10.758375ms35822026/11/12 16:58:00 INFO Vacuumed table table=pending_objects35832026/11/12 16:58:00 INFO Vacuumed table table=multipart_uploads35842026/11/12 16:58:00 INFO Received uploads request method=POST path=/35852026/11/12 16:58:00 INFO Vacuumed table table=closures35862026/11/12 16:58:00 INFO Received uploads request method=POST path=/35872026/11/12 16:58:00 INFO Received complete push request method=POST path=/35882026/11/12 16:58:00 INFO Vacuumed table table=objects35892026/11/12 16:58:00 INFO Received push request method=POST path=/35902026/11/12 16:58:00 INFO Vacuumed table table=objects35912026/11/12 16:58:00 INFO Received uploads request method=POST path=/35922026/11/12 16:58:00 INFO Received uploads request method=POST path=/35932026/11/12 16:58:00 INFO Received uploads request method=POST path=/35942026/11/12 16:58:00 INFO Aborted multipart uploads count=0 kept=035952026/11/12 16:58:00 INFO Received uploads request method=POST path=/35962026/11/12 16:58:00 INFO Received complete upload request method=POST path=/35972026/11/12 16:58:00 INFO Received complete push request method=POST path=/35982026/11/12 16:58:00 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."35992026/11/12 16:58:00 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."36002026/11/12 16:58:00 INFO Completed push id=1 objects=5 duration=52.6375ms36012026/11/12 16:58:00 INFO Received uploads request method=POST path=/36022026/11/12 16:58:00 INFO Completed push id=15 objects=5 duration=642.5µs36032026/11/12 16:58:00 INFO Completed upload id=5 objects=7 duration=1.143084ms36042026/11/12 16:58:00 INFO Received push request method=POST path=/36052026/11/12 16:58:00 INFO Received uploads request method=POST path=/36062026/11/12 16:58:00 INFO Aborted multipart uploads count=0 kept=036072026/11/12 16:58:00 INFO Received uploads request method=POST path=/36082026/11/12 16:58:00 WARN Force mode enabled - objects will be deleted immediately without grace period36092026/11/12 16:58:00 INFO Received push request method=POST path=/36102026/11/12 16:58:00 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=3 objects-failed-to-delete=036112026/11/12 16:58:00 INFO Vacuumed table table=pending_closures36122026/11/12 16:58:00 INFO Vacuumed table table=commit_receipts36132026/11/12 16:58:00 INFO Vacuumed table table=pending_objects36142026/11/12 16:58:00 INFO Vacuumed table table=multipart_uploads36152026/11/12 16:58:00 INFO Vacuumed table table=closures36162026/11/12 16:58:00 INFO Received complete push request method=POST path=/36172026/11/12 16:58:00 INFO Vacuumed table table=objects36182026/11/12 16:58:00 INFO Received uploads request method=POST path=/36192026/11/12 16:58:00 INFO Completed push id=7 objects=24 duration=1.295167ms36202026/11/12 16:58:00 INFO Received complete upload request method=POST path=/36212026/11/12 16:58:00 INFO Received uploads request method=POST path=/36222026/11/12 16:58:00 INFO Completed upload id=21 objects=7 duration=975.291µs36232026/11/12 16:58:00 INFO Aborted multipart uploads count=0 kept=036242026/11/12 16:58:00 INFO Received uploads request method=POST path=/36252026/11/12 16:58:01 WARN Force mode enabled - objects will be deleted immediately without grace period36262026/11/12 16:58:01 INFO Received complete upload request method=POST path=/36272026/11/12 16:58:01 INFO Aborted multipart uploads count=0 kept=036282026/11/12 16:58:01 INFO Received push request method=POST path=/36292026/11/12 16:58:01 INFO Received uploads request method=POST path=/36302026/11/12 16:58:01 INFO Received complete push request method=POST path=/36312026/11/12 16:58:01 INFO Completed upload id=6 objects=9 duration=504.166µs36322026/11/12 16:58:01 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036332026/11/12 16:58:01 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036342026/11/12 16:58:01 INFO Received push request method=POST path=/36352026/11/12 16:58:01 INFO Completed push id=17 objects=22 duration=682.667µs36362026/11/12 16:58:01 INFO Received uploads request method=POST path=/36372026/11/12 16:58:01 INFO Vacuumed table table=pending_closures36382026/11/12 16:58:01 INFO Received uploads request method=POST path=/36392026/11/12 16:58:01 INFO Received complete upload request method=POST path=/36402026/11/12 16:58:01 INFO Vacuumed table table=pending_closures36412026/11/12 16:58:01 INFO Received uploads request method=POST path=/36422026/11/12 16:58:01 INFO Received uploads request method=POST path=/36432026/11/12 16:58:01 INFO Vacuumed table table=commit_receipts36442026/11/12 16:58:01 INFO Vacuumed table table=commit_receipts36452026/11/12 16:58:01 INFO Vacuumed table table=pending_objects36462026/11/12 16:58:01 INFO Completed upload id=10 objects=4 duration=1.079667ms36472026/11/12 16:58:01 INFO Received complete upload request method=POST path=/36482026/11/12 16:58:01 INFO Received complete upload request method=POST path=/36492026/11/12 16:58:01 INFO Received complete push request method=POST path=/36502026/11/12 16:58:01 INFO Received complete push request method=POST path=/36512026/11/12 16:58:01 INFO Vacuumed table table=pending_objects36522026/11/12 16:58:01 INFO Vacuumed table table=multipart_uploads36532026/11/12 16:58:01 INFO Received complete upload request method=POST path=/36542026/11/12 16:58:01 INFO Received push request method=POST path=/36552026/11/12 16:58:01 INFO Completed upload id=3 objects=5 duration=335.084µs36562026/11/12 16:58:01 INFO Completed upload id=27 objects=4 duration=422.583µs36572026/11/12 16:58:01 INFO Completed push id=13 objects=2 duration=342.625µs36582026/11/12 16:58:01 INFO Vacuumed table table=multipart_uploads36592026/11/12 16:58:01 INFO Vacuumed table table=closures36602026/11/12 16:58:01 INFO Completed upload id=16 objects=17 duration=799.25µs36612026/11/12 16:58:01 INFO Received push request method=POST path=/36622026/11/12 16:58:01 INFO Received complete push request method=POST path=/36632026/11/12 16:58:01 INFO Received complete upload request method=POST path=/36642026/11/12 16:58:01 INFO Aborted multipart uploads count=0 kept=036652026/11/12 16:58:01 INFO Vacuumed table table=objects36662026/11/12 16:58:01 INFO Received push request method=POST path=/36672026/11/12 16:58:01 INFO Aborted multipart uploads count=0 kept=036682026/11/12 16:58:01 INFO Completed upload id=27 objects=13 duration=356.083µs36692026/11/12 16:58:01 INFO Completed push id=13 objects=11 duration=540µs36702026/11/12 16:58:01 INFO Vacuumed table table=closures36712026/11/12 16:58:01 INFO Received complete push request method=POST path=/36722026/11/12 16:58:01 WARN Force mode enabled - objects will be deleted immediately without grace period36732026/11/12 16:58:01 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=036742026/11/12 16:58:01 INFO Received uploads request method=POST path=/36752026/11/12 16:58:01 INFO Received uploads request method=POST path=/36762026/11/12 16:58:01 INFO Aborted multipart uploads count=0 kept=036772026/11/12 16:58:01 INFO Completed push id=3 objects=5 duration=45.246792ms36782026/11/12 16:58:01 INFO Received complete push request method=POST path=/36792026/11/12 16:58:01 INFO Vacuumed table table=objects36802026/11/12 16:58:01 INFO Received uploads request method=POST path=/36812026/11/12 16:58:01 INFO Completed push id=13 objects=12 duration=1.289125ms36822026/11/12 16:58:01 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036832026/11/12 16:58:01 INFO Vacuumed table table=pending_closures36842026/11/12 16:58:01 INFO Completed push id=3 objects=5 duration=471.042µs36852026/11/12 16:58:01 INFO Vacuumed table table=commit_receipts36862026/11/12 16:58:01 INFO Vacuumed table table=pending_closures36872026/11/12 16:58:01 INFO Aborted multipart uploads count=0 kept=036882026/11/12 16:58:01 INFO Vacuumed table table=commit_receipts36892026/11/12 16:58:01 INFO Aborted multipart uploads count=0 kept=036902026/11/12 16:58:01 INFO Received push request method=POST path=/36912026/11/12 16:58:01 INFO Received uploads request method=POST path=/36922026/11/12 16:58:01 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=036932026/11/12 16:58:01 INFO Received complete push request method=POST path=/36942026/11/12 16:58:01 INFO Vacuumed table table=pending_objects36952026/11/12 16:58:01 INFO Received push request method=POST path=/36962026/11/12 16:58:01 INFO Vacuumed table table=multipart_uploads36972026/11/12 16:58:01 INFO Vacuumed table table=pending_objects36982026/11/12 16:58:01 INFO Vacuumed table table=pending_closures36992026/11/12 16:58:01 WARN Force mode enabled - objects will be deleted immediately without grace period37002026/11/12 16:58:01 INFO Completed push id=9 objects=9 duration=1.279709ms37012026/11/12 16:58:01 INFO Aborted multipart uploads count=0 kept=037022026/11/12 16:58:01 INFO Vacuumed table table=closures37032026/11/12 16:58:01 INFO Vacuumed table table=multipart_uploads37042026/11/12 16:58:01 INFO Vacuumed table table=commit_receipts37052026/11/12 16:58:01 INFO Vacuumed table table=closures37062026/11/12 16:58:01 INFO Vacuumed table table=pending_objects37072026/11/12 16:58:01 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037082026/11/12 16:58:01 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037092026/11/12 16:58:01 INFO Vacuumed table table=objects37102026/11/12 16:58:01 INFO Vacuumed table table=multipart_uploads37112026/11/12 16:58:01 INFO Vacuumed table table=objects37122026/11/12 16:58:01 INFO Vacuumed table table=pending_closures37132026/11/12 16:58:01 INFO Received uploads request method=POST path=/37142026/11/12 16:58:01 INFO Vacuumed table table=closures37152026/11/12 16:58:01 INFO Vacuumed table table=commit_receipts37162026/11/12 16:58:01 INFO Received push request method=POST path=/37172026/11/12 16:58:01 INFO Vacuumed table table=pending_objects37182026/11/12 16:58:01 INFO Vacuumed table table=multipart_uploads37192026/11/12 16:58:01 INFO Vacuumed table table=objects37202026/11/12 16:58:01 INFO Received push request method=POST path=/37212026/11/12 16:58:01 INFO Vacuumed table table=closures37222026/11/12 16:58:01 INFO Vacuumed table table=pending_closures37232026/11/12 16:58:01 INFO Received uploads request method=POST path=/37242026/11/12 16:58:01 INFO Received push request method=POST path=/37252026/11/12 16:58:01 INFO Vacuumed table table=objects37262026/11/12 16:58:01 INFO Vacuumed table table=commit_receipts37272026/11/12 16:58:01 INFO Received complete upload request method=POST path=/37282026/11/12 16:58:01 INFO Received uploads request method=POST path=/37292026/11/12 16:58:01 INFO Received complete push request method=POST path=/37302026/11/12 16:58:01 INFO Received complete upload request method=POST path=/37312026/11/12 16:58:01 INFO Completed upload id=21 objects=8 duration=421.417µs37322026/11/12 16:58:01 INFO Received uploads request method=POST path=/37332026/11/12 16:58:01 INFO Received complete upload request method=POST path=/37342026/11/12 16:58:01 INFO Completed push id=14 objects=3 duration=913.5µs37352026/11/12 16:58:01 INFO Completed upload id=5 objects=5 duration=886.709µs37362026/11/12 16:58:01 INFO Received complete upload request method=POST path=/37372026/11/12 16:58:01 INFO Completed upload id=29 objects=4 duration=602.416µs37382026/11/12 16:58:01 INFO Received complete push request method=POST path=/37392026/11/12 16:58:01 INFO Received uploads request method=POST path=/37402026/11/12 16:58:01 INFO Received complete push request method=POST path=/37412026/11/12 16:58:01 INFO Vacuumed table table=pending_objects37422026/11/12 16:58:01 INFO Completed upload id=11 objects=14 duration=601.375µs37432026/11/12 16:58:01 INFO Received uploads request method=POST path=/37442026/11/12 16:58:01 INFO Completed push id=22 objects=6 duration=353.334µs37452026/11/12 16:58:01 INFO Completed push id=4 objects=4 duration=497.375µs37462026/11/12 16:58:01 INFO Vacuumed table table=multipart_uploads37472026/11/12 16:58:01 INFO Received uploads request method=POST path=/37482026/11/12 16:58:01 INFO Received push request method=POST path=/37492026/11/12 16:58:01 INFO Received uploads request method=POST path=/37502026/11/12 16:58:01 INFO Vacuumed table table=closures37512026/11/12 16:58:01 INFO Received complete push request method=POST path=/37522026/11/12 16:58:01 INFO Received push request method=POST path=/37532026/11/12 16:58:01 INFO Received push request method=POST path=/37542026/11/12 16:58:01 INFO Received push request method=POST path=/37552026/11/12 16:58:01 INFO Vacuumed table table=objects37562026/11/12 16:58:01 INFO Completed push id=24 objects=8 duration=383.292µs37572026/11/12 16:58:01 INFO Received uploads request method=POST path=/37582026/11/12 16:58:01 INFO Received complete push request method=POST path=/37592026/11/12 16:58:01 INFO Received push request method=POST path=/37602026/11/12 16:58:01 INFO Received complete push request method=POST path=/37612026/11/12 16:58:01 INFO Received complete upload request method=POST path=/37622026/11/12 16:58:01 INFO Completed push id=5 objects=3 duration=560.584µs37632026/11/12 16:58:01 INFO Received uploads request method=POST path=/37642026/11/12 16:58:01 INFO Aborted multipart uploads count=0 kept=037652026/11/12 16:58:01 INFO Completed push id=2 objects=10 duration=961µs37662026/11/12 16:58:01 INFO Completed upload id=25 objects=9 duration=286.458µs37672026/11/12 16:58:01 INFO Received complete push request method=POST path=/37682026/11/12 16:58:01 INFO Aborted multipart uploads count=0 kept=037692026/11/12 16:58:01 INFO Received complete push request method=POST path=/37702026/11/12 16:58:01 WARN Force mode enabled - objects will be deleted immediately without grace period37712026/11/12 16:58:01 INFO Aborted multipart uploads count=0 kept=037722026/11/12 16:58:01 INFO Received complete upload request method=POST path=/37732026/11/12 16:58:01 INFO Received push request method=POST path=/37742026/11/12 16:58:01 INFO Completed push id=16 objects=14 duration=1.772959ms37752026/11/12 16:58:01 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=4 objects-deleted-after-grace-period=0 objects-failed-to-delete=037762026/11/12 16:58:01 INFO Completed push id=21 objects=9 duration=513.167µs37772026/11/12 16:58:01 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037782026/11/12 16:58:01 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=037792026/11/12 16:58:01 INFO Completed upload id=27 objects=3 duration=270.042µs37802026/11/12 16:58:01 INFO Received complete push request method=POST path=/37812026/11/12 16:58:01 INFO Aborted multipart uploads count=0 kept=037822026/11/12 16:58:01 INFO Vacuumed table table=pending_closures37832026/11/12 16:58:01 INFO Vacuumed table table=pending_closures37842026/11/12 16:58:01 INFO Vacuumed table table=commit_receipts37852026/11/12 16:58:01 INFO Completed push id=8 objects=7 duration=496.042µs37862026/11/12 16:58:01 WARN Force mode enabled - objects will be deleted immediately without grace period37872026/11/12 16:58:01 INFO Received uploads request method=POST path=/37882026/11/12 16:58:01 INFO Received complete upload request method=POST path=/37892026/11/12 16:58:01 INFO Received complete push request method=POST path=/37902026/11/12 16:58:01 INFO Vacuumed table table=pending_objects37912026/11/12 16:58:01 INFO Completed upload id=28 objects=6 duration=334.5µs37922026/11/12 16:58:01 INFO Vacuumed table table=multipart_uploads37932026/11/12 16:58:01 INFO Completed push id=11 objects=6 duration=804.708µs37942026/11/12 16:58:01 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037952026/11/12 16:58:01 INFO Received uploads request method=POST path=/37962026/11/12 16:58:01 INFO Received complete push request method=POST path=/37972026/11/12 16:58:01 INFO Received push request method=POST path=/37982026/11/12 16:58:01 INFO Vacuumed table table=pending_closures37992026/11/12 16:58:01 INFO Vacuumed table table=commit_receipts38002026/11/12 16:58:01 INFO Completed push id=30 objects=4 duration=368.791µs38012026/11/12 16:58:01 INFO Vacuumed table table=pending_closures38022026/11/12 16:58:01 INFO Vacuumed table table=commit_receipts38032026/11/12 16:58:01 INFO Received push request method=POST path=/38042026/11/12 16:58:01 INFO Vacuumed table table=closures38052026/11/12 16:58:01 INFO Received complete push request method=POST path=/38062026/11/12 16:58:01 INFO Vacuumed table table=pending_objects38072026/11/12 16:58:01 INFO Vacuumed table table=pending_objects38082026/11/12 16:58:01 INFO Completed push id=31 objects=7 duration=375.584µs38092026/11/12 16:58:01 INFO Vacuumed table table=objects38102026/11/12 16:58:01 INFO Vacuumed table table=multipart_uploads38112026/11/12 16:58:01 INFO Vacuumed table table=multipart_uploads38122026/11/12 16:58:01 INFO Received complete upload request method=POST path=/38132026/11/12 16:58:01 INFO Received complete upload request method=POST path=/38142026/11/12 16:58:01 INFO Vacuumed table table=commit_receipts38152026/11/12 16:58:01 INFO Received uploads request method=POST path=/38162026/11/12 16:58:01 INFO Vacuumed table table=closures38172026/11/12 16:58:01 INFO Completed upload id=32 objects=5 duration=316.417µs38182026/11/12 16:58:01 INFO Completed upload id=13 objects=8 duration=353.875µs38192026/11/12 16:58:01 INFO Received complete push request method=POST path=/38202026/11/12 16:58:01 INFO Vacuumed table table=objects38212026/11/12 16:58:01 INFO Vacuumed table table=closures38222026/11/12 16:58:01 INFO Received complete push request method=POST path=/38232026/11/12 16:58:01 INFO Completed push id=14 objects=3 duration=1.219583ms38242026/11/12 16:58:01 INFO Received uploads request method=POST path=/38252026/11/12 16:58:01 INFO Aborted multipart uploads count=0 kept=038262026/11/12 16:58:01 INFO Vacuumed table table=objects38272026/11/12 16:58:01 INFO Completed push id=20 objects=13 duration=316.709µs38282026/11/12 16:58:01 INFO Received complete push request method=POST path=/38292026/11/12 16:58:01 INFO Vacuumed table table=pending_objects38302026/11/12 16:58:01 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038312026/11/12 16:58:01 INFO Received uploads request method=POST path=/38322026/11/12 16:58:01 INFO Vacuumed table table=multipart_uploads38332026/11/12 16:58:01 INFO Vacuumed table table=pending_closures38342026/11/12 16:58:01 INFO Completed push id=5 objects=6 duration=711.208µs38352026/11/12 16:58:01 INFO Received complete push request method=POST path=/38362026/11/12 16:58:01 INFO Received push request method=POST path=/38372026/11/12 16:58:01 INFO Vacuumed table table=commit_receipts38382026/11/12 16:58:01 INFO Received complete upload request method=POST path=/38392026/11/12 16:58:01 INFO Completed push id=22 objects=4 duration=450.167µs38402026/11/12 16:58:01 INFO Aborted multipart uploads count=0 kept=038412026/11/12 16:58:01 INFO Vacuumed table table=pending_objects38422026/11/12 16:58:01 INFO Vacuumed table table=closures38432026/11/12 16:58:01 INFO Vacuumed table table=multipart_uploads38442026/11/12 16:58:01 INFO Completed upload id=9 objects=16 duration=770.959µs38452026/11/12 16:58:01 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=8 objects-deleted-after-grace-period=8 objects-failed-to-delete=038462026/11/12 16:58:01 WARN Force mode enabled - objects will be deleted immediately without grace period38472026/11/12 16:58:01 INFO Vacuumed table table=closures38482026/11/12 16:58:01 INFO Received push request method=POST path=/38492026/11/12 16:58:01 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038502026/11/12 16:58:01 INFO Vacuumed table table=pending_closures38512026/11/12 16:58:01 INFO Vacuumed table table=objects38522026/11/12 16:58:01 INFO Vacuumed table table=commit_receipts38532026/11/12 16:58:01 INFO Vacuumed table table=objects38542026/11/12 16:58:01 INFO Received push request method=POST path=/38552026/11/12 16:58:01 INFO Vacuumed table table=pending_objects38562026/11/12 16:58:01 INFO Received push request method=POST path=/38572026/11/12 16:58:01 INFO Vacuumed table table=multipart_uploads38582026/11/12 16:58:01 INFO Received complete push request method=POST path=/38592026/11/12 16:58:01 INFO Received uploads request method=POST path=/38602026/11/12 16:58:01 INFO Vacuumed table table=closures38612026/11/12 16:58:01 INFO Completed push id=10 objects=3 duration=479.084µs38622026/11/12 16:58:01 INFO Vacuumed table table=pending_closures38632026/11/12 16:58:01 INFO Received push request method=POST path=/38642026/11/12 16:58:01 INFO Vacuumed table table=commit_receipts38652026/11/12 16:58:01 INFO Received uploads request method=POST path=/38662026/11/12 16:58:01 INFO Received complete upload request method=POST path=/38672026/11/12 16:58:01 INFO Vacuumed table table=objects38682026/11/12 16:58:01 INFO Received push request method=POST path=/38692026/11/12 16:58:01 INFO Completed upload id=25 objects=6 duration=826.25µs38702026/11/12 16:58:01 INFO Received uploads request method=POST path=/38712026/11/12 16:58:01 INFO Vacuumed table table=pending_objects38722026/11/12 16:58:01 INFO Aborted multipart uploads count=0 kept=038732026/11/12 16:58:01 INFO Received push request method=POST path=/38742026/11/12 16:58:01 INFO Received complete upload request method=POST path=/38752026/11/12 16:58:01 INFO Vacuumed table table=multipart_uploads38762026/11/12 16:58:01 WARN Force mode enabled - objects will be deleted immediately without grace period38772026/11/12 16:58:01 INFO Received complete push request method=POST path=/38782026/11/12 16:58:01 INFO Completed upload id=23 objects=7 duration=433.042µs38792026/11/12 16:58:01 INFO Vacuumed table table=closures38802026/11/12 16:58:01 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038812026/11/12 16:58:02 INFO Received push request method=POST path=/38822026/11/12 16:58:02 INFO Received uploads request method=POST path=/38832026/11/12 16:58:02 INFO Vacuumed table table=pending_closures38842026/11/12 16:58:02 INFO Completed push id=19 objects=6 duration=1.721917ms38852026/11/12 16:58:02 INFO Received complete push request method=POST path=/38862026/11/12 16:58:02 INFO Vacuumed table table=objects38872026/11/12 16:58:02 INFO Received uploads request method=POST path=/38882026/11/12 16:58:02 INFO Received complete push request method=POST path=/38892026/11/12 16:58:02 INFO Completed push id=24 objects=5 duration=364.541µs38902026/11/12 16:58:02 INFO Vacuumed table table=commit_receipts38912026/11/12 16:58:02 INFO Received uploads request method=POST path=/38922026/11/12 16:58:02 INFO Completed push id=11 objects=9 duration=852µs38932026/11/12 16:58:02 INFO Aborted multipart uploads count=0 kept=038942026/11/12 16:58:02 INFO Aborted multipart uploads count=0 kept=038952026/11/12 16:58:02 INFO Received complete upload request method=POST path=/38962026/11/12 16:58:02 INFO Vacuumed table table=pending_objects38972026/11/12 16:58:02 INFO Received uploads request method=POST path=/38982026/11/12 16:58:02 INFO Completed upload id=25 objects=2 duration=526.416µs38992026/11/12 16:58:02 INFO Vacuumed table table=multipart_uploads39002026/11/12 16:58:02 WARN Force mode enabled - objects will be deleted immediately without grace period39012026/11/12 16:58:02 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039022026/11/12 16:58:02 INFO Vacuumed table table=closures39032026/11/12 16:58:02 INFO Vacuumed table table=pending_closures39042026/11/12 16:58:02 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."39052026/11/12 16:58:02 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."39062026/11/12 16:58:02 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."39072026/11/12 16:58:02 INFO Received complete upload request method=POST path=/39082026/11/12 16:58:02 INFO Vacuumed table table=objects39092026/11/12 16:58:02 INFO Vacuumed table table=commit_receipts39102026/11/12 16:58:02 INFO Completed upload id=26 objects=20 duration=10.709125ms39112026/11/12 16:58:02 INFO Vacuumed table table=pending_objects39122026/11/12 16:58:02 INFO Aborted multipart uploads count=0 kept=039132026/11/12 16:58:02 INFO Vacuumed table table=multipart_uploads39142026/11/12 16:58:02 WARN Force mode enabled - objects will be deleted immediately without grace period39152026/11/12 16:58:02 INFO Vacuumed table table=closures39162026/11/12 16:58:02 INFO Received uploads request method=POST path=/39172026/11/12 16:58:02 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."39182026/11/12 16:58:02 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."39192026/11/12 16:58:02 INFO Vacuumed table table=objects39202026/11/12 16:58:02 INFO Received complete upload request method=POST path=/39212026/11/12 16:58:02 INFO Received uploads request method=POST path=/39222026/11/12 16:58:02 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."39232026/11/12 16:58:02 INFO Received complete push request method=POST path=/39242026/11/12 16:58:02 INFO Completed upload id=4 objects=7 duration=577.709µs39252026/11/12 16:58:02 INFO Received push request method=POST path=/39262026/11/12 16:58:02 INFO Received push request method=POST path=/39272026/11/12 16:58:02 INFO Completed push id=15 objects=24 duration=482.625µs39282026/11/12 16:58:02 INFO Received push request method=POST path=/39292026/11/12 16:58:02 INFO Received uploads request method=POST path=/39302026/11/12 16:58:02 INFO Received complete push request method=POST path=/39312026/11/12 16:58:02 INFO Received complete upload request method=POST path=/39322026/11/12 16:58:02 INFO Received uploads request method=POST path=/39332026/11/12 16:58:02 INFO Completed upload id=19 objects=2 duration=297.792µs39342026/11/12 16:58:02 INFO Completed push id=8 objects=6 duration=10.7065ms39352026/11/12 16:58:02 INFO Received push request method=POST path=/39362026/11/12 16:58:02 INFO Received complete push request method=POST path=/39372026/11/12 16:58:02 INFO Received complete upload request method=POST path=/39382026/11/12 16:58:02 INFO Received push request method=POST path=/39392026/11/12 16:58:02 INFO Received uploads request method=POST path=/39402026/11/12 16:58:02 INFO Completed push id=28 objects=7 duration=354.958µs39412026/11/12 16:58:02 INFO Completed upload id=24 objects=7 duration=313.625µs39422026/11/12 16:58:02 INFO Received complete push request method=POST path=/39432026/11/12 16:58:02 INFO Completed push id=26 objects=3 duration=314.208µs39442026/11/12 16:58:02 INFO Received push request method=POST path=/39452026/11/12 16:58:02 INFO Received complete upload request method=POST path=/39462026/11/12 16:58:02 INFO Completed upload id=28 objects=8 duration=418.459µs39472026/11/12 16:58:02 INFO Aborted multipart uploads count=0 kept=039482026/11/12 16:58:02 WARN Force mode enabled - objects will be deleted immediately without grace period39492026/11/12 16:58:02 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039502026/11/12 16:58:02 INFO Received complete upload request method=POST path=/39512026/11/12 16:58:02 INFO Received complete upload request method=POST path=/39522026/11/12 16:58:02 INFO Completed upload id=29 objects=10 duration=293.792µs39532026/11/12 16:58:02 INFO Completed upload id=17 objects=6 duration=860.416µs39542026/11/12 16:58:02 INFO Received complete upload request method=POST path=/39552026/11/12 16:58:02 INFO Completed upload id=30 objects=20 duration=317.167µs39562026/11/12 16:58:02 INFO Received push request method=POST path=/39572026/11/12 16:58:02 INFO Received complete push request method=POST path=/39582026/11/12 16:58:02 INFO Completed push id=31 objects=11 duration=286.833µs39592026/11/12 16:58:02 INFO Received complete upload request method=POST path=/39602026/11/12 16:58:02 INFO Completed upload id=32 objects=2 duration=290.25µs39612026/11/12 16:58:02 INFO Vacuumed table table=pending_closures39622026/11/12 16:58:02 INFO Received complete upload request method=POST path=/39632026/11/12 16:58:02 INFO Received push request method=POST path=/39642026/11/12 16:58:02 INFO Received complete push request method=POST path=/39652026/11/12 16:58:02 INFO Completed upload id=30 objects=6 duration=232.291µs39662026/11/12 16:58:02 INFO Received uploads request method=POST path=/39672026/11/12 16:58:02 INFO Completed push id=33 objects=3 duration=305.875µs39682026/11/12 16:58:02 INFO Received complete push request method=POST path=/39692026/11/12 16:58:02 INFO Received complete upload request method=POST path=/39702026/11/12 16:58:02 INFO Received push request method=POST path=/39712026/11/12 16:58:02 INFO Received complete upload request method=POST path=/39722026/11/12 16:58:02 INFO Received complete upload request method=POST path=/39732026/11/12 16:58:02 INFO Vacuumed table table=commit_receipts39742026/11/12 16:58:02 INFO Completed push id=31 objects=5 duration=349.083µs39752026/11/12 16:58:02 INFO Received complete push request method=POST path=/39762026/11/12 16:58:02 INFO Completed upload id=10 objects=6 duration=907.125µs39772026/11/12 16:58:02 INFO Received uploads request method=POST path=/39782026/11/12 16:58:02 INFO Completed upload id=7 objects=10 duration=404.291µs39792026/11/12 16:58:02 INFO Completed upload id=34 objects=3 duration=271µs39802026/11/12 16:58:02 INFO Completed push id=5 objects=28 duration=501.375µs39812026/11/12 16:58:02 INFO Vacuumed table table=pending_objects39822026/11/12 16:58:02 INFO Received complete upload request method=POST path=/39832026/11/12 16:58:02 INFO Received push request method=POST path=/39842026/11/12 16:58:02 INFO Vacuumed table table=multipart_uploads39852026/11/12 16:58:02 INFO Completed upload id=15 objects=9 duration=539.042µs39862026/11/12 16:58:02 INFO Received complete upload request method=POST path=/39872026/11/12 16:58:02 INFO Aborted multipart uploads count=0 kept=039882026/11/12 16:58:02 INFO Received complete push request method=POST path=/39892026/11/12 16:58:02 INFO Received complete push request method=POST path=/39902026/11/12 16:58:02 INFO Completed upload id=34 objects=24 duration=371.083µs39912026/11/12 16:58:02 WARN Force mode enabled - objects will be deleted immediately without grace period39922026/11/12 16:58:02 INFO Completed push id=8 objects=6 duration=646.834µs39932026/11/12 16:58:02 INFO Completed push id=10 objects=2 duration=370.875µs39942026/11/12 16:58:02 INFO Vacuumed table table=closures39952026/11/12 16:58:02 INFO Aborted multipart uploads count=0 kept=039962026/11/12 16:58:02 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039972026/11/12 16:58:02 WARN Force mode enabled - objects will be deleted immediately without grace period39982026/11/12 16:58:02 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039992026/11/12 16:58:02 INFO Vacuumed table table=pending_closures40002026/11/12 16:58:02 INFO Vacuumed table table=commit_receipts40012026/11/12 16:58:02 INFO Vacuumed table table=objects40022026/11/12 16:58:02 INFO Vacuumed table table=pending_closures40032026/11/12 16:58:02 INFO Vacuumed table table=commit_receipts40042026/11/12 16:58:02 INFO Vacuumed table table=pending_objects40052026/11/12 16:58:02 INFO Vacuumed table table=multipart_uploads40062026/11/12 16:58:02 INFO Vacuumed table table=closures40072026/11/12 16:58:02 INFO Vacuumed table table=objects40082026/11/12 16:58:02 INFO Vacuumed table table=pending_objects40092026/11/12 16:58:02 INFO Received uploads request method=POST path=/40102026/11/12 16:58:02 INFO Vacuumed table table=multipart_uploads40112026/11/12 16:58:02 INFO Received uploads request method=POST path=/40122026/11/12 16:58:02 INFO Vacuumed table table=closures40132026/11/12 16:58:02 INFO Received uploads request method=POST path=/40142026/11/12 16:58:02 INFO Vacuumed table table=objects40152026/11/12 16:58:02 INFO Received complete push request method=POST path=/40162026/11/12 16:58:02 INFO Received complete push request method=POST path=/40172026/11/12 16:58:02 INFO Completed push id=12 objects=7 duration=657.166µs40182026/11/12 16:58:02 INFO Received uploads request method=POST path=/40192026/11/12 16:58:02 INFO Completed push id=21 objects=17 duration=597.209µs40202026/11/12 16:58:02 INFO Aborted multipart uploads count=0 kept=040212026/11/12 16:58:02 INFO Aborted multipart uploads count=0 kept=040222026/11/12 16:58:02 INFO Received complete push request method=POST path=/40232026/11/12 16:58:02 INFO Received complete upload request method=POST path=/40242026/11/12 16:58:02 INFO Received complete push request method=POST path=/40252026/11/12 16:58:02 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=040262026/11/12 16:58:02 INFO Completed upload id=17 objects=5 duration=464.084µs40272026/11/12 16:58:02 INFO Completed push id=32 objects=26 duration=573.875µs40282026/11/12 16:58:02 INFO Completed push id=19 objects=7 duration=804.875µs40292026/11/12 16:58:02 WARN Force mode enabled - objects will be deleted immediately without grace period40302026/11/12 16:58:02 INFO Received complete upload request method=POST path=/40312026/11/12 16:58:02 INFO Received push request method=POST path=/40322026/11/12 16:58:02 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=4 objects-failed-to-delete=040332026/11/12 16:58:02 INFO Completed upload id=33 objects=2 duration=266.833µs40342026/11/12 16:58:02 INFO Vacuumed table table=pending_closures40352026/11/12 16:58:02 INFO Vacuumed table table=commit_receipts40362026/11/12 16:58:02 INFO Vacuumed table table=pending_closures40372026/11/12 16:58:02 INFO Received complete upload request method=POST path=/40382026/11/12 16:58:02 INFO Aborted multipart uploads count=0 kept=040392026/11/12 16:58:02 INFO Completed upload id=34 objects=5 duration=444.5µs40402026/11/12 16:58:02 INFO Vacuumed table table=commit_receipts40412026/11/12 16:58:02 WARN Force mode enabled - objects will be deleted immediately without grace period40422026/11/12 16:58:02 INFO Vacuumed table table=pending_objects40432026/11/12 16:58:02 INFO Received complete upload request method=POST path=/40442026/11/12 16:58:02 INFO Completed upload id=35 objects=11 duration=349.875µs40452026/11/12 16:58:02 INFO Vacuumed table table=multipart_uploads40462026/11/12 16:58:02 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040472026/11/12 16:58:02 INFO Received complete upload request method=POST path=/40482026/11/12 16:58:02 INFO Vacuumed table table=pending_closures40492026/11/12 16:58:02 INFO Vacuumed table table=closures40502026/11/12 16:58:02 INFO Completed upload id=36 objects=3 duration=243.375µs40512026/11/12 16:58:02 INFO Vacuumed table table=pending_objects40522026/11/12 16:58:02 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=8 objects-deleted-after-grace-period=8 objects-failed-to-delete=040532026/11/12 16:58:02 INFO Vacuumed table table=commit_receipts40542026/11/12 16:58:02 INFO Vacuumed table table=objects40552026/11/12 16:58:02 INFO Aborted multipart uploads count=0 kept=040562026/11/12 16:58:02 INFO Vacuumed table table=multipart_uploads40572026/11/12 16:58:02 INFO Vacuumed table table=pending_closures40582026/11/12 16:58:02 WARN Force mode enabled - objects will be deleted immediately without grace period40592026/11/12 16:58:02 INFO Received complete upload request method=POST path=/40602026/11/12 16:58:02 INFO Vacuumed table table=closures40612026/11/12 16:58:02 INFO Vacuumed table table=pending_objects40622026/11/12 16:58:02 INFO Vacuumed table table=commit_receipts40632026/11/12 16:58:02 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040642026/11/12 16:58:02 INFO Completed upload id=10 objects=6 duration=1.1535ms40652026/11/12 16:58:02 INFO Vacuumed table table=objects40662026/11/12 16:58:02 INFO Vacuumed table table=multipart_uploads40672026/11/12 16:58:02 INFO Vacuumed table table=pending_objects40682026/11/12 16:58:02 INFO Received complete push request method=POST path=/40692026/11/12 16:58:02 INFO Vacuumed table table=closures40702026/11/12 16:58:02 INFO Aborted multipart uploads count=0 kept=040712026/11/12 16:58:02 INFO Vacuumed table table=multipart_uploads40722026/11/12 16:58:02 INFO Completed push id=19 objects=3 duration=548.667µs40732026/11/12 16:58:02 INFO Vacuumed table table=objects40742026/11/12 16:58:02 INFO Received uploads request method=POST path=/40752026/11/12 16:58:03 INFO Vacuumed table table=closures40762026/11/12 16:58:03 INFO Vacuumed table table=pending_closures40772026/11/12 16:58:03 WARN Force mode enabled - objects will be deleted immediately without grace period40782026/11/12 16:58:03 INFO Vacuumed table table=commit_receipts40792026/11/12 16:58:03 INFO Vacuumed table table=objects40802026/11/12 16:58:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040812026/11/12 16:58:03 INFO Received uploads request method=POST path=/40822026/11/12 16:58:03 INFO Vacuumed table table=pending_closures40832026/11/12 16:58:03 INFO Received uploads request method=POST path=/40842026/11/12 16:58:03 INFO Aborted multipart uploads count=0 kept=040852026/11/12 16:58:03 INFO Vacuumed table table=commit_receipts40862026/11/12 16:58:03 INFO Vacuumed table table=pending_objects40872026/11/12 16:58:03 INFO Received uploads request method=POST path=/40882026/11/12 16:58:03 WARN Force mode enabled - objects will be deleted immediately without grace period40892026/11/12 16:58:03 INFO Vacuumed table table=multipart_uploads40902026/11/12 16:58:03 INFO Vacuumed table table=pending_objects40912026/11/12 16:58:03 INFO Received uploads request method=POST path=/40922026/11/12 16:58:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040932026/11/12 16:58:03 INFO Vacuumed table table=multipart_uploads40942026/11/12 16:58:03 INFO Vacuumed table table=closures40952026/11/12 16:58:03 INFO Vacuumed table table=pending_closures40962026/11/12 16:58:03 INFO Received push request method=POST path=/40972026/11/12 16:58:03 INFO Vacuumed table table=closures40982026/11/12 16:58:03 INFO Vacuumed table table=objects40992026/11/12 16:58:03 INFO Vacuumed table table=commit_receipts41002026/11/12 16:58:03 INFO Vacuumed table table=objects41012026/11/12 16:58:03 INFO Vacuumed table table=pending_objects41022026/11/12 16:58:03 INFO Aborted multipart uploads count=0 kept=041032026/11/12 16:58:03 INFO Received uploads request method=POST path=/41042026/11/12 16:58:03 INFO Vacuumed table table=multipart_uploads41052026/11/12 16:58:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041062026/11/12 16:58:03 INFO Received push request method=POST path=/41072026/11/12 16:58:03 INFO Vacuumed table table=closures41082026/11/12 16:58:03 INFO Vacuumed table table=pending_closures41092026/11/12 16:58:03 INFO Vacuumed table table=objects41102026/11/12 16:58:03 INFO Received complete upload request method=POST path=/41112026/11/12 16:58:03 INFO Aborted multipart uploads count=0 kept=041122026/11/12 16:58:03 INFO Vacuumed table table=commit_receipts41132026/11/12 16:58:03 WARN Force mode enabled - objects will be deleted immediately without grace period41142026/11/12 16:58:03 INFO Received complete upload request method=POST path=/41152026/11/12 16:58:03 INFO Vacuumed table table=pending_objects41162026/11/12 16:58:03 INFO Vacuumed table table=multipart_uploads41172026/11/12 16:58:03 INFO Completed upload id=15 objects=3 duration=8.185583ms41182026/11/12 16:58:03 INFO Received complete push request method=POST path=/41192026/11/12 16:58:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041202026/11/12 16:58:03 INFO Vacuumed table table=closures41212026/11/12 16:58:03 INFO Completed push id=9 objects=24 duration=486.542µs41222026/11/12 16:58:03 INFO Vacuumed table table=objects41232026/11/12 16:58:03 INFO Received push request method=POST path=/41242026/11/12 16:58:03 INFO Received uploads request method=POST path=/41252026/11/12 16:58:03 INFO Vacuumed table table=pending_closures41262026/11/12 16:58:03 INFO Received uploads request method=POST path=/41272026/11/12 16:58:03 INFO Received uploads request method=POST path=/41282026/11/12 16:58:03 INFO Vacuumed table table=commit_receipts41292026/11/12 16:58:03 INFO Received push request method=POST path=/41302026/11/12 16:58:03 INFO Completed upload id=1 objects=15 duration=57.15825ms41312026/11/12 16:58:03 INFO Received complete push request method=POST path=/41322026/11/12 16:58:03 INFO Received complete upload request method=POST path=/41332026/11/12 16:58:03 INFO Received complete upload request method=POST path=/41342026/11/12 16:58:03 INFO Received uploads request method=POST path=/41352026/11/12 16:58:03 INFO Completed upload id=10 objects=10 duration=279.959µs41362026/11/12 16:58:03 INFO Completed push id=15 objects=7 duration=607.125µs41372026/11/12 16:58:03 INFO Completed upload id=12 objects=26 duration=1.015791ms41382026/11/12 16:58:03 INFO Received uploads request method=POST path=/41392026/11/12 16:58:03 INFO Vacuumed table table=pending_objects41402026/11/12 16:58:03 INFO Received uploads request method=POST path=/41412026/11/12 16:58:03 INFO Aborted multipart uploads count=0 kept=041422026/11/12 16:58:03 INFO Vacuumed table table=multipart_uploads41432026/11/12 16:58:03 INFO Received uploads request method=POST path=/41442026/11/12 16:58:03 INFO Received uploads request method=POST path=/41452026/11/12 16:58:03 INFO Received uploads request method=POST path=/41462026/11/12 16:58:03 INFO Received complete push request method=POST path=/41472026/11/12 16:58:03 INFO Vacuumed table table=closures41482026/11/12 16:58:03 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."41492026/11/12 16:58:03 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."41502026/11/12 16:58:03 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."41512026/11/12 16:58:03 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."41522026/11/12 16:58:03 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."41532026/11/12 16:58:03 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."41542026/11/12 16:58:03 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."41552026/11/12 16:58:03 INFO Received push request method=POST path=/41562026/11/12 16:58:03 INFO Received complete push request method=POST path=/41572026/11/12 16:58:03 INFO Vacuumed table table=objects41582026/11/12 16:58:03 INFO Completed push id=20 objects=4 duration=535.042µs41592026/11/12 16:58:03 INFO Completed push id=16 objects=8 duration=565.458µs41602026/11/12 16:58:03 INFO Received push request method=POST path=/41612026/11/12 16:58:03 INFO Aborted multipart uploads count=0 kept=041622026/11/12 16:58:03 INFO Received push request method=POST path=/41632026/11/12 16:58:03 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."41642026/11/12 16:58:03 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."41652026/11/12 16:58:03 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."41662026/11/12 16:58:03 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."41672026/11/12 16:58:03 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."41682026/11/12 16:58:03 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."41692026/11/12 16:58:03 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."41702026/11/12 16:58:03 INFO Aborted multipart uploads count=0 kept=041712026/11/12 16:58:03 INFO Received complete upload request method=POST path=/41722026/11/12 16:58:03 WARN Force mode enabled - objects will be deleted immediately without grace period41732026/11/12 16:58:03 INFO Received push request method=POST path=/41742026/11/12 16:58:03 INFO Completed upload id=22 objects=2 duration=437.667µs41752026/11/12 16:58:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041762026/11/12 16:58:03 INFO Received uploads request method=POST path=/41772026/11/12 16:58:03 INFO Vacuumed table table=pending_closures41782026/11/12 16:58:03 INFO Vacuumed table table=commit_receipts41792026/11/12 16:58:03 INFO Received push request method=POST path=/41802026/11/12 16:58:03 INFO Aborted multipart uploads count=0 kept=041812026/11/12 16:58:03 INFO Received complete push request method=POST path=/41822026/11/12 16:58:03 INFO Completed push id=14 objects=3 duration=302.25µs41832026/11/12 16:58:03 INFO Vacuumed table table=pending_objects41842026/11/12 16:58:03 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=041852026/11/12 16:58:03 INFO Aborted multipart uploads count=0 kept=041862026/11/12 16:58:03 INFO Vacuumed table table=multipart_uploads41872026/11/12 16:58:03 INFO Vacuumed table table=pending_closures41882026/11/12 16:58:03 INFO Vacuumed table table=closures41892026/11/12 16:58:03 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."41902026/11/12 16:58:03 INFO Received push request method=POST path=/41912026/11/12 16:58:03 INFO Vacuumed table table=commit_receipts41922026/11/12 16:58:03 INFO Vacuumed table table=objects41932026/11/12 16:58:03 INFO Vacuumed table table=pending_objects41942026/11/12 16:58:03 INFO Received uploads request method=POST path=/41952026/11/12 16:58:03 INFO Vacuumed table table=multipart_uploads41962026/11/12 16:58:03 INFO Received push request method=POST path=/41972026/11/12 16:58:03 INFO Vacuumed table table=closures41982026/11/12 16:58:03 INFO Aborted multipart uploads count=0 kept=041992026/11/12 16:58:03 INFO Aborted multipart uploads count=0 kept=042002026/11/12 16:58:03 INFO Vacuumed table table=objects42012026/11/12 16:58:03 WARN Force mode enabled - objects will be deleted immediately without grace period42022026/11/12 16:58:03 INFO Received uploads request method=POST path=/42032026/11/12 16:58:03 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042042026/11/12 16:58:03 INFO Received push request method=POST path=/42052026/11/12 16:58:03 INFO Received push request method=POST path=/42062026/11/12 16:58:03 INFO Received push request method=POST path=/42072026/11/12 16:58:03 INFO Aborted multipart uploads count=0 kept=042082026/11/12 16:58:03 INFO Vacuumed table table=pending_closures42092026/11/12 16:58:03 INFO Vacuumed table table=commit_receipts42102026/11/12 16:58:03 INFO Received complete upload request method=POST path=/42112026/11/12 16:58:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042122026/11/12 16:58:03 INFO Completed upload id=11 objects=7 duration=496.208µs42132026/11/12 16:58:03 INFO Vacuumed table table=pending_closures42142026/11/12 16:58:03 INFO Received uploads request method=POST path=/42152026/11/12 16:58:03 INFO Vacuumed table table=pending_objects42162026/11/12 16:58:03 INFO Vacuumed table table=commit_receipts42172026/11/12 16:58:03 INFO Received uploads request method=POST path=/42182026/11/12 16:58:03 INFO Vacuumed table table=multipart_uploads42192026/11/12 16:58:03 INFO Received complete push request method=POST path=/42202026/11/12 16:58:03 INFO Vacuumed table table=closures42212026/11/12 16:58:03 INFO Completed push id=11 objects=6 duration=1.140541ms42222026/11/12 16:58:03 INFO Vacuumed table table=objects42232026/11/12 16:58:03 INFO Vacuumed table table=pending_objects42242026/11/12 16:58:03 INFO Received complete push request method=POST path=/42252026/11/12 16:58:03 INFO Vacuumed table table=multipart_uploads42262026/11/12 16:58:03 INFO Completed push id=19 objects=8 duration=6.910208ms42272026/11/12 16:58:03 INFO Vacuumed table table=closures42282026/11/12 16:58:03 INFO Received complete upload request method=POST path=/42292026/11/12 16:58:03 INFO Received complete upload request method=POST path=/42302026/11/12 16:58:03 INFO Received uploads request method=POST path=/42312026/11/12 16:58:03 INFO Received push request method=POST path=/42322026/11/12 16:58:03 INFO Vacuumed table table=objects42332026/11/12 16:58:03 INFO Completed upload id=22 objects=8 duration=372.208µs42342026/11/12 16:58:03 INFO Completed upload id=6 objects=24 duration=1.322208ms42352026/11/12 16:58:03 INFO Aborted multipart uploads count=0 kept=042362026/11/12 16:58:03 INFO Received uploads request method=POST path=/42372026/11/12 16:58:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042382026/11/12 16:58:03 INFO Vacuumed table table=pending_closures42392026/11/12 16:58:03 INFO Received push request method=POST path=/42402026/11/12 16:58:03 INFO Received complete push request method=POST path=/42412026/11/12 16:58:03 INFO Received uploads request method=POST path=/42422026/11/12 16:58:03 INFO Received push request method=POST path=/42432026/11/12 16:58:03 INFO Vacuumed table table=commit_receipts42442026/11/12 16:58:03 INFO Completed push id=23 objects=5 duration=368.625µs42452026/11/12 16:58:03 INFO Aborted multipart uploads count=0 kept=042462026/11/12 16:58:03 INFO Vacuumed table table=pending_objects42472026/11/12 16:58:03 INFO Received push request method=POST path=/42482026/11/12 16:58:03 INFO Received complete upload request method=POST path=/42492026/11/12 16:58:03 INFO Vacuumed table table=multipart_uploads42502026/11/12 16:58:03 INFO Received complete upload request method=POST path=/42512026/11/12 16:58:03 WARN Force mode enabled - objects will be deleted immediately without grace period42522026/11/12 16:58:03 INFO Completed upload id=24 objects=8 duration=278.958µs42532026/11/12 16:58:03 INFO Completed upload id=11 objects=6 duration=787.542µs42542026/11/12 16:58:03 INFO Vacuumed table table=closures42552026/11/12 16:58:03 INFO Received uploads request method=POST path=/42562026/11/12 16:58:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042572026/11/12 16:58:03 INFO Received complete upload request method=POST path=/42582026/11/12 16:58:03 INFO Received uploads request method=POST path=/42592026/11/12 16:58:03 INFO Received complete push request method=POST path=/42602026/11/12 16:58:03 INFO Vacuumed table table=objects42612026/11/12 16:58:03 INFO Completed push id=18 objects=6 duration=396.333µs42622026/11/12 16:58:03 INFO Completed upload id=28 objects=5 duration=492.833µs42632026/11/12 16:58:03 INFO Vacuumed table table=pending_closures42642026/11/12 16:58:03 INFO Received uploads request method=POST path=/42652026/11/12 16:58:03 INFO Aborted multipart uploads count=0 kept=042662026/11/12 16:58:03 INFO Vacuumed table table=commit_receipts42672026/11/12 16:58:03 WARN Force mode enabled - objects will be deleted immediately without grace period42682026/11/12 16:58:03 INFO Received uploads request method=POST path=/42692026/11/12 16:58:03 INFO Aborted multipart uploads count=0 kept=042702026/11/12 16:58:03 INFO Vacuumed table table=pending_objects42712026/11/12 16:58:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042722026/11/12 16:58:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042732026/11/12 16:58:03 INFO Received push request method=POST path=/42742026/11/12 16:58:03 INFO Received push request method=POST path=/42752026/11/12 16:58:03 INFO Vacuumed table table=multipart_uploads42762026/11/12 16:58:03 INFO Vacuumed table table=pending_closures42772026/11/12 16:58:03 INFO Vacuumed table table=pending_closures42782026/11/12 16:58:03 INFO Aborted multipart uploads count=0 kept=042792026/11/12 16:58:03 INFO Vacuumed table table=commit_receipts42802026/11/12 16:58:03 INFO Vacuumed table table=closures42812026/11/12 16:58:03 INFO Vacuumed table table=commit_receipts42822026/11/12 16:58:03 INFO Received complete push request method=POST path=/42832026/11/12 16:58:03 INFO Vacuumed table table=pending_objects42842026/11/12 16:58:03 WARN Force mode enabled - objects will be deleted immediately without grace period42852026/11/12 16:58:03 INFO Received complete push request method=POST path=/42862026/11/12 16:58:03 INFO Vacuumed table table=objects42872026/11/12 16:58:03 INFO Vacuumed table table=pending_objects42882026/11/12 16:58:03 INFO Vacuumed table table=multipart_uploads42892026/11/12 16:58:03 INFO Completed push id=8 objects=6 duration=8.023167ms42902026/11/12 16:58:03 INFO Completed push id=9 objects=5 duration=7.987834ms42912026/11/12 16:58:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042922026/11/12 16:58:03 INFO Vacuumed table table=multipart_uploads42932026/11/12 16:58:03 INFO Received uploads request method=POST path=/42942026/11/12 16:58:03 INFO Vacuumed table table=closures42952026/11/12 16:58:03 INFO Vacuumed table table=closures42962026/11/12 16:58:03 INFO Vacuumed table table=pending_closures42972026/11/12 16:58:03 INFO Aborted multipart uploads count=0 kept=042982026/11/12 16:58:03 INFO Vacuumed table table=objects42992026/11/12 16:58:03 INFO Received uploads request method=POST path=/43002026/11/12 16:58:03 INFO Vacuumed table table=commit_receipts43012026/11/12 16:58:03 INFO Vacuumed table table=objects43022026/11/12 16:58:03 WARN Force mode enabled - objects will be deleted immediately without grace period43032026/11/12 16:58:03 INFO Received push request method=POST path=/43042026/11/12 16:58:03 INFO Vacuumed table table=pending_objects43052026/11/12 16:58:03 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043062026/11/12 16:58:03 INFO Received uploads request method=POST path=/43072026/11/12 16:58:03 INFO Received complete upload request method=POST path=/43082026/11/12 16:58:03 INFO Received uploads request method=POST path=/43092026/11/12 16:58:03 INFO Vacuumed table table=multipart_uploads43102026/11/12 16:58:03 INFO Vacuumed table table=pending_closures43112026/11/12 16:58:03 INFO Completed upload id=20 objects=3 duration=825.25µs43122026/11/12 16:58:03 INFO Received uploads request method=POST path=/43132026/11/12 16:58:03 INFO Received complete upload request method=POST path=/43142026/11/12 16:58:03 INFO Vacuumed table table=commit_receipts43152026/11/12 16:58:03 INFO Vacuumed table table=closures43162026/11/12 16:58:03 INFO Received uploads request method=POST path=/43172026/11/12 16:58:03 INFO Received complete push request method=POST path=/43182026/11/12 16:58:03 INFO Received push request method=POST path=/43192026/11/12 16:58:03 INFO Completed upload id=30 objects=5 duration=927.5µs43202026/11/12 16:58:03 INFO Vacuumed table table=objects43212026/11/12 16:58:03 INFO Vacuumed table table=pending_objects43222026/11/12 16:58:03 INFO Received uploads request method=POST path=/43232026/11/12 16:58:03 INFO Received push request method=POST path=/43242026/11/12 16:58:03 INFO Vacuumed table table=multipart_uploads43252026/11/12 16:58:03 INFO Completed push id=17 objects=22 duration=851.583µs43262026/11/12 16:58:03 INFO Vacuumed table table=closures43272026/11/12 16:58:03 INFO Received uploads request method=POST path=/43282026/11/12 16:58:03 INFO Received complete push request method=POST path=/43292026/11/12 16:58:03 INFO Received complete upload request method=POST path=/43302026/11/12 16:58:03 INFO Received uploads request method=POST path=/43312026/11/12 16:58:03 INFO Received uploads request method=POST path=/43322026/11/12 16:58:03 INFO Completed upload id=24 objects=7 duration=334.208µs43332026/11/12 16:58:03 INFO Vacuumed table table=objects43342026/11/12 16:58:03 INFO Completed push id=18 objects=6 duration=926.917µs43352026/11/12 16:58:03 INFO Aborted multipart uploads count=0 kept=043362026/11/12 16:58:03 INFO Aborted multipart uploads count=0 kept=043372026/11/12 16:58:03 INFO Received complete upload request method=POST path=/43382026/11/12 16:58:03 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043392026/11/12 16:58:03 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043402026/11/12 16:58:03 INFO Received push request method=POST path=/43412026/11/12 16:58:03 INFO Received complete upload request method=POST path=/43422026/11/12 16:58:03 INFO Received push request method=POST path=/43432026/11/12 16:58:03 INFO Completed upload id=9 objects=3 duration=455.459µs43442026/11/12 16:58:03 INFO Vacuumed table table=pending_closures43452026/11/12 16:58:03 INFO Vacuumed table table=pending_closures43462026/11/12 16:58:03 INFO Received complete push request method=POST path=/43472026/11/12 16:58:03 INFO Completed upload id=23 objects=9 duration=302.125µs43482026/11/12 16:58:03 INFO Vacuumed table table=commit_receipts43492026/11/12 16:58:03 INFO Vacuumed table table=commit_receipts43502026/11/12 16:58:03 INFO Completed push id=19 objects=14 duration=667.583µs43512026/11/12 16:58:03 INFO Received push request method=POST path=/43522026/11/12 16:58:03 INFO Vacuumed table table=pending_objects43532026/11/12 16:58:03 INFO Received push request method=POST path=/43542026/11/12 16:58:03 INFO Vacuumed table table=pending_objects43552026/11/12 16:58:03 INFO Received uploads request method=POST path=/43562026/11/12 16:58:03 INFO Aborted multipart uploads count=0 kept=043572026/11/12 16:58:03 INFO Vacuumed table table=multipart_uploads43582026/11/12 16:58:03 INFO Vacuumed table table=multipart_uploads43592026/11/12 16:58:03 INFO Received uploads request method=POST path=/43602026/11/12 16:58:03 INFO Vacuumed table table=closures43612026/11/12 16:58:03 INFO Vacuumed table table=closures43622026/11/12 16:58:03 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."43632026/11/12 16:58:03 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."43642026/11/12 16:58:03 INFO Vacuumed table table=objects43652026/11/12 16:58:03 INFO Vacuumed table table=objects43662026/11/12 16:58:03 INFO Received complete push request method=POST path=/43672026/11/12 16:58:03 INFO Received uploads request method=POST path=/43682026/11/12 16:58:03 INFO Received uploads request method=POST path=/43692026/11/12 16:58:03 INFO Received push request method=POST path=/43702026/11/12 16:58:03 INFO Completed push id=31 objects=3 duration=276.458µs43712026/11/12 16:58:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=043722026/11/12 16:58:03 INFO Received uploads request method=POST path=/43732026/11/12 16:58:03 INFO Vacuumed table table=pending_closures43742026/11/12 16:58:03 INFO Received complete push request method=POST path=/43752026/11/12 16:58:03 INFO Aborted multipart uploads count=0 kept=043762026/11/12 16:58:03 INFO Received complete push request method=POST path=/43772026/11/12 16:58:03 INFO Completed push id=16 objects=26 duration=381.834µs43782026/11/12 16:58:03 INFO Completed push id=12 objects=3 duration=9.149208ms43792026/11/12 16:58:03 WARN Force mode enabled - objects will be deleted immediately without grace period43802026/11/12 16:58:03 INFO Aborted multipart uploads count=0 kept=043812026/11/12 16:58:03 INFO Received uploads request method=POST path=/43822026/11/12 16:58:03 WARN Force mode enabled - objects will be deleted immediately without grace period43832026/11/12 16:58:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043842026/11/12 16:58:03 INFO Received uploads request method=POST path=/43852026/11/12 16:58:03 INFO Vacuumed table table=commit_receipts43862026/11/12 16:58:03 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043872026/11/12 16:58:03 INFO Vacuumed table table=pending_closures43882026/11/12 16:58:03 INFO Vacuumed table table=pending_closures43892026/11/12 16:58:03 INFO Vacuumed table table=pending_objects43902026/11/12 16:58:03 INFO Vacuumed table table=commit_receipts43912026/11/12 16:58:03 INFO Received complete upload request method=POST path=/43922026/11/12 16:58:03 INFO Vacuumed table table=multipart_uploads43932026/11/12 16:58:03 INFO Completed upload id=33 objects=5 duration=6.332959ms43942026/11/12 16:58:03 INFO Vacuumed table table=commit_receipts43952026/11/12 16:58:03 INFO Vacuumed table table=pending_objects43962026/11/12 16:58:03 INFO Vacuumed table table=pending_objects43972026/11/12 16:58:03 INFO Vacuumed table table=multipart_uploads43982026/11/12 16:58:03 INFO Vacuumed table table=multipart_uploads43992026/11/12 16:58:03 INFO Vacuumed table table=closures44002026/11/12 16:58:03 INFO Vacuumed table table=closures44012026/11/12 16:58:04 INFO Vacuumed table table=objects44022026/11/12 16:58:04 INFO Received complete upload request method=POST path=/44032026/11/12 16:58:04 INFO Vacuumed table table=objects44042026/11/12 16:58:04 INFO Vacuumed table table=closures44052026/11/12 16:58:04 INFO Completed upload id=20 objects=6 duration=5.696209ms44062026/11/12 16:58:04 INFO Received complete push request method=POST path=/44072026/11/12 16:58:04 INFO Completed push id=27 objects=6 duration=1.045959ms44082026/11/12 16:58:04 INFO Received uploads request method=POST path=/44092026/11/12 16:58:04 INFO Vacuumed table table=objects44102026/11/12 16:58:04 INFO Received complete upload request method=POST path=/44112026/11/12 16:58:04 INFO Received complete upload request method=POST path=/44122026/11/12 16:58:04 INFO Completed upload id=15 objects=2 duration=280.375µs44132026/11/12 16:58:04 INFO Completed upload id=21 objects=2 duration=276.708µs44142026/11/12 16:58:04 INFO Received complete upload request method=POST path=/44152026/11/12 16:58:04 INFO Received uploads request method=POST path=/44162026/11/12 16:58:04 INFO Received push request method=POST path=/44172026/11/12 16:58:04 INFO Completed upload id=16 objects=23 duration=279.5µs44182026/11/12 16:58:04 INFO Received complete upload request method=POST path=/44192026/11/12 16:58:04 INFO Received uploads request method=POST path=/44202026/11/12 16:58:04 INFO Aborted multipart uploads count=0 kept=044212026/11/12 16:58:04 INFO Completed upload id=15 objects=6 duration=223.167µs44222026/11/12 16:58:04 WARN Force mode enabled - objects will be deleted immediately without grace period44232026/11/12 16:58:04 INFO Received uploads request method=POST path=/44242026/11/12 16:58:04 INFO Received uploads request method=POST path=/44252026/11/12 16:58:04 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044262026/11/12 16:58:04 INFO Vacuumed table table=pending_closures44272026/11/12 16:58:04 INFO Received complete upload request method=POST path=/44282026/11/12 16:58:04 INFO Vacuumed table table=commit_receipts44292026/11/12 16:58:04 INFO Completed upload id=26 objects=8 duration=355.708µs44302026/11/12 16:58:04 INFO Received complete push request method=POST path=/44312026/11/12 16:58:04 INFO Received complete push request method=POST path=/44322026/11/12 16:58:04 INFO Vacuumed table table=pending_objects44332026/11/12 16:58:04 INFO Completed push id=23 objects=28 duration=1.466167ms44342026/11/12 16:58:04 INFO Received complete push request method=POST path=/44352026/11/12 16:58:04 INFO Vacuumed table table=multipart_uploads44362026/11/12 16:58:04 INFO Completed push id=32 objects=6 duration=361.584µs44372026/11/12 16:58:04 INFO Completed push id=28 objects=14 duration=275.584µs44382026/11/12 16:58:04 INFO Vacuumed table table=closures44392026/11/12 16:58:04 INFO Received complete upload request method=POST path=/44402026/11/12 16:58:04 INFO Received complete upload request method=POST path=/44412026/11/12 16:58:04 INFO Vacuumed table table=objects44422026/11/12 16:58:04 INFO Received push request method=POST path=/44432026/11/12 16:58:04 INFO Completed upload id=33 objects=7 duration=291µs44442026/11/12 16:58:04 INFO Completed upload id=27 objects=8 duration=482.791µs44452026/11/12 16:58:04 INFO Received uploads request method=POST path=/44462026/11/12 16:58:04 INFO Received uploads request method=POST path=/44472026/11/12 16:58:04 INFO Received complete upload request method=POST path=/44482026/11/12 16:58:04 INFO Received complete upload request method=POST path=/44492026/11/12 16:58:04 INFO Received push request method=POST path=/44502026/11/12 16:58:04 INFO Aborted multipart uploads count=0 kept=044512026/11/12 16:58:04 INFO Completed upload id=34 objects=3 duration=201.25µs44522026/11/12 16:58:04 INFO Completed upload id=17 objects=7 duration=6.588709ms44532026/11/12 16:58:04 WARN Force mode enabled - objects will be deleted immediately without grace period44542026/11/12 16:58:04 INFO Received complete push request method=POST path=/44552026/11/12 16:58:04 INFO Received push request method=POST path=/44562026/11/12 16:58:04 INFO Aborted multipart uploads count=0 kept=044572026/11/12 16:58:04 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044582026/11/12 16:58:04 INFO Completed push id=35 objects=11 duration=352.417µs44592026/11/12 16:58:04 INFO Aborted multipart uploads count=0 kept=044602026/11/12 16:58:04 INFO Received complete push request method=POST path=/44612026/11/12 16:58:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044622026/11/12 16:58:04 INFO Received complete push request method=POST path=/44632026/11/12 16:58:04 INFO Vacuumed table table=pending_closures44642026/11/12 16:58:04 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."44652026/11/12 16:58:04 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."44662026/11/12 16:58:04 INFO Aborted multipart uploads count=0 kept=044672026/11/12 16:58:04 INFO Completed push id=6 objects=26 duration=1.126792ms44682026/11/12 16:58:04 INFO Vacuumed table table=pending_closures44692026/11/12 16:58:04 INFO Vacuumed table table=commit_receipts44702026/11/12 16:58:04 WARN Force mode enabled - objects will be deleted immediately without grace period44712026/11/12 16:58:04 INFO Completed push id=24 objects=24 duration=1.150083ms44722026/11/12 16:58:04 INFO Vacuumed table table=commit_receipts44732026/11/12 16:58:04 INFO Aborted multipart uploads count=0 kept=044742026/11/12 16:58:04 INFO Vacuumed table table=pending_objects44752026/11/12 16:58:04 INFO Garbage collection completed failed-uploads-deleted=6 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044762026/11/12 16:58:04 INFO Vacuumed table table=multipart_uploads44772026/11/12 16:58:04 WARN Force mode enabled - objects will be deleted immediately without grace period44782026/11/12 16:58:04 INFO Vacuumed table table=closures44792026/11/12 16:58:04 INFO Aborted multipart uploads count=0 kept=044802026/11/12 16:58:04 INFO Vacuumed table table=pending_objects44812026/11/12 16:58:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044822026/11/12 16:58:04 INFO Vacuumed table table=multipart_uploads44832026/11/12 16:58:04 WARN Force mode enabled - objects will be deleted immediately without grace period44842026/11/12 16:58:04 INFO Vacuumed table table=pending_closures44852026/11/12 16:58:04 INFO Vacuumed table table=objects44862026/11/12 16:58:04 INFO Vacuumed table table=closures44872026/11/12 16:58:04 INFO Vacuumed table table=commit_receipts44882026/11/12 16:58:04 INFO Vacuumed table table=pending_closures44892026/11/12 16:58:04 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044902026/11/12 16:58:04 INFO Vacuumed table table=objects44912026/11/12 16:58:04 INFO Vacuumed table table=commit_receipts44922026/11/12 16:58:04 INFO Vacuumed table table=pending_closures44932026/11/12 16:58:04 INFO Received uploads request method=POST path=/44942026/11/12 16:58:04 INFO Aborted multipart uploads count=0 kept=044952026/11/12 16:58:04 INFO Vacuumed table table=pending_objects44962026/11/12 16:58:04 INFO Received complete upload request method=POST path=/44972026/11/12 16:58:04 INFO Vacuumed table table=commit_receipts44982026/11/12 16:58:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044992026/11/12 16:58:04 INFO Received complete upload request method=POST path=/45002026/11/12 16:58:04 INFO Vacuumed table table=multipart_uploads45012026/11/12 16:58:04 INFO Completed upload id=14 objects=6 duration=234.375µs45022026/11/12 16:58:04 INFO Vacuumed table table=pending_closures45032026/11/12 16:58:04 INFO Vacuumed table table=closures45042026/11/12 16:58:04 INFO Vacuumed table table=pending_objects45052026/11/12 16:58:04 INFO Completed upload id=21 objects=9 duration=884.209µs45062026/11/12 16:58:04 INFO Received push request method=POST path=/45072026/11/12 16:58:04 INFO Vacuumed table table=commit_receipts45082026/11/12 16:58:04 INFO Vacuumed table table=multipart_uploads45092026/11/12 16:58:04 INFO Vacuumed table table=pending_objects45102026/11/12 16:58:04 INFO Vacuumed table table=pending_objects45112026/11/12 16:58:04 INFO Vacuumed table table=objects45122026/11/12 16:58:04 INFO Received complete push request method=POST path=/45132026/11/12 16:58:04 INFO Vacuumed table table=multipart_uploads45142026/11/12 16:58:04 INFO Completed push id=25 objects=10 duration=6.09325ms45152026/11/12 16:58:04 INFO Vacuumed table table=closures45162026/11/12 16:58:04 INFO Vacuumed table table=multipart_uploads45172026/11/12 16:58:04 INFO Vacuumed table table=closures45182026/11/12 16:58:04 INFO Vacuumed table table=closures45192026/11/12 16:58:04 INFO Vacuumed table table=objects45202026/11/12 16:58:04 INFO Vacuumed table table=objects45212026/11/12 16:58:04 INFO Received uploads request method=POST path=/45222026/11/12 16:58:04 INFO Received complete upload request method=POST path=/45232026/11/12 16:58:04 INFO Vacuumed table table=objects45242026/11/12 16:58:04 INFO Completed upload id=29 objects=5 duration=800.5µs45252026/11/12 16:58:04 INFO Received complete upload request method=POST path=/45262026/11/12 16:58:04 INFO Completed upload id=25 objects=16 duration=1.006334ms45272026/11/12 16:58:04 INFO Received complete upload request method=POST path=/45282026/11/12 16:58:04 INFO Received uploads request method=POST path=/45292026/11/12 16:58:04 INFO Aborted multipart uploads count=0 kept=045302026/11/12 16:58:04 INFO Aborted multipart uploads count=0 kept=045312026/11/12 16:58:04 WARN Force mode enabled - objects will be deleted immediately without grace period45322026/11/12 16:58:04 INFO Completed upload id=30 objects=2 duration=7.778541ms45332026/11/12 16:58:04 WARN Force mode enabled - objects will be deleted immediately without grace period45342026/11/12 16:58:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045352026/11/12 16:58:04 INFO Received complete push request method=POST path=/45362026/11/12 16:58:04 INFO Received uploads request method=POST path=/45372026/11/12 16:58:04 INFO Completed push id=31 objects=20 duration=465.417µs45382026/11/12 16:58:04 INFO Vacuumed table table=pending_closures45392026/11/12 16:58:04 INFO Vacuumed table table=commit_receipts45402026/11/12 16:58:04 INFO Received complete upload request method=POST path=/45412026/11/12 16:58:04 INFO Vacuumed table table=pending_objects45422026/11/12 16:58:04 INFO Received complete upload request method=POST path=/45432026/11/12 16:58:04 INFO Received push request method=POST path=/45442026/11/12 16:58:04 INFO Completed upload id=14 objects=16 duration=809.292µs45452026/11/12 16:58:04 INFO Completed upload id=32 objects=5 duration=388.25µs45462026/11/12 16:58:04 INFO Vacuumed table table=multipart_uploads45472026/11/12 16:58:04 INFO Received uploads request method=POST path=/45482026/11/12 16:58:04 INFO Received complete upload request method=POST path=/45492026/11/12 16:58:04 INFO Received complete upload request method=POST path=/45502026/11/12 16:58:04 INFO Completed upload id=33 objects=3 duration=327.583µs45512026/11/12 16:58:04 INFO Vacuumed table table=closures45522026/11/12 16:58:04 INFO Completed upload id=7 objects=3 duration=200.125µs45532026/11/12 16:58:04 INFO Aborted multipart uploads count=0 kept=045542026/11/12 16:58:04 INFO Vacuumed table table=objects45552026/11/12 16:58:04 WARN Force mode enabled - objects will be deleted immediately without grace period45562026/11/12 16:58:04 INFO Aborted multipart uploads count=0 kept=045572026/11/12 16:58:04 INFO Received push request method=POST path=/45582026/11/12 16:58:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045592026/11/12 16:58:04 WARN Force mode enabled - objects will be deleted immediately without grace period45602026/11/12 16:58:04 INFO Received push request method=POST path=/45612026/11/12 16:58:04 INFO Vacuumed table table=pending_closures45622026/11/12 16:58:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045632026/11/12 16:58:04 INFO Vacuumed table table=commit_receipts45642026/11/12 16:58:04 INFO Vacuumed table table=pending_objects45652026/11/12 16:58:04 INFO Received complete upload request method=POST path=/45662026/11/12 16:58:04 INFO Vacuumed table table=pending_closures45672026/11/12 16:58:04 INFO Received uploads request method=POST path=/45682026/11/12 16:58:04 INFO Vacuumed table table=multipart_uploads45692026/11/12 16:58:04 INFO Completed upload id=21 objects=25 duration=417.292µs45702026/11/12 16:58:04 INFO Vacuumed table table=closures45712026/11/12 16:58:04 INFO Vacuumed table table=commit_receipts45722026/11/12 16:58:04 INFO Received uploads request method=POST path=/45732026/11/12 16:58:04 INFO Received complete upload request method=POST path=/45742026/11/12 16:58:04 INFO Vacuumed table table=objects45752026/11/12 16:58:04 INFO Completed upload id=22 objects=5 duration=576.208µs45762026/11/12 16:58:04 INFO Received push request method=POST path=/45772026/11/12 16:58:04 INFO Received complete push request method=POST path=/45782026/11/12 16:58:04 INFO Vacuumed table table=pending_objects45792026/11/12 16:58:04 INFO Vacuumed table table=multipart_uploads45802026/11/12 16:58:04 INFO Completed push id=28 objects=3 duration=7.860667ms45812026/11/12 16:58:04 INFO Vacuumed table table=closures45822026/11/12 16:58:04 INFO Vacuumed table table=objects45832026/11/12 16:58:04 INFO Received complete push request method=POST path=/45842026/11/12 16:58:04 INFO Completed push id=29 objects=15 duration=432.5µs45852026/11/12 16:58:04 INFO Received complete push request method=POST path=/45862026/11/12 16:58:04 INFO Completed push id=30 objects=6 duration=310.916µs45872026/11/12 16:58:04 INFO Received complete upload request method=POST path=/45882026/11/12 16:58:04 INFO Completed upload id=31 objects=3 duration=176.667µs45892026/11/12 16:58:04 INFO Received complete push request method=POST path=/45902026/11/12 16:58:04 INFO Completed push id=18 objects=23 duration=532.292µs45912026/11/12 16:58:04 INFO Aborted multipart uploads count=0 kept=045922026/11/12 16:58:04 WARN Force mode enabled - objects will be deleted immediately without grace period45932026/11/12 16:58:04 INFO Received uploads request method=POST path=/45942026/11/12 16:58:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045952026/11/12 16:58:04 INFO Received complete upload request method=POST path=/45962026/11/12 16:58:04 INFO Vacuumed table table=pending_closures45972026/11/12 16:58:04 INFO Completed upload id=23 objects=7 duration=8.721125ms45982026/11/12 16:58:04 INFO Vacuumed table table=commit_receipts45992026/11/12 16:58:04 INFO Received complete upload request method=POST path=/46002026/11/12 16:58:04 INFO Completed upload id=26 objects=5 duration=282.083µs46012026/11/12 16:58:04 INFO Vacuumed table table=pending_objects46022026/11/12 16:58:04 INFO Vacuumed table table=multipart_uploads46032026/11/12 16:58:04 INFO Vacuumed table table=closures46042026/11/12 16:58:04 INFO Vacuumed table table=objects46052026/11/12 16:58:04 INFO Received complete push request method=POST path=/46062026/11/12 16:58:04 INFO Completed push id=27 objects=3 duration=536.458µs46072026/11/12 16:58:04 INFO Received complete upload request method=POST path=/46082026/11/12 16:58:04 INFO Completed upload id=28 objects=5 duration=287.917µs46092026/11/12 16:58:04 INFO Received complete push request method=POST path=/46102026/11/12 16:58:04 INFO Received complete push request method=POST path=/46112026/11/12 16:58:04 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."46122026/11/12 16:58:04 INFO Completed push id=11 objects=9 duration=536.541µs46132026/11/12 16:58:04 INFO Completed push id=29 objects=17 duration=337.583µs46142026/11/12 16:58:04 INFO Received complete push request method=POST path=/46152026/11/12 16:58:04 INFO Completed push id=17 objects=11 duration=1.358291ms46162026/11/12 16:58:04 INFO Received push request method=POST path=/46172026/11/12 16:58:04 INFO Aborted multipart uploads count=0 kept=046182026/11/12 16:58:04 WARN Force mode enabled - objects will be deleted immediately without grace period46192026/11/12 16:58:04 INFO Received push request method=POST path=/46202026/11/12 16:58:04 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046212026/11/12 16:58:04 INFO Received push request method=POST path=/46222026/11/12 16:58:04 INFO Received uploads request method=POST path=/46232026/11/12 16:58:04 INFO Aborted multipart uploads count=0 kept=046242026/11/12 16:58:04 INFO Received push request method=POST path=/46252026/11/12 16:58:04 WARN Force mode enabled - objects will be deleted immediately without grace period46262026/11/12 16:58:04 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046272026/11/12 16:58:04 INFO Vacuumed table table=pending_closures46282026/11/12 16:58:04 INFO Vacuumed table table=pending_closures46292026/11/12 16:58:04 INFO Vacuumed table table=commit_receipts46302026/11/12 16:58:04 INFO Received complete upload request method=POST path=/46312026/11/12 16:58:04 INFO Vacuumed table table=commit_receipts46322026/11/12 16:58:04 INFO Completed upload id=26 objects=6 duration=499.042µs46332026/11/12 16:58:04 INFO Vacuumed table table=pending_objects46342026/11/12 16:58:04 INFO Vacuumed table table=multipart_uploads46352026/11/12 16:58:04 INFO Received uploads request method=POST path=/46362026/11/12 16:58:04 INFO Vacuumed table table=closures46372026/11/12 16:58:04 INFO Received complete upload request method=POST path=/46382026/11/12 16:58:04 INFO Vacuumed table table=objects46392026/11/12 16:58:04 INFO Vacuumed table table=pending_objects46402026/11/12 16:58:04 INFO Completed upload id=29 objects=5 duration=294.75µs46412026/11/12 16:58:04 INFO Vacuumed table table=multipart_uploads46422026/11/12 16:58:04 INFO Received uploads request method=POST path=/46432026/11/12 16:58:04 INFO Vacuumed table table=closures46442026/11/12 16:58:04 INFO Vacuumed table table=objects46452026/11/12 16:58:04 INFO Aborted multipart uploads count=0 kept=046462026/11/12 16:58:04 WARN Force mode enabled - objects will be deleted immediately without grace period46472026/11/12 16:58:04 INFO Received push request method=POST path=/46482026/11/12 16:58:05 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046492026/11/12 16:58:05 INFO Received complete upload request method=POST path=/46502026/11/12 16:58:05 INFO Completed upload id=12 objects=6 duration=502.333µs46512026/11/12 16:58:05 INFO Vacuumed table table=pending_closures46522026/11/12 16:58:05 INFO Vacuumed table table=commit_receipts46532026/11/12 16:58:05 INFO Received complete upload request method=POST path=/46542026/11/12 16:58:05 INFO Vacuumed table table=pending_objects46552026/11/12 16:58:05 INFO Received complete push request method=POST path=/46562026/11/12 16:58:05 INFO Received complete push request method=POST path=/46572026/11/12 16:58:05 INFO Completed upload id=23 objects=15 duration=808.916µs46582026/11/12 16:58:05 INFO Vacuumed table table=multipart_uploads46592026/11/12 16:58:05 INFO Completed push id=17 objects=3 duration=519.416µs46602026/11/12 16:58:05 INFO Vacuumed table table=closures46612026/11/12 16:58:05 INFO Completed push id=5 objects=9 duration=7.82925ms46622026/11/12 16:58:05 INFO Received push request method=POST path=/46632026/11/12 16:58:05 INFO Vacuumed table table=objects46642026/11/12 16:58:05 INFO Received push request method=POST path=/46652026/11/12 16:58:05 INFO Received push request method=POST path=/46662026/11/12 16:58:05 INFO Received uploads request method=POST path=/46672026/11/12 16:58:05 INFO Received complete push request method=POST path=/46682026/11/12 16:58:05 INFO Completed push id=13 objects=15 duration=726.375µs46692026/11/12 16:58:05 INFO Received uploads request method=POST path=/46702026/11/12 16:58:05 INFO Received uploads request method=POST path=/46712026/11/12 16:58:05 INFO Aborted multipart uploads count=0 kept=046722026/11/12 16:58:05 INFO Received push request method=POST path=/46732026/11/12 16:58:05 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."46742026/11/12 16:58:05 INFO Received complete upload request method=POST path=/46752026/11/12 16:58:05 INFO Aborted multipart uploads count=0 kept=046762026/11/12 16:58:05 INFO Received complete upload request method=POST path=/46772026/11/12 16:58:05 INFO Completed upload id=19 objects=6 duration=1.1245ms46782026/11/12 16:58:05 INFO Received complete push request method=POST path=/46792026/11/12 16:58:05 WARN Force mode enabled - objects will be deleted immediately without grace period46802026/11/12 16:58:05 INFO Completed push id=23 objects=5 duration=7.094916ms46812026/11/12 16:58:05 INFO Completed upload id=31 objects=3 duration=8.658ms46822026/11/12 16:58:05 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046832026/11/12 16:58:05 INFO Vacuumed table table=pending_closures46842026/11/12 16:58:05 INFO Vacuumed table table=commit_receipts46852026/11/12 16:58:05 INFO Vacuumed table table=pending_objects46862026/11/12 16:58:05 INFO Received complete upload request method=POST path=/46872026/11/12 16:58:05 INFO Received complete upload request method=POST path=/46882026/11/12 16:58:05 INFO Vacuumed table table=multipart_uploads46892026/11/12 16:58:05 INFO Completed upload id=22 objects=2 duration=5.889375ms46902026/11/12 16:58:05 INFO Completed upload id=24 objects=3 duration=321.375µs46912026/11/12 16:58:05 INFO Aborted multipart uploads count=0 kept=046922026/11/12 16:58:05 WARN Force mode enabled - objects will be deleted immediately without grace period46932026/11/12 16:58:05 INFO Received complete push request method=POST path=/46942026/11/12 16:58:05 INFO Completed push id=25 objects=4 duration=243µs46952026/11/12 16:58:05 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046962026/11/12 16:58:05 INFO Vacuumed table table=pending_closures46972026/11/12 16:58:05 INFO Vacuumed table table=closures46982026/11/12 16:58:05 INFO Vacuumed table table=commit_receipts46992026/11/12 16:58:05 INFO Vacuumed table table=pending_objects47002026/11/12 16:58:05 INFO Vacuumed table table=objects47012026/11/12 16:58:05 INFO Vacuumed table table=multipart_uploads47022026/11/12 16:58:05 INFO Vacuumed table table=closures47032026/11/12 16:58:05 INFO Aborted multipart uploads count=0 kept=047042026/11/12 16:58:05 INFO Vacuumed table table=objects47052026/11/12 16:58:05 WARN Force mode enabled - objects will be deleted immediately without grace period47062026/11/12 16:58:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047072026/11/12 16:58:05 INFO Received complete upload request method=POST path=/47082026/11/12 16:58:05 INFO Received complete upload request method=POST path=/47092026/11/12 16:58:05 INFO Vacuumed table table=pending_closures47102026/11/12 16:58:05 INFO Vacuumed table table=commit_receipts47112026/11/12 16:58:05 INFO Completed upload id=18 objects=2 duration=6.220417ms47122026/11/12 16:58:05 INFO Completed upload id=22 objects=7 duration=6.174083ms47132026/11/12 16:58:05 INFO Vacuumed table table=pending_objects47142026/11/12 16:58:05 INFO Vacuumed table table=multipart_uploads47152026/11/12 16:58:05 INFO Vacuumed table table=closures47162026/11/12 16:58:05 INFO Received push request method=POST path=/47172026/11/12 16:58:05 INFO Vacuumed table table=objects47182026/11/12 16:58:05 INFO Received complete push request method=POST path=/47192026/11/12 16:58:05 INFO Completed push id=24 objects=9 duration=437.583µs47202026/11/12 16:58:05 INFO Received complete push request method=POST path=/47212026/11/12 16:58:05 INFO Received push request method=POST path=/47222026/11/12 16:58:05 INFO Completed push id=15 objects=24 duration=1.16075ms47232026/11/12 16:58:05 INFO Received push request method=POST path=/47242026/11/12 16:58:05 INFO Received uploads request method=POST path=/47252026/11/12 16:58:05 INFO Received uploads request method=POST path=/47262026/11/12 16:58:05 INFO Received uploads request method=POST path=/47272026/11/12 16:58:05 INFO Received uploads request method=POST path=/47282026/11/12 16:58:05 INFO Received complete push request method=POST path=/47292026/11/12 16:58:05 INFO Completed push id=21 objects=3 duration=270.708µs47302026/11/12 16:58:05 INFO Received complete upload request method=POST path=/47312026/11/12 16:58:05 INFO Received uploads request method=POST path=/47322026/11/12 16:58:05 INFO Completed upload id=19 objects=25 duration=857.292µs47332026/11/12 16:58:05 INFO Received complete push request method=POST path=/47342026/11/12 16:58:05 INFO Completed push id=20 objects=3 duration=357.667µs47352026/11/12 16:58:05 INFO Aborted multipart uploads count=0 kept=047362026/11/12 16:58:05 INFO Aborted multipart uploads count=0 kept=047372026/11/12 16:58:05 WARN Force mode enabled - objects will be deleted immediately without grace period47382026/11/12 16:58:05 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=3 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047392026/11/12 16:58:05 INFO Vacuumed table table=pending_closures47402026/11/12 16:58:05 INFO Received push request method=POST path=/47412026/11/12 16:58:05 INFO Vacuumed table table=commit_receipts47422026/11/12 16:58:05 INFO Received complete upload request method=POST path=/47432026/11/12 16:58:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=047442026/11/12 16:58:05 INFO Received push request method=POST path=/47452026/11/12 16:58:05 INFO Received complete push request method=POST path=/47462026/11/12 16:58:05 INFO Completed upload id=6 objects=6 duration=471.292µs47472026/11/12 16:58:05 INFO Vacuumed table table=pending_closures47482026/11/12 16:58:05 INFO Received push request method=POST path=/47492026/11/12 16:58:05 INFO Vacuumed table table=commit_receipts47502026/11/12 16:58:05 INFO Vacuumed table table=pending_objects47512026/11/12 16:58:05 INFO Vacuumed table table=pending_objects47522026/11/12 16:58:05 INFO Completed push id=11 objects=11 duration=1.402ms47532026/11/12 16:58:05 INFO Received complete upload request method=POST path=/47542026/11/12 16:58:05 INFO Aborted multipart uploads count=0 kept=047552026/11/12 16:58:05 INFO Vacuumed table table=multipart_uploads47562026/11/12 16:58:05 INFO Vacuumed table table=multipart_uploads47572026/11/12 16:58:05 INFO Received complete upload request method=POST path=/47582026/11/12 16:58:05 INFO Completed upload id=22 objects=6 duration=6.225916ms47592026/11/12 16:58:05 INFO Completed upload id=13 objects=7 duration=245.584µs47602026/11/12 16:58:05 WARN Force mode enabled - objects will be deleted immediately without grace period47612026/11/12 16:58:05 INFO Vacuumed table table=closures47622026/11/12 16:58:05 INFO Vacuumed table table=closures47632026/11/12 16:58:05 INFO Vacuumed table table=objects47642026/11/12 16:58:05 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047652026/11/12 16:58:05 INFO Vacuumed table table=objects47662026/11/12 16:58:05 INFO Vacuumed table table=pending_closures47672026/11/12 16:58:05 INFO Received uploads request method=POST path=/47682026/11/12 16:58:05 INFO Vacuumed table table=commit_receipts47692026/11/12 16:58:05 INFO Received complete upload request method=POST path=/47702026/11/12 16:58:05 INFO Vacuumed table table=pending_objects47712026/11/12 16:58:05 INFO Received push request method=POST path=/47722026/11/12 16:58:05 INFO Completed upload id=20 objects=4 duration=267.416µs47732026/11/12 16:58:05 INFO Vacuumed table table=multipart_uploads47742026/11/12 16:58:05 INFO Received push request method=POST path=/47752026/11/12 16:58:05 INFO Received uploads request method=POST path=/47762026/11/12 16:58:05 INFO Received complete upload request method=POST path=/47772026/11/12 16:58:05 INFO Received push request method=POST path=/47782026/11/12 16:58:05 INFO Vacuumed table table=closures47792026/11/12 16:58:05 INFO Completed upload id=23 objects=7 duration=256.791µs47802026/11/12 16:58:05 INFO Received uploads request method=POST path=/47812026/11/12 16:58:05 INFO Received complete push request method=POST path=/47822026/11/12 16:58:05 INFO Vacuumed table table=objects47832026/11/12 16:58:05 INFO Completed push id=30 objects=4 duration=877.667µs47842026/11/12 16:58:05 INFO Received push request method=POST path=/47852026/11/12 16:58:05 INFO Received push request method=POST path=/47862026/11/12 16:58:05 INFO Received uploads request method=POST path=/47872026/11/12 16:58:05 INFO Received uploads request method=POST path=/47882026/11/12 16:58:05 INFO Received uploads request method=POST path=/47892026/11/12 16:58:05 INFO Received push request method=POST path=/47902026/11/12 16:58:05 INFO Received push request method=POST path=/47912026/11/12 16:58:05 INFO Received push request method=POST path=/47922026/11/12 16:58:05 INFO Received push request method=POST path=/47932026/11/12 16:58:05 INFO Received push request method=POST path=/47942026/11/12 16:58:05 INFO Aborted multipart uploads count=0 kept=047952026/11/12 16:58:05 INFO Received uploads request method=POST path=/47962026/11/12 16:58:05 WARN Force mode enabled - objects will be deleted immediately without grace period47972026/11/12 16:58:05 INFO Received uploads request method=POST path=/47982026/11/12 16:58:05 INFO Received uploads request method=POST path=/47992026/11/12 16:58:05 INFO Received complete push request method=POST path=/48002026/11/12 16:58:05 INFO Completed push id=32 objects=7 duration=528.666µs48012026/11/12 16:58:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048022026/11/12 16:58:05 INFO Vacuumed table table=pending_closures48032026/11/12 16:58:05 INFO Received complete push request method=POST path=/48042026/11/12 16:58:05 INFO Received uploads request method=POST path=/48052026/11/12 16:58:05 INFO Vacuumed table table=commit_receipts48062026/11/12 16:58:05 INFO Received complete push request method=POST path=/48072026/11/12 16:58:05 INFO Completed push id=26 objects=4 duration=292.75µs48082026/11/12 16:58:05 INFO Completed push id=16 objects=4 duration=6.532125ms48092026/11/12 16:58:05 INFO Vacuumed table table=pending_objects48102026/11/12 16:58:05 INFO Received complete upload request method=POST path=/48112026/11/12 16:58:05 INFO Received uploads request method=POST path=/48122026/11/12 16:58:05 INFO Vacuumed table table=multipart_uploads48132026/11/12 16:58:05 INFO Completed upload id=28 objects=13 duration=285.334µs48142026/11/12 16:58:05 INFO Received uploads request method=POST path=/48152026/11/12 16:58:05 INFO Vacuumed table table=closures48162026/11/12 16:58:05 INFO Vacuumed table table=objects48172026/11/12 16:58:05 INFO Received complete push request method=POST path=/48182026/11/12 16:58:05 INFO Completed push id=29 objects=3 duration=223.208µs48192026/11/12 16:58:05 INFO Received uploads request method=POST path=/48202026/11/12 16:58:05 INFO Received complete upload request method=POST path=/48212026/11/12 16:58:05 INFO Received push request method=POST path=/48222026/11/12 16:58:05 INFO Completed upload id=20 objects=5 duration=5.231042ms48232026/11/12 16:58:05 INFO Received complete upload request method=POST path=/48242026/11/12 16:58:05 INFO Completed upload id=31 objects=2 duration=271.75µs48252026/11/12 16:58:05 INFO Received uploads request method=POST path=/48262026/11/12 16:58:05 INFO Received uploads request method=POST path=/48272026/11/12 16:58:05 INFO Received push request method=POST path=/48282026/11/12 16:58:05 INFO Received complete push request method=POST path=/48292026/11/12 16:58:05 INFO Completed push id=26 objects=23 duration=479.792µs48302026/11/12 16:58:05 INFO Received complete push request method=POST path=/48312026/11/12 16:58:05 INFO Received push request method=POST path=/48322026/11/12 16:58:05 INFO Aborted multipart uploads count=0 kept=048332026/11/12 16:58:05 INFO Completed push id=33 objects=6 duration=225.625µs48342026/11/12 16:58:05 WARN Force mode enabled - objects will be deleted immediately without grace period48352026/11/12 16:58:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048362026/11/12 16:58:05 INFO Aborted multipart uploads count=0 kept=048372026/11/12 16:58:05 INFO Received push request method=POST path=/48382026/11/12 16:58:05 INFO Vacuumed table table=pending_closures48392026/11/12 16:58:05 WARN Force mode enabled - objects will be deleted immediately without grace period48402026/11/12 16:58:05 INFO Vacuumed table table=commit_receipts48412026/11/12 16:58:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048422026/11/12 16:58:05 INFO Vacuumed table table=pending_objects48432026/11/12 16:58:05 INFO Received push request method=POST path=/48442026/11/12 16:58:05 INFO Vacuumed table table=multipart_uploads48452026/11/12 16:58:05 INFO Vacuumed table table=closures48462026/11/12 16:58:05 INFO Vacuumed table table=pending_closures48472026/11/12 16:58:05 INFO Vacuumed table table=objects48482026/11/12 16:58:05 INFO Aborted multipart uploads count=0 kept=048492026/11/12 16:58:05 INFO Vacuumed table table=commit_receipts48502026/11/12 16:58:05 WARN Force mode enabled - objects will be deleted immediately without grace period48512026/11/12 16:58:05 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048522026/11/12 16:58:05 INFO Aborted multipart uploads count=0 kept=048532026/11/12 16:58:05 INFO Vacuumed table table=pending_closures48542026/11/12 16:58:05 INFO Vacuumed table table=commit_receipts48552026/11/12 16:58:05 WARN Force mode enabled - objects will be deleted immediately without grace period48562026/11/12 16:58:05 INFO Vacuumed table table=pending_objects48572026/11/12 16:58:05 INFO Vacuumed table table=multipart_uploads48582026/11/12 16:58:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048592026/11/12 16:58:05 INFO Vacuumed table table=pending_objects48602026/11/12 16:58:05 INFO Vacuumed table table=closures48612026/11/12 16:58:05 INFO Vacuumed table table=pending_closures48622026/11/12 16:58:05 INFO Vacuumed table table=multipart_uploads48632026/11/12 16:58:05 INFO Vacuumed table table=commit_receipts48642026/11/12 16:58:05 INFO Vacuumed table table=objects48652026/11/12 16:58:05 INFO Vacuumed table table=closures48662026/11/12 16:58:05 INFO Vacuumed table table=pending_objects48672026/11/12 16:58:05 INFO Vacuumed table table=objects48682026/11/12 16:58:05 INFO Vacuumed table table=multipart_uploads48692026/11/12 16:58:05 INFO Vacuumed table table=closures48702026/11/12 16:58:05 INFO Vacuumed table table=objects48712026/11/12 16:58:05 INFO Aborted multipart uploads count=0 kept=048722026/11/12 16:58:05 INFO Received complete push request method=POST path=/48732026/11/12 16:58:05 INFO Received push request method=POST path=/48742026/11/12 16:58:05 INFO Completed push id=4 objects=6 duration=6.27675ms48752026/11/12 16:58:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048762026/11/12 16:58:05 INFO Vacuumed table table=pending_closures48772026/11/12 16:58:05 INFO Vacuumed table table=commit_receipts48782026/11/12 16:58:05 INFO Vacuumed table table=pending_objects48792026/11/12 16:58:05 INFO Vacuumed table table=multipart_uploads48802026/11/12 16:58:05 INFO Vacuumed table table=closures48812026/11/12 16:58:05 INFO Vacuumed table table=objects48822026/11/12 16:58:05 INFO Received push request method=POST path=/48832026/11/12 16:58:05 INFO Received push request method=POST path=/48842026/11/12 16:58:05 INFO Aborted multipart uploads count=0 kept=048852026/11/12 16:58:05 WARN Force mode enabled - objects will be deleted immediately without grace period48862026/11/12 16:58:05 INFO Received complete push request method=POST path=/48872026/11/12 16:58:05 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048882026/11/12 16:58:05 INFO Vacuumed table table=pending_closures48892026/11/12 16:58:05 INFO Completed push id=27 objects=15 duration=4.774333ms48902026/11/12 16:58:05 INFO Vacuumed table table=commit_receipts48912026/11/12 16:58:05 INFO Vacuumed table table=pending_objects48922026/11/12 16:58:05 INFO Vacuumed table table=multipart_uploads48932026/11/12 16:58:05 INFO Vacuumed table table=closures48942026/11/12 16:58:05 INFO Vacuumed table table=objects48952026/11/12 16:58:05 INFO Received complete upload request method=POST path=/48962026/11/12 16:58:05 INFO Completed upload id=24 objects=18 duration=308.75µs48972026/11/12 16:58:05 INFO Received push request method=POST path=/48982026/11/12 16:58:05 INFO Received complete push request method=POST path=/48992026/11/12 16:58:05 INFO Completed push id=25 objects=2 duration=568.75µs49002026/11/12 16:58:05 INFO Received push request method=POST path=/49012026/11/12 16:58:05 INFO Received complete push request method=POST path=/49022026/11/12 16:58:05 INFO Received complete push request method=POST path=/49032026/11/12 16:58:05 INFO Completed push id=27 objects=4 duration=3.903ms49042026/11/12 16:58:05 INFO Completed push id=31 objects=14 duration=796.416µs49052026/11/12 16:58:05 INFO Received uploads request method=POST path=/49062026/11/12 16:58:05 INFO Received complete upload request method=POST path=/49072026/11/12 16:58:05 INFO Completed upload id=9 objects=2 duration=374.708µs49082026/11/12 16:58:05 INFO Received complete push request method=POST path=/49092026/11/12 16:58:05 INFO Received complete upload request method=POST path=/49102026/11/12 16:58:05 INFO Completed push id=8 objects=3 duration=367.291µs49112026/11/12 16:58:05 INFO Completed upload id=23 objects=16 duration=683.958µs49122026/11/12 16:58:05 INFO Received push request method=POST path=/49132026/11/12 16:58:05 INFO Aborted multipart uploads count=0 kept=049142026/11/12 16:58:05 WARN Force mode enabled - objects will be deleted immediately without grace period49152026/11/12 16:58:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049162026/11/12 16:58:05 INFO Received uploads request method=POST path=/49172026/11/12 16:58:05 INFO Vacuumed table table=pending_closures49182026/11/12 16:58:05 INFO Vacuumed table table=commit_receipts49192026/11/12 16:58:05 INFO Received complete upload request method=POST path=/49202026/11/12 16:58:05 INFO Completed upload id=13 objects=10 duration=233.25µs49212026/11/12 16:58:05 INFO Vacuumed table table=pending_objects49222026/11/12 16:58:05 INFO Vacuumed table table=multipart_uploads49232026/11/12 16:58:05 INFO Vacuumed table table=closures49242026/11/12 16:58:05 INFO Aborted multipart uploads count=0 kept=049252026/11/12 16:58:05 WARN Force mode enabled - objects will be deleted immediately without grace period49262026/11/12 16:58:05 INFO Vacuumed table table=objects49272026/11/12 16:58:05 INFO Received uploads request method=POST path=/49282026/11/12 16:58:05 INFO Received complete push request method=POST path=/49292026/11/12 16:58:05 INFO Completed push id=28 objects=26 duration=457.5µs49302026/11/12 16:58:06 INFO Received complete push request method=POST path=/49312026/11/12 16:58:06 INFO Completed push id=30 objects=3 duration=192.75µs49322026/11/12 16:58:06 INFO Received complete push request method=POST path=/49332026/11/12 16:58:06 INFO Completed push id=32 objects=19 duration=209.25µs49342026/11/12 16:58:06 INFO Aborted multipart uploads count=0 kept=049352026/11/12 16:58:06 WARN Force mode enabled - objects will be deleted immediately without grace period49362026/11/12 16:58:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049372026/11/12 16:58:06 INFO Vacuumed table table=pending_closures49382026/11/12 16:58:06 INFO Vacuumed table table=commit_receipts49392026/11/12 16:58:06 INFO Received complete push request method=POST path=/49402026/11/12 16:58:06 INFO Vacuumed table table=pending_objects49412026/11/12 16:58:06 INFO Completed push id=16 objects=7 duration=7.943875ms49422026/11/12 16:58:06 INFO Vacuumed table table=multipart_uploads49432026/11/12 16:58:06 INFO Vacuumed table table=closures49442026/11/12 16:58:06 INFO Received complete push request method=POST path=/49452026/11/12 16:58:06 INFO Vacuumed table table=objects49462026/11/12 16:58:06 INFO Completed push id=28 objects=19 duration=367.5µs49472026/11/12 16:58:06 INFO Received complete push request method=POST path=/49482026/11/12 16:58:06 INFO Completed push id=7 objects=7 duration=623.042µs49492026/11/12 16:58:06 INFO Received complete push request method=POST path=/49502026/11/12 16:58:06 INFO Completed push id=12 objects=6 duration=352.25µs49512026/11/12 16:58:06 INFO Received complete push request method=POST path=/49522026/11/12 16:58:06 INFO Completed push id=23 objects=6 duration=552.25µs49532026/11/12 16:58:06 INFO Received complete push request method=POST path=/49542026/11/12 16:58:06 INFO Completed push id=24 objects=6 duration=342.417µs49552026/11/12 16:58:06 INFO Received complete push request method=POST path=/49562026/11/12 16:58:06 INFO Completed push id=29 objects=10 duration=342.833µs49572026/11/12 16:58:06 INFO Received complete upload request method=POST path=/49582026/11/12 16:58:06 INFO Completed upload id=30 objects=6 duration=533.25µs49592026/11/12 16:58:06 INFO Received complete push request method=POST path=/49602026/11/12 16:58:06 INFO Completed push id=25 objects=7 duration=578µs49612026/11/12 16:58:06 INFO Received complete upload request method=POST path=/49622026/11/12 16:58:06 INFO Completed upload id=31 objects=5 duration=290.375µs49632026/11/12 16:58:06 INFO Received complete upload request method=POST path=/49642026/11/12 16:58:06 INFO Completed upload id=34 objects=3 duration=445.75µs49652026/11/12 16:58:06 INFO Aborted multipart uploads count=0 kept=049662026/11/12 16:58:06 WARN Force mode enabled - objects will be deleted immediately without grace period49672026/11/12 16:58:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049682026/11/12 16:58:06 INFO Vacuumed table table=pending_closures49692026/11/12 16:58:06 INFO Vacuumed table table=commit_receipts49702026/11/12 16:58:06 INFO Vacuumed table table=pending_objects49712026/11/12 16:58:06 INFO Vacuumed table table=multipart_uploads49722026/11/12 16:58:06 INFO Vacuumed table table=closures49732026/11/12 16:58:06 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=11 objects-failed-to-delete=049742026/11/12 16:58:06 INFO Vacuumed table table=pending_closures49752026/11/12 16:58:06 INFO Vacuumed table table=objects49762026/11/12 16:58:06 INFO Vacuumed table table=commit_receipts49772026/11/12 16:58:06 INFO Vacuumed table table=pending_objects49782026/11/12 16:58:06 INFO Vacuumed table table=multipart_uploads49792026/11/12 16:58:06 INFO Received complete push request method=POST path=/49802026/11/12 16:58:06 INFO Vacuumed table table=closures49812026/11/12 16:58:06 INFO Completed push id=7 objects=15 duration=7.253833ms49822026/11/12 16:58:06 INFO Vacuumed table table=objects49832026/11/12 16:58:06 INFO Received complete push request method=POST path=/49842026/11/12 16:58:06 INFO Received push request method=POST path=/49852026/11/12 16:58:06 INFO Completed push id=13 objects=22 duration=841.917µs49862026/11/12 16:58:06 INFO Received push request method=POST path=/49872026/11/12 16:58:06 INFO Received complete upload request method=POST path=/49882026/11/12 16:58:06 INFO Completed upload id=26 objects=24 duration=452.292µs49892026/11/12 16:58:06 INFO Received complete push request method=POST path=/49902026/11/12 16:58:06 INFO Received push request method=POST path=/49912026/11/12 16:58:06 INFO Completed push id=21 objects=6 duration=341.791µs49922026/11/12 16:58:06 INFO Received push request method=POST path=/49932026/11/12 16:58:06 INFO Received complete upload request method=POST path=/49942026/11/12 16:58:06 INFO Completed upload id=22 objects=7 duration=344.791µs49952026/11/12 16:58:06 INFO Received complete push request method=POST path=/49962026/11/12 16:58:06 INFO Completed push id=24 objects=4 duration=246.083µs49972026/11/12 16:58:06 INFO Received complete push request method=POST path=/49982026/11/12 16:58:06 INFO Completed push id=25 objects=9 duration=195.708µs49992026/11/12 16:58:06 INFO Received complete push request method=POST path=/50002026/11/12 16:58:06 INFO Completed push id=14 objects=4 duration=1.034667ms50012026/11/12 16:58:06 INFO Received complete upload request method=POST path=/50022026/11/12 16:58:06 INFO Completed upload id=26 objects=3 duration=139.958µs50032026/11/12 16:58:06 INFO Received push request method=POST path=/50042026/11/12 16:58:06 INFO Received uploads request method=POST path=/50052026/11/12 16:58:06 INFO Aborted multipart uploads count=0 kept=050062026/11/12 16:58:06 WARN Force mode enabled - objects will be deleted immediately without grace period50072026/11/12 16:58:06 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=050082026/11/12 16:58:06 INFO Vacuumed table table=pending_closures50092026/11/12 16:58:06 INFO Vacuumed table table=commit_receipts50102026/11/12 16:58:06 INFO Vacuumed table table=pending_objects50112026/11/12 16:58:06 INFO Received complete upload request method=POST path=/50122026/11/12 16:58:06 INFO Vacuumed table table=multipart_uploads50132026/11/12 16:58:06 INFO Completed upload id=27 objects=7 duration=379.583µs50142026/11/12 16:58:06 INFO Vacuumed table table=closures50152026/11/12 16:58:06 INFO Vacuumed table table=objects50162026/11/12 16:58:06 INFO Received complete upload request method=POST path=/50172026/11/12 16:58:06 INFO Completed upload id=28 objects=3 duration=217.459µs50182026/11/12 16:58:06 INFO Received complete push request method=POST path=/50192026/11/12 16:58:06 INFO Completed push id=29 objects=23 duration=357.125µs50202026/11/12 16:58:06 INFO Received complete push request method=POST path=/50212026/11/12 16:58:06 INFO Completed push id=30 objects=6 duration=229.375µs50222026/11/12 16:58:06 INFO Received complete upload request method=POST path=/50232026/11/12 16:58:06 INFO Completed upload id=27 objects=26 duration=281.792µs50242026/11/12 16:58:06 INFO Received complete push request method=POST path=/50252026/11/12 16:58:06 INFO Completed push id=28 objects=6 duration=274.375µs50262026/11/12 16:58:06 INFO Received complete push request method=POST path=/50272026/11/12 16:58:06 INFO Completed push id=29 objects=6 duration=241.375µs50282026/11/12 16:58:06 INFO Received complete push request method=POST path=/50292026/11/12 16:58:06 INFO Completed push id=33 objects=3 duration=283.75µs50302026/11/12 16:58:06 INFO Received complete push request method=POST path=/50312026/11/12 16:58:06 INFO Completed push id=34 objects=6 duration=221.584µs50322026/11/12 16:58:06 INFO Received complete push request method=POST path=/50332026/11/12 16:58:06 INFO Completed push id=35 objects=8 duration=216.125µs50342026/11/12 16:58:06 INFO Received complete upload request method=POST path=/50352026/11/12 16:58:06 INFO Completed upload id=36 objects=9 duration=201.417µs50362026/11/12 16:58:06 INFO Aborted multipart uploads count=0 kept=050372026/11/12 16:58:06 WARN Force mode enabled - objects will be deleted immediately without grace period50382026/11/12 16:58:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=050392026/11/12 16:58:06 INFO Vacuumed table table=pending_closures50402026/11/12 16:58:07 INFO Vacuumed table table=commit_receipts50412026/11/12 16:58:07 INFO Received complete push request method=POST path=/50422026/11/12 16:58:07 INFO Vacuumed table table=pending_objects50432026/11/12 16:58:07 INFO Vacuumed table table=multipart_uploads50442026/11/12 16:58:07 INFO Received complete push request method=POST path=/50452026/11/12 16:58:07 INFO Vacuumed table table=closures50462026/11/12 16:58:07 INFO Completed push id=19 objects=14 duration=641.584µs50472026/11/12 16:58:07 INFO Completed push id=16 objects=16 duration=7.236875ms50482026/11/12 16:58:07 INFO Vacuumed table table=objects50492026/11/12 16:58:07 INFO Aborted multipart uploads count=0 kept=050502026/11/12 16:58:07 WARN Force mode enabled - objects will be deleted immediately without grace period50512026/11/12 16:58:07 INFO Received complete upload request method=POST path=/50522026/11/12 16:58:07 INFO Aborted multipart uploads count=0 kept=050532026/11/12 16:58:07 INFO Completed upload id=30 objects=8 duration=227.667µs50542026/11/12 16:58:07 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."50552026/11/12 16:58:07 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."50562026/11/12 16:58:07 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."50572026/11/12 16:58:07 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."50582026/11/12 16:58:07 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."50592026/11/12 16:58:07 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."50602026/11/12 16:58:07 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."50612026/11/12 16:58:07 INFO Received uploads request method=POST path=/50622026/11/12 16:58:07 INFO Received uploads request method=POST path=/50632026/11/12 16:58:07 INFO Received push request method=POST path=/50642026/11/12 16:58:07 INFO Received uploads request method=POST path=/50652026/11/12 16:58:07 INFO Aborted multipart uploads count=0 kept=050662026/11/12 16:58:07 WARN Force mode enabled - objects will be deleted immediately without grace period50672026/11/12 16:58:07 INFO Received complete upload request method=POST path=/50682026/11/12 16:58:07 INFO Completed upload id=31 objects=6 duration=231.25µs50692026/11/12 16:58:07 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."50702026/11/12 16:58:07 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."50712026/11/12 16:58:07 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."50722026/11/12 16:58:07 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."50732026/11/12 16:58:07 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."50742026/11/12 16:58:07 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."50752026/11/12 16:58:07 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."50762026/11/12 16:58:07 INFO Received complete upload request method=POST path=/50772026/11/12 16:58:07 INFO Completed upload id=32 objects=4 duration=1.094792ms50782026/11/12 16:58:07 INFO Received complete upload request method=POST path=/50792026/11/12 16:58:07 INFO Completed upload id=21 objects=3 duration=191.459µs50802026/11/12 16:58:07 INFO Aborted multipart uploads count=0 kept=050812026/11/12 16:58:07 WARN Force mode enabled - objects will be deleted immediately without grace period50822026/11/12 16:58:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=7 objects-failed-to-delete=050832026/11/12 16:58:07 INFO Vacuumed table table=pending_closures50842026/11/12 16:58:07 INFO Vacuumed table table=commit_receipts50852026/11/12 16:58:07 INFO Vacuumed table table=pending_objects50862026/11/12 16:58:07 INFO Vacuumed table table=multipart_uploads50872026/11/12 16:58:07 INFO Vacuumed table table=closures50882026/11/12 16:58:07 INFO Vacuumed table table=objects50892026/11/12 16:58:07 INFO Received complete upload request method=POST path=/50902026/11/12 16:58:07 INFO Completed upload id=33 objects=10 duration=219.625µs50912026/11/12 16:58:07 INFO Aborted multipart uploads count=0 kept=050922026/11/12 16:58:07 WARN Force mode enabled - objects will be deleted immediately without grace period50932026/11/12 16:58:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=050942026/11/12 16:58:07 INFO Vacuumed table table=pending_closures50952026/11/12 16:58:07 INFO Vacuumed table table=commit_receipts50962026/11/12 16:58:07 INFO Vacuumed table table=pending_objects50972026/11/12 16:58:07 INFO Vacuumed table table=multipart_uploads50982026/11/12 16:58:07 INFO Vacuumed table table=closures50992026/11/12 16:58:07 INFO Vacuumed table table=objects51002026/11/12 16:58:07 INFO Received push request method=POST path=/51012026/11/12 16:58:07 INFO Received complete push request method=POST path=/51022026/11/12 16:58:07 INFO Completed push id=22 objects=3 duration=823.75µs51032026/11/12 16:58:07 INFO Received uploads request method=POST path=/51042026/11/12 16:58:07 INFO Received push request method=POST path=/51052026/11/12 16:58:07 INFO Received complete push request method=POST path=/51062026/11/12 16:58:07 INFO Completed push id=24 objects=3 duration=270.209µs51072026/11/12 16:58:07 INFO Received push request method=POST path=/51082026/11/12 16:58:07 INFO Aborted multipart uploads count=0 kept=051092026/11/12 16:58:07 WARN Force mode enabled - objects will be deleted immediately without grace period51102026/11/12 16:58:07 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=051112026/11/12 16:58:07 INFO Received complete upload request method=POST path=/51122026/11/12 16:58:07 INFO Vacuumed table table=pending_closures51132026/11/12 16:58:07 INFO Vacuumed table table=commit_receipts51142026/11/12 16:58:07 INFO Completed upload id=34 objects=23 duration=331.666µs51152026/11/12 16:58:07 INFO Vacuumed table table=pending_objects51162026/11/12 16:58:07 INFO Vacuumed table table=multipart_uploads51172026/11/12 16:58:07 INFO Vacuumed table table=closures51182026/11/12 16:58:07 INFO Vacuumed table table=objects51192026/11/12 16:58:07 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=051202026/11/12 16:58:07 INFO Vacuumed table table=pending_closures51212026/11/12 16:58:07 INFO Vacuumed table table=commit_receipts51222026/11/12 16:58:07 INFO Vacuumed table table=pending_objects51232026/11/12 16:58:07 INFO Vacuumed table table=multipart_uploads51242026/11/12 16:58:07 INFO Vacuumed table table=closures51252026/11/12 16:58:07 INFO Vacuumed table table=objects51262026/11/12 16:58:07 INFO Received push request method=POST path=/51272026/11/12 16:58:07 INFO Received complete push request method=POST path=/51282026/11/12 16:58:07 INFO Completed push id=35 objects=6 duration=678.041µs51292026/11/12 16:58:07 INFO Aborted multipart uploads count=0 kept=051302026/11/12 16:58:07 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=051312026/11/12 16:58:07 INFO Vacuumed table table=pending_closures51322026/11/12 16:58:07 INFO Vacuumed table table=commit_receipts51332026/11/12 16:58:07 INFO Vacuumed table table=pending_objects51342026/11/12 16:58:07 INFO Vacuumed table table=multipart_uploads51352026/11/12 16:58:07 INFO Vacuumed table table=closures51362026/11/12 16:58:07 INFO Vacuumed table table=objects51372026/11/12 16:58:07 INFO Received uploads request method=POST path=/51382026/11/12 16:58:07 INFO Aborted multipart uploads count=0 kept=051392026/11/12 16:58:07 WARN Force mode enabled - objects will be deleted immediately without grace period51402026/11/12 16:58:07 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=051412026/11/12 16:58:07 INFO Vacuumed table table=pending_closures51422026/11/12 16:58:07 INFO Vacuumed table table=commit_receipts51432026/11/12 16:58:07 INFO Vacuumed table table=pending_objects51442026/11/12 16:58:07 INFO Vacuumed table table=multipart_uploads51452026/11/12 16:58:07 INFO Vacuumed table table=closures51462026/11/12 16:58:07 INFO Vacuumed table table=objects51472026/11/12 16:58:07 INFO Aborted multipart uploads count=0 kept=051482026/11/12 16:58:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=051492026/11/12 16:58:07 INFO Vacuumed table table=pending_closures51502026/11/12 16:58:07 INFO Vacuumed table table=commit_receipts51512026/11/12 16:58:07 INFO Vacuumed table table=pending_objects51522026/11/12 16:58:07 INFO Vacuumed table table=multipart_uploads51532026/11/12 16:58:07 INFO Vacuumed table table=closures51542026/11/12 16:58:07 INFO Vacuumed table table=objects51552026/11/12 16:58:07 INFO Received uploads request method=POST path=/51562026/11/12 16:58:07 INFO Received complete upload request method=POST path=/51572026/11/12 16:58:07 INFO Completed upload id=26 objects=2 duration=508.583µs51582026/11/12 16:58:07 INFO Aborted multipart uploads count=0 kept=051592026/11/12 16:58:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=051602026/11/12 16:58:07 INFO Vacuumed table table=pending_closures51612026/11/12 16:58:07 INFO Vacuumed table table=commit_receipts51622026/11/12 16:58:07 INFO Vacuumed table table=pending_objects51632026/11/12 16:58:07 INFO Vacuumed table table=multipart_uploads51642026/11/12 16:58:07 INFO Vacuumed table table=closures51652026/11/12 16:58:07 INFO Vacuumed table table=objects51662026/11/12 16:58:07 INFO Received push request method=POST path=/51672026/11/12 16:58:07 INFO Aborted multipart uploads count=0 kept=051682026/11/12 16:58:07 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."51692026/11/12 16:58:07 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."51702026/11/12 16:58:07 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."51712026/11/12 16:58:07 INFO Received push request method=POST path=/51722026/11/12 16:58:07 INFO Received push request method=POST path=/51732026/11/12 16:58:07 INFO Received complete push request method=POST path=/51742026/11/12 16:58:07 INFO Completed push id=28 objects=11 duration=618.417µs51752026/11/12 16:58:07 INFO Received complete push request method=POST path=/51762026/11/12 16:58:07 INFO Completed push id=29 objects=24 duration=671.083µs51772026/11/12 16:58:07 INFO Aborted multipart uploads count=0 kept=051782026/11/12 16:58:07 WARN Force mode enabled - objects will be deleted immediately without grace period51792026/11/12 16:58:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=051802026/11/12 16:58:07 INFO Vacuumed table table=pending_closures51812026/11/12 16:58:07 INFO Vacuumed table table=commit_receipts51822026/11/12 16:58:07 INFO Vacuumed table table=pending_objects51832026/11/12 16:58:07 INFO Vacuumed table table=multipart_uploads51842026/11/12 16:58:07 INFO Vacuumed table table=closures51852026/11/12 16:58:07 INFO Vacuumed table table=objects51862026/11/12 16:58:07 INFO Received complete upload request method=POST path=/51872026/11/12 16:58:07 INFO Completed upload id=13 objects=14 duration=1.287792ms51882026/11/12 16:58:07 INFO Aborted multipart uploads count=0 kept=051892026/11/12 16:58:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=051902026/11/12 16:58:07 INFO Vacuumed table table=pending_closures51912026/11/12 16:58:07 INFO Vacuumed table table=commit_receipts51922026/11/12 16:58:07 INFO Vacuumed table table=pending_objects51932026/11/12 16:58:07 INFO Vacuumed table table=multipart_uploads51942026/11/12 16:58:07 INFO Vacuumed table table=closures51952026/11/12 16:58:07 INFO Vacuumed table table=objects51962026/11/12 16:58:07 INFO Received complete push request method=POST path=/51972026/11/12 16:58:07 INFO Completed push id=31 objects=6 duration=1.199333ms51982026/11/12 16:58:07 INFO Received uploads request method=POST path=/51992026/11/12 16:58:07 INFO Received complete push request method=POST path=/52002026/11/12 16:58:07 INFO Completed push id=1 objects=9 duration=1.554292ms52012026/11/12 16:58:08 INFO Received complete upload request method=POST path=/52022026/11/12 16:58:08 INFO Completed upload id=2 objects=7 duration=1.407625ms52032026/11/12 16:58:08 INFO Received complete upload request method=POST path=/52042026/11/12 16:58:08 INFO Completed upload id=8 objects=6 duration=1.33125ms52052026/11/12 16:58:08 INFO Received complete upload request method=POST path=/52062026/11/12 16:58:08 INFO Completed upload id=12 objects=9 duration=1.403459ms52072026/11/12 16:58:08 INFO Received complete upload request method=POST path=/52082026/11/12 16:58:08 INFO Completed upload id=18 objects=22 duration=1.430875ms52092026/11/12 16:58:08 INFO Received complete upload request method=POST path=/52102026/11/12 16:58:08 INFO Completed upload id=21 objects=8 duration=656.667µs52112026/11/12 16:58:08 INFO Received complete upload request method=POST path=/52122026/11/12 16:58:08 INFO Completed upload id=22 objects=7 duration=497.833µs52132026/11/12 16:58:08 INFO Received complete push request method=POST path=/52142026/11/12 16:58:08 INFO Completed push id=24 objects=5 duration=409.834µs52152026/11/12 16:58:08 INFO Received complete upload request method=POST path=/52162026/11/12 16:58:08 INFO Completed upload id=25 objects=8 duration=300.791µs52172026/11/12 16:58:08 INFO Received complete push request method=POST path=/52182026/11/12 16:58:08 INFO Completed push id=26 objects=4 duration=326.708µs52192026/11/12 16:58:08 INFO Received complete push request method=POST path=/52202026/11/12 16:58:08 INFO Completed push id=30 objects=2 duration=233.041µs52212026/11/12 16:58:08 INFO Received complete push request method=POST path=/52222026/11/12 16:58:08 INFO Completed push id=32 objects=7 duration=273.833µs52232026/11/12 16:58:08 INFO Received complete upload request method=POST path=/52242026/11/12 16:58:08 INFO Completed upload id=33 objects=6 duration=189.333µs52252026/11/12 16:58:08 INFO Received complete upload request method=POST path=/52262026/11/12 16:58:08 INFO Completed upload id=34 objects=3 duration=155.208µs52272026/11/12 16:58:08 INFO Aborted multipart uploads count=0 kept=052282026/11/12 16:58:08 WARN Force mode enabled - objects will be deleted immediately without grace period52292026/11/12 16:58:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=052302026/11/12 16:58:08 INFO Vacuumed table table=pending_closures52312026/11/12 16:58:08 INFO Vacuumed table table=commit_receipts52322026/11/12 16:58:08 INFO Vacuumed table table=pending_objects52332026/11/12 16:58:08 INFO Vacuumed table table=multipart_uploads52342026/11/12 16:58:08 INFO Vacuumed table table=closures52352026/11/12 16:58:08 INFO Vacuumed table table=objects5236--- PASS: FuzzPushGC (0.01s)5237 --- PASS: FuzzPushGC/seed#3 (7.56s)5238 --- PASS: FuzzPushGC/seed#0 (8.33s)5239 --- PASS: FuzzPushGC/seed#5 (11.48s)5240 --- PASS: FuzzPushGC/seed#9 (11.53s)5241 --- PASS: FuzzPushGC/seed#11 (11.87s)5242 --- PASS: FuzzPushGC/seed#8 (13.18s)5243 --- PASS: FuzzPushGC/seed#6 (13.42s)5244 --- PASS: FuzzPushGC/seed#15 (13.56s)5245 --- PASS: FuzzPushGC/seed#1 (13.74s)5246 --- PASS: FuzzPushGC/seed#14 (14.42s)5247 --- PASS: FuzzPushGC/seed#7 (14.89s)5248 --- PASS: FuzzPushGC/seed#13 (15.37s)5249 --- PASS: FuzzPushGC/seed#4 (15.79s)5250 --- PASS: FuzzPushGC/seed#12 (16.12s)5251 --- PASS: FuzzPushGC/seed#2 (16.37s)5252 --- PASS: FuzzPushGC/seed#10 (17.74s)5253PASS5254{"timestamp":"2026-11-12T16:58:08.980584Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:64608","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(5)"}5255{"timestamp":"2026-11-12T16:58:08.980933Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:64607","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(11)"}52562026-11-12 16:58:09.265 UTC [8743] LOG: received smart shutdown request52572026-11-12 16:58:09.266 UTC [8743] LOG: background worker "logical replication launcher" (PID 8754) exited with exit code 152582026-11-12 16:58:10.384 UTC [8748] LOG: checkpoint complete: wrote 3201 buffers (19.5%), wrote 5 SLRU buffers; 0 WAL file(s) added, 0 removed, 33 recycled; write=34.423 s, sync=0.844 s, total=35.532 s; sync files=43137, longest=0.001 s, average=0.001 s; distance=533007 kB, estimate=533007 kB; lsn=0/28095128, redo lsn=0/22010C1852592026-11-12 16:58:10.386 UTC [8748] LOG: shutting down52602026-11-12 16:58:10.386 UTC [8748] LOG: checkpoint starting: shutdown immediate52612026-11-12 16:58:10.983 UTC [8748] LOG: checkpoint complete: wrote 9238 buffers (56.4%), wrote 2 SLRU buffers; 0 WAL file(s) added, 0 removed, 6 recycled; write=0.535 s, sync=0.053 s, total=0.598 s; sync files=3086, longest=0.003 s, average=0.001 s; distance=98833 kB, estimate=489589 kB; lsn=0/280951A0, redo lsn=0/280951A052622026-11-12 16:58:10.993 UTC [8743] LOG: database system is shut down5263Running OIDC tests...5264=== RUN TestAudienceForIssuer5265=== PAUSE TestAudienceForIssuer5266=== RUN TestProviderClientGivesUpOnSilentIssuer5267=== PAUSE TestProviderClientGivesUpOnSilentIssuer5268=== RUN TestGlobMatch5269=== PAUSE TestGlobMatch5270=== RUN TestValidateToken_ValidToken5271=== PAUSE TestValidateToken_ValidToken5272=== RUN TestValidateToken_WrongAudience5273=== PAUSE TestValidateToken_WrongAudience5274=== RUN TestValidateToken_Expired5275=== PAUSE TestValidateToken_Expired5276=== RUN TestValidateToken_BoundClaimsMismatch5277=== PAUSE TestValidateToken_BoundClaimsMismatch5278=== RUN TestValidateToken_BoundSubjectMismatch5279=== PAUSE TestValidateToken_BoundSubjectMismatch5280=== RUN TestValidateToken_MultipleProviders5281=== PAUSE TestValidateToken_MultipleProviders5282=== RUN TestValidateToken_UnknownIssuerFetchesNoKeys5283=== PAUSE TestValidateToken_UnknownIssuerFetchesNoKeys5284=== RUN TestValidateToken_GoogleSchemelessIssuer5285=== PAUSE TestValidateToken_GoogleSchemelessIssuer5286=== RUN TestValidateToken_KubernetesServiceAccount5287=== PAUSE TestValidateToken_KubernetesServiceAccount5288=== RUN TestNewValidator_KubernetesRequiresCA5289=== PAUSE TestNewValidator_KubernetesRequiresCA5290=== RUN TestValidateToken_KubernetesIssuerFromOwnToken5291=== PAUSE TestValidateToken_KubernetesIssuerFromOwnToken5292=== RUN TestPins_ReservedForMatchingRule5293=== PAUSE TestPins_ReservedForMatchingRule5294=== RUN TestPins_TopLevelShorthand5295=== PAUSE TestPins_TopLevelShorthand5296=== RUN TestPins_ConfigValidation5297=== PAUSE TestPins_ConfigValidation5298=== RUN TestScopes_LegacyProviderDefaultsToWrite5299=== PAUSE TestScopes_LegacyProviderDefaultsToWrite5300=== RUN TestScopes_Rules5301=== PAUSE TestScopes_Rules5302=== RUN TestScopes_ConfigValidation5303=== PAUSE TestScopes_ConfigValidation5304=== CONT TestAudienceForIssuer5305--- PASS: TestAudienceForIssuer (0.00s)5306=== CONT TestValidateToken_WrongAudience5307=== CONT TestValidateToken_Expired5308=== CONT TestProviderClientGivesUpOnSilentIssuer5309=== CONT TestPins_TopLevelShorthand5310=== CONT TestScopes_LegacyProviderDefaultsToWrite5311=== CONT TestValidateToken_MultipleProviders5312=== CONT TestPins_ConfigValidation5313=== CONT TestValidateToken_BoundSubjectMismatch5314=== CONT TestValidateToken_BoundClaimsMismatch5315--- PASS: TestProviderClientGivesUpOnSilentIssuer (0.00s)5316=== CONT TestValidateToken_UnknownIssuerFetchesNoKeys53172026/11/12 16:58:13 INFO OIDC provider initialized name=known issuer=https://known.example5318--- PASS: TestPins_ConfigValidation (0.01s)5319=== CONT TestValidateToken_GoogleSchemelessIssuer5320=== CONT TestPins_ReservedForMatchingRule5321=== CONT TestValidateToken_KubernetesIssuerFromOwnToken5322=== CONT TestNewValidator_KubernetesRequiresCA5323=== CONT TestValidateToken_KubernetesServiceAccount5324=== CONT TestGlobMatch5325=== RUN TestGlobMatch/foo_foo5326=== PAUSE TestGlobMatch/foo_foo5327=== RUN TestGlobMatch/foo_bar5328=== PAUSE TestGlobMatch/foo_bar5329=== RUN TestGlobMatch/*_5330=== PAUSE TestGlobMatch/*_5331=== RUN TestGlobMatch/*_anything5332=== PAUSE TestGlobMatch/*_anything5333=== RUN TestGlobMatch/foo*_foo5334=== PAUSE TestGlobMatch/foo*_foo5335=== RUN TestGlobMatch/foo*_foobar5336=== PAUSE TestGlobMatch/foo*_foobar5337=== RUN TestGlobMatch/foo*_bar5338=== PAUSE TestGlobMatch/foo*_bar5339=== RUN TestGlobMatch/*bar_bar5340=== PAUSE TestGlobMatch/*bar_bar5341=== RUN TestGlobMatch/*bar_foobar5342=== PAUSE TestGlobMatch/*bar_foobar5343=== RUN TestGlobMatch/*bar_foo5344=== PAUSE TestGlobMatch/*bar_foo5345=== RUN TestGlobMatch/foo*bar_foobar5346=== PAUSE TestGlobMatch/foo*bar_foobar5347=== RUN TestGlobMatch/foo*bar_foo123bar5348=== PAUSE TestGlobMatch/foo*bar_foo123bar5349=== RUN TestGlobMatch/foo*bar_foobarbaz5350=== PAUSE TestGlobMatch/foo*bar_foobarbaz5351=== RUN TestGlobMatch/*/*_foo/bar5352=== PAUSE TestGlobMatch/*/*_foo/bar5353=== RUN TestGlobMatch/*/*_foo5354=== PAUSE TestGlobMatch/*/*_foo5355=== RUN TestGlobMatch/refs/heads/*_refs/heads/main5356=== PAUSE TestGlobMatch/refs/heads/*_refs/heads/main5357=== RUN TestGlobMatch/refs/heads/*_refs/tags/v1.05358=== PAUSE TestGlobMatch/refs/heads/*_refs/tags/v1.05359=== RUN TestGlobMatch/refs/*/main_refs/heads/main5360=== PAUSE TestGlobMatch/refs/*/main_refs/heads/main5361=== RUN TestGlobMatch/fo?_foo5362=== PAUSE TestGlobMatch/fo?_foo5363=== RUN TestGlobMatch/fo?_fo5364=== PAUSE TestGlobMatch/fo?_fo5365=== RUN TestGlobMatch/fo?_fooo5366=== PAUSE TestGlobMatch/fo?_fooo5367=== RUN TestGlobMatch/?oo_foo5368=== PAUSE TestGlobMatch/?oo_foo5369=== RUN TestGlobMatch/?oo_boo5370=== PAUSE TestGlobMatch/?oo_boo5371=== RUN TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5372=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5373=== RUN TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5374=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5375=== CONT TestGlobMatch/foo_foo5376=== CONT TestValidateToken_ValidToken5377=== CONT TestGlobMatch/*/*_foo/bar5378=== CONT TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5379=== CONT TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5380=== CONT TestGlobMatch/?oo_boo5381=== CONT TestGlobMatch/?oo_foo5382=== CONT TestGlobMatch/fo?_fooo5383=== CONT TestGlobMatch/fo?_fo5384=== CONT TestGlobMatch/fo?_foo5385=== CONT TestGlobMatch/refs/*/main_refs/heads/main5386=== CONT TestGlobMatch/refs/heads/*_refs/tags/v1.05387=== CONT TestGlobMatch/refs/heads/*_refs/heads/main5388=== CONT TestGlobMatch/*/*_foo5389=== CONT TestGlobMatch/foo*bar_foobarbaz5390=== CONT TestGlobMatch/foo*bar_foo123bar5391=== CONT TestGlobMatch/foo*_bar5392=== CONT TestGlobMatch/*_anything5393=== CONT TestGlobMatch/*_5394=== CONT TestGlobMatch/foo_bar5395=== CONT TestScopes_Rules5396=== CONT TestGlobMatch/foo*_foobar5397=== CONT TestGlobMatch/*bar_foo5398=== CONT TestGlobMatch/foo*_foo5399=== CONT TestGlobMatch/foo*bar_foobar5400=== CONT TestGlobMatch/*bar_bar5401=== CONT TestScopes_ConfigValidation5402=== CONT TestGlobMatch/*bar_foobar5403--- PASS: TestGlobMatch (0.00s)5404 --- PASS: TestGlobMatch/foo_foo (0.00s)5405 --- PASS: TestGlobMatch/*/*_foo/bar (0.00s)5406 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main (0.00s)5407 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main (0.00s)5408 --- PASS: TestGlobMatch/?oo_boo (0.00s)5409 --- PASS: TestGlobMatch/?oo_foo (0.00s)5410 --- PASS: TestGlobMatch/fo?_fooo (0.00s)5411 --- PASS: TestGlobMatch/fo?_fo (0.00s)5412 --- PASS: TestGlobMatch/fo?_foo (0.00s)5413 --- PASS: TestGlobMatch/refs/*/main_refs/heads/main (0.00s)5414 --- PASS: TestGlobMatch/refs/heads/*_refs/tags/v1.0 (0.00s)5415 --- PASS: TestGlobMatch/refs/heads/*_refs/heads/main (0.00s)5416 --- PASS: TestGlobMatch/*/*_foo (0.00s)5417 --- PASS: TestGlobMatch/foo*bar_foobarbaz (0.00s)5418 --- PASS: TestGlobMatch/foo*bar_foo123bar (0.00s)5419 --- PASS: TestGlobMatch/*_anything (0.00s)5420 --- PASS: TestGlobMatch/*_ (0.00s)5421 --- PASS: TestGlobMatch/foo*_foobar (0.00s)5422 --- PASS: TestGlobMatch/foo_bar (0.00s)5423 --- PASS: TestGlobMatch/*bar_foo (0.00s)5424 --- PASS: TestGlobMatch/foo*_foo (0.00s)5425 --- PASS: TestGlobMatch/foo*bar_foobar (0.00s)5426 --- PASS: TestGlobMatch/*bar_bar (0.00s)5427 --- PASS: TestGlobMatch/*bar_foobar (0.00s)5428 --- PASS: TestGlobMatch/foo*_bar (0.00s)5429--- PASS: TestScopes_ConfigValidation (0.00s)54302026/11/12 16:58:13 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:49215/oidc54312026/11/12 16:58:13 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:49213/oidc5432--- PASS: TestPins_TopLevelShorthand (0.05s)5433--- PASS: TestScopes_LegacyProviderDefaultsToWrite (0.06s)54342026/11/12 16:58:13 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:49218/oidc54352026/11/12 16:58:13 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:49220/oidc5436--- PASS: TestValidateToken_BoundClaimsMismatch (0.07s)5437--- PASS: TestPins_ReservedForMatchingRule (0.07s)54382026/11/12 16:58:13 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:49222/oidc5439--- PASS: TestValidateToken_ValidToken (0.06s)54402026/11/12 16:58:13 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:49224/oidc54412026/11/12 16:58:13 INFO OIDC provider initialized name=kubernetes issuer=https://oidc.eks.invalid/id/ABC1235442--- PASS: TestValidateToken_Expired (0.09s)5443--- PASS: TestValidateToken_KubernetesIssuerFromOwnToken (0.08s)5444--- PASS: TestValidateToken_UnknownIssuerFetchesNoKeys (0.10s)54452026/11/12 16:58:13 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:49229/oidc5446--- PASS: TestValidateToken_BoundSubjectMismatch (0.10s)54472026/11/12 16:58:13 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:49233/oidc54482026/11/12 16:58:13 INFO OIDC provider initialized name=kubernetes issuer=https://127.0.0.1:492315449--- PASS: TestValidateToken_WrongAudience (0.13s)54502026/11/12 16:58:13 INFO OIDC provider initialized name=google issuer=https://accounts.google.com5451--- PASS: TestValidateToken_GoogleSchemelessIssuer (0.12s)5452--- PASS: TestValidateToken_KubernetesServiceAccount (0.11s)54532026/11/12 16:58:13 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:49237/oidc5454--- PASS: TestScopes_Rules (0.12s)54552026/11/12 16:58:13 INFO OIDC provider initialized name=provider1 issuer=http://127.0.0.1:49217/oidc54562026/11/12 16:58:13 INFO OIDC provider initialized name=provider2 issuer=http://127.0.0.1:49241/oidc5457--- PASS: TestValidateToken_MultipleProviders (0.16s)54582026/11/12 16:58:13 http: TLS handshake error from 127.0.0.1:49240: read tcp 127.0.0.1:49239->127.0.0.1:49240: use of closed network connection5459--- PASS: TestNewValidator_KubernetesRequiresCA (0.15s)5460PASS5461Running hook tests...5462=== RUN TestSendPathsEmpty5463=== PAUSE TestSendPathsEmpty5464=== RUN TestQueueEnqueueAndFetch5465=== PAUSE TestQueueEnqueueAndFetch5466=== RUN TestQueueDeduplication5467=== PAUSE TestQueueDeduplication5468=== RUN TestQueueRemove5469=== PAUSE TestQueueRemove5470=== RUN TestQueueFetchBatchLimit5471=== PAUSE TestQueueFetchBatchLimit5472=== RUN TestQueueRetryMovesToBack5473=== PAUSE TestQueueRetryMovesToBack5474=== RUN TestQueueFetchRemoveLifecycle5475=== PAUSE TestQueueFetchRemoveLifecycle5476=== RUN TestQueueConcurrentWriters5477=== PAUSE TestQueueConcurrentWriters5478=== RUN TestQueueRemoveLargeClosure5479=== PAUSE TestQueueRemoveLargeClosure5480=== RUN TestServerClientIntegration5481--- PASS: TestServerClientIntegration (0.00s)5482=== RUN TestServerQueueError54832026/11/12 16:58:14 ERROR Failed to queue paths error="permission denied" count=15484--- PASS: TestServerQueueError (0.00s)5485=== RUN TestGetListenerSocketActivation5486 server_test.go:147: systemd socket activation does not exist on darwin5487--- SKIP: TestGetListenerSocketActivation (0.00s)5488=== RUN TestServerStalledClientDoesNotBlockShutdown5489=== PAUSE TestServerStalledClientDoesNotBlockShutdown5490=== RUN TestServerBacksOffOnAcceptErrors5491=== PAUSE TestServerBacksOffOnAcceptErrors5492=== RUN TestDrainIsolatesPoisonPath5493=== PAUSE TestDrainIsolatesPoisonPath5494=== RUN TestRunNotBlockedByPoisonHead5495=== PAUSE TestRunNotBlockedByPoisonHead5496=== RUN TestDrainGivesUpWhenServerDown5497=== PAUSE TestDrainGivesUpWhenServerDown5498=== RUN TestFailedPathPrunedByLaterClosure5499=== PAUSE TestFailedPathPrunedByLaterClosure5500=== RUN TestWorkerUploadsAndRemoves5501=== PAUSE TestWorkerUploadsAndRemoves5502=== RUN TestWorkerSkipsGCdPaths5503=== PAUSE TestWorkerSkipsGCdPaths5504=== RUN TestWorkerPrunesClosureDeps5505=== PAUSE TestWorkerPrunesClosureDeps5506=== RUN TestDrainTimeout5507=== PAUSE TestDrainTimeout5508=== RUN TestWorkerRemoveFailureIsNotProgress5509=== PAUSE TestWorkerRemoveFailureIsNotProgress5510=== RUN TestWorkerKeepsPathItCannotStat5511=== PAUSE TestWorkerKeepsPathItCannotStat5512=== CONT TestSendPathsEmpty5513=== CONT TestRunNotBlockedByPoisonHead5514=== CONT TestDrainIsolatesPoisonPath5515=== CONT TestWorkerKeepsPathItCannotStat5516=== CONT TestWorkerRemoveFailureIsNotProgress5517=== CONT TestDrainTimeout5518=== CONT TestWorkerPrunesClosureDeps5519=== CONT TestWorkerSkipsGCdPaths5520=== CONT TestWorkerUploadsAndRemoves5521=== CONT TestFailedPathPrunedByLaterClosure5522=== CONT TestDrainGivesUpWhenServerDown5523--- PASS: TestSendPathsEmpty (0.00s)5524=== CONT TestQueueRetryMovesToBack5525=== CONT TestQueueFetchBatchLimit5526=== CONT TestQueueRemove5527=== CONT TestQueueDeduplication5528=== CONT TestQueueFetchRemoveLifecycle5529=== CONT TestQueueEnqueueAndFetch5530=== CONT TestServerBacksOffOnAcceptErrors5531=== CONT TestServerStalledClientDoesNotBlockShutdown55322000/01/01 00:00:00 ERROR Accept failed error="too many open files"55332000/01/01 00:00:00 ERROR Accept failed error="too many open files"5534=== CONT TestQueueRemoveLargeClosure5535=== CONT TestQueueConcurrentWriters55362000/01/01 00:00:00 ERROR Accept failed error="too many open files"55372000/01/01 00:00:00 ERROR Accept failed error="too many open files"55382000/01/01 00:00:00 ERROR Accept failed error="too many open files"55392000/01/01 00:00:00 ERROR Accept failed error="too many open files"55402000/01/01 00:00:00 ERROR Accept failed error="too many open files"55412000/01/01 00:00:00 ERROR Accept failed error="too many open files"5542--- PASS: TestServerBacksOffOnAcceptErrors (0.00s)55432026/11/12 16:58:14 INFO Upload queue status pending=255442026/11/12 16:58:14 WARN Store path no longer exists (garbage collected?), removing from queue path=/private/var/tb/2/scratch/777606c6228c2b4e/build/TestWorkerSkipsGCdPaths198785515/002/nonexistent55452026/11/12 16:58:14 INFO Upload queue status pending=255462026/11/12 16:58:14 INFO Uploading batch count=25547--- PASS: TestQueueFetchBatchLimit (0.01s)55482026/11/12 16:58:14 INFO Upload queue status pending=255492026/11/12 16:58:14 INFO Uploading batch count=155502026/11/12 16:58:14 INFO Uploading batch count=155512026/11/12 16:58:14 INFO Uploading batch count=255522026/11/12 16:58:14 ERROR Upload failed error="upload failed" count=255532026/11/12 16:58:14 ERROR Upload failed, will retry later error="upload failed" path=/private/var/tb/2/scratch/777606c6228c2b4e/build/TestDrainGivesUpWhenServerDown3660773595/002/a55542026/11/12 16:58:14 INFO Uploading batch count=455552026/11/12 16:58:14 ERROR Upload failed error="upload failed" count=455562026/11/12 16:58:14 ERROR Upload failed, will retry later error="upload failed" path=/private/var/tb/2/scratch/777606c6228c2b4e/build/TestDrainGivesUpWhenServerDown3660773595/002/b55572026/11/12 16:58:14 INFO Upload queue status pending=355582000/01/01 00:00:00 INFO Uploading batch count=255592026/11/12 16:58:14 INFO Uploading batch count=155602026/11/12 16:58:14 ERROR Upload failed error="upload failed" count=155612026/11/12 16:58:14 WARN Cannot stat store path, will retry later path=/private/var/tb/2/scratch/777606c6228c2b4e/build/TestWorkerKeepsPathItCannotStat2612280587/002/locked/aaa error="stat /private/var/tb/2/scratch/777606c6228c2b4e/build/TestWorkerKeepsPathItCannotStat2612280587/002/locked/aaa: permission denied"55622026/11/12 16:58:14 INFO Uploading batch count=155632026/11/12 16:58:14 ERROR Upload failed error="upload failed" count=155642000/01/01 00:00:00 ERROR Upload failed error="context deadline exceeded" count=255652026/11/12 16:58:14 ERROR Upload failed, will retry later error="upload failed" path=/private/var/tb/2/scratch/777606c6228c2b4e/build/TestDrainIsolatesPoisonPath3656740643/002/bbb55662000/01/01 00:00:00 ERROR Drain finished with paths left in queue remaining=45567--- PASS: TestQueueRetryMovesToBack (0.01s)55682026/11/12 16:58:14 INFO Uploading batch count=255692026/11/12 16:58:14 ERROR Upload failed error="upload failed" count=255702026/11/12 16:58:14 ERROR Upload failed, will retry later error="upload failed" path=/private/var/tb/2/scratch/777606c6228c2b4e/build/TestDrainGivesUpWhenServerDown3660773595/002/c5571--- PASS: TestQueueEnqueueAndFetch (0.01s)5572--- PASS: TestQueueFetchRemoveLifecycle (0.01s)5573--- PASS: TestQueueDeduplication (0.01s)55742026/11/12 16:58:14 INFO Uploading batch count=155752026/11/12 16:58:14 ERROR Upload failed, will retry later error="upload failed" path=/private/var/tb/2/scratch/777606c6228c2b4e/build/TestDrainGivesUpWhenServerDown3660773595/002/d55762026/11/12 16:58:14 WARN Store path no longer exists (garbage collected?), removing from queue path=/private/var/tb/2/scratch/777606c6228c2b4e/build/TestWorkerRemoveFailureIsNotProgress3504799644/002/nonexistent55772026/11/12 16:58:14 INFO Uploading batch count=155782026/11/12 16:58:14 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=155792026/11/12 16:58:14 WARN Store path no longer exists (garbage collected?), removing from queue path=/private/var/tb/2/scratch/777606c6228c2b4e/build/TestWorkerRemoveFailureIsNotProgress3504799644/002/nonexistent55802026/11/12 16:58:14 WARN Cannot stat store path, will retry later path=/private/var/tb/2/scratch/777606c6228c2b4e/build/TestWorkerKeepsPathItCannotStat2612280587/002/locked/aaa error="stat /private/var/tb/2/scratch/777606c6228c2b4e/build/TestWorkerKeepsPathItCannotStat2612280587/002/locked/aaa: permission denied"55812026/11/12 16:58:14 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=155822026/11/12 16:58:14 INFO Uploading batch count=255832026/11/12 16:58:14 WARN Store path no longer exists (garbage collected?), removing from queue path=/private/var/tb/2/scratch/777606c6228c2b4e/build/TestWorkerRemoveFailureIsNotProgress3504799644/002/nonexistent55842026/11/12 16:58:14 ERROR Upload failed error="upload failed" count=255852026/11/12 16:58:14 ERROR Upload failed, will retry later error="upload failed" path=/private/var/tb/2/scratch/777606c6228c2b4e/build/TestDrainGivesUpWhenServerDown3660773595/002/e5586--- PASS: TestQueueRemove (0.02s)55872026/11/12 16:58:14 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=155882026/11/12 16:58:14 INFO Uploading batch count=155892026/11/12 16:58:14 ERROR Upload failed error="upload failed" count=155902026/11/12 16:58:14 ERROR Drain finished with paths left in queue remaining=155912026/11/12 16:58:14 WARN Cannot stat store path, will retry later path=/private/var/tb/2/scratch/777606c6228c2b4e/build/TestWorkerKeepsPathItCannotStat2612280587/002/locked/aaa error="stat /private/var/tb/2/scratch/777606c6228c2b4e/build/TestWorkerKeepsPathItCannotStat2612280587/002/locked/aaa: permission denied"55922026/11/12 16:58:14 ERROR Upload failed, will retry later error="upload failed" path=/private/var/tb/2/scratch/777606c6228c2b4e/build/TestDrainGivesUpWhenServerDown3660773595/002/f55932026/11/12 16:58:14 INFO Uploading batch count=155942026/11/12 16:58:14 ERROR Upload failed error="upload failed" count=155952026/11/12 16:58:14 ERROR Drain finished with paths left in queue remaining=155962026/11/12 16:58:14 ERROR Drain finished with paths left in queue remaining=1055972026/11/12 16:58:14 INFO Uploading batch count=155982026/11/12 16:58:14 ERROR Upload failed error="upload failed" count=15599--- PASS: TestDrainTimeout (0.02s)5600--- PASS: TestWorkerRemoveFailureIsNotProgress (0.02s)56012026/11/12 16:58:14 ERROR Drain finished with paths left in queue remaining=15602--- PASS: TestFailedPathPrunedByLaterClosure (0.02s)5603--- PASS: TestWorkerKeepsPathItCannotStat (0.02s)5604--- PASS: TestDrainGivesUpWhenServerDown (0.02s)5605--- PASS: TestDrainIsolatesPoisonPath (0.02s)5606--- PASS: TestWorkerSkipsGCdPaths (0.03s)5607--- PASS: TestWorkerPrunesClosureDeps (0.03s)5608--- PASS: TestWorkerUploadsAndRemoves (0.03s)5609--- PASS: TestQueueRemoveLargeClosure (0.05s)5610--- PASS: TestQueueConcurrentWriters (0.05s)56112026/11/12 16:58:14 ERROR Failed to decode request error="read unix /private/var/tb/2/scratch/777606c6228c2b4e/build/hook3463791108/test.sock->: i/o timeout"56122026/11/12 16:58:14 ERROR Failed to write response error="write unix /private/var/tb/2/scratch/777606c6228c2b4e/build/hook3463791108/test.sock->: i/o timeout"5613--- PASS: TestServerStalledClientDoesNotBlockShutdown (0.20s)56142026/11/12 16:58:15 INFO Uploading batch count=156152026/11/12 16:58:15 INFO Uploading batch count=156162026/11/12 16:58:15 INFO Uploading batch count=156172026/11/12 16:58:15 ERROR Upload failed error="upload failed" count=156182026/11/12 16:58:15 INFO Uploading batch count=156192026/11/12 16:58:15 ERROR Upload failed error="upload failed" count=156202026/11/12 16:58:15 INFO Uploading batch count=156212026/11/12 16:58:15 ERROR Upload failed error="upload failed" count=156222026/11/12 16:58:15 INFO Uploading batch count=156232026/11/12 16:58:15 ERROR Upload failed error="upload failed" count=156242026/11/12 16:58:15 ERROR Drain finished with paths left in queue remaining=15625--- PASS: TestRunNotBlockedByPoisonHead (1.05s)5626PASS5627Running signing tests...5628=== RUN TestExternalSigner5629=== PAUSE TestExternalSigner5630=== RUN TestExternalSignerWrongPublicKey5631=== PAUSE TestExternalSignerWrongPublicKey5632=== RUN TestExternalSignerErrors5633=== PAUSE TestExternalSignerErrors5634=== RUN TestExternalSignerRestartsAfterCrash5635=== PAUSE TestExternalSignerRestartsAfterCrash5636=== RUN TestExternalSignerProgramErrorKeepsProcess5637=== PAUSE TestExternalSignerProgramErrorKeepsProcess5638=== RUN TestExternalSignerClose5639=== PAUSE TestExternalSignerClose5640=== RUN TestExternalSignerRejectsInvalidPublicKey5641=== PAUSE TestExternalSignerRejectsInvalidPublicKey5642=== RUN TestExternalSignerCancellation5643=== PAUSE TestExternalSignerCancellation5644=== RUN TestGenerateFingerprint5645=== PAUSE TestGenerateFingerprint5646=== RUN TestParseSigningKey5647=== PAUSE TestParseSigningKey5648=== RUN TestSign5649=== PAUSE TestSign5650=== RUN TestSignBatch5651=== PAUSE TestSignBatch5652=== RUN TestNilKeyPublicKey5653=== PAUSE TestNilKeyPublicKey5654=== RUN TestSignErrors5655=== PAUSE TestSignErrors5656=== CONT TestExternalSigner5657=== CONT TestExternalSignerCancellation5658=== CONT TestSignBatch5659=== CONT TestSignErrors5660=== CONT TestExternalSignerProgramErrorKeepsProcess5661=== CONT TestNilKeyPublicKey5662--- PASS: TestNilKeyPublicKey (0.00s)5663=== CONT TestExternalSignerErrors5664=== RUN TestExternalSignerErrors/exit_at_startup5665=== CONT TestParseSigningKey5666=== RUN TestParseSigningKey/valid_32-byte_key5667=== PAUSE TestParseSigningKey/valid_32-byte_key5668=== RUN TestParseSigningKey/valid_32-byte_key_with_different_name5669=== PAUSE TestParseSigningKey/valid_32-byte_key_with_different_name5670=== RUN TestParseSigningKey/no_colon5671=== PAUSE TestParseSigningKey/no_colon5672=== RUN TestParseSigningKey/empty_name5673=== PAUSE TestParseSigningKey/empty_name5674=== RUN TestParseSigningKey/invalid_base645675=== PAUSE TestParseSigningKey/invalid_base645676=== RUN TestParseSigningKey/wrong_length5677=== PAUSE TestParseSigningKey/wrong_length5678=== CONT TestParseSigningKey/valid_32-byte_key5679=== CONT TestExternalSignerClose5680=== CONT TestParseSigningKey/invalid_base645681=== CONT TestExternalSignerRestartsAfterCrash5682=== CONT TestGenerateFingerprint5683=== RUN TestGenerateFingerprint/basic_with_references5684=== PAUSE TestGenerateFingerprint/basic_with_references5685=== RUN TestGenerateFingerprint/no_references5686=== PAUSE TestGenerateFingerprint/no_references5687=== RUN TestGenerateFingerprint/unsorted_references_get_sorted5688=== PAUSE TestGenerateFingerprint/unsorted_references_get_sorted5689=== RUN TestGenerateFingerprint/invalid_nar_hash_prefix5690=== PAUSE TestGenerateFingerprint/invalid_nar_hash_prefix5691=== RUN TestGenerateFingerprint/invalid_nar_hash_length5692=== PAUSE TestGenerateFingerprint/invalid_nar_hash_length5693=== RUN TestGenerateFingerprint/invalid_store_path_prefix5694=== PAUSE TestGenerateFingerprint/invalid_store_path_prefix5695=== RUN TestGenerateFingerprint/invalid_reference_prefix5696=== PAUSE TestGenerateFingerprint/invalid_reference_prefix5697=== CONT TestGenerateFingerprint/basic_with_references5698=== CONT TestExternalSignerWrongPublicKey5699=== CONT TestGenerateFingerprint/invalid_nar_hash_length5700=== CONT TestGenerateFingerprint/unsorted_references_get_sorted5701=== CONT TestGenerateFingerprint/invalid_reference_prefix5702=== CONT TestSign5703=== PAUSE TestExternalSignerErrors/exit_at_startup5704=== RUN TestExternalSignerErrors/wrong_id5705=== PAUSE TestExternalSignerErrors/wrong_id5706=== RUN TestExternalSignerErrors/missing_signatures5707=== PAUSE TestExternalSignerErrors/missing_signatures5708=== RUN TestExternalSignerErrors/wrong_signatures_shape5709=== PAUSE TestExternalSignerErrors/wrong_signatures_shape5710=== RUN TestExternalSignerErrors/missing_signature_name5711=== PAUSE TestExternalSignerErrors/missing_signature_name5712=== RUN TestExternalSignerErrors/wrong_signature_name5713=== CONT TestParseSigningKey/wrong_length5714=== CONT TestParseSigningKey/no_colon5715=== CONT TestParseSigningKey/empty_name5716=== CONT TestParseSigningKey/valid_32-byte_key_with_different_name5717=== CONT TestGenerateFingerprint/invalid_store_path_prefix5718=== CONT TestGenerateFingerprint/no_references5719=== CONT TestGenerateFingerprint/invalid_nar_hash_prefix5720--- PASS: TestGenerateFingerprint (0.00s)5721 --- PASS: TestGenerateFingerprint/basic_with_references (0.00s)5722 --- PASS: TestGenerateFingerprint/invalid_nar_hash_length (0.00s)5723 --- PASS: TestGenerateFingerprint/unsorted_references_get_sorted (0.00s)5724 --- PASS: TestGenerateFingerprint/invalid_reference_prefix (0.00s)5725 --- PASS: TestGenerateFingerprint/invalid_store_path_prefix (0.00s)5726 --- PASS: TestGenerateFingerprint/no_references (0.00s)5727 --- PASS: TestGenerateFingerprint/invalid_nar_hash_prefix (0.00s)5728=== CONT TestExternalSignerRejectsInvalidPublicKey5729=== RUN TestExternalSignerRejectsInvalidPublicKey/#005730=== PAUSE TestExternalSignerErrors/wrong_signature_name5731=== PAUSE TestExternalSignerRejectsInvalidPublicKey/#005732=== RUN TestExternalSignerRejectsInvalidPublicKey/no-colon5733=== PAUSE TestExternalSignerRejectsInvalidPublicKey/no-colon5734=== RUN TestExternalSignerErrors/exit_on_request5735=== RUN TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5736=== PAUSE TestExternalSignerErrors/exit_on_request5737=== PAUSE TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5738=== RUN TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5739=== RUN TestExternalSignerErrors/wrong_shape5740=== PAUSE TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5741=== RUN TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5742=== PAUSE TestExternalSignerErrors/wrong_shape5743=== PAUSE TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5744=== CONT TestExternalSignerRejectsInvalidPublicKey/#005745=== CONT TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5746=== CONT TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5747=== CONT TestExternalSignerRejectsInvalidPublicKey/no-colon5748=== CONT TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5749--- PASS: TestExternalSignerRejectsInvalidPublicKey (0.00s)5750 --- PASS: TestExternalSignerRejectsInvalidPublicKey/#00 (0.00s)5751 --- PASS: TestExternalSignerRejectsInvalidPublicKey/test-key:YQ== (0.00s)5752 --- PASS: TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= (0.00s)5753 --- PASS: TestExternalSignerRejectsInvalidPublicKey/no-colon (0.00s)5754 --- PASS: TestExternalSignerRejectsInvalidPublicKey/test-key:invalid! (0.00s)5755=== RUN TestExternalSignerErrors/null_signatures5756=== PAUSE TestExternalSignerErrors/null_signatures5757=== RUN TestExternalSignerErrors/too_few_signatures5758=== PAUSE TestExternalSignerErrors/too_few_signatures5759=== RUN TestExternalSignerErrors/invalid_signature_base645760=== PAUSE TestExternalSignerErrors/invalid_signature_base645761=== RUN TestExternalSignerErrors/invalid_JSON5762=== PAUSE TestExternalSignerErrors/invalid_JSON5763=== RUN TestExternalSignerErrors/wrong_signature_length5764=== PAUSE TestExternalSignerErrors/wrong_signature_length5765=== RUN TestExternalSignerErrors/program_error5766=== PAUSE TestExternalSignerErrors/program_error5767=== RUN TestExternalSignerErrors/null5768=== PAUSE TestExternalSignerErrors/null5769=== RUN TestExternalSignerErrors/unrequested_key5770=== PAUSE TestExternalSignerErrors/unrequested_key5771=== CONT TestExternalSignerErrors/exit_at_startup5772=== CONT TestExternalSignerErrors/null_signatures5773=== CONT TestExternalSignerErrors/wrong_signature_length5774=== CONT TestExternalSignerErrors/null5775=== CONT TestExternalSignerErrors/invalid_JSON5776=== CONT TestExternalSignerErrors/wrong_shape5777=== CONT TestExternalSignerErrors/program_error5778=== CONT TestExternalSignerErrors/invalid_signature_base645779=== CONT TestExternalSignerErrors/exit_on_request5780=== CONT TestExternalSignerErrors/too_few_signatures5781=== CONT TestExternalSignerErrors/unrequested_key5782=== RUN TestSignErrors/nil_key5783=== PAUSE TestSignErrors/nil_key5784=== CONT TestExternalSignerErrors/wrong_signatures_shape5785=== CONT TestExternalSignerErrors/missing_signature_name5786=== CONT TestExternalSignerErrors/wrong_signature_name5787--- PASS: TestSign (0.00s)5788=== CONT TestExternalSignerErrors/missing_signatures5789=== CONT TestExternalSignerErrors/wrong_id5790--- PASS: TestParseSigningKey (0.00s)5791 --- PASS: TestParseSigningKey/invalid_base64 (0.00s)5792 --- PASS: TestParseSigningKey/wrong_length (0.00s)5793 --- PASS: TestParseSigningKey/no_colon (0.00s)5794 --- PASS: TestParseSigningKey/empty_name (0.00s)5795 --- PASS: TestParseSigningKey/valid_32-byte_key (0.00s)5796 --- PASS: TestParseSigningKey/valid_32-byte_key_with_different_name (0.00s)5797=== RUN TestSignErrors/nil_narinfo5798--- PASS: TestSignBatch (0.00s)5799=== PAUSE TestSignErrors/nil_narinfo5800=== RUN TestSignErrors/invalid_narinfo5801=== PAUSE TestSignErrors/invalid_narinfo5802=== RUN TestSignErrors/canceled5803=== PAUSE TestSignErrors/canceled5804=== CONT TestSignErrors/nil_key5805=== CONT TestSignErrors/invalid_narinfo5806=== CONT TestSignErrors/canceled5807=== CONT TestSignErrors/nil_narinfo5808--- PASS: TestSignErrors (0.00s)5809 --- PASS: TestSignErrors/nil_key (0.00s)5810 --- PASS: TestSignErrors/invalid_narinfo (0.00s)5811 --- PASS: TestSignErrors/canceled (0.00s)5812 --- PASS: TestSignErrors/nil_narinfo (0.00s)5813--- PASS: TestExternalSignerCancellation (0.24s)5814--- PASS: TestExternalSignerProgramErrorKeepsProcess (0.42s)5815--- PASS: TestExternalSignerClose (1.25s)5816--- PASS: TestExternalSignerWrongPublicKey (1.62s)5817--- PASS: TestExternalSignerRestartsAfterCrash (2.03s)5818/private/var/tb/2/scratch/777606c6228c2b4e/build/TestExternalSignerRestartsAfterCrash3377725149/001/sign program: line 7: /private/var/tb/2/scratch/777606c6228c2b4e/build/TestExternalSignerRestartsAfterCrash3377725149/001/sign program.starts: No such file or directory5819--- PASS: TestExternalSigner (2.22s)5820--- PASS: TestExternalSignerErrors (0.00s)5821 --- PASS: TestExternalSignerErrors/unrequested_key (0.61s)5822 --- PASS: TestExternalSignerErrors/null_signatures (0.80s)5823 --- PASS: TestExternalSignerErrors/program_error (0.99s)5824 --- PASS: TestExternalSignerErrors/wrong_id (1.43s)5825 --- PASS: TestExternalSignerErrors/wrong_signature_length (1.81s)5826 --- PASS: TestExternalSignerErrors/wrong_shape (2.40s)5827 --- PASS: TestExternalSignerErrors/too_few_signatures (2.59s)5828 --- PASS: TestExternalSignerErrors/exit_at_startup (2.79s)5829 --- PASS: TestExternalSignerErrors/exit_on_request (2.98s)5830 --- PASS: TestExternalSignerErrors/wrong_signatures_shape (3.18s)5831 --- PASS: TestExternalSignerErrors/null (3.37s)5832 --- PASS: TestExternalSignerErrors/invalid_signature_base64 (3.56s)5833 --- PASS: TestExternalSignerErrors/missing_signatures (3.75s)5834 --- PASS: TestExternalSignerErrors/invalid_JSON (3.96s)5835 --- PASS: TestExternalSignerErrors/missing_signature_name (4.15s)5836 --- PASS: TestExternalSignerErrors/wrong_signature_name (4.35s)5837PASS5838Running OpenBao signer tests...5839=== RUN TestSign5840=== PAUSE TestSign5841=== RUN TestSignErrors5842=== PAUSE TestSignErrors5843=== RUN TestServe5844=== PAUSE TestServe5845=== RUN TestServeRejectsGarbage5846=== PAUSE TestServeRejectsGarbage5847=== CONT TestSign5848=== CONT TestSignErrors5849=== CONT TestServe5850=== RUN TestSignErrors/http_error5851=== PAUSE TestSignErrors/http_error5852=== RUN TestSignErrors/invalid_json5853=== PAUSE TestSignErrors/invalid_json5854=== RUN TestSignErrors/no_prefix5855=== PAUSE TestSignErrors/no_prefix5856=== RUN TestSignErrors/invalid_base645857=== PAUSE TestSignErrors/invalid_base645858=== CONT TestServeRejectsGarbage5859--- PASS: TestServeRejectsGarbage (0.00s)5860=== CONT TestSignErrors/http_error5861=== CONT TestSignErrors/invalid_json5862=== CONT TestSignErrors/invalid_base645863=== CONT TestSignErrors/no_prefix5864--- PASS: TestServe (0.00s)5865--- PASS: TestSign (0.00s)5866--- PASS: TestSignErrors (0.00s)5867 --- PASS: TestSignErrors/invalid_base64 (0.00s)5868 --- PASS: TestSignErrors/http_error (0.00s)5869 --- PASS: TestSignErrors/invalid_json (0.00s)5870 --- PASS: TestSignErrors/no_prefix (0.00s)5871PASS