niks3-go-unit-tests
checks.aarch64-darwin.go-unit-tests
· build #508
· raw
1tribuchet: building on mac022Running client tests...3=== RUN TestDoServerRequestAttachesToken4=== PAUSE TestDoServerRequestAttachesToken5=== RUN TestUploadBuildLog_FileBodyReplayedOnRetry6=== PAUSE TestUploadBuildLog_FileBodyReplayedOnRetry7=== RUN TestRegisterUploadedObjectReusesConnections8=== PAUSE TestRegisterUploadedObjectReusesConnections9=== RUN TestRunGarbageCollection_FinishedOnAnotherReplica10=== PAUSE TestRunGarbageCollection_FinishedOnAnotherReplica11=== RUN TestRunGarbageCollection_NotFoundAfterLocalRun12=== PAUSE TestRunGarbageCollection_NotFoundAfterLocalRun13=== RUN TestCaseHackSuffix14=== PAUSE TestCaseHackSuffix15=== RUN TestFilterOversizedClosures16=== PAUSE TestFilterOversizedClosures17=== RUN TestUploadMultipart_PartsInParallel18=== PAUSE TestUploadMultipart_PartsInParallel19=== RUN TestUploadMultipart_ReturnsAfterDroppedBody20=== PAUSE TestUploadMultipart_ReturnsAfterDroppedBody21=== RUN TestUploadMultipart_FailedProducerIsNotCompleted22=== PAUSE TestUploadMultipart_FailedProducerIsNotCompleted23=== RUN TestPartSizeForNAR24=== PAUSE TestPartSizeForNAR25=== RUN TestDumpPathCaseHackMatchesNix26--- PASS: TestDumpPathCaseHackMatchesNix (0.05s)27=== RUN TestDumpPathCaseHackCollision28--- PASS: TestDumpPathCaseHackCollision (0.00s)29=== RUN TestDumpPathMatchesNix30=== PAUSE TestDumpPathMatchesNix31=== RUN TestDumpPathSingleFile32=== PAUSE TestDumpPathSingleFile33=== RUN TestDumpPathWriterError34=== PAUSE TestDumpPathWriterError35=== RUN TestEncodeNixBase3236=== PAUSE TestEncodeNixBase3237=== RUN TestEncodeNixBase32WithRealHash38=== PAUSE TestEncodeNixBase32WithRealHash39=== RUN TestConvertHashToNix3240=== PAUSE TestConvertHashToNix3241=== RUN TestGetStorePathHash42=== PAUSE TestGetStorePathHash43=== RUN TestPathInfoHashCompatibility44=== PAUSE TestPathInfoHashCompatibility45=== RUN TestParsePathInfoJSON46=== PAUSE TestParsePathInfoJSON47=== RUN TestParsePathInfoJSONMultiplePaths48=== PAUSE TestParsePathInfoJSONMultiplePaths49=== RUN TestPathInfoCACompatibility50=== PAUSE TestPathInfoCACompatibility51=== RUN TestRateLimiterFeedback52=== PAUSE TestRateLimiterFeedback53=== RUN TestRateLimiterFeedback_400DoesNotCountAsSuccess54=== PAUSE TestRateLimiterFeedback_400DoesNotCountAsSuccess55=== RUN TestRegisterUploadedObject_BoundedAgainstSilentServer56=== PAUSE TestRegisterUploadedObject_BoundedAgainstSilentServer57=== RUN TestRegisterUploadedObject_OutlivesPush58=== PAUSE TestRegisterUploadedObject_OutlivesPush59=== RUN TestResolveStorePath60=== PAUSE TestResolveStorePath61=== RUN TestDoWithRetry_BodyReplayedViaGetBody62=== PAUSE TestDoWithRetry_BodyReplayedViaGetBody63=== RUN TestRetryTimeoutOutlastsMaxRetries64=== PAUSE TestRetryTimeoutOutlastsMaxRetries65=== RUN TestDoWithRetry_FinalResponseBodyReadable66=== PAUSE TestDoWithRetry_FinalResponseBodyReadable67=== RUN TestShellSplit68=== PAUSE TestShellSplit69=== RUN TestShellSplitErrors70=== PAUSE TestShellSplitErrors71=== RUN TestStreamPushReportsEveryPath72=== PAUSE TestStreamPushReportsEveryPath73=== RUN TestStreamPushBatchesUnderLoad74=== PAUSE TestStreamPushBatchesUnderLoad75=== RUN TestStreamPushIsolatesFailures76=== PAUSE TestStreamPushIsolatesFailures77=== RUN TestStreamPushGivesUpOnDeadServer78=== PAUSE TestStreamPushGivesUpOnDeadServer79=== RUN TestStreamPushRequestLine80=== PAUSE TestStreamPushRequestLine81=== RUN TestStreamPushReportsSignatures82=== PAUSE TestStreamPushReportsSignatures83=== RUN TestStreamPushReportsSkippedPaths84=== PAUSE TestStreamPushReportsSkippedPaths85=== RUN TestClientSignaturesByStorePath86=== PAUSE TestClientSignaturesByStorePath87=== RUN TestStreamPushAnswersEveryLineWhenCanceled88=== PAUSE TestStreamPushAnswersEveryLineWhenCanceled89=== RUN TestStreamPushStopsWhenCanceled90=== PAUSE TestStreamPushStopsWhenCanceled91=== RUN TestSetClientTLS92=== PAUSE TestSetClientTLS93=== RUN TestSetClientTLSDoesNotMutateDefaultTransport94=== PAUSE TestSetClientTLSDoesNotMutateDefaultTransport95=== RUN TestSetClientTLSErrors96=== PAUSE TestSetClientTLSErrors97=== RUN TestStaticToken98=== PAUSE TestStaticToken99=== RUN TestFileTokenReadsAndCaches100=== PAUSE TestFileTokenReadsAndCaches101=== RUN TestFileTokenMissing102=== PAUSE TestFileTokenMissing103=== RUN TestFileTokenEmpty104=== PAUSE TestFileTokenEmpty105=== RUN TestScriptTokenNoExpiryRerunsEveryCall106=== PAUSE TestScriptTokenNoExpiryRerunsEveryCall107=== RUN TestScriptTokenCachesUntilRefresh108=== PAUSE TestScriptTokenCachesUntilRefresh109=== RUN TestScriptTokenEmptyToken110=== PAUSE TestScriptTokenEmptyToken111=== RUN TestScriptTokenBadJSON112=== PAUSE TestScriptTokenBadJSON113=== RUN TestScriptTokenScriptFails114=== PAUSE TestScriptTokenScriptFails115=== RUN TestScriptTokenEmptyCommand116=== PAUSE TestScriptTokenEmptyCommand117=== CONT TestDoServerRequestAttachesToken118=== CONT TestScriptTokenNoExpiryRerunsEveryCall119=== CONT TestDoWithRetry_BodyReplayedViaGetBody120=== CONT TestStreamPushReportsSignatures121=== CONT TestSetClientTLSDoesNotMutateDefaultTransport122=== CONT TestDumpPathWriterError123=== CONT TestFileTokenReadsAndCaches124=== CONT TestUploadMultipart_PartsInParallel125=== CONT TestFileTokenEmpty126=== CONT TestSetClientTLS127=== CONT TestStreamPushRequestLine128=== CONT TestStreamPushGivesUpOnDeadServer129=== CONT TestStreamPushIsolatesFailures130=== CONT TestStreamPushBatchesUnderLoad131=== RUN TestStreamPushBatchesUnderLoad/oneAtATime=false132=== PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=false1332026/11/12 11:45:54 ERROR Upload failed error=boom count=1134=== RUN TestStreamPushBatchesUnderLoad/oneAtATime=true1352026/11/12 11:45:54 ERROR Upload failed error="connection refused" count=201362026/11/12 11:45:54 ERROR Server seems unavailable, giving up on batch untried=17137=== PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=true138=== CONT TestStreamPushReportsEveryPath139=== CONT TestShellSplitErrors140=== CONT TestShellSplit1412026/11/12 11:45:54 ERROR Upload failed error=boom count=1142=== CONT TestDoWithRetry_FinalResponseBodyReadable143=== CONT TestRetryTimeoutOutlastsMaxRetries144=== CONT TestStreamPushStopsWhenCanceled1452026/11/12 11:45:54 ERROR Upload failed error="bad path" count=3146=== CONT TestStreamPushAnswersEveryLineWhenCanceled147=== CONT TestClientSignaturesByStorePath148=== CONT TestStreamPushReportsSkippedPaths149=== CONT TestRegisterUploadedObject_OutlivesPush150--- PASS: TestStreamPushReportsSignatures (0.00s)151--- PASS: TestStreamPushIsolatesFailures (0.00s)152--- PASS: TestShellSplit (0.00s)153--- PASS: TestStreamPushGivesUpOnDeadServer (0.00s)154--- PASS: TestStreamPushStopsWhenCanceled (0.00s)155--- PASS: TestShellSplitErrors (0.00s)156--- PASS: TestStreamPushReportsEveryPath (0.00s)157--- PASS: TestFileTokenEmpty (0.00s)158--- PASS: TestClientSignaturesByStorePath (0.00s)159=== CONT TestResolveStorePath160--- PASS: TestStreamPushReportsSkippedPaths (0.00s)161=== CONT TestFileTokenMissing1622000/01/01 00:00:00 ERROR Upload failed error="context canceled" count=21632026/11/12 11:45:54 WARN Rate limiter enabled after throttle name=server-test rate=51642026/11/12 11:45:54 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:509701652026/11/12 11:45:54 WARN Rate limiter backed off name=server-test rate=51662026/11/12 11:45:54 WARN Request returned retryable status, retrying attempt=2 backoff=0s status=503 url=http://127.0.0.1:509701672026/11/12 11:45:54 WARN Rate limiter enabled after throttle name=server-test rate=51682026/11/12 11:45:54 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:50975169--- PASS: TestStreamPushAnswersEveryLineWhenCanceled (0.00s)1702026/11/12 11:45:54 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:509741712026/11/12 11:45:54 WARN Rate limiter backed off name=server-test rate=5172--- PASS: TestDoWithRetry_BodyReplayedViaGetBody (0.01s)173=== CONT TestScriptTokenBadJSON174--- PASS: TestDoWithRetry_FinalResponseBodyReadable (0.00s)175=== CONT TestScriptTokenEmptyCommand176--- PASS: TestScriptTokenEmptyCommand (0.00s)177=== CONT TestScriptTokenScriptFails178--- PASS: TestDoServerRequestAttachesToken (0.01s)179--- PASS: TestSetClientTLSDoesNotMutateDefaultTransport (0.01s)180=== CONT TestRunGarbageCollection_NotFoundAfterLocalRun181=== CONT TestGetStorePathHash182=== RUN TestGetStorePathHash/valid_store_path183=== PAUSE TestGetStorePathHash/valid_store_path184=== RUN TestGetStorePathHash/basename_without_hyphen_should_error185=== PAUSE TestGetStorePathHash/basename_without_hyphen_should_error186=== RUN TestGetStorePathHash/hash_with_invalid_characters_should_error187=== PAUSE TestGetStorePathHash/hash_with_invalid_characters_should_error188=== RUN TestGetStorePathHash/hash_with_wrong_length_should_error189=== PAUSE TestGetStorePathHash/hash_with_wrong_length_should_error190=== CONT TestParsePathInfoJSON191=== CONT TestCaseHackSuffix192=== CONT TestEncodeNixBase32193=== RUN TestEncodeNixBase32/test_string_hash194=== CONT TestScriptTokenEmptyToken195=== RUN TestParsePathInfoJSON/Nix_format196=== PAUSE TestParsePathInfoJSON/Nix_format197=== CONT TestStaticToken198=== CONT TestRunGarbageCollection_FinishedOnAnotherReplica199=== CONT TestRateLimiterFeedback_400DoesNotCountAsSuccess200=== CONT TestPathInfoCACompatibility201=== CONT TestParsePathInfoJSONMultiplePaths202=== CONT TestConvertHashToNix32203=== CONT TestRateLimiterFeedback204=== CONT TestPathInfoHashCompatibility205--- PASS: TestFileTokenMissing (0.00s)206--- PASS: TestFileTokenReadsAndCaches (0.01s)207--- PASS: TestStaticToken (0.00s)208=== RUN TestConvertHashToNix32/SRI_format_to_Nix32209=== PAUSE TestConvertHashToNix32/SRI_format_to_Nix32210=== RUN TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)211--- PASS: TestRegisterUploadedObject_OutlivesPush (0.00s)212=== RUN TestConvertHashToNix32/already_Nix32_format213=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)214=== RUN TestPathInfoHashCompatibility/old_string_format_with_colon215=== RUN TestRateLimiterFeedback/429_enables_limiter216=== PAUSE TestConvertHashToNix32/already_Nix32_format217=== RUN TestConvertHashToNix32/invalid_format218=== CONT TestFilterOversizedClosures219=== PAUSE TestConvertHashToNix32/invalid_format2202026/11/12 11:45:54 WARN Rate limiter enabled after throttle name=server-test rate=5221=== RUN TestFilterOversizedClosures/no_limit_keeps_everything222=== RUN TestParsePathInfoJSON/Lix_format223=== CONT TestRegisterUploadedObject_BoundedAgainstSilentServer224=== RUN TestPathInfoCACompatibility/null_ca_field225=== PAUSE TestPathInfoCACompatibility/null_ca_field226=== RUN TestPathInfoCACompatibility/old_string_format_-_text227=== RUN TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths228=== PAUSE TestPathInfoCACompatibility/old_string_format_-_text229=== RUN TestPathInfoCACompatibility/old_string_format_-_fixed_recursive230=== PAUSE TestPathInfoCACompatibility/old_string_format_-_fixed_recursive231=== RUN TestPathInfoCACompatibility/new_structured_format_-_text232=== CONT TestRegisterUploadedObjectReusesConnections233=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_colon2342026/11/12 11:45:54 INFO Garbage collection started235=== PAUSE TestRateLimiterFeedback/429_enables_limiter236=== RUN TestRateLimiterFeedback/503_enables_limiter237=== PAUSE TestEncodeNixBase32/test_string_hash2382026/11/12 11:45:54 INFO Garbage collection started239=== CONT TestUploadBuildLog_FileBodyReplayedOnRetry2402026/11/12 11:45:54 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:50974241=== PAUSE TestFilterOversizedClosures/no_limit_keeps_everything242=== RUN TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped243=== PAUSE TestParsePathInfoJSON/Lix_format244=== RUN TestEncodeNixBase32/empty_input245=== RUN TestParsePathInfoJSON/empty_input246=== PAUSE TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths247=== RUN TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths248=== PAUSE TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths249=== PAUSE TestEncodeNixBase32/empty_input250=== CONT TestScriptTokenCachesUntilRefresh251=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_text252=== RUN TestPathInfoCACompatibility/new_structured_format_-_nar_method253=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_nar_method254=== CONT TestSetClientTLSErrors255=== RUN TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI256=== PAUSE TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI257=== RUN TestPathInfoHashCompatibility/new_structured_format_with_sha512258=== CONT TestUploadMultipart_FailedProducerIsNotCompleted259=== RUN TestSetClientTLS/rejects_connection_without_client_cert260=== RUN TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part261=== PAUSE TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part262=== RUN TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary263=== PAUSE TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary264=== PAUSE TestPathInfoHashCompatibility/new_structured_format_with_sha512265=== CONT TestDumpPathSingleFile266=== PAUSE TestSetClientTLS/rejects_connection_without_client_cert267=== PAUSE TestRateLimiterFeedback/503_enables_limiter268=== RUN TestSetClientTLS/succeeds_with_client_cert_and_CA269=== PAUSE TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped270=== RUN TestFilterOversizedClosures/all_closures_skipped271=== PAUSE TestFilterOversizedClosures/all_closures_skipped272=== RUN TestRateLimiterFeedback/200_does_not_enable_limiter273=== PAUSE TestRateLimiterFeedback/200_does_not_enable_limiter274=== RUN TestRateLimiterFeedback/400_does_not_enable_limiter275=== PAUSE TestRateLimiterFeedback/400_does_not_enable_limiter276=== CONT TestUploadMultipart_ReturnsAfterDroppedBody277=== PAUSE TestParsePathInfoJSON/empty_input278=== RUN TestParsePathInfoJSON/whitespace_only279=== PAUSE TestParsePathInfoJSON/whitespace_only280=== RUN TestParsePathInfoJSON/invalid_JSON281=== PAUSE TestParsePathInfoJSON/invalid_JSON282=== CONT TestDumpPathMatchesNix283=== CONT TestEncodeNixBase32WithRealHash284--- PASS: TestEncodeNixBase32WithRealHash (0.00s)285=== CONT TestStreamPushBatchesUnderLoad/oneAtATime=false286=== CONT TestStreamPushBatchesUnderLoad/oneAtATime=true287--- PASS: TestRunGarbageCollection_NotFoundAfterLocalRun (0.00s)288=== PAUSE TestSetClientTLS/succeeds_with_client_cert_and_CA289=== RUN TestSetClientTLS/preserves_debug_logging_transport290=== PAUSE TestSetClientTLS/preserves_debug_logging_transport291=== CONT TestGetStorePathHash/valid_store_path292=== CONT TestGetStorePathHash/hash_with_wrong_length_should_error293=== CONT TestGetStorePathHash/basename_without_hyphen_should_error294=== CONT TestGetStorePathHash/hash_with_invalid_characters_should_error295--- PASS: TestGetStorePathHash (0.00s)296 --- PASS: TestGetStorePathHash/valid_store_path (0.00s)297 --- PASS: TestGetStorePathHash/hash_with_wrong_length_should_error (0.00s)298 --- PASS: TestGetStorePathHash/basename_without_hyphen_should_error (0.00s)299 --- PASS: TestGetStorePathHash/hash_with_invalid_characters_should_error (0.00s)300=== CONT TestConvertHashToNix32/SRI_format_to_Nix32301=== CONT TestConvertHashToNix32/invalid_format302=== CONT TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths303=== CONT TestConvertHashToNix32/already_Nix32_format304--- PASS: TestConvertHashToNix32 (0.00s)305 --- PASS: TestConvertHashToNix32/SRI_format_to_Nix32 (0.00s)306 --- PASS: TestConvertHashToNix32/invalid_format (0.00s)307 --- PASS: TestConvertHashToNix32/already_Nix32_format (0.00s)308--- PASS: TestResolveStorePath (0.00s)309=== CONT TestEncodeNixBase32/test_string_hash310=== CONT TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths311--- PASS: TestParsePathInfoJSONMultiplePaths (0.00s)312 --- PASS: TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths (0.00s)313 --- PASS: TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths (0.00s)314--- PASS: TestStreamPushBatchesUnderLoad (0.00s)315 --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=false (0.00s)316 --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=true (0.00s)317=== CONT TestPathInfoCACompatibility/new_structured_format_-_nar_method318=== CONT TestPathInfoCACompatibility/new_structured_format_-_text319=== CONT TestPathInfoCACompatibility/old_string_format_-_text320=== CONT TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part321=== CONT TestPathInfoCACompatibility/null_ca_field322=== CONT TestPathInfoCACompatibility/old_string_format_-_fixed_recursive323--- PASS: TestPathInfoCACompatibility (0.00s)324 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_nar_method (0.00s)325 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_text (0.00s)326 --- PASS: TestPathInfoCACompatibility/old_string_format_-_text (0.00s)327 --- PASS: TestPathInfoCACompatibility/null_ca_field (0.00s)328 --- PASS: TestPathInfoCACompatibility/old_string_format_-_fixed_recursive (0.00s)329=== CONT TestEncodeNixBase32/empty_input330--- PASS: TestEncodeNixBase32 (0.00s)331 --- PASS: TestEncodeNixBase32/test_string_hash (0.00s)332 --- PASS: TestEncodeNixBase32/empty_input (0.00s)333=== CONT TestFilterOversizedClosures/no_limit_keeps_everything334=== CONT TestRateLimiterFeedback/429_enables_limiter335=== CONT TestFilterOversizedClosures/all_closures_skipped336=== CONT TestPartSizeForNAR337=== RUN TestPartSizeForNAR/zero_stays_at_minimum338=== PAUSE TestPartSizeForNAR/zero_stays_at_minimum339=== RUN TestPartSizeForNAR/small_stays_at_minimum340=== PAUSE TestPartSizeForNAR/small_stays_at_minimum341=== RUN TestPartSizeForNAR/80_GiB_fits_at_minimum342=== PAUSE TestPartSizeForNAR/80_GiB_fits_at_minimum343=== RUN TestPartSizeForNAR/115_GiB_needs_larger_parts344=== PAUSE TestPartSizeForNAR/115_GiB_needs_larger_parts345=== RUN TestPartSizeForNAR/1_TiB346=== PAUSE TestPartSizeForNAR/1_TiB347=== CONT TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped3482026/11/12 11:45:54 INFO Garbage collection progress phase=running_on_another_replica failed_uploads_deleted=0 old_closures_deleted=0 objects_marked=0 objects_deleted=0 objects_failed=03492026/11/12 11:45:54 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper nar_size=100 max_nar_size=50350=== RUN TestSetClientTLSErrors/missing_cert_file351=== CONT TestParsePathInfoJSON/Nix_format352=== PAUSE TestSetClientTLSErrors/missing_cert_file353=== RUN TestSetClientTLSErrors/missing_key_file354=== PAUSE TestSetClientTLSErrors/missing_key_file355=== RUN TestSetClientTLSErrors/missing_ca_file356=== PAUSE TestSetClientTLSErrors/missing_ca_file357=== RUN TestSetClientTLSErrors/invalid_ca_file358=== PAUSE TestSetClientTLSErrors/invalid_ca_file359=== CONT TestRateLimiterFeedback/200_does_not_enable_limiter360=== CONT TestRateLimiterFeedback/400_does_not_enable_limiter3612026/11/12 11:45:54 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb-vm-image nar_size=5000 max_nar_size=2000362=== RUN TestPartSizeForNAR/5_TiB_S3_max_object363--- PASS: TestFilterOversizedClosures (0.00s)364 --- PASS: TestFilterOversizedClosures/no_limit_keeps_everything (0.00s)365 --- PASS: TestFilterOversizedClosures/all_closures_skipped (0.01s)366 --- PASS: TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped (0.01s)367=== PAUSE TestPartSizeForNAR/5_TiB_S3_max_object368=== CONT TestRateLimiterFeedback/503_enables_limiter369=== RUN TestPartSizeForNAR/capped_at_5_GiB370=== PAUSE TestPartSizeForNAR/capped_at_5_GiB371=== CONT TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary3722026/11/12 11:45:54 WARN Rate limiter enabled after throttle name=server-test rate=53732026/11/12 11:45:54 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=429 url=http://127.0.0.1:510253742026/11/12 11:45:54 WARN Garbage collection finished on another replica. Its statistics are not available here375--- PASS: TestRunGarbageCollection_FinishedOnAnotherReplica (0.01s)376=== CONT TestPathInfoHashCompatibility/new_structured_format_with_sha512377=== CONT TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI378=== CONT TestPathInfoHashCompatibility/old_string_format_with_colon379=== CONT TestSetClientTLS/rejects_connection_without_client_cert380=== CONT TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)381--- PASS: TestPathInfoHashCompatibility (0.00s)382 --- PASS: TestPathInfoHashCompatibility/new_structured_format_with_sha512 (0.00s)383 --- PASS: TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI (0.00s)384 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_colon (0.00s)385 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI) (0.00s)386=== CONT TestParsePathInfoJSON/whitespace_only387=== CONT TestParsePathInfoJSON/invalid_JSON388=== CONT TestSetClientTLS/preserves_debug_logging_transport389--- PASS: TestUploadMultipart_ReturnsAfterDroppedBody (0.01s)390=== CONT TestSetClientTLS/succeeds_with_client_cert_and_CA3912026/11/12 11:45:54 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:509743922026/11/12 11:45:54 WARN Rate limiter enabled after throttle name=server-test rate=53932026/11/12 11:45:54 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:51070394=== CONT TestParsePathInfoJSON/empty_input395=== CONT TestParsePathInfoJSON/Lix_format396--- PASS: TestParsePathInfoJSON (0.00s)397 --- PASS: TestParsePathInfoJSON/Nix_format (0.00s)398 --- PASS: TestParsePathInfoJSON/whitespace_only (0.00s)399 --- PASS: TestParsePathInfoJSON/invalid_JSON (0.00s)400 --- PASS: TestParsePathInfoJSON/empty_input (0.00s)401 --- PASS: TestParsePathInfoJSON/Lix_format (0.00s)402=== CONT TestSetClientTLSErrors/missing_cert_file403=== CONT TestSetClientTLSErrors/missing_ca_file404=== CONT TestSetClientTLSErrors/invalid_ca_file405--- PASS: TestUploadMultipart_FailedProducerIsNotCompleted (0.00s)406 --- PASS: TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part (0.01s)407 --- PASS: TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary (0.00s)408=== CONT TestSetClientTLSErrors/missing_key_file4092026/11/12 11:45:54 WARN Rate limiter backed off name=server-test rate=5410=== CONT TestPartSizeForNAR/5_TiB_S3_max_object411=== CONT TestPartSizeForNAR/1_TiB412=== CONT TestPartSizeForNAR/115_GiB_needs_larger_parts413=== CONT TestPartSizeForNAR/80_GiB_fits_at_minimum414=== CONT TestPartSizeForNAR/small_stays_at_minimum415=== CONT TestPartSizeForNAR/zero_stays_at_minimum416=== CONT TestPartSizeForNAR/capped_at_5_GiB417--- PASS: TestPartSizeForNAR (0.01s)418 --- PASS: TestPartSizeForNAR/5_TiB_S3_max_object (0.00s)419 --- PASS: TestPartSizeForNAR/1_TiB (0.00s)420 --- PASS: TestPartSizeForNAR/115_GiB_needs_larger_parts (0.00s)421 --- PASS: TestPartSizeForNAR/80_GiB_fits_at_minimum (0.00s)422 --- PASS: TestPartSizeForNAR/small_stays_at_minimum (0.00s)423 --- PASS: TestPartSizeForNAR/zero_stays_at_minimum (0.00s)424 --- PASS: TestPartSizeForNAR/capped_at_5_GiB (0.00s)425--- PASS: TestSetClientTLSErrors (0.01s)426 --- PASS: TestSetClientTLSErrors/missing_cert_file (0.00s)427 --- PASS: TestSetClientTLSErrors/missing_ca_file (0.00s)428 --- PASS: TestSetClientTLSErrors/invalid_ca_file (0.00s)429 --- PASS: TestSetClientTLSErrors/missing_key_file (0.00s)4302026/11/12 11:45:54 WARN Rate limiter enabled after throttle name=s3-test rate=54312026/11/12 11:45:54 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:50998432--- PASS: TestScriptTokenScriptFails (0.01s)4332026/11/12 11:45:54 WARN Rate limiter backed off name=server-test rate=54342026/11/12 11:45:54 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:50974435--- PASS: TestUploadBuildLog_FileBodyReplayedOnRetry (0.01s)436--- PASS: TestStreamPushRequestLine (0.02s)437--- PASS: TestRateLimiterFeedback (0.00s)438 --- PASS: TestRateLimiterFeedback/200_does_not_enable_limiter (0.00s)439 --- PASS: TestRateLimiterFeedback/429_enables_limiter (0.01s)440 --- PASS: TestRateLimiterFeedback/400_does_not_enable_limiter (0.00s)441 --- PASS: TestRateLimiterFeedback/503_enables_limiter (0.01s)442--- PASS: TestScriptTokenBadJSON (0.02s)4432026/11/12 11:45:54 WARN Request returned retryable status, retrying attempt=3 backoff=1ms status=502 url=http://127.0.0.1:509744442026/11/12 11:45:54 WARN Request returned retryable status, retrying attempt=4 backoff=1ms status=502 url=http://127.0.0.1:509744452026/11/12 11:45:54 WARN Request returned retryable status, retrying attempt=5 backoff=1ms status=502 url=http://127.0.0.1:509744462026/11/12 11:45:54 WARN Request returned retryable status, retrying attempt=6 backoff=1ms status=502 url=http://127.0.0.1:509744472026/11/12 11:45:54 WARN Request returned retryable status, retrying attempt=7 backoff=1ms status=502 url=http://127.0.0.1:509744482026/11/12 11:45:54 WARN Request returned retryable status, retrying attempt=8 backoff=1ms status=502 url=http://127.0.0.1:509744492026/11/12 11:45:54 WARN Request returned retryable status, retrying attempt=9 backoff=1ms status=502 url=http://127.0.0.1:509744502026/11/12 11:45:54 WARN Request returned retryable status, retrying attempt=10 backoff=1ms status=502 url=http://127.0.0.1:509744512026/11/12 11:45:54 WARN Request returned retryable status, retrying attempt=11 backoff=1ms status=502 url=http://127.0.0.1:509744522026/11/12 11:45:54 WARN Request returned retryable status, retrying attempt=12 backoff=1ms status=502 url=http://127.0.0.1:50974453--- PASS: TestScriptTokenEmptyToken (0.04s)454--- PASS: TestRegisterUploadedObjectReusesConnections (0.04s)4552026/11/12 11:45:54 WARN Request returned retryable status, retrying attempt=13 backoff=1ms status=502 url=http://127.0.0.1:509744562026/11/12 11:45:54 http: TLS handshake error from 127.0.0.1:51075: remote error: tls: bad certificate4572026/11/12 11:45:54 WARN Request returned retryable status, retrying attempt=14 backoff=1ms status=502 url=http://127.0.0.1:50974458--- PASS: TestScriptTokenNoExpiryRerunsEveryCall (0.05s)459--- PASS: TestSetClientTLS (0.01s)460 --- PASS: TestSetClientTLS/preserves_debug_logging_transport (0.00s)461 --- PASS: TestSetClientTLS/succeeds_with_client_cert_and_CA (0.01s)462 --- PASS: TestSetClientTLS/rejects_connection_without_client_cert (0.03s)4632026/11/12 11:45:54 WARN Request returned retryable status, retrying attempt=15 backoff=1ms status=502 url=http://127.0.0.1:509744642026/11/12 11:45:54 WARN Request returned retryable status, retrying attempt=16 backoff=1ms status=502 url=http://127.0.0.1:509744652026/11/12 11:45:54 WARN Request returned retryable status, retrying attempt=17 backoff=1ms status=502 url=http://127.0.0.1:50974466--- PASS: TestRetryTimeoutOutlastsMaxRetries (0.05s)467--- PASS: TestScriptTokenCachesUntilRefresh (0.05s)468--- PASS: TestDumpPathWriterError (0.06s)469--- PASS: TestDumpPathSingleFile (0.06s)470--- PASS: TestCaseHackSuffix (0.06s)471--- PASS: TestDumpPathMatchesNix (0.08s)4722026/11/12 11:45:54 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:50985/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:50985/api/uploads/complete4732026/11/12 11:45:54 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"4742026/11/12 11:45:54 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:50985/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:50985/api/uploads/complete4752026/11/12 11:45:54 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"4762026/11/12 11:45:54 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:50985/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:50985/api/uploads/complete4772026/11/12 11:45:54 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"478--- PASS: TestRegisterUploadedObject_BoundedAgainstSilentServer (0.10s)479--- PASS: TestUploadMultipart_PartsInParallel (0.62s)480--- PASS: TestRateLimiterFeedback_400DoesNotCountAsSuccess (1.00s)481PASS482Running server tests...483The files belonging to this database system will be owned by user "_tribuchetbld1".484This user must also own the server process.485486The database cluster will be initialized with locale "C".487The default database encoding has accordingly been set to "SQL_ASCII".488The default text search configuration will be set to "english".489490Data page checksums are enabled.491492creating directory /private/var/tb/1/scratch/d6c7014d22e1e3f1/build/postgres1637863116/data ... ok493creating subdirectories ... ok494selecting dynamic shared memory implementation ... posix495selecting default "max_connections" ... 100496selecting default "shared_buffers" ... 128MB497selecting default time zone ... UTC498creating configuration files ... ok499running bootstrap script ... ok500performing post-bootstrap initialization ... ok501syncing data to disk ... ok502503initdb: warning: enabling "trust" authentication for local connections504initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.505506Success. You can now start the database server using:507508 pg_ctl -D /private/var/tb/1/scratch/d6c7014d22e1e3f1/build/postgres1637863116/data -l logfile start5095102026-11-12 11:45:56.286 UTC [49288] LOG: starting PostgreSQL 18.6 on aarch64-apple-darwin25.6.0, compiled by clang version 21.1.8, 64-bit5112026-11-12 11:45:56.287 UTC [49288] LOG: listening on Unix socket "/private/var/tb/1/scratch/d6c7014d22e1e3f1/build/postgres1637863116/.s.PGSQL.5432"5122026-11-12 11:45:56.289 UTC [49295] LOG: database system was shut down at 2026-11-12 11:45:56 UTC5132026-11-12 11:45:56.289 UTC [49296] FATAL: the database system is starting up514/private/var/tb/1/scratch/d6c7014d22e1e3f1/build/postgres1637863116:5432 - rejecting connections5152026-11-12 11:45:56.290 UTC [49288] LOG: database system is ready to accept connections516/private/var/tb/1/scratch/d6c7014d22e1e3f1/build/postgres1637863116:5432 - accepting connections517{"timestamp":"2026-11-12T11:45:56.544903Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"eb4090ad-1096-4708-8182-3e263e0b67d1","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"GET","uri":"/health/ready","status_code":503,"suppressed_errors":0,"duration_ms":11,"result":"server_error","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":502,"threadName":"rustfs-worker","threadId":"ThreadId(6)"}518{"timestamp":"2026-11-12T11:45:56.652764Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"a3b329c3-ff92-48b0-b6bd-bd39c25c14a0","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"GET","uri":"/health/ready","status_code":503,"suppressed_errors":0,"duration_ms":0,"result":"server_error","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":502,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}519{"timestamp":"2026-11-12T11:45:56.756217Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"873a8bb7-c778-4bca-8826-35d33505d9d3","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"GET","uri":"/health/ready","status_code":503,"suppressed_errors":0,"duration_ms":0,"result":"server_error","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":502,"threadName":"rustfs-worker","threadId":"ThreadId(7)"}520=== RUN TestAbortSucceeded521=== PAUSE TestAbortSucceeded522=== RUN TestService_AuthMiddleware523=== PAUSE TestService_AuthMiddleware524=== RUN TestService_AuthMiddleware_MTLSProxyHeader525=== PAUSE TestService_AuthMiddleware_MTLSProxyHeader526=== RUN TestService_AuthMiddleware_MTLSBoundSubjects527=== PAUSE TestService_AuthMiddleware_MTLSBoundSubjects528=== RUN TestService_ReadAuthMiddleware529=== PAUSE TestService_ReadAuthMiddleware530=== RUN TestService_AuthMiddleware_OIDC531=== PAUSE TestService_AuthMiddleware_OIDC532=== RUN TestService_RequireScope_OIDC533=== PAUSE TestService_RequireScope_OIDC534=== RUN TestService_ReadScope_PublicByDefault535=== PAUSE TestService_ReadScope_PublicByDefault536=== RUN TestCacheConfigHandler537=== PAUSE TestCacheConfigHandler538=== RUN TestCacheStatsHandler539=== PAUSE TestCacheStatsHandler540=== RUN TestClientCADerivations541=== PAUSE TestClientCADerivations542=== RUN TestClientErrorHandling543=== PAUSE TestClientErrorHandling544=== RUN TestClientIntegration545=== PAUSE TestClientIntegration546=== RUN TestClientMultipleUploads547=== PAUSE TestClientMultipleUploads548=== RUN TestClientWithDependencies549=== PAUSE TestClientWithDependencies550=== RUN TestClientSharedPathCommittedMidPush551=== PAUSE TestClientSharedPathCommittedMidPush552=== RUN TestPinProtectsFromGC553=== PAUSE TestPinProtectsFromGC554=== RUN TestClientPushesUseOnePush555=== PAUSE TestClientPushesUseOnePush556=== RUN TestClientFallsBackToClosures557=== PAUSE TestClientFallsBackToClosures558=== RUN TestDeleteClosuresRacingPins559=== PAUSE TestDeleteClosuresRacingPins560=== RUN TestConcurrentCommitsSharingObjectsDoNotDeadlock561=== PAUSE TestConcurrentCommitsSharingObjectsDoNotDeadlock562=== RUN TestResolveDBConnectionString563=== PAUSE TestResolveDBConnectionString564=== RUN TestConnectWaitsForAPeerMigration565=== PAUSE TestConnectWaitsForAPeerMigration566=== RUN TestLeadElectsOneAndHandsOver567=== PAUSE TestLeadElectsOneAndHandsOver568=== RUN TestLeadIncumbentWinsAfterRestart5692026/11/12 11:45:57 INFO lead: acquired remote=192.0.2.1:12345702026/11/12 11:45:57 INFO lead: released remote=192.0.2.1:12345712026/11/12 11:45:57 INFO lead: acquired remote=192.0.2.1:12345722026/11/12 11:45:58 INFO lead: released remote=192.0.2.1:1234573--- PASS: TestLeadIncumbentWinsAfterRestart (2.09s)574=== RUN TestLeadUnannouncedStandbyYieldsToIncumbent575=== PAUSE TestLeadUnannouncedStandbyYieldsToIncumbent576=== RUN TestLeadIncumbentAnnouncesAtOnce577=== PAUSE TestLeadIncumbentAnnouncesAtOnce578=== RUN TestLeadEndsOnShutdown579=== PAUSE TestLeadEndsOnShutdown580=== RUN TestLeadEndsWhenItsConnectionHangs5812026/11/12 11:45:59 INFO lead: acquired remote=192.0.2.1:12345822026-11-12 11:46:00.532 UTC [49348] FATAL: terminating connection due to administrator command5832026/11/12 11:46:00 INFO lead: acquired remote=192.0.2.1:12345842026/11/12 11:46:01 WARN lead: lock connection lost error="timeout: context deadline exceeded"5852026/11/12 11:46:01 INFO lead: released remote=192.0.2.1:12345862026/11/12 11:46:01 INFO lead: released remote=192.0.2.1:1234587--- PASS: TestLeadEndsWhenItsConnectionHangs (2.97s)588=== RUN TestGCAdvisoryLockBlocksConcurrentRun589=== PAUSE TestGCAdvisoryLockBlocksConcurrentRun590=== RUN TestGCBugBareHashReferences591=== PAUSE TestGCBugBareHashReferences592=== RUN TestGCMetrics593=== PAUSE TestGCMetrics594=== RUN TestPushDedupSurvivesConcurrentGC595=== PAUSE TestPushDedupSurvivesConcurrentGC596=== RUN TestDeduplicatedObjectsRecordedAsPending597=== PAUSE TestDeduplicatedObjectsRecordedAsPending598=== RUN TestGCSweepSkipsPendingObjects599=== PAUSE TestGCSweepSkipsPendingObjects600=== RUN TestTombstonedObjectOfferedWithoutWaiting601=== PAUSE TestTombstonedObjectOfferedWithoutWaiting602=== RUN TestGCSweepDeliversEachKeyOnce603=== PAUSE TestGCSweepDeliversEachKeyOnce604=== RUN TestCreatePendingClosureVerifyS3FailureReleasesConnection605=== PAUSE TestCreatePendingClosureVerifyS3FailureReleasesConnection606=== RUN TestForceGCDuringPushOffersSweptObject607=== PAUSE TestForceGCDuringPushOffersSweptObject608=== RUN TestFailedS3DeleteKeepsTombstone609=== PAUSE TestFailedS3DeleteKeepsTombstone610=== RUN TestSweepSkipsRowsLockedByPush611=== PAUSE TestSweepSkipsRowsLockedByPush612=== RUN TestRegisterCompletedObjectNeedsPendingRow613=== PAUSE TestRegisterCompletedObjectNeedsPendingRow614=== RUN TestGetExistingObjectsLocksRows615=== PAUSE TestGetExistingObjectsLocksRows616=== RUN TestSweepRechecksPendingAfterLocking617=== PAUSE TestSweepRechecksPendingAfterLocking618=== RUN TestPushWaitsForSweepHoldingRow619=== PAUSE TestPushWaitsForSweepHoldingRow620=== RUN TestCommitRacingPendingCleanupKeepsObjects621=== PAUSE TestCommitRacingPendingCleanupKeepsObjects622=== RUN TestGCEndsOnShutdown623=== PAUSE TestGCEndsOnShutdown624=== RUN TestGCTaskStore_StartNew625=== PAUSE TestGCTaskStore_StartNew626=== RUN TestGCTaskStore_DeduplicateSameParams627=== PAUSE TestGCTaskStore_DeduplicateSameParams628=== RUN TestGCTaskStore_ConflictDifferentParams629=== PAUSE TestGCTaskStore_ConflictDifferentParams630=== RUN TestGCTaskStore_GetEmpty631=== PAUSE TestGCTaskStore_GetEmpty632=== RUN TestGCTaskStore_GetReturnsLatest633=== PAUSE TestGCTaskStore_GetReturnsLatest634=== RUN TestGCTaskStore_CompletedAllowsNewTask635=== PAUSE TestGCTaskStore_CompletedAllowsNewTask636=== RUN TestGCTaskStore_PhaseUpdates637=== PAUSE TestGCTaskStore_PhaseUpdates638=== RUN TestGCTaskStore_Fail639=== PAUSE TestGCTaskStore_Fail640=== RUN TestGracefulShutdownDrainsInflight641=== PAUSE TestGracefulShutdownDrainsInflight642=== RUN TestService_healthCheckHandler643=== PAUSE TestService_healthCheckHandler644=== RUN TestService_readinessHandler645=== PAUSE TestService_readinessHandler646=== RUN TestGenerateLandingPage647=== PAUSE TestGenerateLandingPage648=== RUN TestEnsureAbortUploadsRule649=== PAUSE TestEnsureAbortUploadsRule650=== RUN TestCacheConfigHandlerMaxNarSize651=== PAUSE TestCacheConfigHandlerMaxNarSize652=== RUN TestCreatePendingClosureRejectsOversizedNAR653=== PAUSE TestCreatePendingClosureRejectsOversizedNAR654=== RUN TestNARDeduplicationMetadataUploadBug655=== PAUSE TestNARDeduplicationMetadataUploadBug656=== RUN TestMetricsInventory657=== PAUSE TestMetricsInventory658=== RUN TestMetricsCommit659=== PAUSE TestMetricsCommit660=== RUN TestService_NativeMTLS661=== PAUSE TestService_NativeMTLS662=== RUN TestServerTLSConfig663=== PAUSE TestServerTLSConfig664=== RUN TestMultipartCleanup665=== PAUSE TestMultipartCleanup666=== RUN TestMultipartUploadAbortedWhenCancelledBeforeRecorded667=== PAUSE TestMultipartUploadAbortedWhenCancelledBeforeRecorded668=== RUN TestPendingCleanupUsesOneCutoff669=== PAUSE TestPendingCleanupUsesOneCutoff670=== RUN TestObjectStatsTrigger671=== PAUSE TestObjectStatsTrigger672=== RUN TestObjectStatsUpdatedOncePerCommit673=== PAUSE TestObjectStatsUpdatedOncePerCommit674=== RUN TestOrphanedObjectsGC675=== PAUSE TestOrphanedObjectsGC676=== RUN TestOrphanedObjectsGCStressTest677=== PAUSE TestOrphanedObjectsGCStressTest678=== RUN TestCreatePin_ReservedPins679=== PAUSE TestCreatePin_ReservedPins680=== RUN TestPresentReportsOnlyClosureRoots681=== PAUSE TestPresentReportsOnlyClosureRoots682=== RUN TestPresentRefreshesOnlyStaleClosures683=== PAUSE TestPresentRefreshesOnlyStaleClosures684=== RUN TestPresentNotReportedWhileGCDeletesClosure685=== PAUSE TestPresentNotReportedWhileGCDeletesClosure686=== RUN TestParseSingleRange687=== PAUSE TestParseSingleRange688=== RUN TestProxyHeadersOnlyTrustedOnSocket6892026/11/12 11:46:02 INFO Starting HTTP server address=127.0.0.1:511056902026/11/12 11:46:02 INFO Starting HTTP server address=proxy.sock6912026/11/12 11:46:02 WARN mTLS auth: subject not in bound subjects subject="CN=someone"6922026/11/12 11:46:02 INFO Shutdown signal received, draining in-flight requests timeout=10s693--- PASS: TestProxyHeadersOnlyTrustedOnSocket (0.17s)694=== RUN TestIsValidCachePath695=== PAUSE TestIsValidCachePath696=== RUN TestReadProxyNarinfo697=== PAUSE TestReadProxyNarinfo698=== RUN TestReadProxyNarinfoAlreadyDecompressed699=== PAUSE TestReadProxyNarinfoAlreadyDecompressed700=== RUN TestReadProxyContentNegotiation701=== PAUSE TestReadProxyContentNegotiation702=== RUN TestReadProxyNarStreaming703=== PAUSE TestReadProxyNarStreaming704=== RUN TestReadProxy404705=== PAUSE TestReadProxy404706=== RUN TestReadProxyInvalidPath707=== PAUSE TestReadProxyInvalidPath708=== RUN TestReadProxyHead709=== PAUSE TestReadProxyHead710=== RUN TestReadProxyOutlastsServerWriteTimeout711=== PAUSE TestReadProxyOutlastsServerWriteTimeout712=== RUN TestReadProxyConditionalGet713=== PAUSE TestReadProxyConditionalGet714=== RUN TestReadProxyRootRedirectsToIndexHTML715=== PAUSE TestReadProxyRootRedirectsToIndexHTML716=== RUN TestReadProxyDisabled717=== PAUSE TestReadProxyDisabled718=== RUN TestReadRedirectNar719=== PAUSE TestReadRedirectNar720=== RUN TestReadRedirectKeepsNarinfoProxied721=== PAUSE TestReadRedirectKeepsNarinfoProxied722=== RUN TestReadProxyRangeRequest723=== PAUSE TestReadProxyRangeRequest724=== RUN TestReadRedirectUsesPublicS3URL725=== PAUSE TestReadRedirectUsesPublicS3URL726=== RUN TestPush_OverlappingRootsStoreOneRowPerKey727=== PAUSE TestPush_OverlappingRootsStoreOneRowPerKey728=== RUN TestPush_CompleteCommitsEveryRoot729=== PAUSE TestPush_CompleteCommitsEveryRoot730=== RUN TestPush_RepeatedRootCommitsOnce731=== PAUSE TestPush_RepeatedRootCommitsOnce732=== RUN TestPush_CompleteWaitsForCleanupOfThePush733=== PAUSE TestPush_CompleteWaitsForCleanupOfThePush734=== RUN TestPush_RootsAreLockedInByteOrder735=== PAUSE TestPush_RootsAreLockedInByteOrder736=== RUN TestPush_SkippedKeySurvivesGCBeforeCommit737=== PAUSE TestPush_SkippedKeySurvivesGCBeforeCommit738=== RUN TestPush_RejectsBadRequests739=== PAUSE TestPush_RejectsBadRequests740=== RUN TestPush_SignsNarinfosOfItsPendingObjects741=== PAUSE TestPush_SignsNarinfosOfItsPendingObjects742=== RUN TestCommitCanBeRepeatedUntilCleanup743=== PAUSE TestCommitCanBeRepeatedUntilCleanup744=== RUN TestSharedNarBothUploadsComplete745=== PAUSE TestSharedNarBothUploadsComplete746=== RUN TestCompleteMultipartUpload_ErrorButObjectExists747=== PAUSE TestCompleteMultipartUpload_ErrorButObjectExists748=== RUN TestCompletedNarNotReofferedAcrossClosures749=== PAUSE TestCompletedNarNotReofferedAcrossClosures750=== RUN TestPresignedUploadRegisteredBeforeCommit751=== PAUSE TestPresignedUploadRegisteredBeforeCommit752=== RUN TestService_Rustfstest753=== PAUSE TestService_Rustfstest754=== RUN TestParseSize755=== PAUSE TestParseSize756=== RUN TestSkippedUploadsHandler757=== PAUSE TestSkippedUploadsHandler758=== RUN TestSystemdListenerNotActivated759--- PASS: TestSystemdListenerNotActivated (0.00s)760=== RUN TestWatchdogBeatsWhenHealthy761--- PASS: TestWatchdogBeatsWhenHealthy (0.02s)762=== RUN TestWatchdogSkipsWhenUnhealthy7632026/11/12 11:46:02 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7642026/11/12 11:46:02 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7652026/11/12 11:46:02 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7662026/11/12 11:46:02 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7672026/11/12 11:46:02 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7682026/11/12 11:46:02 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7692026/11/12 11:46:02 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7702026/11/12 11:46:02 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7712026/11/12 11:46:02 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7722026/11/12 11:46:02 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"773--- PASS: TestWatchdogSkipsWhenUnhealthy (0.20s)774=== RUN TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle775=== PAUSE TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle776=== RUN TestProxyWriteTimeout777=== PAUSE TestProxyWriteTimeout778=== RUN TestPendingClosureWriteTimeout779=== PAUSE TestPendingClosureWriteTimeout780=== RUN TestIsValidUploadKey781=== PAUSE TestIsValidUploadKey782=== RUN TestUploadHandlersRejectInvalidKeys783=== PAUSE TestUploadHandlersRejectInvalidKeys784=== RUN TestUploadHandlersRejectOversizedBody785=== PAUSE TestUploadHandlersRejectOversizedBody786=== RUN TestService_cleanupPendingClosuresHandler787=== PAUSE TestService_cleanupPendingClosuresHandler788=== RUN TestService_createPendingClosureHandler789=== PAUSE TestService_createPendingClosureHandler790=== RUN TestService_verifyS3Integrity791=== PAUSE TestService_verifyS3Integrity792=== RUN TestService_verifyS3ReuploadsStaleNarinfo793=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo794=== RUN TestCompleteMultipartUnregistered795=== PAUSE TestCompleteMultipartUnregistered796=== RUN TestCreatePendingClosure_SmallNARUsesSimplePUT797=== PAUSE TestCreatePendingClosure_SmallNARUsesSimplePUT798=== CONT TestIsValidCachePath799=== CONT TestOrphanedObjectsGC800=== RUN TestIsValidCachePath/narinfo801=== PAUSE TestIsValidCachePath/narinfo802=== RUN TestIsValidCachePath/narinfo_all_nix_base32_chars803=== CONT TestAbortSucceeded804--- PASS: TestAbortSucceeded (0.00s)805=== CONT TestService_createPendingClosureHandler806=== CONT TestService_Rustfstest807=== CONT TestParseSingleRange808=== RUN TestParseSingleRange/none809=== CONT TestPresentNotReportedWhileGCDeletesClosure810=== PAUSE TestParseSingleRange/none811=== CONT TestPresentRefreshesOnlyStaleClosures812=== RUN TestParseSingleRange/unknown_unit813=== RUN TestPresentNotReportedWhileGCDeletesClosure/stale814=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/stale815=== CONT TestPush_RejectsBadRequests816=== CONT TestCreatePendingClosure_SmallNARUsesSimplePUT817=== CONT TestCreatePin_ReservedPins818=== CONT TestCompleteMultipartUnregistered819=== CONT TestOrphanedObjectsGCStressTest820=== CONT TestService_verifyS3ReuploadsStaleNarinfo821=== RUN TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed822=== CONT TestService_verifyS3Integrity823=== CONT TestService_cleanupPendingClosuresHandler824=== CONT TestUploadHandlersRejectOversizedBody825=== CONT TestUploadHandlersRejectInvalidKeys826=== CONT TestIsValidUploadKey827=== CONT TestPendingClosureWriteTimeout828=== CONT TestProxyWriteTimeout829=== CONT TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle830=== CONT TestSkippedUploadsHandler831=== CONT TestParseSize832=== PAUSE TestIsValidCachePath/narinfo_all_nix_base32_chars833=== PAUSE TestParseSingleRange/unknown_unit834=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info835=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info836=== RUN TestProxyWriteTimeout/narinfo837=== CONT TestPresentReportsOnlyClosureRoots838--- PASS: TestParseSize (0.00s)839=== CONT TestPresignedUploadRegisteredBeforeCommit840=== RUN TestPresentNotReportedWhileGCDeletesClosure/fresh841=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed842=== PAUSE TestProxyWriteTimeout/narinfo843=== RUN TestIsValidUploadKey/narinfo844=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal845=== RUN TestParseSingleRange/multi-range_ignored846=== RUN TestPendingClosureWriteTimeout/empty847=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/fresh848=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal849=== PAUSE TestParseSingleRange/multi-range_ignored850=== PAUSE TestPendingClosureWriteTimeout/empty851=== CONT TestCompletedNarNotReofferedAcrossClosures852=== PAUSE TestIsValidUploadKey/narinfo8532026/11/12 11:46:02 INFO Client skipped oversized paths paths=3 nar_bytes=5000000000854=== RUN TestIsValidCachePath/nar_zst855=== PAUSE TestIsValidCachePath/nar_zst856=== RUN TestIsValidCachePath/nar_xz857=== PAUSE TestIsValidCachePath/nar_xz858=== RUN TestIsValidCachePath/nar_bz2859=== RUN TestProxyWriteTimeout/1_GiB_nar860=== PAUSE TestProxyWriteTimeout/1_GiB_nar861=== RUN TestProxyWriteTimeout/10_GiB_nar862=== PAUSE TestProxyWriteTimeout/10_GiB_nar863=== RUN TestProxyWriteTimeout/unknown_size864=== PAUSE TestProxyWriteTimeout/unknown_size865=== RUN TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed866=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed867=== RUN TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost868=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost869=== RUN TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed870=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed871=== RUN TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed872=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed873=== PAUSE TestIsValidCachePath/nar_bz2874=== RUN TestIsValidCachePath/nar_uncompressed875=== PAUSE TestIsValidCachePath/nar_uncompressed876=== RUN TestIsValidCachePath/ls877=== PAUSE TestIsValidCachePath/ls878=== RUN TestIsValidCachePath/log879=== PAUSE TestIsValidCachePath/log880=== RUN TestIsValidCachePath/realisation881--- PASS: TestSkippedUploadsHandler (0.04s)882=== PAUSE TestIsValidCachePath/realisation883=== CONT TestCommitCanBeRepeatedUntilCleanup884=== RUN TestIsValidCachePath/nix-cache-info885=== PAUSE TestIsValidCachePath/nix-cache-info886=== RUN TestParseSingleRange/malformed_no_dash887=== PAUSE TestParseSingleRange/malformed_no_dash888=== RUN TestParseSingleRange/malformed_both_empty889=== PAUSE TestParseSingleRange/malformed_both_empty890=== RUN TestParseSingleRange/malformed_end_before_start891=== PAUSE TestParseSingleRange/malformed_end_before_start892=== RUN TestCommitCanBeRepeatedUntilCleanup/push893=== PAUSE TestCommitCanBeRepeatedUntilCleanup/push894=== RUN TestCommitCanBeRepeatedUntilCleanup/pending_closure895=== PAUSE TestCommitCanBeRepeatedUntilCleanup/pending_closure896=== RUN TestPendingClosureWriteTimeout/400_objects897=== PAUSE TestPendingClosureWriteTimeout/400_objects898=== RUN TestPendingClosureWriteTimeout/670k_objects899=== PAUSE TestPendingClosureWriteTimeout/670k_objects900=== RUN TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline901=== PAUSE TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline902=== CONT TestPush_SignsNarinfosOfItsPendingObjects903=== CONT TestReadProxyDisabled904=== RUN TestIsValidUploadKey/nar_zst905=== PAUSE TestIsValidUploadKey/nar_zst906=== RUN TestIsValidUploadKey/nar_xz907=== PAUSE TestIsValidUploadKey/nar_xz908=== RUN TestIsValidUploadKey/nar_plain909=== PAUSE TestIsValidUploadKey/nar_plain910=== RUN TestIsValidUploadKey/listing911=== PAUSE TestIsValidUploadKey/listing912=== RUN TestIsValidUploadKey/build_log913=== PAUSE TestIsValidUploadKey/build_log914=== RUN TestIsValidUploadKey/build_log_home-manager_file915=== PAUSE TestIsValidUploadKey/build_log_home-manager_file916=== RUN TestIsValidUploadKey/build_log_plus_in_name917=== PAUSE TestIsValidUploadKey/build_log_plus_in_name918=== RUN TestIsValidUploadKey/build_log_question_mark919=== PAUSE TestIsValidUploadKey/build_log_question_mark920=== RUN TestIsValidUploadKey/build_log_equals921=== PAUSE TestIsValidUploadKey/build_log_equals922=== RUN TestIsValidUploadKey/realisation923=== PAUSE TestIsValidUploadKey/realisation924=== RUN TestIsValidUploadKey/realisation_plus_in_output925=== PAUSE TestIsValidUploadKey/realisation_plus_in_output926=== RUN TestIsValidUploadKey/nix-cache-info927=== PAUSE TestIsValidUploadKey/nix-cache-info928=== RUN TestIsValidUploadKey/index.html929=== PAUSE TestIsValidUploadKey/index.html930=== RUN TestIsValidUploadKey/narinfo_key,_nar_type931=== PAUSE TestIsValidUploadKey/narinfo_key,_nar_type932=== RUN TestIsValidUploadKey/nar_key,_narinfo_type933=== PAUSE TestIsValidUploadKey/nar_key,_narinfo_type934=== RUN TestIsValidUploadKey/listing_key,_narinfo_type935=== PAUSE TestIsValidUploadKey/listing_key,_narinfo_type936=== RUN TestIsValidUploadKey/traversal937=== PAUSE TestIsValidUploadKey/traversal938=== RUN TestIsValidUploadKey/traversal_nar939=== PAUSE TestIsValidUploadKey/traversal_nar940=== RUN TestIsValidUploadKey/absolute941=== PAUSE TestIsValidUploadKey/absolute942=== RUN TestIsValidUploadKey/empty_key943=== PAUSE TestIsValidUploadKey/empty_key944=== RUN TestIsValidUploadKey/unknown_type945=== PAUSE TestIsValidUploadKey/unknown_type946=== RUN TestIsValidCachePath/index.html947=== PAUSE TestIsValidCachePath/index.html948=== RUN TestIsValidCachePath/traversal_parent949=== PAUSE TestIsValidCachePath/traversal_parent950=== RUN TestIsValidCachePath/traversal_in_middle951=== PAUSE TestIsValidCachePath/traversal_in_middle952=== RUN TestIsValidCachePath/invalid_char_e953=== PAUSE TestIsValidCachePath/invalid_char_e954=== RUN TestIsValidCachePath/invalid_char_u955=== PAUSE TestIsValidCachePath/invalid_char_u956=== RUN TestIsValidCachePath/random_path957=== PAUSE TestIsValidCachePath/random_path958=== RUN TestIsValidCachePath/empty959=== PAUSE TestIsValidCachePath/empty960=== RUN TestIsValidCachePath/leading_slash961=== PAUSE TestIsValidCachePath/leading_slash962=== RUN TestIsValidCachePath/wrong_extension963=== PAUSE TestIsValidCachePath/wrong_extension964=== RUN TestIsValidCachePath/short_hash965=== PAUSE TestIsValidCachePath/short_hash966=== CONT TestPush_SkippedKeySurvivesGCBeforeCommit967=== CONT TestSharedNarBothUploadsComplete968=== CONT TestPush_RootsAreLockedInByteOrder969=== RUN TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key970=== PAUSE TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key971=== CONT TestCompleteMultipartUpload_ErrorButObjectExists972=== RUN TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key973=== PAUSE TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key974=== CONT TestPush_CompleteWaitsForCleanupOfThePush975=== RUN TestParseSingleRange/closed976=== PAUSE TestParseSingleRange/closed977=== RUN TestParseSingleRange/open-ended978=== PAUSE TestParseSingleRange/open-ended979=== RUN TestParseSingleRange/end_clamped_to_size980=== PAUSE TestParseSingleRange/end_clamped_to_size981=== RUN TestParseSingleRange/suffix982=== PAUSE TestParseSingleRange/suffix983=== RUN TestParseSingleRange/suffix_exceeds_size984=== PAUSE TestParseSingleRange/suffix_exceeds_size985=== RUN TestParseSingleRange/single_byte986=== PAUSE TestParseSingleRange/single_byte987=== RUN TestParseSingleRange/start_past_EOF988=== PAUSE TestParseSingleRange/start_past_EOF989=== RUN TestParseSingleRange/start_far_past_EOF990=== PAUSE TestParseSingleRange/start_far_past_EOF991=== CONT TestCreatePendingClosureVerifyS3FailureReleasesConnection992=== RUN TestUploadHandlersRejectOversizedBody/create_pending_closure993=== PAUSE TestUploadHandlersRejectOversizedBody/create_pending_closure994=== RUN TestUploadHandlersRejectOversizedBody/complete_multipart995=== PAUSE TestUploadHandlersRejectOversizedBody/complete_multipart996=== RUN TestUploadHandlersRejectOversizedBody/request_more_parts997=== PAUSE TestUploadHandlersRejectOversizedBody/request_more_parts998=== CONT TestPush_RepeatedRootCommitsOnce9992026/11/12 11:46:02 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:51108/oidc10002026/11/12 11:46:02 INFO Received uploads request method=POST path=/api/pending_closures10012026/11/12 11:46:02 INFO Received uploads request method=POST path=/api/pending_closures10022026/11/12 11:46:02 INFO Received uploads request method=POST path=/api/pending_closures1003--- PASS: TestService_Rustfstest (0.83s)1004=== CONT TestObjectStatsUpdatedOncePerCommit10052026/11/12 11:46:03 INFO Received uploads request method=POST path=/api/pending_closures1006--- PASS: TestCreatePendingClosure_SmallNARUsesSimplePUT (1.26s)1007=== CONT TestPush_CompleteCommitsEveryRoot1008=== RUN TestPush_RejectsBadRequests/no_roots1009=== PAUSE TestPush_RejectsBadRequests/no_roots1010=== RUN TestPush_RejectsBadRequests/no_objects1011=== PAUSE TestPush_RejectsBadRequests/no_objects1012=== RUN TestPush_RejectsBadRequests/bad_root1013=== PAUSE TestPush_RejectsBadRequests/bad_root1014=== RUN TestPush_RejectsBadRequests/root_not_in_objects1015=== PAUSE TestPush_RejectsBadRequests/root_not_in_objects1016=== CONT TestObjectStatsTrigger1017--- PASS: TestPresentRefreshesOnlyStaleClosures (2.22s)1018=== CONT TestPush_OverlappingRootsStoreOneRowPerKey10192026/11/12 11:46:04 INFO Received complete multipart upload request method=POST path=/api/multipart/complete10202026/11/12 11:46:04 INFO Completed multipart upload object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst upload_id=ZjliZmY4ZjktNjNjOC00N2M0LWE2ZWEtN2VmMTBiZWZmZjU2LjhhOWUxMGRhLTQ2OGEtNGUzYy1hMTBlLTA5OTM3N2Q4Mzk5NHgxNzk0NDgzOTYzMDExMDUxMDAw parts=1010212026/11/12 11:46:04 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete10222026/11/12 11:46:04 INFO Completed upload id=1 objects=2 duration=34.698ms10232026/11/12 11:46:04 INFO Received get closure request method=GET path=/api/closures/0000000000000000000000000000000010242026/11/12 11:46:04 INFO Received uploads request method=POST path=/api/pending_closures10252026/11/12 11:46:04 INFO Starting cleanup of old closures method=DELETE path=/api/closures10262026/11/12 11:46:04 INFO Aborted multipart uploads count=0 kept=010272026/11/12 11:46:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=01028--- PASS: TestPresentReportsOnlyClosureRoots (2.67s)1029=== CONT TestReadRedirectUsesPublicS3URL10302026/11/12 11:46:05 INFO Vacuumed table table=pending_closures10312026/11/12 11:46:05 INFO Vacuumed table table=commit_receipts10322026/11/12 11:46:05 INFO Vacuumed table table=pending_objects10332026/11/12 11:46:05 INFO Vacuumed table table=multipart_uploads10342026/11/12 11:46:05 INFO Vacuumed table table=closures10352026/11/12 11:46:05 INFO Vacuumed table table=objects10362026/11/12 11:46:05 INFO Received get closure request method=GET path=/api/closures/000000000000000000000000000000001037--- PASS: TestService_createPendingClosureHandler (2.87s)1038=== CONT TestPendingCleanupUsesOneCutoff10392026/11/12 11:46:05 INFO Received cleanup request method=DELETE path=/api/pending_closures10402026/11/12 11:46:05 INFO Aborted multipart uploads count=0 kept=010412026/11/12 11:46:05 INFO Received uploads request method=POST path=/api/pending_closures10422026/11/12 11:46:05 INFO Received cleanup request method=DELETE path=/api/pending_closures10432026/11/12 11:46:05 INFO Aborted multipart uploads count=1 kept=010442026/11/12 11:46:05 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete10452026-11-12 11:46:05.891 UTC [49443] ERROR: Closure does not exist: id=110462026-11-12 11:46:05.891 UTC [49443] CONTEXT: PL/pgSQL function commit_pending_closure(bigint) line 17 at RAISE10472026-11-12 11:46:05.891 UTC [49443] STATEMENT: -- name: CommitPendingClosure :one1048 SELECT commit_pending_closure($1::bigint)::bigint1049 1050--- PASS: TestService_cleanupPendingClosuresHandler (3.56s)1051=== CONT TestReadProxyRangeRequest10522026/11/12 11:46:06 INFO Received uploads request method=POST path=/api/pending_closures10532026/11/12 11:46:06 INFO Registered completed upload object_key=nar/jjjjjjjjjjjjjjjjjjjjjjjjjjjjjj0100000000000000000000.nar.zst10542026/11/12 11:46:06 INFO Received uploads request method=POST path=/api/pending_closures1055--- PASS: TestPresignedUploadRegisteredBeforeCommit (4.07s)1056=== CONT TestMultipartUploadAbortedWhenCancelledBeforeRecorded1057=== NAME TestOrphanedObjectsGC1058 orphaned_objects_gc_test.go:287: GC Test Summary:1059 orphaned_objects_gc_test.go:288: - Kept: 2 objects from closure A1060 orphaned_objects_gc_test.go:289: - Deleted: 2 objects from closure B1061 orphaned_objects_gc_test.go:290: - Deleted: 6 orphaned chain objects (X1->X2->X3)1062 orphaned_objects_gc_test.go:291: - Deleted: 2 orphaned single objects (Y)1063 orphaned_objects_gc_test.go:292: - Total deleted: 10 objects1064--- PASS: TestOrphanedObjectsGC (4.10s)1065=== CONT TestReadRedirectKeepsNarinfoProxied10662026/11/12 11:46:06 INFO Received push request method=POST path=/api/pushes10672026/11/12 11:46:06 INFO Received complete push request method=POST path=/api/pushes/1/complete10682026/11/12 11:46:06 INFO Completed push id=1 objects=4 duration=44.644166ms10692026/11/12 11:46:06 INFO Received push request method=POST path=/api/pushes10702026/11/12 11:46:06 INFO Aborted multipart uploads count=0 kept=010712026/11/12 11:46:06 WARN Force mode enabled - objects will be deleted immediately without grace period10722026/11/12 11:46:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=2 objects-deleted-after-grace-period=2 objects-failed-to-delete=01073--- PASS: TestReadProxyDisabled (4.58s)1074=== CONT TestMultipartCleanup10752026/11/12 11:46:07 INFO Vacuumed table table=pending_closures10762026/11/12 11:46:07 INFO Vacuumed table table=commit_receipts10772026/11/12 11:46:07 INFO Vacuumed table table=pending_objects10782026/11/12 11:46:07 INFO Vacuumed table table=multipart_uploads10792026/11/12 11:46:07 INFO Vacuumed table table=closures10802026/11/12 11:46:07 INFO Vacuumed table table=objects10812026/11/12 11:46:07 INFO Received complete push request method=POST path=/api/pushes/2/complete10822026/11/12 11:46:07 INFO Completed push id=2 objects=4 duration=10.911291ms1083--- PASS: TestPush_SkippedKeySurvivesGCBeforeCommit (4.79s)1084=== CONT TestReadRedirectNar10852026/11/12 11:46:07 INFO Received uploads request method=POST path=/api/pending_closures1086--- PASS: TestCreatePendingClosureVerifyS3FailureReleasesConnection (5.13s)1087=== CONT TestServerTLSConfig1088=== RUN TestServerTLSConfig/no_client_CA1089=== PAUSE TestServerTLSConfig/no_client_CA1090=== RUN TestServerTLSConfig/missing_CA_file1091=== PAUSE TestServerTLSConfig/missing_CA_file1092=== RUN TestServerTLSConfig/not_a_PEM_file1093=== PAUSE TestServerTLSConfig/not_a_PEM_file1094=== CONT TestGCTaskStore_GetReturnsLatest1095--- PASS: TestGCTaskStore_GetReturnsLatest (0.00s)1096=== CONT TestSweepRechecksPendingAfterLocking10972026/11/12 11:46:07 INFO Received uploads request method=POST path=/api/pending_closures10982026/11/12 11:46:07 INFO Received uploads request method=POST path=/api/pending_closures10992026/11/12 11:46:08 INFO Received complete multipart upload request method=POST path=/api/multipart/complete11002026/11/12 11:46:08 INFO Received complete multipart upload request method=POST path=/api/multipart/complete11012026/11/12 11:46:08 WARN CompleteMultipartUpload errored but object exists; treating as success error="The specified key does not exist." object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=ZjliZmY4ZjktNjNjOC00N2M0LWE2ZWEtN2VmMTBiZWZmZjU2LmI0OGRhMzgyLWQyNTgtNDIzNy1hM2Q3LWViMTc3NGRiZWE0NXgxNzk0NDgzOTY3OTMwNzYzMDAw11022026/11/12 11:46:08 INFO Received uploads request method=POST path=/api/pending_closures11032026/11/12 11:46:08 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=ZjliZmY4ZjktNjNjOC00N2M0LWE2ZWEtN2VmMTBiZWZmZjU2LmI0OGRhMzgyLWQyNTgtNDIzNy1hM2Q3LWViMTc3NGRiZWE0NXgxNzk0NDgzOTY3OTMwNzYzMDAw parts=11104--- PASS: TestCompleteMultipartUpload_ErrorButObjectExists (5.90s)1105=== CONT TestGCTaskStore_CompletedAllowsNewTask1106--- PASS: TestGCTaskStore_CompletedAllowsNewTask (0.00s)1107=== CONT TestGetExistingObjectsLocksRows11082026/11/12 11:46:08 INFO Received uploads request method=POST path=/api/pending_closures11092026/11/12 11:46:08 INFO Received uploads request method=POST path=/api/pending_closures11102026/11/12 11:46:09 INFO Received push request method=POST path=/api/pushes11112026-11-12 11:46:09.366 UTC [49504] ERROR: could not obtain lock on row in relation "closures"11122026-11-12 11:46:09.366 UTC [49504] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT11132026-11-12 11:46:09.369 UTC [49504] ERROR: could not obtain lock on row in relation "closures"11142026-11-12 11:46:09.369 UTC [49504] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT11152026-11-12 11:46:09.376 UTC [49504] ERROR: could not obtain lock on row in relation "closures"11162026-11-12 11:46:09.376 UTC [49504] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT11172026-11-12 11:46:09.384 UTC [49504] ERROR: could not obtain lock on row in relation "closures"11182026-11-12 11:46:09.384 UTC [49504] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT11192026-11-12 11:46:09.385 UTC [49504] ERROR: could not obtain lock on row in relation "closures"11202026-11-12 11:46:09.385 UTC [49504] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT1121--- PASS: TestPush_RootsAreLockedInByteOrder (7.06s)1122=== CONT TestRegisterCompletedObjectNeedsPendingRow11232026/11/12 11:46:09 INFO Received push request method=POST path=/api/pushes11242026/11/12 11:46:09 INFO Received complete push request method=POST path=/api/pushes/1/complete11252026/11/12 11:46:09 INFO Completed push id=1 objects=4 duration=69.598625ms1126--- PASS: TestPush_RepeatedRootCommitsOnce (7.39s)1127=== CONT TestSweepSkipsRowsLockedByPush11282026/11/12 11:46:10 INFO Received complete multipart upload request method=POST path=/api/multipart/complete11292026/11/12 11:46:10 ERROR Failed to get multipart upload error="no rows in result set" upload_id=does-not-exist object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst1130--- PASS: TestCompleteMultipartUnregistered (7.70s)1131=== CONT TestFailedS3DeleteKeepsTombstone11322026/11/12 11:46:10 INFO Received complete multipart upload request method=POST path=/api/multipart/complete11332026/11/12 11:46:10 INFO Received push request method=POST path=/api/pushes11342026/11/12 11:46:10 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst upload_id=ZjliZmY4ZjktNjNjOC00N2M0LWE2ZWEtN2VmMTBiZWZmZjU2LmM2YjRhNjA0LTgwNDgtNGFiMi1hN2E4LTc1ZDhmNTNlZTBkOHgxNzk0NDgzOTY4MzQzOTc3MDAw parts=1011352026/11/12 11:46:10 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete11362026/11/12 11:46:10 INFO Completed upload id=1 objects=2 duration=49.328541ms11372026/11/12 11:46:10 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign11382026/11/12 11:46:10 INFO Received uploads request method=POST path=/api/pending_closures11392026/11/12 11:46:10 INFO Signed narinfos id=1 count=111402026/11/12 11:46:10 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign11412026/11/12 11:46:10 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign11422026/11/12 11:46:10 INFO Received uploads request method=POST path=/api/pending_closures11432026/11/12 11:46:10 ERROR Failed to sign narinfos signer="index 1" error="signing key cannot be nil"11442026/11/12 11:46:10 INFO Object in database but missing from S3 key=dadb44fdadb44fdadb44fdadb44f0000.narinfo11452026/11/12 11:46:10 WARN Found objects in DB but missing from S3, will re-upload count=111462026/11/12 11:46:10 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign1147--- PASS: TestService_verifyS3Integrity (8.30s)1148=== CONT TestService_NativeMTLS11492026/11/12 11:46:10 ERROR Failed to sign narinfos signer=niks3-test-1 error="signer failed"11502026/11/12 11:46:10 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign11512026/11/12 11:46:10 ERROR Signer returned unexpected signature count signer=niks3-test-1 got=0 want=111522026/11/12 11:46:10 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign11532026/11/12 11:46:10 ERROR Signer returned unexpected object key signer=niks3-test-1 object_key=unexpected.narinfo1154--- PASS: TestPush_SignsNarinfosOfItsPendingObjects (8.27s)1155=== CONT TestForceGCDuringPushOffersSweptObject1156=== RUN TestForceGCDuringPushOffersSweptObject/before_pending_rows1157=== PAUSE TestForceGCDuringPushOffersSweptObject/before_pending_rows1158=== RUN TestForceGCDuringPushOffersSweptObject/after_presence_check1159=== PAUSE TestForceGCDuringPushOffersSweptObject/after_presence_check1160=== CONT TestMetricsCommit11612026/11/12 11:46:10 INFO Received push request method=POST path=/api/pushes11622026/11/12 11:46:11 INFO Received uploads request method=POST path=/api/pending_closures11632026/11/12 11:46:11 INFO Received complete multipart upload request method=POST path=/api/multipart/complete11642026-11-12 11:46:11.218 UTC [49525] ERROR: Push does not exist: id=111652026-11-12 11:46:11.218 UTC [49525] CONTEXT: PL/pgSQL function commit_push(bigint) line 15 at RAISE11662026-11-12 11:46:11.218 UTC [49525] STATEMENT: SELECT commit_push($1)1167--- PASS: TestPush_CompleteWaitsForCleanupOfThePush (8.84s)1168=== CONT TestMetricsInventory11692026/11/12 11:46:11 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0100000000000000000000.nar.zst upload_id=ZjliZmY4ZjktNjNjOC00N2M0LWE2ZWEtN2VmMTBiZWZmZjU2LmRhYzllYjE3LWU4YmMtNDVlOC04NzcyLTkzMjEzYTBjNWI4MngxNzk0NDgzOTY4NzYyNzk1MDAw parts=1211702026/11/12 11:46:11 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux11712026/11/12 11:46:11 WARN Refused reserved pin name=worker-x86_64-linux11722026/11/12 11:46:11 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux11732026/11/12 11:46:11 INFO Received create pin request method=POST path=/api/pins/my-app11742026/11/12 11:46:11 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux1175--- PASS: TestCreatePin_ReservedPins (9.16s)1176=== CONT TestNARDeduplicationMetadataUploadBug1177--- PASS: TestObjectStatsTrigger (8.38s)1178=== CONT TestCreatePendingClosureRejectsOversizedNAR11792026/11/12 11:46:12 INFO Received uploads request method=POST path=/api/pending_closures1180--- PASS: TestCreatePendingClosureRejectsOversizedNAR (0.01s)1181=== CONT TestPushWaitsForSweepHoldingRow11822026/11/12 11:46:12 INFO Received push request method=POST path=/api/pushes11832026/11/12 11:46:12 WARN Rate limiter enabled after throttle name=s3-test rate=511842026/11/12 11:46:12 WARN S3 rate limit hit operation="complete multipart upload" error="Please reduce your request rate."1185=== NAME TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle1186 throttle_test.go:213: Proxy stats: total=13, throttled=10, completeMultipart=101187 throttle_test.go:215: Rate limiter: enabled=true, rate=5.001188--- PASS: TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle (10.09s)11892026/11/12 11:46:12 INFO Received complete push request method=POST path=/api/pushes/1/complete1190=== CONT TestCacheConfigHandlerMaxNarSize1191--- PASS: TestCacheConfigHandlerMaxNarSize (0.00s)1192=== CONT TestGCTaskStore_StartNew1193--- PASS: TestGCTaskStore_StartNew (0.00s)1194=== CONT TestEnsureAbortUploadsRule11952026/11/12 11:46:12 INFO Completed push id=1 objects=6 duration=73.69775ms1196--- PASS: TestPush_CompleteCommitsEveryRoot (8.90s)1197=== CONT TestGCTaskStore_DeduplicateSameParams1198--- PASS: TestGCTaskStore_DeduplicateSameParams (0.00s)1199=== CONT TestGCEndsOnShutdown1200=== RUN TestGCEndsOnShutdown/before_the_run1201=== PAUSE TestGCEndsOnShutdown/before_the_run1202=== RUN TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete1203=== PAUSE TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete1204=== RUN TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark1205=== PAUSE TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark1206=== CONT TestGenerateLandingPage1207--- PASS: TestGenerateLandingPage (0.00s)1208=== CONT TestGCTaskStore_GetEmpty1209--- PASS: TestGCTaskStore_GetEmpty (0.00s)1210=== CONT TestCommitRacingPendingCleanupKeepsObjects12112026/11/12 11:46:13 INFO Received push request method=POST path=/api/pushes1212--- PASS: TestObjectStatsUpdatedOncePerCommit (9.92s)1213=== CONT TestService_readinessHandler1214--- PASS: TestPush_OverlappingRootsStoreOneRowPerKey (8.70s)1215=== CONT TestGCTaskStore_ConflictDifferentParams1216--- PASS: TestGCTaskStore_ConflictDifferentParams (0.00s)1217=== CONT TestService_healthCheckHandler12182026/11/12 11:46:13 INFO Received complete multipart upload request method=POST path=/api/multipart/complete12192026/11/12 11:46:13 INFO Received complete multipart upload request method=POST path=/api/multipart/complete1220--- PASS: TestReadRedirectUsesPublicS3URL (8.57s)1221=== CONT TestGracefulShutdownDrainsInflight12222026/11/12 11:46:13 INFO Starting HTTP server address=127.0.0.1:5121112232026/11/12 11:46:13 INFO Shutdown signal received, draining in-flight requests timeout=10s12242026/11/12 11:46:13 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0100000000000000000000.nar.zst upload_id=ZjliZmY4ZjktNjNjOC00N2M0LWE2ZWEtN2VmMTBiZWZmZjU2LjUzOTQ4ZWRmLWU3NmUtNGFlMy1iOTVhLWQzYzMyOGZiNmYzNHgxNzk0NDgzOTY4ODM0NDA4MDAw parts=121225--- PASS: TestSharedNarBothUploadsComplete (11.26s)1226=== CONT TestGCTaskStore_Fail1227--- PASS: TestGCTaskStore_Fail (0.00s)1228=== CONT TestGCTaskStore_PhaseUpdates1229--- PASS: TestGCTaskStore_PhaseUpdates (0.00s)1230=== CONT TestGCAdvisoryLockBlocksConcurrentRun1231--- PASS: TestGracefulShutdownDrainsInflight (0.09s)1232=== CONT TestGCSweepDeliversEachKeyOnce12332026/11/12 11:46:13 INFO Completed multipart upload object_key=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhh0100000000000000000000.nar.zst upload_id=ZjliZmY4ZjktNjNjOC00N2M0LWE2ZWEtN2VmMTBiZWZmZjU2LmQ3OGZjNzY3LTljMzAtNDBjMy04YzRlLWExODg4YTBlMDg4YngxNzk0NDgzOTcxMjAzNDI0MDAw parts=1212342026/11/12 11:46:13 INFO Received uploads request method=POST path=/api/pending_closures1235--- PASS: TestCompletedNarNotReofferedAcrossClosures (11.36s)1236=== CONT TestTombstonedObjectOfferedWithoutWaiting12372026/11/12 11:46:13 INFO Received uploads request method=POST path=/api/pending_closures12382026/11/12 11:46:14 INFO Received uploads request method=POST path=/api/pending_closures12392026/11/12 11:46:14 INFO Received cleanup request method=DELETE path=/api/pending_closures1240--- PASS: TestReadProxyRangeRequest (8.52s)1241=== CONT TestGCSweepSkipsPendingObjects1242--- PASS: TestReadRedirectKeepsNarinfoProxied (8.29s)1243=== CONT TestDeduplicatedObjectsRecordedAsPending12442026/11/12 11:46:14 INFO Received uploads request method=POST path=/api/pending_closures1245--- PASS: TestMultipartUploadAbortedWhenCancelledBeforeRecorded (8.73s)1246=== CONT TestPushDedupSurvivesConcurrentGC12472026/11/12 11:46:15 INFO Received uploads request method=POST path=/api/pending_closures12482026/11/12 11:46:15 INFO Received cleanup request method=DELETE path=/api/pending_closures12492026/11/12 11:46:15 WARN Failed to abort upload, keeping its closure key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst error="Get \"http://127.0.0.1:1/bucket37/?location=\": dial tcp 127.0.0.1:1: connect: connection refused" code=""12502026/11/12 11:46:15 INFO Aborted multipart uploads count=0 kept=112512026/11/12 11:46:15 INFO Received cleanup request method=DELETE path=/api/pending_closures12522026/11/12 11:46:15 INFO Aborted multipart uploads count=1 kept=01253--- PASS: TestMultipartCleanup (8.69s)1254=== CONT TestGCMetrics1255--- PASS: TestReadRedirectNar (8.60s)1256=== CONT TestGCBugBareHashReferences12572026/11/12 11:46:16 INFO Aborted multipart uploads count=0 kept=012582026/11/12 11:46:16 WARN Force mode enabled - objects will be deleted immediately without grace period12592026/11/12 11:46:16 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=012602026/11/12 11:46:16 INFO Vacuumed table table=pending_closures12612026/11/12 11:46:16 INFO Vacuumed table table=commit_receipts12622026/11/12 11:46:16 INFO Vacuumed table table=pending_objects12632026/11/12 11:46:16 INFO Vacuumed table table=multipart_uploads12642026/11/12 11:46:16 INFO Vacuumed table table=closures12652026/11/12 11:46:16 INFO Vacuumed table table=objects1266--- PASS: TestSweepRechecksPendingAfterLocking (8.86s)1267=== CONT TestClientFallsBackToClosures1268--- PASS: TestGetExistingObjectsLocksRows (8.14s)1269=== CONT TestCacheStatsHandler1270--- PASS: TestRegisterCompletedObjectNeedsPendingRow (7.27s)1271=== CONT TestClientPushesUseOnePush12722026/11/12 11:46:17 INFO Aborted multipart uploads count=0 kept=012732026/11/12 11:46:17 WARN Force mode enabled - objects will be deleted immediately without grace period12742026/11/12 11:46:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=012752026/11/12 11:46:17 INFO Vacuumed table table=pending_closures12762026/11/12 11:46:17 INFO Vacuumed table table=commit_receipts12772026/11/12 11:46:17 INFO Vacuumed table table=pending_objects12782026/11/12 11:46:17 INFO Vacuumed table table=multipart_uploads12792026/11/12 11:46:17 INFO Vacuumed table table=closures12802026/11/12 11:46:17 INFO Vacuumed table table=objects12812026/11/12 11:46:17 INFO Aborted multipart uploads count=0 kept=012822026/11/12 11:46:17 WARN Force mode enabled - objects will be deleted immediately without grace period12832026/11/12 11:46:17 INFO Received uploads request method=POST path=/api/pending_closures12842026/11/12 11:46:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=012852026/11/12 11:46:17 INFO Vacuumed table table=pending_closures12862026/11/12 11:46:17 INFO Vacuumed table table=commit_receipts12872026/11/12 11:46:17 INFO Vacuumed table table=pending_objects12882026/11/12 11:46:17 INFO Vacuumed table table=multipart_uploads12892026/11/12 11:46:17 INFO Vacuumed table table=closures12902026/11/12 11:46:17 INFO Vacuumed table table=objects1291--- PASS: TestSweepSkipsRowsLockedByPush (7.81s)1292=== CONT TestPinProtectsFromGC12932026/11/12 11:46:17 INFO Aborted multipart uploads count=0 kept=012942026/11/12 11:46:17 WARN Force mode enabled - objects will be deleted immediately without grace period12952026/11/12 11:46:17 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."12962026/11/12 11:46:17 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."1297--- PASS: TestFailedS3DeleteKeepsTombstone (7.66s)1298=== CONT TestLeadElectsOneAndHandsOver12992026/11/12 11:46:17 INFO Received push request method=POST path=/api/pushes13002026/11/12 11:46:17 INFO Received complete push request method=POST path=/api/pushes/1/complete13012026/11/12 11:46:17 INFO Completed push id=1 objects=4 duration=57.523708ms13022026/11/12 11:46:17 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete13032026/11/12 11:46:17 INFO Completed upload id=2 objects=2 duration=2.404667ms1304--- PASS: TestMetricsCommit (7.33s)1305=== CONT TestLeadEndsOnShutdown13062026/11/12 11:46:18 WARN mTLS auth: subject not in bound subjects subject="CN=reader"13072026/11/12 11:46:18 WARN mTLS auth: subject not in bound subjects subject="CN=reader"1308--- PASS: TestService_NativeMTLS (7.47s)1309=== CONT TestConnectWaitsForAPeerMigration1310=== NAME TestNARDeduplicationMetadataUploadBug1311 metadata_upload_test.go:48: First store path: /private/var/tb/1/scratch/d6c7014d22e1e3f1/build/TestNARDeduplicationMetadataUploadBug3533313867/001/store/qfc4xbnh3ky28n3zs6wfq2li27zvwnl3-file1.txt1312--- PASS: TestMetricsInventory (7.74s)1313=== CONT TestResolveDBConnectionString1314=== RUN TestResolveDBConnectionString/flag_wins1315=== PAUSE TestResolveDBConnectionString/flag_wins1316=== RUN TestResolveDBConnectionString/file_when_flag_empty1317=== PAUSE TestResolveDBConnectionString/file_when_flag_empty1318=== RUN TestResolveDBConnectionString/missing_file_is_an_error1319=== PAUSE TestResolveDBConnectionString/missing_file_is_an_error1320=== RUN TestResolveDBConnectionString/PGHOST_allows_empty1321=== PAUSE TestResolveDBConnectionString/PGHOST_allows_empty1322=== RUN TestResolveDBConnectionString/nothing_configured1323=== PAUSE TestResolveDBConnectionString/nothing_configured1324=== CONT TestLeadIncumbentAnnouncesAtOnce13252026/11/12 11:46:19 INFO Received push request method=POST path=/api/pushes13262026/11/12 11:46:19 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)13272026/11/12 11:46:19 INFO Uploading qfc4xbnh3ky28n3zs6wfq2li27zvwnl3-file1.txt (160B)13282026/11/12 11:46:19 WARN Failed to register uploaded object key=nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst error="server returned 404: 404 page not found\n"13292026/11/12 11:46:19 WARN Failed to register uploaded object key=qfc4xbnh3ky28n3zs6wfq2li27zvwnl3.ls error="server returned 404: 404 page not found\n"13302026/11/12 11:46:19 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign13312026/11/12 11:46:19 INFO Signed narinfos id=1 count=113322026/11/12 11:46:19 INFO Uploading 1 narinfos13332026/11/12 11:46:19 INFO Received uploads request method=POST path=/api/pending_closures13342026/11/12 11:46:19 INFO Received complete push request method=POST path=/api/pushes/1/complete13352026/11/12 11:46:19 WARN Failed to register uploaded object key=qfc4xbnh3ky28n3zs6wfq2li27zvwnl3.narinfo error="server returned 404: 404 page not found\n"13362026/11/12 11:46:19 INFO Completed push id=1 objects=3 duration=57.311291ms13372026/11/12 11:46:19 INFO Upload complete. (399ms)1338=== NAME TestNARDeduplicationMetadataUploadBug1339 metadata_upload_test.go:54: Retrieved narinfo from S3:1340 StorePath: /private/var/tb/1/scratch/d6c7014d22e1e3f1/build/TestNARDeduplicationMetadataUploadBug3533313867/001/store/qfc4xbnh3ky28n3zs6wfq2li27zvwnl3-file1.txt1341 URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst1342 Compression: zstd1343 NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1344 NarSize: 1601345 References: 1346 CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1347 metadata_upload_test.go:55: Retrieved .ls file from S3 (compressed size: 77 bytes)1348 metadata_upload_test.go:55: Decompressed .ls content (64 bytes):1349 {"version":1,"root":{"type":"regular","size":44,"narOffset":96}}1350 metadata_upload_test.go:64: Second store path (same content): /private/var/tb/1/scratch/d6c7014d22e1e3f1/build/TestNARDeduplicationMetadataUploadBug3533313867/001/store/6yzm6rw65d84g0x8mz901px1qxm2brdf-file2.txt1351=== NAME TestOrphanedObjectsGCStressTest1352 orphaned_objects_gc_test.go:422: Created 10 active closures, 5 to-delete closures, 20 orphaned chains1353 orphaned_objects_gc_test.go:440: Marked 210 objects for deletion13542026/11/12 11:46:19 INFO Received push request method=POST path=/api/pushes13552026/11/12 11:46:19 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)13562026/11/12 11:46:19 INFO Installed lifecycle rule aborting incomplete NAR uploads bucket=bucket49 days=21357--- PASS: TestEnsureAbortUploadsRule (7.29s)1358=== CONT TestConcurrentCommitsSharingObjectsDoNotDeadlock13592026/11/12 11:46:19 INFO Received sign narinfos request method=POST path=/api/pushes/2/sign13602026/11/12 11:46:19 WARN Failed to register uploaded object key=6yzm6rw65d84g0x8mz901px1qxm2brdf.ls error="server returned 404: 404 page not found\n"13612026/11/12 11:46:19 INFO Signed narinfos id=2 count=113622026/11/12 11:46:19 INFO Uploading 1 narinfos1363--- PASS: TestPushWaitsForSweepHoldingRow (7.70s)1364=== CONT TestLeadUnannouncedStandbyYieldsToIncumbent13652026/11/12 11:46:19 INFO Received complete push request method=POST path=/api/pushes/2/complete13662026/11/12 11:46:19 WARN Failed to register uploaded object key=6yzm6rw65d84g0x8mz901px1qxm2brdf.narinfo error="server returned 404: 404 page not found\n"13672026/11/12 11:46:19 INFO Completed push id=2 objects=3 duration=787.667µs13682026/11/12 11:46:19 INFO Upload complete. (190ms)1369=== NAME TestNARDeduplicationMetadataUploadBug1370 metadata_upload_test.go:76: Retrieved narinfo from S3:1371 StorePath: /private/var/tb/1/scratch/d6c7014d22e1e3f1/build/TestNARDeduplicationMetadataUploadBug3533313867/001/store/6yzm6rw65d84g0x8mz901px1qxm2brdf-file2.txt1372 URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst1373 Compression: zstd1374 NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1375 NarSize: 1601376 References: 1377 CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1378 metadata_upload_test.go:77: Retrieved .ls file from S3 (compressed size: 62 bytes)1379 metadata_upload_test.go:77: Decompressed .ls content (49 bytes):1380 {"version":1,"root":{"type":"regular","size":44}}1381--- PASS: TestNARDeduplicationMetadataUploadBug (8.43s)1382=== CONT TestDeleteClosuresRacingPins13832026/11/12 11:46:20 INFO Aborted multipart uploads count=1 kept=013842026/11/12 11:46:20 INFO Aborted multipart uploads count=0 kept=013852026/11/12 11:46:20 WARN Force mode enabled - objects will be deleted immediately without grace period13862026/11/12 11:46:20 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=013872026/11/12 11:46:20 INFO Received cleanup request method=DELETE path=/api/pending_closures13882026/11/12 11:46:20 INFO Aborted multipart uploads count=1 kept=01389--- PASS: TestPendingCleanupUsesOneCutoff (14.99s)1390=== CONT TestReadProxyNarinfoAlreadyDecompressed13912026/11/12 11:46:20 INFO Vacuumed table table=pending_closures13922026/11/12 11:46:20 INFO Vacuumed table table=commit_receipts13932026/11/12 11:46:20 INFO Vacuumed table table=pending_objects13942026/11/12 11:46:20 INFO Vacuumed table table=multipart_uploads13952026/11/12 11:46:20 INFO Vacuumed table table=closures13962026/11/12 11:46:20 WARN readiness check failed error="closed pool"13972026/11/12 11:46:20 INFO Vacuumed table table=objects1398--- PASS: TestService_readinessHandler (7.18s)1399=== CONT TestReadProxy4041400--- PASS: TestCommitRacingPendingCleanupKeepsObjects (7.81s)1401=== CONT TestReadProxyRootRedirectsToIndexHTML1402--- PASS: TestService_healthCheckHandler (7.36s)1403=== CONT TestReadProxyNarStreaming14042026/11/12 11:46:21 ERROR failed to check GC advisory lock error="query pg_locks: failed to connect to `user=_tribuchetbld1 database=niks3`: /nonexistent/.s.PGSQL.5432 (/nonexistent): dial error: dial unix /nonexistent/.s.PGSQL.5432: connect: no such file or directory"1405--- PASS: TestGCAdvisoryLockBlocksConcurrentRun (7.42s)1406=== CONT TestReadProxyConditionalGet14072026/11/12 11:46:21 INFO Aborted multipart uploads count=0 kept=014082026/11/12 11:46:21 WARN Force mode enabled - objects will be deleted immediately without grace period14092026/11/12 11:46:21 INFO Received uploads request method=POST path=/api/pending_closures1410--- PASS: TestTombstonedObjectOfferedWithoutWaiting (8.27s)1411=== CONT TestReadProxyContentNegotiation14122026/11/12 11:46:22 INFO Received uploads request method=POST path=/api/pending_closures14132026/11/12 11:46:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=014142026/11/12 11:46:22 INFO Vacuumed table table=pending_closures14152026/11/12 11:46:22 INFO Vacuumed table table=commit_receipts14162026/11/12 11:46:22 INFO Vacuumed table table=pending_objects14172026/11/12 11:46:22 INFO Vacuumed table table=multipart_uploads14182026/11/12 11:46:22 INFO Vacuumed table table=closures14192026/11/12 11:46:22 INFO Vacuumed table table=objects1420--- PASS: TestGCSweepDeliversEachKeyOnce (8.85s)1421=== CONT TestReadProxyOutlastsServerWriteTimeout14222026/11/12 11:46:22 INFO Aborted multipart uploads count=0 kept=014232026/11/12 11:46:22 WARN Force mode enabled - objects will be deleted immediately without grace period14242026/11/12 11:46:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=014252026/11/12 11:46:22 INFO Vacuumed table table=pending_closures14262026/11/12 11:46:22 INFO Vacuumed table table=commit_receipts14272026/11/12 11:46:22 INFO Vacuumed table table=pending_objects14282026/11/12 11:46:22 INFO Vacuumed table table=multipart_uploads14292026/11/12 11:46:22 INFO Vacuumed table table=closures14302026/11/12 11:46:22 INFO Vacuumed table table=objects1431--- PASS: TestGCSweepSkipsPendingObjects (8.46s)1432=== CONT TestClientSharedPathCommittedMidPush14332026/11/12 11:46:22 INFO Received uploads request method=POST path=/api/pending_closures1434--- PASS: TestDeduplicatedObjectsRecordedAsPending (8.22s)1435=== CONT TestClientIntegration14362026/11/12 11:46:23 INFO Received uploads request method=POST path=/api/pending_closures14372026/11/12 11:46:23 INFO Aborted multipart uploads count=0 kept=014382026/11/12 11:46:23 INFO Aborted multipart uploads count=0 kept=014392026/11/12 11:46:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=014402026/11/12 11:46:23 WARN Force mode enabled - objects will be deleted immediately without grace period14412026/11/12 11:46:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=014422026/11/12 11:46:23 INFO Vacuumed table table=pending_closures14432026/11/12 11:46:23 INFO Vacuumed table table=pending_closures14442026/11/12 11:46:23 INFO Vacuumed table table=commit_receipts14452026/11/12 11:46:23 INFO Vacuumed table table=pending_objects14462026/11/12 11:46:23 INFO Vacuumed table table=multipart_uploads14472026/11/12 11:46:23 INFO Vacuumed table table=closures14482026/11/12 11:46:23 INFO Vacuumed table table=objects14492026/11/12 11:46:23 INFO Vacuumed table table=commit_receipts1450--- PASS: TestGCMetrics (8.07s)1451=== CONT TestReadProxyHead14522026/11/12 11:46:23 INFO Vacuumed table table=pending_objects14532026/11/12 11:46:23 INFO Vacuumed table table=multipart_uploads14542026/11/12 11:46:23 INFO Vacuumed table table=closures14552026/11/12 11:46:23 INFO Vacuumed table table=objects14562026/11/12 11:46:23 INFO Aborted multipart uploads count=0 kept=014572026/11/12 11:46:23 WARN Force mode enabled - objects will be deleted immediately without grace period14582026/11/12 11:46:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=014592026/11/12 11:46:23 INFO Vacuumed table table=pending_closures14602026/11/12 11:46:23 INFO Vacuumed table table=commit_receipts14612026/11/12 11:46:23 INFO Vacuumed table table=pending_objects14622026/11/12 11:46:23 INFO Vacuumed table table=multipart_uploads14632026/11/12 11:46:23 INFO Vacuumed table table=closures14642026/11/12 11:46:23 INFO Vacuumed table table=objects1465--- PASS: TestPushDedupSurvivesConcurrentGC (8.78s)1466=== CONT TestClientErrorHandling1467=== RUN TestClientErrorHandling/InvalidStorePath1468=== PAUSE TestClientErrorHandling/InvalidStorePath1469=== RUN TestClientErrorHandling/InvalidAuthToken1470=== PAUSE TestClientErrorHandling/InvalidAuthToken1471=== RUN TestClientErrorHandling/ServerNotAvailable1472=== PAUSE TestClientErrorHandling/ServerNotAvailable1473=== CONT TestClientWithDependencies1474--- PASS: TestGCBugBareHashReferences (8.49s)1475=== CONT TestReadProxyInvalidPath1476--- PASS: TestCacheStatsHandler (8.21s)1477=== CONT TestClientMultipleUploads14782026/11/12 11:46:24 INFO Received uploads request method=POST path=/api/pending_closures14792026/11/12 11:46:24 INFO Received uploads request method=POST path=/api/pending_closures14802026/11/12 11:46:24 INFO Uploading 2 paths to 127.0.0.1 (1 already cached)14812026/11/12 11:46:24 INFO Uploading jzvihp064mmd89a05gpb1755pggf1fn0-b (256B)14822026/11/12 11:46:24 INFO Uploading hl0zzyk20p9rsc7wis79wzq14n43ryfp-shared-dep (136B)14832026/11/12 11:46:24 WARN Failed to register uploaded object key=yzns3ngqvi21v92nxvl3amha0s3v0msx.ls error="server returned 404: 404 page not found\n"14842026/11/12 11:46:24 WARN Failed to register uploaded object key=nar/1fbvx1bv55gm78gc6wzj3ijynry2lsvkbkyxwk0gggp23mbkwm32.nar.zst error="server returned 404: 404 page not found\n"14852026/11/12 11:46:24 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"14862026/11/12 11:46:24 WARN Failed to register uploaded object key=jzvihp064mmd89a05gpb1755pggf1fn0.ls error="server returned 404: 404 page not found\n"14872026/11/12 11:46:24 WARN Failed to register uploaded object key=hl0zzyk20p9rsc7wis79wzq14n43ryfp.ls error="server returned 404: 404 page not found\n"14882026/11/12 11:46:24 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign14892026/11/12 11:46:24 INFO Signed narinfos id=1 count=214902026/11/12 11:46:24 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign14912026/11/12 11:46:24 INFO Signed narinfos id=2 count=214922026/11/12 11:46:24 INFO Uploading 4 narinfos14932026/11/12 11:46:25 WARN Failed to register uploaded object key=hl0zzyk20p9rsc7wis79wzq14n43ryfp.narinfo error="server returned 404: 404 page not found\n"14942026/11/12 11:46:25 WARN Failed to register uploaded object key=yzns3ngqvi21v92nxvl3amha0s3v0msx.narinfo error="server returned 404: 404 page not found\n"14952026/11/12 11:46:25 WARN Failed to register uploaded object key=jzvihp064mmd89a05gpb1755pggf1fn0.narinfo error="server returned 404: 404 page not found\n"14962026/11/12 11:46:25 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete14972026/11/12 11:46:25 WARN Failed to register uploaded object key=hl0zzyk20p9rsc7wis79wzq14n43ryfp.narinfo error="server returned 404: 404 page not found\n"14982026/11/12 11:46:25 INFO Completed upload id=1 objects=6 duration=51.213666ms14992026/11/12 11:46:25 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete15002026/11/12 11:46:25 INFO Completed upload id=2 objects=6 duration=848.875µs15012026/11/12 11:46:25 INFO Upload complete. (375ms)1502=== NAME TestClientFallsBackToClosures1503 client_pushes_test.go:112: Retrieved narinfo from S3:1504 StorePath: /private/var/tb/1/scratch/d6c7014d22e1e3f1/build/TestClientFallsBackToClosures1341439402/001/store/hl0zzyk20p9rsc7wis79wzq14n43ryfp-shared-dep1505 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1506 Compression: zstd1507 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821508 NarSize: 1361509 References: 1510 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n1511 client_pushes_test.go:112: Retrieved narinfo from S3:1512 StorePath: /private/var/tb/1/scratch/d6c7014d22e1e3f1/build/TestClientFallsBackToClosures1341439402/001/store/yzns3ngqvi21v92nxvl3amha0s3v0msx-a1513 URL: nar/1fbvx1bv55gm78gc6wzj3ijynry2lsvkbkyxwk0gggp23mbkwm32.nar.zst1514 Compression: zstd1515 NarHash: sha256:1fbvx1bv55gm78gc6wzj3ijynry2lsvkbkyxwk0gggp23mbkwm321516 NarSize: 2561517 References: /private/var/tb/1/scratch/d6c7014d22e1e3f1/build/TestClientFallsBackToClosures1341439402/001/store/hl0zzyk20p9rsc7wis79wzq14n43ryfp-shared-dep1518 CA: text:sha256:0dcb0dil5qv13jimxc14h2j7n91vmz8gq0wi5m045m1zg6gkrnqa1519 client_pushes_test.go:112: Retrieved narinfo from S3:1520 StorePath: /private/var/tb/1/scratch/d6c7014d22e1e3f1/build/TestClientFallsBackToClosures1341439402/001/store/jzvihp064mmd89a05gpb1755pggf1fn0-b1521 URL: nar/1fbvx1bv55gm78gc6wzj3ijynry2lsvkbkyxwk0gggp23mbkwm32.nar.zst1522 Compression: zstd1523 NarHash: sha256:1fbvx1bv55gm78gc6wzj3ijynry2lsvkbkyxwk0gggp23mbkwm321524 NarSize: 2561525 References: /private/var/tb/1/scratch/d6c7014d22e1e3f1/build/TestClientFallsBackToClosures1341439402/001/store/hl0zzyk20p9rsc7wis79wzq14n43ryfp-shared-dep1526 CA: text:sha256:0dcb0dil5qv13jimxc14h2j7n91vmz8gq0wi5m045m1zg6gkrnqa1527--- PASS: TestClientFallsBackToClosures (8.90s)1528=== CONT TestService_AuthMiddleware_MTLSBoundSubjects15292026/11/12 11:46:25 INFO lead: acquired remote=192.0.2.1:123415302026/11/12 11:46:25 INFO Received push request method=POST path=/api/pushes15312026/11/12 11:46:25 INFO Uploading 2 paths to 127.0.0.1 (1 already cached)1532=== NAME TestPinProtectsFromGC15332026/11/12 11:46:25 INFO Uploading mhpalnhy7jl2sbvqfs7hv90r4yk60jfr-shared-dep (136B)1534 client_integration_test.go:791: Pinned store path: /private/var/tb/1/scratch/d6c7014d22e1e3f1/build/TestPinProtectsFromGC2178966503/001/store/fv4nal9gmi559lvlk1k49knih5x4i6qq-pinned-file.txt1535 client_integration_test.go:792: Unpinned store path: /private/var/tb/1/scratch/d6c7014d22e1e3f1/build/TestPinProtectsFromGC2178966503/001/store/2j8kfdlwfnwcyb2lvysvssym2haj8s3w-unpinned-file.txt15362026/11/12 11:46:25 INFO Uploading swfnmksaag5zh9xn9p7l4zn7643jvilj-b (256B)15372026/11/12 11:46:25 WARN Failed to register uploaded object key=mhpalnhy7jl2sbvqfs7hv90r4yk60jfr.ls error="server returned 404: 404 page not found\n"15382026/11/12 11:46:25 WARN Failed to register uploaded object key=axj1f5c1hzfr48faab4z55z84yy777zf.ls error="server returned 404: 404 page not found\n"15392026/11/12 11:46:25 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"15402026/11/12 11:46:25 WARN Failed to register uploaded object key=swfnmksaag5zh9xn9p7l4zn7643jvilj.ls error="server returned 404: 404 page not found\n"15412026/11/12 11:46:25 WARN Failed to register uploaded object key=nar/1pfxwi09dayjvnn62mac68n5fily9yvdd61slmw191bgk4z9p3cm.nar.zst error="server returned 404: 404 page not found\n"15422026/11/12 11:46:25 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign15432026/11/12 11:46:25 INFO Signed narinfos id=1 count=315442026/11/12 11:46:25 INFO Uploading 3 narinfos15452026/11/12 11:46:25 INFO Received push request method=POST path=/api/pushes15462026/11/12 11:46:25 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)15472026/11/12 11:46:25 INFO Uploading fv4nal9gmi559lvlk1k49knih5x4i6qq-pinned-file.txt (128B)15482026/11/12 11:46:25 WARN Failed to register uploaded object key=mhpalnhy7jl2sbvqfs7hv90r4yk60jfr.narinfo error="server returned 404: 404 page not found\n"15492026/11/12 11:46:25 WARN Failed to register uploaded object key=swfnmksaag5zh9xn9p7l4zn7643jvilj.narinfo error="server returned 404: 404 page not found\n"15502026/11/12 11:46:25 INFO Received complete push request method=POST path=/api/pushes/1/complete15512026/11/12 11:46:25 WARN Failed to register uploaded object key=axj1f5c1hzfr48faab4z55z84yy777zf.narinfo error="server returned 404: 404 page not found\n"15522026/11/12 11:46:25 WARN Failed to register uploaded object key=fv4nal9gmi559lvlk1k49knih5x4i6qq.ls error="server returned 404: 404 page not found\n"15532026/11/12 11:46:25 INFO Completed push id=1 objects=8 duration=50.84125ms15542026/11/12 11:46:25 INFO Upload complete. (347ms)1555=== NAME TestClientPushesUseOnePush1556 client_pushes_test.go:97: Retrieved narinfo from S3:1557 StorePath: /private/var/tb/1/scratch/d6c7014d22e1e3f1/build/TestClientPushesUseOnePush4064559449/001/store/mhpalnhy7jl2sbvqfs7hv90r4yk60jfr-shared-dep1558 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1559 Compression: zstd1560 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y8215612026/11/12 11:46:25 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign1562 NarSize: 1361563 References: 1564 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n15652026/11/12 11:46:25 WARN Failed to register uploaded object key=nar/0qmhsg42pqsz7x6hh02j8gznnac3yi3qpjhrqbwvz88i194jbb5s.nar.zst error="server returned 404: 404 page not found\n"15662026/11/12 11:46:25 INFO Signed narinfos id=1 count=11567 client_pushes_test.go:97: Retrieved narinfo from S3:1568 StorePath: /private/var/tb/1/scratch/d6c7014d22e1e3f1/build/TestClientPushesUseOnePush4064559449/001/store/axj1f5c1hzfr48faab4z55z84yy777zf-a1569 URL: nar/1pfxwi09dayjvnn62mac68n5fily9yvdd61slmw191bgk4z9p3cm.nar.zst1570 Compression: zstd1571 NarHash: sha256:1pfxwi09dayjvnn62mac68n5fily9yvdd61slmw191bgk4z9p3cm1572 NarSize: 2561573 References: /private/var/tb/1/scratch/d6c7014d22e1e3f1/build/TestClientPushesUseOnePush4064559449/001/store/mhpalnhy7jl2sbvqfs7hv90r4yk60jfr-shared-dep1574 CA: text:sha256:0knspqap7v4mzp7bm0vsjvql8mwdzrrhw7lq4iqyws2mzvrx8y1r15752026/11/12 11:46:25 INFO Uploading 1 narinfos1576 client_pushes_test.go:97: Retrieved narinfo from S3:1577 StorePath: /private/var/tb/1/scratch/d6c7014d22e1e3f1/build/TestClientPushesUseOnePush4064559449/001/store/swfnmksaag5zh9xn9p7l4zn7643jvilj-b1578 URL: nar/1pfxwi09dayjvnn62mac68n5fily9yvdd61slmw191bgk4z9p3cm.nar.zst1579 Compression: zstd1580 NarHash: sha256:1pfxwi09dayjvnn62mac68n5fily9yvdd61slmw191bgk4z9p3cm1581 NarSize: 2561582 References: /private/var/tb/1/scratch/d6c7014d22e1e3f1/build/TestClientPushesUseOnePush4064559449/001/store/mhpalnhy7jl2sbvqfs7hv90r4yk60jfr-shared-dep1583 CA: text:sha256:0knspqap7v4mzp7bm0vsjvql8mwdzrrhw7lq4iqyws2mzvrx8y1r15842026/11/12 11:46:25 INFO lead: acquired remote=192.0.2.1:123415852026/11/12 11:46:25 INFO Received complete push request method=POST path=/api/pushes/1/complete15862026/11/12 11:46:25 WARN Failed to register uploaded object key=fv4nal9gmi559lvlk1k49knih5x4i6qq.narinfo error="server returned 404: 404 page not found\n"15872026/11/12 11:46:25 INFO Completed push id=1 objects=3 duration=40.628916ms15882026/11/12 11:46:25 INFO Upload complete. (332ms)1589--- PASS: TestClientPushesUseOnePush (9.26s)1590=== CONT TestService_ReadAuthMiddleware15912026/11/12 11:46:26 INFO Received push request method=POST path=/api/pushes15922026/11/12 11:46:26 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)15932026/11/12 11:46:26 INFO Uploading 2j8kfdlwfnwcyb2lvysvssym2haj8s3w-unpinned-file.txt (128B)15942026/11/12 11:46:26 WARN Failed to register uploaded object key=2j8kfdlwfnwcyb2lvysvssym2haj8s3w.ls error="server returned 404: 404 page not found\n"15952026/11/12 11:46:26 INFO Received sign narinfos request method=POST path=/api/pushes/2/sign15962026/11/12 11:46:26 WARN Failed to register uploaded object key=nar/0gh0wdnr6gmk4r5sfzzh12isnl14rgmkrwy1ws41hpsrlndclg5h.nar.zst error="server returned 404: 404 page not found\n"15972026/11/12 11:46:26 INFO Signed narinfos id=2 count=115982026/11/12 11:46:26 INFO Uploading 1 narinfos15992026/11/12 11:46:26 INFO Received complete push request method=POST path=/api/pushes/2/complete16002026/11/12 11:46:26 WARN Failed to register uploaded object key=2j8kfdlwfnwcyb2lvysvssym2haj8s3w.narinfo error="server returned 404: 404 page not found\n"16012026/11/12 11:46:26 INFO Completed push id=2 objects=3 duration=749.959µs16022026/11/12 11:46:26 INFO Upload complete. (192ms)16032026/11/12 11:46:26 INFO lead: acquired remote=192.0.2.1:123416042026/11/12 11:46:26 INFO lead: released remote=192.0.2.1:12341605--- PASS: TestLeadIncumbentAnnouncesAtOnce (7.27s)1606=== CONT TestService_AuthMiddleware16072026/11/12 11:46:26 INFO Starting cleanup of old closures method=DELETE path=/api/closures16082026/11/12 11:46:26 INFO Garbage collection started16092026/11/12 11:46:26 INFO Aborted multipart uploads count=0 kept=016102026/11/12 11:46:26 WARN Force mode enabled - objects will be deleted immediately without grace period16112026/11/12 11:46:26 INFO lead: acquired remote=192.0.2.1:123416122026/11/12 11:46:26 INFO lead: released remote=192.0.2.1:123416132026/11/12 11:46:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=016142026/11/12 11:46:26 INFO lead: acquired remote=192.0.2.1:123416152026/11/12 11:46:26 INFO Vacuumed table table=pending_closures16162026/11/12 11:46:26 INFO Vacuumed table table=commit_receipts16172026/11/12 11:46:26 INFO Vacuumed table table=pending_objects16182026/11/12 11:46:27 INFO Vacuumed table table=multipart_uploads16192026/11/12 11:46:27 INFO lead: released remote=192.0.2.1:12341620--- PASS: TestLeadEndsOnShutdown (9.04s)1621=== CONT TestService_AuthMiddleware_MTLSProxyHeader16222026/11/12 11:46:27 INFO Vacuumed table table=closures1623--- PASS: TestConnectWaitsForAPeerMigration (8.94s)1624=== CONT TestClientCADerivations16252026/11/12 11:46:27 INFO Vacuumed table table=objects16262026/11/12 11:46:27 INFO lead: released remote=192.0.2.1:123416272026/11/12 11:46:27 INFO lead: acquired remote=192.0.2.1:12341628--- PASS: TestReadProxyNarinfoAlreadyDecompressed (7.53s)1629=== CONT TestReadProxyNarinfo16302026/11/12 11:46:27 INFO lead: given back to the incumbent remote=192.0.2.1:123416312026/11/12 11:46:27 INFO lead: acquired remote=192.0.2.1:123416322026/11/12 11:46:27 INFO lead: released remote=192.0.2.1:12341633--- PASS: TestLeadElectsOneAndHandsOver (10.22s)1634=== CONT TestService_AuthMiddleware_OIDC16352026/11/12 11:46:28 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:51371/oidc16362026/11/12 11:46:28 INFO Garbage collection progress phase="" failed_uploads_deleted=0 old_closures_deleted=1 objects_marked=3 objects_deleted=3 objects_failed=016372026/11/12 11:46:28 INFO Received create pin request method=POST path=/api/pins/myapp16382026/11/12 11:46:28 INFO Created/updated pin name=myapp store_path=/private/var/tb/1/scratch/d6c7014d22e1e3f1/build/TestPinProtectsFromGC2178966503/001/store/fv4nal9gmi559lvlk1k49knih5x4i6qq-pinned-file.txt narinfo_key=fv4nal9gmi559lvlk1k49knih5x4i6qq.narinfo16392026/11/12 11:46:28 INFO Starting cleanup of old closures method=DELETE path=/api/closures16402026/11/12 11:46:28 INFO Garbage collection started16412026/11/12 11:46:28 INFO Aborted multipart uploads count=0 kept=016422026/11/12 11:46:28 WARN Force mode enabled - objects will be deleted immediately without grace period1643--- PASS: TestReadProxyRootRedirectsToIndexHTML (8.03s)16442026/11/12 11:46:28 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=01645=== CONT TestService_ReadScope_PublicByDefault16462026/11/12 11:46:28 INFO Vacuumed table table=pending_closures16472026/11/12 11:46:28 INFO Vacuumed table table=commit_receipts16482026/11/12 11:46:28 INFO Vacuumed table table=pending_objects16492026/11/12 11:46:28 INFO Vacuumed table table=multipart_uploads16502026/11/12 11:46:28 INFO Vacuumed table table=closures16512026/11/12 11:46:28 INFO Vacuumed table table=objects1652--- PASS: TestConcurrentCommitsSharingObjectsDoNotDeadlock (8.81s)1653=== CONT TestCacheConfigHandler1654=== RUN TestCacheConfigHandler/full_config,_no_issuer1655=== PAUSE TestCacheConfigHandler/full_config,_no_issuer1656=== RUN TestCacheConfigHandler/no_cache_url_configured1657=== PAUSE TestCacheConfigHandler/no_cache_url_configured1658=== RUN TestCacheConfigHandler/no_signing_keys1659=== PAUSE TestCacheConfigHandler/no_signing_keys1660=== RUN TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1661=== PAUSE TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1662=== CONT TestService_RequireScope_OIDC16632026/11/12 11:46:28 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:51379/oidc1664--- PASS: TestReadProxy404 (8.35s)1665=== CONT TestPresentNotReportedWhileGCDeletesClosure/stale1666--- PASS: TestReadProxyNarStreaming (8.15s)1667=== CONT TestPresentNotReportedWhileGCDeletesClosure/fresh1668--- PASS: TestReadProxyConditionalGet (8.02s)1669=== CONT TestProxyWriteTimeout/narinfo1670=== CONT TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed1671--- PASS: TestReadProxyContentNegotiation (7.58s)1672=== CONT TestProxyWriteTimeout/unknown_size1673=== CONT TestProxyWriteTimeout/10_GiB_nar1674=== CONT TestProxyWriteTimeout/1_GiB_nar1675--- PASS: TestProxyWriteTimeout (0.04s)1676 --- PASS: TestProxyWriteTimeout/narinfo (0.00s)1677 --- PASS: TestProxyWriteTimeout/unknown_size (0.00s)1678 --- PASS: TestProxyWriteTimeout/10_GiB_nar (0.00s)1679 --- PASS: TestProxyWriteTimeout/1_GiB_nar (0.00s)1680=== CONT TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed16812026/11/12 11:46:29 INFO lead: released remote=192.0.2.1:12341682--- PASS: TestLeadUnannouncedStandbyYieldsToIncumbent (9.87s)1683=== CONT TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed1684=== NAME TestPinProtectsFromGC1685 client_integration_test.go:900: Pin successfully protected closure from garbage collection1686--- PASS: TestPinProtectsFromGC (12.87s)1687=== CONT TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost16882026/11/12 11:46:30 INFO Received push request method=POST path=/api/pushes16892026/11/12 11:46:30 INFO Uploading 2 paths to 127.0.0.1 (0 already cached)16902026/11/12 11:46:30 INFO Uploading syikkcikpddz281by6h7141gf650j66y-shared-dep (136B)16912026/11/12 11:46:30 INFO Uploading gvajy9fg2hgzls84mdfwz0fa3gb65ydp-top (264B)16922026/11/12 11:46:30 WARN Failed to register uploaded object key=syikkcikpddz281by6h7141gf650j66y.ls error="server returned 404: 404 page not found\n"16932026/11/12 11:46:30 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"16942026/11/12 11:46:30 WARN Failed to register uploaded object key=nar/1sayhw3kz2n3fprkrg9pn96qsmswrdb0klaa403q1r8zn8ch1ih1.nar.zst error="server returned 404: 404 page not found\n"16952026/11/12 11:46:30 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign16962026/11/12 11:46:30 WARN Failed to register uploaded object key=gvajy9fg2hgzls84mdfwz0fa3gb65ydp.ls error="server returned 404: 404 page not found\n"16972026/11/12 11:46:30 INFO Signed narinfos id=1 count=216982026/11/12 11:46:30 INFO Uploading 2 narinfos16992026/11/12 11:46:30 WARN Failed to register uploaded object key=syikkcikpddz281by6h7141gf650j66y.narinfo error="server returned 404: 404 page not found\n"17002026/11/12 11:46:30 INFO Received complete push request method=POST path=/api/pushes/1/complete17012026/11/12 11:46:30 WARN Failed to register uploaded object key=gvajy9fg2hgzls84mdfwz0fa3gb65ydp.narinfo error="server returned 404: 404 page not found\n"1702=== NAME TestClientIntegration1703 client_integration_test.go:292: Created store path: /private/var/tb/1/scratch/d6c7014d22e1e3f1/build/TestClientIntegration3584928457/002/store/nikpix870b47hz0bw0z7rxqdqglrd4n2-test-file.txt1704--- PASS: TestReadProxyHead (7.27s)1705=== CONT TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed17062026/11/12 11:46:31 INFO Completed push id=1 objects=6 duration=36.087084ms17072026/11/12 11:46:31 INFO Upload complete. (320ms)1708=== NAME TestClientSharedPathCommittedMidPush1709 client_integration_test.go:740: Retrieved narinfo from S3:1710 StorePath: /private/var/tb/1/scratch/d6c7014d22e1e3f1/build/TestClientSharedPathCommittedMidPush2710623467/001/store/syikkcikpddz281by6h7141gf650j66y-shared-dep1711 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1712 Compression: zstd1713 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821714 NarSize: 1361715 References: 1716 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n1717 client_integration_test.go:740: Retrieved narinfo from S3:1718 StorePath: /private/var/tb/1/scratch/d6c7014d22e1e3f1/build/TestClientSharedPathCommittedMidPush2710623467/001/store/gvajy9fg2hgzls84mdfwz0fa3gb65ydp-top1719 URL: nar/1sayhw3kz2n3fprkrg9pn96qsmswrdb0klaa403q1r8zn8ch1ih1.nar.zst1720 Compression: zstd1721 NarHash: sha256:1sayhw3kz2n3fprkrg9pn96qsmswrdb0klaa403q1r8zn8ch1ih11722 NarSize: 2641723 References: /private/var/tb/1/scratch/d6c7014d22e1e3f1/build/TestClientSharedPathCommittedMidPush2710623467/001/store/syikkcikpddz281by6h7141gf650j66y-shared-dep1724 CA: text:sha256:0pbfrdpb2w0bkdwj7p81w8qvl2xjfnslklsk6ay41rx4fgv7hdbm1725--- PASS: TestClientSharedPathCommittedMidPush (8.18s)1726=== CONT TestCommitCanBeRepeatedUntilCleanup/push17272026/11/12 11:46:31 INFO Received push request method=POST path=/api/pushes17282026/11/12 11:46:31 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)17292026/11/12 11:46:31 INFO Uploading nikpix870b47hz0bw0z7rxqdqglrd4n2-test-file.txt (152B)17302026/11/12 11:46:31 WARN Failed to register uploaded object key=nikpix870b47hz0bw0z7rxqdqglrd4n2.ls error="server returned 404: 404 page not found\n"17312026/11/12 11:46:31 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign17322026/11/12 11:46:31 WARN Failed to register uploaded object key=nar/11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1.nar.zst error="server returned 404: 404 page not found\n"17332026/11/12 11:46:31 INFO Signed narinfos id=1 count=117342026/11/12 11:46:31 INFO Uploading 1 narinfos17352026/11/12 11:46:31 INFO Received complete push request method=POST path=/api/pushes/1/complete17362026/11/12 11:46:31 WARN Failed to register uploaded object key=nikpix870b47hz0bw0z7rxqdqglrd4n2.narinfo error="server returned 404: 404 page not found\n"17372026/11/12 11:46:31 INFO Completed push id=1 objects=3 duration=40.1745ms17382026/11/12 11:46:31 INFO Upload complete. (354ms)17392026/11/12 11:46:31 INFO All 1 paths already cached1740=== NAME TestClientIntegration1741 client_integration_test.go:318: Retrieved narinfo from S3:1742 StorePath: /private/var/tb/1/scratch/d6c7014d22e1e3f1/build/TestClientIntegration3584928457/002/store/nikpix870b47hz0bw0z7rxqdqglrd4n2-test-file.txt1743 URL: nar/11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1.nar.zst1744 Compression: zstd1745 NarHash: sha256:11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk11746 NarSize: 1521747 References: 1748 CA: fixed:r:sha256:11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk11749 client_integration_test.go:319: Retrieved .ls file from S3 (compressed size: 77 bytes)1750 client_integration_test.go:319: Decompressed .ls content (64 bytes):1751 {"version":1,"root":{"type":"regular","size":39,"narOffset":96}}1752--- PASS: TestReadProxyInvalidPath (7.43s)1753=== CONT TestPendingClosureWriteTimeout/empty1754=== CONT TestCommitCanBeRepeatedUntilCleanup/pending_closure17552026/11/12 11:46:31 INFO Received push request method=POST path=/api/pushes17562026/11/12 11:46:31 INFO Object in database but missing from S3 key=nikpix870b47hz0bw0z7rxqdqglrd4n2.ls17572026/11/12 11:46:31 WARN Found objects in DB but missing from S3, will re-upload count=117582026/11/12 11:46:31 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)17592026/11/12 11:46:31 INFO Received complete push request method=POST path=/api/pushes/2/complete17602026/11/12 11:46:31 WARN Failed to register uploaded object key=nikpix870b47hz0bw0z7rxqdqglrd4n2.ls error="server returned 404: 404 page not found\n"17612026/11/12 11:46:31 INFO Completed push id=2 objects=3 duration=601.5µs17622026/11/12 11:46:31 INFO Upload complete. (149ms)1763=== NAME TestClientIntegration1764 client_integration_test.go:336: Retrieved .ls file from S3 (compressed size: 62 bytes)1765 client_integration_test.go:336: Decompressed .ls content (49 bytes):1766 {"version":1,"root":{"type":"regular","size":39}}1767=== NAME TestClientWithDependencies1768 client_integration_test.go:661: Built derivation: /private/var/tb/1/scratch/d6c7014d22e1e3f1/build/TestClientWithDependencies140813118/001/store/ka3yf54ip3hbcrxglhy0lg6v78x1034n-test-script1769 client_integration_test.go:663: Found 1 dependencies (including self)17702026/11/12 11:46:32 INFO Received push request method=POST path=/api/pushes17712026/11/12 11:46:32 INFO Received push request method=POST path=/api/pushes17722026/11/12 11:46:32 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)17732026/11/12 11:46:32 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)17742026/11/12 11:46:32 INFO Uploading ka3yf54ip3hbcrxglhy0lg6v78x1034n-test-script (136B)17752026/11/12 11:46:32 INFO Received complete push request method=POST path=/api/pushes/3/complete17762026/11/12 11:46:32 WARN Failed to register uploaded object key=nikpix870b47hz0bw0z7rxqdqglrd4n2.ls error="server returned 404: 404 page not found\n"17772026/11/12 11:46:32 WARN Failed to register uploaded object key=ka3yf54ip3hbcrxglhy0lg6v78x1034n.ls error="server returned 404: 404 page not found\n"17782026/11/12 11:46:32 INFO Completed push id=3 objects=3 duration=822.416µs17792026/11/12 11:46:32 INFO Upload complete. (148ms)1780=== NAME TestClientIntegration1781 client_integration_test.go:353: Retrieved .ls file from S3 (compressed size: 62 bytes)17822026/11/12 11:46:32 WARN Failed to register uploaded object key=nar/1s7yghixzpinyqm7g59kyn3sav81pmnsdi5v63m6x4r2vacy907j.nar.zst error="server returned 404: 404 page not found\n"1783 client_integration_test.go:353: Decompressed .ls content (49 bytes):1784 {"version":1,"root":{"type":"regular","size":39}}1785 client_integration_test.go:364: Testing garbage collection...17862026/11/12 11:46:32 WARN Failed to register uploaded object key=log/afkna6awn4hz36zywgqw3r0prpnp4ldz-test-script.drv error="server returned 404: 404 page not found\n"17872026/11/12 11:46:32 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign17882026/11/12 11:46:32 INFO Signed narinfos id=1 count=117892026/11/12 11:46:32 INFO Uploading 1 narinfos1790=== NAME TestClientMultipleUploads17912026/11/12 11:46:32 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted,O=Other"17922026/11/12 11:46:32 WARN mTLS auth: bound subjects configured but subject DN unavailable17932026/11/12 11:46:32 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted"1794 client_integration_test.go:406: Created store path 0: /private/var/tb/1/scratch/d6c7014d22e1e3f1/build/TestClientMultipleUploads1186417644/001/store/qpkdnqb6bq3nhagn2ng70nsj9bj5ryq3-test-file-0.txt1795--- PASS: TestService_AuthMiddleware_MTLSBoundSubjects (7.14s)1796=== CONT TestIsValidUploadKey/narinfo1797=== CONT TestIsValidCachePath/narinfo1798=== CONT TestPendingClosureWriteTimeout/670k_objects1799=== CONT TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline18002026/11/12 11:46:32 INFO Starting cleanup of old closures method=DELETE path=/api/closures18012026/11/12 11:46:32 INFO Garbage collection started18022026/11/12 11:46:32 INFO Received complete push request method=POST path=/api/pushes/1/complete18032026/11/12 11:46:32 WARN Failed to register uploaded object key=ka3yf54ip3hbcrxglhy0lg6v78x1034n.narinfo error="server returned 404: 404 page not found\n"18042026/11/12 11:46:32 INFO Aborted multipart uploads count=0 kept=018052026/11/12 11:46:32 WARN Force mode enabled - objects will be deleted immediately without grace period1806=== NAME TestClientMultipleUploads1807 client_integration_test.go:406: Created store path 1: /private/var/tb/1/scratch/d6c7014d22e1e3f1/build/TestClientMultipleUploads1186417644/001/store/6wv97yhq3j7xy5k5zkwccjxsq99ah2wl-test-file-1.txt18082026/11/12 11:46:32 INFO Completed push id=1 objects=4 duration=45.557917ms18092026/11/12 11:46:32 INFO Upload complete. (373ms)18102026/11/12 11:46:32 INFO Aborted multipart uploads count=0 kept=01811 client_integration_test.go:406: Created store path 2: /private/var/tb/1/scratch/d6c7014d22e1e3f1/build/TestClientMultipleUploads1186417644/001/store/z1abnqryzhckfn6chlk5ipsvwgyfiq68-test-file-2.txt18122026/11/12 11:46:32 WARN Force mode enabled - objects will be deleted immediately without grace period18132026/11/12 11:46:32 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=018142026/11/12 11:46:32 INFO Vacuumed table table=pending_closures18152026/11/12 11:46:32 INFO Received push request method=POST path=/api/pushes18162026/11/12 11:46:32 INFO Vacuumed table table=commit_receipts18172026/11/12 11:46:32 INFO Uploading 3 paths to 127.0.0.1 (0 already cached)18182026/11/12 11:46:32 INFO Uploading qpkdnqb6bq3nhagn2ng70nsj9bj5ryq3-test-file-0.txt (160B)18192026/11/12 11:46:32 INFO Vacuumed table table=pending_objects18202026/11/12 11:46:32 INFO Uploading 6wv97yhq3j7xy5k5zkwccjxsq99ah2wl-test-file-1.txt (160B)18212026/11/12 11:46:32 INFO Uploading z1abnqryzhckfn6chlk5ipsvwgyfiq68-test-file-2.txt (160B)18222026/11/12 11:46:32 INFO Vacuumed table table=multipart_uploads1823--- PASS: TestService_ReadAuthMiddleware (6.83s)1824=== CONT TestIsValidUploadKey/nar_zst1825=== CONT TestIsValidUploadKey/unknown_type1826=== CONT TestIsValidUploadKey/empty_key1827=== CONT TestIsValidUploadKey/absolute1828=== CONT TestIsValidUploadKey/traversal_nar1829=== CONT TestIsValidUploadKey/traversal1830=== CONT TestIsValidUploadKey/listing_key,_narinfo_type1831=== CONT TestIsValidUploadKey/nar_key,_narinfo_type1832=== CONT TestIsValidUploadKey/narinfo_key,_nar_type1833=== CONT TestIsValidUploadKey/index.html18342026/11/12 11:46:32 WARN Failed to register uploaded object key=qpkdnqb6bq3nhagn2ng70nsj9bj5ryq3.ls error="server returned 404: 404 page not found\n"1835=== CONT TestIsValidUploadKey/nix-cache-info1836=== CONT TestIsValidUploadKey/realisation_plus_in_output18372026/11/12 11:46:32 WARN Failed to register uploaded object key=z1abnqryzhckfn6chlk5ipsvwgyfiq68.ls error="server returned 404: 404 page not found\n"1838=== CONT TestIsValidUploadKey/realisation1839=== CONT TestIsValidUploadKey/build_log_equals18402026/11/12 11:46:32 WARN Failed to register uploaded object key=6wv97yhq3j7xy5k5zkwccjxsq99ah2wl.ls error="server returned 404: 404 page not found\n"1841=== CONT TestIsValidUploadKey/build_log_question_mark1842=== CONT TestIsValidUploadKey/build_log_plus_in_name1843=== CONT TestIsValidUploadKey/build_log_home-manager_file1844=== CONT TestIsValidUploadKey/build_log1845=== CONT TestIsValidUploadKey/listing1846=== CONT TestIsValidUploadKey/nar_plain1847=== CONT TestIsValidUploadKey/nar_xz18482026/11/12 11:46:32 WARN Failed to register uploaded object key=nar/13p9pqi0ga6fx2apcwls9hjnv7nmf740ggpn5zvfzdsp5vz2bbg4.nar.zst error="server returned 404: 404 page not found\n"1849--- PASS: TestIsValidUploadKey (0.05s)1850 --- PASS: TestIsValidUploadKey/narinfo (0.00s)1851 --- PASS: TestIsValidUploadKey/nar_zst (0.00s)1852 --- PASS: TestIsValidUploadKey/unknown_type (0.00s)1853 --- PASS: TestIsValidUploadKey/empty_key (0.00s)1854 --- PASS: TestIsValidUploadKey/absolute (0.00s)1855 --- PASS: TestIsValidUploadKey/traversal_nar (0.00s)1856 --- PASS: TestIsValidUploadKey/traversal (0.00s)1857 --- PASS: TestIsValidUploadKey/listing_key,_narinfo_type (0.00s)1858 --- PASS: TestIsValidUploadKey/nar_key,_narinfo_type (0.00s)1859 --- PASS: TestIsValidUploadKey/narinfo_key,_nar_type (0.00s)1860 --- PASS: TestIsValidUploadKey/index.html (0.00s)1861 --- PASS: TestIsValidUploadKey/nix-cache-info (0.00s)1862 --- PASS: TestIsValidUploadKey/realisation_plus_in_output (0.00s)1863 --- PASS: TestIsValidUploadKey/realisation (0.00s)1864 --- PASS: TestIsValidUploadKey/build_log_equals (0.00s)1865 --- PASS: TestIsValidUploadKey/build_log_question_mark (0.00s)1866 --- PASS: TestIsValidUploadKey/build_log_plus_in_name (0.00s)1867 --- PASS: TestIsValidUploadKey/build_log_home-manager_file (0.00s)1868 --- PASS: TestIsValidUploadKey/build_log (0.00s)1869 --- PASS: TestIsValidUploadKey/listing (0.00s)1870 --- PASS: TestIsValidUploadKey/nar_plain (0.00s)1871 --- PASS: TestIsValidUploadKey/nar_xz (0.00s)1872=== CONT TestIsValidCachePath/traversal_parent1873=== CONT TestIsValidCachePath/ls1874=== CONT TestIsValidCachePath/index.html1875=== CONT TestIsValidCachePath/nix-cache-info18762026/11/12 11:46:32 WARN Failed to register uploaded object key=nar/1823g6vx4q5zqvr73m7lkq1wpk40fin7bsc792di430yk7sg4lka.nar.zst error="server returned 404: 404 page not found\n"1877=== CONT TestIsValidCachePath/realisation18782026/11/12 11:46:32 INFO Vacuumed table table=closures1879=== CONT TestIsValidCachePath/log18802026/11/12 11:46:32 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign1881=== CONT TestIsValidCachePath/nar_xz1882=== CONT TestIsValidCachePath/nar_uncompressed18832026/11/12 11:46:32 WARN Failed to register uploaded object key=nar/0ahq3izw3h0b0c08p3xdfclh758fwxnp85ccvflasb4vnfsv50ri.nar.zst error="server returned 404: 404 page not found\n"1884=== CONT TestIsValidCachePath/nar_bz21885=== CONT TestIsValidCachePath/empty1886=== CONT TestIsValidCachePath/nar_zst1887=== CONT TestIsValidCachePath/invalid_char_u1888=== CONT TestIsValidCachePath/random_path18892026/11/12 11:46:32 INFO Signed narinfos id=1 count=31890=== CONT TestIsValidCachePath/wrong_extension18912026/11/12 11:46:32 INFO Uploading 3 narinfos1892=== CONT TestIsValidCachePath/short_hash1893=== CONT TestIsValidCachePath/invalid_char_e1894=== CONT TestIsValidCachePath/traversal_in_middle1895=== CONT TestIsValidCachePath/leading_slash1896=== CONT TestIsValidCachePath/narinfo_all_nix_base32_chars18972026/11/12 11:46:33 INFO Vacuumed table table=objects1898--- PASS: TestIsValidCachePath (0.05s)1899 --- PASS: TestIsValidCachePath/narinfo (0.00s)1900 --- PASS: TestIsValidCachePath/traversal_parent (0.00s)1901 --- PASS: TestIsValidCachePath/ls (0.00s)1902 --- PASS: TestIsValidCachePath/index.html (0.00s)1903 --- PASS: TestIsValidCachePath/nix-cache-info (0.00s)1904 --- PASS: TestIsValidCachePath/realisation (0.00s)1905 --- PASS: TestIsValidCachePath/log (0.00s)1906 --- PASS: TestIsValidCachePath/nar_xz (0.00s)1907 --- PASS: TestIsValidCachePath/nar_uncompressed (0.00s)1908 --- PASS: TestIsValidCachePath/nar_bz2 (0.00s)1909 --- PASS: TestIsValidCachePath/empty (0.00s)1910 --- PASS: TestIsValidCachePath/nar_zst (0.00s)1911 --- PASS: TestIsValidCachePath/invalid_char_u (0.00s)1912 --- PASS: TestIsValidCachePath/random_path (0.00s)1913 --- PASS: TestIsValidCachePath/wrong_extension (0.00s)1914 --- PASS: TestIsValidCachePath/short_hash (0.00s)1915 --- PASS: TestIsValidCachePath/invalid_char_e (0.00s)1916 --- PASS: TestIsValidCachePath/traversal_in_middle (0.00s)1917 --- PASS: TestIsValidCachePath/leading_slash (0.00s)1918 --- PASS: TestIsValidCachePath/narinfo_all_nix_base32_chars (0.00s)1919=== CONT TestPendingClosureWriteTimeout/400_objects1920=== CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info19212026/11/12 11:46:33 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=019222026/11/12 11:46:33 INFO Received uploads request method=POST path=/1923=== CONT TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key19242026/11/12 11:46:33 INFO Received complete multipart upload request method=POST path=/1925=== NAME TestClientWithDependencies19262026/11/12 11:46:33 WARN Failed to register uploaded object key=qpkdnqb6bq3nhagn2ng70nsj9bj5ryq3.narinfo error="server returned 404: 404 page not found\n"1927 client_integration_test.go:677: Skipping nix copy test - isolated store (/private/var/tb/1/scratch/d6c7014d22e1e3f1/build/TestClientWithDependencies140813118/001/store) requires matching store prefix1928=== CONT TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key19292026/11/12 11:46:33 WARN Failed to register uploaded object key=6wv97yhq3j7xy5k5zkwccjxsq99ah2wl.narinfo error="server returned 404: 404 page not found\n"19302026/11/12 11:46:33 INFO Received request for more parts method=POST path=/1931=== CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal19322026/11/12 11:46:33 INFO Received complete push request method=POST path=/api/pushes/1/complete19332026/11/12 11:46:33 WARN Failed to register uploaded object key=z1abnqryzhckfn6chlk5ipsvwgyfiq68.narinfo error="server returned 404: 404 page not found\n"19342026/11/12 11:46:33 INFO Received uploads request method=POST path=/1935--- PASS: TestUploadHandlersRejectInvalidKeys (0.05s)1936 --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info (0.01s)1937 --- PASS: TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key (0.01s)1938 --- PASS: TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key (0.00s)1939 --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal (0.00s)1940=== CONT TestParseSingleRange/none1941=== CONT TestParseSingleRange/start_far_past_EOF1942=== CONT TestParseSingleRange/start_past_EOF1943=== CONT TestParseSingleRange/single_byte1944=== CONT TestParseSingleRange/suffix_exceeds_size1945=== CONT TestParseSingleRange/suffix1946=== CONT TestParseSingleRange/end_clamped_to_size1947=== CONT TestParseSingleRange/open-ended1948=== CONT TestParseSingleRange/closed1949=== CONT TestParseSingleRange/malformed_end_before_start1950=== CONT TestParseSingleRange/malformed_both_empty1951=== CONT TestParseSingleRange/malformed_no_dash1952=== CONT TestParseSingleRange/multi-range_ignored1953=== CONT TestParseSingleRange/unknown_unit1954--- PASS: TestParseSingleRange (0.06s)1955 --- PASS: TestParseSingleRange/none (0.00s)1956 --- PASS: TestParseSingleRange/start_far_past_EOF (0.00s)1957 --- PASS: TestParseSingleRange/start_past_EOF (0.00s)1958 --- PASS: TestParseSingleRange/single_byte (0.00s)1959 --- PASS: TestParseSingleRange/suffix_exceeds_size (0.00s)1960 --- PASS: TestParseSingleRange/suffix (0.00s)1961 --- PASS: TestParseSingleRange/end_clamped_to_size (0.00s)1962 --- PASS: TestParseSingleRange/open-ended (0.00s)1963 --- PASS: TestParseSingleRange/closed (0.00s)1964 --- PASS: TestParseSingleRange/malformed_end_before_start (0.00s)1965 --- PASS: TestParseSingleRange/malformed_both_empty (0.00s)1966 --- PASS: TestParseSingleRange/malformed_no_dash (0.00s)1967 --- PASS: TestParseSingleRange/multi-range_ignored (0.00s)1968 --- PASS: TestParseSingleRange/unknown_unit (0.00s)1969=== CONT TestUploadHandlersRejectOversizedBody/create_pending_closure19702026/11/12 11:46:33 INFO Received uploads request method=POST path=/19712026/11/12 11:46:33 INFO Vacuumed table table=pending_closures19722026/11/12 11:46:33 INFO Completed push id=1 objects=9 duration=41.602291ms19732026/11/12 11:46:33 INFO Upload complete. (448ms)1974=== NAME TestClientMultipleUploads1975 client_integration_test.go:417: Uploaded 3 paths in 483.449792ms19762026/11/12 11:46:33 INFO Vacuumed table table=commit_receipts1977--- PASS: TestClientWithDependencies (9.16s)1978=== CONT TestUploadHandlersRejectOversizedBody/request_more_parts19792026/11/12 11:46:33 INFO Received request for more parts method=POST path=/19802026/11/12 11:46:33 INFO Vacuumed table table=pending_objects19812026/11/12 11:46:33 INFO Vacuumed table table=multipart_uploads1982=== CONT TestUploadHandlersRejectOversizedBody/complete_multipart19832026/11/12 11:46:33 INFO Received complete multipart upload request method=POST path=/19842026/11/12 11:46:33 INFO Vacuumed table table=closures1985--- PASS: TestClientMultipleUploads (8.53s)1986=== CONT TestPush_RejectsBadRequests/no_roots19872026/11/12 11:46:33 INFO Received push request method=POST path=/api/pushes1988=== CONT TestPush_RejectsBadRequests/bad_root19892026/11/12 11:46:33 INFO Received push request method=POST path=/api/pushes1990=== CONT TestPush_RejectsBadRequests/no_objects19912026/11/12 11:46:33 INFO Received push request method=POST path=/api/pushes1992=== CONT TestPush_RejectsBadRequests/root_not_in_objects19932026/11/12 11:46:33 INFO Received push request method=POST path=/api/pushes1994--- PASS: TestPush_RejectsBadRequests (1.34s)1995 --- PASS: TestPush_RejectsBadRequests/no_roots (0.00s)1996 --- PASS: TestPush_RejectsBadRequests/bad_root (0.00s)1997 --- PASS: TestPush_RejectsBadRequests/no_objects (0.00s)1998 --- PASS: TestPush_RejectsBadRequests/root_not_in_objects (0.00s)1999=== CONT TestServerTLSConfig/not_a_PEM_file2000=== CONT TestServerTLSConfig/no_client_CA
loading 897 hidden lines…
28982026/11/12 11:46:53 INFO Completed push id=19 objects=6 duration=1.033583ms28992026/11/12 11:46:53 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029002026/11/12 11:46:53 INFO Received complete push request method=POST path=/29012026/11/12 11:46:53 INFO Aborted multipart uploads count=0 kept=029022026/11/12 11:46:53 INFO Vacuumed table table=pending_closures29032026/11/12 11:46:53 INFO Vacuumed table table=commit_receipts29042026/11/12 11:46:53 INFO Completed push id=14 objects=6 duration=6.627292ms29052026/11/12 11:46:53 INFO Vacuumed table table=commit_receipts29062026/11/12 11:46:53 INFO Aborted multipart uploads count=0 kept=029072026/11/12 11:46:53 INFO Received push request method=POST path=/29082026/11/12 11:46:53 INFO Vacuumed table table=pending_objects29092026/11/12 11:46:53 WARN Force mode enabled - objects will be deleted immediately without grace period29102026/11/12 11:46:53 INFO Vacuumed table table=commit_receipts29112026/11/12 11:46:53 INFO Received complete push request method=POST path=/29122026/11/12 11:46:53 INFO Vacuumed table table=pending_objects29132026/11/12 11:46:53 WARN Force mode enabled - objects will be deleted immediately without grace period29142026/11/12 11:46:53 INFO Vacuumed table table=multipart_uploads29152026/11/12 11:46:53 INFO Received push request method=POST path=/29162026/11/12 11:46:53 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=3 objects-failed-to-delete=029172026/11/12 11:46:53 INFO Vacuumed table table=multipart_uploads29182026/11/12 11:46:53 INFO Vacuumed table table=pending_objects29192026/11/12 11:46:53 INFO Vacuumed table table=closures29202026/11/12 11:46:53 INFO Completed push id=9 objects=4 duration=987.583µs29212026/11/12 11:46:53 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."29222026/11/12 11:46:53 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."29232026/11/12 11:46:53 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."29242026/11/12 11:46:53 INFO Aborted multipart uploads count=0 kept=029252026/11/12 11:46:53 INFO Vacuumed table table=objects29262026/11/12 11:46:53 INFO Vacuumed table table=multipart_uploads29272026/11/12 11:46:53 INFO Vacuumed table table=closures29282026/11/12 11:46:53 INFO Vacuumed table table=pending_closures29292026/11/12 11:46:53 INFO Vacuumed table table=closures29302026/11/12 11:46:53 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=1 objects-deleted-after-grace-period=0 objects-failed-to-delete=029312026/11/12 11:46:53 INFO Vacuumed table table=pending_closures29322026/11/12 11:46:53 INFO Vacuumed table table=objects29332026/11/12 11:46:53 INFO Vacuumed table table=commit_receipts29342026/11/12 11:46:53 INFO Aborted multipart uploads count=0 kept=029352026/11/12 11:46:53 INFO Received uploads request method=POST path=/29362026/11/12 11:46:53 INFO Vacuumed table table=objects29372026/11/12 11:46:53 INFO Vacuumed table table=pending_objects29382026/11/12 11:46:53 INFO Vacuumed table table=commit_receipts29392026/11/12 11:46:53 INFO Vacuumed table table=multipart_uploads29402026/11/12 11:46:53 WARN Force mode enabled - objects will be deleted immediately without grace period29412026/11/12 11:46:53 INFO Vacuumed table table=closures29422026/11/12 11:46:53 INFO Vacuumed table table=objects29432026/11/12 11:46:53 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."29442026/11/12 11:46:53 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."29452026/11/12 11:46:53 INFO Received push request method=POST path=/29462026/11/12 11:46:53 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."29472026/11/12 11:46:53 INFO Received push request method=POST path=/29482026/11/12 11:46:53 INFO Received uploads request method=POST path=/29492026/11/12 11:46:53 INFO Received push request method=POST path=/29502026/11/12 11:46:53 INFO Vacuumed table table=pending_objects29512026/11/12 11:46:53 INFO Received push request method=POST path=/29522026/11/12 11:46:53 INFO Vacuumed table table=multipart_uploads29532026/11/12 11:46:53 INFO Received uploads request method=POST path=/29542026/11/12 11:46:53 INFO Received uploads request method=POST path=/29552026/11/12 11:46:53 INFO Received push request method=POST path=/29562026/11/12 11:46:53 INFO Received complete push request method=POST path=/29572026/11/12 11:46:53 INFO Received push request method=POST path=/29582026/11/12 11:46:53 INFO Completed push id=12 objects=2 duration=936.75µs29592026/11/12 11:46:53 INFO Vacuumed table table=closures29602026/11/12 11:46:53 INFO Received uploads request method=POST path=/29612026/11/12 11:46:53 INFO Received push request method=POST path=/29622026/11/12 11:46:53 INFO Vacuumed table table=objects29632026/11/12 11:46:53 INFO Received uploads request method=POST path=/29642026/11/12 11:46:53 INFO Received uploads request method=POST path=/29652026/11/12 11:46:53 INFO Received uploads request method=POST path=/29662026/11/12 11:46:53 INFO Aborted multipart uploads count=0 kept=029672026/11/12 11:46:53 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029682026/11/12 11:46:53 INFO Vacuumed table table=pending_closures29692026/11/12 11:46:53 INFO Received push request method=POST path=/29702026/11/12 11:46:53 INFO Vacuumed table table=commit_receipts29712026/11/12 11:46:53 INFO Received uploads request method=POST path=/29722026/11/12 11:46:53 INFO Received push request method=POST path=/29732026/11/12 11:46:53 INFO Received push request method=POST path=/29742026/11/12 11:46:53 INFO Vacuumed table table=pending_objects29752026/11/12 11:46:53 INFO Vacuumed table table=multipart_uploads29762026/11/12 11:46:53 INFO Received complete upload request method=POST path=/29772026/11/12 11:46:53 INFO Received push request method=POST path=/29782026/11/12 11:46:53 INFO Vacuumed table table=closures29792026/11/12 11:46:53 INFO Received push request method=POST path=/29802026/11/12 11:46:53 INFO Completed upload id=10 objects=4 duration=796.792µs29812026/11/12 11:46:53 INFO Vacuumed table table=objects29822026/11/12 11:46:53 INFO Received uploads request method=POST path=/29832026/11/12 11:46:53 INFO Received uploads request method=POST path=/29842026/11/12 11:46:53 INFO Received uploads request method=POST path=/29852026/11/12 11:46:53 INFO Received push request method=POST path=/29862026/11/12 11:46:53 INFO Received push request method=POST path=/29872026/11/12 11:46:53 INFO Received complete upload request method=POST path=/29882026/11/12 11:46:53 INFO Aborted multipart uploads count=0 kept=029892026/11/12 11:46:53 INFO Received push request method=POST path=/29902026/11/12 11:46:53 INFO Received push request method=POST path=/29912026/11/12 11:46:53 INFO Received uploads request method=POST path=/29922026/11/12 11:46:53 INFO Completed upload id=5 objects=5 duration=439.917µs29932026/11/12 11:46:53 WARN Force mode enabled - objects will be deleted immediately without grace period29942026/11/12 11:46:53 INFO Received push request method=POST path=/29952026/11/12 11:46:53 INFO Received push request method=POST path=/29962026/11/12 11:46:53 INFO Aborted multipart uploads count=0 kept=029972026/11/12 11:46:53 INFO Received uploads request method=POST path=/29982026/11/12 11:46:53 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029992026/11/12 11:46:53 INFO Received uploads request method=POST path=/30002026/11/12 11:46:53 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030012026/11/12 11:46:53 INFO Vacuumed table table=pending_closures30022026/11/12 11:46:53 INFO Aborted multipart uploads count=0 kept=030032026/11/12 11:46:53 INFO Received uploads request method=POST path=/30042026/11/12 11:46:53 INFO Vacuumed table table=commit_receipts30052026/11/12 11:46:53 INFO Received complete push request method=POST path=/30062026/11/12 11:46:53 WARN Force mode enabled - objects will be deleted immediately without grace period30072026/11/12 11:46:53 INFO Received push request method=POST path=/30082026/11/12 11:46:53 INFO Vacuumed table table=pending_closures30092026/11/12 11:46:53 INFO Completed push id=6 objects=8 duration=6.606292ms30102026/11/12 11:46:53 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030112026/11/12 11:46:53 INFO Vacuumed table table=pending_objects30122026/11/12 11:46:53 INFO Vacuumed table table=commit_receipts30132026/11/12 11:46:53 INFO Vacuumed table table=multipart_uploads30142026/11/12 11:46:53 INFO Vacuumed table table=pending_closures30152026/11/12 11:46:53 INFO Received uploads request method=POST path=/30162026/11/12 11:46:53 INFO Received uploads request method=POST path=/30172026/11/12 11:46:53 INFO Vacuumed table table=closures30182026/11/12 11:46:53 INFO Vacuumed table table=commit_receipts30192026/11/12 11:46:53 INFO Vacuumed table table=objects30202026/11/12 11:46:53 INFO Vacuumed table table=pending_objects30212026/11/12 11:46:53 INFO Received uploads request method=POST path=/30222026/11/12 11:46:53 INFO Vacuumed table table=multipart_uploads30232026/11/12 11:46:53 INFO Received complete push request method=POST path=/30242026/11/12 11:46:53 INFO Received uploads request method=POST path=/30252026/11/12 11:46:53 INFO Vacuumed table table=closures30262026/11/12 11:46:53 INFO Vacuumed table table=pending_objects30272026/11/12 11:46:53 INFO Completed push id=22 objects=3 duration=1.0935ms30282026/11/12 11:46:53 INFO Vacuumed table table=objects30292026/11/12 11:46:53 INFO Received uploads request method=POST path=/30302026/11/12 11:46:53 INFO Aborted multipart uploads count=0 kept=030312026/11/12 11:46:53 INFO Vacuumed table table=multipart_uploads30322026/11/12 11:46:53 INFO Aborted multipart uploads count=0 kept=030332026/11/12 11:46:53 INFO Received complete upload request method=POST path=/30342026/11/12 11:46:53 INFO Vacuumed table table=closures30352026/11/12 11:46:53 INFO Completed upload id=7 objects=4 duration=6.219667ms30362026/11/12 11:46:53 WARN Force mode enabled - objects will be deleted immediately without grace period30372026/11/12 11:46:53 INFO Vacuumed table table=objects30382026/11/12 11:46:53 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=030392026/11/12 11:46:53 INFO Vacuumed table table=pending_closures30402026/11/12 11:46:53 INFO Received push request method=POST path=/30412026/11/12 11:46:53 INFO Received push request method=POST path=/30422026/11/12 11:46:53 INFO Aborted multipart uploads count=0 kept=030432026/11/12 11:46:53 INFO Received push request method=POST path=/30442026/11/12 11:46:53 WARN Force mode enabled - objects will be deleted immediately without grace period30452026/11/12 11:46:53 INFO Aborted multipart uploads count=0 kept=030462026/11/12 11:46:53 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030472026/11/12 11:46:53 INFO Received complete push request method=POST path=/30482026/11/12 11:46:53 INFO Vacuumed table table=commit_receipts30492026/11/12 11:46:53 WARN Force mode enabled - objects will be deleted immediately without grace period30502026/11/12 11:46:53 INFO Vacuumed table table=pending_closures30512026/11/12 11:46:53 INFO Vacuumed table table=commit_receipts30522026/11/12 11:46:53 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030532026/11/12 11:46:53 INFO Vacuumed table table=pending_objects30542026/11/12 11:46:53 INFO Vacuumed table table=pending_closures30552026/11/12 11:46:53 INFO Vacuumed table table=multipart_uploads30562026/11/12 11:46:53 INFO Vacuumed table table=pending_objects30572026/11/12 11:46:53 INFO Vacuumed table table=commit_receipts30582026/11/12 11:46:53 INFO Received complete upload request method=POST path=/30592026/11/12 11:46:53 INFO Vacuumed table table=multipart_uploads30602026/11/12 11:46:53 INFO Completed push id=1 objects=24 duration=37.045875ms30612026/11/12 11:46:53 INFO Vacuumed table table=pending_objects30622026/11/12 11:46:53 INFO Completed upload id=11 objects=14 duration=335.542µs30632026/11/12 11:46:53 INFO Vacuumed table table=multipart_uploads30642026/11/12 11:46:53 INFO Vacuumed table table=closures30652026/11/12 11:46:53 INFO Vacuumed table table=closures30662026/11/12 11:46:53 INFO Received push request method=POST path=/30672026/11/12 11:46:53 INFO Vacuumed table table=objects30682026/11/12 11:46:53 INFO Vacuumed table table=closures30692026/11/12 11:46:53 INFO Vacuumed table table=objects30702026/11/12 11:46:53 INFO Vacuumed table table=objects30712026/11/12 11:46:53 INFO Received push request method=POST path=/30722026/11/12 11:46:53 INFO Received push request method=POST path=/30732026/11/12 11:46:53 INFO Aborted multipart uploads count=0 kept=030742026/11/12 11:46:53 INFO Received uploads request method=POST path=/30752026/11/12 11:46:53 INFO Received push request method=POST path=/30762026/11/12 11:46:53 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030772026/11/12 11:46:53 INFO Received complete upload request method=POST path=/30782026/11/12 11:46:53 INFO Aborted multipart uploads count=0 kept=030792026/11/12 11:46:53 WARN Force mode enabled - objects will be deleted immediately without grace period30802026/11/12 11:46:53 INFO Completed upload id=9 objects=7 duration=11.39575ms30812026/11/12 11:46:53 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030822026/11/12 11:46:53 INFO Vacuumed table table=pending_closures30832026/11/12 11:46:53 INFO Vacuumed table table=commit_receipts30842026/11/12 11:46:53 INFO Vacuumed table table=pending_objects30852026/11/12 11:46:53 INFO Vacuumed table table=multipart_uploads30862026/11/12 11:46:53 INFO Vacuumed table table=pending_closures30872026/11/12 11:46:53 INFO Aborted multipart uploads count=0 kept=030882026/11/12 11:46:53 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."30892026/11/12 11:46:53 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."30902026/11/12 11:46:53 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."30912026/11/12 11:46:53 INFO Vacuumed table table=closures30922026/11/12 11:46:53 INFO Vacuumed table table=objects30932026/11/12 11:46:53 INFO Vacuumed table table=commit_receipts30942026/11/12 11:46:53 INFO Received uploads request method=POST path=/30952026/11/12 11:46:53 INFO Received uploads request method=POST path=/30962026/11/12 11:46:53 INFO Received complete push request method=POST path=/30972026/11/12 11:46:53 INFO Received complete push request method=POST path=/30982026/11/12 11:46:53 INFO Completed push id=10 objects=2 duration=589.208µs30992026/11/12 11:46:53 INFO Completed push id=14 objects=3 duration=1.477166ms31002026/11/12 11:46:53 INFO Vacuumed table table=pending_objects31012026/11/12 11:46:53 INFO Vacuumed table table=multipart_uploads31022026/11/12 11:46:53 INFO Received complete push request method=POST path=/31032026/11/12 11:46:53 INFO Completed push id=5 objects=6 duration=418.125µs31042026/11/12 11:46:53 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."31052026/11/12 11:46:53 INFO Aborted multipart uploads count=0 kept=031062026/11/12 11:46:53 INFO Received uploads request method=POST path=/31072026/11/12 11:46:53 WARN Force mode enabled - objects will be deleted immediately without grace period31082026/11/12 11:46:53 INFO Vacuumed table table=closures31092026/11/12 11:46:53 INFO Received uploads request method=POST path=/31102026/11/12 11:46:53 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031112026/11/12 11:46:53 INFO Vacuumed table table=pending_closures31122026/11/12 11:46:53 INFO Received uploads request method=POST path=/31132026/11/12 11:46:53 INFO Vacuumed table table=commit_receipts31142026/11/12 11:46:53 INFO Vacuumed table table=pending_objects31152026/11/12 11:46:53 INFO Vacuumed table table=multipart_uploads31162026/11/12 11:46:53 INFO Aborted multipart uploads count=0 kept=031172026/11/12 11:46:53 INFO Vacuumed table table=closures31182026/11/12 11:46:53 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."31192026/11/12 11:46:53 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."31202026/11/12 11:46:53 INFO Vacuumed table table=objects31212026/11/12 11:46:53 INFO Vacuumed table table=objects31222026/11/12 11:46:53 INFO Received complete push request method=POST path=/31232026/11/12 11:46:53 INFO Received complete upload request method=POST path=/31242026/11/12 11:46:53 INFO Received push request method=POST path=/31252026/11/12 11:46:53 INFO Completed upload id=12 objects=21 duration=418.583µs31262026/11/12 11:46:53 INFO Completed push id=15 objects=7 duration=594.459µs31272026/11/12 11:46:53 INFO Aborted multipart uploads count=0 kept=031282026/11/12 11:46:53 INFO Received complete push request method=POST path=/31292026/11/12 11:46:53 INFO Received push request method=POST path=/31302026/11/12 11:46:53 INFO Received push request method=POST path=/31312026/11/12 11:46:53 WARN Force mode enabled - objects will be deleted immediately without grace period31322026/11/12 11:46:54 INFO Completed push id=3 objects=21 duration=6.662417ms31332026/11/12 11:46:54 INFO Received uploads request method=POST path=/31342026/11/12 11:46:54 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=3 objects-failed-to-delete=031352026/11/12 11:46:54 INFO Received uploads request method=POST path=/31362026/11/12 11:46:54 INFO Received uploads request method=POST path=/31372026/11/12 11:46:54 INFO Vacuumed table table=pending_closures31382026/11/12 11:46:54 INFO Vacuumed table table=commit_receipts31392026/11/12 11:46:54 INFO Received push request method=POST path=/31402026/11/12 11:46:54 INFO Vacuumed table table=pending_objects31412026/11/12 11:46:54 INFO Received push request method=POST path=/31422026/11/12 11:46:54 INFO Vacuumed table table=multipart_uploads31432026/11/12 11:46:54 INFO Vacuumed table table=closures31442026/11/12 11:46:54 INFO Received push request method=POST path=/31452026/11/12 11:46:54 INFO Vacuumed table table=objects31462026/11/12 11:46:54 INFO Received complete upload request method=POST path=/31472026/11/12 11:46:54 INFO Received uploads request method=POST path=/31482026/11/12 11:46:54 INFO Received uploads request method=POST path=/31492026/11/12 11:46:54 INFO Received complete upload request method=POST path=/31502026/11/12 11:46:54 INFO Completed upload id=22 objects=2 duration=442.625µs31512026/11/12 11:46:54 INFO Received push request method=POST path=/31522026/11/12 11:46:54 INFO Received complete push request method=POST path=/31532026/11/12 11:46:54 INFO Completed push id=13 objects=23 duration=451.166µs31542026/11/12 11:46:54 INFO Completed upload id=21 objects=7 duration=1.105708ms31552026/11/12 11:46:54 INFO Aborted multipart uploads count=0 kept=031562026/11/12 11:46:54 INFO Received uploads request method=POST path=/31572026/11/12 11:46:54 INFO Received uploads request method=POST path=/31582026/11/12 11:46:54 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=031592026/11/12 11:46:54 INFO Vacuumed table table=pending_closures31602026/11/12 11:46:54 INFO Vacuumed table table=commit_receipts31612026/11/12 11:46:54 INFO Vacuumed table table=pending_objects31622026/11/12 11:46:54 INFO Received push request method=POST path=/31632026/11/12 11:46:54 INFO Received uploads request method=POST path=/31642026/11/12 11:46:54 INFO Aborted multipart uploads count=0 kept=031652026/11/12 11:46:54 INFO Received complete push request method=POST path=/31662026/11/12 11:46:54 INFO Vacuumed table table=multipart_uploads31672026/11/12 11:46:54 INFO Received complete push request method=POST path=/31682026/11/12 11:46:54 INFO Vacuumed table table=closures31692026/11/12 11:46:54 INFO Completed push id=15 objects=5 duration=756.792µs31702026/11/12 11:46:54 INFO Received uploads request method=POST path=/31712026/11/12 11:46:54 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=031722026/11/12 11:46:54 INFO Completed push id=7 objects=15 duration=823.916µs31732026/11/12 11:46:54 INFO Vacuumed table table=objects31742026/11/12 11:46:54 INFO Received uploads request method=POST path=/31752026/11/12 11:46:54 INFO Received uploads request method=POST path=/31762026/11/12 11:46:54 INFO Received push request method=POST path=/31772026/11/12 11:46:54 INFO Received uploads request method=POST path=/31782026/11/12 11:46:54 INFO Received push request method=POST path=/31792026/11/12 11:46:54 INFO Vacuumed table table=pending_closures31802026/11/12 11:46:54 INFO Received complete upload request method=POST path=/31812026/11/12 11:46:54 INFO Vacuumed table table=commit_receipts31822026/11/12 11:46:54 INFO Completed upload id=27 objects=4 duration=694.833µs31832026/11/12 11:46:54 INFO Received push request method=POST path=/31842026/11/12 11:46:54 INFO Received push request method=POST path=/31852026/11/12 11:46:54 INFO Received uploads request method=POST path=/31862026/11/12 11:46:54 INFO Received complete push request method=POST path=/31872026/11/12 11:46:54 INFO Received push request method=POST path=/31882026/11/12 11:46:54 INFO Completed push id=13 objects=11 duration=487.75µs31892026/11/12 11:46:54 INFO Vacuumed table table=pending_objects31902026/11/12 11:46:54 INFO Aborted multipart uploads count=0 kept=031912026/11/12 11:46:54 INFO Received push request method=POST path=/31922026/11/12 11:46:54 INFO Received uploads request method=POST path=/31932026/11/12 11:46:54 INFO Vacuumed table table=multipart_uploads31942026/11/12 11:46:54 WARN Force mode enabled - objects will be deleted immediately without grace period31952026/11/12 11:46:54 INFO Aborted multipart uploads count=0 kept=031962026/11/12 11:46:54 INFO Vacuumed table table=closures31972026/11/12 11:46:54 INFO Aborted multipart uploads count=0 kept=031982026/11/12 11:46:54 INFO Received complete upload request method=POST path=/31992026/11/12 11:46:54 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032002026/11/12 11:46:54 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032012026/11/12 11:46:54 INFO Vacuumed table table=objects32022026/11/12 11:46:54 INFO Vacuumed table table=pending_closures32032026/11/12 11:46:54 INFO Vacuumed table table=pending_closures32042026/11/12 11:46:54 INFO Vacuumed table table=commit_receipts32052026/11/12 11:46:54 INFO Vacuumed table table=commit_receipts32062026/11/12 11:46:54 WARN Force mode enabled - objects will be deleted immediately without grace period32072026/11/12 11:46:54 INFO Received complete upload request method=POST path=/32082026/11/12 11:46:54 INFO Vacuumed table table=pending_objects32092026/11/12 11:46:54 INFO Received push request method=POST path=/32102026/11/12 11:46:54 INFO Vacuumed table table=multipart_uploads32112026/11/12 11:46:54 INFO Completed upload id=5 objects=7 duration=6.088417ms32122026/11/12 11:46:54 INFO Vacuumed table table=pending_objects32132026/11/12 11:46:54 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032142026/11/12 11:46:54 INFO Vacuumed table table=closures32152026/11/12 11:46:54 INFO Vacuumed table table=multipart_uploads32162026/11/12 11:46:54 INFO Vacuumed table table=pending_closures32172026/11/12 11:46:54 INFO Received uploads request method=POST path=/32182026/11/12 11:46:54 INFO Vacuumed table table=objects32192026/11/12 11:46:54 INFO Completed upload id=1 objects=15 duration=43.200542ms32202026/11/12 11:46:54 INFO Vacuumed table table=commit_receipts32212026/11/12 11:46:54 INFO Vacuumed table table=pending_objects32222026/11/12 11:46:54 INFO Received uploads request method=POST path=/32232026/11/12 11:46:54 INFO Received push request method=POST path=/32242026/11/12 11:46:54 INFO Received uploads request method=POST path=/32252026/11/12 11:46:54 INFO Vacuumed table table=multipart_uploads32262026/11/12 11:46:54 INFO Vacuumed table table=closures32272026/11/12 11:46:54 INFO Vacuumed table table=closures32282026/11/12 11:46:54 INFO Vacuumed table table=objects32292026/11/12 11:46:54 INFO Aborted multipart uploads count=0 kept=032302026/11/12 11:46:54 INFO Received complete upload request method=POST path=/32312026/11/12 11:46:54 INFO Received push request method=POST path=/32322026/11/12 11:46:54 INFO Vacuumed table table=objects32332026/11/12 11:46:54 INFO Completed upload id=15 objects=4 duration=854.416µs32342026/11/12 11:46:54 INFO Received complete upload request method=POST path=/32352026/11/12 11:46:54 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."32362026/11/12 11:46:54 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."32372026/11/12 11:46:54 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."32382026/11/12 11:46:54 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."32392026/11/12 11:46:54 INFO Completed upload id=6 objects=9 duration=729.834µs32402026/11/12 11:46:54 INFO Received push request method=POST path=/32412026/11/12 11:46:54 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."32422026/11/12 11:46:54 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."32432026/11/12 11:46:54 INFO Received uploads request method=POST path=/32442026/11/12 11:46:54 INFO Received uploads request method=POST path=/32452026/11/12 11:46:54 INFO Received uploads request method=POST path=/32462026/11/12 11:46:54 INFO Received uploads request method=POST path=/32472026/11/12 11:46:54 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."32482026/11/12 11:46:54 INFO Received uploads request method=POST path=/32492026/11/12 11:46:54 INFO Aborted multipart uploads count=0 kept=032502026/11/12 11:46:54 INFO Received complete upload request method=POST path=/32512026/11/12 11:46:54 INFO Received uploads request method=POST path=/32522026/11/12 11:46:54 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."32532026/11/12 11:46:54 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."32542026/11/12 11:46:54 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."32552026/11/12 11:46:54 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."32562026/11/12 11:46:54 INFO Received complete upload request method=POST path=/32572026/11/12 11:46:54 INFO Completed upload id=16 objects=3 duration=428.75µs32582026/11/12 11:46:54 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."32592026/11/12 11:46:54 INFO Received push request method=POST path=/32602026/11/12 11:46:54 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."32612026/11/12 11:46:54 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."32622026/11/12 11:46:54 INFO Completed upload id=3 objects=5 duration=383.209µs32632026/11/12 11:46:54 INFO Aborted multipart uploads count=0 kept=032642026/11/12 11:46:54 INFO Aborted multipart uploads count=0 kept=032652026/11/12 11:46:54 WARN Force mode enabled - objects will be deleted immediately without grace period32662026/11/12 11:46:54 INFO Aborted multipart uploads count=0 kept=032672026/11/12 11:46:54 INFO Received push request method=POST path=/32682026/11/12 11:46:54 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032692026/11/12 11:46:54 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032702026/11/12 11:46:54 INFO Aborted multipart uploads count=0 kept=032712026/11/12 11:46:54 INFO Vacuumed table table=pending_closures32722026/11/12 11:46:54 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=032732026/11/12 11:46:54 INFO Received push request method=POST path=/32742026/11/12 11:46:54 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032752026/11/12 11:46:54 INFO Vacuumed table table=commit_receipts32762026/11/12 11:46:54 INFO Received complete push request method=POST path=/32772026/11/12 11:46:54 INFO Vacuumed table table=pending_closures32782026/11/12 11:46:54 INFO Aborted multipart uploads count=0 kept=032792026/11/12 11:46:54 INFO Vacuumed table table=pending_closures32802026/11/12 11:46:54 INFO Vacuumed table table=pending_objects32812026/11/12 11:46:54 INFO Vacuumed table table=commit_receipts32822026/11/12 11:46:54 INFO Vacuumed table table=commit_receipts32832026/11/12 11:46:54 INFO Vacuumed table table=multipart_uploads32842026/11/12 11:46:54 INFO Completed push id=16 objects=14 duration=1.164583ms32852026/11/12 11:46:54 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."32862026/11/12 11:46:54 INFO Vacuumed table table=pending_objects32872026/11/12 11:46:54 INFO Vacuumed table table=pending_closures32882026/11/12 11:46:54 INFO Vacuumed table table=closures32892026/11/12 11:46:54 INFO Received complete push request method=POST path=/32902026/11/12 11:46:54 INFO Vacuumed table table=multipart_uploads32912026/11/12 11:46:54 INFO Vacuumed table table=commit_receipts32922026/11/12 11:46:54 INFO Vacuumed table table=pending_objects32932026/11/12 11:46:54 INFO Aborted multipart uploads count=0 kept=032942026/11/12 11:46:54 INFO Received uploads request method=POST path=/32952026/11/12 11:46:54 INFO Vacuumed table table=objects32962026/11/12 11:46:54 INFO Completed push id=14 objects=3 duration=6.758542ms32972026/11/12 11:46:54 INFO Vacuumed table table=closures32982026/11/12 11:46:54 INFO Vacuumed table table=multipart_uploads32992026/11/12 11:46:54 WARN Force mode enabled - objects will be deleted immediately without grace period33002026/11/12 11:46:54 INFO Vacuumed table table=pending_objects33012026/11/12 11:46:54 INFO Vacuumed table table=objects33022026/11/12 11:46:54 INFO Vacuumed table table=closures33032026/11/12 11:46:54 INFO Vacuumed table table=multipart_uploads33042026/11/12 11:46:54 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033052026/11/12 11:46:54 INFO Vacuumed table table=objects33062026/11/12 11:46:54 INFO Vacuumed table table=closures33072026/11/12 11:46:54 INFO Vacuumed table table=pending_closures33082026/11/12 11:46:54 INFO Received uploads request method=POST path=/33092026/11/12 11:46:54 INFO Vacuumed table table=commit_receipts33102026/11/12 11:46:54 INFO Vacuumed table table=objects33112026/11/12 11:46:54 INFO Aborted multipart uploads count=0 kept=033122026/11/12 11:46:54 INFO Aborted multipart uploads count=0 kept=033132026/11/12 11:46:54 INFO Vacuumed table table=pending_objects33142026/11/12 11:46:54 INFO Aborted multipart uploads count=0 kept=033152026/11/12 11:46:54 WARN Force mode enabled - objects will be deleted immediately without grace period33162026/11/12 11:46:54 INFO Received uploads request method=POST path=/33172026/11/12 11:46:54 INFO Vacuumed table table=multipart_uploads33182026/11/12 11:46:54 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033192026/11/12 11:46:54 INFO Vacuumed table table=closures33202026/11/12 11:46:54 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033212026/11/12 11:46:54 INFO Vacuumed table table=pending_closures33222026/11/12 11:46:54 INFO Received complete upload request method=POST path=/33232026/11/12 11:46:54 INFO Vacuumed table table=objects33242026/11/12 11:46:54 INFO Received complete push request method=POST path=/33252026/11/12 11:46:54 INFO Vacuumed table table=pending_closures33262026/11/12 11:46:54 INFO Vacuumed table table=commit_receipts33272026/11/12 11:46:54 INFO Received complete push request method=POST path=/33282026/11/12 11:46:54 INFO Completed upload id=18 objects=4 duration=601.917µs33292026/11/12 11:46:54 INFO Vacuumed table table=commit_receipts33302026/11/12 11:46:54 INFO Aborted multipart uploads count=0 kept=033312026/11/12 11:46:54 INFO Vacuumed table table=pending_objects33322026/11/12 11:46:54 INFO Completed push id=21 objects=9 duration=500.209µs33332026/11/12 11:46:54 INFO Vacuumed table table=pending_objects33342026/11/12 11:46:54 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033352026/11/12 11:46:54 INFO Received complete push request method=POST path=/33362026/11/12 11:46:54 INFO Vacuumed table table=multipart_uploads33372026/11/12 11:46:54 INFO Completed push id=3 objects=8 duration=36.18975ms33382026/11/12 11:46:54 INFO Vacuumed table table=multipart_uploads33392026/11/12 11:46:54 INFO Vacuumed table table=closures33402026/11/12 11:46:54 INFO Vacuumed table table=pending_closures33412026/11/12 11:46:54 INFO Completed push id=17 objects=6 duration=589.791µs33422026/11/12 11:46:54 INFO Vacuumed table table=closures33432026/11/12 11:46:54 INFO Vacuumed table table=commit_receipts33442026/11/12 11:46:54 INFO Vacuumed table table=objects33452026/11/12 11:46:54 INFO Vacuumed table table=objects33462026/11/12 11:46:54 INFO Received uploads request method=POST path=/33472026/11/12 11:46:54 INFO Received push request method=POST path=/33482026/11/12 11:46:54 INFO Received push request method=POST path=/33492026/11/12 11:46:54 INFO Vacuumed table table=pending_objects33502026/11/12 11:46:54 INFO Received push request method=POST path=/33512026/11/12 11:46:54 INFO Received complete push request method=POST path=/33522026/11/12 11:46:54 INFO Vacuumed table table=multipart_uploads33532026/11/12 11:46:54 INFO Completed push id=2 objects=10 duration=1.19025ms33542026/11/12 11:46:54 INFO Received complete push request method=POST path=/33552026/11/12 11:46:54 INFO Aborted multipart uploads count=0 kept=033562026/11/12 11:46:54 INFO Received push request method=POST path=/33572026/11/12 11:46:54 INFO Vacuumed table table=closures33582026/11/12 11:46:54 INFO Completed push id=4 objects=7 duration=482.458µs33592026/11/12 11:46:54 INFO Received push request method=POST path=/33602026/11/12 11:46:54 INFO Vacuumed table table=objects33612026/11/12 11:46:54 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033622026/11/12 11:46:54 INFO Received uploads request method=POST path=/33632026/11/12 11:46:54 INFO Received push request method=POST path=/33642026/11/12 11:46:54 INFO Received complete push request method=POST path=/33652026/11/12 11:46:54 INFO Completed push id=8 objects=7 duration=452.541µs33662026/11/12 11:46:54 INFO Vacuumed table table=pending_closures33672026/11/12 11:46:54 INFO Received push request method=POST path=/33682026/11/12 11:46:54 INFO Vacuumed table table=commit_receipts33692026/11/12 11:46:54 INFO Received uploads request method=POST path=/33702026/11/12 11:46:54 INFO Received uploads request method=POST path=/33712026/11/12 11:46:54 INFO Received uploads request method=POST path=/33722026/11/12 11:46:54 INFO Vacuumed table table=pending_objects33732026/11/12 11:46:54 INFO Vacuumed table table=multipart_uploads33742026/11/12 11:46:54 INFO Received complete upload request method=POST path=/33752026/11/12 11:46:54 INFO Vacuumed table table=closures33762026/11/12 11:46:54 INFO Completed upload id=25 objects=6 duration=4.303ms33772026/11/12 11:46:54 INFO Received complete upload request method=POST path=/33782026/11/12 11:46:54 INFO Received push request method=POST path=/33792026/11/12 11:46:54 INFO Vacuumed table table=objects33802026/11/12 11:46:54 INFO Received complete upload request method=POST path=/33812026/11/12 11:46:54 INFO Completed upload id=33 objects=5 duration=1.083875ms33822026/11/12 11:46:54 INFO Received push request method=POST path=/33832026/11/12 11:46:54 INFO Aborted multipart uploads count=0 kept=033842026/11/12 11:46:54 INFO Completed upload id=13 objects=8 duration=838.625µs33852026/11/12 11:46:54 INFO Received uploads request method=POST path=/33862026/11/12 11:46:54 WARN Force mode enabled - objects will be deleted immediately without grace period33872026/11/12 11:46:54 INFO Received uploads request method=POST path=/33882026/11/12 11:46:54 INFO Received complete push request method=POST path=/33892026/11/12 11:46:54 INFO Received push request method=POST path=/33902026/11/12 11:46:54 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."33912026/11/12 11:46:54 INFO Received uploads request method=POST path=/33922026/11/12 11:46:54 INFO Completed push id=11 objects=9 duration=447.833µs33932026/11/12 11:46:54 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."33942026/11/12 11:46:54 INFO Received push request method=POST path=/33952026/11/12 11:46:54 INFO Received push request method=POST path=/33962026/11/12 11:46:54 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."33972026/11/12 11:46:54 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."33982026/11/12 11:46:54 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."33992026/11/12 11:46:54 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."34002026/11/12 11:46:54 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."34012026/11/12 11:46:54 INFO Received push request method=POST path=/34022026/11/12 11:46:54 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."34032026/11/12 11:46:54 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."34042026/11/12 11:46:54 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."34052026/11/12 11:46:54 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."34062026/11/12 11:46:54 INFO Aborted multipart uploads count=0 kept=034072026/11/12 11:46:54 INFO Received push request method=POST path=/34082026/11/12 11:46:54 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034092026/11/12 11:46:54 INFO Received complete push request method=POST path=/34102026/11/12 11:46:54 INFO Vacuumed table table=pending_closures34112026/11/12 11:46:54 INFO Received complete push request method=POST path=/34122026/11/12 11:46:54 INFO Completed push id=19 objects=14 duration=596.5µs34132026/11/12 11:46:54 INFO Aborted multipart uploads count=0 kept=034142026/11/12 11:46:54 INFO Vacuumed table table=commit_receipts34152026/11/12 11:46:54 INFO Received complete push request method=POST path=/34162026/11/12 11:46:54 INFO Completed push id=8 objects=2 duration=394.666µs34172026/11/12 11:46:54 INFO Completed push id=15 objects=24 duration=562.958µs34182026/11/12 11:46:54 WARN Force mode enabled - objects will be deleted immediately without grace period34192026/11/12 11:46:54 INFO Received uploads request method=POST path=/34202026/11/12 11:46:54 INFO Aborted multipart uploads count=0 kept=034212026/11/12 11:46:54 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034222026/11/12 11:46:54 INFO Received complete upload request method=POST path=/34232026/11/12 11:46:54 INFO Vacuumed table table=pending_objects34242026/11/12 11:46:54 INFO Received complete upload request method=POST path=/34252026/11/12 11:46:54 WARN Force mode enabled - objects will be deleted immediately without grace period34262026/11/12 11:46:54 INFO Vacuumed table table=pending_closures34272026/11/12 11:46:55 INFO Completed upload id=19 objects=2 duration=361.208µs34282026/11/12 11:46:55 INFO Vacuumed table table=multipart_uploads34292026/11/12 11:46:55 INFO Completed upload id=9 objects=16 duration=945.75µs34302026/11/12 11:46:55 INFO Received uploads request method=POST path=/34312026/11/12 11:46:55 INFO Vacuumed table table=commit_receipts34322026/11/12 11:46:55 INFO Received complete upload request method=POST path=/34332026/11/12 11:46:55 INFO Vacuumed table table=closures34342026/11/12 11:46:55 INFO Received push request method=POST path=/34352026/11/12 11:46:55 INFO Received complete push request method=POST path=/34362026/11/12 11:46:55 INFO Vacuumed table table=pending_objects34372026/11/12 11:46:55 INFO Completed upload id=24 objects=7 duration=473.417µs34382026/11/12 11:46:55 INFO Vacuumed table table=multipart_uploads34392026/11/12 11:46:55 INFO Vacuumed table table=objects34402026/11/12 11:46:55 INFO Completed push id=17 objects=22 duration=797µs34412026/11/12 11:46:55 INFO Received push request method=POST path=/34422026/11/12 11:46:55 INFO Vacuumed table table=closures34432026/11/12 11:46:55 INFO Received complete push request method=POST path=/34442026/11/12 11:46:55 INFO Received push request method=POST path=/34452026/11/12 11:46:55 INFO Received complete push request method=POST path=/34462026/11/12 11:46:55 INFO Received complete upload request method=POST path=/34472026/11/12 11:46:55 INFO Completed push id=26 objects=3 duration=300.666µs34482026/11/12 11:46:55 INFO Vacuumed table table=objects34492026/11/12 11:46:55 INFO Received uploads request method=POST path=/34502026/11/12 11:46:55 INFO Completed push id=10 objects=3 duration=430.375µs34512026/11/12 11:46:55 INFO Completed upload id=6 objects=24 duration=900.625µs34522026/11/12 11:46:55 INFO Received complete upload request method=POST path=/34532026/11/12 11:46:55 INFO Received uploads request method=POST path=/34542026/11/12 11:46:55 INFO Received push request method=POST path=/34552026/11/12 11:46:55 INFO Received uploads request method=POST path=/34562026/11/12 11:46:55 INFO Completed upload id=28 objects=8 duration=836.083µs34572026/11/12 11:46:55 INFO Received push request method=POST path=/34582026/11/12 11:46:55 INFO Received complete push request method=POST path=/34592026/11/12 11:46:55 INFO Received push request method=POST path=/34602026/11/12 11:46:55 INFO Received complete upload request method=POST path=/34612026/11/12 11:46:55 INFO Completed push id=26 objects=14 duration=1.089708ms34622026/11/12 11:46:55 INFO Aborted multipart uploads count=0 kept=034632026/11/12 11:46:55 INFO Completed upload id=29 objects=10 duration=278.958µs34642026/11/12 11:46:55 WARN Force mode enabled - objects will be deleted immediately without grace period34652026/11/12 11:46:55 INFO Received push request method=POST path=/34662026/11/12 11:46:55 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=034672026/11/12 11:46:55 INFO Received complete upload request method=POST path=/34682026/11/12 11:46:55 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034692026/11/12 11:46:55 INFO Vacuumed table table=pending_closures34702026/11/12 11:46:55 INFO Completed upload id=30 objects=20 duration=270.5µs34712026/11/12 11:46:55 INFO Vacuumed table table=pending_closures34722026/11/12 11:46:55 INFO Vacuumed table table=commit_receipts34732026/11/12 11:46:55 INFO Vacuumed table table=pending_objects34742026/11/12 11:46:55 INFO Received complete push request method=POST path=/34752026/11/12 11:46:55 INFO Received uploads request method=POST path=/34762026/11/12 11:46:55 INFO Vacuumed table table=multipart_uploads34772026/11/12 11:46:55 INFO Completed push id=31 objects=11 duration=344.333µs34782026/11/12 11:46:55 INFO Vacuumed table table=closures34792026/11/12 11:46:55 INFO Aborted multipart uploads count=0 kept=034802026/11/12 11:46:55 INFO Vacuumed table table=commit_receipts34812026/11/12 11:46:55 INFO Received complete upload request method=POST path=/34822026/11/12 11:46:55 INFO Vacuumed table table=objects34832026/11/12 11:46:55 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034842026/11/12 11:46:55 INFO Vacuumed table table=pending_objects34852026/11/12 11:46:55 INFO Completed upload id=32 objects=2 duration=244.417µs34862026/11/12 11:46:55 INFO Vacuumed table table=multipart_uploads34872026/11/12 11:46:55 INFO Received complete push request method=POST path=/34882026/11/12 11:46:55 INFO Vacuumed table table=pending_closures34892026/11/12 11:46:55 INFO Received uploads request method=POST path=/34902026/11/12 11:46:55 INFO Completed push id=8 objects=6 duration=846.625µs34912026/11/12 11:46:55 INFO Received complete push request method=POST path=/34922026/11/12 11:46:55 INFO Vacuumed table table=commit_receipts34932026/11/12 11:46:55 INFO Received uploads request method=POST path=/34942026/11/12 11:46:55 INFO Completed push id=33 objects=3 duration=234µs34952026/11/12 11:46:55 INFO Vacuumed table table=pending_objects34962026/11/12 11:46:55 INFO Vacuumed table table=multipart_uploads34972026/11/12 11:46:55 INFO Received complete upload request method=POST path=/34982026/11/12 11:46:55 INFO Vacuumed table table=closures34992026/11/12 11:46:55 INFO Received push request method=POST path=/35002026/11/12 11:46:55 INFO Completed upload id=34 objects=3 duration=340.833µs35012026/11/12 11:46:55 INFO Vacuumed table table=objects35022026/11/12 11:46:55 INFO Received uploads request method=POST path=/35032026/11/12 11:46:55 INFO Aborted multipart uploads count=0 kept=035042026/11/12 11:46:55 INFO Vacuumed table table=closures35052026/11/12 11:46:55 WARN Force mode enabled - objects will be deleted immediately without grace period35062026/11/12 11:46:55 INFO Received push request method=POST path=/35072026/11/12 11:46:55 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035082026/11/12 11:46:55 INFO Received push request method=POST path=/35092026/11/12 11:46:55 INFO Vacuumed table table=objects35102026/11/12 11:46:55 INFO Received push request method=POST path=/35112026/11/12 11:46:55 INFO Vacuumed table table=pending_closures35122026/11/12 11:46:55 INFO Vacuumed table table=commit_receipts35132026/11/12 11:46:55 INFO Received complete push request method=POST path=/35142026/11/12 11:46:55 INFO Received complete upload request method=POST path=/35152026/11/12 11:46:55 INFO Received push request method=POST path=/35162026/11/12 11:46:55 INFO Completed upload id=9 objects=3 duration=357.625µs35172026/11/12 11:46:55 INFO Received complete push request method=POST path=/35182026/11/12 11:46:55 INFO Received push request method=POST path=/35192026/11/12 11:46:55 INFO Vacuumed table table=pending_objects35202026/11/12 11:46:55 INFO Completed push id=5 objects=28 duration=1.232041ms35212026/11/12 11:46:55 INFO Vacuumed table table=multipart_uploads35222026/11/12 11:46:55 INFO Aborted multipart uploads count=0 kept=035232026/11/12 11:46:55 INFO Received complete push request method=POST path=/35242026/11/12 11:46:55 INFO Vacuumed table table=closures35252026/11/12 11:46:55 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."35262026/11/12 11:46:55 INFO Completed push id=1 objects=7 duration=49.808833ms35272026/11/12 11:46:55 INFO Completed push id=10 objects=2 duration=413.375µs35282026/11/12 11:46:55 INFO Vacuumed table table=objects35292026/11/12 11:46:55 INFO Received complete upload request method=POST path=/35302026/11/12 11:46:55 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."35312026/11/12 11:46:55 INFO Received complete upload request method=POST path=/35322026/11/12 11:46:55 INFO Received uploads request method=POST path=/35332026/11/12 11:46:55 INFO Completed upload id=17 objects=6 duration=1.020208ms35342026/11/12 11:46:55 INFO Completed upload id=21 objects=25 duration=872.542µs35352026/11/12 11:46:55 INFO Received uploads request method=POST path=/35362026/11/12 11:46:55 INFO Received push request method=POST path=/35372026/11/12 11:46:55 INFO Received uploads request method=POST path=/35382026/11/12 11:46:55 INFO Received uploads request method=POST path=/35392026/11/12 11:46:55 INFO Received complete push request method=POST path=/35402026/11/12 11:46:55 INFO Received push request method=POST path=/35412026/11/12 11:46:55 INFO Completed push id=12 objects=3 duration=565.916µs35422026/11/12 11:46:55 INFO Received complete push request method=POST path=/35432026/11/12 11:46:55 INFO Received uploads request method=POST path=/35442026/11/12 11:46:55 INFO Received complete push request method=POST path=/35452026/11/12 11:46:55 INFO Received uploads request method=POST path=/35462026/11/12 11:46:55 INFO Completed push id=27 objects=25 duration=1.331916ms35472026/11/12 11:46:55 INFO Received push request method=POST path=/35482026/11/12 11:46:55 INFO Received uploads request method=POST path=/35492026/11/12 11:46:55 INFO Aborted multipart uploads count=0 kept=035502026/11/12 11:46:55 INFO Received uploads request method=POST path=/35512026/11/12 11:46:55 INFO Received complete upload request method=POST path=/35522026/11/12 11:46:55 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035532026/11/12 11:46:55 INFO Completed push id=3 objects=22 duration=43.535708ms35542026/11/12 11:46:55 INFO Completed upload id=17 objects=5 duration=468.792µs35552026/11/12 11:46:55 INFO Received push request method=POST path=/35562026/11/12 11:46:55 INFO Vacuumed table table=pending_closures35572026/11/12 11:46:55 INFO Received complete upload request method=POST path=/35582026/11/12 11:46:55 INFO Received push request method=POST path=/35592026/11/12 11:46:55 INFO Vacuumed table table=commit_receipts35602026/11/12 11:46:55 INFO Completed upload id=15 objects=9 duration=402.916µs35612026/11/12 11:46:55 INFO Vacuumed table table=pending_objects35622026/11/12 11:46:55 INFO Vacuumed table table=multipart_uploads35632026/11/12 11:46:55 INFO Received push request method=POST path=/35642026/11/12 11:46:55 INFO Vacuumed table table=closures35652026/11/12 11:46:55 INFO Vacuumed table table=objects35662026/11/12 11:46:55 INFO Aborted multipart uploads count=0 kept=035672026/11/12 11:46:55 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035682026/11/12 11:46:55 INFO Received complete push request method=POST path=/35692026/11/12 11:46:55 INFO Completed push id=24 objects=5 duration=835.625µs35702026/11/12 11:46:55 INFO Received uploads request method=POST path=/35712026/11/12 11:46:55 INFO Received complete upload request method=POST path=/35722026/11/12 11:46:55 INFO Completed upload id=16 objects=23 duration=390.333µs35732026/11/12 11:46:55 INFO Received complete upload request method=POST path=/35742026/11/12 11:46:55 INFO Vacuumed table table=pending_closures35752026/11/12 11:46:55 INFO Received complete push request method=POST path=/35762026/11/12 11:46:55 INFO Completed upload id=15 objects=6 duration=450.917µs35772026/11/12 11:46:55 INFO Received complete push request method=POST path=/35782026/11/12 11:46:55 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=035792026/11/12 11:46:55 INFO Vacuumed table table=commit_receipts35802026/11/12 11:46:55 INFO Completed push id=19 objects=3 duration=351.875µs35812026/11/12 11:46:55 INFO Vacuumed table table=pending_closures35822026/11/12 11:46:55 INFO Completed push id=21 objects=17 duration=550.792µs35832026/11/12 11:46:55 INFO Received uploads request method=POST path=/35842026/11/12 11:46:55 INFO Vacuumed table table=commit_receipts35852026/11/12 11:46:55 INFO Received complete push request method=POST path=/35862026/11/12 11:46:55 INFO Completed push id=19 objects=7 duration=476.166µs35872026/11/12 11:46:55 INFO Vacuumed table table=pending_objects35882026/11/12 11:46:55 INFO Vacuumed table table=multipart_uploads35892026/11/12 11:46:55 INFO Received push request method=POST path=/35902026/11/12 11:46:55 INFO Vacuumed table table=closures35912026/11/12 11:46:55 INFO Vacuumed table table=pending_objects35922026/11/12 11:46:55 INFO Vacuumed table table=objects35932026/11/12 11:46:55 INFO Vacuumed table table=multipart_uploads35942026/11/12 11:46:55 INFO Vacuumed table table=closures35952026/11/12 11:46:55 INFO Vacuumed table table=objects35962026/11/12 11:46:55 INFO Received push request method=POST path=/35972026/11/12 11:46:55 INFO Aborted multipart uploads count=0 kept=035982026/11/12 11:46:55 INFO Received uploads request method=POST path=/35992026/11/12 11:46:55 WARN Force mode enabled - objects will be deleted immediately without grace period36002026/11/12 11:46:55 INFO Aborted multipart uploads count=0 kept=036012026/11/12 11:46:55 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036022026/11/12 11:46:55 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036032026/11/12 11:46:55 INFO Received push request method=POST path=/36042026/11/12 11:46:55 INFO Vacuumed table table=pending_closures36052026/11/12 11:46:55 INFO Vacuumed table table=pending_closures36062026/11/12 11:46:55 INFO Received uploads request method=POST path=/36072026/11/12 11:46:55 INFO Vacuumed table table=commit_receipts36082026/11/12 11:46:55 INFO Vacuumed table table=commit_receipts36092026/11/12 11:46:55 INFO Received complete upload request method=POST path=/36102026/11/12 11:46:55 INFO Vacuumed table table=pending_objects36112026/11/12 11:46:55 INFO Received complete upload request method=POST path=/36122026/11/12 11:46:55 INFO Vacuumed table table=multipart_uploads36132026/11/12 11:46:55 INFO Vacuumed table table=pending_objects36142026/11/12 11:46:55 INFO Completed upload id=17 objects=7 duration=469.708µs36152026/11/12 11:46:55 INFO Completed upload id=2 objects=2 duration=490.416µs36162026/11/12 11:46:55 INFO Vacuumed table table=closures36172026/11/12 11:46:55 INFO Vacuumed table table=multipart_uploads36182026/11/12 11:46:56 INFO Vacuumed table table=objects36192026/11/12 11:46:56 INFO Aborted multipart uploads count=0 kept=036202026/11/12 11:46:56 INFO Vacuumed table table=closures36212026/11/12 11:46:56 INFO Received complete upload request method=POST path=/36222026/11/12 11:46:56 INFO Vacuumed table table=objects36232026/11/12 11:46:56 INFO Received uploads request method=POST path=/36242026/11/12 11:46:56 INFO Completed upload id=2 objects=10 duration=592.834µs36252026/11/12 11:46:56 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."36262026/11/12 11:46:56 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."36272026/11/12 11:46:56 INFO Received push request method=POST path=/36282026/11/12 11:46:56 INFO Received complete push request method=POST path=/36292026/11/12 11:46:56 INFO Received complete upload request method=POST path=/36302026/11/12 11:46:56 INFO Received complete push request method=POST path=/36312026/11/12 11:46:56 INFO Completed push id=21 objects=26 duration=744.667µs36322026/11/12 11:46:56 INFO Completed push id=20 objects=4 duration=382.875µs36332026/11/12 11:46:56 INFO Received complete push request method=POST path=/36342026/11/12 11:46:56 INFO Completed push id=6 objects=5 duration=758.25µs36352026/11/12 11:46:56 INFO Received complete upload request method=POST path=/36362026/11/12 11:46:56 INFO Received push request method=POST path=/36372026/11/12 11:46:56 INFO Completed upload id=12 objects=6 duration=341.625µs36382026/11/12 11:46:56 INFO Received push request method=POST path=/36392026/11/12 11:46:56 INFO Completed upload id=1 objects=9 duration=74.054792ms36402026/11/12 11:46:56 INFO Received complete push request method=POST path=/36412026/11/12 11:46:56 INFO Completed push id=17 objects=3 duration=684.834µs36422026/11/12 11:46:56 INFO Received push request method=POST path=/36432026/11/12 11:46:56 INFO Received push request method=POST path=/36442026/11/12 11:46:56 INFO Received complete push request method=POST path=/36452026/11/12 11:46:56 INFO Received uploads request method=POST path=/36462026/11/12 11:46:56 INFO Completed push id=6 objects=26 duration=1.594833ms36472026/11/12 11:46:56 INFO Received uploads request method=POST path=/36482026/11/12 11:46:56 INFO Aborted multipart uploads count=0 kept=036492026/11/12 11:46:56 INFO Received complete upload request method=POST path=/36502026/11/12 11:46:56 WARN Force mode enabled - objects will be deleted immediately without grace period36512026/11/12 11:46:56 INFO Completed upload id=14 objects=6 duration=253µs36522026/11/12 11:46:56 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036532026/11/12 11:46:56 INFO Received push request method=POST path=/36542026/11/12 11:46:56 INFO Received push request method=POST path=/36552026/11/12 11:46:56 INFO Vacuumed table table=pending_closures36562026/11/12 11:46:56 INFO Aborted multipart uploads count=0 kept=036572026/11/12 11:46:56 INFO Vacuumed table table=commit_receipts36582026/11/12 11:46:56 INFO Received push request method=POST path=/36592026/11/12 11:46:56 INFO Vacuumed table table=pending_objects36602026/11/12 11:46:56 INFO Vacuumed table table=multipart_uploads36612026/11/12 11:46:56 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036622026/11/12 11:46:56 INFO Received complete push request method=POST path=/36632026/11/12 11:46:56 INFO Vacuumed table table=closures36642026/11/12 11:46:56 INFO Vacuumed table table=pending_closures36652026/11/12 11:46:56 INFO Received complete upload request method=POST path=/36662026/11/12 11:46:56 INFO Completed push id=23 objects=5 duration=387.333µs36672026/11/12 11:46:56 INFO Vacuumed table table=objects36682026/11/12 11:46:56 INFO Received complete upload request method=POST path=/36692026/11/12 11:46:56 INFO Completed upload id=28 objects=17 duration=332.459µs36702026/11/12 11:46:56 INFO Received complete upload request method=POST path=/36712026/11/12 11:46:56 INFO Completed upload id=12 objects=26 duration=1.630125ms36722026/11/12 11:46:56 INFO Received uploads request method=POST path=/36732026/11/12 11:46:56 INFO Received complete push request method=POST path=/36742026/11/12 11:46:56 INFO Completed upload id=24 objects=3 duration=460.459µs36752026/11/12 11:46:56 INFO Vacuumed table table=commit_receipts36762026/11/12 11:46:56 INFO Received uploads request method=POST path=/36772026/11/12 11:46:56 INFO Completed push id=29 objects=2 duration=433.959µs36782026/11/12 11:46:56 INFO Vacuumed table table=pending_objects36792026/11/12 11:46:56 INFO Received complete upload request method=POST path=/36802026/11/12 11:46:56 INFO Aborted multipart uploads count=0 kept=036812026/11/12 11:46:56 INFO Received complete push request method=POST path=/36822026/11/12 11:46:56 INFO Received complete push request method=POST path=/36832026/11/12 11:46:56 INFO Received complete push request method=POST path=/36842026/11/12 11:46:56 INFO Vacuumed table table=multipart_uploads36852026/11/12 11:46:56 INFO Received complete upload request method=POST path=/36862026/11/12 11:46:56 INFO Completed upload id=7 objects=3 duration=287.958µs36872026/11/12 11:46:56 INFO Completed push id=25 objects=4 duration=562µs36882026/11/12 11:46:56 INFO Completed push id=31 objects=6 duration=731.625µs36892026/11/12 11:46:56 INFO Completed push id=16 objects=8 duration=706.584µs36902026/11/12 11:46:56 WARN Force mode enabled - objects will be deleted immediately without grace period36912026/11/12 11:46:56 INFO Completed upload id=7 objects=10 duration=1.006084ms36922026/11/12 11:46:56 INFO Received complete push request method=POST path=/36932026/11/12 11:46:56 INFO Aborted multipart uploads count=0 kept=036942026/11/12 11:46:56 INFO Received complete upload request method=POST path=/36952026/11/12 11:46:56 INFO Received complete upload request method=POST path=/36962026/11/12 11:46:56 INFO Vacuumed table table=closures36972026/11/12 11:46:56 INFO Completed push id=32 objects=9 duration=426.292µs36982026/11/12 11:46:56 INFO Completed upload id=10 objects=23 duration=618.292µs36992026/11/12 11:46:56 WARN Force mode enabled - objects will be deleted immediately without grace period37002026/11/12 11:46:56 INFO Completed upload id=22 objects=8 duration=391.125µs37012026/11/12 11:46:56 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037022026/11/12 11:46:56 INFO Vacuumed table table=pending_closures37032026/11/12 11:46:56 INFO Received complete push request method=POST path=/37042026/11/12 11:46:56 INFO Received uploads request method=POST path=/37052026/11/12 11:46:56 INFO Vacuumed table table=commit_receipts37062026/11/12 11:46:56 INFO Completed push id=23 objects=5 duration=385.834µs37072026/11/12 11:46:56 INFO Vacuumed table table=pending_objects37082026/11/12 11:46:56 INFO Vacuumed table table=objects37092026/11/12 11:46:56 INFO Received complete upload request method=POST path=/37102026/11/12 11:46:56 INFO Vacuumed table table=multipart_uploads37112026/11/12 11:46:56 INFO Received complete push request method=POST path=/37122026/11/12 11:46:56 INFO Completed upload id=24 objects=8 duration=448.041µs37132026/11/12 11:46:56 INFO Vacuumed table table=closures37142026/11/12 11:46:56 INFO Completed push id=3 objects=5 duration=630.083µs37152026/11/12 11:46:56 INFO Received complete upload request method=POST path=/37162026/11/12 11:46:56 INFO Vacuumed table table=objects37172026/11/12 11:46:56 INFO Completed upload id=28 objects=5 duration=470.25µs37182026/11/12 11:46:56 INFO Received uploads request method=POST path=/37192026/11/12 11:46:56 INFO Received push request method=POST path=/37202026/11/12 11:46:56 INFO Aborted multipart uploads count=0 kept=037212026/11/12 11:46:56 WARN Force mode enabled - objects will be deleted immediately without grace period37222026/11/12 11:46:56 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037232026/11/12 11:46:56 INFO Vacuumed table table=pending_closures37242026/11/12 11:46:56 INFO Received complete upload request method=POST path=/37252026/11/12 11:46:56 INFO Vacuumed table table=commit_receipts37262026/11/12 11:46:56 INFO Received complete upload request method=POST path=/37272026/11/12 11:46:56 INFO Completed upload id=11 objects=6 duration=405.708µs37282026/11/12 11:46:56 INFO Vacuumed table table=pending_objects37292026/11/12 11:46:56 INFO Received complete upload request method=POST path=/37302026/11/12 11:46:56 INFO Vacuumed table table=multipart_uploads37312026/11/12 11:46:56 INFO Received push request method=POST path=/37322026/11/12 11:46:56 INFO Completed upload id=33 objects=9 duration=257µs37332026/11/12 11:46:56 INFO Vacuumed table table=closures37342026/11/12 11:46:56 INFO Vacuumed table table=objects37352026/11/12 11:46:56 INFO Aborted multipart uploads count=0 kept=037362026/11/12 11:46:56 INFO Completed upload id=3 objects=6 duration=52.034375ms37372026/11/12 11:46:56 INFO Received complete push request method=POST path=/37382026/11/12 11:46:56 INFO Received push request method=POST path=/37392026/11/12 11:46:56 WARN Force mode enabled - objects will be deleted immediately without grace period37402026/11/12 11:46:56 INFO Aborted multipart uploads count=0 kept=037412026/11/12 11:46:56 INFO Completed push id=18 objects=23 duration=627µs37422026/11/12 11:46:56 INFO Aborted multipart uploads count=0 kept=037432026/11/12 11:46:56 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037442026/11/12 11:46:56 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037452026/11/12 11:46:56 INFO Received uploads request method=POST path=/37462026/11/12 11:46:56 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.narinfo error="Access Denied."37472026/11/12 11:46:56 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."37482026/11/12 11:46:56 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.narinfo error="Access Denied."37492026/11/12 11:46:56 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."37502026/11/12 11:46:56 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."37512026/11/12 11:46:56 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."37522026/11/12 11:46:56 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."37532026/11/12 11:46:56 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."37542026/11/12 11:46:56 ERROR failed to remove object object=nar/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.nar.zst error="Access Denied."37552026/11/12 11:46:56 ERROR failed to remove object object=nar/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.nar.zst error="Access Denied."37562026/11/12 11:46:56 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."37572026/11/12 11:46:56 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."37582026/11/12 11:46:56 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."37592026/11/12 11:46:56 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."37602026/11/12 11:46:56 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."37612026/11/12 11:46:56 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."37622026/11/12 11:46:56 INFO Vacuumed table table=pending_closures37632026/11/12 11:46:56 INFO Vacuumed table table=commit_receipts37642026/11/12 11:46:56 INFO Aborted multipart uploads count=0 kept=037652026/11/12 11:46:56 INFO Vacuumed table table=pending_objects37662026/11/12 11:46:56 INFO Received complete push request method=POST path=/37672026/11/12 11:46:56 INFO Vacuumed table table=pending_closures37682026/11/12 11:46:56 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=16 objects-failed-to-delete=037692026/11/12 11:46:56 INFO Vacuumed table table=multipart_uploads37702026/11/12 11:46:56 INFO Vacuumed table table=closures37712026/11/12 11:46:56 INFO Vacuumed table table=commit_receipts37722026/11/12 11:46:56 INFO Vacuumed table table=pending_closures37732026/11/12 11:46:56 INFO Completed push id=11 objects=6 duration=10.232208ms37742026/11/12 11:46:56 INFO Received complete upload request method=POST path=/37752026/11/12 11:46:56 INFO Vacuumed table table=objects37762026/11/12 11:46:56 INFO Received complete push request method=POST path=/37772026/11/12 11:46:56 INFO Completed upload id=30 objects=5 duration=823µs37782026/11/12 11:46:56 INFO Vacuumed table table=pending_objects37792026/11/12 11:46:56 INFO Vacuumed table table=multipart_uploads37802026/11/12 11:46:56 INFO Received complete push request method=POST path=/37812026/11/12 11:46:56 INFO Vacuumed table table=closures37822026/11/12 11:46:56 INFO Completed push id=5 objects=9 duration=5.13925ms37832026/11/12 11:46:56 INFO Received push request method=POST path=/37842026/11/12 11:46:56 INFO Completed push id=4 objects=4 duration=432.75µs37852026/11/12 11:46:56 INFO Vacuumed table table=objects37862026/11/12 11:46:56 INFO Vacuumed table table=commit_receipts37872026/11/12 11:46:56 INFO Received push request method=POST path=/37882026/11/12 11:46:56 INFO Vacuumed table table=pending_objects37892026/11/12 11:46:56 INFO Aborted multipart uploads count=0 kept=037902026/11/12 11:46:56 INFO Vacuumed table table=multipart_uploads37912026/11/12 11:46:56 INFO Received push request method=POST path=/37922026/11/12 11:46:56 INFO Received complete push request method=POST path=/37932026/11/12 11:46:56 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037942026/11/12 11:46:56 INFO Vacuumed table table=pending_closures37952026/11/12 11:46:56 INFO Completed push id=12 objects=15 duration=6.588959ms37962026/11/12 11:46:56 INFO Received push request method=POST path=/37972026/11/12 11:46:56 INFO Vacuumed table table=commit_receipts37982026/11/12 11:46:56 INFO Received uploads request method=POST path=/37992026/11/12 11:46:56 INFO Received push request method=POST path=/38002026/11/12 11:46:56 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."38012026/11/12 11:46:56 INFO Received complete push request method=POST path=/38022026/11/12 11:46:56 INFO Received complete push request method=POST path=/38032026/11/12 11:46:56 INFO Vacuumed table table=closures38042026/11/12 11:46:56 INFO Vacuumed table table=pending_objects38052026/11/12 11:46:56 INFO Received uploads request method=POST path=/38062026/11/12 11:46:56 INFO Completed push id=11 objects=9 duration=403.375µs38072026/11/12 11:46:56 INFO Completed push id=5 objects=3 duration=483.083µs38082026/11/12 11:46:56 INFO Vacuumed table table=multipart_uploads38092026/11/12 11:46:56 INFO Vacuumed table table=closures38102026/11/12 11:46:56 INFO Received complete push request method=POST path=/38112026/11/12 11:46:56 INFO Vacuumed table table=objects38122026/11/12 11:46:56 INFO Received push request method=POST path=/38132026/11/12 11:46:56 INFO Received push request method=POST path=/38142026/11/12 11:46:56 INFO Completed push id=18 objects=6 duration=491.375µs38152026/11/12 11:46:56 INFO Received push request method=POST path=/38162026/11/12 11:46:56 INFO Aborted multipart uploads count=0 kept=038172026/11/12 11:46:56 INFO Aborted multipart uploads count=0 kept=038182026/11/12 11:46:56 INFO Aborted multipart uploads count=0 kept=038192026/11/12 11:46:56 INFO Received uploads request method=POST path=/38202026/11/12 11:46:56 INFO Received push request method=POST path=/38212026/11/12 11:46:56 INFO Vacuumed table table=objects38222026/11/12 11:46:56 INFO Received uploads request method=POST path=/38232026/11/12 11:46:56 INFO Received complete push request method=POST path=/38242026/11/12 11:46:56 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=4 objects-deleted-after-grace-period=0 objects-failed-to-delete=038252026/11/12 11:46:56 WARN Force mode enabled - objects will be deleted immediately without grace period38262026/11/12 11:46:56 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038272026/11/12 11:46:56 INFO Received uploads request method=POST path=/38282026/11/12 11:46:56 INFO Aborted multipart uploads count=0 kept=038292026/11/12 11:46:56 INFO Completed push id=31 objects=3 duration=466.208µs38302026/11/12 11:46:56 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038312026/11/12 11:46:56 INFO Vacuumed table table=pending_closures38322026/11/12 11:46:56 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038332026/11/12 11:46:56 INFO Received uploads request method=POST path=/38342026/11/12 11:46:56 INFO Aborted multipart uploads count=0 kept=038352026/11/12 11:46:56 INFO Vacuumed table table=commit_receipts38362026/11/12 11:46:56 INFO Vacuumed table table=pending_closures38372026/11/12 11:46:56 INFO Vacuumed table table=pending_closures38382026/11/12 11:46:56 INFO Vacuumed table table=commit_receipts38392026/11/12 11:46:56 INFO Vacuumed table table=commit_receipts38402026/11/12 11:46:56 INFO Vacuumed table table=pending_objects38412026/11/12 11:46:56 WARN Force mode enabled - objects will be deleted immediately without grace period38422026/11/12 11:46:56 INFO Vacuumed table table=pending_objects38432026/11/12 11:46:56 INFO Received complete upload request method=POST path=/38442026/11/12 11:46:56 INFO Completed upload id=15 objects=6 duration=267.458µs38452026/11/12 11:46:56 INFO Vacuumed table table=multipart_uploads38462026/11/12 11:46:56 INFO Vacuumed table table=multipart_uploads38472026/11/12 11:46:56 INFO Vacuumed table table=pending_objects38482026/11/12 11:46:56 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038492026/11/12 11:46:56 INFO Vacuumed table table=closures38502026/11/12 11:46:56 INFO Vacuumed table table=closures38512026/11/12 11:46:56 INFO Received push request method=POST path=/38522026/11/12 11:46:56 INFO Vacuumed table table=multipart_uploads38532026/11/12 11:46:56 INFO Received complete push request method=POST path=/38542026/11/12 11:46:56 INFO Vacuumed table table=pending_closures38552026/11/12 11:46:56 INFO Vacuumed table table=objects38562026/11/12 11:46:56 INFO Vacuumed table table=objects38572026/11/12 11:46:56 INFO Vacuumed table table=pending_closures38582026/11/12 11:46:56 INFO Vacuumed table table=commit_receipts38592026/11/12 11:46:56 INFO Received complete upload request method=POST path=/38602026/11/12 11:46:56 INFO Completed upload id=20 objects=3 duration=724.417µs38612026/11/12 11:46:56 INFO Vacuumed table table=closures38622026/11/12 11:46:56 INFO Vacuumed table table=pending_objects38632026/11/12 11:46:56 INFO Vacuumed table table=multipart_uploads38642026/11/12 11:46:56 INFO Received push request method=POST path=/38652026/11/12 11:46:56 INFO Vacuumed table table=objects38662026/11/12 11:46:56 INFO Vacuumed table table=closures38672026/11/12 11:46:57 INFO Completed push id=1 objects=7 duration=44.30875ms38682026/11/12 11:46:57 INFO Vacuumed table table=commit_receipts38692026/11/12 11:46:57 INFO Received complete upload request method=POST path=/38702026/11/12 11:46:57 INFO Completed upload id=24 objects=7 duration=316.417µs38712026/11/12 11:46:57 INFO Vacuumed table table=objects38722026/11/12 11:46:57 INFO Aborted multipart uploads count=0 kept=038732026/11/12 11:46:57 INFO Received uploads request method=POST path=/38742026/11/12 11:46:57 WARN Force mode enabled - objects will be deleted immediately without grace period38752026/11/12 11:46:57 INFO Aborted multipart uploads count=0 kept=038762026/11/12 11:46:57 INFO Received uploads request method=POST path=/38772026/11/12 11:46:57 INFO Received push request method=POST path=/38782026/11/12 11:46:57 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038792026/11/12 11:46:57 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038802026/11/12 11:46:57 INFO Received push request method=POST path=/38812026/11/12 11:46:57 INFO Vacuumed table table=pending_closures38822026/11/12 11:46:57 INFO Received complete push request method=POST path=/38832026/11/12 11:46:57 INFO Vacuumed table table=pending_closures38842026/11/12 11:46:57 INFO Vacuumed table table=commit_receipts38852026/11/12 11:46:57 INFO Vacuumed table table=pending_objects38862026/11/12 11:46:57 INFO Received push request method=POST path=/38872026/11/12 11:46:57 INFO Completed push id=26 objects=23 duration=472.208µs38882026/11/12 11:46:57 INFO Vacuumed table table=commit_receipts38892026/11/12 11:46:57 INFO Vacuumed table table=pending_objects38902026/11/12 11:46:57 INFO Vacuumed table table=multipart_uploads38912026/11/12 11:46:57 INFO Received push request method=POST path=/38922026/11/12 11:46:57 INFO Vacuumed table table=multipart_uploads38932026/11/12 11:46:57 INFO Vacuumed table table=pending_objects38942026/11/12 11:46:57 INFO Received push request method=POST path=/38952026/11/12 11:46:57 INFO Received complete upload request method=POST path=/38962026/11/12 11:46:57 INFO Vacuumed table table=closures38972026/11/12 11:46:57 INFO Vacuumed table table=multipart_uploads38982026/11/12 11:46:57 INFO Received complete upload request method=POST path=/38992026/11/12 11:46:57 INFO Completed upload id=23 objects=15 duration=894.708µs39002026/11/12 11:46:57 INFO Aborted multipart uploads count=0 kept=039012026/11/12 11:46:57 INFO Vacuumed table table=objects39022026/11/12 11:46:57 INFO Vacuumed table table=closures39032026/11/12 11:46:57 INFO Vacuumed table table=closures39042026/11/12 11:46:57 INFO Completed upload id=13 objects=14 duration=1.159542ms39052026/11/12 11:46:57 INFO Received complete push request method=POST path=/39062026/11/12 11:46:57 INFO Aborted multipart uploads count=0 kept=039072026/11/12 11:46:57 INFO Vacuumed table table=objects39082026/11/12 11:46:57 INFO Completed push id=9 objects=7 duration=12.335375ms39092026/11/12 11:46:57 INFO Received complete push request method=POST path=/39102026/11/12 11:46:57 WARN Force mode enabled - objects will be deleted immediately without grace period39112026/11/12 11:46:57 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039122026/11/12 11:46:57 INFO Received uploads request method=POST path=/39132026/11/12 11:46:57 INFO Completed push id=32 objects=6 duration=265.167µs39142026/11/12 11:46:57 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039152026/11/12 11:46:57 INFO Vacuumed table table=pending_closures39162026/11/12 11:46:57 INFO Aborted multipart uploads count=0 kept=039172026/11/12 11:46:57 INFO Vacuumed table table=objects39182026/11/12 11:46:57 INFO Vacuumed table table=commit_receipts39192026/11/12 11:46:57 INFO Received complete upload request method=POST path=/39202026/11/12 11:46:57 WARN Force mode enabled - objects will be deleted immediately without grace period39212026/11/12 11:46:57 INFO Vacuumed table table=pending_objects39222026/11/12 11:46:57 INFO Completed upload id=33 objects=7 duration=208.833µs39232026/11/12 11:46:57 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039242026/11/12 11:46:57 INFO Vacuumed table table=multipart_uploads39252026/11/12 11:46:57 INFO Vacuumed table table=closures39262026/11/12 11:46:57 INFO Received complete upload request method=POST path=/39272026/11/12 11:46:57 INFO Received complete upload request method=POST path=/39282026/11/12 11:46:57 INFO Vacuumed table table=pending_closures39292026/11/12 11:46:57 INFO Received complete upload request method=POST path=/39302026/11/12 11:46:57 INFO Vacuumed table table=objects39312026/11/12 11:46:57 INFO Vacuumed table table=commit_receipts39322026/11/12 11:46:57 INFO Completed upload id=11 objects=7 duration=493.333µs39332026/11/12 11:46:57 INFO Completed upload id=19 objects=6 duration=516.375µs39342026/11/12 11:46:57 INFO Completed upload id=34 objects=3 duration=561.75µs39352026/11/12 11:46:57 INFO Received push request method=POST path=/39362026/11/12 11:46:57 INFO Vacuumed table table=pending_closures39372026/11/12 11:46:57 INFO Received uploads request method=POST path=/39382026/11/12 11:46:57 INFO Received complete push request method=POST path=/39392026/11/12 11:46:57 INFO Vacuumed table table=pending_objects39402026/11/12 11:46:57 INFO Completed push id=35 objects=11 duration=6.195292ms39412026/11/12 11:46:57 INFO Vacuumed table table=multipart_uploads39422026/11/12 11:46:57 INFO Received uploads request method=POST path=/39432026/11/12 11:46:57 INFO Vacuumed table table=closures39442026/11/12 11:46:57 INFO Aborted multipart uploads count=0 kept=039452026/11/12 11:46:57 INFO Vacuumed table table=objects39462026/11/12 11:46:57 INFO Received uploads request method=POST path=/39472026/11/12 11:46:57 WARN Force mode enabled - objects will be deleted immediately without grace period39482026/11/12 11:46:57 INFO Vacuumed table table=commit_receipts39492026/11/12 11:46:57 INFO Garbage collection completed failed-uploads-deleted=6 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039502026/11/12 11:46:57 INFO Received complete push request method=POST path=/39512026/11/12 11:46:57 INFO Received complete push request method=POST path=/39522026/11/12 11:46:57 INFO Completed push id=31 objects=6 duration=2.404833ms39532026/11/12 11:46:57 INFO Received push request method=POST path=/39542026/11/12 11:46:57 INFO Completed push id=16 objects=27 duration=726.208µs39552026/11/12 11:46:57 INFO Received uploads request method=POST path=/39562026/11/12 11:46:57 INFO Received uploads request method=POST path=/39572026/11/12 11:46:57 INFO Vacuumed table table=pending_closures39582026/11/12 11:46:57 INFO Received uploads request method=POST path=/39592026/11/12 11:46:57 INFO Received uploads request method=POST path=/39602026/11/12 11:46:57 INFO Received complete push request method=POST path=/39612026/11/12 11:46:57 INFO Vacuumed table table=commit_receipts39622026/11/12 11:46:57 INFO Received uploads request method=POST path=/39632026/11/12 11:46:57 INFO Vacuumed table table=pending_objects39642026/11/12 11:46:57 INFO Completed push id=11 objects=11 duration=1.266791ms39652026/11/12 11:46:57 INFO Received push request method=POST path=/39662026/11/12 11:46:57 INFO Received complete upload request method=POST path=/39672026/11/12 11:46:57 INFO Vacuumed table table=multipart_uploads39682026/11/12 11:46:57 INFO Completed upload id=10 objects=7 duration=716.583µs39692026/11/12 11:46:57 INFO Received complete upload request method=POST path=/39702026/11/12 11:46:57 INFO Received uploads request method=POST path=/39712026/11/12 11:46:57 INFO Completed upload id=22 objects=7 duration=447.875µs39722026/11/12 11:46:57 INFO Vacuumed table table=pending_objects39732026/11/12 11:46:57 INFO Vacuumed table table=multipart_uploads39742026/11/12 11:46:57 INFO Vacuumed table table=closures39752026/11/12 11:46:57 INFO Vacuumed table table=closures39762026/11/12 11:46:57 INFO Received uploads request method=POST path=/39772026/11/12 11:46:57 INFO Received complete push request method=POST path=/39782026/11/12 11:46:57 INFO Received complete push request method=POST path=/39792026/11/12 11:46:57 INFO Vacuumed table table=objects39802026/11/12 11:46:57 INFO Completed push id=24 objects=9 duration=489.167µs39812026/11/12 11:46:57 INFO Received complete upload request method=POST path=/39822026/11/12 11:46:57 INFO Completed push id=23 objects=28 duration=903.208µs39832026/11/12 11:46:57 INFO Completed upload id=18 objects=4 duration=403.875µs39842026/11/12 11:46:57 INFO Received push request method=POST path=/39852026/11/12 11:46:57 INFO Received complete upload request method=POST path=/39862026/11/12 11:46:57 INFO Received push request method=POST path=/39872026/11/12 11:46:57 INFO Completed upload id=27 objects=8 duration=345.375µs39882026/11/12 11:46:57 INFO Received complete upload request method=POST path=/39892026/11/12 11:46:57 INFO Vacuumed table table=objects39902026/11/12 11:46:57 INFO Received push request method=POST path=/39912026/11/12 11:46:57 INFO Completed upload id=19 objects=25 duration=990.875µs39922026/11/12 11:46:57 INFO Received uploads request method=POST path=/39932026/11/12 11:46:57 INFO Aborted multipart uploads count=0 kept=039942026/11/12 11:46:57 INFO Received complete push request method=POST path=/39952026/11/12 11:46:57 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039962026/11/12 11:46:57 INFO Completed push id=27 objects=15 duration=448.541µs39972026/11/12 11:46:57 INFO Aborted multipart uploads count=0 kept=039982026/11/12 11:46:57 WARN Force mode enabled - objects will be deleted immediately without grace period39992026/11/12 11:46:57 INFO Vacuumed table table=pending_closures40002026/11/12 11:46:57 INFO Vacuumed table table=commit_receipts40012026/11/12 11:46:57 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=040022026/11/12 11:46:57 INFO Vacuumed table table=pending_closures40032026/11/12 11:46:57 INFO Received complete push request method=POST path=/40042026/11/12 11:46:57 INFO Vacuumed table table=commit_receipts40052026/11/12 11:46:57 INFO Completed push id=21 objects=3 duration=290.709µs40062026/11/12 11:46:57 INFO Vacuumed table table=pending_objects40072026/11/12 11:46:57 INFO Vacuumed table table=pending_objects40082026/11/12 11:46:57 INFO Received uploads request method=POST path=/40092026/11/12 11:46:57 INFO Vacuumed table table=multipart_uploads40102026/11/12 11:46:57 INFO Vacuumed table table=multipart_uploads40112026/11/12 11:46:57 INFO Vacuumed table table=closures40122026/11/12 11:46:57 INFO Aborted multipart uploads count=0 kept=040132026/11/12 11:46:57 INFO Vacuumed table table=closures40142026/11/12 11:46:57 INFO Vacuumed table table=objects40152026/11/12 11:46:57 INFO Vacuumed table table=objects40162026/11/12 11:46:57 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=3 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040172026/11/12 11:46:57 INFO Aborted multipart uploads count=0 kept=040182026/11/12 11:46:57 INFO Vacuumed table table=pending_closures40192026/11/12 11:46:57 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040202026/11/12 11:46:57 INFO Received push request method=POST path=/40212026/11/12 11:46:57 INFO Vacuumed table table=commit_receipts40222026/11/12 11:46:57 INFO Received complete push request method=POST path=/40232026/11/12 11:46:57 INFO Vacuumed table table=pending_closures40242026/11/12 11:46:57 INFO Received complete push request method=POST path=/40252026/11/12 11:46:57 INFO Received uploads request method=POST path=/40262026/11/12 11:46:57 INFO Vacuumed table table=commit_receipts40272026/11/12 11:46:57 INFO Completed push id=6 objects=5 duration=5.896292ms40282026/11/12 11:46:57 INFO Completed push id=18 objects=10 duration=6.013166ms40292026/11/12 11:46:57 INFO Vacuumed table table=pending_objects40302026/11/12 11:46:57 INFO Vacuumed table table=multipart_uploads40312026/11/12 11:46:57 INFO Vacuumed table table=pending_objects40322026/11/12 11:46:57 INFO Vacuumed table table=closures40332026/11/12 11:46:57 INFO Received push request method=POST path=/40342026/11/12 11:46:57 INFO Vacuumed table table=multipart_uploads40352026/11/12 11:46:57 INFO Vacuumed table table=objects40362026/11/12 11:46:57 INFO Received uploads request method=POST path=/40372026/11/12 11:46:57 INFO Aborted multipart uploads count=0 kept=040382026/11/12 11:46:57 INFO Vacuumed table table=closures40392026/11/12 11:46:57 INFO Received complete push request method=POST path=/40402026/11/12 11:46:57 INFO Aborted multipart uploads count=0 kept=040412026/11/12 11:46:57 WARN Force mode enabled - objects will be deleted immediately without grace period40422026/11/12 11:46:57 INFO Received complete upload request method=POST path=/40432026/11/12 11:46:57 INFO Received push request method=POST path=/40442026/11/12 11:46:57 INFO Vacuumed table table=objects40452026/11/12 11:46:57 INFO Garbage collection completed failed-uploads-deleted=5 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040462026/11/12 11:46:57 WARN Force mode enabled - objects will be deleted immediately without grace period40472026/11/12 11:46:57 INFO Completed upload id=25 objects=16 duration=1.072416ms40482026/11/12 11:46:57 INFO Completed push id=2 objects=7 duration=1.373583ms40492026/11/12 11:46:57 INFO Received push request method=POST path=/40502026/11/12 11:46:57 INFO Aborted multipart uploads count=0 kept=040512026/11/12 11:46:57 INFO Aborted multipart uploads count=0 kept=040522026/11/12 11:46:57 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040532026/11/12 11:46:57 INFO Received push request method=POST path=/40542026/11/12 11:46:57 WARN Force mode enabled - objects will be deleted immediately without grace period40552026/11/12 11:46:57 INFO Received uploads request method=POST path=/40562026/11/12 11:46:57 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040572026/11/12 11:46:57 INFO Received push request method=POST path=/40582026/11/12 11:46:57 INFO Received complete upload request method=POST path=/40592026/11/12 11:46:57 INFO Received complete push request method=POST path=/40602026/11/12 11:46:57 INFO Vacuumed table table=pending_closures40612026/11/12 11:46:57 INFO Completed push id=16 objects=4 duration=6.36725ms40622026/11/12 11:46:57 INFO Completed upload id=10 objects=6 duration=6.3955ms40632026/11/12 11:46:57 INFO Vacuumed table table=commit_receipts40642026/11/12 11:46:57 INFO Received complete push request method=POST path=/40652026/11/12 11:46:57 INFO Vacuumed table table=pending_closures40662026/11/12 11:46:57 INFO Received uploads request method=POST path=/40672026/11/12 11:46:57 INFO Received complete push request method=POST path=/40682026/11/12 11:46:57 INFO Vacuumed table table=commit_receipts40692026/11/12 11:46:57 INFO Vacuumed table table=pending_objects40702026/11/12 11:46:57 INFO Completed push id=30 objects=4 duration=1.655125ms40712026/11/12 11:46:57 INFO Vacuumed table table=multipart_uploads40722026/11/12 11:46:57 INFO Completed push id=2 objects=25 duration=833µs40732026/11/12 11:46:57 INFO Received complete upload request method=POST path=/40742026/11/12 11:46:57 INFO Received uploads request method=POST path=/40752026/11/12 11:46:57 INFO Vacuumed table table=pending_closures40762026/11/12 11:46:57 INFO Received push request method=POST path=/40772026/11/12 11:46:57 INFO Completed upload id=20 objects=5 duration=304.042µs40782026/11/12 11:46:57 INFO Vacuumed table table=pending_objects40792026/11/12 11:46:57 INFO Received push request method=POST path=/40802026/11/12 11:46:57 INFO Vacuumed table table=closures40812026/11/12 11:46:57 INFO Received push request method=POST path=/40822026/11/12 11:46:57 INFO Received complete push request method=POST path=/40832026/11/12 11:46:57 INFO Vacuumed table table=multipart_uploads40842026/11/12 11:46:57 INFO Received uploads request method=POST path=/40852026/11/12 11:46:57 INFO Vacuumed table table=objects40862026/11/12 11:46:57 INFO Vacuumed table table=commit_receipts40872026/11/12 11:46:57 INFO Completed push id=8 objects=6 duration=685.959µs40882026/11/12 11:46:57 INFO Received push request method=POST path=/40892026/11/12 11:46:57 INFO Vacuumed table table=closures40902026/11/12 11:46:57 INFO Aborted multipart uploads count=0 kept=040912026/11/12 11:46:57 INFO Received push request method=POST path=/40922026/11/12 11:46:57 INFO Received push request method=POST path=/40932026/11/12 11:46:57 INFO Vacuumed table table=objects40942026/11/12 11:46:57 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040952026/11/12 11:46:57 INFO Received complete push request method=POST path=/40962026/11/12 11:46:57 INFO Received push request method=POST path=/40972026/11/12 11:46:57 INFO Received push request method=POST path=/40982026/11/12 11:46:57 INFO Completed push id=32 objects=7 duration=399.667µs40992026/11/12 11:46:57 INFO Received uploads request method=POST path=/41002026/11/12 11:46:57 INFO Received complete push request method=POST path=/41012026/11/12 11:46:57 INFO Completed push id=26 objects=4 duration=393.5µs41022026/11/12 11:46:57 INFO Received complete push request method=POST path=/41032026/11/12 11:46:57 INFO Vacuumed table table=pending_objects41042026/11/12 11:46:57 INFO Vacuumed table table=multipart_uploads41052026/11/12 11:46:57 INFO Completed push id=4 objects=6 duration=817µs41062026/11/12 11:46:57 INFO Received push request method=POST path=/41072026/11/12 11:46:57 INFO Aborted multipart uploads count=0 kept=041082026/11/12 11:46:57 INFO Vacuumed table table=pending_closures41092026/11/12 11:46:57 INFO Received complete upload request method=POST path=/41102026/11/12 11:46:57 INFO Received uploads request method=POST path=/41112026/11/12 11:46:57 WARN Force mode enabled - objects will be deleted immediately without grace period41122026/11/12 11:46:57 INFO Aborted multipart uploads count=0 kept=041132026/11/12 11:46:57 INFO Received complete push request method=POST path=/41142026/11/12 11:46:57 INFO Vacuumed table table=closures41152026/11/12 11:46:57 INFO Completed upload id=28 objects=13 duration=576.166µs41162026/11/12 11:46:57 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041172026/11/12 11:46:57 WARN Force mode enabled - objects will be deleted immediately without grace period41182026/11/12 11:46:57 INFO Received complete upload request method=POST path=/41192026/11/12 11:46:57 INFO Completed push id=12 objects=7 duration=597.791µs41202026/11/12 11:46:57 INFO Completed upload id=22 objects=5 duration=5.850917ms41212026/11/12 11:46:57 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041222026/11/12 11:46:57 INFO Received complete push request method=POST path=/41232026/11/12 11:46:57 INFO Completed push id=29 objects=3 duration=368.625µs41242026/11/12 11:46:57 INFO Vacuumed table table=pending_closures41252026/11/12 11:46:57 INFO Vacuumed table table=commit_receipts41262026/11/12 11:46:57 INFO Vacuumed table table=commit_receipts41272026/11/12 11:46:57 INFO Vacuumed table table=pending_closures41282026/11/12 11:46:57 INFO Aborted multipart uploads count=0 kept=041292026/11/12 11:46:57 INFO Received complete push request method=POST path=/41302026/11/12 11:46:57 INFO Received complete upload request method=POST path=/41312026/11/12 11:46:57 INFO Vacuumed table table=commit_receipts41322026/11/12 11:46:57 INFO Vacuumed table table=pending_objects41332026/11/12 11:46:57 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=041342026/11/12 11:46:57 INFO Completed upload id=31 objects=2 duration=361.875µs41352026/11/12 11:46:57 INFO Completed push id=28 objects=3 duration=582.375µs41362026/11/12 11:46:57 INFO Vacuumed table table=multipart_uploads41372026/11/12 11:46:57 INFO Vacuumed table table=objects41382026/11/12 11:46:57 INFO Vacuumed table table=pending_objects41392026/11/12 11:46:57 INFO Received complete push request method=POST path=/41402026/11/12 11:46:57 INFO Vacuumed table table=closures41412026/11/12 11:46:57 INFO Received complete push request method=POST path=/41422026/11/12 11:46:57 INFO Vacuumed table table=pending_objects41432026/11/12 11:46:57 INFO Vacuumed table table=multipart_uploads41442026/11/12 11:46:57 INFO Vacuumed table table=pending_closures41452026/11/12 11:46:57 INFO Completed push id=33 objects=6 duration=261.667µs41462026/11/12 11:46:57 INFO Vacuumed table table=objects41472026/11/12 11:46:57 INFO Vacuumed table table=closures41482026/11/12 11:46:57 INFO Vacuumed table table=multipart_uploads41492026/11/12 11:46:57 INFO Vacuumed table table=commit_receipts41502026/11/12 11:46:57 INFO Completed push id=29 objects=15 duration=397.125µs41512026/11/12 11:46:57 INFO Vacuumed table table=objects41522026/11/12 11:46:57 INFO Aborted multipart uploads count=0 kept=041532026/11/12 11:46:57 INFO Aborted multipart uploads count=0 kept=041542026/11/12 11:46:57 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041552026/11/12 11:46:57 INFO Received complete push request method=POST path=/41562026/11/12 11:46:57 WARN Force mode enabled - objects will be deleted immediately without grace period41572026/11/12 11:46:57 INFO Vacuumed table table=pending_closures41582026/11/12 11:46:57 INFO Completed push id=30 objects=6 duration=422.375µs41592026/11/12 11:46:57 INFO Received push request method=POST path=/41602026/11/12 11:46:57 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041612026/11/12 11:46:57 INFO Vacuumed table table=commit_receipts41622026/11/12 11:46:57 INFO Vacuumed table table=closures41632026/11/12 11:46:57 INFO Received complete push request method=POST path=/41642026/11/12 11:46:57 INFO Vacuumed table table=pending_objects41652026/11/12 11:46:57 INFO Received complete upload request method=POST path=/41662026/11/12 11:46:57 INFO Vacuumed table table=pending_objects41672026/11/12 11:46:57 INFO Completed push id=28 objects=19 duration=367.583µs41682026/11/12 11:46:57 INFO Received push request method=POST path=/41692026/11/12 11:46:57 INFO Vacuumed table table=multipart_uploads41702026/11/12 11:46:57 INFO Completed upload id=31 objects=3 duration=421.166µs41712026/11/12 11:46:57 INFO Vacuumed table table=pending_closures41722026/11/12 11:46:57 INFO Vacuumed table table=multipart_uploads41732026/11/12 11:46:57 INFO Vacuumed table table=closures41742026/11/12 11:46:57 INFO Received push request method=POST path=/41752026/11/12 11:46:57 INFO Vacuumed table table=commit_receipts41762026/11/12 11:46:57 INFO Aborted multipart uploads count=0 kept=041772026/11/12 11:46:58 INFO Vacuumed table table=closures41782026/11/12 11:46:58 INFO Vacuumed table table=objects41792026/11/12 11:46:58 INFO Aborted multipart uploads count=0 kept=041802026/11/12 11:46:58 INFO Vacuumed table table=objects41812026/11/12 11:46:58 INFO Vacuumed table table=pending_objects41822026/11/12 11:46:58 INFO Vacuumed table table=objects41832026/11/12 11:46:58 INFO Vacuumed table table=multipart_uploads41842026/11/12 11:46:58 WARN Force mode enabled - objects will be deleted immediately without grace period41852026/11/12 11:46:58 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041862026/11/12 11:46:58 INFO Vacuumed table table=closures41872026/11/12 11:46:58 INFO Received complete upload request method=POST path=/41882026/11/12 11:46:58 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041892026/11/12 11:46:58 INFO Received push request method=POST path=/41902026/11/12 11:46:58 INFO Vacuumed table table=objects41912026/11/12 11:46:58 INFO Completed upload id=9 objects=2 duration=6.389ms41922026/11/12 11:46:58 INFO Received uploads request method=POST path=/41932026/11/12 11:46:58 INFO Received push request method=POST path=/41942026/11/12 11:46:58 INFO Vacuumed table table=pending_closures41952026/11/12 11:46:58 INFO Received complete push request method=POST path=/41962026/11/12 11:46:58 INFO Received push request method=POST path=/41972026/11/12 11:46:58 INFO Vacuumed table table=commit_receipts41982026/11/12 11:46:58 INFO Vacuumed table table=pending_closures41992026/11/12 11:46:58 INFO Completed push id=8 objects=3 duration=515.625µs42002026/11/12 11:46:58 INFO Vacuumed table table=commit_receipts42012026/11/12 11:46:58 INFO Received push request method=POST path=/42022026/11/12 11:46:58 INFO Aborted multipart uploads count=0 kept=042032026/11/12 11:46:58 WARN Force mode enabled - objects will be deleted immediately without grace period42042026/11/12 11:46:58 INFO Received push request method=POST path=/42052026/11/12 11:46:58 INFO Received uploads request method=POST path=/42062026/11/12 11:46:58 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042072026/11/12 11:46:58 INFO Aborted multipart uploads count=0 kept=042082026/11/12 11:46:58 INFO Received uploads request method=POST path=/42092026/11/12 11:46:58 INFO Received uploads request method=POST path=/42102026/11/12 11:46:58 INFO Vacuumed table table=pending_objects42112026/11/12 11:46:58 INFO Vacuumed table table=pending_closures42122026/11/12 11:46:58 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042132026/11/12 11:46:58 INFO Received complete upload request method=POST path=/42142026/11/12 11:46:58 INFO Received uploads request method=POST path=/42152026/11/12 11:46:58 INFO Vacuumed table table=multipart_uploads42162026/11/12 11:46:58 INFO Vacuumed table table=pending_objects42172026/11/12 11:46:58 INFO Vacuumed table table=closures42182026/11/12 11:46:58 INFO Vacuumed table table=pending_closures42192026/11/12 11:46:58 INFO Vacuumed table table=commit_receipts42202026/11/12 11:46:58 INFO Completed upload id=13 objects=10 duration=6.298583ms42212026/11/12 11:46:58 INFO Vacuumed table table=multipart_uploads42222026/11/12 11:46:58 INFO Vacuumed table table=commit_receipts42232026/11/12 11:46:58 INFO Aborted multipart uploads count=0 kept=042242026/11/12 11:46:58 INFO Vacuumed table table=objects42252026/11/12 11:46:58 INFO Vacuumed table table=closures42262026/11/12 11:46:58 WARN Force mode enabled - objects will be deleted immediately without grace period42272026/11/12 11:46:58 INFO Vacuumed table table=objects42282026/11/12 11:46:58 INFO Vacuumed table table=pending_objects42292026/11/12 11:46:58 INFO Vacuumed table table=pending_objects42302026/11/12 11:46:58 INFO Received push request method=POST path=/42312026/11/12 11:46:58 INFO Vacuumed table table=multipart_uploads42322026/11/12 11:46:58 INFO Vacuumed table table=multipart_uploads42332026/11/12 11:46:58 INFO Vacuumed table table=closures42342026/11/12 11:46:58 INFO Vacuumed table table=closures42352026/11/12 11:46:58 INFO Vacuumed table table=objects42362026/11/12 11:46:58 INFO Vacuumed table table=objects42372026/11/12 11:46:58 INFO Received complete push request method=POST path=/42382026/11/12 11:46:58 INFO Received uploads request method=POST path=/42392026/11/12 11:46:58 INFO Completed push id=29 objects=10 duration=325.167µs42402026/11/12 11:46:58 INFO Received uploads request method=POST path=/42412026/11/12 11:46:58 INFO Received push request method=POST path=/42422026/11/12 11:46:58 INFO Received push request method=POST path=/42432026/11/12 11:46:58 INFO Received complete push request method=POST path=/42442026/11/12 11:46:58 INFO Received push request method=POST path=/42452026/11/12 11:46:58 INFO Received uploads request method=POST path=/42462026/11/12 11:46:58 INFO Completed push id=4 objects=4 duration=1.266459ms42472026/11/12 11:46:58 INFO Aborted multipart uploads count=0 kept=042482026/11/12 11:46:58 INFO Received complete push request method=POST path=/42492026/11/12 11:46:58 WARN Force mode enabled - objects will be deleted immediately without grace period42502026/11/12 11:46:58 INFO Received uploads request method=POST path=/42512026/11/12 11:46:58 INFO Completed push id=31 objects=14 duration=761.25µs42522026/11/12 11:46:58 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042532026/11/12 11:46:58 INFO Received complete upload request method=POST path=/42542026/11/12 11:46:58 INFO Received uploads request method=POST path=/42552026/11/12 11:46:58 INFO Completed upload id=23 objects=18 duration=916.417µs42562026/11/12 11:46:58 INFO Vacuumed table table=pending_closures42572026/11/12 11:46:58 INFO Received uploads request method=POST path=/42582026/11/12 11:46:58 INFO Vacuumed table table=commit_receipts42592026/11/12 11:46:58 INFO Received complete push request method=POST path=/42602026/11/12 11:46:58 INFO Received push request method=POST path=/42612026/11/12 11:46:58 INFO Completed push id=1 objects=9 duration=455.542µs42622026/11/12 11:46:58 INFO Received complete upload request method=POST path=/42632026/11/12 11:46:58 INFO Completed upload id=22 objects=22 duration=447.917µs42642026/11/12 11:46:58 INFO Vacuumed table table=pending_objects42652026/11/12 11:46:58 INFO Received push request method=POST path=/42662026/11/12 11:46:58 INFO Vacuumed table table=multipart_uploads42672026/11/12 11:46:58 INFO Received complete push request method=POST path=/42682026/11/12 11:46:58 INFO Vacuumed table table=closures42692026/11/12 11:46:58 INFO Completed push id=26 objects=5 duration=5.856708ms42702026/11/12 11:46:58 INFO Received complete upload request method=POST path=/42712026/11/12 11:46:58 INFO Vacuumed table table=objects42722026/11/12 11:46:58 INFO Completed upload id=30 objects=6 duration=1.04175ms42732026/11/12 11:46:58 INFO Received complete upload request method=POST path=/42742026/11/12 11:46:58 INFO Completed upload id=24 objects=14 duration=194.083µs42752026/11/12 11:46:58 INFO Received complete upload request method=POST path=/42762026/11/12 11:46:58 INFO Received complete push request method=POST path=/42772026/11/12 11:46:58 INFO Completed upload id=31 objects=5 duration=207.791µs42782026/11/12 11:46:58 INFO Completed push id=19 objects=3 duration=499.959µs42792026/11/12 11:46:58 INFO Received complete upload request method=POST path=/42802026/11/12 11:46:58 INFO Completed upload id=34 objects=3 duration=206.417µs42812026/11/12 11:46:58 INFO Received uploads request method=POST path=/42822026/11/12 11:46:58 INFO Aborted multipart uploads count=0 kept=042832026/11/12 11:46:58 WARN Force mode enabled - objects will be deleted immediately without grace period42842026/11/12 11:46:58 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042852026/11/12 11:46:58 INFO Received complete push request method=POST path=/42862026/11/12 11:46:58 INFO Received uploads request method=POST path=/42872026/11/12 11:46:58 INFO Completed push id=6 objects=6 duration=486.416µs42882026/11/12 11:46:58 INFO Vacuumed table table=pending_closures42892026/11/12 11:46:58 INFO Vacuumed table table=commit_receipts42902026/11/12 11:46:58 INFO Received push request method=POST path=/42912026/11/12 11:46:58 INFO Received complete upload request method=POST path=/42922026/11/12 11:46:58 INFO Completed upload id=25 objects=4 duration=283.75µs42932026/11/12 11:46:58 INFO Received push request method=POST path=/42942026/11/12 11:46:58 INFO Vacuumed table table=pending_objects42952026/11/12 11:46:58 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."42962026/11/12 11:46:58 INFO Received complete push request method=POST path=/42972026/11/12 11:46:58 INFO Vacuumed table table=multipart_uploads42982026/11/12 11:46:58 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."42992026/11/12 11:46:58 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."43002026/11/12 11:46:58 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."43012026/11/12 11:46:58 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."43022026/11/12 11:46:58 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."43032026/11/12 11:46:58 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."43042026/11/12 11:46:58 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."43052026/11/12 11:46:58 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."43062026/11/12 11:46:58 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."43072026/11/12 11:46:58 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."43082026/11/12 11:46:58 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."43092026/11/12 11:46:58 INFO Completed push id=20 objects=8 duration=349.958µs43102026/11/12 11:46:58 INFO Vacuumed table table=closures43112026/11/12 11:46:58 INFO Received uploads request method=POST path=/43122026/11/12 11:46:58 INFO Received uploads request method=POST path=/43132026/11/12 11:46:58 INFO Received push request method=POST path=/43142026/11/12 11:46:58 INFO Vacuumed table table=objects43152026/11/12 11:46:58 INFO Received push request method=POST path=/43162026/11/12 11:46:58 INFO Received uploads request method=POST path=/43172026/11/12 11:46:58 INFO Received push request method=POST path=/43182026/11/12 11:46:58 INFO Received push request method=POST path=/43192026/11/12 11:46:58 INFO Received complete push request method=POST path=/43202026/11/12 11:46:58 INFO Received complete push request method=POST path=/43212026/11/12 11:46:58 INFO Aborted multipart uploads count=0 kept=043222026/11/12 11:46:58 INFO Completed push id=1 objects=11 duration=489µs43232026/11/12 11:46:58 INFO Completed push id=27 objects=11 duration=259µs43242026/11/12 11:46:58 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=043252026/11/12 11:46:58 INFO Received complete upload request method=POST path=/43262026/11/12 11:46:58 INFO Vacuumed table table=pending_closures43272026/11/12 11:46:58 INFO Completed upload id=29 objects=26 duration=207.333µs43282026/11/12 11:46:58 INFO Vacuumed table table=commit_receipts43292026/11/12 11:46:58 INFO Received complete push request method=POST path=/43302026/11/12 11:46:58 INFO Vacuumed table table=pending_objects43312026/11/12 11:46:58 INFO Completed push id=30 objects=8 duration=335.333µs43322026/11/12 11:46:58 INFO Vacuumed table table=multipart_uploads43332026/11/12 11:46:58 INFO Received complete upload request method=POST path=/43342026/11/12 11:46:58 INFO Completed upload id=31 objects=5 duration=192.334µs43352026/11/12 11:46:58 INFO Received complete push request method=POST path=/43362026/11/12 11:46:58 INFO Vacuumed table table=closures43372026/11/12 11:46:58 INFO Completed push id=32 objects=3 duration=290.125µs43382026/11/12 11:46:58 INFO Received complete push request method=POST path=/43392026/11/12 11:46:58 INFO Vacuumed table table=objects43402026/11/12 11:46:58 INFO Aborted multipart uploads count=0 kept=043412026/11/12 11:46:58 INFO Completed push id=19 objects=8 duration=1.865958ms43422026/11/12 11:46:58 WARN Force mode enabled - objects will be deleted immediately without grace period43432026/11/12 11:46:58 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043442026/11/12 11:46:58 INFO Received uploads request method=POST path=/43452026/11/12 11:46:58 INFO Received push request method=POST path=/43462026/11/12 11:46:58 INFO Aborted multipart uploads count=0 kept=043472026/11/12 11:46:58 INFO Received complete upload request method=POST path=/43482026/11/12 11:46:58 INFO Received complete upload request method=POST path=/43492026/11/12 11:46:58 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043502026/11/12 11:46:58 INFO Completed upload id=16 objects=3 duration=393.583µs43512026/11/12 11:46:58 INFO Vacuumed table table=pending_closures43522026/11/12 11:46:58 INFO Vacuumed table table=pending_closures43532026/11/12 11:46:58 INFO Completed upload id=10 objects=2 duration=11.113708ms43542026/11/12 11:46:58 INFO Received push request method=POST path=/43552026/11/12 11:46:58 INFO Vacuumed table table=commit_receipts43562026/11/12 11:46:58 INFO Vacuumed table table=commit_receipts43572026/11/12 11:46:58 INFO Received push request method=POST path=/43582026/11/12 11:46:58 INFO Vacuumed table table=pending_objects43592026/11/12 11:46:58 INFO Aborted multipart uploads count=0 kept=043602026/11/12 11:46:58 INFO Vacuumed table table=multipart_uploads43612026/11/12 11:46:58 INFO Received push request method=POST path=/43622026/11/12 11:46:58 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=043632026/11/12 11:46:58 INFO Vacuumed table table=pending_closures43642026/11/12 11:46:58 INFO Vacuumed table table=commit_receipts43652026/11/12 11:46:58 INFO Vacuumed table table=closures43662026/11/12 11:46:58 INFO Vacuumed table table=pending_objects43672026/11/12 11:46:58 INFO Vacuumed table table=pending_objects43682026/11/12 11:46:58 INFO Received complete push request method=POST path=/43692026/11/12 11:46:58 INFO Vacuumed table table=multipart_uploads43702026/11/12 11:46:58 INFO Vacuumed table table=multipart_uploads43712026/11/12 11:46:58 INFO Completed push id=7 objects=27 duration=596.041µs43722026/11/12 11:46:58 INFO Vacuumed table table=objects43732026/11/12 11:46:58 INFO Vacuumed table table=closures43742026/11/12 11:46:58 INFO Vacuumed table table=closures43752026/11/12 11:46:58 INFO Vacuumed table table=objects43762026/11/12 11:46:58 INFO Vacuumed table table=objects43772026/11/12 11:46:58 INFO Aborted multipart uploads count=0 kept=043782026/11/12 11:46:58 WARN Force mode enabled - objects will be deleted immediately without grace period43792026/11/12 11:46:58 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043802026/11/12 11:46:58 INFO Vacuumed table table=pending_closures43812026/11/12 11:46:58 INFO Vacuumed table table=commit_receipts43822026/11/12 11:46:58 INFO Received complete push request method=POST path=/43832026/11/12 11:46:58 INFO Vacuumed table table=pending_objects43842026/11/12 11:46:58 INFO Received push request method=POST path=/43852026/11/12 11:46:58 INFO Completed push id=7 objects=7 duration=382.25µs43862026/11/12 11:46:58 INFO Vacuumed table table=multipart_uploads43872026/11/12 11:46:58 INFO Vacuumed table table=closures43882026/11/12 11:46:58 INFO Received complete push request method=POST path=/43892026/11/12 11:46:58 INFO Vacuumed table table=objects43902026/11/12 11:46:58 INFO Completed push id=12 objects=6 duration=432.416µs43912026/11/12 11:46:58 INFO Received push request method=POST path=/43922026/11/12 11:46:58 INFO Received uploads request method=POST path=/43932026/11/12 11:46:58 INFO Received uploads request method=POST path=/43942026/11/12 11:46:58 INFO Received uploads request method=POST path=/43952026/11/12 11:46:58 INFO Received push request method=POST path=/43962026/11/12 11:46:58 INFO Received uploads request method=POST path=/43972026/11/12 11:46:58 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=11 objects-failed-to-delete=043982026/11/12 11:46:58 INFO Vacuumed table table=pending_closures43992026/11/12 11:46:58 INFO Vacuumed table table=commit_receipts44002026/11/12 11:46:58 INFO Vacuumed table table=pending_objects44012026/11/12 11:46:58 INFO Vacuumed table table=multipart_uploads44022026/11/12 11:46:58 INFO Vacuumed table table=closures44032026/11/12 11:46:58 INFO Vacuumed table table=objects44042026/11/12 11:46:59 INFO Received push request method=POST path=/44052026/11/12 11:46:59 INFO Received push request method=POST path=/44062026/11/12 11:46:59 INFO Received complete push request method=POST path=/44072026/11/12 11:46:59 INFO Received complete upload request method=POST path=/44082026/11/12 11:46:59 INFO Completed push id=13 objects=15 duration=729.333µs44092026/11/12 11:46:59 INFO Completed upload id=3 objects=9 duration=660.5µs44102026/11/12 11:46:59 INFO Aborted multipart uploads count=0 kept=044112026/11/12 11:46:59 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."44122026/11/12 11:46:59 INFO Received complete push request method=POST path=/44132026/11/12 11:46:59 INFO Received complete upload request method=POST path=/44142026/11/12 11:46:59 INFO Completed push id=14 objects=4 duration=1.325625ms44152026/11/12 11:46:59 INFO Completed upload id=4 objects=14 duration=1.522ms44162026/11/12 11:46:59 INFO Received complete upload request method=POST path=/44172026/11/12 11:46:59 INFO Completed upload id=2 objects=7 duration=419.917µs44182026/11/12 11:46:59 INFO Received push request method=POST path=/44192026/11/12 11:46:59 INFO Received uploads request method=POST path=/44202026/11/12 11:46:59 INFO Aborted multipart uploads count=0 kept=044212026/11/12 11:46:59 WARN Force mode enabled - objects will be deleted immediately without grace period44222026/11/12 11:46:59 INFO Received complete push request method=POST path=/44232026/11/12 11:46:59 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044242026/11/12 11:46:59 INFO Completed push id=5 objects=6 duration=485.459µs44252026/11/12 11:46:59 INFO Aborted multipart uploads count=0 kept=044262026/11/12 11:46:59 INFO Aborted multipart uploads count=0 kept=044272026/11/12 11:46:59 INFO Received complete push request method=POST path=/44282026/11/12 11:46:59 WARN Force mode enabled - objects will be deleted immediately without grace period44292026/11/12 11:46:59 INFO Vacuumed table table=pending_closures44302026/11/12 11:46:59 WARN Force mode enabled - objects will be deleted immediately without grace period44312026/11/12 11:46:59 INFO Completed push id=11 objects=8 duration=1.180542ms44322026/11/12 11:46:59 INFO Vacuumed table table=commit_receipts44332026/11/12 11:46:59 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044342026/11/12 11:46:59 INFO Aborted multipart uploads count=0 kept=044352026/11/12 11:46:59 INFO Vacuumed table table=pending_closures44362026/11/12 11:46:59 INFO Vacuumed table table=commit_receipts44372026/11/12 11:46:59 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044382026/11/12 11:46:59 INFO Vacuumed table table=pending_closures44392026/11/12 11:46:59 INFO Vacuumed table table=pending_objects44402026/11/12 11:46:59 INFO Vacuumed table table=multipart_uploads44412026/11/12 11:46:59 INFO Vacuumed table table=commit_receipts44422026/11/12 11:46:59 INFO Vacuumed table table=pending_objects44432026/11/12 11:46:59 INFO Vacuumed table table=closures44442026/11/12 11:46:59 INFO Vacuumed table table=multipart_uploads44452026/11/12 11:46:59 INFO Vacuumed table table=pending_objects44462026/11/12 11:46:59 INFO Vacuumed table table=closures44472026/11/12 11:46:59 INFO Vacuumed table table=objects44482026/11/12 11:46:59 INFO Vacuumed table table=multipart_uploads44492026/11/12 11:46:59 INFO Vacuumed table table=closures44502026/11/12 11:46:59 INFO Vacuumed table table=objects44512026/11/12 11:46:59 INFO Vacuumed table table=objects44522026/11/12 11:46:59 INFO Received push request method=POST path=/44532026/11/12 11:46:59 INFO Aborted multipart uploads count=0 kept=044542026/11/12 11:46:59 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044552026/11/12 11:46:59 INFO Vacuumed table table=pending_closures44562026/11/12 11:46:59 INFO Vacuumed table table=commit_receipts44572026/11/12 11:46:59 INFO Received complete push request method=POST path=/44582026/11/12 11:46:59 INFO Completed push id=20 objects=3 duration=463.125µs44592026/11/12 11:46:59 INFO Vacuumed table table=pending_objects44602026/11/12 11:46:59 INFO Vacuumed table table=multipart_uploads44612026/11/12 11:46:59 INFO Vacuumed table table=closures44622026/11/12 11:46:59 INFO Vacuumed table table=objects44632026/11/12 11:46:59 INFO Received complete push request method=POST path=/44642026/11/12 11:46:59 INFO Received push request method=POST path=/44652026/11/12 11:46:59 INFO Completed push id=13 objects=22 duration=420.459µs44662026/11/12 11:46:59 INFO Received push request method=POST path=/44672026/11/12 11:46:59 INFO Received complete push request method=POST path=/44682026/11/12 11:46:59 INFO Completed push id=4 objects=6 duration=728.958µs44692026/11/12 11:46:59 INFO Received complete push request method=POST path=/44702026/11/12 11:46:59 INFO Completed push id=21 objects=6 duration=349.583µs44712026/11/12 11:46:59 INFO Received push request method=POST path=/44722026/11/12 11:46:59 INFO Received uploads request method=POST path=/44732026/11/12 11:46:59 INFO Received complete upload request method=POST path=/44742026/11/12 11:46:59 INFO Received complete upload request method=POST path=/44752026/11/12 11:46:59 INFO Received push request method=POST path=/44762026/11/12 11:46:59 INFO Completed upload id=22 objects=7 duration=430.041µs44772026/11/12 11:46:59 INFO Completed upload id=22 objects=6 duration=304.625µs44782026/11/12 11:46:59 INFO Received complete push request method=POST path=/44792026/11/12 11:46:59 INFO Aborted multipart uploads count=0 kept=044802026/11/12 11:46:59 INFO Received uploads request method=POST path=/44812026/11/12 11:46:59 INFO Completed push id=24 objects=4 duration=249.959µs44822026/11/12 11:46:59 INFO Received complete push request method=POST path=/44832026/11/12 11:46:59 WARN Force mode enabled - objects will be deleted immediately without grace period44842026/11/12 11:46:59 INFO Received complete push request method=POST path=/44852026/11/12 11:46:59 INFO Completed push id=5 objects=8 duration=2.346958ms44862026/11/12 11:46:59 INFO Completed push id=25 objects=9 duration=299.375µs44872026/11/12 11:46:59 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044882026/11/12 11:46:59 INFO Received complete upload request method=POST path=/44892026/11/12 11:46:59 INFO Received push request method=POST path=/44902026/11/12 11:46:59 INFO Received uploads request method=POST path=/44912026/11/12 11:46:59 INFO Vacuumed table table=pending_closures44922026/11/12 11:46:59 INFO Completed upload id=14 objects=16 duration=1.126792ms44932026/11/12 11:46:59 INFO Received complete upload request method=POST path=/44942026/11/12 11:46:59 INFO Vacuumed table table=commit_receipts44952026/11/12 11:46:59 INFO Completed upload id=26 objects=3 duration=1.207833ms44962026/11/12 11:46:59 INFO Received uploads request method=POST path=/44972026/11/12 11:46:59 INFO Received push request method=POST path=/44982026/11/12 11:46:59 INFO Vacuumed table table=pending_objects44992026/11/12 11:46:59 INFO Vacuumed table table=multipart_uploads45002026/11/12 11:46:59 INFO Received complete upload request method=POST path=/45012026/11/12 11:46:59 INFO Vacuumed table table=closures45022026/11/12 11:46:59 INFO Completed upload id=8 objects=6 duration=6.223584ms45032026/11/12 11:46:59 INFO Vacuumed table table=objects45042026/11/12 11:46:59 INFO Received push request method=POST path=/45052026/11/12 11:46:59 INFO Aborted multipart uploads count=0 kept=045062026/11/12 11:46:59 INFO Received uploads request method=POST path=/45072026/11/12 11:46:59 INFO Received uploads request method=POST path=/45082026/11/12 11:46:59 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045092026/11/12 11:46:59 INFO Vacuumed table table=pending_closures45102026/11/12 11:46:59 INFO Vacuumed table table=commit_receipts45112026/11/12 11:46:59 INFO Vacuumed table table=pending_objects45122026/11/12 11:46:59 INFO Aborted multipart uploads count=0 kept=045132026/11/12 11:46:59 INFO Received uploads request method=POST path=/45142026/11/12 11:46:59 INFO Vacuumed table table=multipart_uploads45152026/11/12 11:46:59 INFO Received complete push request method=POST path=/45162026/11/12 11:46:59 WARN Force mode enabled - objects will be deleted immediately without grace period45172026/11/12 11:46:59 INFO Vacuumed table table=closures45182026/11/12 11:46:59 INFO Completed push id=13 objects=6 duration=5.77625ms45192026/11/12 11:46:59 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045202026/11/12 11:46:59 INFO Vacuumed table table=objects45212026/11/12 11:46:59 INFO Received uploads request method=POST path=/45222026/11/12 11:46:59 INFO Received push request method=POST path=/45232026/11/12 11:46:59 INFO Vacuumed table table=pending_closures45242026/11/12 11:46:59 INFO Vacuumed table table=commit_receipts45252026/11/12 11:46:59 INFO Received push request method=POST path=/45262026/11/12 11:46:59 INFO Vacuumed table table=pending_objects45272026/11/12 11:46:59 INFO Received uploads request method=POST path=/45282026/11/12 11:46:59 INFO Received uploads request method=POST path=/45292026/11/12 11:46:59 INFO Vacuumed table table=multipart_uploads45302026/11/12 11:46:59 INFO Received uploads request method=POST path=/45312026/11/12 11:46:59 INFO Received uploads request method=POST path=/45322026/11/12 11:46:59 INFO Received complete upload request method=POST path=/45332026/11/12 11:46:59 INFO Vacuumed table table=closures45342026/11/12 11:46:59 INFO Received push request method=POST path=/45352026/11/12 11:46:59 INFO Completed upload id=27 objects=7 duration=254.125µs45362026/11/12 11:46:59 INFO Vacuumed table table=objects45372026/11/12 11:46:59 INFO Received complete upload request method=POST path=/45382026/11/12 11:46:59 INFO Received push request method=POST path=/45392026/11/12 11:46:59 INFO Completed upload id=8 objects=3 duration=1.596083ms45402026/11/12 11:46:59 INFO Received complete upload request method=POST path=/45412026/11/12 11:46:59 INFO Aborted multipart uploads count=0 kept=045422026/11/12 11:46:59 INFO Completed upload id=28 objects=3 duration=462.584µs45432026/11/12 11:46:59 INFO Received uploads request method=POST path=/45442026/11/12 11:46:59 WARN Force mode enabled - objects will be deleted immediately without grace period45452026/11/12 11:46:59 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045462026/11/12 11:46:59 INFO Vacuumed table table=pending_closures45472026/11/12 11:46:59 INFO Vacuumed table table=commit_receipts45482026/11/12 11:46:59 INFO Received complete push request method=POST path=/45492026/11/12 11:46:59 INFO Vacuumed table table=pending_objects45502026/11/12 11:46:59 INFO Received complete push request method=POST path=/45512026/11/12 11:46:59 INFO Completed push id=16 objects=16 duration=1.203917ms45522026/11/12 11:46:59 INFO Vacuumed table table=multipart_uploads45532026/11/12 11:46:59 INFO Completed push id=6 objects=6 duration=495.708µs45542026/11/12 11:46:59 INFO Vacuumed table table=closures45552026/11/12 11:46:59 INFO Vacuumed table table=objects45562026/11/12 11:46:59 INFO Received push request method=POST path=/45572026/11/12 11:46:59 INFO Aborted multipart uploads count=0 kept=045582026/11/12 11:46:59 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."45592026/11/12 11:46:59 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."45602026/11/12 11:46:59 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."45612026/11/12 11:46:59 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."45622026/11/12 11:46:59 INFO Received complete push request method=POST path=/45632026/11/12 11:46:59 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."45642026/11/12 11:46:59 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."45652026/11/12 11:46:59 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."45662026/11/12 11:46:59 INFO Completed push id=29 objects=23 duration=371.666µs45672026/11/12 11:46:59 INFO Received complete push request method=POST path=/45682026/11/12 11:46:59 INFO Received uploads request method=POST path=/45692026/11/12 11:46:59 INFO Received complete push request method=POST path=/45702026/11/12 11:46:59 INFO Completed push id=17 objects=11 duration=536.75µs45712026/11/12 11:46:59 INFO Completed push id=30 objects=6 duration=231.875µs45722026/11/12 11:46:59 INFO Received uploads request method=POST path=/45732026/11/12 11:46:59 INFO Received push request method=POST path=/45742026/11/12 11:46:59 INFO Received push request method=POST path=/45752026/11/12 11:46:59 INFO Received uploads request method=POST path=/45762026/11/12 11:46:59 INFO Received push request method=POST path=/45772026/11/12 11:46:59 INFO Aborted multipart uploads count=0 kept=045782026/11/12 11:46:59 WARN Force mode enabled - objects will be deleted immediately without grace period45792026/11/12 11:46:59 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."45802026/11/12 11:46:59 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."45812026/11/12 11:46:59 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."45822026/11/12 11:46:59 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."45832026/11/12 11:46:59 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."45842026/11/12 11:46:59 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."45852026/11/12 11:46:59 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."45862026/11/12 11:46:59 INFO Received complete upload request method=POST path=/45872026/11/12 11:46:59 INFO Completed upload id=26 objects=6 duration=239.792µs45882026/11/12 11:46:59 INFO Received complete upload request method=POST path=/45892026/11/12 11:46:59 INFO Completed upload id=21 objects=3 duration=213.333µs45902026/11/12 11:46:59 INFO Received uploads request method=POST path=/45912026/11/12 11:46:59 INFO Aborted multipart uploads count=0 kept=045922026/11/12 11:46:59 INFO Received complete upload request method=POST path=/45932026/11/12 11:46:59 WARN Force mode enabled - objects will be deleted immediately without grace period45942026/11/12 11:46:59 INFO Completed upload id=29 objects=5 duration=236.458µs45952026/11/12 11:46:59 INFO Received complete push request method=POST path=/45962026/11/12 11:46:59 INFO Received complete push request method=POST path=/45972026/11/12 11:46:59 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=7 objects-failed-to-delete=045982026/11/12 11:46:59 INFO Completed push id=33 objects=3 duration=410.75µs45992026/11/12 11:46:59 INFO Completed push id=10 objects=15 duration=6.329709ms46002026/11/12 11:46:59 INFO Received complete push request method=POST path=/46012026/11/12 11:46:59 INFO Received complete upload request method=POST path=/46022026/11/12 11:46:59 INFO Completed push id=34 objects=6 duration=878.666µs46032026/11/12 11:46:59 INFO Vacuumed table table=pending_closures46042026/11/12 11:46:59 INFO Aborted multipart uploads count=0 kept=046052026/11/12 11:46:59 INFO Vacuumed table table=commit_receipts46062026/11/12 11:46:59 INFO Completed upload id=17 objects=25 duration=6.331083ms46072026/11/12 11:46:59 WARN Force mode enabled - objects will be deleted immediately without grace period46082026/11/12 11:46:59 INFO Received complete push request method=POST path=/46092026/11/12 11:46:59 INFO Completed push id=35 objects=8 duration=297.125µs46102026/11/12 11:46:59 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046112026/11/12 11:46:59 INFO Received push request method=POST path=/46122026/11/12 11:46:59 INFO Vacuumed table table=pending_closures46132026/11/12 11:46:59 INFO Vacuumed table table=pending_objects46142026/11/12 11:46:59 INFO Vacuumed table table=commit_receipts46152026/11/12 11:46:59 INFO Received complete upload request method=POST path=/46162026/11/12 11:46:59 INFO Aborted multipart uploads count=0 kept=046172026/11/12 11:46:59 INFO Vacuumed table table=multipart_uploads46182026/11/12 11:46:59 INFO Completed upload id=36 objects=9 duration=276.708µs46192026/11/12 11:46:59 INFO Vacuumed table table=pending_objects46202026/11/12 11:46:59 INFO Received complete upload request method=POST path=/46212026/11/12 11:46:59 INFO Vacuumed table table=multipart_uploads46222026/11/12 11:46:59 INFO Completed upload id=21 objects=2 duration=577µs46232026/11/12 11:46:59 INFO Vacuumed table table=closures46242026/11/12 11:46:59 INFO Vacuumed table table=closures46252026/11/12 11:46:59 WARN Force mode enabled - objects will be deleted immediately without grace period46262026/11/12 11:46:59 INFO Aborted multipart uploads count=0 kept=046272026/11/12 11:46:59 WARN Force mode enabled - objects will be deleted immediately without grace period46282026/11/12 11:46:59 INFO Vacuumed table table=objects46292026/11/12 11:46:59 INFO Vacuumed table table=objects46302026/11/12 11:46:59 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046312026/11/12 11:46:59 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=046322026/11/12 11:46:59 INFO Aborted multipart uploads count=0 kept=046332026/11/12 11:46:59 WARN Force mode enabled - objects will be deleted immediately without grace period46342026/11/12 11:46:59 INFO Vacuumed table table=pending_closures46352026/11/12 11:46:59 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046362026/11/12 11:46:59 INFO Vacuumed table table=commit_receipts46372026/11/12 11:46:59 INFO Vacuumed table table=pending_closures46382026/11/12 11:46:59 INFO Vacuumed table table=pending_closures46392026/11/12 11:46:59 INFO Received push request method=POST path=/46402026/11/12 11:46:59 INFO Vacuumed table table=commit_receipts46412026/11/12 11:46:59 INFO Vacuumed table table=commit_receipts46422026/11/12 11:46:59 INFO Aborted multipart uploads count=0 kept=046432026/11/12 11:46:59 INFO Vacuumed table table=pending_objects46442026/11/12 11:46:59 WARN Force mode enabled - objects will be deleted immediately without grace period46452026/11/12 11:46:59 INFO Received uploads request method=POST path=/46462026/11/12 11:46:59 INFO Vacuumed table table=multipart_uploads46472026/11/12 11:46:59 INFO Received complete push request method=POST path=/46482026/11/12 11:46:59 INFO Vacuumed table table=pending_objects46492026/11/12 11:46:59 INFO Vacuumed table table=pending_objects46502026/11/12 11:46:59 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046512026/11/12 11:46:59 INFO Vacuumed table table=multipart_uploads46522026/11/12 11:46:59 INFO Completed push id=17 objects=6 duration=5.862791ms46532026/11/12 11:46:59 INFO Vacuumed table table=pending_closures46542026/11/12 11:46:59 INFO Vacuumed table table=multipart_uploads46552026/11/12 11:46:59 INFO Vacuumed table table=closures46562026/11/12 11:46:59 INFO Vacuumed table table=commit_receipts46572026/11/12 11:46:59 INFO Vacuumed table table=closures46582026/11/12 11:46:59 INFO Vacuumed table table=closures46592026/11/12 11:46:59 INFO Received complete upload request method=POST path=/46602026/11/12 11:46:59 INFO Vacuumed table table=objects46612026/11/12 11:46:59 INFO Completed upload id=31 objects=3 duration=5.683333ms46622026/11/12 11:46:59 INFO Vacuumed table table=pending_objects46632026/11/12 11:46:59 INFO Vacuumed table table=objects46642026/11/12 11:46:59 INFO Vacuumed table table=objects46652026/11/12 11:46:59 INFO Vacuumed table table=multipart_uploads46662026/11/12 11:46:59 INFO Received complete upload request method=POST path=/46672026/11/12 11:46:59 INFO Received push request method=POST path=/46682026/11/12 11:46:59 INFO Vacuumed table table=closures46692026/11/12 11:46:59 INFO Completed upload id=22 objects=2 duration=6.941959ms46702026/11/12 11:46:59 INFO Vacuumed table table=objects46712026/11/12 11:46:59 INFO Received push request method=POST path=/46722026/11/12 11:46:59 INFO Aborted multipart uploads count=0 kept=046732026/11/12 11:46:59 INFO Received uploads request method=POST path=/46742026/11/12 11:46:59 WARN Force mode enabled - objects will be deleted immediately without grace period46752026/11/12 11:47:00 INFO Aborted multipart uploads count=0 kept=046762026/11/12 11:47:00 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046772026/11/12 11:47:00 INFO Received complete push request method=POST path=/46782026/11/12 11:47:00 INFO Received complete upload request method=POST path=/46792026/11/12 11:47:00 INFO Vacuumed table table=pending_closures46802026/11/12 11:47:00 WARN Force mode enabled - objects will be deleted immediately without grace period46812026/11/12 11:47:00 INFO Completed push id=22 objects=3 duration=919.417µs46822026/11/12 11:47:00 INFO Completed upload id=12 objects=9 duration=250.291µs46832026/11/12 11:47:00 INFO Vacuumed table table=commit_receipts46842026/11/12 11:47:00 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."46852026/11/12 11:47:00 INFO Received uploads request method=POST path=/46862026/11/12 11:47:00 INFO Vacuumed table table=pending_objects46872026/11/12 11:47:00 INFO Vacuumed table table=multipart_uploads46882026/11/12 11:47:00 INFO Vacuumed table table=closures46892026/11/12 11:47:00 INFO Received push request method=POST path=/46902026/11/12 11:47:00 INFO Vacuumed table table=objects46912026/11/12 11:47:00 INFO Received complete push request method=POST path=/46922026/11/12 11:47:00 INFO Received complete upload request method=POST path=/46932026/11/12 11:47:00 INFO Completed push id=24 objects=3 duration=643.834µs46942026/11/12 11:47:00 INFO Completed upload id=18 objects=2 duration=1.153042ms46952026/11/12 11:47:00 INFO Received push request method=POST path=/46962026/11/12 11:47:00 INFO Received push request method=POST path=/46972026/11/12 11:47:00 INFO Aborted multipart uploads count=0 kept=046982026/11/12 11:47:00 WARN Force mode enabled - objects will be deleted immediately without grace period46992026/11/12 11:47:00 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047002026/11/12 11:47:00 INFO Vacuumed table table=pending_closures47012026/11/12 11:47:00 INFO Vacuumed table table=commit_receipts47022026/11/12 11:47:00 INFO Received complete upload request method=POST path=/47032026/11/12 11:47:00 INFO Completed upload id=14 objects=8 duration=734.125µs47042026/11/12 11:47:00 INFO Vacuumed table table=pending_objects47052026/11/12 11:47:00 INFO Vacuumed table table=multipart_uploads47062026/11/12 11:47:00 INFO Vacuumed table table=closures47072026/11/12 11:47:00 INFO Received uploads request method=POST path=/47082026/11/12 11:47:00 INFO Vacuumed table table=objects47092026/11/12 11:47:00 INFO Received complete push request method=POST path=/47102026/11/12 11:47:00 INFO Aborted multipart uploads count=0 kept=047112026/11/12 11:47:00 WARN Force mode enabled - objects will be deleted immediately without grace period47122026/11/12 11:47:00 INFO Completed push id=16 objects=7 duration=6.8155ms47132026/11/12 11:47:00 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=047142026/11/12 11:47:00 INFO Vacuumed table table=pending_closures47152026/11/12 11:47:00 INFO Aborted multipart uploads count=0 kept=047162026/11/12 11:47:00 INFO Vacuumed table table=commit_receipts47172026/11/12 11:47:00 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=047182026/11/12 11:47:00 INFO Received complete push request method=POST path=/47192026/11/12 11:47:00 INFO Vacuumed table table=pending_objects47202026/11/12 11:47:00 INFO Vacuumed table table=multipart_uploads47212026/11/12 11:47:00 INFO Completed push id=4 objects=7 duration=5.559834ms47222026/11/12 11:47:00 INFO Vacuumed table table=closures47232026/11/12 11:47:00 INFO Received uploads request method=POST path=/47242026/11/12 11:47:00 INFO Vacuumed table table=objects47252026/11/12 11:47:00 INFO Vacuumed table table=pending_closures47262026/11/12 11:47:00 INFO Vacuumed table table=commit_receipts47272026/11/12 11:47:00 INFO Received complete upload request method=POST path=/47282026/11/12 11:47:00 INFO Completed upload id=13 objects=7 duration=328.167µs47292026/11/12 11:47:00 INFO Received uploads request method=POST path=/47302026/11/12 11:47:00 INFO Vacuumed table table=pending_objects47312026/11/12 11:47:00 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=8 objects-deleted-after-grace-period=8 objects-failed-to-delete=047322026/11/12 11:47:00 INFO Vacuumed table table=multipart_uploads47332026/11/12 11:47:00 INFO Received complete upload request method=POST path=/47342026/11/12 11:47:00 INFO Received uploads request method=POST path=/47352026/11/12 11:47:00 INFO Vacuumed table table=closures47362026/11/12 11:47:00 INFO Completed upload id=20 objects=4 duration=6.199292ms47372026/11/12 11:47:00 INFO Vacuumed table table=objects47382026/11/12 11:47:00 INFO Vacuumed table table=pending_closures47392026/11/12 11:47:00 INFO Received complete upload request method=POST path=/47402026/11/12 11:47:00 INFO Vacuumed table table=commit_receipts47412026/11/12 11:47:00 INFO Received uploads request method=POST path=/47422026/11/12 11:47:00 INFO Completed upload id=23 objects=7 duration=525.458µs47432026/11/12 11:47:00 INFO Vacuumed table table=pending_objects47442026/11/12 11:47:00 INFO Received complete upload request method=POST path=/47452026/11/12 11:47:00 INFO Vacuumed table table=multipart_uploads47462026/11/12 11:47:00 INFO Vacuumed table table=closures47472026/11/12 11:47:00 INFO Completed upload id=26 objects=2 duration=5.548542ms47482026/11/12 11:47:00 INFO Vacuumed table table=objects47492026/11/12 11:47:00 INFO Aborted multipart uploads count=0 kept=047502026/11/12 11:47:00 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047512026/11/12 11:47:00 INFO Aborted multipart uploads count=0 kept=047522026/11/12 11:47:00 INFO Received complete push request method=POST path=/47532026/11/12 11:47:00 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047542026/11/12 11:47:00 INFO Completed push id=23 objects=6 duration=358.875µs47552026/11/12 11:47:00 INFO Received complete push request method=POST path=/47562026/11/12 11:47:00 INFO Vacuumed table table=pending_closures47572026/11/12 11:47:00 INFO Vacuumed table table=pending_closures47582026/11/12 11:47:00 INFO Vacuumed table table=commit_receipts47592026/11/12 11:47:00 INFO Received complete push request method=POST path=/47602026/11/12 11:47:00 INFO Completed push id=7 objects=24 duration=1.802667ms47612026/11/12 11:47:00 INFO Vacuumed table table=commit_receipts47622026/11/12 11:47:00 INFO Vacuumed table table=pending_objects47632026/11/12 11:47:00 INFO Completed push id=24 objects=6 duration=268.125µs47642026/11/12 11:47:00 INFO Received uploads request method=POST path=/47652026/11/12 11:47:00 INFO Vacuumed table table=multipart_uploads47662026/11/12 11:47:00 INFO Vacuumed table table=closures47672026/11/12 11:47:00 INFO Received complete upload request method=POST path=/47682026/11/12 11:47:00 INFO Vacuumed table table=objects47692026/11/12 11:47:00 INFO Completed upload id=16 objects=17 duration=6.193542ms47702026/11/12 11:47:00 INFO Vacuumed table table=pending_objects47712026/11/12 11:47:00 INFO Received uploads request method=POST path=/47722026/11/12 11:47:00 INFO Vacuumed table table=multipart_uploads47732026/11/12 11:47:00 INFO Received push request method=POST path=/47742026/11/12 11:47:00 INFO Aborted multipart uploads count=0 kept=047752026/11/12 11:47:00 INFO Vacuumed table table=closures47762026/11/12 11:47:00 WARN Force mode enabled - objects will be deleted immediately without grace period47772026/11/12 11:47:00 INFO Received uploads request method=POST path=/47782026/11/12 11:47:00 INFO Vacuumed table table=objects47792026/11/12 11:47:00 INFO Received complete push request method=POST path=/47802026/11/12 11:47:00 INFO Received push request method=POST path=/47812026/11/12 11:47:00 INFO Completed push id=8 objects=6 duration=1.1075ms47822026/11/12 11:47:00 INFO Received push request method=POST path=/47832026/11/12 11:47:00 INFO Received uploads request method=POST path=/47842026/11/12 11:47:00 INFO Received push request method=POST path=/47852026/11/12 11:47:00 INFO Received complete push request method=POST path=/47862026/11/12 11:47:00 INFO Completed push id=13 objects=2 duration=325.458µs47872026/11/12 11:47:00 INFO Received complete upload request method=POST path=/47882026/11/12 11:47:00 INFO Aborted multipart uploads count=0 kept=047892026/11/12 11:47:00 INFO Completed upload id=27 objects=13 duration=5.913208ms47902026/11/12 11:47:00 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."47912026/11/12 11:47:00 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."47922026/11/12 11:47:00 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."47932026/11/12 11:47:00 INFO Received complete push request method=POST path=/47942026/11/12 11:47:00 INFO Aborted multipart uploads count=0 kept=047952026/11/12 11:47:00 INFO Received push request method=POST path=/47962026/11/12 11:47:00 INFO Completed push id=25 objects=7 duration=349.959µs47972026/11/12 11:47:00 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047982026/11/12 11:47:00 INFO Vacuumed table table=pending_closures47992026/11/12 11:47:00 INFO Received push request method=POST path=/48002026/11/12 11:47:00 INFO Vacuumed table table=commit_receipts48012026/11/12 11:47:00 INFO Vacuumed table table=pending_objects48022026/11/12 11:47:00 INFO Vacuumed table table=multipart_uploads48032026/11/12 11:47:00 INFO Received complete push request method=POST path=/48042026/11/12 11:47:00 INFO Vacuumed table table=closures48052026/11/12 11:47:00 INFO Completed push id=28 objects=11 duration=737.459µs48062026/11/12 11:47:00 INFO Vacuumed table table=objects48072026/11/12 11:47:00 INFO Received uploads request method=POST path=/48082026/11/12 11:47:00 INFO Received push request method=POST path=/48092026/11/12 11:47:00 INFO Received push request method=POST path=/48102026/11/12 11:47:00 INFO Received uploads request method=POST path=/48112026/11/12 11:47:00 INFO Received uploads request method=POST path=/48122026/11/12 11:47:00 INFO Received complete upload request method=POST path=/48132026/11/12 11:47:00 INFO Completed upload id=29 objects=4 duration=581.416µs48142026/11/12 11:47:00 INFO Received uploads request method=POST path=/48152026/11/12 11:47:00 INFO Received uploads request method=POST path=/48162026/11/12 11:47:00 INFO Received complete push request method=POST path=/48172026/11/12 11:47:00 INFO Completed push id=11 objects=6 duration=672.208µs48182026/11/12 11:47:00 INFO Received complete upload request method=POST path=/48192026/11/12 11:47:00 INFO Completed upload id=24 objects=18 duration=276.834µs48202026/11/12 11:47:00 INFO Received complete push request method=POST path=/48212026/11/12 11:47:00 INFO Completed push id=25 objects=2 duration=431.208µs48222026/11/12 11:47:00 INFO Received complete push request method=POST path=/48232026/11/12 11:47:00 INFO Completed push id=27 objects=4 duration=233.583µs48242026/11/12 11:47:00 INFO Received complete push request method=POST path=/48252026/11/12 11:47:00 INFO Completed push id=20 objects=13 duration=437.5µs48262026/11/12 11:47:00 INFO Received complete push request method=POST path=/48272026/11/12 11:47:00 INFO Completed push id=22 objects=4 duration=324.5µs48282026/11/12 11:47:00 INFO Received complete upload request method=POST path=/48292026/11/12 11:47:00 INFO Completed upload id=26 objects=24 duration=317.334µs48302026/11/12 11:47:00 INFO Received complete push request method=POST path=/48312026/11/12 11:47:00 INFO Received complete push request method=POST path=/48322026/11/12 11:47:00 INFO Completed push id=7 objects=7 duration=390.375µs48332026/11/12 11:47:00 INFO Completed push id=29 objects=24 duration=5.74925ms48342026/11/12 11:47:00 INFO Received uploads request method=POST path=/48352026/11/12 11:47:00 INFO Aborted multipart uploads count=0 kept=048362026/11/12 11:47:00 WARN Force mode enabled - objects will be deleted immediately without grace period48372026/11/12 11:47:00 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=048382026/11/12 11:47:00 INFO Received complete upload request method=POST path=/48392026/11/12 11:47:00 INFO Vacuumed table table=pending_closures48402026/11/12 11:47:00 INFO Vacuumed table table=commit_receipts48412026/11/12 11:47:00 INFO Completed upload id=23 objects=7 duration=10.240167ms48422026/11/12 11:47:00 INFO Received complete push request method=POST path=/48432026/11/12 11:47:00 INFO Completed push id=24 objects=5 duration=312.25µs48442026/11/12 11:47:00 INFO Vacuumed table table=pending_objects48452026/11/12 11:47:00 INFO Vacuumed table table=multipart_uploads48462026/11/12 11:47:00 INFO Vacuumed table table=closures48472026/11/12 11:47:00 INFO Received complete push request method=POST path=/48482026/11/12 11:47:00 INFO Received complete upload request method=POST path=/48492026/11/12 11:47:00 INFO Received complete push request method=POST path=/48502026/11/12 11:47:00 INFO Completed push id=28 objects=26 duration=469.167µs48512026/11/12 11:47:00 INFO Vacuumed table table=objects48522026/11/12 11:47:00 INFO Completed upload id=25 objects=2 duration=278.417µs48532026/11/12 11:47:00 INFO Completed push id=9 objects=24 duration=599.208µs48542026/11/12 11:47:00 INFO Received complete push request method=POST path=/48552026/11/12 11:47:00 INFO Received complete upload request method=POST path=/48562026/11/12 11:47:00 INFO Completed push id=30 objects=3 duration=297.041µs48572026/11/12 11:47:00 INFO Completed upload id=26 objects=20 duration=5.963042ms48582026/11/12 11:47:01 INFO Received uploads request method=POST path=/48592026/11/12 11:47:01 INFO Received complete push request method=POST path=/48602026/11/12 11:47:01 INFO Completed push id=32 objects=19 duration=257.834µs48612026/11/12 11:47:01 INFO Received uploads request method=POST path=/48622026/11/12 11:47:01 INFO Aborted multipart uploads count=0 kept=048632026/11/12 11:47:01 WARN Force mode enabled - objects will be deleted immediately without grace period48642026/11/12 11:47:01 INFO Received complete upload request method=POST path=/48652026/11/12 11:47:01 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048662026/11/12 11:47:01 INFO Completed upload id=10 objects=10 duration=6.180292ms48672026/11/12 11:47:01 INFO Received uploads request method=POST path=/48682026/11/12 11:47:01 INFO Vacuumed table table=pending_closures48692026/11/12 11:47:01 INFO Received push request method=POST path=/48702026/11/12 11:47:01 INFO Vacuumed table table=commit_receipts48712026/11/12 11:47:01 INFO Vacuumed table table=pending_objects48722026/11/12 11:47:01 INFO Vacuumed table table=multipart_uploads48732026/11/12 11:47:01 INFO Received complete push request method=POST path=/48742026/11/12 11:47:01 INFO Received push request method=POST path=/48752026/11/12 11:47:01 INFO Received complete upload request method=POST path=/48762026/11/12 11:47:01 INFO Received complete upload request method=POST path=/48772026/11/12 11:47:01 INFO Completed upload id=18 objects=22 duration=7.192333ms48782026/11/12 11:47:01 INFO Vacuumed table table=closures48792026/11/12 11:47:01 INFO Completed push id=28 objects=7 duration=59.093333ms48802026/11/12 11:47:01 INFO Completed upload id=18 objects=11 duration=28.353833ms48812026/11/12 11:47:01 INFO Vacuumed table table=objects48822026/11/12 11:47:01 INFO Received complete upload request method=POST path=/48832026/11/12 11:47:01 INFO Received uploads request method=POST path=/48842026/11/12 11:47:01 INFO Completed upload id=21 objects=8 duration=264.209µs48852026/11/12 11:47:01 INFO Received complete upload request method=POST path=/48862026/11/12 11:47:01 INFO Received uploads request method=POST path=/48872026/11/12 11:47:01 INFO Completed upload id=27 objects=26 duration=1.027166ms48882026/11/12 11:47:01 INFO Received push request method=POST path=/48892026/11/12 11:47:01 INFO Received complete push request method=POST path=/48902026/11/12 11:47:01 INFO Received complete push request method=POST path=/48912026/11/12 11:47:01 INFO Completed push id=28 objects=6 duration=6.732792ms48922026/11/12 11:47:01 INFO Completed push id=14 objects=3 duration=312.333µs48932026/11/12 11:47:01 INFO Aborted multipart uploads count=0 kept=048942026/11/12 11:47:01 INFO Received complete push request method=POST path=/48952026/11/12 11:47:01 INFO Received push request method=POST path=/48962026/11/12 11:47:01 WARN Force mode enabled - objects will be deleted immediately without grace period48972026/11/12 11:47:01 INFO Completed push id=29 objects=6 duration=246.542µs48982026/11/12 11:47:01 INFO Received push request method=POST path=/48992026/11/12 11:47:01 INFO Received complete upload request method=POST path=/49002026/11/12 11:47:01 INFO Completed upload id=30 objects=6 duration=643.417µs49012026/11/12 11:47:01 INFO Aborted multipart uploads count=0 kept=049022026/11/12 11:47:01 WARN Force mode enabled - objects will be deleted immediately without grace period49032026/11/12 11:47:01 INFO Received complete push request method=POST path=/49042026/11/12 11:47:01 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049052026/11/12 11:47:01 INFO Completed push id=31 objects=5 duration=6.460458ms49062026/11/12 11:47:01 INFO Received complete upload request method=POST path=/49072026/11/12 11:47:01 INFO Completed upload id=22 objects=7 duration=234.917µs49082026/11/12 11:47:01 INFO Vacuumed table table=pending_closures49092026/11/12 11:47:01 INFO Vacuumed table table=commit_receipts49102026/11/12 11:47:01 INFO Received complete push request method=POST path=/49112026/11/12 11:47:01 INFO Completed push id=24 objects=5 duration=262.458µs49122026/11/12 11:47:01 INFO Vacuumed table table=pending_objects49132026/11/12 11:47:01 INFO Vacuumed table table=multipart_uploads49142026/11/12 11:47:01 INFO Vacuumed table table=closures49152026/11/12 11:47:01 INFO Vacuumed table table=objects49162026/11/12 11:47:01 INFO Received push request method=POST path=/49172026/11/12 11:47:01 INFO Received uploads request method=POST path=/49182026/11/12 11:47:01 INFO Received uploads request method=POST path=/49192026/11/12 11:47:01 INFO Aborted multipart uploads count=0 kept=049202026/11/12 11:47:01 WARN Force mode enabled - objects will be deleted immediately without grace period49212026/11/12 11:47:01 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049222026/11/12 11:47:01 INFO Vacuumed table table=pending_closures49232026/11/12 11:47:01 INFO Vacuumed table table=commit_receipts49242026/11/12 11:47:01 INFO Vacuumed table table=pending_objects49252026/11/12 11:47:01 INFO Vacuumed table table=multipart_uploads49262026/11/12 11:47:01 INFO Vacuumed table table=closures49272026/11/12 11:47:01 INFO Vacuumed table table=objects49282026/11/12 11:47:01 INFO Received uploads request method=POST path=/49292026/11/12 11:47:01 INFO Received complete upload request method=POST path=/49302026/11/12 11:47:01 INFO Completed upload id=25 objects=8 duration=315.625µs49312026/11/12 11:47:01 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=8 objects-deleted-after-grace-period=8 objects-failed-to-delete=049322026/11/12 11:47:01 INFO Vacuumed table table=pending_closures49332026/11/12 11:47:01 INFO Received uploads request method=POST path=/49342026/11/12 11:47:01 INFO Received complete push request method=POST path=/49352026/11/12 11:47:01 INFO Vacuumed table table=commit_receipts49362026/11/12 11:47:01 INFO Completed push id=26 objects=4 duration=6.652458ms49372026/11/12 11:47:01 INFO Vacuumed table table=pending_objects49382026/11/12 11:47:01 INFO Vacuumed table table=multipart_uploads49392026/11/12 11:47:01 INFO Received complete push request method=POST path=/49402026/11/12 11:47:01 INFO Received uploads request method=POST path=/49412026/11/12 11:47:01 INFO Vacuumed table table=closures49422026/11/12 11:47:01 INFO Completed push id=30 objects=2 duration=294.375µs49432026/11/12 11:47:01 INFO Vacuumed table table=objects49442026/11/12 11:47:01 INFO Received complete push request method=POST path=/49452026/11/12 11:47:01 INFO Completed push id=32 objects=26 duration=1.201833ms49462026/11/12 11:47:01 INFO Aborted multipart uploads count=0 kept=049472026/11/12 11:47:01 WARN Force mode enabled - objects will be deleted immediately without grace period49482026/11/12 11:47:01 INFO Received complete upload request method=POST path=/49492026/11/12 11:47:01 INFO Received uploads request method=POST path=/49502026/11/12 11:47:01 INFO Completed upload id=33 objects=2 duration=218.042µs49512026/11/12 11:47:01 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049522026/11/12 11:47:01 INFO Vacuumed table table=pending_closures49532026/11/12 11:47:01 INFO Received complete upload request method=POST path=/49542026/11/12 11:47:01 INFO Vacuumed table table=commit_receipts49552026/11/12 11:47:01 INFO Completed upload id=34 objects=5 duration=6.041208ms49562026/11/12 11:47:01 INFO Vacuumed table table=pending_objects49572026/11/12 11:47:01 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."49582026/11/12 11:47:01 INFO Vacuumed table table=multipart_uploads49592026/11/12 11:47:01 INFO Vacuumed table table=closures49602026/11/12 11:47:01 INFO Vacuumed table table=objects49612026/11/12 11:47:01 INFO Received complete push request method=POST path=/49622026/11/12 11:47:01 INFO Received complete upload request method=POST path=/49632026/11/12 11:47:01 INFO Completed upload id=35 objects=11 duration=252.75µs49642026/11/12 11:47:01 INFO Received complete upload request method=POST path=/49652026/11/12 11:47:01 INFO Completed upload id=30 objects=8 duration=234.75µs49662026/11/12 11:47:01 INFO Completed push id=18 objects=6 duration=907.959µs49672026/11/12 11:47:01 INFO Received uploads request method=POST path=/49682026/11/12 11:47:01 INFO Received complete upload request method=POST path=/49692026/11/12 11:47:01 INFO Completed upload id=15 objects=3 duration=981.167µs49702026/11/12 11:47:01 INFO Received complete upload request method=POST path=/49712026/11/12 11:47:01 INFO Completed upload id=36 objects=3 duration=191.958µs49722026/11/12 11:47:01 INFO Received complete upload request method=POST path=/49732026/11/12 11:47:01 INFO Received complete push request method=POST path=/49742026/11/12 11:47:01 INFO Completed upload id=23 objects=9 duration=239.541µs49752026/11/12 11:47:01 INFO Completed push id=32 objects=7 duration=277.667µs49762026/11/12 11:47:01 INFO Aborted multipart uploads count=0 kept=049772026/11/12 11:47:01 WARN Force mode enabled - objects will be deleted immediately without grace period49782026/11/12 11:47:01 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049792026/11/12 11:47:01 INFO Received push request method=POST path=/49802026/11/12 11:47:01 INFO Received push request method=POST path=/49812026/11/12 11:47:01 INFO Vacuumed table table=pending_closures49822026/11/12 11:47:01 INFO Vacuumed table table=commit_receipts49832026/11/12 11:47:01 INFO Received uploads request method=POST path=/49842026/11/12 11:47:01 INFO Vacuumed table table=pending_objects49852026/11/12 11:47:01 INFO Vacuumed table table=multipart_uploads49862026/11/12 11:47:01 INFO Vacuumed table table=closures49872026/11/12 11:47:01 INFO Vacuumed table table=objects49882026/11/12 11:47:01 INFO Received complete push request method=POST path=/49892026/11/12 11:47:01 INFO Received complete upload request method=POST path=/49902026/11/12 11:47:01 INFO Completed upload id=31 objects=6 duration=299.375µs49912026/11/12 11:47:01 INFO Completed push id=15 objects=4 duration=6.78425ms49922026/11/12 11:47:01 INFO Received push request method=POST path=/49932026/11/12 11:47:01 INFO Received push request method=POST path=/49942026/11/12 11:47:01 INFO Received complete upload request method=POST path=/49952026/11/12 11:47:01 INFO Completed upload id=32 objects=4 duration=563.542µs49962026/11/12 11:47:01 INFO Received complete upload request method=POST path=/49972026/11/12 11:47:01 INFO Completed upload id=33 objects=6 duration=296µs49982026/11/12 11:47:01 INFO Received uploads request method=POST path=/49992026/11/12 11:47:01 INFO Received complete push request method=POST path=/50002026/11/12 11:47:01 INFO Completed push id=16 objects=26 duration=386.709µs50012026/11/12 11:47:01 INFO Received complete upload request method=POST path=/50022026/11/12 11:47:01 INFO Completed upload id=34 objects=3 duration=191.666µs50032026/11/12 11:47:01 INFO Aborted multipart uploads count=0 kept=050042026/11/12 11:47:01 WARN Force mode enabled - objects will be deleted immediately without grace period50052026/11/12 11:47:01 INFO Aborted multipart uploads count=0 kept=050062026/11/12 11:47:01 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=050072026/11/12 11:47:01 WARN Force mode enabled - objects will be deleted immediately without grace period50082026/11/12 11:47:01 INFO Vacuumed table table=pending_closures50092026/11/12 11:47:01 INFO Vacuumed table table=commit_receipts50102026/11/12 11:47:01 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=050112026/11/12 11:47:01 INFO Vacuumed table table=pending_objects50122026/11/12 11:47:01 INFO Vacuumed table table=multipart_uploads50132026/11/12 11:47:01 INFO Vacuumed table table=closures50142026/11/12 11:47:01 INFO Vacuumed table table=objects50152026/11/12 11:47:01 INFO Vacuumed table table=pending_closures50162026/11/12 11:47:01 INFO Received complete upload request method=POST path=/50172026/11/12 11:47:01 INFO Vacuumed table table=commit_receipts50182026/11/12 11:47:01 INFO Completed upload id=33 objects=10 duration=322.125µs50192026/11/12 11:47:01 INFO Vacuumed table table=pending_objects50202026/11/12 11:47:01 INFO Vacuumed table table=multipart_uploads50212026/11/12 11:47:01 INFO Vacuumed table table=closures50222026/11/12 11:47:01 INFO Received complete push request method=POST path=/50232026/11/12 11:47:01 INFO Vacuumed table table=objects50242026/11/12 11:47:01 INFO Completed push id=27 objects=6 duration=6.154042ms50252026/11/12 11:47:01 INFO Received complete upload request method=POST path=/50262026/11/12 11:47:01 INFO Completed upload id=15 objects=2 duration=256.667µs50272026/11/12 11:47:01 INFO Received push request method=POST path=/50282026/11/12 11:47:01 INFO Received uploads request method=POST path=/50292026/11/12 11:47:01 INFO Received uploads request method=POST path=/50302026/11/12 11:47:01 INFO Received complete upload request method=POST path=/50312026/11/12 11:47:01 INFO Completed upload id=26 objects=8 duration=239.333µs50322026/11/12 11:47:01 INFO Received complete push request method=POST path=/50332026/11/12 11:47:01 INFO Completed push id=28 objects=14 duration=235.917µs50342026/11/12 11:47:01 INFO Received push request method=POST path=/50352026/11/12 11:47:01 INFO Received uploads request method=POST path=/50362026/11/12 11:47:01 INFO Received complete push request method=POST path=/50372026/11/12 11:47:01 INFO Aborted multipart uploads count=0 kept=050382026/11/12 11:47:01 INFO Completed push id=14 objects=3 duration=263.5µs50392026/11/12 11:47:01 INFO Received complete upload request method=POST path=/50402026/11/12 11:47:01 INFO Completed upload id=34 objects=23 duration=226.542µs50412026/11/12 11:47:01 WARN Force mode enabled - objects will be deleted immediately without grace period50422026/11/12 11:47:01 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=050432026/11/12 11:47:01 INFO Vacuumed table table=pending_closures50442026/11/12 11:47:01 INFO Vacuumed table table=commit_receipts50452026/11/12 11:47:01 INFO Vacuumed table table=pending_objects50462026/11/12 11:47:01 INFO Vacuumed table table=multipart_uploads50472026/11/12 11:47:01 INFO Vacuumed table table=closures50482026/11/12 11:47:01 INFO Vacuumed table table=objects50492026/11/12 11:47:01 INFO Received complete upload request method=POST path=/50502026/11/12 11:47:01 INFO Completed upload id=20 objects=4 duration=214.666µs50512026/11/12 11:47:01 INFO Received complete push request method=POST path=/50522026/11/12 11:47:01 INFO Received uploads request method=POST path=/50532026/11/12 11:47:01 INFO Completed push id=17 objects=22 duration=533.75µs50542026/11/12 11:47:01 INFO Received uploads request method=POST path=/50552026/11/12 11:47:01 INFO Received complete upload request method=POST path=/50562026/11/12 11:47:01 INFO Received complete push request method=POST path=/50572026/11/12 11:47:01 INFO Completed push id=35 objects=6 duration=256.791µs50582026/11/12 11:47:01 INFO Completed upload id=21 objects=9 duration=5.737333ms50592026/11/12 11:47:01 INFO Aborted multipart uploads count=0 kept=050602026/11/12 11:47:01 INFO Received complete push request method=POST path=/50612026/11/12 11:47:01 WARN Force mode enabled - objects will be deleted immediately without grace period50622026/11/12 11:47:01 INFO Completed push id=25 objects=10 duration=418.958µs50632026/11/12 11:47:01 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=050642026/11/12 11:47:01 INFO Vacuumed table table=pending_closures50652026/11/12 11:47:01 INFO Vacuumed table table=commit_receipts50662026/11/12 11:47:01 INFO Received complete upload request method=POST path=/50672026/11/12 11:47:01 INFO Completed upload id=29 objects=5 duration=239µs50682026/11/12 11:47:01 INFO Vacuumed table table=pending_objects50692026/11/12 11:47:01 INFO Vacuumed table table=multipart_uploads50702026/11/12 11:47:01 INFO Vacuumed table table=closures50712026/11/12 11:47:01 INFO Vacuumed table table=objects50722026/11/12 11:47:01 INFO Received complete upload request method=POST path=/50732026/11/12 11:47:01 INFO Received push request method=POST path=/50742026/11/12 11:47:01 INFO Completed upload id=30 objects=2 duration=244.625µs50752026/11/12 11:47:01 INFO Received complete push request method=POST path=/50762026/11/12 11:47:01 INFO Received complete push request method=POST path=/50772026/11/12 11:47:01 INFO Completed push id=19 objects=14 duration=373.792µs50782026/11/12 11:47:01 INFO Completed push id=31 objects=20 duration=292.875µs50792026/11/12 11:47:01 INFO Received uploads request method=POST path=/50802026/11/12 11:47:01 INFO Received complete upload request method=POST path=/50812026/11/12 11:47:01 INFO Completed upload id=32 objects=5 duration=455.917µs50822026/11/12 11:47:01 INFO Received complete push request method=POST path=/50832026/11/12 11:47:01 INFO Completed push id=16 objects=5 duration=244.333µs50842026/11/12 11:47:01 INFO Received uploads request method=POST path=/50852026/11/12 11:47:01 INFO Received complete upload request method=POST path=/50862026/11/12 11:47:01 INFO Completed upload id=33 objects=3 duration=154.917µs50872026/11/12 11:47:01 INFO Aborted multipart uploads count=0 kept=050882026/11/12 11:47:01 INFO Aborted multipart uploads count=0 kept=050892026/11/12 11:47:01 WARN Force mode enabled - objects will be deleted immediately without grace period50902026/11/12 11:47:01 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=050912026/11/12 11:47:01 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=050922026/11/12 11:47:01 INFO Vacuumed table table=pending_closures50932026/11/12 11:47:01 INFO Vacuumed table table=commit_receipts50942026/11/12 11:47:01 INFO Vacuumed table table=pending_closures50952026/11/12 11:47:01 INFO Vacuumed table table=commit_receipts50962026/11/12 11:47:01 INFO Vacuumed table table=pending_objects50972026/11/12 11:47:01 INFO Vacuumed table table=multipart_uploads50982026/11/12 11:47:01 INFO Vacuumed table table=closures50992026/11/12 11:47:01 INFO Vacuumed table table=pending_objects51002026/11/12 11:47:01 INFO Vacuumed table table=multipart_uploads51012026/11/12 11:47:01 INFO Vacuumed table table=objects51022026/11/12 11:47:01 INFO Vacuumed table table=closures51032026/11/12 11:47:01 INFO Vacuumed table table=objects51042026/11/12 11:47:01 INFO Aborted multipart uploads count=0 kept=051052026/11/12 11:47:01 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=051062026/11/12 11:47:01 INFO Vacuumed table table=pending_closures51072026/11/12 11:47:01 INFO Vacuumed table table=commit_receipts51082026/11/12 11:47:01 INFO Vacuumed table table=pending_objects51092026/11/12 11:47:01 INFO Vacuumed table table=multipart_uploads51102026/11/12 11:47:01 INFO Received complete upload request method=POST path=/51112026/11/12 11:47:01 INFO Vacuumed table table=closures51122026/11/12 11:47:01 INFO Vacuumed table table=objects51132026/11/12 11:47:01 INFO Completed upload id=20 objects=6 duration=287.709µs51142026/11/12 11:47:01 INFO Received uploads request method=POST path=/51152026/11/12 11:47:01 INFO Received uploads request method=POST path=/51162026/11/12 11:47:01 INFO Received uploads request method=POST path=/51172026/11/12 11:47:01 INFO Received push request method=POST path=/51182026/11/12 11:47:01 INFO Aborted multipart uploads count=0 kept=051192026/11/12 11:47:01 INFO Received complete upload request method=POST path=/51202026/11/12 11:47:01 INFO Completed upload id=21 objects=2 duration=193.958µs51212026/11/12 11:47:01 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=051222026/11/12 11:47:01 INFO Vacuumed table table=pending_closures51232026/11/12 11:47:01 INFO Vacuumed table table=commit_receipts51242026/11/12 11:47:01 INFO Vacuumed table table=pending_objects51252026/11/12 11:47:01 INFO Vacuumed table table=multipart_uploads51262026/11/12 11:47:01 INFO Vacuumed table table=closures51272026/11/12 11:47:01 INFO Vacuumed table table=objects51282026/11/12 11:47:01 INFO Received uploads request method=POST path=/51292026/11/12 11:47:01 INFO Aborted multipart uploads count=0 kept=051302026/11/12 11:47:01 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=051312026/11/12 11:47:01 INFO Vacuumed table table=pending_closures51322026/11/12 11:47:01 INFO Vacuumed table table=commit_receipts51332026/11/12 11:47:01 INFO Vacuumed table table=pending_objects51342026/11/12 11:47:01 INFO Vacuumed table table=multipart_uploads51352026/11/12 11:47:01 INFO Vacuumed table table=closures51362026/11/12 11:47:01 INFO Vacuumed table table=objects51372026/11/12 11:47:01 INFO Aborted multipart uploads count=0 kept=051382026/11/12 11:47:01 WARN Force mode enabled - objects will be deleted immediately without grace period51392026/11/12 11:47:01 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=051402026/11/12 11:47:01 INFO Vacuumed table table=pending_closures51412026/11/12 11:47:01 INFO Vacuumed table table=commit_receipts51422026/11/12 11:47:01 INFO Vacuumed table table=pending_objects51432026/11/12 11:47:01 INFO Vacuumed table table=multipart_uploads51442026/11/12 11:47:01 INFO Vacuumed table table=closures51452026/11/12 11:47:01 INFO Vacuumed table table=objects51462026/11/12 11:47:01 INFO Received push request method=POST path=/51472026/11/12 11:47:01 INFO Received push request method=POST path=/51482026/11/12 11:47:01 INFO Received uploads request method=POST path=/51492026/11/12 11:47:01 INFO Received uploads request method=POST path=/51502026/11/12 11:47:01 INFO Received push request method=POST path=/51512026/11/12 11:47:01 INFO Received push request method=POST path=/51522026/11/12 11:47:01 INFO Received complete push request method=POST path=/51532026/11/12 11:47:01 INFO Completed push id=24 objects=24 duration=838.875µs51542026/11/12 11:47:01 INFO Aborted multipart uploads count=0 kept=051552026/11/12 11:47:01 INFO Aborted multipart uploads count=0 kept=051562026/11/12 11:47:01 WARN Force mode enabled - objects will be deleted immediately without grace period51572026/11/12 11:47:01 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=051582026/11/12 11:47:01 INFO Vacuumed table table=pending_closures51592026/11/12 11:47:01 INFO Vacuumed table table=commit_receipts51602026/11/12 11:47:01 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=051612026/11/12 11:47:01 INFO Vacuumed table table=pending_closures51622026/11/12 11:47:01 INFO Vacuumed table table=pending_objects51632026/11/12 11:47:01 INFO Vacuumed table table=multipart_uploads51642026/11/12 11:47:01 INFO Vacuumed table table=commit_receipts51652026/11/12 11:47:01 INFO Vacuumed table table=closures51662026/11/12 11:47:01 INFO Vacuumed table table=objects51672026/11/12 11:47:01 INFO Vacuumed table table=pending_objects51682026/11/12 11:47:01 INFO Vacuumed table table=multipart_uploads51692026/11/12 11:47:01 INFO Vacuumed table table=closures51702026/11/12 11:47:01 INFO Vacuumed table table=objects51712026/11/12 11:47:01 INFO Aborted multipart uploads count=0 kept=051722026/11/12 11:47:01 WARN Force mode enabled - objects will be deleted immediately without grace period51732026/11/12 11:47:01 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=051742026/11/12 11:47:01 INFO Vacuumed table table=pending_closures51752026/11/12 11:47:01 INFO Vacuumed table table=commit_receipts51762026/11/12 11:47:01 INFO Vacuumed table table=pending_objects51772026/11/12 11:47:01 INFO Vacuumed table table=multipart_uploads51782026/11/12 11:47:01 INFO Vacuumed table table=closures51792026/11/12 11:47:01 INFO Received uploads request method=POST path=/51802026/11/12 11:47:01 INFO Vacuumed table table=objects51812026/11/12 11:47:01 INFO Received complete upload request method=POST path=/51822026/11/12 11:47:01 INFO Completed upload id=19 objects=2 duration=746.292µs51832026/11/12 11:47:01 INFO Received uploads request method=POST path=/51842026/11/12 11:47:01 INFO Received uploads request method=POST path=/51852026/11/12 11:47:01 INFO Received complete push request method=POST path=/51862026/11/12 11:47:01 INFO Completed push id=33 objects=7 duration=424.417µs51872026/11/12 11:47:01 INFO Received push request method=POST path=/51882026/11/12 11:47:01 INFO Received complete upload request method=POST path=/51892026/11/12 11:47:01 INFO Completed upload id=26 objects=3 duration=172.208µs51902026/11/12 11:47:01 INFO Received uploads request method=POST path=/51912026/11/12 11:47:01 INFO Aborted multipart uploads count=0 kept=051922026/11/12 11:47:01 WARN Force mode enabled - objects will be deleted immediately without grace period51932026/11/12 11:47:01 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=051942026/11/12 11:47:01 INFO Vacuumed table table=pending_closures51952026/11/12 11:47:01 INFO Vacuumed table table=commit_receipts51962026/11/12 11:47:01 INFO Vacuumed table table=pending_objects51972026/11/12 11:47:01 INFO Vacuumed table table=multipart_uploads51982026/11/12 11:47:01 INFO Vacuumed table table=closures51992026/11/12 11:47:01 INFO Aborted multipart uploads count=0 kept=052002026/11/12 11:47:01 INFO Vacuumed table table=objects52012026/11/12 11:47:01 WARN Force mode enabled - objects will be deleted immediately without grace period52022026/11/12 11:47:02 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=052032026/11/12 11:47:02 INFO Vacuumed table table=pending_closures52042026/11/12 11:47:02 INFO Vacuumed table table=commit_receipts52052026/11/12 11:47:02 INFO Vacuumed table table=pending_objects52062026/11/12 11:47:02 INFO Vacuumed table table=multipart_uploads52072026/11/12 11:47:02 INFO Vacuumed table table=closures52082026/11/12 11:47:02 INFO Vacuumed table table=objects52092026/11/12 11:47:02 INFO Received push request method=POST path=/52102026/11/12 11:47:02 INFO Received complete upload request method=POST path=/52112026/11/12 11:47:02 INFO Completed upload id=23 objects=7 duration=666.792µs52122026/11/12 11:47:02 INFO Received complete upload request method=POST path=/52132026/11/12 11:47:02 INFO Completed upload id=26 objects=5 duration=166.833µs52142026/11/12 11:47:02 INFO Received complete push request method=POST path=/52152026/11/12 11:47:02 INFO Completed push id=27 objects=3 duration=318.792µs52162026/11/12 11:47:02 INFO Received complete upload request method=POST path=/52172026/11/12 11:47:02 INFO Completed upload id=28 objects=5 duration=140.958µs52182026/11/12 11:47:02 INFO Received complete push request method=POST path=/52192026/11/12 11:47:02 INFO Completed push id=29 objects=17 duration=195.166µs52202026/11/12 11:47:02 INFO Aborted multipart uploads count=0 kept=052212026/11/12 11:47:02 WARN Force mode enabled - objects will be deleted immediately without grace period52222026/11/12 11:47:02 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=052232026/11/12 11:47:02 INFO Vacuumed table table=pending_closures52242026/11/12 11:47:02 INFO Vacuumed table table=commit_receipts52252026/11/12 11:47:02 INFO Vacuumed table table=pending_objects52262026/11/12 11:47:02 INFO Vacuumed table table=multipart_uploads52272026/11/12 11:47:02 INFO Vacuumed table table=closures52282026/11/12 11:47:02 INFO Vacuumed table table=objects52292026/11/12 11:47:02 INFO Received complete push request method=POST path=/52302026/11/12 11:47:02 INFO Completed push id=9 objects=9 duration=1.065541ms52312026/11/12 11:47:02 INFO Received complete upload request method=POST path=/52322026/11/12 11:47:02 INFO Completed upload id=21 objects=8 duration=231.375µs52332026/11/12 11:47:02 INFO Received complete push request method=POST path=/52342026/11/12 11:47:02 INFO Completed push id=22 objects=6 duration=230.75µs52352026/11/12 11:47:02 INFO Received complete push request method=POST path=/52362026/11/12 11:47:02 INFO Completed push id=24 objects=8 duration=209µs52372026/11/12 11:47:02 INFO Received complete upload request method=POST path=/52382026/11/12 11:47:02 INFO Completed upload id=25 objects=9 duration=150µs52392026/11/12 11:47:02 INFO Received complete upload request method=POST path=/52402026/11/12 11:47:02 INFO Completed upload id=27 objects=3 duration=137.334µs52412026/11/12 11:47:02 INFO Received complete upload request method=POST path=/52422026/11/12 11:47:02 INFO Completed upload id=28 objects=6 duration=117.458µs52432026/11/12 11:47:02 INFO Received complete push request method=POST path=/52442026/11/12 11:47:02 INFO Completed push id=30 objects=4 duration=165.125µs52452026/11/12 11:47:02 INFO Received complete push request method=POST path=/52462026/11/12 11:47:02 INFO Completed push id=31 objects=7 duration=172.875µs52472026/11/12 11:47:02 INFO Received complete upload request method=POST path=/52482026/11/12 11:47:02 INFO Completed upload id=32 objects=5 duration=118.125µs52492026/11/12 11:47:02 INFO Received complete upload request method=POST path=/52502026/11/12 11:47:02 INFO Completed upload id=34 objects=24 duration=228.792µs52512026/11/12 11:47:02 INFO Aborted multipart uploads count=0 kept=052522026/11/12 11:47:02 WARN Force mode enabled - objects will be deleted immediately without grace period52532026/11/12 11:47:02 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=052542026/11/12 11:47:02 INFO Vacuumed table table=pending_closures52552026/11/12 11:47:02 INFO Vacuumed table table=commit_receipts52562026/11/12 11:47:02 INFO Vacuumed table table=pending_objects52572026/11/12 11:47:02 INFO Vacuumed table table=multipart_uploads52582026/11/12 11:47:02 INFO Vacuumed table table=closures52592026/11/12 11:47:02 INFO Vacuumed table table=objects5260--- PASS: FuzzPushGC (0.00s)5261 --- PASS: FuzzPushGC/seed#5 (8.17s)5262 --- PASS: FuzzPushGC/seed#0 (9.46s)5263 --- PASS: FuzzPushGC/seed#8 (10.08s)5264 --- PASS: FuzzPushGC/seed#14 (10.74s)5265 --- PASS: FuzzPushGC/seed#15 (10.83s)5266 --- PASS: FuzzPushGC/seed#13 (11.24s)5267 --- PASS: FuzzPushGC/seed#3 (11.51s)5268 --- PASS: FuzzPushGC/seed#4 (12.65s)5269 --- PASS: FuzzPushGC/seed#2 (13.67s)5270 --- PASS: FuzzPushGC/seed#7 (13.83s)5271 --- PASS: FuzzPushGC/seed#11 (14.27s)5272 --- PASS: FuzzPushGC/seed#10 (14.37s)5273 --- PASS: FuzzPushGC/seed#12 (14.54s)5274 --- PASS: FuzzPushGC/seed#6 (14.59s)5275 --- PASS: FuzzPushGC/seed#1 (14.77s)5276 --- PASS: FuzzPushGC/seed#9 (14.95s)5277PASS5278{"timestamp":"2026-11-12T11:47:02.268556Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:51179","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(7)"}5279{"timestamp":"2026-11-12T11:47:02.268709Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:51621","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(8)"}5280{"timestamp":"2026-11-12T11:47:02.268767Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:51437","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(5)"}5281{"timestamp":"2026-11-12T11:47:02.269061Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:51589","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(8)"}52822026-11-12 11:47:02.416 UTC [49288] LOG: received smart shutdown request52832026-11-12 11:47:02.416 UTC [49288] LOG: background worker "logical replication launcher" (PID 49299) exited with exit code 152842026-11-12 11:47:03.368 UTC [49293] LOG: checkpoint complete: wrote 3283 buffers (20.0%), wrote 5 SLRU buffers; 0 WAL file(s) added, 0 removed, 33 recycled; write=27.514 s, sync=0.831 s, total=28.412 s; sync files=43134, longest=0.001 s, average=0.001 s; distance=535492 kB, estimate=535492 kB; lsn=0/280A9378, redo lsn=0/2227E07052852026-11-12 11:47:03.371 UTC [49293] LOG: shutting down52862026-11-12 11:47:03.371 UTC [49293] LOG: checkpoint starting: shutdown immediate52872026-11-12 11:47:03.862 UTC [49293] LOG: checkpoint complete: wrote 9021 buffers (55.1%), wrote 2 SLRU buffers; 0 WAL file(s) added, 0 removed, 6 recycled; write=0.441 s, sync=0.047 s, total=0.492 s; sync files=3013, longest=0.001 s, average=0.001 s; distance=96428 kB, estimate=491585 kB; lsn=0/280A93F0, redo lsn=0/280A93F052882026-11-12 11:47:03.870 UTC [49288] LOG: database system is shut down5289Running OIDC tests...5290=== RUN TestAudienceForIssuer5291=== PAUSE TestAudienceForIssuer5292=== RUN TestProviderClientGivesUpOnSilentIssuer5293=== PAUSE TestProviderClientGivesUpOnSilentIssuer5294=== RUN TestGlobMatch5295=== PAUSE TestGlobMatch5296=== RUN TestValidateToken_ValidToken5297=== PAUSE TestValidateToken_ValidToken5298=== RUN TestValidateToken_WrongAudience5299=== PAUSE TestValidateToken_WrongAudience5300=== RUN TestValidateToken_Expired5301=== PAUSE TestValidateToken_Expired5302=== RUN TestValidateToken_BoundClaimsMismatch5303=== PAUSE TestValidateToken_BoundClaimsMismatch5304=== RUN TestValidateToken_BoundSubjectMismatch5305=== PAUSE TestValidateToken_BoundSubjectMismatch5306=== RUN TestValidateToken_MultipleProviders5307=== PAUSE TestValidateToken_MultipleProviders5308=== RUN TestValidateToken_UnknownIssuerFetchesNoKeys5309=== PAUSE TestValidateToken_UnknownIssuerFetchesNoKeys5310=== RUN TestValidateToken_GoogleSchemelessIssuer5311=== PAUSE TestValidateToken_GoogleSchemelessIssuer5312=== RUN TestValidateToken_KubernetesServiceAccount5313=== PAUSE TestValidateToken_KubernetesServiceAccount5314=== RUN TestNewValidator_KubernetesRequiresCA5315=== PAUSE TestNewValidator_KubernetesRequiresCA5316=== RUN TestValidateToken_KubernetesIssuerFromOwnToken5317=== PAUSE TestValidateToken_KubernetesIssuerFromOwnToken5318=== RUN TestPins_ReservedForMatchingRule5319=== PAUSE TestPins_ReservedForMatchingRule5320=== RUN TestPins_TopLevelShorthand5321=== PAUSE TestPins_TopLevelShorthand5322=== RUN TestPins_ConfigValidation5323=== PAUSE TestPins_ConfigValidation5324=== RUN TestScopes_LegacyProviderDefaultsToWrite5325=== PAUSE TestScopes_LegacyProviderDefaultsToWrite5326=== RUN TestScopes_Rules5327=== PAUSE TestScopes_Rules5328=== RUN TestScopes_ConfigValidation5329=== PAUSE TestScopes_ConfigValidation5330=== CONT TestAudienceForIssuer5331--- PASS: TestAudienceForIssuer (0.00s)5332=== CONT TestScopes_ConfigValidation5333=== CONT TestValidateToken_Expired5334=== CONT TestValidateToken_BoundSubjectMismatch5335=== CONT TestValidateToken_BoundClaimsMismatch5336=== CONT TestValidateToken_ValidToken5337=== CONT TestValidateToken_GoogleSchemelessIssuer5338=== CONT TestScopes_Rules5339=== CONT TestScopes_LegacyProviderDefaultsToWrite5340=== CONT TestPins_ConfigValidation5341=== CONT TestPins_TopLevelShorthand5342--- PASS: TestPins_ConfigValidation (0.01s)5343=== CONT TestPins_ReservedForMatchingRule5344--- PASS: TestScopes_ConfigValidation (0.01s)5345=== CONT TestValidateToken_KubernetesIssuerFromOwnToken5346=== CONT TestNewValidator_KubernetesRequiresCA5347=== CONT TestValidateToken_KubernetesServiceAccount5348=== CONT TestGlobMatch5349=== RUN TestGlobMatch/foo_foo5350=== PAUSE TestGlobMatch/foo_foo5351=== RUN TestGlobMatch/foo_bar5352=== PAUSE TestGlobMatch/foo_bar5353=== RUN TestGlobMatch/*_5354=== PAUSE TestGlobMatch/*_5355=== RUN TestGlobMatch/*_anything5356=== PAUSE TestGlobMatch/*_anything5357=== RUN TestGlobMatch/foo*_foo5358=== PAUSE TestGlobMatch/foo*_foo5359=== RUN TestGlobMatch/foo*_foobar5360=== PAUSE TestGlobMatch/foo*_foobar5361=== RUN TestGlobMatch/foo*_bar5362=== PAUSE TestGlobMatch/foo*_bar5363=== RUN TestGlobMatch/*bar_bar5364=== PAUSE TestGlobMatch/*bar_bar5365=== RUN TestGlobMatch/*bar_foobar5366=== PAUSE TestGlobMatch/*bar_foobar5367=== RUN TestGlobMatch/*bar_foo5368=== PAUSE TestGlobMatch/*bar_foo5369=== RUN TestGlobMatch/foo*bar_foobar5370=== PAUSE TestGlobMatch/foo*bar_foobar5371=== RUN TestGlobMatch/foo*bar_foo123bar5372=== PAUSE TestGlobMatch/foo*bar_foo123bar5373=== RUN TestGlobMatch/foo*bar_foobarbaz5374=== PAUSE TestGlobMatch/foo*bar_foobarbaz5375=== RUN TestGlobMatch/*/*_foo/bar5376=== PAUSE TestGlobMatch/*/*_foo/bar5377=== RUN TestGlobMatch/*/*_foo5378=== PAUSE TestGlobMatch/*/*_foo5379=== RUN TestGlobMatch/refs/heads/*_refs/heads/main5380=== PAUSE TestGlobMatch/refs/heads/*_refs/heads/main5381=== RUN TestGlobMatch/refs/heads/*_refs/tags/v1.05382=== PAUSE TestGlobMatch/refs/heads/*_refs/tags/v1.05383=== RUN TestGlobMatch/refs/*/main_refs/heads/main5384=== PAUSE TestGlobMatch/refs/*/main_refs/heads/main5385=== RUN TestGlobMatch/fo?_foo5386=== PAUSE TestGlobMatch/fo?_foo5387=== RUN TestGlobMatch/fo?_fo5388=== PAUSE TestGlobMatch/fo?_fo5389=== RUN TestGlobMatch/fo?_fooo5390=== PAUSE TestGlobMatch/fo?_fooo5391=== RUN TestGlobMatch/?oo_foo5392=== PAUSE TestGlobMatch/?oo_foo5393=== RUN TestGlobMatch/?oo_boo5394=== PAUSE TestGlobMatch/?oo_boo5395=== RUN TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5396=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5397=== RUN TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5398=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5399=== CONT TestGlobMatch/foo_foo5400=== CONT TestValidateToken_WrongAudience5401=== CONT TestGlobMatch/*/*_foo/bar5402=== CONT TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5403=== CONT TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5404=== CONT TestGlobMatch/?oo_boo5405=== CONT TestGlobMatch/?oo_foo5406=== CONT TestGlobMatch/fo?_fooo5407=== CONT TestGlobMatch/fo?_fo5408=== CONT TestGlobMatch/fo?_foo5409=== CONT TestGlobMatch/refs/*/main_refs/heads/main5410=== CONT TestGlobMatch/refs/heads/*_refs/tags/v1.05411=== CONT TestGlobMatch/refs/heads/*_refs/heads/main5412=== CONT TestGlobMatch/*/*_foo5413=== CONT TestGlobMatch/*bar_bar5414=== CONT TestGlobMatch/foo*bar_foobarbaz5415=== CONT TestGlobMatch/foo*bar_foo123bar5416=== CONT TestGlobMatch/foo*bar_foobar5417=== CONT TestGlobMatch/*bar_foo5418=== CONT TestGlobMatch/*bar_foobar5419=== CONT TestGlobMatch/foo*_foo5420=== CONT TestGlobMatch/foo*_bar5421=== CONT TestGlobMatch/foo*_foobar5422=== CONT TestGlobMatch/*_5423=== CONT TestGlobMatch/*_anything5424=== CONT TestGlobMatch/foo_bar5425--- PASS: TestGlobMatch (0.00s)5426 --- PASS: TestGlobMatch/foo_foo (0.00s)5427 --- PASS: TestGlobMatch/*/*_foo/bar (0.00s)5428 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main (0.00s)5429 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main (0.00s)5430 --- PASS: TestGlobMatch/?oo_boo (0.00s)5431 --- PASS: TestGlobMatch/?oo_foo (0.00s)5432 --- PASS: TestGlobMatch/fo?_fooo (0.00s)5433 --- PASS: TestGlobMatch/fo?_fo (0.00s)5434 --- PASS: TestGlobMatch/fo?_foo (0.00s)5435 --- PASS: TestGlobMatch/refs/*/main_refs/heads/main (0.00s)5436 --- PASS: TestGlobMatch/refs/heads/*_refs/tags/v1.0 (0.00s)5437 --- PASS: TestGlobMatch/refs/heads/*_refs/heads/main (0.00s)5438 --- PASS: TestGlobMatch/*/*_foo (0.00s)5439 --- PASS: TestGlobMatch/*bar_bar (0.00s)5440 --- PASS: TestGlobMatch/foo*bar_foobarbaz (0.00s)5441 --- PASS: TestGlobMatch/foo*bar_foo123bar (0.00s)5442 --- PASS: TestGlobMatch/foo*bar_foobar (0.00s)5443 --- PASS: TestGlobMatch/*bar_foo (0.00s)5444 --- PASS: TestGlobMatch/*bar_foobar (0.00s)5445 --- PASS: TestGlobMatch/foo*_foo (0.00s)5446 --- PASS: TestGlobMatch/foo*_bar (0.00s)5447 --- PASS: TestGlobMatch/foo*_foobar (0.00s)5448 --- PASS: TestGlobMatch/*_ (0.00s)5449 --- PASS: TestGlobMatch/*_anything (0.00s)5450 --- PASS: TestGlobMatch/foo_bar (0.00s)5451=== CONT TestValidateToken_MultipleProviders5452=== CONT TestValidateToken_UnknownIssuerFetchesNoKeys5453=== CONT TestProviderClientGivesUpOnSilentIssuer54542026/11/12 11:47:06 INFO OIDC provider initialized name=known issuer=https://known.example5455--- PASS: TestProviderClientGivesUpOnSilentIssuer (0.01s)54562026/11/12 11:47:06 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:52377/oidc5457--- PASS: TestValidateToken_Expired (0.04s)54582026/11/12 11:47:06 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:52380/oidc5459--- PASS: TestPins_TopLevelShorthand (0.05s)54602026/11/12 11:47:06 INFO OIDC provider initialized name=kubernetes issuer=https://oidc.eks.invalid/id/ABC12354612026/11/12 11:47:06 INFO OIDC provider initialized name=google issuer=https://accounts.google.com54622026/11/12 11:47:06 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:52382/oidc5463--- PASS: TestValidateToken_GoogleSchemelessIssuer (0.07s)5464--- PASS: TestValidateToken_UnknownIssuerFetchesNoKeys (0.04s)5465--- PASS: TestValidateToken_KubernetesIssuerFromOwnToken (0.06s)5466--- PASS: TestValidateToken_WrongAudience (0.06s)54672026/11/12 11:47:06 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:52389/oidc5468--- PASS: TestValidateToken_ValidToken (0.08s)54692026/11/12 11:47:06 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:52391/oidc54702026/11/12 11:47:06 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:52393/oidc54712026/11/12 11:47:06 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:52395/oidc54722026/11/12 11:47:06 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:52400/oidc5473--- PASS: TestValidateToken_BoundClaimsMismatch (0.09s)54742026/11/12 11:47:06 INFO OIDC provider initialized name=kubernetes issuer=https://127.0.0.1:523965475--- PASS: TestValidateToken_BoundSubjectMismatch (0.09s)5476--- PASS: TestPins_ReservedForMatchingRule (0.08s)5477--- PASS: TestValidateToken_KubernetesServiceAccount (0.07s)5478--- PASS: TestScopes_Rules (0.09s)54792026/11/12 11:47:06 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:52402/oidc5480--- PASS: TestScopes_LegacyProviderDefaultsToWrite (0.10s)54812026/11/12 11:47:06 INFO OIDC provider initialized name=provider1 issuer=http://127.0.0.1:52399/oidc54822026/11/12 11:47:06 INFO OIDC provider initialized name=provider2 issuer=http://127.0.0.1:52404/oidc5483--- PASS: TestValidateToken_MultipleProviders (0.08s)54842026/11/12 11:47:06 http: TLS handshake error from 127.0.0.1:52408: read tcp 127.0.0.1:52407->127.0.0.1:52408: use of closed network connection5485--- PASS: TestNewValidator_KubernetesRequiresCA (0.12s)5486PASS5487Running hook tests...5488=== RUN TestSendPathsEmpty5489=== PAUSE TestSendPathsEmpty5490=== RUN TestQueueEnqueueAndFetch5491=== PAUSE TestQueueEnqueueAndFetch5492=== RUN TestQueueDeduplication5493=== PAUSE TestQueueDeduplication5494=== RUN TestQueueRemove5495=== PAUSE TestQueueRemove5496=== RUN TestQueueFetchBatchLimit5497=== PAUSE TestQueueFetchBatchLimit5498=== RUN TestQueueRetryMovesToBack5499=== PAUSE TestQueueRetryMovesToBack5500=== RUN TestQueueFetchRemoveLifecycle5501=== PAUSE TestQueueFetchRemoveLifecycle5502=== RUN TestQueueConcurrentWriters5503=== PAUSE TestQueueConcurrentWriters5504=== RUN TestQueueRemoveLargeClosure5505=== PAUSE TestQueueRemoveLargeClosure5506=== RUN TestServerClientIntegration5507--- PASS: TestServerClientIntegration (0.00s)5508=== RUN TestServerQueueError55092026/11/12 11:47:06 ERROR Failed to queue paths error="permission denied" count=15510--- PASS: TestServerQueueError (0.00s)5511=== RUN TestGetListenerSocketActivation5512 server_test.go:147: systemd socket activation does not exist on darwin5513--- SKIP: TestGetListenerSocketActivation (0.00s)5514=== RUN TestServerStalledClientDoesNotBlockShutdown5515=== PAUSE TestServerStalledClientDoesNotBlockShutdown5516=== RUN TestServerBacksOffOnAcceptErrors5517=== PAUSE TestServerBacksOffOnAcceptErrors5518=== RUN TestDrainIsolatesPoisonPath5519=== PAUSE TestDrainIsolatesPoisonPath5520=== RUN TestRunNotBlockedByPoisonHead5521=== PAUSE TestRunNotBlockedByPoisonHead5522=== RUN TestDrainGivesUpWhenServerDown5523=== PAUSE TestDrainGivesUpWhenServerDown5524=== RUN TestFailedPathPrunedByLaterClosure5525=== PAUSE TestFailedPathPrunedByLaterClosure5526=== RUN TestWorkerUploadsAndRemoves5527=== PAUSE TestWorkerUploadsAndRemoves5528=== RUN TestWorkerSkipsGCdPaths5529=== PAUSE TestWorkerSkipsGCdPaths5530=== RUN TestWorkerPrunesClosureDeps5531=== PAUSE TestWorkerPrunesClosureDeps5532=== RUN TestDrainTimeout5533=== PAUSE TestDrainTimeout5534=== RUN TestWorkerRemoveFailureIsNotProgress5535=== PAUSE TestWorkerRemoveFailureIsNotProgress5536=== RUN TestWorkerKeepsPathItCannotStat5537=== PAUSE TestWorkerKeepsPathItCannotStat5538=== CONT TestSendPathsEmpty5539=== CONT TestDrainTimeout5540=== CONT TestDrainGivesUpWhenServerDown5541--- PASS: TestSendPathsEmpty (0.00s)5542=== CONT TestWorkerKeepsPathItCannotStat5543=== CONT TestWorkerRemoveFailureIsNotProgress5544=== CONT TestDrainIsolatesPoisonPath5545=== CONT TestServerBacksOffOnAcceptErrors55462000/01/01 00:00:00 ERROR Accept failed error="too many open files"55472000/01/01 00:00:00 ERROR Accept failed error="too many open files"55482000/01/01 00:00:00 ERROR Accept failed error="too many open files"55492000/01/01 00:00:00 ERROR Accept failed error="too many open files"55502000/01/01 00:00:00 ERROR Accept failed error="too many open files"55512000/01/01 00:00:00 ERROR Accept failed error="too many open files"55522000/01/01 00:00:00 ERROR Accept failed error="too many open files"55532000/01/01 00:00:00 ERROR Accept failed error="too many open files"5554=== CONT TestServerStalledClientDoesNotBlockShutdown5555--- PASS: TestServerBacksOffOnAcceptErrors (0.00s)5556=== CONT TestQueueFetchBatchLimit5557=== CONT TestQueueRemove5558=== CONT TestQueueDeduplication5559=== CONT TestQueueRetryMovesToBack5560=== CONT TestQueueEnqueueAndFetch5561=== CONT TestQueueRemoveLargeClosure5562=== CONT TestQueueConcurrentWriters5563=== CONT TestQueueFetchRemoveLifecycle5564=== CONT TestWorkerUploadsAndRemoves5565=== CONT TestWorkerPrunesClosureDeps5566=== CONT TestWorkerSkipsGCdPaths5567=== CONT TestFailedPathPrunedByLaterClosure5568=== CONT TestRunNotBlockedByPoisonHead55692026/11/12 11:47:06 INFO Upload queue status pending=255702026/11/12 11:47:06 WARN Store path no longer exists (garbage collected?), removing from queue path=/private/var/tb/1/scratch/d6c7014d22e1e3f1/build/TestWorkerSkipsGCdPaths948458161/002/nonexistent55712026/11/12 11:47:06 INFO Uploading batch count=155722026/11/12 11:47:06 ERROR Upload failed error="upload failed" count=155732026/11/12 11:47:06 INFO Upload queue status pending=255742026/11/12 11:47:06 INFO Uploading batch count=155752026/11/12 11:47:06 INFO Uploading batch count=155762026/11/12 11:47:06 INFO Uploading batch count=155772026/11/12 11:47:06 INFO Uploading batch count=455782026/11/12 11:47:06 ERROR Upload failed error="upload failed" count=455792026/11/12 11:47:06 INFO Upload queue status pending=355802026/11/12 11:47:06 INFO Upload queue status pending=255812026/11/12 11:47:06 INFO Uploading batch count=155822026/11/12 11:47:06 ERROR Upload failed error="upload failed" count=155832026/11/12 11:47:06 INFO Uploading batch count=25584--- PASS: TestQueueRemove (0.01s)55852026/11/12 11:47:06 INFO Uploading batch count=155862000/01/01 00:00:00 INFO Uploading batch count=255872000/01/01 00:00:00 ERROR Upload failed error="context deadline exceeded" count=255882026/11/12 11:47:06 ERROR Upload failed, will retry later error="upload failed" path=/private/var/tb/1/scratch/d6c7014d22e1e3f1/build/TestDrainIsolatesPoisonPath1084953330/002/bbb55892026/11/12 11:47:06 INFO Uploading batch count=255902026/11/12 11:47:06 ERROR Upload failed error="upload failed" count=255912026/11/12 11:47:06 ERROR Upload failed, will retry later error="upload failed" path=/private/var/tb/1/scratch/d6c7014d22e1e3f1/build/TestDrainGivesUpWhenServerDown2825478553/002/a55922026/11/12 11:47:06 WARN Cannot stat store path, will retry later path=/private/var/tb/1/scratch/d6c7014d22e1e3f1/build/TestWorkerKeepsPathItCannotStat3624339391/002/locked/aaa error="stat /private/var/tb/1/scratch/d6c7014d22e1e3f1/build/TestWorkerKeepsPathItCannotStat3624339391/002/locked/aaa: permission denied"5593--- PASS: TestQueueFetchRemoveLifecycle (0.01s)55942000/01/01 00:00:00 ERROR Drain finished with paths left in queue remaining=45595--- PASS: TestQueueRetryMovesToBack (0.01s)5596--- PASS: TestQueueFetchBatchLimit (0.02s)55972026/11/12 11:47:06 ERROR Upload failed, will retry later error="upload failed" path=/private/var/tb/1/scratch/d6c7014d22e1e3f1/build/TestDrainGivesUpWhenServerDown2825478553/002/b55982026/11/12 11:47:06 WARN Cannot stat store path, will retry later path=/private/var/tb/1/scratch/d6c7014d22e1e3f1/build/TestWorkerKeepsPathItCannotStat3624339391/002/locked/aaa error="stat /private/var/tb/1/scratch/d6c7014d22e1e3f1/build/TestWorkerKeepsPathItCannotStat3624339391/002/locked/aaa: permission denied"55992026/11/12 11:47:06 INFO Uploading batch count=156002026/11/12 11:47:06 WARN Cannot stat store path, will retry later path=/private/var/tb/1/scratch/d6c7014d22e1e3f1/build/TestWorkerKeepsPathItCannotStat3624339391/002/locked/aaa error="stat /private/var/tb/1/scratch/d6c7014d22e1e3f1/build/TestWorkerKeepsPathItCannotStat3624339391/002/locked/aaa: permission denied"56012026/11/12 11:47:06 ERROR Upload failed error="upload failed" count=156022026/11/12 11:47:06 WARN Store path no longer exists (garbage collected?), removing from queue path=/private/var/tb/1/scratch/d6c7014d22e1e3f1/build/TestWorkerRemoveFailureIsNotProgress1811402841/002/nonexistent56032026/11/12 11:47:06 INFO Uploading batch count=256042026/11/12 11:47:06 ERROR Upload failed error="upload failed" count=256052026/11/12 11:47:06 ERROR Upload failed, will retry later error="upload failed" path=/private/var/tb/1/scratch/d6c7014d22e1e3f1/build/TestDrainGivesUpWhenServerDown2825478553/002/c56062026/11/12 11:47:06 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=156072026/11/12 11:47:06 WARN Store path no longer exists (garbage collected?), removing from queue path=/private/var/tb/1/scratch/d6c7014d22e1e3f1/build/TestWorkerRemoveFailureIsNotProgress1811402841/002/nonexistent56082026/11/12 11:47:06 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=156092026/11/12 11:47:06 ERROR Upload failed, will retry later error="upload failed" path=/private/var/tb/1/scratch/d6c7014d22e1e3f1/build/TestDrainGivesUpWhenServerDown2825478553/002/d56102026/11/12 11:47:06 INFO Uploading batch count=156112026/11/12 11:47:06 ERROR Upload failed error="upload failed" count=156122026/11/12 11:47:06 WARN Store path no longer exists (garbage collected?), removing from queue path=/private/var/tb/1/scratch/d6c7014d22e1e3f1/build/TestWorkerRemoveFailureIsNotProgress1811402841/002/nonexistent56132026/11/12 11:47:06 ERROR Drain finished with paths left in queue remaining=156142026/11/12 11:47:06 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=156152026/11/12 11:47:06 ERROR Drain finished with paths left in queue remaining=156162026/11/12 11:47:06 INFO Uploading batch count=256172026/11/12 11:47:06 ERROR Upload failed error="upload failed" count=256182026/11/12 11:47:06 ERROR Upload failed, will retry later error="upload failed" path=/private/var/tb/1/scratch/d6c7014d22e1e3f1/build/TestDrainGivesUpWhenServerDown2825478553/002/e5619--- PASS: TestQueueDeduplication (0.02s)56202026/11/12 11:47:06 INFO Uploading batch count=156212026/11/12 11:47:06 ERROR Upload failed error="upload failed" count=15622--- PASS: TestFailedPathPrunedByLaterClosure (0.02s)56232026/11/12 11:47:06 ERROR Upload failed, will retry later error="upload failed" path=/private/var/tb/1/scratch/d6c7014d22e1e3f1/build/TestDrainGivesUpWhenServerDown2825478553/002/f5624--- PASS: TestQueueEnqueueAndFetch (0.02s)56252026/11/12 11:47:06 ERROR Drain finished with paths left in queue remaining=156262026/11/12 11:47:06 ERROR Drain finished with paths left in queue remaining=105627--- PASS: TestDrainTimeout (0.02s)5628--- PASS: TestWorkerRemoveFailureIsNotProgress (0.02s)5629--- PASS: TestWorkerKeepsPathItCannotStat (0.02s)5630--- PASS: TestDrainIsolatesPoisonPath (0.02s)5631--- PASS: TestDrainGivesUpWhenServerDown (0.02s)5632--- PASS: TestWorkerSkipsGCdPaths (0.03s)5633--- PASS: TestWorkerPrunesClosureDeps (0.03s)5634--- PASS: TestWorkerUploadsAndRemoves (0.03s)5635--- PASS: TestQueueRemoveLargeClosure (0.05s)5636--- PASS: TestQueueConcurrentWriters (0.06s)56372026/11/12 11:47:06 ERROR Failed to decode request error="read unix /private/var/tb/1/scratch/d6c7014d22e1e3f1/build/hook1063873260/test.sock->: i/o timeout"56382026/11/12 11:47:06 ERROR Failed to write response error="write unix /private/var/tb/1/scratch/d6c7014d22e1e3f1/build/hook1063873260/test.sock->: i/o timeout"5639--- PASS: TestServerStalledClientDoesNotBlockShutdown (0.20s)56402026/11/12 11:47:07 INFO Uploading batch count=156412026/11/12 11:47:07 INFO Uploading batch count=156422026/11/12 11:47:07 INFO Uploading batch count=156432026/11/12 11:47:07 ERROR Upload failed error="upload failed" count=156442026/11/12 11:47:07 INFO Uploading batch count=156452026/11/12 11:47:07 ERROR Upload failed error="upload failed" count=156462026/11/12 11:47:07 INFO Uploading batch count=156472026/11/12 11:47:07 ERROR Upload failed error="upload failed" count=156482026/11/12 11:47:07 INFO Uploading batch count=156492026/11/12 11:47:07 ERROR Upload failed error="upload failed" count=156502026/11/12 11:47:07 ERROR Drain finished with paths left in queue remaining=15651--- PASS: TestRunNotBlockedByPoisonHead (1.03s)5652PASS5653Running signing tests...5654=== RUN TestExternalSigner5655=== PAUSE TestExternalSigner5656=== RUN TestExternalSignerWrongPublicKey5657=== PAUSE TestExternalSignerWrongPublicKey5658=== RUN TestExternalSignerErrors5659=== PAUSE TestExternalSignerErrors5660=== RUN TestExternalSignerRestartsAfterCrash5661=== PAUSE TestExternalSignerRestartsAfterCrash5662=== RUN TestExternalSignerProgramErrorKeepsProcess5663=== PAUSE TestExternalSignerProgramErrorKeepsProcess5664=== RUN TestExternalSignerClose5665=== PAUSE TestExternalSignerClose5666=== RUN TestExternalSignerRejectsInvalidPublicKey5667=== PAUSE TestExternalSignerRejectsInvalidPublicKey5668=== RUN TestExternalSignerCancellation5669=== PAUSE TestExternalSignerCancellation5670=== RUN TestGenerateFingerprint5671=== PAUSE TestGenerateFingerprint5672=== RUN TestParseSigningKey5673=== PAUSE TestParseSigningKey5674=== RUN TestSign5675=== PAUSE TestSign5676=== RUN TestSignBatch5677=== PAUSE TestSignBatch5678=== RUN TestNilKeyPublicKey5679=== PAUSE TestNilKeyPublicKey5680=== RUN TestSignErrors5681=== PAUSE TestSignErrors5682=== CONT TestExternalSigner5683=== CONT TestExternalSignerCancellation5684=== CONT TestExternalSignerWrongPublicKey5685=== CONT TestSignBatch5686=== CONT TestSignErrors5687=== CONT TestNilKeyPublicKey5688--- PASS: TestNilKeyPublicKey (0.00s)5689=== CONT TestParseSigningKey5690=== RUN TestParseSigningKey/valid_32-byte_key5691=== CONT TestSign5692=== CONT TestExternalSignerProgramErrorKeepsProcess5693=== CONT TestExternalSignerRejectsInvalidPublicKey5694=== RUN TestExternalSignerRejectsInvalidPublicKey/#005695=== PAUSE TestExternalSignerRejectsInvalidPublicKey/#005696=== RUN TestExternalSignerRejectsInvalidPublicKey/no-colon5697=== PAUSE TestExternalSignerRejectsInvalidPublicKey/no-colon5698=== RUN TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5699=== PAUSE TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5700=== RUN TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5701=== PAUSE TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5702=== RUN TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5703=== PAUSE TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5704=== CONT TestExternalSignerRejectsInvalidPublicKey/#005705=== CONT TestExternalSignerClose5706=== CONT TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5707=== CONT TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5708=== CONT TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5709=== CONT TestExternalSignerRejectsInvalidPublicKey/no-colon5710--- PASS: TestExternalSignerRejectsInvalidPublicKey (0.00s)5711 --- PASS: TestExternalSignerRejectsInvalidPublicKey/#00 (0.00s)5712 --- PASS: TestExternalSignerRejectsInvalidPublicKey/test-key:invalid! (0.00s)5713 --- PASS: TestExternalSignerRejectsInvalidPublicKey/test-key:YQ== (0.00s)5714 --- PASS: TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= (0.00s)5715 --- PASS: TestExternalSignerRejectsInvalidPublicKey/no-colon (0.00s)5716=== CONT TestExternalSignerErrors5717=== RUN TestExternalSignerErrors/missing_signatures5718=== CONT TestExternalSignerRestartsAfterCrash5719=== CONT TestGenerateFingerprint5720=== RUN TestGenerateFingerprint/basic_with_references5721=== PAUSE TestGenerateFingerprint/basic_with_references5722=== RUN TestGenerateFingerprint/no_references5723=== PAUSE TestGenerateFingerprint/no_references5724=== RUN TestGenerateFingerprint/unsorted_references_get_sorted5725=== PAUSE TestGenerateFingerprint/unsorted_references_get_sorted5726=== RUN TestGenerateFingerprint/invalid_nar_hash_prefix5727=== PAUSE TestGenerateFingerprint/invalid_nar_hash_prefix5728=== RUN TestGenerateFingerprint/invalid_nar_hash_length5729=== PAUSE TestParseSigningKey/valid_32-byte_key5730=== RUN TestParseSigningKey/valid_32-byte_key_with_different_name5731=== PAUSE TestExternalSignerErrors/missing_signatures5732=== PAUSE TestParseSigningKey/valid_32-byte_key_with_different_name5733=== RUN TestExternalSignerErrors/missing_signature_name5734=== RUN TestParseSigningKey/no_colon5735=== PAUSE TestExternalSignerErrors/missing_signature_name5736=== PAUSE TestParseSigningKey/no_colon5737=== PAUSE TestGenerateFingerprint/invalid_nar_hash_length5738=== RUN TestExternalSignerErrors/invalid_JSON5739=== RUN TestGenerateFingerprint/invalid_store_path_prefix5740=== PAUSE TestExternalSignerErrors/invalid_JSON5741=== RUN TestExternalSignerErrors/wrong_id5742=== PAUSE TestGenerateFingerprint/invalid_store_path_prefix5743=== RUN TestParseSigningKey/empty_name5744=== PAUSE TestExternalSignerErrors/wrong_id5745=== RUN TestGenerateFingerprint/invalid_reference_prefix5746=== RUN TestExternalSignerErrors/program_error5747=== PAUSE TestParseSigningKey/empty_name5748=== PAUSE TestExternalSignerErrors/program_error5749=== RUN TestExternalSignerErrors/exit_at_startup5750=== PAUSE TestExternalSignerErrors/exit_at_startup5751=== RUN TestExternalSignerErrors/exit_on_request5752=== RUN TestParseSigningKey/invalid_base645753=== PAUSE TestGenerateFingerprint/invalid_reference_prefix5754=== PAUSE TestExternalSignerErrors/exit_on_request5755=== RUN TestExternalSignerErrors/wrong_shape5756=== PAUSE TestExternalSignerErrors/wrong_shape5757=== CONT TestGenerateFingerprint/invalid_store_path_prefix5758=== CONT TestGenerateFingerprint/basic_with_references5759=== CONT TestGenerateFingerprint/invalid_nar_hash_length5760=== CONT TestGenerateFingerprint/invalid_nar_hash_prefix5761=== CONT TestGenerateFingerprint/unsorted_references_get_sorted5762=== CONT TestGenerateFingerprint/no_references5763=== RUN TestExternalSignerErrors/null5764=== PAUSE TestExternalSignerErrors/null5765=== RUN TestExternalSignerErrors/null_signatures5766=== PAUSE TestExternalSignerErrors/null_signatures5767=== RUN TestExternalSignerErrors/wrong_signatures_shape5768=== PAUSE TestExternalSignerErrors/wrong_signatures_shape5769=== RUN TestExternalSignerErrors/wrong_signature_name5770=== PAUSE TestExternalSignerErrors/wrong_signature_name5771=== CONT TestGenerateFingerprint/invalid_reference_prefix5772--- PASS: TestGenerateFingerprint (0.00s)5773 --- PASS: TestGenerateFingerprint/invalid_store_path_prefix (0.00s)5774 --- PASS: TestGenerateFingerprint/basic_with_references (0.00s)5775 --- PASS: TestGenerateFingerprint/invalid_nar_hash_length (0.00s)5776 --- PASS: TestGenerateFingerprint/invalid_nar_hash_prefix (0.00s)5777 --- PASS: TestGenerateFingerprint/unsorted_references_get_sorted (0.00s)5778 --- PASS: TestGenerateFingerprint/no_references (0.00s)5779 --- PASS: TestGenerateFingerprint/invalid_reference_prefix (0.00s)5780=== PAUSE TestParseSigningKey/invalid_base645781=== RUN TestParseSigningKey/wrong_length5782=== PAUSE TestParseSigningKey/wrong_length5783=== CONT TestParseSigningKey/valid_32-byte_key5784=== CONT TestParseSigningKey/invalid_base645785=== RUN TestExternalSignerErrors/invalid_signature_base645786=== CONT TestParseSigningKey/no_colon5787=== PAUSE TestExternalSignerErrors/invalid_signature_base645788=== RUN TestExternalSignerErrors/too_few_signatures5789=== CONT TestParseSigningKey/valid_32-byte_key_with_different_name5790=== PAUSE TestExternalSignerErrors/too_few_signatures5791=== CONT TestParseSigningKey/wrong_length5792=== RUN TestExternalSignerErrors/unrequested_key5793=== CONT TestParseSigningKey/empty_name5794=== PAUSE TestExternalSignerErrors/unrequested_key5795=== RUN TestExternalSignerErrors/wrong_signature_length5796=== PAUSE TestExternalSignerErrors/wrong_signature_length5797=== CONT TestExternalSignerErrors/missing_signatures5798=== CONT TestExternalSignerErrors/null5799=== CONT TestExternalSignerErrors/invalid_signature_base645800=== CONT TestExternalSignerErrors/wrong_signature_name5801=== CONT TestExternalSignerErrors/wrong_shape5802--- PASS: TestSign (0.00s)5803=== CONT TestExternalSignerErrors/wrong_signatures_shape5804=== RUN TestSignErrors/nil_key5805=== CONT TestExternalSignerErrors/unrequested_key5806=== PAUSE TestSignErrors/nil_key5807=== RUN TestSignErrors/nil_narinfo5808=== PAUSE TestSignErrors/nil_narinfo5809=== RUN TestSignErrors/invalid_narinfo5810=== PAUSE TestSignErrors/invalid_narinfo5811=== RUN TestSignErrors/canceled5812=== PAUSE TestSignErrors/canceled5813=== CONT TestSignErrors/nil_key5814=== CONT TestExternalSignerErrors/wrong_signature_length5815=== CONT TestExternalSignerErrors/invalid_JSON5816=== CONT TestExternalSignerErrors/wrong_id5817--- PASS: TestSignBatch (0.00s)5818=== CONT TestExternalSignerErrors/null_signatures5819--- PASS: TestParseSigningKey (0.00s)5820 --- PASS: TestParseSigningKey/invalid_base64 (0.00s)5821 --- PASS: TestParseSigningKey/no_colon (0.00s)5822 --- PASS: TestParseSigningKey/wrong_length (0.00s)5823 --- PASS: TestParseSigningKey/empty_name (0.00s)5824 --- PASS: TestParseSigningKey/valid_32-byte_key (0.00s)5825 --- PASS: TestParseSigningKey/valid_32-byte_key_with_different_name (0.00s)5826=== CONT TestExternalSignerErrors/program_error5827=== CONT TestExternalSignerErrors/too_few_signatures5828=== CONT TestSignErrors/canceled5829=== CONT TestExternalSignerErrors/missing_signature_name5830=== CONT TestExternalSignerErrors/exit_on_request5831=== CONT TestSignErrors/invalid_narinfo5832=== CONT TestExternalSignerErrors/exit_at_startup5833=== CONT TestSignErrors/nil_narinfo5834--- PASS: TestSignErrors (0.00s)5835 --- PASS: TestSignErrors/nil_key (0.00s)5836 --- PASS: TestSignErrors/canceled (0.00s)5837 --- PASS: TestSignErrors/invalid_narinfo (0.00s)5838 --- PASS: TestSignErrors/nil_narinfo (0.00s)5839--- PASS: TestExternalSignerCancellation (0.24s)5840--- PASS: TestExternalSignerProgramErrorKeepsProcess (0.41s)5841--- PASS: TestExternalSignerWrongPublicKey (0.60s)5842--- PASS: TestExternalSignerClose (1.74s)5843--- PASS: TestExternalSignerRestartsAfterCrash (1.95s)5844/private/var/tb/1/scratch/d6c7014d22e1e3f1/build/TestExternalSignerRestartsAfterCrash4252718091/001/sign program: line 7: /private/var/tb/1/scratch/d6c7014d22e1e3f1/build/TestExternalSignerRestartsAfterCrash4252718091/001/sign program.starts: No such file or directory5845--- PASS: TestExternalSigner (2.71s)5846--- PASS: TestExternalSignerErrors (0.00s)5847 --- PASS: TestExternalSignerErrors/too_few_signatures (0.79s)5848 --- PASS: TestExternalSignerErrors/missing_signature_name (0.98s)5849 --- PASS: TestExternalSignerErrors/invalid_JSON (1.17s)5850 --- PASS: TestExternalSignerErrors/program_error (1.36s)5851 --- PASS: TestExternalSignerErrors/wrong_signature_name (1.55s)5852 --- PASS: TestExternalSignerErrors/wrong_signatures_shape (2.12s)5853 --- PASS: TestExternalSignerErrors/null_signatures (2.32s)5854 --- PASS: TestExternalSignerErrors/missing_signatures (2.50s)5855 --- PASS: TestExternalSignerErrors/invalid_signature_base64 (2.89s)5856 --- PASS: TestExternalSignerErrors/null (3.08s)5857 --- PASS: TestExternalSignerErrors/wrong_shape (3.27s)5858 --- PASS: TestExternalSignerErrors/wrong_signature_length (3.46s)5859 --- PASS: TestExternalSignerErrors/unrequested_key (3.66s)5860 --- PASS: TestExternalSignerErrors/exit_at_startup (3.85s)5861 --- PASS: TestExternalSignerErrors/exit_on_request (4.04s)5862 --- PASS: TestExternalSignerErrors/wrong_id (4.23s)5863PASS5864Running OpenBao signer tests...5865=== RUN TestSign5866=== PAUSE TestSign5867=== RUN TestSignErrors5868=== PAUSE TestSignErrors5869=== RUN TestServe5870=== PAUSE TestServe5871=== RUN TestServeRejectsGarbage5872=== PAUSE TestServeRejectsGarbage5873=== CONT TestSign5874=== CONT TestServe5875=== CONT TestSignErrors5876=== RUN TestSignErrors/http_error5877=== CONT TestServeRejectsGarbage5878=== PAUSE TestSignErrors/http_error5879=== RUN TestSignErrors/invalid_json5880--- PASS: TestServeRejectsGarbage (0.00s)5881=== PAUSE TestSignErrors/invalid_json5882=== RUN TestSignErrors/no_prefix5883=== PAUSE TestSignErrors/no_prefix5884=== RUN TestSignErrors/invalid_base645885=== PAUSE TestSignErrors/invalid_base645886=== CONT TestSignErrors/http_error5887=== CONT TestSignErrors/no_prefix5888=== CONT TestSignErrors/invalid_base645889=== CONT TestSignErrors/invalid_json5890--- PASS: TestServe (0.00s)5891--- PASS: TestSign (0.00s)5892--- PASS: TestSignErrors (0.00s)5893 --- PASS: TestSignErrors/no_prefix (0.00s)5894 --- PASS: TestSignErrors/http_error (0.00s)5895 --- PASS: TestSignErrors/invalid_base64 (0.00s)5896 --- PASS: TestSignErrors/invalid_json (0.00s)5897PASS