niks3-go-unit-tests
checks.aarch64-linux.go-unit-tests
· build #493
· raw
1tribuchet: building on eliza2Running client tests...3=== RUN TestDoServerRequestAttachesToken4=== PAUSE TestDoServerRequestAttachesToken5=== RUN TestUploadBuildLog_FileBodyReplayedOnRetry6=== PAUSE TestUploadBuildLog_FileBodyReplayedOnRetry7=== RUN TestRegisterUploadedObjectReusesConnections8=== PAUSE TestRegisterUploadedObjectReusesConnections9=== RUN TestRunGarbageCollection_FinishedOnAnotherReplica10=== PAUSE TestRunGarbageCollection_FinishedOnAnotherReplica11=== RUN TestRunGarbageCollection_NotFoundAfterLocalRun12=== PAUSE TestRunGarbageCollection_NotFoundAfterLocalRun13=== RUN TestCaseHackSuffix14=== PAUSE TestCaseHackSuffix15=== RUN TestFilterOversizedClosures16=== PAUSE TestFilterOversizedClosures17=== RUN TestUploadMultipart_PartsInParallel18=== PAUSE TestUploadMultipart_PartsInParallel19=== RUN TestUploadMultipart_ReturnsAfterDroppedBody20=== PAUSE TestUploadMultipart_ReturnsAfterDroppedBody21=== RUN TestUploadMultipart_FailedProducerIsNotCompleted22=== PAUSE TestUploadMultipart_FailedProducerIsNotCompleted23=== RUN TestPartSizeForNAR24=== PAUSE TestPartSizeForNAR25=== RUN TestDumpPathCaseHackMatchesNix26--- PASS: TestDumpPathCaseHackMatchesNix (1.57s)27=== RUN TestDumpPathCaseHackCollision28--- PASS: TestDumpPathCaseHackCollision (0.00s)29=== RUN TestDumpPathMatchesNix30=== PAUSE TestDumpPathMatchesNix31=== RUN TestDumpPathSingleFile32=== PAUSE TestDumpPathSingleFile33=== RUN TestDumpPathWriterError34=== PAUSE TestDumpPathWriterError35=== RUN TestEncodeNixBase3236=== PAUSE TestEncodeNixBase3237=== RUN TestEncodeNixBase32WithRealHash38=== PAUSE TestEncodeNixBase32WithRealHash39=== RUN TestConvertHashToNix3240=== PAUSE TestConvertHashToNix3241=== RUN TestGetStorePathHash42=== PAUSE TestGetStorePathHash43=== RUN TestPathInfoHashCompatibility44=== PAUSE TestPathInfoHashCompatibility45=== RUN TestParsePathInfoJSON46=== PAUSE TestParsePathInfoJSON47=== RUN TestParsePathInfoJSONMultiplePaths48=== PAUSE TestParsePathInfoJSONMultiplePaths49=== RUN TestPathInfoCACompatibility50=== PAUSE TestPathInfoCACompatibility51=== RUN TestRateLimiterFeedback52=== PAUSE TestRateLimiterFeedback53=== RUN TestRateLimiterFeedback_400DoesNotCountAsSuccess54=== PAUSE TestRateLimiterFeedback_400DoesNotCountAsSuccess55=== RUN TestRegisterUploadedObject_BoundedAgainstSilentServer56=== PAUSE TestRegisterUploadedObject_BoundedAgainstSilentServer57=== RUN TestRegisterUploadedObject_OutlivesPush58=== PAUSE TestRegisterUploadedObject_OutlivesPush59=== RUN TestResolveStorePath60=== PAUSE TestResolveStorePath61=== RUN TestDoWithRetry_BodyReplayedViaGetBody62=== PAUSE TestDoWithRetry_BodyReplayedViaGetBody63=== RUN TestRetryTimeoutOutlastsMaxRetries64=== PAUSE TestRetryTimeoutOutlastsMaxRetries65=== RUN TestDoWithRetry_FinalResponseBodyReadable66=== PAUSE TestDoWithRetry_FinalResponseBodyReadable67=== RUN TestShellSplit68=== PAUSE TestShellSplit69=== RUN TestShellSplitErrors70=== PAUSE TestShellSplitErrors71=== RUN TestStreamPushReportsEveryPath72=== PAUSE TestStreamPushReportsEveryPath73=== RUN TestStreamPushBatchesUnderLoad74=== PAUSE TestStreamPushBatchesUnderLoad75=== RUN TestStreamPushIsolatesFailures76=== PAUSE TestStreamPushIsolatesFailures77=== RUN TestStreamPushGivesUpOnDeadServer78=== PAUSE TestStreamPushGivesUpOnDeadServer79=== RUN TestStreamPushRequestLine80=== PAUSE TestStreamPushRequestLine81=== RUN TestStreamPushReportsSignatures82=== PAUSE TestStreamPushReportsSignatures83=== RUN TestStreamPushReportsSkippedPaths84=== PAUSE TestStreamPushReportsSkippedPaths85=== RUN TestClientSignaturesByStorePath86=== PAUSE TestClientSignaturesByStorePath87=== RUN TestStreamPushStopsWhenCanceled88=== PAUSE TestStreamPushStopsWhenCanceled89=== RUN TestSetClientTLS90=== PAUSE TestSetClientTLS91=== RUN TestSetClientTLSDoesNotMutateDefaultTransport92=== PAUSE TestSetClientTLSDoesNotMutateDefaultTransport93=== RUN TestSetClientTLSErrors94=== PAUSE TestSetClientTLSErrors95=== RUN TestStaticToken96=== PAUSE TestStaticToken97=== RUN TestFileTokenReadsAndCaches98=== PAUSE TestFileTokenReadsAndCaches99=== RUN TestFileTokenMissing100=== PAUSE TestFileTokenMissing101=== RUN TestFileTokenEmpty102=== PAUSE TestFileTokenEmpty103=== RUN TestScriptTokenNoExpiryRerunsEveryCall104=== PAUSE TestScriptTokenNoExpiryRerunsEveryCall105=== RUN TestScriptTokenCachesUntilRefresh106=== PAUSE TestScriptTokenCachesUntilRefresh107=== RUN TestScriptTokenEmptyToken108=== PAUSE TestScriptTokenEmptyToken109=== RUN TestScriptTokenBadJSON110=== PAUSE TestScriptTokenBadJSON111=== RUN TestScriptTokenScriptFails112=== PAUSE TestScriptTokenScriptFails113=== RUN TestScriptTokenEmptyCommand114=== PAUSE TestScriptTokenEmptyCommand115=== CONT TestDoServerRequestAttachesToken116=== CONT TestScriptTokenEmptyCommand117--- PASS: TestScriptTokenEmptyCommand (0.00s)118=== CONT TestScriptTokenScriptFails119=== CONT TestScriptTokenBadJSON120=== CONT TestScriptTokenEmptyToken121=== CONT TestScriptTokenCachesUntilRefresh122=== CONT TestScriptTokenNoExpiryRerunsEveryCall123=== CONT TestFileTokenEmpty124=== CONT TestFileTokenMissing125--- PASS: TestFileTokenEmpty (0.00s)126--- PASS: TestFileTokenMissing (0.00s)127=== CONT TestFileTokenReadsAndCaches128--- PASS: TestScriptTokenCachesUntilRefresh (0.01s)129=== CONT TestPathInfoCACompatibility130=== RUN TestPathInfoCACompatibility/null_ca_field131=== PAUSE TestPathInfoCACompatibility/null_ca_field132=== RUN TestPathInfoCACompatibility/old_string_format_-_text133=== PAUSE TestPathInfoCACompatibility/old_string_format_-_text134=== RUN TestPathInfoCACompatibility/old_string_format_-_fixed_recursive135=== PAUSE TestPathInfoCACompatibility/old_string_format_-_fixed_recursive136=== RUN TestPathInfoCACompatibility/new_structured_format_-_text137=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_text138=== RUN TestPathInfoCACompatibility/new_structured_format_-_nar_method139=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_nar_method140=== CONT TestPathInfoCACompatibility/null_ca_field141--- PASS: TestFileTokenReadsAndCaches (0.00s)142=== CONT TestParsePathInfoJSONMultiplePaths143=== RUN TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths144=== PAUSE TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths145=== RUN TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths146=== PAUSE TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths147=== CONT TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths148=== CONT TestStaticToken149--- PASS: TestStaticToken (0.00s)150=== CONT TestSetClientTLSErrors151=== CONT TestParsePathInfoJSON152=== RUN TestParsePathInfoJSON/Nix_format153=== PAUSE TestParsePathInfoJSON/Nix_format154=== RUN TestParsePathInfoJSON/Lix_format155=== PAUSE TestParsePathInfoJSON/Lix_format156=== CONT TestShellSplit157--- PASS: TestShellSplit (0.00s)158=== CONT TestRegisterUploadedObjectReusesConnections159=== RUN TestParsePathInfoJSON/empty_input160=== PAUSE TestParsePathInfoJSON/empty_input161=== RUN TestParsePathInfoJSON/whitespace_only162=== PAUSE TestParsePathInfoJSON/whitespace_only163=== RUN TestParsePathInfoJSON/invalid_JSON164=== PAUSE TestParsePathInfoJSON/invalid_JSON165--- PASS: TestScriptTokenScriptFails (0.93s)166=== CONT TestSetClientTLSDoesNotMutateDefaultTransport167=== CONT TestPathInfoHashCompatibility168=== CONT TestGetStorePathHash169=== CONT TestConvertHashToNix32170=== CONT TestEncodeNixBase32WithRealHash171=== CONT TestEncodeNixBase32172=== CONT TestDumpPathWriterError173=== CONT TestDumpPathSingleFile174=== CONT TestDumpPathMatchesNix175=== CONT TestPartSizeForNAR176=== CONT TestUploadMultipart_FailedProducerIsNotCompleted177=== CONT TestUploadMultipart_ReturnsAfterDroppedBody178=== CONT TestUploadMultipart_PartsInParallel179=== CONT TestFilterOversizedClosures180=== CONT TestCaseHackSuffix181=== CONT TestRunGarbageCollection_NotFoundAfterLocalRun182=== CONT TestRunGarbageCollection_FinishedOnAnotherReplica183--- PASS: TestScriptTokenBadJSON (0.93s)184=== CONT TestUploadBuildLog_FileBodyReplayedOnRetry185=== CONT TestSetClientTLS186=== RUN TestSetClientTLSErrors/missing_cert_file187=== RUN TestPartSizeForNAR/zero_stays_at_minimum188=== PAUSE TestPartSizeForNAR/zero_stays_at_minimum189=== RUN TestPartSizeForNAR/small_stays_at_minimum190=== PAUSE TestPartSizeForNAR/small_stays_at_minimum191=== RUN TestPartSizeForNAR/80_GiB_fits_at_minimum192--- PASS: TestSetClientTLSDoesNotMutateDefaultTransport (1.10s)193=== CONT TestStreamPushStopsWhenCanceled194=== PAUSE TestSetClientTLSErrors/missing_cert_file195=== PAUSE TestPartSizeForNAR/80_GiB_fits_at_minimum196=== RUN TestConvertHashToNix32/SRI_format_to_Nix32197=== RUN TestSetClientTLSErrors/missing_key_file198=== RUN TestFilterOversizedClosures/no_limit_keeps_everything199=== PAUSE TestFilterOversizedClosures/no_limit_keeps_everything200=== RUN TestPartSizeForNAR/115_GiB_needs_larger_parts201=== PAUSE TestPartSizeForNAR/115_GiB_needs_larger_parts202=== RUN TestPartSizeForNAR/1_TiB203=== PAUSE TestPartSizeForNAR/1_TiB204=== RUN TestPartSizeForNAR/5_TiB_S3_max_object205=== PAUSE TestConvertHashToNix32/SRI_format_to_Nix32206--- PASS: TestStreamPushStopsWhenCanceled (0.00s)207=== RUN TestEncodeNixBase32/test_string_hash208=== PAUSE TestEncodeNixBase32/test_string_hash209=== RUN TestEncodeNixBase32/empty_input210=== CONT TestStreamPushReportsSkippedPaths211=== RUN TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped212=== RUN TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)213=== PAUSE TestPartSizeForNAR/5_TiB_S3_max_object214=== RUN TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary215=== PAUSE TestSetClientTLSErrors/missing_key_file216=== CONT TestStreamPushReportsSignatures217=== CONT TestStreamPushRequestLine218=== CONT TestClientSignaturesByStorePath219=== RUN TestGetStorePathHash/valid_store_path220=== CONT TestStreamPushGivesUpOnDeadServer221--- PASS: TestEncodeNixBase32WithRealHash (0.00s)222--- PASS: TestScriptTokenNoExpiryRerunsEveryCall (4.28s)223=== CONT TestStreamPushIsolatesFailures224=== PAUSE TestEncodeNixBase32/empty_input225=== CONT TestStreamPushBatchesUnderLoad226=== RUN TestStreamPushBatchesUnderLoad/oneAtATime=false227=== PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=false228--- PASS: TestStreamPushReportsSkippedPaths (0.00s)229--- PASS: TestClientSignaturesByStorePath (0.00s)230--- PASS: TestScriptTokenEmptyToken (4.29s)231=== PAUSE TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped232=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)233=== RUN TestPathInfoHashCompatibility/old_string_format_with_colon234=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_colon235=== RUN TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI236=== RUN TestPartSizeForNAR/capped_at_5_GiB237=== PAUSE TestPartSizeForNAR/capped_at_5_GiB238=== PAUSE TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary239=== RUN TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part240=== PAUSE TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part241=== CONT TestStreamPushReportsEveryPath242=== RUN TestSetClientTLSErrors/missing_ca_file243=== PAUSE TestSetClientTLSErrors/missing_ca_file244=== RUN TestSetClientTLSErrors/invalid_ca_file245=== PAUSE TestSetClientTLSErrors/invalid_ca_file246=== CONT TestShellSplitErrors247--- PASS: TestShellSplitErrors (0.00s)248=== CONT TestDoWithRetry_FinalResponseBodyReadable249=== CONT TestParsePathInfoJSON/Nix_format2502026/11/12 00:01:07 INFO Garbage collection started2512026/11/12 00:01:07 ERROR Upload failed error=boom count=12522026/11/12 00:01:07 ERROR Upload failed error=boom count=1253--- PASS: TestStreamPushReportsSignatures (0.00s)2542026/11/12 00:01:07 ERROR Upload failed error="connection refused" count=20255=== CONT TestRetryTimeoutOutlastsMaxRetries2562026/11/12 00:01:07 INFO Garbage collection started2572026/11/12 00:01:07 ERROR Server seems unavailable, giving up on batch untried=17258=== RUN TestConvertHashToNix32/already_Nix32_format259=== PAUSE TestConvertHashToNix32/already_Nix32_format260=== RUN TestConvertHashToNix32/invalid_format261=== PAUSE TestConvertHashToNix32/invalid_format262=== CONT TestDoWithRetry_BodyReplayedViaGetBody2632026/11/12 00:01:07 ERROR Upload failed error="bad path" count=3264=== PAUSE TestGetStorePathHash/valid_store_path265=== RUN TestGetStorePathHash/basename_without_hyphen_should_error266=== PAUSE TestGetStorePathHash/basename_without_hyphen_should_error267=== RUN TestGetStorePathHash/hash_with_invalid_characters_should_error268=== PAUSE TestGetStorePathHash/hash_with_invalid_characters_should_error269=== RUN TestGetStorePathHash/hash_with_wrong_length_should_error270=== CONT TestRateLimiterFeedback271=== RUN TestRateLimiterFeedback/429_enables_limiter272=== PAUSE TestRateLimiterFeedback/429_enables_limiter2732026/11/12 00:01:07 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:40563274=== CONT TestPathInfoCACompatibility/new_structured_format_-_nar_method2752026/11/12 00:01:07 WARN Rate limiter enabled after throttle name=server-test rate=52762026/11/12 00:01:07 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:40437277=== CONT TestPathInfoCACompatibility/new_structured_format_-_text2782026/11/12 00:01:07 WARN Rate limiter enabled after throttle name=server-test rate=5279=== CONT TestPathInfoCACompatibility/old_string_format_-_fixed_recursive2802026/11/12 00:01:07 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:39777281--- PASS: TestDoServerRequestAttachesToken (4.31s)282=== RUN TestRateLimiterFeedback/503_enables_limiter2832026/11/12 00:01:07 WARN Rate limiter backed off name=server-test rate=5284=== PAUSE TestRateLimiterFeedback/503_enables_limiter285=== RUN TestRateLimiterFeedback/200_does_not_enable_limiter286=== PAUSE TestRateLimiterFeedback/200_does_not_enable_limiter287=== RUN TestRateLimiterFeedback/400_does_not_enable_limiter288=== PAUSE TestRateLimiterFeedback/400_does_not_enable_limiter289--- PASS: TestStreamPushGivesUpOnDeadServer (0.00s)290--- PASS: TestStreamPushIsolatesFailures (0.00s)291--- PASS: TestUploadMultipart_ReturnsAfterDroppedBody (0.04s)292--- PASS: TestStreamPushReportsEveryPath (0.00s)2932026/11/12 00:01:07 WARN Rate limiter backed off name=server-test rate=5294=== CONT TestRateLimiterFeedback_400DoesNotCountAsSuccess2952026/11/12 00:01:07 WARN Request returned retryable status, retrying attempt=2 backoff=0s status=503 url=http://127.0.0.1:39777296--- PASS: TestDoWithRetry_FinalResponseBodyReadable (0.00s)297=== CONT TestParsePathInfoJSON/invalid_JSON298=== CONT TestParsePathInfoJSON/empty_input299=== CONT TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths3002026/11/12 00:01:07 WARN Rate limiter enabled after throttle name=server-test rate=5301--- PASS: TestParsePathInfoJSONMultiplePaths (0.00s)302 --- PASS: TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths (0.00s)303 --- PASS: TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths (0.00s)304=== PAUSE TestGetStorePathHash/hash_with_wrong_length_should_error305=== CONT TestParsePathInfoJSON/Lix_format306=== CONT TestEncodeNixBase32/test_string_hash307=== PAUSE TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI308=== RUN TestPathInfoHashCompatibility/new_structured_format_with_sha512309=== RUN TestSetClientTLS/rejects_connection_without_client_cert3102026/11/12 00:01:07 INFO Garbage collection progress phase=running_on_another_replica failed_uploads_deleted=0 old_closures_deleted=0 objects_marked=0 objects_deleted=0 objects_failed=0311=== PAUSE TestSetClientTLS/rejects_connection_without_client_cert3122026/11/12 00:01:07 WARN Rate limiter enabled after throttle name=s3-test rate=5313=== RUN TestSetClientTLS/succeeds_with_client_cert_and_CA3142026/11/12 00:01:07 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:33577315=== PAUSE TestSetClientTLS/succeeds_with_client_cert_and_CA316=== RUN TestSetClientTLS/preserves_debug_logging_transport317=== PAUSE TestSetClientTLS/preserves_debug_logging_transport3182026/11/12 00:01:07 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:40563319--- PASS: TestDoWithRetry_BodyReplayedViaGetBody (0.00s)320=== CONT TestRegisterUploadedObject_BoundedAgainstSilentServer321=== CONT TestResolveStorePath322=== RUN TestStreamPushBatchesUnderLoad/oneAtATime=true323=== CONT TestPathInfoCACompatibility/old_string_format_-_text324=== CONT TestParsePathInfoJSON/whitespace_only325=== CONT TestPartSizeForNAR/5_TiB_S3_max_object326=== CONT TestPartSizeForNAR/1_TiB327=== CONT TestPartSizeForNAR/115_GiB_needs_larger_parts328=== CONT TestPartSizeForNAR/80_GiB_fits_at_minimum329=== CONT TestPartSizeForNAR/small_stays_at_minimum330=== CONT TestSetClientTLSErrors/missing_cert_file331=== CONT TestSetClientTLSErrors/missing_ca_file332=== CONT TestPartSizeForNAR/capped_at_5_GiB333=== CONT TestSetClientTLSErrors/invalid_ca_file334=== CONT TestSetClientTLSErrors/missing_key_file335=== CONT TestConvertHashToNix32/SRI_format_to_Nix32336=== CONT TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part337=== RUN TestFilterOversizedClosures/all_closures_skipped338=== PAUSE TestFilterOversizedClosures/all_closures_skipped339=== CONT TestConvertHashToNix32/invalid_format340=== CONT TestRateLimiterFeedback/429_enables_limiter341=== CONT TestRateLimiterFeedback/503_enables_limiter342=== PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=true343=== CONT TestRegisterUploadedObject_OutlivesPush344=== CONT TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary345=== CONT TestRateLimiterFeedback/400_does_not_enable_limiter3462026/11/12 00:01:07 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:40563347=== CONT TestGetStorePathHash/hash_with_wrong_length_should_error348=== CONT TestEncodeNixBase32/empty_input349=== PAUSE TestPathInfoHashCompatibility/new_structured_format_with_sha512350=== CONT TestConvertHashToNix32/already_Nix32_format351=== CONT TestGetStorePathHash/hash_with_invalid_characters_should_error352=== CONT TestSetClientTLS/preserves_debug_logging_transport3532026/11/12 00:01:07 WARN Garbage collection finished on another replica. Its statistics are not available here3542026/11/12 00:01:07 WARN Rate limiter enabled after throttle name=server-test rate=53552026/11/12 00:01:07 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:42449356=== CONT TestSetClientTLS/succeeds_with_client_cert_and_CA357=== CONT TestFilterOversizedClosures/no_limit_keeps_everything358=== CONT TestFilterOversizedClosures/all_closures_skipped3592026/11/12 00:01:07 WARN Rate limiter enabled after throttle name=server-test rate=53602026/11/12 00:01:07 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=429 url=http://127.0.0.1:46205361=== CONT TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped362=== CONT TestStreamPushBatchesUnderLoad/oneAtATime=false3632026/11/12 00:01:07 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb-vm-image nar_size=5000 max_nar_size=50364=== CONT TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)3652026/11/12 00:01:07 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb-vm-image nar_size=5000 max_nar_size=2000366--- PASS: TestParsePathInfoJSON (0.00s)367 --- PASS: TestParsePathInfoJSON/Nix_format (0.00s)368 --- PASS: TestParsePathInfoJSON/invalid_JSON (0.00s)369 --- PASS: TestParsePathInfoJSON/empty_input (0.00s)370 --- PASS: TestParsePathInfoJSON/Lix_format (0.00s)371 --- PASS: TestParsePathInfoJSON/whitespace_only (0.00s)372=== CONT TestGetStorePathHash/basename_without_hyphen_should_error373=== CONT TestPartSizeForNAR/zero_stays_at_minimum3742026/11/12 00:01:07 WARN Rate limiter backed off name=server-test rate=53752026/11/12 00:01:07 WARN Rate limiter backed off name=server-test rate=5376=== CONT TestPathInfoHashCompatibility/old_string_format_with_colon377=== CONT TestRateLimiterFeedback/200_does_not_enable_limiter378=== CONT TestSetClientTLS/rejects_connection_without_client_cert379--- PASS: TestPartSizeForNAR (2.27s)380 --- PASS: TestPartSizeForNAR/5_TiB_S3_max_object (0.00s)381 --- PASS: TestPartSizeForNAR/1_TiB (0.00s)382 --- PASS: TestPartSizeForNAR/115_GiB_needs_larger_parts (0.00s)383 --- PASS: TestPartSizeForNAR/80_GiB_fits_at_minimum (0.00s)384 --- PASS: TestPartSizeForNAR/small_stays_at_minimum (0.00s)385 --- PASS: TestPartSizeForNAR/capped_at_5_GiB (0.00s)386 --- PASS: TestPartSizeForNAR/zero_stays_at_minimum (0.00s)387--- PASS: TestUploadBuildLog_FileBodyReplayedOnRetry (0.00s)3882026/11/12 00:01:07 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:40563389--- PASS: TestRunGarbageCollection_NotFoundAfterLocalRun (0.00s)390=== CONT TestGetStorePathHash/valid_store_path391=== CONT TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI392=== CONT TestStreamPushBatchesUnderLoad/oneAtATime=true393--- PASS: TestRunGarbageCollection_FinishedOnAnotherReplica (0.05s)394--- PASS: TestRegisterUploadedObject_OutlivesPush (0.00s)395--- PASS: TestResolveStorePath (0.00s)396=== CONT TestPathInfoHashCompatibility/new_structured_format_with_sha512397--- PASS: TestConvertHashToNix32 (0.00s)398 --- PASS: TestConvertHashToNix32/SRI_format_to_Nix32 (0.00s)399 --- PASS: TestConvertHashToNix32/invalid_format (0.00s)400 --- PASS: TestConvertHashToNix32/already_Nix32_format (0.00s)401--- PASS: TestEncodeNixBase32 (0.00s)402 --- PASS: TestEncodeNixBase32/test_string_hash (0.00s)403 --- PASS: TestEncodeNixBase32/empty_input (0.00s)404--- PASS: TestGetStorePathHash (0.04s)405 --- PASS: TestGetStorePathHash/hash_with_wrong_length_should_error (0.00s)406 --- PASS: TestGetStorePathHash/hash_with_invalid_characters_should_error (0.00s)407 --- PASS: TestGetStorePathHash/basename_without_hyphen_should_error (0.00s)408 --- PASS: TestGetStorePathHash/valid_store_path (0.00s)409--- PASS: TestFilterOversizedClosures (0.00s)410 --- PASS: TestFilterOversizedClosures/no_limit_keeps_everything (0.00s)411 --- PASS: TestFilterOversizedClosures/all_closures_skipped (0.00s)412 --- PASS: TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped (0.00s)413--- PASS: TestUploadMultipart_FailedProducerIsNotCompleted (0.00s)414 --- PASS: TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part (0.00s)415 --- PASS: TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary (0.00s)416--- PASS: TestPathInfoCACompatibility (0.00s)417 --- PASS: TestPathInfoCACompatibility/null_ca_field (0.00s)418 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_nar_method (0.00s)419 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_text (0.00s)420 --- PASS: TestPathInfoCACompatibility/old_string_format_-_fixed_recursive (0.00s)421 --- PASS: TestPathInfoCACompatibility/old_string_format_-_text (0.00s)422--- PASS: TestPathInfoHashCompatibility (0.01s)423 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI) (0.00s)424 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_colon (0.00s)425 --- PASS: TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI (0.00s)426 --- PASS: TestPathInfoHashCompatibility/new_structured_format_with_sha512 (0.00s)427--- PASS: TestStreamPushBatchesUnderLoad (0.00s)428 --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=false (0.00s)429 --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=true (0.00s)430--- PASS: TestSetClientTLSErrors (4.28s)431 --- PASS: TestSetClientTLSErrors/missing_cert_file (0.00s)432 --- PASS: TestSetClientTLSErrors/missing_key_file (0.00s)433 --- PASS: TestSetClientTLSErrors/missing_ca_file (0.00s)434 --- PASS: TestSetClientTLSErrors/invalid_ca_file (0.00s)435--- PASS: TestRateLimiterFeedback (0.00s)436 --- PASS: TestRateLimiterFeedback/400_does_not_enable_limiter (0.00s)437 --- PASS: TestRateLimiterFeedback/503_enables_limiter (0.00s)438 --- PASS: TestRateLimiterFeedback/429_enables_limiter (0.00s)439 --- PASS: TestRateLimiterFeedback/200_does_not_enable_limiter (0.00s)4402026/11/12 00:01:07 WARN Request returned retryable status, retrying attempt=3 backoff=1ms status=502 url=http://127.0.0.1:405634412026/11/12 00:01:07 WARN Request returned retryable status, retrying attempt=4 backoff=1ms status=502 url=http://127.0.0.1:405634422026/11/12 00:01:07 WARN Request returned retryable status, retrying attempt=5 backoff=1ms status=502 url=http://127.0.0.1:405634432026/11/12 00:01:07 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:33323/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:33323/api/uploads/complete4442026/11/12 00:01:07 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"4452026/11/12 00:01:07 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:33323/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:33323/api/uploads/complete4462026/11/12 00:01:07 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"4472026/11/12 00:01:07 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:33323/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:33323/api/uploads/complete4482026/11/12 00:01:07 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"449--- PASS: TestRegisterUploadedObject_BoundedAgainstSilentServer (0.10s)450--- PASS: TestRateLimiterFeedback_400DoesNotCountAsSuccess (1.00s)4512026/11/12 00:01:09 WARN Request returned retryable status, retrying attempt=6 backoff=1ms status=502 url=http://127.0.0.1:405634522026/11/12 00:01:09 WARN Request returned retryable status, retrying attempt=7 backoff=1ms status=502 url=http://127.0.0.1:405634532026/11/12 00:01:09 WARN Request returned retryable status, retrying attempt=8 backoff=1ms status=502 url=http://127.0.0.1:405634542026/11/12 00:01:09 WARN Request returned retryable status, retrying attempt=9 backoff=1ms status=502 url=http://127.0.0.1:405634552026/11/12 00:01:09 WARN Request returned retryable status, retrying attempt=10 backoff=1ms status=502 url=http://127.0.0.1:405634562026/11/12 00:01:09 WARN Request returned retryable status, retrying attempt=11 backoff=1ms status=502 url=http://127.0.0.1:405634572026/11/12 00:01:09 WARN Request returned retryable status, retrying attempt=12 backoff=1ms status=502 url=http://127.0.0.1:40563458--- PASS: TestDumpPathSingleFile (2.00s)459--- PASS: TestCaseHackSuffix (2.00s)4602026/11/12 00:01:09 WARN Request returned retryable status, retrying attempt=13 backoff=1ms status=502 url=http://127.0.0.1:405634612026/11/12 00:01:09 http: TLS handshake error from 127.0.0.1:50134: remote error: tls: bad certificate462--- PASS: TestSetClientTLS (0.00s)463 --- PASS: TestSetClientTLS/succeeds_with_client_cert_and_CA (1.98s)464 --- PASS: TestSetClientTLS/preserves_debug_logging_transport (1.98s)465 --- PASS: TestSetClientTLS/rejects_connection_without_client_cert (2.00s)4662026/11/12 00:01:09 WARN Request returned retryable status, retrying attempt=14 backoff=1ms status=502 url=http://127.0.0.1:40563467--- PASS: TestRegisterUploadedObjectReusesConnections (6.28s)4682026/11/12 00:01:09 WARN Request returned retryable status, retrying attempt=15 backoff=1ms status=502 url=http://127.0.0.1:405634692026/11/12 00:01:09 WARN Request returned retryable status, retrying attempt=16 backoff=1ms status=502 url=http://127.0.0.1:405634702026/11/12 00:01:09 WARN Request returned retryable status, retrying attempt=17 backoff=1ms status=502 url=http://127.0.0.1:40563471--- PASS: TestRetryTimeoutOutlastsMaxRetries (2.01s)472--- PASS: TestDumpPathWriterError (2.05s)473--- PASS: TestStreamPushRequestLine (2.01s)474--- PASS: TestUploadMultipart_PartsInParallel (2.63s)475--- PASS: TestDumpPathMatchesNix (4.57s)476PASS477Running server tests...478The files belonging to this database system will be owned by user "nixbld".479This user must also own the server process.480481The database cluster will be initialized with locale "C".482The default database encoding has accordingly been set to "SQL_ASCII".483The default text search configuration will be set to "english".484485Data page checksums are enabled.486487creating directory /build/postgres385024851/data ... ok488creating subdirectories ... ok489selecting dynamic shared memory implementation ... posix490selecting default "max_connections" ... 100491selecting default "shared_buffers" ... 128MB492selecting default time zone ... UTC493creating configuration files ... ok494running bootstrap script ... ok495performing post-bootstrap initialization ... ok496syncing data to disk ... ok497498initdb: warning: enabling "trust" authentication for local connections499initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.500501Success. You can now start the database server using:502503 pg_ctl -D /build/postgres385024851/data -l logfile start504505/build/postgres385024851:5432 - no response506/build/postgres385024851:5432 - no response507/build/postgres385024851:5432 - no response508/build/postgres385024851:5432 - no response509/build/postgres385024851:5432 - no response510/build/postgres385024851:5432 - no response511/build/postgres385024851:5432 - no response512/build/postgres385024851:5432 - no response513/build/postgres385024851:5432 - no response514/build/postgres385024851:5432 - no response515/build/postgres385024851:5432 - no response516/build/postgres385024851:5432 - no response5172026-11-12 00:01:24.978 UTC [128] LOG: starting PostgreSQL 18.6 on aarch64-unknown-linux-gnu, compiled by clang version 21.1.8, 64-bit5182026-11-12 00:01:24.979 UTC [128] LOG: listening on Unix socket "/build/postgres385024851/.s.PGSQL.5432"5192026-11-12 00:01:24.985 UTC [146] LOG: database system was shut down at 2026-11-12 00:01:23 UTC5202026-11-12 00:01:24.992 UTC [128] LOG: database system is ready to accept connections521/build/postgres385024851:5432 - accepting connections522{"timestamp":"2026-11-12T00:01:38.297301168Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"4434d5a4-31b8-4b9c-9480-5316663edec5","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"GET","uri":"/health/ready","status_code":503,"suppressed_errors":0,"duration_ms":14,"result":"server_error","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":502,"threadName":"rustfs-worker","threadId":"ThreadId(196)"}523{"timestamp":"2026-11-12T00:01:38.413366436Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"1ebd7608-6dc5-4ae0-8d6d-b3d415a8ef5c","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"GET","uri":"/health/ready","status_code":503,"suppressed_errors":0,"duration_ms":0,"result":"server_error","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":502,"threadName":"rustfs-worker","threadId":"ThreadId(205)"}524=== RUN TestAbortSucceeded525=== PAUSE TestAbortSucceeded526=== RUN TestService_AuthMiddleware527=== PAUSE TestService_AuthMiddleware528=== RUN TestService_AuthMiddleware_MTLSProxyHeader529=== PAUSE TestService_AuthMiddleware_MTLSProxyHeader530=== RUN TestService_AuthMiddleware_MTLSBoundSubjects531=== PAUSE TestService_AuthMiddleware_MTLSBoundSubjects532=== RUN TestService_ReadAuthMiddleware533=== PAUSE TestService_ReadAuthMiddleware534=== RUN TestService_AuthMiddleware_OIDC535=== PAUSE TestService_AuthMiddleware_OIDC536=== RUN TestService_RequireScope_OIDC537=== PAUSE TestService_RequireScope_OIDC538=== RUN TestService_ReadScope_PublicByDefault539=== PAUSE TestService_ReadScope_PublicByDefault540=== RUN TestCacheConfigHandler541=== PAUSE TestCacheConfigHandler542=== RUN TestCacheStatsHandler543=== PAUSE TestCacheStatsHandler544=== RUN TestClientCADerivations545=== PAUSE TestClientCADerivations546=== RUN TestClientErrorHandling547=== PAUSE TestClientErrorHandling548=== RUN TestClientIntegration549=== PAUSE TestClientIntegration550=== RUN TestClientMultipleUploads551=== PAUSE TestClientMultipleUploads552=== RUN TestClientWithDependencies553=== PAUSE TestClientWithDependencies554=== RUN TestClientSharedPathCommittedMidPush555=== PAUSE TestClientSharedPathCommittedMidPush556=== RUN TestPinProtectsFromGC557=== PAUSE TestPinProtectsFromGC558=== RUN TestClientPushesUseOnePush559=== PAUSE TestClientPushesUseOnePush560=== RUN TestClientFallsBackToClosures561=== PAUSE TestClientFallsBackToClosures562=== RUN TestDeleteClosuresRacingPins563=== PAUSE TestDeleteClosuresRacingPins564=== RUN TestConcurrentCommitsSharingObjectsDoNotDeadlock565=== PAUSE TestConcurrentCommitsSharingObjectsDoNotDeadlock566=== RUN TestResolveDBConnectionString567=== PAUSE TestResolveDBConnectionString568=== RUN TestConnectWaitsForAPeerMigration569=== PAUSE TestConnectWaitsForAPeerMigration570=== RUN TestLeadElectsOneAndHandsOver571=== PAUSE TestLeadElectsOneAndHandsOver572=== RUN TestLeadIncumbentWinsAfterRestart5732026/11/12 00:01:46 INFO lead: acquired remote=192.0.2.1:1234 generation=15742026/11/12 00:01:47 INFO lead: released remote=192.0.2.1:1234 generation=15752026/11/12 00:01:47 INFO lead: acquired remote=192.0.2.1:1234 generation=25762026/11/12 00:01:48 INFO lead: released remote=192.0.2.1:1234 generation=2577--- PASS: TestLeadIncumbentWinsAfterRestart (10.19s)578=== RUN TestLeadUnannouncedStandbyYieldsToIncumbent579=== PAUSE TestLeadUnannouncedStandbyYieldsToIncumbent580=== RUN TestLeadIncumbentAnnouncesAtOnce581=== PAUSE TestLeadIncumbentAnnouncesAtOnce582=== RUN TestLeadGenerationGrowsWithEveryGrant583=== PAUSE TestLeadGenerationGrowsWithEveryGrant584=== RUN TestLeadEndsOnShutdown585=== PAUSE TestLeadEndsOnShutdown586=== RUN TestLeadEndsWhenItsConnectionHangs5872026/11/12 00:01:52 INFO lead: acquired remote=192.0.2.1:1234 generation=15882026-11-12 00:01:53.560 UTC [393] FATAL: terminating connection due to administrator command5892026/11/12 00:01:53 INFO lead: acquired remote=192.0.2.1:1234 generation=25902026/11/12 00:01:54 WARN lead: lock connection lost error="timeout: context deadline exceeded"5912026/11/12 00:01:54 INFO lead: released remote=192.0.2.1:1234 generation=15922026/11/12 00:01:55 INFO lead: released remote=192.0.2.1:1234 generation=2593--- PASS: TestLeadEndsWhenItsConnectionHangs (6.42s)594=== RUN TestGCAdvisoryLockBlocksConcurrentRun595=== PAUSE TestGCAdvisoryLockBlocksConcurrentRun596=== RUN TestGCBugBareHashReferences597=== PAUSE TestGCBugBareHashReferences598=== RUN TestGCMetrics599=== PAUSE TestGCMetrics600=== RUN TestPushDedupSurvivesConcurrentGC601=== PAUSE TestPushDedupSurvivesConcurrentGC602=== RUN TestDeduplicatedObjectsRecordedAsPending603=== PAUSE TestDeduplicatedObjectsRecordedAsPending604=== RUN TestGCSweepSkipsPendingObjects605=== PAUSE TestGCSweepSkipsPendingObjects606=== RUN TestTombstonedObjectOfferedWithoutWaiting607=== PAUSE TestTombstonedObjectOfferedWithoutWaiting608=== RUN TestGCSweepDeliversEachKeyOnce609=== PAUSE TestGCSweepDeliversEachKeyOnce610=== RUN TestCreatePendingClosureVerifyS3FailureReleasesConnection611=== PAUSE TestCreatePendingClosureVerifyS3FailureReleasesConnection612=== RUN TestForceGCDuringPushOffersSweptObject613=== PAUSE TestForceGCDuringPushOffersSweptObject614=== RUN TestFailedS3DeleteKeepsTombstone615=== PAUSE TestFailedS3DeleteKeepsTombstone616=== RUN TestSweepSkipsRowsLockedByPush617=== PAUSE TestSweepSkipsRowsLockedByPush618=== RUN TestRegisterCompletedObjectNeedsPendingRow619=== PAUSE TestRegisterCompletedObjectNeedsPendingRow620=== RUN TestGetExistingObjectsLocksRows621=== PAUSE TestGetExistingObjectsLocksRows622=== RUN TestSweepRechecksPendingAfterLocking623=== PAUSE TestSweepRechecksPendingAfterLocking624=== RUN TestPushWaitsForSweepHoldingRow625=== PAUSE TestPushWaitsForSweepHoldingRow626=== RUN TestCommitRacingPendingCleanupKeepsObjects627=== PAUSE TestCommitRacingPendingCleanupKeepsObjects628=== RUN TestGCEndsOnShutdown629=== PAUSE TestGCEndsOnShutdown630=== RUN TestGCTaskStore_StartNew631=== PAUSE TestGCTaskStore_StartNew632=== RUN TestGCTaskStore_DeduplicateSameParams633=== PAUSE TestGCTaskStore_DeduplicateSameParams634=== RUN TestGCTaskStore_ConflictDifferentParams635=== PAUSE TestGCTaskStore_ConflictDifferentParams636=== RUN TestGCTaskStore_GetEmpty637=== PAUSE TestGCTaskStore_GetEmpty638=== RUN TestGCTaskStore_GetReturnsLatest639=== PAUSE TestGCTaskStore_GetReturnsLatest640=== RUN TestGCTaskStore_CompletedAllowsNewTask641=== PAUSE TestGCTaskStore_CompletedAllowsNewTask642=== RUN TestGCTaskStore_PhaseUpdates643=== PAUSE TestGCTaskStore_PhaseUpdates644=== RUN TestGCTaskStore_Fail645=== PAUSE TestGCTaskStore_Fail646=== RUN TestGracefulShutdownDrainsInflight647=== PAUSE TestGracefulShutdownDrainsInflight648=== RUN TestService_healthCheckHandler649=== PAUSE TestService_healthCheckHandler650=== RUN TestService_readinessHandler651=== PAUSE TestService_readinessHandler652=== RUN TestGenerateLandingPage653=== PAUSE TestGenerateLandingPage654=== RUN TestEnsureAbortUploadsRule655=== PAUSE TestEnsureAbortUploadsRule656=== RUN TestCacheConfigHandlerMaxNarSize657=== PAUSE TestCacheConfigHandlerMaxNarSize658=== RUN TestCreatePendingClosureRejectsOversizedNAR659=== PAUSE TestCreatePendingClosureRejectsOversizedNAR660=== RUN TestNARDeduplicationMetadataUploadBug661=== PAUSE TestNARDeduplicationMetadataUploadBug662=== RUN TestMetricsInventory663=== PAUSE TestMetricsInventory664=== RUN TestMetricsCommit665=== PAUSE TestMetricsCommit666=== RUN TestService_NativeMTLS667=== PAUSE TestService_NativeMTLS668=== RUN TestServerTLSConfig669=== PAUSE TestServerTLSConfig670=== RUN TestMultipartCleanup671=== PAUSE TestMultipartCleanup672=== RUN TestMultipartUploadAbortedWhenCancelledBeforeRecorded673=== PAUSE TestMultipartUploadAbortedWhenCancelledBeforeRecorded674=== RUN TestPendingCleanupUsesOneCutoff675=== PAUSE TestPendingCleanupUsesOneCutoff676=== RUN TestObjectStatsTrigger677=== PAUSE TestObjectStatsTrigger678=== RUN TestObjectStatsUpdatedOncePerCommit679=== PAUSE TestObjectStatsUpdatedOncePerCommit680=== RUN TestOrphanedObjectsGC681=== PAUSE TestOrphanedObjectsGC682=== RUN TestOrphanedObjectsGCStressTest683=== PAUSE TestOrphanedObjectsGCStressTest684=== RUN TestCreatePin_ReservedPins685=== PAUSE TestCreatePin_ReservedPins686=== RUN TestPresentReportsOnlyClosureRoots687=== PAUSE TestPresentReportsOnlyClosureRoots688=== RUN TestPresentRefreshesOnlyStaleClosures689=== PAUSE TestPresentRefreshesOnlyStaleClosures690=== RUN TestPresentNotReportedWhileGCDeletesClosure691=== PAUSE TestPresentNotReportedWhileGCDeletesClosure692=== RUN TestParseSingleRange693=== PAUSE TestParseSingleRange694=== RUN TestProxyHeadersOnlyTrustedOnSocket6952026/11/12 00:01:56 INFO Starting HTTP server address=127.0.0.1:415936962026/11/12 00:01:56 INFO Starting HTTP server address=proxy.sock6972026/11/12 00:01:56 WARN mTLS auth: subject not in bound subjects subject="CN=someone"6982026/11/12 00:01:56 INFO Shutdown signal received, draining in-flight requests timeout=10s699--- PASS: TestProxyHeadersOnlyTrustedOnSocket (1.28s)700=== RUN TestIsValidCachePath701=== PAUSE TestIsValidCachePath702=== RUN TestReadProxyNarinfo703=== PAUSE TestReadProxyNarinfo704=== RUN TestReadProxyNarinfoAlreadyDecompressed705=== PAUSE TestReadProxyNarinfoAlreadyDecompressed706=== RUN TestReadProxyContentNegotiation707=== PAUSE TestReadProxyContentNegotiation708=== RUN TestReadProxyNarStreaming709=== PAUSE TestReadProxyNarStreaming710=== RUN TestReadProxy404711=== PAUSE TestReadProxy404712=== RUN TestReadProxyInvalidPath713=== PAUSE TestReadProxyInvalidPath714=== RUN TestReadProxyHead715=== PAUSE TestReadProxyHead716=== RUN TestReadProxyOutlastsServerWriteTimeout717=== PAUSE TestReadProxyOutlastsServerWriteTimeout718=== RUN TestReadProxyConditionalGet719=== PAUSE TestReadProxyConditionalGet720=== RUN TestReadProxyRootRedirectsToIndexHTML721=== PAUSE TestReadProxyRootRedirectsToIndexHTML722=== RUN TestReadProxyDisabled723=== PAUSE TestReadProxyDisabled724=== RUN TestReadRedirectNar725=== PAUSE TestReadRedirectNar726=== RUN TestReadRedirectKeepsNarinfoProxied727=== PAUSE TestReadRedirectKeepsNarinfoProxied728=== RUN TestReadProxyRangeRequest729=== PAUSE TestReadProxyRangeRequest730=== RUN TestReadRedirectUsesPublicS3URL731=== PAUSE TestReadRedirectUsesPublicS3URL732=== RUN TestPush_OverlappingRootsStoreOneRowPerKey733=== PAUSE TestPush_OverlappingRootsStoreOneRowPerKey734=== RUN TestPush_CompleteCommitsEveryRoot735=== PAUSE TestPush_CompleteCommitsEveryRoot736=== RUN TestPush_RepeatedRootCommitsOnce737=== PAUSE TestPush_RepeatedRootCommitsOnce738=== RUN TestPush_CompleteWaitsForCleanupOfThePush739=== PAUSE TestPush_CompleteWaitsForCleanupOfThePush740=== RUN TestPush_RootsAreLockedInByteOrder741=== PAUSE TestPush_RootsAreLockedInByteOrder742=== RUN TestPush_SkippedKeySurvivesGCBeforeCommit743=== PAUSE TestPush_SkippedKeySurvivesGCBeforeCommit744=== RUN TestPush_RejectsBadRequests745=== PAUSE TestPush_RejectsBadRequests746=== RUN TestPush_SignsNarinfosOfItsPendingObjects747=== PAUSE TestPush_SignsNarinfosOfItsPendingObjects748=== RUN TestSharedNarBothUploadsComplete749=== PAUSE TestSharedNarBothUploadsComplete750=== RUN TestCompleteMultipartUpload_ErrorButObjectExists751=== PAUSE TestCompleteMultipartUpload_ErrorButObjectExists752=== RUN TestCompletedNarNotReofferedAcrossClosures753=== PAUSE TestCompletedNarNotReofferedAcrossClosures754=== RUN TestPresignedUploadRegisteredBeforeCommit755=== PAUSE TestPresignedUploadRegisteredBeforeCommit756=== RUN TestService_Rustfstest757=== PAUSE TestService_Rustfstest758=== RUN TestParseSize759=== PAUSE TestParseSize760=== RUN TestSkippedUploadsHandler761=== PAUSE TestSkippedUploadsHandler762=== RUN TestSystemdListenerNotActivated763--- PASS: TestSystemdListenerNotActivated (0.00s)764=== RUN TestWatchdogBeatsWhenHealthy765--- PASS: TestWatchdogBeatsWhenHealthy (0.02s)766=== RUN TestWatchdogSkipsWhenUnhealthy7672026/11/12 00:01:56 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7682026/11/12 00:01:56 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7692026/11/12 00:01:56 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7702026/11/12 00:01:56 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7712026/11/12 00:01:56 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7722026/11/12 00:01:56 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7732026/11/12 00:01:56 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7742026/11/12 00:01:56 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7752026/11/12 00:01:56 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7762026/11/12 00:01:56 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"777--- PASS: TestWatchdogSkipsWhenUnhealthy (0.20s)778=== RUN TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle779=== PAUSE TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle780=== RUN TestProxyWriteTimeout781=== PAUSE TestProxyWriteTimeout782=== RUN TestPendingClosureWriteTimeout783=== PAUSE TestPendingClosureWriteTimeout784=== RUN TestIsValidUploadKey785=== PAUSE TestIsValidUploadKey786=== RUN TestUploadHandlersRejectInvalidKeys787=== PAUSE TestUploadHandlersRejectInvalidKeys788=== RUN TestUploadHandlersRejectOversizedBody789=== PAUSE TestUploadHandlersRejectOversizedBody790=== RUN TestService_cleanupPendingClosuresHandler791=== PAUSE TestService_cleanupPendingClosuresHandler792=== RUN TestService_createPendingClosureHandler793=== PAUSE TestService_createPendingClosureHandler794=== RUN TestService_verifyS3Integrity795=== PAUSE TestService_verifyS3Integrity796=== RUN TestService_verifyS3ReuploadsStaleNarinfo797=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo798=== RUN TestCompleteMultipartUnregistered799=== PAUSE TestCompleteMultipartUnregistered800=== RUN TestCreatePendingClosure_SmallNARUsesSimplePUT801=== PAUSE TestCreatePendingClosure_SmallNARUsesSimplePUT802=== CONT TestProxyWriteTimeout803=== CONT TestService_createPendingClosureHandler804=== CONT TestAbortSucceeded805=== CONT TestCreatePendingClosure_SmallNARUsesSimplePUT806=== CONT TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle807--- PASS: TestAbortSucceeded (0.00s)808=== CONT TestPresentReportsOnlyClosureRoots809=== CONT TestCompleteMultipartUnregistered810=== CONT TestSkippedUploadsHandler811=== CONT TestService_verifyS3ReuploadsStaleNarinfo812=== RUN TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed813=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed814=== CONT TestParseSize815--- PASS: TestParseSize (0.00s)816=== CONT TestService_verifyS3Integrity817=== CONT TestService_Rustfstest818=== CONT TestUploadHandlersRejectInvalidKeys819=== CONT TestPresignedUploadRegisteredBeforeCommit820=== CONT TestService_cleanupPendingClosuresHandler821=== CONT TestCompletedNarNotReofferedAcrossClosures822=== CONT TestUploadHandlersRejectOversizedBody823=== CONT TestCompleteMultipartUpload_ErrorButObjectExists824=== CONT TestIsValidUploadKey825=== CONT TestSharedNarBothUploadsComplete826=== CONT TestPush_SignsNarinfosOfItsPendingObjects827=== CONT TestGCTaskStore_CompletedAllowsNewTask828=== CONT TestGCTaskStore_GetReturnsLatest829=== CONT TestPresentRefreshesOnlyStaleClosures830=== CONT TestGCTaskStore_GetEmpty831=== RUN TestProxyWriteTimeout/narinfo832=== RUN TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed833=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed834=== RUN TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost835=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost836=== RUN TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed837=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed838=== RUN TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed839=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed840=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info841=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info842=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal843=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal844=== RUN TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key845=== PAUSE TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key846=== RUN TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key847=== PAUSE TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key848=== CONT TestOrphanedObjectsGCStressTest8492026/11/12 00:01:56 INFO Client skipped oversized paths paths=3 nar_bytes=5000000000850--- PASS: TestGCTaskStore_GetReturnsLatest (0.00s)851=== CONT TestGCTaskStore_DeduplicateSameParams852--- PASS: TestGCTaskStore_DeduplicateSameParams (0.00s)853=== CONT TestOrphanedObjectsGC854=== PAUSE TestProxyWriteTimeout/narinfo855=== RUN TestProxyWriteTimeout/1_GiB_nar856=== PAUSE TestProxyWriteTimeout/1_GiB_nar857=== RUN TestProxyWriteTimeout/10_GiB_nar858=== PAUSE TestProxyWriteTimeout/10_GiB_nar859=== RUN TestProxyWriteTimeout/unknown_size860=== PAUSE TestProxyWriteTimeout/unknown_size861=== CONT TestCreatePin_ReservedPins862=== RUN TestIsValidUploadKey/narinfo863=== PAUSE TestIsValidUploadKey/narinfo864=== RUN TestIsValidUploadKey/nar_zst865=== PAUSE TestIsValidUploadKey/nar_zst866=== RUN TestIsValidUploadKey/nar_xz867=== PAUSE TestIsValidUploadKey/nar_xz868=== RUN TestIsValidUploadKey/nar_plain869=== PAUSE TestIsValidUploadKey/nar_plain870=== RUN TestIsValidUploadKey/listing871=== PAUSE TestIsValidUploadKey/listing872=== RUN TestIsValidUploadKey/build_log873--- PASS: TestGCTaskStore_CompletedAllowsNewTask (0.00s)874=== PAUSE TestIsValidUploadKey/build_log875=== RUN TestIsValidUploadKey/build_log_home-manager_file876=== PAUSE TestIsValidUploadKey/build_log_home-manager_file877=== CONT TestGCTaskStore_StartNew878--- PASS: TestGCTaskStore_StartNew (0.00s)879=== CONT TestObjectStatsTrigger880=== CONT TestGCEndsOnShutdown881=== RUN TestGCEndsOnShutdown/before_the_run882=== PAUSE TestGCEndsOnShutdown/before_the_run883=== RUN TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete884=== PAUSE TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete885=== RUN TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark886=== PAUSE TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark887=== CONT TestObjectStatsUpdatedOncePerCommit888=== CONT TestPresentNotReportedWhileGCDeletesClosure889=== RUN TestIsValidUploadKey/build_log_plus_in_name890=== RUN TestPresentNotReportedWhileGCDeletesClosure/fresh891=== PAUSE TestIsValidUploadKey/build_log_plus_in_name892=== CONT TestGCTaskStore_ConflictDifferentParams893=== CONT TestPendingCleanupUsesOneCutoff894--- PASS: TestGCTaskStore_GetEmpty (0.00s)895--- PASS: TestGCTaskStore_ConflictDifferentParams (0.00s)896=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/fresh897=== RUN TestPresentNotReportedWhileGCDeletesClosure/stale898=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/stale899=== RUN TestIsValidUploadKey/build_log_question_mark900=== PAUSE TestIsValidUploadKey/build_log_question_mark901=== RUN TestIsValidUploadKey/build_log_equals902=== PAUSE TestIsValidUploadKey/build_log_equals903=== CONT TestPush_RejectsBadRequests904=== RUN TestIsValidUploadKey/realisation905=== PAUSE TestIsValidUploadKey/realisation906=== RUN TestIsValidUploadKey/realisation_plus_in_output907=== PAUSE TestIsValidUploadKey/realisation_plus_in_output908=== RUN TestIsValidUploadKey/nix-cache-info909=== PAUSE TestIsValidUploadKey/nix-cache-info910=== RUN TestIsValidUploadKey/index.html911=== PAUSE TestIsValidUploadKey/index.html912=== RUN TestIsValidUploadKey/narinfo_key,_nar_type913=== PAUSE TestIsValidUploadKey/narinfo_key,_nar_type914=== RUN TestIsValidUploadKey/nar_key,_narinfo_type915=== PAUSE TestIsValidUploadKey/nar_key,_narinfo_type916=== RUN TestIsValidUploadKey/listing_key,_narinfo_type917=== PAUSE TestIsValidUploadKey/listing_key,_narinfo_type918=== RUN TestIsValidUploadKey/traversal919=== PAUSE TestIsValidUploadKey/traversal920=== RUN TestIsValidUploadKey/traversal_nar921=== PAUSE TestIsValidUploadKey/traversal_nar922=== RUN TestIsValidUploadKey/absolute923=== PAUSE TestIsValidUploadKey/absolute924=== RUN TestIsValidUploadKey/empty_key925=== PAUSE TestIsValidUploadKey/empty_key926=== RUN TestIsValidUploadKey/unknown_type927=== PAUSE TestIsValidUploadKey/unknown_type928=== CONT TestCommitRacingPendingCleanupKeepsObjects929=== RUN TestUploadHandlersRejectOversizedBody/create_pending_closure930=== PAUSE TestUploadHandlersRejectOversizedBody/create_pending_closure931=== RUN TestUploadHandlersRejectOversizedBody/complete_multipart932=== PAUSE TestUploadHandlersRejectOversizedBody/complete_multipart933=== RUN TestUploadHandlersRejectOversizedBody/request_more_parts934=== PAUSE TestUploadHandlersRejectOversizedBody/request_more_parts935=== CONT TestMultipartUploadAbortedWhenCancelledBeforeRecorded936--- PASS: TestSkippedUploadsHandler (3.17s)937=== CONT TestPush_SkippedKeySurvivesGCBeforeCommit9382026/11/12 00:02:00 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:39833/oidc939=== RUN TestPush_RejectsBadRequests/no_roots940=== PAUSE TestPush_RejectsBadRequests/no_roots941=== RUN TestPush_RejectsBadRequests/no_objects942=== PAUSE TestPush_RejectsBadRequests/no_objects943=== RUN TestPush_RejectsBadRequests/bad_root944=== PAUSE TestPush_RejectsBadRequests/bad_root945=== RUN TestPush_RejectsBadRequests/root_not_in_objects946=== PAUSE TestPush_RejectsBadRequests/root_not_in_objects947=== CONT TestMultipartCleanup9482026/11/12 00:02:00 INFO Received uploads request method=POST path=/api/pending_closures9492026/11/12 00:02:00 INFO Received uploads request method=POST path=/api/pending_closures9502026/11/12 00:02:00 INFO Received uploads request method=POST path=/api/pending_closures951=== CONT TestEnsureAbortUploadsRule952--- PASS: TestPresentRefreshesOnlyStaleClosures (1.82s)9532026/11/12 00:02:00 INFO Aborted multipart uploads count=0 kept=0954--- PASS: TestObjectStatsUpdatedOncePerCommit (1.83s)955=== CONT TestPush_RootsAreLockedInByteOrder9562026/11/12 00:02:00 INFO Received uploads request method=POST path=/api/pending_closures9572026/11/12 00:02:00 INFO Received uploads request method=POST path=/api/pending_closures958--- PASS: TestMultipartUploadAbortedWhenCancelledBeforeRecorded (1.73s)959=== CONT TestGenerateLandingPage960=== CONT TestPush_CompleteWaitsForCleanupOfThePush961--- PASS: TestGenerateLandingPage (4.28s)9622026/11/12 00:02:04 INFO Received cleanup request method=DELETE path=/api/pending_closures9632026/11/12 00:02:04 INFO Aborted multipart uploads count=0 kept=09642026/11/12 00:02:04 INFO Received uploads request method=POST path=/api/pending_closures9652026/11/12 00:02:04 WARN Force mode enabled - objects will be deleted immediately without grace period9662026/11/12 00:02:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=09672026/11/12 00:02:04 INFO Vacuumed table table=pending_closures9682026/11/12 00:02:04 INFO Vacuumed table table=pending_objects9692026/11/12 00:02:04 INFO Vacuumed table table=multipart_uploads9702026/11/12 00:02:04 INFO Vacuumed table table=closures9712026/11/12 00:02:04 INFO Vacuumed table table=objects9722026/11/12 00:02:04 INFO Received cleanup request method=DELETE path=/api/pending_closures9732026/11/12 00:02:04 INFO Aborted multipart uploads count=1 kept=09742026/11/12 00:02:04 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete9752026-11-12 00:02:04.594 UTC [467] ERROR: Closure does not exist: id=19762026-11-12 00:02:04.594 UTC [467] CONTEXT: PL/pgSQL function commit_pending_closure(bigint) line 13 at RAISE9772026-11-12 00:02:04.594 UTC [467] STATEMENT: -- name: CommitPendingClosure :one978 SELECT commit_pending_closure($1::bigint)::bigint979 980--- PASS: TestService_cleanupPendingClosuresHandler (6.23s)981=== CONT TestService_readinessHandler9822026/11/12 00:02:04 INFO Received complete multipart upload request method=POST path=/api/multipart/complete9832026/11/12 00:02:04 WARN CompleteMultipartUpload errored but object exists; treating as success error="The specified key does not exist." object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=NTQwMDM1NWYtMTEzNC00NTU1LWJlZGQtNzAyNDcwZjVhNWQ3LmI2NjY0ZGVhLTM1YmMtNGIwNy1iN2EyLWUzNzBmNDg2ZTE1YngxNzk0NDQxNzIwMjY0MTAyNjY29842026/11/12 00:02:04 INFO Received push request method=POST path=/api/pushes9852026/11/12 00:02:04 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=NTQwMDM1NWYtMTEzNC00NTU1LWJlZGQtNzAyNDcwZjVhNWQ3LmI2NjY0ZGVhLTM1YmMtNGIwNy1iN2EyLWUzNzBmNDg2ZTE1YngxNzk0NDQxNzIwMjY0MTAyNjY2 parts=19862026/11/12 00:02:04 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign987--- PASS: TestCompleteMultipartUpload_ErrorButObjectExists (6.27s)988=== CONT TestPush_RepeatedRootCommitsOnce9892026/11/12 00:02:04 INFO Signed narinfos id=1 count=19902026/11/12 00:02:04 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign9912026/11/12 00:02:04 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign9922026/11/12 00:02:04 ERROR Failed to sign narinfos signer="index 1" error="signing key cannot be nil"9932026/11/12 00:02:04 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign9942026/11/12 00:02:04 ERROR Failed to sign narinfos signer=niks3-test-1 error="signer failed"9952026/11/12 00:02:04 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign9962026/11/12 00:02:04 ERROR Signer returned unexpected signature count signer=niks3-test-1 got=0 want=19972026/11/12 00:02:04 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign9982026/11/12 00:02:04 ERROR Signer returned unexpected object key signer=niks3-test-1 object_key=unexpected.narinfo999--- PASS: TestPush_SignsNarinfosOfItsPendingObjects (6.28s)1000=== CONT TestService_healthCheckHandler1001--- PASS: TestObjectStatsTrigger (6.33s)1002=== CONT TestPush_CompleteCommitsEveryRoot1003=== NAME TestOrphanedObjectsGC1004 orphaned_objects_gc_test.go:287: GC Test Summary:1005 orphaned_objects_gc_test.go:288: - Kept: 2 objects from closure A1006 orphaned_objects_gc_test.go:289: - Deleted: 2 objects from closure B1007 orphaned_objects_gc_test.go:290: - Deleted: 6 orphaned chain objects (X1->X2->X3)1008 orphaned_objects_gc_test.go:291: - Deleted: 2 orphaned single objects (Y)1009 orphaned_objects_gc_test.go:292: - Total deleted: 10 objects1010--- PASS: TestOrphanedObjectsGC (6.34s)1011=== CONT TestGracefulShutdownDrainsInflight10122026/11/12 00:02:04 INFO Starting HTTP server address=127.0.0.1:4179510132026/11/12 00:02:04 INFO Shutdown signal received, draining in-flight requests timeout=10s1014--- PASS: TestService_Rustfstest (8.07s)1015=== CONT TestPush_OverlappingRootsStoreOneRowPerKey1016--- PASS: TestGracefulShutdownDrainsInflight (0.07s)10172026/11/12 00:02:04 INFO Received uploads request method=POST path=/api/pending_closures10182026/11/12 00:02:04 INFO Received uploads request method=POST path=/api/pending_closures10192026/11/12 00:02:04 INFO Received uploads request method=POST path=/api/pending_closures1020=== CONT TestCacheConfigHandlerMaxNarSize1021--- PASS: TestCacheConfigHandlerMaxNarSize (0.00s)1022=== CONT TestReadRedirectUsesPublicS3URL10232026/11/12 00:02:04 INFO Received uploads request method=POST path=/api/pending_closures10242026/11/12 00:02:04 INFO Received uploads request method=POST path=/api/pending_closures10252026/11/12 00:02:04 INFO Received complete multipart upload request method=POST path=/api/multipart/complete10262026/11/12 00:02:04 INFO Received cleanup request method=DELETE path=/api/pending_closures10272026/11/12 00:02:04 ERROR Failed to get multipart upload error="no rows in result set" upload_id=does-not-exist object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst1028--- PASS: TestCompleteMultipartUnregistered (8.19s)1029=== CONT TestServerTLSConfig1030=== RUN TestServerTLSConfig/no_client_CA1031=== PAUSE TestServerTLSConfig/no_client_CA1032=== RUN TestServerTLSConfig/missing_CA_file1033=== PAUSE TestServerTLSConfig/missing_CA_file1034=== RUN TestServerTLSConfig/not_a_PEM_file1035=== PAUSE TestServerTLSConfig/not_a_PEM_file1036=== CONT TestReadProxyRangeRequest10372026/11/12 00:02:04 INFO Received uploads request method=POST path=/api/pending_closures10382026/11/12 00:02:04 INFO Received complete multipart upload request method=POST path=/api/multipart/complete10392026/11/12 00:02:04 INFO Received complete multipart upload request method=POST path=/api/multipart/complete10402026/11/12 00:02:04 INFO Received uploads request method=POST path=/api/pending_closures10412026/11/12 00:02:04 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0100000000000000000000.nar.zst upload_id=NTQwMDM1NWYtMTEzNC00NTU1LWJlZGQtNzAyNDcwZjVhNWQ3LmFjZjgwZjczLWRmMDgtNDZmNC05ZWIwLWQ2NmJjZGQ2M2NlNXgxNzk0NDQxNzIwMTMwMzA4NzMw parts=1210422026/11/12 00:02:04 INFO Completed multipart upload object_key=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhh0100000000000000000000.nar.zst upload_id=NTQwMDM1NWYtMTEzNC00NTU1LWJlZGQtNzAyNDcwZjVhNWQ3LjM0NmRiYmIwLWY1ZjAtNDNjZS04MWY1LTExMjYyY2RhYzA2ZXgxNzk0NDQxNzIwMTU3OTIyNTUz parts=1210432026/11/12 00:02:04 INFO Received uploads request method=POST path=/api/pending_closures1044--- PASS: TestCompletedNarNotReofferedAcrossClosures (6.59s)1045=== CONT TestService_NativeMTLS10462026/11/12 00:02:04 INFO Registered completed upload object_key=nar/jjjjjjjjjjjjjjjjjjjjjjjjjjjjjj0100000000000000000000.nar.zst10472026/11/12 00:02:04 INFO Received uploads request method=POST path=/api/pending_closures1048--- PASS: TestPresignedUploadRegisteredBeforeCommit (8.30s)1049=== CONT TestReadRedirectKeepsNarinfoProxied1050--- PASS: TestPresentReportsOnlyClosureRoots (8.31s)1051=== CONT TestMetricsCommit10522026/11/12 00:02:05 INFO Received uploads request method=POST path=/api/pending_closures10532026/11/12 00:02:05 INFO Received complete multipart upload request method=POST path=/api/multipart/complete1054--- PASS: TestCreatePendingClosure_SmallNARUsesSimplePUT (8.36s)1055=== CONT TestReadRedirectNar10562026/11/12 00:02:05 INFO Received uploads request method=POST path=/api/pending_closures10572026/11/12 00:02:05 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux10582026/11/12 00:02:05 WARN Refused reserved pin name=worker-x86_64-linux10592026/11/12 00:02:05 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux10602026/11/12 00:02:05 INFO Received create pin request method=POST path=/api/pins/my-app10612026/11/12 00:02:05 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux1062--- PASS: TestCreatePin_ReservedPins (6.73s)1063=== CONT TestMetricsInventory10642026/11/12 00:02:05 INFO Received push request method=POST path=/api/pushes10652026/11/12 00:02:05 INFO Received complete push request method=POST path=/api/pushes/1/complete10662026/11/12 00:02:05 INFO Received uploads request method=POST path=/api/pending_closures10672026/11/12 00:02:05 INFO Received complete multipart upload request method=POST path=/api/multipart/complete10682026/11/12 00:02:05 INFO Completed multipart upload object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst upload_id=NTQwMDM1NWYtMTEzNC00NTU1LWJlZGQtNzAyNDcwZjVhNWQ3LjIwM2EwYzRjLWQ1NGYtNGJlMC05YmM3LTJiY2EzMjg3YmIxNngxNzk0NDQxNzI0Nzk5NzI4OTU5 parts=1010692026/11/12 00:02:05 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete10702026/11/12 00:02:05 INFO Completed upload id=1 objects=2 duration=4.579724ms10712026/11/12 00:02:05 INFO Received get closure request method=GET path=/api/closures/0000000000000000000000000000000010722026/11/12 00:02:05 INFO Received uploads request method=POST path=/api/pending_closures10732026/11/12 00:02:05 INFO Starting cleanup of old closures method=DELETE path=/api/closures10742026/11/12 00:02:05 INFO Aborted multipart uploads count=0 kept=010752026/11/12 00:02:05 INFO Received cleanup request method=DELETE path=/api/pending_closures10762026/11/12 00:02:05 WARN Failed to abort upload, keeping its closure key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst error="Get \"http://127.0.0.1:1/bucket28/?location=\": dial tcp 127.0.0.1:1: connect: connection refused" code=""10772026/11/12 00:02:05 INFO Aborted multipart uploads count=0 kept=110782026/11/12 00:02:05 INFO Received cleanup request method=DELETE path=/api/pending_closures10792026/11/12 00:02:05 INFO Aborted multipart uploads count=1 kept=01080--- PASS: TestMultipartCleanup (5.24s)1081=== CONT TestReadProxyDisabled10822026/11/12 00:02:05 INFO Received complete multipart upload request method=POST path=/api/multipart/complete10832026/11/12 00:02:05 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst upload_id=NTQwMDM1NWYtMTEzNC00NTU1LWJlZGQtNzAyNDcwZjVhNWQ3LmQxMTc0OTY3LTg4ZWUtNGU2ZS1iZjI0LWQ1YjA3OTZhYzk4N3gxNzk0NDQxNzI1MDUxNjk4MDQz parts=1010842026/11/12 00:02:05 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete10852026/11/12 00:02:05 INFO Completed upload id=1 objects=2 duration=7.446651ms10862026/11/12 00:02:05 INFO Received complete multipart upload request method=POST path=/api/multipart/complete10872026/11/12 00:02:05 INFO Received uploads request method=POST path=/api/pending_closures10882026/11/12 00:02:05 INFO Received uploads request method=POST path=/api/pending_closures10892026/11/12 00:02:05 INFO Object in database but missing from S3 key=dadb44fdadb44fdadb44fdadb44f0000.narinfo10902026/11/12 00:02:05 WARN Found objects in DB but missing from S3, will re-upload count=110912026/11/12 00:02:05 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0100000000000000000000.nar.zst upload_id=NTQwMDM1NWYtMTEzNC00NTU1LWJlZGQtNzAyNDcwZjVhNWQ3LjlkZjMzNGQ4LWFiMDktNGI0NC1hY2FhLWRkZTZlNDgyMzQyZHgxNzk0NDQxNzIwMTUzNjYyMjEz parts=121092--- PASS: TestService_verifyS3Integrity (8.81s)1093=== CONT TestNARDeduplicationMetadataUploadBug1094--- PASS: TestSharedNarBothUploadsComplete (7.13s)1095=== CONT TestReadProxyRootRedirectsToIndexHTML10962026/11/12 00:02:06 INFO Completed push id=1 objects=4 duration=1.336279669s10972026/11/12 00:02:06 INFO Received push request method=POST path=/api/pushes10982026/11/12 00:02:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=01099--- PASS: TestCommitRacingPendingCleanupKeepsObjects (8.12s)1100=== CONT TestCreatePendingClosureRejectsOversizedNAR11012026/11/12 00:02:06 INFO Received uploads request method=POST path=/api/pending_closures1102--- PASS: TestCreatePendingClosureRejectsOversizedNAR (0.00s)1103=== CONT TestReadProxyConditionalGet11042026/11/12 00:02:06 INFO Vacuumed table table=pending_closures11052026/11/12 00:02:06 INFO Vacuumed table table=pending_objects11062026/11/12 00:02:06 INFO Vacuumed table table=multipart_uploads11072026/11/12 00:02:06 INFO Vacuumed table table=closures11082026/11/12 00:02:06 INFO Vacuumed table table=objects11092026/11/12 00:02:06 INFO Received get closure request method=GET path=/api/closures/000000000000000000000000000000001110=== NAME TestOrphanedObjectsGCStressTest1111 orphaned_objects_gc_test.go:422: Created 10 active closures, 5 to-delete closures, 20 orphaned chains1112--- PASS: TestService_createPendingClosureHandler (9.89s)1113=== CONT TestReadProxyOutlastsServerWriteTimeout11142026/11/12 00:02:06 INFO Aborted multipart uploads count=0 kept=011152026/11/12 00:02:06 WARN Force mode enabled - objects will be deleted immediately without grace period1116--- PASS: TestReadProxyDisabled (1.32s)1117=== CONT TestReadProxyContentNegotiation1118--- PASS: TestReadRedirectKeepsNarinfoProxied (1.66s)1119=== CONT TestReadProxyNarinfoAlreadyDecompressed11202026/11/12 00:02:06 INFO Received push request method=POST path=/api/pushes11212026/11/12 00:02:06 INFO Received complete push request method=POST path=/api/pushes/1/complete11222026/11/12 00:02:06 INFO Received push request method=POST path=/api/pushes11232026/11/12 00:02:06 WARN mTLS auth: subject not in bound subjects subject="CN=reader"11242026/11/12 00:02:06 WARN mTLS auth: subject not in bound subjects subject="CN=reader"1125=== CONT TestReadProxyHead1126--- PASS: TestService_NativeMTLS (1.82s)11272026/11/12 00:02:06 INFO Received complete push request method=POST path=/api/pushes/1/complete11282026/11/12 00:02:06 INFO Received push request method=POST path=/api/pushes1129--- PASS: TestReadProxyRootRedirectsToIndexHTML (1.33s)1130=== CONT TestReadProxyNarinfo11312026/11/12 00:02:06 INFO Received push request method=POST path=/api/pushes1132--- PASS: TestPush_OverlappingRootsStoreOneRowPerKey (2.14s)1133=== CONT TestIsValidCachePath1134=== RUN TestIsValidCachePath/narinfo1135=== PAUSE TestIsValidCachePath/narinfo1136=== RUN TestIsValidCachePath/narinfo_all_nix_base32_chars1137=== PAUSE TestIsValidCachePath/narinfo_all_nix_base32_chars1138=== RUN TestIsValidCachePath/nar_zst1139=== PAUSE TestIsValidCachePath/nar_zst1140=== RUN TestIsValidCachePath/nar_xz1141=== PAUSE TestIsValidCachePath/nar_xz1142=== RUN TestIsValidCachePath/nar_bz21143=== PAUSE TestIsValidCachePath/nar_bz21144=== RUN TestIsValidCachePath/nar_uncompressed1145=== PAUSE TestIsValidCachePath/nar_uncompressed1146=== RUN TestIsValidCachePath/ls1147=== PAUSE TestIsValidCachePath/ls1148=== RUN TestIsValidCachePath/log1149=== PAUSE TestIsValidCachePath/log1150=== RUN TestIsValidCachePath/realisation1151=== PAUSE TestIsValidCachePath/realisation1152=== RUN TestIsValidCachePath/nix-cache-info1153=== PAUSE TestIsValidCachePath/nix-cache-info1154=== RUN TestIsValidCachePath/index.html1155=== PAUSE TestIsValidCachePath/index.html1156=== RUN TestIsValidCachePath/traversal_parent1157=== PAUSE TestIsValidCachePath/traversal_parent1158=== RUN TestIsValidCachePath/traversal_in_middle1159=== PAUSE TestIsValidCachePath/traversal_in_middle1160=== RUN TestIsValidCachePath/invalid_char_e1161=== PAUSE TestIsValidCachePath/invalid_char_e1162=== RUN TestIsValidCachePath/invalid_char_u1163=== PAUSE TestIsValidCachePath/invalid_char_u1164=== RUN TestIsValidCachePath/random_path1165=== PAUSE TestIsValidCachePath/random_path1166=== RUN TestIsValidCachePath/empty1167=== PAUSE TestIsValidCachePath/empty1168=== RUN TestIsValidCachePath/leading_slash1169=== PAUSE TestIsValidCachePath/leading_slash1170=== RUN TestIsValidCachePath/wrong_extension1171=== PAUSE TestIsValidCachePath/wrong_extension1172=== RUN TestIsValidCachePath/short_hash1173=== PAUSE TestIsValidCachePath/short_hash1174=== CONT TestParseSingleRange1175=== RUN TestParseSingleRange/none1176=== PAUSE TestParseSingleRange/none1177=== RUN TestParseSingleRange/unknown_unit1178=== PAUSE TestParseSingleRange/unknown_unit1179=== RUN TestParseSingleRange/multi-range_ignored1180=== PAUSE TestParseSingleRange/multi-range_ignored1181=== RUN TestParseSingleRange/malformed_no_dash1182=== PAUSE TestParseSingleRange/malformed_no_dash1183=== RUN TestParseSingleRange/malformed_both_empty1184=== PAUSE TestParseSingleRange/malformed_both_empty1185=== RUN TestParseSingleRange/malformed_end_before_start1186=== PAUSE TestParseSingleRange/malformed_end_before_start1187=== RUN TestParseSingleRange/closed1188=== PAUSE TestParseSingleRange/closed1189=== RUN TestParseSingleRange/open-ended1190=== PAUSE TestParseSingleRange/open-ended1191=== RUN TestParseSingleRange/end_clamped_to_size1192=== PAUSE TestParseSingleRange/end_clamped_to_size1193=== RUN TestParseSingleRange/suffix1194=== PAUSE TestParseSingleRange/suffix1195=== RUN TestParseSingleRange/suffix_exceeds_size1196=== PAUSE TestParseSingleRange/suffix_exceeds_size1197=== RUN TestParseSingleRange/single_byte1198=== PAUSE TestParseSingleRange/single_byte1199=== RUN TestParseSingleRange/start_past_EOF1200=== PAUSE TestParseSingleRange/start_past_EOF1201=== RUN TestParseSingleRange/start_far_past_EOF1202=== PAUSE TestParseSingleRange/start_far_past_EOF1203=== CONT TestPendingClosureWriteTimeout1204=== RUN TestPendingClosureWriteTimeout/empty1205=== PAUSE TestPendingClosureWriteTimeout/empty1206=== RUN TestPendingClosureWriteTimeout/400_objects1207=== PAUSE TestPendingClosureWriteTimeout/400_objects1208=== RUN TestPendingClosureWriteTimeout/670k_objects1209=== PAUSE TestPendingClosureWriteTimeout/670k_objects1210=== RUN TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline1211=== PAUSE TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline1212=== CONT TestGCTaskStore_PhaseUpdates1213--- PASS: TestGCTaskStore_PhaseUpdates (0.00s)1214=== CONT TestReadProxy4041215--- PASS: TestReadRedirectNar (1.87s)1216=== CONT TestGCSweepSkipsPendingObjects12172026/11/12 00:02:06 INFO Received push request method=POST path=/api/pushes1218--- PASS: TestReadProxyRangeRequest (2.10s)1219=== CONT TestSweepRechecksPendingAfterLocking1220--- PASS: TestMetricsInventory (1.89s)1221=== CONT TestGetExistingObjectsLocksRows12222026/11/12 00:02:07 INFO Received push request method=POST path=/api/pushes12232026/11/12 00:02:07 INFO Installed lifecycle rule aborting incomplete NAR uploads bucket=bucket42 days=21224--- PASS: TestEnsureAbortUploadsRule (6.83s)1225=== CONT TestRegisterCompletedObjectNeedsPendingRow12262026/11/12 00:02:07 INFO Received complete push request method=POST path=/api/pushes/1/complete1227--- PASS: TestReadRedirectUsesPublicS3URL (2.28s)1228=== CONT TestSweepSkipsRowsLockedByPush12292026/11/12 00:02:07 WARN readiness check failed error="closed pool"1230--- PASS: TestService_readinessHandler (2.47s)1231=== CONT TestFailedS3DeleteKeepsTombstone1232--- PASS: TestService_healthCheckHandler (2.44s)1233=== CONT TestForceGCDuringPushOffersSweptObject1234=== RUN TestForceGCDuringPushOffersSweptObject/before_pending_rows1235=== PAUSE TestForceGCDuringPushOffersSweptObject/before_pending_rows1236=== RUN TestForceGCDuringPushOffersSweptObject/after_presence_check1237=== PAUSE TestForceGCDuringPushOffersSweptObject/after_presence_check1238=== CONT TestCreatePendingClosureVerifyS3FailureReleasesConnection12392026-11-12 00:02:07.234 UTC [625] ERROR: Push does not exist: id=112402026-11-12 00:02:07.234 UTC [625] CONTEXT: PL/pgSQL function commit_push(bigint) line 11 at RAISE12412026-11-12 00:02:07.234 UTC [625] STATEMENT: SELECT commit_push($1)1242--- PASS: TestPush_CompleteWaitsForCleanupOfThePush (2.68s)1243=== CONT TestGCSweepDeliversEachKeyOnce12442026-11-12 00:02:07.773 UTC [611] ERROR: could not obtain lock on row in relation "closures"12452026-11-12 00:02:07.773 UTC [611] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT12462026-11-12 00:02:07.774 UTC [611] ERROR: could not obtain lock on row in relation "closures"12472026-11-12 00:02:07.774 UTC [611] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT12482026-11-12 00:02:07.774 UTC [611] ERROR: could not obtain lock on row in relation "closures"12492026-11-12 00:02:07.774 UTC [611] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT12502026-11-12 00:02:07.774 UTC [611] ERROR: could not obtain lock on row in relation "closures"12512026-11-12 00:02:07.774 UTC [611] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT12522026-11-12 00:02:07.775 UTC [611] ERROR: could not obtain lock on row in relation "closures"12532026-11-12 00:02:07.775 UTC [611] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT12542026/11/12 00:02:07 INFO Completed push id=1 objects=4 duration=728.078206ms12552026/11/12 00:02:07 INFO Completed push id=1 objects=6 duration=992.547289ms12562026/11/12 00:02:07 INFO Completed push id=1 objects=4 duration=1.097157707s1257--- PASS: TestPush_RepeatedRootCommitsOnce (3.13s)1258=== CONT TestTombstonedObjectOfferedWithoutWaiting1259=== NAME TestOrphanedObjectsGCStressTest1260 orphaned_objects_gc_test.go:440: Marked 210 objects for deletion1261--- PASS: TestPush_CompleteCommitsEveryRoot (3.08s)1262=== CONT TestConnectWaitsForAPeerMigration1263--- PASS: TestPush_RootsAreLockedInByteOrder (7.57s)1264=== CONT TestDeduplicatedObjectsRecordedAsPending12652026/11/12 00:02:07 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete12662026/11/12 00:02:07 INFO Completed upload id=2 objects=2 duration=3.363912ms1267--- PASS: TestMetricsCommit (2.82s)1268=== CONT TestPushDedupSurvivesConcurrentGC12692026/11/12 00:02:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=2 objects-deleted-after-grace-period=2 objects-failed-to-delete=01270=== NAME TestNARDeduplicationMetadataUploadBug1271 metadata_upload_test.go:48: First store path: /build/TestNARDeduplicationMetadataUploadBug4247417565/001/store/ahblc6fyfz8w6i36hh8fms4d1i5s83mg-file1.txt12722026/11/12 00:02:07 INFO Vacuumed table table=pending_closures12732026/11/12 00:02:07 INFO Vacuumed table table=pending_objects12742026/11/12 00:02:07 INFO Vacuumed table table=multipart_uploads12752026/11/12 00:02:07 INFO Vacuumed table table=closures12762026/11/12 00:02:07 INFO Vacuumed table table=objects12772026/11/12 00:02:07 INFO Received complete push request method=POST path=/api/pushes/2/complete12782026/11/12 00:02:07 INFO Completed push id=2 objects=4 duration=6.3413ms1279--- PASS: TestPush_SkippedKeySurvivesGCBeforeCommit (8.03s)1280=== CONT TestGCMetrics12812026/11/12 00:02:07 INFO Received uploads request method=POST path=/api/pending_closures12822026/11/12 00:02:07 INFO Aborted multipart uploads count=0 kept=01283--- PASS: TestGetExistingObjectsLocksRows (0.96s)1284=== CONT TestGCBugBareHashReferences1285--- PASS: TestRegisterCompletedObjectNeedsPendingRow (0.97s)1286=== CONT TestGCAdvisoryLockBlocksConcurrentRun12872026/11/12 00:02:07 INFO Aborted multipart uploads count=0 kept=01288--- PASS: TestReadProxy404 (1.15s)1289=== CONT TestLeadEndsOnShutdown12902026/11/12 00:02:08 INFO Received uploads request method=POST path=/api/pending_closures12912026/11/12 00:02:08 INFO Aborted multipart uploads count=0 kept=012922026/11/12 00:02:08 INFO Aborted multipart uploads count=0 kept=01293--- PASS: TestReadProxyConditionalGet (1.60s)1294=== CONT TestLeadGenerationGrowsWithEveryGrant12952026/11/12 00:02:08 INFO Received uploads request method=POST path=/api/pending_closures12962026/11/12 00:02:08 INFO Aborted multipart uploads count=0 kept=01297--- PASS: TestReadProxyNarinfoAlreadyDecompressed (1.52s)1298=== CONT TestLeadIncumbentAnnouncesAtOnce1299--- PASS: TestReadProxyHead (1.43s)1300=== CONT TestLeadUnannouncedStandbyYieldsToIncumbent1301--- PASS: TestCreatePendingClosureVerifyS3FailureReleasesConnection (1.16s)1302=== CONT TestLeadElectsOneAndHandsOver13032026/11/12 00:02:08 ERROR Failed to decode narinfo key=3hcdxyjf9yiq7qf3i4548drb6sjmwa1v.narinfo limit=16777216 error=<nil>1304--- PASS: TestReadProxyNarinfo (1.42s)1305=== CONT TestClientPushesUseOnePush1306--- PASS: TestReadProxyContentNegotiation (1.69s)1307=== CONT TestResolveDBConnectionString1308=== RUN TestResolveDBConnectionString/flag_wins1309=== PAUSE TestResolveDBConnectionString/flag_wins1310=== RUN TestResolveDBConnectionString/file_when_flag_empty1311=== PAUSE TestResolveDBConnectionString/file_when_flag_empty1312=== RUN TestResolveDBConnectionString/missing_file_is_an_error1313=== PAUSE TestResolveDBConnectionString/missing_file_is_an_error1314=== RUN TestResolveDBConnectionString/PGHOST_allows_empty1315=== PAUSE TestResolveDBConnectionString/PGHOST_allows_empty1316=== RUN TestResolveDBConnectionString/nothing_configured1317=== PAUSE TestResolveDBConnectionString/nothing_configured1318=== CONT TestConcurrentCommitsSharingObjectsDoNotDeadlock1319=== NAME TestOrphanedObjectsGCStressTest1320 orphaned_objects_gc_test.go:503: Stress test completed successfully:1321 orphaned_objects_gc_test.go:504: - Active objects preserved: 201322 orphaned_objects_gc_test.go:505: - Objects deleted: 2101323 orphaned_objects_gc_test.go:506: - Total GC'd: 2101324--- PASS: TestOrphanedObjectsGCStressTest (12.02s)1325=== CONT TestDeleteClosuresRacingPins13262026/11/12 00:02:09 WARN Force mode enabled - objects will be deleted immediately without grace period13272026/11/12 00:02:09 WARN Force mode enabled - objects will be deleted immediately without grace period13282026/11/12 00:02:09 WARN Force mode enabled - objects will be deleted immediately without grace period13292026/11/12 00:02:09 WARN Force mode enabled - objects will be deleted immediately without grace period13302026/11/12 00:02:09 WARN Force mode enabled - objects will be deleted immediately without grace period13312026/11/12 00:02:09 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=013322026/11/12 00:02:09 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=013332026/11/12 00:02:09 INFO Vacuumed table table=pending_closures13342026/11/12 00:02:09 INFO Vacuumed table table=pending_objects13352026/11/12 00:02:09 INFO Vacuumed table table=multipart_uploads13362026/11/12 00:02:09 INFO Vacuumed table table=closures13372026/11/12 00:02:09 INFO Vacuumed table table=pending_closures13382026/11/12 00:02:09 INFO Vacuumed table table=pending_objects13392026/11/12 00:02:09 INFO Vacuumed table table=objects13402026/11/12 00:02:09 INFO Vacuumed table table=multipart_uploads13412026/11/12 00:02:09 INFO Vacuumed table table=closures13422026/11/12 00:02:09 INFO Vacuumed table table=objects13432026/11/12 00:02:09 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=013442026/11/12 00:02:09 INFO Vacuumed table table=pending_closures13452026/11/12 00:02:09 INFO Vacuumed table table=pending_objects13462026/11/12 00:02:09 INFO Vacuumed table table=multipart_uploads13472026/11/12 00:02:09 INFO Vacuumed table table=closures13482026/11/12 00:02:09 INFO Vacuumed table table=objects1349--- PASS: TestSweepRechecksPendingAfterLocking (2.28s)13502026/11/12 00:02:09 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."1351=== CONT TestClientFallsBackToClosures13522026/11/12 00:02:09 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."1353--- PASS: TestFailedS3DeleteKeepsTombstone (2.18s)1354=== CONT TestService_AuthMiddleware_MTLSBoundSubjects13552026/11/12 00:02:09 INFO Aborted multipart uploads count=0 kept=01356--- PASS: TestGCSweepSkipsPendingObjects (2.34s)1357=== CONT TestService_AuthMiddleware_OIDC13582026/11/12 00:02:09 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=013592026/11/12 00:02:09 INFO Vacuumed table table=pending_closures13602026/11/12 00:02:09 INFO Vacuumed table table=pending_objects13612026/11/12 00:02:09 INFO Vacuumed table table=multipart_uploads13622026/11/12 00:02:09 INFO Vacuumed table table=closures13632026/11/12 00:02:09 INFO lead: acquired remote=192.0.2.1:1234 generation=113642026/11/12 00:02:09 INFO Vacuumed table table=objects1365--- PASS: TestGCSweepDeliversEachKeyOnce (2.06s)1366=== CONT TestService_ReadAuthMiddleware13672026/11/12 00:02:09 INFO lead: acquired remote=192.0.2.1:1234 generation=113682026/11/12 00:02:09 INFO lead: released remote=192.0.2.1:1234 generation=11369--- PASS: TestLeadIncumbentAnnouncesAtOnce (1.18s)1370=== CONT TestService_RequireScope_OIDC13712026/11/12 00:02:09 INFO lead: acquired remote=192.0.2.1:1234 generation=113722026/11/12 00:02:09 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:42367/oidc13732026/11/12 00:02:09 INFO Aborted multipart uploads count=0 kept=013742026/11/12 00:02:09 INFO lead: acquired remote=192.0.2.1:1234 generation=113752026/11/12 00:02:09 INFO Received uploads request method=POST path=/api/pending_closures1376--- PASS: TestTombstonedObjectOfferedWithoutWaiting (1.66s)1377=== CONT TestCacheConfigHandler1378=== RUN TestCacheConfigHandler/full_config,_no_issuer1379=== PAUSE TestCacheConfigHandler/full_config,_no_issuer1380=== RUN TestCacheConfigHandler/no_cache_url_configured1381=== PAUSE TestCacheConfigHandler/no_cache_url_configured1382=== RUN TestCacheConfigHandler/no_signing_keys1383=== PAUSE TestCacheConfigHandler/no_signing_keys1384=== RUN TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1385=== PAUSE TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1386=== CONT TestService_ReadScope_PublicByDefault13872026/11/12 00:02:09 INFO Received uploads request method=POST path=/api/pending_closures13882026/11/12 00:02:09 ERROR failed to check GC advisory lock error="query pg_locks: failed to connect to `user=nixbld database=niks3`: /nonexistent/.s.PGSQL.5432 (/nonexistent): dial error: dial unix /nonexistent/.s.PGSQL.5432: connect: no such file or directory"1389--- PASS: TestGCAdvisoryLockBlocksConcurrentRun (1.48s)1390=== CONT TestReadProxyNarStreaming13912026/11/12 00:02:09 INFO Aborted multipart uploads count=0 kept=013922026/11/12 00:02:09 INFO Received uploads request method=POST path=/api/pending_closures1393--- PASS: TestDeduplicatedObjectsRecordedAsPending (1.76s)1394=== CONT TestService_AuthMiddleware_MTLSProxyHeader13952026/11/12 00:02:09 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:42117/oidc13962026/11/12 00:02:09 WARN Rate limiter enabled after throttle name=s3-test rate=513972026/11/12 00:02:09 WARN S3 rate limit hit operation="complete multipart upload" error="Please reduce your request rate."1398=== NAME TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle1399 throttle_test.go:213: Proxy stats: total=13, throttled=10, completeMultipart=101400 throttle_test.go:215: Rate limiter: enabled=true, rate=5.001401--- PASS: TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle (13.28s)1402=== CONT TestClientWithDependencies14032026/11/12 00:02:10 INFO lead: released remote=192.0.2.1:1234 generation=11404--- PASS: TestLeadEndsOnShutdown (2.35s)1405=== CONT TestPinProtectsFromGC14062026/11/12 00:02:10 INFO lead: released remote=192.0.2.1:1234 generation=114072026/11/12 00:02:10 INFO lead: acquired remote=192.0.2.1:1234 generation=214082026/11/12 00:02:10 INFO lead: given back to the incumbent remote=192.0.2.1:1234 generation=214092026/11/12 00:02:10 INFO lead: released remote=192.0.2.1:1234 generation=114102026/11/12 00:02:10 INFO lead: acquired remote=192.0.2.1:1234 generation=214112026/11/12 00:02:10 INFO lead: acquired remote=192.0.2.1:1234 generation=314122026/11/12 00:02:10 INFO Aborted multipart uploads count=1 kept=014132026/11/12 00:02:10 INFO Received cleanup request method=DELETE path=/api/pending_closures14142026/11/12 00:02:10 INFO Aborted multipart uploads count=1 kept=01415--- PASS: TestPendingCleanupUsesOneCutoff (12.52s)1416=== CONT TestClientSharedPathCommittedMidPush1417=== NAME TestPinProtectsFromGC1418 client_integration_test.go:779: Pinned store path: /build/TestPinProtectsFromGC1660759056/001/store/qa3g9999gl3pnc87bs5ybf1jfb9rb67z-pinned-file.txt1419 client_integration_test.go:780: Unpinned store path: /build/TestPinProtectsFromGC1660759056/001/store/l6p4l6lvsn6iyfgkd2sd029ds4qzj4ny-unpinned-file.txt1420=== NAME TestClientWithDependencies1421 client_integration_test.go:661: Built derivation: /build/TestClientWithDependencies2903173064/001/store/gffri7dp6v7vfcd2gc9695zb97x670li-test-script1422 client_integration_test.go:663: Found 1 dependencies (including self)14232026/11/12 00:02:11 INFO Received push request method=POST path=/api/pushes14242026/11/12 00:02:11 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)14252026/11/12 00:02:11 INFO Uploading qa3g9999gl3pnc87bs5ybf1jfb9rb67z-pinned-file.txt (128B)14262026/11/12 00:02:11 WARN Failed to register uploaded object key=nar/0qmhsg42pqsz7x6hh02j8gznnac3yi3qpjhrqbwvz88i194jbb5s.nar.zst error="server returned 404: 404 page not found\n"14272026/11/12 00:02:11 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign14282026/11/12 00:02:11 WARN Failed to register uploaded object key=qa3g9999gl3pnc87bs5ybf1jfb9rb67z.ls error="server returned 404: 404 page not found\n"14292026/11/12 00:02:11 INFO Signed narinfos id=1 count=114302026/11/12 00:02:11 INFO Uploading 1 narinfos14312026/11/12 00:02:11 INFO Received complete push request method=POST path=/api/pushes/1/complete14322026/11/12 00:02:11 WARN Failed to register uploaded object key=qa3g9999gl3pnc87bs5ybf1jfb9rb67z.narinfo error="server returned 404: 404 page not found\n"14332026/11/12 00:02:11 INFO Completed push id=1 objects=3 duration=7.32867ms14342026/11/12 00:02:11 INFO Upload complete. (95ms)14352026/11/12 00:02:11 INFO Received push request method=POST path=/api/pushes14362026/11/12 00:02:11 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)14372026/11/12 00:02:11 INFO Uploading gffri7dp6v7vfcd2gc9695zb97x670li-test-script (136B)14382026/11/12 00:02:11 WARN Failed to register uploaded object key=gffri7dp6v7vfcd2gc9695zb97x670li.ls error="server returned 404: 404 page not found\n"14392026/11/12 00:02:11 WARN Failed to register uploaded object key=nar/1s7yghixzpinyqm7g59kyn3sav81pmnsdi5v63m6x4r2vacy907j.nar.zst error="server returned 404: 404 page not found\n"14402026/11/12 00:02:11 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign14412026/11/12 00:02:11 WARN Failed to register uploaded object key=log/s3lybvb0y03ivslazirzma9d4hni3rbx-test-script.drv error="server returned 404: 404 page not found\n"14422026/11/12 00:02:11 INFO Signed narinfos id=1 count=114432026/11/12 00:02:11 INFO Uploading 1 narinfos14442026/11/12 00:02:11 INFO Received complete push request method=POST path=/api/pushes/1/complete14452026/11/12 00:02:11 WARN Failed to register uploaded object key=gffri7dp6v7vfcd2gc9695zb97x670li.narinfo error="server returned 404: 404 page not found\n"14462026/11/12 00:02:11 INFO Completed push id=1 objects=4 duration=5.30055ms14472026/11/12 00:02:11 INFO Upload complete. (75ms)1448 client_integration_test.go:665: Skipping nix copy test - isolated store (/build/TestClientWithDependencies2903173064/001/store) requires matching store prefix1449--- PASS: TestClientWithDependencies (1.20s)1450=== CONT TestService_AuthMiddleware14512026/11/12 00:02:11 INFO Received push request method=POST path=/api/pushes14522026/11/12 00:02:11 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)14532026/11/12 00:02:11 INFO Uploading l6p4l6lvsn6iyfgkd2sd029ds4qzj4ny-unpinned-file.txt (128B)14542026/11/12 00:02:11 WARN Failed to register uploaded object key=nar/0gh0wdnr6gmk4r5sfzzh12isnl14rgmkrwy1ws41hpsrlndclg5h.nar.zst error="server returned 404: 404 page not found\n"14552026/11/12 00:02:11 INFO Received sign narinfos request method=POST path=/api/pushes/2/sign14562026/11/12 00:02:11 WARN Failed to register uploaded object key=l6p4l6lvsn6iyfgkd2sd029ds4qzj4ny.ls error="server returned 404: 404 page not found\n"14572026/11/12 00:02:11 INFO Signed narinfos id=2 count=114582026/11/12 00:02:11 INFO Uploading 1 narinfos14592026/11/12 00:02:11 INFO Received complete push request method=POST path=/api/pushes/2/complete14602026/11/12 00:02:11 WARN Failed to register uploaded object key=l6p4l6lvsn6iyfgkd2sd029ds4qzj4ny.narinfo error="server returned 404: 404 page not found\n"14612026/11/12 00:02:11 INFO Completed push id=2 objects=3 duration=3.599154ms14622026/11/12 00:02:11 INFO Upload complete. (78ms)14632026/11/12 00:02:11 INFO lead: released remote=192.0.2.1:1234 generation=214642026/11/12 00:02:11 INFO lead: acquired remote=192.0.2.1:1234 generation=314652026/11/12 00:02:11 INFO lead: released remote=192.0.2.1:1234 generation=31466--- PASS: TestLeadGenerationGrowsWithEveryGrant (3.56s)1467=== CONT TestClientMultipleUploads1468--- PASS: TestGCBugBareHashReferences (4.43s)1469=== CONT TestClientIntegration14702026/11/12 00:02:12 WARN Force mode enabled - objects will be deleted immediately without grace period14712026/11/12 00:02:12 WARN Force mode enabled - objects will be deleted immediately without grace period14722026/11/12 00:02:12 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=014732026/11/12 00:02:12 INFO Vacuumed table table=pending_closures14742026/11/12 00:02:12 INFO Vacuumed table table=pending_objects14752026/11/12 00:02:12 INFO Vacuumed table table=multipart_uploads14762026/11/12 00:02:12 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=014772026/11/12 00:02:12 INFO Vacuumed table table=closures14782026/11/12 00:02:12 INFO Vacuumed table table=objects14792026/11/12 00:02:12 INFO Vacuumed table table=pending_closures1480--- PASS: TestGCMetrics (4.54s)1481=== CONT TestClientCADerivations14822026/11/12 00:02:12 INFO Vacuumed table table=pending_objects14832026/11/12 00:02:12 INFO Vacuumed table table=multipart_uploads14842026/11/12 00:02:12 INFO Vacuumed table table=closures14852026/11/12 00:02:12 INFO Vacuumed table table=objects14862026/11/12 00:02:12 INFO Aborted multipart uploads count=0 kept=014872026/11/12 00:02:12 INFO lead: released remote=192.0.2.1:1234 generation=31488--- PASS: TestLeadUnannouncedStandbyYieldsToIncumbent (4.22s)1489=== CONT TestPushWaitsForSweepHoldingRow14902026/11/12 00:02:12 WARN Force mode enabled - objects will be deleted immediately without grace period14912026/11/12 00:02:12 INFO Starting cleanup of old closures method=DELETE path=/api/closures14922026/11/12 00:02:12 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=014932026/11/12 00:02:12 INFO Garbage collection started14942026/11/12 00:02:12 INFO Vacuumed table table=pending_closures14952026/11/12 00:02:12 INFO Vacuumed table table=pending_objects14962026/11/12 00:02:12 INFO Vacuumed table table=multipart_uploads14972026/11/12 00:02:12 INFO Aborted multipart uploads count=0 kept=01498--- PASS: TestReadProxyOutlastsServerWriteTimeout (5.90s)1499=== CONT TestGCTaskStore_Fail1500--- PASS: TestGCTaskStore_Fail (0.00s)1501=== CONT TestReadProxyInvalidPath15022026/11/12 00:02:12 INFO Vacuumed table table=closures15032026/11/12 00:02:12 INFO Vacuumed table table=objects1504--- PASS: TestPushDedupSurvivesConcurrentGC (4.67s)1505=== CONT TestClientErrorHandling1506=== RUN TestClientErrorHandling/InvalidStorePath1507=== PAUSE TestClientErrorHandling/InvalidStorePath1508=== RUN TestClientErrorHandling/InvalidAuthToken1509=== PAUSE TestClientErrorHandling/InvalidAuthToken1510=== RUN TestClientErrorHandling/ServerNotAvailable1511=== PAUSE TestClientErrorHandling/ServerNotAvailable1512=== CONT TestCacheStatsHandler15132026/11/12 00:02:12 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=015142026/11/12 00:02:12 INFO Vacuumed table table=pending_closures15152026/11/12 00:02:12 INFO Vacuumed table table=pending_objects15162026/11/12 00:02:12 INFO Vacuumed table table=multipart_uploads15172026/11/12 00:02:12 INFO Vacuumed table table=closures15182026/11/12 00:02:12 INFO Vacuumed table table=objects1519--- PASS: TestSweepSkipsRowsLockedByPush (5.49s)1520=== CONT TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed15212026/11/12 00:02:12 WARN Authentication failed token_preview=wrongtoken token_length=10 reason="static API token mismatch"1522--- PASS: TestService_AuthMiddleware (1.41s)1523=== CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info15242026/11/12 00:02:12 INFO Received uploads request method=POST path=/1525=== CONT TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key15262026/11/12 00:02:12 INFO Received complete multipart upload request method=POST path=/1527=== CONT TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key15282026/11/12 00:02:12 INFO Received request for more parts method=POST path=/1529=== CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal15302026/11/12 00:02:12 INFO Received uploads request method=POST path=/1531--- PASS: TestUploadHandlersRejectInvalidKeys (0.00s)1532 --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info (0.00s)1533 --- PASS: TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key (0.00s)1534 --- PASS: TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key (0.00s)1535 --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal (0.00s)1536=== CONT TestProxyWriteTimeout/narinfo1537=== CONT TestProxyWriteTimeout/unknown_size1538=== CONT TestProxyWriteTimeout/10_GiB_nar1539=== CONT TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed1540--- PASS: TestService_AuthMiddleware_MTLSProxyHeader (3.05s)1541=== CONT TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed1542--- PASS: TestReadProxyNarStreaming (3.40s)1543=== CONT TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost1544--- PASS: TestService_ReadScope_PublicByDefault (3.43s)1545=== CONT TestProxyWriteTimeout/1_GiB_nar1546--- PASS: TestProxyWriteTimeout (1.71s)1547 --- PASS: TestProxyWriteTimeout/narinfo (0.00s)1548 --- PASS: TestProxyWriteTimeout/unknown_size (0.00s)1549 --- PASS: TestProxyWriteTimeout/10_GiB_nar (0.00s)1550 --- PASS: TestProxyWriteTimeout/1_GiB_nar (0.00s)1551=== CONT TestGCEndsOnShutdown/before_the_run1552=== RUN TestService_RequireScope_OIDC/builder_may_write1553=== PAUSE TestService_RequireScope_OIDC/builder_may_write1554=== RUN TestService_RequireScope_OIDC/builder_may_not_admin1555=== PAUSE TestService_RequireScope_OIDC/builder_may_not_admin1556=== RUN TestService_RequireScope_OIDC/ops_may_admin1557=== PAUSE TestService_RequireScope_OIDC/ops_may_admin1558=== RUN TestService_RequireScope_OIDC/ops_may_not_write1559=== PAUSE TestService_RequireScope_OIDC/ops_may_not_write1560=== RUN TestService_RequireScope_OIDC/reader_may_not_write1561=== PAUSE TestService_RequireScope_OIDC/reader_may_not_write1562=== RUN TestService_RequireScope_OIDC/static_token_may_admin1563=== PAUSE TestService_RequireScope_OIDC/static_token_may_admin1564=== RUN TestService_RequireScope_OIDC/static_token_may_write1565=== PAUSE TestService_RequireScope_OIDC/static_token_may_write1566=== RUN TestService_RequireScope_OIDC/reader_may_read1567=== PAUSE TestService_RequireScope_OIDC/reader_may_read1568=== RUN TestService_RequireScope_OIDC/writer_implies_read1569=== PAUSE TestService_RequireScope_OIDC/writer_implies_read1570=== RUN TestService_RequireScope_OIDC/anonymous_may_not_read1571=== PAUSE TestService_RequireScope_OIDC/anonymous_may_not_read1572=== RUN TestService_RequireScope_OIDC/reader_may_read_with_netrc1573=== PAUSE TestService_RequireScope_OIDC/reader_may_read_with_netrc1574=== RUN TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1575=== PAUSE TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1576=== RUN TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read1577=== PAUSE TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read1578=== RUN TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1579=== PAUSE TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1580=== CONT TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed1581=== RUN TestService_AuthMiddleware_OIDC/valid_OIDC_token1582=== PAUSE TestService_AuthMiddleware_OIDC/valid_OIDC_token1583=== RUN TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1584=== PAUSE TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1585=== RUN TestService_AuthMiddleware_OIDC/malformed_token_rejected1586=== PAUSE TestService_AuthMiddleware_OIDC/malformed_token_rejected1587=== RUN TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1588=== PAUSE TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1589=== CONT TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark1590--- PASS: TestService_ReadAuthMiddleware (3.63s)1591=== CONT TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete15922026/11/12 00:02:13 INFO lead: acquired remote=192.0.2.1:1234 generation=115932026/11/12 00:02:13 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted,O=Other"15942026/11/12 00:02:13 WARN mTLS auth: bound subjects configured but subject DN unavailable15952026/11/12 00:02:13 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted"1596--- PASS: TestService_AuthMiddleware_MTLSBoundSubjects (3.81s)1597=== CONT TestPresentNotReportedWhileGCDeletesClosure/fresh1598=== CONT TestPresentNotReportedWhileGCDeletesClosure/stale1599--- PASS: TestConnectWaitsForAPeerMigration (5.90s)16002026/11/12 00:02:13 WARN Force mode enabled - objects will be deleted immediately without grace period1601=== NAME TestClientMultipleUploads1602 client_integration_test.go:406: Created store path 0: /build/TestClientMultipleUploads1724765138/001/store/2rkwcg3aadq1ll55lb44x1xjnbqmjwf7-test-file-0.txt16032026/11/12 00:02:13 INFO Received push request method=POST path=/api/pushes16042026/11/12 00:02:13 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)16052026/11/12 00:02:13 INFO Uploading ahblc6fyfz8w6i36hh8fms4d1i5s83mg-file1.txt (160B)1606--- PASS: TestConcurrentCommitsSharingObjectsDoNotDeadlock (5.44s)1607=== CONT TestIsValidUploadKey/narinfo1608=== CONT TestIsValidUploadKey/unknown_type1609=== CONT TestIsValidUploadKey/empty_key1610=== CONT TestIsValidUploadKey/absolute1611=== CONT TestIsValidUploadKey/traversal_nar1612=== CONT TestIsValidUploadKey/traversal1613=== CONT TestIsValidUploadKey/listing_key,_narinfo_type1614=== CONT TestIsValidUploadKey/nar_key,_narinfo_type1615=== CONT TestIsValidUploadKey/narinfo_key,_nar_type1616=== CONT TestIsValidUploadKey/index.html1617=== CONT TestIsValidUploadKey/nix-cache-info1618=== CONT TestIsValidUploadKey/realisation_plus_in_output1619=== CONT TestIsValidUploadKey/realisation1620=== CONT TestIsValidUploadKey/build_log_equals1621=== CONT TestIsValidUploadKey/build_log_question_mark1622=== CONT TestIsValidUploadKey/build_log_plus_in_name1623=== CONT TestIsValidUploadKey/build_log_home-manager_file1624=== CONT TestIsValidUploadKey/build_log1625=== CONT TestIsValidUploadKey/listing1626=== CONT TestIsValidUploadKey/nar_plain1627=== CONT TestIsValidUploadKey/nar_xz1628=== CONT TestIsValidUploadKey/nar_zst1629=== CONT TestUploadHandlersRejectOversizedBody/create_pending_closure1630--- PASS: TestIsValidUploadKey (0.01s)1631 --- PASS: TestIsValidUploadKey/narinfo (0.00s)1632 --- PASS: TestIsValidUploadKey/unknown_type (0.00s)1633 --- PASS: TestIsValidUploadKey/empty_key (0.00s)1634 --- PASS: TestIsValidUploadKey/absolute (0.00s)1635 --- PASS: TestIsValidUploadKey/traversal_nar (0.00s)1636 --- PASS: TestIsValidUploadKey/traversal (0.00s)1637 --- PASS: TestIsValidUploadKey/listing_key,_narinfo_type (0.00s)1638 --- PASS: TestIsValidUploadKey/nar_key,_narinfo_type (0.00s)1639 --- PASS: TestIsValidUploadKey/narinfo_key,_nar_type (0.00s)1640 --- PASS: TestIsValidUploadKey/index.html (0.00s)1641 --- PASS: TestIsValidUploadKey/nix-cache-info (0.00s)1642 --- PASS: TestIsValidUploadKey/realisation_plus_in_output (0.00s)1643 --- PASS: TestIsValidUploadKey/realisation (0.00s)1644 --- PASS: TestIsValidUploadKey/build_log_equals (0.00s)1645 --- PASS: TestIsValidUploadKey/build_log_question_mark (0.00s)1646 --- PASS: TestIsValidUploadKey/build_log_plus_in_name (0.00s)1647 --- PASS: TestIsValidUploadKey/build_log_home-manager_file (0.00s)1648 --- PASS: TestIsValidUploadKey/build_log (0.00s)1649 --- PASS: TestIsValidUploadKey/listing (0.00s)1650 --- PASS: TestIsValidUploadKey/nar_plain (0.00s)1651 --- PASS: TestIsValidUploadKey/nar_xz (0.00s)1652 --- PASS: TestIsValidUploadKey/nar_zst (0.00s)16532026/11/12 00:02:13 INFO Received uploads request method=POST path=/16542026/11/12 00:02:13 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign16552026/11/12 00:02:13 WARN Failed to register uploaded object key=nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst error="server returned 404: 404 page not found\n"16562026/11/12 00:02:13 WARN Failed to register uploaded object key=ahblc6fyfz8w6i36hh8fms4d1i5s83mg.ls error="server returned 404: 404 page not found\n"16572026/11/12 00:02:13 INFO Signed narinfos id=1 count=116582026/11/12 00:02:13 INFO Uploading 1 narinfos16592026/11/12 00:02:13 WARN Failed to register uploaded object key=ahblc6fyfz8w6i36hh8fms4d1i5s83mg.narinfo error="server returned 404: 404 page not found\n"16602026/11/12 00:02:13 INFO Received complete push request method=POST path=/api/pushes/1/complete16612026/11/12 00:02:13 INFO Received uploads request method=POST path=/api/pending_closures16622026/11/12 00:02:13 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=016632026/11/12 00:02:13 INFO Vacuumed table table=pending_closures16642026/11/12 00:02:13 INFO Completed push id=1 objects=3 duration=12.191056ms16652026/11/12 00:02:13 INFO Upload complete. (1.404s)16662026/11/12 00:02:13 INFO Vacuumed table table=pending_objects16672026/11/12 00:02:13 INFO Vacuumed table table=multipart_uploads1668=== NAME TestNARDeduplicationMetadataUploadBug1669 metadata_upload_test.go:54: Retrieved narinfo from S3:1670 StorePath: /build/TestNARDeduplicationMetadataUploadBug4247417565/001/store/ahblc6fyfz8w6i36hh8fms4d1i5s83mg-file1.txt1671 URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst1672 Compression: zstd1673 NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1674 NarSize: 1601675 References: 1676 CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf16772026/11/12 00:02:13 INFO Vacuumed table table=closures1678 metadata_upload_test.go:55: Retrieved .ls file from S3 (compressed size: 77 bytes)1679 metadata_upload_test.go:55: Decompressed .ls content (64 bytes):1680 {"version":1,"root":{"type":"regular","size":44,"narOffset":96}}16812026/11/12 00:02:13 INFO Vacuumed table table=objects16822026/11/12 00:02:13 INFO Received uploads request method=POST path=/api/pending_closures16832026/11/12 00:02:13 INFO Received uploads request method=POST path=/api/pending_closures16842026/11/12 00:02:13 INFO Received uploads request method=POST path=/api/pending_closures16852026/11/12 00:02:13 INFO Received uploads request method=POST path=/api/pending_closures16862026/11/12 00:02:14 INFO Aborted multipart uploads count=0 kept=016872026/11/12 00:02:14 INFO Aborted multipart uploads count=0 kept=01688=== CONT TestUploadHandlersRejectOversizedBody/request_more_parts16892026/11/12 00:02:14 INFO Received request for more parts method=POST path=/16902026/11/12 00:02:14 INFO lead: released remote=192.0.2.1:1234 generation=116912026/11/12 00:02:14 INFO lead: acquired remote=192.0.2.1:1234 generation=216922026/11/12 00:02:14 INFO Received complete multipart upload request method=POST path=/api/multipart/complete16932026/11/12 00:02:14 INFO Received complete multipart upload request method=POST path=/api/multipart/complete16942026/11/12 00:02:14 INFO Received complete multipart upload request method=POST path=/api/multipart/complete16952026/11/12 00:02:14 INFO Received complete multipart upload request method=POST path=/api/multipart/complete16962026/11/12 00:02:14 INFO Received complete multipart upload request method=POST path=/api/multipart/complete16972026/11/12 00:02:14 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000300000000000000000000.nar.zst upload_id=NTQwMDM1NWYtMTEzNC00NTU1LWJlZGQtNzAyNDcwZjVhNWQ3LjE5NGJhOGVkLTZmMjAtNDFlMy1iOTRhLWYyZGM5YzZkYTU4M3gxNzk0NDQxNzMzODYzNTM0NjUz parts=1016982026/11/12 00:02:14 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000400000000000000000000.nar.zst upload_id=NTQwMDM1NWYtMTEzNC00NTU1LWJlZGQtNzAyNDcwZjVhNWQ3LmQ2NTViZjdmLTA4OWEtNDc1Ny05M2I4LTdlNmJhYzUxNTA5OHgxNzk0NDQxNzMzOTUzMzk0NDkx parts=1016992026/11/12 00:02:14 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete17002026/11/12 00:02:14 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete17012026/11/12 00:02:14 INFO Completed upload id=1 objects=3 duration=3.125409ms17022026/11/12 00:02:14 INFO Received uploads request method=POST path=/api/pending_closures17032026/11/12 00:02:14 INFO Completed upload id=1 objects=3 duration=3.447033ms17042026/11/12 00:02:14 INFO Received uploads request method=POST path=/api/pending_closures1705=== CONT TestUploadHandlersRejectOversizedBody/complete_multipart17062026/11/12 00:02:14 INFO Received complete multipart upload request method=POST path=/1707=== CONT TestPush_RejectsBadRequests/no_roots17082026/11/12 00:02:14 INFO Received push request method=POST path=/api/pushes1709=== CONT TestPush_RejectsBadRequests/bad_root17102026/11/12 00:02:14 INFO Received push request method=POST path=/api/pushes1711=== CONT TestPush_RejectsBadRequests/no_objects17122026/11/12 00:02:14 INFO Received push request method=POST path=/api/pushes1713=== CONT TestPush_RejectsBadRequests/root_not_in_objects17142026/11/12 00:02:14 INFO Received push request method=POST path=/api/pushes1715=== CONT TestServerTLSConfig/no_client_CA1716=== CONT TestServerTLSConfig/not_a_PEM_file1717--- PASS: TestPush_RejectsBadRequests (1.68s)1718 --- PASS: TestPush_RejectsBadRequests/no_roots (0.00s)1719 --- PASS: TestPush_RejectsBadRequests/bad_root (0.01s)1720 --- PASS: TestPush_RejectsBadRequests/no_objects (0.00s)1721 --- PASS: TestPush_RejectsBadRequests/root_not_in_objects (0.00s)1722=== CONT TestServerTLSConfig/missing_CA_file1723=== CONT TestIsValidCachePath/narinfo1724=== CONT TestIsValidCachePath/short_hash1725=== CONT TestIsValidCachePath/wrong_extension1726=== CONT TestIsValidCachePath/leading_slash1727=== CONT TestIsValidCachePath/empty1728--- PASS: TestServerTLSConfig (0.00s)1729 --- PASS: TestServerTLSConfig/no_client_CA (0.00s)1730 --- PASS: TestServerTLSConfig/not_a_PEM_file (0.01s)1731 --- PASS: TestServerTLSConfig/missing_CA_file (0.00s)1732=== CONT TestIsValidCachePath/log1733=== CONT TestIsValidCachePath/ls1734=== CONT TestIsValidCachePath/nar_uncompressed1735=== CONT TestIsValidCachePath/nar_bz21736=== CONT TestIsValidCachePath/nar_xz1737=== CONT TestIsValidCachePath/nar_zst1738=== CONT TestIsValidCachePath/narinfo_all_nix_base32_chars1739=== CONT TestIsValidCachePath/traversal_in_middle1740=== CONT TestIsValidCachePath/random_path1741=== CONT TestIsValidCachePath/invalid_char_u1742=== CONT TestIsValidCachePath/invalid_char_e1743=== CONT TestIsValidCachePath/index.html1744=== CONT TestIsValidCachePath/traversal_parent1745=== CONT TestIsValidCachePath/nix-cache-info1746=== CONT TestIsValidCachePath/realisation1747--- PASS: TestIsValidCachePath (0.00s)1748 --- PASS: TestIsValidCachePath/narinfo (0.00s)1749 --- PASS: TestIsValidCachePath/short_hash (0.00s)1750 --- PASS: TestIsValidCachePath/wrong_extension (0.00s)1751 --- PASS: TestIsValidCachePath/leading_slash (0.00s)1752 --- PASS: TestIsValidCachePath/empty (0.00s)1753 --- PASS: TestIsValidCachePath/log (0.00s)1754 --- PASS: TestIsValidCachePath/ls (0.00s)1755 --- PASS: TestIsValidCachePath/nar_uncompressed (0.00s)1756 --- PASS: TestIsValidCachePath/nar_bz2 (0.00s)1757 --- PASS: TestIsValidCachePath/nar_xz (0.00s)1758 --- PASS: TestIsValidCachePath/nar_zst (0.00s)1759 --- PASS: TestIsValidCachePath/narinfo_all_nix_base32_chars (0.00s)1760 --- PASS: TestIsValidCachePath/traversal_in_middle (0.00s)1761 --- PASS: TestIsValidCachePath/random_path (0.00s)1762 --- PASS: TestIsValidCachePath/invalid_char_u (0.00s)1763 --- PASS: TestIsValidCachePath/invalid_char_e (0.00s)1764 --- PASS: TestIsValidCachePath/index.html (0.00s)1765 --- PASS: TestIsValidCachePath/traversal_parent (0.00s)1766 --- PASS: TestIsValidCachePath/nix-cache-info (0.00s)1767 --- PASS: TestIsValidCachePath/realisation (0.00s)1768=== CONT TestParseSingleRange/none1769=== CONT TestParseSingleRange/start_past_EOF1770=== CONT TestParseSingleRange/single_byte1771=== CONT TestParseSingleRange/suffix_exceeds_size1772=== CONT TestParseSingleRange/suffix1773=== CONT TestParseSingleRange/end_clamped_to_size1774=== CONT TestParseSingleRange/open-ended1775=== CONT TestParseSingleRange/closed1776=== CONT TestParseSingleRange/malformed_end_before_start1777=== CONT TestParseSingleRange/malformed_both_empty1778=== CONT TestParseSingleRange/start_far_past_EOF1779=== CONT TestParseSingleRange/malformed_no_dash1780=== CONT TestParseSingleRange/unknown_unit1781=== CONT TestParseSingleRange/multi-range_ignored1782--- PASS: TestParseSingleRange (0.00s)1783 --- PASS: TestParseSingleRange/none (0.00s)1784 --- PASS: TestParseSingleRange/start_past_EOF (0.00s)1785 --- PASS: TestParseSingleRange/single_byte (0.00s)1786 --- PASS: TestParseSingleRange/suffix_exceeds_size (0.00s)1787 --- PASS: TestParseSingleRange/suffix (0.00s)1788 --- PASS: TestParseSingleRange/end_clamped_to_size (0.00s)1789 --- PASS: TestParseSingleRange/open-ended (0.00s)1790 --- PASS: TestParseSingleRange/closed (0.00s)1791 --- PASS: TestParseSingleRange/malformed_end_before_start (0.00s)1792 --- PASS: TestParseSingleRange/malformed_both_empty (0.00s)1793 --- PASS: TestParseSingleRange/start_far_past_EOF (0.00s)1794 --- PASS: TestParseSingleRange/malformed_no_dash (0.00s)1795 --- PASS: TestParseSingleRange/unknown_unit (0.00s)1796 --- PASS: TestParseSingleRange/multi-range_ignored (0.00s)1797=== CONT TestPendingClosureWriteTimeout/empty1798=== CONT TestPendingClosureWriteTimeout/670k_objects1799=== CONT TestPendingClosureWriteTimeout/400_objects1800=== CONT TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline18012026/11/12 00:02:15 INFO lead: released remote=192.0.2.1:1234 generation=21802--- PASS: TestLeadElectsOneAndHandsOver (7.16s)1803=== CONT TestForceGCDuringPushOffersSweptObject/before_pending_rows1804=== CONT TestForceGCDuringPushOffersSweptObject/after_presence_check1805=== NAME TestNARDeduplicationMetadataUploadBug1806 metadata_upload_test.go:64: Second store path (same content): /build/TestNARDeduplicationMetadataUploadBug4247417565/001/store/w37f6nxls1gqpczb5hcng9mh55yzmqcv-file2.txt18072026/11/12 00:02:18 WARN Force mode enabled - objects will be deleted immediately without grace period1808=== NAME TestClientMultipleUploads1809 client_integration_test.go:406: Created store path 1: /build/TestClientMultipleUploads1724765138/001/store/asqichd98f3gbpj5k1zghmgx1fvx4qyg-test-file-1.txt1810=== NAME TestClientIntegration1811 client_integration_test.go:292: Created store path: /build/TestClientIntegration2187357540/002/store/lmx6y3g062n42d3avqdz8iwj1jhb2cbs-test-file.txt18122026/11/12 00:02:18 INFO Garbage collection progress phase="" failed_uploads_deleted=0 old_closures_deleted=1 objects_marked=3 objects_deleted=3 objects_failed=018132026/11/12 00:02:18 WARN Force mode enabled - objects will be deleted immediately without grace period18142026/11/12 00:02:18 INFO Received create pin request method=POST path=/api/pins/myapp18152026/11/12 00:02:18 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000100000000000000000000.nar.zst upload_id=NTQwMDM1NWYtMTEzNC00NTU1LWJlZGQtNzAyNDcwZjVhNWQ3LmJlYTg1M2FiLWIxYjAtNDQ3NS04ZDlhLWMwODcxZTE1ZDcxZXgxNzk0NDQxNzMzODg5ODAwMDg0 parts=1018162026-11-12 00:02:18.781 UTC [1190] ERROR: canceling statement due to user request18172026-11-12 00:02:18.781 UTC [1190] STATEMENT: -- name: MarkStaleObjects :execrows1818 WITH RECURSIVE ct AS (1819 SELECT timezone('UTC', now()) AS now1820 ),1821 closure_reach AS (1822 -- Start with all closure keys1823 SELECT o.key, o.refs1824 FROM objects o1825 INNER JOIN closures c ON o.key = c.key1826 UNION1827 -- Recursively add all referenced objects1828 SELECT o.key, o.refs1829 FROM objects o1830 INNER JOIN closure_reach cr ON o.key = ANY(cr.refs)1831 ),1832 reachable_objects AS (1833 SELECT DISTINCT key FROM closure_reach1834 ),1835 stale_objects AS (1836 SELECT o.key1837 FROM objects AS o, ct1838 WHERE1839 NOT EXISTS (1840 SELECT 11841 FROM reachable_objects ro1842 WHERE ro.key = o.key1843 )1844 AND NOT EXISTS (1845 SELECT 11846 FROM pending_objects AS po1847 WHERE po.key = o.key1848 )1849 AND o.deleted_at IS NULL -- Only mark fresh objects1850 ORDER BY o.key COLLATE "C" -- lock in key order, like commit_pending_closure1851 FOR UPDATE1852 )1853 UPDATE objects1854 SET1855 deleted_at = ct.now,1856 first_deleted_at = COALESCE(first_deleted_at, ct.now)1857 FROM stale_objects, ct1858 WHERE objects.key = stale_objects.key1859 18602026/11/12 00:02:18 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000200000000000000000000.nar.zst upload_id=NTQwMDM1NWYtMTEzNC00NTU1LWJlZGQtNzAyNDcwZjVhNWQ3LjRhMTU4MmNiLTU3YWYtNGVhNy1iOTViLWNlZjA2OTRlYTEyN3gxNzk0NDQxNzMzODM4MDA2Nzkw parts=1018612026/11/12 00:02:18 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000500000000000000000000.nar.zst upload_id=NTQwMDM1NWYtMTEzNC00NTU1LWJlZGQtNzAyNDcwZjVhNWQ3LjRmM2I5MTM1LTE3NTItNDk1ZS1hYWM5LTRmNjk4YzllODdmMHgxNzk0NDQxNzMzOTE0MDg5Nzk1 parts=1018622026/11/12 00:02:18 ERROR failed to remove object object=nar/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.nar.zst error="Post \"http://localhost:34089/bucket95/?delete=\": context canceled"1863=== CONT TestResolveDBConnectionString/flag_wins1864=== CONT TestResolveDBConnectionString/PGHOST_allows_empty1865=== CONT TestResolveDBConnectionString/missing_file_is_an_error1866--- PASS: TestGCEndsOnShutdown (0.00s)1867 --- PASS: TestGCEndsOnShutdown/before_the_run (1.24s)1868 --- PASS: TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete (5.87s)1869 --- PASS: TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark (5.90s)1870=== CONT TestResolveDBConnectionString/nothing_configured1871=== CONT TestCacheConfigHandler/full_config,_no_issuer1872=== CONT TestResolveDBConnectionString/file_when_flag_empty1873=== CONT TestCacheConfigHandler/no_signing_keys1874=== CONT TestCacheConfigHandler/no_cache_url_configured1875=== CONT TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1876=== CONT TestClientErrorHandling/InvalidStorePath1877--- PASS: TestCacheConfigHandler (0.00s)1878 --- PASS: TestCacheConfigHandler/full_config,_no_issuer (0.00s)1879 --- PASS: TestCacheConfigHandler/no_signing_keys (0.00s)1880 --- PASS: TestCacheConfigHandler/no_cache_url_configured (0.00s)1881 --- PASS: TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator (0.00s)1882=== CONT TestClientErrorHandling/InvalidAuthToken1883--- PASS: TestResolveDBConnectionString (0.00s)1884 --- PASS: TestResolveDBConnectionString/flag_wins (0.00s)1885 --- PASS: TestResolveDBConnectionString/PGHOST_allows_empty (0.00s)1886 --- PASS: TestResolveDBConnectionString/missing_file_is_an_error (0.00s)1887 --- PASS: TestResolveDBConnectionString/nothing_configured (0.00s)1888 --- PASS: TestResolveDBConnectionString/file_when_flag_empty (0.00s)18892026/11/12 00:02:18 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete18902026/11/12 00:02:18 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete18912026/11/12 00:02:18 INFO Created/updated pin name=myapp store_path=/build/TestPinProtectsFromGC1660759056/001/store/qa3g9999gl3pnc87bs5ybf1jfb9rb67z-pinned-file.txt narinfo_key=qa3g9999gl3pnc87bs5ybf1jfb9rb67z.narinfo18922026/11/12 00:02:18 INFO Completed upload id=1 objects=3 duration=5.140529ms18932026/11/12 00:02:18 INFO Completed upload id=1 objects=3 duration=6.036818ms18942026/11/12 00:02:18 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete18952026/11/12 00:02:18 INFO Starting cleanup of old closures method=DELETE path=/api/closures18962026/11/12 00:02:18 INFO Garbage collection started18972026/11/12 00:02:18 INFO Completed upload id=1 objects=3 duration=4.795666ms18982026/11/12 00:02:18 INFO Received uploads request method=POST path=/api/pending_closures18992026/11/12 00:02:18 INFO Aborted multipart uploads count=0 kept=01900=== CONT TestClientErrorHandling/ServerNotAvailable1901--- PASS: TestReadProxyInvalidPath (6.38s)1902=== CONT TestService_RequireScope_OIDC/builder_may_write1903=== CONT TestService_RequireScope_OIDC/reader_may_read1904=== CONT TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1905=== CONT TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read19062026/11/12 00:02:18 WARN Authentication failed token_preview=wrong token_length=5 oidc_error="no provider could verify the token (signature or issuer mismatch)" oidc_provider="" tried_providers=[test]1907=== CONT TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1908=== CONT TestService_RequireScope_OIDC/reader_may_read_with_netrc1909=== CONT TestService_RequireScope_OIDC/anonymous_may_not_read1910=== CONT TestService_RequireScope_OIDC/reader_may_not_write1911=== CONT TestService_RequireScope_OIDC/static_token_may_write1912=== CONT TestService_RequireScope_OIDC/static_token_may_admin1913=== CONT TestService_RequireScope_OIDC/ops_may_admin1914=== CONT TestService_RequireScope_OIDC/ops_may_not_write1915=== CONT TestService_RequireScope_OIDC/writer_implies_read1916=== CONT TestService_RequireScope_OIDC/builder_may_not_admin1917=== CONT TestService_AuthMiddleware_OIDC/valid_OIDC_token1918--- PASS: TestService_RequireScope_OIDC (3.55s)1919 --- PASS: TestService_RequireScope_OIDC/builder_may_write (0.00s)1920 --- PASS: TestService_RequireScope_OIDC/reader_may_read (0.00s)1921 --- PASS: TestService_RequireScope_OIDC/empty_netrc_password_may_not_read (0.00s)1922 --- PASS: TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read (0.00s)1923 --- PASS: TestService_RequireScope_OIDC/static_token_may_admin_with_netrc (0.00s)1924 --- PASS: TestService_RequireScope_OIDC/reader_may_read_with_netrc (0.00s)1925 --- PASS: TestService_RequireScope_OIDC/anonymous_may_not_read (0.00s)1926 --- PASS: TestService_RequireScope_OIDC/reader_may_not_write (0.00s)1927 --- PASS: TestService_RequireScope_OIDC/static_token_may_write (0.00s)1928 --- PASS: TestService_RequireScope_OIDC/static_token_may_admin (0.00s)1929 --- PASS: TestService_RequireScope_OIDC/ops_may_admin (0.00s)1930 --- PASS: TestService_RequireScope_OIDC/ops_may_not_write (0.00s)1931 --- PASS: TestService_RequireScope_OIDC/writer_implies_read (0.00s)1932 --- PASS: TestService_RequireScope_OIDC/builder_may_not_admin (0.00s)1933=== CONT TestService_AuthMiddleware_OIDC/malformed_token_rejected19342026/11/12 00:02:18 WARN Authentication failed token_preview=not-a-valid-jwt token_length=15 oidc_error="no provider could verify the token (signature or issuer mismatch)" oidc_provider="" tried_providers=[test]1935=== CONT TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected19362026/11/12 00:02:18 WARN Authentication failed token_preview=eyJhbGciOi...tQX7Je5iug token_length=701 oidc_error="no rule matched: claim \"repository_owner\" value [otherorg] not in allowed patterns [myorg]" oidc_provider=test tried_providers=[test]1937=== CONT TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1938--- PASS: TestService_AuthMiddleware_OIDC (3.65s)1939 --- PASS: TestService_AuthMiddleware_OIDC/valid_OIDC_token (0.00s)1940 --- PASS: TestService_AuthMiddleware_OIDC/malformed_token_rejected (0.00s)1941 --- PASS: TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected (0.00s)1942 --- PASS: TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured (0.00s)19432026/11/12 00:02:18 INFO Received uploads request method=POST path=/api/pending_closures19442026/11/12 00:02:18 INFO Received uploads request method=POST path=/api/pending_closures19452026/11/12 00:02:18 INFO Object in database but missing from S3 key=nar/dadb44fdadb44fdadb44fdadb44f000500000000000000000000.nar.zst19462026/11/12 00:02:18 WARN Found objects in DB but missing from S3, will re-upload count=119472026/11/12 00:02:18 INFO Object in database but missing from S3 key=nar/dadb44fdadb44fdadb44fdadb44f000100000000000000000000.nar.zst19482026/11/12 00:02:18 WARN Narinfo references a NAR missing from S3, will re-upload narinfo=dadb44fdadb44fdadb44fdadb44f0001.narinfo1949--- PASS: TestCacheStatsHandler (6.45s)19502026/11/12 00:02:18 INFO Received uploads request method=POST path=/api/pending_closures19512026/11/12 00:02:18 INFO Received uploads request method=POST path=/api/pending_closures19522026/11/12 00:02:18 INFO Aborted multipart uploads count=0 kept=019532026/11/12 00:02:18 INFO Received uploads request method=POST path=/api/pending_closures19542026/11/12 00:02:19 INFO Received uploads request method=POST path=/api/pending_closures19552026/11/12 00:02:19 INFO Aborted multipart uploads count=0 kept=019562026/11/12 00:02:19 INFO Received complete multipart upload request method=POST path=/api/multipart/complete19572026/11/12 00:02:19 INFO Received complete multipart upload request method=POST path=/api/multipart/complete19582026/11/12 00:02:19 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f100300000000000000000000.nar.zst upload_id=NTQwMDM1NWYtMTEzNC00NTU1LWJlZGQtNzAyNDcwZjVhNWQ3LmU2ZGEwZmE2LTgyNGItNDM5MS1hN2VhLWI0OTMyOTI4ZTE5ZXgxNzk0NDQxNzM4Nzk2OTQyMDgx parts=1019592026/11/12 00:02:19 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete19602026/11/12 00:02:19 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f100400000000000000000000.nar.zst upload_id=NTQwMDM1NWYtMTEzNC00NTU1LWJlZGQtNzAyNDcwZjVhNWQ3LmU2ODRhMTllLWUyZGItNDYwMC05NDk1LTllMjExZjk5MDNjNXgxNzk0NDQxNzM4Nzk2OTM0MjIx parts=1019612026/11/12 00:02:19 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete19622026/11/12 00:02:19 INFO Completed upload id=2 objects=3 duration=1.493355ms19632026/11/12 00:02:19 INFO Completed upload id=2 objects=3 duration=1.962898ms19642026/11/12 00:02:19 INFO Received uploads request method=POST path=/api/pending_closures19652026/11/12 00:02:19 INFO Received uploads request method=POST path=/api/pending_closures19662026/11/12 00:02:19 INFO Object in database but missing from S3 key=nar/dadb44fdadb44fdadb44fdadb44f000300000000000000000000.nar.zst19672026/11/12 00:02:19 WARN Narinfo references a NAR missing from S3, will re-upload narinfo=dadb44fdadb44fdadb44fdadb44f0003.narinfo19682026/11/12 00:02:19 INFO Object in database but missing from S3 key=nar/dadb44fdadb44fdadb44fdadb44f000400000000000000000000.nar.zst19692026/11/12 00:02:19 WARN Narinfo references a NAR missing from S3, will re-upload narinfo=dadb44fdadb44fdadb44fdadb44f0004.narinfo1970--- PASS: TestService_verifyS3ReuploadsStaleNarinfo (0.00s)1971 --- PASS: TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed (5.96s)1972 --- PASS: TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed (6.28s)1973 --- PASS: TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed (6.32s)1974 --- PASS: TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed (6.71s)1975 --- PASS: TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost (6.41s)1976--- PASS: TestUploadHandlersRejectOversizedBody (0.17s)1977 --- PASS: TestUploadHandlersRejectOversizedBody/complete_multipart (0.07s)1978 --- PASS: TestUploadHandlersRejectOversizedBody/request_more_parts (1.62s)1979 --- PASS: TestUploadHandlersRejectOversizedBody/create_pending_closure (5.57s)1980--- PASS: TestPushWaitsForSweepHoldingRow (7.01s)1981--- PASS: TestPresentNotReportedWhileGCDeletesClosure (0.00s)1982 --- PASS: TestPresentNotReportedWhileGCDeletesClosure/fresh (1.61s)1983 --- PASS: TestPresentNotReportedWhileGCDeletesClosure/stale (5.83s)1984--- PASS: TestPendingClosureWriteTimeout (0.00s)1985 --- PASS: TestPendingClosureWriteTimeout/empty (0.00s)1986 --- PASS: TestPendingClosureWriteTimeout/670k_objects (0.00s)1987 --- PASS: TestPendingClosureWriteTimeout/400_objects (0.00s)1988 --- PASS: TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline (5.05s)19892026/11/12 00:02:20 INFO Garbage collection progress phase=cleanup_pending_uploads failed_uploads_deleted=0 old_closures_deleted=0 objects_marked=0 objects_deleted=0 objects_failed=019902026/11/12 00:02:21 WARN Force mode enabled - objects will be deleted immediately without grace period19912026/11/12 00:02:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=019922026/11/12 00:02:21 INFO Vacuumed table table=pending_closures19932026/11/12 00:02:21 INFO Vacuumed table table=pending_objects19942026/11/12 00:02:21 INFO Vacuumed table table=multipart_uploads19952026/11/12 00:02:21 INFO Vacuumed table table=closures19962026/11/12 00:02:21 INFO Vacuumed table table=objects19972026/11/12 00:02:24 WARN Force mode enabled - objects will be deleted immediately without grace period19982026/11/12 00:02:24 WARN Force mode enabled - objects will be deleted immediately without grace period19992026/11/12 00:02:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=020002026/11/12 00:02:24 INFO Vacuumed table table=pending_closures
loading 686 hidden lines…
26872026/11/12 00:02:49 INFO Vacuumed table table=multipart_uploads26882026/11/12 00:02:49 INFO Vacuumed table table=closures26892026/11/12 00:02:49 INFO Vacuumed table table=objects26902026/11/12 00:02:49 INFO Received uploads request method=POST path=/26912026/11/12 00:02:49 INFO Received uploads request method=POST path=/26922026/11/12 00:02:49 INFO Received push request method=POST path=/26932026/11/12 00:02:49 INFO Received complete upload request method=POST path=/26942026/11/12 00:02:49 INFO Completed upload id=15 objects=6 duration=1.582375ms26952026/11/12 00:02:49 INFO Received complete push request method=POST path=/26962026/11/12 00:02:49 INFO Completed push id=9 objects=5 duration=2.08096ms26972026/11/12 00:02:49 INFO Received complete upload request method=POST path=/26982026/11/12 00:02:49 INFO Aborted multipart uploads count=0 kept=026992026/11/12 00:02:49 INFO Completed upload id=3 objects=25 duration=3.16789ms27002026/11/12 00:02:49 INFO Aborted multipart uploads count=0 kept=027012026/11/12 00:02:49 WARN Force mode enabled - objects will be deleted immediately without grace period27022026/11/12 00:02:49 WARN Force mode enabled - objects will be deleted immediately without grace period27032026/11/12 00:02:49 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027042026/11/12 00:02:49 INFO Vacuumed table table=pending_closures27052026/11/12 00:02:49 INFO Received push request method=POST path=/27062026/11/12 00:02:49 INFO Vacuumed table table=pending_objects27072026/11/12 00:02:49 INFO Vacuumed table table=multipart_uploads27082026/11/12 00:02:49 INFO Vacuumed table table=closures27092026/11/12 00:02:49 INFO Vacuumed table table=objects27102026/11/12 00:02:49 INFO Aborted multipart uploads count=0 kept=027112026/11/12 00:02:49 INFO Received push request method=POST path=/27122026/11/12 00:02:49 INFO Received complete push request method=POST path=/27132026/11/12 00:02:49 INFO Completed push id=16 objects=27 duration=2.351163ms27142026/11/12 00:02:49 INFO Received uploads request method=POST path=/27152026/11/12 00:02:49 INFO Received push request method=POST path=/27162026/11/12 00:02:49 INFO Received complete push request method=POST path=/27172026/11/12 00:02:49 INFO Completed push id=18 objects=10 duration=2.255582ms27182026/11/12 00:02:49 INFO Aborted multipart uploads count=0 kept=027192026/11/12 00:02:49 WARN Force mode enabled - objects will be deleted immediately without grace period27202026/11/12 00:02:49 INFO Received complete push request method=POST path=/27212026/11/12 00:02:49 INFO Completed push id=15 objects=24 duration=4.783946ms27222026/11/12 00:02:49 INFO Received uploads request method=POST path=/27232026/11/12 00:02:49 INFO Received uploads request method=POST path=/27242026/11/12 00:02:49 INFO Received uploads request method=POST path=/27252026/11/12 00:02:49 INFO Received complete upload request method=POST path=/27262026/11/12 00:02:49 INFO Completed upload id=6 objects=6 duration=1.434134ms27272026/11/12 00:02:49 INFO Aborted multipart uploads count=0 kept=027282026/11/12 00:02:49 WARN Force mode enabled - objects will be deleted immediately without grace period27292026/11/12 00:02:50 INFO Garbage collection completed failed-uploads-deleted=5 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027302026/11/12 00:02:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027312026/11/12 00:02:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=1 objects-deleted-after-grace-period=0 objects-failed-to-delete=027322026/11/12 00:02:50 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027332026/11/12 00:02:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=4 objects-deleted-after-grace-period=0 objects-failed-to-delete=027342026/11/12 00:02:50 INFO Vacuumed table table=pending_closures27352026/11/12 00:02:50 WARN Force mode enabled - objects will be deleted immediately without grace period27362026/11/12 00:02:50 WARN Force mode enabled - objects will be deleted immediately without grace period27372026/11/12 00:02:50 WARN Force mode enabled - objects will be deleted immediately without grace period27382026/11/12 00:02:50 INFO Vacuumed table table=pending_closures27392026/11/12 00:02:50 WARN Force mode enabled - objects will be deleted immediately without grace period27402026/11/12 00:02:50 WARN Force mode enabled - objects will be deleted immediately without grace period27412026/11/12 00:02:50 INFO Vacuumed table table=pending_closures27422026/11/12 00:02:50 INFO Vacuumed table table=pending_closures27432026/11/12 00:02:50 INFO Vacuumed table table=pending_objects27442026/11/12 00:02:50 WARN Force mode enabled - objects will be deleted immediately without grace period27452026/11/12 00:02:50 INFO Vacuumed table table=multipart_uploads27462026/11/12 00:02:50 INFO Vacuumed table table=pending_objects27472026/11/12 00:02:50 WARN Force mode enabled - objects will be deleted immediately without grace period27482026/11/12 00:02:50 INFO Vacuumed table table=multipart_uploads27492026/11/12 00:02:50 INFO Vacuumed table table=pending_closures27502026/11/12 00:02:50 INFO Vacuumed table table=pending_objects27512026/11/12 00:02:50 INFO Vacuumed table table=multipart_uploads27522026/11/12 00:02:50 INFO Vacuumed table table=closures27532026/11/12 00:02:50 INFO Vacuumed table table=pending_objects27542026/11/12 00:02:50 INFO Vacuumed table table=closures27552026/11/12 00:02:50 INFO Vacuumed table table=closures27562026/11/12 00:02:50 INFO Vacuumed table table=multipart_uploads27572026/11/12 00:02:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027582026/11/12 00:02:50 INFO Vacuumed table table=objects27592026/11/12 00:02:50 INFO Vacuumed table table=pending_closures27602026/11/12 00:02:50 INFO Vacuumed table table=objects27612026/11/12 00:02:50 INFO Vacuumed table table=objects27622026/11/12 00:02:50 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027632026/11/12 00:02:50 INFO Vacuumed table table=pending_objects27642026/11/12 00:02:50 INFO Vacuumed table table=pending_objects27652026/11/12 00:02:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027662026/11/12 00:02:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027672026/11/12 00:02:50 INFO Vacuumed table table=closures27682026/11/12 00:02:50 INFO Vacuumed table table=multipart_uploads27692026/11/12 00:02:50 INFO Vacuumed table table=multipart_uploads27702026/11/12 00:02:50 INFO Vacuumed table table=closures27712026/11/12 00:02:50 INFO Vacuumed table table=pending_closures27722026/11/12 00:02:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027732026/11/12 00:02:50 INFO Vacuumed table table=pending_closures27742026/11/12 00:02:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027752026/11/12 00:02:50 INFO Vacuumed table table=objects27762026/11/12 00:02:50 INFO Vacuumed table table=objects27772026/11/12 00:02:50 INFO Vacuumed table table=closures27782026/11/12 00:02:50 INFO Vacuumed table table=pending_objects27792026/11/12 00:02:50 INFO Vacuumed table table=multipart_uploads27802026/11/12 00:02:50 INFO Vacuumed table table=closures27812026/11/12 00:02:50 INFO Vacuumed table table=objects27822026/11/12 00:02:50 INFO Vacuumed table table=pending_closures27832026/11/12 00:02:50 INFO Vacuumed table table=objects27842026/11/12 00:02:50 INFO Vacuumed table table=pending_objects27852026/11/12 00:02:50 INFO Vacuumed table table=pending_closures27862026/11/12 00:02:50 INFO Vacuumed table table=pending_closures27872026/11/12 00:02:50 INFO Vacuumed table table=multipart_uploads27882026/11/12 00:02:50 INFO Vacuumed table table=pending_objects27892026/11/12 00:02:50 INFO Vacuumed table table=multipart_uploads27902026/11/12 00:02:50 INFO Vacuumed table table=closures27912026/11/12 00:02:50 INFO Vacuumed table table=pending_objects27922026/11/12 00:02:50 INFO Vacuumed table table=objects27932026/11/12 00:02:50 INFO Vacuumed table table=pending_objects27942026/11/12 00:02:50 INFO Vacuumed table table=closures27952026/11/12 00:02:50 INFO Vacuumed table table=multipart_uploads27962026/11/12 00:02:50 INFO Vacuumed table table=multipart_uploads27972026/11/12 00:02:50 INFO Vacuumed table table=closures27982026/11/12 00:02:50 INFO Vacuumed table table=objects27992026/11/12 00:02:50 INFO Vacuumed table table=closures28002026/11/12 00:02:50 INFO Vacuumed table table=objects28012026/11/12 00:02:50 INFO Vacuumed table table=objects28022026/11/12 00:02:50 INFO Received uploads request method=POST path=/28032026/11/12 00:02:50 INFO Received push request method=POST path=/28042026/11/12 00:02:50 INFO Received push request method=POST path=/28052026/11/12 00:02:50 INFO Received push request method=POST path=/28062026/11/12 00:02:50 INFO Received push request method=POST path=/28072026/11/12 00:02:50 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=3 objects-failed-to-delete=028082026/11/12 00:02:50 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=5 objects-failed-to-delete=028092026/11/12 00:02:50 INFO Aborted multipart uploads count=0 kept=028102026/11/12 00:02:50 WARN Force mode enabled - objects will be deleted immediately without grace period28112026/11/12 00:02:50 INFO Vacuumed table table=pending_closures28122026/11/12 00:02:50 INFO Vacuumed table table=pending_closures28132026/11/12 00:02:50 INFO Received uploads request method=POST path=/28142026/11/12 00:02:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028152026/11/12 00:02:50 INFO Vacuumed table table=pending_objects28162026/11/12 00:02:50 INFO Vacuumed table table=pending_closures28172026/11/12 00:02:50 INFO Vacuumed table table=multipart_uploads28182026/11/12 00:02:50 INFO Vacuumed table table=closures28192026/11/12 00:02:50 INFO Vacuumed table table=pending_objects28202026/11/12 00:02:50 INFO Vacuumed table table=pending_objects28212026/11/12 00:02:50 INFO Vacuumed table table=multipart_uploads28222026/11/12 00:02:50 INFO Vacuumed table table=multipart_uploads28232026/11/12 00:02:50 INFO Vacuumed table table=closures28242026/11/12 00:02:50 INFO Vacuumed table table=objects28252026/11/12 00:02:50 INFO Vacuumed table table=objects28262026/11/12 00:02:50 INFO Vacuumed table table=closures28272026/11/12 00:02:50 INFO Vacuumed table table=objects28282026/11/12 00:02:50 INFO Received push request method=POST path=/28292026/11/12 00:02:50 INFO Received push request method=POST path=/28302026/11/12 00:02:50 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."28312026/11/12 00:02:50 INFO Received complete push request method=POST path=/28322026/11/12 00:02:50 INFO Received uploads request method=POST path=/28332026/11/12 00:02:50 INFO Aborted multipart uploads count=0 kept=028342026/11/12 00:02:50 INFO Aborted multipart uploads count=0 kept=028352026/11/12 00:02:50 INFO Completed push id=6 objects=5 duration=3.431172ms28362026/11/12 00:02:50 INFO Received uploads request method=POST path=/28372026/11/12 00:02:50 WARN Force mode enabled - objects will be deleted immediately without grace period28382026/11/12 00:02:50 INFO Received push request method=POST path=/28392026/11/12 00:02:50 WARN Force mode enabled - objects will be deleted immediately without grace period28402026/11/12 00:02:50 INFO Received push request method=POST path=/28412026/11/12 00:02:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028422026/11/12 00:02:50 INFO Received uploads request method=POST path=/28432026/11/12 00:02:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028442026/11/12 00:02:50 INFO Vacuumed table table=pending_closures28452026/11/12 00:02:50 INFO Vacuumed table table=pending_closures28462026/11/12 00:02:50 INFO Vacuumed table table=pending_objects28472026/11/12 00:02:50 INFO Vacuumed table table=multipart_uploads28482026/11/12 00:02:50 INFO Vacuumed table table=pending_objects28492026/11/12 00:02:50 INFO Vacuumed table table=closures28502026/11/12 00:02:50 INFO Vacuumed table table=objects28512026/11/12 00:02:50 INFO Vacuumed table table=multipart_uploads28522026/11/12 00:02:50 INFO Vacuumed table table=closures28532026/11/12 00:02:50 INFO Received uploads request method=POST path=/28542026/11/12 00:02:50 INFO Vacuumed table table=objects28552026/11/12 00:02:50 INFO Received push request method=POST path=/28562026/11/12 00:02:50 INFO Aborted multipart uploads count=0 kept=028572026/11/12 00:02:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028582026/11/12 00:02:50 INFO Received push request method=POST path=/28592026/11/12 00:02:50 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."28602026/11/12 00:02:50 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.narinfo error="Access Denied."28612026/11/12 00:02:50 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."28622026/11/12 00:02:50 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."28632026/11/12 00:02:50 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."28642026/11/12 00:02:50 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."28652026/11/12 00:02:50 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."28662026/11/12 00:02:50 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."28672026/11/12 00:02:50 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."28682026/11/12 00:02:50 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."28692026/11/12 00:02:50 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."28702026/11/12 00:02:50 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."28712026/11/12 00:02:50 ERROR failed to remove object object=nar/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.nar.zst error="Access Denied."28722026/11/12 00:02:50 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."28732026/11/12 00:02:50 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."28742026/11/12 00:02:50 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."28752026/11/12 00:02:50 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."28762026/11/12 00:02:50 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."28772026/11/12 00:02:50 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."28782026/11/12 00:02:50 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."28792026/11/12 00:02:50 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."28802026/11/12 00:02:50 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."28812026/11/12 00:02:50 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."28822026/11/12 00:02:50 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."28832026/11/12 00:02:50 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."28842026/11/12 00:02:50 INFO Vacuumed table table=pending_closures28852026/11/12 00:02:50 INFO Vacuumed table table=pending_objects28862026/11/12 00:02:50 INFO Received push request method=POST path=/28872026/11/12 00:02:50 INFO Vacuumed table table=multipart_uploads28882026/11/12 00:02:50 INFO Vacuumed table table=closures28892026/11/12 00:02:50 INFO Vacuumed table table=objects28902026/11/12 00:02:50 INFO Received push request method=POST path=/28912026/11/12 00:02:50 INFO Received push request method=POST path=/28922026/11/12 00:02:50 INFO Received push request method=POST path=/28932026/11/12 00:02:50 INFO Received complete upload request method=POST path=/28942026/11/12 00:02:50 INFO Received complete push request method=POST path=/28952026/11/12 00:02:50 INFO Received uploads request method=POST path=/28962026/11/12 00:02:50 INFO Received uploads request method=POST path=/28972026/11/12 00:02:50 INFO Aborted multipart uploads count=0 kept=028982026/11/12 00:02:50 INFO Completed upload id=6 objects=7 duration=2.324722ms28992026/11/12 00:02:50 WARN Force mode enabled - objects will be deleted immediately without grace period29002026/11/12 00:02:50 INFO Completed push id=1 objects=5 duration=5.975917ms29012026/11/12 00:02:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029022026/11/12 00:02:50 INFO Vacuumed table table=pending_closures29032026/11/12 00:02:50 INFO Received complete upload request method=POST path=/29042026/11/12 00:02:50 INFO Received complete upload request method=POST path=/29052026/11/12 00:02:50 INFO Vacuumed table table=pending_objects29062026/11/12 00:02:50 INFO Vacuumed table table=multipart_uploads29072026/11/12 00:02:50 INFO Vacuumed table table=closures29082026/11/12 00:02:50 INFO Completed upload id=19 objects=25 duration=3.227811ms29092026/11/12 00:02:50 INFO Vacuumed table table=objects29102026/11/12 00:02:50 INFO Completed upload id=2 objects=7 duration=6.085698ms29112026/11/12 00:02:50 INFO Received push request method=POST path=/29122026/11/12 00:02:50 INFO Received complete upload request method=POST path=/29132026/11/12 00:02:50 INFO Received push request method=POST path=/29142026/11/12 00:02:50 INFO Aborted multipart uploads count=0 kept=029152026/11/12 00:02:50 INFO Received push request method=POST path=/29162026/11/12 00:02:50 INFO Received push request method=POST path=/29172026/11/12 00:02:50 INFO Completed upload id=1 objects=9 duration=10.948424ms29182026/11/12 00:02:50 WARN Force mode enabled - objects will be deleted immediately without grace period29192026/11/12 00:02:50 INFO Received push request method=POST path=/29202026/11/12 00:02:50 INFO Aborted multipart uploads count=0 kept=029212026/11/12 00:02:50 INFO Received uploads request method=POST path=/29222026/11/12 00:02:50 WARN Force mode enabled - objects will be deleted immediately without grace period29232026/11/12 00:02:50 INFO Received complete push request method=POST path=/29242026/11/12 00:02:50 INFO Aborted multipart uploads count=0 kept=029252026/11/12 00:02:50 INFO Received push request method=POST path=/29262026/11/12 00:02:50 INFO Completed push id=1 objects=2 duration=4.017539ms29272026/11/12 00:02:50 INFO Received uploads request method=POST path=/29282026/11/12 00:02:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029292026/11/12 00:02:50 INFO Received complete push request method=POST path=/29302026/11/12 00:02:50 INFO Received push request method=POST path=/29312026/11/12 00:02:50 INFO Vacuumed table table=pending_closures29322026/11/12 00:02:50 INFO Completed push id=6 objects=8 duration=4.033759ms29332026/11/12 00:02:50 INFO Vacuumed table table=pending_objects29342026/11/12 00:02:50 INFO Received push request method=POST path=/29352026/11/12 00:02:50 INFO Vacuumed table table=multipart_uploads29362026/11/12 00:02:50 INFO Vacuumed table table=closures29372026/11/12 00:02:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=029382026/11/12 00:02:50 INFO Received complete upload request method=POST path=/29392026/11/12 00:02:50 INFO Vacuumed table table=objects29402026/11/12 00:02:50 INFO Received uploads request method=POST path=/29412026/11/12 00:02:50 INFO Vacuumed table table=pending_closures29422026/11/12 00:02:50 INFO Vacuumed table table=pending_objects29432026/11/12 00:02:50 INFO Completed upload id=4 objects=14 duration=2.991628ms29442026/11/12 00:02:50 INFO Vacuumed table table=multipart_uploads29452026/11/12 00:02:50 INFO Vacuumed table table=closures29462026/11/12 00:02:50 INFO Vacuumed table table=objects29472026/11/12 00:02:50 INFO Received push request method=POST path=/29482026/11/12 00:02:50 INFO Received uploads request method=POST path=/29492026/11/12 00:02:50 INFO Received uploads request method=POST path=/29502026/11/12 00:02:50 INFO Received push request method=POST path=/29512026/11/12 00:02:50 INFO Aborted multipart uploads count=0 kept=029522026/11/12 00:02:50 INFO Received uploads request method=POST path=/29532026/11/12 00:02:50 INFO Aborted multipart uploads count=0 kept=029542026/11/12 00:02:50 INFO Aborted multipart uploads count=0 kept=029552026/11/12 00:02:50 WARN Force mode enabled - objects will be deleted immediately without grace period29562026/11/12 00:02:50 WARN Force mode enabled - objects will be deleted immediately without grace period29572026/11/12 00:02:50 INFO Received complete upload request method=POST path=/29582026/11/12 00:02:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029592026/11/12 00:02:50 INFO Received push request method=POST path=/29602026/11/12 00:02:50 INFO Received complete push request method=POST path=/29612026/11/12 00:02:50 INFO Vacuumed table table=pending_closures29622026/11/12 00:02:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029632026/11/12 00:02:50 INFO Vacuumed table table=pending_objects29642026/11/12 00:02:50 INFO Vacuumed table table=multipart_uploads29652026/11/12 00:02:50 INFO Vacuumed table table=pending_closures29662026/11/12 00:02:50 INFO Received complete upload request method=POST path=/29672026/11/12 00:02:50 INFO Vacuumed table table=closures29682026/11/12 00:02:50 INFO Vacuumed table table=pending_objects29692026/11/12 00:02:50 INFO Vacuumed table table=multipart_uploads29702026/11/12 00:02:50 INFO Aborted multipart uploads count=0 kept=029712026/11/12 00:02:50 INFO Received uploads request method=POST path=/29722026/11/12 00:02:50 INFO Completed upload id=5 objects=5 duration=3.273071ms29732026/11/12 00:02:50 INFO Received push request method=POST path=/29742026/11/12 00:02:50 INFO Vacuumed table table=objects29752026/11/12 00:02:50 INFO Completed upload id=1 objects=15 duration=8.540481ms29762026/11/12 00:02:50 INFO Completed push id=3 objects=22 duration=7.750994ms29772026/11/12 00:02:50 INFO Vacuumed table table=closures29782026/11/12 00:02:50 WARN Force mode enabled - objects will be deleted immediately without grace period29792026/11/12 00:02:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029802026/11/12 00:02:50 INFO Vacuumed table table=objects29812026/11/12 00:02:50 INFO Vacuumed table table=pending_closures29822026/11/12 00:02:50 INFO Received uploads request method=POST path=/29832026/11/12 00:02:50 INFO Vacuumed table table=pending_objects29842026/11/12 00:02:50 INFO Vacuumed table table=multipart_uploads29852026/11/12 00:02:50 INFO Vacuumed table table=closures29862026/11/12 00:02:50 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."29872026/11/12 00:02:50 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."29882026/11/12 00:02:50 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."29892026/11/12 00:02:50 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."29902026/11/12 00:02:50 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."29912026/11/12 00:02:50 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."29922026/11/12 00:02:50 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."29932026/11/12 00:02:50 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."29942026/11/12 00:02:50 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."29952026/11/12 00:02:50 INFO Vacuumed table table=objects29962026/11/12 00:02:50 INFO Received push request method=POST path=/29972026/11/12 00:02:50 INFO Received uploads request method=POST path=/29982026/11/12 00:02:50 INFO Aborted multipart uploads count=0 kept=029992026/11/12 00:02:50 INFO Received uploads request method=POST path=/30002026/11/12 00:02:50 INFO Received push request method=POST path=/30012026/11/12 00:02:50 INFO Received uploads request method=POST path=/30022026/11/12 00:02:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030032026/11/12 00:02:50 INFO Vacuumed table table=pending_closures30042026/11/12 00:02:50 INFO Vacuumed table table=pending_objects30052026/11/12 00:02:50 INFO Vacuumed table table=multipart_uploads30062026/11/12 00:02:50 INFO Vacuumed table table=closures30072026/11/12 00:02:50 INFO Vacuumed table table=objects30082026/11/12 00:02:50 INFO Received uploads request method=POST path=/30092026/11/12 00:02:50 INFO Aborted multipart uploads count=0 kept=030102026/11/12 00:02:50 INFO Received uploads request method=POST path=/30112026/11/12 00:02:50 WARN Force mode enabled - objects will be deleted immediately without grace period30122026/11/12 00:02:50 INFO Aborted multipart uploads count=0 kept=030132026/11/12 00:02:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030142026/11/12 00:02:50 INFO Received uploads request method=POST path=/30152026/11/12 00:02:50 INFO Vacuumed table table=pending_closures30162026/11/12 00:02:50 WARN Force mode enabled - objects will be deleted immediately without grace period30172026/11/12 00:02:50 INFO Vacuumed table table=pending_objects30182026/11/12 00:02:50 INFO Vacuumed table table=multipart_uploads30192026/11/12 00:02:50 INFO Vacuumed table table=closures30202026/11/12 00:02:50 INFO Vacuumed table table=objects30212026/11/12 00:02:50 INFO Received uploads request method=POST path=/30222026/11/12 00:02:50 INFO Received uploads request method=POST path=/30232026/11/12 00:02:50 INFO Received complete upload request method=POST path=/30242026/11/12 00:02:50 INFO Completed upload id=7 objects=4 duration=1.172471ms30252026/11/12 00:02:50 INFO Received uploads request method=POST path=/30262026/11/12 00:02:50 INFO Received uploads request method=POST path=/30272026/11/12 00:02:50 INFO Received push request method=POST path=/30282026/11/12 00:02:50 INFO Received uploads request method=POST path=/30292026/11/12 00:02:50 INFO Received complete upload request method=POST path=/30302026/11/12 00:02:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=9 objects-failed-to-delete=030312026/11/12 00:02:50 INFO Vacuumed table table=pending_closures30322026/11/12 00:02:50 INFO Completed upload id=3 objects=24 duration=2.612025ms30332026/11/12 00:02:50 INFO Received push request method=POST path=/30342026/11/12 00:02:50 INFO Vacuumed table table=pending_objects30352026/11/12 00:02:50 INFO Vacuumed table table=multipart_uploads30362026/11/12 00:02:50 INFO Received complete push request method=POST path=/30372026/11/12 00:02:50 INFO Vacuumed table table=closures30382026/11/12 00:02:50 INFO Received push request method=POST path=/30392026/11/12 00:02:50 INFO Vacuumed table table=objects30402026/11/12 00:02:50 INFO Received push request method=POST path=/30412026/11/12 00:02:50 INFO Aborted multipart uploads count=0 kept=030422026/11/12 00:02:50 INFO Completed push id=1 objects=24 duration=7.759894ms30432026/11/12 00:02:50 INFO Received push request method=POST path=/30442026/11/12 00:02:50 INFO Aborted multipart uploads count=0 kept=030452026/11/12 00:02:50 INFO Received push request method=POST path=/30462026/11/12 00:02:50 WARN Force mode enabled - objects will be deleted immediately without grace period30472026/11/12 00:02:50 INFO Received push request method=POST path=/30482026/11/12 00:02:50 INFO Received uploads request method=POST path=/30492026/11/12 00:02:50 INFO Aborted multipart uploads count=0 kept=030502026/11/12 00:02:50 INFO Received uploads request method=POST path=/30512026/11/12 00:02:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030522026/11/12 00:02:50 INFO Vacuumed table table=pending_closures30532026/11/12 00:02:50 INFO Aborted multipart uploads count=0 kept=030542026/11/12 00:02:50 INFO Vacuumed table table=pending_objects30552026/11/12 00:02:50 INFO Received uploads request method=POST path=/30562026/11/12 00:02:50 INFO Vacuumed table table=multipart_uploads30572026/11/12 00:02:50 INFO Vacuumed table table=closures30582026/11/12 00:02:50 WARN Force mode enabled - objects will be deleted immediately without grace period30592026/11/12 00:02:50 INFO Received uploads request method=POST path=/30602026/11/12 00:02:50 INFO Vacuumed table table=objects30612026/11/12 00:02:50 INFO Received push request method=POST path=/30622026/11/12 00:02:50 INFO Received uploads request method=POST path=/30632026/11/12 00:02:50 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."30642026/11/12 00:02:50 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."30652026/11/12 00:02:50 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."30662026/11/12 00:02:50 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."30672026/11/12 00:02:50 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."30682026/11/12 00:02:50 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."30692026/11/12 00:02:50 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."30702026/11/12 00:02:50 INFO Received complete push request method=POST path=/30712026/11/12 00:02:50 INFO Completed push id=2 objects=25 duration=2.453003ms30722026/11/12 00:02:50 INFO Received push request method=POST path=/30732026/11/12 00:02:50 INFO Received complete upload request method=POST path=/30742026/11/12 00:02:50 INFO Completed upload id=23 objects=18 duration=2.543164ms30752026/11/12 00:02:51 INFO Received push request method=POST path=/30762026/11/12 00:02:51 INFO Received complete upload request method=POST path=/30772026/11/12 00:02:51 INFO Aborted multipart uploads count=0 kept=030782026/11/12 00:02:51 INFO Completed upload id=2 objects=2 duration=1.589195ms30792026/11/12 00:02:51 INFO Received complete upload request method=POST path=/30802026/11/12 00:02:51 INFO Received complete push request method=POST path=/30812026/11/12 00:02:51 INFO Completed upload id=10 objects=6 duration=2.245821ms30822026/11/12 00:02:51 INFO Completed push id=5 objects=8 duration=3.553114ms30832026/11/12 00:02:51 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."30842026/11/12 00:02:51 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."30852026/11/12 00:02:51 INFO Received complete upload request method=POST path=/30862026/11/12 00:02:51 INFO Completed upload id=23 objects=16 duration=2.161741ms30872026/11/12 00:02:51 INFO Aborted multipart uploads count=0 kept=030882026/11/12 00:02:51 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."30892026/11/12 00:02:51 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."30902026/11/12 00:02:51 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."30912026/11/12 00:02:51 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."30922026/11/12 00:02:51 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."30932026/11/12 00:02:51 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."30942026/11/12 00:02:51 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."30952026/11/12 00:02:51 INFO Received uploads request method=POST path=/30962026/11/12 00:02:51 INFO Received push request method=POST path=/30972026/11/12 00:02:51 INFO Received push request method=POST path=/30982026/11/12 00:02:51 INFO Received uploads request method=POST path=/30992026/11/12 00:02:51 INFO Received complete push request method=POST path=/31002026/11/12 00:02:51 INFO Received push request method=POST path=/31012026/11/12 00:02:51 INFO Completed push id=6 objects=5 duration=1.663176ms31022026/11/12 00:02:51 INFO Received push request method=POST path=/31032026/11/12 00:02:51 INFO Received push request method=POST path=/31042026/11/12 00:02:51 INFO Aborted multipart uploads count=0 kept=031052026/11/12 00:02:51 INFO Received complete push request method=POST path=/31062026/11/12 00:02:51 WARN Force mode enabled - objects will be deleted immediately without grace period31072026/11/12 00:02:51 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031082026/11/12 00:02:51 INFO Completed push id=7 objects=20 duration=2.454683ms31092026/11/12 00:02:51 INFO Vacuumed table table=pending_closures31102026/11/12 00:02:51 INFO Received uploads request method=POST path=/31112026/11/12 00:02:51 INFO Vacuumed table table=pending_objects31122026/11/12 00:02:51 INFO Aborted multipart uploads count=0 kept=031132026/11/12 00:02:51 INFO Vacuumed table table=multipart_uploads31142026/11/12 00:02:51 INFO Vacuumed table table=closures31152026/11/12 00:02:51 INFO Received complete upload request method=POST path=/31162026/11/12 00:02:51 INFO Vacuumed table table=objects31172026/11/12 00:02:51 INFO Received push request method=POST path=/31182026/11/12 00:02:51 INFO Completed upload id=3 objects=8 duration=1.853877ms31192026/11/12 00:02:51 INFO Received uploads request method=POST path=/31202026/11/12 00:02:51 INFO Received push request method=POST path=/31212026/11/12 00:02:51 INFO Received push request method=POST path=/31222026/11/12 00:02:51 INFO Aborted multipart uploads count=0 kept=031232026/11/12 00:02:51 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031242026/11/12 00:02:51 INFO Received push request method=POST path=/31252026/11/12 00:02:51 INFO Received push request method=POST path=/31262026/11/12 00:02:51 INFO Vacuumed table table=pending_closures31272026/11/12 00:02:51 INFO Vacuumed table table=pending_objects31282026/11/12 00:02:51 INFO Vacuumed table table=multipart_uploads31292026/11/12 00:02:51 INFO Received uploads request method=POST path=/31302026/11/12 00:02:51 INFO Received uploads request method=POST path=/31312026/11/12 00:02:51 INFO Received push request method=POST path=/31322026/11/12 00:02:51 INFO Received uploads request method=POST path=/31332026/11/12 00:02:51 INFO Vacuumed table table=closures31342026/11/12 00:02:51 INFO Aborted multipart uploads count=0 kept=031352026/11/12 00:02:51 INFO Vacuumed table table=objects31362026/11/12 00:02:51 INFO Received complete upload request method=POST path=/31372026/11/12 00:02:51 INFO Completed upload id=22 objects=22 duration=886.889µs31382026/11/12 00:02:51 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031392026/11/12 00:02:51 INFO Vacuumed table table=pending_closures31402026/11/12 00:02:51 INFO Received uploads request method=POST path=/31412026/11/12 00:02:51 INFO Vacuumed table table=pending_objects31422026/11/12 00:02:51 INFO Vacuumed table table=multipart_uploads31432026/11/12 00:02:51 INFO Vacuumed table table=closures31442026/11/12 00:02:51 INFO Vacuumed table table=objects31452026/11/12 00:02:51 INFO Received push request method=POST path=/31462026/11/12 00:02:51 INFO Aborted multipart uploads count=0 kept=031472026/11/12 00:02:51 INFO Received complete push request method=POST path=/31482026/11/12 00:02:51 INFO Completed push id=8 objects=6 duration=2.553905ms31492026/11/12 00:02:51 INFO Received complete push request method=POST path=/31502026/11/12 00:02:51 INFO Received push request method=POST path=/31512026/11/12 00:02:51 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031522026/11/12 00:02:51 INFO Vacuumed table table=pending_closures31532026/11/12 00:02:51 INFO Received push request method=POST path=/31542026/11/12 00:02:51 INFO Completed push id=16 objects=4 duration=2.362062ms31552026/11/12 00:02:51 INFO Vacuumed table table=pending_objects31562026/11/12 00:02:51 INFO Received push request method=POST path=/31572026/11/12 00:02:51 INFO Received push request method=POST path=/31582026/11/12 00:02:51 INFO Vacuumed table table=multipart_uploads31592026/11/12 00:02:51 INFO Vacuumed table table=closures31602026/11/12 00:02:51 INFO Vacuumed table table=objects31612026/11/12 00:02:51 INFO Aborted multipart uploads count=0 kept=031622026/11/12 00:02:51 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."31632026/11/12 00:02:51 INFO Received uploads request method=POST path=/31642026/11/12 00:02:51 INFO Received complete push request method=POST path=/31652026/11/12 00:02:51 INFO Received complete push request method=POST path=/31662026/11/12 00:02:51 INFO Completed push id=12 objects=7 duration=1.852497ms31672026/11/12 00:02:51 INFO Received push request method=POST path=/31682026/11/12 00:02:51 INFO Received complete push request method=POST path=/31692026/11/12 00:02:51 INFO Completed push id=13 objects=12 duration=3.560874ms31702026/11/12 00:02:51 INFO Received uploads request method=POST path=/31712026/11/12 00:02:51 INFO Completed push id=26 objects=5 duration=1.844677ms31722026/11/12 00:02:51 INFO Received complete push request method=POST path=/31732026/11/12 00:02:51 INFO Received uploads request method=POST path=/31742026/11/12 00:02:51 INFO Completed push id=5 objects=3 duration=3.269631ms31752026/11/12 00:02:51 INFO Received push request method=POST path=/31762026/11/12 00:02:51 INFO Aborted multipart uploads count=0 kept=031772026/11/12 00:02:51 INFO Aborted multipart uploads count=0 kept=031782026/11/12 00:02:51 INFO Aborted multipart uploads count=0 kept=031792026/11/12 00:02:51 INFO Received complete upload request method=POST path=/31802026/11/12 00:02:51 INFO Completed upload id=20 objects=5 duration=976.73µs31812026/11/12 00:02:51 INFO Received uploads request method=POST path=/31822026/11/12 00:02:51 INFO Aborted multipart uploads count=0 kept=031832026/11/12 00:02:51 INFO Received complete upload request method=POST path=/31842026/11/12 00:02:51 INFO Completed upload id=24 objects=14 duration=915.989µs31852026/11/12 00:02:51 INFO Received uploads request method=POST path=/31862026/11/12 00:02:51 WARN Force mode enabled - objects will be deleted immediately without grace period31872026/11/12 00:02:51 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031882026/11/12 00:02:51 INFO Aborted multipart uploads count=0 kept=031892026/11/12 00:02:51 INFO Vacuumed table table=pending_closures31902026/11/12 00:02:51 INFO Vacuumed table table=pending_objects31912026/11/12 00:02:51 INFO Vacuumed table table=multipart_uploads31922026/11/12 00:02:51 INFO Vacuumed table table=closures31932026/11/12 00:02:51 INFO Vacuumed table table=objects31942026/11/12 00:02:51 INFO Aborted multipart uploads count=0 kept=031952026/11/12 00:02:51 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=031962026/11/12 00:02:51 INFO Vacuumed table table=pending_closures31972026/11/12 00:02:51 INFO Received push request method=POST path=/31982026/11/12 00:02:51 INFO Received uploads request method=POST path=/31992026/11/12 00:02:51 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=032002026/11/12 00:02:51 INFO Vacuumed table table=pending_objects32012026/11/12 00:02:51 INFO Vacuumed table table=multipart_uploads32022026/11/12 00:02:51 INFO Vacuumed table table=pending_closures32032026/11/12 00:02:51 INFO Vacuumed table table=closures32042026/11/12 00:02:51 INFO Vacuumed table table=objects32052026/11/12 00:02:51 INFO Vacuumed table table=pending_objects32062026/11/12 00:02:51 INFO Vacuumed table table=multipart_uploads32072026/11/12 00:02:51 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032082026/11/12 00:02:51 INFO Received complete push request method=POST path=/32092026/11/12 00:02:51 INFO Vacuumed table table=closures32102026/11/12 00:02:51 INFO Vacuumed table table=pending_closures32112026/11/12 00:02:51 INFO Completed push id=19 objects=3 duration=1.855098ms32122026/11/12 00:02:51 INFO Vacuumed table table=objects32132026/11/12 00:02:51 INFO Vacuumed table table=pending_objects32142026/11/12 00:02:51 INFO Vacuumed table table=multipart_uploads32152026/11/12 00:02:51 INFO Vacuumed table table=closures32162026/11/12 00:02:51 INFO Vacuumed table table=objects32172026/11/12 00:02:51 INFO Received complete upload request method=POST path=/32182026/11/12 00:02:51 INFO Received push request method=POST path=/32192026/11/12 00:02:51 INFO Completed upload id=8 objects=3 duration=2.885067ms32202026/11/12 00:02:51 INFO Received push request method=POST path=/32212026/11/12 00:02:51 INFO Received push request method=POST path=/32222026/11/12 00:02:51 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=032232026/11/12 00:02:51 INFO Vacuumed table table=pending_closures32242026/11/12 00:02:51 INFO Vacuumed table table=pending_objects32252026/11/12 00:02:51 INFO Vacuumed table table=multipart_uploads32262026/11/12 00:02:51 INFO Received complete push request method=POST path=/32272026/11/12 00:02:51 INFO Vacuumed table table=closures32282026/11/12 00:02:51 INFO Aborted multipart uploads count=0 kept=032292026/11/12 00:02:51 INFO Received push request method=POST path=/32302026/11/12 00:02:51 INFO Completed push id=4 objects=4 duration=3.710696ms32312026/11/12 00:02:51 INFO Received uploads request method=POST path=/32322026/11/12 00:02:51 INFO Vacuumed table table=objects32332026/11/12 00:02:51 WARN Force mode enabled - objects will be deleted immediately without grace period32342026/11/12 00:02:51 INFO Received complete push request method=POST path=/32352026/11/12 00:02:51 INFO Aborted multipart uploads count=0 kept=032362026/11/12 00:02:51 INFO Completed push id=7 objects=15 duration=2.441743ms32372026/11/12 00:02:51 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032382026/11/12 00:02:51 INFO Vacuumed table table=pending_closures32392026/11/12 00:02:51 INFO Vacuumed table table=pending_objects32402026/11/12 00:02:51 INFO Received complete push request method=POST path=/32412026/11/12 00:02:51 INFO Vacuumed table table=multipart_uploads32422026/11/12 00:02:51 INFO Vacuumed table table=closures32432026/11/12 00:02:51 INFO Received push request method=POST path=/32442026/11/12 00:02:51 INFO Vacuumed table table=objects32452026/11/12 00:02:51 INFO Completed push id=8 objects=6 duration=1.845698ms32462026/11/12 00:02:51 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032472026/11/12 00:02:51 INFO Received uploads request method=POST path=/32482026/11/12 00:02:51 INFO Vacuumed table table=pending_closures32492026/11/12 00:02:51 INFO Vacuumed table table=pending_objects32502026/11/12 00:02:51 INFO Vacuumed table table=multipart_uploads32512026/11/12 00:02:51 INFO Vacuumed table table=closures32522026/11/12 00:02:51 INFO Vacuumed table table=objects32532026/11/12 00:02:51 INFO Received uploads request method=POST path=/32542026/11/12 00:02:51 INFO Received push request method=POST path=/32552026/11/12 00:02:51 INFO Received complete push request method=POST path=/32562026/11/12 00:02:51 INFO Received push request method=POST path=/32572026/11/12 00:02:51 INFO Completed push id=6 objects=6 duration=2.07222ms32582026/11/12 00:02:51 INFO Received push request method=POST path=/32592026/11/12 00:02:51 INFO Received push request method=POST path=/32602026/11/12 00:02:51 INFO Received uploads request method=POST path=/32612026/11/12 00:02:51 INFO Received push request method=POST path=/32622026/11/12 00:02:51 INFO Received uploads request method=POST path=/32632026/11/12 00:02:51 INFO Received complete push request method=POST path=/32642026/11/12 00:02:51 INFO Received complete push request method=POST path=/32652026/11/12 00:02:51 INFO Received uploads request method=POST path=/32662026/11/12 00:02:51 INFO Completed push id=6 objects=6 duration=2.258602ms32672026/11/12 00:02:51 INFO Received uploads request method=POST path=/32682026/11/12 00:02:51 INFO Completed push id=19 objects=6 duration=3.749216ms32692026/11/12 00:02:51 INFO Received push request method=POST path=/32702026/11/12 00:02:51 INFO Received uploads request method=POST path=/32712026/11/12 00:02:51 INFO Received complete push request method=POST path=/32722026/11/12 00:02:51 INFO Received push request method=POST path=/32732026/11/12 00:02:51 INFO Received complete upload request method=POST path=/32742026/11/12 00:02:51 INFO Aborted multipart uploads count=0 kept=032752026/11/12 00:02:51 INFO Completed upload id=25 objects=4 duration=1.01453ms32762026/11/12 00:02:51 INFO Received push request method=POST path=/32772026/11/12 00:02:51 WARN Force mode enabled - objects will be deleted immediately without grace period32782026/11/12 00:02:51 INFO Completed push id=5 objects=9 duration=5.717994ms32792026/11/12 00:02:51 INFO Received uploads request method=POST path=/32802026/11/12 00:02:51 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032812026/11/12 00:02:51 INFO Vacuumed table table=pending_closures32822026/11/12 00:02:51 INFO Vacuumed table table=pending_objects32832026/11/12 00:02:51 INFO Vacuumed table table=multipart_uploads32842026/11/12 00:02:51 INFO Vacuumed table table=closures32852026/11/12 00:02:51 INFO Vacuumed table table=objects32862026/11/12 00:02:51 INFO Aborted multipart uploads count=0 kept=032872026/11/12 00:02:51 INFO Received complete push request method=POST path=/32882026/11/12 00:02:51 INFO Received push request method=POST path=/32892026/11/12 00:02:51 INFO Received push request method=POST path=/32902026/11/12 00:02:51 INFO Completed push id=7 objects=27 duration=2.06894ms32912026/11/12 00:02:51 INFO Received push request method=POST path=/32922026/11/12 00:02:51 WARN Force mode enabled - objects will be deleted immediately without grace period32932026/11/12 00:02:51 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032942026/11/12 00:02:51 INFO Received complete push request method=POST path=/32952026/11/12 00:02:51 INFO Vacuumed table table=pending_closures32962026/11/12 00:02:51 INFO Received push request method=POST path=/32972026/11/12 00:02:51 INFO Vacuumed table table=pending_objects32982026/11/12 00:02:51 INFO Vacuumed table table=multipart_uploads32992026/11/12 00:02:51 INFO Aborted multipart uploads count=0 kept=033002026/11/12 00:02:51 INFO Vacuumed table table=closures33012026/11/12 00:02:51 INFO Completed push id=3 objects=21 duration=4.838606ms33022026/11/12 00:02:51 INFO Vacuumed table table=objects33032026/11/12 00:02:51 INFO Received push request method=POST path=/33042026/11/12 00:02:51 INFO Received push request method=POST path=/33052026/11/12 00:02:51 WARN Force mode enabled - objects will be deleted immediately without grace period33062026/11/12 00:02:51 INFO Received complete push request method=POST path=/33072026/11/12 00:02:51 INFO Received uploads request method=POST path=/33082026/11/12 00:02:51 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."33092026/11/12 00:02:51 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."33102026/11/12 00:02:51 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."33112026/11/12 00:02:51 INFO Completed push id=10 objects=15 duration=2.220861ms33122026/11/12 00:02:51 INFO Received uploads request method=POST path=/33132026/11/12 00:02:51 INFO Received complete upload request method=POST path=/33142026/11/12 00:02:51 INFO Aborted multipart uploads count=0 kept=033152026/11/12 00:02:51 INFO Completed upload id=10 objects=17 duration=1.268312ms33162026/11/12 00:02:51 INFO Received uploads request method=POST path=/33172026/11/12 00:02:51 INFO Aborted multipart uploads count=0 kept=033182026/11/12 00:02:51 INFO Received complete upload request method=POST path=/33192026/11/12 00:02:51 INFO Completed upload id=3 objects=9 duration=1.276552ms33202026/11/12 00:02:51 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033212026/11/12 00:02:51 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033222026/11/12 00:02:51 INFO Vacuumed table table=pending_closures33232026/11/12 00:02:51 INFO Vacuumed table table=pending_closures33242026/11/12 00:02:51 INFO Vacuumed table table=pending_objects33252026/11/12 00:02:51 INFO Vacuumed table table=multipart_uploads33262026/11/12 00:02:51 INFO Vacuumed table table=pending_objects33272026/11/12 00:02:51 INFO Vacuumed table table=closures33282026/11/12 00:02:51 INFO Received complete push request method=POST path=/33292026/11/12 00:02:51 INFO Vacuumed table table=multipart_uploads33302026/11/12 00:02:51 INFO Received push request method=POST path=/33312026/11/12 00:02:51 INFO Vacuumed table table=objects33322026/11/12 00:02:51 INFO Aborted multipart uploads count=0 kept=033332026/11/12 00:02:51 INFO Completed push id=20 objects=8 duration=1.527274ms33342026/11/12 00:02:51 INFO Vacuumed table table=closures33352026/11/12 00:02:51 WARN Force mode enabled - objects will be deleted immediately without grace period33362026/11/12 00:02:51 INFO Vacuumed table table=objects33372026/11/12 00:02:51 INFO Received push request method=POST path=/33382026/11/12 00:02:51 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."33392026/11/12 00:02:51 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."33402026/11/12 00:02:51 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."33412026/11/12 00:02:51 INFO Received uploads request method=POST path=/33422026/11/12 00:02:51 INFO Received push request method=POST path=/33432026/11/12 00:02:51 INFO Aborted multipart uploads count=0 kept=033442026/11/12 00:02:51 INFO Received push request method=POST path=/33452026/11/12 00:02:51 INFO Received uploads request method=POST path=/33462026/11/12 00:02:51 WARN Force mode enabled - objects will be deleted immediately without grace period33472026/11/12 00:02:51 INFO Received push request method=POST path=/33482026/11/12 00:02:51 INFO Received complete push request method=POST path=/33492026/11/12 00:02:51 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033502026/11/12 00:02:51 INFO Received uploads request method=POST path=/33512026/11/12 00:02:51 INFO Completed push id=19 objects=14 duration=3.20391ms33522026/11/12 00:02:51 INFO Received push request method=POST path=/33532026/11/12 00:02:51 INFO Vacuumed table table=pending_closures33542026/11/12 00:02:51 INFO Vacuumed table table=pending_objects33552026/11/12 00:02:51 INFO Vacuumed table table=multipart_uploads33562026/11/12 00:02:51 INFO Vacuumed table table=closures33572026/11/12 00:02:51 INFO Aborted multipart uploads count=0 kept=033582026/11/12 00:02:51 INFO Vacuumed table table=objects33592026/11/12 00:02:51 INFO Received complete push request method=POST path=/33602026/11/12 00:02:51 INFO Received push request method=POST path=/33612026/11/12 00:02:51 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033622026/11/12 00:02:51 INFO Completed push id=5 objects=6 duration=1.565154ms33632026/11/12 00:02:51 INFO Vacuumed table table=pending_closures33642026/11/12 00:02:51 INFO Vacuumed table table=pending_objects33652026/11/12 00:02:51 INFO Vacuumed table table=multipart_uploads33662026/11/12 00:02:51 INFO Received push request method=POST path=/33672026/11/12 00:02:51 INFO Vacuumed table table=closures33682026/11/12 00:02:51 INFO Vacuumed table table=objects33692026/11/12 00:02:51 INFO Aborted multipart uploads count=0 kept=033702026/11/12 00:02:51 INFO Received uploads request method=POST path=/33712026/11/12 00:02:51 WARN Force mode enabled - objects will be deleted immediately without grace period33722026/11/12 00:02:51 INFO Received complete push request method=POST path=/33732026/11/12 00:02:51 INFO Aborted multipart uploads count=0 kept=033742026/11/12 00:02:51 INFO Received push request method=POST path=/33752026/11/12 00:02:51 INFO Received push request method=POST path=/33762026/11/12 00:02:51 INFO Completed push id=14 objects=6 duration=1.901998ms33772026/11/12 00:02:51 INFO Aborted multipart uploads count=0 kept=033782026/11/12 00:02:51 WARN Force mode enabled - objects will be deleted immediately without grace period33792026/11/12 00:02:51 INFO Aborted multipart uploads count=0 kept=033802026/11/12 00:02:51 WARN Force mode enabled - objects will be deleted immediately without grace period33812026/11/12 00:02:51 INFO Received complete push request method=POST path=/33822026/11/12 00:02:51 WARN Force mode enabled - objects will be deleted immediately without grace period33832026/11/12 00:02:51 INFO Completed push id=19 objects=8 duration=3.220031ms33842026/11/12 00:02:51 INFO Received complete push request method=POST path=/33852026/11/12 00:02:51 INFO Completed push id=15 objects=7 duration=1.787377ms33862026/11/12 00:02:51 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=033872026/11/12 00:02:51 INFO Received uploads request method=POST path=/33882026/11/12 00:02:51 INFO Vacuumed table table=pending_closures33892026/11/12 00:02:51 INFO Received push request method=POST path=/33902026/11/12 00:02:51 INFO Vacuumed table table=pending_objects33912026/11/12 00:02:51 INFO Received complete push request method=POST path=/33922026/11/12 00:02:51 INFO Vacuumed table table=multipart_uploads33932026/11/12 00:02:51 INFO Received complete push request method=POST path=/33942026/11/12 00:02:51 INFO Vacuumed table table=closures33952026/11/12 00:02:51 INFO Completed push id=27 objects=11 duration=1.231052ms33962026/11/12 00:02:51 INFO Received uploads request method=POST path=/33972026/11/12 00:02:51 INFO Vacuumed table table=objects33982026/11/12 00:02:51 INFO Completed push id=15 objects=5 duration=2.315542ms33992026/11/12 00:02:51 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=034002026/11/12 00:02:51 INFO Vacuumed table table=pending_closures34012026/11/12 00:02:51 INFO Received uploads request method=POST path=/34022026/11/12 00:02:51 INFO Vacuumed table table=pending_objects34032026/11/12 00:02:51 INFO Vacuumed table table=multipart_uploads34042026/11/12 00:02:51 INFO Vacuumed table table=closures34052026/11/12 00:02:51 INFO Vacuumed table table=objects34062026/11/12 00:02:51 INFO Received push request method=POST path=/34072026/11/12 00:02:51 INFO Received push request method=POST path=/34082026/11/12 00:02:51 INFO Received complete upload request method=POST path=/34092026/11/12 00:02:51 INFO Completed upload id=29 objects=26 duration=959.409µs34102026/11/12 00:02:51 INFO Aborted multipart uploads count=0 kept=034112026/11/12 00:02:51 INFO Received uploads request method=POST path=/34122026/11/12 00:02:51 INFO Received uploads request method=POST path=/34132026/11/12 00:02:51 WARN Force mode enabled - objects will be deleted immediately without grace period34142026/11/12 00:02:51 INFO Aborted multipart uploads count=0 kept=034152026/11/12 00:02:51 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034162026/11/12 00:02:51 INFO Vacuumed table table=pending_closures34172026/11/12 00:02:51 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034182026/11/12 00:02:51 INFO Vacuumed table table=pending_objects34192026/11/12 00:02:51 INFO Vacuumed table table=multipart_uploads34202026/11/12 00:02:51 INFO Vacuumed table table=closures34212026/11/12 00:02:51 INFO Vacuumed table table=pending_closures34222026/11/12 00:02:51 INFO Received push request method=POST path=/34232026/11/12 00:02:51 INFO Aborted multipart uploads count=0 kept=034242026/11/12 00:02:51 INFO Vacuumed table table=objects34252026/11/12 00:02:51 INFO Vacuumed table table=pending_objects34262026/11/12 00:02:51 INFO Vacuumed table table=multipart_uploads34272026/11/12 00:02:51 INFO Vacuumed table table=closures34282026/11/12 00:02:51 INFO Vacuumed table table=objects34292026/11/12 00:02:51 INFO Aborted multipart uploads count=0 kept=034302026/11/12 00:02:51 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034312026/11/12 00:02:51 INFO Vacuumed table table=pending_closures34322026/11/12 00:02:51 INFO Received complete push request method=POST path=/34332026/11/12 00:02:51 INFO Vacuumed table table=pending_objects34342026/11/12 00:02:51 INFO Completed push id=30 objects=8 duration=933.529µs34352026/11/12 00:02:51 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=8 objects-deleted-after-grace-period=8 objects-failed-to-delete=034362026/11/12 00:02:51 INFO Vacuumed table table=multipart_uploads34372026/11/12 00:02:51 INFO Vacuumed table table=closures34382026/11/12 00:02:51 INFO Vacuumed table table=objects34392026/11/12 00:02:51 INFO Vacuumed table table=pending_closures34402026/11/12 00:02:51 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=1 objects-deleted-after-grace-period=0 objects-failed-to-delete=034412026/11/12 00:02:51 INFO Vacuumed table table=pending_closures34422026/11/12 00:02:51 INFO Vacuumed table table=pending_objects34432026/11/12 00:02:51 INFO Vacuumed table table=multipart_uploads34442026/11/12 00:02:51 INFO Vacuumed table table=pending_objects34452026/11/12 00:02:51 INFO Vacuumed table table=closures34462026/11/12 00:02:51 INFO Vacuumed table table=multipart_uploads34472026/11/12 00:02:51 INFO Vacuumed table table=closures34482026/11/12 00:02:51 INFO Vacuumed table table=objects34492026/11/12 00:02:51 INFO Vacuumed table table=objects34502026/11/12 00:02:51 INFO Received uploads request method=POST path=/34512026/11/12 00:02:51 INFO Aborted multipart uploads count=0 kept=034522026/11/12 00:02:51 WARN Force mode enabled - objects will be deleted immediately without grace period34532026/11/12 00:02:51 INFO Received uploads request method=POST path=/34542026/11/12 00:02:51 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034552026/11/12 00:02:51 INFO Received complete upload request method=POST path=/34562026/11/12 00:02:51 INFO Vacuumed table table=pending_closures34572026/11/12 00:02:51 INFO Completed upload id=31 objects=5 duration=942.349µs34582026/11/12 00:02:51 INFO Vacuumed table table=pending_objects34592026/11/12 00:02:51 INFO Vacuumed table table=multipart_uploads34602026/11/12 00:02:51 INFO Vacuumed table table=closures34612026/11/12 00:02:51 INFO Vacuumed table table=objects34622026/11/12 00:02:51 INFO Received push request method=POST path=/34632026/11/12 00:02:51 INFO Received push request method=POST path=/34642026/11/12 00:02:51 INFO Received push request method=POST path=/34652026/11/12 00:02:51 INFO Received complete push request method=POST path=/34662026/11/12 00:02:51 INFO Aborted multipart uploads count=0 kept=034672026/11/12 00:02:51 INFO Completed push id=9 objects=7 duration=3.801156ms34682026/11/12 00:02:51 INFO Aborted multipart uploads count=0 kept=034692026/11/12 00:02:51 INFO Received push request method=POST path=/34702026/11/12 00:02:51 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034712026/11/12 00:02:51 INFO Received complete push request method=POST path=/34722026/11/12 00:02:51 WARN Force mode enabled - objects will be deleted immediately without grace period34732026/11/12 00:02:51 INFO Vacuumed table table=pending_closures34742026/11/12 00:02:51 INFO Completed push id=32 objects=3 duration=1.151751ms34752026/11/12 00:02:51 INFO Vacuumed table table=pending_objects34762026/11/12 00:02:51 INFO Vacuumed table table=multipart_uploads34772026/11/12 00:02:51 INFO Vacuumed table table=closures34782026/11/12 00:02:51 INFO Vacuumed table table=objects34792026/11/12 00:02:51 INFO Received push request method=POST path=/34802026/11/12 00:02:51 INFO Received complete upload request method=POST path=/34812026/11/12 00:02:51 INFO Completed upload id=22 objects=2 duration=1.106691ms34822026/11/12 00:02:51 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."34832026/11/12 00:02:51 INFO Received push request method=POST path=/34842026/11/12 00:02:51 INFO Received uploads request method=POST path=/34852026/11/12 00:02:51 INFO Aborted multipart uploads count=0 kept=034862026/11/12 00:02:51 INFO Aborted multipart uploads count=0 kept=034872026/11/12 00:02:51 INFO Received uploads request method=POST path=/34882026/11/12 00:02:51 INFO Received complete upload request method=POST path=/34892026/11/12 00:02:51 WARN Force mode enabled - objects will be deleted immediately without grace period34902026/11/12 00:02:51 INFO Aborted multipart uploads count=0 kept=034912026/11/12 00:02:51 INFO Received complete push request method=POST path=/34922026/11/12 00:02:51 INFO Completed upload id=10 objects=7 duration=1.775597ms34932026/11/12 00:02:51 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034942026/11/12 00:02:51 INFO Vacuumed table table=pending_closures34952026/11/12 00:02:51 INFO Completed push id=31 objects=14 duration=2.828527ms34962026/11/12 00:02:51 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=034972026/11/12 00:02:51 INFO Vacuumed table table=pending_objects34982026/11/12 00:02:51 INFO Vacuumed table table=pending_closures34992026/11/12 00:02:51 INFO Vacuumed table table=multipart_uploads35002026/11/12 00:02:51 INFO Vacuumed table table=closures35012026/11/12 00:02:51 INFO Vacuumed table table=pending_objects35022026/11/12 00:02:51 INFO Vacuumed table table=multipart_uploads35032026/11/12 00:02:51 INFO Vacuumed table table=objects35042026/11/12 00:02:51 INFO Received uploads request method=POST path=/35052026/11/12 00:02:51 INFO Received uploads request method=POST path=/35062026/11/12 00:02:51 INFO Vacuumed table table=closures35072026/11/12 00:02:51 INFO Vacuumed table table=objects35082026/11/12 00:02:51 INFO Received uploads request method=POST path=/35092026/11/12 00:02:51 INFO Received uploads request method=POST path=/35102026/11/12 00:02:51 INFO Received complete upload request method=POST path=/35112026/11/12 00:02:51 INFO Completed upload id=14 objects=8 duration=2.009019ms35122026/11/12 00:02:51 INFO Received uploads request method=POST path=/35132026/11/12 00:02:51 INFO Received uploads request method=POST path=/35142026/11/12 00:02:51 INFO Received complete push request method=POST path=/35152026/11/12 00:02:51 INFO Completed push id=12 objects=2 duration=3.080449ms35162026/11/12 00:02:51 INFO Received uploads request method=POST path=/35172026/11/12 00:02:51 INFO Received push request method=POST path=/35182026/11/12 00:02:51 INFO Received uploads request method=POST path=/35192026/11/12 00:02:51 INFO Received uploads request method=POST path=/35202026/11/12 00:02:51 INFO Received complete upload request method=POST path=/35212026/11/12 00:02:51 INFO Completed upload id=13 objects=14 duration=2.984949ms35222026/11/12 00:02:51 INFO Received push request method=POST path=/35232026/11/12 00:02:51 INFO Received uploads request method=POST path=/35242026/11/12 00:02:51 INFO Received uploads request method=POST path=/35252026/11/12 00:02:51 INFO Received complete upload request method=POST path=/35262026/11/12 00:02:51 INFO Completed upload id=18 objects=4 duration=1.465714ms35272026/11/12 00:02:51 INFO Aborted multipart uploads count=0 kept=035282026/11/12 00:02:51 WARN Force mode enabled - objects will be deleted immediately without grace period35292026/11/12 00:02:51 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035302026/11/12 00:02:51 INFO Vacuumed table table=pending_closures35312026/11/12 00:02:51 INFO Vacuumed table table=pending_objects35322026/11/12 00:02:51 INFO Received push request method=POST path=/35332026/11/12 00:02:51 INFO Vacuumed table table=multipart_uploads35342026/11/12 00:02:51 INFO Vacuumed table table=closures35352026/11/12 00:02:51 INFO Aborted multipart uploads count=0 kept=035362026/11/12 00:02:51 INFO Vacuumed table table=objects35372026/11/12 00:02:51 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035382026/11/12 00:02:51 INFO Received uploads request method=POST path=/35392026/11/12 00:02:51 INFO Vacuumed table table=pending_closures35402026/11/12 00:02:51 INFO Vacuumed table table=pending_objects35412026/11/12 00:02:51 INFO Vacuumed table table=multipart_uploads35422026/11/12 00:02:51 INFO Vacuumed table table=closures35432026/11/12 00:02:51 INFO Vacuumed table table=objects35442026/11/12 00:02:51 INFO Received complete push request method=POST path=/35452026/11/12 00:02:51 INFO Received push request method=POST path=/35462026/11/12 00:02:51 INFO Completed push id=7 objects=7 duration=1.519074ms35472026/11/12 00:02:51 INFO Aborted multipart uploads count=0 kept=035482026/11/12 00:02:51 INFO Aborted multipart uploads count=0 kept=035492026/11/12 00:02:51 WARN Force mode enabled - objects will be deleted immediately without grace period35502026/11/12 00:02:51 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035512026/11/12 00:02:51 INFO Vacuumed table table=pending_closures35522026/11/12 00:02:51 INFO Vacuumed table table=pending_objects35532026/11/12 00:02:51 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=035542026/11/12 00:02:51 INFO Vacuumed table table=multipart_uploads35552026/11/12 00:02:51 INFO Vacuumed table table=closures35562026/11/12 00:02:51 INFO Aborted multipart uploads count=0 kept=035572026/11/12 00:02:51 INFO Vacuumed table table=pending_closures35582026/11/12 00:02:51 INFO Received complete push request method=POST path=/35592026/11/12 00:02:51 INFO Received complete push request method=POST path=/35602026/11/12 00:02:51 INFO Vacuumed table table=pending_objects35612026/11/12 00:02:51 INFO Vacuumed table table=objects35622026/11/12 00:02:51 INFO Completed push id=12 objects=6 duration=15.826931ms35632026/11/12 00:02:51 INFO Vacuumed table table=multipart_uploads35642026/11/12 00:02:51 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035652026/11/12 00:02:51 INFO Vacuumed table table=closures35662026/11/12 00:02:51 INFO Completed push id=31 objects=6 duration=3.024969ms35672026/11/12 00:02:51 INFO Vacuumed table table=objects35682026/11/12 00:02:51 INFO Vacuumed table table=pending_closures35692026/11/12 00:02:51 INFO Vacuumed table table=pending_objects35702026/11/12 00:02:51 INFO Vacuumed table table=multipart_uploads35712026/11/12 00:02:51 INFO Vacuumed table table=closures35722026/11/12 00:02:51 INFO Vacuumed table table=objects35732026/11/12 00:02:51 INFO Received uploads request method=POST path=/35742026/11/12 00:02:51 INFO Received push request method=POST path=/35752026/11/12 00:02:51 INFO Received push request method=POST path=/35762026/11/12 00:02:51 INFO Received uploads request method=POST path=/35772026/11/12 00:02:51 INFO Received uploads request method=POST path=/35782026/11/12 00:02:51 INFO Received push request method=POST path=/35792026/11/12 00:02:51 INFO Received uploads request method=POST path=/35802026/11/12 00:02:51 INFO Received complete push request method=POST path=/35812026/11/12 00:02:51 INFO Completed push id=14 objects=3 duration=7.824315ms35822026/11/12 00:02:51 INFO Received complete upload request method=POST path=/35832026/11/12 00:02:51 INFO Completed upload id=14 objects=16 duration=2.646126ms35842026/11/12 00:02:51 INFO Received complete push request method=POST path=/35852026/11/12 00:02:51 INFO Completed push id=13 objects=22 duration=1.882658ms35862026/11/12 00:02:51 INFO Received complete upload request method=POST path=/35872026/11/12 00:02:51 INFO Completed upload id=16 objects=17 duration=5.801615ms35882026/11/12 00:02:51 INFO Aborted multipart uploads count=0 kept=035892026/11/12 00:02:51 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035902026/11/12 00:02:51 INFO Vacuumed table table=pending_closures35912026/11/12 00:02:51 INFO Vacuumed table table=pending_objects35922026/11/12 00:02:51 INFO Vacuumed table table=multipart_uploads35932026/11/12 00:02:51 INFO Vacuumed table table=closures35942026/11/12 00:02:51 INFO Vacuumed table table=objects35952026/11/12 00:02:51 INFO Received uploads request method=POST path=/35962026/11/12 00:02:51 INFO Received complete push request method=POST path=/35972026/11/12 00:02:51 INFO Received complete push request method=POST path=/35982026/11/12 00:02:51 INFO Completed push id=21 objects=6 duration=1.749917ms35992026/11/12 00:02:51 INFO Completed push id=21 objects=9 duration=2.608184ms36002026/11/12 00:02:51 INFO Received complete push request method=POST path=/36012026/11/12 00:02:51 INFO Received complete push request method=POST path=/36022026/11/12 00:02:51 INFO Aborted multipart uploads count=0 kept=036032026/11/12 00:02:51 INFO Completed push id=1 objects=11 duration=2.887227ms36042026/11/12 00:02:51 INFO Completed push id=22 objects=3 duration=2.680186ms36052026/11/12 00:02:51 WARN Force mode enabled - objects will be deleted immediately without grace period36062026/11/12 00:02:51 INFO Received complete upload request method=POST path=/36072026/11/12 00:02:51 INFO Completed upload id=22 objects=7 duration=2.696745ms36082026/11/12 00:02:51 INFO Received uploads request method=POST path=/36092026/11/12 00:02:51 INFO Aborted multipart uploads count=0 kept=036102026/11/12 00:02:51 INFO Received complete push request method=POST path=/36112026/11/12 00:02:51 INFO Received complete push request method=POST path=/36122026/11/12 00:02:51 INFO Completed push id=24 objects=4 duration=1.141791ms36132026/11/12 00:02:51 INFO Completed push id=17 objects=11 duration=8.811404ms36142026/11/12 00:02:51 INFO Received complete upload request method=POST path=/36152026/11/12 00:02:51 INFO Received complete push request method=POST path=/36162026/11/12 00:02:51 INFO Completed upload id=16 objects=3 duration=1.313272ms36172026/11/12 00:02:51 WARN Force mode enabled - objects will be deleted immediately without grace period36182026/11/12 00:02:51 INFO Completed push id=1 objects=9 duration=1.799777ms36192026/11/12 00:02:51 INFO Received uploads request method=POST path=/36202026/11/12 00:02:51 INFO Received complete push request method=POST path=/36212026/11/12 00:02:51 INFO Received push request method=POST path=/36222026/11/12 00:02:51 INFO Received push request method=POST path=/36232026/11/12 00:02:51 INFO Completed push id=25 objects=9 duration=1.954898ms36242026/11/12 00:02:51 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."36252026/11/12 00:02:51 INFO Received push request method=POST path=/36262026/11/12 00:02:51 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=8 objects-deleted-after-grace-period=8 objects-failed-to-delete=036272026/11/12 00:02:51 INFO Vacuumed table table=pending_closures36282026/11/12 00:02:51 INFO Vacuumed table table=pending_objects36292026/11/12 00:02:51 INFO Vacuumed table table=multipart_uploads36302026/11/12 00:02:51 INFO Vacuumed table table=closures36312026/11/12 00:02:51 INFO Vacuumed table table=objects36322026/11/12 00:02:51 INFO Received push request method=POST path=/36332026/11/12 00:02:51 INFO Received complete upload request method=POST path=/36342026/11/12 00:02:51 INFO Completed upload id=26 objects=3 duration=8.473381ms36352026/11/12 00:02:51 INFO Received uploads request method=POST path=/36362026/11/12 00:02:51 INFO Aborted multipart uploads count=0 kept=036372026/11/12 00:02:51 INFO Received push request method=POST path=/36382026/11/12 00:02:51 WARN Force mode enabled - objects will be deleted immediately without grace period36392026/11/12 00:02:51 INFO Received push request method=POST path=/36402026/11/12 00:02:51 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036412026/11/12 00:02:51 INFO Vacuumed table table=pending_closures36422026/11/12 00:02:51 INFO Vacuumed table table=pending_objects36432026/11/12 00:02:51 INFO Vacuumed table table=multipart_uploads36442026/11/12 00:02:51 INFO Vacuumed table table=closures36452026/11/12 00:02:51 INFO Vacuumed table table=objects36462026/11/12 00:02:51 INFO Aborted multipart uploads count=0 kept=036472026/11/12 00:02:51 WARN Force mode enabled - objects will be deleted immediately without grace period36482026/11/12 00:02:51 INFO Received complete upload request method=POST path=/36492026/11/12 00:02:51 INFO Received complete upload request method=POST path=/36502026/11/12 00:02:51 INFO Received push request method=POST path=/36512026/11/12 00:02:51 INFO Completed upload id=26 objects=6 duration=1.725536ms36522026/11/12 00:02:51 INFO Completed upload id=27 objects=7 duration=2.004599ms36532026/11/12 00:02:51 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036542026/11/12 00:02:51 INFO Vacuumed table table=pending_closures36552026/11/12 00:02:51 INFO Vacuumed table table=pending_objects36562026/11/12 00:02:51 INFO Vacuumed table table=multipart_uploads36572026/11/12 00:02:51 INFO Vacuumed table table=closures36582026/11/12 00:02:51 INFO Vacuumed table table=objects36592026/11/12 00:02:51 INFO Received complete upload request method=POST path=/36602026/11/12 00:02:51 INFO Completed upload id=2 objects=7 duration=10.46406ms36612026/11/12 00:02:51 INFO Received uploads request method=POST path=/36622026/11/12 00:02:51 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036632026/11/12 00:02:51 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036642026/11/12 00:02:51 INFO Vacuumed table table=pending_closures36652026/11/12 00:02:51 INFO Vacuumed table table=pending_objects36662026/11/12 00:02:51 INFO Vacuumed table table=multipart_uploads36672026/11/12 00:02:51 INFO Vacuumed table table=pending_closures36682026/11/12 00:02:51 INFO Vacuumed table table=pending_objects36692026/11/12 00:02:51 INFO Vacuumed table table=multipart_uploads36702026/11/12 00:02:51 INFO Vacuumed table table=closures36712026/11/12 00:02:51 INFO Vacuumed table table=objects36722026/11/12 00:02:51 INFO Vacuumed table table=closures36732026/11/12 00:02:51 INFO Received uploads request method=POST path=/36742026/11/12 00:02:51 INFO Vacuumed table table=objects36752026/11/12 00:02:51 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036762026/11/12 00:02:51 INFO Received complete upload request method=POST path=/36772026/11/12 00:02:51 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=036782026/11/12 00:02:51 INFO Vacuumed table table=pending_closures36792026/11/12 00:02:51 INFO Vacuumed table table=pending_objects36802026/11/12 00:02:51 INFO Vacuumed table table=multipart_uploads36812026/11/12 00:02:51 INFO Vacuumed table table=pending_closures36822026/11/12 00:02:51 INFO Vacuumed table table=closures36832026/11/12 00:02:51 INFO Vacuumed table table=pending_objects36842026/11/12 00:02:51 INFO Vacuumed table table=objects36852026/11/12 00:02:51 INFO Vacuumed table table=multipart_uploads36862026/11/12 00:02:51 INFO Completed upload id=15 objects=3 duration=3.117309ms36872026/11/12 00:02:51 INFO Vacuumed table table=closures36882026/11/12 00:02:51 INFO Vacuumed table table=objects36892026/11/12 00:02:51 INFO Received uploads request method=POST path=/36902026/11/12 00:02:51 INFO Received complete upload request method=POST path=/36912026/11/12 00:02:51 INFO Completed upload id=28 objects=3 duration=990.469µs36922026/11/12 00:02:51 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."36932026/11/12 00:02:51 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."36942026/11/12 00:02:51 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."36952026/11/12 00:02:51 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."36962026/11/12 00:02:51 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."36972026/11/12 00:02:51 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."36982026/11/12 00:02:51 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."36992026/11/12 00:02:51 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."37002026/11/12 00:02:51 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."37012026/11/12 00:02:51 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."37022026/11/12 00:02:51 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."37032026/11/12 00:02:51 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."37042026/11/12 00:02:51 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."37052026/11/12 00:02:51 ERROR failed to remove object object=realisations/sha256:aaaaaaaa!out.doi error="Access Denied."37062026/11/12 00:02:51 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."37072026/11/12 00:02:51 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."37082026/11/12 00:02:51 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."37092026/11/12 00:02:51 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."37102026/11/12 00:02:51 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."37112026/11/12 00:02:51 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."37122026/11/12 00:02:51 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037132026/11/12 00:02:51 INFO Vacuumed table table=pending_closures37142026/11/12 00:02:51 INFO Aborted multipart uploads count=0 kept=037152026/11/12 00:02:52 WARN Force mode enabled - objects will be deleted immediately without grace period37162026/11/12 00:02:52 INFO Vacuumed table table=pending_objects37172026/11/12 00:02:52 INFO Vacuumed table table=multipart_uploads37182026/11/12 00:02:52 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037192026/11/12 00:02:52 INFO Vacuumed table table=closures37202026/11/12 00:02:52 INFO Vacuumed table table=pending_closures37212026/11/12 00:02:52 INFO Received uploads request method=POST path=/37222026/11/12 00:02:52 INFO Vacuumed table table=pending_objects37232026/11/12 00:02:52 INFO Vacuumed table table=objects37242026/11/12 00:02:52 INFO Vacuumed table table=multipart_uploads37252026/11/12 00:02:52 INFO Received push request method=POST path=/37262026/11/12 00:02:52 INFO Vacuumed table table=closures37272026/11/12 00:02:52 INFO Vacuumed table table=objects37282026/11/12 00:02:52 INFO Received push request method=POST path=/37292026/11/12 00:02:52 INFO Received complete upload request method=POST path=/37302026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=037312026/11/12 00:02:52 INFO Completed upload id=29 objects=5 duration=1.546775ms37322026/11/12 00:02:52 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037332026/11/12 00:02:52 INFO Vacuumed table table=pending_closures37342026/11/12 00:02:52 INFO Vacuumed table table=pending_objects37352026/11/12 00:02:52 INFO Vacuumed table table=multipart_uploads37362026/11/12 00:02:52 INFO Vacuumed table table=closures37372026/11/12 00:02:52 INFO Received push request method=POST path=/37382026/11/12 00:02:52 INFO Vacuumed table table=objects37392026/11/12 00:02:52 INFO Received complete push request method=POST path=/37402026/11/12 00:02:52 INFO Received complete push request method=POST path=/37412026/11/12 00:02:52 INFO Received complete upload request method=POST path=/37422026/11/12 00:02:52 INFO Received uploads request method=POST path=/37432026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=037442026/11/12 00:02:52 INFO Completed push id=29 objects=23 duration=1.659616ms37452026/11/12 00:02:52 INFO Completed upload id=9 objects=7 duration=2.996308ms37462026/11/12 00:02:52 INFO Completed push id=8 objects=6 duration=4.16454ms37472026/11/12 00:02:52 INFO Received complete upload request method=POST path=/37482026/11/12 00:02:52 INFO Completed upload id=8 objects=6 duration=1.781317ms37492026/11/12 00:02:52 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."37502026/11/12 00:02:52 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."37512026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=037522026/11/12 00:02:52 WARN Force mode enabled - objects will be deleted immediately without grace period37532026/11/12 00:02:52 INFO Received complete push request method=POST path=/37542026/11/12 00:02:52 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."37552026/11/12 00:02:52 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."37562026/11/12 00:02:52 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."37572026/11/12 00:02:52 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."37582026/11/12 00:02:52 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."37592026/11/12 00:02:52 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."37602026/11/12 00:02:52 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."37612026/11/12 00:02:52 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."37622026/11/12 00:02:52 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."37632026/11/12 00:02:52 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."37642026/11/12 00:02:52 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."37652026/11/12 00:02:52 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."37662026/11/12 00:02:52 INFO Completed push id=30 objects=6 duration=1.466694ms37672026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=037682026/11/12 00:02:52 INFO Received complete push request method=POST path=/37692026/11/12 00:02:52 WARN Force mode enabled - objects will be deleted immediately without grace period37702026/11/12 00:02:52 INFO Completed push id=4 objects=6 duration=2.020619ms37712026/11/12 00:02:52 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037722026/11/12 00:02:52 INFO Vacuumed table table=pending_closures37732026/11/12 00:02:52 INFO Received complete push request method=POST path=/37742026/11/12 00:02:52 INFO Received complete upload request method=POST path=/37752026/11/12 00:02:52 INFO Vacuumed table table=pending_objects37762026/11/12 00:02:52 INFO Vacuumed table table=multipart_uploads37772026/11/12 00:02:52 INFO Vacuumed table table=closures37782026/11/12 00:02:52 INFO Vacuumed table table=objects37792026/11/12 00:02:52 INFO Completed upload id=33 objects=5 duration=2.966368ms37802026/11/12 00:02:52 INFO Received push request method=POST path=/37812026/11/12 00:02:52 INFO Completed push id=11 objects=11 duration=4.100179ms37822026/11/12 00:02:52 INFO Received uploads request method=POST path=/37832026/11/12 00:02:52 INFO Received push request method=POST path=/37842026/11/12 00:02:52 INFO Received uploads request method=POST path=/37852026/11/12 00:02:52 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=16 objects-failed-to-delete=037862026/11/12 00:02:52 INFO Vacuumed table table=pending_closures37872026/11/12 00:02:52 INFO Received complete push request method=POST path=/37882026/11/12 00:02:52 INFO Vacuumed table table=pending_objects37892026/11/12 00:02:52 INFO Completed push id=33 objects=3 duration=1.601356ms37902026/11/12 00:02:52 INFO Vacuumed table table=multipart_uploads37912026/11/12 00:02:52 INFO Vacuumed table table=closures37922026/11/12 00:02:52 INFO Received push request method=POST path=/37932026/11/12 00:02:52 INFO Vacuumed table table=objects37942026/11/12 00:02:52 INFO Received push request method=POST path=/37952026/11/12 00:02:52 INFO Received complete push request method=POST path=/37962026/11/12 00:02:52 INFO Completed push id=17 objects=22 duration=2.526404ms37972026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=037982026/11/12 00:02:52 WARN Force mode enabled - objects will be deleted immediately without grace period37992026/11/12 00:02:52 INFO Received complete upload request method=POST path=/38002026/11/12 00:02:52 INFO Received uploads request method=POST path=/38012026/11/12 00:02:52 INFO Completed upload id=13 objects=8 duration=1.579575ms38022026/11/12 00:02:52 INFO Received push request method=POST path=/38032026/11/12 00:02:52 INFO Received uploads request method=POST path=/38042026/11/12 00:02:52 INFO Received push request method=POST path=/38052026/11/12 00:02:52 INFO Received complete push request method=POST path=/38062026/11/12 00:02:52 INFO Completed push id=34 objects=6 duration=1.07689ms38072026/11/12 00:02:52 INFO Received complete upload request method=POST path=/38082026/11/12 00:02:52 INFO Received uploads request method=POST path=/38092026/11/12 00:02:52 INFO Completed upload id=12 objects=9 duration=1.363033ms38102026/11/12 00:02:52 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=3 objects-failed-to-delete=038112026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=038122026/11/12 00:02:52 INFO Received push request method=POST path=/38132026/11/12 00:02:52 INFO Received push request method=POST path=/38142026/11/12 00:02:52 INFO Vacuumed table table=pending_closures38152026/11/12 00:02:52 INFO Received uploads request method=POST path=/38162026/11/12 00:02:52 INFO Vacuumed table table=pending_objects38172026/11/12 00:02:52 INFO Received complete push request method=POST path=/38182026/11/12 00:02:52 INFO Vacuumed table table=multipart_uploads38192026/11/12 00:02:52 INFO Vacuumed table table=closures38202026/11/12 00:02:52 INFO Vacuumed table table=objects38212026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=038222026/11/12 00:02:52 INFO Completed push id=4 objects=6 duration=2.792867ms38232026/11/12 00:02:52 WARN Force mode enabled - objects will be deleted immediately without grace period38242026/11/12 00:02:52 INFO Received complete upload request method=POST path=/38252026/11/12 00:02:52 INFO Received complete push request method=POST path=/38262026/11/12 00:02:52 INFO Completed push id=35 objects=8 duration=1.005689ms38272026/11/12 00:02:52 INFO Received complete upload request method=POST path=/38282026/11/12 00:02:52 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."38292026/11/12 00:02:52 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."38302026/11/12 00:02:52 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."38312026/11/12 00:02:52 INFO Received uploads request method=POST path=/38322026/11/12 00:02:52 INFO Completed upload id=21 objects=25 duration=1.735416ms38332026/11/12 00:02:52 INFO Completed upload id=6 objects=24 duration=3.631334ms38342026/11/12 00:02:52 INFO Received push request method=POST path=/38352026/11/12 00:02:52 INFO Received uploads request method=POST path=/38362026/11/12 00:02:52 INFO Received complete push request method=POST path=/38372026/11/12 00:02:52 INFO Received uploads request method=POST path=/38382026/11/12 00:02:52 INFO Completed push id=9 objects=24 duration=1.687216ms38392026/11/12 00:02:52 INFO Received complete upload request method=POST path=/38402026/11/12 00:02:52 INFO Received push request method=POST path=/38412026/11/12 00:02:52 INFO Completed upload id=21 objects=7 duration=2.807387ms38422026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=038432026/11/12 00:02:52 INFO Received complete upload request method=POST path=/38442026/11/12 00:02:52 INFO Received push request method=POST path=/38452026/11/12 00:02:52 INFO Completed upload id=36 objects=9 duration=905.949µs38462026/11/12 00:02:52 INFO Received uploads request method=POST path=/38472026/11/12 00:02:52 INFO Received uploads request method=POST path=/38482026/11/12 00:02:52 INFO Received push request method=POST path=/38492026/11/12 00:02:52 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=038502026/11/12 00:02:52 INFO Vacuumed table table=pending_closures38512026/11/12 00:02:52 INFO Vacuumed table table=pending_objects38522026/11/12 00:02:52 INFO Vacuumed table table=multipart_uploads38532026/11/12 00:02:52 INFO Vacuumed table table=closures38542026/11/12 00:02:52 INFO Received uploads request method=POST path=/38552026/11/12 00:02:52 INFO Received complete upload request method=POST path=/38562026/11/12 00:02:52 INFO Vacuumed table table=objects38572026/11/12 00:02:52 INFO Received complete upload request method=POST path=/38582026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=038592026/11/12 00:02:52 WARN Force mode enabled - objects will be deleted immediately without grace period38602026/11/12 00:02:52 INFO Completed upload id=9 objects=2 duration=1.519634ms38612026/11/12 00:02:52 INFO Received uploads request method=POST path=/38622026/11/12 00:02:52 INFO Completed upload id=31 objects=3 duration=2.428983ms38632026/11/12 00:02:52 INFO Received push request method=POST path=/38642026/11/12 00:02:52 INFO Received complete push request method=POST path=/38652026/11/12 00:02:52 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038662026/11/12 00:02:52 INFO Completed push id=8 objects=6 duration=2.697126ms38672026/11/12 00:02:52 INFO Vacuumed table table=pending_closures38682026/11/12 00:02:52 INFO Vacuumed table table=pending_objects38692026/11/12 00:02:52 INFO Vacuumed table table=multipart_uploads38702026/11/12 00:02:52 INFO Received push request method=POST path=/38712026/11/12 00:02:52 INFO Received uploads request method=POST path=/38722026/11/12 00:02:52 INFO Vacuumed table table=closures38732026/11/12 00:02:52 INFO Received push request method=POST path=/38742026/11/12 00:02:52 INFO Received push request method=POST path=/38752026/11/12 00:02:52 INFO Vacuumed table table=objects38762026/11/12 00:02:52 INFO Received complete push request method=POST path=/38772026/11/12 00:02:52 INFO Completed push id=8 objects=3 duration=1.591975ms38782026/11/12 00:02:52 INFO Received uploads request method=POST path=/38792026/11/12 00:02:52 INFO Received complete push request method=POST path=/38802026/11/12 00:02:52 INFO Received complete upload request method=POST path=/38812026/11/12 00:02:52 INFO Completed upload id=22 objects=2 duration=1.007809ms38822026/11/12 00:02:52 INFO Completed push id=10 objects=2 duration=2.265462ms38832026/11/12 00:02:52 INFO Received complete upload request method=POST path=/38842026/11/12 00:02:52 INFO Completed upload id=12 objects=6 duration=1.216852ms38852026/11/12 00:02:52 INFO Received complete push request method=POST path=/38862026/11/12 00:02:52 INFO Received complete upload request method=POST path=/38872026/11/12 00:02:52 INFO Completed push id=19 objects=14 duration=1.395073ms38882026/11/12 00:02:52 INFO Completed upload id=9 objects=3 duration=1.015789ms38892026/11/12 00:02:52 INFO Received push request method=POST path=/38902026/11/12 00:02:52 INFO Received complete upload request method=POST path=/38912026/11/12 00:02:52 INFO Received complete upload request method=POST path=/38922026/11/12 00:02:52 INFO Received push request method=POST path=/38932026/11/12 00:02:52 INFO Received complete upload request method=POST path=/38942026/11/12 00:02:52 INFO Completed upload id=10 objects=10 duration=1.127831ms38952026/11/12 00:02:52 INFO Completed upload id=5 objects=7 duration=2.580944ms38962026/11/12 00:02:52 INFO Completed upload id=10 objects=2 duration=2.565464ms38972026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=038982026/11/12 00:02:52 INFO Received complete push request method=POST path=/38992026/11/12 00:02:52 INFO Received complete push request method=POST path=/39002026/11/12 00:02:52 INFO Completed push id=5 objects=28 duration=2.10154ms39012026/11/12 00:02:52 WARN Force mode enabled - objects will be deleted immediately without grace period39022026/11/12 00:02:52 INFO Received complete upload request method=POST path=/39032026/11/12 00:02:52 INFO Completed push id=17 objects=3 duration=1.773697ms39042026/11/12 00:02:52 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039052026/11/12 00:02:52 INFO Received uploads request method=POST path=/39062026/11/12 00:02:52 INFO Received push request method=POST path=/39072026/11/12 00:02:52 INFO Completed upload id=12 objects=21 duration=1.180851ms39082026/11/12 00:02:52 INFO Received uploads request method=POST path=/39092026/11/12 00:02:52 INFO Received complete upload request method=POST path=/39102026/11/12 00:02:52 INFO Vacuumed table table=pending_closures39112026/11/12 00:02:52 INFO Received push request method=POST path=/39122026/11/12 00:02:52 INFO Received push request method=POST path=/39132026/11/12 00:02:52 INFO Vacuumed table table=pending_objects39142026/11/12 00:02:52 INFO Completed upload id=18 objects=22 duration=1.485434ms39152026/11/12 00:02:52 INFO Vacuumed table table=multipart_uploads39162026/11/12 00:02:52 INFO Received uploads request method=POST path=/39172026/11/12 00:02:52 INFO Vacuumed table table=closures39182026/11/12 00:02:52 INFO Vacuumed table table=objects39192026/11/12 00:02:52 INFO Received complete upload request method=POST path=/39202026/11/12 00:02:52 INFO Completed upload id=6 objects=9 duration=1.600915ms39212026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=039222026/11/12 00:02:52 INFO Received complete push request method=POST path=/39232026/11/12 00:02:52 INFO Received uploads request method=POST path=/39242026/11/12 00:02:52 INFO Completed push id=10 objects=2 duration=2.082299ms39252026/11/12 00:02:52 INFO Received push request method=POST path=/39262026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=039272026/11/12 00:02:52 INFO Received complete upload request method=POST path=/39282026/11/12 00:02:52 INFO Received complete upload request method=POST path=/39292026/11/12 00:02:52 INFO Received uploads request method=POST path=/39302026/11/12 00:02:52 INFO Received complete upload request method=POST path=/39312026/11/12 00:02:52 INFO Received complete push request method=POST path=/39322026/11/12 00:02:52 INFO Completed upload id=21 objects=8 duration=1.098251ms39332026/11/12 00:02:52 INFO Completed upload id=13 objects=10 duration=1.405653ms39342026/11/12 00:02:52 INFO Received uploads request method=POST path=/39352026/11/12 00:02:52 INFO Completed push id=23 objects=5 duration=1.715157ms39362026/11/12 00:02:52 INFO Completed upload id=18 objects=2 duration=2.142021ms39372026/11/12 00:02:52 INFO Received complete upload request method=POST path=/39382026/11/12 00:02:52 INFO Completed upload id=20 objects=6 duration=1.427414ms39392026/11/12 00:02:52 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."39402026/11/12 00:02:52 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."39412026/11/12 00:02:52 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=039422026/11/12 00:02:52 INFO Vacuumed table table=pending_closures39432026/11/12 00:02:52 INFO Received complete upload request method=POST path=/39442026/11/12 00:02:52 INFO Vacuumed table table=pending_objects39452026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=039462026/11/12 00:02:52 INFO Vacuumed table table=multipart_uploads39472026/11/12 00:02:52 INFO Completed upload id=27 objects=4 duration=1.842437ms39482026/11/12 00:02:52 INFO Vacuumed table table=closures39492026/11/12 00:02:52 INFO Received complete upload request method=POST path=/39502026/11/12 00:02:52 INFO Received push request method=POST path=/39512026/11/12 00:02:52 WARN Force mode enabled - objects will be deleted immediately without grace period39522026/11/12 00:02:52 INFO Received push request method=POST path=/39532026/11/12 00:02:52 INFO Received complete upload request method=POST path=/39542026/11/12 00:02:52 INFO Received complete push request method=POST path=/39552026/11/12 00:02:52 INFO Received push request method=POST path=/39562026/11/12 00:02:52 INFO Received uploads request method=POST path=/39572026/11/12 00:02:52 INFO Completed upload id=24 objects=3 duration=2.132681ms39582026/11/12 00:02:52 INFO Completed upload id=17 objects=5 duration=1.763917ms39592026/11/12 00:02:52 INFO Vacuumed table table=objects39602026/11/12 00:02:52 INFO Received complete upload request method=POST path=/39612026/11/12 00:02:52 INFO Completed push id=13 objects=23 duration=2.896288ms39622026/11/12 00:02:52 INFO Completed upload id=22 objects=7 duration=1.091831ms39632026/11/12 00:02:52 INFO Received uploads request method=POST path=/39642026/11/12 00:02:52 INFO Received complete push request method=POST path=/39652026/11/12 00:02:52 INFO Received complete push request method=POST path=/39662026/11/12 00:02:52 INFO Completed push id=13 objects=11 duration=2.300381ms39672026/11/12 00:02:52 INFO Completed push id=25 objects=4 duration=930.689µs39682026/11/12 00:02:52 INFO Received complete upload request method=POST path=/39692026/11/12 00:02:52 INFO Received push request method=POST path=/39702026/11/12 00:02:52 INFO Received complete push request method=POST path=/39712026/11/12 00:02:52 INFO Received uploads request method=POST path=/39722026/11/12 00:02:52 INFO Completed upload id=3 objects=5 duration=2.106901ms39732026/11/12 00:02:52 INFO Completed push id=24 objects=5 duration=1.106611ms39742026/11/12 00:02:52 INFO Received push request method=POST path=/39752026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=039762026/11/12 00:02:52 INFO Received uploads request method=POST path=/39772026/11/12 00:02:52 INFO Received uploads request method=POST path=/39782026/11/12 00:02:52 INFO Received complete push request method=POST path=/39792026/11/12 00:02:52 INFO Completed push id=19 objects=3 duration=1.220052ms39802026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=039812026/11/12 00:02:52 INFO Received complete upload request method=POST path=/39822026/11/12 00:02:52 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=039832026/11/12 00:02:52 INFO Completed upload id=13 objects=7 duration=1.625135ms39842026/11/12 00:02:52 INFO Received complete push request method=POST path=/39852026/11/12 00:02:52 INFO Vacuumed table table=pending_closures39862026/11/12 00:02:52 INFO Vacuumed table table=pending_objects39872026/11/12 00:02:52 INFO Vacuumed table table=multipart_uploads39882026/11/12 00:02:52 INFO Received complete upload request method=POST path=/39892026/11/12 00:02:52 INFO Completed push id=12 objects=3 duration=2.05464ms39902026/11/12 00:02:52 INFO Vacuumed table table=closures39912026/11/12 00:02:52 INFO Completed upload id=25 objects=8 duration=949.409µs39922026/11/12 00:02:52 INFO Vacuumed table table=objects39932026/11/12 00:02:52 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=039942026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=039952026/11/12 00:02:52 INFO Vacuumed table table=pending_closures39962026/11/12 00:02:52 WARN Force mode enabled - objects will be deleted immediately without grace period39972026/11/12 00:02:52 INFO Vacuumed table table=pending_objects39982026/11/12 00:02:52 INFO Vacuumed table table=multipart_uploads39992026/11/12 00:02:52 INFO Received uploads request method=POST path=/40002026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=040012026/11/12 00:02:52 INFO Vacuumed table table=closures40022026/11/12 00:02:52 INFO Received complete upload request method=POST path=/40032026/11/12 00:02:52 INFO Vacuumed table table=objects40042026/11/12 00:02:52 INFO Received complete upload request method=POST path=/40052026/11/12 00:02:52 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=11 objects-failed-to-delete=040062026/11/12 00:02:52 INFO Completed upload id=21 objects=2 duration=1.537174ms40072026/11/12 00:02:52 INFO Completed upload id=20 objects=4 duration=1.07275ms40082026/11/12 00:02:52 INFO Vacuumed table table=pending_closures40092026/11/12 00:02:52 INFO Received push request method=POST path=/40102026/11/12 00:02:52 INFO Received uploads request method=POST path=/40112026/11/12 00:02:52 INFO Received complete push request method=POST path=/40122026/11/12 00:02:52 INFO Vacuumed table table=pending_objects40132026/11/12 00:02:52 INFO Vacuumed table table=multipart_uploads40142026/11/12 00:02:52 INFO Received complete push request method=POST path=/40152026/11/12 00:02:52 INFO Vacuumed table table=closures40162026/11/12 00:02:52 INFO Completed push id=26 objects=4 duration=919.369µs40172026/11/12 00:02:52 INFO Vacuumed table table=objects40182026/11/12 00:02:52 INFO Completed push id=11 objects=8 duration=3.434733ms40192026/11/12 00:02:52 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=040202026/11/12 00:02:52 INFO Received complete push request method=POST path=/40212026/11/12 00:02:52 INFO Received complete push request method=POST path=/40222026/11/12 00:02:52 INFO Vacuumed table table=pending_closures40232026/11/12 00:02:52 INFO Completed push id=20 objects=4 duration=1.527795ms40242026/11/12 00:02:52 INFO Completed push id=26 objects=23 duration=1.839937ms40252026/11/12 00:02:52 INFO Vacuumed table table=pending_objects40262026/11/12 00:02:52 INFO Vacuumed table table=multipart_uploads40272026/11/12 00:02:52 INFO Vacuumed table table=closures40282026/11/12 00:02:52 INFO Vacuumed table table=objects40292026/11/12 00:02:52 INFO Received complete upload request method=POST path=/40302026/11/12 00:02:52 INFO Received uploads request method=POST path=/40312026/11/12 00:02:52 INFO Received push request method=POST path=/40322026/11/12 00:02:52 INFO Completed upload id=23 objects=7 duration=999.29µs40332026/11/12 00:02:52 INFO Received complete push request method=POST path=/40342026/11/12 00:02:52 INFO Received uploads request method=POST path=/40352026/11/12 00:02:52 INFO Completed push id=14 objects=3 duration=1.191671ms40362026/11/12 00:02:52 INFO Received push request method=POST path=/40372026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=040382026/11/12 00:02:52 INFO Received complete push request method=POST path=/40392026/11/12 00:02:52 INFO Completed push id=30 objects=2 duration=1.053211ms40402026/11/12 00:02:52 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040412026/11/12 00:02:52 INFO Vacuumed table table=pending_closures40422026/11/12 00:02:52 INFO Vacuumed table table=pending_objects40432026/11/12 00:02:52 INFO Vacuumed table table=multipart_uploads40442026/11/12 00:02:52 INFO Vacuumed table table=closures40452026/11/12 00:02:52 INFO Vacuumed table table=objects40462026/11/12 00:02:52 INFO Received uploads request method=POST path=/40472026/11/12 00:02:52 INFO Received push request method=POST path=/40482026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=040492026/11/12 00:02:52 INFO Received push request method=POST path=/40502026/11/12 00:02:52 INFO Received push request method=POST path=/40512026/11/12 00:02:52 INFO Received complete push request method=POST path=/40522026/11/12 00:02:52 WARN Force mode enabled - objects will be deleted immediately without grace period40532026/11/12 00:02:52 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040542026/11/12 00:02:52 INFO Completed push id=2 objects=10 duration=2.722546ms40552026/11/12 00:02:52 INFO Vacuumed table table=pending_closures40562026/11/12 00:02:52 INFO Vacuumed table table=pending_objects40572026/11/12 00:02:52 INFO Received complete upload request method=POST path=/40582026/11/12 00:02:52 INFO Vacuumed table table=multipart_uploads40592026/11/12 00:02:52 INFO Received uploads request method=POST path=/40602026/11/12 00:02:52 INFO Vacuumed table table=closures40612026/11/12 00:02:52 INFO Completed upload id=22 objects=8 duration=1.347893ms40622026/11/12 00:02:52 INFO Vacuumed table table=objects40632026/11/12 00:02:52 INFO Received complete push request method=POST path=/40642026/11/12 00:02:52 INFO Completed push id=27 objects=15 duration=1.155631ms40652026/11/12 00:02:52 INFO Received complete push request method=POST path=/40662026/11/12 00:02:52 INFO Completed push id=32 objects=7 duration=1.249912ms40672026/11/12 00:02:52 INFO Received complete push request method=POST path=/40682026/11/12 00:02:52 INFO Received complete push request method=POST path=/40692026/11/12 00:02:52 INFO Received complete upload request method=POST path=/40702026/11/12 00:02:52 INFO Completed upload id=24 objects=18 duration=1.382613ms40712026/11/12 00:02:52 INFO Completed push id=13 objects=6 duration=3.12213ms40722026/11/12 00:02:52 INFO Received complete push request method=POST path=/40732026/11/12 00:02:52 INFO Completed push id=14 objects=4 duration=4.418703ms40742026/11/12 00:02:52 INFO Completed push id=23 objects=5 duration=933.148µs40752026/11/12 00:02:52 INFO Received uploads request method=POST path=/40762026/11/12 00:02:52 INFO Received push request method=POST path=/40772026/11/12 00:02:52 INFO Received complete upload request method=POST path=/40782026/11/12 00:02:52 INFO Received push request method=POST path=/40792026/11/12 00:02:52 INFO Received push request method=POST path=/40802026/11/12 00:02:52 INFO Received complete upload request method=POST path=/40812026/11/12 00:02:52 INFO Completed upload id=16 objects=23 duration=1.500614ms40822026/11/12 00:02:52 INFO Completed upload id=33 objects=6 duration=1.02005ms40832026/11/12 00:02:52 INFO Received complete push request method=POST path=/40842026/11/12 00:02:52 INFO Completed push id=25 objects=2 duration=1.990559ms40852026/11/12 00:02:52 INFO Received uploads request method=POST path=/40862026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=040872026/11/12 00:02:52 INFO Received complete upload request method=POST path=/40882026/11/12 00:02:52 INFO Completed upload id=24 objects=8 duration=882.588µs40892026/11/12 00:02:52 WARN Force mode enabled - objects will be deleted immediately without grace period40902026/11/12 00:02:52 INFO Received push request method=POST path=/40912026/11/12 00:02:52 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040922026/11/12 00:02:52 INFO Vacuumed table table=pending_closures40932026/11/12 00:02:52 INFO Received push request method=POST path=/40942026/11/12 00:02:52 INFO Vacuumed table table=pending_objects40952026/11/12 00:02:52 INFO Vacuumed table table=multipart_uploads40962026/11/12 00:02:52 INFO Received complete push request method=POST path=/40972026/11/12 00:02:52 INFO Received complete upload request method=POST path=/40982026/11/12 00:02:52 INFO Vacuumed table table=closures40992026/11/12 00:02:52 INFO Completed upload id=15 objects=6 duration=1.08291ms41002026/11/12 00:02:52 INFO Vacuumed table table=objects41012026/11/12 00:02:52 INFO Received complete upload request method=POST path=/41022026/11/12 00:02:52 INFO Received uploads request method=POST path=/41032026/11/12 00:02:52 INFO Completed push id=8 objects=7 duration=2.046959ms41042026/11/12 00:02:52 INFO Received complete push request method=POST path=/41052026/11/12 00:02:52 INFO Completed upload id=34 objects=3 duration=968.409µs41062026/11/12 00:02:52 INFO Completed push id=27 objects=4 duration=1.03519ms41072026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=041082026/11/12 00:02:52 INFO Received complete push request method=POST path=/41092026/11/12 00:02:52 INFO Received push request method=POST path=/41102026/11/12 00:02:52 WARN Force mode enabled - objects will be deleted immediately without grace period41112026/11/12 00:02:52 INFO Received complete upload request method=POST path=/41122026/11/12 00:02:52 INFO Completed push id=28 objects=19 duration=1.573535ms41132026/11/12 00:02:52 INFO Completed upload id=28 objects=5 duration=884.628µs41142026/11/12 00:02:52 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041152026/11/12 00:02:52 INFO Received uploads request method=POST path=/41162026/11/12 00:02:52 INFO Vacuumed table table=pending_closures41172026/11/12 00:02:52 INFO Received uploads request method=POST path=/41182026/11/12 00:02:52 INFO Vacuumed table table=pending_objects41192026/11/12 00:02:52 INFO Vacuumed table table=multipart_uploads41202026/11/12 00:02:52 INFO Received uploads request method=POST path=/41212026/11/12 00:02:52 INFO Vacuumed table table=closures41222026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=041232026/11/12 00:02:52 INFO Vacuumed table table=objects41242026/11/12 00:02:52 INFO Received complete upload request method=POST path=/41252026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=041262026/11/12 00:02:52 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041272026/11/12 00:02:52 INFO Completed upload id=15 objects=4 duration=2.568685ms41282026/11/12 00:02:52 INFO Vacuumed table table=pending_closures41292026/11/12 00:02:52 WARN Force mode enabled - objects will be deleted immediately without grace period41302026/11/12 00:02:52 INFO Vacuumed table table=pending_objects41312026/11/12 00:02:52 INFO Vacuumed table table=multipart_uploads41322026/11/12 00:02:52 INFO Vacuumed table table=closures41332026/11/12 00:02:52 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041342026/11/12 00:02:52 INFO Vacuumed table table=objects41352026/11/12 00:02:52 INFO Received complete push request method=POST path=/41362026/11/12 00:02:52 INFO Vacuumed table table=pending_closures41372026/11/12 00:02:52 INFO Completed push id=24 objects=24 duration=3.270052ms41382026/11/12 00:02:52 INFO Vacuumed table table=pending_objects41392026/11/12 00:02:52 INFO Vacuumed table table=multipart_uploads41402026/11/12 00:02:52 INFO Vacuumed table table=closures41412026/11/12 00:02:52 INFO Received complete push request method=POST path=/41422026/11/12 00:02:52 INFO Vacuumed table table=objects41432026/11/12 00:02:52 INFO Received complete upload request method=POST path=/41442026/11/12 00:02:52 INFO Completed push id=28 objects=26 duration=1.791657ms41452026/11/12 00:02:52 INFO Received push request method=POST path=/41462026/11/12 00:02:52 INFO Received push request method=POST path=/41472026/11/12 00:02:52 INFO Completed upload id=30 objects=5 duration=968.97µs41482026/11/12 00:02:52 INFO Received complete push request method=POST path=/41492026/11/12 00:02:52 INFO Received complete upload request method=POST path=/41502026/11/12 00:02:52 INFO Received complete push request method=POST path=/41512026/11/12 00:02:52 INFO Received push request method=POST path=/41522026/11/12 00:02:52 INFO Completed push id=29 objects=10 duration=1.307413ms41532026/11/12 00:02:52 INFO Completed upload id=17 objects=7 duration=1.541955ms41542026/11/12 00:02:52 INFO Completed push id=17 objects=6 duration=2.171841ms41552026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=041562026/11/12 00:02:52 INFO Received complete upload request method=POST path=/41572026/11/12 00:02:52 INFO Completed upload id=9 objects=16 duration=2.0132ms41582026/11/12 00:02:52 WARN Force mode enabled - objects will be deleted immediately without grace period41592026/11/12 00:02:52 INFO Received complete push request method=POST path=/41602026/11/12 00:02:52 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041612026/11/12 00:02:52 INFO Completed push id=30 objects=3 duration=1.255052ms41622026/11/12 00:02:52 INFO Vacuumed table table=pending_closures41632026/11/12 00:02:52 INFO Received uploads request method=POST path=/41642026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=041652026/11/12 00:02:52 INFO Received uploads request method=POST path=/41662026/11/12 00:02:52 INFO Vacuumed table table=pending_objects41672026/11/12 00:02:52 INFO Vacuumed table table=multipart_uploads41682026/11/12 00:02:52 INFO Vacuumed table table=closures41692026/11/12 00:02:52 INFO Vacuumed table table=objects41702026/11/12 00:02:52 INFO Received complete push request method=POST path=/41712026/11/12 00:02:52 INFO Completed push id=31 objects=3 duration=1.327613ms41722026/11/12 00:02:52 WARN Force mode enabled - objects will be deleted immediately without grace period41732026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=041742026/11/12 00:02:52 INFO Received push request method=POST path=/41752026/11/12 00:02:52 INFO Received complete upload request method=POST path=/41762026/11/12 00:02:52 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041772026/11/12 00:02:52 INFO Completed upload id=30 objects=6 duration=1.354473ms41782026/11/12 00:02:52 INFO Vacuumed table table=pending_closures41792026/11/12 00:02:52 INFO Received complete push request method=POST path=/41802026/11/12 00:02:52 INFO Vacuumed table table=pending_objects41812026/11/12 00:02:52 INFO Vacuumed table table=multipart_uploads41822026/11/12 00:02:52 INFO Vacuumed table table=closures41832026/11/12 00:02:52 INFO Completed push id=32 objects=19 duration=1.517554ms41842026/11/12 00:02:52 INFO Vacuumed table table=objects41852026/11/12 00:02:52 INFO Received complete upload request method=POST path=/41862026/11/12 00:02:52 INFO Received complete push request method=POST path=/41872026/11/12 00:02:52 INFO Received complete push request method=POST path=/41882026/11/12 00:02:52 INFO Completed upload id=16 objects=3 duration=3.513993ms41892026/11/12 00:02:52 INFO Completed push id=4 objects=7 duration=3.432433ms41902026/11/12 00:02:52 INFO Completed push id=9 objects=4 duration=3.637714ms41912026/11/12 00:02:52 INFO Received uploads request method=POST path=/41922026/11/12 00:02:52 INFO Received uploads request method=POST path=/41932026/11/12 00:02:52 INFO Received push request method=POST path=/41942026/11/12 00:02:52 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."41952026/11/12 00:02:52 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."41962026/11/12 00:02:52 INFO Received complete upload request method=POST path=/41972026/11/12 00:02:52 INFO Completed upload id=31 objects=5 duration=822.648µs41982026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=041992026/11/12 00:02:52 WARN Force mode enabled - objects will be deleted immediately without grace period42002026/11/12 00:02:52 INFO Received uploads request method=POST path=/42012026/11/12 00:02:52 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042022026/11/12 00:02:52 INFO Received complete push request method=POST path=/42032026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=042042026/11/12 00:02:52 INFO Vacuumed table table=pending_closures42052026/11/12 00:02:52 INFO Completed push id=32 objects=6 duration=1.237312ms42062026/11/12 00:02:52 INFO Received complete push request method=POST path=/42072026/11/12 00:02:52 INFO Vacuumed table table=pending_objects42082026/11/12 00:02:52 INFO Completed push id=10 objects=3 duration=2.051039ms42092026/11/12 00:02:52 INFO Vacuumed table table=multipart_uploads42102026/11/12 00:02:52 INFO Received uploads request method=POST path=/42112026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=042122026/11/12 00:02:52 INFO Received complete upload request method=POST path=/42132026/11/12 00:02:52 INFO Vacuumed table table=closures42142026/11/12 00:02:52 INFO Completed upload id=34 objects=3 duration=866.148µs42152026/11/12 00:02:52 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042162026/11/12 00:02:52 WARN Force mode enabled - objects will be deleted immediately without grace period42172026/11/12 00:02:52 INFO Vacuumed table table=objects42182026/11/12 00:02:52 INFO Received uploads request method=POST path=/42192026/11/12 00:02:52 INFO Vacuumed table table=pending_closures42202026/11/12 00:02:52 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042212026/11/12 00:02:52 INFO Vacuumed table table=pending_objects42222026/11/12 00:02:52 INFO Vacuumed table table=pending_closures42232026/11/12 00:02:52 INFO Vacuumed table table=multipart_uploads42242026/11/12 00:02:52 INFO Vacuumed table table=closures42252026/11/12 00:02:52 INFO Vacuumed table table=pending_objects42262026/11/12 00:02:52 INFO Vacuumed table table=objects42272026/11/12 00:02:52 INFO Vacuumed table table=multipart_uploads42282026/11/12 00:02:52 INFO Vacuumed table table=closures42292026/11/12 00:02:52 INFO Vacuumed table table=objects42302026/11/12 00:02:52 INFO Received complete upload request method=POST path=/42312026/11/12 00:02:52 INFO Completed upload id=33 objects=7 duration=884.248µs42322026/11/12 00:02:52 INFO Received uploads request method=POST path=/42332026/11/12 00:02:52 INFO Received complete upload request method=POST path=/42342026/11/12 00:02:52 INFO Completed upload id=14 objects=6 duration=1.05619ms42352026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=042362026/11/12 00:02:52 INFO Received complete push request method=POST path=/42372026/11/12 00:02:52 WARN Force mode enabled - objects will be deleted immediately without grace period42382026/11/12 00:02:52 INFO Received uploads request method=POST path=/42392026/11/12 00:02:52 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042402026/11/12 00:02:52 INFO Completed push id=16 objects=16 duration=4.057799ms42412026/11/12 00:02:52 INFO Vacuumed table table=pending_closures42422026/11/12 00:02:52 INFO Received complete upload request method=POST path=/42432026/11/12 00:02:52 INFO Received push request method=POST path=/42442026/11/12 00:02:52 INFO Vacuumed table table=pending_objects42452026/11/12 00:02:52 INFO Vacuumed table table=multipart_uploads42462026/11/12 00:02:52 INFO Completed upload id=34 objects=3 duration=1.467994ms42472026/11/12 00:02:52 INFO Vacuumed table table=closures42482026/11/12 00:02:52 INFO Vacuumed table table=objects42492026/11/12 00:02:52 INFO Received push request method=POST path=/42502026/11/12 00:02:52 INFO Received push request method=POST path=/42512026/11/12 00:02:52 INFO Received uploads request method=POST path=/42522026/11/12 00:02:52 INFO Received complete push request method=POST path=/42532026/11/12 00:02:52 INFO Completed push id=35 objects=11 duration=1.406214ms42542026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=042552026/11/12 00:02:52 INFO Received uploads request method=POST path=/42562026/11/12 00:02:52 WARN Force mode enabled - objects will be deleted immediately without grace period42572026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=042582026/11/12 00:02:52 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042592026/11/12 00:02:52 INFO Vacuumed table table=pending_closures42602026/11/12 00:02:52 INFO Received complete push request method=POST path=/42612026/11/12 00:02:52 INFO Vacuumed table table=pending_objects42622026/11/12 00:02:52 INFO Received complete upload request method=POST path=/42632026/11/12 00:02:52 INFO Vacuumed table table=multipart_uploads42642026/11/12 00:02:52 INFO Vacuumed table table=closures42652026/11/12 00:02:52 INFO Completed upload id=7 objects=3 duration=1.178851ms42662026/11/12 00:02:52 INFO Received uploads request method=POST path=/42672026/11/12 00:02:52 INFO Received complete upload request method=POST path=/42682026/11/12 00:02:52 INFO Vacuumed table table=objects42692026/11/12 00:02:52 INFO Completed push id=7 objects=7 duration=3.686055ms42702026/11/12 00:02:52 INFO Completed upload id=18 objects=4 duration=1.743017ms42712026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=042722026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=042732026/11/12 00:02:52 WARN Force mode enabled - objects will be deleted immediately without grace period42742026/11/12 00:02:52 WARN Force mode enabled - objects will be deleted immediately without grace period42752026/11/12 00:02:52 INFO Received uploads request method=POST path=/42762026/11/12 00:02:52 INFO Garbage collection completed failed-uploads-deleted=6 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042772026/11/12 00:02:52 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042782026/11/12 00:02:52 INFO Received uploads request method=POST path=/42792026/11/12 00:02:52 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."42802026/11/12 00:02:52 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."42812026/11/12 00:02:52 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."42822026/11/12 00:02:52 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."42832026/11/12 00:02:52 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."42842026/11/12 00:02:52 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."42852026/11/12 00:02:52 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."42862026/11/12 00:02:52 INFO Received uploads request method=POST path=/42872026/11/12 00:02:52 INFO Vacuumed table table=pending_closures42882026/11/12 00:02:52 INFO Vacuumed table table=pending_closures42892026/11/12 00:02:52 INFO Vacuumed table table=pending_objects42902026/11/12 00:02:52 INFO Vacuumed table table=multipart_uploads42912026/11/12 00:02:52 INFO Vacuumed table table=closures42922026/11/12 00:02:52 INFO Vacuumed table table=objects42932026/11/12 00:02:52 INFO Vacuumed table table=pending_objects42942026/11/12 00:02:52 INFO Vacuumed table table=multipart_uploads42952026/11/12 00:02:52 INFO Vacuumed table table=closures42962026/11/12 00:02:52 INFO Vacuumed table table=objects42972026/11/12 00:02:52 INFO Received complete push request method=POST path=/42982026/11/12 00:02:52 INFO Completed push id=18 objects=23 duration=2.291542ms42992026/11/12 00:02:52 INFO Received complete push request method=POST path=/43002026/11/12 00:02:52 INFO Received uploads request method=POST path=/43012026/11/12 00:02:52 INFO Completed push id=17 objects=6 duration=1.876458ms43022026/11/12 00:02:52 INFO Received uploads request method=POST path=/43032026/11/12 00:02:52 INFO Received complete upload request method=POST path=/43042026/11/12 00:02:52 INFO Completed upload id=18 objects=11 duration=1.518295ms43052026/11/12 00:02:52 INFO Received uploads request method=POST path=/43062026/11/12 00:02:52 INFO Received uploads request method=POST path=/43072026/11/12 00:02:52 INFO Received uploads request method=POST path=/43082026/11/12 00:02:52 INFO Received complete push request method=POST path=/43092026/11/12 00:02:52 INFO Received push request method=POST path=/43102026/11/12 00:02:52 INFO Completed push id=17 objects=22 duration=2.538204ms43112026/11/12 00:02:52 INFO Received uploads request method=POST path=/43122026/11/12 00:02:52 INFO Received complete push request method=POST path=/43132026/11/12 00:02:52 INFO Received push request method=POST path=/43142026/11/12 00:02:52 INFO Received push request method=POST path=/43152026/11/12 00:02:52 INFO Completed push id=11 objects=9 duration=2.137821ms43162026/11/12 00:02:52 INFO Received complete upload request method=POST path=/43172026/11/12 00:02:52 INFO Received push request method=POST path=/43182026/11/12 00:02:52 INFO Received push request method=POST path=/43192026/11/12 00:02:52 INFO Completed upload id=23 objects=7 duration=5.758015ms43202026/11/12 00:02:52 INFO Received push request method=POST path=/43212026/11/12 00:02:52 INFO Received complete push request method=POST path=/43222026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=043232026/11/12 00:02:52 INFO Completed push id=18 objects=6 duration=4.441262ms43242026/11/12 00:02:52 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043252026/11/12 00:02:52 INFO Vacuumed table table=pending_closures43262026/11/12 00:02:52 INFO Vacuumed table table=pending_objects43272026/11/12 00:02:52 INFO Vacuumed table table=multipart_uploads43282026/11/12 00:02:52 INFO Vacuumed table table=closures43292026/11/12 00:02:52 INFO Vacuumed table table=objects43302026/11/12 00:02:52 INFO Received complete upload request method=POST path=/43312026/11/12 00:02:52 INFO Received uploads request method=POST path=/43322026/11/12 00:02:52 INFO Received push request method=POST path=/43332026/11/12 00:02:52 INFO Completed upload id=26 objects=5 duration=1.06081ms43342026/11/12 00:02:52 INFO Received complete upload request method=POST path=/43352026/11/12 00:02:52 INFO Completed upload id=23 objects=9 duration=1.340253ms43362026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=043372026/11/12 00:02:52 INFO Received uploads request method=POST path=/43382026/11/12 00:02:52 INFO Received complete push request method=POST path=/43392026/11/12 00:02:52 INFO Received uploads request method=POST path=/43402026/11/12 00:02:52 WARN Force mode enabled - objects will be deleted immediately without grace period43412026/11/12 00:02:52 INFO Completed push id=27 objects=3 duration=2.612945ms43422026/11/12 00:02:52 INFO Received complete upload request method=POST path=/43432026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=043442026/11/12 00:02:52 INFO Received push request method=POST path=/43452026/11/12 00:02:52 INFO Completed upload id=28 objects=5 duration=2.666905ms43462026/11/12 00:02:52 INFO Received complete upload request method=POST path=/43472026/11/12 00:02:52 WARN Force mode enabled - objects will be deleted immediately without grace period43482026/11/12 00:02:52 INFO Completed upload id=7 objects=10 duration=2.182501ms43492026/11/12 00:02:52 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043502026/11/12 00:02:52 INFO Vacuumed table table=pending_closures43512026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=043522026/11/12 00:02:52 INFO Vacuumed table table=pending_objects43532026/11/12 00:02:52 INFO Vacuumed table table=multipart_uploads43542026/11/12 00:02:52 INFO Vacuumed table table=closures43552026/11/12 00:02:52 WARN Force mode enabled - objects will be deleted immediately without grace period43562026/11/12 00:02:52 INFO Vacuumed table table=objects43572026/11/12 00:02:52 INFO Received complete push request method=POST path=/43582026/11/12 00:02:52 INFO Completed push id=29 objects=17 duration=1.361613ms43592026/11/12 00:02:52 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."43602026/11/12 00:02:52 INFO Received uploads request method=POST path=/43612026/11/12 00:02:52 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."43622026/11/12 00:02:52 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."43632026/11/12 00:02:52 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."43642026/11/12 00:02:52 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."43652026/11/12 00:02:52 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."43662026/11/12 00:02:52 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."43672026/11/12 00:02:52 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."43682026/11/12 00:02:52 INFO Received uploads request method=POST path=/43692026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=043702026/11/12 00:02:52 INFO Received complete upload request method=POST path=/43712026/11/12 00:02:52 INFO Completed upload id=17 objects=6 duration=2.555444ms43722026/11/12 00:02:52 INFO Received uploads request method=POST path=/43732026/11/12 00:02:52 WARN Force mode enabled - objects will be deleted immediately without grace period43742026/11/12 00:02:52 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043752026/11/12 00:02:52 INFO Vacuumed table table=pending_closures43762026/11/12 00:02:52 INFO Vacuumed table table=pending_objects43772026/11/12 00:02:52 INFO Vacuumed table table=multipart_uploads43782026/11/12 00:02:52 INFO Vacuumed table table=closures43792026/11/12 00:02:52 INFO Vacuumed table table=objects43802026/11/12 00:02:52 INFO Received complete upload request method=POST path=/43812026/11/12 00:02:52 INFO Completed upload id=21 objects=3 duration=1.04551ms43822026/11/12 00:02:52 INFO Received push request method=POST path=/43832026/11/12 00:02:52 INFO Received push request method=POST path=/43842026/11/12 00:02:52 INFO Received push request method=POST path=/43852026/11/12 00:02:52 INFO Received push request method=POST path=/43862026/11/12 00:02:52 INFO Received complete push request method=POST path=/43872026/11/12 00:02:52 INFO Completed push id=15 objects=4 duration=1.499294ms43882026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=043892026/11/12 00:02:52 WARN Force mode enabled - objects will be deleted immediately without grace period43902026/11/12 00:02:52 INFO Received push request method=POST path=/43912026/11/12 00:02:52 INFO Received complete upload request method=POST path=/43922026/11/12 00:02:52 INFO Completed upload id=23 objects=15 duration=2.332602ms43932026/11/12 00:02:52 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=7 objects-failed-to-delete=043942026/11/12 00:02:52 INFO Vacuumed table table=pending_closures43952026/11/12 00:02:52 INFO Received complete push request method=POST path=/43962026/11/12 00:02:52 INFO Received push request method=POST path=/43972026/11/12 00:02:52 INFO Vacuumed table table=pending_objects43982026/11/12 00:02:52 INFO Vacuumed table table=multipart_uploads43992026/11/12 00:02:52 INFO Vacuumed table table=closures44002026/11/12 00:02:52 INFO Completed push id=16 objects=26 duration=1.560915ms44012026/11/12 00:02:52 INFO Vacuumed table table=objects44022026/11/12 00:02:52 INFO Received uploads request method=POST path=/44032026/11/12 00:02:52 INFO Received complete upload request method=POST path=/44042026/11/12 00:02:52 INFO Completed upload id=11 objects=7 duration=1.746137ms44052026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=044062026/11/12 00:02:52 INFO Received push request method=POST path=/44072026/11/12 00:02:52 WARN Force mode enabled - objects will be deleted immediately without grace period44082026/11/12 00:02:52 INFO Received push request method=POST path=/44092026/11/12 00:02:52 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044102026/11/12 00:02:52 INFO Vacuumed table table=pending_closures44112026/11/12 00:02:52 INFO Vacuumed table table=pending_objects44122026/11/12 00:02:52 INFO Vacuumed table table=multipart_uploads44132026/11/12 00:02:52 INFO Vacuumed table table=closures44142026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=044152026/11/12 00:02:52 INFO Vacuumed table table=objects44162026/11/12 00:02:52 INFO Received uploads request method=POST path=/44172026/11/12 00:02:52 INFO Received complete upload request method=POST path=/44182026/11/12 00:02:52 WARN Force mode enabled - objects will be deleted immediately without grace period44192026/11/12 00:02:52 INFO Completed upload id=19 objects=6 duration=1.149851ms44202026/11/12 00:02:52 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044212026/11/12 00:02:52 INFO Vacuumed table table=pending_closures44222026/11/12 00:02:52 INFO Vacuumed table table=pending_objects44232026/11/12 00:02:52 INFO Vacuumed table table=multipart_uploads44242026/11/12 00:02:52 INFO Received complete push request method=POST path=/44252026/11/12 00:02:52 INFO Vacuumed table table=closures44262026/11/12 00:02:52 INFO Vacuumed table table=objects44272026/11/12 00:02:52 INFO Completed push id=14 objects=3 duration=1.180191ms44282026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=044292026/11/12 00:02:52 INFO Received uploads request method=POST path=/44302026/11/12 00:02:52 INFO Received complete push request method=POST path=/44312026/11/12 00:02:52 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044322026/11/12 00:02:52 INFO Vacuumed table table=pending_closures44332026/11/12 00:02:52 INFO Received uploads request method=POST path=/44342026/11/12 00:02:52 INFO Completed push id=27 objects=6 duration=3.510334ms44352026/11/12 00:02:52 INFO Vacuumed table table=pending_objects44362026/11/12 00:02:52 INFO Vacuumed table table=multipart_uploads44372026/11/12 00:02:52 INFO Received push request method=POST path=/44382026/11/12 00:02:52 INFO Vacuumed table table=closures44392026/11/12 00:02:52 INFO Vacuumed table table=objects44402026/11/12 00:02:52 INFO Received complete upload request method=POST path=/44412026/11/12 00:02:52 INFO Completed upload id=22 objects=7 duration=1.325832ms44422026/11/12 00:02:52 INFO Received push request method=POST path=/44432026/11/12 00:02:52 INFO Received complete push request method=POST path=/44442026/11/12 00:02:52 INFO Received push request method=POST path=/44452026/11/12 00:02:52 INFO Completed push id=22 objects=3 duration=3.523154ms44462026/11/12 00:02:52 INFO Received uploads request method=POST path=/44472026/11/12 00:02:52 INFO Received complete upload request method=POST path=/44482026/11/12 00:02:52 INFO Completed upload id=20 objects=4 duration=846.868µs44492026/11/12 00:02:52 INFO Received complete upload request method=POST path=/44502026/11/12 00:02:52 INFO Completed upload id=15 objects=2 duration=909.189µs44512026/11/12 00:02:52 INFO Received uploads request method=POST path=/44522026/11/12 00:02:52 INFO Received uploads request method=POST path=/44532026/11/12 00:02:52 INFO Received complete upload request method=POST path=/44542026/11/12 00:02:52 INFO Completed upload id=15 objects=9 duration=1.14439ms44552026/11/12 00:02:52 INFO Received push request method=POST path=/44562026/11/12 00:02:52 INFO Received complete push request method=POST path=/44572026/11/12 00:02:52 INFO Received complete push request method=POST path=/44582026/11/12 00:02:52 INFO Completed push id=24 objects=9 duration=1.990099ms44592026/11/12 00:02:52 INFO Completed push id=16 objects=5 duration=1.127411ms44602026/11/12 00:02:52 INFO Received push request method=POST path=/44612026/11/12 00:02:52 INFO Received complete push request method=POST path=/44622026/11/12 00:02:52 INFO Completed push id=26 objects=14 duration=2.602905ms44632026/11/12 00:02:52 INFO Received uploads request method=POST path=/44642026/11/12 00:02:52 INFO Received push request method=POST path=/44652026/11/12 00:02:52 INFO Received complete push request method=POST path=/44662026/11/12 00:02:52 INFO Received complete push request method=POST path=/44672026/11/12 00:02:52 INFO Completed push id=21 objects=17 duration=2.599605ms44682026/11/12 00:02:52 INFO Completed push id=24 objects=3 duration=1.081711ms44692026/11/12 00:02:52 INFO Received push request method=POST path=/44702026/11/12 00:02:52 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044712026/11/12 00:02:52 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044722026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=044732026/11/12 00:02:52 INFO Vacuumed table table=pending_closures44742026/11/12 00:02:52 INFO Vacuumed table table=pending_closures44752026/11/12 00:02:52 INFO Received push request method=POST path=/44762026/11/12 00:02:52 INFO Vacuumed table table=pending_objects44772026/11/12 00:02:52 INFO Vacuumed table table=multipart_uploads44782026/11/12 00:02:52 INFO Vacuumed table table=pending_objects44792026/11/12 00:02:52 INFO Vacuumed table table=multipart_uploads44802026/11/12 00:02:52 INFO Vacuumed table table=closures44812026/11/12 00:02:52 INFO Vacuumed table table=closures44822026/11/12 00:02:52 INFO Vacuumed table table=objects44832026/11/12 00:02:52 INFO Vacuumed table table=objects44842026/11/12 00:02:52 INFO Received push request method=POST path=/44852026/11/12 00:02:52 INFO Received push request method=POST path=/44862026/11/12 00:02:52 INFO Received complete push request method=POST path=/44872026/11/12 00:02:52 INFO Completed push id=19 objects=7 duration=1.495034ms44882026/11/12 00:02:52 INFO Received complete push request method=POST path=/44892026/11/12 00:02:52 INFO Completed push id=13 objects=15 duration=2.180761ms44902026/11/12 00:02:52 INFO Received uploads request method=POST path=/44912026/11/12 00:02:52 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=044922026/11/12 00:02:52 INFO Received uploads request method=POST path=/44932026/11/12 00:02:52 INFO Vacuumed table table=pending_closures44942026/11/12 00:02:52 INFO Received push request method=POST path=/44952026/11/12 00:02:52 INFO Vacuumed table table=pending_objects44962026/11/12 00:02:52 INFO Vacuumed table table=multipart_uploads44972026/11/12 00:02:52 INFO Vacuumed table table=closures44982026/11/12 00:02:52 INFO Received uploads request method=POST path=/44992026/11/12 00:02:52 INFO Vacuumed table table=objects45002026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=045012026/11/12 00:02:52 INFO Received uploads request method=POST path=/45022026/11/12 00:02:52 INFO Received uploads request method=POST path=/45032026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=045042026/11/12 00:02:52 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."45052026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=045062026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=045072026/11/12 00:02:52 WARN Force mode enabled - objects will be deleted immediately without grace period45082026/11/12 00:02:52 WARN Force mode enabled - objects will be deleted immediately without grace period45092026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=045102026/11/12 00:02:52 INFO Received uploads request method=POST path=/45112026/11/12 00:02:52 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045122026/11/12 00:02:52 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045132026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=045142026/11/12 00:02:52 INFO Vacuumed table table=pending_closures45152026/11/12 00:02:52 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045162026/11/12 00:02:52 INFO Vacuumed table table=pending_closures45172026/11/12 00:02:52 INFO Vacuumed table table=pending_objects45182026/11/12 00:02:52 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045192026/11/12 00:02:52 INFO Vacuumed table table=multipart_uploads45202026/11/12 00:02:52 INFO Vacuumed table table=pending_objects45212026/11/12 00:02:52 WARN Force mode enabled - objects will be deleted immediately without grace period45222026/11/12 00:02:52 INFO Vacuumed table table=closures45232026/11/12 00:02:52 INFO Vacuumed table table=pending_closures45242026/11/12 00:02:52 INFO Vacuumed table table=pending_closures45252026/11/12 00:02:52 INFO Vacuumed table table=multipart_uploads45262026/11/12 00:02:52 INFO Vacuumed table table=objects45272026/11/12 00:02:52 INFO Vacuumed table table=closures45282026/11/12 00:02:52 INFO Vacuumed table table=pending_objects45292026/11/12 00:02:52 INFO Vacuumed table table=pending_objects45302026/11/12 00:02:52 INFO Vacuumed table table=multipart_uploads45312026/11/12 00:02:52 INFO Vacuumed table table=objects45322026/11/12 00:02:52 INFO Vacuumed table table=multipart_uploads45332026/11/12 00:02:52 INFO Vacuumed table table=closures45342026/11/12 00:02:52 INFO Vacuumed table table=objects45352026/11/12 00:02:52 INFO Vacuumed table table=closures45362026/11/12 00:02:52 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045372026/11/12 00:02:52 INFO Vacuumed table table=objects45382026/11/12 00:02:52 INFO Vacuumed table table=pending_closures45392026/11/12 00:02:52 INFO Vacuumed table table=pending_objects45402026/11/12 00:02:52 INFO Vacuumed table table=multipart_uploads45412026/11/12 00:02:52 INFO Vacuumed table table=closures45422026/11/12 00:02:52 INFO Vacuumed table table=objects45432026/11/12 00:02:52 INFO Received complete push request method=POST path=/45442026/11/12 00:02:52 INFO Received complete upload request method=POST path=/45452026/11/12 00:02:52 INFO Completed upload id=26 objects=8 duration=1.035549ms45462026/11/12 00:02:52 INFO Completed push id=16 objects=14 duration=2.722526ms45472026/11/12 00:02:52 INFO Received uploads request method=POST path=/45482026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=045492026/11/12 00:02:52 WARN Force mode enabled - objects will be deleted immediately without grace period45502026/11/12 00:02:52 INFO Received uploads request method=POST path=/45512026/11/12 00:02:52 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045522026/11/12 00:02:52 INFO Vacuumed table table=pending_closures45532026/11/12 00:02:52 INFO Vacuumed table table=pending_objects45542026/11/12 00:02:52 INFO Received complete push request method=POST path=/45552026/11/12 00:02:52 INFO Vacuumed table table=multipart_uploads45562026/11/12 00:02:52 INFO Vacuumed table table=closures45572026/11/12 00:02:52 INFO Vacuumed table table=objects45582026/11/12 00:02:52 INFO Completed push id=20 objects=3 duration=1.855298ms45592026/11/12 00:02:52 INFO Received complete push request method=POST path=/45602026/11/12 00:02:52 INFO Completed push id=28 objects=14 duration=1.180291ms45612026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=045622026/11/12 00:02:52 INFO Received uploads request method=POST path=/45632026/11/12 00:02:52 WARN Force mode enabled - objects will be deleted immediately without grace period45642026/11/12 00:02:52 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045652026/11/12 00:02:52 INFO Vacuumed table table=pending_closures45662026/11/12 00:02:52 INFO Received push request method=POST path=/45672026/11/12 00:02:52 INFO Vacuumed table table=pending_objects45682026/11/12 00:02:52 INFO Vacuumed table table=multipart_uploads45692026/11/12 00:02:52 INFO Vacuumed table table=closures45702026/11/12 00:02:52 INFO Vacuumed table table=objects45712026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=045722026/11/12 00:02:52 INFO Received push request method=POST path=/45732026/11/12 00:02:52 INFO Received push request method=POST path=/45742026/11/12 00:02:52 INFO Received push request method=POST path=/45752026/11/12 00:02:52 INFO Received complete upload request method=POST path=/45762026/11/12 00:02:52 INFO Received complete upload request method=POST path=/45772026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=045782026/11/12 00:02:52 INFO Received complete push request method=POST path=/45792026/11/12 00:02:52 INFO Completed upload id=12 objects=26 duration=2.876408ms45802026/11/12 00:02:52 INFO Completed push id=21 objects=3 duration=1.297232ms45812026/11/12 00:02:52 INFO Completed upload id=17 objects=25 duration=2.597365ms45822026/11/12 00:02:52 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045832026/11/12 00:02:52 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=045842026/11/12 00:02:52 INFO Vacuumed table table=pending_closures45852026/11/12 00:02:52 INFO Vacuumed table table=pending_closures45862026/11/12 00:02:52 INFO Vacuumed table table=pending_objects45872026/11/12 00:02:52 INFO Vacuumed table table=multipart_uploads45882026/11/12 00:02:52 INFO Vacuumed table table=closures45892026/11/12 00:02:52 INFO Vacuumed table table=pending_objects45902026/11/12 00:02:52 INFO Vacuumed table table=objects45912026/11/12 00:02:52 INFO Vacuumed table table=multipart_uploads45922026/11/12 00:02:52 INFO Vacuumed table table=closures45932026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=045942026/11/12 00:02:52 INFO Vacuumed table table=objects45952026/11/12 00:02:52 INFO Received uploads request method=POST path=/45962026/11/12 00:02:52 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045972026/11/12 00:02:52 INFO Vacuumed table table=pending_closures45982026/11/12 00:02:52 INFO Vacuumed table table=pending_objects45992026/11/12 00:02:52 INFO Vacuumed table table=multipart_uploads46002026/11/12 00:02:52 INFO Vacuumed table table=closures46012026/11/12 00:02:52 INFO Vacuumed table table=objects46022026/11/12 00:02:52 INFO Received push request method=POST path=/46032026/11/12 00:02:52 INFO Received push request method=POST path=/46042026/11/12 00:02:52 INFO Received uploads request method=POST path=/46052026/11/12 00:02:52 INFO Received push request method=POST path=/46062026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=046072026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=046082026/11/12 00:02:52 INFO Received push request method=POST path=/46092026/11/12 00:02:52 WARN Force mode enabled - objects will be deleted immediately without grace period46102026/11/12 00:02:52 INFO Received uploads request method=POST path=/46112026/11/12 00:02:52 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046122026/11/12 00:02:52 INFO Vacuumed table table=pending_closures46132026/11/12 00:02:52 INFO Vacuumed table table=pending_objects46142026/11/12 00:02:52 INFO Received uploads request method=POST path=/46152026/11/12 00:02:52 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046162026/11/12 00:02:52 INFO Vacuumed table table=multipart_uploads46172026/11/12 00:02:52 INFO Vacuumed table table=closures46182026/11/12 00:02:52 INFO Vacuumed table table=pending_closures46192026/11/12 00:02:52 INFO Received complete upload request method=POST path=/46202026/11/12 00:02:52 INFO Vacuumed table table=objects46212026/11/12 00:02:52 INFO Vacuumed table table=pending_objects46222026/11/12 00:02:52 INFO Vacuumed table table=multipart_uploads46232026/11/12 00:02:52 INFO Completed upload id=21 objects=2 duration=974.209µs46242026/11/12 00:02:52 INFO Vacuumed table table=closures46252026/11/12 00:02:52 INFO Vacuumed table table=objects46262026/11/12 00:02:52 INFO Received uploads request method=POST path=/46272026/11/12 00:02:52 INFO Received push request method=POST path=/46282026/11/12 00:02:52 INFO Received complete upload request method=POST path=/46292026/11/12 00:02:52 INFO Received complete push request method=POST path=/46302026/11/12 00:02:52 INFO Completed upload id=26 objects=2 duration=1.333753ms46312026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=046322026/11/12 00:02:52 INFO Received uploads request method=POST path=/46332026/11/12 00:02:52 INFO Completed push id=16 objects=8 duration=1.850637ms46342026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=046352026/11/12 00:02:52 INFO Received complete upload request method=POST path=/46362026/11/12 00:02:52 WARN Force mode enabled - objects will be deleted immediately without grace period46372026/11/12 00:02:52 INFO Completed upload id=25 objects=6 duration=1.775357ms46382026/11/12 00:02:52 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046392026/11/12 00:02:52 INFO Vacuumed table table=pending_closures46402026/11/12 00:02:52 INFO Vacuumed table table=pending_objects46412026/11/12 00:02:52 INFO Vacuumed table table=multipart_uploads46422026/11/12 00:02:52 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=3 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046432026/11/12 00:02:52 INFO Received push request method=POST path=/46442026/11/12 00:02:52 INFO Vacuumed table table=pending_closures46452026/11/12 00:02:52 INFO Vacuumed table table=closures46462026/11/12 00:02:52 INFO Received push request method=POST path=/46472026/11/12 00:02:52 INFO Vacuumed table table=objects46482026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=046492026/11/12 00:02:52 INFO Vacuumed table table=pending_objects46502026/11/12 00:02:52 INFO Vacuumed table table=multipart_uploads46512026/11/12 00:02:52 INFO Vacuumed table table=closures46522026/11/12 00:02:52 INFO Vacuumed table table=objects46532026/11/12 00:02:52 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046542026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=046552026/11/12 00:02:52 INFO Vacuumed table table=pending_closures46562026/11/12 00:02:52 INFO Vacuumed table table=pending_objects46572026/11/12 00:02:52 INFO Vacuumed table table=multipart_uploads46582026/11/12 00:02:52 WARN Force mode enabled - objects will be deleted immediately without grace period46592026/11/12 00:02:52 INFO Vacuumed table table=closures46602026/11/12 00:02:52 INFO Vacuumed table table=objects46612026/11/12 00:02:52 INFO Received complete upload request method=POST path=/46622026/11/12 00:02:52 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046632026/11/12 00:02:52 INFO Vacuumed table table=pending_closures46642026/11/12 00:02:52 INFO Completed upload id=21 objects=9 duration=2.605965ms46652026/11/12 00:02:52 INFO Vacuumed table table=pending_objects46662026/11/12 00:02:52 INFO Vacuumed table table=multipart_uploads46672026/11/12 00:02:52 INFO Vacuumed table table=closures46682026/11/12 00:02:52 INFO Received push request method=POST path=/46692026/11/12 00:02:53 INFO Vacuumed table table=objects46702026/11/12 00:02:53 INFO Received complete upload request method=POST path=/46712026/11/12 00:02:53 INFO Completed upload id=22 objects=6 duration=1.081211ms46722026/11/12 00:02:53 INFO Received push request method=POST path=/46732026/11/12 00:02:53 INFO Received push request method=POST path=/46742026/11/12 00:02:53 INFO Received push request method=POST path=/46752026/11/12 00:02:53 INFO Received complete push request method=POST path=/46762026/11/12 00:02:53 INFO Completed push id=25 objects=10 duration=1.722656ms46772026/11/12 00:02:53 INFO Received uploads request method=POST path=/46782026/11/12 00:02:53 INFO Received uploads request method=POST path=/46792026/11/12 00:02:53 INFO Received uploads request method=POST path=/46802026/11/12 00:02:53 INFO Received push request method=POST path=/46812026/11/12 00:02:53 INFO Received push request method=POST path=/46822026/11/12 00:02:53 INFO Received complete upload request method=POST path=/46832026/11/12 00:02:53 INFO Completed upload id=29 objects=5 duration=1.076691ms46842026/11/12 00:02:53 INFO Received push request method=POST path=/46852026/11/12 00:02:53 INFO Received complete push request method=POST path=/46862026/11/12 00:02:53 INFO Completed push id=11 objects=9 duration=1.587195ms46872026/11/12 00:02:53 INFO Received complete push request method=POST path=/46882026/11/12 00:02:53 INFO Completed push id=30 objects=4 duration=3.21721ms46892026/11/12 00:02:53 INFO Received complete upload request method=POST path=/46902026/11/12 00:02:53 INFO Completed upload id=30 objects=2 duration=934.589µs46912026/11/12 00:02:53 INFO Received uploads request method=POST path=/46922026/11/12 00:02:53 INFO Received complete push request method=POST path=/46932026/11/12 00:02:53 INFO Received complete push request method=POST path=/46942026/11/12 00:02:53 INFO Completed push id=27 objects=25 duration=3.22399ms46952026/11/12 00:02:53 INFO Aborted multipart uploads count=0 kept=046962026/11/12 00:02:53 INFO Completed push id=11 objects=6 duration=3.691335ms46972026/11/12 00:02:53 INFO Received uploads request method=POST path=/46982026/11/12 00:02:53 INFO Received complete push request method=POST path=/46992026/11/12 00:02:53 INFO Completed push id=15 objects=24 duration=1.641915ms47002026/11/12 00:02:53 INFO Aborted multipart uploads count=0 kept=047012026/11/12 00:02:53 INFO Received complete push request method=POST path=/47022026/11/12 00:02:53 INFO Completed push id=31 objects=20 duration=1.310853ms47032026/11/12 00:02:53 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047042026/11/12 00:02:53 INFO Vacuumed table table=pending_closures47052026/11/12 00:02:53 INFO Received push request method=POST path=/47062026/11/12 00:02:53 INFO Vacuumed table table=pending_objects47072026/11/12 00:02:53 INFO Vacuumed table table=multipart_uploads47082026/11/12 00:02:53 INFO Vacuumed table table=closures47092026/11/12 00:02:53 INFO Received complete push request method=POST path=/47102026/11/12 00:02:53 INFO Vacuumed table table=objects47112026/11/12 00:02:53 INFO Aborted multipart uploads count=0 kept=047122026/11/12 00:02:53 INFO Completed push id=7 objects=24 duration=4.319822ms47132026/11/12 00:02:53 INFO Received complete upload request method=POST path=/47142026/11/12 00:02:53 INFO Completed upload id=19 objects=2 duration=979.83µs47152026/11/12 00:02:53 INFO Received push request method=POST path=/47162026/11/12 00:02:53 INFO Received complete push request method=POST path=/47172026/11/12 00:02:53 INFO Completed push id=18 objects=6 duration=1.584595ms47182026/11/12 00:02:53 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047192026/11/12 00:02:53 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."47202026/11/12 00:02:53 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."47212026/11/12 00:02:53 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."47222026/11/12 00:02:53 INFO Vacuumed table table=pending_closures47232026/11/12 00:02:53 INFO Vacuumed table table=pending_objects47242026/11/12 00:02:53 INFO Received complete upload request method=POST path=/47252026/11/12 00:02:53 INFO Vacuumed table table=multipart_uploads47262026/11/12 00:02:53 INFO Aborted multipart uploads count=0 kept=047272026/11/12 00:02:53 INFO Vacuumed table table=closures47282026/11/12 00:02:53 INFO Completed upload id=32 objects=5 duration=1.140471ms47292026/11/12 00:02:53 INFO Vacuumed table table=objects47302026/11/12 00:02:53 WARN Force mode enabled - objects will be deleted immediately without grace period47312026/11/12 00:02:53 INFO Received push request method=POST path=/47322026/11/12 00:02:53 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047332026/11/12 00:02:53 INFO Vacuumed table table=pending_closures47342026/11/12 00:02:53 INFO Vacuumed table table=pending_objects47352026/11/12 00:02:53 INFO Vacuumed table table=multipart_uploads47362026/11/12 00:02:53 INFO Vacuumed table table=closures47372026/11/12 00:02:53 INFO Vacuumed table table=objects47382026/11/12 00:02:53 INFO Received complete upload request method=POST path=/47392026/11/12 00:02:53 INFO Completed upload id=24 objects=7 duration=943.509µs47402026/11/12 00:02:53 INFO Received uploads request method=POST path=/47412026/11/12 00:02:53 INFO Received push request method=POST path=/47422026/11/12 00:02:53 INFO Received push request method=POST path=/47432026/11/12 00:02:53 INFO Aborted multipart uploads count=0 kept=047442026/11/12 00:02:53 INFO Received complete upload request method=POST path=/47452026/11/12 00:02:53 INFO Completed upload id=33 objects=3 duration=643.686µs47462026/11/12 00:02:53 INFO Received complete push request method=POST path=/47472026/11/12 00:02:53 INFO Received uploads request method=POST path=/47482026/11/12 00:02:53 INFO Received complete upload request method=POST path=/47492026/11/12 00:02:53 INFO Completed push id=24 objects=5 duration=2.654085ms47502026/11/12 00:02:53 INFO Completed upload id=19 objects=2 duration=2.346903ms47512026/11/12 00:02:53 INFO Received complete push request method=POST path=/47522026/11/12 00:02:53 INFO Completed push id=26 objects=3 duration=965.55µs47532026/11/12 00:02:53 INFO Received complete push request method=POST path=/47542026/11/12 00:02:53 INFO Received uploads request method=POST path=/47552026/11/12 00:02:53 INFO Completed push id=32 objects=7 duration=1.09897ms47562026/11/12 00:02:53 INFO Aborted multipart uploads count=0 kept=047572026/11/12 00:02:53 WARN Force mode enabled - objects will be deleted immediately without grace period47582026/11/12 00:02:53 INFO Received uploads request method=POST path=/47592026/11/12 00:02:53 INFO Received uploads request method=POST path=/47602026/11/12 00:02:53 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047612026/11/12 00:02:53 INFO Vacuumed table table=pending_closures47622026/11/12 00:02:53 INFO Received complete upload request method=POST path=/47632026/11/12 00:02:53 INFO Received push request method=POST path=/47642026/11/12 00:02:53 INFO Completed upload id=28 objects=8 duration=801.748µs47652026/11/12 00:02:53 INFO Vacuumed table table=pending_objects47662026/11/12 00:02:53 INFO Vacuumed table table=multipart_uploads47672026/11/12 00:02:53 INFO Vacuumed table table=closures47682026/11/12 00:02:53 INFO Vacuumed table table=objects47692026/11/12 00:02:53 INFO Received complete push request method=POST path=/47702026/11/12 00:02:53 INFO Completed push id=26 objects=4 duration=957.249µs47712026/11/12 00:02:53 INFO Received uploads request method=POST path=/47722026/11/12 00:02:53 INFO Received uploads request method=POST path=/47732026/11/12 00:02:53 INFO Received complete push request method=POST path=/47742026/11/12 00:02:53 INFO Completed push id=33 objects=7 duration=1.731037ms47752026/11/12 00:02:53 INFO Received uploads request method=POST path=/47762026/11/12 00:02:53 INFO Received complete upload request method=POST path=/47772026/11/12 00:02:53 INFO Received complete push request method=POST path=/47782026/11/12 00:02:53 INFO Completed upload id=29 objects=10 duration=879.829µs47792026/11/12 00:02:53 INFO Completed push id=28 objects=11 duration=2.221721ms47802026/11/12 00:02:53 INFO Received complete upload request method=POST path=/47812026/11/12 00:02:53 INFO Completed upload id=28 objects=13 duration=1.000089ms47822026/11/12 00:02:53 INFO Received complete upload request method=POST path=/47832026/11/12 00:02:53 INFO Received push request method=POST path=/47842026/11/12 00:02:53 INFO Received uploads request method=POST path=/47852026/11/12 00:02:53 INFO Completed upload id=26 objects=3 duration=917.289µs47862026/11/12 00:02:53 INFO Received complete upload request method=POST path=/47872026/11/12 00:02:53 INFO Completed upload id=30 objects=20 duration=853.868µs47882026/11/12 00:02:53 INFO Received complete push request method=POST path=/47892026/11/12 00:02:53 INFO Completed push id=29 objects=3 duration=1.098731ms47902026/11/12 00:02:53 INFO Aborted multipart uploads count=0 kept=047912026/11/12 00:02:53 WARN Force mode enabled - objects will be deleted immediately without grace period47922026/11/12 00:02:53 INFO Received uploads request method=POST path=/47932026/11/12 00:02:53 INFO Received uploads request method=POST path=/47942026/11/12 00:02:53 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047952026/11/12 00:02:53 INFO Received complete push request method=POST path=/47962026/11/12 00:02:53 INFO Completed push id=31 objects=11 duration=925.749µs47972026/11/12 00:02:53 INFO Vacuumed table table=pending_closures47982026/11/12 00:02:53 INFO Vacuumed table table=pending_objects47992026/11/12 00:02:53 INFO Vacuumed table table=multipart_uploads48002026/11/12 00:02:53 INFO Vacuumed table table=closures48012026/11/12 00:02:53 INFO Vacuumed table table=objects48022026/11/12 00:02:53 INFO Received complete upload request method=POST path=/48032026/11/12 00:02:53 INFO Completed upload id=31 objects=2 duration=895.228µs48042026/11/12 00:02:53 INFO Received complete upload request method=POST path=/48052026/11/12 00:02:53 INFO Received complete push request method=POST path=/48062026/11/12 00:02:53 INFO Received push request method=POST path=/48072026/11/12 00:02:53 INFO Completed upload id=32 objects=2 duration=845.548µs48082026/11/12 00:02:53 INFO Completed push id=13 objects=2 duration=1.169911ms48092026/11/12 00:02:53 INFO Received complete push request method=POST path=/48102026/11/12 00:02:53 INFO Completed push id=33 objects=6 duration=916.588µs48112026/11/12 00:02:53 INFO Received complete push request method=POST path=/48122026/11/12 00:02:53 INFO Received complete upload request method=POST path=/48132026/11/12 00:02:53 INFO Completed push id=33 objects=3 duration=1.05723ms48142026/11/12 00:02:53 INFO Completed upload id=27 objects=13 duration=1.415614ms48152026/11/12 00:02:53 INFO Received complete push request method=POST path=/48162026/11/12 00:02:53 INFO Completed push id=21 objects=26 duration=2.13298ms48172026/11/12 00:02:53 INFO Received complete push request method=POST path=/48182026/11/12 00:02:53 INFO Aborted multipart uploads count=0 kept=048192026/11/12 00:02:53 INFO Received complete upload request method=POST path=/48202026/11/12 00:02:53 INFO Completed upload id=34 objects=3 duration=716.947µs48212026/11/12 00:02:53 INFO Aborted multipart uploads count=0 kept=048222026/11/12 00:02:53 INFO Completed push id=29 objects=24 duration=1.906998ms48232026/11/12 00:02:53 WARN Force mode enabled - objects will be deleted immediately without grace period48242026/11/12 00:02:53 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048252026/11/12 00:02:53 INFO Vacuumed table table=pending_closures48262026/11/12 00:02:53 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048272026/11/12 00:02:53 INFO Vacuumed table table=pending_objects48282026/11/12 00:02:53 INFO Vacuumed table table=multipart_uploads48292026/11/12 00:02:53 INFO Vacuumed table table=pending_closures48302026/11/12 00:02:53 INFO Vacuumed table table=closures48312026/11/12 00:02:53 INFO Vacuumed table table=pending_objects48322026/11/12 00:02:53 INFO Vacuumed table table=objects48332026/11/12 00:02:53 INFO Vacuumed table table=multipart_uploads48342026/11/12 00:02:53 INFO Received complete upload request method=POST path=/48352026/11/12 00:02:53 INFO Vacuumed table table=closures48362026/11/12 00:02:53 INFO Received complete push request method=POST path=/48372026/11/12 00:02:53 INFO Vacuumed table table=objects48382026/11/12 00:02:53 INFO Completed upload id=28 objects=17 duration=1.177012ms48392026/11/12 00:02:53 INFO Completed push id=9 objects=9 duration=2.798707ms48402026/11/12 00:02:53 INFO Aborted multipart uploads count=0 kept=048412026/11/12 00:02:53 INFO Aborted multipart uploads count=0 kept=048422026/11/12 00:02:53 WARN Force mode enabled - objects will be deleted immediately without grace period48432026/11/12 00:02:53 WARN Force mode enabled - objects will be deleted immediately without grace period48442026/11/12 00:02:53 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048452026/11/12 00:02:53 INFO Vacuumed table table=pending_closures48462026/11/12 00:02:53 INFO Vacuumed table table=pending_objects48472026/11/12 00:02:53 INFO Vacuumed table table=multipart_uploads48482026/11/12 00:02:53 INFO Vacuumed table table=closures48492026/11/12 00:02:53 INFO Vacuumed table table=objects48502026/11/12 00:02:53 INFO Received complete push request method=POST path=/48512026/11/12 00:02:53 INFO Completed push id=16 objects=7 duration=1.919218ms48522026/11/12 00:02:53 INFO Received uploads request method=POST path=/48532026/11/12 00:02:53 INFO Received complete push request method=POST path=/48542026/11/12 00:02:53 INFO Completed push id=29 objects=2 duration=1.456794ms48552026/11/12 00:02:53 INFO Received complete upload request method=POST path=/48562026/11/12 00:02:53 INFO Completed upload id=21 objects=8 duration=1.289492ms48572026/11/12 00:02:53 INFO Received push request method=POST path=/48582026/11/12 00:02:53 INFO Received complete push request method=POST path=/48592026/11/12 00:02:53 INFO Received complete push request method=POST path=/48602026/11/12 00:02:53 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=048612026/11/12 00:02:53 INFO Completed push id=22 objects=6 duration=1.04879ms48622026/11/12 00:02:53 INFO Completed push id=31 objects=6 duration=1.121011ms48632026/11/12 00:02:53 INFO Vacuumed table table=pending_closures48642026/11/12 00:02:53 INFO Received complete push request method=POST path=/48652026/11/12 00:02:53 INFO Vacuumed table table=pending_objects48662026/11/12 00:02:53 INFO Completed push id=23 objects=6 duration=1.06631ms48672026/11/12 00:02:53 INFO Vacuumed table table=multipart_uploads48682026/11/12 00:02:53 INFO Vacuumed table table=closures48692026/11/12 00:02:53 INFO Vacuumed table table=objects48702026/11/12 00:02:53 INFO Received push request method=POST path=/48712026/11/12 00:02:53 INFO Received complete push request method=POST path=/48722026/11/12 00:02:53 INFO Completed push id=24 objects=8 duration=1.03857ms48732026/11/12 00:02:53 INFO Received complete push request method=POST path=/48742026/11/12 00:02:53 INFO Completed push id=32 objects=9 duration=1.134491ms48752026/11/12 00:02:53 INFO Received complete push request method=POST path=/48762026/11/12 00:02:53 INFO Completed push id=24 objects=6 duration=913.668µs48772026/11/12 00:02:53 INFO Received uploads request method=POST path=/48782026/11/12 00:02:53 INFO Received complete upload request method=POST path=/48792026/11/12 00:02:53 INFO Completed upload id=25 objects=9 duration=970.35µs48802026/11/12 00:02:53 INFO Received complete upload request method=POST path=/48812026/11/12 00:02:53 INFO Completed upload id=33 objects=9 duration=884.488µs48822026/11/12 00:02:53 INFO Received complete push request method=POST path=/48832026/11/12 00:02:53 INFO Received uploads request method=POST path=/48842026/11/12 00:02:53 INFO Completed push id=25 objects=7 duration=1.319652ms48852026/11/12 00:02:53 INFO Received complete upload request method=POST path=/48862026/11/12 00:02:53 INFO Completed upload id=27 objects=3 duration=878.248µs48872026/11/12 00:02:53 INFO Aborted multipart uploads count=0 kept=048882026/11/12 00:02:53 INFO Received complete upload request method=POST path=/48892026/11/12 00:02:53 WARN Force mode enabled - objects will be deleted immediately without grace period48902026/11/12 00:02:53 INFO Received complete upload request method=POST path=/48912026/11/12 00:02:53 INFO Completed upload id=29 objects=4 duration=1.647736ms48922026/11/12 00:02:53 INFO Completed upload id=28 objects=6 duration=858.048µs48932026/11/12 00:02:53 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048942026/11/12 00:02:53 INFO Vacuumed table table=pending_closures48952026/11/12 00:02:53 INFO Vacuumed table table=pending_objects48962026/11/12 00:02:53 INFO Vacuumed table table=multipart_uploads48972026/11/12 00:02:53 INFO Vacuumed table table=closures48982026/11/12 00:02:53 INFO Vacuumed table table=objects48992026/11/12 00:02:53 INFO Received complete push request method=POST path=/49002026/11/12 00:02:53 INFO Completed push id=30 objects=4 duration=936.649µs49012026/11/12 00:02:53 INFO Received uploads request method=POST path=/49022026/11/12 00:02:53 INFO Received complete upload request method=POST path=/49032026/11/12 00:02:53 INFO Completed upload id=26 objects=24 duration=1.400853ms49042026/11/12 00:02:53 INFO Received complete push request method=POST path=/49052026/11/12 00:02:53 INFO Completed push id=31 objects=7 duration=1.06149ms49062026/11/12 00:02:53 INFO Received complete upload request method=POST path=/49072026/11/12 00:02:53 INFO Completed upload id=32 objects=5 duration=821.287µs49082026/11/12 00:02:53 INFO Received uploads request method=POST path=/49092026/11/12 00:02:53 INFO Received complete upload request method=POST path=/49102026/11/12 00:02:53 INFO Received complete push request method=POST path=/49112026/11/12 00:02:53 INFO Completed upload id=27 objects=26 duration=1.08463ms49122026/11/12 00:02:53 INFO Completed push id=11 objects=6 duration=1.664056ms49132026/11/12 00:02:53 INFO Received complete push request method=POST path=/49142026/11/12 00:02:53 INFO Completed push id=28 objects=6 duration=924.489µs49152026/11/12 00:02:53 INFO Received complete push request method=POST path=/49162026/11/12 00:02:53 INFO Completed push id=20 objects=13 duration=1.404614ms49172026/11/12 00:02:53 INFO Received complete upload request method=POST path=/49182026/11/12 00:02:53 INFO Completed upload id=34 objects=24 duration=969.929µs49192026/11/12 00:02:53 INFO Received complete push request method=POST path=/49202026/11/12 00:02:53 INFO Completed push id=29 objects=6 duration=1.172711ms49212026/11/12 00:02:53 INFO Received complete push request method=POST path=/49222026/11/12 00:02:53 INFO Completed push id=22 objects=4 duration=1.564815ms49232026/11/12 00:02:53 INFO Aborted multipart uploads count=0 kept=049242026/11/12 00:02:53 WARN Force mode enabled - objects will be deleted immediately without grace period49252026/11/12 00:02:53 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049262026/11/12 00:02:53 INFO Vacuumed table table=pending_closures49272026/11/12 00:02:53 INFO Vacuumed table table=pending_objects49282026/11/12 00:02:53 INFO Vacuumed table table=multipart_uploads49292026/11/12 00:02:53 INFO Vacuumed table table=closures49302026/11/12 00:02:53 INFO Vacuumed table table=objects49312026/11/12 00:02:53 INFO Received complete upload request method=POST path=/49322026/11/12 00:02:53 INFO Completed upload id=23 objects=7 duration=920.929µs49332026/11/12 00:02:53 INFO Received complete upload request method=POST path=/49342026/11/12 00:02:53 INFO Completed upload id=30 objects=8 duration=2.794126ms49352026/11/12 00:02:53 INFO Received complete push request method=POST path=/49362026/11/12 00:02:53 INFO Completed push id=24 objects=5 duration=886.448µs49372026/11/12 00:02:53 INFO Received complete upload request method=POST path=/49382026/11/12 00:02:53 INFO Completed upload id=31 objects=6 duration=1.214552ms49392026/11/12 00:02:53 INFO Received complete upload request method=POST path=/49402026/11/12 00:02:53 INFO Completed upload id=25 objects=2 duration=827.508µs49412026/11/12 00:02:53 INFO Received complete upload request method=POST path=/49422026/11/12 00:02:53 INFO Completed upload id=32 objects=4 duration=820.948µs49432026/11/12 00:02:53 INFO Received complete upload request method=POST path=/49442026/11/12 00:02:53 INFO Completed upload id=26 objects=20 duration=789.747µs49452026/11/12 00:02:53 INFO Received complete upload request method=POST path=/49462026/11/12 00:02:53 INFO Completed upload id=33 objects=10 duration=1.151351ms49472026/11/12 00:02:53 INFO Received complete push request method=POST path=/49482026/11/12 00:02:53 INFO Completed push id=28 objects=7 duration=1.09099ms49492026/11/12 00:02:53 INFO Received complete upload request method=POST path=/49502026/11/12 00:02:53 INFO Completed upload id=34 objects=23 duration=959.369µs49512026/11/12 00:02:53 INFO Received complete upload request method=POST path=/49522026/11/12 00:02:53 INFO Completed upload id=30 objects=6 duration=1.500975ms49532026/11/12 00:02:53 INFO Received complete push request method=POST path=/49542026/11/12 00:02:53 INFO Completed push id=35 objects=6 duration=1.09653ms49552026/11/12 00:02:53 INFO Received complete push request method=POST path=/49562026/11/12 00:02:53 INFO Completed push id=31 objects=5 duration=978.889µs49572026/11/12 00:02:53 INFO Aborted multipart uploads count=0 kept=049582026/11/12 00:02:53 WARN Force mode enabled - objects will be deleted immediately without grace period49592026/11/12 00:02:53 INFO Received complete push request method=POST path=/49602026/11/12 00:02:53 INFO Completed push id=32 objects=26 duration=1.639536ms49612026/11/12 00:02:53 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=049622026/11/12 00:02:53 INFO Vacuumed table table=pending_closures49632026/11/12 00:02:53 INFO Vacuumed table table=pending_objects49642026/11/12 00:02:53 INFO Vacuumed table table=multipart_uploads49652026/11/12 00:02:53 INFO Vacuumed table table=closures49662026/11/12 00:02:53 INFO Received complete upload request method=POST path=/49672026/11/12 00:02:53 INFO Completed upload id=33 objects=2 duration=788.307µs49682026/11/12 00:02:53 INFO Vacuumed table table=objects49692026/11/12 00:02:53 INFO Received complete upload request method=POST path=/49702026/11/12 00:02:53 INFO Completed upload id=34 objects=5 duration=2.12856ms49712026/11/12 00:02:53 INFO Received complete upload request method=POST path=/49722026/11/12 00:02:53 INFO Completed upload id=35 objects=11 duration=1.467534ms49732026/11/12 00:02:53 INFO Received complete upload request method=POST path=/49742026/11/12 00:02:53 INFO Completed upload id=36 objects=3 duration=890.489µs49752026/11/12 00:02:53 INFO Aborted multipart uploads count=0 kept=049762026/11/12 00:02:53 WARN Force mode enabled - objects will be deleted immediately without grace period49772026/11/12 00:02:53 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049782026/11/12 00:02:53 INFO Vacuumed table table=pending_closures49792026/11/12 00:02:53 INFO Vacuumed table table=pending_objects49802026/11/12 00:02:53 INFO Vacuumed table table=multipart_uploads49812026/11/12 00:02:53 INFO Vacuumed table table=closures49822026/11/12 00:02:53 INFO Vacuumed table table=objects49832026/11/12 00:02:53 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049842026/11/12 00:02:53 INFO Vacuumed table table=pending_closures49852026/11/12 00:02:53 INFO Vacuumed table table=pending_objects49862026/11/12 00:02:53 INFO Vacuumed table table=multipart_uploads49872026/11/12 00:02:53 INFO Vacuumed table table=closures49882026/11/12 00:02:53 INFO Vacuumed table table=objects49892026/11/12 00:02:53 INFO Received complete upload request method=POST path=/49902026/11/12 00:02:53 INFO Completed upload id=20 objects=3 duration=3.632315ms49912026/11/12 00:02:53 INFO Received push request method=POST path=/49922026/11/12 00:02:53 INFO Received complete upload request method=POST path=/49932026/11/12 00:02:53 INFO Completed upload id=24 objects=7 duration=841.608µs49942026/11/12 00:02:53 INFO Aborted multipart uploads count=0 kept=049952026/11/12 00:02:53 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049962026/11/12 00:02:53 INFO Vacuumed table table=pending_closures49972026/11/12 00:02:53 INFO Vacuumed table table=pending_objects49982026/11/12 00:02:53 INFO Vacuumed table table=multipart_uploads49992026/11/12 00:02:53 INFO Vacuumed table table=closures50002026/11/12 00:02:53 INFO Vacuumed table table=objects50012026/11/12 00:02:53 INFO Received uploads request method=POST path=/50022026/11/12 00:02:53 INFO Aborted multipart uploads count=0 kept=050032026/11/12 00:02:53 WARN Force mode enabled - objects will be deleted immediately without grace period50042026/11/12 00:02:53 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=050052026/11/12 00:02:53 INFO Vacuumed table table=pending_closures50062026/11/12 00:02:53 INFO Vacuumed table table=pending_objects50072026/11/12 00:02:53 INFO Vacuumed table table=multipart_uploads50082026/11/12 00:02:53 INFO Vacuumed table table=closures50092026/11/12 00:02:53 INFO Vacuumed table table=objects50102026/11/12 00:02:53 INFO Received complete push request method=POST path=/50112026/11/12 00:02:53 INFO Completed push id=23 objects=28 duration=2.919508ms50122026/11/12 00:02:53 INFO Received complete upload request method=POST path=/50132026/11/12 00:02:53 INFO Completed upload id=27 objects=8 duration=932.409µs50142026/11/12 00:02:53 INFO Received push request method=POST path=/50152026/11/12 00:02:54 INFO Aborted multipart uploads count=0 kept=050162026/11/12 00:02:54 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=050172026/11/12 00:02:54 INFO Vacuumed table table=pending_closures50182026/11/12 00:02:54 INFO Vacuumed table table=pending_objects50192026/11/12 00:02:54 INFO Vacuumed table table=multipart_uploads50202026/11/12 00:02:54 INFO Vacuumed table table=closures50212026/11/12 00:02:54 INFO Vacuumed table table=objects50222026/11/12 00:02:54 INFO Aborted multipart uploads count=0 kept=050232026/11/12 00:02:54 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=050242026/11/12 00:02:54 INFO Vacuumed table table=pending_closures50252026/11/12 00:02:54 INFO Vacuumed table table=pending_objects50262026/11/12 00:02:54 INFO Vacuumed table table=multipart_uploads50272026/11/12 00:02:54 INFO Vacuumed table table=closures50282026/11/12 00:02:54 INFO Vacuumed table table=objects50292026/11/12 00:02:54 INFO Received complete upload request method=POST path=/50302026/11/12 00:02:54 INFO Completed upload id=25 objects=16 duration=1.859918ms50312026/11/12 00:02:54 INFO Aborted multipart uploads count=0 kept=050322026/11/12 00:02:54 WARN Force mode enabled - objects will be deleted immediately without grace period50332026/11/12 00:02:54 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=050342026/11/12 00:02:54 INFO Vacuumed table table=pending_closures50352026/11/12 00:02:54 INFO Vacuumed table table=pending_objects50362026/11/12 00:02:54 INFO Vacuumed table table=multipart_uploads50372026/11/12 00:02:54 INFO Vacuumed table table=closures50382026/11/12 00:02:54 INFO Vacuumed table table=objects50392026/11/12 00:02:54 INFO Received push request method=POST path=/50402026/11/12 00:02:54 INFO Received push request method=POST path=/50412026/11/12 00:02:54 INFO Received uploads request method=POST path=/50422026/11/12 00:02:54 INFO Received complete upload request method=POST path=/50432026/11/12 00:02:54 INFO Completed upload id=22 objects=5 duration=1.670876ms50442026/11/12 00:02:54 INFO Received complete push request method=POST path=/50452026/11/12 00:02:54 INFO Completed push id=28 objects=3 duration=2.538844ms50462026/11/12 00:02:54 INFO Received complete push request method=POST path=/50472026/11/12 00:02:54 INFO Completed push id=29 objects=15 duration=1.10081ms50482026/11/12 00:02:54 INFO Received complete push request method=POST path=/50492026/11/12 00:02:54 INFO Completed push id=30 objects=6 duration=969.349µs50502026/11/12 00:02:54 INFO Received complete upload request method=POST path=/50512026/11/12 00:02:54 INFO Completed upload id=31 objects=3 duration=895.708µs50522026/11/12 00:02:54 INFO Aborted multipart uploads count=0 kept=050532026/11/12 00:02:54 WARN Force mode enabled - objects will be deleted immediately without grace period50542026/11/12 00:02:54 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=050552026/11/12 00:02:54 INFO Vacuumed table table=pending_closures50562026/11/12 00:02:54 INFO Vacuumed table table=pending_objects50572026/11/12 00:02:54 INFO Vacuumed table table=multipart_uploads50582026/11/12 00:02:54 INFO Vacuumed table table=closures50592026/11/12 00:02:54 INFO Vacuumed table table=objects5060--- PASS: FuzzPushGC (0.01s)5061 --- PASS: FuzzPushGC/seed#3 (3.71s)5062 --- PASS: FuzzPushGC/seed#4 (4.47s)5063 --- PASS: FuzzPushGC/seed#10 (4.69s)5064 --- PASS: FuzzPushGC/seed#7 (4.75s)5065 --- PASS: FuzzPushGC/seed#13 (4.78s)5066 --- PASS: FuzzPushGC/seed#8 (4.82s)5067 --- PASS: FuzzPushGC/seed#1 (4.98s)5068 --- PASS: FuzzPushGC/seed#6 (5.43s)5069 --- PASS: FuzzPushGC/seed#14 (5.51s)5070 --- PASS: FuzzPushGC/seed#5 (5.52s)5071 --- PASS: FuzzPushGC/seed#2 (5.55s)5072 --- PASS: FuzzPushGC/seed#0 (5.61s)5073 --- PASS: FuzzPushGC/seed#9 (5.70s)5074 --- PASS: FuzzPushGC/seed#12 (5.92s)5075 --- PASS: FuzzPushGC/seed#11 (6.01s)5076 --- PASS: FuzzPushGC/seed#15 (6.55s)5077PASS50782026-11-12 00:02:54.842 UTC [128] LOG: received smart shutdown request50792026-11-12 00:02:54.851 UTC [128] LOG: background worker "logical replication launcher" (PID 149) exited with exit code 150802026-11-12 00:02:56.652 UTC [144] LOG: checkpoint complete: wrote 3846 buffers (23.5%), wrote 8 SLRU buffers; 0 WAL file(s) added, 0 removed, 33 recycled; write=7.229 s, sync=0.443 s, total=8.905 s; sync files=42370, longest=0.004 s, average=0.001 s; distance=533979 kB, estimate=533979 kB; lsn=0/26F24EF0, redo lsn=0/220DCBA050812026-11-12 00:02:56.674 UTC [144] LOG: shutting down50822026-11-12 00:02:56.675 UTC [144] LOG: checkpoint starting: shutdown immediate50832026-11-12 00:02:57.190 UTC [144] LOG: checkpoint complete: wrote 8203 buffers (50.1%), wrote 2 SLRU buffers; 0 WAL file(s) added, 0 removed, 4 recycled; write=0.171 s, sync=0.335 s, total=0.516 s; sync files=2558, longest=0.003 s, average=0.001 s; distance=80160 kB, estimate=488597 kB; lsn=0/26F24F68, redo lsn=0/26F24F6850842026-11-12 00:02:57.382 UTC [128] LOG: database system is shut down5085Running OIDC tests...5086=== RUN TestAudienceForIssuer5087=== PAUSE TestAudienceForIssuer5088=== RUN TestGlobMatch5089=== PAUSE TestGlobMatch5090=== RUN TestValidateToken_ValidToken5091=== PAUSE TestValidateToken_ValidToken5092=== RUN TestValidateToken_WrongAudience5093=== PAUSE TestValidateToken_WrongAudience5094=== RUN TestValidateToken_Expired5095=== PAUSE TestValidateToken_Expired5096=== RUN TestValidateToken_BoundClaimsMismatch5097=== PAUSE TestValidateToken_BoundClaimsMismatch5098=== RUN TestValidateToken_BoundSubjectMismatch5099=== PAUSE TestValidateToken_BoundSubjectMismatch5100=== RUN TestValidateToken_MultipleProviders5101=== PAUSE TestValidateToken_MultipleProviders5102=== RUN TestValidateToken_NoMatchingProvider5103=== PAUSE TestValidateToken_NoMatchingProvider5104=== RUN TestValidateToken_KubernetesServiceAccount5105=== PAUSE TestValidateToken_KubernetesServiceAccount5106=== RUN TestNewValidator_KubernetesRequiresCA5107=== PAUSE TestNewValidator_KubernetesRequiresCA5108=== RUN TestValidateToken_KubernetesIssuerFromOwnToken5109=== PAUSE TestValidateToken_KubernetesIssuerFromOwnToken5110=== RUN TestPins_ReservedForMatchingRule5111=== PAUSE TestPins_ReservedForMatchingRule5112=== RUN TestPins_TopLevelShorthand5113=== PAUSE TestPins_TopLevelShorthand5114=== RUN TestPins_ConfigValidation5115=== PAUSE TestPins_ConfigValidation5116=== RUN TestScopes_LegacyProviderDefaultsToWrite5117=== PAUSE TestScopes_LegacyProviderDefaultsToWrite5118=== RUN TestScopes_Rules5119=== PAUSE TestScopes_Rules5120=== RUN TestScopes_ConfigValidation5121=== PAUSE TestScopes_ConfigValidation5122=== CONT TestAudienceForIssuer5123--- PASS: TestAudienceForIssuer (0.00s)5124=== CONT TestScopes_ConfigValidation5125--- PASS: TestScopes_ConfigValidation (0.00s)5126=== CONT TestScopes_Rules5127=== CONT TestScopes_LegacyProviderDefaultsToWrite5128=== CONT TestPins_ConfigValidation5129=== CONT TestPins_TopLevelShorthand5130=== CONT TestPins_ReservedForMatchingRule5131--- PASS: TestPins_ConfigValidation (0.01s)5132=== CONT TestValidateToken_KubernetesIssuerFromOwnToken5133=== CONT TestValidateToken_BoundClaimsMismatch5134=== CONT TestValidateToken_WrongAudience5135=== CONT TestValidateToken_ValidToken5136=== CONT TestValidateToken_MultipleProviders5137=== CONT TestValidateToken_KubernetesServiceAccount5138=== CONT TestGlobMatch5139=== RUN TestGlobMatch/foo_foo5140=== PAUSE TestGlobMatch/foo_foo5141=== RUN TestGlobMatch/foo_bar5142=== CONT TestNewValidator_KubernetesRequiresCA5143=== CONT TestValidateToken_BoundSubjectMismatch5144=== CONT TestValidateToken_Expired5145=== CONT TestValidateToken_NoMatchingProvider5146=== PAUSE TestGlobMatch/foo_bar5147=== RUN TestGlobMatch/*_5148=== PAUSE TestGlobMatch/*_5149=== RUN TestGlobMatch/*_anything5150=== PAUSE TestGlobMatch/*_anything5151=== RUN TestGlobMatch/foo*_foo5152=== PAUSE TestGlobMatch/foo*_foo5153=== RUN TestGlobMatch/foo*_foobar5154=== PAUSE TestGlobMatch/foo*_foobar5155=== RUN TestGlobMatch/foo*_bar5156=== PAUSE TestGlobMatch/foo*_bar5157=== RUN TestGlobMatch/*bar_bar5158=== PAUSE TestGlobMatch/*bar_bar5159=== RUN TestGlobMatch/*bar_foobar5160=== PAUSE TestGlobMatch/*bar_foobar5161=== RUN TestGlobMatch/*bar_foo5162=== PAUSE TestGlobMatch/*bar_foo5163=== RUN TestGlobMatch/foo*bar_foobar5164=== PAUSE TestGlobMatch/foo*bar_foobar5165=== RUN TestGlobMatch/foo*bar_foo123bar5166=== PAUSE TestGlobMatch/foo*bar_foo123bar5167=== RUN TestGlobMatch/foo*bar_foobarbaz5168=== PAUSE TestGlobMatch/foo*bar_foobarbaz5169=== RUN TestGlobMatch/*/*_foo/bar5170=== PAUSE TestGlobMatch/*/*_foo/bar5171=== RUN TestGlobMatch/*/*_foo5172=== PAUSE TestGlobMatch/*/*_foo5173=== RUN TestGlobMatch/refs/heads/*_refs/heads/main5174=== PAUSE TestGlobMatch/refs/heads/*_refs/heads/main5175=== RUN TestGlobMatch/refs/heads/*_refs/tags/v1.05176=== PAUSE TestGlobMatch/refs/heads/*_refs/tags/v1.05177=== RUN TestGlobMatch/refs/*/main_refs/heads/main5178=== PAUSE TestGlobMatch/refs/*/main_refs/heads/main5179=== RUN TestGlobMatch/fo?_foo5180=== PAUSE TestGlobMatch/fo?_foo5181=== RUN TestGlobMatch/fo?_fo5182=== PAUSE TestGlobMatch/fo?_fo5183=== RUN TestGlobMatch/fo?_fooo5184=== PAUSE TestGlobMatch/fo?_fooo5185=== RUN TestGlobMatch/?oo_foo5186=== PAUSE TestGlobMatch/?oo_foo5187=== RUN TestGlobMatch/?oo_boo5188=== PAUSE TestGlobMatch/?oo_boo5189=== RUN TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5190=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5191=== RUN TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5192=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5193=== CONT TestGlobMatch/foo_foo5194=== CONT TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5195=== CONT TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5196=== CONT TestGlobMatch/?oo_boo5197=== CONT TestGlobMatch/?oo_foo5198=== CONT TestGlobMatch/fo?_fooo5199=== CONT TestGlobMatch/fo?_fo5200=== CONT TestGlobMatch/fo?_foo5201=== CONT TestGlobMatch/refs/*/main_refs/heads/main5202=== CONT TestGlobMatch/refs/heads/*_refs/tags/v1.05203=== CONT TestGlobMatch/refs/heads/*_refs/heads/main5204=== CONT TestGlobMatch/*/*_foo5205=== CONT TestGlobMatch/*/*_foo/bar5206=== CONT TestGlobMatch/foo*bar_foobarbaz5207=== CONT TestGlobMatch/foo*bar_foo123bar5208=== CONT TestGlobMatch/foo*bar_foobar5209=== CONT TestGlobMatch/*bar_foo5210=== CONT TestGlobMatch/*bar_foobar5211=== CONT TestGlobMatch/*bar_bar5212=== CONT TestGlobMatch/foo*_bar5213=== CONT TestGlobMatch/foo*_foobar5214=== CONT TestGlobMatch/foo*_foo5215=== CONT TestGlobMatch/*_anything5216=== CONT TestGlobMatch/*_5217=== CONT TestGlobMatch/foo_bar5218--- PASS: TestGlobMatch (0.00s)5219 --- PASS: TestGlobMatch/foo_foo (0.00s)5220 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main (0.00s)5221 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main (0.00s)5222 --- PASS: TestGlobMatch/?oo_boo (0.00s)5223 --- PASS: TestGlobMatch/?oo_foo (0.00s)5224 --- PASS: TestGlobMatch/fo?_fooo (0.00s)5225 --- PASS: TestGlobMatch/fo?_fo (0.00s)5226 --- PASS: TestGlobMatch/fo?_foo (0.00s)5227 --- PASS: TestGlobMatch/refs/*/main_refs/heads/main (0.00s)5228 --- PASS: TestGlobMatch/refs/heads/*_refs/tags/v1.0 (0.00s)5229 --- PASS: TestGlobMatch/refs/heads/*_refs/heads/main (0.00s)5230 --- PASS: TestGlobMatch/*/*_foo (0.00s)5231 --- PASS: TestGlobMatch/*/*_foo/bar (0.00s)5232 --- PASS: TestGlobMatch/foo*bar_foobarbaz (0.00s)5233 --- PASS: TestGlobMatch/foo*bar_foo123bar (0.00s)5234 --- PASS: TestGlobMatch/foo*bar_foobar (0.00s)5235 --- PASS: TestGlobMatch/*bar_foo (0.00s)5236 --- PASS: TestGlobMatch/*bar_foobar (0.00s)5237 --- PASS: TestGlobMatch/*bar_bar (0.00s)5238 --- PASS: TestGlobMatch/foo*_bar (0.00s)5239 --- PASS: TestGlobMatch/foo*_foobar (0.00s)5240 --- PASS: TestGlobMatch/foo*_foo (0.00s)5241 --- PASS: TestGlobMatch/*_anything (0.00s)5242 --- PASS: TestGlobMatch/*_ (0.00s)5243 --- PASS: TestGlobMatch/foo_bar (0.00s)52442026/11/12 00:03:02 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:36893/oidc52452026/11/12 00:03:02 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:41419/oidc5246--- PASS: TestPins_ReservedForMatchingRule (0.88s)5247--- PASS: TestScopes_Rules (0.93s)52482026/11/12 00:03:02 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:35475/oidc52492026/11/12 00:03:05 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:33819/oidc5250--- PASS: TestScopes_LegacyProviderDefaultsToWrite (4.08s)52512026/11/12 00:03:05 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:43213/oidc5252--- PASS: TestValidateToken_BoundSubjectMismatch (3.18s)5253--- PASS: TestPins_TopLevelShorthand (4.07s)52542026/11/12 00:03:05 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:42833/oidc5255--- PASS: TestValidateToken_ValidToken (3.19s)52562026/11/12 00:03:05 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:34741/oidc52572026/11/12 00:03:05 http: TLS handshake error from 127.0.0.1:34232: remote error: tls: bad certificate5258--- PASS: TestNewValidator_KubernetesRequiresCA (3.21s)5259--- PASS: TestValidateToken_BoundClaimsMismatch (3.21s)52602026/11/12 00:03:05 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:37409/oidc5261--- PASS: TestValidateToken_Expired (3.23s)52622026/11/12 00:03:05 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:43413/oidc5263--- PASS: TestValidateToken_WrongAudience (3.29s)52642026/11/12 00:03:05 INFO OIDC provider initialized name=provider1 issuer=http://127.0.0.1:41615/oidc5265--- PASS: TestValidateToken_NoMatchingProvider (3.39s)52662026/11/12 00:03:05 INFO OIDC provider initialized name=kubernetes issuer=https://127.0.0.1:449695267--- PASS: TestValidateToken_KubernetesServiceAccount (3.41s)52682026/11/12 00:03:05 INFO OIDC provider initialized name=provider1 issuer=http://127.0.0.1:39135/oidc52692026/11/12 00:03:05 INFO OIDC provider initialized name=provider2 issuer=http://127.0.0.1:36191/oidc5270--- PASS: TestValidateToken_MultipleProviders (3.46s)52712026/11/12 00:03:05 INFO OIDC provider initialized name=kubernetes issuer=https://oidc.eks.invalid/id/ABC1235272--- PASS: TestValidateToken_KubernetesIssuerFromOwnToken (4.42s)5273PASS5274Running hook tests...5275=== RUN TestSendPathsEmpty5276=== PAUSE TestSendPathsEmpty5277=== RUN TestQueueEnqueueAndFetch5278=== PAUSE TestQueueEnqueueAndFetch5279=== RUN TestQueueDeduplication5280=== PAUSE TestQueueDeduplication5281=== RUN TestQueueRemove5282=== PAUSE TestQueueRemove5283=== RUN TestQueueFetchBatchLimit5284=== PAUSE TestQueueFetchBatchLimit5285=== RUN TestQueueRetryMovesToBack5286=== PAUSE TestQueueRetryMovesToBack5287=== RUN TestQueueFetchRemoveLifecycle5288=== PAUSE TestQueueFetchRemoveLifecycle5289=== RUN TestQueueConcurrentWriters5290=== PAUSE TestQueueConcurrentWriters5291=== RUN TestQueueRemoveLargeClosure5292=== PAUSE TestQueueRemoveLargeClosure5293=== RUN TestServerClientIntegration5294--- PASS: TestServerClientIntegration (0.00s)5295=== RUN TestServerQueueError52962026/11/12 00:03:08 ERROR Failed to queue paths error="permission denied" count=15297--- PASS: TestServerQueueError (0.00s)5298=== RUN TestGetListenerSocketActivation5299 server_test.go:220: === RUN TestGetListenerSocketActivation5300 --- PASS: TestGetListenerSocketActivation (0.00s)5301 PASS5302 5303--- PASS: TestGetListenerSocketActivation (3.50s)5304=== RUN TestServerStalledClientDoesNotBlockShutdown5305=== PAUSE TestServerStalledClientDoesNotBlockShutdown5306=== RUN TestServerBacksOffOnAcceptErrors5307=== PAUSE TestServerBacksOffOnAcceptErrors5308=== RUN TestDrainIsolatesPoisonPath5309=== PAUSE TestDrainIsolatesPoisonPath5310=== RUN TestRunNotBlockedByPoisonHead5311=== PAUSE TestRunNotBlockedByPoisonHead5312=== RUN TestDrainGivesUpWhenServerDown5313=== PAUSE TestDrainGivesUpWhenServerDown5314=== RUN TestFailedPathPrunedByLaterClosure5315=== PAUSE TestFailedPathPrunedByLaterClosure5316=== RUN TestWorkerUploadsAndRemoves5317=== PAUSE TestWorkerUploadsAndRemoves5318=== RUN TestWorkerSkipsGCdPaths5319=== PAUSE TestWorkerSkipsGCdPaths5320=== RUN TestWorkerPrunesClosureDeps5321=== PAUSE TestWorkerPrunesClosureDeps5322=== RUN TestDrainTimeout5323=== PAUSE TestDrainTimeout5324=== RUN TestWorkerRemoveFailureIsNotProgress5325=== PAUSE TestWorkerRemoveFailureIsNotProgress5326=== RUN TestWorkerKeepsPathItCannotStat5327=== PAUSE TestWorkerKeepsPathItCannotStat5328=== CONT TestSendPathsEmpty5329--- PASS: TestSendPathsEmpty (0.00s)5330=== CONT TestServerStalledClientDoesNotBlockShutdown5331=== CONT TestQueueRetryMovesToBack5332=== CONT TestQueueDeduplication5333=== CONT TestWorkerUploadsAndRemoves5334=== CONT TestWorkerRemoveFailureIsNotProgress5335=== CONT TestDrainTimeout5336=== CONT TestWorkerPrunesClosureDeps5337=== CONT TestWorkerSkipsGCdPaths5338=== CONT TestDrainGivesUpWhenServerDown5339=== CONT TestFailedPathPrunedByLaterClosure5340=== CONT TestQueueConcurrentWriters5341=== CONT TestQueueRemoveLargeClosure5342=== CONT TestRunNotBlockedByPoisonHead5343=== CONT TestQueueRemove5344=== CONT TestQueueFetchBatchLimit5345=== CONT TestDrainIsolatesPoisonPath5346=== CONT TestQueueFetchRemoveLifecycle5347=== CONT TestQueueEnqueueAndFetch5348=== CONT TestWorkerKeepsPathItCannotStat5349=== CONT TestServerBacksOffOnAcceptErrors53502000/01/01 00:00:00 ERROR Accept failed error="too many open files"53512000/01/01 00:00:00 ERROR Accept failed error="too many open files"53522000/01/01 00:00:00 ERROR Accept failed error="too many open files"53532000/01/01 00:00:00 ERROR Accept failed error="too many open files"53542000/01/01 00:00:00 ERROR Accept failed error="too many open files"53552000/01/01 00:00:00 ERROR Accept failed error="too many open files"53562000/01/01 00:00:00 ERROR Accept failed error="too many open files"53572000/01/01 00:00:00 ERROR Accept failed error="too many open files"5358--- PASS: TestServerBacksOffOnAcceptErrors (0.00s)53592000/01/01 00:00:00 INFO Uploading batch count=253602000/01/01 00:00:00 ERROR Upload failed error="context deadline exceeded" count=253612026/11/12 00:03:11 INFO Upload queue status pending=253622000/01/01 00:00:00 ERROR Drain finished with paths left in queue remaining=453632026/11/12 00:03:11 INFO Uploading batch count=253642026/11/12 00:03:11 INFO Uploading batch count=153652026/11/12 00:03:11 ERROR Upload failed error="upload failed" count=15366--- PASS: TestQueueDeduplication (0.03s)5367--- PASS: TestDrainTimeout (0.03s)5368--- PASS: TestQueueRemove (0.03s)53692026/11/12 00:03:11 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat1988536315/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat1988536315/002/locked/aaa: permission denied"53702026/11/12 00:03:11 INFO Uploading batch count=253712026/11/12 00:03:11 ERROR Upload failed error="upload failed" count=253722026/11/12 00:03:11 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown1609915945/002/a53732026/11/12 00:03:11 INFO Upload queue status pending=253742026/11/12 00:03:11 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat1988536315/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat1988536315/002/locked/aaa: permission denied"53752026/11/12 00:03:11 INFO Upload queue status pending=253762026/11/12 00:03:11 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerSkipsGCdPaths933769633/002/nonexistent53772026/11/12 00:03:11 INFO Uploading batch count=153782026/11/12 00:03:11 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown1609915945/002/b53792026/11/12 00:03:11 INFO Uploading batch count=153802026/11/12 00:03:11 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat1988536315/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat1988536315/002/locked/aaa: permission denied"53812026/11/12 00:03:11 ERROR Drain finished with paths left in queue remaining=153822026/11/12 00:03:11 INFO Uploading batch count=253832026/11/12 00:03:11 ERROR Upload failed error="upload failed" count=253842026/11/12 00:03:11 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown1609915945/002/c5385--- PASS: TestQueueFetchRemoveLifecycle (0.04s)5386--- PASS: TestQueueRetryMovesToBack (0.04s)53872026/11/12 00:03:11 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown1609915945/002/d53882026/11/12 00:03:11 INFO Uploading batch count=453892026/11/12 00:03:11 ERROR Upload failed error="upload failed" count=453902026/11/12 00:03:11 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainIsolatesPoisonPath681516249/002/bbb5391--- PASS: TestWorkerKeepsPathItCannotStat (0.04s)53922026/11/12 00:03:11 INFO Uploading batch count=15393--- PASS: TestQueueFetchBatchLimit (0.04s)53942026/11/12 00:03:11 INFO Uploading batch count=153952026/11/12 00:03:11 INFO Uploading batch count=253962026/11/12 00:03:11 ERROR Upload failed error="upload failed" count=253972026/11/12 00:03:11 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown1609915945/002/e53982026/11/12 00:03:11 INFO Upload queue status pending=353992026/11/12 00:03:11 INFO Uploading batch count=154002026/11/12 00:03:11 ERROR Upload failed error="upload failed" count=154012026/11/12 00:03:11 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown1609915945/002/f5402--- PASS: TestFailedPathPrunedByLaterClosure (0.04s)54032026/11/12 00:03:11 ERROR Drain finished with paths left in queue remaining=105404--- PASS: TestWorkerUploadsAndRemoves (0.05s)5405--- PASS: TestDrainGivesUpWhenServerDown (0.04s)5406--- PASS: TestWorkerSkipsGCdPaths (0.05s)5407--- PASS: TestWorkerPrunesClosureDeps (0.05s)54082026/11/12 00:03:12 ERROR Failed to decode request error="read unix /build/hook3331414052/test.sock->@: i/o timeout"54092026/11/12 00:03:12 ERROR Failed to write response error="write unix /build/hook3331414052/test.sock->@: i/o timeout"5410--- PASS: TestServerStalledClientDoesNotBlockShutdown (0.20s)54112026/11/12 00:03:12 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress2244777836/002/nonexistent5412--- PASS: TestQueueEnqueueAndFetch (0.70s)54132026/11/12 00:03:12 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=154142026/11/12 00:03:12 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress2244777836/002/nonexistent54152026/11/12 00:03:12 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=154162026/11/12 00:03:12 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress2244777836/002/nonexistent54172026/11/12 00:03:12 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=154182026/11/12 00:03:12 ERROR Drain finished with paths left in queue remaining=154192026/11/12 00:03:12 INFO Uploading batch count=154202026/11/12 00:03:12 ERROR Upload failed error="upload failed" count=154212026/11/12 00:03:12 INFO Uploading batch count=154222026/11/12 00:03:12 ERROR Upload failed error="upload failed" count=15423--- PASS: TestWorkerRemoveFailureIsNotProgress (0.70s)54242026/11/12 00:03:12 INFO Uploading batch count=154252026/11/12 00:03:12 ERROR Upload failed error="upload failed" count=154262026/11/12 00:03:12 ERROR Drain finished with paths left in queue remaining=15427--- PASS: TestDrainIsolatesPoisonPath (0.70s)5428--- PASS: TestQueueRemoveLargeClosure (0.79s)5429--- PASS: TestQueueConcurrentWriters (0.81s)54302026/11/12 00:03:13 INFO Uploading batch count=154312026/11/12 00:03:13 INFO Uploading batch count=154322026/11/12 00:03:13 INFO Uploading batch count=154332026/11/12 00:03:13 ERROR Upload failed error="upload failed" count=154342026/11/12 00:03:13 INFO Uploading batch count=154352026/11/12 00:03:13 ERROR Upload failed error="upload failed" count=154362026/11/12 00:03:13 INFO Uploading batch count=154372026/11/12 00:03:13 ERROR Upload failed error="upload failed" count=154382026/11/12 00:03:13 INFO Uploading batch count=154392026/11/12 00:03:13 ERROR Upload failed error="upload failed" count=154402026/11/12 00:03:13 ERROR Drain finished with paths left in queue remaining=15441--- PASS: TestRunNotBlockedByPoisonHead (1.72s)5442PASS5443Running signing tests...5444=== RUN TestExternalSigner5445=== PAUSE TestExternalSigner5446=== RUN TestExternalSignerWrongPublicKey5447=== PAUSE TestExternalSignerWrongPublicKey5448=== RUN TestExternalSignerErrors5449=== PAUSE TestExternalSignerErrors5450=== RUN TestExternalSignerRestartsAfterCrash5451=== PAUSE TestExternalSignerRestartsAfterCrash5452=== RUN TestExternalSignerProgramErrorKeepsProcess5453=== PAUSE TestExternalSignerProgramErrorKeepsProcess5454=== RUN TestExternalSignerClose5455=== PAUSE TestExternalSignerClose5456=== RUN TestExternalSignerRejectsInvalidPublicKey5457=== PAUSE TestExternalSignerRejectsInvalidPublicKey5458=== RUN TestExternalSignerCancellation5459=== PAUSE TestExternalSignerCancellation5460=== RUN TestGenerateFingerprint5461=== PAUSE TestGenerateFingerprint5462=== RUN TestParseSigningKey5463=== PAUSE TestParseSigningKey5464=== RUN TestSign5465=== PAUSE TestSign5466=== RUN TestSignBatch5467=== PAUSE TestSignBatch5468=== RUN TestNilKeyPublicKey5469=== PAUSE TestNilKeyPublicKey5470=== RUN TestSignErrors5471=== PAUSE TestSignErrors5472=== CONT TestExternalSigner5473=== CONT TestExternalSignerCancellation5474=== CONT TestExternalSignerProgramErrorKeepsProcess5475=== CONT TestExternalSignerRejectsInvalidPublicKey5476=== RUN TestExternalSignerRejectsInvalidPublicKey/#005477=== PAUSE TestExternalSignerRejectsInvalidPublicKey/#005478=== RUN TestExternalSignerRejectsInvalidPublicKey/no-colon5479=== PAUSE TestExternalSignerRejectsInvalidPublicKey/no-colon5480=== RUN TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5481=== CONT TestExternalSignerClose5482=== PAUSE TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5483=== CONT TestSignBatch5484=== CONT TestSignErrors5485=== CONT TestNilKeyPublicKey5486--- PASS: TestNilKeyPublicKey (0.00s)5487=== CONT TestParseSigningKey5488=== RUN TestParseSigningKey/valid_32-byte_key5489=== PAUSE TestParseSigningKey/valid_32-byte_key5490=== RUN TestParseSigningKey/valid_32-byte_key_with_different_name5491=== PAUSE TestParseSigningKey/valid_32-byte_key_with_different_name5492=== RUN TestParseSigningKey/no_colon5493=== PAUSE TestParseSigningKey/no_colon5494=== CONT TestSign5495=== CONT TestGenerateFingerprint5496=== CONT TestExternalSignerErrors5497=== CONT TestExternalSignerRestartsAfterCrash5498=== CONT TestExternalSignerWrongPublicKey5499=== RUN TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5500=== PAUSE TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5501=== RUN TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5502=== PAUSE TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5503=== CONT TestExternalSignerRejectsInvalidPublicKey/#005504=== RUN TestParseSigningKey/empty_name5505=== PAUSE TestParseSigningKey/empty_name5506=== RUN TestParseSigningKey/invalid_base645507=== PAUSE TestParseSigningKey/invalid_base645508=== RUN TestParseSigningKey/wrong_length5509=== PAUSE TestParseSigningKey/wrong_length5510=== CONT TestParseSigningKey/valid_32-byte_key5511=== RUN TestGenerateFingerprint/basic_with_references5512=== PAUSE TestGenerateFingerprint/basic_with_references5513=== RUN TestGenerateFingerprint/no_references5514=== PAUSE TestGenerateFingerprint/no_references5515=== RUN TestGenerateFingerprint/unsorted_references_get_sorted5516=== PAUSE TestGenerateFingerprint/unsorted_references_get_sorted5517=== RUN TestGenerateFingerprint/invalid_nar_hash_prefix5518=== PAUSE TestGenerateFingerprint/invalid_nar_hash_prefix5519=== RUN TestGenerateFingerprint/invalid_nar_hash_length5520=== PAUSE TestGenerateFingerprint/invalid_nar_hash_length5521=== RUN TestGenerateFingerprint/invalid_store_path_prefix5522=== PAUSE TestGenerateFingerprint/invalid_store_path_prefix5523=== RUN TestGenerateFingerprint/invalid_reference_prefix5524=== PAUSE TestGenerateFingerprint/invalid_reference_prefix5525=== CONT TestGenerateFingerprint/basic_with_references5526=== CONT TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5527=== RUN TestExternalSignerErrors/null_signatures5528=== PAUSE TestExternalSignerErrors/null_signatures5529=== RUN TestExternalSignerErrors/wrong_signatures_shape5530=== PAUSE TestExternalSignerErrors/wrong_signatures_shape5531=== RUN TestExternalSignerErrors/unrequested_key5532=== PAUSE TestExternalSignerErrors/unrequested_key5533=== RUN TestExternalSignerErrors/missing_signature_name5534=== PAUSE TestExternalSignerErrors/missing_signature_name5535=== RUN TestExternalSignerErrors/wrong_shape5536=== PAUSE TestExternalSignerErrors/wrong_shape5537=== RUN TestExternalSignerErrors/wrong_id5538=== PAUSE TestExternalSignerErrors/wrong_id5539=== RUN TestExternalSignerErrors/missing_signatures5540=== PAUSE TestExternalSignerErrors/missing_signatures5541=== RUN TestExternalSignerErrors/too_few_signatures5542=== PAUSE TestExternalSignerErrors/too_few_signatures5543=== RUN TestExternalSignerErrors/invalid_signature_base645544=== PAUSE TestExternalSignerErrors/invalid_signature_base645545=== RUN TestExternalSignerErrors/program_error5546=== PAUSE TestExternalSignerErrors/program_error5547=== RUN TestExternalSignerErrors/invalid_JSON5548=== PAUSE TestExternalSignerErrors/invalid_JSON5549=== RUN TestExternalSignerErrors/wrong_signature_length5550=== PAUSE TestExternalSignerErrors/wrong_signature_length5551=== RUN TestExternalSignerErrors/exit_on_request5552=== PAUSE TestExternalSignerErrors/exit_on_request5553=== RUN TestExternalSignerErrors/wrong_signature_name5554=== PAUSE TestExternalSignerErrors/wrong_signature_name5555=== RUN TestExternalSignerErrors/exit_at_startup5556=== PAUSE TestExternalSignerErrors/exit_at_startup5557=== RUN TestExternalSignerErrors/null5558=== PAUSE TestExternalSignerErrors/null5559=== CONT TestExternalSignerErrors/null_signatures5560=== CONT TestParseSigningKey/empty_name5561=== RUN TestSignErrors/nil_key5562=== PAUSE TestSignErrors/nil_key5563=== RUN TestSignErrors/nil_narinfo5564=== CONT TestExternalSignerErrors/null5565=== CONT TestExternalSignerErrors/wrong_signature_name5566--- PASS: TestSign (0.00s)5567=== CONT TestParseSigningKey/invalid_base645568=== CONT TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5569=== CONT TestExternalSignerErrors/exit_on_request5570=== CONT TestParseSigningKey/no_colon5571=== CONT TestExternalSignerErrors/wrong_signature_length5572=== CONT TestParseSigningKey/valid_32-byte_key_with_different_name5573=== CONT TestGenerateFingerprint/invalid_reference_prefix5574=== CONT TestExternalSignerErrors/program_error5575--- PASS: TestSignBatch (0.00s)5576=== CONT TestExternalSignerErrors/invalid_signature_base645577=== CONT TestGenerateFingerprint/invalid_store_path_prefix5578=== CONT TestExternalSignerErrors/wrong_shape5579=== CONT TestGenerateFingerprint/invalid_nar_hash_length5580=== CONT TestExternalSignerErrors/missing_signatures5581=== CONT TestGenerateFingerprint/invalid_nar_hash_prefix5582=== CONT TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5583=== CONT TestGenerateFingerprint/unsorted_references_get_sorted5584=== CONT TestGenerateFingerprint/no_references5585=== CONT TestExternalSignerRejectsInvalidPublicKey/no-colon5586=== CONT TestExternalSignerErrors/exit_at_startup5587=== PAUSE TestSignErrors/nil_narinfo5588=== CONT TestExternalSignerErrors/too_few_signatures5589=== CONT TestExternalSignerErrors/missing_signature_name5590--- PASS: TestGenerateFingerprint (0.00s)5591 --- PASS: TestGenerateFingerprint/basic_with_references (0.00s)5592 --- PASS: TestGenerateFingerprint/invalid_reference_prefix (0.00s)5593 --- PASS: TestGenerateFingerprint/invalid_store_path_prefix (0.00s)5594 --- PASS: TestGenerateFingerprint/invalid_nar_hash_length (0.00s)5595 --- PASS: TestGenerateFingerprint/invalid_nar_hash_prefix (0.00s)5596 --- PASS: TestGenerateFingerprint/no_references (0.00s)5597 --- PASS: TestGenerateFingerprint/unsorted_references_get_sorted (0.00s)5598=== CONT TestExternalSignerErrors/unrequested_key5599--- PASS: TestExternalSignerRejectsInvalidPublicKey (0.00s)5600 --- PASS: TestExternalSignerRejectsInvalidPublicKey/#00 (0.00s)5601 --- PASS: TestExternalSignerRejectsInvalidPublicKey/test-key:invalid! (0.00s)5602 --- PASS: TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= (0.00s)5603 --- PASS: TestExternalSignerRejectsInvalidPublicKey/test-key:YQ== (0.00s)5604 --- PASS: TestExternalSignerRejectsInvalidPublicKey/no-colon (0.00s)5605=== CONT TestParseSigningKey/wrong_length5606=== RUN TestSignErrors/invalid_narinfo5607=== PAUSE TestSignErrors/invalid_narinfo5608=== CONT TestExternalSignerErrors/wrong_id5609=== RUN TestSignErrors/canceled5610--- PASS: TestParseSigningKey (0.00s)5611 --- PASS: TestParseSigningKey/empty_name (0.00s)5612 --- PASS: TestParseSigningKey/valid_32-byte_key (0.00s)5613 --- PASS: TestParseSigningKey/invalid_base64 (0.00s)5614 --- PASS: TestParseSigningKey/no_colon (0.00s)5615 --- PASS: TestParseSigningKey/valid_32-byte_key_with_different_name (0.00s)5616 --- PASS: TestParseSigningKey/wrong_length (0.00s)5617=== CONT TestExternalSignerErrors/invalid_JSON5618=== PAUSE TestSignErrors/canceled5619=== CONT TestExternalSignerErrors/wrong_signatures_shape5620=== CONT TestSignErrors/nil_key5621=== CONT TestSignErrors/invalid_narinfo5622=== CONT TestSignErrors/canceled5623=== CONT TestSignErrors/nil_narinfo5624--- PASS: TestSignErrors (0.01s)5625 --- PASS: TestSignErrors/nil_key (0.00s)5626 --- PASS: TestSignErrors/canceled (0.00s)5627 --- PASS: TestSignErrors/invalid_narinfo (0.00s)5628 --- PASS: TestSignErrors/nil_narinfo (0.00s)5629--- PASS: TestExternalSignerClose (0.01s)5630--- PASS: TestExternalSignerProgramErrorKeepsProcess (0.01s)5631--- PASS: TestExternalSignerCancellation (0.02s)5632--- PASS: TestExternalSignerRestartsAfterCrash (0.02s)5633--- PASS: TestExternalSignerWrongPublicKey (0.02s)5634--- PASS: TestExternalSigner (0.02s)5635--- PASS: TestExternalSignerErrors (0.00s)5636 --- PASS: TestExternalSignerErrors/program_error (0.01s)5637 --- PASS: TestExternalSignerErrors/null_signatures (0.01s)5638 --- PASS: TestExternalSignerErrors/null (0.01s)5639 --- PASS: TestExternalSignerErrors/wrong_signature_name (0.01s)5640 --- PASS: TestExternalSignerErrors/exit_at_startup (0.01s)5641 --- PASS: TestExternalSignerErrors/unrequested_key (0.01s)5642 --- PASS: TestExternalSignerErrors/wrong_signature_length (0.02s)5643 --- PASS: TestExternalSignerErrors/missing_signatures (0.02s)5644 --- PASS: TestExternalSignerErrors/invalid_signature_base64 (0.02s)5645 --- PASS: TestExternalSignerErrors/wrong_shape (0.02s)5646 --- PASS: TestExternalSignerErrors/missing_signature_name (0.02s)5647 --- PASS: TestExternalSignerErrors/wrong_id (0.01s)5648 --- PASS: TestExternalSignerErrors/wrong_signatures_shape (0.01s)5649 --- PASS: TestExternalSignerErrors/exit_on_request (0.02s)5650 --- PASS: TestExternalSignerErrors/too_few_signatures (0.02s)5651 --- PASS: TestExternalSignerErrors/invalid_JSON (0.01s)5652PASS5653Running OpenBao signer tests...5654=== RUN TestSign5655=== PAUSE TestSign5656=== RUN TestSignErrors5657=== PAUSE TestSignErrors5658=== RUN TestServe5659=== PAUSE TestServe5660=== RUN TestServeRejectsGarbage5661=== PAUSE TestServeRejectsGarbage5662=== CONT TestSign5663=== CONT TestServeRejectsGarbage5664--- PASS: TestServeRejectsGarbage (0.00s)5665=== CONT TestServe5666--- PASS: TestServe (0.00s)5667=== CONT TestSignErrors5668=== RUN TestSignErrors/http_error5669=== PAUSE TestSignErrors/http_error5670=== RUN TestSignErrors/invalid_json5671=== PAUSE TestSignErrors/invalid_json5672=== RUN TestSignErrors/no_prefix5673=== PAUSE TestSignErrors/no_prefix5674=== RUN TestSignErrors/invalid_base645675=== PAUSE TestSignErrors/invalid_base645676=== CONT TestSignErrors/http_error5677=== CONT TestSignErrors/invalid_base645678=== CONT TestSignErrors/no_prefix5679=== CONT TestSignErrors/invalid_json5680--- PASS: TestSign (0.78s)5681--- PASS: TestSignErrors (0.00s)5682 --- PASS: TestSignErrors/invalid_base64 (0.77s)5683 --- PASS: TestSignErrors/invalid_json (0.77s)5684 --- PASS: TestSignErrors/http_error (0.77s)5685 --- PASS: TestSignErrors/no_prefix (0.77s)5686PASS